Marcelo Freitas Caetano
email: caetano@ircam.fr

Xavier Rodet
email: rodet@ircam.fr

INDEPENDENT MANIPULATION OF HIGH-LEVEL SPECTRAL ENVELOPE SHAPE FEATURES FOR SOUND MORPHING BY MEANS OF EVOLUTIONARY COMPUTATION

d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

There is a burgeoning interest in the search for computational techniques that allow the user to obtain perceptually relevant sound transformations and seamless transitions because computer sound manipulations are widespread in audio applications. Feature-based synthesis is a promising candidate to attain perceptually relevant sound transformations when the features closely capture salient perceptual dimensions [START_REF] Verfaille | Adaptive Digital Audio Effects (a-DAFx): A New Class of Sound Transformations[END_REF]. Among the many different possible transformations [START_REF] Amatriain | Content-Based Transformations[END_REF], we will focus specifically on morphing acoustic musical instrument sounds. There seems to be no consensus on what sound morphing is. Most authors seem to agree that morphing involves the hybridization of two (or more) sounds by blending auditory features. One frequent requirement is that the result should fuse into a single percept, somewhat ruling out simply mixing the sources [START_REF] Fitz | Cell-Utes and Flutter-Tongued Cats: Sound Morphing Using Loris and the Reassigned Bandwidth-Enhanced Model[END_REF], [START_REF] Williams | Perceptually-Motivated Audio Morphing: Softness[END_REF], because the ear is still usually capable of distinguishing them due to a number of cues and auditory processes. Although many different methods are described as morphing [START_REF] Caetano | Automatic Timbral Morphing of Musical Instrument Sounds by High-Level Descriptors, to appear[END_REF], the result is usually associated with what many authors describe as timbre interpolation [START_REF] Williams | Perceptually-Motivated Audio Morphing: Softness[END_REF], [START_REF] Tellman | Timbre Morphing of Sounds with Unequal Numbers of Features[END_REF], [START_REF] Slaney | Automatic Audio Morphing[END_REF]. The goal of timbre interpolation is to obtain a hybrid sound that is perceived to come from more than one source at the same time. For instance, we could seek to get a hybrid between a violin and a trumpet sound. We should bear in mind that this definition of morphing supposes that timbre is the perceptual phenomenon responsible for sound source identification [START_REF] Handel | Timbre perception and auditory object identification[END_REF], somewhat ignoring that the same sound can present timbral variations related to the perceptual dimensions of timbre unveiled by psychoacoustic experiments [START_REF] Mcadams | Perceptual Scaling of Synthesized Musical Timbres: Common Dimensions, Specifities and latent subject Classes[END_REF]. A crescendo trumpet note, for example, becomes perceptually brighter as it gets louder.

Most morphing techniques described in the literature consist in interpolating the parameters of a model used to represent both sounds we wish to morph between, regardless of features [START_REF] Fitz | Cell-Utes and Flutter-Tongued Cats: Sound Morphing Using Loris and the Reassigned Bandwidth-Enhanced Model[END_REF], [START_REF] Tellman | Timbre Morphing of Sounds with Unequal Numbers of Features[END_REF] [7], [START_REF] Boccardi | Sound Morphing with Gaussian Mixture Models[END_REF], [START_REF] Ezzat | Morphing Spectral Envelopes using Audio Flow[END_REF], [START_REF] Ahmad | Morphing of transient sounds based on shift-invariant discrete wavelet transform and singular value decomposition[END_REF]. These techniques usually aim at obtaining a sound with an intermediate timbre [START_REF] Fitz | Cell-Utes and Flutter-Tongued Cats: Sound Morphing Using Loris and the Reassigned Bandwidth-Enhanced Model[END_REF], [START_REF] Tellman | Timbre Morphing of Sounds with Unequal Numbers of Features[END_REF]. The basic idea behind the interpolation principle is that if we can represent different sounds by simply adjusting the parameters of a model, we should obtain a somewhat smooth transition between two (or more) sounds by interpolating between these parameters. Interpolation of sinusoidal modeling is amongst the most common approaches [START_REF] Fitz | Cell-Utes and Flutter-Tongued Cats: Sound Morphing Using Loris and the Reassigned Bandwidth-Enhanced Model[END_REF], [START_REF] Williams | Perceptually-Motivated Audio Morphing: Softness[END_REF], [START_REF] Tellman | Timbre Morphing of Sounds with Unequal Numbers of Features[END_REF], [START_REF] Boccardi | Sound Morphing with Gaussian Mixture Models[END_REF]. The sinusoidal parameters can be directly interpolated [START_REF] Fitz | Cell-Utes and Flutter-Tongued Cats: Sound Morphing Using Loris and the Reassigned Bandwidth-Enhanced Model[END_REF], [START_REF] Tellman | Timbre Morphing of Sounds with Unequal Numbers of Features[END_REF], or by means of another technique [START_REF] Boccardi | Sound Morphing with Gaussian Mixture Models[END_REF]. A few authors have proposed to detach the spectral envelope from the pitch information and interpolate them separately [START_REF] Slaney | Automatic Audio Morphing[END_REF], [START_REF] Ahmad | Morphing of transient sounds based on shift-invariant discrete wavelet transform and singular value decomposition[END_REF]. Even more interesting seems to be the approach of designing the spectral envelope separately [START_REF] Caetano | Evolutionary Spectral Envelope Morphing by Spectral Shape Descriptors[END_REF], [START_REF] Ezzat | Morphing Spectral Envelopes using Audio Flow[END_REF] and imposing the result later for synthesis [START_REF] Arfib | Source-Filter processing[END_REF].

Our main motivation is to find a morphed sound that would not only be perceived as a hybridization of the sources, but would also be perceptually intermediate with respect to known salient timbre dimensions, such as brightness [START_REF] Mcadams | Perceptual Scaling of Synthesized Musical Timbres: Common Dimensions, Specifities and latent subject Classes[END_REF]. In other words, instead of simply obtaining morphed sounds, we want to control the morphing process perceptually. We want to be able to decide how much of each timbrally related feature we will include from each source. So, for example, when morphing between a bright trumpet sound and a duller clarinet sound, we want to be able to control the perceived brightness of the trumpet-clarinet hybrid. For this, we need to be able to independently manipulate individual features. There have been different proposals in the literature to use features to guide synthesis and transformations. Yee-King [START_REF] Yee-King | Synthbot: An Unsupervised Software Synthesizer Programmer[END_REF] uses a genetic algorithm to tune the parameters of an FM synthesis model according to target MFCC values. Hoffman [START_REF] Hoffman | Feature-based Synthesis: Mapping from Acoustic and Perceptual Features to Synthesis Parameters[END_REF] presents preliminary results on an MFCC-based synthesis module that uses some descriptors as guides, while Le Groux [START_REF] Groux | Perceptsynth: Mapping Perceptual Musical Features to Sound Synthesis Parameters[END_REF] uses a support vector machine approach to map an additive synthesis PCA-reduced model to descriptors such as fundamental frequency and loudness. In turn, Park [START_REF] Park | Feature Modulation Synthesis[END_REF] proposes ways of modulating various descriptor-based features, although not independently, such that varying one feature also changes the others in unexpected ways. Verfaille [START_REF] Verfaille | Adaptive Digital Audio Effects (a-DAFx): A New Class of Sound Transformations[END_REF] details a general framework to manipulate low-level features to obtain sound transformations that control certain perceptually related features.

However, for morphing, only recently did we start to take perceptual aspects into consideration [START_REF] Williams | Perceptually-Motivated Audio Morphing: Softness[END_REF], [START_REF] Caetano | Automatic Timbral Morphing of Musical Instrument Sounds by High-Level Descriptors, to appear[END_REF], [START_REF] Caetano | Evolutionary Spectral Envelope Morphing by Spectral Shape Descriptors[END_REF], and the result is the addition of one more step in the process, feature calculation. In most models proposed, linear variation of interpolation parameters does not produce perceptually linear morphs [START_REF] Caetano | Automatic Timbral Morphing of Musical Instrument Sounds by High-Level Descriptors, to appear[END_REF]. For the moment, we are interested in being able to control perceptual features of sounds related to the spectral shape, so we will study spectral envelope morphing [START_REF] Ezzat | Morphing Spectral Envelopes using Audio Flow[END_REF] and manipulation techniques, such as that proposed by Caetano [START_REF] Caetano | Evolutionary Spectral Envelope Morphing by Spectral Shape Descriptors[END_REF]. In this work, we aim at spectral envelope design guided by high-level features, such that in general terms our approach consists of first obtaining an envelope with the general desired number of formant peaks and then shaping it so as to manipulate the high-level features. More spe-DAFX-2 cifically, we will study a method that produces more perceptually relevant morphed envelopes with the desired intermediate number of formant peaks guided by high-level spectral shape descriptors. High-level descriptors are acoustic correlates of timbral dimensions, such that manipulation of the descriptors corresponds to potentially interesting timbral variations. So we define the most suitable representation of the spectral envelope that allows manipulating the spectral envelope shape while retaining the number of peaks. Finally, we describe a technique that uses a genetic algorithm (GA) [START_REF] Davis | Handbook of Genetic Algorithms[END_REF] to independently manipulate the timbral dimensions guided by the descriptors.

The next section presents the basic notion behind the spectral envelope and sound source identification. Then we describe how slight variations of a spectral envelope can be perceived as presenting slightly different timbral features (different neighboring points in timbre space), while still being associated with the same instrument. Next, we introduce the high-level spectral shape descriptors, which capture the acoustic correlates of timbre dimensions. We proceed with the description of the methods to obtain morphed spectral envelopes and study their perceptual impact. The next step is to introduce a technique that uses evolutionary computation to independently manipulate timbral features associated with an envelope. Finally, we describe the experiment we devised to validate our proposal, followed by an evaluation of the results and the conclusions and future perspectives.

SOURCE-FILTER MODEL AND SOUND SOURCE IDENTIFICATION

Listeners use many acoustical properties to identify sonic events, such as the spectral shape, formant frequencies, attack and/or onset and decay and/or offset, noise, among others [START_REF] Handel | Timbre perception and auditory object identification[END_REF]. The cues to identification and timbre vary across notes, durations, intensities and tempos. One model of sound production is based on two possibly interactive components, the source and the filter. The basic notion is that the source applies excitation energy to generate a vibration pattern composed of several vibration modes (modeled as sinusoidal components). This pattern is imposed on the filter, which acts to modify the relative amplitudes of the components of the source input. Resonators, by their nature, tend to amplify certain frequencies louder than others. These resonant frequency regions, or formant peaks, are uniquely related to the size and shape of the instrument and its resonator. We obtain estimates of the excitation and the filter by calculating the spectral envelope, which is a smooth curve that approximately matches the peaks of the spectrum. The peaks of the spectral envelope (also called formants in voice research) correspond roughly to the vibration modes of the source-filter model. The number and absolute position of spectral peaks in frequency is important for musical instrument (sound source) identification. However, we cannot underestimate the perceptual impact of slight variations of spectral shape. The relationship between fundamental frequency and timbre, for example, is readily apparent in some acoustic instruments. The clarinet, for instance, has three distinct registers, that is, three distinct pitch ranges with three different timbral characteristics. It is remarkable that a single instrument can have such a variety of timbres, but the example of the clarinet proves the impact of variations of spectral shape on an instrument's timbre and even temporal evolution. The relationship between applied energy and timbre is relatively clear. As more energy is input to the instrument, higher modes of vibration are achieved such that more partials are present in the frequency spectrum. This is why a note played forte is not just louder than piano, but also brighter in timbre. A classical example is Risset's discovery that brassy trumpet sounds, usually described as bright, present a broader spectrum resulting from the appearance of higher partials. The spectral centroid, defined later in Section 4, was found [START_REF] Mcadams | Perceptual Scaling of Synthesized Musical Timbres: Common Dimensions, Specifities and latent subject Classes[END_REF], [START_REF] Grey | Perceptual Evaluations of Synthesized Musical Instrument Tones[END_REF] to be highly correlated with the dimension of timbre usually verbally labeled as brightness. Therefore, brassy trumpet sounds that are perceived as bright can be characterized at the signal level as presenting a high spectral centroid value. Notably, a crescendo trumpet note would exhibit an increasing spectral centroid, while roughly preserving the general position of the first formant peaks (because other peaks appear at high frequency regions).

The vast majority of research in sound perception has focused either on the acoustic properties of musical instruments [START_REF] Risset | Computer Study of Trumpet Tones[END_REF] or on the perception of sounds as unveiled by psychoacoustic experiments [START_REF] Mcadams | Perceptual Scaling of Synthesized Musical Timbres: Common Dimensions, Specifities and latent subject Classes[END_REF]. The challenge we face today is to find the link between the two in order to be able to manipulate the sounds in a more perceptually meaningful way.

ACOUSTIC CORRELATES OF TIMBRE SPACES

In this section we briefly present timbre perception, timbre spaces and the most relevant acoustic correlates of timbral dimensions obtained in the literature of timbre perception. The concept of timbre is related to the subjective response to the perceptual qualities of sound objects and events [START_REF] Handel | Timbre perception and auditory object identification[END_REF]. We know that source identification is not reduced to waveform memorization because the intrinsic dynamic nature of the sources produces variations [START_REF] Handel | Timbre perception and auditory object identification[END_REF]. Timbre perception is inherently multidimensional, involving features such as the attack, spectral shape, and harmonic content. Historically, Helmholtz was the first to propose an acoustic model of musical instrument sounds. Helmholtz characterized what he called musical tone as a waveform that follows an amplitude envelope that consists of the attack, the steady state and the decay, as shown in Figure 1. During the attack, the amplitude increases from zero to its peak value. In the steady state portion the amplitude is constant and finally decreases back to zero during the decay. Helmholtz concluded that sounds that evoke the sensation of pitch possess fixed waveforms that do not change in the course of the tone, apart from the amplitude envelope, whose temporal evolution has great impact on the perception of the tone, according to him. The classical Helmholtz model breaks down when we examine musical instrument sounds on a small scale. When the harmonic content of sound is examined with the STFT over small time periods, we discover that, contrary to the Helmholtz model, a sound's spectrum changes profoundly over time. During the attack portion of a sound, harmonic content may change rapidly and unpredictably. This phenomenon is called the initial transient. During the release, upper partials tend to disappear more quickly before the entire sounds fades away. While the sustain portion of the sound, when it exists, is certainly more stable than the attack or decay, it is hardly as static as Helmholtz would suggest. Clearly, the basic premise of the classical Helmholtz model -a static spectral envelope with a fixed amplitude envelope temporal evolution -is by no means an accurate and robust characterization of a wide range of acoustic musical instrument sounds. Since the pioneering work of Helmholtz, multidimensional scaling techniques figure among the most prominent when trying to quantitatively describe timbre. Handel [START_REF] Handel | Timbre perception and auditory object identification[END_REF] gives a comprehensive review of the early timbre space studies. Grey [START_REF] Grey | Perceptual Evaluations of Synthesized Musical Instrument Tones[END_REF] investigated the multidimensional nature of the perception of musical instrument timbre and constructed a three-dimensional timbre space and proposed acoustic correlates for each dimension. He concluded that the first dimension corresponded to spectral energy distribution (measured by the spectral centroid), the second and third dimensions were related to the temporal variation of the notes (onset synchronicity). Krumhansl [START_REF] Krumhansl | Why is Musical Timbre So Hard to Understand?[END_REF] conducted a similar study using synthesized sounds and also found three dimensions related to attack, synchronicity and brightness (spectral energy distribution). Krimphoff [START_REF] Krimphoff | Caractérisation du Timbre des sons Complexes. II: Analyses Acoustiques et Quantification Psychophysique[END_REF] studied acoustic correlates of timbre dimensions and concluded that brightness is correlated with the spectral centroid and rapidity of attack with rise time in a logarithmic scale. McAdams [START_REF] Mcadams | Perceptual Scaling of Synthesized Musical Timbres: Common Dimensions, Specifities and latent subject Classes[END_REF] conducted similar experiments with synthesized musical instrument timbres and concluded that the most salient dimensions were log rise time, spectral centroid and degree of spectral variation. More recently, Caclin [START_REF] Caclin | Acoustic correlates of timbre space dimensions: A confirmatory study using synthetic tones[END_REF] studied the perceptual relevance of a number of acoustic correlates of timbre-space dimensions with MDS techniques and concluded that listeners use attack time, spectral centroid and spectrum fine structure in dissimilarity rating experiments.

These results suggest that slight changes in the spectral shape produce perceptual changes in timbre that can be measured with high-level spectral shape descriptors. Notably, a morphed sound/spectral envelope with intermediate descriptors should be perceived not only as a hybrid of the source envelopes, but especially as perceptually intermediate.

HIGH-LEVEL DESCRIPTORS

In this section we present the general scheme used to calculate the descriptors used in this work, depicted in Figure 2. The sound signal is highlighted with a dark background, all the purely signal processing stages have white background and the steps where we calculate the descriptors present a light background. Peeters [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF] describes exhaustively how to calculate all the descriptors we use in this work and proposes to use them in audio classification tasks instead of traditional MFCCs. We are going to present every step of the descriptor extraction scheme with emphasis on the descriptor calculation procedures. The basic signal processing step is the STFT, which refers to the blocks that read "signal frame" and "FFT".

Spectral Shape

The calculation of the spectral shape descriptors consists of three steps, spectral envelope estimation, application of the perceptual model, and finally calculation of the spectral shape descriptors, namely, spectral centroid, spread, skewness, kurtosis and slope [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF]. For every frame, we calculate the spectral envelope using a cepstral smoothing technique, called true envelope [START_REF] Röbel | Efficient Spectral Envelope Estimation and its Application To Pitch Shifting And Envelope Preservation[END_REF]. Next, we apply the perceptual model, which consists of the mid-ear filter shown in Figure 3 evaluated on the mel frequency scale. We should notice that this calculation is like the MFCC-based spectral envelope used in [START_REF] Slaney | Automatic Audio Morphing[END_REF] without critical band smoothing, such that we do not lose information. Finally we calculate the spectral shape descriptors with the mid-ear attenuated, melwarped spectral envelope. The spectral shape descriptors considered are calculated as if the magnitude spectrum were a probability distribution. So we associate the frequency bins i of the DFT with the sample space and the probabilities to observe them with the magnitude of the normalized spectral envelope, given by

( ) ( ) ( ) ∑ = k k S k S k p (1)
such that the spectral shape descriptors are defined as the moments of p(k), where k is the frequency index. The spectral centroid is measured as the mean of p(k) and the spectral spread as the standard deviation, shown in equations ( 2) and (3) respectively.

( ) ∑ = k k kp µ (2) ( ) ∑ = k k p k 2 2 σ (3)
The third and forth standardized moments are respectively skewness and kurtosis, shown in equations ( 4) and ( 5).

( ) ( )

3 3 3 σ µ γ ∑ - = k k p k (4) ( ) ( ) 4 4 4 σ µ γ ∑ - = k k p k (5)
And finally the spectral slope is given by equation ( 6), where i is the FFT frequency bin index

∑ ∑ ∑ ∑ ∑ ∑       - - = i i i i i i i k i k N i k i p i k i p N i p 2 2 ) ( ) ( ) ( ) ( ) ( ) ( ) ( 1 λ (6)
Proc. of the 13 th Int. Conference on Digital Audio Effects (DAFx-10), Graz, Austria, September 6-10, 2010 DAFX-4

SPECTRAL ENVELOPE MORPHING BY HIGH-LEVEL DESCRIPTORS

In this section we explain our motivation for using spectral shape descriptors as guides in obtaining perceptually relevant morphed spectral envelopes. The aim of morphing spectral envelopes is to obtain a result that is perceived not only as a hybrid between the original sounds, but especially perceptually intermediate between them. Slaney [START_REF] Slaney | Automatic Audio Morphing[END_REF] explains the concept by analogy with image morphing, where the aim is to gradually change from one image to the other, producing perceptually convincing intermediates (or hybrids) along the way. Other authors have proposed the same analogy [START_REF] Fitz | Cell-Utes and Flutter-Tongued Cats: Sound Morphing Using Loris and the Reassigned Bandwidth-Enhanced Model[END_REF]. Figure 4 shows such an example of image morphing with faces. Clearly, it is not enough to blindly interpolate parameters (pixels, for instance, for the images) since there are a number of important features in the faces that we must take into account. Finding those features is an important task, and developing techniques to obtain intermediate (hybrid) images that use those features as cues is the key to a successful morph. The analogy with spectral envelope morphing is immediate. Each frame of the STFT is interpreted as a snapshot of the spectrum of the sounds seen through a time window. So the task of morphing spectral envelopes becomes similar to image morphing, each hybrid envelope must present intermediate features to be perceptually convincing. Here we argue that high-level descriptors capture salient timbre dimensions of sounds, so we use them as a guide to morph spectral shapes. An important concept that can be inferred from Figure 4 is the fact that there are many possible intermediate steps between the two images shifting from the first (S 1 ) to the second (S 2 ) spectral envelope. So, if we consider each intermediate hybrid spectral envelope as the result of a different combination of S 1 and S 2 , this convex combination can be mathematically expressed as equation ( 7)

( ) [ ] 2 1
) ( 1 ) ( , S t S t t M α α α - + = (7)
and each step is characterized by one value of a single parameter (α), called morphing factor, as shown at the bottom of Figure 4.

The morphing factor should vary between 0 and 1, such that α = 1 and α = 0 produces S 1 and S 2 respectively.

Usually, morphing techniques propose to interpolate the parameters of a model without making sure that the results will (perceptually) correspond in the feature space. Moreover, depending on the parameters we interpolate, the morphed spectral envelope will not present the desired number and position of peaks. Ideally, we want the morphed spectral envelope to have an intermediate number and position of peaks and to match as closely as possible the values of target perceptually related features, in this case, spectral shape descriptors. So we present in Figure 5 an example of the perceptual impact of morphing spectral envelopes by three different methods proposed in the literature, namely, linear predictive coefficients (LPC) [START_REF] Caetano | Evolutionary Spectral Envelope Morphing by Spectral Shape Descriptors[END_REF], [START_REF] Makhoul | Linear prediction: A tutorial review[END_REF] line spectral frequencies (LSF) [START_REF] Caetano | Automatic Timbral Morphing of Musical Instrument Sounds by High-Level Descriptors, to appear[END_REF], [START_REF] Mcloughlin | Line spectral pairs[END_REF], [START_REF] Pfitzinger | DFW-based Spectral Smoothing for Concatenative Speech Synthesis[END_REF], and dynamic frequency warping (DFW) [START_REF] Ezzat | Morphing Spectral Envelopes using Audio Flow[END_REF], [START_REF] Pfitzinger | DFW-based Spectral Smoothing for Concatenative Speech Synthesis[END_REF]. Figure 5 compares the result of varying the interpolation factor from 0 to 1 for LPC, LSF and DFW both in number of peaks and in shape for two very different spectral envelopes, labeled clavi and tuba. Notice that, for this example, LPC (top) does not interpolate well either the peaks or the shape. Neither the spectral envelopes nor the descriptors vary smoothly from one extreme to the other as desired. Although DFW (middle) visually seems to render a satisfactory shift, the descriptors vary in unexpected ways. LSFs (bottom) interpolate better both the peaks and the shape. Ideally, we expect the hybrid spectral envelope to smoothly change from source to target, with peaks shifting, appearing or splitting and disappearing or merging. The change is also very satisfactory in the descriptor domain, varying almost as straight lines. Ideally, we want a spectral envelope morphing method that generates hybrid envelopes with not only an intermediate number of peaks, but also with intermediate values of features, as measured by the descriptors. When we set α = 0.5, we seek for a morphed envelope whose descriptors are halfway between those of S 1 and S 2 . Table 1 shows an example of the accuracy of the three methods for α = 0.5. We show the target value for each descriptor considered, calculated applying equation (7) using the descriptor values as S 1 and S 2 , and the value measured for the spectral envelope produced by each morphing method. We consider a method that generates morphed envelopes with descriptor values closer to the target values to render more perceptually relevant morphed spectral envelopes [START_REF] Caetano | Automatic Timbral Morphing of Musical Instrument Sounds by High-Level Descriptors, to appear[END_REF]. This is an objective measure of the perceptual impact of the morphed envelopes obtained with each method.

Descriptor

Target 3.14 1.12 1.54 3.12 Table 1. Target (α = 0.5) and measured spectral shape descriptors.

INDEPENDENT MANIPULATION OF FEATURES BY EVOLUTIONARY COMPUTATION

In this section we explain the technique we use to manipulate perceptually relevant timbral features of the morphed envelopes to produce interesting timbral variants. In other words, we are still looking for a morphed envelope that is perceived halfway (α = 0.5) between a clarinet and a trombone, but now we dot not want all the timbral features to be also automatically in the middle. We want to be able to control them independently and obtain a hybrid clarinet-trombone morphed envelope that sounds as bright as the original trombone, for example. This would correspond to obtaining a hybrid image morph of the Bush-Obama faces where all the features are halfway, except the nose, which still resembles Obama's. The technique consists in generating a prototype morphed envelope with intermediate features, and then manipulating the features independently to match new target values set for each descriptor. The variants are obtained by setting the morphing factor α independently for each descriptor, representing each feature we want to control. The prototype morphed envelope can potentially be obtained with any method because we can convert the representation of a spectral envelope. This means that we can obtain the LSF representation of an envelope estimated with true envelope [START_REF] Röbel | Efficient Spectral Envelope Estimation and its Application To Pitch Shifting And Envelope Preservation[END_REF], linear prediction [START_REF] Makhoul | Linear prediction: A tutorial review[END_REF], or any other method. Particularly, we can calculate the LSF representation of an envelope generated with DFW [START_REF] Pfitzinger | DFW-based Spectral Smoothing for Concatenative Speech Synthesis[END_REF]. Therefore, we will verify if there is one prototyping method that outperforms the other between prototype envelopes generated with LSF and DFW. We clearly need a suitable representation that allows local manipulation of the spectral shape without completely changing the overall prototype envelope (i.e., the number and location of peaks).

Line Spectral Frequency Pairs

Line spectral frequency pairs (LSFs) are an alternative parameterization of LPC [START_REF] Makhoul | Linear prediction: A tutorial review[END_REF] with a one to one correspondence. The two LSF polynomials are given by ( ) ( ) ( ) ( )

1 1 - + - + = z A z z A z P p (8) ( ) ( ) ( ) ( ) 1 1 - + - - = z A z z A z Q p (9)
where A(z) is the linear prediction polynomial of order p [START_REF] Makhoul | Linear prediction: A tutorial review[END_REF]. The roots of the polynomials in equations ( 8) and (9) determine the LSFs. If A(z) is minimum phase, the roots of P(z) and Q(z) are on the unit circle, are real, interleaved with each other, and always lead to stable envelopes when arranged in ascending order [START_REF] Mcloughlin | Line spectral pairs[END_REF]. LSFs also present the useful tendency to be located where the peaks of the envelope they represent are. Figure 6 shows that each pair tends to be close together when near a peak of the spectral envelope and far apart when not, depicting another useful property of LSFs. The closer the line spectrum pair is, the narrower the peak. Based on these properties of LSFs, McLoughlin [START_REF] Mcloughlin | Line spectral pairs[END_REF] exemplifies how we can manipulate the LSFs to produce small changes in the shape of the spectral envelope and Morris [START_REF] Morris | Modification of Formants in the Line Spectrum Domain[END_REF] presents a method for modifying formant peak locations and bandwidths in the line spectrum domain. Figure 6 shows the original spectral envelope in grey and a modified envelope (solid line) with its corresponding LSFs. Since there are LSF pairs that correspond roughly to specific spectral peaks, we generally can make changes to a specific peak without changing much the overall spectral envelope. With this in mind, for a prototype morphed envelope, there must be a variant with slightly different LSFs whose shape matches more closely the independently set target spectral shape descriptors. Because fine tuning the LSFs to match all descriptors at once is a difficult task (highly nonlinear mapping), we use a genetic algorithm (GA) to perform the search. The GA will manipulate the LSF representation of prototype morphed envelopes obtained with both LSF and DFW.

Genetic Algorithms

Genetic algorithms (GAs) are the most commonly used paradigm of evolutionary computation due to the robustness with which they explore complex search spaces. They codify the parameters of a model into a chromosome-like structure so that each individual corresponds to a point in the parameter space, depending on the values of the parameters. The resulting search space contains the candidate solutions, and the evolutionary operators will implement exploration and exploitation of the search space aiming at finding quasi-global optima. The GA iteratively manipulates populations of individuals at a given generation by means of the simple genetic operations of selection, crossover and mutation. The evolutionary process combines survival of the fittest with the exchange of information in a structured yet random way. The standard genetic algorithm [START_REF] Davis | Handbook of Genetic Algorithms[END_REF]

Codification and Evolutionary Operators

In this section we explain how we apply the steps presented in Table 2 to the population of candidate prototype spectral envelopes in the search for the variant envelope model that best matches the independent target envelope shape descriptor values.

Initialization

We initialize the population of N candidate solutions by producing variants of a prototype spectral envelope. We obtain the prototype morphed envelope with either LSF or DFW. The variants are obtained by adding a perturbation vector (normal distribution N(0,10 -3 )) to the LSFs of the prototype morphed envelope, so that each individual is a variant of the prototype with a slightly different spectral shape. Each individual in the population is codified as a chromosome that lists LSF pairs in ascending order, as represented on the top of Figure 7. This initialization process is intended to sample the search space (spectral shape descriptors) while restricting the positions of the peaks of the envelopes.

Crossover

Crossover is responsible for the exploitation of regions of interest of the search space by means of the exchange of information between individuals of a population. Crossover consists of selecting two parent individuals, the crossover points, and swapping the chromosome segments (represented by different shades in Figure 7) between them, thus generating two offspring. We mate each individual of the current population with one randomly chosen partner (uniform distribution) using a one-point crossover operator with a uniform distribution [START_REF] Davis | Handbook of Genetic Algorithms[END_REF]. Both offspring are inserted in the population and the parents are also kept. We use a onepoint crossover operator, which consists of selecting one mating partner for each individual in the population (those are the parent chromosomes), randomly (uniform distribution) choosing a crossover point and swapping the segments between the parents, thus generating the offspring chromosomes shown in the middle of Figure 7, where each segment is represented by a different shade. The result of the crossover operation is two offspring individuals comprised of LSFs that come from both parents. Thus the offspring consist of variant spectral envelopes whose spectral shape inherits characteristics of both parents. It is important to notice that both offspring generated are inserted in the population and the parents are also kept, thus increasing the number of candidate solutions by 2. That is, now we have three times the initial number of individuals in the population, or 3N, because for each individual we mate, they produce two offspring that are inserted in the current generation and we keep the current individuals in the population.

Mutation

Mutation is responsible for the exploration of the search space by randomly replacing the value of one randomly selected pair of LSFs, thus allowing different regions of the search space to be investigated. The mutation operation, applied to all individuals in this increased population, is depicted at the bottom of Figure 7 and consists of randomly (uniform distribution) choosing a mutation point, represented in black in Figure 7, and adding a perturbation to it (normal distribution N(0,10 -3 )). Thus mutation is a kind of perturbation of the LSF pair selected, resulting in a slightly different shape.

Fitness Evaluation

Next we measure the fitness of all individuals in the current population using the fitness function (ff) in equation [START_REF] Boccardi | Sound Morphing with Gaussian Mixture Models[END_REF]. The fitness function operates on the feature space (the spectral shape descriptors) and here it is a very simple error (or distance) measure between the target and the calculated descriptor values. Equation 10 below shows the computation as the absolute value of the difference between target descriptors (T) and the descriptor values (c i ) calculated for each individual in the current generation, weighted (ω i ) and normalized by the target value T for each spectral shape descriptor used in the method so that they are all dimensionless and therefore can be compared.

( ) ∑ ∑ - = i i i i i i i T c T ff ω ω (10)

Selection

Finally, the selection operator discards individuals with low fitness values, only keeping individuals that correspond to promising regions of the search space for the next generation. Selection is done by sorting the individuals of the population of the current generation by increasing values of fitness and selecting the first N as the population for the next generation. We use a strategy called elitism that consists of keeping the best individual found so far in all generations even if it is lost in the current population due to crossover and mutation. The termination criterion is met when either a minimum fitness threshold or the maximum number of generations is reached.

EXPERIMENT AND RESULTS

We aim to show that the GA allows us to obtain hybrid morphed envelopes that closely match the target perceptual descriptor values even if we use independent morphing factors for the descriptors. We also want to verify which method used to obtain the prototype envelope, DFW or LSF, renders resultant envelopes that retain the desired overall number and position of peaks. So we set a variable morphing factor (α = [0.1, 0.3, 0.5, 0.7, 0.9]) for the spectral centroid and kept the others constant at α = 0.5. We expect the results to show hybrid spectral envelopes whose centroid varies as desired while keeping the other descriptors considered unchanged. We measure the quality of the results as how closely they match the values of target perceptual spectral shape descriptors while retaining the desired overall number and position of peaks of the prototype spectral envelope. Table 3 shows the target and measured descriptor values for each morphing factor α indicated, applied only to the perceptual centroid. 1), the GA compromises the original number and position of peaks in favor of shape. Table 3, on the other hand, shows that, for LSFs, as the spectral centroid shifts as expected, the other values vary slightly. In general, LSFs outperformed DFW in matching the target descriptors. We do not control the individual accuracy of descriptors; therefore, because they all have different ranges, the precision of matching the spectral centroid differs from the other descriptors. Without the weights, descriptors with smaller ranges tend to be matched with greater precision. The weights allow us to tip the scales and focus on the descriptors of interest. If we compare the corresponding columns in Table 1 with the column labelled α = 0.5 in Table 3, we see that DFW performed poorly in matching the target descriptors even after the application of the GA. On the other hand, for LSFs, in this case and all others presented, even though the GA unmatched the other descriptors a little, the descriptor of interest (centroid) is always a perfect match. We do not directly compare the results quantitatively with the manipulation of LPCs by a GA [START_REF] Caetano | Evolutionary Spectral Envelope Morphing by Spectral Shape Descriptors[END_REF] because qualitatively they are not equivalent. The method presented in [START_REF] Caetano | Evolutionary Spectral Envelope Morphing by Spectral Shape Descriptors[END_REF] does not manipulate LSFs and therefore would probably not render results with the desired number and position of peaks for, as we showed in section 5, the interpolation of LPCs is highly unstable.

We did not find any studies on how the accuracy of the spectral shape descriptors affects the perception of the features they are related to, such that, it is not possible as of now to decide how perceptually relevant the values are and how accurate the matching should be. We would need to study whether there is a sort of just noticeable difference (JND) for the descriptors in order to infer how their individual accuracies affect perception. The most important aspect of the results lies in the independent control of the descriptors given that the relevance of the descriptors values is only relative and there is no scale at present with which to perform a deep quantitative analysis. Controlling the descriptors individually is a first step toward the study of perceptually motivated feature-based sound transformations such as morphing. All the results presented in this article are available online http://recherche.ircam.fr/anasyn/caetano/morph.html.

CONCLUSION AND FUTURE PERSPECTIVES

The aim of sound morphing is to find a morphed sound that would not only be perceived as a hybridization of the original sounds, but would also be perceptually intermediate with respect to known salient timbre dimensions. We studied methods to obtain a morphed spectral envelope that contains both the desired intermediate number and position of resonant peaks, but also corresponding intermediate perceptual features, measured by highlevel spectral shape descriptors. High-level descriptors are acoustic correlates of timbral dimensions, such that manipulation of the descriptors corresponds to potentially interesting timbral variations. We compared the morphed spectral envelopes obtained with LPC [START_REF] Makhoul | Linear prediction: A tutorial review[END_REF], DFW [START_REF] Pfitzinger | DFW-based Spectral Smoothing for Concatenative Speech Synthesis[END_REF] and LSFs [START_REF] Mcloughlin | Line spectral pairs[END_REF] and found that LSFs generate more smoothly varying hybrids according to two criteria, number and position of spectral peaks and values of spectral shape descriptors.

Then, we defined LSFs as the most suitable representation of the spectral envelope that allows manipulating the spectral envelope shape while roughly retaining the desired number and position of peaks of the prototype morphed envelope we wish to transform. Finally, we described a technique that uses a genetic algorithm (GA) to independently manipulate perceptually relevant timbral features of the morphed spectral envelopes guided by the descriptors to produce interesting timbral variants.

We verified that we have finer control over sound transformations perceptually if we are able to control how individual features behave under each transformation. Using our proposed technique, we can perform transformations where only one descriptor changes, or they all change in different ways, so there would be many different possible transformations between two different sounds regarding the perceptual shape, instead of only one. For example, we could obtain a morphed clarinet-trumpet sound and impose the brightness of the trumpet on the morphed result.

However, we would need to perform listening tests to verify how these differences manifest perceptually, since the perceptual impact of manipulating such nuances of timbral perception is still unclear.

Future perspectives of this work include investigating the impact of the timbral variants on the spectral shape of the origi- Proc. of the 13 th Int. Conference on Digital Audio Effects (DAFx-10), Graz, Austria, September 6-10, 2010 DAFX-8 nal sounds alone. We could investigate if the timbral variationtechnique presented here could render a trumpet sound brighter, like it was played louder. Also, we could verify whether improvements on the perceptual model used to calculate the descriptors would render more perceptually relevant results. Finally, there are many possible variations of the application of the GA that should be investigated, such as a different crossover operator, mutation, selection, fitness function, among others. One could even test other optimization methods or even different mapping strategies to obtain the variant spectral envelopes.

Figure 1 :

 1 Figure 1: Classical acoustic model of musical instrument sounds. Original figure from [22].

Figure 2 .

 2 Figure 2. Simplified scheme to calculate the descriptors.

Figure 3 :

 3 Figure 3: Mid-ear filter applied to extracted spectral envelopes

Figure 4 :Figure 5 .

 45 Figure 4: Depiction of image morphing to exemplify the aim of sound morphing

Figure 6 :

 6 Figure 6: Depiction of LSFs and corresponding spectral envelope. Original figure from [30].

6 Figure 7 .

 67 Figure 7. Depiction of the chromosome (top), crossover (middle) and mutation operations (bottom).

Figure 8

 8 depicts the resultant envelopes for LSF and DFW proto-LSF Targ DFW LSF Targ DFW LSF Targ DFW LSF

Figure 8 .

 8 Figure 8. Spectral envelopes resulting from the application of the GA with a varying morphing factor independently controlling the perceptual spectral shape descriptors for LSF and DFW prototype envelopes.

  consists of the steps shown in Table2.

	(*Initialize Population*)
	(*Main Cycle*) generations
	repeat
	(*Competition Cycle*)
	Crossover
	Mutation
	Fitness Evaluation
	Selection
	until termination criterion met

Table 2 .

 2 Standard Genetic Algorithm.

Table 3 .

 3 Target and measured perceptual spectral shape descriptor values for DFW and LSF after the application of the GA with morphing factor independently set. type spectral envelopes. If we compare the number and position of peaks of the results presented in Figure8with the corresponding morphed prototypes in Figure5, we readily see that the desired intermediate number and position of peaks was retained for LSF. Although DFW generates morphed prototypes with smoothly varying intermediate number and position of peaks, since the corresponding descriptor values are farther from the target (see Table

	Centr (x10 +3 )	2.35 3.53	2.35 2.78 2.78	2.78 3.20 2.69	3.20 3.63 2.43	3.63 4.05 2.42	4.05
	Spread (x10 +6 )	2.70 4.52	1.66 2.70 3.91	2.08 2.70 2.71	2.57 2.70 1.94	2.99 2.70 1.44	3.49
	Skew	1.27 1.07	1.82 1.27 1.86	1.47 1.27 2.48	1.17 1.27 2.69	0.93 1.27 1.93	0.68
	Kurt	12.4 5.35	16.6 12.4 8.64	12.6 12.4 14.9	9.52 12.4 21.2	7.49 12.4 20.7	5.84
	Slope (-1x10 -16 ) 3.14 5.63	1.89 3.14 5.84	2.64 3.14 5.11	3.39 3.14 3.54	3.97 3.14 1.85	4.39

ACKNOWLEDGEMENTS

This work is supported by the Brazilian Governmental Research Agency CAPES (process 4082-05-2).


Marcelo Freitas Caetano

Juan Jose Burred

Xavier Rodet

Audionamix

AUTOMATIC SEGMENTATION OF THE TEMPORAL EVOLUTION OF ISOLATED ACOUSTIC MUSICAL INSTRUMENT SOUNDS USING SPECTRO-TEMPORAL CUES

whether they are published or not. The documents may come

INTRODUCTION

The temporal evolution of musical instrument sounds plays a conspicuous role on the perception of their most important features [START_REF] Handel | Timbre perception and auditory object identification[END_REF]. Sound modeling and manipulation techniques could be greatly improved by the correct segmentation of musical instrument sounds taking into account the different characteristics of each perceptually different region. Notably, time stretching a sound ignoring its temporal evolution results in a perceptually different sound because the attack region, for example, is transformed by the same factor as the rest of the sound, even if it is well known [START_REF] Iverson | Isolating the Dynamic Attributes of Musical Timbre[END_REF] that the attack transients are perceived differently, being one of the most perceptually salient dimensions of musical timbre as unveiled by psycho-acoustical experiments [START_REF] Mcadams | Perceptual Scaling of Synthesized Musical Timbres: Common Dimensions, Specifities and latent subject Classes[END_REF]. The morphing of musical instrument sounds (especially those based on different modes of excitation) could largely benefit from a more accurate model of temporal evolution. A correct segmentation would also be beneficial to the modeling of musical instrument sounds for synthesis or classification purposes. Notably, the MPEG-7 standard [START_REF] Peeters | Instrument Sound Description in the Context of MPEG-7[END_REF] relies heavily on the estimation of perceptually important temporal events such as attack time to accomplish tasks such as automatic classification and similarity of sounds. Statistical and sinusoidal modeling could benefit from the segmentation as well. Computational [START_REF] Collins | Investigating Computational Models of Perceptual Attack Time[END_REF] models of perceptual attack time (PAT) [START_REF] Gordon | The Perceptual Attack Time of Musical Tones[END_REF] must rely on an accurate estimation of attack time. However, to automatically estimate perceptually relevant features of musical sounds, such as the attack, from the signal we must rely on a model that utilizes features that exhibit a characteristic behavior during the events we wish to detect. The automatic segmentation task consists in the detection of events such as onset, attack, decay, sustain, release, and offset. There have been many proposals to the detection of some of these isolated events, especially onset [START_REF] Schloss | On the Automatic Transcription of Percussive Music -From Acoustic Signal to High-Level Analysis[END_REF], [START_REF] Bello | A tutorial on onset detection in music signals[END_REF] and attack time [START_REF] Collins | Investigating Computational Models of Perceptual Attack Time[END_REF]. However, the segmentation problem has been more rarely addressed [START_REF] Jensen | Envelope Model of Isolated Musical Sounds[END_REF], [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF]. Usually, the problem is attacked in two steps: extraction of a detection function that supposedly contains information about the events to be detected followed by a technique responsible for automatically picking them out. The correct segmentation depends heavily on a model that accounts for as many different types of sounds as possible because not all instruments necessarily generate all the events. Also, due to the different nature of some of the events we wish to detect, it is almost naïve to expect one detection function to carry information about all of them. Historically, Helmholtz was the first to propose the segmentation of an acoustic musical instrument sound according to its temporal evolution [START_REF] Helmholtz | On the Sensations of Tone[END_REF], as early as 1885. This model led to the development of the attack-decay-sustain-release (ADSR) envelope [START_REF] Pinch | Analog Days: The Invention and Impact of the Moog Synthesizer[END_REF], which was originally proposed for synthesis. Notably, it takes only amplitude information into account and supposes all sounds evolve the same way. Luce and Clark [START_REF] Luce | Durations of Attack Transients of Nonpercussive Orchestral Instruments[END_REF] suggested that the attack depends on the rise time of the amplitude envelope. Consequently, most detection techniques proposed use different estimations of the amplitude envelope to try and detect the events dictated by the ADSR model. Hajda [START_REF] Hajda | A New Model for Segmenting the Envelope of Musical Signals: The relative Salience of Steady State versus Attack, Revisited[END_REF] proposed a new model for the segmentation of isolated sustained (nonpercussive) musical instrument sounds that uses the relationship between the amplitude envelope and the temporal evolution of the spectral centroid to define the theoretical boundaries between perceptually salient segments of musical sounds. This model was coined the amplitude/centroid trajectory (ACT).

In this work, we propose to use the ACT model to automatically segment isolated acoustic musical instrument sounds. Our main goal is the automatic detection of the boundaries of the regions defined in the model for many types of instruments with different modes of excitation. For such, we will verify whether the model applies to some percussive instruments as well. An important part of our contribution lies in an improved technique to accurately estimate the amplitude envelope evolution of sounds by means of cepstral smoothing with a method known as true envelope [START_REF] Röbel | Efficient Spectral Envelope Estimation and its Application To Pitch Shifting And Envelope Preservation[END_REF], usually employed to estimate the spectral en velope. However, our main contribution lies in the automatic segmentation of the sounds by detecting significant changes in Marcelo Caetano, Juan José Burred * , Xavier Rodet Analysis/Synthesis Team IRCAM {caetano,burred,rodet}@ircam.fr the features proposed in the ACT model, that only theoretically defines their boundaries. We compare our results with the baseline method proposed by Peeters [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF]. The next section introduces the classical acoustic model of the temporal evolution of musical instrument sounds and the following section presents techniques to estimate the amplitude envelope and two segmentation techniques. We also introduce the improved true amplitude envelope estimation. Then, we proceed by a description of the signal level manifestation of the acoustic phenomena that generate the events we wish to detect. This exposition leads to the presentation of the improved ACT model. We finally present our technique to automatically detect the boundaries of the regions, followed by the conclusions and future perspectives.

THE CLASSICAL ACOUSTIC MODEL

Historically, Helmholtz was the first to propose the segmentation of isolated acoustic musical instrument sounds according to their temporal evolution [START_REF] Helmholtz | On the Sensations of Tone[END_REF]. Helmholtz characterized what he called musical tone as a waveform that follows an amplitude envelope that consists of the attack, the steady state and the decay, as shown in Figure 1. During the attack, the amplitude increases from zero to its peak value. In the steady state portion the amplitude is constant and finally decreases back to zero during the decay. Helmholtz concluded that sounds that evoke the sensation of pitch possess fixed waveforms that do not change in the course of the tone, apart from the amplitude envelope, whose temporal evolution has great impact on the perception of the tone, according to him. We should notice that this model only takes into account temporal cues provided by the amplitude envelope to define perceptually salient features such as the attack, steady state and decay.

Later on, Robert Moog devised the ADSR envelope model, shown in Figure 2, while developing his synthesizer and it quickly became the standard way to describe the amplitude envelope generator functions [START_REF] Pinch | Analog Days: The Invention and Impact of the Moog Synthesizer[END_REF]. We should bear in mind that it was developed for synthesis purposes and, as such, it usually does not describe well the temporal evolution of most acoustical instrument sounds. However, most segmentation techniques [START_REF] Jensen | Envelope Model of Isolated Musical Sounds[END_REF], [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF] rely on the detection of these events/regions based solely on the use of the amplitude envelope. Particularly, the attack is notoriously thought as being dependent on the rise time of the amplitude envelope [START_REF] Luce | Durations of Attack Transients of Nonpercussive Orchestral Instruments[END_REF] and some authors use it as its definition [START_REF] Bello | A tutorial on onset detection in music signals[END_REF]. Therefore, we will present some amplitude envelope estimation techniques usually used in the detection of some of these events.

AMPLITUDE ENVELOPE ESTIMATION

The classic Helmholtz model led to the development of some segmentation techniques that only take temporal cues into ac count [START_REF] Jensen | Envelope Model of Isolated Musical Sounds[END_REF], [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF]. Notably, these methods rely on the estimation of the amplitude (or energy, which is amplitude squared) envelope and use it as detection function to estimate the boundaries of the regions defined by the model. An early attempt [START_REF] Schloss | On the Automatic Transcription of Percussive Music -From Acoustic Signal to High-Level Analysis[END_REF] consisted of a piece-wise linear approximation of the waveform. The amplitude envelope is created by finding and connecting the peaks of the waveform in a window that moves through the data. Jensen [START_REF] Jensen | Envelope Model of Isolated Musical Sounds[END_REF] proposed a method that fits curve shape approximations to model the amplitude envelope of the partials of an additive model of instrument sounds and later Skowronek [START_REF] Skowronek | Features for Audio Classification: Percussiveness of Sounds[END_REF] applied it to approximate the global amplitude envelope. In this section we will compare the most widely used techniques to estimate the amplitude envelope and we will present a much more reliable amplitude envelope estimation method that optimally finds a smooth function that approximately matches the peaks of the waveform using true envelope cepstral smoothing [START_REF] Röbel | Efficient Spectral Envelope Estimation and its Application To Pitch Shifting And Envelope Preservation[END_REF]. The amplitude envelope estimation techniques that will be presented are the classical low-pass filtering, RMS energy, and analytic signal amplitude demodulation, as well as frequency-domain linear prediction [START_REF] Athineos | Frequency-Domain Linear Prediction for Temporal Features[END_REF] (FDLP) and our own proposal, true amplitude envelope (TAE). The aim of this section is to show that the TAE estimation technique leads to better results for all cases shown.

Low-Pass Filtering (LPF)

Low-pass filtering is the most straightforward way of obtaining a smooth signal that follows the amplitude evolution of the original waveform. It is based on a classical amplitude demodulation envelope follower technique [START_REF] Bello | A tutorial on onset detection in music signals[END_REF], that low-pass filters a half-wave (hwr) or full-wave rectified (fwr) version of an amplitude modulated (AM) signal. The principle of amplitude modulation (AM) is that the amplitude changes of the signal carry the information we seek. There are many possible filter designs with different characteristics and the choice affects the quality of the final envelope. For instance, Jensen [START_REF] Jensen | Envelope Model of Isolated Musical Sounds[END_REF] proposes convolving the waveform with a Gaussian window function, resulting in a suboptimal estimation. Also, the cut-off frequency of the filter has a major impact on the result. High cut-off frequencies will likely produce an amplitude envelope with ripples and very low cut-off frequencies are less responsive to sudden amplitude changes.

RMS Energy

The RMS energy envelope is perhaps the most popular [START_REF] Tzanetakis | Sound analysis using MPEG compressed audio[END_REF], [START_REF] Xu | Automatic music summarization based on temporal, spectral and cepstral features[END_REF], [START_REF] Shao | Kankanhalli Automatic Music Summarization in Compressed Domain Proc[END_REF], [START_REF] Hajda | A New Model for Segmenting the Envelope of Musical Signals: The relative Salience of Steady State versus Attack, Revisited[END_REF] method for estimating the temporal evolution of the signal energy. The RMS energy envelope is based on the root mean square energy calculation and can be easily adapted to obtain an estimate of the amplitude envelope by simply applying it with a sliding rectangular window, as shown in equation (1) ( ) ( )

t x T t RMS T i i ∑ = = 1 2 1 (1)
where x i (t) is the i th local sample of the signal centered around t as seen through the window, t is the number of samples the analysis window moves, and T is the window length. The RMS is a special case of the generalized mean with exponent p=2 and as such, also functions as a sort of moving average, low-pass filter that smoothes out the signal. The analysis step t imposes a tradeoff between the temporal sampling rate of the envelope and how much information it represents. Small values of t react sooner to sudden changes in amplitude, while presenting ripple in more steady regions and larger values smooth out the ripples but tend to lag behind abrupt energy changes.

Analytic Signal

The Hilbert transform is part of a signal processing technique for amplitude demodulation [START_REF] Potamianos | A Comparison of the Energy Operator and the Hilbert Transform Approach to Signal and Speech Demodulation[END_REF]. The Hilbert transform of a signal x(t) is defined as

( ) ( ) t t x t x π 1 ˆ * = (2)
where * stands for convolution. Using equation ( 2), we can define the analytic signal z(t) as

( ) ( ) ( ) ( ) () [ ] t j t r t x j t x t z θ exp ˆ= + = (3)
The analytic signal is useful for envelope detection since its modulus r(t) and time derivative of the phase θ(t) can serve as estimates for the amplitude envelope and instantaneous frequency of x(t) under certain conditions. Notably, if the Hilbert transform of x(t) is equal to its quadrature signal [START_REF] Potamianos | A Comparison of the Energy Operator and the Hilbert Transform Approach to Signal and Speech Demodulation[END_REF], then the estimates are equal to the actual information signals [START_REF] Potamianos | A Comparison of the Energy Operator and the Hilbert Transform Approach to Signal and Speech Demodulation[END_REF]. Synthetic (i.e., AM) signals can be constructed to have this property, but there is no reason to expect that acoustic musical instrument sounds also present it. A more realistic condition is verified when we are dealing with narrowband signals, which is rarely the case for musical instrument sounds. The Hilbert transform can be effectively used to extract the amplitude envelope of individual partials if applied to each frequency bin of the STFT, but when applied to the whole signal it is equivalent to trying to demodulate several AM signals at the same time.

Frequency-Domain Linear Prediction (FDLP)

Traditional linear prediction [START_REF] Makhoul | Linear prediction: A tutorial review[END_REF] estimates the spectral envelope from the time-domain signal. The idea behind FDLP [START_REF] Athineos | Frequency-Domain Linear Prediction for Temporal Features[END_REF] is to exploit time-frequency duality to extract the temporal amplitude envelope by applying linear prediction to a spectral representation. In particular, the used spectral representation is the discrete cosine transform (DCT), since it is real-valued. The envelope peaks, whose number and width are determined by the model order, will now be their frequency domain counterparts, the rectified waveform peaks. Thus, the model order has to be adjusted with respect to the temporal structure of the signal, and not to the formant structure of the spectrum.

True Amplitude Envelope (TAE)

True envelope [START_REF] Röbel | Efficient Spectral Envelope Estimation and its Application To Pitch Shifting And Envelope Preservation[END_REF] is a method for spectral envelope estimation that has shown better performance than linear prediction or cepstral methods such as discrete cepstrum, both in terms of accuracy and ease of model order selection. It is based on iteratively estimating the cepstrum and adapting it in such a way that the peak matching is maximized and inter-peak valleys are avoided. Here, we propose to use a dual of true envelope in the time domain. The time domain signal is subjected to the algorithm in stead of the Fourier spectrum. In this way, the amplitude envelope is expected to match the amplitude peaks more closely than the previously introduced methods. It is important to note that the optimal model order will be directly proportional to the fundamental frequency, rather than inversely proportional as in spectral envelope estimation.

Figure 3 shows a full-wave rectified (fwr) version of a bass clarinet and acoustic guitar waveforms and the amplitude envelope estimates. We are looking for the estimation that best fits the model waveform, that is, that follows the amplitude evolution most closely matching the peaks. It should be noted that all curves, except TAE, were normalized and scaled to the maximum of the fwr waveform and that RMS was also low-pass filtered to eliminate the ripples during mostly the sustain part. Upon close inspection, Figure 3 shows that TAE renders the best fit, closely following the peaks without ripples.

Segmentation Techniques Using the Amplitude Envelope

Finally, we present two previously proposed techniques to automatically segment individual musical instrument sounds based solely on the amplitude envelopes, namely derivatives [START_REF] Jensen | Envelope Model of Isolated Musical Sounds[END_REF], [START_REF] Skowronek | Features for Audio Classification: Percussiveness of Sounds[END_REF] and efforts [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF]. Both methods try to detect the inflection points of the amplitude envelope based on the assumption that the amplitude envelope changes correspond to the boundaries of the regions we are looking for. Notably, these models define the attack as the rise time of the amplitude envelope, like other authors [START_REF] Bello | A tutorial on onset detection in music signals[END_REF], [START_REF] Luce | Durations of Attack Transients of Nonpercussive Orchestral Instruments[END_REF]. Skowronek [START_REF] Skowronek | Features for Audio Classification: Percussiveness of Sounds[END_REF] proposed a segmentation method based on their attack-decay-and-sustain-release (A-D-&-S-R) model. They obtain an approximation of the amplitude envelope and use it together with its first derivative to estimate the boundaries of the three regions defined as start of attack (soa) and end of attack (eoa); and start of release (sor) and end of release (eor), as exemplified on the left of Figure 4. The right of Figure 4 shows the method of efforts, introduced by Peeters [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF]. The method of efforts segments musical instrument sounds using the attack/rest model, whereby we separate the attack portion from the rest of the sound, be it sustained or percussive. First we divide the slope DAFX-4 Figure 4: Depiction of the method of derivatives (on the left) and efforts (on the right) for the automatic segmentation. Figures respectively from [START_REF] Skowronek | Features for Audio Classification: Percussiveness of Sounds[END_REF] and [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF].

corresponding to the rise time into N equal intervals according to the amplitude incremental values (called thresholds). We could define the start and end of the attack as fixed percentages of the minimum and maximum amplitude values, very much like Luce and Clark [START_REF] Luce | Durations of Attack Transients of Nonpercussive Orchestral Instruments[END_REF]. Instead, in this method, the start of attack and end of attack are estimated according to the slope of the rise region. So we calculate a piecewise measure of the slope for each threshold jump by measuring how long it takes to go from one threshold to the next (called efforts). The selected threshold is the one whose value is smaller than M times the mean threshold for both the start and end of the attack. They recommend to use M = 3 [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF]. We will show in the next section why methods that rely solely on the amplitude fail to segment sounds into perceptually meaningful events because they use restricted information.

TEMPORAL EVOLUTION

The segmentation of musical instrument sounds depends on the correct detection of the boundaries of the regions. Clearly we need a good definition of the regions to be detected in order to be able to estimate them. The first problem we face is that not all instruments contain the same temporal events, so we cannot expect, for example, to be able to estimate the sustain part for a percussive instrument sound. This is where a robust model plays a key role in defining the segments and their boundaries. The most important aspect to be taken into account is a clear separation of cause and effect. The amplitude envelope is merely the description of one of the results of the source-filter interaction. It is fruitless to attempt to detect the boundaries of the events we want to estimate without a proper causal description. We must find the signal level counterparts of the physical events to properly estimate them. The technique we present in Section 5 uses spectro-temporal cues at the signal level left by the physical/gestural events to correctly segment them. The difficulty in this approach is that each instrument has its own particularities. Ideally, we search for a model that is robust enough to describe the signal level manifestations of as many types of instruments as possible. We will begin by describing the general model we will consider, namely, the source filter model, originally developed for speech; and then the physical characteristics of the events we aim to describe, and then later in Section 5 we will present the ACT model [START_REF] Hajda | A New Model for Segmenting the Envelope of Musical Signals: The relative Salience of Steady State versus Attack, Revisited[END_REF] for specific classes of instruments. Finally, Section 6 shows how we use the ACT model to obtain significant estimates of the boundaries from spectro-temporal traces left by the physical gestures.

Source-Filter Model

Listeners use many acoustical properties to identify sonic events, such as the spectral shape, formant frequencies, attack and/or onset and decay and/or offset, noise, among others [START_REF] Handel | Timbre perception and auditory object identification[END_REF]. The cues to identification and timbre vary across notes, durations, intensities and tempos. One model of sound production is based on two possibly interactive components, the source and the filter. The basic notion is that the source applies excitation energy to generate a vibration pattern composed of several vibration modes (modeled as sinusoidal components). This pattern is imposed on the filter, which acts to modify the relative amplitudes of the components of the source input. We obtain estimates of the source and the filter by calculating the spectral envelope, which is a smooth curve that approximately matches the peaks of the spectrum. The peaks of the spectral envelope (also called formants in voice research) correspond roughly to the vibration modes of the source-filter model. The number and absolute position of spectral peaks in frequency is important for musical instrument (source) identification. The relationship between fundamental frequency and timbre is readily apparent in some acoustic instruments. The clarinet, for instance, has three distinct registers, that is, three distinct pitch ranges with three different timbral characteristics. It is remarkable that a single instrument can have such a variety of timbres, but the example of the clarinet proves the impact of a resonating body on an instrument's timbre and temporal evolution. Resonators, by their nature, tend to amplify certain frequencies louder than others. These resonant frequency regions, called formants, are uniquely related to the size and shape of the instrument and its resonator. The relationship between applied energy and timbre is relatively clear. As more energy is input to the instrument, higher modes of vibration are achieved such that more partials are present in the frequency spectrum. This is why a note played forte is not just louder than piano, but also brighter in timbre.

In this work, we are primarily interested in the temporal evolution of the sounds, so we are going to examine the source-filter model from a temporal perspective. Figure 5 shows a simplified schematic view of the temporal evolution of the excitation (dotted line) and the resulting amplitude envelope followed by the sound (solid line) for two markedly distinct classes of excitation Step-like excitation corresponds to playing modes whose energy is supplied for some length of time before being interrupted, while impulse-like is when energy is supplied in a short burst. The former typically applies to sustained sounds resulting from bowed strings and blown pipes, and the latter to percussive excitations such as plucked strings or struck instruments, although blown or bowed staccato notes would probably be better described by it. The beginning and end of the energy supply for each excitation mode are highlighted by long arrows and the short arrow marks the maximum amplitude attained by the sound. Next, we try to make a connection between the mainly physical events such as onset, attack, decay, sustain, release and offset and its model counterparts in connection with the excitation and resulting temporal evolution presented earlier. The idea is to find signal level manifestations of the physical gestures. Our main goal is to show that these events cannot be solely described by the amplitude envelope of most sounds, such that we need a more complete model to appropriately segment them.

Signal-Level Manifestation of Physical Events

Musical instruments are mechanical systems that by themselves are at equilibrium. They need an external source of energy input to produce sound. In general terms, all acoustic musical instruments have one (or more) method for applying mechanical energy to the system, herein termed the excitation method. Pianos have keys connected to hammers that strike a set of tuned strings. Violins have strings that are bowed or plucked. Clarinets have a mouthpiece with a single reed that, when blown, creates a vibrating column of air. Different modes of excitation will generally lead to perceptually different attacks.

The connection between intensity (dynamics), frequency and amplitude envelope is far less obvious. Before the onset, the instrument is in a state of equilibrium. As with all mechanical systems, there is a certain amount of resistance or inertia that keeps the instrument from vibrating on its own. Performers must overcome that inertia before their instrument will sound properly. The more energy a performer uses, the faster the resistance is overcome and the faster the instrument reaches its steady state vibration. For example, different fingerings on a wind instrument produce different lengths of air columns -longer columns mean more mass to vibrate. We know that large masses have more inertia to overcome, but also have more momentum once they are in motion. Thus, low notes have a longer attack and a longer release than high ones. In this section we define the physical/gestural events that generate/define each perceptually different region of the temporal evolution of musical instrument sounds and, more importantly, we find signal-level manifestations of the physical events.

Attack

The attack is perhaps the only event that is present in all sounds independent of the mode of excitation. The attack corresponds to the initial excitation of the instrument. The beginning of the attack is perhaps best characterized by the transition between no event and event (or more properly background noise and event for recordings, i.e., signals). This is usually termed onset. The end of the attack is more difficult to define since it depends on the physical gesture. Notably, transients occur until a permanent resonance mode is attained. For some instruments, we can make a clear distinction between the end of the attack and the beginning of the resonance. The time period when a hammer touches the piano strings would be the attack and the moment the standing wave pattern establishes itself in the string marks the beginning of the resonance mode. For a bowed string it is similar.

From the moment when the bow first touches the string (onset) until the string enters a resonance regime with the bow we can devise two physically and perceptually distinct events. The end of the attack happens before the resonance. For a tube (blown instruments) the situation is similar.

Decay

The decay supposedly corresponds to a decrease of energy after the attack during which the permanent excitation regime (such as a standing wave vibration pattern) is already established. This is the region where the amplitude evolution of a percussive instrument sound constantly decays due to losses and strays from constant (step-like) excitation patterns (blown/bowed strings), where energy is repeatedly input to the instrument during a period of time. When we look closely, the decay remarkably contains standing wave patterns, even though the amplitude is decreasing.

In plucked strings, for example, there is a clear spectral pattern that remains constant throughout and that is perceptually important. The decay contrasts with the amplitude evolution of a blown or bowed instrument, whose standing wave vibration pattern coincides with a more or less constant amplitude.

Sustain

The sustain part usually corresponds to the region where the system (musical instrument) is constantly exited with external energy. It is usually defined in terms of approximately constant amplitude. Perceptually, though, it is not reasonable to expect the region where a standing wave vibration pattern manifests as spectrally constant resonances (similar spectral shape) to be sufficiently described solely by the amplitude. Therefore we suggest that constant excitation instruments (bowed and blown, among others), where the energy and the spectral information remain roughly constant, present a sustained part.

Release

The release phase admits several interpretations, and its definition has not been consistent among authors. On the one hand, it can refer to the release of the excitation, such that the release segment is the interval between the time instant where the energy ceases to be supplied and the vibrations dying out (offset). This definition is common for sustained sounds (step-like excitation), but not always used for non-sustained sounds (impulse-like excitation), because in the latter case release would be equivalent to decay. On the other hand, it can refer to an intentional interruption of the vibration by the player. Most notably, in stringed keyboard instruments, this corresponds to the release of a key, which causes the damper to stop string vibrations. To avoid confusion between these very different physical events, we will use the following conventions. Release will correspond to the release of excitation in sustained instruments. Then, we will introduce a new segment called interruption to account for the case of intentional interruption of vibrations in non-sustained instruments. 2DAFX-6

So we see that although these regions share common spectrotemporal aspects, they cannot be defined simply in terms of energy evolution. Also, the accuracy of estimation depends on a detection function that captures the essence of the model and exhibits peculiar behavior at the boundaries of the regions that allow for a robust automatic detection.

AMPLITUDE/CENTROID TRAJECTORY MODEL

The classical Helmholtz model breaks down when we examine musical instrument sounds on a small scale. When the harmonic content of sound is examined with the STFT over small time periods, we discover that, contrary to the Helmholtz model, a sound's spectrum changes profoundly over time. During the attack portion of a sound, harmonic content may change rapidly and unpredictably. This phenomenon is called the initial transient. During the release, upper partials tend to disappear more quickly before the entire sounds fades away. While the sustain portion of the sound, when it exists, is certainly more stable than the attack or decay, it is hardly as static as Helmholtz would suggest. Clearly, the basic premise of the classical Helmholtz model -a static spectral envelope with a fixed amplitude envelope temporal evolution -is by no means an accurate and robust characterization of a wide range of acoustic musical instrument sounds. All these facts suggest that, in order to better understand the temporal evolution of sounds, we need a model that accounts for spectro-temporal changes. The vast majority of research in sound perception has focused either on the acoustic properties of musical instruments [START_REF] Risset | Exploration of timbre by analysis and synthesis, The Psychology of Music[END_REF] or on the perception of sounds as unveiled by psycho-acoustic experiments [START_REF] Mcadams | Perceptual Scaling of Synthesized Musical Timbres: Common Dimensions, Specifities and latent subject Classes[END_REF]. The challenge we face today is to find the link between the two in order to be able to manipulate the sounds in a more perceptually meaningful way. A classical example is Risset's discovery that brassy trumpet sounds present a broader spectrum. The spectral centroid, defined in equation (4) reflects spectro-temporal acoustic properties at the signal level. Therefore, brassy sounds can be characterized as presenting a higher centroid value.

( ) (

= = = M b b M b b b t a t a t f t C 1 1 (4) ) ( ) ( ) ∑ ∑
Here C(t) is the time-varying centroid, f b (t) is the frequency in Hz and a b (t) is the amplitude of frequency band b up to the M th band computed.

Hajda [START_REF] Hajda | A New Model for Segmenting the Envelope of Musical Signals: The relative Salience of Steady State versus Attack, Revisited[END_REF] proposed a segmentation model he dubbed the amplitude/centroid trajectory (ACT) that relies on both the amplitude envelope and temporal evolution of the spectral centroid. In this model, the spectral centroid gives information about the excitation indirectly. The sudden transition characteristic of the onset reflects as a brief broadening and narrowing of the spectrum, causing the centroid to drop until the steady state resonance establishes itself, bringing the centroid up again to a somewhat steady value. For step-like excitation, the release is the moment when the player stops supplying energy to the instrument. This reflects a new drop in the centroid because the higher partials tend to fade before the lower ones, until the sound/note fades away, characterizing the offset. Figure 6 depicts the re gions (letters) and boundaries (numbers) of the ACT model for sustained (step-like excitation) sounds in part a) and an extension of the ACT model for percussive (impulse-like excitation) sounds in part b). In the figure, BN stands for background noise, A for attack, T for transition, S is sustain, D is decay, R is release and I is interruption. The boundaries are the onset [START_REF] Handel | Timbre perception and auditory object identification[END_REF], end of attack (2), begin of transient ( 3), begin of release/interruption (4) and offset [START_REF] Collins | Investigating Computational Models of Perceptual Attack Time[END_REF]. Using this model, Hajda defines the attack as that part of the signal from onset during which the amplitude increases and the centroid decreases. Pre-attack noise is indicated by more or less uncorrelated fluctuations of both amplitude and centroid. According to the model, the attack ends when the centroid slope changes direction. A new segment, the attack/steady state transition, is defined as that segment immediately following the attack during which the amplitude continues to increase and the centroid increases overall. The sustain begins when the amplitude has achieved a local maximum; during this segment, the amplitude and centroid vary in a more or less monotonic fashion [START_REF] Hajda | A New Model for Segmenting the Envelope of Musical Signals: The relative Salience of Steady State versus Attack, Revisited[END_REF]. The release (or interruption) begins when both the amplitude and centroid decrease. Our approach uses the ACT model proposed by Hajda to automatically segment the temporal evolution of acoustic musical instrument sounds by detecting these five boundaries for as many types of sounds as possible, so we will test the model with sustained and percussive sounds, even though Hajda concluded that the ACT model does not seem to apply to most percussive sounds. Our technique uses the improved true amplitude envelope estimation that we developed instead of the RMS originally used [START_REF] Hajda | A New Model for Segmenting the Envelope of Musical Signals: The relative Salience of Steady State versus Attack, Revisited[END_REF].

AUTOMATIC SPECTRO-TEMPORAL SEGMENTATION

Here Figure 7 presents the regions automatically detected by our proposed method and compare our results with the baseline method by Peeters [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF]. The importance of these figures is twofold. Firstly they should enable us to verify whether the ACT model works for the sustained (blown and bowed) and percussive (struck and plucked) sounds tested. Secondly, we will present the results of our automatic segmentation method and compare them to the baseline method. So we show the fwr waveform outlined by the TAE (solid line) and the centroid (dashed line) at the top and the corresponding spectrogram at the bottom. We included the spectrogram as an important visual aid in interpreting the segmentation results. That is, the spectrogram is intended to allow an intuitive visual inspection of the segments. The segmentation boundaries are shown as five solid vertical lines for our method and two dashed lines for the baseline method. The five boundaries correspond to those shown in Figure 6 while the two detected by the baseline method should represent what they de- fined as the attack (rise time), roughly corresponding to the onset and beginning of steady state in our model. We use an automatic onset detection method [START_REF] Röbel | A New Approach to Transient Processing in the Phase Vocoder[END_REF] to accurately define the first boundary, the onset (1). Then, the offset ( 5) is defined as the last point the TAE attains the same energy (amplitude squared) as the onset. The next step is the definition of the beginning of the sustain (3) and release (4) segments with a modified adaptive version of the effort method, explained bellow. Finally, we find the first local minimum between (1) and ( 3) and define it as the end of the attack [START_REF] Iverson | Isolating the Dynamic Attributes of Musical Timbre[END_REF]. We should notice that we allow the same point to define the boundary of two distinct contiguous regions. This could happen remarkably during the transition between the end of the attack (2) and the beginning of sustain (3) and we interpret it as merely signifying that the transition is too short to be detected as a separate segment. We employed adaptive efforts to independently measure the initial and final slopes because they exhibit typically different behaviors. This adaptive version is based on the original method of efforts proposed by Peeters [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF] because it is more robust than the derivatives [START_REF] Jensen | Envelope Model of Isolated Musical Sounds[END_REF], [START_REF] Skowronek | Features for Audio Classification: Percussiveness of Sounds[END_REF], too sensitive to ripples. The adaptive effort is an indirect measure of the slope. We measure from the onset forward how many efforts summed are larger than the mean effort. Then we use this number as M in the original method. We do the same from the offset backwards. The results shown for the baseline method correspond to the original proposal [START_REF] Peeters | A large set of audio features for sound description (similarity and classification) in the CUIDADO project[END_REF].

We chose the instruments to try to display the result of the automatic segmentation for the four main classes considered, namely, blown pipe and bowed string (sustained), and plucked string and struck (percussive). These are challenging choices if we consider the original proposed application of the model and are meant to test its robustness. We will examine the results case by case. The clarinet (blown pipe) represents the best case scenario. It clearly fits the model perfectly and shows that the automatic segmentation accurately detects the boundaries because they are strongly present. Notably, the end of the attack (2), beginning of sustain (3), and begin of release (4) agree well with a visual inspection of the spectrogram. This seems to be the case for all blown instruments we tested, confirming the original model. We should notice that the baseline method detects both (2) and (3) with a considerable lag.

The cello shows that both the amplitude and the centroid might not behave like predicted for slow attacks typical of bowed strings. If we examine the spectrogram, the onset (1), release (4) and offset (5) were accurately detected. However, the end of the attack (2) is more difficult to determine, both visually and from the detection function. The beginning of the sustain is remarkably difficult to estimate for this case because neither the amplitude nor the centroid seem to confirm the visual inspection of the spectrogram. Bowed strings typically present a crescendo behavior reflected in the amplitude evolution that usually does not correspond to the vibration pattern measured by the centroid. Also, the centroid does not remain stable during the excitation mainly due to the vibrato, also typically present in bowed strings. It is evident, though, that the baseline method presents estimates that are far off. All bowed strings that we tested exhibited similar behavior.

The guitar shows that some plucked strings might exhibit a typical behavior that roughly fits the model. The steady vibrational pattern captured by the centroid is a typical example of how the decay cannot be characterized solely by the amplitude. However, since the attack is very fast, the boundaries between onset (1), end of attack (2) and beginning of decay (3) are blurred. This is a typical example of when (2) and (3) coincide. Also, it correctly detects the interruption (4), as previously explained. The baseline method also presents considerable detection lags in this case, overestimating the attack. Not all plucked strings tested, however, presented such clearly steady vibrational patterns that could be correctly identified by the model.

Finally, the marimba is a clear example of when the model assumptions break down. Even though the spectrogram would lead us to visually guess the boundaries of some of the events, neither the centroid nor the amplitude envelope present a typical enough behavior to allow a correct detection of the regions according to the model. Notably, the vibration of the stick used to strike the key is clearly audible during the attack portion of the sound, constituting an important perceptual cue to the identification of struck percussions. When the vibrations of the instrument itself dominate, it is already too late to try and detect events. There are only two visible boundaries detected by our method. The onset (1) seems to be the only one correctly detected. End of attack (2) and beginning of decay (3) were bundled up. There is no beginning of interruption [START_REF] Peeters | Instrument Sound Description in the Context of MPEG-7[END_REF]. In this case, the baseline method does not seem to find two separate points either, detecting the beginning too late. We would probably need to deconvolute the vibration of the stick from the key to be able to apply the model.

CONCLUSIONS AND FUTURE PERSPECTIVES

This work presents a method for automatically segmenting the temporal evolution of isolated acoustic musical instrument sounds using a model that takes spectro-temporal cues to correctly detect the boundaries of the regions. Even though the ACT model was originally proposed for sustained instrument sounds, we tested whether it can also be applied to some types of percussive sounds. An important contribution of this work is an improved amplitude envelope estimation technique (TAE) based on true envelope that optimally fits a curve that approximately matches the peaks of the waveform. TAE proved to be superior to the other methods tested. We found that the ACT model can be applied to plucked strings within certain boundaries. Finally, the automatic detection technique proposed performed better than the baseline method for most examples shown. We verified empirically that it outperforms the baseline method for all cases when the model fits well the type of instrument tested. More examples on http://recherche.ircam.fr/anasyn/caetano/seg.html.

Future perspectives of this work could include more robust detection techniques that potentially apply universally to all sounds, sustained or otherwise, or even separate automatic detection techniques devoted to each type of excitation.

Figure 1 :

 1 Figure 1: The Helmholtz model of the temporal evolution of acoustic musical instrument sounds. Helmholtz defined that the amplitude envelope can be divided into attack, steady state and decay. Figure from [17].

Figure 2 :

 2 Figure 2: ADSR model applied to a wind instrument to explain its temporal evolution.

Figure 3 :

 3 Figure 3: Full-wave rectified (fwr) waveform and amplitude envelope estimation methods; true amplitude envelope (TAE), frequency-domain linear prediction (FDLP), root-mean square (RMS) and low-pass filtering (LPF).

Figure 5 :

 5 Figure 5: Simplified temporal evolution of the excitation (dotted line) and amplitude envelope (solid line) for two distinct classes of excitation modes. In a) we see the typical excitation and amplitude envelope resulting from the step-like excitation (e.g., blown/bowed) and in b) for impulse-like excitation (e.g., plucked/struck). The beginning and end of the excitation are marked with long arrows, while the short arrow shows the maximum amplitude attained by the resulting amplitude envelope.

Figure 6 :

 6 Figure 6: Temporal evolution of the spectral centroid and amplitude envelope for two classes of excitation modes. The figure shows the regions (letters) and their boundaries (numbers).

Figure 7 :

 7 Figure 7: Full-wave rectified (fwr) waveform, TAE and centroid (top) and spectrogram (bottom) for sustained (blown and bowed) and percussive (struck and plucked) instruments.

Note that, while rare, it is also possible for the release phase of a sustained instrument to be followed by an interruption phase, such as when a violinist intentionally interrupts the vibrations of the strings after having stopped supplying energy to them by bowing.

ACKNOWLEDGMENTS

This work is supported by the Brazilian Governmental Research Agency CAPES (process 4082-05-2). The first author would like to thank Stephen McAdams for the inspiration.