Jun Wu
email: wu@hil.t.u-tokyo.ac.jp

Stanisław Emmanuel Vincent
email: emmanuel.vincent@inria.fr

Stanislaw Andrzej Raczynski
email: raczyns-ki@hil.t.u-tokyo.ac.jp

Takuya Nishimoto
email: nishimotz@olarbee.com

Nobutaka Ono

Shigeki Sagayama
email: sagayama@hil.t.u-tokyo.ac.jp

Stanisław A Raczyński

Andrzej Raczyński

Keywords: Instrument identification, Harmonic model, attack model, EM algorithm, PCA, SVM I

come   L'archive ouverte pluridisciplinaire

INTRODUCTION

olyphonic musical instrument identification consists of estimating the pitch, the onset time and the instrument associated with each note in a music recording involving several instruments at a time. This is often addressed by conducting multiple pitch estimation first, then classifying each note into an instrument class using suitable timbre features [START_REF] Burred | Dynamic spectral envelope modeling for timbre analysis of musical instrument sounds[END_REF][START_REF] Klapuri | Analysis of musical instrument sounds by source-filter-decay model[END_REF][START_REF] Kinoshita | Musical sound source identification based on frequency component adaptation[END_REF][START_REF] Eggink | Application of missing feature theory to the recognition of musical instruments in polyphonic audio[END_REF].

Multiple pitch estimation is the task of estimating the fundamental frequencies and the onset times of the musical notes simultaneously present in a given musical signal. It is considered to be a difficult problem mainly due to the overlap between the harmonics of different pitches, a phenomenon common in Western music, where combinations of sounds that share some partials are preferred. Several approaches have been proposed, including perceptually motivated [START_REF] Hartmann | Pitch, periodicity, and auditory organization[END_REF][START_REF] Klapuri | Multiple fundamental frequency estimation based on harmonicity and spectral smoothness[END_REF][START_REF] Wu | A multipitch tracking algorithm for noisy speech[END_REF][START_REF] Tolonen | A computationally efficient multipitch analysis model[END_REF], parametric signal model-based [START_REF] Davy | Bayesian analysis of western tonal music[END_REF][START_REF] Chazan | Optimal multi-pitch estimation using the EM algorithm for co-channel speech separation[END_REF], classification-based [START_REF] Poliner | A discriminative model for polyphonic piano transcription[END_REF] and parametric spectrum model-based [START_REF] Goto | A real-time music-scene-description system: Predominant-F0 estimation for detecting melody and bass lines in real-world audio signals[END_REF][START_REF] Raczynski | Multipitch analysis with harmonic nonnegative matrix approximation[END_REF][START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF][START_REF] Vincent | Adaptive harmonic spectral decomposition for multiple pitch estimation[END_REF][START_REF] Yeh | Multiple fundamental frequency estimation and polyphony inference of polyphonic music signals[END_REF] algorithms. Parametric spectrum model-based algorithms represent the power spectrum or the magnitude spectrum of the observed signal as the sum or the mixture of individual note spectra or harmonic partial spectra and perform parameter estimation in the Maximum Likelihood (ML) sense. These algorithms are particularly suitable in the context of polyphonic instrument identification since they do not only provide the pitch of each note but also additional parameters encoding part of its timbre.

Timbre features have been widely investigated for the classification of isolated notes or single-instrument recordings and gradually applied to polyphonic recordings. Typical features computed on the signal as a whole include power spectra [START_REF] Vincent | Instrument identification in solo and ensemble music using independent subspace analysis[END_REF], spectral or cepstral features [START_REF] Brown | Computer identification of musical instruments using pattern recognition with cepstral coefficients as features[END_REF][START_REF] Agostini | Musical instrument timbres classification with spectral features[END_REF] as well as temporal features [START_REF] Eronen | Musical instrument recognition using cepstral coefficients and temporal features[END_REF]. These features are not directly computable from the parameters of a multiple pitch estimation model. By contrast, timbre features have been derived in an unsupervised fashion from the amplitudes of the harmonic partials in [START_REF] Loureiro | Timbre classification of a single musical instrument[END_REF][START_REF] Hourdin | A multidimensional scaling analysis of musical instruments' time-varying spectra[END_REF][START_REF] Sandell | Perceptual evaluation of principal-component-based synthesis of musical timbres[END_REF] either via Multidimensional Scaling (MDS) or Principal Component Analysis (PCA). Supervised timbre models involving a source-filter-decay model or a dynamic statistical model of the amplitudes of the partials trained over labeled training data were also considered in [START_REF] Burred | Dynamic spectral envelope modeling for timbre analysis of musical instrument sounds[END_REF][START_REF] Klapuri | Analysis of musical instrument sounds by source-filter-decay model[END_REF]. Classification is then performed either via the Euclidean distance between the feature vectors or via maximum likelihood (ML) under the above models. In addition to their ease of use in the context of multiple pitch estimation, these algorithms reduce the dimension of the timbre parameter set, resulting in increased robustness with respect to parameter estimation errors. Feature weighting techniques were proposed in [START_REF] Kinoshita | Musical sound source identification based on frequency component adaptation[END_REF][START_REF] Eggink | Application of missing feature theory to the recognition of musical instruments in polyphonic audio[END_REF][START_REF] Kitahara | Instrument identification in polyphonic music: Feature weighting to minimize influence of sound overlaps[END_REF] to further improve robustness by associating a smaller weight to the parameters of overlapping partials, which are likely to be less accurately estimated.

While the attack part of musical notes is essential for timbre perception [START_REF] Eronen | Musical instrument recognition using cepstral coefficients and temporal features[END_REF], the above multiple pitch estimation and timbre feature models have focused on the representation of harmonic partials only. The attack part consists of an inharmonic sound and may be characterized in particular by its spectral envelope and its power, both of which depend on the instrument. Designing an instrument model able to deal both with harmonic and inharmonic features is essential for reflecting the timbre Polyphonic pitch estimation and instrument identification by joint modeling of sustained and attack sounds characteristics of any musical instrument. In [START_REF] Itoyama | Integration and adaptation of harmonic and inharmonic models for separating polyphonic musical signals[END_REF], a joint parametric harmonic and non-parametric inharmonic model was proposed and used for source separation given the pitch and instrument of all notes. In [START_REF] Wu | Musical instrument identification based on harmonic temporal timbre features[END_REF], we defined a joint parametric model of harmonic and attack sounds but considered timbre features derived from the harmonic part only. Therefore, attack timbre features have not been exploited for polyphonic musical instrument identification to date.

In this article, we propose an algorithm for polyphonic pitch estimation and instrument identification by joint modeling of harmonic and attack sounds. At first a flexible harmonic model is proposed to model the harmonic and attack parameters of musical notes via a mixture of time-frequency Gaussian distributions. These parameters are then estimated from a given recording together with the time-varying fundamental frequency using the Expectation-Maximization (EM) algorithm. Timbre features are subsequently derived by PCA from the model parameters after suitable logarithmic transformation and normalization. Finally, instrument classification is performed for each note via a Support Vector Machine (SVM)-based classifier instead of Euclidean distance or likelihood. We thereby extend our preliminary paper [START_REF] Wu | Musical instrument identification based on harmonic temporal timbre features[END_REF] by providing a more detailed treatment of the model, defining more efficient timbre features and separately evaluating the resulting performance in terms of pitch estimation and instrument identification. Experimental results show that the proposed features outperform the features in [START_REF] Wu | Musical instrument identification based on harmonic temporal timbre features[END_REF].

The overall flowchart of the proposed system is illustrated in Figure 1. The output of the proposed system is the estimated collection of pitches underlying the musical signal and the different colors represent different instruments. The structure of the rest of this article is as follows. In Section II, the joint model of sustained and attack sounds is introduced. In Section III, parameter estimation and classification algorithms are presented. Experimental results on synthetic and real-world data are shown in Section IV. Finally, the conclusion is made in Section V.

II. J OINT MODELING OF SUSTAINED AND ATTACK SOUNDS

We adopt the same two-stage approach as a majority of algorithms [START_REF] Burred | Dynamic spectral envelope modeling for timbre analysis of musical instrument sounds[END_REF][START_REF] Klapuri | Analysis of musical instrument sounds by source-filter-decay model[END_REF][START_REF] Eggink | Application of missing feature theory to the recognition of musical instruments in polyphonic audio[END_REF][START_REF] Kitahara | Instrument identification in polyphonic music: Feature weighting to minimize influence of sound overlaps[END_REF]: a multipitch estimation stage provides the estimated pitch of all notes in the recording and an instrument identification stage classifies each note into a specific instrument category. However, while most algorithms rely on a different model for each stage, we use the same model for both stages. This model describes both the spectral and the temporal envelope by a mixture of Gaussian distributions as in [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF] with significant improvements detailed hereafter. The main difficulty of polyphonic musical instrument identification is the overlapping of observed partials from different timbres. So an applicable model should also be able to associate the corresponding partials with specific timbres.

In the following, we assume that the input signal is sampled at 16 kHz and represented by its power constant-Q transform [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF]. The transform is computed using Gabor-wavelet basis functions with a time resolution of 16 ms for the lowest subband. The time resolution is set to 16 ms for all subbands. The lower bound of the frequency range and the frequency resolution are 60 Hz and one semitone, respectively, as in [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF].

Denoting by x and t the frequency bin and time frame indexes respectively, the proposed model approximates the observed nonnegative power spectrogram W(x,t) by a mixture of K nonnegative parametric models, each of which represents a single musical note. Every note model is composed of a harmonic part, itself consisting of N harmonic partials, and an attack part. Figure 2 depicts the spectrogram of a piano note with the attack part being marked with a rectangle. The power spectrogram of the kth note is represented as .

(

) 1
where is the total energy of the harmonic part, represents the spectrogram of the nth harmonic partial and the spectrogram of the attack part. The list of model parameters is shown in Table 1. Bandwidth of the partials of the kth note Coefficient of the spectral envelope of the attack of the kth note, jth frequency band Table 1. Free parameters of the proposed model.

A. Harmonic Model

The proposed model for the harmonic part is similar to [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF]. However, in contrast to [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF], the time-domain envelope is assumed to be different for each partial. This modification has significant impact on instrument identification since differences between the temporal evolution of the partials contribute to the characterization of timbre [START_REF] Klapuri | Analysis of musical instrument sounds by source-filter-decay model[END_REF].

The harmonic model of each partial is defined as the product of a spectral model and a temporal model . Due to the use of a Gabor constant-Q transform, the spectral harmonic model follows a Gaussian distribution, as illustrated in Figure 3. The bandwidth is approximately equal for all partials on a log scale so a constant standard deviation can be used. Given the fundamental log-frequency of the kth note, the log-frequency of the nth partial is given by . This results in [START_REF] Klapuri | Analysis of musical instrument sounds by source-filter-decay model[END_REF] where is the relative power of the nth partial satisfying where is the mean of the first Gaussian, which is considered as the onset time, is the weight parameter for each time instant, which allows the temporal envelope to have a variable shape for each harmonic partial, and is the spacing between successive sampling instants, which is proportional to the note duration . The weight parameters are normalized as 1 .

(5) The Dirichlet distribution is used as a prior distribution over and (6) [START_REF] Wu | A multipitch tracking algorithm for noisy speech[END_REF] where Γ is the gamma function, and denote the expected values of and and and regulate the strength of the priors.

B. Attack Model

We now define the attack model as the product of a spectral model and a temporal model . Our model differs from the nonparametric inharmonic model in [START_REF] Itoyama | Integration and adaptation of harmonic and inharmonic models for separating polyphonic musical signals[END_REF] in two ways: it does not represent sustained inharmonic sounds but the attack part only and it involves much fewer parameters due to its parametric expression. These two differences make sense in our application context, where no prior information is available contrary to the informed source separation context in [START_REF] Itoyama | Integration and adaptation of harmonic and inharmonic models for separating polyphonic musical signals[END_REF] where pitch, onset, duration and instrument are known.

The temporal attack model is expressed by a single Gaussian [START_REF] Tolonen | A computationally efficient multipitch analysis model[END_REF] Because the attack occurs at the same time as the onset of the harmonic partials, this distribution is equal to the first Gaussian component of the temporal harmonic model. The spectral attack model is represented by a GMM with constrained means, where the number of Gaussians is fixed to J and the means are uniformly spaced over the whole log-frequency axis. This gives [START_REF] Davy | Bayesian analysis of western tonal music[END_REF] where the means and standard deviation satisfy + and the weights encode the spectral envelope.

C. Overall model

The whole proposed model including the harmonic part and attack part is illustrated in Figure 5. The harmonic model part is a GMM in the time and log-frequency direction while the attack model part is a GMM in the log-frequency direction. Overall, this can be expressed as [START_REF] Chazan | Optimal multi-pitch estimation using the EM algorithm for co-channel speech separation[END_REF] where z indexes Gaussians representing either the harmonic part (one Gaussian per partial n and per time sampling instant y) or the attack part (one Gaussian per subband j) and θ denotes the full set of parameters of all notes. Therefore the whole signal is also represented as a mixture of spectro-temporal Gaussian distributions .

III. PARAMETER ESTIMATION AND CLASSIFICATION ALGORITHMS

A. Inference with the EM algorithm

We subsequently employ the EM algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] to estimate the parameters of our model. We assume that the observed power density W(x,t) has an unknown fuzzy membership to the kth note, represented by a spectro-temporal mask . To minimize the difference between the observed spectrogram W(x,t) and the note models, we use the Kullback-Leibler (KL) divergence as the global cost function [START_REF] Poliner | A discriminative model for polyphonic piano transcription[END_REF] where D denotes the whole time-frequency plane. Therefore the problem is regarded as the minimization of (11) under the constraints (12) [START_REF] Raczynski | Multipitch analysis with harmonic nonnegative matrix approximation[END_REF] The parameters of the note models and the corresponding masks are both unknown and must be estimated. These quantities are initialized as described in Section IV.B and iteratively optimized using the EM algorithm, where the E-step updates with fixed and the M-step updates with fixed. The number of notes K is also estimated as explained in Section IV.B.

Since each note model is composed of several Gaussians , we use a complementary set of masks to represent the fuzzy membership of to the zth Gaussian. By apply Jensen's inequality, we get [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF] Equality holds when [START_REF] Vincent | Adaptive harmonic spectral decomposition for multiple pitch estimation[END_REF] satisfying the following conditions: [START_REF] Yeh | Multiple fundamental frequency estimation and polyphony inference of polyphonic music signals[END_REF] .

(

) 17
The E-step is achieved by setting

The M-step consists of updating each parameter in turn, where the updates can be obtained analytically using Lagrange mul-tipliers. The update equations are given in Appendix. The computation time of the proposed approach is about 1.1 times that of the original HTC algorithm [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF].

B. Feature extraction

Assuming that the model parameters have been estimated, we now exploit these parameters to derive relevant features for instrument identification. By contrast with previous approaches, we extract features jointly from harmonic and attack parameters. Also, contrary to [START_REF] Wu | Musical instrument identification based on harmonic temporal timbre features[END_REF], we do not consider the parameters themselves but apply a logarithmic transformation which increases correlation with subjective timbre perception [START_REF] Klapuri | Analysis of musical instrument sounds by source-filter-decay model[END_REF] and makes their distribution closer to Gaussian [START_REF] Burred | Dynamic spectral envelope modeling for timbre analysis of musical instrument sounds[END_REF], as needed by PCA. The impact of these choices is analyzed in Section IV.

For each note k, we extract a large feature vector consisting of the following six categories of features:

1. note energy feature log( ), 2. relative partial energy features log( ) for all n, 3. partial bandwidth feature log( ), 4. harmonic temporal envelope features log( ) for all n and y, 5. note duration feature log( ), 6. attack spectral envelope features log( ) for all j. Note that the choice of a GMM as the temporal model for the harmonic part enables the extraction of a fixed number of harmonic temporal envelope features from all notes, regardless of their duration.

C. PCA for dimension reduction

While this feature vector encodes relevant timbre information, it cannot be directly used as the input to an instrument classifier. Indeed, its large dimension makes it sensitive to overfitting and to outliers, due to e.g. possible misestimation of the parameters of overlapping partials. These issues are classically addressed by dimension reduction techniques [START_REF] Loureiro | Timbre classification of a single musical instrument[END_REF][START_REF] Hourdin | A multidimensional scaling analysis of musical instruments' time-varying spectra[END_REF][START_REF] Sandell | Perceptual evaluation of principal-component-based synthesis of musical timbres[END_REF].

We here use PCA to transform the above feature vector into a low-dimension vector. This transformation is carried over the whole feature vector, so as to account for possible redundancies between harmonic and attack features. Because centering and normalization play a crucial role in PCA (features with low variance are discarded even when they are discriminative), we subtract the mean of each feature and normalize it by its largest absolute value over the training data beforehand so that it ranges from -1 to 1.

In order to illustrate the result, we computed the proposed features for five instruments among the training data of Section IV and plot the first three principal components of the feature set without attack features in Figure 6 and of the full feature set with attack features in Figure 7. These figures show that harmonic features allow some discrimination of the instruments to a certain extent, but that attack features contribute to increasing the margin between certain pairs of instruments, e.g. alto sax and piano or piano and violin. In order to increase discrimination, a larger number of components is used in our experiments. We attempted a qualitative interpretation of these components. However, due to the normalization step, most features were active in some component, so that there was no obvious interpretation.

D. SVM for instrument classification

For each note k, instrument identification is achieved by classifying the corresponding low-dimension feature vector into one instrument class. To this aim, we use a set of SVM classifiers with radial basis function (RBF) kernel [START_REF] Suykens | Nonlinear modeling and support vector machines[END_REF] where x is the feature vector composed of the values in Section III-B. SVMs are state-of-the art classifiers which maximize the margin between two classes of feature vectors in a high-dimensional space associated with the kernel. In order to solve the multi-class classification problem at hand, we use the one-versus-all approach: we train a SVM to classify each instrument versus all others and select the class which yields the greatest margin.

Training is performed on feature vectors extracted from isolated notes of each instrument. In order to account for the dependency of timbre features on pitch, a separate set of SVMs is trained for each pitch on the semitone scale. Since the accuracy of an SVM largely depends on the selection of the kernel parameters, we use 10-fold cross-validation to optimize the parameter of the RBF kernel on the training database.

IV. EXPERIMENTS

Since the proposed system aims to address both pitch estimation and instrument identification, we evaluate it according to three complementary tasks, namely multiple pitch estimation, instrument identification given the true pitches, and joint pitch estimation and instrument identification.

A. Training and test data

Training is performed on isolated notes from 9 instruments taken from three databases: the RWC database [START_REF] Goto | RWC music database: Popular, classical, and jazz music database[END_REF], McGill University Master Samples CD library [30] and the UIowa database [31]. The number of notes from each database is listed in Table 2.

Testing is performed on both synthetic mixtures of isolated notes and on real-world data. For each instrument of each database, we randomly generate 60 signals of 6 s duration. Each signal contains more than two notes and consists of both notes with similar onset times and notes in a sequence. We then randomly sum with each other the signals of different instruments within the same database so as to obtain 45 synthetic polyphonic test mixtures with the same duration. In addition, we use the real-world development data of the Multiple Fundamental Frequency Estimation & Tracking track of the 2007 Music Information Retrieval Exchange (MIREX) [32]. These data consist of five synchronized woodwind tracks, which we randomly cut to 6 s and sum together in order to obtain 30 real-world polyphonic test mixtures.

Since the timbre features of each instrument depend on the recording conditions, it is essential to use different databases for training and testing. In the following, we evaluate multiple pitch estimation and instrument identification performance on each of the three above databases (RWC, McGill or UIowa), while using the remaining two for learning. The results are then averaged over the three databases.

McGill

B. Model settings

The proposed model includes a number of hyper-parameters, which are either fixed or estimated from the data as follows. The number of harmonic partials N and the number of time sampling instants Y are fixed to 20 and 10, respectively. The number of coefficients J of the attack model is set to 20, since we found it to provide the best accuracy experimentally. Fol-lowing [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF], the parameters of the prior distributions , , and are set to 0.6547 , 0.2096 , 0.04 and 0.04, respectively.

The other model parameters are initialized as in [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF]. In particular, the number of note models K is initialized as 60 and the fundamental log-frequency and the onset time of each note are initialized to the log-frequency and time frame of the K largest peaks in the observed spectrogram.

is initialized as 2.0, is initialized as 5.0. After the EM algorithm has converged, the notes k whose energy per unit time is smaller than the average energy per unit time over all notes are discarded. This procedure allows automatic determination of the number of notes K.

Finally, we then extract the first 20 principal components of the feature vector. This number of components accounts for 99.3% of the variance of the training data and was found to provide good results experimentally.

Figure 8.b illustrates the result of the proposed algorithm with the above setting on an excerpt from the song RM-J012 in the RWC database [START_REF] Goto | RWC music database: Popular, classical, and jazz music database[END_REF].

C. Evaluation of multiple pitch estimation

In a first experiment, we assess multiple pitch estimation performance alone using the MIREX note tracking criteria [32]. A returned pitch-onset pair is considered as correct if it is within 1/4 tone and 50ms of a ground-truth note. The proportion of deleted and inserted notes is measured in terms of recall R and precision P. The F-measure is calculated from these two values as F = 2RP/(R + P).

We compare the proposed model with the NMF algorithm in [START_REF] Raczynski | Multipitch analysis with harmonic nonnegative matrix approximation[END_REF] and the original HTC algorithm in [START_REF] Kameoka | A multipitch analyzer based on harmonic temporal structured clustering[END_REF]. The parameters of NMF are set as in [START_REF] Raczynski | Multipitch analysis with harmonic nonnegative matrix approximation[END_REF] and those of HTC as in Section IV.B. To detect notes in the coefficient matrix of NMF, we use the procedure in [START_REF] Raczynski | Multipitch analysis with harmonic nonnegative matrix approximation[END_REF] based on median filtering, thresholding and discarding of notes with short duration.

The results are shown in Table 3. Our algorithm outperforms NMF and HTC both in terms of recall and precision. The resulting improvement in terms of F-measure is equal to 13% and 6% on synthetic data and 15% and 6% on real-world data, respectively. This improvement is due in particular to the introduction of the attack model, which avoids errors due to fitting of inharmonic sounds by harmonic partials.

D. Evaluation of instrument identification given the true pitches

In a second experiment, we assume that the pitch and onset time of each note are known. We use the proposed multiple pitch estimation algorithm to estimate the remaining unknown parameters of each note and assess the subsequent instrument identification performance alone. The estimated instrument is considered as correct if it is the ground truth instrument. The resulting accuracy is the percentage of notes associated with the correct instrument.

The proposed algorithm is compared with conventional 12dimensional Mel-Frequency Cepstral Coefficients (MFCCs) [START_REF] Zheng | Comparison of different implementations of MFCC[END_REF], with the source-filter model of harmonic partials in [START_REF] Klapuri | Analysis of musical instrument sounds by source-filter-decay model[END_REF] and with the harmonic features proposed in our previous work [START_REF] Wu | Musical instrument identification based on harmonic temporal timbre features[END_REF]. MFCCs are extracted from the power spectrum of each note and classified by SVM. Source-filter features are classified by ML using the likelihood function defined in [START_REF] Klapuri | Analysis of musical instrument sounds by source-filter-decay model[END_REF]. Finally, in order to directly compare SVM and ML, we also classify the proposed features by ML, where the likelihood function stems from the Gaussian model underlying PCA. We calculated the Euclidean distance between the training data and testing data, for every testing note the smallest Euclidean distance is obtained when the note is projected into the correct category. The results over synthetic data and real-world data are shown in Tables 4 and5 as a function of the number of instruments in the test signals. The proposed algorithm based on joint harmonic and attack features and SVM outperforms all other algorithms on all tasks. The resulting improvement is equal to 22%, 5% and 7% compared to MFCCs, source-filter features and our previous features on average. Including the attack features or using a SVM classifier improves the accuracy compared to considering harmonic features only or using ML classification, but only using both attack features and the SVM classifier provides the best performance for all test data.

E. Evaluation of joint pitch estimation and instrument identification

Finally, as a third experiment, we use the proposed multiple pitch estimation algorithm to estimate all note parameters and jointly evaluate multiple pitch estimation and instrument identification. An estimated note is considered as correct when its pitch, onset and instrument are all correct. The proposed algo-rithm features are compared with the same alternative features and classifiers as in the second experiment.

The results over synthetic data and real-world data are shown in Tables 6 and7. Again, the proposed algorithm outperforms all other algorithms on all tasks. The resulting improvement is equal to 20% and 6% compared to MFCCs and source-filter features on average.

Number

V. CONCLUSION

In this article, we proposed an algorithm for polyphonic pitch estimation and instrument identification based on joint modeling of sustained and attack sounds. The proposed algorithm is based on a spectro-temporal GMM model of each note, whose parameters are estimated by the EM algorithm. These parameters are then subject to a logarithmic transformation and to PCA so as to obtain a low-dimension timbre feature vector. Finally, SVM classifiers are trained from the extracted features and used for musical instrument recognition. The proposed algorithm was shown to outperform certain state-of-the-art algorithms based on harmonic modeling alone both for multiple pitch estimation and instrument identification. Future work will focus on explicitly accounting for overlapping partials so as to further improve the robustness of the proposed timbre features. In these equations, and denote when the zth Gaussian encodes the nth harmonic partial at instant y or the jth frequency subband of the attack, respectively. Furthermore, the value of y in ( 21) and ( 22) is assumed to be 0 for those Gaussians associated with the attack.

Figure 1 .

 1 Figure 1. Flow chart of the proposed system.

Figure 2 .

 2 Figure 2. Spectrogram of a piano note signal. The rectangle marks the attack part of the note.

( 3 )Figure 3 .

 33 Figure 3. Representation of the spectral models of all partials n.The temporal model of each partial is designed as a Gaussian Mixture Model (GMM) with constrained means representing time sampling instants as shown in Figure4. More precisely, the number of Gaussians is fixed to Y and the means are uniformly spaced over the duration of the note, resulting in[START_REF] Eggink | Application of missing feature theory to the recognition of musical instruments in polyphonic audio[END_REF]

Figure 4 .

 4 Figure 4. Representation of the temporal model of one partial n.

Figure 5 .

 5 Figure 5. Overall representation of the proposed model.

Figure 6 .

 6 Figure 6. First three principal components of the proposed feature set without attack features.

Figure 7 .

 7 Figure 7. First three principal components of the proposed feature set with attack features.

Figure 8 .

 8 Figure 8. Comparison of the ground truth pitches (a) and the estimated pitches (b) for song RM-J012 of the RWC database. Piano notes are represented in blue and flute notes in yellow.

Table 2 .

 2 Number of isolated notes from databases.

			RWC	UIowa	Total
	bassoon	16	112	113	241
	cello	40	430	337	807
	clarinet	47	120	423	590
	flute	90	36	226	352
	oboe	27	34	104	165
	piano	67	88	88	243
	tuba	16	90	111	217
	viola	32	467	271	770
	violin	93	45	283	421
	Total	428	1422	1956	3806

Table 3 .

 3 Multiple pitch estimation performance

			Synthetic data		real-world data
		P (%) R (%) F (%) P (%) R (%) F (%)
	NMF	72.5	74.4	73.4	44.1	46.6	45.3
	HTC	82.0	78.7	80.3	57.4	51.3	54.2
	Proposed	85.3	86.5	85.9	59.7	61.4	60.5

Table 5 .

 5 Accuracy (%) for instrument identification given the true pitches (real-world data).

	Number of Instruments	1	2	3	4	Average
	MFCC + SVM	66.5	60.8	53.4	44.1	56.2
	Source-filter + ML	78.7	74.3	69.2	67.5	72.4
	Harmonic fea-tures [26]	77.5	72.8	66.4	66.3	70.8
	Proposed features
	(without attack) +	79.5	73.8	70.4	68.7	73.1
	ML
	Proposed features
	(without attack) +	82.7	78.5	72.0	70.2	75.9
	SVM
	Proposed features
	(with attack) +	82.3	77.4	71.9	70.4	75.5
	ML
	Proposed features
	(with attack) +	84.5	80.7	73.8	72.7	77.9
	SVM
	Table 4. Accuracy (%) for instrument identification given the
		true pitches (synthetic data).

Table 6 .

 6 F-measure (%) for joint pitch estimation and instrument identification (synthetic data).

	Number of instruments	1	2	3	4	Average
	MFCC + SVM	58.5	52.1	46.0	34.3	47.7
	Source-filter + ML	70.4	63.8	58.4	54.0	61.7
	Proposed features
	(without attack) +	71.4	65.1	60.5	56.4	63.3
	ML
	Proposed features
	(without attack) +	73.5	69.5	63.6	59.8	66.6
	SVM
	Proposed features
	(with attack) +	72.6	68.4	63.4	59.0	65.9
	ML
	Proposed features
	(with attack) +	75.4	70.2	65.7	61.9	68.3
	SVM

Table 7 .

 7 F-measure (%) for joint pitch estimation and instrument identification (real-world data).

	of instruments	1	2	3	4	Average
	MFCC + SVM	35.8	32.0	27.7	20.5	29.0
	Source-filter + ML	47.5	45.4	40.1	36.0	42.3
	Proposed features
	(without attack) +	48.0	44.3	39.4	36.5	42.1
	ML
	Proposed features
	(without attack) +	51.4	48.5	44.2	40.9	46.3
	SVM
	Proposed features
	(with attack) +	50.5	49.2	43.2	39.0	45.5
	ML
	Proposed features
	(with attack) +	52.7	50.5	47.0	42.4	48.2
	SVM

Copyright IEEE 2011. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.

ACKNOWLEDGMENT

The authors would like to thank Anssi Klapuri for providing the code of his source-filter model [2]. This work was supported by INRIA under the Associate Team Program VERSAMUS (http://versamus.inria.fr/).

APPENDIX

The update equations of the parameters are as follows.

Joint harmonic and attack parameters: [START_REF] Agostini | Musical instrument timbres classification with spectral features[END_REF]


Nicolas Obin
email: nobin@ircam.fr

Pierre Lanchantin
email: lanchantin@ircam.fr

Anne Lacheret
email: anne.lacheret@u-paris10.fr

Xavier Rodet
email: rodet@ircam.fr

Reformulating Prosodic Break Model into Segmental HMMs and Information Fusion

Keywords: speech prosody, prosodic break, segmental-HMM, Dempster-Shafer fusion

In this paper, a method for prosodic break modelling based on segmental-HMMs and Dempster-Shafer fusion for speech synthesis is presented, and the relative importance of linguistic and metric constraints in prosodic break modelling is assessed 1 . A context-dependent segmental-HMM is used to explicitly model the linguistic and the metric constraints. Dempster-Shafer fusion is used to balance the relative importance of the linguistic and the metric constraints into the segmental-HMM. A linguistic processing chain based on surface and deep syntactic parsing is additionally used to extract linguistic informations of different nature. An objective evaluation proved evidence that the optimal combination of the linguistic and the metric constraints significantly outperforms both the conventional HMM (linguistic information only) and segmental-HMM (equal balance of linguistic and metric constraints), and confirmed that the linguistic constraint is prior to the metric.

Introduction

Linguistic studies generally assume that the production of a prosodic punctuation markerprosodic break -results from the integration of various and potentially conflictual constraints, in particular the syntactic and the metric constraints [START_REF] Selrik | Phonology and Syntax: The Relation between Sound and Structure[END_REF][START_REF] Dell | L'accentuation dans les phrases en franc ¸ais[END_REF][START_REF] Bailly | Integration of rhythmic and syntactic constraints in a model of generation of French prosody[END_REF][START_REF] Delais-Roussarie | Vers une nouvelle approche de la structure prosodique[END_REF]. A prosodic break is primarily produced by speakers and can be used by listeners to clarify the structure of the utterance. Simultaneously, secondary cognitive constraints (performance constraints) tend to produce a segmentation into prosodic breaks with an optimal configuration [START_REF] Gee | Performance structures: a psycholinguistic and linguistic appraisal[END_REF], in particular with respect to the metric regularity [START_REF] Dell | L'accentuation dans les phrases en franc ¸ais[END_REF]. These constraints conflict in the production of a prosodic structure, and secondary extra-linguistic constraints often override the primary linguistic constraint.

In speech synthesis, the adequate insertion of prosodic breaks guarantees the intelligibility, the naturalness, and the variety of the synthesized speech. Statistical methods have been proposed to combine linguistic and metric constraints based on segmental models [START_REF] Ostendorf | A hierarchical stochastic model for automatic prediction of prosodic boundary location[END_REF][START_REF] Schmid | New statistical methods for phrase break prediction[END_REF][START_REF] Bell | Adaptation of prosodic phrasing models[END_REF]) in the modelling and adaptation of prosodic breaks. However, the proposed methods remain generally based on surface syntactic informations (POS) solely, while deep syntactic informations are ignored. Additionally, the relative importance of the linguistic and the metric constraints is not considered, or inadequately formulated.

In this study, a statistical method to combine linguistic and metric constraints in the modelling of prosodic breaks is proposed based on segmental HMMs and Dempster-Shafer fusion, and the relative importance of the linguistic and the metric constraints is assessed depending on the nature of the linguistic information. A discrete segmental HMM is used in which prosodic breaks are modelled conditionally to the linguistic context in which they are observed, and the distance across successive prosodic breaks (length of a prosodic phrase) is explicitly modelled. Dempster-Shafer fusion is additionally employed to balance the relative importance of the linguistic constraint and the metric constraint into the segmental HMM. Segmental HMMs are objectively evaluated with respect to different sets of linguistic contexts, and the relative importance of the linguistic and the metric constraints is assessed. This paper is organized as follows: segmental HMMs and their application to prosodic break modelling are presented in section 2, Dempster-Shafer fusion is presented in section 3. The evaluation is described and discussed in sections 4 and 5.

Segmental HMMs

Segmental HMMs [START_REF] Russel | Explicit modeling of state occupancy in hidden Markov models for automatic speech recognition[END_REF][START_REF] Levinson | Continuously variable duration hidden Markov models for automatic speech recognition[END_REF][START_REF] Gales | Segmental HMMs for speech recognition[END_REF][START_REF] Ostendorf | From hmm's to segment models: a unified view of stochastic modeling for speech recognition[END_REF] were introduced in speech recognition in which state sequences are explicitly represented as segments with an explicit modelling of the segment stateoccupancy. Segmental HMM is a generalization of hidden Markov model (HMM) that addresses two principal limitations of the conventional hidden Markov model: 1) state duration modelling, and 2) assumption of conditional independence of the observation sequence given the state sequence.

The reformulation of prosodic break modelling into a segment model requires to reformulate prosodic breaks as segments. Actually, a prosodic break instantiates a prosodic segment (prosodic phrase) that is defined as the segment left/right bounded by a prosodic break. Thus, the modelling of prosodic breaks can reformulated in terms of prosodic segments.

Let define q = [q1, . . . , qT ] the sequence of linguistic contexts of length T, where qt = [qt(1), . . . , qt(L)] ⊤ is the (Lx1) linguistic context vector which describes the linguistic characteristics associated with the t-th syllable, l = [l1, . . . , lT ] the corresponding sequence of prosodic labels, where lt denotes the prosodic label associated with the t-th syllable, s = [s1, ...sK ] the associated sequence of prosodic phrases of length K, and d = [d1, ...dK ] the corresponding segment state-durations, where d k denotes the length of prosodic phrase s k .

In prosodic break modelling, the segment model can be simplified as follows:

1. one segment:

s k = [ l [t k-1 +1:t k -1] = b, lt k = b ]
2. segment transition = 1 where: t = [t1, . . . , tK ] denotes the sequence of segment boundaries, and b denotes a prosodic break and b the absence of a prosodic break.

Parameters Estimation

During the training, the estimation of the context-dependent segmental HMM parameters is simplified, and the parameters of the linguistic model λ (linguistic) and segment duration model λ (metric) are estimated separately.

λ = " λ (linguistic) , λ (metric) " (1)
The linguistic model λ (linguistic) is estimated using the context-dependent discrete HMM described in [START_REF] Obin | HMM-based prosodic structure model using rich linguistic context[END_REF].

First, linguistic contexts are clustered so as to derive a context-dependent tree. Then, a context-dependent HMM λ (linguistic) = (λ

(linguistic) S 1 , . . . , λ (linguistic) S M
) is constructed from the set of terminal contexts S = (S1, . . . , SM ) of the decision-tree, where λS m denotes the HMM parameters associated with the context Sm.

The segment duration model λ (metric) is estimated with a normal distribution.

Parameters Inference

During the synthesis, the segment sequence (s, d) is determined so as to maximize the conditional probability of the segment sequence s and the segment duration sequence d given the linguistic context sequence q:

(s, d) = argmax s,d p(s, d|q) (2)
The determination of the segment sequence (s, d) can be proved to be equivalent to the determination of the prosodic break sequence b l as follows:

b l = argmax l K Y k=1 p(l [t-d k +1:t-1] = b, lt = b |q [t-d k +1:t] ) p(l [t-d k +1:t-1] = b, lt = b) × p(d k |l [t-d k +1:t-1] = b, lt = b) (3) = argmax l K Y k=1 po(t k ) | {z } observation probability ps(t k ) | {z } segment probability (4)
where

ps(lt k ) = p(d k | l [t-d k :t-1] = b, lt = b)
denotes the partial probability that the k-th segment with duration d k ends at time t k , and po(lt

k ) ∝ p(l [t-d k +1:t] = b, lt k = b |q [t-d k +1:t]
) the partial observation probability over the k-th segment with duration d k .

The solution to this problem is achieved with a reformulation of the conventional Viterbi Algorithm (VA) for segmental HMMs [START_REF] Ostendorf | From hmm's to segment models: a unified view of stochastic modeling for speech recognition[END_REF].

Dempster-Shafer Fusion

In the formulation of segmental HMMs, the segment probability and the observation probability are equally considered. However, linguistic studies pointed out that the linguistic and the metric constraints are not of equal importance in the production of a prosodic break. In particular, the metric constraint is generally assumed to be secondary compared to the linguistic constraint.

Dempster-Shafer theory [START_REF] Shafer | A Mathematical Theory of Evidence[END_REF] is a mathematical theory commonly used for information fusion in statistical processing.

In particular, Dempster Shafer theory provides a proper probabilistic formulation for information fusion, in which the reliability that can be conferred to different sources of information can be explicitly formulated. In the Dempster-Shafer fusion, PDFs can be reformulated into mass functions (MFs) to account for the reliability that can be conferred to each PDF, and then combined with the Dempster-Shafer fusion rule. Mass functions are defined on P(Ω), where Ω denotes the state alphabet, and P(Ω) the total set of combinations of Ω.

In order to balance the relative importance of the linguistic constraint po(lt) and the metric constraint ps(lt) into the segmental HMM, one of the PDFs is alternatively replaced by a mass function (MF), while the other remains a PDF:

mo(lt) = α po(lt) mo(Ω) = 1 -α (5) ms(lt) = β ps(lt) ms(Ω) = 1 -β (6)
where α and β denote the reliability that is associated with the observation probability po(lt) and the segment probability ps(lt) respectively, and mo(Ω) and ms(Ω) the model ignorance.

The Dempster-Shafer fusion of mo and ms is then given by:

(mo ⊕ ms)(lt) ∝ α(1 -β) po(lt) + αβ po(lt) ps(lt) + β(1 -α) ps(lt) (7)
Hence,

(m1 ⊕ m2)(lt) ∝ 8 < : po(lt), α = 1, β = 0 1 ps(lt), α = 0, β = 1 2 po(lt) ps(lt), α = 1, β = 1 3
1 denotes that the observation probability is considered only (conventional HMM), 2 denotes that the segment probability is considered only, and 3 denotes that the segment and observation probabilities are equally considered (conventional segmental HMM). In the latter case, the expression is equivalent to the conventional Bayes combination rule.

Finally, the relative confidence α and β are rewritten into a single weight (α, β) so that the relative importance of the linguistic and the segment probabilities is linearly interpolated from the metric constraint solely to the linguistic constraint solely. Thus: (α, β) = -1 will refer to α = 0 and β = 1, (α, β) = 0 to α = 1 and β = 1, and (α, β) = +1 to α = 1 and β = 0.

Evaluation

The evaluation was conducted to assess the relative importance the linguistic and the metric constraints, and their combination in prosodic break modelling. In particular, a large range of combination of linguistic contexts was used to estimate contextdependent segmental HMMs, and various combinations of the linguistic and metric constraints were compared. Two baseline models were used for the comparison: the conventional punctuation rule-based model (P) in which a prosodic break is inserted after each punctuation marker, and the segmental HMM estimated with the conventional morpho-syntactic linguistic context (M).

Evaluation scheme

The comparison of context-dependent segmental HMMs was conducted using different set of linguistic contexts and different combination of the linguistic and the metric constraints. Evaluation was conducted according to a 10-fold cross-validation. F-measure was used as performance measure and a paired Student t-test [START_REF] Box | Statistics for Experimenters[END_REF] was employed to assess whether a significant difference exists between the models being compared.

Speech Material

Two French read-speech databases were compared: a laboratory and a multi-media corpus. The laboratory corpus is composed of short sentences that were selected in order to design a phonetically well-balanced speech database. Each sentence was separately read by a non-professional French speaker (9 hours). The multi-media corpus is the novel "Du côté de Chez Swann" ("Swann's Way") by the French writer Marcel Proust. The text was read by a professional actor in the context of an audio-book format (7 hours). The laboratory corpus consists of simple linguistic structures and controlled prosody, while the multi-media corpus consists of complex linguistic structures and a rich prosody variety.

Linguistic Contexts

Linguistic informations were extracted from text using a French linguistic processing chain that includes surface and deep syntactic parsing [START_REF] Sagot | The Lefff, a freely available and large-coverage morphological and syntactic lexicon for French[END_REF][START_REF] De | From metagrammars to factorized TAG/TIG parsers[END_REF]. The extracted syntactic features were classified into different sets according to the nature of the syntactic information. Morpho-syntactic (M) informations correspond to the conventional syntactic information (POS) used in speech prosody modelling. Dependency (D) and constituency (C) were compared in their relevancy in speech prosody modelling. Additionally, adjunction (A) covers a large variety of syntactic constructions potentially related to speech prosody (e.g., relative clause, incise) and was introduced for comparison. Finally, segmental-HMMs were compared with respect to any combination of the different linguistic feature sets. A detailed description of the linguistic contexts used is presented in [START_REF] Obin | Towards improved HMM-based speech synthesis using high-level syntactical features[END_REF][START_REF] Obin | HMM-based prosodic structure model using rich linguistic context[END_REF].

Results & Discussion

Table 1 summarizes the mean performance obtained for the laboratory and multi-media speech databases depending on the linguistic feature set, and the comparison of the metric model only, the conventional segmental-HMM, the linguistic model only, and the optimal configuration of linguistic and metric constraints into the segmental-HMM. The optimal configuration significantly outperforms the conventional segmental-HMM and the linguistic model for 1: Ranked F-measure for the optimal configuration, the metric model only ps, the conventional segmental-HMM pspo, and the linguistic model only po. Significance test for the comparison of the optimal configuration, the conventional segmental-HMM and the linguistic model. all of the linguistic feature sets, and corresponds to a prior importance of the linguistic constraint over the metric constraint. The balance of the linguistic and the metric constraints varies depending on the relevancy of the linguistic or/and the metric constraint. In particular, the optimal configuration gradually tends to the linguistic constraint when the linguistic information increase in reliability ( (α, β) varies from +0.23 to +0.56 for the laboratory corpus in correlation with a linguistic performance which varies from 74.2% to 95%), and is very close to the linguistic constraint when the metric constraint is not reliable ((α, β) varies from +0.58 to +0.73 in correlation with a metric performance of 39%).

-1 D M M D C M C M D C D C A M A M D A D A D C A M C A C A M D C A e x t
D M M D C M C M D C D C A M A M D A D A D C A M C A C A M D C A 0.5 0.6 0.7 0.8 0.9 1 recal l 0.91 POUET M M D C M C M D C D C A M A M D A D A D C A M C A C A M D C A
The relevancy of the different linguistic features in prosodic break modelling (figure 1) confirms and refines observations reported in [START_REF] Obin | HMM-based prosodic structure model using rich linguistic context[END_REF]. Adjunction (A) and to a less extent constituency (C) are the most relevant single linguistic contexts (91.7% and 83.8% for the laboratory corpus, 63.2% and 65.3% for the multi-media corpus), and their combination (CA) is strongly relevant (96.0% and 73.7% for the laboratory and the multi-media speech databases respectively). Morpho-syntactic (M) and dependency (D) are slightly relevant linguistic contexts (78.3% and 73.8% for the laboratory corpus, 59.2% and 52.0% for the multi-media corpus). The optimal performance is obtained with the combination of all of the linguistic contexts (MDCA) (96.3% and 75.3% for the laboratory and multi-media speech databases respectively).

Additionally, recall and precision mutually increase when the linguistic description is enriched (figure 2). However, the increase in recall is large (from 62% to 92% for the laboratory corpus, and from 42% to 67% for the multi-media corpus) compared to the increase in precision (from 92% to 97% for the laboratory corpus, and from 76% to 82% for the multi-media corpus). Thus, the enrichment of the linguistic description significantly decreases the omission of prosodic breaks, while the false insertion of prosodic breaks remains globally marginal regardless to the linguistic description.

The increase in performance obtained with the enrichment of the linguistic description is significantly larger compared to that obtained with the integration of the metric constraint. For the linguistic constraint, the increase in performance is of 18% by comparison of the conventional morpho-syntactic context (78.3%) and the optimal linguistic context (96.3%). For the combination of the linguistic and the metric constraints, the increase in performance does not exceed 4% (74.2% and 78.3% for the conventional morpho-syntactic context), and 2% with a rich linguistic description (95% and 93.3% for the optimal linguistic context).

Finally, the performance significantly varies depending on the speech database. The overall performance, and the increase in performance due to the enrichment of the linguistic description and the combination of the linguistic and the metric constraints are significant larger for the laboratory corpus compared to the multi-media corpus. The difference may be simply interpreted in terms of the reliability of the syntactic analysis, and eventually by the difference in prosodic variety of the speakers.

Conclusion

In this paper, a statistical method that combines linguistic and metric constraints in the modelling of prosodic breaks was proposed based on segmental HMMs and Dempster-Shafer fusion, and the relative importance of the linguistic and the metric constraints was assessed depending on the nature of the linguistic informations. The optimal combination of the linguistic and the metric constraints into segmental-HMM was proved to significantly outperform the conventional segmental-HMM and the linguistic model only. The linguistic constraint was shown to be prior to the metric constraint, and the optimal configuration to gradually tend to the linguistic constraint when the linguistic description is enriched, or when the metric constraint is slightly reliable. Finally, the increase in performance obtained by the integration of the metric constraint and its combination with the linguistic constraint remains slight compared to that obtained with the enrichment of the linguistic description. In further studies, the segment model will be refined to improve the modelling of the metric constraint and its combination with the linguistic constraint, and will be evaluated for the modelling of various speaking styles.

Figure 1 :

 1 Figure1: Laboratory corpus: F1 measure depending on the linguistic context and the balance α,β of metric and linguistic constraints. The dotted line denotes the segmental-HMM, and the red line the optimal combination of the metric and linguistic constraints.

Figure 2 :

 2 Figure 2: Laboratory corpus: precision and recall of the optimal model depending on the linguistic context.

Table

	context	popt	α, β	ps	pspo	t-test	po	t-test
	laboratory
	MDCA	96.3	+0.44	65.4	92.1	<0.001	95.0	<0.001
	MCA	96.0	+0.48	65.4	92.1	<0.001	94.7	<0.001
	CA	96.0	+0.54	65.4	92.0	<0.001	94.6	<0.001
	DCA	95.8	+0.56	65.4	91.7	<0.001	94.6	<0.001
	DA	94.1	+0.41	65.4	89.1	<0.001	92.6	<0.001
	. . .	. . .	. . .	. . .	. . .	. . .	. . .	. . .
	M	78.3	+0.23	65.4	75.5	<0.001	74.2	<0.001
	P	66.3	-	-	-	-	-	-
	multi-media
	MDCA	75.3	+0.70	39.0	70.0	<0.001	74.0	0.03
	MCA	75.2	+0.58	39.0	69.6	<0.001	73.6	0.04
	DCA	74.2	+0.68	39.0	68.6	<0.001	72.8	0.02
	CA	73.7	+0.73	39.0	67.4	<0.001	72.6	0.2
	MC	69.6	+0.65	39.0	65.5	<0.001	68.0	0.1
	. . .	. . .	. . .	. . .	. . .	. . .	. . .	. . .
	M	59.2	+0.62	39.0	56.7	0.03	58.7	0.5
	P	55.1	-	-	-	-	-	-

This study was partially funded by "La Fondation Des Treilles", and supported by ANR Rhapsodie 07 Corp-030-01; reference prosody corpus of spoken French; French National Agency of research;

2008- 2012.


Nicolas Obin
email: nobin@ircam.fr

Pierre Lanchantin
email: lanchantin@ircam.fr

Anne Lacheret
email: anne.lacheret@u-paris10.fr

Xavier Rodet
email: rodet@ircam.fr

Discrete/Continuous Modelling of Speaking Style in HMM-based Speech Synthesis: Design and Evaluation

Keywords: speaking style, speech synthesis, speech prosody, average modelling

This paper assesses the ability of a HMM-based speech synthesis systems to model the speech characteristics of various speaking styles 1 . A discrete/continuous HMM is presented to model the symbolic and acoustic speech characteristics of a speaking style. The proposed model is used to model the average characteristics of a speaking style that is shared among various speakers, depending on specific situations of speech communication. The evaluation consists of an identification experiment of 4 speaking styles based on delexicalized speech, and compared to a similar experiment on natural speech. The comparison is discussed and reveals that discrete/continuous HMM consistently models the speech characteristics of a speaking style.

Introduction

Each speaker has his own speaking style which constitutes his vocal signature, and a part of his identity. Nevertheless, a speaker continuously adapt his speaking style according to specific communication situations, and to his emotional state. In particular, each situational context determines a specific mode of production associated with it -a genre -which is defined by a set of conventions of form and content that are shared among all of its productions [START_REF] Simon | Les phonostyles: une description prosodique des styles de parole en franc ¸ais[END_REF]. In particular, a specific discourse genre (DG) relates to a specific speaking style. Consequently, a speaker adapts his speaking style to each specific situation depending on the formal conventions that are associated with the situation, his a-priori knowledge about these conventions, and his competence to adapt his speaking style. Thus, each communication act instantiates a style which is composed of a style that depends on the speaker identity, and a conventional speaking style that is conditioned by a specific situation. In speech synthesis, methods have been proposed to model and adapt the symbolic [START_REF] Schmid | New statistical methods for phrase break prediction[END_REF][START_REF] Bell | Adaptation of prosodic phrasing models[END_REF] and acoustic speech characteristics of a speaking style, with application to emotional speech synthesis [START_REF] Yamagishi | HMM-based expressive speech synthesis -Towards TTS with arbitrary speaking styles and emotions[END_REF]. However, no study exists on the joint modelling of the symbolic and acoustic characteristics of speaking style, and speaking style acoustic modelling generally limits to the modelling of emotion, with rare extensions to other sources of speaking styles variations [START_REF] Krstulović | An HMM-based speech synthesis system applied to german and its adaptation to a limited set of expressive football announcements[END_REF]. This paper presents an average discrete/continuous HMM which is applied to the speaking style modelling of various discourse genres in speech synthesis, and assesses whether the model adequately captures the speech prosody characteristics of a speaking style. Incidentally, the robustness of the HMMbased speech synthesis is evaluated in the conditions of realworld applications. The paper is organized as follows: the speaking style corpus design is described in section 2; the average discrete/continuous HMM model is presented in section 3; the evaluation is presented and discussed in sections 4 and 5.

Speech & Text Material

Corpus Design

For the purpose of speaking style speech synthesis, a 4-hour multi-speakers speech database was designed. The speech database consists of four different DG's: catholic mass ceremony, political, journalistic, and sport commentary. In order to reduce the DG intra-variability, the different DGs were restricted to specific situational contexts (see list below) and to male speakers only. The following is a description of the four selected DG's: mass: Christian church sermon (pilgrimage and Sunday highmass sermons); single speaker monologue, no interaction. political: New Year's French president speech; single speaker monologue; no interaction. journal: radio review (press review; political, economical, technological chronicles); almost single speaker monologue with a few interactions with a lead journalist. sport commentary: soccer; two speakers engaged in monologues with speech overlapping during intense soccer sequences and speech turn changes; almost no interaction. The speech database consists of natural speech multi-media audio contents with strongly variable audio quality (background noise: crowd, audience, recording noise, and reverberation). The speech prosody characteristics of the speech databased are illustrated in figure 1.

Speaking Style Model

A speaking style model λ (style) is composed of discrete/continuous context-dependent HMMs that model the symbolic/acoustic speech characteristics of a speaking style.

λ (style) = " λ (style) symbolic , λ (style) acoustic " (1)
During the training, the discrete/continuous context-dependent HMMs are estimated separately. During the synthesis, the symbolic/acoustic parameters are generated in cascade, from the symbolic representation to the acoustic variations. Additionally, a rich linguistic description of the text characteristics is automatically extracted using a linguistic processing chain [START_REF] De | From metagrammars to factorized TAG/TIG parsers[END_REF] and used to refine the context-dependent HMM modelling (see [START_REF] Obin | Towards improved HMM-based speech synthesis using high-level syntactical features[END_REF] and [START_REF] Lacheret | Design and Evaluation of Shared Prosodic Annotation for Spontaneous French Speech: From Expert Knowledge to Non-Expert Annotation[END_REF] for a detailed description of the enriched linguistic contexts).

Training of the Discrete/Continuous Models

Discrete HMM

For each speaking style, an average context-dependent discrete HMM λ

(style) symbolic is estimated from the pooled speakers associated with the speaking style.

The prosodic grammar consists of a hierarchical prosodic representation that was experimented as an alternative to ToBI [START_REF] Silverman | ToBI: a standard for labeling english prosody[END_REF] for French prosody labelling [START_REF] Obin | Expectations for Speaking Style Identification: a Prosodic Study[END_REF]. The prosodic grammar is composed of major prosodic boundaries (FM, a boundary which is right bounded by a pause), minor prosodic boundaries (Fm, an intermediate boundary), and prosodic prominences (P).

Let R be the number of speakers from which an average model λ (style) symbolic is to be estimated. Let l = (l (1) , . . . , l (R) ) the total set of prosodic symbolic observations, and l (r) = [l (r) (1), . . . , l (r) (Nr)] the prosodic symbolic sequence associated with speaker r, where l (r) (n) is the prosodic label associated with the n-th syllable.

Let q = (q (1) , . . . , q (R) ) the total set of linguistic contexts observations, and q (r) = [q (r) (1), . . . , q (r) (Nr)] the linguistic context sequence associated with speaker r, where

q (r) (n) = [q (r) 1 (n), . . . , q (r) L (n)] ⊤
is the (Lx1) linguistic context vector which describes the linguistic characteristics associated with the n-th syllable.

An average context-dependent discrete HMM λ (style) symbolic is estimated from the pooled speakers observations. Firstly, an average context-dependent tree T (style) symbolic is derived so as to minimize the information entropy of the prosodic symbolic labels l conditionally to the linguistic contexts q . Then, a contextdependent HMM model λ (style) symbolic is estimated for each terminal node of the context-dependent tree T (style) symbolic .

Continuous HMM

For each speaking style, an average acoustic model λ (style) acoustic that includes source/filter variations, f0 variations, and statedurations, is estimated from the pooled speakers associated with the speaking style based on the conventional HTS system [START_REF] Yoshimura | Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis[END_REF].

Let R be the number of speakers from which an average model is to be estimated. Let o = (o (1) , . . . , o (R) ) the total set of observations, and o (r) = [o (r) (1), . . . , o (r) (Tr)] the observation sequences associated with speaker r, where

o (r) (t) = [o (r) t (1), . . . , o (r)
t (D)] ⊤ is the (Dx1) observation vector which describes the acoustical property at time t. Let q = (q (1) , . . . , q (R) ) the total set of linguistic contexts observations, and q (r) = [q (r) (1), . . . , q (r) (Tr)] the linguistic context sequence associated with speaker r, where q (r) (t) = [q (r) 1 (t), . . . , q (r) L (t)] ⊤ is the (Lx1) linguistic context vector which describes the linguistic properties at time t.

An average context-dependent HMM acoustic model λ (style) symbolic is estimated from the pooled speakers observations. Firstly, a context-dependent HMM model is estimated for each of the linguistic contexts. Then, an average context-dependent tree T (style) acoustic is derived so as to minimize the description length of the context-dependent HMM model λ (style) acoustic .

The acoustic module models simultaneously source/filter variations, f0 variations, and the temporal structure associated with a speaking style. Speakers f0 were normalized with respect to the speaking style prior to modelling. Source, filter, and normalized f0 observation vectors and their dynamic vectors are used to estimate context-dependent HMM models λ (style) acoustic . Context-dependent HMMs are clustered into acoustically similar models using decision-tree-based context-clustering (ML-MDL [START_REF] Yoshimura | Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis[END_REF]). Multi-Space probability Distributions (MSD) [START_REF] Tokuda | Hidden Markov models based on multi-space probability distribution for pitch pattern modeling[END_REF] are used to model continuous/discrete parameter f0 sequence to manage voiced/unvoiced regions properly. Each contextdependent HMM is modelled with a state duration probability density functions (PDFs) to account for the temporal structure of speech [START_REF] Zen | Hidden semi-Markov model based speech synthesis[END_REF]. Finally, speech dynamic is modelled according to the trajectory model and the global variance (GV) that model local and global speech variations over time [START_REF] Toda | A speech parameter generation algorithm considering global variance for HMM-based speech synthesis[END_REF].

Generation of the Speech Parameters

During the synthesis, the text is first converted into a concatenated sequence of context-dependent HMM models λ (style) symbolic associated with the linguistic context sequence q = [q1, . . . , qN ], where qn = [q1, . . . , qL] ⊤ denotes the (Lx1) linguistic context vector associated with the n-th phoneme.

Firstly, the prosodic symbolic sequence b l is determined so as to maximize the likelihood of the prosodic symbolic sequence l conditionally to the linguistic context sequence q and the model λ

Then, the linguistic context sequence q augmented with the prosodic symbolic sequence b l is converted into a concatenated sequence of context-dependent models λ (style) acoustic .

The acoustic sequence b o is inferred so as to maximize the likelihood of the acoustic sequence o conditionally to the model λ q is determined so as to maximize the likelihood of the state sequence conditionally to the model λ (style) acoustic . Then, the observation sequence b c is determined so as to maximize the likelihood of the observation sequence conditionally to the state sequence b q, the model λ

(style) acoustic under dynamic constraint o = Wc. R b q b c = r b q (4)
where:

R b q = W ⊤ Σ -1 b q W. ( 5
)
r b q = W ⊤ Σ -1 b q µ b q . ( 6
)
and Σ b q and µ b q are respectively the covariance matrix and the mean vector for the state sequence b q.

⇓ sentence Longtemps , je me suis couché de bonne heure .

SPEECH SYNTHESIS

˜˜˜F igure 2: Generation of discrete/continuous speech parameters for the sentence: "Longtemps, je me suis couché de bonne heure" ("For a long time I used to go to bed early").

Evaluation

The proposed model has been evaluated based on a speaking style identification perceptual experiment, and compared to a speaking style identification experiment with natural speech [START_REF] Obin | HMM-based prosodic structure model using rich linguistic context[END_REF]. For the purpose of such a comparison, it was necessary to provide a single evaluation scheme for both experiments. In particular, it was not possible to control the linguistic content of natural speech utterances which provides evident cues for DG's identification (a single keyword would be sufficient to identify a DG). Thus, such a comparison required to remove lexical access and to focus on the prosodic dimension only.

Experimental Setup

40 speech utterances (10 per DG) were selected in the speaking style corpus and removed from the training set. Lexical access was removed using a band-pass filter that insured that the lowest frequency of the fundamental frequency and the highest frequency of its first harmonic was included.

Subjective Evaluation

The evaluation consists of a multiple choice identification task from speech prosody perception. The evaluation was conducted according to crowd-sourcing technique using social networks. 50 subjects (including 25 native French speakers, 15 non-native French speakers, 10 non-French speakers; 34 expert and 16 naïve listeners) participated in this experiment. Participants were given a brief description of the different speaking styles. Then, they were asked to associate a speaking style to each of the speech utterances. For this purpose, participants were given three options: total confidence: select only one speaking style when certain of the choice; confusion: select two different speaking styles when two speaking styles are possible; total indecision: select "indecision" when completely unsure. Subjects were asked to use this possibility only as a very last resort. Additional informations were gleaned from the participants: speech expertise (expert, naïve), language (native French speaker, non-native French speaker, non-French speaker), age, and listening condition (headphones or not). Expert participants were actually coming from various domains (speech and audio technologies, linguistics, musicians). Participants were encouraged to use headphones.

Results & Discussion

Identification performance was estimated using a measure based on Cohen's Kappa statistic [START_REF] Cohen | A coefficient of agreement for nominal scales[END_REF]. Cohen's Kappa statistic measures the proportion of agreement between two raters with correction for random agreement. Our measure monitors the agreement between the ratings of the participants and the ground truth. The measure varies from -1 to 1: -1 is perfect disagreement; 0 is chance; 1 is perfect agreement. Confusion ratings were considered as equally possible ratings. Total indecision ratings were relatively rare (3% of the total ratings) and removed. Figure 3 presents the identification confusion matrix. Overall score reveals fair identification performance (κ = 0.38 ± 0.04) which is comparable to that observed for identification from natural speech (κ natural = 0.45 ± 0.03). The identification performance significantly depends on the speaking style (figure 4): sport commentary is substantially identified (κ = 0.68 ± 0.05), journal fairly identified (κ = 0.50 ± 0.06), political discourse moderately identified (κ = 0.28±0.07), and mass only slightly identified (κ = 0.12 ± 0.06). In comparison with identification from natural speech, the identification is comparable in the case of the sport commentary and the journal speaking styles (κ natural = 0.70 ± 0.03 and κ natural = 0.54 ± 0.05, respectively). However, there is a drop in identification for the political and the mass speaking styles which is especially significant for the mass style (κ natural = 0.34±0.05 and κ natural = 0.38 ± 0.04, respectively). This indicates that the model somehow failed to capture the relevant cues of the corresponding speaking style. Nevertheless, a large confusion exists between the political and the mass speech that is inherent to a similarity in the speaking style and the formal situation in which the speech occurs. Additionally, the conventional HMMbased speech synthesis system failed into modelling adequately the breathyness and the creakyness that is specific to the political speaking style, especially within unvoiced segments. ANOVA analysis was conducted to assess whether the identification performance depends on the language of the participants. Analysis reveals a significant effect of the language (F(2, 59) = 15, p < 0.001) (F(48,2)=5.9, p-value=0.005), and confirms results obtained for natural speech. This confirms evidence that there exists variations of a speaking style depending on the language and/or cultural background. Finally, an informal evaluation of the quality of the synthesized speech suggests that the speaking style modelling is robust to the large variety of audio quality.

Conclusion

In this study, the ability and the robustness of a HMM-based speech synthesis system to model the speech characteristics of various speaking styles were assessed. A discrete/continuous HMM was presented to model the symbolic and acoustic speech characteristics of a speaking style, and used to model the average characteristics of a speaking style that is shared among various speakers, depending on specific situations of speech communication. The evaluation consisted of an identification experiment of 4 speaking styles based on delexicalized speech, and compared with a similar experiment on natural speech. The evaluation showed that the discrete/continuous HMM consis- tently models the speech characteristics of a speaking style, and is robust to the differences in audio quality. This proves evidence that the discrete/continuous HMM speech synthesis system successfully models the speech characteristics of a speaking style in the conditions of real-world applications.

Figure 1 :

 1 Figure 1: Prosodic description of the speaking styles depending on the speaker. Mean and variance of f0 and speech rate (syllable per second).

  ## je me suis cou-ché de bonne heure ## Table 4.2: Illustration of the text-to-prosodic-structure conversion.

Figure 3 :

 3 Figure 3: Identification confusion matrices. Rows represent synthesized speaking style. Columns represent identified speaking style.

Figure 4 :

 4 Figure 4: Mean identification scores and 95% confidence interval obtained for natural and synthesized speech.

This study was partially funded by "La Fondation Des Treilles", and supported by ANR Rhapsodie 07 Corp-030-01; reference prosody corpus of spoken French; French National Agency of research;

2008- 2012.