Julien Dehos
email: julien.dehos@gmail.com

Eric Zeghers
email: zeghers@iut.u-clermont1.fr

Christophe Renaud
email: renaud@lil.univ-littoral.fr

Laurent Sarry
email: laurent.sarry@u-clermont1.fr

Radiometric compensation for a low-cost immersive projection system

Keywords: CR Categories: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism-virtual reality, I.4.0 [Image Processing and Computer Vision]: General-Image displays, I.4.1 [Image Processing and Computer Vision]: Digitization and Image Capture-Radiometry, I.4.8 [Image Processing and Computer Vision]: Scene Analysis-Photometry, radiometric compensation, mixed reality, virtual reality, immersive projection

is a low-cost projection system aiming at making mixed reality (virtual, augmented or diminished reality) affordable. It combines a videoprojector, a camera and a convex mirror and works in a non-specific room. This system displays an immersive environment by projecting an image onto the different parts of the room. However, the presence of an uncalibrated projector, heterogeneous materials and light inter-reflections influence the colors of the environment displayed in the room. Radiometric compensation of the projection process enables the system to reduce this problem.

In this paper, we present our low-cost immersive projection system and propose a radiometric model and a compensation method which handle the projector response, surface materials and interreflections between surfaces. Our method works in two stages. First, the radiometric response of the projection process is evaluated. Then, this radiometric response is used to compensate the projection process in the desired environments.

Introduction

In virtual reality frameworks, an immersive environment is a partially or totally artificial environment created around the user. Immersive environments have many applications such as training simulations (flight and driving simulators), architectural or engineering prototyping, virtual offices, entertainment environments, games, art installations. . . Immersive environments can be displayed by two kinds of systems: wearable devices such as head mounted display devices [START_REF] Sutherland | Seminal graphics: poineering efforts that shaped the field[END_REF]] and large screen systems such as CAVEs [START_REF] Sandin | Surround-screen projection-based virtual reality: the design and implementation of the cave[END_REF].

Everyday technology makes it possible to develop low-cost mixed reality systems. Catopsys (for catadioptric projection system) is a research project which aims at bringing mixed reality to Figure 1: The Catopsys projection system. A projector (P) aims at a convex mirror (M). A beam of light (B) coming from P is reflected by M toward the room (R) (wall, floor, ceiling, furniture. . . ). The system can thus display immersive environments. For calibration purposes, a camera (C) can see the whole screen by rotating around the optical axis of P.

the home by developing a low-cost immersive projection system. This system is composed of a projector, a convex mirror and a rotating camera (see Fig. 1). Once in a room, the system calibrates automatically the geometry of its components and of the room [START_REF] Astre | Automatic calibration of a single-projector catadioptric display system[END_REF]]. Immersive projection can then be done. In [START_REF] Yuen | Inexpensive immersive projection[END_REF], a system using the same devices is presented. However, it uses a specific screen and, consequently, has different applications.

Our projection system requires radiometric compensation (i.e. color correction). Indeed, since the projection is over non-specific surfaces, images must be modified to really appear as desired, after projection. The projection is influenced by the projector, surface materials and inter-reflections between surfaces. We may notice that inter-reflections are very important for our application because images are projected onto the whole room. To perform radimetric compensation, the system provides a panoramic viewpoint (camera) and a dynamic omnidirectional light source (projector + mirror).

In this paper, we present a radiometric compensation technique adapted to a low-cost immersive projection display. Our method accounts for projector non-linearity, color mixing and interreflections. It has been implemented in a self-calibrating system.

Related work

Recently, a lot of work has been done to project images faithfully on everyday surfaces [START_REF] Bimber | The visual computing of projector-camera systems[END_REF]]. Here, we only consider a few two-stage compensation techniques using a camera-projector system. Such techniques first evaluate the radiometric response of the projection process off-line and then, inverte this response to compute compensation images on-line. [Nayar et al. 2003] proposes a model of projection on nonspecific screens. The authors also give a calibration/compensation method based on this model. This method separates the projection process into linear color mixing and a per-channel non-linear response. However, the screen is supposed to be flat. [START_REF] Bimber | Compensating indirect scattering for immersive and semi-immersive projection displays[END_REF]] proposes a method to compensate projection when inter-reflections occur. The authors present a mathematical and algorithmic formulation of a reverse radiosity approach. However, they do not explain to get the geometry and the reflectances of the scene automatically. Moreover, they assume the projector response is linear.

Finally, an exhaustive method is proposed in [START_REF] Wetzstein | Radiometric compensation through inverse light transport[END_REF]]. This method consists in measuring the light transport matrix i.e. the influence of each projector pixel on each camera pixel. Compensated images can be computed using the inverted matrix. However, the non-linearity of the projector response is not taken into account. Furthermore, the matrix measurement requires a huge number of camera captures.

Catopsys projection system

The Catopsys projection system aims at democratizing mixed reality. To reach this goal, the system has to be low-cost and achieve immersive projection and self-calibration. Our solution is a catadioptric projector-camera system (see Fig. 1). A convex mirror (M) is fixed to the ceiling of the room. A videoprojector (P) is placed under the mirror and points toward it. Thus, a beam of light (B) emitted by P is reflected by M into the room (R). Finally, a wide-angle camera (C) is attached above the projector and oriented toward the walls of the room. The camera can rotate around the optical axis of the projector thanks to a stepper motor. The projector, the camera and the stepper motor are controlled by a computer.

This system fulfills the desired features. First, it is composed of a few low-cost devices. Indeed, over the past few years, projectors have become less and less expensive and can reasonably be bought for home applications. In the same way, affordable models of computer-controlled camera exist. The remaining components of the system are inexpensive. In our implementation, we use a surveillance mirror, a home-made stand for the projector-camera system and a cheap stepper motor which we control through an IEEE 1284 card. In a final product, these elements would be manufactured, which would reduce costs even more.

The system also provides the second desired feature i.e. immersive projection. Indeed, according to the settings of the projector, the position of the mirror and the configuration of the room, the projection can be done onto the whole room (4π steradians).

Finally, the system is able to automatically calibrate itself. Thanks to the projector-camera combination, the system can project an image and capture the result. More precisely, the system has to be calibrated geometrically (relative positions and orientations of devices, geometry of the room), optically (distortions of optics) and radiometrically (light scattering, spectral responses of devices). In this paper, we deal with radiometric compensation and consider that geometrical and optical calibrations have been done [START_REF] Astre | Automatic calibration of a single-projector catadioptric display system[END_REF]]. Thus, we know onto which triangle of the room geometry, each pixel is projected. We also have an atomic function which gives, for each pixel of the projected image, the color seen by the camera (see section 6 for examples).

Radiometric model

Projection process

Consider an image projected by our system (see Fig. 2). On the computer, this image is coded in a color basis. The projector converts it according to its internal color channels and emits the corresponding light. Then, this light strikes screen surfaces (walls, ceiling or floor of the room) and bounces many times between surfaces (inter-reflections). Finally, light is captured by the camera sensors to produce another image, on the computer.

The process involves intra-channel non-linearity (projector and camera responses), linear inter-channel mixing (because projector

Radiometric model

Let us consider a pixel. We call x its color in the linearized color basis of the projector and y its color in the linearized color basis of the camera. Let t be the transition matrix from projector basis to camera basis, then y = tx. We define u and v as

uij = tij -1 if i = j 0 otherwise , vij = 1 if i = j tij otherwise Therefore t = u + v.
We can see u as the intra-channel projectorcamera basis transition. Hence the following radiometric model, based on the one given in [Nayar et al. 2003]. Let a be the color of the pixel in the input image (sent to the projector). In the following, we call x λ the value of the channel λ of the color x. The per-channel per-pixel non-linear function w λ models the non-linear response, the linear material response and uii (where i is the index corresponding to the channel λ in u). Let b be the light emitted by the projector and reflected a single time by surfaces of the room, in a scaled linearized projector color basis.

b λ = w λ (a λ ) (1)
Let c be the light after the first reflection, in the linearized camera color basis.

c = vb (2)
We model inter-reflections by the following formula, in linearized camera color basis.

D λ = C λ + H λ C λ ( 3
)
C λ is the vector containing the c λ value of the pixels (or group pixels) seen by the camera.. It represents the light emitted by surfaces of the room i.e. the light coming from the projector and reflected a single time. H λ is a surface-to-surface light exchange matrix, accounting for both geometry and radiometry: λij is the part of light emitted from surface i, scattered in the room by interreflection and then reflected by surface j toward the camera. Therefore, C λ is the light going directly to the camera and H λ C λ is the light going to the camera after inter-reflection. Finally, D λ contains the full light seen by the camera (direct light and interreflections). Consequently, if surface materials are diffuse, light is reflected equally in all directions and equation 3 is valid for any viewpoint. However, if the room contains specular materials, this equation is valid only if the viewpoint is close to the camera.

Finally, the per-channel per-pixel non-linear function g models the camera response. Thus, given a projector pixel, if we call d the corresponding light coming from the room (after projection and reflections) toward the camera and e the corresponding color that the camera has captured, we have, for the channel λ, e λ = g λ (d λ )

(4)

Compensation algorithm

In the following, we assume that the camera response is known and corrected. Our radiometric compensation method is of two steps. In the first step, the model is evaluated to characterize a specific projection system i.e. to find the v, w λ and H λ which model the system. In the second step, the evaluated model is inverted to compute compensated images. Once projected, such images are supposed to appear like the desired images.

Calibration

We use the method described in [Nayar et al. 2003] to determine v and w λ . In the following, we assume that the devices have three channels. Let us consider a pixel. Using the notations of the previous section, we have

a =   ar ag a b   , b =   br bg b b   =   wr(ar) wg(ag) w b (a b )   (5) v =   1 vrg v rb vgr 1 v gb v br v bg 1   , c =   cr cg c b   = vb (6)
To determine the elements of v, for example vgr and v br , we project two images which only differ in one channel. For example, if we change the red channel of the projected color, we have

a (1) =    a (1) r a (1) g a (1) b    , a (2) =    a (2) r a (1) g a (1) b    (7)
The corresponding captured color c verifies The other unknown elements of v can be computed, in the same way, by changing the green channel and then the blue channel. Since all pixels can be processed in parallel, all v matrices can be computed by projecting only four uniform images. The function w λ can be estimated by projecting every value of a λ . Indeed, if we project a λ , we can capture λ and compute b = v -1 c. Since b λ = w λ (a λ ), this gives w λ for a λ . By projecting gray images we can evaluate wr, wg and w b using the same images. To reduce the number of acquisitions, we carry out a few projection-captures only. The remaining values can then be estimated by interpolation.

   c (1) r c (1) g c (1) b    = v    b (1) r b (1) g b (1) b    ,    c (2) r c (2) g c (2) b    = v   b (2) r b (1) g b (1) b    (
This method makes it possible to compensate projection when no inter-reflection occurs. To adapt it to our system, we have to cancel inter-reflections in captured images. To achieve that, we capture direct illumination only, by using the technique proposed in [START_REF] Nayar | Fast separation of direct and global components of a scene using high frequency illumination[END_REF]]. This technique enables the system to separate direct and indirect illumination of a scene uniformly lit by a projector. Basically, it consists in projecting a checkerboard. Thus a point of the room corresponding to a pixel in a black square of the checkerboard is not directly lit. Therefore, the corresponding light captured by the camera is due to indirect lighting only. By translating the checkerboard, we can get the indirect light for every pixel. Finally, we can get the direct light for a pixel by considering a colored square. Here, the captured light is the sum of direct light and indirect light. Since we have characterized indirect light, we can get direct light by subtraction.

The last calibration step aims at determining λ where H λij is the part of light (of the channel λ) leaving i which is scattered in room and then reflected by j toward the camera. [START_REF] Sen | Dual photography[END_REF] proposes a hierarchical method to determine such a matrix. Basically, this method works by considering pixel pairs i.e. it determines the contribution of each projector pixel onto each camera pixel. In our implementation, we simply group pixels which are projected onto the same triangles of the room geometry.

Thus, to determine H λij , we project an image where all pixels matching the triangle i are white (other pixels are black). Knowing the projected image and a part of the projection response (v and w λ ), we compute the light which is effectively emitted by the triangle i in the room. This enables the system to normalize the captured image. By summing the value of all normalized captured pixels matching the triangle j, we obtain H λij .

Compensation

First, we have to compensate for inter-reflections. Equation 3 is equivalent to

C λ = F λ D λ (10)
where

F λ = (I + H λ ) -1 = I -H λ + H 2 λ -. . . (11)
Equation 10 gives the light each triangle should emit to compensate for inter-reflections. Since this equation considers triangles, we have to set a relation between pixels and triangles. We obtain the value of a triangle in D λ by summing the value of the corresponding pixels. Then, we compute equation 10. Finally, we can compensate for inter-reflections in the image by weighting each pixel with the value of the corresponding triangle in C λ . However, this will show discontinuities between triangles. Therefore we use the classic Gouraud method [START_REF] Gouraud | Seminal graphics: poineering efforts that shaped the field[END_REF]] to take back the values from triangles to pixels. This gives an image that compensates for inter-reflections.

Finally, we have to inverte equation 2 and equation 1 with

b = v -1 c, a λ = w -1 λ (b) (12)
Projecting pixel colors a should make the camera see the desired colors d. This radiometrically compensates the projection system.

Implementation

Unlike calibration, compensation has to be computed for every image to project. Therefore, it must be done as quickly as possible. By precomputing some steps (v -1 and F ) during calibration, we can reduce compensation computations. Consider we use RGB color spaces. Let T and N be the number of triangles and the number of pixels, respectively. Since F has been precomputed, the first compensation step requires three products of a T × T matrix by a T vector and Gouraud smoothing. Similarly, v -1 has been precomputed, therefore the second step requires the product of a 3 × 3 matrix by a vector, for each pixel (i.e. N products). Finally, for each pixel, compensating for w λ resumes to invert a sparsely discretized function i.e. find the neighboring samples and interpolate inverse values.

Results

We have tested these algorithms with our system (SXGA DLP projector, XGA 3-CCD camera). Our test room composed of specular white-painted walls onto which are hung sheets of colored paper. The calibration step requires about ten hours. This time is almost exclusively devoted to image acquisition, since the number of required captures is very significant. Compensation can be done quickly (about 5 images per second), using the described precomputations. We call error of a captured image, the evaluation of

1 N p cp -ĉp 2 2
where N is the number of pixels and cp (resp. ĉp) the Luv color of the pixel p in the desired image (resp. in the captured image).

Fig. 3 presents some of our results. These images give for each pixel of the projector, the color seen by the camera. Our test room (a) has specular material and colored area. The desired image (b) is a panoramic landscape. Without compensation (c), the projected environment is unacceptably too dark. A basic uniform compensation (d) makes the environment become more visible. However, we can still notice the materials of the room (colored area, specular highlights) and the projector response (color shifts, contrast attenuation). Our method (e) compensates these phenomena and manages to display an environment which is close to the desired one.

Conclusion and future works

In this paper, we have presented the Catopsys projection system. This system combines a videoprojector, a mirror and a rotating camera. It is low-cost, self-calibrating and performs immersive projection. Then, we have given a radiometric model of the projection process. This model accounts for non-linearity, color mixing and inter-reflections. Finally, we have given a calibration algorithm and a compensation algorithm, based on this model, which make it possible to compensate the projection process.

In future works, we will try to improve the accuracy of the method. Using high dynamic range images should avoid camerasensor saturation and thus extends the range of colors which can be compensated for. Using a perceptually-uniform color space, such as Luv, may also give more faithful results. Finally, we will try to explicitly characterize the materials of the scene to account for specular components.

Figure 2 :

 2 Figure 2: Camera-projector radiometric process.

Figure 3 :

 3 Figure 3: Compensation results for a virtual reality application. (a) room. (b) desired image. (c) uncompensated projection (error = 0.064). (d) uniformly compensated projection (error = 0.054). (e) compensated projection (error = 0.049).

Acknowledgements

This work is founded by the Audiovisual and Multimedia Program of the French Agency for Research (Catopsys project). The authors are grateful to Christophe LOHOU and Susan ARBON for help in preparing this paper.


Alexey Ozerov
email: alexey.ozerov@inria.fr

Emmanuel Vincent
email: emmanuel.vincent@inria.fr

Using the FASST source separation toolbox for noise robust speech recognition

Keywords: speech separation, source separation, general flexible framework, noise robust speech recognition

We describe our submission to the 2011 CHiME Speech Separation and Recognition Challenge. Our speech separation algorithm was built using the Flexible Audio Source Separation Toolbox (FASST) we developed recently. This toolbox is an implementation of a general flexible framework based on a library of structured source models that enable the incorporation of prior knowledge about a source separation problem via userspecifiable constraints. We show how to use FASST to develop an efficient speech separation algorithm for the CHiME dataset. We also describe the acoustic model training and adaptation strategies we used for this submission. Altogether, as compared to the baseline system, we obtain an improvement of keyword recognition accuracies in all conditions. The best improvement of about 40 % is achieved in the worst condition of -6 dB Signalto-Noise-Ratio (SNR), where 18 % of this improvement is due to the speech separation. The improvement decreases when the SNR increases. These results indicate that audio source separation can be very helpful to improve speech recognition in noisy or multi-source environments.

Introduction

The CHiME Speech Separation and Recognition Challenge aims to tackle speech separation and recognition in typical everyday listening conditions. The challenge employs noise background that has been collected from a real family living room using binaural microphones. Target speech commands have been mixed into the environment at a fixed position using genuine room impulse responses. The task is to separate the speech and recognize the commands being spoken using systems that have been trained on noise-free commands and room noise recordings. For more details about the challenge and the corresponding datasets, see the challenge web page 1 .

We here describe our submission to the challenge based on the Flexible Audio Source Separation Toolbox (FASST), developed in Matlab, that enables the incorporation of prior knowledge about source separation problems to build efficient source separation algorithms. To design a speech separation algorithm for the CHiME challenge we have used the following prior knowledge about the source separation problem:

1. The speaker identity and availability of clean speech signals for each speaker.

2.

A rough idea about the target speech source direction.

This work was supported in part by the Quaero Programme, funded by OSEO.

1 http://www.dcs.shef.ac.uk/spandh/chime/ challenge.html 3. Knowledge that the background noise can involve multiple sources and that there are usually no more than 4 active sources at the same time.

4. Knowledge of the temporal location of the target speech sentences within the noisy background.

Note that points 1 and 4 are used as well by the CHiME baseline recognizer. We here do not give any details about the toolbox and the underlying general flexible audio source separation framework, and rather refer the reader to visit the FASST web page2 and to read the corresponding article [START_REF] Ozerov | A general flexible framework for the handling of prior information in audio source separation[END_REF]. The rest of this paper is organized as follows. The speech separation algorithm based on FASST is presented in section 2. Section 3 describes the acoustic model training and adaptation strategies we employed. The results are presented in section 4 and the conclusions are drawn in section 5.

Speech separation using FASST

Our speech separation algorithm based on FASST consists in the following steps:

1. Speech spectral power model: For each of the 34 speakers, a speaker dependent 32-component nonnegative matrix factorization (NMF) model [START_REF] Ozerov | A general flexible framework for the handling of prior information in audio source separation[END_REF] is learned from 60 sentences randomly selected from the training set and converted to mono. The spectral patterns of the NMF model are first initialized by K-means clustering of the short term power spectra, and then reestimated using FASST.

2. Speech spatial model: An initial rank-1 convolutive filter is estimated from the clean reverberated speech of the development test set.

Background noise model:

The noise is modeled as a sum of 4 sources. Each source is given a rank-1 convolutive spatial model and an 8-component NMF spectral power model. This multi-source model is initialized randomly and learned using FASST from 20 seconds of speechfree background noise (10 seconds before and 10 seconds after each sentence), which can be extracted thanks to the availability of the temporal location of the target speech sentences.

Mixture model:

A mixture model consisting of 5 source models (1 for speech and 4 for noise) is created for the separation of the target speech. The spatial models and the NMF spectral patterns are initialized by those of the corresponding models described in the three previous steps. from each noisy speech sentence using FASST. Finally the speech source signal is extracted using FASST.

All learning operations are performed in the maximum likelihood (ML) sense.

We did some informal listening of the separated signals. The separated speech is usually quite clean with some artifacts and some interferences, especially from the concurrent speech sources coming from about the same direction. However, there is still some speech in the separated noisy background. Some separation examples are available from the demo web page3 .

Training and adaptation

Speech recognition is achieved using speaker-dependent acoustic models estimated from the clean speech of the training set. First, we have tried the models provided within the challenge. Second, we have tried to re-train these models by learning speaker independent models and fixing the acoustic Gaussian mixture model (GMM) variances and weights while adapting them to each speaker, i.e., only the means were reestimated. This constrained training was achieved by adding the -u m option to the HERest HTK function used for the reestimation of speaker-dependent model parameters.

Since the acoustic models are trained from clean speech and our goal is to recognize either noisy speech or separated speech (corrupted by source separation errors), the speech recognition performance can be improved by adapting the acoustic models to these perturbations. An adaptation set consisting of noisy speech sentences (60 sentences per speaker) was provided within the challenge. However, this set was provided without the corresponding speech-free background. Thus, we were not able to use this set, since it was not possible to separate it by our speech separation algorithm (in order to reproduce similar perturbations) that relies on the knowledge of the neighbouring speech-free background. We have thus created our own adaptation set consisting of 200 sentences per speaker randomly selected from 500 sentences of the training set and randomly mixed with the training set backgrounds (only the mixtures with Signal-to-Noise-Ratio (SNR) between -5 and 10 dB were retained). We then used the maximum a posteriori (MAP) adaptation script provided within the challenge to adapt the acoustic models to this adaptation set as it is, if no speech separation is used, or to its separated version, if speech separation is applied. Knowledge of the test or development SNR was not used. Whatever the SNR, the models were adapted on the whole adaptation set for each speaker.

Results

We have tested all the 8 possible combinations of the proposed options, i.e., with or without constrained training (Sec. 3), with or without MAP adaptation (Sec. 3) and with or without speech separation (Sec. 2). The results for both the development and test sets are given in Table 1. We see that, as compared to the baseline system (first line of Tab. 1), all three tested options lead to some improvement in almost all conditions and whatever their combination. On the development set, the best improvement of about 39 % is achieved in the worst condition of -6 dB SNR, with 16 % due to the constrained training (see lines 1 and 5 of Tab. 1), 5 % due to the MAP adaptation (see lines 5 and 7 of Tab. 1), and the remaining 18 % due to source separation (see lines 7 and 8 of Tab. 1). The improvement decreases but remains positive when the SNR increases.

Conclusion

First, our results indicate that audio source separation can be very helpful to improve speech recognition in noisy or multisource environment. Second, building a new algorithm for this particular source separation problem using FASST was really fast since no additional coding is needed. Indeed, it took us only two weeks to prepare this submission, all steps included. This demonstrates once again the usefulness of the FASST source separation toolbox and the underlying general flexible framework [START_REF] Ozerov | A general flexible framework for the handling of prior information in audio source separation[END_REF].

Table 1 :

 1 Speech recognition performance expressed in terms of keyword recognition accuracy (in %) for both the development and test sets and for different configurations. Abbreviations: "Constr. train." = constrained acoustic model training, "MAP. adapt." = MAP acoustic model adaptation, and "Src. sep." = using source separation.

					Development set				Test set
	train.	adapt. sep.	-6 dB -3 dB	0 dB	3 dB	6 dB	9 dB	-6 dB -3 dB	0 dB	3 dB	6 dB	9 dB
	No	No	No	31.08 36.75 49.08 64.00 73.83 83.08	30.33 35.42 49.50 62.92 75.00 82.42
	No	No	Yes	51.08 59.83 69.50 75.92 81.25 84.00	48.50 56.58 66.67 74.33 82.17 86.33
	No	Yes	No	44.08 51.17 62.17 71.92 81.08 88.08	43.58 50.08 62.50 73.25 82.08 87.83
	No	Yes	Yes	63.42 72.83 79.42 83.50 87.75 89.50	64.92 69.92 77.58 82.75 86.67 87.58
	Yes	No	No	47.00 49.83 61.25 73.58 82.83 88.25	44.00 50.08 63.33 73.50 83.25 90.17
	Yes	No	Yes	65.75 71.00 78.50 85.17 88.42 90.08	65.50 73.08 80.08 86.25 89.00 92.83
	Yes	Yes	No	51.67 57.17 70.08 78.17 86.83 89.58	52.92 59.50 69.75 79.92 85.75 91.67
	Yes	Yes	Yes	70.00 77.17 84.33 88.33 91.58 93.17	71.08 76.58 81.08 88.58 90.33 90.67

The NMF temporal activations are initialized randomly. All the parameters, except the NMF spectral patterns that are kept fixed, are separately reestimated Constr. MAP Src.

http://bass-db.gforge.inria.fr/fasst/

http://www.irisa.fr/metiss/ozerov/chime_ ssep_demo.html

Acknowledgements

The authors would like to thank the CHiME'11 organizers for very well-prepared challenge and especially Ning Ma for a constant support, interesting discussions and for his suggestion about the constrained acoustic model training we used.


Alexey Ozerov
email: alexey.ozerov@inria.fr

Mathieu Lagrange
email: mathieu.lagrange@ircam.fr

Emmanuel Vincent
email: emmanuel.vincent@inria.fr

GMM-based classification from noisy features

Keywords: Uncertainty-based classification, Gaussian mixture model, expectation maximization algorithm

We consider Gaussian mixture model (GMM)-based classification from noisy features, where the uncertainty over each feature is represented by a Gaussian distribution. For that purpose, we first propose a new GMM training and decoding criterion called log-likelihood integration which, as opposed to the conventional likelihood integration criterion, does not rely on any assumption regarding the distribution of the data. Secondly, we introduce two new Expectation Maximization (EM) algorithms for the two criteria, that allow to learn GMMs directly from noisy features. We then evaluate and compare the behaviors of two proposed algorithms with a categorization task on artificial data and speech data with additive artificial noise, assuming the uncertainty parameters are known.

Experiments demonstrate the superiority of the likelihood integration criterion with the newly proposed EM learning in all tested configurations, thus giving rise to a new family of learning approaches that are insensitive to the heterogeneity of the noise characteristics between testing and training data.

Introduction

Classification and detection systems generally have to face a wide variety of data distortion phenomena. This results in noisy observations or features that the system has to handle as robustly as possible. In this paper, we focus on classification from noisy data, whatever the type of noise (e.g., additive or convolutive). While our approach is quite general, we mostly consider classification of audio data in the experimental part and in the examples throughout the paper.

In order to reduce the sensitivity of the classifier to noise, many approaches can be taken at different levels. At the signal level, one can respectively apply noise suppression [START_REF] Ephraim | Statistical-model-based speech enhancement systems[END_REF] or source separation [START_REF] Ozerov | A general modular framework for audio source separation[END_REF] techniques, for stationary or nonstationary additive noise. At the feature level, one can define features that are robust to certain noises (e.g., additive or convolutive noise) [START_REF] Nadeu | Filtering time sequences of spectral parameters for speech recognition[END_REF] or to the interferences and artifacts produced by a noise suppression or a source separation algorithm applied at the signal level. Finally, at the classifier level, one can account for possible distortion of the features, given some information about this distortion [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF][START_REF] Deng | Dynamic compensation of HMM variances using the feature enhancement uncertainty computed from a parametric model of speech distortion[END_REF]. In this paper, we focus on the latter approach considering a Gaussian mixture model (GMM)-based generative classification task [START_REF] Reynolds | Large population speaker identification using clean and telephone speech[END_REF].

When facing non-stationary distortions, the features are sometimes completely masked. In this case, the features are effectively missing and should be disregarded. Assuming that the This work was supported in part by the Quaero Programme, funded by OSEO.

GMMs have been trained from clean data, some works have derived decoding systems that are able to cope with the so-called missing data problem [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF][START_REF] Barker | Decoding speech in the presence of other sources[END_REF][START_REF] Srinivasan | Transforming binary uncertainties for robust speech recognition[END_REF][START_REF] Shao | A computational auditory scene analysis system for speech segregation and robust speech recognition[END_REF].

Even though the availability of clean data for model training is a reasonable assumption in some scenarios, many others have to deal with noisy data at the learning stage. Indexing television or radio shows is an example, where we need to recognize which speaker is speaking at a given time. In this case, recordings of the speakers of interest in a controlled environment are usually not available. Moreover, one could also want to adapt available models (potentially learned on clean data) to the noisy data received during the decoding or to learn them directly from noisy data. In that case, one needs to define learning algorithms that are able to cope with the missing feature problem such as the one proposed in [START_REF] Ghahramani | Supervised learning from incomplete data via an EM approach[END_REF].

In practice, the features are usually neither completely clean nor completely masked. In order to benefit from this range of uncertainty, one may represent the uncertainty over the features by a probability distribution. For each feature we here assume a Gaussian distribution with its mean and covariance matrix representing, respectively, the expected value of the feature and its uncertainty. Again, assuming that the GMMs have been trained on clean data, one only needs to take the uncertainty into account at the decoding stage as proposed in [START_REF] Deng | Dynamic compensation of HMM variances using the feature enhancement uncertainty computed from a parametric model of speech distortion[END_REF][START_REF] Kolossa | Independent component analysis and time-frequency masking for speech recognition in multitalker conditions[END_REF]. However, the approach in [START_REF] Deng | Dynamic compensation of HMM variances using the feature enhancement uncertainty computed from a parametric model of speech distortion[END_REF][START_REF] Kolossa | Independent component analysis and time-frequency masking for speech recognition in multitalker conditions[END_REF] suffers from the following issues:

1. it is assumed that the clean data underlying the noisy observations have been generated by the GMMs, and 2. uncertainty is taken into account only at the decoding stage, assuming that the GMMs were trained from some clean data that are not always available, as mentioned above.

We address the first issue by defining a new data-driven criterion (log-likelihood integration) for GMM learning and decoding that, as opposed to the conventional model-driven criterion from [START_REF] Deng | Dynamic compensation of HMM variances using the feature enhancement uncertainty computed from a parametric model of speech distortion[END_REF][START_REF] Kolossa | Independent component analysis and time-frequency masking for speech recognition in multitalker conditions[END_REF] (likelihood integration), does not rely on any assumption regarding the distribution of the data. To address the second issue, we here derive two new Expectation Maximization (EM) algorithms [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] allowing learning GMMs from noisy data with Gaussian uncertainty for the conventional and the newly proposed criteria, respectively. The first EM algorithm generalizes both the algorithm proposed in [START_REF] Ghahramani | Supervised learning from incomplete data via an EM approach[END_REF], that is restricted to binary uncertainty, and the algorithm proposed in [START_REF] Arberet | Blind spectral-GMM estimation for underdetermined instantaneous audio source separation[END_REF]. The algorithm in [START_REF] Arberet | Blind spectral-GMM estimation for underdetermined instantaneous audio source separation[END_REF] was not investigated in the context of classification and it is restricted to the case of uncertainty with diagonal covariances and zero-mean GMMs with diagonal covariances.

In summary, we provide two contributions:

1. validation of the model-driven uncertainty decoding approach in [START_REF] Deng | Dynamic compensation of HMM variances using the feature enhancement uncertainty computed from a parametric model of speech distortion[END_REF][START_REF] Kolossa | Independent component analysis and time-frequency masking for speech recognition in multitalker conditions[END_REF] compared to a data-driven approach,

introduction of new uncertainty training algorithms enabling accurate training on noisy data.

The remaining of the paper is organized as follows. After a brief review of uncertainty decoding in Section 2, we introduce in Section 3 the two criteria to be optimized for training and classification over noisy data as well as the corresponding algorithms derived from the EM framework. Their behaviors are then evaluated and compared in Section 4 with a categorization task on artificial data and speech data with additive artificial noise.

GMM decoding from noisy data

Classification is the problem of assigning a sequence of observation vectors 1 , where i = 1, . . . , I are state indices, and µ i , Σ i and ω i ( P i ω i = 1) are respectively the mean, the covariance matrix and the weight of the i-th state. In other words, each vector xn is modeled as follows:

x = {xn} N n=1 (xn ∈ R M ) to a class C. Each class C is modeled by a GMM θ = {µ i , Σ i , ω i } I i=1
(xn|qn = i) ∼ N (µ i , Σi), P(qn = i) = ωi, ( 1
)
where qn denotes the state at time n. Under this model the likelihood of the observation sequence x is given by

p(x|θ) = Y N n=1 p(xn|θ), ( 2
) with p(x n |θ) = X I i=1 ω i N (x n |µ i , Σ i ), (3)
where

N (x n |µ i , Σ i ) 1 p (2π) M |Σ i | » - (x n -µ i ) T Σ -1 i (x n -µ i ) 2 - . (4)
Within this framework the common generative classification approach [START_REF] Reynolds | Large population speaker identification using clean and telephone speech[END_REF] consists in the following two steps:

1. Training: For each class C the corresponding parameters θ are estimated from some sequence of training vectors by maximizing the likelihood (1).

2. Classification: An observation sequence x is assigned to the class C for which the likelihood (1) is maximum.

Note, however, that one does not need the GMMs to represent perfectly the data distribution: the most important for classification is to obtain a good separator between the classes. This was confirmed by considering discriminative training of GMMs for classification [START_REF] Del Alamo | Discriminative training of GMM for speaker identification[END_REF].

Binary uncertainty

In the case where some components x n are missing and the remaining are available (xn = [x a n , x m n ] ), the probabilistic likelihood p(xn|θ) (3) cannot be evaluated in the usual manner. A first approach, called marginalization, is to approximate the likelihood by p(x n |θ) p(x a n |θ) [START_REF] Ghahramani | Supervised learning from incomplete data via an EM approach[END_REF]. A second approach, called data imputation, is to interpolate missing values so that the likelihood can be computed in the usual manner (p(xn|θ)

p([x a n , xm n ]|θ)). Numerous imputation methods can be considered from the simplest one, i.e., interpolating a given component with the mean value of this component over the training set, to more refined ones, like drawing interpolated values from the GMM conditionally to the evaluated class [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF].

The first approach is theoretically more seducing and empirically demonstrated to be more powerful in many cases as it roots the classification decision over observed data only. However, it should be noticed that it requires an adaptation of the classification mechanism which may not be feasible in some situations. In the latter case, only imputation is feasible.

Probabilistic uncertainty

By contrast with the binary uncertainty framework, we assume that x n is unknown and distributed as

xn ∼ N (yn, Σn), (5)
where the parameters y n and Σn are known.

For example, y n can be a feature computed from a distorted signal and Σn an estimate of the corresponding zero-mean distortion2 covariance. Alternatively, yn can be a feature computed from a signal provided by some source separation algorithm or an estimate of this feature obtained using the underlying source separation model [START_REF] Adiloglu | An uncertainty estimation approach for the extraction of individual source features in multisource recordings[END_REF]. In this case Σn is a covariance matrix of the corresponding estimation error. In this paper we have the latter application in mind rather than the former.

Since x is unknown, one cannot directly compute the likelihood [START_REF] Ozerov | A general modular framework for audio source separation[END_REF], and this quantity should be redefined so as to take the uncertainty specified by (5) into account. We consider two cases presented in the following subsections.

Likelihood integration

In a generative framework, the uncertainty equation ( 5) can be also rewritten as

y n ∼ N (x n , Σn ), (6)
or, in other words, it is assumed that

y n = x n + e n , ( 7
)
where e n is a zero-mean Gaussian noise with covariance matrix Σn that is independent from xn distributed as specified by [START_REF] Nadeu | Filtering time sequences of spectral parameters for speech recognition[END_REF]. Indeed, it is reasonable to assume that the noise and the features are independent. With these assumptions the joint likelihood of x and y is well-defined and it can be written as follows:

p(x, y|θ, Σ) = p(x|θ)p(y|x, Σ), (8)
where y = {y n } N n=1 and Σ = ˘Σ n ¯N n=1 . The corresponding Bayesian network is shown in Figure 1.

The likelihood integration approach consists in marginalizing the joint likelihood (8) over the missing features x, i.e., in considering the likelihood of y. The likelihood p(x|θ) from ( 2) is replaced by the following objective function [START_REF] Deng | Dynamic compensation of HMM variances using the feature enhancement uncertainty computed from a parametric model of speech distortion[END_REF][START_REF] Kolossa | Independent component analysis and time-frequency masking for speech recognition in multitalker conditions[END_REF]:

fLI(y, Σ|θ) = Z R M ×N p(y|x, Σ)p(x|θ)dx (9) = N Y n=1 I X i=1 ω i N (y n |µ i , Σ i + Σn ). ( 10
)
This approach is compatible with the marginalization approach for binary uncertainty [START_REF] Ghahramani | Supervised learning from incomplete data via an EM approach[END_REF] (see Sec.

Log-likelihood integration

As a generative approach, likelihood integration assumes that the distribution of the hidden data x n is accurately modeled by the GMM of the corresponding class, which may not always be the case in practice. We propose a new approach called loglikelihood integration that is totally data-driven. It does not rely on any assumption regarding the distribution of x, but makes as if all values encoded by (5) were actually observed. Note also that, in contrast to the likelihood integration, this approach does not have any probabilistic formulation that can be represented by a Bayesian network. The log-likelihood log p(x|θ) from ( 2) is replaced by its expectation over the observations [START_REF] Deng | Dynamic compensation of HMM variances using the feature enhancement uncertainty computed from a parametric model of speech distortion[END_REF], which results in the following objective function:

f LLI (y, Σ|θ) = E x ˆlog p(x|θ)|y, Σ˜= Z R M ×N p(x|y, Σ) log p(x|θ)dx = N X n=1 Z R M p(x n |y n , Σn ) log I X i=1 ω i N (x n |µ i , Σ i )dx n . (11)
Unfortunately, a closed form computation of the integral in [START_REF] Kolossa | Independent component analysis and time-frequency masking for speech recognition in multitalker conditions[END_REF] is only possible in the case of a single-state (I = 1) GMM. An approximate expression may be obtained by assuming that all data xn drawn from (5) in a given time frame n correspond to the same (unknown) state i, resulting in

fLLI(y, Σ|θ) ≈ N X n=1 log Z R M p(xn|yn, Σn) I X i=1 ωiN (xn|µ i , Σi)dxn = N X n=1 log I X i=1 ωiN (yn|µ i , Σi)e -1 2 tr(Σ -1 i Σn) . ( 12
)
Note also that, using Jensen's inequality [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], this approximation can also be shown to be an upper bound of (11).

GMM learning from noisy data

As discussed in the introduction, there are many application scenarios where it is desirable to train the models over noisy data.

Likelihood integration

To optimize criterion [START_REF] Ghahramani | Supervised learning from incomplete data via an EM approach[END_REF] we propose an EM algorithm, considering the true features x and the GMM state indices q = {q n } N n=1 as latent variables. The resulting EM updates are summarized in Algorithm 1.

Algorithm 1 One iteration of the EM algorithm for the likelihood integration-based GMM learning from noisy data. E step. Conditional expectations of natural statistics:

γ i,n ∝ ω i N (y n |µ i , Σ i + Σn ),
and

X i γ i,n = 1, ( 13
)
xi,n = Wi,n (yn

-µ i ) + µ i , ( 14
) b R xx,i,n = xi,n xT i,n + (I -W i,n ) Σ x,i , ( 15
)
where

W i,n = Σ i ˆΣi + Σn ˜-1 . ( 16
)
M step. Update GMM parameters:

ω i = 1 N N X n=1 γ i,n , ( 17
)
µ i = 1 P N n=1 γ i,n N X n=1 γ i,n xi,n , ( 18
) Σi = 1 P N n=1 γi,n N X n=1 γi,n b Rxx,i,n -µ i µ T i . ( 19
)

Log-likelihood integration

To optimize criterion [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] we propose an EM algorithm, considering only the GMM state indices q = {q n } N n=1 as latent variables. The resulting EM updates are summarized in Algorithm 2.

Discussion

These two algorithms were derived using different sets of latent variables: {x, q} for likelihood integration and q for loglikelihood integration. This is due to the fact that the complete data criterion corresponding to criterion (12) can be optimized in closed form, considering only q as latent variables, while it is not possible in general for criterion [START_REF] Ghahramani | Supervised learning from incomplete data via an EM approach[END_REF]. Note also that both EM algorithms reduce to the classical EM algorithm for GMM learning [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] when the covariance matrices Σn are all zero. Moreover, Algorithm 1 reduces asymptotically to the binary uncertainty-based EM proposed in [START_REF] Ghahramani | Supervised learning from incomplete data via an EM approach[END_REF] when the covariance matrices Σn are diagonal with the entries corresponding to observed data being zero and the entries corresponding to the missing data tending to infinity.

Experiments

We now consider two evaluation frameworks based on a classification task in order to compare the benefits of the two uncertainty handling methods. The first experiment considers artificial data sampled from GMMs whereas the second one considers features extracted from speech utterances of the TIMIT database [START_REF] Garofolo | DARPA TIMIT acoustic phonetic continuous speech corpus CDROM[END_REF].

In these two cases, the uncertainty over each feature, i.e., the covariance matrix Σn , is assumed correspond to the actual amount of noise in the data. Even though this cannot be realistically assumed in practice, this study focuses on validating Algorithm 2 One iteration of the EM algorithm for the loglikelihood integration-based GMM learning from noisy data. E step. Conditional expectations of natural statistics:

γi,n ∝ ωiN (yn|µ i , Σi)e -1 2 tr(Σ -1 i Σn) ,
and

X i γ i,n = 1, (20)
M step. Update GMM parameters:

ωi = 1 N N X n=1 γi,n, ( 21
)
µ i = 1 P N n=1 γi,n N X n=1 γ i,n y n , ( 22
)
Σi = P N n=1 γ i,n (y n -µ i )(y n -µ i ) T + Σn P N n=1 γ i,n . ( 23
)
the methodological aspects of the proposed approaches. For that purpose, controlling the characteristics of the uncertainty is valuable as it will be shown in the remaining of this section.

For the two experiments reported here, a wide range of possible setups have been considered. Finally, to ease the control of the uncertainty characteristics, all the covariance matrices Σn are considered as diagonal, i.e.,

Σn = diag ˘[σ 2 m,n ] m ¯. (24)
However, our framework is not limited to this specific case. The uncertainty in these experiments is controlled by two parameters. The level of noise called Feature to Noise Ratio (FNR) is expressed in dB as FNR = 10 log 10

P n xn 2 P n x n -y n 2 , ( 25
)
where the summations are over all features in all classes. As previously stated, interfering sources are highly non-stationary. As a consequence, a high level of variability of noise can be assumed from a feature component to another and from a time frame to the next. In order to reflect this phenomenon, we introduce a variability parameter called Noise Variation Level (NVL) and computed as the standard deviation of variances from (24) expressed in dB:

NVL = stdev " ˘10 log 10 σ2 m,n ¯m,n " . ( 26
)
As the noise over the data can be different for training and classification, we considered four parameters: This gives a total number of 375 setups. For each of these setups, training and classification are performed and the average number of correct classification is recorded. For each setup specified by its FNR and NVL, artificial noise is generated in the following manner. First, the logvariances {log σ2 m,n } m,n are drawn from the zero-mean unit variance Gaussian random distribution and scaled so as to satisfy (25). Second, the expected values y of the missing features are drawn from the Gaussian distribution (6), given x and Σn = { Σn } n defined by ( 24), and both y -x and {σ m,n } m,n are scaled by the same factor in order to satisfy (26). In order to remove additional variability the seed of the random number generator was re-initialized to the same value for each setup.

Artificial data

In the first experiment, two-dimensional artificial features were generated from three 4-states GMMs representing the corresponding three classes (Fig. 2, top, left). For each setup and each class a training sequence of length 300 and 100 test sequences of length 100 were randomly drawn from distributions of the corresponding GMMs. Artificial noise was generated and added as explained above. Figure 2 shows an example of clean data x and two examples of noisy data y with the same FNR = 10 dB and with different NVLs (0 and 8 dB). One can note that, as compared to low NVL, the distribution of the data with high NVL is closer to the clean data, but at the same time there are more outliers. Thus, for a given FNR, it should be easier to classify and to learn from data with large NVL, since the influence of the outliers can be diminished by taking into account the uncertainty.

Three approaches including likelihood integration, loglikelihood integration and the baseline conventional GMMbased classification [START_REF] Reynolds | Large population speaker identification using clean and telephone speech[END_REF] that does not take the uncertainty into account were evaluated for all 375 setups described above. To investigate the impact of the FNR and the NVL on the classification performance, selected results are shown in Figure 2.

The left part of Figure 2 shows the impact of the FNR at the training and testing stage. The likelihood integration approach is the most powerful. It gets the best correct classification rate and has the lowest dependency to the heterogeneity of the noise level between the testing and training sets. Compared to the Approaches: "Like int" = likelihhod integration, "Log like int" = log-likelihhod integration, and "No uncrt" = conventional GMM-based classification [START_REF] Reynolds | Large population speaker identification using clean and telephone speech[END_REF] that does not take into account the uncertainty. baseline approach that does not consider the uncertainty, the log-likelihood integration approach gets slightly better results in terms of accuracy but remains sensitive to the heterogeneity of the noise level between training and testing data. For example, to classify a test set with an FNR at 0 dB, a classifier trained with an FNR at 0 dB is almost three times better than a classifier trained with a cleaner dataset (FNR at 20 dB).

The right part of Figure 2 shows the impact of the NVL both at the training and testing stage. The likelihood approach behaves as expected (c.f., the above informal comparison of two noisy datasets represented at the bottom of Figure 2), i.e., the larger the NVL at train and test, the better the accuracy. The two other approaches do not benefit so much from a large NVL and face the same dependency to the above discussed heterogeneity. Indeed, for both the log-likelihood and baseline approaches, a classifier trained over a dataset at 0 dB NVL is better at classifying a test-set at 0 dB NVL than a classifier trained over a dataset at 8 dB NVL.

Speech data

The second experiment is mostly based on the one considered in [START_REF] Reynolds | Large population speaker identification using clean and telephone speech[END_REF]. The task is to recognize the speaker who pronounced the test sentence among the speakers for which GMMs have been trained. A subset of 10 male speakers of the TIMIT database [START_REF] Garofolo | DARPA TIMIT acoustic phonetic continuous speech corpus CDROM[END_REF] is considered. For each speaker and each setup, a fullcovariance 16-states GMM is trained using the two sa sentences and three si sentences of this speaker. The remaining five sx sentences were cut into 4 pieces of equal length that were used for testing. This gives a total of 200 test sequences of approximately half a second.

The Mel frequency cepstral coefficients (MFCCs) [START_REF] Vergin | Generalized mel frequency cepstral coefficients for large-vocabulary speakerindependent continuous-speech recognition[END_REF] are usually used as features for the speaker recognition task [START_REF] Reynolds | Large population speaker identification using clean and telephone speech[END_REF]. One of the advantages of the MFCCs is that they can reasonably be assumed to be decorrelated, and thus can be modeled by GMMs with diagonal covariance matrices, which decreases computational complexity. However, the error introduced by Figure 4: Speech data results: impact of the FNR in train and test (left) and impact of the NVL in train and test (right). Approaches: "Like int" = likelihhod integration, "Log like int" = log-likelihhod integration, and "No uncrt" = conventional GMM-based classification [START_REF] Reynolds | Large population speaker identification using clean and telephone speech[END_REF] that does not take into account the uncertainty. a source separation algorithm cannot be assumed to be decorrelated over MFCC vector dimensions, as we assume within this methodological evaluation framework (see ( 24)). This error would be rather decorrelated on the logarithm of Mel-frequency filter-bank outputs (LMFFB). In fact, at least for probabilistic model-based source separation approaches (see e.g., [START_REF] Ozerov | A general modular framework for audio source separation[END_REF]), the estimation error is usually decorrelated between different frequency bands [START_REF] Adiloglu | An uncertainty estimation approach for the extraction of individual source features in multisource recordings[END_REF], thus it is nearly decorrelated between the dimensions of the LMFFBs (there is still a small correlation due to the overlap of Mel-frequency filters). Thus, we here consider the LMFFBs with diagonal uncertainty covariance matrices and model them by GMMs with full covariance matrices. Since the MFCCs are computed from the LMFFBs by the discrete cosine transform (DCT), which is an orthogonal transform, this setting is very similar to the usual setting with the MFCCs, where the features are modeled by GMMs with diagonal covariance matrices and the uncertainty covariance matrices should be nondiagonal. We use LMFFB vectors of dimension 20.

The same experiments as in the case of artificial data were carried out and selected results are shown on Figure 2.

The left part of Figure 4 shows the impact of the FNR at the training and testing stage. The conclusions made over the synthetic dataset still hold: the likelihood integration approach also gets the best correct classification rate and has the lowest dependency to the heterogeneity of the noise level between the testing and training sets.

The right part of Figure 4 shows the impact of the NVL both at the training and testing stage. The likelihood approach behaves as expected, i.e., the higher the NVL at train and test, the better the accuracy. On contrary, the accuracy of the other two approaches appears to be rather independent of the NVL and it is just slightly superior to that of a random classifier.

Conclusions

We studied in this paper the learning of GMMs models over noisy data using two criteria. The first criterion called likeli-hood integration, proposed in [START_REF] Deng | Dynamic compensation of HMM variances using the feature enhancement uncertainty computed from a parametric model of speech distortion[END_REF], assumes that the data have been generated by a GMM. We proposed a second criterion called log-likelihood integration that does not make such assumption, which theoretically seemed beneficial while facing real data. For both criteria, we proposed EM algorithms in order to estimate the parameters of the GMMs over noisy data.

Experimental evaluation conducted over synthetic and speech data with knowledge of the uncertainty demonstrated the superiority of likelihood integration over log-likelihood integration and the standard approach that does not consider uncertainty information, thus validating the approach taken in [START_REF] Deng | Dynamic compensation of HMM variances using the feature enhancement uncertainty computed from a parametric model of speech distortion[END_REF] for both the training and testing stages. The experiments also demonstrated that considering the uncertainty allows us to: (i) handle the heterogeneity of noise between the training and testing sets, (ii) exploit the variability of noise for improved performance.

Future work will consider realistic uncertainty estimates from source separation algorithms over a larger evaluation dataset in order to evaluate the practical benefit of this new classification scheme. Another interesting research direction would be to consider the log-likelihood integration within a GMM-based classification framework with discriminative training [START_REF] Del Alamo | Discriminative training of GMM for speaker identification[END_REF].

  3.3).

Figure 1 :

 1 Figure 1: Bayesian network representing GMM-modeled missing features under the uncertainty model (6).

  FNR train and NVL train control the noise over the training data and FNR test and NVLtest control the noise over the data to be classified. The following values are considered: FNRtrain, FNRtest = {-20, -10, 0, 10, 20}, NVL train = {0, 4, 8}, NVL test = {0, 2, 4, 6, 8}.

Figure 2 :

 2 Figure 2: Artificial data examples: GMMs used for clean data generation (top, left), clean data x (top, right), noisy data y with zero NVL (bottom, left), and noisy data y with high NVL (bottom, right).

  Impact of FNR (NVL train = NVL test = 0 dB) Like int (FNR train = 0 dB) Like int (FNR train = 20 dB) Log like int (FNR train = 0 dB) Log like int (FNR train = 20 dB) No uncrt (FNR train = 0 dB) No uncrt (FNR train = 20 dB) Impact of NVL (FNR train = FNR test = -10 dB) Like int (NVL train = 0 dB) Like int (NVL train = 8 dB) Log like int (NVL train = 0 dB) Log like int (NVL train = 8 dB) No uncrt (NVL train = 0 dB) No uncrt (NVL train = 8 dB)

Figure 3 :

 3 Figure3: Artificial data results: impact of the FNR in train and test (left) and impact of the NVL in train and test (right). Approaches: "Like int" = likelihhod integration, "Log like int" = log-likelihhod integration, and "No uncrt" = conventional GMM-based classification[START_REF] Reynolds | Large population speaker identification using clean and telephone speech[END_REF] that does not take into account the uncertainty.

  Impact of FNR (NVL train = NVL test = 0 dB) Like int (FNR train = 10 dB) Like int (FNR train = 20 dB) Log like int (FNR train = 10 dB) Log like int (FNR train = 20 dB) No uncrt (FNR train = 10 dB) No uncrt (FNR train = 20 dB) Impact of NVL (FNR train = FNR test = 0 dB) Like int (NVL train = 0 dB) Like int (NVL train = 8 dB) Log like int (NVL train = 0 dB) Log like int (NVL train = 8 dB) No uncrt (NVL train = 0 dB) No uncrt (NVL train = 8 dB)

For the sake of brevity we omit here the class label C in the set of the model parameters θ.

Note that considering zero-mean distortion does not mean reducing the generality of the approach. Indeed, in case of distortion with nonzero mean μn one should simply consider y n -μn instead of y n .