Loic Barrault

Renato De Mori
email: renato.demori@univ-avignon.fr

Roberto Gemello
email: roberto.gemello@loquendo.com

Franco Mana
email: franco.mana@loquendo.com

Driss Matrouf
email: driss.matrouf@univ-avignon.fr

Loquendo

VARIABILITY OF AUTOMATIC SPEECH RECOGNITION SYSTEMS USING DIFFERENT FEATURES

The paper describes the use of two recognizers fed by different acoustic features. The first recognizer performs Multiple Resolution Analysis (MRA) while the other recognizer computes JRASTA Perceptual Linear Prediction Coefficients (JRASTAPLP). The two recognizers use the same denoising method but perform different partitions of their acoustic spaces. Experiments with the Italian and Spanish components of the AURORA3 corpus show that the two systems provide, in a significant proportion of cases, substantially different posterior probabilities for the same phoneme in the same time interval. A decision rule is proposed when two different words are hypothesized by the two recognizers. It is based on the probability that a hypothesis is correct, given the identity of the word hypotheses that are in competition. Significant word error rate (WER) reductions have been found for the CH1 proportion of the Italian and Spanish components of the AURORA3 corpus.

INTRODUCTION

Attempts have been recently reported [START_REF] Zhang | Word level confidence annotation using combinations of features[END_REF] [START_REF] Utsuro | Confidence of agreement among multiple LVCSR models and model combination by SVM[END_REF] on the use of neural networks, decision trees and other machine learning techniques to combine the results of Automatic Speech Recognition (ASR) systems in order to reduce word error rates (WER). In [START_REF] Zolnay | acoustic feature combination for robust speech recognition[END_REF], log-linear model combination and feature combination models are proposed to enhance ASR performance. In [START_REF] Sankar | Bayesian model combination (baycom) for improved recognition[END_REF] a Bayesian model combination for ASR outputs is proposed. It computes the likelihood that a sentence hypothesized by each system is correct given system hypotheses and their confidence scores. Independence is assumed among systems and correctness probabilities depend on the overall system performance without considering which phonemes of words are hypothesized by each system. . Other recent work focuses on factors affecting WER in ASR systems. In [START_REF] Kim_ | Why Speech Recognizers Make Errors? A Robustness View[END_REF], it is shown that a combination of utterancebased Signal-to-Noise Ratio (SNR) and its local variations provide useful predictions of recognition error rates.

The objective of the research described in this paper is to understand when and how different sets of features affect recognition performance. Specific probabilities are introduced that a phoneme hypothesis is correct given the phonemes that would be hypothesized with two different feature sets and their posterior probabilities. In this way, correctness probabilities explicitly depend on the competing word and phoneme hypotheses generated by systems using different feature sets. Two versions of an ANN/HMM hybrid recognizer are used. The two recognizers are fed by different sets of acoustic features, but have the same topology. The feature sets are those obtained by Multi Resolution Analysis (MRA) followed by Principal Component Analysis (PCA) described in [START_REF] Gemello | Multiple Resolution Analysis for Robust Automatic Speech Recognition[END_REF] and by Perceptual Linear Prediction (PLP) described in [START_REF] Hermansky | RASTA Processing of Speech[END_REF] followed by RASTA filtering. The latter features will be indicated as JRASTAPLP. The same denoising technique, described in [START_REF] Gemello | Multiple Resolution Analysis for Robust Automatic Speech Recognition[END_REF] is used for both feature sets. The ANNs are trained to recognize phonemes and transitions using a corpus of phonetically balanced sentences which are completely independent from the test data. It is important to notice that combining the scores of the two recognizers does not change the recognition results if their most likely hypotheses are the same.

Experiments were carried out with the test sets of the Italian and Spanish portions of the AURORA3 corpus. Intervals of phoneme posterior probabilities computed by the two systems are defined in section 2. Statistics of joint values of posterior probabilities for each phoneme class are reported showing important differences in the behavior of the two systems. An analysis of the consensus among the recognizers is presented in section 3. The probability that they generate correct hypotheses is high when their outputs are the same. A specific decision strategy is proposed in section 4 for the situation in which there is a discrepancy between the words hypothesized by the recognizers. Experimental results with this strategy are reported in section 5. Unfortunately, even if this is unlikely, different recognizers may generate the same wrong hypothesis. Diagnosis in this case is difficult, because situations of this type are rare and the output discrepancies cannot be used to guide further analysis. This aspect is not treated in this paper. . Such a space can be partitioned as in Figure 1. Counts for the occupation of zones in this space can be collected for each pair of phonemes j m f , f

PERFORMANCE COMPARISON OF THE FEATURE SETS

. Each unit of the count represents the posterior probabilities generated by the two recognizers for the two phonemes when the same time frame is considered.

(nT) Y f P j 0.5 0.1 0.1 0.5 (nT) Y f P m A B H C M D L E F Figure 1.
-Partition of the space of posterior probabilities.

In order to investigate feature variability for each phoneme and each set of features, the case j m f f has been considered and statistics have been collected for each phoneme f. This has been obtained by performing forced alignment of data in the AURORA3 training set (not used to train the ANNs) and considering the intersection of the time intervals hypothesized by the two systems for the same phoneme.

Zone H represents the case in which a phoneme is the most likely hypothesis with both feature sets. In this case the phoneme should be correctly hypothesized. Zone L correspond to the case in which hypothesizing the phoneme with both feature sets is similar to doing it with a random choice among candidates with uniform distribution, Zone M represents the case in which features indicate a possibility for the phoneme but that a decision would not be very reliable. Zones A, B and C indicate that JRASTAPLP better represents a phoneme than MRA and zones D, E and F indicate the contrary. Statistics collected about the fact that probabilities for a phoneme are represented by points in the above defined zones, may indicate possible confusions due to the inadequacy of the features or of the models to represent a phoneme. Moreover, other useful diagnostic results could be obtained when different modeling techniques or acoustic models in different languages are used.

An experiment was carried out with the training sets of the CH0 and CH1 data of the Spanish and Italian components of the AURORA3 corpus. The results, grouped by classes of phonemes, are reported in Table I. After forced alignment, phoneme segment hypotheses are generated for each stream of features. The average posterior probabilities, for each segment and for each feature set, have been computed and the pair was represented by a label in Figure 1.

Results clearly show that nonsonorant phonemes are more difficult to recognize than sonorant ones especially because plosives are short in time and features for fricatives are often distorted by denoising. Vowels are less affected by denoising because, in general, their segments have a fairly high SNR.

MRA appears to provide better discrimination for many phonemes, but is particularly weak on semivowels. The reason is that these phonemes are often very short and a fine frequency resolution is required for their distinction. An analysis frame of long duration is required for obtaining the necessary frequency resolution with MRA. The average frame SNR after denoising varies between 6.5 and 14.8 for vowels, 3.6 and 10.4 for nonsonorant consonants, 6.0 and 10.2 for sonorant consonants.

The most impressive result is that there are many cases in which probabilities with one set of features are high while they are low with the other set of features. In the lines labeled with DEF, MRA features show better performance, while in the lines labeled with ABC, the JRASTAPLP features show better performance. It is rather unlikely that this result is only due to limits of the modeling techniques especially because there are many cases in which one probability is above 0.5 and the other is below 0.1. Likely, part of the discrepancies (cases A,B,D,F) are due to intrinsic feature variability which makes it difficult to infer appropriate probability distributions leading to good discrimination. The following remarks are also worth to be mentioned:

the ANNs take into account a fairly long acoustic context, experiments were performed on a noisy small vocabulary corpus, thus phoneme recognition refer to limited phonetic contexts, ANNs were not trained on the application considered for the experiment nor on car data, rather they were trained on standard telephone environments, in general, the Italian corpus shows lower performance, especially on vowels in spite of the fact that the average SNR is greater than 10 after denoising for all the vowels but /uh/, the number of cases in D,E,F is generally higher than that in A,B,C showing that it is more frequent that the most likely phoneme with MRA is correct.

SITUATION DEFINITION

A sequence of words W generates a path in m and a path in j . The two paths are related by the fact that they are two representations of the same signal. The two recognizers label each path with a word sequence, namely W m and W j . Several situations can be defined based on the types of consensus between the words in W m and W j . The two sequences may be the same or they may differ by a variable number of words. In the latter case, it may be useful to identify the time segments in which the difference appears and analyze the types of discrepancies between the two recognizer outputs.

An initial set of situations or result comparison states are defined as follows: Q1: Q3: two segments of W m and W j have approximately the same time bounds but without any word in common.

When the state of the recognition results is Q1, combining the scores of hypotheses generated by the two recognizers would produce the same result. It will be shown that in Q1 the WER is low. Diagnosis reveals that deletion errors in Q1 are essentially due to the imprecision of voice activity detection or denoising and not to the capability of the feature sets to discriminate among phonemes. If a portion of a word signal is considered as a non-speech segment, then the remaining part is often attached to a neighbor word which is likely to be misrecognized. Segmentation errors are a frequent cause of errors in Q1. Insertions are often due to the fact that background noise is considered as a speech segment.

Using the test sets of the CH1 data of the Spanish and Italian components of the AURORA3 corpus, in the case of full sentence consensus, the coverage is 72.66% for Spanish with a WER of 0.16% and 63.16% for Italian with a WER of 2%.

For the cases corresponding to state Q2, the WER is also low. Errors are essentially caused by substitutions. Some errors are due to segmentation and denoising but others reveal poor discrimination power among phonemes. This weakness is common to both feature sets in certain zones of their acoustic spaces, especially those corresponding to low SNR.

It is unlikely that correction of errors in the case of word consensus can be obtained with better strategies and scoring methods. Some errors can be avoided by the use of good language and lexical models. This aspect is not considered in this paper whose main objective is to report on the comparative study of feature performance.

In the absence of consensus, the oracle WER reveals that many errors could potentially be avoided by a scoring and decision strategy conceived to maximize the probability of selecting the correct candidate when it is proposed by only one of the two recognizers.

TYPES OF DIVERGENCE BETWEEN SYSTEM OUTPUTS

When there is no consensus among the hypotheses generated by the two recognizers, then their most likely hypotheses are aligned. Experiments described in detail in [START_REF] Gemello | Multiple Resolution Analysis for Robust Automatic Speech Recognition[END_REF] show that the MRA recognizer has better performance in both languages than the other recognizer. So its hypotheses are considered as reference for aligning the results of the two recognizers when they disagree. Let The situations defined in Table II, describing discrepancies among the recognizers, are worth considering.

Table II The following decision strategy is proposed when there is no sentence consensus. The MRA word candidate is selected, except for situation sb for which the decision rule introduced below is applied.

represents the fact that w is correct when the hypotheses with the highest score obtained with the MRA and the JRASTAPLP feature sets are respectively m w and j w .

The decision rule is:

) w ,

AUTOMATIC SPEECH RECOGNITION EXPERIMENTS

ASR experiments were conducted with the test sets of the AURORA3 corpus. Only CH1 data were used for the Italian and Spanish portions. The recognizers were trained with reallife telephone data and not with the AURORA3 training corpus. Denoising was performed by non-linear spectral subtraction as described in [START_REF] Gemello | Multiple Resolution Analysis for Robust Automatic Speech Recognition[END_REF].

After aligning the best word sequences generated by the two recognizers, the following decision strategy is used. If there is consensus at the word level, then the word hypothesis is validated, otherwise each word hypothesis ) e , b ( w m generated by the recognizer using the MRA features is considered for validation. This is motivated by the fact that this recognizer has a lower WER than the one using JRASTAPLP features. III. The WER reduction is 13.67% for Italian and 18.24% for Spanish.

The strategy compares

CONCLUSIONS AND FUTURE WORK

Two ASR systems with different feature sets have been used to perform diagnosis and combination of results for improving recognition accuracy. With forced alignment of the reference sentence, it was observed that the phoneme posterior probabilities obtained with the two feature sets for the same phoneme and the same time interval may be very different in a substantial proportion of cases. This may be due to the fact that probabilities are computed by different models, by the effect of denoising or by a different discrimination power of the feature sets for different phoneme classes. Future work will investigate whether or not the trend is confirmed with different modeling techniques. If the trend will be confirmed, then attention will be paid to the features in the attempt to characterize confusions due to intrinsic variability.

A new strategy for combining ASR system results has been proposed. It is based on the probability that a hypothesis is correct, given the identity of the word hypotheses that are in competition. Significant WER reductions have been found for the CH1 proportion of the Italian and Spanish components of the AURORA3 corpus. Future work will investigate how this approach can be extended to tasks with large vocabularies.

  or a sequence of words hypothesized by the MRA recognizer in the time interval (b,esequence of phonemes hypothesized by the JRASTAPLP recognizer in a time segment with substantial overlapping (more than 50%) with (b,e).

  and insertion (si m ) All the other cases multiple discrepancies(md mj )

  more than 50% of the frames, 2. special case of j-deletion: a non-speech hypothesis is generated by the recognizer using JRASTAPLP features in the time interval of the training set of Aurora 3 and used for selecting between the competing hypotheses. The decision rule is the (3). Results, in terms of WER, are reported in Table

Table I -

 I Distributions of the probability zones for the CH1 data of the Spanish and Italian components of the AURORA3 corpus (comma represents union)

		Language	Non	Sonorant	Vowels
			sonorant	consonant
	D,E,F	SPANISH 20,74%	13,39%	23,12%
		ITALIAN 20,49%	18,14%	12,59%
	A,B,C	SPANISH 15,05%	13,8%	5,3%
		ITALIAN 16,01%	11,09%)	14,85%
	H	SPANISH 40,36%	63,94%	66,4%
		ITALIAN 37,54%	61,04%	58,02%
	M	SPANISH	2,31%	1,79%	1,53%
		ITALIAN	8,29%	3,87%	7,01%
	L	SPANISH 21,55%	7,08%	3,64%
		ITALIAN 17,67%	5,86%	7,54%

Table III -

 III Performance, in terms of WER, of the new decision strategy compared with that of the best system

		Italian	Spanish
	MRA system	20.34%	15.19%
	New Strategy -substitutions	6.2%	5.71%
	New Strategy -deletions	9.79%	1.99%
	New Strategy -insertions	1.57%	4.72%
	New Strategy -overall WER	17.56%	12.42%

ACKNOWLEDMENTS

The work described in this paper is part of a research effort carried out in the DIVINES project on feature extraction and diagnosis for Automatic Speech Recognition. This project is partially funded by a programme of the Human Language Technology Division of the European Community.


Driss Matrouf
email: driss.matrouf@univ-avignon.fr

Loic Barrault
email: loic.barrault@univ-avignon.fr

Renato De Mori
email: renato.demori@univ-avignon.fr

A GENERAL METHOD FOR COMBINING ACOUSTIC FEATURES IN AN AUTOMATIC SPEECH RECOGNITION SYSTEM

A general method for the use of different types of features in Automatic Speech Recognition (ASR) systems is presented. A gaussian mixture model (GMM) is obtained in a reference acoustic space. A specific feature combination or selection is associated to each gaussian of the mixture and used for computing symbol posterior probabilities. Symbols can refer to phonemes, phonemes in context or states of a Hidden Markov Model (HMM). Experimental results are presented of applications to phoneme and word rescoring after verification. Two corpora were used, one with small vocabularies in Italian and Spanish and one with very large vocabulary in French.

INTRODUCTION

It is known that Automatic Speech Recognition (ASR) systems make errors (see, for example, [START_REF] Sarikaya | Semantic Confidence Measurement for Spoken Dialog Systems[END_REF]). This is due to the imperfection of the various models used, on the limitations of the feature extracted and on the apporximations of the recognition engines.

With the purpose of increasing robustness, recent ASR systems combine streams of different acoustic measurements, such as multi-resolution spectral/time correlates. This is motivated by the assumption that some characteristics that are de-emphasized by a particular feature are emphasized by another feature, and therefore the combined feature streams capture complementary information present in individual features ( [START_REF] Vaseghi | Multi resolution phonetic/segmental features and models for HMM-based speech recognition[END_REF], [START_REF] Kirchhoff | Combining articulatory and acoustic information for speech recognition in noise and reverberant environments[END_REF], [START_REF] Hon | Combining frame and segment based models for large vocabulary continuous speech recognition[END_REF], [START_REF] Jiang | Acoustic feature selection using speech recognizers[END_REF], [START_REF] Hermansky | RASTA Processing of Speech[END_REF], [START_REF] Hariharan | Noise robust speech parameterization using multiresolution feature extraction[END_REF], [START_REF] Kingsbury | Robust speech recognition in noisy environments: The 2001 IBM spine evaluation system[END_REF], [START_REF] Kleinschmidt | Improving word accuracy with Gabor feature extraction[END_REF], [START_REF] Morgan | TRAPping Conversational Speech: Extending TRAP/Tandem approaches to conversational telephone speech recognition[END_REF], [START_REF] Zhu | On Using MLP Features in LVCSR[END_REF], [START_REF] Pujol | Comparison and combination of features in a hybrid HMM/MLP and a HMM/GMM speech recognition system[END_REF], [START_REF] Hegde | Speech processing using joint features derived from the modified Group delay function[END_REF]).

At another level, attempts have been recently reported ( [START_REF] Zhang | Word level confidence annotation using combinations of features[END_REF], [START_REF] Utsuro | Confidence of agreement among multiple LVCSR models and model combination by SVM[END_REF]) on the use of neural networks, decision trees and other machine learning techniques to combine the results of ASR systems fed by different feature streams or using different models in order to reduce word error rates (WER). In ( [START_REF] Zolnay | Acoustic feature combination for robust speech recognition[END_REF]) it is shown that loglinear combination provides good results when used for integrating probabilities provided by acoustic models.

Other approaches integrate some specific parameters into a single stream of features ( [START_REF] Thomson | Use of periodicity and jitter as speech recognition feature[END_REF], [START_REF] Zolnay | Robust speech recognition using a voiced-unvoiced feature[END_REF], [START_REF] Stephenson | Speech recognition with auxiliary information[END_REF]). A generalization of this approach consists in concatenating different sets of acoustic features into a single stream. In order to reduce modelling complexity, algorithms have been described to select subsets of features in a long stream using a criterion that optimizes automatic classification of speech data into phonemes or phonetic features. Unfortunately, pertinent algorithms are computationally intractable with these types of classes as stated in ( [START_REF] Omar | Maximum Mutual Information Based Acoustic-Features Representation of Phonlogical Features For Speech Recognition[END_REF]), where a sub-optimal solution is proposed. Such a solution consists in selecting a set of acoustic measurement that guarantees a high value of the mutual information between acoustic measurements and phonetic distinctive features.

An analysis of the factors affecting WER in ASR systems is ceretainly useful for an effective use of differentr feature sets. In ( [START_REF] Kim | Why Speech Recognizers Make Errors? A Robustness View[END_REF]) ASR errors of spoken dialog data collected from various telephony-based customer care services are analyzed. It is shown that a combination of utterance-based Signal-to-Noise Ratio (SNR) and its local variations provide useful predictions of recognition error rate. The correlation between SNR and WER is also made evident in the study described in ( [START_REF] Siohan | Constructing ensembles of asr systems using randomized decision trees[END_REF]) regarding the English portion of the MALACH corpus for which an influence on WER of the number of syllables per second is also reported. In ( [START_REF] Hunt | Speech recognition, sylabification and statistical phonetics[END_REF]) the importance of correctly defining syllable boundaries is dis-cussed and it is shown that ASR errors for certain consonants depend on whether a consonant is in a prevocalic or a postvocalic position in a syllable.

The approach described in this paper considers the possibility of dynamically combining different feature sets and acoustic models in an ASR system. Given a sampled input signal S = s(kτ ), where τ is the sampling period, let us consider the sequence of samples in a time window of length T and represent such a sequence for the n-th window as follows:

Y n = [s(kτ )] (n+1)T nT , n = 0, 1, ......, N
For each value of n, the window sequence [s(kτ )] (n+1)T nT of signal samples is transformed into a feature vector Y a (nT ) represented in a feature space a . Features have an intrinsic variability with respect to the symbols q ∈ Q which describe a spoken message. Variability may cause equivocation represented by the fact that different symbols of Q may be coded into signal segments leading to the same vector Y a (nT ). Equivocation varies from point to point of a given feature space. In order to reduce equivocation, it is thus interesting to vary the choice or the use of features depending on the sample sequence based on which symbol hypotheses are hypothesized.

INTEGRATING DIFFERENT FEATURES AND MODELS

Given feature samples Y a (nT ), hypotheses about symbols q ∈ Q are generated by computing the posterior probabilities P µ [q|Y a (nT )]. Symbols may represent phonemes, phonemes in context, transients or other phonetic descriptors. Computation of these probabilities is performed using acoustic models µ. If different models and features are available, then posterior probabilities can be obtained with log-linear interpolation as follows:

log P [q|Y n ] = µ,a w a µ [Y r (nT )] log P µ [q|Y a (nT )] (1)
Where w a µ [Y r (nT )] are weights depending on the feature sample in a reference space indicated by the superscript r. Initially, speech analysis is performed in the reference space r . The reference features can be the ones that produce the best ASR results or good results with minimal computation time.

Features Y r (nT ) may be reliable in certain zones of the acoustic space r and less reliable in other zones. Furthermore, in certain zones different models may provide better probability approximations than others.

The choice and use of models and features in a given point of r depends on the values of the weights

w a µ [Y r (nT )]. Let g(nT ) = w a µ [Y r (nT )
] be a vector of weights corresponding to a point of r .

In practice, only a limited number of vectors of weights can be considered in a system. Let G be the set of these vectors. If only one feature stream is available with symbol posterior probabilities computed with two models, for example Artificial Neural Networks (ANN) and Gaussian Mixture Models (GMM) and the symbols are states of a Hidden Markov Model (HMM) then the vectors contain the coefficients of a log-linear combination of the probabilities provided by the two models. If vector elements are binary variables, and only one of them is equal to one, then changing vectors corresponds to switch between a feature stream and another.

In practice, vector values can be estimated or determined by experiments only in certain points of the reference space r and the symbol posterior probability in a point of the acoustic space has to be estimated by smoothing the probabilities estimated with different vectors of weight values. For this purpose, a probability density P [g|Y r (nT )] is introduced. It indicates the probability that vector g ∈ G provides the right weight model for computing log P (g|Y n ) with the (1). Different weight models for computing a symbol posterior probability are then used as follows:

P (q|Y n ) = g∈G P [qg|Y n ] = g∈G P g [q|Y n ]P [g|Y r (nT )]
(2) where P g [q|Y n ] indicates the posterior symbol probability is computed with the (1), using the weight model g.

Probability P [g|Y r (nT )] can be computed by associating to each weight model g a Gaussian distribution as follows:

P [g|Y r (nT )] = N (µ g , Σ g , Y r (nT )) γ∈G N (µ γ , Σ γ , Y r (nT )) (3)
There are many possible uses of the (1) and the (2). An initial attempt has been made by considering two sets of features, one of which is used as reference. Zones of the acoustic space r where the reference set of features is unreliable are determined and a new set of features is used in those zones. Hypothesis generation in these zones depends on the two feature streams. The (2) is used, in this case, as follows. A set of Gaussian distributions P [g|Y r (nT )] is obtained in the reference space. In each point Y r of r there is a Gaussian distribution N (Y r ) with the highest value. If features in r are reliable, then the features in reference space r are associated to N (Y r ), i.e. g(Y r ) = {features in r }, otherwise the new set of features is associated to the point. Other solutions are possible by associating to g(Y r ) a log linear combination of models and/or features according to the [START_REF] Barrault | Characterizing Feature Variability in Automatic Speech Recognition Systems[END_REF].

A method for predicting the reliability of features based on their variability in a point Y r is described in ( [START_REF] Barrault | Characterizing Feature Variability in Automatic Speech Recognition Systems[END_REF]).

EXPERIMENTAL RESULTS ON PHONEME RECOGNITION

Two experiments have been conducted on the use of the (2) for computing posterior phoneme probabilities.

They have been conducted on the French ESTHER corpus [START_REF] Galliano | The ESTER phase II evaluation campaign for the rich transcription of French broadcast news[END_REF] and on the Italian portion of the CH1 part of AURORA3. The French ESTHER corpus has a very large vocabulary and contains 80000 broadcast news sentences in French. A set of 10000 sentences have been isolated to infer a mixture of 1024 distribution probabilities P [g|Y r (nT ) in a reference space of PLP features. With Maximum A Posteriori (MAP) probability estimation, a mixture of 1024 Gaussians has been obtained for each of the 37 phonemes in French. After elimination of the Gaussians which did not have samples associated to them, a total of 10000 Gaussians were kept. A portion of the corpus consisting of about 60000 sentences was used to estimate a vector P g [q|Y n ] associated to each Gaussian g of the mixture. This was obtained by introducing a counter per phoneme and incrementing, after forced alignment and for each time frame, the counter of the phoneme f corresponding to a segment by a quantity equal to P g [f |Y n ]. The contents of phoneme counters have been normalized to ensure that the same number of frames is used for each phoneme.

A test set containing 20000 phonemes was used. After forced alignment, classification was performed on each segment, ignoring the phonemes used for the alignment. In each segment, the (2) was applied and the phoneme with the highest posterior probability was considered as the recognition result. Such a result was compared with the one obtained using an HMM model per phoneme.

Notice that probabilities P g [f |Y n ] can be computed once forever using different features for different Gaussians g .

A similar test was conducted with the CH1 Italian portion of AURORA3. The training set was used for obtaining the GMM in the reference space and log-linear interpolation was used for computing P g [q|Y n ]. The results are shown in Table 1. corpus using the (2) using HMMs Italian CH1 9.2 14 French ESTER 30.5 37.7

Table 1. Results in terms of phoneme error rates(%) using the (2) and phoneme HMMs With a KLD lower than 0.004, then 63% of the Italian digits are validated with a phoneme error rate of 0.59%.

HYPOTHESIS VERIFICATION AND FEATURE SWITCHING

A simple application of the ( 1) is for hypothesis verification. The features in r are initially used for generating word hypotheses.

An hybrid system consisting of an Artificial Neural Network and a set of Hidden Markov acoustic Models (HMM) is used. The recognizer uses a feature set obtained by Multi Resolution Analysis (MRA) followed by Principal Component Analysis (PCA). A denoising technique described in ( [START_REF] Gemello | Multiple resolution analysis for robust automatic speech recognition[END_REF]) is used. The stream of acoustic features will be indicated as {Y m (nT )}. The value of T is 10 msecs and the feature set contains seven analysis frames centered on the frame at nT . Let m be the space of the MRA features. In this case r = m .

The ANN is trained to recognize phonemes and transitions using a corpus of phonetically balanced sentences which are completely independent from the test data. The ANNs have 636 outputs, one for each phoneme and each transition between two successive phonemes.

A new set of features is obtained with Perceptual Linear Prediction (PLP) followed by RASTA filtering. These features will be called JRASTAPLP and the corresponding vector will be indicated as {Y j (nT )} belonging to the acoustic space j . The vectors Y m (nT ) and Y j (nT ) represent two different observations of a speech segment Y n centered on the same sample.

The two ASR systems separately fed by the two feature streams use acoustic models which are trained with a general telephone corpus without using any data of the application which is being tested. The ANNs of the two systems have the same topology and he same denoising algorithm is applied to the two feature streams.

Let W = w 1 . . . w h . . . w H be the sequence of word and pause hypotheses generated by an initial decoder using features of r . Let w h = h 1 . . . h k . . . H K(h) be the sequence of phonemes given in the lexicon of w h .

Initially only feature streams Y m (nT ) are used for recognition, but the two feature streams are used for verification. The hypothesis w h is generated in the time interval (t hb , t he ), with segments labelled with phoneme hypotheses. Let assume that phoneme h k is hypothesized in the time interval (t h k b , t h k e ).

Two posterior probability streams

P m G [q|SEG(h k b, h k e)] and P j G [q|SEG(h k b, h k e)] are computed in each segment SEG(h k b, h k e).
The probabilities are computed using segment Gaussian Mixture Models. The Kullback-Leibler distance (KLD) between the two streams is then computed:

KLD[SEG(h k b, h k e)] = D P m G [q|SEG(h k b, h k e)]||P j G [q|SEG(h k b, h k e)] = q∈Q P m G [q|SEG(h k b, h k e)] log P m G [q|SEG(h k b,h k e)] P j G [q|SEG(h k b,h k e)] (4)
The features of r are likely to be the cause of a wrong hypothesization of w h if the probability

P [w h |Y n (t hb , t he )] is not low.
The symbol w h indicates the fact that hypothesis w h is not correct. If the probability P [w h |Y n (t hb , t he )] is above a given threshold for one word or for a time segment containing a sequence of words and pauses, then the set of features Y j (nT ) is considered in that segment and recognition is also performed using feature vectors P g (q|Y n ) in the [START_REF] Barrault | Characterizing Feature Variability in Automatic Speech Recognition Systems[END_REF] with a new selection g.

The following approximation is proposed for computing P [w h |Y n (t hb , t he )]:

P [w h |σ kld ] = 1 H(K h ) H(K h k=1 P (h k |σ kld ) (5)
where

P (h k |σ kld ) = 1 -P (h k |σ kld ) and σ kld = KLD[SEG(h k b, h k e)].
The hypothesis w h is accepted if P [w h |Y n (t hb , t he )] is below a given threshold. Other specific thresholds are used for dealing with the cases of word insertion and deletion.

It is possible that contiguous word hypotheses generated with features {Y m (nT )} are incorrect. In this case, the segment corresponding to the sequence is processed with the new feature set {Y j (nT )} leading to a sequence of word hypotheses with a different number of words.

When word hypotheses are generated with feature vectors {Y j (nT )}, it is possible that there is a word consensus with hypotheses generated with the reference features. It is possible to investigate whether or not there is a consensus on an error. This has not been done yet and the problem will be investigated in future work. In the absence of word consensus, a decision is made based on the hypothesis with the lowest probability of being wrong.

EXPERIMENTAL RESULTS ON VERIFICATION

Experiments have been performed with the Italian components of the Aurora3 database (connected digits collected in car environment). The acoustic models employed were hybrid HMM-ANN trained on large corpora completely disjoint from Aurora3 namely the domain independent, phonetically balanced SpeechDat1-2 corpora. The training corpora are made of telephonic read speech and were recorded in quiet environments. Different HMM-ANN models were trained, one for J-RASTA PLP and one for MRA, with the same training set for each language. The Aurora3 corpus contains a set of close-talking utterances indicated as CH0 and a set of hand-free utterances, indicated as CH1. Utterances of CH0 are nearly clean, as the close-talking microphone collects little environmental noise, while utterances of CH1 are quite noisy as the hand-free microphone gathers a lot of car noise. Aurora3 is divided into training and test components. The test corpus was used for producing the results, in terms of WER, reported in Table 2.

Baseline results were obtained with MRA features which resulted to perform better than JRASTAPLP features for this task and with this setting ( [START_REF] Gemello | Multiple resolution analysis for robust automatic speech recognition[END_REF]). Oracle re-sults refer to what is obtained by comparing MRA and JRASTAPLP result with the reference and always deciding for the correct result if it is produced by at least one of the systems. This strategy results are the results obtained with the strategy proposed in this paper. Table 2. Results in terms of WER with the baseline, the Oracle and the strategy proposed in this paper

CONCLUSIONS AND FUTURE WORK

A general method for the combined use of different feature sets has been proposed. Specific symbol posterior probability computation is associated to each gaussian of a mixture. These probabilities can be computed once forever and simply retrieved during decoding or rescoring. Examples of application have been presented showing that the proposed method of computing phoneme posterior probabilities is effective even when a single feature set is used.

As an extension of the (2), it possible to consider two acoustic spaces as reference. In this case, symbol posterior probabilities are computed as follows:

P (q|Y n ) =
7. ACKNOWLEDMENTS

  gm,g j ∈G P g [q|Y m n , Y j n ] * P [g m |Y m n ] * P [g j |Y j n ]

This work was supported by European Commission 6th

Framework Program project DIVINES under the contract number FP6-002034.