Hans-Guenter Hirsch
email: hans-guenter.hirsch@hs-niederrhein.de

Harald Finster

A New Approach for the Adaptation of HMMs to Reverberation and Background Noise

Keywords: Robust Speech Recognition, HMM Adaptation, Hands-free Speech Input

Looking at practical application scenarios of speech recognition systems several distortion effects exist that have a major influence on the speech signal and can considerably deteriorate the recognition performance. So far, mainly the influence of stationary background noise and of unknown frequency characteristics has been studied. A further distortion effect is the handsfree speech input in a reverberant room environment.

A new approach is presented to adapt the energy and spectral parameters of HMMs as well as their time derivatives to the modifications by the speech input in a reverberant environment. The only parameter, needed for the adaptation, is an estimate of the reverberation time. The usability of this adaptation technique is shown by presenting the improvements for a series of recognition experiments on reverberant speech data. The approach for adapting the time derivatives of the acoustic parameters can be applied in general for all different types of distortions and is not restricted to the case of a hands-free input.

The use of a hands-free speech input comes along with the recording of any background noise that is present in the room. Thus there exists the need of combining the adaptation to reverberant conditions with the adaptation to background noise and unknown frequency characteristics. A combined adaptation scheme for all mentioned effects is presented in this paper. The adaptation is based on an estimation of the noise characteristics before the beginning of speech is detected. The estimation of the distortion parameters is based on signal processing techniques. The applicability is demonstrated by showing the improvements on artificially distorted data as well as on real recordings in rooms.

Introduction

The use of speech recognition as alternative input device is especially of interest where the user has not his hands available for controlling a keyboard or mouse. Quite often this input mode comes along with the need of a hands-free speech input. For reasons of practical usage and personal comfort it is especially of interest without the need of wearing a close-talking microphone.

The drawback of a hands-free speech input is a modification of the speech by the acoustic environment when the input takes place in a room. The influence of transmitting speech in a room can be modeled as superposition of the sound on the direct path from the talker's mouth to the recording microphone and multiple reflections of the sound at the walls and any equipment inside the room. For stationary conditions the transmission can be modeled as convolution of the speech with a room impulse response (RIR). But the RIR changes as soon as the talker moves in the room or room conditions change like in case of opening a door or a window or other people moving in the room. Thus the adaptive estimation of the RIR is a quite difficult and complex task.

Several approaches exist for enhancing speech that has been recorded in hands-free mode inside rooms. Some of these approaches have also been applied for the improvement of speech recognition. The methods can be separated in single and multi-channel processing techniques. Most of the multi-channel approaches, e.g. [START_REF] Omologo | Environmental conditions and acoustic transduction in hands-free speech recognition[END_REF][START_REF] Bitzer | Multi microphone noise reduction techniques for hands-free speech recognition -a comparative study[END_REF][START_REF] Liu | Blind deconvolution of reverberated signals[END_REF][START_REF] Seltzer | Likelihood-maximizing beamforming for robust hands-free speech recognition[END_REF], are based on a beamforming technique to reduce the influence of the reflected sound or on a correlation based multi-channel processing. A variety of different techniques are the basis for the single channel approaches, e.g. [START_REF] Avendano | Study on the dereverberation of speech based on temporal filtering[END_REF][START_REF] Kingsbury | Perceptually inspired signal processing strategies for robust speech recognition in reverberant environments[END_REF][START_REF] Yegnanarayana | Enhancement of reverberant speech using LP residual signals[END_REF][START_REF] Gelbart | Double the trouble: Handling noise and reverberation in far-field automatic speech recognition[END_REF][START_REF] Tashev | Reverberation reduction for improved speech recognition[END_REF][START_REF] Wu | A two-stage algorithm for enhancement of reverberant speech[END_REF][START_REF] Kinshita | Efficient blind dereverberation framework for automatic speech recognition[END_REF]. Some are based on modifying the envelope contours of subband energies.

Only a few approaches exist that try to improve speech recognition by modifying the pattern matching process [START_REF] Couvreur | Fast adaptation for robust speech recognition in reverberant environments[END_REF][START_REF] Palomäki | Missing data speech recognition in reverberant conditions[END_REF]. One method [START_REF] Raut | Model adaptation by state splitting of HMM for long reverberation[END_REF] is also based on an adaptation and modification of HMM parameters as in the approach presented here.

Looking at HMMs that are used for modeling speech in recognition systems, the detailed knowledge about the transmission as it is given by a RIR is not needed. Each state of a HMM represents a short speech segment with several tenth of milliseconds duration. The state contains information about the distribution of some spectral parameters within the segment. Usually a type of MEL filterbank is applied. In general HMMs describe speech with a quite low resolution with respect to time and frequency in comparison to the detailed description with a RIR. Thus the estimation of the RIR is not really needed to include the modifications of the HMM parameters that are caused by the hands-free speech input.

A new approach is presented in the next chapter for adapting the parameters of HMMs to the speech transmission inside a room. The method is based on the description of a room transmission by an impulse response with an exponentially decaying envelope as approximation for a real RIR. This approximation is applied to the fairly rough modeling of speech as a sequence of HMM states. As consequence of the exponentially decaying shape of the impulse response, the acoustic excitation at a certain point in time will also be seen at later time segments. The effect of reverberation is an temporal extension of an acoustic excitation. This extension is modeled by adding contributions of earlier states with respect to energy and spectral parameters. The only parameter, needed for the adaptation, is an estimate of the reverberation time T60 that defines the contour of the exponentially decaying RIR. Several recognition experiments have been performed to proof the usability of the new approach.

In most hands-free speech input situations background noise is present in the room. Thus, the HMM adaptation for the effects of a hands-free speech input is only useful when it can be combined with a technique for compensating the influence of stationary background noise and of an unknown frequency characteristic. During the recent years a lot of investigations have been carried out to reduce the deterioration of the recognition performance due to additive background noise and unknown frequency characteristics. The approaches are either based on the extraction of robust features in the front-end, e.g. [START_REF] Macho | Evaluation of a noise robust DSR front-end on Aurora databases[END_REF][START_REF] Gadrudadri | Qualcomm-ICSI-OGI features for ASR[END_REF]ETSI, 2003), or on the adaptation of the HMM parameters to the noise conditions, e.g. [START_REF] Gauvain | Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains[END_REF][START_REF] Minami | Adaptation method based on HMM composition and EM algorithm[END_REF][START_REF] Sankar | A Maximum-likelihood approach to stochastic matching for robust speech recognition[END_REF][START_REF] Gales | Robust continuous speech recognition using parallel model combination[END_REF][START_REF] Gales | Nice Model-Based Compensation Schemes for Robust Speech Recognition[END_REF][START_REF] Woodland | Speaker adaptation for continuous density HMMs: A review[END_REF]. Most of the adaptation techniques try to estimate some kind of HMM parameter mapping with a maximization of the likelihood score as optimization criterion. This work uses signal processing approaches for estimating the distortion effects. The estimated distortion parameters are taken for the adaptation on the basis of a signal processing model that describes the spectral modifications due to the distortions. A frequency weighting can be caused, e.g. by the special characteristics of the recording microphone. In case of a hands-free speech input a frequency weighting might also occur due to the frequency-dependency of T60. This is not covered by the new approach which assumes one frequency independent value for T60 so far. But it can be compensated with an additional adaptation to unknown frequency characteristics.

This paper shows how the new adaptation technique can be combined with the adaptation of HMMs to additive noise and unknown frequency characteristics (Hirsch, 2001a). This approach is based on the well known PMC method [START_REF] Gales | Model based techniques for noise robust speech recognition[END_REF]. The results of several recognition experiments are presented in the last chapter that proof the usability of the combined adaptation to all distortion effects as they can occur in real application scenarios of speech recognition systems. The achieved results are compared to the application of the well known MLLR (maximum likelihood linear regression) approach [START_REF] Leggeter | Maximum likelihood linear regression for speaker adaptation of continuous density Hidden Markov Models[END_REF] as an alternative adaptation technique.

Adaptation to hands-free speech input

The new approach for adapting the energy and spectral parameters of HMMs will be derived in this chapter. It is based on the approach of modeling the transmission in a reverberant room by an impulse response with an ideal, exponentially decaying shape. This will be presented in the first section. The next section describes the adaptation of the static energy and spectral parameters of HMMs derived from the ideal modeling of the RIR. Finally the last section presents a new technique to adapt also the time derivatives of the energy and spectral HMM parameters.

Modeling the influence of a hands-free speech input

The multiple reflections of sound in a room can be ideally described by an exponential decay of the acoustic energy which has been the result of early investigations in room acoustics [START_REF] Kuttruff | Room Acoustics[END_REF]. This leads to a room impulse response h(t) with an exponentially decaying shape. The only parameter for the description of the exponential shape is the reverberation time T60 that takes approximately values in the range of about 0,2 to 0,4 seconds for smaller rooms and of about 0,4 to 0,8 seconds for larger rooms. It can take values above 1 second for very large rooms like churches. The reverberation time depends on the interior in the room and the individual absorption characteristics of the walls.

( )

The RIR can be transformed to the room transfer function by means of a Fourier transform. The room transfer function has a contour that changes very fast along frequency. Usually only the envelope of the room transfer function is of interest when looking at the filterbank approaches that are applied for extracting acoustic features in speech recognition. This effect can be covered by an adaptation to an unknown frequency characteristic.

More important for the frame based analysis in speech recognition is the influence on the contour of the short-term energy along time. The energy contours of a speech signal are shown in figure 1 before and after the transmission in a room. The energy is usually estimated as short-term energy in frames of about 20 ms duration. It can be seen that the reverberation leads to an artificial extension of each sound contribution. This extension occurs as so called reverberation tail with the exponentially decaying shape of the RIR. The same effect will also be seen when looking at the energy contours in single subbands of a MEL based filterbank that is usually applied in the front-end of a speech recognition system. [START_REF] Houtgast | Predicting speech intelligibility in rooms from the modulation transfer function[END_REF] which can be mathematically described as ( )

2 60 10 ln 6 2 1 1 ) ( ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⋅ ⋅ ⋅ ⋅ + = T F F m π (2)
The low pass characteristic of the modulation transfer function is shown in figure 2 for different values of T60. The cut-off frequency of the low pass characteristic is shifting to lower values of the modulation frequency for increasing values of T60. This corresponds to longer reverberation tails for higher values of T60. The artificial extension of sound contributions can lead to masking the acoustic parameters of low energy sounds by the parameters of a preceding sound with higher energy.

Adaptation of static parameters

Looking at a sequence of HMM states the acoustic excitation described by the parameters of a single state will also occur in succeeding states at a certain attenuation. This is based on the assumption that the HMMs have been trained on clean speech, recorded with a close talking microphone. Figure 3 tries to visualize this effect.

Figure 3: The distribution of energy at state S 1 due to reverberation Each state S i of a HMM describes a speech segment with an average duration dur(S i ) that can be derived from the transition probability p(S i | S i ) to remain in this state.

( )

states NR i with S states all for t S S p S dur i shift i i i _ 1 | 1 1 ) ( ≤ ≤ ⋅ - = , ( 3
)
where t shift is the time for shifting the analysis window in the feature extraction and NR_states is the number of states of an individual HMM. t shift takes a value of 10 ms for all analysis schemes applied in these investigations. This corresponds to a frame rate of 100 Hz.

The energy contribution of the acoustic excitation in the first state to a succeeding state S i can be estimated by integrating the squared RIR as defined by equation 1 over the time segment of the succeeding state

( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 1 0 2 1 1 2 1 , = ∂ + = = ∂ = ∫ ∑ ∫ ∞ - = t t
i i α (4)
Given an estimate of the reverberation time T60 the contribution factors can be individually calculated for all states of all HMMs. The approach for estimating T60 will be described later.

Usually each state of a HMM is defined by a set of spectral parameters like the MEL frequency cepstral coefficients (MFCCs) and an energy parameter. These parameters are the means of Gaussian distributions. The corresponding variances are needed as further parameters to completely define the shape of the Gaussians.

The mean of an energy parameter at an individual state S i can be adapted by adding the energy contributions of the state itself and the preceding states

( ) ( ) ( ) ( ) ( ) ∑ = - - - - ⋅ = + ⋅ + ⋅ + ⋅ = i j j j i i i i i i i i i i i S E S E S E S E S E 1 , 2 2 , 1 1 , , ~α α α α (5)
In the same way the means of the power density spectra can be adapted. In case of using MFCCs, the cepstral coefficients have to be transformed back to the spectral domain first.

{ } ( ) ( ) ( ) { } { } mel NR EXP mel NR IDCT cep NR X X X X X X C C C C _ 2 1 _ 2 1 _ 2 1 0 ,..., , log ,..., log , log ,..., , , ⎯ ⎯ → ⎯ ⎯ ⎯ → ⎯ (6)
where NR_cep is the highest index of the cepstral coefficients and NR_mel is the number of bands in the MEL frequency range. For NR_cep a value of 12 is chosen and NR_mel takes a value of 24 in our realization. |X k | represents the value of the magnitude spectrum in the MEL band with index k.

Then the power spectra can be adapted in the same way as the energy parameter

( ) ( ) ( ) ( ) ( ) mel NR k for S X S X S X S X S X i j j k j i i k i i i k i i i k i i i k _ 1 ~1 2 , 2 2 2 , 2 1 1 , 2 , 2 ≤ ≤ ⋅ = + ⋅ + ⋅ + ⋅ = ∑ = - - - - α α α α (7)
The adapted spectra X ~ have to be transformed to the MFCCs again. In practice, mainly 2 to 3 preceding HMM states have an influence on the current state. This depends on the reverberation time and on the average durations of the HMM states.

The variances are not adapted. It turned out in earlier investigations [START_REF] Gales | Model based techniques for noise robust speech recognition[END_REF]Hirsch, 2001a) that the modification of the variances has only a minor influence on the improvement of the recognition performance.

The effects of this adaptation approach are visualized in figure 4

) ( ) ( ) { } ( ) ( ) ( ) { } … … , 20 , 10 , 0 , , , 3 2 1 ms X ms X X t X t X t X k k k Spline S k S k S k ⎯ ⎯ → ⎯ (9)
Thus the spectrum can be recreated at a frame rate of 100 Hz as it is also defined by the window shift of 10 ms in the feature extraction. The spectrogram of the clean HMM is shown in graph a) of figure 4 as it has been trained with the utterances of the TIDigits data base [START_REF] Leonard | A Database for speaker-independent digit recognition[END_REF]. Only the utterances containing a single digit have been taken for the training. The contributions of the fricatives at the end can be clearly seen in the high frequency region where the formants of the vowel are visible in the middle of the word.

The spectrogram in graph b) represents the HMM that has been trained on the TIDigits data after applying an artificial reverberation to the training utterances. The reverberation tails can be clearly seen when looking at the contours in individual MEL bands.

The spectrogram in graph c) of figure 4 represents the HMM after adapting the clean HMM with the new approach. A fixed value is chosen for the reverberation time T60. The reverberation tails can also be seen in this figure . Comparing it with the spectrogram trained on reverberated data, a lot of similarities are visible. This shows that the new approach allows an adaptation of the static parameters that is comparable with training the HMM on data that have been recorded under reverberant conditions.

In practice each HMM state tries to model a certain speech segment with a mixture of Gaussian distributions for each feature component. In this case the energy parameter at state S i and for the mixture component with the index mix j is adapted by calculating

( ) ( ) ( ) ( ) ( )
i i i j i i i i i i i i i j i i i j i _ 1 _ 1 , , , ~1 1 , , 2 2 , 1 1 , , ≤ ≤ ≤ ≤ ⋅ + ⋅ = + ⋅ + ⋅ + ⋅ = ∑ - = - - - - α α α α α where ( ) S E
is the average energy at state S by weighting the energies of the different mixture components with the corresponding mixture weighting factors

( ) ( ) ( ) ( ) ∑ ∑ = = = ⋅ = mix NR j j j mix NR j j mix w with mix S E mix w S E _ 1 _ 1 1 , (11)
NR_mix is the number of Gaussian distributions for modeling this acoustic parameter in each individual HMM state. Thus, the influence of an earlier state is considered by using the average energy at this earlier state.

In the same way the power density spectra of individual mixture components are adapted by transforming back the set of average cepstral coefficients to the spectral domain first and adapting the subband energy in each Mel band in the same way as the energy parameter.

( ) ( ) cep NR m for mix C mix w C j m mix NR j j m _ 1 _ 1 ≤ ≤ ⋅ = ∑ = (12) { } ( ) ( ) ( ) { } { } mel NR EXP mel NR IDCT cep NR X X X X X X C C C C _ 2 1 _ 2 1 _ 2 1 0 ,..., , log ,..., log , log ,..., , , ⎯ ⎯ → ⎯ ⎯ ⎯ → ⎯ ( ) ( ) ( ) ∑ - = ⋅ + ⋅ = 1 1 2 , 2 , 2 , , ~i k i j i k i i j i k S X mix S X mix S X α α (14)
The adapted power density spectrum is transformed to the cepstral domain again.

Estimation of T60

The estimated reverberation time T60 is the only parameter that is needed for the adaptation as it has been described in the previous section. The recognition of an utterance is done with a set of adapted HMMs where the applied value of T60 has been estimated from the recognition of the previous utterance. T60 is estimated after the recognition of an utterance by a search for this set of adapted HMMs that leads to a maximum likelihood for another forced recognition of the already recognized sequence of HMMs. The restriction to the forced recognition of the already recognized HMM sequence is introduced to limit the computational costs. This iterative process is visualized in figure 5. The sequence of buffered feature vectors is used to perform the match with the previously recognized HMM sequence. At the beginning this match is performed with adapted HMMs for the previous estimate of T60 and for values of T60 that differ by ±20 ms. The values for the probability, that the feature vectors match with the sequence of adapted HMMs, are taken as input for the search of this T60 value that leads to a maximum likelihood. Dependent on the achieved probabilities the estimated value of T60 is lowered or increased by another 20 ms or the search process is stopped in case the previous estimate of T60 leads to the maximum likelihood. In case the search process is continued, another set of adapted HMMs is derived from the clean HMMs (13) (10) where the adaptation can be restricted to the previously recognized HMMs. This newly adapted HMMs are used for another forced match and the search for the maximum likelihood. Because the hands-free conditions will usually alter only slowly in practical applications, the modification of T60 is restricted to the range of ±40 ms from the previous estimate. Thus only a few matches are needed, so that the computational costs are fairly low.

It turns out that the estimated value of T60 varies for different speakers even though the hands-free condition is the same. This seems to be dependent on the speaking rate. Thus, this adaptation technique includes also a kind of duration modeling to some extent.

Adaptation of Delta parameters

Comparing the contours of the clean and the reverberant HMM at individual Mel bins it becomes obvious that also the Delta and Delta-Delta parameters as time derivatives of the static parameters are modified by the influence of the hands-free speech input. This can be seen for example in figure 4 where a "valley" is visible between the vowel and the succeeding phoneme for the clean HMM. This "valley" is filled by the reverberation tails for the reverberant HMM versions. This indicates that also the time derivatives will be different in this region.

The Delta parameters are calculated in the feature extraction for the frame at time i t as sum of weighted differences between the static parameters of preceding and succeeding frames [START_REF] Young | The HTK book (version 3.3)[END_REF]. For example the calculation of the Delta logarithmic energy

( ) i t E log ∆ is done as ( ) ( ) ( ) [ ] ∑ ∑ = = - + ⋅ = - ⋅ = ∆ 3 1 2 3 1 2 log log log j j j i j i i j norm with norm t E t E j t E (15)
where , , , ,

1 1 + - i i i t t t
describe the window shift by 10 ms.

The Delta parameters of the reverberant speech are estimated as described below by looking at the adapted static parameters of all HMM states. The average logarithmic frame energies of all states are considered for a single HMM. A Spline interpolation is applied to recreate the average energy contour at a frame rate of 100 Hz.

( ) ( ) ( ) { } ( ) ( ) ( ) { } … … , log , , 10 log , 0 log , log , log , log 3 2 1 i Spline S S S t E ms E E t E t E t E ⎯ ⎯ → ⎯ (16)
In the same way an interpolated version of the average energy contour can be calculated for the average frame energies of the clean HMM.

The procedure for calculating the Deltas in the feature extraction, as described by equation 15, can be applied to the interpolated energy contours of the clean and the adapted HMM. Thus the average logarithmic Delta energies ( )

i clean t E log ∆
) ( ) ( ) , 20 , 10 , 0 log log log ms ms t all for t E t E t E i i clean i adapted i diff = ∆ - ∆ = ∆ (17)
These values describe the average differences between the Delta logarithmic energies of the adapted and the clean HMM at each frame. By means of a Spline interpolation the average differences are calculated for all HMM states.

( ) ( ) ( ) { } ( ) ( ) ( ) { } … … , log , log , log , log , , 10 log , 0 log 3 2 1 S diff S diff S diff Spline i diff diff diff t E t E t E t E ms E E ∆ ∆ ∆ ⎯ ⎯ → ⎯ ∆ ∆ ∆ (18)
A weighted version of these average differences is added to the corresponding Delta parameters of the clean HMM to create a set of adapted Delta parameters.

i S diff j S clean j S i i i _ 1 _ 1 log , log , log ≤ ≤ ≤ ≤ ∆ ⋅ + ∆ = ∆ β (19)
This is done individually for each state and for each mixture component. A factor β is introduced for the weighted summation of the differences. During recognition experiments we found a value of 0.7 for β to achieve highest performance.

The Delta cepstral parameters can be adapted in the same way. The average logarithmic Mel spectral values are taken as basis as they can be calculated by equations 12 and 13 from the average cepstral coefficients for each HMM state. A Spline interpolation can be applied to recreate the contour of the logarithmic Mel magnitude in each Mel band at the frame rate of 100 Hz.

( ) ( ) ( ) { } ( ) ( ) ( ) { } mel NR k for t X ms X X t X t X t X i k k k Spline S k S k S k _ , , 2 , 1 , log , , 10 log , 0 log , log , log , log 3 2 1 … … = ⎯ ⎯ → ⎯
The logarithmic spectral domain seems to be the right domain for applying the Spline interpolation even though the interpolation could be also immediately applied to the average cepstral parameters. The interpolated average logarithmic spectra are transformed to the cepstral domain.

( ) ( ) ( ) ( ) ( ) ( ) { } ( ) ( ) ( ) { } , 10 , 0 ,..., , log ,..., log , log _ 1 0 _ 2 1 ms t for t C t C t C t X t X t X i i cep NR i i DCT i mel NR i i = ⎯ ⎯ → ⎯ (21)
The Delta coefficients can be calculated for the contour of each individual average cepstral coefficient. This can be done again for the clean as well as for the adapted HMM so that the difference between these two versions can be estimated.

( ) ( ) ( ) cep NR m for and ms ms t all for t C t C t C i i m i m i m clean adapted diff _ 1 , 20 , 10 , 0 ≤ ≤ = ∆ - ∆ = ∆ (22)
These values describe the average differences between the Delta cepstral coefficients of the adapted and the clean HMM at each frame. By means of a Spline interpolation the average differences are calculated individually for each cepstral coefficient for all HMM states.

( ) ( ) ( ) { } ( ) ( ) ( ) { } cep NR m for t C t C t C t C ms C C S m S m S m Spline i m m m diff diff diff diff diff diff _ 1 , , , , , , 10 , 0 3 2 1 ≤ ≤ ∆ ∆ ∆ ⎯ ⎯ → ⎯ ∆ ∆ ∆ … … (23)
The adapted cepstral coefficients can be calculated by adding a weighted version of the average differences to the Delta coefficients of the clean HMM.

i i m j i m j i m diff clean _ 1 _ 1 _ 1 , , ~≤ ≤ ≤ ≤ ≤ ≤ ∆ ⋅ + ∆ = ∆ β (24)
The adaptation of each cepstral coefficient is done individually for each state and for each mixture component. The value of β is the same as applied in equation 18 for the energy.

Figure 6 summarizes and visualizes this new technique for calculating the differences between the Delta coefficients that can be derived from the static parameters of the clean and the adapted HMMs. It is not only applicable in case of adapting HMMs to the conditions of a hands-free speech input. This method can be applied in any case of adapting the static parameters of HMMs to the influence of distortion effects.

The Delta-Delta parameters can be adapted in the same way as the Delta parameters. The Delta-Delta parameters are calculated from the Delta parameters in the same way the Delta parameters are determined from the static parameters. Looking at equation 15 the only difference is a value of 2 for the higher summation index. This results in a calculation of the Delta-Delta parameters over 5 sets of Delta parameters. Otherwise the adaptation of the Delta-Delta parameters is done as described by equations 16 to 24.

Restrictions in case of a connected word recognition

The complete adaptation scheme as described above works well for whole word HMMs in case of an isolated word recognition. For the recognition of connected words the adaptation will not be perfect when uttering a sequence of words without even short pauses between the words. The beginning of a word is modified by the reverberation tail of the acoustic excitation at the ending of the preceding word. These effects can be taken into account only in a type of online adaptation. To get knowledge about the preceding word, the log likelihood is observed at the final states of all HMMs during the frame-wise recognition with the Viterbi algorithm. The adaptation of the first frames of all HMMs can be done when a high log likelihood is computed for the final states of a HMM so that is very likely that the corresponding word was spoken. Thus the energy and spectral parameters, that are contained in the final states of the HMM with the high likelihood, can be used to adapt the first states of all HMMs. The implementation of this online technique is quite complex because it is based on a frame-wise decision process whether and which HMM creates a high likelihood at its final states. The authors implemented this approach but could find only small improvements for a connected digit recognition with respect to word accuracy. Also because of the high computational effort the approach was not investigated further.

Adaptation of triphone HMMs

Most often triphone HMMs are used in case of a phoneme based recognition which is used for the recognition of large vocabularies. A triphone HMM is applied to model the acoustic characteristics of a phoneme in the context of a specified preceding and a specified succeeding phoneme. Thinking about the adaptation of this triphone HMM, the knowledge about the preceding phoneme and the succeeding phoneme can be taken to apply the adaptation approach as it was presented for the application to whole word HMMs in the previous sections. Looking at a single triphone HMM as it is done for the triphone "s-Ik" in figure 7, the number of possible preceding and succeeding triphone HMMs is restricted. The triphone HMM "s-I-k" for the vowel "i" in the context of a preceding "s" and a succeeding "k" will be preceded by a triphone HMM "unknown-s-I" for the fricative "s" with the vowel "i" following. Only the preceding phoneme of "s" is not defined. In the same way a triphone HMM "I-k-unknown" for the "k" will succeed the triphone HMM for the "i". This knowledge about the sequence of HMMs and hence also about the complete sequence of states enables the applicability of the new approach for the adaptation to reverberation. Looking at the sequence of 3 triphones as shown in figure 7, the complete sequence of 9 HMM states for all 3 models is used for the adaptation. It is done in the same way as it has been described for the whole-word HMMs before.

Thus, the problem of being unable to adapt the first states of a whole word HMM, does not exist in case of triphone models where a certain knowledge about the preceding model is available. This preceding model "unknown-s-I" is chosen from all available triphone HMMs by looking at the spectral similarity between the last state of each triphone HMM "unknown-s-I" and the first state of the HMM "s-I-k". The spectral similarity is estimated by calculating the Euclidean distance between the average cepstral coefficients of the first state of "s-I-k" and the corresponding coefficients of the last state of the preceding phoneme. The triphone HMM with the smallest spectral distance is selected. In the same way the preceding triphone is chosen by comparing the last state of "s-I-k" with the first state of all triphones "Ik-unknown".

{ } ( ) { } ( ) { } [ ] ∑ = ∈ - - - - - cep NR m m last m phonemes all unknown unknown k I S C k I s S C MIN _ 1 2 1 " " " " (27)
The static and the Delta parameters are adapted for the states of the HMM "s-I-k" by looking at the whole state sequence of all 3 consecutive triphone HMMs and applying the adaptation technique as described for the whole-word HMMs. By transforming back the 9 sets of cepstral coefficients to the spectral domain and applying a Spline interpolation the spectrogram for the complete segment of the 3 triphones is available. The knowledge of the preceding phoneme has the advantage that the energy and spectral information of these states can be used to adapt the succeeding states of the HMM "s-I-k" with respect to reverberation. Hence, the drawback of not knowing the preceding HMM for the adaptation of the first states does not exist as it occurs in case of whole-word HMMs. The knowledge of the succeeding triphone is only used for the better estimation of the adapted Delta parameters.

This adaptation technique can be applied to all triphone HMMs that are used for the recognition. In case of using state tying a further tying can be applied after the adaptation. This has not been investigated here. The authors only intended to demonstrate the principal applicability of the new adaptation method to the modeling with triphone HMMs.

Recognition experiments on hands-free speech input

Recognition experiments have been run to proof the applicability of the new adaptation approach and to quantify the improvements that can be achieved.

Some details about the applied feature extraction and the recognition are presented at the beginning of this chapter. After this the results are shown for a series of experiments with the intention to demonstrate the applicability to a word recognition based on whole word HMMs. Finally the improvements are presented separately for the recognition of a large vocabulary based on the use of triphone HMMs.

Feature Extraction

The acoustic features are extracted from the speech signal by a cepstral analysis scheme that is similar to many realizations in this field. A pre-emphasis is applied to the speech signal by means of a first order FIR filter where the value of the preceding sample is weighted by a factor of 0.95 before subtracting it from the value of each sample. Short segments of speech are extracted with a 25 ms Hamming window. The window is shifted by 10 ms which corresponds to a frame rate of 100 Hz. Each speech frame is transformed to the spectral domain by means of a 256 point DFT (Discrete Fourier Transform). The so called MEL spectrum is estimated by weighting the values of the DFT magnitude spectrum with triangular shaped functions and summing up the spectral magnitudes for each triangular. Thus, a MEL spectrum is computed for 24 nonlinearly distributed frequency bands in the range from 200 Hz up to 4000 Hz. The 24 logarithmic MEL spectral values are transformed to the cepstral domain by means of a DCT (Discrete Cosine Transform). 13 cepstral coefficients C0 to C12 are calculated. Thus, C0 is available as acoustic parameter in each state of all HMMs. C0 is only needed to transform back the cepstral coefficients to the spectral domain as part of the adaptation process. But C0 is not used for the recognition. Instead of C0 an energy parameter is estimated from the preemphasised and Hamming weighted speech samples. A preemphasis factor of -1 is applied here which leads to a slightly higher attenuation of the low frequency components. This is of advantage in the presence of background noise with its main energy at low frequencies. The short-term energy is calculated by summing up the squared values of all samples in each 25ms frame.

The described analysis technique is applied to speech data sampled at 8 kHz. In case of data sampled at 16 kHz the same MEL filterbank is applied in the frequency range up to 4000 Hz. All filter characteristics like e.g. the preemphasis filtering or the MEL filters and all other individual settings like e.g. the frame length or the FFT length are chosen in an appropriate way so that the final cepstral coefficients are almost identical to the coefficients which result from an analysis of the same utterance sampled at 8 kHz. This approach has been investigated in earlier work (Hirsch, 2001b). It allows for example the recognition of speech data sampled at 8 kHz with HMMs that have been trained on data sampled at 16 kHz.

Besides the 13 cepstral coefficients and the energy parameter in the range up to 4 kHz two further parameters are calculated in case of data sampled at 16 kHz. These are 2 energy parameters that describe the energy in the frequency range from 4 to 5.5 kHz respectively in the range from 5.5 to 8 kHz. This is realized by summing up the corresponding components of the FFT power density spectrum. These additional coefficients can help to increase the recognition performance a bit in comparison to the case of recognizing data sampled at 8 kHz.

12 cepstral coefficients C1 to C12 and the logarithm of the energy parameters are used as acoustic parameters for the recognition. Furthermore Delta and Delta-Delta coefficients are added as additional features where the Delta parameters are calculated as described by equation 15. The Delta calculation corresponds to the way of estimating Delta parameters in the HTK software package [START_REF] Young | The HTK book (version 3.3)[END_REF].

Thus, finally a feature vector consists of 39 components in case of speech data sampled at 8 kHz and it consists of 45 components in case of data sampled at 16 kHz.

The parameters of the HMMs are determined by applying the available training tools of the HTK software package. The recognition is done either with an own C implementation of a Viterbi recognizer or with the corresponding tool of the HTK package. It has been verified that the own implementation leads to the same recognition results as the HTK recognizer. Furthermore the Viterbi recognizer has also been implemented as Matlab module. This was helpful during the development process of the adaptation algorithms due to the easier software development with Matlab and its graphical visualization properties.

The adaptation techniques have been implemented as Matlab and as C modules. The adaptation is individually applied to each speech utterance when detecting the beginning of speech. The applied voice activity detector takes the MEL magnitude spectrum as input. It will be described a bit more in detail later. The C modules for the analysis and the recognition are designed for an application in a real-time recognition and dialogue system [START_REF] Hirsch | HMM adaptation for telephone applications[END_REF]. This means that the Viterbi match can be started and run in parallel to the feature extraction after the detection of speech. Techniques like a cepstral mean normalization on the whole utterance are not considered here because they would delay the beginning of the Viterbi match till the detection of the end of speech.

Recognition with whole-word HMMs

The TIDigits data base is taken as basis for the experiments on isolated and connected word recognition. A version of the TIDigits is used that has been downsampled at 8 kHz. All utterances from the adult speakers designated for training are taken to determine 2 gender dependent HMMs for each word. Each HMM consists of 16 states where each state is described by the mixture of 2 Gaussian distributions for each of the 39 acoustic features. A single state HMM with a mixture of 8 Gaussians is used for modeling the pauses. The HMMs are defined as left-to-right models without skips over states. The recognizer is set up to recognize any sequence of digits with the restriction that a sequence contains only models from the same gender.

Recognition of single digits

A first series of experiments focused on these test utterances that contain only a single digit. These are about 2500 utterances in total. This is done to avoid the inter-word effects between fluently spoken words without pauses between the words. As already mentioned before, the reverberation will influence the beginning of a word by the ending of the preceding word.

The word error rates are shown in figure 8 where the recognition of connected words is still enabled so that also insertion errors can occur. Results are presented for 3 different conditions. Besides the clean data the TIDigits have been processed with a tool for simulating the influence of a hands-free speech input [START_REF] Hirsch | The simulation of realistic acoustic input scenarios for speech recognition systems[END_REF]. This tool creates fairly natural sounding reverberation. A Web interface exists to experience this tool [START_REF] Finster | Web interface to experience the simulation of acoustic scenarios[END_REF]. Test data have been created for the simulation of two different rooms, an office room with a reverberation time of about 0.4 s and a living room with a reverberation time of about 0.6 s. As expected the error rates are higher for the longer reverberation time.

For each condition four different processing methods are compared. The first one is based on the advanced front-end as it has been standardized by ETSI (ETSI, 2003). Robust acoustic features are extracted with this front-end. The term robustness refers to the presence of background noise and unknown frequency characteristics. This is realized by extending a cepstral analysis scheme by two further processing steps. The first one contains a Wiener filtering based on an estimation of the noise spectrum to reduce the influence of stationary background noise. A blind estimation and equalization of unknown frequency characteristics has been integrated as second processing block. Each feature vector consists of 39 parameters. These are 12 Mel frequency cepstral coefficients and an energy parameter as well as the corresponding Delta and Delta-Delta parameters. Feature vectors are computed at a rate of 100 Hz. This front-end is considered as a representative for a robust feature extraction and is taken as reference for comparing the results with the HMM adaptation. The cepstral analysis scheme as it has been described in the previous section is investigated as second method. Word error rates are presented for the 3 cases where the recognition is done without any adaptation or with adaptation of the static parameters only or with adaptation of the static and the Delta and Delta-Delta parameters.

The error rates for the clean data are in the range of 0.4 to 0.5 %. It can be seen that the influence of the reverberation leads to a considerable deterioration of the recognition rates for both feature extraction schemes. The high error rate of the cepstral analysis scheme in comparison to the ETSI front-end in case of the living room condition is due to a high number of insertion errors. The number of substitutions is even lower for the cepstral analysis scheme in comparison to the ETSI scheme.

Error rates can be reduced by applying the adaptation methods. Adapting also the Delta and Delta-Delta parameters leads to an additional gain in both reverberant situations.

The efficiency of the adaptation scheme is investigated by comparing the obtained results to the case of training the HMMs on reverberated data. Therefore a set of HMMs is trained with all TIDigits training utterances after applying the simulation of a reverberation in the living room.

Results are listed in the first two lines of table 1 for the cases without adaptation and with adaptation of the static and Delta parameters and taking HMMs trained on clean data only. These are the error rates as already shown in figure 8. The third line of table 1 contains the error rates for the case of applying the HMMs trained on reverberated data and without any adaptation. The error rate decreases from about 7% to 1,7% for the living room condition when moving from the training on clean data and applying no adaptation to the training on reverberant data. Applying the adaptation scheme to the clean HMMs leads to an error rate quite close to the case of training on reverberated data. This can be taken as further proof for the usefulness of the applied adaptation method.

The drawback of training the HMMs on reverberant data is a considerable increase of the error rate to about 11 % for the recognition of clean data. This indicates that the training has to be done on a mixture of conditions for a practical application. And this can only be done if the whole range of conditions is known in advance.

Connected word recognition

The word error rates are shown in figure 9 for the recognition of all TIDigits utterances that have been designated for recognition. These are 8700 utterances containing about 28000 digits in total. The results are presented for the 4 conditions that have also been shown in the previous figure .  First of all the error rates for the robust ETSI front-end are higher in comparison to applying the cepstral analysis scheme without any adaptation. For the condition of a hands-free speech input in reverberant environments it looks like the ETSI frontend does not work as efficient as it does in the presence of background noise.

The application of the adaptation method leads also to a reduction of the error rates for the case of recognizing sequences of connected words. The relative improvement is not as high as in case of recognizing single digits. The authors mainly regard the superposition effect at the beginning of words as responsible for this. The acoustic information at the beginning of a word is modified by the acoustic information of the preceding word due to the reverberation. These "inter-word" modifications occur especially when sequences of words are spoken fluently with coarticulation effects. In general the speaking rate considerably varies between speakers when uttering a sequence of digits. This effect can only be approximately covered by modeling with HMMs with multiple mixture components. Analyzing the errors a bit more in detail, it turns out that about half of the errors are due to deletions in case of recognizing the living room data with adaptation. It seems to be difficult to recognize especially the "fast" speakers which create these co-articulation effects. The "inter-word" modifications are not compensated by this adaptation technique.

We observe again that the additional adaptation of the Delta and Delta-Delta parameters causes a further gain in recognition performance.

Further recognition results are presented in figure 10 for varying the reverberation time in the living room condition. The tool for simulating the hands-free speech input in noisy environments allows the variation of the reverberation time in a certain range. The RIR for the living room simulation is modified inside the tool so that it reflects the desired reverberation time. As expected the error rate increases for higher reverberation time. The error rates for a reverberation time of 0.6 s are the ones shown in the previous figure. Again the recognition performance is slightly worse for applying the robust ETSI front-end in comparison to the cepstral analysis scheme. The improvement due to the adaptation is visible over the whole range of the reverberation time.

Recognition with triphone HMMs

Triphone HMMs are used for the recognition of a large vocabulary. The "Wall Street Journal" data base (WSJ0) [START_REF] Ldc | Speech Data Base CSR-I (WSJ0)[END_REF] is taken as basis for these investigations as it has also been used for the evaluations inside the ETSI working group Aurora [START_REF] Picone | Performance analysis of the Aurora large vocabulary experiment[END_REF]. Approximately 7200 utterances that have been recorded at a sampling rate of 16 kHz with a high quality microphone are taken for the training of triphone models. The triphones are modeled as HMMs with 3 states where each acoustic parameter of each state is described by a mixture of 4 Gaussian distributions. Training and recognition are done with HTK as it has been defined in [START_REF] Au Yeung | Improved performance of Aurora-4 using HTK and unsupervised MLLR adaptation[END_REF]. The training procedure includes state tying to model the triphones with a of about 3200 different states. The recognition is based on the usage of a dictionary containing the phoneme description of about 5000 words. The possible sequences of words are defined by a "bigram" model. The recognition process is speeded up by a state based pruning.

The cepstral analysis scheme is applied as it has been described before for data sampled at 16 kHz. We achieve a word error rate of 11.21% for the recognition of the 330 clean utterances that have been designated for testing. The word error rate can be reduced by applying HMMs that model acoustic parameters with a higher number of distributions. Because of the high computational costs, only experiments have been run with HMMs modeling with a mixture of 4 distributions.

All 330 test utterances have been processed with the simulation tool to investigate the recording in an office room with a reverberation time of about 0,4 s. The word error rates for recognizing these data are listed in table 2 The word error rate increases considerably to a value of 48.8 % in the hands-free mode.

The adaptation of the triphone HMMs is applied as described before. As the own implementation of the Viterbi recognizer does not support the use of complex language models, HTK is employed for the recognition. Thus, the estimation of the reverberation time T60 as well as the individual adaptation for each utterance is not applicable. The whole set of triphone HMMs is adapted once at the beginning with a fixed value for the estimated reverberation time T60 instead. The adaptation is implemented as Matlab functions. The intention of the authors is only the proof that the new adaptation approach can be applied to triphone HMMs in principal. The word error rate decreases by about 10 % when applying the set of adapted triphone HMMs.

Combined adaptation to different distortion effects

The hands-free speech input in a room comes along with the recording of background noise as it is present in almost all applications of speech recognition systems. Furthermore the spectrum of the speech is modified by the frequency characteristics of the microphone and of an additional transmission channel, e.g. in case of transmitting the speech via telephone to a remote recognition system. This creates the need to compensate also these distortion effects.

In earlier work (Hirsch, 2001a) the authors developed an adaptation scheme based on the well known PMC (parallel model combination) approach. This scheme consists of an adaptation of the static Mel frequency cepstral coefficients. The cepstral coefficients are transformed back to the Mel spectral domain where the adaptation can be realized by a multiplication with a frequency weighting function as estimate for the frequency characteristics and by adding the estimated noise spectrum. The cepstral coefficients of all HMMs are individually adapted for each speech utterance when the beginning of speech is detected. Furthermore the energy parameter can be adapted with an estimate of the noise energy.

We present a short overview about the techniques for estimating the spectrum of the background noise and the frequency weighting function in the next section. Having these estimates as well as an estimation of T60, it will be shown that the earlier adaptation approach (Hirsch, 2001a) can be combined with the new method of adapting the spectra to a hands-free speech input.

Estimation of distortion parameters

The spectrum of the background noise is estimated by looking at a smoothed version of the Mel magnitude spectrum as it is calculated in the feature extraction. The contour of the spectral magnitude values is smoothed in each Mel subband by applying a first order recursive filtering.

( ) ( ) ( ) ( ) … , 20 , 10 , 0 _ 1 1 1 ms ms t and mel NR k for t Xsmooth t Xi t Xsmooth i i k i k i k = ≤ ≤ ⋅ + ⋅ - = - α α where ( ) i k t Xi
is the Mel spectrum of the analysis frame at time i t as calculated in the feature extraction.

(28)

A VAD (voice activity detector) is applied that takes the Mel spectra as input. A speech onset is detected when the estimated signal-to-noise ratios exceed an adaptive threshold in several subbands for a certain umber of frames. The VAD was developed for earlier investigations. More details can be found in [START_REF] Hirsch | Noise estimation techniques for robust speech recognition[END_REF]Hirsch, 2001a).

When the beginning of speech is detected the estimated noise spectrum is set to the smoothed spectrum of the last analysis frame that is marked as pause frame.

( )

mel NR k for frame pause last Xsmooth N k k _ 1 ≤ ≤ = (29)
Furthermore the energy of the noise is estimated as energy of the last pause frame.

(

)

frame pause last Ei Enoise = , ( 30
)
where ( )

i t Ei
is the energy of the analysis frame at time i t as calculated in the feature extraction.

The detection of speech begin is also taken as trigger point to perform the adaptation of all HMMs. For the simulation experiments we take the acoustic parameters of all frames from a recorded utterance as input for the Viterbi recognition. For the real-time version of the recognizer as it is applied in a speech dialogue system, we start the recognition process 5 frames earlier than the first frame detected as speech. Thus, the Viterbi calculation can be run almost in parallel with the feature extraction.

The frequency weighting function is estimated after the recognition of an utterance. It is applied for the recognition of the next utterance. This is based on the assumption that the frequency characteristics of the whole speech transmission will not change rapidly. Usually the microphone and the other transmission conditions do not change during a recognition session. The weighting function is estimated by comparing the long-term spectra of the noisy input speech and of the clean speech. The "best" sequence of HMM states is considered as it is available after the Viterbi match by backtracking the path with the highest likelihood. The long-term spectrum of the noisy input speech is calculated for all analysis frames that are mapped on speech HMMs excluding the frames that are mapped on the pause model.

( ) { }

∈ ⋅ = ∑ _ 1 (31)
where NR_speech is the total number of vectors mapped on speech HMMs.

In a similar way the long-term spectrum of the clean speech is estimated by looking at the spectral information contained in the HMM states at the path with highest likelihood. A set of adapted HMMs is used for the recognition. But for the estimation of the clean spectrum the spectral information is extracted from the corresponding clean HMMs. The cepstral coefficients of the corresponding clean HMM states are transformed back to the Mel spectral domain. In case of HMMs with multiple mixture components, the spectrum of this mixture component with the smallest spectral distance to the corresponding spectrum of the input signal is taken. Therefore, the estimated noise spectrum is subtracted from the spectrum of the input signal to compare it with the spectrum contained in a clean HMM. The spectral similarity is calculated as City block distance. So, the long-term spectrum of the clean speech can be estimated. sil N is the Mel spectrum that can be derived from the single state pause model. Calculating the average cepstral values of the pause state according to equation 12, the spectrum can be determined by transforming the cepstral values to the spectral domain (equation 13). The pause model contains the spectral information of the background noise that was present during the recording of the training data. In case of "clean" training data, sil N takes only small values. It is subtracted here to compensate its presence in the spectral parameters of all HMMs. In the rare case of getting a negative value after the subtraction the result is set to a fixed small positive value.

Subtracting the estimated noise spectrum from the long-term spectrum of the noisy input speech, the frequency weighting function can be estimated.

mel NR k for Slong N Xlong W k k k k _ 1 ≤ ≤ - = (33)
It turned out in earlier investigations that this type of estimating the spectral difference between the input signal and the clean HMMs works well. Because of comparing the spectral information from the input signal and the clean HMMs, the weighting function does not only contain the spectral characteristics of the recording equipment and the transmission line but also the frequency characteristics of the individual speaker to some extent.

In the same way the difference between the energy contours of the input speech and the best HMM sequence can be calculated. The energy values of the input signal are accumulated for those frames mapped on speech HMMs.

i i ∈ - ⋅ = ∑ , , _ 1 (35)
The average energy sil E of the single state pause model is subtracted to compensate the presence of background noise in the training data.

A weighting factor can be calculated that describes the average energy difference between the input signal and the energies contained in the sequence of HMM states on the best path.

Eclean Enoise

Einput we - = (36)
This factor contains information about the loudness of the individual speaker.

Combined Adaptation

Having estimates for the noise spectrum, the frequency weighting function and the reverberation time, the Mel spectra of the clean HMMs are adapted as shown in figure 11.

The cepstral coefficients of each state and mixture component are transformed back to the linear Mel spectrum for all clean HMMs. The Mel spectra are adapted to the estimated reverberation condition as described by equation 14. The estimated weighting function and the estimated noise spectrum are applied for the further adaptation.

( ) ( ) mel NR k for N mix S X W mix S X k j i k k j i k _ 1 , , ˆ≤ ≤ + ⋅ = (37)
The adapted Mel spectra are transformed to the cepstral domain again. In the same way the energy parameter that has been adapted to the reverberation as stated in equation 10, is used as input for the further adaptation. The adaptation to reverberation and noise is visualized by the spectrograms in figure 12. The spectrogram is shown in graph a) as it can be calculated from the HMM of the word six trained on clean data. In graph b) the adapted version of this HMM is visualized. The adapted HMM has been extracted during the recognition of artificially distorted TIDigits data. These data have been created from a simulation of the hands-free recording in a noisy living room environment. The noise spectrum as it is estimated for the individual input utterance, becomes visible as shift of the complete spectrogram. The reverberation tails can also be seen when looking at the contours along time in individual subbands.

) ( ) Enoise mix S E we mix S E j i j i k + ⋅ = , , ˆ (38)

Recognition experiments on hands-free speech input in noisy environments

Again the recognition of connected digits is employed to proof the applicability of the combined adaptation to several distortion effects. A data base called Aurora-2 [START_REF] Hirsch | The Aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions[END_REF] exists that consists of noisy versions of the TIDigits. Noise signals have been artificially added at different SNRs. Furthermore a few test sets exist where the frequency characteristics have been modified to simulate the recording with audio devices in the telecommunication area. But Aurora-2 does not cover the effect of a hands-free speech input in noisy environments.

Thus new sets of distorted versions have been artificially created from the clean TIDigits by applying the already mentioned simulation tool [START_REF] Finster | Web interface to experience the simulation of acoustic scenarios[END_REF]. To make these data available for the research community, they have been put together as data base and have been combined with HTK based recognition experiments. They will become available under the title Aurora-5 (Aurora, 2006).

A few details about the new data base will be listed in the next section before presenting the recognition results for these data. Finally results will be shown for the recognition of data that have been recorded in a reverberant meeting room in hands-free mode.

Distorted data of the Aurora-5 experiment

The new data base focuses on 2 application scenarios for speech recognition. The first one is the application inside the noisy interior of a car, the second one the hands-free speech input in an office or a living room, to control e.g. electronic devices like a telephone or audio/video equipment. In comparison to where only 1000 utterances were selected, each test set contains all 8700 utterances here.

The usage of telephone like devices is assumed in general for the car scenario by filtering all data with a G.712 frequency characteristic first [START_REF] Campos-Neto | The ITU-T software library[END_REF]. G.712 is a characteristic that attenuates all frequency components outside the range from about 300 to 3400 Hz. Car noise is added to the filtered data at different SNRs in the range from 0 to 15 dB. The noise segment for distorting a single utterance is randomly selected out of 8 recordings that were made in different cars and under different conditions like e.g. windows open or closed. Three different versions exist for the car noise condition. The first version contains additive noise only according to the recording with a close talking microphone. The second one considers the recording with a hands-free microphone. The third version is like the second one but containing a further transmission over a GSM cellular telephone network. This reflects the usage of an information retrieval system located at a remote position in the telephone network. The GSM transmission is simulated by randomly selecting an AMR (adaptive multi-rate) speech coding mode and the channel conditions of the cellular channel. These options exist as part of the simulation tool. In total 15 test sets exist for the 5 different SNR conditions including the clean case and the 3 versions.

For the second scenario randomly selected noise segments are added from 5 recordings inside different rooms like e.g. an office room or a restaurant. The same range of SNRs is considered. Three different versions exist. The first one looks at additive noise only simulating the recording with a close talking microphone. The second one considers the recording in an office room where the reverberation time is randomly varied in the range from 0.3 to 0.4 seconds. In the third version the recording in a living room is simulated where the reverberation time randomly varies in the range from 0.5 to 0.6 seconds. This comes up again to 15 test sets in total.

In general the Aurora-5 data contain a bigger variance of the distortion conditions inside each test set in comparison to Aurora-2. For example only a single noise recording has been taken for Aurora-2 to create one test set.

Recognition of artificially distorted digits

Cepstral parameters are extracted again as acoustic features, as described and applied before for the experiments with reverberation as the single distortion effect. Also the same gender dependent HMMs are used that have been created with a training on the clean TIDigits. The word error rates are presented in figure 13 for the three different versions containing car noise.

Looking at the condition with additive noise only, shown in graph a), the expected improvement can be seen when comparing the results for the robust ETSI front-end against the results for a cepstral analysis. Further small improvements are achieved when adapting the HMMs to all distortion effects as described in the previous chapter. Furthermore the error rates are shown for the unsupervised HMM adaptation with MLLR as it is available as part of the HTK Viterbi recognizer. An incremental MLLR is performed after each utterance. We observed a worse recognition performance when applying MLLR every 2 or more utterances. The adaptation is performed on the HMMs containing the features of the cepstral analysis so that the results can be immediately compared to the new adaptation approach. The error rates for MLLR are only a little bit worse in this case.

The improvement, comparing the new adaptation approach against the ETSI front-end, is higher when looking at the condition of a hands-free speech input in the noisy car environment. This is shown in the graph b). The reverberation time is fairly small in a car in comparison to rooms. The major impact of the hands-free recording inside a car is a modification of the frequency characteristics. The adaptation seems to compensate these effects to a higher extent than the robust feature extraction except for the low SNR of 0 dB. The error rates for MLLR are again slightly worse.

shown in figure 15 for an isolated word recognition using the set of HMMs that have been trained on the clean TIDigits. For the cepstral analysis and the HMMs trained on clean TIDigits the recognition performance is low with an error rate of about 30 %. The error rate can considerably be reduced with the robust ETSI front-end to a rate less than 10 %. Applying the adaptation technique leads to a further relative error rate reduction by about 70 % in comparison to the error rate for the robust front-end.

MLLR adaptation is applied on the HMMs for the case of cepstral analysis, where these results are not shown in the figure. We obtain the highest performance with an error rate of 12 % when performing the adaptation every 5 utterances. An interesting result is achieved when applying the MLLR adaptation on the HMMs trained on the features of the ETSI front-end. We obtain an error rate of 2,3 % when performing the adaptation for each utterance. This is even slightly better than the new adaptation technique. It might indicate that it is possible to combine a robust feature extraction with an additional adaptation. Another experiment is run on some recordings of the meeting recorder project [START_REF] Janin | The ICSI meeting corpus[END_REF]. Speech data have been recorded during meetings in a meeting room where the microphones were placed in the middle of a table. Thus these data contain reverberation besides a low amount of background noise. The speakers uttered also sequences of English digits which are used for this experiment. Only the recordings of native English speakers are used resulting in about 2350 utterances with about 7700 digits in total. Word error rates are shown in figure 16.

In this case the performance of the robust front-end is slightly worse in comparison to a standard cepstral analysis. This effect has already been observed in most of the experiments on reverberant data presented in the preceding sections. Applying the adaptation scheme, the error rate can be reduced by about 70 % in relation to the cepstral analysis without adaptation. We achieve almost the same performance when performing MLLR adaptation for each utterance.

Conclusion

We present a new technique for adapting the acoustic parameters of HMMs to the condition of hands-free speech input in reverberant rooms. This approach can be combined with existing techniques for the adaptation on noise and unknown frequency characteristics. Furthermore we introduce a new method for adapting the Delta parameters based on a preceding adaptation of the static parameters.

Applying the new adaptation approach on artificially distorted data or on real recordings in noisy conditions, we achieve a considerably higher recognition performance in comparison to the case without adaptation. The error rates are also lower than the ones that are achieved with the robust ETSI front-end that can be considered as representative for a robust feature extraction.

Especially in conditions where additive noise and reverberation distort the speech signal, we obtain a higher recognition performance with the new adaptation technique in comparison to MLLR adaptation. The linear regression seems to compensate the nonlinear distortion effects worse than the new approach.

In the future we will investigate whether and how robust feature extraction schemes can be combined with HMM adaptation techniques.

Figure 1 :

 1 Figure 1: Energy contours of a speech signal in clean and reverberant condition

Figure 2 :

 2 Figure 2: Modulation transfer functions for different values of T60

  in the spectral domain by comparing the spectrograms as representations of different HMMs. Spectrograms are shown in a three dimensional visualization mode. The spectrogram of a HMM is estimated by transforming the MFCCs back to the linear spectral domain for all states. a state are taken to model the average duration of this state as defined by equation 3. In figure 4 three spectrograms of different HMM versions are shown for the word "six". Each HMM consists of 16 states where a single state is described by a set of cepstral coefficients including the zeroth cepstral coefficient C0. The spectrum of an individual HMM state is positioned with respect to its point in time i S t in the middle of the segment that is described by this state.

  interpolation is applied to the contour of the magnitude spectral values in each MEL band.

(

Figure 4 :

 4 Figure 4: Different spectrograms of HMMs for the word "six"

Figure 5 :

 5 Figure 5: Estimation of T60 by an iterative search of the maximum likelihood

.

  the clean and for the adapted HMM version at the frame rate of 100 Hz As the number of frames is equal for the clean and the adapted HMM of an individual word class the difference between the clean and the adapted Delta energies can be calculated.

(

Figure 6 :

 6 Figure 6: Scheme for estimating the differences between the Delta parameters of clean and adapted HMMs

Figure 7 :

 7 Figure 7: Possible preceding and succeeding triphone HMMs for one selected triphone HMM

  modeling with a mixture of Gaussian distributions the average cepstral coefficients are calculated by taking into account the mixture weights.

Figure 8 :

 8 Figure 8: Word error rates for these TIDigits utterances that contain only single digits

Figure 9 :

 9 Figure 9: Word error rates for the recognition of all TIDigits

  Figure 10: Word error rates for a variation of the reverberation time in the living room

(

Figure 11 :

 11 Figure 11: Scheme for adapting HMMs to all distortion effects

Figure 12 :

 12 Figure 12: Spectrograms of the clean and the adapted HMMs for the word "six"

Figure 15 :

 15 Figure 15: Word error rates for the Bellcore single digits with HMMs trained on the TIDigits

Figure 16 :

 16 Figure 16: Word error rates for the meeting recorder digits with HMMs trained on TIDigits

Table 1 :

 1 Word error rates for the recognition of single TIDigits applying the cepstral analysis

			Condition
		Clean	Office room	Living room
	without adaptation	0.44 %	3.49 %	6.94 %
	Adaptation to reverberation &	0.44 %	1.57 %	2.05 %
	Deltas
	HMMs trained on living room	10.98 %	1.93 %	1.73 %

  .

	clean	Office without adaptation	Office with fixed adaptation
	11.2 %	48.8 %	39.8 %

Table 2 :

 2 Word error rates for the recognition of the "Wall Street Journal" large vocabulary

  The energy parameters contained in the clean HMMs on the best path are accumulated as estimate for the clean energy. Models, states and mixture components are selected as described before (equation 32).

	Einput	=	NR	_	speech 1	⋅	( ) t frames Ei i ∑ speech	for	t	i	∈	{	feature	vectors	mapped	on	speech	} HMMs	(34)
	Eclean		( ) ( ) ( ) ] HMMs speech on mapped } sil E t mix t S t Mclean [ vectors E frames feature speech { t for i speech NR i

Acknowledgments

The investigations have been partly carried out during a research stay at the International Computer Science Institute in Berkeley, CA 94704, USA. H.G. Hirsch would like to thank Nelson Morgan and the whole speech group for fruitful discussions and their support.

This work is supported by the German research funding organization DFG (Deutsche Forschungsgemeinschaft).

The adaptation scheme shows its usability also for the case of an additional transmission over the GSM cellular network as shown by the results in graph c). In this case the speech is further modified by the encoding and decoding and the transmission errors on the cellular channel. The adaptation technique seems to cover this type of distortion considerably better than the robust ETSI front-end. The performance of MLLR is extremely low for the SNR of 0 dB. This has been observed in several experiments where the performance without adaptation was already quite low. MLLR seems to be unable to find the right feature mapping in such cases and seems to adapt the features in the wrong direction. The cases with car noise do not include the major effects of a hands-free speech input in a reverberant room environment. The word error rates presented in figure 14 do include such effects. These experiments investigate recordings of speech inside a noisy room environment.

In the case of additive noise only, shown in graph a), the new adaptation scheme leads to similar error rates like the robust front-end. In general the recognition performance is lower in comparison to the case with car noise because the interior noise signals contain more non stationary segments.

A considerable improvement is observed when comparing the new adaptation technique against the robust front-end for the cases of a hands-free speech input in an office or a living room as shown in graphs b) and c). The additional adaptation to reverberation causes this improvement. MLLR adaptation leads to worse results especially for SNRs below 15 dB. It looks like the mapping on the basis of a linear regression is not able to completely compensate the sum of spectral modifications caused by background noise and reverberation. While additive noise and a frequency weighting can be modeled as a stationary modification of each frame, reverberation includes modifications along the time axis. In sum this can not be completely compensated with a linear mapping. As already observed for the car noise conditions, MLLR seems to adapt into the wrong direction in case we obtain a low performance without adaptation.

Recognition of digits recorded in application scenarios

All results presented so far have been derived from a recognition of artificially distorted speech data. The authors believe that their simulation of recording conditions represents the situation of applying a recognition system in a real scenario quite well.

Thus the improvements on artificially distorted data should also be visible in real application scenarios. This is proofed by recognizing speech data that have been recorded in different situations.

The first experiment is run on the so called Bellcore digits. These are speech data that have been recorded over telephone lines. There exist the recordings of 220 American speakers that have spoken the 10 English digits as isolated words. The recordings contain the usual distortions that occur in case of transmitting speech over the telephone. This includes the usage of telephone devices with different frequency characteristics and the presence of some background noise. Word error rates are