Roberto Gemello
email: roberto.gemello@loquendo.com

Franco Mana
email: francomana@loquendo.com

Stefano Scanzio
email: stefano.scanzio@polito.it

Pietro Laface
email: pietro.laface@polito.it

Renato De Mori
email: renato.demori@lia.univ-avignon.fr

Linear Hidden Transformations for Adaptation of Hybrid ANN/HMM Models

Keywords: Automatic Speech Recognition, Speaker Adaptation, Neural Network Adaptation, Catastrophic Forgetting

This paper focuses on the adaptation of Automatic Speech Recognition systems using Hybrid models combining Artificial Neural Networks (ANN) with Hidden Markov Models (HMM). Most adaptation techniques for ANNs reported in the literature consist in adding a linear transformation network connected to the input of the ANN. This paper describes the application of linear transformations not only to the input features, but also to the outputs of the internal layers. The motivation is that the outputs of an internal layer represent discriminative features of the input pattern suitable for the classification performed at the output of the ANN.

In order to reduce the effect due to the lack of adaptation samples for some phonetic units we propose a new solution, called Conservative Training. Supervised adaptation experiments with different corpora and for different types of adaptation are described. The results show that the proposed approach always outperforms the use of transformations in the feature space and yields even better results when combined with linear input transformations.

Introduction

The literature on adaptation of speaker, environment, and application is rich of techniques for refining Automatic Speech Recognition (ASR) systems by adapting the acoustic features and the parameters of stochastic models [START_REF] Gauvain | Maximum a posteriori estimation for multivariate gaussian mixture observations of Markov chain[END_REF][START_REF] Gales | Maximum likelihood linear transformations for HMM-based speech recognition[END_REF][START_REF] Kuhn | Rapid speaker adaptation in eigenvoice space[END_REF][START_REF] Sagayama | Analytic methods for acoustic model adaptation: A review[END_REF][START_REF] Lee | On adaptive decision rules and decision parameter adaptation for automatic speech recognition[END_REF]. More recently, particular attention has been paid to discriminative training techniques and their application to the acoustic feature transformation [START_REF] Hsiao | Discriminative feature transformation by guided discriminative training[END_REF][START_REF] Liu | Model complexity control and compression using discriminative growth functions[END_REF]. Since discriminative methods are also used to train the acoustic-phonetic Artificial Neural Networks (ANN) models, it is worth exploring methods for adapting their features and model parameters. Several solutions to this problem have been proposed. Some of these techniques for adapting neural networks are compared in [START_REF] Abrash | Connectionist speaker normalization and adaptation[END_REF][START_REF] Neto | Speaker-adaptation for hybrid HMM-ANN continuous speech recognition system[END_REF]. A classical approach consists in adding a linear transformation network (LIN) that acts as a pre-processor to the main network. Alternatively, it could be possible to simply adapt all the weights of the original network. A tied-posterior approach is proposed in [START_REF] Stadermann | Two-stage speaker adaptation of hybrid tied-posterior acoustic models[END_REF] to combine Hidden Markov Models (HMM) with ANN adaptation strategies. The weights of a hybrid ANN/HMM system are adapted by optimizing the training set cross entropy. A sub-set of the hidden units is selected for this purpose. The adaptation data are propagated through the original ANN. The nodes that exhibit the highest variances are selected, since hidden nodes with a high variance transfer a larger amount of information to the output layer. Then, only the weights of the links coming out of the selected nodes are adapted. Recent adaptation techniques have been proposed with the useful properties of not requiring to store the previously used training data, and to be effective even with a small amount of adaptation data. Methods based on speaker space adaptation [START_REF] Gales | Maximum likelihood linear transformations for HMM-based speech recognition[END_REF] and eigenvoices [START_REF] Kuhn | Rapid speaker adaptation in eigenvoice space[END_REF] are of this type and can be applied both to Gaussian Mixture HMMs as well as to the ANN inputs as proposed in [START_REF] Dupont | Fast speaker adaptation of artificial neural networks for automatic speech recognition[END_REF]. The parameters of the transformations are considered the components of a vector in a parameter adaptation space. The principal components of this space define a speaker space. Rapid adaptation consists in finding the values of the coordinates of a specific speaker point in the speaker space. Another approach is the regularized adaptation proposed in [START_REF] Li | Regularized adaptation of discriminative classifiers[END_REF], where the original weights of the networks, trained with unadapted data, are the a priori knowledge used to control the degree of adaptation, to avoid overfitting on adaptation data. This paper explores a new possibility consisting in adapting ANN models with transformations of an entire set of internal model features. Values for these features are collected at the output of a hidden layer for which the number of outputs is usually of the order of a few hundreds. These features are supposed to represent an internal structure of the input pattern. As for input feature transformation, a linear network can be used for hidden layer feature transformation. In both cases, the estimation of the parameters of the adaptation networks can be done with error Back-Propagation by keeping unchanged the values of the parameters of the ANN.

A problem, however, occurs in distributed connectionist learning when a network, trained with a large set of patterns, has to be adapted to classify input patterns that differ in some aspects from the ones used originally to train the network. A problem called "catastrophic forgetting" [START_REF] French | Catastrophic forgetting in connectionist networks: causes, consequences and solutions[END_REF] arises when a network is adapted with new data that do not adequately represent the knowledge included in the original training data. This causes a particularly severe performance degradation. This happens when adaptation data do not contain examples for a subset of the output classes. A review of several approaches that has been proposed to solve this problem is presented in [START_REF] French | Catastrophic forgetting in connectionist networks: causes, consequences and solutions[END_REF]. One of them uses a set of pseudo-patterns, i.e. random patterns, associated to the output values produced by the connectionist network before adaptation. These pseudo-patterns are added to the set of the new patterns to be learned [START_REF] Robins | Catastrophic forgetting, rehearsal, and pseudo-rehearsal[END_REF]. The attempt is to keep stable the classification boundaries related to classes that have few or no samples in the new set of patterns. This effectively decreases the catastrophic forgetting of the knowledge provided by originally learned patterns. Tests of this solution have been reported with small networks and low dimensional artificial input patterns. Unfortunately, they do not scale well because it is difficult to generate effective pseudo-patterns when the dimensionality of the input features is high. For this reason, it has been proposed [START_REF] Benzeghiba | Hybrid HMM/ANN and GMM combination for usercustomized password speaker verification[END_REF] to include examples of the missing classes, taken from the training set, in the adaptation set. However, the addition of a small subset of training examples related to the missing classes could redefine the class boundaries according to the distribution of these small subsets. This distribution would be different from the one of the complete training set. Moreover, this approach has a main practical problem: it is mandatory to store training set samples for the adaptation step. The number of samples should be large enough to provide a good preservation of the class boundaries. Finally, since the task independent network could be adapted to several applications, different sets of training patterns would be necessary to compensate classes missing in different adaptation sets.

This paper proposes a solution to this problem by introducing Conservative Training, a variation to the standard method of assigning the target values, which compensates for the lack of adaptation samples in some classes. The key idea of Conservative Training is that the probability of classes with no adaptation samples available should be replaced by the best available estimations of their real values. The only way to obtain these estimations is with the model provided by the original network.

Experimental results on the adaptation test for the Wall Street Journal task [START_REF] Pallett | Benchmark tests for the ARPA spoken language program[END_REF], using the proposed approaches, compare favorably with published results on the same task [START_REF] Stadermann | Two-stage speaker adaptation of hybrid tied-posterior acoustic models[END_REF][START_REF] Pallett | Benchmark tests for the ARPA spoken language program[END_REF].

The paper is organized as follows: Section 2 gives a short overview of the acoustic-phonetic models of the ANN used by the ASR system, and presents the Linear Hidden Networks, which transform the features at the output of hidden layers. Section 3 is devoted to the illustration of the problem of catastrophic forgetting in connectionist learning, and proposes our Conservative Training approach as a possible solution. Section 4 illustrates the benefits of Conservative Training using an artificial classification task of 16 classes. Section 5 reports the experiments performed on several databases with the aim of clarifying the behavior of the new adaptation techniques with respect to the classical LIN approach. Finally, the conclusions and future developments are presented in the last Section.

Feature transformations

The LOQUENDO-ASR decoder uses a hybrid combination of Hidden Markov Models (HMM) and a 4-layer Multi Layer Perceptron (MLP), where each phonetic unit is described in terms of a single or double state left-to-right automaton with self-loops. The HMM transition probabilities are uniform and fixed, and the emission probabilities are computed by a MLP [START_REF] Albesano | Hybrid HMM-NN modeling of stationary-transitional units for continuous speech recognition[END_REF]. The MLP has an input layer of 273 units (39 parameters of a 7 frame context), a first hidden layer of 315 units, a second hidden layer of 300 units and an output layer including a variable number of units, which is language dependent (600 to 1000). The advantage of using two hidden layers, rather than a larger single hidden layer, is that the total number of connections is reduced. Moreover, this architecture allows to consider the activation values of each hidden layer as a progressively refined projection of the input pattern in a space of features more suitable for classification. The acoustic models are based on a set of vocabulary and gender independent units, including stationary context-independent phones and diphone-transition coarticulation models [START_REF] Albesano | Hybrid HMM-NN modeling of stationary-transitional units for continuous speech recognition[END_REF]. The models have been trained using large 8 kHz telephone speech databases, e.g. the SpeechDat corpora. The training set includes mainly phonetically balanced sentences, and some samples of application lists (e.g. digits, currency, yes-no). The weights of the ANN/HMM system are adapted by optimizing the training set cross entropy. These models are the acoustic models of the 15 languages available with the LOQUENDO-ASR recognizer. They are used as seed models for the adaptation experiments of Section 5, unless differently specified.

Input feature transformations

The simplest and more popular approach to speaker adaptation with ANNs is Linear Input Transformation (LIN) [START_REF] Abrash | Connectionist speaker normalization and adaptation[END_REF][START_REF] Neto | Speaker-adaptation for hybrid HMM-ANN continuous speech recognition system[END_REF]. A linear transformation rotates the input space to reduce the discrepancy between target and training conditions. A LIN, as shown in Figure 1, performs this transformation. The LIN weights are initialized with an identity matrix, and they are trained by minimizing the error at the output of the ANN system keeping fixed the weights of the original ANN. Using few training data, the performance of the combined architecture LIN/ANN is usually better than the one obtained by adapting the weights of the whole network, because the adaptation involves a lower number of parameters.

Input Layer

In a layered neural network, the activation values of each hidden layer are a progressively refined projection of the input pattern in a space of features more suitable for classification. The weights between the last hidden layer and the output layer perform a linear discrimination of the output classes. For this purpose, the weights of the lower layers of the network are trained to produce activations of the last hidden units that are linearly separable. These activations can be considered as new features obtained by a non-linear discriminant analysis. Because of their properties, they are used -properly decorrelated -as observation features for Continuous Densities Gaussian Mixture models in the TANDEM approach [START_REF] Hermansky | Tandem connectionist feature extraction for conventional HMM systems[END_REF]. Since the activation values of a hidden layer represent an internal structure of the input pattern in a space more suitable for classification, it is worth considering the adaptation of these features. A Linear Hidden Network (LHN) performs such an adaptation. Exactly as in the LIN case, the values of an identity matrix initialize the weights of the LHN. The weights are estimated using a standard Back-Propagation algorithm keeping frozen the weights of the original network. It is worth noting that, since the LHN performs a linear transformation, when the adaptation process is completed, the LHN can be removed. This can be done combining the LHN weights with the ones leading to the nodes of the next layer using the following simple matrix operations:

SI LHN SI a SI LHN a W B B B W W W × + = × = (1)
where W a and B a are the weights and the biases of the adapted layer, W SI and B SI are the weights and biases of the layer above the LHN in the original Speaker Independent network, and W LHN and B LHN are the adapted weights and the biases of the linear hidden network.

In our experiments the LHN has been applied to the last hidden layer, but since the outputs of an internal layer can be considered as features more discriminative than the original ones, the LHN can be applied to whatever internal layer.

Catastrophic Forgetting

It is well known that in connectionist learning, acquiring new information in the adaptation process may cause a partial or total oblivion of the previously learned information [START_REF] French | Catastrophic forgetting in connectionist networks: causes, consequences and solutions[END_REF][START_REF] Robins | Catastrophic forgetting, rehearsal, and pseudo-rehearsal[END_REF]. This effect must be taken into account when adapting an ANN with a limited amount of data, i.e. when the probability of the absence of samples for some acoustic-phonetic units is high. The problem is more severe in the ANN modeling framework than in the classical Gaussian Mixture HMMs. The reason is that an ANN uses discriminative training to estimate the posterior probability of each acoustic-phonetic unit. The minimization of the output error is performed by means of the Back-Propagation algorithm that penalizes the units with no observations in the adaptation set by setting to zero the target value of the their output units for every adaptation frame. This target assignment policy reduces the ANN capability of correctly classifying the corresponding acoustic-phonetic units. On the contrary, the Gaussian Mixture models with little or no observations remain un-adapted, or share some adaptation transformations of their parameters with other similar acoustic models, maintaining the knowledge acquired before adaptation.

To mitigate the just introduced oblivion problem, it has been proposed [START_REF] Benzeghiba | Hybrid HMM/ANN and GMM combination for usercustomized password speaker verification[END_REF] to include in the adaptation set examples of the missing classes taken from the training set. The disadvantage of this approach is that a substantial amount of the training set must be stored in order to have enough examples of the missing classes for each adaptation task. In [START_REF] Robins | Catastrophic forgetting, rehearsal, and pseudo-rehearsal[END_REF], it has been proposed to approximate the real patterns with pseudo-patterns rather than using the training set. A pseudo-pattern is a pair of a random input activation and its corresponding output. These pseudo-patterns are included in the set of the new patterns to be learned to prevent catastrophic forgetting of the original patterns. The proposed solutions have problems when applied to the adaptation of large ANNs. In fact, there are no criteria for selecting adaptation samples from the training data which are often not available when adaptation is performed. Moreover, the selected data should share some characteristics that make the adaptation environment different from the training one, but the elements of such a difference are often unknown. Furthermore, it is unclear how effective pseudo-patterns can be generated when the dimensionality of the input features is high.

A solution, called Conservative Training (CT), is now proposed to mitigate the forgetting problem. Since the Back-Propagation technique used for MLP training is discriminative, the units for which no observations are available in the adaptation set will have zero as a target value for all the adaptation samples. Thus, during adaptation, the weights of the MLP will be biased to favor the output activations of the units with samples in the adaptation set and to weaken the other units, which will always have a posterior probability getting closer to zero. Conservative Training does not set to zero the value of the targets of the missing units; it uses instead the outputs computed by the original network as target values. Regularization as proposed in [START_REF] Li | Regularized adaptation of discriminative classifiers[END_REF] is another solution to the forgetting problem. Regularization has theoretical justifications and affects all the ANN outputs by constraining the network weight variations. Unfortunately, regularization does not directly address the problem of classes that do not appear in the adaptation set. We tested the regularization approach in a preliminary set of experiments, obtaining minor improvements. Furthermore, we found difficult to tune a single regularization parameter that could perform the adaptation avoiding catastrophic forgetting. Conservative Training, on the contrary, takes explicitly all the output units into account, by providing target values that are estimated by the original ANN model using samples of units available in the adaptation set.

Let F p be the set of phonetic units included in the adaptation set (p indicates presence), and let F m be the set of the missing units. In Conservative Training the target values are assigned as follows: O . Thus, a phonetic unit that is missing in the adaptation set will keep the value that it would have had with the original un-adapted network, rather than obtaining a zero target value for each input pattern. This policy, like many other target assignment policies, is not optimal. Nevertheless, it has the advantage of being applicable in practice to large and very large vocabulary ASR systems using information from the adaptation environment, and avoiding the destruction of the class boundaries of missing classes. It is worth noting that in badly mismatched training and adaptation conditions, for example in some environmental adaptation tasks, acoustically mismatched adaptation samples may produce unpredictable activations in the target network. This is a real problem for all adaptation approaches: if the adaptation data are scarce and they have largely different characteristics -SNR, channel, speaker age, etc. -other normalization techniques have to be used for transforming the input patterns to a domain similar to the original acoustic space. Although different strategies of target assignment can be devised, the experiments reported in the next sections have been performed using only this approach. Possible variations, within the same framework, include the fuzzy definition of missing classes and the interpolation of the original network output with the standard 0/1 targets.

0 . 0 )) | ( ! & | ( ) | ( _ _ 0 . 1 ( )) | ( & | ( ) | ( _ _ ) | ( = ∈ - = ∈ = ∈ ∈ t i t p i F j t j t i t p i t i t m i O f correct O F f T O f NN ORIGINAL OUTPUT O f correct O F f T O f NN ORIGINAL OUTPUT O F f T m (2)

Experimental results on artificial data

An artificial two-dimensional classification task has been used to investigate the effectiveness of the Conservative Training technique. The examples have been designed to illustrate the basic dynamics of the class boundaries. They reproduce the problems due to missing classes in the adaptation set, emphasizing them.

An MLP has been used to classify points belonging to 16 classes having the rectangular shapes shown by the green borders in Figure 3. The MLP has 2 input units, two 20 node hidden layers, and 16 output nodes. It has been trained using 2500 uniformly distributed patterns for each class. was greater than 0.5. MLP outputs have also been plotted for test points belonging to regions that have not been trained, and outside the green rectangles: they are at the left and right sides of Figure 3. The average classification rate for all classes, and the classification rate for classes 6 and 7, is reported in the first row of Table 1. Afterward, an adaptation set was defined to simulate an adaptation condition where only two of the 16 classes appear. The 5000 points in this set define a border between classes 6 and 7 shifted toward the left, as shown in Figure 4. In the first adaptation experiment, all the 760 MLP weights and 56 biases of the network were adapted. The catastrophic forgetting behavior of the adapted network is evident in Figure 4, where a blue grid has been superimposed to indicate the original class boundaries learned by full training. Classes 6 and 7 do actually show a relevant increase of their correct classification rate, but they have a tendency to invade the neighbor classes. Moreover, a marked shift toward the left affects the classification regions of all classes, even the ones that are distant from the adapted classes. This undesired shift of the boundary surfaces induced by the adaptation process damages the overall average classification rate, as shown in the second row of Table 1.

To mitigate the catastrophic forgetting problem, the adaptation of the network has been performed using Conservative Training. Figure 5 shows how the trend of classes 6 and 7 to invade neighbor classes is largely reduced, Class 6 and 7 fit well their true classification regions, and although the left shift syndrome is still present, the adapted network performs better as shown by the average classification rate in the third row of Table 1.

Our artificial test-bed is not well suited to LIN adaptation because the classes cover rectangular regions: thus, a linear transformation matrix that is able to perform a single global rotation of the input features is ineffective. Moreover, the degree of freedom of this LIN is really poor: the LIN includes 4 weights and 2 biases only. These considerations are confirmed by the results reported in line 4 of Table 1. Classes 6 and 7 are well classified, but the average classification is very bad because the adaptation of the LIN weights to fit the The mitigation of these effects introduced by Conservative Training is shown in Figure 6, and in line 5 of Table 1. The shift toward left syndrome is still visible, but the horizontal boundary surfaces are correct.

If we add, instead, a LHN between last hidden layer and the output layer, and we adapt its 420 weights plus biases only, we obtain better results than LIN adaptation (see line 6 of Table 1). However, as Figure 7 shows, the class separation surfaces are ugly. Class 6, and especially class 7 are spread out, class 3 is split, and thus the average classification rate is unacceptable.

Conservative Training does again a very good job, as shown in Figure 8 and in last line of Table 1, even if class 12 does not present high scores.

Experimental results on speech recognition tasks

Adaptation to a specific application may involve the speakers, the channel, the environmental noise and the vocabulary, especially if the application uses specific lists of terms. The proposed techniques have been tested on a variety of cases requiring different types of adaptation. The adaptation tasks that have been considered are listed in sub-section 5.1 below. The LOQUENDO default speaker and task independent Italian models, described in Section 2, were the seed models for the adaptation.

The results of our experiments show that the problem of forgetting is dramatic especially when the adaptation set is not characterized by a good coverage of the phonemes of the language. The use of Conservative Training mitigates the forgetting problem, allowing adaptation with a limited performance decrease of the model on other tasks (some performance reductions are inevitable because the ANN is adapted to a specific condition and thus it is less general).

Tests on various adaptation tasks

Application adaptation: Directory Assistance

We tested the performance of models adapted to a Directory Assistance application. The corpus includes spontaneous utterances of the 9325 Italian city names. The adaptation set has 53713 utterances; the test set includes 3917 utterances.

Vocabulary adaptation: Command words

The lists A1-2-3 of the SpeechDat-2 Italian corpus, containing 30 command words, have been used. The adaptation and the test sets include 6189 and 3094 utterances respectively.

Channel-Environment adaptation: Aurora-3

The benchmark is the standard Aurora3 Italian corpus. The Well-Matched train set has been used for adaptation (2951 utterances), while the results on the Well-Matched test set (the noisy channel, ch1) are reported (654 utterances). The results on these tests, reported in Table 2, show that a linear transform on hidden units (LHN) always outperforms a linear transform on the input space (LIN). This indicates that the hidden units represent a projection of the input pattern in a space where it is easier to learn or adapt the classification expected at the output of the MLP. The whole network row in the table corresponds to the adaptation of all the ANN weights by incremental training of the original network. This adaptation is feasible only if many adaptation data are available, and it is less effective than LHN.

Conservative training imposes some constraints to the adaptation process because it tries to prevent forgetting missing classes. Thus, the performance of the LIN/LHN models, adapted on a given task and tested on sentences of the same domain, are slightly better than the performance of the corresponding models adapted by means of CT in addition to LIN/LHN. This happens because, using a large adaptation set, there are enough samples for most of the outputs. Thus, there is little or no risk of catastrophic forgetting, and there is no need for the estimates obtained when input samples of different units are applied at the network input.

Conservative training, however, preserves the generality of the adapted model. This claim has been assessed by using a model adapted on a task and evaluating its recognition performance on a different generic common task. The generic task is large vocabulary (9.4k words) continuous speech recognition of 4296 phonetically balanced sentences uttered by 966 speakers. The average sentence length is 6 words. The results are obtained without language modeling. The adaptation tasks and the reference word error rate (29.3 %) achieved using unadapted acoustic models on the generic task are given in the first and last row of Table 3 respectively. Since the adapted models have been specialized to a specific condition, they are not expected to perform as well as the original model -task independent -on a continuous speech task. It was interesting, however, to have a measure of the generalization loss after adapting the standard model with different approaches. Table 3 highlights the effects of catastrophic forgetting, which takes place when the vocabulary of the adaptation set is small and has a poor phonetic coverage. This is particularly evident for the models adapted on the Command words and Aurora 3 tasks whose results on the generic continuous speech recognition task are emphasized in italics.

If the adaptation set has enough data for each class appearing in the test set of the new task, then LIN or LHN approach perform well without CT. The results reported in the first column of Table 3 confirm that the use of CT is not relevant when the model has been obtained using the phonetically rich adaptation set provided by the utterances collected for a Directory Assistance task. On the other hand, if the test set includes phonetic classes that are missing in the adaptation set, the use of CT in addition to LIN/LHN produces better results avoiding the catastrophic forgetting. This is shown in the second and third columns of Table 3, where the test set is still large vocabulary continuous speech, but the models have been adapted using command words and digits respectively.

Conservative Training, thus, mitigates the forgetting problem, preserving an acceptable performance of the adapted model on the task for which the original network was trained (open vocabulary, task independent speech recognition). The phonetic classes that were rarely represented, or were missing, in the adaptation set can be still reasonably well recognized by the adapted model.

Speaker Adaptation

Further experiments have been performed on the WSJ0 speaker adaptation test in several conditions. Three baseline models have been used:

the default LOQUENDO 8 kHz telephone speech model (trained with LDC MACROPHONE [19] -referred as MCRP in the Tables); a model trained with the WSJ0 train set (SI-84), 16 kHz.

a model trained with the WSJ0 train set (SI-84), down-sampled to 8 kHz. Furthermore, we tested two architectures for each type of models: the standard one (STD), described in sub-section 2.1 and an improved (IMP) architecture, characterized by a wider input window modeling a time context of 250 ms [START_REF] Dupont | A study of implicit and explicit modeling of coarticulation and pronunciation variation[END_REF], and by the presence a third 300 units hidden layer. The results, reported in Tables 4 and5, show that also in these tests LHN always achieves better performance that LIN. The combination of LIN and LHN (trained simultaneously) is usually better than the use of LHN alone. Conservative training effects are of minor importance in these tests because the adaptation set has a good phonetic coverage and the problem of unseen phonetic classes is not dramatic. Nevertheless, its use improves the performance (compare Standard LIN and LIN+CT), because it avoids the adaptation of prior probabilities of the phonetic classes on the (poor) prior statistics of the adaptation set. Finally, Figure 9 shows how CT influences the performance of models adapted with a different amount of adaptation data. The results, obtained with a bigram language model, refer to LHN adaptation of the standard models (STD) using up to 40 sentences of each speaker. The sentences are down-sampled to 8 kHz. The first point in the graph shows the 13.4% WER achieved using the standard model without adaptation.

Conservative training is particularly useful when the adaptation set is very small (a few sentences), because in that case the problem of the missing phonetic classes is more relevant. In particular, using a single sentence the error rate for LHN adaptation alone actually increases.

The experiments show an overall benefit of CT, considering that the minimal performance degradation reported for the improved network models adapted with LIN has been obtained using the trigram LM, which could mask the acoustic quality of the models.

Conclusions

A method has been proposed for adapting all the outputs of the hidden layer of ANN acoustic models and for reducing the effects of catastrophic forgetting when the adaptation set does not contain examples for some classes. Experiments have been performed for the adaptation of an existing ANN to a new application, a new vocabulary, a new noisy environment and new speakers. They show the benefits of CT, and that LHN outperforms LIN. Furthermore, experiments on speaker adaptation show that further improvements are obtained by the simultaneous use of LHN and LIN showing that linear transformations at different levels produce different positive effects that can be effectively combined. An overall WER of 5% after adaptation on WSJ0 using the standard trigram LM and without across word specific acoustic models compares favorably with published results.

Fig. 1 .Fig. 2 .

 12 Fig. 1. Artificial Neural Network including a linear input layer

Figure 3 Fig. 3 .Fig. 4 .

 334 Fig. 3. Training 16 classes on a 4-layer network with 760 weights

Fig. 5 .

 5 Fig. 5. Conservative Training adaptation of all the network weights.

Fig. 6 .Fig. 7 .Fig. 8

 678 Fig. 6. Conservative Training LIN adaptation.

  6 and 7, has the catastrophic forgetting effect of enlarging the regions of all classes.

Figure 9

 9 Figure 9 Word error rate on the WSJ0 8 kHz tests as a function of the amount of adaptation data (1, 5, 10, 20, 30, and 40 sentences).

Table 1 Correct

 1

	Adaptation	Forgetting	Average	Class 6	Class 7
	method	mitigation	classification rate	classification	classification
		technique	(%)	rate (%)	rate (%)
	1. None	None	95.9	98.5	93.3
	2. Whole network	None	83.1	100.0	98
	3. Whole network	CT	89.8	97.8	94.8
	4. LIN	None	42.6	100	95.7
	5. LIN	CT	69.0	99.0	91.8
	6. LHN	None	65.4	99.6	97.2
	7. LHN	CT	86.7	98.0	93.3

classification rates on the artificial data task.

Table 2

 2 Adaptation results (WER %) on different tasks using various adaptation methods. The seed adaptation models are the standard LOQUENDO telephone models. The model are adapted on a given task and tested on sentences of the same domain.

	Adaptation
	task	Application	Vocabulary	Channel-Environment
	Adaptation	Directory Assistance	Command Words	Aurora3 Ch 1
	method
	No adaptation	14.6	3.8	24.0
	Whole network	10.5	3.2	10.7
	LIN	11.2	3.4	11.0
	LIN + CT	12.4	3.4	15.3
	LHN	9.6	2.1	9.8
	LHN + CT	10.1	2.3	10.4
	Adaptation
	task	Directory Assistance	Command Words	Aurora3 Ch1
	Adaptation	Adapted Models	Adapted Models	Adapted Models
	method
	Whole network	36.3	63.9	126.6
	LIN	36.3	42.7	108.6
	LIN + CT	36.5	35.2	42.1
	LHN	40.6	63.7	152.1
	LHN + CT	40.7	45.3	44.2
	No adaptation		29.3

Table 3

 3 Evaluation of the forgetting problem: recognition results (WER%) on Italian continuous speech with models adapted on different tasks.

  The test set is the standard SI 5K read NVP Senneheiser microphone (si_et_05, 8 speakers x ~40 utterances). The bigram or trigram standard Language Models provided by Lincoln Labs have been used in these experiments.

	Train Set Net type Adaptation method Bigram LM Trigram LM
			NO adaptation	16.4	13.6
			Standard LIN	14.6	11.6
	MCRP	STD	LIN+CT	13.9	11.3
			LHN+CT	12.1	9.9
			LIN+LHN+CT	11.2	9.0
			NO adaptation	13.4	10.8
			Standard LIN	14.2	11.6
	WSJ0	STD	LIN+CT	11.8	9.7
			LHN+CT	10.4	8.3
			LIN+LHN+CT	9.7	7.9
			NO adaptation	10.8	8.8
			Standard LIN	9.8	7.6
	WSJ0	IMP	LIN + CT	9.8	7.7
			LHN + CT	8.5	6.6
			LIN+LHN+CT	8.3	6.3
	Table 4
	Speaker Adaptation word error rate on the WSJ0 8 kHz tests using different adaptation
	approaches.
	Train Set Net type Adaptation method Bigram LM Trigram LM
			NO adaptation	10.5	8.4
			Standard LIN	9.9	7.9
		STD	LIN+CT	9.4	7.1
			LHN+CT	8.4	6.6
	WSJ0		LIN+LHN+CT NO adaptation	8.6 8.5	6.3 6.5
			Standard LIN	7.2	5.6
		IMP	LIN+CT	7.1	5.7
			LHN+CT	7.0	5.6
			LIN+LHN+CT	6.5	5.0
	Table 5
	Speaker Adaptation word error rate on the WSJ0 16 kHz tests using different adaptation
	approaches.
	The adaptation set is the standard adaptation set of WSJ0 (si_et_ad, 8 speakers, 40
	utterances per speaker), down-sampled to 8 kHz when necessary.

This work was supported by the EU FP-6 IST Projects DIVINES and HIWIRE


Katarina Bartkova

Denis Jouvet
email: denis.jouvet@francetelecom.com

On using units trained on foreign data for improved multiple accent speech recognition

Keywords: Non-native speech, Speech recognition, Adaptation, Multilingual units, Non-Native Speech Recognition

Foreign accented speech recognition systems have to deal with the acoustic realization of sounds produced by non-native speakers that does not always match with native speech models. As the standard native speech modeling alone is generally not adequate, it is usually extended with models of phonemes estimated from speech data of foreign languages, and often complemented with extra pronunciation variants. In this paper, the focus is set on the speech recognition of multiple non-native accents. The speech corpus used was recorded from speakers originated from 24 different countries. The introduction of models of phonemes of the target language adapted on foreign speech data is presented and detailed. For the recognition of non-native speech comprising multiple foreign accents, this approach provides better performance than the introduction of standard foreign units. The selection of the most frequent acoustic variants for each phoneme is also discussed as this method makes recognition results more homogenous across speaker language groups. Furthermore, the adaptation of the acoustic models on non-native speech data is studied. Results show that detailed models, which include the modeling of extra pronunciation variants through acoustic units estimated on foreign data, benefit more from the task and accent adaptation process than baseline standard models used for native speech recognition. In addition, experiments show that an adaptation of the acoustic models on a limited set of foreign accents provides speech recognition performance improvements even on foreign accents absent from the adaptation data.

Non-Native Speech Recognition 3 / 23 January 12, 2007

List of Figures

List of Tables

INTRODUCTION

Efficient handling of foreign accent is getting crucial in automatic speech recognition systems as speech enabled information services are increasingly used by non-native speakers in an ever more cosmopolitan society. However, although the recognition of native speech reaches in many cases an acceptable level, speech recognition performance is considerably lowered when the system has to deal with words or sentences uttered with an altered pronunciation due to a foreign accent. While investigating the variability between speakers through statistical analysis methods, [START_REF] Huang | Analysis of speaker variability[END_REF] found that the first two principal components correspond to the gender and accent respectively.

Accented speech is associated to a shift within the feature space of phonemes [START_REF] Van Compernolle | Recognizing speech of goats, wolves, sheep and … non-natives[END_REF]. For native accents, the shift, applied by large groups of speakers, is global and more or less important, but on overall, the acoustic confusability is not changed significantly. In many cases, most of the regional variants of a language are handled in a blind way through a global training of the speech recognition system using speech data that cover all regional variants. Accent classification has been studied since many years [START_REF] Arslan | Language accent classification in American English[END_REF] based either on phone models [START_REF] Kumpf | Automatic accent classification of foreign accented Australian English speech[END_REF]King, 1996, Teixeira et al., 1996) or specific acoustic features [START_REF] Fung | Fast accent identification and accented speech recognition[END_REF]. Also, good classification results between regional accents are reported in [START_REF] Draxler | Identification of regional variants of high German from digit sequences in German telephone speech[END_REF] for human listeners on German SpeechDat data, and in [START_REF] Lin | Phoneme-less hierarchical accent classification[END_REF] for automatic classification between American and British accents. To handle variants occurring in regional accents, enriched modeling is often used through multiple acoustic models associated to large groups of speakers as in [START_REF] Beattie | An integrated multidialect speech recognition system with optional speaker adaptation[END_REF][START_REF] Van Compernolle | Speaker clustering for dialectic robustness in speaker independent speech recognition[END_REF], or through the introduction of detailed pronunciation variants at the phonetic level [START_REF] Adda-Decker | Pronunciation variants across systems, languages and speaking styles[END_REF]Lamel, 1998, Humphries et al., 1996). However, adding too many systematic pronunciation variants may be harmful [START_REF] Strik | Modeling pronunciation variation for ASR: A survey of the literature[END_REF] and experiments showed that it was preferable to have models only for a small number of large speaker populations than for many small groups [START_REF] Beattie | An integrated multidialect speech recognition system with optional speaker adaptation[END_REF][START_REF] Van Compernolle | Speaker clustering for dialectic robustness in speaker independent speech recognition[END_REF].

Compared to native speech recognition, performance degrades when recognizing accented speech and even more when recognizing non-native speech [START_REF] Kubala | Comparative experiments on large vocabulary speech recognition[END_REF][START_REF] Lawson | Effect of foreign accent on speech recognition in the NATO N-4 Corpus[END_REF]. Foreign accent is harder to handle than regional accent and its processing remains among the most difficult speech recognition tasks [START_REF] Goronzy | Generating non-native pronunciation variants for lexicon adaptation[END_REF]. Non-native accent is less homogenous than native accent as it is influenced both by the native language of the speaker and by the level of his proficiency, that is, his capability to imitate the pronunciation of the target language. As the foreign accent shift from the standard pronunciation depends on theses two factors, non-native speech variability is important.

One of the reasons of the degradation of the recognition performance observed on foreign accented speech is that the acoustic models are usually trained only on speech with standard native pronunciations. Non-native speech recognition is not properly handled by native speech models, no matter how much accented data is included in the training [START_REF] Beattie | An integrated multidialect speech recognition system with optional speaker adaptation[END_REF]. This is due to the fact that non-native speakers can replace an unfamiliar phoneme in the target language, which is absent in their native language phoneme inventory, with the one considered as the closest in their native language phoneme set [START_REF] Flege | Interaction between the native and second language phonetic subsystems[END_REF]. In addition, January 12, 2007 differences between foreign accented speech and native speech occur also at the phonological level when a phoneme of the target language is replaced with a different phoneme of the same language [START_REF] Bartkova | Language based phone model combination for ASR adaptation to foreign accent[END_REF]. The replacement of some sounds by others, as well as their insertion or deletion, cannot be handled by the usual triphone-based modeling [START_REF] Jurafsky | What kind of pronunciation variation is hard for triphones to model?[END_REF]. Therefore, for dealing efficiently with foreign accented speech, speech recognition systems should handle variants occurring both at the acoustic level [START_REF] Witt | Off-line acoustic modelling of non-native accents[END_REF] and at the phonological level [START_REF] Bonaventura | Speech recognition methods for non-native pronunciation variants[END_REF]. Introducing multiple phonetic transcriptions that handle alterations produced by non-native speakers is a usual approach, and is generally associated to a combination of phone models of the native language with phone models of the target language [START_REF] Bartkova | Language based phone model combination for ASR adaptation to foreign accent[END_REF][START_REF] Bonaventura | Speech recognition methods for non-native pronunciation variants[END_REF][START_REF] Witt | Off-line acoustic modelling of non-native accents[END_REF]. When a single foreign accent is handled, some accented data can be used for training or adapting the acoustic models [START_REF] Aalburg | Foreign-accented speaker-independent speech recognition[END_REF], He and Zhao, 2003[START_REF] Liu | MLLR-based accent model adaptation without accented data[END_REF][START_REF] Uebler | Recognition of non-native german speech with multilingual recognizers[END_REF]. Alteration rules can be defined from phonetic knowledge or estimated from some accented data [START_REF] Livescu | Lexical modeling of non-native speech for automatic speech recognition[END_REF], or inferred using only native speech of both languages [START_REF] Goronzy | Generating non-native pronunciation variants for lexicon adaptation[END_REF]. [START_REF] Raux | Automated Lexical Adaptation and Speaker Clustering based on Pronunciation Habits for Non-Native Speech Recognition[END_REF] investigates the adaptation of the lexicon according to preferred phonetic variants.

Only adding accented speech into the training data base may not solve the problem, as modeling together too "heterogeneous" pronunciations may not be efficient for any foreign accent, while the resulting modeling may be less efficient for standard pronunciation. It was found that a separate modeling of foreign accents is needed to capture accurately non-native pronunciations [START_REF] Beattie | An integrated multidialect speech recognition system with optional speaker adaptation[END_REF]. When dealing with various foreign accents, phone models of several languages can be used simultaneously with the phone models of the target language (Bartkova and Jouvet, 2004a), multilingual units can be used [START_REF] Uebler | Recognition of non-native german speech with multilingual recognizers[END_REF] or specialized models for different speaker groups can be elaborated [START_REF] Cincarek | Speech recognition for multiple non-native accent groups with speaker-group-dependent acoustic models[END_REF].

In this paper, the focus is set on the recognition of non-native speech comprising multiple accents. The application context corresponds to speech-activated interactive vocal services, where the identity of the speaker, thus its non-native accent, is not known. The paper is organized as follows. Section 2 describes the speech corpus comprising multiple non-native accents as well as the baseline performance obtained with standard native speech models. Section 3 details and discusses the introduction of modeling variants. Several types of variants are considered. Phonological rules and introduction of standard foreign units are first recalled. Then the introduction of models of phonemes of the target language adapted on foreign data is detailed. This section ends by a presentation and a discussion of the recognition results. Section 4 investigates the selection of variants. Finally section 5 analyses the behavior of these approaches when the acoustic models are adapted on non-native speech data. In addition, the adaptation of the models on non-native speech using various subsets of foreign accents is discussed.

SPEECH DATA AND BASELINE OVERVIEW

The speech corpus used in this study was collected from speakers originating from various countries and pronouncing French words or expressions. Thus, this corpus exhibits several types of foreign accents.

Speech Modeling

The speech modeling used in this study is HMM-based, and relies on a context-dependent modeling of the phonemes [START_REF] Jouvet | On the Modelization of Allophones in an HMM based Speech Recognition System[END_REF]. This modeling approach of contextdependent units is similar to the triphone approach, but the sharing of the contextual parameters (i.e. output densities associated to entry and exit states of the phoneme models) is defined beforehand according to phonological knowledge. Hence left and right contexts are defined as classes of phonemes sharing same feature sets (place of articulation, voiced/unvoiced, plosive/fricative/nasal/...) and having a similar influence on the acoustic realization of the sound. About 20 phoneme classes are used for specifying the left and right contexts. Being defined according to a priori knowledge, the contexts are compatible among different languages, thus making possible a simultaneous use of contextual models from different languages with an adequate handling of contexts at phoneme boundaries.

The acoustic analysis computes MFCC features and a frame synchronous line adaptation process is also applied which plays a similar role as the mean cepstral subtraction (C. Mokbel et al., 1996). Mixtures of Gaussian densities are used for the modeling of the acoustic features: MFCC and energy, plus their first and second order temporal derivatives.

In all the reported experiments the recognition vocabulary consisted of 83 French words and expressions. These are typical commands for vocal interactive services, such as the digits, the names of days and months and command words such as "aide" (help), "suivant" (next), "précédent" (previous), etc. A few expressions are also present in the vocabulary, such as "mode d'emploi" (instructions for use), "après-demain" (the day after tomorrow), etc. Some vocabulary entries had rather similar phonetic forms, as for example /m.w.a/ for the word "mois" (month) and /m.w. / for the word "moins" (less). All the words are common usual words, and many of them contain only one or two syllables.

Non-Native Speech Corpus

The speech corpus contains isolated utterances of those 83 French words and expressions collected over the telephone. It was recorded from speakers originating from 24 countries. Each speaker uttered the items in a random order according to vocal prompts. The corpus was split into two parts, one part used as an adaptation set (in section 5), and the other as a test set. The types of non-native accents of the test set are detailed below. The adaptation set has a similar size; it exhibits the same types of accents, but was collected from different speakers. This adaptation set was also used for selecting modeling variants in section 4.

For analyzing the speech recognition performance with respect to the accents, the test set was divided into subsets, each one corresponding to the French utterances pronounced by speakers originating from a given language group. 11 language groups were defined.

• French group: 94 speakers from France, Belgium, Switzerland, Canada, Guadeloupe, Reunion, ... • Spanish group: 35 speakers from Spain.

• English group: 96 speakers from USA, UK, Ireland, & Australia.

• German group: 113 speakers from Germany & Austria.

ACCEPTED MANUSCRIPT

Non-Native Speech Recognition 7 / 23

January 12, 2007

• Italian group: 56 speakers from Italy.

• Portuguese group: 17 speakers from Portugal.

• African group: 50 speakers from Senegal, Congo, Mali, Cameroon 1 , …

• Arabic group: 53 speakers from Algeria, Tunisia & Morocco.

• Turkish group: 53 speakers from Turkey.

• Cambodian group: 48 speakers from Cambodia.

• Asian group: 69 speakers from China & Vietnam 2 .

Eventually two language group clusters were also defined. The first one named "EsEnDe" corresponds to the merge of the Spanish, English and German groups. This grouping was driven by the availability of trained standard foreign units for these three languages. The second one named "Other" corresponds to the merge of the remaining groups of foreign speakers: Italian, Portuguese, African, Arabic, Turkish, Cambodian and Asian groups.

Baseline Performance

The baseline recognition results are reported in Figure 3 (native model M1.A1 -leftmost bar). They were obtained with standard context-dependent French acoustic models trained on a large corpus of telephone French speech data (about 190 hours of signal, corresponding to words and sentences collected from several thousands speakers) and standard native pronunciation descriptions of the French vocabulary words. In the figures, the names Mn.Ak refer to both the type of modeling (M1 for baseline, M2 with phonological rules, and so on) and the data on which the acoustic models are trained (A1 for standard training of generic units, A2 to A5 for task adaptation with various subsets of non-native data). Table 1 summarizes the notations used. In Figure 3 and similar figures, the left part reports the word error rates on each cluster, namely French speakers, then EsEnDe cluster (that is Spanish, English and German speakers), and finally for the other speakers, that is speakers belonging to the 7 other language groups. The right part reports the error rates for each language group.

As expected, the French speakers -i.e. native speakers -obtain the lowest error rate; however large differences are observed in the error rates among the various language groups ranging from less than 6% for German speakers up to 12% and more for Spanish and English speakers.

The reason of the good recognition performance obtained on German speakers can be partly explained by the fact that the German language as well as the French language have a tense way of pronunciation (Delattre, 1966), and that they share closed and open vowels as well as rounded front vowels (absent from Spanish and English) and the velar [ . 1 The reason for this group is the following: all these countries are francophone; however there was no indication of the speakers' mother tongue. 2 The reason of maintaining two separate groups between Cambodian and Asian stems from the fact that Chinese and Vietnamese languages are tonal ones while the Cambodian Khmer language is not.

ADDING MODELING VARIANTS

In order to handle non-native speech accents, extra pronunciation variants were introduced in the lexicon descriptions. As non native speakers may utter phonemes either as they are pronounced in the target language or as they are pronounced in their mother tongue, pronunciation variants can be defined using models of phonemes of the mother tongue. This performs well in a bilingual approach, as described in [START_REF] Bartkova | Language based phone model combination for ASR adaptation to foreign accent[END_REF] and [START_REF] Witt | Off-line acoustic modelling of non-native accents[END_REF], when the mother tongue of the speaker is known. However, when the speaker mother tongue is not known (e.g. non-native speakers calling a speech enabled service) it is only possible to rely on an enriched modeling using various available foreign models. Another way of enriching the modeling consists in using models of phonemes of the target language adapted on speech data from foreign languages, as presented in (Bartkova and Jouvet, 2004a). Furthermore, phonological rules are useful for generating pronunciation variants [START_REF] Bartkova | Language based phone model combination for ASR adaptation to foreign accent[END_REF], 2004a[START_REF] Bonaventura | Speech recognition methods for non-native pronunciation variants[END_REF] by replacing some phonemes of the target language by others (often influenced by native language pronunciation).

Phonological Rules

Several sets of phonological rules were defined and used to generate pronunciation variants for the vocabulary words.

The first set of rules deals with vowels having two apertures and no significant timber differences. The French vowels targeted by these rules are [ ] [ ] and [ ]. Once these rules applied, no difference is made between open and closed vowels, that is, at every occurrence of a closed vowel its open counterpart is also added into the word phonetic description (and vice versa). For example the non-native pronunciation for the word "aide" (help) will be /( + ) / instead of the standard native pronunciation / /. The two aperture vowel processing is achieved through the application of 3 pairs of rules (one pair for each of the vowels having two apertures, as indicated before), like the following rules:

( + ) ( + )
In fact, it is supposed that a foreigner, whose native language does not contain a closed / open vocalic opposition, or at least not in the same phonotactic distribution as in French, would have difficulty to make the difference between them. Thus, maintaining systematically the two vowels should allow the speaker to use any of them.

The second set of rules deals with denasalizing nasal vowels, that is, a nasal vowel can be decomposed into the oral vowel which corresponds to the same vocalic timber as the nasal one (no matter which letter occurs in the spelling form) followed by a nasal consonant which articulation place depends on the adjacent consonant: [ ] before apical consonants, [ ] before labial consonants and [ ] before velar consonants. Therefore, for example, the digit "cinq" (five) will have the non-native pronunciation / ( + ) / instead of its standard native phonetic form which is / /. The denasalization is achieved through the application of a set of 4 rules (one for each French nasal vowel), like the following: The reason to these transformations is the following: nasal vowels are seldom present in the inventory of the world languages [START_REF] Valée | Système vocalique: topologie et tendances universelles[END_REF] and most of the speakers are unfamiliar with the correct oral plus nasal channel coupling. It is worth mentioning that nasal vowel decomposition exists even in French south dialect where the nasal vowel is decomposed into oral vowel plus a nasal appendix.

The third and last set of rules contains two rules to deal with the French front round vowel and semi-vowel. According to this rule, the front rounded vowel [ ] can be replaced by the back rounded vowel [ ] and the French front rounded semi-vowel [ ] can be replaced by the back rounded semi-vowel [ ]. In fact, speakers having a heavy accent in French often show difficulty uttering front rounded vowel when their native language does not contain such a vowel. These speakers (for example of Spanish, Italian or English origin) make such replacement especially when the same or a very close phonetic form of a word already exists in their own language and when it contains the same phoneme shift (back rounded versus front rounded).

Thus, for example, once all the above phonological rules are applied, the non-native pronunciation description of the French word "continuer" (to follow) becomes / ( + ) ( + )( + )/ whereas the standard native form is / /.

Phone Models from Foreign Languages

The acoustic modeling was enriched using standard acoustic models of phonemes of a few foreign languages. These standard models of the phonemes in each language were trained in a classical way that is, using large corpora of telephone speech data of the given language and associated pronunciation descriptions (about 60 hours of signal for German data, 80 hours for Spanish data and 140 hours for English data; each corpus -SpeechDat or similar type of corpus -contains words and sentences pronounced by many speakers). For example, English models of the English phonemes were trained using English speech data pronounced by English speakers, as indicated on the left part of Figure 2.

Then, for each phoneme several models were used: on the one hand, the model of the phoneme in the target language (for example _fr_FR for the model of the French phoneme / / estimated on French speech data), and on the other hand the models of the corresponding phonemes in other languages. In the reported experiments, the foreign standard units used correspond to the Spanish, English and German languages as represented on the left part of In this approach, multiple acoustic models are used for each phoneme, each of them coming from a different language. Hence, it is important to recall that the context-dependent modeling of the phonemes rely on an a priori sharing of contextual parameters according to phonological knowledge. As mentioned before in section 2.1, because of such phonologically-based knowledge, the context-dependent models from different languages can be glued together with a proper handling of the context dependencies.

Having multiple acoustic models of different languages in parallel for each phoneme enables the Viterbi-based decoder to choose for each phoneme the acoustic model that provides the best match according to the input signal, reflecting the degree of foreign accent of the speaker. There are no constraints whatsoever to use units of the same category all along the decoding path (i.e. complete path with native units only or with foreign units only) since such constraints would not be relevant as non-native accent affects phonemes independently of each other, in a speaker and language dependent way.

Target Phone Models Adapted on Foreign Data

Another way of enriching the modeling consists in using models of phonemes of the target language adapted on speech data from foreign languages (right part of Figure 2). It is expected that such acoustic models will exhibit, at least to some extend, the pronunciation of the target phonemes by non-native speakers. and French units adapted on foreign data (right).

Through the application of adequate correspondences between the phoneme sets of the foreign language and of the target language, foreign speech data was used to adapt the acoustic models of the target language. The following example explains how French phoneme models were adapted on English data. As an example, let's take the words "Paris" and "message" from the English corpus.

_us . _us

The suffix '_US' indicates that the utterances belong to the English corpus, and '_us' indicates that the English pronunciations of the words are given in term of American-English phoneme units. In order to match these English units on the French phoneme units, either a simple correspondence between the phoneme units was used, such as: _us _fr or a more complex one, as for example: _us _fr . _fr Applying these transformations on every lexicon pronunciation led to the following pronunciation descriptions of these two words: These transformed pronunciation descriptions were then used for adapting the acoustic models of the French phonemes on English speech data. The acoustic models were adapted using an incremental enrollment procedure which is equivalent to segmental MAP adaptation January 12, 2007 with specific choice of priors (Mokbel and Collin, 1999). In the following, such foreign speech adapted units will be denoted as _fr_US, for example, where the first suffix, '_fr', refers to the language of the phoneme (here French, that is the target language) and the second suffix, '_US', specifies the origin of the speech data used for adapting the model parameters.

The few phonemes which do not have similar counterparts in the target language can be either associated to some garbage units, or more simply, the related sentences can be ignored during the adaptation process.

Subsequently, for the recognition process, each phoneme was modeled simultaneously by the acoustic model corresponding to the target language and a few acoustic models resulting from the adaptation of the target phoneme models on several foreign languages, as represented on the right part of Figure 1. In the reported experiments, the adapted models introduced were the models of the French phonemes adapted on Spanish data (for example _fr_SP), English data ( _fr_US) and German data ( _fr_DE): these are the same 3 languages as before.

Experiments and Discussions

Figure 3 reports the recognition results obtained with the various modeling approaches described before: baseline native acoustic and native pronunciation modeling (M1), baseline acoustic modeling and pronunciation lexicon enriched through the application of phonological rules (M2), then with inclusion of standard foreign units (M3) or French units adapted on foreign data (M4). The last model (M5) will be discussed later in chapter 4. In all cases, generic acoustic models (A1) were used (i.e. these acoustic models are not specific to the recognition task under consideration). The notations used are summarized in Table 1. When phonological rules were applied (model M2, second bar) significant recognition performance improvements were observed for some speaker language groups, such as for Spanish, English, Italian, Portuguese and Turkish speakers. The performance improvement can be explained partly by the fact that the first four languages (Spanish, English, Italian and January 12, 2007

Portuguese) share with French a great amount of lexical units, having similar spelling forms but with some pronunciation variations. Some of these pronunciation variations were taken into account precisely through the application of the phonologic rules, hence the improvement of the recognition performance for these speaker language groups. However, the approach seemed ineffective for several other language groups. Nevertheless, the introduction of phonological rules improved the overall recognition performance. Introducing foreign standard units in the modeling, besides the phonological rules (model M3, third bar), improved the speech recognition performance for Spanish and English speakers, that is, two of the languages corresponding to the added foreign units; however no improvement was observed for the other language groups, except for Italian speakers. Though German modeling units were also added into the recognition procedure, degradation was observed for this speaker group. One possible explanation of such a less satisfactory performance can be the following: French and German (unlike French and English or French and Spanish) have very similar vowel inventories. Therefore the French modeling units are rather adequate for the recognition of accented speech produced by German speakers. Adding foreign modeling units of other languages augments the perplexity in the decoding procedure and therefore increases the probability of errors as well. In fact, the recognition performance obtained on German speakers evolves in a rather similar way as the performance obtained on French speakers (see Figure 3).

The results obtained with model M4 (fourth bar) show that recognition performance improves on many language groups when acoustic models of the target language phonemes (here French phonemes) adapted on foreign data are used together with phonological rules.

Comparing to baseline recognition performance, large error rate reductions are observed on many speaker language groups while a small degradation is observed only for German speakers.

Experiments were also conducted with modeling variants corresponding to the introduction of foreign standard units or the introduction of models of French phonemes adapted on foreign data without the application of phonological rules. Results reported in (Bartkova & Jouvet, Non-Native Speech Recognition 14 / 23 January 12, 2007January 12, 2006) ) showed small improvements for a few speaker language groups only. Hence this clearly demonstrated that recognition performance improvements on non-native speech result from a simultaneous application of the phonological rules and the introduction of units trained on foreign data.

SELECTING MODELING VARIANTS

Adding pronunciation variants, through the introduction of units trained on foreign data in the modeling, improved significantly the recognition performance for non-native speakers corresponding to the languages of the added units. However the improvement was smaller, or even inexistent, for some other language groups and some degradation was observed for French speaking speakers. One can presume that some of the added variants are not useful, or may even be harmful for some speaker categories. Hence the of selection processes for optimizing the variants.

Preliminary Experiments

In (Bartkova and Jouvet, 2004b), an analysis of the frequency of usage of foreign units in the modeling of non-native French speech was initiated. This was achieved through a forced alignment of the speech signal with an enriched modeling as the one represented in Figure 1, left part. In this model each phoneme of the target language pronunciation description was represented by a plurality of acoustic units associated to different languages. It appeared that French units were the most often used by all speakers of all language groups and that the second most frequently used units were generally those, corresponding to the native language of the speakers.

It was also observed that the usage of foreign standard units was not evenly spread over the phonemes. It was higher for vowels than for consonants. Therefore, experiments were conducted in which foreign standard units or foreign-adapted units were introduced as variants only for the vowels while the consonants were modeled with the French standard acoustic models only. Some preliminary experiments were reported in [START_REF] Bartkova | Using multilingual units for improved modeling of pronunciation variants[END_REF]. Limiting the introduction of variants to the vowels led to recognition performance improvements for some language groups, including for native French speakers, but performance degradations were observed on some other groups.

A detailed analysis of the usage of the variants showed that, across the speaker language groups, not always the same phoneme was the most frequently replaced by its foreign counterpart. For example, the Spanish model for /b/ was frequently used by Spanish speakers speaking French, while the English and German models for /b/ were seldom used by English and German speakers speaking French. This finding led to investigating a phoneme dependent selection of the relevant variants.

Most Frequently Used

As unrelevant variants may be harmful for the recognition process, a method was elaborated for selecting for each phoneme the most relevant variants. This was investigated in a multiple acoustic modeling framework, using acoustic models of the French phonemes adapted on foreign data. In order to determine which variants are used, the utterances of a development January 12, 2007 set are aligned (Viterbi forced alignment) on the non-native pronunciation descriptions which include all variants: application of phonological rules and acoustic modeling variants as in the right part of Figure 1. Then, for each phoneme, frequencies of usage of the acoustic variants were estimated.

Two selection processes were experimented. In the first selection process, only the n-most frequent variants, observed on the development set, were kept with n equal 1, 2, 3, and so on. Whereas, in the second selection process, only the acoustic variants that were used at least x% of the time on the development set were kept. As the selection is applied separately for each phoneme, this second approach leads to a number of variants that may vary from one phoneme to another. For the results reported in Figure 4, x took the values 5%, 15% and 25%.

The lower this threshold was the higher was the number of variants used for each phoneme.

Only results obtained with the second selection process are detailed below, however, results obtained with the first proposed selection process were quite similar.

( ) ) ) ! Error rate & # ' ! # $ % ! " !
Figure 4 -Error rates for each language group using phonological rules and various selections of foreign-adapted units.

Figure 4 displays recognition error rates per speaker's language group. Baseline results are reported on the left, and results using phonological rules and all the foreign-adapted acoustic units, as well as acoustic variants for each phoneme, are reported on the right. Intermediate results correspond to the selection of different amounts of the most frequently used acoustic variants. The curves show a smooth evolution with respect to the amount of variants selected.

When the amount of acoustic variants increases (e.g. from point 25% up to "All"), performance is getting more homogenous across language groups. Results corresponding to the 15% threshold are also reported in Figure 3 (model M5, last bar).

An interesting point is that for groups for which performance degraded when all variants were used (for example French and German speakers), the degradation was less important when only a limited amount of variants were introduced (e.g. point 15%). For Spanish and English speakers, the performance improvements were reduced when the global amount of variants got limited. As for the languages for which no adapted units were available, Figure 4 shows that by limiting the amount of foreign-adapted variants (here those used more than 15% of the January 12, 2007 of priors (Mokbel and Collin, 1999). It should be noted that for each utterance of the adaptation set, the Viterbi alignment selects the most adequate (best matching) variant among all the possible variants of the current model.

French units _fr_FR

Standard foreign units _us_US, _es_ES, … The recognition results obtained with the task and accent adapted models are reported in Figure 6. Obviously, the adaptation improves the recognition performance for all types of modeling. Even after task and all accent adaptation, when the baseline modeling, which consists of only French units with no application of any phonological rules, is used (model M1.A5) the error rates for non-native speakers remains much higher than for French native speakers. Adding few pronunciation variants through phonological rules (model M2.A5) provides better results for almost all language groups, hence proving their importance in such a recognition task, even when models are adapted using non-native speech data.

As for the results obtained with models relying on units initially trained on foreign data, their behavior after task and all accents adaptation are somewhat similar to the behavior of the results obtained with the standard generic models: the introduction of units corresponding to the adaptation of French phonemes on foreign data (model M4.A5) leads to better recognition performance than the introduction of standard units of foreign languages (model M3.A5). Moreover, using the foreign adapted French units and restricting the variants to those that are the most frequently observed on forced alignments of the adaptation set data, remains the best performing solution, even after task and all accents adaptation (model M5.A5).

However, the recognition performance obtained for each speaker language group with model M5 (selection of units) compared to performance obtained with model M4 (all variants) does not vary in exactly the same way for adapted models (Figure 3) and generic models (Figure 6). This is due to the adaptation on accented data, which adjust the acoustic parameters of the modeling without using any a priori knowledge about the type of accent they are supposed to model. Moreover, a mixed of non-native accents (adaptation set) are used in the adaptation process. Nevertheless overall results are similar, the selection of units leads to a small performance degradation on average on the group of Spanish, English and German speakers (group of languages corresponding to the added units), while performance improvements are observed for French speakers as as speakers from other languages. Global results are summarized in Table 3.

Subset of Accents

This second set of experiments aims at studying the impact of the task and accent adaptation when only subsets of accents are present in the adaptation data. In order to do so, the adaptation set was split into several subsets, according to the same language criteria as defined in section 2.2 for the test set: data uttered by French speakers only; data from Spanish, English and German speakers ("EsEnDe" subset); and data from speakers of other language groups.

January 12, 2007 Results obtained for the best modeling variants (i.e. model M5: application of phonological rules and selection of added foreign-adapted French units) are reported in Figure 7. Results show that the adaptation on French speakers provides some improvement for almost all the language groups. The adaptation on data from non-native speakers (for example models M5.A3 and M5.A4) also provides some improvement for the French group data. This improvement can be considered as the effect of the task adaptation, as recording conditions were similar for French and for non-native speakers.

The adaptation using Spanish, English and German speakers provides noticeable improvements for these speaker language groups. The very interesting point is that such a limited accent adaptation yielded also significant recognition performance improvements on the other speaker language groups. Nevertheless, the adaptation on the subset "Other" reveals that the best results are still obtained when the adaptation and testing sets correspond to the same type of accents. Finally, using the full adaptation set, that is all available types of accents, leads to the best recognition performance. Global results are summarized in Table 4.

Let us also recall that the modeling relies on multiple sets of units for each phoneme, and that the percentage of usage of each of them (along the best Viterbi alignment path) depends on the speaker language group. However, in every case the amount of usage of native French units is important (Bartkova and Jouvet, 2004b). Hence native French units were adapted in each case, which may explain part of the systematic performance improvement observed after adaptation whatever subset was used for adaptation. Also as the frequency of usage vary according to the speaker language group, the more frequently used models on a given group were better adapted with speech data from that group (than with speech data of another group), which resulted in better performance improvements for speakers of the corresponding group.

CONCLUSION

In this paper, recognition of non-native speech, comprising multiple foreign accents, was analyzed. Modeling pronunciation variants proved to be necessary for handling non-native speech variabilities. The application of phonological rules helped handling the replacement of some phonemes by others. Furthermore, introducing models of phonemes in foreign languages provided a way of modeling the realization of sounds by non-native speakers. Such a modeling is efficient when units from languages, corresponding to the origin of the non-native speakers, can be used. A new approach, based on the adaptation of models of French phonemes on foreign speech data, was introduced, and it proved to be effective and robust, even for speakers from other language groups.

Previous studies analyzing the usage of foreign units on non-native speech led to investigating the reduction of the amount of variants by selecting the most relevant ones. It was observed, for example, that foreign models were more frequently used for vowels than for consonants. Therefore automatic selection methods were developed. One of them consisted in keeping for each phoneme only the variants, among the adapted units, that were the most frequently used on a development set. This approach reduced the amount of variants that were introduced, and led to the best and most homogenous recognition results across various speaker language groups.

The adaptation of the acoustic models on various sets of non-native speech was also studied.

Results showed that detailed models, including phonological rules and extra units, benefited more from the task and accent adaptation process than baseline standard models used for native speech recognition. Moreover, after adaptation on a limited set of foreign accents, recognition performance was improved even on other types of foreign accents. This interesting result proves that the detailed modeling, relying on the use of phonological rules and models of target phonemes adapted on a few foreign languages, is useful, and takes benefit of the available task accented data through classical adaptation processes, even if all accents are not present in the adaptation data.

Figure 1 -Figure 2 -Figure 3 -Figure 4 -Figure 5 -

 12345 Figure 1 -Adding foreign standard units (left) or French units adapted on foreign data (right). ................................................................................................................10 Figure 2 -Training scheme for standard foreign units (left) and French units adapted on foreign data (right). ............................................................................................11 Figure 3 -Error rates for each language group for baseline and modeling variants, before task and accent adaptation. .................................................................................13 Figure 4 -Error rates for each language group using phonological rules and various selections of foreign-adapted units......................................................................15 Figure 5 -Task and accent adaptation scheme..................................................................17

Figure 6 -

 6 Figure 6 -Error rates for each language group for baseline and modeling variants, after task and accent adaptation. .................................................................................17

Figure 7 -

 7 Figure 7 -Error rates for each language group for the best modeling variant, after various accents adaptation. .............................................................................................19

  realization of 'N' is [ ], [ ] or [ ] depending on the right context (following consonant).

Figure 1 -

 1 Figure 1 -Adding foreign standard units (left) or French units adapted on foreign data (right).

Figure 2 -

 2 Figure 2 -Training scheme for standard foreign units (left)and French units adapted on foreign data (right).

Figure 3 -

 3 Figure3-Error rates for each language group for baseline and modeling variants, before task and accent adaptation.

Figure 5 -

 5 Figure 5 -Task and accent adaptation scheme.

Figure 6 -

 6 Figure6-Error rates for each language group for baseline and modeling variants, after task and accent adaptation.

Table 1 -

 1 Summary of notations for pronunciation modeling and acoustic parameters ....12

Table 2 -

 2

Summary of results (error rates) with generic acoustic models.........................16

Table 3 -Summary of results (error rates) with adapted acoustic models. .......................18 Table 4 -Summary of results (error rates) for various non-native accent adaptations. .....20

  Their standard English pronunciations are:

	Paris_US	_us .	_us . _us .	_us .	_us
	Message_US	_us .	_us .	_us .

Table 1 -

 1 Summary of notations for pronunciation modeling and acoustic parameters

	Pronunciation modeling	Acoustic parameters
	M1 Native pronunciations only	A1 Generic acoustic models
	M2 M1 (native pronunciations) plus variants	A2 Task adaptation using data from French
	derived through phonological rules	speakers only
	M3 M2 plus variants associated with standard	A3 Task adaptation using data from Spanish,
	foreign units	English & German speakers
	M4 M2 plus variants associated with French	A4 Task adaptation using data from speakers
	units adapted on foreign data	corresponding to other languages
	M5 M4, but only most frequent variants are	A5 Task adaptation using data from speakers
	kept for each phoneme	corresponding to all types of accent

Table 3 -

 3 Summary of results (error rates) with adapted acoustic models.

		French	Span., Engl. & Germ.	Other language groups
		1227 utt.	3106 utt.	4322 utt.
	M1.A5: Baseline,	3.2 %	5.6 %	5.9 %
	adapted on all types of accents	(+/-1.0 %)	(+/-0.8 %)	(+/-0.7 %)
	M4.A5: phon. rules & all units; adapted on all types of accents	4.6 %	5.0 %	5.6 %
	M5.A5: phon. rules & select. units; adapted on all types of accents	4.2 %	5.2 %	5.2 %

Table 4 -

 4 Summary of results (error rates) for various non-native accent adaptations.

	Non-Native Speech Recognition	20 / 23	January 12, 2007

Non-Native Speech Recognition

In Table2, as well as in following tables, the 95% confidence interval is given only for the reference results corresponding to the first line; here the results obtained with the baseline models. This avoids overloading the tables with too many figures.

Acknowledgements ..............................................................................................................21 References............................................................................................................................21

ACKNOWLEDGEMENTS

This work has received research funding from the EU 6th Framework Programme, under contract number IST-2002-002034 (DIVINES -Diagnostic and Intrinsic Variabilities in Natural Speech). The views expressed here are those of the authors only. The Community is not liable for any use that may be made of the information contained therein.

Non-Native Speech Recognition 16 / 23

January 12, 2007 time on the development set), recognition performance improved on several language groups (e.g. Portuguese, Italian, African, Asian, …) compared to the usage of all available variants.

As summarized in left part of Figure 3 and in Table 2 3 , with such a selection, the recognition performance degradation on the French speakers was reduced, as compared to the usage of all acoustic variants. Though improvement on the language groups corresponding to the added units was a slightly smaller, better results were obtained on the other language groups.

ADAPTATION ON NON-NATIVE SPEECH

This section investigates the behavior of the various modeling approaches described before, when the acoustic models are adapted on non-native speech data. Let us recall that the adaptation set had roughly the same size as the test set and that the distribution of the speakers among the language groups was very similar to the one described for the test set in section 2.2.

All Accents

In a first set of experiments, the acoustic models were adapted using all the data in the adaptation set which exhibited similar non-native accents as those present in the test set. The resulting models were thus adapted to the task environment and non-native accents conditions.

The training scheme is represented in Figure 5. The top of the figure refers to the models of phonemes (French, English, Spanish and German languages) that were previously estimated on French or foreign data as described in section 3. The second line recalls the modeling variants (M1 to M5) that were considered, and the arrows indicate the units involved in each case. Every model (M1 to M5) was then adapted to the multiple foreign accent and task conditions using the adaptation set which contains the pronunciation of French words by French and foreign speakers. The acoustic parameters were adapted using an incremental enrollment procedure which is equivalent to segmental MAP adaptation with specific choice