Leonardo Tomazeli Duarte
email: leonardo.duarte@lis.inpg.fr

Christian Jutten
email: christian.jutten@inpg.fr

A MUTUAL INFORMATION MINIMIZATION APPROACH FOR A CLASS OF NONLINEAR RECURRENT SEPARATING SYSTEMS

In this work, we deal with nonlinear blind source separation. Our contribution is the derivation of a learning strategy that minimizes the mutual information between the outputs of a class of nonlinear recurrent separating systems. By using the concept of the differential of the mutual information, we obtain an algorithm that does not need a precise knowledge of the source distributions, in contrast to the one obtained by a direct derivation of the minimum mutual information framework, or equally the maximum likelihood approach, for the considered model. The validity of our approach is supported by simulations.

INTRODUCTION

The problem of blind source separation (BSS) concerns the retrieval of an unknown set of source signals by using only samples that are mixtures of these original signals. A myriad of methods were proposed for the case in which the mixing process is of linear nature [START_REF] Hyvärinen | Independent component analysis[END_REF]. The basis of the majority of these techniques is the independent component analysis (ICA) [START_REF] Comon | Independent component analysis, a new concept?[END_REF]. When the mixing system is nonlinear, the BSS problem becomes more difficult given that in this situation the recovery of the independence, which is the very essence of ICA, does not guarantee, as a rule, source separation. In view of this limitation, a more reasonable approach is to consider constrained mixing systems as, for example, postnonlinear (PNL) mixtures [START_REF] Taleb | Source separation in postnonlinear mixtures[END_REF] and linear-quadratic mixtures [START_REF] Hosseini | Blind separation of linear-quadratic mixtures of real sources using a recurrent structure[END_REF].

In this work, we deal with the problem of BSS in a particular class of nonlinear systems which is related to a chemical sensing application. Our approach is based on a recursive separating system and the main contribution of this work is the derivation of a learning rule based on the ICA paradigm of mutual information minimization. The paper is organized as follows: in Section 2, we begin with a description of the mixing model as well as of the chosen separating Leonardo Tomazeli Duarte would like to thank CNPq (Brazil) for the financial support. structure. In Section 3, we discuss the application of the minimum mutual information paradigm on the considered separating structure. Firstly, in subsection 3.1, we notice through an example that a mutual information-based algorithm resulted from a direct derivation may not be able to provide source separation. Then, in subsection 3.2, a learning rule based on the concept of the differential of the mutual information is proposed. In Section 4, simulations are carried out in order to verify the viability of our proposal. Finally, in Section 5, we state our conclusions and remarks.

PROBLEM STATEMENT

Blind source separation methods have been proved to be a very promising approach in chemical sensing applications. In [START_REF] Bermejo | Isfet source separation: foundations and techniques[END_REF][START_REF] Bedoya | Improving semiconductor-based chemical sensor arrays using advanced algorithms for blind source separation[END_REF], for instance, BSS algorithms were used in the problem of estimating the concentrations of several ions in a solution. A remarkable feature in this kind of application is that the sensors have a nonlinear response and, thus, one must resort to nonlinear source separation techniques. A first approach in this spirit was presented in [START_REF] Bedoya | Improving semiconductor-based chemical sensor arrays using advanced algorithms for blind source separation[END_REF], in which a post-nonlinear separation algorithm was applied on a set of mixtures provided by a sensor array of ion-sensitive fieldeffect transistors (ISFET). Despite the good results obtained by this proposal, it is limited to situations in which all the ions in the solution have the same valence.

In the present work, we envisage the situation in which the valences are different. We take as our starting point the Nikolsky-Eisenman model [START_REF] Bermejo | Isfet source separation: foundations and techniques[END_REF], which gives a simple and yet adequate description of potentiometric-based ion concentration sensors, as the ISFET ones. According to this model, the response of the i-th ISFET sensor is given by:

x i = c i1 + c i2 log s i + j,j =i a ij s z i z j j , (1)
where s i and s j are, respectively, the concentration of the ion of interest and the concentration of the j-th interfering ion; and where z i and z j denote the valences of the ions i and j, respectively. The selective coefficients a ij model the interference process; c i1 and c i2 are constants that depends on some physical parameters.

Assuming that c i1 and c i2 are previously estimated and considering a mixture of two ions, the model (1) can be simplified to the following one that will be considered in this work

x 1 = s 1 + a 12 s k 2 x 2 = s 2 + a 21 s 1 k 1 , (2)
where x i and s j denote the i-th mixture and the j-th source, respectively. In the context of this application, k is related to the ratio of the valences of the two considered ions.

Adopted separating system

For separating sources s i from mixtures (2), we adopt the following recurrent network as separating system:

y 1 (n + 1) = x 1 -w 12 y 2 (n) k y 2 (n + 1) = x 2 -w 21 y 1 (n) 1 k , (3)
where the vector w = [w 12 w 21 ] T denotes the parameters to be adjusted and y(n) = [y 1 (n) y 2 (n)] T corresponds to the separating system outputs at time n. For each sample of the mixtures, and for a given value w, the system outputs are obtained after the convergence of the dynamics (3).

In order to understand how this structure works, let s = [s 1 s 2 ] T denote a sample of the sources. By considering (2), one can easily check that when [w 12 w 21 ] T = [a 12 a 21 ] T , then y(n + 1) = y(n) = s correspond to an equilibrium point of [START_REF] Taleb | Source separation in postnonlinear mixtures[END_REF], that is to say, it becomes possible to counterbalance the action of the mixing system without relying on its direct inversion. This sort of approach was firstly developed in [START_REF] Jutten | Blind separation of sources, part i: an adapative algorithm based on neuromimetic architecture[END_REF] regarding linear BSS. Its extension to the problem of nonlinear BSS was proposed by Hosseini and Deville [START_REF] Hosseini | Blind separation of linear-quadratic mixtures of real sources using a recurrent structure[END_REF] in the context of source separation of linear-quadratic mixtures.

Given that the adopted separating system corresponds to a nonlinear dynamic, a crucial point concerns the stability of the separating solution y = s. In [START_REF] Duarte | Blind source separation of a class of nonlinear mixtures[END_REF], it was shown that, for each sample of the source, a necessary stability condition is given by:

|a 12 a 21 s ( 1 k -1) 1 s k-1 2 | < 1. (4)
In this same work, we described a ICA learning rule for (3) based on the cancellation of the nonlinear correlations between their outputs. The main advantage of this technique relies on its simplicity, given that only the estimation of higher-order (4th) moments is required. On the other hand, there are convergence problems which are exacerbated when the sources are close to the stability boundary. Aiming to obtain a more robust learning algorithm, we discuss, in the next section, the paradigm of mutual information minimization for the considered mixing model.

MUTUAL INFORMATION MINIMIZATION APPROACH

Firstly, let us recall the definition of mutual information, here expressed for the outputs of ( 3)

I(y) = H(y 1 ) + H(y 2 ) -H(y 1 , y 2 ), (5)
where H(•) denotes the differential entropy [START_REF] Cover | Elements of Information Theory[END_REF]. A common trick to avoid the estimation of H(y 1 , y 2 ) consists in expressing it as a function of the joint entropy of the mixtures. Thus, considering that the mapping provided by (3) after the convergence is invertible in the region containing the mixed signals and applying the entropy transformation law [START_REF] Hyvärinen | Independent component analysis[END_REF], one obtains:

I(y) = H(y 1 ) + H(y 2 ) -H(x) -E{log(| det J|)}, ( 6
)
where J is the Jacobian matrix associated to the separating system mapping. Since we are interested in the minimization of ( 6) with respect to w and given that H(x) does not depend on these parameters, then we may adopt the following expression as a separating criterion 1

C(y) = H(y 1 ) + H(y 2 ) -E{log(| det J|)}. (7)
In appendix 6.1, we derive the gradient of ( 7) with respect to w, which is given by:

∂C(y) ∂w = E det J ψ(y 1 )(-y k 2 ) + ψ(y 1 )(ka 12 y 1 k 1 y k-1 2 ) + ψ(y 2 )( 1 k a 21 y 1 k -1 1 y k 2 ) -a 21 y 1 k -1 1 y k-1 2 ψ(y 2 )(-y 1 k 1 ) -a 12 y 1 k -1 1 y k-1 2 (8)
where ψ(y i ) = -log(p(y i )) ′ denotes the marginal score function of y i . From this expression, one obtains the following learning rule that would minimize the mutual information:

w ← w -µ C(y) ∂w , (9)
where µ is the learning rate. Expression (8) could equally be derived through the maximal likelihood framework (see [START_REF] Hosseini | Blind maximum likelihood separation of a linear-quadratic mixture[END_REF], for instance). Usually in this approach, the true distributions of the sources are approximated by previously defined nonlinear functions [START_REF] Cardoso | Blind signal separation: statistical principles[END_REF] or by estimations obtained from the reconstructed sources provided by the separating system. It is well known that the separation task is accomplished in the linear case even when the sources distributions are roughly approximated [START_REF] Cardoso | Blind signal separation: statistical principles[END_REF]. However, this is not true for nonlinear mixing systems. Indeed, 1 In a nonlinear context, although equivalent in a theoretical standpoint, expressions ( 6) and ( 7) may lead to different practical algorithms due to the different statistical properties of these two estimators, as demonstrated for the case of PNL models in [START_REF] Achard | Criteria based on mutual information minimization for blind source separation in post nonlinear mixtures[END_REF]. we present, in the next section, an example that indicates that for the mixing system (2), even a satisfactory estimation of the score functions in (8) may not result in source separation.

Example of the influence of the score functions estimations on the performance of (9)

Let us consider a scenario with two sources uniformly distributed between [0.1, 1.1], k = 2 and mixing coefficients given by a 12 = a 21 = 0.5. This set of parameters satisfies the structural stability condition given by [START_REF] Hosseini | Blind separation of linear-quadratic mixtures of real sources using a recurrent structure[END_REF]. As it will be clarified in the sequel, it is useful to describe the pdf of each uniform source as

p(s i ) = 1 2 lim b→∞ tanh(b(s i -0.1)) -tanh(b(s i -1.1)).
(10) Therefore, it is not difficult to show that the score function of s i is given by:

ψ(s i ) = lim b→∞ b tanh 2 (b(s i -1.1)) -b tanh 2 (b(s i -0.1)) tanh(b(s i -0.1)) -tanh(b(s i -1.1)
) .

(11) In fact, this limit tends to a sum of two Dirac delta functions placed at 0.1 and 1.1, respectively.

In order to verify the influence of the choice of the score function on (9), we executed this learning rule considering different values for b in [START_REF] Hosseini | Blind maximum likelihood separation of a linear-quadratic mixture[END_REF]. Also we considered the case in which the score functions are estimated from the reconstructed sources during the adaptation stage by using two distinct methods: a kernel-based approach [START_REF] Pham | Fast algorithm for estimating mutual information, entropies and score functions[END_REF] and a MSE estimation approach [3] using a fourth-order polynomial as a parametric structure. For matter of illustration, we present, in Fig. 1, the resulting functions [START_REF] Hosseini | Blind maximum likelihood separation of a linear-quadratic mixture[END_REF] for several values of b as well as the source score function estimated with the kernel and MSE methods. In all executions, we considered as initial conditions for w the theoretical solution of the problem perturbed by a small random vector. A typical evolution of a performance index (see (18) below) for each approach is depicted in Fig. 2. It is clear that there is a performance degradation as b decreases. When b = 5 (not depicted in the figure), for example, the adaptation goes toward a instable region with respect to the separating system. Also, it is worth special attention that, for this scenario, even the application of a satisfactory score function estimation method, such as the kernel-based one, could not be able to maintain the good initial solution.

An approach based on the notion of the mutual information differential

As discussed in the above example, a reasonable performance of the mutual information approach (or equally the maximum likelihood one) based on the minimization of ( 7) is depending on a very accurate knowledge of the source score functions. Evidently, this demand is a very contrasting one to the idea of blind source separation. In the sequel, aiming to overcome this requirement, we shall develop an alternative rule that directly minimizes [START_REF] Bedoya | Improving semiconductor-based chemical sensor arrays using advanced algorithms for blind source separation[END_REF]. By proceeding in this way, one obtains a learning algorithm that is based on the so-called score function difference vector (see [START_REF] Pham | Fast algorithm for estimating mutual information, entropies and score functions[END_REF] below).

The cornerstone of our approach is the notion of the differential of the mutual information, proposed in [START_REF] Babaie-Zadeh | Differential of the mutual information[END_REF]. In this work, it was proved that a small variation ∆ of a given random vector y results, up to higher-order terms (expressed by o(∆)), in the following variation of the mutual information

I(y + ∆) -I(y) = E{∆ T β y (y)} + o(∆), (12)
where β y (y) is the score function difference vector associated with the random variable y. In view of this result, it can be interpreted as the gradient of the mutual information with respect to y. In mathematical terms, the i-th element of β y (y) is given by

β yi (y i ) = - ∂ log p(y) ∂y i -- d log p(y i ) dy i , (13)
i.e. the difference between the i-th element of the joint score function of y and the marginal score function of y i . It is not difficult to prove that y have independent components if, and only if, β yi (y i ) = 0 for every i.

The result expressed in [START_REF] Cardoso | Blind signal separation: statistical principles[END_REF] provides the guidelines for the design of a learning algorithm according to the minimum mutual information idea. The first step is to determine how a small variation of the separating system parameters, denoted by the vector ∆w = [∆w 12 ∆w 21 ] T , affects their outputs. Then, the "gradient" of the mutual information with respect to w is estimated by [START_REF] Cardoso | Blind signal separation: statistical principles[END_REF], by substituting the calculated small variations of the outputs in this expression.

In order to determine the variation ∆y = [∆y 1 ∆y 2 ] T given a small variation ∆w, one may consider a linearized version of the separating system (3) with respect to w. Considering two sources, this is expressed by (15) As stated above, the score function difference β y (y) may be interpreted as the gradient of [START_REF] Cardoso | Blind signal separation: statistical principles[END_REF]. Hence, considering [START_REF] Cardoso | Blind signal separation: statistical principles[END_REF] and (15), one may argue that the "gradient" of the mutual information with respect to the parameters w is

∂I ∂w = E ∂y ∂w T β y (y) . (16)
Therefore, it expected that the following learning rule minimizes the mutual information between the reconstructed sources

w ← w -µE ∂y ∂w T β y (y) , ( 17
)
where µ denotes the learning rate.

EXPERIMENTAL RESULTS

In order to assess the performance of the learning algorithm (17), experiments were conducted for two scenarios: k = 2 and k = 3. In both cases, the efficacy of the obtained solutions was quantified according to the following index:

SN R i = 10 log E{s 2 i } E{(s i -y i ) 2 } . (18)
Thus, SN R = 0.5(SN R 1 +SN R 2 ) defines a global index.

Regarding the estimation of the score function difference vector, we considered the method proposed in [START_REF] Pham | Fast algorithm for estimating mutual information, entropies and score functions[END_REF]. In short, this is a kernel-based method which differs from the classical approaches in two points: the estimation is done over a regular grid and a cardinal spline is used as kernel function. As a consequence, one obtain a much faster algorithm than the classical kernel method.

First scenario -k = 2

In a first scenario, we consider the separation of two sources uniformly distributed between [0.1, 1.1]. The mixing parameters are given by a 12 = 0.5 and a 21 = 0.5. Concerning the separation system, a set of 2000 samples of the mixtures was considered. The number of iterations was 2000 with a learning rate µ = 0.01. The initial conditions of the dynamics (3) were chosen as [y 1 (1) y 2 (1)] T = [0 0] T . The results of this first case are expressed in the first row of Table 1.

Second scenario -k = 3

In this situation, the separation of two sources uniformly distributed between [0.2, 1.2] is conducted. The other parameters were defined in the same way as in the first scenario, except the learning step which is µ = 0.008. In this situation, there are some samples of the sources that do not satisfy the stability condition (4). As a consequence, the adopted separating system will never converge to these samples which means that they cannot be retrieved. Also, these samples act as a kind of noise in the adaptation stage, in the sense that even for an ideal separating system, there are outputs that are still mixtures of the sources which may disturb the separating criteria.

Regardless the problem discussed above, our proposal does well in this scenario as can be noticed in the second row of Table 1. The high standard deviation in this case is due to the fact that the algorithm did not converge in 2 among the 50 experiments. In Fig. 3, the joint distributions of the mixtures and of the retrieved signals are depicted for a typical case. Note that the outputs of the separating system are almost uniformly distributed, which indicates that the separation task was fulfilled.

CONCLUSIONS

The objective of this work was to develop a mutual information framework for the adjustment of a class of recurrent separating systems. In a first moment, we showed, through an example, that the minimization of (7) leads to a learning rule that may be very dependent on an extremely accurate score function estimation stage. Then, an alternative method based on the notion of the differential of the mutual information was proposed. The obtained experimental results confirmed the efficacy of our proposal.

A first perspective of this work would be the application of the derived method on a real problem of chemical sensing. Indeed, in such an application, there are some informations, such as the positivity of the sources, that could be taken into account in the design of a separating method for the most general case of the Nikolsky-Eisenman model expressed in [START_REF] Hyvärinen | Independent component analysis[END_REF].

Also, there are two other interesting subjects for future work: 1) a theoretical analysis of the experimental result presented in section (3.1), and 2) an investigation of the cases for which our proposal did not converge. Despite the fact that just a very small number of trials did not converge, this problem may be exacerbated in the generalization of our method to scenarios with a greater number of sources.

APPENDIX

Derivation of (8)

Taking the derivative of ( 7) with respect to w, one obtains:

∂C(y) ∂w = ∂H(y 1 ) ∂w + ∂H(y 2 ) ∂w -E 1 det J ∂ det J ∂w . (19) As proved in [3] ∂H(y i ) ∂w = E ψ(y i ) ∂y i ∂w , (20)
where ψ(y i ) denotes the score function of y 1 , that is to say:

ψ(y i ) = - d log p(y i ) dy i = - p ′ (y i ) p(y i ) . (21)
By substituting (20) in ( 19), one obtains

∂C(y) ∂w = E ψ(y 1 ) ∂y 1 ∂w + E ψ(y 2 ) ∂y 2 ∂w -E 1 det(J) ∂ det(J) ∂w . (22)
In appendix 6.2, the expressions for ∂yi ∂w are determined The next step is to calculate the derivative of the Jacobian determinant associated with the mapping performed by the separating system. For a given value of w, the separating system system outputs satisfy the following equation when (3) converges

x 1 = y 1 + w 12 y k 2 x 2 = y 2 + w 21 y 1 k 1 . (23)
In fact, this is the inverse mapping with relation to the separating system. Therefore, it is possible to calculate J by determining the Jacobian of this inverse mapping, which will be denoted by J ′ . Obviously, there is a tacit assumption in this procedure, i.e., we assume that the mapping is invertible in the domain of the mixtures.

(23), it straightforward to show that

J ′ = ∂x ∂y = 1 kw 12 y k-1 2 1 k w 21 y 1 k -1 1 1 . ( 24
)
Given that J = J ′ -1 , then the following expression holds det(J) = 1 det(J ′ ) = 1

1w 12 w 21 y

1 k -1 1 y k-1 2 . ( 25
)
From this expression, it is straightforward to show that .

(

) 27
After the convergence of (3), the mapping performed by the separating system is given by (23). Therefore, the derivatives in (26) can be calculated by applying the chain rule property on (23). For instance, it is not difficult to verify from that:

∂y 1 ∂w 12 = -(y k 2 + w 12 ky k-1 2 ∂y 2 ∂w 12 ). (28)
Given that

∂y 2 ∂w 12 = - 1 k w 21 y 1 k -1 1 ∂y 1 ∂w 12 , (29)
and substituting this expression in (28), one obtains:

Fig. 1 .

 1 Fig. 1. Score functions.

Fig. 2 .

 2 Fig. 2. Evolution of SNR for distinct score function estimation approaches.

  the appendix 6.2 the elements of ∂y ∂w are determined. If one considers in (12) that ∆ = ∆y, then one readily obtains that: I(y + ∆y) -I(y) = E ∆w T ∂y ∂w T β y (y) + o(∆y).

Fig. 3 .

 3 Fig. 3. Second scenario -k = 3.

2 . 6 . 2 .

 262 (26) Hence, considering (22), (26) and the results derived in the appendix 6.2, expression (8) is obtained. Calculation of ∂y ∂w In this appendix, we are interested in the determination of

∂y 1 ∂w 12 = -y k 2 1 -w 12 w 21 y 1 k -1 1 y k-1 2 . 2 k( 1 -w 12 w 21 y 1 k -1 1 y k-1 2 )

 12211221112

Table 1 .

 1 Average SNR results over 50 experiments and standard deviation (STD).

	SN R 1 SN R 2 SN R ST D(SN R)
	k = 2 44.40	40.92 42.66	7.71
	k = 3 39.41	36.40 37.90	14.18

k


Sarah Samson

Laurent Besacier
email: laurent.besacier@imag.fr

Benjamin Lecouteux
email: benjamin.lecouteux@imag.fr

Tien-Ping Tan
email: tienping@cs.usm.my

Merging of Native and Non-native Speech for Low-resource Accented ASR

Keywords: automatic speech recognition, cross-lingual acoustic modelling, non-native speech, low-resource system, multi-accent SGMM, accent-specific DNN

This paper presents our recent study on low-resource automatic speech recognition (ASR) system with accented speech. We propose multi-accent Subspace Gaussian Mixture Models (SGMM) and accent-specific Deep Neural Networks (DNN) for improving non-native ASR performance. In the SGMM framework, we present an original language weighting strategy to merge the globally shared parameters of two models based on native and non-native speech respectively. In the DNN framework, a native deep neural net is fine-tuned to non-native speech. Over the non-native baseline, we achieved relative improvement of 15% for multi-accent SGMM and 34% for accent-specific DNN with speaker adaptation.

Introduction

Performance of non-native automatic speech recognition (ASR) is poor when few (or no) non-native speech is available for training / adaptation. Many approaches have been suggested for handling accented-speech in ASR, such as acoustic model merging ( [START_REF] Morgan | Making a Speech Recognizer Tolerate Non-native Speech through Gaussian Mixture Merging[END_REF], [START_REF] Bouselmi | Fully Automated Non-native Speech Recognition using Confusion-based Acoustic Model Intergration[END_REF], [START_REF] Tan | Acoustic Model Interpolation for Non-native Speech Recognition[END_REF], [START_REF] Tan | Acoustic Model Merging using Acoustic Models from Multilingual Speakers for Automatic Speech Recognition[END_REF]), applying maximum likelihood linear regression (MLLR) for adapting models to each non-native speaker [START_REF] Huang | Accent Modeling based on Pronunciation Dictionary Adaptation for Large Vocabulary Mandarin Speech Recognition[END_REF], or adapting lexicon ( [START_REF] Arslan | A Study of the Temporal Features and Frequency Characteristics in American English Foreign Accent[END_REF], [START_REF] Goronzy | Robust Adaptation to Non-native Accents in Automatic Speech Recognition[END_REF]).

Lately, Subspace Gaussian Mixture Models (SGMMs) ( [START_REF] Povey | Subspace Gaussian Mixture Models for Speech Recognition[END_REF], [START_REF] Povey | The Subspace Gaussian Mixture Model -a Structured Model for Speech recognition[END_REF]) have shown to be very promising for ASR in limited training conditions (see [START_REF] Lu | Cross-lingual Subspace Gaussian Mixture Models for Low-resource Speech recognition[END_REF] and [START_REF] Imseng | Using Out-of-language Data to Improve Under-resourced Speech recognizer[END_REF]). In SGMM modelling, the acoustic units are all derived from a common GMM called Universal Background Model (UBM). This UBM, which in some way represents the acoustic space of the training data, can be estimated on large amount of untranscribed data from one or several languages. The globally shared parameters do not need the knowledge about the phone units used in the source language(s). Without this constraint of source-target mapping of acoustic units, the UBM can be well used in cross-lingual or multilingual (multi-accent) settings.

In the mean time, Deep Neural Networks (DNNs) have been increasingly employed for building efficient ASR systems. HMM/DNN hybrid systems clearly outperform HMM/(S)GMM systems for many ASR tasks [START_REF] Hinton | Deep Neural Networks for Acoustic Modeling in Speech recognition[END_REF] which include dealing with low-resource systems ( [START_REF] Miao | Improving Low-resource CD-DNN-HMM using Dropout and Multilingual DNN Training[END_REF], [START_REF] Vu | Multilingual Deep Neural Network based Acoustic Modeling for Rapid Language Adaptation[END_REF], [START_REF] Huang | Cross-language Knowledge Transfer using Multilingual Deep Neural Network with Shared Hidden Layers[END_REF]). Several studies have shown that multilingual DNNs can be achieved by utilizing multilingual data for conducting unsupervised pretraining [START_REF] Swietojanski | Unsupervised Cross-lingual Knowledge Transfer in DNN-based LVCSR[END_REF] or training the whole network simultaneously ( [START_REF] Huang | Cross-language Knowledge Transfer using Multilingual Deep Neural Network with Shared Hidden Layers[END_REF], [START_REF] Vu | Multilingual Deep Neural Network based Acoustic Modeling for Rapid Language Adaptation[END_REF], [START_REF] Heigold | Multilingual Acoustic Models using Distributed Deep Neural Networks[END_REF]).

In the above techniques, acoustic model merging can easily be conducted through sharing the UBMs (for SGMM) and hidden layers (for DNN) with other systems. But what is the optimal way to do so? Can we merge a large amount of native speech with a small quantity of non-native data? This paper tries to respond to these questions using both SGMM (less efficient than DNNs but more compact for embedded applications) and DNN (state-of-the-art) frameworks. We apply our methods to Malaysian English ASR, where a large amount of native (English) data is available (TED-LIUM corpus [START_REF] Rousseau | TED-LIUM: An Automatic Speech Recognition Dedicated Corpus[END_REF]), while only 2h of non-native speech is available. More precisely, we propose one strategy for each framework:

(1) language weighting for multi-accent SGMMs and (2) accent-specific top layer for DNN. The first strategy is novel and involves manipulating the number of Gaussians of each native / non-native model for (multi-accent) UBM merging. In the second approach, we build accent-specific DNN similarly to last year's work of [START_REF] Huang | Multi-accent Deep Neural Network Acoustic Model with Accent-specific Top Layer using the KLD-regularized Model Adaptation[END_REF] but we make it work for a very low-resource setting and with speaker adaptation on top of it.

The rest of the paper is organized as follows. In Section 2 we describe the background of SGMM and DNN as well as their application to multilingual and multi-accent ASR. Section 3 presents the experimental setup for building native and non-native systems as well as the results of our baselines. In Section 4 and 5, we describe the proposed strategies and show their benefits to low-resource accented ASR. Last but not least, Section 6 concludes this paper.

2 Background of acoustic modelling for cross-lingual or accented ASR

Subspace Gaussian Mixture Models

The GMM and SGMM acoustic models are similar since each emission probability of each HMM state is modelled with a Gaussian mixture model. However, in the SGMM approach, the Gaussian means and mixture component weights are generated from the phonetic and speaker subspaces along with a set of weight projections. For SGMM, the state probabilities are defined following the equations below [START_REF] Povey | The Subspace Gaussian Mixture Model -a Structured Model for Speech recognition[END_REF]:

p(x|j) = Mj m=1 c jm I i=1 w jmi N (x; µ jmi , Σ i ) (1)
µ jmi = M i v jm , (2)
w jmi = exp w T i v jm I i =1 exp w T i v jm (3)
where x ∈ R D denotes the D-dimensional feature vector, j ∈ {1..J} is the HMM state, i is the Gaussian index, m is the substate and c jm is the substate weight. Each state j is associated to a vector v jm ∈ R S (S is the phonetic subspace dimension) which derives the means, µ jmi and mixture weights, w jmi , I is the number of Gaussians for each state. The phonetic subspace M i , weight projections w T i and covariance matrices Σ i , i.e., the globally shared parameters Φ i = {M i , w T i , Σ i } are common across all states. These parameters can be shared and estimated over multiple language data. [START_REF] Lu | Cross-lingual Subspace Gaussian Mixture Models for Low-resource Speech recognition[END_REF] and [START_REF] Imseng | Using Out-of-language Data to Improve Under-resourced Speech recognizer[END_REF] presented cross-lingual and multilingual work using SGMM for improving ASR with very limited training data. In both studies, the cross-lingual approach was carried out by porting the UBM which was trained using source language data, to SGMM training of target language. Basically, the SGMM model was derived from the UBM of source language. For the second approach, the strategy involved training UBM using more than one language data and then employed the multilingual UBM for SGMM training of a specific language. Applying both methods improved ASR performance of monolingual system.

This idea motivates us to investigate a multi-accent approach using this framework. We propose to build SGMM models which are derived from merged UBMs, rather than carrying out the SGMM training in a multilingual fashion (see studies on non-native SGMM in [START_REF] Mohan | Dealing with Acoustic Mismatch for Training Multlingual Subspace Gaussian Mixture Models for Speech Recognition[END_REF], [START_REF] Tong | Subspace Gaussian Mixture Models for Computer-assisted Language Learning[END_REF]). The method is particularly appealing if one wishes to consider borrowing UBMs of other systems. Our strategy and experiments are described in Section 4.

Deep Neural Networks

Deep Neural Network (DNN) for ASR is a feedforward neural network with hidden layers. Mathematically, each output (node) of the l-th layer of a DNN can be defined as

x l = σ(b l + W l x l-1 ), f or 1 ≤ l < L (4)
where W l is the connection weight from x l and x l-1 , the output of the (l -1)-th layer, while b l is the bias. The hidden output x l is a sigmoid function defined as σ(x) = (1 + exp(-x)) -1 . The last (L-th) layer of the DNN uses a softmax function to obtain the posterior probability of each HMM state j given the acoustic observation o t at time t:

p(j|o t ) = exp(x L ) Σ j exp(x L ) . (5)
Optimizing hidden layers can be done by pretraining the network using Restricted Boltzmann Machines (RBM) [START_REF] Hinton | A Practical Guide to Training Restricted Boltzmann Machines[END_REF]. The generative pretraining strategy builds stacks of RBMs corresponding to the number of desired hidden layers and provides better starting point (weights) for DNN fine-tuning through backpropagation algorithm. Pretraining a DNN can be carried out in a unsupervised manner because it does not involve specific knowledge (labels, phone set) of a target language1 . Only the softmax layer is sensitive to the target language. It is added on top of the hidden layers during fine-tuning and its ouput corresponds to the HMM states of the target language.

As shown in [START_REF] Swietojanski | Unsupervised Cross-lingual Knowledge Transfer in DNN-based LVCSR[END_REF], using untranscribed data for RBM pretraining as a multilingual strategy has little effect on improving monolingual ASR performance. The transfer learning [START_REF] Heigold | Multilingual Acoustic Models using Distributed Deep Neural Networks[END_REF] approach has shown large recognition accuracy improvements. The method involves removing the top layer of a multilingual DNN and fine-tuning the hidden layers to a specific language.

Recently, a multi-accent DNN with accent specific softmax layer has been proposed for improving decoding performance of English ASR for British and Indian accents [START_REF] Huang | Multi-accent Deep Neural Network Acoustic Model with Accent-specific Top Layer using the KLD-regularized Model Adaptation[END_REF]. The accent adaptation approach yielded better decoding results compared to non-adapted DNNs. Another attempt to improve ASR performance on non-native task was done by [START_REF] Chen | Deep Neural Network Acoustic Modeling for Native and Non-native Mandarin Speech Recognition[END_REF] for Mandarin language. They also proved the interest of adapting non-native accents over the baseline DNN model.

In this paper, we investigate a method similar to [START_REF] Huang | Multi-accent Deep Neural Network Acoustic Model with Accent-specific Top Layer using the KLD-regularized Model Adaptation[END_REF] to build the accent-specific network models, but we apply it in a very low-resource setting. Previously, the method has been tested with larger amount of non-native speech (x10 or x100 compared to our experimental conditions). Hence, we try to measure the effectiveness of the approach when the ratio between non-native data and native data is largely unbalanced. In addition, we develop a strategy to handle cross-lingual DNNs with different feature transforms for speaker adaptation.

Experimental Setup

The ASR experiments were conducted on Kaldi speech recognition toolkit [START_REF] Povey | The Kaldi Speech Recognition Toolkit[END_REF]. This section reports non-native and native speech databases used in our investigation. Besides that, we present the baseline results for non-native ASR based on GMM, SGMM and DNN.

Data

The non-native speech corpus contains 15h of English speech spoken by 24 Malaysians (of Malay, Chinese and Indian origin). The data were collected by Universiti Sains Malaysia for conducting research on acoustic model merging for ASR (see [START_REF] Tan | Acoustic Model Merging using Acoustic Models from Multilingual Speakers for Automatic Speech Recognition[END_REF] for more details). Table 1 shows the amount of data used to train and evaluate the non-native ASR. We employed 2h of transcribed data for training the system and evaluate its performance on 4h of transcribed speech. For SGMM training, 9h of untranscribed data were added to the 2h of transcribed speech to build the UBM. Our system used the CMU pronunciation dictionary (no non-native adaptation of the lexicon) which has more than 100k words. Furthermore, we used a trigram language model for decoding. The model was trained on news data, taken from a local English news website2 . After evaluating the LM on the test transcription data, the LM perplexity is 189 while the OOV rate is 2.5%. To obtain a baseline for native ASR, we used the first release of TED-LIUM [20] corpus 3 . The transcriptions of this corpus were generated by the Laboratoire d'Informatique at Université du Maine (LIUM) for the International Workshop on Spoken Language Translation (IWSLT) evaluation campaign in 2011. The corpus contains speeches that were excerpted from video talks of the TED website. We used 118h to train the system and 4h for evaluation. Besides that, we used a pronunciation dictionary which was included in the package. For decoding, we used a trigram language model which was built on TED and WMT11 (Workshop on Machine Translation 2011) data. The model perplexity is 220 after estimation on the test data.

Baseline systems

For the non-native ASR system, we trained a triphone acoustic model (39 MFCC with deltas and deltas deltas) using 776 states and 10K Gaussians. Then, we trained SGMM using the same decision trees as in the previous system. The SGMM was derived from a UBM with 500 Gaussians and phonetic subspace dimension was S = 40. The UBM was trained on 11h data. We built a DNN based on state-level minimum Bayes risk [START_REF] Kingsbury | Lattice-based Optimization of Sequence Classification Criteria for Neural Network Acoustic Modeling[END_REF] (sMBR) and the network had 7 layers, each of the 6 hidden layers had 1024 hidden units. The network was trained from 11 consecutive frames (5 preceding and 5 following frames) of the same MFFCs as in the GMM system. Besides that, the same HMM states were used as targets of the DNN. The initial weights for the network were obtained using Restricted Boltzmann Machines (RBMs) that resulted in a deep belief network with 6 stacks of RBMs. Fine tuning was done using Stochastic Gradient Descent with per-utterance updates, and learning rate 0.00001 which was kept constant for 4 epochs. To run our DNN experiments, we utilized a GPU machine and CUDA toolkit to speed up the computations.

For the native ASR system, we built a triphone acoustic model with 3304 states and 40K Gaussians. Subsequently, we built SGMM system using the same decision trees and 500 UBM Gaussians. Lastly, we trained a DNN with 7 layers using the same setting for building non-native DNN. The three systems were evaluated on native speech (TED task) and we achieved the following WER results: 30.55% for GMM, 28.05% for SGMM and 19.10% for DNN. Table 2 presents the baseline results of systems that used non-native and native acoustic models, evaluated on accented speech. For non-native acoustic modelling, SGMM and DNN systems outperformed the GMM system. The systems gave 3% and 22% relative improvement, respectively. Using these non-native models (trained on 2h only!) to decode non-native speech resulted lower word error rate (WER) compared to the pure native ASR systems (trained on 118h). In the following sections, we try to take advantage of both corpora (large native and small non-native) by merging acoustic models (or data) efficiently.

Language weighting for multi-accent Subspace

Gaussian Mixture Models

Proposed Method

In SGMM, the system is initialized by a Universal Background Model (UBM) which is a mixture of full-covariance Gaussians. This single GMM is trained on all speech classes that are pooled together. The advantage of this model is that it can be trained on large amount of untranscribed data or multiple languages, as shown in [START_REF] Lu | Cross-lingual Subspace Gaussian Mixture Models for Low-resource Speech recognition[END_REF] for cross-lingual SGMM in low-resource conditions. The authors showed that the SGMM global parameters are transferable between languages, especially when the parameters are trained in multilingual fashion. Thus, this gives an opportunity for low-resource systems to borrow UBM trained from other sources. Figure 1 illustrates the process of UBM merging through language weighting. The first step is to choose a language weight, α to L 1 in order to determine the number of Gaussians to be kept for merging ((1 -α) is given to L 2 ). Intuitively, a larger α should be given to the less represented source data. Then, we use data that are representative of the ASR task in order to find the top αN Gaussians in L 1 UBM using maximum likelihood criterion. The same process is done for the L 2 UBM but only (1-α)N Gaussians are selected. The final step applies weight normalization before merging all the Gaussians in a single GMM. The final UBM should have the same number of Gaussians if both initial UBMs are the same size. For experiments, we built a multi-accent UBM by merging native and non-native models using our language weighting strategy. To implement this, we used UBM of native speech (trained on 118h) and UBM of non-native speech (trained on 11h). Each of the UBMs has 500 Gaussians. Using the two models, we employed the language weighting approach for obtaining several multi-accent UBMs. Thereafter, these UBMs were used to estimate the parameters of non-native SGMM systems. Subsequently, we trained multi-accent SGMMs with different numbers of substates, ranging from 800 to 8750. By doing this, we obtained several SGMMs for each multi-accent UBM applied. We summarize our results by reporting only the highest (maximum), average and best (minimum) SGMM results, as shown in Figure 2.

Results

Our findings show that using the proposed strategy resulted in significant improvement from the SGMM baseline. We reach the lowest WER when the SGMM system was obtained from a multi-accent UBM with 250 Gaussians from native and 250 Gaussians from non-native (α = 0.5, WER=37.71%). This result proves that carefully controlling the contribution of two (unbalanced) data as sources for UBM training is a way to optimize ASR performance. In this experiment, the optimal α obtained tells us that non-native (Malaysian) data (in small quantity but very representative of the ASR task) and native (TED-LIUM) data (bigger corpus with speaker diversity) contribute equally to the acoustic space representation.

Furthermore, we did not gain WER improvements when the amount of substates increased. The minimum WERs shown in the figure are results for SGMMs with 800 substates. We extended our investigation to evaluate ASR performance for very compact (smaller number of Gaussians) UBM. The UBM was built with only 50 Gaussians using native/non-native data and then we applied the same language weighting strategy to obtain multi-accent UBMs. The non-native system significantly improved after applying this method. Table 3 shows the comparison between multi-accent SGMM with UBM=500 and UBM=50. For α = 0.5, the new multi-accent SGMM outperformed the one with more UBM Gaussians by 9% relative improvement on the WER. The result shows that deriving SGMM from a compact UBM gives better performance in very low-resource conditions. We also tried even smaller UBM but the WERs started to go back up (39.85% for UBM with 5 Gaussians!).

L 1 0 . 9 L 1 0 . 8 L 1 0 . 7 L 1 0 . 6 L 1 0 . 5 L 1 0 . 4 L 1 0 . 3 L 1 0 . 2 L 1 0 .
5 Accent-specific top layer for DNN

Proposed Method

Figure 3 illustrates the training process for obtaining an accent-specific DNN. We began with a network that was fine-tuned on native speech (last line and last column in Table 2). Then, we removed the softmax layer of native (source) DNN. Subsequently, a new softmax layer was added through fine-tuning the whole network on the non-native (target) training data. For this condition, we built the DNN on the GMM baseline for non-native. We also built a second system for evaluating this approach. The system was speaker adapted and built upon new HMM/GMM acoustic models. First, we trained new native and non-native triphone models on new feature vectors using linear discriminant analysis (LDA) and maximum likelihood linear transform (MLLT), as well as speaker adaptive training using feature-space maximum likelihood linear regression (fMLLR). One important trick is to use feature transforms that were acquired from the native corpus (with large number of speakers), during LDA+MLLT training of non-native system. If not done this way, we observed no improvement with speaker adaptation (merging non-native and native DNNs with different feature transforms is not good). Then, we trained DNN for native and later we removed the top layer of the model. Subsequently, we fine-tuned the remaining DNN layers on the non-native data.

Results

We tested the DNNs on the same non-native evaluation data (4h test). Table 4 presents our findings. Both results are significantly better than the pure non-native DNN baseline (last line in Table 2). For example, we achieved 24% and 34% relative improvement respectively over the non-native DNN baseline (32.52%). Thus, the hidden layers of the native DNN proved to be useful for improving the low-resource non-native ASR. Besides that, our approach for building DNN with speaker adaptation and accent-specific top layer provided the best result. We obtained 14% relative improvement over the accent-specific DNN without speaker adaptation.

Conclusions

We have proposed two approaches for optimal merging of native and non-native data in order to improve accented ASR with limited training data. The first approach introduced a language weighting strategy for constructing multi-accent compact SGMM acoustic models. In this approach, we used language weights to control the number of Gaussians of each UBM involved in the merging process. Improvement of the ASR performance was observed with language weighting. The second approach involved fine-tuning the hidden layers of native DNN on the non-native training data. We applied this approach for obtaining accent-specific DNN with and without speaker adaptation. For the former, we trained the DNN on HMM/GMMs that had feature transforms of the native speech data. Both DNNs outperformed the DNN baseline. Overall, the approaches used in this study resulted in encouraging improvement in WER. Over the non-native baseline, we achieved relative improvement of 15% for SGMM (multi-accent UBM50) and 34% for DNN (accent-specific with speaker adaptation).

Fig. 1 .

 1 Fig. 1. An Illustration of UBM merging through language weighting

Fig. 3 .

 3 Fig. 3. Process of obtaining accent-specific DNN (right) using hidden layers of native DNN

Table 1 .

 1 Statistics of the non-native speech data for ASR.

	Train		Test
	Untranscribed Transcribed
	9h	2h	4h

Table 2 .

 2 Word error rates (WER %) of ASR with non-native (2h) and native (118h) acoustic models on the non-native evaluation data (4h test) -same pronunciation dictionary and language model for both system.

	Acoustic Models Non-native Native
	GMM	41.47	57.09
	SGMM	40.41	45.84
	DNN	32.52	40.70

Table 3 .

 3 A summary of results from the SGMM experiments on the non-native ASR (4h test). Different UBMs were employed for building SGMM with 2h of non-native training data.

	SGMM	WER (%)
	Non-native UBM500	40.41 (baseline)
	Native UBM500	41.13
	For α = 0.5,
	a. Multi-accent UBM500	37.71
	b. Multi-accent UBM50	34.24

Table 4 .

 4 WERs of accent-specific DNN on the non-native ASR task (4h test).

	DNN with accent-specific top layer WER (%)
	a. No speaker adaptation	24.89
	b. Speaker adaptation	21.48

In that sense, RBM pretraining (for DNN) and UBM training (for SGMM) are both unsupervised methods to get an initial representation of the acoustic space before modelling the speech units

http://www.thestar.com.my/

We are aware that TED-LIUM is not a truly native English corpus (non-native speakers of multiple origins) but we consider here that the corpus permit to build an efficient system to decode native English ASR. Thus, in this paper we call it "excessively" a native corpus.


L Besacier

B Lecouteux

N Q Luong

N T Le

SPOKEN LANGUAGE TRANSLATION GRAPHS RE-DECODING USING AUTOMATIC QUALITY ASSESSMENT

Keywords: Quality estimation, Word confidence estimation (WCE), Spoken Language Translation (SLT), Search graph re-decoding

This paper investigates how automatic quality assessment of spoken language translation (SLT), also named confidence estimation (CE), can help re-decoding SLT output graphs and improve the overall speech translation performance. Our graph redecoding method can be seen as a second-pass of translation. For this, a robust word confidence estimator for SLT is required. We propose several estimators based on our estimation of transcription (ASR) quality, translation (MT) quality, or both (combined ASR+MT). Using these word confidence measures to re-decode the spoken language translation graph leads to a significant BLEU improvement (more than 2 points) compared to our SLT baseline, for a French-English SLT task. These results could be applied to interactive speech translation or computer-assisted translation of speeches and lectures.

INTRODUCTION

Automatic quality assessment of spoken language translation (SLT), also named confidence estimation (CE), is an important topic because it allows to know if a system produces (or not) user-acceptable outputs. In interactive speech to speech translation, CE helps to judge if a translated turn is uncertain (and ask the speaker to rephrase or repeat). For speech-to-text applications, CE may tell us if output translations are worth being corrected or if they require retranslation from scratch. Building a method which is capable of pointing out the correct parts as well as detecting the errors in a speech translated output is crucial to tackle above issues.

In this paper, we claim that an accurate CE can also help to improve SLT itself through a second-pass N-best list reranking or search graph re-decoding, as it has already been done for text translation by [START_REF] Bach | Goodness: A method for measuring machine translation confidence[END_REF] or more recently by [START_REF] Luong | Word Confidence Estimation for SMT Nbest List Re-ranking[END_REF].

Given signal x f in the source language, the spoken language translation (SLT) consists in finding the most probable target language sequence ê = (e 1 , e 2 , ..., e N ) so that ê = argmax e {p(e/x f , f )} [START_REF] Bach | Goodness: A method for measuring machine translation confidence[END_REF] where f = (f 1 , f 2 , ..., f M ) is the transcription of x f . Now, if we perform confidence estimation at the "words" level, the problem is called Word-level Confidence Estimation (WCE) and we can represent this information as a sequence q (same length N of ê) where q = (q 1 , q 2 , ..., q N ) and q i ∈ {good, bad} 1 .

Then, integrating automatic quality assessment in our SLT process can be done as following:

ê = argmax e q
p(e, q/x f , f )

ê = argmax e q p(q/x f , f, e) * p(e/x f , f )

ê ≈ argmax e {max q {p(q/x f , f, e) * p(e/x f , f )}} (4)

In the product of (4), the SLT component p(e/x f , f ) and the WCE component p(q/x f , f, e) contribute together to find the best translation output ê.

Contributions Following this formalisation, our paper investigates how WCE can help re-decoding SLT output graphs and improve the overall speech translation performance. Our graph redecoding method (discussed in section 2) can be seen as a second-pass of translation. One requirement for this is the availability of a robust confidence estimator based on both ASR and MT features. Our WCE system using such multiple features is discussed in section 3. The experiments described in sections 4 and 5 show how robust WCE helps re-decoding speech translation graphs leading to significant BLEU improvements (more than 2 points). An analysis of the SLT hypotheses with and w/o second pass (based on WCE) is also provided at the end of this paper.

SLT GRAPH RE-DECODING

Our approach

For practical implementation, we made the following choices for evaluating equation ( 4):

• We want to maximize the product of probabilities over all sequences q of quality tags. This would require applying WCE to the SLT graph but due to technical constraints, our WCE system (detailed later in the paper) can be only applied word sequences. Thus, for each sentence to translate, we tag N-best hypotheses from the SLT system in order to approximate the maximization over q.

• For the same reason, the use of WCE labels cannot be integrated directly into the MT decoder so we apply a second pass were labels related to word confidence prediction are integrated into the first-pass search graph (SG) to re-determine the best hypothesis. To do this, our intuition is that all parts of hypotheses corresponding to words labeled as good should be appreciated while those labeled as bad should be weakened.

Consequently, the additional steps (for the second pass) are the following:

• Firstly, apply a WCE classifier on the SLT N-best list to assign the quality labels (good or bad) along with the confidence probabilities for each word.

• Secondly, for each word in the N-best list, update the cost of all SG's hypotheses containing it by adding the updated score to their cost.

• Thirdly, search again on the updated SG for the cheapest-cost hypothesis and thus find the new best translation.

We assume that the decoder generates N best hypotheses e N = {e 1 , e 2 , ..., e N } at the end of the first pass. Using the WCE system, we are able to assign the j-th word in the hypothesis e i , denoted by e ij , with one appropriate quality label, q ij . Then, the second pass is carried out by considering every word e ij and its labels q ij . Our principal idea is that, if e ij is a correct translation, i.e. q ij = good, all hypotheses H k ∈ SG containing it in the SG should be "rewarded" by reducing their cost. On the contrary, those containing incorrect translation will be "penalized". Let reward(e ij ) and penalty(e ij ) denote the reward or penalty score of e ij , the new transition cost of H k after being updated is formally defined by:

transition (H k ) = transition(H k )+ reward(e ij ) if q ij = good penalty(e ij ) otherwise (5)
The update finishes when all words in the N-best list have been considered. We then re-compute the new score of complete hypotheses by tracing backward via back-pointers and aggregating the transition cost of all their edges. Essentially, the re-decoding pass reorders SG hypotheses following the rule: the more good words (predicted by WCE system) they contain, the more cost reduction will be made. In this work, the updated scores used are defined as follows:

penalty(e ij ) = -reward(e ij ) = β * score(H k ) #words(H k ) (6)
Where #words(H k ) is the number of target words in H k , the positive coefficient β accounts for the impact level of this score on the hypothesis's final cost and can be optimized during experiments (in this work, due to corpus constraints, we apply a cross-validation procedure where optimization is done on the first half of the test corpus and applied to the second half, and vice-versa). Here, penalty(e ij ) gets negative sign (since score(H k ) < 0) and will be added to the transition cost of all hypotheses containing e ij in case this word is labeled as bad; whereas reward(e ij ) (same value, opposite sign) is used in the other case.

BUILDING AN EFFICIENT QUALITY ASSESSMENT (WCE) SYSTEM

The WCE component solves the equation:

q = argmax q {p(q/x f , f, e)} (7)
where q = (q 1 , q 2 , ..., q N ) is the sequence of quality labels on the target language. This is a sequence labelling task that can be solved with several machine learning techniques such as Conditional Random Fields (CRF) [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting et labeling sequence data[END_REF]. However, for that, we need a large amount of training data for which a quadruplet (x f , f, e, q) is available.

While a corpus containing such a quadruplet was recently made available by [START_REF] Besacier | Word confidence estimation for speech translation[END_REF], it contains only 2.5k utterances which is not sufficient to accurately train the model.

Since this is much easier to obtain data containing either the triplet (x f , f, q) (automatically transcribed speech with manual references and quality labels infered from word error rate estimation) or the triplet (f, e, q) (automatically translated text with manual post-editions and quality labels infered using tools such as TERpA [START_REF] Snover | Terp system description[END_REF]) we can recast the WCE problem with the following equation :

q = argmax q {p ASR (q/x f , f ) α * p M T (q/e, f ) 1-α } (8)
where α is a weight giving more or less importance to W CE ASR (quality assesment on transcription) compared to W CE M T (quality assesment on translation). It is important to note that p ASR (q/x f , f ) corresponds to the quality estimation of the words in the target language based on features calculated on the source language (ASR). For that, what we do is projecting source quality labels to the target using wordalignment information between e and f sequences. Our two components W CE ASR and W CE M T are more precisley detailed in next subsections (French-English SLT task).

WCE for speech transcription

In this work, we extract several types of features, which come from the ASR graph, from language model scores and from a morphosyntactic analysis. These features are listed below (more details can be found in [START_REF] Besacier | Word confidence estimation for speech translation[END_REF]):

• Acoustic features: word duration (F-dur).

• Graph features (extracted from the ASR word confusion networks): number of alternative (F-alt) paths between two nodes; word posterior probability (F-post).

• Linguistic features (based on probabilities by the language model): word itself (F-word), 3-gram probability (F-3g), back-off level of the targeted word (F-back), as proposed in [START_REF] Fayolle | Crf-based combination of contextual features to improve a posteriori word-level confidence measures[END_REF],

• Lexical Features: words Part-Of-Speech (F-POS).

We use a variant of boosting classification algorithm in order to combine features. The used classifier is bonzaiboost [START_REF] Laurent | Boosting bonsai trees for efficient features combination : application to speaker role identification[END_REF]. It implements the boosting algorithm Adaboost.MH over deeper trees.

For each word, we estimate the 7 features (F-Word; F-3g; F-back; F-alt; F-post; F-dur; F-POS) previously described. The WCE estimator is trained on a separate french corpus (BREF 120 [START_REF] Lamel | Bref, a large vocabulary spoken corpus for french1[END_REF]) which was entirely transcribed automatically to obtain label examples (1M words with labels in total).

WCE for machine translation

We employ the Conditional Random Fields [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting et labeling sequence data[END_REF] (CRFs) as our machine learning method, with WAPITI toolkit [START_REF] Lavergne | Practical very large scale crfs[END_REF], to train the WCE estimator. A separate corpus of 10000 (frenchenglish) MT post-editions (see [START_REF] Potet | Collection of a large database of french-english smt output corrections[END_REF]) is used as training set.

The reason why we use different machine learning technique for confidence estimation in ASR (boosting) and in MT (CRFs) is due to the fact that these systems were already available before this work. However, our short term goal is to use an unified approach for confidence estimation (for ASR and MT) based only on CRF.

A number of knowledge sources are employed for extracting features, in a total of 25 major feature types:

• Target side: target word; bigram (trigram) backward sequences; number of occurrences

• Source side: source word(s) aligned to the target word [START_REF] Bach | Goodness: A method for measuring machine translation confidence[END_REF]: the combinations of the target (source) word and all aligned source (target) words in the window ±2

• Alignment context
• Word posterior probability [START_REF] Ueffing | Confidence measures for statistical machine translation[END_REF] • Pseudo-reference (Google Translate): does the word appear in the pseudo reference or not?

• Graph topology [START_REF] Luong | Word confidence estimation and its integration in sentence quality estimation for machine translation[END_REF]: number of alternative paths in the confusion network, maximum and minimum values of posterior probability distribution

• Language model (LM) based: length of the longest sequence of the current word and its previous ones in the target (resp. source) LM.

• Lexical features: word Part-Of-Speech (POS); sequence of POS of all its aligned source words; POS bigram (trigram) backward sequences; punctuation; proper name; numerical value.

• Syntactic features: null link [START_REF] Xiong | Error detection for statistical machine translation using linguistic features[END_REF]; constituent label; depth in the constituent tree after parsing the target hypothesis.

• Semantic features: number of word senses in WordNet (on the target hypothesis).

A very similar feature set was used in our English -Spanish WCE system submitted to WMT 2013 and WMT 2014 Quality Estimation shared task and obtained very good performances [START_REF] Luong | Word confidence estimation and its integration in sentence quality estimation for machine translation[END_REF]. Our experience in participating to the WCE shared task in 2013 and 2014 lead us to the following observation: while feature processing is very important to achieve good performance, it requires to call a set of heterogeneous NLP tools (for lexical, syntactic, semantic analyses). Thus, we recently proposed to unify the feature processing, together with the call of machine learning algorithms, in order to facilitate the design of confidence estimation systems. The opensource toolkit proposed (written in Python and made available on github2 ) integrates some standard as well as in-house features that have proven useful for WCE (based on our experience in WMT 2013 and 2014). To our knowledge, this is the first toolkit dedicated to word confidence estimation. We also believe that the integrated feature processing of our toolkit could be used for other cross-lingual NLP tasks.

EXPERIMENTAL SETTINGS

Our dev/test corpus contains 2643 transcribed French speech utterances (x f and f ref -5 hours -news domain) translated into English (e ref ) for which we obtained speech transcription output (f hyp ) and speech translation output (e hyp ). Since this corpus is rather small, we apply a cross-validation on it by tuning the re-decoding parameters on half of it while evaluating on the other half, and vice-versa.

To obtain the speech transcripts (f hyp ), we built an ASR system based on KALDI toolkit [START_REF] Povey | The kaldi speech recognition toolkit[END_REF]. The 3-gram language model was trained on the French ESTER corpus as well as French Gigaword (vocabulary size is 55k). SGMM-based acoustic models were trained using the same ESTER corpussee details in [START_REF] Galliano | Corpus description of the ester evaluation campaign for the rich transcription of french broadcast news[END_REF]. In addition, automatic post-processing was needed at the output of the ASR system in order to match requirements of standard input for machine translation (number conversion, recasing, re-punctuating, converting full words back to abbreviations and restoring special characters). With this post-processing, the output of our ASR system, scored against the f ref reference is 26.6% WER. This WER may appear as rather high according to the task (transcribing read news) but these news contain a lot of foreign named entities (part of the data is extracted from French newspapers dealing with european economy in many EU countries).

To obtain the speech translations (e hyp ), we used a French-English phrase-based translation system based on moses toolkit [START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF]. This medium-sized system was trained on Europarl and News parallel corpora for a former WMT evaluation shared-task (system more precisely described in [START_REF] Potet | The lig machine translation system for wmt 2010[END_REF] -1.6M parallel sentences and 48M monolingual sentences in target language).

In order to evaluate our WCE system, we obtained a sequence q of quality labels (recall that q = (q 1 , q 2 , ..., q N ) and q i ∈ {good, bad}) using TERp-A toolkit [START_REF] Snover | Terp system description[END_REF]. Each word or phrase in the hypothesis e hyp is aligned to a word or phrase in the reference (e ref ) with different types of edit: "I" (insertions), "S" (substitutions), "T" (stem matches), "Y" (synonym matches), "P" (phrasal substitutions) and "E" (exact match). Then, we re-categorize the obtained 6-label set into binary set: the E, T and Y belong to the good, whereas the S, P and I belong to the bad category.

Table 1 summarizes MT (translation from manual transcripts f ref ) and SLT (translation from automatic transcripts f hyp ) performances obtained on our corpus, as well as the distribution of good and bad labels inferred for both tasks (these labels will be considered as our reference to evaluate WCE later on). Logically, the percentage of (B) labels increases from MT to SLT task in the same conditions.

EXPERIMENTS ON SLT GRAPH RE-DECODING

Robust estimation of word confidence for a speech translation task

We first report in Table 2 the baseline results obtained by individual WCE system for a single ASR task (second column of the table). Then, we evaluate the performance of 3 WCE systems for the SLT task:

• The first system (SLT sys. / MT feat.) is the one described in section 3.2 and uses only MT features. • The second system (SLT sys. / ASR feat.) is the one described in section 3.1 and uses only ASR features (so this is predicting SLT output confidence using only ASR confidence features!). Word alignment information between f hyp and e hyp is used to project the WCE scores coming from ASR, to the SLT output, • The third system (SLT sys. / MT+ASR feat.) combines the information from the two previous WCE systems.

In this work, the ASR-based confidence score of the source is projected to the target SLT output and combined with the MT-based confidence score as shown in equation ( 8) (we did not tune the α coefficient and set it a priori to 0.5).

The results of these 3 systems are given in the last 3 columns of Table 2. They are obtained on the whole test set (all the results are given using a good/bad decision threshold which is a priori set to 0.7). The evaluation metric is the average between the F-measure for good labels and the F-measure for bad labels. From these results, we see that the use of both ASR-based and MT-based confidence scores improve the averaged F-score from 58,25% (MT only features) and 57,20% (ASR only features) to 60,75% (MT+ASR features). MT feat. ASR feat. MT+ASR feat. p(q/x f , f ) p(q/f, e) p(q/x f , f ) p(q/x f , f, e) projected to e Fmes 62,56 % 58,25% 57,20% 60,75%

Table 2. WCE performance with diff. feat. sets

Re-decoding results

Table 3 compares our 1-pass SLT baseline, to the 2-pass (graph re-decoding) strategy used with three different word confidence estimators: one based on ASR features only; one based on MT features only; and one based on joint MT+ASR features. We see that re-decoding the SLT search graph using WCE improves the translation performance measured with BLEU. The use of joint ASR+MT features for WCE lead to the best performance and the improvement over the 1-pass baseline (more than 2 BLEU points) is significant (p ∈ [0.00; 0.01] evaluated according to [START_REF] Clark | Better hypothesis testing for statistical machine translation: Controlling for optimizer instability[END_REF]). Comparing the 2-pass results, we observe that ASR+MT features clearly overpass MT features (BLEU is 32.82% instead of 31.89%) which may seem surprising because results of table 2 did not show a huge difference between both WCE methods. A first explanation may be related to the fact that for re-decoding, the confidence estimator is not applied on 2643 sentences only, but on 2643 * N sentences (in this case N = 100 best hypotheses) and differences in performance between MT and ASR+MT may be more important in this case. Another explanation may be also that even a small improvement in error detection (words whose label is bad) can lead to a significant gain of BLEU score. Finally, the use of ASR features only improves the performance compared to a single-pass system (31.12% instead of 30.60%) but this is the weakest improvement observed.

system baseline redecoding redecoding redecoding WCE feat.

none ASR MT SLT p(q/x f , f ) p(q/f, e) p(q/x f , f, e) Perf.

30.60% 31.12% 31.89% 32.82% Table 3. SLT perf. (BLEU) after 2d pass (2643 utt.) (without trying to analyze fine differences between MT and ASR+MT estimators -so, line indicating with re-decoding corresponds to the use of one or the other estimator).

Analysis of SLT hypotheses

Example 1 illustrates a first case where re-decoding allows slightly improving the translation hypothesis. Analysis of the labels from the confidence estimator indicates that the words a (start of sentence) and penalty were labeled as bad here. Thus, a better hypothesis arised from the second pass, although the transcription error could not be recovered (since only the 1-best ASR hypothesis is translated so far -not the full ASR graph).

In example 2, the confidence estimator labeled as bad the following word sequences: it has, speech that was and post route. Better translation hypothesis is found after re-decoding (correct pronoun, better quality at the end of sentence).

Finally, example 3 shows a case where, this time, the end of the first pass translation deteriorated after re-decoding. Analysis of confidence estimator output shows that the phrase to open was (correctly) labeled as bad, but the re-decoding gave rise to an even worse hypothesis. This last example illustrates some limitations of our current approach, which in this case would -in any event -have been unable to recover the named entity opel which was not present in the translation graph.

RELATED WORK

Several previous works tried to propose effective confidence measures in order to detect errors on ASR outputs. Confidence measures are introduced for Out-Of-Vocabulary (OOV) detection by [START_REF] Asadi | Automatic detection of new words in a large vocabu-lary continuous speech recognition system[END_REF]. [START_REF] Young | Recognition confidence measures: Detection of misrecognitions and out-of-vocabulary words[END_REF] extends the previous work and introduces the use of word posterior probability (WPP) as a confidence measure for speech recognition. Posterior probability of a word is most of the time computed using the hypothesis word graph [START_REF] Kemp | Estimating confidence using word lattices[END_REF]. Also, recent approaches [START_REF] Lecouteux | Combined low level and high level features for out-of-vocabulary word detection[END_REF] for confidence measure estimation use side-information extracted from the recognizer: normalized likelihoods (WPP), number of competitors at the end of a word (hypothesis density), decoding process behavior, linguistic features, acoustic (acoustic stability, duration features) and semantic features. In parallel, the Workshop on Machine Translation (WMT) introduced in 2013 a WCE task for machine translation. [START_REF] Li | Quality estimation for machine translation using the joint method of evaluation criteria and statistical modeling[END_REF] [24] employed the Conditional Random Fields (CRF) [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting et labeling sequence data[END_REF] model as their Machine Learning method to address the problem as a sequence labelling task. Meanwhile, [START_REF] Bicici | Referential translation machines for quality estimation[END_REF] extended the global learning model by dynamic training with adaptive weight updates in the perceptron training algorithm. As far as prediction indicators are concerned, [START_REF] Bicici | Referential translation machines for quality estimation[END_REF] proposed seven word feature types and found among them the "common cover links" (links that point from the leaf node containing this word to other leaf nodes in the same subtree of the syntactic tree) the most outstanding. [START_REF] Li | Quality estimation for machine translation using the joint method of evaluation criteria and statistical modeling[END_REF] focused only on various n-gram combinations of target words. Inheriting most of previously-recognized features, [START_REF] Luong | LIG system for WMT13 QE task: Investigating the usefulness of features in word confidence estimation for MT[END_REF] integrated a number of new indicators relying on graph topology, pseudo reference, syntactic behavior (constituent label, distance to the semantic tree root) and polysemy characteristic. To our knowledge, the first attempt to design WCE for speech translation, using joint ASR and MT features, is the work published very recently at IWSLT 2014 by [START_REF] Besacier | Word confidence estimation for speech translation[END_REF].

Concerning second-pass MT, several works dealt with automatic post-edition [START_REF] Parton | Can automatic post-editing make mt more meaningful?[END_REF] or N-best re-ranking [START_REF] Duh | Beyond log-linear models: Boosted minimum error rate training for n-best re-ranking[END_REF][START_REF] Bach | Goodness: A method for measuring machine translation confidence[END_REF][START_REF] Zhang | Distributed language modeling for n-best list re-ranking[END_REF].

[29] compared N-best re-ranking with graph re-decoding for statistical machine translation and have shown that graph redecoding is more efficient. This is why this paper focus on graphs instead of N-best lists.

If we focus on graph re-decoding, [START_REF] Zens | N-gram posterior probabilities for statistical machine translation[END_REF] proposes a 2-pass translation system which uses, in the second pass, sentence length and word sequence probability parameters. Experiments for Mandarin-English MT (NIST task) show a significant improvement in performance. Moreover, [START_REF] Tromble | Lattice minimum bayes risk decoding for statistical machine translation[END_REF] proposes to re-decode the MT graph using minimization of Bayes risk (MBR). The experimental results on Arabic-English, Mandarin-English and English-Mandarin show that the approach outperforms a N-best re-ranking baseline (also using MBR criterion). Finally, [START_REF] Venugopal | An efficient two-pass approach to synchronous-cfg driven statistical mt[END_REF] uses a N-gram language model (with longer history) to re-decode a translation graph obtained using probabilistic context free grammar (PCFG) MT. More recently, [START_REF] Vaswani | Decoding with large-scale neural language models improves translation[END_REF] integrated a large-scale neural language model into a machine translation system both by reranking N-best lists and by direct integration into the decoder. All these works were applied to MT graphs re-decoding only (no actual speech translation).

In addition to being applied to spoken language (SLT), our approach is different of the above mentionned since we are using, for second-pass decoding, external information features gathered via a robust confidence estimator (based on combined ASR and MT features). However, it present some similarities with [START_REF] Zens | N-gram posterior probabilities for statistical machine translation[END_REF] but using a much bigger number of features (for assessing word confidence).

CONCLUSION

We have proposed a formalisation of the use of quality assessment in speech translation. Moreover, experiments have shown that WCE (word confidence estimation) labels can be used to successfully re-decode the speech translation graphs and significantly improve speech translation performance. The use of combined MT and ASR features for robust word confidence estimation in SLT lead to the best performance in second-pass re-decoding. Some perspectives of this work are the following: train a unique WCE system for SLT (evaluating p(q/x f , f, e) as in equation ( 7)) using joint ASR+MT features and see if more SLT errors can be accurately detected, re-decoding speech translation graph obtained after translating the full ASR lattice (so far, a SLT graph is obtained by translating only the 1-best of the ASR), use our approach for real interactive speech translation scenarios such as news or lectures subtitling.

  une démobilisation des employés peut déboucher sur une démoralisation mortifère f hyp une démobilisation des employés peut déboucher sur une démoralisation mort y faire e hyp baseline a demobilisation employees can lead to a penalty demoralisation e hyp with redecoding a demobilisation of employees can lead to a demoralization death e ref demobilization of employees can lead to a deadly demoralization example 2 f ref celui-ci a indiqué que l'intervention s'était parfaitement bien déroulée et que les examens postopératoires étaient normaux f hyp celui-ci a indiqué que l' intervention c'était parfaitement bien déroulés , et que les examens post opératoire étaient normaux . e hyp baseline it has indicated that the speech that was well conducted , and that the tests were normal post route e hyp with redecoding he indicated that the intervention is very well done , and that the tests after operating were normal e ref he indicated that the operation went perfectly well and the post-operative tests were normal example 3 f ref general motors repousse jusqu'en janvier le plan pour opel f hyp general motors repousse jusqu' en janvier le plan pour open e hyp baseline general motors postponed until january the plan to open e hyp with redecoding general motors puts until january terms to open e ref general motors postponed until january the plan for opel

Table 1 .

 1 Baseline MT and SLT performance on 2643 utt.

	task	ASR	MT	% good	% bad
		(WER)	(BLEU)
	MT	0%	52.8%	82.5%	17.5%
	SLT	26.6%	30.6%	65.5%	34.5%

Table 4 .

 4 Exemples of French SLT hyp with and w/o redecoding

	Examples of speech translation hypotheses (SLT) ob-
	tained with or without graph re-decoding are given in table 4

q i could be also more than

labels, or even scores but this paper only deals with error detection (binary set of labels)

http://github.com/besacier/WCE-LIG