Ngoc-Tien Le

Benjamin Lecouteux

Laurent Besacier

Automatic Quality Estimation for Speech Translation Using Joint ASR and MT Features

Keywords:

This paper addresses the automatic quality estimation of spoken language translation (SLT). This relatively new task is defined and formalized as a sequence-labeling problem where each word in the SLT hypothesis is tagged as good or bad according to a large feature set. We propose several word confidence estimators (WCE) based on our automatic evaluation of transcription (ASR) quality, translation (MT) quality, or both (combined ASR+MT). This research work is possible because we built a specific corpus, which contains 6.7k utterances comprising the quintuplet: ASR output, verbatim transcript, text translation, speech translation, and post-edition of the translation. The conclusion of our multiple experiments using joint ASR and MT features for WCE is that MT features remain the most influential while ASR features can bring interesting complementary information. In addition, the last part of the paper proposes to disentangle ASR errors and MT errors where each word in the SLT hypothesis is tagged as good, asr_error or mt_error. Robust quality estimators for SLT can be used for re-scoring speech translation graphs or for providing feedback to the user in interactive speech translation or computer-assisted speech-to-text scenarios.

Introduction

Automatic quality assessment of spoken language translation (SLT), also named confidence estimation (CE), is an important topic because it allows us to know whether a system produces user-acceptable outputs or not. In interactive speech-to-speech translation, CE helps to judge whether a translated term is uncertain (in which case we can ask the speaker to rephrase or repeat the term). For speech-to-text applications, CE may tell us whether output translations are worth correcting, or whether they require retranslation from scratch. Moreover, an accurate CE can also help to improve SLT itself through a second-pass Nbest list re-ranking or search graph re-decoding, as has already been done for text translation in [START_REF] Bach | Goodness: A method for measuring machine translation confidence[END_REF] and Luong et al (2014b), or for speech translation in [START_REF] Besacier | Spoken language translation graphs re-decoding using automatic quality assessment[END_REF]. Consequently, building a method which is capable of pointing out the correct parts as well as detecting the errors in a speech-translated output is crucial to tackle the above issues.

Given a signal x f in the source language, spoken language translation (SLT) consists of finding the most probable target language sequence ê = (e 1 , e 2 , ..., e N ), as in (1):

ê = argmax e {p(e|x f , f )} (1)
where f = (f 1 , f 2 , ..., f M ) is the transcription of x f . Now, if we perform confidence estimation at the "word" level, the problem is called Word-level Confidence Estimation (WCE) and we can represent this information as a sequence q (same length N of ê) where q = (q 1 , q 2 , ..., q N ) and q i ∈ {good, bad}. 1 Then, integrating automatic quality assessment into our SLT process (q is defined above) can be done as in ( 2)-( 4):

ê = argmax e q
p(e, q|x f , f )

ê = argmax e q p(q|x f , f, e) * p(e|x f , f )

ê ≈ argmax e {max q {p(q|x f , f, e) * p(e|x f , f )}} (4)

In the product of (4), the SLT component p(e|x f , f ) and the WCE component p(q|x f , f, e) contribute together to find the best translation output ê. In the past, WCE has been treated separately in ASR or MT contexts and we propose here a joint estimation of word confidence for an SLT task involving both ASR and MT.

This journal paper is an extended version of a paper published at ASRU 2015 [START_REF] Besacier | Spoken language translation graphs re-decoding using automatic quality assessment[END_REF], but here we focus more on the WCE component and on the best approaches to accurately estimate p(q|x f , f, e).

Contributions A corpus (distributed to the research community) 2 dedicated to WCE for SLT was initially published in [START_REF] Besacier | Word confidence estimation for speech translation[END_REF]. In this paper, we present its extension from 2643 to 6693 speech utterances. In addition, while our previous work on quality assessment was based on two separate WCE classifiers (one for quality assessment in ASR and one in MT), we propose here a unique joint model based on different feature types (ASR and MT features). This joint model allows us to operate feature selection and analyze which features (from ASR or MT) are the most efficient for quality assessment in speech translation. We also experiment with two ASR systems that have different performance in order to analyze the behaviour of our SLT qualityassessment algorithms at different levels of word error rate (WER) (?). The last part of this paper proposes to disentangle ASR and MT errors in speech translation by automatically detecting the origin of SLT errors, either due to ASR or to MT.

Outline The outline of this paper is as follows: Section 2 reviews the stateof-the-art on confidence estimation for ASR and MT. Our word-confidence estimation (WCE) system using multiple features is then described in Section 3. The experimental setup (namely our specific WCE corpus) is presented in Section 4 while Section 5 evaluates our joint WCE system. Feature selection for quality assessment in speech translation is analyzed in Section 6. Section 7 proposes to disentangle ASR and MT errors in SLT output and finally, Section 8 concludes this work and gives some future perspectives.

Related Work on Confidence Estimation for ASR and MT

Several previous works tried to propose effective confidence measures in order to detect errors on ASR outputs. Confidence measures are introduced for Out-Of-Vocabulary (OOV) detection by [START_REF] Asadi | Automatic detection of new words in a large vocabulary continuous speech recognition system[END_REF]. [START_REF] Young | Recognition confidence measures: Detection of misrecognitions and out-of-vocabulary words[END_REF] extends this previous work and introduces the use of word posterior probability (WPP) as a confidence measure for speech recognition. The posterior probability of a word is most of the time computed using the hypothesis word graph [START_REF] Kemp | Estimating confidence using word lattices[END_REF]. More recent approaches [START_REF] Lecouteux | Combined low level and high level features for out-of-vocabulary word detection[END_REF] for confidence measure estimation use side-information extracted from the recognizer: normalized likelihoods (WPP), the number of competitors at the end of a word (hypothesis density), decoding process behaviour, linguistic features, acoustic features (acoustic stability, duration features), and semantic features. In addition, ASR quality estimation has been addressed in several recent studies [START_REF] De Souza | Multitask learning for adaptive quality estimation of automatically transcribed utterances[END_REF][START_REF] Zamani | Reference-free and confidence-independent binary quality estimation for automatic speech recognition[END_REF][START_REF] Jalalvand | Transcrater: a tool for automatic speech recognition quality estimation[END_REF]. Re-scoring the output of the ASR N -best list using ASR and MT features was also proposed by Ng et al (2015b[START_REF] Ng | Groupwise learning for ASR kbest list reranking in spoken language translation[END_REF].

In parallel, the Workshop on Machine Translation (WMT)3 introduced in 2013 a WCE task for Machine Translation. [START_REF] Han | Quality estimation for machine translation using the joint method of evaluation criteria and statistical modeling[END_REF] and Luong et al (2013b) employed the Conditional Random Fields (CRF) [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting et labeling sequence data[END_REF] model as their machine-learning method to address the problem as a sequencelabelling task. Meanwhile, [START_REF] Bicici | Referential translation machines for quality estimation[END_REF] extended their initial proposition by dynamic training with adaptive weight updates in their neural network classifier. As far as prediction indicators are concerned, [START_REF] Bicici | Referential translation machines for quality estimation[END_REF] proposed seven word feature types and found among them the "common cover links" (the links that point from the leaf node containing this word to other leaf nodes in the same subtree of the syntactic tree) the most outstanding. [START_REF] Han | Quality estimation for machine translation using the joint method of evaluation criteria and statistical modeling[END_REF] focused only on various N -gram combinations of target words. Inheriting most of the previously recognized features, Luong et al (2013b) integrated a number of new indicators relying on graph topology, pseudo reference, syntactic behaviour (constituent label, distance to the semantic tree root) and polysemy characteristic. The estimation of the confidence score mainly uses classifiers like CRF [START_REF] Han | Quality estimation for machine translation using the joint method of evaluation criteria and statistical modeling[END_REF]Luong et al, 2014a), Support Vector Machines [START_REF] Langlois | Loria system for the wmt12 quality estimation shared task[END_REF], or neural networks [START_REF] Bicici | Referential translation machines for quality estimation[END_REF]. Some investigations were also conducted to determine which features seem to be the most relevant. [START_REF] Langlois | Loria system for the wmt12 quality estimation shared task[END_REF] proposed to filter features using a forward-backward algorithm to discard linearly correlated features. Using boosting as the learning algorithm, [START_REF] Luong | Towards accurate predictors of word quality for machine translation: Lessons learned on french -english and english -spanish systems[END_REF] were able to take advantage of the most significant features.

Finally, several toolkits for WCE have recently been proposed: TranscRater for ASR [START_REF] Jalalvand | Transcrater: a tool for automatic speech recognition quality estimation[END_REF], 4 QuEst++ for MT [START_REF] Specia | Multi-level translation quality prediction with quest++[END_REF], 5 MARMOT for MT (Logacheva et al, 2016), 6 as well as the WCE toolkit [START_REF] Servan | An Open Source Toolkit for Word-level Confidence Estimation in Machine Translation[END_REF] 7 that is used to extract MT features in the experiments of this paper.

To the best of our knowledge, the first attempt to design WCE for speech translation, using both ASR and MT features in a single classifier, was our own work [START_REF] Besacier | Word confidence estimation for speech translation[END_REF][START_REF] Besacier | Spoken language translation graphs re-decoding using automatic quality assessment[END_REF] which is further extended in this paper.

Building an Efficient Quality Assessment (WCE) System

The WCE component solves equation ( 5):

q = argmax q {p SLT (q|x f , f, e)} (5)
where q = (q 1 , q 2 , ..., q N ) is the sequence of quality labels on the target language. This is a sequence-labelling task that can be solved with several machinelearning techniques such as CRF. However, for that, we need a large amount of training data for which a quadruplet (x f , f, e, q) is available. In this work, we will use a corpus extended from [START_REF] Besacier | Word confidence estimation for speech translation[END_REF] which contains 6.7k utterances. We will investigate if this amount of data is enough to evaluate and test a joint model p SLT (q|x f , f, e).

As it is much easier to obtain data containing either the triplet (x f , f, q) (automatically transcribed speech with manual references and quality labels 4. https://github.com/hlt-mt/TranscRater 5. http://www.quest.dcs.shef.ac.uk/ 6. https://github.com/qe-team/marmot 7. https://github.com/besacier/WCE-LIG inferred from WER estimation), or the triplet (f, e, q) (automatically translated text with manual post-edits and quality labels inferred using tools such as TERp-A [START_REF] Snover | Terp system description[END_REF])), we can also recast the WCE problem as in (6):

q = argmax q {p ASR (q|x f , f ) α * p M T (q|e, f ) 1-α } (6)
where α is a weight giving more or less importance to W CE ASR (quality assessment on transcription) compared to W CE M T (quality assessment on translation). It is important to note that p ASR (q|x f , f ) corresponds to the quality estimation of the words in the target language based on features calculated on the source language (ASR). For that, what we do is project source quality scores to the target using word alignment information between e and f sequences. This alternative approach (Equation ( 6)) will be also evaluated in this work even if it corresponds to a different optimization problem than Equation ( 5). In particular, the choice of α is only set a priori in our experiments (to 0.5) which is probably not the best option.

In both approaches -joint (p SLT (q|x f , f, e)) and combined (p ASR (q|x f , f ) + p M T (q|e, f )) -some features need to be extracted from ASR and MT modules. They are more precisely detailed in the next subsections.

WCE Features for Speech Transcription (ASR)

In this work, we extract several types of features, which come from the ASR graph, from language model scores and from a morphosyntactic analysis. These features are listed below (more details can be found in [START_REF] Besacier | Word confidence estimation for speech translation[END_REF]):

• Acoustic features: word duration (F-dur).

• Graph features (extracted from the ASR word confusion networks): number of alternative (F-alt) paths between two nodes; word posterior probability (F-post).

• Linguistic features (based on probabilities by the language model): the word itself (F-word), 3-gram probability (F-3g), log probability (F-log), back-off level of the word (F-back), as proposed in [START_REF] Fayolle | Crf-based combination of contextual features to improve a posteriori word-level confidence measures[END_REF].

• Lexical Features: Part-Of-Speech (POS) of the word (F-POS).

• Context Features: Part-Of-Speech tags in the neighborhood of a given word (F-context).

For each word in the ASR hypothesis, we estimate these 9 features: F-Word; F-3g; F-back; F-log; F-alt; F-post; F-dur; F-POS; and F-context.

In a preliminary experiment, we evaluate these features for quality assessment in ASR only (W CE ASR task). Two different classifiers will be used: a variant of boosting classification algorithm called bonzaiboost [START_REF] Laurent | Boosting bonsai trees for efficient features combination : application to speaker role identification[END_REF], which implements the boosting algorithm Adaboost.MH over deeper trees, and CRF.

WCE Features for Machine Translation (MT)

A number of knowledge sources are employed for extracting features, in a total of 24 major feature types, as depicted in Table 1.

It is important to note that we extract features regarding tokens in the translated hypothesis (MT or SLT). In other words, one feature is extracted for each token in the MT output. So in Table 1, target refers to the feature coming from the translated hypothesis and source refers to a feature extracted from the source word aligned to the considered target word. More details on some of these features are given in the next subsections.

Internal Features

These features are given by the MT system, which outputs additional data like an N -best list.

Word Posterior Probability (WPP) and Nodes features are extracted from a confusion network, which comes from the output of the MT N -best list. WPP Exact is the WPP value for each word concerned at the exact same position in the graph. WPP Any extracts the same information at any position in the graph. WPP Min gives the smallest WPP value concerned by the transition and WPP Max its maximum.

External Features

Below is the list of the external features used:

• Proper Name: indicates if a word is a proper name; the same binary features are extracted to know if a token is Numerical, Punctuation, or a Stop Word.

• Unknown Stem: informs whether the stem of the considered word is known or not.

• Number of Word/Stem Occurrences: counts the occurrences of a word/stem in the sentence.

• Alignment context features: these features (#11-13 in Table 1) are based on collocations and were proposed by [START_REF] Bach | Goodness: A method for measuring machine translation confidence[END_REF]. Collocations could be an indicator for judging whether a target word is generated by a particular source word. We also apply the reverse, the collocations regarding the source side (#7 in Table 1, simply called Alignment Features):

Source alignment context features: the combinations of the target word, the source word (with which it is aligned), and one source word before and one source word after (left and right contexts, respectively).

Target alignment context features: the combinations of the source word, the target word (with which it is aligned), and one target word before and one target word after.

• Longest Target (or Source) N -gram Length: we seek to compute the length (n + 1) of the longest left sequence (w i-n ) concerned by the current word (w i ) and known by the language model (LM) concerned (source and target sides). For example, if the longest left sequence w i-2 , w i-1 , w i appears in the target LM, the longest target N -gram value for w i will be 3. This value ranges from 0 to the max order of the LM concerned. We also extract a redundant feature called Backoff Behaviour Target.

• The target word's constituent label (Constituent Label) and its depth in the constituent tree (Distance to Root) are extracted using a syntactic parser.

• Target Polysemy Count: we extract the polysemy count, which is the number of meanings of a word in a given language.

• Occurrences in Google Translate and Occurrences in Bing Translator: in the translation hypothesis, we (optionally) test the presence of the target word in on-line translations given respectively by Google Translate and Bing Translator. 8

A very similar feature set was used for a simple W CE M T task (English-Spanish MT, WMT 2013, 2014 quality-estimation shared task) and obtained very good performance (Luong et al, 2013a). This preliminary experience in participating to the WCE shared task in 2013 and 2014 led us to the following observation: while feature processing is very important to achieve good performance, it requires a set of heterogeneous NLP tools (for lexical, syntactic, and semantic analysis). Thus, we recently proposed to unify the feature processing, together with the call of machine-learning algorithms, in order to facilitate the design of confidence-estimation systems. The open-source toolkit proposed (written in Python and made available on github) 9 integrates some standard 8. Using this kind of feature is controversial, but we observed that such features are available in general scenarios, so we decided to include them in our experiments. Contrastive results without these two features will be also given later on. 9. http://github.com/besacier/WCE-LIG as well as in-house features that have proven useful for WCE (based on our experience in WMT 2013 and 2014).

In this paper, we use only CRF as our machine-learning method, with the WAPITI toolkit [START_REF] Lavergne | Practical very large scale crfs[END_REF], to train our WCE estimator based on both MT and ASR features. For a French-English translation task, we used our SMT system to obtain the translation hypotheses for 10,881 source sentences taken from news corpora from WMT evaluation campaigns from 2006 to 2010. Post-editions were obtained from non professional translators using a crowdsourcing platform. More details on the baseline SMT system used can be found in [START_REF] Potet | The lig machine translation system for wmt 2010[END_REF], and more details on the post-edited corpus can be found in [START_REF] Potet | Collection of a large database of french-english smt output corrections[END_REF]. It is worth mentioning, however, that a subset (311 sentences) of these collected posteditions was assessed by a professional translator and 87.1% of the post-edits were judged to improve the hypothesis. Then, the word-label setting for WCE was done using the TERp-A toolkit [START_REF] Snover | Terp system description[END_REF]. Table 2 illustrates the labels generated by TERp-A for one hypothesis and post-edition pair. Each word or phrase in the hypothesis is aligned to a word or phrase in the post-edition with different types of edit operations: "I" (insertions), "S" (substitutions), "T" (stem matches), "Y" (synonym matches), and "P" (phrasal substitutions). The lack of a symbol indicates an exact match and will be replaced by "E" thereafter. We do not consider words marked with "D" (deletions) since they appear only in the reference. However, later on, we will have to train binary classifiers (good/bad) so we re-categorize the obtained 6-label set into a binary set: E, T and Y belong to the good (G) class, whereas S, P and I belong to the bad (B) category.

Extending the Corpus with Speech Recordings and Transcripts

The dev set and tst set of this corpus were recorded by French native speakers. Each sentence was uttered by 3 speakers, leading to 2643 and 4050 speech recordings for the dev set and tst set, respectively. For each speech utterance, a quintuplet containing ASR output (f hyp ), verbatim transcript (f ref ), English text-translation output (e hypmt ), speech-translation output (e hyp slt ) and postedition of translation (e ref ) was made available. This corpus is available on a github repository.10 More details are given in Table 3. The total length of the dev and tst speech corpora obtained are 16h52, since some utterances were pretty long.

ASR Systems

To obtain the speech transcripts (f hyp ), we built a French ASR system based on KALDI toolkit [START_REF] Povey | The kaldi speech recognition toolkit[END_REF]. Acoustic models are trained using several corpora (ESTER, REPERE, ETAPE and BREF120) representing more than 600 hours of French transcribed speech.

The baseline GMM system is based on mel-frequency cepstral coefficient (MFCC) acoustic features (13 coefficients expanded with delta and double delta features and energy: 40 features) with various feature transformations including linear discriminant analysis (LDA), maximum likelihood linear transformation (MLLT), and feature space maximum likelihood linear regression (fMLLR) with speaker adaptive training (SAT). The GMM acoustic model makes initial phoneme alignments of the training data set for the following DNN acoustic model training.

The speech transcription process is carried out in two passes: an automatic transcript is generated with a GMM-HMM model of 43,182 states and 250,000 Gaussians. Then word-graph outputs obtained during the first pass are used to compute a fMLLR-SAT transform on each speaker. The second pass is performed using DNN acoustic model trained on acoustic features normalized with the fMLLR matrix.

CD-DNN-HMM acoustic models are trained (43,182 context-dependent states) using a GMM-HMM topology.

We propose to use two 3-gram language models trained on the French ES-TER corpus [START_REF] Galliano | Corpus description of the ester evaluation campaign for the rich transcription of french broadcast news[END_REF] as well as on French Gigaword (vocabulary sizes are 62k and 95k, respectively). The LM weight parameters of the ASR systems are tuned via WER on the dev corpus. Details on these two language models can be found in Table 4.

Table 4 -Details on language models (LMs) used in our two ASR systems.

LM

1-gram 2-grams 3-grams

small (ASR1) 62K 1M 59M big (ASR2) 95K 49M 301M
In our experiments, we propose two ASR systems based on the previously described language models. The first system (ASR1) uses the small language model allowing a fast ASR system (about 2x Real Time), while in the second system lattices are rescored with a big language model (about 10x Real Time) during a third pass.

Table 5 presents the performances obtained by two above ASR systems. These WER scores may appear rather high for the task of transcribing read news. A deeper analysis shows that these news items contain a lot of foreign named entities, especially in our dev set. This part of the data is extracted from French media dealing with the European economy. This could also explain why the scores are significantly different between the dev and tst sets. In addition, automatic post-processing is applied to the ASR output in order to match the requirements of standard input for MT.

SMT System

We used the Moses phrase-based translation toolkit [START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF] to translate French ASR into English (e hyp ). This medium-sized system was trained using a subset of data provided for IWSLT 2012 evaluation [START_REF] Federico | Overview of the IWSLT 2012 evaluation campaign[END_REF]: Europarl, Ted and News-Commentary corpora, with a total amount of about 60M words. We used an adapted target language model trained on specific data (News Crawled corpora) similar to our evaluation corpus (see [START_REF] Potet | The lig machine translation system for wmt 2010[END_REF]). This standard SMT system is used in all experiments reported in this paper.

Obtaining Quality Assessment Labels for SLT

After building an ASR system, we have a new element of our desired quintuplet: the ASR output f hyp , which is the noisy version of our already available verbatim transcripts called f ref . This ASR output (f hyp ) is then translated by the exact same SMT system [START_REF] Potet | The lig machine translation system for wmt 2010[END_REF] mentioned in subsection 4.3. This new output translation is called e hyp slt and is a degraded version of e hypmt (translation of f ref ).

At this point, a strong assumption we made has to be revealed: we reused the post-editions obtained from the text-translation task (called e ref ), to infer the quality (G, B) labels of our speech translation output e hyp slt . The word-label setting for WCE is done using TERp-A toolkit [START_REF] Snover | Terp system description[END_REF]) between e hyp slt and e ref . This assumption, and the fact that initial MT postedition can also be used to infer labels of a SLT task, is reasonable regarding results (presented later in Tables 8 and9) where it is shown that there is not a huge difference between the MT and SLT performance (evaluated with BLEU).

The above remark is important and this is what makes this corpus valuable. For instance, other corpora such as the TED corpus could be used to obtain a quintuplet with ASR output, verbatim transcript, MT output, SLT output and target translation, but there are two main differences: first in TED, the target translation is a manual translation of the prior subtitles so this is not a post-edition of an automatic translation (and we have no guarantee that the good/bad labels extracted from this would be reliable for WCE training and testing); secondly, in our corpus, each sentence is uttered by 3 different speakers which introduces speaker variability in the database and allows us to deal with different ASR outputs for a single source sentence.11

Final Corpus Statistics

The final corpus obtained is summarized in Table 6, where we also clarify how the WCE labels were obtained. For the test set, we now have all the data needed to evaluate WCE for 3 tasks:

• ASR: extract good/bad labels by calculating WER between f hyp and f ref ,

• MT: extract good/bad labels by calculating TERp-A between e hypmt and e ref ,

• SLT: extract good/bad labels by calculating TERp-A between e hyp slt and e ref .

Table 7 gives an example of the quintuplet available in our corpus. One transcript (f hyp1 ) has 1 error while the other one (f hyp2 ) has 4. This leads to respectively 2 B labels (e hyp slt1 ) and 4 B labels (e hyp slt2 ) in the speech translation output, while e hypmt has only one B label.

Tables 8 and9 summarize the baseline ASR, MT and SLT performances obtained on our corpora, as well as the distribution of good (G) and bad (B) labels inferred for both tasks. Logically, the percentage of (B) labels increases from the MT to the SLT task under the same conditions. • The first and second systems (WCE for ASR/ASR feat.) use ASR features described in Section 3.1 with two different classifiers (CRF or Boosting).

• The third system (WCE for SLT/MT feat.) uses only the MT features described in Section 3.2 with the CRF classifier.

• The fourth system (WCE for SLT/ASR feat.) uses only the ASR features described in Section 3.1 with the CRF classifier, i.e. predicting SLT output confidence using only ASR confidence features! Word-alignment information between f hyp and e hyp is used to project the WCE scores coming from ASR to the SLT output.

In all experiments reported in this paper, we evaluate the performance of our classifiers by using the average between the F-measure for good labels and the F-measure for bad labels that are calculated by the common evaluation metrics: Precision, Recall and F-measure for good/bad labels. Since two ASR systems are available, F-mes1 is obtained for SLT based on ASR1, whereas F-mes2 is obtained for SLT based on ASR2. For the results in Table 11, the classifier is evaluated on the tst part of our corpus and trained on the dev part. Concerning WCE for ASR, we observe that F-measure decreases when ASR WER is lower F-mes2 <F-mes1 while W ER ASR2 < W ER ASR1 , i.e. quality assessment in ASR seems to become harder as the ASR system improves. This could be due to the fact that the ASR1 errors recovered by a bigger LM in the ASR2 system were easier to detect. Anyway, this conclusion should be considered with caution since both results (F-mes1 and F-mes2 ) are not directly comparable because they are evaluated on different references (the proportion of good/bad labels differ as the ASR systems themselves differ). The effect of the classifier (CRF or Boosting) is not conclusive since CRF is better for F-mes1 and worse for F-mes2. In any case, we decide to use CRF for all our future experiments since this is the classifier integrated in the WCE-LIG toolkit [START_REF] Servan | An Open Source Toolkit for Word-level Confidence Estimation in Machine Translation[END_REF].

To assess WCE for SLT, the observed F-measure is better using MT features rather than ASR features, i.e. quality assessment for SLT is more dependent on MT features than ASR features. Again, F-measure decreases when ASR WER is lower (F-mes2 <F-mes1 while W ER ASR2 < W ER ASR1 ). For MT features, removing OccurInGoogleTranslate and OccurInBingTranslate features lead to 63.09% and 62.33% for F-mes1 and F-mes2, respectively. Finally, it is worth mentioning that the perfomance obtained by the classifiers in Table 11 are above the random baselines in Table 10.

In the next subsection, we investigate whether the use of both MT and ASR features improves quality assessment for SLT.

SLT Quality Assessment Using both MT and ASR Features

We report in Table 13 the WCE results for SLT obtained using both MT and ASR features. More precisely, we evaluate two different approaches (combination and joint):

• The first system (WCE for SLT/MT+ASR feat.) combines the output of two separate classifiers based on ASR and MT features. In this approach, the ASR-based confidence score of the source is projected to the target SLT output and combined with the MT-based confidence score as shown in Equation (6) (we did not tune the α coefficient and set it a priori to 0.5).

• The second system (joint feat.) trains a single WCE system for SLT (evaluating p(q|x f , f, e) as in Equation ( 5) using joint ASR features and MT features. All ASR features are projected to the target words using automatic word alignments. However, a problem occurs when a target word does not have any source word aligned to it. In this case, we decide to duplicate the ASR features of its previous target word. Another problem occurs when a target word is aligned to more than one source word. In that case, there are several strategies we can use to infer the 9 ASR features: average or max over numerical values, selection or concatenation over symbolic values (for F-word and F-POS), etc. Three different variants of these strategies (shown in Table 12) are evaluated here.

Table 12 -Different strategies to project ASR features to a target word when it is aligned to more than one source word. It should be noted that F-context features are the combinations of the source word (F-word) and one POS of the source word (F-POS) before and one POS of the source word (F-POS) after. The results in Table 13 show that joint ASR and MT features only slightly improve WCE performance: F-mes1 is slightly better than one set-up in Table 11 (WCE for SLT/MT features only). We also observe that simple combination (MT+ASR) degrades WCE performance. This may be due to the different behaviour of the W CE M T and W CE ASR classifiers which makes the weighted combination ineffective. The relatively disappointing performance of our joint classifier may be due to an insufficient training set (only 2643 utterances in dev ). Finally, removing OccurInGoogleTranslate and OccurInBingTranslate features for Joint lowered F-mes between 1% and 2%.

ASR Feat Joint 1 Joint 2 Joint 3 F-post avg(F-post1, F-post2) avg(F-post1, F-post2) avg(F-post1, F-post2) F-log avg(F-log1, F-log2) avg(F-log1, F-log2) avg(F-log1, F-log2) F-back avg(F-back1, F-back2) avg(F-back1, F-back2) avg(F-back1, F-back2) F-dur max(F-dur1, F-dur2) max(F-dur1, F-dur2) max(F-dur1, F-dur2) F-3g max(F-3g1, F-3g2) max(F-3g1, F-3g2) max(F-3g1, F-3g2) F-alt max(F-alt1, F-alt2) max(F-alt1, F-alt2) max(F-alt1, F-alt2) F-word F-word1 F-word2 F-word1_F-word2 F-POS F-POS1 F-POS2 F-POS1_F-POS2 F-context F-context* F-context* F-context*
These observations lead us to investigate the behaviour of our WCE approaches for a large range of good/bad decision thresholds. While the previous tables provided WCE performance for a single point of interest (good/bad decision threshold set to 0.5), the curves in Figures 1 and2 show the full picture of our WCE systems (for SLT) using speech-transcription systems ASR1 and ASR2, respectively. We observe that the classifier based on ASR features has a very different behaviour than the classifier based on MT features which explains why their simple combination (MT+ASR) does not work very well for the default decision threshold (0.5). However, for thresholds above 0.75, the use of joint ASR and MT features is slightly beneficial compared to MT features only. This is interesting because higher thresholds improve the F-measure on bad labels and so improve error detection. Both curves are similar whatever the ASR system used. These results suggest that with enough development data for appropriate threshold tuning (which we do not have for this very new task), the use of both ASR and MT features should improve error detection in speech translation (blue and red curves are above the green curve for higher decision thresholds).12 Although not reported here, we also analyzed the F-measure curves for bad and good labels separately: if we consider, for instance, the ASR1 system, for decision threshold equal to 0.75, the F-measure on bad labels is equivalent (52%) for 3 systems (Joint, MT+ASR and MT ), while the F-measure on good labels is 76% when using MT features only, 78% when using Joint features and 77% when using MT+ASR features. In other words, for a fixed performance on bad labels, the F-measure on good labels is improved using all information available (ASR and MT features). Finally, if we focus on Joint versus MT+ASR, we notice that the range of the threshold where performance is stable is larger for Joint than for MT+ASR.

Feature Selection

In this section, we try to better understand the contribution of each (ASR or MT) feature by applying feature selection on our joint WCE classifier. In these experiments, we decide to keep OccurInGoogleTranslate and OccurInBing-Translate features.

We choose the Sequential Backward Selection (SBS) algorithm [START_REF] Aha | A Comparative Evaluation of Sequential Feature Selection Algorithms[END_REF] which is a top-down algorithm starting from a feature set noted Y k (which denotes the set of all features) and sequentially removing the most irrelevant one (x) that maximizes the Mean F-Measure, M F (Y k -x). In our work, we examine until the set Y k contains only one remaining feature. Algorithm 1 summarizes the whole process.

Algorithm 1 Sequential Backward Selection (SBS) algorithm for feature selection. Y k denotes the set of all features and x is the feature removed at each step of the algorithm.

while size of

Y k > 0 do maxval = 0 for x ∈ Y k do if maxval < M F (Y k -x) then maxval ← M F (Y k -x) worstf eat ← x end if end for remove worstf eat from Y k end while
The results of the SBS algorithm can be found in Table 14 which ranks all joint features used in WCE for SLT by order of importance after applying the algorithm on dev. We can see that the SBS algorithm is not very stable and is clearly influenced by the ASR system (ASR1 or ASR2) considered in SLT. In any case, if we focus on the 10-best features in both cases, we find that the most relevant ones are:

• Alignment Features (source and target collocation features)

• Occur in Google Translate and Occur in Bing Translate (diagnostic from other MT systems),

• Longest Source N -gram Length, Target Backoff Behaviour (source or target N -gram features),

• Word Posterior Probability Max (WPP Max) (graph topology feature)

We also observe that the most relevant ASR features (in Table 14) are Fback, F-3g and F-context (linguistic and context features) whereas ASR lexical, acoustic and graph-based features are among the worst (F-POS, F-dur and Fpost). Accordingly, in our experimental setting, it seems that MT features are more influential than ASR features. Interestingly, "source and target collocation features" (Alignment Features) and "Occur in Bing Translate" are the most prominent features (rank 1 and rank 2, respectively) when applied to dev corpus for both ASR1 and ASR2. Besides, the graph-topology feature extracted from a confusion network WPP Max outperforms the others such as Nodes and WPP Min. Nevertheless, two other features including WPP Exact and WPP any are proven to be weak in accordance with their bottom-most positions against the two above systems, whereas we were expecting to see them among the top features (as shown in [START_REF] Luong | Towards accurate predictors of word quality for machine translation: Lessons learned on french -english and english -spanish systems[END_REF] where WPP Any is among the best features for WCE in MT). Figure 3 presents the evolution of WCE performance for dev and tst corpora when feature selection using the SBS algorithm is made on dev, for the ASR1 system; the same shape is observed for the ASR2 system. In other words, for this figure, we apply our SBS algorithm on dev which means that feature selection is done on dev with classifiers trained on tst. After that, the best feature subsets (using 33, 32, 31 until 1 feature only) are applied to the tst corpus (with classifiers trained on dev ). 13In the figure, we observe that half of the features only contribute to the WCE process since the best performance is observed with only 15 to 25 features. We selection for dev application on tst Figure 3 -Evolution of WCE performance for dev (features selected) and tst corpora when feature selection using SBS algorithm is made on dev (ASR1 system only; the same shape is observed for ASR2).

also see that optimal WCE performance is not necessarily obtained with the full feature set, as it can be obtained with a subset of it.

7 Disentangling ASR and MT Errors

In the previous section, we only extract good/bad labels from the SLT output while it might be interesting to move from a 2-class problem to a 3-class problem in order to label our SLT hypotheses with one of the 3 following labels: good (G), asr-error (B_ASR) and mt-error (B_MT ). Before training automatic systems for error detection, we need to set such 3-class labels for our dev and test corpora. For that, in the next subsections we propose two slightly different methods to extract them. The first one is based on the word alignments from SLT to MT, and the second is based on a subtraction between SLT and MT errors.

Method 1: Using word alignments between MT and SLT

In MT, the fertility of a source word denotes how many output words it translates as. If we transpose this definition to our disentangling problem, then fertility of an MT error denotes how many erroneous words -in the SLT output -it is aligned to. From this simple definition, we derive our first way (Method 1 ) of generating 3-class annotations.

Let êslt = (e 1 , e 2 , . . . , e n ) be the set of SLT hypotheses (e hyp slt ); e kj denotes the j th word in the sentence e k , where 1 ≤ k ≤ n Let êmt = (e 1 , e 2 , . . . , e n ) be the set of MT hypotheses (e hypmt ); e ki denotes the i th word in the sentence e k , where 1 ≤ k ≤ n Let L = (l 1 , l 2 , . . . , l n ) be the set of word alignments from sentences in e hyp slt to related sentences in e hypmt , where l k contains the word alignments from sentence e k to the relevant sentence e k , 1 ≤ k ≤ n; (e kj , e ki ) = True, if there is one word alignment between e kj and e ki ; (e kj , e ki ) = False, otherwise.

Our algorithm for Method 1 is defined as Algorithm 2. This method relies on word alignments and uses MT labels. We also propose a simpler method in the next section.

Method 2: Subtraction between SLT and MT errors

Our second way to extract 3-class labels (Method 2 ) focuses on the differences between the SLT hypothesis (e hyp slt ) and the MT hypothesis (e hypmt ). We call it subtraction between SLT and MT errors because we simply consider that errors present in SLT and not present in MT are due to ASR. This method has one main difference with the previous one, namely that it does not rely on the extracted labels for MT.

Our intuition is that the number of mt-error s estimated will be slightly lower than for Method 1 since we first estimate the number of asr-error s and the rest is considered -by default -as mt-error s.

We use the same notations as Method 1, except that L = (l 1 , l 2 , . . . , l n ) is the set of alignments through edit distance between e hyp slt and e hypmt , where l ki corresponds to "Insertion", "Substitution", "Deletion" or "Exact". Our algorithm for Method 2 is defined in Algorithm 3.

Example with 3-label setting

Table 15 gives the edit distance between an SLT and an MT hypothesis, while Table 16 shows how Method 1 and Method 2 set 3-class labels to the SLT hypothesis. One transcript (f hyp ) has 1 error. This drives 3 B labels on SLT output (e hyp slt ), while e hypmt has only 2 B labels. As can be seen for Method 1 and Method 2, we respectively have (1 B_ASR, 2 B_MT) and (2 B_ASR, 1 B_MT). These differences might be related to the word alignments from the SLT hypothesis to the relevant MT hypothesis. As Table 16 presents, "is" (SLT hypothesis) is aligned to "have" (MT hypothesis) and "have" (MT hypothesis) is labeled with "B". It can be assumed, therefore, that "is" (SLT hypothesis) should be annotated with word-level labels by B_MT according to Method 1. However, using Method 2, "is" (SLT hypothesis) could be labeled B_ASR because the type of word alignment between "is" (SLT hypothesis) and "have" (MT hypothesis) is substitution (S), as shown in Table 15.

Statistics with 3-label setting on the whole corpus

Tables 17 and18 present the summary statistics for the distribution of good (G), asr-error (B_ASR) and mt-error (B_MT) labels obtained with both labelextraction methods. We see that both methods give similar statistics but slightly different rates of B_ASR and B_MT. Comparing Table 17 and Table 18, it is interesting to note that while the ASR system improves from ASR1 to ASR2, the rate of B_ASR labels logically decreases by more than 2 points, while the rate of B_MT remains almost stable (less than 1 point difference) which makes sense since the MT system is the same in both Table 17 andTable 18. These statistics show that the intersection between both methods is probably a good estimation of disentangling ASR and MT errors in SLT.

Qualitative Analysis of SLT Errors

Our new 3-label setting procedure allows us to analyze the behaviour of our SLT system. We omit examples here, but they are made available as supplementary material to this paper on a Web link. 14 Nonetheless, we can observe sentences with few ASR and MT errors leading to many SLT errors. Indeed, this is a good way of detecting flaws in the SLT pipeline such as bad post-processing of the SLT output (numerical or text dates, for instance). In contrast, there are cases where many ASR errors lead to few SLT errors (ASR errors with few consequences such as morphological substitutions (for instance in French: de/des, déficit/déficits, budgétaire/budgétaires). Finally, some ASR errors have different consequences on SLT quality (on a sample sentence, 2 ASR errors in Systems 1 and 2 lead to 14 and 9 SLT errors, respectively).

Experiments on 3-class error detection

We report in Table 19 our first attempt to build an error-detection system in SLT as a 3-class problem (joint approach only). We conducted our experiment by training and evaluating the model on Intersection(m1, m2) which corresponds to high confidence in the labels. 15 In addition to giving a better informed error detection (B ASR and B M T instead of B), we note that 3-class error detection leads to overall similar results if we backoff to good/bad decision (F avg becomes 62.5 on ASR1 and 61.00 on ASR2 in that case). 15. However, we observed that the use of different label sets (Method 1, Method 2, Intersection(Method 1, Method 2)) does not have a strong influence on the results, so we omit these results here.

Conclusion

Main contributions

In this paper, we introduced a new quality-assessment task: word confidence estimation (WCE) for spoken language translation (SLT). A specific corpus distributed to the research community was built for this purpose. We formalized WCE for SLT and proposed several approaches based on several types of features: MT-based features, ASR-based features, as well as combined or joint features using both ASR and MT information. The proposal of a unique joint classifier based on different feature types (ASR and MT features) allowed us to operate feature selection and analyze which features (from ASR or MT) are the most efficient for quality assessment in speech translation. Our experiments have shown that MT features remain the most influential, while ASR features can bring interesting complementary information. For the purpose of reproducible research, our toolkit has been made available on a GitHub repository under the licence GPL V3. We hope that the availability of our corpus and toolkit could lead, in the near future, to a new shared task dedicated to quality estimation for speech translation. Such a shared task could be proposed in avenues such as IWSLT or WMT for instance. Towards the end of the paper, we proposed to disentangle ASR and MT errors and recast WCE as a 3-label setting problem.

Perspectives

A direct application of this work is the use of WCE labels to re-decode speech-translation graphs and (hopefully) improve speech-translation performance. Preliminary results have already been obtained and published by the authors of this paper [START_REF] Besacier | Spoken language translation graphs re-decoding using automatic quality assessment[END_REF]. The main idea is to carry a second speech translation pass by considering every word and its quality-assessment label, as shown in Equation (4).

In addition to re-decoding SLT graphs, our quality-assessment system can be used in interactive speech-translation scenarios such as news or lecture subtitling, to improve human translator productivity by giving them feedback on automatic transcription and translation quality. Another application would be the adaptation of our WCE system to interactive speech-to-speech translation scenarios where feedback on transcription and translation modules is needed to improve communication. Finally, in this paper, engineered features were used for WCE; a natural perspective is also to learn the WCE features as it is now possible with deep neural networks, for instance.

  Point: an Existing MT Post-Edition Corpus

Figure 1 -

 1 Figure 1 -Evolution of system performance (y-axis -F-mes1 -ASR1) for tst corpus (4050 utt) along decision threshold variation (x-axis). Training is made on dev corpus (2643 utt).

Figure 2 -

 2 Figure 2 -Evolution of system performance (y-axis -F-mes2 -ASR2) for tst corpus (4050 utt) along decision-threshold variation (x-axis). Training is made on dev corpus (2643 utt).

Table 1 -

 1 List of MT features extracted.

	1 Proper Name	10 Stop Word	19 WPP max
	2 Unknown Stem	11 Word context Alignments	20 Nodes
	3 Num. of Word Occ.	12 POS context Alignments	21 Constituent Label
	4 Num. of Stem Occ.	13 Stem context Alignments	22 Distance To Root
	5 Polysemy Count -Target	14 Longest Target N -gram Length 23 Numeric
	6 Backoff Behaviour -Target 15 Longest Source N -gram Length 24 Punctuation
	7 Alignment Features	16 WPP Exact
	8 Occur in Google Translate	17 WPP Any
	9 Occur in Bing Translate	18 WPP min

Table 2 -

 2 Example of training label obtained using TERp-A.

	Reference	The	consequence of	the fundamentalist
		E	S	E	E	S
	Hyp After Shift The	result	of	the hard-line
	Reference	movement		also has its importance	.
		Y	I	E	D	P	E
	Hyp After Shift trend	is	also		important	.

Table 3 -

 3 Details on our dev and tst corpora for SLT.

	Corpus	#sentences #speech recordings #speakers	Duration
	dev	881	2643	15 ( 9 women + 6 men)	5h51
	tst	1350	4050	27 (11 women + 16 men) 11h01

Table 5 -

 5 ASR performance (WER) on our dev and tst set for the two different ASR systems

	Task	dev set tst set
	ASR1 21.86%	17.37%
	ASR2 16.90%	12.50%

Table 6 -

 6 Overview of our post-edition corpus for SLT.

	Data	# dev utt # tst utt # dev words # tst words method to obtain
						WCE labels
	f ref	881	1350	21 988	36 404
	f hyp1	881*3	1350*3	66 435	108 332	wer(f hyp1 , f ref )
	f hyp2	881*3	1350*3	66 834	108 598	wer(f hyp2 , f ref )
	e hypmt	881	1350	22 340	35 213	terpa(e hypmt , e ref )
	e hyp slt1 881*3	1350*3	61 787	97 977	terpa(e hyp slt1 , e ref )
	e hyp slt2 881*3	1350*3	62 213	97 804	terpa(e hyp slt2 , e ref )
	e ref	881	1350	22 342	34 880

Table 7 -

 7 Example of quintuplet with associated labels.

	f ref	quand	notre cerveau chauffe
	f hyp1	comme	notre cerveau chauffe
	labels ASR B	G	G	G
	f hyp2	qu'	entre serbes	au chauffe
	labels ASR B	B	B	B G
	e hypmt	when	our	brains	chauffe
	labels MT	G	G	G	B
	e hyp slt1	as	our	brains	chauffe
	labels SLT B	G	G	B
	e hyp slt2	between serbs in	chauffe
	labels SLT B	B	B	B
	e ref	when	our	brain	heats up

Table 8 -

 8 MT and SLT performances on our dev set.We first report in Table11the baseline WCE results obtained using MT or ASR features separately. In short, we evaluate the performance of 4 WCE

	Task	ASR (WER) MT (BLEU) % G (good) % B (bad)
	MT	0%	49.13%	76.93%	23.07%
	SLT (ASR1) 21.86%	26.73%	62.03%	37.97%
	SLT (ASR2) 16.90%	28.89%	63.87%	36.13%
	5 Experiments on WCE for SLT
	5.1 SLT Quality Assessment Using only MT or ASR Fea-
	tures

Table 9 -

 9 MT and SLT performances on our tst set.

	Task	ASR (WER) MT (BLEU) % G (good) % B (bad)
	MT	0%	57.87%	81.58%	18.42%
	SLT (ASR1) 17.37%	36.21%	70.59%	29.41%
	SLT (ASR2) 12.50%	38.97%	72.61%	27.39%
	systems for different tasks:

Table 10 -

 10 WCE performance baseline (%F G , %F B , %F-mes) on ASR1 and on ASR2 for tst set (random classifier generating G or B)

	task ASR1		WCE for ASR		WCE for SLT
		%F G	%F B	%F-mes %F G	%F B	%F-mes
	Baseline	62.99% 23.11% 43.05%	58.27% 36.70% 47.49%
	task ASR2		WCE for ASR		WCE for SLT
		%F G	%F B	%F-mes %F G	%F B	%F-mes
	Baseline	64.31% 17.65% 40.98%	59.37% 35.56% 47.47%

Table 11 -

 11 WCE performance with different feature sets for tst set (training is made on dev set); *for MT feat, removing OccurInGoogleTranslate and Occur-

	InBingTranslate features lead to 63.09% and 62.33% for F-mes1 and F-mes2,
	respectively.
	task	WCE for ASR WCE for ASR WCE for SLT WCE for SLT
	feat. type ASR feat.	ASR feat.	MT feat.	ASR feat.
		p(q|x f , f )	p(q|x f , f )	p(q|f, e)	p ASR (q|x f , f )
		(CRFs)	(Boosting)		projected to e
	F-mes1	68.71%	64.27%	64.69%*	53.85%
	F-mes2	59.83%	62.61%	64.48%*	48.67%

Table 13 -

 13

	task	WCE for SLT WCE for SLT WCE for SLT WCE for SLT
	feat. type MT+ASR feat.	Joint feat. 1	Joint feat. 2	Joint feat. 3
		p ASR (q|x f , f ) α	p(q|x f , f, e)	p(q|x f , f, e)	p(q|x f , f, e)
		* p M T (q|e, f ) 1-α
	F-mes1	58.07%	64.90%*	64.84%	64.86%
	F-mes2	53.66%	64.17%*	64.11%	63.87%

WCE performance with combined (MT+ASR) or joint (MT, ASR) feature sets for tst set (training is made on dev set); *for Joint 1 feat, removing OccurInGoogleTranslate and OccurInBingTranslate features lead to 63.31% and 62.16% for F-mes1 and F-mes2, respectively.

Table 14 -

 14 Rank of each feature according to the Sequential Backward Selection algorithm on the WCE for SLT task using Joint (ASR,MT) features. Feature selection is applied to the dev corpus for both ASR1 and ASR2. ASR features are in bold.

	Rank	Rank Feature	Rank	Rank Feature
	ASR1 ASR2	ASR1 ASR2
	1	1 Alignment Features	18	20 Unknown Stem
	2	2 Occur in Bing Translate	19	29 Number of Word Occurrences
	3	4 Longest Source N -gram Length	20	28 Polysemy Count -Target
	4	3 WPP Max	21	19 F-dur
	5	6 Occur in Google Translate	22	12 Punctuation
	6	24 F-back	23	21 Constituent Label
	7	11 F-context	24	25 F-word
	8	27 F-alt	25	23 Longest Target N -gram Length
	9	7 Target Backoff Behaviour	26	10 POS Context Alignment
	10	5 Word Context Alignment	27	26 WPP Exact
	11	30 Stem Context Alignment	28	18 WPP Any
	12	31 Numeric	29	22 Proper Name
	13	13 Distance to Root	30	8 Number of Stem Occurrences
	14	9 F-3g	31	16 F-POS
	15	17 Stop Word	32	33 F-post
	16	15 Nodes	33	32 F-log
	17	14 WPP Min

  Algorithm 2 Method 1 : Using word alignments between MT and SLT list_labels_result ← empty_list for each sentence e k ∈ êslt do list_labels_sent ← empty_list for j ← 1 to N umberOf W ords(e k ) do if label(e kj ) = 'G' then add 'G' to list_labels_sent else if Existed Word Alignment (e kj , e ki ) and label(e ki )='B' then

	add 'B_M T ' to list_labels_sent
	else
	add 'B_ASR' to list_labels_sent
	end if
	end for
	add list_labels_sent to list_labels_result
	end for

Table 15 -

 15 Example of edit distance between SLT and MT

	e hyp slt	surgeons in los angeles it	is	said
	e hypmt	surgeons in los angeles ** have said
	edit op. E	E E	E	I	S	E

Table 16 -

 16 Example of quintuplet with 2-label and 3-label.

	f ref	les chirurgiens de	los angeles ont		dit
	f hyp	les chirurgiens de	los angeles on		dit
	labels ASR	G G	G	G	G	B		G
	e hypmt	surgeons	in	los angeles		have	said
	labels MT	G	B	G	G		B	G
	e hyp slt	surgeons	in	los angeles it	is	said
	labels SLT (2-label)	G	B	G	G	B	B	G
	labels SLT (Method 1 ) G	B_MT G	G	B_ASR B_MT	G
	labels SLT (Method 2 ) G	B_MT G	G	B_ASR B_ASR G
	e ref	the surgeons	of	los angeles			said

Table 17 -

 17 Statistics with 3-label setting for ASR1

	Task	dev set %G %B_ASR %B_MT	tst set %G %B_ASR %B_MT
	label/m1:Method 1	62.03	19.09	18.89 70.59	14.50	14.91
	label/m2:Method 2	62.03	22.49	15.49 70.59	16.62	12.79
	label/same(m1, m2) 62.03	18.09	14.49 70.59	13.58	11.88
	label/diff(m1, m2)	0	1.00	4.40	0	0.92	3.03
	Table 18 -Statistics with 3-label setting for ASR2
	Task	dev set %G %B_ASR %B_MT	tst set %G %B_ASR %B_MT
	label/m1:Method 1	63.87	16.89	19.23 72.61	11.92	15.47
	label/m2:Method 2	63.87	19.78	16.34 72.61	13.58	13.81
	label/same(m1, m2) 63.87	16.05	15.50 72.61	11.12	13.01
	label/diff(m1, m2)	0	0.84	3.73	0	0.80	2.46

Table 19 -

 19 Error-Detection Performance (2-labels vs 3-labels) on SLT output for the tst set. Training is performed on the dev set.

	Error	2-class			3-class
	Detection
		ASR1	ASR2		ASR1	ASR2
	F G	81.79	83.17	F G	85.00	85.00
	F B	48.00	45.17	F B_ASR 44.00	42.00
				F B_M T	14.00	15.00
	F avg	64.90	64.17	F avg	47.67	47.33

14. http://tienhuong.weebly.com/examples-for-the-paper.html

cf. http://www.statmt.org/wmt17/ for the most recent such instance.

https://github.com/besacier/WCE-SLT-LIG/

These 3 alternative utterances are simply added to the corpus as 3 examples and are used independently of each other.

Corresponding to optimization of the F-measure on bad labels (errors).

In principle, three data sets would have been needed to (a) train classifiers, (b) apply feature selection, (c) evaluate WCE performance. Since we only have a dev and a tst set, we found this procedure acceptable.


Luc Libralesso

Florian Larramendy

Vincent Jost

Frédéric Maffray

Thibault Honegger

Thibault Honegger How

How to design brains on a chip

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

How to design brains on a chip

LUC LIBRALESSO (*,**) -FLORIAN LARRAMENDY (*) -VINCENT JOST (**) -FRÉDÉRIC MAFFRAY (**) -THIBAULT HONEGGER (*) (*) Univ. Grenoble Alpes, CNRS, LTM, 38000 Grenoble, France (**) Univ. Grenoble Alpes, CNRS, G-SCOP, 38000 Grenoble, France CONTACT:luc.libralesso@grenoble-inp.fr

Typical workflow

Goals

• Chip design is very complex, until now, designs were made by hand

• This work aims to use mathematical and computer tools to automate this process

Why a computer approach ?

Too much constraints and possibilities to be handled by a human

• Lots of choices to make:

-Chamber positions -Lots of possible plannar embeddings of a 3D structure -Channel dimentions (width, length, height)

-Channel routing

• Microfluidic constraints:

-Navier-stokes equations.

∆ Z = ηQ ρg λ in L in W in H 3 in + λ ch L ch W ch H 3 ch + λ out L out W out H 3 out • Fabrication constraints:
-Two chambers should be not too close

• Neural-engineering constraints:

-Channel width and length related to their brain equivalent -Each node contains a specific number of neurons

Chip Architecture

Work done so far

• Formal representation of chip requirements and chip designs

• Semi-automatic chip design from specifications

• Already done on several practical cases:

-Basal ganglia loop (5 nodes)

-Several toy models (2 to 4 nodes)

Perspectives

• Continue to implement brains models

• Study algorithms to optimize chamber placement

• Implement heuristics to route channels automatically

Bibliography

[1] Sung-Joo Lim, Julie A Fiez, and Lori L Holt. How may the basal ganglia contribute to auditory categorization and speech perception?

Frontiers in neuroscience, 8, 2014.