Ngoc-Quang Luong
email: ngoc-quang.luong@imag.fr

Laurent Besacier
email: laurent.besacier@imag.fr

Benjamin Lecouteux
email: benjamin.lecouteux@imag.fr

Word Confidence Estimation and its Integration in Sentence Quality Estimation for Machine Translation

Keywords: Machine Translation, Confidence Measure, Confidence Estimation, Conditional Random Fields, Boosting

This paper proposes some ideas to build an effective estimator, which predicts the quality of words in a Machine Translation (MT) output. We integrate a number of features of various types (systembased, lexical, syntactic and semantic) into the conventional feature set, for our baseline classifier training. Once having experiments with all features, we deploy a "Feature Selection" strategy to filter the best performing ones. Then, a method that combines multiple "weak" classifiers to build a strong "composite" classifier by taking advantage of their complementarity allows us achieve a better performance in term of F score. Finally, we exploit word confidence scores for improving the estimation system at sentence level.

Introduction

Statistical Machine Translation (SMT) systems in recent years have marked impressive breakthroughs with numerous fruitful achievements, as they produced more and more user-acceptable outputs. Nevertheless the users still face with some open questions: are these translations ready to be published as they are? Are they worth to be corrected or do they require retranslation? It is undoubtedly that building a method which is capable of pointing out the correct parts as well as detecting the translation errors in each MT hypothesis is crucial to tackle these above issues. If we limit the concept "parts" to "words", the problem is called Word-level Confidence Estimation (WCE). The WCE's objective is to judge each word in the MT hypothesis as correct or incorrect by tagging it with an appropriate label. A classifier which has been trained beforehand calculates the confidence score for the MT output word, and then compares it with a pre-defined threshold. All words with scores that exceed this threshold are categorized in the Good label set; the rest belongs to the Bad label set.

The contributions of WCE for the other aspects of MT are incontestable. First, it assists the post-editors to quickly identify the translation errors, determine whether to correct the sentence or retranslate it from scratch, hence improve their productivity. Second, the confidence score of words is a potential clue to re-rank the SMT N-best lists. Last but not least, WCE can also be used by the translators in an interactive scenario [START_REF] Gandrabur | Confidence Estimation for Text Prediction[END_REF].

This article conveys ideas towards a better word quality prediction, including: novel features integration, feature selection and Boosting technique. It also investigates the usefulness of using WCE in a sentence-level confidence estimation (SCE) system. After reviewing some related researches in Section 2, we depict all the features used for the classifier construction in Section 3. The settings and results of our preliminary experiments are reported in Section 4. Section 5 explains our feature selection procedure. Section 6 describes the Boosting method to improve the system performance. The role of WCE in SCE is discussed in Section 7. The last section concludes the paper and points out some perspectives.

Previous Work Review

To cope with WCE, various approaches have been proposed, aiming at two major issues: features and model to build the classifier. In [START_REF] Blatz | Confidence Estimation for Machine Translation[END_REF], the authors combine a considerable number of features, then train them by the Neural Network and naive Bayes learning algorithms. Among these features, Word Posterior Probability (henceforth WPP) proposed by [START_REF] Ueffing | Confidence Measures for Statistical Machine Translation[END_REF] is shown to be the most effective system-based features. Moreover, its combination with IBM-Model 1 features is also shown to overwhelm all the other ones, including heuristic and semantic features [START_REF] Blatz | Confidence Estimation for Machine Translation[END_REF].

A novel approach introduced in [START_REF] Ueffing | Word-level Confidence Estimation for Machine Translation Using Phrased-based Translation Models[END_REF] explicitly explores the phrase-based translation model for detecting word errors. A phrase is considered as a contiguous sequence of words and is extracted from the word-aligned bilingual training corpus. The confidence value of each target word is then computed by summing over all phrase pairs in which the target part contains this word. Experimental results indicate that the method yields an impressive reduction of the classification error rate compared to the state-of-the-art on the same language pairs. Xiong et al. [START_REF] Xiong | Error Detection for Statistical Machine Translation Using Linguistic Features[END_REF] integrate the POS of the target word with another lexical feature named "Null Dependency Link" and train them by Maximum Entropy model. In their results, linguistic features sharply outperform WPP feature in terms of F-score and classification error rate. Similarly, 70 linguistic features guided by three main aspects of translation: accuracy, fluency and coherence are applied in [START_REF] Felice | Linguistic Features for Quality Estimation[END_REF]. Results reveal that these features are helpful, but need to be carefully integrated to reach better performance.

Unlike most of previous work, the authors in [START_REF] Soricut | Trustrank: Inducing Trust in Automatic Translations via Ranking[END_REF] apply solely external features with the hope that their classifier can deal with various MT approaches, from statistical-based to rule-based. Given a MT output, the BLEU score is predicted by their regression model. Results show that their system maintains consistent performance across various language pairs. Nguyen et al. [START_REF] Nguyen | Goodness: A Method for Measuring Machine Translation Confidence[END_REF] study a method to calculate the confidence score for both words and sentences relied on a feature-rich classifier. The novel features employed include source side information, alignment context, and dependency structure. Their integration helps to augment marginally in F-score as well as the Pearson correlation with human judgment.

Features

This section depicts in detail 25 features exploited to train our classifier. Among them, those marked with a ℗ symbol are proposed by us, and the remaining comes from the previous work.

3.1 System-based Features (directly extracted from SMT system) Target Side Features We take into account the information of every word (at position i in the MT output), including:

-The word itself.

-The sequences formed between it and a word before (i -1/i) or after it (i/i + 1). -The trigram sequences formed by it and two previous and two following words (including: i -2/i -1/i; i -1/i/i + 1; and i/i + 1/i + 2). -The number of occurrences in the sentence. Source Side Features Using the alignment information, we can track the source words which the target word is aligned to. To facilitate the alignment representation, we apply the BIO1 format: if multiple target words are aligned with one source word, the first word's alignment information will be prefixed with symbol "B-" (means "Begin"); meanwhile "I-" (means "Inside") will be added at the beginning of the alignment information for each of the remaining ones. The target words which are not aligned with any source word will be represented as "O" (means "Outside"). Table 1 shows an example for this representation, in Table 1. Example of using BIO format to represent the alignment information case of the hypothesis is "The public will soon have the opportunity to look again at its attention.", given its source: "Le public aura bientôt l'occasion de tourner à nouveau son attention.". Since two target words "will" and "have" are aligned to "aura" in the source sentence, the alignment information for them will be "B-aura" and "I-aura" respectively. In case a target word has multiple aligned source words (such as "again"), we separate these words by the symbol "|" after putting the prefix "B-" at the beginning.

Alignment Context Features These features are proposed by [START_REF] Nguyen | Goodness: A Method for Measuring Machine Translation Confidence[END_REF] in regard with the intuition that collocation is a believable indicator for judging if a target word is generated by a particular source word. We also apply them in our experiments, containing:

-Source alignment context features: the combinations of the target word and one word before (left source context) or after (right source context) the source word aligned to it.

-Target alignment context features: the combinations of the source word and each word in the window ±2 (two before, two after) of the target word.

For instance, in case of "opportunity" in Table 1, the source alignment context features are: "opportunity/l' " and "opportunity/de"; while the target alignment context features are: "occasion/have", "occasion/the", "occasion/opportunity", "occasion/to" and "occasion/look".

Word Posterior Probability WPP [START_REF] Ueffing | Confidence Measures for Statistical Machine Translation[END_REF] is the likelihood of the word occurring in the target sentence, given the source sentence. To calculate it, the key point is to determine sentences in N-best lists that contain the word e under consideration in a fixed position i. In this work, we exploit the graph that represents MT hypotheses [START_REF] Ueffing | Generation of Word Graphs in Statistical Machine Translation[END_REF]. From this, the WPP of word e in position i (denoted by WPP exact) can be calculated by summing up the probabilities of all paths containing an edge annotated with e in position i of the target sentence. Another form is "WPP any" in case we sum up the probabilities of all paths containing an edge annotated with e in any position of the target sentence. In this paper, both forms are employed.

Graph topology features They are based on the N-best list graph merged into a confusion network. On this network, each word in the hypothesis is labelled with its WPP, and belongs to one confusion set. Every completed path passing through all nodes in the network represents one sentence in the N-best, and must contain exactly one link from each confusion set. Looking into a confusion set (which the hypothesis word belongs to), we find some information that can be the useful indicators, including: the number of alternative paths it contains (called Nodes℗) , and the distribution of posterior probabilities tracked over all its words (most interesting are maximum and minimum probabilities, called Max ℗ and Min℗) . We assign three above numbers as features for the hypothesis word.

Language Model Based Features Applying SRILM toolkit [START_REF] Stolcke | Srilm -an Extensible Language Modeling Toolkit[END_REF] on the bilingual corpus, we build 4-gram language models for both target and source side, which permit to compute two features: the "longest target n-gram length"℗ and "longest source n-gram length"℗ (length of the longest sequence created by the current word and its previous ones in the language model). For example, with the target word w i : if the sequence w i-2 w i-1 w i appears in the target language model but the sequence w i-3 w i-2 w i-1 w i does not, the n-gram value for w i will be 3. The value set for each word hence ranges from 0 to 4. Similarly, we compute the same value for the word aligned to w i in the source language model. Additionally, we consider also the backoff behaviour [START_REF] Raybaud | This sentence is wrong. Detecting errors in machine-translated sentences[END_REF] of the target language model to the word w i , according to how many time it has to back-off in order to assign a probability to the word sequence.

Lexical Features

A prominent lexical feature that has been widely explored in WCE researches is word's Part-Of-Speech (POS). We use TreeTagger2 toolkit for POS annotation task and obtain the following features for each target word:

-Its POS -Sequence of POS of all source words aligned to it -Bigram and trigram sequences between its POS and the POS of previous and following words. Bigram sequences are P OSi-1, P OSi and P OSi, P OSi+1 and trigram sequences are: P OSi-2, P OSi-1, P OSi; P OSi-1, P OSi, P OSi+1 and P OSi, P OSi+1, P OSi+2

In addition, we also build four other binary features that indicate whether the word is a: stop word (based on the stop word list for target language), punctuation symbol, proper name or numerical.

Syntactic Features

The syntactic information about a word is a potential hint for predicting its correctness. If a word has grammatical relations with the others, it will be more likely to be correct than those which has no relation. In order to obtain the links between words, we select the Link Grammar Parser3 as our syntactic parser, allowing us to build a syntactic structure for each sentence in which each pair of grammar-related words is connected by a labeled link. Based on this structure, we get a binary feature called"Null Link": 0 in case of word has at least one link with the others, and 1 if otherwise. Another benefit yielded by this parser is the "constituent" tree, representing the sentence's grammatical structure (showing noun phrases, verb phrases, etc.). This tree helps to produce more word syntactic features, including its constituent label ℗ and its depth in the tree℗ (or the distance between it and the tree root).

Figure 1 represents the syntactic structure as well as the constituent tree for a MT output: "The government in Serbia has been trying to convince the West to defer the decision until by mid 2007.". It is intuitive to observe that the words in Fig. 1. Example of parsing result generated by Link Grammar brackets (including "until " and "mid ") have no link with the others, meanwhile the remaining ones have. For instance, the word "trying" is connected with "to" by the link "TO" and with "been" by the link "Pg*b". Hence, the value of "Null Link" feature for "mid " is 1 and for "trying" is 0. The figure also brings us the constituent label and the distance to the root of each word. In case of the word "government", these values are "NP" and "2", respectively.

Semantic Features

The word semantic characteristic that we study is its polysemy. We hope that the number of senses of each target word given its POS can be a reliable indicator for judging if it is the translation of a particular source word. The feature "Polysemy count"℗ is built by applying a Perl extension named Lingua::WordNet4 , which provides functions for manipulating WordNet database.

Baseline WCE Experiments

4.1 Experimental Settings SMT System Our French -English SMT system is constructed using the Moses toolkit [START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF]. We keep the Moses's default setting: log-linear model with 14 weighted feature functions. The translation model is trained on the Europarl and News parallel corpora used for WMT105 evaluation campaign (1,638,440 sentences). Our target language model is a standard n-gram language model trained using the SRI language modeling toolkit [START_REF] Stolcke | Srilm -an Extensible Language Modeling Toolkit[END_REF] on the news monolingual corpus (48,653,884 sentences). More details on this system can be found in [START_REF] Potet | The LIG Machine Translation System for WMT[END_REF].

Corpus Preparation We used our SMT system to obtain the translation hypothesis for 10,881 source sentences taken from news corpora of the WMT evaluation campaign (from 2006 to 2010). Our post-editions were generated by using a crowdsourcing platform: Amazon Mechanical Turk [START_REF] Potet | Collection of a Large Database of French-English SMT Output Corrections[END_REF]. We extract 10,000 triples (source, hypothesis and post edition) to form the training set, and keep the remaining 881 triples for the test set.

Word Label Setting This task is performed by TERp-A toolkit [START_REF] Snover | Terp System Description[END_REF]. Table 2 illustrates the labels generated by TERp-A for one hypothesis and reference pair. Each word or phrase in the hypothesis is aligned to a word or phrase in the reference with different types of edit: I (insertions), S (substitutions), T (stem matches), Y (synonym matches), and P (phrasal substitutions). The lack of a symbol indicates an exact match and will be replaced by E thereafter. We do not consider the words marked with D (deletions) since they appear only in the reference. Then, to train a binary classifier, we re-categorize the obtained 6-label set into binary set: The E, T and Y belong to the Good (G), whereas the S, P and I belong to the Bad (B) category. Finally, we observed that out of total words (train and test sets) are 85% labeled G, 15% labeled B.

Reference

The consequence of the fundamentalist movement also has its importance . S S Y I D P Hyp After Shift The result of the hard-line trend is also important .

Table 2. Example of training label obtained using TERp-A.

Classifier Selection We apply several conventional models, such as: Decision Tree, Logistic Regression and Naive Bayes using KNIME platform6 . However, since our intention is to treat WCE as a sequence labeling task, we employ also the Conditional Random Fields (CRF) model [START_REF] Lafferty | Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data[END_REF]. Among CRF based toolkits, we selected WAPITI [START_REF] Lavergne | Practical Very Large Scale CRFs[END_REF] to train our classifier. We also compare our classifier with two naive baselines: in Baseline 1, all words in each MT hypothesis are classified into G label. In Baseline 2, we assigned them randomly: 85% into G and 15% into B label (similar to the percentage of these labels in the corpus).

Preliminary Results and Analysis

We evaluate the performance of our classifier by using common evaluation metrics: Precision (Pr), Recall (Rc) and F-score (F). We perform our preliminary experiment by training a CRF classifier with the combination of all 25 features. The classification task is then conducted multiple times, corresponding to a threshold increase from 0.300 to 0.975 (step = 0.025). When threshold = α, all words in the test set which the probability for G class exceeds α will be labelled as "G", and otherwise, "B". The values of Pr and Rc for G and B label are tracked along this threshold variation. The results observed show that in case of B label, Rc increases gradually from 0.285 to 0.492 whereas Pr falls from 0.438 to 0.353. With G label, the variation occurs in the opposite direction: Rc drops almost regularly from 0.919 to 0.799, meanwhile Pr augments slightly from 0.851 to 0.876. Table 3. Average Pr, Rc and F for labels of all-feature system and two baselines.

these labels in the all-feature system and the baseline systems (correspond to the above threshold variation). These values imply that in our system: (1) Good label is much better predicted than Bad label, (2) The combination of features helped to detect the translation errors significantly above the "naive" baselines.

In an attempt of investigating the performance of CRF model, we compare it ones, so it is fair to say that with our classifiers, detecting a translation error should be more appreciated than identifying a good translated word. Therefore, we propose a "composite" score called F * putting more weight on the system capability of detecting B words: F * = 0.70 * F score(B) + 0.30 * F score(G). We track all scores along to the threshold variation and then plot them in Figure 2.

The topmost position of CRF curve shown in the figure reveals that the CRF model performs better than all the remaining ones, and it is more suitable to deal with our features and corpus. Another notable observation is that the "optimal" threshold (which gives the best F * ) for each classifier is different from the others: 0.975 for CRF, 0.925 for Decision Tree, 0.800 for Logistic Regression and 0.300 for Naive Bayes classifier. In the next sections, which propose ideas to improve the prediction capability, we work only with the CRF classifier.

Feature Selection for WCE

In Section 4, the all-feature system yielded promising F scores for G label, but not very convincing F scores for B label. That can be originated from the risk that not all of features are really useful, or in other words, some are poor predictors and might be the obstacles weakening the other ones. In order to prevent this drawback, we propose a method to filter the best features based on the "Sequential Backward Selection" algorithm 7 . We start from the full set of N features, and in each step sequentially remove the most useless one. To do that, all subsets of (N-1) features are considered and the subset that leads to the best performance gives us the weakest feature (not included in the considered set). Obviously, the discarded feature is not considered in the following steps. We iterate the process until there is only one remaining feature in the set, and use the following score for comparing systems: F avg (all) = 0.30 * F avg (G) + 0.70 * F avg (B), where F avg (G) and F avg (B) are the averaged F scores for G and B label, respectively, when threshold varies from 0.300 to 0.975. This strategy enables us to sort the features by descending order of importance, as displayed in Table 4. Figure 3 shows the evolution of the WCE performance as more and more features are removed, and the details of 3 best feature subsets yielding the highest F-scores. 4. The rank of each feature (in term of usefulness) in the set. The letter represents category: "S" for system-based , "L" for lexical, "T" for syntactic, and "M" for semantic feature; and the symbol "*" indicates our proposed features. 4 reveals that in our system, system-based and lexical features seemingly outperform the other types in terms of usefulness, since in top 10, they contribute 8 (5 system-based + 3 lexical). However, 2 out of 3 syntactic features appear in top 10, indicating that their role cannot be disdained. Observation in 10-best and 10-worst performing features suggests that features belonging to word origin (word itself, POS) perform very well, meanwhile those from word statistical knowledge sources (target and source language models) are much less beneficial. In addition, in Figure 3, when the size of feature set is small (from 1 to 7), we can observe sharply the growth of the system performance (F avg (all)). Nevertheless the scores seem to saturate as the feature set increases from 8 up to 25. This phenomenon raises a hypothesis about our classifier's learning capability when coping with a large number of features, hence drives us to an idea for improving the classification scores. This idea is detailed in the next section.

Classifier Performance Improvement Using Boosting

The results and observations in Section 5 lead to a question: If we build a number of "weak" (or "basic") classifiers by using subsets of our features and a machine learning algorithm (such as Boosting), should we get a single "strong" classifier? If deploying this idea, our hope is that multiple models can complement each other as one feature set might be specialized in a part of the data where the others do not perform very well.

First, we prepare 23 feature subsets (F 1 , F 2 , ..., F 23 ) to train 23 basic classifiers, in which: F 1 contains all features, F 2 is the Top 17 in Table 4 and F i (i = 3..23) contains 9 randomly chosen features. Next, the 10-fold cross validation is applied on our usual 10K training set. We divide it into 10 equal subsets (S 1 , S 2 , . . . , S 10 ). In the loop i (i = 1..10), S i is used as the test set and the remaining data is trained with 23 feature subsets. After each loop, we obtain the results from 23 classifiers for each word in S i . Finally, the concatenation of these results after 10 loops gives us the training data for Boosting. The detail of this algorithm is described below: 5. Comparison of the average Pr, Rc and F between CRF and Boosting systems scores suggest that using Boosting algorithm on our CRF classifiers' output is an efficient way to make them predict better: on the one side, we maintain the already good achievement on G class (only 0.05% lost), on the other side we augment 2.89% the performance in B class. It is likely that Boosting enables different models to better complement one another, in terms of the later model becomes experts for instances handled wrongly by the previous ones. Another advantage is that Boosting algorithm weights each model by its performance (rather than treating them equally), so the strong models (come from all features, top 17, etc.) can make more dominant impacts than the others.

Using WCE in Sentence Confidence Estimation (SCE)

WCE helps not only in detecting translation errors, but also in improving the sentence level prediction when combined with other sentence features. To verify this, firstly we build a SCE system (called SYS1) based on our WCE outputs (prediction labels). The seven features used to train SYS1 are:

-The ratio of number of good words to total number of words. (1 feature) -The ratio of number of good nouns to total number of nouns. The similar ones are also computed for other POS: verb, adjective and adverb. (4 features) -The ratio of number of n consecutive good word sequences to total number of consecutive word sequences. Here, n=2 and n=3 are applied. (2 features)

Then, we inherit the script used in WMT12 9 for extracting 17 sentence features, to build an another SCE system (SYS2). In both SYS1 and SYS2, each sentence training label is an integer score from 1 to 5, based on its TER score, as following:

score(s) =                5 if TER(s) ≤ 0.1 4 if 0.1 < TER(s) ≤ 0.3 3 if 0.3 < TER(s) ≤ 0.5 2 if 0.5 < TER(s) ≤ 0.7 1 if TER(s) > 0.7 (1)
Two conventional metrics are employed to measure the SCE system's performance: Mean Absolute Error (MAE) and Root of Mean Square Error (RMSE) 10 .

To observe the impact of WCE on SCE, we design a third system (called SYS1+SYS2), which takes the results yielded by SYS1 and SYS2, post-processes them and makes final decision. For each sentence, SYS1 and SYS2 generate five probabilities for five integer labels it can be assigned, then select the label which highest probability as official result. Meanwhile, SYS1+SYS2 collects probabilities come from both systems, then updates the probability for each label by the sum of two appropriate values in SYS1 and SYS2. Similarly, the label with highest likelihood is assigned to this sentence. The results are shown in Table 6 on our WCE are separately exploited, they yield acceptable performance. And, their contributions are definitively proven when they are combined with WMT12 features: The combination system SYS1+SYS2 sharply reduces MAE and RMSE of both single systems. It demonstrates that in order to judge effectively a sentence, besides global and general indicators, information synthesized from the quality of each word are very useful.

Conclusions and Perspectives

We proposed some ideas to deal with WCE for MT, starting with the integration of our proposed features into the existing features to build the classifier. The first experiment's results show that precision and recall obtained in G label are very promising, and B label reaches acceptable performance. A feature selection strategy is then deployed to identify the valuable features, find out the best performing subset. One more contribution we made is the protocol of applying Boosting algorithm, training multiple "weak" classifiers, taking advantage of their complementarity to get a "stronger" one. Especially, the integration with SCE enlightens the WCE contribution in judging the sentence quality.

In the future, we will take a deeper look into linguistic features of word, such as the grammar checker, dependency tree, semantic similarity, etc. Besides, we would like to investigate the segment-level confidence estimation, which exploits the context relation between surrounding words to make the prediction more accurate. Moreover, a methodology to conclude the sentence confidence relied on the word-and segment-level confidence will be also deeply considered.

Fig. 2 .

 2 Fig. 2. Performance comparison (F * ) among different classifiers with several other models, as stated in Section 4.1. The pivotal problem is how to define an appropriate metric to compare them efficiently? Due to the fact that in our training corpus, the number of G words sharply outperforms the B

Fig. 3 .

 3 Fig. 3. Evolution of system performance (Favg(all)) during Feature Selection process Table4reveals that in our system, system-based and lexical features seemingly outperform the other types in terms of usefulness, since in top 10, they contribute 8 (5 system-based + 3 lexical). However, 2 out of 3 syntactic features appear in top 10, indicating that their role cannot be disdained. Observation in 10-best and 10-worst performing features suggests that features belonging to word origin (word itself, POS) perform very well, meanwhile those from word statistical knowledge sources (target and source language models) are much less beneficial. In addition, in Figure3, when the size of feature set is small (from 1 to 7), we can observe sharply the growth of the system performance (F avg (all)). Nevertheless the scores seem to saturate as the feature set increases from 8 up to 25. This phenomenon raises a hypothesis about our classifier's learning capability when coping with a large number of features, hence drives us to an idea for improving the classification scores. This idea is detailed in the next section.

  Algorithm to build Boosting training datafor i :=1 to 10 do begin TrainSet(i) := ∩Sj (j = 1..10, j = i) TestSet(i) := Si for j := 1 to 23 do begin Classifier Cj := Train TrainSet(i) with Fj Result Rj := Use Cj to test Si Column Pj := Extract the "probability of word to be G label" in Rj end Subset Di (23 columns) := {Pj} (j = 1..23) end Boosting training set D := ∩Di (i = 1..10)Next, Bonzaiboost toolkit 8 (based on decision trees and implements Boosting algorithm) is used for building Boosting model. In the training command, we invoked: algorithm = "AdaBoost", and number of iterations = 300. The Boosting test set is prepared as follows: we train 23 feature sets with the usual 10K training set to obtain 23 classifiers, then use them to test the CRF test set, finally extract the 23 probability columns (like in the above pseudo code). In the testing phase, similar to what we did in Section 5, the averaged Pr, Rc and F scores against threshold variation for G and B labels are tracked as seen in

  Table 3 reports the average values of Precision, Recall and F-score of

	System	Label Pr(%) Rc(%) F(%)
	All features Good 85.99 88.18 87.07
		Bad	40.48 35.39 37.76
	Baseline 1 Good 81.78 100.00 89.98
		Bad	-	0	-
	Baseline 2 Good 81.77 85.20 83.45
		Bad	18.14 14.73 16.26

Table 5 .

 5 The Boosting 90.10 84.13 87.02 34.33 49.83 40.65 CRF (all) 85.99 88.18 87.07 40.48 35.39 37.76 Table

	System Pr(G) Rc(G) F(G) Pr(B) Rc(B) F(B)

Table 6 .

 6 . Scores observed reveal that when WMT12 baseline features and those based Scores of 3 different SCE systems.

	System	MAE RMSE
	SYS1	0.5584 0.9065
	SYS2	0.5198 0.8707
	SYS1+SYS2 0.4835 0.8415

http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/

http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/

http://www.link.cs.cmu.edu/link/

http://search.cpan.org/dist/Lingua-Wordnet/Wordnet.pm

http://www.statmt.org/wmt10/

http://www.knime.org/knime-desktop

http://research.cs.tamu.edu/prism/lectures/pr/pr l11.pdf

http://bonzaiboost.gforge.inria.fr/x1-20001

https://github.com/lspecia/QualityEstimation/blob/master/baseline system

http://www.52nlp.com/mean-absolute-error-mae-and-mean-square-error-mse/


N El Ghossein

A Sari

P Venet

Interpretation of the Particularities of Lithium-Ion Capacitors and Development of a Simple Circuit Model

Keywords: Lithium-ion capacitor, Electrical model, Energy storage, Nonlinear capcitance evolution, Electrochemical behavior

The expeditious development of electric vehicles and hybrid electric vehicles relies on using appropriate energy storage systems such as supercapacitors and lithium-ion batteries. The new technology that combines both conventional energy storage systems is the lithium-ion capacitor. The aim of this component is to fill the gaps between supercapacitor's low energy density and lithium-ion battery's low power density. This paper presents the main electrical characteristics, deduced from frequency domain measurements, of such a technology. The nonlinear capacitance evolution of a lithium-ion capacitor cell with respect to the voltage was analyzed based on its physicochemical properties. Moreover, the lithium-ion capacitor was modeled using a simple model derived from the frequency measurements.

INTRODUCTION

Automobiles used in people's daily lives produce emissions that are extremely harmful. The cumulative effects of exhaust gas threaten the environment and human life severely. Thus, vehicle electrification becomes inevitable in order to reduce the amount of undesired pollutants generated by the combustion of fossil fuels. The environmentally friendly energy storage systems are integrated into transport facilities in order to struggle against the pollution.

Lithium-ion batteries have proven propitious properties such as high specific energy and high power density compared to other battery types. Among this family of batteries exist many technologies that are related to the positive and negative electrodes' materials: lithium-iron sulphide (FeS), lithiumtitanate (LiTiO/NiMnO2), lithium-iron phosphate (LiFePO4), lithium-ion polymer (LiPo), … [START_REF] Tie | A review of energy sources and energy management system in electric vehicles[END_REF]. They are characterized by their low discharge rate and high energy density. Because of their significant energy and power capabilities, they are suitable for several applications such as Plug-in Hybrid Electric Vehicles (PHEV) [START_REF] Vazquez | Energy storage systems for transport and grid applications[END_REF]. For further increase in fuel economy, Electrochemical Double Layer Capacitors (EDLCs), known as supercapacitors, were introduced. They provide a power density higher than that of Lithium-Ion Batteries (LIBs). This is due to their low resistance less than 1 mΩ. On the other hand, they possess a long cycle life and they have proven a better behavior in negative temperatures than LIBs. However, their energy capabilities are low [START_REF] Burke | Ultracapacitor technologies and application in hybrid and electric vehicles[END_REF].

Therefore, the search for a hybrid energy storage system that combines the advantages of both technologies had risen during the past years. Several energy management strategies for associating EDLCs with LIBs in one system have been carried out in the field of transportation [START_REF] Allegre | Different energy management strategies of hybrid energy storage system (HESS) using batteries and supercapacitors for vehicular applications[END_REF][START_REF] Cao | A new battery/ultra-capacitor hybrid energy storage system for electric, hybrid and plug-in hybrid electric vehicles[END_REF]. In terms of power supply, the role of the battery concerned the average power while the EDLC supplied the change in power. This would increase the lifetime of the battery and decrease the cost of the system [START_REF] Dougal | Power and life extension of battery/ultracapacitor hybrids[END_REF][START_REF] Lahyani | Optimal hybridization and amortized cost study of battery/supercapacitors system under pulsed loads[END_REF].

Instead of using two distinct systems, Lithium-Ion Capacitors (LICs) are the new technology that aims to deliver high energy and high power while assuring a long cycle life. Its composition is inspired from that of EDLCs and LIBs; thus, it provides features intermediate between both systems [START_REF] Ronsmans | Combining energy with power: Lithiumion capacitors[END_REF]. Based on the Ragone plot (Fig. 1) describing energy and power densities of energy storage systems present in the market, LICs possess characteristics closer to those of EDLCs. However, their energy densities are higher since their maximum voltage (3.8 V) is greater than that of EDLCs (usually 2.7 V). This paper presents in section II the operating principle of a LIC and analyzes its electrical behavior based on data collected from measurements in the frequency domain. In order to simulate this behavior of a LIC, it is modelled using an electrical model in section III.

The tested product is the Ultimo prismatic cell shown in Fig. 2. This LIC is developed by JM Energy and has the properties summarized in Table I.

Capacity (F) 3300

Energy density (Wh/kg) 13

Equivalent series resistance at 1 kHz (mΩ) 0.7

Power density (kW/kg) 10

Maximum charge current (A) 125

Maximum discharge current (A) 360

II. OPERATING PRINCIPLE AND ELECTRICAL CHARACTERISTICS

The positive electrode of a LIC is similar to the positive electrode of an EDLC. Similarly, the negative electrode of a LIC is the same as that of the majority of LIBs. Therefore, in order to understand the main electrochemical processes happening in a LIC, it is important to recall the ones taking place in an EDLC and in a LIB as in [START_REF] German | Improved supercapacitor floating ageing interpretation through multi-pore impedance model parameters evolution[END_REF][START_REF] Smith | Electrochemical control of lithium-ion batteries [applications of control[END_REF].

A. Working Principle

A commercial LIC cell is composed of a positive electrode of activated carbon and a negative electrode of a carbon material pre-doped with lithium. The pre-lithiation of carbon decreases the electrode's potential in order to increase the cell's overall voltage. There are several methods used for intercalating lithium ions into carbon layers and the level of pre-lithiation has an important impact on the cell specific capacitance and cycle stability [START_REF] Sivakkumar | Evaluation of lithium-ion capacitors assembled with pre-lithiated graphite anode and activated carbon cathode[END_REF][START_REF] Kumagai | Cycle performance of lithiumion capacitors using graphite negative electrodes at different prelithiation levels[END_REF]. Both electrodes are placed in an organic electrolyte containing a lithium salt: 1M LiPF6 in 1:1 mixture of ethyl carbonate (EC) and dimethyl carbonate (DMC). The same configuration of the electrolyte is usually used in LIBs. A separator between electrodes is employed to isolate the association electronically while allowing the migration of ions.

During the charge from 2.2 V to 3 V, the cations Li + are desorbed from the surface of pores of the activated carbon and intercalated into the carbon layers. From 3 V to 3.8 V, the anions PF6 -are adsorbed on the surface of the activated carbon positive electrode while the cations Li + are further intercalated into the negative electrode of carbon as seen in Fig. 3 [START_REF] Sivakkumar | Evaluation of lithium-ion capacitors assembled with pre-lithiated graphite anode and activated carbon cathode[END_REF]. During the discharge, the opposite process happens. Therefore, for a LIC, at the positive electrode a non-faradaic process occurs while at the negative electrode faradaic reactions take place according to the following equation:

+ + -+6 ⇋ x 6 (1)
The voltage at the negative electrode barely changes during the charge and the discharge of the cell. Thus, the adsorption and desorption of ions at the surface of the positive electrode have the most significant effect on voltage and capacitance changes of a LIC.

B. Electrical Characteristics

The performance of LICs was investigated using frequency domain measurements. Impedance data were collected in order to analyze the electrical behavior of this technology.

Galvanostatic Electrochemical Impedance Spectroscopy (GEIS) measurements were applied to a LIC using a sinusoidal current having an amplitude of 5 A in the frequency range of 10 mHz to 100 kHz at 25°C. The well-defined protocol in Fig. 4 for the succession of measurements at five voltage values is followed in order to increase the accuracy of the results. The potential of the cell, at which the GEIS is applied, is maintained at a Constant Voltage (CV) for 30 min before starting the measurement process. In addition, reaching the required voltage at each measuring step is done by charging the LIC using a Constant Current (CC) of 10 A. In order to show the measured complex impedance, Nyquist plots, in Fig. 5, illustrate the opposite of the imaginary part of the impedance with respect to its real part for five voltage values. At high frequencies, the voltage does not have a major influence on the impedance values. However, at low frequencies, both imaginary and real parts of the impedance depend on the voltage. Therefore, in order to investigate furthermore the influence of the cell's potential, additional measurements were done for a step voltage of 0.1 V in the range of 2.2-3.8 V.

Nyquist plots for different cell potentials are illustrated in Fig. 6 at low frequencies. Observing the impedance for different potentials leads to evaluating the changes of the capacity as a function of cell's potentials. Fig. 7 shows the resulting graph of the measured capacity at 10 mHz with respect to the voltage. Considering both electrodes as two capacitances in series, the total capacitance C t of the LIC cell is expressed by: -

1 1 1 t C C C + = + ( 2)
where, C -is the capacitance of the negative electrode and C + is the capacitance of the positive electrode. However, the phenomena of the Li + intercalation and deintercalation into carbon layers are very slow compared to the adsorption and desorption of ions at the surface of the activated carbon [START_REF] Sivakkumar | Rate capability of graphite materials as negative electrodes in lithium-ion capacitors[END_REF].Therefore, the change in C t is mainly affected by the variation of C + if the changes in C -are considered negligible. From Fig. 7, one can conclude that while charging the LIC from 2.2 V to 3 V, C t decreases with voltage increase. According to the working principle explained before, in this range of voltage, the cations Li + leave the surface of the activated carbon thus C + diminishes. This explains the decrease in C t . In addition, charging the LIC from 3 V to 3.8 V causes the increase of C t which seems also related to the operating principle stating that within this range of voltage the anions are adsorbed on the surface of the positive electrode. Thus, the capacitance of the LIC is increased.

In order to study the effects of high temperatures on the dependence of the capacitance on the cell's voltage, GEIS measurements were done at 40°C and 60°C for the five potential values used before. The LIC cell was placed in a climatic chamber for 4 hours at the temperature of the test prior to the start of the measurement procedure. Results of the measured capacity at 10 mHz were compared to the values found previously at 25°C as seen in Fig. 8. The decrease of the capacity in the voltage window of 2.2 V to 3 V and its increase in the voltage window of 3 V to 3.8 V are also observed at high temperatures. However, at 2.2 V, the capacitance of the LIC decreases with the temperature increase while at 3 V and 3.8 V an increase in the capacitance value is detected. This confirms the particularity of the behavior of the LIC. In fact, in the case of EDLCs, high temperatures induce a small increase in the capacitance for all potentials [START_REF] Diab | Electrical, frequency and thermal measurement and modelling of supercapacitor performance[END_REF].

III. ELECTRICAL MODELLING

For testing the possibility of integrating this technology in applications requiring management systems, it is important to model LIC cells. Several models of LIBs and EDLCs were used to monitor their respective behaviors. The efficiency of the model is related to its ability to describe the performance of the system under different operating conditions.

An EDLC during charge can be modeled by a simple RC circuit. The capacitance of the EDLC is considered to be linear with respect to the voltage. In the case of a LIC, the same model can be used during charge as seen in Fig. 9. However, the capacitance is not assumed to be linear with the voltage according to Fig. 7. The impedance of the LIC Z LIC (f,V LIC ,T) depends on the frequency f of the sinusoidal current passing through the cell, on the DC voltage V LIC across the cell and on the operating temperature. It is then described by the following equation:

j T V f C f T V f R T V f Z LIC i LIC i LIC LIC × × × × + = ) , , ( 2 1 ) , , ( ) , , ( π (3)
where, R i (f,V LIC ,T) and C i (f,V LIC ,T) are the frequency, voltage and temperature dependent resistance and capacitance of the LIC respectively. From impedance spectroscopy measurements illustrated in Fig. 6, one can extract the parameters R i (f,V LIC ,T) and C i (f,V LIC ,T) for different frequencies and voltages using the following equations:

1 2 I m ( ) ( , ) ( , )
, ,

LIC i L I C LIC C f Z f V f V T T π - = × × × Re( ) ( , ) ( , )
, ,

i L I C LIC LIC R Z f V f V T T = (4) (5)
where, Im(Z LIC (f,V LIC ,T)) and Re(Z LIC (f,V LIC ,T)) are the imaginary and the real parts of the impedance Z LIC (f,V LIC ,T) respectively.

Another simple frequency-independent RC circuit can be considered for describing the behavior of LICs. The resistance would be equal to the resistance of the LIC at the resonant frequency at which the imaginary part of Z LIC (f,V LIC ,T) is equal to zero. For example, at 3.8 V and at 25°C, this resistance is equal to 0.7 mΩ. In addition, the capacitance at 10 mHz would be the best choice for the capacitance value of the circuit model since it is the best to describe real time application of the LIC.

IV. DISCUSSION

LICs seem as an interesting technology that combines the advantages of EDLCs and LIBs. Its particularity remains in the nonlinear dependence of its capacitance on the voltage. Thus, choosing the optimal operating voltage of the cell must take into account the evolution of its capacitance.

Since the voltage of the pre-lithiated carbon is very low, the cell voltage is approximately equal to the potential of the activated carbon electrode. Therefore, at 3 V, one can conclude that the activated carbon electrode is at its neutral state since its potential is equal to its potential VS Li/Li + . This explains the minimum value of the capacitance around 3 V.

V. CONCLUSION

The development of hybrid components targets improving the characteristics of energy storage systems. LICs can be employed in high energy and high power applications because of their significant energy and power densities. Their specifications were introduced in this paper using frequency measurements that highlighted the nonlinear dependence of the capacitance on the voltage of the cell. A Physicochemical analysis was conducted in order to interpret this exceptional relationship between the capacitance and the voltage. These measurements were also the first step for modeling these cells. The depicted model is very simple and describes the behavior of the LIC during charge. However, further work is being done for improving this model and increasing its complexity so it would better suit the behavior of LICs while providing a high accuracy. Moreover, a model derived from time domain measurements is also under study in order to compare the efficiencies of both modeling methodologies.

Fig. 1 .

 1 Fig. 1. Ragone plot comparing characteristics of commercial energy storage systems.

Fig. 2 .

 2 Fig. 2. Ultimo prismatic cell 3300 F.

Fig. 3 .

 3 Fig. 3. The composition of the LIC cell at the end of the charge.

Fig. 4 .Fig 5 .

 45 Fig. 4. The protocol followed during the application of GEIS measurements.

Fig. 6 .

 6 Fig. 6. Comparison of Nyquist plots for different cell potentials at low frequencies (10 mHz to 1 Hz) at 25°C.

Fig. 7 .

 7 Fig. 7. Capacity evolution with respect to LIC's voltage.

Fig. 8 .

 8 Fig. 8. Capacity evolution with respect to LIC's voltage at 3 different temperatures.

Fig. 9 .

 9 Fig. 9. A simple model that describes the behavior of a LIC.

TABLE I

 I

	.	LIC PROPERTIES
	Voltage range (V)	2.2 -3.8


B Lecouteux
email: benjamin.lecouteux@univ-avignon.fr

G Linarès

B Favre
email: favre@icsi.berkeley.edu

Combined low level and high level features for Out-Of-Vocabulary Word detection

Keywords: OOV word detection, confidence measures, speech recognition

This paper addresses the issue of Out-Of-Vocabulary (OOV) words detection in Large Vocabulary Continuous Speech Recognition (LVCRS) systems. We propose a method inspired by confidence measures, that consists in analyzing the recognition system outputs in order to automatically detect errors due to OOV words. This method combines various features based on acoustic, linguistic, decoding graph and semantics. We evaluate separately each feature and we estimate their complementarity. Experiments are conducted on a large French broadcast news corpus from the ESTER evaluation campaign. Results show good performances in real conditions: the method obtains an accuracy of 45%-90% in OOV word detection with 2.5%-17.5% of false detection.

Introduction

Large Vocabulary Continuous Speech Recognition (LVCSR) systems are restricted by the limited size of the lexicon, since Out-Of-Vocabulary (OOV) words are frequently meaningful, and may be potentially critical for speech mining or indexing systems. The increase of lexical coverage by an extensive growth of the size of the vocabulary is not suitable because it introduces noise and it requires additional computing resources for handling huge vocabularies. Some authors proposed domain-specific methods for augmenting the lexicon. These approaches generally rely on a semantic analysis of the spoken contents. [START_REF] Oger | Local methods for on-demand out-of-vocabulary word retrieval[END_REF] proposed to use local context to build webqueries allowing to retrieve missing words.

One of the main difficulty in designing efficient methods for OOV word retrieval is to automatically detect them. Most of previous work addresses this problem by integrating fillers in the language models that are supposed to absorb unexpected speech segments. These methods demonstrated good performances on small or medium vocabulary tasks but the integration of filler models requires a fine tuning of the language models, especially on large vocabulary tasks. Others proposed a posteriori approaches, where the intermediate Automatic Speech Recognition (ASR) system outputs are analyzed in order to locate the transcripts areas where errors are due to OOV words. In this paper, we present such an a posteriori method that aims at detecting OOVs by analysing the output of a first recognition pass. We propose various features that are independently evaluated and combined in a statistical classifier.

In the next section, we draw an overview of related works about OOV word detection. In section 3, we present our method, which is derived from confidence measure estimation approaches. We focus on specific aspects of OOV detection in comparison with classical confidence measures.

In Section 4 we define the experimental framework. We describe the ESTER corpus on which experiments are conducted and we detail the experimental protocol.

In Section 5, we propose various features for OOV detection. We evaluate their relevance their complementarity for OOV word detection. We also study the impact of Word Error Rate (WER) on our OOV detection methods. Finally we present results on the whole test corpus in Section 5. The last section presents conclusions and prospects.

Related work

The detection of OOV words has been explored in several ways; some research groups [START_REF] Asadi | Automatic detection of new words in a large vocabulary continuous speech recognition system[END_REF][START_REF] Bazzi | Modeling out-of-vocabularywords for robust speech recognition[END_REF][START_REF] Boulianne | Out-of-vocabulary word modeling using multiple lexical fillers[END_REF] proposed to model OOV words through lexical fillers or generic word models: the goal is to cover all OOV word pronunciations, by representing them with sub-word units. Unfortunately, these methods often absorb parts of the speech corresponding to known words, and require to be finely tuned. Furthermore, these approaches are often used with small lexicons.

High-level information (syntactical, semantic) is used for both OOV detection and confidence measures estimation. [START_REF] Boros | Semantic processing of out-of-vocabulary words in a spoken dialogue system[END_REF] proposes to detect OOV words in spoken dialog system. Experiments are conducted in closed domains, but show the benefits of distant context for detecting named entities. [START_REF] Cox | A semantically-based confidence measure for speech recognition[END_REF] proposes to use Latent Semantic Analysis (LSA) for estimating confidence measures in LVCSR systems; results show a good accuracy but relatively low recall rates.

Other works [START_REF] White | Confidence estimation, oov detection and language id using phone-to-word transduction and phone-level alignments[END_REF][START_REF] Lin | Oov detection by joint word/phone lattice alignment[END_REF][START_REF] Hayamizu | Detection of unknown words in large vocabulary speech recognition[END_REF] use, in different ways, the edit distance between the phoneme lattice and the decoded words, misalignment being supposed to be more frequent in OOV words.

These methods yield good accuracy, but do not take directly into account semantic features or graph topology.

In other articles [START_REF] Sun | Using word confidence measure for oov words detection in a spontaneous spoken dialog system[END_REF][START_REF] Cai | Oov rejection algorithm based on class-fusion support vector machine for speech recognition[END_REF][START_REF] Young | Recognition confidence measures:detection of misrecognitions and out-of-vocabulary words[END_REF] OOV words are identified by using confidence measures. These approaches allow one to introduce more information sources extracted from the LVCSR system. These methods obtain better accuracy than the fillermodel, but they are limited to spoken dialog systems or isolated word recognition. Moreover, they do not use robust linguistics features.

Recent approaches for confidence measure estimation use side-information extracted from the recognizer: the number of competitors at the end of a word, normalized likelihoods, linguistics features, decoding process behavior, etc. Some work addressing the use of confidence measures show the prominence of language models: [START_REF] Mauclair | Automatic detection of well recognizedwords in automatic speech transcriptions[END_REF] proposes to combine acoustic and linguistic features such as language model back-off behavior and posteriors.

Detecting OOV words

Principle

The proposed method consists in 3 stages. The first one extracts features related to acoustic, linguistic, search graph topology. Then, a first OOV detection hypothesis is produced by a classifier based on the boosting algorithm. Finally, a semantic module refines this detection process. The next subsections detail this 3 steps detection method.

Extracted features

Each word from the hypothesis is represented by a feature vector composed of 23 features, that are grouped into 3 classes.

OOV word utterance induces probably acoustic distortions between the hypothesis and the best phonetic sequence. We use acoustic features wich consist of the acoustic log-likelihood of the word, the averaged log-likelihood per frame, the difference between the word log-likelihood and the unconstrained acoustic decoding of the corresponding speech segment.

Linguistic features are based on probabilities estimated by the 3-gram language model used by the ASR system. We use the 3-gram probability, the perplexity of the word in the window, the unigram probability. We also add the index, proposed in [START_REF] Mauclair | Automatic detection of well recognizedwords in automatic speech transcriptions[END_REF], that represents the current backoff level of the targeted word. This value is set to 3 if the current trigam occurred in the train corpus, 2 if only the current bigram occurred, 1 overwise.

Graph features are based on the analysis of the word confusion networks. The use of these features is motivated by the idea that when an OOV occurs, the search algortihm probably explores various alternative paths that are similarly scored. Backtracking behavior and the distribution of posterior probabilities could be a good index of unknown word detection. We use the number of alternative paths in the word section, and the posterior probability. We also include values related to the distribution of posteriors probabilities in the word section: the 2 extremum values, the mean of the posteriors from the sausage section, the number of null links before and after the current section. A last parameter represents the mean duration of words, that is estimated in a 500ms window.

Features combination and classification

We use a boosting classification algorithm in order to combine features, as detailed in [START_REF] Moreno | A boosting approach fo confidence scoring[END_REF]. The classifier is a variant of Adaptive Boosting (Adaboost): icsiboost 1 . The algorithm consists in an exhaustive search for a linear combination of classifiers by overweighting misclassified examples. An advantage of this classifier is its ability to provide posteriors, which allows for an intuitive interpretation of the results.

Input vectors are estimated on a window composed by 3 consecutive words. On each word, we estimate the 23 features previouslay described. We finally obtain a 69 coefficient vector including previous, current and next word descriptors.

The classifier is trained on a specific training corpus, that was not included in the ASR system training. Each word from this corpus is tagged as in or out of vocabulary, according to the ASR system lexicon. 1 available on http://code.google.com/p/icsiboost/

Latent Semantic Analysis and web trigrams for OOV words

Latent Semantic Analysis is a technique that allows to associate words that tend to co-occur within documents with a semantic relation. The assumption is that co-occurrent words within the same document are semantically linked.

In our system, a semantically consistent word sequence may be considered as unexpected by the ASR language model, due to the limits of the n-gram language models. Therefore, we estimate an index of semantic consistency (SC) that allows to validate (or to reject) OOV detections previously performed by the classifier. This measure is not included in the classifier because of the introduced noise in non-OOV words. It is only applied on detected words, as a last filtering process.

This filter combines two measures based on the web and the French corpus Gigaword. The first estimates the probability of a word co-occurences as a ratio of Google hits [START_REF] Oger | Local methods for on-demand out-of-vocabulary word retrieval[END_REF]. This measure is estimated on a local window of 3 words (including the targeted one) filtered by a stop list. The second uses the Latent Semantic Analysis (LSA) to estimate how much the targeted word is semantically close to the current segment [START_REF] Cox | A semantically-based confidence measure for speech recognition[END_REF] : for each targeted word, the LSA module selects the 100 closest words. The cardinality of the intersection of this bag of words and the current segment is normalized by the segment size, and the resulting value is linearly combined with google-based scores.

Experimental framework 4.1. The LIA broadcast news system

Experiments are carried out by using the LIA broadcast news (BN) system which was used in the ESTER evaluation campaign. This system relies on the HMM-based decoder developed at LIA, Speeral [START_REF] Nocera | The lia's french broadcast news transcription system[END_REF]. Speeral is an asynchronous decoder operating on a phoneme lattice; acoustic models are HMMbased, context dependent with cross word triphones. The language models are classical trigrams estimated on about 200M words from the French newspaper Le Monde and from the ES-TER broadcast news corpus (about 1M words). The lexicon contains 67K words. In these experiments, only one decoding pass is performed in 3x Real Time (RT).

The ESTER corpus

The training and development parts of the data set are based on the training corpus provided for the ESTER evaluation campaign. The ESTER corpus consists of French radio broadcasts of the Radio-France group. The training corpus contains 15K in-vocabulary words randomly selected among 1M words and 15K OOV words extracted from the whole training corpora.

We test our approach on 7 hours of speech extracted from the ESTER test set. The number of OOV words is about 900, which, since the corpus contains 65,000 words, represents an OOV rate of about 1.5%. The mean WER is 28.9 because only the first pass of decoding is performed.

Detection protocol

OOV words have been manually specified by selecting all the reference words not available in the lexicon. During the detection, if a marked OOV word overlaps with a true OOV word, the OOV word is considered as true detection. All the other selected words are considered as false detection.

Experiments

We performed three experiments to estimate the relevance of the features and their complementarity. Moreover, we estimate the system robustness by evaluating the detection rates according to the WER.

Relevance of the features

Classification depending on each parameter

In this section, we draw an overview of the classification ability for each set of features. We always use the same model trained with icsiboost, but we disable unwanted features during classification. We use Receiver Operating Characteristic (ROC) curves to display OOV true and false detection rates. Curves are traced by varying the threshold on classifier outputs. The ROC curves, shown in Figure 1, present the output of the classifier for each separate feature compared to the combination of all features. Results show that the most relevant descriptors are the linguistic and graph-topology features, while acoustic features and posteriors seem weaker in comparison to others. Linguistic scores, language model back-off behavior are predictors of linguistics breaking. This break is characteristic of incoherences due to unknown word or recognition errors. An interesting thing is the quality of the language model back-off behavior, which presents results close to language model probabilities.

Graph topology is also a interesting feature, because it is related to recognition process difficulties. A surprising result is the high discriminative capacity of the unigram probability. This may be due to fact that the system trends to backoff, when unknown words are encountred, to small but infrequent words that are acoustically matching to the signal. However, the combination of all the features show a significant improvement. In spite of the feature heterogeneity, we observe a relatively high complementary.

Linguistic context of OOV words

The experiments reported in the previous section show the relevance of the OOV word contexts (Figure 2). Our topology includes the previous and next word for each spotted word. An unexpected result is that the previous and next word seem to be more informative than the spotted word with a prevalence for the previous word. The previous word is probably more relevant, concerning linguistic shifts, than the middle and next words: we must not forget that the features of the previous word hold linguistic information about the middle word. Yet, the combination of the three sets of feature of the words leads to a 10% relative improvement, whereas the use of a window bigger than 3 words leads to worse results.

Complementarity of features

After the overview of the feature relevance, we study their complementarity. The baseline curve uses the best observed set of features in the previous section. Then, we enable each new set of features according to their relevance, from the most relevant to the last relevant. The ROC curves (Figure 3) show the complementarity of each set of features.

We measure the complementarity by using the Equal Error Rate (EER). The EER is the point where OOV word detection rate and false detection rate are equal. In the table of Figure 3 we show the EER evolution for each added set of features.

Each set provides some discriminative information, except from the unigram probability that seems to be redundant with the language model scores. An unexpected result is the acoustic significance. This feature shows the best improvement, while it exhibits the worst performance when other features are not used. However, the classifier allows one to relate different acoustic features. Sets of features are therefore complementary and mutually dependent.

OOV words and WER

These last experiments examine the behavior of the detection according to the WER obtained in transcription. We have sorted the test segments in 6 slices of WER (Figure 4).

In Figure 4 we show results for the different slices of WER. We can remark a correlation between WER and OOV rate. ROC Figure 4: Results for each WER slice curves show three trends: an excellent classification for WER between 10% and 30%, a stable classification for WER between 0% and 10% or 30% and 50%, and a degradation beyond 50%.

The 0%-10% slice is surprising because it is far from the best slices. It corresponds to speech segments with very low OOV rate (0.25%), that may increase the detection difficulty : high lexical coverage and good WER indicates acoustic and linguistic contexts probably close to the training conditions, on which our features may be less informative.

However, between 10% and 30%, the classification rates are greater than the ones observed on other slices. It probably corresponds to the most frequent decoding context, on which the test condition matches the training conditions and where features are significants. Beyond 50%, noise decreases the accuracy. Globaly, experiments show a relatively good robustness against WER, except when ASR system dramatically fails.

Semantic Filtering of detected OOV words

In the final pass, we propose to use semantic information for filtering the detected OOV words. We use the semantic module only on detected OOV words, with the purpose to refine detection on semantically coherent words. The ROC curves are presented in Figure . Results show that the filter reduces the EER by 4% relatively (15.2% to 14.6), while the false detection rate decreases of about 5% relative.

Conclusion and future work

In this paper we presented a method for the detection of OOV words in Large Vocabulary Continuous Speech Recognition systems. Our approach consists in extracting multiple features during the speech recognition process: acoustic features, linguistic features, topology graph features. Finally, a semantic module performs a last filtering pass to reduce the false detection rates.

Our experiments showed promising results: OOV word detection seems to be homogeneous, despite the inconstant WER. The proposed method allows one to detect 45% of the unknown words with a 2.5% false acceptance rate, or 90% for 17.5% false acceptance. Experiments show that linguistic and graph-based features are the most relevant predictors. However, acoustic features associated to the others make the detection more robust. Finally, semantic filtering provides a slight but significant improvment.

Presently, these results rely only on the one-best word hypothesis; we plan to extend the semantic module in order to retrieve alternative paths in the graph, for invalidating false OOV word detections. We wish to add more semantic analysis, such as the automatic detection of named entities that usually compose a large part of unknown words.

Figure 1 :

 1 Figure 1: Relevance of each feature

Figure 2 :

 2 Figure 2: Relevance of the words in an observation window

Figure 3 :

 3 Figure 3: Complementarity of the features

Figure 5 :

 5 Figure 5: Results after the semantic filter