Benjamin Lecouteux
email: benjamin.lecouteux@univ-avignon.fr

Georges Linarès
email: georges.linares@univ-avignon.fr

Pascal Nocéra
email: pascal.nocera@univ-avignon.fr

Jean-Franc ¸ois Bonastre
email: jean-francois.bonastre@univ-avignon.fr

Imperfect transcript driven speech recognition

In many cases, textual information can be associated to speech signals such as movie subtitles, theater scenarios, broadcast news summaries etc. These informations could be considered as approximated transcripts and correspond rarely to the exact word utterances. The goal of this work is to use this kind of information to improve the performance of an automatic speech recognition (ASR) system. Multiple applications are possible: follow a play with closed caption aligned to the voice signal (while respecting to performer variations) for helping deaf people, watch a movie in another language using aligned and corrected closed captions, etc. We propose in this paper a method combining a linguistic analysis of the imperfect transcripts and a dynamic synchronization of these transcripts inside the search algorithm.

The proposed technique is based on language model adaptation and on-line synchronization of the search algorithm. Experiments are carried out on an extract of the ESTER evaluation campaign [4] database, using the LIA Broadcast News System. The results show that the transcript-driven system outperforms significantly both the original recognizer and the imperfect transcript itself.

Introduction

In many cases, external information sources associated to speech signal are available: for instance movies subtitle or broadcast news with summary, theater scenario, journalist prompt. These information could be used in order to improve the performance of an automatic speech recognition (ASR) system.

Howether, when the available transcript differ from the correct one, speech-to-text alignment methods fail and large speech segments may be removed. In many cases, an automatic speech recognizer could provide useful but probably erroneous transcripts. Nevertheless, the speech recognition system can take benefit of the provided transcripts.

The use of imperfect transcripts for audio decoding was already studied within the framework of audio/video automatic indexing [START_REF] Chih-Wei | Automatic closed caption alignment based on speech recognition transcripts[END_REF] or for acoustic model re-estimation with non transcribed data [START_REF] Photina | Improving acoustic models with captioned multimedia speech[END_REF] [START_REF] Lamel | Lightly supervised and unsupervised acoustic models training[END_REF]. These works are frequently motivated by the availability of large corpora which are partially or poorly transcribed.

This task is presented in literature as a speech-to-text synchronization problem, or as an imperfect transcript correction problem. In both cases the main difficulty resides in the low quality of the provided transcripts: P. Placeway [START_REF] Placeway | Cheating with imperfect transcripts[END_REF] measures a WER of 10 to 20% between movie subtitles and the exact transcript. These divergences increase considerably the alignment difficulty.

In this paper, we first present issues involved in imperfect text alignment on a word stream. Then, we describe our method for speech recognition driven by imperfect transcripts. We especially investigate the integration of this source of information to an asynchronous stack decoder. Finally, the undertaken experiments and the results obtained are detailed and some conclusions and future prospects are presented.

Speech-to-text alignment

Alignment with exact transcripts

Speech-to-text alignment has been widely studied during last decades. Several authors proposed methods based on a forced Viterbi decoding, sometimes driven by heuristics reducing the algorithm complexity.

In [START_REF] Moreno | A recursive algorithm for the forced alignment of very long audio segments[END_REF], P.J. Moreno proposes to align long audio documents with their exact transcripts within the framework of automatic indexing of multimedia documents. His method is based on the search of well synchronized areas, called small confidence island. Initially, a language model is estimated on the exact transcript. Synchronized areas are isolated from the extracted segments with an high matching between the a priori transcript and the automatic transcript. Documents are then segmented according these small confidence islands; on each part, a specific language model is estimated. The algorithm is launched recursively on unaligned parts until reaching the convergence point. This method, restricted to exact transcripts, obtains excellent results: 99% of the words are correctly aligned.

Alignment with imperfect transcripts

Alignment on partially incorrect transcripts has been mainly used to deal with corpora where only low quality transcripts are available. The goal is to improve the database usability, generally by extracting well-transcribed speech segments.

In [START_REF] Lamel | Lightly supervised and unsupervised acoustic models training[END_REF], L. Lamel evaluates a method for acoustic model training on low quality transcribed databases. This technique consists in automatic decoding the training database and comparing the resulting hypothesis to the imperfect transcripts. Matching segments are then used for acoustic models re-estimation. This process is iteratively performed until convergence. In this paper, we want not only to find well-transcribed segments but also to improve the transcripts on segments where no correct transcripts are available. This issue was tackled by P. Placeway [START_REF] Placeway | Cheating with imperfect transcripts[END_REF] to take advantage of subtitles with a synchronous decoder (Sphinx-3, [START_REF] Eskenazi | The 1996 hub-4 sphinx-3 system[END_REF]). Their experiments are based on an English broadcast news database. They propose to estimate a language model on the subtitles and align them to the audio stream.

In order to combine information from closed captions and language models, P. Placeway [START_REF] Placeway | Cheating with imperfect transcripts[END_REF] proposes to interpolate a generic model and a model estimated on the subtitles. This interpolated model is then used by the ASR. Their experiments are carried out on 9.7% WER subtitles. Using a beam-search decoder, this technique improves the performance of 15% relative WER (Word Error Rate) compared to the initial decoding (from 55.8% to 47.2%). Moreover, they propose to integrate a time warping algorithm in addition to the model interpolation technique: as the decoder progresses and selects a small list of candidate words, the words corresponding to the imperfect transcript are favored in the hypothesis beam. This method brings a 37% relative gain in WER compared to initial decoding (while decreasing the absolute WER to 35%). The final result remains however definitely lower than the quality of the imperfect transcript provided to the system (9.7% of WER).

Dealing with imperfect transcripts

Our objective is to exploit imperfect transcripts with an asynchronous decoder based on the A* algorithm, within the framework of a broadcast news (BN) system. First, we present the characteristics of this kind of decoder. Then we show how the information from imperfect transcripts can be integrated in the decoding process.

Two methods exploiting the imperfect transcript are proposed. The first one consists in combining a generic language model and a language model estimated on the imperfect transcript. The second one integrates a Viterbi-based alignment algorithm within the A* algorithm, by on-demand synchronization and estimate function rescoring.

Language model adaptation

In our approach, linguistic variability can be reduced thanks to the contribution of an exact or imperfect transcript. A profit can be obtained by reducing overall linguistic space. This can be achieved by estimating a language model on the transcript itself. However, such a language model would be probably too specific when the speaker deviates from the original transcript. Therefore, this model is interpolated with a generic language model.

Anatomy of SPEERAL decoder

LIA has developed a large vocabulary continuous speech recognition system named SPEERAL [START_REF] Nocera | Phoneme lattice based a* search algorithm for speech recognition[END_REF]. This decoder is derived from the A* search algorithm operating on a phone lattice. The exploration of the graph is supervised by an estimate function F (hn) which evaluates the probability of the hypothesis hn crossing the node n:

F (hn) = g(hn) + p(hn) (1)
where g(hn) is the probability of the current hypothesis which results from the partial exploration of the search graph (from the starting point to the current node n); p(hn) is the probe which estimates the probability of the best hypothesis from the current node n to the ending node.

In SPEERAL, the probe p combines the acoustic probability and a linguistic look-ahead score [START_REF] Linarès | Scalable language model look-ahead for lvcsr[END_REF]. The acoustic term is com-puted by an acoustic decoding carried out by the Viterbi-back algorithm operating on a phone lattice.

The graph exploration is based on the function of estimate F (). Indeed, the stack of hypothesis is ordered on each node according to F (). The best paths are then explored firstly. This deep search refines the evaluation of the current hypothesis. Lowprobability paths are cutted-off, leading to search backtrack. In such situations, search is desynchronized from the audio stream.

In order to be able to take into account information resulting from the imperfect transcript, the F () function is modified to influence the score of the current hypothesis. In fact, this mechanism drives the search by dynamically rescoring g(hn) according to the alignment scores. This algorithm consists in two parts: synchronization of the transcript and integration of the synchronized transcript inside the A * evaluation function.

Audio stream synchronization to the imperfect transcript

The speech recognition system generates hypothesies as it advances in the phonemes lattice. The best hypothesies at time t are extended according to the current hypothesis probability and the probe results. In order to locate an anchorage point into the transcript, each evaluated word is aligned to the reference word stream using an algorithm of temporal alignment (Dynamic Time Warping [START_REF] Berndt | Using dynamic time warping to find patterns in time series[END_REF]). A partial hypothesis is built by collecting the current word and its history from the path found during the search process. The best hypothesis-to-reference matching provides a synchronization point which will be used for hypothesis rescoring.

Considering the low complexity of word alignment, this ondemand synchronization process requires low CPU resources. Moreover, the additional alignment cost may be balanced by the speed-up provided on well-transcribed sections.

The figure 1 illustrates the dynamic synchronization of the search driven by alignment on an imperfect transcript.

Weighting of the current hypothesis according to alignment

Once the hypothesis is synchronized with the transcript, the algorithm estimates the matching transcript-to-hypothesis score (noticed α). This score is based on the number of words in the shortterm history which are correctly aligned with the transcript: only three values are used, corresponding respectively to a full alignment of the current trigram, a full alignment of the current bigram and an alignment of one word only. α is maximum when the trigram is aligned and decreases according to the misalignments of the history. Values of α are determined empirically using a development corpus. Then, linguistic probabilities are modified using the following rescoring rule:

P (wi|wi-2, wi-3) = P 1-α (wi|wi-2, wi-3) (2)
where P (wi|wi-2, wi-3) is the updated trigram probability of a the word wi knowing the history wi-2, wi-3, and P (wi|wi-2, wi-3) is the initial probability of the trigram.

Experiments

Experimental context

The experiments are carried out with the "Broadcast news" system developed by the LIA for the French evaluation campaign ESTER [START_REF] Linarès | Le système de transcription du lia pour ester-2005[END_REF].

Corpora and imperfect transcript

The system is assessed on 3 hours of radio broadcast (France Inter 1, France Inter 2, France Info) extracted from the ESTER development corpus. Imperfect transcripts are generated by adding errors manually in the initial transcript, while ensuring a correct journalistic form in order to respect the traditional style of a radio broadcast: 10% WER are introduced in 2 show transcripts, and 20% WER are introduced in the last show transcript.

Language model interpolation

Some preliminary experiments are undertaken in order to identify the potential benefits of the proposed methods. First, a language model is estimated on the exact transcript. Then, this model is combined with a generic language model (65000 words learned on the newspaper Le Monde). The objective is to measure the real effect of the suggested techniques on the decoder recognition performance 1 . The table 1 shows the results of the interpolation of a language model trained on the exact transcript with a generic language model. For comparison, a baseline decoding is performed by using a generic language model, we obtain a WER of 22.7%. WER FrInter 1: ML-G alone 22.7% FrInter 1: ML-TrEx alone 5.2% FrInter 1: ML-G 70% + ML-TrEx 30% 13.0% FrInter 1: ML-G 50% + ML-TrEx 50% 11.5% FrInter 1: 30% + ML-TrEx 70% 10.8% These experiments show that a decoding based on a language model estimated on the imperfect transcripts improves significantly the WER. However, without another information source, the speech recognition system converges to errors contained in the 1 The words not found in the ASR lexicon are extracted from the transcript and added to the language model transcript. This technique does not allow to obtain good recognition rates. Indeed, using a language model estimated on the exact transcript, the WER remains under 15% despite the availability of the perfect transcripts. So we need to use more efficiently transcript information.

Experiments with model interpolation and dynamic synchronization

This section present experiments using the decoding strategy based on both dynamic synchronization and linguistic rescoring.

The experiments combining the interpolation of the language models with an alignment on the exact transcript are presented in table 3. WER FrInter 1: ML-G alone + alignment TrEx 3.7% FrInter 1: ML-TrEx alone + alTrEx 3.7% FrInter 1: ML-G70%+ML-TrEx30%+alTrEx 4.9% FrInter 1: ML-G50%+ML-TrEx50%+alTrEx 3.5% FrInter 1: ML-G30%+ML-TrEx70%+alTrEx 3.7% Table 3: Interpolation of the generic language model (ml-G) with the model trained on the exact transcript (ml-TrEx) and alignment to the exact transcript (alTrEx)

We obtain in this case a WER of 3.5%. This level of error can be considered as minimal for a method re-estimating the concurrent hypothesis without modifying the content of the hypothesis stack.

Table 4 shows the experiments replacing the exact transcript by the imperfect transcript. WER FrInter 1: ML-TrErr + alTrErr 9.9% FrInter 1: ML-G + alTrErr 7.7% FrInter 1: ML-G70%+ML-TrEr30%+alTrErr 7.2% FrInter 1: ML-G50%+ML-TrEr50%+alTrErr 7.4% FrInter 1: ML-G30%+ML-TrEr70%+alTrErr 8.6% Table 4: Interpolation of the generic language model (ml-G) with the model trained on the imperfect transcript (ml-TrErr -10% WER) and alignment to imperfect transcript (alTrErr -10% WER)

Although this approach removes some of the limits observed in the model combination, potential sources of error remain. In particular, heuristics are used in the decoder to reduce the search space and to accelerate decoding. In usual conditions, pruning should introduce only few errors; however, when the acoustic context is of low quality, the best hypothesis can be excluded from the stack of available hypothesis. This occurs more frequently in real time configurations of the system, for which pruning is more strict. In this case, a strategy based on the promotion of the synchronized hypothesis does not allow error recovering. One can quantify these phenomena by using the recognition engine to perform a text-tospeech forced alignment.

The best result is obtained by combining the generic language model (with a 70% weight) and the model estimated on the imperfect transcript (with 30% weight) and by carrying out an alignment on the latter. Alignment reduces WER down to 7.2%. It makes possible to bring a temporal information which is poorly taken into account by the language model. The use of an DTW alignment associated with the interpolation of the models shows a new gain.

In order to validate these results, we test the best configuration of the system on a larger corpus. Two hours are processed using the same evaluation protocol described in the last sections. Results are reported in the table 5. We observe that the gain in performance seems to be relatively independent from the quality of the initial transcript. Nevertheless, the best improvement is obtained on the FrInfo show, which is poorly transcribed. This result suggests a good robustness of the proposed method.

Shows

These experiments show that imperfect information can be favorably exploited during the decoding process.

Conclusion

A well-known advantage of A * algorithm lies in the possibility to incorporate various information source into the recognition process. Nevertheless, it is an asynchronous algorithm and its application to alignment tasks can be difficult. We have proposed a ondemand synchronization which allows to combine asynchronous recognition and text-to-speech alignment. The system takes advantage of the approximate transcript when it allows a gain and switches in free-recognition mode when the acoustic observations do not match the suggested transcript.

We evaluated our approach step-by-step. For each phases we estimated the system performance and we compared the results with the maximum performance bound allowed by the proposed approach.

The first evaluated technique consists in extracting linguistic information from the script by estimating a language model on the imperfect transcripts. Our experiments showed that the interpolation of this model with the generic language model leads to significant improvements. Nevertheless this method does not allow to go further than the quality of the imperfect transcript. This drawback may limit the interest of that kind of approach.

The second method presented in this paper consists in driving the search algorithm with the provided imperfect transcript. The method relies on a synchronization between the current search hypothesis and the provided information.

This method allows a significant gain in term of WER even if the quality of the provided transcripts is low. The obtained relative WER improvement is between 28% and 40%.

Moreover, we observe that the modified algorithm improves slightly the decoding speed, in spite of the additional computational cost due to the search synchronization. This profit in term of execution time is due to the earlier exploration of the best paths on well transcribed sections. However, search efficiency can be improved by introducing heuristics in the probe itself.

Although these first results show the interest of an alignment on an approached transcript, these experiments were carried out under controlled conditions: a relative low level of noise, transcript closed to the exact transcript and BN speech style and topics. We plan to use our methods under more difficult conditions, for example on automatic subtitling of movies or theater plays.

Figure 1 :

 1 Figure 1: Synchronization of the search beams with the imperfect transcript by the DTW algorithm during asynchronous decoding

Table 1

 1

	: Interpolation of generic language model (ml-G) with a
	model trained on the exact transcript (ml-TrEx)
	Then, starting from the imperfect transcript (10% WER), a
	language model is also generated. The experiments using this lan-
	guage model combined with the generic model are presented in
	table 2.
		WER
	FrInter 1: ML-TrErr alone	16.3%
	FrInter 1: ML-G 70% + ML-TrErr 30%	16.2%
	FrInter 1: ML-G 50% + ML-TrErr 50%	15.4%
	FrInter 1: ML-G 30% + ML-TrErr 70%	15.2%
	Table 2: Interpolation of generic the language model (ml-G) with
	the model trained on the imperfect transcript (ml-TrErr -10%
	WER)


Grégory Senay
email: gregory.senay@univ-avignon.fr

Georges Linarès
email: georges.linares@univ-avignon.fr

Benjamin Lecouteux
email: benjamin.lecouteux@univ-avignon.fr

A SEGMENT-LEVEL CONFIDENCE MEASURE FOR SPOKEN DOCUMENT RETRIEVAL

Keywords: Speech recognition, confidence measures, spoken document retrieval

This paper presents a semantic confidence measure that aims to predict the relevance of automatic transcripts for a task of Spoken Document Retrieval (SDR). The proposed predicting method relies on the combination of Automatic Speech Recognition (ASR) confidence measure and a Semantic Compacity Index (SCI), that estimates the relevance of the words considering the semantic context in which they occurred. Experiments are conducted on the French Broadcast news corpus ESTER, by simulating a classical SDR usage scenario : users submit text-queries to a search engine that is expected to return the most relevant documents regarding the query. Results demonstrate the interest of using semantic level information to predict the transcription indexability.

INTRODUCTION

Typical approaches of speech retrieval associate Automatic Speech Recognition (ASR) and information retrieval techniques. One of the major issues of this processing scheme is the impact of recognition errors on the SDR system performance : ASR systems suffer from a lack of robustness on unexpected conditions where Word Error Rates (WER) may be upper than 30%, impacting significantly the accuracy of speech search [START_REF] Oard | Building an information retrieval test collection for spontaneous conversational speech[END_REF][START_REF] Whittaker | Scanmail: a voicemail interface that makes speech browsable, readable and searchable[END_REF][START_REF] Hansen | Speechfind: Advances in spoken document retrieval for a national gallery of the spoken word[END_REF]. On controlled conditions, TREC-SDR track concluded that errors do not corrupt SDR engines results [START_REF] Garofolo | The TREC spoken document retrieval track: A success story[END_REF].

Considering that the perfect ASR system is not yet a shortterm perspective, many recent studies on SDR focus on errortolerant indexing methods, based on word-lattices or N-Best list representations of ASR outputs [START_REF] Saraclar | Lattice-based search for spoken utterance retrieval[END_REF][START_REF] Chang | Latent semantic retrieval of spoken documents over position specific posterior lattices[END_REF], indexing strategies [START_REF] Chelba | Soft indexing of speech content for search in spoken documents[END_REF][START_REF] Kurimo | Retrieving speech correctly despite the recognition errors[END_REF][START_REF] Siegler | Integration of Continuous Speech Recognition and Information Retrieval for Mutually Optimal Performance[END_REF] and the handling of out-of-vocabulary words.

For industrial applications, a realistic way could be to identify the speech segments where ASR system fails, not only in terms of WER but also by considering the final objective of speech retrieval. The erroneous segments could then be checked and corrected by an human operator. In this semi-automatic scenario, the availability of a self-diagnostic tool -that helps the operator to identify corrupted segments-is critical for the global cost of the indexing process. This paper describes such a method that is expected to predict how much an erroneous transcription impacts the global SDR system performance.

First section describes the task and the metric we use for evaluating retrieval errors due to the ASR. Section 2 introduces a segment-level Confidence Measure (CM) that aims to predict this metric. This method relies on the combination of ASR confidence measure and a local semantic compacity index. In Section 3, we present the experimental protocol. Section 4 reports results of experiments. Finally, the paper ends with conclusion and some perspectives.

TASK AND EVALUATION METRIC

Evaluation of the WER impact on SDR have been studied and discussed in many papers ( [START_REF] Chelba | Retrieval and browsing of spoken content[END_REF]). TREC evaluation protocols compare the results provided by the SDR system and a reference ranking provided by human experts. Another way consists in comparing the ranking obtained by automatic transcription of spoken documents to the ones obtained by correct transcriptions. These evaluations are conducted by using a large set of queries, that are submitted to a search engine operating on the whole test set. The performance of the SDR system are obtained by Mean Average Precision (MAP) or R-Precision scores.

Here, our goal is to predict, at a the segment level, the impact of errors on the global SDR process. The next section presents how this indexability measure is estimated, the method we propose to predict indexability being described in section 5.

Indexability Estimate

Segments are extracted according to the speech pauses, with a maximum length of 30s. Each of them is considered as a document by the SDR system. Considering that one (ore more) error in the segment potentially impacts all search results (for all queries), each segment indexability estimate requires a full run of SDR evaluation.

Therefore, the indexability Idx(s) of a segment s is computed by a 3 step process:

• (1) the targeted speech segment s is automatically transcribed by the ASR system,

• (2) for each test-query, search is performed on the whole speech database by using correct transcriptions for all segments, except for s which is automatically transcribed,

• (3) the resulting ranks are compared to the ones obtained by searching the full reference transcription set. Finally, indexability Idx(s) of the segment s is obtained by computing the F-measure on the top-20 ranked segments, relatively to the top-20 ranking reference (i.e. the ranking on the correct transcripts).

This algorithm estimates the individual impact of the targeted segment transcription on the global SDR process, knowing the targeted ranking. The next section presents a method to predict this indexability metric.

PREDICTING INDEXABILITY

The proposed method aims to predict the impact of recognition errors on the indexing process. This is achieved by combining word-level confidence measures and a semantic compacity index on the one-best hypothesis from ASR. Combination is achieved by using a classical multi-layer perceptron. These main components are described in the next 3 sections

Confidence measure from ASR

The ASR confidence scores are computed in 2 stages. The first one extracts low level features related to acoustic and search graph topology, and high level features related to linguistics. In the second step, a first error detection hypothesis is produced by a classifier based on the boosting algorithm. Each word from the hypothesis is represented by a feature vector composed of 23 features, that are grouped into 3 classes:

• Acoustic features consist of the acoustic log-likelihood of the word, the averaged log-likelihood per frame, the difference between the word log-likelihood and the unconstrained acoustic decoding of the corresponding speech segment.

• Linguistic features are based on probabilities estimated by the 3-gram language model used in the ASR system. We use the 3-gram probability, the word perplexity and the unigram probability. We also add an index that represents the current back-off level of the targeted word.

• Graph features are based on the analysis of the word confusion networks: the number of alternative paths in the word section and values related to the distribution of posteriors probabilities.

We use a boosting classification algorithm in order to combine word features, as detailed in [START_REF] Moreno | A boosting approach for confidence scoring[END_REF]. The algorithm consists in an exhaustive search for a linear combination of classifiers by overweighting misclassified examples. The classifier is trained on a specific training corpus, that was not included in the ASR system training. Each word from this corpus is tagged as correct or erroneous, according to the ASR system reference. This measure obtains a Normalised Cross Entropy of 0.373 on dev and 0.282 on test.

The confidence score of a document is computed by filtering the low meaningful words with a stop-list, scores of remaining words being averaged to obtain the segment-level confidence measure.

Semantic Compacity Index

The use of semantic-level information for indexability prediction is motivated by the fact that a query usually targets documents according to their semantic contents (topic or finer granularity concepts). Some papers proposed to use such high level-features for the estimate of confidence measures [START_REF] Cox | High-level approaches to confidence estimation in speech recognition[END_REF][START_REF] Hakkani-Tür | Error prediction in spoken dialog: from signal-to-noise ratio to semantic confidence scores[END_REF]. Most of the authors concluded that such an approach does not significantly improve the CM accuracy for ASR. Nevertheless, meaningful words are critical for SDR and WER criterion does not evaluate the interpretability of transcriptions.

Our proposal is to estimate a semantic compacity index SCI(s) for each segment s and to use it as an input feature of the predictor. This segment score is obtained by averaging the local semantic correlations sc(w i , w j ) of its word pairs (w i , w j ) estimated on a large corpus.

We focus on short-term correlations between meaningful words. Therefore, cue words are removed according to a stoplist. Moreover, the remaining terms are lemmatised in both corpus and transcription segments. Then, word-pair semantic scores are computed by using lemma co-occurences frequencies weighted by a TF-IDF index:

sc(wi,wj) = T F (li, c).IDF (li).δ c (wj)+T F (li, c).IDF (lj).δ c (wi) (1)
where l i is the lemmatised form of w i word, T F (l i , c) the frequency of the lemma l i in the context c, IDF (l k ) the inverse frequency of the lemma l k on the whole corpus, and δ c (w i ) = 1 if w i ∈ c, 0 else. Semantic compacity sci(c) are estimated in a sliding window of 5 lemma, each corresponding to a context c:

sci(c) = X c k X (w i ,w j )∈c k v u u u t sc(wi, wj) * IDF (wi)IDF (wj) n P k=1 IDF (w k ) (2)
In our experiments, statistics are computed on a the French part of Wikipedia corpus which offers the advantage of covering large topics and subjects.

Scores combination

ASR confidence measures and semantic index are combined to predict the indexability score. In order to determine this mapping function from CM and SCI measures, a classical multilayer perceptron is trained [START_REF] Rosenblatt | Principles of neurodynamics: Perceptrons and the theory of brain mechanisms[END_REF]. This neural network is 3-layer network trained by using the back-propagation algorithm. Input, medium and output layers contain respectively two, ten and one cells. One of the eight hours of the ESTER test set is dedicated to train (220 of the 1694 segments).

EXPERIMENTAL FRAMEWORK

ASR system

Experiments are conducted by using the LIA ASR system (Speeral). It is an asynchronous decoder operating on a phoneme lattice; acoustic models are HMM-based, contextdependent with cross word triphones. The language models are classical trigrams estimated on about 200M words from the French newspaper Le Monde and from the ESTER broadcast news corpus (about 1M words). The lexicon contains 67K words. Since the full system runs 3 passes including unsupervised speaker adaptation, we use here only the system 2xRT, without any speaker adaptation, that performs 35.1% word error rate on the 8 hours test set of ESTER campaign.

Search engine and query set

As our goal is to evaluate the data quality rather than the search strategy, we used the standard TF-IDF-based search engine Lucene in our experiments. Queries are directly extracted from the headlines of the newspaper Le Monde published during the same period as the test corpus. Indeed, this query set is composed of 160k unique queries, every one of which can produce at least one result.

EXPERIMENTS

The first experiment consists in evaluating the prediction error rates (P ER) on the 7 hours test corpus. In order to estimate the individual contribution of each feature, we trained neuronal predictors base respectively on confidence measure (CM ), semantic compacity (SCI) and the combination of CM and SCI, noted as CM + SCI in the table 1.

P ER the distortions between P Idx and Idx are evaluated as follow:

Q1 = 1 τ τ X j=1 |P Idx(j) -Idx(j)| Idx(j) (3)
RM S = v u u t 1 τ τ X j=1 (P Idx(j) -Idx(j)) 2 (4)
Q1 and RM S respectively presents the general distortion and root mean squared (RM S) error. In all results, the CM + SCI score significantly outperforms both confidence measures and semantic consistencies. We can even see that the absolute difference of semantic prediction is much better than confidence measure (27% relative).

CM SCI CM + SCI Q1

1,65 1,74 1,59 RM S 0,25 0,32 0,22 Table 1. Mean Prediction error by using neural prediction based respectively on confidence measure only (CM ), on semantic compacity index SCI and the combination of the 2 index ones (CM + SCI). Prediction errors rates (P ER) are computed by using arithmetic error (Q1) and root mean square metrics (RM S).

In a second experiment, we investigate the interest of the proposed indexability predictor in the particular usage scenario where a metric is supposed to indicate, to a human operator, the segments that should be manually checked. It is a document classification task where each document is tagged as indexable or unindexable by the system. We estimate the classification performance by comparing the 2-classes tagging performed by using the real indexability score and predicted score, according to a common fixed threshold. A document is tagged as well-classified, if indexability and predicted indexability are both under or above the same threshold T . Threshold fluctuates between 10% and 90%. Results (cf. fig. 1) at the limit-conditions correspond to expectations: in the [10%,40%] interval, the worst documents are detected; beyond 70%, the system detects only the best ones. Depending of the chosen tradeoff between indexing-quality and cost, the threshold can be adjusted.

The confidence measure CM yields good classification for an indexability threshold under 50% (never below 80% of classification), but classification performances dramatically decrease under 60%. CM predictor outperforms the SCIbased system, but the combined approach CM + SCI clearly provides the best results. It enables to classify documents correctly with T in [50, 90] %. The best improvement is observed for a T value at 70%: confidence measure and semantic prediction obtain only 62% at 61% at this point, but the combination reaches 74%, corresponding to an improvement of about Fig. 1. Indexable/unindexable document classification according to the indexability threshold, by using the predicted indexability based on confidence measure (CM), semantic compacity index (SCI) and the combination of CM and SCI (CM+SCI). 13% relative. In most of the cases, this combined system correctly tags more than 70% of the speech segments.

CONCLUSION AND PERSPECTIVES

In this study, we investigated the interest of semantic level information to prior estimate of the transcription quality in the specific scope of spoken document retrieval. We introduced an method for indexability prediction that combines confidence measure from ASR and a local semantic compacity index. Results demonstrate that semantic information is a useful feature for estimating data-quality for SDR; even if its own prediction performance is worse than the one based on confidence measure, complementarity yields a significant improvement: the prediction error rates are improved of about 13% relative with the combined approach. We plan now to investigate various strategies for semantic modeling; here, we focused on local semantic compacity. Context widening and the use other modeling paradigm (such Latent Dirichlet Allocation) could improve the extraction and identification of the latent concepts in the speech stream.

  4.1. Speech databaseExperiments are conducted on the ESTER database. This database is composed of about 100 hours French radio broadcasts that were manually annotated. The test set is composed by 8 hours from 4 different radio stations. This corpus is split in two parts: the first part is one hour to learn the neural predictor of indexability. The second part is used to test the system on 7 hours (1474 documents)

This research was supported by the ANR agency (Agence Nationale de la Recherche), AVISON project (ANR-007-014)