Camille Guinaudeau
email: camille.guinaudeau@inria.fr

Guillaume Gravier
email: guillaume.gravier@irisa.fr

Pascale Sébillot
email: pascale.sebillot@irisa.fr

Enhancing lexical cohesion measure with confidence measures, semantic relations and language model interpolation for multimedia spoken content topic segmentation

Keywords: Topic segmentation, lexical cohesion, confidence measures, semantic relations, language model interpolation, TV broadcasts

Enhancing lexical cohesion measure with confidence measures, semantic relations and language model interpolation for multimedia spoken content topic segmentation

Introduction

Structuring video feeds has become a requirement and is a highly challenging issue. Indeed, with the proliferation of video-sharing websites and the increasing number of television channels, the quantity of video that users can access has become so important that it is necessary to develop methods to structure this material and enable users to navigate inside the stream. Since videos available are of different kinds (movies, news, talk shows, etc.) and in order to avoid the use of several domain specific methods, such structuring approaches must necessarily be generic enough to treat various types of video. A crucial structuring stage is the segmentation into successive TV shows on the one hand, and of the shows into topically homogeneous segments on the other hand. Topic segmentation of informative content TV shows can rely on the speech pronounced in the programs. In this case, the segmentation is based on the analysis of the distribution of words within the speech, a topic change being detected when the vocabulary changes significantly. With the improvement of automatic speech recognition (ASR) systems in recent years [START_REF] Ostendorf | Speech segmentation and spoken document processing[END_REF], topic segmentation of spoken documents can now be performed on automatic transcripts of the speech material. However, most of the work in this direction directly apply methods developed for textual topic segmentation to automatic transcripts of spoken language. These methods are often based on the notion of lexical cohesion which corresponds to the lexical relations that exist within a text, mainly enforced by reiterations, i.e., repetitions of the same words [START_REF] Mulbregt | Segmentation of automatically transcribed broadcast news text[END_REF][START_REF] Utiyama | A statistical model for domain-independent text segmentation[END_REF]. Alternately, discourse markers, obtained from a preliminary expert or learning process, can also be used to identify topic boundaries [START_REF] Beeferman | Statistical models for text segmentation[END_REF][START_REF] Christensen | Maximum entropy segmentation of broadcast news[END_REF].

Christensen et al. [START_REF] Christensen | Maximum entropy segmentation of broadcast news[END_REF] have established that transcription errors have little effect on the performance of a supervised segmentation algorithm using discourse markers. However, we have observed a large gap of performance between manual and automatic transcripts in previous work on topic segmentation of radio broadcasts using an unsupervised approach based on lexical cohesion [START_REF] Huet | Un modèle multi-sources pour la segmentation en sujets de journaux radiophoniques[END_REF]. This difference is mostly due to the specifics of the material on which we focus, i.e., TV shows. Indeed, automatic transcripts of TV shows have certain peculiarities that are detrimental to topic segmentation and, in general, to natural language processing. Firstly, the error rate of the ASR system used, even if it remains reasonable for news, can be as high as 70% for challenging programs such as talk shows or debates. Moreover, TV programs are composed of topic segments that can be very short and contain few repetitions of vocabulary, particularly in news where journalists make use of synonyms to avoid reiterations. In our corpus, we have measured that a word occurs on average 1.8 times in a topically coherent segment in broadcast news and 2.0 times in reports on current affairs (for more details, cf. Section 4.3). In order to overcome difficulties related to transcription errors, some studies have suggested to add features specific to spoken documents to the sole concept of lexical cohesion. For example, [START_REF] Amaral | Topic indexing of TV broadcast news programs[END_REF] exploits speaker detection to locate the anchor speaker in news program, relating anchor speaker occurences with new reports and hence with topic changes. In [START_REF] Stolcke | Combining words and speech prosody for automatic topic segmentation[END_REF], prosody is used in addition to automatic transcription. However, such clues are seldom used in practice because their automatic extraction is difficult. Moreover, they are highly dependent on the type of documents and therefore imply domain and genre specific knowledge.

The aim of this paper is to propose a segmentation method able to deal with peculiarities of professional videos (transcription errors, possibly short segments and limited number of repetitions) while remaining generic enough. This method is based on the criterion of lexical cohesion which is not dependent on a type of document since it is mainly enforced by word repetitions. However lexical cohesion is not efficient when the number of reiterations is low-i.e., when synonyms are used or topic segments are very short-and is sensitive to transcription errors, two characteristics of our video material. We propose several extensions to a measure of lexical cohesion, based on generalized probabilities using a unigram language model, in order to make this criterion more robust to spoken content. On the one hand, the measure of the lexical cohesion is modified by an original technique that incorporates two sources of additional information: semantic relations, highlighting the semantic proximity between words, and confidence measures. On the other hand, we propose to use language model interpolation techniques so as to provide better estimates of the lexical cohesion on short segments.

The paper is organized as follows: we first present the topic segmentation method, based on lexical cohesion, developed for the segmentation of textual documents and used as a baseline in this work. In Section 3, extensions of the probabilistic lexical cohesion measure to improve robustness to TV program specifics are described. The experimental setup is presented in Section 4. Finally, experimental results are extensively discussed in Section 5, before the presentation of future work.

Topic segmentation

Within the framework of TV stream structuring, the objective of topic segmentation is to split relevant shows (broadcast news and reports on current affairs in this work) into segments that deal with a single topic. Topic segmentation algorithms can be based on the criterion of lexical cohesion. In this case, the segmentation relies on the analysis of the distribution of words within the text, a topic change being detected when the vocabulary changes significantly [START_REF] Utiyama | A statistical model for domain-independent text segmentation[END_REF][START_REF] Hearst | TextTiling: Segmenting text into multi-paragraph subtopic passages[END_REF].

In this section, the lexical cohesion criterion is first described, as it is usually used in the context of topic segmentation, before the presentation of the topic segmentation method of Utiyama and Isahara [START_REF] Utiyama | A statistical model for domain-independent text segmentation[END_REF] which serves as a baseline in this work.

Lexical cohesion

The notion of lexical cohesion refers to lexical relations that exist within a text to provide a certain unity. Lexical cohesion is created by repetitions of the same words, co-references, and the use of sets of semantically related words [START_REF] Halliday | Cohesion in English[END_REF]. As lexical cohesion is a guide to the organization of the flow of ideas in the text, this criterion is widely used in many fields of the natural language processing domain. In discourse analysis, [START_REF] Xingwei | The relationship between cohesion and coherence[END_REF] studies the relationship between cohesion and coherence in texts while [START_REF] Klebanov | Lexical cohesion analysis of political speech[END_REF] makes a stylistic analysis of political speeches. Others studies use lexical cohesion for word sense disambiguation [START_REF] Manabu | Word sense disambiguation and text segmentation based on lexical cohesion[END_REF], and automatic summarization [START_REF] Barzilay | Using lexical chains for text summarization[END_REF][START_REF] Boguraev | Lexical cohesion, discourse segmentation and document summarization[END_REF]. Error identification and correction applications can also rely on lexical cohesion to detect errors by identifying tokens that are semantically unrelated to their context. These methods can be applied on regular text, as in [START_REF] Hirst | Correcting real-word spelling errors by restoring lexical cohesion[END_REF], or on automatic transcripts [START_REF] Inkpen | Semantic similarity for detecting recognition errors in automatic speech transcripts[END_REF].

In [START_REF] Utiyama | A statistical model for domain-independent text segmentation[END_REF], the lexical cohesion computation of a segment S i consists in the evaluation of the ability of a language model ∆ i whose parameters are estimated from words in S i to predict the words in the segment. In this framework, two important steps are needed: the estimation of the language model ∆ i and the computation of the generalized probability of words in S i , reflecting the ability of the language model ∆ i to predict the words of S i .

Language model. The language model ∆ i estimated on S i is a unigram language model [START_REF] Utiyama | A statistical model for domain-independent text segmentation[END_REF] which specifies a distribution over all the words of the text (or transcript) to be segmented. The calculation of the language model of the segment S i is formalized, for a Laplace smoothing, by

∆ i = P i (u) = C i (u) + 1 z i , ∀ u ∈ V K , (1)
with V K the vocabulary of the text of size K and C i (u) the count of word u in S i corresponding to the number of occurrences of u in S i . The probability distribution is smoothed by incrementing the count of each word by 1. The normalization term z i ensures that ∆ i is a probability mass function and, in the particular case of Eq. 1, z i = K + n i with n i the number of words in S i .

Generalized probability. The language model ∆ i is used to compute the generalized probability of the words in S i as a measure of lexical cohesion according to

ln P [S i ; ∆ i ] = n i j=1 ln P [w i j ; ∆ i ] , (2)
where w i j denotes the j th word in S i . Intuitively the probability favors lexically consistent segments since its value is greater when words appear several times within the segment and decreases if many words are different.

Topic segmentation

The topic segmentation method used, introduced by Utiyama and Isahara [START_REF] Utiyama | A statistical model for domain-independent text segmentation[END_REF] for textual documents, was chosen in this context of transcript-based TV program segmentation for two main reasons. It is currently one of the best performing method that makes no assumption on a particular domain (no discourse markers, no topic models, etc.). Moreover, contrary to many methods based on local measures of the lexical cohesion, the global criterion used in [START_REF] Utiyama | A statistical model for domain-independent text segmentation[END_REF] makes it possible to account for the high variability in segment lengths.

The method consists in searching the segmentation that produces the most consistent segments from a lexical point of view, while respecting a prior distribution of the segment lengths. Cast in a probabilistic framework, the principle is to find the most probable segmentation of a sequence of l basic units (words or sentences) W = W l 1 among all possible segmentations, Ŝ = argmax

S m 1 P [W |S] P [S] . (3)
Assuming that P [S m 1 ] = n -m , with n the number of words in the text and m the number of segments, and that segments are independent, the probability of a text W for a segmentation S = S m 1 is given by

Ŝ = argmax S m 1 m i=1 ln(P [W b i a i |S i ]) -α ln(n) , (4)
where P [W b i a i |S i ] denotes the generalized probability of the sequence of basic units corresponding to S i as given by Eq. 2. The parameter α allows for different trade-offs between lexical cohesion and segment lengths.

In this paper, the basic units used are utterances as given by the partitioning step of the ASR system, thus limiting possible topic boundaries to utterance boundaries, and the text W is only composed of lemmatized1 nouns, adjectives and non modal verbs.

Robust topic segmentation for spoken multimedia contents

The language model based lexical cohesion measure, as defined in Section 2.1, relies only on word repetitions. However, this can turn out to be insufficient in the case of automatically transcribed video material. Indeed, two occurrences of a word can be erroneously recognized as two different words and therefore not considered for lexical cohesion. Moreover, due to potentially small segment lengths and to the use of synonyms, the number of repetitions can be very low.

To adjust the computation of lexical cohesion to spoken documents, we propose several extensions. In the first ones, additional information is incorporated to the lexical cohesion measure while, in the second one, language model interpolation techniques are used so as to povide better language model estimates.

Integration of additional sources of information

Different kinds of additional information, such as prosody or confidence measures, can be used to improve the generalized probability measure of lexical cohesion. In this work, confidence measures and semantic relations are employed. Confidence measures, associated with each word by the ASR system, correspond to (an estimation of) the probability that a word has been correctly transcribed. They are considered to reduce the contribution of non properly transcribed words in the lexical cohesion calculation. Semantic relations, which represent the semantic proximity between words, are used to consider the fact that two different words can be semantically related and seen as a repetition. Such information can be accounted for either during the language model estimation step or during the calculation of the generalized probability.

Using confidence measures

Confidence measures can be accounted for at the language model level by replacing the count C i (u) by the sum of the confidences over all occurrences of u, i.e.,

C ′ i (u) = w i j =u c(w i j ) δ 1 , (5)
where c(w i j ) ∈ [0, 1] corresponds to the value of the confidence measure for the j th word in S i and δ 1 is a parameter used to reduce the weight of words whose confidence measure is low. Indeed, the higher δ 1 , the lower the impact of words with a low confidence measure.

Confidence measures can also be accounted for during the generalized probability computation. In this case, the log-likelihood of the occurrence of a word in a segment is multiplied by the confidence measure of the word occurrence,

ln P [S i ; ∆ i ] = n i j=1 c(w i j ) δ 2 ln P [w i j ; ∆ i ] , (6)
with δ 2 equivalent to δ 1 . Eq. 6 allows to reduce the contribution to the lexical cohesion of a word whose confidence measure is low. In this case, the language model ∆ i can be either estimated from the counts C i (u), thus limiting the use of confidence measures to the probability computation, or from the modified counts C ′ i (u). Note that in Eq. 6, ln P [S i ; ∆ i ] is not strictly speaking a log probability. However, if δ 2 = 1, ln P [S i ; ∆ i ] can be seen as the joint probability for the word w i j to be correctly transcribed and to be represented by the language model ∆ i .

Using semantic relations

Many topic segmentation techniques, based on the lexical cohesion criterion, use semantic relations as additional information to take into account the semantic links that exist between words. Indeed, the primary goal of semantic relations is obviously to ensure that two semantically related words, e.g., "car" and "drive", contribute to the lexical cohesion, thus avoiding erroneous topic boundaries between two such words. These different methods can use semantic relations that are manually defined by experts, as in [START_REF] Morris | Lexical cohesion computed by thesaural relations as an indicator of the structure of text[END_REF], or extracted automatically from corpora [START_REF] Ferret | Approches endogène et exogène pour améliorer la segmentation thématique de documents[END_REF][START_REF] Jobbins | Text segmentation using reiteration and collocation[END_REF]. For example, in [START_REF] Ferret | Approches endogène et exogène pour améliorer la segmentation thématique de documents[END_REF], Ferret uses a network of lexical co-occurrences built from a large corpus to improve the results of a topic segmenter based on lexical reiteration. The algorithm in [START_REF] Jobbins | Text segmentation using reiteration and collocation[END_REF] compares adjacent windows of sentences and determines their lexical similarity, based on repetitions of words and collocations, to detect topic boundaries.

In our work, as in [START_REF] Morris | Lexical cohesion computed by thesaural relations as an indicator of the structure of text[END_REF][START_REF] Ferret | Approches endogène et exogène pour améliorer la segmentation thématique de documents[END_REF][START_REF] Jobbins | Text segmentation using reiteration and collocation[END_REF], semantic relations are employed to overcome the limited number of vocabulary repetitions. But semantic relations are also expected to limit the impact of recognition errors. Indeed, contrary to correctly transcribed words, misrecognized words are unlikely to be semantically linked to other words in a topic segment [START_REF] Inkpen | Semantic similarity for detecting recognition errors in automatic speech transcripts[END_REF]. As a consequence, the contribution of non properly transcribed words will be less important than the one of correct words.

As for confidence measures, accounting for semantic relations can be achieved by modifying the counts in the language model estimation step. Counts, which normally reflect how many times a word appears in a segment, are extended so as to emphasize the probability of a word based on its number of occurrences as well as on the occurrences of related words. More formally, the counts C i in Eq. 2 are amended according to

C ′′ i (u) = C i (u) + n i j=1,w i j =u r(w i j , u) , (7)
where r(w i j , u) ∈ [0, 1] denotes the semantic proximity between w i j and u, close to 1 for highly related words and null for non related words. More details about the computation of r are given in Section 4.2.

Unlike confidence measures, semantic relations cannot be accounted for during generalized probability computation. Indeed, in this case, it does not really make sense to multiply the probability for a word to appear in a segment S i by the sum of the relations the word maintains with other words in S i .

To conclude this section, it is important to note that, in general, semantic relations are obtained from domain specific corpora. They can also be learned on a general purpose corpus but will in any case fail to address all domains. One strong point of our technique is that when semantic relations are not adequate for a particular document-e.g., when segmenting a document from a domain not in the semantic relations training data-C ′′ i (u) will remained unchanged with respect to C i (u) since r(u, v) is null between any two words of the document. Put differently, out of domain relations will have no impact on topic segmentation, contrarily to latent semantic approaches [START_REF] Deerwester | Indexing by latent semantic analysis[END_REF][START_REF] Landauer | Introduction to latent semantic analysis[END_REF].

Combining global and local measurements of lexical cohesion

In TV programs, and especially in news, topic segments can be very short. An easy, and unfortunately accurate, criticism of Eq. 1 is that for small segments, the language model ∆ i is poorly estimated. Therefore, in order to deal with short segments, the use of a more sophisticated estimation method is needed. To skirt this problem, we propose to interpolate the language model at the segment level with a more robust language model estimated on the entire transcript. This allow to take into account the whole text to be segmented in order to have a better language model estimation for short segments. Two interpolation strategies are studied-interpolation of the probabilities [START_REF] Jelinek | Interpolated estimation of Markov source parameters from sparse data[END_REF] and interpolation of the counts [START_REF] Bacchiani | Unsupervised language model adaptation[END_REF].

Linear interpolation of probabilities

The first interpolation technique is a basic probability interpolation. In this case, the lexical cohesion of a segment S i , given the segment S i and the text T , is measured according to ln P [S i ; S i , T ] =

n i j=1 ln(λP [w i j ; ∆ i ] + (1 -λ)P [w i j ; ∆ t ]) = n i j=1 ln     λ C i (w i j ) + ξ u∈V T C i (u) + ξ + (1 -λ) C t (w i j ) u∈V T C t (u)     , (8)
where ∆ i is the language model estimated on S i and ∆ t the one estimated on T . C t (u) is the count of word u in T and C i (u) the count of that word in S i . ξ is a count smoothing bias that corresponds to the Laplace smoothing when ξ = 1.

Count interpolation

Rather than interpolating probabilities, language model interpolation can be based on the interpolation of counts. In this case, the lexical cohesion of a segment S i is defined as

ln P [S i ; S i , T ] = n i j=1 ln P [w i j ; ∆ it ] = n i j=1 ln     λ(C s (w i j ) + ξ) + (1 -λ)C t (w i j ) u∈V T λ(C i (u) + ξ) + (1 -λ)C t (u)     , (9)
where ∆ it is the interpolated language model of the segment S i and the text T . As for linear interpolation, frequent words in T will get a high probability regardless of their frequency in S i while non frequent ones will always get a low probability-depending on λ. However, because of the renormalization by the sum of all the counts, this fact might be less detrimental than for probability interpolation and the behavior of this interpolation technique more likely to be consistent with what is expected.

Experimental setup

Experiments were carried out with our speech recognition system on a comprehensive corpus. Before presenting the corpus, a brief description of the ASR system is provided and the acquisition and selection of semantic relations is discussed.

ASR system and confidence measures

All TV programs were transcribed using our Irene ASR system, originally developed for broadcast news transcription. Irene implements a multiple pass strategy, progressively narrowing the set of candidate transcriptsthe search space-in order to use more complex models. In the final steps, a 4-gram language model over a vocabulary of 65,000 words is used with context-dependent phone models to generate a list of 1,000 sentence transcription hypotheses. Morphosyntactic tagging, using a tagger specifically designed for ASR transcripts, is used in a post-processing stage to generate a final transcription by consensus from a confusion network, combining the acoustic, language model and morphosyntactic scores [START_REF] Huet | Morpho-syntactic post-processing of Nbest lists for improved French automatic speech recognition[END_REF]. Confusion network posterior probabilities are used directly as confidence measures.

Acoustic models were trained on about 250 hours of broadcast news material from the French ESTER 2 data [START_REF] Galliano | The ESTER 2 evaluation campaign for the rich transcription of French radio broadcasts[END_REF]. The language model probabilities were estimated on 500 million words from French newspapers and interpolated with language model probabilities estimated over 2 million words corresponding to the reference transcription of radio broadcast news shows. The system exhibits a word error rate (WER) of 16 % on the non accented news programs of the ESTER 2 evaluation campaign. As far as TV contents are concerned, we estimated word error rates ranging from 15 % on news programs to more than 70 % on talk shows or movies.

Semantic relations

Semantic relations were automatically extracted from text corpora and relevant relations were selected so as to avoid abusive enforcement of the lexical cohesion.

Acquisition

Automatic extraction of semantic relations from text corpora has been widely studied [START_REF] Manning | Foundations of statistical natural language processing[END_REF][START_REF] Claveau | From effiency to portability: Acquisition of semantic relations by semi-supervised machine learning[END_REF][START_REF] Grefenstette | Corpus-derived first, second and third-order word affinities[END_REF]. Two main types of relations might be of interest for lexical cohesion, namely syntagmatic and paradigmatic relations.

Syntagmatic relations correspond to relations of contiguity that words maintain within a given syntactic context (sentence, chunk, fixed length window, etc.), two words being related if they often appear together. A popular criterion to measure the degree of syntagmatic proximity between two words u and v is the mutual information (MI) which compares the probability of observing the two words u and v together with the probability of observing these two words separately [START_REF] Grefenstette | Corpus-derived first, second and third-order word affinities[END_REF]. Several variants of the mutual information criterion have been proposed. For example, in [START_REF] Daille | Study and implementation of combined techniques for automatic extraction of terminology[END_REF], the mutual information is cubed to avoid emphasizing rare associations. In [START_REF] Church | Word association norms, mutual information, and lexicography[END_REF], order between words is taken into account.

Paradigmatic relations link two words having an important common component from a meaning point of view. These relations, corresponding to synonyms, hyperonyms, antonyms, etc., are typically calculated by means of context vectors for each word, grouping together words that appear in the same contexts. The context vector of a word u describes the distribution of words in its vicinity and contains, for each word v, its frequency of occurrence in the neighborhood of u, possibly normalized by its average frequency in the neighborhood of any word. The semantic proximity between two terms can be defined thanks to the Jaccard index as in [START_REF] Grefenstette | Sextant: Exploring unexplored contexts for semantic extraction from syntactic analysis[END_REF] or as the angular distance between their respective context vectors. In this work, the MI 3 criterion for syntagmatic relations and the cosine distance between normalized context vectors for paradigmatic ones were used. Relations were extracted from a corpus of articles from French newspapers (about 800 million words) and from reference transcripts of radio broadcast news shows (about 2 million words). All the data were lemmatized, keeping only nouns, adjectives, and non modal verbs. Semantic proximity scores given either by the MI 3 criterion or by the angular distance were normalized in [0,1]. Table 1 shows, for the word "cigarette", the five related words with the highest semantic proximity score, for syntagmatic and for paradigmatic relations.

Selection

To prevent the creation of too many links between words, a selection step is implemented to choose relevant syntagmatic and paradigmatic relations among all the existing ones so as to introduce only the best ones in the segmentation algorithm. Two selection strategies were evaluated, either retaining globally the N relations with the highest scores (global selection strategy) or retaining for each word the M best relations (word level selection strategy). Moreover, common usage words such as "go" or "year" were found to be related to many words, thus jeopardizing topic segmentation by abusively enforcing lexical cohesion. Semantic relations for words whose number of related words exceeds a given threshold are therefore discarded, the threshold being equal to the average number of relations associated with a word multiplied by a parameter γ ∈ [0, 1].

Corpus

Results are reported on two distinct corpora. The first one, a news corpus, is made up of 57 news programs (≈ 1/2 hour each) broadcasted in February and March 2007 on the French television channel France 2, and the second one is a reports corpus composed of 16 reports on current affairs "Sept à Huit" (≈ 1 hour each) transmitted on the French channel TF1 between September 2008 and February 2009. In the reports corpus, reports are longer (around 10-15 minutes) and possibly on non news topics, while the news corpus follows the classical scheme of rather short reports (usually 2-3 minutes). Having two distinct corpora makes it possible to study the behavior of topic segmentation on data sets with different characteristics. Indeed, in addition to different durations, the average number of topics and the average number of segments per show vary greatly between news and reports. Moreover, the number of repetitions is less important in news programs than in reports ones, as reported in Table 2, while the transcription error rate is higher on the latter due to a larger amount of non professional speakers.

In each show, headlines and closing remarks were removed, these two particular parts disturbing the segmentation algorithm and being easily detectable from audiovisual clues. A reference segmentation was established by considering a topic change associated with each report, the start and end boundaries being respectively placed at the beginning of the report's introduction and at the end of the report's closing remarks. Note that in the news corpus, considering a topic change between each report is a choice that can be questioned as, in most cases, the first reports all refer to the main news of the day and are therefore dealing with the same broad topic.

Results

The goal of the article is to study to which extent confidence measures, semantic relations and interpolation techniques can help to make the lexical cohesion criterion more robust for professional video processing. We therefore study the influence of each technique and each parameter on the entire data set, reporting results for optimal parameter values. Even though the resulting figures are optimistic and do not reflect real-life behavior, their comparison with a baseline system clearly demonstrates the impact of each technique on topic segmentation.

Recall and precision on topic boundaries are considered for evaluation purposes after alignment between reference and hypothesized boundaries, with a tolerance on boundary locations of respectively 10 and 30 s for news and reports, while different trade-offs between precision and recall are obtained by varying α in Eq. 4. To compare the different parameters (δ, γ, etc.), the tables in this section contain the best F1-measure obtained for an optimal value of α, i.e., the one leading to the segmentation with average segment length closest to that of the reference segmentation.

Confidence measures

Results for the integration of confidence measures (CMs) are presented in Tables 3 and4 for the news and reports corpus respectively. In these tables, the grey line stands for the values obtained when CMs are introduced during the language model estimation alone, with δ 1 varying from 0 to 4, while the grey column represents their integration during the generalized probability computation alone. The dark grey cell therefore corresponds to the baseline method (δ 1 = δ 2 = 0). Recall and precision curves are reported on Figure 1.

We can observe that confidence measures integration leads to an improvement in the topic segmentation quality. Indeed, it can be seen that the best value of the F1-measure is improved by 2 points for the news corpus and by 3.7 points for the reports corpus.

Moreover, results show that the behavior of the integration of confidence measures is different for the two corpora depending on whether CMs are integrated during generalized probability or language model computation. For the news corpus, results are better when confidence measures are integrated during the generalized probability computation alone rather than during the language model computation alone; while for the reports corpus, experiments lead to the opposite conclusion. This phenomenon is also observed on the recall/precision curves, where the integration during the generalized probability calculation (green curve) gives better results for the news corpus, while integration during the language model computation is more efficient for reports.

Another difference between the two corpora is the optimal values of δ 1 and δ 2 . For the news corpus, the highest F1-measure is obtained when δ 1 and δ 2 are quite small (both equal to 1) while for the reports corpus the optimal values for δ 1 and δ 2 are greater. This difference can be explained by the fact that for high values of δ, c(w i j ) δ becomes negligible except for words whose confidence measure is very close to 1. As the proportion of words with a CM less than 0.9 is more important in the reports data, the impact of the confidence measures is more perceptible on this data set and higher values of δ lead to a greater improvement.

Similarly, the impact of the transcription quality on the efficiency of the integration of confidence measures is also highlighted by the difference between the improvements observed for the two corpora. Figure 1 shows that confidence measures have more impact on the reports corpus than on the news corpus. This difference shows that confidence measures are of utmost importance when transcription quality decreases. From a more qualitative point of view, we also observed that accounting for confidence measures not only increases the number of correct boundaries detected but also improves boundary locations. Indeed, boundary locations are more precise when using confidence measures, even if this fact does not show in the recall/precision curves because of the tolerated gap on the boundary positions. We also observed that the quality of confidence measures have an impact on boundary precision. Indeed, using confidence measures improved thanks to high-level linguistic features with a classifier [START_REF] Fayolle | CRF-based combination of contextual features to improve a posteriori word-level confidence measures[END_REF] resulted in more accurate boundary locations.

To conclude, experiments carried out on the two corpora confirm the hy-pothesis that the integration of confidence measures in the lexical cohesion computation allows the criterion to be less sensitive to transcription errors. Indeed, amending the calculation method of lexical cohesion to include CMs provides a significant improvement in the quality of topic segmentation. Confidence measures seem also to lead to an increase in the number of correct boundaries detected but also to the displacement of previously recognized borders, moving them closer to reference boundaries.

Semantic relations

Various tests, on the choice of the type and number of semantic relations introduced and on the use of the selection strategy (global or word level) or the filtering technique have produced a large number of results. The most convincing ones are discussed here.

Tables 5 and6 summarize the results on the news and reports corpus respectively, with values for syntagmatic relations in the leftmost part of the tables and results for paradigmatic ones in the rightmost part. For each kind of semantic relations, the two selection strategies are evaluated for different numbers of semantic relations, 2 to 10 per word for the word-level strategy and 5,000 to 90,000 for the global strategy. Finally, the results for the filtering technique, used to discard semantic relations for words whose number of related words exceeds a given threshold, are presented for values of parameter γ between 0.2 and 1. In Figure 2, recall/precision curves for the integration of paradigmatic relations are presented.

For the news corpus, semantic relations can help the lexical cohesion criterion to be more robust to the limited number of vocabulary repetitions due to short segments and/or the use of synonyms or related words. Indeed, the best F1-measure is improved by 1.4 when syntagmatic relations are introduced and by 0.6 for paradigmatic relations (Table 5). Moreover, even if the best values of the F1-measure are reached for syntagmatic relations, the global improvement is higher with paradigmatic ones. For the reports corpus, the improvement is much smaller than for the news one. Indeed, Figure 2 shows that results for the integration of paradigmatic relationswhich are comparable to those for syntagmatic relations, cf. Table 6-are equivalent to the baseline. The difference in results for both corpora be explained by the fact that in the reports corpus the number of reiterations and the segments lengths are more important than in the news corpus. Thus, the use of semantic relations is not as effective for this corpus. Moreover, in this work, semantic relations were extracted from a corpus composed of news articles and are therefore less suited for the reports corpus. However, it is interesting to note that using non appropriate relations does not degrade results.

Table 6 shows that the integration of semantic relations damages the segmentation quality when a too large number of relations is introducedi.e., when the parameter γ is high-degradation which is proportional to the number of relations included. This behavior can also be observed for the news corpus but for higher values of γ. Therefore, the filtering technique is crucial to avoid considering relations that seem less adapted in the context of topic segmentation because they introduce some noise (i.e., they connect words and segments that should not be). The stronger effect of the filtering technique on the reports corpus can be explained by the fact that many relations are out of domain for this corpus. Therefore, more general relations-associated with words such as "year" and "go"-that introduce noise appear in a higher proportion and have more impact on reports than on news.

Finally, concerning the technique used for the semantic relations selection, no difference was found between the global selection strategy and the word level selection strategy for the news corpus. In contrast, for the reports corpus, the global selection strategy is better, both for paradigmatic and syntagmatic relations. We believe that the word level strategy selects relations which are the most characteristic of the corpus they are learned from, while the global strategy is more likely to pick more general relations. Thus, as relations are extracted from a corpus composed of news articles, the ones selected by the word level strategy are more suitable for the news corpus but less adapted for the reports one.

To conclude, the introduction of semantic relations can improve topic segmentation when the number of vocabulary repetitions is low. However, it is essential to limit the number of relations considered.

Interpolation

Finally, results for language model interpolation are given in Table 7. Values for the count interpolation strategy are presented in the leftmost part of the table while results for the linear interpolation of probabilities are located in the rightmost one.

For the two corpora, we can observe that the improvements in the topic segmentation quality are higher with the interpolation of the counts than with the linear interpolation of probabilities. Indeed, the best value for the F1-measure is increased by 4.9 against 2.3 for the news corpus and by 0.7 against 0 for the reports corpus. This behavior is explained by the fact that, as mentioned in Section 3.2, for the linear interpolation of probabilities, frequent words in the text get a high probability regardless of their frequencies in the segment, while non frequent ones always get a low probability. However, because of the renormalization in Eq. 9, this fact is less detrimental for count interpolation than for probability interpolation. Table 7, as well as the recall/precision curves in Figure 3, shows that the interpolation techniques are much less effective for the reports corpus than for the news corpus. This observation can be easily explained by the fact that segments in reports are four times longer than the ones in the news. Thus, the use of interpolation has no effect for the calculation of lexical cohesion in the reports segments since the initial language models measure more robust to TV program specifics while being generic enough to be effective on different kinds of programs. First, it has been shown that the use of semantic relations and interpolation techniques improve the topic segmentation quality of TV programs divided in short segments and in which lexical repetition rates are low. It was also pointed out that the integration of confidence measures has more impact when the transcription quality is lower, as for the reports corpus. Finally, we demonstrated that these different elements can be used together to combine their advantages. Interestingly, it was also found that this combination never leads to a deterioration of topic segmentation.

These results clearly lead to the integration of other features, such as prosodic ones, to detect lexical stresses or speaker changes for example. Moreover, as the integration of confidence measures has a positive impact, we think it would be interesting to investigate the application of the topic segmentation method on intermediary outputs of the ASR system (such as word graphs or confusion networks) rather than on the final transcripts. Finally, as mentioned in Section 4.3, the structure of news programs is clearly hierarchical as, in most cases, the first reports all refer to the main news of the day and are therefore dealing with the same broad topic. However, in this work, only linear topic segmentation has been considered. Therefore, a long view prospect of this paper is to develop a method that can handle a hierarchical topic segmentation of our data.

Figure 1 :

 1 Figure 1: Integration of confidence measures -Recall/Precision curves (figures in the legend correspond to resp. δ 1 and δ 2 )

Table 1 :

 1 Words with the highest association scores, in decreasing order, for the word "cigarette", as extracted automatically.

	syntagmatic	paradigmatic
	fumer (to smoke)	cigare (cigar )
	paquet (pack )	gitane 2
	allumer (to light)	gauloise 2
	contrebande (smuggling)	clope (ciggy)
	fabricant (producer )	tabac (tobacco)

Table 2 :

 2 Comparison of the news and reports corpora

		average	average	average	average	#topic
		#repetitions confidence	#words	#words boundaries
			measure	per segment per file
	news	1.82	0.62	106.4	2,243	1,202
	reports	2.01	0.57	424.1	2,495	86

Table 3 :

 3 Integration of confidence measures -news corpus -best F1-measure

	δ 2 \ δ 1	0	0.5	1	1.5	2	2.5	3	3.5	4
	0	59.7 60.4 60.7 61.0 61.1 60.9 60.9 59.7 59.1
	0.5	60.4 60.7 61.5 61.5 60.9 60.9 60.8 59.5 58.8
	1	61.1 61.5 61.7 61.1 60.9 60.9 60.6 58.9 58.8
	1.5	61.2 61.6 61.3 60.9 60.5 60.1 60.2 58.9 58.8
	2	61.3 61.4 61.4 60.8 60.2 59.9 59.9 58.9 58.7
	2.5	61.2 61.5 61.1 60.5 60.1 59.6 59.6 58.6 58.1
	3	61.6 61.5 60.8 60.4 60.0 59.5 59.4 58.1 57.9
	3.5	60.6 60.5 59.3 58.8 58.8 58.3 58.1 57.8 57.8
	4	60.9 60.6 59.3 59.0 58.9 58.1 57.8 57.8 57.7

Table 4 :

 4 Integration of confidence measures -reports corpus -best F1-measure

	δ 2 \ δ 1	0	0.5	1	1.5	2	2.5	3	3.5	4
	0	54.9 55.0 56.8 54.6 54.7 55.6 58.3 59.9 59.1
	0.5	53.4 56.1 55.3 56.1 54.7 54.7 55.0 57.9 58.2
	1	55.3 56.5 55.4 55.6 54.7 55.0 55.0 59.1 58.2
	1.5	54.4 55.6 55.6 56.3 54.7 55.0 55.0 56.9 58.6
	2	54.9 56.0 56.3 55.0 55.3 55.3 57.0 56.1 56.1
	2.5	55.1 55.7 56.3 55.0 55.3 57.0 57.0 56.1 56.5
	3	55.3 56.8 56.3 55.0 56.7 57.0 58.6 57.7 58.1
	3.5	55.3 56.8 56.3 56.3 58.3 58.6 58.6 57.7 57.7
	4	54.5 56.8 56.3 59.9 59.5 58.6 57.7 57.7 57.1

Table 5 :

 5 Integration of semantic relations -news corpus

				Syntagmatic					Paradigmatic
		Word level	Global		Word level	Global
	γ	2	3	10	5k	20k	90k	2	3	10	5k	20k	90k
	0.2	59.7	59.7	59.7	59.7	59.7	59.7	59.4	59.7	59.7	59.7	59.7	59.7
	0.4	59.7	59.7	60.8	59.7	59.8	59.7	59.7	59.7	59.7	59.7	59.7	60.0
	0.5	59.7	59.7	60.1	59.7	60.3	60.4	59.7	59.7	60.0	59.7	59.7	60.3
	0.6	59.7	59.6	60.5	59.7	60.0	60.1	59.7	59.7	60.2	59.7	60.1	60.1
	0.8	60.0	59.8	60.9	60.5	60.6	60.6	59.8	60.3	60.0	59.9	60.6	60.3
	1	58.9	61.1	59.8	60.5	60.8	61.0	60.9	60.4	60.2	60.0	60.3	60.3

Table 6 :

 6 Integration of semantic relations -reports corpus

				Syntagmatic					Paradigmatic
		Word level	Global		Word level		Global
	γ	2	3	10	5k	20k	90k	2	3	10	5k	20k	90k
	0.2	54.9	54.9	54.9	54.9	54.9	54.9	54.9	54.9	54.9	54.9	54.9	54.9
	0.4	54.9	54.9	55.7	54.9	55.5	55.5	54.9	54.9	55.5	54.9	54.9	54.9
	0.5	54.9	54.9	54.0	54.9	55.5	55.5	54.9	54.9	55.1	54.9	54.9	55.1
	0.6	54.9	54.9	53.3	54.9	54.4	54.4	54.9	54.9	54.0	54.9	54.9	54.4
	0.8	56.5	55.7	53.4	55.5	53.1	53.1	55.8	53.2	53.6	54.9	55.8	54.8
	1	54.7	54.6	51.6	55.1	54.0	54.0	55.2	53.0	53.0	54.9	52.3	54.0

A lemma is an arbitrary canonical form grouping all inflections of a word in a grammatical category, e.g., the infinitive form for verbs, the masculine singular form for adjectives, etc.

Brand name.

Acknowledgments

This work was partially funded by OSEO, French state agency for innovation, in the framework of the Quaero project.

were already well estimated. This can also be seen from the optimal value of parameter λ, representing the weight of the language model of the segment in the interpolation, that is greater for reports than for news.

Combination

Confidence measures, semantic relations and language model interpolation can be used in conjunction. In this part, the joint integration of these Confidence + Interpolation. When confidence measures and interpolation techniques are integrated together, the improvements observed for both are combined for the news corpus. In this case, the CMs introduced are the ones that exhibit the most important amelioration (with δ 1 = δ 2 = 1) and the interpolation technique is the interpolation of counts with λ equals to 0.2. Figure 4 shows that the combination of the two cues leads to a greater improvement in the topic segmentation quality than the sole interpolation, especially for high recall values. Thus, we can conclude that the problems inherent in TV programs, i.e., transcription errors, potentially short segments and limited number of reiterations, are partially handled for the news corpus. For reports, unsurprisingly, comparable results are obtained with or without interpolation when confidence measures are used (cf. Fig. 4), interpolation alone yielding no improvement on this corpus. However it is important to note that the combination does not damage topic segmentation compared to the sole use of CMs.

Confidence + Semantics. Confidence measures can also be combined with semantic relations, whether syntagmatic or paradigmatic. For the corpus composed of broadcast news, the relations introduced are paradigmatic ones, selected thanks to the word level selection technique (2 relations per word). In this case, topic segmentation is slightly improved compared to the use of confidence measures or semantic relations alone. For reports, the combination does not lead to a better topic segmentation, compared with the use of confidence measures alone (cf. Table 8). This observation is not surprising as the semantic relations do not give real improvements for this corpus as explained in Section 5.2. However, as before, the use of two different cues does not damage the topic segmentation quality.

Semantics + Interpolation. Finally, semantic relations can also be combined with interpolation techniques. This combination, possibly associated with the use of confidence measures, does not improve the topic segmentation quality, either for the news or the reports corpus. As semantic relations and especially interpolation techniques both allow to overcome problems related to small segments, their combination is redundant.

Conclusion and future work

In this paper, we have improved and extended a probabilistic measure of lexical cohesion to include confidence measures and semantic relations, making use of language model interpolation. This results in a lexical cohesion