Johann Poignant

Laurent Besacier

Viet Bac Le

Sophie Rosset

Georges Quénot

Unsupervised naming of speakers in broadcast TV: using written names, pronounced names or both ?

Keywords: Speaker identification, OCR, ASR

Persons identification in video from TV broadcast is a valuable tool for indexing them. However, the use of biometric models is not a very sustainable option without a priori knowledge of people present in the videos. The pronounced names (PN) or written names (WN) on the screen can provide hypotheses names for speakers. We propose an experimental comparison of the potential of these two modalities (names pronounced or written) to extract the true names of the speakers. The names pronounced offer many instances of citation but transcription and named-entity detection errors halved the potential of this modality. On the contrary, the written names detection benefits of the video quality improvement and is nowadays rather robust and efficient to name speakers. Oracle experiments presented for the mapping between written names and speakers also show the complementarity of both PN and WN modalities.

Introduction

Nowadays, with the growing number of audio-visual content available, the automatic identification of people appears as very useful for searching and browsing in this type of data. Such person identification may for instance be based on speaker recognition technology. However, training biometric models of speakers requires costly manual annotations of video contents.

As we can not consider the manual annotation of each new video source as a viable option, an interesting alternative is the use of unsupervised approaches for naming people in multimedia documents. To this end, we can automatically classify each speech turn with an anonymous label (i.e. speakers clustering or diarization) and use others sources of information that provide the real person names for at least some of the clusters. When dealing with TV broadcast, at least two different modalities can provide the real names of the persons speaking: (i) the names extracted from the speech transcript (ASR output) and (ii) the names written on the screen by the show to introduce a person (name in OCR output written in a title block 1 ).

In Figure 1, we can see an example from a TV news show including an anchor, a journalist and a person interviewed. In this example, the arrows represent the citation links from written names and pronounced names to a person appearing/talking This work was partly realized as part of the Quaero Program and the QCompere project, respectively funded by OSEO (French State agency for innovation) and ANR (French national research agency). 1 Title block: spatial position used by the show to write a name in order to introduce the corresponding person. in the show. Indeed, in this example, there is a correlation between names pronounced or written and the audio-visual presence of this person in the adjacent speech turns or shots.

Naming people in television programs using automated systems can allow to further address several tasks: Automatic annotation: it can help/complement/replace the manual annotation. This task is recall and precision oriented. Creating models: The automatic extraction of audio segments can be used to build speakers models. Such a task is driven by the accuracy of the speaker models that must be as pure as possible while having enough signal to generate them. Information retrieval: The answer to a query propose several video segments where a person is present. This task is recall oriented and should be able to hande the maximum number of persons (even those for which no a priori model is available).

This paper focuses on naming speakers in TV broadcast. The pronounced names and written names provide both relevant information to answer to these questions. Previous works mainly used pronounced names. The names written on the screen were seldom used due to the poor image quality which lead to low performance text detection and transcription systems. But the evolution of video quality available brings us to reassess the use of this modality. We therefore propose a comparative study of the potential of pronounced names (obtained via ASR) and written names (obtained via OCR) to identify/name a speaking person in a TV broadcast.

This article begins with an overview of the literature on naming people in broadcast radios and videos. More particularly, we focus on the methods for the extraction of hypothesis names (clustering methods and association name-person being outside the scope of this literature review). Then, we continue with a presentation of the REPERE corpus on which we experimented. Next, we compare the quality of the extraction of pronounced names (ASR) and written names (OCR) using automatic systems. Finally, we evaluate both modalities for an unsupervised speaker detection task on TV broadcast. This is done with an oracle mapping (adjacent speech turns) between person-name whatever the time stamp of the name citation.

State-of-the-art

Previous works concerning the unsupervised naming of people in television or radio, use essentially the same framework:

• Persons clustering (diarization).

• Hypothesis names extraction for each person.

• Hypothesis names/persons mapping (or association). Thereafter, we will focus on methods of extracting hypothesis names for each person. Pronounced names (PN) are mostly used in the state of the art due to the poor quality of the written names (WN) transcription. In the papers we reviewed, three steps, either manual or automatic, were generally used:

PREPRESS PROOF FILE CAUSAL PRODUCTIONS

• Detection and transcription of the speech or written text on the screen.

• Person names detection in the transcription.

• Mapping of each hypothesis name to a speaker.

The first works were proposed by Canseco et al. in [START_REF] Canseco-Rodriguez | Speaker diarization from speech transcripts[END_REF] and [START_REF] Canseco | A Comparative Study Using Manual and Automatic Transcriptions for Diarization[END_REF]. The authors use linguistic patterns set manually in order to determine to which a pronounced name refers: the current speaker ("Hello, I am Joie Chen"), following ("This is Candy Crowley") or previous ("thank you Candy Crowley"). Tranter et al. [START_REF] Tranter | Who Really Spoke When? Finding Speaker Turns and Identities in Broadcast News Audio[END_REF] replace manual rules by a learning phase of n-grams sequences with associated probabilities. Mauclair et al. [START_REF] Mauclair | Speaker diarization : about whom the speaker is talking?[END_REF] use a semantic classification tree trained to associate a pronounced name to a speaker. Estève et al. [START_REF] Estève | Extracting true speaker identities from transcriptions[END_REF] compare these two techniques. They conclude that the semantic classification trees are less sensitive than the sequences of n-grams when using automatic speech transcriptions. Jousse et al. [START_REF] Jousse | Automatic named identification of speakers using diarization and ASR systems[END_REF] improve the using of the semantic classification trees with a local decision (affiliate a name to a nearby speech turn) and a global decision (propagation names into speaker clusters). They also show a performance degradation between 19.5% and 70% relative (speakers identification error rate) when using automatic speech transcriptions instead of manual transcriptions. More recently, in [START_REF] Poignant | Unsupervised Speaker Identification using Overlaid Texts in TV Broadcast[END_REF] we proposed three propagation methods to map written names to speakers clusters. These unsupervised methods, inherently multi-modal, get much better performance than a mono-modal supervised solution. We have shown that automatic mapping of written names to speakers clusters lead to an accuracy of 98.9% when the diarization is considered as perfect.

The use of automatically extracted pronounced names (PN) faces several challenges: (i) transcription errors. (ii) errors in the person names detection: missing name parts, false detection or adding/removing words (name = "Here John Chan"). (iii) Mapping (affiliation) errors: to which speaker associate a name? The current speaker, the next one, the previous one ?

The use of automatically extracted written names (WN) faces the same difficulties: (a) transcription errors: the increase of the video quality reduces these errors. (b) errors in the person names detection: each show uses a template with specific locations to write texts. The difficulty lies in the detection of spatial positions of title blocks. (c) Mapping (affiliation) errors: usually a name is written on the screen while person is talking.

REPERE Corpus

Our comparison of these two modalities for unsupervised speaker detection will be based on the REPERE corpus [START_REF] Giraudel | The REPERE Corpus : a Multimodal Corpus for Person Recognition[END_REF]. This corpus is composed of 7 different shows (news, talks, debates) recorded on two French TV channels. The quality of these recordings (720*576, mpeg2) allows us to use the texts written on the screen. For the experiments, we use the training part (phase 1) of this corpus (58 hours of raw video, 24 hours annotated). For the REPERE challenge these videos were partially annotated on UEM segments2 . On these segments, the speech transcription was completely manually annotated. 555 speakers were named while 255 others were not. These unknown speakers correspond to 25 minutes of speech time among the 1440 minutes annotated. In this paper, we are interested only in speakers who have been identified during the annotation. The written texts are partially annotated (in average one image every ten seconds, i.e. so manual annotation can miss names). These texts are cut out and transcribed, the names of people have been labeled. Raw videos are longer than the annotated segments: advertisings and extra programs may be contained in the raw video. This additional (un-annotated) signal can help to extract more names, to have more occurrences of a name, or to find rarely pronounced or rarely written names (which is actually the case for anchors).

Automatic names extraction 4.1. Written names (WN)

To detect the names written on the screen used to introduce a person, a detection and transcription system is needed. For this task we used LOOV [START_REF] Poignant | From Text Detection in Videos to Person Identification[END_REF] (LIG Overlaid OCR in Video). This system has been previously evaluated on another broadcast news corpus with low-resolution videos. We obtained a character error rate (CER) of 4.6% for any type of text and 2.6% for names written on the screen to introduce a person. From the transcriptions, we use a simple technique for detecting the spatial positions of title blocks. This technique compares each transcript with a list of famous names (list extracted from Wikipedia, 175k names). Whenever a transcription corresponds to a famous name, we add its spatial position in a list. With the repeating positions in this list we find the spatial positions of title blocks used to introduce a person. However, these text boxes detected do not always contain a name. A simple filtering based on some linguistic rules allows us to filter false positives. Transcription errors are corrected using our Wikipedia list when the edit distance is small (207 corrections with 4 errors).

Pronounced names (PN)

A state-of-the-art off-the-shelf Speech-To-Text system for French [START_REF] Lamel | Speech Recognition for Machine Translation in Quaero[END_REF] was used to transcribe the audio data without specific model adaptation on our corpus. The recognizer uses the same basic statistical modeling techniques and decoding strategy as in the LIMSI English BN system [START_REF] Gauvain | The LIMSI Broadcast News Transcription System[END_REF]. Prior to transcription, segmentation and clustering [START_REF] Adda | Partitioning and Transcription of Broadcast News Data[END_REF] are performed. Word decoding is carried out in a 1xRT single decoding pass. Each decoding pass produces a word lattice with cross-word, wordposition dependent acoustic models, followed by consensus decoding with a 4-gram language model and pronunciation probabilities (35-phone set, 65k word vocabulary). This system obtained a word error rate of 16.87% (on around 36k words) during the first evaluation campaign of the REPERE challenge. For named-entity detection, we trained specific independent CRF models on the Quaero data. These models used the same features as those presented in [START_REF] Dinarelli | Models Cascade for Tree-Structured Named Entity Detection[END_REF]: (1) Standard features like word prefixes and suffixes. (2) Morpho-syntactic features extracted as in [START_REF] Allauzen | Training and evaluation of pos taggers on the french multitag corpus[END_REF]. (3) Features extracted from a multilevel analyzer used in the LIMSI question-answering systems [START_REF] Bernard | LIMSI participation in the QAst 2009 track[END_REF].

Comparison of WN and PN systems quality

The use of LOOV pipelined with our written names detection technique allows us to obtain 97.7% of names (see Table 1), with a precision of 95.7%. The few remaining errors are due to transcription or filtering errors. Extracting pronounced names generates more errors. The main difficulty lies in the transcription and the detection of unknown names (we do not have any a priori knowledge of names that could be pronounced). Despite the lower precision and recall of the PN relative to WN, they provide more hypothesis names (see Table 2). We can observe that there are about twice more pronounced names compared to written names, whether we analyze raw videos or UEM only. This proportion is valid for the number of names occurrences or the number of different persons. 5. Naming Speaker using Pronounced Names (PN) and/or Written Names (WN)

Modalities Precision

Modalities

Metrics used

The number of nameable speakers has been evaluated for each video (mono-video):

N pmono = #videos where p∈P hr #videos where p∈P r

With : p : a person P r : a set of persons p speaking P hr : P r with their names writ./pron. We also evaluated in a cross-video propagation mode :

N pcross = 1 If p ∈ P hr 0 else In other words, the N pcross of person p is equal to 1 if at least in one video the name of p is written/pronounced when the corresponding person speaks in this video, 0 otherwise.

Overall, for all persons, mono-and cross-video scores are:

Nmono = p∈P r N pmono #p ∈ P r Ncross = p∈P r N pcross #p ∈ P r
If we look at this example with three videos (VA, VB, VC ) and five speakers (S1 to S5). The name of these speakers can be written or pronounced in each video (N1 to N5) :

VA VB VC Speaker : S1, S2, S3 S1, S3, S4 S1, S5 N ames : N1, N2 N3, N5 N5, N4
We obtain these scores for each speaker and for all the speakers:

S1 S2 S3 S4 S5 Global N pmono 1/3 1/1 1/2 0/1 1/1 Nmono = 0.57 N pcross 1 1 1 0 1 Ncross = 0.8
S1 speaks in the three videos but can be named only in video V1. Therefore the corresponding N pmono is equal to 1/3, and N pcross is equal to 1. The name of S4 has never been pronounced in the video where he speaks, so this speaker is considered as not nameable (N pmono=N pcross=0).

In addition we also count the occurrence number: Occ : occurrences number of names written/pronounced Occpv : #Occ when the corresponding person speaks in the UEM segments A larger number of occurrences may help the name-person mapping. However, since the manual annotation of the written text is not complete (only one image every ten seconds is annotated in average), Occpv is probably under-evaluated but can be used at least to compare the potential of WN versus PN.

In the following tables, we use the following notations: Muem : manual annotations on UEM segments Auem : automatic annotations on UEM segments Araw : automatic annotations on raw videos W N : written names P N : pronounced names

Unsupervised Speaker Naming

In Table 3, we observe that the written names extracted automatically can name 73.5% of the 555 speakers. The manual annotation of W N is not complete (1 image / 10 sec only), which explains the higher score of the automatic system (73.5%) compared to manual annotations (60.5%). The combined use of the two modalities (WN+PN) enhances the score (+19.9 % in the case of manual annotations -MUEM but fewer when automated systems are used (+2.3 % for Auem)). A cross-video propagation increases the Ncross approximatively by 4% on average.

The use of the raw videos (Araw) increases the occurrences number of speakers name (Occpv from Auem = 2262 to Araw = 2781) without significantly increasing the number of speakers nameable in the UEM segments (Auem = 75.8 % to Araw = 76.9 %). But, the additional occurrences of names may facilitate the name-person mapping.

Finally, it is important to mention that the percentages of Occpv for Araw are underevaluated, ground truth annotation involving only UEM segments. So we can not say if names do match to a person speaking outside the UEM segments . We can see that the name of the 84 anchors/reporters are relatively rarely pronounced (Occpv to Muem = 414, Auem = 58) or written (Occpv to Muem = 91, Auem = 174). Indeed, anchors/journalists are often cited by their first names. In addition, their names are difficult to transcribe because they may be unknown to the automated systems (we do not use a priori knowledge of the anchor/reporters names). People in R1 are quite difficult to automatically name while they represent 45% of the speech time. Concerning the percentage of nameable speakers, 79.4% of people in R1 have their names pronounced but only 13.9 % can be retrieved automatically. People in R2 seem to be more automatically nameable than those of R1 (W N +41%, P N +14.8%). The combined use of both modalities (PN+WN) can increase the nameable speakers and occurrences number whatever the type of role taken into account or the propagation mode (mono-or cross-videos).

P N W N Occ Occpv

Name-to-Speaker Mapping: Oracle Experiments

Until now, we collected percentage of nameable speakers and number of occurrences figures which gave us an idea of the potential of both WN and PN modalities for unsupervised naming of persons in video. But the name-to-speaker mapping step was not considered yet. State of the art systems which do that are currently restricted to adjacent speech turns to affiliate a name to a speaker. In this section, we compare the mapping (affiliation) ability of names written or pronounced to the correct speaker with the help of an oracle. For written names (WN), the oracle considers that the name-person mapping is correct if he speaks during the display of the name. For pronounced names (PN), the oracle considers that the mapping is correct if the right person speaks during the current, previous or the next speech turn. The results in table 6 are to be compared with those in table 3. When the affiliation is restricted to adjacent speech turns, the performance reduces (in absolute) from 4.9 % to 8.8 % depending on the system used and on the propagation considered. The reduction is, however, less important in the case of written names (reduction from 0.3 % to 0.8 %). Despite this, the table shows the complementarity of both PN and WN modalities.

Conclusion

The pronounced names (PN) and written names (WN) on the screen are an important source of information to name people in broadcast TV. Despite a larger number of PN in the manual speech transcription in our video corpus, speech transcription and named-entities errors reduce the potential of this modality for naming speakers. On the contrary, with our WN detection and transcription system, we were able to obtain twice nameable speakers as the one obtained using PN. Also, it is worth mentioning that the mapping (affiliation) of WN to the right speakers is inherently simpler than for PN. Despite these differences, both methods were shown to be complementary and unsupervised multi-modal name propagation should be developed in the future to improve speaker indexing of TV shows.

Figure 1 :

 1 Figure 1: Pronounced names and written names in a TV broadcast video

Table 1 :

 1

			Recall F1-measure
	WN	95,7%	97,7%	96,7%
	PN	73,5%	50%	59,5%

Quality of names extraction, WN: written names, PN: pronounced names, UEM segments only (24 hours of video)

Table 2 :

 2 Number of written (WN) and pronounced names (PN)

		Segment	#Occurrences #Persons w/o of names duplicates
	WN	UEM (24h) Raw (58h)	1407 2090	458 629
	PN	UEM (24h) Raw (58h)	2905 4922	736 1156

Table 3 :

 3 Mono-and Cross-video scores, and number of occurrences of named speakers, for PN and WN modalities -555 manually annotated spkrs -24h (UEM) or 58h (Raw) of video5.3. Detail per speaker's roleIn the REPERE corpus, five different speaker categories have been defined to classify people (anchor, columnist, reporter, guest, other). In view of the detailed results, we merged categories with similar behavior for a better readability. The first three were grouped into the role R1: anchor/journalist, the last two in the role R2: guest/other. Table4shows the speaker distribution according to their roles. A role has been assigned to each person identified in the videos, a person may have different roles depending on the show. Speakers of R1 cover 45% of speech time while they represent only 15% of the speakers.

					Nmono Ncross
	Muem	-	4273 1863 (43,6%)	62,2	66,5
	-	Muem 1049 1022 (97,4%)	60,5	65,9
	Muem Muem 5322 2885 (54,2%)	80,4	83,6
	Auem	-	2905	914 (31,5%)	26,7	30,8
	-	Auem 1407 1348 (95,8%)	73,5	76,8
	Auem	Auem 4312 2262 (52,5%)	75,8	78,7
	Araw	-	4922 1104 (22.4%)	27.9	32.3
	-	Araw	2090 1677 (80.2%)	74.8	77.5
	Araw	Araw	7012 2781 (39.7%)	76.9	79.3

Table 4 :

 4 Speakers distribution according to their roles. R1: anchor/journalist, R2: guest/other.

Table 5

 5

	details the speakers' nameability depending on the role:
	P N	W N	Occpv R1 R2	Nmono R1 R2	Ncross R1 R2
	Muem	-	414 1449 79.4 59.0 86.9 62.7
	-	Muem	91	931	23.5 66.6 35.7 70.7
	Muem Muem 505 2380 81.5 80.0 89.3 82.3
	Auem	-	58	856	13.9 28.7 16.7 33.1
	-	Auem 174 1174 38.3 79.3 47.6 81.5
	Auem	Auem 232 2030 43.3 81.1 52.4 82.9
	Table 5: Mono-and Cross-video scores, as well as number of
	Occurrences of named speakers, for both PN and WN according
	to roles (R1: 84 anchor/journalist, R2: 475 guest/other)

Table 6 :

 6 Oracle Name-to-Speaker Mapping Performance for the 555 speakers (UEM segments only -24h video).

	P N	W N	Oracle(% spk correctly named) mono cross
	Muem	-	53.4	58.9
	-	Muem	60.2	65.6
	Muem Muem	76.9	80.5
	Auem	-	21.3	25.9
	-	Auem	72.9	76.0
	Auem	Auem	75.2	78.4

UEM: Evaluation unpartitioned Map


Johann Poignant

Hervé Bredin

Laurent Besacier

Georges Quénot

Claude Barras

Towards a better integration of written names for unsupervised speakers identification in videos

Keywords: speaker identification, speaker diarization, written names, multimodal fusion, TV broadcast

Existing methods for unsupervised identification of speakers in TV broadcast usually rely on the output of a speaker diarization module and try to name each cluster using names provided by another source of information: we call it "late naming". Hence, written names extracted from title blocks tend to lead to high precision identification, although they cannot correct errors made during the clustering step.

In this paper, we extend our previous "late naming" approach in two ways: "integrated naming" and "early naming". While "late naming" relies on a speaker diarization module optimized for speaker diarization, "integrated naming" jointly optimize speaker diarization and name propagation in terms of identification errors. "Early naming" modifies the speaker diarization module by adding constraints preventing two clusters with different written names to be merged together.

While "integrated naming" yields similar identification performance as "late naming" (with better precision), "early naming" improves over this baseline both in terms of identification error rate and stability of the clustering stopping criterion.

Introduction

Knowing "who said what" in broadcast TV programs is very useful to provide efficient information access to large video collections. Therefore, the identification of speakers is important for the search and browsing in this type of data. Conventional approaches are supervised with the use of voice biometric models. However, the use of biometric models faces two main problems: 1) manual annotations: generating biometric models is very costly because of the great number of recognizable persons in video collections; 2) lack of prior knowledge on persons appearing in videos (except for journalists and anchors): a very large amount of a priori trained speaker models (several hundreds or more) is needed for covering only a decent percentage of speakers in a show.

A solution to these problems is to use other information sources for naming speakers in a video. This is called unsupervised naming of speakers and most approaches for that can be decomposed into the three steps:

1. Speaker clustering (or diarization), 2. Extraction of hypothesis names from the video (or from the collection of videos),

This work was partly realized as part of the Quaero Program and the QCompere project, respectively funded by OSEO (French State agency for innovation) and ANR (French national research agency).

Mapping (or association) between hypothesis names and speaker clusters.

Speaker diarization is the process of partitioning the audio stream into homogeneous clusters without prior knowledge on the speakers' voice. Each cluster must correspond to only one speaker and vice versa. Most systems use a bottom-up approach which tries to merge speech turns into clusters that are the purest as possible using a distance metric (with a distance-based criterion to stop the clustering).

Two modalities, intrinsic to the video, can provide the name of speakers in broadcast TV: pronounced names and names written on the screen (see figure 1). Most state-of-the-art ap- They also show a performance drop from 19.5% to 85% in speaker identification error rate when using automatic speech transcription instead of (perfect) manual transcriptions and named entities detection. More recently, we proposed three propagation methods to propagate written names to speaker clusters [START_REF] Poignant | Unsupervised Speaker Identification using Overlaid Texts in TV Broadcast[END_REF]. These unsupervised multi-modal methods yield much better performance than mono-modal ones. We also show that these methods lead to 98.9% accuracy with perfect speaker diarization. The use of automatically extracted pronounced names faces several challenges: (i) transcription errors; (ii) named entity detection errors (missing first/last name, false alarms, etc.); (iii) mapping errors (current, previous or next speech turn).

The use of automatically extracted written names faces similar difficulties: (a) transcription errors -though better video quality reduces these errors; (b) detection errors -fewer because each TV show uses specific spatial position for title blocks 1 ; (c) mapping errors -though a name is usually written on the screen while the person is talking, yielding easier affiliation.

This paper addresses other errors that can impact results: the errors made during the clustering process (during speaker diarization). For instance, the incorrect merging of two clusters containing different speakers can severely impact the speaker naming performance. Tuning the stopping criterion for hierarchical clustering is important to avoid such a problem. In this paper, we rely on the hypothesis that the high precision of written names to identify the current speaker can help us improve the diarization process in order to avoid the problems mentioned earlier. We limit our study to the use of written names for unsupervised speaker identification in videos and propose an extension of [START_REF] Poignant | Unsupervised Speaker Identification using Overlaid Texts in TV Broadcast[END_REF]. In this previous work, we proposed three methods for "late naming" of speakers which are highly dependent on the quality of speaker diarization. In this article, we present two novel approaches to overcome this issue: "integrated naming" aims at better choosing the value of the stopping criterion in order to minimize the speaker identification error while "early naming" adds written names-driven constraints to speaker diarization.

The outline of the paper is as follows. Section 2 presents the experimental setup as well as the speaker diarization module and the written names extraction module used in our experiments. Then, we describe our speaker naming methods in Section 3. Section 4 presents our experiments. Finally, we conclude this work and give some perspectives.

Experimental setup

The REPERE [START_REF] Kahn | A presentation of the REPERE challenge[END_REF] evaluation campaign phase 1 took place in January 2013. The main objective of this challenge is to answer the two following questions at any instant of the video: "who is speaking?" "who is seen?". In this paper, we try to answer the first question in an unsupervised way.

REPERE Corpus

The dataset used in our experiments is extracted from a corpus created for the REPERE challenge [START_REF] Giraudel | The REPERE Corpus : a Multimodal Corpus for Person Recognition[END_REF], which addresses multimodal person identification in videos. Videos are recorded from seven different shows (including news and talk shows) broadcasted on two French TV channels. An overview of the data is presented in Table 1 Though raw videos were provided to the participants (including the whole show, adverts and part of surrounding shows), only excerpts of the target shows were manually annotated for the evaluation. 1 Title block: spatial position used in the TV show to write a name and introduce the corresponding person.

Our evaluation is performed on test set. It is important to note that, although the whole test set is processed, the performance is measured only on the annotated frames. Figure 2 shows some statistics of the test set (duration and number of videos) for each TV show available in the REPERE corpus.

Evaluation Metrics

Alongside the usual precision P and recall R, the official REPERE metric is also used for evaluation. It is called the Estimated Global Error Rate (EGER). This metric is defined as: EGER = #fa + #miss + #conf #total where #total is the number of person utterances to be detected, #conf the number of utterances wrongly identified, #miss the number of missed utterances and #fa the number of false alarms.

To evaluate speaker diarization performance, we also used the diarization error rate (DER) defined by:

DER = d f a + dmiss + d conf d total
where d total is the total speech time, d f a the duration of false alarm, dmiss the duration of missed speech and d conf the duration of the speech time where hypothesis and reference disagree. As identities of speakers are not considered, hypothesis and reference are aligned 1-to-1 to minimized d conf .

Audio and Video Processing Modules

Speaker Diarization

Speaker diarization consists in segmenting the audio stream into speaker turns and tagging each turn with a label specific of the speaker. Given that no a priori knowledge of the speaker's voice is available in the unsupervised condition, only anonymous speaker labels can be provided at this stage.

After splitting the signal into acoustically homogeneous segments, we calculate a similarity score matrix between each pair of segments using the BIC criterion [START_REF] Chen | Environment and Channel Change Detection and Clustering via the Bayesian Information Criterion[END_REF] with single fullcovariance Gaussians. This similarity matrix is then given as input of a complete-link agglomerative clustering. Depending on the similarity threshold used as stopping criterion, several clustering results can be obtained.

It is worth mentioning that the matrix is not updated after each merging of clusters, as this is usually the case for regular BIC clustering.

We are aware that hierarchical clustering based on BIC distance is less efficient than hierarchical clustering with CLR distance [START_REF] Barras | Multi-Stage Speaker Diarization of Broadcast News[END_REF] but our goal, here, is to do a fair comparison of several speaker naming methods, independently of the similarity measure (BIC or CLR).

Written names extraction

To detect the names written on the screen used to introduce a person, a detection and transcription system is needed. For this task we used LOOV [START_REF] Poignant | From Text Detection in Videos to Person Identification[END_REF] (LIG Overlaid OCR in Video). This system has been previously evaluated on another broadcast news corpus with low-resolution videos. We obtained a character error rate (CER) of 4.6% for any type of text and of 2.6% for names written on the screen to introduce a person. From the transcriptions, we use a simple technique in order to detect the spatial positions of title blocks. This technique compares each transcript with a list of famous names (list extracted from Wikipedia, 175k names). Whenever a transcription corresponds to a famous name, we add its spatial position to a list. With the repeating positions in this list we find the spatial positions of title blocks used to introduce a person. However, these text boxes detected do not always contain a name. A simple filtering based on some linguistic rules allows us to filter false positives.

Unsupervised Naming of Speakers

We propose three methods for unsupervised (i.e. with no prior biometric models) naming of speakers with written names.

Late naming (LN)

Late naming is based on our previous work [START_REF] Poignant | Unsupervised Speaker Identification using Overlaid Texts in TV Broadcast[END_REF] (method M3). Speaker diarization and overlaid names recognition are run independently from each other. Speaker diarization is tuned to achieve the best diarization performance (i.e. minimize the diarization error rate, DER) as shown in Figure 3. The mapping between written names and speaker clusters is based on the following observations:

• when only one name is written on screen, any cooccurring speech turn is very likely (95% precision according to the train set) to be uttered by this person; • the speaker diarization system can produce oversegmented speaker clusters, i.e. split speech turns from one speaker into two or more clusters.

Therefore, this method proceeds in two steps. First, speech turns with exactly one co-occurring name are tagged. Then, each remaining unnamed speech turn is tagged cluster-wise using the following criteria:

f (s) = argmax n∈N TF (s, n) • IDF (n)
where the Term-Frequency Inverse Document Frequency (TF-IDF) [START_REF] Robertson | Relevance weighting of search terms[END_REF][START_REF] Fang | A formal study of information retrieval heuristics[END_REF] coefficient -made popular by the information retrieval research community -is adapted to our problem as follows: TF (s, n) = duration of name n in cluster s total duration of all names in cluster s

IDF (n) = # speaker clusters # speaker clusters co-occurring with n

In other words, speaker clusters are analogous to textual documents, whose words are detected written names.

Late naming is based on this method but there is a slight update that needs to be mentioned: we reduce the temporal scope of each written name to the more co-occurring speech turn, this can correct the time offset between audio and written names segmentation. It is important to note that the diarization can be different before and after the name-clusters association: some clusters may be merged (same name) or split (speech turn with a different name). Therefore, the scoring of the diarization can marginally change.

Integrated naming (IN)

One limitation of the late naming method is that the threshold used to stop hierarchical clustering is optimized in terms of diarization error rate (DER), while the ultimate objective is speaker identification, not diarization. Obviously, optimizing DER does not necessarily lead to the lower identification error rate (EGER). Therefore, "integrated naming" is a simple extension of "late naming" where the stopping criterion threshold is tuned in order to minimize the EGER. We will show later in the experiments that the resulting threshold is generally higher than the one selected to minimize DER (i.e. agglomerative clustering is stopped earlier) Figure 4: Integrated naming In practice, as shown in Figure 4, we keep multiple clustering outputs, on which we apply the same method as in the "late naming" strategy described before. The threshold optimizing EGER on the training set is chosen.

Early naming (EN)

As already stated, when one or more names are written on the screen, there is a very high probability that the name of the current speaker corresponds to the written name on screen. Therefore, in "early naming", we use the information provided by written names during the clustering process.

Before clustering, we associate each written name n to the more co-occurring speech turns. At this stage, a speech turn can have several names if several names are written on the screen at the same time. Then, regular agglomerative clustering (based on speech turn similarity) is performed with the constraint that merging two clusters s without at least one name n in common is forbidden. For example, two clusters s1 and s2 can be merged into a new one snew in the following case (the list of associated names is shown between brackets):

• s1(∅) ∪ s2(∅) ⇒ snew(∅) • s1(n1) ∪ s2(∅) ⇒ snew(n1) • s1(n1, n2) ∪ s2(∅) ⇒ snew(n1, n2) • s1(n1, n2) ∪ s2(n1) ⇒ snew(n1)
Below are examples where the two clusters cannot be merged:

• s1(n1) ∪ s2(n2) ⇒ Forbidden • s1(n1, n3) ∪ s2(n2) ⇒ Forbidden
The clustering is stopped according to the optimal (minimizing EGER) threshold learned on the training set.

Results

In this section we compare the ability of our naming methods to correctly identify speakers in TV broadcast and more particularly their sensitivity to the value of the stopping criterion threshold.

Learning the threshold as stop criterion

We used the training set to learn the stopping criterion threshold. However, in order to be less dependent on manual annotations, we did not use the whole 24 hours training set and selected 100 subsets randomly from it. These subsets were chosen to match the test set (duration, balance between shows, and number of videos for each show). As expected, Table 2 shows that the optimal threshold for IN is higher than those for LN. It means that IN stops earlier in the agglomerative clustering though split clusters may end up with the same name.

The constrained clustering of EN stops at a lower threshold. The standard deviation for EN is very high compared to the two others methods, it is possible to interpreted that EN is less sensitive to the threshold value. For the rest of the paper, we chose to use the median as global threshold.

Speaker Identification

For all the following experiences, it is important to note that the stopping criterion thresholds are learned on the training set while the results are displayed for the test set. Figure 6 shows the evolution of EGER with respect to the selected threshold and should be read from right to left as a smaller threshold value means that the agglomerative clustering stops later. LN and IN curves overlap but differ in the optimal stopping criterion threshold: threshold a aims at minimizing the DER (late naming) while b focuses on minimizing EGER (integrated naming). EN behaves very differently. 1 shows the impact of the written name constraints and c the threshold learned to minimize the EGER.

Table 3 summarizes the performance of the three methods. The integrated naming has a lower EGER but the difference is very small, yet this method has better precision due to its higher threshold. As far as EN is concerned, the clustering constraint helps keeping the same precision (80.4%) though the threshold is lower. It allows to correctly merge some additional clusters and therefore increases the recall to 68.3%. For IN and EN, minimizing the EGER still allows to maximize other metrics like precision, providing at least enough speech duration to build speakers models.

Naming strategy

Thr

Speaker Diarization

Figure 7 shows the evolution of DER as a function of the threshold. The baseline "before naming" corresponds to an audioonly diarization. As explained in section 3.2 the diarization is different before and after the late naming. speech turns with the same name. At the end of the clustering 3 , this step removes from clusters some speech turns with a different name. 1 shows the effect of the constraints preventing clusters with different names from being merged. a corresponds to the threshold tuned to minimize the DER. We obtain an 18.11% DER on the test set without written names (see Table 4). "Integrated naming" has a higher threshold but some clusters end up merged (thanks to their identical associated names), leading to a lower DER of 17.5%. The constrained clustering shows only a small variation of DER (from 18.7% to 20.2%, with a minimum of 16.37%) over the [0-1800] threshold range: it appears to be much less sensitive to the threshold choice (see figure 7). The x-axis of Figure 8 summarizes the range of variation of this optimal threshold over the 100 training subsets (e.g. 1440 to 1680 for late naming strategy), as already introduced in Table 2. The y-axis reports the corresponding average identification error rate (EGER) and its standard deviation on the test set. This figure points out that late and integrated naming strategies are more dependent on the training set and may therefore suffer from over fitting. Their respective identification error rates (EGER) has a standard deviation of 1.2% and 0.8%, while standard deviation of early naming EGER is only 0.2% (though the range of optimal thresholds over the 100 training subsets is much bigger).

Show-dependent threshold

The test corpus is composed of seven different types of shows (as illustrated in Figure 2). While a global show-independent threshold (Thr. corpus) can be trained, we also investigate the use of a show-dependent threshold (Thr. per show) and report the outcome of this experiment in Figure 9. Thr. oracle corresponds to the best possible performance in case an oracle is able to predict the best threshold. The robustness of a particular naming strategy can be inferred by the difference between the thresholds tuned on the whole training set (Thr. corpus and Thr. per show) and the optimal threshold (Thr oracle). Figure 9 shows that there is a difference of behavior between DER minimization (late naming) or EGER minimization (integrated or early naming). On one hand, DER minimization aims at associating one specific cluster to each speaker, whether they can be named or not. On the other hand, EGER minimization tries to associate its name to every speaker. Anonymous speakers can remain in the same cluster or split into several clusters as it has no influence on the final value of the identification error rate (EGER).

The REPERE corpus is composed of various types of shows. Some contains numerous speakers (up to 18 for news show BFM Story) whose names are usually displayed only once. Others, like the debate Pile Et Face, only have three speakers (two guests and the anchor) whose names are displayed 24 times on average over the duration of each show. For this particular type of show, the optimal DER threshold is 1300 while the EGER one is 1560. As a matter of fact, since speaker names are written multiple times, it is not worth trying to get exactly one cluster per speaker. A speaker cluster can be split into multiple smaller clusters as long as those clusters are named correctly.

Finally, we highlight that oracle results show almost identical performance for the three strategies. However, since early naming is less sensitive to the chosen threshold, it leads to much better identification performance (very close to the oracle one).

Conclusions

In this paper, we introduced and analyzed two naming strategies for unsupervised speaker identification in TV broadcast. Integrated naming is a simple extension of our previous work [START_REF] Poignant | Unsupervised Speaker Identification using Overlaid Texts in TV Broadcast[END_REF] that improves precision (+1.1%) while keeping the same identification error rate (32.4%). Early naming relies on the knowledge of overlaid names during the clustering process. This information is used to constrain clustering by preventing two clusters named by different written names from being merged. This method leads to better identification error rate (29.9%) and is less sensitive to the choice of the stopping criterion threshold. These two methods allow maximizing the metric associated to the target task. Future works will focus on the integration of additional sources of information like pronounced names or face clustering.

Figure 1 :

 1 Figure 1: Exemple of written names on the screen proaches rely on pronounced names due to the poor quality of written names transcription observed in the past. Naming speakers with pronounced names has been proposed by Canseco et al. [1, 2] and Charhad et al. [3]. Manually-designed linguistic patterns indicate whether a name refers to the speaker of the current speech turn, the following or the previous one. Tranter et al. [4] learn these patterns as sequences of n-grams. Mauclair et al. [5] use semantic classification trees (SCT) to match names and speaker turns. Estève et al. [6] compare these two techniques. They conclude that SCTs are less sensitive to automatic speech transcriptions errors than sequences of ngrams. Jousse et al.[START_REF] Jousse | Automatic named identification of speakers using diarization and ASR systems[END_REF] improved over the SCT baseline: first, each name is attached locally to a nearby speech turn; names are then propagated globally to speaker clusters. They also show a performance drop from 19.5% to 85% in speaker identification error rate when using automatic speech transcription instead of (perfect) manual transcriptions and named entities detection. More recently, we proposed three propagation methods to propagate written names to speaker clusters[START_REF] Poignant | Unsupervised Speaker Identification using Overlaid Texts in TV Broadcast[END_REF]. These unsupervised multi-modal methods yield much better performance than mono-modal ones. We also show that these methods lead to 98.9% accuracy with perfect speaker diarization.The use of automatically extracted pronounced names faces several challenges: (i) transcription errors; (ii) named entity detection errors (missing first/last name, false alarms, etc.); (iii)

Figure 2 :

 2 Figure 2: Duration and number of videos for the various TV shows available in the REPERE collection

Figure 3 :

 3 Figure 3: Late naming

Figure 5 :

 5 Figure 5: Early naming

Figure 6 :

 6 Figure 6: Influence of the stopping criterion threshold ( a , b , c learned on train set) on identification error rate on test set, for the three naming strategies.

Figure 7 :

 7 Figure 7: Influence of the stopping criterion threshold on diarization error rate on test set, before and after naming.

Table 4 :

 4 DER depending on the threshold 4.4. Sensitivity to the training set Threshold tuning is achieved by randomly selecting 100 subsets from the training set and choosing the best threshold value for each of them.

Figure 8 :

 8 Figure 8: Average and standard deviation of the EGER on test set depending on the subsets used to learn the threshold

Figure 9 :

 9 Figure 9: Identification error rate (EGER) for a show-dependent or show-independent stopping criterion.

Table 1 :

 1 . Train and test sets statistics

		Train	Test
	Raw video	58h	15h
	Annotated part	24h	3h
	Number of annotated frames 8766 1229

Table 2 :

 2

	Naming strategy	median	min	max	standard deviation
	LN: lower DER	1540	1440 1680	54
	IN: lower EGER	1620	1520 1740	44
	EN: lower EGER	1260	300	1640	277

Threshold learned on 100 subsets of the train set, to minimize the DER or the EGER, LN: Late naming, IN: Integrate naming, EN: Early naming

Table 3 :

 3 Trained stopping criterion threshold learned on the train set and the corresponding identification error (EGER), precision (P) and recall (R) obtained on test set.

		.	EGER (%) P (%) R (%)
	Late (LN)	a 1540	32.1	80.4	66.0
	Integrated (IN)	b 1620	32.4	81.5	65.3
	Early (EN)	c 1260	29.9	80.4	68.3

and

show the influence of the direct speech turn tagging step. At the start of the clustering 2 , this step merges


Hervé Bredin
email: bredin@limsi.fr

Johann Poignant
email: johann.poignant@imag.fr

Keywords: speaker diarization, integer linear programming, speaker identification, multimodal fusion, optical character recognition

Most state-of-the-art approaches address speaker diarization as a hierarchical agglomerative clustering problem in the audio domain. In this paper, we propose to revisit one of them: speech turns clustering based on the Bayesian Information Criterion (a.k.a. BIC clustering). First, we show how to model it as an integer linear programming (ILP) problem. Its resolution leads to the same overall diarization error rate as standard BIC clustering but generates significantly purer speaker clusters. Then, we describe how this approach can easily be extended to the audiovisual domain and TV broadcast in particular. The straightforward integration of detected overlaid names (used to introduce guests or journalists, and obtained via video OCR) into a multimodal ILP problem yields significantly better speaker diarization results. Finally, we explain how this novel paradigm can incidentally be used for unsupervised speaker identification (i.e. not relying on any prior acoustic speaker models). Experiments on the REPERE TV broadcast corpus show that it achieves performance close to that of an oracle capable of identifying any speaker as long as their name appears on screen at least once in the video.

Integer Linear Programming for Speaker Diarization and Cross-Modal Identification in TV Broadcast

Introduction

Speaker diarization is the task of partitioning and labeling an audio stream into homogeneous speech segments according to the identity of the speaker. Most state-of-the-art approaches address it as a hierarchical (either agglomerative, divisive or a combination of both [START_REF] Evans | A Comparative Study of Bottom-Up and Top-Down Approaches to Speaker Diarization[END_REF]) clustering problem in the audio domain [START_REF] Tranter | An Overview of Automatic Speaker Diarization Systems[END_REF][START_REF] Reynolds | A Study of New Approaches to Speaker Diarization[END_REF][START_REF] Miro | Speaker Diarization: A Review of Recent Research[END_REF]. In this paper, we propose to revisit one of them: speech turns clustering based on the Bayesian Information Criterion (a.k.a. BIC clustering) [START_REF] Chen | Speaker, Environment and Channel Change Detection and Clustering via the Bayesian Information Criterion[END_REF].

One of the main limitations of this type of approach is that each clustering iteration is performed locally and does not guarantee global optimality. Two clusters are merged because they are close to each other, independently of how similar (or dissimilar) they are to other clusters. New approaches based on spectral clustering [START_REF] Ng | On Spectral Clustering: Analysis and an Algorithm[END_REF] were recently introduced to try and cope with this limitation [START_REF] Ning | A Spectral Clustering Approach to Speaker Diarization[END_REF][START_REF] Luque | On the use of Agglomerative and Spectral Clustering in Speaker Diarization of Meetings[END_REF][START_REF] Shum | On the Use of Spectral and Iterative Methods for Speaker Diarization[END_REF]. Indeed, spectral clustering techniques rely on the complete speech turns similarity matrix to project each of them into a low-dimensional manifold more suited for clustering. However, they still rely on a subsequent agglomerative clustering step with the same limitations as above.

Inspired by Dupuy et al. [START_REF] Dupuy | i-Vectors and ILP Clustering Adapted to Cross-Show Speaker Diarization[END_REF] who recently proposed a global optimization framework based on Integer Linear Programming (ILP), we investigate the use of ILP as a replacement for BIC clustering. Our approach differs from [START_REF] Dupuy | i-Vectors and ILP Clustering Adapted to Cross-Show Speaker Diarization[END_REF] both in the actual formulation of the ILP clustering problem (introduced in Section 2), and in the fact that we use it in place of BIC clustering (Section 3) while they still rely on BIC clustering as a preliminary step.

Section 4 describes how the monomodal (audio-only) ILP problem is extended to the audiovisual domain and TV broadcast in particular [START_REF] Noulas | Multimodal Speaker Diarization[END_REF]. Building on our previous related work [START_REF] Poignant | Unsupervised Speaker Identification using Overlaid Texts in TV Broadcast[END_REF], we integrate overlaid person names (automatically detected by video OCR) into the ILP problem in order to jointly improve speaker diarization performance and achieve unsupervised speaker identification (i.e. not relying on any prior acoustic speaker models).

We present the results of our experiments, both for speaker diarization and cross-modal speaker identification, in Section 5. Section 6 concludes the paper.

Clustering as an ILP Problem

Clustering has been addressed in numerous scientific fields in the past: from graph mining and community detection [START_REF] Newman | Modularity and Community Structure in Networks[END_REF] to natural language processing and co-reference resolution [START_REF] Finkel | Enforcing Transitivity in Coreference Resolution[END_REF]. Classical clustering algorithms include K-means and hierarchical (agglomerative or divisive) clustering [START_REF] Jain | Data Clustering: A Review[END_REF].

We describe the clustering of N items by a function δ:

δ : 1, N 2 → {0, 1} (i, j) → δij =   
1 if items i and j are in the same cluster (H hypothesis) 0 otherwise (H hypothesis) A few constraints ensure that δ codes for a valid clustering:

Reflexivity. Every element must belong to its own cluster:

∀i ∈ 1, N , δii = 1 (1)
Symmetry. If i is in the same cluster as j, then j should be in the same cluster as i:

∀ (i, j) ∈ 1, N 2 δij = δji (2)
Transitivity. If i is in the same cluster as j (δij = 1) and j is in the same cluster as k (δ jk = 1), then i must be in the same cluster as k (δ ik = 1):

∀ (i, j, k) ∈ 1, N 3 δij + δ jk -δ ik ≤ 1 (3)
Let pij = p (H | dij) be the posterior probability (conditionally to their similarity dij) that elements i and j are in the same cluster. Finkel & Manning [START_REF] Finkel | Enforcing Transitivity in Coreference Resolution[END_REF] propose to use Integer Linear Programming to find the optimal clustering function δ * that maximize the intra-cluster similarity while simultaneously minimizing the inter-cluster one:

δ * = argmax δ f (δ) (4) f (δ) = α • i,j δij • pij intra-cluster similarity + (1 -α) • i,j (1 -δij) • (1 -pij)
inter-cluster dissimilarity

The influence and choice of parameter α ∈ [0, 1] will be discussed later in Section 3.3. The details towards the solution of this ILP problem is out of the scope of this paper. For this work, we use the Gurobi solver [START_REF] Gurobi Optimization | Gurobi Optimizer Reference Manual[END_REF] implementation of the branchand-bound algorithm [START_REF] Land | An Automatic Method of Solving Discrete Programming Problems[END_REF].

Application to Speaker Diarization

Our first contribution is to revisit the standard BIC clustering approach [START_REF] Chen | Speaker, Environment and Channel Change Detection and Clustering via the Bayesian Information Criterion[END_REF] by replacing its agglomerative clustering step with the ILP formulation described in previous section.

Baseline BIC clustering

Our baseline BIC clustering approach is derived from the multistage speaker diarization system introduced in [START_REF] Barras | Multi-Stage Speaker Diarization of Broadcast News[END_REF].

Segmentation

Feature extraction yields one 38-dimensional vectors every 30ms, made of the concatenation of 12 LPC-based cepstral coefficients [START_REF] Hermansky | Perceptual Linear Predictive (PLP) Analysis of Speech[END_REF], their first-and second-order derivatives, and those of the log-energy. Speech activity detection is achieved based on Viterbi decoding with one 64-Gaussians Mixture Model (GMM) for speech, noisy speech, speech over music, pure music, and silence or noise. Speech segments are chopped into short pure segments (with a minimum duration of 2.5 seconds) using divergence-based segmentation. After training one GMM for each segment, Viterbi segmentation is used to refine segment boundaries.

Clustering

Agglomerative clustering is initialized with one cluster per segment, modeled with one Gaussian with full covariance matrix Σ trained on the D = 12-dimensional Mel Frequency Cepstral Coefficients (MFCC) and energy. The BIC criterion ∆BICij [START_REF] Chen | Speaker, Environment and Channel Change Detection and Clustering via the Bayesian Information Criterion[END_REF] defines the similarity between clusters i and j:

∆BICij = (ni + nj) log |Σ| -ni log |Σi| -nj log |Σj| - 1 2 • λ • D + 1 2 D (D + 1) log (ni + nj)
where n k is the number of samples in cluster k and λ the penalty weighting coefficient. Each iteration merges the two most similar clusters i and j until the stopping criterion ∆BIC < 0.

ILP BIC clustering

We propose to replace the clustering step of paragraph 3.1.2 by the ILP clustering introduced in Section 2. For fair comparison, we also use the BIC criterion as similarity measure (dij = ∆BICij). We apply Bayes' theorem to obtain the posterior probability pij = p (H | dij) used in Equation 4:

p (H | d) = 1 1 + p(d | H) p(d | H) p(H) p(H) (5)

Role of parameter α

Though detailed experimental results will be presented in Section 5, we here provide a general discussion about the behavior of the proposed ILP approach. In particular, Figure 2 illustrates the influence of parameter α on both cluster purity, cluster coverage [START_REF] Chen | Speaker, Environment and Channel Change Detection and Clustering via the Bayesian Information Criterion[END_REF] and resulting diarization error rate [START_REF] Fiscus | Results of the Fall 2004 STT and MDE Evaluation[END_REF]. It shows that smaller values of α tend to generate small pure clusters while larger ones will result in large covering clusters at the expense of purity. Rewriting Equation 4 in both extreme regions (α ≈ 0 and α ≈ 1) explains this expected behavior:

[α ≈ 0] δ * ≈ argmin δ ij δij • (1 -pij) = argmin δ ij δij [α ≈ 1] δ * ≈ argmax δ ij δij • pij = argmax δ ij δij
The optimal value for diarization error rate is around α = 0.6 though we notice a plateau for α ∈ [0.5, 0.7] common to both our development and test sets.

Cross-Modal Speaker Diarization

Most state-of-the-art speaker diarization approaches address this problem in the audio domain -the baseline and ILP BIC clustering are no exception. This is illustrated in the upper part of the graph in Figure 3 where each pair of speech turn vertices i and j is connected with an edge weighted by the posterior probability pij. However, in the case of TV broadcast such as TV news or talk-shows, other sources of information can be used to jointly improve speaker diarization and perform speaker identification. As a matter of fact, guests or reporters are often introduced to the viewer using overlaid title blocks containing their name, such as the ones shown in Figure 4. In this section, we propose to augment the speech turn graph with written name vertices (middle part of Figure 3) and identity vertices (bottom).

Written names

We use the video Optical Character Recognition (OCR) system proposed by Poignant et al. in [START_REF] Poignant | From Text Detection in Videos to Person Identification[END_REF] to automatically extract this information. Overlaid text boxes are first detected using a coarse-to-fine approach with temporal tracking. We then use the open-source Tesseract OCR system [START_REF] Smith | An Overview of the Tesseract OCR Engine[END_REF] to provide one transcription every ten frames. The transcriptions are finally merged to produce one transcription for each text box. As seen in Figure 4, not all detected text boxes are used to introduce a person: some provide other kind of information (name of the show, news flash, etc.). In order to only extract title blocks with person names, we use a list of names extracted from Wikipedia and the annotated training set introduced in Section 5 to automatically learn the spatial positions the most likely to be used for title blocks.

Cross-Modal Probability Graph

One vertex is added per written name occurrence. As shown by → traffic signs in Figure 3, each of them is constrained (δij = 1) to be in the same cluster as the corresponding identity vertex.

Moreover, any two identity vertices are prevented from being in the same cluster by an additional constraint ( \ sign, δij = 0).

To benefit from the influence of written names for speech turns clustering, cross-modal edges are added between cooccurring speech turns and written names. As a matter of fact, when a speech turn i and one (or more) written name(s) j occur simultaneously, the probability pij that the latter corresponds to the former is very high. This probability is learned using the annotated training set and depends on the number of cooccurring written names, as illustrated in Figure 4. For instance, in case two names are written while one person is speaking, the corresponding speech turn vertex is connected to both written name vertices with probability p = 0.996.

Since the resulting cross-modal probability graph is no longer complete, the objective function of Equation 4 is updated to take missing edges into account:

f (δ) = α • i,j ∃ i↔j δij • pij + (1 -α) • i,j ∃ i↔j (1 -δij) • (1 -pij)
where ∃ i ↔ j means that vertices i and j are connected.

Speaker Identification

The right part of Figure 3 describes the expected output of the resulting multimodal ILP clustering process: one cluster per person containing every corresponding vertex (speech turn, written name and identity when available). Therefore, the proposed framework can also be used for speaker identification.

After optimization, each speech turn is simply given the identity of the unique identity vertex belonging to the same cluster. Since the ILP problem is designed to prevent identity vertices from ending up in the same cluster (using \ constraints), there can be at most one identity vertex in every cluster. In Figure 3, speech turns #1 and #2 are given the identity ID1 and speech turns #4 and #5 the identity ID2. However, it may happen that a cluster does not contain any identity vertex: speech turns simply remain with unknown identity (as does speech turn #3).

Experiments

Corpora

Figure 5 provides a graphical overview of the REPERE video corpus used in our experiments [START_REF] Giraudel | The REPERE Corpus: a Multimodal Corpus for Person Recognition[END_REF]. It contains 188 videos (30 hours) recorded from 7 different shows broadcast by the French TV channels BFM TV and LCP. The audio stream is fully annotated with labeled speech turns ("who speaks when?"). [START_REF] Giraudel | The REPERE Corpus: a Multimodal Corpus for Person Recognition[END_REF] provides a comprehensive description of the corpus and the associated annotation process.

Experimental Protocol

The whole corpus is divided into three corpora. The training set is used to estimate the log-likelihood and prior probability ratios introduced in Equation 5. While speaker audio annotations are complete, visual annotations are only available for one frame every 10 seconds. Therefore, we ran our written name extraction system on the training set and used its output to estimate the cross-modal probabilities of Figure 4. The development set is used to select the optimal value for parameters λ (for standard BIC clustering) and α (for ILP BIC clustering). The test set is used for evaluation. Depending on the evaluated task (speaker diarization or speaker identification), parameters are optimized with respect to two evaluation metrics: diarization error rate (DER) for the former and identification error rate (IER) for the latter [START_REF] Fiscus | Results of the Fall 2004 STT and MDE Evaluation[END_REF].

Late, Intermediate or Early Fusion

For comparison purposes, we also evaluated two other multimodal speaker identification approaches (dubbed "Poignant et al. [START_REF] Poignant | Unsupervised Speaker Identification using Overlaid Texts in TV Broadcast[END_REF]" and "Constrained HAC" in Tables 1 and2). They differ from the proposed ILP approach in how early in the fusion pipeline the audio and visual modalities are combined.

The late fusion approach by Poignant et al. [START_REF] Poignant | Unsupervised Speaker Identification using Overlaid Texts in TV Broadcast[END_REF] consists in first applying standard BIC clustering and then propagating written names onto each speaker clusters and/or speech turns. The early fusion approach "Constrained HAC" relies on standard Hierarchical Agglomerative Clustering (HAC) with average linkage [START_REF] Jain | Data Clustering: A Review[END_REF] based on ∆BIC speech turn similarity defined in Section 3.1. Starting with one cluster per speech turn, each iteration merges the two closest clusters for which there is no conflict in their cooccurring written names. Our proposed audiovisual ILP approach falls into the intermediate fusion category.

We also define two oracles (OSD for speaker diarization and OSI for identification) that provide error rate lower bounds.

Speaker Diarization

Table 1 summarizes the performance in terms of speaker diarization. The oracle OSD relies on the same automatic preprocessing as the proposed approaches but differs in the clustering process which is perfect. The non-zero DER obtained by OSD in Table 1 is mostly due to the lack of an overlapping speech detection module in pre-processing. Indeed, the REPERE corpus contains TV news and talk-shows where guests tend to speak simultaneously. Overall, it leads to a high missed detection error rate (only one speaker detected instead of two).

Audio-only ILP clustering achieves performance as good as standard BIC clustering. Yet, it yields much purer clusters (92% vs. 94%). Therefore, we expect that integrating this ILP approach into a multi-stage (BIC+CLR/i-vector [START_REF] Dehak | Front-End Factor Analysis for Speaker Verification[END_REF]) one will lead to a sensible improvement in performance. Moreover, the integration of the visual modality leads to a significant improvement (-1% absolute DER) mostly due to an increase in cover- age. It was expected as two small clusters cooccurring with the same written name tend to be grouped together into a larger one.

Speaker Identification

Table 2 summarizes the performance in terms of speaker identification. The oracle OSI is able to correctly identify any speaker as long as its name appears at least once in the same video, according to our written name extraction system. It does not have to appear simultaneously with any of the speaker speech turns: such an oracle is therefore very optimistic. The high IER obtained by OSI in Table 2 is mostly due to the fact that anchors are very rarely introduced by overlaid names and therefore cannot be recognized in any way.

Looking at both IER and F-Measure, all three speaker identification approaches have very similar performance. The room for improvement to reach the oracle performance is also very limited. In the future, we will therefore try to lower the oracle performance bound by integrating supervised speaker identification approaches to better recognize anchors (for which training data can easily be collected). This can be achieved seamlessly by adding edges connecting speech turn and identity vertices, weighted by a probability derived from the identification score.

Though they have similar performance, the various approaches behave very differently from each other. Audio-visual ILP has much higher precision than constrained HAC for instance. One could try to combine them by first applying audiovisual ILP identification and then benefiting from the nearly perfect recall of the constrained HAC approach to identify the remaining unknown speakers.

Conclusion and Future Works

In this paper, we revisited BIC clustering as an Integer Linear Programming problem. We showed that standard and ILPbased BIC clusterings perform equally well with the exception that the latter yields much purer clusters. We therefore envision significant improvement over state-of-the-art approaches once it is integrated into a full-fledged multi-stage diarization system [START_REF] Barras | Multi-Stage Speaker Diarization of Broadcast News[END_REF].

Then, we showed how easily the proposed framework can be extended to the audiovisual domain and lead to significant performance improvement. Overlaid written names are not the only source of information available in TV broadcast. For instance, we plan to extend this approach to multimodal person recognition [START_REF] Tapaswi | Knock! Knock! Who is it?" Probabilistic Person Identification in TV-Series[END_REF] by adding face track vertices to the graph, or to named speaker identification [START_REF] Jousse | Automatic Named Identification of Speakers using Diarization and ASR Systems[END_REF] using addressee name vertices extracted from the automatic speech transcription with named entity detection.

Figure 1 :

 1 Figure 1: Estimation of posterior probability p (H | d) on the training set. Top: likelihood under hypothesis H (rightmost distribution, green) and H (leftmost distribution, red). Middle: estimated log-likelihood ratio (•) and linear regression. Bottom: estimated posterior probability.

Figure 2 :

 2 Figure 2: Influence of parameter α on the development set. The likelihood ratio p(d | H)/p(d | H) is estimated using the training set described in Section 5. As shown in Figure 1, linear regression by minimization of the sum of squared error is used to fit an affine function to the log-likelihood ratio. Finally, the prior probability ratio is estimated using the same training set. We obtain p(H) = 4.97×p(H): this leads to a shift to the right of the abscissa of p (H | d) = p H | d = 0.5 in Figure 1.

2 ?Figure 3 :Figure 4 :

 234 Figure 3: Cross-modal probability graph & expected clusters. One name: p = 0.956 Two names: p = 0.996

Figure 5 :

 5 Figure 5: Training, development and test sets each contain 7 different types of shows (A to G).

Table 1 :

 1 Speaker diarization.

	Approach	DER	Purity	Coverage F-measure
	Oracle OSD	7.4%	100.0%	96.5%	98.2%
	BIC clustering	19.8%	92.1%	86.8%	89.4%
	ILP (audio)	19.9%	94.1%	85.0%	89.3%
	ILP (audio-visual)	19.0%	94.2%	85.7%	89.7%
	Constrained HAC	21.6%	88.3%	88.8%	88.5%
	Approach	IER	Precision Recall F-Measure
	Oracle OSI	39.3%	100.0%	62.1%	76.6%
	Poignant et al. [12]	44.4%	79.8%	59.4%	68.1%
	ILP (audio-visual)	44.9%	90.6%	58.2%	70.9%
	Constrained HAC	42.2%	83.9%	61.2%	70.7%

Table 2 :

 2 Speaker identification.

Acknowledgments. This work was realized as part of the OSEO Quaero Program and the ANR QCompere project.