Paul Gay

Sylvain Meignier

Jean-Marc Odobez
email: odobez@idiap.ch

Paul Del Église

CRF based context modeling for person identification in broadcast videos

Keywords: Face identification, speaker identification, broadcast videos, conditional random field, face clustering, speaker diarization

published or not. The documents may come

CRF based context modeling for person identification in broadcast videos

INTRODUCTION

For the last two decades, researchers have been trying to create indexing and fast search and browsing tools capable of handling the growing amount of available video collections. Among the associated possibilities, person identification is an important one. Indeed, video contents can often be browsed through the appearances of their different actors. Moreover, the availability of each person intervention allows easier access to video structure elements such as the scene segmentation. Both motivations are especially verified in the case of news collections. The focus of this paper is therefore to develop a program able to identify persons in broadcast videos. That is, the program must be able to provide all temporal segments corresponding to each face and speaker.

Person identification can be supervised. A face and/or a speaker model of the queried person is then learned over manually labeled training data. However, this raises the problem of annotation cost. An unsupervised and complementary approach consists in using the naming information already present in the documents. Such resources include overlaid texts, speech transcripts and metadata. Motivated by this et al. (1999) to the development of more complex news browsing systems exploiting this paradigm [START_REF] Jou | Structured exploration of who, what, when, and where in heterogeneous multimedia news sources[END_REF]), or thanks to sponsored competitions [START_REF] Giraudel | The REPERE corpus: a multimodal corpus for person recognition[END_REF]). Whatever the source of naming information, it must tackle two main obstacles: associate the names to co-occurring speech and face segments, and propagate this naming information from the co-occurring segments to the other segments of this person.

There are several challenges related to this task. First, the named entities need to be recognized and an association step must decide if the name corresponds to people co-occurring in the document. Ambiguities arise when multiple audiovisual (AV) segments co-occur with one name. This is illustrated in Fig 1c where

there is more than one face in the image. This situation is becoming more common with modern video editing. Regarding the identity propagation, it can be done with speaker and face diarization techniques (detecting and clustering person interventions). However, these two tasks have been active fields of research for more than a decade and thus are difficult problems to solve. Indeed, a person may appear in different contexts thus introducing huge intraperson variabilities. We can distinguish them in function of the modalities and the different types of videos. For the speaker diarization, the main challenge in broadcast news is background noise such as music, or a noisy environment during outside reports. If we consider debates in studio where the speech is more spontaneous, the bottleneck becomes the overlapping speech and short speech segments. Regarding face diarization, report videos usually exhibit the largest variations as location and time may change between two scenes, and so will be the illumination conditions. For the debate and studio scenes, variations come essentially from changes in the facial poses.

In this paper, we assume that closed-captions are not available as this is the case in European medias.

Instead, we focus on Overlaid Person Names (OPNs) which are used to introduce the speakers as illustrated in Fig. 1a. Such names are appealing since their extraction is much more reliable than pronounced names obtained through Automatic Speech Recognition (ASR). Moreover, their association with face or speech segments is in general easier than analyzing whether pronounced names in ASR transcripts refer to people appearing in the video. The identification systems submitted at the recent REPERE campaign (Bredin et al. (2013); [START_REF] Bechet | Multimodal understanding for person recognition in video broadcasts[END_REF]; [START_REF] Poignant | Unsupervised speaker identification in TV broadcast based on written names[END_REF]) mainly rely on such names.

Our approach offers several advantages. Faces are identified by alternating between a clustering step of faces and audio speech segments and a naming step of the resulting AV clusters. Each step is performed by a dedicated CRF. The use of CRF enables us to include heterogeneous context cues in our modeling. The use of such cues is challenging because they must use as little specific prior information as possible in order to achieve generalization over the different types of videos. In this paper, we include different generic context cues. First, we have AV association scores which enable to associate overlapping speaker and face segments when they correspond to the same person. Then, we use uniqueness constraints between simultaneously appearing pairs of faces. Furthermore, one of the main contribution is a background recurrence descriptor which attributes a soft role to each segment. It enables to distinguish the persons which are announced by the OPNs such as guests or journalists from the anonymous persons appearing around them. Last but not least, the names contained in the OPNs are included to guide the clustering by using the probabilities obtained with the naming CRF. These different cues enable to improve the clustering by reducing errors due to monomodal intracluster variations such as facial pose or audio background noise. Eventually, the CRF formulation avoids hard local decisions by providing a joint probability distribution over all the segments.

The first CRF performs jointly the clustering of face tracks and speaker segments thanks to AV association as introduced in Gay et al. (2014c). In practice, AV association is initialized in a pre-processing step based on temporal co-occurrence and then refined inside the CRF thanks to talking head detection scores and the previously described contextual cues. The second CRF assigns a name to each cluster by using cooccurrence statistics and a uniqueness constraint preventing any two faces on the same image to receive the same name. In [START_REF] Gay | A conditional random field approach for face identification in broadcast news using overlaid text[END_REF], this approach was designed for face identification. In the present case, we extend this approach for the AV case and provide results for the final evaluation of the REPERE campaign. Identification performances are discussed by investigating the algorithm behavior in different types of shows (reports, news, debates, celebrity magazines) and the relations with the clustering quality.

The rest of the article is organized as follows: Section 2 reviews related work on unsupervised identification. Then, Section 3 presents the proposed CRF-based system. Experiments and results are presented in section 4. Finally, Section 5 sums up our main findings and concludes the paper.

RELATED WORK

As stated in the introduction, unsupervised people identification must address the problems of local person/name association and propagation to the video parts where the names are absent. The association is conducted via the use of co-occurrence statistics between the names present in the document and the detected persons. The propagation can be seen as a clustering problem. Clustering methods can regularly benefit from new improvements in speaker and face representations. At the time of writing, the ivector approach is one of the most successful [START_REF] Rouvier | An open-source state-of-the-art toolbox for broadcast news diarization[END_REF]) for the speaker diarization task. Regarding face representation, recent advances include encodings [START_REF] Simonyan | Fisher vector faces in the wild[END_REF]), metric learning [START_REF] Bhattarai | Some faces are more equal than others: Hierarchical organization for accurate and efficient large-scale identity-based face retrieval[END_REF]) and feature learning by deep Convolutionnal Neural Networks [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF]). However, most of the systems require explicit face alignment to obtain frontal views which is not always feasible.

The work published in Zhang et al. (2015) suggests that using only face representation is a great limitation when dealing with unconstrained views of persons. For this reason, we believe that investigation into context-assisted clustering is justified, especially for broadcast news videos which exhibit a strong structure.

To identify the faces, most approaches try to solve the association and the propagation problems jointly.

On one hand, co-occurrence statistics at cluster level are more discriminant and accurate than just describing a face locally with namedness features (like face position or talking activity) to assess whether the detected name should be associated. On the other hand, name/face co-occurrences are used as a contextual cue to improve the face clustering process. These principles have been used intensively since the seminal works of [START_REF] Berg | Names and faces in the news[END_REF] and [START_REF] Everingham | Hello! my name is... buffy-automatic naming of characters in tv video[END_REF] which applied to two representative use-cases: captioned images, as examplified by the Yahoo News! dataset, and soap series with the buffy dataset. The first case study consists of news articles with images illustrating the subject. The initial approach described in Berg et al. ( 2004) is an EM clustering where the update of the model parameters takes into account the name/face co-occurrences. In this context, the work of [START_REF] Ozkan | Interesting faces: A graph-based approach for finding people in news[END_REF] exploits the fact that a textual query enables to retrieve faces where the queried person holds the majority. The problem of finding those faces is posed as finding the densest component in a graph. This idea was later extented in [START_REF] Guillaumin | Multiple instance metric learning from automatically labeled bags of faces[END_REF] where the distance within clusters is minimized with respect to a cannot-link constraint which implies that two faces must belong to different clusters if their captions contain different names. However, those co-occurrence statistics can fail when group of people co-occur in a similar fashion, a situation commonly encountered in TV programs. In soap series, the names of the speakers can be obtained with the transcripts and the subtitles. Works in [START_REF] Cour | Learning from partial labels[END_REF]; [START_REF] Wohlhart | Multiple instance boosting for face recognition in videos[END_REF]; [START_REF] Bauml | Semi-supervised learning with constraints for person identification in multimedia data[END_REF] use those names as weak labels to improve supervised classifiers. They choose a learning setting which takes into account the label ambiguities, for example: multiple instance learning [START_REF] Wohlhart | Multiple instance boosting for face recognition in videos[END_REF]) and semi-supervised strategies [START_REF] Bauml | Semi-supervised learning with constraints for person identification in multimedia data[END_REF]). Talking head detection [START_REF] Everingham | Hello! my name is... buffy-automatic naming of characters in tv video[END_REF]; [START_REF] Cour | Learning from partial labels[END_REF]) and dialogue cues [START_REF] Cour | Talking pictures: Temporal grouping and dialogsupervised person recognition[END_REF]) are also used to solve the ambiguities in the face/name association. Note that in the previous two case studies, the naming co-occurrence statistics are quite different to those in broadcast videos where the OPNs are more sporadic. Indeed, the OPN of a given person only appears one or a few times (usually for the first time utterance). This scarsity increases the dependence of the identification performance on the clustering quality.

Originally, unsupervised speaker identification in broadcast news was conducted by first performing a speaker diarization (i.e. clustering) step of the audio track and then assigning the names extracted from the transcription to the speaker clusters by using semantic classification trees [START_REF] Jousse | Automatic named identification of speakers using diarization and ASR systems[END_REF]) or

Maximum-entropy classifiers [START_REF] Ma | Finding speaker identities with a conditional maximum entropy model[END_REF]). More recently, the idea of constrained speaker clustering has been exploited in Bredin and Poignant (2013); [START_REF] Poignant | Unsupervised speaker identification in TV broadcast based on written names[END_REF]. The system described in Bredin and Poignant (2013) defines a graph where the nodes are speaker segments and OPNs. OPNs are used to express must-link and cannot-link constraints between the utterances. The clustering and the naming of those segments is done using an Integer Linear Programming formulation. As first investigated by [START_REF] Li | Person identification in tv programs[END_REF], the case study of videos allows to exploit the complementarity of audio and video modalities. AV cues such as talking head detection scores can be used to match faces and speakers and to improve the monomodal speaker and face diarizations. The scores of such cues are computed by estimating motion in the region of the lips. In addition, features such as the face size, the face position or the number of faces in the image are extracted and given to a supervised classifier (El [START_REF] El Khoury | Audiovisual diarization of people in video content[END_REF]; [START_REF] Vallet | A multimodal approach to speaker diarization on TV talk-shows[END_REF]) to further refine the talking assessment. In order to bring corrections to the initial monomodal diarizations, the talking head detection scores should be reliable where monomodal errors are present. Moreover, the audio and video will also be more complementary if they make errors at different moments. In other words, the improvements of the AV diarization depend on the performances of the initial monomodal ones. The work of [START_REF] Noulas | Multimodal speaker diarization[END_REF] integrates faces and speech segments in a factorial hidden markov model. The assignment of a segment to a cluster label is based on biometric model and on AV links with co-occurrent segments from the other modality. The use of a graphical model enables to express dependences between variables with a global probabilistic formulation which can then be optimized jointly. In order to jointly identify faces and speakers, the authors of [START_REF] Poignant | Naming multi-modal clusters to identify persons in tv broadcast[END_REF] proposed a constrained multimodal clustering. They use the simple idea that two segments which co-occur with different names implies that they should be assigned to different clusters. The authors also showed that their multimodal clustering of faces and speakers can make use of talking head detection scores to correct errors present in the monomodal systems.

Paul Gay et al.

person identification in videos

The work of [START_REF] Bechet | Multimodal understanding for person recognition in video broadcasts[END_REF], an interesting yet not detailed contribution to the field, reports the intensive use of multimodal scene understanding cues. First, speaker diarization is performed and speakers are identified using OPNs or pre-trained models. Then, identities are propagated from the speakers to the

METHOD

The method will be first described globally in section 3.1. In the section 3.2, we introduce the notations.

We then describe how we extract LFB and AV association features in sections 3.3 and 3.4. In section 3.5, the diarization CRF which clusters face and speech segments is presented, and in section 3.6 the naming CRF which is in charge of identifying the clusters. To conclude this part, we describe how the full system is used and optimized in section 3.7.

Method overview

The general approach is summarized in

Notations

The pre-processing includes obtaining initial monomodal face and speaker clusters, a set of OPNs and extracting the features from those elements. First, faces are detected [START_REF] Viola | Robust real-time face detection[END_REF]) and tracked within each shot, resulting in a set of face tracks denoted as Finally, the audio stream is segmented into a set A = {A i , i = 1 . . . N A } of continuous speech segments called utterances, each described by a set of acoustic features x a i . Features are 12 MFCCs with first order derivatives. Each frame is normalized with a short-term windowed mean and variance. Feature warping is also applied. In addition, a set of boolean features {x lfba i (k), k 2 K} is extracted indicating whether A i is co-occurring with a recurrent LFB, as described in the next section. Finally, talking head detection features

V = {V i , i = 1 . . . N V }. Each face track V i is
x av ij are extracted between each couple (A i , V j ) with a non-zero overlap as described in section 3.4.

Local Face Background recurrence

We want to capture whether a face appears with a recurrent visual background. This feature will be included in the diarization CRF. To this end, we focus on an area around each face track V i to capture the background context of this face. We do not consider full images as the same image might include different

Talking head detection features

In order to integrate AV association information in the CRF, we detect talking heads. To characterize talking heads, we use the following measures. These features are extracted for each overlapping utterance/face track couple and include:

• lip activity: the lip activity of a given face at frame k is computed as described in El Khoury et al.

(2012) and consists in the mean intensity difference between frame k and k + 1 after local image registration in predefined regions corresponding to the lips. In addition, we focus on the relative lip activity by dividing by the sum of all the lip activities measured from all people in the image.

• Head size: the interest of this feature relies on the hypothesis that the face of the speaker is usually larger than the faces of other people in the image. Put simply, we take the diagonal size of the detection bounding boxes. We also use the relative head size.

The previous features are computed from each frame of the face track. Eventually, the final feature x av ij is an average over all values from the frames included in the overlap between the utterance A i and the face track

V j . This corresponds to the method used in Gay et al. (2014c). To assess whether a couple of utterance/face track corresponds to a talking head given the features, we use an SVM with gaussian kernel denoted as h.

Audio-visual (AV) person diarization CRF

The clustering of face tracks and utterances defines itself by estimating the label field E d = {e a i , i = 1 . . . N A , e v j , j = 1 . . . N V } as such, the same person index is used for e a i and e v j when the utterance A i and the face track V j correspond to the same person. The labels e a i and e v j take value in the set of possible person indices denoted as P. To achieve this, let G be an undirected graph over the set of random variables A, V , O, and E d . We then seek to maximize the CRF posterior probability formulated as:

P (E d |A, V, O) = 1 Z(A, V, O) ⇥ exp ⇢ X i2F X c2G i i f i (A c , V c , O c , E d c ) (1)
where each triplet The association function f av favors the association of talking heads to utterances. The function is defined on all overlapping utterance/face track couples {(i, j)/t(A i , V j ) 6 = 0} where t(A i , V j ) is the overlapping time duration between segments A i and V j : The visual feature function f v (V i , e v i ), defined for all face tracks V i 2 V , indicates how likely the visual features x surf i of V i should be labeled with the person index e v i . This is a face modeling task in which for each label e i , we need to define a visual model that is learned from the data currently associated to the label.

(f i , G i , i ) is composed of a feature function f i ,
f av (A i , V j , e a i , e v j ) = ( t(A i , V j )h(x av ij ) if e a i = e v j t(A i , V j )h(x av ij ) otherwise (2)
Practically, f v computes as score between V i and a label e v i the 10th percentile SURF vector distances between x surf i and all the SURF features of the current face tracks associated with this label. The distance between two face tracks is computed following (El [START_REF] El Khoury | Face-and-clothing based people clustering in video content[END_REF]). Although the use of SURF features could be discussed regarding other more modern representations, we observe that their matching power is useful for similar faces of the same person viewed from a similar view point. The previous work in [START_REF] Gay | A conditional random field approach for face identification in broadcast news using overlaid text[END_REF] uses an average of the distances. By using the percentile, we found a slight improvement for the diarization task (0.2 points on the development REPERE corpus). We believe that the use of a percentile instead of averaging enables to merge 2 clusters of the same identity but containing samples whose poses are dominantly from different poses.

The acoustic function f a (A i , e a i ), defined over all utterances A i 2 A, is the audio equivalent of f v . We chose a 512 GMM-UBM with diagonal covariance following [START_REF] Ben | Speaker diarization using bottom-up clustering based on a parameter-derived distance between adapted GMMs[END_REF]. We did not use ivectors since we might need to learn a model on small clusters containing only a few seconds of speech. f a (A i , e a i )

computes the likelihood score of the features x a i given the GMM model learned over the data currently associated to the cluster label e a i .

The LFB feature function is driven by the assumption that faces inside a recurrent LFB are likely to correspond to a speaker announced by an OPN. To favor face tracks identified as recurrent LFB to join a person cluster which could be named, we define the following feature function. For each face track V i ,

f lfbv k (V i , e v i ) = ⇢ 1 if x lfbv i (k) and e v i 2 E opn 0 otherwise (3)
where E opn is the set of person clusters indices co-occurring with an OPN, i.e. the set of clusters which are currently associated with a name.

This principle is extended to each utterance A i with the function f lfba k which employs the feature x lfba i (k).

To this end, we assume that the utterances co-occurring with a recurrent LFB should be assigned a cluster label from the set E opn . Thus, as discussed in section 3.3,

x lfba i (k) is set to true if utterance A i is overlapping
with a face track V j such that x lfbv j (k) is true. We then introduce the same function as in the video case:

f lfba k (A i , e a i ) = ⇢ 1 if x lfba i (k) and e a i 2 E opn 0 otherwise (4)
Interestingly, these functions act as a namedness feature [START_REF] Pham | Linking names and faces: Seeing the problem in different ways[END_REF]) in the sense that they favor the naming of the corresponding face tracks and utterances. They also softly constrain the number of clusters. In other words, the clusters whose labels belong to E opn will attract the segments identified as recurrent LFB. Note that if the constraint was strictly enforced, each concerned audio or visual segment would only be assigned to a member of E opn .

The OPN feature functions bring a special treatment to the segments co-occurring with OPNs. The idea is to favor segments (face tracks or utterances) co-occurring with an OPN O j to be assigned to a person cluster likely to be labeled with the name x opn j . Thus, we define: cluster label e v i . This probability is computed with the naming CRF as defined in section 3.6.

f opn alone (V i , O j , e v i ) = ⇢ p(e c i = x opn j |C, P ) if V i is
Similarly, we use f opn multi if V i co-occurs with other faces:

f opn multi (V i , O j , e v i ) = ⇢ p(e c i = x opn j |C, P ) if V i co-occurs with OPN O j 0 otherwise (6)
We also define f opn audio for each co-occurring couple (A i , O j ):

f opn audio (A i , O j , e a i ) = ⇢ p(e c i = x opn j |C, P ) if A i co-occurs with OPN O j 0 otherwise (7)
Differentiating these 3 cases enables to learn specific weights so that the model behavior is adapted to each situation. The uniqueness feature function ensures two faces that co-occur in the same shot to have different labels [START_REF] Berg | Names and faces in the news[END_REF]; [START_REF] Pham | Naming persons in video: Using the weak supervision of textual stories[END_REF]). For such a pair V i , V j :

f uniq (V i , V j , e v i , e v j ) = ⇢ Inf if e v i = e v j 0 otherwise (8)
It is crucial to use this function because due to the OPN feature functions, multiple faces co-occurring with the same OPN will tend to be assigned to the same person cluster.

Cluster identification

The previous diarization CRF provides us a set of AV person clusters C = {C i , i = 1 . . . N C }. Thus, in the naming step, the goal incorporates estimating the label field

E N = {e c i , i = 1 . . . N C } such that
the label e c i corresponds to the name of the cluster C i . The label e c i takes value in the set of names M augmented by an anonymous label which should be assigned to anonymous persons. For this naming CRF, the posterior probability uses 6 feature functions:

P (E N |C, O) = 1 Z(C, O) ⇥ exp ⇢ 6 X i=1 X c2G i i f i (E N c , C c , O c ) (9

person identification in videos

on one hand, and with the audio segments of C i on the other hand. Moreover, we exploit the assumption that a person does not usually appear or speak before the first apparition of his name in an OPN to define f before (e c i , C i , O j ), which returns the number of audio segments from cluster C i that occur before the first apparition of the name x opn j associated to the OPN O j .

f before (e c i , C i , O j ) = ⇢ ]{A i 2 C i , end(A i ) < start(O j )} if x opn j = e c i 0 otherwise (11)
We also introduce prior knowledge over the anonymous label by defining a fifth feature function f ano (e c i , C i ) which returns 1 if e c i is the anonymous label. When applied, it allows the model to penalize the fact of not identifying a person and improves the recall.

Lastly, we define a uniqueness function f uniq (e c i , C i , e c j , C j ) over visually overlapping clusters just as in the diarization step. For each cluster pair (C i , C j ) with overlapping face tracks:

f uniq (e c i , C i , e c j , C j ) = ⇢ 1 if e c i = e c j 0 otherwise (12)

Optimization

The joint use of the two CRFs is conducted by applying the following steps: i) the diarization labels are firstly initialized by separately performing audio and video clustering and then associating the clusters to obtain the potential AV person labels P (audio and face cluster couples). The association is conducted using the Hungarian algorithm [START_REF] Kuhn | The hungarian method for the assignment problem[END_REF]) where the cost for a cluster couple is defined as the sum of the scores from the function f av over all its utterance/face track pairs. ii) For each resulting person label p i , biometric models are learned from their associated data and naming probabilities for each label are estimated by using the naming CRF. iii) Given these models, we run the loopy belief propagation inference to get the most probable diarization labels E d by solving

E d = arg max E d P (E d |A, V, O).
Eventually, Steps ii) and iii) are iterated in a Expectation-Maximization style by alternating model updates and inference. Ideally, one would iterate until convergence, i.e. when the label for each segment becomes stable. In practice, as there is no guarantee that the algorithm converges, a fixed number of iterations is tuned over the development set since we observe only small modifications after a few iterations.

The computational bottleneck with the Loopy Belief Propagation algorithm occurs in the presence of big cliques. This is the case, in our graphs, when the uniqueness constraint is applied to images where there are more than 20 faces. In such cases, uniqueness constraints can be dropped from the graph during inference and enforced in a post-processing step.

RESULTS AND DISCUSSION

This section will firstly present our experimental set-up: the corpus (section 4.1), implementation details (section 4.2) and the metrics used for the evaluation (section 4.3). Then in section 4.4, we present our results showing identification and clustering performances in function of the different parts of the model.

Corpus description

We used the REPERE corpus (Giraudel et We evaluate our approach on the final test set which contains 37 hours during which 10 are annotated. A development set is used to optimize the number of LFB functions and the number of iterations between the two CRFs. It consists of 28 hours among which 6 are annotated. The SVM h used in the f av feature function and the CRF parameters are learned on the test set of the first REPERE evaluation composed of 3 hours of annotated data.

Parameter settings and algorithm details

We set the K value to {3, 4, 5} for the LFB feature functions. We set the number of iterations between the two CRFs to 3 as we noticed that no major changes usually occur after that point. It is important to note that these CRF parameters are learned on automatic detections and automatic clusters and not on cleanly segmented ones. Therefore, it enables us to take into account the noise present at test time. We use the GRMM toolbox [START_REF] Mccallum | MALLET: A machine learning for language toolkit[END_REF]) for the CRF implementation.

The initial speaker diarization system is the LiumSpkDiarization toolbox 1 which combines ivector representation and ILP clustering [START_REF] Rouvier | An open-source state-of-the-art toolbox for broadcast news diarization[END_REF]). It has achieved state-of-the-art results in several speaker diarization benchmarks [START_REF] Rouvier | A global optimization framework for speaker diarization[END_REF]). The initial face diarization uses the system described in [START_REF] Khoury | Fusing matching and biometric similarity measures for face diarization in video[END_REF]) which combines SURF based distances and DCT features whose distribution is modeled with GMMs. This system has been evaluated on the public Buffy dataset [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF]) and compares favorably to other metric learning methods. The use of state-of-the-art systems enables us to verify that our CRF is able to correct errors which are proven difficult to solve in the monomodal case.

Performance measures

The overall identification performance is measured with the Estimated Global Error Rate (EGER) which is the REPERE evaluation metric. It is defined as follows: . Extract of an evaluation file for face identification. The second row is the reference name list and the third row is the predicted list. Augusta ADA KING will be counted as correct. One of the two remaining names will be counted as confusion with Barack OBAMA, and the third one will be a false alarm. Since there are 2 persons in the reference and the system made 2 errors, the corresponding EGER of this example is 1.

EGER = ] conf + ] miss + ] false ] total (

Speaker diarization results

Initial 1.Speaker and face diarization performances in terms of CER. The first column presents the initial monomodal systems [START_REF] Khoury | Fusing matching and biometric similarity measures for face diarization in video[END_REF]; [START_REF] Rouvier | An open-source state-of-the-art toolbox for broadcast news diarization[END_REF]. The second one is the diarization CRF presented in this paper. The third one is the same as the second one, however, we remove the OPN related functions f lfb and f opn .

where the audio file is divided in continuous segments at each speaker change and:

• dur(seg) is the duration of the segment seg.

• N Ref (seg) is the number of active speakers during segment seg.

• N Sys (seg) is the number of speakers detected by the system.

• N Correct (seg) is the number of speakers correctly detected by the system. A match needs to be made between the clusters and the speaker references in order to compute this term.

We applied this measure to the face clustering task. With the audio CER, a detected speech segment is matched to a reference during their temporal overlap. The only modification to tackle visual modality is that a face detection must have a temporal AND spatial overlap to be matched with a reference. In addition, note that we do not consider false alarms and missed detections that are usually considered in NIST to compare the effects of the different systems since the only error that changes with methods given the setup (fixed face tracks and utterances) is due to the final clustering of the face and speech segments. Thus, miss detections and false alarms are identical.

Identification and clustering results with the CRF combination

Diarization results: we first describe the diarization results presented in Table 1. We can see that the full CRF model has a slightly lower error rate over the whole corpus than the initial monomodal systems (6.8% vs 7.4% for the speakers and 5.0% vs 5.2% for the faces). On the other hand, the performances depend

function f a f v f av f lfbv k f lfba k a v av lfbv k lfba k
value 0.4 1.8 0.2 1.9 1.7 Table 2.The parameter values for some of the feature functions used by the diarization CRF. For lfbv k and lfba k , the value of k is 5, which is the highest parameter value. It corresponds to the most common case as 90% of the segments are inside background clusters which contain more than 5 elements.

strongly on the type of shows. For instance, an important part of the global improvement comes from the debate videos (4.0% vs 6.6% for the speakers and 1.9% vs 4.9% for the faces). In debates, most of the scenes are in the same studio thereby reducing the visual variability of the background image and most of the persons present are speakers announced by an OPN. Thus, most faces and utterances are featured as recurrent (i.e. x lfbv i is set to true) and the f lfb functions have a positive impact on the diarization. They enable to solve clustering confusion errors by constraining the number of clusters toward the number of detected OPNs. Indeed, if we remove the OPN related functions f lfb and f opn (cf third column of the Table 1), most of the improvements are lost. It appears that the use of multimodality does not help to correct clustering errors. This is somewhat surprising as past works (Gay et al. (2014c)) reports improvements in the audio modality with this very system on the same type of data. The difference with this previous work is that our initial monomodal speaker diarization system has become much more efficient, essentially thanks to a careful selection of the data used to train the generic speaker model UBM. This way, there are much fewer errors to correct.

In the case of celebrity magazines, the diarization CRF increases the error rate (15.1% vs 14.6% for the speakers and 7.9% vs 3.5% for the faces). Those videos contain very few OPNs and essentially short outdoor scenes. Thus, the f lfb functions cannot help the CRF to take appropriate decisions. Moreover, previous experiments reported in Gay et al. (2014c) showed that the use of the biometric person models inside the CRF framework appears to be less efficient than when it is used in the hierarchical monomodal systems.

The importance of the OPN related functions is also visible if we consider the parameters learned by the CRF in Table 2. During training, the weight av are indeed set to a relatively low value as compared to the other terms (although those values are ponderated by the amplitude of the feature functions). We have found that for a majority of segments, the f lfb function is dominant. This is further illustrated in table 4.

Identification results: We now turn to the identification results reported in Table 3. We compare 3 systems:

we denote by N the naming CRF applied on top of the initial monomodal diarizations described in Khoury Adding the diarization CRF permits to globally reduce the error rates in both modalities (31.4% vs 33.4% for the speakers and 52.2% vs 54.5% for the faces), especially for debate and parliament videos. This is not surprising as we previously showed that the diarization CRF have less confusion errors for studio scenes than the initial monomodal systems.

Regarding news videos, although we saw that clustering confusion errors were not reduced globally, the use of the diarization CRF also improves the identification. This is probably due to the correction of confusion errors in studio scenes which have a greater impact on the identification than errors concerning anonymous persons in reports. Audio Visual N 33.4% 54.5%

N + D (f a + f v + f av ) 34.1% 56.2% N + D (f a + f v + f av + f opn )
33.9% 56.4% N + D (f a + f v + f av + f opn + f uniq ) 33.9% 55.6% N + D (f a + f v + f av + f opn + f uniq + f lfb ) 31.4% 52.7% Table 4. Contribution of the different diarization model components on the naming task (results in EGER).

As in table 3, the system N is the naming CRF with the monomodal diarizations. The other lines correspond to the combination of the naming and the diarization CRF, using as feature functions in the diarization CRF those given in parenthesis.

full model provides the best performances. It should also be noticed that, although it might generates big cliques in some cases, the uniqueness function is essential to benefit from the f opn feature functions. If not applied, an OPN will be propagated to all the faces overlapping with him.

Comparison with state of the art and discussion: on the same dataset, the system described in [START_REF] Bechet | Multimodal understanding for person recognition in video broadcasts[END_REF] obtains an EGER of 30.9% for the speakers and 39.4% for the faces. Thus, it proves to have a better performance especially regarding the faces. This is possible with the help of pre-trained models for each show which enable to indicate how many faces should be present on screen and what their roles are. For instance, when it detects the configuration shown in Fig 1c, it deduces that the announced guest is present on the right even if no faces have been detected. In fact, this approach does not even use a face diarization module. However, it requires a large amount of learning and a priori information.

By comparison, our method is much simpler to implement, especially since it has better generalization capabilities, we learn one single model over a large and diverse corpus, and what is more, it requires less annotations if we need to process a new type of show.

The constrained hierarchical clustering detailed in [START_REF] Poignant | Naming multi-modal clusters to identify persons in tv broadcast[END_REF] obtains an EGER of 35.9% for the speakers and 44.3% for the faces. Compared with our system, it has better performances on the faces, but worst for the speakers. As we do, they only rely on OPNs without other specific supervised information on the show. According to their paper, it seems that their constrained multimodal clustering, which avoids clustering together faces which co-occur with different OPN names, is one of the contributions which improves results and that we do not use, and could explain the difference. Nevertheless, the influence of each pre-processing (speaker and face detections, monomodal clusterings and OPN detection) makes it hard to analyse the performance difference.

CONCLUSION

In this paper, we presented our contribution for AV person diarization and identification from OPNs. Our system uses an iterative combination of 2 CRFs. One performing the AV diarization at a person level, and a second one associating the names and the clusters. Several context modeling cues are used to solve the person/name association problem and the diarization issues. While it is clear that more supervised learning and a priori information on the context can improve the performances, our approach provides an interesting trade-off between performance on one hand and generalization/low annotation cost on the other hand. The principal contextual cue consists in the face image background. It allows us to distinguish the faces and the speakers which are announced by OPNs and guide the clustering accordingly.

In this work, we did not address the issue of non-frontal face detection. As a short term perspective, it would be interesting to increase the recall of the face detector, for instance, by adding a profile view detector. This would render the face clustering task more challenging and the potential benefit from context modeling would be greater. Secondly, our context modeling assumes that speakers are announced by an OPN the first time they talk. For the REPERE dataset, this is the case. However, this assumption could be sensible to broadcaster's editing policies. Actually, the optimal choice of the context for unsupervised person identification is a difficult problem if we want to avoid the need for specific annotations for each show. One solution to consider is to learn the setting of each show or a part of the setting from a corpus in an unsupervised way.

Figure 1 .

 1 Figure 1. Example frames from the REPERE corpus showing the variety of the visual conditions (pose, camera viewpoint, illumination) and the name face association challenges such as: multiface images (image c) and name propagation (from a to b). Images a and c show an example of OPNs.

  faces. Scene segmentation, role detection, and pre-trained visual models for each TV set (and sometimes for each camera) are used to indicate how many faces are present on screen and what their roles are. Such a fine-grain modeling enables them to report the best identification on the REPERE campaign. Indeed, it permits to tell which persons are present without detecting the faces by detecting the specific shot (up to which studio camera is used). Thus, profile views and persons seen from the back can be identified. However, to learn those models, manual annotations have been made for each show. This poses the problem of human labor cost and lack of generalization. More generally, several researchers focus on exploiting the context surrounding the faces. The work in Zhang et al. (2013) uses clothes, image background, cluster co-occurrences and attribute classifiers and Tapaswi et al. (2014) build must-link and cannot-link constraints deduced from shot threads (sequence of shots obtained from the same camera angle). Contributions: in this paper, we leverage on different contextual cues present in the state of the art, introduce new ones, and include them in our CRF model. First, instead of conducting speaker and face clustering separately [15, 5], we perform a joint clustering of face tracks and speaker segments which also benefits from the OPNs information. To be more precise, we compute Local Face visual Backgrounds (LFBs) around each face track and cluster them. This provides us with a signature for each face track characterizing the level of recurrence of its LFB in the data. Intuitively, recurrent a LFB correspond to people who are important and can be seen as a soft role assignment distinguishing faces to be named from faces of figurative people. Concretely, it enables to encourage faces tracks with recurrent LFBs to join named clusters, i.e. overlapping an OPN. Secondly, a naming CRF performs the joint identification of all person clusters, thus allowing to account for uniqueness constraints and co-occurrence statistics between clusters and OPNs. Unlike previous works which rely on extensive annotations (Bechet et al. (2014)), those elements of context have better generalization capabilities, since we can learn one single model over a large and diversified corpus, and require less annotations if we want to learn a new type of show. Thanks to the flexibility of the CRF formulation, new contextual cues could be added in the future to further improve the performances.

  Fig 2. First, the different modalities are processed separately:monomodal speaker[START_REF] Rouvier | An open-source state-of-the-art toolbox for broadcast news diarization[END_REF]) and face[START_REF] Khoury | Fusing matching and biometric similarity measures for face diarization in video[END_REF]) diarizations are performed, LFBs are extracted around each face and clustered, Optical Character Recognition (OCR) is performed to extract the overlaid texts[START_REF] Chen | Video text recognition using sequential monte carlo and error voting methods[END_REF]) and named entities are detected(Gay et al. (2014a)).In the second part, we perform the AV clustering and the naming of the persons. Initially, we use the Hungarian algorithm to associate face and speaker clusters based on their temporal overlap. Naming

Figure 2 .

 2 Figure 2. Overview of the system. First, face tracks and speech utterances are detected and clustered separately. They are then used with the OPNs in an iterative combination of the two CRFs. A first one to refine the diarization, and the other one to identify the clusters. The latter is eventually used for the final cluster identification.

  characterized by a set of visual features x surf i (set of Speeded-Up-Robust Features (SURF) extracted in up to 9 images of the face track (El Khoury et al. (2010))) and a set of boolean features {x lfbv i (k), k 2 K} indicating whether V i corresponds to a recurrent LFB as explained in the next section 3.3. Second, OCR (Chen and Odobez (2005)) and Named Entity Detection techniques based on string matching against external resources (predefined lists, freebase database, Google hits. . . ) are applied as described in Gay et al. (2014a) to extract the set O = {O i , i = 1 . . . N O } of OPNs. Each OPN O i is characterized by its duration d opn i and its name x opn i 2 M where M = {n j , j = 1 . . . N M } denotes the set of unique names extracted from the video.

Figure 3 .

 3 Figure 3. Left: predefined spatial template for the background face selection area given a head detection. Center and right: two examples with head detections and the selected background areas.

Figure 4 .

 4 Figure 4. Top: original images. Bottom: clusters of recurrent local face backgrounds (LFB) automatically cropped from each image and clustered given the localised faces. Each row corresponds to a local background cluster. Clusters 1 and 2 are recurrent backgrounds and correspond to speakers. Clusters 3 and 4 are not recurrent. They contain non-speaking faces which appear occasionally in the video.

  face visual contexts (see the first, fourth and fifth images from the left in the top row of Fig 4). Instead, we select a rectangle area around each face as local face background (LFB) representative by following a predefined spatial template between the face and this rectangle as can be seen in Fig 3. In practice, the fixed proportions were chosen manually so as to avoid a potential overlap with other parts of the images in typical edited videos like in the 4th and 5th images from the left on the top of Fig 4. We then characterize Paul Gay et al. person identification in videos each obtained rectangle area with SURF features and, in order to cluster them, we use a hierarchical clustering approach (El Khoury et al. (2010)). Then, we set x lfbv i (k) to true if face track V i belongs to a local background cluster whose number of elements is higher than k. In practice, multiple values of k can be used to characterize different levels of recurrence and reduce the importance of the stopping criterion of the hierarchical clustering. Fig 4 shows examples of obtained recurrent and non-recurrent patterns.

  a weight i learned at training time and the set G i of cliques where this function is defined. (A c , V c , O c , E c ) denotes the set of nodes contained in the clique c. F is a set of abstract functions indices. We use 6 types of feature functions which will be described in the next sections. A graphical representation of this model is illustrated on Fig 5.

Figure 5 .

 5 Figure5. Factor graph illustrating the diarization CRF using talking head information (f av ) and the context from the OPNs (f opn , f lfb ). The blank circle nodes correspond to hidden variables, the shaded circle nodes correspond to observations while the squares represent the feature functions. The x-axis represents time and the drawing shows also segments corresponding to the time intervals during which a specific observation (track, utterance, OPN) occurs. Red dot segments illustrate the face track temporal segments while blue plain segments the utterances.

  alone in the image and co-occurs with OPN O j P ) is the probability that the name contained in the OPN O j corresponds to the cluster label e v i given the clustering C and the set of OPNs P . Here, we denote as e c i the naming label of

Figure 6 .

 6 Figure 6. Factor graph illustrating the naming CRF using the co-occurrence functions with the OPNs and the uniqueness constraint. The conventions are the same as in Fig 5.

)Fig 6

 6 Fig 6 represents an illustration of this. This naming model exploits four different co-occurrence statistics between clusters and OPNs. The first function f alone is defined over each triplet (e c i , C i , O j ), where the OPN O j must co-occur with a face track which belongs to C i and which is alone in the image. Let us denote as (C i , O j ) the co-occurring time between the face tracks which occurs alone in the cluster C i and O j . Then, we have:

  al. (2012)) for our experiments. It involves broadcast data videos containing 4 main types of shows: i) debates in indoor studio (Fig 1a,b); ii) modern format information shows which contain reports and interviews with dynamic picture compositions (Fig 1c,d); iii) extracts from parliamentary sessions "Questions to the government" (Fig 1e) iv); celebrity news (Fig 1f).

  Figure7. Extract of an evaluation file for face identification. The second row is the reference name list and the third row is the predicted list. Augusta ADA KING will be counted as correct. One of the two remaining names will be counted as confusion with Barack OBAMA, and the third one will be a false alarm. Since there are 2 persons in the reference and the system made 2 errors, the corresponding EGER of this example is 1.

  Fig 8). Adding the diarization CRF permits to globally reduce the error rates in both modalities (31.4% vs

Figure 9 .

 9 Figure 9. The Y-axis is the EGER difference between the CRF combination and the naming CRF alone measured for the faces. The X-axis is the DER difference between the diarization CRF and the initial monomodal face diarization Khoury et al. (2013).

Paul Gay et al. person identification in videos

This is a provisional file, not the final typeset article

Frontiers

 3. Identification performances measured in EGER. The system N is the naming CRF on top of the monomodal diarizations and the system N + D is the naming and diarization CRF combination. The structure of celebrity magazines differs from the other shows as it contains very few OPNs and recurrent LFB. In those cases, the diarization CRF degrades both diarization and identification performances.

We design an oracle on the diarization and the identification to measure the potential improvements. It uses automatic face/speech segment detections and automatic OPN extraction. Then, the association between these segments and the OPNs is done with the manual reference. Thus, the errors made by the oracle correspond to missing OPNs or missing segment detections. In the case of celebrity shows, with an error rate of 75.3%, the OPN-based approach is clearly not suitable.

Error analysis: the proportion of the different error types can be visualized globally on the pie charts in We also illustrate the correlation between diarization and identification performances in figure 9. We plot the performance differences for each video between the full system (N + D) and the CRF naming alone (N ). We observe that they are unique to their type of show. The debate videos appear in the top-right part of the plane, which means that the diarization CRF improves the diarization and the identification.

Concerning news and parliament videos, the correlation between CER and DER is not as strong. The presence of anonymous persons and off voices imply that a change in the diarization does not necessarily correspond to a change in identification performances.

Finally, the table 4 shows the performance of the model when adding the different components of the diarization CRF one by one. If we focus on the first and second lines, we see that the CRF with only 3 feature functions degrades the performances compared to the monomodal diarizations. We find that, used alone, the monomodal representations present in the CRF (see the f a and f v functions) do not compare favourably with the monomodal diarization frameworks. This could be improved in a future work by using


Gaël Le Lan

Sylvain Meignier

Delphine Charlet

Paul Deléglise

Gaël Le Lan

Speaker Diarization With Unsupervised Training Framework

Keywords: Speaker diarization, speaker linking, unsupervised training

published or not. The documents may come   L'archive ouverte pluridisciplinaire

INTRODUCTION

Speaker diarization, as defined in the NIST evaluations on Rich Transcription [START_REF]Fall 2004 rich transcription (RT-04F) evaluation plan[END_REF], aims to uniquely identify speakers across an audio stream, without a priori knowledge about the speakers. Until recently, speaker diarization was applied on individual shows.

With the increasing amount of data and the need for indexation of multimedia documents, performing diarization has become mandatory, not only on individual shows but on collections of shows. This task is known as cross-show speaker diarization [START_REF] Dupuy | I-vectors and ILP Clustering Adapted to Cross-Show Speaker Diarization[END_REF], speaker linking [START_REF] Ferràs | Speaker Diarization and Linking of Large Corpora[END_REF] or speaker attribution [START_REF] Ghaemmaghami | Speaker attribution of multiple telephone conversations using a complete-linkage clustering approach[END_REF]. Thus, some implementations have been proposed [START_REF] Dupuy | I-vectors and ILP Clustering Adapted to Cross-Show Speaker Diarization[END_REF][START_REF] Ferràs | Speaker Diarization and Linking of Large Corpora[END_REF][START_REF] Tran | Comparing Multi-Stage Approaches for Cross-Show Speaker Diarization[END_REF][START_REF] Yang | Investigation of Cross-show Speaker Diarization[END_REF][START_REF] Dupuy | Is incremental crossshow speaker diarization efficient to process large volumes of data?[END_REF]. The usual approach consists in applying multistage clusterings, considering shows one by one and then merging the speaker clusters in the cross-show step. The ETAPE and REPERE campaigns [START_REF] Galibert | The ETAPE Speech Processing Evaluation[END_REF][START_REF] Galibert | The first official repere evaluation[END_REF] addressed the problem of speaker diarization on French TV broadcast news data and an implementation has been proposed, where the system was trained on annotated data [START_REF] Dupuy | Is incremental crossshow speaker diarization efficient to process large volumes of data?[END_REF]. The MGB Challenge [START_REF] Bell | The mgb challenge: Evaluating multi-genre broadcast media transcription[END_REF] addressed the same problem on English TV broadcast data, but no annotations were provided for training data.

The state of the art speaker modeling is based on i-vector/PLDA frameworks, which require data labeled by speakers, indicating who speaks when. Such frameworks are based on inter session speaker variability modeling, thus training corpora require multiple occurrences of a same speaker across different shows. Such labeled data are expensive to produce and are not always available for target data, resulting in a mismatch of language and/or noise conditions. Solutions to this problem have already been proposed in the context of speaker verification, based on domain adaptation [START_REF] Shum | Unsupervised clustering approaches for domain adaptation in speaker recognition systems[END_REF] or unsupervised iterative PLDA training [START_REF] Liu | An iterative framework for unsupervised learning in the plda based speaker verification[END_REF]. For both systems, training was done on mono-speaker unlabeled data.

In this paper we address the problem of cross-show diarization, using a priori models trained on unlabeled data. We propose an unsupervised framework for data selection, in order to train models needed to extract i-vectors (Universal Background Model (UBM) and Total Varibility (TV) matrix) and to compute PLDA (eigenvoice and noise covariance matrices). Those models are trained offline and their performances on single and cross-show speaker diarization are compared with supervised models, trained on the same audio data, but using speakers labels.

Subsequent sections are organized as follows: first, we describe the baseline single-show and cross-show diarization systems. Then, we present a solution for the data selection problem for model training and conclude with the experimental results using unsupervised and supervised models.

DIARIZATION FRAMEWORK

Figure 1 describes the diarization process, which is detailed below. This process can be based on a i-vector/PLDA system trained in a supervised or unsupervised way.

BIC segmentation and clustering

As presented in the first part of Figure 1, after a Viterbi-based speech/non speech detection, a two pass acoustic segmentation, followed by a hierarchical agglomerative clustering (HAC), is applied over the speech segments. The segmentations and the clustering are described in [START_REF] Meignier | LIUM SpkDiarization: An opensource toolkit for diarization[END_REF] where the standard BIC is replaced by the segmental square-root BIC described in [START_REF] Stafylakis | The segmental bayesian information criterion and its applications to speaker diarization[END_REF]. From previous experience, the segmental square-root BIC outperforms the standard BIC on target data.

Single Show Speaker clustering

At this point, each cluster is pure and supposed to represent a single speaker; however, several clusters can be related to the same speaker. In the previous steps, features were not normalized because the channel contribution was useful to differentiate the speakers. In this step, the channel contribution is removed with a zero mean and unit variance normalization computed for each cluster. A final clustering stage is then performed in order to obtain one cluster by speaker.

Speaker Modeling: i-vector/PLDA

Speaker clusters are modeled with i-vectors, normalized by Spherical Nuisance Normalization [START_REF] Bousquet | Variance-spectra based normalization for i-vector standard and probabilistic linear discriminant analysis[END_REF]. PLDA is used to calculate log likelihood ratio for pairs of i-vectors [START_REF] Kenny | Bayesian speaker verification with heavy tailed priors[END_REF], which we call PLDA scores or PLDA distances in the following of the paper. The term PLDA matrix indicates the matrix of the log likelihood ratios. I-vector and PLDA are computed using the Alize toolkit [START_REF] Larcher | ALIZE 3.0open source toolkit for state-of-the-art speaker recognition[END_REF]. The dimension of the feature vectors is 60: 20 MFCCs including c0 coefficient supplemented with the first and second order derivatives. The GMM is composed of 256 Gaussians with diagonal covariance matrix, the dimension of the ivectors is 200 and PLDA eigenvoice matrix has a dimension of 100.

For experimental purposes, we also present a system which does not use PLDA nor SNN, where the scores between i-vectors are the cosine distance between them.

Clustering Process: CC+HAC

For the last step, a graph based clustering followed by a HAC is performed on the PLDA matrix. The negative of the PLDA distance matrix 1 can be interpreted as a connected graph, illustrated in Figure 2, where the clusters are represented by the nodes, and scores between clusters are represented by the links. This approach is similar to the ILP clustering presented in [START_REF] Dupuy | Recent improvements towards ILP-based clustering for broadcast news speaker diarization[END_REF].

This graph can be simplified by removing all the unnecessary links corresponding to scores above a threshold δ. After this simplification, the graph now contains a set of subgraphs corresponding to the connected components (CC) of the graph.

A complete-linkage HAC is then performed on each complex subgraph (ie. not a star-graph). In this case, the HAC tries to find classes within a diameter less than a threshold β.

Cross-show Speaker Diarization

Once speaker diarization has been applied to each show separately, the collection of shows is considered as a global single-show prob- 1 we change the sign of PLDA score to minimize modifications in the classification algorithm that usually works with distances. lem and the single-show clustering process is reapplied on newly formed clusters. Each single-show cluster is modeled with an ivector, a distance matrix is computed and CC+HAC clustering is applied in a global way.

EXPERIMENTAL CONTEXT

Contrastive models for diarization systems were trained on manually annotated corpus. In this corpus the speakers are identified by their first and last names, providing several sessions for a large set of speakers. About 200 hours of French broadcast news drawn from REPERE [START_REF] Galibert | The first official repere evaluation[END_REF], ETAPE [START_REF] Galibert | The ETAPE Speech Processing Evaluation[END_REF] and ESTER [START_REF] Galliano | The ESTER 2 evaluation campaign for the rich transcription of French radio broadcasts[END_REF] evaluation campaigns were used to build three corpora. The shows were broadcast between 1998 and 2007, each show during between 10 minutes and an hour. The corpora also contain some broadcasts of Moroccan radio, in French language. For each show in the corpus, multiple episodes are available.

Train corpus

The train corpus is composed of 344 audio files from train and development corpora of the three previously cited campaigns, for a total of 1772 hours of speech duration. For each show, all available episodes are taken, meaning many speakers appear in more than one episode. Some speakers also appear in different shows (politicians, for example). The corpus contains 3888 unique speakers. Among those speakers, 929 appear in at least two different episodes (from the same or from different shows), we call them Cross-Show (CS) speakers, as opposed to Single-Show speakers (SS), who only speak in one audio. Thus, this corpus is well suited for a i-vector PLDA system training.

Test corpora

DATA SELECTION FOR UNSUPERVISED MODEL TRAINING

The proposed selection method has been inspired by the IDIAP method [START_REF] Khoury | Hierarchical speaker clustering methods for the nist i-vector challenge[END_REF] presented during the NIST i-vector Machine Learning Challenge 2014 for speaker recognition. In this challenge, unlabeled i-vectors are directly provided, without audio data. The training corpora need to be automatically constructed, using clustering methods to train session compensation, UBM, TV matrix and PLDA. Our task is more demanding than the i-vector challenge: the i-vectors need to be extracted from unsegmented audio data.

According to the diarization system described in section 2, required models for diarization are a Universal Background Model (UBM), a Total Variability matrix (TV) and a PLDA matrix. For the data selection task, only the audio of the training corpus is usable, annotations provided in the corpus are not authorized.

Universal Background Model

The Universal Background Model (UBM) is required for the Baum-Welch statistics needed during i-vector extraction. It needs to be trained on clean speech segments with few background noise that are representative of test set in terms of duration, gender and channel. Thus, the output of the GMM based speech / non-speech detector on the training corpus is used to train the UBM, which contains 256 Gaussians with diagonal covariance matrices.

Total Variability

The Total Variability (TV) matrix is required to extract i-vectors. It must be trained over segments containing a unique speaker. On the test corpus, most of the classes produced by the BIC diarization system are pure (they contain only one speaker). According to this observation, we consider each class produced by the BIC diarization applied to the train corpus as a segment representing a unique speaker and the total variability matrix is trained on those with a dimension of 200.

PLDA

Spherical Nuisance Normalization requires the mean vector and the within-class co-variance matrix of the i-vectors, whereas PLDA is trained using normalized speaker labeled i-vectors. Both systems are designed to compensate the inter-session speaker variability. In both cases, the training data must be labeled by speaker and session. Several sessions are needed per speaker to obtain robust models.

Hence, we need to perform a cross-show speaker diarization on the training corpus to automatically obtain speaker clusters, containing data from the same speaker and from different audios. PLDA will then be trained over the i-vectors extracted from the automatically obtained speaker clusters.

An i-vector for each class produced by the BIC diarization is extracted using the 256-UBM and TV matrix trained over the training corpus. A cosine distance matrix between all the i-vectors is computed, without normalization, and a CC+HAC clustering is applied, thresholds δ and β being set at the same value (see 2.2.2). In the resulting clusters, we only keep the ones containing at least 3 initial classes. The kept clusters are considered as containing multiple occurrences of a single speaker.

EXPERIMENTS

The metric used to measure performance in speaker diarization is the Diarization Error Rate (DER). DER was introduced by the NIST as the fraction of speaking time which is not attributed to the correct speaker, using the best match between references and hypothesis speaker labels. The scoring tool is employed for single-show and cross-show speaker diarization. In this last case, a cross-show speaker would always be labeled in the same way, in every recording that composes a collection.

Data selection

Supervised PLDA was trained using labeled data from the train corpus, over speakers appearing in at least 3 episodes. From previous experience, keeping speakers with a least 2 occurrences gives slightly lower performances. This process allowed us to retrieve 303 different speakers. Those speakers represent 61, 34% of the total speech time of the train corpus. We call that proportion the data coverage (proportion of the train data used for PLDA modeling).

For the unsupervised PLDA, CC+HAC clustering is performed on the train corpus. Figure 4 shows the clustering results, depending on the clustering threshold applied on cosine distances between i-vectors representing the initial BIC classes. Resulting clusters are used to generate the unsupervised P LDAunsup. The results are plot in terms of number of automatically extracted clusters, their average purity and the data coverage (proportion of initial data kept) they represent. The figure also presents single-show DER results on the EST ERtest corpus, for each generated PLDA.

The DER is lower than 8.5% for a wide amount of threshold, varying from -0.7 to 0.1, for a purity varying from 93% to 40% and for a data coverage varying from 40% to 100%. At the lowest threshold, the model is still performing well although the data coverage is lower than for the supervised PLDA (40%), generating 395 different clusters, with high purity (93%). On the other hand, at the highest threshold, the number or clusters is around 200, purity is relatively low, all data from the corpus are kept but the added data do not affect too much the performances. What seems to be important for the PLDA to be effective is generating enough clusters, which a relative high purity. When the number of clusters falls under 200, too many different speakers are agglomerated and the modeling is too poor.

The unsupervised PLDA which gives the lowest DER on the EST ERtest corpus is achieved for a threshold of -0.4. For this threshold, the single-show DER is 7.17%, the purity is 85.98% and the data coverage is 73.96%. The clustering process at this threshold generates 440 clusters (i.e. supposed recurring speakers) from the train corpus. Analyzing the composition of those clusters, we note that the average recall of all speakers is 90%, meaning that all classes from a same speaker are in majority regrouped in a same cluster. But we also note that 17% of the speakers have a recall less than 90%, representing 27% of the total speech time, some of the main speakers of the corpus being split in different clusters.

Single and cross-show diarization

Table 2 shows single and cross-show DER on both test corpora, using cosine and PLDA distance metrics (see 2).

TV matrix influence

Cosine distance is only based on i-vectors, without normalization. Thus, performances with cosine distance show the difference between supervised and unsupervised modeling of the UBM and TV matrix. Performances between both modelings are similar, except for the cross-show DER on EST ERtest.

PLDA modeling

PLDA-based scoring results allow to compare the full supervised and unsupervised systems. Single-show DER is quite similar on both configurations on EST ERtest, while unsupervised modeling provides better results on BF Mtest. The unsupervised system performs slightly better than the baseline on the cross-show task, despite the imperfect purity (85.98%) of the generated clusters for the unsupervised PLDA modeling.

We suppose the difference of performance between the supervised and unsupervised PLDA-based system is explained by the fact that some speakers with a high speech time and many occurrences (see 5.1, having a high inter-session variability, are considered as separate speakers in the unsupervised system, and give a better modeling : the amount of clusters for PLDA training is higher when generated automatically (440 versus 303 speakers generated from labels).

While previous work [START_REF] Liu | An iterative framework for unsupervised learning in the plda based speaker verification[END_REF] shows that an unsupervised PLDAbased performs less than a supervised PLDA-based system on the speaker verification task on telephone data, our experiments show that both supervised and unsupervised PLDA-based systems give similar results on TV/Radio data. We suppose it is due to the fact that our unsupervised system is trained on data which is processed in the same way than the test data (clustered BIC diarization classes), contrary to the supervised system, where training data is obtained on manual annotations. In the NIST i-vector challenge, there is no segmentation problem, i-vectors used for unsupervised PLDA training are already provided, while in our system, they need to be produced from unsegmented audio data. Table 2. single (SS) and cross-show (CS) DER on test corpora.

CONCLUSION

While ESTER and REPERE campaigns allowed to train a single and cross-show diarization system based on i-vector/PLDA frameworks trained on labeled data, we proposed to train the i-vector/PLDA framework models over automatically selected data from the training corpus, using data from those campaigns. We compared it with supervised models (e.g. trained on the same corpus, using provided manual labels) on the cross-show speaker diarization task. We show that having manual labels to build a performing system is not mandatory, and that automatically select training data allows to build a sufficient i-vector/PLDA framework for cross-show speaker diarization on TV and radio broadcast news data.

Further work should be dedicated to better understanding of the composition of the unsupervised clusters, and to iterative unsupervised domain adaptation for processing new types of unlabeled collections of data.

Fig. 1 .

 1 Fig. 1. Overview of speaker diarization systems.

Fig. 2 .

 2 Fig. 2. Graph clustering using sub-components. The dashed circle indicates a sub-graph, for which a HAC needs to be performed; the colored clusters are identified as star-graph centers.

Fig. 3 .

 3 Fig. 3. Unsupervised selection methods for train corpus.

Fig. 4 .

 4 Fig. 4. Metrics on the unsupervised i-vector cosine clustering of training data for P LDAunsup generation

  In this paper, we define two test corpora built from REPERE and ESTER test corpora. The first one, named EST ERtest, is the test corpus from ESTER campaigns, it contains Radio broadcasts. The second test corpus, named BF Mtest, is the collection of all available episodes of the TV news talk-show BF M Story. It has been selected because it is the one with the highest number of episodes (42), and there is a large amount of CS speakers, who speak for more than 55% of the total speech duration of the collection. Numerical details about the two corpora are presented in table 1.

	Corpus	EST ERtest	BF Mtest
	Episodes	24	42
	SS speakers	454	358
	CS speakers	42	79
	Total speakers	496	437
	Total duration	14h43m00s	43h12m17s
	SS speakers speech proportion	74,60%	44,62%
	CS speakers speech proportion	25,40%	55,38%

Table 1 .

 1 Composition of test corpora.


Sylvain Meignier
email: sylvain.meignier@lia.univ-avignon.fr

Jean-François Bonastre
email: jean-francois.bonastre@lia.univ-avignon.fr

Stéphane Igounet
email: stephane.igounet@univ-avignon.fr.

Jean-Franc ¸ois Bonastre

E-HMM

Introduction

Seeking within a recording the speech sequences uttered by a given speaker is one of the main tasks of document indexing. Segmentation systems first detect breaks in audio streams and then cluster in homogeneous sound classes the segments according to those breaks.

In automatic speaker recognition, segmentation consist in finding all the speakers, as well as the beginning and the end of their contributions. The speaker segmentation problem is commonly approached by two methods.

The first method (described in [1] and [START_REF] Gish | Segregation of speakers for speech recognition and speaker identification[END_REF]) is composed of two steps. The former locates the signal breaks which are caused by speakers changes. The latter determines and labels the utterances using a clustering algorithm.

The second method (as done in [START_REF] Wilcox | Audio indexing using speaker identification[END_REF] and [1]) uses an automatic speaker recognition (ASR) system. Breaks detections and clustering tasks are carried out simultaneously. The system has to determine speakers present within a given message as well as the utterances of each of them.

No a priori information on speakers is used in these two approaches, i.e. the speaker models have to be built during the process. Therefore these methods are well adapted to the tasks of blind segmentation.

In this article, we propose a system adapted from the second method for blind segmentation tasks. The conversation is modeled by a Markov Model (like [START_REF] Wilcox | Audio indexing using speaker identification[END_REF]). During the segmentation process, the Markov Model is expanded with each new detection of sound class.

The proposed system was tested on N-speaker segmentation task of NIST 2001 evaluation campaign [START_REF]The NIST Year 2001 Speaker Recognition Evaluation Plan[END_REF] which uses the CALLHOME database. The experiments in this paper are done using a half of this database to select the best fitting parameters for speaker models learning. The other half remains to validate the choice of those parameters.

Segmentation model

Structure of the segmentation model

The signal to segment consists in a sequence of observa-

tion vectors O = (o 1 , o 2 , ..., o T ).
The changes of sound classes are represented by a hidden Markov model (HMM). In this application the sound classes represent a speaker. Each HMM state characterize a class of sound and their transitions model the changes of classes.

The HMM λ is defined by (E, A, B) :

• Let E = {1, 2, ..., N } be a set of states.

• Let A = {a i,j } be a set of transition probabilities between the states.

• Let B be the set of {b i }. Let the state i be associated with a sound model C i of the sound class C i . Each state i is then associated with a set b i of emission probabilities according to C i . Let o t be an observation from O, b i (o t ) is the probability calculated from C i for o t .

The HMM is fully connected.

Transition probabilities are established according to a set of rules complying with the three following conditions:

           ∀i, a i,i = γ ∀(i, j), i = j, a i,j = 1-γ N -1 0 < γ < 1 (1)

Detection of sound classes and segmentation model building

The Segmentation model is generated by an iterative process, which detected and added a new state i at each stage (i.e. i). We refer to evolutive HMM as E-HMM [START_REF] Meignier | Evolutive HMM for Multi-Speaker Tracking System[END_REF]. At the process initialization stage i = 1 figure 2, HMM is

λ 1 = (E 1 , A 1 , B 1
). E 1 = {1} is composed of a single state "1", which is associated with a sound model C 1 ( 1 ) learned from the whole signal O. At the end of the initialization process, a first trivial segmentation s 1 = (s 1 1 , ..., s 1 T ) = (1, ..., 1) is generated. In fact, each observation o i is simply labeled with the only sound class C 1 . This segmentation s 1 is composed of a single segment which will be challenged at the following stages. The process (e.g. stage 2 & 3 figure 2) is divided in 4 steps (figure 1) for each stage i (i > 1) :

Step 1 A new state i is added to the set E i-1 (E i = E i-1 ∪ i).
Transition probabilities are adapted to take into account the new number of states. Then, the new HMM

λ i = (E i , A i , B i-1 ) is obtained.
Step 2 The sound model C i is estimated from a subset of observation 2 (o r , o r+1 , ..., o r+t ). r is selected such as :

       r = ArgMax j∈L   j+t k=j b 1 (o k )   L = {j ∈ {1, ..., T }|s i-1 j = s i-1 j+1 = ... = s i-1 j+t = 1}
(2) then, r ∈ L is the rank of the subset (o r , o r+1 , ..., o r+t ) maximizing the probabilities product for the sound class C 1 .

Moreover, the segmentation s

i is computed: the subset (o r , o r+1 , ..., o r+t ) is relabeled to the sound class C i .    s i j = s i-1 j ∀j / ∈ {r, ..., r + t} s i r = s i r+1 = ... = s i r+t = i (3)
1 C 1 is the sound model for the not yet detected speakers (i.e. all). 2 In this work, each subset has a 3 sec. duration (i.e. t is fixed).

Step 3 In this step, the process iteratively adapts the parameters of HMM λ i :

(a) For each k in {1, ..., i}, the sound model C k is adapted according to data which were affected to it in the segmentation s i .

(b) The set B i of emission probabilities are recomputed.

(c) Viterbi algorithm is applied to obtain a new version of segmentation s i according to the HMM.

The Viterbi path P (

s i |A i , B i , O) is com- puted. P (s i |A i , B i , O) = b s i 1 (o 1 )× T j=2 (a s i j-1 ,s i j ×b s i j (o j )) (4
) If a gain is observed between two loops in 3, the process returns to (a)

Step 4 Lastly, the stop criterion is assessed: if

P (s i |A i , B i , O) > P (s i-1 |A i , B i-1 , O) (5)
then a new stage starts back to step "1".

Note: the probability of s i-1 is reestimated with the transition A i of the model λ i , because topologies of segmentation models λ i and λ i-1 must be comparable.

Automatic Speaker recognition System

The sound models and emission probabilities are calculated by the AMIRAL ASR system developed at LIA [START_REF] Fredouille | AMIRAL: a block-segmental multi-recognizer approach for Automatic Speaker Recognition[END_REF]. Emission probabilities are computed on fixed-length blocks of 0.3 second. Each emission probability is normalized by the world model.

Acoustic parameterization (16 cepstral coefficients and 16 ∆-cepstral coefficients) is carried out using the SPRO module developed by the ELISA consortium3 [START_REF]Overview of the ELISA consortium research activities, Odyssey[END_REF].

The sound classes are modeled by gaussian mixture models (GMM) with 128 components and diagonal covariance matrices [START_REF] Reynolds | Speaker identification and verification using gaussian mixture speaker models[END_REF], adapted from a background model.

The sound model C is first estimated over a subsequence of 3 seconds (sec. 2.2 -Step 2). Then, the sound model C is adapted from the segments which are labeled by the sound class C (sec. 2.2 -Step 3a).

The adaptation scheme for training speaker models is based on the maximum a posteriori method (MAP). For each Gaussian g, Mean µ g of sound model C is a linear combination between the estimated µ g and the corresponding mean µ W g in background model W . Mean µ g is estimated according to the data of sound class C :

   µ g = αµ W g + (1 -α) µ g α > 0 (6)
Neither the weights, nor covariance matrices are adapted. The sound model C uses the weights and covariance matrices of the background model.

Experiments

The proposed approach was experimented on the Nspeakers segmentation task during NIST 2001 evaluation campaign [START_REF]The NIST Year 2001 Speaker Recognition Evaluation Plan[END_REF]. The results are shown in sec. 4.5. Moreover, development experiments on learning method are reported in sec. 4.4.

Databases

N-speakers evaluation corpus (described in [START_REF]The 2000 NIST Speaker Recognition Evaluation Plan[END_REF] and [START_REF]The NIST Year 2001 Speaker Recognition Evaluation Plan[END_REF]) is composed of 500 conversational speech tests drawn from CALLHOME corpus. Tests of varying length (< 10 minutes) are taken from 6 languages. The exact number of speakers is not provided (but is less than 10).

The NIST Corpus is divided in two parts of 250 tests named Dev and Eva.

NIST provided a separate development corpus (named train ch) composed of 48 conversational speech samples extracted from CALLHOME corpus. The train ch corpus permitted to learn the background model (wld ch).

A second separate development data set (train sb) is composed of 472 trials made of up to 100 speakers from Switchboard 2. train sb permitted to learn the background model wld sb.

Experiments

Experiments were carried out to estimate the influence of α parameter of MAP learning, applied on both wld ch and wld sb background models.

Moreover, a reference experiment based on a trivial segmentation (only one segment) named trivial is generated.

The results are obtained with parameters:

• The transition probabilities are estimated with γ = 0.6 (Eq. 1).

• The MAP parameters (Eq. 6) are:

α ∈ {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}

Scoring measures

Two evaluation measures are considered:

• The mean m diff of differences between the estimated number of speakers e n and the real number of speakers r n in the test n.

m diff = N n=1 e n -r n N (7)
• NIST speaker segmentation scoring is described in [START_REF]The NIST Year 2001 Speaker Recognition Evaluation Plan[END_REF]. Scoring is computed on the NIST reference segments with only one speaker speaking. This score corresponds to a segmentation error.

Results

In order to compare the influence of α parameter, figures 3 and 4 are shown respectively the segmentation scores and the mean m diff obtained with both background models. The best results are presented in tables 1 and 2 (respectively for Dev and Eva).

The results on Eva corpus is close to the results obtained on Dev corpus.

When the weight is close to 1, the result becomes equivalent to the trivial segmentation. The systems mainly attribute the data to only one class, besides the process does not add new speaker models (m diff is near -1.5).

When the weight is 0, the adaptation learning becomes equivalent to a EM-ML training with one iteration [START_REF] Dempster | Maximum likelihood from incomplete data via EM algorithm[END_REF], but initialized using the corresponding background model. This weight gives the best result (24.01%) for the wld sb background model. Although sufficient data is provided to compute background model, the train sb data is very different from the data of CALLHOME corpus (Dev and Eva).

The background model wld cl take advantage of the MAP learning method. The best result (25.5%) is obtained for an α = 0.3. Few data is used to learn this background model which is not efficient to generalize Dev and Eva data of CALLHOME corpus.

For both background models, the mean m diff of differences between the estimated number of speakers and the real number of speakers is quite good (near 0.5 for best results with wld ch and wld sb).

NIST 2001 results

Two systems were presented to NIST 2001 N-speaker segmentation. They use wld sb background model to adapt speaker models with an α parameter equal to 0. The difference between the system is the value of γ parameter used to compute HMM transition probabilities. In the first system LIA10, γ is equal to 0.6, In the second one, γ = 0.5. Note: α and γ parameter was estimated before NIST 2001 on train ch corpus according to the evaluation rules.

Tables 3 and4 show the results of LIA10 and LIA20 as well as the results of the trivial segmentation.

Tables 3 show the scores computed by the number of speakers present. Systems are well adapted for tests where a lot of speakers speak. The number of speaker are detected correctly. However the scores (22% and 24%) is close to the score of the trivial system (26%) for 2 speakers tests.

LIA10 and LIA20 scores is almost equal for the different speaker languages. The chosen learning method is well adapted when speaker language is unknown.

Summary

In this article, the segmentation system uses an evolutive HMM to model the conversation and to determine automatically the sound classes present in messages. The approach is based on an iterative algorithm which detects and adds the sound models one by one. At each stage, a segmentation is proposed, according to available knowledge. This segmentation is called into question at the following iteration until the optimal segmentation is reached.

Within sight of the results, the system behaves satisfactorily. Experiments showed that MAP training is well adapted for the selected task of segmentation. As for the weight between the background and the estimated sound model, it has a influence on the segmentation error.

Further work will focus on this two points, by adapting the background model data of CALLHOME to the SWITCHBOARD background model and by introducing an explicit duration model into the HMM to improve the speaker detection.

Figure 1 :

 1 Figure 1: Diagram of the segmentation process.

Figure 2 :

 2 Figure 2: Example of segmentation for a 2 speakers test.

Figure 3 :

 3 Figure 3: NIST N-speakers segmentation score (%): influence of α parameter.

Figure 4 :

 4 Figure 4: NIST N-speakers m diff : influence of α parameter on MAP.

Table 1 :

 1 Best results for the background models

	Corpus background model	α	score (%) m diff
	Dev	wld sb	0	24.01	0.58
	Dev	wld ch	0.3	25.50	0.71
	Corpus background model	α	score (%) m diff
	Eva	wld sb	0	23.42	0.54
	Eva	wld ch	0.3	25.17	0.44

Table 2 :

 2 Validation of the results obtain on Dev Corpus

ELISA consortium is composed of European research laboratories which work on a common platform. Members of ELISA for the participation to NIST 2001 is : ENST (France), IRISA (France), LIA (France), RMA (Belgium).