Nicolas Audibert

Anthony Larcher

Christophe Lévy

Juliette Kahn

Solange Rossato
email: solange.rossato@imag.fr

Driss Matrouf

Jean-François Bonastre

Jean-Francois Bonastre

LIA human-based system description for NIST HASR 2010

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

This paper describes the participation of the LIA laboratory to the Human Assisted Speaker Recognition (HASR) evaluation, which is part of the NIST-SRE 2010 campaign. The submission of the LIA for this task is based on a human decision. Samples were rated by three listeners, system decision being based on majority voting. Confidence scores were defined by mapping human decision to scores distribution of a SVM-based automatic system. This paper describes in section 3, the automatic system used for scores mapping is presented. In section 2 the algorithms used for listening stimuli generation and the protocol for samples listening and rating. Subsections 2.1 and 2.2 describe the algorithms used for automatic extracts selection from each model or test segment, and for extracts normalisation and concatenation. Subsection 2.3 describes the listeners involved and the listening protocol. Subsection 2.4 presents the calculations made on human decisions to obtain scores submitted to NIST. Finally, the characteristics of the submitted system are summarized and perspectives for future work are presented in section 5.

Human evaluation protocol

Extracts selection

In order to perform extracts selection, recordings are pre-processed by using Linear-Frequency Cepstral Coefficients (LFCC). Extraction of these parameters is described in Section 3.1. Once the LFCC parameters are computed, the energy coefficients are first normalised using a mean removal and variance normalisation in order to fit a 0-mean and 1-variance distribution. The energy component is then used to train a three component GMM, which aims at selecting informative frames. The frames carrying the highest level of energy are selected through the GMM and labeled speech. Once the speech segments of a signal are selected, a final process is applied in order to refine the speech segmentation:

1. Overlapping speech segments between both sides of a conversation are removed; 2. Morphological rules are applied on speech segments to discard segments too short to be used in a listening task, by adding or removing speech frames.

For each trial, 6 seconds-long extracts are automatically selected from the model and test segments and concatenated to build the audio stimulus presented to listeners. Extracts selection was guided by 2 criteria:

• Selected extracts should include a proportion as large as possible of speech frames;

• For both model and test segments, the extracts should include only speech frames corresponding to the interviewee, excluding speech turns of the interviewer audible in the channel of interest.

This selection is achieved by means of tools implemented in the MISTRAL/ALIZE [START_REF] Bonastre | ALIZE/SpkDet: a state-of-the-art open source software for speaker recognition[END_REF] toolkit.

Once the signal parameterization and speech frames detection described supra is performed, 6 seconds-long extracts are extracted from the channel of interest of the model and test segments. Interventions of the interviewer are excluded thanks to the speech frames redundancy across channels. For each original file (model and test segment), the initial threshold is set to a minimum of 70% speech frames not appearing in the other channel and iteratively decreased when necessary to end up with a minimum number of selected segments of 7 (i.e. a minimum total duration of 42 seconds for each file of a model/test pair). Although this method generally succeeds in selecting extracts that mainly include speech As a result, selection of appropriate 6 seconds-long extracts is performed manually by one of the listeners for these 3 files.

Rules for the generation of stimuli

Selected extracts are then combined in the audio stimulus, generated using the Praat software [START_REF] Boersma | Praat: Doing phonetics by computer (Version 5.0. 38)[END_REF]. Extracts from the model segment in the one hand and from the test segment in the other hand are chosen alternatively. A 1000 Hz, 50 milliseconds-long beep surrounded by two 75 milliseconds-long silent parts is inserted between consecutive segments to signal inter-extract switching. All extracts included in the generated stimulus are normalised to the same acoustic intensity. This level of normalised intensity ias set to 70 dB, and lowered when necessary (down to 66 dB for 4 model/test pairs) to avoid clipping.

Human evaluation participants and protocol

Three native French listeners (2 female aged 25 and 36, 1 male aged 31) with experience in phonetics and speech analysis, and without any known hearing impairment, evaluated independently the 150 stimuli generated. For each trial, they were requested to decide whether the extracts alternated in the stimulus had been uttered by the same speaker or not. Although this information was not directly used in submitted results, listeners were also requested to rate their confidence in this judgment in a 0-5 scale for further analyses.

Listeners evaluated the stimuli in a quiet environment using closed headphones. For stimuli comprising a large amount of noise (especially low-frequency noise) in either the model or test segment, they were given the possibility to band-filter the signal using the Praat software [START_REF] Boersma | Praat: Doing phonetics by computer (Version 5.0. 38)[END_REF] after visual inspection of the power spectrum, in order to reduce the perceptual heterogeneity caused by the difference of recording channels. When they considered a single listening as not sufficient for decision making, they were allowed to listen to the generated stimulus by selecting parts or as a whole as many times as necessary. Listeners took 12 to 180 seconds for decision making (mean: 66 seconds).

System scoring

For each trial, the decision of the human system submitted to NIST is defined by majority voting among the decisions taken by the three listeners. The confidence score submitted to NIST for HASR evaluation is defined from the level of agreement between listeners. In order to make possible comparisons between human performances and that of the SVMbased automatic system described in section 3, the inter-listener level of agreement is mapped to the impostor and client scores distribution obtained with this system and SRE 2008 data, as summarised in table 1.

An experiment is performed by running the SVM-based system (cf. Section 3) according to the NIST-SRE 08 short2-short3 protocol. In order to map the human decision on the automatic speaker recognition framework, mean and variance of both the client and impostor scores distributions are estimated. The mapping values presented in Table 1 are selected according to these score distributions. Table 1 also indicates the number of trials corresponding to each inter-listener level of agreement for each target speaker gender. Overall, the three listeners took the same decision on 51% of trials.

Description of the automatic system

The speaker recognition system chosen to determine the human decision scores is a classical GMM-SVM system (Gaussian Mixture Model -Support Vector Machine) using Latent Factor Analysis (LFA) [START_REF] Matrouf | A straightforward and efficient implementation of the factor analysis model for speaker verification[END_REF], [START_REF] Kenny | Factor analysis simplified[END_REF] for session variability modelling. This system is based on the open-source biometric platform MISTRAL/ALIZE [START_REF] Bonastre | ALIZE/SpkDet: a state-of-the-art open source software for speaker recognition[END_REF].

Front-end processing

Parameters extracted from speech signal (using the open source SPro toolkit [START_REF] Gravier | SPro: speech signal processing toolkit[END_REF]) are based on a filter-bank analysis (linear filter). Feature vectors are composed of 19 Linear-Frequency Cepstral Coefficients (20ms window, 10ms shift), its derivatives, the first 11 second derivatives and the delta energy. The frequency window is restricted to 300-3400 Hz. An energy labelling is performed on the signal and only the frames deemed to be speech are processed by the speaker recognition engine. Then simple feature normalisation is applied, so that the distribution of each cepstral coefficient is 0-mean and 1-variance for a given utterance.

World model

The UBM model size is set to 512 components (with diagonal covariance matrix). The UBM consists of a GMM trained on telephone conversations from the Fisher English database [START_REF] Cieri | The Fisher Corpus: a Resource for the Next Generations of Speech-to-Text[END_REF] and microphone recordings from the NIST-SRE 2005 database.

Speaker model using Factor Analysis

According to the Latent Factor Analysis (LFA) modelling [START_REF] Matrouf | A straightforward and efficient implementation of the factor analysis model for speaker verification[END_REF], speaker models are formed of three different components: a speaker and session independent background model, a speaker dependent and a session dependent components [START_REF] Vogt | Modelling session variability in text-independent speaker verification[END_REF], [START_REF] Kenny | Factor analysis simplified[END_REF]. The resulting model can be written as:

m (h,s) = m + Dy s + U x (h,s) (1)
where m (h,s) is the session-speaker dependent mean super-vector, D is S × S diagonal matrix (S is the dimension of the supervector), y s the speaker vector, U is the eigenchannel matrix of low rank R (a S × R matrix) and x (h,s) are the session factors. Both y s and x (h,s) are normally distributed among N (0, 1). D satisfies the following equation

I = τ D t Σ -1 D
where τ is the relevance factor required in the standard MAP adaptation.

SVM modelling

According to Equation 1, the Factor Analysis model estimates speaker supervectors normalised with respect to the session variability. A distance between GMMs is computed by using a probabilistic kernel K [START_REF] Campbell | Support Vector Machines Using GMM Supervectors for Speaker Verification[END_REF]. This distance, well suited for a SVM classifier when only mean parameters of the GMM models are adapted, is given by Equation 2for two sequences X s and X s respectively spoken by speakers s and s .

K(X s , X s ) = M g=1 ( √ α g Σ -1 2 g m g s ) t ( √ α g Σ -1 2 g m g s ) (2)
where m s is taken form the model in Equation 1(m s = m + Dy s ), and Σ g is the covariance matrix of the component g shared by the two models. The LIA SpkDet toolkit benefits from the LIB-SVM library [START_REF] Chang | LIBSVM: a library for support vector machines[END_REF] to estimate SVMs and classify instances. SVM are trained with an infinite (very large in practice) C parameter thus avoiding classification error on the training data (hard margin behaviour). The negative labelled examples are speakers form the normalisation cohort.

Automatic system performance

The system was developed on NIST SRE 2008 data. Performance of this system are reported in

Table 1 :

 1 Mapping of human decision and SVM-based automatic system scores, for each target speaker gender and each inter-listener level of agreement . frames corresponding to the interviewee, for 3 model/test pairs out of 150 (2 male target speakers, 1 female) it is unable to find in either the model or test segment extracts including enough useful information to make human decision possible.

	Listeners decision System decision	Confidence score calculation	Male score (N=36) Female score (N=114)
	3 false	Certain false	Average impostor score -2 σ imp	-2.45 (N=12)	-2.13 (N=33)
	2 false, 1 true	Uncertain false	Average impostor score	0.62 (N=8)	0.69 (N=20)
	1 false, 2 true	Uncertain true	Average client score	6.51 (N=7)	5.44 (N=39)
	3 true	Certain true	Average client score + 2 σ target	12.19 (N=9)	10.85 (N=22)

Table 2

 2

	for the 8 conditions

Table 2 :

 2 Performance (% EER and DCFmin) of the SVM system used for HASR score mapping.

Table 3 :

 3 4. Computation timeEach model segment is approximately 180 seconds-long, while each test segment is approximately 300 seconds-long. In addition to the time required by human processing presented in section 2.3, table 3 presents the computation time required for a trial by each step of the automatic processing of speech signals. The parameterization of speech signals, used both for listening stimuli generation and by the automatic system, is performed only once. Computation time mean per trial and standard deviation, for each automatic processing step. Mean computation time is also indicated as a multiple of real time.

	Automatic processing step	Mean time (seconds) σ (seconds) Time (xRT)
	Signal parametrization	3.70	0.08	0.008 xRT
	Listening stimulus building	0.65	0.14	0.001 xRT
	Automatic speaker verification	26.88	0.94	0.056 xRT

Conclusions

The results submitted for the HASR part of the HASR 2010 evaluation were based on majority voting by 3 listeners, after automatic selection of extracts of interests from the model and test segments and their concatenation in a listening stimulus for each of the 150 trials. Submitted confidence scores were obtained by mapping human decision to scores distribution obtained on SRE 2008 data with the SVM-based automatic system presented in section 3. Comparison of human vs. automatic system performances will be presented at the NIST SRE workshop, together with an analysis of human performances. Moreover, human performance analysis will be extended by using individual confidence scores and by evaluating differences between the model and test segments of each trial according to numerous perceptual dimensions, including channel differences, specific phonetic and prosodic features, and speakers affective states.