Pablo L Sordo Martínez

Benoît Fauve
email: benoit.fauve@validsoft.com

Anthony Larcher
email: alarcher@i2r.a-star.edu.sg

John S D Mason

SPEAKER VERIFICATION PERFORMANCE WITH CONSTRAINED DURATIONS

Keywords: Speaker verification, GMM/UBM, ivectors, LDA, PLDA, short duration

Over the last decade speaker recognition has witnessed significant advances, with successful developments in Factor Analysis (FA) and more recently i-vectors, more than halving the error rates achieved by the classical UBM/GMM approach. However when very short duration utterances are considered, it is known that these improvements are much less. This paper begins with a review of the recent developments of i-vector systems with a focus on short test duration, in the region of 10 s or less. Experimental results are then presented showing that error rates rise from approximately 5% to 18% when the test duration is systematically reduced from 30 s to just 3 s. Interestingly, with the 30 s condition the i-vector error rate is in the region of half that of the corresponding UBM/GMM system. Nevertheless, when the test segments are just 3 seconds duration then the error rates of the 2 systems systems are very similar. All experiments relate to the short-short condition of the NIST 2008 SRE, but with the test duration systematically reduced.

INTRODUCTION

The growth of interest for telephony based authentication in applications such as mobile banking [START_REF] Kong Aik Lee | Speaker verification makes its debut in smartphone[END_REF] has brought new challenges to speaker verification. User convenience invariably means that systems should operate with just a few seconds of speech, certainly in the authentication (testing) phase. This is in contrast to the majority of speaker recognition research which has tended to focus on much longer durations, although in recent times significant effort has been directed towards short duration testing, including the work of [START_REF] Fauve | Influence of task duration in textindependent speaker verification[END_REF][START_REF] Vogt | Factor Analysis Modelling for Speaker Verification with Short Utterances[END_REF].

I-vectors have become the state-of-the-art in speaker recognition since 2010. They have improved accuracy and robustness while simplifying the classification task, bringing it into a low-dimensional space. A considerable number of works tested the performance of i-vectors in short duration conditions [START_REF] Dehak | Front-End Factor Analysis for Speaker Verification[END_REF] and recently some improvements have been made [START_REF] Hasan | Duration mismatch compensation for ivector based speaker recognition systems[END_REF][START_REF] Kumar Sarkar | Study of the Effect of I-vector Modeling on Short and Mismatch Ut-terance Duration for Speaker Verification[END_REF]. Given all the existing works, limitation of the speech material is still a challenging constraint.

In this paper, we first present a brief review of some of the latest developments in i-vector based systems when evaluated under short duration conditions. We then present somewhat related results with on a direct comparison with a conventional UBM/GMM as a benchmark against one using ivectors. The main goal is to compare error rate trends of these arrangements as the quantity of speech data is reduced. Here we keep the enrolment (training) data constant and systematically reduce the amount of verification (test) data.

The paper is structured as follows. Section 2 presents a brief survey of recent developments involving i-vector schemes specifically encompassing short duration conditions while Section 3 follows with an account of the related strategies and techniques. The experimental configuration, in particular the test data shortening process, follows in Section 4, while experimental results are presented in Section 5. Finally, conclusions are drawn in Section 6.

I-VECTORS IN A SHORT DURATION CONTEXT

Since its appearance [START_REF] Dehak | Front-End Factor Analysis for Speaker Verification[END_REF], i-vector framework rapidly became the state-of-the-art in speaker recognition. Thanks to the intensive use of development data provided by the NIST and the Linguistic Data Consortium (LDC), i-vector systems outperformed most of the other configuration involved in the last two speaker recognition evaluations (2010 and 2012). As a consequence of this enormous support from these two institutions, i-vectors were initially only exploited in textindependent contexts [START_REF] Bousquet | Intersession Compensation and Scoring Methods in the i-vectors Space for Speaker Recognition[END_REF][START_REF] Senoussaoui | Mixture of PLDA Models in I-Vector Space for Gender-Independent Speaker Recognition[END_REF][START_REF] Jiang | PLDA Modeling in I-Vector and Supervector Space for Speaker Verification[END_REF].

Recently, i-vectors have proved to be efficient in some other areas such as language recognition [START_REF] Martínez | Language Recognition in iVectors Space[END_REF] and textdependent speaker verification [START_REF] Larcher | Phonetically-constrained PLDA modeling for textdependent speaker verification with multiple short utterances[END_REF][START_REF] Stafylakis | Text-dependent speaker recognition using PLDA with uncertainty propagation[END_REF]. Regarding the latter, the inclusion and development of new text-dependent databases [START_REF] Larcher | RSR2015: Database for Text-Dependent Speaker Verification using Multiple Pass-Phrases[END_REF] has led to a new framework where short du-ration is a requirement.

During time, the natural mainstream has been to evaluate the robustness of i-vectors against different sources of variability, among which duration mismatch was one of the primary options. Several works [START_REF] Kanagasundaram | PLDA based speaker recognition on short utterances[END_REF][START_REF] Kanagasundaram | i-vector based speaker recognition on short utterances[END_REF] proved that duration mismatch between enrolment and test data is less harmful than shortening both sets equally.

In the classical UBM/GMM configuration [START_REF] Reynolds | Speaker verification using Adapted Gaussian mixture models[END_REF] it has long been known [START_REF] Fauve | Tackling Variabilities in Speaker Verification with a Focus on Short Durations[END_REF] that, when the two speech segments under consideration are of meaningfully different durations then it proves beneficial to generate the GMM on the longer segment and carry out the UBM/GMM scoring on the shorter segment. Of course in the context of the i-vector approach, where two i-vectors are derived, one from test segment GMM and one from training segment GMM, then no such separation exists and the models presented to the classification process are structurally identical (and interchangeable) for the two segments. This leads to the challenging question of how to address the imbalance of data in the i-vector context when one speech segment is meaningfully different in duration to the second segment, see for example the recent work of [START_REF] Hasan | Duration mismatch compensation for ivector based speaker recognition systems[END_REF].

SPEAKER VERIFICATION SYSTEMS

In the experimental work presented here speaker verification scores for two i-vectors configurations are compared directly with those derived from a conventional UBM/GMM. The latter is now well understood [START_REF] Reynolds | Speaker verification using Adapted Gaussian mixture models[END_REF][START_REF] Bimbot | A tutorial on text-independent speaker verification[END_REF] with research and application applied over almost two decades. Thus here we address the much more recent i-vector approach which is then to be directly compared with the UBM/GMM when tested against different speech durations.

The i-vectors framework

The i-vector paradigm has been motivated by inconsistencies found in the Joint Factor Analysis (JFA) [START_REF] Kenny | Joint factor analysis of speaker and session variability: Theory and algorithms[END_REF] framework. Indeed, it was shown in [START_REF] Dehak | Front-End Factor Analysis for Speaker Verification[END_REF] the the assumption of speaker and session variabilities laying in different subspaces is not true and that the session subspace contains information on the speaker identity.

In this paradigm, it is assumed that a speech segment can be represented by a single vector, the i-vector, in a lowdimensional space referred to as total variability space [START_REF] Dehak | Front-End Factor Analysis for Speaker Verification[END_REF]. Then, a GMM super-vector can be decomposed as:

m h,s = µ + T • w h,s (1)
where µ is the speaker-and-session-independent component, i.e. the mean super-vector coming of the UBM. A basis of the total-variability subspace is given by the rows of T, which is a rectangular matrix of low rank and w h,s is a vector normally distributed with parameters N (0, I). Extracting an ivector w h,s , is essentially a Maximum a-Posteriori adaptation (MAP) in the subspace defined by T.

In this paper, a pooled total-variability approach is utilized for convenience, as considered by McLaren and van Leeuwen [START_REF] Mclaren | Improved speaker recognition when using i-vectors from multiple speech sources[END_REF]. Therefore, all available training speech has been compiled into a dataset regardless its source (telephone, microphone or interview).

Linear Discriminant Analysis

Linear Discriminant Analysis (LDA), was used in [START_REF] Dehak | Front-End Factor Analysis for Speaker Verification[END_REF] to minimize the intra-class variance and maximize the between-class variance in the total variability space. These techniques attempt to project i-vectors onto a new set of orthogonal axes, so that those which belong to the same speaker lay into the same region, and apart from others. This problem is defined according to the Rayleigh coefficient:

J(v) = v t • S b • v v t • S w • v (2)
where v represents a space direction, S b is the between-class variance and S w is the within-class variance. Therefore, J(v) is proportional to the quality of the LDA performance. To calculate the variances:

S b = S s=1 (w s -w) • (w s -w) t (3)
S w = S s=1 1 n s ns i=1 (w s i -w s ) • (w s i -w s ) t (4)
where w s is the centroid of the class s, i.e. the mean of the ivectors of each speaker, S is the number of speakers involved, and n s is the number of sessions per speaker s. LDA seeks a projection matrix which consists in the eigenvectors whose eigenvalues are the highest from the general equation:

S b • v = λ • S w • v (5)

Mahalanobis distance scoring

The Mahalanobis distance originates from the Euclidean metric concept and has been proved to outperform the classical cosine distance [START_REF] Larcher | ALIZE 3.0open source toolkit for state-of-the-art speaker recognition[END_REF]. As explained by Bousquet et al. [START_REF] Bousquet | Intersession Compensation and Scoring Methods in the i-vectors Space for Speaker Recognition[END_REF],

given a new observation w representing an i-vector, the goal of a statistical classifier is to identify to which class it belongs. Assuming equality of class covariances and Gaussian conditional density models, an i-vector w is assigned to that particular class which minimizes:

(w -w s ) t • W -1 • (w -w s ) = w -w s 2 W -1 (6)
where W is the within class covariance matrix. w s represents the same as in [START_REF] Dehak | Front-End Factor Analysis for Speaker Verification[END_REF]. Note that this score is proportional to the log -probability that w belongs to the class s. Therefore, the Mahalanobis metric between two i-vectors w 1 and w 2 is:

score(w 1 , w 2 ) = -w 1 -w 2 2 W -1 (7)
with W being the within-class covariance matrix of any class of interest. In this work, the Mahalanobis metric is used after LDA.

Probabilistic Linear Discriminant Analysis

The aim of the approach is to define a set of factors which directly model session and speaker variability in that subspace. The generative model [START_REF] Kenny | Bayesian Speaker Verification with Heavy-Tailed Priors[END_REF] is:

w r = w + U 1 • x 1 + U 2 • x 2r + r (8)
where w r is a feature vector with r = 1, ..., R, being R the number of recordings of a speaker; U 1 is the eigenvoice matrix and U 2 is the eigenchannel matrix. x 1 , x 2r and r are respectively the speaker, channel and residual factors.

SYSTEM CONFIGURATIONS AND EXPERIMENTS

Data and evaluation protocol

Unlike previous editions, NIST SRE 2012 [START_REF]The NIST year 2012 speaker Recognition evaluation plan[END_REF] core task involved duration mismatch [START_REF] Hasan | Duration mismatch compensation for ivector based speaker recognition systems[END_REF]. This increasing interest has led us to choose data which include a more constrained task in terms of duration.

Here we have taken data from a previous NIST evaluation (2008) and the "short-short" condition for which the speech durations are typically 180 s. The shortened test segments are obtained by systematically and successively utilising a lower and lower percentage of the original test segment. Follow the procedure proposed in [START_REF] Fauve | Influence of task duration in textindependent speaker verification[END_REF][START_REF] Fauve | Improving the performance of text-independent short duration SVMand GMM-based speaker verification[END_REF], which takes portions of active speech. This keeps an x% of frames from the original test excerpts, being 100% the actual NIST 2008 SRE short2 -short3 condition. The mean duration and mode are shown in the Table 1 and the distributions of the utterances' length for each subset are presented in the Figure 1. The standard deviation is in all cases approximately just over 30% of the mode value.

Performance was evaluated using the equal error rate (EER) and the minimum decision cost function (minDCF) with the values proposed by NIST SRE 2008, i.e. C miss = 10, C F A = 1 and P target = 0.01, and for the evaluation condition DET6. Evaluation involved for 12,511 trials employing 1,270 enrolled speakers and 2,528 test segments.

System configuration

The baseline configuration used for all experiments utilizes an energy-based voice activity detector (VAD) [START_REF] Fauve | Influence of task duration in textindependent speaker verification[END_REF][START_REF] Fauve | Improving the performance of text-independent short duration SVMand GMM-based speaker verification[END_REF] with 19 dimensional feature-warped linear frequency cepstral coefficients (LFCC) and appended delta [START_REF] Kenny | Joint factor analysis of speaker and session variability: Theory and algorithms[END_REF], double delta [START_REF] Larcher | Phonetically-constrained PLDA modeling for textdependent speaker verification with multiple short utterances[END_REF] and the delta energy coefficients, for an overall sum of 50. A gender-dependent UBM of 512 Gaussians has been used, trained on NIST 2004 SRE corpus, specifically 219 male speakers. The total variability matrix and the PLDA esti- In order to show the sensitivity of the configurations to short duration excerpts, the number of remaining dimensions after LDA has been varied from 50 to 400 in steps of 50. The number of eigenvoices for PLDA has been changed in the same manner while the number of eigenchannels have fixed to 400 throughout. For both, LDA and PLDA, 3 iterations of Eigen Factor Radial (EFR) [START_REF] Bousquet | Variance-Spectra based Normalization for I-vector Standard and Probabilistic Linear Discriminant Analysis[END_REF] have been applied for i-vector normalization. Each of these iterations is equivalent to length normalization [START_REF] Garcia | Analysis of i-vector length normalization in speaker recognition systems[END_REF].

PERFORMANCE COMPARISONS

Figure 2 shows results in the form of DET plots for the 3 systems, namely the UBM/GMM and the 2 i-vector systems, LDA and PLDA. Each plot has 6 profiles reflecting the different durations of the test segments, from 2% upto 100%, which is the original recording in full. The immediate difference in the three plots is the bunching of the profiles in the case of the UBM/GMM. This claims that the initial performance on the full original segments is far superior for both the LDA and Fig. 2: DET plots for three systems: (a) a standard UBM/GMM providing a benchmark (b) an i-vector system and LDA, and (c) an i-vector system and PLDA. In all 3 cases the profiles relate to data reduction in the test segment only, from 100% (original duration) down to 2% of the original. Note the profiles for the 2% condition are similar in all 3 cases, with an EER just below 20%

PLDA i-vector systems.

A second observation is that the upper profile in each case (the one representing just 2% of the original test data) is similar in all 3 cases at just below 20%. So the performance of these three systems in terms of EER, certainly tends to converge when just 2 to 3 seconds are available at the test stage. In contrast, for the full duration (1 to 2 minutes of speech), the performances of the i-vector systems are far better than that of the UBM/GMM, with EER's of approximately 5% for the latter 2 compared with 10% for the UBM/GMM. These duration performances are shown more clearly in Figure 3, with the convergence of the i-vector schemes with the UBM/GMM at 2.5 seconds well illustrated. Following we consider variations in the i-vector parameters, with particular attention to the shortest duration performance, specifically to see if there are any simple gains to be made in this area.

Figure 4 shows a series of profiles for the i-vector LDA and PLDA configurations where each one reflect variations in i-vector subspace dimension, from 50 up to 400. For PLDA, dimension of the speaker factor (represented by x 1 in equation 8) is taken as a parameter. In the case of LDA, the variable to be changed is the number of eigenvectors which form the projection matrix, eq. 5. The profiles show minimal variations, certainly for the higher dimensions, 150 up to 400. This changes at 5% profile and is quite marked at the shortest duration of 2%. Here, in the case of LDA the error rates fall slightly and in the case of PLDA increase. This is perhaps an interesting trend, worthy of further investigation.

Finally, Figure 5 compares both i-vector systems, when only 2% of test data remains. The parameter involved from each scheme is the same as the one varied in the previous figure, respectively. We can observe a knee point in the middle of both graphs, which marks a sustainable change. Considering the profiles, it can be stated that LDA shows a more consistent performance than PLDA, which deteriorates abruptly when only a few dimensions are present.

CONCLUSIONS

The work described here focuses on short duration segments in the test stage of speaker recognition; it assumes well trained models are available from adequate quantities of enrolment speech.

In this paper we have compared the behaviour or two state-of-the-art i-vector systems and a classical GMM/UBM engine when varying the speech duration. Experimental results confirm the fact that i-vector based systems are indeed markedly better than the UBM/GMM, when sufficient test data are available. Here for instance, the first scores 5% EER against the UBM/GMM of 10% EER with full length utter-ances. However, such a huge improvement decreases consistently when short duration utterances are involved. In this regard, it is observed a knee point in the region of 10 sec. to 20 sec. and when data is reduced further to the region of 2 sec. to 3 sec., the performance of the two type of systems (i-vector system and UBM/GMM) converge to give very similar scores just below 20% EER. Furthermore, if we consider the most restrictive test condition represented by the blue solid line on 2a, it is remarkable that UBM/GMM gives a performance not very far from the i-vectors one. This observation has led to consider that i-vector does not bring the same improvement for all durations and hereby, it is more sensitive to the lack of data than the UBM/GMM. In the same way, LDA has proved to be more consistent than PLDA when very short utterances are evaluated and dimension reduction is carried out in both approaches.

Fig. 1 :

 1 Fig. 1: Distributions of the given utterance sets after shortening in 8 stages down to approximately 2% of original durations. mation involved utterances from NIST-SRE 2004, 2005 and 2006 as well as Switchboard II, Phases II and III and Switchboard Cellular, Parts I and II. In total, 1,289 male speakers and 16,969 sessions have been used. The rank of the total variability matrix (i.e. the dimension of the i-vectors) has been set to 400.In order to show the sensitivity of the configurations to short duration excerpts, the number of remaining dimensions after LDA has been varied from 50 to 400 in steps of 50. The number of eigenvoices for PLDA has been changed in the same manner while the number of eigenchannels have fixed to 400 throughout. For both, LDA and PLDA, 3 iterations of Eigen Factor Radial (EFR)[START_REF] Bousquet | Variance-Spectra based Normalization for I-vector Standard and Probabilistic Linear Discriminant Analysis[END_REF] have been applied for i-vector normalization. Each of these iterations is equivalent to length normalization[START_REF] Garcia | Analysis of i-vector length normalization in speaker recognition systems[END_REF].

Fig. 5 :

 5 Fig. 5: Comparison of LDA and PLDA systems at 2% of active speech. Performances vary according to the values of the space dimension after LDA and the number of eigenvoices in PLDA

Table 1 :

 1 The percentage of speech frames from the original and the mean duration of the utterances involved.

			Percentage of		Utterances
			remaining frames mode duration
			2%		2.14 s
			5%		5.12 s
			10%		10.3 s
			20%		20.8 s
			30%		30.6 s
			40%		40.7 s
			50%		50.8 s
			75%		74.5 s
			100%		99.9 s
		1
	relative frequency	0.2 0.4 0.6 0.8
		0 0	50	100	150	200
				seconds

ACKNOWLEDGEMENTS

The results in this paper have been produced thanks to the ALIZE/SpkDETopen source toolkit [21], and in collaboration with the University of Avignon (LIA), France.


R Saeidi

K A Lee

T Kinnunen

T Hasan

B Fauve

P. -M Bousquet

E Khoury

P L Sordo Martinez

J M K Kua

C H You

H Sun

A Larcher

P Rajan

V Hautamäki

C Hanilci

B Braithwaite

R Gonzales-Hautamäki

S O Sadjadi

G Liu

H Boril

N Shokouhi

D Matrouf

L El Shafey

P Mowlaee

J Epps

T Thiruvaran

D A Van Leeuwen

B Ma

H Li

J H L Hansen

J. -F Bonastre

S Marcel

J Mason

E Ambikairajah

I4U submission to NIST SRE 2012: A large-scale collaborative effort for noise-robust speaker verification

Keywords: Speaker Verification, NIST SRE 2012, I4U, ivector

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I4U submission to NIST SRE 2012:

A large-scale collaborative effort for noise-robust speaker verification

Introduction

The I4U submission to National Institute of Standards and Technology (NIST) speaker recognition evaluation 2012 (SRE '12) [START_REF]NIST speaker recognition evaluation[END_REF] is a result of active exchange of information between the coalition participants across nine institutions. The name of the institutes and corresponding system identifiers are provided in Table 1. The submitted results are based on the fusion of multiple classifiers. The optimization of the component classifiers and the fusion device were done with development sets jointly designed within the I4U coalition with multiple design iterations, refinement of noise adding protocol and various other details. Different from previous SREs, the task of SRE'12 involves:

Handling noisy test segments: This required speech enhancement algorithms and employing mixed training or parallel model combination techniques.

Imbalanced multi-session training:

There are tens of segments available for training some speaker models while only a single segment for some other speakers. Open-set identification: SRE'12 evaluation protocol allows the use of knowledge of all target speakers in each detection trials which resulted in utilizing compound log-likelihood ratio. This paper is organized as follows: In Section 2, we present the strategies taken to make a development set coping with SRE'12 new conditions. Details of the submitted systems and the component classifiers, together with the strategies to deal with the new challenges listed above are described in Section 3. One of the motivations underlying the I4U coalition is to experiment with the fusion of large numbers of sub-systems. Results for the individual and the fused system are presented in Section 4.

Development sets

The development sets were generated to help I4U team members in developing their speaker recognition systems considering the special conditions in SRE'12 including multiple segments training for a speaker 1 . All the members of I4U coalition helped in refining the lists with respect to detecting empty or otherwise problematic segments with conflicts in gender and speaker PIN (there are issues with pre-SRE'12 lists like multiple-genders or wrong genders for some speakers). The latest lists from NIST were utilized and speech segments for all 1918 target speakers were fetched from SRE'06, SRE'08 and SRE'10 corpora and corresponding meta-data were extracted. To be able to assess both the recognition systems' generalization and calibration performance, separate development (DEV) and evaluation (EVAL) sets were created. The number of segments, speakers and trials for each set are given in Table 2. In designing these sets, the following criteria were considered:

• Test segments are disjoint for DEV-test and EVAL-test.

• Most of the train segments in DEV-train are added to EVALtrain. The number of train segments in EVAL-train is almost twice the number of segments in DEV-train. This design choice is made to evaluate the systems performance under the condition that speaker and channel spaces are already trained but the number of enrollment segments for target speaker modeling has increased.

• The segments from train to test have all different LDC-IDs to avoid testing against same session from training.

• Two disjoint sets of speakers from SRE'06 data that do not appear in SRE'12 are added to DEV-test and EVAL-test to form unknown non-target trials.

• For those speakers having telephone and microphone data, both types of channels were included in the train set so that systems could benefit from having different channels in training.

• Considering noisy segments inclusion in NIST SRE'12, for every original NIST segment, two noisy versions were generated. Noise adding was carried out using FaNT 3 . We have used ten noise segments for each HVAC (heating, ventilation and air-conditioning) and crowd noise type. Noise signals used to contaminate the speech segments were different from train to test and from DEV to EVAL. Noises are added at two SNR-levels 6dB and 15dB. The mean measured SNR-levels were 40dB, 15dB and 10dB for original, 15dB and 6dB segments, respectively 4 . Since there are two noisy versions of each clean segment being utilized in DEV and EVAL sets, the performance of the developed systems are optimized to perform well under noisy condition rather than clean ("not altered") condition.

Recognition systems

The systems developed in the I4U coalition were based on state-of-the-art: 1) i-vector system [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF] with probabilistic linear discriminant analysis (PLDA) [START_REF] Prince | Probabilistic linear discriminant analysis for inferences about identity[END_REF] modeling, or 2) Gaussian supervector representation and joint factor analysis (JFA) [START_REF] Kenny | A study of inter-speaker variability in speaker verification[END_REF][START_REF] Kenny | Speaker and session variability in GMM-based speaker verification[END_REF], or support vector machine (SVM) modeling. All 16 kHz audio data were down-sampled to 8 kHz to match to the existing 8 kHz background data. Energy-based speech activity detection (SAD) was applied to telephone segments, while for interview segments a dual-channel SAD is employed. The automatic speech recognition (ASR) transcripts from NIST for interview segments in SRE'08 and SRE'10 were used to refine 3 http://dnt.kr.hsnr.de/download.html 4 The SNR is measured using stnr tool from NIST the SAD labels. All of the systems are gender-dependent. The components and data usage of sub-systems are presented in Tables 3, 4 and 5 for features, transform and classifier, respectively. Sys1: Validsoft's i-vector system uses spectral subtraction to enhance energy profile for SAD. Test i-vectors are scored against all target segment i-vectors followed by score averaging.

Sys2: Swansea's i-vectors are normalized with eigenfactors radial (EFR) method [START_REF] Bousquet | Variance-spectra based normalization for i-vector standard and probabilistic linear discriminant analysis[END_REF] utilizing total covariance matrix of the background data. LDA-reduced 200-dimensional ivectors are averaged for each target speaker and used with Mahalanobis scoring.

Sys3: LIA's system uses two fused sub-subsystems. The first uses LDA reduction preceded by iterative i-vector normalization according to the covariance matrix [START_REF] Bousquet | Variance-spectra based normalization for i-vector standard and probabilistic linear discriminant analysis[END_REF] and twocovariance scoring; the second uses PLDA preceded by spherical nuisance normalization with within-class covariance matrix [START_REF] Bousquet | Variance-spectra based normalization for i-vector standard and probabilistic linear discriminant analysis[END_REF]. Score is computing as a) the average score of the test ivector against all target i-vectors and b) an equal-weights combination of these scores according to multiple PLDA subspace dimensions (from 50 to 400 in steps of 50).

Sys 4: RUN's i-vector PLDA system uses dynamic noise suppression within a Wiener filter applied both for speech enhancement and SAD. Noise estimation uses improved minima controlled recursive averaging (IMCRA) [START_REF] Cohen | Noise spectrum estimation in adverse environments: improved minima controlled recursive averaging[END_REF][START_REF] Martin | Noise power spectral density estimation based on optimal smoothing and minimum statistics[END_REF] which averages the previous estimate of the noise power spectra and has proven robust against input SNR and different noise types due to rapid noise tracking. The noise power spectral density estimate is used for decision-directed a-priori SNR estimation, which further defines a Wiener filter applied for magnitude enhancement.

Sys5-10: The CRSS's i-vector systems use combinations of two different front-ends and three back-ends [START_REF] Liu | An investigation on back-end for speaker recognition in multi-session enrollment[END_REF][START_REF] Hasan | CRSS Systems for 2012 NIST Speaker Recognition Evaluation[END_REF]. Gaussianized cosine-distance scoring (GCDS) and a discriminative back-end using L2-regularized logistic regression (using LIB-LINEAR [START_REF] Fan | LIBLINEAR: A library for large linear classification[END_REF]) are used. The enrollment i-vectors are averaged and then Gaussianized using mean and variance of devset. LDA dimensionality reduction and cosine scoring are used.

Sys11: UEF contributed the overall fusion component for I4U [START_REF] Hautamäki | Variational Bayes logistic regression as regularized fusion for nist sre 2010[END_REF][START_REF] Hautamaki | Sparse classifier fusion for speaker verification[END_REF] and developed a robust utterance-adaptive SAD [START_REF] Kinnunen | A practical, self-adaptive voice activity detector for speaker verification with noisy telephone and microphone data[END_REF] Switchboard II Phase 2 and 3, Switchboard cellular part 1 and 2, Fis: Fisher, -D: diagonal covariance, -F: full covariance, TV: total variability [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF], NAP: nuisance attribute projection [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF], ISV: inter session variability [START_REF] Wallace | Inter-session variability modelling and joint factor analysis for face authentication[END_REF]. are obtained from reliable frames with the help of aggressive spectral oversubtraction. The recognizer is a standard i-vector PLDA system and, unlike most of the other I4U system, does not use multicondition training.

Sys12 by I2R whitens the first-order sufficient statistics using UBM covariances, which speeds up estimation of the posterior distribution during the total variability matrix (T-matrix) training and i-vector extraction [START_REF] Kenny | A small foot-print i-vector extractor[END_REF]. T-matrix estimation uses two subspaces, T tel and Tmic, where T tel is trained from telephone data and Tmic from microphone data following decoupled method on [START_REF] Senoussaoui | An i-vector extractor suitable for speaker recognition with both microphone and telephone speech[END_REF]. This enables easy control of the dimensionality of the subspaces in T = [T tel , Tmic] and avoids the problem of data type imbalance encountered when all data are pooled for T-matrix training in one go. For details of the PLDA implementation, refer to [START_REF] Jiang | PLDA modeling in i-vector and supervector space for speaker verification[END_REF].

Sys 13 by I2R is a GMM supervector system with KL divergence kernel [START_REF] Campbell | Support vector machines using GMM supervectors for speaker verification[END_REF]. Utterance GMM is obtained via MAP adaptation of the UBM means that are concatenated and normalized by the UBM standard deviation and square root of the mixture weights. Nuisance attribute projection (NAP) [START_REF] Campbell | SVM based speaker verification using a GMM supervector kernel and NAP variability compensation[END_REF] and tz-norm are applied for channel and score normalization, respectively.

Sys 14 is an anti-model variant of Sys 13. The use of other target speakers is allowed in SRE'12 which leads to an openset identification problem. The anti-model approach of [START_REF] Matejka | Use of antimodels to further improve state-of-the-art PRLM language recognition system[END_REF] is adopted for increased discrimination between target and unseen non-targets. SVM for each target speaker is trained using the supervectors of the other target speakers as the SVM background together with additional data drawn from SRE'04 for the unseen speakers.

Sys 15 is a Bhattacharyya-kernel GMM-SVM system with data-dependent relevance factor [START_REF] You | GMM-SVM kernel with a bhattacharyya-based distance for speaker recognition[END_REF][START_REF] You | A GMM-supervector approach to language recognition with adaptive relevance factor[END_REF] and zt-norm. Sys 16, in turn, uses joint factor analysis (JFA) implementation for I2R's SRE'10 submission [START_REF] Ma | IIR system description for the 2010 nist speaker recognition evaluation submission[END_REF]. It is composed of 300 speaker factors, 200 channel factors (100 for telephone, 50 for microphone, 50 for interview), and full rank diagonal matrix. For eigenchannel training, the tel, mic and interview channels were separately trained and concatenated into an eigenchannel matrix. Enrollment and scoring (with zt-norm) are as in Sys 15.

Sys 17: IDIAP's system is a single classifier with intersession variability (ISV) modeling technique of [START_REF] Wallace | Inter-session variability modelling and joint factor analysis for face authentication[END_REF]. It is implemented using Bob 6 , an open-source signal processing and machine learning toolbox. ISV is similar to JFA with linear scoring approximation [START_REF] Glembek | Comparison of scoring methods used in speaker recognition with joint factor analysis[END_REF] but with merged eigen-voice and -channel spaces. Scores are normalized using zt-norm.

System performance

We analyze and compare system performance on the core task of NIST SRE'12 using the equal error rate (EER) and primary cost. The notion of EER is commonly known. What is new in SRE'12 is the use of the so-called primary cost Cprimary, defined as the average cost at two specific points on the DET curve. At either of these points, the detection cost function (DCF) is defined in normalized form (such that the maximum value is one), as follows

CNorm(θ) = Pmiss(θ) + 1-Ptar
Ptar ×

[P fa (θ|known)+P f a (θ|unknown)] 2

. Here, Ptar is the a priori probability that a trial is a target trial, while Pmiss(θ) and P fa (θ) are, respectively, the probability of miss and false alarm at threshold θ. Notice that P fa (θ) consists of two components computed separately from the known and unknown non-target trials. Now, let θA be the threshold which gives the CNorm(θA) with Ptar = 0.01 and θB be the threshold which gives the CNorm(θB) with Ptar = 0.001, the primary detection cost is defined as the average cost between the points on the detection error trade-off (DET) curve, as follows

Cprimary = C Norm (θ A )+C Norm (θ B ) 2
Table 6 shows the absolute performance of all 17 systems and their fusion for common conditions 1 and 3 as defined in SRE'12. One obvious point to note here is that, the PLDA ivector systems give consistently better performance in terms of EER and minimum Cprimary when the test signal is collected over clean (CC1) and noisy (CC3) interview sessions. It is also obvious that, the GMM-SVM (Sys 13, 14, and 15) and JFA (Sys 16) give equally good performance compared to, and for some instances better than i-vector based systems.

The fusion of large ensemble of recognition systems was by itself a challenging issue, for instance, over-fitting may easily degrade the performance. We followed the recent work in [START_REF] Hautamäki | Variational Bayes logistic regression as regularized fusion for nist sre 2010[END_REF][START_REF] Hautamaki | Sparse classifier fusion for speaker verification[END_REF] whereby fusion weights are trained using regularization to avoid over fitting. Different regularizers were systematically evaluated, and ridge regression (L2-norm regularization) was chosen. Instead of cross-validating the regularization factor λ, we decided to use a simple Bayesian method that allows automatic selection of λ, as described in [START_REF] Foo | A majorization minimization algorithm for (multiple) hyperparameter learning[END_REF]. This method integrates out λ and the resulting non-convex optimization problem is solved via majorization-minimization approach. Convergence was assumed after two iterations. The fusion results are shown in Table 6 with Fusion1-3. Though effective on our DEV set, the ridge-regression regularization (Fusion1 and 2) does not always give improved performance over the single best system, Figure 1: Analysis of excluding one system at a time in fusion using Focal and employing compound log-likelihood ratio [START_REF] Van Leeuwen | Knowing the non-target speakers: The effect of the i-vector population for PLDA training in speaker recognition[END_REF] for P known = 0.5. Using the full ensemble of classifiers results in actual Cprimary of 0.3959 and 0.2836 for first two common conditions (CC1 and CC2) respectively in SRE'12 for the pooled scores of males and females. A positive relative change indicates increased actual Cprimary by excluding a system in fusion resulting in fusion performance drop. Systems number 2 and 3 are not considered for this analysis. while the original FoCal7 fusion (Fusion3) does. One possible insight that we might draw here is that regularization might hamper effective training of fusion parameters when the development data is sufficient. This is a point for future research. The results for Fusion1 are slightly inferior to Fusion2 because of some mis-labeled scores during the evaluation which are corrected for post-evaluation (Fusion2). An analysis of individual systems importance in fusion is provided in Fig. 1. Comparing between interview (CC1) and telephone (CC2) conditions in Fig. 1, the most influensive systems in fusion are not the same across different conditions.

Conclusion

This paper provides an overview of fusion of 17 systems submitted to NIST SRE'12 by different sites in I4U coalition. The collaboration of over 30 researchers within the coalition benefited all the sites in preparing robust speaker recognition systems. It is hard to compare the individual subsystems and determine the strengths of each system but in a very general prospective, the systems that utilized more recent features and employ speech enhancement in the front-end were more successful. Averaging the enrollment i-vectors gave about the same performance as averaging the scores of i-vectors. Discriminative training schemes, such as SVMs, using a proper distance kernel on Gaussian supervector representation was found to outperform generative i-vector representation with PLDA classification. The new paradigm shift in NIST SRE'12 is expected to emphasize the discriminative training in modeling and even i-vector representation.

  This work was partly supported by Academy of Finland (proj. no 253120 and 132129), Swiss National Science Foundation under the LOBI project, contract no. SNSF-235 and European Community's Seventh Framework Program (FP7/2007-2013) under grant agreement no. 238803.

Table 1 :

 1 I4U Coalition and assigned system indexes

	Site	System index
	ValidSoft Ltd (VLD)	Sys1
	Swansea University (UWS)	Sys2
	University of Avignon (LIA)	Sys3
	Radboud University Nijmegen (RUN)	Sys4
	University of Texas at Dallas (CRSS)	Sys5-10
	University of Eastern Finland (UEF)	Sys11
	Institute for Infocomm Research (IIR)	Sys12-16
	Idiap Research Institute (IDIAP)	Sys17

Table 2 :

 2 Number of speakers, speech segments and trials in the development sets.

		Number of speakers	Number of segments		Number of trials
		DEV	EVAL	DEV	EVAL		DEV	EVAL
		Train Test Train Test Train Test	Train Test	True	False	True	False
	Males	680 868 763 804 16941 19866 29961 21837 14589 13494291 15483 16646148
	Females 1039 1243 1155 1102 24693 25980 43119 28548 19863 26973357 20763 32952177

Table 3 :

 3 Feature extraction setup for the systems in I4U, CMVN: cepstral mean and variance normalization, RFCC: repartitioned frequency cepstral coefficients[START_REF] Boril | Data-driven design of front-end filter bank for Lombard speech recognition[END_REF], MHEC: mean Hilbert envelope coefficients[START_REF] Sadjadi | Hilbert envelope based features for robust speaker identification under reverberant mismatched conditions[END_REF].

		Features	SAD	Speech enhancement Features post-processing
	Sys1-3	19 LFCCs + ∆ + ∆E + first 11 ∆∆	Energy-based	Spectral subtraction	CMVN
	Sys4	19 MFCCs + E + ∆ + ∆∆	Energy-based [4]	Wiener filtering[5]	Feature warping [6]
	Sys5-7	12 MHEC + logE + ∆ + ∆∆	Voicing feature [7]	-	RASTALP [8] + CMVN
	Sys8-10	12 RFCC + c0 + ∆ + ∆∆	Statistical SAD [9]	-	Feature warping
	Sys11	18 MFCCs + ∆ + ∆∆	Adaptive SAD [10]	-	RASTA + CMVN
	sys12-14	18 MFCCs + logE + ∆ + ∆∆	Energy-based	Qualcomm-ICSI-OGI 2	RASTA + CMVN
	sys15-16 19 LPCCs + ∆ + ∆∆ + 12 MFCCs	Energy-based	Qualcomm-ICSI-OGI	RASTA + CMVN
	Sys17	19 MFCCs + logE + ∆ + ∆∆	Energy-based	Qualcomm-ICSI-OGI	CMVN

Table 4 :

 4 5 where 16-component speech and non-speech codebooks are trained from 12 MFCCs including c0. Training labels Transform details for sub-systems in I4U. Numbers in data columns are standing for corresponding NIST SRE corpus, SW:

Table 5 :

 5 Classifier details for i-vector based systems in I4U: Lnorm, Length normalization[START_REF] Garcia-Romero | Analysis of i-vector length normalization in speaker recognition systems[END_REF], EFR: eigen-factors radial normalization[START_REF] Bousquet | Variance-spectra based normalization for i-vector standard and probabilistic linear discriminant analysis[END_REF], <IV> and <scores>: average over i-vectors or scores in multi-session training.

		Background	IV pre-processing	scoring	#Voice #Channel Scoring strategy
		data for IV
		processing
	Sys1	DEV train	EFR (W)	PLDA	300	50	<scores>
	Sys2	DEV train	EFR (C), LDA(300)	Mahanabolis	-	-	<IV>
	Sys3	DEV train	i. EFR (C), LDA (50 to 400) ii. EFR (W) i. 2Cov ii. PLDA 50 to 400	400	<IV>
	Sys4	DEV train	LDA(200), centering, WCCN, Lnorm	PLDA	200	50	<IV>
	Sys5 and Sys8	04, 05, 06, SW,	LDA(400), centering, Lnorm	PLDA	400	400	<IV>
		DEV train
	Sys6 and Sys9	04, 05, 06, SW,	LDA(400), Gaussianization, Lnorm	Cosine	-	-	<IV>
		DEV train
	Sys7 and Sys 10 04, 05, 06, SW,	L2-regularized [18]	linear regression	-	-	<IV>
		DEV train
	Sys11	04, 06, 08, 10	-	PLDA	200	0	<IV>
		and SW
	Sys12	04, 06, 08, 10,	LDA(400)	PLDA	200	50	<IV> and Snorm
		SW, MIXER5
		and DEV train

Table 6 :

 6 Analysis of system performance based on equal error rate (EER) and minimum Cprimary (minC) for P known = 0. NIST SRE'12 common conditions include multi-session in train and specific channel in test; CC1: interview and CC3: added noise interview. Fusion:1) Auto Ridge[START_REF] Hautamäki | Variational Bayes logistic regression as regularized fusion for nist sre 2010[END_REF] submitted to SRE'12 as I4U submission 2) Auto Ridge post evaluation 3) FoCal post evaluation.

		Males		Females
		CC1	CC3	CC1	CC3
		EER minC EER minC EER minC EER minC
	Sys1	5.55 0.2674 4.22 0.4154 4.26 0.1674 4.07 0.5600
	Sys2	5.44 0.2633 4.27 0.4246 4.77 0.1950 4.27 0.5663
	Sys3	12.10 0.4998 10.90 0.5579 11.50 0.4363 10.50 0.5498
	Sys4	5.75 0.2670 4.83 0.3741 4.86 0.1580 4.09 0.3018
	Sys5	4.73 0.2669 4.14 0.3635 4.53 0.1373 3.52 0.3072
	Sys6	4.28 0.2168 3.79 0.3053 4.05 0.1118 3.43 0.2420
	Sys7	9.71 0.4742 9.32 0.6071 5.81 0.3083 5.18 0.3840
	Sys8	4.81 0.3051 4.28 0.3918 4.65 0.1167 3.27 0.3094
	Sys9	4.86 0.2374 4.22 0.2894 4.15 0.0948 3.38 0.2346
	Sys10	9.84 0.4251 9.61 0.5714 5.56 0.1635 5.13 0.4124
	Sys11	13.30 0.5276 9.72 0.6316 12.60 0.3985 7.48 0.5316
	Sys12	3.74 0.2765 3.29 0.3322 4.01 0.2290 3.62 0.3877
	Sys13	4.77 0.4440 4.50 0.3587 4.36 0.3055 3.35 0.2470
	Sys14	5.45 0.3474 5.74 0.3618 4.52 0.1351 3.52 0.1591
	Sys15	4.85 0.3347 5.57 0.3751 4.55 0.1708 4.10 0.2188
	Sys16	3.78 0.2333 5.56 0.3415 5.17 0.1906 5.31 0.4245
	Sys17	9.03 0.4932 7.88 0.4302 8.48 0.4189 5.66 0.3797
	Fusion1 3.62 0.2306 3.25 0.3162 3.96 0.1196 2.81 0.2470
	Fusion2 3.38 0.2267 3.75 0.3075 4.06 0.0760 2.92 0.2140
	Fusion3 3.48 0.2020 2.67 0.2767 3.87 0.0719 2.78 0.2277

The lists are available via http://cls.ru.nl/ ˜saeidi/ file_library/I4U.tgz

SAD available at http://cs.uef.fi/pages/tkinnu/VQVAD/VQVAD.zip

http://idiap.github.com/bob/

http://niko.brummer.googlepages.com/focal


Anthony Larcher
email: alarcher@i2r.a-star.edu.sg

Jean-François Bonastre

Benoît Fauve

Aik Kong

Christophe Lee

Haizhou Levy

John Li

Jean-Yves Mason

Parfait

Jean-François Bonastre

Benoît Fauve

Levy

Alize

Jean-Francois Bonastre

Benoit Fauve

Kong Aik Lee

Christophe Lévy

Haizhou Li

John S D Mason

Jean-Yves Parfait

ALIZE 3.0-Open Source Toolkit for State-of-the-Art Speaker Recognition

Keywords: speaker recognition, open-source platform, ivector

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

As indicated by the number of applications developed recently, speech technologies have now reached a level of performance that makes them attractive for distributed and embedded applications. Following this trend, NIST speaker recognition evaluations (SREs) have seen their number of participants increase significantly since the first edition. These campaigns clearly illustrate the substantial improvements in performance that have been achieved in the last few years. Speaker verification systems have benefited from a number of developments in noise and channel robustness [START_REF] Solomonoff | Advances in channel compensation for svm speaker recognition[END_REF][START_REF] Matrouf | A straightforward and efficient implementation of the factor analysis model for speaker verification[END_REF][START_REF] Bousquet | Variance-Spectra based Normalization for I-vector Standard and Probabilistic Linear Discriminant Analysis[END_REF] and new paradigms such as Joint Factor Analysis [START_REF] Kenny | Joint factor analysis versus eigenchannels in speaker recognition[END_REF] and i-vectors [START_REF] Dehak | Language Recognition via Ivectors and Dimensionality Reduction[END_REF]. At the same time, techniques developed in the field of speaker verification have been shown to be useful for other areas [START_REF] Fredouille | Back-and-forth methodology for objective voice quality assessment: from/to expert knowledge to/from automatic classification of dysphonia[END_REF][START_REF] Li | Spoken language recognition: from fundamentals to practice[END_REF][START_REF] Vaquero | Intra-session variability compensation and a hypothesis generation and selection strategy for speaker segmentation[END_REF].

State-of-the-art techniques are now based on intensive use of corpus and computational resources [START_REF] Glembeck | Simplification and optimization of I-Vector extraction[END_REF][START_REF] Cumani | Memory and Computation Trade-Offs for Efficient I-Vector Extraction[END_REF]. The continual improvement in performance calls for enormous number of trials to maintain confidence in the results. For instance, the number of trials from the core task of NIST-SRE evaluation has increased from about 24,000 in 2006 to more than 1.88 millions in 2012 (or 88 millions for the extended task) and participation to such an event has become a true engineering challenge. The rapidly growing effort needed to keep up-to-date with state-ofthe-art performance has strongly motivated an increasing number of collaborations between sites. However, system development often remains a challenge and large scale implementation is resource consuming. In this context, collaborative opensource software offers a viable solution as it can be used to reduce the individual development effort and offer a baseline system implementation [START_REF] Li | TechWare: Speaker and Spoken Language Recognition Resources[END_REF].

The ALIZE project has been initiated in 2004 by the University of Avignon LIA within the ELISA consortium [START_REF] Magrin-Chagnolleau | Overview of the 2000-2001 ELISA consortium research activities[END_REF] with the aim to create an open-source C++ library for speaker recognition. Since then, many research institutes and companies have contributed to the toolkit through research projects or by sharing source code. More recently the development has been supported by the BioSpeak project, part of the EU-funded Eurostar/Eureka program 1 . Based on the ALIZE core library, high level functionalities dedicated to speaker recognition are available through the LIA RAL package. All the code from the toolkit is distributed through open source software licenses (LPGL) and has been tested on different platform including Windows, Linux and Mac-OS.

Recent developments include Joint Factor Analysis [START_REF] Kenny | Joint factor analysis versus eigenchannels in speaker recognition[END_REF], ivector modelling and Probabilistic Linear Discriminant Analysis [START_REF] Prince | Probabilistic linear discriminant analysis for inferences about identity[END_REF]. These developments stem mainly from a collaboration between LIA and the Institute for Infocomm Research (I2 R).

This paper presents an overview of the ALIZE toolkit. Section 2 gives a description of the collaborative tools and details about the toolkit implementation. In Section 3 we describe the main functions available in the LIA RAL package. Section 4 presents the performance of i-vector systems based on ALIZE for the NIST-SRE 2010 extended task. Finally, Section 5 discusses the future evolution of the project.

ALIZE: an Open Source Platform

A Community of Users

A number of tools are available for dissemination, exchange and collaborative work through a web portal 2 . To federate the community, this portal collects and publishes scientific work and industrial realisations related to ALIZE. The users can register to the mailing list that allows them to be informed of the latest developments and to share their experience with the community. A LinkedIn group also provides a way to know about the facilities and the people working in the field of speaker recognition.

Documentation, wiki and tutorials are available on the website to get started with the different parts of the toolkit. The official release of the toolkit can be downloaded from the website and the latest version of the sources are available through a SVN server. quickly run regression tests in order to increase the reliability of future releases and to make the code easier to maintain. Test cases include low level unit tests on the core ALIZE and the most important algorithmic classes as well as an integration test level on the high-level executable tools. Doxygen documentation is available on line and can be compiled from the sources.

Source Code

ALIZE Toolkit

The platform includes a Visual Studio solution and autotools for compilation under Windows and UNIX-like platforms. A large part of the LIA RAL functions use parallel processing for speed The multi-thread implementation based on the standard POSIX library is fully compatible with the most common platforms. The LIA RAL library can be linked to the well known Lapack3 library to give high accuracy in matrix manipulations.

All sources are available under LPGL licence that impose minimal restriction on the redistribution of the software.

High Level Executables

The goal of this paper is to show the steps to set-up a state-of-the art biometric engine using the different components of ALIZE. For more details, excellent tutorials on speaker recognition can be found in the literature [START_REF] Campbell | Speaker recognition: a tutorial[END_REF][START_REF] Bimbot | A tutorial on text-independent speaker verification[END_REF][START_REF] Kinnunen | An overview of text-independent speaker recognition: From features to supervectors[END_REF].

Figure 1 shows the general architecture of a speaker recognition engine. LIA RAL high level toolkit provides a number of executables that can be used to achieve the different functions depicted in this diagram. The rest of this section gives an overview of the main functionnalities of the LIA RAL toolkit with the corresponding executables.

Front-End

The first stage of a speaker recognition engine consists of a feature extraction module that transforms the raw signal into a sequence of low dimension feature vectors. ALIZE interfaces to features generated by SPRO 4 and HTK 5 and also accepts "raw" format.

Once the acoustic features have been generated, they can be normalised to remove the contribution of slowly varying convolutive noises (mean subtraction) as well as reduced (variance normalization) by using the function NormFeat. Low energy frames, corresponding to silence and noise, can then be discarded with EnergyDetector. This executable computes a bi-or tri-Gaussian model of the feature vector or of the log-energy distribution and selects the features belonging to "highest" mean distribution. Finally, it is possible to smooth a selection of feature vectors. Applied to a single channel recording, LabelFusion smooths the selection of frames with a morphological window.

When applied to a two-channel recording, LabelFusion removes overlapping sections of high energy features.

Enrolment

In speaker recognition, the enrolment module generates a statistical model from one or several sequences of features. Although it is possible to generate one model for each recording session, depending on the system's architecture, it is common to consider a single model to represent a speaker. By extension, we refer to this as the speaker model for the remainder of the paper.

State-of-the-art speaker recognition engines are mainly based on three types of speaker models. Although these three models are all related to a Gaussian mixture model (GMM), we distinguish between a first first type of model that explicitly makes use of a GMM and the two other types that represent the speaker or session as a fixed-length vector derived from a GMM. This vector can be a super-vector [START_REF] Campbell | SVM based speaker verification using a GMM supervector kernel and NAP variability compensation[END_REF] (concatenation of the mean parameters of the GMM) or a more compact representation known as an i-vector [START_REF] Dehak | Language Recognition via Ivectors and Dimensionality Reduction[END_REF]. Each of the three models can be obtained from the corresponding executable of the LIA RAL toolkit.

Robustness to inter-session variability, that could be due to channel or noise, is one of the main issue in speaker recognition. Therefore, ALIZE includes the most common solutions to this challenge for each type of model described below.

TrainTarget generates GMM models given one or more feature sequences. The GMMs can be adapted from a universal background model (UBM) M by using a maximum a posteriori (MAP) criterion [START_REF] Gauvain | Maximum a posteriori estimation for multivariate gaussian mixture observations of Markov chains[END_REF][START_REF] Reynolds | Robust Text-Independent Speaker Identification Using Gaussian Mixture Speaker Models[END_REF] or a maximum likelihood linear regression (MLLR) [START_REF] Leggetter | Maximum likelihood linear regression for speaker adaptation of continuous density hidden markov models[END_REF]. Noise and channel robust representations can be obtained by using Factor-Analysis (FA) based approaches. Factor Analysis for speaker recognition has been introduced in [START_REF] Kenny | Disentangling speaker and channel effects in speaker verification[END_REF] and assumes that the variability due to the speaker and channel both lie in distinct low dimension subspaces. Different flavours of FA have been proposed and two are available in ALIZE. The more general one is known as Joint Factor Analysis [START_REF] Kenny | Joint factor analysis versus eigenchannels in speaker recognition[END_REF], in which the super-vector m (s,n) of the session and speaker dependent GMM is a sum of three terms given by Eq.1.

m (s,n) = M + Vy (s) + Ux (s,n) + Dz (s) (1)
In this formulation, V and U are "factor loaded matrices", D is a diagonal MAP matrix while y (s) and x (s,n) are respectively called speaker and channel factors. y (s) , x (s,n) and z (s) are assumed to be independent and have standard normal distributions. A simplified version of this model, often referred to as EigenChannel or Latent Factor Analysis [START_REF] Matrouf | A straightforward and efficient implementation of the factor analysis model for speaker verification[END_REF] is also available in the TrainTarget. In this version, the simplified genera-tive equation becomes:

m (s,n) = M + Ux (s,n) + Dz (s) (2)
ModeToSv extracts super-vectors from GMMs. A supervector is the representation of a speaker in a high dimension space that has been popularised by the development of Support Vector Machines (SVM) for speaker recognition [START_REF] Campbell | SVM based speaker verification using a GMM supervector kernel and NAP variability compensation[END_REF]. LibSVM library [START_REF] Chang | LIBSVM: A library for support vector machines[END_REF] has been integrated into the ALIZE SVM executable. Nuisance Attribute Projection (NAP) [START_REF] Solomonoff | Advances in channel compensation for svm speaker recognition[END_REF] aims to attenuate the channel variability in the super-vector space by rejecting a subspace that contains most of the channel variability (nuisance effect). The most straightforward approach consists of estimating the within-session co-variance matrix W of a set of speakers and to compute the projection m(s,n) of super-vector m (s,n) such that:

m(s,n) = (I -SS t )m (s,n) (3)
where S contains the first eigenvectors resulting from the singular value decomposition of W .

IvExtractor extracts a low-dimensional i-vector [START_REF] Dehak | Language Recognition via Ivectors and Dimensionality Reduction[END_REF] from a sequence of feature vectors. An i-vectors is generated according to

m (s,n) = M + Tw (s,n) ( 4
)
where T is a low rank rectangular matrix called the Total Variability matrix and the i-vector w (s,n) is the probabilistic projection of the super-vector m (s,n) onto the Total Variability space, defined by the columns of T. Many normalization techniques have been proposed to compensate for the session variability into the Total Variability space. The IvNorm executable can be used to apply normalizations based on the Eigen Factor Radial (EFR) method [START_REF] Bousquet | Intersession compensation and scoring methods in the i-vectors space for speaker recognition[END_REF]. EFR iteratively modifies the distribution of i-vectors such that it becomes standard normal and the i-vectors have a unitary norm. Given a development set T of i-vectors, of mean µ and total co-variance matrix Σ, an i-vectors is modified according to:

w ← Σ 1 2 (w -µ) |Σ 1 2 (w -µ)| (5)
After this transformation has been applied to all i-vectors from the development set T and from the test data, the mean, µ, and co-variance matrix, Σ, are re-estimated to perform the next iteration. Note that the length-norm proposed in [START_REF] Garcia-Romero | Analysis of i-vector length normalization in speaker recognition systems[END_REF] is equivalent to one iteration of the EFR algorithm.

A variation of the EFR, proposed later in [START_REF] Bousquet | Variance-Spectra based Normalization for I-vector Standard and Probabilistic Linear Discriminant Analysis[END_REF] as Spherical Nuisance Normalization (sphNorm), is also available in the AL-IZE toolkit. For sphNorm, the total co-variance matrix Σ is replaced by the within class co-variance matrix of the development set T . After normalization of their norm, all i-vectors lie on a sphere and it is therefore difficult to estimate a relevant within-class co-variance matrix. Spherical Nuisance Normalization is then used to project the i-vectors onto a spherical surface while assuring that there is no principal direction for the session variability.

Other standard techniques such as Within Class Covariance Normalization (WCCN) and Linear Discriminant Analysis (LDA) [START_REF] Dehak | Language Recognition via Ivectors and Dimensionality Reduction[END_REF] are also available in ALIZE.

Pattern Matching

Given a test utterance, X , and a target speaker model, the matching module returns a score that reflects the confidence of the system in X being spoken by the target speaker. The nature and computation of this score vary depending on the type of speaker model and the different assumptions made. Similarly for the enrolment module, LIA RAL includes three executables, each dedicated to a specific type of model.

ComputeTest, given a sequence of acoustic features, X = {xt}t∈T of length T , computes a log-likelihood ratio between the UBM and a speaker dependent GMM. If no channel compensation method is applied, the log-likelihood of utterance X over a model s is computed as the average log-likelihood of features xt such that:

log P (X |s) = T t=1 log C c=1 γcN (xt, µ c , Σc) ( 6
)
where C is the number of distribution in s and γc, µ c and Σc are the weight, mean and co-variance matrix of the c th distribution respectively.

For the case of Joint Factor Analysis where it is difficult to integrate out the channel effect, ComputeTest can compute two approximations of the log-likelihood. The first one is adapted directly from [START_REF] Kenny | Joint factor analysis of speaker and session variability: Theory and algorithms[END_REF] and uses a MAP point estimate for the channel factor and the second is the linear scoring proposed in [START_REF] Glembek | Comparison of Scoring Methods used in Speaker Recognition with Joint Factor Analysis[END_REF]. A detailed description of both approaches can be found in [START_REF] Glembek | Comparison of Scoring Methods used in Speaker Recognition with Joint Factor Analysis[END_REF].

SVM returns a score that reflects the distance of a test supervector to the hyper-plan defined by the classifier. Different kernels such as GLDS [START_REF] Campbell | Support vector machines for speaker and language recognition[END_REF] or GSL derived from the Kullback-Liebler divergence [START_REF] Campbell | SVM based speaker verification using a GMM supervector kernel and NAP variability compensation[END_REF] are available through the LibSVM library.

IvTest is dedicated to i-vector comparison. The i-vector paradigm offers an attractive low-dimensional representation of speech segments that enables standard classification techniques to be applied for speaker recognition. Amongst the most popular scoring methods, four have been implemented in IvTest: cosine [START_REF] Dehak | Language Recognition via Ivectors and Dimensionality Reduction[END_REF] and Mahalanobis [START_REF] Bousquet | Intersession compensation and scoring methods in the i-vectors space for speaker recognition[END_REF] scoring, two-co-variance scoring (2cov) [START_REF] Brummer | The speaker partitioning problem[END_REF] as well as two versions of the Probabilistic Linear Discriminant Analysis (PLDA) scoring [START_REF] Prince | Probabilistic linear discriminant analysis for inferences about identity[END_REF].

In the remainder of this section, W , B and µ are respectively the within-and between-class co-variance matrices and the mean of a large set of i-vector.

Cosine similarity has been proposed in [START_REF] Dehak | Language Recognition via Ivectors and Dimensionality Reduction[END_REF] to compute the similarity between two i-vectors w1 and w2. In the same paper, the authors compensate the session variability through Within Class Co-variance Normalization (WCCN) and Linear Discriminant Analysis (LDA). Considering that Υ is the Cholesky decomposition of the within-class co-variance matrix W calculated over a large data set and that Λ is the LDA matrix computed on the same data set, the cosine similarity score is given by:

S(w1, w2) = < Υ t Λ t w1|Υ t Λ t w2 > ||Υ t Λ t w1|| ||Υ t Λ t w2|| (7)
Mahalanobis distance is a generalisation of the Euclidian distance for the case where the data are not following a standard normal distribution. The Mahalanobis score is given by: S(w1, w2) = (w1 -w2) t W -1 (w1 -w2)

The two-co-variance model, described in [START_REF] Brummer | The speaker partitioning problem[END_REF], can be seen as a special case of the PLDA. It consists of a simple linear-Gaussian generative model in which an i-vector w can be decomposed as w = y s + where the speaker and noise components y s and follow respectively normal distributions given by P (y s ) = N (µ, B) and P ( |y s ) = N (y s , W ). The resulting score can be expressed as:

s = N (w1|y, W ) N (w2|y, W ) N (y|µ, B) dy i=1,2
N (wi|y, W ) N (y|µ, B) dy

PLDA [START_REF] Prince | Probabilistic linear discriminant analysis for inferences about identity[END_REF] is one of the most recent addition to the AL-IZE toolkit. The generative model of PLDA considers that an i-vector w is a sum of three terms:

w (s,n) = µ + Fφ (s) + Gψ (s,n) + ( 10
)
where F and G are low rank speaker and channel "factor loaded matrices". is a normally distributed additive noise of full covariance matrix. ALIZE implementation of the PLDA follows the work of [START_REF] Jiang | PLDA Modeling in I-vector and Supervector Space for Speaker Verification[END_REF]. Two scoring methods, described by Figure 2, have been implemented. The first is based on the native PLDA scoring while the second one is using the mean of the L enrolment i-vectors, w. Note that both methods allow multiple enrolment sessions. More details can be found in [START_REF] Larcher | Phonetically-Constrained PLDA Modeling for Text-Dependent Speaker Verification with Multiple Short Utterances[END_REF] and in a companion paper [START_REF] Lee | Multisession PLDA Scoring of I-vector for Partially Open-set Speaker Detection[END_REF].

Background Modelling

Speaker recognition is a data-driven technology and all approaches implemented in ALIZE rely on a background knowledge learned from a large quantity of development data. Estimation of the knowledge component is computationally intense. Efficient tools have been developed in the toolkit to optimize and simplify the development efforts. The UBM can be trained efficiently by using the TrainWorld that uses a random selection of features to speed up the iterative learning process based on the EM algorithm. Meta-parameters of JFA and LFA models can be trained by using EigenVoice, EigenChannel and EstimateDmatrix while the TotalVariability has been especially optimised to deal with the computational constraints of learning the Total Variability matrix for i-vector extraction. The implementation follows the work described in [START_REF] Kenny | Disentangling speaker and channel effects in speaker verification[END_REF] with additional minimum divergence described in [START_REF] Brümmer | The em algorithm and minimum divergence[END_REF]. Nuisance Attribute Projection matrices can be trained using CovIntra and, for i-vector systems, normalization and PLDA meta-parameters can be trained by respectively using PLDA and IvNorm. PLDA estimation follows the algorithm described in [START_REF] Jiang | PLDA Modeling in I-vector and Supervector Space for Speaker Verification[END_REF].

Performance of i-Vector Systems

This section presents the performance of different i-vector systems based on the ALIZE toolkit on the Condition 5 of the NIST-SRE'10 extended task for male speakers [START_REF]Speaker recognition evaluation plan[END_REF]. 50 dimension MFCC vectors are used as input features (19 MFCC, 19∆, 11∆∆ and ∆E). High energy frames are retained and normalized so that the distribution of each cepstral coefficient is 0mean and 1-variance for a given utterance. A 2048-distribution UBM with diagonal co-variance matrix is trained on 6,687 male sessions from NIST-SRE 04, 05 and 06 telephone and microphone data. The same data augmented with Fisher and Switchboard databases (28,524 sessions) are used to train a Total Variability matrix of rank 500. All meta-parameters required for ivector normalization and scoring are estimated from 710 speakers from NIST-SRE 04,05 and 06 with a total of 11,177 sessions. Rank of the F and G matrices from the PLDA model are set to 400 and 0 respectively. When applied, two iterations of Eigen Factor Radial and 3 iterations of Spherical Nuisance Normalization are performed.

Figure 3: Performance of ALIZE i-vector systems on NIST-SRE10 extended male tel-tel task (condition 5) given in terms of (% EER, minDCF2010).

Figure 3 shows the performance of seven systems using different i-vector normalization and scoring functions. The performance of these systems are consistent with the current stateof-the-art considering that simple acoustic features have been used.

Discussion

We have described ALIZE, an open-source speaker recognition toolkit. This toolkit includes most of the standard algorithms recently developed in the field of speaker recognition, including Joint Factor Analysis, i-vector modelling and Probabilistic Linear Discriminant Analysis. The aim of ALIZE collaborative project is to pool development efforts and to make efficient implementation of standard algorithms available for the community. In the future, efforts will be concentrated on the documentation of the toolkit through online help and tutorials.

  ALIZE software architecture is based on UML modelling and strict code conventions in order to facilitate collaborative development and maintenance of the code. An open-source and cross-platform test suite enables ALIZEs contributors to

Figure 1 :

 1 Figure 1: General structure of a speaker verification system.

Figure 2 :

 2 Figure 2: Graphical model of the two PLDA scoring implementations in ALIZE for L enrolment i-vectors.

3. 5 .

 5 Score NormalizationDifferent combinations of score normalization based on Z-and T-norm are available through ComputeNorm,[START_REF] Li | Normalizations and selection of speech segments for speaker recognition scoring[END_REF][START_REF] Auckenthaler | Score Normalization for Text-Independent Speaker Verification System[END_REF].

http://www.eurekanetwork.org/activities/eurostars

http://alize.univ-avignon.fr

http://www.netlib.org/lapack/

http://www.irisa.fr/metiss/guig/spro/

http://htk.eng.cam.ac.uk/


Ville Hautamäki

Kong Aik Lee

David Van Leeuwen

Rahim Saeidi

Anthony Larcher

Tomi Kinnunen

Taufiq Hasan

Seyed Omid Sadjadi

Gang Liu

Hynek Boril

John H L Hansen

Benoit Fauve

Automatic Regularization of Cross-entropy Cost for Speaker Recognition Fusion

In this paper we study automatic regularization techniques for the fusion of automatic speaker recognition systems. Parameter regularization could dramatically reduce the fusion training time. In addition, there will not be any need for splitting the development set into different folds for cross-validation. We utilize majorization-minimization approach to automatic ridge regression learning and design a similar way to learn LASSO regularization parameter automatically. By experiments we show improvement in using automatic regularization.

Introduction

Score level fusion of multiple sub-systems has been the most effective way of obtaining very high accuracy in speaker verification. This trend is observed in the NIST SRE evaluations for many years now. These systems utilize multiple classifiers in order to produce the final score on whether the person speaking in the test segment is the hypothesized speaker [START_REF] Kinnunen | An overview of text-independent speaker recognition: from features to supervectors[END_REF]. The goal is to select a small set of base classifiers that are believed to be complementary in order to improve the discriminiation ability of the whole ensemble. To improve the generalization ability of the ensemble, regularization is a commonly used approach [START_REF] Hautamäki | Sparse classifier fusion for speaker verification[END_REF].

A regularizer imposes an additional constraint on the fusion weight optimization problem, where a regularization parameter defines the maximum norm of fusion weights [START_REF] Hautamäki | Regularized logistic regression fusion for speaker verification[END_REF]. Then unconstrained optimization task is turned into a constraint optimization task, where role of the constraint is to avoid fusion training to overfit on the training data. In the Lagrangian formulation, the role of the constraint is played by the well known Langrange multiplier, λ. The meaning of the λ is that the larger it is, the smaller search space is placed on the optimizer. Constraint is origin centric. So regularization is commonly called shrinkage, as it tends to shrink the weight coefficients towards zero.

The functional form of the regularizer also plays a role in overfit avoidance, as it defines the shape of the constraint region. The most common is the Ridge regularizer, which is just quadratic function of the norm (l2), thus constraint is an origin centric hypersphere [START_REF] Hastie | The Elements of Statistical Learning -Data Mining, Inference, and Prediction[END_REF]. Bigger λ will then shrink all classifier weights fairly equally. Least absolute shrinkage (LASSO) [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] is a regularizer, where l1 norm leads to a origin centric diamod as a constraint. LASSO is so called sparsity promoting regu- larizer, because if the corner of the diamond is optimum, then some of the weights will turn out to be be exactly zero.

Regularized fusion could be carried out in a two-stage process. First, a range of possible λ values are selected, then the weights are estimated for all these selected values. In this regard, a fusion training set is split into two parts for weight optimization and cross-validation estimation, respectively. If the training set is large, as was the case in NIST SRE 2012 [START_REF]The NIST year 2012 speaker recognition evaluation plan[END_REF], this requires a large computational effort [START_REF] Figueiredo | Adaptive sparseness for supervised learning[END_REF].

From the Bayesian perspective, we see that regularized fusion training is actually the maximum a posteriori (MAP) estimation of the fusion parameters consisting of the weight vector w and bias b [START_REF] Figueiredo | Adaptive sparseness for supervised learning[END_REF]. The regularizer is then the prior and the λ is the hyper-parameter of the prior distribution. An improvement to the frequentist interpretation is observed by using an "integrating out" approach to estimate the regularization parameters [START_REF] Figueiredo | Adaptive sparseness for supervised learning[END_REF], for instance, via variational Bayes. There, the goal is to estimate the full posterior, while integrating out everything else [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]. Hyperparameters, are iteratively re-estimated, with no need for cross-validation. In [START_REF] Hautamäki | Variational Bayes logistic regression as regularized fusion for nist sre 2010[END_REF], it was noticed that in speaker verification fusion, variational Bayes provides stable results over different ensemble sizes, but the fused score is not well calibrated as the synthetic prior cannot be easily included into the optimization cost, in contrast to cross-entropy where it is an additive term.

In case of using sparsity promoting priors, such as l1, few Bayesian approaches have been reported in the literature, where the λ is integrated out. In [START_REF] Figueiredo | Adaptive sparseness for supervised learning[END_REF], hierarchial Bayesian model was derived and non-informative and improper Jeffreys prior was placed in the last stage. An expectation maximization (EM) algorithm was specifically derived for this purpose. In [START_REF] Cawley | Sparse multinomial logistic regression via Bayesian l1 regularization[END_REF], Jeffreys prior was placed directly to the parameter λ, resulting in a very tractable integral, but the resulting solution is less flexible than [START_REF] Figueiredo | Adaptive sparseness for supervised learning[END_REF]. The integration by [START_REF] Cawley | Sparse multinomial logistic regression via Bayesian l1 regularization[END_REF] results in an estimate that λ is the number of non-zero weights. In the case of [START_REF] Cawley | Sparse multinomial logistic regression via Bayesian l1 regularization[END_REF], the hyperparameter is not estimated from the observed data.

In this work we apply the method for Bayesian ridge regression [START_REF] Foo | A majorization-minimization algorithm for (multiple) hyperparamtere learning[END_REF] to regularized classifier fusion training. In that method λ is integrated out, but still, the selected method, the majorization minimization lends to a possibility to use as any tool that finds regularized optimum of the cross-entropy cost. In addition, the method gives an estimate of the λ based on the training data. In this work we find the closed form solution to l1 regularized optimization cost, where λ has been integrated out. We also attempt to optimize the weights using the majorization minimization scheme. Additional benefit of such a scheme is that not only is computational time reduced1 , but no training set splitting is needed and all data can be used for training.

Regularized cross-entropy

Given a development set of trials D = {(sn, yn), n = 1, 2, . . . , N dev } containing N dev score vectors from L base classifiers, we are interested to find a linear model w t s + b that minimizes errors on the development set and generalizes well to an unseen corpus. Here, w = (w1, w2, . . . , wL) t are the weights to be applied to the L base classifier scores, b is the bias term added to the fused scores, and the class label y is defined to take values 0 and +1 for non-target and target trial, respectively.

For any class-conditional densities that follow exponential family and share the dispersion parameter [START_REF] Jordan | Why the logistic function? a tutorial discussion on probabilities and neural networks[END_REF], the logistic regression model can be written as:

p(y = 1|s) = (1 + exp{-(w t s + b)}) -1 = σ(w t s + b).
In addition, the bias term b can be absorbed into w via a standard trick, by adding a default system that produces score 1 for each trial. Then, the likelihood of the logistic regression model can be written as:

p(D|w) = N dev n=1 σ(w t sn) yn σ(-w t sn) 1-yn . (1)
Taking the negative logarithm of (1), we obtain the cost function [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]:

-

N dev n=1 yn ln σ(w t sn) + (1 -yn) ln σ(-w t sn) . (2)
In a speaker detection task, it is common to train the fusion to be sensitive to a predefined cost function. Taking into account the Bayes risk optimal decision, the cost parameters (Cmiss, CFA, Ptar) can be summarized as an effective prior, P eff [START_REF] Brümmer | Application-independent evaluation of speaker detection[END_REF] such that:

P eff = PtarCmiss PtarCmiss + (1 -Ptar)C fa (3)
Finally, due to the fact that the posterior probability of the target trials is different from the synthetic Ptar, the cost function in (2) was further modified in [START_REF] Brümmer | Fusion of heterogeneous speaker recognition systems in the STBU submission for the NIST speaker recognition evaluation 2006[END_REF] as follows:

E(w, D) = P eff Nt N t i=1 log 1 + e -w t s i -logit P eff + 1 -P eff N f N f j=1 log 1 + e w t s j +logit P eff , ( 4
)
where the summation is over the Nt target score vectors si, and the N f non-target score vectors sj, respectively. Regularization can be added to the cost (4) in order to improve the generalization ability of the classifier. In the case of p-norm regularizer we will obtain

min w E(w, D) s.t. w q p ≤ t. (5)
Then, the method of Lagrange multipliers will give us

min w E(w, D) + λ w q p . (6)
In most typical settings, p and q are both set to 2 for the standard Ridge regression, and set to 1 for the LASSO [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF].

Integrating out the hyperparameters

The solution to (6) can be interpreted as an MAP estimate of [START_REF] Foo | A majorization-minimization algorithm for (multiple) hyperparamtere learning[END_REF]:

wMAP = arg min w E(w, D) -log p(w|λ) (7)
We can simplify [START_REF] Figueiredo | Adaptive sparseness for supervised learning[END_REF] by marginalizing λ out from the prior, i.e., p(w) = λ p(w|λ)p(λ)dλ. Depending on the choice of regularizer, we now need to define the p(λ) and compute the integral.

In [START_REF] Foo | A majorization-minimization algorithm for (multiple) hyperparamtere learning[END_REF], for the case of Ridge regression (p = 2), prior is defined as an isotropic Gaussian p(w|λ) ∝ exp{-1 2 λ w 2 2 } and λ ∼ Ga(α, β), a Gamma distribution. Then, solving marginalization and removing the terms independent of w leads to [START_REF] Foo | A majorization-minimization algorithm for (multiple) hyperparamtere learning[END_REF]:

log p(w) = - L 2 + α log 1 2 w 2 2 + β (8)

Sparsity promoting priors

In the case of p = 1, a Laplacian distribution p(w|λ) = λ 2 exp{-λ w 1} is an appropriate representation of LASSO regularization in the prior form. In [START_REF] Figueiredo | Adaptive sparseness for supervised learning[END_REF], non-informative, also improper at the same time, Jeffreys hyperprior was placed on λ ∼ 1 λ Then a simple EM algorithm was derived that optimizes the model parameters in the hierarchial Bayes framework. In this work, we assume λ ∼ Ga(α, β). Similar approach was taken in the context of variational Bayes approach by [START_REF] Kabán | On Bayesian classification with Laplacian priors[END_REF].

We follow the example of [START_REF] Foo | A majorization-minimization algorithm for (multiple) hyperparamtere learning[END_REF] in deriving log p(w) in the case of Laplacian prior. We start by assuming that

λ ∼ p(λ|α, β) = Ga(α, β) = β α 1 Γ(α) λ α-1 exp{-λβ} (
= λ 2 exp{-λ w 1}β α 1 Γ(α) λ α-1 exp{-λβ}dλ = β α 1 2Γ(α) λ α exp{-λ( w 1 + β)}dλ = β α Γ(α + 1) 2Γ(α) ( w 1 + β) -(1+α) I(λ)dλ, (10)
where

I(λ) = λ α ( w 1 + β) α+1 Γ(α + 1) exp{-λ( w 1 + β)}, (11)
which happens to be a Gamma distribution with parameters Ga(α+1, w 1 +β), so the last integral in (10) is 1. Finally, by taking the logarithm of the result and removing additive terms where w does not appear, we obtain:

log p(w) = -(1 + α) log( w 1 + β). (12)

Majorization-minimization

the results in ( 12) and ( 7) we arrive at the cost function to be minimized:

f (w) = E(w, D) + L 2 + α log 1 2 w 2 + β , (13)
which is differentiable, but unfortunately not convex. The solution proposed in [START_REF] Foo | A majorization-minimization algorithm for (multiple) hyperparamtere learning[END_REF] is to use the majorization-minimization (MM) [START_REF] Lange | Optimization transfer using surrogate objective functions[END_REF] algorithmic framework to solve the optimization task in [START_REF] Brümmer | Application-independent evaluation of speaker detection[END_REF]. Majorization-minimization is based on the idea that we solve the optimization problem iteratively, where in each iteration we form a surrogate cost g(w, w ′ ), which is an upper bound of the original cost function f (w) ≤ g(w, w ′ ). Minimum of g(w, w ′ ) is found w.r.t. w, and the new bound is constructed. Algorithm converges when minimum, whether global or local, is reached. For convergence, we require that equality is reached only when f (w ′ ) ≤ g(w ′ , w ′ ). Convergence is then assured by the so called descent property [START_REF] Lange | Optimization transfer using surrogate objective functions[END_REF]:

f (w ′ ) = g(w ′ , w ′ ) ≤ g(w ′′ , w ′ ) ≤ f (w ′′ ) (14)
The reader might now notice that EM framework is actually a special case of MM optimization framework.

In [START_REF] Foo | A majorization-minimization algorithm for (multiple) hyperparamtere learning[END_REF], a surrogate function g(w, w ′ ) is constructed in the following way. For notational convenience, we denote the terms inside the log in (8) as h(w) = 1 2 w2 + β. Then, taking the first order Taylor approximation around w ′ of the term, we notice that

log(h(w)) ≤ log(h(w) ′ ) + h(w) h(w ′ ) -1, (15)
for any w ′ and equality is obtained if and only if w = w ′ . Using the result in [START_REF] Kabán | On Bayesian classification with Laplacian priors[END_REF], g(w, w ′ ) can be written as

g(w, w ′ ) = E(w, D) + L/2 + α 1/2 w ′ 2 + β 1 2 w 2 , ( 16
) where terms independent of w been removed. Obtained algorithm is presented in Algorithm 1. We notice first of all that, there is no need to manually (or by cross-validation) set the λ-parameter. But what is even more interesting is that we read an estimate of the λ directly from the g(w, w ′ ). Using the same strategy as in [START_REF] Kabán | On Bayesian classification with Laplacian priors[END_REF] for a LASSO regularizer, we can do away with the log operation in f (w).

f (w) = -log p(D|w) + (1 + α) log ( w 1 + β) . (18)
Which will lead to λ = 1+α w (t) 1 +β as a new λ estimate in t th iteration. However, the approach taken here requires that the bound is exact if and only if w w (t) , and this unfortunately is not the case with the l1 norm. It will lead to non-convergent algorithm, however, in preliminary experiments we found out that after a few iterations, differences between consecutive λ's is not very large.

Algorithm 1 Majorization-minimization for fusion training

1: t = 0; 2: repeat 3:

λ q (t) = 1 if t = 0 L/2+α 1 2 w (t) 2 +β otherwise. ( 17
)
4:

w (t+1) = arg minw E(w + 1 2 λ (t) w 2 5: until convergence

Experiments

Various solutions were put in place as part of the I4U 2 submission consisting of 17 base classifiers of which 15 were used in these experiments These include the GMM-SVM [START_REF] Campbell | Support vector machines using GMM supervectors for speaker verification[END_REF], JFA [START_REF] Kenny | A study of inter-speaker variability in speaker verification[END_REF], and the most advanced i-vector based classifier [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF]. Briefly, the ensemble consists of 1 GMM-UBM, 3 GMM-SVMs, 1 JFA, and 13 i-vector based classifiers. In essence, all of these classifiers are based on the use of Gaussian mixture model (GMM) in a GMM-UBM framework. In other words, these classifiers require and have their own UBM for extracting the Baum-Welch statistics. To this end, UBMs are trained mainly using data drawn from SRE04, though SRE05, SRE06, Switchboard and Fisher were also used. The channel compensation components (NAP for GMM-SVM classifiers, channel factors in JFA, LDA and PLDA for i-vector) were trained using both the telephone and microphone data drawn from SRE04-SRE10. Some i-vector based classifiers use cosine scoring, while most classifiers use PLDA, such as back-ends of CRSS systems [START_REF] Hasan | CRSS systems for 2012 NIST speaker recognition evaluation[END_REF].

To enable the researchers in the I4U coalition to optimize speaker recognition systems performance on a condition similar to SRE'12 evaluation plan, the development sets were generated 3 . Two sets called development (DEV) and evaluation (EVAL) are made to assess the absolute performance as well as calibration performance of the systems. A disjoint set of speakers from SRE'06 are held out from our DEV and EVAL training sets and only included in test phase to simulate the unknown non-target trials. Number of segments, speakers and trials for each set are given in Table 1. For the SRE12 fusion experiments, we train the fusion device on DEV portion of the I4U and apply it as is to common conditions (CC) 1 and 3.

We experimented with the maximum likelihood optimized logistic regression (cross-entropy cost) 4 , Ridge regularized cross-entropy, and LASSO regularized cross-entropy. In the case of LASSO regularization, non-differentiability of the prior is handled by the optimizer package we utilized. More details on the optimizer can be found in [START_REF] Hautamäki | Sparse classifier fusion for speaker verification[END_REF]. Regularized variants are the implementation of Section 3.2, where λ is first integrated out from the optimization cost leading to non-convex cost function. This cost function is then optimized by a local search scheme, namely majorization-minimization. The MM estimates gives as a side product the estimate of the λ. In all experiments p known = 0 and we report only the equal error rate (EER) and minCprimary. In both regularized variants, parameters of Gamma distribution were set to Ga(0, 1). We also show best individual base classifier, based on the performance on the dev set, namely GMM-SVM utilizing an anti model [START_REF] Sun | Anthi-Model KL-SVM-NAP System for NIST SRE 2012 Evaluation[END_REF].

In Table 2, we notice that majorization-minimization optimized Ridge and l1 improves the minCprimaryover nonregularized approach and best individual classifier. However, for EER, regularized approaches do not bring improvement over non-regularized and even in the case of Female subset EER is worse than best individual. Estimated λ Ridge and l1 similar for both Female and Male subsets, indicat- In Tables 3 and4 we see equal error rate and minCprimaryfor SRE12 evaluation corpus subset CC1 and CC2. We notice that the ridge and l1 give practically identical results for these sets. In terms of both EER and minCprimarywe note that regularization is better only in one case out of eight, in other cases regularization does not help. This is an indication that optimizing fusion weights on I4U DEV did not lead to an overfit, which would make it necessary to regularize in order to avoid it.

Conclusions

In this paper, we have compared different regularization techniques for fusion of speaker verification systems when optimizing a cross-entropy function. We proposed to remove the need of setting a hyper-parameter and to apply cross-validation by integrating out the regularization parameter. Several regularization constraint have been compared. For the case of Ridge regularized logistic regression, we applied an existing marginalization technique in complement of majorization-minimization algorithm to learn the regularization parameters. We also derived a similar method for the case of sparsity promoting prior focusing on the l1 regularization as sparse fusion. As a future we plan to investigate an other bounds in the LASSO case, so that majorization-minimization algorithm converges in all cases.

  [START_REF] Hautamäki | Variational Bayes logistic regression as regularized fusion for nist sre 2010[END_REF] and we can then write p(w) = p(w|λ)p(λ|α, β)dλ

Table 1 :

 1 Number of speakers, speech segments and trials in the development sets.

		Number of speakers		Number of segments			Number of trials
		DEV		EVAL	DEV		EVAL		DEV	EVAL
	Train	Test	Train	Test	Train	Test	Train	Test	True	False	True	False
	Males	680	868	763	804	16941 19866 29961 21837 14589 13494291 15483 16646148
	Females 1039 1243 1155 1102 24693 25980 43119 28548 19863 26973357 20763 32952177
	Table 2: Fusion performance on the I4U evalset
	Method	EER minCprimary Estimated
			(%)				λ
				Female
	Best ind.	0.54		0.0667		n/a
	No Regul. 0.33		0.0524		n/a
	Ridge		0.63		0.0449	4.0749
	l1		0.63		0.0448	0.2963
				Male
	Best ind.	0.68		0.0656		n/a
	No Regul. 0.40		0.0483		n/a
	Ridge		0.49		0.0399	4.0835
	l1		0.49		0.0398	0.2759
	Table 3: Results on SRE12 evaluation corpus, common condi-
	tion 1 (CC1) subset.
	Method	EER (%) minCprimary
				Female
	Best ind.		5.74	0.36179
	No regul.	2.67	0.27674
	Ridge		3.75	0.30752
	l1			3.74	0.30775
				Male
	Best ind.		5.45	0.34738
	No regul.	3.48	0.20204
	Ridge		3.38	0.22670
	l1			3.39	0.22670
	ing close similarity in the trainingset score distributions. The
	majorization-minimization algorithm converged for Ridge reg-
	ularizer in three iterations and for l1 in two iterations.

Table 4 :

 4 Results on SRE12 evaluation corpus, common condition 3 (CC3) subset.

	Method	EER (%) minCprimary
		Female
	Best ind.	3.52	0.15913
	No regul.	2.78	0.22766
	Ridge	2.92	0.21396
	l1	2.92	0.21396
		Male
	Best ind.	4.52	0.13511
	No regul.	3.87	0.07187
	Ridge	4.06	0.07604
	l1	4.06	0.07604

In practice, the optimizer is needed to run only few times, versus computing it for each λ candidate.

This is a collaboration of the following institutes: (1) Institute for Infocomm Research, Singapore, (2) University of Eastern Finland, Finland, (3) Radboud University Nijmegen (RUN), The Netherlands, (4) CRSS, University of Texas at Dallas, USA (4) ValidSoft Ltd, London, UK, (5) LIA, University of Avignon, France, (6) Idiap Research Institute, Switzerland, (7) Swansea University, UK, (8) University of New South Wales,

Australia[START_REF] Hautamäki | Regularized logistic regression fusion for speaker verification[END_REF] The lists are available via http://lands.let.ru.nl/

˜saeidi/I4U.tgz[START_REF] Hastie | The Elements of Statistical Learning -Data Mining, Inference, and Prediction[END_REF] Similar to software package FoCal, but a different implementation