Fabien Ringeval
email: fabien.ringeval@uni-passau.de

Erik Marchi

Charline Grossard

Jean Xavier

Mohamed Chetouani

David Cohen

Björn Schuller

Automatic Analysis of Typical and Atypical Encoding of Spontaneous Emotion in the Voice of Children

Keywords: affective computing, spontaneous emotions, autism spectrum disorders, language impairments

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The ability to communicate with speech requires the acquisition of codes that link acoustic realisation to both linguistic [START_REF] Schuller | The linguistic status of prosodic and paralinguistic features[END_REF] and socio-affective related meanings [START_REF]Prosodic and paralinguistic correlates of social categories[END_REF][START_REF] Marchi | Autism and Speech, Language, and Emotion -a Survey[END_REF]. The acquisition and correct use of such codes, which are supposed to be functional in the early stages of a child's life [START_REF] Kuhl | Early language acquisition: cracking the speech code[END_REF], play an essential role in the inter-subjective development and social interaction abilities of children. As a consequence, most children presenting speech or developmental disorders have limited social interactions, which contributes to social isolation [START_REF] Müller | Social challenges and supports from the perspective of individuals with Asperger syndrome and other autism spectrum disabilities[END_REF].

International classifications differentiate Specific Language Impairment (SLI) from those that are symptomatic of a developmental disorder, such as ASD. The former can affect both expressive and receptive language and is defined as a 'pure' language impairment. The latter, ASD, is characterised by severe deficits and pervasive impairment in several areas of development such as reciprocal social interactions, communication skills and stereotyped behaviours, interests and activities [START_REF] Rapin | Developmental language: nosological consideration[END_REF]. Because of the clinical heterogeneity of ASD, the recent DSM-5 decided to adopt a single diagnosis and to specify some dimensional features [START_REF] Xavier | Autism spectrum disorders: An historical synthesis and a multidimensional assessment toward a tailored therapeutic program[END_REF]. The DSM-4 distinguished ASD subtypes [START_REF]Diagnostic and Statistical Manual of mental disorders[END_REF]: e.g. Autism Disorders (AD), with symptoms in all areas that characterise Pervasive Developmental Disorders (PDD); or Pervasive Developmental Disorder-Not Otherwise Specified (PDD-NOS), which is characterised by social, communicative and/or stereotypic impairments that are less severe than in AD and appear later in life [START_REF] Xavier | Autism spectrum disorders: An historical synthesis and a multidimensional assessment toward a tailored therapeutic program[END_REF]. Some authors have shown that AD, PDD-NOS and SLI had relied on different linguistic mechanisms and that expressive syntax, pragmatic skills, and some intonation features could be considered as language differential markers of pathology [START_REF] Demouy | Differential language markers of pathology in Autism, Pervasive Developmental Disorder Not Otherwise Specified and Specific Language Impairment[END_REF][START_REF] Ringeval | Automatic intonation recognition for prosodic assessment of language impaired children[END_REF].

Both clinicians and researchers are facing a huge increase in the prevalence of ASD, resulting from the expansion of the diagnostic criteria, but also from a better awareness of the condition and the acceptance that ASD is a lifelong condition [START_REF] Matson | The increasing prevalence of autism spectrum disorders[END_REF]. Recently, systems have been proposed to provide objective measurements of acoustic features used by children suffering from ASD to encode non-verbal information in speech by the use of prosody [START_REF] Ringeval | Automatic intonation recognition for prosodic assessment of language impaired children[END_REF][START_REF] Marchi | Emotion in the Speech of Children with Autism Spectrum Conditions: Prosody and Everything Else[END_REF][START_REF] Bone | The psychologist as an interlocutor in autism spectrum disorder assessment: Insights from a study of spontaneous prosody[END_REF][START_REF] Marchi | Typicality and Emotion in the Voice of Children with Autism Spectrum Condition: Evidence Across Three Languages[END_REF]. Analyses can be then performed indirectly, by assessing the performance of a child on a given task, e.g., producing specific prosodic contours to convey sentence modality [START_REF] Ringeval | Automatic intonation recognition for prosodic assessment of language impaired children[END_REF], or emotion [START_REF] Marchi | Emotion in the Speech of Children with Autism Spectrum Conditions: Prosody and Everything Else[END_REF][START_REF] Marchi | Typicality and Emotion in the Voice of Children with Autism Spectrum Condition: Evidence Across Three Languages[END_REF]. In this case, the system is tuned for each group of children, e.g., typically developing (TD) and ASD, and performance can be compared between the groups to provide cues regarding the observed atypicalities of ASD. Analyses can also be performed directly, as an automatic diagnosis, by comparing the children's groups in the task [START_REF] Marchi | Voice-enabled assistive robots for handling autism spectrum conditions: an examination of the role of prosody[END_REF][START_REF] Marchi | Speech, Emotion, Age, Language, Task, and Typicality: Trying to Disentangle Performance and Feature Relevance[END_REF]. In this case, the system is tuned to search for differences in speech production between each group of children, which can also be a mean to identify the particularities of ASD.

Such systems can be used to help clinicians to improve the diagnosis, but also to develop tools based on information and communication technology, which enable users to access professional support on-line [START_REF] Schuller | Recent developments and results of ASC-Inclusion: An Integrated Internet-Based Environment for Social Inclusion of Children with Autism Spectrum Conditions[END_REF]. Recent studies have indeed shown that it is possible to improve emotional skills of ASD children in both emotion perception and production, by providing them interactive tools integrating affective computing [START_REF] Schuller | Recent developments and results of ASC-Inclusion: An Integrated Internet-Based Environment for Social Inclusion of Children with Autism Spectrum Conditions[END_REF]. However, the automatic processing of children's speech is challenging, as they present significant differences compared to the voice of adults [START_REF] Potamianos | A review of the acoustic and linguistic properties of childrens speech[END_REF], and even more when they are affected by ASD.

Contribution of this work

The present study focuses on the recognition of spontaneous emotional expressions in the voice of AD, PDD-NOS, SLI, and Figure 1: Extracts from the book "Frog where are you?" [START_REF] Mayer | Frog where are you? New York: Dial Books for young readers[END_REF] that were used for recording spontaneous emotional speech production from children. We hypothesised that images with Neutral valence (left picture, the beginning of the story: the boy is sleeping in his room while his frog is escaping), Negative valence (middle picture, the middle of the story: the boy has been captured by a deer while searching for his frog) and Positive valence (right picture, the end of the story: the boy finally finds his frog) correlated with emotion production. typically developing (TD) children. We investigate the classification performances with expert-based reduced feature sets against large sets of features that include a vast number of spectral-, source-, and duration-related features. This study further focuses on the automatic discrimination of typicality between TD children and children suffering from AD, PDD-NOS, and SLI. For this purpose, a new database -Child Pathological and Emotional Speech database (CPESD) -is introduced, which will be made available to academic researchers. To the best knowledge of the authors, this is the very first study that performs automatic analysis of spontaneous emotion in children's speech with AD, PDD-NOS, and SLI. We hypothesised that we could distinguish: (i) different types of emotion based on prosodic features; (ii) patients with speech impairments from TD controls; (iii) different profiles within patients with speech impairments since some authors suggested that AD, SLI, and/or PDD-NOS may have different profile of emotional impairments [START_REF] Mazefsky | The Role of Emotion Regulation in Autism Spectrum Disorder[END_REF][START_REF] Xavier | Reliability and diagnostic efficiency of the Diagnostic Inventory for Disharmony (DID) in Youths with Pervasive Developmental Disorder and Multiple Complex Developmental Disorder[END_REF][START_REF] Lai | Subgrouping the Autism Spectrum: Reflections on DSM-5[END_REF].

Child Pathological & Emotional Speech Database

We received approval by the Ethical Committee of the Pitié-Salpétrière Hospital to conduct recruitment and speech recording of children. Consents were obtained from parents or legal caregivers of all participants. Thirty-five monolingual participants with communicative verbal skills were recruited in two university departments of child and adolescent psychiatry located in Paris, France. They consulted for ASD and/or SLI which were diagnosed as AD, PDD-NOS, or SLI, according to DSM IV criteria [START_REF]Diagnostic and Statistical Manual of mental disorders[END_REF] academic grades, and lexical abilities. Socio-demographic and clinical characteristics of the participants are available in [START_REF] Ringeval | Automatic intonation recognition for prosodic assessment of language impaired children[END_REF].

We also recruited a group of 70 TD children matched for age and sex (1 patient for 2 TD) in elementary schools. A questionnaire was used to exclude children with learning disorders, an history of speech, language, hearing, or general learning problems.

Our main goal was to compare children's abilities to use prosody to encode pragmatic and affect in speech. A first task was based on the reproduction of intonation contour and was analysed in a previous study [START_REF] Ringeval | Automatic intonation recognition for prosodic assessment of language impaired children[END_REF]. The second task was based on a story telling of a pictured book "Frog where are you?" [START_REF] Mayer | Frog where are you? New York: Dial Books for young readers[END_REF], wherein a little boy tries to find his escaped frog during the night. The task was originally developed to assess language production in a standardised but unconstrained manner. Here, we assume that the child is supposed to produce prosodic cues during the story telling that are correlated to the levels of the emotional valence, which was categorised in three categories by a psychologist: Negative/Neutral/Positive. In total, the pictured book included 15 emotionally negative, 6 emotionally neutral and 5 emotionally positive pictures, cf. Figure 1 We collected nearly 10 hours of recording: 7 h 38 min for TD children, 1 h 35 min for children with AD, 1 h 12 min for children with PDD-NOS, and 1 h 56 min for children with SLI. Recordings were then segmented automatically into groups of breaths, using the energy contour. As many sources of perturbation appeared during the recordings (e. g., false-starts, repetitions or noise from the environment), the obtained speech segments were further manually processed; only utterances that had a complete prosodic contour, i. e., whatever the pronounced words, were kept. Statistics (number, relative proportion, and mean duration) on those utterances are provided for each emotion category, and all, in Table 3. Those data already provide some interesting insights: all TD children produced utterances which are significantly longer than AD, PDD-NOS, and SLI children for all emotion categories (p < 0.5, two-tailed t test); we observed the opposite on the constrained task of intonation contour imitation [START_REF] Ringeval | Automatic intonation recognition for prosodic assessment of language impaired children[END_REF]. Moreover, spontaneous speech production of PDD-NOS children focused significantly more on positive emotions compared to all other groups (p < 0.5).

Experiments

Two main tasks were performed: automatic recognition of typicality (direct analysis), and emotion (indirect analysis). The typicality task concerns the classification of TD children vs all other children. Additionally, we performed the classification of each group of children (diagnosis). The emotion task covers the recognition of the three classes of emotional valence, i. e., positive, neutral, and negative. This task was performed either on each group separately, or with models trained on TD children.

Acoustic features

Acoustic features were automatically extracted from the speech waveform on the utterances by using our open-source openS-MILE feature extractor in its recent 2.2 release [START_REF] Eyben | Recent developments in openSMILE, the Munich open-source multimedia feature extractor[END_REF]. Five different feature sets were investigated: large brute-forced feature sets (IS09, IS10, IS11, and ComParE), which have all been used for paralinguistic information retrieval, and a smaller, expert knowledge based feature set (eGeMAPS). Those feature sets cover spectral-, source-and duration-related feature space with 1. The first four sets, i. e., IS09, IS10, IS11, and ComParE, show a clear tendency in enlarging the feature space over the years, by including further low-level acoustic descriptors and associated functionals. Recently, this "brute-forcing" approach has been revisited, with investigations on a small, expert knowledge based feature set, eGeMAPS [START_REF] Eyben | The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing[END_REF]. A detailed description and implementation of these feature sets, which is impossible to provide here, is given in [START_REF] Eyben | Real-time speech and music classification by large audio feature space extraction[END_REF].

Setup and evaluation

We used Support Vector Machines (SVMs) for the classification tasks with LIBSVM [START_REF] Chang | LIBSVM: A library for support vector machines[END_REF], as they are a well known standard method that can handle both high and low dimensional data. The SVM training has been made with the three different kernels: linear, polynomial (3rd order), and Gaussian (γ parameter was set to default value); the complexity parameter was set to default value (C = 1). Results are always presented with the best kernel. To ensure speaker independent evaluations, we performed a Leave-One-Speaker-Out (LOSO) crossvalidation in all experiments. Because all data sets are unbalanced, we applied upsampling of the under-represented classes in all the evaluation experiments. For the same reason, we used the unweighted average recall (UAR) of the classes as scoring metric. ues are normalised to zero-mean and unit standard deviation, was performed for each for all emotion recognition tasks. Whereas for both typicality and diagnosis tasks, we standardised the features of all speakers with the on-line approach, i. e., mean and standard-deviation were computed on the training partition and applied on both training and testing data.

Results

Typicality & Diagnosis

Results obtained on typicality and diagnosis are given for each emotion class and each feature set in Table 2. One may note that all obtained performance are far above the chance level, in agreement with [START_REF] Marchi | Typicality and Emotion in the Voice of Children with Autism Spectrum Condition: Evidence Across Three Languages[END_REF], despite being lower than on the constrained task, i. e., intonation contour imitation [START_REF] Schuller | The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism[END_REF]. In order to gain further insights, we performed automatic recognition of typicality and diagnosis with each different feature subset, i. e., spectral-, source-, and duration-related features. Results show that, spectral-related features are the most contributing in the two tasks, and can perform even better when taken alone for diagnosis, cf. Table 4.

Emotion

Results obtained on the automatic recognition of the emotional valence are given in Table 5. Our hypothesis that the spontaneous description of the pictured book will be correlated with the emotional valence depicted in the images is validated by the experiments, because the system performs significantly better than chance for all groups of children. Obviously, TD children obtained the best performance, and all other children obtained a significantly lower performance (p < 0.5); two-tailed t test. A detailed analysis of each feature subset shows that, (i) spectralrelated features provide the best performance for all groups of children, and (ii) the minimalistic feature set, i. e., eGeMAPS, performed remarkably well on all groups, especially for sourcerelated features, cf. formance was improved further for the PDDs, by combining spectral-and duration-related features for AD, and all feature subsets for PDD-NOS, cf. Table 7.

Finally, in order to investigate how the models obtained on TD children could generalise on the others groups of children, we performed a mismatched evaluation, by training models on TD and testing on AD, PDD-NOS, and SLI. Results show that, there are some specific associations between feature space and pathology; models obtained from TD children generalised best on AD with source-related features, PDD-NOS with spectralrelated features, and SLI with duration-related features. Moreover, PDD-NOS children obtained systematically the best performance for all combinations of the feature subsets, cf. Table 8. This result supports the hypothesis that PDD-NOS may express more emotion than AD [START_REF] Mazefsky | The Role of Emotion Regulation in Autism Spectrum Disorder[END_REF][START_REF] Xavier | Reliability and diagnostic efficiency of the Diagnostic Inventory for Disharmony (DID) in Youths with Pervasive Developmental Disorder and Multiple Complex Developmental Disorder[END_REF][START_REF] Lai | Subgrouping the Autism Spectrum: Reflections on DSM-5[END_REF].

Conclusions

A new speech database of spontaneous emotions produced by AD, PDD-NOS, SLI and TD speaking children has been introduced: CPESD. Extensive experiments have been performed on this database, showing that all groups of children can be differentiated directly (typicality, diagnosis) and indirectly (emotion) with an automatic recognition system. A detailed analysis has shown that, large scale features are necessary to differentiate TD from the other groups, whereas the eGeMAPS set provides the best results for the emotion recognition task.

Table 1 :

 1 Distribution of acoustic features in spectral/energy-

	Feature set Spectral Source Duration Total
	eGeMAPS	48	48	6	102
	IS09	336	48	0	384
	IS10	1216	212	154	1582
	IS11	2808	272	1288	4368
	ComParE	4366	397	1610	6373

related, source/excitation-related and duration-related feature sets for different configurations of openSMILE.

Table 2 :

 2

	Feature set Negative Neutral Positive	All
		a. Typicality
	eGeMAPS	83.1	82.6	79.0	83.2
	IS09	83.2	80.5	79.7	84.0
	IS10	87.1	85.2	85.2	87.6
	IS11	89.1	87.5	87.4	89.4
	ComParE	88.0	85.1	86.5	86.3
		b. Diagnosis
	eGeMAPS	46.2	48.2	44.9	48.2
	IS09	50.8	45.8	46.7	51.2
	IS10	51.8	47.1	50.1	53.7
	IS11	54.7	52.2	51.3	56.4
	ComParE	53.2	51.2	50.9	56.2

. Patients were matched for age, sex, UAR -typicality (2 classes) or diagnosis (4 classes) recognition for each feature set and emotion category.

Table 3 :

 3 . Three pictures considered ambivalent because of ambiguous interpreta-Number, relative proportion, and mean duration of utterances per emotion class and group. * = p < 0.05: alternative hypothesis is true when comparing data between children groups, i. e., A, N, S, and T; AD (A): autism disorders; NOS (N): developmental disorders not-otherwise specified; SLI (S): specific language impairment; TD (T): typically developing. Sour. Dur. Spec. Sour. Spec. Sour. Dur. Spec. Sour. Dur. Spec. Sour. Dur.

	Group

Table 4 :

 4 UAR -typicality (2 classes) or diagnosis (4 classes) recognition for each feature subset and all emotion categories.

tion were excluded.

Table 6 :

 6 Sour. Dur. Spec. Sour. Spec. Sour. Dur. Spec. Sour. Dur. Spec. Sour. Dur. UAR -spontaneous emotion recognition (3 classes) for each feature subset and groups of children; AD: autism disorders; NOS: pervasive developmental disorders not-otherwise specified; SLI: specific language impairment; TD: typically developing.

		eGeMAPS	IS09		IS10			IS11			ComParE
	Group Spec. AD 34.8	37.4 37.7	37.0	37.4	35.9	34.8	33.6	39.8	37.3 36.8	38.7	33.5	35.8
	NOS	38.5	39.1	30.2	38.4	37.1	41.4	36.5	39.2	34.1	36.9	37.0	33.4	35.0	37.0
	SLI	43.7	38.5	38.0	38.5	37.1	42.1	38.2	34.9	40.4	37.4	38.7	40.5	37.5	38.9
	TD	46.0	41.3	34.6	43.1	36.0	43.7	38.3	38.9	44.4	38.3	42.0	44.2	39.9	42.3
	Feature set		AD NOS SLI	TD
	Spectral + Source	40.0 39.7 43.3 44.7
	Source + Duration	37.3 37.3 39.5 42.8
	Spectral + Duration	40.6 41.5 39.0 43.2
	Spec. + Sour. + Dur. 39.8 42.1 39.1 43.2
	Best group		39.8 41.4 43.7 46.0
	Table 7: UAR -spontaneous emotion recognition with 3
	classes, combination of the best feature subsets; AD: autism dis-
	orders; NOS: pervasive developmental disorders not-otherwise
	specified; SLI: specific language impairment; TD: typically de-
	veloping.

Standardisation of the features, i. e., feature val-

Table 6 .

 6 Results obtained with different combinations of the best feature subsets show that, the per-

	Feature set	AD NOS SLI	TD
	Spectral	35.8 41.7 35.8 46.0
	Source	39.3 38.4 33.9 41.3
	Duration	34.2 37.8 38.0 42.3
	Spectral + Source	38.0 43.5 33.2 44.7
	Source + Duration	36.3 39.2 37.5 42.8
	Spectral + Duration	34.7 40.5 38.7 43.2
	Spec. + Sour. + Dur. 35.1 40.7 38.2 43.2
	IS11	38.3 40.8 39.6 44.5

Table 8 :

 8 UAR -spontaneous emotional valence recognition with 3 classes and model training on TD; AD: autism disorders; NOS: pervasive developmental disorders not-otherwise specified; SLI: specific language impairment; TD: typically developing.

Acknowledgements

The research leading to these results has been funded by the European Community's Seventh Framework Programme through the ERC Starting Grant No. 338164 (iHEARu), and the European's Union's Horizon 2020 Programme through the Research Innovative Action #688835 (DE-ENIGMA).


Dan Istrate

Corinne Fredouille

Sylvain Meignier

Laurent Besacier

Jean François Bonastre

NIST RT'05S Evaluation : Pre-Processing Techniques and Speaker Diarization on Multiple Microphone Meetings

This paper presents different pre-processing techniques, coupled with three speaker diarization systems in the framework of the NIST 2005 Spring Rich Transcription campaign (RT'05S). The pre-processing techniques aim at providing a signal quality index in order to build unique "virtual" signal obtained from all the microphone recordings available for a meeting. The unique "virtual" signal relies on a weighted sum of the different microphones while the signal quality index is given according to a signal to noise ratio. Two methods are used in this paper to compute the instantaneous signal to noise ratio: speech activity detection based approach and a noise spectrum estimate. The speaker diarization task is performed using systems developed by different labs: the LIA, LIUM and CLIPS. Among the different system submissions made by these three labs, the best system obtained 24.5 % speaker diarization error for the conference subdomain and 18.4 % for lecture subdomain.

Introduction

The goal of speaker diarization is to segment a N-speaker audio document in homogeneous parts containing the voice of only one speaker and to associate the resulting segments by matching those belonging to a same speaker. In speaker diarization the intrinsic difficulty of the task increases according to the target data: telephone conversations, broadcast news, meeting data.

This paper is related to speaker diarization on meeting data in the framework of the NIST 2005 Spring Rich Transcription (RT'05S) campaign. Meeting data present three main specificities compared to broadcast news data:

• meeting conversations are recorded with multiple microphones which implies redundancies, different qualities of the same speech record. The use of information from all channels seems to be an important issue; • the meeting room recording conditions associated with distant microphones lead to noisy recordings, including background noises, reverberations and distant speakers;

• the speech is fully-spontaneous, highly interactive across participants and presents a large number of disfluencies as well as speaker segment overlaps.

This paper is focused on the extraction of pertinent information issued from the different multiple microphone recordings in the particular task of speaker diarization. Indeed, signal processing techniques are applied on the different distant microphone signal recordings in order to determine pertinent portions of signal and to build a unique "virtual" signal. This virtual signal is then used as input for the speaker diarization systems. Basically, the unique "virtual" signal is based on a weighted sum of the multiple microphone signals. The weights of this sum are estimated according to a signal quality index based on a signal to noise ratio estimate.

Two main factors will be studied in this paper; the efficiency of the preprocessing techniques to build a unique "virtual" signal in the context of speaker diarization will be first investigated as well as the robustness of speaker diarization systems, only tuned on broadcast news data, when handling meeting data.

Concerning the last point, different speaker diarization systems will be tested in this study. Developed in three different labs: the LIA, LIUM and CLIPS, these systems have been tuned and evaluated during the French ESTER Rich Transcription evaluation campaign (organized in January 2005 and sponsored by the French ministry), dedicated to Broadcast news data [START_REF] Galliano | The ester phase ii evaluation campaign for the rich transcription of french broadcast news[END_REF]. No particular tuning has been made on the meeting data in order to evaluate whether a reliable preprocessing on multi-channel recordings may be sufficient in order to maintain the performance of Broadcast News speaker diarization systems.

Finally, the RT'05S evaluation campaign has initiated a new task, based on the "Speech Activity Detection" (SAD). This processing is classically implemented in both the speech transcription and speaker diarization systems but never scored individually. This paper will present the SAD system proposed by the authors for the RT'05S evaluation campaign and their results.

Section 2 presents the Speech Activity Detection algorithm. Section 3 is devoted to the pre-processing techniques used in order to obtain a unique signal from the multi-channel recordings. In section 4, the LIA, LIUM and CLIPS speaker diarization systems are presented, followed by a brief description of all the systems submitted for the RT'05S evaluation campaign. Section 5 presents the experimental protocols and results and finally, section 6 concludes this work.

Speech Activity Detection task

Only considered until now as a sub-part of speech transcription or speaker diarization systems, "Speech Activity Detection" has been evaluated in the RT'05S evaluation campaign as an entire task. Speech Activity Detection is not trivial in a multiple microphone environment. For instance, the portions of silence might be different from one microphone to another. Besides, energy based SAD systems have some difficulties while dealing with background voices.

The Speech Activity Detection (SAD) system, used by most of the systems presented in this paper, was developed by the LIA. It is based on the ALIZE platform [START_REF] Bonastre | Alize, a free toolkit for speaker recognition[END_REF] and relies on two passes:

• to apply a speech activity detector process on each individual channel for a given meeting, provided speech and non-speech segments; • to keep the non-speech segments, shared over ALL the channels.

The speech activity detector process used in the first pass is based on the energy modelling and works as follows:

1. The energy coefficients are first normalized using a mean removal and variance normalization in order to fit a 0-mean and 1-variance distribution. 2. They are then used to train a three component GMM, which aims at selecting speech frames. Indeed, X% of the most energized frames are selected through the GMM, with: X = w 1 + (λ * α * w 2 ) where: w 1 the weight of the highest (energy) gaussian component, w 2 the weight of the middle component, λ an integer ranging from 0 to 1, α a weighting parameter, empirically fixed to 0.6 on the development set. The value of λ is decided according to the likelihood loss when merging the gaussian components 1 and 2 and the components 2 and 3. If the loss is higher for components 1 and 2, λ is set to 0 else to 1. 3. Once all the frames of a signal are labelled as speech or non-speech and concatenated to form segments according to their labels, a final process is applied in order to refine the speech detection. This last process is based on two morphological rules, which consist in constraining the minimum duration of both the speech and non-speech segments (minimun length is 0.3 s).

Meeting Pre-Processing algorithms

Meeting signals are recorded in smart rooms with multichannel acquisition systems. According to the distant microphone position in the table, the quality of signal may hugely differ from one microphone to another. For instance, the main speaker utterances may be caught by one or two distant microphones while the other microphones mainly provide background voices, long silence, or background noise only. The aim of this pre-processing system is to use redundant channel information in order to extract pertinent information for an enhanced output "virtual" signal. This output signal is a weighted mix of all channels available for a given meeting. For each channel a quality measure (signal to noise ratio -SNR) is estimated in order to adapt channel weights. The sum of weights is equal to 1 and the channel weights w i are computed following equation ( 1), where N is the number of channels.

wi = SN Ri/ N j=1

SN Rj

(1)

Fig. 1. Example of SNR estimate

To obtain a reliable quality measure it is necessary to estimate the noise energy, for which two methods have been considered: the use of a speech activity detector (SAD) and the noise spectrum estimate.

If a speech activity detector is used, the labelled non-speech segments are used to compute the average noise energy E noise for each channel. The SNR is estimated at each 32 ms on frames of 64 ms (L=1024 samples) using equation [START_REF] Bonastre | Alize, a free toolkit for speaker recognition[END_REF].

SN R = 10 log 10 L i=0 s 2 i -E noise /E noise [dB] (2)
where s i is a signal sample at instant i.

In the second case, an estimate of the noise spectrum is used in order to eliminate the speech activity detector errors and to have an instantaneous noise energy value instead of an averaged one. The algorithm is based on a minimum statistics tracking method [START_REF] Cui | A noise-robust asr back-end technique based on weighted viterbi recognition[END_REF]. Assuming the noisy speech power is the summation of clean speech and background noise power, tracking power spectral minima can provide a fairly accurate estimate of the background noise power and then a good estimate of SNR [START_REF] Hirsh | Estimation of noise spectrum and its application to snr-estimation and speech enhancement[END_REF]. Also, by tracking minimum statistics, this algorithm can deal with nonstationary background noise with slowly changing statistical characteristics. The noise spectrum is estimated every 2s using signal power spectrum histogram. An example of signal to noise ratio estimate for a part of channel 1 signal from "NIST 20020305-1007" file is presented in Figure 1.

In this case, the SNR is estimated using the signal power spectrum and noise power spectrum, like in equation [START_REF] Cui | A noise-robust asr back-end technique based on weighted viterbi recognition[END_REF].

SN R = 10 * log 10   M i=0 S i / M j=0 N j   (3)
where S i is signal spectral amplitude at frequency i and N j is noise spectral amplitude at frequency j.

In order to evaluate the influence of these pre-processing techniques, an unweighted mix (w i = 1 N ) has also been computed.

Three speaker diarization systems are involved in this work, developed individually by the LIUM, the CLIPS and the LIA laboratories. No particular tuning on the meeting data has been carried out for these systems to participate at the RT'05S evaluation campaign. Indeed, all these speaker diarization systems have participated to the French Rich Transcription evaluation campaign called ESTER (organized in January 2005 and sponsored by the French ministry), dedicated to Broadcast news data [START_REF] Galliano | The ester phase ii evaluation campaign for the rich transcription of french broadcast news[END_REF]. Testing these systems on meeting data without any further tuning will allow to evaluate the robustness of these systems to environment changes, especially if pre-processing techniques are applied beforehand on multiple microphone signals in order to extract pertinent information.

The LIA system

The LIA speaker diarization system has been entirely redeveloped by using the free ALIZE toolkit, detained by the LIA and firstly, dedicated to speaker recognition [START_REF] Meignier | Alize, a free toolkit for speaker recognition[END_REF]. Its performance has been evaluated firstly during the ESTER evaluation campaign on Broadcast News data. Despite this new development environment, the system still uses a one step algorithm based on E-HMM (Evolutive HMM) [START_REF] Moraru | The ELISA consortium approaches in broadcast news speaker segmentation during the NIST 2003 rich transcription evaluation[END_REF][START_REF] Meignier | Evolutive HMM for speaker tracking system[END_REF]. Each E-HMM state characterizes a particular speaker and the transitions represent the speaker changes. All possible changes between speakers are authorized. This algorithm has 2 stages: segmentation and resegmentation. The segmentation stage has 4 steps:

1. Initialisation: A first model, named L 0 , is estimated on all speech data. The HMM has one state, L 0 state. 2. New speaker detection: A new speaker is detected in the segments labelled L 0 as follows: a segment is selected among all the L 0 segments by likelihood maximization criterion. This selected segment is then used to estimate the model of the new speaker, named L x , which is added to the HMM. 3. Adaptation/Decoding loop: The objective is to detect all segments belonging to the new speaker L x . All speaker models are re-estimated through an adaptation process according to the actual segmentation. A Viterbi decoding pass is done in order to obtain a new segmentation. This loop adaptation/decoding is re-iterated while the segmentation is not stable. 4. Speaker model validation and stop criterion: The current segmentation is analyzed in order to decide if the new added speaker, L x , is relevant. In this case the decision is made according to heuristical rules on speaker L x segment duration. The stop criterion is reached if there is no more segment available in L 0 . On the contrary, the process goes to the step 2.

The resegmentation stage aims at refining the boundaries and at deleting unreliable speakers. This stage is based only on the third step of the segmentation process. A HMM is generated from the segmentation and the adaptation/decoding loop is launched. At the end of each iteration, speakers with too short duration are deleted.

Concerning the front end-processing, the signal is characterized by 20 linear cepstral features (LFCC), computed every 10ms using a 20ms window. The cepstral features are augmented by the energy. No frame removal or any coefficient normalization is applied at this stage.

The LIUM system

The LIUM speaker diarization system is based upon a BIC framework similar to [START_REF] Siu | An unsupervised, sequential learning algorithm for segmentation of speech waveforms with multi speakers[END_REF][START_REF] Chen | Speaker, environment and channel change detection and clustering via the bayesian information criterion[END_REF], composed of three modules: (1) the signal is first split into small homogeneous segments; (2)the segments are clustered per speaker without changing the boundaries; (3) the boundaries are adjusted.

The initial segment boundaries are determined according to a Generalized Likelihood Ratio (GLR) computed over two consecutive windows of 2s sliding over the features (12MFCC+E). No threshold is employed, except for the minimal segment length which is set to 2.5s. The signal is over-segmented in order to minimize miss detection of boundaries but the minimum segment length is set long enough for a correct estimate of a speaker model using a diagonal Gaussian.

The clustering is based upon a bottom-up hierarchical clustering. In the initial set of clusters, each segment is a cluster. The two closest clusters are then merged at each iteration until the BIC stop criterion is met. The speaker, ie the cluster, is modelled by a full covariance Gaussian as in the segmentation process. The BIC penalty factor is computed over the length of the two candidate clusters instead of the standard penalty computed over the length of the whole signal ( [START_REF] Zhu | Combining speaker identification and bic for speaker diarization[END_REF]). To minimized the clustering time, a first pass of clustering is performed only over adjacent clusters. The lambda parameter is fixed to 2 for the first pass and to 7.5 for the second pass.

A Viterbi decoding is performed to adjust segment boundaries. A speaker is modeled by a one-state HMM containing a diagonal covariance GMM of 8 components learned by EM-ML over the set of speaker segments. The log-penalty of switching between two speakers is fixed experimentally to 100.

The CLIPS system

The CLIPS system is based on a BIC [START_REF] Delacourt | DISTBIC: A speaker based segmentation for audio data indexing[END_REF] (Bayesian Information Criterion) speaker change detector followed by an hierarchical clustering. The clustering stop condition is the estimate of the number of speakers using a penalized BIC criterion. The BIC approach is used to define first potential speaker changes. A BIC curve is extracted by computing a distance between two 1.75s adjacent windows that go along the signal. Mono-Gaussian models with diagonal covariance matrices are used to model the two windows. A threshold is then applied on the BIC curve to find the most likely speaker change points which correspond to the local maximums of the curve. Clustering starts by first training a 32 components GMM background model (with diagonal covariance matrices) on the entire test file maximizing a ML criterion thanks to a classical EM algorithm. Segment models are then trained using MAP adaptation of the background model (means only). Next, BIC distances are computed between segment models and the closest segments are merged at each step of the algorithm until N segments are left (corresponding to the N speakers in the conversation).

The number of speakers (N Sp ) is estimated using a penalized BIC (Bayesian Information Criterion). The number of speakers is constrained between 1 and 15. The upper limit is related to the recording duration. The number of speakers (N Sp ) is selected to maximize the equation 4.

BIC(M ) = log L(X; M ) -λ(m/2)N Sp * log(N X ) ( 4
)
where M is the model composed of the N Sp speaker models, N X is the total number of speech frames involved, m is a parameter that depends on the complexity of the speaker models and λ is a tuning parameter equal to 0.6. The signal is characterized by 16 mel Cepstral features (MFCC) computed every 10 ms on 20 ms windows using 56 filter banks. Then the Cepstral features are augmented by energy. No frame removal or any coefficient normalization is applied here.

The entire speaker segmentation process is largely described in [START_REF] Moraru | The ELISA consortium approaches in speaker segmentation during the NIST 2002 speaker recognition evaluation[END_REF][START_REF] Moraru | The ELISA consortium approaches in broadcast news speaker segmentation during the NIST 2003 rich transcription evaluation[END_REF].

Proposed systems

Different systems have been submitted for the RT'05S campaign. Most of them relies on the following scheme -composition of 3 modules -as summarized in table 1:

1. The Pre-processing module can consist in:

• The weighted mix of the multiple microphone signals in which channel weights depend on SNR, estimated either using the speech activity detector (Weighted Mix -SAD) or by applying the noise spectrum algorithm (Weighted Mix -Noise spectrum).

• The unweighted mix of the multiple microphone signals (Mix). 2. A speaker diarization module, which can be based on the LIA, LIUM or CLIPS systems. 3. The LIA ReSegmentation module since different studies have shown that a resegmentation phase leads to performance improvement [START_REF] Gauvain | Audio partitioning and transcription for broadcast data indexation[END_REF][START_REF] Reynolds | The Lincoln speaker recognition system: NIST EVAL[END_REF][START_REF] Adami | A new speaker change detection method for two-speaker segmentation[END_REF][START_REF] Meignier | Alize, a free toolkit for speaker recognition[END_REF].

Experiments

This section presents the protocols and results obtained by the different techniques proposed in this paper and submitted to the RT'05S evaluation campaign.

Protocols

For RT'05S, the speaker diarization task was evaluated on two subdomains: recordings issued from conference rooms (similar to RT'04S) and from lecture rooms.

As for any evaluation campaign, two corpora were available:

• a development corpus: composed of RT'04S development and evaluation corpora (12 meetings of about 10mn each), plus some additional meetings including new recording sites. • two evaluation corpora, one for each subdomain, composed of 10 meetings for the conference subdomain of about 10mn each and 29 meetings for the lecture subdomain of about 3mn each.

In this paper, only the RT'04S data (development and evaluation) is used as the development corpus, and will be referred to as dev corpus in the next sections. On the other hand, the RT'05S evaluation data will be referred to as eva-conf for conference data and eva-lect for lecture data in the next sections.

Analysis of the different corpora leads to the following observations. Regarding the dev corpus, we may note that:

• short silence periods, which implies some difficulties to estimate the noise spectrum or the noise energy; • low SNR (minimum average SNR -5.4 dB; 23.75% of files with SNR < 0 dB and 65% of files with SNR < 3 dB); • a variable recording level and a bad use of the input scale (a file with a maximum level of 2% of scale and 58.75% of files with a maximum level <50% of scale); • several speakers with overlapped speaking segments.

A similar observation can be made on the evaconf corpus (same subdomains):

• short silence periods, with similar consequences; • low SNR (minimum average SNR -1.95 dB; 7.5% of files with SNR < 0 dB and 6.2% of files with SNR < 3 dB); • a variable recording level and a bad use of the input scale (a file with a maximum level of 11% of scale and 35% of files with a maximum level <50% of scale); • several speakers with overlapped speaking segments.

Finally, the evalect corpus reveals some marginal characteristics, enforced by the shortness of the excerpts:

• low SNR (minimum average SNR -2.1 dB; 6.2% of files with SNR < 0 dB and 15.17% of files with SNR < 3 dB); • predominantly one speaker per record;

• better use of input signal scale.

Results and discussion

SAD task. Table 2 shows the performance of the Speech Activity Detection system on the evaconf and evalect corpora in terms of Missed Speaker Error (MiE) and False Alarm Speaker Error (FaE) rates.

We can observe that the SAD obtains comparable performance on the evaconf and evalect test sets but presents, on both, large Missed Speaker Error rates (≈5.4%). For comparison, the best SAD system has obtained about 5% in terms of both Missed and False Alarm Speaker error rates during RT05s.

Speaker diarization task. All the submitted speaker diarization systems have been evaluated on the dev corpus as presented in Table 3. Here, the system performance is expressed in terms of Missed speaker Error (MiE), False Alarm speaker Error (FaE) and Speaker Diarization Error (SDE) rates. Details on each meeting are provided as well as the global performance on the dev corpus.

Two directions are evaluated: the pre-processing algorithms on the LIA speaker diarization system (WMixSpectrum, WMix and MixLIA) and the robustness of broadcast news speaker diarization systems on the Meetings recordings (MixLIA, MixCLIPS and MixLIUM ).

The use of the multi-channel information (WMixSpectrum and WMix ), extracted thanks to the pre-processing techniques does not improve globally the speaker diarization performance on the dev corpus but obtains very close results from the baseline system (simple sum of the multiple microphone signals: MixLIA). Nevertheless, signal analysis shows that the pre-processing algorithms improve the global SNR of resulting "virtual" signals; for example, in the case of LDC 20011121-1700 meeting the unweighted mix leads to a global SNR of -3.88 dB (SNR∈ [-10.1;2]dB) to be compared with -0.1 dB (SNR∈ [-6.2;5.69]dB) for Weighted Mix -SAD algorithm and with -0.59 dB (SNR∈ [-5.0;5.34]dB) for Weighted Mix -Noise spectrum algorithm. The improvement of the SNR on the unique "virtual" signal does not seem to be helpful for the speaker diarization systems. Table 4 presents the results obtained globally on the RT'05S evaluation test Conference (eva-conf corpus) and Lecture Room (eva-lect corpus) recordings. It is important to note that the WMixSpectrum system has not been applied on the Conference subdomain test set because of lack of time.

In the case of RT'05S evaluation, the best results have been obtained using the two proposed pre-processing techniques on the contrary of the RT'04S Meeting data (dev corpus). Comparison between the simple unweighted sum method and the weighted ones shows a gain of 15% (in relative) on the eva-conf corpus and of 56% on the eva-lect. The better quality of RT'05S data, in terms of SNR, can explain this performance gain for weighted sum based systems; in fact the same SNR gain observed on both RT'04S and RT'05S does not have the same influence in terms of speaker diarization performance according to the initial signal quality. This result tends to demonstrate the reliability of the proposed strategy: design of a "virtual" signal based on a weighted sum of the multiple microphone recordings.

Concerning the robustness of the different speaker diarization systems against environment changes, it may be observed that their overall performance has significantly decreased on meeting data (about 21% Speaker Error rate) compared with broadcast news (about 12% Speaker Error rate [START_REF] Galliano | The ester phase ii evaluation campaign for the rich transcription of french broadcast news[END_REF]), even though it is often difficult to compare results obtained on different databases.

Unfortunately, pre-processing techniques applied on multiple microphone signal do not seem to be sufficient to deal with meeting data issues and to avoid specific speaker diarization systems.

Conclusions

This paper is concerned with the speaker diarization task in the specific context of meeting data. More precisely, the focus is made on the handling of multiple microphone signals available per meeting. In this framework, a novel approach is experimented based on the rebuilding of a unique and virtual signal, composed of the most pertinent portions of signals issued from the different multiple microphone recordings. The extraction of these pertinent portions is based on signal quality index based on the signal to noise ratio estimate.

Coupled with different speaker diarization systems developped by three different labs: the LIA, LIUM and CLIPS, the proposed approach has been submitted for the NIST 2005 Spring Rich Transcription evaluation campaign (RT'05S). According to the results obtained on the RT05s evaluation, the use of this preprocessing strategy, which takes benefits of the multi-channel information, seems to have a slight positive influence on the speaker diarization performance.

This study was also focused on the behavior of speaker diarization systems, tuned on broadcast news and tested on meeting data. One assumption was that the application of the pre-processing techniques and the production of the unique and "virtual" signal would be sufficient to ensure similar performance between broascast news and meeting corpora. Nevertheless, the level of performance is quite different between both of them. Even though the pre-processing techniques proposed in this paper may be still improved to provide more pertinent "virtual" signal, further investigation has to be done to study the other particularities of the meeting data (like spontaneous speech, overlap, ...) which take a large responsability in the speaker diarization system perturbations.

Table 1 .

 1 Proposed diarization systems

	Systems	Pre-processing	Seg/Re-Seg
	WMixSpectrum Weighted Mix -Noise spectrum LIA/LIA
	WMix	Weighted Mix -SAD	LIA/LIA
	MixLIA	Mix	LIA/LIA
	MixCLIPS	Mix	CLIPS/LIA
	MixLIUM	Mix	LIUM/LIA

Table 2 .

 2 Results of SAD on RT'05S

	Task	MiE FaE
	eva -conf 5.3 2.1
	eva -lect 5.4 1.2

Table 3 .

 3 Results on development corpus (dev)

	Meetings 4	SAD WMixSpectrum WMix MixLIA MixCLIPS MixLIUM MiE FaE SDE SDE SDE SDE SDE
	CMU 20020319-1400 0.5 5.5	57.9	57.9	57.9	46.9	46.9
	CMU 20020320-1500 0.1 5.3	20.2	20.2	20.2	18.5	18.5
	ICSI 20010208-1430 0.4 3.1	16.5	17.0	19.3	22.5	13.4
	ICSI 20010322-1450 0.4 1.4	19.6	13.6	16.7	17.0	24.6
	LDC 20011116-1400 0.4 2.9	4.5	15.4	8.0	6.9	7.8
	LDC 20011116-1500 0.4 1.6	18.7	12.2	8.1	15.8	13.3
	NIST 20020214-1148 0.2 8.1	25.4	16.8	17.3	22.8	27.2
	NIST 20020305-1007 0.0 3.5	33.0	47.8	44.6	9.4	19.0
	CMU 20030109-1530 0.1 0.7	34.2	34.2	34.2	27.9	32.2
	CMU 20030109-1600 2.5 1.3	33.5	33.5	33.5	20.7	33.5
	ICSI 20000807-1000 0.0 3.6	21.2	17.1	16.2	17.1	16.3
	ICSI 20011030-1030 0.0 3.4	41.4	37.0	32.3	51.8	49.4
	LDC 20011121-1700 0.0 2.2	32.0	6.7	3.3	28.7	39.6
	LDC 20011207-1800 0.0 8.6	26.5	40.3	44.2	35.7	34.7
	NIST 20030623-1409 0.0 1.1	18.9	18.4	24.7	30.5	11.6
	NIST 20030925-1517 0.4 16.3	64.3	52.0	51.8	70.7	48.6
	Global performance 0.3 4.1	27.8	26.6	26.2	25.7	26.0

Table 4 .

 4 Results on RT'05S Meeting (on eva-conf and eva-lect corpora)

	Show	SAD WMixSpectrum WMix MixLIA MixCLIPS MixLIUM MiE FaE SDE SDE SDE SDE SDE
	eva-conf 4.0 3.0	-	24.5	27.7	25.0	30.5
	eva-lect 5.6 1.3	18.4	21.4	34.2	35.3	20.0

No multi-channel processing is applied on the first two CMU meetings since only one channel was available