Imene Zangar
email: imene.zangar@enit.utm.tn

Zied Mnasri
email: zied.mnasri@enit.utm.tn

Vincent Colotte
email: vincent.colotte@loria.fr

Denis Jouvet
email: denis.jouvet@loria.fr

Amal Houidhek
email: amal.houidhek@loria.fr

Duration modeling using DNN for Arabic speech synthesis

Keywords: Arabic TTS, phoneme duration modeling, HTS, MERLIN, DNN

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Text-to-speech synthesis (TTS) has become a useful component in many voice applications, such as online translators and text message readers. Furthermore, TTS is nowadays available for most widely spoken languages all over the world on the main online services. Hence, it is important to have high quality TTS for Arabic language since it represents a large market with more than 300 million potential users.

Following the development of the PSOLA technique [START_REF] Moulines | Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones[END_REF], concatenative TTS has been the dominant method as it made it possible to build automatically a good-quality synthetic voice using a relatively small corpus with an appropriate segmentation, mainly into diphones [START_REF] Moulines | Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones[END_REF]. While concatenating the adjacent segments, spectrum smoothing is performed using time or frequency-domain pitch synchronous overlap and add (TD/FD-PSOLA).

Less than a decade after, a higher quality of TTS was obtained by unit selection [START_REF] Hunt | Unit selection in a concatenative speech synthesis system using a large speech database[END_REF]. Unit selection TTS is based on the selection and the concatenation of original speech segments, such as phonemes, diphones or demisyllables [START_REF] Black | Optimal utterance selection for unit selection speech synthesis databases[END_REF]. The selection and the concatenation of units are based on minimizing the weighted sum of two costs namely the target cost, that is the difference between a candidate unit and the target, and to the concatenation cost, which evaluates the quality of joining consecutive units [START_REF] Hunt | Unit selection in a concatenative speech synthesis system using a large speech database[END_REF]. Hence, unit selection does not require any prosodic or spectral modification of the selected units, which leads to a very high level of naturalness of the generated speech. However, a huge database is required to provide units covering most of the speech units, unless the TTS system is designed for a specific application.

In the last decade, taking advantage of the success of stochastic speech modeling, especially for automatic speech recognition [START_REF] Young | The HTK HMM toolkit: Design and philosophy[END_REF], statistical parametric speech synthesis (SPSS) based on hidden Markov models (HMM) was successfully developed [START_REF] Yoshimura | Simultaneous modeling of spectrum, pitch and duration in hmm-based speech synthesis[END_REF]. This approach has many advantages in comparison to the concatenative speech synthesis technique such as stable quality of speech synthesis, robustness for speaker adaptation [START_REF] Yamagishi | Robust speaker-adaptive HMM-based text-tospeech synthesis[END_REF] and the flexibility to change voice characteristics [START_REF] Zen | Statistical parametric speech synthesis[END_REF]. This technique is the backbone of HTS system [START_REF] Yoshimura | Simultaneous modeling of spectrum, pitch and duration in hmm-based speech synthesis[END_REF], which has successfully been transformed into a multilingual TTS system [START_REF] Zen | Statistical parametric speech synthesis[END_REF].

Since a few years, deep learning and mainly deep neural network (DNN) technique has been growing so fast that it becomes the must-have in most data-driven systems. DNN allows modeling large data sets with high accuracy. Therefore, DNN have been recently used to design new generation TTS systems [START_REF] Zen | Statistical parametric speech synthesis using deep neural networks[END_REF], such as MERLIN [START_REF] Wu | Merlin: An open source neural network speech synthesis system[END_REF] and WAVENET [START_REF] Oord | Wavenet: A generative model for raw audio[END_REF].

As far as Arabic speech is concerned, Arabic TTS systems have been developed since the beginning of TTS technology. Unit selection TTS was successfully adapted to Arabic [START_REF] Abdelmalek | High quality Arabic text-tospeech synthesis using unit selection[END_REF] as well as HTS [START_REF] Abdel-Hamid | Improving Arabic HMM based speech synthesis quality[END_REF]. Recently, a new set of linguistic features, that takes care of Arabic phonemes specificities, such as vowel quantity and consonant gemination, has been successfully introduced into the HTS system, to better fit it to Arabic language [START_REF] Houidhek | Statistical modelling of speech units in HMM-based speech synthesis for Arabic[END_REF].

HTS prosody modeling is quite general. Indeed, though HTS has been successfully adapted for many languages, including Arabic [START_REF] Abdel-Hamid | Improving Arabic HMM based speech synthesis quality[END_REF], using almost the same set of features, the quality of the generated speech is usually less appreciated than unit-selection-generated speech. Actually, HTS suffers mainly from uniformly-distributed state/phoneme durations and over-smoothed F0 contours. To cope with such issues, it would be interesting to look for more accurate prosody models, possibly more language specific. Note that MERLIN [START_REF] Wu | Merlin: An open source neural network speech synthesis system[END_REF], which is based on DNN modeling, relies also on generic approaches, whatever the language is.

Modern Standard Arabic (MSA), which is widely used among all Arab-speaking countries as the official and literary language, has 28 consonants and three vowels, /a/, /u/ and /i/ [START_REF] Newman | The phonetics of Arabic[END_REF]. Most consonants could be geminated (doubled) which is indicated in writing through adding the specific diacritic sign (shadda) whereas each vowel has a short and a long version. Vowel quantity affects the meaning of the word, e.g. /ذهبdahaba/ (he went) and /ذهباdahaba:/ (they went (when concerning two people)) [START_REF] Baloul | Développement d'un système automatique de synthèse de la parole à partir du texte arabe standard voyellé[END_REF]. Consonant gemination is another important phenomenon in MSA, which may also modify the meaning, e.g. /darasa/ (he studied) and /darrasa/ (he taught). Both phenomena should be considered while modeling prosody, and particularly duration. Recently both phenomena were successfully taken into account in Arabic TTS using HTS, which was proved by subjective listening tests [START_REF] Houidhek | Statistical modelling of speech units in HMM-based speech synthesis for Arabic[END_REF].

Therefore, this paper investigates the modeling of phoneme duration for Arabic language. Several modeling approaches based on DNN, and including recent advances such as LSTM and BLSTM architectures [START_REF] Zen | Unidirectional long short-term memory recurrent neural network with recurrent output layer for lowlatency speech synthesis[END_REF], are studied and analyzed on four important phoneme classes: short vowels, long vowels, simple consonants, and geminated consonants. Then a class-specific modeling is achieved by using for each class the approach that is performing the best in a given development set. This class-specific approach is compared to the duration modeling achieved by well-known TTS toolkits, such as HTS (HMM-based modeling) and MERLIN (DNNbased modeling). A fourth modeling, based on artificial neural networks (ANN), which was previously proposed for Arabic [START_REF] Mnasri | Modeling Segmental Duration by Statistical Learning for an Arabic Text-to-Speech System[END_REF], is also included in the comparisons.

The rest of the paper is organized as follows. Section 2 describes duration modeling using HMM and DNN. Section 3 presents the LSTM and bidirectional-LSTM architectures developed for phoneme duration modeling for Arabic TTS. Section 4 details the experimental results with the associated objective evaluations. Finally, discussion and conclusion are presented in section 5.

Duration modeling

Prosodic parameters, i.e. duration, F0 and intensity are the physical manifestations of phonological phenomena of speech. Particularly, duration plays a major role (a) to define the speech rhythm and accentuation and (b) to preserve the speech meaning, as for some languages, such as Arabic, a long or a short vowel changes the meaning of the word. Therefore, an accurate duration model is needed to ensure that the synthesized speech is well perceived.

Duration modeling for speech synthesis has been the subject of many studies, such as in [START_REF] Klatt | Linguistic uses of segmental duration in English: Acoustic and perceptual evidence[END_REF], where segment duration is determined by explicit formulas, considering some theoretical hypothesis, which assume the existence of an inherent duration for every phoneme and the existence of common compression/extension factor for all the phonemes within a syllable. Another explicit model was developed in [START_REF] Van Santen | Prosodic modelling in text-to-speech synthesis[END_REF] where the duration of a segment is calculated as the sum of products of some contextual features.

However, with the ability of machine learning algorithms to lead to accurate models provided the right features are used, recent phoneme duration models are now based on HMM and DNN approaches.

Duration modeling based on HMM

In HTS system, duration modeling is performed using a dedicated module, where state duration distributions are predicted separately. Then the phoneme duration is the sum of the predicted state durations. To model the set of state durations of each phoneme HMM, a multi-dimensional Gaussian distribution is used. The state duration distributions are clustered using a decision-tree-based clustering technique [START_REF] Yoshimura | Duration Modeling in HMM based Speech Synthesis System[END_REF]. The state duration are determined in the synthesis stage by the state duration of the relevant HMM models. The decision trees are constructed by taking into account linguistic features, that include phoneme identity and class (vowel/consonant), phonological features such as stress and accentuation, and positional features like the relative positions of different segment levels (phoneme, syllable, word) inside higher level segments. Finally, the yielded model maps the phone segment linguistic features to the matching duration [START_REF] Yoshimura | Simultaneous modeling of spectrum, pitch and duration in hmm-based speech synthesis[END_REF].

Duration modeling based on DNN

Artificial neural networks (ANN) have been used to model prosodic parameters, particularly segment duration since many years, as in [START_REF] Campbell | Syllable-based segmental duration -Talking machines: Theories, models, and designs[END_REF] for English, [START_REF] Mixdorff | Building an integrated prosodic model of German[END_REF] for German, and [START_REF] Mnasri | Modeling Segmental Duration by Statistical Learning for an Arabic Text-to-Speech System[END_REF] for Arabic. Nevertheless, these models were not too accurate in comparison to HMM, mainly because former ANN input dimension and number of hidden layers were limited.

Meanwhile, DNN have been gaining much interest for their ability to approximate any real continuous function. Therefore, DNN are now the state of the art for speech synthesis systems, such as MERLIN [START_REF] Wu | Merlin: An open source neural network speech synthesis system[END_REF], and Wavenet [START_REF] Oord | Wavenet: A generative model for raw audio[END_REF].

Since segmental duration is a continuous value, DNN are used as a regression tool, which is trained to minimize the root mean squared prediction error (RMSE) [START_REF] Henter | Robust tts duration modelling using dnns[END_REF]. Furthermore, recurrent networks architecture, like long short-term memory (LSTM) and Bidirectional-LSTM (BLSTM) are powerful models in sequential modeling. Therefore, deep LSTM and BLSTM are investigated to model the relationship between linguistic features and phoneme duration [START_REF] Chen | Discrete duration model for speech synthesis[END_REF]. The input features could be the same as those used in HTS, since they cover most of the phonological, linguistic and contextual data with the addition of Arabic specific features regarding vowel quantity and consonant gemination. Input features need to undertake some preprocessing to reduce the data scattering, including normalization and/or saturation of some of the positional input values.

Output duration targets could be the state or the phoneme durations. To enhance the prediction quality of phoneme duration, it is recommended to normalize its distribution, using an adequate transform [START_REF] Rosen | Analysis of speech segmental duration with the lognormal distribution: A basis for unification and comparison[END_REF].

Experiment and result

Experimental environment

In order to train the duration model, a phonetically-balanced MSA corpus was used [START_REF] Halabi | Phonetic inventory for an Arabic speech corpus[END_REF]. It consists of 1597 utterances corresponding to news bulletin read by a native-Arabic male speaker in a neutral style. The speech signals are sampled at 48 KHz with a16-bit precision.

The corpus is divided into three subsets, 1150 utterances for training, 287 for validation and 160 for test. The training set contains 37872 simple consonant occurrences, 23367 short vowel occurrences, 11565 long vowel occurrences, 4040 geminated consonant occurrences and 2458 pause segments. As mentioned before, input linguistic features and output duration targets have been pre-processed as required. Here we are using the same set of input features as proposed for HTS for Arabic synthesis in [START_REF] Houidhek | Statistical modelling of speech units in HMM-based speech synthesis for Arabic[END_REF], for example the identity of the two previous phoneme and the two next phoneme of the actual phoneme, the position of actual phoneme in the syllable, word and phrase. Actually, the type of an input feature can be binary, like stressed/not-stressed, discrete like the phoneme identity, or numeric like the phoneme position. All the input features are encoded into a 445-coefficient vector that includes all the binary and numerical features. Target duration outputs, i.e. phoneme durations have been analyzed to check their distributions, and a log-transform has been applied to normalize the values [START_REF] Rosen | Analysis of speech segmental duration with the lognormal distribution: A basis for unification and comparison[END_REF]. Pauses LSTM all the phonemes 3 LSTM layers with 1024, 512 and 512 units.

Implemented duration models

Since our work is focused on modeling phoneme duration with DNN, several architectures have been implemented. This includes, feed-forward DNN using only dense layers, and recurrent DNNs based on LSTM and on BLSTM layers. For each model, various numbers of hidden layers, of nodes and of activation functions have been tried. The RMSprop optimizer was adopted in the experiments, as well as early stopping to avoid the over-fitting problem. But above all, the novelty of this work consists in determining the best model for each class of phonemes, considering two major characteristics of Arabic speech, i.e. vowel quantity and consonant gemination. Therefore, each type of neural network, i.e. DNN, LSTM and BLSTM has been trained on several subsets of data. For each class of sounds (e.g., simple consonants, consonants, …), a specific model is trained using only the segments corresponding to that class. Actually, 8 classes are considered: all phonemes including pauses between words, all phonemes only (i.e., without pauses), all consonants only, all vowels only, simple consonants, geminated consonants, short vowels, and long vowels. Hence, for each class, different models have been trained and evaluated, and the architecture leading to the most accurate prediction on the development set was selected, see Table 1. As the size of training corpus is different from one class to another one, this may explain why it is not the same model architecture that leads to the most accurate prediction of phoneme durations on the different classes of sounds. Consequently, in the following, we define the classspecific modeling as the fact of using, for predicting the duration of each sound, the model which is the most accurate for the corresponding class, as defined in Table 1.

Objective evaluation

The objective evaluation consists in comparing the performance of the class-specific DNN modeling to state-ofart models, i.e. HMM model as used in HTS, DNN model as used in MERLIN, and a former ANN model developed for Arabic [START_REF] Mnasri | Modeling Segmental Duration by Statistical Learning for an Arabic Text-to-Speech System[END_REF]. The DNN model as used in MERLIN is composed by 6 hidden layers with 1024 units each and tanh as activation transfer function. This model relies on the same set of features as HTS. The ANN model from [START_REF] Mnasri | Modeling Segmental Duration by Statistical Learning for an Arabic Text-to-Speech System[END_REF], contains 2 hidden layers with 26 units each, and uses sigmoid and tanh as activation functions. This model does not use exactly the same set of linguistic features as HTS. Evaluation measures are reported for each class of sounds in Table 2, and globally in Table 3. Table 2 shows that for each phoneme class, a class specific DNN modeling enhances the prediction accuracy of the phoneme duration, on the test set data, as measured by the various criteria: root mean square error (RMSE), mean absolute error (MAE) and correlation coefficient between original and predicted duration. Results show that for each class of sounds, the novel class-specific DNN modeling performs better than the HMM modeling from HTS, the DNN modeling from MERLIN, and the former ANN model: the root mean square error and the mean absolute error are lower, and the correlation between predicted and reference duration values is higher. When the accuracy are computed globally, i.e., on all the classes of sounds, as in Table 3, results show that using a class-specific modeling leads to a global improvement, compared to state of the art models, when all the phonemes are considered together in the evaluation, and also when considering all the phonemes and the pauses. This confirms that optimizing the duration modeling on the development set, for each class separately, allows achieving the overall best performance on the test set.

Figure 1 shows the distributions of phoneme durations for the original and predicted values on the test set, for each class of phonemes. Results show a good match for simple consonants, short vowels and long vowels. For the geminated consonants, the distribution of the predicted values is sharper, i.e. has a lower standard deviation, and slightly shifted towards higher durations. This means that the model slightly over-estimates the durations of long vowels.

Discussion and conclusions

This paper has investigated the modeling of the duration of the Arabic sounds for text-to-speech synthesis. Various DNNbased architectures have been developed and evaluated. Each model have been trained on various subsets of the training data corresponding to classes of sounds, as for example training on all phoneme segments, training on vowel segments only, training on short vowel segments only, etc. The various modeling architectures trained on various subsets of sounds have been compared on the development data. It appears that it is not the same model and training data, which leads to the best prediction accuracy on the various classes of phonemes (short vowels, long vowels, simple consonants and geminated consonants). This led us to define a class-specific modeling approach, which for each sound, uses the model that performs the best on the validation set. This class-specific modeling approach has been compared on the Arabic test set, to several state of the art modeling approaches, as the HMM-based modeling from the HTS toolkit and the DNN-based modeling from the MERLIN toolkit. Objective evaluations show that the proposed approach leads to a better prediction of the sound durations. The next steps will consist in integrating this classspecific duration modeling into state of the art TTS toolkits, in order to produce corresponding speech signals necessary for subjective evaluation through listening tests.

Figure 1 :

 1 Figure 1: Comparison of the distributions of the phoneme durations between original and predicted values.

Table 1 :

 1 Description of the model architecture leading to the best accuracy on the development set, for each phoneme class, and for pauses.

	Phoneme	Model	Training	Model
	Class		set	description
				2 Dense layers
				with 512 units,
	Simple consonant	DNN-BLSTM	simple consonant	activation function tanh, plus 2
				BLSTM layers
				with 128 units.
				2 Dense layers
				with 16 units,
	Geminated	DNN-	geminated	activation
	consonant	BLSTM	consonant	function tanh,
				plus 2 BLSTM
				layers with 16
				units.
				2 Dense layers
				with 512 units,
	Short	DNN-	short	activation
	vowel	BLSTM	vowel	function tanh,
				plus 2 BLSTM
				layers with 128
				units.
				2 DNN layers
	Long vowel	DNN	long vowel	with 512 and 256 units,
				activation
				function tanh.

Table 2 :

 2 Comparison of RMSE, MAE and correlation between predicted durations and reference durations on test set, for each phoneme class and for the variousmodeling approaches.

	Phoneme	Duration	RMSE	MAE	Corr
	class	modeling	(ms)	(ms)
		HMM from HTS	25	18	0.76
	Simple	Class-specific DNN	25	17	0.77
	consonant	DNN-MERLIN	26	18	0.75
		ANN from [17]	35	25	0.50
	Geminated consonant	HMM from HTS Class-specific DNN DNN from MERLIN	43 42 54	31 32 40	0.43 0.51 0.15
		ANN from [17]	62	50	0.42
	Short vowel	HMM from HTS Class-specific DNN DNN from MERLIN	22 22 26	16 16 19	0.82 0.84 0.81
		ANN from [17]	26	19	0.78
	Long vowel	HMM from HTS Class-specific DNN DNN from MERLIN	49 40 54	34 28 38	0.68 0.77 0.66
		ANN from [17]	68	52	0.07
		HMM from HTS	109	73	0.54
	Pauses	Class-specific DNN	109	70	0.54
		DNN from MERLIN	146	110	0.60
		ANN from [17]	188	158	0.56

Table 3 :

 3 Comparison of RMSE, MAE and correlation between predicted durations and reference durationson test set, for the various modeling approaches.

	Phoneme	Duration	RMSE	MAE	Corr
	Class	modeling	(ms)	(ms)
		HMM from HTS	30	20	0.83
	All	Class-specific DNN	28	19	0.85
	phonemes	DNN from MERLIN	33	22	0.80
		ANN from [17]	41	28	0.66
	All	HMM from HTS	40	24	0.93
	phonemes	Class-specific DNN	39	22	0.93
	+pauses	DNN from MERLIN	50	28	0.92
		ANN from [17]	63	37	0.87

Acknowledgements

This research work was conducted under PHC-Utique Program, grant N°15G1405.


Sami Sieranoja

Md Sahidullah

Tomi Kinnunen

Jukka Komulainen

Abdenour Hadid

Audiovisual Synchrony Detection with Optimized Audio Features

Keywords: Audiovisual Synchrony, Presentation Attack Detection, Multimodal Processing, Feature Extraction, Mel-Frequency Cepstral Coefficients (MFCCs)

Audiovisual speech synchrony detection is an important part of talking-face verification systems. Prior work has primarily focused on visual features and joint-space models, while standard mel-frequency cepstral coefficients (MFCCs) have been commonly used to present speech. We focus more closely on audio by studying the impact of context window length for delta feature computation and comparing MFCCs with simpler energy-based features in lip-sync detection. We select state-of-the-art hand-crafted lip-sync visual features, spacetime auto-correlation of gradients (STACOG), and canonical correlation analysis (CCA), for joint-space modeling. To enhance joint space modeling, we adopt deep CCA (DCCA), a nonlinear extension of CCA. Our results on the XM2VTS data indicate substantially enhanced audiovisual speech synchrony detection, with an equal error rate (EER) of 3.68%. Further analysis reveals that failed lip region localization and beardedness of the subjects constitutes most of the errors. Thus, the lip motion description is the bottleneck, while the use of novel audio features or joint-modeling techniques is unlikely to boost lip-sync detection accuracy further.

I. INTRODUCTION

Nowadays, most mobile devices are equipped with a microphone and a front-facing video camera, enabling nonintrusive audiovisual user authentication. Although integrated face and voice modalities can increase recognition accuracy, both are highly vulnerable to presentation attacks (spoofing attacks) [START_REF] Marcel | Handbook of Biometric Anti-Spoofing: Trusted Biometrics Under Spoofing Attacks[END_REF]. For instance, presentation of prerecorded audio clip (replay) together with a still photograph is enough to circumvent talking face verification relying on late fusion [START_REF] Bredin | Making talking-face authentication robust to deliberate imposture[END_REF]. Multi-biometrics by itself is not inherently robust to spoofing attacks since successful spoofing of just one modality may compromise the entire system [3].

One approach to counter audiovisual presentation attacks is to independently validate face [4] and voice [5] liveness. Another approach is to determine whether the content and timing of the captured audible and visual speech match (see Fig. 1). Such audiovisual speech synchrony detection can be performed using both text-independent [START_REF] Bredin | Making talking-face authentication robust to deliberate imposture[END_REF], [6], [START_REF] Boutellaa | Audiovisual synchrony assessment for replay attack detection in talking face biometrics[END_REF], [START_REF] Eveno | A speaker independent "liveness" test for audio-visual biometrics[END_REF], [START_REF] Slaney | Facesync: A linear operator for measuring synchronization of video facial images and audio tracks[END_REF], and text-dependent [START_REF] Kollreider | Realtime face detection and motion analysis with application in "liveness" assessment[END_REF], [11] methods. The former are effective in detecting attacks whereby the attacker uses separate audio and video recordings (or photo) of the target person. These methods, however, are powerless under pre-recorded video replay attacks with synchronized audiovisual speech. Text-dependent synchrony assessment methods tackle this issue by utilizing challenge-response approach by prompting the user a randomly selected sentence [START_REF] Kollreider | Realtime face detection and motion analysis with application in "liveness" assessment[END_REF], [11] (challenge) and then verifies whether the preassigned utterance can be recognized in both modalities within a specific time window (response).

Audiovisual speech synchrony studies largely focus on visual features and joint-space models, leaving an open question whether, and how much, improvement could be obtained by optimization of the audio features. Standard melfrequency cepstral coefficients (MFCCs) are commonly used (with some exceptions [START_REF] Slaney | Facesync: A linear operator for measuring synchronization of video facial images and audio tracks[END_REF]). Speech activity detection (SAD) is another audio front-end component whose usefulness has not been directly addressed in prior work. For these reasons, it is difficult to tell what are the bottlenecks in the existing lip-sync detection pipelines, i.e. the used audio or visual features, or the joint modeling and synchrony detection of the two modalities.

In this work, we study the use of energy and delta features that we believe to be useful for audio-visual synchrony detection. We fix the visual features to space-time autocorrelation of gradients (STACOG) and the joint-space analysis to canonical correlation analysis (CCA), that have formed the state-of-the-art in audiovisual speech synchrony detection [START_REF] Boutellaa | Audiovisual synchrony assessment for replay attack detection in talking face biometrics[END_REF], [12]. To gain further insight into the importance of feature choice versus joint-space modeling, we consider deep CCA (DCCA), a nonlinear extension of CCA [13]. Finally, we analyze the misclassified test cases with an aim to explain the reasons behind these errors in audiovisual synchrony detection task.

II. RELATED WORK

Audiovisual speech synchrony detection studies focus primarily on back-end synchrony measures between audio and video. MFCCs [14] are commonly used to present speech [START_REF] Boutellaa | Audiovisual synchrony assessment for replay attack detection in talking face biometrics[END_REF], [15], [16], [START_REF] Aides | Text-dependent audiovisual synchrony detection for spoofing detection in mobile person recognition[END_REF], with a few alternatives such as frame energy [START_REF] Hershey | Audio vision: Using audiovisual synchrony to locate sounds[END_REF]. For video, discrete cosine transform (DCT) [6], lip measurements [16] and multi-channel gradient model (MCGM) [19] are commonly used. In [START_REF] Boutellaa | Audiovisual synchrony assessment for replay attack detection in talking face biometrics[END_REF], STACOG were found to outperform DCT features in measuring audio and visual speech correlation. Very recently, also convolutional neural network (CNN) architectures have been proposed for both audio and visual speech feature extraction [START_REF] Chung | Out of time: Automated lip sync in the wild[END_REF], [21]. As a back-end, CCA is the default choice [START_REF] Slaney | Facesync: A linear operator for measuring synchronization of video facial images and audio tracks[END_REF], with alternatives such as coinertia analysis (CoIA) [START_REF] Eveno | A speaker independent "liveness" test for audio-visual biometrics[END_REF], generalized bimodal linear prediction [16], kernel CCA [19] and deep neural networks (DNNs) [START_REF] Aides | Text-dependent audiovisual synchrony detection for spoofing detection in mobile person recognition[END_REF], [START_REF] Chung | Out of time: Automated lip sync in the wild[END_REF] where it has demonstrated superior performance and computation efficiency over similar methods [START_REF] Kobayashi | Motion recognition using local auto-correlation of space-time gradients[END_REF].

Following up [START_REF] Boutellaa | Audiovisual synchrony assessment for replay attack detection in talking face biometrics[END_REF], [12], we use dlib 1 to determine eye and mouth locations in every video frame, following the strategy of [24] to obtain a good approximation of the whole mouth region. The resulting rectangular mouth image is resized to 70 × 40 pixels from 1584-dimensional STACOG features using default parameters 2 . The features represent dynamics between three consecutive video frames (excluding endpoints).

B. Audio features

MFCCs [14] are computed from short-term discrete Fourier transform (DFT). If we denote the DFT power spectrum of one frame as a vector s ∈ R N/2+1 , where N is the size of discrete Fourier transform (after zero padding), MFCC vector is c = D log(Hs), where H ∈ R Q×(N/2+1) is a filterbank matrix containing Q filter responses, log(•) operates element-wise and D is a DCT matrix.

Prior to MFCC extraction, we downsample the audio to 8 kHz as most voiced speech energy lies below 4 kHz. Speech activity detection [START_REF] Sohn | A statistical model-based voice activity detection[END_REF] is optionally used to eliminate silence at the beginning and end of utterances. Then 20 MFCCs are extracted using Q = 20 filters from 40 ms frames with no frame overlap 3 . To measure local spectral dynamics [START_REF] Soong | On the use of instantaneous and transitional spectral information in speaker recognition[END_REF], we append MFCCs with their delta features. If c t denotes the MFCCs of t th frame, the corresponding delta vector using a time context of f = 2τ + 1 frames is ∆c t = τ m=-τ mc t+m . Double deltas, ∆ 2 c t , are computed 1 http://dlib.net/ 2 https://staff.aist.go.jp/takumi.kobayashi/codes.html#STACOG 3 This somewhat unconventionally long speech window is chosen to match the 40ms associated with the 25fps video frame rate (1/25fps=40ms).

C. Joint-space modeling

As STACOG features are produced at 25fps starting from the third video frame, the first two audio feature vectors are excluded to synchronize the two modalities. To measure the degree of synchrony between speech and video features, we use the same CCA [27] joint-modeling and synchrony measures as described in [START_REF] Boutellaa | Audiovisual synchrony assessment for replay attack detection in talking face biometrics[END_REF], [12]. Given two multidimensional random variables X and Y , CCA finds orthogonal linear projections w i and z i that are maximally correlated and ordered from highest to lowest correlating:

(w * i , z * i ) = arg max (wi,zi) corr(Xw i , Y z i ) (1)
Training consists of computing CCA projection matrices (X and Y ) for both modalities (Fig. 2). In the test phase, these are used to project audio and video features to a common space. For each test video, correlation between audio and video is calculated for each dimension of the projected features. The synchrony score is the sum of the K highest correlating dimensions for all the frames:

S W,Z (X, Y ) = 1 K K k=1 |corr(Xw k , Y z k )|, (2)
where K is a control parameter.

For more advanced joint-space modeling, we consider deep CCA (DCCA) [13]. It works the same way as CCA, but Xw i and Y z i (in Eq. 1) are replaced with nonlinear functions f X (X; θ w ) and f Y (Y ; θ z ), where for v ∈ {x, y}, f v is a deep feedforward neural net of L layers with parameters θ v consisting of the weights and biases of all the layers. The network is trained with backpropagation on the gradient of correlation; for more details, see [13].

IV. EXPERIMENTAL SETUP

A. Dataset and evaluation protocol

To provide comparable results with [START_REF] Boutellaa | Audiovisual synchrony assessment for replay attack detection in talking face biometrics[END_REF] and [12], we use the XM2VTS dataset [28] to evaluate the lip-sync detectors. XM2VTS contains audiovisual sequences of 295 subjects recorded in four sessions. The audio was recorded at 32 kHz and video at 25 fps. The same sentence ("Joe took fathers green shoe bench is repeated in all videos. Logical access attack scenarios were created by combining audio and video from different sessions of the same person uttering the same sentence, thus high level of synchrony (but not perfect) is present in the audiovisual speech.

B. Evaluation

We measure the performance of audiovisual synchrony detection based on the score S W,Z (X, Y ) using equal error rate (EER), the operating threshold with equal miss and false alarm rates 4 . We follow exactly the same evaluation protocol as in [START_REF] Boutellaa | Audiovisual synchrony assessment for replay attack detection in talking face biometrics[END_REF], [12], where the dataset is split into two equal size subject-disjoint halves. The lip-sync detection models are trained on real videos of one group and the resulting model is evaluated on the other group. The process is repeated by alternating the role of the two folds and the reported EER is the average of the two tests. The parameter K is optimized for both folds separately, with optimal K for first fold used for scoring the second fold, and vice versa.

V. RESULTS

A. Effect of audio feature configuration

The results for audio feature optimization with CCA backend are illustrated in Figs. 3 and4, correspondingly for MFCCs and energy features (sum of mel filterbank output). A summary is provided in Table I along with the baseline method [12] in the first row. The best result of EER 3.68% is obtained using MFCCs with deltas and double deltas and three-frame context, though performance with 5-frame context is similar. The trends in Figs. 3 and4 are consistent -less frames in delta computation is better. The energy features are behind MFCCs (5.22%>3.68%) as one might expect, but provide nevertheless low EERs. This suggest that most of the detection accuracy (even with MFCCs) results from detecting the synchrony of the amount of sound energy and the lip motion. Inclusion of speech activity detection had very little effect (possibly due to short utterances); with MFCCs with deltas, EER changed from 4.07% to 3.97%, and with delta plus double deltas from 3.70% to 3.68%.

B. Deep CCA back-end

The effect of DCCA parameters on EER is shown in I indicate that DCCA performs better than CCA with MFCC features, having the best result of EER 3.52% and also performing better with other MFCC feature combinations.

DCCA performs worse in combination with the energy features. This might be due to a conflict between the high dimensional visual (D = 1584) STACOG features and the low dimensional (D = 1, 2, 3) energy features leading to overfit. This interpretation is backed by additional experiments with DCCA and energy features where we increased the DCCA regularization parameter from the default 10 -4 to 10 -2 . This improved the results for the 1-dimensional energy-stat features from EER 12.29% to 9.14%, and for Energy-stat+∆+∆ 2 from 6.19% to 5.88%.

C. Error case analysis

The obtained error rates are reasonably low already with classic CCA back-end. To gain further insight into the remaining issues to be solved in audio-visual synchrony detection, we study the errors of CCA with MFCCs with deltas and double deltas (with f =3). We fix the detection threshold to the EER operating point (at 3.68%). The cases that contained synchronous video, but were falsely rejected as asynchronous, 27% contained incorrectly detected mouth positions and 67% of subjects had a beard or moustache -see Fig. 6. In the opposite case of asynchronous video wrongly classified as synchronous, there were no above normal levels of beardedness. In summary, video of a person with a beard or a moustache would be more likely wrongly classified as a spoofing attack. Possible reasons include; (1) as beard covers a part of the mouth, it conceals some lip movements that would be visible on a shaved face; (2) as bearded persons are a minority in the dataset, the amount of training data to model their features might be insufficient.

VI. CONCLUSION

We studied MFCC delta configurations to provide a suitable counterpart for the visual STACOG features, leading to audiovisual speech synchrony detection with an EER 3.68%, surpassing the selected baseline [12]. We additionally used simpler energy measure that yielded an EER of 5.22%, suggesting that most of the recognition accuracy is based on detecting if the amount of speech energy produced is in synchrony with the lip movements. Finally, we found that beardedness of subjects and failed lip region localization explains most errors. Therefore, in conclusion, the lip motion description seems to be the bottleneck, while the use of other audio features or advanced joint-modeling, such as [START_REF] Aides | Text-dependent audiovisual synchrony detection for spoofing detection in mobile person recognition[END_REF], [22], may not increase accuracy further.

Figure 1 .

 1 Figure 1. Synchronous lip region frames and audio spectrograms extracted from phrase Joe took. Audiovisual synchrony detection seeks to detect discrepancy of the two modalities (an indication of a spoofing attack).

  , [21], [22]. III. METHODOLOGY A. Video features STACOG [23] is a motion feature extraction method that encodes the local auto-correlations of space-time gradients in a video for capturing the local geometric characteristics of moving objects. Originally, STACOG has been successfully used for e.g. hand gesture and human action recognition

Figure 2 .

 2 Figure 2. Joint-space model using CCA

Figure 3 .

 3 Figure3. Effect of window size for MFCC delta calculation. Best result is obtained with three frames (equal to STACOG). The baseline system has the same computation pipeline as [12], but with revised MFCC extractor and EER estimation.

Fig. 5 .Figure 5 .

 55 Figure 5. Effect of varying the number of latent variables, layers and nodes per layer in DCCA. latent variables, five hidden layers, 2048 nodes per layer, sigmoid activations and perform more extensive training with full batch size. The results included in TableIindicate that DCCA performs better than CCA with MFCC features, having the best result of EER 3.52% and also performing better with other MFCC feature combinations.DCCA performs worse in combination with the energy features. This might be due to a conflict between the high dimensional visual (D = 1584) STACOG features and the low dimensional (D = 1, 2, 3) energy features leading to overfit. This interpretation is backed by additional experiments with DCCA and energy features where we increased the DCCA regularization parameter from the default 10 -4 to 10 -2 . This improved the results for the 1-dimensional energy-stat features from EER 12.29% to 9.14%, and for Energy-stat+∆+∆ 2 from 6.19% to 5.88%.

Figure 6 .

 6 Figure 6.Sample faces from synchronous videos wrongly classified as asynchronous.

ACKNOWLEDGMENTS

The project was partially funded by Academy of Finland and the Finnish Foundation for Technology Promotion.


L Gremillard

C Wei

J Chevalier

K Hans

T Oberbach

A fast, stepwise procedure to assess time-temperature equivalence for hydrothermal ageing of zirconia-based materials

Hydrothermal ageing is one lifetime-limiting phenomenon of zirconia-based ceramics and composites for many applications, from biomedical implants to ferrules and watches. Predicting hydrothermal ageing at use-temperature implies a set of accelerated ageing experiments conducted under water vapour at several, high temperatures (usually between 100 °C and 140 °C). From these data, the activation energy of ageing can be determined, and thus ageing can be predicted at any temperature. However, obtaining precise extrapolations with this procedure requires the use of lower temperatures (70 or 80 °C), leading to a rather long procedure (up to a few thousand hours), and should involve at least one specimen for each test temperature. This article presents a new procedure that allows the determination of all ageing parameters and the estimation of kinetics at usetemperatures with the use of a single specimen within a shorter time, accurate enough for fast screening of new materials.

Introduction

Zirconia ceramics can exist in three crystallographic phases at atmospheric pressure: cubic, tetragonal and monoclinic. The phase transition temperatures depend on many parameters, among which one of the most important is the nature and amount of stabilising oxides present in the ceramics. 3Y-TZP (tetragonal zirconia polycrystal stabilized with 3 mol% Y 2 O 3 ) is the most used monolithic zirconia ceramic for biomedical applications, thanks to a combination of interesting properties: high toughness and strength (up to 6 MPa m 1/2 and 1200 MPa respectively), perfect biocompatibility, white colour and translucency. These good mechanical properties mainly come from Phase Transformation Toughening (PTT) that involves the tetragonalto-monoclinic (t-m) phase transformation around the tip of propagating cracks, the resulting volume increase acting to close the cracks and slow them, reinforcing the material [START_REF] Chevalier | The tetragonal-monoclinic transformation in Zirconia: lessons learned and future trends[END_REF]. However, the same t-m transformation can occur on surfaces in contact with water, and lead to roughening and microcracking [START_REF] Chevalier | Low-temperature degradation of Zirconia and implications for biomedical implants[END_REF]. This degradation phenomenon is called hydrothermal ageing (or Low Temperature Degradation) and is responsible for the decline of 3Y-TZP components in orthopaedics in the 2000 s. To prevent it, several strategies exist. In particular aluminatoughened zirconia (ATZ) presents similar mechanical properties, while being less sensitive to ageing [START_REF] Schneider | Low-temperature aging behavior of alumina-toughened zirconia[END_REF]. ZTA (standing for zirconia-toughened alumina) is also highly developed because of a higher resistance to ageing [START_REF] Chevalier | On the kinetics and impact of tetragonal to monoclinic transformation in an alumina/zirconia composite for arthroplasty applications[END_REF].

Ageing may occur at any temperature where tetragonal phase is metastable (for 3Y-TZP, anywhere below ∼500 °C). Over a certain temperature range (for 3Y-TZP between room temperature and more than 150 °C) it is a thermally activated phenomenon, thus faster at high temperature and slower at low temperature. To assess ageing at room (or body) temperature, it is thus convenient to extrapolate it from experiments made at higher temperature (typically in water and in autoclave, between 80 and 150 °C). The typical procedure thus consists in measuring the evolution of monoclinic fraction with time of the surface of samples exposed to water (or steam) at different temperature, one sample being exposed to only one temperature until saturation of the monoclinic fraction. Then, from the ageing kinetics measured at (at least) two temperatures, an extrapolation to the temperature of interest can be conducted.

Any procedure to extrapolate ageing kinetics at low temperature is based on the fact that hydrothermal ageing is cumulative, and on two hypotheses:

1. Hydrothermal ageing is thermally activated in the temperature range used for the tests, thus the kinetics follow the same laws at 37 °C as at higher temperatures. 2. The ageing exponent, n, is constant vs temperature.

⁎ Corresponding author. E-mail address: laurent.gremillard@insa-lyon.fr (L. Gremillard).

If these hypotheses are satisfied, the hydrothermal ageing kinetic can be expressed as the evolution of the monoclinic volume fraction (V) vs time by:

= - - ⋅ = - V t T b T t t b ( , ) 1 exp [ ( ( ) ) ] with b ( ) exp ( ) n Q RT 0 (1)
Thus the knowledge of the three ageing parameters b 0 , n and Q allows the calculation of V(t,T) for any ageing time and temperature.

The usual method [START_REF] Chevalier | Low-temperature aging of Y-TZP ceramics[END_REF] requires the determination of V(t,T) at different temperatures, and determines analytically b 0 , n and Q. Indeed, on can write Eq. ( 1) as:

⎜ ⎟ ⎜ ⎟ - =- ⋅ ⎛ ⎝ ⎛ ⎝ - ⎞ ⎠ ⎞ ⎠ = + V t T b T t V t T n b T n t ln(1 ( , )) ( ( ) ) or ln ln 1 1 (, ) ln( ( )) ln( ) n (2)
Thus, after Eq. ( 2), for each temperature, plotting

- ( ) ( ) ln ln V t T 1 1 (, )
vs ln(t) gives a straight line of slope n. Then a common value of n is chosen for all temperatures (usually the average of the values obtained at each temperature), and for each temperature a value of b(T) is determined after the intercept of

- ( ) ( ) ln ln V t T 1 1 (, )
vs ln(t) (this intercept is: n•b(T)). Then plotting ln(b(T)) vs 1/T gives again a straight line of intercept ln(b 0 ) and of slope -Q/R. This is a well-established method that gives correct results, but suffers from a few limitations that make it not always easy to use. In particular, it was shown [START_REF] Gremillard | Modeling the aging kinetics of zirconia ceramics[END_REF] that n may vary during a same kinetic. Thus the determination of n and b may depend on the person who makes the calculations. Another, numerical, method starts from the same data (V(t,T) at different temperatures), but minimizes the global error function ε between the measured values and the ones predicted by Eq. ( 1) by optimizing the parameters b 0 , n and Q.

∑ ∑ = ⎛ ⎝ - ⎡ ⎣ - - ⋅ ⎤ ⎦ -⎞ ⎠ = ( ) ( ) ε b t f N 1 exp exp i j N Q RT j n ij i 1 0 2 i i (3)
Eq. (3) presents such an error function, in which each kinetic is given the same weight whatever the number of time points. In Eq. ( 3), i relates to the number of temperatures tested and j to the number of time points for each temperature (for 1 temperature T i , N i time points are measured); f ij are the measured monoclinic fraction at temperature T i and time t j . Many other error functions can be used, the other most frequent one being the one in which each experimental point is given the same weight. Optimization of b 0 , n and Q can be conducted numerically using a solver. This method presents the advantage to optimize all three parameters at once, and is less sensitive to measurement uncertainties. On the other hand, it must be considered with care since it will not directly reveal potential deviations from Eq. ( 1), contrary to the analytical method.

We propose here a new, stepwise method to extrapolate ageing kinetics to room temperature. It is based on ageing measurements at different temperature on a single sample, and can largely reduce the time necessary for correct extrapolations. In its principle it is similar to Locati procedure for the determination of mechanical fatigue limit [7], insofar as it calls on different levels of stress applied on a single sample (the "level of stress" being here a temperature) and on a "theoretical shape" of the physical quantity to characterize (fatigue limit follows a Wohler curve for Locati procedure, and here monoclinic content follows a Mehl Avrami Johnson law (Eq. ( 1)) for hydrothermal ageing). There are of course some differences between the method we propose here and Locati procedure. The main difference is that fatigue tests only rely on one single measurement: the number of cycles to fracture; they don't allow the evaluation of the damage parameter during the test. On the contrary, our procedure allows, and necessitates, the evaluation of the monoclinic fraction at every step of the test.

Materials and methods

Materials

Experiments were conducted on commercially available aluminatoughened zirconia balls (ceramys ® , Mathys Orthopaedie, Moersdorf, Germany).

Ageing procedure

To assess the ageing procedure proposed here, hydrothermal ageing was conducted in autoclave (Wolf Sanoclav, Germany), in water vapour at different temperatures between 70 and 141 °C, according to the schedules shown in Table 1. Schedules 1 and 2 are used to apply the stepwise procedure (thus each sample is submitted to the whole schedule), whereas the Usual schedule refers to one single temperature per sample until saturation of the monoclinic fraction.

The monoclinic fractions were determined using X-Ray Diffraction (D8-Advance, Bruker, Germany) at every time point of the same schedule. XRD diagrams were acquired between 26 and 33 deg. 2θ, exposing the (-111) and (111) monoclinic peaks, and (101) tetragonal peak. The volume monoclinic fraction (f m ) was deduced from these peaks intensities using Garvie and Nicholson's equation [START_REF] Garvie | Phases analysis in zirconia systems[END_REF] modified by Toraya [START_REF] Toraya | Calibration curves for the quantitative analysis of the monoclinic-tetragonal ZrO2 system by X-ray diffraction[END_REF]:

= + = + + + f X X X I I I I I 1.
All ageing schedules were performed on 3 samples. All measurements of the monoclinic fraction were performed on two locations (pole and equator) of each sample.

Theory/calculation

Extrapolation procedures

The usual extrapolation procedures are described in the introduction. Here they are referred to as UA (usual schedule, analytical determination of the ageing parameters using Eq. ( 2)) and US (usual schedule, determination of the ageing parameters by global error minimisation using Eq. ( 3)).

The stepwise procedure presented here does not rely anymore on complete kinetics measured at different, constant temperatures. Instead, a single piece of zirconia-based material is submitted successively to different ageing temperatures. The single ageing kinetics obtained this way allows the determination of all three ageing parameters.

The method is based on a set of three equations obtained from Eq. ( 1), and on preliminary approach used to describe ageing at two alternating temperatures [START_REF] Gremillard | Combining ageing and wear to assess the durability of zirconia-based ceramic heads for total hip arthroplasty[END_REF]. It is developed below. Let's assume a completely tetragonal material before hydrothermal ageing (V(0,T 0 ) = 0 whatever T 0 ). Then a first accelerated ageing in autoclave is conducted at temperature T 1 during time Δt 1 . The resulting monoclinic fraction is V 1 and could be calculated by Eq. ( 1). Then the temperature is changed to T 2 and the material is aged again for a duration Δt 2 . The resulting monoclinic fraction is V 2 . One cannot represent the points (Δt 1 ,V 1 ) and (Δt 1 + Δt 2 , V 2 ) on the same curve, since they were obtained at different temperatures. Instead, one can estimate the "virtual time" t 2 that would have been necessary to reach the monoclinic fraction V 1 while ageing at the temperature T 2 . This time is given by:

⎜ ⎟ = - - = ⎛ ⎝ -⎞ ⎠ t V b b b Q RT ln(1 ) , with exp 2 1 2 2 0 2 n ( 5
)
Then, V 2 could be calculated with Eq. ( 6):

= - -⋅ + V b t Δt 1 exp[ ( ( )) ] n 2 2 2 2 (6)
This leads to an iterative procedure, in which the state of the material after the i th iteration (ageing at T i during Δt i ) is described by the following set of equations:

= - -⋅ + V b t Δt 1 exp[ ( ( )) ] i i i i n (7a) = - -- t with i V b ln(1 ) i n i 1 (7b) = - ( ) b b and exp i Q RT 0 i (7c)
Note that for a material that retains a non-zero monoclinic fraction after sintering (V 0 ) and whose monoclinic fraction saturates at a fraction V M the set of Eqs. (7a)-(7c) can be changed to the set of Eqs. (8a)-(8c):

= + - ⋅ - -⋅ + V V V V b t Δt ( ){ 1 e x p [ ( ( ) )] } i M i i i n 0 0 (8a) = - -- ( ) t with ln i b V V V V 1 i M M i n 0 1 (8b) = - ( ) b b and exp i Q RT 0 i (8c)
The result of an ageing experiment following this procedure is thus a list of [(T i , Δt i , V i ), i = 1,2,…] (list in which all T i are not necessarily different). From this list one can extract n, Q and b 0 (and V 0 and V M if necessary) either analytically or using a numerical optimization procedure (simply put, a solver).

Indeed, if one uses twice the same temperature (T 2 = T 1 ) for the two first iterations (t 1 = 0), it follows that b 2 = b 1 , and the situation can be described by:

= - -⋅ = - -⋅ + V b Δ t V b Δt Δt 1 exp[ ( ( )) ] and 1 exp[ ( ( )) ] n n 1 1 1 2 1 1 2 (9)
Which leads to the knowledge of n by:

= - - + - - ( ) ( ) ( ) ( ) n Δt Δt Δt
ln ln ln ln ln( ) ln( )

V V 1 1 1 1 1 1 2 1 2 (10)
Moreover, incorporating a time increment Δt i of 0 in Eq. (7a) leads to:

= = - -⋅ - V V b t 1 exp[ ( ) ] i i i i n 1 (11)
Combined with Eq. (7a):

- - = -⋅ -⋅ + = ⋅ + -⋅ - V V b t b t Δt b t Δt b t 1 1 exp[ ( ) ] exp[ ( ( )) ] exp[( ( )) ( ) ] i i i i n i i i n i i i n i i n 1 (12a) ⎜ ⎟ ⎛ ⎝ - - ⎞ ⎠ = ⋅ + -⋅ - V V b t Δt b t or ln 1 1 ( ( )) ( ) i i i i i n i i n 1 (12b)
If Δt i is small enough as compared to t i , then V i is very close to V i-1 , then a first order limited development of the exponential, a classical equivalent of the logarithm and a combination with Eq. (7c) give:

⎜ ⎟ ⎜ ⎟ ⎛ ⎝ - - ⎞ ⎠ ∼ ∼ ⎛ ⎝ -⎞ ⎠ - Δt V V nb nb Q RT 1 ln 1 1 exp i i i i i 1 0 (13a) ⎜ ⎟ ⎡ ⎣ ⎢ ⎛ ⎝ - - ⎞ ⎠ ⎤ ⎦ ⎥ ∼ - - Δt V V nb Q RT ln 1 ln 1 1 ln( ) i i i i 1 0 (13b) Thus plotting ⎡ ⎣ ⎤ ⎦ - - - ( ) ln ln Δt V V 1 1 1 i i i 1 versus RT 1 i
between two close points (small Δt i as compared to t i ) gives a straight line of slope -Q and intercept nb ln( ) 0 , thus b 0 can be calculated thanks to the previous knowledge of n.

Results

Fig. 1 shows the determination of n, b0 and Q using the classical, analytical approach (three complete kinetics measured at 90, 111 and 134 °C, analysed using Eq. ( 2)). It confirms that n is almost independent on the temperature. Here a value of 0.94 was chosen for n (average of the three slopes reported on Fig. 1(a)). From these parameters (reported in Table 2 in the column ZTA-UA) one can calculate the ageing kinetics at the same temperatures and extrapolate at 37 °C. The result is shown on Fig. 2 minimize the error function described in Eq. ( 3) to obtain directly another set of parameters (reported in Table 2 in the column ZTA-US). The kinetics calculated using this new set are reported in Fig. 2(b). In both cases, a good fit of all experimental data is obtained.

(a). Using the same set of experimental points, one can

Fig. 3 shows the results obtained from the schedule 1 of the stepwise procedure (see SW1 on Table 1). The calculation of the ageing kinetics parameters was conducted using a solver, and minimizing the difference between the experimental values of the monoclinic fraction and the calculated ones. A very good agreement is reached. However, since only one point was used for each temperature, an analytical determination of the ageing parameters was not possible. The values of n, b 0 and Q found for the best fit are reported in Table 2 (column SW1).

Schedule 2 (see Table 1, SW2) was devised so that the measurements at lower temperatures occur during the fastest increase of monoclinic fraction with ageing time (above 30% monoclinic fraction). This decreases again the total ageing time by a factor of almost 1.5 as compared to schedule 1. Moreover, an analytical determination of n, b 0 and Q was made possible by incorporating intermediate time points at several temperatures (see Table 1). The results are shown on Fig. 4 and were exploited using two procedures.

In the first one, n was determined from the two first points at 141 °C according to Eq. [START_REF] Gremillard | Combining ageing and wear to assess the durability of zirconia-based ceramic heads for total hip arthroplasty[END_REF]. Knowing n, it was possible to plot

⎡ ⎣ ⎤ ⎦ - - - ( ) ln ln Δt V V 1 1 1 i i i 1 versus RT 1 i
at the different temperatures (Fig. 4(b)) to determine the activation energy (100 kJ/mol) and b 0 according to Eq. (13b). However it can be seen that the agreement between measured and calculated monoclinic fractions is not very good. These results are reported in the column SW2A of Table 2.

Using the same dataset the second procedure called upon a numerical optimization procedure (solver) to minimize the global error by adjusting b 0 , Q and n (V 0 and V Max being known from UA or US procedures). The resulting kinetics are displayed on Fig. 5, and show a much better agreement between calculations and experiment. Once again the ageing kinetics parameters are reported in Table 2 (column SW2S).

From the parameters reported on Table 2, it is possible to extrapolate all the measured kinetics at 134 °C and at body temperature (37 °C) (Fig. 6). At 134 °C, all (but one) extrapolated kinetics agree reasonably well with the measured kinetics. Only SW2A set of parameters results in an incorrect extrapolation that underestimate the ageing kinetics. At 37 °C no experimental determination of the ageing kinetics of ATZ is possible in a reasonable timeframe. Taking as a reference the extrapolation realised using the classical, analytical procedure (UA), it can be seen that all other extrapolations lie below although they are in the same order of magnitude. For example, at 200 000 h (23 years) the calculated monoclinic fractions lie between 0.23 (SW2A) and 0.39 (US). Conversely the times necessary to reach 40% monoclinic fraction lie between 200 000 h (23 years, UA) and 500 000 h (57 years, SW2A). All other extrapolations lie between UA and SW2A.

Discussion

The parameters shown in Table 2 reveal some interesting findings:

-First, the experimental values of V 0 and V M (resp. 0. In fact, since n is the same for all extrapolation procedures (except SW2A), the parameter b (at 37 °C) directly controls the speed of ageing at 37 °C. This explains why the values of b (at 37 °C) are ordered in the same way as the ageing kinetics extrapolated at the same temperature: SW2A < SW2S < SW1 < US < UA (where the sign " < " indicates a smaller value of b and a slower kinetic). On Fig. 6, the grey band indicates an interval of confidence calculated from the data of the classical procedure. Except for SW2A and SW2S, all extrapolation procedures lead to equivalent ageing kinetics at 37 °C (all kinetics fall inside the interval of confidence).

This leads us to wonder why the two SW2 procedures are outside this band. SW2A proves to be the least precise. It is probably related to the way n is calculated (Eq. ( 10)), taking into account only the two first experimental points. Moreover, ensuring that Δt i is small enough as compared to t i means that V i and V i-1 are similar. Thus small, unavoidable errors on the measurement of V i will cause large relative errors on V i -V i-1 , thus a large imprecision on Q and n•b 0 obtained by Eq. (13b) if one tries this analytical approach. In conclusion the SW2A procedure should not be used alone; however it can prove useful to provide initial parameters for the global error minimization of SW2S.

The difference between SW2S and the other procedures (UA, US, SW1) may be attributed to the less important weight given in this procedure to the lowest temperatures: in SW2S (and SW2A) 15% of the total increase of the monoclinic fraction during ageing can be attributed to temperatures equal to or lower than 100 °C. In SW1, this figure rises to 27%, and to 32% in UA and US (taking into account all kinetics at all temperatures). On the other hand, schedule 1 does not lead to saturation of the monoclinic fraction.

SW1 and SW2 procedures present several advantages over the classical ones: It is rather difficult to decide what extrapolation procedure gives the most precise results. One might decide on choosing the worst case. But it is doubtful that the worst case will always be given by the UA procedure.

In our opinion, the UA procedure may prove to be the most accurate, provided it's conducted at both low (down to 80 or 70 °C) and high (up to 150 °C) temperatures, which is not the case here. On the other hand, this procedure will be so long that it cannot be used to assess numerous new materials. In the framework of zirconia-based materials development, schedule SW1 may be a good option to compare relatively quickly between several materials. Of course, the schedules presented here are not the only ones possible, and further developments could be implemented in the case of completely unknown materials (for example, start with a few hours at 134 °C, and change the ageing temperature after each 10% increase of the monoclinic fraction). In all cases, it is always advisable to start with the measurement of a whole kinetics at high temperature, thus gaining access to more precise initial values of n, V 0 and V Max , while b 0 and Q will be known after a SW schedule.

Finally, it is worthy to point out that although the measurements of monoclinic fraction after each ageing step are mandatory, X-ray diffraction is not the only method available. The stepwise procedure presented here should give similar results when using Raman spectroscopy [START_REF] Clarke | Measurement of the crystallographically transformed zone produced by fracture in ceramics containing tetragonal zirconia[END_REF][START_REF] Pezzotti | Raman spectroscopic analysis of advanced ceramic composite for hip prosthesis[END_REF] or any other 'evolution indicator' to measure the monoclinic fractions.

Conclusion

This article presents a stepwise procedure to determine the parameters of hydrothermal ageing of zirconia-containing ceramics. This procedure relies on ageing at different temperatures applied on a same sample. It is shown to be faster than usual ones (relying on a single temperature per sample) and less costly (necessitating less time, only one sample and fewer measurement of the monoclinic fraction). The extrapolation to 37 °C realised using both usual and stepwise procedures lie in the same order of magnitude. However they are not exactly alike. It is suggested that the stepwise procedure should be used for fast screening of new materials, while a more precise extrapolation may be provided by the usual procedure extended to lower temperatures.

Fig. 1 .

 1 Fig. 1. Calculation of n, b0 and Q according to the analytical procedure.

Fig. 2 .

 2 Fig. 2. Aging kinetics of ATZ using the classical procedure. (a): fit parameters derived from an analytical approach (UA); (b): fit parameters derived from a numerical global error minimization (US).

Fig. 3 .

 3 Fig. 3. Ageing kinetics of ATZ calculated after the SW1 procedure, using schedule 1 and a numerical optimization; experimental datapoints are represented by blue triangles; calculated datapoints are shown as open circles; calculated kinetics are shown as continuous lines. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

  013 and 0.75, measured during a whole ageing kinetics at 134 °C) are adequate for all fitting procedure. -Second, the values of n are also very close: from 0.94 to 0.98, with one exception (SW2A for which n = 0.84, due to the fact that for SW2A n is determined from only 2 early time points, whereas for the other procedures all time points are taken into account). -Third, there is more scattering in the values of the activation energy and of b 0 . However, when using b 0 and Q to calculate b at 37 °C, all values of b are in the same range (1.4•10 -6 (SW2A) to 3.5•10 -6 (UA)). Since b is closely related to the transformation speed (high b means fast transformation), all kinetics at 37 °C also lie in the same range.

Fig. 4 .

 4 Fig. 4. Ageing kinetics calculated after the stepwise procedure (schedule 2, analytical).

Fig. 5 .

 5 Fig. 5. Ageing kinetics calculated after the stepwise procedure, schedule 2, with a numerical optimization.

Fig. 6 .

 6 Fig. 6. Calculations of the monoclinic fraction vs time at 134 °C and at 37 °C using the different ageing procedures. Experimental points are measurements taken at 134 °C during the Usual Schedule.

Table 1

 1 Ageing kinetics schedules for the stepwise (SW) extrapolation procedure; Schedule SW1 and schedule SW2: all samples undergo the whole schedule; Usual schedule: three samples for each temperature.

	Schedule 1: SW1	Schedule 2: SW2A, SW2S	Usual Schedule: UA, US
	T i (°C)	Δt i (h)	T i (°C)	Δt i (h)	T (°C)	Total time (h)
		0		0		0
	141	4	141	4 + 2	134	150
	134	3 + 4	134	3 + 4	111	650
	121	15	111	15 + 20	90	3200
	111	40	75	200 + 300
	100	120	90	100
	85	400	85	200
	75	1000	100	50 + 100
			121	40 + 40
			134	10 + 10
	TOTAL	1586		1098		4000

Table 2

 2 Parameters for ageing of ATZ according to the different ageing procedures.

		Analytical (UA)	Solver (US)	Stepwise Schedule 1 solver (SW1)	Stepwise Schedule 2 Analytical (SW2A)	Stepwise Schedule 2 solver (SW2S)
	V 0	0.013	0.013	0.013	0.013	0.013
	V M	0.75	0.75	0.75	0.75	0.75
	b 0 (h -1 )	3.6•10 10	3.6•10 10	4.5•10 11	1.1•10 11	1.1•10 12
	Q (kJ/mol)	95.0	95.3	102.8	100.2	105.7
	b at 37 °C	3.5•10 -6	3.1•10 -6	2.1•10 -6	1.4•10 -6	1.7•10 -6
	n	0.94	0.98	0.95	0.84	0.96
	Overall error	5.3•10 -3	1.2•10 -2	1.2•10 -4	2.2•10 -1	2.8•10 -3
	Error/point	1.9•10 -4	4.4•10 -4	1.3•10 -5	1.4•10 -2	1.8•10 -4
	time in autoclave (h)	4000	4000	1586	1098	1098
	Nb of XRD measurements	28	28	9	17	17

Acknowledgment

C. Wei was supported by a grant from the Chinese Science Council.

Appendix A. Supplementary data

Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.jeurceramsoc.2017.08. 018.


Katarina Bartkova

Denis Jouvet

Analysis of prosodic correlates of emotional speech data

Keywords: expressive speech, emotions, prosodic groups, prosodic correlates

The study of expressive speech styles remains an important topic as to their parameters detection or prediction in speech processing. In this paper, we analyze prosodic correlates for six emotion styles (anger, disgust, joy, fear, surprise and sadness), using data uttered by two speakers. The analysis is focused on the way pronunciations and prosodic parameters are modified in emotional speech, compared to neutral style. The analysis concerns speech pronunciation modifications, presence of pauses in sentences, and local prosodic behavior, with an emphasis set on the analysis of the prosody over prosodic groups and breathing groups.

Introduction

Prosody not only conveys information on the linguistic content of the vocal messages, but also on speaker's attitude and emotional state. In speech sciences, expressive speech is now attracting a lot of interest, as for example for speech synthesis [START_REF] Schröder | Expressive speech synthesis: Past, present, and possible futures[END_REF]) and for automatic recognition of emotions [START_REF] Lanjewar | Speech Emotion Recognition: A Review[END_REF]. For such studies, the collection of emotional speech data plays an important role. Several approaches have been envisaged for collecting emotional data; this includes the recording of natural emotions, the recording through induced situations, and the recording of acted speech by professional actors [START_REF] Scherer | Vocal communication of emotion: A review of research paradigms[END_REF]. This last mode is currently the most frequently used. Expressive speech synthesis has been developed for corpusbased [START_REF] Iida | A corpus-based speech synthesis system with emotion[END_REF] and parametric-based approaches [START_REF] Yamagishi | HMM-based expressive speech synthesis-Towards TTS with arbitrary speaking styles and emotions[END_REF]). As the recording of emotional data is difficult, several techniques have been investigated for adapting speech synthesis systems using small amount of emotional data [START_REF] Inanoglu | Data-driven emotion conversion in spoken English[END_REF] or for converting neutral speech into emotional speech [START_REF] Tao | Prosody conversion from neutral speech to emotional speech[END_REF]). It should be noted that parametric speech synthesis, needs to know the sequence of units, that is the sequence of sounds and pauses, for being able to produce the synthesized speech signal.

Following a previous study [START_REF] Bartkova | Prosodic Parameters and Prosodic Structures of French Emotional Data[END_REF]) that has analyzed prosodic features on a global level, this paper focus on analyzing more local distributions associated to prosodic groups and breathing groups.

Corpus and features

The speech corpus used contains French sentences in six emotional styles (anger, disgust, joy, fear, sadness, and surprise), uttered by two professional speakers, one male and one female. Every emotional style contains about 50 sentences of various length. Each speaker has also uttered the same sentences in a neutral, reading style. This makes possible to study how pronunciations and prosodic parameters vary from neutral to emotional style.

For each sentence, the speech material has been aligned with its corresponding text using an automatic speech-text forced alignment procedure, and then manually checked and corrected if necessary. An automatic process has also been applied to localize prosodic boundaries, based on vowel duration, F0 slope, F0 delta, and pause occurrences,

Statistics on prosodic groups

Emotional speech is generally faster than neutral speech. Compared to neutral speech, the speaking is significantly higher (up to 28% faster) for fear and anger, slightly higher for disgust, joy and surprise, but significantly lower (about 20% slower) for sadness.

As the same sentences have been pronounced in neutral and emotional styles, it was possible to compare the presence and position of pauses. About 95% of the pauses observed in emotional data appears at the same place in the neutral pronunciation. However, except for sadness, 13% to 30% of the pauses observed in neutral pronunciation disappear in emotional data, which explains the increase in speaking rate for these emotional styles. Compared to neutral speech, the most noticeable differences in F0 range distributions are observed for anger and sadness (displayed in Figure 1). Larger F0 ranges are much more frequent for anger, and slightly more frequent for fear, surprise and joy (not represented in Figure 1 for lack of space); and smaller F0 ranges are more frequently observed for sadness.

Analysis of delta F0 at end of breathing groups

The histograms in Figure 2 display the distributions of the delta F0 measures at the end of the breathing groups (for lack of space, only two emotions are displayed). To better see the occurrences of falling and rising delta F0 values at the end of the breathing groups, the horizontal axis is symmetric (from -15 up to +15 semi-tones). Anger and fear styles comprise larger amounts of negative delta F0 values than the neutral style. The other noticeable difference is observed for sadness, displaying a sharp histogram of delta F0 values centered around zero; exhibiting rather flat F0 patterns. A tendency of using relatively flat or slightly falling F0 pattern is also observed in disgust.

Segmental level analysis

Sequences of phone segments associated to each word have been compared between emotion and neutral data to obtain statistics on the modification of the pronunciation of the words from neutral to emotional speech. Unlike other studies [START_REF] Tahon | Optimal Feature Set and Minimal Training Size for Pronunciation Adaptation in TTS[END_REF], few phoneme changes are observed in the data.

The most common phoneme changes concern phonetic feature assimilations (mostly nasalization or voice feature assimilation) that are slightly more frequent in emotional speech than in neutral style. Some cases of liquid omissions are also observed, mainly in consonantal clusters. However, the main difference is mostly the omission of the schwa like vowel. In the emotional style a high number of schwa are omitted, and this vowel omission is place sensitive. In fact, the first and the last breathing groups of each record (a record corresponds to one or a few sentences) contain the highest number of schwa omissions (compared to neutral speech), 24% of the omissions are observed in the first breathing group and 27% in the last breathing group. Also, the number of schwa omissions is slightly emotion dependent, the highest percentage of schwa omissions is observed for disgust, fear and joy.

Conclusion

This paper has presented an analysis of prosodic correlates of emotional speech using a corpus of acted emotional speech. In comparison to neutral data, for the anger emotion, the speaking rate is higher, the number of pauses is significantly lower, larger F0 ranges over the prosodic groups are more frequent, as well as large negative delta F0 at end of breathing groups. Joy, disgust and fear emotions exhibit also higher speaking rate than neutral style. However, sadness exhibits a quite different behavior: compared to neutral speech, the speaking rate is lower, with a high number of inserted pauses.

Unlike the other five emotional styles, in sadness style, there is a shift of the F0 range histogram towards lower values. As for the delta F0 at end of breathing groups, small values (rather flat F0 pattern) are more frequent.

Figure 1 :

 1 Figure 1: Histograms of F0 ranges over prosodic groups (average over the two speakers) in emotional and neutral speech styles

Figure 2 :

 2 Figure 2: Histograms of delta F0 values at end of breathing groups (average over the two speakers) in emotional and neutral speech styles

Acknowledgments

This work has been partially supported by the CPER LCHN (Contrat Plan Etat Région "Langues, Connaissances et Humanités Numériques").