Ioan Doré Landau

Tudor-Bogdan Airimitoaie

Raul Melendez

Gabriel Buche

Luc Dugard

Experimental evaluation of a new class of algorithms for active noise attenuation

Keywords: active noise control, adaptive feedforward compensation, Youla-Kučera parametrization, positive feedback coupling

Historically adaptive feedforward noise compensation using the so called LMS adaptation algorithm has been considered as the basic solution for active noise attenuation despite a number of drawbacks. In fact one can reject unknown and time-varying multiple narrow-band noises without requiring information upon the noise using an adaptive feedback active attenuation scheme. The implementation of this type of scheme uses the Internal Model Principle and the Youla-Kučera parametrization as well as improved adaptation algorithms. For attenuation of broad-band noise, while one still uses a feedforward compensation scheme, new adaptation algorithms have been developed which take into account the interaction between the measurement of the incoming noise and the operation of the compensation device (leading to an internal positive feedback loop which may cause instability). The use of the Youla-Kučera parametrization allows to separate the problem of the stability of the internal positive feedback loop from the problem of optimization of the parameters of the feedforward compensator. This allows also to go beyond the limitations of the length of the compensation path. Experimental results are presented in the last part of the paper.

Introduction

Adaptive feedforward noise attenuation is widely used when a well correlated signal with the disturbance (image of the disturbance) is available ([3, 6, 17]). The first references go back roughly to 1985 ( [START_REF] Widrow | Adaptive Signal Processing[END_REF]). In most of the systems, there is a positive acoustic coupling between the feedforward compensation system and the measurement of the image of the disturbance. This often leads to the instability of the system. In the context of this inherent "positive" feedback, the adaptive feedforward compensator should minimize the effect of the disturbance while simultaneously assuring the stability of the internal positive feedback loop. This problem has been clearly identified by the mid nineties [START_REF] Kuo | Active noise control systems-Algorithms and DSP implementation[END_REF][START_REF] Jacobson | Stability of active noise control algorithms[END_REF]. Figure 1 gives the basic block diagram of the adaptive feedforward compensation in the presence of the internal pos- itive coupling between the output of the compensator and the measurement of the image of the incoming noise. The incoming noise propagates through the so called primary path T and its effect is compensated by a secondary noise source through the secondary path G driven by a feedforward compensator N . The input to the feedforward compensator is the sum of the image of the incoming noise and of the internal acoustical positive feedback through M . The residual noise is used to emulate the adaptation of the feedforward compensator.

A major component of such a system is the PAA (parameter adaptation algorithm). In the field of ANC (active noise control), the first algorithm used was the so called LMS (least mean squares, see [START_REF] Widrow | Adaptive Signal Processing[END_REF]) derived from a local minimization of a quadratic criterion in terms of the residual noise. Many contributions have been done on the analysis of the properties of this algorithm1 and the improvement of the algorithm. Filtering of the regressor vector was one of the ways for improving the adaptation algorithm. The FULMS algorithm [START_REF] Eriksson | Development of the filtered-U LMS algorithm for active noise control[END_REF] 2 which seems to be the most used algorithm in adaptive feedforward compensation can be viewed as a particular approximation of the algorithms derived from stability considerations in [START_REF] Dore Landau | Adaptive feedforward compensation algorithms for active vibration control with mechanical coupling[END_REF].

A major step forward in the design of the adaptive feedforward compensators in the presence of the internal positive feedback coupling was the introduction of the Youla-Kučera parametrization allowing to separate the stabilization of the internal positive feedback loop from the optimization of the parameters of the feedforward compensator. This ideea has been prompted out for the first time in [START_REF] Zeng | Recursive filter estimation for feedforward noise cancellation with acoustic coupling[END_REF]. A full synthesis procedure has been presented in [START_REF] Doré Landau | A Youla-Kučera parametrized adaptive feedforward compensator for active vibration control with mechanical coupling[END_REF][START_REF] Landau | IIR Youla-Kučera parameterized adaptive feedforward compensators for active vibration control with mechanical coupling[END_REF]. Examples of applications in active noise control can be found in [START_REF] Landau | Why one should use youla-kucera parametrization in adaptive feedforward noise attenuation?[END_REF][START_REF] Landau | Robust and adaptive feedback noise attenuation in ducts[END_REF]. At the end of the nineties, adaptive feedback noise control emerged as an efficient solution for cancelling single or multiple tonal disturbances ( [START_REF] Ben Amara | Adaptive sinusoidal disturbance rejection in linear discrete-time systems -Part I: Theory[END_REF]), ( [START_REF] Ben Amara | Adaptive sinusoidal disturbance rejection in linear discretetime systems -Part II: Experiments[END_REF]) taking advantage of the internal model principle and the Youla-Kučera parametrization of the feedback controller. Fig. 2 gives the block diagram of such a system.

+ + - Residual force (disturbance)
Primary path The feedback approach is mainly dedicated to the attenuation of unknown time-varying multiple narrow-band noises while the feedforward approach is dedicated for the attenuation of broad-band noise with unknown and time-varying characteristics.

Secondary path Controller

Experimental Setup

The views of the various configurations of the test-bench used for experiments are shown in Fig. 3 and its detailed scheme is given in Fig. 4. The speaker used as the source of disturbances is labelled as 1, while the control speaker is marked as 2. At pipe's open end, the microphone that measures the system's output (residual noise e(t)) is denoted as 3. Inside the pipe, close to the source of disturbances, the second microphone, labelled as 4, measures the perturbations' image, denoted as y(t). Additionally, we denote u(t) the control signal, and s(t) the disturbance. The transfer function between the disturbance's speaker and the microphone (1→3) is called Global Primary Path, while the transfer function between the control speaker and the microphone (2→3) is denoted Secondary Path. The transfer function between microphones (4→3) is called Primary Path. The internal coupling found between (2→4) is denoted Reverse Path. These marked paths have a double differentiator behaviour, since as input we have the voice coil displacement and as output the air acoustic pressure. For the feedback attenuation configuration the microphone 4 is unnecessary. The speakers are isolated inside wood boxes filled with special foam in order to create anechoic chambers and reduce the radiation noise produced. Both speakers are connected to a xPC Target computer with Simulink Real-time R environment through a pair of high definition power amplifiers and a data acquisition board. A second computer is used for development, design and operation with Matlab R . The sampling frequency has been chosen in accordance with the recommendations given in [START_REF] Doré Landau | Adaptive and Robust Active Vibration Control-Methodology and Tests[END_REF]. Taking into account that disturbances up to 400 Hz may need to be attenuated, a sampling frequency f s = 2500 Hz has been chosen (T s = 0.0004 sec), i.e., approximately six times the maximum frequency to attenuate. The frequency characteristics of the identified models for the primary 3 , secondary and reverse 4 paths are shown in Fig. 5 for the version V1 of the test-bench. These characteristics present multiple resonances (low damped complex poles) and anti-resonances (low damped complex zeros).

For the other configurations while the location of the resonances and anti-resonances will be different the general feature will be similar.

It is important to note that most of the implementations of the adaptive feedforward compensation systems are close to a co-location of the residual noise measurement and of the secondary source used for compensation. A ratio of 1/6 to 2/6 between the length of the secondary path and the length of the primary path is generally used. Nevertheless, there are new potential applications areas (exhaust noise reduction on boats, trucks, cars) where thermal constraints will not allow to have a configuration close to a colocation. For this reason the ratio between the length of the secondary path and the length of the primary path has been chosen about 5/6 (close to the theoretical limit) as well beyond (7/6) for the version V3 (see Fig. 3).

Adaptive feedforward noise attenuation

Figure 1 gives the basic structures of the adaptive feedforward attenuation schemes where the feedforward compensator can have a FIR or an IIR structure. The compensator N has the form N = R S . The estimated polynomials R and S will be denoted R(q -1 ) and Ŝ(q -1 ) (for S = 1 one has a FIR compensator). The corresponding block diagram for the adaptive feedforward compensation using IIR Youla-Kučera parametrization 3 The primary path model has been exclusively used for simulation purposes. 4 In the feedback approach this path is not used since there is no a second microphone of the feedforward compensator is shown in Fig. 6. ŷ(t) denotes the effective output provided by the measurement device and which will serve as input to the adaptive feedforward filter N . in this case the adaptive feeforward compensator is formed by the central controller R 0 /S 0 and the adjustable estimated filter BQ / ÂQ . The output of this filter denoted by û(t) is applied to the actuator through an amplifier.

For A Q = 1 one gets an FIR Youla-Kučera parametrization.

In this case the poles of the internal positive feedback loop will be defined by the choice of S 0 and R 0 and will remain unchanged independently of the values of the parameters of the estimated polynomial BQ . Details can be found in [START_REF] Doré Landau | A Youla-Kučera parametrized adaptive feedforward compensator for active vibration control with mechanical coupling[END_REF][START_REF] Landau | IIR Youla-Kučera parameterized adaptive feedforward compensators for active vibration control with mechanical coupling[END_REF].

Adaptive feedback noise attenuation

The adaptive feedback noise attenuation configuration using the Youla-Kučera parametrization is shown in Fig. 7 where S 0 (q -1 ) and R 0 (q -1 ) define the central controller. The characteristic polynomial, which specifies the desired closed loop poles of the system is given by (see also [START_REF] Landau | Digital control systems -Design, identification and implementation[END_REF]): 5

P 0 (z -1 ) = A G (z -1 )S 0 (z -1 ) + z -d B G (z -1 )R 0 (z -1 ). (1)
When the Youla-Kučera parametrization is used with a FIR Q polynomial the poles of the closed loop defined by (1) remain unchanged independently of the values of the parameters of the Q filter. In this case the controller's polynomials become:

R = R 0 + A G Q, (2)
S = S 0 -z -d B G Q, (3)
A key aspect of this methodology is the use of the Internal Model Principle (IMP). It is supposed that p(t) is a deterministic disturbance given by

p(t) = N p (q -1 ) D p (q -1 ) • δ(t), (4)
where δ(t) is a Dirac impulse and N p , D p are coprime polynomials of degrees n Np and n Dp , respectively 6 . In the case of stationary narrow-band disturbances, the roots of D p (z -1 ) are on the unit circle.

Internal Model Principle [8]:

The effect of the disturbance given in (4) upon the output is given by e(t) = A G (q -1 )S(q -1 ) P (q -1 )

• N p (q -1 ) D p (q -1 ) • δ(t), (5)
where D p (z -1 ) is a polynomial with roots on the unit circle and P (z -1 ) is an asymptotically stable polynomial. y(t) in Eq. ( 5) converges asymptotically towards zero if and only if the polynomial S(z -1 ) in the (equivalent) RS controller has the form: S(z -1 ) = D p (z -1 )S (z -1 ).

One can see that the polynomial S of the controller incorporates in this case the denominator of the model of the disturbance. It turns out that using the Youla-Kučera parametrization the introduction of the internal model of the disturbance in the controller in the presence of unknown disturbances can be achieved by the direct adaptation of the parameters of the Q polynomial. Details can be found in [START_REF] Dore Landau | Robust and adaptive feedback noise attenuation in ducts[END_REF].

Parameter Adaptation Algorithms

The PAA used in all the schemes has the general form: 5 It is assumed that a reliable model identification is achieved and therefore the estimated model is assumed to be equal to the true model. 6 Throughout the paper, n X denotes the degree of the polynomial X.

θ(t + 1) = θ(t) + F (t)ψ(t)ν(t + 1) ; (7)
ν(t + 1) = ν • (t + 1) 1 + ψ T (t)F (t)ψ(t) ; (8)
F (t + 1) = 1 λ 1 (t)   F (t) - F (t)ψ(t)ψ T (t)F (t) λ1(t) λ2(t) + ψ T (t)F (t)ψ(t)   (9) 1 ≥ λ 1 (t) > 0; 0 ≤ λ 2 (t) < 2; F (0) > 0 (10) ψ(t) = φ f (t) = L(q -1 )φ(t), (11)
where λ 1 (t) and λ 2 (t) allow to obtain various time profiles for the matrix adaptation gain F (t) (see [START_REF] Landau | Adaptive control[END_REF]). By taking λ 2 (t) ≡ 0 and λ 1 (t) ≡ 1, one gets a constant adaptation gain matrix.

θ(t + 1) = θ(t) + F (t)ψ(t) ν • (t + 1) 1 + ψ T (t)F (t)ψ(t) (12) ν • (t) = -e • (t) (13)
Two types of adaptation gain allowing to operate in an "adaptive" regime are mostly used:

• Constant trace algorithm. λ 1 (t) and λ 2 (t) are adjusted continuously to maintain constant the trace of the adaptation gain matrix. This allows to move in the optimal direction while maintaining the adaptation capabilities.

• Constant scalar adaptation gain. This is obtained by taking λ 1 (t) ≡ 1, λ 2 (t) ≡ 0 and F (t) = γI, γ > 0 where I is the identity matrix.

The values of λ 1 (t) and λ 2 (t) in order to maintain constant the trace of the adaptation gain matrix are determined from the equation:

tr (F (t + 1)) = 1 λ 1 (t) tr F (t) - F (t)ψ(t)ψ T (t)F (t) δ(t) + ψ T (t)F (t)ψ(t) fixing the ratio δ(t) = λ 1 (t)/λ 2 (t) = const. Typical value: δ = 1.
The updating of matrix F(t) is done using the U-D factorization for numerical robustness reasons. The details of this algorithm 7 are given in [START_REF] Doré Landau | Adaptive and Robust Active Vibration Control-Methodology and Tests[END_REF]Appendix B]. By taking F (t) = γI, where I is the identity matrix, one gets a scalar adaptation gain. The equation [START_REF] Landau | IIR Youla-Kučera parameterized adaptive feedforward compensators for active vibration control with mechanical coupling[END_REF] for updating the parameter vector becomes:

θ(t + 1) = θ(t) + γψ(t) ν • (t + 1) 1 + γψ T (t)ψ(t) . (14)
When using scalar adaptation gain, for very small values of γ one can approximate the above equation by

θ(t + 1) = θ(t) + γψ(t)ν • (t + 1) (15)
In the FULMS and FXLMS algorithms, since the adaptation gain is small and therefore the residual error will vary slowly, the quantity ψ(t)ν

• (t + 1) is approximated by ψ(t -1)ν • (t)
leading to:

θ(t + 1) = θ(t) + γψ(t -1)ν • (t) (16)

Stability conditions

The a posteriori adaptation error ν is governed by an equation of the form:

ν(t + 1) = H ν (q -1 )[θ -θ]φ(t) (17)
7 Routines for the implementation of the algorithm can be downloaded from http://www.gipsa-lab.grenoble-inp.fr/~ioandore.landau/ adaptivecontrol/ where φ(t) is the observation (measurement) vector. This vector is always filtered by a filter L and (17) will take the form:

ν(t + 1) = H(q -1 )[θ -θ]ψ(t) (18)
where H(q -1 ) = H ν (q -1 )/L(q -1)

Taking into account (18) for the a posteriori adaptation error and the equations for the PAA [START_REF] Landau | IIR Youla-Kučera parameterized adaptive feedforward compensators for active vibration control with mechanical coupling[END_REF] through [START_REF] Landau | Digital control systems -Design, identification and implementation[END_REF] one has the following condition for global asymptotic stability [START_REF] Landau | Adaptive control[END_REF]:

H (z -1 ) = H(z -1 ) - λ 2 2 , max t (λ 2 (t)) ≤ λ 2 < 2 (20)
should be a strictly positive real (SPR) transfer function. For constant adaptation gains (λ 2 (t) = 0) the stability condition becomes: H(z -1 ) should be SPR. A key role in the stability of the various adaptation algorithms is played by the filter L operating on the observation vector φ. It helps to satisfy the "strictly positive real condition" for asymptotic stability.

Experimental Results

The objective of this section is to illustrate the performance which can be obtained with the techniques presented.

Adaptive feedback noise attenuation 6.1.1 Interference test

By interference, one refers to the effect occurring when two distinct waves with very close frequencies act together, creating periodic outbursts in the resulting wave magnitude. In this context two interferences are considered simultaneously and the frequencies are supposed unknown and timevarying.

The protocol is as follows: for 1 s, the system operates in open loop and without any disturbance in order to get a reference for the ambient noise.

Step changes in frequencies

In this experiment, step changes in the frequencies of a pair of tonal noise disturbances are considered, starting from their nominal values of 170 Hz and 285 Hz. The steps are of ±10 Hz and applied every 6.2 s. The system is operated in open loop from 0 s to 1 s.

The performance of the adaptive controller is illustrated in Fig. 10. The residual noise is close to the ambient noise. The adaptation transients are visible but very short. The evolution of the Q-parameters is shown in Fig. 11. 60 parameters (numerator : 30, denominator: 30) using the FUSBA algorithm [START_REF] Doré Landau | Adaptive and Robust Active Vibration Control-Methodology and Tests[END_REF] (matrix adaptation gain). Adaptation starts at t=15s. The disturbance has an almost flat spectrum between 70 and 270 Hz. Attenuation (also displayed on this figure) reaches almost the steady state value at 600s. The attenaution achieved is 35.72 dB. Figure 13 shows the PSD (power spectral density) for this configuration both in open loop and with the adptive feeforward compensator. Figure 14 shows the adaptation capabilities of the IIRYK adaptive compensator with 30/30 parameters. One switch at t=180s from a flat disturbance between 170 to 270 Hz to a aflat disturbance between 70 to 170 Hz. For the case when the length of the secondary path is larger than the length of the primary path (structure V3) experimental results can be found in [START_REF] Landau | Robust and adaptive feedback noise attenuation in ducts[END_REF].

Adaptive feedforward noise attenuation

Concluding remarks

A new methodology for adaptive active noise attenuation has emerged and extensive experimental results have been carried on to evaluate the performance. This new methodology can improve the performance of current solution for active noise attenuation an opens the way for new applications.

Figure 1 :

 1 Figure 1: Feedforward AVC with adaptive feedforward compensator.

Figure 2 :

 2 Figure 2: Adaptive feedback noise attenuation.

Figure 3 :

 3 Figure 3: Duct active noise control test bench-Configuration (Photo): configuration V1 (top), configuration V2 (middle), configuration V3 (bottom)

Figure 4 :

 4 Figure 4: Duct active noise control test-bench diagram.

Figure 5 :

 5 Figure 5: Frequency characteristics of the Primary, Secondary and Reverse paths identified models.

Figure 6 :

 6 Figure 6: Feedforward ANVC with IIRYK adaptive feedforward compensator.

Figure 7 :

 7 Figure 7: Direct adaptive feedback scheme for rejection of unknown disturbances.

From 1 s

 1 to 10 s, the test bench works in open loop, in the presence of two pairs of sinusoidal noise disturbances located at 170 Hz and 170.5 Hz and 285 Hz and 285.5 Hz respectively. At 10 s, the loop is closed and the controller begins to counteract the disturbance effect. The frequencies of the four signals are then increased at 21 s by 10 Hz. The corresponding new values are 180 Hz and 180.5 Hz for the first pair and 295 Hz and 295.5 Hz for the second pair .

Figure 8

 8 Figure 8 presents the results obtained with the adaptive feedback Youal-Kučera configuration. The number of adjustable parameters in the Q-filter is 4 (n Q = 3) and an adaptation algorithm with constant trace adaptation gain is used. It can be seen that after a negligible transient, an excellent attenuation is obtained. The global attenuation obtained is 70.56 dB. Excellent levels of attenuation are also obtained once the disturbances frequencies move away by 10 Hz (global attenuation 67.65 dB), with a negligible adaptation transient Figure 9 displays the evolution of the Q-parameters. From 0 s to 10 s, all the parameters have values equal to zero

Figure 8 :

 8 Figure 8: Acoustic interference attenuation using an adaptive controller. Noise frequencies:170+170.5 Hz, 285+285.5 Hz then 180+180.5 Hz, 295+295.5 Hz. Loop closed at 10 s.

Figure 9 :

 9 Figure 9: Parameters evolution for acoustic interference test using an adaptive controller.

Figure 12 Figure 10 :

 1210 Figure 12 illustrates the evolution of the residual noise and of the attenuation over an horizon of 600 s for the Youla-Kučera feedforward compensator with an IIR Q filter with

Figure 11 :

 11 Figure 11: Evolution of the parameters of the adaptive controller in the presence of step changes in disturbances frequencies.

Figure 12 :

 12 Figure 12: Residual noise using the IIRYK 30/30 adaptive compensators with FUSBA matrix adaptation (70-270 Hz disturbance, 600 s experiments).

Figure 13 :

 13 Figure 13: PSD of the IIRYK 30/30 adaptive compensators using FUSBA matrix adaptation (70-270 Hz disturbance, 600 s experiments).

Figure 14 :

 14 Figure 14: Residual noise of the IIRYK 30/30 adaptive feedforward compensator for a change of disturbance from 170 -270 Hz to 70 -170 Hz at 180 s.

However these attempts have not solved the stability problem in the presence of the internal "positive" feedback.

It is used with an IIR structure of the feedforward compensator. When it is used with a FIR feedforward compensator it is denoted FXLMS.


Mehmet Gazaloğlu

Carole Camarasa

Sylvie Dequin

Elke Nevoigt

Enzymatic diversity among pectinolytic activities in yeast

Enzymatic diversity among pectinolytic activities in yeast

Mehmet Gazaloğlu 1 , Carole Camasara 2 , Sylvie Dequin 2 , and Elke Nevoigt 1

Background

• Yeast strains from different species were studied for phenotypic diversity in polygalacturonase enzyme secretion under optimum and winemaking conditions. • The 14 strains that are PG-positive under winemaking conditions will be characterized for the quantification of their PG activity, their aroma contributions and their impact on winemaking processing such as filtration time, clarity and color for future studies. • To understand the genetic basis of the enzymatic and metabolic information acquired so far, the genomic DNA of superior strains will be subjected to whole-genome sequencing (WGS), and bioinformatics analysis will be conducted to search also for other potential CAZYmes. It has been reported that certain yeast species display pectinolytic activities, which mainly result from the activity of polygalacturonases.

Conclusions and Future Perspectives

These yeasts have reported positive effects on taste complexity and organoleptic richness in winemaking and cocoa production.

Nevertheless, a detailed understanding of how pectinolytic yeasts contribute to wine and cocoa quality needs further scientific work.

In this work, strains from different wine-associated yeast species will be screened for their native polygalacturonase activities.

Method

In total, 306 yeast strains from 17 different yeast species have been screened for polygalacturonase activity.

PG POSITIVE YEASTS ON WINE-MIMICIKING MEDIUM

••

  provided by • Jacobs University -AG Nevoigt Yeast Collection, Bremen , Germany • Yeast Collection Sciences pour l'Oenologie, SupAgro, INRA, Montpellier, France • Lallemand Inc., Ontario, Canada C 68 of the 306 yeast strains showed polygalacturonase activity in Rich Medium pH 5.0 at 30°C 14 of the 68 yeast strains also showed polygalacturonase activity in Synthtetic Grape Must Medium pH 3.5 at 20°C Complete fermentation of must sugars to ethanol due to high ethanol tolerance and low pH (3.0-4.0 pH) Produce broad range of secondary metabolites • Secretion of extracellular enzymes 1 Department of Life Sciences and Chemistry, Jacobs University Bremen, Campus Ring 1, 28759, Bremen, Germany 2 Sciences pour l'Oenologie, INRA, Montpellier SupAgro, 2 Place Pierre Viala, 34060, Montpellier, France Aim: Screen pectinolytic yeasts for improving clarification and organoleptic properties in winemaking This project has received funding from the European Union'

Winemaking Process Yeast in winemaking Pectinolytic enzymes in winemaking Sampling Harvest Crushing / Pressing Fermentation Stabilizing Aging Filtration Bottling Consumption Yeast Enzyme

	• Increase juice yield
	• Increase concentration & variety of flavor
	compounds
	• Greater colour stability
	• Easy filterability
	• Speeds up settling
	• Shortens filtration time and stabilization
	processes
	• Color and flavor extraction


Angela Maria Coral Medina

Carole Camarasa

John Morrissey

Contribution of the interaction between environment and

come

Contribution of the interaction between environment and genotype to flavour and aroma

Angela Maria Coral Medina 1,2 , Carole Camarasa 2 & John Morrissey 1 1 Microbiology Department, University College Cork, Ireland; 2 INRA, SupAgro, Universite de Montpellier,

France

Saccharomyces uvarum has interesting properties that can be exploited for products of fermented beverage. Particularly, the cryotolerance and capacity to produce high amounts of volatile compounds offers new opportunities for the fermentation industry. Besides the contribution of the nitrogen source to protein formation, some nitrogen compounds are precursors of volatile molecules that produce aroma.

The nitrogen compounds assimilated by yeast are classified in rich, intermediate and poor nitrogen sources depending on how they affect the specific growth rate. In S. cerevisiae, the nitrogen metabolism is well understood but less is known about these pathways in S. uvarum, nor whether there are regulatory differences between the yeasts. The aim of the project is to understand how nitrogen metabolism is regulated in S. uvarum and what the effects of the nitrogen source are on the production of volatiles at low temperature. The focus is on temperatures below 20°C since is relevant for prodution of wine or cider. First, it was determined what nitrogen sources are preferred and non-preferred by S. uvarum MTF 3098 relative to S. cerevisiae. Next, we will determine how the nitrogen source affects the expression of key genes potentially involved in aroma production. Transcripome analysis under particular nitrogen and temperature conditions will be performed. Future work includes the correlation of the transcriptome and volatile metabolome to understand links between temperature, nitrogen regulation and aroma production in S. uvarum MTF 3098.


Rachid Riad
email: rachid.riad@ens.fr

Anne-Catherine Bachoud-Lévi

Frank Rudzicz

Emmanuel Dupoux
email: emmanuel.dupoux@gmail.com

Identification of primary and collateral tracks in stuttered speech

Keywords: evaluation metrics, disfluency, stuttering, speech processing, audio features

Introduction

Around 6% percent of spoken words in non-pathological speech are categorised as disfluent [START_REF] Tree | The effects of false starts and repetitions on the processing of subsequent words in spontaneous speech[END_REF] and this increases with the cognitive load of the speaker [START_REF] Bortfeld | Disfluency rates in conversation: Effects of age, relationship, topic, role, and gender[END_REF][START_REF] Lindström | The effect of cognitive load on disfluencies during in-vehicle spoken dialogue[END_REF]. Speaking in real time is a demanding activity, subject to cognitive constraints and pragmatic settings. Under time pressure, a word may not be retrieved, part of a sentence may be revised, unfilled and filled pauses may be inserted, words or part of words may be repeated. Some of these deviations can be viewed as the symptoms of sentence planning problems [START_REF] Mcroberts | The role of lexical access in spontaneous speech disfluencies[END_REF] or as the results of some strategies [START_REF] Clark | Repeating words in spontaneous speech[END_REF] unfolded under speaker's control to signal something. Stuttering is a severe case of speech pathology, that interrupts at a higher rate the flow of speech than in typical Speech Production. Indeed, in addition to the 'classic' disfluencies, stutterers can produce other forms of disfluency that are both quantitatively and qualitatively distinguishable from typical forms [START_REF] Lickley | Disfluency in typical and stuttered speech. Fattori sociali e biologici nella variazione fonetica-Social and biological factors in speech variation[END_REF]. Speech pathologists need to quantify all the disfluency events for clinical screening but also to assess potential treatments [START_REF] Yaruss | Clinical measurement of stuttering behaviors[END_REF]. A large number of factors and speaking settings influence stuttering behaviours and occurrences of disfluencies: interlocutor's characteristics (ex: age, relationship with the speaker), conversational settings (at home, at the hospital, at work), speaking tasks (ex: reading, dialogues, descriptions of scenes). The clinical assessment still rely heavily on subjective and one on one evaluation [START_REF] Yaruss | Overall assessment of the speaker's experience of stuttering (oases): Documenting multiple outcomes in stuttering treatment[END_REF]). An automatic, reliable procedure would provide Speech Pathologists an objective comparison between clinical facilities and treatments. Besides, detecting automatically disfluencies and stuttering symptoms from speech, in different settings, could unlock *Part of the work done at the Vector Institute during RR's summer internship in-home assessments and more frequent trainings for patients. Two main issues arise from the literature around disfluency detection. The first one is the lack of public pathological annotated datasets. The second issue is the absence of a clear evaluation protocol for the automatic detection of disfluencies. This might be due to the fact that different communities have different applications in mind. Speech pathologists are interested in the automatic classification of type and duration of disfluencies [START_REF] Yaruss | Clinical measurement of stuttering behaviors[END_REF]. Most used proprietary datasets, with patients performing reading tasks and where disfluent and non disfluent parts have been balanced [START_REF] Nöth | Automatic stuttering recognition using hidden markov models[END_REF][START_REF] Yildirim | Automatic detection of disfluency boundaries in spontaneous speech of children using audio-visual information[END_REF][START_REF] Oue | Automatic dysfluency detection in dysarthric speech using deep belief networks[END_REF]. Researchers in the NLP community are interested in modelling the disfluencies for several reasons [START_REF] Shriberg | To 'errrr'is human: ecology and acoustics of speech disfluencies[END_REF]): text normalisation for downstream tasks such as Dependency Parsing or Semantic Role Labeling. In addition, NLP researchers are also interested in disfluency detection for affective computing [START_REF] Tian | Emotion recognition in spontaneous and acted dialogues[END_REF] applications. They use features derived from transcribed text [START_REF] Honal | Correction of disfluencies in spontaneous speech using a noisy-channel approach[END_REF]) using Shriberg's formalism [START_REF] Shriberg | Preliminaries to a theory of speech disfluencies[END_REF], and focus on the detection of disfluencies in non-pathological speech in telephonic conversation using datasets like Switchboard [START_REF] Godfrey | Switchboard: Telephone speech corpus for research and development[END_REF]). Yet, the work from [START_REF] Goldwater | Which words are hard to recognize? prosodic, lexical, and disfluency factors that increase speech recognition error rates[END_REF] demonstrated that words preceding disfluent interruption points also have high error rates for speech recognition systems. Finally, psycholinguists and clinicians are interested in the distribution and type of disfluencies, which could inform on speech and language production systems [START_REF] Jackson | Responses of adults who stutter to the anticipation of stuttering[END_REF] [START_REF] Fromkin | The non-anomalous nature of anomalous utterances[END_REF] as well as diagnosis. Obviously, for this kind of application, running interview-like speech with minimal annotations would be preferable. Since several hybrid text/speech systems have been proposed [START_REF] Tran | Parsing speech: A neural approach to integrating lexical and acoustic-prosodic information[END_REF][START_REF] Yildirim | Automatic detection of disfluency boundaries in spontaneous speech of children using audio-visual information[END_REF], we believe that a common evaluation method would be beneficial to bridge the gap between these research communities. First, we introduce a new framework for the evaluation of disfluency detection which would be relevant both for spontaneous and/or pathological speech using metrics combining insights from both NLP and Speech Technologies (ST) communities (Section 2). Second, we test these metrics on a new dataset obtained by force-aligning an annotated corpus of pathological speech (Section 3). All annotations in Praat format and code for evaluation will be released on the GitHub repository of the first author RR 1 . Third, we compare the performance of different baseline systems across textual and speech inputs on this dataset that were usually used in NLP and ST (Section 4). Four, to bridge the gap in performance between NLP and ST methods, we introduce new audio features that improve on the different frame-based baselines (Section 5).

Metrics: primary and collateral tracks

We take inspiration from the H. Clark's theory of speech performance (Clark, 1996, p. 255), which states that speakers communicate using two parallel tracks. The primary track contains the traditional linguistic content of the discourse while the collateral track contains additional signals regulating the communication channel itself. Among these signals we find delays, (un)filled pauses, rephrasing, mistakes, laughs, or vocal noises. The extraction of these two tracks from continuous speech can be decomposed in several engineering tasks (see Figure 1). First, a diarization [START_REF] Anguera | Speaker diarization: A review of recent research[END_REF] component involves assigning stretches of signal (turns) to each speaker. This component is well studied and existing evaluation metrics can be used (see [START_REF] Bredin | Pyannote. metrics: a toolkit for reproducible evaluation, diagnostic, and error analysis of speaker diarization systems[END_REF] for a suite of diagnostic tools). Second, each turn is analysed in terms which sub-part contains collateral information (collateral detection task). This allows us to quantify how well the collateral track (only disfluencies in this work) is detected without specifying their category. The parsed segments can be evaluated in terms of a gold lexicon and a gold alignment. For this, we report the Detection Error Rate and the Detection F1-score. Third, each collateral sub-part is categorised into a small number of categories, which we restrict here to disfluency types Table 1: Metrics used for the detection and identification of the collateral track. T false alarm is the duration of false alarm (e.g. primary track classified as collateral), T miss detection is the duration of missed detection (e.g. collateral track classified as primary), T Collateral Track is the total duration of the collateral track in the reference, T confusion is the total duration of the confusion between disfluency labels. Detection F1-score is computed as there are only two classes (primary and collateral). Precision i and Recall i are computed as the detection formula where the positive class is the i-th disfluency Table 2 (identification task). For that purpose we use the Identification F1-score and the Identification Error Rate. The formula to obtain these metrics are summarised in the Table 2.. The two error rate metrics are defined with the similar formula to that used in the Diarization Error Rate, which means that they can go over 100% (the denominator being restricted here to the collateral track). All these metrics were coded using the python toolkit pyannote.metrics [START_REF] Bredin | Pyannote. metrics: a toolkit for reproducible evaluation, diagnostic, and error analysis of speaker diarization systems[END_REF]. Another motivation for this framework comes from speech pathology research on Stuttering evaluation. Indeed, from these timing prediction of the primary and collateral track, it can be computed automatically the Speech Efficiency Score (SES) introduced in [START_REF] Amir | The speech efficiency score (ses): A time-domain measure of speech fluency[END_REF]. This study demonstrated that this score, which is based on a timedomain analysis is closely equivalent to stuttering severity ratings done by speech pathologists. By solving the diarization task, and the disfluency detection task mentioned above, it is possible to obtain an estimation of the SES (see below the formula for the equivalence between our framework and their notations).

SES = TPrimary Track

TPrimary Track+TCollateral Track * 100 = TEfficient time TTotal time-TSilence * 100

Dataset

We built on FluencyBank, a large-scale open source audiovisual dataset primarily used by clinical researchers to study fluency [START_REF] Bernstein | Fluency bank: A new resource for fluency research and practice[END_REF], from which we selected and forced-aligned a consistent subset focused on stuttering. FluencyBank contains a collection of sub-datasets collected by different research groups to study typical and disordered fluency in infants and adults.

Figure 2: Distribution of disfluency in the FluencyBank-AWS per speaker. See table 2 for definitions of disfluencies.

We selected the Adult-who-Stutter(AWS) sub-dataset of2 FluencyBank, which contains video recordings focused on patients. We excluded the recordings where the annotation was lacking, and we obtained 22 speaker video interviews (1429 utterances and 24693 words). The original recordings were done while the participants answered questions of the OASES elicitation protocol [START_REF] Yaruss | Overall assessment of the speaker's experience of stuttering (oases): Documenting multiple outcomes in stuttering treatment[END_REF], and transcriptions and disfluencies were done in the original dataset at the sentence level. Table 2 provides the five classes of disfluencies that we consider here. We provide some examples for each class of patients answering some questions of the protocol. In this work, we did not consider blocks, syllable repetitions or prolongations. Yet, our formulation with the primary and collateral tracks can easily be extended to these disfluencies. We obtained the timings of the primary and collateral tracks by force-alignment at the phone level with the kaldi toolkit [START_REF] Povey | The kaldi speech recognition toolkit[END_REF] with a HMM-GMM model. Figure 2 shows the distribution of disfluencies per speaker in the dataset. The total number of disfluencies and their types vary greatly across speakers.

Multi-Repetition (MR)

I'm fortunate to be be be be in graduate school. Phrase Repetition (PR) they are they are so sweet. Retracing or Revision (RT) I ended when I was it ended when I was seventeen.

Baselines: text versus speech predictions

Here, we provide two different kind of baseline systems with the purpose of comparing textual and acoustic approaches on the same metrics: (1) word-based systems, which assume that the input speech has been segmented

4.1. Detection from aligned speech: word-based systems

Word-based systems can incorporate both textual and acoustic features (See Table 3). As for textual features, we used token and span features which are common in the NLP community [START_REF] Shriberg | Preliminaries to a theory of speech disfluencies[END_REF][START_REF] Charniak | Edit detection and parsing for transcribed speech[END_REF].

As for acoustic/prosodic, we use summary statistics on duration energy and F0. All statistics and pooling are done using the timing alignment of each word w i . The semantic representation and part-of-speech tags are extracted with [START_REF] Honnibal | spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing[END_REF]. The number of syllables and phones are extracted with [START_REF] Bernard | bootphon/phonemizer: phonemizer-1[END_REF].

w i == w i+k , k ∈ [-15, +15] * 30 p i == p i+k , k ∈ [-15, +15] * 30 w i , w i+1 == w i+k , w i+k+1 8 k ∈ [-4, +4] * p i , p i+1 == p i+k , p i+k+1 8 k ∈ [-4, +4] * Acoustic word duration (s) 1 Prosodic
number of syllables 1 number of phones 1 high, low and total energy 3 in filterbanks F0 mean, std, median, min, max 9 5%, 25%, 75% & 95% percentiles surrounding pause times (s) 2 pitch breaks inside 3 and around word limits Such hand-crafted features have been used previously in the literature [START_REF] Zayats | Disfluency detection using a bidirectional lstm[END_REF][START_REF] Ferguson | Disfluency detection with a semi-markov model and prosodic features[END_REF] and have shown to improve the prediction performance. Indeed, neighbour words and prosodic cues are very informative about the disfluency events [START_REF] Shriberg | Preliminaries to a theory of speech disfluencies[END_REF]. In [START_REF] Yildirim | Automatic detection of disfluency boundaries in spontaneous speech of children using audio-visual information[END_REF], they obtained that interrupting points in disfluencies are 98% associated with a pitch break. We obtained a similar result in the Fluen-cyBank AWS dataset with 95% of the disfluent boundaries that match with a pitch break in a 100 ms vicinity. All these features are normalized with the M axAbsScaler from scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine Learning in Python[END_REF] to avoid the loss of sparsity (specially in the span features). We compared 5 different models (see Table 4). The latest work is focusing on sequence tagging prediction with Recurrent Neural Network architectures [START_REF] Zayats | Disfluency detection using a bidirectional lstm[END_REF][START_REF] Tran | Parsing speech: A neural approach to integrating lexical and acoustic-prosodic information[END_REF]. We compared Forward and Bi-Directional architectures with Long-Short-Term-Memory (LSTM) Networks for disfluency detection and identification. The hidden dimension of the recurrent networks is set at 20. All the experiments are carried using the Adam training procedure with the default parameters [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] and early-stopping on a held-out validation set of 20% of spoken utterances of the training speakers. In addition, we used a discriminative approach with classical machine learning classifiers. Every word is supposed to be independent: the goal is now to predict each word individually, without its neighbour's representation or prediction. The information listed in Table 3 is already obtained by the aggregation of local and long range information and could be sufficient to make predictions. We compare a standard classic Support Vector Machine with linear kernel (SVM) and a penalty term of C = 0.025, a L2 regularised logistic regression and a classic deep forward neural network (DNN) with 2 hidden layers with 100 and 50 hidden units. In the results, we also report the Token F1 score 4 at the word level to compare the predictions. This differs from the F1-score reported for disfluency detection in the NLP community [START_REF] Godfrey | Switchboard: Telephone speech corpus for research and development[END_REF]. Usually, NLP models tries to detect the disfluency and identify the subparts of each disfluency (reperandum, interegnum, repair), not the different categories of disfluencies.

Detection from raw speech: frame-based systems

Frame-based systems have no other information than speech. In principle, they are closer to what would be useful for clinical purposes, but obviously, the task is much harder. For comparison purposes we propose 3 baseline frame-based systems. Here, we evaluate frame-level prediction for the disfluency detection and identification as in [START_REF] Oue | Automatic dysfluency detection in dysarthric speech using deep belief networks[END_REF]. The patterns of the disfluencies in the speech signal can range from very local phenomenon (filled pauses) to long time-scales (retracing). Here, we investigate the predictions are made every 10 ms, in a bottom-up manner, using only local features. Speech represented using a bank of 40 log-energy Melscale filters representing 25 ms of speech (Hamming windowed) every 10 ms. The Mel features are mean-variance normalised per file, using the VAD information. Besides, we extract prosodic features with the F0 trajectory and its first derivatives in a 50 ms window (obtaining a 56dimension vector as F0 is computed every 1.8ms). These spectral and prosodic representation are concatenated to obtain a final 106-dimension vector representation every 10 ms. All the frame-based systems use a window of 7 stacked frames [START_REF] Oue | Automatic dysfluency detection in dysarthric speech using deep belief networks[END_REF].

Based on these representations directly extracted from the signal, we follow a similar procedure as in the word-based predictions: we compare a standard classic Support Vector Machine with linear kernel (SVM) and a penalty term of C = 0.025, with a classic deep forward neural network (DNN) with 2 hidden layers with 100 and 50 hidden units. These approaches have been previously used in stuttering detection literature (Chee et al., ).

As in many machine learning problems [START_REF] Lemaître | Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning[END_REF], disfluency datasets have the attribute to be very imbalanced. The number of frames that are labelled fluent exceeds by a large margin all the others classes (92.7% of the frames are labelled as fluent). We evaluate a random undersampler technique [START_REF] Lemaître | Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning[END_REF] that discards randomly a large number of the majority class (here fluent) before training each model. This undersampling strategy has been used in Speech Technologies, yet, systems had not been evaluated on running speech datasets.

Audio Span Features

We want to improve the frame-based system using information over a long time span and replace the textual features with equivalent ones directly from the raw speech. We introduce here our Audio Span Features. The goal of these features is to obtain similar information as the span features from the word-based systems (Table 3).

Our main assumptions for disfluency events are: (1) Repetition-like disfluency events exhibit a common underlying structure property in the frequency domain, (2) filled pauses exhibit specific acoustic correlates with a steady frequency signature [START_REF] Gabrea | Detection of filled pauses in spontaneous conversational speech[END_REF], (3) these filled pauses have usually adjacent unfilled pauses/silences [START_REF] Daly | Acoustic study of verbal hesitations, filled pauses, and unfilled pauses in spontaneous speech[END_REF].

That is why we posit that local neighbour-similarities in the frequency domain can approximate the span features for the word comparisons from Table 3. Besides, different chunk size can also inform on the different type of disfluencies (close and more local similarities are triggered by fillers versus spaced and long range similarities are triggered by repetitions of words). Therefore, for every time-step t, for a given window-scale s, we compute the similarity ψ(t, s, i) of the frequency representation x t centered on t with its i-th closest neighbours.

We compute this similarity with the N previous neighbours and the N next. The frequency representation x t ∈ R 40 is still the bank of 40 log-energy Mel-scale filters computed every δ = 10ms. These neighbours are centered centered every

t s i = t + s • i • δ.
The scale s is the (odd) number of stacked frames. So we denote by x s t ∈ R 40•s the concatenation of the s frames around t:

x s t =        x t-( s-1 2 )•δ x t-( s-1 2 +1)•δ ... x t+( s-1 2 -1)•δ x t+( s-1 2 )•δ       
Finally, our Audio Span Features can be computed:

∀i ∈ [[-N, N ]] * , ψ(t, s, i) = x s t • x s t s i • 1 n s (1)
We divided the similarity by n s = 40 • s to normalise in the scale dimension and not privilege the bigger stacked frames similarities. We computed this similarity for 8 different scales with a logarithmic spacing for the different scales between 30 ms and 1 s (s ∈ [101,61,37,23,13,9,5,3]).

We choose these numbers to capture different orders of magnitude that characterise disfluent segments of speech: phones (∼ 30ms), words (∼ 100ms), sentences (∼ 1s). We finally chose N = 4 for the number of neighbours (i ∈ [-4, -3, -2, -1, 1, 2, 3, 4]). Now, for every time-step t we concatenate the neighbour cross-similarities at different scales and obtain the final vector ψ(t) ∈ R 4•2•8=64 . See Figure 3 for a schematic representation of the computations. We evaluate these new Audio Span Features alone and along the acoustic and prosodic representation described in subsection 4.2. We report the evaluation with the Standard Sampler and the DNN model as in 4.2.

Results and Discussions

Table 4 shows performances on the detection and identification of primary and collateral tracks. We first review the results from the word-based predictions methods when we take all features as input. Overall, we observed that Sequence-to-sequence models underperform compared to more classical machine learning classifiers. We hypothesise that if the data gets larger the LSTMs architectures might catch on compared to the classifiers. With respect to the F1-score in the detection and identification tasks, the LSTMs architecture are actually not that far from classifiers. Yet, there is an important drop in performance on Error Rates. This highlights the importance to take into account more than one composite score. Among these classifiers, good old SVM-linear model yields the best performances in almost all metrics (except in the Detection recall for the Logistic Regression). proves by a large margin the Detection Recall. By contrast, the Precision metrics, the Detection Error Rate and Identification Error Rate are way above 100%. This shows that with extreme class imbalance, frame-based methods that were previously shown reasonable performance in balanced datasets fail in a spectacular fashion. This highlights the importance of addressing the issue of the detection of disfluencies using running speech rather than artificially bal-anced datasets or read-speech.

The new Audio Span Features alone demonstrate really poor performance and are missing almost all the disfluency events (Detection Recall and F1 lower than 0.001). However, the Audio Span Features along the acoustic and prosodic representation show the best performance on the frame-based system, especially in the identification task (Identification F1 0.118 and Error Rate below 1). The system misses a number of disfluent events (Low Detection Recall 0.063), but maintain a good precision level in comparison to the other frame-based baselines (Detection Precision 0.488 and Identification Precision 0.450). The Audio Span Features do not capture all the necessary information and can be improved. Especially, our Audio Span Features do not have grammatical information captured by the wordbased span features. One of our hypothesis is that the Audio Span Features fail also to detect the revision/retracing disfluent events. To better understand the impact of the input features, we ran an ablation study for the word-based predictions, see Table 5. We compare the different combinations of features as defined in Table 3. First, the acoustic/prosodic features are not informative on their own to predict disfluencies. Span are better but still not reach the full model performance. They might be more suitable to detect the repetition-like disfluencies but not necessarily for the Filled pauses. Obviously, the token based representations have a clear advantage especially for Filled pauses. The acoustic/prosodic features provide a little gain for the span and token representation, but the combination of span and token is already sufficient on its own and gets very close to the combination of all features. This study could orient future work to bridge the gap in performance between our frame-based predictions and wordbased predictions. Indeed, the semantic and grammatical part-of-speech features play an important part in the good results of the word-based systems. To obtain such features from the signal, we could build an Automatic-Speech-Recognition pipeline suited for Stutterers and obtain the word2vec representations [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. Or we can obtain such semantic information directly from the signal [START_REF] Chung | Speech2vec: A sequence-to-sequence framework for learning word embeddings from speech[END_REF].

Conclusions

In this work, we investigated a framework to evaluate the disfluencies detection system in stuttered speech. First, we prepared and adapted an open dataset of Adult-Who-Stutter used by clinical researchers, for the task of disfluency detection from running speech. We provided a suite of metrics based on the forced alignment, that enables to compare word-based predictions and frame based-predictions. This allows the direct comparison between different type of approaches. Finally, we compared different baselines systems with textual or acoustic input features, and using word-or frame based pooling of information. The wordbased systems show superior performance, illustrating the need (1) to improve frame-based aggregation of information over a long time span and (2) replace textual features with equivalent ones that can be derived automatically from raw speech. Finally, we introduced new Audio Span Features that show the best performances for the frame-based methods.

Figure 1 :

 1 Figure 1: Schematic diagram of identification of the primary and collateral tracks of communication. In black the primary track (fluent), in other colors the words in the collateral track, in blue a filler, in green a phrase repetition.

T

  false alarm +T missed detection +T confusion T Collateral Track

  Figure 3: Audio Span Features: It is the concatenation of similarities between the current representation with the 2 • N closest neighbours filterbank representations at different scales s.

Table 2 :

 2 Collateral signals taxonomy (usually called disfluency) under consideration here in the FluencyBank dataset: Italic for the primary track and Bold for the collateral track.

	Disfluency	Example
	Filled pause (F)	I was primarily uh focused
		on fluency.
	Single word Repetition (R) I I don't like switch word.

Table 3 :

 3 List of features in the word-based prediction.

	Type	Core features and dimension
	Token	semantic representation	384
		part-of-speech (pos): p i	19
		word position	1
	Span

Table 4 :

 4 Results of the evaluation of detection and identification of primary and collateral track for the different approaches described in Section 4. The best scores for each metric for each condition (word vs frame based) are in bold, best metrics overall are underlined. For the evaluation of the Audio Span Features, we report the performance with a DNN model trained with the Standard sampler.

	Model	NLP		Detection			Identification
		Token	P	R	F1	Error	P	R	F1	Error
		F1				Rate				Rate
	Word-based 4.1 (all features)
	Forward LSTM	0.416 0.823 0.595	0.691	0.623	0.717 0.518	0.601	0.701
	Bi-LSTM	0.417 0.786 0.605	0.684	0.731	0.701 0.537	0.608	0.799
	SVM-Linear	0.569 0.966 0.642	0.771	0.381	0.905 0.599	0.721	0.424
	Logistic Regression	0.544 0.846 0.645	0.732	0.513	0.762 0.576	0.656	0.581
	DNN	0.485 0.958 0.611	0.746	0.417	0.855 0.544	0.665	0.484
	Frame-based 4.2 (Baseline Signal features only)
	Standard sampler + DNN	-	0.312 0.014	0.026	1.005	0.182 0.010	0.020	1.008
	Undersampler + DNN	-	0.073 1.000	0.136	15.286 0.038 0.520	0.069	15.766
	Standard sampler + SVM	-	0.150 0.086	0.109	1.502	0.116 0.067	0.086	1.520
	Undersampler + SVM	-	0.077 0.838	0.140	12.529 0.025 0.288	0.048	13.079
	Frame-based 5 (Audio Span Features)
	Audio Span Features +	-	0.864 ≤ 1e-3 ≤ 1e-3 1.000	0.818 ≤ 1e-3 ≤ 1e-3 1.003
	Standard Sampler + DNN
	Audio Span Features +	-	0.488 0.063	0.112	0.986	0.450 0.059	0.105	0.990
	baselines features + Stan-
	dard Sampler + DNN

Table 5 :

 5 Results of the evaluation of detection and identification of primary and collateral track for the different input features in the word-based predictions with a SVM model with linear kernel. The best error rates for each metric overall are in bold.

	Features	Detection	Identification
		F1	Error	F1	Error
			Rate		Rate
	Word-based (SVM-Linear)
	Token only	0.639	0.537 0.622	0.550
	Span only	0.313	0.826 0.263	0.856
	Acoustic only	≤ 0.001 1.000 ≤ 0.001 1.000
	Acoustic+Span 0.314	0.825 0.269	0.852
	Acoustic+Token 0.646	0.529 0.628	0.543
	Token+Span	0.772	0.382 0.720	0.427
	All (line 3	0.771	0.381 0.721	0.424
	from Table 4)

https://fluency.talkbank.org/access/Voices-AWS.html into words, and aggregate textual and/or acoustic features over the entire span of each word (2), frame-based system which make decision on a frame-by frame basis from raw speech. Obviously, the latter kind of system cannot use textual features. All evaluations are performed with leaveone-speaker-out cross-validation, so that we can assess the generalisation to unseen speaker.

Acknowledgements

This work is funded through a Facebook AI Research grant, and supported by INRIA, as well as grants ANR-10-IDEX-0001-02 (PSL*) and ANR-17-EURE-0017 and grants from FacebookAI Research (Research Grant), Google (Faculty ResearchAward) and Microsoft Research (Azure Credits and Grant).


Sara Akodad
email: sara.akodad@ims-bordeaux.fr

Lionel Bombrun
email: lionel.bombrun@ims-bordeaux.fr

Yannick Berthoumieu
email: yannick.berthoumieu@ims-bordeaux.fr

Christian Germain
email: christian.germain@ims-bordeaux.fr

CLUSTER KERNEL FOR LEARNING SIMILARITIES BETWEEN SYMMETRIC POSITIVE DEFINITE MATRIX TIME SERIES

Keywords: Multivariate times series, classification, second-order statistics, log-Euclidean, remote sensing

The launch of the last generation of Earth observation satellites has yield to a great improvement in the capabilities of acquiring Earth surface images, providing series of multitemporal images. To process these time series images, many machine learning algorithms have been proposed in the literature such as warping based methods and recurrent neural networks (LSTM, . . . ). Recently, based on an ensemble learning approach, the time series cluster kernel (TCK) has been proposed and has shown competitive results compared to the state-of-the-art. Unfortunately, it does not model the spectral/spatial dependencies. To overcome this problem, this paper aims at extending the TCK approach by modeling the time series of second-order statistical features (SO-TCK). Experimental results are conducted on different benchmark datasets, and land cover classification with remote sensing satellite time series over the Reunion Island.

INTRODUCTION

A time series is a sequence of data points which has been acquired during an ordered time segment. Many real-world pattern recognition tasks deal with time-series analysis [START_REF] Box | Time Series Analysis, Forecasting and Control[END_REF] such as biomedical signals (e.g. EEG and ECG), financial data (e.g. stock market and currency exchange rates), industrial devices (e.g. gas sensors and laser excitation), biometrics (e.g. voice, signature and gesture), video processing, music mining, forecasting, etc. Recently, the launch of the last generation of Earth observation satellites such as Sentinel-1 and 2 has yield more recurrent acquisition of Earth surface images. These sensors allow the acquisition of multivariate time series such as spectral surface reflectance in several wavelengths. The availability of these multivariate time series has raised the interest of the image processing community to develop novel machine learning strategies for supervised classification. In particular, many deep learning approaches have recently been proposed [START_REF] Ismail Fawaz | Deep learning for time series classification: a review[END_REF]. They include convolutional neural networks (CNN) [START_REF] Krizhevsky | ImageNet classification with deep convolutional neural networks[END_REF] and deep recurrent neural networks (RNN) such as long short-term memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF][START_REF] Ienco | Land cover classification via multitemporal spatial data by deep recurrent neural networks[END_REF] and gated recurrent units (GRU) [START_REF] Cho | On the properties of neural machine translation: Encoder-decoder approaches[END_REF]. In the following, these ap-proaches will not be considered since the training process involves large data for avoiding overfitting and generalization purpose, while the aim of this paper is to work with relatively small datasets. Other non deep learning approaches can be decomposed into three categories: distance-based time series classifiers, feature-based time series classifiers and ensemble classifiers. Distance-based strategies rely on a point-to-point distance between time series which is then fed in a conventional classifier such as a k nearest neighbor or SVM. In this family, the most popular approach is certainly the dynamic time warping (DTW) which measures the similarity between two time series by aligning them [START_REF] Sakoe | Dynamic programming algorithm optimization for spoken word recognition[END_REF][START_REF] Berndt | Using dynamic time warping to find patterns in time series[END_REF]. It has the ability to match time series that are distorted and shifted along the temporal axis. Inspired by the principle of DTW, some closely related approaches have been proposed such as derivative DTW (DDTW) [START_REF] Keogh | Derivative dynamic time warping[END_REF] or weighted DTW (WDTW) [START_REF] Jeong | Weighted dynamic time warping for time series classification[END_REF]. Kernel based methods have also been introduced such as the global alignment kernel (GAK) [START_REF] Cuturi | Fast global alignment kernels[END_REF] for the family of DTW distances. More recently, many feature-based methods have been proposed. It consists in extracting features such as wavelet coefficients or 1D SIFT descriptors from multivariate time series before the classification step. Among these feature-based methods, codebook based representations have raised an interest [START_REF] Wang | Bag-of-words representation for biomedical time series classification[END_REF]. For example, the bag-of-words (BoW) model has been used in [START_REF] Bailly | Dense bag-of-temporal-sift-words for time series classification[END_REF] to obtain an histogram representation of the time series by encoding SIFT features in a codebook. The third family concerns ensemble based classifier systems. The basic idea relies on the combination of multiple classifiers in order to obtain more accurate and robust decisions. Once again, many approaches can be considered. For example, a random forest classifier trains a single base classifier (i.e. decision tree) on different subsets of training data (sample, attributes and/or temporal subsets). Another strategy consists in using different classifiers on the same dataset such as in the collective of transformation-based ensembles (COTE) [START_REF] Bagnall | Time-series classification with cote: The collective of transformation-based ensembles[END_REF] and its extension based on a hierarchical vote (HIVE-COTE) [START_REF] Lines | Hive-cote: The hierarchical vote collective of transformation-based ensembles for time series classification[END_REF] where 35 and 37 standalone classifiers are respectively considered. Even if this later has demonstrated successful results and is considered as the reference for time series classification [START_REF] Bagnall | The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances[END_REF], it suffers from an high computation cost since each classifier should be trained on the whole dataset. Since then, in order to get benefit of the ad-vantages of kernel methods, codebook based representations and ensemble learning strategies, Mikalsen et al. have introduced the time series cluster kernel (TCK) method in [START_REF] Mikalsen | Time series cluster kernel for learning similarities between multivariate time series with missing data[END_REF] which has demonstrated competitive results for times series classification.

At the same time, second-order features have shown a great interest for many image processing applications including person re-identification, texture recognition, material categorization or EEG classification in brain-computer interfaces to cite a few of them [START_REF] Faraki | More about VLAD: A leap from Euclidean to Riemannian manifolds[END_REF][START_REF] Barachant | Classification of covariance matrices using a Riemannian-based kernel for BCI applications[END_REF][START_REF] Said | Texture classification using Rao's distance on the space of covariance matrices[END_REF]. For example, the use of covariance matrices has demonstrated to be successful in [START_REF] Akodad | Image classification based on log-Euclidean Fisher vectors for covariance matrix descriptors[END_REF] for remote sensing scene classification. This strategy, Hybrid LE FV, achieves better results than its analog Hybrid FV [START_REF] Li | Integrating multilayer features of convolutional neural networks for remote sensing scene classification[END_REF] which exploits only first-order features. Since then, the main contribution of this paper is to present how TCK proposed in [START_REF] Mikalsen | Time series cluster kernel for learning similarities between multivariate time series with missing data[END_REF] for multivariate time series can be extended to work with second-order feature time series. The paper is structured as follows. Section 2 presents and discusses the principle of TCK. Then, Section 3 introduces the proposed extension of TCK for the modeling of the time series of second-order statistics (SO-TCK). An application on different datasets is next presented in Section 4. Finally, Section 5 concludes this paper and provides some perspectives of this work.

TIME SERIES CLUSTER KERNEL (TCK)

TCK has recently been introduced in [START_REF] Mikalsen | Time series cluster kernel for learning similarities between multivariate time series with missing data[END_REF] for the classification of multivariate time series. It exploits the power of kernel methods, codebook based representations and ensemble learning strategies. The global principle is explained in Fig. 1. The main idea behind this method is to compute a positive semi-definite similarity measure (i.e. a kernel) between two multivariate times series (MTS). For that, a GMM model is first trained on the training set, which is next used to encode each multivariate time series. Moreover, to ensure robustness, an ensemble learning strategy is considered. The next subsection presents the main steps during training and testing.

Training phase

An MTS X is represented as a matrix of dimension V × T where V is the number of attributes and T is the time length. It is a finite sequence of V univariate time series, i.e. X = x v ∈ R T for v = 1, . . . , V . During this stage, N MTS are considered for training, and X (n) represents the n th MTS sample. To cope with missing data, a second MTS is considered. R (n) is a binary MTS defined by r

(n) v (t) = 0 if the value x (n) v (t) is missing and r (n) v (t) = 1 otherwise.
As explained before, TCK is based on an ensemble learning approach. Practically, Q subsets are considered, where each subset is a subsample of N q data, V q attributes and T q time segments extracted from the training set. Then, for each subset, a codebook is created by learning a GMM model which is next used to encode each MTS.

Codebook: GMM modeling

For the q th subset, the training set is composed of N q MTS X (n) q of dimension V q × T q with their associated binary MTS R (n) q . A GMM model with G components is considered to learn a codebook, where its probability density function for the incompletely observed MTS (X

(n) q , R (n)
q ) is given by:

p(X (n) q |R (n) q , Θ) = G g=1 ωg Vq v=1 Tq t=1 N (x (n) v (t)|µgv(t), σgv) r (n) v (t) , (1)
where Θ = {ω g , µ gv , σ gv } for g = 1, . . . , G. MTS are assumed to have time-dependent means, i.e. {µ gv ∈ R Tq } for each attribute v. To enforce regularity, a kernel-based Gaussian prior is defined for the mean. In addition, the covariance matrix for each Gaussian component is assumed to be diagonal and constant, that is Σ g = diag{σ 2 g1 , . . . , σ 2 gVq } and σ 2 gv is the variance of attribute v for data belonging to the g th cluster. To estimate the GMM parameters Θ, a MAP-EM algorithm is considered. For more details, the interested reader is referred to Algorithm 1 in [START_REF] Mikalsen | Time series cluster kernel for learning similarities between multivariate time series with missing data[END_REF]. The set composed by the estimated GMM parameters represents the codebook. To ensure even more robustness, the number of GMM components G and the initialization of the MAP-EM algorithm are randomly selected for each subset.

Coding

The encoding of the n th MTS X (n) q for subset q is carried out by computing the posterior assignment Π (n) q , obtained by:

Π (n) q (g) = ωg Vq v=1 Tq t=1 N (x (n) v (t)|µgv(t), σgv) r (n) v (t) G g=1 ωg Vq v=1 Tq t=1 N (x (n) v (t)|µgv(t), σgv) r (n) v (t) . (2)
In the following, vector Π

q containing the G posterior probabilities Π (n) q (g) is considered to define the feature map used in the kernel.

Testing step

To compute a similarity measure between a training MTS X (n) and a testing MTS X (m) , a kernel based approach is considered. It is built on the basis of an inner product between two posterior distributions to form a linear kernel in the space of posterior distributions as:

K X (n) , X (m) = K nm = Q q=1 Π (n) q T Π (m) q , (3)
where Π

(n) q and Π (m) (q) are respectively the vector of posterior probabilities for the training and testing MTS obtained with (2). In the end, a nearest neighbor classifier is used with the induced distance d between X (n) and X (m) Fig. 1. General principle of the time series cluster kernel (TCK) [START_REF] Mikalsen | Time series cluster kernel for learning similarities between multivariate time series with missing data[END_REF].

given by:

d 2 X (n) , X (m) = K X (n) , X (n) -2K X (n) , X (m) + K X (m) , X (m) . ( 4
)
As observed in ( 3) and ( 4), two MTS will be similar if their posterior probability vectors Π

(n) q and Π (m) q are similar for each subset q. When a low number G of GMM components are considered, the kernel measures the similarity at large scale. While for large G, it focuses more locally. Hence, varying the number of components in the GMM model allows to capture both local and global structures in the data.

TIME SERIES CLUSTER KERNEL FOR SECOND-ORDER STATISTICAL FEATURES (SO-TCK)

In the literature, second-order features, in particular covariance matrices features, have proved to play an important role in different tasks related to visual recognition process [START_REF] Julesz | Inability of humans to discriminate between visual textures that agree in second-order statistics-revisited[END_REF]. Compared to first-order feature based classification algorithms, many authors have shown the interest of exploiting second-order statistics such as covariance matrix attributes [START_REF] Faraki | More about VLAD: A leap from Euclidean to Riemannian manifolds[END_REF][START_REF] Barachant | Classification of covariance matrices using a Riemannian-based kernel for BCI applications[END_REF][START_REF] Said | Texture classification using Rao's distance on the space of covariance matrices[END_REF]. This kind of data has a particular structure, they are symmetric positive-definite (SPD) matrices. This section's focus is twofold. First, we present how second-order features are extracted from a MTS to form SPD matrix time series (SPD-MTS). Secondly, we introduce the log-Euclidean representation of SPD-MTS in order to exploit them in TCK, yielding to the proposed SO-TCK method.

SPD matrix time series (SPD-MTS)

In this work, each considered time series is represented by a set of time-dependent second-order features which constitutes a SPD matrix time series (SPD-MTS) as shown in Fig. 2. Those latter are computed on a sliding temporal window of dimension ∆t. Hence, for the q th subset, we associate for each MTS of dimension V q × T q a SPD-MTS of dimension V q × V q × (T q -∆t + 1).

Fig. 2. Illustration of SPD-MTS computation.

In order to improve the feature representation and enhance classification performance, a full local Gaussian descriptor can be exploited. In this model, the local mean vector µ is jointly exploited with the V q ×V q covariance matrix M which gives the augmented SPD matrix of dimension (V q + 1) × (V q + 1) as proposed in [START_REF] Lovric | Multivariate normal distributions parametrized as a riemannian symmetric space[END_REF]:

M augmented = |M| -1 Vq +1 M + µµ T µ µ T 1 . (5)

Log-Euclidean representation of SPD-MTS

In order to adapt TCK to work with SPD-MTS, the geometry of the space P d of d × d symmetric and positive definite (SPD) matrices should be considered. In case of using covariance matrices, d = V q , while for the augmented SPD matrix:

d = V q + 1.
As observed in Fig. 2, SPD matrices lie in a convex cone which is a Riemannian manifold. Tools developed in the context of Euclidean geometry are hence not adapted to manipulate these data points. A Riemannian metric is better suited such as the log-Euclidean (LE) one [START_REF] Arsigny | Log-Euclidean metrics for fast and simple calculus on diffusion tensors[END_REF]. It consists in projecting the set of SPD matrices on a tangent space defined at a reference point, classically considered at the identity matrix. After being projected on the tangent space, tools of the Euclidean geometry can used such as the MAP-EM algorithm defined in Section 2.1.1 to estimate GMM parameters and the Kernel method presented in Section 2.2. Practically, each SPD matrix M is mapped on the tangent space by applying the following operations:

m = Vec(logm(M)), (6)
where logm() is the matrix logarithm and Vec() the vectorization operator:

x = Vec(X) = X 11 , √ 2X 12 , . . . , √ 2X 1d , X 22 , √ 2X 23 , . . . , X dd . ( 7
)
To summarize, these operations lead to a transformation from a SPD matrix M ∈ R d×d to a vector m ∈ R d(d+1) 2

.

EXPERIMENTS

In this section, we illustrate the potential of the proposed approach to multivariate time series classification on four benchmark datasets from UCI machine learning repository [START_REF] Dua | UCI machine learning repository[END_REF] and one on a remote sensing (RS) application. For this latter, the goal is to predict 9 land cover classes from a set composed by a series of T = 23 Landsat 8 images above the Reunion Island. It contains V = 10 attributes (7 surface reflectances and 3 vegetation/water indices) for each pixel at each timestamp. Table 1 gives the main characteristics of these datasets.

In the following, performance are measured in terms of mean overall accuracy evaluated over 5 runs. Fig. 3 draws the evolution of the overall accuracy on the Libras dataset as a function of the temporal support ∆t for the proposed SO-TCK approach. Three kind of second-order descriptors are considered: the covariance matrix (in red), the second order moment (E xx T , in blue) and the full local Gaussian descriptor (Augmented SPD matrix in green). As observed, best results are obtained for this latter. In the following, SO-TCK will refer to the classification performance obtained with this full local Gaussian descriptor. Note also that the choice of ∆t reflects a trade-off. It should be small enough to preserve a good temporal localisation, and at the same time sufficiently large for estimation purpose.

Datasets

Secondly, a classification comparison is conducted over five datasets. Two state-of-the-art approaches are employed, namely DTW [START_REF] Sakoe | Dynamic programming algorithm optimization for spoken word recognition[END_REF] and TCK [START_REF] Mikalsen | Time series cluster kernel for learning similarities between multivariate time series with missing data[END_REF]. We have adopted the same experimental setup as the one used in [START_REF] Mikalsen | Time series cluster kernel for learning similarities between multivariate time series with missing data[END_REF]. For reproducibility purpose, TCK has been launched with the authors Matlab implementation. As observed in Table 2, the best results are Fig. 3. Classification comparison between three considered second-order statistics for SPD-MTS computation (V q = 2).

obtained for the proposed SO-TCK approach with a gain of 2 to 15 % compared to DTW and TCK, hence illustrating its potential for various application on time series classification.

Datasets DTW [START_REF] Sakoe | Dynamic programming algorithm optimization for spoken word recognition[END_REF] TCK [START_REF] Mikalsen | Time series cluster kernel for learning similarities between multivariate time series with missing data[END_REF]

CONCLUSION

Inspired by the time series cluster kernel (TCK) and the potential of second-order statistical descriptors for many classification tasks, this paper has extended the formalism of TCK to second-order features: SPD matrix time series (SPD-MTS). For that, the log-Euclidean metric has been considered to represent a SPD-MTS as a multivariate time series where the principle of TCK can be employed. Experimental results on benchmark datasets and land cover classification with remote sensing data have shown the potential of the proposed method compared to state-of-the-art times series classification algorithm. Future works will include the evaluation of the proposed method for forest health monitoring using Sentinel-2 time series images.

Table 1 .

 1 Presentation of the five time series datasets.

		V	T Train Test Classes
	Libras	2 23	180	180	15
	Natops	3 51	180	180	6
	Char.Traj.	3 23	300	2 558	20
	RacketSports 6 30	151	152	4
	RS	10 23	900	900	9

Table 2 .

 2 Comparison of classification performance on five benchmark datasets.

	SO-TCK

ACKNOWLEDGMENT

This work was financially supported by the "PHC Sakura" program (project number 45095SK), implemented by the French Ministry for Europe and Foreign Affairs, the French Ministry of Higher Education, Research and Innovation and the Japan Society for Promotion of Science. The authors would also acknowledge the financial support of Bordeaux Science Agro and the Regional Council of Nouvelle Aquitaine.