Laetitia Le Pourhiet
email: laetitia.lepourhiet@upmc.fr

Dave A May

Lucas Huille

Louise Watremez

Sylvie Leroy

A Genetic Link Between Transform and Hyper-Extended Margins

Keywords: continental rifting, passive margins, polyphasing, palaeogeography, geodynamic modelling

The similarity between the geometry of the West African and South American coastlines is among one of the strongest natural observations supporting the plate tectonic paradigm. However, using classical plate tectonic approaches to model these conjugate transform margins results in a high degree of variability in palaeogeographic reconstructions. Using state-of-the-art 3D coupled thermo-mechanical numerical models, we simulate for the first time, crustal deformation at the onset of oceanisation along large offset oblique margins. Our models show that obliquity causes oceanic rift propagation to stall, resulting in an apparent polyphased tectonic evolution, and in some circumstances leads to the formation of hyper-extended margins. As a result, conjugate margins located at the edge of future fracture zones are highly asymmetric from rifting to spreading, with their lengths differing by a factor of 5 to 10, before the the final phase of break-up occurs.

Accounting for this discrepancy should ameliorate future palaeogeographic reconstructions.

Introduction

The conceptual model of a transform margin (Fig. 1) is an adaptation from the kinematic model of a transform plate boundary of [START_REF] Francheteau | Marginal Fracture Zones as Structural Framework of Continental Margins in South Atlantic Ocean[END_REF]. In this model, the interior of the plates are assumed to be rigid, except along diverging boundary segments (dark grey regions in Fig. 1). Even in deforming areas, the direction of transport is everywhere parallel to the relative plate motion. This model was subsequently adopted by marine geologists to understand the dynamics of extremely oblique passive margins [START_REF] Mascle | Evidence for transform margin evolution from the Ivrory Coast-Ghana continental margin[END_REF].

Extending this idea, [START_REF] Basile | The Ivory Coast-Ghana transform margin: a marginal ridge structure deduced from seismic data[END_REF] recognised that the edge of the divergent and transform segments display different morphologies and vertical displacement history. They introduced the terms of concave and convex transform divergent segments based on the shape of the of oceanic domain (Fig. 1A). Within the rigid transform margin models, these differences are explained by the interaction of the margin with the ridge during the early stage of drifting (Fig. 1B) which causes a secondary heat pulse and uplift [START_REF] Rüpke | Basin modelling of a transform margin setting: structural, thermal and hydrocarbon evolution of the Tano Basin, Ghana[END_REF]. This process also affects the transform margin between the stages depicted in Fig. 1B and Fig. 1C, and according to the rigid kinematic model, without resulting in any horizontal deformation.

Using the transform margin model to reconstruct plates motion is known to fail at large offsets.

Attempts to geometrically match conjugate margins typically lead to isolated pockets of oceanic crust along rifted segments [START_REF] Bullard | The fit of the continents around the Atlantic[END_REF][START_REF] Rabinowitz | The Mesozoic South Atlantic Ocean and evolution of its continental margins[END_REF][START_REF] Unternehr | South Atlantic fits and intraplate boundaries in Africa and South America[END_REF], or overlapping regions of continental blocks [START_REF] Moulin | A new starting point for the South and Equatorial Atlantic Ocean[END_REF]. The failure of the classical model is most frequently explained by pre break-up deformation [START_REF] Torsvik | A new scheme for the opening of the South Atlantic Ocean and the dissection of an Aptian salt basin[END_REF], or, as suggested by [START_REF] Vink | Continental rifting and the implications for plate tectonic reconstructions[END_REF]; [START_REF] Turner | Structure and evolution of an obliquely sheared continental margin: Rio Muni, West Africa[END_REF], by a delay in the break-up along the oblique segment. Both explanations clearly indicate that tectonic plates deviate from the rigid kinematic block theory outlined in Fig. 1.

Paleogeographic reconstructions can account for distributed strain, but they require quantitative constraints on the rate of deformation, and the obliquity of the structures which accommodate the strain. Recently, high-resolution two-dimensional dynamic models of break-up have been utilised to better understand the syn-rift deformation and provide constraints for paleogeographic reconstructions [START_REF] Brune | Rift migration explains continental margin asymmetry and crustal hyper-extension[END_REF][START_REF] Brune | Abrupt plate accelerations shape rifted continental margins[END_REF]. Nevertheless, the conclusions drawn from such models need to be demonstrated in oblique rifting context. Modelling oblique setting naturally mandates the usage of three-dimensional dynamic models.

To date, two classes of 3D rift models have been considered. The first class have been designed to focus on the localisation of the deformation within large oblique weak zone [START_REF] Tron | Experiments on oblique rifts in brittle-ductile systems[END_REF][START_REF] Mart | Analogue experiments of propagation of oblique rifts[END_REF][START_REF] Mcclay | Extensional fault systems in sedimentary basins: a review of analogue model studies[END_REF][START_REF] Van Wijk | Role of weak zone orientation in continental lithosphere extension[END_REF][START_REF] Brune | Evolution of stress and fault patterns in oblique rift systems: 3-D numerical lithospheric-scale experiments from rift to breakup[END_REF][START_REF] Heine | Oblique rifting of the Equatorial Atlantic: Why there is no Saharan Atlantic Ocean[END_REF], whilst the second class focus on the strain localisation within the transfer zone between two en-echellon grabens by using two offset weak zones [START_REF] Vendeville | Physical models of normal-fault relays between variably offset grabens[END_REF][START_REF] Gerya | Dynamical instability produces transform faults at mid-ocean ridges[END_REF][START_REF] Allken | Three-dimensional numerical modeling of upper crustal extensional systems[END_REF][START_REF] Allken | Factors controlling the mode of rift interaction in brittle-ductile coupled systems: A 3D numerical study[END_REF][START_REF] Allken | 3D numerical modelling of graben interaction and linkage: a case study of the Canyonlands grabens, Utah[END_REF][START_REF] Liao | From continental rifting to seafloor spreading: Insight from 3d thermomechanical modeling[END_REF].

Performing dynamic numerical models of large offset (100's km), magma poor, oblique passive margins such as Rio Muni in West Africa [START_REF] Turner | Structure and evolution of an obliquely sheared continental margin: Rio Muni, West Africa[END_REF], Alula Fartak in the Gulf of Aden [START_REF] Leroy | From rifting to oceanic spreading in the Gulf of Aden: a synthesis[END_REF], or the Ghana transform margin [START_REF] Basile | The Ivory Coast-Ghana transform margin: a marginal ridge structure deduced from seismic data[END_REF], requires three-dimensional experiments which include: full thermo-mechanical coupling, large deformation spanning at least 30 Myr of evolution and sufficiently high spatial resolution to capture accurately the deformation across a 300 km × 200 km transfer zone.

Due to technical limitations, none of the previous 3D numerical studies could simultaneously satisfy all of these requirements. Time scales of evolution rarely exceeded 5 Myr [START_REF] Choi | Thermomechanics of mid-ocean ridge segmentation[END_REF][START_REF] Gerya | Dynamical instability produces transform faults at mid-ocean ridges[END_REF][START_REF] Gerya | Three-dimensional thermomechanical modeling of oceanic spreading initiation and evolution[END_REF][START_REF] Allken | Three-dimensional numerical modeling of upper crustal extensional systems[END_REF][START_REF] Allken | Factors controlling the mode of rift interaction in brittle-ductile coupled systems: A 3D numerical study[END_REF][START_REF] Allken | 3D numerical modelling of graben interaction and linkage: a case study of the Canyonlands grabens, Utah[END_REF], and the offset across the diverging segments reached, at maximum 50 km, within a model domain (in map view) of 100 km × 100 km [START_REF] Choi | Thermomechanics of mid-ocean ridge segmentation[END_REF][START_REF] Gerya | Dynamical instability produces transform faults at mid-ocean ridges[END_REF][START_REF] Gerya | Three-dimensional thermomechanical modeling of oceanic spreading initiation and evolution[END_REF][START_REF] Allken | Factors controlling the mode of rift interaction in brittle-ductile coupled systems: A 3D numerical study[END_REF][START_REF] Liao | From continental rifting to seafloor spreading: Insight from 3d thermomechanical modeling[END_REF]. [START_REF] Heine | Oblique rifting of the Equatorial Atlantic: Why there is no Saharan Atlantic Ocean[END_REF] modelled oblique rifting at the scale of the equatorial Atlantic, however their models did not have sufficient spatial resolution to resolve fault patterns in the crust. In this work, we utilise a newly developed, state-of-the-art massively parallel computational framework for studying 3D lithospheric deformation to surmount all of the aforementioned technical challenges required to simulate models of oblique passive margins.

Method

Model set-up

The original kinematic model [START_REF] Mascle | Evidence for transform margin evolution from the Ivrory Coast-Ghana continental margin[END_REF] supports that transform margins are expected form at bridging rift segments, therefore we adopted the model design which includes two offset weak zones. We consider a lithosphere scale model domain (Fig. 2A) representing 600×150×1200 km 3 . Apart from the dimensions of the physical domain, the model configuration is similar to other models designed to simulate strain localisation of oblique transfer zones [START_REF] Vendeville | Physical models of normal-fault relays between variably offset grabens[END_REF][START_REF] Gerya | Dynamical instability produces transform faults at mid-ocean ridges[END_REF][START_REF] Allken | Three-dimensional numerical modeling of upper crustal extensional systems[END_REF][START_REF] Allken | 3D numerical modelling of graben interaction and linkage: a case study of the Canyonlands grabens, Utah[END_REF][START_REF] Allken | Factors controlling the mode of rift interaction in brittle-ductile coupled systems: A 3D numerical study[END_REF][START_REF] Liao | From continental rifting to seafloor spreading: Insight from 3d thermomechanical modeling[END_REF] and incorporates a minimum amount of complexity.

All models consist of an initially horizontally layered hypothetical lithosphere which includes mantle, lower crust and upper crust (Fig. 2A). The Moho is originally located at 40 km depth and the boundary between lower and upper crust is located at 20 km depth. To simulate extensional systems, we impose a normal velocity with magnitude 1 cm/yr, along each face whose normal points in the x direction. This magnitude of velocity is quite standard for modelling continental extension and was previously used by [START_REF] Watremez | Buoyancy and localizing properties of continental mantle lithosphere: Insights from thermomechanical models of the eastern gulf of aden[END_REF] to model the opening of the Gulf of Aden, a young example of oblique continental extension. Material leaving the domain across these faces is compensated by a constant infilling normal velocity (2.5 mm/yr) at the base of the models. Both the inflow and outflow boundaries prescribed that the shear stress is zero. Zero normal velocity and shear stress (free slip) are prescribed on the domain faces with normal pointing in the z-direction. The top boundary of the domain is defined to be a free surface (i.e. zero stress).

The initial crustal density structure is the same in all the models and corresponds to a density of 2700 kg/m 3 in the upper crust and 2800 kg/m 3 in the lower crust. The initial geotherm we used corresponds to a half-space cooling model after 300 Myr (see Fig. 2C). All three regions employ an approximate Arrhenius type flow law (see Appendix A, Eq. (A.7)). The reference viscosity of the lower crust was varied in order to test whether lithosphere previously involved in orogenic processes reacts differently from a regular (REG) lithosphere. The Post Orogenic (PO) lithosphere assumes that the lower crust constitutes sedimentary material that has previously been buried underneath a regular crust [START_REF] Huet | Post-orogenic extension and metamorphic core complexes in a heterogeneous crust: the role of crustal layering inherited from collision. application to the cyclades (aegean domain)[END_REF] and therefore possesses a viscosity lower than that in the regular lower crust. This produces an inverted rheological stratification of the crust (Fig. 2D). The viscosity profiles obtained for the REG and PO lithosphere models are shown in Fig. 2D In order to avoid unrealistically high stresses and topography, material strength is limited by frictional brittle failure with an additional plastic yield strength set to 300 MPa [START_REF] Watremez | Buoyancy and localizing properties of continental mantle lithosphere: Insights from thermomechanical models of the eastern gulf of aden[END_REF]. Localisation is enhanced by frictional softening with strain (Fig. 2B). Softening is enhanced in the mantle (see Table 1) to approximate the effect of serpentinisation on the brittle behaviour.

Initially, each material point in the domain is assigned a random plastic strain value between 0 and 0.03. In the domain, we impose two weak zones, offset by a distance W , each defining a 100 × 120 × 100 km 3 volume (depicted in Fig. 2A). Within each weak zone, the amplitude of the initial random plastic strain is ten times larger (i.e. varies between 0 and 0.3) compared with the initial damage defined throughout the rest of the domain. All parameters are listed in the Table 1.

We will first present the results of 8 simulations, examining how: (i)

ρ 0 density kg m -3 2.7/2.8 × 10 3 2.7/2.8 × 10 3 3.3 × 10 3 α thermal expansion • C -1 2 × 10 -5 2 × 10 -5 2 × 10 -5 β compressibility Pa -1 0 0 3 × 10 -12 κ thermal diffusivity m 2 s -1 10 -6 10 -6 10 -6
Table 1: Physical properties of the different materials. Regular (REG) crust is more viscous than Post Orogenic crust (PO). The relationship between parameters listed and the equations solved by the thermo-mechanical numerical model are elaborated upon in Appendix A.

between the weak zones (see Fig. 2A); and (ii) the initial mechanical coupling between the crust and lithospheric mantle (REG versus PO in Fig. 2B) influences the linkage between rift segments.

Following that, we concentrate on the dynamics of divergent segments which link at moderate to large offset values.

Modelling

This study uses pTatin3D [START_REF] May | pTatin3D: high-performance methods for longterm lithospheric dynamics[END_REF][START_REF] May | A scalable, matrix-free multigrid preconditioner for finite element discretizations of heterogeneous Stokes flow[END_REF] -a highly scalable, massively parallel implementation of the finite element method which employs an Arbitrary Lagrangian Eulerian discretisation, together with the material point method. The equations governing viscous flow (see Appendix A) together with a free surface boundary condition are accurately resolved through the use of high order stable elements (Q 2 -P 1 ). Material properties and history variables are assigned to material points contained within the computational domain. Material points are allowed to exit (or enter) the model domain according to boundary conditions. Details related to the non-linear solver are provided in Appendix B.

Owing to this new generation code, it has become possible to discretise a 600 × 150 × 1200 km 3 lithosphere domain using 128 × 32 × 256 elements, leading to spatial resolution of 5 km. This discrete system of algebraic equations defines a highly non-linear problem (due to the rheology used) with over 29 × 10 6 degrees of freedom. Each simulation integrates deformation over a time span of 25 to 35 Myr, which corresponds to ∼ 2000 time steps on average.

Post-processing

In this study, we are interested in the obliquity of tectonic structure with the stretching direction. Hence, we have chosen to represent the simulation results primarily in map view. The results are presented in terms of surface elevation, Moho elevation and active fault trends map.

As the models have a free surface, the topography is simply the elevation of the nodes located at the surface of the model relative to the upper right corner of each map.

The different lithologies are stored on markers in the code and used to compute effective viscosity and density (see Appendix A). In order to reduce the amount of data per stored time step to approximately 500 MBytes (250 GBytes per simulations), we only store the density, viscosity and plastic strain averaged over each element.

In order to extract the Moho, we examine all cells with an average density larger than 3200 kg/m 3 and extract the cells average y coordinate. For a given column in the x -z plane, the Moho is defined via the maximum of these y coordinates (refer to Fig. 2A for coordinate system used). As the models include neither melt nor, true oceanic crust, we define the ocean-continent boundary by attributing a continental affinity to all surface cells in which densities are lower than 3200 kg/m 3 .

Faults in these numerical models are shear bands with a finite thickness, typically on the order of one to two elements. Here we split the components of shear that corresponds to vertical direction, i.e. the dip slip component ( εxy and εyz ) from the component of shear in the horizontal direction, i.e. the transform component ( εxz ) and we only visualise the zones where these components are larger than the background strain rate (5 × 10 -16 s -1 ) as these regions correspond to locations where strain rate has localised.

Results and discussion

Rift linkage

To first order, models with a weak lower crust (PO) display distributed normal faulting with faults trending orthogonal to the direction of stretching while this distributed small wavelength deformation is absent in the strong lower crust models (REG). This contrasting behaviour is predicted by previous 2D numerical models [START_REF] Huismans | Roles of lithospheric strain softening and heterogeneity in determining the geometry of rifts and continental margins[END_REF]. More surprising is that the propagation of divergent rift segments occurs faster in the decoupled lithosphere (Fig. 3, see PO models at 10 Myr), compared to the coupled lithosphere model (Fig. 3, see REG models at 10 Myr).

The final outcome of the models considered fall into one of four cases outlined by the different colours of the small insets panels in Fig. 3. At small (100 km) and extremely large initial offset (400 km), the rheology of the lower crust does not greatly influence the results within the first 25 Myr of evolution. With an offset of 100 km, oblique margins form rapidly, followed by oblique spreading. For offsets of 400 km, propagating rifts do not interact mechanically.

Obliquity is quantified as the angle between the strike and the stretching direction, a low angle therefore corresponds to a high obliquity. For moderate to large initial offsets, the rheology controls the transition from an oblique margin (45 o ), to an extremely oblique margin (< 30 o ), and a failed oblique rift (Fig. 3). For an offset of 200 km, the obliquity of the transfer zone is higher when using a PO lithosphere, compared to when using a REG lithosphere. The maximum offset between two linking weak zones increases with the strength of the lithosphere (compare the models with 300 km offset in Fig. 3).

Examining the snapshots of the models at 25 Myr (Fig. 3), it is clear that, the larger the offset, the longer it takes to localise the deformation on the oblique segment. As a result, the delay between the break-up on the propagating divergent rift segment, measured as the first break-up time in the pre-existing weak zones and the final oblique break-up, increases from 6 Myr for 100 km offset to 25 Myr for a 300 km initial offset (see supplementary movies of the experiments).

During this period, mid-ocean ridge propagation is stalled, but distributed continental extension accommodates the divergence at the tip of the ridge.

The models in which early faults (those active at 10 Myr in Fig. 3) strike perpendicular to the stretching direction result in a larger final obliquity than models which develop early structures that are oblique to the stretching direction. In all the models, the orientation and existence of early structures (those active at 10 Myr in Fig. 3) is controlled by the rheology of the lower crust, and, to a lesser extent, by the initial offset. This led us to speculate that, in nature, pre-existing structures reactivated during the early stages of rifting [START_REF] Bellahsen | Fault reactivation and rift localization: Northeastern Gulf of Aden margin[END_REF] Supplementary movies are provided for experiments REG-100 km, REG-300 km and PO-100 km, PO-300 km using the same colour scales as this figure .  influence the final obliquity of the margin.

Initial weak zones spacing and linkage

Previous studies with no thermal coupling have reported that linkage does not occur when notch spacing exceeds the thickness of the brittle layer [START_REF] Vendeville | Physical models of normal-fault relays between variably offset grabens[END_REF][START_REF] Allken | Factors controlling the mode of rift interaction in brittle-ductile coupled systems: A 3D numerical study[END_REF]. At lithospheric scale, this limit would correspond to W ≤ 100 km. In our models which account for thermal coupling, the linkage between divergent segments is obtained for much larger offsets, upto 3-4 times the thickness of the brittle layer.

In analogue models, the interactions between grabens has been found to be less efficient when the weak zones overlap, but no significant differences were observed between models with nonoverlapping weak zones and models with underlapping weak zones (Le Calvez and [START_REF] Le Calvez | Experimental designs to model along-strike fault interaction. Analogue modelling of large-scale tectonic processes[END_REF]. Using dynamic numerical models to study these geometrical effects in more details, [START_REF] Allken | 3D numerical modelling of graben interaction and linkage: a case study of the Canyonlands grabens, Utah[END_REF] showed that a larger amount of underlap favours linkage because more diffuse strain can occur in the transfer zone before the imposed rift segment propagates. In a previous study, [START_REF] Allken | Factors controlling the mode of rift interaction in brittle-ductile coupled systems: A 3D numerical study[END_REF] showed that linkage occurs with a larger offset if a weak and thick ductile layer is present at the base of the crust. With our lithospheric scale models, we find that it is easier to link lithospheric scale rift segment when the lower crust is resistant. This is somewhat contradictory to [START_REF] Allken | Factors controlling the mode of rift interaction in brittle-ductile coupled systems: A 3D numerical study[END_REF] conclusions. Yet, we note that their models neglected the strain localisation in the lithospheric mantle and were devoid of any thermo-mechanical coupling.

The two studies can be reconciled by recognising that at the scale of our models, the weak lower crust of [START_REF] Allken | Factors controlling the mode of rift interaction in brittle-ductile coupled systems: A 3D numerical study[END_REF] corresponds to our weak asthenosphere. The final strain rate and linkage pattern of our models actually always occurred within the low viscosity lower crust class defined in [START_REF] Allken | Factors controlling the mode of rift interaction in brittle-ductile coupled systems: A 3D numerical study[END_REF], independently of the viscosity of the lower crust. We now focus on the dynamics of models in which linkage occur at large off-set in our parametric study.

Dynamics of large offset break-up

Before oblique localisation occurs (Fig. 4A), the two divergent segments are always separated by en-echellon deep intra-continental basins as clearly indicated by the topography. Depending on the dynamics of break-up, oblique or extremely oblique, their thermal and tectonic histories are different. The vertical loads associated with the presence of hot asthenospheric mantle and lateral variation in crustal thickness result in the flexural bending of plates. Where crustal thinning does not coincide with lithospheric mantle thinning, a topographic ridge forms at the moment of break-up. For the model which results in oblique break-up (REG-200 km) in Fig. 4, the latest stage of oblique localisation occurs within the center of the early oblique basin. Looking at the accumulated plastic strain as a proxy for major fault orientation, one observes that the oblique break-up does not cut across any pre-existing structures with a different orientation. The rotation of the active structure with time (Fig. 4A,B and supplementary movie shows that the early structures rotated continuously between the two diverging segments to reach their final orientations.

As a result, the remnants of the early deep basins forms an oblique margin which is separated from the divergent segment by topographic ridges which are orthogonal to the direction of extension.

In models which result in extremely oblique break-up (PO-200 km and REG-300 km) in Fig. 4, strike slip localisation occurs along a strike slip structure, which cross-cut the initial en-echelon basins (Fig. 4B). The supplementary movies clearly indicate, this structure did not form by continuous rotation of the earlier structures. At the onset of localization in the mantle, no oblique structures are visible at the surface (Fig. 4A), yet small portion of the early basins deforms with strike slip kinematics. These local spots of green shades outline the direction of the still deep seated mantle necking structure which will reach the surface at the stage depicted on Fig. 4B.

The accumulated plastic strain (Fig. 4C) show that extremely oblique break-up segments cut across the early rift structures. For the REG-300 km model, the shade of white in the plastic strain highlights the orientation of the structures during break-up and indicates that break up occurs on pure transform segments separated by divergent segments. As crustal thinning, inherited from the earlier phase, is not parallel to lithospheric thinning, a topographic ridge forms parallel to the oblique margins.

The delay between the divergent margin break-up and the onset of oblique break-up causes the formation of a large oceanic domain prior to complete break-up. They are displayed as transparent on Fig. 4C. Their size increases with increasing initial offset and decreasing lower crustal strength.

All the models show that these oceanic domains are unevenly distributed across the midoceanic ridge and that the asthenospheric flow feeding these ridges is also asymmetric (Fig. 4C).

This suggests that early spreading is not purely divergent in the model. The magnetic anomalies from the Gulf of Aden (Fig. 5A) and West Africa (Fig. 5B) show the presence of oblique stripes, and ductile shear indicators such as S-C structures, in good agreement with the prediction of our models.

For strong lower crustal rheology (REG-models, Fig. 4C the asymmetric spreading occur in the oceanic domain. For weak lower crustal rheology (PO-model, Fig. 4C), the lower crust is remobilized and spreads to accommodate the asymmetry. This suggest that the rheology of the lower crust controls whether re-mobilized crustal material [START_REF] Ranero | Sequential faulting explains the asymmetry and extension discrepancy of conjugate margins[END_REF][START_REF] Unternehr | Hyper-extended crust in the South Atlantic: in search of a model[END_REF], or exhumed mantle material [START_REF] Pérez-Gussinyé | Serpentinization and magmatism during extension at non-volcanic margins: the effect of initial lithospheric structure[END_REF], accommodates the latest stage of deformation.

The Moho is very shallow at the tip of the oceanic domain (Fig. 6A), highlighting a large amount of continental extension. The 3D models show that large initial offset causes an asymmetry in extension and the formation of hyper-extended margins. However, these hyper-extended basins only mark the tip of en-echellon diverging rift segments which have stalled during the long-lasting strain localisation along the oblique segments. They do not indicate the direction of propagation of oceanisation as suggested in the Gulf of Aden [START_REF] Manighetti | Propagation rifling along the Arabia-Somalia plate boundary: The Gulfs of Aden and Tadjoura[END_REF]. In order to perform the reconstruction correctly, the initial conditions, the pre-rift and the early normal faults located on the transform segment must be used to restore the onset of break-up stage which is displayed in Fig. 6A.

A new model for transform margins

This should leave behind large remnants of divergent oceanic domain, exhumed lower crustal domain and hyper-extended margins, which can be overlapped based on the magnetic anomaly of the purely divergent segments. From the results of our numerical models, a good proxy for the timing of the final break-up is the formation of the marginal ridge which cross-cut the early basins. The flexural uplift should be marked by an unconformity between the syn-rift and post-rift sediments in the deep basins located at its back. The deep basins formed in the early stage of rifting present a good potential for hydrocarbons because they were tectonically isolated from the well oxygenated water that fills the basins forming along the pre-imposed divergent segments. However, this potential also depends on the dynamics of the break-up, i.e. oblique or extremely oblique. In the first case, they rotate to align with the oblique mid-oceanic ridge. In the second case, some of these basins are preserved in the intrado of the topographic ridge, proctecting them from heat and promoting very early migration. As a result, the concave transform divergent margin (zone 3 in Fig. 1 and6A) is constituted by deep, potentially hydrocarbon rich, basins formed in the early stages of rifting which are absent from the convex divergent margins.

Conclusions

Using modern, state-of-the-art, high performance computing tools has allowed us to simulate dynamic, 3D lithospheric scale oblique rifting scenarios with the necessary spatial resolution. Our results show that simple set-up with cylindrical and constant boundary conditions produce a level of complexity in term of asymmetry and poly-phasing of deformation that could only be obtained with more complex model configurations or boundary conditions in 2D [START_REF] Brune | Rift migration explains continental margin asymmetry and crustal hyper-extension[END_REF]Huismans andBeaumont, 2011, 2007).

Our models predict that the break-up of extremely oblique segments occurs 10 to 25 Myr after the break-up of the divergent segments. During this period, stalling normal rift propagation leads to the formation of deep anoxic basins, along with micro continental ribbons which form patterns similar to those observed in northern Atlantic profiles (Péron-Pinvidic and Manatschal, 2010) and in Angola-like hyper-extended margins [START_REF] Unternehr | Hyper-extended crust in the South Atlantic: in search of a model[END_REF].

Natural fracture zones and transform margins are not defined by line segments, but rather a 100 km diffuse zone of deformation (Fig. 5) where significant asymmetry can be accommodated by exhumation of lower crustal material, and/or asymmetry of the spreading. These deformation zones record the temporal history of several phases of deformation occurring prior to oblique break-up.

Consequently, attempting to reconstruct plate motions with rigid plate transform margin model is impossible. Our new high resolution 3D dynamic models can help predict where, and when, significant amounts of stretching and wrenching are expected to occur. Such results will help to further improve the reliability and accuracy of future paleogeographic reconstructions.

average viscosity over each Q 2 element using the harmonic average of the effective viscosity associated with all material points contained in a given element. The element density is computed in a similar manner, but use the arithmetic average of marker densities.

We apply the Flexible-GMRES [START_REF] Saad | A flexible inner-outer preconditioned GMRES algorithm[END_REF] Krylov method to solve Eq. (A.1). The system is right preconditioned using an upper block triangular preconditioner [START_REF] Elman | Finite Elements and Fast Iterative Solvers[END_REF]. The viscous block is preconditioned using a hybrid geometric multigrid implementation [START_REF] May | A scalable, matrix-free multigrid preconditioner for finite element discretizations of heterogeneous Stokes flow[END_REF].

The temperature T in Eq. (A.6) is discretised using Q 1 finite elements which overlap the Q 2 velocity space. The advection-diffusion equation is stabilized via SUPG [START_REF] Brooks | Streamline upwind/Petrov-Galerkin formulations for convection dominated flows with particular emphasis on the incompressible Navier-Stokes equations[END_REF]. The Stokes problem and energy equation are solved in a de-coupled manner, thus the temperature dependence in the rheology (Eq. (A.7)) and forcing function (Eq. (A.5)) does not introduce an additional non-linearity.

Support for all parallel linear algebra, in the form of matrices, vectors, preconditioners, Krylov methods and nonlinear solvers is provided by PETSc (Balay et al., 2013b,a). Parallelism is achieved by spatially decomposing the structured finite element (FE) mesh containing M × N × P elements, into structured sub-domains containing m × n × p elements. The underlying mesh data structure we uses is defined by DMDA object provided by PETSc.

The simulation results were performed using 512 cores an SGI ICE 8200 with compute nodes consisting of 2×6 Intel Xeon (Series 5600) cores running at 2.8 GHz. Each model required approximately 2000 time steps summing to approximately 2 days of computation time for a mesh resolution of 256 × 32 × 128 Q 2 elements. The multigrid preconditioner was configured with three levels, from the coarsest to the finest level: 32 × 16 × 16, 128 × 32 × 64, 256 × 32 × 128. We used conjugate gradient and incomplete LU as the coarse grid solver. The coarse grid solver was terminated after 25 iterations, or if the initial residual was reduced by a factor of 10 4 . The non linear solver uses picard iterations with rtol 10 -2 or atol 10 -4 . pTatin3D [START_REF] May | pTatin3D: high-performance methods for longterm lithospheric dynamics[END_REF][START_REF] May | A scalable, matrix-free multigrid preconditioner for finite element discretizations of heterogeneous Stokes flow[END_REF] is an open source code copyrighted under GNU General Public License v.3 and may be accessed 1 at https://bitbucket.org/jedbrown/ptatin3d. pTatin3D heavily leverages functionality from PETSc, which can be freely downloaded at http://www.mcs.anl.gov/petsc.

Input files are available upon request from the corresponding author.

Figure 1 :

 1 Figure 1: Illustration of the kinematic model of transform margin formation. Modified from Francheteau and Le Pichon (1972) and Basile et al. (1993). Light grey represents normal thickness continental crust, dark grey regions represent the thinned continental crust and light blue regions indicate oceanic crust. A) Post rift stage illustrating the location of concave and convex transform divergent margins; B) Latest stage of transform margin activity showing that the mid-ocean ridge interacts thermally with the concave transform margin, long after break-up; C) Formation of the transform margin.

Figure 2 :

 2 Figure 2: Model definition. A) 3D domain with extensional boundary conditions and the outline of pre-existing weak zones offset by a distance W ; B) Rheological softening law in the crust; C) Initial geotherm; D) Viscosity profile for the coupled (REG) and decoupled (PO) lithosphere obtained using the initial geotherm.

Figure 3 :

 3 Figure 3: Results of the parametric study. Map view of the topography of all the models after 10 Myr (large insets) and 25 Myr (small insets) with major active faults outlined as shade of red (dip slip kinematics)and green (strike slip kinematic). At 10 Myr, distributed small wave length faults striking perpendicular to the direction of stretching forms in the post orogenic models (Left) while large oblique normal faults forms in the regular crust models (Right). At 25 Myr, coloured boxes indicate models with oblique margins (45 o , red), with extremely oblique margins (< 30 o , black), abandoned transfer zone (dashed) and in which segments did not link at all (blue).

Figure 4 :

 4 Figure 4: Map view of the topography with major active faults outlined via a shade of red (dip-slip kinematics) and green (strike slip kinematic). Timing corresponds to A) onset of strike slip strain localisation in the transfer zone and B) incipient break-up. C) Accumulated plastic strain (main faults) in the upper crust. The oceanic lithosphere has been removed to outline the ocean-continent boundary and to highlight the asymmetry in asthenospheric flow line feeding the mid-oceanic ridges. Exhumed lower crustal material is shown in brown.

Figure 5 :

 5 Figure 5: textbfMap view of the gradient of magnetic field. A) In the Gulf of Aden at the Corner of Alula Fartak fracture zone (AfFZ)[START_REF] Leroy | Contrasted styles of rifting in the eastern Gulf of Aden: A combined wide-angle, multichannel seismic, and heat flow survey[END_REF]. B) Along the West African margin[START_REF] Maus | Emag2: A 2-arc min resolution earth magnetic anomaly grid compiled from satellite, airborne, and marine magnetic measurements[END_REF] nearby the Saint Paul Fracture Zone (sPFZ), the Romanche Fracture Zone (RFZ) and the Chain Fracture Zone (CFZ). In both cases that the magnetic anomalies (dashed cyan line) are not perpendicular to the direction of spreading (white arrow), nor to the fracture zone (yellow line) close to the transform margin.

Fig.

  Fig. 6A highlights the coastlines (cyan lines) from the conjugate convex (3) and concave (2) transform divergent margins produced by model REG-300 km at the end of continental break-up.The coastlines are of very different length. The concave margin is only a few tens of kilometers long, whilst its convex conjugate can be as long as 200 km. Fig.6Brepresents a hypothetical passive margin that would result from model REG-300 km. Combining the ∼ 250 km of the oblique segment (yellow line) and the 200 km segment of the convex margin, the coastline which trends extremely oblique to the direction of stretching, reaches a length of ∼ 450 km. This is comparable in length to the transform margins in the equatorial Atlantic (Fig.5B).Due to the anti-symmetric extension occurring on the opposite segment, a coastline of similar length exists on the right block. However, trying to reassemble the left and right blocks using the rigid transform margin model (Fig.6C) shifts the location of the early deep basins, while our models indicate these features were aligned during the 15 to 25 Myr of extension that preceded oblique break-up (Figs.4 and 6A).

Figure 6

 6 Figure 6: A) Map view of the Moho depth for the REG model with a 300 km offset with the main structures; 1, 2 and 3 indicate respectively the divergent, convex and concave transform divergent margins; B) Representation of the anticipated passive margin, long time after continental rifting with annotations corresponding to the original model represented on Fig. 1A; C) Schematic reconstruction using the rigid plate kinematic model and showing the misfit of the early rift structures if one does not first reconstruct the break-up stage displayed in A)

  the initial offset W

	quantity	name	unit	REG crust	PO crust	mantle
	η 0	reference viscosity	Pa s	10 27	10 26	10 30
	θ	char. temperature	• C -1	2 × 10 -2	2 × 10 -2 1.8 × 10 -2
	φ 0	friction angle	•	30	30	30
	C 0	cohesion	Pa	2 × 10 7	2 × 10 7	2 × 10 7
	τ mms	maximum strength	Pa	2 × 10 8	2 × 10 8	3 × 10 8
	φ ∞	min. friction angle	•	6	6	6
	ε p ∞	softening char. strain	-	0.3	0.3	0.15171

Acknowledgements

Maps were realised with GMT, post processing with Paraview. The paper benefited of discussions with C. Basile, D. Aslanian and P. De Clarens through the Action Marge Program. D. A. May and L. Le Pourhiet acknowledge the Swiss National Supercomputing Centre (CSCS) for compute time on Piz Dora, obtained under the production proposal grant 20131011-2135237.

Appendix A. Governing Equations

The mechanical component of pTatin3D solves the conservation of momentum for incompressible, highly non-linear viscous fluids. The conservation of momentum for such a fluid is given by 2η εij (u) ,j -p ,i = f i , (A.1) where u, p are the fluid velocity and pressure, η is the shear viscosity, f i is the body force and the strain rate operator εij is given by εij (u) := 1 2 (u i,j + u j,i ) .

(A.

2)

The total stress σ ij is given by

Simplified mass conservation is achieved through assuming incompressibility

The forcing term in the momentum equation reflects changes in buoyancy associated with thermal contraction/expansion (Boussinesq approximation), or compositional variations between different rock types following

where ρ 0 is the compositional density of the rock as measured at the reference temperature T 0 and pressure p 0 , ĝ is the gravity vector, α, β are the coefficients of thermal expansion and adiabatic compressibility respectively and T is the temperature. The evolution of temperature is given by the following conservation equation

where κ = k/(ρC p ) is the thermal diffusivity.

Thermally activated creep mechanisms defining the viscous strength of rock, η are approximated by the Frank-Kamenetskii flow rule

where the parameters η 0 , θ account for the local chemical composition of the rock. These coefficients are also carried by the markers. Pressure and strain rate dependence are introduced by limiting the effective viscosity so that stresses in the model do not exceed the Drucker-Prager yield stress

which is limited a maximum Mises stress, τ mms . The cohesion C 0 is regarded to be constant, however the internal friction angle φ is assumed to decrease as a function of the accumulated plastic strain p according to

Eq. (A.9) defines an empirical softening relation which reduces the friction angle linearly with accumulated plastic strain. Here φ 0 defines the initial friction angle. p ∞ represents the measure of plastic strain after which complete softening is achieved and internal friction angle reaches φ ∞ .

Plastic strain represents an integrated, tensorial invariant measure of the deformation which has occurred due to plastic yielding. Thus, the quantity p can be regarded as a simplified measure of material damage. The evolution of plastic strain is given by; (piece-wise linear) basis which is discontinuous across element boundaries.

The material-point-method is used to discretise different material regions. In the materialpoint-method, Lagrangian particles are introduced within each finite element. The particles are advected with the velocity computed from the solution of the discrete Stokes problem. Each particle stores an indexing used to identify to which material region it belongs to. This index is used to determine the value of the rheological constants and coefficients related to Boussinesq.

The particles are also used to store the plastic strain p .

At each material point we evaluate the viscous flow-law and the Drucker-Prager yield stress.

The effective viscosity (see Eq. (A.11)) is also evaluated at each material point. We define an


Zixing Zhang
email: zixing.zhang@uni-passau.de

Fabien Ringeval

Jing Han

Jun Deng

Erik Marchi

Björn Schuller

Facing Realism in Spontaneous Emotion Recognition from Speech: Feature Enhancement by Autoencoder with LSTM Neural Networks

Keywords: emotion recognition, spontaneous speech, additive and convolutional noises, feature enhancement, autoencoder, LSTM Neural Networks

During the last decade, speech emotion recognition technology has matured well enough to be used in some real-life scenarios. However, these scenarios require an almost silent environment to not compromise the performance of the system. Emotion recognition technology from speech thus needs to evolve and face more challenging conditions, such as environmental additive and convolutional noises, in order to broaden its applicability to real-life conditions. This contribution evaluates the impact of a front-end feature enhancement method based on an autoencoder with long short-term memory neural networks, for robust emotion recognition from speech. Support Vector Regression is then used as a back-end for time-and value-continuous emotion prediction from enhanced features. We perform extensive evaluations on both non-stationary additive noise and convolutional noise, on a database of spontaneous and natural emotions. Results show that the proposed method significantly outperforms a system trained on raw features, for both arousal and valence dimensions, while having almost no degradation when applied to clean speech.

Introduction

Technology for automatic emotion recognition from speech (ERS) has gained increasing commercial attention in the last decade. Rapid progress of this technology has indeed enabled application of ERS in various domains, such as, health care [START_REF] Tacconi | Activity and emotion recognition to support early diagnosis of psychiatric diseases[END_REF], education [START_REF] Calvo | Frontiers of affect-aware learning technologies[END_REF], serious games [START_REF] Schuller | Recent developments and results of asc-inclusion: An integrated internet-based environment for social inclusion of children with autism spectrum conditions[END_REF], robotics [START_REF] Marchi | Voice-enabled assistive robots for handling autism spectrum conditions: An examination of the role of prosody[END_REF], and call-centers [START_REF] Petrushin | Emotion in speech: Recognition and application to call centers[END_REF]. However, while good performance has been reported in research papers under laboratory conditions [START_REF] Schuller | Acoustic emotion recognition: A benchmark comparison of performances[END_REF], or with systems tailored towards specific databases [START_REF] Ringeval | Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data[END_REF], real-life applications of ERS still remain an open challenge. Indeed, various factors make this task highly challenging, which can be grouped into three main categories: (i) the contextual dependencies of the meaning and significance of affective expressions across different speakers, languages and cultures [START_REF] Sauter | Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations[END_REF], (ii) the presence of varying and degraded acoustic conditions caused by reverberation, background noise, and acoustic properties of the recording devices used, and (iii) the necessity to use distributed systems in a client-server architecture, which introduce some latency and distortion in the data [START_REF] Zhang | Distributing recognition in computational paralinguistics[END_REF].

Stationary, non-stationary, and convolutional noise severely degrade performance of systems, and affect consequently the user experience in real-life conditions [START_REF] Schuller | Emotion recognition in the noise applying large acoustic feature sets[END_REF][START_REF] Tawari | Speech emotion analysis in noisy realworld environment[END_REF][START_REF] Huang | Speech emotion recognition under white noise[END_REF]. Therefore, many studies have been performed for speech and acoustic fea-ture enhancement (FE), especially for automatic speech recognition (ASR). Recurrent Neural Networks (RNN) are widely used in this field to enhance corrupted features, which is an application of the de-noising autoencoder [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF] principle: neural networks are trained to map noisy features to clean features. This method has recently also been exploited for speech enhancement in the time domain [START_REF] Lu | Speech enhancement based on deep denoising autoencoder[END_REF][START_REF] Xia | Speech enhancement with weighted denoising auto-encoder[END_REF]. RNN have been also studied for blind non-linear source separation, with the aim to enhance the acoustic features by separating noise and speech sources [START_REF] Tan | Nonlinear blind source separation using a radial basis function network[END_REF][START_REF] Zhang | Channel mapping using bidirectional long short-term memory for dereverberation in hand-free voice controlled devices[END_REF]. In the context of speech enhancement, the authors in [START_REF] Lu | Speech enhancement based on deep denoising autoencoder[END_REF] use deep neural networks to map noisy to clean Mel features, but the network output is synthesised directly into a time domain signal, instead of constructing a filter based on speech and noise magnitudes. A combination of unsupervised noise estimation and Deep Neural Network (DNN) based speech power spectrum estimation is used in [START_REF] Xia | Speech enhancement with weighted denoising auto-encoder[END_REF] to construct a Wiener filter. Supervised training of deep neural networks was performed to predict the ideal ratio mask in an uncertainty decoding framework for ASR [START_REF] Narayanan | Ideal ratio mask estimation using deep neural networks for robust speech recognition[END_REF].

Studies on noise robustness for ERS are much more sparse, despite being necessary for real-life applications of this technology. To the best of our knowledge, only a few studies have addressed this issue so far. Large acoustic feature sets were investigated in [START_REF] Schuller | Emotion recognition in the noise applying large acoustic feature sets[END_REF]. Adaptive noise cancellation was proposed as a front end in [START_REF] Tawari | Speech emotion analysis in noisy realworld environment[END_REF]. Speech enhancement based on spectral subtraction and masking properties was studied in [START_REF] Huang | Speech emotion recognition under white noise[END_REF]. Wavelet decomposition [START_REF] Vasquez-Correa | Emotion recognition from speech under environmental noise conditions using wavelet decomposition[END_REF] and feature selection techniques [START_REF] Eyben | Affect recognition in real-life acoustic conditions -a new perspective on feature selection[END_REF] have also been proposed. Additionally, supervised Nonnegative Matrix Factorization (NMF) was investigated for the robustness of emotion recognition engines [START_REF] Weninger | Recognition of nonprototypical emotions in reverberated and noisy speech by nonnegative matrix factorization[END_REF].

One may note that most existing work on noise robustness for ERS has been performed on acted emotions, which are rarely observable in real-life. Furthermore, only a few of those studies have analysed the impact of reverberated noise, which is known to impact severely the performance of ASR systems [START_REF] Weninger | Recognition of nonprototypical emotions in reverberated and noisy speech by nonnegative matrix factorization[END_REF][START_REF] Benesty | Speech Enhancement[END_REF]. In this light, this present contribution studies the impact of non-stationary additive noise and convolutional noise on the automatic recognition of spontaneous emotions from speech. We propose the use of a FE method based on a memory-enhanced recurrent Denoising Autoencoder (rDA) as a front end, and show that this method can significantly improve the performance, while having almost no degradation when applied to clean speech.

The following paper is structured as follows: the proposed FE method based on rDA is introduced in Section 2, then extensive experiments on spontaneous emotions are described in Section 3, and a conclusion with future work is given in Section 4. hidden layer input layer output layer

{x n t : x n t ∈ X n } {x c t : x c t ∈ X c } LSTM blocks

Feature enhancement

In the past few years, Long Short-Term Memory (LSTM) models have been widely applied to a variety of pattern recognition tasks [START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF][START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF], and show a powerful capability of learning longrange contextual information. In this section, we give a quick overview of such a memory-enhanced RNN, on which the proposed rDA is built.

Memory-enhanced recurrent neural networks

Compared to conventional RNN, the LSTM-RNN model proposed by Hochreiter and Schmidhuber [START_REF] Hochreiter | Long short-term memory[END_REF] uses one or multiple LSTM blocks to replace hidden neurons. Every memory block consists of self-connected linear memory cells c and three multiplicative gate units: an input gate i, a forget gate f , and a output gate o, which are responsible for writing, reading, and resetting the memory cell values, respectively. Given an input xt at the step time t, the activations of the input gate it, the forget gate ft, the memory cell state ct, and the output gate ot are separately updated by the following formulas:

it = fg(Wxixt + W hi ht-1 + Wcict-1 + bi), (1) ft
= fg(W xf xt + W hf ht-1 + W cf ct-1 + b f ), (2) ct = it • fi(Wxcxt + W hc ht-1 + bc) + ft • ct-1, (3) ot = fg(Wxoxt + W ho ht-1 + Wcoct + bo), (4) ht = ot • fo(ct), (5)
where fg, fi, and fo denote the logistic sigmoid, tanh, and tanh activation functions, respectively; W is a weight matrix of the mutual connections; ht presents the output of the hidden block; b indicates the block bias. From the equations mentioned above, it is observed that the values of all memory cells and block outputs in the previous time step t -1 will certainly affect the activations of all three gates, even the input units in the current time step t in the same layer, except for the case between memory cell and output gate. More details about the memory structure can be found in [START_REF] Graves | Supervised sequence labelling with recurrent neural networks[END_REF].

The main advantage of using such a memory-enhanced block over a traditional neuron in a RNN is that the cell state in a LSTM block sums activations over time. Since derivatives distribute over sums, the backpropagated error does not blow up or decay over time (the vanishing gradient problem) [START_REF] Hochreiter | Long short-term memory[END_REF][START_REF] Graves | Supervised sequence labelling with recurrent neural networks[END_REF].

The general structure of rDA with memory-enhanced neural networks proposed in this paper is illustrated in Fig. 1, which includes an input layer, an output layer, and one or multiple hid-den layers that are implemented by the LSTM blocks. In comparison with the conventional DA given in [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF] where the DA is modelled with Feedforward Neural Networks (FNN), the presented DA is structured with the above described LSTM-RNN in the hidden layers. Additionally, it also should be noticed that the recurrent autoencoder also differs from the ones described in [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Srivastava | Unsupervised learning of video representations using LSTMs[END_REF], where an encoder is used to map an input sequence into a fixed length representation, and a decoder is used to decode the target sequence from the representation.

Feature enhancement by an autoencoder

As discussed in Section 1, the speech signal s(k) is easily distorted by the environmental noise and recording devices when facing realistic application scenarios with the Acoustic Impulse Response (AIR) r(k) of finite length T60 and the background additional noise n(k). Therefore, the distorted speech signal ŝ(k) can be expressed as

ŝ(k) = s(k) * r(k) + n(k). (6)
The signal in the time domain ŝ(k) can be approximatedly transformed into the spectrum domain as

| Ŝ(f )| 2 ≈ |S(f )| 2 • |R(f )| 2 + |N (f )| 2 (7)
by applying a Short-Time Discrete Fourier Transform (STDFT) with three assumptions: 1) T60 is shorter than the analysis window size w; 2) The power spectrum of the additive noise in each analysis window w is a slowly varying process, which means that the additive noise is assumed to be stationary in each analysis window; 3) The phase of different analysis windows are non-correlated.

To extract the feature vectors in the ceptral domain such as Mel-Frequency Cepstrum Coefficients (MFCC) for emotion recognition from speech, logarithms and Discrete Cosine Transform (DCT) are performed over the above spectrum. Therefore, Eq. ( 7) can be further formulated into

D(ln| Ŝ(f )| 2 ) ≈D(ln|S(f )| 2 ) + D(ln|R(f )| 2 ) + D(ln(1 + |N (f )| 2 |S(f )| 2 • |R(f )| 2 )). (8)
From Eq. ( 8), we can see that the goal of denoising is to eliminate the impact of the last two terms. For the nonstationary noise, however, the cepstrum does not only fluctuate over time, but is also involved with the original speech spectrum which is non-stationary as well. Therefore, the last term in Eq. ( 8) cannot be simply subtracted due to its non-linear property.

To tackle this non-linear problem, we choose the memoryenhanced rDA as described in Section 2.1 with the purpose of exploiting its advantage of accessing long-range contextual information. The goal of the DA is to reconstruct the features x c in the clean speech feature domain X c from the corresponding features x n in the corrupted speech feature domain X n , as shown in Fig. 1. When providing these corrupted features as the input x n to the first layer, we want the output xn to be highly similar to the clean features x c . To learn the required mapping between noisy and clean features, an objective function -Mean Squared Error (MSE) -is defined to mimise the reconstruction error during training:

J (θ) = 1 T T t=1 (x n t -x c t ) 2 , ( 9
)
where T is the number of frames of the training set.

Experiments and results

In the following, we firstly describe the selected spontaneous emotion database, then evaluate the performance of the proposed FE method based on the rDA with LSTM neural networks for time-and value-continuous emotion recognition.

RECOLA and noise database

For the experiments, we chose the REmote COLlaborative and Affective (RECOLA) database [START_REF] Ringeval | Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions[END_REF] which was used as a database for the 5 th Audio/Visual + Emotion Challenge (AV + EC 2015) [START_REF] Ringeval | AV + EC 2015 -The first affect recognition challenge bridging across audio, video, and physiological data[END_REF]. The motivation of the database collection was to study the complex phenomena, especially emotion, portrayed by humans during social interactions in daily-life.

To generate additive noisy speech, we added the CHiME15 database [START_REF] Barker | The third 'CHiME' speech separation and recognition challenge: Dataset, task and baselines[END_REF] into the clean (raw or original) speech with various levels of SNR (i.e., 0-12 dB at a step of 3 dB). This database was used for the 3rd CHiME Challenge [START_REF] Barker | The third 'CHiME' speech separation and recognition challenge: Dataset, task and baselines[END_REF], and was collected in five different locations, such as booth, bus, cafe, pedestrian area, and street junction. The goal of this database is to simulate emotional speech in different places with various additive background noises.

To generate convolution noise, we applied (via convolution) the Microphone Impulse Response (MIR) of the Google Nexus One smartphone to the recordings from RECOLA using the Audio Degradation Toolbox (ADT) [START_REF] Mauch | The audio degradation toolbox and its application to robustness evaluation[END_REF]. The goal is to simulate reverberant speech being recorded with a smartphone. Moreover, other noises are further simulated via the MIR, by applying the Room Impulse Response (RIR) of classroom or grand hall as the second convolutional noise. This aims to simulate the scenarios that someone speaks on the phone in different environments.

Note that, when adding the CHiME noise, each noise recording was firstly normalised to 0 dB peak energy and concatenated according to the type of noise. Then, the recording was cut into three partitions of the same length for the training, the validation, and the test sets, respectively. Finally, for each recording of RECOLA, we randomly chose an excerpt of the concatenated noise signal from the relevant partition, to ensure both speaker and noise independent partitions.

Experimental setups

At the front-end of the emotion recognition system, 13 Low-Level Descriptors (i.e., MFCCs 0-12) were firstly extracted. In detail, the feature vectors of x n t and x c t were separately extracted from the distorted speech signals and the original clean speech signals at every 10 ms using a window size of 25 ms. Before training the rDA, the global means and variances were calculated of the noisy and clean speech. Then, standardisation was performed over the network inputs and targets using the means and variances from the corresponding training sets, respectively.

For the rDA, both input and output node numbers are equal to the dimension of the feature vector (13 in our case). Two bidirectional LSTM hidden layers were chosen, and each layer consists of 30 memory blocks. During network training, gradient descent was implemented with a learning rate of 10 -6 and a momentum of 0.9. Zero mean Gaussian noise with standard deviation 0.1 was added to the input activations in the training phase such as to improve generalisation. All weights were randomly initialised in the range from -0.1 to 0.1. Note that, all these parameters were optimised on the validation set. Finally, the early stopping strategy was used, i.e., training was stopped when no improvement of the MSE on the validation set has been observed during 20 epochs or the predefined maximum number of training epochs (200 in our case) has been executed. Further, to accelerate the training process, we updated the network weights after running every mini batch of 8 sequences for computation in parallel. The training was performed with our CURRENNT toolkit [START_REF] Weninger | Introducing CUR-RENNT: The munich open-source CUDA RecurREnt Neural Network Toolkit[END_REF].

After the procedure of FE, functionals -mean and variance -were applied over each of the enhanced MFCCs with a window size of 8 s at a step of 0.04 s, which leads to 26 attributes for each window. These statistical features were then fed into the back-end of the system used for emotion recognition, where L2-regularised L2-loss Support Vector Regression (SVR) implemented in the LIBLINEAR toolbox [START_REF] Fan | LIBLINEAR: A library for large linear classification[END_REF] was used. The complexity value of SVR was optimised by the best performance of the validation set, i.e., C = 5 • 10 -5 for arousal and C = 5 • 10 -3 for valence in our experiments.

For the performance evaluation, we choose the Concordance Correlation Coefficient (CCC) [START_REF] Lawrence | A concordance correlation coefficient to evaluate reproducibility[END_REF]. Compared to Pearson's Correlation Coefficient (PCC), CCC can estimate not only the linear correlation, but also the difference of the bias between two variables. Formally, CCC is formulated as

ρc = 2ρσxσy σ 2 x + σ 2 y + (µx -µy) 2 , ( 10
)
where ρ is the correlation coefficient between the two variables; µx and µy are the means of the two variables; and σ 2 x and σ 2 y are the corresponding variances. Moreover, it is worth noting that the gold standard ratings for all the recordings were shifted back in time with a four seconds delay. This is due to the reaction delay of human for continuous emotion annotation [START_REF] He | Multimodal affective dimension prediction using deep bidirectional long short-term memory recurrent neural networks[END_REF].

Results and Discussion

To verify the effectiveness of the rDA with LSTM Neural Networks for FE, we separately performed two experiments: 1) on the non-stationary additive noisy speech only, i.e., by adding CHiME15 noise; 2) on the smartphone related convolutional noisy speech, i.e., the speech is distorted by applying MIR only (smartphone), or additionally by applying RIR of the classroom/hall (+classroom/hall), or additionally by adding various levels of CHiME noise (+CH).

Apart from that, we carried out two FE strategies -1) matched FE: Several FE models are trained separately on the data sets with different noise conditions. For example, when testing on clean speech, the same quality of speech is used, i.e., clean speech is employed to train the rDA; 2) mixed FE: One FE model is trained on a data set with mixed noise conditions. Therefore, the distinction between the two FE strategies is based on the noise condition of the training data that can match or not with the one of the testing data. For example, when testing the clean speech, the mixed conditional speech, i.e., all kinds of CHiME noisy speech or the smartphone related noisy speech together with the clean speech, are utilised to train the rDA.

Table 1 shows the performance of the non-enhanced and enhanced speech (with the matched or mixed FE strategies) evaluated on the emotion recognition model trained on the clean speech for both, arousal and valence regression. In almost all cases, the proposed FE method significantly outperforms the system trained on the non-enhanced noisy speech (baseline).

Taking the additive noisy speech (CHiME15) for example, the average CCC over the recordings at different levels of SNRs on the test set is boosted from 0.563 to 0.596 and 0.594, respectively, by matched and mixed FE for arousal, and from 0.176 to 0.223 and 0.199, respectively, by matched and mixed FE for valence. Furthermore, it is expected that the matched FE outperforms the mixed FE, since the matched FE uses different FE models for denoising corresponding noisy data, whereas the mixed FE trains only one FE model for denoising all kinds of noisy data.

Specifically, the performance obtained on the clean speech condition almost does not degrade when executing the matched FE method, but this conclusion is not supported by performing the mixed FE method. This should be mainly due to huge mismatch between the clean speech and the mixed noisy speech. However, this can be easily solved by inserting a noise detector at the front-end to distinguish whether the signal is noisy or clean [START_REF] Garner | Robust noise detection for speech detection and enhancement[END_REF]. If it was clean, the speech signal can directly be fed into the recognition model without any procedures of FE. Moreover, for the convolutional noise of smartphone in the classroom or in the hall, we can see that our proposed method does not work quite well, however, for the convolutional noise of the smartphone with CHiME noise, the proposed method can surprisingly improve the baseline. This may be because LSTM-RNN does not work efficiently for a linear problem, as the T60 of the MIR used for these experiments is short and the convolutinal noise can be regarded as a constant value in the spectral domain (see Eq. ( 8)).

To further investigate the efficiency of the proposed FE, we calculated the CCC between the enhanced (by matched FE)/non-enhanced speech and the clean speech over the whole test set with the CHiME noise or the smartphone-in-the-hall noise, as illustrated in Fig. 2. It can be seen that the enhanced speech could deliver higher correlation coefficients with the speech, which possibly contributes to the better emotion recognition performance.

Conclusions

We presented a feature enhancement method based on a denoising autoencoder with Long Short-Term Memory neural networks for spontaneous emotion recognition from speech. Extensive experiments were carried out with non-stationary additive noise and convolutional noise. The results show that, the presented feature enhancement method is significantly superior to the baseline without any enhancement methods. With the fast development of deep learning technologies, there are many possibilities that could be used to further improve the robustness performance of emotion recognition systems from speech. For example, Convolutional Neural Networks are good at reducing spectral variation for the clean speech, which could also be effective for noisy speech. Methods combined with Deep Neural Networks in an end-to-end structure [START_REF] Trigeorgis | Adieu features? End-to-end speech emotion recognition using a Deep Convolutional Recurrent Network[END_REF] is worth evaluating as well in future. Further, some other traditional denoising approaches, e.g., minimum mean square error [START_REF] Deng | Highperformance robust speech recognition using stereo training data[END_REF], may be also of interest in this task.

Figure 1 :

 1 Figure 1: Structure of a recurrent denoising autoencoder with LSTM neural networks.

  Figure 2: Concordance Correlation Coefficient (CCC) of 13 Low-Level Descriptors (MFCC 0-12) between the enhanced/non-enhanced speech and the clean speech over the whole test set with the CHiME noise (a) or the smartphone-inthe-hall noise (b).

Table 1 :

 1 Performance (Concordance Correlation Coefficient [CCC]) of the validation and test sets for the proposed matched and mixed feature enhancement (FE) model on the CHiME15 noisy speech only or on the smartphone related noisy speech, in the evaluation of arousal and valence emotional tasks. class.: classroom; CH.: the average CCC over five different 'smartphone + CHiME' noisy speeches with 0-12 dB of SNRs at a step of 3 dB. The mark of "/": no other noise is added onto the smartphone related noisy speech.

						CHiME15				smartphone +
		CCC		clean 12dB	9dB	6dB	3dB	0dB	mean clean	/	class.	hall	CH.	mean
						arousal on the validation set
		baseline	0.736 0.680 0.657 0.626 0.584 0.526 0.635 0.736 0.726 0.629 0.634 0.436 0.545
		matched FE 0.735 0.715 0.710 0.692 0.666 0.627 0.691 0.735 0.723 0.641 0.662 0.675 0.682
		mixed FE	0.693 0.721 0.711 0.691 0.648 0.594 0.676 0.690 0.686 0.650 0.651 0.599 0.630
							arousal on the test set
		baseline	0.732 0.628 0.590 0.542 0.480 0.404 0.563 0.732 0.719 0.618 0.609 0.356 0.495
		matched FE 0.729 0.658 0.646 0.611 0.510 0.422 0.596 0.729 0.713 0.614 0.694 0.682 0.684
		mixed FE	0.717 0.683 0.651 0.598 0.499 0.418 0.594 0.712 0.690 0.612 0.600 0.532 0.586
						valence on the validation set
		baseline	0.402 0.304 0.276 0.246 0.213 0.180 0.270 0.402 0.359 0.306 0.302 0.156 0.239
		matched FE 0.383 0.335 0.299 0.259 0.257 0.223 0.293 0.383 0.335 0.236 0.331 0.187 0.246
		mixed FE	0.201 0.227 0.275 0.275 0.262 0.205 0.249 0.253 0.243 0.251 0.253 0.215 0.230
							valence on the test set
		baseline	0.278 0.190 0.172 0.155 0.139 0.124 0.176 0.278 0.237 0.212 0.205 0.003 0.105
		matched FE 0.269 0.227 0.258 0.214 0.200 0.172 0.223 0.269 0.211 0.126 0.152 0.140 0.162
		mixed FE	0.171 0.210 0.217 0.214 0.202 0.179 0.199 0.195 0.175 0.208 0.209 0.085 0.135
		0.6
		0.5
		0.4
	CCC	0.3
		0.2
		0.1	enhanced non-enhanced
		0 2 4 6 8 10 12
				MFCCs

Acknowledgements

The research leading to these results was supported by the EC's 7th Framework Programme through the ERC Starting Grant No. 338164 (iHEARu), the EU's Horizon 2020 Programme through the Innovative Action No. 644632 (MixedEmotions), No. 645094 (SEWA), and the Research Innovative Action No. 645378 (ARIA-VALUSPA), and by the German Federal Ministry of Education, Science, Research and Technology (BMBF) under grant agreement #16SV7213 (EmotAsS). We further thank the NVIDIA Corporation for their support of this research by Tesla K40-type GPU donation.


Xuan Zhu

Claude Barras
email: barras@limsi.fr

Sylvain Meignier
email: meignier@limsi.fr

Jean-Luc Gauvain
email: gauvain@limsi.fr

COMBINING SPEAKER IDENTIFICATION AND BIC FOR SPEAKER DIARIZATION

This paper describes recent advances in speaker diarization by incorporating a speaker identification step. This system builds upon the LIMSI baseline data partitioner used in the broadcast news transcription system. This partitioner provides a high cluster purity but has a tendency to split the data from a speaker into several clusters, when there is a large quantity of data for the speaker. Several improvements to the baseline system have been made. Firstly, a standard Bayesian information criterion (BIC) agglomerative clustering has been integrated replacing the iterative Gaussian mixture model (GMM) clustering. Then a second clustering stage has been added, using a speaker identification method with MAP adapted GMM. A final post-processing stage refines the segment boundaries using the output of the transcription system. On the RT-04f and ES-TER evaluation data, the improved multi-stage system provides between 40% and 50% reduction of the speaker error, relative to a standard BIC clustering system.

INTRODUCTION

Speaker diarization is the process of partitioning an input audio stream into homogeneous segments according to the speaker identity and the background and channel conditions. Unlike speaker identification or tracking tasks where a priori knowledge of the speaker's voice is provided and an absolute identification is required, the speaker diarization task is relative to a given show, and thus only a relative, show-internal speaker identification is output by the system.

Speaker diarization is a useful preprocessing step for an automatic speech transcription system. By separating out speech and non-speech segments, the recognizer only needs to process audio segments containing speech, thus reducing the computation time. By clustering segments of the same acoustic nature, condition specific models can be used to improve the recognition performance. By clustering segments from the same speaker, the amount of data available for unsupervised speaker adaptation is increased, which can significantly improve the transcription performance. Speaker diarization can also improve readability of an automatic transcription by structuring the audio stream into speaker turns and in some cases by providing the identity of the speakers. Such information can also be of interest for the indexation of multimedia documents.

For most speaker diarization tasks, the number of speakers and the speaker characteristics are unknown a priori, and need This work was partially fi nanced by the European Commission under the FP6 Integrated Project IP 506909 CHIL.

now with the Laboratoire d'Informatique de l'Universite du Maine to be automatically determined. There are two predominant approaches to the speaker diarization problem. The first approach relies on a two step procedure [START_REF] Siegler | Automatic Segmentation, Classification, and Clustering of Broadcast News Audio[END_REF][START_REF] Chen | Speaker, Environment and Channel Change Detection and Clustering via the Bayesian Information Criterion[END_REF][START_REF] Hain | Segment generation and clustering in the HTK broadcast news transcription system[END_REF]. First is the segmentation step, which locates segment boundaries based on acoustic changes in the signal. Second is the clustering step, which regroups segments coming from the same speaker into a cluster. A limitation of this method is that errors made in the segmentation step are not only difficult to correct later, but can also degrade the performance of the subsequent clustering step. An alternative is to optimize jointly the segmentation and the clustering, via, for example, an iterative segmentation and clustering procedure as described in [START_REF] Gauvain | Partitioning and Transcription of Broadcast News Data[END_REF] which uses a set of Gaussian Mixture Models (GMMs). An iterative method based on an ergodic hidden Markov model (HMM) is also proposed in [START_REF] Meignier | E-HMM approach for learning and adapting sound models for speaker indexing[END_REF][START_REF] Ajmera | A robust speaker clustering algorithm[END_REF]. The remainder of this paper is organized as follows: Section 2 describes the baseline partitioning system, and Section 3 describes the Bayesian information criterion (BIC) clustering and speaker identification (SID) clustering used to improve the partitioning system. The experimental results are presented in Section 4 followed by some conclusions.

BASELINE PARTITIONING SYSTEM

The baseline data partitioning system is the first stage of the system developed for the LIMSI English broadcast news transcription system [START_REF] Gauvain | Partitioning and Transcription of Broadcast News Data[END_REF]. This baseline partitioner processes 38 dimensional feature vectors (12 Mel frequency cepstral coefficients, and -coefficients plus the and -energy), and is structured as follows (cf. Figure 1): Speech Activity Detection (SAD): Speech is extracted from the signal with a Viterbi decoding using Gaussian Mixture Models (GMM) for speech, speech over music, music, silence and noise. The GMMs, each with 64 Gaussians, were trained on about 1 hour of data.

Chopping into small segments: Segmentation of the signal is performed by taking the maxima of a local Gaussian divergence measure between two adjacent sliding windows of 0.5 seconds, similar to [START_REF] Siegler | Automatic Segmentation, Classification, and Clustering of Broadcast News Audio[END_REF]. A single diagonal Gaussian is used for each window.

Iterative GMM segmentation/clustering procedure: Each initial segment is used to seed one cluster, and a 8 components GMM with diagonal covariance matrix is trained on the segment data. The algorithm alternates Viterbi resegmentation and GMMs reestimation steps until maximization of the objective function: where is the likelihood of the segments given the models , and and are segment and cluster penalties.

Viterbi resegmentation: The segment boundaries are refined using the last set of GMMs and an additional relative energy-based boundary, within a 1 second interval, so as to avoid cutting words. Bandwidth (studio or telephone) and gender (male or female) labeling is performed on the segments using 4 GMMs with 64 diagonal covariance matrices.

MULTI-STAGE PARTITIONING

In recent research on the speaker diarization task, BIC clustering methods have been widely used with a good performance [START_REF] Moh | Towards domain independent speaker clustering[END_REF][START_REF] Tranter | Speaker diarisation for broadcast news[END_REF]. We therefore tested a modified system, replacing the iterative GMM clustering with BIC-based clustering (cf. Figure 1, (c-bic)). We also pipelined the output of the system into a second clustering stage which uses a speaker identification module (c-sid). Finally, a SAD post-filtering stage was added in order to take into account short pauses. The other parts of the system were kept unchanged.

BIC clustering

Agglomerative clustering is applied to the segments output by one iteration of GMM resegmentation. At the beginning, each segment seeds one cluster, modeled by a single Gaussian with a full covariance matrix trained on the 12 Mel cepstrum coefficients and the energy (but without the coefficients). At each step, the two nearest clusters are merged until the stop criterion is reached. The BIC criterion [START_REF] Chen | Speaker, Environment and Channel Change Detection and Clustering via the Bayesian Information Criterion[END_REF] is used both for the inter-cluster distance measure and the stop criterion.

In order to decide whether to merge two clusters and , the value is computed as:

where is the covariance matrix of the merged cluster ( and ), of cluster , of cluster , and and are respectively the number of the acoustic frames in cluster and . The penalty is:

where d is the dimension of the feature vector space. The merging criterion is that two clusters should be merged if . The clustering procedure terminates when . In our BIC clustering procedure, the size of the two merged clusters, i.e.

, is used in the penalty for the BIC criterion, as described in [START_REF] Cettolo | Segmentation, Classification and Clustering of an Italian Broadcast News Corpus[END_REF]. We refer to this as a local BIC penalty. But in general the size of the whole set of clusters, i.e.

has to be used in the penalty, which we refer to as a global BIC penalty. Since the BIC criterion is used as the distance measure for merging the clusters, using the total size makes the penalty constant, so the decision to merge two clusters is decided just by the increase in likelihood for the global BIC case. The local BIC thus seems to be a better choice for a merging criterion, even if it is not optimal as a stop criterion.

SID clustering

Speaker clustering methods performed by either the iterative GMM or the BIC agglomerative clustering procedures have to deal in the beginning of the process with short duration segments, and thus use a limited set of parameters per cluster. After several iterations, the amount of data per cluster increases, so a more complex model can be used. In addition, purely acoustic clustering tends to split a speaker's data into several clusters as a function of the various background conditions (clean speech, speech with noise, speech with music etc.), so an acoustic background normalization is necessary to regroup the data for a given speaker.

After the BIC clustering stage, state-of-the-art speaker recognition methods [START_REF] Reynolds | Speaker verification using adapted gaussian mixture models[END_REF][START_REF] Barras | Feature and score normalization for speaker verification of cellular data[END_REF] were used to improve the quality of the speaker clustering. The feature vector consists of 15 Mel frequency cepstral coefficients plus delta coefficients and delta energy with a feature warping normalization [START_REF] Pelecanos | Feature warping for robust speaker verification[END_REF]. For each gender (male, female) and each channel condition (studio, telephone) combination, a Universal Background Model (UBM) with 128 diagonal Gaussians is trained on the 1996/1997 English Broadcast News data. For each initial cluster, maximum a posteriori (MAP) adaptation of the means of the matching UBM is performed.

Agglomerative clustering is performed separately for each gender and bandwidth condition, using a cross log-likelihood ratio as in [START_REF] Reynolds | Blind clustering of speech utterances based on speaker and language characteristics[END_REF]. For each cluster , its model is MAP adapted from the gender and channel matched UBM using the feature vectors belonging to the cluster. Then, given two clusters and , the cross log-likelihood ratio is defined as: where is the likelihood of the acoustic frames given the model , normalized by the length of the signal. This is a symmetric similarity measure. After each merge, a new model is trained for the cluster . The clustering stops when the cross log-likelihood ratio between all clusters is below a given threshold estimated on development data.

SAD post-filtering

The output of the LIMSI Broadcast News Speech-To-Text system is used in a post-processing stage to filter out shortduration silence segments that are not detected by the initial speech detection step. Only inter-word silences lasting at least 1 second are filtered out, this value being determined on development data.

EXPERIMENTS AND RESULTS

Several configurations were tested for the systems. By default, the configuration used is the one that provided the best results on development data, i.e. for c-std, for c-bic and for c-sid and p-asr. A local BIC merging and stop criterion was also used.

Databases

The experiments were conducted on the development database (dev1) and the evaluation database used in NIST RT-04f (Fall 2004 Rich Transcription Evaluation) [START_REF]Fall 2004 Rich Transcription (RT-04f) Evaluation Plan[END_REF] and on the databases of the French ESTER broadcast news evaluation [START_REF] Gravier | The ESTER evaluation campaign of rich transcription of french broadcast news[END_REF]

Performance measures

The speaker diarization task performance is measured via an optimum mapping between the reference speaker IDs and the hypotheses. The primary metric for the task is the fraction of speaker time that is not attributed to the correct speaker, given the optimum speaker mapping. In addition to this speaker error, the overall speaker diarization error includes also the missed and false alarm speaker times, thus taking speech/non-speech detection errors into account [START_REF]Fall 2004 Rich Transcription (RT-04f) Evaluation Plan[END_REF].

In order to analyze better the performance of speaker clustering methods, average frame-level cluster purity and cluster coverage are used as defined by [START_REF] Gauvain | Partitioning and Transcription of Broadcast News Data[END_REF]. Cluster purity is defined as the ratio between the number of frames by the dominating speaker in a cluster and the total number of frames in the cluster. Cluster coverage accounts for the dispersion of a given speaker's data across clusters.

Results on the RT-04f development data

As expected, the standard partitioner c-std in its default configuration provides a high purity, but a relatively poor coverage, resulting in a high overall diarization error over 30% on dev1 data (cf. Table 1). Setting the penalty and to optimize these values reduces this error below 25%. The c-bic system also provides a high purity, with much better coverage (resp. 97% and 90%), reducing the overall error rate by almost 50%. The c-sid system achieves a large increase of the coverage without degradation of the purity, resulting in a global error rate about 7%, a reduction of almost 50% compared to c-bic system.

A global BIC merging and stop criterion was also tested, but always performed worse compared to the local BIC criterion in our experiments, as can be seen for c-bic system on RT-04f dev1 (cf. Table 2). A similar result was observed in [START_REF] Tranter | Speaker diarisation for broadcast news[END_REF]. This result remains to be further interpreted but may be due to an inadequacy between the BIC modelization and the real distribution of the data. Thus only the local criterion was used in the remaining experiments.

Looking in more detail at the performance of the c-sid system, a large variation of the speaker error across shows is observed, ranging from the lowest error of 0.8% for MNB show to over 12% for ABC and NBC shows (cf.

Results on the ESTER development data

For ESTER, SAD was performed using the same acoustic models for RT-04f plus an additional speech over music model trained on French broadcast news data. The optimal threshold for the SID clustering on development data was . Similar to the results obtained on the RT-04f dev1 dataset, the csid system provides also high cluster purity and coverage (resp. 95.3% and 94.8%) on the ESTER development dataset (cf. Table 1). A 50% reduction of overall error rate is gained by adding the c-sid system to the c-bic system.

Results on the evaluation data

On the RT-04f evaluation dataset, the trend observed on the development data was confirmed, with a slight increase in overall diarization error to 17% for the c-bic system and to 9.1% for the c-sid system. The final SAD post-processing stage gives an improvement of 0.6%, mainly by reducing false alarms in speech detection (cf. Table 4). As mentioned in [START_REF] Reynolds | Approaches and Applications of Audio Diarization[END_REF], the p-asr system had the best performance in all the participants of the RT-04f evaluation. Postevaluation experiments show that the speaker error for the c-sid system could be reduced from 6.9% to 6.0% with the SID clustering threshold . On the ESTER evaluation dataset, with the setting optimized 4: Performances of c-bic, c-sid and p-asr systems on the evaluation data of RT-04f and ESTER (*these systems are the primary systems submitted to the evaluations.).

on the development database, overall diarization error was reduced from 13.8% for the c-bic system to 11.5% for the c-sid system (cf. Table 4). Postevaluation experiments on the evaluation data illustrate that the c-sid system has a still better result (9.1% overall diarization error) with .

CONCLUSIONS

In this paper, we have presented the LIMSI improved speaker diarization system. Several modifications to the baseline system have been explored by replacing the iterative GMM clustering with the combination of an agglomerative BIC clustering and a second clustering using the state-of-the-art speaker identification techniques.

The improved system performs much better for the diarization task. On the RT-04f development data, the improved system has a relative speaker error reduction of over 75% compared to the baseline system. An overall diarization error under 10% was obtained with the c-sid system on the RT-04f evaluation data, while the performance of BIC-based systems was at least 15%. In the ESTER evaluation, the c-sid system has 8.0% overall diarization error on the development dataset and 9.1% overall error could be obtained on the evaluation dataset with an optimal threshold. This dramatic improvement over the baseline system results from several changes: a model complexity which increases with the average amount of speech data per cluster, and the combination of two different clustering stages and models, each one focusing on a different acoustic aspect.

Several issues remain to be investigated in order to improve the robustness and the efficiency of the system. It was observed that the clustering threshold needs to be set according to the length of the audio document, and that the system still has a large variability across individual shows. Only with a large amount of files can statistically consistent results be obtained. Finally, as most speaker diarization systems rely on a purely acoustic segmentation and clustering, combining the acoustic with the linguistic layer as explored in [START_REF] Canseco-Rodriguez | Speaker diarization from speech transcripts[END_REF] would improve the robustness of a speaker diarization system and make it more exploitable by a human reader.

Figure 1 :

 1 Figure 1: Standard baseline LIMSI partitioning system (c-std on the left side of the diagram) and improved speaker diarization system (p-asr to the right, along with c-bic and c-sid intermediate steps).

  . The development database (dev1) used in English RT-04f consists of 6 audio files recorded in February 2001. The evaluation database consists of 12 audio files recorded in December 2003. All the audio files last 30 minutes and were extracted from different US radio and television broadcast news shows. The development database for ESTER consists of 8 hours audio data from 4 radio broadcast news shows in French (France Inter, France Info, RFI, RTM); the evaluation database of ES-TER consists of 10 hours data from the same 4 sources plus 2 new sources. The audio files last from 15 minutes to one hour.

Table 3 )

 3 .

	system		cluster	coverage overall
			purity (%)	(%)	error
		RT-04f dev1 dataset
	c-std (	)	95.0	71.6	32.3
	c-std (	)	90.6	82.1	24.8
	c-bic (	)	97.1	90.2	13.2
	c-sid (	)	97.9	95.8	7.1
		ESTER development dataset
	c-bic (	)	92.8	89.4	15.8
	c-sid (	)	95.3	94.8	8.0

Table 1 :

 1 The cluster purity, cluster coverage and the overall diarization error from the systems c-std (both in initial configuration and best configuration), c-bic and c-sid on the RT-04f dev1 dataset and the ESTER development dataset.

	BIC	overall	BIC		overall
	criterion	error	criterion		error
		5.0 13.3%		5.0	16.4
	local	6.0 12.8%	global	6.0	15.5
		7.0 13.8%		7.0	18.2

Table 2 :

 2 The overall diarization error for c-bic system on the RT-04f dev1 database, as a function of the penalty weight for the local and global BIC criterion.

Table 3 :

 3 Performance of c-sid system on the RT-04f dev1

	show REF SYS MS FA SPK DIA
	average	-	-	0.4 1.3	5.4	7.1
	ABC	27	37	1.6 1.3 12.4 15.2
	VOA	20	22	0.3 1.2	2.2	3.7
	PRI	27	30	0.1 0.9	2.8	3.8
	NBC	21	35	0.1 1.1 12.0 13.2
	CNN	16	21	0.5 1.4	5.6	7.6
	MNB	10	16	0.2 1.8	0.8	2.8

dataset, scores are given for miss (MS), false alarm (FA), speaker error (SPK) and overall diarization error (DIA), REF and SYS are respectively the reference and system speaker number.