Charlotte Baey

Paul-Henry Cournède

Using a hierarchical segmented model to assess the dynamics of leaf appearance in plant populations

Keywords: nonlinear mixed model, segmented regression, variability, sugar beet

Modeling inter-individual variability in plant populations is a key issue to enhance the predictive capacity of plant growth models at field level. In sugar beet, this variability is well illustrated by the phyllochron (thermal time elapsing between two successive leaf appearances): even if the mean phyllochron remains stable within a given variety, there is a high heterogeneity between individuals. When considering the dynamics of leaf appearance as a function of thermal time in sugar beet, two linear phases can be observed, leading to the definition of a hierarchical segmented model with four random parameters varying from one individual to another: thermal time of initiation, first phyllochron, rupture thermal time and second phyllochron. The SAEM-MCMC algorithm is used to estimate the model parameters.

Introduction

A new trend in crop modeling is the development of individual-based plant growth models such as functional-structural plant models (FSPM), combining the description of plant architecture and its physiological functioning (Vos et al. [START_REF] Vos | Functional-Structural plant modeling in crop production[END_REF], de Reffye et al. [START_REF] De Reffye | Plant Growth Models[END_REF]).

The extrapolation of individual-based models to the field scale is still at its early stages. Due to the complexity to describe the growth of all individual plants in the field, an average plant is usually considered for the model calibration process and prediction at field level, as illustrated by Lemaire et al. [START_REF] Lemaire | A morphogenetic crop model for sugar beet (Beta vulgaris L.)[END_REF] for sugar beet. Beside the difficulty of assessing this 'average plant', the strong variability among individuals makes the average plant prediction quite restrictive, since it only gives a partial characterization of field production.

This variability could arise from three different sources identified by de Reffye et al. [START_REF] De Reffye | Modeling Inter-individual Variability in Sugar Beet Populations[END_REF]: the growth initial conditions (typically, the seed weight and the thermal time of initiation), the environmental conditions (the growth of a given plant varies with local environmental conditions), and the genetic variability (in the case of sugar beet, populations are not pure bred, resulting in genetic variations between individuals).

In this paper, we focus on the phenomenon of plant organogenesis (i.e. the dynamics of appearance of plant organs) since its right description is crucial for the development of FSPM and since it is affected by several sources of variability. For most crops, the thermal time (corresponding to the accumulation of daily temperature above a base temperature) elapsing between the successive appearances of two phytomers (the elementary structural unit of a plant) usually shows a good range of stability, even though it may be strongly affected by environmental factors, see Clerget et al. [START_REF] Clerget | Variability of Phyllochron, Plastochron and Rate of Increase in Height in Photoperiod-sensitive Sorghum Varieties[END_REF] for a short review on the main results for different plants. In given conditions, the phyllochron may reveal constant for each individual of a population, while showing a strong variability between individuals within a cultivar, see Frank and Bauer[8]. This is typically the case for sugar beet (see Lemaire et al. [START_REF] Lemaire | A morphogenetic crop model for sugar beet (Beta vulgaris L.)[END_REF]), which is not pure-bred. Likewise, seedling emergence may strongly vary within a population, potentially inducing important variations in the final yield, see Liu et al. [START_REF] Liu | Response of Corn Grain Yield to Spatial and Temporal Variability in Emergence[END_REF], if the younger plants do not manage to close the gap with the older ones. The development of late plants could also be slowed down by the shadow resulting from the extra-leaves of the early plants.

The sugar beet plant is specifically chosen as test plant for our study. As shown by Lemaire et al. [START_REF] Lemaire | A morphogenetic crop model for sugar beet (Beta vulgaris L.)[END_REF], and already observed by Milford et al. [START_REF] Milford | An analysis of leaf growth in sugar beet. I. Leaf appearance and expansion in relation to temperature under controlled conditions[END_REF], two phases can be observed in the development of new leaves by the sugar beet, leading to the definition of two different phyllochrons. We will thus study the inter-individual variability of the thermal time of initiation (seedling emergence), of the two phyllochrons (controlling the rate of leaf appearance in these two phases), and of the rupture thermal time, i.e. the transition thermal time corresponding to the setting up of the second phase.

To take into account these two phases, a segmented regression will be used, with an unknown break-point corresponding to the rupture thermal time. The variability of the parameters will be assessed using a nonlinear mixed model. We first introduce our methodology of analysis. In section 2, a description of the hierarchical segmented regression model is given, and inference method is described in section 3. We then apply the methodology to experimental data for the sugar beet in section 4, and we also show how the model can be used to build comparisons between different plant populations.

The model

Nonlinear mixed models are of particular interest for the analysis of repeated measures data, in many research fields such as pharmacokinetics, agriculture, epidemiology ... In such models, the functional form of the model linking the response variable to time (in the case of longitudinal data) is the same for all individuals, but some parameters are allowed to vary among individuals (see Lindstrom and Bates [START_REF] Lindstrom | Nonlinear Mixed Effects Models[END_REF], Davidian and Giltinan [START_REF] Davidian | Nonlinear Models for Repeated Measurement Data[END_REF], Pinheiro and Bates [START_REF] Pinheiro | Approximations to the Log-Likelihood Function in the Nonlinear Mixed-Effects Model[END_REF] and the references therein).

The sugar beet organogenesis model can be described as a two-stage hierarchical model. In the first stage, the number of leaves according to the thermal time is modeled for a given plant. In the second stage, the variability between plants is assessed by considering each parameter as a random variable.

First stage : intra-individual variation Let y ij denote the number of leaves of plant i at thermal time t j . Then we have:

y ij = f (t j , φ i ) + e ij (1)
with φ i the vector of parameters specific to individual i, and e ij a random error term following a normal distribution N (0, σ 2 ). Thus, f characterize the systematic variation whereas e ij represents the random variation of measurements from individual i.

In our case, f is a two-linear phases function defined as follow:

f (t j , φ i ) = φ 1,i (t j -φ 0,i ) + φ 3,i (t j -φ 2,i ) 1 tj ≥φ2,i (2)
with φ 0,j the thermal time of initiation, φ 1,i the first slope (the inverse of the first phyllochron), φ 2,i the rupture thermal time and φ 3,i the difference in slopes between the two phases for plant i. The parameter vector is

φ i = (φ 0,i , φ 1,i , φ 2,i , φ 3,i ).
With this formulation, we model the change in slopes between the two phases, rather than the two distinct slopes, and we force the two lines to join at the rupture thermal time.

Second stage : inter-individual variation In this stage, the inter-individual variation is taken into account by the following model for the parameter vector φ i :

φ i = β + b i , b i ∼ N (0, D) (3)
where β = (β 0 , β 1 , β 2 , β 3 ) is a vector of fixed parameters, b i is a vector of random effects associated with individual i, and D is a 4×4 covariance matrix. It is assumed to be diagonal with elements ω 2 0 , ω 2 1 , ω 2 2 and ω 2 3 . This formulation corresponds to the one described by Lindstrom and Bates [START_REF] Lindstrom | Nonlinear Mixed Effects Models[END_REF], although more general forms can be used to model the interindividual variations (see Davidian and Giltinan [START_REF] Davidian | Nonlinear Models for Repeated Measurement Data[END_REF]). Similarly, we make here the normal assumption for the random components b i , but a more relaxed hypothesis can be assumed.

Inference method

A lot of inference methods have been proposed for the nonlinear mixed models, most of them based on maximum likelihood estimation, with a likelihood function based on the joint density of the observations given the covariates.

However, because of the nonlinearity of f , this density has an integral form which is in general analytically intractable.

A list of the different methods can be found in a review by Davidian and Giltinan [START_REF] Davidian | Nonlinear Models for Repeated Measurment Data: An Overview and Update[END_REF]. The most popular methods are based on an approximation of the likelihood, and are well implemented in classical statistical softwares (for example via the SAS procedure proc nlmixed or the S-PLUS/R package nlme). However, the likelihood approximation on which they are based may be poor, for example if the number of observations per subject n i is not large enough or when the Gaussian assumption no longer holds (see Makowski and Lavielle [START_REF] Makowski | Using SAEM to estimate parameters of models of response to applied fertilizer[END_REF]).

An alternative to these methods is to use "exact" or "direct" methods, in which the likelihood function is maximized directly using EM-algorithm, for example (see Walker [START_REF] Walker | An EM Algorithm for Nonlinear Random Effects Models[END_REF]). Advances in computational power have made these methods more and more appealing. Delyon et al. [START_REF] Delyon | Convergence of a stochastic approximation version of the EM algorithm[END_REF] proposed a stochastic approximation of the EM-algorithm, which converges under very general conditions to a local maximum of the function. Kuhn and Lavielle [START_REF] Kuhn | Coupling a stochastic approximation version of EM with an MCMC procedure[END_REF] showed that the convergence to maximum likelihood estimates holds when the algorithm is coupled to a MCMC procedure. This method has the advantage of being quicker than an usual EM-algorithm (see Kuhn and Lavielle [START_REF] Kuhn | Maximum likelihood estimation in nonlinear mixed effects models[END_REF]), so that the convergence can be obtained within a few seconds, and observed likelihood and Fisher Information Matrix could also be estimated. The SAEM-MCMC is implemented in the free software MONOLIX [START_REF] Team | User Guide to Monolix[END_REF].

Results

In 2008, field experiments were conducted in France, under three different density conditions: low (5.4 plants per m 2 ), standard (10.9 plants per m 2 ) and high (16.4 plants per m 2 ). The number of leaves of the 45 sugar beet crops (15 plants in each density) was collected at a maximum of 14 different dates. Daily mean values of air temperature ( • C) were obtained from French meteorological advisory services (Météo France) five kilometers away from the experimental site to compute the thermal time after sowing.

The model was first applied to the reference density in sugar beet crops, and then a comparison between the three densities was performed. It has been shown by Milford et al. [START_REF] Milford | An analysis of leaf growth in sugar beet. I. Leaf appearance and expansion in relation to temperature under controlled conditions[END_REF] and observed by Lemaire et al. [START_REF] Lemaire | A morphogenetic crop model for sugar beet (Beta vulgaris L.)[END_REF] that the phyllochron of the first phase of development remains quite constant among plant densities, whereas the duration of this first phase was subject to change. This change in phyllochron is observed approximately at full leaf cover, when the competition for light increase. In our model, we thus assumed that the thermal time of initiation and the first slope do not depend on the population density, but we let the rupture thermal time and the second phyllochron vary among the cropping densities.

To estimate the values of θ = (β, σ, D), a set of initial values is required for β. We used a first set of values from a previous study of leaf appearance in sugar beet, described in Lemaire [START_REF] Lemaire | Systme dynamique de la croissance et du dveloppement de la betterave sucrire (Beta vulgaris L.)[END_REF]. Other sets of initial values were tested, leading to similar results. Results of the model with the highest likelihood are presented in Table 1. Table1. Results of the SAEM algorithm for the standard density (10.9 pl/m 2 )

Parameter
The average thermal time of initiation in the population is estimated at 99 • Cdays, and the average rupture thermal time is estimated at 894 • Cdays. The two mean phyllochrons can be easily calculated by taking the inverse of the slopes, and their variances can be approximated using the delta method: if X is a random variable

Var 1 X ≈ Var(X) E(X) 4 (4)
We find for the first phyllochron a mean value of 36.1 • Cdays and a standard deviation of 1 • Cdays in the population, and for the second phyllochron, a mean value of 68.5 • Cdays and a standard deviation of 6.9 • Cdays in the population (we calculated the variance of the second slope by taking the sum of the two variances ω 2 1 and ω 2 3 , assuming that the two parameters φ 1 and φ 3 are independent).

To compare the different densities, two dummy variables I s,i and I h,i were added to the model, equal to 1 if the i-th plant grew in standard and high density respectively, and 0 otherwise.

The function f remains unchanged, but the parameters model is now:

φ i = A i β + b i , b i ∼ N (0, D) (5)
with:

A i =    
1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 I s,i I h,i 0 0 0 0 0 0 0 0 1

I s,i I h,i     (6)
and β = (β 0 , β 1 , β 2,l , δ 2,s , δ 2,h , β 3,l , δ 3,s , δ 3,h ) t .

Parameters δ 2,s , δ 3,s , (resp. δ 2,h , δ 3,h ) represent the difference between the parameter values in the standard density (resp. the high density) and the parameter values in the low density condition. Wald tests can be used to test for a significant difference between the three densities. Different sets of initial values were used for the parameter vector β, and results of the model with the highest likelihood are presented in Table 2 Table2. Comparison of the three densities.

No differences were observed between the standard and the high density. The rupture thermal time was significantly different between low and standard density (1280 • Cdays vs. 1001 • Cdays), and between low and high density (1280 • Cdays vs. 786 • Cdays). The second phyllochron was estimated at 74.1 • Cdays for the low density, at 61 • Cdays for the standard density, and at 65.3 • Cdays for the high density. A significant difference for the second slope between standard and low density was observed, but not between standard and high density.

Conclusion

The hierarchical segmented regression model used here allows for a better handling of the plant heterogeneity, and thus a better statistical description of the population. All the available data can be used as we no longer resort to an average plant, and comparison between different cropping densities can be performed, taking into account the variability between plants. The mean population value of the parameters and their inter-individual variability can be used as inputs of functional-structural plant models, as described in de Reffye et al. [START_REF] De Reffye | Modeling Inter-individual Variability in Sugar Beet Populations[END_REF]. The main issue is then to compute the propagation of these sources of probabilistic uncertainty in dynamic systems of plant growth. This issue has already been studied in the context of crop models by Monod et al. [START_REF] Monod | Uncertainty and Sensitivity Analysis for Crop Models[END_REF]. However, the proposed method to study the output distribution is based on Monte Carlo sampling. It bears strong limitations for assessing individual variability by inverse methods, since it would be necessary to run the Monte Carlo simulations a large number of times. There exist several other methods widely used in other domains (like unscented filtering, see Julier et al. [START_REF] Julier | A new method for the nonlinear transformation of means and covariances in filters and estimators[END_REF] and Bevington and Robinson [START_REF] Bevington | Data reduction and error analysis for the physical science[END_REF]) that overcome this difficulty. The next step of our work is the implementation of these methods to fully develop a full functional-structural plant population model and the associated estimation procedures.

Acknowledgement

We are grateful to ITB (sugar beet research institute, Paris) for providing the experimental data to test our methodology. This study is funded by the research project CANTIA (competitiveness cluster IAR, Picardie and Champagne-Ardennes regions).

  .

	Parameter Estimate Standard error Wald tests (p-value)
	β0	70.3	16	-
	β1	0.0267	0.00064	-
	β 2,l	1280	7.6	-
	δ2,s	-279	22	< 0.0001
	δ 2,h	-494	15	< 0.0001
	β 3,l	-0.0132	0.0012	-
	δ3,s	-0.00289	0.0013	0.022
	δ 3,h	-0.00182	0.0012	0.13
	ω0	13.6	6	-
	ω1	0.00246	0.0003	-
	ω2	8.11	5.3	-
	ω3	0.00234	0.00044	-
	σ	1.53	0.052	-
	Log-likelihood		-1167.9


Ingmar Steiner

Slim Ouni

Towards an articulatory tongue model using 3D EMA

Within the framework of an acoustic-visual (AV) speech synthesizer, we describe a preliminary tongue model that is both simple and flexible, and which is controlled by 3D electromagnetic articulography (EMA) data through an animation interface, providing realistic tongue movements for improved visual intelligibility. Data from a pilot study is discussed and deemed encouraging, and the integration of the tongue model into the AV synthesizer is outlined.

Introduction

It is well known that the human tongue plays a significant role in speech production. It participates in several kinds of sounds, and its position is critical for several, if not most, phonemes. Even though it is only partially visible from outside the mouth, the tongue helps to improve the intelligibility of audiovisual speech. This is especially important for lip reading since it gives useful information for dental and liquid sounds, for example. Within the framework of developing a talking head, adding a tongue model increases drastically the overall intelligibility of the visual articulation, compared to talking heads displaying only the face and lips.

The tongue is a complex organ that is highly flexible, expandable, compressible, can be curved and displays a fine degree of articulation (e.g. [START_REF] Abd-El-Malek | Observations on the morphology of the human tongue[END_REF][START_REF] Bole | Electromyography of the genioglossus muscles in man[END_REF][START_REF] Carpentier | La langue: un ensemble musculaire complexe[END_REF]. The tongue and its muscles are laterally symmetrical: a median septum divides the organ into two halves. The dynamic aspects of the tongue articulation (mainly coarticulation) are important as this has an influence on the intelligibility of the overall gestures.

In previous work, several approaches have been taken to model the shape, as well as the dynamic properties, of the tongue during speech, with varying complexity (e.g. [START_REF] Pelachaud | Modeling and animating the human tongue during speech production[END_REF][START_REF] King | Creating speech-synchronized animation[END_REF][START_REF] Gerard | 3D biomechanical tongue modeling to study speech production[END_REF][START_REF] Lu | From experiments to articulatory motion -a three dimensional talking head model[END_REF], and while many use vocal tract MRI to obtain static tongue shapes, the dynamics of running speech are more successfully observed using modalities such as electromagnetic articulography (EMA). Most specifically, [START_REF] Engwall | Combining MRI, EMA & EPG measurements in a three-dimensional tongue model[END_REF] presents a MRI-adapted, 3D parametric tongue model, which can be externally controlled by EMA data, but this data is only two-dimensional in nature, and the mapping from the EMA coils to model control parameters is somewhat simplified.

Within the scope of a project developing an acoustic-visual (AV) speech synthesizer,1 our aim is to implement and integrate a tongue model into the synthesizer's animated talking head that will improve the intelligibility and naturalness of the resulting AV speech output. In particular, the movements of the tongue model should appear as natural as possible.

Modeling a realistic tongue with high precision in articulation presents a significant challenge. In developing the AV synthesizer, we mainly focus on visual intelligibility. In this context, the tongue is partially visible and thus we consider that the dynamics are more important than finely modeling the shape or position of the tongue. For these reasons, the objective of this work is to build a simple dynamic tongue model.

The AV synthesizer concatenates units selected from a multimodal corpus of facial motion-capture data and simultaneous acoustic recordings [START_REF] Toutios | Towards a true acoustic-visual speech synthesis[END_REF]. The motion-capture was performed with a stereoscopic high-speed camera array tracking 252 facial marker points at a rate approaching 200 Hz. The acoustic data is processed and used directly for speech synthesis by waveform concatenation in a unit selection paradigm.

Apart from the facial motion-capture, no further articulatory data is available for the AV corpus, because the instrumentation of modalities such as EMA would have interfered with the optical acquisition. While simultaneous optical and articulatory tracking has been demonstrated in the past (e.g. [START_REF] Yehia | Quantitative association of vocaltract and facial behavior[END_REF][START_REF] Engwall | Resynthesis of 3D tongue movements from facial data[END_REF][START_REF] Kroos | Measurement accuracy in 3D electromagnetic articulography (Carstens AG500)[END_REF][START_REF] Lu | From experiments to articulatory motion -a three dimensional talking head model[END_REF], invasive components such as wires and transducer coils would have significantly affected the quality of the acoustic recordings, and the duration of the recording session (and thus, the size of the resulting multimodal corpus) would have been limited by EMA-inherent practicalities (e.g. coil detachment).

Nevertheless, with the AV synthesizer driven entirely by data captured from a real human speaker, we explore the possibility of animating the tongue through articulatory data in an analogous manner. Of course, for this to be possible, an auxiliary corpus of such data must be available, and multimodal synthesis from both corpora must be synchronized and merged appropriately. The former precondition is the subject of this paper, while the integration of facial/acoustic and tongue movement synthesis will be addressed in a later stage of the project.

The key difference between the facial motion-capture data in the primary AV corpus and tongue surface data in the auxiliary articulatory corpus is one of granularity. The 252 marker points represent the vertices of a mesh sufficiently fine to serve as the basis for the surface of the synthetic face. The movements of the tongue surface, on the other hand, cannot be tracked using such a fine sampling, since the technical limitations of EMA restrict the number of transducer coils. Therefore, EMA offers high temporal resolution, but at best only a sparse representation of the tongue surface.2 Conversely, the five-dimensional measurement data of the Carstens AG500 EMA system provides not only the position of each coil in space, but also its orientation [START_REF] Kaburagi | Three-dimensional electromagnetic articulography: A measurement principle[END_REF].

In this paper, we present how the 3D positions and the 2 rotation angles of the EMA data can be used to provide information for tongue animation. We start from a very simple geometric tongue model. Then, the articulatory data, acquired using the AG500, is used to control the tongue shape.

Tongue model and Animation

To integrate a tongue model into the AV synthesizer, using the same approach for tongue and facial animation, articulatory point tracking data at a high temporal resolution is required. However, unlike the facial motion-capture data, which provides 3D movement data for hundreds of points on the speaker's face, enough to use these points directly as the vertices of the facial mesh in the talking head, only a very small number of transducer coils (by comparison), and hence, fleshpoints on the tongue can be tracked using EMA. The resulting data yields an insufficient number of vertices to be used for the tongue surface in the same way as the AV motion-capture data for the face.

To address this issue, we reduce the dimensionality of the tongue control in such a way that the tongue mesh is not wholly defined by, but only controlled by, the EMA data. This corresponds to the fact that the human speaker's tongue is not controlled by the EMA coils, but that the coils represent observations of fleshpoints on the tongue surface, and move in synchrony with them (by virtue of being physically attached).

An advantage of 3D EMA which counters some of the drawbacks of data sparsity, lies in the fact that along with the spatial positions of the transducer coils, their orientation in spherical coordinates is measured as well. This additional information is well-suited for transformational projection onto geometric structures larger than the coils themselves.

The tongue mesh itself is defined by a set of vertices arranged in a shape that happens to resemble that of a human tongue, but could in fact just as well be a sphere, cylinder, or similar construct. The critical fact is that it moves in unison with the EMA data controlling it. Each EMA coil can be associated with a subset of the tongue mesh vertices, and this association can furthermore be weighted either individually or by vertex groups. Hence, the movement and rotation of each EMA coil can be transferred to the vertices, thereby animating the tongue mesh. This can be described as a form of the skeletal animation technique.

However, both the position on the surface of the tongue and the small, pointlike shape, of the EMA coils limit the suitability of the coils for direct use in skeletal animation. For this reason, we allow the skeletal animation to be controlled by the EMA coils only indirectly, by means of a weighted mapping to a pseudo-skeleton, or articulated armature, rigged inside the tongue mesh in a neutral bind pose. In line with conventional skeletal animation, the armature is in turn mapped to vertex groups on the tongue mesh by a weight map which can be defined automatically from the envelope of the armature's shape and manually adjusted if required, and manipulating the armature's components deforms the surrounding mesh accordingly. Figure 1 shows a simple armature and its automatic vertex weight map, where the influence of the armature's components on the mesh is visualized on a color scale, ranging from blue (low) to red (high).

The resulting tongue animation preserves the high temporal resolution of the controlling EMA data, and is both simple and flexible, as shown in Figure 2. Moreover, the details of the animation can be fine-tuned through the armature-coil mapping, as well as through the armature structure itself, and the vertex groups associated with it. In addition, the modular design allows the simple structure of the preliminary tongue model to be refined and improved at a later stage by replacing the mesh and rebinding the armature.

A distinct advantage of this approach to tongue modeling for AV synthesis lies in the fact that it employs well-developed and state-of-the-art concepts and techniques, and can be prototyped with relative ease using off-the-shelf 3D modeling software and engines, even addressing more advanced issues such as collision detection or soft body dynamics, by accessing the corresponding physics subsystems.

Pilot study

As a pilot study and to provide preliminary data during the development of the tongue model, a small corpus of 3D EMA data was recorded using the Carstens AG500 Artic- ulograph. Since our focus was on the tongue surface, there was no need to record lip movements; instead, a total of eight coils were attached to the speaker's tongue: four coils in the mid-sagittal plane (tongue tip, blade, mid, and back center), as well as two on each side (tongue blade left/right, and tongue back left/right); the resulting layout is shown with the tongue model in Figure 3. In addition, the jaw aperture was captured by a coil on the lower incisors. Reference coils for the normalization with respect to head movement were placed on the upper incisors, and behind both ears. However, using the alternative normalization approach proposed by [START_REF] Kroos | Using sensor orientation information for computational head stabilisation in 3D electromagnetic articulography (EMA)[END_REF], it might be possible to increase the number of available measurement coils on the tongue even further by using fewer reference coils.

The unusually large number of tongue coils and their corresponding wires presented a practical challenge, and we did not manage to attach all of the coils in the optimal orientation. Ideally, for coils arranged along the mid-sagittal contour of the tongue, the coil axes should lie in the mid-sagittal plane, while for the lateral coils, a transverse axis orientation tends to provide a more robust indication of tongue grooving. In addition, we are currently uncertain whether the close proximity of coils on the tongue may have interfered with the measurement accuracy for some of the EMA channels (see below).

The recorded material includes CVC sequences, several utterances from a French prompt list, a 3D palate trace (using the LInc coil, attached to a pen), and one sweep during which the speaker explored various unusual tongue configurations without speaking.

This EMA data was visualized in a 3D environment, with the tongue model animated through the interface described above. Preliminary assessment seems to indicate that the tongue model can indeed be animated in a satisfactory way using the design described here, but a number of issues remain to be addressed. For instance, the optimal smoothing of the noise in the raw EMA trajectories must be determined to remove vis-ible jitter without negatively affecting the animation. Another issue is the reliability of the EMA data; a few coils in the pilot data exhibit intermittent out-of-bounds positions. Such unrealistic movements correlate with RMS spikes for the corresponding channels, but some dimensions seem to be more sensitive than others. The source of the problem might be related to the coil hardware itself, or the coil layout (in particular, their orientation and close proximity), or perhaps the method used to extract the positions from the raw data. While it may be possible to debug and reduce such errors, it is more than likely that they cannot be fully eliminated, and so the tongue model's control interface must be extended to robustly and gracefully handle these invalid positions.

Integration

While the tongue model is yet to be integrated with the AV synthesizer described in Section 1, several approaches are conceivable to generate the EMA trajectories required to control the tongue model for unseen utterances, i.e. within the final text-to-speech (TTS) application. All of them make use of a planned corpus, containing EMA recordings for a short prompt list providing at least minimal phonetic coverage.

By applying acoustic-to-articulatory inversion techniques, it may be possible to enrich the full AV corpus with reconstructed articulatory data offline, which can be used at synthesis time for tongue model animation in a way analogous to EMA data. However, it remains to be explored whether the additional dimensions required can be recovered in a satisfactory way by inversion techniques (most of which were developed using 2D EMA). If such an approach proved feasible, then the integration of the tongue model into the multimodal unit selection synthesizer would be straightforward, as corresponding articulatory trajectories would be bundled with every selected unit and could be used to animate the tongue along with the face for the corresponding segments of synthesized speech.

An alternative approach for the generation of articulatory control data lies in trajectory synthesis to generate the required control data at synthesis time, either using statistical models (e.g. [START_REF] Ling | HMM-based text-to-articulatory-movement prediction and analysis of critical articulators[END_REF] or by direct concatenation (e.g. [START_REF] Engwall | Evaluation of a system for concatenative articulatory visual speech synthesis[END_REF] in a separate, but parallel unit-selection thread fed by the auxiliary corpus. In the latter case, the units would not necessarily have to match those selected from the primary AV corpus for speech and facial animation, but could be handled in a syllable frame or as kinematic n-phones [START_REF] Okadome | Generation of articulatory movements by using a kinematic triphone model[END_REF].

Conclusion

We have described a tongue model designed for integration into an AV speech synthesizer, whose primary goal is to increase the visual intelligibility of a talking head. The model is simple but flexible, composed of a 3D mesh controlled by 3D EMA data through a skeletal animation interface. Initial tests with a small corpus of pilot data show encouraging results.

Future work includes refining the interface to robustly handle noisy data with intermittent measurement errors, recording and processing a larger, phonetically balanced EMA corpus, and integrating the tongue model into the AV synthesizer. An open question is whether the animation of the face and tongue, which so far are completely separate, could be partially combined, perhaps with one or both informing the other (e.g. [START_REF] Yehia | Quantitative association of vocaltract and facial behavior[END_REF][START_REF] Engwall | Resynthesis of 3D tongue movements from facial data[END_REF]. Finally, an evaluation study is planned to assess the performance of the tongue model.

Figure 1 .

 1 Figure 1. Top view of armature (six components) and an automatically assigned vertex weight map, with the colors indicating the degree of influence of each component on the vertices; the tongue mesh is visible as a wireframe (tip on right).

Figure 2 .

 2 Figure 2. Side view (slightly oblique) of tongue mesh (tip on right) and armature, superimposed as six stick-like shapes, and two deformations of the mesh (tongue back towards the viewer) obtained by manipulating the armature. The center and right panels show a bunched and retroflex configuration of the tongue model, respectively.

Figure 3 .

 3 Figure 3. Top view of tongue model (tip on right) and layout of EMA coils on the tongue in pilot study. Coils, shown in aqua color, with a short line indicating the orientation, along the center (left to right): TBackC, TMidC, TBladeC, TTipC, and TBackL, TBladeL (top) and TBackR, TBladeR (bottom).

http://visac.loria.fr/

Some previous studies[START_REF] Kaburagi | Determination of sagittal tongue shape from the positions of points on the tongue surface[END_REF][START_REF] Qin | Reconstructing the full tongue contour from EMA/X-ray microbeam[END_REF] have explored the possibility of reconstructing the tongue contour from EMA data, but only in the mid-sagittal plane.