A Bouënard
email: abouenard@gmail.com

S Gibet

M M Wanderley

Hybrid Inverse Motion Control for Virtual Characters Interacting with Sound Synthesis Application to Percussion Motion

Keywords: Physics-based Computer Animation, Motion Control, Motion-Sound Interaction, Sound Synthesis

The ever growing use of virtual environments requires more and more engaging elements for enhancing user experiences. Specifically regarding sounding virtual environments, one promising option to achieve such realism and interactivity requirements is the use of virtual characters interacting with sounding objects. In this paper, we focus as a case study on virtual characters playing virtual music instruments. We address more specially the real-time motion control and interaction of virtual characters with their sounding environment for proposing engaging and compelling virtual music performances. Combining physics-based simulation with motion data is a recent approach to finely represent and modulate this motion-sound interaction, while keeping the realism and expressivity of the original captured motion. We propose a physically-enabled environment in which a virtual percussionist interacts with a physics-based sound synthesis algorithm. We introduce and extensively evaluate the Hybrid Inverse Motion Control (HIMC ), a motion-driven hybrid control scheme dedicated to the synthesis of upper-body

Introduction

Playing a musical instrument involves complex human behaviors. Instrumental gestures are indeed gestures that require extreme rapidity, accuracy, and physical engagement with the instrument. While performing, a skilled musician is able to accurately control his motion, as well as perceive both the reaction of the instrument to his own actions and the resulting sound. The tight relationship between sound and the performer actions is crucial for understanding the mechanisms underlying musical learning and performance, and for finally proposing realistic and engaging virtual music performances. Transposing these real-world experiences into virtual environments offers the possibility of exploring novel solutions where virtual characters interacting with virtual musical instruments are central. We believe that these simulation experiments provide valuable clues for better modeling of instrumental gestures, and by extension for designing multimedia environments where animated entities interact realistically with sounding objects.

Most studies that focus on animating expressive and dexterous gestures use data-driven kinematics approaches, but these fail in modeling collisions that are involved in movements. However, the fine control of gesturesound interaction with relevant musical nuances requires the modeling of effects due to human motor control. To compensate the lack of physical realism of these pure kinematic approaches, a class of methods for animating virtual characters that interact realistically with the environment propose physical approaches. Such approaches rarely address the simulation of instumental gestures, and colliding effects to the corresponding sound production. Finally, simulating virtual characters interacting with musical instruments entails combining the principles of both kinematics and dynamics approaches, as a means of reflecting and taking into account the richness of motion strategies during music performances. In this work, we more specifically address the influence of motion control on the produced sound, and we highlight three major issues in the case of percussing gestures: i) we adopt a physics-based approach, both for animating the virtual character and for synthesizing sound; we also model the subtle physical interactions occurring as the drumstick makes contact with the drum membrane. We assume that this physical modeling gives rise to compelling contact information that is shown to be one of the sources of motionsound realism and variations [START_REF] Gaver | How Do We Hear in the World? Explorations in Ecological Acoustics[END_REF];

ii) we use motion capture data for driving our simulations. This type of control ensures that the produced motion remains fluid and expressive, and also provides significant clues for constraining the produced movements;

iii) at a planning level, we focus on the preparatory motion which is very relevant in percussion practice [START_REF] Bouënard | Gesture Control of Sound Synthesis: Analysis and Classification of Percussion Gestures[END_REF]. The short duration of beat impacts moreover emphasizes the existence of motion strategies to rapidly form adequate contact profiles for various musical nuances [START_REF] Wagner | Analysis of Drumbeats: Interaction between Drummer, Drumstick and Instrument[END_REF].

The major contributions of this paper for synthesizing compelling percussion motion are in the direction of these three points. We propose a novel motion control approach, the so-called Hybrid Inverse Motion Control (HIMC ), which combines a physics-based controller and a kinematics planner driven by motion capture data. Although this motion control scheme uses existing components from the animation community, its originality comes from the embedded nature of kinematic and dynamic controllers. Furthermore, we use at the planning level a reduced set of three-dimensional captured data, thus enabling a more intuitive conducting of the virtual character. Our aim is to show the feasibility, from an animation point of view, of a hybrid motion control scheme that integrates the inherent constraints of human data. We also propose an original real-time framework for animating the virtual character, and interacting with a sound-producing system. This multisensory environment is used to explore the interaction between motion and sound, by modifying the parameters at different levels: planning, motion and sound control. This opens new perspectives for a compositional system of gestures, editing gesture trajectories and producing in real time visual, audio and gestural outputs.

The paper is organized as follows. We begin in section 2 by providing an overview of our multisensory system, and exposing our motivations, for motion control, sound synthesis and multisensory framework. Section 3 describes our HIMC scheme, an evaluation is also conducted by comparing qualitative and quantitative results with traditional inverse kinematics and inverse dynamics strategies. Finally, in section 4, we propose an interaction scheme with sound synthesis, and present simulation results. A discussion is outlined in section 5.

Overview and motivation

Three main issues are generally addressed when considering the animation and interaction of virtual characters with sound synthesis processes. One first issue is to preserve the realism of the simulated motion in comparison with captured motion examples, while enabling the physical interaction of the virtual character with its surrounding environment. It generally involves physically-based simulations driven by motion capture data, to ensure the physical integrity of the simulated motion even under external disturbances. A second critical issue is the sound synthesis technique to be used, so that a compelling and effective interaction can be defined between motion parameter outputs and sound parameter inputs. Finally, a third issue to be considered is the synchronization of these different modalities -motion and sound -and its use in interactive applications. Finally, we conclude this section by highlighting the specificities of drumming gestures.

Motion control

The adaptive and responsive control of virtual characters has been intensively investigated in physics-based computer animation research. Most of contributions have addressed the control of articulated figures using roboticsinspired proportional-derivative (PD) controllers [START_REF] Raibert | Legged Robots that Balance[END_REF]. This has inspired many works for handling different types of motor tasks, such as walking, running, swimming or boxing [START_REF] Hodgins | Animating Human Athletics[END_REF][START_REF] Yang | Layered Dynamic Control for Interactive Character Swimming[END_REF][START_REF] Zordan | Dynamic Response for Motion Capture Animation[END_REF]. A lot of contributions have then focused on the main three issues concerning physicsbased animation using controllers: reusability, tuning, and low/high level control. The first two issues are generally addressed either by composing controllers dedicated to specific motor tasks [START_REF] Faloutsos | Composable Controllers for Physics-Based Character Animation[END_REF][START_REF] Coros | Robust Taskbased Control Policies for Physics-based Characters[END_REF][START_REF] Dasilva | Linear Bellman Combination for Control of Character Animation[END_REF], or helping to cope with the hard and time-consuming process of tuning such PD controllers [START_REF] Allen | On the Beat! Timing and Tension for Dynamic Characters[END_REF].

Our work is more related to the third issue, i.e. how to provide higher-level control policies in physics-based animation. This approach is motivated by the fact that most physical controllers use multi-dimensional postures as input, which prevents from intuitive control.

One solution is the use of hybrid methods, combining physics-based controllers and motion data, in such a way that we may associate the advantage of user controllability of kinematic methods and the responsiveness of dynamic controllers. Previous contributions have followed this direction. A solution is to use procedural methods combined with physics simulation [START_REF] Ko | Animating Human Locomotion with Inverse Dynamics[END_REF]. Transitioning between kinematic and dynamic controllers has also been studied, either by using simple sequencing policies, or by using searching/indexing methods [START_REF] Shapiro | Hybrid Control For Interactive Character Animation[END_REF][START_REF] Zordan | Dynamic Response for Motion Capture Animation[END_REF] based on motion capture data. Another recent contribution is the use of approximated equations of motions [START_REF] Mordatch | Robust Physicsbased Locomotion using Low-dimensional Planning[END_REF].

Our work differs from previous works by proposing a higher-level and intuitive control scheme, composed of a cascaded collaboration between inverse kinematics (IK ) and inverse dynamics (ID). More specifically, this new method provides a solution to the control of a physics-based virtual character only by specifying endjoint positions. IK has also been used in conjonction with ID control in [START_REF] Zordan | Tracking and modifying upperbody human motion data with dynamic simulation[END_REF], but only as a pre-process step rather than in the core of the control algorithm. Regarding percussion motion, the use of captured data of reduced dimension as inputs to our HIMC control scheme is particularly consistent, since: i) drumstick extremity trajectories have been identified as motion parameters that characterize motion variability and may be used to discriminate different performances [START_REF] Bouënard | Gesture Control of Sound Synthesis: Analysis and Classification of Percussion Gestures[END_REF];

ii) cognitive experiments show that the shape of the three-dimensional mallet trajectory prefigures the type of stroke and the nature of the produced sound [START_REF] Dahl | Playing the Accent: Comparing Striking Velocity and Timing in Ostinato Rhythm Performed by Four Drummers[END_REF];

iii) such higher-level representation of motion is closer to the planning level than the whole posture of motion, and facilitates the edition of motion, according to a motion score composed of beat sequences [START_REF] Bouënard | Advantages and Limitations of Simulating Percussion Gestures for Sound Synthesis[END_REF][START_REF] Bouënard | Virtual Gesture Control and Synthesis of Music Performances: Qualitative Evaluation of Synthesized Timpani Exercises[END_REF].

Our approach for controlling the virtual character uses motion capture data. Driving the simulation by real motion data is a means of preserving the variability and expressiveness of real percussion gestures. We have already shown that expressive cues may be retained by drumstick extremity trajectories [START_REF] Bouënard | Gesture Control of Sound Synthesis: Analysis and Classification of Percussion Gestures[END_REF] and are due to various percussion playing variations, such as using different drumstick grips, motion variations reflecting musical nuances, as well as various beat impact locations.

We propose in this paper an original control strategy extracting end-joint trajectories to drive a combined inverse kinematics and dynamics systems, as shown by pipeline 2 in figure 1, and partially introduced in [START_REF] Bouënard | Hybrid Motion Control combining Inverse Kinematics and Inverse Dynamics Controllers for Simulating Percussion Gestures[END_REF]. Such a dual inversion process has not been addressed for motion control in computer animation. In previous contributions, traditional inverse kinematics applied to a geometrical model of a virtual character has been proposed, as illustrated by pipeline 1 in figure 1, but it does not allow physical interactions with a virtual instrument. Other approaches such as inverse dynamics directly exploit inner-joint orientations to drive a physical model, as shown by pipeline 3 in figure 1 [START_REF] Zordan | Tracking and modifying upperbody human motion data with dynamic simulation[END_REF][START_REF] Bouënard | Enhancing the Visualization of Percussion Gestures by Virtual Character Animation[END_REF], but unfortunately offer a lower control level due to the high dimension of input motion poses. Conversely, our dual inversion scheme (HIMC ) provides a higherlevel control still with physical interaction. HIMC is particularly uneasy to model and implement, as it provides potential instabilities and tuning difficulties. In our approach, we avoid these instabilities by exploiting captured data to statistically determine the joint limits and to compute biomechanical parameters (center of mass, length, mass, density and inertia of the bodies). We also provide a comparative assessment of our HIMC model with conventional ones: classical inverse kinematics (IK ), inverse dynamics (ID) from postures, and real human data.

Sound synthesis

Along with the dynamic animation of virtual characters, physically-motivated sound synthesis methods have been widely studied, with the analogous goal of creating sounds that can adapt to actions and changes in environment or object properties. Since the early interest of the graphics community in modeling the vibrations of objects for sound synthesis, most works have either focused on the direct use of vibrational modes based on measurements [START_REF] Van Den Doel | FoleyAutomatic: Physically-Based Sound Effects for Interactive Simulation and Animation[END_REF][START_REF] Corbett | Timbrefields: 3d Interactive Sound Models for Real-Time Audio[END_REF], or the simulation of surface vibrations using the finite element method [START_REF] O'brien | Synthetizing Sounds from Rigid Body Simulations[END_REF]. More recently, the high interest on modal synthesis [START_REF] Loy | The Mathematical Foundations of Music[END_REF] has led to acceleration algorithms for handling complex environments or objects [START_REF] Raghuvanshi | Interactive Sound Synthesis for Large Scale Environments[END_REF][START_REF] Bonneel | Fast Modal Sounds with Scalable Frequency-Domain Synthesis[END_REF][START_REF] Picard | A Robust and Multi-scale Modal Analysis for Sound Synthesis[END_REF]. Our work involves in a similar manner physical models based on modal synthesis, but we point out and address the difficulty of relating these sound synthesis schemes to their cause in the context of music performance, i.e. the instrumental motion of a virtual performer. Here the focus is therefore on the simulation of percussion gestures, as well as on the interaction between motion and sound synthesis.

Multisensory architecture

We propose a real-time architecture for exploring the physical interaction between motion and sound synthesis. Figure 2 shows a block diagram of our framework. It is divided into three main parts: the motion simulation of the virtual character, the interaction process, and the sound synthesis process. Biomechanical data are first extracted from the motion database and used to initialize the physical model of the virtual character. Motion capture data are also the input of the control policy involving our HIMC model. The motion simulation of the virtual character is then used to realize interacting experiments with a simulated sound synthesis system.

Our framework synchronizes different modalities and heterogenous data: captured motion, motion simulation and sound control parameters used by the simulation processes; visual and audio data used to visualize the results of the simulation. Previous contributions have also adopted a motion-driven approach for the synchronized generation of soundtracks from animations [START_REF] Takala | Sound Rendering[END_REF][START_REF] Cardle | Sound-by-Numbers: Motion-Driven Sound Synthesis[END_REF]. However as demonstrated in many audio-visual works, despite many recent improvements, most computer animation and simulation frameworks are still reluctant to integrate such interaction or synchronization methods, therefore moving away from the interactive realism and presence that sound can give to computer animation applications. Our goal is to show how physical modeling of the motion-sound interaction, combined with a carefully-designed system architecture can be of interest to obtain engaging and realistic virtual environments. Moreover, we do not focus on a specific sound synthesis model, but we explore different interaction schemes attached to modal synthesis principles.

Specificities of percussion motion

There is a large amount of percussion instruments, which vary in size, shape, quality of the drumming surface. There is also a great variability among drumming gestures, such as the placement, the involvement of the whole body or not, the force needed for striking, and the type of the grip. Percussion performances involve ballistic gestures that precede the strike, the so-called "preparatory gesture", and impulse-based gestures during strike. Moreover, the contact with the instrument is limited to a few millisecond [START_REF] Wagner | Analysis of Drumbeats: Interaction between Drummer, Drumstick and Instrument[END_REF], consequently the striking force and effects on the produced sound have to be induced by the entire gesture. Finally the use of a striking tool such as a drumstick, a mallet, a hammer, or the hands allows the percussionist to excite the instrument with varying forces and velocities, and to alter the timbre of the produced sound by changing the shape, hardness, weight, and stiffness of the tool. All these features make drumming gestures a very rich subject of study which has not been previously explored before in computer animation, both for the expressiveness and the variability of the performance, and for the control of synthesized sounds. We chose to work on timpani gestures which have the specificity to involve the whole upper-body during performance, and are characterized by movements of large amplitude during the preparatory phase. For this class of gestures we have captured a large amount of data expressing different styles and nuances. Our captured database covers fourty five sequences of drumming, for three percussionists, with five playing modes (legato, tenuto, accent, vertical accent and staccato), and three impact locations on the timpani, each sequence containing at least 10 strikes, both for individual training exercices and improvisation sequences. This study required the participation of two students and one professional percussionists to make a selective choice of drumming sequences and to record the different performances.

Motion simulation: virtual character modeling and motion control

We describe in this section the approach that we adopt for physically modeling and controlling percussion motion from pre-recorded motion capture data. It involves a muscular-skeleton model of the virtual character depicted in figure 3, by merging the two traditional kinematics and dynamics representations [START_REF] Van Welbergen | Real-time Animation of Virtual Humans: a Trade-off between Naturalness and Control[END_REF]. Our novel motion control paradigm takes advantage of this double representation by the definition of a hybrid controller driven by motion capture data according to two control modes. The first control scheme uses simple ID controllers similar to previous contributions, whereas the second HIMC mode involves a novel hybrid motion control scheme including a cascaded process of IK and ID controllers. This latter allows for the control of the physical model of the virtual character solely by the specification and editing of drumstick extremity trajectories.

Virtual character modeling

The muscular-skeleton model of the virtual performer is composed of two skeleton layers: a dynamic layer and a kinematic layer, equation [START_REF] Allen | On the Beat! Timing and Tension for Dynamic Characters[END_REF].

D = [ s D = {m, s, d, I} [1,N ] , j D = {k s , k d } [1,M ] ] K = [ j K = {q, q} [1,M ] , l K = {λ, e} [1,M ] ] (1)
The dynamical skeleton D models the physical properties of the virtual character, and is composed of a set of rigid bodies s D articulated by mechanical joints j D , figure 3(b) shows the considered capsule-shaped skeleton. Motion capture data of percussion performances are used to physically model and parameterize both the anthropometry and the mechanical joints of the virtual character, making a direct yet simplified correspondence between the recorded performer and the virtual character.

The physical properties of each rigid body composing the virtual character, such as mass (m), size of the limbs (s), density (d) and inertia matrix (I) are consistent with the anthropometry extracted from motion capture data. Table 1 specifies the mass percentage of each rigid body as regards to the total mass of the virtual character, from which the other dynamic parameters are derived using human anthropometry knowledge, for instance [START_REF] Dempster | Properties of Body Segments based on Size and Weight[END_REF]. Note that drumsticks are integral parts of the dynamic model D of the virtual character, so that they can be considered as proper body limbs, consequently upper-body end-effectors will be referred to as drumsticks extremities.

Mechanical joints are parameterized by proportional (k s ) and derivative (k d ) coefficients that traditionally represent the tension/relaxation between rigid bodies in PD inverse dynamics control.

The kinematic layer K of the virtual character as shown in figure 3(a) represents the observation of the kinematic state of D, i.e. the linear-angular positions and velocities (q, q) of the joints articulating the dynamic representation. State features q and q are obtained by integrating the set of motion equations over time through a forward dynamic scheme, using the Lagrangian formalism d dt ( ∂L ∂ qi) ) -∂L ∂qi = Q i , q i and qi being the generalized coordinates (quaternion angular position and angular velocity), and Q i the generalized effort for each degree of freedom i of the system.

In addition, each mechanical joint has three rotational degrees of freedom, restricted to the angular limits of the human body, in order to avoid non realistic motion. Two formulations are available to extract the "lower" and "upper" limits of the angular joints, based on a statistical analysis of the motion capture data. The first formulation uses the Euler angle representation, and computes basic statistical features to characterize joint limits. The singularity of this represention is however frequent (gimbal lock ), therefore we propose another formulation based on the quaternionic representation presented in [START_REF] Johnson | Exploiting Quaternions to Support Expressive Interactive Character Motion[END_REF].

Joint limits are processed in the quaternion space, as detailed by algorithm 1. The rotational (quaternionic) trajectory Q of a joint over time is transposed in the tangential space of its quaternionic mean q, in which a Singular Value Decomposition is applied. The resulting eigen values and axes (λ, e) are used for representing the joint limits l of a joint by the specification of the bounding ellipsoid around the considered quaternionic mean.

!"#$"% &'#'(')* +"%#,"- Xt !" !# qt !" qt (q, q, X) $%&%'
During the simulation, the respect of joint limits from bounding ellipsoids is then achieved by verifying if each joint orientation transposed in its respective mean tangent space is valid (i.e. contained in ellipsoid). If it is not valid, torques are computed so that mechanical joints achieve the orientation corresponding to the tangent space vector optimizing the quadratic distance on the considered ellipsoid surface. This minimum-distance in the tangent space corresponds to the minimum-path on the unit sphere in the quaternion space. Table 1 also specifies the eigen values and axes representing the bounding ellipsoid for the upperbody joints, resp. neck, shoulder, elbow and wrist. These parameters have been obtained from the different percussion recordings, allowing to constrain these parameters for the specific task of percussion.

Motion control

We propose two ways for realizing the motion control of the muscular-skeleton model, as illustrated in figure 4. The first control paradigm proceeds to the traditional tracking of motion capture data in the joint space (quaternionic trajectories). The novel HIMC scheme that we propose is motivated by the expressive factor of drumstick control, and proceeds to the tracking of endeffector trajectories in the 3D Cartesian space. While tracking motion capture data in the joint space requires ID control, HIMC tracking in the end-effector space requires a cascaded collaboration of both IK and ID controllers. In the latter case, both inversion processes are strongly linked.

ID motion control

ID control is related to tracking motion capture data by using PD controllers. Kinematic postures of the joints composing the virtual character (qua-ternionic trajectories q t ) are first extracted from motion capture data, and used to drive the motion of the physics-based model of the virtual character.

We use traditional PD feedback controllers, modeled as damped springs and parameterized by manuallytuned stiffness and damping coefficients (k s , k d ). Knowing the current state of the mechanical joints (q, q) and the angular target q t to be reached, the needed torque τ is computed and exerted on the corresponding articulated bodies according to equation [START_REF] Aubry | Modeling Joint Synergies to Synthesize Realistic Movements[END_REF].

τ = k s .(q t -q) -k d . q
(2)

HIMC: Hybrid Inverse Motion Control

The Hybrid Inverse Motion Control (HIMC ) is a more intuitive physics control of the virtual character combining IK and ID controllers. Instead of directly tracking quaternionic trajectories (q t ) from the motion capture database, HIMC consists in tracking end-effector positions in the 3D Cartesian space (drumsticks extremities X t for arms, for instance). Knowing the Jacobian J q of the system to be controlled, as well as its Cartesian target X t and current state X, an IK method computes a kinematic posture q t according to equation (3). The kinematic posture q t is then used similarly to the previous point as the desired input of PD controllers, thus providing the required torques τ to control the physical character, as shown in figure 4.

∆q t = J -1 q .(X t -X), q t = q + ∆q t τ = k s .(q t -q) -k d . q (3)
Algorithm 2 shows the principle of the HIMC control scheme, composed of two nested loops. The first one concerns the task specification (drumstick extremity position). Between two task updates, the second loop involves the cooperation of the IK and ID formulations. For example, for a task frequency of 250 Hz and a simulation frequency of 10000 Hz, the second simulation loop is called 10000/250=40 times, which is experimentally sufficient for the IK convergence, and consequently the ID convergence.

Alg. 2 HIMC combining IK and ID controllers. In F T , F S Task and simulation frequencies (q, q)

Current kinematic posture X Current end-effector position X t

Targetted end-effector position

Out

Jq

Jacobian matrix q t

Targetted kinematic posture τ Resulting torques

while Read task at F T do X t ← New drumstick extremity steps ← 0 repeat -Inverse Kinematics Jq ← ComputeJacobian(q) q t ← InverseKinematics(Jq, X, X t )

-Inverse Dynamics τ ← InverseDynamics(q, q, q t ) -Forward Dynamics ApplyTorques(τ ) Simulation()

steps ← steps + 1 until steps = F S /F T end
The modularity of the presented approach for combining IK and ID controllers allows the consideration of any method, so that our system supports at the momemt several IK formulations that can be combined with simple ID formulations: transpose, pseudo-inverse, damped-least-squares. Recent advances using learning techniques [START_REF] Gibet | Expressive Gesture Animation Based on Non Parametric Learning of Sensory-Motor Models[END_REF][START_REF] Aubry | Modeling Joint Synergies to Synthesize Realistic Movements[END_REF] could also be considered.

Using an IK formulation necessitates the computation of the Jacobian matrix, underlining the need of the double kinematic and dynamic representation of the virtual character. The main difficulty with the coupling of both kinematic and dynamic controllers is that the convergence of the IK algorithm is added to the difficulty of tuning the parameters of the PD dynamic controllers. However, once adequate PD parameters are found, this approach enables the manipulation of motion capture data in the 3D Cartesian space (configuration X t ) instead of the joint space (q t ), which is more consistent and intuitive for controlling percussion motion. Using end-effector trajectories such as drumstick extremities is a means of compressing the motion, and may therefore facilitate the edition of percussion gestures to be synthesized.

Results

The physical model of the virtual character is composed of 19 joints totalling 57 degrees of freedom. The presented results focus more specifically on the upperbody, which counts 11 bodies articulated by 11 joints totalling 33 degrees of freedom. The Open Dynamics Engine (ODE ) physics simulation toolkit 1 is used for the forward dynamic simulation of the motion equations. The ID implementation used in these experiments also exploits ODE 's internal force/torque maxima, for preserving simulation stability while not sacrificing simulation quality with over-damped behaviours.

The HIMC tracking mode can take advantage and can be combined to various inverse kinematic methods. These results have been obtained using an implementation of the damped-least-squares method [START_REF] Wampler | Manipulator Inverse Kinematic Solutions based on Vector Formulations and Damped Least Squares[END_REF], a simple yet more robust adaptation of the pseudo-inverse method regarding the singularity of the inverse kinematic problem.

The results presented in this section compare the HIMC control mode to both traditional IK and ID control modes. It should be noted that to make such a comparison possible between HIMC and ID control modes, parameters that could drastically change the resulting simulations have been kept constant. Such parameters include for instance the simulation rate, the recorded 1 Open Dynamics Engine, http://www.ode.org motion to be simulated, as well as the parameterization of the damped springs composing the dynamic layer of the virtual character.

A qualitative evaluation is first conducted. The results obtained by the two tracking modes used to physically control the virtual percussionist are compared: a) motion capture tracking in the joint space, involving only ID controllers, and b) motion capture tracking in the Cartesian space, using the HIMC combination of IK and ID controllers. Then we extensively and quantitatively evaluate the two tracking modes by quantifying the errors made during the simulation of various motion variations in each case.

A critical issue to consider is the metric used for evaluating the effectiveness of HIMC compared to traditional ID control. Previous works have addressed the question of the user sensitivity to various motion metrics [START_REF] Reitsma | Perceptual Metrics for Character Animation: Sensitivity to Errors in Ballistic Motion[END_REF][START_REF] Ren | A Data-driven Approach to Quantifying Natural Human Motion[END_REF]. In this work, the ballistic nature of percussion motion does however make it easy to conduct an extensive evaluation by focusing on drumstick extremity trajectories. We also show that our solution leads to a coherent motion in the joint space.

Qualitative evaluation

For this qualitative evaluation, we ran the simulation of a set of pre-recorded percussion motions. Figure 5 compares the original motion to the resulting simulated motion using the three considered control modes: IK only, ID only and HIMC.

The hybrid control scheme tracks a sequence of percussion gestures for synthesizing whole arm movements only from the specification of the drumstick tips. original data and data generated by the three control modes for wrist, elbow and shoulder flexion angle trajectories. These results show that data generated by the IK involved in the hybrid control mode are consistent with real ones, especially for wrist and elbow trajectories. IK techniques do not generally guarantee that angular data are exactly following mocap data, however IK gives coherent results thanks to its embedding with ID in the core control process, and thanks to a tracking process using close successive targets. Data generated by our hybrid solution is even more accurate compared to ID control, the latter shows indeed a motion exageration that is avoided by our solution. More specifically for the shoulder flexion angle, it seems that the IK process alters the original motion. Joints closer to the skeleton root are generally considered as more important than the others, for instance for locomotion. However this is not necessary the case for the considered percussion tasks. When the percussionist faces his instrument, the position of the root does not move very much, so that the motion of his arms (elbow, wrist) become more important than shoulder motion. This loss is therefore of lower importance for percussion motion, as argued in [START_REF] Bouënard | Analysis of Percussion Grip for Physically-based Character Animation[END_REF][START_REF] Bouënard | Gesture Control of Sound Synthesis: Analysis and Classification of Percussion Gestures[END_REF] that wrist and elbow angle trajectories are important degrees of freedom in percussion (timpani) arm mechanisms, in comparison to shoulder angle trajectories.

We also present the comparison of the two control modes (ID control only and HIMC ) for height trajectories of the drumstick extremity, wrist and elbow in figures 5(d), 5(e) and 5(f). One interesting issue is the accuracy of the hybrid control mode compared to the simple ID control. The motion exageration previously observed in joint flexion trajectories in the case of ID control is propagated on joint position trajectories. Conversely, HIMC leads to a more accurate tracking in joint position trajectories, as shown in the animation (Online Resource 1). This observation lies in the fact that the convergence of motion capture tracking is processed in the joint space in the case of ID control, adding and amplifying multiple errors on the different joints and leading to a greater error than processing the convergence in the Cartesian space for the hybrid control.

One limitation however arises in elbow position trajectories. While the ID control mode results again in an amplified motion, our hybrid control solution re-sults in a loss in the motion pattern of original data. The elbow position trajectory resulting from the hybrid control mode leads to a stiffer and somewhat constant motion. This can be related to the inverse kinematic process which does not include any secondary goals, as already pointed out in [START_REF] Klein | Review of Pseudoinverse Control for use with Kinematically Redundant Manipulators[END_REF]. This may be explained also by the fact that the IK algorithm favorizes the accuracy of the most distal joints compared to joints at the basis of the articulated chain. Another reason for this limitation could be the influence of the physical simulation on the convergence of the inverse kinematic scheme.

Although such a cascaded chain of IK and ID controllers runs in real-time, the main drawback of this improvement is however the additional computational cost of the IK algorithm which is processed at every simulation step. It nevertheless provides a more flexible motion edition technique for controlling a physics-based virtual character solely by end-effector trajectories. It also provides a consistent control scheme as regards to current knowledge about the control mechanisms occuring in real percussion performances [START_REF] Dahl | Playing the Accent: Comparing Striking Velocity and Timing in Ostinato Rhythm Performed by Four Drummers[END_REF][START_REF] Bouënard | Gesture Control of Sound Synthesis: Analysis and Classification of Percussion Gestures[END_REF].

Quantitative evaluation

An extensive evaluation of ID and HIMC control modes is conducted, taking into account the test of the two control algorithms on 100 simulation trials for various motion variations (from top to bottom: legato, tenuto, accent, vertical accent and staccato).

Figure 6(a) shows examples of depth/height phase trajectories of the drumstick extremity position for each motion variation. Motion variations present drastically different dynamics in drumstick motion, especially regarding its depth and height amplitudes. For example, accent variation differs from the others through a stiffer motion, characterized by a higher height amplitude and a more reduced range of motion in the saggital direction. Conversely vertical accent and staccato variations are special compared to the others as they are characterized with a much greater depth motion amplitude. These differences in motion dynamics play an important role in the expressiveness of percussion motion, as illustrated in the corresponding animation (Online Resource 1).

Results of the quantitative evaluation are illustrated in figure 6(b). The experience consisted in the generation of 100 simulation trials for each motion variation. Each simulation trial leads to the computation of the root mean square error (RMS-E ) and standard deviation (RMS-SD) as regards to the drumstick extremity from the corresponding motion initially chosen in the motion capture database. From the results presented in figure 6(b), one can easily conclude that the cascaded -E ) and standard deviation (RMS-SD) between captured and simulated drumstick trajectories for various motion variations, using the two motion control schemes: ID and HIMC. RMS-E (blue for ID, red for HIMC ) and RMS-SD (black) are obtained from 100 simulation trials for each motion variation (from top to bottom: legato, tenuto, accent, vertical accent and staccato).

hybrid combination of IK and ID controllers leads to a more accurate minimization of the error made on drumstick extremity trajectories of about 2% of motion amplitude, compared to simple ID controllers. The standard deviation along simulation trials are also more reduced for the hybrid motion paradigm compared to the ID control. Such a difference of 2% in the average errors between the two control modes could be considered as negligeable. However, drumstick extremity information around beat impacts can change drastically the nature of the resulting sounds, that is why such amelioration is of paramount importance for proposing compelling contact information both for motion simulation and sound synthesis.

Discussion

Two interesting points about HIMC that are worth mentionning are the interest of drumstick extremity control for motion simulation on the one hand, and on the other hand for sound synthesis.

HIMC is a means of reducing the dimensionality of motion capture data for controlling a physically respon-sive virtual character. Only 3D Cartesian trajectories are necessary compared to traditional high-dimensional body postures. HIMC is therefore characterized by a meaningfull, intuitive yet real-time scheme for physics tracking. One critical point in HIMC is whether or not to embed IK inside the physics control loop (cf. algorithm 2). A first approach could consist in separating the IK process from the physical simulation, resulting in the preprocess of end-effector targets before the simulation. Our choice with HIMC is to embed and combine IK and ID at the physical simulation level. It does not hypothesize any pre-processing while achieving real-time simulation. HIMC makes therefore room for the real-time modification of the desired end-effector tasks performed by the virtual character depending on physical interactions occuring during the simulation, for instance the membrane displacement fed back to the virtual performer.

HIMC is also carefully designed to fulfill some of the requirements of sound synthesis. Its physical nature provides any sound synthesis model with the essential contact parameters that give an adequate sound feedback to motion simulation. Moreover HIMC 's capability to control and modify in real-time end-effector tasks is important in the case of percussion sound synthesis. Physics-based sound synthesis of percussive sounds are generally based on simulating vibrations. Particularly among membranophones, membrane vibrations play an important role during the drumstick/membrane interaction and finally in the resulting sound. Vibrations depend on membrane physical properties (mass, tension, Young's modulus), as well as the applied force, and in return vibration modes will act on and disturb differently the drumstick. With HIMC, it is therefore possible to explore this phenomenon, for instance by exploring models accounting for the drumstick reaction towards membrane vibrations, although this was not implemented in the current simulation.

Interaction and sound synthesis

In this section, we present the explicit interaction of percussion motion simulations with a modal/mechanical model of a drum membrane. We then propose a general architecture for running simultaneously and asynchronously the physical simulation of percussion gestures, graphics and sound rendering processes, as well as handling their interaction in real-time. This architecture is effective for managing heterogenous data types characterized by various frame rates, and for specifying at the physics level the mapping between these modalities.

M S .q(t) + K S .q(t) = S T .f sound with : S = [s 1 , ..., s N ], z = S.q M S = S T .M.S K S = S T .K.S

(5)

F s = k.[c(x 0 , y 0 , M, t)] α with : k ∝ F m c(x 0 , y 0 , t) = p(t) -d(x 0 , y 0 , M, t) α = f (φ) (6)
The factorization of the model governed by equation (4) leads to a solution of the form z = [s 1 , ..., s N ].q where q = [q i ] i∈ [1,N ] are sinusoidal oscillators (or modes) and s i = [s i,j ] j∈ [1,N ] are referred to as the modal shapes of the system. Including the latter expression in equation (4) which leads to considering the modal shapes [s 1 , ..., s N ] as an orthogonal basis. The expression of the initial problem yields an uncoupled equivalent formulation in this orthogonal basis, equation [START_REF] Bouënard | Hybrid Motion Control combining Inverse Kinematics and Inverse Dynamics Controllers for Simulating Percussion Gestures[END_REF], where M S and K S are diagonal matrices. Equation ( 5) explicitly shows that the modal shapes of the system (matrix S) defines how external force f sound acts on the modes. Such a physically-based approach enables the direct modeling of contact force impacts. This force mainly depends on the state (displacement and velocity) of the colliding modal objects. A collision detection algorithm associated with the motion physics simulation retrieves the physical features f motion of any contact event produced when the drum is excited during the simulation.

It specifically provides information on impact location (x 0 , y 0 ), contact shape φ, contact compression c, as well as force amplitude F m . Contact compression is specifically expressed as the difference between drumstick and membrane vertical displacement (resp. p and d), with d depending on the impact location and membrane properties M . This contact information can be put in correspondence to the previously presented inputs to the modal/mechanical model of the drum membrane according to a direct one-to-one mapping. Equation ( 6) details a possible mapping following Hertz's contact law. Force profile F s acting on the drum model depends both on a force stiffness term k and compression c. Force stiffness k is determined from force amplitude F m during the simulation, whereas c is powered by a term α that characterizes the contact geometry (φ).

Asynchronous client-server architecture

Multimodal processes such as graphics and sound rendering are fundamentally different, and achieving a realtime interaction between the two can be hazardous. The first difficulty that appears when managing such media is their discrepency in time constants: graphics rendering is usually admitted effective at about 33Hz, whereas sound rendering needs a much higher sampling rate around 44kHz. Moreover, the graphics rendering is the visible layer of a far more demanding process, that of the physical simulation, requiring the handling of two other different time rates: the original motion capture time rate (250 Hz) and the time step of the physical simulation (10000 Hz). A solution could consist in running synchronously every manager at the fastest time rate (sound), but such an approach would fall short in real-time considerations.

We propose an asynchronous client-server scheme for handling the interaction between the physics, graphics and sound managers as depicted in figure 7. We adopt the Open Sound Control (OSC ) communication Fig. 8 Application: simulation of timpani motion using the HIMC control mode and its interaction with the modal sound synthesis model. The virtual character is controlled by the specification of drumstick extremity trajectories (yellow spheres). An exerpt of a timpani improvisation including various motion and rythm variations is presented, consisting of five percussion strokes with corresponding impact locations on the drum membrane as well as impact forces and resulting sound waveforms.

protocol 2 which is traditionally used for allowing the real-time exchange of multimedia data flows. The physical simulation and graphics rendering are embedded in OSC clients that are registered to the interaction and sound synthesis managers defining OSC servers. Such architecture allows for the asynchronous interaction of motion simulation, visual and sound renderings, namely by sending contact information to sound synthesis processes when needed. This also enables the possible distribution of the different managers on distinct platforms, thus reducing the computational cost of each manager and its impact on the others. The asynchronous exchange of data is materialized here by the mapping between the output of motion synthesis and the input of sound synthesis. Events produced during the physical simulation are detected by the interaction manager (collision detection). It is then in charge of processing and propagating these information through the OSC protocol so that the graphics and sound managers can react and produce the corresponding sound and visual outputs.

Application: simulation of a timpani improvisation

An application of our motion control simulation and interaction with modal synthesis is presented in this section, with a special focus on timpani percussion motion (figure 8). An improvisation made of various motion 2 Open Sound Control, http://www.opensoundcontrol.org variations (legato, tenuto, accent, vertical accent and staccato) and rythmn variations of timpani motion is simulated and shown in the corresponding animation (Online Resource 1).

The 3D rendering of the timpani percussion instrument is the visible representation of the modal/mechanical model of the drum membrane, with which the virtual percussionist can interact. The drum model is characeterized by a membrane diameter of 23" and has been modeled using Modalys3 . The motion control and simulation using ODE includes a collision detection module, which once triggered by a collision between drumsticks and the drum membrane sends contact information (impact location, contact shape and compression, force amplitude) both to graphics and sound managers. In this application, motion/interaction and sound managers have been deployed on two different machines following our system architecture, cf. figure 7. This results in the real-time visualization of simulated beat impacts on the drum membrane, as well as corresponding sound feedback.

Figure 8 shows an exerpt of percussion impacts, with their corresponding impact locations on the drum membrane as well as impact forces and resulting sound waveforms. Motion control is performed using HIMC and implies only the specification of drumstick extremity trajectories (yellow spheres seen in screenshots). The specification of drumstick extremity trajectories facili-tates the edition of the motion to be simulated, since the control of whole upper-body motion is reduced to two 3D position trajectories. Moreover, HIMC control mode from drumstick trajectories allows for the specification of various impact locations on the drum membrane, and therefore produces adequate variations of sound rendering. Figure 8 also shows the capability of our system to take into account varying impact forces. The first three percussion impacts show an increase in force impacts with an adequate increase in sound amplitude. Specifically, on the third impact a force of 106.9N is generated, and as a result a much greater waveform amplitude is synthesized compared to the other impacts. The next impact is characterized by a lower force amplitude of 65N, but takes advantage of the resonance of drum vibrations created by the previous impact to produce a medium sound amplitude, compared to the fifth impact with a similar force (62.3N), but with a lower sound amplitude.

Conclusion and Future Works

In this paper, we proposed a physically-enabled environment in which a virtual percussionist can be physically controlled and interact in real-time with physicsbased sound synthesis processes. The control of the virtual character uses motion data recorded during real percussion performances, therefore guaranteeing to maintain the main characteristics of human motion data while keeping the physical coherence of the interaction with the simulated instrument. For the range of our timpani expressive gestures, we have shown that the motion control method is efficient and robust. This can be explained by the incorporation within our control scheme of constraints statistically computed from motion captured data, that is biomechanical parameters and joint limits. HIMC control mode, combining IK and ID controllers, leads to a more accurate and controllable simulation as regards to drumstick extremity trajectories. The generation of consistent poses over time is also ensured by the dynamic tracking of these trajectories. Finally, driving the virtual character using such data as input provides users with a more intuitive way of composing the motion to be simulated, by editing and manipulating these trajectories as a gesture partition. We also proposed a modal/mechanical model of a drum membrane with which the physicsbased model of the virtual percussionist can interact. Finally, our asynchronous client-server architecture takes advantage of motion and sound physical formulations, generating in real-time virtual percussion performances.

This framework has been implemented in a modular way, so that we can substitute each module dedicated to motion or sound synthesis. In the near future, other more elaborate schemes of inverse kinematics will be used, in particular those exploiting learning of motion captured data. A major work will also involve the extension of HIMC to full-body motion synthesis to the light of recent achievements. Balance strategies similar to [START_REF] Macchietto | Momentum Control for Balance[END_REF] could therefore be used for simulating our virtual performer playing with a set of several percussion instruments, while keeping the effectiveness of HIMC for simulating upper-body motion. This would also allow to explore a large variety of percussion gestures and also different styles of percussionists. Among other perspectives of such work is the improvement of the interaction between motion simulation and sound synthesis. This can involve the design of a more powerful collision detection module. If we consider the physical simulation of the instrument, it might be interesting to include mechanical interactions, such as hammered or plucked interactions, or even to model the mechanical structure of a finger or a hand. We also plan to extend the mapping possibilities beween motion parameters and sound input parameters, and to develop a user interface for editing percussion motion in the 3D Cartesian space. This would lead to the possibility of building a new pedagogical tool dedicated to percussion performance analysis and simulation.

Fig. 1

 1 Fig. 1 Pipelines used for motion control: (1) Inverse Kinematics (IK ); (2) Inverse Kinematics + Inverse Dynamics (HIMC ); (3) Inverse Dynamics (ID). Commented dashed lines illustrate the data used for each pipeline.

  !"#$%&'())*+%,-./$0*'())*+%,-

Fig. 2

 2 Fig. 2 System architecture and multimodal outputs. The Motion Simulation step involves a Motion Capture Database and results in the Visual Feedback. The Interaction expresses the mapping between the motion simulation and the Sound Synthesis, which results in the Sound Feedback.

Fig. 3

 3 Fig. 3 Virtual character model: merging of kinematic and dynamic representations. (a) The underlying kinematic representation is represented by the implicit limbs (white lines) linked by a set of joints (white spheres), among which arm end-effectors (yellow spheres) are of special interest in HIMC as they represent the extremities of the drumsticks. (b) The underlying dynamic representation is characterized by explicit limbs (red capsule-shaped rigid bodies) linked by mechanical joints (blue spheres).

Tab. 1

 1 Muscular-skeleton model: body mass percentages, joint ellipsoid parameters. Body m [%] Joint l [λ, (e X , e Y , e Z )] , (0.337, -0.684, 0.647) Pelvis 18 8.9, (-0.889, -0.004, 0.458) Up. legs 14 4.4, (-0.311, -0.729, -0.609) Lo. legs 5 Wrist 19.9, (0.231, -0.932, 0.278) Feet 1.5 9.1, (0.003, -0.285, -0.958) 2.1, (-0.973, -0.222, 0.063)

Alg. 1

 1 Quaternionic joint limits. In Q Joint quaternionic trajectory Out (λ, e) Eigen values and axes l Joint limits (bounding ellipsoid) -Mean of Q q ← mean(Q) -Motion transposition in the tangent space of q Q * ← log(q * Q) -Eigen values and axes (λ, e) ← SVD(Q * ) -Joint limits: bounding ellipsoid l ← ellipsoid(λ, e)

(Fig. 4

 4 Fig.4Hybrid physics-based motion control and synthesis. The Motion Database is used for the physics modeling of the virtual character, as well as for expressing two modes of Control. The two modes correspond to pipelines (2) and (3) depicted in figure1. These allow for the physics-based motion tracking, (3) in the Joint Space (ID) from angular quaternionic trajectories q t , or (2) in the Drumstick Space (HIMC ) from end-effector trajectories X t . HIMC involves the combination of inverse kinematic (IK ) and inverse dynamic controllers (ID).

Fig. 5

 5 Fig. 5 Comparison of captured (black) and simulated trajectories for several percussion strokes, using IK (green), ID (blue) and HIMC (red) -(a-b-c) wrist, elbow and shoulder flexion angles, (d-e-f) drumstick extremity, wrist and elbow height positions.

  Figures 5(a), 5(b) and 5(c) present the comparison between

Fig. 6

 6 Fig.6(a) Depth/height phase trajectories of drumstick extremity for 5 motion variations, from left to right: legato, tenuto, accent, vertical accent and staccato. (b) Comparison of root mean square error (RMS-E ) and standard deviation (RMS-SD) between captured and simulated drumstick trajectories for various motion variations, using the two motion control schemes: ID and HIMC. RMS-E (blue for ID, red for HIMC ) and RMS-SD (black) are obtained from 100 simulation trials for each motion variation (from top to bottom: legato, tenuto, accent, vertical accent and staccato).

Tab. 2

 2 Mapping parameters between f motion and f sound . f motion f sound Impact location (x 0 , y 0 ) Membrane inner properties Contact shape (φ) (M : e.g. size, tension) Contact compression (c) Force amplitude (Fm) Force profile (Fs) 4.1 Motion-sound physics mapping In this paper, we consider the modal synthesis technique, which is an efficient way of representing the vibrations of resonating objects as the motion simulation of systems composed of masses connected with springs and dampers. A practical mapping between the force generated by the motion simulation f motion and the interaction force f sound acting on a modal/mechanical model of a drum membrane is shown in table 2. The drum membrane model is governed by the vertical motion z = [z i ] i∈[1,N ] of N mechanical oscillators of mass M = [m i ] i∈[1,N ] and spring coefficients K = [k i ] i∈[1,N ] , that can be excited by an external force f sound = [f i ] i∈[1,N ] according to equation (4). This model is characterized by coupled equations since matrices M and K are not necessarily diagonal. M.z(t) + K.z(t) = f sound

Fig. 7

 7 Fig. 7 Asynchronous server-client architecture of our system.

Modalys, http://support.ircam.fr/doc-modalys

Acknowledgments

This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (Discovery and Special Research Opportunity grants), the Agence Nationale de la Recherche and the Pôle de Compétitivité Images & Réseaux (France).


Abdullah Almaksour
email: abdullah.almaksour@irisa.fr

Eric Anquetil
email: eric.anquetil@irisa.fr

Improving Premise Structure in Evolving Takagi-Sugeno Neuro-Fuzzy Classifiers

Keywords: Incremental learning, Takagi-Sugeno, neuro-fuzzy

We present in this paper a new method for the design of evolving neuro-fuzzy classifiers. The presented approach is based on a first-order Takagi-Sugeno neuro-fuzzy model. We propose a modification on the premise structure in this model and we provide the necessary learning formulas, with no problemdependent parameters. We demonstrate by the experimental results the positive effect of this modification on the overall classification performance.

I. INTRODUCTION

Classification techniques appear frequently in many application areas, and become the basic tool for almost any pattern recognition task. The main problem in classification is to induce a classifier from a set of data samples. A large amount of samples is needed to set up and evaluate a classification system that can achieve a high accuracy, and it is practically very difficult to have such number of samples covering the whole feature space. Therefore, lifelong classifier adaptation becomes more and more an essential point. Moreover, in many application contexts, the classifier needs to take into account new unseen classes and to integrate them in the classification process, which increases the need for "evolving" classifiers. One good example of such applications is the use of online handwriting gesture classifiers which aim at facilitating interactions with computers using pen-based interfaces like whiteboards, tablet PCs, PDA...Etc. The main drawback in the current existing systems is that they are trained "offline" on a specific group of gestures and then implemented to operate without changing their structure during the use. This fixed structure does not allow the user to choose his own set of gestures or to add new ones to assign them to new interactive commands (according to his special needs), for example.

In our work, we aim at building a handwriting classifier, on-the-fly, from scratch and using only few data. Thus, the classifier will be incrementally adapted to achieve high recognition rates as soon as possible and to keep the system robust when introducing new unseen classes at any moment in the life-long learning process.

An incremental learning algorithm is defined in [?] as being one that meets the following criteria: it should be able to learn additional information from new data; it should not require access to the original data (i.e. data used to train the existing classifier); it should preserve previously acquired knowledge (it should not suffer from catastrophic forgetting, significant loss of original learned knowledge); and it should be able to accommodate new classes that may be introduced with new data. Many of the existing "incremental learning" algorithms are not truly incremental because at least one of the mentioned criteria is violated. These criteria can be briefly expressed by the so called "plasticity-stability dilemma"[?], which is that a system must be able to learn in order to adapt to a changing environment but that constant change can lead to an unstable system that can learn new information only by forgetting everything it has so far learned.

We can distinguish two main types of incremental learning algorithms: algorithms for parameter incremental learning and algorithms for structure incremental learning. The incremental learning of parameters can be considered as "adaptation" algorithm. The structure in such systems is fixed and initialized at the beginning of the learning process, and the system parameters are learned incrementally according to newly available data. Some examples of these systems are presented in [?] and [?].

Most of structure incremental learning algorithms are based on the principle of the ART clustering algorithm [?], such as [?], [?], [?]. The main problem of these systems is that they are sensitive to the selection of the vigilance parameter, to noise level in the training data and to the order in which training data are presented.

In an online incremental learning algorithm, the training set is not available a priori, since the learning examples come over time. Although online learning systems can continuously update and improve their models, not all of them are necessarily based on a real incremental approach. For some systems the model is completely rebuilt at each step of learning using all available data, they are systems with "full instance memory" [?]. On the other hand, if the learning algorithm modifies the model using only the last available learning example, it is called a system with "no instance memory" [?] [?]. A third category is that of systems with "partial instance memory", which select and maintain a reduced subset of learning examples to use them in the next learning step [?].

Evolving neuro-fuzzy models [?] have been successfully used in many modeling and classification problems during the last years. Several incremental learning algorithms have been proposed for the evolving structure identification and the recursive parameters learning (Denfis[?], Flexfis[?], eTS[?], eTS+[?] etc.). One of the most used neurofuzzy models is the First-order Takagi-Sugeno fuzzy model. It usually has a linear consequent part and its premise part can be learned from data (no expert is needed). These models can address problems with either single output or multi-output. It consists of a set of fuzzy rules of the following form:

Rule i : IF x is close to P i THEN y 1 i = l 1 i ( x), ..., y k i = l k i ( x) (1)
where l m i ( x) is the linear consequent function of the rule i for the class m. The Prototype P is defined by a center and a fuzzy zone of influence. A membership function must be defined in order to calculate the "closeness" degree between x and P (considering its center and its fuzzy zone of influence). In the existing approaches, the zone of influence has often a hyper-spherical form [?] [?]. It has then the same radius for all the feature space dimensions, and it can be represented by a diagonal covariance matrix with identical values on the diagonal. More sophisticated form is presented in [?] that allows the zone radius to have different values for the different dimensions (still diagonal covariance matrix but with different values on the diagonal). This ability results in hyperelliptical zones where the ellipses are still parallel to the feature space axes. In this paper, we go a step ahead in the structure of the fuzzy prototypes in first order TS models by allowing them to have a hyper-elliptical form non-parallel to the feature space axes. This form enables the model to take into consideration the correlations that can exist between the features, and is represented by normal (nondiagonal) covariance matrices. We calculate the prototype centers and covariance matrices in a recursive way with zero problem-dependent parameters. We use a membership function based on a well-known multivariate probability distribution function. By experimenting this structure on our handwritten gesture problem and other benchmark multi-class problems, we show its useful effect on the overall system output and the linear consequent learning. The rest of the paper is organized as follows: We describe in Section ?? the architecture of our neuro-fuzzy classifier. Then, the different elements of the used online incremental learning algorithm are detailed in Section III. We present our experimental results in Section IV before concluding in Section ??.

II. SYSTEM ARCHITECTURE

As aforementioned, our system is based on firstorder Takagi-Sugeno fuzzy inference system. It consists of a set of fuzzy rules of the following form:

Rule i : IF x is close to P i THEN y 1 i = l 1 i ( x), ..., y k i = l k i ( x) (2)
where l m i ( x) is the linear consequent function of the rule i for the class m:

l m i ( x) = π m i x = a m i0 + a m i1 x 1 + a m i2 x 2 + ... + a m in xn (3)
where n is the size of the input vector. To find the class of x, its membership degree β i ( x) to each fuzzy prototype is first computed. After normalizing these membership degrees, the sum-product inference is used to compute the system output for each class:

y m ( x) = r i=1 βi ( x) l m i ( x) (4)
where r is the number of fuzzy rules in the system. The winning class label is given by finding the maximal output and taking the corresponding class label as response:

class( x) = y = argmax y m ( x) m = 1, .., k (5)
The membership degree can be calculated in many ways. For hyper-spherical or axes-parallel hyperelliptical prototypes, the membership degree can be computed depending on the prototype center µ i and the radius value σ i (the same value in all the dimensions for the former, and different values for the latter). In this case, the Gaussian membership function is generally used. The value of β i ( x) can then be computed as follows:

β i ( x) = n j=1 exp(- x -µ i 2 j 2σ 2 ij ) (6)
It must then be normalized as follows:

βi ( x) = β i ( x) r j=1 β j ( x) (7)
In our model, where the fuzzy influence zones of the prototype take a rotated hyper-elliptical form, the membership degree is computed by the prototype center µ i and its variance-covariance matrix A i :

A i =     σ 2 1 c 12 ... c 1n c 21 σ 2 2 ... c 2n ... ... ... ... c n1 c n2 ... σ 2 n     i (8)
where c 12 (= c 21 ) is the covariance of x 1 and x 2 , and so on. In order to calculate the membership degree, we use the multivariate Cauchy probability distribution. The value of β i ( x) is computed in our model as follows:

β i ( x) = 1 2π |A i | 1 + ( x -µ i ) t A -1 i ( x -µ i ) -n+1 2 (9) III. ON-LINE INCREMENTAL LEARNING ALGORITHM
The incremental learning algorithm of our model consists of three different tasks: the creation of new rules, the adaptation of the existing rule's premises, and the tuning of the linear consequent parameters. These three tasks must be done in an online incremental mode and all the needed calculation must be completely recursive. We will describe in this section these different parts of the learning algorithm.

A. Incremental clustering

When introducing a new training sample in an online learning mode, it will either reinforce the information contained in the previous data and represented by the current clustering, or bring enough information to form a new cluster or modify an existing one. The importance of a given sample in the clustering process can be evaluated by its potential value. The potential of a sample is defined as inverse of the sum of distances between a data sample and all the other data samples [?]:

P ot k ( x(k)) = 1 1 + k-1 i=1 x(k) -x(i) 2 (10)
A recursive method for the calculation of the potential of a new sample was introduced in [?], which made this technique a promised solution for any incremental clustering problem. The recursive formula avoids memorizing the whole previous data but keeps -using few variables -the density distribution in the feature space based on the previous data:

P k ( x(k)) = k -1 (k -1)α(k) + γ(k) -2ζ(k) + k -1 (11)
where

α(k) = n j=1 x 2 j (k) (12) γ(k) = γ(k -1) + α(k -1), γ(1) = 0 (13) ζ(k) = n j=1 x j (k)η j (k), η j (k) = η j (k-1)+x j (k-1), η j (1) = 0 (14)
Introducing a new sample affects the potential values of the centers of the existing clusters, which can be recursively updated by:

P k (µ i ) = (k -1)P k-1 (µ i ) k -2 + P k-1 (µ i ) + P k-1 (µ i ) n j=1 µ i -x(k -1) 2 j (15)
If the potential of the new sample is higher than the potential of the existing centers then this sample will be a center of a new cluster and a new fuzzy rule will be formed in the case of our neuro-fuzzy model. So, the center of the new prototype µ r+1 = x k and its covariance matrix A r+1 = ǫI, where I is the identity matrix of size n and ǫ is a problemindependent parameter and can generally be set to 10 -2 .

B. Premise adaptation

This adaptation process allows to incrementally update the prototype centers coordinates according to each new available learning data, and to recursively compute the prototype covariance matrices in order to give them the rotated hyper-elliptical form. For each new sample x k , the center and the covariance matrix of the prototype that has the highest activation degree are updated.

The center coordination of the selected prototype is recalculated as follows:

µ i = (1 - 1 s i + 2 ) µ i + 1 s i + 2 ( x k -µ i ) ( 16
)
where s i represents the number of updates that have been already applied on this prototype. The covariance matrix is recursively computed as follows:

A i = (1 - 1 s i + 2 )A i + 1 s i + 1 ( x k -µ i )( x k -µ i ) t (17)
For practical issues, since the membership degree can be calculated using only A -1 (|A| = 1

|A -1 | ), and in order to avoid any matrix inversion, we use an updating rule for A -1 directly [?]:

A -1 i = A -1 i 1 -α - α 1 -α • (A -1 i ( x k -µ i )) • (A -1 i ( x k -µ i )) t 1 + α(( x k -µ i ) t A -1 i ( x k -µ i )) (18)
where a = 1 si+1 .

C. Linear consequent tuning

The tuning of the linear consequent parameters in a first-order TS model can be done by the weighted Recursive Least Square method (wRLS). Let Π i be the linear consequent parameters of the rule i:

Π i = π 1 i π 2 i ... π m i t (19)
where

π c i = [a c i0 a c i1 ...a c in ].
It can be recursively estimated as follows:

Π i = Π i + C i βi ( x k ) x k (Y k -x k Π i ), Π init = 0 (20)
C i = C i - βi ( x k )C i x k x t k C i 1 + βi ( x k ) x t k C i x k , C init = ΩI (21)
where Ω is a large positive number, and I is the identity matrix.

IV. EXPERIMENTAL RESULTS

A. Handwritten gesture recognition

We will particularly focus in our experiments on the rapidity of the performance improvement in the beginning of the incremental learning process, and on the stability and the recovery speed of the performance when introducing new unseen classes. We led the experiments on the "SIGN" database, which is a database of on-line handwritten gestures (figure ??). It is composed of 25 different gestures drawn by 17 different writers on Tablet PCs. Each writer has drawn 100 samples of each gesture, i.e. 1,700 gestures in each writer-specific dataset. The dataset (and additional information on the data collection protocol) can be found in [?]. Each gesture is described by a set of 10 features. The presented Figure 1: The used handwritten gestures results are the average of results of 17 different tests for the 17 writers. In order to get the results unbiased by the data order effect, we repeat the experiment for each writer 40 times with different random data orders and the mean results are considered. We used about half of the database for the incremental learning process and the rest is used to estimate the evolution of the performance during the learning process. Two fuzzy incremental learning models are compared in these experiments:

• Model I: evolving first-order TS classifier with parallel-to-axes hyper-elliptical prototypes, • Model II: our extended version with rotated hyper-elliptical prototypes (covariance between features are considered).

We introduce the 25 gestures in the beginning of the learning process. Two new samples from each class are presented to the system between each two consecutive evaluation points.

B. UCI repository datasets

Besides the SIGN dataset, we evaluate the presented neuro-fuzzy model on some benchmark problems form the UCI machine learning repository [?]. We have focused on problems with more than two classes. The learning process is done in an online mode.

• Letter : The objective is to recognize the input vector as one of the 26 English letters. Each letter is represented by a vector of 16 attributes extracted from raster scan image of the letter. The dataset contains 20000 instances. We use 1800 ones in the incremental learning process and the rest is used as evaluation dataset. We can note from Figure ?? that our enhanced neuro-fuzzy model outperforms the classic one. Thanks to the improved premise structure, the incremental learning process goes faster and the misclassification rate generally decreases between 10% 35% depending on the classification problem. We resume in Table ?? the results achieved by the proposed model at the end of the incremental learning process, compared to the classic one and also to two different batch classifiers (Multi-Layer Perceptron (MLP, with one hidden layer) and a K-Nearest Neighbor (K-NN, with K=5)). We note from the table that the presented evolving neuro-fuzzy classifier with a one-pass learning algorithm is even able to outperform in many cases some well-known batch classifiers.

V. CONCLUSION

An improvement on the premise structure of the first-order Takagi-Sugeno neuro-fuzzy classifier had been presented in this paper. The rotated fuzzy zones of influence in the presented model enable the premise "layer" to consider the correlations that may exist between input features and boost the overall performance of the classifier. An appropriate recursive premise adaptation method is coupled with a known incremental clustering technique and used with the recursive least squares method to learn the classifier in one-pass incremental mode. The proposed enhancement reduces the misclassification rate by 30% for our specific classification problem (handwritten gesture recognition), and these good results are also obtained on several benchmark classification problems.

Figure 2 :

 2 Figure 2: Performance evaluation during the incremental learning process (the horizontal axis represents the number of learning samples per class, and the vertical axis represents the misclassification rate)

Table I :

 I Misclassification rates (%)

		SIGN	Let.	Cov.	Veh. Dig.	Seg.
	Model I	6.1	23.0	21.0	20.4	3.0	9.0
	Model II	4.1	16.0	19.7	16.3	2.0	7.3
	MLP	7.4	21.2	17.4	18.6	6.1	4.6
	K-NN	4.6	20.9	18.5	28.7	2.3	8.0