Alexandre Bouënard

Marcelo M Wanderley

Sylvie Gibet

Virtual Gesture Control of Sound Synthesis: Analysis and Synthesis of Percussion Gestures

In recent years, the control of virtual instruments or sound-synthesis processes by natural gestures has become an important research field, both for building new audio-visual tools and for exploring gesture-sound relationships. Such multimodal and interactive tools typically present two advantages, on the one hand they provide realistic virtual instruments whose response can be compared to existing musical instruments, and on the other hand they give the possibility to vary characteristics of natural gestures, while ensuring a certain coherence between gesture and sound parameters.

In this paper, we present and evaluate a new framework for explicitly expressing the characteristics of natural percussion gestures used for modeling, controlling and finally synthesizing new percussion gestures. A preliminary analysis of pre-recorded gestures leads to the identification of significant parameters, and their evaluation using a classification approach. This analysis shows that a reduced-dimension representation of captured motion can be used to control a virtual character. Furthermore, the simulated gestures provide dynamical variables that can be used to control sound synthesis through a mapping-interaction process.

Introduction and Motivation

While playing a musical instrument, a musician establishes a more or less continuous interaction with the instrument. Such interaction is based on complex mechanisms, allowing the fine-tuning of the sound-producing gestures via sensorimotor loops, including audio, visual and haptic/proprioceptive feedback. This sensory information is directly influenced by the semiotic information contained in the musical phrases, and may change the motor commands that produce the gesture. Virtual musical instruments controlled by natural gestures try to realistically reproduce this sensorimotor situation with the aims of approaching real instrumental situations and exploring possible gesture-sound relationships. Reality may then be extended through new paradigms where users can interact in real-time with sound processes.

Traditional acoustic models provide a formal representation of the underlying physical mechanisms that are at the origin of the produced sounds. Such models have been proposed for a wide range of musical instruments, such as strings [START_REF] Bensa | The Simulation of Piano String Vibration: from Physical Models to Finite Difference Schemes and Digital Waveguides[END_REF], single-reed [START_REF] Kergomard | Elementary Considerations on Reed-instruments Oscillations[END_REF] or brass instruments [START_REF] Msallam | Physical Model of the Trombone Including Nonlinear Effects. Application to the Sound Synthesis of Loud Tones[END_REF]. These contributions imply more generally inversion processes, starting from the produced sound to obtain physical parameters for controlling virtual instruments models, which are themselves driven by instrumental gesture excitations. In fact, one of the main issues of the above approaches is to characterize the inversion processes, and especially to map the physical parameters of the virtual instrument with interaction gestures.

Previous works have focused on the analysis and modeling of interaction gestures for particular instrumental gestures, namely for taking into account the gestural signals that are responsible for the sound production. For example in the case of bowing gestures, related works have focused on the identification and use of interaction profiles [START_REF] Demoucron | On the Control of Virtual Violins: Physical Modelling and Control of Bowed String Instruments[END_REF][START_REF] Rasamimanana | Geste Instrumental du Violoniste en Situation de Jeu : Analyse et Modélisation[END_REF], as well as gesture following [START_REF] Bevilacqua | Continuous Realtime Gesture Following and Recognition[END_REF]. These works are generally related to the analysis of the considered instrumental gestures [START_REF] Dahl | Playing the Accent: Comparing Striking Velocity and Timing in Ostinato Rhythm Performed by Four Drummers[END_REF][START_REF] Rasamimanana | Effort-based Analysis of Bowing Movements: Evidence of Anticipation Effects[END_REF] for identifying gesture profiles of interest, but rarely address the modeling of the equivalent gesture actions that are at the origin of these profiles. Our work sensibly differs from these previous works in the sense that the proposed system produces the gestural signals that are responsible for the sound production, with a special focus on timpani gestures.

We propose to introduce a complete modeling of gesture, by designing a human-like character endowed with realistic and expressive behavior. In order to reproduce the main characteristics of the control exerted on a real instrument, and more specifically the efforts involved in the interaction, we adopt a physics-based approach for modeling both the 1 1 Introduction and Motivation !"#$"%&'()(*+#+), -$*+%,$"%&)+./0#$"% 12(3/(#$"%&45& 63(,,$70(#$"% 8+,#/)+&95%#:+,$, 9"/%.&95%#:+,$,

!"#$%&'& (%")*+&'& ,-)'-"./#)#

Fig. 1: Approach: gesture dimension reduction by extracting motion parameters, evaluation of motion parameters and synthesized motion data, and finally interaction between synthesized motion data and sound synthesis.

virtual performer and the sound-synthesis process. The main interest of a biomechanics-based modeling of gesture lies in the dynamical understanding of the mechanical links established between the instrumentalist and the instrument. Furthermore, this physical approach may also provide insights about the various parameters that are responsible for producing specific performances with a desired degree of expressiveness. These parameters may characterize for example the joint constraints in the form of angular limits or biomechanical parameters such as stiffness and damping of the joints. Finally, such a physics modeling approach allows to represent not only what occurs during the mechanical interaction between the instrumentalist and the instrument, but also the whole gesture production system, including the preparatory and the retraction phases that precede and follow the interaction process. In a similar way to sound-synthesis research that focuses on effect-centered (sampling and spectral modeling) or cause-centered (physical modeling) synthesis methods, the control of virtual characters involves either effect-centered (kinematics) [START_REF] Tolani | Real-Time Inverse Kinematics Techniques for Anthropomorphic Limbs[END_REF][START_REF] Kovar | Motion Graphs[END_REF]) or cause-centered (physics) [START_REF] Zordan | Dynamic Response for Motion Capture Animation[END_REF][START_REF] Jain | Optimization-Based Interactive Motion Synthesis[END_REF]) solutions. To take advantage of both methods, we define a cascaded combination of two inversion models (kinematics and physics) allowing the control of the virtual performer through a sensorimotor control loop from motion data of reduced dimension compared to related contributions [START_REF] Zordan | Interactive Dynamic Response for Games[END_REF]. This controller uses sensory information, for instance mallet trajectories or more generally visual or proprioceptive information, to compute the forces/torques that drive the physical model of the virtual performer.

However, without real data, inferring the forces and moments applied to joints and bones of the virtual character still remains a difficult inverse problem. As illustrated in Fig. 1, we propose to solve it by using motion captured data recorded during real performances. The idea is to extract from these data a segmented and reduced-dimension representation of motion, and then to edit and assemble these motion chunks in order to control the virtual character. One key issue of our work is therefore to propose an original methodology which evaluates a reduced-representation of the recorded motion data, and to demonstrate that this reduced-representation may be used to synthesize the movement through an inversion process. Concerning the reduced-dimension process, we observe that the trajectories of the mallet's end-extremity follow cyclic patterns which drastically change with the type of the attack depending among other factors on the type of the technique, the style of the instrumentalist, and the different playing modes. We make the hypothesis that these patterns can be considered as traces of the movement that contain most of the necessary features to reconstruct the movement. To go further, we propose to characterize these patterns by a limited number of parameters representing the extremities of the kinematic trajectories and their corresponding timing.The methodology is divided into two parts: first, it evaluates this reduced representation, thanks to a classification process. Second, the same classification method is applied on the synthesized gestures. In addition, we propose an interaction scheme between gesture and sound, so that the impact forces and the whole gestures produced during the simulations may be taken into account to control the sound synthesis process through a mapping strategy. We finally show how such models can be used to simulate virtual percussion performances, and evaluated using auditory and sound feedback.

This paper is organized as follows. Section 2 presents the analysis part of the work, by extracting a set of relevant parameters that characterize percussion performances, and by evaluating these parameters using a classification/recognition method. We then propose in section 3, first a method for modeling and controlling virtual gestures from reduced-dimension motion data, according to the results of the analysis, and second a general scheme for interacting with sound synthesis. Results are presented in section 4: they concern the evaluation of synthesized gestures compared to real ones, as well as the evaluation of virtual percussion performances, from visual and auditory output. We conclude and draw perspectives of this work in section 5.

Percussion Performance Analysis

In this section, we present an original analysis methodology to evaluate the relevance of gesture parameters extracted from recording percussion playing techniques, and especially timpani performances.

Timpani Data Collection

Timpani Playing Techniques

Timpani-related equipment is mainly composed of a bowl, a head and mallets. In general, timpanists have to cope with several timpani (usually four) with bowls varying in size. As for timpani mallets, they consist of a shaft and a head that can be designed in a wide range of lengths, weights, thicknesses and materials [START_REF] Cook | Teaching Percussion[END_REF].

Timpani playing is characterized by varied playing techniques. We focus here on three main techniques: percussion grips, gesture variations and beat impact locations. First, there are two main strategies for holding mallets: the French grip (also called "thumbs-up") and the German grip (or "matched" grip). Timpani players also use several variations of a gesture: we selected five of them (Legato, Tenuto, Accent, Vertical Accent and Staccato) which are associated with related sound effects, typically characterized by timbre and resonance differences. Players also use three distinct locations of impacts: One-third of membrane radius, Center and Rim of the membrane. The most used is the One-third location, while Center and Rim are used less often. These playing techniques define the timpani gesture typology used for building a motion capture protocol from which our data collection is derived.

Motion Capture Protocol and Database

We captured the motion of several performers using a Vicon 460 system based on Infra-Red camera tracking, as well as a standard DV camera allowing the synchronization of both captured gestures and sounds [START_REF] Bouënard | Enhancing the Visualization of Percussion Gestures by Virtual Character Animation[END_REF]. Timpanists wore a lycra suit fitted with markers placed according to the marker position setup of Vicon's Plug-in Gait. Mallets have also been augmented with markers, so that beat impacts can be reliably retrieved. In this study, we have also restricted the drumset to an unique timpani.

Three percussionists were asked to perform a pre-defined capture protocol consisting of a single stroke roll for each gesture variation. For each gesture, performers were asked to change the location of the beat impact according to the three locations previously described. In total, thirty examples of timpani exercises were performed for each percussionist (each with six beats per hand). Performers had various musical backgrounds and playing characteristics. Main differences included their degree of expertise (Teacher, Master student or Bachelor student), their preferred grip used (French or German), dominant (Left or Right) hand, and gender.

One of the main issues using such hardware solutions is the choice of the sampling rate used for the capture of percussive gestures, mainly because of the short duration of the beat impact [START_REF] Wagner | Analysis of Drumbeats -Interaction between Drummers, Drumstick and Instrument[END_REF]. With high sampling rates (500 Hz and above), one can expect to more accurately retrieve beat attacks, but the capture space range is significantly reduced so that it may be difficult to capture the whole body of the performer. For this project, a compromise was chosen by setting the cameras at 250 Hz, allowing both full-body capture as well as a reasonably high sampling rate for reliably capturing mallet beat impacts.

Analysis

The analysis of timpani performance gestures collected in the database focuses on the study of mallet extremity trajectories, the result of the percussionist action during musical performance. An intuitive hypothesis consists typically in stating that percussionists more specifically control the motion of mallets over time, even if studies have underlined strategies involved in the motion of shoulders, elbows and wrists [START_REF] Dahl | Playing the Accent: Comparing Striking Velocity and Timing in Ostinato Rhythm Performed by Four Drummers[END_REF][START_REF] Bouënard | Analysis of Percussion Grip for Physically Based Character Animation[END_REF]. The aim of this section is to provide a quantitative analysis of this hypothesis, as well as to give relevant parameters that are of particular interest for the modeling and synthesis part of our work (section 3). In order to highlight this asumption, we conduct below a study based on mallet height trajectories.

Method

The parameters used to characterize mallet extremity trajectories are local extrema extracted from position trajectories during beat-to-beat preparatory phases, as well as local extrema from velocity and acceleration beat-centered profiles as represented on Figure 2 (a,b) for a particular legato stroke for the French grip. Variations of mallet trajectories (position, velocity and acceleration) from which local extrema are extracted are given in Figure 2 (c-e) for both grips and for each gesture variation.

The evaluation of such a parameterization is conducted by a quantitative analysis based on a classification/recognition scheme. The relevance of these parameters is measured using the Support Vector Machine (SVM) classification method, with the use of Radial Basis Functions (RBF) as kernel functions. The scope of this evaluation initially concerns parameters related to the type of percussion grip, then several gesture variations. For each case, a combination of parameters is chosen and forms a reduced-dimension and refined data set of the motion capture database, on which the classification/recognition scheme will be based on. This refined data set is then divided into two sub-sets, an excerpt of each is randomly extracted for representing determined classes that will train a classifier, whereas the remaining data will consist in queries submitted to the classifier. The relevance of the selected parameters will be estimated accordingly to their recognition success by the classifier.

In this quantitative study, the determined classes to be recognized will typically correspond to two classes for percussion grips, and five classes for gesture variations, so that a multi-class SVM approach has been adopted. The Tab. 1: Statistical features computed on mallet trajectories and extracted extrema: vertical Mean, Variance and Range of Motion (RoM), as well as the average temporal (in percentage of gesture's duration) and spatial characterization of extracted extrema presented in Figure 2. classical approach to multi-class SVM is to construct several binary one-versus-rest classifiers, each being characterized by a decision hyperplane to discriminate the corresponding class to the others [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF]. The approach adopted in this work is to characterize the multi-class problem by a piecewise decision hyperplane, enabled with a decision function that can train data without errors compared to a set of one-versus-rest classifiers as argued in [START_REF] Weston | Support Vector Machines for Multi-class Pattern Recognition[END_REF]. The presented classification analysis has been conducted based on the Spider library [START_REF] Weston | The Spider Library[END_REF], and uses default parameters of RBF kernel functions.

Statistics / Grips Mean [m] Variance [m] RoM [m] E 1 [%/m] E 2 [%/m] E 3 [%/m] French

Percussion Grips

We initially focus on the analysis of the influence of French and German grips on mallet trajectories. Quantitative features (Table 1) processed on the vertical component of the extremity of the mallet show that French grip-related data performs the same timpani gesture with much more amplitude. The mean of the mallet extremity height is about twenty centimeters higher than its German grip counterpart, with a variance twice higher. This fact is strengthened by the vertical range of motion (RoM) of the extremity of the stick for French grip-related data that is about twice higher than for German grip data. Moreover the mean of mallet extremity height for German grip data shows that the extremity of the stick is in average closer to the timpani membrane. Specific local extrema can also be observed during preparatory gestures. Figure 2 presents examples of the vertical component of the preparatory gesture between two beat attacks, and the identification of three characteristic extrema denoted E 1 , E 2 and E 3 . These extrema are temporally (temporal apparition in percentage of gesture's duration) and spatially characterized in Table 1.

Vertical extrema E 1 , E 2 and E 3 are temporally equi-distributed for the French grip-related data showing a continuous preparatory gesture, whereas local extrema for German grip-related data denote three distinct parts. In the latter case, E 1 corresponds to the reaction to the previous beat attack, between E 1 and E 2 the extremity of the mallet seems to seek a rest position (during more than the half of the whole movement duration) just above the timpani membrane, while E 2 and E 3 correspond to the amplitude of the German grip-related data around the following beat attack. Table 1 also quantifies the effect of the french and german grips on the vertical amplitude of the extrema.

French and German grips influence the spatial and temporal characteristics of the extracted extrema from height trajectories of the extremity of the mallets. In order to evaluate the relevance of such parameters for discrimining percussion grips, we chose to use a classification/recognition process. The training set is randomly composed of only 1/8 th (68) of the total number of available data (540), and the query set is composed of the remaining data.

The high recognition rates of these extrema (superior to 97% in average), as shown by the confusion matrix in Table 2, indicate that such a parameterization is well-suited for characterizing the effect of percussion grips on the height trajectories of mallets.

Tab. 3: SVM recognition percentage: French grip gesture variations using the combination of mallet velocity and acceleration extrema presented in Figure 2 . Tab. 4: SVM recognition percentage: German grip gesture variations using the combination of mallet position and acceleration extrema presented in Figure 2 .

Training
Training

Gesture Variations

Following the same methodology, the considered set of extracted parameters is enhanced for taking into account more timpani playing techniques, namely the different gesture variations available in the motion capture database for each percussion grip sub-group, as described in section 2.1.1. These additional parameters are composed of the previously presented mallet height extrema (Figure 2(a)), as well as local extrema extracted from mallet height velocity and acceleration beat-centered profiles (Figure 2(b)). Beat-centered profiles are the truncation of motion to a window of 120 ms, 60 ms before and after the beat impact. In both situations for discriminating gesture variations inside percussion grips, the training set is randomly composed of only 1/4 th (45) of the total number of available data (180), and the query set is composed of the remaining data. The discrimination of gesture variations related to the French grip is achieved by considering the combination of velocity and acceleration extrema. The results obtained with such a parameterization are presented by the confusion matrix in Table 3, with an average recognition rate superior to 94%.

As for gesture variations related to the German grip, the discrimination is achieved by combining both position and acceleration extrema. The results obtained with such a parameterization are presented by the confusion matrix in Table 4, with an average recognition rate superior to 93%.

Discussion

Regarding the nature of the spatio-temporal parameters highlighted through this analysis, we believe that they are relevant according to the percussion task, since both the height and the timing of gestures are highly controlled during percussion performances. The introduction of velocity and acceleration characteristics for discriminating between gesture variations among French and German grip-related data can be interpreted as the parameterization of the musical expressiveness intrinsically related to each gesture variation.

More interestingly is the use of different parameter combinations (velocity/acceleration for French grip gesture variations, and position/acceleration for German grip gesture variations), related to the statistical features presented in section 2.2.2. As shown in Table 1 the mallet range of motion is much more constrained for the German grip, attesting the importance of position parameters for discriminating gesture variations in this case. Conversely, the continuous and equi-distributed preparatory gesture shown for the French grip underlines a less stiff constraint on mallet position, so that the way velocity is involved is predominant for discriminating gesture variations in this particular situation.

!"#$#%&'( )*+,-., !"#$#%&'( /-'#01%-.,

!"#$%&'(%)%&**+%#)&%,,"& -.$/.,%0$1",")%#**-%2",

X S X T Θ T G, F G τ D -1 K -1 Θ S
, ΘS S P S K B Fig. 3: Virtual gesture control: physical control of the musculo-skeleton model of the virtual performer from mallet extremity trajectories.

Acceleration parameters are important features for both grips as they may be related to the dynamics of the strike.

Virtual Gesture Modeling and Interaction with Sound Synthesis

In this section, we first present the physics-based modeling of the virtual character and the way the mallet extremity trajectories can be used to control the performance, as stated in the previous section. Then we derive an interaction mechanism to make possible the virtual performer control sound synthesis.

Virtual Gesture Modeling and Control

The virtual performer is represented by a physical model, composed of two skeleton layers: a physics layer (S P ), and a kinematics layer (S K ), as depicted in Figure 3. The skeleton physics S P is composed of a system of rigid body segments (limbs) linked by mechanical joints. These latter form the biomechanical properties B of the model, constraining the allowed rotational degrees of freedom between the articulated bodies. Each rigid body is characterized by physical properties, such as mass and inertia. S P is put into motion by a forward dynamics scheme according to Newton's motion laws, where rigid bodies acceleration and angular velocities are inferred by the application of forces and torques. Such a musculo-skeleton representation of the virtual performer can therefore be responsive to any application of forces and torques during a simulation. This is the starting point of the motion control formulations presented here, which aim at applying the right physical torques τ on S P only by the specification of end-effector cartesian targets describing the motion of mallet extremity trajectories X T . This motion control scheme inferring physical torques from kinematic trajectories is characterized by the inversion of two cascaded problems, the inverse kinematics (K -1 ) and inverse dynamics (D -1 ) formulations. In a first time, given the target trajectory of the mallet extremity X T and its state X S , the inverse kinematics scheme processes automatically a kinematic pose Θ T that realizes the target X T , and representing the orientation of each joint composing S K (and specifically the arms). In a second time, given the kinematic pose Θ T and joints current state (Θ S , ΘS ), the inverse dynamics scheme computes automatically the torques τ to be applied on the rigid bodies composing S P . Such cascaded strategy therefore achieves the physical control of the virtual percussionist by the reduced-dimension and intuitive specification of mallet extremity trajectories available in our database. The method used to link both inverse schemes is detailed in [START_REF] Bouënard | Hybrid Motion Control conbining Inverse Kinematics and Inverse Dynamics Controllers for Simulating Percussion Gestures[END_REF][START_REF] Bouënard | Synthesis of Music Performances: Virtual Character Animation as a Controller of Sound Synthesis[END_REF].

Interaction with Sound Synthesis

In this paper, we are interested in physically based sound models, which produce sound that respond to physical input parameters. One of the main interests of this physical approach is its interactivity and ease in mapping gesture to sound control. In our work, we adopt the modal synthesis formalism [START_REF] Adrien | The Missing Link: Modal Synthesis[END_REF], which describes the vibrational model of the M.

z(t) + K.z(t) = F S (1)
Such a model is typically characterized by physics parameters S including the size, tension, mass and Young's modulus of the membrane. S also includes other parameters related to sound synthesis such as the number of modes taken into account as well as dispersion terms. Finally, a force density F S is exerted by the mallet on the drum membrane model. This force represents the excitation of the drum and is responsible for the selection and weighting of the modes involved in the sound production. The interaction between gesture and sound synthesis can be represented by the gesture outputs (G, F G ) mapped to the sound inputs (S, F S ), as shown in Figure 4.

Gesture outputs are characterized by parameters (G, F G ) revealing the nature of the impact during the simulation. G includes the impact location (x 0 , y 0 ) and velocity, as well as the attack angle Φ of the mallet on the membrane. F G is an impact force which results from a collision detection between the mallet and the drum membrane during the gesture physics simulation. Physical models of impacts can be a source of knowledge for determining the excitation force F G , such as the Hertz's law of contact as described by Equation (2),

F G (t) = k.[c(x 0 , y 0 ,t)] α with c(x 0 , y 0 ,t) = p(t) -ω(x 0 , y 0 ,t) (2)
where p(t) is the displacement of the mallet's head, and ω the vertical component of the displacement of the membrane at the impact point (x 0 , y 0 ). The parameter k characterizes the force stiffness and can be determined by the force impulse F G coming from the gesture synthesis. The exponent α is determined by the contact geometry between the mallet and the membrane, and can be affected by the contact angle Φ.

Gesture parameters G and F G are then used to control sound parameters through a gesture-sound mapping. The excitation force F S (x 0 , y 0 , x, y,t) is limited in time and distributed over a certain width. Some models assume that the time and space dependence can be separated [START_REF] Chaigne | Numerical Simulation of Xylophones. I. Time-domain Modeling of the Vibrating Bars[END_REF], and that there exists a relationship between F G and F S , Equation (3),

F S (x, y, x 0 , y 0 ,t) = F G (t).H (g(x, y, x 0 , y 0 )) (3)
where g(x, y, x 0 , y 0 ) is a spatial window that accounts for the contact width of the mallet with the membrane. Other models consider that the spatial dependency is negligible, and consequently the excitation force F S can be defined as the impact force F G [START_REF] Avanzini | Physical Modeling of Impacts: Theory and Experiments on Contact Time and Spectral Centroid[END_REF]. This mapping-interaction approach gives the possibility to simulate real physical mechanismsboth for gesture control and sound synthesis-as well as propose variations that affect the numerical simulations. In particular, sound parameters S can be controlled from gesture parameters as a means of extrapolating different gesture output and sound input parameters, as well as various mapping schemes, thus allowing us to explore the relationship between gesture and sound. This flexibility can be interesting for instance in virtual reality appications.

Simulation Results

The gesture modeling and control system is implemented by using the Open Dynamic Engine (ODE) library [START_REF] Smith | Open Dynamics Engine[END_REF] which simulates elementary rigid body dynamics on which our control algorithms are built. This library includes an integrated collision detection scheme with friction. The model of the virtual character is composed of 19 bodies and 19 ball-and-socket joints, each with three degrees of freedom. The control model is only applied on the two hand-arm systems with a stick-mallet. It uses advanced joint types that allow us to specify the stiffness and damping coefficients that characterize the shoulder, elbow and wrist joints. We call these joints' coefficients, as well as the links' parameters (size, mass), biomechanical parameters. They can be changed for simulating variations of the gestures. The collision detection module of ODE produces impact forces F G (t) which are instantaneous impulse functions applied on the vibrating surface at some specific point (x 0 , y 0 ). We use this position as well as the initial velocity ċ0 of the mallet's head at this point, at the time where it comes in contact with the surface. We may also use the motion of the mallet before the impact, as it is simulated by the physical model.

Quantitative results are first presented, by analyzing the synthesized gestures, using the same methodology as the one described in section 2. We then qualitatively evaluate the virtual percussion performances, using visual and auditory feedback.

Evaluation of Synthesized Gestures

In order to evaluate the quality of the synthesized gestures generated by our framework, we first compute the error made on upper-body member trajectories between synthesized and real gestures. Figure 5 presents the root mean squared error (RMS) computed on two hundred simulation trials compared to recorded data specifically for French grip and for each gesture variation. In each case a gesture unit representing the motion of the mallet extremity has been chosen as the target trajectory to be reproduced. Figure 5 shows for each gesture variation the errors resulting from the synthesis, not only for the mallet extremity trajectories, but also for the trajectories of the wrist, elbow and shoulder joints. Among the simulated gesture variations, one can identify differences in the RMS errors. These can be explained by the fact that the biomechanical parameterization B of the physical model (mechanical joints, Figure 3) has been kept constant for all simulations, while dedicated biomechanical parameters for each gesture variation may lead to more accurate results. However, the low errors seen on mallet extremity and joints trajectories (less than 1cm in average) attest to the ability of our motion control scheme to accurately reproduce the different gesture variations. Second, we evaluate the timing of velocity and acceleration of synthesized gestures. Although we only specify mallet extremity position over time in our motion control scheme (section 3.2), we obtain interesting results regarding velocity and acceleration profiles. In an analogous manner to the analysis evaluation work made in section 2.2, mallet position, velocity and acceleration parameters at extremity points (as shown in Figure 2) have been extracted from two hundred simulation trials for each gesture variation. The same classification/recognition approach (SVM method with radial basis functions) was used to evaluate the recognition of the French-grip gesture variations, i.e. a combination of velocity and acceleration parameters, and their specific time values. The training set is composed of parameters related to motion capture data, whereas the query set is represented by the parameters related to the simulation trials. Table 5 presents the confusion matrix of the recognition results. They show that gesture variations are similarly well recognized, with results comparable to the those obtained in the analysis study (Table 3). This result shows that synthesized gestures can be characterized (both spatially and temporally) in the same way as captured motion data.

Virtual Percussion Performance: Visual and Auditory Feedback

As for the sound synthesis, we use the Modalys implementation [START_REF] Adrien | The Missing Link: Modal Synthesis[END_REF][START_REF] Modalys | IRCAM[END_REF]. A first difficulty is defining the exact shape of the function F S (x 0 , y 0 , x, y,t) since no study makes available physics knowledge about timpani force profiles. We use as excitation forces with Modalys simple mathematical functions such as the ADSR (attack, decay, sustain, release) function, that has the advantage to clearly define the gesture-sound interaction at the physical level. In addition, an interface based on an asynchronous client-server architecture that uses the OSC [START_REF] Freed | Features and Future of Open Sound Control version 1.1 for NIME[END_REF] communication protocol makes possible the modification of the drum membrane properties (mass, tension, dispersion), as well as the multimodal integration, interaction and synchronization of visual and auditory output within our framework.

With such a simulation framework, we have synthesized various gestures performed by the virtual character, by considering as input of the gesture control model the trajectories of the mallet's head. We simulated various gesture profiles (video examples of simulations are available at http://www.youtube.com/SamsaraUBS), according to the nature of the grips and to the five playing variations [START_REF] Bouënard | Advantages and Limitations of Simulating Percussion Gestures for Sound Synthesis[END_REF]. This leads to synthesized gestures which are deeply influenced by the percussionist style techniques and the playing modes. The animations can provide the user an appropriate visualization of the simulated percussion gestures with the possibility to modify the 3D rendering of the virtual character, or offering different views of the mallet trajectories or beat impacts. Figure 6 shows particular results for the simulation of Legato beat impacts for the French grip. It highlights (top) the comparison between real and synthesized trajectories (mallet extremity and joints positions), as well as the produced sound (middle) and visual feedback (bottom) resulting from the interaction between mallet and timpani physical impacts.

Conclusions and Future Work

We have presented in this paper a complete framework for analyzing and synthesizing new percussion performances from real pre-recorded gestures, and for facilitating the interaction process between gesture and sound. An initial analysis step on mallet trajectories has led to the identification of consistent control parameters (position, velocity and acceleration with their corresponding time stamps). Such a parameterization of timpani gestures proves to be pertinent, as shown by the high recognition rates obtained with the classification/recognition methodology. The synthesis step provides a sensorimotor control loop for controlling a physical model of a virtual performer using only mallet trajectories. The cascaded control scheme solves two inversion problems which lead to the calculation of appropriate forces and torques applied to the physical model. The evaluation of the synthesized model is done both quantitatively and qualitatively. The quantitative evaluation compares synthesized and real gestures. We also evaluate the simulated gestures through the analysis/recognition method used in the previous analysis step. This comparison shows that the proposed framework can accurately reproduce the mechanisms involved during percussion performances and that the various gestures produced may be easily discriminated. The control of sound synthesis processes is also made easier, as the physical models can be used to dynamically modify the interaction parameters that are applied to the drum model. The mapping-interaction scheme describes the general physical mechanisms involved in the interaction, but it can also provide ways to build new experiences that can be useful in virtual reality. There are many directions for extending this work, mainly related to the synthesis by analysis schemes, and the building of new interaction paradigms in order to simulate percussion performances. Our analysis could be extended for a larger database of gestures, including other dynamical and stylistic variations, such as dynamics variations in music (pp, mf and ff ). Furthermore, an accurate analysis of timpani gestures could lead to the identification of the stiffness and damping coefficients involved in our synthesis models, thus characterizing various biomechanical constraints applied on arm joints, depending on musical variations. Further work on synthesis of percussion performances mainly relies on the interaction process that is made programmable with our system. Our interaction scheme currently focuses on the influence of the synthesized percussion gestures on a physical drum model. Inversely, the interaction process could also involve the influence of the drum model on the gesture simulation. This would involve the development of other gesture control models for dealing with different action/reaction schemes, depending on various physical characteristics of the vibrating models.

Fig. 2 :

 2 Fig. 2: Motion profiles for the French grip. First raw: local extrema extracted from height trajectories of the mallet extremity, one particular legato beat for French grip: (a) position (with E 1 , E 2 and E 3 extrema), (b) velocity and acceleration. Second raw: profile variations of (c) position, (d) velocity and (e) acceleration trajectories for the French grip, and for each gesture variation (black: legato, red: tenuto, blue: accent, green: vertical accent and cyan: staccato).

3Fig. 4 :

 4 Fig. 4: (a) Interaction with sound synthesis: mapping between gesture parameters (G, F G ) and sound parameters (S, F S ). (b) Gesture features G from which F G can be derived: impact location (x 0 , y 0 ), compression displacement (p, ω), mallet-membrane angle (Φ).

Fig. 5 :

 5 Fig. 5: Comparison of simulated and real (captured) gesture variations: root mean square error (RMS) for shoulder (blue), elbow (green), wrist (red) and mallet extremity (black), computed on 200 simulation trials for each gesture variation.

Fig. 6 :

 6 Fig. 6: Results from the simulation of Legato impacts for the French grip. Top: comparison of recorded and simulated trajectories (mallet extremity and joints positions). Middle and bottom: resulting sound and visual feedback from the physical interaction between the mallet and the timpani membrane.

  Tab. 2: SVM recognition percentage: timpani grips using mallet position extrema presented in Figure2.

		1.1	0.1	0.4	21 / 1.3	47 / 1.1	68 / 1.3
	German	0.9	0.05	0.2	12 / 0.9	68 / 0.8	85 / 1.0
			Training / Test	French	German
			French	98.2	1.8
			German	2.7	97.3

5

  Conclusions and Future WorkTab. 5: SVM recognition percentage: simulated French grip gesture variations using the combination of mallet velocity and acceleration extrema presented in Figure2.

	Training / Test	Legato	Tenuto	Accent	Vert. Accent	Staccato
	Legato	92.6	4.8	0	2.6	0
	Tenuto	3.7	94.2	1.2	0.9	0
	Accent	2.5	0	96.4	0	1.1
	Vert. Accent	0.8	0.6	0	93.9	4.7
	Staccato	0	0	0.5	4.4	95.1


) for the control of virtual percussion instruments, by modeling and simulating virtual percussionists gestures. By positioning the virtual performer at the center of the gesture-sound synthesis system, we aim at providing original tools to analyze and synthesize instrumental gesture performances. Our physics-based approach for gesture simulation brings some insight into the effect of biomechanical parameters of the gesture on the instrumental performance. Simulating both gesture and sound by physical models leads also to a coherent and human-centered interaction and provides new ways of exploring the mapping between gesture and sound. The use of motion capture data enables the realistic synthesis of both prerecorded and novel percussion sequences from the specification of gesture scores. Such scores involve motion editing techniques applied to simple beat attacks. We therefore propose an original gesture language based on the instrumental playing techniques. This language is characterized by expressivity, interactivity with the user, and the possibility to take into account co-articulation between gesture units. Finally, providing 3D visual rendering synchronized with sound rendering allows us to observe virtual performances to the light of real ones, and to qualitatively evaluate both pedagogical and compositional capabilities of such a system.

Background

Digital musical instruments have been widely studied during the past decades, focusing mostly on the elaboration of new interfaces for controlling sound processes [START_REF] Miranda | New Digital Music Instruments: Control and Interaction beyong the Keyboard[END_REF]. The design of these new musical instruments relies fundamentally on the input gesture they can track and both sensor and camerabased motion capture systems have become widespread solutions for tracking instrumental gestures [START_REF] Kapur | The Electronic Tabla Controller[END_REF]. More specifically about percussion tracking, one can find comprehensive comparisons of tracking solutions [START_REF] Tindale | A Comparison of Sensor Strategies for Capturing Percussive Gestures[END_REF][START_REF] Benning | A Comparative Study on Wearable Sensors for Signal Processing on the North Indian Tabla[END_REF]. Such systems may also be used for the analysis of performer gestures [START_REF] Dahl | Playing the Accent: Comparing Striking Velocity and Timing in Ostinato Rhythm Performed by Four Drummers[END_REF], Bouënard et al 2010a), where a good understanding of movements leads to the identification of gesture parameters that may be used for Computer Music Journal interacting with sound synthesis processes.

Nevertheless, motion capture data intrinsically present several drawbacks.

The recorded motion is dependent on the uniqueness of the interaction situation under study, in the sense that it is difficult to extrapolate it to new instrumental situations. With such data-based approaches, it is indeed far from straightforward to go beyond the recorded data and reuse it to synthesize adaptative and realistic new performances. Moreover, these systems do retrieve kinematic motion data but fall short in retrieving the physical aspects of the recorded situation. The interaction between such kinematic data with sound relies then on non-intuitive multidimensional correspondences [START_REF] Dobrian | The 'E' in NIME: Musical Expression with New Computer Interfaces[END_REF]. Therefore, a promising research trend consists in providing motion models that could interact with sound synthesis processes.

Developments in sound synthesis have given rise to various methods to generate percussive sounds. Specifically, physics-based synthesis of percussive sounds has involved the modeling of a hammer, collision and sliding excitations [START_REF] Avanzini | Synthesis of Environmental Sounds in Interactive Multimodal Systems[END_REF]Rochesso 2004, Avanzini 2007), as well as drum skins [START_REF] Chuchacz | Physical Models and Musical Controllers: Designing a Novel Electronic Percussion Instrument[END_REF]. However, their main limitation seems to lie in the way they are controlled.

Despite a few early attempts to approach this issue, it is still not clear how to formally relate these models to the excitation by a (real or virtual) performer.

Most of works have involved the interaction of a virtual percussion with a real performer [START_REF] Mäki-Patola | User Interface Comparison for Virtual Drums[END_REF], and contributions exploring the modeling of the equivalent gestures by the definition of a synthetic performer are fairly new. These are either based on real-world robotic models or virtual characters.

Robotic models have been used for synthesizing percussion performances [START_REF] Weinberg | Toward Robotic Musicianship[END_REF][START_REF] Hoffman | Shimon: An Interactive Improvisational Robotic Marimba Player[END_REF][START_REF] Kapur | The Machine Orchestra[END_REF]. Such models generally include the design of mechanical actuators driven by various control solutions (electro-valve, solenoids, motors) for putting robots into motion, Computer Music Journal while multi-agents approaches can be involved to specify or make emerge collaborative rules between robots [START_REF] Eigenfeldt | An Agent-based System for Robotic Musical Performance[END_REF].

More related to our work among virtual models are the contributions involving a slowly evolving mechanical model [START_REF] Gibet | Instrumental Gesture Modeling by Identification with Time-varying Mechanical Models[END_REF], as well as the simulation of a simplified articulated arm hitting a vibrating membrane [START_REF] Gibet | Gestural Control of Sound Synthesis[END_REF]. More recents attempts to overcome this limitation involve the animation of virtual models from MIDI data flows [START_REF] Hänninen | Virtual Concert Performance -Synthetic Animated Musicians playing in an Acoustically Simulated Room[END_REF][START_REF] Lytle | Pipe Dream[END_REF], or from rigid body simulations with the corresponding sound synthesis [START_REF] Van Den Doel | FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Simulation[END_REF][START_REF] O'brien | Synthesizing Sounds from Rigid Body Simulations[END_REF].

Regarding either regarding robotics-inspired or virtual solutions, their main limitation seems to be related to the difficulty to compare to motion data extracted from real performances. Furthermore these do not address a complete physics-based modeling of instrumental actions and subtleties that a virtual performer could use.

Our contribution addresses these issues and proposes a physics-based model of a percussionist that can be driven by real percussion (timpani) data. It enables the synthesis of pre-recorded as well as novel percussion sequences by the specification of a gesture score, the latter consisting of a choice of pre-recorded gesture units.

The paper is organized as follows. In a first time, we present the basic techniques related to timpani performance as well as the building of a motion capture database. We then introduce the global architecture of our framework, from the editing of a gesture score and the real-time simulation of percussion gestures, to its mapping to sound synthesis processes for synthesizing whole percussion performances. A qualitative evaluation of our framework is then conducted, which consists in the synthesis of percussion sequences composed of gesture units from the database. These sequences reflect both percussion exercises that are already available in the database but also exercises that were not initially captured. We then discuss the advantages and limitations of our approach, and finally conclude with

Computer Music Journal

Timpani Performance

In this work, we focus on a particular type of percussion instrument, timpani (Figure 2). Timpani related equipment is mainly composed of a bowl, a head and mallets. In general, timpanists have to cope with several timpani (usually four) with bowls varying in size [START_REF] Friese | The Complete Timpani Method[END_REF]Lepak 1985, Instrumentalist 1984). As for timpani mallets, they consist of a shaft and a head, and are designed in a multitude of lengths, weights, thicknesses and materials, their choice being of great importance (Instrumentalist 1984).

Timpani performers use a wide range of playing techniques. First, there are two main strategies for holding mallets: the French grip (also called "thumbs-up") and the German grip (or "matched" grip). These strategies imply different positions of the hand (vertical palm with the French grip, horizontal with the German grip), and thus different motions of the wrist and of the fingers. Moreover, the use of the limbs differ: the French grip imposes larger movement amplitudes of the arms and forearms. It should be noted however that many timpanists use techniques Computer Music Journal combining some features of the two main techniques described here.

Performers commonly use various gesture articulation modes (legato, tenuto, accent, vertical accent and staccato), as well as three distinct locations of impacts (one-third of membrane radius, center and rim). The most used is the one-third location, producing a full sound with substantial resonance. The center (sharp and muffled attack with barely any resonance) and rim (metallic sound and a resonance bringing out mostly high frequencies) are used less often, but have been adopted by composers since Elliott Carter [START_REF] Carter | Eight Pieces for Four Timpani (One Player)[END_REF]).

Motion Capture Protocol and Database

As initially proposed in (Bouënard et al 2008a), we captured the motion of several timpani performers using a Vicon infra-red camera tracking system and a standard DV camera that allow both the retrieval of gesture and sound. Percussionists used a lycra suit fitted with markers placed according to the marker positions of Vicon's Plug-in Gait model. In order to retrieve beat impacts, markers have also been placed on the mallets. It should be noted that the placement of markers on mallets can have an impact on the recorded performance since it can slightly change their balance.

The balance between left and right mallets can also be altered since different placements have been used, as a means of recognizing left and right mallets during the processing of motion data. A careful choice of the size of the timpani has been done as regards to capture conditions. A 23"timpani has therefore been chosen to minimize the occlusion of markers by the timpani bowl.

Three percussionists were asked to perform a pre-defined capture protocol consisting of a single stroke roll for each playing mode. For each mode, performers were asked to change the location of the beat impact according to the three locations described in the previous section. In total, fifteen examples of timpani exercises were performed for each percussionist (each with five beats per hand). The performers Computer Music Journal had varied degrees of musical expertise (from Professor to Master Student or Undergraduate Student), used different grip strategies (French or German), and also differed in dominant handedness.

The choice of the sampling rate used for the acquisition of percussive gestures is important for the quality of the capture data, mostly because of the short duration of the beat impact [START_REF] Wagner | Analysis of Drumbeats -Interaction between Drummer, Drumstick and Instrument[END_REF]. With high sampling rates (500Hz and above), one can expect to retrieve beat articulations more accurately, but the spatial range is significantly reduced so that it may be difficult to capture the whole body of the performer in a limited performance space (medium sized room), given a reasonable number of IR cameras (in our case, six cameras were used). For this project, a compromise was chosen by setting the cameras at 250Hz, allowing both full-body capture as well as a reasonably high sampling rate for capturing reliably mallet beat impacts.

Gesture-Sound Synthesis System

The proposed system integrates both visual and sound feedback through the realtime simulation of instrumental percussion gestures from a user-defined sequence of gestures, a "gesture score". The global architecture (Figure 1) involves three steps: a) the editing of a sequence of instrumental gestures using data from a motion capture database, b) the physics simulation of the chosen instrumental gestures, and c) interaction strategies between simulated gesture and the sound synthesis process.

Editing and Composition of Gesture Scores

The basis of the editing process is highly inspired from existing event-based works in representing music-related materials. One of the most widespread examples is the approach that was adopted in the design of the MIDI protocol. Similarly, we define a set of events that can be used in our system for synthesizing percussion Computer Music Journal performances. However, the drastic difference is that our system uses the manipulation of elementary gesture events. In fact, these gesture events consist of motion signals coming directly from collected data presented in section "Timpani Performance", and we will therefore refer to them as gesture units. Figure 1 depicts the use of such gesture units in our synthesis system. The editing and assembling of these canonical gesture units lead to a gesture score, whose equivalent signal is then simulated by our system.

As proposed in [START_REF] Ramstein | Analyse, Représentation et Traitement du Geste Instrumental[END_REF], Bouënard et al 2008b), such gesture units can be obtained from the segmentation of each multi-beat percussion signal into single beat-to-beat units (i.e. the motion between two beat impacts) by examining the physical activity of the performer-instrument interaction. However, in a simulation context, problems may arise from the assembling of such beat-to-beat gesture units. This segmentation creates two types of problems. The first one is related to the fact that linking two gesture units at the moment of the beat impact may create unwanted kinematic discontinuities, concerning the position and orientation (and their derivatives) of both mallet extremities and body segments. Another problem that can occur is the alteration of beat impact profiles. Linking two gesture units at the beat impact would lead to separate impact profiles in two phases, which may result in the modification of the action/reaction mechanical nature of the impact.

Finally, placing the articulation at the exact moment of the beat impact is at odds with the usual decomposition of instrumental gestures into preparatory, interaction and retraction gestures. Such gesture units are analogous to movemes [START_REF] Bregler | Learning and Recognizing human Dynamics in Video Sequences[END_REF] and dynemes [START_REF] Green | Quantifying and Recognizing Human Movement Patterns from Monocular Video Images: Part I[END_REF] in motion pattern recognition, as they are the source of high-level parameters retaining motion variations that can distinguish between different executions of a motion (Bouënard et al 2010a).

We therefore translate gesture units from beat-to-beat to beat-centered units.

The articulation point is then placed between the retraction phase of the previous

Computer Music Journal gesture unit and the preparatory phase of the next one. Because even such a unit representation may lead to discontinuities in position and velocity between two gesture units, we ensure a C 1 continuity on drumstick extremity trajectories.

Examples of beat-centered gesture units for legato and accent articulation modes are presented in Figure 1, with the corresponding simulation of the two-units assembling.

Virtual Performer Simulation

The physics modeling and control of the virtual percussionist is divided in two parts: (a) an offline process for specifying the physics model from the anthropometry of the captured percussionist, and (b) the real-time physics-based control loop achieving the simulation of the previously edited gesture score.

The physics-based model of the virtual percussionist is composed of a set of rigid solids linked by mechanical joints, considered as the virtual performer skeleton. The biomechanical properties of this skeleton (solids' length, mass, density, inertia) are initialized by motion capture data. The motion of every solid composing the physics model is simulated by solving the physical motion equations thanks to the Open Dynamics Engine library. The time step used for simulating the synthesized exercises presented in this paper is about 1 milli-second.

Our control paradigm operates directly from the mallet extremity trajectories infers joints angles (such as shoulder, elbow, wrist) from mallet trajectories. In a extracted from the gesture score, and solves two cascaded inverse problems (Bouënard et al 2009a(Bouënard et al , 2009b(Bouënard et al , 2010b)). In a first time, an Inverse Kinematics module second time, an Inverse Dynamics module exploits the joints angles previously computed for driving the mechanical joints of the virtual performer. For each mechanical joint, the physics control loop involves its angular state and target, and computes the necessary torques to be applied on the linked solids for reaching the Computer Music Journal

Multimodal Simulation and Interaction

The physics layer is exploited to express the mapping between the simulation of percussion gestures and sound synthesis methods at the physics level. The interaction between simulated percussion gestures and sound synthesis involves a collision detection algorithm that retrieves information on the impact position, velocity and force (direction and amplitude). Such interaction model can therefore be applied to various types of sound output: pre-recorded percussive sounds, signal-based (wavetable) and physics-based sound synthesis. For the physics-based sound synthesis, we experimented with a modal synthesis approach, using Modalys [START_REF] Adrien | Etudes de Structures Complexes Vibrantes, Application à la Synthèse par Modèles Physiques[END_REF][START_REF] Ellis | Modalys Demonstration[END_REF].

Two levels of interaction are involved in the simulations presented in this paper. The first one is a one-to-one mapping between the virtual performer simulation and the sound simulation. The vibrating membrane parameters are its size, stiffness, Young's modulus, and are shared between the two simulations. Beat

Computer Music Journal impact information is composed of the location on the membrane as well as the attack velocity and force. The second level of interaction is the communication of such beat impact parameters in real-time. This communication is achieved by using the Open Sound Control protocol, therefore helping the broadcast of parameters independently of the number and type of machines involved for the simulations.

As for visual outputs, users may explore the percussion performance space and visualize different features of interest (Figure 3), such as static information attached to the joints, mallets' 3D trajectories, varied camera views, virtual character rendering, as well as the visualization of impact forces and positions.

Musical Evaluation of Virtual Percussion Performances

One of the most interesting outcomes of such framework is the possibility of handling and assembling heterogeneous performances using a combination of a few percussion gesture units. Thanks to the physics simulation of the virtual performer, the issue of gesture articulation between movement units is partly addressed by the physics engine, thus leading to a more natural sequence of movements. Although, the resulting articulation is not necessarily equivalent to real performer techniques, it will nevertheless be a physically plausible solution to the problem. To evaluate the proposed framework, we have simulated an extensive set of musical examples consisting of several timpani exercises. The percussion exercises to be simulated as well as the resulting simulations were chosen and evaluated by the last author (who is a university percussion professor and active performer) by focusing both on the visual and auditory feedback, first simultaneously and then separately. The resulting exercises are available in the DVD submitted jointly to the paper. These exercises are divided in two main categories: validation and extrapolation exercises.

Throughout this section, all percussion exercises exploit gesture units for the French grip from the motion capture database described in section "Timpani

Computer Music Journal Performance". The notation corresponding to timpani playing variations is described in Table 1, and is used as the standard notation throughout the rest of this paper for describing percussion exercise scores.

Validation Exercises

The first category, validation exercises, aims at verifying the accuracy with which the proposed framework can synthesize percussion movements (and sounds) similar to the captured movements in the database. They consist in independent exercises that explore variations on the type of grip, the type of playing mode, as well as the position of the impact on the drum membrane (Figure 4). In this section we analyze how the system manages to connect independent gesture units taken from the database, where each exercise consists on a sequence of similar types of strokes. We particularly simulated the exercises depicted in Figure 4 Computer Music Journal

Extrapolation Exercises

The second category, extrapolation exercises, consists in musical excerpts that go beyond those obtained with motion capture. Such exercises were not performed by the musicians and therefore no corresponding articulation data are available. They typically mix various types of gestures and impact positions in one excerpt, as well as variations on the tempo of the performances, as depicted on Figure 5. We

Qualitative Evaluation

A first musical evaluation was achieved for determining the overall degree of expertise of the virtual performer. From the resulting simulations, we can see that the virtual timpanist's performance is similar to that of a beginner/intermediate performer. Exercises have been performed with a wide and accurate collection of variations in impact locations and in the motion of the arm and forearm, even if only parts of the articulations were performed correctly.

Before going more deeply into the evaluation of the validation and extrapolation exercises, three general issues have been identified. A first issue, the capture limitation, is related to the data collected during our motion capture sessions. The capture limitation involves the lack of available data for the fingers

Computer Music Journal Computer Music Journal which are key components for the characterization of the grips. This can be an issue for highly-trained timpanists for evaluating how accurate the grip simulation is. The second issue, the gesture simulation limitation, is related to our physics-based simulation framework which apparently may cause exaggeration in upper-body movements, or conversely fixed movements for lower-body parts. It can then be difficult to discriminate between articulation modes, or to judge the body involvement of the virtual performer. Finally, a third issue, the sound limitation, is related to the Modalys membrane model that may not show as much variability as real timpani instrument. This can be difficult for instance for discriminating between some of impact locations. These three limitations can be found at different levels both for validation and extrapolation exercises.

validation exercises, the resulting simulations proved to show significant variations in the quality of the gesture articulations -exercises depicted by Figure 4(a-e). A much more satisfying comment is about impact locations simulations -exercises depicted by Figure 4(g-h) -where impact points were judged as accurate. However, the gesture simulation limitation has also been identified by an exaggeration of shoulders motion. A particular attention has also been paid on the sound outcome which was judged plausible and differential for the articulation modes, although the sound simulation limitation was pointed out as a factor of drifts in attack saturation and resonance compared to real timpani sounds.

As for extrapolation exercises, a first source of satisfaction is the comments obtained for exercises depicted by Figure 5(a-b). The transition between articulation modes (legato, tenuto, accent, vertical accent and staccato) has been judged as fairly smooth and realistic, with changes perceptible throughout the simulation.

Discrepancies have however been noticed particularly for legato-tenuto and tenutoaccent pairs, so that visual differences between these are sometimes not very clear.

This can be attributed to a combination of capture and gesture simulation

Computer Music Journal limitations, as unavailable finger effects as well as motion exaggeration may have altered part of the fine subtleties between these gesture modes. More interestingly about the accelerando-decelerando exercise, we can identify an height amplitude decrease of the mallet motion for respecting the beat velocity, as attested by the middle portion of the arrow in Figure 5(d). As for the quality of the gesture-sound interaction, a crescendo occurs naturally during the accelerando by setting the timpani membrane in resonance. It should be noticed that these features are also used by real timpanists to cope with roll speed and build-up a crescendo during a roll. This fact also attests to the natural response of the Modalys model of the drum membrane in response to the simulated legato strokes under an accelerando tempo variation. It shows moreover that the interaction model, and especially the collision detection of beat impacts, offers realistic features such as contact durations and forces in comparison to real timpani performances. This exercise also test the limits of our system, the speed/quality ratio of our system is somewhat reached with this exercise, since an irregular motion flow appears when approaching the maximum beat velocity, consequently the impact locations substantially change, causing inconsistency in the quality of the sounds produced. Finally, the exercise testing the sequencing of beat impact locations -Figure 5(e) -gave interesting results as differences both in location and sound output were judged as perceptible, and more specifically with respect to the end of the exercise, i.e. the two last groups of four strokes when switching between one-third, center, one-third and rim locations.

However, it has been noted that center locations should have been more obvious among the three types of location, specifically for the centered strokes. The major reason for this effect is that the original recorded performances that were used to build and simulate this exercise were not performed accurately enough with respect to impact position. As we will discuss in the next section, the recorded beat impact locations -hollow symbols on Figure 6 -for the center location are too near to those

Computer Music Journal corresponding to the one-third location. This illustrates the important influence of the captured performance on our system, which will conserve performer's style but also drifts if any.

Discussion

Simulating instrumental percussion gestures for controlling sound synthesis processes presents advantages and limitations, and these are of different orders when considering each module of our framework. In the following sections we will analyze in detail some of the sources of variability that are produced by our simulation system and may be at the origin of unwanted artifacts, as well as new interesting possibilities. We therefore consider sequentially the advantages and limitations of, firstly, the instrumental gesture simulation and secondly the gesture editing step (cf. Figure 1).

Virtual Performer Simulation

We here underline the importance of the parameterization of the underlying physics model of virtual character. A non-optimal parameterization can lead to artifacts in the resulting synthesis of virtual percussionist gestures, as well as to effects on synthesized sounds. We also highlight the interest of simulating instrumental gestures with respect to the possible modulation of synthesized gestures while preserving the initial style and expressive characteristics of real percussion performances.

Advantages

From a simulation point of view, once an acceptable parameterization is achieved, it has been shown that associating a motion capture database and the physics-based synthesis of instrumental gestures yields accurate simulations (Bouënard 2009c, slightly modifying the instrumental gesture may produce a form of expressiveness.

Computer Music Journal
The propagation (not the amplification) of such numerical drifts is also influenced by the editing and the articulation of gesture units. These small differences in gestural trajectories therefore lead to variations in the impact locations on the membrane, thus producing variations in corresponding synthesized sound. These different effects may be explored musically, and actually lead to non-mechanical performances by the virtual percussionist. An example is presented in Figure 6, comparing recorded and simulated legato strokes for the French grip. One can note the comparable variability of the (real) percussionist's strokes from motion capture (small hollow symbols), and the resulting variability of the synthesized strokes (small bold symbols) from multiple simulations of each gesture unit.

Limitations

Depending on the sequence and speed of the selected gestures, as well as to the fine

Computer Music Journal tuning of the mechanical joints, simulation artifacts can appear and result in large variations in beat impact positions and forces. These variations are due to the adaptation of the physics simulation to the constraints in the movement data, and may yield unexpected sound phenomena. An example is presented in Figure 7, where the virtual musician performs a sequence of four beats played legato with different sets of physical constraints. The parameterization of the mass and density of the different segments composing the physical model of the virtual percussionist should be chosen carefully, so that the inertial behavior of the system is comparable to that of humans. Otherwise unexpected artifacts may be obtained, cf. the physics model for achieving this posture control that may lead to numerical instabilities. Moreover, the German grip requires more control from the wrist and fingers, features that are not taken into account in our model. Figure 7(c) shows the effect of this issue on the resulting sound. The virtual performer is given a task to perform a sequence of accent strokes using captured data of the German grip. From the resulting sounds, one can notice that at some moments, (1) the sound amplitude is larger than average, and (2) the resulting beats are not regularly spaced, although no dynamics or tempo variation was present in the score. Issue (1) characterizes a larger stroke as an artifact from the sound synthesis generation, i.e. the simulated gesture presents strokes that are very close temporally and spatially. The collision detection events, performed in a short lapse time, are then interpreted by the Modalys system which "glues" them together as a single yet much stronger stroke.

As for issue (2), it can be explained by the limitations of both the capture database and the simulation framework, as discussed above.

Motion Database and Gesture Editing

Our framework depends highly on the motion clips initially recorded during the acquisition process of motion data. The editing processes associated with the motion database are off-line components that are of particular importance when considering the composition of a gesture score. We initially highlight the advantages of our framework in considering the simulation of percussion performances at the gestural level. We then discuss the completeness of the available motion capture database, as well as the editing method presented in this paper.

Advantages

In this work, we adopted a sketching approach to the composition of gesture scores made of elementary gesture units. Beginning and ending points of a gesture unit are

Computer Music Journal determined from mallet height trajectories. This constitutes an intuitive process where the user only needs to select and assemble a limited amount of canonical gesture units, one for representing each timpani playing variation. The whole gesture score can then be formed from copy-and-paste and concatenation operations on these gesture units.

Although the presented model does not take into account the subtle mechanisms occurring in real percussion performances, a simple articulation of gesture units is implicitly achieved by the internal physics simulation when the gesture score simulation switches from a gesture unit to another. Our simple yet effective sketching process for composing gesture scores solely based on mallet extremity trajectories has the advantage of offering the articulation mechanism transparently to the user.

Another advantage of our framework in considering the composition at the gestural level is the preservation of the expressive features inherent to real percussion performances. As the composition process of gesture scores relies highly on real motion data, this insures that simulations will preserve the expressive components of the gesture units used in the gesture score. It allows consequently to preserve the style of the real performer in the simulated percussion performances.

Extrapolation exercises can then be considered as supplementary exercises that we could have asked the percussionist to perform.

The presented motion capture database in section "Timpani Performance" focuses on a particular excerpt among the multitude of percussion techniques used by timpani performers. Our system provides the interesting possibility of enriching this limited database by the editing and simulation of novel percussion performances. One can therefore consider our approach as a mean of preventing the tedious and time-consuming task of capturing an exhaustive motion database containing all the possible combinations of playing techniques.

Computer Music Journal

Limitations

The motion capture database presented in section "Timpani Performance" focuses on a specific set of timpani playing techniques, i.e. it provides gesture examples only for these playing techniques. As seen before, this fact limits the possible simulations to be performed with the proposed system. It also raises the question of the quality of the original captured motion. For instance we can see in Figure 6 the distribution of beat impacts from captured data is not as accurate as expected, with center locations too near from one-third locations. It leads to simulations showing the same trend, as pointed out in the qualitative evaluation.

Another limiting issue is the choice of the set of gesture units used during the editing step (section "Editing and Composition of Gesture Scores"). Such issue yields inevitably to the question of the existence of a best-suited gesture unit as well as its automatic recognition, compared to other examples of the same playing variation. If such "best" gesture unit exists, how can one identify it among the numerous replicates available in the motion database? Otherwise, does it make sense to average multiple gesture units so that a "mean" gesture unit can be used?

We believe that using an average gesture might lead to the loss of some of the expressive features implicitly contained in the gesture signal. We therefore chose one specific gesture example within the sequence, without trying to find an optimized criteria for automatically identifying and selecting it.

A final issue related to the editing process is the concatenation of gesture units, which is the basis of the composition process. The gesture composition presented in this paper reveals only a sketching approach for composing gesture scores. It could be improved by considering higher-level planning processes responsible for the sequencing of gestures. In real percussion playing, high-level control mechanisms are indeed in charge of the articulation between playing modes, depending on many factors such as playing style, tempo variations as well as

Computer Music Journal expressive nuances.

Conclusion and Future Work

We presented a system for synthesizing percussion performances, from the editing process of gesture scores, to the physics simulation of a virtual percussionist that controls sound synthesis processes. The approach was applied to the simulation of both validation and extrapolation exercises consisting in pre-recorded as well as new edited percussion sequences. The qualitative evaluation of both validation and extrapolation exercises by a university percussion professor have attested the overall realism of the resulting simulations. Our system is capable of handling realistic motion strategies, such as performing gesture articulations separately, as well as their smooth sequencing. Our system can also cope with motion-sound strategies that are tightly linked. The best example is the accelerando-decelerando exercise which showed that the virtual performer could adapt to high tempi and adequately decrease his mallet height while performing a sound crescendo, which are realworld properties of a percussion roll. timing properties. The evaluation also pointed out three major issues that make room for improvements for our gesture-sound synthesis system.

The capture limitation yielded to difficulties to recognize some gesture articulations in our simulated exercises. Although our motion capture setup does capture the orientation of hands and mallets, such information appears to be unsufficient for modeling accurately the grasp subtleties that could enhance the realism of the resulting grip simulations. This issue could be overcome by adding sensors on performer's fingers, and therefore monitoring the grasp mechanisms involving the fingers and the mallets. Of course we can consequently anticipate an additional work for extending our virtual performer modeling and simulation to take into account such additional data.

Computer Music Journal

An other issue is related to the gesture simulation limitation, leading to upper-body motion exaggeration, as well as fixed movements for lower-body parts.

Motion exaggeration can be explained by the dynamic properties of the body parts of the virtual percussionist. When the physical model is put into motion, these body parts seem to acquire too much inertia, so that even if the overall simulated motion is accurate enough compared to the recorded gesture units (Bouënard 2009c, Bouënard et al 2010b), an exaggeration of motion appears. An exploration of the dynamic properties of the physics model could solve this issue, more specifically joints linking limbs could involve an increase in stiffness. Moreover, fixed movements of the lower-body are related to balance strategies, i.e. how a timpanist involves his center of mass for helping him to switch between playing modes. In fact no balance strategies are involved in the simulation of the percussion exercises, although information on performers' center of mass was captured in the motion capture sessions. This simulation choice explains the remark on the quite static bottom part of the virtual performer. Balance strategies could also help us to extend the repertoire of the virtual performer, as they are important if we want to simulate exercises where the virtual performer plays on several timpani instruments.

A final issue is the sound simulation limitation, which sometimes leads to resulting sounds without the metallic grain of real timpani sounds, or with too much attack saturation and not enough resonance. The lack of sound resonance is easily explained by the fact that the drum membrane model does not feature a resonance body. Moreover, the non-metallic nature of the resulting sounds may also come from a combination of imprecisions both from the drum membrane model itself since no metallic rim is included in the Modalys model, and from beat impact locations inaccuracies. A solution would then involve considering an extended model of the timpani instrument with Modalys.

The overall musical perspectives of our gesture-sound synthesis system are

Figure 1 :

 1 Figure 1: Global framework, from the off-line editing of a gesture score with its corresponding input signal, to the real-time simulation with visual and sound feedback.

Figure 2 :

 2 Figure 2: Timpani performance, (left) timpani instrument and mallets, (middle) French (top) and German (bottom) grips, (right) theoretical beat impact locations.

Figure 3 :

 3 Figure 3: Visualization of (a) mallet trajectories and (b) impact position on the drum membrane

Figure 4 :

 4 Figure 4: Validation exercises. (a) Sequence of legato strokes. (b) Sequence of tenuto strokes played at the rim. (c) Sequence of accent strokes. (d) Sequence of vertical accent strokes. (e) Sequence of staccato strokes. (f) Sequence of legato strokes played at the one-third. (g) Sequence of legato strokes played at the center. (h) Sequence of legato strokes played at the rim.

  typically simulated several sequences of different gesture articulations. The major difference between exercises presented in Figure5(a) and Figure5(b) is that the former aims at testing the smooth sequencing of the gesture articulations, whereas the latter aims at testing the velocity and fidelity of quick changes between three particular articulations. We also simulated a well-known exercise consisting of an accelerando-decelerando of legato strokes -Figure5(c) and Figure5(d). This exercise aims at testing the ratio between the maximum velocity and the gesture fidelity our system can cope with. A final exercise -Figure5(e) -consists in testing the sequencing of different beat impact locations for a given gesture articulation.

Figure 5 :

 5 Figure 5: Extrapolation exercises. (a) Five gesture units repeated four times: staccato, legato, tenuto, accent and vertical accent. (b) Variations between legato and accent strokes, ending with a vertical accent. (c) Accelerando-decelerando of legato strokes. Note that the timpani database only has samples at a given tempo. (d) Height of the mallet extremity, the top arrow shows an amplitude increase, as well as a small drop when approaching the maximum accelerando velocity. (e) Sequence of legato strokes played at various locations on the drum membrane.

Figure 6 :

 6 Figure 6: Comparison of impact locations: small hollow, large hollow and small bold symbols represent resp. strokes from the database, chosen strokes in the database and simulated impact locations.

Figure 7 :

 7 Figure 7: Sound waveform amplitude and intensity, from (a) a simulation artifact and (b) a stable simulation for the French grip. (c) As discussed in the text, a possible artifact for the German grip is when two close impacts are "glued" together by the sound synthesis, oval (1), and when impacts are finally heard as two, oval (2).

Figure

  has been observed that percussion grips may have an influence on the quality of the simulations. While French grip-related simulations work well, German ones are less satisfactory. This can be explained by the fact that the German grip involves generally a technique requiring that the arms stay relatively immobile during a large portion of the gesture unit. This ends with a very stiff configuration of Computer Music Journal

Table 1 :

 1 Timpani playing notation

Computer Music Journal

Computer Music Journal Bouënard et 2010b).

A first advantage of the physics-based framework for simulating percussion gestures is the retrieval of the physical aspects (beat impacts) of the recorded instrumental gestures. No information was initially been recorded concerning beat impact durations and forces, our system therefore starts from pure kinematic data (joints position) to re-build a whole performance with physical interactions. A second advantage of our system is its virtual nature. Indeed, the proposed framework allows the modeling of realistic phenomena occurring during real percussion performances. Furthermore, it also allows going beyond reality, and provides the possibility of modifying the parameters of the virtual environment as wanted for creating new percussion performances. It is possible to explore changes in the physical model of the virtual percussionist, as well as in the interaction schemes between synthesized gestures and sound synthesis inputs through the contact information provided by the physical framework.

From a more musical point of view, an issue to take into account is the small variations in movements that are important to produce expressive performances. Indeed, deadpan virtual performances where movements are kept as close to an ideal movement target as possible are generally considered mechanical and nonexpressive. Although our focus is more on the technical capabilities of our virtual performer, the smooth and non-mechanical interpretation of the simulated exercises is important. By choosing a single gesture unit for representing each playing mode in the various simulations (see section "Editing and Composition of Gesture Scores"), we might actually expect to eliminate the natural variability of movements from the original performance by the (real) percussionists. Nevertheless, thanks to the physics simulation layer, starting from a unique gesture unit for each gesture type will produce synthesized gestures with inherent variability due to the dynamic characteristic of the virtual character. Exploiting (while controlling) such drifts for Computer Music Journal encouraging as we plan also to apply it both for musical pedagogy and composition.

A pedagogical tool based on our system would offer a multimodal interface for studying timpani gestures of a percussionist, being a student or a professor. Of course it would first imply a minimal amount of motion capture sessions, which can involve a heavy and time-consuming setup, but we believe it is a price to pay for accessing such pedagogical capabilities. A compositional tool can also be derived, with which composers could then choose for a substantial palette of percussion variations, such as gesture articulations, beat locations and tempi, yet from a minimal amount of gesture units. Gesture score inputs could be graphically implemented in a user interface for easying the editing process of composers, while giving them both visual and audio feedback of their simulated composition.