Jules Françoise

Gabriel Meseguer-Brocal

Frédéric Bevilacqua

Movement Analysis and Decomposition with the Continuous Wavelet Transform

Keywords: CCS Concepts:, Human-centered computing → Human computer interaction (HCI); Gestural input Movement, Rhythmic Gesture, Wavelet, Continuous Wavelet Transform, Recognition, Vocalization ACM Reference

Human movements support communication, and can be used to imitate actions or physical phenomenons. Observing gestural imitations of short sounds, we found that such gestures can be categorized by their frequency content. To analyse such movements, we propose an analysis method based on wavelet analysis for clustering or recognizing movement characteristics. Our technique draws upon the continuous wavelet transform to derive a time-frequency representation of movement information. We propose several global descriptors based on statistical descriptors, frequency tracking, or non-negative matrix factorization, that can be used for recognition or clustering to highlight relevant movement qualities. Additionally, we propose a real-time implementation of the continuous wavelet transform based on a set of approximations, that enables its use in interactive applications. Our method is evaluated on a database of gestures co-executed with vocal imitations of recorded sounds.

INTRODUCTION

The capture and description of human movements has an intertwined history between Sciences and Arts. The early endeavours of pioneers such as Étienne-Jules Marey, physiologist and physician, and Eadweard Muybridge, photographer, is exemplary of the different disciplines that were initially involved for studying human movements. Later, the possibility of quantifying human movement parameters opened the development of mathematical and computational modelling.

Movement analysis can be performed at different levels [START_REF] Camurri | The Dancer in the Eye: Towards a Multi-Layered Computational Framework of Qualities in Movement[END_REF], from low-level features related to kinematics to high-level movement and gesture representations linked to semantic, affective or expressive qualities [START_REF] Camurri | Communicating expressiveness and affect in multimodal interactive systems[END_REF]. High-level qualitative representations have also been investigated in various fields, and in particular in non-verbal communication [START_REF] Mcneill | Gesture and thought[END_REF] and artistic practices such as dance and music. Nevertheless, important work remains to be performed to establish quantitative mid-and high-level movement representations. For example, while dance and somatic practitioners have developed powerful concepts to analyze how movements are performed, generally denoted as movement qualities [START_REF] Fdili | Movement Qualities as Interaction Modality[END_REF][START_REF] Volpe | Computational models of expressive gesture in multimedia systems[END_REF], it remain difficult to translate such concepts in general computational models [START_REF] Fdili Alaoui | Seeing, Sensing and Recognizing Laban Movement Qualities[END_REF].

In this paper, we investigate the use of time-frequency representations for computational movement analysis from wearable sensor data. Considering that movement qualities are often related to movement dynamics and rhythmic patterns, we hypothesize that such representations would be helpful for deriving features in a variety of tasks including 2.1 Computational Movement Analysis: from Trajectories to Qualities Developing operational representations of movements is key to success in movement analysis and interaction design. Such representations build upon features that characterize some of the aspects of movement that researchers, engineers or designers aim to capture. These movement characteristics can cover low-level parameters such as trajectories, or higher-level aspects that relate to movement's expressive, affective or emotional qualities. Camurri et al. [START_REF] Camurri | The Dancer in the Eye: Towards a Multi-Layered Computational Framework of Qualities in Movement[END_REF] proposed a multi-layered computational framework for movement analysis with a focus on extracting movement qualities.

Their framework includes several successive layers of analysis, from signal-level to gesture-level and semantics-level descriptors.

Low-level movement representations build upon the raw data originating from either motion capture systems or wearable sensors. Common preprocessing operations include resampling, scaling, filtering, quantization, coordinate change, and numeric derivatives. In this paper, we focus on the use of Inertial Measurement Units (IMUs). For a single IMU, there are typically 6 or 9 dimensions from a 3D accelerometer, a 3D gyroscope and optionally a 3D magnetometer. Using several IMUs on the body scales the number of inputs. Cameras provide an even larger dimensionality considering pixels as raw data. Many mid-level features have been proposed within the field of movement computing to represent positional motion capture data, including curvature [START_REF] Glowinski | Toward a minimal representation of affective gestures[END_REF], Kinetic Energy [START_REF] Glowinski | Toward a minimal representation of affective gestures[END_REF], the contraction index [START_REF] Volpe | Computational models of expressive gesture in multimedia systems[END_REF], the silhouette [START_REF] Volpe | Computational models of expressive gesture in multimedia systems[END_REF], or the quantity of motion [START_REF] Volpe | Computational models of expressive gesture in multimedia systems[END_REF]. Other approaches focus on a geometrical description of postures, with features such as bounding space [START_REF] Larboulette | A review of computable expressive descriptors of human motion[END_REF], center of mass [START_REF] Kapadia | Efficient motion retrieval in large motion databases[END_REF], directness [START_REF] Volpe | Computational models of expressive gesture in multimedia systems[END_REF], or Müller's Full Body Relational Features [START_REF] Müller | Efficient and robust annotation of motion capture data[END_REF].

Such mid-level features can be used to derive representations capturing higher-level concepts such as emotion and movement qualities [START_REF] Camurri | Communicating expressiveness and affect in multimodal interactive systems[END_REF][START_REF] Fdili | Movement Qualities as Interaction Modality[END_REF][START_REF] Fdili Alaoui | Seeing, Sensing and Recognizing Laban Movement Qualities[END_REF]. Movement qualities are widely used in dance practice and choreography, and consider "how" a movement is performed (with what intention, expression or emotion). Laban Movement Analysis (LMA), developed by Rudolf Laban in the early twentieth century and pursued by theorists such as Bartenieff, is the most widespread theoretical model of movement that has been used for computational modeling. In particular, its "Effort" category is used to describe movement qualities through four dimensions of Space, Weight, Time and Flow [START_REF] Fdili Alaoui | Seeing, Sensing and Recognizing Laban Movement Qualities[END_REF]. In early research, affinities between such efforts and spatial representations have been proposed [START_REF] Volpe | Computational models of expressive gesture in multimedia systems[END_REF]. A number of methods based on machine learning have also been developed to recognize such movement qualities from sensor data [START_REF] Niewiadomski | Low-Intrusive Recognition of Expressive Movement Qualities[END_REF][START_REF] Silang Maranan | Designing for Movement: Evaluating Computational Models Using LMA Effort Qualities[END_REF]. A second approach is to work with movement qualities defined by choreographers themselves, and modeled through physical modeling techniques, as proposed by Alaoui et al. [START_REF] Fdili Alaoui | Interactive Visuals as Metaphors for Dance Movement Qualities[END_REF].

Time-Frequency Representations for Movement Analysis

We are interested in exploring movement qualities through the perspective of movement dynamics. Gestures can be performed with different expression by altering their temporal dynamics to convey intent [START_REF] Fdili | Movement Qualities as Interaction Modality[END_REF][START_REF] Fdili Alaoui | Seeing, Sensing and Recognizing Laban Movement Qualities[END_REF]. We now review computational methods capturing temporal and/or spectral characteristics of human movement.

Fourier

Transforms. The Fourier transform has been successfully used in the analysis of human movement and applied to gesture analysis and recognition. Usually, the Windowed Fourier Transform (WFT) is applied on segments of movement data, and computed using the Fast Fourier Transform (FFT). Shiqi Yu et al. [START_REF] Yu | Gait Analysis for Human Identification in Frequency Domain[END_REF] used the normalized power FFT on a set of points of the silhouette contour. The authors argue that the use of normalization allows for scale invariance, which improves the recognition of gait patterns. Makihara et al. [START_REF] Makihara | Gait Recognition Using a View Transformation Model in the Frequency Domain[END_REF] combined auto-correlation with spectral analysis of gait movements. They propose to evaluate the gait period from auto-correlation, and subsequently apply a WFT on each gait cycle to extract spectral features. The Fourier transform has also been applied to measure movement smoothness. Melendez-Calderon et al. [START_REF] Melendez-Calderon | Estimating Movement Smoothness From Inertial Measurement Units[END_REF] report a systematic evaluation of smoothness measures from IMUs relying on the spectral arc length. They show that thee method is valid when applied on velocity data or on rotational velocity data. Finally, the Fourier transform has been used in conjunction with convolutional neural networks for sensor-based activity recognition [START_REF] Jiang | Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks[END_REF].

Wavelet

Transforms. The Continuous Wavelet Transform (CWT) has been marginally applied to the analysis of human movement. Côté-Allard et al. [START_REF] Côté-Allard | Deep learning for electromyographic hand gesture signal classification using transfer learning[END_REF] used the CWT together with Convolutional Neural Networks (CNNs) and transfer learning for gesture recognition from electromyography (EMG). However, EMG data has different statistical properties compared to IMUs. Recently, Nedorubova et al. [START_REF] Nedorubova | Human Activity Recognition using Continuous Wavelet Transform and Convolutional Neural Networks[END_REF] proposed to use the CWT for activity recognition. The CWT is used to extract time-frequency representations of activities captured with IMUs. The resulting images are fed to a CNN for the recognition of daily activities. The CWT has been previously applied to multimodal discourse analysis [START_REF] Xiong | Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis[END_REF]. Xiong and Quek proposed to use the CWT with a Morlet Basis to compute a time-frequency representation of speech-related gestures. From the extraction of frequency ridges, the authors propose a method for analyzing the relationship between oscillatory gestures and speech content.

MOVEMENT ANALYSIS WITH THE CONTINUOUS WAVELET TRANSFORM

The Continuous Wavelet Transform (CWT) is a time-frequency analysis method allowing for optimal time-frequency localization [START_REF] Paul S Addison | Wavelet transforms and the ECG: a review[END_REF][START_REF] Mallat | A Wavelet Tour of Signal Processing[END_REF]. In this section, we consider how the wavelet transform can be used for characterizing dynamic behaviors in hand gestures. For more extensive tutorials on wavelet analysis, we refer the reader to the dedicated literature [START_REF] Paul S Addison | Wavelet transforms and the ECG: a review[END_REF][START_REF] Mallat | A Wavelet Tour of Signal Processing[END_REF][START_REF] Torrence | A practical guide to wavelet analysis[END_REF].

Motivation

In Fourier analysis, the spectrum of a signal is estimated by convoluting the signal with a set of harmonic plane waves.

The Windowed Fourier Transform (WFT) -or Short-term Fourier Transform (STFT) -computes a time-frequency representation of signals by performing a Discrete Fourier Transform on a sliding window along the signal. As all digital signals are finite, in practice the plane waves -or equivalently, the signals, -are multiplied by a window function to avoid artifacts due to border effects. The WFT therefore assumes a fixed window size, which determines the bandwidth of each frequency band in the time-frequency representation. One of the main limitation of the WFT is the inaccuracy resulting from the imposition of a scale or 'response interval' into the analysis [START_REF] Torrence | A practical guide to wavelet analysis[END_REF]. Indeed, the WFT aliases all frequency components that do not fall within the frequency range of the window.

On the contrary, instead of assuming a fixed window size, the wavelet transform both translates and dilates a wavelet function with short-term influence. The dilation of the wavelet implies that the analysis window is expanded as the carrier frequency of the wavelet decreases. As a result of the Heisenberg's inequality, time-frequency resolution varies: the temporal window is short in high frequencies while the bandwidth is narrower in low frequencies.

A rapid analysis of typical setups for capturing and representing movements further motivates the use of multiresolution analysis. Typically, motion capture systems such as inertial sensors or low-cost motion capture devices have framerates of about 50 to 500 Hz. Human movements are typically in the range of a few hertz, from 0.2 -0.5 Hz to 10 -15 Hz for smooth and periodic movements, and up to 50 Hz for impacts. If one considers that the frequency resolution necessary to derive an accurate analysis around 1 Hz should be at most of order 0.1 Hz, then the minimal window size required with the WFT is 10 s. In this case, any transient high-frequency phenomenon (of duration typically inferior to a second) will be blurred by the size of the analysis window. On the contrary, imposing a window size of 1 s to guarantee an acceptable time localization restricts the resolution in the frequency domain to a bandwidth of 1 Hz, which is insufficient for capturing low-frequency phenomena. The multiresolution analysis solution provided by the wavelet transform allows us to derive an arbitrary high localization both in time and frequency [START_REF] Paul S Addison | Wavelet transforms and the ECG: a review[END_REF].

Formulation of the Continuous Wavelet Transform

The Continuous Wavelet Transform (CWT) of a discrete sequence 𝒙 1:𝑁 = {𝑥 1 • • • 𝑥 𝑁 } sampled at period 𝛿𝑡 for a wavelet function Ψ 0 is defined as the convolution of the sequence with a scaled and dilated version of the base wavelet [START_REF] Torrence | A practical guide to wavelet analysis[END_REF]:

𝑊 𝑛 (𝑠) = 𝑁 ∑ 𝑛 ′ =0 𝑥 𝑛 ′ Ψ * (𝑛 ′ -𝑛)𝛿𝑡 𝑠 , ∀𝑛 = 1 • • • 𝑁 (1)
where Ψ * is the complex conjugate of the normalized wavelet Ψ:

Ψ (𝑛 ′ -𝑛)𝛿𝑡 𝑠 = 𝛿 𝑡 𝑠 1/2 Ψ 0 (𝑛 ′ -𝑛)𝛿𝑡 𝑠 ( 2
)
and 𝑠 is the scale parameter. The time-frequency representation can be constructed by translating the wavelet along the time axis (varying 𝑛) and dilating the wavelet with the scale parameter 𝑠. By analogy with the term 'spectrogram' for the WFT, the 'scalogram' can be computed by taking the power representation of spectral information in the scale

domain |𝑊 𝑛 (𝑠)| 2 .
For most wavelets, there exist an analytical formulation of their Fourier Transform, which reduces the computational cost to a single inverse FFT per frequency band that simultaneously estimates the scalogram at all time steps.

3.2.1 Wavelet Functions. While the WFT imposes windowed plane waves as set of base functions, wavelet analysis offers flexibility in the choice of the base functions for analysis. To be admissible as a wavelet, a function must meet three conditions: the must have zero-mean, finite energy, and their Fourier transform must be real and vanish in negative frequencies [START_REF] Paul S Addison | Wavelet transforms and the ECG: a review[END_REF]. Several factors are to be considered in the choice of a wavelet basis, including orthogonality, real vs complex wavelets, width and shape [START_REF] Torrence | A practical guide to wavelet analysis[END_REF].

In this research, we mostly experimented with the Complex Morlet wavelet -sometimes called Gabor Wavelet,which has the property of optimal localization in time and frequency. It is defined as a plane wave modulated by a Gaussian window1 :

Ψ 0 (𝜂) = 𝜋 -1/4 𝑒 𝑖𝜔 0 𝜂 -𝑒 -𝜔 2 0 /2 𝑒 -𝜂 2 /2 (3)
where 𝜔 0 is the carrier frequency of the Wavelet. The spectrum of the Morlet wavelet is a unit Gaussian centered around its carrier frequency.

Choice of scales.

In comparison with the WFT, the CWT offer a great flexibility in the choice of the analysis domain. In particular, while the WFT restricts the analysis domain to the set of harmonic bands, one can choose a subset of arbitrary scales for wavelet analysis. We follow a convention to distribute the scales as fractional powers of two:

𝑠 𝑗 = 𝑠 0 2 𝑗/𝑏 , 𝑗 = 𝐽 𝑚𝑖𝑛 , 𝐽 𝑚𝑖𝑛 + 1, • • • , 𝐽 𝑚𝑎𝑥 (4)
where 𝑏 is the number of bands per octave and 𝐽 𝑚𝑖𝑛 and 𝐽 𝑚𝑎𝑥 can be specified from a target frequency range. The smallest resolvable scale 𝑠 0 can be estimated from the equivalent Fourier frequency according to the Nyquist frequency 1/2𝛿𝑡. This representation is convenient for selecting the analysis domain, whose specification is therefore reduced to a frequency range and a number of bands per octave.

Example

In this section, we report an illustrative example comparing the proposed method with the Windowed Fourier Transform.

We consider movements captured using inertial sensors. Our approach to multidimensional analysis using the wavelet transform is based on a late fusion of the scalograms. When analyzing dynamic movements with an IMU sensor fixed on the wrist, our goal is to derive a representation of the spectral behavior that is invariant to the movement's direction and amplitude. In this case, we compute the scalogram for each axis of the accelerometer independently, and we represent gestures by the sum of the scalograms on each axis. This approach provides a representation that is independent from the movement direction, and that can take into account different sensors (accelerometers and gyroscopes). However, it is possible to consider scalograms independently on each axis in order to preserve the spatial information.

Figure 1 shows a comparison of the CWT with the WFT on human movement captured with IMUs. We observe that oscillatory behaviors appear with a higher contrast in the scalogram (CWT) than in the spectrogram (WFT). Moreover, the low frequency component is also better defined in the scalogram.

IMPLEMENTATION

The computation of CWT can be performed either offline or online. Both are described below, with more attention to the online implementation which might necessitate different approximations. Before describing these two cases, we describe the important notion of cone of influence.

Cone of Influence

As for the WFT, the CWT is subject to edge effects resulting from the use of finite signals. While this problem can be partially resolved by padding with the edge values before performing the analysis, this process still introduces discontinuities at endpoints. The cone of influence is defined as the region where the edge effects become important, and can be defined in terms of e-folding time. According to Torrence and Compo [START_REF] Torrence | A practical guide to wavelet analysis[END_REF] "This e-folding time is chosen so that the wavelet power for a discontinuity at the edge drops by a factor 𝑒 -2 and ensures that the edge effects are negligible beyond this point".

The e-folding time for scale 𝑠 with the Morlet wavelet is √ 2𝑠, and the relationship between the scale and the equivalent Fourier frequency can be computed as

𝑓 = 𝜔 0 + √ 2 + 𝜔 2 0 4𝜋𝑠 (5)
In the remainder of this study, we consider a sufficient padding of the signals so that the scalogram can be estimated without edge effects at all scales. We also use the e-folding time as criterion for the online approximation of the CWT (section 4.3).

Offline Computation

For offline estimation, the CWT can be efficiently computed using the Fast Fourier Transform to evaluate the convolution as a product in the spectral domain:

𝑊 𝑛 (𝑠) = 𝑁 -1 𝑘=0 x𝑘 Ψ * (𝑠𝜔 𝑘 )𝑒 𝑖𝜔 𝑘 𝑛𝛿𝑡
with angular frequencies

𝜔 𝑘 = 2𝜋𝑘 𝑁 𝛿𝑡 if 𝑘 ≤ 𝑁 /2 -2𝜋𝑘 𝑁 𝛿𝑡 if 𝑘 > 𝑁 /2 (6)

An Online Approximation of the CWT

The CWT is typically computed offline, when the entire signal is available. In order to use in interactive applications, we propose an online approximation of the CWT. Our implementation is based on a finite-length approximation of the wavelet depending on the cone of influence in each frequency band. We propose to use a computation window as small as possible at each scale to get a good approximation of the scalogram with a minimal delay in each frequency band. Additionally, we propose two optimization schemes based on a multi-rate representation of the wavelet and of the incoming signal.

Formulation.

We consider a finite-length windowing method where the window size is specified for each frequency band depending on the wavelet's energy decrease. The CWT is implemented as a filter bank with minimal delay per frequency band, and the computations are done in the temporal domain. At each new frame of the signal, we estimate only the central value of the scalogram on a sliding window with minimal size with regards to the e-folding time. This process is illustrated in Figure 2.

For each scale 𝑠, the value of the scalogram with delay 𝑁 𝑠 /2 is estimated from a new observation value as

𝑊 𝑠 [𝑡 -𝑁 𝑠 /2] = 𝑁 𝑠 ∑ 𝑘=0 𝑥 [𝑡 -𝑁 𝑠 + 𝑘] • Ψ * (𝑘 -𝑁 𝑠 /2)𝛿𝑡 𝑠 (7)
where the window size 𝑁 𝑠 can be estimated from the wavelet's e-folding time 𝜏 𝑠 as 𝑁 𝑠 = 𝜆𝜏 𝑠 /𝛿𝑡 with 𝜆 a constant determined experimentally we call the windowing factor. For a Morlet wavelet with carrier frequency 𝜔 0 , the window size can be estimated at a given scale from the equivalent Fourier frequency 𝑓 as The complexity per frequency band is therefore 𝑁 𝑠 multiplications and 𝑁 𝑠 -1 additions. For the Morlet wavelet with scales distributed in powers of two, the total complexity is exponential in the number of bands and can be estimated as

𝑁 𝑠 = 𝜆 √ 2 • 𝜔 0 + √ 2 + 𝜔 2 0 4𝜋 𝑓 𝛿𝑡 (8
𝐶 = 2 𝐽 𝑚𝑎𝑥 ∑ 𝑗=𝐽 𝑚𝑖𝑛 𝜆 √ 2𝑠 0 2 𝑗/𝑏 -1 = 2 √ 2𝜆𝑠 0 (2 1/𝑏 ) 𝐽 𝑚𝑎𝑥 -(2 1/𝑏 ) 𝐽 𝑚𝑖𝑛 2 1/𝑏 -1 ( 9
)
where 𝑏 is the number of bands per octave and 𝑠 0 =

𝜔 0 + √ 2+𝜔 2 0 4𝜋 •2𝛿𝑡
is the highest resolvable scale. Standard Optimization: The number of computations per frequency band can be reduced by considering that low-frequency components can be approximated by a downsampled version of the wavelet. We propose to downsample the wavelets by an integer factor depending on the ratio of their equivalent Fourier frequency with the Nyquist frequency. As a result, we guarantee that the wavelet's samplerate is sufficient to avoid aliasing the corresponding frequencies, while reducing the number of computation by an integer factor. Note that in this case, the incoming signal is not decimated and we still evaluate the CWT at each frame for all frequency bands.

To avoid aliasing in low frequencies, the signal is passed to a bank of low-pass filters for each decimation factor.

Aggressive Optimization Further optimization can be achieved by decimating not only the wavelet, but also the incoming signal. In this case, instead of evaluating the CWT at each frame for all frequency bands, we apply a As the standard optimization scheme keeps a frame-based computation of the scalogram, the estimation of the wavelet power spectrum remains smooth, which is particularly suited to interactive applications. On the contrary, aggressive optimization involves a severe downsampling of the incoming signal and the wavelet spectrum for lowfrequency components is evaluated at larger time steps. This latter optimization scheme can be interesting when a sparse representation of the scalogram is desired.

In our experiments, standard and aggressive optimization techniques lead to a gain of a factor 20 and 80 in computation time, respectively. Note that the gain with standard optimization is made at the cost of an increased memory footprint, as it is required to store several versions of the incoming signal with different lowpass filtering. In both cases, the major drawback of the optimization methods is the introduction of an additional delay due to the low-pass filtering of the signal.

Figure 3 shows an example of the online approximation of the continuous wavelet transform computed over an acceleration signal. The same plot with the realigned scalograms is showed in Figure 4, where the delay introduced by the online approximation is compensated in each frequency band, for comparison with the online estimate. While the online approximation without optimization introduces a delay, that increases as the scale increases, it can be seen that the distortion of the scalogram is relatively small compared with the offline estimate.

We evaluated the quality of the approximation of the scalogram using the online implementation of the CWT. Table 1 reports the Normalized Mean Squared Error (NMSE) or the online approximation of the scalogram as a function of the windowing factor. Each error is estimated with respect to the offline estimation of the scalogram, and is averaged over the gestures from all participants from the dataset presented in Section 5. For comparison with the true estimate, the estimation delay of the power spectrum was compensated in each frequency band.

A relative window size of 3 -with respect to the e-folding of the wavelet in each frequency band, -provides a NMSE inferior to 5%. This approximation is sufficient in most interactive applications, as the approximation of the scalogram is not distorted.

Software.

We implemented the online CWT as a freely available external for Cycling'74 Max within the MuBu package2 [START_REF] Schnell | MuBu and friends-assembling tools for content based real-time interactive audio processing in Max/MSP[END_REF]. It is implemented as PiPo module that can used for real-time processing or off-line processing of data buffers. An experimental JavaScript implementation is also available within the CODA live-coding environment3 [START_REF] Françoise | CO/DA: Live-Coding Movement-Sound Interactions for Dance Improvisation[END_REF].

Two modes are possible for the real-time analysis of movement data within Max. In the first mode, the values of the wavelet power spectrum are outputted with a minimal delay in each frequency band. In this case, each band of the filterbank is evaluated with a different delay. While this approach does not guarantee a correct alignment of the different frequency bands, it provides a high reactivity in high frequencies, which can be interesting for interactive applications where impulsive gestures should be identified with low-latency while lowfrequency components are longer to establish. In the second mode, the wavelet spectrum is outputted with the same delay in all frequency bands, guaranteeing a correct alignment of the various components. However, this implies that the delay must be aligned on the largest delay corresponding to the lowest frequency component.

APPLICATION TO MOVEMENT QUALITIES MODELING

In this section, we present the application of the method to movement analysis. We used the CWT for both recognizing or clustering movement qualities on a dataset of gestural imitations of sounds. and Machine sounds. The protocol used to constitute the dataset is summarized hereafter, and it is fully described in [START_REF] Lemaitre | Rising tones and rustling noises: Metaphors in gestural depictions of sounds[END_REF].

The experiment was divided in two blocks where participants were instructed to jointly perform a vocal and gestural imitation of the referent sound. We only used the second block where participants' hand movements are recorded, using two inertial measurements units attached to wrists, measuring 3D accelerometer data at a sampling rate of 200 Hz.

The choice of the wavelet movement analysis method on the SkAT-VG imitation dataset was driven by the insights gathered through qualitative studies on participants' strategies [START_REF] Lemaitre | Rising tones and rustling noises: Metaphors in gestural depictions of sounds[END_REF]. In particular, considering the case of abstract sound where no identifiable pitch could be perceived, participants did not favor any specific direction in the movement. The use of specific rhythmic behaviors for different sounds proved to be the most salient feature in the sense that it is shared across most participants.

5.1.2 Manual Annotation of the Dataset. While the dataset was initally annotated with sound categories, observations of the participants' gestures highlighted the importance of temporal and rhythmic patterns. Therefore, we reframed the analysis problem as the extraction of "movement qualities", independent from spatial directions but characterized by their frequency content, instead of the recognition of sound categories from gestures. Importantly, we define these "movement qualities" for our specific evaluation, and their validity is restricted to the specific dataset we use.

First, gestures were segmented into three phases of Preparation, Stroke and Recovery -inspired by similar representations proposed in the literature [START_REF] Kendon | Gesture: Visible action as utterance[END_REF]:

Preparation refers to the phase that leads from the relaxed position to the stroke.

Stroke is where the actual expression of the gesture is accomplished.

Recovery covers the phase from the stroke to the final relaxed state.

Then, we defined six categories to characterize the Stroke:

Steady gestures are postures that remain constant over time. They include completely still postures and gestures that evolve very slowly.

Smooth movements are fluid and include gradual changes in posture.

Dynamic gestures involve abrupt, energetic and rapid actions.

Impulse describe single and sudden gestures.

Periodic gestures refer to movements exhibiting periodicity

Shaky is a specific class for the dataset that involves the hand shaking.

The dataset was manually annotated by two researchers, using a dedicated interface created with Max/MSP, where video recordings and signals were displayed to facilitate annotation. We evaluated the inter-rater agreement between annotators using Cohen's kappa coefficient. We obtained a value of 0.67 for this coefficient, that indicates a substantial conformity. To understand where disagreement occur, we analyzed a confusion matrix between the two annotators. It revealed that most disagreements occur between the dynamic and smooth categories. This is due to the fact that the threshold to consider a gesture as fluid or energetic depends on the annotator's judgment.

Statistical Gesture Descriptors for Recognition

In this section, we describe two approaches to the representation of gestures based on gesture-level descriptors derived from wavelet analysis. The first approach is based on a description of the scalogram by its normalized moments (centroid, variance, skewness, kurtosis) in the wavelet spectrum domain and in the temporal domain. The second approach draws upon the multi-target tracking of the frequency ridges.

Spectral and Temporal Moments.

We propose to describe the scalogram at a high level from the distribution of the wavelet power spectrum and of the energy envelope. The global wavelet power spectrum for an entire gesture can be obtained through the time-averaged scalogram (Figure 5, right):

P (𝑠) = 𝑁 -1 ∑ 𝑛=0 |𝑊 𝑛 (𝑠)| 2 , ∀𝑠 ∈ {𝑠 𝑚𝑖𝑛 , • • • , 𝑠 𝑚𝑎𝑥 } (10)
Similarly, the energy envelope of the gesture can be obtained from the scale-averaged scalogram (Figure 5, bottom): We further abstract the temporal and spectral information by taking the normalized moments of each distribution, raising 8 parameters for the description of a single gesture: Spectral Centroid, Spectral Variance, Spectral Skewness, Spectral Kurtosis, Temporal Centroid, Temporal Variance, Temporal Skewness, Temporal Kurtosis. While this description remains simple, it is invariant to the scale of the movement data. This invariance is essential to the consistency of the analysis, considering that participants' energy in the gesture performance is extremely variable in the database.

Ē𝑛 = 𝑠 𝑚𝑎𝑥 ∑ 𝑠=𝑠 𝑚𝑖𝑛 |𝑊 𝑛 (𝑠)| 2 , ∀𝑛 ∈ {0, • • • , 𝑁 -1} (

Average Ridge Descriptors.

A similar description can be derived from features obtained through the tracking of frequency ridges. The wavelet spectrum representation of movement data provides rich information about the frequential content of dynamic gestures. In particular, periodic and impulsive gestures result in clear ridges in the scalogram centered around the fundamental frequency of the movement. We developed a method for tracking frequency ridges in the scalogram, drawing upon methods from multi-target tracking in computer vision and radar applications.

Our method is described in detail in Appendix A.

As result of the tracking process, several target ridges are identified and their amplitude Ā, frequency F , and variance W are continuously estimated along the gesture. This allows to separate more clearly the contribution of concurrent frequential modes in the gesture. As before, we can consider time-averaged ridge descriptors as the basis for high-level description of gestures. In particular, for a set of 𝐾 ridges, we can compute 3𝐾 parameters per ridge as:

F (𝑘) = 𝑁 1 𝑛=0 𝐹 (𝑘) 𝑛 ∀𝑘 ∈ {1, • • • , 𝐾 } Ā(𝑘) = 𝑁 1 𝑛=0 𝐴 (𝑘) 𝑛 ∀𝑘 ∈ {1, • • • , 𝐾 } W (𝑘) = 𝑁 1 𝑛=0 𝑊 (𝑘) 𝑛 ∀𝑘 ∈ {1, • • • , 𝐾 } (12)
The description of the temporal envelope can be obtained independently for each ridge from the estimated ridge amplitude 𝐴 (𝑘)

1:𝑁 -1 .

Recognition of Movement

Qualities. We evaluated whether our method could be used as a mean to automatically recognize the movement qualities defined manually. For this we followed a standard 5-fold cross-validation method where the 35 participants used in the experiment are randomly assigned to 5 groups. This guarantees that the gestures of a same participant cannot be found both in the training and test sets. The classification was performed using Gaussian Mixture Models (GMMs) [START_REF] Murphy | Machine learning: a probabilistic perspective[END_REF]. We used 5 full-covariance Gaussian components per GMM, with variance regularization.

As baseline, we computed the accuracy score between the annotation of the two human annotators. Once again, the inter-annotator accuracy of 0.67 emphasizes a relative agreement of the annotators, with disagreements regarding the smooth and dynamic categories.

We compare several gesture-level features, derived either from the spectral and temporal moments of the scalogram, or from the ridge tracking algorithm. For the moments representation, we use the first two moments of the scalogram averaged either in time or scale, raising four descriptors per gesture: the temporal and wavelet spectral centroid and variance. Gesture-level descriptors derived from the ridge tracking algorithm have a similar form, with simplified temporal modeling. The best scores were obtained by using only the duration information of each gesture with the average frequency of each ridge -amplitude and spread did not significantly improve the classification accuracy. For consistency, in multi-target tracking, the ridges are ordered in descending intensity.

Table 2 reports the classification accuracy with regards to each annotator's reference labels. With a 2s window size, the WFT gives significantly lower accuracy scores than the description based on the continuous wavelet transform.

The results obtained using continuous ridge tracking are superior to the accuracy obtained using the moment-based description. This might indicate that ridge tracking is more robust to noise than the moments. However, no significant difference in accuracy is found between 1, 2 and 3 ridges for the recognition of gesture primitives on the Abstract categories, which might indicate that all gestures are sufficiently represented using a single frequential mode.

Descriptor

Annotator 1 Annotator 2 CWT WFT CWT WFT moments 0,63 0,58 0,57 0,54 1 ridge 0,68 0,42 0,62 0,43 2 ridges 0,67 0,25 0,59 0,24 3 ridges 0,66 0,59

Table 2. Classification accuracy of movement qualities from gesture-level descriptors. The classification was performed using GMMs with 5 full-covariance Gaussian components. Each accuracy score was computed on 5-fold cross-validation on the imitations of Abstract sounds from 35 participants, where participants from the test set are absent from the training set.

While the recognition might appear low, the confusion matrix provide us with a more general view of the results. recognition rate is due to specific confusions. Specifically, we found that most of the recognition errors occur between the periodic and shaky categories. This suggest that both categories could be merged, which is supported by further qualitative analysis: some gestures are visually associated to the shaky categories, due to apparent 'random' motion of the hand and finger movements, but the wrist movements, which are actually the ones measured, remain globally periodic. Merging the periodic and shaky categories led to an improvement of the accuracy using all descriptors, that reaches about 80% using ridge tracking descriptors. The other sources of inaccuracy arise from the definition of the boundary between smooth and dynamic gestures. Once again, smooth and dynamic categories are defined using a threshold that is subjective to the annotator, while their union forms a continuous space with varying frequency and intensity.

Representation Learning using Non-negative Matrix Factorization

The task of recognition of human-defined movement qualities is known to be difficult, as seen in the previous section.

Another approach is to use data to derive "movement qualities" using a non-supervised task. We refer this as representation learning. Thus, we investigated whether scalograms can represent first basic gestures primitives, then leading to define movement qualities by using clustering techniques . As previously reported, one possibility is to simply apply statistics to the scalogram to extract a global representation. In order to better characterize specific patterns appearing in the scalogram, we introduce an unsupervised method based on Non-negative Matrix Factorization (NMF) [START_REF] Daniel | Learning the parts of objects by non-negative matrix factorization[END_REF]. NMF is similar to Principal Component Analysis (PCA), but working strictly with non-negative matrices. The idea is to find a limited set of components (i.e. primitives) from a dataset of scalograms stored in a non-negative matrix 𝑉 (the scalogram is non-negative by definition). The components, represented by a non-negative matrix 𝐻 , allow for expressing any scalogram from the dataset (and by extension other scalogram) as a weighted sum of the components. The weights are also represented as a non-negative matrix. The mathematical description of the method is available in Appendix B.

The amplitude of each scalogram was normalized between 0 and 255 (treated as an image of float values, with a fixed size of 64 scale bands × 500 normalized time steps. Since both time is resampled to 500 points and amplitude is normalized to 255, both the average time and amplitude information are ignored (for this step, but these parameters will be reintroduced in the clustering operation, see next section) Then, the 𝑉 matrix is obtained by concatenating scalogram, flattened to vectors of size 32000 (64 × 500). Finally, using equations 19 and 20, the first 𝑘 = 25 components are computed. They are reported in Figure 7.

Clustering.

Each gesture is represented by a vector containing the weights of the primitives, along with the average time and energy (total energy of the original scalogram). In order to discover significant gestures primitives, the 27 dimensions 5 feature space previously described is clustered. In order to apply the clustering algorithm, we transform the initial descriptors distribution, which are close to an exponential distribution, to a distribution that is close to a normal distribution, by applying the function 𝑙𝑛(𝑥 + 1). Finally, the dataset is standardized by centering and scaling each feature independently (zero mean and unit variance distribution).

The clustering is performed using a K-Means algorithm on all gestures in the dataset. The cluster number was set to six to compare the automatic clustering to the six primitives defined manually. The six primitives are reported in Fig. 8 as reconstructed scalograms. It appears that most clusters occupy specific time-frequency regions of the scalograms.

While clusters 1 and 3 (left column of Figure 8 occupy similar time-frequency regions, they are distinct in term of average time and energy. .

These primitives extracted using unsupervised learning can be compared to the movement qualities defined manually using a confusion matrix, as displayed in Figure 9. Overall, all the manual movement qualities can be expressed as a combination of our manually defined primitives. Specifically, we found that the primitive 1 (see Figure 7) can be considered mainly as the combination of impulse and dynamic gestures, primitive 2 as steady gestures, primitive 3 being equivalent to impulse, primitive 4 is the combination of periodic and dynamic, primitive 5 is periodic and shaky, primitive 6 is a combination of smooth and dynamic. This confirms that this unsupervised approach seems to be a valuable method to discover and quantify movement qualities from data. In particular, the association of data-driven primitives and manually defined movement qualities is consistent with the confusion matrix obtained on a recognition task in the previous section.

DISCUSSION AND CONCLUSION

We presented in this paper an approach to movement analysis based on the Continuous Wavelet Transform. While using such a well established signal processing technique is not novel per se, we believe that we provided researchers and artists in the MOCO community with several key elements to facilitate experimentation with this approach. We summarize below the main contributions and provide elements of discussion.

First, in contrast to fields such as audio processing that have well established several low level analysis methods for sound, the movement and computing community cannot currently rely on widely shared methods. Towards such a goal, we first recall why the Continuous Wavelet Transform presents several advantages in comparison with the Windowed Fourier Transform when working with movement and gesture data. The CWT provides an elegant solution to guarantee workable resolution from low to high frequencies. An important difference of movement data compared with audio signals relies in the importance of low frequency components (< 1 Hz) that make the WFT difficult to use in practice. The CWT indeed offers a better compromise to characterize movement data that includes smooth movements with important low frequency content, and more impulsive gestures presenting time-localized high frequency content.

Our results on the recognition of manually annotated movement qualities confirm that the CWT outperforms the WFT.

Overall, we believe that the CWT offers a promising low-level representation that forms the base for descriptors of particular temporal and spectral movement patterns. We proposed several approaches using either a simple statistical description of the scalogram, frequency tracking, or unsupervised learning of particular patterns through Non-negative Matrix Factorization. We believe that the prospect of using the CWT to derive high-level movement properties using non-supervised approaches is particularly promising. Interestingly, the various representations can be computed in a way that fuses information from the sensor's different dimensions, effectively making the representation invariant to orientation or sensor placement. The proposed approach still has a number of limitations. First, the fusion of scalograms across sensor channels makes for a representation that is somehow invariant to orientation. However, many applications involve gestures where spatial content is meaningful, in this case the CWT would need to be combined with other descriptors characterizing trajectories and spatial aspects of movement. Second, even with the proposed online approximation, the computational complexity of the continuous wavelet transform remains high, and its application to embedded systems with low computing power might be difficult. Finally, while the proposed methods represent a first step in exploiting the potential of the CWT, more elaborate approaches building could be developed, in particular by including phase information to complement the scalogram. Yet, the CWT seems particularly appropriate for movement presenting some level of periodicity, and further research would be necessary to understand its potential for more transient gestures.

We also elaborated on practical issues regarding the implementation of the CWT either offline or online for movement analysis. We provide open-source code as well as a Max external to facilitate its use in interactive application. Our online implementation can be used in real-time through a set of approximations and optimization methods. It has been used in several interactive applications, notably for tracking the fundamental frequency of walking or breathing [START_REF] Prpa | Attending to Breath: Exploring How the Cues in a Virtual Environment Guide the Attention to Breath and Shape the Quality of Experience to Support Mindfulness[END_REF],

and in an interactive system for music conducting dedicated to a young public 6 , where the average tempo is estimated from gestures, and used to control the speed of an orchestra recording. One characteristics of our implementation is that we use a minimal delay in each frequency band. While this distorts the spectral representation because the frequency bands are not aligned temporally, in our experience this characteristics can be interesting in interactive scenarios. Indeed, while latency is not essential in low-frequency movements, having low latency in higher-frequency movement enables the design of interactions that are reactive with impulsive gestures.

To conclude, we believe that the CWT holds promise as a low-level representation that could be used to build higher-level movement descriptors, in particular for modeling movement qualities. We hope that the work reported in this article can foster further investigations of the method in the movement and computing community.

A FREQUENCY RIDGE EXTRACTION USING TARGET TRACKING

In this section, we describe the implementation of a multi-target tracker for characterizing multiple frequency modes in dynamic gestures.

A.1 Multi-target Track-before-detect in the Wavelet Domain

Our approach follows recent developments in multi-target tracking in Radar tracking and computer vision systems. In particular, we consider track-before-detect (TBD) approaches to multiple object tracking where the target detection and tracking is performed jointly [START_REF] Boers | Multitarget particle filter track before detect application[END_REF][START_REF] Samuel | Multiple transition mode multiple target track-before-detect with partitioned sampling[END_REF][START_REF] Qiu | A Survey of Motion-Based Multitarget Tracking Methods[END_REF][START_REF] Salmond | A particle filter for track-before-detect[END_REF]. Such approaches differ from standard detect-before-track methods where a set of potential targets are first detected -e.g. from image segmentation in the case of object tracking, -and then filtered using dynamic system modeling.

In TBD, the decision is made at the end of the processing chain, and therefore integrates all information over time.

As a result, the tracking is less sensitive to errors in the detection step. Moreover, recursive approaches to TBD avoid the classical problem of data association between tracked targets and detected candidates: no thresholding is necessary, which removes the need for explicit association. Finally, it makes it possible to estimate additional parameters about the target, such as its intensity or size.

A.1.1 Problem Definition. We aim at simultaneously tracking a maximum number 𝐾 of frequency components of the movement -thereafter called 'targets', -from measurements of the wavelet power spectrum computed from one or several sensors. We assume that each target 𝑘 ∈ {1, • • • , 𝐾 }, if present, is specified by a time-varying position 𝑥 (𝑘) in the wavelet scale domain -which is associated with the movement's fundamental frequency, -and a time-varying amplitude 𝑎 (𝑘) . We assume that each frequential component is projected over the wavelet spectrum as a Gaussian, centered around 𝑥 (𝑘) and spread on the wavelet space with width Σ (𝑘) . Our goal is to estimate, from measurements of wavelet power spectrum, the number of components present at the current time step as well as their characteristic parameters 𝑓 (𝑘) , 𝑎 (𝑘) , Σ (𝑘) 𝐾 𝑘=1 .

A.1.2 System Dynamics. We assume that the position of each target evolves according to linear dynamics. We further assume a constant velocity model, meaning that, on a short time scale, each ridge evolves linearly in the wavelet domain. 7 For each target 𝑘 ∈ {1, • • • , 𝐾 }, we assume the following time-invariant state equation:

𝒔 𝑡 +1 = 𝒇 (𝒔 𝑡 ) + 𝒈(𝒔 𝑡 )𝒘 𝑡 (13)
where the process noise 𝒘 𝑡 is assumed to be standard Gaussian white noise. The hidden state 𝒔 𝑡 at time step 𝑡 for a given target is composed by the target's position, velocity, amplitude, and spread:

𝒔 𝑡 =           𝑥 𝑡 𝑣 𝑡 𝑎 𝑡 Σ 𝑡           (14)
The system dynamics function 𝒇 and process noise input model 𝒈 can be defined under a time-invariance assumption as

𝑓 (𝒔 𝑡 ) = 1 𝛿𝑡 0 0 0 1 0 0 0 0 1 0 0 0 0 1 𝒔 𝑡 , 𝑔(𝒔 𝑡 ) = 𝜎 𝑥 𝛿𝑡 3 /3 𝜎 𝑥 𝛿𝑡 2 /2 0 0 𝜎 𝑥 𝛿𝑡 2 /2 𝜎 𝑥 𝛿𝑡 0 0 0 0 𝜎 𝑎 𝛿𝑡 0 0 0 0 𝜎 Σ 𝛿𝑡 ( 15
)
where 𝛿𝑡 is the sampling period and 𝜎 𝑥 , 𝜎 𝑎 and 𝜎 Σ are the expected maximum acceleration of the target, maximum change in amplitude, and maximum change in spread.

A.1.3 Measurement Model. By analogy with radar sensors or optical system, we consider the measured power as reflected on a unidimensional observation space in the wavelet domain. A measurement 𝒛 𝑡 therefore consists of 𝑁 power measurements 𝑧 𝑖 𝑡 in the scale domain, where 𝑁 is the number of bands of the CWT. In a multi-target setting, we hypothesize that the power measurements in the wavelet spectrum result from the superposition of the power emitted by each target, if present, yielding the following form:

𝒛 𝑡 = 𝐾 ∑ 𝑘=1 𝛿 𝑘 𝒉 𝒔 (𝑘) 𝑡 + 𝒏 𝑡 ( 16
)
where 𝛿 𝑘 is a binary indicator variable specifying if target 𝑘 is present, and 𝒉 𝒔 (𝑘) 𝑡

is the reflected power of the 𝑘th target in the wavelet space, assumed to be Gaussian:

ℎ 𝑖 𝒔 (

A.2 Particle Filter Implementation

We derived a a Bayesian solution to the estimation problem introduced in the previous section. While the state dynamics are linear and could be optimally solved analytically, the non-linear nature of the observation model makes exact inference intractable. The track-before-detect can be solved using Sequential Monte Carlo estimation -also called particle filtering. Our particle filter implementation follows a standard Particle Filter implementation, as described in [START_REF] Boers | Multitarget particle filter track before detect application[END_REF]. An example of ridge tracking from the scalogram is shown in Figure 10.

where k is a number of basis components. 𝑉 ∈ R 𝑓 ×𝑡 is the original non-negative data, 𝑊 ∈ R 𝑓 ×𝑘 the basis vector decomposition and 𝐻 ∈ R 𝑘×𝑡 the weight matrix. Each column of V represents the whole dataset. Here, each matrix row represents one scalogram. In W, each column is referred to as a basis vector, representing the k components. Rows of H stand for the gain of corresponding basis vector. k is chosen so that (𝑓 + 𝑡) × 𝑘 < (𝑓 × 𝑡). The product 𝑊 × 𝐻 presents a compressed form of the original data V.

Technically, W and H are calculated resolving the equation: min

There are several approaches to solve [START_REF] Lin | Projected gradient methods for nonnegative matrix factorization[END_REF]. In this paper, we used the Alternating Nonnegative Least Squares Matrix Factorization Using Projected Gradient or LSNMF as described in [START_REF] Lin | Projected gradient methods for nonnegative matrix factorization[END_REF]. It converges fast and provides small approximation errors.

Then, each gesture scalogram 𝑏 (new or already known) is described as a linear combination of the k basis vectors, where x is the weight of each component to be calculated by resolving:

arg min 𝑥 = ∥𝐴𝑥 -𝑏 ∥ 2 for 𝑥 >= 0 [START_REF] Makihara | Gait Recognition Using a View Transformation Model in the Frequency Domain[END_REF] where A the vector of components W. Only additive combinations are allowed. Any given scalogram does not have to use all the available components.

Fig. 1 .

 1 Fig.1. Time-frequency power spectrum of an acceleration signal (i.e. raw signal from a 3D accelerometer in arbitrary units) using the CWT and WFT. We used a window size of 1s and 2s for the WFT, and 8 bands per octave on frequency range [0.2; 50] Hz for the CWT.

Fig. 2 .

 2 Fig. 2. Illustration of the online implementation of the Continuous Wavelet Transform.

4. 3 . 2

 32 Optimization by Multi-rate Approximation. Computing the online CWT can be intensive at high framerates as the number of bands per octave increases, and as the analysis requires low-frequency components involving large window sizes. To alleviate this issue, we propose two optimization schemes of the online transform based on a multi-rate representation of the wavelets and of the signal.

Fig. 3 .

 3 Fig. 3. Example of online approximation of the continuous wavelet transform computed over an acceleration signal (i.e. from an accelerometer) from a dataset of gestural imitations of sounds. The approximation was computed with carrier frequency 𝜔 0 = 5 and windowing factor 𝜆 = 3. The x-axis unit the sample number, using sampling rate of 200 Hz. The y-axis unit of the acceleration signal is in 𝑔

Fig. 4 .

 4 Fig. 4. Example of online approximation of the continuous wavelet transform computed over an acceleration signal from from a dataset of gestural imitations of sounds. The delays in each frequency band are compensated for comparison with the offline estimate. The approximation was computed with carrier frequency 𝜔 0 = 5 and windowing factor 𝜆 = 3. The x-axis unit the sample number, using sampling rate of 200 Hz. The y-axis unit of the acceleration signal is in 𝑔

Fig. 5 .

 5 Fig.5. Description of a gesture through its scalogram's energy envelope and power spectrum. Global descriptors can be extracted using the moments of these distributions (centroid, variance, skewness, kurtosis).

Figure 6 Fig. 6 .

 66 Figure6depicts the confusion matrix associated with ridge-based classification. First, it shows that the some movement qualities are better recognised than others (impulse is recognized at approximately 90%), and that the relatively low

Fig. 7 .

 7 Fig. 7. The 25 basic components of the dataset. Each one represents a gesture primitive spatially localized (time and amplitude) in our gesture scalogram dataset. The components are represented as images of 64 scale bands × 500 normalized time steps.

Fig. 8 .

 8 Fig. 8. Each primitive can be represented as a scalograms reconstructed from the NMF descriptors, and average Time and Energy descriptors. Top: primitives 1, 2, and 3 Bottom: primitives 4, 5 and 6. Average times and energy are (respectively for primitives 1 to 6): Time 0.86, 5.87, 0.67,4.70, 4.09,2.93 [s], and Energy & 1.86𝑒 + 09, 1.18𝑒 + 09, 1.00𝑒 + 09, 8.33𝑒 + 09, 6.01𝑒 + 09, 1.73𝑒 + 09

Fig. 9 .

 9 Fig. 9. Comparison of the unsupervised movement qualities using NMF with the manual qualities.

,

  ∀𝑖 ∈ {0, • • • , 𝑁 -1}(17)A.1.4 Track Birth and Death. We propose to model the possibility of birth and death of the frequency ridges using a jump Markov process, as often proposed in multi-target tracking systems[START_REF] Boers | Multitarget particle filter track before detect application[END_REF][START_REF] Samuel | Multiple transition mode multiple target track-before-detect with partitioned sampling[END_REF]. Instead of using a single mode state variable, as proposed by Boers et al.[START_REF] Boers | Multitarget particle filter track before detect application[END_REF], we keep a vector of binary indicator variable 𝜹 = [𝛿 1 , • • • , 𝛿 𝐾 ] ⊤ . We assume that track birth and death are ruled a first-order Markov process where the probability of switching between different modes -i.e. configurations of the targets present at a given time steps -only depends on the mode at the previous time step. Constraints on track birth and death are then specified through a transition matrix that allows the birth of a new track with probability 𝑝 birth and the death of a new track with probability 𝑝 death . Additionally, it prevents the simultaneous birth and death of two tracks as well as the simultaneous birth of several tracks, in order to avoid target switching.

Fig. 10 .

 10 Fig.10. Example of Ridge tracking from the scalogram. The top plot represents the scalogram computed from the 3D acceleration of an example gesture from a dataset of gestural imitations of sounds. The middle and bottom plots respectively represent the tracked ridge frequency and intensity. We used a single target with 100 particles.

(

  𝑉 𝑖,𝑗 -(𝑊 𝐻 ) 𝑖,𝑗 ) 2 subject to 𝑊 𝑖,𝑎 ≥ 0, 𝐻 𝑏,𝑗 ≥ 0, ∀𝑖, 𝑎, 𝑏, 𝑗 .

Table 1 .

 1 Normalized Mean Squared Error of the online approximation of the scalogram compared with the standard estimate. Results are averaged over all imitations from all participants on the Abstract sound family of the ANON dataset (see Section 5. The results are displayed according to the size of the approximation window relative to the wavelet's e-folding time in each frequency band.

	Win. Factor NMSE (%) Comp. Time (10 -3 ms/loop)
	1	54.676228	1.867
	2	21.404965	3.687
	3	4.632857	5.502
	4	1.293430	7.362
	6	0.008430	11.035
	similar decimation of the signal. The wavelet power spectrum is therefore evaluated at a lower framerate for
	low-frequency components, which provides a sparser representation of the scalogram.

We report here the 'complete' for of the Morlet Wavelet. As discussed by Addison et al.[START_REF] Paul S Addison | Wavelet transforms and the ECG: a review[END_REF], considering that the correction term is negligible for 𝜔 0 > 5., most articles in the literature use a truncated form for the Morlet wavelet: Ψ 0 (𝜂) = 𝜋 -1/4 𝑒 𝑖𝜔 0 𝜂 𝑒 -𝜂

/2 . However, in some cases it can be interesting to use smaller values of 𝜔 0 in order to get a high temporal localization

https://forum.ircam.fr/projects/detail/mubu/

https://github.com/julesFrancoise/coda

See https://www.ircam.fr/projects/blog/multimodal-database-of-vocal-and-gestural-imitations-elicited-by-sounds/. This dataset was built during the Skat-VG project (https://cordis.europa.eu/project/rcn/110562/factsheet/en)

the 25 basic vectors of the NMF + average time and energy.

see https://ircamamplify.com/en/achievements/maestra-maestro-with-la-philharmonie-des-enfants/, accessed May 05 2022

We assume that the scales are distributed as fractional powers of 2, meaning that the associated distribution in the Fourier domain is logarithmic in frequency.

ACKNOWLEDGMENTS

This research was supported by the project SkAT-VG with financial support of the Future and Emerging Technologies (FET) programme within the Seventh Framework Programme for Research of the European Commission, under FET-Open grant number: 618067. This research was partially supported by the ELEMENT project (ANR-18-CE33-0002) from the French National Research Agency.


Imen Trabelsi

Sensor-based Activity Recognition using Deep Learning: A Comparative Study

Keywords: CCS Concepts:, Human-centered computing → Gestural input;, Computing methodologies → Machine learning Human Activity recognition, Inertial Sensors, Movement Computing, Continuous Wavelet Transform, Deep Learning, Convolutional Neural Networks

With the wide availability of inertial sensors in smartphones and connected objects, interest in sensor-based activity recognition has risen. Yet, recognizing human actions from inertial data remains a challenging task because of the complexity of human movements and of inter-individual differences in movement execution. Recently, approaches based on deep neural networks have shown success on standardized activity recognition datasets, yet few works investigate systematically how these models generalize to other protocols for data collection. We present a study that evaluates the performance of various deep learning architectures for activity recognition from a single inertial measurement unit, on a recognition task combining data from six publicly available datasets. We found that the best performance on this combined dataset is obtained with an approach combining the continuous wavelet transform and 2D convolutional neural networks.

INTRODUCTION

Computational movement modeling is a central topic in the Movement and Computing (MOCO) community, as it is the foundation for the design of interactive systems involving gesture and whole-body motion. The MOCO community has witnessed an increasing number of contributions to new conceptual frameworks for movement analysis [START_REF] Camurri | The Dancer in the Eye: Towards a Multi-Layered Computational Framework of Qualities in Movement[END_REF], signal processing methods [START_REF] Glowinski | Toward a minimal representation of affective gestures[END_REF][START_REF] Larboulette | A review of computable expressive descriptors of human motion[END_REF][START_REF] Volpe | Computational models of expressive gesture in multimedia systems[END_REF] and applications of Machine Learning (ML) to movement analysis and recognition [START_REF] Coupeté | A User-Adaptive Gesture Recognition System Applied to Human-Robot Collaboration in Factories[END_REF][START_REF] Françoise | Movement Sequence Analysis Using Hidden Markov Models[END_REF].

Yet, using ML for analyzing movement data from wearable sensors remains challenging, especially in applications domains where collecting and annotating data remains time-consuming or impractical, for instance in music and performing arts. Approaches such as interactive machine learning have shown success for designing gesture recognition systems from few examples [START_REF] Bevilacqua | Continuous Realtime Gesture Following and Recognition[END_REF][START_REF] Fiebrink | Human Model Evaluation in Interactive Supervised Learning[END_REF][START_REF] Françoise | Motion-Sound Mapping through Interaction: An Approach to User-Centered Design of Auditory Feedback Using Machine Learning[END_REF][START_REF] Françoise | Probabilistic Models for Designing Motion and Sound Relationships[END_REF]]. Yet, these approaches often rely on hand-crafted features and standard ML models, which can limit their use for modeling higher-level movement characteristics and concepts such as movement qualities [START_REF] Camurri | Computational Models of Expressive Movement Qualities in Dance[END_REF][START_REF] Fdili | Movement Qualities as Interaction Modality[END_REF][START_REF] Fdili Alaoui | Seeing, Sensing and Recognizing Laban Movement Qualities[END_REF].

In this project, we are interested in drawing upon Deep Neural Networks (DNNs) to learn representations of movement from inertial sensor data. While our goal is to apply such techniques to a broad spectrum of movement and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2022 Association for Computing Machinery. Manuscript submitted to ACM computing applications, we consider in this paper the task of sensor-based Human Activity Recognition (HAR), where deep learning has shown success in recent years [START_REF] Chen | Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities[END_REF][START_REF] Ignatov | Real-time human activity recognition from accelerometer data using Convolutional Neural Networks[END_REF][START_REF] Mekruksavanich | Lstm networks using smartphone data for sensor-based human activity recognition in smart homes[END_REF][START_REF] Ordóñez | Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition[END_REF][START_REF] Qi | A fast and robust deep convolutional neural networks for complex human activity recognition using smartphone[END_REF].

HAR based on wearable sensors finds applications in a wide range of areas such as health an well being, monitoring physical activity or surveillance. Sensor-based HAR entails recognising a person's activity (e.g. walking, running, etc.) from their physical movements by analysing data generated from on-body inertial sensors. These wearable sensors usually embed accelerometers, gyroscopes and magnetometers. Traditional approaches to HAR apply classic ML modelsincluding Random Forests (RF), Bayesian Network (BN) or Support Vector Machines (SVM), -to movement features that are engineered by hand. In recent years, DNN-based methods have achieved tremendous success in sensor-based HAR, and several architectures have been proposed, including convolutional and recurrent neural networks [START_REF] Chen | Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities[END_REF][START_REF] Ignatov | Real-time human activity recognition from accelerometer data using Convolutional Neural Networks[END_REF][START_REF] Muralidharan | 1D Convolution approach to human activity recognition using sensor data and comparison with machine learning algorithms[END_REF][START_REF] Qi | A fast and robust deep convolutional neural networks for complex human activity recognition using smartphone[END_REF]. Yet, to date few approaches have investigated the potential of time-frequency representations using the wavelet transform in conjunction with convolutional neural networks (CNNs) which are extremely successful in computer vision. Considering that human activities captured with wearable sensors often involve characteristic temporal patterns, we hypothesize that the Continuous Wavelet Transform (CWT), that provides an accurate time-frequency representation, would result in a valuable representation for recognizing activities. Furthermore, algorithms are usually evaluated on standard datasets that are publicly available, that are built with a fixed protocol and a limited number of activities.

Our contributions in this paper are twofold. First, rather than considering only individual datasets, we propose a more challenging task by combining activities from different datasets. This process introduces more variability in data collection protocols and sensor placement, and challenges the generalization power of HAR methods. Second, we propose a new method where accelerometer and gyroscope signals are first analyzed using the Continuous Wavelet Transform (CWT) to generate a 2D image representation, and then classified using a CNN 2D. We demonstrate that wavelet CNNs achieve competitive accuracies and outperformed the traditional statistical and other deep learning methods when combining activities from different datasets.

The remainder of this paper is structured as follows: Section 2 summarizes the related work. Section 3 presents the proposed methods and metrics. Section 4 describes the experimental setup and presents the results, Section 5 contains a discussion of the obtained results, and Section 6 concludes this paper and proposes future improvements.

RELATED WORK

HAR methods consists in monitoring and analyzing a person's activity behaviors to enable computing systems to proactively assist users in various tasks. Significant progress has been made by conventional ML methods. However, with the rapid development of DL, there is an emerging interest and effort in applying it. Figure 1 presents a typical flowchart of HAR from inertial measurement unit (IMU) sensor data using approaches based on feature engineering or approaches based on deep neural networks.

Approaches based on Feature Engineering

Conventional approaches to HAR often draw upon classic learning algorithms applied to handcrafted features.

De Leonardis et al. [START_REF] De Leonardis | Human Activity Recognition by Wearable Sensors: Comparison of different classifiers for real-time applications[END_REF] explored five classifiers (SVM, decision tree, K-Nearest Neighbor (KNN), Feedforward Neural Network (FNN) and Naïve Bayes (NB) for classifying eight activities performed by fifteen subjects from their own dataset. The results showed that all classifiers were able to correctly recognize more than 90% of activities, and that best performances were obtained by KNN. Catal et al. [START_REF] Catal | On the use of ensemble of classifiers for accelerometer-based activity recognition[END_REF] conducted several experiments on the Wisdm dataset with decision trees, logistic regression, and multi-layer perceptron algorithms, and combines these classifiers with the average of probabilities combination rule. Gao et al. [START_REF] Gao | Evaluation of accelerometer based multi-sensor versus single-sensor activity recognition systems[END_REF] evaluated the effect of multiple sensors on HAR. They While conventional approaches have made progress in HAR, they remain limited in several aspects. First, traditional methods require handcrafted features, which heavily relies on human expertise. This lengthy process is task-dependent and for more general tasks, this will show performance degradation [START_REF] Muhammad Attique Khan | Hand-crafted and deep convolutional neural network features fusion and selection strategy: an application to intelligent human action recognition[END_REF]. Second, handcrafted features often refer to some statistical information. They can only be used to recognize basic physical activities such as walking or standing, and can hardly infer high-level or context-aware activities [START_REF] Li | Human Activity Recognition Based on Residual Network and BiLSTM[END_REF].

Approaches based on Deep Neural Networks

Unlike conventional techniques, approaches based on DNNs can directly process raw data and have outperformed conventional approaches especially for the case of heterogeneous, large and unfamiliar datasets [START_REF] Georgiou | A survey of traditional and deep learning-based feature descriptors for high dimensional data in computer vision[END_REF][START_REF] Niall | Deep learning vs. traditional computer vision[END_REF]. There are important reasons behind deep learning's success: the availability of big data, advances in learning algorithms and hardware, and increase in computing power. Chen et al. [START_REF] Chen | Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities[END_REF] provide a recent review of sensor-based human activity recognition using deep learning. Muralidharan et al. [START_REF] Muralidharan | 1D Convolution approach to human activity recognition using sensor data and comparison with machine learning algorithms[END_REF] compared classic approaches based on feature engineering with standard ML algorithms, as introduced in the previous section, with 1D Convolutional Neural Network (CNNs) on UCI HAR dataset. They found that the SVM and the proposed 1D convolution neural network were the best-performing models. Qi et al. [START_REF] Qi | A fast and robust deep convolutional neural networks for complex human activity recognition using smartphone[END_REF] proposed a fast and robust CNN to perform complex activity recognition from smartphone sensors and the accuracy reached 94.18% in the experiment. Ignatov [START_REF] Ignatov | Real-time human activity recognition from accelerometer data using Convolutional Neural Networks[END_REF] proposed using a 1D CNN for local feature extraction together with simple statistical features. The proposed CNN architecture consists of a convolutional layer, a max-pooling layer, the output of the max-pooling layer is then flattened and stacked together with additional statistical features, followed by a fully-connected layer. The authors obtained 97.62% accuracy on the UCI HAR dataset with the proposed method.

In [START_REF] Mekruksavanich | Lstm networks using smartphone data for sensor-based human activity recognition in smart homes[END_REF][START_REF] Zebin | Human activity recognition from inertial sensor time-series using batch normalized deep LSTM recurrent networks[END_REF], the authors presented an LSTM network for the classification of six activities from UCI HAR dataset.

The first one achieves 92% average accuracy, while the second achieves 91.61% of accuracy. To improve accuracy of predictions, researchers in [START_REF] Brownlee | Deep learning for time series forecasting: predict the future with MLPs, CNNs and LSTMs in Python[END_REF][START_REF] Mekruksavanich | Lstm networks using smartphone data for sensor-based human activity recognition in smart homes[END_REF][START_REF] Ordóñez | Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition[END_REF] merged functionalities of CNN and LSTM architectures to create a new one, the so-called CNN-LSTM. They used the outputs of CNN as input data for LSTM and obtained good results.

Alemayoh et al. [START_REF] Tsige Tadesse Alemayoh | New sensor data structuring for deeper feature extraction in human activity recognition[END_REF] proposed a CNN with a double-channeled time-domain input to identify activities. In the proposed architecture, data were shaped into double-channels to extract deep temporal and spatial features of the signals. In [START_REF] Wagner | Activity recognition using inertial sensors and a 2-D convolutional neural network[END_REF], the authors split the sensor data into small segments of 128 data points. A frequency representation is taken from these windows by applying the Discrete Fourier transform (DFT). A pre-trained deep neural network (AlexNet) extracts features from the produced images. For the classification task, a SVM is choosen with linear and nonlinear kernels. An overall accuracy of 94% was reached on the UCI HAR dataset. While a number of neural network architectures have been proposed for sensors-based HAR, few works have investigated the potential of time-frequency representations such as the continuous wavelet transform [START_REF] Nedorubova | Human Activity Recognition using Continuous Wavelet Transform and Convolutional Neural Networks[END_REF]. Moreover, most algorithms are evaluated on published dataset in isolation. In this paper, we propose a new method for sensor-based HAR that applies 2D CNN to scalograms computed from IMU data. We evaluate the proposed method against state of the art algorithms, both on isolated datasets and on a combined dataset that aggregates data from various sources.

METHODS

This Section describes the adopted four baseline approaches, as well as our method combining CWT and CNNs.

Baseline Approaches

3.1.1 Random Forests on Statistical Features. We first consider a conventional baseline that applies a standard classifier on a large set of both time-domain and frequency-domain features. Frequency domain signals were obtained by applying a Fast Fourier Transform (FFT). Following Anguita et al. [START_REF] Anguita | A public domain dataset for human activity recognition using smartphones[END_REF], we used 561 features to train a classifier. We experimented with several popular algorithms (KNN, Adaboost, NB and RF) and found that Random Forests (RF) gave higher accuracy in this circumstance. RF classifiers are efficient on huge data and can manage hundreds of input variable without deletion [START_REF] Breiman | Random forests[END_REF]. It is also preferred because of its interpretability and low computational cost.

Convolutional Neural Network (CNN 1D).

CNNs have been actively used and achieved impressive performance in many research area that include activity recognition [START_REF] Münzner | CNN-based sensor fusion techniques for multimodal human activity recognition[END_REF]. CNNs consist of combinations of convolutional, pooling, normalization, and fully connected layers that extract progressive yet semantically rich features from the input data.

CNNs were developed for image classification problems, where the model learns an internal representation of the input image, and produces a batch of 2D feature, in a process referred to as feature learning. This same process can be harnessed on one-dimensional sequences of data, such as in the case of acceleration and gyroscopic sensors. The model learns to extract features from sequences of observations and how to map the internal features to different activity types. CNN are able to capture the spatial information and connections of multimodal sensory data.

In this paper, we adopt the architecture proposed by Ignatov [START_REF] Ignatov | Real-time human activity recognition from accelerometer data using Convolutional Neural Networks[END_REF], that consists of a convolutional layer, followed by one Max pooling layer, followed by a flattening convolutional layer, then one fully-connected layer and finally a softmax classifier. It is denoted CNN1D in the following. The choice of this method is motivated by the fact that this algorithm is computationally efficient, and the obtained results significantly outperforms baseline approaches [START_REF] Ignatov | Real-time human activity recognition from accelerometer data using Convolutional Neural Networks[END_REF].

3.1.3 Long-Short Term Memory Networks (LSTM). Long Short-Term Memory (LSTM) are a special case of Recurrent Neutral Networks (RNN), having the capability to extract dependencies in time-series data and incrementally learn information through time intervals. Hence, they are appropriate for streaming sensory data in human activity recognition.

The internal architecture of LSTM includes several gates where each gate processes the input from the previous gate and forward it to the next gate thereby controlling the flow of information towards the final output.

We compare two LSTM architectures. The LSTM architecture consists of one LSTM layer, followed by one fullyconnected layer and finally a dense layer with a softmax activation function [START_REF] Brownlee | Deep learning for time series forecasting: predict the future with MLPs, CNNs and LSTMs in Python[END_REF]. The CNN-LSTM architecture involves CNN layers for feature extraction on input data combined with LSTMs to support sequence prediction. The architecture seeks to leverage the combined power of both networks [START_REF] Mutegeki | A CNN-LSTM approach to human activity recognition[END_REF][START_REF] Xia | LSTM-CNN architecture for human activity recognition[END_REF]. The architecture consists of two sets of convolutional layers, followed by one Max pooling layer, followed by a flattening convolutional layer. These features are then passed through one LSTM layer, followed by one fully-connected layer and finally a dense layer with a softmax activation function for classification [START_REF] Brownlee | Long short-term memory networks with python: develop sequence prediction models with deep learning[END_REF].

Proposed Method Combining CWT and 2D CNNs

We propose a new approach that draws upon a time-frequency representation of IMU data, combined with 2D convolutional networks, as illustrated in Figure 2. A similar approach has previously been proposed using the windowed Fourier transform [START_REF] Wagner | Activity recognition using inertial sensors and a 2-D convolutional neural network[END_REF]. This approach is motivated by the fact that many daily activities such as walking, running, involve specific temporal patterns that could be analyzed in the spectral domain. Time-frequency representations can then be considered images, and we can take advantage of methods developed in computer vision to analyze spectrograms as images. Rather than using the Short-Term Fourier Transform (STFT), we propose to consider the Continuous Wavelet Transform (CWT), which addresses limitations of the STFT [START_REF] Paul S Addison | Wavelet transforms and the ECG: a review[END_REF][START_REF] Mallat | [END_REF][START_REF] Torrence | A practical guide to wavelet analysis[END_REF]. Wavelets could be considered as viable options when considering feature extraction in activity recognition, as evidenced in [START_REF] Shah Syed | A hierarchical approach to activity recognition and fall detection using wavelets and adaptive pooling[END_REF]. Rather than imposing a fixed window length for analysis, the CWT translates and dilates a wavelet function with short-term influence, raising better resolution in both time and frequency [START_REF] Torrence | A practical guide to wavelet analysis[END_REF]. The CWT also gives flexibility in the choice of the Wavelet basis used for analysis, and in the choice of the frequency range and resolution to use for analysis.

In this paper, we use the Complex Morlet wavelet, which is defined as a plane wave modulated by a Gaussian window.

Using the complex version of the Morlet wavelet enables us to compute the scalogram (analog to the spectrogram with the Fourier transform), by taking the power of the transform. We used 32 frequency bands distributed in log scale, with a fixed number of 4 bands per octave. Applied to signals sampled at 50 Hz, the corresponding frequency range is [0.11, 25] Hz. Since each signal has six components, each signal will also have six scalograms. The scalograms are placed on top of each other to create one single image with six channels, of shape (128, 32, 6). For computational efficiency and reduced memory, the scalograms are downsampled by a factor 4 along the time axis, raising images of shape [START_REF] Mekruksavanich | Lstm networks using smartphone data for sensor-based human activity recognition in smart homes[END_REF][START_REF] Mekruksavanich | Lstm networks using smartphone data for sensor-based human activity recognition in smart homes[END_REF][START_REF] Bevilacqua | Continuous Realtime Gesture Following and Recognition[END_REF].

We can therefore use the images resulting from the mapping from the input sensors into a 2-dimensional space via CWT as input of a 2D CNN.

Given that images have small dimension and low complexity (compared to pictures), we used a simple 2D CNN architecture, summarized in Figure 2-(c), with 2 convolutional layers with 64 filters of size 3x3, interleaved with max pooling layers with kernel size 2 and stride 2, a flattening layer is used then to convert the output of the pooling layer, followed by two dense layers containing 1024, 128 units separately with ReLU activation, dropouts are applied after each dense layer. And the last dense layer, uses softmax as the activation function.

RESULTS

Experimental Protocol

We assessed the performance of the methods presented in Section 3.1 to predict human activities on several datasets. In order to ensure the reproducibility of our results, we chose to train and evaluate our methods on six publicly available datasets, that are summarized in Table 1. To further assess the generalization power of the various methods, we also created a new "combined" dataset aggregating data from these six datasets across five common activities (Walking, Upstairs, Downstairs, Sitting, Standing).

Each dataset was divided into a training set and for test set with a 70/30% split across users, guaranteeing that different users are used for training and evaluation. For the combined dataset, we kept the train and test split from each dataset to constitute the train and test sets.

For consistency, we used similar preprocessing for all of methods. We only used accelerometer and gyroscope data from a single IMU. Signals were resampled at a fixed frame rate of 50Hz, and segmented into 2.56s windows (128 samples) with 50% overlap. We used Scikit-Learn1 for RF and all deep learning algorithms were implemented using Keras. 2 Models were trained using the Adam optimizer with a learning rate of 0.001, with a batch size of 128, for a maximum of 100 epochs with an additional early stopping mechanism.

Activity Recognition on Individual Datasets

In this Section, we examine a comparison between the different methods on individual datasets, including all activities from each dataset. We report the accuracy on the test set of each of the methods in Table 2. Overall, the classifiers performed better with the UCI HAR dataset followed by the MotionSense dataset.

Random Forest performs relatively better than the other methods, obtaining the best scores on 4 datasets: MotionSense, PAMAP2, w-HAR and Wisdm Watch. The proposed CWT-CNN2D yielded an accuracy of 92.46% for UCI HAR and 48.4%

for Wisdm Phone. The results are poor on the Wisdm dataset for all methods. PAMAP2 showed also poor performances, with an accuracy inferior to 63%. We observe an increase in recognition rates for all methods. The best results were given by our proposed method CWT-CNN2D on UCI HAR, w-HAR, Wisdm Phone, and Wisdm Watch datasets, followed by RF applied on statistical features. We also note that the MotionSense dataset has the highest recognition rate, with 93.9%. CWT-CNN2D gives the best recognition rate on the Combined Dataset. In particular, this result demonstrated an increase of approximately 9% over the RF method.

DISCUSSION

We assessed the performance of the several deep learning and conventional methods to predict human activities from IMU data on separate datasets. Random Forests performs relatively better than the other methods followed by our proposed method CWT-CNN2D. The performance of the handcrafted method confirms the years of research experience in HAR community performed by standard HAR procedure: manually extracted features, feature selection and feature classification. Nevertheless, for Wisdm Phone dataset, Wisdm Watch dataset and PAMAP2, the results were relatively poor with all the methods. In fact, the Wisdm dataset have some specific characteristics. It is composed of complex activities. This complexity increases the difficulty of the recognition because of multiple transitions between different motions. Moreover, for this dataset, previous works usually considered user-dependent solutions in which, the models are trained and evaluated for a user using just her/his own data, and trained their models on a subset of activities [START_REF] Mekruksavanich | Enhanced hand-oriented activity recognition based on smartwatch sensor data using lstms[END_REF].

For the PAMAP2 dataset, the poor accuracies are related to the high number of activities but also, because it includes basic movements, household activities and more unusual activities (e.g. rope jumping).

The proposed method combining the CWT with CNNs reached higher accuracy on 4 out of 6 datasets when selecting only common activities across different datasets. Interestingly, this method outperformed all other approaches on the combined dataset. This dataset offers a much larger training and testing database and embeds significant variability, because protocols for data capture differ dramatically across the original datasets. In particular, the combined dataset aggregates IMU data from different types of sensors (e.g. smartphones and smartwatches), placed on different body parts.

This makes the recognition task more difficult than with isolated datasets, and requires models to further generalize.

The CWT-CNN2D approach is particularly efficient with this data, and might be more robust in real-world settings where users' devices might be heterogeneous in type, quality and placement. We hypothesize that the CWT could present a better ability to capture temporal patterns while remaining invariant to positioning and direction. Yet, its performance decreases with a higher number of activities, which means that it might generalize well to new data when the number of activities is limited, but might fail to capture nuances in movement execution when working with a large number of classes. Finally, the proposed method combining CWT with 2D convolutional networks has higher computational complexity than other approaches based on 1D CNNs. Training remains efficient once scalograms are pre-computed for the entire database, and in our experiments the method is still efficient enough for real-time activity recognition, using 1s windows with 50% overlap.

CONCLUSION

In this work, we compared different deep learning techniques for sensor-based human activity recognition under three different scenarios: activities from individual datasets, five common activities from individual datasets and five common activities from a combined dataset. We used data from UCI HAR dataset, MotionSense dataset, PAMAP2 dataset, w-HAR dataset, Wisdm Phone dataset and Wisdm Watch dataset.

We proposed a new method combining the CWT, that provides an accurate time-frequency representation of the original signal, with 2D CNNs that have been extensively studied in image processing. The main findings from the direct comparison of our proposed method CWT-CNN2D against the baseline models is that: (i) CWT-CNN2D reaches a higher recognition rate, especially when the number of activities is small; (ii) it offers better generalization capacity illustrated by a higher recognition rate; (iii) conventional methods based on the combination between statistical feature analysis and Random Forest remain powerful even compared with modern deep learning architectures.

Technically, there is no model which achieves the best results in all tasks, so it is recommended to choose models based on the scenarios. The proposed CWT-CNN2D method appears more robust to variations in sensors placement than other techniques, yet it might remain limited when a large number of activities need to be recognized. We believe that this architecture could form the base of other movement analysis methods which require analyzing temporal dynamics independently from trajectories and directions. In future research, the proposed method will be extended to include a transfer learning approach in order to facilitate the design of movement analysis and recognition systems able to learn from few examples.

Fig. 1 .

 1 Fig. 1. HAR from IMU sensor data comprises three main parts, (a) data capture and pre-processing (accelerometer and gyroscope data); (b) modeling and training, using either feature engineering with standard classifiers or deep learning extract representations from raw data, and (c) activity recognition.

Fig. 2 .

 2 Fig. 2. CWT-CNN2D framework comprises three main parts, (a) data pre-processing for sensor-based HAR (accelerometer and gyroscope); (b) continuous wavelet transform applied on data; (c) training phase, which utilizes CNN2D, and (d) activities recognition.

Table 1 .

 1 HAR Datasets used for evaluation. The combined dataset was created using common activities from all six datasets.

	Dataset	Activities number	Sensor placement	Frequency Users
	w-Har [7]	7	Ankle	250	22
	UCI HAR[4]	6	Pocket	50	30
	PAMAP2 [41]	18	Wrist	100	9
	MotionSense [30]	6	Pocket	50	24
	Wisdm (smartphone) [46] 18	Pocket	20	51
	Wisdm (watch)[46]	18	Wrist	20	5
	Combined dataset	5	multiple	100	136

Table 2 .

 2 Classification results on different HAR datasets. Bold font indicates best results.

		CNN1D LSTM CNN-LSTM CWT-CNN2D RF
	UCI HAR	92.2	87.5	89.8	92.46	86
	MotionSense	82.6	80.08	86.1	88.9	91.52
	PAMAP2	52.4	51.8	53.9	54.4	62.17
	w-HAR	64.2	59.9	64.7	71.3	74.33
	Wisdm Phone 29.8	26.9	40.75	48.4	38.52
	Wisdm watch 54.2	49.7	47.4	49.4	57.52
	4.3 Activity Recognition on a Combined Dataset

Table 3

 3 reports the recognition rates obtained by the different methods, when reducing the number of activities to the ones that are common across all six datasets: laying, sitting, walking, walking upstairs and walking downstairs.

Table 3 .

 3 Classification results on HAR datasets, for the common activities. The last row represents the classfication results for the Combined Dataset. Bold font indicates best results.

		CNN1D LSTM CNN-LSTM CWT-CNN2D RF
	UCI HAR	93.8	89.8	93	93.9	87.79
	MotionSense	87.8	87.08	86.3	89.1	93.9
	PAMAP2	77.1	76.1	78.1	74.3	92.20
	w-HAR	75.2	69.8	80.8	84.2	78.68
	Wisdm Phone	89.1	87.7	86.3	91.8	83.43
	Wisdm Watch	64.2	63.18	69.3	70.04	63.17
	Combined Dataset 63.8	65.2	71.4	82.5	74

https://scikit-learn.org/stable/

https://keras.io/

ACKNOWLEDGMENTS

This research was supported by the ELEMENT project (ANR-18-CE33-0002) from the French National Research Agency.

We want to acknowledge all members of the project for fruitful discussions.