Julien Cumin
email: julien1.cumin@orange.com

Grégoire Lefebvre
email: gregoire.lefebvre@orange.com

A Priori Data and A Posteriori Decision Fusions for Human Action Recognition

Keywords: Action Recognition, Decision Fusion, Voting Methods, Dempster-Shafer Theory, Possibility Theory

In this paper, we tackle the challenge of human action recognition using multiple data sources by mixing a priori data fusion and a posteriori decision fusion. Our strategy applied from 3 main classifiers (Dynamic Time Warping, Multi-Layer Perceptron and Siamese Neural Network) using several decision fusion methods (Voting, Stacking, Dempster-Shafer Theory and Possibility Theory) on two databases (MHAD (Ofli et al., 2013) and ChAirGest (Ruffieux et al., 2013)) outperforms state-of-the-art results with respectively 99.85% ± 0.53 and 96.40% ± 3.37 of best average correct classification when evaluating a leave-one-subject-out protocol.

INTRODUCTION

In the last decades, human action recognition based on inertial or visual data sources has been an active area of research due to its success in robotics, video games, surveillance, etc. Nevertheless, some challenging difficulties still exist, caused by what is called the "3V" issues [START_REF] Ibm | Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data[END_REF]: the action Velocity, the action Variety and the action Volume.

Action recognition requires instantaneous responses from the system, moreover if it is an interactive application where actions are used as interaction controllers. In this regard, the velocity of data generation is a major problem, imposing constraints of execution times on algorithms.

Moreover, there is great variability about the way people produce actions. Dynamic variations occur when people produce intense or phlegmatic, slow or fast gestures. Different shapes, orientations and directions may then be captured from body trajectories. These variations exist between people but also for a single user producing the same set of actions (e.g. human ability, left or right-handed, on the move, in different use contexts, etc.).

Finally, volumetric variations are challenging as well, ranging from one user in a close world paradigm to multi-users in an open world paradigm.

Consequently, when designing a pattern recognition system, several steps are needed to deal with these issues: processing input data in order to reduce noise and to enhance salient information, clustering data to reduce the dimensionality of the problem, and learning a specific action classifier.

In this paper, we propose a human action classification system mixing a priori data fusion and a posteriori decision-level fusion (i.e. classifier fusion) methods.

This paper is organized as follows. In Section 2, we present some main literature methods on action recognition and decision fusion. In Section 3, we explain in details our fusion strategy. Then, Section 4 shows our experimental configurations and results. Finally, our conclusions are drawn and perspectives are presented in the last section.

STATE OF THE ART

Action Recognition

Human action recognition has been deeply studied for the past ten years. Some studies are based on inertial sensors, others on visual skeleton acquisitions, or sometimes both simultaneously.

Based on inertial data, three main strategies can be identified. The first action recognition strategy relies on similarity metrics between unknown actions to be classified and class reference instances. One main representative [START_REF] Akl | Accelerometer-based gesture recognition via dynamic-time warping, affinity propagation, & compressive sensing[END_REF]) is a model constructed from the Dynamic Time Warping (DTW) similarity distance and a K Nearest Neighbor (KNN) classifier. Others studies by [START_REF] Berlemont | Siamese neural network based similarity metric for inertial gesture classification and rejection[END_REF] proposed a non-linear metric learning strategy based on Siamese Neural Networks (SNN). The second strategy consists in a statistic modeling approach with Hidden Markov Models (HMM), as in [START_REF] Pylvänäinen | Accelerometer Based Gesture Recognition Using Continuous HMMs Pattern Recognition and Image Analysis[END_REF] in order to model correlations between temporal data samples. Finally, the last strategy implies machine learning methods in order to model class features, such as Support Vector Machines (SVM) [START_REF] Wu | Gesture recognition with a 3-d accelerometer[END_REF], Bayesian Networks [START_REF] Cho | Twostage Recognition of Raw Acceleration Signals for 3-D Gesture-Understanding Cell Phones[END_REF] or Recurrent Neural Networks [START_REF] Lefebvre | Inertial gesture recognition with BLSTM-RNN[END_REF].

Using visual feature data, these three main strategies are still relevant. Firstly, [START_REF] Zhou | Generalized time warping for multi-modal alignment of human motion[END_REF] present a Generalized Time Warping (GTW) algorithm, which is an extension of the DTW algorithm to temporally align multi-modal sequences from multiple subjects performing similar activities. Secondly, [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] present an approach for human action recognition with histograms of 3D joints locations. These features are projected using Linear Discriminant Analysis (LDA) and clustered into several posture visual words. The temporal evolutions of those visual words are then modeled by a discrete HMM. Thirdly, a study by [START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a Lie group[END_REF] uses a SVM classifier to build an action recognition system. Their approach is based on a skeletal representation modeling the 3D geometric relationships between body parts using rotations and translations in 3D space. Since 3D rigid body motions are members of the Special Euclidean group SE(3), human actions can be modeled as curves in this Lie group.

These previous strategies focus on one sensor and one classifier to increase action recognition rates. This can be viewed as a classifier selection. Few studies take into account multi-modal sources and several classifiers to build a more robust system. [START_REF] Chen | Improving human action recognition using fusion of depth camera and inertial sensors[END_REF] present a two-level fusion approach based on two modality sensors consisting of a depth camera and an inertial body sensor. In the feature-level fusion, features generated from the two differing modality sensors are merged before classification. In the decision-level fusion, outcomes from two classifiers are combined with decision fusion (in their article, using Dempster-Shafer Theory (DST)).

Inspired by this recent study, we propose to fuse a posteriori decisions taken by classifiers, on the one hand from individual data, and on the other hand from a priori combined data.

Decision Fusion

In the following (see Figure 1), we assume that the final classification should be made between c i classes, with i 2 {1,. .., I}. We have available C j classifiers, with j 2 {1,. .., J}, each giving a decision x j i,k 2 [0, 1] for a class c i about a gesture instance G k , k 2 {1,. .., K}. A decision x j i,k closer to 1 indicates a high confidence that the instance belongs to the class c i , whereas a decision closer to 0 indicates a low confidence. The final decision taken by the decision fusion method is denoted as c d .

classifier. Others studies by [START_REF] Berlemont | Siamese neural network based similarity metric for inertial gesture classification and rejection[END_REF] proposed a non-linear metric learning strategy based on Siamese Neural Networks (SNN). The second strategy consists in a statistic modeling approach with Hidden Markov Models (HMM), as in [START_REF] Pylvänäinen | Accelerometer Based Gesture Recognition Using Continuous HMMs Pattern Recognition and Image Analysis[END_REF] in order to model correlations between temporal data samples. Finally, the last strategy implies machine learning methods in order to model class features, such as Support Vector Machines (SVM) [START_REF] Wu | Gesture recognition with a 3-d accelerometer[END_REF], Bayesian Networks [START_REF] Cho | Twostage Recognition of Raw Acceleration Signals for 3-D Gesture-Understanding Cell Phones[END_REF] or Recurrent Neural Networks [START_REF] Lefebvre | Inertial gesture recognition with BLSTM-RNN[END_REF].

Using visual feature data, these three main strategies are still relevant. Firstly, [START_REF] Zhou | Generalized time warping for multi-modal alignment of human motion[END_REF] present a Generalized Time Warping (GTW) algorithm, which is an extension of the DTW algorithm to temporally align multi-modal sequences from multiple subjects performing similar activities. Secondly, [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] present an approach for human action recognition with histograms of 3D joints locations. These features are projected using Linear Discriminant Analysis (LDA) and clustered into several posture visual words. The temporal evolutions of those visual words are then modeled by a discrete HMM. Thirdly, a study by [START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a Lie group[END_REF] uses a SVM classifier to build an action recognition system. Their approach is based on a skeletal representation modeling the 3D geometric relationships between body parts using rotations and translations in 3D space. Since 3D rigid body motions are members of the Special Euclidean group SE(3), human actions can be modeled as curves in this Lie group.

These previous strategies focus on one sensor and one classifier to increase action recognition rates. This can be viewed as a classifier selection. Few studies take into account multi-modal sources and several classifiers to build a more robust system. [START_REF] Chen | Improving human action recognition using fusion of depth camera and inertial sensors[END_REF] present a two-level fusion approach based on two modality sensors consisting of a depth camera and an inertial body sensor. In the feature-level fusion, features generated from the two differing modality sensors are merged before classification. In the decision-level fusion, outcomes from two classifiers are combined with decision fusion (in their article, using Dempster-Shafer Theory (DST)).

Inspired by this recent study, we propose to fuse a posteriori decisions taken by classifiers, on the one hand from individual data, and on the other hand from a priori combined data.

Decision Fusion

In the following (see Figure 1), we assume that the final classification should be made between c i classes, with i 2 {1,...,I}. We have available C j classifiers, with j 2 {1,...,J}, each giving a decision x j i,k 2 [0, 1] for a class c i about a gesture instance G k , k 2 {1,...,K}. A decision x j i,k closer to 1 indicates a high confidence that the instance belongs to the class c i , whereas a decision closer to 0 indicates a low confidence. The final decision taken by the decision fusion method is denoted as c d .

G k C 1 C 2 C J x 1 1,k x 1 I,k ... x 2 1,k x 2 I,k ... x J 1,k x J I,k ... . . .

Decision c d Fusion

Figure 1: The decision fusion process.

Voting Methods

Voting methods are based on the following principle: each classifier C j adds a vote V j i for each class c i . The class decided by the fuser is then the one that collects the most votes. In case of a tie, a class at random from those with the most votes is chosen.

Voting by Majority (VM) C j adds a vote of 1 for the class it has the most confidence in (i.e. the class c i for which x j i,k is closest to 1), and a vote of 0 for all other classes.

Voting by Borda Count (VBC) C j adds a vote of P 2 {1,...,I} for the class it has the most confidence in, P 1 for the second most confident class, . . . , 1 for the P th most confident class, and 0 for all remaining classes for which it has less confidence. If P = 1, this VBC method is identical to VM.

Weighted Votes (VW) C j adds a vote of V j i = j(x j i,k ), with j a weighting function, taking into account the decision value. As in the Borda Count voting method, only the P 2 {1,...,I} classes the classifier has the most confidence in can vote, while the remaining classes receive a vote of 0.

Voting by Kumar and Raj (VK) A weighted vote approach is also presented by [START_REF] Kumar | Unsupervised fusion weight learning in multiple classifier systems[END_REF]. We define the positive set X + i (i.e. instances belonging to a class c i ), the negative set X i (i.e. instances

Voting Methods

Voting methods are based on the following principle: each classifier C j adds a vote V j i for each class c i . The class decided by the fuser is then the one that collects the most votes. In case of a tie, a class at random from those with the most votes is chosen.

Voting by Majority (VM) C j adds a vote of 1 for the class it has the most confidence in (i.e. the class c i for which x j i,k is closest to 1), and a vote of 0 for all other classes.

Voting by Borda Count (VBC) C j adds a vote of P 2 {1,. .., I} for the class it has the most confidence in, P 1 for the second most confident class, . . . , 1 for the P th most confident class, and 0 for all remaining classes for which it has less confidence. If P = 1, this VBC method is identical to VM.

Weighted Votes (VW) C j adds a vote of V j i = j(x j i,k ), with j a weighting function, taking into account the decision value. As in the Borda Count voting method, only the P 2 {1,. .., I} classes the classifier has the most confidence in can vote, while the remaining classes receive a vote of 0.

Voting by Kumar and Raj (VK) A weighted vote approach is also presented by [START_REF] Kumar | Unsupervised fusion weight learning in multiple classifier systems[END_REF]. We define the positive set X + i (i.e. instances belonging to a class c i ), the negative set X i (i.e. instances belonging to all the other classes) and b 2 R a regularization parameter. x j i,k is set to 0 if c i is not the best decided class for C j (meaning only the best class for each classifier is voting) and the total number of votes V i of a class c i is then:

V i = I Â i=1 w j i,k x j i,k , where (1)
w j i,k = argmax w |R(x j i,k , w)|, with (2) R(x j i,k , w) = 1 |X i | Â u2X i 1 1 + e bw | (x j i,u x j i,k ) 1 |X + i | Â u2X + i 1 1 + e bw | (x j i,k x j i,u ) . (3)

Stacking Methods

Stacking methods [START_REF] Wolpert | Stacked generalization[END_REF] are based around the fact that decision fusion is equivalent to a classification task: a fusion classifier has to correctly classify an instance using the decisions of initial classifiers as inputs. We then build a feature vector to be learnt by stacking methods as the concatenation of the firstlevel decisions. The fusion classifier can be trained on the same data set that was used to train the initial classifiers. A multitude of different classifiers can be used to perform stacking. We use two in this paper: MultiLayer Perceptron and Support Vector Machine, respectively referenced by SMLP and SSVM in the experimental section (see section 4).

Dempster-Shafer Theory (DST)

Dempster-Shafer Theory, also called Evidence Theory, models data uncertainty and imprecision with what is called mass functions.

Let Q = {c 1 , c 2 ,. .., c n } be the set of classes of the problem, and let 2 Q = { / 0, {c 1 }, {c 2 }, {c 1 , c 2 }, {c 3 }, {c 1 , c 3 },. .., Q} be the power set of Q. We define a mass function m j associated to a source C j , as follows:

m j : 2 Q 7 ! [0, 1], Â A22 Q m j (A) = 1. ( 4
)
In practice here, we use :

m j (A) = 8 < : x j i,k  J j=1 x j i,k if A = {c i } 0 otherwise . ( 5
)
We can combine the mass functions of all classifiers into a single mass function m. Using Smets' combination rule [START_REF] Smets | The combination of evidence in the transferable belief model[END_REF], we have:

8A 2 2 Q , m(A) = Â B 1 \...\B k =A J ' j=1 m j (B j ). (6)
The decided class c d is then the one that maximizes the plausibility function Pl: 9.

c d = argmax c i ,i2{1,...,I} Pl({c i }) (7) = argmax c i ,i2{1,...,I} 0 @ Â B22 Q ,B\{c i }6 = / 0 m(B) 1 A . ( 8
c d = argmax i2{1,...,I} (µ i,k ). (9)
As in [START_REF] Fauvel | Decision fusion for hyperspectral classification[END_REF], µ i,k can be evaluated as:

µ i,k = v u u u u t J Â j=1 (w j µ j i,k ) 2

J

, where (10)

w j = J Â p=0,p6 = j H 0,5 (p p k ) (J 1) J Â p=0 H 0,5 (p p k )
, and where (11)

H a (p j k ) = 1 2 2a I I Â i=1 (µ j i,k ) a (1 µ j i,k ) a . ( 12
)
3 OUR STRATEGY Suppose we have 3 data sources A, B and C to record an instance we want to classify (in our case, a human action), and suppose we have 2 classifiers C 1 and C 2 at our disposal. A basic decision fusion approach to classify one instance would be to naively combine (e.g. concatenate) the data produced by the 3 sources into data ABC, and then classify the instance using C 1 and C 2 on this new combined data. The 2 sets of decisions X 1 ABC and X 2 ABC are then combined with any of the decision fusion methods presented in 2.2 to obtain the final class label of the instance. This is the strategy we will call a priori fusion in this paper. Rather than naively combining data a priori, we can also directly use C 1 and C 2 on A, B and C independently, resulting in 6 sets of decisions:

X 1 A , X 1 B , X 1 C , X 2 A , X 2
B and X 2 C , which we will denote A + B + C. This is the strategy we will call a posteriori fusion in this paper. We could have also only combined two data sources (AB or AC or BC), each resulting in two sets of decisions.

We propose to use all those different approaches simultaneously in the decision fusion process. Rather than feeding a decision fusion method with only 2 sets of decisions on ABC, or 6 sets of decisions on A + B +C, we can use A + B +C + AB + AC + BC + ABC, resulting in 14 sets of decisions to combine. This approach only exploits available data and does not require to use any additional source.

This strategy is motivated by the idea that some data sources can be more discriminant for certain classes than others. For example, for an action consisting of a hand rotation around the axis of the forearm, the skeleton data will probably not describe the gesture well, because the skeleton is itself the axis of rotation. A gyrometer or accelerometer sensor placed on the hand of the user, on the other hand, will produce data that will feature well the gesture since it will be subject to the rotation of the hand. Conversely, there can be actions that would be easily described with skeleton data, while inertial data would be less useful to classifiers. If we combine a priori both of those data sources, the classifiers can use both information to reach better classification performances than when using only one of the sources, but the task of identifying which of the two data is more helpful to classify this specific instance relies only on the classifiers. Using decisions taken on data fused a priori, as well as on each data source independently, the task of identifying which data is helpful to classify the instance is shared between the classifiers and the decision fusion method, since it now has access to decisions taken on each data source independently. While the use of decisions taken on each data source independently may worsen the classification rates of the decision fusion method for certain classes that are well featured by all sources, we believe it can significantly improve the rates of classification of the decision fusion method for classes that are only wellcaptured by some sources, and weakly featured by others.

This process can thus exploit the complementarity of data sources, which will better discriminate classes depending on their nature (e.g. accelerations vs joint trajectories) or their placement (e.g. two accelerometers placed on the hand and on the hip will not equally discriminate a hand rotation). The Berkeley MHAD (Multimodal Human Action Database [START_REF] Ofli | Berkeley MHAD: A comprehensive Multimodal Human Action Database[END_REF], see Figure 2) contains 11 actions performed by 7 male and 5 female subjects in the range 23-30 years of age except for one elderly subject. All the subjects performed 5 repetitions of each action, yielding about 660 action sequences which correspond to about 82 minutes of total recording time.

EXPERIMENTAL RESULTS

Experiments on MHAD

The specified set of actions consists of the following: (1) actions with movement in both upper and lower extremities (e.g. Jumping in place, Jumping jacks, Throwing, etc.), (2) actions with high dynamics in upper extremities (e.g. Waving hands, Clapping hands, etc.), and (3) actions with high dynamics in lower extremities (e.g. Sit down, Stand up).

Protocols and Method Configurations

We use a leave-one-subject-out cross-validation process to test our fusion strategy. Thus, for each crossvalidation, all samples from 11 subjects (i.e. 605 instances) are used as a training set, and all samples from the remaining subject constitute the test set (i.e. 55 instances). A validation set is extracted from 10% of the training set to optimize b of the VK method (it can also be used to optimize certain parameters of other fusion methods, which is not done here for the datasets presented). Thus the training data for the fusion methods only consist of 90% of the actual total training set.

After some preliminary studies on the available 6 accelerometers data and 27 joints skeleton data, we keep 4 main data sources to build our system: the left hand accelerometer (noted A1) and the right hip accelerometer (A4), as in [START_REF] Chen | Improving human action recognition using fusion of depth camera and inertial sensors[END_REF], as well as the right hand joint trajectory (M20) and the left hand joint trajectory (M27), which are natural joints to track in an action recognition system.

Our fusion method evaluation is based on 3 main action classifiers: DTW, MLP and SNN. The following classifiers and decision fusion methods use the corresponding parameters :

• DTW: low-pass filter of parameter 0.8 on raw data and KNN using K=1;

• MLP: fixed input vectors of size 100 from the concatenation of normalized raw data, 45 hidden neurons with a learning rate of 0.005 and 200 epochs;

• SNN [START_REF] Berlemont | Siamese neural network based similarity metric for inertial gesture classification and rejection[END_REF]: fixed input vectors of size 100 from the concatenation of normalized raw data, 45 hidden neurons with a learning rate of 0.00005 and 200 epochs;

• VM: V c = 1;

• VBC: P = 11 voting classes;

• VW : V c = 0.15+ 0.85 1+e 7.68(x 0.68) and P = 11 voting classes;

• VK: b is selected in the set {10 4 ,. .., 10 5 } to maximize classification rates on the validation set;

• SMLP [START_REF] Hall | The weka data mining software: An update[END_REF]: 45 hidden neurons, learning rate = 0.2, momentum = 0.1, 100 epochs;

• SSVM [START_REF] Hall | The weka data mining software: An update[END_REF]: C = 256, g = 0.001;

• PT: a = 0.5.

Those parameters were experimentally evaluated (using the same 10% validation set logic) on a purely symbolic gesture database containing accelerometer and gyrometer data only, and were not chosen to be optimal for the MHAD database (or the ChAirGest database presented after).

Inertial and Vision-based Classification

Table 1 presents the average classification rates and standard deviations on isolated data: A1, A4, M20 and M27. It is remarkable to see that for each data source, multiple strategies of decision fusion give better results than the best first-level classifier. For instance, VBC achieves the best results on A1, VM on A4 and M20, and SMLP on M27 data. The best overall classification rate is achieved by VM on M20 with 92.55% ± 7.14, which is better than the best original classifier (SNN) with 91.34% ± 6.74 on M20.

A priori Data Fusion

Table 2 presents, from concatenated data, the average classification rates and standard deviations of decision fusion methods from 3 classifiers (DTW, MLP, SNN) on A1 and A4 concatenated, on M20 and M27 concatenated, and on A1, A4, M20 and M27 concatenated. This table shows that most decision fusion methods benefit from a priori data concatenation. For example, SSVM achieves 87.97% ± 7.36 from A1A4 data which is better than the results it obtained on A1 (87.81% ± 9.18) or A4 (65.49% ± 10.84) (see Table 1). The methods that show worse classification rates are actually weakly impacted compared to the results on A1, which shows that decision fusion methods are relatively robust to data that contains non-discriminative parts. The best overall classification rate is achieved by VK on all concatenated data A1A4M20M27 with 96.95% ± 3.25.

A posteriori Classifier Fusion

Table 3 presents, from separated data, the average classification rates and standard deviations with classifier fusion methods. A posteriori decision fusion gives here better average classification rates for each configuration. For instance, when we fuse first-level classifier decisions on separated M20 and M27 data, we obtain 95.90% ± 4.32 for VK, which is higher than 95.13% ± 4.87 presented before (see Table 2) on M20M27 concatenation. The best overall classifica-tion rate is achieved by VW with 99.39% ± 1.11 on the A1+A4+M20+M27 configuration.

Mixed Fusion Strategies

Table 4 presents our strategy from mixed data using both a priori data and a posteriori decision fusion. The best average classification rate is reached at 99.85% ± 0.53 by the VW decision fusion method on separated data (A1+A4+M20+M27), plus partially concatenated data (A1A4+M20M27), plus all concatenated data (A1A4M20M27). These results are significantly better than 99.39% ± 1.11 presented in Table 3. [START_REF] Chen | Improving human action recognition using fusion of depth camera and inertial sensors[END_REF] propose on this database a fusion approach based on two differing modality sensors (depth and inertial data). Their best results for a leave-one-subject-out evaluation is a classification rate of 99.54% fusing Kinect depth stream and A1 and A4 inertial sensors data with a SRC (Sparse Representation Classifier) method. Our strategy is consequently challenging with 99.85% ± 0.53 correct classification at best. We then prove on a second dataset the repeatability of our strategy.

Experiments on ChAirGest

Protocols and Method Configurations

As in the previous experiment, we use a leave-onesubject-out cross-validation to test our system. Thus, for each cross-validation, all samples from 9 subjects (i.e. 450 instances) are used as a training set, and all samples from the remaining subject constitute the test set (i.e. 50 instances). Classifiers and fusion methods configurations are identical to the ones used on MHAD, described in section 4.1.1. We use here the 4 accelerometers data (A1, A2, A3 and A4) and 3 joint skeleton data K2, K6, and K10, corresponding respectively to the head, the left hand and the right hand. Table 5 shows a comparison between a priori data and a posteriori decision fusion based on 4 accelerometers. The best average classification rate for the second approach is VBC with 89.00% ± 5.35, as opposed to the first approach where it is SMLP with 81.40% ± 9.89. We see that each decision fusion method is significantly better on A1+A2+A3+A4 compared to A1A2A3A4, which is a trend that was already existing on the MHAD dataset.

Our Results

Table 6 proposes similar results on Kinect data with again better classification rates for the second approach (e.g. 94.80% ± 3.68 for VM) compared to the first one (93.80% ± 4.05 for VK). However, standard deviations are bigger in the second approach; both approaches are thus quite equivalent for those data.

Figure 4 presents means and standard deviations for each method when applying decision fusion on separated data A1+A2+A3+A4+K2+K6+K10. It is remarkable to see that all decision fusion methods but DST outperform a first-level classifier selection. For instance, the DTW classifier achieves a classification rate of 91.00% ± 4.64 on K6, while VM achieves 95.40% ± 4.12 using all separated data sources.

The final results are presented in Table 7. The best overall average classification rate is 96.40% ± 3.37 for the VM method, with our strategy mixing both a priori data and a posteriori decision fusion. [START_REF] Cao | Multi-modal learning for gesture recognition[END_REF] published results on the ChAirGest dataset. With the exact same leave-one-subject-out cross-validation strategy we used in this paper, they attain at best a classification rate of 91.84% ± 5,76. Our approach performs significantly better for all decision fusion methods, bar DST, with results as high as 96.40% ± 3.37 for the VM fusion method. [START_REF] Yin | Gesture spotting and recognition using salience detection and concatenated hidden markov models[END_REF]) also published previous results on the ChAirGest dataset. They attain an average final classification rate of 91.16%. Here, our evaluation protocol are quite different (we believe that ours is more challenging), nevertheless our best configuration performs better (96.40% ± 3.37).

Previous published results

(

CONCLUSIONS AND PERSPECTIVES

In this paper, we tackle the challenge of human action recognition by mixing a priori data fusion and a posteriori decision fusion. Our strategy applied from 3 main classifiers (DTW, MLP and SNN) on two databases (MHAD [START_REF] Ofli | Berkeley MHAD: A comprehensive Multimodal Human Action Database[END_REF] and ChAirGest [START_REF] Ruffieux | ChAirGest: A Challenge for Multimodal Mid-air Gesture Recognition for Close HCI[END_REF]) matches or even outperforms state-of-the-art results. Note that the classification rates consistently increased at each step, from standard decision fusion all the way up to our mixed fusion strategy, for all decision fusion methods we studied, which highlights the benefits of our approach.

Our perspectives are to extend our solution to disambiguate some human actions: two gesture classes where frontiers are fuzzy (e.g. heart and clockwise symbolic gestures); two gesture classes where one source is relevant and the other data source gives no salient information (e.g. limbs rotations, identifiable by inertial systems but not with skeleton based trajectories); or using both strong and weak classifiers, in order to evaluate the impact of extremely low performance classifiers on our approach of decision fusion.

Figure 1 :

 1 Figure 1: The decision fusion process.

  } be the set of membership degrees of an instance G k to classes c i with classifier C j . The decided class c d is here the one with the maximum merged membership degree µ d,k , as in Equation

Figure 2 :

 2 Figure 2: Snapshots from all the actions available in the Berkeley MHAD displayed together with the corresponding point clouds obtained from the Kinect depth data. Actions (from left to right): jumping, jumping jacks, bending, punching, waving two hands, waving one hand, clapping, throwing, sit down/stand up, sit down, stand up.

Figure 3 :

 3 Figure 3: Two sample images captured by the Kinect RGB stream, where 4 IMUs are fixed on the participant's arm.

Figure 4 :

 4 Figure 4: Average classification rates and standard deviations of classifiers on individual data and of decision fusion methods on A1+A2+A3+A4+K2+K6+K10.

Table 1 :

 1 Isolated data: average classification rates and standard deviation for standard decision fusion.

		A1	A4	M20	M27
	DTW MLP SNN VM VBC VW VK SMLP SSVM DST PT	79.01 ± 10.69 81.75 ± 9.40 82.05 ± 9.91 85.39 ± 9.46 87.81 ± 9.18 85.08 ± 7.60 86.59 ± 9.53 84.03 ± 8.09 83.72 ± 8.24 83.72 ± 7.75 83.42 ± 8.15	56.54 ± 14.45 61.10 ± 10.29 62.76 ± 10.46 65.49 ± 10.84 61.56 ± 15.04 65.20 ± 10.87 64.90 ± 11.09 65.35 ± 12.20 64.58 ± 10.14 62.31 ± 10.51 62.47 ± 10.71	88.75 ± 7.55 89.51 ± 9.27 91.34 ± 6.74 92.55 ± 7.14 92.10 ± 7.77 90.42 ± 8.61 92.40 ± 5.61 90.58 ± 8.28 90.42 ± 8.11 90.27 ± 8.48 90.42 ± 8.47	61.24 ± 8.89 76.44 ± 11.68 75.22 ± 10.07 75.53 ± 10.56 76.44 ± 10.75 76.73 ± 11.39 74.76 ± 9.90 79.78 ± 10.50 78.41 ± 11.30 77.04 ± 11.40 76.43 ± 11.67

Table 2 :

 2 Concatenated data: average classification rates and standard deviations on a priori decision fusion.

		A1A4	M20M27	A1A4M20M27
	VM VBC VW VK SMLP SSVM DST PT	82.07 ± 8.43 83.73 ± 8.95 85.39 ± 7.90 80.86 ± 8.93 87.52 ± 7.77 87.97 ± 7.36 85.85 ± 6.52 85.40 ± 7.61	95.13 ± 4.87 94.83 ± 5.14 93.76 ± 6.12 94.38 ± 5.61 93.91 ± 6.42 93.61 ± 6.58 93.91 ± 6.45 94.06 ± 6.37	96.64 ± 3.96 96.04 ± 3.39 94.97 ± 5.69 96.95 ± 3.25 94.81 ± 5.74 94.81 ± 5.76 94.66 ± 6.07 94.97 ± 5.46

Table 3 :

 3 Separated data: average classification rates and standard deviations on a posteriori decision fusion.

		A1+A4	M20+M27	A1+A4+M20+M27
	VM VBC VW VK SMLP SSVM DST PT	90.11 ± 6.60 89.04 ± 7.62 92.09 ± 5.63 92.54 ± 5.08 91.02 ± 5.79 90.41 ± 5.13 88.89 ± 8.03 91.33 ± 5.18	91.35 ± 8.24 91.95 ± 7.38 94.98 ± 5.33 95.90 ± 4.32 94.82 ± 5.50 93.45 ± 6.36 94.08 ± 6.28 94.68 ± 5.43	98.18 ± 2.01 97.71 ± 3.22 99.39 ± 1.11s 96.34 ± 3.02 98.32 ± 2.21 98.17 ± 2.32 96.66 ± 4.35 98.93 ± 1.67

Table 4 :

 4 Mixed data: Average classification rates and stan-

	dard deviations of decision fusion methods from 3 classi-
	fiers (DTW, MLP, SNN) on A1, A4, M20 and M27 mixed.
		A1+A4+M20+M27 +A1A4+M20M27	A1+A4+M20+M27 +A1A4+M20M27 +A1A4M20M27
	VM VBC VW VK SMLP SSVM DST PT	98.48 ± 2.43 98.63 ± 2.93 99.24 ± 1.20 96.80 ± 3.13 99.24 ± 2.62 98.78 ± 2.73 97.57 ± 4.99 98.93 ± 2.74	98.94 ± 2.12 98.78 ± 2.73 99.85 ± 0.53 97.56 ± 2.85 99.09 ± 2.63 99.09 ± 2.63 97.87 ± 4.51 99.54 ± 1.13
	4.1.6 Previous published results

Table 5 :

 5 Accelerometer data: Average classification rates and standard deviations.

		A1A2A3A4	A1+A2+A3+A4
	VM VBC VW VK SMLP SSVM DST PT	76.40 ± 11.23 79.40 ± 9.93 75.80 ± 10.26 77.20 ± 10.63 81.40 ± 9.89 73.20 ± 8.95 57.40 ± 9.52 64.20 ± 7.97	87.00 ± 6.48 89.00 ± 5.35 87.40 ± 4.90 82.20 ± 4.85 83.20 ± 6.74 78.20 ± 7.20 79.00 ± 9.20 82.80 ± 5.27

Table 6 :

 6 Kinect data: Average classification rates and standard deviations.

K2K6K10 K2+K6+K10 VM 93.40 ± 3.27 94.80 ± 3.68 VBC 93.60 ± 3.77 93.60 ± 3.97 VW 93.40 ± 3.53 93.20 ± 3.91 VK 93.80 ± 4.05 94.00 ± 5.33 SMLP 90.40 ± 3.75 92.00 ± 3.89 SSVM 89.80 ± 3.94 91.20 ± 3.68 DST 90.40 ± 4.60 90.20 ± 5.37 PT 90.60 ± 3.78 92.20 ± 4.57

Table 7 :

 7 Mixed data: Average classification rates and standard deviations.

				A1+A2+A3+A4
		A1A2A3A4 K2K6K10	A1+A2+A3+A4 +K2+K6+K10	+K2+K6+K10 +A1A2A3A4 +K2K6K10
				+A1A2A3A4K2K6K10
	VM VBC VW VK SMLP SSVM DST PT	89.60 ± 6.02 90.40 ± 4.88 88.40 ± 7.35 89.80 ± 6.14 91.00 ± 4.45 90.00 ± 4.11 81.20 ± 11.04 83.00 ± 9.39	95.40 ± 4.15 94.60 ± 4.62 94.20 ± 3.82 91.80 ± 4.62 93.60 ± 4.30 92.40 ± 4.20 88.60 ± 6.47 93.60 ± 3.86	96.40 ± 3.37 96.20 ± 3.58 96.20 ± 3.46 93.20 ± 4.02 95.20 ± 4.13 94.40 ± 3.75 89.60 ± 6.31 95.60 ± 3.62


Grégoire Lefebvre
email: gregoire.lefebvre@orange.com

Julien Cumin
email: julien1.cumin@orange.com

Recognizing Human Actions based on Extreme Learning Machines

Keywords:

In this paper, we tackle the challenge of action recognition by building robust models from Extreme Learning Machines (ELM). Applying this approach from reduced preprocessed feature vectors on the Microsoft Research Cambridge-12 (MSRC-12) Kinect gesture dataset outperforms the state-of-the-art results with an average correct classification rate of 0.953 over 20 runs, when splitting in two equal subsets for training and testing the 6, 244 action instances. This ELM based proposal using a multi-quadric radial basis activation function is compared to other classical classification strategies such as Support Vector Machines (SVM) and Multi-Layer Perceptron (MLP) and advancements are also presented in terms of execution times.

INTRODUCTION

Human activity recognition is one of the main challenging topics in computer vision research. Its importance may be explained by many successes for instance in video games, video surveillance, sport analysis, back and neck pain control, sign language understanding, human-robot interaction, etc. Various types of activities indeed appear in everyday life, such as gestures, actions, human-object interactions, social interactions and group activities.

Nevertheless, some issues still exist in human action recognition automation: the first ones are inherent to user gesture productions, the other ones come from data acquisitions. The classical recognition methods are often biased by numerous factors: dynamical variations (dynamic and phlegmatic users), temporal variations (slow and fast users), physical variations (device weight, user morphology, left or right-handed users, contextual environment, parallel user activities, etc.), paradigm variations (mono versus multi users, open or closed world paradigm, etc.), and cultural interpretation variations (i.e. one gesture may have different meanings regarding several cultures). Other difficulties impacting the data acquisition system are the presence of occlusions, nonrigid motions, view-point changes, background interferences, etc.

Therefore, a human action recognition system is designed to deal with these issues. The early methods propose pose estimators as action features and build appearance models based on shape [START_REF] Blank | Actions as space-time shapes[END_REF] or motion [START_REF] Efros | Recognizing action at a distance[END_REF] information. Then, in order to avoid segmentation and object tracking issues, local features and bag-of-features representations strategies are proposed. This offers compact action descriptors to be classified as dense trajectories (Wang et al., 2011a) or sparse motion vectors [START_REF] Kantorov | Efficient feature extraction, encoding, and classification for action recognition[END_REF]. In contrast, other studiesdeal with mid-level representations [START_REF] Jain | Representing videos using mid-level discriminative patches[END_REF] and temporal models [START_REF] Amer | Monte carlo tree search for scheduling activity recognition[END_REF]. On one side, the main objectives are to group point trajectories into tentative action parts by similarity in motion and appearance and then learn discriminative models with latent assignment of action parts. This allows in particular to localize discriminative action parts. On the other side, temporal models describe actions as a sparse sequence of spatio-temporally localized frames. This approach models longer events by temporal logical composition of simple actions.

On action data, preprocessing steps such as spatiotemporal segmentation and description are time consuming. Likewise, parameterizing and training a dedicated classifier can be very challenging. That is why, in this paper, we propose an action classification system based on Extreme Learning Machines [START_REF] Huang | Extreme learning machine: a new learning scheme of feedforward neural networks[END_REF] in order to operate fast and robust recognitions on action data with compact feature vectors. This paper is organized as follows. Section 2 presents related works on action recognition. In Section 3, we explain in details Extreme Learning Machines (ELM). Then, Section 4 describes our results with a comparison to classical approaches. Finally, our conclusions and perspectives are drawn.

Designing an automatic action recognition system based on video analysis is already challenging, but when users want an interactive system based on natural human interactions, the challenge is even higher. Indeed, when designing an iterative action recognition system, the two main priorities are reducing latency when users are interacting with it and maximizing the action recognition scores in order to offer better user experiences. These two properties are influenced by the action feature vector dataset to be modeled in a robustness and speed classifier.

In [START_REF] Ellis | Exploring the trade-off between accuracy and observational latency in action recognition[END_REF], the two aspects are taken into account with a latency-aware learning formulation in order to train a logistic regression model (LRM). The contribution is to distinguish canonical poses from body motions to reduce ambiguity for action recognition. Their experimental results on the Microsoft Research Cambridge-12 (MSRC-12) Kinect gesture dataset [START_REF] Fothergill | Instructing people for training gestural interactive systems[END_REF] show improvements (i.e. an average correct classification rate of 0.912 for a 4-cross validation) in opposition to a bags of visual words and a conditional random field strategy. The trade-off between latency and accuracy is then guided by a Frame Based Descriptor (FBD) of size 2276. The information support from the 3D body joint motions appears then to be crucial.

Likewise, in [START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF], the construction of discriminative descriptors from 3D skeleton data is obtained in order to optimize the recognition system. Encoding the correlation between joint trajectories during space and time, they propose a Temporal Hierarchy of Covariance Descriptor (THCD). This descriptor has a fixed size of 7320 and the final classification is operated by a linear SVM classifier. On the MSRC-12 dataset, the experiments give an average correct classification rate of 0.917 for a 20-cross validation when splitting 50% of the dataset for training and testing steps.

A more recent study by [START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a lie group[END_REF]) also uses a SVM classifier to build an action recognition system. Consequently, it is again the choice of the feature descriptors which influences the global recognition accuracy. The authors propose to model the geometric relationships between 3D body parts. Human actions are then modeled as curves in this Lie group based on 3D rigid body motions which are members of the Special Euclidean Group SE(3). The feature vector is then of size 2280 and the correct classification rate are very challenging with 0.9088 on the Florence3D-Action dataset [START_REF] Seidenari | Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses[END_REF].

In this paper, we propose a more reduced feature vector of size 1200 and a fast ELM classification. Deep learning methods are nowadays widely used in machine learning. With recent successes on visual object recognition [START_REF] Farabet | Learning hierarchical features for scene labeling. Pattern Analysis and Machine Intelligence[END_REF] or speech recognition [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF] and iconic gesture recognition [START_REF] Lefebvre | Inertial gesture recognition with BLSTM-RNN[END_REF], deep architectures using feed-forward neural networks (e.g. Convolutional Neural Networks [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]) or recurrent neural networks (e.g. Long Short Term Memory [START_REF] Hochreiter | Long short-term memory[END_REF]) are limited to some issues: a relatively slow training speed, overfitting problems and a relatively high number of parameters needed to be tuned.

EXTREME LEARNING MACHINES

Inspired by previous studies on Radial Basis Function (RBF) networks (see [START_REF] Broomhead | Multivariable Functional Interpolation and Adaptive Networks[END_REF][START_REF] Chen | Orthogonal least squares learning algorithm for radial basis function networks[END_REF]), Huang et al. present ELM in (Huang et al., 2004) in order to deal with these issues for classification and regression problems. An ELM is a single layer feed-forward network (SLFN) with only two parameters: the number of hidden nodes and the choice of an activation function (see Figure 1). In comparison to the traditional backpropagation algorithm based on a gradient descent in order to minimize the network error between the produced output and the desired target, the ELM method initializes randomly the network weights and then computes directly a Moore-Penrose pseudo-inverse matrix to generate a unique solution. In various fields (e.g. regression problems, medical diagnosis application, diabetes protein sequence classification), ELM shows some promising features such as generalization ability, robustness, controllability and a fast learning process (see [START_REF] Huang | Trends in extreme learning machines: A review[END_REF]).

Numerous studies propose some extensions to address some ELM issues, such as the network hid- den layer output matrix singularity. In [START_REF] Wang | A study on effectiveness of extreme learning machine[END_REF], the Effective ELM (EELM) algorithm exploits the strictly dominant criterion for non-singular matrices. In [START_REF] Huang | Extreme learning machine for regression and multiclass classification. Systems, Man, and Cybernetics[END_REF], an Optimizationbased regularized ELM (ORELM) enhances the generalization properties of ELM. Likewise, improvements are published in [START_REF] Iosifidis | Minimum variance extreme learning machine for human action recognition[END_REF] in order to exploit the training data dispersion with a Minimum Variance ELM (MVELM).

Let's introduce some notations. For N arbitrary distinct samples (x j , t j ), we note data samples by x j = [x 1 j , x 2 j ,...,x n j ] | 2 R n and the relative targets by

t j = [t 1 j ,t 2 j ,...,t m j ] | 2 R m . Let be a i = [a 1 i , a 2 i ,.
..,a n i ] | the weight vector connecting the i th hidden node and the input nodes, and b i 2 R is the respective bias of the i th hidden node. The weight vector connecting the i th hidden node and the output nodes is b

i = [b 1 i , b 2 i ,...,b m i ]
| and the activation function h is usually a Sigmoid or a Radial Basis Function. Consequently, a SLFN with L hidden neurons is defined by:

8 j 2 {1,. .., n}, t j = L Â i=1 b i h(a i • x j + b i ) (1)
Equation 1 can be written with H 2 R N⇥L the hidden layer output matrix of the network, B 2 R L⇥m and T 2 R N⇥m , as defined by: HB = T.

(2)

Therefore, the SFLN learning process corresponds now to find the Equation 2 least squares solution. The solution is provided by: B = H † T.

(3)

In Equation 3, H † represents Moore-Penrose generalized inverse of the hidden layer output matrix H. As described in [START_REF] Zhang | An algorithm research for prediction of extreme learning machines based on rough sets[END_REF], the optimal solution B has the following features:

1. this algorithm can gain the minimal training error; 2. this algorithm can get the optimal generalization capability of the minimum paradigm of the output connection weights and network;

3. B is unique and can avoid local optimal solutions.

Consequently, the ELM learning speed is fast and can reach the solution straightforward without facing local minima or over-fitting issues. Recent studies [START_REF] Minhas | Human action recognition using extreme learning machine based on visual vocabularies[END_REF][START_REF] Iosifidis | Minimum variance extreme learning machine for human action recognition[END_REF] proposed ELM for Human Action Recognition based on bags of visual words. Here, we build our system on specific feature vectors using directly four joint trajectories, described hereafter.

EXPERIMENTS

Protocols

Our experiments are based on the Microsoft Research Cambridge-12 (MSRC-12) Kinect gesture dataset (see for details [START_REF] Fothergill | Instructing people for training gestural interactive systems[END_REF] and an action sample in Figure 2). This dataset is composed of 594 sequences of human actions from 30 people performing 12 gestures. The gesture set is composed of 6 iconic gestures: crouch and hide (duck), shoot with a pistol, throw an object, change weapon, kick to attack, put on night vision goggles and 6 metaphoric gestures: start the music (lift outstretched arms), navigate to next menu (push right), wind up the music, take a bow, protest the music, lay down the song tempo (beat both arms). In total, 6, 244 gesture instances are stored using 20 body joint positions captured at 30Hz. Each body joint is 3-dimensional.

In this study, we evaluate the effectiveness of our approach with a reduced training set using 50% of the MSRC-12 dataset (i.e. 3, 122 training instances and 3, 122 test instances). This configuration appears to be the less advantageous and the more challenging one according to [START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF]. In order to compare our results with two state-of-theart methods: Frame-Based Descriptors and Logistic Regression Models [START_REF] Ellis | Exploring the trade-off between accuracy and observational latency in action recognition[END_REF] (FBD+LRM) and Temporal Hierarchy of Covariance Descriptors and Support Vector Machines [START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF] (THCD+SVM), we perform a 20-cross validation as in [START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF] and the comparison is then based on the average correct classification and the associated standard deviation. We evaluate as well two other classical classifiers (i.e. SVM and MLP) with the Weka data mining software [START_REF] Hall | The weka data mining software: An update[END_REF] on our feature vectors (FV) to prove the relevance of our (FV+ELM) approach.

Our feature vectors are obtained from the concatenation of the four 3D body joints corresponding to the two hands and the two feet, for each time period. We first filter the 12-D signal with a low-pass filter of parameter 0.7. Then a normalization is operated and an interpolation is computed to get an equal duration of 3333 ms for all actions. The feature vector is finally the concatenation of all values through time, giving a dimension of 1200. This can be viewed as a prior fusion process.

The choice of all parameters in the following study is given by analyzing a preliminary study with 1200 training instances and 1200 validation instances extracted from the original 3, 122 training instances.

For instance, Figure 3 shows some tested configurations for all classifiers. This figure gives the best ELM performance with 400 hidden nodes and a multi-quadric radial basis activation function as opposed to other ELM configurations with sigmoid and hyperbolic tangent activation functions. Likewise, the preliminary results on the good fit of our feature vectors with a SVM classifier give better performances with a RBF kernel with g = 0.005 and c = 4, compared to a polynomial kernel. Our evaluation on MLP gives a best configuration with 100 hidden neurons, 600 learning epochs and a learning rate of 0.005 with a hyperbolic tangent as an activation function.

Consequently, the final configurations of the classifiers reported in the next section are the following:

• SVM uses a RBF kernel with g = 0.005 and c = 4;

• MLP uses one layer of 100 hidden neurons, 600 learning epochs and a learning rate of 0.005 with a hyperbolic tangent as an activation function;

• ELM uses 400 hidden nodes and a multi-quadric radial basis activation function.

Classification Results

Table 1 outlines the global performances of each human action recognition strategy. Our solution based on our feature vectors FV and an ELM classifier gives the best average correct classification rate of 0.953 with a reduced standard deviation of 0.003. The results previously published were respectively 0.912 for FBD+LRM [START_REF] Ellis | Exploring the trade-off between accuracy and observational latency in action recognition[END_REF] (n.b. the authors used only a 4 folds cross-validation) and 0.917 for THCD+SVM [START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF]. Consequently, these results show that feature vectors extracted only from four joint data can be an adequate information support for human action recognition. Moreover, we demonstrate on our protocol the effectiveness of the ELM classifier as opposed to the SVM and MLP classifiers (with respectively 0.898 ± 0.002 and 0.906 ± 0.008 of correct classification rate).

A main conclusion of an analysis of confusion matrices (see Figure 4 showing one tested configuration) is that our FV+ELM method presents 20 misclassification between wind up the music (wind) and lay down the tempo (beat). In fact, these two actions share common trajectories with respectively circular movements with both arms, in front of the body, for the first action, and hand beat movements in the air for the second action. Likewise, we can observe some confusions between duck and shoot, start and beat, start and wind up, and enough and shoot.

Computing Times

In addition, we report the computing times for the 3 methods using the same feature vectors in order to learn 3, 122 gestures and testing 3, 122 gestures of the MSRC-12 database (c.f. Table 2). This experiment was executed on an Intel Core i5 CPU clocked at 2.67 GHz with 3.42 GB of RAM.

These experimental results show that the computing times for the ELM based solution is fast (i.e. around 4.865 seconds in average). On the contrary, the FV+SVM based solution requires more time to learn and test the 6, 244 gestures (i.e. around 8.875 seconds in average). The FV+MLP based method needs around 793.883 seconds in average but this is mainly due to the learning process using 600 epochs to build the best neural model in this case.

Consequently, our proposed FV+ELM system, achieving the best result performances in a multi-user configuration with a fast recognition computing time, is a very challenging solution.

CONCLUSIONS AND PERSPECTIVES

In this article, we presented a human action recognition system based on specific feature vectors FV and a ELM classifier. Using the MSRC-12 dataset in order to cover real world challenging cases, we showed that the proposed approach proves to be superior to some state-of-the-art methods. It also outmatches its neural counterpart MLP, both in classification and processing time on our experiments. Some perspectives would be to experiment Echo State Networks (ESN [START_REF] Tong | Learning grammatical structure with echo state networks[END_REF]) with raw temporal input data to preserve the temporal correlation and detect gestural grammar inside each human action.

Figure 1 :

 1 Figure 1: Single feed-forward network.

Figure 2 :

 2 Figure 2: Human action from MSRC-12 Kinect gesture dataset: Wind up the music.

Figure 3 :

 3 Figure 3: Classification rate on an evaluation dataset (1200 training instances, 1200 evaluation instances).

Figure 4 :

 4 Figure 4: One resulting confusion matrix on MSRC-12, when applying our FV+ELM strategy.

Table 1 :

 1 Average correct Classification Rates (ACR) and Standard Deviations (SD) on MSRC-12.

	Database	MSRC-12
	Methods	ACR & SD
	FBD+LRM (Ellis et al., 2013)	0.912 ± -
	THCD+SVM (Hussein et al., 2013)	0.917 ± -
	FV+SVM FV+MLP FV+ELM	0.898 ± 0.002 0.906 ± 0.008 0.953 ± 0.003

Table 2 :

 2 Average Computing Times (ACT) and Standard Deviation (SD) in seconds on MSRC-12.

	Database	MSRC-12
	Methods	ACT & SD
	FV+SVM FV+MLP 793.883 ± 0.194 8.875 ± 0.182
	FV+ELM	4.865 ± 0.102