Adnan Salih

A L Alwani

Youssef Chahir
email: youssef.chahir@unicaen.fr

Spatiotemporal Representation of 3D Skeleton Joints-Based Action Recognition using Modified Spherical Harmonics

Action recognition based on the 3D coordinates of body skeleton joints is an important topic in computer vision applications and humanrobot interaction. At present, most 3D data are captured using recently introduced economical depth sensors. In this study, we explore a new method for skeleton-based human action recognition. In this novel framework, the normalized angles of local joints are first extracted, and then the modified spherical harmonics (MSHs) are used to explicitly model the angular skeleton by projecting the spherical angles onto the unit sphere basis. This process decomposes the skeleton representation into a set of basis functions. A spatiotemporal system of the spherical angles is adopted to construct the static pose and joint displacement over a human action sequence. Consequently, the MSH coe cients of the joints are used as the discriminative descriptor of the sequence. The extreme learning machine (ELM) classifier and recently published 3D action datasets are used to validate the proposed method. The experimental results show that the proposed approach performs better than many classical methods.

Introduction

Action representation and recognition from 3D data are among the topics widely discussed in pattern recognition. In recent years, research trends have mainly shifted toward the action recognition of sequences captured by RGB-D cameras. These ranging cameras feature the facility of 3D imaging technologies to provide 2D images and depth maps, and simplify the acquisition of 3D human posture through skeleton joints (Shotton et al.) . Moreover, the popularity of low-cost depth sensors, such as the Kinect sensor, has led to the development of e cient methods for specific action recognition applications [START_REF] Raptis | Real-time classification of dance gestures from skeleton animation[END_REF][START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF]. The studies recently conducted by [START_REF] Alnowami | A quantitative assessment of using the kinect for xbox360 for respiratory surface motion tracking[END_REF][START_REF] Obdrzalek | Accuracy and robustness of kinect pose estimation in the context of coaching of elderly population[END_REF][START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF] indicate that, compared with traditional 2D image data, the representation of skeleton joints only for action recognition provides better results. Therefore, the current study uses skeleton joint data as an initial input for skeleton representation. With RGB-D stationary sensors, the sequences of depth maps can be registered as a set of skeleton joints in 3D coordinates (Shotton et al.). In other words, the application of RGB-sensors allows human action recognition problems to be handled directly as 3D positions of joints, which cannot be realized with a silhouette-based imagery system. Importantly, depth sensors not only achieve fast capturing speed and good view invariance, but they also provide accurate depth maps [START_REF] Li | Action recognition based on a bag of 3d points[END_REF][START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF]. Skeleton-based action recognition methods typically rely on the direct use of either the absolute position of skeleton joints or the connection between these joints (rigid segments between joints). Joint position approaches consider the human skeleton simply as a set of articulated points, in which only the skeleton joints are used to abstract body motion in 3D space. Likewise, these approaches investigate human motion either by individual joints or are relational between joints using various features, including joint positions [START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF][START_REF] Raptis | Flexible dictionaries for action classification, in: The 1st International Workshop on Machine Learning for Visionbased Motion Analysis -MLVMA[END_REF], joint orientations with respect to a reference coordinate axis [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] and pairwise relative joint positions [START_REF] Wang | Recognition of human actions using motion capture data and support vector machine[END_REF][START_REF] Yang | Eigenjoints-based action recognition using navebayes-nearest-neighbor[END_REF].

With the e↵orts of numerous researchers to reliably improve skeleton-based recognition algorithms, research attention is now directed toward using skeleton joint data to recognize human actions. Loss of accuracy and e↵ective feature representation remains a major problem in the field. This issue should be addressed to robustly handle the task of human action recognition by either improving the discriminative feature descriptions or pose representation or by relying on additional post-processing of skeletal data. Several di culties remain in undertaking recognition tasks based on 3D joint representation (e.g., occlusion of body parts, lack of precision, and errors in data acquisition).

In this study, we present a novel skeleton joint-based representation of 3D human action in a spatiotemporal manner. We employ the spherical angles of body joints computed from the 3D coordinates of skeleton joints. The proposed feature representation is a combination of the modified spherical harmonics (MSHs) and the spatiotemporal model of sequence level. To estimate the human pose, the spherical harmonics (SHs) of spherical angles provide a distinctive feature description. As such, the problem of skeleton joint representation is addressed in a spatiotemporal approach using MSHs. The proposed model simply incorporates two mechanisms to e ciently capture the temporal dynamic of joints, namely, the application of MSHs in the computed spherical angles of each pose and the construction of MSHs in a hierarchical scheme. MSHs are computed in multi-levels, in which each level encodes the time window of an action sequence.

In the proposed representation of 3D human action, the selected MSHs are adopted to characterize the features in multilevels and capture the harmonic frequency of function in a twosphere space S 2 . Given this condition, the defined spherical angle vector of the selected joints may be projected onto (S 2 ). However, the principle computation required in this space is extremely large because each selected joint is sampled by the feature vectors of M J = M 1 , , M k , M 2 R N⇤N ; where M is an MSH matrix of k levels, J joint numbers, and N number of frames in each level. Considering that the desired descriptor dimensionality aims to expedite the classification phase as well as reduce the noise and redundant feature sizes, we apply dynamic time wrapping (DTW) to determine the optimal alignment between the sub-levels of hierarchical MSHs.

Unlike recent works that rely directly on individual joint locations, our study is related to explicit skeleton model approaches. In particular, this study capitalizes on a new feature space that has not been previously considered. Therefore, all the human skeleton joints are represented as a collection of measured features in static pose and joint motion nature. An action classification is performed using the extreme learning machine (ELM) classifier and SH-based skeleton representation. The proposed method is evaluated based on recent skeletonbased 3D action datasets.

In this paper, we present an improved skeleton representation by applying the MSHs and spatiotemporal modeling of skeleton joints along the sequence level. The ELM is also compared with other classification algorithms using the same datasets.

Related Work

In this section, we briefly summarize various skeleton-based human action recognition approaches. In particular, the methods related to our developed technique are reviewed because they rely on skeleton joint data only. For a recent detailed survey on human motion analysis from depth data, see references [START_REF] Aggarwal | Human activity recognition from 3d data: A review[END_REF], and [START_REF] Ye | A survey on human motion analysis from depth data[END_REF]. 3D pose-based approaches have been explored by various researchers. [START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF] indicated that the application of skeleton data (e.g., positions, velocities and angles of a joint from a human articulated body) outperforms gray-based features captured by 2D cameras in an indoor environment scenario. In general, many useful features can be initially extracted from RGB-D skeletal data. The majority of these features can be divided into two: those that are based on the angular characteristics of joints and those that are based on the generic 3D coordinates of joints. In certain action recognition methods, the features are developed in complex models to form the representation of the motion sequences.

3D joint positions are commonly extracted as features through four mechanisms. First, raw 3D data are recognized directly without any further processing [START_REF] Raptis | Flexible dictionaries for action classification, in: The 1st International Workshop on Machine Learning for Visionbased Motion Analysis -MLVMA[END_REF][START_REF] Shimada | Gesture recognition using sparse code of hierarchical som[END_REF][START_REF] Wang | Recognition of human actions using motion capture data and support vector machine[END_REF]. Second, these data are further processed to address certain challenges [START_REF] Barnachon | A real-time system for motion retrieval and interpretation[END_REF][START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF][START_REF] Zhao | Human action recognition based on semi-supervised discriminant analysis with global constraint[END_REF]. Third, the distances between each joint can be used as a distance-based feature vector for each frame [START_REF] Antnio | Distance matrices as invariant features for classifying mocap data[END_REF]. Fourth, the features for the selected joints can be simply calculated with reference to the relative distance between joints [START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF].

In [START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF], the human body skeleton was interpreted by directly constructing 3D skeleton joint locations as covariance descriptors, and the temporal evolutions of the action dynamic were modeled using a temporal hierarchy of covariance descriptors. In [START_REF] Lv | Recognition and segmentation of 3-d human action using hmm and multi-class adaboost[END_REF], the 3D coordinates of the joints were used for a skeleton representation of the human body. Correspondingly, the temporal nature of the action sequence was modeled with a generative discrete hidden Markov model (HMM), and action recognition was performed using the multiclass AdaBoost. The view-invariant representation of the human skeleton was proposed in [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] by partitioning the 3D spherical coordinates into angular spaced bins based on the aligned orientations with respect to a coordinate system registered at the hip center. A generative HMM classifier, which addresses the temporal nature of pose observations, was then used to classify each visual code word identified with the cluster method.

The proposed work in [START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF] applied the idea of the pairwise relative locations of joints to represent the human skeleton. The temporal displacement of this representation was characterized using the coe cients of a Fourier pyramid hierarchy. Moreover, the researchers proposed an action let-based approach, in which the e↵ective joint combinations were selected using a multiple kernel learning approach. In [START_REF] Yang | Eigenjoints-based action recognition using navebayes-nearest-neighbor[END_REF], the skeleton joints were represented by combining the temporal and spatial joint relations. To explicitly model the motion displacement, the researchers adopted a method for skeleton representation on the basis of relative joint positions, temporal motion of joints and o↵set of joints with respect to the reference frame. The resulting descriptors were projected onto eigenvectors using Principle Component Analysis PCA . In this case, each frame was described by an EigenJoint descriptor, and action recognition was performed using the nave Bayes nearest neighbor. The same scheme was used for the skeleton representation in [START_REF] Zhu | Fusing spatiotemporal features and joints for 3d action recognition[END_REF], in which action recognition was achieved by adopting the random forest classifier. The view-invariant action representation framework was proposed by [START_REF] Evangelidis | Skeletal quads: Human action recognition using joint quadruples[END_REF]. In this work, the quadbased skeletal feature was adopted to encode the local relation between joints in quadruple form. Consequently, the 3D similarity invariance was achieved. The researchers also adopted a Fisher kernel representation based on a Gaussian mixture model. Such a representation generates the skeletal quads and invokes a multilevel splitting of sequences into segments to integrate the order of subactions into the vector representation. In (Raviteja et al.), a human skeleton was presented as points in the Lie group. The proposed representation explicitly models the 3D geometric relationships among various body parts using rotations and translations. Given that the Lie group was a curved manifold, the researchers mapped all action curves from the Lie group to its Lie algebra, and the temporal evolutions were modeled with DTW. Our proposed method is di↵erent in many aspects over the previously published methods. To more explicitly examine these issue, we now include the comparison in terms of number of joints used, choice of classifier, computation complexity and dataset variability. In term of number of joints, several techniques such as [ [START_REF] Yang | Eigenjoints-based action recognition using navebayes-nearest-neighbor[END_REF][START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF][START_REF] Raviteja | Human action recognition by representing 3d skeletons as points in a lie group[END_REF][START_REF] Evangelidis | Skeletal quads: Human action recognition using joint quadruples[END_REF][START_REF] Ohn Bar | Joint angles similarities and hog2 for action recognition[END_REF][START_REF] Wang | Recognition of human actions using motion capture data and support vector machine[END_REF]] adopted all skeleton joints for feature extraction. Whereas, our method adopted only nine joints. For the classification task, we used ELM which provides a very high recognition accuracy with very fast training time compared to many other classification methods such as :

• (SVM) used [START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF], • (NBNN) used in [START_REF] Yang | Eigenjoints-based action recognition using navebayes-nearest-neighbor[END_REF] and [START_REF] Seidenari | Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses[END_REF].

• (AdaBoost ) used in [START_REF] Bloom | G3d: A gaming action dataset and real time action recognition evaluation framework[END_REF], • (HHM) used in [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF].

The calculation of our features are computationally light than many previous works which try to classify a large 3D point, such in [START_REF] Raptis | Real-time classification of dance gestures from skeleton animation[END_REF], (Raviteja et al.), [START_REF] Ohn Bar | Joint angles similarities and hog2 for action recognition[END_REF], [START_REF] Zhu | Fusing spatiotemporal features and joints for 3d action recognition[END_REF], and [START_REF] Li | Action recognition based on a bag of 3d points[END_REF]. Moreover, in our method, we perform the evaluation on a variety of datasets compared to many approaches ( [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF], [START_REF] Yang | Eigenjoints-based action recognition using navebayes-nearest-neighbor[END_REF], and [START_REF] Seidenari | Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses[END_REF]). The main contribution of our work was to reveal the potential of features that can be extracted from the 3D joint locations, without requiring the additional processing of the entire depth maps of a sequence, as in [START_REF] Yang | Eigenjoints-based action recognition using navebayes-nearest-neighbor[END_REF], [START_REF] Zhu | Fusing spatiotemporal features and joints for 3d action recognition[END_REF][START_REF] Ohn Bar | Joint angles similarities and hog2 for action recognition[END_REF][START_REF] Ohn Bar | Joint angles similarities and hog2 for action recognition[END_REF].

Methodology

Overview of Spherical Harmonics

SHs are Fourier series defined on the basis of a 2-sphere. Given that a Fourier series is a set of mathematical tools for expanding trigonometric functions, SHs are used to organize unit-sphere functions by angular frequency in terms of spherical coordinates. An illustration of a basis functions by means of their harmonics is configured into a set of rows as shown in Fig. 1. The colors in the Fig. 1 represent positive and negative values of SHs. SHs are adopted in various fields to solve specific types of differential equations, such as the representation of gravitational fields and the modeling of geoscience computational problems. SHs have also been adopted to solve various problems in the context of computer vision applications. Such as, face recognition under unknown lighting challenge was adopted by [START_REF] Zhang | Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics[END_REF], as well as 3D object retrieval [START_REF] Bustos | Featurebased similarity search in 3d object databases[END_REF], and volumetric descriptors [START_REF] Vranic | An improvement of rotation invariant 3d-shape descriptor based on functions on concentric spheres[END_REF]. In this section, we briefly explain SH theory. For a general introduction to SH transform, see [START_REF] Freeden | Spherical Functions of Mathematical Geosciences: A Scalar, Vectorial, and Tensorial Setup[END_REF][START_REF] Silverman | Special Functions and their Applications[END_REF], which present the classical material on SHs. Let (r, ✓, ) : r 2 R + , ✓ 2 [0, 2⇡], 2 [0, ⇡] be the spherical coordinates, and f (✓, ) be the homogeneous harmonic functions on R 3 . In this study, we aim to determine the homogeneous solutions of Laplaces equation, r 2 f = 0, in spherical coordinates. Likewise, we intend to explain how these solutions correspond to the decomposition of eigenfunctions in space L 2 (S 2 ), S 2 = (✓, , r) 2 R 3 . In this case, SH is the generalization of Fourier series to 2-sphere by projecting the square-integrable function S 2 onto Hilbert space L 2 (S 2 ). However, for the spherical coordinates:

x = r sin ✓ cos y = r sin ✓ sin z = r cos ✓. (1)
where the Laplacian of a harmonic function on two-sphere using the spherical coordinates is given by

S 2 f = 1 sin ✓ @ @✓ sin ✓ @ f @✓ ! + 1 sin 2 ✓ @ 2 f @ 2 .
(2)

The final solution of Laplacian in R 3 ( the detailed solution can be found in [START_REF] Silverman | Special Functions and their Applications[END_REF] ) is a set of Legendre function and eigenfunctions expressed as follows:

f (✓, ) = Q(Z m n (cos✓))(exp( jm )). ( 3
)
where Q is a constant. The first term in Eq. 3 is a set of Legendre polynomials, and the second term is the eigenfunctions of the Laplacian on a sphere with an eigenvalue of n(n + 1). The notation of the preceding equation represents the SHs in complex form. In this context, we adopt the notion of real SHs with the degree of n and order of m > 0. Thus, we set

y m n (✓, ) = p 2Q m n cos(m )Z m n (cos ✓). (4)
Q m n are the scaling factors expressed as

Q m n = s (2n + 1)(n |m|)! 4m(n + |m|)! .
(5) [START_REF] Silverman | Special Functions and their Applications[END_REF] specified that any function of the form f (✓, ) can be represented by a set of expansion coe cients on the unit sphere. The complete harmonic basis functions are indexed by two integer constants (i.e., the degree n and the order m). The sampling frequencies of the basis functions over the unit sphere are defined by the values of the order n  m  n. In general, there are 2n + 1 bases. As an illustration example, the visual representations of the real SHs for the azimuth and elevation directions are shown in Fig. 1. The blue portions represent the positive harmonic functions, while the red portions depict the negative ones. The distance of the surface from the origin (rows) indicates the value of harmonics in angular direction (✓, ).

Modified spherical harmonics ( MSHs)

this section presents a proposed feature extraction framework, in which the modified real part notation of SHs is used to represent the spatiotemporal features of skeleton joints and improve human action recognition. The notation cos(m ) denotes the real part of SHs. For the special case of degree m = 2, the modified SHs can be computed given that cos(2 ) is explicitly expressed as:

cos(2 ) = 2 cos 2 1. ( 6
)
Substituting Eq. 6 in Eq. 4 and rearranging the terms in the latter establish the modified real SHs as follows:

y m n = Q m n [2 cos 2 1]Z m n cos ✓. (7)
The quadratic term in 6 captures the angular velocity of joint displacement. This velocity is useful to di↵erentiate the actions involved in a curved motion, such as waving or shape drawing. Thus, for a given action, the angular quantities (e.g., relative angular speed and changes in directions of these joints) can be more stable across objects than their actual 3D positions. However, the MSHs of the local 3D skeleton joints capture discriminant information about di↵erent actions. In other words, the quadratic term in MSHs describes the direction and angular speed of joint motions. Experiments reported in section 6.2 have proven that introducing the quadratic angular velocity and direction of joint dynamics significantly improves the use of the standard SHs.

3D Skeleton Coordinate

The raw data (3D coordinates of the joints) contain useful information about the motion sequence of a human. To estimate the skeleton features, a depth sensor is generally used to easily develop a skeleton model consisting of K joints. To make the 3D joint locations invariant to sensor parameters, the joint positions must be mapped into a unique coordinate system. However, Since only the global 3D coordinates of the skeleton joints are available. Thus, to align the body skeleton with the reference coordinate system, we consider the origin of body coordinates indicated by the hip center and set the horizontal reference vector as the vector directed from the left hip to the right hip. Moreover, the 3D subject coordinates comprise three orthogonal vectors ( , ⇢, ) as depicted in Fig. 2. The first axis (horizontal axis) is from the right hip to the left one. The second axis (perpendicular axis) ⇢ is always directed toward the head and aligned with the vertical dimension of the torso. The second vector is computed by rotating vector by 90 . The third axis stems from the cross product of the two bases ( = ⇥ ⇢). The selection of the hip centerand Right/Le f thip joints ensures an acceptable approximation of the subject coordinates along the joint movement. At this preprocessing stage, a significant reduction of joints is usually performed because these joints are not involved in the final action representation.

Features Extraction

Body pose and local joint displacement must be incorporated into the 3D skeleton-based action descriptor. Therefore, we appropriately organize the skeleton joints in a manner where both static pose and joint displacement can be reliably handled at a specific instance in time. In this context, the skeleton joints are represented in terms of the spherical angles relatively measured with respect to the fixed coordinates, which are more accurate than the joint coordinates or joint di↵erence. To characterize the body pose properties, the spherical angles are quantified in the spherical coordinate. All angles are computed corresponding to the origin reference (i.e., the origin of the spherical coordinate system is placed at the hip center joint coordinate). Only a primitive set of the supported joints is used for the 3D pose representation. In this case, only the joints that correspond to Right/Left elbow, Right/Left wrist, Right/Left knee, Right/Left foot, and Head are selected. For every selected joint J i , the following spherical angles are estimated over the action sequence:

✓(t) = arctan( ⇢ ) (t) = arccos p (⇢ 2 + 2 + 2 ) ! . ( 8
)
where t is the frame index, and ✓ i and i are the estimated spherical angles.

Spatiotemporal-based skeleton joint representation

To further analyze the 3D skeleton joints in terms of their spatiotemporal domain, we propose a new joint representation using MSHs which it is an extension of SHs. SHs are a frequency domain basis for characterizing homogeneous functions defined over a two-sphere. They are the extension of the 1D Fourier series on spherical coordinates. As previously explained, SH defines a set of harmonic functions by solving the angular Laplace equation in spherical coordinates. In our method, we separately project the time series of the spherical angle vector f (✓, ) of local joints and the spatial pose onto a two-sphere.

Let the entire skeleton body be represented by J joints (i.e., J = (1, 2, , j), and the action be performed over T frames. Also, let x t (⇢, ✓, ) 2 R 3 denote the spherical coordinates of the skeleton joints in the human body at each frame. The spherical angle system of the entire action sequence can be constructed as a spatiotemporal system expressed below.

F s2A (✓, ) = Pose ? ? ? ? ? ? ? ? ? y J 1 J 2 . . .

J k

2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4

(✓, ) 1,1 (✓, ) 1,2 . . . (✓, ) 1,T ✓, ) 2,1 (✓, ) 2,2 . . . (✓, ) 2,T . . . . . . . . . . . . (✓, ) J,1 (✓, ) J,2 . . . (✓, ) J,T
3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5

.

(9) where s is the specific action, T is the total number of frames in the action sequence, and J is the number of joints in skeleton sequence.

The spatiotemporal system in 9 combine the spherical angles for temporal displacement(row) and spatial distribution (column) of local join, and pose in the action sequence, respectively.

However, using the system in equation (9 and MSHs concept, the static pose description can be calculated by projecting each column in equation 9 onto the basis function of MSHs [START_REF] Silverman | Special Functions and their Applications[END_REF].

f (! q ) = inf X l=0 l X m= l f m n , Y m n (! q ). ( 10
)
where f (! p ) 2 R is the real valued spherical angle vector. Recall that ! q , q 2 {0, ..., j 1} is the vector pointing at the angle of colatitude ✓ q 2 [0, ⇡] measured down from the upper pole, and the angle of longitude 2 [0, 2⇡] is the argument of the S 2 in spherical coordinates. The expansion coe cients are calculated using

f m n = 4⇡/n n 1 X ↵=0 f (! q ) Y m n (! q ). ( 11
)
where Y m n (! q ) is the modified real SH basis function defined as

Y m n (! q ) = p 2Q m n [2 cos 2 1]Z m n (cos ✓), m = 2. ( 12
)
In Eq. ( 12), Q is the scaling factor, and Z is the associated Legendre polynomials given as.

Z m n (x) = 1 m (2 n n!) (1 + x 2 ) m 2 (d n+m ) (dx n+m ) (x 2 1) n . ( 13
)
The estimated MSHs for the body pose at time t (column of Eq. 9) represent the spatial features of static pose.

Temporal Construction of MSHs in a Hierarchical Model

In 3D skeleton-based action recognition, a compact skeletonbased descriptor should encode the static pose information and the temporal evolution or joint motion at a given time segment. The static pose and joint displacement features of a given skeleton body sequence contain discriminative data about the human action over a time segment. In the previous section, the MSHs capture the spatial dependency of the holistic joints (i.e., pose in frame) and the motion of the local joint properties over the time sequence. To e ciently encode the temporal variation of the local joints over time, each SH of these joints is constructed in a hierarchical manner. The idea of hierarchical construction is inspired by the spatial pyramid matching introduced by [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF] to achieve matching in 2D images. Relying on determining the MSHs calculated in the previous section, we construct the MSHs of the local joints in a multi-level approach. Each MSH covers a specific time window of the action sequence. The MSHs are computed over the entire video sequence from the top level and over the smaller windows at the lower levels. Window overlapping is used to increase the ability of the proposed representation to di↵erentiate multiple actions by sliding from one window to the half of the next one, as depicted in Fig 4 .  Regardless of whether the multiple levels of SHs are used, differentiating the local temporal sequences of various action categories is a di cult task because of numerous issues, including the frame rate variations and the temporal independence in each sublevel. To address these issues, DTW (Muller) is used to compute for a distance between the multiple levels of MSHs for each action category. DTW is a time series alignment algorithm. It compute a path between two sequences by warping the time axis iteratively until an optimal match between the two sequences is found. Similarly, DTW is used to identify the nominal distances between the MSHs of consecutive levels for each local joint. The distance vector for each local joint displacement is then formed. The temporal model of the skeleton joints is encoded for each action category as a concatenation of the distance vector D t = [T 1 , , T J ] . Through the computation of the pose and motion feature vectors of the skeleton joints, an action sequence is represented by a combination of these vectors to form a skeleton representation feature vector.

S = H s + D t . ( 14
)
where and  are the weighting parameters.

The static pose and temporal dynamic of the harmonics contain information about the spatiotemporal function over a time sequence of an action. Therefore, this type of harmonic information can be considered a compact representation of the body skeleton joint and can be used to reliably classify actions.

Action Classification

ELM is a multi-class classifier recently introduced for pattern recognition. The proposed action recognition system incorporates this classifier, which is a version of the feedforward neural network [START_REF] Huang | Extreme learning machine: Theory and applications[END_REF]. Compared with other classifiers, ELM provides significant performances, such as fast learning time and recognition accuracy. In [START_REF] Minhas | Human action recognition using extreme learning machine based on visual vocabularies[END_REF], ELM was adopted for human activity recognition from video data. In recent years, this learning algorithm has been applied to solve skeleton-based human action recognition problems (Chen and Koskela) and many other computer vision problems. In this section, we present a brief review of the theory underlying this type of machine learning. For more details about the classical materials of ELM, see [START_REF] Huang | Extreme learning machine: Theory and applications[END_REF].

Extreme Learning Machine ELM

ELM has been extensively employed for learning singlehidden layer feedforward neural networks [START_REF] Huang | Extreme learning machine: Theory and applications[END_REF]. The hidden nodes in ELM are randomly initialized and do not have to be iteratively tuned. In fact, these nodes remain fixed after initialization. As such, only the input weight parameters must be learned. When the training sample A is given by (x j , y j ), j = [1, , q], in which x j 2 R N and y j 2 R M , the output function of ELM model with L hidden neurons can be expressed as follows:

f l (x) = L X i=1 g i ! i (x) = ⌦(x) G. ( 15
)
where G = [g 1 , . . . , g L ] is the output weight vector relating the L hidden nodes to the m > 1 output nodes, and

⌦(x) = [! 1 (x), ..! L (x)
] is a nonlinear activation function (Huang et al.,Fig. 5. Marked skeleton joints as captured by the Kinect sensor 2006). The system ⌦ i (x) can be written in an explicit form presented as follows:

⌦ i (x) = (⌧ i .x + ✏ i ), ⌧ i 2 R d , ✏ i 2 R. ( 16
)
where ( .) is an activation function with hidden layer parameters (⌧, ✏). In the second stage of ELM learning, the error minimization between training data and output weight ⌦ is solved by using the least square norm depicted below.

mink⌦G Hk 2 , G 2 R N⇤M . ( 17
)
where⌦ defines the system of the layer of hidden neurons given as

⌦ = 2
The optimal solution for minimizing the training error in (17) practically assumes that the number of hidden neurons L is less than that of the training set (i.e., L < Q). Therefore, in using the MoorePenrose generalized inverse of matrix ⌦, the optimal solution for ( 17) is [START_REF] Huang | Extreme learning machine for regression and multiclass classification[END_REF].

G ⇤ = ⌦ ⇤ H. ( 20
)
Where ⌦ ⇤ is the inverse of ⌦.

Alternative Body Skeleton Features

Alternative skeleton representations are adopted as an anther abstraction of the skeleton features which are used for further performance evaluation of our method. These skeleton representations are as follows:

Joint Location ( JL): simply concatenates all joint coordinates in one vector.

pairwise joints di↵erences ( PJDs): concatenation y f = {p i p j |i, j = (1, 2, , K), i j} of all frames.

Magnitude of the Position Velocity ( MPV): the velocity between the same joints of enter frame defined as Y t1,t2 = ||p i,t1 p i,t2 ||. These skeleton representations are fed directly into the classifier to directly compare the proposed MSHs method with the alternative representation schemes (i.e. JL, PJDs, and MPV).

Experimental Setup

To evaluate the e↵ectiveness of the proposed method, we perform action recognition on the proposed feature representation and recently published datasets (i.e., MSR-Action 3D [START_REF] Li | Action recognition based on a bag of 3d points[END_REF],G3D [START_REF] Bloom | G3d: A gaming action dataset and real time action recognition evaluation framework[END_REF], Florence 3D Action [START_REF] Seidenari | Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses[END_REF], and UTKinect-Action [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF]). These datasets are used as benchmarks in the experiment. The action complexity of these datasets varies from simple to complex sequences. In addition to depth data, skeleton data are also provided by these datasets using a Kinect sensor http://msdn.microsoft.com/en-us/library/ hh855352.aspx as required. In all experiments, an ELM classifier is used with the proposed representation. For each dataset, the state-of-the-art skeleton-based methods are extensively compared with the proposed approach. The number of hidden neurons of ELM is experimentally tuned for each dataset. To simplify the computation in each experiment, we set the waiting parameters as = 0.5 and  = 0.5. The frequency bands of MSHs are equal to 2. We consider the cross subject protocol for the test setting in all datasets. In particular, half of the subjects are used for training, and the other half for testing. We also divide the MSR-Action dataset into further subsets, AS1, AS2 and AS3 , similar to [START_REF] Li | Action recognition based on a bag of 3d points[END_REF]. Each subset consists of eight actions, and action recognition is separately performed on each subset. In all experiments, we use nine joints from the body skeletal as the initial input joints to our proposed method, as shown in Fig 5 . The features from these joints are initially used for the skeleton feature representation. All the results in this paper were reported in term of an average accuracy.

Results

Comparison with Various Skeleton Features

The performance of various skeleton representations is evaluated on all datasets, and the e ciency of the proposed MSHs method is compared with the various skeleton representations. Table 1 reports the accuracy of the proposed approach with the corresponding results of di↵erent representation methods based on the MSR-Action dataset. Our findings presented in this table are achieved using three levels of MSHs, while the window overlap in the second and third levels is preserved. Compared with other skeleton representations, the proposed method provides satisfactory results. In particular, the MSHs method improves the average accuracies over JL, PDJs, and MPV by 16.07 %, 18.62 %, and 9.84 % , respectively. These observations clearly indicate the superiority of the proposed MSHs representation over existing skeleton representations. The experiments on MSR dataset indicate that the MSHs-based method have better performance on subsets AS1, AS2, AS3 in modeling complex actions and di↵erentiating similar actions. Table 2 summarizes the recognition accuracies of various skeleton representations on the UTKinect-Action, Florence 3D Action, and G3D datasets. The results reveal that MSHs method significantly outperforms the other skeleton representations on these datasets. In using UTKinect dataset, the average accuracy of the proposed representation is 10.5. % better than that of JL, 9.92% better than that of PJDs, and 5.42 % better than that of MPV. In the case of the Florence dataset, the average accuracy of the proposed representation is 9.54 % better than the average accuracy of, 15.8 % better than the average accuracy of PJDs, and 2.43 % better than the average accuracy of MPV. respectively. In the case of the G3D dataset, the average accuracy of the MSHs-based skeleton representation is 13.83 % better than the average accuracy of JL, 12.47 % better than the the average accuracy of PJDs, and 10.79 % better than the average accuracy of MPV. It is interesting to note that the results from MSHs clearly demonstrate the superiority of the MSHsbased method over various skeleton features on Florence, and G3D datasets.

Comparison with the state-of-the-art

The same datasets are used to compare the performance of the proposed method with those of existing state-of-the-art methods. For each data set, the hidden neurons are reported separately. In all experiments, the results correspond to using three levels of hierarchical SHs, while preserving the overlap in the last two levels. Several recognition results on the MSR-Action 3D dataset are already available in the literature. Table 3 presents the recognition rate of the proposed approach along with those of the corresponding current methods. As indicated in this table, the proposed approach obtains the best results compared with those of most existing methods. In particular, our method provides good Approaches Accuracy HO3DJ [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] 78.97 EigenJoints [START_REF] Yang | Eigenjoints-based action recognition using navebayes-nearest-neighbor[END_REF] 82.30 Joint angles similaritiesOhn [START_REF] Ohn Bar | Joint angles similarities and hog2 for action recognition[END_REF] 83.53 Fusion spatiotemporal [START_REF] Zhu | Fusing spatiotemporal features and joints for 3d action recognition[END_REF] 90.90 Covariance Descriptor [START_REF] Hussein | Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations[END_REF] 90.53 Skeletal Quads [START_REF] Evangelidis | Skeletal quads: Human action recognition using joint quadruples[END_REF] 89.86 Lie Group Raviteja et al. 92.46 SHs Al Alwani and Chahir (2015) 90.94 proposed approach 90.98 results in line with those of some existing methods but outperforms the others. Moreover, The proposed approach generates the best results over the stat-of-the-arts reported in [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF]; [START_REF] Yang | Eigenjoints-based action recognition using navebayes-nearest-neighbor[END_REF]; Ohn [START_REF] Ohn Bar | Joint angles similarities and hog2 for action recognition[END_REF]; [START_REF] Evangelidis | Skeletal quads: Human action recognition using joint quadruples[END_REF]]. We can see that our result is in line with recent stat-of-the-arts such as [START_REF] Zhu | Fusing spatiotemporal features and joints for 3d action recognition[END_REF]; Hussein et al. ( 2013); Al [START_REF] Al Alwani | 3-d skeleton joints-based action recognition using covariance descriptors on discrete spherical harmonics transform[END_REF]], but is inferior to Raviteja et al.. This is probably because these methods either, incorporates a spatiotemporal skeleton joints structures with huge numbers of trees in the classification stage as in [START_REF] Zhu | Fusing spatiotemporal features and joints for 3d action recognition[END_REF], or incorporates geometry relation between local joints in order to models the rotation and translation invariants of the features as lie groups as in Raviteja et al.. Some skeleton based methods like Yang and Tian (2012) use skeleton features based on pairwise di↵erences between joints. However, results obtained on MSR Action 3D dataset show that integrating the evolution of the whole skeleton during the sequence is more discriminative than taking into consideration the joints separately.

In addition, the methods proposed in Yang and Tian (2012); [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF]; Ohn [START_REF] Ohn Bar | Joint angles similarities and hog2 for action recognition[END_REF] have the lack of information about temporal nature of the action,making the recognition less e↵ective compared to our method. In this case, 780 hidden neurons are observed in ELM.

For further evaluation, the proposed approach is applied to the skeleton sequences from UTKinect-Action, Florence, and G3D Action datasets. The performance of the proposed approach in this experiment is also compared with those of the corresponding methods. Table 4 compares our method with various state-of-the-art skeleton-based human action recognition approaches on the UTKinect dataset. The proposed approach gives comparable results. The average accuracy of the proposed representation is 5.10% better than that given in [START_REF] Zhu | Fusing spatiotemporal features and joints for 3d action recognition[END_REF] and 2.08% better than that in [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF]. The number of hidden neurons in this experiment is 640.

Table 5 reports the average recognition accuracies in the case of the Florence dataset. The results reveal that the accuracy of the proposed method is slightly higher than that citepd in [START_REF] Seidenari | Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses[END_REF]. In particular, the performance of the proposed approach is superior over that of the state-of-the-art methods by 4.13%. Our results in this table correspond to 500 hidden neurons for ELM. Fusion spatiotemporal [START_REF] Zhu | Fusing spatiotemporal features and joints for 3d action recognition[END_REF] 87.90 HO3DJ [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] 90.92 Space-time pose Rep. [START_REF] Devanne | Space-time pose representation for 3d human action recognition[END_REF] 91.5 SHs Al [START_REF] Al Alwani | 3-d skeleton joints-based action recognition using covariance descriptors on discrete spherical harmonics transform[END_REF] 91.65 Proposed approach 93.0 The performance of the proposed method is also assessed based on the G3D-Action dataset. Table 6 demonstrates the results, which indicate that our method evidently outperforms the existing skeletal joint-based state-of-the-art methods by achieving better accuracy by 0.59%. In this experiment, 700 hidden neurons exist in the ELM.

Benefit of modified SHs

Table 7 demonstrates that the addition of dynamic features expressed by the second-order term of the real SHs dramatically increases the recognition accuracy compared with the standard SHs (Al [START_REF] Al Alwani | 3-d skeleton joints-based action recognition using covariance descriptors on discrete spherical harmonics transform[END_REF]. The e ciency of using MSHs becomes evident when we compare them with the standard SH descriptors. In Table 4, the recognition accuracies of MSHs are used and compared with those of the standard SHs given in (Al [START_REF] Al Alwani | 3-d skeleton joints-based action recognition using covariance descriptors on discrete spherical harmonics transform[END_REF]. The explicit estimation of angular speed and directions in terms of the second-order function presents a significant performance. For example, in the MSR-Action 3D dataset, the use of the quadratic term in MSHs improves the recognition accuracy by a substantial .04% margin over the standard SHs. In the case of the UTKinect and G3D datasets, the MSHs add a significant improvement of 1.35% and 0.59% to their recognition accuracies respectively. Contrarily, in the Florence dataset, the recognition rate is decreased from 87.5% for SHs to 86.13% for MSHs. Our findings a rm that the angular speed component of the quadratic function in MSHs model is extremely important for action representation with curved displacement. Such a displacement cannot be fitted by the spatiotemporal features of the standard real SHs. To further asses the performance of the proposed approach, we also compare the performance of ELM with that of a support vector machine (SVM) classifier and report the obtained recognition accuracies on the same datasets. We simply use a linear SVM [START_REF] Chang | LIBSVM: A library for support vector machines[END_REF] to compare the recognition algorithms. The recognition accuracies corresponding to ELM and SVM are reported in Table 8. Our finding indicates that ELM performs better than SVM based on the MSR dataset, achieving a recognition accuracy of 90.98% (as opposed to the 86.36 % attained by SVM). Nonetheless, both classifiers exhibit distinctive results in the case of the UTKinect dataset. The performance of SVM based on the Florence dataset is slightly lower than that of ELM; the recognition rate of the latter is 6.07% higher than that of the former. For the G3D dataset, the result of ELM conforms to that of SVM.

Conclusion

In this study, we introduced a novel framework for action recognition based on an explicit model of 3D skeleton joints in a spatiotemporal domain. SH transform was used to explicitly model the angular speed and direction of joints. A novel MSH was also proposed based on the quadratic function of the real part of SHs. According to our framework, all body joints were registered into a body coordinate system to extract the spherical angles of joints expressed in 3D body coordinates. The spatiotemporal system of human action was then constructed and encoded by a set of MSHs of static poses and local joint displacement over time. The temporal evolution of the skeleton joints was characterized in a hierarchical manner. An appropriate skeleton representation of an action was formulated as a vector of combined poses and joint motion features. For the action recognition, ELM was used. The performance of the proposed method was evaluated based on recent 3D skeletonbased datasets. We compared the proposed method with the existing state-of-the-art methods either by adopting pure skeleton data or by directly relying on depth data. The experimental results revealed that depending on the used datasets, the proposed method can obtain results similar to those of the extant methods or outperform them. The findings also indicated that MSHs are well suited for action representation with curved movement and angular direction changes. In summary, our newly proposed method is in line with the recently presented methods for 3D skeleton-based pose representation. The angular direction estimated from skeleton data and its derived SHs are relevant for action recognition and can be successfully used to capture temporal changes in action and obtain a high recognition rate.

Fig. 1 .

 1 Fig. 1. Plots of the spherical harmonic basis functions. Blue indicates positive values and red indicates negative values.(Top), SHs for frequency l=3, and order m=2.(Bottom), frequency l=4, and order m=3.

Fig. 2

 2 Fig. 2. 3-dimensional coordinates corresponding to a human body skeleton. (left) real world coordinate , (right) body reference coordinate transformation.

Fig. 3 .

 3 Fig. 3. Spherical harmonics basis are computed for local joints. The temporal motion of each local joint is mapped onto a unit sphere. The unit spheres in the plot represent individual joint of a person performs a tennis swing action. (Top panel, left to right):Elbow R/L, Wrist R/L, Knee Right. (Bottom panel, left to right ) Knee left, Foot R/L , and Head Joints respectively

Fig. 4 .

 4 Fig. 4. A 3-level representation of Temporal construction of the SHs, SHs l j is the j th Spherical harmonics in the l th level of the hierarchy

Table 1

 1

	. Recognition rates ( in %) for various skeletal representations on
	MSR-Action3D dataset
	subset	JL	PDJ MPV Proposed method
	AS1	72.2 76.22 80.23	89.76
	AS2	69.83 80.47 79.15	91.7
	AS3	82.7	71.4 84.06	92.5
	Average 74.91 72.36 81.14	90.98

Table 2

 2

	. Recognition rates (in %)for various skeletal representations on
	UTKinect Action, Florence3D Action, and G3D Action datasets
	Dataset	JL	PDJ MPV Proposed
	UTKinect	82.5 83.08 87.58	93.0
	Florence3D 76.59 70.33 83.7	86.13
	G3D	79	80.36 82.04	92.89

Table 3 .

 3 Comparison of recognition rates (in %) with the state-of-the-art results on MSR-Action3D dataset

Table 4 .

 4 Comparison of Recognition rates (in %)with the state-of-the-art results using UTKinect dataset

Table 5 .

 5 Comparison of recognition rates (in %) with the state-of-the-art results, using Florence dataset

	Multi-part bagSeidenari et al. (2013)	82.00
	SHs Al Alwani and Chahir (2015)	87.50
	Proposed approach	86.13

Table 6 .

 6 Comparison of recognition rates (in %) with the state-of-the-art results, using G3D dataset

	Bloom et al. (2012)2012	71.04
	Alwani et al. (2014)	80.55
	SHs Al Alwani and Chahir (2015) 92.30
	Proposed approach	92.89

Table 7 .

 7 Comparison of recognition rates (in %) with the SHs-based stateof-the-art results

	Datasets	SHs MSHs
	MSR Action 3D 90.94 90.98
	UTKnect	91.65 93.00
	Florence	87.50 86,13
	G3D	92.30 92.89
	Table 8. Recognition rate (in %)for di↵erent classifiers
	Dataset	SVM ELM
	MSR-action	86.36 90.98
	UTKinect	89.94	93.0
	Florence3D	79.06 86.13
	G3D	91.5	92.89
	6.4. Comparison with other Classifier


Adnan Al Alwani
email: adnan.alalwani@unicaen.fr

Youssef Chahir
email: youssef.chahir@unicaen.fr

3-D SKELETON JOINTS-BASED ACTION RECOGNITION USING COVARIANCE DESCRIPTORS ON DISCRETE SPHERICAL HARMONICS TRANSFORM

Keywords: Spherical Harmonics Transform, Skeleton Joints, Covariance Descriptor, Extrem Learning Machine

In this paper, we explore a new method for skeleton-based human action recognition. First, the normalized angles of local joints are extracted and Spherical Harmonics Transform (SHT) can then be used to explicitly model the angular skeleton by projecting the spherical angles onto unit sphere basis. This enables that the skeleton representation can be decomposed into a basis functions. We adopt the spatiotemporal covariance matrix of the spherical harmonic to capture joints orientations over the human action sequence. Thus, the covariance coefficients of joints are used as a discriminative descriptor for the sequence. We validate the proposed method using Extreme Learning Machine (ELM) classifier and recent published 3D action datasets. Experimental results show that our method performs better than many classical methods.

INTRODUCTION

Human action can be captured by either 2D RGB camera or recent release of 3D sensor. In the former case the holistic or body parts of human are used for human action recognition. however Despite the finding of these approaches, the monocular RGB data is still restricted by the various factors like shading, body part occlusion and background noise. Moreover, 2D video sensors cannot fully register the 3D motion of the human using single camera. On the other hand, the 3D information captured by RGB-Depth (RGB-D) sensor are used recently for human action recognition. The RGB-D such as kinect sensor provides both the 2D image as well as the depth map.

According to the motivation of [START_REF] Shotton | Real-Time Human Pose Recognition in Parts from a Single Depth Image[END_REF] and the articulated structure of human body, human actions can be abstracted by a set of 3-D joints locations of poses. In which, the detection of the human posture by means of skeleton joints is practically achieved through further processing of the depth data. Based on the study of [START_REF] Yao | Does human action recognitio benefit from pose estimation?[END_REF] , it obvious that using joints locations alone may provides a good human motion representation for action recognition task. However, in the field of 3D-based action recognition the trend has been focused on using the depth and skeleton joints data to develop an efficient methods for specific task recognition [START_REF] Ohn-Bar | Joint Angles Similarities and HOG2 for Action Recognition[END_REF], and computer vision applications, etc... To this end, features estimation from skeleton body joints is relatively faster due to low dimensionality of the features vector and encodes a better view invariance. This paper attempts to address the sequences of skeletaljoints representation problem in an explicit model. In this model, a novel feature descriptor is used based on the Spherical Harmonic Transform (SHT) of temporally local joints and the covariance coefficients. The main objective of our approach is based on the calculation of the SHT of spherical angles of local joints to explicitly model the displacement of each individual joint. Unlike the works in [START_REF] Hussein | Human Action Recognition Using a Temporal Hierarchy of Covariance Descriptors on 3D Joint Locations[END_REF][START_REF] Vemulapalli | Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group[END_REF][START_REF] Evangelidis | Skeletal Quads: Human Action Recognition Using Joint Quadruples[END_REF]] that consider the spatial-relation between individuals joints. While the present study is related to recent approaches in skeleton descriptor [START_REF] Hussein | Human Action Recognition Using a Temporal Hierarchy of Covariance Descriptors on 3D Joint Locations[END_REF], it capitalizes on a new feature space, which was not considered in these earlier studies.

Let a spherical coordinates of skeleton joint Ji denoted by (θ, φ), the model of temporal evolution ofJi can be represented using spherical harmonic of θ and φ orientation respectively. Then, to handle frame length variations, for each action category, we introduce the covariance technique to compute the covariance coefficients of each SHs matrix. Collecting the computed covariance coefficients of all selected local joints forms the skeleton features representation for an action sequence. Finally, we perform the evaluation procedure using ELM classifier and multiple 3D action datasets.

RELATED WORKS

With advance imaging techniques, such as Microsoft Kinect, an action recognition is abstracted by a set of simple extracted and low dimensional features. Moreover, using skeleton joints locations captured by this sensor, the body pose estimation, as well as action recognition can be achieved efficiently. In this section, we summarize various methods that only use skeleton data, and more related to our approach. A rich material of the human motion analysis from depth data, can be found in [START_REF] Aggarwal | Human activity recognition from 3d data: A review[END_REF].

Human skeleton was represented in [START_REF] Hussein | Human Action Recognition Using a Temporal Hierarchy of Covariance Descriptors on 3D Joint Locations[END_REF] using 3D skeleton joint locations and the temporal evolutions were modeled using a temporal hierarchy of covariance descriptors. In [START_REF] Lv | Recognition and Segmentation of 3D Human Action Using HMM and Multi-class Adaboost[END_REF], 3D coordinates of joints were used and the action sequence was modeled with a generative discrete HMM. Action recognition was performed using multi-class Adaboost. The proposed work in [START_REF] Wang | Mining Actionlet Ensemble for Action Recognition with Depth Cameras[END_REF] used the idea of pairwise relative locations of the joints in order to represent human skeleton. The temporal displacement were characterized using a coefficients of Fourier pyramid hierarchy. The authors proposed an actionlet-based approach in which learning kernel approach was used in order to effectively candidates the meaningful joint combinations. In [ [START_REF] Yang | EigenJoints-based Action Recognition Using Nave-Bayes-Nearest-Neighbor[END_REF]], the authors adopted a representation based on eigenjoint descriptor calculated from each frame. The action recognition was performed using the Naive-Bayes nearest neighbor. In [START_REF] Zhu | Fusing Spatiotemporal Features and Joints for 3D Action Recognition[END_REF] the task of action recognition is achieved by random forests classifier. A view invariant representation of human skeleton was proposed in [START_REF] Xia | View Invariant Human Action Recognition Using Histograms of 3D Joints[END_REF] by partitioning the 3D spherical coordinates into angular spaced bins, based on the aligned orientations with respect to a coordinate system registered at the hip center. Then, a generative HMMs classifier classifies each visual code word. Besides that, the authors of [START_REF] Evangelidis | Skeletal Quads: Human Action Recognition Using Joint Quadruples[END_REF] used the idea of the skeletal quad to encodes the relation of local joint in a quadruples form. The skeletal quads are generated by a Fisher kernel representation based on Gaussian mixture model. In [START_REF] Vemulapalli | Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group[END_REF] a human skeleton was represented as points in the Lie group. The proposed representation explicitly models the 3D geometric relationships between various body parts, using rotations and translations. Since the Lie group is a curved manifold, they map all the action curves from the Lie group to its Lie algebra and the temporal evolutions were modeled using dynamic time warping (DTW).

SHPERICAL HARMONICS

In this section, we briefly discuss the spherical harmonics transform, the classical introduction of SHs can be found in [START_REF] Freeden | Spherical Functions of Mathematical Geosciences ; A Scalar, Vectorial, and Tonsorial Setup[END_REF]. Spherical harmonics are the solution to a variety of problems that relay on an orthonormal basis s 2 . SHs was used for solving PDEs in geophysics, quantum mechanics, as well as a host of computer vision and computer graphics related applications [START_REF] Zhang | Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics[END_REF][START_REF] Bustos | Feature-based similarity search in 3-D object databases[END_REF].

Suppose that the SHs denoted by y m l , are the angular solution that satisfies Laplace's formula in spherical coordinates: radial distance r ∈ R + , azimuth angle θ ∈ [0, 2π] and elevation angle φ ∈ [0, π]. Then, the standard SHs have the expression [START_REF] Freeden | Spherical Functions of Mathematical Geosciences ; A Scalar, Vectorial, and Tonsorial Setup[END_REF]:

Y m l (θ, φ) = K m l P m l cos(θ) exp(jmφ). (1)
Where P m l cos(θ) are the associated Legendre polynomials of degree l and orderm, defined by the differential equation as

P m l = -1 m (2 l l, !) (1 + x 2 ) m 2 (d l+m ) (dx l+m ) (x 2 -1) l . ( 2
)
The term K m l is normalization constant, equal to

K m l = (l + 1 4m (l -|m|)! (l + |m|)! . (3)
Since the spherical harmonics is analogue to Fourier series on the unit sphere, any functionf may be defined by a set of a linear combination of the harmonics basis. For this , the function x(θ.φ) is decomposed in terms of the spherical harmonics as :

x(θ, φ) = l max l=0 l m=-l C m l Y m l (θ, φ). (4)
Where C m l is the expansion coefficients, and Y m l (.) is the real coefficients of the spherical harmonic given by :

Y m l (θ, φ) = 2K m l cos(mφ)P m l (x), for m > 0. (5)

3D POSE DESCRIPTOR

Overview

Regardless of the skeleton structure being used, temporal sequence discrimination into different action classes is a difficult task due to challenges like frame numbers variations in each action, and temporal joints dependency. To address these problems for each action class, we propose a highly discriminative 3D pose descriptor. Particularly, we introduce a novel skeleton-joints descriptor that is based on covariance between local joints. As shown in Fig. 1, the descriptor is designed by finding the covariance coefficients on the spherical harmonics of local joints. We sample these coefficients over the time of the action sequence. The idea of covariance descriptor was first adopted by [START_REF] Tuzel | Region covariance: a fast descriptor for detection and classification[END_REF] as a region descriptor of an image and texture-based classification [START_REF] Tuzel | Pedestrian detection via classification on Riemannian manifolds[END_REF]. The idea of spatio-temporal patch-based covariance descriptor is recently introduced as an action recognition framework [START_REF] Sanin | Spatio-temporal covariance descriptors for action and gesture recognition[END_REF]. In our work, we compute the spatio-temporal covariance coefficients between local joints along the time sequence.

Covariance-based descriptor for body skeleton joints

Suppose we have the entire skeleton structure is represented by Q joints, and the action is performed over T time sequence (frame). Let H denote harmonics data matrix of a set of spherical harmonics {h 1 , ..h n }. Because sets of related spherical harmonics of Q joints are considered for whole action, the 2-D SHs h i of length m = v × u is expressed in column vector i.e. = vect(h). Thus, the harmonic data H is anM × Q matrix, and defined as H = {h 1 , ..., h Q } where typically,M > Q with fixed Q. Having obtained the harmonic data matrixH, the covariance elements over the sequence T is given by [START_REF] Hussein | Human Action Recognition Using a Temporal Hierarchy of Covariance Descriptors on 3D Joint Locations[END_REF] :

C(H) = 1 T -1 T t=1 (H -H)(H -H). ( 6
)
Where H is the sample mean of H.

In our case, we sample the lower part elements of the covariance matrix C(.). Thus, the length of the descriptor is Q(Q + 1)/2. Where Q is the number of skeleton joints used to represent the action sequence.

ACTION CLASSIFICATION

We employ Extreme Learning Machine ELM for the action classification. Recently, this learning algorithm has been applied to solve recent skeleton-based human action recognition problem [START_REF] Chen | Skeleton-Based Action Recognition with Extreme Learning Machines[END_REF]. ELM has been extensively devoted for learning single hidden layer feedforward neural networks SLFNs [START_REF] Huang | Extreme learning machine for regression and multiclass classification[END_REF]providing fast learning time and accurate results. We start the learning stage by assuming that there are M actions A = A 1 , ...A M and the row vector y = [y 1 , ...y M ] indicates the action that the sequence belongs to. Note that, each action sequence c is represented by the features of its frames, calculated as described in section 4 i.e (c, y) form a set of training pairs for the classifier.

For the training samples P {x, y} where x i ∈ R n and y i ∈ R m , the output function of ELM model with N hidden neurons can be represented as [START_REF] Huang | Extreme learning machine for regression and multiclass classification[END_REF] : Where Ω = [ω 1 , , ω N ] is the output weight vector relate the N hidden nodes to the m > 1 output nodes, and

f n (x) = N i=1 ω i ψ i (x) = Ψ(x)Ω. (7)
Ψ(x) = [ψ 1 (x), ..ψ N (x)
] is a nonlinear activation function [START_REF] Huang | Extreme learning machine for regression and multiclass classification[END_REF]. In particular, the system ψ i (x) can be written in explicit from as :

ψ i (x) = β(τ i .x + i ), τ i ∈ R d , i ∈ R. ( 8
)
Where β(.) is a mapping function, with hidden layer parameters (τ, ). In the second stage of ELM learning, the errors between training data and the output weight Ω, is solved by minimizing the solution of the following term

min ΨΩ -T 2 , Ω ∈ R N * M . (9)
WhereΨ define the system of the hidden neurons layer given as

Ψ =    ψ(x 1 ) . . . ψ(x N )    . ( 10
)
And T is the training data matrix denoted as

T =    t T 1 . . . t T N    . (11)
Using the Moore-Penrose generalized inverse of matrix Ψ, the optimal solution to (9) can be found as [START_REF] Huang | Extreme learning machine for regression and multiclass classification[END_REF].

Ω * = Ψ * T. (12)
Where Ψ * denotes the inverse of Ψ.

EXPERIMENTS

In this section, quantitative results of action recognition are reported and compared. Four public RGB-D datasets, acquired using a Kinect sensor, were used as benchmarks in the experiment. These datasets are: MSR-Action3D dataset [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF], UTKinect-Action dataset [START_REF] Xia | View Invariant Human Action Recognition Using Histograms of 3D Joints[END_REF], Florence3D-Action dataset [START_REF] Seidenari | Recognizing Actions from Depth Cameras as Weakly Aligned Multi-part Bag-of-Poses[END_REF] and gaming G3D dataset [START_REF] Bloom | G3D: a Gaming action dataset and real time action recognition evaluation framework[END_REF]. In all experiments, we used a ELM classifier with the covariance descriptor. Before computing the descriptor, we first need to project the 3D joints data into a common coordinate system to make the joints' coordinates. To achieve this, we select the hip-center as the origin point, and use its coordinates as the common basis. Then, we project and transform all the other skeleton joints on the new center.

Evaluation Settings and Parameters : For MSR-Action3D dataset, the protocol of cross subject test setting was used similar to [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF]. We further divided the dataset into subsets AS1, AS2 and AS3 each consisting of 8 sub-actions. The recognition task was performed on each subset separately and we averaged the results. For the remaining data sets, we divide each dataset into half of the subjects for training and the rest are used for the testing task. We selected nine joints from the body skeletal as shown in Figure 2. These joints were used as an initial features input for descriptor. The number of hidden neurons were selected by experiment to perform high accuracies and our results are compared with state-of-thearts methods that rely only on the skeleton joints description.

RESULTS

Previous recognition results have already been reported in the literature using the MSRAction3D dataset. Table 1 shows the recognition rate per action subset along with the corresponding results of methods that rely on skeleton joints. As we can see, our method gives a good results. More specifically, our method outperforms some of the state-of-the-art methods on this dataset. Similar to [START_REF] Zhu | Fusing Spatiotemporal Features and Joints for 3D Action Recognition[END_REF], we experimented with our approach on a UTKinect-Action and Florence3D-Action datasets, and we Table 1. Comparison of Recognition rates with the state-ofthe-art results on MSR action dataset Histograms of 3D joints [START_REF] Xia | View Invariant Human Action Recognition Using Histograms of 3D Joints[END_REF] 78.97 EigenJoints [START_REF] Yang | EigenJoints-based Action Recognition Using Nave-Bayes-Nearest-Neighbor[END_REF] 82.30 Joint angle similarities [START_REF] Ohn-Bar | Joint Angles Similarities and HOG2 for Action Recognition[END_REF] 83.53 Covariance descriptors [START_REF] Hussein | Human Action Recognition Using a Temporal Hierarchy of Covariance Descriptors on 3D Joint Locations[END_REF] 90.53 Random forests [START_REF] Zhu | Fusing Spatiotemporal Features and Joints for 3D Action Recognition[END_REF] 90.90 Joints as special Lie algebra [START_REF] Vemulapalli | Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group[END_REF] use the same setup in [START_REF] Bloom | G3D: a Gaming action dataset and real time action recognition evaluation framework[END_REF] on a G3D Action dataset. Table 2 summarizes the recognition accuracies of our method compared with current skeleton-based method using UTKinect dataset. In this case the proposed approach gives the best results on these datasets. For example, the average accuracy of our method outperforms the average accuracy of [START_REF] Xia | View Invariant Human Action Recognition Using Histograms of 3D Joints[END_REF] and [START_REF] Zhu | Fusing Spatiotemporal Features and Joints for 3D Action Recognition[END_REF] by 0.73% and 3.75%, respectively. We further evaluate our method using Florence dataset, the recognition rates compared with various methods were reported in table 3, The proposed method gives the best over the results of [START_REF] Seidenari | Recognizing Actions from Depth Cameras as Weakly Aligned Multi-part Bag-of-Poses[END_REF] by 5.5%.

The last experiment was carried out on G3D-Action dataset. The average accuracy of our representation reported in table 4 is 21.26%. This result is better than the average accuracy of [START_REF] Bloom | G3D: a Gaming action dataset and real time action recognition evaluation framework[END_REF]. These results clearly demonstrate the performance of our proposed method over a number of existing skeletal joints-base approaches.

CONCLUSION

The problem of skeleton body representation was explicitly modeled in this paper. We have presented an efficient approach for skeleton-based human action recognition. We adopted the spherical harmonics and covariance technique. We used the spatio-temporal spherical harmonics that characterize the spherical angles of local joints over the entire action sequence. We exploited the idea of covariance components in order to capture the dynamic of the action and provide a relevant descriptor with the a fixed length.

The experimental results tested on a various datasets prove the effectiveness of the proposed method. Results demonstrate that our method can be successfully used for capturing temporal changes in action and achieve a higher recognition rate. In future studies, we will enhance our method for classifying and recognizing different other behaviors.

Fig. 1 .

 1 Fig. 1. Construction of the 3D joints descriptor

Fig. 2 .

 2 Fig. 2. Marked skeleton joints as captured by the Kinect sensor

Table 3 .

 3 Comparison of Recognition rates with the state-ofthe-art results, using Florence dataset

		92.46
	Proposed approach	90.94
	Table 2. Comparison of Recognition rates with the state-of-
	the-art results using UTKinect dataset
	Random forests [12]	87.90
	Histograms of 3D joints [18] 90.92
	Proposed approach	91.65

Table 4 .

 4 Comparison of Recognition rates with the state-of-

	the-art results, using G3D dataset
	Hybrid joints feature + adaboost [9] 71.04
	Proposed approach	92.30