Xuan Son Nguyen
email: xuan-son.nguyen@inria.fr

Thanh Phuong Nguyen
email: thanh-phuong.nguyen@univ-tln.fr

Improving Surface Normals Based Action Recognition in Depth Images

In this paper, we propose a new local descriptor for action recognition in depth images. Our proposed descriptor jointly encodes the shape and motion cues using surface normals in 4D space of depth, time, spatial coordinates and higher-order partial derivatives of depth values along spatial coordinates. In a traditional Bag-of-words (BoW) approach, local descriptors extracted from a depth sequence are encoded to form a global representation of the sequence. In our approach, local descriptors are encoded using Sparse Coding (SC) and Fisher Vector (FV), which have been recently proven effective for action recognition. Action recognition is then simply performed using a linear SVM classifier. Our proposed action descriptor is evaluated on two public benchmark datasets, MSRAction3D and MSRGesture3D. The experimental result shows the effectiveness of the proposed method on both the datasets.

Introduction

Approaches for human action recognition in depth images have received a large attention in recent years thanks to the rich information provided by depth sensors.

These approaches usually exploit depth information to build highly discriminative low-level descriptors, or use skeletal data which can be more easily obtained using depth images to build high-level descriptors. Although many approaches have achieved impressive results, they still face a number of challenges, e.g. rate variations, temporal misalignment, composite actions, noise, human-object interaction. Moreover, most of them require high computation time to extract features and recognize actions.

In this paper, we propose a new local descriptor that relies on surface normals in 4D space of depth, time, spatial coordinates and higher-order partial derivatives of depth values along spatial coordinates. The advantage of our proposed descriptor is that it is low-dimensional while being able to jointly capture the shape and motion cues. In order to perform action recognition, we follow the traditional BoW approach and use Sparse Coding [START_REF] Lee | Efficient Sparse Coding Algorithms[END_REF] and Fisher Vector [START_REF] Sanchez | Image Classification with the Fisher Vector: Theory and Practice[END_REF] for obtaining a global representation of depth sequences.

This paper is organized as follows. Section 2 introduces the related work on action recognition in depth images. Section 3 explains our proposed local descriptor and the methods for feature encoding using SC and FV. Section 4 presents the experimental evaluation of the proposed method. Finally, Section 5 offers some conclusions.

Related Work

Existing approaches for action recognition in depth images can be broadly grouped into three main categories: skeleton-based, depth map-based and hybrid approaches. Xia et al. [START_REF] Xia | View Invariant Human Action Recognition Using Histograms of 3D Joints[END_REF] partitioned the 3D space using a spherical coordinate defined at the hip joint position. Each 3D joint casted a vote into a bin to generate a histogram of 3D joint. These histograms were then used to construct visual words whose temporal evolutions were modeled using discrete Hidden Markov Models [START_REF] Rabiner | A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition[END_REF]. Yang and Tian [START_REF] Yang | EigenJoints-based Action Recognition Using Naive-Bayes-Nearest-Neighbor[END_REF] learned EigenJoints from differences of joint positions and used Naïve-Bayes-Nearest-Neighbor [START_REF] Boiman | In Defense of Nearest-Neighbor Based Image Classification[END_REF] for action classification. Zanfir et al. [START_REF] Zanfir | The Moving Pose: An Efficient 3D Kinematics Descriptor for Low-Latency Action Recognition and Detection[END_REF] relied on the configuration, speed, and acceleration of joints to construct the action descriptor. A modified kNN classifier was then used for action classification. Vemulapalli et al. [START_REF] Vemulapalli | Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group[END_REF] used rotations and translations to represent 3D geometric relationships of body parts in a Lie group [START_REF] Murray | A Mathematical Introduction to Robotic Manipulation[END_REF], and then employed Dynamic Time Warping [START_REF] Müller | Information Retrieval for Music and Motion[END_REF] and Fourier Temporal Pyramid [START_REF] Wang | Mining Actionlet Ensemble for Action Recognition with Depth Cameras[END_REF] to model the temporal dynamics. Du et al. [START_REF] Du | Hierarchical Recurrent Neural Network for Skeleton Based Action Recognition[END_REF] divided the human skeleton into five parts, and fed them to five subnets of a recurrent neural network [START_REF] Schmidhuber | Deep Learning in Neural Networks: An Overview[END_REF]. The representations extracted by the subnets at a layer were hierarchically fused to be the inputs of higher layers. Once the final representations of skeleton sequences have been obtained, actions were classified using a fully connected layer and a softmax layer.

Depth map-based approaches usually rely on low-level features from the spacetime volume of depth sequences to compute action descriptors. Li et al. [START_REF] Li | Action Recognition Based on A Bag of 3D Points[END_REF] proposed a bag of 3D points to capture the shape of the human body and used an action graph [START_REF] Li | Expandable Data-Driven Graphical Modeling of Human Actions Based on Salient Postures[END_REF] to model the dynamics of actions. Representative 3D points used to describe a posture were sampled from a very small set of points in depth maps. Kurakin et al. [START_REF] Kurakin | A Real-Time System for Dynamic Hand Gesture Recognition with A Depth Sensor[END_REF] proposed cell occupancy-based and silhouette-based features which were then used with action graphs for gesture recognition. Wang et al. [START_REF] Wang | Robust 3D Action Recognition with Random Occupancy Patterns[END_REF] introduced random occupancy patterns which were computed from subvolumes of the space-time volume of depth sequences with different sizes and at different locations. Yang et al. [START_REF] Yang | Recognizing Actions Using Depth Motion Mapsbased Histograms of Oriented Gradients[END_REF] projected depth maps onto three orthogonal Cartesian planes to obtain Depth Motion Maps (DMMs) which were used to extract HOG descriptors [START_REF] Mikolajczyk | A Performance Evaluation of Local Descriptors[END_REF] for action recognition. Xia and Aggarwal [START_REF] Xia | Spatio-temporal Depth Cuboid Similarity Feature for Activity Recognition Using Depth Camera[END_REF] proposed a filtering method to extract local spatio-temporal interest points of depth sequences. The histograms of depth pixels in 3D cuboids centered around the extracted interest points were calculated and used in a BoW approach for action recognition. Wang et al. [START_REF] Wang | An Approach to Pose-Based Action Recognition[END_REF] represented actions by histograms of spatial-part-sets and temporal-part-sets, where spatial-part-sets are sets of frequently co-occurring spatial configurations of body parts in a single frame, and temporal-part-sets are co-occurring sequences of evolving body parts. Oreifej and Liu [START_REF] Oreifej | HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences[END_REF] and Yang and Tian [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] relied on surface normals in 4D space of depth, time, and spatial coordinates to capture the shape and motion cues in depth sequences. However, they used different methods to construct action descriptors. The method of [START_REF] Oreifej | HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences[END_REF] used polychorons to quantize possible directions of 4D normals, while the method of [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] used SC to compute visual words and spatial average pooling and temporal max pooling to aggregate local descriptors into a global representation of depth sequences. Chen et al. [START_REF] Chen | Action Recognition from Depth Sequences Using Depth Motion Maps-based Local Binary Patterns[END_REF] proposed a real-time approach that used local binary patterns [START_REF] Wang | Texture Classification Using Texture Spectrum[END_REF] computed for overlapped blocks in the DMMs of depth sequences to construct action descriptors. Action recognition was performed using feature-level fusion and decision-level fusion.

Hybrid approaches combine skeletal data and depth maps to create action descriptors. Wang et al. [START_REF] Wang | Mining Actionlet Ensemble for Action Recognition with Depth Cameras[END_REF] introduced local occupancy patterns computed in spatiotemporal cells around 3D joints which were treated as the depth appearance of these joints. They proposed Actionlet Ensemble Model where each actionlet is a particular conjunction of the features for a subset of 3D joints. An action was then represented as a linear combination of a set of discriminative actionlets which were learned using data mining. Zhu et al. [START_REF] Zhu | Fusing Spatiotemporal Features and Joints for 3D Action Recognition[END_REF] fused spatio-temporal features based on 3D interest point detectors and skeleton joints features using pair-wise joint distances in one frame and joints difference between two consecutive frames.

3 Our Method

Local Feature Descriptor

Our local descriptor is computed using surface normals in 4D space of depth, time, and spatial coordinates [START_REF] Oreifej | HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences[END_REF][START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF]. Those are the extensions of surface normals in 3D space of depth and spatial coordinates [START_REF] Tang | Histogram of Oriented Normal Vectors for Object Recognition with a Depth Sensor[END_REF]. The depth sequence can be considered as a function R 3 → R 1 : z = f (x, y, t), which constitutes a surface in 4D space represented as a set of points (x, y, z, t) satisfying S(x, y, t, z) = f (x, y, t)z = 0. Denote by p t the pixel at frame t of the depth sequence with the spatial coordinates (x, y) and the depth value z. We form a 5-dimensional local descriptor associated with p t as follows:

l(p t ) = [ ∂z ∂x ; ∂z ∂y ; ∂z ∂t ; ∂ 2 z ∂x 2 ; ∂ 2 z ∂y 2 ].
Note that the first three components of l(p t ) correspond to the three components of the normal to the surface S at point (x, y, z, t), which have been shown [START_REF] Oreifej | HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences[END_REF][START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] to capture important shape and motion cues.

Denote by

{p t-n-1 2 , p t-n-1 2 +1 , . . . , p t+ n-1 2
} the set of n neighboring pixels of p t in the temporal dimension, i.e. these pixels have the same spatial coordinates as p t but they are at n neighboring frames of frame t. In order to further introduce the local motion cue, the final feature descriptor v extracted at pixel p t is formed by clustering the 5-dimensional descriptors extracted at pixels

p t-n-1 2 , p t-n-1 2 +1 , . . . , p t+ n-1 2 as follows: v = [l(p t-n-1 2 ); l(p t-n-1 2 +1 ); . . . ; l(p t+ n-1 2
)].

In our experiments, the number of neighboring pixels n is set to n = 5. Our local descriptors are thus 25-dimensional vectors. In comparison with the local descriptors proposed in [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] (termed polynormals) which are 81-dimensional vectors, the dimensionality of our local descriptors is much lower than that of polynormals. As we will see in the next section, we use the method proposed in [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] for feature encoding, which results in action descriptors whose dimensions are proportional to the product of the dimension of a local descriptor and the number of visual words in the codebook. Thus, using the same size for the codebook and the same classifier for action recognition, our method is much more efficient than the method of [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF]. Moreover, we show in the experimental evaluation that our method also outperforms the method of [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] in terms of recognition accuracy.

Feature Encoding

Suppose that for each sequence, we obtain a set of whitened vectors X = {v t , t = 1, . . . , T }, where v t ∈ R M is a local descriptor, M is the dimension of a local descriptor, T is the number of local descriptors extracted in the sequence. In order to construct a global representation of the sequence, we use SC and FV which are explained in more detail in the next sections.

Sparse Coding

Yang and Tian [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] used SC to train a codebook and code local descriptors. In this method, the codebook is computed by solving the following constrained optimization problem:

min D,α α α 1 N N i=1 1 2 ||v i -Dα α α i || 2 2 + λ||α α α|| 1 , subject to d T k d k ≤ 1, ∀k = 1, . . . , K, where {v 1 , . . . , v N } is a subset of X, D in R M ×K is the codebook, each column (d k ) K
k=1 represents a visual word; α α α in R K×N is the coefficients of sparse decomposition; λ is the sparsity inducing regularizer.

Each column (α α α i ) N i=1 is l 1 -normalized to obtain the soft assignment α k,i of v i to the k th visual word. Assuming that the depth sequence is partitioned into different spatio-temporal grids. Then a local descriptor is computed for each spatiotemporal grid and the final descriptor of the depth sequence is the concatenation of the descriptors from all the grids. For each spatio-temporal grid and each visual word, the average of the coefficient-weighted differences is computed over all 3D points at the same frame of the grid:

u k (f ) = 1 |N f | i∈N f α k,i (v i -d k ),
where N f is the set of 3D points at frame f of the grid.

If

u k (f ) = [u k,1 (f ); . . . ; u k,M (f )], then the vector representation u k = [u k,1 ; . . . ; u k,M
] of the k th visual word for the grid is computed by taking the maximal value over all the frames of the grid for each dimension:

u k,i = max f =1,...,F u k,i (f ), for i = 1, . . . , M,
where the frame indices of the grid are supposed to be 1, . . . , F .

The vector representation U of the grid is the concatenation of the vectors u k :

U = [u 1 ; . . . ; u K ].
In our experiments, we used the spatio-temporal grids proposed in [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] and illustrated in Fig. 1, where the largest spatio-temporal grid corresponds to the bounding box of the action, and adaptive temporal pyramid is used to take into account the variations in motion speed and frequency when different people perform the same action.

Fisher Vector

FV [START_REF] Perronnin | Fisher Kernels on Visual Vocabularies for Image Categorization[END_REF] assumes that the generation process of local descriptors v t can be modeled by a probability density function p(.; θ) with parameters θ. In order to describe the contribution of individual parameters to the generative process, one can compute the gradient of the log-likelihood of the data on the model:

G X θ = 1 T ∇ θ log p(X; θ).
The probability density function is usually modeled by Gaussian Mixture Model (GMM), and θ = {w 1 , µ µ µ 1 , σ σ σ 1 , . . . , w K , µ µ µ K , σ σ σ K } are the model parameters of the K-component GMM, where w k , µ µ µ k and σ σ σ k are respectively the mixture weight, mean vector, and diagonal covariance matrix of Gaussian k. In our work, we use the following formulas [START_REF] Perronnin | Improving the Fisher Kernel for Largescale Image Classification[END_REF] to calculate the FV components:

G X µ µ µ k = 1 T √ w k T t=1 γ t (k) v t -µ µ µ k σ σ σ k , G X σ σ σ k = 1 T √ 2w k T t=1 γ t (k) (v t -µ µ µ k ) 2 σ σ σ 2 k -1 ,
where γ t (k) is the soft assignment of v t to Gaussian k:

γ t (k) = w k N (v t ; µ µ µ k , σ σ σ k ) K i=1 w i N (v t ; µ µ µ i , σ σ σ i ) , where N (v; µ µ µ k , σ σ σ k ) are M -dimensional Gaussian distributions. The FV represen- tation of a sequence is the concatenation of G X µ µ µ k and G X σ σ σ k . Since G X µ µ µ k and G X σ σ σ k are M -dimensional vectors, our FVs are 2M K-dimensional vectors.
The GMM parameters can be trained using the Expectation-Maximization (EM) [START_REF] Dempster | Maximum Likelihood from Incomplete Data via the EM Algorithm[END_REF] algorithm to optimize a Maximum Likelihood criterion. While training a GMM using EM is often more expensive than training a codebook using SC (if the same number of visual words is used in both cases), the advantage of the FV encoding method is that once the GMM has been learned, the calculation of action descriptors using FV is much faster than that using the method in Section 3.2.1.

We apply two normalization steps: l 2 -normalization and power normalization [START_REF] Sanchez | Image Classification with the Fisher Vector: Theory and Practice[END_REF], as they have been shown [START_REF] Perronnin | Improving the Fisher Kernel for Largescale Image Classification[END_REF] to significantly improve the recognition accuracy when the FV is combined with a linear classifier. In the power normalization step, the FV is transformed using the function of the form: z ← sign(z)|z| ρ with 0 < ρ ≤ 1. In our experiments, ρ is set to ρ = 1 2 . In order to further introduce the spatial geometry and temporal order of a depth sequence, we use the spatio-temporal grids given in Fig. 1. We compute the FV representation for each grid, and concatenate the FVs of all the grids to obtain the final representation of the depth sequence.

Action Recognition

Using the two methods presented in Sections 3.2.1 and 3.2.2 to encode local descriptors into a global representation of a depth sequence, we obtain two algorithms termed SN4D-SC and SN4D-FV. For both the algorithms, we rely on a linear SVM trained in an one-versus-all fashion to build a multi-class classifier for action recognition. In our experiments, we use the LIBLINEAR library [START_REF] Fan | LIBLINEAR: A Library for Large Linear Classification[END_REF] that provides the implementation of linear SVMs.

Experiments

In this section, we evaluate SN4D-SC and SN4D-FV on two benchmark datasets: MSRAction3D [START_REF] Li | Action Recognition Based on A Bag of 3D Points[END_REF] and MSRGesture3D [START_REF] Wang | Robust 3D Action Recognition with Random Occupancy Patterns[END_REF], and compare them against several Method Accuracy Bag of 3D Points [START_REF] Li | Action Recognition Based on A Bag of 3D Points[END_REF] 74.70 HOJ3D [START_REF] Xia | View Invariant Human Action Recognition Using Histograms of 3D Joints[END_REF] 79.00 EigenJoints [START_REF] Yang | EigenJoints-based Action Recognition Using Naive-Bayes-Nearest-Neighbor[END_REF] 82.30 Random Occupancy Pattern [START_REF] Wang | Robust 3D Action Recognition with Random Occupancy Patterns[END_REF] 86.50 Actionlet Ensemble [START_REF] Wang | Mining Actionlet Ensemble for Action Recognition with Depth Cameras[END_REF] 88.20 Depth Motion Maps Based HOG [START_REF] Yang | Recognizing Actions Using Depth Motion Mapsbased Histograms of Oriented Gradients[END_REF] 88.73 HON4D [START_REF] Oreifej | HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences[END_REF] 88.89 DSTIP [START_REF] Xia | Spatio-temporal Depth Cuboid Similarity Feature for Activity Recognition Using Depth Camera[END_REF] 89.30 Pose Set [START_REF] Wang | An Approach to Pose-Based Action Recognition[END_REF] 90.00 Moving Pose [START_REF] Zanfir | The Moving Pose: An Efficient 3D Kinematics Descriptor for Low-Latency Action Recognition and Detection[END_REF] 91.70 SNV [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] 93.09 SN4D-SC 95.45 SN4D-FV 91.07 state-of-the-art methods. For SN4D-SC, the number of visual words was set to K = 100. For SN4D-FV, the number of GMM components K was set to K = 50.

MSRAction3D Dataset

The MSRAction3D is an action dataset captured using a depth sensor similar to Kinect. It contains 20 actions performed by 10 different subjects. Each subject performs every action two or three times. For a fair comparison, we used the experimental setting described in [START_REF] Li | Action Recognition Based on A Bag of 3D Points[END_REF]. We divided the 20 actions into three sub-sets AS1, AS2 and AS3, each having 8 actions. Action recognition was performed on each subset separately. We followed the cross-subject test setting, in which subjects 1,3,5,7,9 were used for training and subjects 2,4,6,8,10 were used for testing.

Tab. 1 shows the accuracy of the proposed methods and different state-of-theart methods. As can be observed, SN4D-SC outperforms the other methods, including HON4D and SNV which also use surface normals in 4D space of depth, time, and spatial coordinates to form local descriptors. The accuracy of SN4D- SC, SN4D-FV and some state-of-the-art methods on AS1, AS2, AS3 is given in Tab. 2. SN4D-SC achieves the highest average accuracy. The confusion matrices are shown in Fig. 2. For both SN4D-SC and SN4D-FV, most of the confusion is observed between actions that share similar arm motion like tennis serve and pick up & throw (AS1 and AS3), draw circle and high arm wave (AS2).

Method

MSRGesture3D Dataset

The MSRGesture3D dataset contains 12 dynamic hand gestures defined by the American sign language. Each gesture is performed two or three times by 10 subjects.

For a fair comparison, we used the leave-one-subject-out cross validation scheme proposed by [START_REF] Wang | Robust 3D Action Recognition with Random Occupancy Patterns[END_REF]. Tab. 3 shows the accuracy of our methods and some state-ofthe-art methods. We can observe that SN4D-FV achieves the highest accuracy. SN4D-SC also competes with SNV and outperforms the remaining methods. The confusion matrices are shown in Fig. 3. For SN4D-SC, the most confusion occurs between the actions finish and bathroom. For SN4D-FC, the most confusion occurs between the actions finish and milk.

Effect of Second-Order Partial Derivatives in The Local Descriptor

In this section, we show the effectiveness of ∂ 2 z ∂x 2 and ∂ 2 z ∂y 2 in the proposed local descriptor by comparing SN4D-SC and SN4D-FV against their variants termed SN4D-SC v and SN4D-FV v , where the components ∂ 2 z ∂x 2 and ∂ 2 z ∂y 2 are removed from the local descriptor. Tab. 4 shows the accuracy of the four methods. We can observe that in terms of average accuracy over the two datasets, SN4D-SC performs slightly better than SN4D-SC v and SN4D-FV outperforms SN4D-FV v . This result shows that second-order partial derivatives of depth values along x and y axes carry useful shape cues for improving the recognition accuracy.

Conclusion

In this paper, we proposed a new local descriptor for human action recognition in depth images which jointly captures the shape and motion cues using surface Method Accuracy Action Graph on Silhouette [START_REF] Kurakin | A Real-Time System for Dynamic Hand Gesture Recognition with A Depth Sensor[END_REF] 87.70 Random Occupancy Pattern [START_REF] Wang | Robust 3D Action Recognition with Random Occupancy Patterns[END_REF] 88.50 Depth Motion Maps Based HOG [START_REF] Yang | Recognizing Actions Using Depth Motion Mapsbased Histograms of Oriented Gradients[END_REF] 89.20 HON4D [START_REF] Oreifej | HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences[END_REF] 92.45 SNV [START_REF] Yang | Super Normal Vector for Activity Recognition Using Depth Sequences[END_REF] 94.74 SN4D-SC 94.2 SN4D-FV 95.03 normals in 4D space of depth, time, spatial coordinates and second-order partial derivatives of depth values along spatial coordinates. Using SC and FV for feature encoding, we obtain high-dimensional action descriptors which are then fed into a linear SVM for efficient classification. We experimentally showed that the proposed methods are effective and more accurate than many state-of-the-art methods on two benchmark datasets.

Figure 1 :

 1 Figure 1: The spatio-temporal grids used in our experiments. From left to right: the first, second and third temporal pyramids

Figure 2 :

 2 Figure 2: (Best viewed in color) The confusion matrices of our methods on MSRAction3D (top: SN4D-SC, bottom: SN4D-FV, left: AS1, middle: AS2, right: AS3).

Figure 3 :

 3 Figure 3: (Best viewed in color) The confusion matrices of our methods on MSRGesture3D (left: SN4D-SC, right: SN4D-FV).

Table 1 :

 1 Recognition accuracy comparison of our methods and previous approaches on MSRAction3D.

Table 2 :

 2 Recognition accuracy comparison of our methods and previous approaches on AS1, AS2 and AS3 of MSRAction3D.

	AS1	AS2	AS3	Ave.

Table 3 :

 3 Recognition accuracy comparison of our methods and previous approaches on MSRGesture3D.

		MSRAction3D MSRGesture3D Ave.
	SN4D-SC v	94.04	94.28	94.16
	SN4D-FV v	86.39	94.85	90.62
	SN4D-SC	95.45	94.2	94.82
	SN4D-FV	91.07	95.03	93.05

Table 4 :

 4 Effect of second-order partial derivatives in our local descriptor.