Rim Slama
email: rim.slama@telecom-lille.fr

Hazem Wannous
email: hazem.wannous@telecom-lille.fr

Mohamed Daoudi
email: mohamed.daoudi@telecom-lille.fr

Grassmannian Representation of Motion Depth for 3D Human Gesture and Action Recognition

Recently developed commodity depth sensors open up new possibilities of dealing with rich descriptors, which capture geometrical features of the observed scene. Here, we propose an original approach to represent geometrical features extracted from depth motion space, which capture both geometric appearance and dynamic of human body simultaneously. In this approach, sequence features are modeled temporally as subspaces lying on Grassmannian manifold. Classification task is carried out via computation of probability density functions on tangent space of each class tacking benefit from the geometric structure of the Grassmaniann manifold. The experimental evaluation is performed on three existing datasets containing various challenges, including MSR-action 3D, UT-kinect and MSR-Gesture3D. Results reveal that our approach outperforms the state-of-theart methods, with accuracy of 98.21% on MSR-Gesture3D and 95.25% on UT-kinect, and achieves a competitive performance of 86.21% on MSR-action 3D.

I. INTRODUCTION

Thanks to the advancement in information technologies, effective and inexpensive depth video cameras, like Microsoft Kinect or Asus Xtion PRO LIVE, are increasingly used in the domain of computer vision. One of the most active research area in this domain is human action recognition. The motivation behinds the great interest for action recognition is the large number of possible applications in: consumer interactive entertainment and gaming [START_REF] Fothergill | Instructing people for training gestural interactive systems[END_REF], surveillance systems [START_REF] Lao | Automatic video-based human motion analyzer for consumer surveillance system[END_REF], lifecare systems and smart home systems [START_REF] Jalal | Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home[END_REF].

The main challenges in almost action recognition system are the accuracy of acquisition data and the dynamic modeling of the movements. The major problems in video based human action recognition which can alter the way actions are perceived, and consequently be recognized, are: occlusions, shadows and background extraction, lighting condition variations and viewpoint changes. The recent release of consumer depth cameras, like Microsoft Kinect, has significantly alleviate these difficulties that reduce the action recognition performance in 2D video. These cameras provide, in addition to the RGB image, depth stream allowing to discern changes in depth. In this paper, we address the problem of modeling and analyzing human motion in depth sequences. Particularly, we recast action recognition problem as a statistical problem on Grassmannian manifold.

In recent years, many approaches dealing with human action and gesture recognition in depth sequence have received growing attention. These approaches can be categorized into 3D joint-based approaches, depth-based approaches and hydride approaches.

First methods used for activity recognition from depth sequences have tendency to extrapolate techniques already developed for 2D video sequences to depth ones. These approaches use all points in depth map sequences as a gray pixels in images to extract meaningful spatiotemporal descriptors. In [START_REF] Li | Action recognition based on a bag of 3d points[END_REF], depth maps of a sequence are projected onto the three orthogonal Cartesian planes and the contours of the projections are sampled for each frame as bag-of-points to model the dynamics of the actions as an action graph. Vieira et al. [START_REF] Vieira | Stop: Space-time occupancy patterns for 3d action recognition from depth map sequences[END_REF], represent each depth map sequence as a 4D grid by dividing the space and time axes into multiple segments in order to extract Spatio-Temporal Occupancy Pattern features (STOP). Also in Wang et al. [START_REF] Wang | Robust 3d action recognition with random occupancy patterns[END_REF], the action sequence is considered as a 4D shape and Random Occupancy Pattern (ROP) features are extracted. In [START_REF] Yang | Recognizing actions using depth motion maps-based histograms of oriented gradients[END_REF], the average difference between the depth frames is computed and summarized in a single Depth Motion Maps (DMM), from which a a Histograms of Oriented Gradients features (HOG) are extracted. Each DMM is generated by projecting depth map from the sequence onto three orthogonal Cartesian planes, computing a motion energy by thresholding the difference between two consecutive maps, and stacking the energies for each projection. A classical SVM classifier is finally used for recognition. By the same token, Oreifej et al. [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF] compute a 4D histogram over depth, time, and spatial coordinates capturing the distribution of the surface normal orientation. This histogram is created using 4D projectors allowing quantification of 4D space.

The availability of 3D sensors has recently made possible to estimate 3D positions of body joints using depth sensors through a real-time 3D joint position prediction system [START_REF] Shotton | Real-time human pose recognition in parts from single depth images[END_REF]. Joint-based approaches have then become popular and many approaches in the literature propose to model the dynamic of the action using these features. Xia et al. [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] comput histograms of the locations of 3D joints as a compact representation. Yang et al. [START_REF] Yang | Eigenjoints based action recognition using naive bayes nearest neighbor[END_REF] proposes to use eigenjoints which combine action information including static posture, motion, and offset. Some hybrid approaches combining both 3D joint data features and depth information were recently introduced. They are trying to take benefit from positive aspects of both approaches. Azary et al. [START_REF] Azary | A spatiotemporal descriptor based on radial distances and 3d joint tracking for action classification[END_REF] propose spatiotemporal descriptors as time-invariant action surfaces, combining features extracted using radial distance measures and 3D joint tracking. In [START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF], local features are computed from patches around the body joints. A structure of a particular conjunction of the features is then defined for a subset of the joints as an actionlet. Finally, an action is represented as a linear combination of the actionlets, where the discriminative weights are learnt via a multiple kernel learning method. In Oreifej et al. [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF], a spatiotemporal histogram (HON4D) is computed over depth sequences to encode the distribution of some 4D normals. Similarly to [START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF], HON4D histograms are computed around joints and concatenated in a spatiotemporal descriptor of the sequence to provide the input of an SVM classifier.

In recent years, variety of techniques reformulating computer vision problems over non-Euclidean spaces, such as Riemannian manifolds, have received growing attention. Such state-of-the-art manifold techniques is presented by Turaga et al. [START_REF] Turaga | Locally time-invariant models of human activities using trajectories on the grassmannian[END_REF], [START_REF] Lui | Tangent bundle for human action recognition[END_REF].

In this paper, data representation considers the geometry of space and incorporates the intrinsic nature of the data. In a such framework, which is 3D depth-based, both geometric appearance and dynamic of human body are captured simultaneously. The main contributions of this paper are: First, a novel proposed approach for gesture and activity recognition from depth sequences, in which motion is represented as a local displacement of the normal vector orientation lying on Grassmaniann manifold. Second, a learning algorithm is introduced using the notion of the tangent space on the manifold, where the classification process is then performed as a function of probability density by Truncated Warped Gaussian on specificclass tangent spaces. Finally, it is demonstrated how this representation capture both geometric appearance and dynamic of human body simultaneously without any information about joint position.

The rest of the paper is organized as follows. In Section II, our approach is described and then Grassmannian manifold and learning algorithm are introduced. Section III presents the experimental results and inroduce the datasets used for evaluations. Section IV concludes the paper.

II. APPROACH

Our approach describes a motion of depth images as sequence features modeled temporally as subspaces lying on Grassmannian manifold. First, we compute local oriented displacements of each sequence and represent it as a time series of angle orientations. Second, action sequence represented by its time series is modeled as an autoregressive and moving average model (ARMA). An observability matrix is then computed using ARMA parameters and represented by its orthonormal basis using Gram shmith. This last operation allows representing the action as a subspaces lying on Grassman manifold. Finally, each action on this manifold is learnt by its mean and covariance according to its class-specific tangent space.

A. 3D oriented displacement features

With a depth sensor, the distance between the pixel position and the depth sensor z, is obtained and quantized into 11-bit digits. The depth information captured by a depth sensor is usually called the depth image. We denote each pixel in the depth image as P = (x; y; z). Let I = [I(1), I(2), ..., I(t), I(⌧ )] denotes the depth sequence. This sequence can be seen as a 4D surface S in the 4D space if we considere a function R 3 ! R 1 : z = I(x; y; t) [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF].

Since the orientation of the normal vector, at every surface point, can describe the surface of an object, the local 4D geometry characteristics (Depth + motion) can be represented as a local displacement of the normal vector orientation. The normals of this surface are given by a derivation of S(x, y, z, t) where S(x, y, z, t) = f (x, y, t) z = 0. Thus, n = 5S = ( @z @x ; @z @y ; @z @t ; 1) T = (n

x , n y , n t , 1) if we follow the same demonstration of Tang et al. [START_REF] Tang | Histogram of oriented normal vectors for object recognition with a depth sensor[END_REF]. Experimentally @z @x , @z @y and @z @t are calculated using the finite difference approximation respectively:

@z @x ' I(x Dif f, y, t) I(x + Dif f, y, t) @z @y ' I(x, y Dif f, t) I(x, y + Dif f, t) @z @t ' I(x, y, t) I(x, y, t + 1) (1)
where Dif f is a positif value of displacement on image matrix. Encoding orientation information of this normal is more meaningful for describing the surface, than (x,y,z,t) coordinates. Thus, these local oriented displacements can be parametrized using spherical coordinates represented as 3 angles ⇥, and descibing respectively zenith angle, azimuth angle and inclination angle. These angles, which are illustrated in Figure 1, are computed as follows:

⇥ = tan 1 ( q n 2 x + n 2 y + n 2 t ) = tan 1 ( ny nx ) = tan 1 ( nt p (n 2 x +ny) ) (2)

B. Temporal modeling

After feature extraction step, the sequence of depth images can be represented as a time series model of features:

F = [f (1), f(2), ..., f(⌧ )].
3D oriented displacement features computed on each image are linearized on a vector f (t) for modeling the time series.

Let

(x, y, t) denotes the angle orientation of a pixel computed between I(t) and

I(t + 1). f (t) = [ (1, 1, t), (1, 2, t), • • • (n, m, t)],
with n ⇥ m = p the resolution of the image I. F = f (1), f(2), , f(T ), with T the number of frames -1 and f 2 R p . A motion sequence can then be seen as a matrix representing a time-series from angle features. Dynamics and continuity of movement implies that action can not be resumed as a simply set of oriented 3D normal because of the temporal information contained in the sequence. Instead of directly using original time-series data, we believe that a linear dynamic system, like that often used for dynamic texture modeling, is essential before manifold analysis. Therefore, to capture both the spatial and the temporal dynamics of a motion, linear dynamical system characterized by ARMA models, will be applied to the time-series matrix M . The dynamic captured by the ARMA model during an action sequence M can be represented as:

p(t) = Cz(t) + w(t), w(t) ⇠ N (0, R), z(t + 1) = Az(t) + v(t), v(t) ⇠ N (0, Q) (3)
where z 2 R d is a hidden state vector, A 2 R d⇥d is the transition matrix and C 2 R 3⇤J⇥d the measurement matrix. w and v are noise components modeled as normal with mean equal to zero and covariance matrix R 2 R p⇥p and Q 2 R d⇥d respectively. The goal is to learn parameters of the model (A, C) given by these equations. Let U P

V T be the singular value decomposition of M . Then, the estimated model parameters A and C are given by: Ĉ = U and

 = P V T D 1 V (V T D 2 V ) 1 P 1 ,

C. Grassmann analysis

Manifold analysis has been widely used with success by various disciplines and for several applications including image set matching, face recognition and action recognition. In this work we are interested in Grassmannian manifolds. Two points U 1 and U 2 on G n,d are equivalent if one can be mapped into the other one by d ⇥ d orthogonal matrix [START_REF] Edelman | The geometry of algorithms with orthogonality constraints[END_REF]. In other words, U 1 and U 2 are equivalent if the d column of U 1 are rotations of U 2 The minimum length curve connecting these two points is the geodesic between them computed as :

d geod (U 1 , U 2 ) =k ✓ 1 , ✓ 2 , • • • , ✓ i , • • • , ✓ d k 2 (4)
where ✓ i is the principal angle vector which can be computed through the SVD of U T 1 U 2 . Grassman analysis provides a natural way to deal with the problem of sequence matching. Specially, as G n,d allows to represent a sequence by a point on its manifold and offer tools to compare and do statistics on this manifold. The classification problem of matching sets of motions represented by a collection of features can be transformed to point classification problem on G n,d .

D. Learning on the Grassmann manifold

The Karcher Mean enables computation of a mean representative for a cluster of points on the Grassmann manifold. The algorithm exploits exp and log maps in a predictor/corrector loop until convergence to an expected point [START_REF] Drira | 3d face recognition under expressions, occlusions, and pose variations[END_REF].

The karcher Mean enables computation of a mean representative for a cluster of points on the manifold. This mean should belong to the same space as the given points. In our case, we need karcher mean to compute averages on the Grassman manifold. Let µ denotes a mean obtained by the karcher mean on a set on sequences {U i } i=1:N belonging to the same class of action. In addition to this mean, we look for the standard deviation value between all actions in each class of training data. The must be computed on {V i } i=1:N where V = exp 1 µ (U i ) are the projections of actions from the Grassmannian manifold into the tangent space defined on the mean µ. The key idea here is to use the fact that the tangent space T µ (G n,d ) is a vector space. Thus, we can estimate the parameters of a probability density function such as a Gaussian and then use the exponential map to wrap these parameters back onto the manifold using exponential map operator (see Figure 2). However, the exponential map is not a bijection for the Grassmannian manifold. In fact, a line on tangent space with infinite length, can be warped around the manifold many times. Thus, some points of this line are going to have more than one image on G n,d . It becomes a bijection only if the domain is restricted. Therefore, we can restrict the tangent space by a truncation beyond a radius of ⇡ in T µ (G n,d ). By truncation, the normalization constant changes for multivariate density in T µ (G n,d ). In fact, it gets scaled down depending on how much of the probability mass is left out of the truncation region.

Let f (x) denote the probability density function (pdf) defined on T µ (G n,d ) by :

f (x) = 1 p 2⇡ 2 e
x 2 2 2

(5) After truncation, an approximation of f gives:

f (x) = f (x)1 |x|<⇡ z ( 6
)
where z is the normalization factor :

z = Z ⇡ ⇡ f (x)1 |x|<⇡ dx (7)
Using Monte Carlo estimation it can be demonstrated that the estimation of z is given by: ẑ = 1

N N X i=1 1 |xi|<⇡ (8)
In practice, we employ wrapped Gaussians in each classspecific tangent space. Separate tangent space is considered for each class at its mean computed by Karcher Mean algorithm. Predicted class of an observation point is estimated in these individual tangent spaces. In the training step, the mean, standard deviation and normalization factor in each class of actions are computed. The predicted label of unknown class action is estimated as a function of probability density in classspecific tangent spaces (see Algorithm 1).

III. EXPERIMENTAL RESULTS

This section summarizes our obtained results and provides an analysis of the performances of our proposed approach tested on several datasets and compared with state-of-the-art methods. The evaluation is made on three publicly datasets, containing various challenges, including: MSR-action 3D [START_REF] Li | Action recognition based on a bag of 3d points[END_REF], UT-kinect [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] and MSR-Gesture3D [START_REF] Wang | Robust 3d action recognition with random occupancy patterns[END_REF]. Example frames from these datasets are shown in Figure 3.

A. MSR-Action 3D dataset

MSR-Action 3D [START_REF] Li | Action recognition based on a bag of 3d points[END_REF] is a public dataset of 3D action captured by a depth camera. Sequences of this dataset consist of a set of temporally segmented actions where subjects are facing the camera and the background is pre-processed clearing discontinuities. Despite this, it is also a challenging dataset since many activities appear very similar due to small interclass variation.

Angle normals computation is performed on cropped area around models. For each frame normal angles features computed on cropped area gives 3800 features.

To reduce this feature dimension, we learnt a low dimension features using PCA. This dimension reduction allows working with features with lower size and also avoid the manipulation of long vectors, whose computation is costly, containing redundant information . The feature vector initially contains 3800 features. This feature dimension can be reduced to 500 while kiping 100% of informations. In our experiments we chose to reduce the feature vector to 200 by kipping 87% of the information. This final feature vector is computed on each frame allowing to build the time series that characterize the action. Then, we fit an ARMA model and we compute observability matrix and its basis which represents action as a point on G n⇥d with n = 200 ⇥ m and d = m = 16.

Table I summarized accuracies of the state-of-the-art methods. To evaluate our approach, we followed the same experimental setup as in Oreifej et al. [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF] and Jiang et al. [START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF], where first five actors are used for training and the rest for testing.

Method

accuracy % Histograms of 3D Joints [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] 78.97 Eigen Joints [START_REF] Yang | Eigenjoints based action recognition using naive bayes nearest neighbor[END_REF] 82.33 DMM-HOG [START_REF] Yang | Recognizing actions using depth motion maps-based histograms of oriented gradients[END_REF] 85.52 HON4D [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF] 85.80 Random Occupancy patterns [START_REF] Wang | Robust 3d action recognition with random occupancy patterns[END_REF] 86.50 HOH4D + D disc [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF] 88.89 We firstly choose to test the efficiency of normal angles seperately, then we use the 3 angles as feature for each image.

We note that our method using angles as features to model the time series gives the best recognition rate comparing to ⇥, or even the three angles together as illustrated in II. As summarized in II,our approach achieves an accuracy of 86.21%, just below the best method from the state-ofthe-art proposed by Oreifej et al. [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF]. Knowing that our approach is based on only 3D oriented displacement features without any information about 3D joint poisitions, compared to other approaches, such as [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF] and [START_REF] Wang | Robust 3d action recognition with random occupancy patterns[END_REF] which use the depth information around joint locations. All results in the rest of experiments are obtained using only angle as feature to represent the time series.

Figure 4 gives more details about recognition per class. The first observation is that using our approach about 10 actions are 100% correctly classified. The second observation is on the misclassified actions which are mainly 3 actions: 'Hammer' confused with 'draw X', 'hand catch' confused with 'draw tick' and 'hight serve' with 'hight throw'.

B. Ut-Kinect dataset

Ut-kinect dataset [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] contains 10 different actions, including: walk, sit down, stand up, pick up, carry, throw, push, pull, wave and clap hands.Each action is performed two times by ten subjects. Sequences are taken using one kinect in indoor settings and their length vary from 5 to 120 frames. We use this dataset because it contains several challenges like view change, significant variation in the realisation of the same action, variation in duration of actions and occlusions. The dataset contains both RGB and depth sequence images but for our experiments we use only depth sequences which resolution is 320 ⇥ 240.

To compare our results with state of the art approaches, we follow experiment protocol proposed in [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] and followed in [START_REF] Devanne | Space-time pose representation for 3d human action recognition[END_REF]. The protocol is leave-one-out cross-validation. Table III compared the recognition accuracy produced using our approach and previous systems. As shown, our approach outperforms the tow methods. Indeed, all the actions are correctly classified with a score more than 90%. Some actions in this dataset include human-object interaction (pick-up, carry, throw), which Devanne et al. [START_REF] Devanne | Space-time pose representation for 3d human action recognition[END_REF] fail to correctly classify these actions since their approach rely totally on skeleton features. Thus, actions like throw (action with object interation) and push (action without object iteration) are classified the same. However, our approach, since it is based on features computed on depth images, overcome this problem. The flaw of Xia's method [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF]

C. Gesture 3D dataset

The gesture 3D dataset [START_REF] Kurakin | A real time system for dynamic hand gesture recognition with a depth sensor[END_REF] contains 336 depth sequences of 12 hand gesture defined by American sign language (ASL). These gestures are: bathroom, blue, finish, green, hungry, milk, past, pig, store, where, j, z. Following experiment setup used by Kurakin et al. [START_REF] Kurakin | A real time system for dynamic hand gesture recognition with a depth sensor[END_REF], the protocol used for evaluation is Leaveone-subject-out-cross-validation. We note that the resolution of depth maps is different from one sequence to an other. In order to ensure the consistensly of the scale, each depth sequence is resized to the same size given images with resolution 50 ⇥ 50. Accuracies obtained with our approach and using state of the art approaches are summarized in table IV. Our performance is better than HON4D presented by Oreifej et al. [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF]. This can be explained by the facrt that HON4D computes histograms of 4D normals while we are using directly the normal information and he is segmenting the sequence into fixed number of cells which is very sensitive to change in exceution rate. Finally, using subspaces allows being robust to noise and missing data and in this dataset, several frames are either empty or with noise.

Method accuracy Oreifej et al. [START_REF] Oreifej | Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences[END_REF] 92.45 Jiang et al. [START_REF] Yang | Recognizing actions using depth motion maps-based histograms of oriented gradients[END_REF] 88.50 Yang et al. [START_REF] Wang | Robust 3d action recognition with random occupancy patterns[END_REF] 89.20 Klaser et al. [START_REF] Klaser | A Spatio-Temporal Descriptor Based on 3D-Gradients[END_REF] 85.23 Our approach 98.21

IV. CONCLUSION

This paper addressed the problem of human gesture and action recognition in depth image sequences. We introduced a novel framework, in which sequence of local oriented displacement features are modeled temporally as subspaces lying in Grassmannian manifold. We then formulated our learning algorithm using the notion of the class-specific tangent space on the Grassmannian manifold. Thanks to statistical tools applied on this riemaniann manifold, separate tangent space is considered for each class and the classification process is performed as a function of probability density by Truncated Warped Gaussian on specific-class tangent spaces. The evaluation of our approach en terms of human activity recognition even in presence of object interaction and hand gesture recognition reveals a remarkable efficiency exceeding 95% on UT-kinect and MSR-Gesture3D datasets.

Fig. 1 .

 1 Fig. 1. 3D angles illustration. From the left to the right the angles ⇥, and .

  where D 1 = [0 0, I ⌧ 1 0] and D 2 = [I ⌧ 1 0, 0 0] where I represents the identity matrix. Comparing two ARMA models can be done by simply comparing their observability matrices. The expected observation sequence generated by an ARMA model (A,C) lies in the column space of the extended observability matrix given by T 1 = [C T , (CA) T , (CA 2 ) T , ...] T . This can be approximated by the finite observability matrix T m = [C T , (CA) T , (CA 2 ) T , ..., (CA 2 ) m ] T [17]. The subspace spanned by columns of this finite observability matrix correspond to a point on a Grassmannian manifold G n⇥d .

Fig. 2 .

 2 Fig. 2. Algorithms for estimating class-conditionals on class-specific poles.

Algorithm 1 : 1 -of v i 3 - 5 - 6 -/ẑ i 3 -

 113563 pdf classification by TWG on class-specific tangent space **** Training **** Input: N training actions as points on G n,d , belonging to k classes: D = {U i , l j , } i=1:N,j=1:k Output: M ultiplication factor ( ẑj ) and standard deviation j for each class for j=1 : k do Compute the karcher mean µ j of the j th class 2-Compute the standard deviation j Sample a large number of points from the Gaussian, N (0, j ), estimated by the fitted Gaussian to the set of points {V i } l j . 4-Count N ⇡ points from N generated ones that lie within a distance ⇡ from the origin of T µj (G n,d ) Compute multiplication factor ẑj = N ⇡ /N Adjust normalization factor for the j th class conditional density f **** Testing **** Input: U : unknown action, ẑj , j Output: l: class label for j=1 : k do 1-Compute v j = log µj (U ) 2-Compute the probability of belonging to class j: fj = f (v j )Predict the class label l of U which belong to the class with maximum probability fj .

Fig. 3 .

 3 Fig. 3. Example frames from different actions obtained respectively, from top to bottom, from MSR-action 3D, UT-kinect and MSR-Gesture3D dataset.

Fig. 4 .

 4 Fig. 4. Confusion matrix for the proposed approach on MSR-Action 3D dataset.

TABLE I

 I

	.	RECOGNITION ACCURACY (IN %) FOR THE MSR-ACTION
	3D DATASET OBTAINED BY THE MOST KOWN STATE-OF-THE-ART
		APPROACHES.

  is that complex actions effects adversely his HMM classification when the number of training samples is small.

	Method	accuracy %
	Histogram of 3D joints [10]	90.92
	Space-time Pose Representation [21] 91.5
	Our approach	95.25
	TABLE III.	RECOGNITION ACCURACY (IN %) FOR THE UT-KINECT
	DATASET USING OUR APPROACH COMPARED TO THE PREVIOUS
		APPROACHES.

TABLE IV .

 IV THE PERFORMANCE ON MSR HAND GESTURE 3DDATASET COMPARED TO PREVIOUS APPROACHES.

ACKNOWLEDGEMENTS

The authors would like to thank Anuj Srivastava for his assistance and useful discussions about this work.


Amine Benelallam

Abel Gómez

Gerson Sunyé

Massimo Tisi

David Launay
email: dlaunay@mia-software.com

Neo4EMF, a Scalable Persistence Layer for EMF Models

Several industrial contexts require software engineering methods and tools able to handle large-size artifacts. The central idea of abstraction makes model-driven engineering (MDE) a promising approach in such contexts, but current tools do not scale to very large models (VLMs): already the task of storing and accessing VLMs from a persisting support is currently inefficient. In this paper we propose a scalable persistence layer for the de-facto standard MDE framework EMF. The layer exploits the efficiency of graph databases in storing and accessing graph structures, as EMF models are. A preliminary experimentation shows that typical queries in reverse-engineering EMF models have good performance on such persistence layer, compared to file-based backends.

Introduction

With large-scale software engineering becoming a compelling necessity in several industrial contexts, companies need tools that are capable to scale efficiently. One of such companies is MIA Software, part of the group Sodifrance, working in the field of software modernization.

The emergence of new techniques and tools for building complex, adaptive and distributed systems has raised a need for the modernization of existing software. A software modernization process follows a systematic approach by first building high level abstractions from source code through reverse engineering, and then using these abstractions to understand, evaluate the quality, extract enterprise architectures and finally, improve the system. A natural approach to reverse engineering is to use Model-Driven Engineering (MDE) tools and in particular those based on the Eclipse Modeling Framework (EMF).

Indeed, EMF has become a de facto standard for building MDE tools, providing a common base for different purposes: reverse engineering [START_REF] Bruneliere | Modisco: A generic and extensible framework for model driven reverse engineering[END_REF][START_REF]MoDisco Eclipse Project[END_REF], model transformation [START_REF]ATLAS transformation language[END_REF][START_REF] Omg | MOF 2.0 QVT final adopted specification (ptc/05-11-01)[END_REF], and code generation [START_REF] Bettini | Implementing Domain-Specific Languages with Xtext and Xtend[END_REF][START_REF] Musset | Acceleo user guide[END_REF]. However, EMF was designed to support modeling activities in the first place and has shown clear limits when dealing with large models, which is often the case of automatically generated models.

While several solutions to persist EMF models exist, they are limited for two reasons. First, most of them do not allow partial model load and unload, and hence, the size of the models they can handle is limited by the memory size; and second, models are structurally graphs and most of the existing solutions are based on relational databases, which are not fully adapted to store graphs.

In this paper we identify specific large-model requirements, discuss the limitations of EMF with this respect, and present a scalable persistence layer for EMF models that meets these requirements. Our persistence layer, Neo4EMF, is built on top of the popular graph database Neo4j. Neo4EMF is open-source, publicly available at [START_REF] Atlanmod | Neo4EMF[END_REF] and it can be immediately used by existing EMF-based tools, without modifying them, to improve their applicability to complex industrial contexts.

Neo4EMF provides two main benefits to the state-of-the-art MDE tools: (i) a scalable access to very large models, with on-demand loading of model elements, (ii) the possibility to exploit the enterprise features of Neo4j, like online backups, horizontal scalability and advanced monitoring. To evaluate this aspect we perform a set of queries in the domain of software modernization, and we compare the execution performance of these queries with the de facto standard persistence layers for EMF: XMI and CDO [START_REF]The CDO Model Repository (CDO)[END_REF].

The paper is organized as follows. Section 2 introduces the concept of persistence layer and graph database, Section 3 describes our proposed persistence layer, Section 4 experimentally evaluates the performance of our layer. Section 5 compares our proposal to existing related work, and finally Section 6 concludes and draws the future perspectives of the tool.

Background

Persistence layers

Software developers often need to persist the state of one or more objects using an existing storage support: relational databases, XML files, etc. There are two main approaches to achieve object persistence. The first one is to hard code the persistence behavior in the class. This approach is efficient and adapted to small applications, but increases the coupling between the class and the storage support. The second approach is to use a persistence layer [START_REF] Ambler | The design of a robust persistence layer for relational databases[END_REF], i. e., a robust and adaptable mechanism that hides storage details from developers and reduces coupling between the storage support and classes. The adaptability of this approach is ensured by a mapping that binds the object model, composed of classes, references, and attributes to the storage model: tables, columns, etc. The object and the storage models can evolve independently, provided a mapping between their concepts is possible. The mapping reduces the development cost of persistent classes, but has a significant impact on the performance.

The emergence of code generation techniques allows developers to adopt a third approach that combines the advantages of the two others. It consists on automatically generating an efficient code for persistence using the correspondence mapping as a generation parameter. Contrary to a persistent layer, the adaptability is not ensured at runtime, but at generation time.

Persistence layers for EMF Since the publication of the XMI standard [START_REF] Omg | OMG MOF 2 XMI Mapping Specification version 2[END_REF], XML-based serialization has been the preferred format for storing and sharing models and metamodels. Some tools, such as EMF [START_REF]Eclipse Modeling Framework Project (EMF)[END_REF], have even adopted it as their canonical representation. However, XMI-based serialization in EMF results to be extremely inefficient: (i) XMI files sacrifice compactness in favor of human-readability and (ii) XMI files need to be completely parsed to obtain a navigational model of their contents. The first factor greatly reduces the efficiency in I/O accesses, while the second greatly increases the memory required to load and query models and limits the use of proxies and on-demand loading to inter-document relationships. Moreover, XMI-based implementations do not provide advanced features such as concurrent modifications, model versioning, or access control out-of-the-box.

The design of CDO [START_REF]The CDO Model Repository (CDO)[END_REF], built on top of EMF, solves most of these problems. CDO was initially envisioned, among other things, as a framework to manage large models in a collaborative environment with a low memory footprint. CDO implements a client-server architecture with transactional and notification facilities where model elements are loaded on demand. CDO servers (usually called repositories) are built on top of different data storage solutions. However, in practice, only relational databases are commonly used. Indeed, only DB Store [START_REF]CDO / DB Store[END_REF], which uses a proprietary Object/Relational mapper, supports all the features of CDO and is regularly released in the Eclipse Simultaneous Release [START_REF]CDO / Hibernate Store[END_REF][START_REF]CDO / MongoDB Store[END_REF][START_REF]CDO / Objectivity Store[END_REF].

Graph databases

The volume of data that organizations gather has grown explosively in recent years, showing a need for solutions that scale-out, as well as the limits of relational databases. To overcome these limits, new technologies for data management have raised, the so-called NoSQL databases [START_REF] Stonebraker | Sql databases v. nosql databases[END_REF]. Despite their non-respect of the ACID properties, these database are able to manage large-scale data on highly distributed environments.

Among the different data models used on NoSQL databases (e. g.column, document, or key-value), graph databases are particularly adapted to store EMF models. The graph data model uses graph structures with nodes, edges, and properties to store data and provides index-free adjacency. Although this data model is not new-the research on graph databases was popular back in the 1990s-it became again a topic of interest due to the large amounts of graph data introduced by social networks and the web in general.

Neo4EMF

Neo4EMF is our proposal for scalable model persistence built on top of the EMF framework. Neo4EMF is an open source project that aims at providing a compatibility layer between the EMF API and a graph-based storage subsystem. Specifically, Neo4EMF is built on top of Neo4j [START_REF] Partner | Neo4j in Action[END_REF], a NoSQL database which is distributed under the terms of the (A)GPLv3.

EMF-based models can easily be described in terms of graph concepts, since there is a natural mapping between the two representations. This natural translation is the main motivation that lead us to choose a native graph database instead of another NoSQL database. Since graph databases like Neo4j have shown good performance for connected data operations, we argue that they are a promising platform for model manipulation.

In this section we first briefly provide an overview of the underlying mapping between EMF models and Neo4j artifacts through a running example, then we describe the main design principles of Neo4EMF.

Mapping EMF models and Neo4j graphs

Figure 1 shows a small excerpt of the Java metamodel provided by MoDisco [START_REF]MoDisco Eclipse Project[END_REF]. This metamodel describes Java programs in terms of Packages, ClassDeclarations, BodyDeclarations and Modifiers. A Package is a named container that groups a set of ClassDeclarations through the ownedElements composition. A ClassDeclaration contains a name and a set of BodyDeclarations. Finally, a Body-Declaration contains a name, and its visibility is described by a single Modifier.

Figure 2 shows a simple instance of this metamodel. This instance contains a single Package (package1), containing only one ClassDeclaration (class1). The Class contains only the bodyDecl1 BodyDeclaration, which is public. Figures 1, 2, and 3 show that:

-Model elements are represented as nodes. Nodes p1, c1, d1 and m1 are examples of this, and correspond to the elements p1, c1, d1 and m1 shown in Figure 2. A ROOT element denotes the model element(s) that directly or indirectly references all the other elements in the model. -Element attributes are represented as node properties -a pair of property name, property value contained in the corresponding node. This can be observed in nodes p1, c1, d1, and m1 again. -Metamodel elements are also represented as nodes. Nodes representing metamodel elements are indexed to ease their access. These kind of nodes also contain two node properties. As it can be seen in nodes P, C, B, and M (which correspond to Package, ClassDeclaration, BodyDeclaration, and Modifier on Figure 1), the first property holds the name of the metamodel element, and the second property the metamodel unique identifier (also known as nsURI ).

Neo4EMF design principles

Figure 3 shows the high-level architecture of Neo4EMF. In this section we introduce the different design principles that we respected in the development of Neo4EMF.

Compliance with standard APIs. In order to keep compliance with EMF, Neo4EMF provides a feature to generate an adapted Java code implementation allowing a refined on-demand loading. To allow for a fine-grained on-demand load mechanism even when using the Java generated API, Neo4EMF provides an adapted code generator supporting all the kinds of EMF generation (reflective, virtual, and dynamic). Neo4EMFObject extends the EMF EObject class with additional metadata such as the id. In addition to the default package organization, we generate a Java class containing a map from the model references to the Neo4j Relationships.

On demand loading. Neo4EMF uses an on-demand loading mechanism that reduces memory footprint and allows programs to load and query large models in systems with limited memory. This capabilities are provided for both the Neo4EMF dynamic API and the Neo4EMF generated Java API. These APIs are kept fully compliant with the standard EMF methods to load, navigate, modify, and save models. When a resource is loaded, only the root elements of the model are charged in memory, without any reference to their features. Depending on the user's query, the rest of the model is to be loaded. Thus, when a feature is queried, Neo4EMF checks if the elements already exist in the cache memory, if not they are loaded from the backend store.

Lightweight model change tracking. Saving model changes in XMI is time consuming, especially when dealing with in large models. The standard serialization mechanisms must traverse the whole resource to save a file. Neo4EMF uses an event-driven change notification approach to keep track of the model changes. Every Neo4EMFObject contains an adapter that sends a notification for each change to a shared listener. Notifications are stored in a ChangeLog model, which is asynchronously analyzed to optimize persistence operations. In this case, instead of traversing the whole resource to save the changes, Neo4EMF queries a ChangeLog model, and saves only the modified elements. Here, a model change can either be a creation of a new element, an edition of feature(s) of an existing one, or a deletion. Figure 4 shows the metamodel of the ChangeLog model.

Lightweight first time loading. Neo4EMF Java code generation separates objects data from their objects, in the sense that, every generated class references to an inner class holding all the class features. This allows a light-weight first time loading of Neo4EMF Objects.

Experimental evaluation

In this section, we evaluate how the access time of Neo4EMF scales in increasingly large scenarios, and we compare it against CDO (with H2 as relational database backend) and XMI. These experiments are performed over 3 EMF models that conform to the Java Metamodel proposed in MoDisco [START_REF]MoDisco Eclipse Project[END_REF] and reverse-engineered from existing code using the MoDisco Java Discoverer. As In order to compare the three technologies, we generate three different EMF access APIs, starting from the Java MoDisco Metamodel respectively with 1) EMF standard parameters, 2) CDO parameters, and 3) Neo4EMF generator. We import the 3 experimental models, originally in XMI format to CDO and Neo4EMF, and we verify that all the imported models contain the same data.

Experiment I : Model traversal. In a first experimentation we execute a model visitor that starting from the root of the model traverses the full containment tree in a depth-first order. At each step of the traversal the visitor loads the element content from the backend, and modifies the element (changing its name). Only the standard EMF interface methods are used by the visitor, that is hence agnostic of which backend he is running on. During the traversal we measure the execution times for covering 0.1%, 1%, 10% 50% and 100% of the model. Fig. 5 shows the results of this experimentation over the two largest test models (org.eclipse.jdt.core and org.eclipse.jdt.*). Experiment II : Java reverse engineering. In a second experimentation we execute a set of three simple queries on the Java metamodel that originate from the domain of reverse-engineering Java code. While the first of these queries is a well-known scenario in academic literature, the other two have been selected to mimic typical model access patterns in reverse engineering, according to the experience of our industrial partner. 1. Grabats (GB): it returns the set of classes that holds static method declarations having as return type the holding class (e. g., Singleton) [START_REF] Jouault | An AmmA/ATL Solution for the GraBaTs 2009 Reverse Engineering Case Study[END_REF]. 2. Unused Method Declarations (UnM): it returns the set of method declarations that are private and not internally called. 3. Class-Attribute Map (CA-M): it returns a map associating each Class declaration to the set of its attribute declarations. All these queries start their computation by accessing the list of all the instances of a particular element type, then apply a filtering to this list to select the starting points for navigating the model. In the experience of our industrial partner this pattern covers the quasi-totality of computationaldemanding queries in the reverse-engineering domain. For this reason we added a method getAllInstances to the EMF API and we implemented it in all the three back-ends. In CDO we implemented this method by a native SQL query, achieved through the union of the tables containing elements of the given type and its subtypes. In Neo4EMF the same result is achieved by a native Neo4j query traversing the graph nodes via the relationship INSTANCE_OF, for the given type and all of its subtypes. The user-code of each of the three queries uses this method to start the computation in all implementation, hence remaining backend-agnostic. Fig. 6 illustrates the result of the experimentation.

Discussion

The results of the two experimentations are consistent with each other. Fig. 5 shows that while in XMI the access time to each model element is trascurable with respect to the initial model-loading time, the two backends with on-demand loading mechanisms have a constant access time (giving linear complexity to the query). This shows that the backends can scale well for even larger sizes. In both experiments in Fig. 5 the backends with on-demand loading mechanisms outperform XMI when the part of the model that needs to be accessed is lower than a certain ratio of the full model. The graphs show that this ratio is approximately constant, independently of the size of the model and it amounts to 14.12% and 12.46% for Neo4EMF and 29.54% and 27.84%. for CDO. The CDO backend performs better than Neo4EMF, by an approximately constant factor that in the two experiments is respectively of 1.38 and 2.6. The results from Fig. 6 show that both Neo4EMF and CDO outperform XMI. The test also confirms the previous result, showing execution times from CDO consistently lower than Neo4EMF.

Summarizing, while resulting a better solution than XMI for the industrial use case under study, the current version of Neo4EMF does not exhibit the performance optimizations in caching and prefetching of more mature solutions like CDO. CDO has two complementary ways of caching, one of CDOObjects placed on the client side, and two other caches maintaining CDORevisions (through the revision manager). Moreover CDO supports partial collection loading that gives the possibility to manage the number of elements to be loaded when an elements is fetched for the first time. Likewise, CDO provides a mechanism to decide how and when fetching the target objects asynchronously.

We also remark that the acceptable performances of XMI may be misleading in a real-world scenario: the amount of memory we used allowed to load the whole models in memory, avoiding any swapping in secondary memory that would have made the XMI solution completely unusable for the scenario. Moreover the use of an SSD hard disk significantly improved the loading & saving times from file. On-demand loading allows to use only the necessary amount of primary memory, extending the applicability of MDE tools to these large scenarios.

We did not measure significant differences in memory occupation between CDO and Neo4EMF, but we noticed several problems in importing large models in CDO. For instance CDO failed to import the test model 3 from its initial XMI serialization on a 8Go machine, as a TimeOutException was raised.

Finally, the comparison with relational databases backend should also take into account several other features, besides execution time and memory in a single processor configuration. Neo4EMF allows existing MDE tools to make use from now of the characteristics of graph databases like Neo4j, including clustering, online backups and advanced monitoring.

Related work

The interest on scalable model persistence has grown significantly in recent years, especially with the advent of new solutions for Model-Driven Reverse Engineering (MDRE) and Software Modernization (MDSM). Tools built on top of the EMF, such as MoDisco [START_REF] Bruneliere | Modisco: A generic and extensible framework for model driven reverse engineering[END_REF]17,[START_REF]MoDisco Eclipse Project[END_REF] have shown that models obtained from reverseengineering processes can normally be composed of millions of elements [START_REF] Jouault | An AmmA/ATL Solution for the GraBaTs 2009 Reverse Engineering Case Study[END_REF]. Existing approaches are not suitable to manage this kind of artifacts both in terms of processing and memory consumption requirements.

CDO is the de facto standard solution to handle large models in EMF by storing them in a relational database. However, different experiences have shown that CDO does not scale well to very large models [START_REF] Pagán | Morsa: A scalable approach for persisting and accessing large models[END_REF][START_REF] Pagán | Querying large models efficiently[END_REF][START_REF] Scheidgen | Automated and transparent model fragmentation for persisting large models[END_REF]. Barmpis and Kolovos [START_REF] Barmpis | Comparative analysis of data persistence technologies for large-scale models[END_REF] suggest that NoSQL databases would provide better scalability and performance than relational databases due to the interconnected nature of models.

Morsa [START_REF] Pagán | Morsa: A scalable approach for persisting and accessing large models[END_REF] was one of the first approaches to provide persistence of large scale EMF models using NoSQL databases. As Neo4EMF, Morsa is based on a NoSQL database. Specifically, Morsa uses MongoDB, a document-oriented database, as its persistence backend. Morsa can be used seamlessly to persist large models using the standard EMF mechanisms. As CDO, it was built using a clientserver architecture. Morsa provides on-demand loading capabilities together with incremental updates to maintain a low workload. Performance of the storage backend and their own query language (MorsaQL) has been reported in [START_REF] Pagán | Morsa: A scalable approach for persisting and accessing large models[END_REF] and [START_REF] Pagán | Querying large models efficiently[END_REF]. Neo4EMF is similar to Morsa in several aspects (notably in on-demand loading) but it aims at exploiting the optimized navigation performance offered by graph-databases w.r.t. document-oriented databases.

Mongo EMF [START_REF] Hunt | [END_REF] is another alternative to store EMF models in MongoDB databases. Mongo EMF provides the same standard API than previous approaches. However, according to the documentation, the storage mechanism behaves slightly different than the standard persistence backend (for example, for persisting collections of objects or saving bi-directional cross-document containment references). For this reason, Mongo EMF cannot be used without performing any modification to replace another backend in an existing system.

EMF fragments [START_REF] Scheidgen | EMF fragments[END_REF] is another NoSQL-based persistence layer for EMF aimed at achieving fast storage of new data and fast navigation of persisted models. Supported backends are MongoDB, Apache Hbase and regular files on the file system. EMF fragments principles are simpler than in other similar approaches and those principles are based on the proxy mechanism used by EMF for inter-document relationships. In EMF fragments, models are automatically partitioned in several chunks (fragments). Unlike Neo4EMF, CDO, and Morsa, all data from a single fragment is loaded at a time, and only links to another fragments are loaded on demand. Another difference with other approaches is that artifacts should be specifically adapted: metamodels have to be modified to indicate where the partitions should be made to get the partitioning capabilities. While our approach has the advantage of not requiring metamodel-specific user manipulation or tool adaptation, fragmentation may provide performance benefits that we plan to investigate in future versions of Neo4EMF.

Conclusions and future work

In this paper we present the first version of Neo4EMF, a tool that can improve the applicability of MDE to large-scale scenarios, where on-demand loading, high-performance access and enterprise-level data-management features are needed. Our preliminary experimentation shows that, while Neo4EMF is a beneficial alternative to XMI for these scenarios, its raw performances do not surpass a more mature solution like CDO.

In our future work we plan to improve the tool by implementing performance optimization strategies, starting from a definition of model partitions, i.e., elements that are loaded in a single transaction, to reduce the total number of transactions during execution. We then plan to study the problem of memory unloading, by deriving unloading strategy from a definition of the possible uses of the persisted model. Finally we want to extend the applicability of Neo4EMF to other graph databases by exploiting recent proposals of common APIs among graph-databases [1], making of Neo4EMF a generic graph-database backend like CDO is for relational databases.

Fig. 1 :Fig. 2 :

 12 Fig. 1: Excerpt of the Java metamodel

Fig. 3 :

 3 Fig. 3: Representation of the sample instance of the Java metamodel in Neo4j

Fig. 5 :

 5 Fig. 5: Results for model traversal on test models 2 and 3.

Fig. 6 :

 6 Fig. 6: Results for scenario 2

Table 1 :

 1 Overview of the experimental sets

	# Plugin	Size	Number of elements
	1 org.eclipse.emf.ecore 24.2MB 121.295
	2 org.eclipse.jdt.core 420.6MB 1.557.007
	3 org.eclipse.jdt.*	984.7MB 3.609.454


Y Komura

Y Okabe

J Phys

Aernout C D Van Enter

Silvano Romano
email: silvano.romano@pv.infn.it

Valentin A Zagrebnov
email: zagrebnov@cpt.univ-mrs.fr

Keywords: AMS 2000 subject classification: 82B20, 82B26, 60K35 Generalised XY-model, Kosterlitz-Thouless versus first-order transition, nonuniversality

We point out that the claim of strong universality of [1] is incorrect, as it contradicts known rigorous results.

The generalised lattice ferromagnetic XY -model was introduced by two of us in [2]. Its simplest version involves 3-component unit spins on a D-dimensional lattice, parameterized by standard angles in the spherical coordinates and interacting according to the ferromagnetic nearest-neighbour Hamiltonian H = -J <i,j> (sinθ i sinθ j ) q cos(ϕ iϕ j ) , J > 0 .

When D = 2, the model yields orientational disorder at all finite temperatures, and produces a transition to a low-temperature Berezinskiǐ-Kosterlitz-Thouless (BKT) phase, possessing slow decay of magnetic correlations and infinite susceptibility [2]. In turn, the transition is expected, and was found by simulation [3], to exhibit the usual BKT scenario, at least for small values of q. On the other hand, this scenario does not hold for all values of q, i.e. the BKT transitions are not universal.

Indeed, we proved in [4] that the named transition turns first-order for sufficiently large q. Recall that a hint of this behaviour was already suggested by numerical analysis [3].

Notice that the above rigorous results entail that both existence and the type of transition depend on the parameter q, i.e. they exclude universality.

The above-mentioned generalized XY -model was recently addressed by simulation for various values of q in Ref. [1]. Although the authors found only a certain evidence of the BKT criticality, they claim that this behaviour holds for all values of q. This claim can not be correct since it contradicts to our rigorous results [4], which seem to have been unknown to the authors of the Ref. [1].