Huy Hieu Pham

Houssam Salmane
email: houssam.salmane@cerema.fr

Louahdi Khoudour
email: louahdi.khoudour@cerema.fr

Alain Crouzil
email: alain.crouzil@irit.fr

Pablo Zegers
email: pablozegers@gmail.com

Sergio A Velastin
email: sergio.velastin@ieee.org

A Deep Learning Approach for Real-Time 3D Human Action Recognition from Skeletal Data

Keywords: Action recognition, Skeletal data, Enhanced-SPMF

We present a new deep learning approach for real-time 3D human action recognition from skeletal data and apply it to develop a vision-based intelligent surveillance system. Given a skeleton sequence, we propose to encode skeleton poses and their motions into a single RGB image. An Adaptive Histogram Equalization (AHE) algorithm is then applied on the color images to enhance their local patterns and generate more discriminative features. For learning and classification tasks, we design Deep Neural Networks based on the Densely Connected Convolutional Architecture (DenseNet) to extract features from enhanced-color images and classify them into classes. Experimental results on two challenging datasets show that the proposed method reaches state-of-the-art accuracy, whilst requiring low computational time for training and inference. This paper also introduces CEMEST, a new RGB-D dataset depicting passenger behaviors in public transport. It consists of 203 untrimmed real-world surveillance videos of realistic normal and anomalous events. We achieve promising results on real conditions of this dataset with the support of data augmentation and transfer learning techniques. This enables the construction of real-world applications based on deep learning for enhancing monitoring and security in public transport.

Introduction

Human Action Recognition or HAR for short, plays a crucial role in many computer vision applications such as intelligent surveillance, human-computer interaction or robotics. Although significant progress has been achieved, detecting accurately what humans do in unknown videos is still a challenging task due to numerous challenges, e.g. viewpoint changes, intra-class variation, or surrounding distractions [START_REF] Poppe | A survey on vision-based human action recognition[END_REF]. At present, depth sensor-based HAR is considered as one of the best available methods for overcoming the above obstacles. Cost-effective depth sensors are able to provide 3D structural information of the human body, which is suitable for HAR task. In particular, most of these devices have integrated the real-time skeleton estimation algorithms [START_REF] Shotton | Real-time human pose recognition in parts from single depth images[END_REF] that are robust to surrounding distractions as well as invariant to camera viewpoints. Therefore, exploiting skeletal data for HAR opens up opportunities for addressing the limitations of RGB and depth modalities. In the literature of skeleton-based action recognition, there are two main issues that need to be solved. The first challenge is how to transform the raw skeleton sequences into an effective representation, which is able to capture the spatio-temporal dynamics of human motions. The second is to model and recognize actions using the motion representation obtained from skeletons. Previous works on this topic can be divided into two main groups: HAR based on hand-crafted features and HAR using deep learning models. The first group of methods extracts hand-crafted local features from skeleton joints and uses probabilistic graphical models such as Hidden Markov Model (HMM) [START_REF] Lv | Recognition and segmentation of 3D human action using HMM and multi-class Adaboost[END_REF], Conditional Random Field (CRF) [START_REF] Han | Discriminative human action recognition in the learned hierarchical manifold space[END_REF], and Fourier Temporal Pyramid (FTP) [START_REF] Vemulapalli | Human action recognition by representing 3D skeletons as points in a lie group[END_REF] to model and classify actions. For instance, since the first work on 3D HAR from depth data was introduced [START_REF] Li | Action recognition based on a bag of 3D points[END_REF], many methods for skeleton-based action recognition have been proposed [START_REF] Han | Discriminative human action recognition in the learned hierarchical manifold space[END_REF][START_REF] Luo | Group sparsity and geometry constrained dictionary learning for action recognition from depth maps[END_REF][START_REF] Vemulapalli | Human action recognition by representing 3D skeletons as points in a lie group[END_REF][START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF][START_REF] Wang | Graph based skeleton motion representation and similarity measurement for action recognition[END_REF][START_REF] Wang | Mining mid-level features for action recognition based on effective skeleton representation[END_REF][START_REF] Wu | Leveraging hierarchical parametric networks for skeletal joints based action segmentation and recognition[END_REF][START_REF] Xia | View invariant human action recognition using histograms of 3D joints[END_REF]. The common characteristic of these approaches is that, they extract the geometric features from the 3D coordinates of the skeleton joints and model their temporal information by a generative model. Although promising results have been achieved, most of these approaches are shallow, data-dependent and require a lot of feature engineering. E.g., they require pre-processing input data in which the skeleton sequences need to be segmented or aligned. In contrast, we propose a skeleton-based representation and a learning framework for 3D HAR that learns to recognize actions from the raw skeletons in an end-to-end manner, without dependence on the length of actions.

The second group considers skeleton-based action recognition as a time-series problem and proposes to use Recurrent Neural Networks with Long Short-Term Memory units (RNN-LSTMs) [START_REF] Hochreiter | Long Short-Term Memory[END_REF] to analyze the temporal evolutions of skeletons. They are considered as the most popular deep learning based approach for the HAR task from skeletons and have achieved high-level performance [START_REF] Du | Hierarchical recurrent neural network for skeleton based action recognition[END_REF][START_REF] Liu | Spatio-temporal LSTM with trust gates for 3D human action recognition[END_REF][START_REF] Liu | Global context-aware attention LSTM networks for 3D action recognition[END_REF][START_REF] Shahroudy | NTU RGB+D: A large scale dataset for 3D human activity analysis[END_REF][START_REF] Si | Skeleton-based action recognition with spatial reasoning and temporal stack learning[END_REF][START_REF] Veeriah | Differential recurrent neural networks for action recognition[END_REF][START_REF] Zhu | Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks[END_REF]. The temporal evolutions of skeletons are in fact spatiotemporal patterns. Thus, they can be modeled by memory cells in the structure of RNN-LSTMs. However, RNN-LSTM based methods tend to overemphasize the temporal information and lose spatial information of skeletons [START_REF] Du | Hierarchical recurrent neural network for skeleton based action recognition[END_REF] -an important characteristic for 3D HAR. Another limitation of RNN-LSTM networks is that they just model the overall temporal dynamics of actions without considering the detailed temporal dynamics of them [START_REF] Lee | Ensemble deep learning for skeleton-based action recognition using temporal sliding lstm networks[END_REF]. Additionally, this approach considers skeletons as a kind of low-level feature by feeding raw skeletal data directly into the network. The huge number of input features makes RNN-LSTMs complex, time-consuming and easily lead to overfitting. Furthermore in many cases, RNN-LSTMs act as a classifier, which cannot extract high-level features for the HAR problem [START_REF] Sainath | Convolutional, Long Short-Term Memory, Fully Connected Deep Neural Networks[END_REF]. In this paper, we propose a CNN-based method to extract rich geometric motion features from skeleton sequences and model various temporal dynamics, including both short-term and long-term actions.

Fig. 1. Overview of the proposed approach for real-time 3D action recognition from skeletal data. Each skeleton sequence is encoded as a single RGB image via a skeletonbased representation called SPMF [START_REF] Pham | Skeletal movement to color map: A novel representation for 3D action recognition with Inception Residual networks[END_REF]. The SPMF is built from Pose Feature vectors (PFs) and Motion Feature vectors (MFs), which are estimated from the 3D coordinates of skeletons. A color enhancement technique [START_REF] Pizer | Adaptive histogram equalization and its variations[END_REF] is adopted to enhance the local textures of SPMF to form the enhanced motion maps, namely Enhanced-SPMF. Finally, they are fed to a deep network for learning image features and performing action classification.

In contrast to the existing approaches, we aim to build an end-to-end deep learning framework for real-time action recognition from skeleton sequences. We believe that an effective motion representation is the key factor influencing recognition performance. Therefore, we propose to encode human poses and motions extracted from the 3D coordinates of skeleton joints into color images. These color-coded images are then enhanced in their local textures by an Adaptive Histogram Equalization (AHE) algorithm [START_REF] Pizer | Adaptive histogram equalization and its variations[END_REF] before feeding into Deep Convolutional Neural Networks (D-CNNs), which are built based on the DenseNet architecture [START_REF] Huang | Densely connected convolutional networks[END_REF]. Before that, a smoothing filter is applied to reduce the effects of noise on the input skeletal data. The overview of the proposed method is illustrated in Fig. 1. Generally speaking, four hypotheses that motivate us to build a skeleton-based representation and design DenseNets for 3D HAR include:

(1) human actions can be correctly represented via movements of the skeleton [START_REF] Johansson | Visual perception of biological motion and a model for its analysis[END_REF]; (2) spatio-temporal evolutions of skeletons can be transformed into color images -a kind of 3D tensor that can be effectively learned by D-CNNs [START_REF] Bilen | Action recognition with dynamic image networks[END_REF][START_REF] Choutas | Potion: Pose motion representation for action recognition[END_REF][START_REF] Ding | Investigation of different skeleton features for cnn-based 3d action recognition[END_REF]. This hypothesis was proved in our previous studies [START_REF] Pham | Skeletal movement to color map: A novel representation for 3D action recognition with Inception Residual networks[END_REF][START_REF] Pham | Learning to Recognize 3D Human Action from A New Skeleton-based Representation Using Deep Convolutional Neural Networks[END_REF][START_REF] Pham | Exploiting deep residual networks for human action recognition from skeletal data[END_REF]; (3) compared to RGB and depth modalities, skeletal data has high-level information with much less complexity. This makes the learning model much simpler and requiring less computation, allowing us to build real-time deep learning framework for HAR task; (4) DenseNet is currently one of the most effective CNN architecture for image recognition. It has a densely connected structure allowing maximal information flow and facilitates features reuse as each layer in its architecture has direct access to the features from previous layers. This helps DenseNet to improve its learning performance. Therefore, we explore and optimise this architecture for learning and recognizing human actions on the proposed image-based representation.

The main contributions of this work are three-fold. First, we introduce Enhanced-SPMF (Enhanced Skeleton Pose-Motion Feature) -a 3D motion representation for HAR tasks (Sect. 2). This is an extended representation of SPMF, which was presented in our previous work [START_REF] Pham | Skeletal movement to color map: A novel representation for 3D action recognition with Inception Residual networks[END_REF]. The new representation aims to improve the efficiency of the SPMF by using a smoothing filter on input skeleton sequences and a color enhancement technique that could make the proposed Enhanced-SPMF more robust and discriminative. An ablation study on the Enhanced-SPMF demonstrates that the new representation leads to better overall action recognition performance than the SPMF. Second, we introduce an end-to-end deep framework based on D-CNNs1 for learning and recognizing actions from the Enhanced-SPMFs (Sect. 3). This approach is general in the sense that it can be applied to other data modalities, e.g. mocap data or the output of 3D pose estimation algorithms. The proposed method is evaluated on two highly competitive benchmark datasets and achieved state-of-the-art performance on both these two benchmark tasks with high computational efficiency (Sect. 4). Finally, we collect and introduce a new RGB-D dataset consisting of real-world surveillance videos for analyzing anomalous and normal events in public transport. Experimental results show that the proposed method achieves promising performance in realistic conditions (Sect. 4).

The rest of this paper is organized as follows: Sect. 2 presents the proposed skeleton-based representation. The proposed deep learning framework is presented in Sect. 3. Datasets and experiments are provided in Sect. 4, including a description of our dataset and the obtained results. Section 5 concludes the paper.

Enhanced Skeleton Pose-Motion Feature

One of the major challenges in exploiting D-CNNs for skeleton-based action recognition is how a skeleton sequence could be effectively represented and fed to the deep networks. As D-CNNs work well on still images [START_REF] Lecun | Deep learning[END_REF], our idea therefore is to encode the spatial and temporal dynamics of skeletons into 2D images [START_REF] Pham | Learning to Recognize 3D Human Action from A New Skeleton-based Representation Using Deep Convolutional Neural Networks[END_REF][START_REF] Pham | Exploiting deep residual networks for human action recognition from skeletal data[END_REF]. Two essential elements for describing an action are static poses and their temporal dynamics. As shown by Zhang et al. in [START_REF] Zhang | On geometric features for skeleton-based action recognition using multilayer LSTM networks[END_REF], the combination of too many geometric features will lead to lower performance than using only a single feature or several main features. Moreover, joint features such as joint distance and joint motion are stronger features than others [START_REF] Yun | Two-person interaction detection using body-pose features and multiple instance learning[END_REF]. Hence, we decide to transform these two important elements into the static spatial structure of a color image. The details of this idea are explained in our previous work [START_REF] Pham | Skeletal movement to color map: A novel representation for 3D action recognition with Inception Residual networks[END_REF], in which the spatio-temporal patterns of a skeleton sequence can be encoded into a single color image as a global representation, namely SPMF, via pose and motion feature vectors. Due to the limited space available, detailed description of the SPMF is not included. Instead, we refer the interested readers to [START_REF] Pham | Skeletal movement to color map: A novel representation for 3D action recognition with Inception Residual networks[END_REF] for further technical details. Figure 2 shows some SPMF representations in form of an image-based representation obtained from MSR Action3D dataset [START_REF] Li | Action recognition based on a bag of 3D points[END_REF]. The color images obtained after the process of encoding mainly reflect the spatio-temporal distribution of skeleton joints. We observe that these images are represented by close contrast values, as can be seen in as Fig. 2. In this case, a color enhancement method can be useful for increasing the contrast of these representations and highlighting the texture and edges of motion maps. This helps to better distinguish similar actions. Therefore, it is necessary to enhance the local features on the generated color images. The Adaptive Histogram Equalization (AHE) [START_REF] Pizer | Adaptive histogram equalization and its variations[END_REF] is a common approach for this task. This technique is capable of enhancing the local features of an image. Mathematically, let I be a given image, represented as a r-by-c matrix of integer pixels with intensity levels in the range [0, L -1]. The histogram of image I will be defined by H k = n k , where n k is the number of pixels with intensity k in I . Hence, the probability of occurrence of intensity level k in I is

p k = n k r × c , (k = 0, 1, 2, ..., L -1). (1)
The histogram equalized image will be formed by transforming the pixel intensities, n, of I by the function

T (n) = floor((L -1) n k=0 p k ), (n = 0, 1, 2, ..., L -1), (2)
The Histogram Equalization (HE) method is used for increasing the global contrast of the image. However, it cannot solve the problem of increasing the local contrast. To do this, the image needs to be divided into R regions and the HE is then applied in each region. This technique is called the Adaptive Histogram Equalization (AHE). Figure 3 shows samples of the enhanced motion maps with R = 8, which we refer to it as Enhanced-SPMF for some actions from the MSR Action 3D dataset [START_REF] Li | Action recognition based on a bag of 3D points[END_REF].

Fig. 3. The corresponding Enhanced-SPMF representations after applying the AHE algorithm [START_REF] Pizer | Adaptive histogram equalization and its variations[END_REF]. This color enhancement step could make the proposed Enhanced-SPMF more robust and discriminative for the representation learning phase with D-CNNs later.

Deep Learning Model

This section reviews the key ideas behind the DenseNet architecture and presents the proposed deep networks for recognizing actions from the Enhanced-SPMFs.

DenseNet Review

DenseNet [START_REF] Huang | Densely connected convolutional networks[END_REF], a recently proposed CNN model, has some interesting properties. Each layer is connected to all the others within a dense block and all layers can access feature maps from their preceding layers. Besides, each layer receives direct information flow from the loss function through shortcut connections. These properties make DenseNet less prone to overfitting for supervised learning problems. Traditional CNN architectures use the output feature maps x l-1 of the (l -1) th layer as input to the l th layer and learn a mapping function x l = H l (x l-1 ). Here, H l (•) is a non-linear transformation that is usually implemented by a series of operations such as Convolution (Conv.), Rectified Linear Unit (ReLU) [START_REF] Glorot | Deep sparse rectifier neural networks[END_REF], Pooling, and Batch Normalization (BN) [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF]. When increasing the depth of the network, the problem of optimization becomes complex due to the vanishing-gradient problem and the degradation phenomenon [START_REF] He | Convolutional neural networks at constrained time cost[END_REF]. To solve these problems, [START_REF] He | Deep residual learning for image recognition[END_REF] introduced ResNet. The key idea behind the ResNet architecture is to add shortcut connections that bypass the non-linear transformations H l (•) with an identity function id(x) = x. Inspired by the philosophy of ResNet, to maximize information flow through layers, Huang et al. [START_REF] Huang | Densely connected convolutional networks[END_REF] proposed DenseNet in which the l th layer in a dense block receives the feature maps of all preceding layers as inputs. That means

x l = H l (concat[x 0 , x 1 , x 2 , ..., x l-1 ]), (3)
where concat[x 0 , x 1 , x 2 , ..., x l-1 ] is a single tensor constructed by concatenation of the previous layer output feature maps. All layers receive direct supervision signal from the loss function through the shortcut connections. Therefore, DenseNets are easy to optimize and resistant to overfitting. In DenseNet, multiple dense blocks are connected via transition layers. Each block with its transition layer produces k feature maps and the parameter k is called as the "growth rate" of the network. The function H l (•) in the original work [START_REF] Huang | Densely connected convolutional networks[END_REF] is a composite function of three consecutive layers: BN-ReLU-Conv.

Network Design

We design D-CNNs based on the DenseNet architecture [START_REF] Huang | Densely connected convolutional networks[END_REF] to learn and classify actions on the Enhanced-SPMF. To study how performance varies with architecture size, we test three different configurations of DenseNet: {DenseNet-16, k = 12}; {DenseNet-28, k = 12}; and {DenseNet-40, k = 12}. Here, the numbers 16, 28, 40 refer to the depth of the network and k is the network growth rate. For computational efficiency, we use three dense blocks on 32 × 32 input images. The H l (•) function is implemented by a sequence of layers: Batch Normalization (BN), advanced activation layer named Exponential Linear Units (ELU) [START_REF] Clevert | Fast and accurate deep network learning by Exponential Linear Units (ELUs)[END_REF] and 3 × 3 Convolution (Conv). Dropout [START_REF] Clevert | Fast and accurate deep network learning by Exponential Linear Units (ELUs)[END_REF] with a rate of 0.2 is used after each Conv. to prevent overfitting. The proposed networks can be trained in an end-to-end manner by gradient descent using Adam update rule [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. During training, we minimize a cross-entropy loss function between the true action label y and the predicted action ŷ over the training samples X , by solving the following optimization problem Arg min

W (L X (y, ŷ)) = Arg min W   - 1 M M i=1 C j=1 y ij log ŷij   , (4)
where W is the set of weights that will be learned by the model, M denotes the number of samples in training set X and C is the number of action classes.

Experiments

The proposed method is first evaluated on two challenging datasets: the MSR Action3D and NTU RGB+D (Sect. 4.1). We then introduce the CEMEST dataset2 and report experimental results on this dataset. The implementation details of the proposed D-CNNs are also provided in this section (Sect. 4.2).

Datasets and Settings

MSR Action3D Dataset [START_REF] Li | Action recognition based on a bag of 3D points[END_REF]: This dataset contains 20 actions performed by 10 subjects. Each skeleton is composed of 20 key joints. Experiments were conducted on 557 action sequences. We follow the protocol proposed by [START_REF] Li | Action recognition based on a bag of 3D points[END_REF]. Specifically, the whole dataset is divided into three subsets: AS1, AS2 and AS3.

For each subset, five subjects are selected for training and the rest are used for testing (see Supplemental Material). Data augmentation techniques including random cropping, vertically flipping, and Gaussian filtering have been applied on this dataset. real-world dataset. The CEMEST was made at a metro station in France without any control of the passenger behavior as well as illumination. It contains three actions including both normal and abnormal behaviors: crossing normally over the barriers, jumping over the barriers, and sneaking under the barriers. These three behaviors are taken into account for acquisition because they have a significant impact on monitoring and management in public transport. As an example, the French National Railway Company (SNCF) reported that they lost e500 million every year through people trying to cheat the ticket system [START_REF]The Local: SNCF increases fines for ticket dodgers[END_REF]. In summary, this dataset provides RGB, depth and skeletal data. The skeleton sequences are extracted by Kinect SDK with 25 key joints for each subject, at a frame rate of 30 FPS. All recorded sequences are manually segmented and labeled. Figure 5 shows some samples from the CEMEST. We carried out two experimental evaluations on this dataset. In the first setting, we randomly chose 67% of the data as training set and the remaining 33% is used for testing. In the second setting, the proposed networks are trained on a combination dataset, which is created from a portion of the MSR Action3D [START_REF] Li | Action recognition based on a bag of 3D points[END_REF] and NTU RGB+D [START_REF] Shahroudy | NTU RGB+D: A large scale dataset for 3D human activity analysis[END_REF] datasets (see Supplementary Material for more details). To ensure the number of samples in each action class is balanced, we augmented samples in the MSR Action3D to match the size of the larger dataset. The pre-trained model is then deployed on the CEMEST dataset in the hope that transfer learning will help to solve overfitting problem when training on small dataset. In both experiments, data augmentation (i.e. cropping, flipping, Gaussian filtering) has been used.

NTU RGB+D Dataset

Implementation Details

The Enhanced-SPMFs are computed directly from skeletons without using a fixed number of frames. The proposed DenseNets were implemented in Python using Keras. For training, we use mini-batches of 64 images. The weights are initialized as [START_REF] He | Delving deep into rectifiers: Surpassing humanlevel performance on ImageNet classification[END_REF]. Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] is used with an initial learning rate η = 3e-4. All networks are trained for 250 epochs from scratch.

Experimental Results and Evaluation

Experimental results and comparison of the proposed method with existing approaches on the MSR Action3D dataset are summarized in Table 1. The DenseNet-40 achieves an average accuracy of 99.10% over three subsets, which outperformed previous approaches by [START_REF] Chen | Real-time human action recognition based on depth motion maps[END_REF][START_REF] Du | Hierarchical recurrent neural network for skeleton based action recognition[END_REF][START_REF] Li | Action recognition based on a bag of 3D points[END_REF][START_REF] Vemulapalli | Human action recognition by representing 3D skeletons as points in a lie group[END_REF][START_REF] Wang | Graph based skeleton motion representation and similarity measurement for action recognition[END_REF][START_REF] Weng | Spatio-temporal Naive-Bayes Nearest-Neighbor (ST-NBNN) for skeleton-based action recognition[END_REF][START_REF] Xu | Spatio-temporal pyramid model based on depth maps for action recognition[END_REF] and surpassed our previous work on SPMF [START_REF] Pham | Skeletal movement to color map: A novel representation for 3D action recognition with Inception Residual networks[END_REF]. Figure 6 (left) shows an example of the learning curves of the network during training on this dataset.

For the NTU RGB+D dataset, as shown in Table 2, the proposed DenseNet-40 achieves an accuracy of 79.95% on the Cross-Subject and 87.52% on Cross-View evaluations, respectively. These results demonstrate the effectiveness of the proposed representation and deep learning framework since they surpassed previous state-of-the-art approaches reported in [START_REF] Du | Hierarchical recurrent neural network for skeleton based action recognition[END_REF][START_REF] Hu | Jointly learning heterogeneous features for RGB-D activity recognition[END_REF][START_REF] Liu | Spatio-temporal LSTM with trust gates for 3D human action recognition[END_REF][START_REF] Liu | Skeleton-based human action recognition with global context-aware attention LSTM networks[END_REF][START_REF] Liu | Enhanced skeleton visualization for view invariant human action recognition[END_REF][START_REF] Shahroudy | NTU RGB+D: A large scale dataset for 3D human activity analysis[END_REF][START_REF] Vemulapalli | Human action recognition by representing 3D skeletons as points in a lie group[END_REF][START_REF] Wang | Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks[END_REF][START_REF] Zhang | On geometric features for skeleton-based action recognition using multilayer LSTM networks[END_REF] as well as a higher level of performance than SPMF [START_REF] Pham | Skeletal movement to color map: A novel representation for 3D action recognition with Inception Residual networks[END_REF]. The experimental results show that the proposed method reached an accuracy of 95.70%, increasing the performance by nearly 5% compared to the first experiment. This could be explained by the fact that since the CEMEST dataset is quite small, it benefits from the knowledge transfer coming from larger datasets such as the MSR Action3D and NTU RGB+D datasets. This result indicates that the use of data augmentation and transfer learning is crucial to address the small amount of samples in real-world datasets. Figure 7 shows learning curves of the proposed deep learning networks on the CEMEST dataset from scratch (Fig. 7a-c), pre-training on the combined dataset (Fig. 7d-f) and fine-tuning on CEMEST dataset (Fig. 7g-i).

An Ablation Study on Enhanced-SPMF

We believe that the use of the smoothing filter and the AHE algorithm [START_REF] Pizer | Adaptive histogram equalization and its variations[END_REF] helps the proposed representation to be more discriminative, which improves recognition accuracy. To verify this hypothesis, we carried out an ablation study on the proposed representation by removing the color enhancement module and seeing how that affects performance. We observed that this kind of transformation is needed for improving learning performance of deep neural networks. Specifically, we trained the proposed DenseNet-40 on both the SPMFs and Enhanced-SPMFs provided by MSR Action3D dataset [START_REF] Li | Action recognition based on a bag of 3D points[END_REF]. During training, the same hyper-parameters and training methodology were applied. The experimental results indicate that the proposed deep network achieves better recognition accuracy when trained on the Enhanced-SPMF (+1.42%). This result validates our hypothesis above.

Computational Efficiency Evaluation

The proposed learning framework comprises three main stages: (1) the computation of Enhanced-SPMF; (2) the training stage; and (3) the inference stage. To evaluate the computational efficiency of this method, we measure the execution time of each stage on the AS1 subset/MSR Action3D dataset with the proposed DenseNet-40 network, which only has 1.0M parameters. With the implementation in Python using Keras and training on a single GTX Ti 1080 GPU, the training process takes less than one hour to reach convergence. While the inference stage, including the stage (1) that is executed on a CPU and the stage (3), takes an average of 0.175 s per sequence without parallel processing. This result verifies the appropriateness of our method in terms of computational cost. Additionally, the computation of the Enhanced-SPMF can be implemented on a GPU for real-time applications.

Conclusions

We introduced a deep learning framework for 3D action recognition from skeletal data. A new motion representation that captures the spatio-temporal patterns of skeleton movements and encodes them into color images has been proposed. Densely connected networks have been designed to learn and recognize actions from the proposed representation in an end-to-end manner. Experiments on two public datasets have demonstrated the effectiveness of our method, both in terms of accuracy as well as computational time. We also introduced CEMEST, a new real-wold surveillance dataset containing both normal and anomalous events for studying human behaviors in public transport. Experimental results on this dataset show that the proposed deep learning based approach achieved promising results. We are currently expanding this study by adding more visual evidence to the network in order to further gains in performance. A new approach for 3D pose estimation will also be studying to replace depth sensors. The preliminary results are encouraging.

Fig. 2 .

 2 Fig. 2. Some SPMF representations obtained from the MSR Action3D dataset [20]. The change in color reflects the change of distance and orientation between the joints.

Fig. 4 .

 4 Fig. 4. A DenseNet block (left). The symbols + + denotes the concatenation operator. We explore and optimize this architecture for learning and recognizing human actions on the proposed image-based representation (right).

Fig. 5 .

 5 Fig. 5. Some samples from the CEMEST dataset: (a), (b) crossing over the barriers, (c), (d) jumping over the ticket barriers and (e), (f) sneaking under ticket barriers.

Figure 6 (

 6 right) shows the training loss and test accuracy of the proposed DenseNet-40 on the NTU RGB+D

Fig. 6 .

 6 Fig. 6. Learning curves of the proposed DenseNet-40 network on the Enhanced-SPMFs obtained from the MSR Action3D and NTU RGB+D

Fig. 7 .

 7 Fig. 7. Learning curves of the three proposed deep networks (DenseNet-16, DenseNet-28, DenseNet-40) on CEMEST dataset when trained from scratch (a)-(b)-(c) and finetuned on CEMEST dataset (d)-(e)-(f). Our best configuration (DenseNet-40) achieved an accuracy of 91.18% when trained on the CEMEST dataset from scratch. With the support of transfer learning, the proposed method reached an accuracy of 95.70%, increasing the recognition accuracy by nearly 5%.

Table 1 .

 1 Experimental results and comparison with the state-the-art approaches on the MSR Action3D dataset[START_REF] Li | Action recognition based on a bag of 3D points[END_REF]. The best accuracies are in italics. Results that surpass previous works are in bold.Enhanced-SPMF DenseNet-40 (ours) 98.88% 99.05% 99.24% 99.10 %

	Method (protocol of [20])	AS1	AS2	AS3	Aver.
	Bag of 3D Points [20]	72.90% 71.90% 71.90% 74.70%
	Depth Motion Maps [2]	96.20% 83.20% 92.00% 90.47%
	Bi-LSTM [36]	92.72% 84.93% 97.89% 91.84%
	Lie Group [39]	95.29% 83.87% 98.22% 92.46%
	Hierarchical RNN [6]	99.33% 94.64% 95.50% 94.49%
	Graph-Based Motion [42]	93.60% 95.50% 95.10% 94.80%
	ST-NBNN [44]	91.50% 95.60% 97.30% 94.80%
	ST-NBMIM [45]	92.50% 95.60% 98.20% 95.30%
	S-T Pyramid [48]	99.10% 92.90% 96.40% 96.10%
	SPMF [27]	97.54% 98.73% 99.41% 98.56%
	Enhanced-SPMF DenseNet-16 (ours) 98.05% 98.38% 98.80% 98.41%
	Enhanced-SPMF DenseNet-28 (ours) 98.44% 98.47% 99.18% 98.70%

Table 2 .

 2 Experimental results and comparison with the state-the-art approaches on the NTU RGB+D dataset[START_REF] Shahroudy | NTU RGB+D: A large scale dataset for 3D human activity analysis[END_REF]. The best accuracies are in italics. Results that surpass previous works are in bold.

	Method (protocol of [33])	Cross-Subject Cross-View
	Lie Group [39]	50.10%	52.80%
	Hierarchical RNN [6]	59.07%	63.97%
	Dynamic Skeletons [13]	60.20%	65.20%
	Two-Layer P-LSTM [33]	62.93%	70.27%
	ST-LSTM Trust Gates [21]	69.20%	77.70%
	Geometric Features [50]	70.26%	82.39%
	Two-Stream RNN [40]	71.30%	79.50%
	Enhanced Skeleton [24]	75.97%	82.56%
	GCA-LSTM [22]	76.10%	84.00%
	SPMF [27]	78.89%	86.15%
	Enhanced-SPMF DenseNet-16 (ours) 77.89%	86.55%
	Enhanced-SPMF DenseNet-28 (ours) 79.07%	86.82%
	Enhanced-SPMF DenseNet-40 (ours) 79.95%	87.52%

Codes and models are available on our GitHub project at https://bit.ly/2EC9vj9.

Created by Cerema and Tisséo public transport in France and available for research purposes from https://bit.ly/2SNbrdE.

Acknowledgements. This research was supported by the Cerema, France. Sergio A. Velastin is grateful for funding from the Universidad Carlos III de Madrid, the EU's 7th Framework Programme for Research, Technological Development and demonstration (grant 600371), Ministerio de Economia, Industria y Competitividad (COFUND2013-51509), Ministerio de Educación, cultura y Deporte (CEI-15-17) and Banco Santander.


[𝑚𝑚]

• quantitative agreement with Tore Supra 𝜆 𝑞 (ITER database)

• quantitative agreement with multi-machine prediction of ITER startup SOL width 𝜏 𝐸 ∝ 𝐼 𝑃 0.2 𝐵 0.6 𝑃 𝑇𝑂𝑇 -0.6 𝑛 𝑒 0.6 𝑀 0.2 𝑅 2.8 𝑎 𝑅 1.9

Scaling law ITER 98:

𝜏 𝐸 ∝ 𝐼 𝑃 1.0 𝐵 0.1 𝑃 𝑇𝑂𝑇 -0.6 𝑛 𝑒 0.4 𝑀 0.2 𝑅 1.9 𝑎 𝑅

••

  >100 probe reciprocations with turbulence (1MHz 𝜃-rake) & profile (tunnel) collectors -ExB velocities of averaged filaments match with isolated filament model -fluctuation amplitudes (𝑛 𝑒 & Φ) and main 𝑘 𝜃 well reproduced -density profile decay length (𝜆 𝑛 ) quantitatively matched model of interchange turbulent transport: on the correlation between scrape off layer width and core confinement in tokamaks N. Fedorczak, Mathieu Peret, Philippe Ghendrih, Patrick Tamain, Hugo Bufferand CEA, IRFM, F-13108 Saint-Paul-Lez-Durance, France  Spectral filament paradigm: self consistent prediction of turbulent spectra (𝑛 𝑒 , Φ) of flux driven 2D isothermal interchange turbulence  Predicts macroscopic transport properties (flux, fluctuations) in scrape-off layer ( SOL width 𝜆 𝑞 ) but also in the core ( confinement time 𝜏 𝐸 )  SOL transport: quantitative agreement with Tore Supra data (circular plasmas), recovers impact of divertor leg length on SOL width (𝜆 𝑞 ) in TCV, recovers multi-machine parametric sensitivity of 𝜆 𝑞  Core confinement: recovers multi-machine parametric sensitivity of 𝜏 𝐸 and correlation between core confinement and heat flux width  Offers a flexible paradigm for addressing the optimization of power exhaust versus core confinement (key: impact of geometry in model) 2D isothermal interchange model (TOKAM2D) . Krasheninnikov et al. J. Plasma Phys. 2008, 74, 679 Density filament: size 𝛿 𝑓 & normalized amplitude 𝑛 𝑓  drift velocity (𝑣 𝑓 ) predicted from vorticity balance Hyp: well shaped filaments, not the case in simulations Solution: construction of averaged filaments  conditional average with event amplitude discrimination Isolated filament model works on TOKAM2D averaged filaments Spectral filament (SF) model N. Fedorczak et al. Contrib. Plasma Phys. 2018;58:471-477 N. Fedorczak et al. Nuc. Mater. & Energy 19 (2019) 433-439 Principle: extend filament model to fluctuation spectra 𝑣 𝑓 𝛿 𝑓 → Φ 𝑘 𝜃 & 𝑛 𝑏 → 𝑛 𝑘 𝜃 Hyp: density & potential spectra in phase quadrature + time independent spectra (it is!) • potential: include effective viscosity from dominant mode 𝑘 4 Φ 𝑘 0 Φ 𝑘 + 𝜎 ∥ Φ 𝑘 = 𝑔𝑘 𝑛 𝑘 • density : mixing length + ad-hoc saturation at low k k Spectral integrals for macroscopic quantities: • radial flux: Γ 𝑟 ≡ 8𝜋 2 ∫ 𝑘Φ 𝑘 𝑛 𝑘 𝑑𝑘 = 41 𝜆 𝑛 scrape-off-layer width: Γ 𝑟 ≡ 𝜆 𝑛 𝜎 ∥  𝜆 𝑛 = 3.9 𝑔 3not diffusive nor convective • intermittency disappears in spectral paradigm • key : model of 𝑔 & 𝜎 ∥ Recovers multi-machine scaling of ITER startup heat flux width Recovers multi-machine scaling sensitivity of 𝜆 𝑞 in H-mode SF model  prediction of heat flux width & confinement time Heat flux decay length Hyp: 𝜆 𝑞 = 𝛼 𝑞 𝜆 𝑛 , 𝑔 = 𝜌 𝐿 /𝑅 & 𝜎 ∥ = 𝜌 𝐿 𝐿 ∥ = 𝜌 𝐿 /𝜋𝑞 𝑐𝑦𝑙 𝑅 , 2-point model for 𝑇 𝑠𝑒𝑝 𝜆 𝑞 = 50𝛼 𝑞 𝑞 𝑐𝑦𝑙 0.63 𝐵 𝑇 -0.65 𝑅 𝑚 0 pressure & particle flux  heat flux vorticity // dissipation still valid (HFS decorrelation of LFS mode) gradient scale length 𝜆 ≈ 𝑎 𝜏 𝐸 = 0.007𝛼 𝑇 𝐼 𝑃 𝑀𝐴 0Hyp: 𝜆 𝑞 = 0.5𝜆 𝑛 (correlation from Tore Supra database)

J

  Gunn et al. J. Nucl. Mater. 438 (Suppl) (2013) S184-S188, N. Fedorczak et al. Nuc. Mater. & Energy 19 (2019) 433-439 J. Horacek et al. Plasma Phys. Controlled Fusion 58 (7) (2016) 074005. Model: 𝜆 𝑞 ∝ 𝑅 0.25 … • 𝜆 𝑞 X3-4 for ITER & DEMO compared to 𝐵 𝜃 scaling • sounds with XGC1 results (different interpretation) T. Eich et al. Nucl. Fusion 53 (9) (2013) 093031 C.S. Chang et al 2017 Nucl. Fusion 57 116023 Recovers impact of divertor leg length on 𝜆 𝑞 in TCV L-mode TCV ohmic pulses: scan of plasma current and divertor geometry gives same 𝜆 𝑞 (𝐿 ∥ ) Model: 𝜆 𝑞 ∝ 𝐿 ∥ 0.63 • sensitivity recovered by the SF model • 3D turbulent simulations show interchange along divertor leg, as the model considers R. Maurizio et al. Nucl. Fusion 58 (2018) 016052 A. Gallo et al. Plasma Phys. Control. Fusion 60 (2018) 014007 Recovers global multi-machine 𝜏 𝐸 sensitivity & correlation with 𝜆 𝑞 Model :

•

  significant difference in 𝐼 𝑃 & 𝐵 • global trend is well matched hints of correlations in exp. scaling Correlation confinement vs 𝜆 𝑞 • experimental correlation 𝜆 𝑞 ∝ ҧ 𝑝 -0.48 across L-I-H modes in C-mod • recovered in exp. H-mode scaling • recovered by model on current machines, but not for ITER/DEMO D. Brunner et al. Nucl. Fusion 58 (2018) 094002