Amor Ben Tanfous

Alice Mme

Rapporteur Caplier

Pr

M Sylvain

Dr Calinon Rapporteur

M Boulbaba

Ben Amor

M Hassen Drira

Examinateur M Josef Kittler

Bernadette Mme

Président Dorizzi

Sparse Representations in the Shape Manifold for Human Trajectories Classification and Generation COMPOSITION DU JURY

come   L'archive ouverte pluridisciplinaire

solution, we exploit notions of Riemannian geometry to encode these trajectories based on sparse coding and dictionary learning. This not only overcomes the problem of nonlinearity of the manifold but also yields sparse representations that lie in vector space, that are more discriminative and less noisy than the original data. We study intrinsic and extrinsic paradigms of sparse coding and dictionary learning in the shape manifold and provide a comprehensive evaluation on their use according to the nature of the data (i.e. face or body in 2D or 3D). Based on these sparse representations, we present two frameworks for 3D human action recognition and 2D micro-and macro-facial expression recognition and show that they achieve competitive performance in comparison to the state-of-the-art. Finally, we design a generative model allowing to synthesize human actions. The main idea is to train a generative adversarial network to generate new sparse representations that are then transformed to pose sequences. This framework is applied to the task of data augmentation allowing to improve the classification performance. In addition, the generated pose sequences are used to guide a second framework to generate human videos by means of pose transfer of each pose to a texture image. We show that the obtained videos are realistic and have better appearance and motion consistency than a recent state-of-the-art baseline.

-3 -

Résumé

Concevoir des systèmes intelligents pour comprendre le contenu des vidéos est devenu un axe de recherche très important car il permet de compenser les capacités humaines limitées de l'analyse efficace des vidéos. En particulier, la compréhension du comportement humain à partir de vidéos suscite un intérêt considérable en raison de ses nombreuses applications potentielles. Au même temps, la détection et le suivi de marqueurs humains dans les flux vidéo sont devenus de plus en plus fiables, et c'est en partie grâce à la disponibilité de capteurs abordables. Cela permet de déduire des données géométriques qui varient dans le temps et qui jouent un rôle important dans l'analyse automatique du mouvement humain. Cependant, une telle analyse reste difficile en raison des énormes variations de vue, de la détection inexacte des marqueurs, des variations importantes des classes ainsi que de l'insuffisance des données annotées. Dans cette thèse, nous proposons de nouvelles méthodes permettant de classifier et de générer des séquences 2D/3D de marqueurs humains. Tout d'abord, nous représentons ces séquences comme étant des trajectoires dans des variétés de formes permettant ainsi une analyse invariante à la vue. Toutefois, ces variétés sont non linéaires et, par conséquent, les algorithmes classiques telles que les techniques d'apprentissage automatique standards ne pourraient pas être appliqués d'une manière directe vu qu'ils sont conçus pour des données de nature linéaire. En guise de solution, nous exploitons des notions de la géométrie Riemannienne pour coder ces trajectoires en appliquant une technique de codage parcimonieux et d'apprentissage de dictionnaires. Cela permet non seulement de résoudre le problème de non-linéarité des variétés de formes mais aussi de représenter les trajectoires comme étant des représentations parcimonieuses qui sont définies dans des espaces vectoriels, qui sont plus discriminantes et moins bruitées que les données originales. Nous étudions des paradigmes intrinsèques et extrinsèques de codage parcimonieux et d'apprentissage de dictionnaire dans les variétés de formes et nous présentons une étude comparative de leur utilisation en fonction de la nature des données -4 -(i.e. visage ou corps en 2D ou 3D). D'autre part, en se basant sur ces représentations parcimonieuses, nous proposons deux approches de reconnaissance d'actions humaines en 3D et de reconnaissance d'expressions faciales en 2D, et nous montrons que les résultats obtenus sont compétitifs avec les méthodes récentes de l'état de l'art. Enfin, nous présentons un modèle génératif permettant de synthétiser des actions humaines dont l'idée principale est de concevoir un réseau antagoniste génératif afin de générer de nouvelles représentations parcimonieuses qui sont ensuite transformées en des séquences de poses. Nous appliquons cette méthode pour l'augmentation des données ce qui permet d'améliorer les performances de la classification d'actions. De plus, les séquences de pose générées sont utilisées pour guider un second modèle générateur dans le but de générer des vidéos humaines par transfert de chaque pose d'une séquence en une image texture. Nous montrons que les vidéos obtenues sont réalistes et présentent une meilleure cohérence en termes d'apparence et de mouvement qu'une méthode récente de l'état de l'art.

-5 -

Table of contents

Exponential map and logarithm map operations on a Riemannian manifold.

The red curve represents the geodesic connecting the points Z and Z 1 . The green arrow represents the tangent vector V on the tangent space T Z (M). . .

List of tables

3.1 Overall recognition accuracy (%) on MSR-Action 3D, Florence Action 3D, and UTKinect 3D datasets. In the first column: (R) : Riemannian approaches;

(N ) : other recent approaches; Last row: our approach. . . . . . . . . . . . . .

3.2

Overall recognition accuracy (%) on NTU-RGB+D following the X-sub and X-view protocols. In the first column: (R) : Riemannian approaches;

(RN ) : RNN-based approaches; -Visual Surveillance Video surveillance cameras can be found everywhere (see On the other hand, using landmark data is expected to reduce the computational complexity of the designed algorithms as instead of treating high-dimensional RGB images that may contain irrelevant information for the desired tasks, landmark data summarize -23 - occlusion. More accurate estimation can be yielded with Motion Capture systems based on body markers for an automatic tracking of 3D skeletons, but this solution is more expensive in cost and time than depth cameras. Recently, pose estimation approaches [START_REF] Andriluka | 2d human pose estimation: New benchmark and state of the art analysis[END_REF][START_REF] Cao | Realtime multi-person 2d pose estimation using part affinity fields[END_REF][START_REF] Bogo | Keep it smpl: Automatic estimation of 3d human pose and shape from a single image[END_REF] have reached impressive performances inferring 2D/3D skeletons that are in many cases robust to occlusion and body profile. On the other hand, facial landmarks are located in different positions of the face such as the mouth, nose, eyes, eyebrows and also the chin in some methods. The latter can be detected from texture images in 2D [START_REF] Asthana | Incremental face alignment in the wild[END_REF] or 3D [START_REF] Baltrusaitis | Openface 2.0: Facial behavior analysis toolkit[END_REF].

Scientific challenges

Human action and facial expression analysis (i.e. recognition and generation) are important but challenging research topics due to several major issues [START_REF] Kong | Human action recognition and prediction: A survey[END_REF][START_REF] Samadiani | A review on automatic facial expression recognition systems assisted by multimodal sensor data[END_REF].

-25 -Intra-and inter-class variations It is well known that people perform the same action in various styles and express their emotions differently through facial expressions. For a given action such as "hand wave", one can use his left or right hand, can rise it slowly or quickly, or even turns while waving. On the other hand, a person can smile with different intensities, can simultaneously close his eyes or move his head. That is to say, one action or facial expression category may contain various styles of motion which results in large intraclass pose variations. Moreover, different categories can present strong similarities. As an example, actions "walking" and "running" involve similar motion patterns. These similarities would result in small inter-class variations which makes the analysis more challenging. As a consequence, these issues should be carefully addressed in the analysis in order to make human motion representations more discriminative.

View-variations Pose sequences belonging to the same categories can be captured from various viewpoints, see Figure 1.6. They can be taken in front of the subjects, on their side, or even on top of them. Similarly, facial expression analysis may encounter several head pose variations making it more complex. In this thesis work, our goal is to go beyond the non-trivial view-variation challenge.

Main contributions

The goal of this dissertation is to analyze dynamic human landmarks of the body and face. We explore sparse representations to model human actions and facial expressions and present new frameworks to classify and generate them. To summarize, we provide the following main contributions:

Action and facial expression representations We introduce sparse representations for landmark-based human actions and facial expressions which are used throughout this thesis work. Tackling the problem of camera view-variations, we represent sequences of landmarks as trajectories in the shape manifold to allow for a view-invariant analysis. Since the resulting manifold is nonlinear, we exploit its well-known Riemannian geometry to study sparse coding and dictionary learning techniques. The application of these approaches to encode trajectories yields representations that lie to vector spaces which are more suitable for the analysis than nonlinear ones. In addition, the obtained time-series are sparse which represents a suitable computational property for several vision problems.

-27 -

Chapitre 1. Introduction

Action and facial expression recognition frameworks We apply the proposed sparse representations to classify 3D actions and 2D facial expressions. We propose two recognition approaches: 1) a deep learning framework in which we apply Long Short-Term Memory (LSTM) to model and classify our sparse representations; and 2) a classical pipeline of methods designed for Euclidean time-series. We perform extensive evaluation on seven commonly-used datasets that present different challenges such as the MSR-Action 3D [START_REF] Li | Action recognition based on a bag of 3D points[END_REF] and the NTU-RGBD [START_REF] Shahroudy | Ntu rgb+d: A large scale dataset for 3d human activity analysis[END_REF] databases for action recognition, and the CK+ [START_REF] Lucey | The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotionspecified expression[END_REF] and the Oulu-CASIA [START_REF] Valstar | Induced disgust, happiness and surprise: an addition to the mmi facial expression database[END_REF] for facial expression recognition. In addition, we evaluate the proposed framework on the more challenging task of micro-expression recognition. We compare our methods with each other and with state-of-the-art approaches where we demonstrate their competitiveness. Moreover, we perform various baseline experiments and ablative studies to evaluate some properties of the presented techniques.

A comparative study of intrinsic and extrinsic coding techniques in the shape manifold In Chapter 2, we study two techniques of Riemannian sparse coding and dictionary learning in the shape manifold known as intrinsic and extrinsic. We use these approaches to model human actions and facial expressions. Based on the classification experiments (Chapter 3), we try to find an answer to the question "Depending on the nature of the data (i.e body or face) and its dimension (i.e 2D or 3D), when and which technique should we apply?". This comparative study is provided in Chapter 3.

A generative framework for human motion synthesis We propose a new framework to generate human pose sequences which is based on the proposed sparse representations.

Specifically, we use a Generative Adversarial Network (GAN) [START_REF] Goodfellow | Generative adversarial nets[END_REF] which we train to generate new sparse representations of human actions. Then, we transform the obtained samples to pose sequences. We conduct extensive experiments to show the effectiveness of adopting a sparse representation in a generative model. In addition, we apply this framework to augment -28 -

Organisation of the dissertation

an action recognition dataset and demonstrate that this improves our classification results.

Furthermore, we use the generated pose sequences to guide the generation of human videos. Specifically, given an input person image, we transfer each pose of a generated sequence to an image to obtain a new video. To this end, we use a recently-proposed person image generation method [START_REF] Zhu | Progressive pose attention transfer for person image generation[END_REF] which we extend in our work to the temporal domain.

Experimentally, we show that our framework yields realistic human videos in terms of motion and content. We also compare our approach with state-of-the-art and demonstrate that it improves the quality of the generated videos.

Organisation of the dissertation

The rest of this dissertation is structured as follows. Chapter 2 begins with formal studies of sparse representations, Riemannian geometry and the Kendall's shape space with various tools that are exploited in this chapter. We also discuss existing solutions to extend sparse coding and dictionary learning to Riemannian manifolds as well as different approaches to represent Riemannian trajectories. Then, we present the human motion modeling frameworks that are used throughout this thesis work. In Chapter 3, we start by providing a brief literature review of 3D action recognition and 2D facial expression recognition approaches. Next, we present the adopted temporal modeling and classification schemes that we apply to our sparse representations. This is followed by an extensive evaluation of the proposed frameworks with a comparison to state-of-the-art. Moreover, we present a deep analysis of the proposed methods and the obtained results which includes a comparison between the adopted intrinsic and extrinsic coding approaches. Chapter 4

begins with a study of generative adversarial networks and a brief literature review of existing human video generation approaches. Then, we describe the proposed pose sequence and video generation frameworks followed by their experimental evaluation with a comparison to stateof-the-art. Finally, Chapter 5 concludes this dissertation, discusses the main limitations of the -29 -proposed methods and presents possible future directions and perspectives of the proposed work.

-30 -

Chapitre 2

Coding Human Motion Trajectories in the Shape Manifold

Introduction

The availability of real-time skeletal data estimation solutions [START_REF] Cao | Realtime multi-person 2d pose estimation using part affinity fields[END_REF][START_REF] Shotton | Real-time human pose recognition in parts from single depth images[END_REF] and reliable facial landmarks detectors [START_REF] Asthana | Incremental face alignment in the wild[END_REF][START_REF] Baltrusaitis | Openface 2.0: Facial behavior analysis toolkit[END_REF][START_REF] Xiong | Supervised descent method and its applications to face alignment[END_REF] has pushed researchers to study shapes of landmark configurations and their dynamics. For instance, 3D skeletons have been widely used to represent human actions due to their ability in summarizing the human motion. Another example is given by the 2D facial landmarks and their tremendous use in facial expression analysis. However, human actions and facial expressions observed from visual sensors are often subject to view variations which makes their analysis complex. Considering this non-trivial problem, an efficient way to analyze these data takes into account viewinvariance properties, giving rise to shape representations often lying to nonlinear shape spaces [START_REF] Ben Amor | Action recognition using rate-invariant analysis of skeletal shape trajectories[END_REF][START_REF] Bryner | 2d affine and projective shape analysis[END_REF][START_REF] Kendall | Shape manifolds, Procrustean metrics, and complex projective spaces[END_REF]. David G. Kendall [START_REF] Kendall | Shape manifolds, Procrustean metrics, and complex projective spaces[END_REF] defines the shape as the geometric information that remains when location, scale, and rotational effects are filtered out from an object.

Accordingly, we represent 2D landmark faces and 3D skeletons as points in the 2D and 3D -31 -

Chapitre 2. Coding Human Motion Trajectories in the Shape Manifold

Kendall's spaces, respectively. Further, when considering the dynamics of these points, the corresponding representations become trajectories in these spaces [START_REF] Ben Amor | Action recognition using rate-invariant analysis of skeletal shape trajectories[END_REF]. However, inferencing such a representation remains challenging due to the nonlinearity of the underlying manifolds. In the literature, two alternatives have been proposed to overcome this problem for different Riemannian manifolds -they are either Intrinsic [START_REF] Ertan | Sparse riemannian manifold clustering for hardi segmentation[END_REF][START_REF] Ertan | Intrinsic mean shift for clustering on stiefel and grassmann manifolds[END_REF][START_REF] Ho | On a nonlinear generalization of sparse coding and dictionary learning[END_REF][START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a lie group[END_REF] or Extrinsic (kernel-based) [START_REF] Harandi | Extrinsic methods for coding and dictionary learning on grassmann manifolds[END_REF][START_REF] Mehrtash | Sparse coding and dictionary learning for symmetric positive definite matrices: A kernel approach[END_REF][START_REF] Jayasumana | A framework for shape analysis via hilbert space embedding[END_REF][START_REF] Li | Log-euclidean kernels for sparse representation and dictionary learning[END_REF]. On one hand, intrinsic solutions tend to project the manifoldvalued data to a tangent space at a reference point [START_REF] Anirudh | Elastic functional coding of human actions: From vector-fields to latent variables[END_REF][START_REF] Ben Amor | Action recognition using rate-invariant analysis of skeletal shape trajectories[END_REF][START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a lie group[END_REF]. While it solves the problem of nonlinearity of the manifold of interest, this solution could introduce distortions, especially when the projected points are far from the reference point. On the other hand, extrinsic solutions are based on embeddings to higher dimensional Reproducing Kernel Hilbert Spaces (RKHS), which are vector spaces where Euclidean geometry applies. These methods bring the advantage that, as evidenced by kernel methods on R n , embedding a lower dimensional space in a higher dimensional one gives a richer representation of the data and helps capturing complex patterns. Nevertheless, to define a valid RKHS, the kernel function must be positive definite according to Mercer's theorem [START_REF] Schölkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF]. Several works in the literature have studied kernels on the 2D Kendall's space. For instance, the Procrustes Gaussian kernel is proposed in [START_REF] Jayasumana | A framework for shape analysis via hilbert space embedding[END_REF] as positive definite. In contrast, to our knowledge, such a kernel has not been explored for the 3D Kendall's space.

Motivated by the success of sparse representations in several recognition tasks [START_REF] Cherian | Riemannian dictionary learning and sparse coding for positive definite matrices[END_REF][START_REF] Harandi | Extrinsic methods for coding and dictionary learning on grassmann manifolds[END_REF][START_REF] Ho | On a nonlinear generalization of sparse coding and dictionary learning[END_REF],

we propose to code shape trajectories using Riemannian sparse coding and dictionary learning (SCDL) in the shape manifold. We will explore both intrinsic and extrinsic paradigms of this technique and provide the main benefits of the resulting representations to model human actions and facial expressions.

In this chapter, we start by presenting some preliminaries and mathematical definitions.

Later on, we review some existing representations of trajectories on Riemannian manifolds along with scientific challenges on the matter. In Section 2.4, we describe the solutions that we adopt to code trajectories in the shape manifold before presenting their properties in the -32 -following section.

Background and definitions

Data encoding techniques have been broadly studied in literature as they play a crucial role in data analysis. In this dissertation, we aim to represent human skeletons and facial landmark configurations with a coding technique that yields sparse representations. This classical approach assumes that the data are defined in Euclidean space. However in our study, we are applying important shape transformations on the data which turn to be elements of a nonlinear space. As a consequence, classical coding approaches could not directly apply to these data and their nonlinear extension is required. As a solution, the Riemannian geometry of the well-known shape manifold can be used, offering several geometric tools which enables the extension of coding to nonlinear spaces. In the following, we start by describing the aforementioned coding technique. Then, we present basic notions of Riemannian geometry with a particular focus on the Kendall's shape space which is our manifold of interest in this dissertation.

Sparse representations

The notion of sparsity as a way to model signals has generated significant interest in the past decade [START_REF] Cherian | Riemannian dictionary learning and sparse coding for positive definite matrices[END_REF][START_REF] Harandi | Extrinsic methods for coding and dictionary learning on grassmann manifolds[END_REF][START_REF] Ho | On a nonlinear generalization of sparse coding and dictionary learning[END_REF]. In particular, sparse representations have proved to be successful in various computer vision and machine learning problems including classification [START_REF] Elad | Image denoising via sparse and redundant representations over learned dictionaries[END_REF],

segmentation [START_REF] Mairal | Discriminative learned dictionaries for local image analysis[END_REF], image denoising and inpainting [START_REF] Shi | A nonconvex relaxation approach to sparse dictionary learning[END_REF], visual tracking [START_REF] Liu | Robust tracking using local sparse appearance model and k-selection[END_REF] and face recognition [START_REF] Zhang | Discriminative k-svd for dictionary learning in face recognition[END_REF], to cite a few. The basic assumption of this model is that, given a dictionary D = {d 1 , d 2 , ..., d N } with N elements called atoms, a query x has a sparse vector representation w ∈ R N with k non-zero elements under the dictionary D: A common approach to estimate the optimal representation w is known as sparse coding which includes a penalty function f (w) in the following optimization problem:

x = Dw, (2.2
l E (x, D) = min w x - N i=1 [w] i d i 2 2 + λf (w), (2.2.2)
where w ∈ R N denotes the vector of codes comprised of {[w] i } N i=1 , f : R N → R is the sparsity inducing function defined as the 1 norm, and λ is the sparsity regularization parameter.

Eq. (2.2.2) seeks to optimally approximate x (by x) as a linear combination of atoms, i.e.,

x = N i=1 [w] i d i , while tacking into account a particular sparsity constraint on the codes, f (w) = w 1 . This sparsity function has the role of forcing x to be represented as only a small number of atoms.

The problem of sparse coding assumes that the dictionary is known. In practice, when the dictionary is learned from the data, significant improvements can be made on the -34 -reconstruction of x. This problem is of wide interest and it is known as the dictionary learning problem. Several techniques have been developed to solve this problem and train dictionaries from data. Popular examples are the Method of Optimal Directions (MOD) [START_REF] Kjersti Engan | Method of optimal directions for frame design[END_REF] and the K-SVD method [START_REF] Aharon | K-svd: An algorithm for designing overcomplete dictionaries for sparse representation[END_REF], which generalizes the K-means clustering algorithm to learn overcomplete dictionaries. Besides, using adaptive dictionaries [START_REF] Rubinstein | Dictionaries for sparse representation modeling[END_REF] is usually the best choice in terms of the reconstruction performance that can be achieved. In addition, it enables to control the amount of induced sparsity over the reconstruction performance.

Formally, given a finite set of t training observations {x 1 , x 2 , ..., x t } in R k , learning adaptive dictionaries [START_REF] Rubinstein | Dictionaries for sparse representation modeling[END_REF] is defined as to jointly minimize the coding cost over all choices of atoms and codes according to:

l E (D) = min D,w t i=1 x i - N j=1 [w i ] j d j 2 2 + λf (w i ). (2.2.3)
To solve this non-convex problem, a common approach alternates between the two sets of variables, D and w, such that: (1) Minimizing over w while D is fixed is a convex problem (i.e., sparse coding). (2) Minimizing Eq. (2.2.3) over D while w is fixed is similarly a convex problem.

In our work, we are interested in the extension of SCDL to the Kendall's shape space.

Before introducing this manifold, in the following section, we present basic notions on Riemannian geometry.

Riemannian geometry

A manifold M is a Hausdorff topological space which locally resembles a Euclidean space R n , where n is the dimension of the manifold. The tangent space at a point on the manifold provides us with a vector space of tangent vectors that give an idea of direction on the manifold. A Riemannian manifold is differential and equipped with a metric on the tangent spaces. A Riemannian metric allows us to compute angle and length of directions (tangent -35 -vectors). A Riemannian metric is a continuous collection of inner products on the tangent space at each point of the manifold. It is usually chosen to provide robustness to some geometrical transformations. Furthermore, it makes it possible to define several geometric notions on a Riemannian manifold such as the geodesic distance between points on the manifold. In fact, points on a Riemannian manifold are connected with smooth curves.

Assuming the Riemannian metric, one can compute the length of a given curve. The curves yielding the minimum distance for any two points of the manifold are called geodesics which are analogous to straight lines in R n . The length of a geodesic defines the geodesic distance.

The geodesic distance induced by the Riemannian metric is the most natural measure of dissimilarity between two points lying on a Riemannian manifold. However, in practice, many other nonlinear distances which do not necessarily arise from Riemannian metrics can also be useful for measuring dissimilarity on manifolds [START_REF] Jayasumana | Kernel methods on riemannian manifolds with gaussian rbf kernels[END_REF]. Two other essential operations on Riemannian manifolds are the logarithm map (log) and exponential map (exp). As illustrated in Figure 2.2, the exponential map exp Z (.) : T Z M → M projects a tangent vector from the tangent space at a point Z to the manifold. It guarantees that the length of the tangent vector is equal to the geodesic distance. The logarithm map log Z (.) = exp -1 (.) : M → T Z M projects a point on the manifold to the tangent space T Z M at another point. The red curve represents the geodesic connecting the points Z and Z 1 . The green arrow represents the tangent vector V on the tangent space T Z (M).

Background and definitions

Several Riemannian manifolds have been widely studied in the computer vision literature.

Examples include the Lie group SO(3) formed by 3D rotation matrices, the unit n-sphere S n formed by normalized histograms, the manifold of symmetric positive definite (SPD) matrices, the Grassmann manifold defined as linear subspaces of R n , and the Kendall's shape space (also known as the shape manifold). For an overview on Riemannian geometry and manifolds, we refer the reader to useful resources on the topic [START_REF] William | An introduction to differentiable manifolds and Riemannian geometry[END_REF][START_REF] Yui | Advances in matrix manifolds for computer vision[END_REF]. In the following section, we will outline the geometric properties of the manifold considered in this work, the Kendall's shape space.

Kendall's shape space

Let us consider a set of n landmarks in R m (m = 2, 3). To represent its shape, Kendall [START_REF] Kendall | Shape manifolds, Procrustean metrics, and complex projective spaces[END_REF] proposed to establish equivalences with respect to shape-preserving transformations that are translations, rotations, and global scaling. Let Z ∈ R n×m represent a configuration of landmarks. To remove the translation variability, we follow [START_REF] Dryden | Statistical shape analysis[END_REF] and introduce the notion of Helmert sub-matrix, a (n-1)×n sub-matrix of a commonly used Helmert matrix, to perform centering of configurations. For any Z ∈ R n×m , the product HZ ∈ R (n-1)×m represents the Euclidean coordinates of the centered configuration. Let C 0 be the set of all such centered

configurations of n landmarks in R m , i.e., C 0 = {HZ ∈ R (n-1)×m |Z ∈ R n×m }. C 0 is a m(n -1)
dimensional vector space and can be identified with R m(n-1) . To remove the scale variability, we define the pre-shape space to be:

C = {Z ∈ C 0 | Z F = 1}; C is a unit sphere in R m(n-1)
and, thus, is m(n -1) -1 dimensional. The tangent space at any pre-shape Z is given by: T is called the shape space of configurations. The tangent space at any shape Z is

Z (C) = {V ∈ C 0 |trace(V T Z) = 0}.
T Z (S) = {V ∈ C 0 |trace(V T Z) = 0, trace(V T ZU ) = 0}
, where U is any m × m skew-symmetric matrix. The first condition makes V tangent to C and the second makes V perpendicular to the rotation orbit. Together, they force V to be tangent to the shape space S. Assuming standard Riemannian metric on S, the geodesic between two points Z1 , Z2 ∈ S is defined as:

α(t) = 1 sin(θ) (sin((1 -t)θ)Z 1 + sin(tθ)Z 2 O * ), (2.2.4)
where

θ = cos -1 ( Z 1 , Z 2 O * ), .
, . is the inner product on S, and O * is the optimal rotation that aligns

Z 2 with Z 1 : O * = argmin O∈SO(m) Z 1 -Z 2 O 2 F .
θ is also the geodesic distance between Z1 and Z2 in the shape space S, representing the optimal deformation to connect Z1 to Z2 in S. For t = 0, α(0) = Z1 and for t = 1 we have α(1) = Z2 . Note that Kendall's shape space is a complete Riemannian manifold such that the logarithm map operator log Z is defined for all Z ∈ S (see Section 2.2.3.1 for its definition). As a consequence, the geodesic distance between two configurations Z1 and Z2 can be computed as d S ( Z1 , Z2 ) = log Z1 ( Z2 ) Z1 , where . Z1 denotes the norm induced by the Riemannian metric at T Z1 (S).

The case of planar shapes For m = 2, a 2D landmark configuration can be initially represented as a n-dimensional complex vector whose real and imaginary parts respectively encode the x and y coordinates of the landmarks. In this case, the pre-shape space is defined, after removing the translation and scale effects, as:

C = {z ∈ C n-1 | z = 1}; C is a complex unit sphere of dimension 2(n -1) -1.
The rotation removal consists of defining, for any

z ∈ C n-1 , an equivalence class z = {zO|O ∈ SO(2)
} that represents all rotations of a configuration z. The final shape space S is the set of all such equivalence classes S = {z|z ∈ C} = C/SO(2). To measure the distance between two shapes z1 and z2 , we define the most popular distance on the 2D Kendall's shape space, named the full Procrustes Distance [START_REF] Kendall | Shape manifolds, Procrustean metrics, and complex projective spaces[END_REF],

as

d F P ( z1 , z2 ) = (1 -| z 1 , z 2 | 2 ) 1/2 , (2.2.5)
where •, • and |.| denote the inner product in S and the absolute value of a complex number, respectively.

-38 -

Background and definitions

Geometric tools on the manifold

Considering the spherical structure of C, analytic expressions of the logarithm and exponential maps are well defined [START_REF] Dryden | Statistical shape analysis[END_REF][START_REF] Kendall | Shape manifolds, Procrustean metrics, and complex projective spaces[END_REF] and can be easily adapted to S. These operations allow to compensate the lack of vector structure in the shape manifold by working on tangent spaces. In this section, we define these operators, then use them to define a useful algorithm allowing to compute the mean shape of manifold-valued points, namely intrinsic mean.

Exponential map allows projection from a tangent space to the manifold. It applies the shooting vector to a source shape and provides the deformed (target) shape. It is defined,

for any V ∈ T Z (S), by, exp Z (V ) = cos(θ)Z + sin(θ) θ V . (2.2.6)
Logarithm map allows to map a point on the manifold to the tangent space at another point. It represents the shooting vector at the first shape (source) to the second shape (target). Its mathematical expression is given explicitly by,

log Z1 ( Z2 ) = θ sin(θ) (Z 2 O * -cos(θ)Z 1 ), (2.2.7)
for source shape Z1 and target Z2 , with θ as above.

Intrinsic mean An important advantage of the Riemannian approach is the ability to compute statistics on a set of manifold-valued points. One can use the notion of Karcher mean [START_REF] Karcher | Riemannian center of mass and mollifier smoothing[END_REF] to define an average shape. This represents an intrinsic mean and can be used as representative of a group of points on the manifold. Let { Z1 , ..., Zk } be a set of points on S.

We define an objective function

Ψ : S → R, Ψ( Z) = k i=1 d S ( Zi , Z) 2 .
The intrinsic mean is obtained by minimizing this objective function, which is commonly solved using a standard algorithm that we describe in Algorithm 1.

As discussed previously, mappings to a tangent space allow to compensate the lack of

Algorithm 1: Computing intrinsic mean on S input : A set of shapes Z = { Zj } m j=1 and 1 , 2 small Initialize μ ← Z 1 , i ← 0 repeat Compute v j ← log μi ( Zj ) Compute average tangent vector v ← 1 k m j=1 v j Update μi according to μi+1 ← exp μi ( 2 v) i ← i + 1 until |v| < 1 return μ, a sample Mean of Z = { Zj } m j=1
vector structure on the shape manifold. We refer to this solution as an intrinsic approach.

Another solution to seek a vector representation of the data is known as an extrinsic approach which embeds the manifold-valued data to a higher-dimensional vector space namely the Hilbert space.

Hilbert space embedding of the manifold

A Hilbert space H is an (often infinite-dimensional) inner product space which is complete with respect to the norm induced by the inner product. A Reproducing Kernel Hilbert Space (RKHS) is a special kind of Hilbert space of functions on some nonempty set S in which all evaluation functionals are bounded and hence continuous. The inner product of an RKHS of functions on S can be defined by a bivariate function on S × S, known as the reproducing kernel of the RKHS [START_REF] Jayasumana | Kernel methods on riemannian manifolds with gaussian rbf kernels[END_REF].

The SCDL algorithms depend only on the notion of inner product, which allows us to measure distances. Therefore, they can be easily extended to Hilbert spaces. The embedding of the shape manifold to RKHS brings the main advantage of transforming the nonlinear manifold into a vector space where one can directly apply standard (Euclidean) algorithms.

In addition, it gives a richer representation of the data in a higher-dimensional space which helps identifying complex patterns. To define an inner product in H, one can use a kernel function f : (S × S) → R which makes the resulting space a RKHS without the need of -40 -computing the actual vectors. This procedure, known as the kernel trick, is commonly used in machine learning when the designed algorithm only relies on measures of similarities, i.e., on inner products. However, to define a valid RKHS, the kernel function must be positive definite according to Mercer's theorem [START_REF] Schölkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF]. In particular, for the 2D shape manifold, the authors in [START_REF] Jayasumana | A framework for shape analysis via hilbert space embedding[END_REF] proved the positive definiteness of the Procrustes Gaussian kernel k P : (S × S) → R which is defined as

k P ( z1 , z2 ) := exp(-d F P 2 ( z1 , z2 )/2σ 2 ), (2.2.8)
where d F P is the full Procrustes Distance defined in Eq.(2.2.5) and σ is the kernel parameter.

This kernel is positive definite for all σ ∈ R. In Section 2.4.2, it will be used for the extension of SCDL to Hilbert space.

Related work

In this section, we provide a brief literature overview of Riemannian SCDL approaches.

Then, we discuss different representations of trajectories on Riemannian manifolds and highlight their limitations.

Riemannian sparse coding and dictionary learning

The basic definition of SCDL as defined in Section 2.2.1 assumes that the query points as well as the dictionary atoms are defined in vector space. However, most suitable image descriptors often lie to nonlinear manifolds [START_REF] Yui | Advances in matrix manifolds for computer vision[END_REF]. Thus, to perform SCDL on these data while respecting the geometric structure of Riemannian manifolds, the classical problem of SCDL needs to be extended to its nonlinear counterpart. The main issue here arises from the fact that the linear combination of atoms is not possible on a nonlinear manifold since the resulting point may not even be in the manifold, as illustrated in Figure 2.3. Previous works addressed this problem [START_REF] Cherian | Riemannian dictionary learning and sparse coding for positive definite matrices[END_REF][START_REF] Guo | Action recognition from video using feature covariance matrices[END_REF][START_REF] Harandi | Riemannian coding and dictionary learning: Kernels to the rescue[END_REF][START_REF] Harandi | Extrinsic methods for coding and dictionary learning on grassmann manifolds[END_REF][START_REF] Ho | On a nonlinear generalization of sparse coding and dictionary learning[END_REF][START_REF] Li | Log-euclidean kernels for sparse representation and dictionary learning[END_REF][START_REF] Çetingül | Segmentation of high angular resolution diffusion mri using sparse riemannian manifold clustering[END_REF]. For instance, a straightforward -41 - solution was proposed in [START_REF] Guo | Action recognition from video using feature covariance matrices[END_REF][START_REF] Yuan | Human Action Recognition under Log-Euclidean Riemannian Metric[END_REF] by embedding the manifolds of interest into vector space, the tangent space at a reference point. However, this solution does not take advantage of the entire Riemannian structure of the manifold since on a tangent space, only distances to the reference point are equal to true geodesic distances. To overcome this problem, Ho et al. [START_REF] Ho | On a nonlinear generalization of sparse coding and dictionary learning[END_REF] proposed a general framework for SCDL in Riemannian manifolds by working on the tangent bundle. Here, each point is coded on its attached tangent space into which the atoms are mapped. By doing so, only distances to the tangent point are needed. Their proposed dictionary learning method includes an iterative update of the atoms using a gradient descent approach along geodesics. This general solution essentially relies on mappings to tangent spaces using the logarithm map operator. Although it is well defined for several manifolds, analytic formulation of the logarithm map is not available or difficult to compute for others.

Therefore, some studies [START_REF] Harandi | Riemannian coding and dictionary learning: Kernels to the rescue[END_REF][START_REF] Harandi | Extrinsic methods for coding and dictionary learning on grassmann manifolds[END_REF][START_REF] Mehrtash | Sparse coding and dictionary learning for symmetric positive definite matrices: A kernel approach[END_REF][START_REF] Li | Log-euclidean kernels for sparse representation and dictionary learning[END_REF] proposed to embed the Riemannian manifold into RKHSs which are vector spaces where linear SCDL becomes possible. Recently, Harandi et al. [START_REF] Harandi | Extrinsic methods for coding and dictionary learning on grassmann manifolds[END_REF] proposed to map the Grassmann manifolds into the space of symmetric matrices to overcome the latter problem and preserve several properties of the Grassmann structure.

They also proposed kernelized versions of the SCDL algorithms to handle the nonlinearity of the data, similarly proposed in [START_REF] Mehrtash | Sparse coding on symmetric positive definite manifolds using bregman divergences[END_REF] for symmetric positive definite matrices. Throughout this dissertation, we will investigate two paradigms of Riemannian SCDL in the shape manifold with the aim of coding trajectories while tackling different challenges that we will discuss in -42 -the following section.

Trajectory representations on Riemannian manifolds

A sequence of points that evolve over time on a manifold can be seen as a time-series. In the case of a Riemannian manifold, a time-series is usually denoted by trajectory. Analysis of these trajectories is challenging due to the nonlinearity of underlying spaces. Several representations of landmark sequences lie to nonlinear manifolds. In many approaches, the Riemannian geometry of these manifolds is exploited to analyze the corresponding representations and a common solution to solve for their nonlinearity consists on mapping the manifold-valued data to a common tangent space. A popular example is given by the Lie group and its use for skeletal trajectory analysis. For instance, Vemulapalli et al. [START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a lie group[END_REF] proposed to represent 3D skeletal sequences in the product space of Special Euclidean (Lie) groups SE(3) n . To this end, for each frame of a sequence, the Euclidean transformation matrices encoding rotations and translations between different joint pairs are computed.

Hence, the dynamics of these matrices is seen as a trajectory on SE(3) × ... × SE(3). To overcome the nonlinear nature of this manifold, this representation is mapped to the Lie algebra se(3) n which is a vector space, the tangent space at the identity element. However, mapping points to a common tangent space may introduce undesirable distortions, especially when the mapped points are far from the tangent point. Aware of this limitation, the authors in [START_REF] Vemulapalli | Rolling rotations for recognizing human actions from 3d skeletal data[END_REF] proposed a mapping of trajectories on Lie groups combining the usual logarithm map with a rolling map that guarantees a better flattening of trajectories on Lie groups. Taking another direction, Anirudh et al. [START_REF] Anirudh | Elastic functional coding of human actions: From vector-fields to latent variables[END_REF] extended the framework of Transported Square-Root Velocity Fields (TSRVF) [START_REF] Su | Statistical analysis of trajectories on riemannian manifolds: Bird migration, hurricane tracking, and video surveillance[END_REF] by modeling trajectories of human actions on the Grassmann manifold and the product space of Lie groups SE(3)×...×SE [START_REF] Andriluka | 2d human pose estimation: New benchmark and state of the art analysis[END_REF]. They tackled the problems of high-dimensionality of the feature space and its nonlinearity and proposed to learn a lowdimensional embedding using a manifold functional variant of principal component analysis.

Hence, each trajectory is mapped to a single point in a low-dimensional Euclidean space.

-43 -

Chapitre 2. Coding Human Motion Trajectories in the Shape Manifold

Another approach [START_REF] Ben Amor | Action recognition using rate-invariant analysis of skeletal shape trajectories[END_REF] proposed a different solution by extending the Kendall's shape theory to trajectories. Accordingly, translation, rotation, and global scaling are filtered out from each skeleton to quantify the shape. Then based on the TSRVF, they defined an elastic metric to jointly align and compare trajectories.

Riemannian trajectories were also considered in the analysis of 2D facial landmark sequences. Taheri et al. [START_REF] Taheri | Towards view-invariant expression analysis using analytic shape manifolds[END_REF] proposed to represent 2D facial landmarks in the Grassmann manifold which makes the resulting representation invariant to affine transformations and hence robust to view variations. To capture facial expressions from these representations, the authors computed the velocity vectors between successive frames using the logarithm map. To obtain velocities in the same tangent space, they applied a parallel transport of these velocity vectors to a common tangent space. However, their method depends on the choice of this tangent space. In another work [START_REF] Kacem | A novel space-time representation on the positive semidefinite cone for facial expression recognition[END_REF], 2D facial landmark sequences were first represented as trajectories of Gram matrices in the manifold of positive semidefinite matrices of rank 2. A similarity measure is then provided by temporally aligning trajectories while taking into account the geometry of the manifold.

Most of the methods described above share a common drawback which consists on mapping all the manifold-valued data to a reference tangent space. The major problem of this strategy is that distortions could be introduced especially when the mapped points are far from the tangent point. Moreover, comparing the resulting tangent vectors by computing distances between them is not accurate since only distances to the tangent point are equal to true geodesics. Therefore, our goal is to go beyond this drawback in the proposed intrinsic representation. On the other hand, to our knowledge, extrinsic approaches were not studied in the literature of human modeling with Riemannian trajectories. Hence, we also aim to explore this direction.

-44 -

Human motion modeling framework

Human motion modeling framework

In this section, we present our human motion modeling, i.e. representation of actions and facial expressions, which is used throughout this thesis work for the tasks of human motion classification and generation. Given a set of sequences of 2D/3D skeletons or facial landmarks, our approach consists in three main steps:

-Embedding each frame (i.e. landmark configuration) of a sequence to the Kendall's pre-shape space C by filtering out translation and scale.

-Learning a dictionary from all training samples (i.e. static pre-shapes on C). In this step, when an operation involves two configurations on C such as the logarithm map or the geodesic distance computation, rotation is filtered out by aligning one pre-shape into the second by applying the Procrustes algorithm [START_REF] Kendall | Shape manifolds, Procrustean metrics, and complex projective spaces[END_REF]. By doing so, we consider dictionary learning on the Kendall's shape space S.

-Using the learned dictionary, each frame (i.e. pre-shapes on C) of a sequence is coded using Riemannian sparse coding in S after filtering out rotation as done in the previous step.

As a main result of applying the proposed framework, each frame of an input sequence gives rise to a sparse code vector and frames that have similar shapes are expected to have similar code vectors. As a consequence, the input sequence will turn to a smoothly-evolving time-series (see Figure 2.4).

Chapitre 2. Coding Human Motion Trajectories in the Shape Manifold

In this work, we investigate two approaches of Riemannian SCDL. The first is intrinsic and is based on projections to tangent spaces, while the second is extrinsic and relies on embeddings to RKHS. We illustrate these two approaches in Figure 2.5. In the following, we describe each of them.

Figure 2.5 -Illustration of the adopted solutions to overcome the nonlinearity of the shape manifold. (A) The intrinsic approach maps the data on the manifold to tangent spaces using the logarithm map opertor. (B) The extrinsic approach embeds the manifold-valued data to RKHS by computing the inner product matrix using a positive definite kernel function.

Intrinsic approach

We propose to adapt a general intrinsic formulation of Riemannian SCDL [START_REF] Ho | On a nonlinear generalization of sparse coding and dictionary learning[END_REF] to the case of Kendall's shape space. This allows to transform a 2D/3D landmark configuration lying on the shape manifold to a sparse vector.

Intrinsic Sparse Coding

Let D = { d1 , d2 , ..., dN } be a dictionary on S, and similarly the query Z is a point on S.

Accordingly, the problem of sparse coding involves the geodesic distance defined on S and thus the Euclidean formulation in Equation (2.2.2) (in Section 2.2.1) becomes

l S ( Z, D) = min w (d S ( Z, F (D, w)) 2 + λf (w)). (2.4.1)
Here, F : S N ×R N → S denotes an encoding function that generates the approximated point Ẑ on S by combining atoms with codes. Note that in the special case of Euclidean space, F (D, w) would be a linear combination of atoms. However, in the Riemannian manifold S,

we have forsaken the structure of vector space which makes the linear combination of atoms lying on S no longer applicable, since the approximated Ẑ may lie out of the manifold. An interesting alternative is the intrinsic formulation of Eq. (2.4.1), when considering that S is a complete Riemannian manifold, thus, the geodesic distance d S ( Z, d) = log Z ( d) Z (as explained in Section 2.2.3). As a consequence, the cost function in (2.4.1) can be written as

l S ( Z, D) = min w N i=1 [w] i log Z ( di ) 2 Z + λf (w), (2.4.2)
where log Z denotes the logarithm map operator that maps each atom d ∈ S to the tangent space T Z (S) at the point Z being coded, and . Z is the norm induced by the Riemannian metric at T Z (S). Mathematically, this allows to partially compensate the lack of vector space structure on S, as illustrated in Figure 2.6. To avoid the solution w = 0, we imposed in Eq. (2.4.2) an important additional affine constraint defined as N i=1 [w] i = 1. By this formulation of sparse coding, we only compute distances to the tangent point, hence we avoid the commonly induced distortions when working in a reference tangent space. By substituting the logarithm map by its explicit formulation in Eq. (2.4.2), we have

l S ( Z, D) = min w N i=1 [w] i θ sin(θ) (d i O * -cos(θ)Z) 2 Z + λf (w). (2.4.3)
In practice, Eq. (2.4.3) is computed by first finding the optimal rotation O * between Z and each atom d i via the Procrustes algorithm [START_REF] Kendall | Shape manifolds, Procrustean metrics, and complex projective spaces[END_REF]. Then, we solve for w using the state-ofthe-art CVXPY optimizer [START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF]. In Algorithm 2, we provide a summary of the sparse coding approach on the shape manifold.

-47 -

* = argmin w N i=1 [w] i V i 2 2 + λf (w) da * = argmin da m j=1 [w a ] log Zj ( da ) 2 Zj + N i=1;i =a [w i ] log Zj ( di ) 2 Zj +λf (w j ) return Dictionary D = { di } N i=1 , di ∈ S
An efficient dictionary initialization for faster learning The performance of sparse coding depends on the number of the dictionary elements N , and an empiric choice of N can be time consuming, especially when it comes to large datasets. As a solution, we propose an initialization step that enables an automatic inference on N and accelerates the convergence of the dictionary learning algorithm. To this end, we propose to cluster the training shapes by adapting the Bayesian clustering of shapes of curves method proposed in [START_REF] Zhang | Bayesian clustering of shapes of curves[END_REF]. The latter brings the advantage of automatically inferring the number of clusters from a set of data. From each cluster, main representatives will then be selected to constitute the initial dictionary.

Kernel-based clustering of shapes for dictionary learning -In Atoms inference from clusters -To select the best representatives of a cluster, the mean shape is a suitable candidate but it is not sufficient to summarize the intra-cluster variability. For that, we propose to apply principal geodesic analysis (PGA), first proposed by [START_REF] Fletcher | Principal geodesic analysis for the study of nonlinear statistics of shape[END_REF]. Specifically, all elements of a culster are mapped to the tangent space at the mean -50 -shape T μ(S ). Then, principal component analysis (PCA) is applied in this vector space to induce the main components (tangent vectors). Finally, the induced vectors are mapped back to the manifold S using the exponential map operator to represent inital atoms. This procedure is applied to all the clusters. Note that an important advantage of performing PGA in each cluster rather than in the whole training set is to avoid the problematic case of having points in the manifold that are far from the tangent point.

Extrinsic approach

The SCDL algorithms only depend on the notion of inner product. In the following, we will discuss how they can be easily extended to RKHS using the Procrustes Gaussian Kernel.

Kernel Sparse Coding

A closed-form solution of kernel sparse coding is proposed in [START_REF] Harandi | Extrinsic methods for coding and dictionary learning on grassmann manifolds[END_REF]. To derive it, let us first define φ : S → H a mapping to RKHS induced by the kernel k( z1 , z2 ) = φ( z1 ) T φ( z2 ), where z1 , z2 ∈ S. For a query shape z ∈ S, extending Eq. 2.2.2 to RKHS yields

l H (z, D) = min w φ(z) - N i=1 [w] i φ( di ) 2 2 + λf (w), (2.4.5)
with N i=1 [w] i = 1. In Eq. 2.4.5, since the sparsity term depends entirely on w, only the reconstruction term needs to be kernelized. Expanding the latter gives

φ(z) - N i=1 [w] i φ( di ) 2 2 = φ(z) T φ(z) = -2 N i=1 [w] i φ( di ) T φ(z) + N i,j=1 [w] i [w] j φ( di ) T φ( dj ) = k(z, z) -2w T k(z, D) + w T K(D, D)w, (2.4.6)
where k(z, D) is the N -dimensional kernel vector computed between the query z and the dictionary atoms, and K(D, D) is the N × N kernel matrix computed between the atoms.

-51 -An efficient solution of kernel sparse coding can be obtained by considering U ΣU T as the SVD of the symmetric positive definite kernel K(D, D), and k(z, z) as a constant term (independent on w). Thus, Eq. 2.4.6 can be written as the least-squares problem in R N :

min w z -Dw 2 2
, where D = Σ 1/2 U T and z = Σ -1/2 U T k(z, D) (we refer to [START_REF] Harandi | Extrinsic methods for coding and dictionary learning on grassmann manifolds[END_REF] for the proof). In this work, this approach is applied in the Kendall's shape space by using the Procrustes Gaussian Kernel defined in Section 2.2.3.2.

Kernel Dictionary learning

Similarly to Euclidean dictionary learning, the extrinsic Riemannian formulation is based on an alternating optimization strategy to update weights and atoms. While the first step is obtained with extrinsic sparse coding presented above, the second is presented in what follows. Given the codes from the first step, the problem of dictionary learning can be viewed as optimizing Eq. (2.4.5) over D. The main idea here is to represent D as a linear combination of the training samples Y in RKHS according to the Representer theorem [START_REF] Schölkopf | A generalized representer theorem[END_REF].

The resulting weights for the M training samples are stacked in a M × N matrix V , which gives φ(D) = φ(Y )V . Since only the first term in Eq. 2.4.5 depends on D, the problem of dictionary update can be written as

U (V ) = φ(Y ) -φ(Y )V W 2 2
, where W is the N × M matrix of sparse codes obtained from the first step. The latter can be expanded to:

U (V ) = T r(φ(Y )(I M -V W )(I M -V A) T φ(Y ) T ) = T r(K(Y, Y )(I M -V W -W T V T + V W W T V T )).
To obtain the updated dictionary that is now defined by V , the gradient of U (V ) is zeroed out w.r.t V . This gives V = (W W T ) -1 W = W † , where † is the pseudo-inverse operator.

-52 -

Properties of the latent space

Sparse coding of trajectories give rise to time-series defined in vector space which we refer to as the latent space. In this section, we describe the main properties of the obtained representations in this space.

Reconstruction of trajectories

As illustrated in Figure 2.8, an important advantage of the intrinsic approach is that it enables to recover a sparse code back to the original manifold. This can be extremely useful for visualization purposes and for certain tasks such as motion generation, as will be studied in Chapter 4. Shape reconstruction is achieved with respect to the pre-learned dictionary by applying the weighted intrinsic mean algorithm described in Algorithm 4. The idea here is based on the intrinsic mean, i.e. Algorithm 1, where we now: 1) Initialize the approximated shape μ as the linear combination of codes and the corresponding atoms from the dictionary.

By doing so, we can obtain a good approximation of the original shape, knowing that the code vector is sparse and supposing that atoms with non-zero coefficients (elements of code vector) are the closest to the tangent point; 2) Compute tangent vectors as log μi (w j Dj ) and the mean tangent vector v. Then update μ by moving v to the average direction. This is done iteratively until the norm of v is sufficiently close to zero.

Algorithm 4: Weighted intrinsic mean for shape reconstruction from code input : A vector of codes shapes

{w j } m j=1 , a dictionary D = { Dj } m j=1 and 1 , 2 small Initialize μ ← 1 m m j=1 w j D j , i ← 0 repeat Compute v j ← log μi (w j Dj ) Compute average tangent vector v ← 1 k m j=1 v j Update μi according to μi+1 ← exp μi ( 2 v) i ← i + 1 until |v| < 1 return μ, the reconstructed shape -53 - Figure 2.
8 -Given the pre-trained dictionary D, skeletal trajectories in the shape manifold can be reconstructed from the space of sparse codes using the weighted intrinsic mean algorithm.

Efficient tangent space projections

As explained in Section 2.3.2, many approaches that model sequences of human landmarks as trajectories on Riemannian manifolds [START_REF] Ben Amor | Action recognition using rate-invariant analysis of skeletal shape trajectories[END_REF][START_REF] Taheri | Towards view-invariant expression analysis using analytic shape manifolds[END_REF][START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a lie group[END_REF] share a major drawback, that of mapping manifold-valued points to a reference tangent space which may introduce distortions especially when points are far from the tangent point. In contrast, in our coding approach, each point is coded on its attached tangent space, where the dictionary atoms are mapped (see Figure 2.9 for illustrations of tangent space approximation strategies). By doing so, we only compute distances to the tangent point which are equal to true geodesics. Furthermore, even though we map all the atoms to a tangent space where some of them may be far from the tangent point, in practice, distortions are usually avoided since our sparse coding scheme tends to code a point using the closest atoms to it and attributes zeros to the rest, distant atoms. As a consequence, assuming that the dictionary is well learned (i.e.

the atoms cover all the space of training shapes), our approach considerably alleviates the non-trivial problem of trajectory distortions caused by tangent space approximations.

-54 -Figure 2.9 -Schematic tangent space projections using our method compared to state-ofthe-art.

Denoising of skeletal shapes

Skeletal joints obtained from low cost sensors are often noisy leading to abnormal skeletal poses. This is a non-trivial issue for applications as action recognition since the performance of a landmark-based recognition approach relies essentially on the accuracy of the extracted landmarks. One advantage of the proposed sparse coding approach is that it naturally enables denoising of skeletons when assuming a clean dictionary. Figure 2.10 presents an illustration of this property using data collected from the state-of-the-art MSR-Action 3D dataset [START_REF] Li | Action recognition based on a bag of 3D points[END_REF].

Here, only certain joints are poorly estimated in a body pose, e.g., right and left knees, hence the global shape is preserved. Sparse coding attempts to approximate this shape using the closest atoms. Assuming that the dictionary does not contain abnormal shapes, the resulting approximation is expected to recover the input abnormal shape. The question now is how to obtain a clean dictionary? Recall that to train a dictionary, skeletons are collected from all training sequences. We point out that in general, noise appears in only certain frames of a sequence. In addition, actions usually evolves smoothly over time. Considering these two information, one can compute distances between successive frames and discard skeletons with a relatively great distance to the previous.

-55 -

On the vector structure of the latent space

Standard notions of statistics (e.g., mean computation, interpolation, etc.) and analysis of time-series (e.g., temporal alignment, temporal modeling, etc.) need significant modification to account for the nonlinearity of the shape manifold. In most cases, these operations become highly involved in terms of computational complexity, and often result in iterative procedures further increasing the computational load. The proposed sparse coding approach allows to exploit the linear nature of the latent variables as well as their low-dimensionality to compute standard statistics on the data and apply standard techniques to process timeseries, rather than performing them directly on the manifold. For instance, one can compute the mean code linearly and recover it back to the manifold and still obtain a point on the manifold that is very close to the intrinsic mean shape of the same data. An illustration of this is given in Figure 2.11. One can also interpolate linearly between latent variables and obtain meaningful interpolates when mapped back to the shape manifold, see Figure 2.12. Now, we turn our attention to our main concern which is to consider time-evolving shapes that represent actions or facial expressions. As explained previously, sparse coding of each -56 -

Summary

In this chapter, we proposed a novel action and facial expression modeling based on Riemannian sparse coding and dictionary learning in the shape manifold. This solution allows to overcome the nonlinear structure of the manifold by mapping a Riemannian trajectory to an Euclidean time-series. In addition to its sparsity and vector structure, this representation allows to reconstruct original trajectory from its latent representation -57 -Chapitre 2. Coding Human Motion Trajectories in the Shape Manifold thanks to the dictionary. In addition, it presents a natural denoising tool allowing to alleviate the noise present in the data. We explored both intrinsic and extrinsic solutions of SCDL.

The first extends sparse coding to tangent spaces while avoiding a commonly encountered problem which is to map all manifold-valued data to a common tangent space. The second is based on embedding the manifold-valued data to Hilbert space via a positive definite kernel function. In Chapter 3, these two approaches will be further studied and evaluated in the context of two recognition tasks: 3D action recognition and 2D facial expression recognition.

In Chapter 4, we will exploit the intrinsic approach to generate new skeletal sequences and show how we can use them to guide the generation of human videos.

-58 -

Chapitre 3

Facial Expression and Action

Recognition with Sparse Representations

Introduction

In the previous chapter, we proposed a novel framework to encode trajectories in the shape manifold. We explored two alternatives of Riemannian SCDL allowing to represent human actions and facial expressions as sparse times-series in vector space. The first is an intrinsic solution where SCDL is performed on the manifold tangent spaces while the second is based on embedding the manifold-valued data to Hilbert spaces. In this chapter, we demonstrate the effectiveness of these sparse representations in two recognition tasks: the 3D action recognition and 2D facial expression recognition (both micro and macro expressions). Specifically, we will show that the intrinsic coding approach is efficient to code 3D shape trajectories while the extrinsic method is suitable for 2D trajectories. In the context of the addressed classification problems, these coding techniques bring two -59 -Chapitre 3. Facial Expression and Action Recognition with Sparse Representations main advantages: (1) Sparse coding of shapes is performed with respect to a Riemannian dictionary. Hence, the resulting sparse times-series are expected to be more discriminative than the data themselves. In addition, they are robust to noise, knowing that SCDL is a powerful denoising tool as demonstrated in Section 2.5.3 of Chapter 2; (2) Using sparse time-series as discriminative features allows us to perform both temporal modeling and classification in vector space, avoiding the more difficult task of classification on the manifold.

To this end, we will study and compare two different pipelines for temporal modeling and classification. The first is based on a standard machine learning technique while the second is a deep learning approach based on RNNs. An overview of the proposed approaches is given in Figure 3.1. -A comparative study on the intrinsic and extrinsic paradigms of Riemannian SCDL in the 2D and 3D shape manifolds. To the best of our knowledge, this work is the first to apply and compare both approaches to dynamic 2D and 3D shapes.

Related work

The typical framework of human motion recognition using skeletons or faces comprises the following phases, see Figure 3 In one hand, we categorise the classical methods into three groups:

Graph-based methods Graph is a powerful tool for modeling structured objects. Since the representation of a human skeleton or face is in essence composed of points that are connected to each other, it is natural to perceive it as a graph. Therefore, several approaches opted for graphs to model the spatial dependencies between human joints as well as their temporal evolution. Probabilistic models such as Hidden Markov models (HMMs), which also falls in this category, have been widely used to model the temporal evolution of human pose sequences.

Kernel methods attempt to compute similarities between data-points (e.g. landmark -62 -configurations, features extracted from them, etc.) using kernel functions. This enable them to operate in a high-dimensional, implicit feature space. The latter is also called the inner product space since only inner products between data-points are computed, without ever computing the coordinates of the data in the new space. These approaches are known to bring a richer representation of the original data since the inner product space is usually higher-dimensional which helps classification methods to identify complex patterns.

Riemannian methods Several representations of landmark sequences may lie to nonlinear manifolds where traditional computational tools and machine learning techniques cannot be directly applied. In fact, in contrast to vector spaces, these manifolds are characterized with nonlinear topology that algorithms have to take into account. As an example, to compute the similarity between two points on a nonlinear manifold, the Euclidean distance is no longer suitable since it does not represent the real proximity between them. As a solution, several approaches studied the Riemannian geometry of these manifolds to define a metric which is obtained by defining a smoothly varying inner product on each tangent space of the manifold. Hence, the vector space structure of these tangent spaces can be exploited to overcome the nonlinearity of Riemannian manifolds.

On the other hand, feature learning using deep learning techniques has been receiving increasing attention in recent years due to their ability of designing powerful features without the need of heavy human labor and domain expert knowledge to develop effective feature extraction methods. Their success is also attributed to the access to Graphics Processing Units (GPUs) as well as to large scale labeled datasets which allow to design networks with millions of parameters. These deep learning techniques consist of multi-layer neural networks with a number of connected units (neurons). These layers represent three types: input layer with input units receiving information to be processed, output layer with output units giving the result of the network, and hidden layers with hidden units which process the data. The training objective of these neural networks consists of learning the weights of the connections between neurons in order to determine the mapping between the input -63 -and the output. The most popular deep learning methods that are commonly used in action and facial expression recognition are convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The former are based on the powerful convolution operation and belong to the category of feed-forward neural networks where the information moves only in one direction, that of the output layer. The latter are a kind of neural networks that takes sequential input and infer sequential output by sharing recurrent connections (parameters) between time steps, in contrast to CNNs.

In the following, we review some examples of methods from the categories discussed above in the context of 3D action recognition and 2D facial expression recognition, with a focus on Riemannian approaches since our proposed framework is also Riemannian. between pairs of two independent frames. Another work [START_REF] Wang | Graph based skeleton motion representation and similarity measurement for action recognition[END_REF] modeled a human action as a set of semantic parts called motionlets obtained by tracking then segmenting the trajectory of each joint. By combining the motionlets and their spatio-temporal correlations, they proposed an undirected complete labeled graph to represent a video, and a subgraph-pattern graph kernel to measure the similarity between graphs, then to classify videos.

3D action recognition

In the category of kernel-based approaches, two kernel-based tensor representations named sequence compatibility kernel (SCK) and dynamics compatibility kernel (DCK) were introduced in [START_REF] Koniusz | Tensor representations via kernel linearization for action recognition from 3d skeletons[END_REF]. These can capture the higher-order relationships between the joints. The -64 -authors in [START_REF] Ben Amor | Action recognition using rate-invariant analysis of skeletal shape trajectories[END_REF] represented skeletal sequences as trajectories in the Kendall's shape space then modeled them using TSRVF. For classification, they computed the mean trajectories of each class and for each trajectory they extracted a feature vector formed by distances to mean trajectories of each class. Finally, they used SVM to classify these feature vectors. Recall that a common drawback of many of these approaches is the mapping of all manifold-valued data to a reference tangent space which may introduce distortions. In contrast, our coding approach in the shape manifold avoids such a problem.

On the other hand, RNNs, which belong to the category of deep learning approaches according to our categorisation in Figure 3.3, have showed promising performance when applied to 3D action recognition. For instance, HBRNN-L [START_REF] Yong Du | Hierarchical recurrent neural network for skeleton based action recognition[END_REF] applied bidirectional RNNs hierarchically by dividing a skeleton into five parts of neighboring joints. Then, each is separately fed into a bidirectional RNN before fusing their outputs to form the upper-body and the lower-body. Similarly, these latter were fed into different RNNs and their outputs fusion form the global body representation. More recently, the spatio-temporal LSTM (ST-LSTM) [START_REF] Liu | Spatio-temporal lstm with trust gates for 3d human action recognition[END_REF] extended LSTM to spatio-temporal domains. To this end, the analysis of a 3D skeleton joint considers spatial information from neighboring joints and temporal information from previous frames. In addition, a tree-structure based method allows to better describe the adjacency properties among the joints. This method is further improved by a gating mechanism to handle noise and occlusion.

2D facial expression recognition

The task of facial expression recognition (FER) consists in recognizing the basic emotions, e.g., fear, surprise, happiness, etc. Recall that facial landmarks are located in certain regions of the face such as the mouth, the eyes, eyebrows, etc. As stated in [23], the motion of these landmarks defining facial expressions allows to characterize the emotional state of humans. Therefore, representing and classifying sequences of facial landmarks has been widely used in the literature of FER.

Falling in the category of graph-based approaches, a geometric approach was proposed in [START_REF] Wang | Capturing complex spatio-temporal relations among facial muscles for facial expression recognition[END_REF] which introduced a unified probabilistic framework based on an interval temporal Bayesian network (ITBN) built from the movements of landmark points. Aware of the small variations along a facial expression, the authors in [START_REF] Jain | Facial expression recognition with temporal modeling of shapes[END_REF] proposed a method to capture the subtle motions within facial expressions using a variant of Conditional Random Fields (CRFs) called Latent-Dynamic CRFs (LDCRFs) on geometric features.

Taking the direction of Riemannian methods, Taheri et al. [START_REF] Taheri | Towards view-invariant expression analysis using analytic shape manifolds[END_REF] proposed to represent 2D facial sequences as parameterized trajectories on the Grassmann manifold of 2-dimensional subspaces in R n (n is the number of landmarks) which is an affine-invariant shape representation. To capture the facial deformations, they used geodesic velocities between facial shapes and finally, classification was performed by applying LDA then SVM. In the work of Kacem et al. [START_REF] Kacem | A novel space-time representation on the positive semidefinite cone for facial expression recognition[END_REF], 2D facial landmark sequences were first represented as trajectories of Gram matrices in the manifold of positive semidefinite matrices of rank 2. A similarity measure is then provided by temporally aligning trajectories while taking into account the geometry of the manifold. This measure is finally used to train a pairwise proximity function SVM.

More recent approaches exploited deep neural networks. In [START_REF] Jung | Joint fine-tuning in deep neural networks for facial expression recognition[END_REF], two neural network architectures were proposed for image videos (DTAN) and 2D facial landmark sequences (DTGN) which are combined (forming DTAGN) to predict final emotions. In particular, DTGN showed to be efficient by using only 2D landmark sequences, when applied separately.

Another approach is proposed in [START_REF] Zhang | Facial expression recognition based on deep evolutional spatial-temporal networks[END_REF] -67 -Although the macro facial expression recognition problem has seen considerable advances, micro-expression recognition is still a relatively challenging task [START_REF] Oh | A survey of automatic facial micro-expression analysis: Databases, methods and challenges[END_REF]. Microexpressions are brief facial movements characterized by short duration, involuntariness and subtle intensity. In the literature, previous methods opted for extracting hand-crafted features from texture videos such as LBP-TOP and HOOF [START_REF] Hao Zheng | A relaxed k-svd algorithm for spontaneous micro-expression recognition[END_REF].

More recently, deep learning methods were proposed to tackle the problem by applying CNNs [START_REF] Breuer | A deep learning perspective on the origin of facial expressions[END_REF][START_REF] Kim | Micro-expression recognition with expression-state constrained spatio-temporal feature representations[END_REF] and RNNs [START_REF] Kim | Micro-expression recognition with expression-state constrained spatio-temporal feature representations[END_REF]. To our knowledge, only the method of [START_REF] Yoon Choi | Recognizing fine facial microexpressions using two-dimensional landmark feature[END_REF] is entirely based on analyzing 2D facial landmark sequences. Their work is based on computing the pointwise distances between adjacent landmark configurations along a sequence which is stacked in a matrix. The latter was seen as an input image to a CNN-LSTM-based classifier. However, their approach was only evaluated on a synthesized dataset produced from a macro-expression dataset. In our work, we will show that we achieve state-of-the-art results on a commonly-used micro-expression dataset using only 2D landmark data.

Temporal modeling and classification

Let { Z1 , Z2 , ..., ZL } be a sequence of skeletons representing a trajectory on S. As described previously, we code each skeleton Zi into a sparse vector w i ∈ R N with respect to a dictionary D. As a consequence, each trajectory is mapped to an N -dimensional function of sparse codes and the problem of classifying trajectories on S is turned to classifying Ndimensional sparse codes functions in Euclidean space, where standard tools designed for Euclidean time-series (e.g., temporal modeling, machine learning techniques, etc.) could be directly applied. We adopt and evaluate two temporal modeling and classification schemes to recognize actions and facial expressions.

-68 -

Dynamic time warping, Fourier pyramid and SVM

A non-trivial challenge in recognizing actions and facial expressions resides in their rate variability since they can be executed at different speed. Thus, a typical landmark sequence representation has to be invariant to the execution rate. In other words, sequences belonging to the same class should typically have similar parametrization. A commonly used technique, namely Dynamic Time Warping (DTW) can temporally align one sequence into another by finding the optimal re-parameterization that minimizes a similarity measure between the two. In our work, we exploit the vector structure of our latent space to directly apply DTW on the sequential data, avoiding the more difficult task of temporal alignment of trajectories on the manifold. Another important post-processing is to further filter out noise present in the data. For instance, Fourier Temporal Pyramid (FTP) [START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF] have shown to be very effective for recognition tasks involving noisy data as it maps a time-series to the Fourier domain and eliminates the high frequency elements. Moreover, FTP is known to be robust to temporal mis-alignment. In our work, we apply a pipeline of DTW and FTP, then classify final features using a one-vs-all linear SVM. By doing so, we handle rate variability, temporal misalignment and noise, and classify final features, respectively.

It is important to note that to be able to apply the FTP approach, the pose sequences should have the same temporal length. For that, we need to apply a temporal re-sampling to all sequences which can be achieved in two alternatives:

-After projecting the input sequences to the Kendall's shape space, we can apply a Riemannian re-sampling algorithm as described in Algorithm 5. This alternative is better applied when the dataset contains long sequences. Thus, one can perform downsampling of trajectories which will reduce the computational cost of the sparse coding step.

-As explained in Section 2.5.4 of Chapter 2, one can exploit the vector structure of the latent space to apply algorithms designed for Euclidean time-series which are in -69 -most cases faster than their nonlinear equivalent. Thus, one can apply SCDL on the input trajectories, then perform a Euclidean temporal re-sampling on the obtained sequences of sparse codes.

Temporal re-sampling of trajectories Temporal up-sampling allows to increase timesteps in a sequence for more accuracy, while temporal down-sampling decreases them to reduce the computation complexity of algorithms as an example. Besides, one may need to have a set of sequences with the same length by applying up or down sampling to each sequence. This can be achieved in the shape manifold by applying the algorithm described in Algorithm. 5.

Algorithm 5: Re-sampling of trajectories on S input : Given a trajectory α(t) t=t 1 ,...,tn , we seek α(s) s=s 1 ,...,s l where l < n for down-sampling and l > n for up-sampling.

for i = 1 to l do Find t i 1 , t i 2 such that t i 1 ≤ s i ≤ t i 2 Compute w 1 = s i -t i 1 t i 2 -t i 1 and w 2 = t i 2 -s i t i 2 -t i 1
x = α(i 1 ), y = α(i 2 ), θ = d S (x, y) then α(s i ) = 1 sin(θ) (sin(w 2 θ)x + sin(w 1 θ)y) return Re-sampled trajectory

Long short-term memory network

Modeling sequential data using recurrent neural networks (RNNs) has been widely used in different computer vision tasks and has led to breakthrough results in natural language processing [START_REF] Socher | Parsing natural scenes and natural language with recursive neural networks[END_REF], speech recognition [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF], etc. RNNs are a kind of neural networks that take sequential input and infer sequential output by sharing parameters between time steps. They are trained using back-propagation over time. However, standard RNNs lack the ability of learning long-term dependencies as they suffer from the problem of vanishing gradient [START_REF] Hochreiter | The vanishing gradient problem during learning recurrent neural nets and problem solutions[END_REF].

Tackling this problem, Long short-term memory (LSTM) network [START_REF] Hochreiter | Long short-term memory[END_REF] is equipped with a gating mechanism that learns which information is relevant to keep or forget during training.

-70 -Thereby they better handle the problem of learning long-term dependencies in sequential data. In the context of action recognition, many works in the literature opted for LSTMs to model and classify actions [START_REF] Liu | Spatio-temporal lstm with trust gates for 3d human action recognition[END_REF][START_REF] Shahroudy | Ntu rgb+ d: A large scale dataset for 3d human activity analysis[END_REF][START_REF] Zhang | View adaptive recurrent neural networks for high performance human action recognition from skeleton data[END_REF]. In [START_REF] Shahroudy | Ntu rgb+ d: A large scale dataset for 3d human activity analysis[END_REF], the authors propose a part-aware LSTM where they divide a skeleton configuration into body parts, hence instead of keeping a longterm memory of the entire body's motion in the cell, they split it to part-based cells. Another work [START_REF] Zhang | View adaptive recurrent neural networks for high performance human action recognition from skeleton data[END_REF] tackles the problem of view variations in the captured actions, similarly to our work but instead of seeking a view-invariant representation then use it for classification, they proposed a view-adaptive LSTM-based network that automatically regulates observation viewpoints during the occurrence of an action. In our work, we aim to take advantage of the view-invariant nature of our representation as well as its compactness (sparsity) and vector space structure to apply an LSTM directly on the SCDL time-series. Further, we explore the use of Bi-directional LSTM (Bi-LSTM). Bi-LSTM is an extension of LSTM that presents each sequence backwards and forwards to two separate recurrent networks, providing context both from the future and past, respectively [START_REF] Graves | Framewise phoneme classification with bidirectional lstm and other neural network architectures[END_REF]. We will experimentally show that Bi-LSTM can achieve slight improvements over the traditional LSTM in recognizing human motion.

Dictionary structure

In the context of classification, one may exploit the important information of data labels to construct more discriminative feature vectors. To this end, we propose to build classspecific dictionaries, similarly to [START_REF] Guha | Learning sparse representations for human action recognition[END_REF]. Formally, let S be a set of labeled sequences on S belonging to q different classes {c 1 , c 2 , ..., c q }, we aim to build q class-specific dictionaries {D 1 , D 2 , ..., D q } in S such that each D j is learned using skeletons belonging to training sequences from the corresponding class c j . In this scenario, coding a query shape Z ∈ S is done with respect to each D j,1≤j≤q , independently. As a result, q vectors of codes are obtained. These vectors are then concatenated to form a global feature vector W . As will be discussed in Section 3.4.3, this yields more discriminative feature vectors for classification.

-71 -

Experimental evaluation

We perform extensive experiments to evaluate the effectiveness of the proposed frameworks in the tasks of: 3D action recognition, 2D macro-and micro-facial expression recognition. We provide comparisons to some state-of-the-art approaches on several publicly available datasets in addition to comparisons between the intrinsic and extrinsic paradigms of the proposed SCDL approach. Moreover, we perform baseline experiments to evaluate some properties of our recognition frameworks.

3D action recognition

Datasets

We evaluate the proposed skeletal representation using four benchmark datasets presenting different challenges: Florence3D-Action [START_REF] Seidenari | Recognizing actions from depth cameras as weakly aligned multi-part bag-ofposes[END_REF], UTKinect-Action [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF], MSR-Action 3D [START_REF] Li | Action recognition based on a bag of 3D points[END_REF], and the large-scale NTU-RGBD dataset [START_REF] Shahroudy | Ntu rgb+d: A large scale dataset for 3d human activity analysis[END_REF].

Florence3D-Action dataset consists of 9 actions performed by 10 subjects. Each subject performed every action two or three times for a total of 215 action sequences. The 3D locations of 15 joints collected using the Kinect sensor are provided. The challenges of this dataset consist of the similarity between some actions and also the high intra-class variations as same action can be performed using left or right hand.

UTKinect-Action dataset consists of 10 actions performed twice by 10 different subjects for a total of 199 action sequences. The 3D locations of 20 different joints captured with a stationary Kinect sensor are provided. The main challenge of this dataset is the variations in the view point.

MSR-Action 3D dataset consists of 20 actions performed by 10 different subjects. Each subject performed every action two or three times for a total of 557 sequences. The 3D locations of 20 different joints captured with a depth sensor similar to Kinect are provided with the dataset. This is a challenging dataset because of the high similarity between many actions (e.g., hammer and hand catch).

NTU-RGB+D is one of the largest 3D human action recognition datasets. It consists of 56, 000 action clips of 60 classes. 40 participants have been asked to perform these actions in a constrained lab environment, with three camera views recorded simultaneously. Each Kinect sensor estimates and records 25 joints coordinates reported in the 3D camera's coordinate system.

Experimental settings

For the first three datasets, we followed the cross-subject test setting of [START_REF] Wang | Robust 3D action recognition with random occupancy patterns[END_REF], in which half of the subjects was used for training and the remaining half was used for testing.

Reported results were averaged over ten different combinations of training and test data.

For Florence3D-Action and UTKinect-Action datasets, we followed an additional setting for each: Leave-one-actor-out (LOAO) [START_REF] Seidenari | Recognizing actions from depth cameras as weakly aligned multi-part bag-ofposes[END_REF][START_REF] Wang | Mining 3d key-pose-motifs for action recognition[END_REF] and Leave-one-sequence-out (LOSO) [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF], respectively. For MSR-Action3D dataset, we also followed [START_REF] Li | Action recognition based on a bag of 3D points[END_REF] and divided the dataset into three subsets AS1, AS2, and AS3, each consisting of 8 actions, and performed recognition on each subset separately, following the cross-subject test setting of [START_REF] Wang | Robust 3D action recognition with random occupancy patterns[END_REF]. The subsets AS1 and AS2 were intended to group actions with similar movements, while AS3 was intended to group complex actions together. In all experiments, we performed recognition based on the two classification schemes: Bi-LSTM and DTW-FTP-SVM.

For NTU-RGB+D, the authors of this dataset recommended two experimental settings that we follow: 1) Cross-subject (X-Sub) benchmark with 39,889 clips from 20 subjects for training and 16,390 from the remaining subjects for testing; 2) Cross-view (X-View) benchmark with 37,462 and 18,817 clips for training and testing. Training clips in this setting come from the camera views 2 and 3 while the testing clips are all from the camera view 1. Due to the huge amount of data in this dataset, we construct dictionaries using the -73 -kernel clustering approach presented in Section 2.4.1.2 since it is less time consuming than the dictionary learning optimization problem. Note that NTU-RGBD dataset contains two types of actions: daily activities where performed by one actor and interactions between two actors. For the latter, we perform sparse coding of each actor's skeleton separately.

Further, since the closeness between the two actors is a relevant information, we compute the Euclidean distance between their center of mass and concatenate it to the feature vector obtained after sparse coding. Moreover, we compute displacement vectors, as described in Section 3.4.2.2 for micro-expressions, and fuse them with the rest of the features. Finally, we perform temporal modeling and classification using Bi-LSTM. For this dataset, we do not suggest using the first classification pipeline as it would be highly consuming in terms of computation of DTW and FTP due to the huge amount of data that NTU-RGBD contains. Comparison to Riemannian methods on MSR-Action, Florence3D and UTKinect datasets The first row of methods in Table 3.1 reports the recognition results of different Riemannian approaches. Since in [START_REF] Ben Amor | Action recognition using rate-invariant analysis of skeletal shape trajectories[END_REF] human actions are also represented as trajectories in the Kendall's shape space, we report additional results of [START_REF] Ben Amor | Action recognition using rate-invariant analysis of skeletal shape trajectories[END_REF] on Florence3D and UTKinect datasets to give more insights about the strength of our coding approach compared to the method of [START_REF] Ben Amor | Action recognition using rate-invariant analysis of skeletal shape trajectories[END_REF]. In Table 3.1, it can be seen that we obtain better results than all Riemannian approaches on the three datasets. We recall that one common drawback of these methods is to map trajectories on manifolds to a reference tangent space, where they compute distances between different points (other than the tangent point). This may introduce distortions, especially when points are not close to the reference point. However, our method avoids such a non-trivial problem as coding of each shape is performed on its attached tangent space and the only measures that we compute are with respect to the tangent point. Now, we discuss our results obtained with the first classification scheme, i.e., DTW-FTP-SVM, similarly used in [START_REF] Anirudh | Elastic functional coding of human actions: From vector-fields to latent variables[END_REF][START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a lie group[END_REF][START_REF] Vemulapalli | Rolling rotations for recognizing human actions from 3d skeletal data[END_REF]. In the three datasets, it is clearly seen that our approach outperforms existing approaches when using the same classification pipeline, which shows the effectiveness of our skeletal representation. For instance, we highlight an improvement of 1.73% on MSR-Action 3D (following protocol [START_REF] Li | Action recognition based on a bag of 3D points[END_REF]) and 1.45% on Florence3D-Action. Now, we discuss the results we obtained using Bi-LSTM. Note that although we do not perform any preprocessing on the sequences of codes before applying Bi-LSTM, our approach still outperforms existing approaches on Florence3D, with 1.64% higher accuracy. However, it performs less well on UTKinect yielding an average accuracy of 96.89% against 97.08%

Results and discussions

obtained in [START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a lie group[END_REF]. In MSR-Action 3D, our approach performs better than the method of [START_REF] Anirudh | Elastic functional coding of human actions: From vector-fields to latent variables[END_REF] using the first protocol. Note that in [START_REF] Anirudh | Elastic functional coding of human actions: From vector-fields to latent variables[END_REF], results were averaged over all 242 possible combinations. However, our average accuracy is lower than other approaches following both protocols on this dataset (around 3.5% in the first and 0.62% in the second). Here, it is important to mention that data provided in MSR-Action 3D are noisy [START_REF] Lo | 3d skeleton-based human action classification[END_REF]. As a consequence, using Bi-LSTM without any additional processing step to handle the noise (e.g., FTP) could -75 -

Chapitre 3. Facial Expression and Action Recognition with Sparse Representations

not achieve state-of-the-art results on this dataset.

Comparison to State-of-the-art We discuss our results with respect to recent non-Riemannian approaches. In all datasets, our approach achieved competitive results.

Florence3D-Action -On this dataset, our method outperforms other methods using Bi-LSTM in the case of LOAO protocol, as shown in Table 3.1. However, using the second protocol, it is 2.19% lower than [START_REF] Koniusz | Tensor representations via kernel linearization for action recognition from 3d skeletons[END_REF]. The authors of [START_REF] Koniusz | Tensor representations via kernel linearization for action recognition from 3d skeletons[END_REF] UTKinect -Following the LOSO setting, our approach achieves the best recognition rate, yielding an improvement of 2.49% compared to the method of [START_REF] Liu | Spatio-temporal lstm with trust gates for 3d human action recognition[END_REF], which is based on an extended version of LSTM. For the second protocol, our best result is competitive to the accuracy of 98.2% obtained in [START_REF] Koniusz | Tensor representations via kernel linearization for action recognition from 3d skeletons[END_REF]. Considering the main challenge of this dataset, i.e., variations in the view point, our approach confirms the importance of the invariance properties gained by adopting the Kendall's representation of shape, hence the relevance of the resulting functions of codes generated using the geometry of the manifold.

MSR-Action 3D -For the experimental setting of [START_REF] Li | Action recognition based on a bag of 3D points[END_REF], our best result is competitive to recent approaches. In particular, on AS3, we report the highest accuracy of 100%. This result shows the efficiency of our approach in recognizing complex actions, as AS3 was intended to group complex actions together. On AS1, we achieved one of the highest accuracies (95.87%).

However, our result on AS2 is about 8.9% lower than state-of-the-art best result. This shows that our approach performs less well when recognizing similar actions, as AS2 was intended to group similar actions together. Although our best result is slightly higher than [START_REF] Koniusz | Tensor representations via kernel linearization for action recognition from 3d skeletons[END_REF], it is lower than the same method when following the experimental setting of [START_REF] Wang | Mining actionlet ensemble for action recognition with depth cameras[END_REF]. This shows that our approach performs better in recognition problems with less classes.

Table 3.2 -Overall recognition accuracy (%) on NTU-RGB+D following the X-sub and X-view protocols. In the first column: (R) : Riemannian approaches; with the exception of [START_REF] Zhang | View adaptive recurrent neural networks for high performance human action recognition from skeleton data[END_REF]. Knowing that we also used an RNN-based model (Bi-LSTM)

for temporal modeling and classification, this shows the efficiency of our action modeling.

In fact, sparse features obtained after SCDL in Kendall's shape space are remarkably more discriminative than the original data. In order to have a better insight into their corresponding data distributions, we used the t-distributed stochastic neighbor embedding (t-SNE)1 to visualize original data and SCDL features. From Fig. 3.5, we can observe that the SCDL features are better clustered than the original data in terms of class labels (colors in the figure). Besides, it is worth noting that SCDL is an efficient denoising tool, which is an important advantage when dealing with the often-noisy skeletons extracted with the Kinect sensor.

Comparison to extrinsic SCDL

To further evaluate the strength of the proposed intrinsic approach in the context of 3D action recognition, we compare it to extrinsic SCDL. Recall that instead of coding on tangent spaces, the extrinsic approach tends to embed the manifold-valued data into Hilbert spaces which are higher dimensional vector spaces where linear coding becomes possible. The main difficulty here arises from the fact that this embedding relies on a kernel function which, according to Mercer's theorem, should be positive definite. For 2D shapes, we presented in Section 2.2.3.2 a positive definite kernel. However, to the best of our knowledge, the existence of such a kernel has not been proved in the literature for 3D shapes. As a remedy, we adapted the extrinsic SCDL formulation by applying the Procrustes Gaussian kernel defined in Section 2.2.3.2, in which we also adapted the full Procrustes distance to 3D shapes as d F P ( Z1 , Z2 ) = sin(θ) (see Section 4.2.1 of [START_REF] Ian | Statistical Shape Analysis: With Applications in R[END_REF]) (θ is the geodesic distance defined in Section 2.2.3). Note that the kernel function relies on a parameter σ. Experimentally, we checked the positive definiteness of the adapted kernel and found out that it is only positive definite for some values of σ. We empirically chose 0.1 for Florence3D, 0.2 for UTKinect, and 0.5 for MSR-Action 3D, as to have valid positive definite kernels. Results reported in Table 3.3 show superiority of the intrinsic method. We argue that this difference comes from the fact that for the 3D case, the positive definiteness constraint on the kernel function reduced the valid space of the kernel parameter σ. Hence, intrinsic SCDL is in this case a better coding solution. In contrast, for the 2D case where we possess a PD kernel, we will show in the next section that extrinsic SCDL is more efficient in 2D recognition tasks.

2D Facial Expression Recognition

In this application, we extract 49 facial landmarks from human faces in 2D and with high accuracy using a state-of-the-art facial landmark detector [START_REF] Asthana | Incremental face alignment in the wild[END_REF]. We first represent the sequences of landmarks as trajectories in the Kendall's shape space. Extrinsic SCDL is then applied to produce sparse time-series that are finally classified in vector space. We evaluate this approach on two different 2D facial expression recognition tasks, the macro and micro.

Macro-Expression Recognition

The task here is to recognize the basic macro emotions, e.g., fear, surprise, happiness, etc.

To this end, we applied our approach on two commonly-used datasets namely the Cohn-Kanade Extended dataset and the Oulu-CASIA dataset. Our obtained results are then discussed with respect to state-of-the-art approaches as well as to intrinsic SCDL. For both datasets, we followed the commonly-used experimental setting in [START_REF] Elaiwat | A spatio-temporal rbm-based model for facial expression recognition[END_REF][START_REF] Jung | Joint fine-tuning in deep neural networks for facial expression recognition[END_REF][START_REF] Liu | Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition[END_REF][START_REF] Zhong | Learning active facial patches for expression analysis[END_REF] consisting on a ten-fold cross validation.

Cohn-Kanade Extended (CK+) dataset [START_REF] Lucey | The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotionspecified expression[END_REF] consists of 327 image sequences performed by 118 subjects with seven emotion labels: anger, contempt, disgust, fear, happiness, sadness, and surprise. Each sequence contains the two first temporal phases of the expression, i.e., neutral and onset (with apex frames).

Oulu-CASIA dataset [START_REF] Valstar | Induced disgust, happiness and surprise: an addition to the mmi facial expression database[END_REF] includes 480 image sequences performed by 80 subjects. They are labeled with one of the six basic emotions (those in CK+, except the contempt). Each sequence begins with a neutral facial expression and ends with the expression apex.

Results and discussions Table 3.4 gives an overview of the obtained results on both datasets. Overall, our approach achieved competitive results compared to the literature. For instance, our best result on CK+ (obtained with Bi-LSTM) is by 1.52% lower than the best state-of-the-art result obtained by the method of [START_REF] Jung | Joint fine-tuning in deep neural networks for facial expression recognition[END_REF]. The latter is based on two neural -80 -network architectures trained on image videos and facial landmark sequences. However, when using only the landmark architecture (DTGN), our approach obtained a higher accuracy.

Similarly, on Oulu-CASIA, our best result is lower than DTAGN and higher than DTGN. On the other hand, the method of [START_REF] Kacem | A novel space-time representation on the positive semidefinite cone for facial expression recognition[END_REF] achieved a better performance on both datasets compared to our method. Comparing the confusion matrices (see Table 3.5), the same method seems to better recognize the sadness expression while our method is clearly more efficient in recognizing the contempt expression. This will be further discussed later on. From Fig. 3.6

and the confusion matrices in Tables 3.6 and 3.5, we can observe that the two expressions:

happiness and surprise are well recognized in the two datasets while the main confusions happened in the two expressions: fear and sadness, conforming to state-of-the-art results [START_REF] Jung | Joint fine-tuning in deep neural networks for facial expression recognition[END_REF][START_REF] Kacem | A novel space-time representation on the positive semidefinite cone for facial expression recognition[END_REF]. Besides, we highlight the superiority of extrinsic SCDL compared to intrinsic SCDL.

The first is performed in RKHS which is a higher dimensional vector space. This helps capturing complex patterns in facial expressions and identifying subtle differences between similar expressions. For instance, an interesting observation could be seen for the contempt expression. As stated in [START_REF] Lucey | The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotionspecified expression[END_REF], the latter is quite subtle and it gets easily confused with other, strong emotions. For this expression, the recognition accuracy obtained with intrinsic SCDL is 55%, compared to 90% obtained with extrinsic SCDL, as shown in Figure 3.6. We argue that this remarkable improvement comes from the mapping to RKHS for the same reasons mentioned above. This observation has pushed us to further evaluate the performance of our approach in the task of micro-expression recognition.

Micro-Expression Recognition

Micro expressions are brief facial movements characterized by short duration, involuntariness and subtle intensity. We argue that to recognize them, in contrast to macro-expressions, we are more interested in detecting subtle shape changes along a sequence. To this end, we applied the extrinsic SCDL framework as in macro-expression recognition, and to further detect the subtle deformations, we computed displacement vectors as the difference between -81 - Recall that previous methods that tackled the problem of micro-expression recognition are appearance-based (i.e., using texture images) and to our knowledge, only [START_REF] Yoon Choi | Recognizing fine facial microexpressions using two-dimensional landmark feature[END_REF] has studied the problem using 2D facial landmarks. However, their approach was only evaluated on a synthesized dataset produced from CK+ (macro) videos, by selecting the three first frames of an expression, then interpolating between them. For this reason, we compare our results with respect to appearance-based methods, as shown in Table 3.7.

We point out the recognition accuracy of 64.62% achieved by our method outperforming state-of-the-art approaches, with the exception of [START_REF] Huang | Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[END_REF]. This shows the effectiveness of the adopted extrinsic SCDL in detecting subtle deformations from 2D landmarks, without any apperance-based information as other approaches in the literature.

Compared to the intrinsic approach, it is clear from Table 3.7 that the extrinsic SCDL method is better in recognizing micro-expressions. Recall that the use of extrinsic SCDL to tackle the problem of micro-expression recognition was driven by its good performance -83 -Table 3.7 -Recognition accuracy on CASME II dataset and comparison with state-ofthe-art methods. In the first column: (A) : Appearance-based approaches. (R) : Riemannian approaches. Last row: our approach.

Method Accuracy (%) (A) STCLQP [START_REF] Huang | Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[END_REF] 58.39 (A) CNN [START_REF] Breuer | A deep learning perspective on the origin of facial expressions[END_REF] 59.47 (A) CNN (LSTM) [START_REF] Kim | Micro-expression recognition with expression-state constrained spatio-temporal feature representations[END_REF] 60.98 (A) LBP-TOP, HOOF [START_REF] Hao Zheng | A relaxed k-svd algorithm for spontaneous micro-expression recognition[END_REF] 63.25 (A) Optical Strain [START_REF] Liong | Spontaneous subtle expression detection and recognition based on facial strain[END_REF] 63.41 (A) DiSTLBP-IIP [START_REF] Huang | Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[END_REF] 64.78

(R) Intrinsic SCDL (SVM) 43.65 (R) Extrinsic SCDL (SVM) 64.62
in recognizing the contempt emotion, in the CK+ dataset which is characterized by subtle changes along the expression. The obtained results on CASME II hence supports our previous claims.

Ablation study

We examine the effectiveness of the proposed Kendall SCDL schemes by performing several baseline experiments on different datasets.

A. Kendall's shape representation -We evaluate the necessity of the Kendall's shape projection. To this end, we perform temporal modeling and classification on raw data, after a scale and translation normalization, against their application on Kendall SCDL features. On NTU-RGB+D, we applied Bi-LSTM while on Florence 3D, MSR Action 3D and UTKinect, we applied the pipeline DTW-FTP-SVM. Performances are reported in the second and fourth rows of Table 3.8. In all datasets, improvements are remarkably gained with the Kendall's space projection. This is clearly seen in particular on the large scale NTU-RBD+D dataset which presents different view-variations and where the improvement is more than 26%.

B. Nonlinear SCDL -In this experiment, we evaluate the importance of the nonlinear formulation of SCDL that we applied on Kendall's space. For that, we compare it to the use of linear SCDL, i.e., by solving for Eq.2.2.2. Obtained results on the four action recognition C. Sparsity regularization -In this experiment, we evaluate the effect of the sparsity regularization parameter λ (in Eq. (2.4.1) and Eq. (2.4.4)) on recognition accuracies obtained using both of the adopted classifiers. To do so, we used half of a training set for learning the dictionary and training the classifiers and the other half for validation. The first graph of Fig. 3.7 shows the impact of increasing λ from 10 -4 to 1 at steps of 10 -2 . Further, we report the average sparsity percentage (i.e., number of non-zero codes divided by the total number of codes) for some values of λ to show the coherence of the obtained codes with the proposed theory. As expected, the sparsity percentage increases when increasing λ. We remark that the accuracy reached a maximum value at λ = 0.01 (37% of sparsity) and λ = 0.02 (49% of sparsity) for SVM and Bi-LSTM, respectively. Note that in all previous experiments, λ was chosen empirically so to correspond to these latter percentages of sparsity.

D. Dictionary structure -As described in Section 3.3.3, we build class-specific dictionaries. To show the relevance of this structure in the context of classification, we compare it to the case of using a global dictionary, e.g., when label(s are not taken into account. The obtained recognition accuracies using Bi-LSTM and following the LOAO setting are 94.48% and 91.53% for class-specific and global dictionary, respectively. These results clearly prove that the adopted structure is better in classifying actions.

E. Dictionary initialization -In this experiment, we evaluate the performance of our proposed initialization step based on Bayesian clustering of shapes and PGA. To this end, we -85 -0,0001 (16%) 0,001 (19%) 0,01 (37%) 0,1 (77%) 0.9 (85%) 80 . compare it to the case of random initialization, where atoms are randomly selected from the training set. We train a class-specific dictionary (for class tight lace in Florence3D dataset)

with the same training data in both cases. For the case of random initialization, we set the number of atoms N to 41 to be equal to that of our proposed initialization. Recall that in our approach, N is automatically inferred to avoid its empiric choice, especially as we build classspecific dictionaries. In Fig. 3.7, on the right graph, we plot the two corresponding dictionary learning objectives over iterations. As it is expected, the proposed initialization shows faster convergence, dividing the overall dictionary learning processing time by approximately two times, when taking into account the execution time of our initialization step.

F. Performance of Bi-LSTM -We compared average accuracies yielded by Bidirectional LSTM and a traditional LSTM. Following LOAO experimental setting, using Bi-LSTM shows an improvement of around 0.7% on Florence Action 3D and 1.2% on NTU-RGBD dataset, indicating the positive effect of learning both future and past contexts to recognize actions.

G. Evaluation on the facial landmark detector

The task of facial expression recognition from landmark data relies essentially on the accuracy of the landmark detector. In this -86 -experiment, we evaluate the performance of the landmark detector that we used in our experiments (i.e., Chehra [START_REF] Asthana | Incremental face alignment in the wild[END_REF]) by comparing it to the newly-released Openface2.0 [START_REF] Baltrusaitis | Openface 2.0: Facial behavior analysis toolkit[END_REF],

which gives the option of extracting either 49 or 68 landmarks. In Table 3.9, we report the classification accuracy obtained by applying the pipeline DTW+FTP+SVM on raw landmark data (after a simple scale and translation normalization). Results obtained on CK+ and Oulu-CASIA datasets clearly show a better performance using landmarks extracted with the Chehra detector.

Discussions

The Kendall's shape representation has proven the efficiency of adopting a view-invariant analysis of the given data. Because of the nonlinearity of the Kendall's manifold, intrinsic and extrinsic solutions of SCDL were comprehensively studied and compared. Regarding the extrinsic solution, the advantage of embedding data from the Kendall's shape space to RKHS is twofold. First, the latter is vector space, thus it enables the extension of linear SCDL to the nonlinear Kendall's space. Second, embedding a lower dimensional space in a higher dimensional one gives a richer representation of the data and helps extracting complex patterns. However, to define a valid RKHS, the kernel function must be positive definite according to Mercer's theorem. On one hand, for the 2D Kendall's space, we have used the Procrustes Gaussian Kernel which is positive definite and shown that for the task of 2D macro facial expression recognition, extrinsic SCDL performs better than intrinsic SCDL. We argue that this is due to the kernel embedding. For instance, we highlight the clear improvement in recognizing the contempt emotion in the CK+ dataset. The latter is -87 -characterized with subtle deformations that are well captured using the extrinsic approach.

This has drove us to evaluate it on the task of 2D micro-expression recognition where the shape deformations along expressions are known to be subtle as well. As expected, the performance of extrinsic SCDL was promising. On the other hand, for the 3D Kendall's space, a positive definite kernel function has not been proposed in the literature. Nevertheless, adapting the PGk to 3D shapes prevented us from exploring the whole space of σ as in this case, this kernel is positive definite for only certain value of this parameter. As a consequence, the performance of extrinsic SCDL in the 3D Kendall's space can be hindered since the quality of the produced codes depends on the value of σ. We argue that this is the main reason behind the better performance obtained using intrinsic SCDL for the task of 3D action recognition. Besides, intrinsic sparse coding of a shape is performed on its attached tangent space, by mapping atoms into it. Compared to Riemannian approaches of the literature, this avoids the common drawback of mapping points to a common tangent space at a reference point which may introduce distortions.

Conclusion

In this chapter, we have used the sparse representations that we presented in the previous chapter to classify 3D actions and 2D micro and macro facial expressions. We have used two temporal modeling and classification schemes on top of the obtained sparse time-series: a deep learning framework based on Bi-LSTM and a pipeline of DTW-FTP-SVM. We have conducted extensive experiments on seven commonly-used datasets and showed that our obtained results are competitive to state-of-the-art. We have compared the two adopted temporal modeling and classification pipelines and discussed the obtained results for different datasets. Further, we presented a comprehensive comparative study on the use of intrinsic and extrinsic SCDL approaches by providing an answer to the question: "Depending on the nature of the data (i.e body or face) and its dimension (i.e 2D or 3D), when and which

Chapitre 4

Pose-guided Human Video Generation with Sparse Representations

Motivation

Due to the emergence of Generative Adversarial Networks, the task of human video generation is attracting increasing attention in recent years as it is suitable for several vision problems. For instance, it can serve as a data augmentation tool which remarkably relieves the burden of manual annotations and thereby contributes to the development of various video understanding tasks such as action and activity recognition. On the other hand, human video synthesis allows for many human-centric applications such as avatar animation. Problems related to realistic image generation with GANs has already shown impressive results [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF][START_REF] Odena | Conditional image synthesis with auxiliary classifier gans[END_REF]. However, the extension from generating images to generating videos turns out to be a highly challenging task with the introduction of the temporal dimension. Considering this fact, a typical generative model needs to learn the plausible -91 -physical motion models of objects in addition to learning their appearance models. To this end, some approaches in the literature opted for a disentangled solution by first generating plausible motion then synthesizing coherent appearances. For instance, Yang et al. [START_REF] Yang | Pose guided human video generation[END_REF] proposed a two stage approach. In the first model [START_REF] Zhu | Progressive pose attention transfer for person image generation[END_REF] to map each generated pose to an image. We differentiate our work as an extension of [START_REF] Zhu | Progressive pose attention transfer for person image generation[END_REF] to video generation, while our main novelty consists on the shapeGAN framework.

-Application of the proposed frameworks to: data augmentation for landmark-based action recognition and human video generation.

Background and related work

Generative Adversarial Networks

In recent years, GANs [START_REF] Goodfellow | Generative adversarial nets[END_REF] have led to breakthroughs in tasks related to realistic image generation. Examples include image synthesis [START_REF] Odena | Conditional image synthesis with auxiliary classifier gans[END_REF] which infers multiple possible outputs for a given input, super-resolution [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF], and image to image translation [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF]. The original GAN framework, as demonstrated in 1. Non-convergence: the generator and the discriminator fail to reach an equilibrium.

2. Mode collapse: the generator model generates similar samples given different inputs.

3. Quick convergence of the discriminator loss to zero: the optimal discriminator will be unable to provide sufficient gradient to update the generator.

To address these issues, several authors suggested heuristic approaches to improve GANs training. For a description of these tricks, we refer the reader to [START_REF] Metz | Unrolled generative adversarial networks[END_REF][START_REF] Salimans | Improved techniques for training gans[END_REF]. More recently, a variant of GANs, called the Wasserstein GAN (WGAN) has shown to be more stable to train [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF]. Different from the above-described vanilla GAN, WGAN optimizes the Wasserstein distance using Kantorovich Rubinstein duality, instead of optimizing the Jensen-Shannon divergence in the standard GAN. However, WGANs can sometimes generate low-quality samples or fail to converge in some settings due to the use of weight clipping to enforce a Lipschitz constraint on the critic. This problem is alleviated in [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] by the use of a gradient penalty. An overview of how GANs and their variants work can be found in [START_REF] Hong | How generative adversarial networks and their variants work: An overview[END_REF]. In this chapter, we will use the improved version of WGAN [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] to train our model.

Human video generation

Early video generation approaches attempted to generate videos directly in pixel space using GANs or Variational Autoencoders (VAEs) [START_REF] Saito | Temporal generative adversarial nets with singular value clipping[END_REF][START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF][START_REF] Vondrick | Generating videos with scene dynamics[END_REF]. By doing so, they model all the structure and scene dynamics at once which may yield uninterpretable results. Recently, many works proposed to guide the video generation using pose sequences making it a twostep approach. In the first, human landmarks are exploited to generate novel pose sequences which, along with an input image, are used to generate frame videos. For instance, Walker et al. [START_REF] Walker | The pose knows: Video forecasting by generating pose futures[END_REF] proposed to guide the video forecasting with pose sequences, breaking the video generation problem into two steps. First, they exploit the effectiveness of landmarks to model human dynamics and used a VAE to generate possible future poses. Then, the latter are used to conditioning a GAN to predict future frames of a video in pixel space. The same idea was applied in [START_REF] Yang | Pose guided human video generation[END_REF] to synthesize human videos with a two-stage pose-guided video generation an adversarial way. Based on the same idea, the authors in [START_REF] Siarohin | Deformable gans for pose-based human image generation[END_REF] introduced deformable skip connections in the generator of the GAN to deal with pixel-to-pixel misalignments caused by the pose differences. In addition, they proposed a nearest-neighbour loss instead of the common L 1 and L 2 losses in order to match the details of the generated image with the target image. More recently, a progressive pose attention transfer was proposed in [START_REF] Zhu | Progressive pose attention transfer for person image generation[END_REF] to generate the person image progressively. To do so, the generator of the network comprises a sequence of Pose-Attentional Transfer Blocks that each transfers certain regions it attends to. This yields more realistic images with better appearance and shape consistency than previous works. given an input image I c of a person, each pose of a generated sequence q is mapped to I c giving rise to a new sequence of images (i.e., video). In the following, we give more details of each step.

The proposed approach

shapeGAN: a pose sequence generation framework

Human landmarks have shown to be effective in several vision tasks including action recognition, as shown in the previous chapter of this thesis. Inspired by its capability to encode human motion dynamics, we aim to use human skeletons to synthesize novel motion dynamics. However, the latter are often characterized with high variability, partly due to the view variations. As a consequence, it is difficult for a generative model to identify shape and motion patterns directly from raw data. Considering this problem, we propose to encode landmark data to a more compact representation that lie to a lower-dimensional space by adopting the sparse coding and dictionary learning scheme that we presented in Chapter 2.

In the context of pose sequence generation, this approach brings three main advantages:

1) Landmark configurations are represented in the shape manifold which discards the view variations within the data (i.e., rotations, translations, and global scaling of configurations), hence keeping only information about the shape and its dynamics; 2) SCDL is performed in the shape manifold and yields a vector space representation while accounting to the nonlinear nature of the data. Hence, the resulting representation is compact and it is expected to be more suitable to train a generative model than the original data; 3) This coding technique enables to recover the latent variables back to the shape manifold. Thereby, we propose to design a generative model that synthesizes new samples in the latent space, then reconstruct them to obtain skeletal sequences in the shape manifold. a human action, we first extract a skeleton from each video frame using a state-of-the-art detector [START_REF] Cao | Realtime multi-person 2d pose estimation using part affinity fields[END_REF] where each skeleton is represented by 18 body landmarks in 2D. Then, training This allows to generate new samples which are then transformed to skeletal sequences in the shape manifold using the weighted intrinsic mean algorithm as described in Algorithm 4 in Chapter 2. This reconstruction procedure is performed with respect to the pretrained dictionary which insures that the obtained samples lie in the space of skeletons avoiding any noisy or corrupted poses. Moreover, the trained GAN model shows its capability to generate smoothly varying time-series which characterizes human actions. More properties of the generated samples will be investigated in Section 4.4.2.

shapeVGAN: a pose-guided video generation framework

Once the shapeGAN model is trained, we are able to generate novel pose sequences that present new variations in terms of style and temporal evolution of human actions. The -98 -

Experimental evaluation

In this section, we conduct extensive experiments to evaluate the efficiency of the proposed frameworks. We first evaluate the performance of the proposed shapeGAN model by applying it in a data augmentation task. For that, we used the Florence 3D action dataset [START_REF] Seidenari | Recognizing actions from depth cameras as weakly aligned multi-part bag-ofposes[END_REF] that we presented in Chapter 3. Later on, we evaluate the shapeVGAN model using data collected from the Weizmann Action database [START_REF] Blank | Actions as space-time shapes[END_REF] which contains 81 videos of 9 people performing 9 actions, including jumping and waving-hands.

-100 -

Implementation details

shapeGAN implementation Recall that shapeGAN has an encoder-GAN architecture.

Regarding the encoder (SCDL), for both datasets, we performed a temporal re-sampling of pose sequences to have equal length l = 32 using Algorithm 5. For each dataset, we constructed a dictionary comprised of k = 32 atoms using the k-means approach as presented in Chapter 2. We set the sparsity parameter λ to 0.1 to perform sparse coding. For a given pose sequence, the sparse coding step would result in a matrix of dimension 32 × 32 which can be seen as a one-channel image that we transform to a tensor of dimension 1 × 32 × 32.

Our GAN implementation is based on the improved Wasserstein variant [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF]. It is built upon the popular Pytorch framework. We used batches of size 4 and we trained our model for 10 5 epochs. We set the dimension of the input random vector z of the generator to 100.

We design the architecture of our generator as follows. The input latent variable z first goes through two residual blocks of linear and LeakyRelu [START_REF] Andrew L Maas | Rectifier nonlinearities improve neural network acoustic models[END_REF] layers, followed by a linear transformation. The obtained vector after these blocks is of dimension sequence length (T )

× feature dimension (dim). It can be seen as T sub-vectors each of dimension dim. Each sub-vector finally goes through a softmax layer which maps each component to the range (0, 1) where the components add up to 1. This output layer is important since it guarantees that the generated sparse code vectors have the same structure as the real ones.

ShapeVGAN implementation We used the Weizmann dataset to validate this model for which we scaled each image frame to 180 × 140. Our implementation is based on that of [START_REF] Zhu | Progressive pose attention transfer for person image generation[END_REF] following the same parameter setting. Similarly, with each sample, the discriminator will receive the same information as input.

Another important criterion that we want to evaluate is the generator ability to create a temporal variability. Recall that in our model, we do not perform any temporal modeling of the data. Nevertheless, our model still captures the temporal variability within the data and is capable of generating new ones. To show our previous claim, in the generated set. Note that for each, the X-axis represents the time frame while the Y-axis represents the codes (or weights) which correspond to the activation of the dictionary atoms.

Comparing these two feature maps, we can notice similar atoms activation in the two while the main difference resides in the temporal shift between them.

The effect of sparsity In our shapeGAN framework, sparsity plays a crucial role in making the data representation more compact which facilitates the task of the generator to learn the training data distribution. In this experiment, we evaluate this property by performing different experiments with different values of the sparsity regularization parameter λ. In Figure 4.9, we present some qualitative results. The training samples are collected from the class hand wave of the Florence 3D dataset. Note that this class presents a considerable variation since actors can perform the action either with the left or the right hand, see the first raw of Figure 4.9. We trained three shapeGAN models with the values 0, 10 -4 and 10 -2 of the parameter λ, respectively. Recall that λ controls the amount of sparsity in the sparse coding step. The obtained results show that the more λ is higher, the more the generated samples are realistic. We can notice that for values 0 and 10 -4 of λ, the generated samples combine the two movements: raising the left hand and raising the right hand in the same action, and even mix both of them by raising the two hands, which do not resemble to real samples. This phenomenon does not occur for relatively higher value of λ.

Quantitative evaluation

To quantitatively evaluate our shapeGAN model, we propose to augment an action recognition dataset, i.e. the Florence 3D dataset, and evaluate the classification performance when using: 1) Original training data; and 2) Training data augmented with an increasing number of generated data. This procedure was proposed in [START_REF] Shmelkov | How good is my gan[END_REF] as a metric to evaluate a GAN's performance.

We use the same action recognition method that we proposed in Chapter 3. For temporal -106 - have seen increasing improvements when increasing the amount of training data. Note that the classification performance reaches saturation when multiplying the amount of training samples by 9. These results demonstrate the efficiency of the proposed shapeGAN model which can be useful as a data augmentation tool for different recognition tasks and potentially for other vision tasks which relieves the burden of manual annotations.

Human video generation

After evaluating the performance of the proposed shapeGAN framework which showed promising results in the generation of human pose sequences, we aim to evaluate the final step of video generation. To this end, in what follows, we present some qualitative results achieved with the shapeVGAN framework. Recall that the generated pose sequences from the first step are used to guide the generation of video frames.

In this condition image, we transfer it to the poses of the generated sequences to obtain the final videos. Visually, we can clearly observe that our obtained results are realistic in terms of appearance and temporal consistency since the human shape in a generated video evolves smoothly over time. This demonstrates the efficiency of the pose transfer approach as well as the advantage of guiding the video generation process with pose sequences. Besides, if we compare the two generated videos frame by frame, we can notice that the way of performing the same action is different which again demonstrates the variability that has been created in the pose motion thanks to the shapeGAN framework.

To further evaluate our method, we took a generated pose sequence and transferred it to two images of two different actors aiming to obtain two videos of actors with the motion.

Figure 4.12 presents the obtained results where we can clearly see that these actors perform the same action with a similar motion. This demonstrates the ability of our framework in preserving the identity of the person (i.e., visual appearance, size of arms, etc.) in the condition image while modifying its motion. Here, it is important to mention that since shapeGAN generates pose sequences in the shape manifold, information regarding scale and the condition image by applying the procrustes algorithm [START_REF] Kendall | Shape manifolds, Procrustean metrics, and complex projective spaces[END_REF].

Comparison to a state-of-the-art method To demonstrate the performance of the presented shapeVGAN framework, we compare it to a state-of-the-art method namely MoCoGAN [START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF]. This method generates a video by mapping a sequence of random vectors to a sequence of video frames where the motion and content are decomposed in the generation process. In contrast to our approach which guides the video generation with pose sequences, MoCoGAN generates videos directly in pixel space.

We train both models using the same videos collected from the Weizmann database.

In Figure 4.13, we show some qualitative results that are randomly taken from generated samples. We can clearly observe that video frames obtained with our approach are more realistic than those obtained with MoCoGAN. In fact, the latter contain some blurry images (e.g. images in last raw of Figure 4.13) as well as some content and motion artifacts (noise).

In contrast, shapeVGAN is able to keep more details in the video images and the person movements evolve more smoothly and naturally over time. This shows the relevance of the proposed pose-guided generation framework.

-109 -

Limitations and ongoing study

Since filtering out translations, rotation and scaling from configurations is a necessary step in our shapeVGAN encoder (to perform SCDL in the shape manifold), these transformations are still important to recover the location of the generated poses in an image, as explained in the previous paragraph. In other words, imagine a video where a person is walking from the left to the right of the scene, its corresponding pose represented in the shape manifold would result in a centered skeleton moving locally in the same position.

As a consequence, in addition to generating pose sequences in the shape manifold, it is necessary to generate the filtered transformations (translations, rotation and scaling) then add them to the generated pose sequences as a post-processing step. As an ongoing study, we aim to encode these transformations separately and feed them to a second generator that learns to generate new translations, rotations and scaling of poses. Since these information are correlated with the pose dynamics, the two generators of our model have to be learned jointly to guarantee a certain coherence in the final pose sequences.

On the other hand, we are now capable of generating pose sequences without any assumption on the class labels. Hence, for data augmentation, we have to train a model for each action class. However, it would be more interesting to condition the shapeGAN model on the class label by using a conditional GAN which showed to be efficient in different image generation tasks. This is part of our ongoing work.

Conclusion

In this chapter, we explored the problem of human motion generation with GANs and

Conclusions and Perspectives

In this dissertation, we have presented a novel framework to model landmark-based human actions and facial expressions. We have demonstrated that the proposed methods are competitive or outperform the state-of-the-art on various and challenging datasets in two recognition tasks: 3D action recognition, 2D micro and macro facial expression recognition.

We also demonstrated that the proposed action modeling can be efficiently integrated in a generative model in order to synthesize novel human actions and videos. We conclude our work pointing out the main contributions in Section 5.1 and their limitations in Section 5.2. Finally, we discuss future perspectives in Section 5.3, indicating interesting directions for future work in this field.

Main contributions

We have proposed sparse representations for landmark-based human actions and facial expressions. We represented sequences of human landmarks as trajectories in the shape manifold to allow for a view-invariant analysis. Trajectories are encoded using Riemannian versions of sparse coding and dictionary learning. This allows to compensate the lack of vector space structure in the shape manifold and yields Euclidean time-series with suitable -113 -computational properties, e.g. sparsity and vector space structure.

Application of the presented frameworks to 3D human action recognition and 2D facial expression recognition We have applied the proposed representations in different classification problems, i.e. 3D action recognition, 2D micro and macro facial expression recognition. We have adopted two temporal modeling and classification schemes, a deep learning framework based on LSTM and a classical pipeline of methods designed for Euclidean time-series. We have conducted extensive evaluation on seven commonly-used datasets that present different challenges which showed the competitiveness of the proposed frameworks to state-of-the-art.

Evaluation and comparison of extrinsic and intrinsic coding approaches in the shape manifold We have explored two paradigms of sparse coding and dictionary learning in the shape manifold, intrinsic and extrinsic.

The first allows to perform coding on the manifold tangent spaces which solves for a commonly encountered problem in the literature of Riemannian methods, i.e. trajectories distortion when mapped to a reference tangent space. It also enables an easy reconstruction back to the original manifold which allows the visual exploration of the latent variables.

The second performs coding in Hilbert space after mapping the manifold-valued data using a kernel embedding which gives a richer representation of the data and helps identifying complex patterns.

We have evaluated and compared these two paradigms in the context of recognition tasks that we addressed. We showed that the extrinsic approach is more efficient to represent 2D trajectories while the intrinsic one is more suitable to encode 3D trajectories.

A generative framework for human motion synthesis We have proposed a generative model to synthesize human pose sequences (namely shapeGAN). shapeGAN first encodes

Limitations

training pose sequences using nonlinear SCDL and uses the obtained latent samples to train a generative adversarial network. The generated samples showed to be realistic while representing new styles of actions. We have used this model to augment a state-of-the-art action dataset which improved the action recognition accuracy.

We have used the generated pose sequences with shapeGAN to guide the generation of human videos. Given an input person image, each pose of a generated sequence is transferred to an image using a state-of-the-art pose transfer framework. This allowed us to synthesize new videos.

Limitations

-The presented techniques assume that human landmarks are available and are sufficiently accurate since all the datasets that we used are collected in controlled lab environments. However, the estimation of landmarks could be more challenging but possible in an uncontrolled setting.

-While the use of an extrinsic approach to represent 2D trajectories in the shape manifold showed to be more efficient for classification than the intrinsic approach, its application in the 3D shape manifold is not usually suitable. In fact, this requires a positive definite kernel to allow mapping manifold-valued data to RKHS. Such valid kernels are only available in the case of 2D shapes.

-The main limitation of the presented human motion generation framework is its disability to generate global shape displacements. In fact, encoding data in the shape manifold discards the global transformations (i.e. translations, rotations, scaling of configurations).

-115 -

Future work

Several aspects can be investigated in future works in order to improve the proposed approaches and further evaluate their performance.

-The presented extrinsic approach to code shape trajectories is only suitable for the 2D shape manifold where a positive definite kernel exists in the literature. This is not the case for the 3D shape manifold where our extension of the Procrustes Gaussian kernel is not always valid (see Section 3.4.1.4). Therefore, we would like to further study the existence of positive definite kernels in the 3D shape manifold.

-The proposed facial expression framework has been evaluated on datasets that were collected in controlled environments and do not present considerable view-variations (e.g. the Cohn-Kanade and the Oulu-Casia datasets). Hence, we would like to evaluate our approach on more challenging datasets such as the AFEW database [START_REF] Dhall | Collecting large, richly annotated facial-expression databases from movies[END_REF] where the data were collected from movies showing close-to-real-world conditions, which simulates the spontaneous expressions in uncontrolled environment. In addition, the view variations for 2D landmarks yield projective transformations which are more complex to filter out, thereby it would be interesting to investigate this problem.

-The presented generative framework, i.e. shapeGAN, is only capable to generate local pose deformations without controlling the skeleton displacement (i.e. location of a pose in the video). We would like to use the transformations that are filtered out in the first step (i.e. rotation, translation, and scale) as input to a second generator in order to generate new ones. The generated samples could then be added up to the corresponding samples generated from the first generator. The two generators have to be trained jointly in order to learn the relations between the two information (i.e. local and global pose deformations).

-We have applied shapeVGAN to generate human actions on the Weizmann dataset only. Hence, we would like to evaluate it on other, more challenging datasets presenting -116 -more action variations such as the UCF-101 dataset [START_REF] Soomro | Ucf101: A dataset of 101 human actions classes from videos in the wild[END_REF]. Our next goal is to also apply the shapeVGAN framework on other problems such as the generation of facial expressions. Moreover, we exploited shapeGAN to augment the Florence3D dataset and we would like to augment other human actions and facial expressions datasets and evaluate recognition performances using different state-of-the-art approaches.

-117 -

1 Introduction 1 . 1 7 - 10 -

 111710 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Scientific challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Main contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Organisation of the dissertation . . . . . . . . . . . . . . . . . . . . . . . . . .2 Coding Human Motion Trajectories in the Shape Manifold 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Background and definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Sparse representations . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Riemannian geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Kendall's shape space . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3.1 Geometric tools on the manifold . . . . . . . . . . . . . . . . 2.2.3.2 Hilbert space embedding of the manifold . . . . . . . . . . . 2.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Riemannian sparse coding and dictionary learning . . . . . . . . . . . 2.3.2 Trajectory representations on Riemannian manifolds . . . . . . . . . . -TABLE OF CONTENTS 4.4.2.2 Quantitative evaluation . . . . . . . . . . . . . . . . . . . . .4.4.3 Human video generation . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Limitations and ongoing study . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conclusions and Perspectives 5.1 Main contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliographie -List of figures 1.1 Examples of video surveillance cameras and a sample monitoring center. . . . 1.2 Outline contours of a walking (right) and a running (left) subject and the corresponding moving light displays attached on the body [56]. . . . . . . . . 1.3 Skeleton estimation from: (a) depth images [84], (b) Motion Capture systems, (c) RGB images [17]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Depth cameras: ASUS Xtion PRO LIVE (left) and the Microsoft Kinect (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Examples of facial landmark detection in: (a) 3D [8] and (b) 2D [7]. . . . . . . 1.6 (a) Samples taken from [94] showing body pose with a variety in camera views. (b) Facial images with different poses. . . . . . . . . . . . . . . . . . . 2.1 Schematic sparse model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. 3 Z 3 . 7

 337 is a data point on the manifold M and the red circles represent the dictionary atoms. The linear approximation Ẑ with the atoms may be out of the manifold M. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . -11 -LIST OF FIGURES 2.4 Example results of the proposed framework. An input sequence is transformed to a smoothly-evolving sparse time-series. The X-axis represents the time frame. (A) 3D skeletal sequence. (B) 2D facial landmark sequence. . . . . . . 2.5 Illustration of the adopted solutions to overcome the nonlinearity of the shape manifold. (A) The intrinsic approach maps the data on the manifold to tangent spaces using the logarithm map opertor. (B) The extrinsic approach embeds the manifold-valued data to RKHS by computing the inner product matrix using a positive definite kernel function. . . . . . . . . . . . . . . . . . 2.6 Pictorial of the sparse coding approach on the pre-shape space C. The approximation of x ∈ C could be viewed as a weighted intrinsic mean of the atoms of a dictionary D = {d i } N i=1 . . . . . . . . . . . . . . . . . . . . . . . 2.7 Pictorial of the proposed clustering approach. Landmark configurations (left) are mapped from the the shape manifold to RKHS by computing the inner product matrix from the data (Middle). Bayesian clustering is then applied on this matrix to construct the final clusters (right) whose number is automatically inferred. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Given the pre-trained dictionary D, skeletal trajectories in the shape manifold can be reconstructed from the space of sparse codes using the weighted intrinsic mean algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Schematic tangent space projections using our method compared to state-ofthe-art. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.10 Denoising of skeletal shapes using sparse coding. Abnormal skeletons are presented in the top left. Code vectors (middle) are obtained after sparse coding. They are then reconstructed with respect to the dictionary (right) to recover the abnormal skeletons. . . . . . . . . . . . . . . . . . . . . . . . . . . -12 -2.11 Mean shape computation on a set of skeletons. Left: Input 3D skeletons in the Kendall's shape space. Middle: Mean shape computed using the intrinsic mean algorithm. Right: Mean shape computed by first sparse coding the input skeletons, then computing the mean code and reconstruct it with the dictionary. 2.12 An example of linear interpolation between two latent variables: source (left) and target (right). Shapes in this figure are the result of mapping the interpolates from the latent space to the manifold. . . . . . . . . . . . . . . . 3.1 Overview of the proposed frameworks. Trajectories of 2D facial expressions (respectively 3D actions) are encoded using extrinsic (respectively intrinsic) SCDL in the Kendall's shape space. Temporal modeling and classification are then performed on the obtained time-series in vector space. . . . . . . . . . . 3.2 Overview of a typical landmark-based action/facial expression recognition approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Our categorisation of state-of-the-art approaches. Representations of 2D/3D sequences of landmarks (skeletons or faces) can be either hand-crafted or learned. The former representations can be categorised into: kernel-based, graphical models or Riemannian. The latter can be achieved using deep learning techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Confusion matrix on the Florence 3D dataset. . . . . . . . . . . . . . . . . . . 3.5 Visualization of 2-dimensional features of the NTU-RGB+D dataset. Left: original data. Right: the corresponding SCDL features. Each class is represented by a different color. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Recognition accuracy achieved for each emotion class in the CK+ (left) and the CASIA (rigth) datasets, and comparison between extrinsic and intrinsic SCDL approaches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Left: Accuracy when varying the sparsity regularization parameter λ (% values in the x-axis represent the average sparsity). Right: Dictionary learning objective over iterations for: (1) Random initialization; (2) Our proposed initialization based on Bayesian clustering and PGA. . . . . . . . . . . . . . . 4.1 Schema of the original GAN framework. The generator G takes a noise vector z as input and outputs generated images x. The discriminator D distinguishes x from real samples x. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Overview of the proposed shapeGAN framework. . . . . . . . . . . . . . . . . 4.3 Generator architecture of the progressive pose attention transfer method [130]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Pose to image transfer. (a): Condition image. (b): Condition pose. (c): Target pose. (d): Target image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Examples of generated pose sequences of actions with shapeGAN. From top to bottom are actions: hand wave, sit-down, and clap hands. . . . . . . . . . . 4.6 2D visualization of training samples (blue) and generated samples (red). . . . 4.7 2D visualization of data belonging to nine action classes of the Florence 3D dataset. Left: training samples. Right: training and generated samples. . . . . 4.8 Visualization of a training (left) and a generated (right) feature maps. . . . . 4.9 Examples of generated samples with shapeVGAN. Different values of the sparsity regularization parameter λ (in the sparse coding) were used in each experiment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.10 (a) Pose sequence generated after training a GAN directly on raw pose sequences. (b) Pose sequence generated with shapeGAN. . . . . . . . . . . . -14 -4.11 Examples of video frames of action wave hands. The first row represents a training video while the two others represent videos generated with our approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.12 Example of a generated pose sequence with shapeGAN (Top row). This sequence is used to guide the generation of the two videos (second and third row) given condition images (in red). . . . . . . . . . . . . . . . . . . . . . . . 109 4.13 Qualitative comparison of (a) Our results obtained with shapeVGAN. (b) Results obtained by training a state-of-the-art approach, i.e. MoCoGAN [104]. 112 -15 -

Chapitre 1 Introduction 1 . 1 Motivation

 111 In recent years, video cameras have become part of our everyday life with the availability of mobile phones, tablets, and digital photo cameras which all can create, store and share high quality videos. With great storage capacities and fast internet access, videos have become easily accessible. Accordingly, designing intelligent systems to understand their content is attracting more and more attention with the availability of powerful and affordable computers. Such intelligent systems are also needed to compensate the limited human capabilities of analyzing videos in an efficient way. In particular, human behavior understanding in videos is receiving a huge interest due its many potential applications.For instance, analyzing the motion of the human body and face plays a key role in several areas such as: Human-Computer Interaction and Entertainment, Visual Surveillance, Video Retrieval, and Healthcare.-Human-Computer Interaction and Entertainment: In computer graphics, video cameras are used for human-machine interaction since they represent a natural and intuitive way to communicate with a device. Therefore, the accurate recognition of -21 -human gestures and facial expressions can guarantee a more natural, less-restrictive and effective human-computer interfaces. In the entertainment industry, a new generation of games based on full body play such as dance and sports games have increased the appeal of gaming to family members of all ages[START_REF] Kong | Human action recognition and prediction: A survey[END_REF]. An example of popular video camera that enables accurate perception of human actions is the Microsoft Kinect sensor (see Figure1.4).

Figure 1 . 1 )

 11 Figure 1.1). Visual surveillance in dynamic scenes attempts to detect, recognize and track certain objects from image sequences, and more generally to understand and describe object behaviors including humans. The aim is to develop intelligent visual surveillance to replace the traditional passive video surveillance that is proving ineffective as the number of cameras exceeds the capability of human operators to monitor them [49]. Therefore, human action and identity recognition are at the core of video surveillance and monitoring systems.

Figure 1 . 1 -

 11 Figure 1.1 -Examples of video surveillance cameras and a sample monitoring center.

Figure 1 . 2 -Figure 1 . 3 - 24 -Figure 1 . 4 -

 12132414 Figure 1.2 -Outline contours of a walking (right) and a running (left) subject and the corresponding moving light displays attached on the body [56].

Figure 1 . 5 shows

 15 examples of 2D facial landmark tracking from RGB images where we can notice the ability to deal with profile faces and occlusion in some cases.

Figure 1 . 5 -

 15 Figure 1.5 -Examples of facial landmark detection in: (a) 3D [8] and (b) 2D [7].

Figure 1 . 6 -

 16 Figure 1.6 -(a) Samples taken from [94] showing body pose with a variety in camera views. (b) Facial images with different poses.

. 1 ) 2 .

 12 Chapitre Coding Human Motion Trajectories in the Shape Manifold with w 0 ≤ k, where . 0 represents the 0 -pseudonorm which counts the number of non-zero elements in a vector. In Equation2.2.1, the query x is represented as a linear combination of k atoms of the dictionary D. The set of indices of these k active atoms is called the support Su. As such, this model assumes that the signal x resides in a low dimensional subspace, which is spanned by the k atoms of D that correspond to the support of w. An illustration of this model is given in Figure 2.1.

Figure 2 . 1 -

 21 Figure 2.1 -Schematic sparse model.

Figure 2 . 2 -

 22 Figure 2.2 -Exponential map and logarithm map operations on a Riemannian manifold. The red curve represents the geodesic connecting the points Z and Z 1 . The green arrow represents the tangent vector V on the tangent space T Z (M).

  To remove the rotation variability, for any Z ∈ C, we define an equivalence class: Z = {ZO|O ∈ SO(m)} that represents all rotations of a configuration Z. The set of all such equivalence classes, S = { Z|Z ∈ C} = C/SO(m)

Figure 2 . 3 -

 23 Figure 2.3 -Z is a data point on the manifold M and the red circles represent the dictionary atoms. The linear approximation Ẑ with the atoms may be out of the manifold M.

Figure 2 . 4 -

 24 Figure 2.4 -Example results of the proposed framework. An input sequence is transformed to a smoothly-evolving sparse time-series. The X-axis represents the time frame. (A) 3D skeletal sequence. (B) 2D facial landmark sequence.

Figure 2 . 6 -Algorithm 2 : 2 2Algorithm 3 :

 26223 Figure 2.6 -Pictorial of the sparse coding approach on the pre-shape space C. The approximation of x ∈ C could be viewed as a weighted intrinsic mean of the atoms of a dictionary D = {d i } N i=1 .

49 -

 49 Figure 2.7, we show a qualitative result of clustering 3D skeletal shapes and 2D facial shapes. To achieve it, an inner product matrix is first computed from the training data based on the kernel function -Chapitre 2. Coding Human Motion Trajectories in the Shape Manifolddefined in Section 2.2.3.2. Note that in the 3D case, this kernel is positive definite for only certain values of the kernel parameter σ. Thus, its empiric choice is required to seek positive definiteness. The inner product matrix is then modeled using a Wishart distribution. To allow for an automatic inference on the number of clusters, prior distributions are carefully assigned to the parameters of the Wishart distribution. Then, posterior is sampled using a Markov chain Monte Carlo procedure based on the Chinese restaurant process for final clustering. For details, we refer the reader to[START_REF] Zhang | Bayesian clustering of shapes of curves[END_REF], where the authors presented the Bayesian clustering method to segment shapes of curves. In our work, we propose to adapt their approach to cluster shapes of 2D/3D landmark configurations where the only difference resides on the computation of the distance matrix.

Figure 2 . 7 -

 27 Figure 2.7 -Pictorial of the proposed clustering approach. Landmark configurations (left) are mapped from the the shape manifold to RKHS by computing the inner product matrix from the data (Middle). Bayesian clustering is then applied on this matrix to construct the final clusters (right) whose number is automatically inferred.

Figure 2 .

 2 Figure 2.10 -Denoising of skeletal shapes using sparse coding. Abnormal skeletons are presented in the top left. Code vectors (middle) are obtained after sparse coding. They are then reconstructed with respect to the dictionary (right) to recover the abnormal skeletons.

Figure 2 . 11 -

 211 Figure 2.11 -Mean shape computation on a set of skeletons. Left: Input 3D skeletons in the Kendall's shape space. Middle: Mean shape computed using the intrinsic mean algorithm. Right: Mean shape computed by first sparse coding the input skeletons, then computing the mean code and reconstruct it with the dictionary.

Figure 2 . 12 -

 212 Figure 2.12 -An example of linear interpolation between two latent variables: source (left) and target (right). Shapes in this figure are the result of mapping the interpolates from the latent space to the manifold.

Figure 3 . 1 -

 31 Figure 3.1 -Overview of the proposed frameworks. Trajectories of 2D facial expressions (respectively 3D actions) are encoded using extrinsic (respectively intrinsic) SCDL in the Kendall's shape space. Temporal modeling and classification are then performed on the obtained time-series in vector space.

  .2. A feature extraction step to capture lower-level information from the data. This step can account to the spatial information only or also considers the temporal dimension. We consider two main categories of features: hand-crafted features and learned features by means of deep learning. Such representations can be used as input to a temporal modeling stage to capture the temporal dependencies in time-series.The output is then fed to a classification stage which can consist on a standard classifier (e.g. k-nearest neighbors, SVM, etc.) or a deep learning framework.

Figure 3 . 2 - 61 -

 3261 Figure 3.2 -Overview of a typical landmark-based action/facial expression recognition approach.

Figure 3 . 3 -

 33 Figure 3.3 -Our categorisation of state-of-the-art approaches. Representations of 2D/3D sequences of landmarks (skeletons or faces) can be either hand-crafted or learned. The former representations can be categorised into: kernel-based, graphical models or Riemannian. The latter can be achieved using deep learning techniques.

  Over the last decade, many techniques have been proposed for action recognition from 3D skeletal data. As shown in Figure3.3, several methods are based on the extraction of hand-crafted features. Examples include the use of spatio-temporal graphical models to represent and classify action sequences. Considering a human action as transitions between body poses over time, G. Hernando et al. [34] proposed a forest-based classifier called transition forests to discriminate both static pose information and temporal transitions

  where a Part-based Hierarchical Bidirectional Recurrent Neural Network (PHRNN) is responsible for analyzing the temporal information of facial landmark sequences and a Multi-Signal Convolutional Neural Network (MSCNN) is designed to extract spatial features from still frames. These two networks are combined to boost the performance of facial expression recognition.

  combine two kernel representations: sequence compatibility kernel (SCK) and dynamics compatibility kernel (DCK) which separately achieved 92.98% and 92.77%, respectively. The proposed approach achieves good performance for most of the actions. However, the main confusions concern very similar actions, e.g., Drink from a bottle and answer phone, as demonstrated by the confusion matrix in Figure 3.4.

Figure 3 . 4 -

 34 Figure 3.4 -Confusion matrix on the Florence 3D dataset.

Figure 3 . 5 -

 35 Figure 3.5 -Visualization of 2-dimensional features of the NTU-RGB+D dataset. Left: original data. Right: the corresponding SCDL features. Each class is represented by a different color.

  successive sparse codes of L-dimensional time-series. Then, the resulting sequences of length L-1 are finally used for classification. We evaluate our approach on the most commonly-used dataset, namely CASME II.CASME II dataset[START_REF] Yan | Casme ii: An improved spontaneous micro-expression database and the baseline evaluation[END_REF] contains 246 spontaneous micro-expression video clips recorded from 26 subjects and regrouped into five classes: happiness, surprise, disgust, repression and others. We performed classification based on the commonly used Leave-one-subject-out protocol.

Figure 3 . 6 -

 36 Figure 3.6 -Recognition accuracy achieved for each emotion class in the CK+ (left) and the CASIA (rigth) datasets, and comparison between extrinsic and intrinsic SCDL approaches.

Figure 3 . 7 -

 37 Figure 3.7 -Left: Accuracy when varying the sparsity regularization parameter λ (% values in the x-axis represent the average sparsity). Right: Dictionary learning objective over iterations for: (1) Random initialization; (2) Our proposed initialization based on Bayesian clustering and PGA..

  stage, pose sequences are used to learn a generative model due to their ability to encode motion dynamics. The newly generated pose sequences are then used to guide the generation of video frames while preserving coherent appearances in the input image. Regarding the first stage, they proposed a pose sequence GAN (PSGAN) to generate skeletal sequences. Their generator transforms an input pose into a pose sequence by adopting a encoder-decoder architecture. The output of the decoder is then fed into a LSTM module for temporal pose modeling. However, their resulting sequences may contain corrupted poses which would affect the synthesis of coherent appearance in the second stage. In this chapter, we follow the same disentangled solution. In the first stage, we propose an encoder-GAN model which generates new samples in the encoder latent space which are then transformed into skeletal sequences. In contrast to previous works, it guarantees the shape and motion consistencies of the generated pose sequences while having a simpler architecture. In the second stage, given an input image, each pose of a sequence is transformed to an image, thereby constitutes the final video. The latter procedure is based on a recently proposed generative model that learns to transfer a person image to new poses[START_REF] Zhu | Progressive pose attention transfer for person image generation[END_REF]. The main contributions in this chapter are, -A novel framework, namely shapeGAN, to generate dynamic human poses. Our model has a encoder-GAN architecture. The encoder aims to represent the data using SCDL in the shape manifold yielding sparse time-series that are defined in a latent space with a vector space structure. These latent variables are then used to train a GAN to generate new samples in the latent space. The generated samples are then mapped back to the shape manifold to represent the generated pose sequences.-A pose-guided human video generation framework, namely shapeVGAN, which exploits the shapeGAN model to generate novel pose dynamics, then uses a pose transfer -92 -

Figure 4 . 1 -

 41 Figure 4.1 -Schema of the original GAN framework. The generator G takes a noise vector z as input and outputs generated images x. The discriminator D distinguishes x from real samples x.

Figure 4 . 93 -

 493 1 consists of a generator and a discriminator, two neural networks that are trained against each other. The generator G aims to generate samples that resemble to real ones, while the discriminator D tries to distinguish whether a sample comes from the training data (i.e. real samples) or from G. In other words, G captures the data distribution by taking a latent variable z sampled from a prior distribution p z (z) (e.g. a Gaussian distribution), and generates a fake image x = G(z). x can be seen as a sample from the learned generative distribution p G . D discriminates between x (as fake data) and x sampled from real data distribution p data (x) (as real data). Accordingly, the -Chapitre 4. Pose-guided Human Video Generation with Sparse Representations training of G and D is similar to a two-player minimax game where D is trained to maximize the probability of assigning the correct label to both real samples and generated samples, and simultaneously, G is trained to minimize log(1 -D(G(z))), according to the following objective function:min G max D L a (G, D) = E x∼p data (x) [log D(x)] + E z∼pz(z) [log(1 -D(G(z)))],(4.2.1) where L a (G, D) denotes the adversarial loss for G and D, and D(x) denotes a probability that x is real. Note that at early training stage, when G only generates poor images, D tends to assign them low probability, which makes log(1-D(G(z))) saturate and unable to provide sufficient gradient to update G. A standard way in practice is to maximize log(D(G(z)) for better training G. Training GANs In practice, training the original GANs can be remarkably challenging and unstable. Empirically, there are three most common failures occurred within the training process [22]:

  method. In the first stage, they train a Pose Sequence GAN (PSGAN) to generate sequences of poses conditioned on the class label. The latter, along with an input image, are used in the second stage to guide the generation of coherent video frames. Since PSGAN may yield corrupted poses, a Semantic Consistent GAN (SCGAN) is proposed to impose semantic consistency between generated pose and ground-truth at a high feature level. In another work, Wang et al.[START_REF] Wang | Every smile is unique: Landmark-guided diverse smile generation[END_REF] proposed a Conditional Multi-Mode Network for a diverse smile video generation, conditioned on landmark sequences. They used a VAE to learn a facial landmark embedding. A recurrent network is then used to generate landmark embedding sequences conditioned on a specific expression. The latter are fed into a multi-mode recurrent landmark generator to produce a set of landmark sequences still associated to the given smile class but clearly distinct from each other. Finally, these landmark sequences are translated into face videos. The latter procedure can be achieved by transferring each frame of a pose -95 -sequence to an image which is know as pose to image translation or pose transfer.Pose to image translation Transferring a person from one pose to another was first proposed by[START_REF] Ma | Pose guided person image generation[END_REF] to synthesize person images in arbitrary poses. Based on an image of a person and novel poses, the generation consists of two steps: pose integration and image refinement. First, the condition image and the target pose are fed into a U-Net-like network to generate an initial but coarse image of the person with the target pose. Then, the image refinement is performed on the initial and blurry result by training a U-Net-like generator in

Given T training videos

  {v i } T i=1 presenting one human action, we aim to train a model that learns the training data distribution and generates a set of new videos {h i } M i=1 . Typically, the generated samples would present variations in terms of the style of performing actions that are different than those of the training videos. This includes variations in terms of the execution rate of actions. To achieve this goal, we propose a two-step approach. In the first, pose sequences {p i } T i=1 are extracted from videos and used to train a generative model, namely shapeGAN, which learns to generate novel motion dynamics {q i } M i=1 . In the second, -96 -

Figure 4 .

 4 Figure 4.2 gives an overview of the proposed approach. Given an input video presenting

Figure 4 . 2 -

 42 Figure 4.2 -Overview of the proposed shapeGAN framework.

Figure 4 . 3 -

 43 Figure 4.3 -Generator architecture of the progressive pose attention transfer method [130].

Figure 4 . 4 -

 44 Figure 4.4 -Pose to image transfer. (a): Condition image. (b): Condition pose. (c): Target pose. (d): Target image.

Figure 4 . 5 -

 45 Figure 4.5 -Examples of generated pose sequences of actions with shapeGAN. From top to bottom are actions: hand wave, sit-down, and clap hands.

4. 4 . 2 Figure 4 . 5 Figure 4 . 7 ,

 424547 Figure 4.5 shows some examples of generated pose sequences with shapeGAN. We can clearly observe that the pose sequences change in a smooth and typical way under each action scenario, without the presence of any corrupted poses (noise) or temporal interruption. Recall that the shapeGAN model learns to generate new samples in the latent space (after coding initial sequences using SCDL in the shape manifold). To help insure that it generalizes well on the training data and creates variability on the generated samples, in Figure 4.6 and Figure 4.7, we show two plots of both training and generated data in the latent space after embedding them in a 2D space. This is achieved by applying the t-SNE embedding that allows to better visualize data distributions. In the first figure, we have data that belong to one class (i.e., hand wave). The training data (in blue) represent eight sequences. Around each of them, we can see the newly generated sequences (in red) which give us evidence on the generalization capability of our shapeGAN model. In the second figure, we have data that belong to nine classes of the Florence 3D dataset. The left panel representing training data (in the encoder latent space) again demonstrates the discrimination ability of our sparse representation since data-points from each class are regrouped together. The right panel shows that the generated samples are also regrouped together in terms of class

Figure 4 . 6 -

 46 Figure 4.6 -2D visualization of training samples (blue) and generated samples (red).

Figure 4 . 7 -

 47 Figure 4.7 -2D visualization of data belonging to nine action classes of the Florence 3D dataset. Left: training samples. Right: training and generated samples.

Figure 4 .

 4 [START_REF] Baltrusaitis | Openface 2.0: Facial behavior analysis toolkit[END_REF], we visualize two feature maps corresponding to two samples in the encoder latent space (i.e., sequences of sparse codes after SCDL). The first is from the training set while the second is from -103 -

Figure 4 . 8 -

 48 Figure 4.8 -Visualization of a training (left) and a generated (right) feature maps.

Figure 4 . 9 -

 49 Figure 4.9 -Examples of generated samples with shapeVGAN. Different values of the sparsity regularization parameter λ (in the sparse coding) were used in each experiment.

Figure 4 .

 4 Figure 4.10 -(a) Pose sequence generated after training a GAN directly on raw pose sequences. (b) Pose sequence generated with shapeGAN.

Figure 4 .

 4 11, we show examples of generated videos with shapeVGAN which belong to the class wave hands of the Weizmann dataset. The first row of this figure represents a training video while the two others are taken from the generated set. The latter are generated -107 -

Figure 4 . 11 -

 411 Figure 4.11 -Examples of video frames of action wave hands. The first row represents a training video while the two others represent videos generated with our approach.

Figure 4 . 12 -

 412 Figure 4.12 -Example of a generated pose sequence with shapeGAN (Top row). This sequence is used to guide the generation of the two videos (second and third row) given condition images (in red).

  proposed a novel framework namely shapeVGAN that allows to generate human videos in a two-stage procedure. In the first stage, we presented the shapeGAN model to generate novel pose sequences which are used in the second stage to guide the video generation. shapeGAN -110 -4.6. Conclusionhas an encoder-GAN architecture. The encoder is based on SCDL in the shape manifold which in addition to being a discriminative representation that showed promising results in classification problems, it has been an efficient representation for the human motion generation task. The GAN generates new samples in the encoder latent space that are then transformed to pose sequences. Each pose of a sequence is then mapped to an image given a condition image to obtain a video. The generated videos with shapeVGAN showed to be realistic and temporally consistent in the case of human actions. As a future work, we aim to generalize this framework to the task of facial expression generation.-111 -

Figure 4 . 13 -

 413 Figure 4.13 -Qualitative comparison of (a) Our results obtained with shapeVGAN. (b) Results obtained by training a state-of-the-art approach, i.e. MoCoGAN [104].

Table 3 .

 3

	1 -Overall recognition accuracy (%) on MSR-Action 3D, Florence Action 3D, and
	UTKinect 3D datasets. In the first column: (R) : Riemannian approaches; (N ) : other recent
	approaches; Last row: our approach.
	Dataset	MSR-Action 3D	Florence 3D	UTKinect 3D
	Protocol	H-H 3 Subsets	H-H LOAO	H-H LOSO
	(R) T-SRVF on Lie group [4]	85.16	-	89.67	-	94.87	-
	(R) T-SRVF on S [9]	89.9	-	-	-	-	-
	(R) Lie Group [107]	89.48	92.46	90.8	-	97.08	-
	(R) Rolling rotations [108]	-	-	91.4	-	-	-
	(R) Gram matrix [58]	-	-	-	88.85	-	98.49
	(N ) Graph-based [114]	-	-	-	91.63	97.44	-
	(N ) ST-LSTM [73]	-	-	-	-	95.0	97.0
	(N ) JLd+RNN [126]	-	-	-	-	95.96	-
	(N ) SCK+DCK [66]	91.45	93.96	95.23	-	98.2	-
	(N ) Transition-Forest [34]	-	94.57	-	94.16	-	-
	(R) Ours (SVM)	90.01	94.19	92.85	92.27	97.39	97.50
	(R) Ours (Bi-LSTM)	86.18	86.18	93.04 94.48	96.89	98.49

Table 3 .

 3

	3 -Comparative evaluation of intrinsic and extrinsic SCDL in recognizing 3D
	actions.
	Dataset	MSR-Action 3D	Florence 3D	UTKinect 3D
	Protocol	H-H	3 Subsets	H-H	LOAO	H-H	LOSO
	Extrinsic SCDL	82.52	88.53	85.76	89.03	93.97	94.97
	Intrinsic SCDL	90.01	94.19	92.85 92.27 97.39 97.50

Table 3 .

 3 4 -Comparison with state-of-the-art on CK+ and Oulu-CASIA datasets. (A) : Appearance-based approaches; (G) : Geometric approaches; (R) : Riemannian approaches; Last row: our approach.

	Method	CK+	Oulu-CASIA
	(A) CSPL [129]	89.89	-
	(A) ST-RBM [31]	95.66	-
	(A) STM-ExpLet [74]	94.19	74.59
	(G) ITBN [116]	86.30	-
	(G) DTGN [57]	92.35	74.17
	(A+G) DTAGN [57]	97.25	81.46
	(R) Shape velocity on Grassmannian [102]	82.80	-
	(R) Shape traj. on Grassmannian [59]	94.25	80.0
	(R) Gram matrix trajectories [59]	96.87	83.13
	(R) Intrinsic SCDL (SVM)	91.26	70.37
	(R) Intrinsic SCDL (Bi-LSTM)	89.43	70.24
	(R) Extrinsic SCDL (SVM)	95.62	77.06
	(R) Extrinsic SCDL (Bi-LSTM)	95.73	73.09

Table 3 .

 3 [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] -Confusion matrix on the CK+ dataset.

	An	Co	Di	Fe	Ha	Sa	Su
	An 97.50

Table 3 .

 3

		An	Di	Fe	Ha	Sa	Su
	An 72.33 12.33	2.11	1.0	12.22	0
	Di 14.22 68.56	6.22	3.0	8.0	0
	Fe	5.22	2.0	72.33	5.11	9.33	6.0
	Ha	4.0	0	9.33	85.67	1.0	0
	Sa 15.22	4.11	6.11	2.0	72.56	0
	Su	0	2.1	5.0	0	2.0	90.89

6 -Confusion matrix on the Oulu-Casia dataset.

Table 3 .

 3 8 -Evaluation of the Kendall's shape space representation. datasets, reported in the third row of Table3.8, clearly show the interest of accounting for the nonlinearity of the manifold when applying SCDL.

	Approach	NTU-RGB+D Florence MSR 3D UTKinect
	Raw data	56.5	84.29	87.36	92.67
	Linear SCDL	79.20	87.94	89.23	93.58
	Kendall SCDL	82.95	92.85	90.01	97.5

Table 3 .

 3 9 -Classification performances when using different landmark detectors.

	Landmark detector	Oulu-CASIA dataset CK+ dataset
	Cherha [7] -49 landmarks	76.41	93.68
	Openface [8] -49 landmarks	70.85	83.73
	Openface [8] -68 landmarks	71.26	82.92

Table 4 .

 4

	1 -Action recognition accuracy (%) using an increasing number of generated
	training data.
	Data	Accuracy
	Training data	93.04
	Augmented data ×3	93.98
	Augmented data ×6	94.56
	Augmented data ×9	95.04
	Augmented data ×12	95.04
	modeling and classification, we choose to use Bi-LSTM rather than the pipeline of DTW-

FTP-SVM since it is known that deep learning methods usually require more data in order to increase their generalization capability compared to classical approaches. Table

4

.1 summarizes the obtained results. Note that, in the different experiments that we performed, the amount of data was multiplied by 3, 6, 9 and 12, respectively. The obtained results

Thèse de Amor Ben Tanfous, Université de Lille, 2019 © 2019 Tous droits réservés. lilliad.univ-lille.fr

t-SNE is a nonlinear dimensionality reduction technique that allows for embedding high-dimensional data into two or three dimensional space, which can then be visualized in a scatter plot.

first captures the spatio-temporal compatibility of joints between two sequences, while the second models a sequence dynamics as the spatio-temporal co-occurrences of the joints.

Tensors are then formed from these kernels to train SVM.

The above-mentioned approaches did not make any manifold assumptions on the data representation. However, several shape representations and their dynamics often lie to nonlinear manifolds. As discussed in the category of Riemannian approaches, many approaches exploited the Riemannian geometry of nonlinear manifolds to analyze skeletal sequences. The latter are considered in different Riemannian manifolds such as the Lie group, the Grassmann manifold, the shape manifold, etc. Since the work presented in this dissertation is based on a representation in the shape manifold, we consider it as part of this category. In Section 2.3.2 of Chapter 2, we described some representations of 3D skeletal sequences on Riemannian manifolds. Here, we discuss their corresponding temporal modeling and classification steps. In [START_REF] Vemulapalli | Human action recognition by representing 3d skeletons as points in a lie group[END_REF], sequences are represented as trajectories in the product space of Lie groups SE(3) × ... × SE(3) then mapped to the Lie algebra se(3) n , the tangent space at the identity element. To handle the rate variability in human actions, the obtained time-series in se(3) n are aligned by means of the popular dynamic time warping (DTW) algorithm. To handle temporal misalignment as well as the noise present in the data, the aligned sequences are processed using Fourier temporal pyramid (FTP). Finally, the obtained features are classified using a one-vs-all SVM. In [START_REF] Vemulapalli | Rolling rotations for recognizing human actions from 3d skeletal data[END_REF], the authors also applied DTW on the Lie group representation of actions then classified final curves using a onevs-all linear SVM after mapping the curves to the Lie algebra using rolling maps which ensures a better flattening of the Lie group. DTW and SVM were also used in [START_REF] Anirudh | Elastic functional coding of human actions: From vector-fields to latent variables[END_REF] to classify their low-dimensional representation of actions which were initially represented as trajectories in the Lie group. Based on the same trajectory representation on SE(3), the authors in [START_REF] Huang | Deep learning on lie groups for skeleton-based action recognition[END_REF] proposed a deep learning framework in Lie groups to recognize actions.

The proposed architecture includes several special layers (e.g. RotMap layer, RotPooling layer, etc.) which accounts to the geometry of the manifold. Taking another direction, the