Wentao Tang

Qinghua Zhang
email: qinghua.zhang@inria.fr

Zhenhua Wang
email: zhenhua.wang@hit.edu.cn

Yi Shen

Ellipsoid bundle and its application to set-membership estimation ⋆

Keywords: Ellipsoid, set-membership estimation, zonotope, ellipsoid bundle

published or not. The documents may come

INTRODUCTION

State estimation is an important topic in control theory and engineering applications. However, a practical system is usually affected by various model uncertainties, for instance, process disturbances and measurement noises. One of the most important problems in state estimation is how to deal with such uncertainties to obtain reliable estimations. Many existing state estimation methods are designed based on stochastic theory, for instance the wellknown Kalman filter [START_REF] Kalman | A new approach to linear filtering and prediction problems[END_REF]. These methods usually assume that uncertainties have known probability distributions. However, in many situations, these probability distributions are unknown. Some disturbances may even not be random. Different from the state estimation methods based on stochastic system theory, set-membership estimation methods only assume that model uncertainties are unknown but bounded. This is a weaker assumption, which can be satisfied in most practical systems. Moreover, set-membership estimation methods can obtain the reachable set of state instead of a single trajectory estimation, which is useful in many applications. Recently, set-membership estimation has received considerable attention and has been applied to various fields such as fault diagnosis [START_REF] Tang | Fault detection and isolation for discrete-time descriptor systems based on H -/L ∞ observer and zonotopic residual evaluation[END_REF][START_REF] Xu | Robust fault detection and set-theoretic UIO for discrete-time LPV systems with state and output equations scheduled by inexact scheduling variables[END_REF] and robust control [START_REF] Canale | Set Membership approximation theory for fast implementation of Model Predictive Control laws[END_REF][START_REF] Efimov | Control of nonlinear and LPV systems: interval observer-based framework[END_REF].

Set-membership estimation aims to construct a compact set enclosing all admissible state values that are consistent with the system model, the input and measurement data. Many geometrical sets, for instance ellipsoids, polytopes and zonotopes, have been used to design set-membership ⋆ This work was supported by National Natural Science Foundation of China (Grant No. 61973098, 61773145), the Key Laboratory Opening Funds of Harbin Institute of Technology under grant HIT.KLOF.2018.073 and the China Scholarship Council (CSC) under grant No.201806120061. estimation methods. For different choices, an important problem is how to make a tradeoff between estimation accuracy and computation complexity. In the pioneering work of [START_REF] Schweppe | Recursive state estimation: unknown but bounded errors and system inputs[END_REF], a recursive set-membership estimation method based on ellipsoids was proposed. The ellipsoid-based methods have low computation complexity and have attracted much attention [START_REF] Fogel | On the value of information in system identification-bounded noise case[END_REF][START_REF] Maksarov | State bounding with ellipsoidal set description of the uncertainty[END_REF][START_REF] Chernousko | Ellipsoidal state estimation for dynamical systems[END_REF][START_REF] Zhang | Ellipsoid-based interval estimation for takagi-sugeno fuzzy systems[END_REF]. However, the estimations by the ellipsoid-based methods are usually conservative since a single ellipsoid is not able to describe a complex reachable set of state. General polytopes can be used to achieve more accurate set-membership estimation [START_REF] Blanchini | Set-theoretic methods in control[END_REF], but the computation complexity may be very high. Zonotopes, a special kind of polytopes, provide a good tradeoff between estimation accuracy and computation complexity. Recently, setmembership estimation methods based on zonotopes have received considerable attention [START_REF] Alamo | Guaranteed state estimation by zonotopes[END_REF][START_REF] Combastel | Zonotopes and kalman observers: Gain optimality under distinct uncertainty paradigms and robust convergence[END_REF][START_REF] Tang | Interval estimation methods for discrete-time linear time-invariant systems[END_REF].

The accuracy of existing methods based on ellipsoids is limited by the shape of a single ellipsoid. However, the ellipsoidal estimation sets have smooth boundaries, which is a good property when the results are used for some optimization purposes. Similarly, in the classical curve fitting problem, splines are often preferred to piecewise linear interpolation. In addition, it is more suitable to use ellipsoids in some specific situations, for instance in target tracking problems [START_REF] Maksarov | State bounding with ellipsoidal set description of the uncertainty[END_REF]. Considering these, this paper proposes a new set representation tool, called ellipsoid bundle, which combines the advantages of ellipsoids and zonotopes. Like zonotopes, ellipsoid bundles are able to describe complex convex sets. Therefore, ellipsoid bundles can be used to design a setmembership estimation method less conservative than the ellipsoid-based method. Moreover, ellipsoid bundles also have smooth boundaries. This paper has the following main contributions. First, we propose a new set representation tool, ellipsoid bundle, which combines the advantages of ellipsoids and zonotopes for uncertainty set representation and computation. Second, the basic properties of ellipsoid bundles are investigated. Third, a novel set-membership estimation method is proposed based on ellipsoid bundles. Finally, simulation results demonstrate the effectiveness of the proposed set-membership estimation method.

PRELIMINARIES

The following notations are standard in this paper. R m×n and R n denote the m × n dimensional and n dimensional Euclidean spaces, respectively. A bold letter represents a set in the rest of this paper. The symbols ≤ and ≥ are understood element-wise. Given a matrix P ∈ R n×n , P ≽ 0 (P ≼ 0) indicates that P is a positive (negative) semidefinite matrix. Definition 1. Given two sets, X 1 ⊂ R n and X 2 ⊂ R n , their Minkowski sum is defined as

X 1 ⊕ X 2 = {x ∈ R n : x = x 1 + x 2 , x 1 ∈ X 1 , x 2 ∈ X 2 }.
Definition 2. An m-order zonotope is an affine transformation of the hypercube

B m = [-1, 1] m , i.e. Z = ⟨p, H⟩ = {x ∈ R n : x = p + Hz, z ∈ B m }, where p ∈ R n
is the center of Z and H ∈ R n×m defines its shape and size. Definition 3. A non-degenerate ellipsoid is defined as

{x ∈ R n : (x -c) T P -1 (x -c) ≤ 1}, (1)
where c ∈ R n is the center of the ellipsoid and P ∈ R n×n is a positive definite matrix which determines its shape and size.

This definition, though widely adopted in the literature, does not cover degenerate ellipsoids, for which the matrix P would be singular. The following definition [START_REF] Durieu | Multiinput multi-output ellipsoidal state bounding[END_REF] covering both degenerate and non-degenerate ellipsoids will be adopted in the remaining part of this paper.

{x ∈ R n : x = c + Lz, z ∈ R m , z T z ≤ 1}, (2) where c ∈ R n and L ∈ R n×m .
For non-degenerate ellipsoids (with non-singular P ), these two definitions are equivalent with P = LL T . Moreover, any matrix L ∈ R n×m such that LL T is equal to the same P defines the same set with (2). The parametrization of an ellipsoid by L is thus clearly redundant. To avoid this redundancy, in what follows, an ellipsoid will be denoted by E(c, P ), with c representing its center and P = LL T . Definition 4. The support function of a convex set S ⊂ R n in terms of a vector l ∈ R n is defined as ρ S (l) = max s∈S l T s.

Lemma 1. [START_REF] Schweppe | Recursive state estimation: unknown but bounded errors and system inputs[END_REF]) Given a non-degenerate ellipsoid E(c, P ) defined as (1), its support function in terms

of l ∈ R n is ρ E(c,P ) (l) = l T c + √ l T P l.
Although Lemma 1 was proposed for non-degenerate ellipsoids in [START_REF] Schweppe | Recursive state estimation: unknown but bounded errors and system inputs[END_REF], it can be easily extended to the ellipsoids defined as (2).

Lemma 2. Give an ellipsoid E(c, P ) defined as ( 2) with

P = LL T , its support function in terms of l ∈ R n is ρ E(c,P ) (l) = l T c + √ l T P l.
Proof. According to the definition of E(c, P ) in ( 2), we have

ρ E(c,P ) (l) = max x∈E(c,P ) l T x = max z T z≤1 (l T c + l T Lz) = l T c + max z T z≤1 (l T L)z = l T c + ρ E(0,I) (L T l) (3)
Since E(0, I) is a non-degenerate ellipsoid, then according to Lemma 1 and (3), we have

ρ E(c,P ) = l T c + √ l T LL T l = l T c + √ l T P l. 2 Lemma 3. (Schweppe, 1968) Given two convex sets, S 1 ⊂ R n , S 2 ⊂ R n , and a vector l ∈ R n , we have ρ S1⊕S2 (l) = ρ S1 (l) + ρ S2 (l).

PROBLEM FORMULATION

Consider the following discrete-time system:

{ x k+1 = A k x k + B k u k + w k y k = C k x k + v k (4)
where x k ∈ R nx , u k ∈ R nu and y k ∈ R ny are the vectors of state, input and measurement output, respectively. w k ∈ R nx and v k ∈ R ny are the process disturbance and measurement noise.

A k ∈ R nx×nx , B k ∈ R nx×nu and C k ∈ R ny×nx are known matrices.
The process disturbance and measurement noise are assumed to be unknown but bounded by known ellipsoids.

w k ∈ Q k = E(0, Q k ), v k ∈ R k = E(0, R k ),
(5) where Q k ∈ R nx×nx and R k ∈ R ny×ny are known positive definite matrices. These ellipsoidal assumptions are particularly suitable in target tracking problems, where uncertainties are naturally related to the Euclidean distance.

In addition, the initial state is also assumed to be unknown but bounded by an ellipsoid.

x 0 ∈ P 0 = E(c 0 , P 0 ), ( 6) where c 0 ∈ R nx and P 0 ∈ R nx×nx is a positive definite matrix.

In this paper, we aim to obtain a convex set X k ⊂ R nx such that

x k ∈ X k , ∀k ≥ 0. In order to increase the tightness of this convex set, we need improve the existing ellipsoid-based methods. In this paper, we will propose a new set representation tool and design a set-membership estimation method based on the proposed tool.

ELLIPSOID BUNDLE

The ellipsoid-based set-membership estimation methods have low computation complexity. However, the estimations obtained by the ellipsoid-based methods may be very conservative since a single ellipsoid has limitation to describe a complex set. Compared to ellipsoids, zonotopes (a special kind of polytopes) are able to describe complex sets with finite parameters. Zonotopes provide a way to make a good tradeoff between estimation accuracy and computation complexity, but they have non-smooth boundaries and are less suitable in some applications, like in target tracking.

In order to overcome the drawbacks of the ellipsoid-based methods, we define a new form of sets which combine some characteristics of ellipsoids and zonotopes. To this end, we need first review the relevant knowledge of zonotopes.

In fact, an m-order zonotope Z = ⟨p, H⟩ as in Definition 2 is also the Minkowski sum of the vector p and m centered line segments as follows.

Z = p ⊕ m ⊕ i=1 L i , ( 7
)
where

L i = {l ∈ R n : l = a i h i , a i ∈ [-1, 1]} and h i is the i-th column of H.
Motivated by ( 7), we define a new form of sets, called ellipsoid bundles, which are defined as follows. Definition 5. An m-order ellipsoid bundle E(c, H) ⊂ R n is the Minkowski sum of the vector c and m centered ellipsoids as follows.

E = c ⊕ m ⊕ i=1 E(0, P i ),
where c ∈ R n and H = [P 1 , . . . , P m ], P i ∈ R n×n , i = 1, . . . , m, are positive semidefinite matrices.

Figure 1 presents a 2-order ellipsoid bundle in 3-dimensional space.

Fig. 1. A 2-order ellipsoid bundle in 3-dimensional space.

Similar to zonotopes, the Minkowski sum and linear transformation of ellipsoid bundles can be implemented by simple operations.

Property 1 (Minkowski sum). Given two ellipsoid bundles, E(c 1 , H 1 ) and E(c 2 , H 2 ), their Minkowski sum satisfies

E 1 ⊕ E 2 = E(c, H), where c = c 1 + c 2 and H = [H 1 , H 2 ].
Property 2 (Linear transformation). Given an m-order ellipsoid bundle E(c, H) and a matrix M ∈ R q×n , we have

M E(c, H) = E ( M c, L (M, H) ) , where L (M, H) = [ P1 , . . . , Pm ] , Pi = M P i M T , i = 1, . . . , m.
Property 1 and Property 2 are direct corollaries of the definition of Minkowski sum and the linear transformation of ellipsoids.

When applying ellipsoid bundles to set-membership estimation, the order of the estimated ellipsoid bundles will increase linearly with the time k, like in zonotope-based estimators. To limit the computational burden, we need to design a reduction method to enclose a high-order ellipsoid bundle by a lower one, again like in the case of zonotopes.

To this end, we first introduce two necessary lemmas. Lemma 4. [START_REF] Durieu | Multiinput multi-output ellipsoidal state bounding[END_REF] Given the ellipsoid bundle E(c, H), we can obtain a parameterized ellipsoid E(c, P α ) enclosing E(c, H), where P α satisfies

P α = m ∑ i=1 1 α i P i ,
where α i > 0 for all i ∈ {1, . . . , m} and ∑ m i=1 α i = 1. Lemma 5. [START_REF] Durieu | Multiinput multi-output ellipsoidal state bounding[END_REF] In the family of E(c, P α ), the minimal-trace ellipsoid containing E(c, H) satisfies

α i = √ tr(P i ) ∑ m i=1 √ tr(P i ) ,
where tr(P i ) denotes the trace of P i .

Based on Lemma 4 and 5, we propose a reduction method, which can be described by the following theorem.

Proof. From Definition 5, the reordering of P i in H does not change the scope of E(c, H). Therefore, E(c, H) = E(c, H).

The reordered ellipsoid bundle E(c, H) can be rewritten as

E(c, H) = E(c, H 1 ) ⊕ E(0, H 2 ), where H 1 = [ P σ(1) , . . . , P σ(s-1) ] , H 2 = [ P σ(s) , . .

. , P σ(m)

] .

According to Lemma 4 and 5, E(0, H 2 ) ⊆ E(0, P ). Then, by using Property 1 (Minkowski sum), we have

E(c, H) ⊆ E(c, H 1 ) ⊕ E(0, P ) = E(c, [ H 1 , P ] ) = E ( c, R s (H)
) .

It follows that E(c, H) ⊆ E ( c, R s (H)

) .

SET-MEMBERSHIP ESTIMATION BASED ON ELLIPSOID BUNDLES

Based on the ellipsoid bundles described in Section 4, we propose a new set-membership estimation method for system (4). The proposed method obtains the setmembership estimation of state by combining a single state trajectory estimator with the reachable set estimation of estimation errors based on ellipsoid bundles.

The single state trajectory estimator has the following observer structure:

xk+1 = A k xk + B k u k + L k (y k -C k xk ), (8)
where xk ∈ R nx is the state estimation and L k ∈ R nx×ny is the observer gain matrix.

Define the estimation error as e k = x k -xk . Subtract ( 8) from ( 4), then we obtain the following error system.

e k+1 = (A k -L k C k )e k + w k -L k v k . ( 9
)
Based on the reachable set analysis of error system (9), we can obtain the set-membership estimation of x k in the form of ellipsoid bundles. The proposed method can be described by the following theorem. Theorem 2. For system (4), x k can be bounded by the ellipsoid bundle E(x k , H k ), where

H k ∈ R nx×(nx×m) satis- fies H k+1 = [ L (A k -L k C k , H k ), Q k , L k R k L T k ] , ( 10
) where

H k = { H k , m ≤ s, R s (H k ), m > s (11)
and H 0 = P 0 , x0 = c 0 (12) and s is a fixed positive integer.

Proof. From ( 6) and ( 12), we have

x 0 ∈ E(x 0 , H 0 ). When k ≥ 0, if x k ∈ E(x k , H k )
, then according to Theorem 1 and (11), we have

E(x k , H k ) ⊆ E(x k , H k ). Therefore, x k ∈ E(x k , H k ). Since e k = x k -xk , it follows that e k ∈ E(0, H k ).
Then, from ( 5) and ( 9), we have

e k+1 ∈ (A k -L k C k )E(0, H k ) ⊕ Q k ⊕ L k R k = (A k -L k C k )E(0, H k ) ⊕ E(0, Q k ) ⊕ L k E(0, R k ). ( 13
)
According to Property 1 and 2, ( 13) implies e k+1 ∈ E(0, H k+1 ). Then, we have

x k+1 = xk+1 + e k+1 ∈ E(x k+1 , H k+1 ). By induction, x k ∈ E(x k , H k ) holds for all k ≥ 0. 2
In order to increase estimation accuracy, we need design L k such that the size of E(x k+1 , H k+1 ) is minimized. Similar to zonotopes, we propose the following size criterion for the ellipsoid bundle E(x k+1 , H k+1 ).

J k+1 ( E(x k+1 , H k+1 ) ) = p+2 ∑ i=1 tr(P k+1 i ). ( 14
)
where

H k+1 = [ P k+1 1 , . . . , P k+1 p+2 ] and p = min{m, s}. J k+1 ( E(x k+1 , H k+1 ) )
is the sum of the squared half-axes of all the ellipsoids E(0, P k+1 i ), i = 1, . . . , p + 2, which determine the shape and size of the ellipsoid bundle E(x k+1 , H k+1 ).

The above size criterion is similar to the F-radius of zonotopes, which can provide an efficient way to design the observer gain in real time. To minimize the size criterion J k+1 , we propose the following theorem. Theorem 3. For the ellipsoid bundle E(x k+1 , H k+1 ) obtained from (10), denote

H k = [ P k 1 , . . . , P k p ]
, the optimal L k minimizing the size criterion J k+1 satisfies

Θ k = p ∑ i=1 P k i , V k = C k Θ k C T k + R k , L k = A k Θ k C T k V -1 k . ( 15
)
Proof. From (10), we have

H k+1 = [ P k+1 1 , . . . , P k+1 p+2 ]
, where

P k+1 i =      (A k -L k C k ) P k i (A k -L k C k ) T , 1 ≤ i ≤ p; Q k , i = p + 1; L k R k L T k , i = p + 2.
Then, from ( 14), we have

J k+1 = p ∑ i=1 tr ( (A k -L k C k ) P k i (A k -L k C k ) T ) + tr(Q k ) + tr(L k R k L T k ).
We can obtain the derivative of J k+1 for L k as follows.

∂J k+1 ∂L k = 2L k V k -2A k P k C T k . ( 16
)
Let ( 16) equal zero, we obtain (15). Therefore, the L k obtained from ( 15) is a stationary point. From (15), we have 15) is the unique stationary point. In addition, J k+1 is a quadratic function of L k and is non-negative for any L k , then J k+1 is a convex function of L k . Therefore, the L k obtained from ( 15) is the optimal solution minimizing the size criterion J k+1 . 2 Remark 1. The assumptions ( 5) and ( 6) can be relaxed as follows.

V k ≽ R k . Since R k is positive definite, V k is invertible. Therefore, the L k obtained from (
w k ∈ E(0, W k ), v k ∈ E(0, V k ), x 0 ∈ E(c 0 , H 0 )
, which can be used to describe more complex sets of uncertainties. In this case, the relevant adjustments of the proposed method are trivial, i.e. P 0 to H

0 , Q k to W k , L k R k L T k to L(L k , V k ) and R k to ∑ nv i=1 R i k , where n v is the order of E(0, V k ) and V k = [ R 1 k . . . R nv k ] .
In fact, an m-order zonotope Z = ⟨p, H⟩ can be represented by an m-order ellipsoid bundle E(p, M ), where

M = [ h 1 h T 1 . . . h m h T m
] and h i is the i-th column of H. This is because that the line segment

L i = {l ∈ R n : l = a i h i , a i ∈ [-1, 1]} is actually the degenerate ellipsoid E(0, h i h T i ).
In this case, zonotopes can be seen as a special kind of ellipsoid bundles. The proposed method can also apply to the case with zonotope-bounded uncertainties. Therefore, the proposed method has wide range of applications.

In some practical applications of set-membership estimation for instance fault diagnosis, for the sake of simplicity, the interval estimation including the upper and lower bounds of the estimated state is usually derived from the estimation set to serve as thresholds. For the proposed method based on ellipsoid bundles, we propose the following theorem to obtain the interval estimation of state. Theorem 4. For system (4), we can obtain an interval vector Proof. According to Theorem 2,

[x k , x k ] such that x k ≤ x k ≤ x k ,
x k ∈ E(x k , H k ). Since e k = x k -xk , we have e k ∈ E(0, H k ).
From Definition 5, we have

E(0, H k ) = m ⊕ i=1 E(0, P k i ). (17)
Define l j ∈ R nx as the vector with its j-th element equal to 1 and the others equal to 0, where j = 1, . . . , n x . For example,

l 1 = [1 0 . . . 0] T .
According to Lemma 2, we have

ρ E(0,P k i ) (l j ) = √ l T j P k i l j = √ P k i (j, j)
, where P k i (j, j) is the j-th diagonal element of P k i . Then, from Lemma 3 and ( 17), we have In this section, a numerical example in the form of ( 4) is used to demonstrate the performance of the proposed method. The example has the following parameters.

ρ E(0,H k ) (l j ) = m ∑ i=1 ρ E(0,P k i ) (l j ) = m ∑ i=1 √ P k i (j, j) = e k (j
A k = [ 0.3 + 0.2 sin(0.5k) -0.7 -0.2 sin(0.25k) 0.6 -0.5 + 0.3 cos(0.5k) ] , B k = [ 1 1 ] , C k = [1 + 0.2 sin(0.2k) 0] .
In the simulation, the input is set as u k = 0.4 sin(0.5k).

Since ellipsoid bundles have many similar characteristics to zonotopes, we compare the proposed method with the Zonotopic Kalman Filter (ZKF) in [START_REF] Combastel | Zonotopes and kalman observers: Gain optimality under distinct uncertainty paradigms and robust convergence[END_REF]. However, zonotope-based methods including ZKF assume that disturbances and noises are bounded by certain zonotopes, usually boxes, which is different from the proposed method. To make comparisons, we consider the following two cases.

Case 1. In the simulation, w k , v k and x 0 are set bounded by ellipsoids as in ( 5) and ( 6) with Q k = 0.0025I 2 , R k = 0.0025I 2 , P 0 = 0.04I 2 . For ZKF, which requires zonotopes bounding w k , c k and x 0 , we use the smallest boxes enclosing the corresponding assumed ball-shaped uncertainty sets:

Q k ⊂ W k = ⟨0, 0.05I 2 ⟩, R k ⊂ V k = ⟨0, 0.05I 2 ⟩, P 0 ⊂ X 0 = ⟨0, 0.2I 2 ⟩.
In the simulation, the reduction order of the proposed method is set as s = 5 and that of ZKF is set as 10. The simulation results are shown in Figure 2, which shows that the proposed method can obtain more accurate estimation results. Therefore, the proposed method is more suitable for the case with ellipsoid-bounded disturbances and noises. Case 2. The assumed ball-shaped uncertainty sets for w k , v k and x 0 in Case 1 put ZKF at a disadvantage w.r.t. the proposed method. To make a fair comparison, in this case, w k , v k and x 0 are randomly generated within the following zonotopes.

w k ∈ W k = ⟨0, 0.05I 2 ⟩, v k ∈ V k = ⟨0, 0.05I 2 ⟩, x 0 ∈ X 0 = ⟨0, 0.2I 2 ⟩.
The proposed method requires ellipsoid bundles bounding w k , c k and x 0 . If we use the smallest ellipsoids enclosing the corresponding assumed square-shaped uncertainty sets, the proposed method will be clearly put at a disadvantage. Note that W k , V k and X 0 can also be represented by ellipsoid bundles as follows.

W k = E(0, W k ), V k = E(0, V k ), X 0 = E(0, H 0 ), where The simulation results are shown in Figure 3 and4. They show that the estimation results obtained by ZKF and the proposed method are close, but the ones by the proposed method are more accurate. Therefore, the proposed method can also deal with the zonotope-bounded uncertainties and can even obtain more accurate estimation results than the zonotope-based method.

Theorem 1 .

 1 Given an m-order ellipsoid bundle E(c, H) ⊂ R n and a fix integer s(0 < s < m). Reordering E(0, P i ) in decreasing trace of P i yields E(c, H), where H = [ P σ(1) , . . . , P σ(m)] and σ(i) is a permutation of i = 1, . . . , m. Then, we can obtain an s-order ellipsoid bundleE ( c, R s (H) ) such that E(c, H) ⊆ E ( c, R s (H) ), where R s (H) = [ P σ(1) , . . . , P σ(s-1)

  where x k and x k satisfy { x k = xk + e k x k = xk + e k where e k = -e k and e k (j) = m ∑ i=1 √ P k i (j, j), j = 1, . . . , n x .

Fig. 2 .

 2 Fig. 2. The interval estimation of x k in case 1.

Fig. 3 .

 3 Fig. 3. The interval estimation of k (1) in case 2.

Fig. 4 .

 4 Fig. 4. The interval estimation of x k (2) in case 2. 7. CONCLUSION In this paper, we propose a new set representation tool, ellipsoid bundle, which combines certain advantages of

  ).

	Since e k ∈ E(0, H k ), then according to Definition 4, we
	have
	e k (j) = l T j e k ≤ ρ E(0,H k ) (l j ), j = 1, . . . , n x .
	It follows that
	e k ≤ e k .	(18)
	Note that E(0, H k ) is centrosymmetric, which implies that
	-e k ∈ E(0, H k ). In the same way, we can obtain -e k ≤ e k .
	It follows that
	e

k ≥ -e k = e k . (19) Since x k = xk + e k , then from (18) and (19), we have xk + e k ≤ x k ≤ xk + e k . 2 6. SIMULATION RESULTS


Huy Hieu Pham

Houssam Salmane
email: psalmane@clayair.io

Louahdi Khoudour
email: louahdi.khoudour@cerema.fr

Alain Crouzil
email: alain.crouzil@irit.fr

Sergio A Velastin
email: sergio.velastin@ieee.org

Pablo Zegers
email: pablozegers@gmail.com

A Unified Deep Framework for Joint 3D Pose Estimation and Action Recognition from a Single RGB Camera

Keywords: human action recognition, 30 pose estimation, RGB sensors, deep learning

We present a deep leaming-based multitask framework for joint 30 human pose estimation and action recognition from RGB sensors using simple cameras. The approach proceeds along two stages. In the first, a real-time 20 pose detector is run to determine the precise pixel location of important keypoints of the human body. A two-stream deep neural network is then designed and trained to map detected 20 keypoints into 30 poses. In the second stage, the Efficient Neural Architecture Search (ENAS) algorithm is deployed to find an optimal network architecture that is used for modeling the spatio-temporal evolution of the estimated 30 poses via an image-based intermediate representation and performing action recognition. Experiments on Human3.6M, MSR Action3O and SBU Kinect Interaction datasets verify the effectiveness of the proposed method on the targeted tasks. Moreover, we show that the method requires a low computational budget for training and inference. In particular, the experimental results show that by using a monocular RGB sensor, we can develop a 30 pose estimation and human action recognition approach that reaches the performance of RGB-depth sensors. This opens up many opportunities for leveraging RGB cameras (which are much cheaper than depth cameras and extensively deployed in private and public places) to build intelligent recognition systems.

Introduction

Hu.man Action Recognition (HAR) from videos has been researched for decades, since this topic plays a key role in various areas such as intelligent surveillance, human-robot interaction, robot vision and so on. Although significant progress has been achieved in the past few years, building an accurate, fast and efficient system for the recognition of actions in unseen videos is still a challenging task due to several obstacles, e.g., changes in camera viewpoint, occlusions, background, speed of motion, etc. Traditional approaches on video-based action recognition [START_REF] Weinland | A Survey of Vision-based Methods for Action Representation, Segmentation and Recognition[END_REF] have focused on extracting hand-crafted local features and building motion descriptors from RGB sensors. Many spatio-temporal representations of human motion have been proposed and widely exploited with success such as Scale Invariant Feature Transform (SIFT) [START_REF] Lowe | Distinctive Image Features from Scale-invariant Keypoints[END_REF], Histograms of Optical Flow (HOF) [START_REF] Laptev | Learning Realistic Human Actions from Movies[END_REF] or Cuboids [START_REF] Dollár | Behavior Recognition via Sparse Spatio-temporal Features[END_REF]. However, one of the major limitations of these approaches is the lack of 3D structure from the scene and recognizing human actions based only on RGB information is not enough to overcome the current challenges in the field.

The rapid development of depth-sensing time-of-flight sensor technology has helped in dealing with this problem, which is considered complex for traditional sensors. Low-cost and easy-to-use depth sensors can provide detailed 3D structural information of human motion. In particular, most of the current depth sensors have integrated real-time skeleton estimation and tracking frameworks [START_REF] Ye | Real-time Simultaneous Pose and Shape Estimation for Articulated Objects using a Single Depth Camera[END_REF], facilitating the collection of skeletal data. This is a high-level representation of the human body, which is suitable for the problem of motion analysis. Hence, exploiting skeletal data for 3D action recognition opens up opportunities for addressing the limitations of RGB-based solutions and many skeleton-based action recognition approaches have been proposed [START_REF] Wang | Mining Actionlet Ensemble for Action Recognition with Depth Cameras[END_REF][START_REF] Xia | View-Invariant Human Action Recognition using Histograms of 3D Joints[END_REF][START_REF] Chaudhry | Bio-inspired Dynamic 3D Discriminative Skeletal Features for Human Action Recognition[END_REF][START_REF] Vemulapalli | Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group[END_REF][START_REF] Ding | Profile HMMs for Skeleton-based Human Action Recognition[END_REF]. However, depth sensors have some significant drawbacks with respect to 3D pose estimation. For instance, they are only able to operate up to a limited distance and within a limited field of view. Moreover, a major drawback of low-cost depth sensors is their inability to work in bright light, especially sunlight [START_REF] Zhang | Microsoft Kinect Sensor and Its Effect[END_REF].

The focus in this paper is therefore to propose a 3D skeleton-based action recognition approach without the need for depth sensors. Specifically, we are interested in building a unified deep framework for both 3D pose estimation and action recognition from RGB video sequences provided by single color sensors. As shown in Figure 1, the approach consists of two stages. In the first, estimation stage, the system recovers the 3D human poses from the input RGB video. In the second, recognition stage, an action recognition approach is developed and stacked on top of the 3D pose estimator in a unified framework, where the estimated 3D poses are used as inputs to learn the spatio-temporal motion features and predict action labels.

Figure 1. Overview of the proposed method. In the estimation stage, a real-time multi-person 2D pose detector, such as OpenPose [START_REF] Cao | Realtime Multi-person 2D Pose Estimation using Part Affinity Fields[END_REF] or AlphaPose [START_REF] Fang | RMPE: Regional Multi-person Pose Estimation[END_REF], is used to generate 2D human body keypoints. A deep neural network is then trained to produce 3D poses from the 2D detections. In the recognition stage, the 3D estimated poses are encoded into a compact image-based representation and finally fed into a deep convolutional network for supervised classification task, which is automatically searched by the ENAS algorithm [START_REF] Pham | Efficient Neural Architecture Search via Parameters Sharing[END_REF].

There are four hypotheses that motivate us to build a deep learning framework for human action recognition from 3D poses. First, actions can be correctly represented through 3D pose movements [START_REF] Johansson | Visual Motion Perception[END_REF][START_REF] Gu | Action and Gait Recognition from Recovered 3D Human Joints[END_REF]. Second, the 3D human pose has a high-level of abstraction with much less complexity compared to RGB and depth streams. This makes the training and inference processes much simpler and faster. Third, depth sensors can provide highly accurate skeletal data for 3D action recognition. However, they are expensive and not always available (e.g., for outdoor scenes).

A fast and accurate approach of 3D pose estimation from only RGB input is highly desirable. Fourth, state-of-the-art 2D pose detectors [START_REF] Cao | Realtime Multi-person 2D Pose Estimation using Part Affinity Fields[END_REF][START_REF] Fang | RMPE: Regional Multi-person Pose Estimation[END_REF][START_REF] Newell | Stacked Hourglass Networks for Human Pose Estimation[END_REF] are able to provide 2D poses with a high degree of accuracy in real time. Meanwhile, deep networks have proved their capacity to learn complex functions from high-dimensional data. Hence, a simple network model can also learn a mapping to convert 2D poses into 3D. The effectiveness of the proposed method is evaluated on public benchmark datasets (Human3.6M [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF] for 3D pose estimation and MSR Action3D [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF] and SBU [START_REF] Yun | Two-person Interaction Detection using Body-pose Features and Multiple Instance Learning[END_REF] for action recognition). Beyond the initial expectations, the experimental results demonstrate state-of-the-art performance on the targeted tasks (Section 4.3) and support the hypotheses above. Furthermore, we show that this approach has a low computational cost (Section 4.4). Overall, our main contributions are as follows:

• First, we present a two-stream, lightweight neural network to recover 3D human poses from RGB images provided by a monocular camera. The proposed method achieves state-of-the-art result on 3D human pose estimation task and benefits action recognition. The novelty of the study is that a very simple deep neural network could be trained effectively to learn a 2D-to-3D mapping for the task of 3D human estimation from color sensors.

•

Second, we propose to put an action recognition approach on top of the 3D pose estimator to form a unified framework for 3D pose-based action recognition. It takes the 3D estimated poses as inputs, encodes them into a compact image-based representation and finally feeds to a deep convolutional network, which is designed automatically by using a neural architecture search algorithm. Surprisingly, the experiments show that we reached state-of-the-art results on this task, even when compared with methods using depth cameras.

The rest of this paper is organized as follows. A review of the related work is presented in Section 2. The proposed method is explained in Section 3. Experiments are provided in Section 4 and Section 5 concludes the paper.

Related Work

This section reviews two main topics that are directly related to the proposed approach, i.e., 3D pose estimation from RGB images and 3D pose-based action recognition. An extensive literature review is beyond the scope of this section. Instead, the interested reader is referred to the surveys of Sarafianos et al. [START_REF] Nikolaos | 3D Human Pose Estimation: A Review of the Literature and Analysis of Covariates[END_REF] for recent advances in 3D human pose estimation and Presti et al. [START_REF] Presti | 3D Skeleton-based Human Action Classification: A Survey[END_REF] for 3D skeleton-based action recognition.

3D Human Pose Estimation

The problem of 3D human pose estimation has been intensively studied in recent years. Almost all early approaches for this task were based on feature engineering [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF][START_REF] Sminchisescu | 3D Human Motion Analysis in Monocular Video Techniques and Challenges[END_REF][START_REF] Ramakrishna | Reconstructing 3D Human Pose from 2D Image Landmarks[END_REF], while the current state-of-the-art methods are based on deep neural networks [START_REF] Li | 3D Human Pose Estimation from Monocular Images with Deep Convolutional Neural Network[END_REF][START_REF] Tekin | Direct Prediction of 3D Body Poses from Motion Compensated Sequences[END_REF][START_REF] Pavlakos | Coarse-to-fine Volumetric Prediction for Single-image 3D Human Pose[END_REF][START_REF] Pavllo | 3D Human Pose Estimation in Video with Temporal Convolutions and Semi-supervised Training[END_REF][START_REF] Mehta | VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera[END_REF][START_REF] Katircioglu | Learning Latent Representations of 3D Human Pose with Deep Neural Networks[END_REF]. Many of them are regression-based approaches that directly predict 3D poses from RGB images via 2D/3D heatmaps. For instance, Li et al. [START_REF] Li | 3D Human Pose Estimation from Monocular Images with Deep Convolutional Neural Network[END_REF] designed a deep convolutional network for human detection and pose regression. The regression network learns to predict 3D poses from single images using the output of a body part detection network. Tekin et al. [START_REF] Tekin | Direct Prediction of 3D Body Poses from Motion Compensated Sequences[END_REF] proposed to use a deep network to learn a regression mapping that directly estimates the 3D pose in a given frame of a sequence from a spatio-temporal volume centered on it. Pavlakos et al. [START_REF] Pavlakos | Coarse-to-fine Volumetric Prediction for Single-image 3D Human Pose[END_REF] used multiple fully convolutional networks to construct a volumetric stacked hourglass architecture, which can recover 3D poses from RGB images. Pavllo et al. [START_REF] Pavllo | 3D Human Pose Estimation in Video with Temporal Convolutions and Semi-supervised Training[END_REF] exploited a temporal dilated convolutional network [START_REF] Fisher | Multi-scale Context Aggregation by Dilated Convolutions[END_REF] for estimating 3D poses. However, this approach led to a significant increase in the number of parameters as well as the required memory. Mehta et al. [START_REF] Mehta | VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera[END_REF] introduced a real-time approach to predict 3D poses from a single RGB sensor. They used ResNets [START_REF] He | Deep Residual Learning for Image Recognition[END_REF] to jointly predict 2D and 3D heatmaps as regression tasks. Recently, Katircioglu et al. [START_REF] Katircioglu | Learning Latent Representations of 3D Human Pose with Deep Neural Networks[END_REF] introduced a deep regression network for predicting 3D human poses from monocular images via 2D joint location heatmaps. This architecture is in fact an overcomplete autoencoder that learns a high-dimensional latent pose representation and accounts for joint dependencies, in which a Long Short-Term Memory (LSTM) network [START_REF] Sepp | Long Short-Term Memory[END_REF] is used to enforce temporal consistency on 3D pose predictions.

To the best of our knowledge, several studies [START_REF] Pavlakos | Coarse-to-fine Volumetric Prediction for Single-image 3D Human Pose[END_REF][START_REF] Mehta | VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera[END_REF][START_REF] Katircioglu | Learning Latent Representations of 3D Human Pose with Deep Neural Networks[END_REF] stated that regressing the 3D pose from 2D joint locations is difficult and not too accurate. However, motivated by Martinez et al. [START_REF] Martinez | A Simple Yet Effective Baseline for 3D Human Pose Estimation[END_REF], we believe that a simple neural network can learn effectively a direct 2D-to-3D mapping. Therefore, this paper aims at proposing a simple, effective and real-time approach for 3D human pose estimation that benefits action recognition. To this end, a two-stream deep neural network that performs 3D pose predictions from the 2D human poses is designed and optimized. These 2D poses are generated by a state-of-the-art 2D detector, which can run in real time for multiple people. We empirically show that although the proposed approach is computationally inexpensive, it is still able to improve the state-of-the-art.

3D Pose-Based Action Recognition

Human action recognition from skeletal data or 3D poses is a challenging task. The methods used in previous works on this topic can be divided into two main groups. The first group [START_REF] Wang | Mining Actionlet Ensemble for Action Recognition with Depth Cameras[END_REF][START_REF] Vemulapalli | Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group[END_REF][START_REF] Lv | Recognition and Segmentation of 3D Human Action Using HMM and Multi-class AdaBoost[END_REF] extracts hand-crafted features and uses probabilistic graphical models, e.g., Hidden Markov Model (HMM) [START_REF] Lv | Recognition and Segmentation of 3D Human Action Using HMM and Multi-class AdaBoost[END_REF] or Conditional Random Field (CRF) [START_REF] Han | Discriminative Human Action Recognition in the Learned Hierarchical Manifold Space[END_REF] to recognize actions. However, almost all these approaches require a lot of feature engineering. The second group [START_REF] Liu | Spatio-temporal LSTM with Trust Gates for 3D Human Action Recognition[END_REF][START_REF] Du | Hierarchical Recurrent Neural Network for Skeleton based Action Recognition[END_REF][START_REF] Shahroudy | A Large Scale Dataset for 3D Human Activity Analysis[END_REF] considers the 3D pose-based action recognition as a time-series problem and proposes to use Recurrent Neural Networks with Long Short-Term Memory units (RNN-LSTMs) [START_REF] Sepp | Long Short-Term Memory[END_REF] for modeling the dynamics of the skeletons. Although RNN-LSTMs are able to model the long-term temporal characteristics of motion and have advanced the state-of-the-art, this approach feeds raw 3D poses directly into the network and just considers them as a kind of low-level feature. The large number of input features makes RNNs very complex and may easily lead to overfitting. Moreover, many RNN-LSTMs act merely as classifiers and cannot extract high-level features for recognition tasks [START_REF] Sainath | Long Short-Term Memory, Fully Connected Deep Neural Networks[END_REF].

In the literature, 3D human pose estimation and action recognition are closely related. However, both problems are generally considered to be two distinct tasks [START_REF] Chéron | Pose-based CNN Features for Action Recognition[END_REF]. Although some approaches have been proposed for tackling the problem of jointly predicting 3D poses and recognizing actions in RGB images or video sequences [START_REF] Yao | Modeling Mutual Context of Object and Human Pose in Human-object Interaction Activities[END_REF][START_REF] Nie | Joint Action Recognition and Pose Estimation from Video[END_REF][START_REF] Luvizon | 2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning[END_REF], they are data-dependent and require a lot of feature engineering, except the work of Luvizon et al. [START_REF] Luvizon | 2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning[END_REF]. Unlike previous studies, a multitask learning framework for 3D pose-based action recognition is proposed here by reconstructing 3D skeletons from RGB images and exploiting them for action recognition in a joint way. Experimental results on public and challenging datasets show that the framework can solve the two tasks in an effective way.

Proposed Method

This section presents the proposed method. First, the approach for 3D human pose estimation is presented, followed by the proposed solution for 3D pose-based action recognition.

Problem Definition

Given an RGB video clip of a person who starts to perform an action at time t = 0 and ends at t = T, the problem studied in this work is to generate a sequence of 3D poses P = (p 0 , ..., p T ), where p i ∈ R 3×M , i ∈ {0, ..., T} at the estimation stage, in which M denotes the number of keypoints for the pose p i . The generated P is then used as input for the recognition stage to predict the corresponding action label A by a supervised learning model. See Figure 1 for an illustration of the problem.

3D Human Pose Estimation

Given an input RGB image I ∈ R W×H×3 , we aim to estimate the body joint locations in the 3-dimensional space, noted as p3D ∈ R 3×M . To this end, we first run any state-of-the-art human 2D pose detector, in this case OpenPose [START_REF] Cao | Realtime Multi-person 2D Pose Estimation using Part Affinity Fields[END_REF], to produce a series of 2D keypoints p 2D ∈ R 2×N . To recover the 3D joint locations, we try to learn a direct 2D-to-3D mapping f r : p 2D f r -→ p3D . This transformation can be implemented by a deep neural network in a supervised manner

p3D = f r (p 2D , θ), ( 1
)
where θ is a set of trainable parameters of the function f r . To optimize f r , the prediction error is minimized over a labelled dataset of C poses by solving the optimization problem arg min

θ 1 C C ∑ n=1 L( f r (x i ), y i ). (2)
Here x i and y i are the input 2D poses and the ground truth 3D poses, respectively; L denotes a loss function. Here, the robust Huber loss [START_REF] Huber | Robust Estimation of a Location Parameter[END_REF] is used to deal with outliers.

Network Design

State-of-the-art deep learning architectures such as ResNet [START_REF] He | Deep Residual Learning for Image Recognition[END_REF], Inception-ResNet-v2 [START_REF] Christian | Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning[END_REF], DenseNet [START_REF] Gao | Densely Connected Convolutional Networks[END_REF], or NASNet [START_REF] Barret | Neural Architecture Search with Reinforcement Learning[END_REF] have achieved an impressive performance in supervised learning tasks with high-dimensional data, e.g., 2D or 3D images. However, the use of these architectures [START_REF] He | Deep Residual Learning for Image Recognition[END_REF][START_REF] Christian | Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning[END_REF][START_REF] Gao | Densely Connected Convolutional Networks[END_REF][START_REF] Barret | Neural Architecture Search with Reinforcement Learning[END_REF] on low-dimensional data like the coordinates of the 2D human joints could lead to overfitting. Therefore, the design is based on a simple and lightweight multilayer network architecture without the convolution operations. The design process exploits some recent improvements in the optimization of the modern deep learning models [START_REF] He | Deep Residual Learning for Image Recognition[END_REF][START_REF] Gao | Densely Connected Convolutional Networks[END_REF]. Concretely, a two-stream network is proposed. Each stream comprises linear layers, Batch Normalization (BN) [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF], Dropout [START_REF] Hinton | Improving Neural Networks by Preventing Co-adaptation of Feature Detectors[END_REF], SELU [START_REF] Klambauer | Self-Normalizing Neural Networks[END_REF] and Identity connections [START_REF] He | Deep Residual Learning for Image Recognition[END_REF]. During the training phase, the first stream takes the ground truth 2D locations as input. The 2D human joints predicted by OpenPose [START_REF] Cao | Realtime Multi-person 2D Pose Estimation using Part Affinity Fields[END_REF] are inputted to the second stream. The outputs of the two streams are then averaged. Figure 2 illustrates the network design. Please note that learning with the ground truth 2D locations for both of these streams could lead to a higher level of performance. However, training with the 2D OpenPose detections could improve the generalization ability of the network and makes it more robust during inference, when only the OpenPose's 2D output is used to deal with action recognition in the wild.

3D Pose-Based Action Recognition

This section describes the integration of the estimation stage with the recognition stage in a unified framework. Specifically, the proposed recognition approach is stacked on top of the 3D pose estimator.

To explore the high-level information of the estimated 3D poses, they are encoded into a compact image-based representation. These intermediate representations are then fed to a Deep Convolutional Neural Network (D-CNNs) for learning and classifying actions. This idea has been proven effective in our previous works [START_REF] Pham | Exploiting Deep Residual Networks for Human Action Recognition from Skeletal Data[END_REF][START_REF] Pham | Skeletal Movement to Color Map: A Novel Representation for 3D Action Recognition with Inception Residual Networks[END_REF][START_REF] Pham | Spatio-Temporal Image Representation of 3D Skeletal Movements for View-Invariant Action Recognition with Deep Convolutional Neural Networks[END_REF]. Thus, the spatio-temporal patterns of a 3D pose sequence are transformed into a single color image as a global representation called Enhanced-SPMF [START_REF] Pham | Spatio-Temporal Image Representation of 3D Skeletal Movements for View-Invariant Action Recognition with Deep Convolutional Neural Networks[END_REF] via two important elements of a human movement: 3D poses and their articulation joint motions as shown in Figure 3. For a detailed technical description of the Enhanced-SPMF the interested reader is referred to the work described in [START_REF] Pham | Spatio-Temporal Image Representation of 3D Skeletal Movements for View-Invariant Action Recognition with Deep Convolutional Neural Networks[END_REF]. Figure 4 visualizes some Enhanced-SPMF representations from samples of the MSR Action3D dataset [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF]. For learning and classifying the obtained images, the use of the Efficient Neural Architecture Search (ENAS) [START_REF] Pham | Efficient Neural Architecture Search via Parameters Sharing[END_REF]-a recent state-of-the-art technique for automatic design of deep neural networks, is proposed. ENAS is in fact an extension of an important advance in deep learning called NAS [START_REF] Barret | Neural Architecture Search with Reinforcement Learning[END_REF], which can automate the designing process of convolutional architectures on a dataset of interest. The method searches for optimal building blocks (called cells, including normal cells and reduction cells) and the final architecture is then constructed from the best cells. Figure 5 shows a typical CNN architecture that is generated by ENAS. In NAS, an RNN is used. It first samples a candidate architecture called child model. This child model is then trained to converge on the desired task and to report its performance. Next, the RNN uses the performance as a guiding signal to find a better architecture. This process is repeated many times, making NAS computationally expensive and time-consuming (e.g., on CIFAR-10, NAS needs 4 days with 450 GPUs to discover the best architecture). The main limitation of NAS is that the training of each child model to convergence requires a significant amount of time and computational resources as it measures model accuracy while throwing away all the trained weights. Therefore, ENAS has been proposed to improve the efficiency of NAS. Its key idea [START_REF] Pham | Efficient Neural Architecture Search via Parameters Sharing[END_REF] is the use of shared parameters among child models, which helps reducing the training times of each child model from scratch to convergence. State-of-the-art performance has been achieved by ENAS on well-known public datasets. We encourage the readers to refer to the original paper [START_REF] Pham | Efficient Neural Architecture Search via Parameters Sharing[END_REF] for more details. Figure 6 illustrates the entire pipeline of our approach for the recognition stage.

Experiments

Datasets and Settings

The proposed method is evaluated on three challenging datasets: Human3.6M, MSR Action3D and SBU Kinect Interaction. Human3.6M is used for evaluating 3D pose estimation. Meanwhile, the other two datasets are used for evaluating action recognition. The characteristics of each dataset are as follows.

Human3.6M [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF]: This is a very large-scale dataset containing 3.6 million different 3D articulated poses captured from 11 actors for 17 actions, under 4 different viewpoints. For each subject, the dataset provides 32 body joints, from which only 17 joints are used for training and computing scores. In particular, 2D joint locations and 3D poses ground truth are available for evaluating supervised learning models. [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF]: This dataset contains 20 actions, performed by 10 subjects. Experiment were conducted on 557 video sequences of the MSR Action3D, in which the whole dataset is divided into three subsets: AS1, AS2, and AS3. There are 8 actions classes for each subset. Half of the data is selected for training and the rest is used for testing. [START_REF] Yun | Two-person Interaction Detection using Body-pose Features and Multiple Instance Learning[END_REF]: This dataset contains a total of 300 interactions, performed by 7 participants for 8 actions. This is a challenging dataset as it contains pairs of actions that are difficult to distinguish such as "exchanging objects-shaking hands" or "pushing-punching". The dataset is randomly split into 5 folds, in which 4 folds are used for training and the remaining 1 fold is used for testing.

MSR Action3D

SBU Kinect Interaction

Implementation Details

The proposed networks were implemented in Python with Keras/TensorFlow backend. The two streams of the 3D pose estimator are trained separately with the same hyperparameters setting, in which mini batches of 128 poses are used with 0.25 dropout rate. The weights are initialized with He initialization [START_REF] He | Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification[END_REF]. Adam optimizer [START_REF] Kingma | A Method for Stochastic Optimization[END_REF] is used with default parameters. The initial learning rate is set to 0.001 and is decreased by a factor of 0.5 after every 50 epochs. The network is trained for 300 epochs from scratch on the Human3.6M dataset [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF]. For action recognition task, OpenPose is run [START_REF] Cao | Realtime Multi-person 2D Pose Estimation using Part Affinity Fields[END_REF] to generate 2D detections on MSR Action3D [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF] and SBU Kinect Interaction [START_REF] Yun | Two-person Interaction Detection using Body-pose Features and Multiple Instance Learning[END_REF]. The 3D pose estimator pre-trained on Human3.6M [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF] is then used to provide 3D poses. Standard data pre-processing and augmentation techniques are used, such as randomly cropping and flipping on these two datasets due to their small sizes. To discover optimal recognition networks, ENAS [START_REF] Pham | Efficient Neural Architecture Search via Parameters Sharing[END_REF] is used with the same parameter setting as the original work. Concretely, the shared parameters ω are trained with Nesterov's accelerated gradient descent [START_REF] Yurii | A Method for Solving a Convex Programming Problem with Convergence Rate O(1/K2)[END_REF] using Cosine learning rate [START_REF] Ilya | Stochastic Gradient Descent with Warm Restarts[END_REF]. The candidate architectures are initialized by He initialization [START_REF] He | Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification[END_REF] and trained by Adam optimizer [START_REF] Kingma | A Method for Stochastic Optimization[END_REF] with a learning rate of 0.00035. Additionally, each search is run for 200 epochs.

Experimental Results and Comparison

Evaluation on 3D Pose Estimation

The effectiveness of the proposed 3D pose estimation network is evaluated using the standard protocol of the Human3.6M dataset [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF][START_REF] Pavlakos | Coarse-to-fine Volumetric Prediction for Single-image 3D Human Pose[END_REF][START_REF] Mehta | VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera[END_REF][START_REF] Martinez | A Simple Yet Effective Baseline for 3D Human Pose Estimation[END_REF]. Five subjects S1, S5, S6, S7, S8 are used for training and the remaining two subjects S9, S11 are used for evaluation. Experimental results are reported by the average error in millimeters between the ground truth and the corresponding predictions over all joints. Much to our surprise, this method outperforms the previous best result from the literature [START_REF] Martinez | A Simple Yet Effective Baseline for 3D Human Pose Estimation[END_REF] by 3.1mm, corresponding to an error reduction of 6.8% even when combining the ground truth 2D locations with the 2D OpenPose detections. This result proves that the network design can learn to recover the 3D pose from the 2D joint locations with a remarkably low error rate, which to the best of our knowledge, has established a new state-of-the-art on 3D human pose estimation (see Table 1 and Figure 7). Table 1. Experimental results (average error in mm) and comparison with previous state-of-the-art 3D pose estimation approaches on the Human3.6M dataset [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF]. The symbol denotes that a 2D detector was used and the symbol † denotes the ground truth 2D joint locations were used. Figure 7. Visualization of 3D output of the estimation stage with some samples on the test set of Human3.6M [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF]. For each example, from left to right are 2D poses, 3D ground truths and the 3D predictions, respectively.

Method

Evaluation on Action Recognition

Table 2 reports the experimental results and comparisons with state-of-the-art methods on the MSR Action3D dataset [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF]. The ENAS algorithm [START_REF] Pham | Efficient Neural Architecture Search via Parameters Sharing[END_REF] is able to explore a diversity of network architectures and the best design is identified based on its validation score. Thus, the final architecture achieved a total average accuracy of 97.98% over three subset AS1, AS2 and AS3. This result outperforms many previous studies [START_REF] Vemulapalli | Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group[END_REF][START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF][START_REF] Liu | Spatio-temporal LSTM with Trust Gates for 3D Human Action Recognition[END_REF][START_REF] Du | Hierarchical Recurrent Neural Network for Skeleton based Action Recognition[END_REF][START_REF] Chen | Real-time Human Action Recognition based on Depth Motion Maps[END_REF][START_REF] Wang | Graph Based Skeleton Motion Representation and Similarity Measurement for Action Recognition[END_REF][START_REF] Weng | Spatio-Temporal Naive-Bayes Nearest-Neighbor (ST-NBNN) for Skeleton-Based Action Recognition[END_REF][START_REF] Xu | Spatio-temporal Pyramid Model based on Depth Maps for Action Recognition[END_REF][START_REF] Lee | Ensemble Deep Learning for Skeleton-based Action Recognition using Temporal Sliding LSTM Networks[END_REF], and among them, many are depth sensor-based approaches. Figure 8 provides a schematic diagram of the best cells and optimal architecture found by ENAS on the AS1 subset [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF]. For the SBU Kinect Interaction dataset [START_REF] Yun | Two-person Interaction Detection using Body-pose Features and Multiple Instance Learning[END_REF], the best model achieved an accuracy of 96.30%, as shown in Table 3. The results reported here indicated an important observation that by using only the 3D predicted poses, it was possible to outperform previous works reported in [START_REF] Liu | Spatio-temporal LSTM with Trust Gates for 3D Human Action Recognition[END_REF][START_REF] Song | An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data[END_REF][START_REF] Weng | Discriminative Spatio-Temporal Pattern Discovery for 3D Action Recognition[END_REF][START_REF] Ke | A New Representation of Skeleton Sequences for 3D Action Recognition[END_REF][START_REF] Yusuf | CNN-based Action Recognition and Supervised Domain Adaptation on 3D Body Skeletons via Kernel Feature Maps[END_REF][START_REF] Wang | Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks[END_REF][START_REF] Liu | Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks[END_REF] and reach state-of-the-art results provided in [START_REF] Pham | Spatio-Temporal Image Representation of 3D Skeletal Movements for View-Invariant Action Recognition with Deep Convolutional Neural Networks[END_REF][START_REF] Zhang | View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition[END_REF], which deploy accurate skeletal data provided by Kinect v2 sensor. on AS1 subset [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF]. They were then used to construct the final network architecture (c). We recommend the interested readers to see [START_REF] Pham | Efficient Neural Architecture Search via Parameters Sharing[END_REF] to better understand this procedure.

Computational Efficiency Evaluation

On a single GeForce GTX 1080Ti GPU with 11GB memory, the runtime of OpenPose [START_REF] Cao | Realtime Multi-person 2D Pose Estimation using Part Affinity Fields[END_REF] is less than 0.1s per frame for an image size of 800 × 450 pixels. On the Human3.6M dataset [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF], the 3D pose estimation stage takes around 15ms to complete a pass (forward + backward) through each stream with a mini batches of size 128. Each epoch was done within 3 min. For the action recognition stage, our implementation of the ENAS algorithm takes about 2 h to find the final architecture (∼2.3M parameters) on each subset of MSR Action3D dataset [START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF], while it takes around 3 h on the SBU Kinect Interaction dataset [START_REF] Yun | Two-person Interaction Detection using Body-pose Features and Multiple Instance Learning[END_REF] to discover the best architecture (∼3M parameters). With small architecture sizes, the discovered networks require low computing time for the inference stage, making the approach more practical for large-scale problems and real-time applications.

Conclusions

In this paper, a unified deep learning framework for joint 3D human pose estimation and action recognition from RGB video sequences has been presented. The proposed method first runs a state-of-the-art 2D pose detector to estimate 2D locations of body joints from a monocular RGB sensor, although the approach is not limited to a particular 2D pose detector. A deep neural network was then designed and trained to learn a direct 2D-to-3D mapping and predict human poses in 3D space. Experimental results demonstrated that the 3D human poses can be effectively estimated by a simple network design and training methodology over 2D keypoints. A novel action recognition approach was also introduced based on a compact image-based representation and automated machine learning, in which an advanced neural architecture search algorithm was exploited to discover the best performing architecture for each recognition task. The experiments on public and challenging action recognition datasets indicated that the proposed framework was able to reach state-of-the-art performance, while requiring less computation budget for training and inference. Despite that, this method naturally depends on the quality of the output of the 2D detectors. Hence, a limitation is that it cannot recover 3D poses from 2D failed output. To tackle this problem, we are currently expanding this study by adding more visual evidence to the network to further gains in performance. The preliminary results are encouraging. Codes and models will be made available on our GitHub project at https://github.com/huyhieupham/.

Figure 2 .

 2 Figure 2. Diagram of the proposed two-stream network for training the 3D pose estimator.

Figure 3 .

 3 Figure 3. Illustration of the Enhanced-SPMF representation. To build an Enhanced-SPMF map from skeletal data, each skeleton sequence is first encoded as a single RGB image via a skeleton-based representation called SPMF (Skeleton Pose-Motion Feature) [53]. The SPMF is built from Pose Feature vectors (PFs) and Motion Feature vectors (MFs), which are calculated from the 3D coordinates of skeletons. Finally, we use a color enhancement technique [55] to enhance the local textures of the SPMF to form the Enhanced-SPMF. This is an image-based global representation for the whole input skeleton sequences. Figure reproduced, by permission from the publishers, from our previous work in [56].

Figure 4 .

 4 Figure 4. Immediate image-based representations for the recognition stage.

Figure 5 .

 5 Figure 5. Illustration of a deep neural network generated by ENAS that contains 3 blocks, each with N optimal convolution cells and one reduction cell.

Figure 6 .

 6 Figure 6. Illustration of the proposed approach for 3D pose-based action recognition.

Figure 8 .

 8 Figure 8. Diagram of the top performing normal cell (a) and reduction cell (b) discovered by ENAS[START_REF] Pham | Efficient Neural Architecture Search via Parameters Sharing[END_REF] on AS1 subset[START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF]. They were then used to construct the final network architecture (c). We recommend the interested readers to see[START_REF] Pham | Efficient Neural Architecture Search via Parameters Sharing[END_REF] to better understand this procedure.

Table 2 .

 2 Test accuracies (%) on the MSR Action3D dataset[START_REF] Li | Action Recognition Based on a Bag of 3D Points[END_REF]. Please note that many previous methods are based on depth sensors.

	Method	AS1	AS2	AS3	Aver.
	Li et al. [19]	72.90	71.90	71.90	74.70
	Chen et al. [67]	96.20	83.20	92.00	90.47
	Vemulapalli et al. [9]	95.29	83.87	98.22	92.46
	Du et al. [38]	99.33	94.64	95.50	94.49
	Liu et al. [37]	N/A	N/A	N/A	94.80
	Wang et al. [68]	93.60	95.50	95.10	94.80
	Wang et al. [69]	91.50	95.60	97.30	94.80
	Xu et al. [70]	99.10	92.90	96.40	96.10
	Lee et al. [71]	95.24	96.43	100.0	97.22
	Pham et al. [54]	98.83	99.06	99.40	99.10
	Ours	97.87	96.81	99.27	97.98

Table 3 .

 3 Test accuracies (%) on the SBU Kinect Interaction dataset[START_REF] Yun | Two-person Interaction Detection using Body-pose Features and Multiple Instance Learning[END_REF]. Please note that many previous methods are based on depth sensors.

	Method	Acc.
	Song et al. [72]	91.51
	Liu et al. [37]	93.30
	Weng et al. [73]	93.30
	Ke et al. [74]	93.57
	Tas et al. [75]	94.36
	Wang et al. [76]	94.80
	Liu et al. [77]	94.90
	Zang et al. [78] (using VA-RNN)	95.70
	Zhang et al. [78] (using VA-CNN)	97.50
	Pham et al. [54]	97.86
	Ours	96.30

Acknowledgments: This research was carried out at the Cerema Research Center and Informatics Research Institute of Toulouse, Paul Sabatier University, France. The authors would like to express our thanks to all the people who have made helpful comments and suggestions on a previous draft. S.A. Velastin is grateful to funding received from the Universidad Carlos III de Madrid, the European Union's Seventh Framework Programme for research, technological development and demonstration under grant agreement no. 600371, el Ministerio de Economía y Competitividad (COFUND2013-51509) and Banco Santander.

Funding: Sergio A. Velastin is grateful for funding received from the Universidad Carlos III de Madrid, the European Union's Seventh Framework Programme for research, technological development and demonstration under grant agreement N 600371, el Ministerio de Economía, Industria y Competitividad (COFUND2013-51509) el Ministerio de Educación, Cultura y Deporte (CEI-15-17) and Banco Santander.

Author Contributions: Conceptualization, methodology, software, validation, investigation, resources, visualization, writing-original draft preparation, H.H.P.; Data curation, formal analysis, writing-review and editing, all authors; Supervision; project administration; funding acquisition, H.S., A.C., P.Z., L.K. and S.A.V. All authors have read and agreed to the published version of the manuscript.

Conflicts of Interest:

The authors declare no conflict of interest.