Y Lee

M Lannoo

N Cavassilas

M Luisier

M Bescond

Efficient quantum modeling of inelastic interactions in nano-devices

This work presents an efficient direct quantum method to model inelastic scattering in nanoelectronic structures including degenerate band extrema. It couples the Born series expansion of the non-equilibrium Green's function (NEGF) to an analytic continuation based on the Padé approximant technique. Using a two-band k • p Hamiltonian, we analyze the electron transport through a linear chain in presence of both optical and acoustic phonons. Results are consistently compared with the usual, computationally expensive, self-consistent Born approximation (SCBA). We find that our approach provides a much better convergence for both types of phonons in presence of strong multi-band coupling. The calculation of the current to the 5 th order in the interactions is sufficient to reproduce the influence of all considered phonon interactions. We also show that the method can be applied to the calculation of the density of carriers which depicts however a slower convergence rate than the current. The capability to efficiently calculate both current and charge density represents a clear advantage in a context of increasing request for atomistic quantum simulations.

I. INTRODUCTION

The great progress of semiconductor industry over the last 30 years has been accompanied by an impressive development of the quantum transport modeling of nanostructures and devices [START_REF] Datta | Electronic Transport in Mesoscopic Systems[END_REF] . Indeed experimental advances in mesoscopic physics and innovations in nanoelectronics require the use of sophisticated quantum transport simulation tools capable to capture the physical properties of such systems. One of the most difficult part in quantum transport modeling is to manage the computational burden induced by the treatment of inelastic interactions between electrons and other particles like phonons or photons. Unfortunately, these interactions are found to have a non-negligible influence and even in some cases can be at the origin of the working principle of the device. This is for instance the case of electron-photon interaction in solar cells.

Several quantum approaches have been intensively developed during the last decade to address this issue [START_REF] Fischetti | [END_REF][3][4][5][6] . Among them the non-equilibrium Green's function (NEGF) formalism appears as one of the most efficient methods to include interactions in quantum transport codes using the concept of self-energies [START_REF] Haug | Quantum Kinetics in Transport and Optics of Semiconductors[END_REF] . This approach has been successfully applied to various nano-structures and devices, including quantum dots 8,9 , molecular junctions 10 , 2D materials 11 , resonant tunneling diodes 12 , semiconductor transistors [13][14][15][16][17][18][19] and solar cells [20][21][22] . Despite significant improvements the NEGF method still requires heavy computational resources when dealing with interactions in realistic nanostructures. Indeed electron-phonon/photon interactions are usually treated within the self-consistent Born approximation (SCBA) through an iterative resolution of the Dyson equation. Self-consistency is often believed to be "the price to pay" to obey all conservation laws and to obtain physically meaningful solutions. However, we recently demonstrated that this is not the case. The calculated self-energies must not be necessarily obtained through a self-consistent scheme, but more importantly via the derivative of a functional φ in the picture of Luttinger and Ward 23,24 . Considering a lowest order approximation (LOA) for the interaction we first demonstrated in Ref. 25 that self-consistency is sufficient, but not necessary to conserve the current. Furthermore, in Ref. 26 we have proposed an infinite family of non-self-consistent, but conserving approaches based on an extension of the LOA relying on the Green's function Born series. This extended LOA technique coupled with an analytic continuation of Padé approximants [START_REF] Hardy | Divergent Series[END_REF][START_REF] Baker | Padé Approximants[END_REF] has been shown to provide excellent expectation values of the current (at least very close to those obtained with SCBA). The first order version of the analytically continuated LOA has been then applied to the calculation of current characteristics with phonon scattering in double-gate [START_REF] Cavassilas | [END_REF] and nanowire 30 n-type transistors.

The aim of the present work is twofold. Firstly, we want to precisely describe the computational scheme of the so-called analytically continuated LOA technique and to assess its limits in providing accurate current expectation values in presence of phonon scattering. To do so we will consider an ideal 1D system described by different Hamiltonians. Two-band k • p models will be used to assess the validity of the approach when multi-band transport becomes predominant. The calculation of the charge density is also an important point in the context of nanostructure quantum transport modeling. Indeed, the calculation of the current requires the determination of the electrostatic potential which is itself computed, in most cases, via a self-consistent loop between the charge density and the Poisson equation. We will therefore describe the computation of charge densities within the analytically continuated LOA approach and will apply it to the Hamiltonians previously mentioned. Such a validated approach represents a significant reduction of the computational complexity with respect to the SCBA since it provides a general recipe to efficiently calculate both current and charge densities in non-equilibrium quantum systems.

The rest of the paper is organized as follows: In Sec.II we will describe the implementation of the LOA starting from the SCBA. Higher order LOA and analytic continuation techniques will be presented in the sub-sections for both current and charge density in the presence of electron-phonon interactions (acoustic and optical). Section III will show the details of the computational implementation of this scheme. Section IV will present the various results for each types of Hamiltonian and phonon interaction intensity. Finally, Section V will summarize the key findings.

II. THEORETICAL FRAMEWORK

In this section, theoretical details of the LOA are introduced starting from the comparison with the usual SCBA. In particular we focus on the physical and computational advantages of the LOA over the SCBA. Thereafter, based on this scheme we will provide the generalized LOA method to calculate the interacting Green's function to any order. In addition, we will apply analytic continuations based on the Padé approximants for the calculation of currents and charge densities.

A. Self-Consistent Born Approximation

Inelastic interactions in the NEGF framework are usually treated using the SCBA in which scattering processes are treated via self-energy contributions [13][14][15]19 . The calculation of these self-energies is performed using the Dyson equation, which can be expressed in the simplified matrix notation [1 = (r 1 , t 1 ), g 0 Σ[G]G = d2 d2 g 0 (1; 2)Σ(2; 2 )G(2 ; 1 )] as

G = g 0 + g 0 Σ[G]G, (1)
where g 0 is the non-interacting (or ballistic) Green's function, Σ is the self-energy and G is the fully interacting Green's function 25 . The resolution of this equation usually requires an iterative scheme which can be written as

G N = [g -1 0 -Σ[G N -1 ]] -1 , (2)
where G N is the Green's function at the N th iteration. It depends on the self-energy calculated from the Green's function at the previous iteration with G 0 = g 0 . When the iterative process converges, i.e. G N = G N -1 , it leads to the SCBA, which has been shown to be current conserving along the device 31 . However, truncated series expansions of Eqs. (1 -2) do not necessarily preserve these conservation laws. Therefore, the SCBA should be treated with specified convergence criteria to obtain the conserving results. This iterative character is definitively the bottleneck in real applications since it may become computationally very expensive. For example, the atomistic full-band simulator described in Ref. 15 requires heavy computational resources (> 3000 CPUs) with up to 100 SCBA iterations to reach good convergences.

B. Lowest Order Approximation (LOA)

Choosing a proper self-energy is the most important task in the NEGF framework since this quantity should accurately describe the physical properties of the system and fulfill the conservation laws. It has been shown that a Φ-derivable self-energy in the Luttinger-Ward picture 23,24 guarantees the conservation laws. The SCBA self-energy is Φ-derivable, but it requires a lot of iterations to converge, as mentioned previously. Therefore, it is of interest to devise an alternative approach satisfying the conservation laws, while avoiding the iterative scheme. The simplest LOA is a one-shot approach that meets these requirements. To build up the LOA one can also start from the Dyson equation and write

G = g 0 + g 0 Σ[G]G = g 0 + δG. (3)
We can consider the non-interacting (or ballistic) Green's function g 0 as an unperturbed term and the other one, g 0 Σ[G]G, as a perturbation δG. By injecting these terms g 0 + δG into the fully interacting Green's function G of Dyson equation and applying a first-order Taylor series expansion to the interacting self energy Σ[g 0 + δG], we get 25

G = g 0 + g 0 Σ[G]G = g 0 + g 0 Σ[g 0 + δG] (g 0 + δG) ≈ g 0 + g 0 Σ[g 0 ]g 0 + g 0 Σ[g 0 ]δG + g 0 ∂Σ[G] ∂G δG g 0 + g 0 ∂Σ[G] ∂G δG 2 , (4)
where the interacting self energy Σ[g

0 + δG] = Σ[g 0 ] + ∂Σ[G]
∂G δG. By restricting Eq. ( 4) to the first-order in interaction, we obtain the LOA Green's function g LOA that only depends on the non-interacting Green's function g 0

g LOA = g 0 + g 0 Σ[g 0 ]g 0 . (5)
This Green's function does not violate the conservation laws since the self-energy Σ[g 0 ] is found to be Φ-derivable. Bescond et al. in Ref. 30 have shown that g LOA is equivalent to the G 1 of the SCBA scheme in Eq. (2) when the perturbation term is very small and that it only includes one interaction process. Finally, this scheme can be the basic building block to generalize LOA for calculating the interacting Green's functions g N th LOA (N > 1), N defining the order in the interaction, which is given in the next section.

C. Generalization of the Lowest Order Approximation

Mera et al. in Ref. 26 have shown that one can construct the Born series of the Green's function starting with the basic building block ∆g 1 = g 0 Σ[g 0 ]g 0 of the LOA. The Green's function for the Born series up to order N in the interaction (g N ≡ g N th LOA , N > 1) can be generalized as

g N = g 0 + N n=1 ∆g n , (6)
where ∆g n is the term of order n in the strength of the interaction. In this paper we focus on the case of the electron-phonon interaction where the self-energy of the interaction is linear in the Green's function and also in the strength of the interaction. This means that Σ [G] can also be expanded in a Born series for which the term of order n is given by

Σ n = Σ [∆g n-1 ] . (7)
Since only the processes of N interactions should be included for the Green's function g N to satisfy the conservation laws, we can then discard higher order perturbation terms. We can now generalize the Born series starting from the Dyson equation ( 1) and expressing the fact that the term of order N , i.e. ∆g N , results from all products of that order contained in Σ [G] G. We thus get, for N 1

∆g N = g 0 N -1 n=0 Σ N -n ∆g n , (8)
with

∆g 1 = g 0 Σ[g 0 ]g 0 , (9)
and

g N = g N -1 + ∆g N . (10)
By using these Eqs. (8 -10), we obtain the carrier density series ρ (j) N = ρ (j) (g N ) and current series I N = I (g N ) to any order in the interaction ρ (j)

N = ρ (j) 0 + N n=1 ∆ρ (j) n , I N = I 0 + N n=1 ∆I n , (11)
where ∆ρ

(j) n = ρ (j) n -ρ (j)
n-1 and ∆I n = I n -I n-1 are defined as the difference between the expectation values of the n th and (n -1)

th order LOA, and ρ (j) 0 and I 0 are the ballistic expectation values. It has to be noted that ρ (j) N represents a carrier density of a local site (j) in the real space. It is defined on each point of the system and then computationally detailed by a vector whose dimension depends on the mesh discretization and the size of the studied system. This series of current and charge density guarantees the conservation laws by construction. However, it may converge or diverge according to the strength of the considered interaction. Therefore, in the next section we also present Padé approximants to analytically continue this series in the case of divergence.

D. Padé Approximants

Some of us has demonstrated 26 that any current expectation values can be calculated by using the Born series, while preserving the conservation laws. However, since this series has a radius of convergence depending on the intensity of the interaction, the energy of the particle, etc., Padé approximants have been introduced as an efficient remedy to treat divergent behavior [START_REF] Hardy | Divergent Series[END_REF][START_REF] Baker | Padé Approximants[END_REF]32 . Padé approximants represent a relevant method in the way they can provide several sets of rational approximations.

Considering a function F (z), the Taylor series is represented as

F N (z) = N k=0 F (k)
k! z k where F (k) is the k th derivative of F . The Padé approximant to this function F (z) is defined as a rational function F l/m (z) in a fractional form composed of two polynomials L l (z), M m (z) such as 33

F l/m (z) = L l (z) M m (z) = l 0 + l 1 z + l 2 z 2 + • • • + l l z l 1 + m 1 z + m 2 z 2 + • • • + m m z m . ( 12
)
By using the relation F N (z)M m (z) -L l (z) = 0, we can determine the unknown coefficients l 0 , . . . , l l and m 1 , . . . , m m up to l + m order. Here, the Padé table of the arbitrary function F (z) constructed up to the second order is given as

F l/m =    f 0 f 0 + f 1 z f 0 + f 1 z + f 2 z 2 f0 1-f1z/f0 f0+(f1-f0f2/f1)z 1-f2z/f1 f0 1-f1z/f0+((f1/f0) 2 -f2/f0)z 2    =   f 0/0 f 1/0 f 2/0 f 0/1 f 1/1 f 0/2   , ( 13
)
where it is noticeable that the first row of the table, i.e. F l/0 (l 2), is exactly equivalent to the Taylor series of the function F (z). Therefore, when the function can be represented by a Taylor series, we can use several rational forms of the Padé approximant. For example, once we apply Eq. ( 11) for the charge density and current series to the first row of the Padé table (13) for the arbitrary function such as

f 0 = ρ (j) 0 , f 1 z = ∆ρ (j) 1
and f 2 z 2 = ∆ρ (j) 2 , we can reconstruct the Padé table for the charge and current as

ρ (j) l/m =      ρ (j) 0 ρ (j) 0 + ∆ρ (j) 1 ρ (j) 0 + ∆ρ (j) 1 + ∆ρ (j) 2 ρ (j) 0 1-∆ρ (j) 1 /ρ (j) 0 ρ (j) 0 +∆ρ (j) 1 -ρ (j) 0 ∆ρ (j) 2 /∆ρ (j) 1 1-∆ρ (j) 2 /∆ρ (j) 1 ρ (j) 0 1-∆ρ (j) 1 /ρ (j) 0 + ∆ρ (j) 1 /ρ (j) 0 2 -ρ (j) 2 /ρ (j) 0      , (14)
I l/m =    I 0 I 0 + ∆I 1 I 0 + ∆I 1 + ∆I 2 I0 1-∆I1/I0 I0+∆I1-I0∆I2/∆I1 1-∆I2/∆I1 I0 1-∆I1/I0+(∆I1/I0) 2 -I2/I0    . ( 15
)
III. COMPUTATIONAL IMPLEMENTATION This section describes in detail the calculation of carrier densities and currents by using the first and higher order LOA. In particular, we apply the formalism in the case of electron-phonon coupling.

A. Calculation of Green's Functions by

Implementing the LOA Algorithm

Let us begin with the definition of the retarded noninteracting/ballistic Green's function g r 0 connected to two contact reservoirs (labelled L for the left contact and R for the right contact) such as

g r 0 = [EI -H -Σ r L -Σ r R ] -1 , ( 16
)
where I is the identity matrix, Σ r L/R are the retarded self-energy for the left/right device contacts [START_REF] Ferry | Transport in Nanostructures[END_REF] and H represents the Hamiltonian of the active region. The corresponding lesser/greater Green's functions are defined as

g ≶ 0 = g r 0 Σ ≶ L + Σ ≶ R g a 0 , (17)
where g a 0 is the advanced Green's function obtained from the Hermitian conjugate of its retarded counterpart g a = g r † and each Σ ≶ function is the lesser/greater selfenergy

Σ < L/R = +iΓ L/R f L/R , (18)
Σ > L/R = -iΓ L/R 1 -f L/R , (19)
f L/R being the Fermi-Dirac distribution function of the left and right reservoirs, respectively and Γ L/R the broadening function calculated according to the following

relation 1 Γ L/R = i Σ > L/R -Σ < L/R = i Σ r L/R -Σ a L/R . ( 20
)
From the lesser/greater ballistic Green's function g ≶ 0 one can calculate the ballistic charge densities such as

ρ (j) 0, e/(h) = - i V dE 2π T r (j) g </(>) 0 (E) , (21)
where V is the volume of the discretized cell and the trace operator T r (j) acts on the terms of the chosen local basis (j). The ballistic current is given by

I 0, (j) = e dE 2π T r Σ ≷ (j) g ≶ 0 (E) -Σ ≶ (j) g ≷ 0 (E) , (22)
with Σ ≶ (j) as the self-energy between slices (j) and (j + 1) along the transport direction, as the reduced Planck constant, e as the electronic charge, and T r as the trace operator on the whole matrix.

We now introduce the electron-phonon scattering by calculating the higher order Born series. The electron mass being much smaller compared to the ion mass [START_REF] Fetter | Quantum Theory of Many-Particle Systems[END_REF] , the phonon bath is assumed to be in thermal equilibrium. Therefore, the self-energy of interaction with optical phonons can be expressed as

Σ ≶ op (E) = M op [N op g ≶ 0 (E ∓ ω) + (N op + 1)g ≶ 0 (E ± ω)], (23)
where M op unit in eV 2 describes the strength of the optical-phonon interaction and N op = 1/(e ω/kBT -1) is the phonon occupation number at the optical phonon energy ω. Interactions with acoustic phonons are assumed to generate elastic processes (E ± w ≈ E) and can be reduced to

Σ ≶ ac (E) = M ac g ≶ 0 (E), (24)
where M ac unit in eV 2 is the intensity of the acousticphonon interaction. We then define an interaction selfenergy

Σ ≶ int = i Σ ≶ op + Σ ≶ ac . (25)
Here, the self-energies for the electron-phonon scattering are treated differently in the SCBA and LOA cases.

For the SCBA, we inject these self-energies Eqs. (23 -25) into Eqs. (16 -17) and iteratively solve the following system

G ≶ N = G r N Σ ≶ L/R + Σ ≶ int [G N -1 ] G a N , (26)
G r N = EI -H -Σ r L/R -Σ r int [G N -1 ] -1 , (27)
Σ r int = [Σ > int -Σ < int ] 2 , ( 28
)
where

G r/ a 0 = g r/ a 0 and G ≶ 0 = g ≶ 0 .
In the case of LOA, the non-interacting Green's function g ≶ 0 can be used for the calculation of the self-energy in Eq. ( 23) as the first order LOA. However, it should be replaced by the perturbation terms ∆g N -1 for the higher order LOA (N 1) such that

Σ ≶ op, N (E) = M op [N op ∆g ≶ N -1 (E ∓ ω) + (N op + 1)∆g ≶ N -1 (E ± ω)], (29)
Σ ≶ ac, N (E) = M ac ∆g ≶ N -1 (E), (30)
where ∆g ≶ 0 = g ≶ 0 is the perturbation term of the firstorder LOA. This process imparts physical interpretations on the LOA meaning that first-order LOA exactly includes one electron-phonon interaction processes, secondorder LOA exactly two electron-phonon interactions, and so on. Now that the non-interacting Green's function and the electron-phonon self-energies are defined, we return to the generalized LOA Eq. ( 8). This equation shows how to compute the retarded Green's function of the LOA to any order. However, to obtain the general formulation of the lesser/greater Green's function of the LOA at any order we need to apply the Langreth theorem 7 to this equation. This theorem states that

G ≶ = g r 0 Σ r int G ≶ + Σ ≶ int G a + Σ ≶ L/R G a . ( 31
)
Noticing that g r 0 Σ

≶ L/R = g ≶ 0 (g a 0 ) -1 and that (g a 0 ) -1 G a = I + Σ a int G a , we can rewrite G ≶ = g ≶ 0 + g r 0 Σ r int G ≶ + Σ ≶ int G a + g ≶ 0 Σ a int G a , (32
) so that the Born series is given by ∆g

≶ N = N -1 n=0 [g r 0 Σ r int, N -n ∆g ≶ n + g r 0 Σ ≶ int, N -n ∆g a n + g ≶ 0 Σ a int, N -n ∆g a n ], (33) with ∆g
≶ N = g ≶ N -g ≶ N -1 . (34)
Finally, once we get the lesser/greater LOA Green's function, calculation of the current and the charge density can be performed by using Eqs. (21 -22)

ρ (j) N, e/ (h) = - i V dE 2π T r (j) g </(>) N (E) , (35)
I N, (j) = e dE 2π T r Σ ≷ (j) g ≶ N (E) -Σ ≶ (j) g ≷ N (E) . ( 36
)
B. Matrix Form of the Padé Approximants

When considering higher order of currents and carrier densities, the Padé approximants in Eq. ( 13) can be rapidly difficult to evaluate. We therefore refer in the following to the matrix form of the Padé approximants. [START_REF] Vajta | Proceedings of the 3rd TEMPUS INTCOM Symposium on Intelligent Systems in Control and Measurement[END_REF] We first turn our attention to Eq. (12). By substituting the Taylor series into this expression, we obtain the following equation

l+m k=0 f k z k = f 0 + f 1 z + f 2 z 2 + • • • + f l+m z l+m = l 0 + l 1 z + l 2 z 2 + • • • + l l z l 1 + m 1 z + m 2 z 2 + • • • + m m z m . (37)
We can then rearrange it and match the same order coefficients

f 0 + (f 0 m 1 + f 1 ) z + (f 0 m 2 + f 1 m 1 + f 2 ) z 2 + • • • = l 0 + l 1 z + l 2 z 2 + • • • , (38)
so that a system of l+m+1 linear homogeneous equations is obtained

l 0 = f 0 , l 1 -f 0 m 1 = f 1 , l 2 -f 0 m 2 -f 1 m 1 = f 2 , . . . l l -f 0 m l -f 1 m l-1 -• • • -f l-1 m 1 = f l , (39) -f l-m+1 m m -f l-m+2 m m-1 -• • • -f l m 1 = f l+1 , -f l-m+2 m m -f l-m+3 m m-1 -• • • -f l+1 m 1 = f l+2 , . . . -f l m m -f l+1 m m-1 -• • • -f l+m-1 m 1 = f l+m .
In terms of the unknown coefficients l 0 , . . . , l l and m 1 , . . . , m m , these linear equations can be transformed in a matrix form as

             1 0 • • • 0 0 0 • • • 0 0 1 • • • 0 -f 0 0 • • • 0 . . . . . . . . . . . . . . . . . . 0 0 • • • 1 -f l-1 -f l-2 • • • 0 0 0 • • • 0 -f l -f l-1 • • • -f l-m+1 0 0 • • • 0 -f l+1 -f l • • • -f l-m+2 . . . . . . . . . . . . . . . . . . 0 0 • • • 0 -f l+m-1 -f l+m-2 • • • -f l                           l 0 l 1 . . . l l m 1 m 2 . . . m m              =              f 0 f 1 . . . f l f l+1 f l+2 . . . f l+m              , (40)
where we can see that it represents a Ax = b linear matrix equation. By solving this equation we can easily determine the coefficients l 0 , . . . , l l and m 1 , . . . , m m for the function

F l/m (z) = l0+l1z+l2z 2 +•••+l l z l 1+m1z+m2z 2 +•••+mmz m .
This approach will be used in the rest of the paper.

IV. RESULTS AND DISCUSSIONS

To validate our approach, we consider a onedimensional (1D) linear atomic chain connected to left and right reservoirs at room temperature, as widely encountered in nanoelectronic devices. We consider a twoband k • p Hamiltonian to assess the capability of such method to treat the impact of multiband transport properties, and we compute both the current and the electron density in presence of electron-phonon scattering.

The active region of the linear chain is 16 nm long and consists of 32 atomic sites along the x-direction (i.e. the lattice constant between atoms is 5 Å). In order to evaluate the impact of multi-band transport, we consider the following two-band k • p Hamiltonian H k•p defined as effective masses and E p is the strength of the coupling between the two bands. By transforming k x → -i∂/∂x, we can discretize this Hamiltonian in the real space. Two different parameter sets are defined in Table I. In the first parameterization the coupling term E p is equal to zero and E 1 = E 2 . The two bands exactly overlap each other, and can be treated independently. Figure 1 (a) shows the resulting band structure defined as TBWOC for "Two-Band WithOut Coupling". The second parametrization, labelled TBWC for "Two-Band With Coupling", assumes a strong coupling E p . The two bands repel each other and generate a negative curvature in the lowest one (Fig. 1 (b)). It gives rise to a camel-like shape which is the characteristics, for instance, of valence band structures in Si nanowires [START_REF] Niquet | [END_REF] .

H k•p =   E 2 + 2 k 2 x 2me2 Ep 2 2m0 k x Ep 2 2m0 k x E 1 + 2 k 2 x 2me1   , (41)
Without any applied voltages on the reservoirs (V RL ), the linear chain is in equilibrium with the left and right chemical potentials µ L/R set 0.1 eV above the bottom of the lowest sub-band (Fig. 1). A non-zero applied bias shifts the two chemical potentials by ∆µ = µ L -µ R = eV RL making the active region out-of-equilibrium. For the sake of simplicity a linear potential profile between the left and right reservoirs is assumed.

We first study the influence of optical phonon interactions with a frequency w = 60 meV and consider a weak electron-phonon coupling (M op = 5 × 10 -4 eV 2 ). Figures 2 (a) and 2 (b) compare the currents obtained in the ballistic regime, the SCBA and for the 5 first LOA for the two TBWOC and TBWC Hamiltonians, respectively. At V RL = 0.2 V, the SCBA results show 11 % current degradation for TBWOC and 4 % current degradation for TBWC with respect to the ballistic currents. We clearly see that the Born series of the current rapidly converges to usual SCBA for both TBWOC and TBWC Hamiltonians. In addition, Figure 2 (b) shows that the LOA well captures the multi-band properties since the current inflection near V RL = 0.07 V induced by the negative curvature of the band structure (Fig. 1 correctly described as in the SCBA.

The corresponding spectral currents at the right-edge of the active region are plotted at V RL = 0.2 V in Figs. 2 (c) -(d) for TBWOC and TBWC, respectively. The ballistic spectrum depicts one peak for TBWOC (Fig. 2 (c)) and two peaks for TBWC (Fig. 2 (d)) corresponding to the band structure features shown in Fig. 1. The SCBA spectrum shows the impact of optical emission and absorption of phonons on the energy distribution of electrons for the two types of Hamiltonians. We also note that at a given order N the LOA includes all the N phonon scattering processes. For such a weak electronphonon coupling, the converging behavior of the series for both types of band structure confirms the relevancy of the LOA, which avoids multiple SCBA iterations without losing accuracy in the current calculation.

The situation differs when considering a larger electron-phonon coupling (M op = 1 × 10 -3 eV 2 ). As shown by the current characteristics in Fig. 3, the reductions between the ballistic and the SCBA are 18 % for TBWOC and 7 % for TBWC at V RL = 0.2 V. Figures 3 (a) and 3 (b) also show that the series of the LOA currents are now diverging with respect to SCBA values. We then apply the Padé approximants to analytically continue these divergent series. Figures 3 (c) and 3 (d) show that most of the Padé approximants successfully reproduce the results of SCBA. We also note that the Padé 1/1 in Fig. 3 (d) exhibits strong singularities. These singularities result from the cancellation of the denominator of the rational expression of the Padé approximants. This behavior occurs when ∆I 1 of Eq. ( 15) is equal to ∆I 2 . To better understand the divergence of the LOA series, Figure 4 shows the current spectra at the edge of the right active region at V RL = 0.2 V. We clearly see that the odd and even orders of the LOA oscillate alternatively in opposite directions, while going away from the SCBA.

We put forward our approach to check the validity for strong optical phonon scattering and then consider an electron-phonon coupling (M op = 3 × 10 -3 eV 2 ). show that current degradations induced by the strong scattering are 34 % for TBWOC and 18 % for TBWC at V RL = 0.2 V. All LOA series strongly diverge (not shown), and we should refer to higher order Padé approximants to successfully reconstruct these divergent series, e.g. Padé 2/3 (requiring the fifth current order) for TBWOC and Padé 1/2 (requiring the third current order) for TBWC.

We now focus on the impact of interactions with acoustic phonons which are defined in the present work within the elastic limit. Study of acoustic phonon scattering is an important aspect since it has been found to accelerate the divergence of the Born series in nanowire transistor 30 . We first consider an electron-phonon coupling M ac = 1 × 10 -3 eV 2 . It corresponds to a scattering which generates a 24 % current degradation for TBWC at V RL = 0.2 V as shown in Fig. 6 (a). Figure 6 (a) also shows that the LOA series diverges. The higher LOA currents (> 3 rd LOA) are even outside the visible range in the scale. However, application of the Padé approximants on the LOA currents is once again capable of providing SCBA results (Fig. 6 (b)). Padé approximants are then a relevant root to reconstruct the divergent series even for a high rate of divergence. Current spectra in Fig. 7 clearly illustrate the reason of the faster divergence of the LOA with the acoustic phonon scattering than with their optical counterparts. Indeed, optical phonon scattering relaxes or excites electrons to lower or higher energies according to the number of scattering processes involved. On the other hand, acoustic phonon scattering, assumed to be elastic, does not modify the electron energy. Therefore, electrons whose energy is near the edge of the subband have a divergent density-of-states due to the Van Hove singularities (compare Fig. 4 with Fig. 7) that drastically increase the electron-phonon interactions.

When increasing the electron-phonon coupling to M ac = 5 × 10 -3 eV 2 , we see that acoustic phonon scattering induces a strong current degradation which is almost 60 % for TBWC at V RL = 0.2 V as shown in Fig. 8 (a). For such a strong scattering, all the LOA currents are highly diverging from the SCBA values. However, applying the Padé approximants to these diverging currents can still successfully reproduce SCBA results (Fig. 8 (b)). In particular, we see that Padé 0/1, 1/2 and 2/3 (i.e. N/N +1) which requires current calculations at the first, third, and fifth orders are more efficient to rapidly converge toward the SCBA results.

Calculation of charge density is another important aspect since it directly determines the potential profile inside the device within the Hartree approximation. We then examine the validity of the analytically continuated LOA method to calculate the charge densities in the linear chain. For the sake of clarity we do not solve the Poisson equation to get a self-consistent potential. The system is first considered at equilibrium, i.e. V RL = 0 V. Figure 9 shows the charge density profiles along the linear chain with the TBWOC and TBWC Hamiltonians for a weak optical phonon coupling (M op = 1 × 10 -4 eV 2 ). As expected for such a bias, the charge density profile in the ballistic regime is constant. On the other hand, those obtained by the SCBA and the LOA show symmetric variations with respect to the middle of the chain. This is due to the fact that the self-energy of the electron-phonon interactions is not included in the Green's functions of the reservoirs, thus generating small reflections at the contacts that are only visible at V RL = 0 V. We also observe that the charge density of the LOA series obtained with TBWC converges more slowly than the one for TBWOC due to the impact of band coupling. These results confirm that the multi-band character of the transport has an important effect on the convergence behavior of the series by increasing the number and the intensity of the scattering processes. However, both LOA series converge towards the SCBA result for this weak optical phonon scattering.

When considering a five times larger coupling, i.e. M op = 5×10 -4 eV 2 , the charge density series for TBWC clearly diverges as it goes to higher LOA (Fig. 10 (a)). Here, we note that Padé approximant is again capable to analytically continue the diverging series. In that case, we need to use higher order Padé approximants ( Padé 3/4) to obtain the same results as with SCBA. However, except for the Padé 0/2 and 2/3, the other order Padé approximants also show good results within ∼ 1 % error to the SCBA result (Fig. 10 (b)).

We now consider a non-zero applied bias V RL = 0.2 V. Figure 11 shows the charge densities obtained for both optical and acoustic scattering with a strong cou-pling M op = M ac = 1 × 10 -3 eV 2 . The applied bias now generates asymmetric charge density profiles since the Poisson equation is not solved to ensure the electroneutrality of the access regions. We first note that the LOA series diverge for both types of phonons (Figs. 11 (a) and 11 (c)). However, like for the current calculations the divergence is faster for acoustic phonons than for optical ones. For the optical phonon scattering (Fig. 11 (a)), active regions near the right reservoir where the forward scattering dominates are diverging slower than the other regions. On the other hand, in the case of the acoustic phonon scattering (Fig. 11 (c)), forward scattering regions (right part) diverge much faster than back scattering regions (left part). These different behaviors of divergence are due to the different elastic and inelastic nature of the acoustic and optical phonon scattering respectively. Padé approximants ( Padé 2/3) applied to the divergent series of the charge density still reproduce the SCBA results for both types of scattering (Figs. 11 (b) and 11 (d)). Although the calculation of the charge density requires higher order Padé approximants, the analytically continuated LOA approach still shows a clear advantage to describe transport properties in quantum simulations without iterative scheme.

Finally, in order to illustrate the numerical efficiency of our approach we compare computational times at V RL = 0.07 V and 0.2 V for the SCBA with those for the 5 first LOA (see Table II) in the case of strong optical and acoustic phonon scattering for TBWC. At V RL = 0.07 V where the SCBA requires more than 40 iterations due to complex band structure features, all LOA calculations outperform the SCBA ones. We also see that computational times of the LOA do not significantly vary with bias, while the number of SCBA iterations depends on it. In total runtime to SCBA, it is shown that the simulation for the optical phonon scattering takes more computational times than that for the acoustic phonon one. This is due to the fact that even though the SCBA for the optical phonon scattering at high bias shows a faster convergence than for the acoustic one, at low bias (V RL 0.05 V) it does not converge after 200 iterations with < 2 % error. In addition, we see that calculating 1 st LOA coupled with the Padé 0/1 is 40 times faster than the SCBA. Calculation of 5 th LOA with Padé 2/3 is still efficient with respect to the SCBA. We then expect that for devices where strong scattering processes are dominant, our approach is much more powerful. Very recently Mera et al. have reported in Ref. 38 an approach based on Gauss hypergeometric resummation to model the Stark effect in the hydrogen atom. Even more recently 39 , they applied this technique to the current calculation in presence of optical phonon scattering in nanostructures. They consider an opposite transport regime as in the present work, i.e. quasi-localized states weakly coupled to electron reservoirs. In this particular regime, the authors have shown that a fourth order hypergeometric approach was able to outperform Padé approximants 1/1 and 2/2. It then represents a relevant technique to treat inelastic scattering in nanodevice quantum simulations. Further investigations should be done along that direction for other general transport regimes.

V. CONCLUSIONS

In this work we presented a numerically efficient approach, coupling the Born series of a given observable with the Padé approximant technique, to treat inelastic scattering in nanostructures. The approach has been applied to the treatment of electron-phonon interactions and has been systematically compared to the usual, time consuming, self-consistent Born approximation. As a fair illustration of systems encountered in nanoelectronics, we considered a linear chain in presence of both acoustic and optical phonons. Two-band k.p Hamiltonians have been implemented to analyze the capability of the approach to capture multi-band scattering. Our results first show that the method is efficient to calculate the current values. Indeed, the Padé approximants constructed with the fifth order terms in interactions are capable to provide the SCBA values for a large variety of coupling intensities and biases. The approach has been also successfully applied to the calculation of the charge density, for which it has been found that the convergence is more difficult to achieve, requiring calculations of higher orders of the electron density Born series. The presented technique remains a relevant alternative to avoid the iterative scheme of the SCBA. Its main advantage is that it can be easily generalized systematically to any order of the interaction and any Padé table. It can then be rapidly implemented in any SCBA quantum transport code. Its main drawback is that second, third, and higher order perturbation terms require important number of matrix operations and a large amount of memory storage. Their implementation in three-dimensional atomistic quantum transport codes to model realistic nano-structures is still numerically challenging and difficult to overcome. However, our results showed that N/N +1 Padé approximants are more reliable in reconstructing the divergent current and density series in the considered system and the simple 0/1 Padé approximant should still be of relative interest in this context.

VI. ACKNOWLEDGEMENT

FIG. 1 .

 1 FIG. 1. (Color online) Band structures in E -kx diagram described by a two-band k • p Hamiltonian: (a) two-bands without coupling (TBWOC) and (b) two-bands with coupling (TBWC). The chemical potentials µ L/R in the left and right reservoirs are set 0.1 eV above the bottom of the lowest sub-band.

FIG. 2 .

 2 FIG. 2. (Color online) Top panel: the current-voltage characteristics in presence of optical phonon scattering (Mop = 5 × 10 -4 eV 2 ): (a) for TBWOC, and (b) for TBWC in the ballistic regime, the SCBA and the 5 first Lowest Order Approximations. Bottom panel: the corresponding currents spectra Jw at the edge of the right active region with VRL = 0.2 V: (c) for TBWOC, and (d) for TBWC in the ballistic regime, the SCBA and the 5 first Lowest Order Approximations. The dotted circle in Fig. 2 (b) refers to the current inflection induced by the negative curvature of the considered band structure.

20 FIG. 3 .

 203 FIG. 3. (Color online) Top panel: the current-voltage characteristics in presence of optical phonon scattering (Mop = 1 × 10 -3 eV 2 ): (a) for TBWOC, and (b) for TBWC in the ballistic regime, the SCBA and the 5 first Lowest Order Approximations. Bottom panel: the current-voltage characteristics reconstructed based on Padé approximants: (c) for TBWOC, and (d) for TBWC. The ballistic and SCBA results are also reported.

FIG. 4 . 20 FIG. 5 FIG. 6 .

 42056 FIG. 4. (Color online) Current spectra Jw of TBWC at the edge of the right active region with VRL = 0.2 V for Mop = 1 × 10 -3 eV 2 in the ballistic regime, the SCBA and the 5 first Lowest Order Approximations. Insets: the currents spectra for the ballistic regime and SCBA.

Figures 5 (

 5 Figures 5 (a) and 5 (b) show that current degradations induced by the strong scattering are 34 % for TBWOC and 18 % for TBWC at V RL = 0.2 V. All LOA series strongly diverge (not shown), and we should refer to higher order Padé approximants to successfully reconstruct these divergent series, e.g. Padé 2/3 (requiring the fifth current order) for TBWOC and Padé 1/2 (requiring the third current order) for TBWC.We now focus on the impact of interactions with acoustic phonons which are defined in the present work within the elastic limit. Study of acoustic phonon scattering is an important aspect since it has been found to accelerate the divergence of the Born series in nanowire transistor30 . We first consider an electron-phonon coupling M ac = 1 × 10 -3 eV 2 . It corresponds to a scattering which generates a 24 % current degradation for TBWC at V RL = 0.2 V as shown in Fig.6 (a). Figure6(a) also shows that the LOA series diverges. The higher LOA currents (> 3 rd LOA) are even outside the visible range in

4 FIG. 7 .

 47 FIG. 7. (Color online) Current spectra Jw of TBWC at the edge of right active region in presence of acoustic phonon scattering with VRL = 0.2 V for Mac = 1 × 10 -3 eV 2 in the ballistic regime, the SCBA, and the 5 first Lowest Order Approximations.

20 FIG. 8 .

 208 FIG. 8. (Color online) Current-voltage characteristics of TBWC in presence of acoustic phonon scattering with Mac = 5 × 10 -3 eV 2 : (a) in the ballistic regime, the SCBA, and the 5 first Lowest Order Approximations, and (b) in the ballistic regime, the SCBA, and the Padé approximants. For clear view of curves, 2 nd ∼ 5 th LOA are not reported in Figure 8 (a).

FIG. 9 .

 9 FIG. 9. (Color online) 1D carrier density profiles along the active region at equilibrium (VRL = 0 V) with Mop = 1 × 10 -4 eV 2 : (a) for TBWOC, and (b) for TBWC in the ballistic regime, the SCBA, and the 5 first Lowest Order Approximations.

16 FIG. 10

 1610 FIG. 10. (Color online) 1D carrier density profiles of TBWC along the active region at equilibrium (VRL = 0 V) with Mop = 5 × 10 -4 eV 2 : (a) in the ballistic regime, the SCBA, and the 7 first Lowest Order Approximations, and (b) where LOA series are replaced by the Padé approximants. 6 th and 7 th LOA are not visible on Figure 10 (a).

16 FIG. 11

 1611 FIG. 11. (Color online) Top panel: 1D carrier density profiles of TBWC with VRL = 0.2 V considering optical phonon scattering with Mop = 1 × 10 -3 eV 2 : (a) in the ballistic regime, the SCBA, and the 5 first Lowest Order approximations, and (b) in the ballistic regime, the SCBA, and the corresponding Padé approximants. Bottom panel: 1D carrier density profile of TBWC with VRL = 0.2 V considering acoustic phonon scattering with Mac = 1 × 10 -3 eV 2 : (c) in the ballistic regime, the SCBA, and the 7 first Lowest Order approximations, and (d) in the ballistic regime, the SCBA, and the corresponding Padé approximants.

TABLE I

 I

	. Parameters for two-band k • p Hamiltonian used in
	this work.
		TBWOC	TBWC
	me1/m0, me2/m0	0.9, 0.9	0.5, 1.0
	E1, E2 (eV)	0, 0	0.0691, 0.1
	Ep	0	0.5

where E 1 and E 2 are the energies of the two bands at k x = 0, m e1 and m e2 represent the corresponding

TABLE II .

 II Comparison of computational times for TBWC at VRL = 0.07 V and 0.2 V for the SCBA with those for 5 first Lowest Order Approximations in presence of strong optical and acoustic phonon scattering with Mop = Mac = 5 × 10 -3 eV 2 . Total computational times for all bias points are also compared.

			SCBA	1 st LOA	2 nd LOA	3 rd LOA	4 th LOA	5 th LOA
	Runtime (s)	Optical	744	17	39	70	111	162
	at VRL = 0.07 V	Acoustic	738	16	33	59	91	132
	Runtime (s)	Optical	191	17	39	69	109	158
	at VRL = 0.2 V	Acoustic	535	15	33	59	91	132
	Total runtime (s)	Optical	13934	357	819	1470	2331	3402
		Acoustic	12166	336	693	1240	1911	2774

This work was supported by the NOODLES contract (ANR-13-NANO-0009) funded by the ANR-French National Research Agency. The author Y. Lee would like to thank the A*MIDEX for financial support. The authors would like to thank Lidiya Yushchenko from ISEN-Toulon for fruitful discussion on Padé approximants.


Théo Voillemin
email: theo.voillemin@univ-lille.fr

Hazem Wannous
email: hazem.wannous@univ-lille.fr

Jean-Philippe Vandeborre
email: jean-philippe.vandeborre@imt-lille-douai.fr

2D Deep Video Capsule Network with Temporal Shift for Action Recognition

Action recognition in continuous video streams is a growing field since the past few years. Deep learning techniques and in particular Convolutional Neural Networks (CNNs) achieved good results in this topic. However, intrinsic CNNs limitations begin to cap the results since 2D CNN cannot capture temporal information and 3D CNN are to much resource demanding for real-time applications. Capsule Network, evolution of CNN, already proves its interesting benefits on small and low informational datasets like MNIST but yet its true potential has not emerged. In this paper we tackle the action recognition problem by proposing a new architecture combining Temporal Shift module over deep Capsule Network. Temporal Shift module permits us to insert temporal information over 2D Capsule Network with a zero computational cost to conserve the lightness of 2D capsules and their ability to connect spatial features. Our proposed approach outperforms or brings near state-of-the-art results on color and depth information on public datasets like First Person Hand Action and DHG 14/28 with a number of parameters 10 to 40 times less than existing approaches.

I. INTRODUCTION

Action recognition and video understanding are hard problems that generally requires a lot of computational cost that make them difficult to process continuous videos. Though, a lot of applications require for efficient and accurate solution of those problems like hand gestures recognition for new Augmented/Virtual Reality interactions.

Deep learning has become one of the most efficient techniques for image and video understanding especially since the appearance of Convolutional Neural Networks (CNNs) [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. They also proved their benefits for real-time applications, at least for relatively small networks, since the learning phase is particularly time consuming, but the inference consists only of a bunch of calculus. However, 3D CNNs, even if they present the benefit to capture both spatial and temporal information, are generally way to large to be applied in real-time with reasonable computing power [START_REF] Ji | 3d convolutional neural networks for human action recognition[END_REF], [START_REF] Molchanov | Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network[END_REF], [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]. Conversely, 2D CNNs, thanks to their weights sharing, bring small computational cost, perfect for fast training and real-time inference, but in their basic implementation, they cannot extract temporal information since they work with only one frame at a time [START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF], [START_REF] Zha | Exploiting image-trained cnn architectures for unconstrained video classification[END_REF], [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF], [START_REF] Zhou | Temporal relational reasoning in videos[END_REF].

To counterbalance this lack of temporal information extraction, previous works presented several approaches taking into account temporal modeling into 2D CNNs [START_REF] Zhou | Temporal relational reasoning in videos[END_REF], [START_REF] Girdhar | Actionvlad: Learning spatio-temporal aggregation for action classification[END_REF], [START_REF] Tran | A closer look at spatiotemporal convolutions for action recognition[END_REF]. Our attention especially focused on recent Temporal Shift Module (TSM) research by Li et al. [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF] that provides new efficient video modeling by adding temporal information into 2D CNN by shifting channels in convolutional layers without deteriorating computational costs.

Besides this lack of temporal information, CNNs seize other limitations as invariance because of pooling, or the incapability to understand spatial features relationship between convolutional layers. A solution was proposed by Sabour et al. [START_REF] Sabour | Dynamic routing between capsules[END_REF] developing a new architecture based on convolution, capsule network that have presented promising results on some datasets like MNIST [START_REF] Lecun | The mnist database of handwritten digits[END_REF] with a far less deeper network than the common CNN solutions. But still, Capsule Network struggles to provide good performance on complex dataset. Recent intuitions, similar to deep convolutional neural network, pushed some research as Rajasegaran et al. [START_REF] Rajasegaran | Deepcaps: Going deeper with capsule networks[END_REF] to experiment deep capsule network (DCN) by stacking capsule layers. They found that simply stacking those layers conduct to a degradation of performance whether in classification precision or in computational cost because of the new routing algorithm, specific to the good learning of capsule layers. In order to address this issue, they proposed to reduce the number of routing iterations on the initial capsule layers, also adding skip connections to improve learning of middle layers and use of 3D convolutions layers to reduce parameters number. It should also be noted that the usage of capsule network have not been yet explored for video understanding, only Duarte et al.'s approach [START_REF] Duarte | Videocapsulenet: A simplified network for action detection[END_REF] can be found. However, we note in this first network of 3D capsules the same problem as for 3DCNN, that is the significant cost of calculation of these solutions. In our exploration of 2D capsule network solutions for video and action understanding, we found some new complications specific to capsule architecture, the weights divergence when large scale information are passed as input of capsule layer, because frame data from video dataset as HMDB51 [START_REF] Kuehne | HMDB: a large video database for human motion recognition[END_REF] or FPAH [START_REF] Garcia-Hernando | First-person hand action benchmark with rgb-d videos and 3d hand pose annotations[END_REF] are much bigger than the 14 x 14 frame of dataset like MNIST [START_REF] Lecun | The mnist database of handwritten digits[END_REF].

To address those problems, we propose in this paper a new architecture combining deep capsule network and temporal shift module for efficient video and action understanding, called later 2D Deep Video Capsule Network (2D DVCN). We explore three different capsule shifts based on Li et al.'s approach [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF], illustrated in Fig. 1. Shift of entire first capsules from a frame to the next one (Fig. 1d), shift of first dimension features for every capsules of a layer of a frame to the capsules of the next one (Fig. 1c) and shift of first dimension features of the first capsule of a layer of a frame to the next one (Fig. 1b).

The implementation of temporal shift on capsule network is motivated by the strength of capsules to understand relationship between spatial features extracted by convolution operations. Our goal is to reinforce the temporal information shifted and shared from a frame to the next one by linking the shifted convolutions with the frame's original ones thanks to the capsules and routing algorithm. We also figured out how to avoid weights divergence when processing large image with capsule network by modifying the original DeepCaps network [START_REF] Rajasegaran | Deepcaps: Going deeper with capsule networks[END_REF] permitting to use input frame 10 times larger than actual frames of datasets usually test on by other capsule networks. The main contributions of our paper are the followings:

• Developing a first 2D capsule network for video understanding (2D DVCN) thanks to the temporal information addition of the temporal shift module. • Proposing three different implementations (Fig. 1) of temporal shift module over capsule layer and a study of their impact on classification rate of our approach. • Evaluation the performance of our approach over challenging action and gesture video datasets as HMBD51 [START_REF] Kuehne | HMDB: a large video database for human motion recognition[END_REF], FPAH [START_REF] Garcia-Hernando | First-person hand action benchmark with rgb-d videos and 3d hand pose annotations[END_REF] and DHG14/28 [START_REF] Smedt | Skeleton-based dynamic hand gesture recognition[END_REF] with outperform or near state-of-the-art results over color and/or depth information but with 10 to 40 times less parameters than other 2D/3D CNN or RNN methods.

The rest of this paper is organized as follows: Discussion of related work over other capsule or 2D/3D convolutional network for video understanding problem in Section II, description of our 2D Deep Video Capsule Network and temporal shift module implementation in Section III, Section IV shows our results and Section V concludes this paper.

II. RELATED WORK

Since the first implementation of Capsule Network by Sabour et al. [START_REF] Sabour | Dynamic routing between capsules[END_REF], some new capsule architectures have been explored. Ma et al. [START_REF] Ma | Forecasting transportation network speed using deep capsule networks with nested lstm models[END_REF] developed deep capsule network joints to LSTM for forecasting transportation. Similar idea for speech recognition but with an LSTM before capsule layer has been explored by Srivastava et al. [START_REF] Srivastava | Identifying aggression and toxicity in comments using capsule network[END_REF]. We can particularly observe that there are few works on capsule network opposite to other deep methods, especially in video understanding. We can find only one implementation of Capsule Network for video understanding in the literature, Duarte et al. [START_REF] Duarte | Videocapsulenet: A simplified network for action detection[END_REF] developed the first 3D Capsule Network especially for action detection and segmentation but we can find the same defect than 3D CNN, the important computational cost compared to 2D Capsule Network.

2D CNN is the lightest method for video recognition and understanding ( [START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF], [START_REF] Zha | Exploiting image-trained cnn architectures for unconstrained video classification[END_REF], [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF], [START_REF] Zhou | Temporal relational reasoning in videos[END_REF]), but these methods have a natural lack to capture temporal information. Karpathy et al. [START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF] designed a two-streams CNN that takes the same image at different resolutions to capture more spatial information. Temporal Segment Network [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF] combines a sparse temporal sampling strategy with video-level supervision. But if 2D CNNs can run efficiently on modest hardware, most of them cannot capture temporal information. 3D CNNs were first developed to offset this lack and most of them surpass 2D CNNs results but for an important computational loss [START_REF] Ji | 3d convolutional neural networks for human action recognition[END_REF], [START_REF] Molchanov | Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network[END_REF], [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]. Molchanov et al. [START_REF] Molchanov | Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network[END_REF] implemented a fusion of 3D CNN to capture spatio-temporal information with a Recurrent Neural Network (RNN) for online analysis of video clip per clip, but this method demanded an important computational power.

Recurrent Neural Networks have been also designed for temporal sequences analysis like speech recognition and action or gesture recognition [START_REF] Murakami | Gesture recognition using recurrent neural networks[END_REF], [START_REF] Smedt | Dynamic hand gesture recognition-from traditional handcrafted to recent deep learning approaches[END_REF], [START_REF] Chen | Motion feature augmented recurrent neural network for skeleton-based dynamic hand gesture recognition[END_REF], [START_REF] Du | Hierarchical recurrent neural network for skeleton based action recognition[END_REF]. In their works, Du et al. [START_REF] Du | Hierarchical recurrent neural network for skeleton based action recognition[END_REF] used skeleton joints of each body members on a hierarchical Bidirectional-RNN network to proceed action recognition. RNN based method have generally better classification results since the skeleton information over time is really informative. Still, we decided to develop our 2D DVCN on color, depth or optical flow images since this kind of information is really easy to obtain. It is thus present on a great majority of databases and do not require a extraction pretreatment as skeleton joints needs. Our motivation is therefore to have the more flexible solution.

Many attempts have been made to model temporal information into deep learning method. We already discussed above about 3D CNN and their natural handling of temporal information. Attention mechanism have been explored, Guo et al. [START_REF] Guo | Attentionbased lstm with semantic consistency for videos captioning[END_REF] used dynamic weighted sum of local 2D CNN and 3D CNN representations to pass extracted features into an LSTM. Zang et al. [START_REF] Zang | Attentionbased temporal weighted convolutional neural network for action recognition[END_REF] have implemented Attentionbased Temporal Weighted to implant visual attention into a multi-stream CNN. Lin et al. [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF] proposed the temporal shift module to deal with lack of temporal information in 2D CNN without computational loss, on each kernel of each convolutional layer, a small proportion of channels is shift on the next or previous frame of a same video sequence that will be treat frame per frame by the CNN. This temporal shifting permits to bring information of a part of the spatial extracted features of the previous frame on the current treated frame, or from the next to the current one in case of offline implementation. This new module, in addition to have zero extra computational cost apart of data movement, improved state-of-the-art performance over multiple action datasets like HMDB51 [START_REF] Kuehne | HMDB: a large video database for human motion recognition[END_REF] or UCF101 [START_REF] Soomro | Ucf101: A dataset of 101 human actions classes from videos in the wild[END_REF] in comparison to other temporal modeling methods. The last work particularly attracted our attention and from which our proposed approach was inspired. A white small cube is for zero-padding since the shift leave empty the first channels. In offline implementation, (a) and (b), channels are shifted in both temporal direction, past and future, since it's impossible in an online solution to get information from the future, an online shift has been implemented to progressively shift only to the future frame.

III. 2D DEEP VIDEO CAPSULE NETWORK (2D DVCN)

2D capsule network architecture for video understanding has not been yet explored. The only works explored to deal with temporal information using capsule used LSTM cells that can only perform well on low dimensional information which is not the case for videos treatment. However, capsule network already proves its potential for spatial-features extraction [START_REF] Sabour | Dynamic routing between capsules[END_REF], [START_REF] Rajasegaran | Deepcaps: Going deeper with capsule networks[END_REF], hence we decided to explore how can we incorporate temporal-features extraction into capsules. We first explain the Temporal Shift Module (TSM) [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF] implementation into capsules, then our 2D Deep Video Capsule Network for video understanding.

A. Shift Convolutional Capsule

Let's represent the shape of the input of a convolutional capsule as : R (w,h,c,n) with w and h respectively be the width and height of the current spatial features, c the number of capsule tensors and n the dimension of the capsule (number of convolutional kernels). By looking at the shape of a convolutional capsule, we can see that, structurally, capsules are just a set of c * n convolution kernels of size w * h. Benefits of capsule network occur by regrouping n convolution kernels into c capsules so that they can represent complex spatial features and with the routing algorithm that produces links between them. We call shift the operation that consists of moving channels or entire capsules from one layer to another one. The same for a video sequence, but from the next (or previous in case of offline) inference of the network that treats another frame. In our approach, we implement and test three different temporal shift operations:

First dimension features of the first capsule. This implementation is a naive adaptation of the temporal shift module [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF] over capsules, since we can see a capsule layer as a R (w,h,cxn) convolution layer if we unroll the capsules. Hence, if we shift only the first convolution kernels of the layer, once roll again, we can see we shifted only the first dimension features of the first capsule (Fig. 1b).

First dimension features of all capsules. In this implementation, we consider capsules as what they are for, a set of convolutional kernels that, together, represent a more complex spatial feature (Fig. 1c). Our motivation is to consider the subspatial features inside each capsules independently to other capsules so that a shift of the first kernels of each capsules permits to capture temporal information of each complex spatial features encapsulated. The temporal shift is then an integral part of the capsule behavior.

Entire first capsules. This implementation is a mixed of the two others. As in the original TSM, they shift a small portion of spatial features extracted by convolutional kernels. In our case, we shift a small portion of complex spatial features extracted by capsules by shifting the entire first capsules (Fig. 1d). Our motivation is then relatively similar, temporal shift is independent to capsules behaviour, we want to share complex spatial features extracted between frames.

Offline vs Online. For offline scenarios, since we can treat an entire video at once, we can shift kernels to future frame, but also from future frame (Fig. 1b and Fig. 1c). This allows us to share a maximum of information inside a video, yet, we cannot get access of future frame for online implementation. Moreover, we can only passed our frame one by one to our network, contrary to offline case where we can pass all video frames in a batch. To address those problems, we can temporally cached kernel of capsule information into a temporal tensor so that we can reuse those information on the next frame.

Every implementation has the same structure in what we called a ShiftConvCaps (Fig. 2). As in TSM [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF], we added a residual branch inside each convolution capsule where the shift and convolution operations are performed. A naive implementation of shift module, where we linearly shift the input

B. Network Architecture

The architecture of our 2D Deep Video Capsule Network with ShiftConvCaps, skettched in Fig. 3, is inspired by the DeepCaps model [START_REF] Rajasegaran | Deepcaps: Going deeper with capsule networks[END_REF]. We conserve the basic structure of a Capsule Network with some first 2D convolution operations followed by convolution capsule and then classification ones. The convolution capsules are replaced with CapsCell [START_REF] Rajasegaran | Deepcaps: Going deeper with capsule networks[END_REF] that performs three convolution capsules and a residual one. All convolution capsules that performed a convolution stride of (1 * 1) are replaced with ShiftConvCaps.

Since we work on much complex problem with video understanding than image classification, we need to use larger reshape input images (128 * 112 * 3) compared to DeepCaps (64 * 64 * 3) or other 2D capsule networks. However, we observe a quick exploding gradient on the deepest capsule layer using such inputs if they are not enough reduced by the previous convolution layers. The original implementation of DeepCaps does not produce this problem since the first convolution and capsule layers produced very small outputs. However, we consider that these first layers reduced to much the dimension of the input and provoked a loss of information that can be really useful to deal with our video understanding problem. To address this problem, we kept reasonably large capsule dimension on the CapsCell by removing convolution strides in the middle of the network. Then, to offset this large scale data for the classification capsules, we added two extra ConvCaps to have treatable data on the classification capsule layer (Fig. 3). We also decided to augment convolution kernels size of ShiftConvCaps to get nearer from to original Capsule Network vision, observing better results in that way.

Finally, a basic reconstruction module with 2D deconvolution was put at the end of the network to reconstruct input image and assist the training by regularization. We also use the margin loss for loss function as described in Sabour et al. [START_REF] Sabour | Dynamic routing between capsules[END_REF]:

L k = T k max(0, m + -||v k ||) 2 +(1-T k )max(0, ||v k ||-m -) 2
(1) With m + = 0.9 and m -= 0.1 and where T k = 1 if the sequence belong to class k and ||v k || is the length of the classification capsules.

IV. EXPERIMENTS AND RESULTS

In this section, we first evaluate our proposed approach on two datasets and compare it with existing state-of-theart methods based only on color and depth images. We then demonstrate the impact of the temporal shift module over capsule network.

A. Datasets and implementation

We use two datasets to test our new architecture, First Person Hand Action (FPHA) [START_REF] Garcia-Hernando | First-person hand action benchmark with rgb-d videos and 3d hand pose annotations[END_REF] and Dynamic Hand Gesture Dataset (DHG28) [START_REF] Smedt | Dynamic hand gesture recognition-from traditional handcrafted to recent deep learning approaches[END_REF].

The FPHA dataset is a collection of RGB-D video sequences with more than 100k frames of 45 dialy hand action categories, involving 26 different objects in several hand configurations. It also contained 21 joints of a hand model via 6 magnetic sensors and inverse kinematic that we are not going to use in our experiments.

The DHG dataset contains sequences of 14 hand gestures performed in two ways : using one finger and the whole hand so we can consider 28 class gestures. Each frame of sequences contains a depth image and the coordinates of 22 joints forming a full hand skeleton both captured with The Intel RealSense short range depth camera. We only use depth image in our case.

We used Keras library [START_REF] Chollet | Keras: The python deep learning library[END_REF] and Tensorflow [START_REF] Abadi | Tensorflow: A system for largescale machine learning[END_REF] as backend for the development of 2D DVCN and trained it on the following configuration: Intel i9 9900k, Nvidia RTX Titan and 32 GB RAM. For all experiments, we used the Adam Optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with a starting learning rate of 0.001 and trained the network over 500 epochs on offline shift version.

B. Classification results

On the color data of the FPHA database [START_REF] Garcia-Hernando | First-person hand action benchmark with rgb-d videos and 3d hand pose annotations[END_REF], in order to avoid overfitting, we considerably reduce the size of our network with only 2 CapsCells and 4 capsules of dimension 8 in the remaining ShiftConvCaps. Such a configuration allows us to have a very small network with only 4 millions parameters and still improving results than with a much deeper network. We reduce the action sequences to 16 images per sequence, which is enough because the database provides high definition color images. Indeed, the action performed by the hand is not only a temporal but also a spatial problem. The subject interacts with an object so there is a spatial context that can counterbalance the necessity to deeply analyse the action over the time dimension. We observe that we obtain better classification results over other state-of-the-art methods on color stream with 45 time less parameters (TABLE I). We apply TSM [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF] method on FPHA to compare our temporal shift on capsule layers over them on convolution layers. We use their code obtained from Github. We can also see the benefit of capsules over traditional convolution with the 5% better classification than TSM [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF] with 5 times less parameters.

For the DHG28 dataset [START_REF] Smedt | Dynamic hand gesture recognition-from traditional handcrafted to recent deep learning approaches[END_REF] we consider the 28 labels cases and use the three CapsCells network as represented in Fig. 3 II). We also observe a 10% classification rate improvement compare to TSM [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF]. We approach state-of-the art method [START_REF] Zhang | Fusion of 2d cnn and 3d densenet for dynamic gesture recognition[END_REF] that uses fusion of 3D CNN and 2D CNN VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and have 20 times more parameters than our method. To prove the benefits of the temporal shift module over capsule network, we also conducted some experimentation by testing the same network with the same training conditions, same architecture, dataset and hyper-parameters but with or without the temporal shift module applied (TABLE III). We then prove that there is directly an improvement by applying two of the temporal shift operations we implement and that the temporal sharing information have a positive impact for classification with at least a 4% accuracy improvement. However, we observe significant degradation of results when applying the temporal shift on every kernel on the first capsule only (Fig. 1b). We can explain this degradation since this shift is the only one that share partial temporal information. Indeed, in this configuration, every other capsule have to deal, with the routing algorithm, with a capsule that have previous and current frame information ; this inconstant information leads to deterioration of the results. On the contrary, the two other implementations offer real improvement. We can see that temporal shift on every kernels of every capsules (Fig. 1c) and shifting all the first capsules (Fig. 1d) lead to similar improvement by outperforming the classification results. We can explain the improvement since the shift over entire capsule is the most logical way to share temporal information on a capsule network. Indeed, a capsule represents a complex spatial information by encoding in a vector, different characteristics of a same spatial features like the orientation or the scale. The two first implementations then, by shifting the firsts elements of these vectors, do not share pertinent information since they shift very specific and not contextual spatial information. Shifting entire capsules however move temporally all spatial information and explain the best improvement from this implementation. We can also see that, regardless of the temporal shift implementation, in addition to the final classification improvement, the training operation is faster to reach same precision in less epochs, even with the temporal shift on every kernels of the first capsule (Fig. 4).

V. CONCLUSION

In this paper, we proposed a new architecture for video understanding, 2D Deep Video Capsule Network that can deal with large image input without exploding gradient contrary to other capsule network architecture. We show that the integration and implementation of temporal shift module over capsule layer inject relevant and benefit temporal information to process video sequences with no additional computational cost. We also prove that shifting all capsules instead of internal kernels provides better results since we share all complex spatial information captured by capsules. Finally, we are outperforming the state-of-the-art on the FPHA dataset with 45 time less parameters and bring near best results on DHG28 but with 20 time less parameters.

Fig. 1 :

 1 Fig. 1: Visual implementation of our temporal shift over capsule layers. A color horizontal slice of a big cube represent a capsule. A big cube represent the same capsule at different temporality / frame. The two big cubes represent a ShiftConvCapsule layer (first and last capsules). Small cube represent a convolutional kernel of size h*w, channels are represented over a line.A white small cube is for zero-padding since the shift leave empty the first channels. In offline implementation, (a) and (b), channels are shifted in both temporal direction, past and future, since it's impossible in an online solution to get information from the future, an online shift has been implemented to progressively shift only to the future frame.

Fig. 2 :

 2 Fig. 2: ConvShiftCapsule representation

Fig. 3 :

 3 Fig. 3: Our 2D Deep Video Capsule network architecture[START_REF] Rajasegaran | Deepcaps: Going deeper with capsule networks[END_REF]

Fig. 4 :

 4 Fig. 4: Evolution of training precision over epochs on different shift operations

  , we obtained a 7M parameters network. Since we

	Method	Parameters Accuracy (%)
	Two stream-color [31]	46M	61.56
	Two stream-flow [31]	46M	69.91
	Two stream-all [31]	181M	75.30
	TSM [11]	24M	71.57
	2D CVNN	4M	76.72

TABLE I

 I

	: Comparing 2D DVCN on FPHA dataset [17] over
	other color stream methods
	have less spatial information, the database being composed
	of gestures only make in front of the camera, without any
	object interactions or background context, we decided to
	consider video sequences of 32 frames to extract as temporal
	information as possible (TABLE

TABLE II :

 II Comparing 2D DVCN on DHG28 databset[START_REF] Smedt | Dynamic hand gesture recognition-from traditional handcrafted to recent deep learning approaches[END_REF] over other depth stream methods

TABLE III :

 III Comparison of all temporal shift operations and no shift on the FPHA database[START_REF] Garcia-Hernando | First-person hand action benchmark with rgb-d videos and 3d hand pose annotations[END_REF]