Souria Hamidouche
email: souria.hamidouche@univ-lorraine.fr

Boris Arcen

Anne Tanière

Numerical prediction of sediment deposition at a river confluence using an Euler-Lagrange method

Keywords: natural confluence, sediment deposition, bed discordance, euler-lagrange method, dispersion model

Sediment deposition near to a river confluence is a crucial challenge for navigation since it can strongly affect the bed morphology and thus the barges traffic. The prediction of particle deposition is thus important in order to optimize the dredging operations which are necessary to maintain a sufficient draft. In the present study, we numerically examine the transport and deposition process of sediments at Scarpe and Escaut confluence (north of France) using an Euler-Lagrange approach. This method is known to be able to accurately predict the behaviour and spatial distribution of inertial particles in turbulent fluid flow. Nonetheless, contrary to the standard method (based on convection-diffusion equation for sediment concentration) usually used to predict the behaviour of sediments in such a complex configuration, it has an higher computational cost. In our study, the mean fluid flow is predicted by solving the 3D Reynolds-Averaged Navier-Stokes (RANS) equations. The trajectory of each sediment is then computed from the particle equation of motion with a stochastic dispersion model which predicts the sediment/turbulence interaction.

Introduction

Spatial distribution of sediment particles deposition in a confluence is an important aspect for the navigation of barges. Therefore, to ensure an appropriate depth for the navigation, it requires an expensive maintenance dredging. The need of controlling and optimising the dredging operations has motivated this study of particle deposition modelling within a natural bed confluence. A better understanding of the flow patterns, sediment deposition and bed evolution is necessary to keep the navigation of barges. However, sediment deposition in confluences is governed by a complex interaction between the convergence of incoming flows and bed morphology. This complexity of flow patterns was firstly studied experimentally by Taylor [START_REF] Taylor | Flow characteristics at rectangular openchannel junctions[END_REF] and was pursued in other studies [START_REF] Webber | An investigation of flow behaviour at the junction of rectangular channel[END_REF][START_REF] Biron | Effects of bed discordance on flow dynamics at open channel confluences[END_REF][START_REF] Bryan | Hydraulic conditions in experimental rill confluences and scour in erodible soils[END_REF][START_REF] Schindfessel | Flow patterns in an open channel confluence with increasingly dominant tributary inflow[END_REF]. The analysis of bed morphology evolution was carried out experimentally with a movable bed by [START_REF] Mosley | An experimental study of channel confluences[END_REF][START_REF] Best | Sediment transport and bed morphology at river channel confluences[END_REF][START_REF] Bryan | Hydraulic conditions in experimental rill confluences and scour in erodible soils[END_REF][START_REF] Guillen-Ludena | Hydro-morphodynamic evolution in a 90° movable bed discordant confluence with low discharge ratio[END_REF]. Field investigations were performed at river confluences primarily to study the major features of mean and turbulent structures of the flow and secondly to evaluate the relevance of experimental and numerical models on flow patterns and bed morphology under natural conditions [START_REF] Biron | Bed morphology and sedimentology at the confluence of unequal depth channels, measurement[END_REF][START_REF] Serres | Three-dimensional structure of flow at a confluence of river channels with discordant beds[END_REF][START_REF] Riley | Influence of junction angle on threedimensionalflow structure and bed morphology at confluent meander bends during different hydrological conditions[END_REF]. Although these field studies provided valuable insight of flow patterns for particular river conditions, they did not give pertinent parameters which control the nature of flow structures and the three-dimensional development of the incoming streams. Numerical modelling provides an alternative method to examine confluence characteristic since a detailed description of 3D patterns of the flow can be done, both for confluences with an artificial bed [START_REF] Schindfessel | Flow patterns in an open channel confluence with increasingly dominant tributary inflow[END_REF][START_REF] Biron | Threedimensional numerical modeling of mixing at river confluences[END_REF] and confluences with a natural bed [START_REF] Bradbrook | Numerical simulation of three-dimensional, timeaveraged flow structure at river channel confluences[END_REF][START_REF] Bradbrook | Large Eddy Simulation of periodic flow characteristics at river channel confluences[END_REF][START_REF] Constantinescu | Structure of turbulent flow at a river confluence with a momentum ratio close to one: insight provided by an eddy-resolving simulation[END_REF]. However, modelling particles dispersion or deposition in an artificial or natural river confluence remains undone. Previous experimental studies used for predicting the sediment deposition in rivers were mainly based on empirical criteria [START_REF] Hjulstrom | Studies of the morphological activity of rivers as illustrated by the River F[END_REF][START_REF] Shields | Application of similarity principles and turbulence research to bed-load movement[END_REF][START_REF] Vanrijn | Principales of sediment transport in rivers, estuaries ans coastal seas. Part I[END_REF] involving the settling particle velocity and bed shear stress. In other studies, authors deduced the particle deposition only on the analyses of the carrier fluid flow velocities. Indeed, authors treated sediment as particle without inertia (passive scalar), the particle deposition is then predicted only through the study of the characteristics of the carrier fluid flow as the mean longitudinal and wall-normal velocities, rms of velocities, etc.) [START_REF] Biron | Effects of bed discordance on flow dynamics at open channel confluences[END_REF][START_REF] Serres | Three-dimensional structure of flow at a confluence of river channels with discordant beds[END_REF]. Recently, nonintrusive image technics were used to obtain instantaneous measures of fluid flow and particles velocities in order to better understand the particle/fluid interaction [START_REF] Radice | Solid transport measurements through image processing[END_REF]. The present study will complement previous ones since the purpose of this paper is to provide a numerical investigation on the particle deposition process at a river confluence using an Euler-Lagrange approach which seems original since only few studies were dedicated particle deposition in rivers [START_REF] Shams | Computational modeling of flow and sediment transport and deposition in meandering rivers[END_REF]. Moreover, no study exists for the case of the confluence zone. This Lagrangian method was largely used to predict the deposition of aerosol particles [START_REF] Li | Dispersion and deposition of spherical particles from point sources in a Turbulent channel flow[END_REF]. In the present study, the carrier phase is calculated by using a 3D RANS approach whereas the second Newton law is used to predict the instantaneous motion of particles. However, only mean fluid flow is predicted with RANS approach whereas the particle equation of motion necessitates instantaneous fluid velocity information which has to be known at the particle location. A dispersion model is thus necessary, i.e. the fluctuating fluid velocity at the particle location must be predicted. As a first approximation, an eddy interaction model (EIM), available on Ansys Fluent code, will be used to generate such randomly fluctuating velocities of the fluid at particle position. The particle/turbulence interactions can be then better modelled since through the resolution of the particle equation, the spatial distribution of particle can be deduced. From the results, we will be able to analyse the effect of the gravity and of the particle /turbulence interactions on particle deposition. A one way coupling assumption is adopted in the present work.

Study site

The study area is the confluence of Scarp and Escaut which is located at Mortagne-du-Nord in France. The confluence has a symmetric planform (Y-shape) with a discordant bed. The two incoming channels join the post-confluence at an angle of about 37-39° (Fig. 1). Bathymetric data revealed the existence of a point bar at the upstream of the junction corner (see dashed black line in Fig. 1).

Figure 1 Confluence between the Scarpe River (SR) and Escaut River (ER). Contours show the confluence bathymetry (the free surface corresponds to z = 0 m).

Simulation procedure

A 3D RANS simulation

The flow of the natural bed confluence is modelling through the use of 3D Reynolds-Averaged Navier-Stokes simulations (RANS). The mean fluid flow is solved using RANS equation using the commercial software Ansys Fluent in Ref. [START_REF] Fluent | Theory Guide[END_REF]. This system involves the continuity equations Eqn (1) and the momentum equation Eqn [START_REF] Webber | An investigation of flow behaviour at the junction of rectangular channel[END_REF].

, 0    i i f x u  (1)  , j i f j i j j i j i j j i f i f u u x x u x u x x p x u u t u                                            (2)
where, i u (i = 1, 2, 3) are mean velocities, i x are coordinates, f  is the water density,  is the dynamic viscosity, p is the mean pressure and

j i u u  
is the Reynolds stresses which must be closed. The Reynolds stresses j i u u   in Eqn. 2 cannot be calculated directly and required additional models. Two model families are chosen, the k-ε and the k-ω which necessitate to solve two supplementary equations. In the k-ε family, the Renormalization Group (RNG) k-ε turbulence model coupled with Standard Wall Functions is used in Ref. [START_REF] Yakhot | Renormalization Group Analysis of Turbulence. I. Basic Theory[END_REF]. This modified version is better suited in flows with significant mean strain as is the case in confluences Ref. [START_REF] Bradbrook | Numerical simulation of three-dimensional, timeaveraged flow structure at river channel confluences[END_REF]. For the second one, the k-ω SST (Shear Stress Transport) model is adopted which is a combination of k-𝜀 and k- models model [START_REF] Menter | Two-equation eddy-viscosity turbulence models for engineering applications[END_REF]. In this model, a k- model is used in the inner boundary layer whereas a k-𝜀 model is used in the outer region of the boundary layer as well as outside of it Ref. [START_REF] Menter | Two-equation eddy-viscosity turbulence models for engineering applications[END_REF]. For more details, refer to Ansys Fluent code in Ref. [START_REF] Fluent | Theory Guide[END_REF].

A finite volume method is used to discretize the incompressible Navier-Stokes equations with steady state flow conditions. Coupling of velocity and pressure is achieved using the Semi-Implicit Method for Pressure-Linked Equations (SIMPLE) algorithm. The computational domain was meshed with about 4 million cells. Natural conditions of the river confluence correspond to the conditions at the site during the year 2010, issued from field measurements (Voies Navigables de France). As initial conditions for inflow sections (ER and SR) the mean mass flow rate released are 19.93 m 3 /s and 5.16 m 3 /s, respectively (See Fig. 1). The channel bed and lateral walls are treated as rough no-slip boundaries, while a symmetry condition is adopted for the free surface.

Particle trajectories

The present simulation deals with spherical, solid particles smaller than the smallest Kolmogorov length scale and with low density ratio (ρ f / ρ p <1, where subscription f and p denote fluid and particle, respectively). Consequently, under these assumptions and taking into account the effects of gravitational acceleration and drag forces, the particle of motion can be reduced to :

    , 1 p f p f p p p g u u dt u d              (3)
where p u  is the velocity of the particle with density p  . f u  is the fluid velocity at the particle location. g  is the gravitational acceleration. p  is the particle relaxation time which is defined as :

, 3 4 f p D p f p p u u C d        (4)
where p d is particle diameter and C D is the drag coefficient which depends on the flow regime. The expression of C D is given by Ref. [START_REF] Morsi | An investigation of particle trajectories in twophase flow systems[END_REF] as follow :

. Re Re

3 2 2 1 K K K C p p D    (5)
Here, the empirical constants K 1 , K 2 and K 3 depend on the particles Reynolds number and are detailed in Ref. [START_REF] Morsi | An investigation of particle trajectories in twophase flow systems[END_REF]. The use of a particle Lagrangian tracking necessitates to generate the fluid velocity fluctuation at particle location. This can be done through the use of a stochastic dispersion model, which is by default in Ansys fluent code, an eddy interaction model (EIM) developed by Ref. [START_REF] Gosman | Aspects of computer simulationof liquid-fueled combustors[END_REF]. The EIM is based on the concept of energy containing turbulent eddies, where turbulent dispersion of particles is modelled as a succession of interaction between these eddies and particles. The turbulent eddies are represented by instantaneous properties consisting of mean and fluctuating quantities. The RANS equations provide the averaged quantity and the fluctuating quantity are assumed to be isotropic and are selected randomly from a Gaussian probability distribution, whose standard deviation ( ii  ) is related to the turbulent kinetic energy (k) as follow :

. 3

2 2 2 2 k w v u ii         (6)
For particle trajectories calculation, a fifth-order Runge-Kutta method developed by Ref. [START_REF] Cash | A variable order Runge-Kutta method for initial value problems with rapidly varying right-hand sides[END_REF] is used. The total number of particles injected, with a diameter of 19.99 µm, in both ER and SR simultaneously is 10382 and their initial velocity is the same as the surrounding fluid. Every particle is tracked till it crosses the outlet boundary condition or hits the bed confluence, where a trap boundary condition is set. The maximum time steps to track every particle are fixed to 50,000,000, which was assumed to be sufficient for the slowest moving particles to have passed through the domain. However, these time steps are variable and the total traveling time is different per particle, therefore.

Results and discussions

Confluence morphology is governed by bed planform (i.e., discordance) and the momentum flux ratio (Mr) between the two incoming flows [START_REF] Serres | Three-dimensional structure of flow at a confluence of river channels with discordant beds[END_REF]. This latter is calculated for local conditions site as :

, ER v f SR v f V Q V Q Mr ER SR    (7)
where, v Q is discharge (m 3 /s) and V is cross-sectional mean velocity (m/s). Since the momentum ratio between the incoming rivers is about 0.23, a great influence hydrodynamic conditions is supposed to occur on discordance bed confluence. As a consequence, the penetration of ER into SR is increased Ref. [START_REF] Riley | Influence of junction angle on threedimensionalflow structure and bed morphology at confluent meander bends during different hydrological conditions[END_REF].

Flow characteristic at river confluence

The presence of a point bar at the upstream of the corner junction (Fig. 1) induces a decrease in the strength of the flow and minimizes the penetration of ER into SR. Results from Figure 2 shows clearly the development of a stagnation zone near the upstream of the junction corner, between section A and section B, where values of velocity magnitude are small. This zone extends into the tributary channel (SR), as it is observed by Ref. [START_REF] Riley | Influence of junction angle on threedimensionalflow structure and bed morphology at confluent meander bends during different hydrological conditions[END_REF]. A region of low velocity magnitude is also observed along the west bank. It can be seen, that these trends are observed whatever the models used (RNG k-ε and k-ω SST in Fig. 2). Figure 3 presents a zoom at the junction, where a recirculation zone is created upstream the junction. This zone is characterised by the presence of vortices.

Velocity (m/s)

RNG k-ε model k-ω SST model

RNG k-ε model k-ω SST model

Analysis of the mixing layer

The convergence of two no-parallel flows of unequal velocities creates a zone of shear, commonly called: zone of mixing layer. According to the literature, the position of this zone depends on the value of Mr (Ref. [START_REF] Serres | Three-dimensional structure of flow at a confluence of river channels with discordant beds[END_REF]), the angle between the two incoming streams (Ref. [START_REF] Best | The morphology of river channel confluences[END_REF]) and the change in bathymetry (i.e., bed discordance) [START_REF] Biron | Effects of bed discordance on flow dynamics at open channel confluences[END_REF][START_REF] Serres | Three-dimensional structure of flow at a confluence of river channels with discordant beds[END_REF]. In the mixing zone and with the presence of bed discordance, a lateral motion of the flow from the deeper channel (ER) towards the shallower channel, enhances this zone and causes the distortion of this latter [START_REF] Best | Mixmg-layer distortion at the confluence of different depth[END_REF][START_REF] Biron | Effects of bed discordance on flow dynamics at open channel confluences[END_REF]. In addition, when Mr <1, the mixing interface shifts to the tributary channel (SR), and this lateral distortion is illustrated by the orientation of high values of k observed in section A and B (Fig. 4 and Fig. 5), trends which have been observed in previous studies [START_REF] Best | Mixmg-layer distortion at the confluence of different depth[END_REF][START_REF] Biron | Effects of bed discordance on flow dynamics at open channel confluences[END_REF][START_REF] Serres | Three-dimensional structure of flow at a confluence of river channels with discordant beds[END_REF].

Near the junction (section A), the shear layer is characterized by a high level of turbulence [START_REF] Best | Mixmg-layer distortion at the confluence of different depth[END_REF][START_REF] Biron | Effects of bed discordance on flow dynamics at open channel confluences[END_REF][START_REF] Bradbrook | Large Eddy Simulation of periodic flow characteristics at river channel confluences[END_REF][START_REF] Ribeiro | Flow and sediment dynamics in channel confluences[END_REF]. Moreover, within this mixing layer, regions of reduced velocities correspond to the highest level of k. According to previous studies [START_REF] Ribeiro | Flow and sediment dynamics in channel confluences[END_REF], this shear layer is tridimensional, since it develops both vertically and laterally (Fig. 4 and Fig. 5). According to the results, the mixing layer is present whatever the models used (RNG k-ε and k-ω SST). However, the prediction of the intensity values of k is different. For the k-ω SST model, highest levels of k are found in section A, where the maximum value is about 2 cm 2 /s 2 (near the junction corner), while it is about 0.7 cm 2 /s 2 in section B. Regarding the RNG k-ε model at section A, the highest value is only 0.8 cm 2 /s 2 and is about 0.4 cm 2 /s 2 for the section B.

In this study, neither a separation zone nor helical cells are observed within the confluence, as it is predicted and presented by Ref. [START_REF] Best | Sediment transport and bed morphology at river channel confluences[END_REF]. According to the experimental study performed by Ref. [START_REF] Bryan | Hydraulic conditions in experimental rill confluences and scour in erodible soils[END_REF], it was found that the flow separation occurs only at 90° angle for symmetrical confluences. As a consequence, the absence of the flow separation can result either or both from: → the presence of depositional bar at the junction corner; → the presence of bed discordance.

In addition, these models (RNG k-ε and k-ω SST) are less successful in solving the secondary flow cells and predicting the details of the secondary flow contrary to a more sophisticated method like a Detached Eddy Simulation recently used for the simulation of a carrier phase [START_REF] Constantinescu | Structure of turbulent flow at a river confluence with a momentum ratio close to one: insight provided by an eddy-resolving simulation[END_REF]. According to the numerical simulations, it is difficult to make a choice about the turbulence model. We present the results of particle deposition using two models, therefore.

Particles deposition at confluence bed

The spatial distribution of particles deposition taking the particle/turbulence interaction and gravity effect (on the bed confluence) into account is shown in Fig 6 . It is seen that the spatial distribution of particles on the bed confluence does not present a significant difference whatever the models used, when particle/turbulence interactions and gravity effect are present. However, if one of the two is absent, the deposition rate or the spatial distribution of particles are greatly affected (Fig. 6c andd). In the case when none of them is present, only 1.3 % and 0.9 % of particles are deposited for RNG k-ε model and k-ω SST model, respectively (see Fig. 7). Indeed, the value of deposition rate is about the same when both gravity effect and particle/turbulence interactions are taken or not into account. (Fig. 7). Moreover, as observed in Ref. [START_REF] Ribeiro | Flow and sediment dynamics in channel confluences[END_REF], the formation of stagnation zone at the upstream of the junction corner of the confluence (Fig. 2 and Fig. 3) causes a sediment transfer between ER and post confluence near the downstream corner, which creates inhomogeneous deposition in bed confluence (Fig. 6a andb). In addition, a deposit trend is observed in the deepest part of the confluence and some of the highest levels of k (see Sections A and B). Some corridors (dashed black lines in Fig. 6) seem to appear. Whatever the results of particle deposition, it is interesting to note that no deposition occurs near the stagnation zone (at the edge of the junction), although this area is characterized by low fluid velocity values. A possible explanation is that a bar deposit (Fig. 1) is already present in this part of confluence.

Conclusion

In this paper, sediment deposition within a symmetrical confluence is investigated using an Eulerian/Lagrangian approach (RANS method coupled with a particle Lagrangian tracking) under the influence of the gravity and turbulence. turbulence kinetic energy in this zone are different according to the turbulence model. About the modelling of particle deposition in a natural confluence river, results show that the choice of the turbulence model has no real influence on the deposition results. Besides, particle/turbulence interactions or gravity effects have to be considered in simulations since they can have a strong influence on particle distribution and on deposition rate on the confluence bed. In the present study, the gravity effect combined to particle/turbulence interactions are the main deposition mechanisms.

Figure 2

 2 Figure 2 Velocity magnitude contours at z = -0.5 m for RANS simulations.

Figure 3

 3 Figure 3 Zoom at the junction. Velocity magnitude contours at z = -0.5 m for RANS simulations.

Figure 4

 4 Figure 4 Turbulent kinetic energy (k at cross sections A and B for RNG k-ε model (sections from Figure 2).

Figure 5

 5 Figure 5 Turbulent kinetic energy at cross sections A and B for k-ω SST model (sections from Figure 2).

Figure 6

 6 Figure 6 Spatial distribution of particle deposition : a-b) with both EIM and gravity: a) RNG k-ε model; b) k-ω SST model, c) RNG k-ε without EIM; d) RNG k-ε without gravity.

Figure 7

 7 Figure 7 Particles deposition rate : ⁺ RNG without EIM, ⁻RNG without gravity and * cases without gravity and EIM.

  Two turbulence models have been tested (RNG k-ε and k-ω SST turbulence models). Results show a well-developed stagnation zone at the upstream of the junction corner whatever the model used. A distortion of the mixing layer appear but values of the 0%


Reda Belaiche
email: reda.belaiche@u-bourgogne.fr

Yu Liu

Cyrille Migniot

Dominique Ginhac

Fan Yang

Cost-effective CNNs for real-time Micro-Expression recognition

Keywords: computer vision, deep learning, optical flow, micro facial expressions, real-time processing

  L'archive ouverte pluridisciplinaire

Introduction

Emotion recognition has received much attention in the research community in recent years.

Among the several sub-fields of emotion analysis, studies of facial expression recognition are particularly active [START_REF] Shan | Facial expression recognition based on local binary patterns: a comprehensive study[END_REF][START_REF] Edwards | Emotion recognition via facial expression and affective prosody in schizophrenia: a methodological review[END_REF][START_REF] Busso | Analysis of emotion recognition using facial expressions, speech and multimodal information[END_REF]. In contrast to the traditional macro-expression, people are less familiar with micro facial expressions [START_REF] Ekman | Nonverbal Leakage and Clues to Deception[END_REF][START_REF] Ekman | Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage[END_REF], and even fewer know how to capture and recognize them.

Micro-Expression (ME) is a rapid and involuntary facial expression that exposes a person's true emotion [START_REF] Haggard | Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy[END_REF]. These subtle expressions usually take place when a person conceals his or her emotions in one of the two scenarios: conscious suppression or unconscious repression. Conscious suppression happens when an individual deliberately prevents oneself from expressing genuine emotions. In contrary, unconscious repression occurs when the subject is not aware of his or her true emotions. In both cases, MEs reveal the subject's true emotions regardless of the subject's awareness. Intuitively, ME recognition has a vast number of potential applications across different sectors, such as security field, neuromarketing [START_REF] Vecchiato | On the use of EEG or MEG brain imaging tools in neuromarketing research[END_REF], automobile drivers' monitoring [START_REF] Nass | Improving Automotive Safety by Pairing Driver Emotion and Car Voice Emotion[END_REF] and lies and deceit detection [START_REF] Ekman | Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage[END_REF].

Psychological research shows that facial MEs generally are transient (e.g., remaining less than 200 ms) and very subtle [START_REF] Ekman | Lie Catching and Micro Expressions[END_REF]. The short duration and subtlety incur great challenges for human to perceive and recognize them. To enable better ME recognition by human, Ekman and his team developed the ME training tool (METT). Even with the help of this training tool, human can barely achieve around 40% accuracy [START_REF] Frank | I see how you feel: training laypeople and professionals to recognize fleeting emotions[END_REF]. Moreover, human's decisions are prone to be influenced by individual's perception varied along different subjects and time, resulting in less objective results. Therefore, a bias-free and high-quality automatic system for facial ME recognition is highly sought after.

A number of earlier solutions to automate facial ME recognition has been based on geometry or appearance feature extraction methods. Specifically, geometric-based features encode geometric information of the face, such as shapes and locations of facial landmarks. On the other hand, appearance-based features describe the skin texture of faces. Most existing methods [START_REF] Wang | Micro expression recognition using color spaces[END_REF][START_REF] Wu | The Machine Knows What You Are Hiding: An Automatic Micro-Expression Recognition System[END_REF] attempt to extract low-level features such as the widely used Local Binary Pattern from Three Orthogonal Planes (LBP-TOP) [START_REF] Pfister | Recognising spontaneous facial micro-expressions[END_REF][START_REF] Yan | CASMEII: an improved spontaneous micro-expression database and the baseline evaluation[END_REF][START_REF] Davison | Micro-facial movements: an investigation on spatio-temporal descriptors[END_REF] from different facial regions, and simply concatenate them for ME recognition.

Nevertheless, transient and subtle ME inherently makes it challenging for low level-features to effectively capture essential movements in ME. At the same time, these features can also be affected by irrelevant information or noise in video clips, which further weakens their discrimination capabilities especially on inactive facial regions with less dynamics [START_REF] He | Multi-task mid-level feature learning for micro-expression recognition[END_REF].

Recently, more approaches based on mid-level and high-level features have been proposed. Among these methods, the pipeline composed of optical flow and deep learning has demonstrated its high effectiveness for MEs recognition in comparison with traditional ones. The studies applying deep learning to tackle the ME classification problem usually considered well-known Convolutional Neural Networks (CNNs) such as ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] and VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. These studies re-purpose the use of off-the-shelf CNNs by giving them input data token from the optical flow extracted from the MEs. While achieving good performance, these neural networks are quite heavy in terms of memory usage and computation.

In specific applications, for example during automobile drivers' monitoring or students' comprehension recognition in virtual education systems, fast and effective processing methods are necessary to capture emotional responses as quickly as possible. Meanwhile, thanks to great progresses in parallel computing, parallelized image processing devices such as embedded systems are easily accessible and affordable. Already well-adopted in diverse domains, these devices possess multiple strengths in terms of speed, embeddability, power consumption and flexibility. These advantages however are often at the cost of limited memory and computing power.

The objective of this work is to design an efficient and accurate ME recognition pipeline for embedded vision purpose. First of all, our design takes into account thorough investigations on different CNN architectures. Next, different optical flow representations for CNN inputs have been studied. Finally, our proposed pipeline achieves competitive accuracy for ME recognition as state-of-the-art approaches while being real-time capable and using less memory. The paper is organized as follows. In Section 2, several recent related work are reviewed. Section 3 explains the proposed methodology in order to establish cost-effective CNNs for fast ME recognition. Section 4 provides experimental results and performance evaluations. Lastly, Section 5 concludes the paper.

Related works

MEs begin at the onset (first frame where the muscles of the facial expressions start to contract), finish at the offset (last frame, where the face returns to its neutral state), and reach their pinnacle at the apex frames (see Figure 1). Because of their very short duration and low intensity, ME recognition and analysis are considered as difficult tasks. Earlier studies proposed using low-level features such as LBP-TOP to address these problems. LBP-TOP is a 3D descriptor extended from the traditional 2D LBP. It encodes the binary patters between image pixels, and the temporal relationship between pixels and their neighboring frames. The resulting histograms are then concatenated to represent the temporal changes over entire videos. LBP-TOP has been widely adopted in several studies. Pfister et al. [START_REF] Pfister | Recognising spontaneous facial micro-expressions[END_REF] applied LBP-TOP for spontaneous ME recognition. Yan et al. [START_REF] Yan | CASMEII: an improved spontaneous micro-expression database and the baseline evaluation[END_REF] achieved 63% ME recognition accuracy on their CASME II database using LBP-TOP. In addition, LBP-TOP has also been used to investigate differences between micro-facial movement sequences and neutral face sequences. Several studies aimed to extend low-level features extracted by LBP-TOP as they still could not reach satisfactory accuracy. For example, Liong et al. [START_REF] Liong | Subtle expression recognition using optical strain weighted features[END_REF] proposed to assign different weights to local features, putting more attention on active facial regions. Wang et al. [START_REF] Wang | Micro expression recognition using color spaces[END_REF] studied the correlation between color and emotions by extracting LBP-TOP from the tensor independent color space (TICS).

Ruiz-Hernandez and Pietikäinen [START_REF] Ruiz-Hernandez | Encoding local binary patterns using re-parameterization of the second order Gaussian jet[END_REF] used the re-parameterization of second order Gaussian jet on the LBP-TOP, achieving promising ME recognition result on the SMIC database [START_REF] Li | A spontaneous micro-expression database: Inducement, collection and baseline[END_REF]. Considering that LBP-TOP consists of redundant information, Wang et al. [START_REF] Wang | LBP with six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition[END_REF] proposed the LBP-Six Intersection Points (LBP-SIP) method which is computationally more efficient and achieves higher accuracy on the CASEME II database. We also note that the STCLQP (SpatioTemporal Completed Local Quantization Patterns) proposed by Huang et al. [START_REF] Huang | Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns[END_REF] achieved a substantial improvement for analyzing facial MEs.

Over the years as research shows that it is non-trivial for low-level features to effectively capture and encode ME's subtle dynamic patterns (especially from inactivate regions), other methods shift to exploit mid-or high-level features. He et al. [START_REF] He | Multi-task mid-level feature learning for micro-expression recognition[END_REF] developed a novel multi-task mid-level feature learning method to enhance the discrimination ability of the extracted low-level features. The mid-level feature representation is generated by learning a set of class-specific feature mappings.

Better recognition performance has been obtained with more available information, features with better discrimination and generalization abilities. A simple and efficient method known as Main Directional Mean Optical-flow (MDMO) was employed by Liu et al. [START_REF] Liu | A Main directional mean optical flow feature for spontaneous micro-expression recognition[END_REF]. They used optical flow to measure the subtle movement of facial regions of interest (ROIs) that were spotted based on the Facial Action Coding System (FACS). Oh et al. [START_REF] Oh | Monogenic Riesz wavelet representation for micro-expression recognition[END_REF] also applied the monogenic Riesz wavelet representation in order to amplify subtle movements of MEs.

The aforementioned methods indicate that the majority of existing approaches heavily rely on hand-crafted features. Inherently, they are not easily transferable as the process of feature crafting and selection depend heavily on domain knowledge and researchers' experience. In addition, methods based on hand-crafted features are not accurate enough to be applied in practice. Therefore, high-level feature descriptors which better describe different MEs and can be automatically learned are desired.

Recently, more and more vision-based tasks have shifted to deep CNN-based solutions due to their superior performance. Recent developments in ME recognition are also inspired by these advancements by incorporating CNN models within the ME recognition framework.

Peng et al. [START_REF] Min | Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition[END_REF] proposed a two-stream convolutional network DTSCNN (Dual Temporal Scale Convolutional Neural Network) to address two aspects: overfitting problem caused by small sizes of existing ME databases and use of high-level features. We can observe four characteristics of DTSCNN:

(i) separate features were first extracted from ME clips from two shallow networks and then fused;

(ii) data augmentation and higher drop-out ratio were applied in each network; (iii) two databases (CASME I and CASME II) were combined to train the network; (iv) the data fed to the networks were optical-flow images instead of raw RGB frames.

Khor et al. [START_REF] Khor | Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[END_REF] studied two variants of an Enriched LRCN (Long-term Recurrent Convolutional Network) model for ME recognition. Spatial enrichment (SE) refers to channel-wise stacking of gray-scale and optical flow images as new input to CNN. On the other hand, temporal enrichment (TE) stacks obtained features. Their TE model achieves better accuracy on a single database, while the SE model is more robust against the cross-domain protocol involving more databases.

Liong et al. [START_REF] Liong | Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition[END_REF] designed a Shallow Triple Stream Three-dimentional CNN (STSTNet). The model takes input stacked optical flow images computed between the onset and apex frames (optical strain, horizontal and vertical flow fields), followed by three shallow convolution layers in parallel and a fusion layer. The proposed method is able to extract rich features from MEs while being computationally light, as the fused features are compact yet discriminative.

Our objective is to realize a fast and high-performance ME recognition pipeline for embedded vision applications under several constraints, such as embeddability, limited memory and restricted computing resources. Inspired by existing works [START_REF] Min | Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition[END_REF][START_REF] Liong | Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition[END_REF], we explore different CNN architectures and several optical flow representations for CNN inputs to find cost-effective neural network architectures that are capable of recognizing MEs in real-time.

Methodology

The studies applying deep learning to tackle the ME classification problem [START_REF] Patel | Selective deep features for micro expression recognition[END_REF][START_REF] Li | Can micro-expression be recognized based on single apex frame?[END_REF][START_REF] Wang | Micro-expression recognition with small sample size by transferring long-term convolutional neural network[END_REF][START_REF] Gan | Off-apexnet on micro-expression recognition system[END_REF] usually used pretrained CNNs such as ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] and VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and applied transfer learning to obtain ME features. In our work, we first select off-the-shelf ResNet18 because it provides the best trade-off between accuracy and speed on the challenging ImageNet classification and is recognized for these performances in transfer learning. ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] explicitly lets the stacked layers fit a residual mapping.

Namely, the stacked non-linear layers are let to fit another mapping of F(x) := H(x)x where H(x) is the desired underlying mapping and x the initial activations. The original mapping is recast into F(x) + x by feedforward neural networks with shortcut connections. ResNet18 has 20 convolutional layers (CL) (17 successive CL and 3 branching ones). Residual links after each pair of successive convolutional units are used and the kernel size after each residual link is doubled. As ResNet18 is designed to extract features from RGB color images, it requires inputs to have 3 channels.

In order to accelerate processing speed in the deep learning domain, the main current trend in decreasing complexity of CNN is to reduce the number of parameters. For example, Hui et al. [START_REF] Hui | LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation[END_REF] proposed a very compact LiteFlowNet which is 30 times smaller in the model size and 1.36 times faster in the running speed in comparison with the state-of-the-art CNNs for optical flow estimation. In Well known CNN is created for specific problems and therefore over calibrated when they are used in other contexts. ResNet18 was made for end-to-end object recognition: the dataset used for training had hundreds of thousands of images for each class and more than a thousand classes in total.

Based on the fact that: (i) ME recognition study considers in maximum 5 classes and the datasets of spontaneous MEs are scarce and contain much fewer samples, and (ii) optical flows are high-level features contrary to low-level color features and so require shallower network, we have reduced the architecture of ResNet18 by iteratively removing residual layers. This allows us to assess the influence of the depth of the network on its classification capacities in our context and therefore to estimate the relevant calibration of the network.

Figure 2 illustrates the reduction protocol: at each step the last residual layer with two CL is removed and the previous one is connected to the fully connected layer. Only networks with an odd number of CL are therefore proposed. As highlighted in Table 1 the decrease in the number of CL poses a significant impact on the number of learnable parameters of the network, which directly affects the forward propagation time. Once the network depth has been correctly estimated, the dimension of the input has to be optimized. In our case, CNNs take optical flows extracted between the onset and apex frames of ME video clips. It is between these two moments that the motion is most likely to be the strongest.

The dimensionality of inputs determines the complexity of the network that uses them since the reduction in input channels dictates the number of filters to be used throughout all following layers of the CNN. The optical flow between the onset (Figure 3-a) and the apex (Figure 3-b) typically has a 3-channel representation to be used in a pretrained architecture designed for 3-channel color images.

This representation however may not be optimal for ME recognition.

Optical flow can be described as the change of structured patterns of light between successive frames to measure movement of a pixel over a period of time. Optical flow estimation techniques are based on the assumption of brightness invariance:

I(x, y, t) = I(x + δ x , y + δ y , t + δ t ) (1)
where I(x, y, t) is the intensity of pixel in position (x, y) at time t.

The optical flow is represented as a vector (Figure 3-c) indicating the direction and intensity of the motion. The projection of the vector on the horizontal axis corresponds to the Vx field (Figure 3-d)

while its projection on the vertical axis is the Vy field (Figure 3-e). The magnitude M is the norm of the vector (Figure 3-f). Figure 4 illustrates this representation of one optical flow vector. The horizontal and vertical components V x and V y of the optical flow correspond to the spatial variation (δ x , δ y ) obtained by minimizing the difference between the left and right term of Equation 1.

In this paper, the optical flow is estimated by the Horn-Schunck method [START_REF] Horn | Determining optical flow[END_REF]. This method assumes that the optical flow is smooth over the entire image. Hence minimizing the following equation estimates the velocity field:

E = ( ∂I ∂x V x + ∂I ∂y V y + ∂I ∂t ) 2 dxdy + α ∇V x 2 + ∇V y 2 dxdy (2)
α is a regularization parameter that controls the degree of smoothness and is usually selected heuristically. This energy is iteratively minimized until convergence from the following equations :

V k+1 x = V k x - ∂I ∂x ( ∂I ∂x V k x + ∂I ∂y V k y + ∂I ∂t ) α 2 + ∂I ∂x 2 + ∂I ∂y 2 (3) V k+1 y = V k y - ∂I ∂y ( ∂I ∂x V k x + ∂I ∂y V k y + ∂I ∂t ) α 2 + ∂I ∂x 2 + ∂I ∂y 2 (4)
where V is the weighted average of V in a neighbourhood. When classifying ME, the resulting matrices Vx , Vy and M are traditionally given as input to the CNN. Nonetheless, the third channel is inherently redundant since M is computed from Vx and Vy. Optical flow composed of the 2-channel Vx and Vy field could already provide all relevant information. Furthermore, we hypothesize that even a single channel motion field itself could be descriptive enough. Hence we have created and evaluated networks taken as input the optical flow in a two-channel representation (Vx-Vy) and in an one-channel representation (M, Vx or Vy). For this purpose, the proposed networks begin by a number of CL related to the depth optimization followed by a batch normalization and ReLU. Then the networks end by a maxpooling layer and a fully connected layer. The Figure 5 presents the architectures used with one to four CL according to the results of the experiments in Section 4. As illustrated in Table 2, a low dimensional input leads to a significant reduction in the number of learnable parameters and therefore in the complexity of the system.

Experiments

Dataset and validation protocol presentation

Two ME databases are used in our experiments. CASME II (Chinese Academy of Sciences Micro-Expression) [START_REF] Yan | CASMEII: an improved spontaneous micro-expression database and the baseline evaluation[END_REF] is a comprehensive spontaneous ME database containing 247 video samples, collected from 26 Asian participants with an average age of 22.03 years old. Compared to the first database, the Spontaneous Actions and Micro-Movements (SAMM) [START_REF] Davison | A Spontaneous Micro-Facial Movement Dataset[END_REF] is a more recent one consisting of 159 micro-movements (one video for each). These videos are collected spontaneously from a demographically diverse group of 32 participants with a mean age of 33.24 years old and a balanced gender split. Originally intended for investigating micro-facial movements, SAMM initially collected the 7 basic emotions.

Both the CASME II and SAMM databases are recorded at a high-speed frame rate of 200 fps. They also both contain "objective classes", as provided in [START_REF] Davison | Objective classes for micro-facial expression recognition[END_REF]. For this reason, Facial MEs Grand Challenge 2018 [START_REF] Yap | Facial Micro-Expressions Grand Challenge 2018 Summary[END_REF] proposed to combine all samples from both databases into a single composite dataset of 253 videos with five emotion classes. It should be noted that the repartition is not very well balanced.

Namely, this composite database is composed of 19.92% "happiness", 11.62% "surprise", 47.30% "anger", 11.20% "disgust" and 9.96% "sadness".

Similar to [START_REF] Yap | Facial Micro-Expressions Grand Challenge 2018 Summary[END_REF], we applied the Leave One Subject Out (LOSO) cross-validation protocol for ME classification, where one subject's data is used as a test set in each fold of the cross-validation. This is done to better reproduce realistic scenarios where the encountered subjects are not present during training of the model. In all experiments, recognition performance is measured by accuracy, which is the percentage of correctly classified video samples out of the total number of samples in the database.

The Horn-Schunck method [START_REF] Horn | Determining optical flow[END_REF] was selected to compute optical flow. This algorithm is widely used for optical flow estimation in many recent studies in virtue of its robustness and efficiency.

Throughout all experiments, we train the CNN models with a mini-batch size of 64 for 150 epochs using the RMSprop optimization. Feature extraction and classification are both handled by the CNN.

Simple data augmentation is applied to double the training size. Specifically, for each ME video clip used for training, in addition to the optical flow between the onset and apex frame, we also include a second flow computed between the onset and apex+1 frame.

ResNet depth study

In order to find the ResNet depth which permits an optimal compromise between the ME recognition performance and the number of learnable parameters, we tested different CNN depths using the method described in Section 3. The obtained accuracies are given in Table 3: We observe that the best score is achieved by ResNet8 which has seven CL. However, the scores achieved by different numbers of CL do not vary much. Furthermore, beyond seven CL, adding more CL doesn't improve the accuracy of the model. The fact that accuracy doesn't increase along with depth confirms that multiple successive CL are not necessary to achieve a respectable accuracy. The most interesting observation is that with a single CL, we achieve a score that is not very far from the optimal score while the size of the model is much more concise. This suggests that instead of deep learning, a more "classical" approach exploiting shallow neural networks presents an interesting field to explore when considering portability and computation efficiency for embedded systems. That is the principal reason we will restrict our study to shallow CNNs.

CNN input study

In this subsection, we study impacts of optical flow representations on ME recognition performance. Two types of CNN have been investigated, one with 1-channel input (Vx, Vy, or M) and the other one using the 2-channel Vx-Vy pair. Due to the fact that off-the-shelf CNNs typically take 3-channel inputs and are pre-trained accordingly, applying transfer learning to adapt to our models is a nontrivial task. Instead, we created custom CNNs and trained them from scratch. Table 4 shows recognition accuracies of different configurations using a small number of CNN layers.

We can observe that the Vx-Vy pair and Vy alone give the best results, both representations achieving 60.17% accuracy. On the other hand, using magnitude alone leads to similar accuracy as those of Vy and Vx-Vy pair with a score of 59.34%. Vx gets the worst results overall, with a maximum score of 54.34%. This observation indicates that the most prominent features for ME classification might indeed be more dominant in vertical movement rather than the horizontal one. This assumption is logical when thinking about the muscle movements happening in each known facial expression. To better visualize the difference in the high-level features present in Vx, Vy and the Magnitude, we did an averaging on all the different samples according to their classes. The result can be seen in

Classification analysis

In order to understand obtained results, we measured cosine similarity of features extracted by three CNNs: ResNet8 (Section 4.2), Vx-Vy-3 CL and Vy-3 CL (Section 4.3). Usually, the convolutional layers of CNNs are considered as different feature extractors; only the last fully connected layer directly performs the classification task. The features just before classification can be represented in vector format. Cosine similarity measures the similarity between two vectors a and b using Equation 5:

cosine(a, b) = a T b a b (5)
Cosine similarity values fall within the range of [- Firstly, we observe that diagonal values (intra-class) across all three CNNs are significantly higher in comparison with other values (inter-class). This illustrates that all three CNNs are capable to separate different ME classes. Secondly, the intra-class cosine similarity of ResNet is closer to 1, suggesting that ResNet features are more discriminative. We hypothesize that our simplified CNNs with reduced layers extract less refined features, resulting in the minor decrease in performance (61.00% vs. 60.17%).

Performance evaluations

In this subsection, we measure our proposed method on three aspects: recognition accuracy, needed memory space and processing speed. Since we obtain optimal results by using the Vy field and 3-layer CNN, further evaluations will concentrate on this particular configuration.

Evaluation on recognition accuracy: we performed an accuracy comparison of 5 objective ME class recognition (see Table 8). Our best CNN reaches similar performance as those of other studies using the same protocol of validation. It is worth mentioning that Peng et al. [START_REF] Peng | From Macro to Micro Expression Recognition: Deep Learning on Small Datasets Using Transfer Learning[END_REF] employed a macro-to-micro transferred ResNet10 model and obtained a better result. Their work used 4 macro-expressions datasets (> 10K images) and some preprocessing such as colour shift, rotation and smoothing. These additional operations make their proposed method difficult for deployment on embedded systems. Seeing the confusion matrix of our model (Figure 7), we can also notice that the distribution of correct assessments for Vy is more balanced than the ones gotten from [START_REF] Khor | Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[END_REF] (Figure 8).

The DTSCNN proposed by Peng et al. in [START_REF] Min | Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition[END_REF] Table 8. Comparison between our method and those of other top-performers from literature.

Method Accuracy

LBP_TOP [START_REF] Khor | Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[END_REF] 42.29% Khor et al [START_REF] Khor | Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[END_REF] 57.00% Peng et al [START_REF] Peng | From Macro to Micro Expression Recognition: Deep Learning on Small Datasets Using Transfer Learning[END_REF] 74.70% Proposed method 60.17% Evaluation on memory space: Table 9 summarizes the number of learnable parameters and used filters according to the dimensionality of the network inputs. The minimum required memory space corresponds to 333,121 parameter storage, which is less than 3.12% of that of off-the-shelf ResNet18. Evaluation on processing speed: we used a mid-range computer with an Intel Xeon processor and an Nvidia GTX 1060 graphic card to carry out all the experiments. The complete pipeline is implemented in MatLAB 2018a with its deep learning toolbox. Our model which achieves the best score is the CNN with a single-channel input and three successive CL. It needs 12.8 ms to classify the vertical component Vy. The optical flow between two frames requires 11.8 ms to compute using our computer, leading to a total runtime to classify an ME video clip of 24.6 ms. In our knowledge, the proposed method outperforms most ME recognition systems in terms of processing speed.

Conclusion and future works

In this paper, we propose cost-efficient CNN architectures to recognize spontaneous MEs. We first investigated the depth of the well-known ResNet18 network to demonstrate that using only a small number of layers is sufficient in our task. Based on this observation, we have experienced several representations at network's input. Finally, we obtained an accuracy of 60.17% with a light CNN design consisting of 3 CL with single-channel inputs Vy. This configuration enables the number of learnable parameters to be reduced by a factor of 32 in comparison with the ResNet18. Moreover, we achieved a processing time of 24.6 ms which is shorter than MEs (40 ms). Our study opens an interesting way to find the trade-off between speed and accuracy in ME recognition. While the results are encouraging, it should be noted that our method does not give a better accuracy than the ones described in the literature. Instead, a compromise has to be made between accuracy and processing time. By minimizing the computation, our proposed method manages to obtain accuracy comparable to the state-of-the-art systems while being compatible with the real-time constraint of embedded vision.

Several future works could further enhance both the speed and accuracy of our proposed ME recognition pipeline. These include more advanced data augmentation techniques to improve recognition performance. Moreover, new ways to automatically optimize the structure of a network to make it lighter have been presented recently. Other networks optimized for efficiency will also be explored. For example, MobileNet [START_REF] Howard | Efficient Convolutional Neural Networks for Mobile Vision Applications[END_REF] uses depth-wise separable convolutions to build light weight CNN. ShuffleNet [START_REF] Zhang | An Extremely Efficient Convolutional Neural Network for Mobile Devices[END_REF] uses pointwise group convolution to reduce computation complexity of 1x1 convolutions and channel shuffle to help the information flowing across feature channels. Our next step of exploration aims to analyze and integrate these new methodologies in our framework.

Figure 1 .

 1 Figure 1. Example of a ME: the maximum movement intensity occurs at the apex frame.

[ 34 ]

 34 , Rieger et al. explored parameter-reduced residual networks on in-the-wild datasets, targeting real-time head pose estimation. They experimented various ResNet architectures with a varying number of layers to handle different image sizes (including low-resolution images). The optimized ResNet achieved state-of-the-art accuracy with real-time speed.

Figure 2 .

 2 Figure 2. Depth reduction of a deep neural network: in the initial network, each residual layer contains two CL (left); the last residual layer is removed (middle) to obtain a shallower network (right).

  onset

Figure 3 .

 3 Figure 3. Optical flow is computed between the onset (a) and the apex (b): vectors obtained for a random sample of pixels (c), Vx field (d), Vy field (e) and magnitude field (f).

Figure 4 .

 4 Figure 4. Visualisation of M, Vx and Vy for one optical flow vector.

Figure 5 .

 5 Figure 5. Proposed networks composed of one to four (from left to right) CL for various representations of the optical flow as input.

Figure 6 .

 6 Figure 6. We observe that Vx exhibits a non-negligible quantity of noise. Magnitude and Vy on the other hand have clear regions of activity for each class. The regions of activity are aligned with the muscles responsible of each facial expression.

Figure 6 .

 6 Figure 6. Average optical flow obtained in the dataset per ME class. Studied classes are in order from left to right: happiness, surprise, anger, disgust and sadness.

  . The STSTNet proposed by Liong et al. in[START_REF] Liong | Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition[END_REF] makes use of three dimensional CNNs which carry out three dimensional convolutions instead of two-dimensional ones (such as ResNet, VGG, the networks presented in[START_REF] Min | Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition[END_REF],[START_REF] Khor | Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[END_REF],[START_REF] Peng | From Macro to Micro Expression Recognition: Deep Learning on Small Datasets Using Transfer Learning[END_REF] and our study). It was tested on three classes: positive, negative, and surprise from a composite database consisting of samples from the SMIC, CASME II and SAMM databases. It achieved an unweighted average recall rate of 76.05% and an unweighted F1-score of 73.53%. Both of these two frameworks are not very suitable for real time embedded applications constrained by limited memory and computing resources.

Figure 7 .

 7 Figure 7. Confusion matrix corresponding to our network with 3 CL and Vy as input.

Figure 8 .

 8 Figure 8. Confusion matrix obtained by the work of [27].

Following

  several previous studies, we fed CNNs with optical flow estimated from the onset and apex of MEs. Different flow representations (horizontal Vx, vertical Vy, Magnitude M and Vx-Vy pair) have been tested and evaluated on a composite dataset (CASME II and SAMM) for recognition of five objective classes. The results obtained on the Vy input alone are more convincing. It is likely due to the fact that such an orientation is more suitable describing ME's motion and its variations between the different expression classes. Experimental results demonstrated that the proposed method can achieve similar recognition rate when compared with state-of-the-art approaches.

Table 1 .

 1 Number of CL and number of learnable parameters in the proposed architectures.

	CL	17	15	13	11	9	7	5	3	1
	Nb. of param. 10,670,932 5,400,725 2,790,149 1,608,965 694,277 398,597 178,309 104,197 91,525

Table 2 .

 2 Number of learnable parameters according to the dimensionality of the input of the network.

	Input	1 CL	2 CL	3 CL	4 CL
	Single channel	82,373	168,997 333,121	712,933
	Double channel 165,541 348,005 709,477 1,620,197

Table 3 .

 3 Accuracies varied by the number of convolution layers (CL) and associated number of learnable parameters.

	Nb. of CL	17	15	13	11	9	7	5	3	1
	Nb. of param. 10,670,932 5,400,725 2,790,149 1,608,965 694,277 398,597 178,309 104,197	91,525
	Accuracy	57.26%	57.26%	60.58%	59.34%	60.17% 61.00% 58.51% 60.17% 58.92%

Table 4 .

 4 Accuracies under various CNN architectures and optical flow representations.

		1 CL	2 CL	3 CL	4 CL
	Vx	52.24% 54.34% 53.92% 53.50%
	Vy	58.09% 59.34% 60.17% 60.17%
	Vx-Vy 58.51% 59.75% 60.17% 58.09%
	M	58.09% 58.92% 59.34% 59.34%

Table 5 .

 5 1, 1]; values closer to 1 indicate higher similarity between two vectors. Tables 5, 6 and 7 display the cosine similarity values: with 2 samples ×5 ME classes, we calculated intra-similarity and average inter-similarity of each class using the same configuration for three CNNs. Cosine similarity for the 3 CL CNN with single-channel input Vy

		happiness surprise anger disgust sadness
	happiness	0.6007	0.1320	0.0574	0.0146	0.1154
	surprise	0.1320	0.5572	0.0485	0.0667	0.1415
	anger	0.0574	0.0485	0.5260	0.0318	0.0698
	disgust	0.0146	0.0667	0.0318	0.5663	0.0159
	sadness	0.1154	0.1415	0.0698	0.0159	0.5099
		happiness surprise anger disgust sadness
	happiness	0.5615	0.1700	0.1171	0.1155	0.1195
	surprise	0.1700	0.5831	0.1432	0.1502	0.1618
	anger	0.1171	0.1432	0.5672	0.1176	0.1503
	disgust	0.1155	0.1502	0.1176	0.5447	0.1225
	sadness	0.1195	0.1618	0.1503	0.1225	0.5443

Table 6 .

 6 Cosine similarity for the 3 CL CNN with double-channel inputs (Vx-Vy)

		happiness surprise anger disgust sadness
	happiness	0.8464	0.3966	0.3860	0.3126	0.2960
	surprise	0.3966	0.8159	0.4040	0.3362	0.3324
	anger	0.3860	0.4040	0.8344	0.3654	0.3307
	disgust	0.3126	0.3362	0.3654	0.8598	0.2363
	sadness	0.2960	0.3324	0.3307	0.2363	0.9343

Table 7 .

 7 Cosine similarity for ResNet8

  opted for two optical flows computed differently from a ME sample, which make the whole network robust to different frame rate of ME videos. In detail, the first optical flow is calculated using 64 frames around the apex to adapt to the frame rate of CASME I. Similarly, the second optical flow is given by the 128 frames around the apex adapted to the frame rate of CASME II. In case the number of frames composing the ME is not sufficient, a linear interpolation method is used to normalize the video clips. Their study uses two CNNs in parallel to extract two separate features before concatenating them. The resulted feature vector is then fed as input to an SVM to be classified. The DTSCNN was tested on four classes (positive, negative, surprise, and other) from a composite dataset consisting of the CASME I and CASME II databases, and it achieves an average recognition rate of 66.67%

Table 9 .

 9 Number of learnable parameters and filters (in brackets) of various network architectures under different input dimensions.

	Input	1 CL	2 CL	3 CL	4 CL
	Single channel	82,373	168,997 333,121	712,933
		(16)	(48)	(112)	(240)
	Double channel 165,541 348,005 709,477 1,620,197
		(32)	(96)	(224)	(480)

Funding: This work was supported by the H2020 Innovative Training Network (ITN) project ACHIEVE (H2020-MSCA-ITN-2017: agreement no. 765866).

Conflicts of Interest:

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:

ME Micro Expression CL

Convolutional Layer M Magnitude