Hassen Drira
email: hassen.drira@telecom-lille1.eu

Boulbaba Ben Amor

Mohamed Daoudi

Anuj Srivastava

Stefano Berretti

3D Dynamic Expression Recognition based on a Novel Deformation Vector Field and Random Forest

This paper proposes a new method for facial motion extraction to represent, learn and recognize observed expressions, from 4D video sequences. The approach called Deformation Vector Field (DVF) is based on Riemannian facial shape analysis and captures densely dynamic information from the entire face. The resulting temporal vector field is used to build the feature vector for expression recognition from 3D dynamic faces. By applying LDA-based feature space transformation for dimensionality reduction which is followed by a Multiclass Random Forest learning algorithm, the proposed approach achieved 93% average recognition rate on BU-4DFE database and outperforms state-of-art approaches.

Introduction and related work

Facial expressions analysis and recognition from 2D and/or 3D data has emerged as an active research topic with applications in several different areas, such as human-centered user interfaces, computer graphics, facial animation, ambient intelligence, etc. In recent years, there has been tremendous interest in tracking and recognizing facial expressions over time, it is suggested that the dynamics of facial expressions provides important cues about the underlying emotions that are not available in static 2D or 3D images.

In 3D-based researches, there are a few works that use 4D data for facial expression analysis. In [START_REF] Sun | Facial expression recognition based on 3d dynamic range model sequences[END_REF], a spatio-temporal expression analysis approach based on 3D dynamic geometric facial model sequences is proposed. The approach integrates a 3D facial surface descriptor and Hidden Markov Models (HMMs) to recognize facial expressions. Experiments were performed on the BU-4DFE and their approach achieved a 90.44% (using spatio-temporal) and 80.04% (using temporal only) recognition rates for distinguishing the six prototypic facial expressions (anger, disgust, fear, happiness, sadness and surprise). The main limitation of this solution resides in the use of 83 proprietary annotated landmarks of the BU-4DFE that are not released for public use. An extension of this work for 4D face recognition appeared in [START_REF] Sun | Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis[END_REF]. Sandbach et al. proposed in [START_REF] Sandbach | A dynamic approach to the recognition of 3d facial expressions and their temporal models[END_REF] to exploit the dynamics of 3D facial motions to recognize observed expressions. First, they captured motion between frames using Free-Form Deformations and extract motion features using a quadtree decomposition of several motion fields. Second, they used HMM models for temporal modeling of the full expression sequence to be represented by 4 segments which are neutral-onset-apex-offset expression segments. In their approach, features selection is derived using GentleBoost technique applied on the onset and offset temporal segments of the expression, the average correct classification results for three basic expressions (happy, angry and surprise) achieved 81.93% in whole sequence-based classification and 73.61% in frame-by-frame classification. An extension of this work is presented in [START_REF] Sandbach | Recognition of 3d facial expression dynamics[END_REF]. Another work on 4D facial expression recognition is proposed by Le et al. [START_REF] Le | Expression recognition from 3d dynamic faces using robust spatio-temporal shape features[END_REF]. They proposed a level curve based approach to capture the shape of 3D facial models. The level curves are parameterized using the arc-length function. The Chamfer distances is applied to measure the distances between the corresponding normalized segments, partitioned from these level curves of two 3D facial shapes. These measures are then used as spatio-temporal features to train Hidden Markov Model (HMM), and since the training data were not sufficient for learning HMM, the authors propose to apply the universal background modeling (UBM) to overcome the over-fitting problem. Using the BU-4DFE database to evaluate their approach, they reached an overall recognition accuracy of 92.22% for three prototypic expressions (happy, sad and surprise), when classifying whole sequences.

In the present work, we present a fully automatic facial expression recognition approach based on 4D (3D+t) data. To capture the deformation across the sequences, we propose a new method called Deformation Vector Filed, obtained via facial surfaces parametrization by collections of radial curves emanating from their tips of the noses. Our paper is organized as follows: In Sect. 2, a face representation model is proposed that captures facial features relevant to categorize expression variations in 3D dynamic sequences. In Sect. 3, the LDA-based feature space transformation and Multiclass Random Forest based classification are addressed. Experimental results and comparative evaluation obtained on the BU-4DFE are reported and discussed in Sect. 4. Finally, conclusions are outlined in Sect. 5.

The Deformation Vector Field

One basic idea to capture facial deformation across 3D video sequences is to track densely meshes' vertices along successive 3D frames. To do so, as the meshes resolutions vary across 3D video frames, establishing a dense matching on consecutive frames is necessary. Sun et al. [START_REF] Sun | Facial expression recognition based on 3d dynamic range model sequences[END_REF] proposed to adapt a generic model (a tracking model) to each 3D frame. However, a set of 83 predefined key-points is required to control the adaptation based on radial basis function. A second solution is presented by Sandbach et al. [START_REF] Sandbach | A dynamic approach to the recognition of 3d facial expressions and their temporal models[END_REF], where the authors used an existing non-rigid registration algorithm (FFD) based on B-splines interpolation between a lattice of control points. The dense matching is a step of preprocessing stage to estimate a motion vector field between frames t and t-1. However, the results provided by the authors are limited to three facial expressions: happy, angry and surprise. To address this problem, we propose to represent facial surfaces by a set of parameterized radial curves emanating from the tip of the nose. Such an approximation of facial surfaces by indexed collection of curves can be seen as solution to facial surface parameterizations which capture locally their shapes.

Shape Deformation Capture

A parameterized curve on the face, β : I → R 3 , where I = [0, 1], is represented mathematically using the square-root velocity function [START_REF] Srivastava | Shape analysis of elastic curves in euclidean spaces[END_REF], denoted by q(t), according to:

q(t) = β(t) √ β(t)
. This specific parametrization has the advantage of capturing the shape of the curve and provides simple calculus. Let define the space of such functions: C = {q : I → R 3 , q = 1} ⊂ L 2 (I, R 3 ), where • implies the L 2 norm. With the L 2 metric on its tangent spaces, C becomes a Riemannian manifold. Given two curves q 1 and q 2 , let ψ denotes a path on the manifold C between q 1 and q 2 , ψ ∈ T ψ (C) is a tangent vector field on the curve ψ ∈ C and < . > denotes the L 2 inner product on the tangent space. In our case, as the elements of C have a unit L 2 norm, C is a Hypersphere in the Hilbert space L 2 (I, R 3 ). The geodesic path between any two points q 1 , q 2 ∈ C is simply given by the minor arc of great circle connecting them on this Hypersphere, ψ * : [0, 1] → C, given by Eq. ( 1):

ψ * (τ ) = 1 sin(θ) (sin((1 -τ )θ)q 1 + sin(θτ )q 2 ) (1)
and θ = d C (q 1 , q 2 ) = cos -1 ( q 1 , q 2 ). We point out that sin(θ) = 0 if the distance between the two curves is null, in other words q 1 = q 2 . In this case, for each τ , ψ * (τ ) = q 1 = q 2 . The tangent vector field on this geodesic dψ * dτ : [0, 1] → T ψ (C) is then given by Eq. ( 2):

dψ * dτ = -θ sin(θ) (cos((1 -τ )θ)q 1 -cos(θτ )q 2 ) (2)
Knowing that on geodesic path, the covariant derivative of its tangent vector field is equal to 0, dψ * dτ is parallel along the geodesic ψ * and we shall represent it with dψ * dτ | τ =0 . Accordingly, Eq. (2) becomes:

dψ * dτ | τ =0 = θ sin(θ) (q 2 -cos(θ)q 1 ) , (3)
with θ = 0. Thus, dψ * dτ | τ =0 is sufficient to represent this vector field, the remaining vectors can be obtained by parallel transport of dψ * dτ | τ =0 along the geodesic ψ * . In practice, the first step to capture the deformation between two given 3D faces S 1 and S 2 is to extract the radial curves. Let β 1 α and β 2 α denote the radial curves that make an angle α with a reference radial curve on faces S 1 and S 2 , respectively. The reference curve is chosen to be the vertical curve as the faces have been rotated to the upright position during the preprocessing step. The tangent vector field ψα * that represents the energy E given in Eq. ( 1) needed to deform β 1 α to β 2 α is then calculated for each index α. We consider the magnitude of this vector field at each point, located in β α and is of index k on this curve, for building a Deformation Vector Field on the facial surface,

V k α = || ψ * α | (τ =0) (k)||
, where α denotes the angle to the vertical radial curve and k denotes a point on this curve. This scalar vector field quantifies the local (on each point) deformation between the faces S 1 and S 2 .

Dynamic Shape Deformation Analysis

To capture the dynamic of facial deformations across 3D face sequences, we consider the deformation map computed between successive frames, as described in section 2.1. Similarly to the recognition schema proposed by Sun et al. [START_REF] Sun | Facial expression recognition based on 3d dynamic range model sequences[END_REF], in order to make possible to come to the recognition system at any time and make the recognition process possible from any frame of a given video, we consider subsequences of n frames. Thus, we chose the first n frames as the first subsequence. Then, we chose n-consecutive frames starting from the second frame as the second subsequence. The process is repeated by shifting the starting index of the sequence every one frame till the end of the sequence. For each sub-sequence, the first frame is considered as reference one and the deformation map is calculated to each of the remaining frames. The feature vector for this subsequence is built based on the average deformation of the n -1 calculated deformation maps. Thus, each subsequence is represented by a feature vector of size the number of points on the face. We point out that we consider 5000 points, 50 for each of the 100 radial curves.

Figure 1 illustrates one subsequence for each expression with n = 6 frames. Each expression is illustrated in two rows, the upper row gives the reference frame of the subsequence and the n -1 successive frames of subsequences. Below are the corresponding deformation maps computed for each frame. The mean deformation map is reported at the right and represent the feature vector for that subsequence. Thus, this deformation map summarizes the temporal deformation undergone by the facial surface when conveying expressions.

Classification

We now represent each subsequence by its Deformation Vector Filed V k α = || ψ * α | (τ =0) (k)||, as described in section 2. Since, the dimensionality of the feature vector is high, we use LDA-based transformation (Linear Discriminant Analysis) to transform the present feature space to an optimal one that is relatively insensitive to different subjects while preserving the discriminative expression information. LDA defines the withinclass matrix S w and the between-class matrix S b . It transforms a n-dimensional feature to an optimized ddimensional feature where d < n. For our experiments, the discriminative classes are 6 expressions, thus the reduced dimension d is 5. For the classification task we used the Multi-class version of Random Forest algorithm. The Random Forest algorithm was proposed by Leo Breiman in [START_REF] Breiman | Random forests[END_REF] and defined as a meta-learner comprised of many individual trees. It was designed to oper- ate quickly over large datasets and more importantly to be diverse by using random samples to build each tree in the forest. Diversity is obtained by randomly choosing attributes at each node of the tree and then using the attribute that provides the highest level of learning.

Experiments and Discussions

To demonstrate the effectiveness of the proposed approach, we perform expression recognition from 3D face sequences following the experimental setting described in [START_REF] Sun | Facial expression recognition based on 3d dynamic range model sequences[END_REF]. The experiments were conducted on a subset of 60 subjects (arbitrarily selected) from the BU-4DFE dataset and includes the 6 prototypic expressions (Angry (An), Disgust (Di), Fear (Fe), Happy (Ha), Sad (Sa), Surprise (Su)). We note that the experiments are conducted in a identity-independent fashion. Following a similar setup as in [START_REF] Sun | Facial expression recognition based on 3d dynamic range model sequences[END_REF], we randomly divided the feature vectors (computed from the subsequences) of 60 subjects into two sets, the training set containing 54 subjects, and the test set containing 6 subjects. We perform on these feature vectors Random Forest algorithm and reported obtained results in Table 1. We note that the reported rates are obtained by averaging the results of the 10-independent and arbitrarily run experiments (10-fold cross validation). The average recognition rate is equal to 93.21%. It can also be observed that the best classified expressions are (Ha) and (Su) with recognition accuracies of 95.47% and 94.53%, respectively, whereas the (Fe) expression is more difficult to classify. This is mainly due to the subtle changes in facial shapes motion of this expression compared to (Ha) and (Su). We note that the proposed approach outperforms state-of-the-art approaches following the same experimental settings. The recognition rates reported in [START_REF] Sun | Facial expression recognition based on 3d dynamic range model sequences[END_REF] based on temporal analysis only was 80.04% and spatio-temporal analysis was 90.44%. In both studies subsequences of constant window width equal to 6 (W in = 6) is defined for experiments. We emphasize that their approach is not completely automatic requiring 83 manually marked key points on the first frame of the sequence to allow accurate model tracking. We can also make comparisons with performances showed in [START_REF] Sandbach | A dynamic approach to the recognition of 3d facial expressions and their temporal models[END_REF] on only three expressions (Ha, An, and Su), their approach achieved an average recognition rate about 73.61% (for each frame) compared to our approach which showed an increase of more than 19% by achieving (93.21%). We note also that the subjects considered on our study are arbitrary selected whereas in [START_REF] Sandbach | A dynamic approach to the recognition of 3d facial expressions and their temporal models[END_REF] sequences are accurately selected. Approaches like [START_REF] Le | Expression recognition from 3d dynamic faces using robust spatio-temporal shape features[END_REF] and [START_REF] Sandbach | A dynamic approach to the recognition of 3d facial expressions and their temporal models[END_REF] reported recognition results on whole facial sequences, this hinders the possibility of the methods to adhere to a real-time protocol. In fact, showing recognition results depends on the preprocessing of whole sequences unlike our approach and the one described in [START_REF] Sun | Facial expression recognition based on 3d dynamic range model sequences[END_REF] which are able to provide recognition results when processing very few 3D frames.

Conclusion

This paper proposes a new Deformation Vector Field (DVF) which accurately describes local deformations across 3D facial sequences. A facial surface parametrization by their radial curves allows the definition of this descriptor on each facial point based on Riemannian Geometry. Then the well known learning algorithm, Random Forest, is performed for the classification task. Experiments conducted on BU-4DFE dataset, following state-of-the-art setting demonstrate the effectiveness of the proposed approach based only on temporal analysis of facial sequences.

Figure 1 .

 1 Figure 1. Calculus of dynamic shape deformation on subsequences taken from the BU-4DFE sequences.

  Once trained, Random Forest classify a new expression from an input feature vector by putting it down each of the trees in the forest. Each tree gives a classification decision by voting for that class. Then, the forest chooses the classification having the most votes (over all the trees in the forest). In our experiments we used Weka Multi-class implementation of Random Forest al-gorithm by considering 40 trees.

Table 1 . Confusion matrix.

 1

	%	An	Di	Fe	Ha	Sa	Su
	An	93.11	2.42	1.71	0.46	1.61	0.66
	Di	2.3	92.46	2.44	0.92	1.27	0.58
	Fe	1.89	1.75	91.24	1.5	1.76	1.83
	Ha	0.57	0.84	1.71	95.47	0.77	0.62
	Sa	1.7	1.52	2.01	1.09	92.46	1.19
	Su	0.71	0.85	1.84	0.72	1.33	94.

53 Average recognition rate = 93.21%


Ben Soltana Wael

Mohsen Ardabilian

Pierre Lemaire

Di Huang

Szeptycki Przemyslaw

Boulbaba Liming Chen

Ben Amor

Drira Hassen

Daoudi Mohamed

Nesli Erdogmus

Wael Ben Soltana

Huang Di

Przemyslaw Szeptycki

Chen Liming

Liris

Lionel Daniel

Jean-Luc Dugelay

Boulbaba Ben Amor

Hassen Drira

Mohamed Daoudi

3D face recognition: A robust multi-matcher approach to data degradations

come   L'archive ouverte pluridisciplinaire

Introduction

3D Face analysis has been an important challenge in computer vision and pattern recognition over the last decades. While humans can recognize faces easily even in degraded conditions, automatic face recognition remains altered by unconstrained environments. Most work in 3D face recognition deals with variations in facial expressions, few others investigates the robustness to quality degradations of 3D face models. Those degradations may have several origins. Firstly, acquisitions conditions can be seen as a source of degradations such as illumination, motion in front of 3D sensor, missing data due to self-occlusions or structured-light absorption, and distance to the 3D scanner. Also, compression, useful when the storage capacity matters, as well as sampling (resolution reduction of the 3D model) can degrade 3D models. A short study of those model degradations was conducted in [START_REF] Faltemier | Flexible and Robust 3D Face Recognition Dissertation[END_REF] and [START_REF] Kakadiaris | Three-dimensional face recognition in the presence of facial expressions: an annotated deformable model approach[END_REF] on the FRGC dataset [START_REF] Phillips | Overview of the face recognition grand challenge[END_REF].

Analyzing the behavior of recognition algorithms in the presence of such degradations is a necessary step before their application in a real environment. Most works in the biometry field recognize the impact of the quality of acquisition data on the performance of algorithms [START_REF] Poh | Benchmarking Quality-dependent and Cost-sensitive Score-level Multimodal Biometric Fusion Algorithms[END_REF]. More specifically, 3D face recognition algorithms reckon with this issue, in that they apply some pre-processing techniques [START_REF] Faltemier | Flexible and Robust 3D Face Recognition Dissertation[END_REF]. For instance, median filtering for removing spikes and interpolation for filling holes have been used. The issue of occlusion was studied in [START_REF] Colombo | Detection and Restoration of Occlusions for 3D Face Recognition[END_REF] and [START_REF] Bronstein | Robust expression-invariant face recognition from partially missing data[END_REF]. In [START_REF] Bronstein | Robust expression-invariant face recognition from partially missing data[END_REF], Bronstein et al. studied the robustness of their 3D face matching approach to occlusion. For this purpose, they generated mild and severe missing data on 3D face. They showed that their recognition accuracy was not affected, and that their verification rate was only slightly affected. However, the test set was limited to 30 persons. In [START_REF] Rodrigues | Robustness of multimodal biometric fusion methods against spoof attacks[END_REF], Rodrigues et al. also simulated missing data on the face by manually excluding some regions of interest. To overcome this issue, they proposed to match local regions of faces together and conclude that the better results were obtained by fusing 28 over the 38 regions.

To our knowledge, there are yet no specific studies on the behavior of 3D face recognition algorithms under 3D face models degradations. In this paper, we study the behavior of four state of the art algorithms for 3D face recognition, subsequently called individual experts, under several frequent quality damages such as noise, decimation and holes. For a more comprehensive study of the impact of degradations, we chose experts that all make use of different properties of 3D face models and exemplify the major classes of 3D face recognition solutions currently proposed in the literature. As these experts capture different geometric properties, we further propose a multi-matcher scheme, which aims at enhancing the individual robustness of experts against listed degradations. This paper is organized as follows. The multi-matching technique for combining similarity measures is described in section 2. The different experts used in this work are introduced in section 3. Section 4 analyzes and discusses experimental results on the FRGC database. Section 5 concludes the paper.

Fusing individual similarity measures for a multi-matcher

Each individual expert generates similarity scores. The role of a fusion step is to weight individual experts between them to produce a stronger expert. The fusion operates on two levels: at first, it must preserve individual expert strengths. Secondly, it must promote the complementarity between them. In this work, we adopt a competitive fusion approach and propose an adaptive score level fusion scheme using a weighted sum rule. The weight associated with each expert is set after an offline and an online weight learning steps. Both steps automatically define the most relevant weights of all scores for each probe matched to the whole gallery set. The basic idea is that more weight is allocated to an expert which performs better according to both its global behavior on a learning dataset (offline learning, see eq. ( 1)) and its actual order of similarity scores (online ranking, see eq. ( 2)). The global behavior of an expert on a learning dataset can be evaluated according to Equal Error Rates (EER), Verification Rate (VR) and Recognition Rate (RR). Both types of weights (offline and online) are further combined to generate a final weight (eq. ( 3)).

Before the fusion step, scores achieved by different experts are first normalized into a common scale. We use a Min-Max normalization [START_REF] Mian | Face recognition using 2d and 3d multimodal local features[END_REF], which maps the matching scores to the range of [0, 1] linearly. During the offline step, we use the approach proposed in [START_REF] Snelick | Large-scale evaluation of multimodal biometric authentication using state-of-the-art systems[END_REF] to assign a weight to scores of a given expert. Let the performance indicator of the m th expert be e m , m = 1, 2, …, M. The corresponding weight P m associated to the scores produced by the m th expert is calculated as:

P ,u ∑ e , ∑ P , P (1)
For instance, when using EER as performance indicator of an expert, its corresponding weight P m is inversely proportional to its corresponding error ( P , and u ∑ ), thereby giving more importance to experts having small EER.

During the online step, each expert f produces a similarity score S g,f between each gallery face and the probe face. All of these similarities S g,f are then sorted in a descending order. We assign to each score S g,f a weight w g,f which is a function of its ordered position p g,f . Specifically, the weight w g,f is defined as:

w , f p ln , (2)
where N g is the number of subjects in the gallery.

This online weighting strategy gives more importance to better ranked scores and aims at discarding matching scores far from the best ones. The final matching score between a face g in the gallery and the probe face takes into account both online and offline matching scores. It is defined by:

S f g ∑ P w , S , (3)
The probe face is recognized as the one in the gallery which obtains the highest final score according to [START_REF] Phillips | Overview of the face recognition grand challenge[END_REF]. As the score ranking p g,f is used in the weighting scheme as in eq. ( 3) through eq. ( 2), this fusion scheme only works for identification scenario (one probe versus N galleries).

Experts

In this section, we present the four experts used in this paper, both for analyzing the impact of 3D model degradation on the 3D face recognition performance, and for illustrating the contribution of our fusion scheme. Those experts were chosen because they all make use of different 3D face properties to perform the 3D face recognition. Specifically, the first expert (E1, namely elastic shape analysis of radial curves), presented in section 3.1, is a hybrid approach. It is neither completely holistic and neither totally local, as it samples a set of radial curves from a facial surface, and measures the geodesic distances computed over each pair of corresponding radial curves. The second expert (E2, namely MS-ELBP + SIFT), presented in section 3.2, is a local feature-based approach which makes use of an extended LBP (ELBP) and SIFT-based matching. Finally, the last two experts, namely E3 and E4, are both holistic approaches. The third expert (E3, namely TPS warping parameters), presented in section 3.3, considers non rigid facial surface matching through Thin-Plate-Spline (TPS) whereas the fourth expert (E4, namely ICP) makes rigid facial surface matching through ICP [START_REF] Besl | A Method for Registration of 3-D Shapes[END_REF].

Hence, we believe that the 4 chosen experts use different geometric properties of 3D faces for the resolution of the same problem. Comparative results will be provided in section 4.

3D face matching algorithm based on elastic shape analysis of radial curves

The first expert performs 3D face matching based on elastic shape analysis of radial curves [START_REF] Drira | Pose and expression-invariant 3d face recognition using elastic radial curves[END_REF]. As illustrated in figure 1, probe and gallery facial surfaces are first aligned using ICP algorithm; radial curves are then extracted from both surfaces; the corresponding probe and gallery curves are then compared within a Riemannian framework, and finally individual scores are fused to produce a general one which represent the degree of similarity between probe and gallery facial surfaces. Within the proposed approach, the authors introduce a quality inspection filter that examines all the extracted radial curves in both gallery and probe models and retains valid ones based on the defined criterion.

Radial curves extraction

The reference curve is chosen to be the vertical curve once the face has been set into the upright position. Each radial curve β α is obtained by slicing the facial surface by a plane P α with the nose tip as its origin, and which makes an angle α with the vertical plane. That is, the intersection of P α with S gives the radial curve β α . This step is repeated to extract radial curves, indexed by the angle α, from the facial surface at equal angular separation. Figure 2 shows examples of some radial curves extraction under different quality degradations.

Radial curves matching framework

The Square-root velocity function is a specific mathematical representation (SRVF), denoted by q(t), used by the authors to analyze the shape of radial curves. It is denoted according to: q t (4) It has been shown in [START_REF] Wang | 3D surface matching and recognition using conformal geometry[END_REF] that the classical elastic metric for comparing shapes of curves becomes the L2-metric under the SRVF representation. The collection of normed SRVF defines the pre-shape space set:

C q: | q I, (5)
The geodesic length between any two points q 1 , q 2 Є C is given by:

d q ,q cos q ,q (6)
The framework allows one to compute geodesic paths denoting optimal deformations between individual curves. Therefore, these deformations are combined to obtain full deformations between faces. The missing data problem is addressed by a quality module that inspects the quality of curves, and discards them if needed. Authors report that, by elastically matching radial curves, deformations due to expression are tackled. Thus, this method is a hybrid approach in between a holistic one such as ICP and a SIFT local feature-based one such as the second expert, introduced in the following subsection.

MS-ELBP + SIFT

The second expert is based on Multi-Scale extended LBP (MS-ELBP) along with local feature SIFT-based matching as detailed in [START_REF] Huang | A Novel Geometric Facial Representation based on Multi-Scale Extended Local Binary Patterns[END_REF]. This expert first generates several 3D facial representations (MS-ELBP) which are followed by a local feature SIFT-based matching and score fusion. It does not require any registration for roughly frontal faces as the ones in the FRGC datasets.

3.2.1

LBP and its descriptive power of local shape variations LBP, a non-parametric algorithm [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF], was first proposed to describe local texture of 2D images. It has been extensively adopted for 2D face recognition in the last several years [START_REF] Ahonen | Face recognition with local binary patterns[END_REF]. However, computationally simple and direct application of LBP on depth images results in unexpected confusion to similar but different local shapes. To address this problem, two complementary solutions were considered. The first one aims at improving the discriminative ability of LBP with Extended LBP coding approach, and the other one focuses on providing a more comprehensive geometric description of the neighborhood by exploiting a Multi-Scale strategy.

Extended Local Binary Patterns

Instead of LBP, ELBP not only extracts relative gray value difference between the central pixel and its neighboring pixels provided by LBP, but also focuses on their absolute difference. Specifically, the ELBP code consists of several LBP codes at multiple layers which encode the exact gray value difference (GD) between the central pixel and its neighboring pixels. The first layer of ELBP is actually the original LBP code encoding the sign of GD. The following layers of ELBP then encode the absolute value of GD. Basically, each absolute GD value is first encoded in its binary representation and then all the binary values at a given layer result in an additional local binary pattern. As a result, when describing similar local shapes, although the first layer LBP is not discriminative enough, the information encoded in the other additional layers can be used to distinguish them. Experimentally, the number of additional layers was set to 4.

Multi-Scale Strategy

The ELBP operator is further extended with different sizes of local neighborhood to handle various scales. The local neighborhood is defined as a set of P sampling points evenly spaced on a circle of radius R that is centered at the pixel to be labeled. For each (P, R) couple and each layer of the ELBP approach, a face is generated regarding the corresponding decimal number of the LBP binary code as the intensity value of each pixel. Such an image is called a MS-ELBP Depth Face (DF). Hence, DFs contain many details of local shapes (Figure 4).

SIFT based local feature matching

Once the MS-ELBP-DFs are computed, the widely-used SIFT based features [START_REF] Lowe | Distinctive image features from scale-invariant key-points[END_REF] are extracted from each DF separately for similarity score calculation and final decision. Because MS-ELBP-DFs highlight local shape characteristics of smooth range images, many more SIFT-based keypoints are detected than those in the original range images. In [START_REF] Huang | A Novel Geometric Facial Representation based on Multi-Scale Extended Local Binary Patterns[END_REF] it is reported that the average number of descriptors extracted from each of DFs is 553, while that of each range image is limited to 41.

Given the features extracted from each MS-ELBP-DF pair of the gallery and probe face scan respectively, two facial keypoint sets can be matched [START_REF] Lowe | Distinctive image features from scale-invariant key-points[END_REF]. Here, NL i (P, R) denotes the number of the matched keypoints in the i th layer of ELBP-DF pair, with a parameter setting of (P, R).

The similarity measure, NL i (P, R), is with a positive polarity (a bigger value means a better matching relationship). A face in the probe set is matched with every face in the gallery, resulting in an n-sized vector. Each of these vectors is then normalized to the interval of [0, 1] using the min-max rule. Matching scores of all scales are fused using a basic weighted sum rule:

S ∑ w P, R N P, R (7)
The corresponding weight, w Li(P,R) , is calculated dynamically during the online step using the scheme in [START_REF] Mian | Keypoint detection and local feature matching for textured 3D face recognition[END_REF]:

w P, R L P, R L P, R L P, R L P, R (8)
where the operators max 1 (V) and max 2 (V) produce the first and second maximum values of the vector V. The gallery face image which has the maximum value in S is declared as the identity of the probe face image. Extended results and analysis are provided in [START_REF] Huang | A Novel Geometric Facial Representation based on Multi-Scale Extended Local Binary Patterns[END_REF]. As we can see, this method is a local feature-based approach which should make it more tolerant to occlusion and missing data.

Recognition via TPS Warping Parameters

The third expert fits a generic model to facial scans in order to extract warping parameters to be used as biometric signatures. Face recognition based on morphable models has been extensively studied in the literature. A flexible model of an object class, which is a linear combination of the example shapes and textures, was introduced in [START_REF] Jones | Model-based matching of line drawings by linear combinations of prototypes[END_REF]. Its extension to 3D was proposed in [START_REF] Blanz | Face recognition based on fitting a 3D morphable model[END_REF] and proved to give promising results. This expert makes use of a generic head model which is strongly deformed to fit facial models in the gallery, using the Thin Plate Spline (TPS) algorithm. Here, the aim is to utilize the discriminative properties of the warping parameters obtained during the fitting process.

TPS method was made popular by Fred L. Bookstein in 1989 in the context of biomedical image analysis [START_REF] Bookstein | Principal warps: Thin-Plate Splines and Decomposition of Deformations[END_REF]. For the 3D surfaces S and T, and a set of corresponding points on each surface, P i and M i respectively, the TPS algorithm computes an interpolation function f(x, y) to compute T', which approximates T by warping S: When a generic model is deformed to fit an individual face in the database, an approximate representation of the facial surface is obtained. The deformation parameters represent the deviation from the generic face, and are therefore claimed to possess dense information about the facial shape.

Before TPS warping, a linear transformation is computed in a least square sense, based on 15 landmark points on both generic and inspected (target face) models. The obtained transformation which includes rotation, translation and isotropic scaling is applied onto the generic model. After that, in addition to the 15 point pairs utilized for alignment, 136 more pairs are generated, by coupling a set of points on the generic model with their closest neighbor on the target face. Using 151 point pairs in total, TPS interpolation is computed for the generic model. Given in [START_REF] Wang | 3D surface matching and recognition using conformal geometry[END_REF], the function f(x,y) includes the warping coefficients: w i , i={1,2,...n} to be utilized. When we transpose the formula for the other two directions, following feature vector is obtained: [(w 1x ,w 1y ,w 1z ), (w 2x ,w 2y ,w 2z ) …(w 151x ,w 151y ,w 151z )]. The whole scheme is summarized in Figure 5, with an illustration on a sample model.

Once the feature vectors are extracted, Cosine and Euclidean distances are calculated between two warping vectors, resulting in two distance vectors of size 151x1. In order to measure the central tendency of these vectors, trimmed mean approach is adopted and hence, sensitivity to outliers is avoided. For trimmed mean method, the portion to be ignored is taken as 10%.

ICP algorithm

In section 4, we also provide results of the baseline algorithm ICP as a comparison. Iterative Closest Point (ICP) algorithm [START_REF] Besl | A Method for Registration of 3-D Shapes[END_REF] is a well-known reference algorithm in matching 3D point clouds through rigid transforms (translation, rotation). It is clearly a holistic matching algorithm in considering only rigid transforms. In our experiments, an automatic detection of the nose was applied, and the face was cropped with a sphere of radius 100 mm centered on the nose tip.

Experiments

As stated in the introduction, 3D models can suffer from various types of degradations having different origins. However, it is not easy to acquire a large dataset representing significantly these different degradations. A fundamental issue here is the quantification of degradations on real data. In this work, we decided to generate degraded data from the FRGC v2.0 dataset which is one of the most comprehensive dataset known so far in the literature. That choice allows us to have a large set of 3D face models under parameterized degradations, thereby testing the behavior of the experts and their fusion towards them. In this section, we first define the evaluation protocol then discuss the experimental results.

Experimental setup

The kinds of degradations that we consider are canonical ones as they typically occur in the acquisition process. They are Gaussian noise on depth, decimation in terms of resolution and holes for missing data. 410 subjects having more than one 3D face models were selected from the FRGC v2.0 database [START_REF] Phillips | Overview of the face recognition grand challenge[END_REF]. For each subject, one model with neutral facial expression was randomly picked to make up the gallery. We also randomly picked another model for each subject, which is used as a probe model. Before we could apply artificial, controlled degradations, we need to ensure that original 3D models are as clean as possible. Hence, gallery and probe sets were first preprocessed to remove spikes and holes as in [START_REF] Phillips | Overview of the face recognition grand challenge[END_REF]. Facial regions were further cropped from these gallery and probe sets based on the nose-tip within a sphere of diameter 100mm. For this purpose, nose-tips were manually located on every face.

From here, each cropped probe face model was then altered to some extent to create new, degraded sets, according to the following degradations: -Gaussian noise corresponds to the injection of an error within a Gaussian distribution on the Z coordinates on the depth image. This tends to emulate the behavior of electronic noise of acquisition devices, albeit a simplistic manner. In our experiments, we set the RMS value of the error respectively to 0.2, 0.4 and 0.8 (mm). -Decimation corresponds to removing vertices from the original data. In this experiment, vertices are picked randomly and removed respectively from a ratio of x2, x4 and x8. -Holes are generated at random locations on the face. At first, we pick a random vertex on the surface of the face. Then, we crop the hole according to a 1 centimeter radius sphere centered on the latter vertex. For each level, we generate respectively 1, 2, 3 holes on the whole face. Figure 6 shows examples of those degradations. In the subsequent, these three types of degradation are respectively denoted decimation (D), missing data (MD) and noise (N) having each three levels (2, 4 and 8). Each individual expert was then benchmarked on the subset of 410 subjects without degradations then using the previously defined degradations. As we will see in the following subsection, every individual expert displays different behavior in terms of performance drop against these degradations. Then, we further propose the fusion of these experts at score level, using the fusion scheme as described in section 2, thereby giving birth to several multi-matchers. The associated experimental protocol, as well as the performance of those multi-matchers, is detailed in subsection 4.2.2.

Results and Analysis

The performances of the four individual experts over the degradations are first analyzed. Then, our proposed fusion scheme (section 2) is compared to the standard sum rule fusion method [START_REF] Ross | Handbook of Multibiometrics[END_REF].

4.2.1

Performances of the individual experts Recall that the four individual experts are: Expert using Elastic Shape Matching (E1), Expert based on MS-ELBP (E2), Expert using TPS (E3), ICP (E4). The 4 algorithms were benchmarked on the subset of 410 subjects as defined previously in subsection 4.1 and the various generated degradations. Figures 789show the rank-one recognition rates of the four experts, respectively under noise (N), decimation (D) and missing data (MD). As we can notice, all the algorithms record to some extent performance drops under degradations. While all the four algorithms resist relatively well to decimation and missing data, their performance drops drastically when the noise is increased to some level, 0.8 mm RMS in this work.

These results also suggest that the local feature-based expert, namely E2, displays the best robustness against the noise and decimation types of degradation. E1, a hybrid method, also displays a good behavior against decimation. Quite interestingly, the holistic approaches (E3 and E4) seem to behave better than other approaches against missing data. Figure 9 shows that holistic approaches, E3 and E4, display quite stable performance in the presence of MD.

4.2.2

Performances of multi-matchers As all the three experts make use of different properties of facial surfaces, we further propose to study several multi-matchers, namely (1) E1+E2, (2) E1+E3, (3) E2+E3 and (4) E1+E2+E3, using the fusion scheme proposed in section 2. The 4 th expert, based on ICP was considered as baseline. Since a learning dataset is needed to set the offline weights based on some global performance indicator of each expert, the whole dataset of 410 subjects was randomly divided into two equal parts, one part for learning and the other one for testing. This experimental setup was then cross-validated 50 times using each time different learning and testing data, and the performance averaged over these 50 cross-validations.

All the four multi-matchers were benchmarked on the degradations, namely decimation (D), missing data (MD) and noise (N) with the three levels (2, 4 and 8).

We experimented with three performance indicators, namely rank-one recognition rate (RR), verification rate at 0.1% FAR (VR) and Equal Error Rate (EER). The best performance being achieved using RR in the fusion scheme, we only report in fig. 10 Fig. 12 depicts the behavior of the four multi-matchers under different levels of missing data compared to the three original individual experts. Once again, all the four multi-matchers display better and stable performance than the three individual experts in almost all cases, the best performance being still achieved by E1+E2+E3 with 97.39%, 96.56% and 96.34% recognition rate respectively under missing data level 2, 4 and 8. As we can see, missing data only affects slightly the overall performance of the multi-matcher E1+E2+E3.

From Table 1, we can notice some performance improvements for the E1+E2+E3 multi-matcher (MM), compared to the standard sum rule (SS) and max rule (Max) fusion methods, for all the degradations at different levels. This proves that the proposed multi-matcher preserves the complementarity between all the experts. For the same experiment, the ICP algorithm seems to be more affected by decimation and Gaussian noise.

While the performances of our multi matcher are affected by degradations, this experiment shows that, within a recognition scenario, it is more robust than any of the engaged individual experts to any type of degradation.

Conclusion and Perspectives

In this paper, we discussed the robustness of four individual experts and their fusion with respect to three sample quality degradation scenarios, namely noise, decimation and missing data. The experimental results demonstrate that: (1) All the individual experts suffer somehow from sample quality degradations and display different behavior depending upon their intrinsic properties; (2) Globally, the performances are more affected by the presence of noise than by the presence of holes or decimation. (3) The multi-matchers using the proposed fusion scheme almost all perform better than individual experts under different degradation scenarios, thereby suggesting that the fusion scheme capitalizes on the individual expert strengths. Furthermore, the decrease in the performances due to quality degradations of the multi-matchers is less important than the decrease of each individual expert.

As a future development work, we would like to study the behavior of our multi matcher scheme against degradations within a verification scenario. This proposition suggests that the weight computation model has to be slightly modified.

Figure 1 :

 1 Figure 1: Block diagram of the studied 3D face rec. method

Figure 2 :

 2 Figure 2: Examples of 3D faces with radial curves under different quality degradation

Figure 3 :

 3 Figure 3: Examples of geodesic paths between source and target Shown in Figure 3 is an example of such geodesic path between source and target faces belonging to the same person under different expressions.The missing data problem is addressed by a quality module that inspects the quality of curves, and discards them if needed. Authors report that, by elastically matching radial curves, deformations due to expression are tackled. Thus, this method is a hybrid approach in between a holistic one such as ICP and a SIFT local feature-based one such as the second expert, introduced in the following subsection.

Figure 4 :

 4 Figure 4: MS-ELBP-DFs of a range face image with different radii from 1 to 8 (from left to right).

Figure 5 :

 5 Figure 5: The proposed feature extraction scheme and an illustration on a sample model: (a) Target face with and without texture (b) generic model before (with landmarks) and after alignment (c) generic model after warping with and without texture

Figure 6 :

 6 Figure 6: An example of degradations applied to one model. From left to right: the Original face, with Noise, Decimation, and Holes

Figure 7 :Figure 8 :

 78 Figure 7: The rank one rec. rates under noise

Figure 9 :

 9 Figure 9: The rank one rec. rates under missing data

  , fig.11 and fig.12 the performance achieved by the four multi-matchers compared to the individual experts E1, E2 and E3 under different degradation scenarios and levels.

Figure 10 :

 10 Figure 10: Rank-one Rec. Rate of the multi-matchers compared to the individual experts under different levels of noise As we can see from fig.10, all the four multi-matchers display better performance under different level of noise, except once for E1+E3 on N(4), the best and stable performance being achieved by E1+E2+E3 which gives 99.77% recognition rate on the original data, 97.01%, 96.68% and 91.13% respectively under the three different noise level. This suggests that the fusion scheme has really capitalized on the complementary properties of individual experts. Notice also that E2+E3, which is the fusion of a local feature-based matcher (E2) with a holistic matcher through TPS (E3), outperforms E1+E2 and E1+E3 which are combinations of a hybrid method (E1) along with a local feature-based expert (E2) and a holistic expert (E3), respectively. Fig.11 and Fig.12 further confirm the previous trend. Fig.11 gives the behavior of the four multi-matchers under different levels of decimation compared to the three

Figure 11 :Figure 12 :

 1112 Figure 11: Rank-one Rec. Rate of the multi-matchers compared to the individual experts under different levels of decimation

Table 1 .

 1 Comparison between Rank-one Recognition Rate (%) of standard sum (SS) rule fusion method , max rule (Max) and our multimatcher (MM) for the E1+E2+E3 (4)

		SS	Max	MM	SS	Max	MM
	N(2)	95.25 94.91 97.01 D(2) 95.60 95.22 98.31
	N(4)	94.60 93.40 96.68 D(4) 94.50 94.37 97.68
	N(8)	88.80 85.56 91.13 D(8) 94.20 93.73 96.60
		SS	Max	MM
	MD(2

) 95.40 95.07 97.39 MD(4) 94.83 94.18 96.56 MD(8) 94.50 94.41 96.34


Dany Obeid

Member, IEEE Gheorghe Zaharia

Sawsan Sadek

Ghaïs El

G El Zein

Microwave Doppler Radar for Heartbeat Detection Vs Electrocardiogram

Keywords: Doppler effect, heart rate variability, remote monitoring, VNA

come

INTRODUCTION

HE utility of microwave Doppler radar has increased for several applications. These include home health care applications, urgent conditions, and hospital needs. Affixed electrodes for traditional electrocardiograms are perturbing for long-duration home monitoring, as well as for patients with conditions such as burn victims or newly born infants. In addition, a touchless technique is necessary to detect life signs for people under rubble [START_REF] Chuang | Automatic cluttercanceller for microwave life-detection systems[END_REF]. Based on the Doppler theory, a target with a quasi-periodic movement reflects the transmitted signal with its phase modulated by the time-varying position of the target [START_REF] Lin | Microwave sensing of physiological movement and volume change: A review[END_REF]. When the target is the person's chest, the reflected signal contains information about the chest displacement due to heartbeat and respiration. However, while holding breath, the reflected signal depends on the chest displacement due to heartbeat alone. At rest, the variation of the chest displacement, caused by respiration, is between 4 and 12 mm [START_REF] De Groote | Chest wall motion during tidal breathing[END_REF], and the chest displacement due to heartbeat alone ranges between 0.2 and 0.5 mm [START_REF] Ramachandran | Three-dimensional reconstruction of cardiac displacement patterns on the chest wall during the P, QRS, and T-segments of the ECG by laser speckle interferometry[END_REF]. The respiration rate corresponds to a frequency that varies between 0.1 and 0.3 Hz, while the heartbeat rate (HR) corresponds to a frequency that varies between 0.83 and 2 Hz [START_REF] Mohammad-Zahed | Contactless Heart Monitoring[END_REF]. Previous works tend to detect life signs [START_REF] Chen | Microwave lifedetection systems for searching human subjects under earthquake rubble and behind barrier[END_REF][START_REF] Droitcour | A microwave radio for Doppler radar sensing of vital signs[END_REF], respiration rates and heartbeat rates, using fixed frequency and fixed power of the transmitted signal. Direct-conversion Doppler radars, operating at 1.6 GHz and 2.4 GHz, have been integrated in 0.25 μm CMOS and BiCMOS technologies [START_REF] Droitcour | 0.25 μm CMOS and BiCMOS single chip direct conversion Doppler radars for remote sensing of vital signs[END_REF]. Heart and respiration activities were detected using a modified Wireless Local Area Network PCMCIA card, and

The authors acknowledge CEDRE (project reference number: 08SciF6/L4), AUF (project reference number: G3-110/1649), and "Région Bretagne" for partially funding this work. a module combining the transmitted and reflected signals [START_REF] Boric-Lubecke | Wireless LAN PC Card Sensing of Vital Signs[END_REF]. Other systems operating in the Ka-Band were described in [START_REF] Xiao | A Ka-band low power Doppler radar system for remote detection of cardiopulmonary motion[END_REF] using a low power double-sideband transmission signal. Similarly, quadrature demodulation architecture is used to detect vital signs from distance [START_REF] Gu | Instrument-Based Noncontact Doppler Radar Vital Sign Detection System Using Heterodyne Digital Quadrature Demodulation Architecture[END_REF]. A new study showing the possibility of detecting the presence of a person through a wall using Ultra-Wideband (UWB) radar is presented in [START_REF] Lubecke | Through-the-wall radar life detection and monitoring[END_REF]. Some experiments are performed for the detection of life signs using the 4 -7 GHz band with 1 mW power and around 7 dB antenna gain [START_REF] Li | Random body movement cancellation in doppler radar vital sign detection[END_REF]. A recent study describes a system operating at 94 GHz that is able to detect the heart and the respiration activities of stationary subjects [START_REF] Mikhelson | Remote sensing of heart rate and patterns of respiration on a stationary subject using 94-GHz millimeter-wave interferometry[END_REF]. Another related published work showed the ability of measuring the heartbeat rate using a Vector Network Analyzer (VNA) and only one antenna [START_REF] Serra | A single on-body as a sensor for cardiopulmonary monitoring[END_REF]. In this paper, a microwave system for heartbeat detection is proposed. The system is based on using a VNA and one (or two) antenna. The aim of this system is to provide a comparative approach in terms of the operational frequency and the transmitted power. This helps identifying the optimum frequency and signal processing technique, for the minimal transmitted power. This study is necessary prior the implementation process. The proposed system, tested for one frequency and several transmitted powers, is validated with an electrocardiogram (ECG) signal. In addition to the installation simplicity, the proposed system has the ability of tuning both frequency and power. Using a VNA, two measurement techniques are compared using respectively one and two antennas. As the transmission of a signal with minimum power would be safer for both patient and medical staff, measurements are performed for several transmitted power levels between 0 and -25 dBm. Both heartbeat rate and Heart Rate Variability (HRV) are extracted and compared to the values obtained by the ECG. The rest of the paper is organized as follows: Section II presents background information, the proposed system with its two versions, and a smoothing technique. Section III presents the heartbeat signals obtained using two versions of the proposed system: one and two antennas respectively. Section IV states the results of extracting the heartbeat rate and the heart rate variability from both phases of S 11 and S 21 . Section V concludes the work.

T MEASUREMENT SPECIFICATIONS AND SMOOTHING TECHNIQUE

This section presents background information about the Doppler radar principle, the proposed system, and a smoothing technique. It is noteworthy to mention that our previous work has demonstrated a feasibility study in the absence of an ECG signal [START_REF] Obeid | Non-Contact Heartbeat Detection at 2.4, 5.8 and 60 GHz: A Comparative Study[END_REF][START_REF] Obeid | Multitunable Microwave System for Touchless Heartbeat Detection and Heart Rate Variability Extraction[END_REF]. In this work, measurements were performed simultaneously with an ECG reference signal. In addition, a recent work shows the possibility of detecting the heartbeat signal using a VNA and one antenna (S 11 ) [START_REF] Serra | A single on-body as a sensor for cardiopulmonary monitoring[END_REF]. This study provides a comparison between using single antenna (S 11 ) and two antennas (S 21 ).

Background Information

The chest displacement varies between 4 mm and 12 mm due to respiration, while it ranges between 0.2 mm and 0.5 mm due to heart beatings. The measurement of this small displacement is the objective of this work. The proposed system detects the heartbeat activity upon the detection of the phase variation of parameters S 11 or S 21 measured by the VNA. The phase variation is directly proportional to the chest displacement and indirectly proportional to the wavelength of the transmitted signal. The relation between the chest displacement ∆x(t) and the phase variation ∆θ(t) is:

4 ( ) ( ) x t t π θ λ Δ Δ = (1)
where λ is the wavelength of the transmitted signal.

System Description

The first version consists of using a VNA and one antenna where the S 11 parameter is considered. The second version consists of using a VNA and two antennas where the S 21 parameter is considered. Measurements were performed while holding the breath for duration of 15 seconds. This duration may vary in the order of milliseconds in order to sustain synchronization between the VNA and the ECG. The number of points taken for the signals measured with the VNA is 20000. The use of a VNA accommodates a quick and simple installation process. Fig. 1 represents the proposed system. Beside simplicity, many benefits are offered upon the use of a VNA (N5230A); these include the following features:

• Full 4-port S-parameter and balanced measurements up to 20 GHz • 120 dB dynamic range at 20 GHz • Trace noise < 0.006 dB at 100 kHz IFBW • Sampling period < 4 μs • Automatic port extension automatically corrects in fixture measurements.

• Advanced connectivity with LAN, USB, and GPIB interfaces One of the most powerful features gained using a VNA is the possibility of tuning both the operational frequency and the transmitted power. In addition, a VNA allows controlling different parameters such as the sweep time and the number of points for a specific trace; thus the sampling rate can be pre-determined.

Two wide band antennas (Q-par Angus Ltd) were used in experiments performed between 2 and 18 GHz. The Qpar Angus Ltd (model number WBH2-18HN/S) has the following specifications:

• Frequency: 2 -18 GHz The operational frequency used in these measurements is 16 GHz. The choice of this frequency is made according to two reasons. The first one is to obtain higher phase variation as it is directly proportional to the frequency, and the second reason is due to the equipment's limits (2 -16 GHz for the antennas). An extension of the system operating at 60 GHz is described in [START_REF] Obeid | Non-Contact Heartbeat Detection at 2.4, 5.8 and 60 GHz: A Comparative Study[END_REF][START_REF] Obeid | Multitunable Microwave System for Touchless Heartbeat Detection and Heart Rate Variability Extraction[END_REF]. The measurements procedure starts by sending a continuous wave signal at the desired frequency. The signal is generated by the VNA and driven to the transmitting antenna, which is directed toward the person's chest. The transmitted signal is reflected off the person chest and received by an antenna similar to the transmitter. The received signal is driven to the VNA where the phase variation of S 21 is computed. This phase corresponds to the difference between the phase of the received signal and the phase of the transmitted signal. The variation of the phase of S 21 is due to the chest displacement. Hence, it contains information about the cardiopulmonary signals when breathing normally, and about the heartbeat signal when holding the breath. The same procedure is used when the phase of S 11 is measured. In this case, only one antenna is used for both transmission and reception.

Smoothing Technique

Since the heartbeat rate is based on the detection of the signal peaks, a smoothing method is applied in order to reduce the number of surrounding peaks. The smoothing method is based on the Newton relation:

m m k k m k a b C a b - = + = ∑ mk ( 2
)
where n = 2m + 1 is the length of the smoothing window and

! , k,n , 0 k n !( )! k n n C k n k = ∈ ≤ ≤ - N (3)
Consider the obtained phase variations represented by a vector P of N samples:

[ (1)...

( )... ( )... ( )... ( )] P p p i m p i p i m p N

= - + (4)
The smoothing technique replaces a given point p(i) by the weighted average of its surrounding points. This is obtained according to the following relation:

2 2 2 0 1 ( ) ( ) 2 m k s m m k p i C p i m k = = - ∑ + (5)
where i should verify the following condition:

1 m i N + ≤ ≤ -m (6)
In general, the length of the smoothing window is chosen according to the following criterion:

1% 20% n N ≤ ≤ (7)
On the other hand, the length of the smoothing window should be decreased accordingly when the corresponding point p(i) is near either sides of the vector P. For example, when 1 ≤ i ≤ m, p(i) is replaced by:

∑ - = - + - = ) 1 ( 2 0 ) 1 ( 2 ) 1 ( 1 4 1 ) ( i k k i k p C i i s p (8) When N-m+1 ≤ i ≤ N, p(i) is replaced by: p s (i ) = 1 4 N -i C 2( N -i ) k p (2i -N + k ) k =0 2( N -i ) ∑ (9)
HEARTBEAT SIGNAL DETECTION This section presents the results obtained upon the detection of the heart activity using two different schemes.

The first scheme is based on measuring the phase of S 11 where single antenna is used for both transmitting and receiving. The second scheme is based on measuring the phase of S 21 where two antennas are used to respectively transmit and receive signals.

One Antenna

Measurements are performed for several transmitted powers between 0 and -25 dBm, and for a distance of 1 m to the person. Fig. 2 (a) shows the heartbeat signal detected using an ECG, while Fig. 2 (b) shows the phase variation of S 11 due to the heartbeat activity. These two signals are obtained simultaneously for the same patient. Fig. 2 (c) shows the heartbeat signal after applying the previously described smoothing technique.

Beside that the heartbeat signal detected with an ECG represents the electrical activity of the heart, the goal of the microwave Doppler system is to measure the heartbeat rate; this corresponds to the average of the R-R intervals of the ECG signal.

Fig. 3 shows the heartbeat signals that correspond to the phase of S 11 detected by the proposed microwave system for several power levels of the transmitted signal: 0 dBm (a), -5 dBm (b), -10 dBm (c), -15 dBm (d), -20 dBm (e), and -25 dBm (f). It can be noticed that as the power decreases, the signal-to-noise ratio (SNR) decreases. However, the peaks of the signals can still be detected. The effect of smoothing these signals is shown in Fig. 4.

Two Antennas

Measurements are performed for several transmitted power values. The phase variation of S 21 is measured. Fig. 5 (a) shows the heartbeat signal detected using an ECG, while Fig. 5 (b) shows the phase variation of S 21 due to heartbeats detected using a transmitted power of -5 dBm. These two signals are obtained simultaneously for the same patient. Fig. 5 (c) shows the heartbeat signal (the phase of S 21 ) after applying a smoothing technique.

Fig. 6 shows the phase variation of S 21 due to the heartbeats detected by the proposed microwave system for several transmitted powers: 0 dBm (a), -5 dBm (b), -10 dBm (c), -15 dBm (d), -20 dBm (e), and -25 dBm (f). As for the phase variation of S 11 , the signal-to-noise ratio (SNR) decreases as the power of the transmitted signal decreases. The peaks of the signals can still be detected even for low power levels of the transmitted signal. The effect of smoothing these signals is shown in Fig. 7.

Note that y-axis of Figs. 2 (b,c) and Figs. 5 (b,c) represent the phase in degrees while the y-axis of Fig. 2 (a) and Fig. 5 (a) stand for the ECG signal represented by 8 bits digital values.

S 11 VERSUS S 21 : HEARTBEAT RATE AND HEART RATE VARIABILITY

Related works tend to extract the average heartbeat rate for a specific window. This does not provide information about the time-variation of the HR, i.e. the HRV. Thus, a peak detection technique is required to track the peaks of the signal. In this work, the heartbeat rate and the HRV of the signals detected for different power levels are extracted. Values obtained from both parameters S 11 and S 21 , for original and smoothed signals, are compared to HR and the HRV obtained from the ECG reference signal.

The Peak Detection Method (PDM) used in this work is based on the following steps:

1. the maximum value of a 2-seconds window starting at time=0 is considered as the first peak 2. the window is shifted to the right by n points and another maximum value is searched 3. the new maximum value is conserved if it satisfies two conditions: the first one is to remain maximum for a duration of 0.5 seconds (0.25 at right and 0.25 at left), and the second one is to have a value greater than a threshold T The size of the window is chosen as the minimum heartbeat rate is 30 beats per minute, hence the maximum duration between 2 beats is 2 seconds. n is chosen according to the number of points to be read at time. In this code n is equal to 10. The value of T is chosen as a mean between the average value and the maximum value of the signal.

Fig. 8 shows the HR relative error of the Doppler radar system, for the phases of both S 11 and S 21 , with respect to the reference ECG signal. The effects of smoothing the signal are also shown in this figure .  It can be noticed that both S 11 and S 21 parameters' can detect the heartbeat signals with relative errors around 1.5% at 0 dBm. However, using the phase of S 11 shows a relative error for the HR error greater than 10% when the power of the transmitted signal becomes less than -10 dBm. On the other hand, S 21 shows the possibility of extracting the HR even at -25 dBm, with a relative error around 4%. The effect of smoothing the signals is higher for S 11 than for S 21 ; this is due to the lower SNR obtained for S 11 . Specifically, the HR relative error for S 11 changes between 1.5% and 26% for transmitted power between 0 and -25 dBm respectively. The smoothing technique changes these values to 2% and 12% respectively. Higher accuracy is obtained when measuring the phase of S 21 where HR relative error between 0 and 3.5% is obtained. Smoothing the signal decreases this relative error to 1.5%.

The HRV is a measure of the beat-to-beat variations (R-R interval for the ECG signal). HRV is regarded as an activity indicator of autonomic regulation of circulatory function; notice that controversy exists over whether this is an accurate metric for analyzing cardiovascular autonomic control [START_REF] Joseph | Effect of reflexological stimulation on heart rate variability[END_REF][START_REF] Al-Hazimi | Time-domain analysis of heart rate variability in diabetic patients with and with-out autonomic neuropathy[END_REF][START_REF] Lin | Modeling heart rate variability in healthy humans: A turbulence analogy[END_REF]. The extraction of the heartbeat rate requires the use of simple averaging over a specific window. However, this lacks providing the time-variation of the heartbeat rate. On the other hand, the extraction of the heart rate variability requires peak-to-peak detection. Hence, a PDM is used. This technique is based on tracking the maximum value in a sliding window, which is greater than a specific threshold. The size of the window is 2 seconds starting at the last peak detected in the preceding window. The minimum and maximum durations between two consecutive peaks are 0.5 and 1.5 seconds respectively. These values correspond to a heartbeat rate between 120 and 40 beats per minute.

It can be noticed that higher accuracy is obtained when using two antennas system, while using one antenna system requires higher directivity and positioning of the antenna.

Fig. 9 shows the HRV relative error of the Doppler radar system, for the phases of both S 11 and S 21 parameters, with respect to the reference ECG signal. The effects of smoothing the signal are shown as well. The proposed system shows the possibility of detecting the peaks of the signal with high accuracy. The HRV relative error for S 11 changes between 2.5% and 28% for transmitted power between 0 and -25 dBm. The HRV relative error for S 21 changes between 1.5% and 21%. On the other hand, smoothing the signal changes the HRV error as follows: 3 -11% interval for S 11 , and 1 -6% for S 21 . CONCLUSION A microwave system is proposed in order to detect the heart beatings of a person from a distance of 1 meter. A comparison between two versions of the proposed system is presented. The phase variations of parameters S 11 and S 21 obtained with a VNA due to the heart activity are extracted and compared to a simultaneous ECG signal. Both schemes, with one and two antennas, show the possibility of detecting the heartbeat activity and allow extracting the heartbeat rate and the heart rate variability. Measurements are performed at 16 GHz and for several transmitted power levels between 0 and -25 dBm. Beside its installation simplicity, the VNA provides the opportunity of testing several transmitted powers. Determining the minimum possible transmitted power would be safer for the patient and the medical staff.

Using one antenna scheme shows the possibility of extracting the heartbeat rate, while it lacks determining the heart rate variability at low power levels of the transmitted signal. This is due to the sensitive positioning of the antenna toward the chest of the person. On the other hand, the two antennas scheme shows higher accuracy in extracting both HR and HRV. A smoothing technique is applied to the signal where high accuracy is observed in extracting the HR and the HRV for both signals obtained with S 11 and S 21 .

Future work will focus on performing measurements for several operational frequencies simultaneously with an ECG. This helps determining the optimum operational frequency for the minimum transmitted power.

Fig. 1 .

 1 Fig. 1. System design: Pc-based ECG and VNA (only the Tx antenna is used for S 11 ).

Fig. 2 .

 2 Fig. 2. ECG signal (a) and phase variations of S 11 due to heartbeats detected at 16 GHz and -5 dBm: original signal (b), smoothed signal (c).

Fig. 3 .

 3 Fig. 3. Phase variations of S 11 due to heartbeats detected at 16 GHz for 0 dBm (a), -5 dBm (b), -10 dBm (c), -15 dBm (d), -20 dBm (e), and -25 dBm (f).

Fig. 4 .

 4 Fig. 4. Phase variations of S 11 , after applying the smoothing technique, due to heartbeats detected at 16 GHz for 0 dBm (a), -5 dBm (b), -10 dBm (c), -15 dBm (d), -20 dBm (e), and -25 dBm (f).

Fig. 5 .

 5 Fig. 5. ECG signal (a) and phase variations of S 21 due to heartbeats detected at 16 GHz and -5 dBm: original signal (b), smoothed signal (c).

Fig. 6 .

 6 Fig. 6. Phase variations of S 21 due to heartbeats detected at 16 GHz for 0 dBm (a), -5 dBm (b), -10 dBm (c), -15 dBm (d), -20 dBm (e), and -25 dBm (f).

Fig. 7 .

 7 Fig. 7. Phase variations of S 21 , after applying the smoothing technique, due to heartbeats detected at 16 GHz for 0 dBm (a), -5 dBm (b), -10 dBm (c), -15 dBm (d), -20 dBm (e), and -25 dBm (f).

Fig. 8 .

 8 Fig. 8. HR relative error of S 11 and S 21 for both original and smoothed signals.

Fig. 9 .

 9 Fig. 9. HRV relative error of S 11 and S 21 for both original and smoothed signals.


Clément Guy

Clément Guy On Model

On Model Subtyping

  Technologies de l'informatique, simulation numérique On Model Subtyping Clément Guy, Triskell and Cairn teams, IRISA-Inria clement.guy@irisa.fr CONTEXT •Rapid increase of the number of modeling languages •More and more model manipulation operators •Need for a systematic engineering •Providing design methods and facilities (reuse of operators and structures, advanced tools…) •Existing approaches remain disconnected from each other Need for a unified theory Model types for graphs MODEL-ORIENTED TYPE SYSTEMS •Model-oriented type systems should provide facilities such as abstraction, reuse, safety, auto-completion… A type of a model is a set of types (MOF classes) of objects which may belong to the model, and their relations. MODEL SUBTYPING RELATIONS •Context of the subtyping relation •Total subtyping: Models typed by A can be safely used everywhere models typed by B are expected •Partial subtyping: Models typed by A can be safely used in a given context in which models typed by B are expected (e.g., a given model transformation) A family of model-oriented type systems CLASSIFICATION OF EXISTING APPROACHES •Underused approaches: •Partial subtyping (i.e., subtyping wrt. a specific context) •Implicit declaration of a subtyping relation •Declaration at the definition of a type •Isomorphic subtyping (too restrictive) •Lack of information on subtyping relations checking Classification of existing approaches for model manipulation reuse

•

  How can we safely use model typed by A where a model typed by B is expected? •Structure of the model types •Isomorphic subtyping: Same names, multiplicities… •Non-isomorphic subtyping: Adaptation from the subtype to Jean-Marc Jézéquel Co-advisors: Benoit Combemale & Steven DerrienPERSPECTIVES•Could we propose a type system providing each one of those features?

Total / Partial Isomorphic/ ¬isomorphic At / After definition Explicit / Implicit Checking Legacy tool reuse

	Varrò et al.	Total	Class renaming	After	Implicit	?	No
	Cuccurru et al.	Total	Class renaming	After	Explicit	?	Yes
	Steel et al.	Total	Class renaming	After	Implicit	At design time,	Yes
			multiplicities			with errors at
			contraction			runtime
	Sanchez Cuadrado	Total	Class renaming	After	Explicit	?	Yes
	et al.		multiplicities
			contraction
	Sen et al.	Partial	Any adaptation	After	Explicit	At design time,	Yes
						with errors at
						runtime
	De Lara et al.	Total	Class renaming	After (Binding)	Explicit	?	No
			navigation and	At (Specialization)
			filtering of properties,
			n-to-1 bindings
	Babau et al.	Total	Isomorphic	After	Explicit	?	Yes