Sana Chaabani
email: schaabani@mondragon.edu

Iñigo Rodriguez

Mikel Cuesta

Yessine Ayed

Pedro José Arrazola

Guénaël Germain

Tool wear and cutting forces when machining inconel 718 under cryogenic conditions: Liquid nitrogen and carbon dioxide

Keywords: Machining, Inconel 718, Cryogenic, Tool wear

published or not. The documents may come   L'archive ouverte pluridisciplinaire

INTRODUCTION

Nickel superalloys reveal excellent mechanical properties at high temperature, high corrosion and creep resistance. Consequently, nickel based alloys are largely employed in the aeronautic applications. However, such alloys are difficult to cut materials due to many reasons such as the strain hardening while machining process [1]. In addition, these alloys present high chemical affinity with most of tool material leading to rapid wear of the tool by adhesion and diffusion mechanisms [2,3]. Moreover, the nickel based alloys exhibit very low thermal conductivity inducing high cutting temperature at the cutting zone [4,5]. For these reasons, to maintain the tool cold as possible and thus to decelerate tool wear evolution, it is relevant to use assisted machining process for instance, high pressure water jet [6,[START_REF] Bouchnak | Etude du comportement en sollicitations extrłmes et de l'usinabilit d'un nouvel alliage de titane aronautique : le ti555-3[END_REF] as well as cryogenic machining process [START_REF] Pusavec | [END_REF]9]. In addition, cryogenic process provides an ecological and safe alternative motivating to industrial applications. In this context, previous works have figured out the effect of cryogenic machining on tool wear when machining hard to cut materials either in the case of using liquid nitrogen LN 2 or carbon dioxide CO 2 . Indeed, Ayed et al [10] have reported the effect of cryogenic cooling on tool wear, especially tool flank wear. Following the tests carried out during the machining of the Ti64 alloy, they have figured out that the evolution of the tool flank wear, is less important in the cryogenic conditions using the liquid nitrogen compared with dry and conventional cooling conditions. Indeed, under dry and conventional conditions, the tool life is lower than 2.5 min and 6 min respectively. In contrast, after 15 min of machining, the tool flank wear does not exceed 0.2 mm under cryogenic conditions when the liquid nitrogen was maintained at high flow rate and high pressure. They have explained this by the fact that the nitrogen jet contributes to lowering the cutting temperature and subsequently to control relatively the wear mechanisms [11].

Thereafter, Kaynak [12] has investigated the performance of Inconel 718 under cryogenic assistance machining by comparing it with dry and MQL machining conditions. Indeed, during the tests, he chose two nozzles to throw the liquid nitrogen at the cutting face and the flank of the tool. The results obtained indicate a good agreement with the results cited above with respect to the reduction in the tool flank wear by comparing the other machining modes namely dry and MQL machining conditions. In fact, the largest measure has been identified in the case of dry machining. However, the MQL approach provided values slightly close to those obtained in cryogenic machining up to 2.5 min and then the tool flank wear increases rapidly to show the same trend as the dry condition.

Recently, Iturbe et al [1] have conducted an extensive research to estimate the extent to which conventional lubrication could be replaced by cryogenic fluid (LN 2 ). To do this, they took over the same configurations exploited by Kaynak (the work material Inconel 718, the cutting parameters, the geometry of the tool). Indeed, the tests were carried out under two conditions namely conventional machining and machining associating MQL and cryogenic fluid. As a result, the findings revealed that the tool life presents a significant difference between conventional lubrication and cryogenic conditions. In the case of conventional machining, the life of the tool exceeds 20 min and shows a homogeneous evolution. However, in the other case (MQL + cryogenic), the evolution of the tool wear occurs quickly and the standard criterion of the life of the tool is reached for shorter times (less than 7 min). This is likely due to the fact that the machined material Inconel 718 has a tendency to hardening excessively under the effect of cryogenic temperatures. Thereby, generating the degradation of the tool and consequently the reduction of its life.

Few researches have studied the effect of cryogenic machining using carbon dioxide CO 2 as a cutting fluid performed on several workpiece materials for instance hardened steel, Ti64 and Inconel 718 [13,14]. Indeed, it has been reported that cryogenic machining using the CO 2 showed better performance in terms of tool life or cutting length in comparison with the LN 2 cryogenic condition, dry and traditional lubrication. For instance, Jerold et al [15] have figured out the effect of cryogenic fluids (LN 2 and CO 2 ) on tool wear considering as a reference the conventional lubrication. Subsequently, Bagherzadeh et al [16] have carried out an extensive study focusing on CO 2 cryogenic cooling approach using different configurations. In fact, during the experiments, they have employed four cooling strategies consisting in: cooling the tool rake face using the CO 2 cryogenic fluid, cooling the rake face using CO 2 and the flank face using MQL, combining the CO 2 fluid and the MQL to cool the tool rake face (CMQL) and using a modified nozzle to cool simultaneously rake and flank faces using the CO 2 cutting fluid. The following table highlights a summary of the results obtained in the previous works carried out when machining several materials under cryogenic conditions focusing on the CO 2 performance compared with dry, MQL or wet conditions (Table 1). [16] Inconel 718 Turning 100 -37.4 * * Cutting length obtained in CO 2 cooling condition. † Reduction of tool wear obtained in CO 2 compared to wet condition. So far, according to the review literature, there is no study that has focused on the comparison between the cryogenic fluids performance namely LN 2 and CO 2 when machining Inconel 718 using the same cutting parameters in order to identify the influence of each cryogenic fluid on Inconel 718 machinability. This paper is dedicated to examine the effect of the latter cryogenic fluids on tool life when machining Inconel 718 considering as a reference the conventional lubrication. Therefore, the experimental methodology is detailed followed by a comparison between the cryogenic configurations performance and traditional lubrication in terms of tool life. Cutting forces are also included.

Experimental methodology Experimental set-up

Machining trials were carried out under conventional coolant and cryogenic cutting fluids using liquid nitrogen LN 2 and carbon dioxide CO 2 . The trials were performed using the same test configuration on a horizontal turning CNC lathe Danumeric 2. Turning tests with the conventional lubrication were conducted cooling the cutting zone with the HOCUT 3380 fluid. As for the cryogenic tests using LN 2 as a coolant, the cryogenic system is composed of the phase separator, the cryogenic control and the liquid nitrogen bottle mounted on the CNC lathe. LN 2 spray was activated before beginning the machining process in order to achieve stable outlet condition. With respect to the CO 2 system, it is composed of a bottle of CO 2 maintained at high pressure equal to 57 bars at room temperature (Fig 1).

Experimental procedure

Longitudinal turning experiments were conducted in finishing operations on forged Inconel 718 bar using the same cutting parameters and the same cutting tool as Iturbe [1], that are a cutting speed of 70 m/min, a feed per revolution of 0.2 mm/rev and a depth of cut of 0.2 mm. However, three cooling ways were employed: LN 2 (the delivery position and the diameter of the nozzle have been changed compared to [1]), CO 2 and conventional cooling. CVD coated carbide inserts from Mitsubishi supplier (DNMG 150612-MS US905) having a tool nose radius of 1.2 mm were exploited in the trials. Table 2 sets out the machining conditions: The experimental tests were conducted until reaching the target machining time (15 min) or when the maximum tool life defined as V bmax = 0.3 mm was reached according to the standard 3685:1993. Each experiment has been carried out two times using the same cutting tool edge under each cooling condition and the average values have been exploited for the analysis. Tool flank wear measurements have been recorded using a LEICA Z16 APO macroscope after each cutting test. Furthermore, the cutting forces were recorded using Kistler 9121 dynamometer.

Results and discussions

Tool wear

Tool flank wear evolution was recorded during the longitudinal turning trials in all cooling strategies (conventional and cryogenic conditions). The experiments were stopped at a machining time of 15 minutes in wet condition even if the maximum flank wear did not exceed the criterion of V bmax = 0.3 mm. Nevertheless, under cryogenic conditions, the experiments were stopped when the tool flank wear achieved the criterion value. As illustrated in Fig 2 ,results showed that the tool life in conventional condition is the longest duration achieved. Indeed, the tool flank wear does not exceed 0.12 mm after 15 minutes of machining in wet condition. Regard the CO 2 cooling condition, the same tendency of tool wear evolution has been observed until reaching 12 min of machining. However, after 12 min, tool flank wear increases notably to exceed the criterion at 15 min. As for LN 2 cryogenic condition, this parameter increased rapidly from the first 2 minutes of machining leading to shorter tool life. Additionally, under LN 2 cryogenic condition, the chipping of the cutting tool was drastically pronounced compared to the conventional lubrication. Furthermore, it should be noted that the tool wear evolution under both cryogenic conditions is quite repeatable at the beginning of the cutting process. However, a notable variability occurred at 11 min and 13 min respectively under LN 2 and CO 2 cooling approaches when the tool flank wear evolved significantly. Overall, under conventional coolant environment, a homogeneous tool flank wear evolution was observed even after longer machining times showing a good repeatability. Nevertheless, in LN 2 cryogenic machining condition, wear peaks were recorded from the beginning of the turning process, revealing that the cutting process is not performed homogeneously whereas this parameter reveals a steady and slow evolution under CO 2 cryogenic condition except the last minutes of machining as shown in Fig 3.

Cutting forces

The cutting forces are crucial factors that indicate the power consumption in the machining process. Actually, the cutting forces components are strongly dependent on several parameters for instance the workpiece properties, the cutting tool material and geometry as well as the coolant strategy. In this work, one is interested to figure out the tendency of cutting forces components when machining Inconel 718 under wet and cryogenic conditions (LN 2 and CO 2 ) that reveal the state of tool wear. Indeed, the cutting, the feed and the passive forces are significantly higher under cryogenic cooling environments than in the case of conventional lubrication (Fig 4 and Actually, the cryogenic fluids namely LN 2 and CO 2 exhibit very low temperature during the machining tests inducing likely higher flow stresses of the workpiece material even before cutting. Probably, this is could be because of a notable cooling effect of the cryogenic fluids preventing the material softening. Consequently, the cutting forces increase drastically. Focusing on the tendency of the three components of the cutting forces obtained in conventional condition that showed lower values when machining Inconel 718 in comparison with LN 2 and CO 2 cryogenic fluids. In fact, the maximum value of cutting forces that has been recorded in conventional condition is 180 N while in LN 2 and CO 2 cooling conditions, this parameter has exceeded 300 N and 250 N respectively. Moreover, the feed forces reveal the same tendency where the maximum values are obtained under LN 2 cryogenic conditions. In contrast, the passive forces showed the highest values measured in all cooling approaches compared to the cutting and feed forces. Results revealed huge values obtained in both cryogenic conditions compared to conventional lubrication. Additionally, it should be noted the strong correlation between the tendency of passive forces and the evolution of tool flank wear. Effectively, one can figure out the trends of the passive forces with tool flank wear state at different machining times under all cooling conditions namely the conventional and the cryogenic cutting fluids. As obviously illustrated in all figures, as the machining time increases, the tool flank wear increased as well. Thereby, passive forces increased relatively to the coolant condition. In particular, passive forces evolution is steady and slow in wet condition unlike the cryogenic conditions where the passive forces rise quite notably. This phenomenon is more pronounced in LN 2 cooling environment than in CO 2 condition. This fact may be due to the rapid tool flank wear evolution in LN 2 cryogenic condition indicating that the passive forces are the most sensitive to tool flank wear.

Conclusion

In this paper, a literature review was reported concerning the study of the poor machinability of nickel-based superalloy Inconel 718. Then, the machining performance of the work material was identified by mentioning the major research that have focused on the cryogenic cooling effect on tool wear when machining Inconel 718. Subsequently, machining trials using three different cooling conditions namely conventional and LN 2 and CO 2 cryogenic conditions have been presented. The experimental results showed the good performance during machining of conventional coolant in terms of tool life reaching 15 min knowing that the flank wear does not achieve the criterion. In contrast, the LN 2 condition indicates shorter tool life revealing a rapid and non homogeneous tool flank wear. With respect to the CO 2 cooling condition, the tool flank wear showed slight difference compared to the conventional lubrication performance only at longer machining time.

Additionally, the cutting forces components showed higher value in the case of cryogenic environments than in conventional condition due to the significant cooling effect. Finally, for enhancement purposes, one is extremely interested in understanding the main tool wear mechanisms under the whole cooling strategies. For this reason, extensive analysis will be conducted using SEM technique as well as the surface integrity investigation (microstructure damage and residual stresses).

FIGURE 1 .

 1 FIGURE 1. Experimental cryogenic systems: LN 2 and CO 2

FIGURE 2 .

 2 FIGURE 2. Tool flank wear evolution in different cooling conditions: Wet, LN 2 and CO 2 .

FIGURE 3 .

 3 FIGURE 3. Comparison of tool flank wear evolution in different cooling conditions: a) Wet, b) LN 2 and c) CO 2 after 10 min of machining.

Fig

FIGURE 4 .

 4 FIGURE 4. Comparison between cutting forces components evolution in different cooling conditions: Wet, LN 2 and CO 2 : Cutting forces (left) and Feed forces (right).

FIGURE 5 .

 5 FIGURE 5. Comparison between passive forces evolution in different cooling conditions: Wet, LN 2 and CO 2 .

TABLE 1 .

 1 Summary of the previous studies showing the effect of the CO 2 performance compared to different cooling methods

	Cooling conditions Work Material	Machining Cutting speed Tool wear Cutting
			operations	(m/min)	reduction	length
					%	(m)
	Wet-LN 2 -CO 2 [15] AISI 1045 steel	Turning	94	91 †
	Wet-LN 2 -CO 2 [15] AISI 1045 steel	Turning	145	81.5 †
	MQL-CO 2 [16]	Ti64	Turning	150	-	42.8 *
	MQL-CO 2

TABLE 2 .

 2 Working conditions

	Workpiece	Geometry	Cylindrical bar
		Material	Inconel 718
	Cutting parameters Cutting speed(m/min) 70
		Depth of cut (mm)	0.2
		Feed (mm/rev)	0.2
	Coolants	Conventional	Hotcut
		Cryogenic	LN 2
		Cryogenic	CO 2


Zhiqi Kang

Mostafa Sadeghi

Radu Horaud

Face Frontalization Based on Robustly Fitting a Deformable Shape Model to 3D Landmarks

Face frontalization consists of synthesizing a frontally-viewed face from an arbitrarily-viewed one. The main contribution of this paper is a robust face alignment method that enables pixel-to-pixel warping. The method simultaneously estimates the rigid transformation (scale, rotation, and translation) and the non-rigid deformation between two 3D point sets: a set of 3D landmarks extracted from an arbitrary-viewed face, and a set of 3D landmarks parameterized by a frontally-viewed deformable face model. An important merit of the proposed method is its ability to deal both with noise (small perturbations) and with outliers (large errors). We propose to model inliers and outliers with the generalized Student's t-probability distribution function -a heavy-tailed distribution that is immune to non-Gaussian errors in the data. We describe in detail the associated expectationmaximization (EM) algorithm that alternates between the estimation of (i) the rigid parameters, (ii) the deformation parameters, and (iii) the t-distribution parameters. We also propose to use the zero-mean normalized cross-correlation, between a frontalized face and the corresponding ground-truth frontally-viewed face, to evaluate the performance of frontalization. To this end, we use a dataset that contains pairs of profile-viewed and frontallyviewed faces. This evaluation, based on direct image-to-image comparison, stands in contrast with indirect evaluation, based on analyzing the effect of frontalization on face recognition. 1

I. INTRODUCTION

The problem of face frontalization is the problem of synthesizing a frontal view of a face from an arbitrary view. Recent research has shown that face frontalization consistently boosts the performance of face analysis. In particular, it has been recently demonstrated that face recognition from frontal views yields better performance than face recognition from unconstrained views [START_REF] Yim | Rotating your face using multi-task deep neural network[END_REF]- [START_REF] Zhao | Towards pose invariant face recognition in the wild[END_REF]. This observation is equally valid for other tasks, such as the analysis of facial expressions [START_REF] Pei | Monocular 3d facial expression features for continuous affect recognition[END_REF] or of lip reading [START_REF] Fernandez-Lopez | Survey on automatic lip-reading in the era of deep learning[END_REF], [START_REF] Adeel | Lip-reading driven deep learning approach for speech enhancement[END_REF].

It is well established that lip movements, as well as tongue and jaw movements, are controlled by speech production and that they are correlated with word pronunciation [START_REF] Schultz | Biosignal-based spoken communication: A survey[END_REF]. Consequently, the capacity to properly analyse facial movements plays an important role in visual and audio-visual speech separation, speech enhancement and speech recognition, e.g. [START_REF] Dupont | Audio-visual speech modeling for continuous speech recognition[END_REF]- [START_REF] Tao | End-to-end audiovisual speech recognition system with multitask learning[END_REF]. This is particularly useful when audio signals are corrupted by ambient noise and by acoustic perturbations. Nevertheless, facial-movement analysis is perturbed by inherent rigid head movements. It is therefore important to separate Z. Kang, M. Sadeghi and R. Horaud are with Inria Grenoble and with Université Grenoble Alpes. 1 Supplemental material for this paper is accessible at https://team.inria.fr/ perception/research/rff/. non-rigid facial movements from rigid head movements, and face frontalization may well be viewed as such a rigid/nonrigid separation process.

In this paper we address face frontalization as the problem of simultaneously estimating the 3D pose and deformation parameters of an arbitrarily-viewed face, e.g. Figure 1. The main contribution is a robust point-set alignment method that enables face frontalization via pixel-to-pixel warping. In details, we propose a method that simultaneously estimates the rigid transformation (scale, rotation, and translation) and the non-rigid deformation between two 3D point sets, namely (i) a set of 3D landmarks extracted from an arbitrary-viewed face and (ii) a set of frontally-viewed 3D landmarks that are parameterized by a deformable face model. An important merit of the proposed alignment estimator is its ability to deal both with noise (small perturbations) and with outliers (large errors), i.e. robust estimation. We propose to model both inliers and outliers with the generalized Student's t-probability distribution function (pdf) -a heavy-tailed distribution that is immune to non-Gaussian errors in the data [START_REF] Sun | Robust mixture clustering using Pearson type VII distribution[END_REF], [START_REF] Forbes | A new family of multivariate heavy-tailed distributions with variable marginal amounts of tailweight: application to robust clustering[END_REF]. We describe in detail the associated expectation-maximization (EM) algorithm that alternates between the estimation of (i) the rigid parameters, (ii) the deformation parameters, and (iii) the pdf parameters. Interestingly, the proposed method can be indifferently applied to align two rigid point sets or to fit and align a deformable point set to a rigid one.

We also propose to use the zero-mean normalized crosscorrelation (ZNCC) coefficient between a frontalized face and the corresponding ground-truth frontally-viewed face, in order to empirically evaluate the performance of face frontalization. To this end, we use a dataset that contains pairs of profileviewed and frontally-viewed faces. This direct evaluation, based on image-to-image comparison between the prediction and the ground-truth, stands in contrast with indirect evaluation, based on analyzing the effect of frontalization on face recognition, e.g. [START_REF] Yim | Rotating your face using multi-task deep neural network[END_REF]- [START_REF] Zhao | Towards pose invariant face recognition in the wild[END_REF].

The remainder of this article is organized as follows. In Section II we review face frontalization methods that were proposed in the recent past. Section III outlines the proposed algorithm and describes in detail the proposed 3D point-set alignment method. Section IV analyzes some technical aspects of the method and provides implementation details. Section V describes the proposed evaluation metrics and shows results obtained with our method and with two state-of-the-art methods. Section VI draws some conclusions. Finally, Appendix A, Appendix B and Appendix C provide mathematical details of the proposed algorithms. Fig. 1: Overview of the proposed robust face frontalization method. 3D landmarks are extracted from both an arbitrarlyviewed input face and from a frontally-viewed deformable shape model. These landmarks are robustly aligned, thus enabling to compute the pose of the input face and to frontalize the input landmarks. The deformable shape model is next fitted to these landmarks, thus obtaining a frontalized shape model, followed by the computation of a frontal dense depth map; the latter is obtained by interpolation of the 3D vertices of the triangulated mesh that describes the shape model. Finally, the input-face pixels are warped onto the output-image pixels.

II. RELATED WORK

As already mentioned, face frontalization consists of synthesizing a frontally-viewed face from an arbitrarily-viewed face. A popular approach has been to train deep neural networks (DNNs) to learn a mapping from an arbitrary view to a frontal view, using a massive collection of input/target pairs of faces. It was shown that architectures based on generative adversarial networks (GANs), e.g. [START_REF] Zhao | Towards pose invariant face recognition in the wild[END_REF], [START_REF] Huang | Beyond face rotation: Global and local perception GAN for photorealistic and identity preserving frontal view synthesis[END_REF]- [START_REF] Rong | Feature-improving generative adversarial network for face frontalization[END_REF] outperform architectures based on convolutional neural networks (CNNs) [START_REF] Yim | Rotating your face using multi-task deep neural network[END_REF] for face recognition. The main drawback of these DNN-based methods is that they are designed to predict frontal faces that are as neutral as possible, in order to improve the performance of face recognition. Hence, there is no guarantee that the predicted output preserves non-rigid facial deformations. These methods are therefore hardly usable for facial expression recognition or for lip reading.

Another approach has been to estimate the 3D pose of an input face with respect to a frontal 3D face model, and then to use the pose parameters to warp the facial pixels from the input image onto a frontal view. Recently proposed face frontalization methods capitalize on 3D pose estimation, e.g. [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF], [START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF]. The problem of 3D pose estimation from point-topoint correspondences has been thoroughly addressed in the past and many solutions have been proposed in the presence of various camera models, e.g. [START_REF] Horaud | Object pose: The link between weak perspective, paraperspective, and full perspective[END_REF]. In [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] it was proposed to use a 3D generic model of a face from which a frontal image is synthesized. Next, 48 2D facial landmarks are extracted from the input face and from the generic frontal face, which provides 2D-to-3D correspondences between the input face and the generic 3D model. This amounts to estimate the projection (or camera) matrices, from the 3D face model onto the input image and onto the frontal image, respectively, and to subsequently warp pixels from one image to another. A similar method was proposed in [START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF] where 68 2D landmarks are extracted from both the input face and from a frontal face associated with a generic 3D model. The 3D pose parame-ters are obtained from 2D-to-3D landmark correspondences between the input face and the 3D model.

Nevertheless, the pose-based methods just cited suffer from a number of limitations. First, the 3D models that they use do not take facial deformations into account. Second, the estimation of 3D pose with an unknown projective camera model is an ill-posed problem in the case of quasi-planar shape models such as faces. Third, the pose estimator itself is not robust to the presence of large errors in landmark detection and localization, and to large non-rigid facial deformations.

The proposed method overcomes these limitations on the following grounds. First, we use a deformable 3D face model that parameterizes faces with large variabilities in appearance and in expression. Deformable shape models were initially introduced in [START_REF] Cootes | Active shape models -their training and application[END_REF], and subsequently used in [START_REF] Blanz | A morphable model for the synthesis of 3D faces[END_REF]- [START_REF] Cao | Facewarehouse: A 3D facial expression database for visual computing[END_REF] to model faces and facial deformations. Second, we use 3D facial landmarks rather than 2D landmarks, e.g. [START_REF] Bulat | How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks)[END_REF]- [START_REF] Zhu | Face alignment in full pose range: A 3d total solution[END_REF]. Consequently, there is no need to estimate the parameters of the camera model, at the price of approximating the latter with a scaled orthographic (or weak perspective) model. However, the latter is a realistic approximation of the true camera model, since human faces are quasi planar surfaces and, hence, their depth range is small relative to the camera-to-face distance [START_REF] Horaud | Object pose: The link between weak perspective, paraperspective, and full perspective[END_REF]. Third, we propose a robust method for aligning a 3D set of landmarks with a 3D deformable face model. The associated EM solver alternates between the estimation of the rigidtransformation parameters, the model-deformation parameters, and the robust statistical-model parameters. This paper builds on a recent study showing that the use of 3D facial landmarks in combination with robust pose estimation yields excellent performance [START_REF] Sadeghi | Unsupervised performance analysis of 3d face alignment[END_REF].

As already mentioned, face frontalization has been essentially used as a pre-processing step for face recognition and the evaluation metrics proposed so far are generally based on the recognition rate [1]- [START_REF] Zhao | Towards pose invariant face recognition in the wild[END_REF], [START_REF] Wang | Facial expressionaware face frontalization[END_REF]. This kind of evaluation lacks direct significance for standalone face frontalization meth- [START_REF] Zhu | High-fidelity pose and expression normalization for face recognition in the wild[END_REF], which biases the results and does not allow to evaluate the extent to which face frontalization preserves non-rigid facial deformations.

III. ROBUST FACE FRONTALIZATION

A. Outline of the Proposed Method

The proposed robust face frontalization (RFF) method is summarized in Algorithm 1 and Figure 1. The core idea of the method is to estimate the 3D pose (scale, rotation, and translation) of an input face based on robust alignment of two 3D point sets, namely a set of 3D landmarks extracted from the input face, and a set of 3D landmarks associated with the frontal pose of a 3D shape model. This is done in the framework of maximum likelihood estimation and, in practice, an expectation-maximization algorithm is being used, i.e. [START_REF] Blanz | A morphable model for the synthesis of 3D faces[END_REF] and Appendix A. This allows to frontalize the input landmarks, i.e. ( 5) and, then, to fit a deformable shape model to the frontalized landmarks, which yields a frontalized shape model of the input face, i.e. [START_REF] Wu | A novel lip descriptor for audio-visual keyword spotting based on adaptive decision fusion[END_REF]. Next, vertex interpolation is used to compute a dense depth map associated with the frontalized shape model, i.e. ( 11) and [START_REF] Tao | End-to-end audiovisual speech recognition system with multitask learning[END_REF]. Eventually, a frontalized face is computed by warping the input face onto a frontal image, i.e. ( 13) and ( 14).

B. Frontalization via Alignment of 3D Point Sets

Let I p be an observed image of a face in an unknown pose. 2 A set X of J = 68 3D landmarks is extracted from I p with image-centered coordinates X 1:J = {X j } J j=1 ⊂ R 3 . Throughout the paper we adopt the notation X j = (X j1 , X j2 , X j3 ) to designate the three coordinates of a point in R 3 . Let Z 1:J = {Z j } J j=1 ⊂ R 3 be the 3D coordinates of a set of landmarks, Z, that correspond to the frontal pose of a 3D deformable face model, e.g. [START_REF] Sadeghi | Unsupervised performance analysis of 3d face alignment[END_REF], and let I f be the frontal image corresponding to this frontal view of the model. The problem of face-pose estimation consists of finding the rigid transformation that best maps X 1:J onto Z 1:J on the premise that the landmarks are in one-to-one correspondence and up to an error (or residual) e j :

e j = Z j -(ρRX j + t), ∀j ∈ {1 . . . J}, (1)
where ρ ∈ R + , R ∈ R 3×3 and t ∈ R 3 are the scale, rotation matrix and translation vector, respectively, associated with the unknown face pose. Because the landmark locations X 1:J are inherently affected by detection noise and outliers, as well as by non-rigid facial deformations, it is suitable to use a robust alignment technique. For this purpose, we assume that the residuals e 1:J are samples of a random variable e drawn from a robust probability distribution function (pdf) P (e; θ), where θ is the set of parameters characterizing the pdf. Then, the problem is cast into maximum likelihood estimation (MLE) or, equivalently into the minimization of the following negative log-likelihood function:

L(θ, ρ, R, t|X 1:J , Z 1:J ) = - 1 2 J j=1 log P (e j ; θ), (2)
The generalized Student's t-distribution [START_REF] Forbes | A new family of multivariate heavy-tailed distributions with variable marginal amounts of tailweight: application to robust clustering[END_REF] is a robust pdf, namely:

P (e; θ) = ∞ 0 N (e; 0, w -1 Σ)G(w; µ, ν)dw (3)
where N () is the normal distribution, G() is the gamma distribution and Σ ∈ R 3×3 is a covariance matrix. w ∈ R is a precision associated with a residual. The precisions w 1:J are modeled as random variables drawn from gamma distributions.

Direct minimization of ( 2) is not practical. An expectationmaximization (EM) formulation is therefore adopted, namely minimization of the expected complete-data negative loglikelihood:

min θ E W [-log P (e 1:J , w 1:J |e 1:J ; θ)], (4)
where EM alternates between the estimation of the means of the precision posteriors, w 1:J and the estimation of the parameters θ = {ρ, R, t, Σ, µ}. As it can be seen in Appendix A, the precisions are precious because they weight the relative importance of the landmarks within the optimization process, and hence they help the parameter estimation process to be robust against badly localized landmarks. Once the pose parameters are estimated, the rigid transformation is applied to the landmarks X 1:J in order to obtain a set Y of frontalized landmarks whose coordinates in I f are denoted Y 1:J ⊂ R 3 , namely:

Y j = ρRX j + t, ∀j ∈ {1 . . . J}. (5)
The next step is to fit a deformable 3D face-shape model to this set of landmarks in order to eventually obtain a frontal dense depth map of the face. For that purpose and without loss of generality, we consider a linear deformation model, e.g. the 3DMM Basel Face Model (BFM) [START_REF] Paysan | A 3D face model for pose and illumination invariant face recognition[END_REF]. The latter consists of a 3D mesh with a set V of N vertices, whose coordinates V 1:N = { V n } N n=1 ⊂ R 3 are parameterized by a statistical linear shape-model in the following way (please consult Appendix B):

V n = V n + W n s, ∀n ∈ {1 . . . N }, (6)
where V 1:N ⊂ R 3 are the vertices of the mean shape model, W 1:N ⊂ R 3×K are reconstruction matrices, and s ∈ R K is a low dimensional embedding of the vertex set, with K 3N , see Appendix B. In order to fit this deformable 3D faceshape model to the frontalized landmarks Y 1:J , we consider a subset of J = 68 vertices with coordinates V 1:J that correspond, one-to-one, to the frontalized landmarks, namely { V j ↔ Y j } J j=1 . 3 For that purpose, the statistical shape model must be scaled, rotated, translated and deformed, such that the vertices V 1:J are optimally aligned with the landmarks Y 1:J . This yields the following negative log-likelihood function:

L(θ, σ, Q, d, s|Y 1:J , V 1:J ) = - 1 2 J j=1 log P (r j ; θ), (7)
where σ, Q and d parameterize the rigid transformation that aligns the two point sets. The residual r j is given by:

r j = Y j -(σQ(V j + W j s) + d), ∀j ∈ {1 . . . J}. (8)
As above, one can use the generalized Student's t-distribution (3) and an expectation-maximization algorithm to robustly estimate the model parameters. The notable difference between (7) and ( 2) is that, in addition to scale, rotation and translation, the deformable shape parameters must be estimated as well.

The shape parameters are estimated with (please consult Appendix C):

s =   J j=1 w j A j Σ -1 A j + κΛ -1   -1   J j=1 w j A j Σ -1 b j   , (9)
where A j = σQW j and b j = Y j -σQV j -d. The shape vertices can now be mapped onto the frontal view, namely Ṽ n = ( Ṽn1 , Ṽn2 , Ṽn3 ), with:

Ṽ n = σQ(V n + W n s) + d, ∀n ∈ {1 . . . N }. (10)
A frontal dense depth map is then computed in the following way. Remember that the shape vertices form a triangulated 3D mesh; therefore the projection of Ṽ 1:N onto the image plane I f form a 2D triangulated mesh whose vertices have ( Ṽn1 , Ṽn2 ) 1:N as 2D coordinates. Let n 1 , n 2 and n 3 be the indexes of the vertices of a mesh triangle. Moreover, let a 1 , a 2 and a 3 be the barycentric coordinates of a pixel (A 1 , A 2 ) that lies inside that triangle, with 0 ≤ a 1 , a 2 , a 3 ≤ 1 and a 1 +a 2 +a 3 = 1. The barycentric coordinates of pixel (A 1 , A 2 ) are computed with:

A 1 A 2 = a 1 Ṽn11 Ṽn12 + a 2 Ṽn21 Ṽn22 + a 3 Ṽn31 Ṽn32 . (11)
3 For the sake of simplifying the notations, we use {1 . . . J} ⊂ {1 . . . N }.

Once the barycentric coordinates are thus determined, the depth A 3 of the pixel is computed by interpolation, as follows:

A 3 = a 1 Ṽn13 + a 2 Ṽn23 + a 3 Ṽn33 (12)
The above procedure is repeated for all the triangles and for all the points inside each triangle, thus obtaining a frontal dense depth map of the face.

The final face frontalization step consists of warping the input face onto a frontal face. The rigid transformation from the frontal view to the profile view is the inverse of the pose, namely the inverse of ( 5): ρ = ρ -1 , R = R , and t = -ρ -1 R t. Assuming scaled orthographic projection, a oneto-one correspondence between a pixel (A 1 , A 2 ) ∈ I f with depth A 3 , and a pixel (B 1 , B 2 ) ∈ I p is obtained with the following formula:

B 1 B 2 = ρ R 11 R 12 R 13 R 21 R 32 R 33   A 1 A 2 A 3   + t 1 t 2 (13)
Finally, face frontalization consists of building an image whose intensities (or colors) are provided by:

I f (A 1 , A 2 ) = I p ([B 1 ], [B 2 ]). (14)
where [•] returns the integer part of a real number.

IV. ALGORITHM ANALYSIS AND IMPLEMENTATION DETAILS

The proposed method is summarized in Algorithm 1. The method starts with 3D facial landmark extraction which is achieved with [START_REF] Bulat | How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks)[END_REF] as it is one of the best-performing methods [START_REF] Sadeghi | Unsupervised performance analysis of 3d face alignment[END_REF]. Moreover, the pose estimation and model fitting steps of Algorithm 1 are achieved with Algorithm 2 (Appendix A) and Algorithm 3 (Appendix C), respectively. Both these two algorithms are EM procedures and hence they have good convergence properties.

All the computations inside these two algorithms are in closed-form, with the notable exception of the estimation of the rotation matrix. The latter is parameterized with a unit quaternion [START_REF] Horn | Closed-form solution of absolute orientation using unit quaternions[END_REF], which allows us to reduce the number of parameters, from nine to four, and to express the orthogonality constraints of the rotation matrix in a much simpler way. The minimization ( 27) is solved using a sequential quadratic problem [START_REF] Bonnans | Numerical optimization: theoretical and practical aspects[END_REF]. More precisely, a sequential least squares programming (SLSQP) solver 4 is used in combination with a rootfinding software package [START_REF] Kraft | A software package for sequential quadratic programming[END_REF]. The SLSQP minimizer found at the previous EM iteration is used to initialize the current EM iteration. At the start of EM, the closed-form method of [START_REF] Horn | Closed-form solution of absolute orientation using unit quaternions[END_REF] is used to initialize the rotation.

The proposed method also requires a deformable shape model, i.e. Appendix B. We consider a set S I = {S I 1 , . . . , S I m , . . . , S I M } ⊂ R 3N of M shapes, where S I m is a concatenation of N 3D coordinates that corresponds to a different face identity, and where each face was scanned in a frontal view and with a neutral expression. We also consider a set S E = {S E 1 , . . . , S E m , . . . , S E M } ⊂ R 3N that contains M scans of the same faces but with facial expressions, associated one-to-one with the M identities. Let S m = S E m -S I m be the expressive-neutral difference of face identity m, namely the expressive offset. A face S can be reconstructed from its identity and expression embeddings, i.e. Appendix B:

Ŝ = S I + S + U I s I + U s , (15)
where S I and S are the means associated with the identityand with the expressive-neutral difference, matrices U I and U contain the K principle eigenvectors, and s I and s are the corresponding embeddings. Note that we use Ŝ to emphasize that the reconstruction is an approximation of S.

In practice we use the publicly available Basel Shape Model (BSM) [START_REF] Paysan | A 3D face model for pose and illumination invariant face recognition[END_REF] augmented with [START_REF] Cao | Facewarehouse: A 3D facial expression database for visual computing[END_REF]. This provides M = 200 registered face scans in a frontal view and with neutral expressions, corresponding to M different identities, as well as M expressive scans of the same identities. Each scan is described by a triangulated mesh with an identical number of vertices, namely N = 53490. The dimension of the embedding is K = 200 and hence we have K 3N . We use the landmark locations associated with the mean identity S I to compute the pose, i.e Z 1:J .

V. EXPERIMENTS

As with any methodological development, performance evaluation is extremely important. In this paper we make recourse to empirical evaluation based on a dataset with associated ground truth. Such a dataset should contain pairs of frontal and profile views of faces for a large number of subjects. Quantitative performance evaluation consists of computing a metric between an image obtained by face frontalization of a profile view of a subject, with an image containing a frontally-viewed face of the same subject. It is desirable that the profile and frontal images are simultaneously recorded with two synchronized cameras. Therefore, the proposed evaluation is based on image-to-image comparison. Several metrics were developed in the past for comparing two images, e.g. featurebased and pixel-based metrics. In this work we use the zero-mean normalized cross correlation (ZNCC) coefficient between two image regions, a measure that has successfully been used for stereo matching, e.g. [START_REF] Sun | Fast stereo matching using rectangular subregioning and 3D maximum-surface techniques[END_REF], because it is invariant to differences in brightness and contrast between the two images, due to the normalization with respect to mean and standard deviation.

Let R f (h, v) ⊂ I f be a region of size H × V whose center coincides with pixel location (h, v) of a frontalized image I f . Similarly, let R t (h, v) ⊂ I t be a region of the same size and whose center coincides with pixel location (h, v) of a ground-truth image I t . The ZNCC coefficient between these two regions writes: where Cov [•, •] is the centered covariance between the two regions, Var [•] is the centered variance of a region, δh and δv are horizontal and vertical shifts, and h and v are the horizontal and vertical shifts that maximize the ZNCC coefficient.

C(h, v,h , v ) = (16) max δh,δv Cov [R f (h, v), R t (h + δh, v + δv)] Var [R f (h, v)] 1/2 Var [R t (h + δh, v + δv)] 1/2 ,
In order to evaluate the performance of the proposed frontalization method and to compare it with state-of-the-art methods, we used a publicly available dataset, namely the OulouVS2 dataset [START_REF] Anina | OuluVS2: a multiview audiovisual database for non-rigid mouth motion analysis[END_REF]. This dataset targets the understanding of speech perception, more precisely, the analysis of nonrigid lip motions that are associated with speech production. The dataset was recorded in an office with ordinary (artificial and natural) lighting conditions. The recording setup consists of five synchronized cameras (2 MP, 30 FPS) placed in five different points of view, namely 0 • , 30 • , 45 • , 60 • and 90 • .

The dataset contains 5 × 780 videos recorded with 53 participants. Each participant was instructed to read loudly several text sequences displayed on a computer monitor placed slightly to the left and behind the 0 • (frontal) camera. The displayed text consists of digit sequences, e.g. "one, seven, three, zero, two, nine", of phrases, e.g. "thank you", "have a good time", and "you are welcome", as well as of sequences from the TIMIT dataset, e.g. "agricultural products are unevenly distributed". While participants were asked to keep their head still, natural uncontrolled head movements and body position changes were inevitable. As a consequence the actual head pose varies from one participant to another and there is no exact match between the head and camera orientations.

In practice, we evaluated the performance of the proposed method and we compared it with two state-of-the-art methods for which the code is publicly available, namely [START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF], [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF]. We applied frontalization to images extracted from the videos recorded with the 30 • camera and compared the results with the "ground-truth", namely the corresponding images extracted from the videos recorded with the 0 • camera. Notice that videos recorded with higher viewing angles, i.e. 45 • , 60 • and 90 • , can be hardly exploited by a frontalization algorithm because half of the face is occluded, e.g. Figure 6. For each frontalized image I f we extract the mouth region R f and we search in the ground-truth image I t for the best-matching region R t . This provides a ZNCC coefficient ( 16) for each test image. Notice that ( 16) only cares about the horizontal and vertical shifts in the image plane and assumes that the frontalized face and the corresponding ground-truth frontal face share the same scale. In practice, different frontalization algorithms output faces at different scales. For this reason and for the sake of fairness, prior to applying (16), we extract facial landmarks from both the frontalized and ground-truth faces and we use a subset of this set of landmarks to estimate the scale factor between the two faces.

We randomly selected 30 video pairs, recorded with the 30 • and 0 • cameras, of 15 participants from the OulouVS2 dataset. Each video contains approximatively 160 images, therefore we used a total of 4800 images in our benchmark. The mean ZNCC coefficients obtained with two state-of-theart methods and with the proposed method are displayed in Table I. Both [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] and [START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF] exploit facial symmetry to fill in the occluded areas, which improves the ZNCC metrics. Indeed, facial symmetry can be used, either by replacing occluded pixels with the mirror-symmetric ones (soft symmetry), or by flipping the visible half of the face (hard symmetry).

We noticed that there were important discrepancies in method performance across participants. In order to better understand this phenomenon, we computed the mean ZNCC coefficient for nine participants and displayed these means and standard deviations as a function of the yaw angle (in degrees), i.e. the horizontal head orientation associated with the 3D head-pose estimated with the proposed method, please refer to Table II. One may notice that there is a wide range of yaw angles, from 19 • and up to 40 • , and that the performance gracefully decreases as the yaw angle increases. Nevertheless, the proposed method yields the best scores for participants #19 and #12 and the second best score for participant #21, and this is without using mirror-symmetric information.

Examples of face frontalization obtained with our method and with the methods of [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] and [START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF] are shown on Figure 2, Figure 2, Figure 3 and Figure 5. As already mentioned, both [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] and [START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF] enforce facial symmetry as a post-processing frontalization step to compensate for the gaps caused by self occlusions. We also show an example of applying frontalization to the 45 • camera, Figure 6. In this example the estimated yaw angle is 57 • and only half of the lip area is visible. Consequently, the ZNCC coefficient is below 0.5 in this case. Notice that when the hard-symmetry strategy of [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] is applied, the ZNCC coefficient increases considerably.

Finally, Figure 7 and Figure 8 show a few examples of frontalization results obtained with faces from the LFW dataset [START_REF] Huang | Labeled faces in the wild: A database for studying face recognition in unconstrained environments[END_REF]. It is worthwhile to notice that our method yields frontal images of a better quality than the methods of [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] and [START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF], and that these two methods produce many artefacts and unrealistic facial deformations.

VI. CONCLUSIONS

In this paper we introduced a robust face frontalization (RFF) method that is based on the simultaneous estimation of the rigid transformation between two 3D point sets and the non-rigid deformation of a 3D face model. This is combined with pixel-to-pixel warping, between an input image of an arbitrarily-viewed face and a synthesized frontal view of the face. The proposed method yields state-of-the-art results, both quantitatively and qualitatively, when compared with two recently proposed methods. Up to now, the performance of face frontalization has been evaluated using face recognition benchmarks. We advocate that direct image-to-image comparison between the predicted output and the associated groundtruth yields a better assessment criterion that is not biased by another task, e.g. face recognition.

It is worthwhile to notice that the performance of face recognition can be boosted by combining face frontalization with the use of facial symmetry. However, the latter is likely to introduce undesired artefacts and unrealistic facial deformations, which are quite damaging for other face analysis tasks, such as expression recognition or lip reading. [START_REF] Huang | Labeled faces in the wild: A database for studying face recognition in unconstrained environments[END_REF]. Both [START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] and [START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF] make use of mirror symmetry, which introduces unrealistic facial deformations.

In the future, we plan to extend the proposed method to image sequences in order to allow robust temporal analysis of faces. In particular, we are interested in combining face frontalization with audio-visual speech enhancement. As already outlined, face frontalization may well be viewed as a process of discriminating between rigid head movements and non-rigid facial deformations, which can be used to eliminate head motions that naturally accompany speech production, and hence leverage the performance of visual speech reading and of audio-visual speech enhancement.

APPENDIX A ROBUST ESTIMATION OF THE RIGID ALIGNMENT BETWEEN TWO 3D POINT SETS

In this appendix, we address the problem of robust alignement between two 3D point sets, X with coordinates

X 1:J = {X j } J j=1 ⊂ R 3×J and Z with coordinates Z 1:J = {Z j } J j=1 ⊂ R 3×J
, respectively. As mentioned in Section III the residuals (1) are samples of a random variable drawn from the generalized Student t-distribution [START_REF] Forbes | A new family of multivariate heavy-tailed distributions with variable marginal amounts of tailweight: application to robust clustering[END_REF]:

P (e; Σ,µ, ν) = ∞ 0 N (e; 0, w -1 Σ)G(w, µ, ν)dw = Γ(µ + 3 2 ) |Σ| 1 2 Γ(µ)(2πν) 3 2 1 + e 2 M 2ν -(µ+ 3 2 ) , (17)
where µ and ν are the parameters of the prior gamma distribution of the precision variable w, and Γ(•) is the gamma function. Without loss of generality, we set ν = 1 [START_REF] Forbes | A new family of multivariate heavy-tailed distributions with variable marginal amounts of tailweight: application to robust clustering[END_REF].

Notice that using the conjugate prior property, the posterior distribution of w is also a gamma distribution, namely the

with parameters:

a = µ + 3 2 , b = 1 + 1 2 e 2 Σ . (19)
The posterior mean of the precision is:

w = E[w|e] = a b . ( 20
)
Direct minimization of the likelihood function is not practical. An expectation-maximization formulation is therefore adopted, namely the minimization of the expected complete-data negative log-likelihood (4), E W [-log P (e 1:J , w 1:J |e 1:J ; θ)] and in this case the parameter vector is a concatenation of the alignment and pdf parameters, namely θ = {ρ, R, t, Σ, µ} since we set ν = 1. This yields the minimization of:

Q(θ) = 1 2 J j=1 w j Y j -ρRX j -t 2 Σ + log |Σ| , (21)
By taking the derivatives of ( 21) with respect to the parameters and equating to zero, we obtain the following analytical expressions for optimal values of the following parameters:

t = Y -ρRX, (22)
ρ = J j=1 w j Y j Σ -1 Y j J j=1 w j (RX j ) Σ -1 (RX j ) 1 2 , (23)
Σ = 1 N J j=1 w j (Y j -ρRX j )(Y j -ρRX j ) , (24)
with the following subsitutions:

X = J j=1 w j X j N j=1 w j , X j = X j -X, (25)
Y = J j=1 w j Y j N j=1 w j , Y j = Y j -Y . (26)
By substituting [START_REF] Paysan | A 3D face model for pose and illumination invariant face recognition[END_REF] in [START_REF] Blanz | A morphable model for the synthesis of 3D faces[END_REF] and using the notations above, the rotation is estimated by minimization of:

R = argmin R 1 2 J j=1 (w j Y j -ρRX j 2 Σ ). (27)
The minimizer ( 27) yields a closed-form solution for an isotropic covariance, i.e. Σ = σI. In the general case, however, one has to make recourse to nonlinear minimization. It is practical to represent the rotation with a unit quaternion and to minimize (27) using a sequential quadratic programming method [START_REF] Bonnans | Numerical optimization: theoretical and practical aspects[END_REF]. The parameter µ is updated by solving the following equation, where Ψ(•) is the digamma function:

µ = Ψ -1   Ψ(a) - 1 n J j=1 log b j   . (28)

APPENDIX B DEFORMABLE SHAPE MODEL

In this appendix we summarize the deformable shape model that we use. The model is based on a training set S = {S 1 , . . . , S m , . . . , S M } of M 3D face scans. Each 3D face scan is described with a triangulated mesh composed of N 3D points, or vertices, V 1:N = {V 1 , . . . , V N }, where the coordinates of each vertex are V n = (V n1 , V n2 , V n3 ) . Hence, a shape in S is described by the vector S:

S = (V 11 , V 12 , V 13 , . . . , V N 1 , V N 2 , V N 3 ) ∈ S ⊂ R 3N .
It is assumed that all the shapes are represented in the same coordinate frame, that they share the same number of vertices and that there is a one-to-one correspondence between their vertices, namely {V nm ↔ V nm } N n=1 , for any two shapes in the training set, S m and S m and for each vertex index n. Notice, however, that the problem of finding a rigid alignment between M shapes based on point-to-point correspondences is not trivial because of the presence of non-rigid shape deformations and of large shape variabilities. In what follows we will treat each shape/face as a point set. The covariance matrix associated with these M registered shapes, {S m } M m=1 , is:

Σ = 1 M M m=1 (S m -S)(S m -S) , (29)
where S = (V 1 , . . . , V N ) ∈ R 3N is the mean shape with, V n = 1/M M m=1 V nm . The covariance Σ ∈ R 3N ×3N is a symmetric semi-definite positive matrix, hence its eigendecomposition writes Σ = UΛU , where U = U 1 . . . U 3N ∈ R 3N ×3N is an orthogonal matrix and

Λ = Diag λ 1 , . . . , λ 3N , with λ 1 ≥ • • • ≥ λ 3N ≥ 0.
It is common practice to select the number K of principal components, or modes, such that the sum of the K largest eigenvalues represents 95% of the total variance, K k=1 λ k = 0.95 3N n=1 λ n . We introduce the following truncated matrices:

U = U 1 . . . U K ∈ R 3N ×K , Λ = Diag λ 1 , . . . , λ K ∈ R K×K ,
with the property U U = I K . Matrix U projects the centered shapes from R 3N onto R K :

s m = U (S m -S), (30)
where s m ∈ R K is a low-dimensional embedding of S m . Conversely, it is possible to reconstruct an approximation of S m , denoted Ŝm , from its low-dimensional embedding:

Ŝm = S + Us m , (31)
which is a concatenation of:

V nm = V n + W n s m , (32)
where the N block-matrices W n ∈ R 3×K are such that:

U =         W 1 . . . W n . . . W N         (33)
The concept of statistical shape model is summarized in Fig. 9. A training set of M shapes is embedded into a lowdimensional space spanned by the K principal eigenvectors of the covariance matrix (29) associated with the training set and is characterized by the K principal (largest) eigenvalues, where an eigenvalue λ k corresponds to the variance along direction u k . Importantly, a valid shape Ŝ is a shape reconstructed from an an embedding s that lies inside an ellipsoid whose half axes are equal to √ λ k , or:

Ŝ = S + Us, s.t. s Λ -1 s ≤ 1. (34)
This guarantees that the reconstructed shape belongs to the family of training shapes with 99% confidence. Fig. 9: A training set of M shapes (black dots), characterized by their mean (orange dot) and covariance, is projected onto the space spanned by the the principal eigenvectors of the shape covariance matrix, U 1 to U K . The number K of principal eigenvectors is selected such that the sum of the associated eigenvalues represents 95% of the total variance. A shape S m projects onto a shape embedding s m . Conversely, shape Ŝm (blue dot), reconstructed from s m , approximates shape S m .

APPENDIX C ROBUST ESTIMATION OF THE ALIGNMENT BETWEEN A DEFORMABLE SHAPE AND A 3D POINT SET

In this appendix we extend the robust alignment method of Appendix A to deformable shapes. The expression (21) can be re-written as:

Q(θ) = 1 2 J j=1 w j Y j -σQ(V j + W j s) -d 2 Σ + log |Σ| + η 2 s Λ -1 s, (35)
where the statistical shape model is scaled, rotated, translated and deformed in order to be aligned with a set of landmarks, and where the last term constrains the embedding s to lie inside a 99% confidence ellipsoid, i.e. [START_REF] Anina | OuluVS2: a multiview audiovisual database for non-rigid mouth motion analysis[END_REF].

By taking the derivatives of [START_REF] Huang | Labeled faces in the wild: A database for studying face recognition in unconstrained environments[END_REF] with respect to the parameters, i.e. θ = {σ, Q, d, Σ, s, µ}, and equating to zero, we obtain the following analytical expressions for the optimal values of the following parameters:

d = Y -σQ V , (36)
σ =   J j=1 w j Y j Σ -1 Y j J j=1 w j (Q V j ) Σ -1 (Q V j )   1 2 , (37)
Σ = 1 N J j=1 w j (Y j -σQ V j )(Y j -σQ V j ) , (38)
where similar notations as in Appendix A and Appendix B were used, i.e ( 25), ( 26) and ( 31). An identical nonlinear optimization problem for the rotation is obtained as well:

Q = argmin Q 1 2 J j=1 (w j Y j -ρQ V j 2 Σ ). (39)
Additionally, there is a closed-form solution for the optimal embedding:

s =   J j=1 w j A j Σ -1 A j + η Λ -1   -1   J j=1 w j A j Σ -1 b j   , (40)
where the following notations were used: A j = σQW j and b j = Y j -σQV j -d. The associated EM algorithm is similar to Algorithm 2; the most notable difference is that the M-step alternates between the estimation of the rigid parameters and the shape parameters, namely Algorithm 3.

Fig. 2 :Fig. 3 :Fig. 4 :Fig. 5 :Fig. 6 :

 23456 Fig. 2: Face frontalization exemples and ZNCC coefficients obtained with our method for participant #02 from the OulouVS2 dataset. Top row: input face recorded with the 30 • camera. The estimated horizontal head orientation (yaw angle) is of 25 • in this case. Middle row: frontalization results with self occluded facial regions displayed in white. Bottom row: ground-truth frontal face recorded with the 0 • camera. The ZNCC coefficients correspond to the mouth bounding boxes shown in red.

Fig. 7 :

 7 Fig. 7: Face frontalization examples from the Labeled Face in the Wild (LFW) dataset[START_REF] Huang | Labeled faces in the wild: A database for studying face recognition in unconstrained environments[END_REF]. Both[START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] and[START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF] make use of mirror symmetry, which introduces unrealistic facial deformations.

Fig. 8 :

 8 Fig. 8: Additional examples from LFW showing the visual impact of using soft symmetry in conjunction with the proposed method. The frontal image (d) is shown for qualitative visual comparison and it is not used as ground truth.

TABLE I :

 I Mean ZNCC coefficients for 15 participants. The best score is in bold and the second best is in slanted bold.

	Method	Post-processing	ZNCC
	Hassner et al. [18]	-	0.723
	Hassner et al. [18]	Soft symmetry	0.780
	Hassner et al. [18]	Hard symmetry	0.722
	Banerjee et al. [2]	-	0.739
	Banerjee et al. [2]	Soft symmetry	0.788
	Proposed	-	0.824

TABLE II :

 II Results for nine participants as a function of estimated yaw angle (in degrees) that corresponds to the horizontal head orientation computed with the proposed 3D head-pose estimator. Both[START_REF] Hassner | Effective face frontalization in unconstrained images[END_REF] and[START_REF] Banerjee | To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?[END_REF] make use of symmetry to fill in the occluded face areas. The best scores are in bold and the second best are in slanted bold.

	Participant	Yaw	[18]	[2]	Proposed
	#31	19.1	0.905	0.856	0.924
	#01	23.5	0.915	0.893	0.913
	#02	24.9	0.888	0.878	0.948
	#10	29.0	0.805	0.812	0.806
	#23	30.0	0.810	0.857	0.842
	#27	32.9	0.685	0.852	0.732
	#19	37.8	0.752	0.650	0.754
	#12	38.5	0.731	0.713	0.774
	#21	40.6	0.632	0.743	0.735
	Mean & std.		0.791±0.100	0.806±0.084	0.825±0.085

  Centered point coordinates X 1:J and Y 1:J Initialization of θ old : Use a closed-form solution for the rotation to evaluate R old and ρ old ; evaluate Σ old using[START_REF] Bulat | How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks)[END_REF]. Provide µ old ; while θ newθ old > do E-step: evaluate a new and b new 1:N using (19) with θ old , then evaluate w new 1:N using (20) ; Update the centered coordinates ; M-scale-step: Evaluate ρ new using (23); M-rotation-step: Estimate R new via minimization of (27) ; M-covariance-step: Evaluate Σ new using (24) ; M-mu-step: Evaluate µ new using (28) ; θ old ← θ new ; end Optimal translation: Evaluate the translation vector using (22); Result: Optimal scale ρ , rotation R , translation t , covariance Σ , and precisions w 1:N .

	This yields Algorithm 2:
	Algorithm 2: Robust estimation of the rigid transfor-
	mation between two 3D point sets.
	Data:

The term face pose refers to a rigid transformation, namely the scale factor, rotation matrix and translation vector that map a face-centered coordinate frame onto a coordinate frame that is aligned with the camera frame.

https://docs.scipy.org/doc/scipy/reference/optimize.html

Algorithm 3: Robust fitting of a deformable shape model to a 3D point set.

Data: Centered landmark coordinates Y 1:J , mean shape V 1:J and shape reconstruction matrices W 1:J Initialization of θ old : Set s old = 0, use the closed-form solution [START_REF] Horn | Closed-form solution of absolute orientation using unit quaternions[END_REF] to evaluate Q old and σ old ; evaluate Σ old using (38). Provide µ old . Compute the centered vertex coordinates V 1:J ; while θ newθ old > do E-step: evaluate a new and b new 1:N using ( 19) with θ old , then evaluate w new 1:N using (20) ; Update the centered coordinates ; M-scale-step: Evaluate σ new using (37); M-rotation-step: Estimate Q new via minimization of (39) ; M-covariance-step: Evaluate Σ new using (38) ; M-shape-step: Evaluate s new using (40) ; M-mu-step: Evaluate µ new using (28) ; θ old ← θ new ; end Optimal translation: Evaluate the translation vector using (36); Result: Optimal scale σ , rotation Q , translation d , covariance Σ , and precisions w 1:N .