Keywords: 3D face modeling, Decoupled generative models, 4D face registration, Surface normal estimation

sortie sans nuire aux performances lors de l'auto-encodage des modalités, en obtenant des résultats de pointe pour la tâche.

1. Anger, disgust, fear, happiness, sadness and surprise.

.8b.

Abstract

Data-driven models of the 3D face are a promising direction for capturing the subtle complexities of the human face, and a central component to numerous applications thanks to their ability to simplify complex tasks. Most data-driven approaches to date were built from either a relatively limited number of samples or by synthetic data augmentation, mainly because of the difficulty in obtaining large-scale and accurate 3D scans of the face. Yet, there is a substantial amount of information that can be gathered when considering publicly available sources that have been captured over the last decade, whose combination can potentially bring forward more powerful models.

This thesis proposes novel methods for building data-driven models of the 3D face geometry, and investigates whether improved performances can be obtained by learning from large and varied datasets of 3D facial scans. In order to make efficient use of a large number of training samples we develop novel deep learning techniques designed to effectively handle three-dimensional face data. We focus on several aspects that influence the geometry of the face: its shape components including fine details, its motion components such as expression, and the interaction between these two subspaces.

We develop in particular two approaches for building generative models that decouple the latent space according to natural sources of variation, e.g.identity and expression. The first approach considers a novel deep autoencoder architecture that allows to learn a multilinear model without requiring the training data to be assembled as a complete tensor. We next propose a novel non-linear model based on adversarial training that further improves the decoupling capacity. This is enabled by a new 3D-2D architecture combining a 3D generator with a 2D discriminator, where both domains are bridged by a geometry mapping layer.

As a necessary prerequisite for building data-driven models, we also address the problem of registering a large number of 3D facial scans in motion. We propose an approach that can efficiently and automatically handle a variety of sequences while making minimal assumptions on the input data. This is achieved by the use of a spatiotemporal model as well as a regression-based initialization, and we show that we can obtain accurate registrations in an efficient and scalable manner.

Finally, we address the problem of recovering surface normals from natural images, with the goal of enriching existing coarse 3D reconstructions. We propose a method that can leverage all available image and normal data, whether paired or not, thanks to a new cross-modal learning architecture. Core to our approach is a novel module that we call deactivable skip connections, which allows to transfer the local details from the image to the output surface without hurting the performance when autoencoding modalities, achieving state-of-the-art results for the task.

Résumé

Les modèles du visage 3D fondés sur des données sont une direction prometteuse pour capturer les subtilités complexes du visage humain, et une composante centrale de nombreuses applications grâce à leur capacité à simplifier des tâches complexes. La plupart des approches basées sur les données à ce jour ont été construites à partir d'un nombre limité d'échantillons ou par une augmentation par données synthétiques, principalement en raison de la difficulté à obtenir des scans 3D à grande échelle. Pourtant, il existe une quantité substantielle d'informations qui peuvent être recueillies lorsque l'on considère les sources publiquement accessibles qui ont été capturées au cours de la dernière décennie, dont la combinaison peut potentiellement apporter des modèles plus puissants.

Cette thèse propose de nouvelles méthodes pour construire des modèles de la géométrie du visage 3D fondés sur des données, et examine si des performances améliorées peuvent être obtenues en apprenant à partir d'ensembles de données vastes et variés. Afin d'utiliser efficacement un grand nombre d'échantillons d'apprentissage, nous développons de nouvelles techniques d'apprentissage profond conçues pour gérer efficacement les données faciales tri-dimensionnelles. Nous nous concentrons sur plusieurs aspects qui influencent la géométrie du visage : ses composantes de forme, y compris les détails, ses composants de mouvement telles que l'expression, et l'interaction entre ces deux sous-espaces.

Nous développons notamment deux approches pour construire des modèles génératifs qui découplent l'espace latent en fonction des sources naturelles de variation, e.g.identité et expression. La première approche considère une nouvelle architecture d'auto-encodeur profond qui permet d'apprendre un modèle multilinéaire sans nécessiter l'assemblage des données comme un tenseur complet. Nous proposons ensuite un nouveau modèle non linéaire basé sur l'apprentissage antagoniste qui davantage améliore la capacité de découplage. Ceci est rendu possible par une nouvelle architecture 3D-2D qui combine un générateur 3D avec un discriminateur 2D, où les deux domaines sont connectés par une couche de projection géométrique.

En tant que besoin préalable à la construction de modèles basés sur les données, nous abordons également le problème de mise en correspondance d'un grand nombre de scans 3D de visages en mouvement. Nous proposons une approche qui peut gérer automatiquement une variété de séquences avec des hypothèses minimales sur les données d'entrée. Ceci est réalisé par l'utilisation d'un modèle spatio-temporel ainsi qu'une initialisation basée sur la régression, et nous montrons que nous pouvons obtenir des correspondances précises d'une manière efficace et évolutive.

Finalement, nous abordons le problème de la récupération des normales de surface à partir d'images naturelles, dans le but d'enrichir les reconstructions 3D grossières existantes. Nous proposons une méthode qui peut exploiter toutes les images disponibles ainsi que les données normales, qu'elles soient couplées ou non, grâce à une nouvelle architecture d'apprentissage cross-modale. Notre approche repose sur un nouveau module qui permet de transférer les détails locaux de l'image vers la surface de

Introduction

The face is our main vehicle for communication, whether verbally, nonverbally, or even involuntarily via micro-expressions. Our ability to produce at least some of the facial expressions is an innate and not a cultural trait [START_REF] Ekman | Universal facial expressions of emotion[END_REF], and thus deeply connected to our most basic instincts. It is also our principal source for recognizing people: it is the face that we will recall first when we think of someone. As such, it has long captivated researchers from numerous domains, ranging from biology and psychology to computer vision and computer graphics.

Digital faces have received similar attention and are equivalently challenging. They play a central role in the film and gaming industry, which has pushed for impressive advances with the goal of creating believable characters. Highly realistic faces can currently be modeled, rendered and animated, although the process is still very complex and expensive, requiring significant manual input from the artists. Digital 3D faces are also ubiquitous in consumer-grade applications. From more trivial tasks such as creating effects during a video-conference, to the more promising applications of telepresence, avatar generation and fully autonomous virtual agents, all of these rely on an underlying model of the three-dimensional face. They are also part of numerous applications in the medical field, including early diagnosis of craniofacial disorders [START_REF] Suttie | Facial dysmorphism across the fetal alcohol spectrum[END_REF], reconstruction of missing parts for implant design [START_REF] Mueller | Missing facial parts computed by a morphable model and transferred directly to a polyamide laser-sintered prosthesis: an innovation study[END_REF], and pain detection [START_REF] Zhang | Three dimensional binary edge feature representation for pain expression analysis[END_REF], to name a few.

The key enabler for the majority of these applications is the underlying parametric model: a function that generates a 3D face based only on a few parameters, thus reducing the complexity of the task at hand. First proposed almost fifty years ago by [START_REF] Parke | A Parametric Model for Human Faces[END_REF], parametric models are widely used in the film industry thanks to their efficiency [Lewis et al., 2014b], as well as in numerous computer vision tasks where the low-dimensional representation can simplify ill-posed problems, e.g.recovering a 3D face from a single image [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF].

Building accurate models of the 3D face as well as its motion is a hard task, due to two main reasons. First, the anatomy of the face is very complex. Its shape is influenced by the underlying bone structure as well as the surrounding tissue, and movement is induced by small muscles that are attached close to the surface skin. When these muscles contract they create subtle but noticeable CHAPTER 1. INTRODUCTION changes that include creases and wrinkles, deforming the surface in a mostly non-rigid manner. To complicate things more, this highly flexible system varies from person to person according to their underlying shape.

A second source of complexity comes from our remarkably tuned ability to read expressions. Already in 1872 Charles Darwin argued that facial expressions are a biological and not a cultural trait [START_REF] Darwin | The expression of the emotions in man and animals[END_REF], and this was shown to be true for at least certain "universal" expressions 1 in the seminal work of psychologist Paul Ekman [START_REF] Ekman | Universal facial expressions of emotion[END_REF]. Since this is a skill more instinctive than learned, we can easily recognize subtle errors made by a computer generated face. This phenomenon has even been formalized in the so-called uncanny valley hypothesis. Introduced by [START_REF] Mori | The uncanny valley[END_REF], the hypothesis relates the level of realism of a character with the emotional response of the observer. The more realistic the better the response, but when it reaches a point of almost-realistic, but not entirely, it produces a highly unpleasant effect due to our ability to recognize the missing aspects. Only when the face is truly realistic one can overcome this "uncanny valley".

Data-driven models are a promising direction for dealing with the aforementioned complexities. The goal here is to learn the particularities of the face from a database of real 3D scans, such that new identities and/or expressions can be generated by manipulating a few parameters based on the statistics of the database. Building such models involves several steps: scanning a large 3D facial dataset with sufficient variation in terms of identities and expressions; establishing correspondences among the captured data, such that they all share the same mathematical representation; and correctly modeling the different factors that affect the facial geometry using a meaningful parameterization. Each of these steps has challenges that remain unsolved.

First, capturing and processing the necessary 3D scans for building datadriven models is a laborious task, which is why most current models were learned from relatively limited datasets. Yet, given the interest that the facial shape has received from the research community, there is a large collection of publicly available databases that were acquired throughout the last decades. Combined, they cover a wide range of identities and expressions, whether static (e.g. [START_REF] Cao | 3d shape regression for real-time facial animation[END_REF][START_REF] Yin | A 3d facial expression database for facial behavior research[END_REF][START_REF] Savran | Bosphorus database for 3d face analysis[END_REF]) or in motion (e.g. [START_REF] Yin | A high-resolution 3d dynamic facial expression database[END_REF][START_REF] Fanelli | A 3-d audio-visual corpus of affective communication[END_REF][START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF], Zhang et al., 2014]), and can even capture fine-scale details (e.g. [START_REF] Stratou | Effect of illumination on automatic expression recognition: a novel 3d relightable facial database[END_REF]). These datasets are a valuable source of information for building data-driven models of the 3D face, and the ability to learn patterns from all of them can potentially result in more powerful models. But handling large 3D datasets coming from various sources presents additional challenges, requiring methods that are at the same time efficient, scalable and robust.

Establishing correspondences among raw scans is also a difficult problem, challenged by noisy data, lack of distinctive structures in large areas of the facial surface, and significant shape variations arising from different ethnicities, 1.1. CONTRIBUTIONS 5 age or expressions. The problem is even harder when dealing with large datasets for which its capture setup is not fully controlled, such as the ones previously mentioned.

Finally, there are many factors that influence the facial shape, and accurately capturing these poses too several challenges. One aspect that is often overlooked, and yet necessary for correctly animating a face, is how to model the interactions that occur between the shape and motion-related components of the face geometry; that is, how to correctly model the expressions taking into account the underlying shape. This is typically approached by either building independent models for the shape and expression spaces (thus ignoring any possible interaction), or by building tensor-based models which, although taking into account these interactions, are hard to scale to large datasets. Another challenging aspect is how to encode and recover detailed surfaces using a datadriven approach, since low-dimensional parametric models usually struggle to recover high-frequency information.

In this thesis we investigate whether better models for the 3D face geometry can be obtained by learning from a large number of real 3D scans. We are interested here in all aspects that influence the geometry of the face: its identityrelated components, including its details, as well as those that arise during motion. To this end, we develop novel techniques for building decoupled models (i.e.those that capture the interaction between the different subspaces) in Chapters 3 and 5; address the problem of establishing dense correspondences among a large number of datasets of 3D faces in motion in Chapter 4; and tackle the problem of recovering finer details through a surface normal representation in Chapter 6. Motivated by the need to handle large-scale datasets coming from publicly available sources, we propose efficient and scalable methods that take advantage of recent deep learning techniques, building performant models that can profit from all available data.

Contributions

This thesis contributes novel methods for building data-driven models of the 3D face, as well as a novel image/shape prior for the problem of surface normal recovery. We focus here on models that can decouple the latent space such that we can improve applications related to the facial motion. In the final chapter we move the focus onto the problem of recovering surface details, an aspect that is typically not encoded in low-dimensional parametric models. In all cases we developed algorithms that can profit from both large and realistic datasets, making use of deep learning frameworks to efficiently handle a large and varied source of information.

We make in particular the following contributions:

• A novel approach for building multilinear models from large datasets, in order to decouple the parameterization in a scalable manner and without the need to assemble the dataset into a complete tensor.

• A novel registration approach for 3D faces in motion, designed to efficiently and robustly put a large number of examples into a common parameterization.

• A novel modeling framework for learning non-linear decoupled models of the 3D face using adversarial learning. To enable this, we contribute a new architecture based on a geometry mapping layer, that allows to perform efficient convolutions on the 3D face while leveraging advances in 2D neural networks.

• A novel method for the estimation of surface normals from unconstrained images, which can learn from large and unpaired datasets of high-quality normals and in-the-wild 2D images of the face. To this end we contribute the deactivable skip connections, a novel module that enhances the capabilities of cross-modal learning and proved to be key for training from unpaired data.

Outline

We begin by reviewing the related work in Chapter 2, including all the necessary steps for building data-driven models.

Chapter 3 presents the multilinear autoencoder, an approach for building tensor-based decoupled models from large datasets that cannot be assembled into a complete tensor. We leverage the capacity of deep neural networks to optimize efficiently over large datasets and propose a new autoencoder architecture that allows to refine an initial multilinear model, such that it can better capture all available data.

Chapter 4 considers the problem of registration, and introduces a novel approach designed to handle large datasets of 3D faces in motion. Key to the method is the use of a spatiotemporal model as well as a regression-based initialization approach, which allows to efficiently process a large number of frames and sequences.

With a large and registered dataset now in hand, Chapter 5 revisits the problem of building decoupled models through a novel framework that makes use of recent generative adversarial learning techniques. This is enabled by a new 3D-2D architecture that allows to generate three-dimensional data while profiting from advances in 2D neural networks, where both domains are bridged by a geometry mapping layer.

In Chapter 6 we move the focus to the problem of estimating surface normals from unconstrained images, and propose a novel approach based on cross-modal learning which allows to learn a rich latent space that encodes 1.3. PUBLICATIONS both 2D facial image and 3D surface normal information, achieving state-ofthe-art results for the task.

Finally, Chapter 7 concludes with a summary of our main contributions as well as considerations for future directions that extend this work.

Publications

The material presented in this thesis is based on the following publications:

Chapter 3: • FERNANDEZ ABREVAYA, V.,

IEEE conference on computer vision and pattern recognition (CVPR). (2020)

2 Background

There are several ways by which we can discretize a 3D surface for computational purposes. A commonly used approach, and the one we follow throughout most of this thesis, is to represent the surface as a triangular mesh: a collection of vertices V = {v i ∈ R 3 , i ∈ [1 . . . n]} and triangular facets F = {(vi, vj, vk), v i , v j , v k ∈ V } that approximate the real, continuous, surface. An example of a facial mesh can be seen in Figure 2.1. This allows to represent the discrete 3D points as encoded by V , as well as any intermediate point p inside a triangle (v i , v j , v k ) through its barycentric coordinates (α, β), such that p = αv i + βv j + γv k , with α + β + γ = 1, α, β, γ ≥ 0. Triangular meshes can also be parameterized into 2D by a one-to-one mapping φ : V → D, in which each vertex v i ∈ V is associated with a coordinate (u, v) i in the unit square domain D. This so-called UV parameterization allows to store in a regular grid the associated color textures, displacement maps, normal maps, or even the geometry [START_REF] Gu | Geometry images[END_REF], encoding dense surface information by interpolating with the barycentric coordinates.

If we want to recover a mesh of n vertices with known triangulation F from an input (e.g.a 2D image) we need to estimate 3n parameters -a large number of degrees of freedom if the mesh has a reasonable resolution. Fortunately, this can be significantly reduced when dealing with a specific class of shapes such as the face. Here the amount of variation among different instances is much less than when dealing with arbitrary objects, and one can presume that the shapes live in a manifold of dimension d << 3n. Numerous methods have been proposed that exploit this fact using dimensionality reduction techniques over a training dataset [Blanz andVetter, 1999, Vlasic et al., 2005] or by hand-crafting a set of d basic facial shapes that can then be linearly combined [Lewis et al., 2014b]. Not only this reduces the dimensionality, but also provides a strong prior knowledge on which facial meshes are plausible that can greatly benefit tasks such as 3D reconstruction from images or real-time facial animation.

Several steps are needed before one can study the properties and common patterns of the face geometry from a dataset of 3D scans. The first step is to acquire such a dataset from real people, which should ideally contain detailed scans of the geometry and be diverse in terms of age, sex, ethnicity, or facial expressions (Section 2.1). The next step is to register the scans such that they share a common parameterization: each face should contain the same number of vertices and triangulation F , and each three-dimensional point should have the same anatomical meaning (Section 2.2). Once in correspondence the dataset can be rigidly aligned (e.g.using Generalized Procrustes Analysis [START_REF] Davies | Statistical models of shape: Optimisation and evaluation[END_REF]), removing rotation, translation, and optionally scale differences1 such that only variations due to shape remain.

With a registered and rigidly aligned dataset we can now study how each of the vertices vary under different types of deformations. There are several aspects of the face that can be captured by a model. Differences in morphological features give place to variations in identity: the traits that distinguish one person from the other. These will be covered in Section 2.3.1. Another important source of variation is that which occurs when performing different expressions: changes that occur when a single identity is set in motion (Section 2.3.2). A particular kind of models, that we call here decoupled models, allow to capture variations in both identity and expression simultaneously, while taking into account the interactions that occur between these two spaces; these are discussed in Section 2.3.3. Another aspect is the encoding of details, as parameteric models usually struggle with higher-frequency components due to the low-dimensional representation; these are reviewed in Section 2.3.4. Finally, the appearance of the face is also crucial for applications such as realistic rendering and analysis-by-synthesis algorithms. We will not review appearance models here as this work is focused on the geometric aspects of the face -interested readers are referred to [START_REF] Klehm | Recent advances in facial appearance capture[END_REF], [START_REF] Egger | 3d morphable face modelspast, present and future[END_REF].

This chapter is intended only as a brief overview that will serve as background information for the chapters to come. For a more detailed treatment we refer to Brunton et al. [2014b], [START_REF] Egger | 3d morphable face modelspast, present and future[END_REF] regarding data-driven facial shape models, Lewis et al. [2014b], [START_REF] Orvalho | A facial rigging survey[END_REF] for 3D facial animation and [START_REF] Zollhöfer | State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications[END_REF] for the application of 3D face reconstruction from monocular input.

Acquisition

There exists a variety of scanners that can accurately measure the 3D surface of the face. A typical classification divides them into active and passive, where the former acquires data by emitting a signal which is then measured by a sensor, while the latter employ sensing devices alone such as two or more RGB cameras. Each of these has its own strengths and weaknesses, as well as its own noise characteristics. Active systems include laser scanners [START_REF] Levoy | The digital michelangelo project: 3d scanning of large statues[END_REF] which were used for faces in e.g. [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF], structured light scanners [START_REF] Geng | Structured-light 3d surface imaging: a tutorial[END_REF], which were used in e.g. [START_REF] Savran | Bosphorus database for 3d face analysis[END_REF] and time-of-flight devices [START_REF] Hansard | Time-of-flight cameras: principles, methods and applications[END_REF], used for example in Cao et al. [2014a]. The accuracy and speed of these sensors can vary greatly, ranging from accurate at the cost of slower frame rates (e.g. the structured light scanner used in [START_REF] Paysan | A 3d face model for pose and illumination invariant face recognition[END_REF]), to real-time at the cost of lower resolution (e.g.the Kinect sensor [START_REF] Mutto | Time-of-flight cameras and microsoft kinect (TM)[END_REF]).

Passive methods estimate depth by reasoning about the reflected light as captured by one or more sensing devices. A commonly used approach operates by triangulating corresponding pixels from two (e.g. [START_REF] Valgaerts | Lightweight binocular facial performance capture under uncontrolled lighting[END_REF]) or more (e.g. [START_REF] Beeler | High-quality singleshot capture of facial geometry[END_REF][START_REF] Bradley | High resolution passive facial performance capture[END_REF]) RGB cameras, a technique known as multi-view stereo. This allows the acquisition of both texture and depth at high frame rates and are thus particularly well suited for dynamic captures -many publicly available motion data such as [START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF][START_REF] Yin | A high-resolution 3d dynamic facial expression database[END_REF] were acquired using this technique. Although the quality of multi-view stereo methods has lagged behind active sensors for many years, recent work e.g. [START_REF] Beeler | High-quality singleshot capture of facial geometry[END_REF][START_REF] Geng | Structured-light 3d surface imaging: a tutorial[END_REF][START_REF] Wu | High-quality shape from multi-view stereo and shading under general illumination[END_REF][START_REF] Fyffe | Multi-view stereo on consistent face topology[END_REF] has demonstrated that high quality 3D scans of the face can be obtained with purely passive approaches, under well-controlled studio setups.

As opposed to the previously mentioned geometric methods which estimate surface positions, photometric methods are concerned with the estimation of the surface normals, and have been used with both active and passive scanners. This is achieved by analyzing the interaction of the light with the surface, either from one image (a technique called shape-from-shading [START_REF] Horn | Shape from shading[END_REF]) or multiple images showing different illuminations, a technique called photometric stereo [START_REF] Woodham | Photometric method for determining surface orientation from multiple images[END_REF]. Both cases estimate dense per-pixel orientations from which the shape can be recovered by integration [START_REF] Quéau | Normal integration: a survey[END_REF], or combined with depth to improve an initial reconstruction [START_REF] Nehab | Efficiently combining positions and normals for precise 3d geometry[END_REF]. These techniques are suitable for acquiring high frequency details and perform well with the facial shape as they do not require a highly textured surface. The most accurate facial acquisitions are typically obtained with a combination of geometric and photometric methods, e.g. Although much research effort has been dedicated to reduce the quality gap between high-end systems and consumer-grade devices (e.g.RGB or RGB-D cameras), good quality acquisitions still need to be done using expensive systems with complex setups and often costly offline computations. Thus, it is to date difficult and expensive to acquire a large dataset of 3D scans that will allow the use of modern deep learning techniques. However, the facial shape has always attracted much research interest, and as a consequence numerous databases of 3D facial scans have been acquired and made publicly available. For example, BU-3DFE [START_REF] Yin | A 3d facial expression database for facial behavior research[END_REF], Bosphorus [START_REF] Savran | Bosphorus database for 3d face analysis[END_REF], ND-2006 [START_REF] Faltemier | Using a multi-instance enrollment representation to improve 3d face recognition[END_REF] or FaceWarehouse [START_REF] Cao | 3d shape regression for real-time facial animation[END_REF] contain thousands of scans showing multiple identites in multiple expressions; ICT-3DRFE [START_REF] Stratou | Effect of illumination on automatic expression recognition: a novel 3d relightable facial database[END_REF] and Photoface [START_REF] Zafeiriou | The photoface database[END_REF] were captured using photometric systems and thus exhibit high frequency details; and dynamic datasets such as BU-4DFE [START_REF] Yin | A high-resolution 3d dynamic facial expression database[END_REF], D3DFACS [START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF], BP4D-Spontaneous [START_REF] Zhang | Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database[END_REF], B3D(AC) 2 [START_REF] Fanelli | A 3-d audio-visual corpus of affective communication[END_REF] or CoMA [START_REF] Ranjan | Generating 3d faces using convolutional mesh autoencoders[END_REF] open up the possibility of analyzing the facial surface in motion. In this work we study techniques to make use of this large but varied source of information in order to learn richer models of the 3D shape, as well as its motion-related components such as expression.

Registration

Capture systems typically return meshes that are inconsistent with each other: each scan contains a different number of vertices, and each of these vertices may have a different anatomical meaning (for example the i-th vertex of one scan might be located at the tip of the nose, while the i-th vertex of another scan might be located in the eye corner). This unordered representation is clearly ill-suited for studying shape variations, as we do not know how each point really varies from one shape to the next. In order to solve this we need to put the data in correspondence, a process that is sometimes referred to as registration.

Following the definition of Van Kaick et al. [2011], given a set of input shapes S 1 , S 2 , . . . , S n the problem of finding correspondences is defined as that of finding a meaningful relation R between the shapes. Correspondences can be sparse, establishing a connection between a few, typically distinctive points; or dense, where the correspondence is defined for all the primitive elements of the shape (e.g.all the vertices of a mesh). Sparse correspondence for 3D faces are commonly referred to as landmarks, and are defined in terms of morphologically relevant and distinguishable features such as the eye corners or the tip of the nose. [START_REF] Gilani | Deep, dense and accurate 3d face correspondence for generating population specific deformable models[END_REF]. We are mostly concerned here with dense correspondences for the 3D face, as this will allow to study the properties of entire facial surfaces. Example works include [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF], Amberg et al. [2007], [START_REF] Li | Global correspondence optimization for non-rigid registration of depth scans[END_REF] to name a few, and will be elaborated in the following paragraphs.

The nature of the relation R also gives rise to different types of algorithms. Rigid registration methods study the case where R is a rigid transformation: the goal here is to find a global rotation R ∈ R 3×3 and translation t ∈ R 3 that 2.2. REGISTRATION 13 will minimize some distance function between two shapes X and Y . A widely used solution is the Iterative Closest Point (ICP) algorithm [START_REF] Besl | Method for registration of 3-d shapes[END_REF], which iterates between selecting the closest point as a matching correspondence, and computing the optimal rigid transformation for these correspondences by minimizing the point-to-point error function:

arg min R∈R 3×3 ,t∈R 3 p i ∈X ||q i -(Rp i + t)|| 2 2 , s.t.R T R = I, det(R) = 1 (2.1)
where q i ∈ Y is the closest point to (Rp i + t) given the current estimate for R, t. The closest point search can be accelerated by using appropriate data structures such as a kd-tree [START_REF] Bentley | Multidimensional binary search trees used for associative searching[END_REF] or by inverse calibration [START_REF] Blais | Registering multiview range data to create 3d computer objects[END_REF]]. An alternative to Equation 2.1 is to minimize the point-to-plane function [Chen andMedioni, 1992, Low, 2004], which considers instead the distance between (Rp i + t) and the tangent plane to the closest point:

arg min R∈R 3×3 ,t∈R 3 p i ∈X n T q i (Rp i + t -q i ) 2 , s.t.R T R = I, det(R) = 1 (2.2)
with n q i the normal vector at the closest point q i ∈ Y . Both cases will converge to a local solution and as such require a good initialization, although it has been shown that the point-to-plane formulation will typically converge faster. Numerous variants have been proposed that tackle different aspects of the algorithm, see e.g. [START_REF] Castellani | 3d shape registration[END_REF].

Non-rigid registration considers the case where R can be an arbitrary relation; a significantly more complex problem with a larger solution space. A common approach when working with a specific class of shapes is to do template fitting: a pre-computed mesh (e.g.an artist-quality facial mesh) is warped towards each of the scans, automatically establishing dense correspondences among all shapes through the common template. Non-rigid variants for this purpose differ mostly in the choice of deformation parameterization. This includes per-vertex affine transformations [Allen et al., 2003, Amberg et al., 2007], vertex displacements [START_REF] Weise | Face/off: Live facial puppetry[END_REF], Salazar et al., 2014], free-form deformations (FFD) [START_REF] Huang | Establishing local correspondences towards compact representations of anatomical structures[END_REF][START_REF] Wang | High resolution acquisition, learning and transfer of dynamic 3-d facial expressions[END_REF], thin-plate splines (TPS) [START_REF] Chui | A new algorithm for non-rigid point matching[END_REF][START_REF] Patel | 3d morphable face models revisited[END_REF][START_REF] Hutton | Dense surface point distribution models of the human face[END_REF], deformation graph [START_REF] Li | Global correspondence optimization for non-rigid registration of depth scans[END_REF], physics-based models [START_REF] Passalis | Evaluation of 3d face recognition in the presence of facial expressions: an annotated deformable model approach[END_REF], gaussian mixture models [START_REF] Myronenko | Non-rigid point set registration: Coherent point drift[END_REF] and gaussian process deformation model [START_REF] Gerig | Morphable face models-an open framework[END_REF]].

An alternative, used particularly for faces, is to perform the registration in the 2D domain by leveraging the associated UV texture map. [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF] proposed the use of optical flow on the texture maps of 200 subjects in neutral expressions, which worked well on their database of uniform ethnicities. To handle more variations, [START_REF] Patel | 3d morphable face models revisited[END_REF] used instead a TPS that interpolates manually annotated landmarks in the UV domain. Manual annotations were replaced by an Active Appearance Model (AAM) in [START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF] and Cheng et al. [2017a]. The work of [START_REF] Booth | Large scale 3d morphable models[END_REF] compared the non-rigid ICP approach of Amberg et al. [2007] with the two UV-based approaches (TPS and optical flow), and showed superior results for non-rigid ICP, in the context of automatic registration of a large number of neutral 3D faces. [START_REF] Dai | A 3d morphable model of craniofacial shape and texture variation[END_REF] proposed instead to combine the two alternatives, initially fitting a template using coherent point drift [START_REF] Myronenko | Non-rigid point set registration: Coherent point drift[END_REF] and afterwards refining using optical flow.

Non-rigid registration can also be aided by the use of a statistical model, such as those outlined in Section 2.3. This was considered in e.g. Albrecht et al. [2008], [START_REF] Schneider | Fast nonrigid mesh registration with a data-driven deformation prior[END_REF], Cheng et al. [2017b] which used a linear model of shape or expression, Brunton et al. [2014a], Bolkart and Wuhrer [2015a] that used a multilinear model, and [START_REF] Lüthi | Gaussian process morphable models[END_REF] that considered a gaussian process model. Note how this is a chicken-and-egg problem: while statistical models can be useful priors for registration, in order to build such a model one requires a training set of registered meshes in the first place. One solution is to use a bootstrapping approach as considered in e.g. [Blanz andVetter, 1999, Li et al., 2017]. Here an initial registration is used to build a statistical model which in turn is used to improve the registrations, iterating for a few times. Another option is to jointly optimize for the registration and the model, a harder problem that was addressed in Bolkart If the same template is used to track multiple motions then correspondences are established both in space and time, expanding the scope of applications. We contribute in Chapter 4 an approach for efficiently registering a large number of 3D facial scans in motion; a more detailed review of the topic will be presented there.

Models

Once registered, each face can be represented as a vector x = (x 1 , y 1 , z 1 , . . . , x n , y n , z n ) ∈ R 3n with n the number of vertices. We are now in position to analyze a population of 3D faces {x 1 , . . . , x m } and study how the surface varies according to the different factors that influence the shape. The next subsections will elaborate on the identity, expression, and high frequency detail models that were built for this purpose.

Modeling Identity Variations

By far the most common approach for modeling identity variations is to use Principal Component Analysis (PCA) [START_REF] Jolliffe | Principal Component Analysis and Factor Analysis[END_REF], assuming a normal distribution of the shapes.

The process of building such a model begins by removing the average face x = 1 m m i=1 x i from each sample, i.e.x i = x ix. The now centered data is assem-

MODELS

bled into the columns of a matrix X ∈ R 3n×m , and an eigenvalue decomposition of the covariance matrix is performed, e.g.by singular value decomposition (SVD) of X. The eigenvectors v 1 , . . . , v p associated with the non-zero eigenvalues provide an orthogonal basis for a new vector space that is decorrelated and is typically of much lower dimensionality compared to R 3n , while the eigenvalues λ 1 , . . . , λ p contain the variance of the data in the direction of each eigenvector. The dimensionality can thus be further reduced by keeping the d < p eigenvectors with largest eigenvalues, such that a certain percentage of variance is retained. This process yields a generative model in which novel faces can be synthesized from a vector w ∈ R d by:

x(w) = x + Bw, (2.3)
where B ∈ R 3n×d contains the d principal eigenvectors and w is the low dimensional representation of x(w) ∈ R 3n . Thanks to the Gaussian assumption we can easily estimate the likelihood of each parameter w i in w, knowing that

P(w i ) = exp[-1/2( w i √ λ i ) 2
]. This can be leveraged for regularization, as was done in e.g. [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF] The seminal work of [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF] was the first to propose the use of this model for a set of densely corresponded faces. Their so-called 3D Morphable Model (3DMM) consisted of two independent PCA models, one for shape and one for texture, trained from 200 scans of Caucasian subjects. An improved version was made publicly available in [START_REF] Paysan | A 3d face model for pose and illumination invariant face recognition[END_REF], known as the Basel face model. This is still widely used to date -see e.g. [Richardson et al., 2016, Tran et Equation 2.3 denotes a global model, in that a single parameter vector generates the entire shape. Local models of identities have also been explored, where several sub-models encode different regions of the face. This allows more diversity in the generation process and better generalization to unseen data, but the representation is less compact and harder to fit to ambiguous input such as 2D images. The work of [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF] already proposed to build PCA models for five regions of the face, and several other works followed this idea, e.g. ter Haar and Veltkamp [2008], De Smet and Van Gool [2011], [START_REF] Brunton | Wavelet model-based stereo for fast, robust face reconstruction[END_REF].

A drawback of PCA models is that they can only represent shapes that are linear combinations of the training data, and hence require a large training set with sufficient coverage. [START_REF] Booth | A 3d morphable model learnt from 10,000 faces[END_REF] built a model from almost 10.000 identities, showing that indeed a larger set can yield models of higher quality. Alternatively, [START_REF] Lüthi | Gaussian process morphable models[END_REF] model shape variations as a Gaussian Process, which allows to compensate for the lack of data with manually designed CHAPTER 2. BACKGROUND kernels. In this thesis we approach this by learning models from a large scale dataset acquired from multiple sources as explained in Section 2.1.

Modeling Expression Variations

Modeling expressions is considerably harder as the movement induced by the muscles on the skin can be highly non-linear.

In the area of facial animation the use of parametric models was introduced by [START_REF] Parke | A Parametric Model for Human Faces[END_REF], and since then many approaches have been proposed for manipulating a 3D face based on a few parameters. Frequently employed in the film industry is the blendshape model [Lewis et al., 2014b], used for example to create Gollum in The Lord of the Rings or Benjamin Button in The Curious Case of Benjamin Button. The blendshape model is based on a set of d meshes in correspondence showing a single subject in multiple expressions, and generates new faces as a linear combination of these:

x(w) = Bw, (2.4)
where B ∈ R 3n×d contains each expression mesh (the blendshapes) on the columns. This is very similar to the PCA model of Equation 2.3 and follows the same goal: to reduce the dimensionality and limit the range of deformations allowed. Yet, there is an important difference: unlike PCA the bases are not orthogonal with each other, but carry instead semantic meaning. For example, the first basis might generate a smile and the second one a blink. This makes it easier for artists to manipulate and animate the facial mesh, modifying expressions by simply "sliding" the components of the parameter vector w. Such semantic control is not possible with PCA, which in exchange offers better compactness since the basis vectors have no redundancy.

The model can also be expressed as displacements from a selected neutral face of the subject x 0 , known as the delta blendshape model:

x(w) = x 0 + Bw, (2.5)
where B now contains displacements from the neutral face, i.e.(x ix 0 ) on each column. This allows a more localized control if the blendshapes are focused on a region, and to easily replace the identity by simply modifying the neutral face x 0 . In order to build the basis vectors the blendshapes can be either manually modeled by an artist (usually following the Facial Action Coding System (FACS) [START_REF] Ekman | Facial action coding system: a technique for the measurement of facial movement[END_REF] 2 ), scanned from an actor (e.g. [START_REF] Weise | Face/off: Live facial puppetry[END_REF]), transferred from the expressions of a different actor [START_REF] Li | Example-based facial rigging[END_REF] or automatically discovered [START_REF] Bouaziz | Online modeling for realtime facial animation[END_REF][START_REF] Li | Realtime facial animation with on-the-fly correctives[END_REF]].

The blendshape model has been widely adopted thanks to its simplicity and semantic parameterization, but can present artifacts due to its linear nature. In particular, it is easy to obtain unrealistic shapes when activating multiple blendshapes at the same time, and care must be taken when combining them so that they do not interfere with each other. To address this some authors proposed to include bilinear "correction" shapes b i,j [Lewis et al., 2014b] such that x(w) = Bw + w i w j b i,j , which are only considered when two blendshapes are simultaneously non-zero. Nonlinear corrections were also considered, e.g. [START_REF] Seol | Weighted pose space editing for facial animation[END_REF], as well as the combination of blendshape models with physical simulation [Barrielle et

Modeling Identity and Expression Variations

Models that can simultaneously encode variations due to identity and expression have clear advantages, as they can generalize to a larger scope of scenarios.

Although it is possible to learn a single PCA model where both novel identities and expressions are synthesized from a unique latent vector, this would require a large amount of training data to properly generalize to unseen instances [START_REF] Yang | Expression flow for 3d-aware face component transfer[END_REF] More importantly, such an entangled representation does not allow to independently control the generation of identities and expressions, excluding applications that take benefit of this such as animation, tracking and recognition.

The most straightfoward and commonly adopted way of combining these two spaces is by addition: The additive model of Equation 2.6 assumes that the expression displacements are the same for all identities, ignoring the fact that the way an expression is performed depends on the underlying shape. Bilinear [Tenenbaum andFreeman, 2000, Chuang et al., 2002] and multilinear [START_REF] Vasilescu | Multilinear analysis of image ensembles: Tensorfaces[END_REF]Terzopoulos, 2002, Vlasic et al., 2005] models address this through the use of factorization techniques.

x(w id , w exp ) = f id (w id ) + f exp (w exp ), (
Multilinear models are mappings from a set of latent variables to the output shape, such that the different factors modulate each other multiplicatively. The model becomes linear when all but one of the factors is held constant.

Let us first recall the basic concepts of multilinear algebra, on which the model is based 3

. A N -th order tensor X ∈ R d 1 ו••×d N is a multi-dimensional
array that is indexed by N integers, generalizing the concept of vectors (firstorder tensors) and matrices (second-order tensors) to higher dimensions. The higher order analog of rows and columns is called a fiber, obtained by fixing all but one of the indices. For example, given a 3-rd order tensor Y ∈ R d 1 ×d 2 ×d 3 , the vector

y i,:,k ∈ R d 2 , with i ∈ [1..d 1 ], k ∈ [1..d 3 ] is a mode-2 fiber of Y . A tensor X ∈ R d 1 ו••×d n ו••×d N can
be converted into a matrix through a process called matricization, unfolding or flattening. The mode-n matricization of X is a matrix

X (n) ∈ R d n ×(d 1 d 2 ...d n-1 d n+1 ...d N )
obtained by arranging the mode-n fibers as columns. The mode-n product between a tensor X and a matrix U ∈ R K×d n , denoted as

X ′ = X × n U, is a tensor X ′ ∈ R d 1 ו••×Kו••×d N obtained
by replacing all mode-n fibers x by Ux.

To build a multilinear model of 3D faces the training set {x 1 , . . . , x m } is assembled into a tensor, where one of the dimensions represents the vertices and the rest is arranged according to the modeled factors of variation. For example, a training set of m i identities each performing m e expressions can be arranged as a tensor Y ∈ R 3n×m i ×m e such that the fiber Y :,i,j contains the vertices of the i th identity and e th expression, and the fiber Y k,:,e contains the k-th vertex of all identities when performing the e-th expression.

Higher-Order SVD (HOSVD) [De Lathauwer et al., 2000a] generalizes the concept of SVD to tensors, and can be applied to the centered version X ′ of X 3. A more detailed review can be found in [START_REF] Kolda | Tensor decompositions and applications[END_REF].

to obtain a decomposition of the form:

X ′ ≈ C × 1 U 1 × 2 U 2 . . . × N U N (2.7)
where U i ∈ R d i ×p i , p i ≤ d i are orthogonal matrices, and C ∈ R p 1 ו••×p N , called the core tensor, models the interaction between the different vector spaces spanned by U i . As with PCA, the training tensor is first centered such that X ′ :,i 1 ,...,i N = X :,i 1 ,...,i Nx, with x the average vertices among the training set. Since the goal is to model vertices as a function of parameters for the different factors (e.g.identity, expression), multilinear models of the face typically do not factor along the mode that corresponds to the vertices [START_REF] Vlasic | Face transfer with multilinear models[END_REF], such that

X ′ ≈ M × 2 U 2 . . . × N U N .
(2.8) A main disadvantage of multilinear models is the need to build a full training data tensor, requiring very careful capture conditions in which e.g.all identities perform all of the expressions. Already the work of [START_REF] Vlasic | Face transfer with multilinear models[END_REF] approached this through the use of Probabilistic PCA for tensor completion. [START_REF] Bolkart | A robust multilinear model learning framework for 3d faces[END_REF] proposed a method to robustly learn a model considering missing data, corrupt data, and incorrect semantic correspondence. Recent work has also explored fully unsupervised approaches: Wang et al.

A
[2017] recover the core tensor using an alternating least squares approach, while [START_REF] Wang | An adversarial neuro-tensorial approach for learning disentangled representations[END_REF] train a deep autoencoder with multilinear structure on three modalities. In Chapter 3 we will present a novel deep autoencoder architecture that allows to train a multilinear model from incomplete data tensors, allowing to handle significantly larger training sets.

Nonlinear models of identity and expression have also been proposed. [START_REF] Wang | High resolution acquisition, learning and transfer of dynamic 3-d facial expressions[END_REF] use manifold learning techniques to model the different expression styles of individuals. More recently, deep learning approaches have allowed to learn such non-linear models in a scalable manner, including the CHAPTER 2. BACKGROUND fusion network of [START_REF] Jiang | Disentangled representation learning for 3d face shape[END_REF], and the work developed in the context of this thesis, which will be introduced in Chapter 5.

Modeling Details

While useful for handling complex problems, low-dimensional parametrizations often restrict the deformations to its coarsest features: mid-scale and highfrequency details are mostly lost in the process. Only recently researchers began to explore a low-dimensional space that can still encode higher-frequency geometry, again thanks to advances in deep learning, e.g. [Tran et al., 2018[START_REF] Tran | Towards high-fidelity nonlinear 3d face morphable model[END_REF]. Still, the usual approach is to complement the coarse shape from a parametric model with one or two extra layers that encode out-of-model details. This can include a medium-scale layer for wrinkles and a fine-scale layer for mesoscopic geometry, and the deformations can be represented directly on the (possibly refined) mesh [START_REF] Garrido | Reconstructing detailed dynamic face geometry from monocular video[END_REF][Garrido et al., , 2016a When available, a database of high-resolution scans can be used to learn a generative model of wrinkle formation. Such an approach was followed for example by [START_REF] Golovinskiy | A statistical model for synthesis of detailed facial geometry[END_REF] Since medium-and fine-scale layers correlate with the deformations of the coarser shape, several works proposed to build a mapping between the coarse shape parameters and the finer layers. This was considered for example in [START_REF] Ma | Facial performance synthesis using deformation-driven polynomial displacement maps[END_REF] through the use of deformation-driven polynomial displacement maps, [START_REF] Bickel | Pose-space animation and transfer of facial details[END_REF], [START_REF] Ichim | Dynamic 3d avatar creation from hand-held video input[END_REF] that train a mapping between edge strain and displacement or normal maps, and Garrido et al. [2016a], Yang et al.

[2020] by regressing displacements from the coefficients of a parametric model.

Fine-scale details have been traditionally obtained using photometric approaches, e.g.shape-from-shading (SfS) [START_REF] Horn | Shape from shading[END_REF] in the case of a single image. This recovers a normal map which can then be integrated [START_REF] Quéau | Normal integration: a survey[END_REF] or used to enhance a pre-computed depth map using methods such as [START_REF] Nehab | Efficiently combining positions and normals for precise 3d geometry[END_REF]. While SfS normally relies on generic priors (such as smoothness or integrability), face-specific models can also be leveraged to alleviate some of its intrinsic limitations. For example, [START_REF] Smith | Recovering facial shape using a statistical model of surface normal direction[END_REF] constrain the output to lie in the space of a statistical model of surface normals. The shading-based refinement methods of Valgaerts et

A Multilinear Autoencoder for 3D Face Model Learning from Large Datasets

Generative models of the 3D facial shape are extensively used in a number of fields that include computer vision -where they serve as priors for ill-posed problems such as 3D reconstruction from images-, computer graphics -where they can be used for the animation of digital characters-, and medical image analysis, where they can be used to distinguish normal from pathological structures. They proved to be benefitial for these tasks as they provide a lowdimensional parameterization for an otherwise complex problem, simplifying both synthesis and inference tasks.

A special kind of models are those that can decouple the changes due to natural factors of variation, for instance identity, expression or even age in the case of faces. Such models provide an (ideally) independent parameterization for each of the factors, incorporating a degree of semantic control that enables numerous applications. For example, a decoupled parameterization allows to transfer the expression from one digital character to another one by simply replicating the expression vector, potentially preserving the target's individuality in the performance of the expression [START_REF] Vlasic | Face transfer with multilinear models[END_REF][START_REF] Dale | Video face replacement[END_REF]. Knowing that the identity is constant allows for strong regularization in tracking [START_REF] Dale | Video face replacement[END_REF][START_REF] Shi | Automatic acquisition of high-fidelity facial performances using monocular videos[END_REF] or learning [START_REF] Sanyal | Learning to regress 3D face shape and expression from an image without 3D supervision[END_REF], Tewari et al., 2019] tasks, a property that will also be leveraged in Chapter 4. Other applications include 3D face and expression recognition [START_REF] Mpiperis | Bilinear models for 3-d face and facial expression recognition[END_REF], Liu et al., 2018], expression rectification [START_REF] Yang | Expression flow for 3d-aware face component transfer[END_REF], automatic blendshape generation [START_REF] Cao | 3d shape regression for real-time facial animation[END_REF], 2014a, 2016, Wang et al., 2020] and synthetic data generation for machine learning applications [START_REF] Han | Deepsketch2face: a deep learning based sketching system for 3d face and caricature modeling[END_REF][START_REF] Shamai | Synthesizing facial photometries and corresponding geometries using generative adversarial networks[END_REF].

Multilinear models were proposed with this task in mind. The model, first employed for 3D faces by [START_REF] Vlasic | Face transfer with multilinear models[END_REF], extends the widely used 3D Morphable Model [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF]] by assembling the training dataset as a tensor that is organized based on the specified factors, and by replacing PCA with a tensor decomposition method, typically Higher-Order SVD (HOSVD)1 . This results in a compact model where the different latent vectors influence only one factor (e.g.identity or expression), and the interaction between these is modeled in a multiplicative manner.

CHAPTER 3. A MULTILINEAR AUTOENCODER FOR 3D FACE MODEL LEARNING FROM LARGE DATASETS

Multilinear models have however one major drawback: in order to build them one needs to assemble the training data into a complete tensor. In the case of faces showing different identities and different expressions, this means that each subject must be scanned performing each of the training expressions. This is clearly not scalable and prohibits the use of scans for which the full set of expressions is not present.

The problem is more pronounced when considering publicly available 3D datasets, whose combination can be large and whose sources and capture protocols can be varied, e.g. [START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF], [START_REF] Savran | Bosphorus database for 3d face analysis[END_REF], [START_REF] Yin | A high-resolution 3d dynamic facial expression database[END_REF]. Considering the effort that is involved in capturing accurate scans of the 3D face, these datasets are a valuable source of information and the ability to learn patterns from them can potentially bring forward more powerful models. Yet, there is evidently no control on the type of data that was captured, and as a consequence one cannot build a complete data tensor to train a multilinear model. The large number of scans also demands for computationally efficient and scalable methods, as well as fully automatic registration approaches that could hence result in partially corrupted training data (either in the geometry or in the semantics). The question is then how to leverage all available training data to build richer multilinear models -without assuming a complete training data tensor, requiring little to no pre-processing, and taking into account that the scans may be corrupted by geometric noise and/or erroneous labels.

This chapter takes a step towards these goals by proposing a novel framework that can learn a multilinear model of the 3D face from such data. This is achieved through a new autoencoder architecture that combines a CNN-based encoder -thus assuring robustness to corrupt and incomplete data-with a multilinear decoder, that can effectively decouple the shape variations over data attributes. It additionally inherits the benefit of scalability that is characteristic of autoencoders. Moreover, using a multilinear model as decoder rather than a generic network allows to explicitly take advantage of redundant training data showing the same factor, and to effectively decouple shape variations in the learned representation.

The proposed approach builds on recent works that use deep neural networks for 3D face modeling. In particular, two of them [START_REF] Laine | Production-level facial performance capture using deep convolutional neural networks[END_REF][START_REF] Tewari | MoFa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction[END_REF] have successfully explored the combination of a CNNbased encoder with a linear generative model as decoder for the task of 3D reconstruction of faces from 2D images. We follow a similar strategy however, unlike [START_REF] Tewari | MoFa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction[END_REF] the decoder is learned with the rest of the network, and unlike [START_REF] Laine | Production-level facial performance capture using deep convolutional neural networks[END_REF], our learned model generalizes to various factors captured for different subjects.

The method takes as input a set of 3D face scans annotated with labels for each factor, e.g.identities and expressions, and provides: (i) A multilinear model, which is able to accurately reconstruct the training data and decouples shape changes due to different factors; (ii) A trained autoencoder capable of regressing from any 3D face scan to the registered model, thus allowing to efficiently compute correspondences for new data.

Our model performs favorably against other recent approaches that learn multilinear face models from incomplete training data tensors, namely [START_REF] Bolkart | A robust multilinear model learning framework for 3d faces[END_REF] and [START_REF] Wang | Learning the multilinear structure of visual data[END_REF]. In particular, we show experimentally that the proposed method is capable of building rich models which achieve a better decoupling of factors. This is demonstrated by a classification rate of synthetically transferred expressions that is over 5% higher than competing methods. While the experiments focus on identity and expression attributes, our formalism readily generalizes to other factors as well.

Related Work

There is an extensive amount of work on 3D human face modeling, many of which were reviewed in Section 2.3. Here we focus the discussion on works that are most closely related to the proposed approach.

Generative modeling of 3D faces Linear models were first introduced to model face shape in neutral expression along with appearance information in [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF], and later extended to include expression change as a linear factor in Amberg et al. [2008]. These linear models are often called 3D morphable models (3DMM), and have recently been learned from large training sets [START_REF] Booth | A 3d morphable model learnt from 10,000 faces[END_REF] and from craniofacial scans [START_REF] Dai | A 3d morphable model of craniofacial shape and texture variation[END_REF]]. These models do not account for correlations of expression and identity spaces.

Multilinear models were introduced to independently represent the influence of different factors on the facial shape, which allows for expression transfer [START_REF] Vlasic | Face transfer with multilinear models[END_REF]. They were later used to edit 2D images and videos with the help of 3D face reconstructions [START_REF] Dale | Video face replacement[END_REF], Cao et al., 2014b]. FaceWarehouse [Cao et al., 2014b] is a popular publicly available multilinear 3D face model. While multilinear models effectively decouple shape variations due to different factors, they require carefully acquired training data where each subject is captured in every factor. [START_REF] Li | Learning a model of facial shape and expression from 4D scans[END_REF] introduced a generative model learned from a large collection of 3D motion sequences of faces. Pose changes due to skeletal motion is modeled using a skinning approach, while shape changes due to identity, expression, and pose correction are modeled as linear factors similar to 3DMM. Interestingly, they note that it is an open problem to extend tensor-based multilinear models to handle dynamic training data.

We take a step in this direction by deriving an efficient method to learn a multilinear model from an incomplete tensor of training data, that effectively decouples factor effects.

Learning a multilinear model from partial or noisy data Traditionally, multilinear models are learned by assembling a dataset into a tensor and performing tensor decomposition [De Lathauwer et al., 2000a]. This requires each training LEARNING FROM LARGE DATASETS face to be present in all factors. Furthermore, noise in the data, registration or labeling affect the quality of the model. While tensor completion methods can be used to solve the problem of incomplete data, they do not scale well in practice, especially if the tensor is dense as in our case [START_REF] Song | Tensor completion algorithms in big data analytics[END_REF].

Two recent methods were proposed to address these problems. [START_REF] Bolkart | A robust multilinear model learning framework for 3d faces[END_REF] introduced a groupwise optimization to handle both missing and noisy data and was shown to outperform tensor completion methods. However, the approach is computationally costly and hence does not scale to large datasets with high dimensionality in two or more factors. Another work proposed an unsupervised method to compute a multilinear model from partial data [START_REF] Wang | Learning the multilinear structure of visual data[END_REF]. While computationally more efficient, it uses a non-standard tensor decomposition that leads to a generative model that does not fully decouple the modes. We will compare to both methods in Section 3.4. [2017] our model is not person-specific and can generalize to arbitrary subjects and expressions. Since then the approach has been adopted by several works. In particular, the simultaneous training of a generative model and a regressor from RGB images holds great potential, as it allows to build fully-unsupervised 3D models from large-scale 2D datasets. In this context, Tewari et al.

Deep neural networks for 3D face modeling

Overview

The goal of this work is to learn a generative model of faces from a set of labeled 3D scans, that are possibly corrupted by both geometric noise and label errors. To achieve this, we propose an autoencoder architecture with a CNN-based encoder and a multilinear model-based decoder, as illustrated in Figure 3.1 and detailed in the following section.

Input Data

To train the autoencoder we consider 3D face scans showing variations in different factors, e.g.identity and expression, along with the corresponding labels. Not all combinations of factors are required in the input scans, and part of the training data can be without labels. The input scans are first registered, enabling reconstruction errors between the output meshes and the input scans to be estimated in a consistent way. These registrations need not be precise, as the global nature of training will ensure that isolated errors are averaged out.

Encoder

The CNN encoder maps each 3D face scan into a low-dimensional representation that decouples the influence of the different factors on the final shape. Extending CNNs to unorganized 3D geometric data is an active field of research (see e.g. [START_REF] Wu | A comprehensive survey on graph neural networks[END_REF]) and beyond the scope of this work. Instead, we take advantage of the fact that 3D faces can be mapped onto 2D images for which regular CNNs apply. Hence, the first step of the encoder is to project input 3D scans into grayscale images that contain depth information. The remainder of the encoder consists of a convolutional neural network followed by fully connected layers, which transform the depth image into a d-dimensional vector with the concatenated coefficients for each mode.

Decoder

The multilinear decoder splits the output of the encoder according to the factors, applies mode-n multiplication between these latent vectors and the core tensor, and adds a previously computed average face, as normally done with multilinear models (see Sections 3.3.1 and 2.3.3). The output of the LEARNING FROM LARGE DATASETS decoder are 3D vertex coordinates that, combined with the connectivity of the average face, define a 3D mesh. The key here is that all of these operations can be written as layers of a neural network, thus allowing to update the values of the core tensor along with the rest of the network.

Training During training both the CNN encoder and the multilinear decoder are optimized. In addition to a generative loss that accounts for reconstruction errors, we optimize a latent loss that measures whether input faces with the same labels are mapped onto close-by points in parameter space, hence enforcing shape variations to be decoupled with respect to the different factors. The space that models face variations is large compared to the available training data and a good initialization is thus required. To this aim, both encoder and decoder are pre-trained, as detailed in Section 3.3.3.

Once the autoencoder has been trained, the multilinear model can be extracted from the decoder and treated as a classic multilinear model. In addition, the trained encoder can be used to regress any 3D scan to the model, thereby allowing to efficiently register new data.

Multilinear Autoencoder

We now describe the proposed autoencoder architecture that allows to learn k modes of variation in the input face data through a multilinear model.

Multilinear Model as a Decoder

In a multilinear model a face is represented by a set of vectors {w 2 , . . . , w k+1 },

w j ∈ R d j ,
where k is the number of linear modes attached to faces in the model. Let x ∈ R 3n be the vector of 3D coordinates associated with the n vertices of a face mesh, then the multilinear model relates the latent k factors w j with the 3D face x by:

x = x + M × 2 w T 2 × 3 w T 3 . . . × k+1 w T k+1 , (3.1)
where x is the average training face, M ∈ R 3n×d 2 ×d 3 ×...d k+1 is a tensor that combines the linear modes w j called the core tensor, and × j is the product of M and a vector along mode j. The model is therefore represented by the entries of M in addition to the set of coefficients w (i) j for the i-th face and the j-th factor in the training set.

An interesting property of tensors states that [START_REF] Kolda | Tensor decompositions and applications[END_REF]]

Y = C × 1 A 1 × 2 A 2 . . . × k+1 A k+1 ⇔ Y (n) = A n C (n) A k+1 ⊗ . . . ⊗ A n+1 ⊗ A n-1 ⊗ . . . A 1 , (3.2)
where Y and C are tensors, ⊗ is the Kronecker product, A n are matrices of appropriate dimensions, and C (n) , Y (n) are the matricizations of C and Y containing

MULTILINEAR AUTOENCODER

29

the mode-n fibers as columns. In particular, for the model vector coefficents w j , a training face mesh x, and mode n = 1,

x = x + C × 1 w T 1 × 2 w T 2 . . . × k+1 w T k+1 ⇔ x = x + w T 1 C (1) w T k+1 ⊗ . . . ⊗ w T 2 . (3.3)
Recall that the vector w 1 corresponds to the vertices factor, and as such it is "absorbed" by the core tensor of the model (see Section 2.3.3). Hence,

M = C × 1 w 1 ⇔ M (1) = w T 1 C (1)
, and thus Equation 3.1 can be written as

x = x + M (1) w T k+1 ⊗ . . . ⊗ w T 2 . (3.4)
By writing the transformation M (1) (⊗ 2 j=k+1 w j ) as layers of a neural network, we can refine the multilinear model M at the same time we optimize for the reconstructions. After training, it suffices to recover M (1) from the network and fold it back into a tensor M to obtain the new multilinear model.

Learning the Multilinear Model

The training process seeks to obtain good reconstructions of the data, while at the same time decoupling the latent representation with respect to the factors of variation. Hence, we will use two loss functions: a geometric loss that measures the reconstruction error, and a latent loss that softly evaluates how decoupled the latent space is, by measuring how close two embeddings with the same label are.

Generative loss Given a training set X of faces, the loss of a multilinear model M over a mini-batch X b ⊆ X is measured as the average error between the reconstructions of the model and the observed faces x i :

L G = 1 |X b | x i ∈X b x i -x + M × 2 w (i) 2 . . . × k+1 w (i) k+1 2 2 , ( 3.5)
or equivalently (Eq. 3.4)

L G = 1 |X b | x i ∈X b x i -x + M (1) 2 ⊗ j=k+1 w (i) j 2 2 .
(3.6)

Note that Equation 3.6 is not a decomposition of the data tensor, but a soft constraint that allows to represent a given label in mode j by different coefficients w (i) j for different faces x i . This can be an advantage when the labeling is not trust-worthy, allowing for flexibility in the factor separation.

Latent loss

We observed that a simple reconstruction loss is not sufficient to ensure a decoupled space, as originally guaranteed by the tensor decomposition. This is expected, since Equation 3.6 does not evaluate the coefficients CHAPTER 3. A MULTILINEAR AUTOENCODER FOR 3D FACE MODEL LEARNING FROM LARGE DATASETS w j directly but the reconstruction they yield, therefore allowing the output vertices to be arbitrarily affected by any mode.

To overcome this, we define a loss function that softly constrains the latent parameters. Given a training batch X b , for each mode j we consider the labeled subset W

(j) lbl = {w (1) j , . . . , w (m) j }, m ≤ |X b |,
of mode-j latent weights in the batch, obtained after encoding X b ; for example, all the batch weights that have an "expression" label associated with it. Let

W(i) j = {w (i 1 ) j , w (i 2 )
j , . . .} be the set of all mode-j coefficients in the full training dataset that share the same label in mode j as w

(i) j ∈ W (j)
lbl (for example, all the training faces that were labeled "happy"). Then the function writes:

L (j) L = 1 |W (j) lbl | w (i) j ∈W (j) lbl 1 | W(i) j | w (p) j ∈ W(i) j w (i) j -w (p) j 2 2 , ( 3.7)
where the average over coefficients accounts for very different sizes of the sets

W (i) j .
The latent loss is then the sum of the loss of each mode:

L L = k+1 j=2 L (j) L . (3.8)
Note that the loss is calculated on each batch over the full training set, and thus the gradient must also be globally computed as:

∂ ∂w (i) j L (j) L = 2 w (p) j ∈ W(i) j 1 | W(i) j | w (i) j -w (p) j -2 w (p) j ∈ W(i) j 1 | W(p) j | w (p) j -w (i) j = 2 w (p) j ∈ W(i) j 1 | W(i) j | w (i) j -w (p) j + 2 w (p) j ∈ W(i) j 1 | W(p) j | w (i) j -w (p) j = 4 1 | W(i) j | w (p) j ∈ W(i) j w (i) j -w (p) j .
(3.9)

Here, the second term accounts for the fact that w (i) j will appear in the set W(p) j , and the last line considers

| W(p) j | = | W(i) j |.

Architecture

CNN Encoder

The encoder transforms the 3D face input data into a vector w ∈ R d 2 +•••+d k+1 that contains the concatenated model coefficients, i.e.the latent parameters of the face. The first layer of the network takes as input a 3D scan and converts it into a 2D image that encodes heights from a fixed plane. The regression from the 2D heightmap to the model coefficients is implemented using a ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF] which reduces the image to a 256-dimensional vector, after which three fully-connected layers perform the regression towards the coefficient vector w of the specified dimensions.

Multilinear Decoder

The multilinear decoder takes as input the vector w, which is seen as a concatenation of mode coefficients w = {w 2 , w 3 , . . . , w k+1 }, and transforms it into 3D vertex coordinates by performing mode multiplications with the core tensor. As explained in Section 3.3.1, this operation can be written as the product between the matricized version of the tensor M (1) and the Kronecker product of each mode coefficient (Equation 3.4). Therefore, in order to learn the parameters of the core tensor M we implement each of these operations as a layer in the network, and allow the linear module M (1) to be optimized with the rest of the parameters. This way we benefit from the capacity of neural networks to robustly summarize the representative aspects of an entire dataset, and from the associated optimization machinery to find the model in a scalable manner.

Estimation

The multilinear autoencoder estimation proceeds in two stages. First, we initialize both CNN encoder and multilinear decoder, as our training data is limited with respect to the number of parameters. Initializing the multilinear decoder with random values did not yield good results in our experiments, particularly in terms of decoupling. Hence we initialize by performing Higher Order Singular Value Decomposition (HOSVD) [De Lathauwer et al., 2000a] on a complete subset of the data, i.e.a subset in which all the factors of variation are present for all elements. Note that this enforces a limit on the dimensionality of the latent vectors, since now they cannot be greater than the amount of samples for each factor in the initial tensor. To subsequently pre-train the CNN encoder, we optimize it separately using the generative loss in Equation 3.5 with the fixed initial multilinear model, and with both registered and unregistered scans to augment the training data.

In the second stage the full network is optimized with all available face data. This is achieved by minimizing the following combined generative and latent loss:

arg min

M (1) ,{w (i) j } L G + λL L , (3.10)
where λ is a scalar that weighs the contribution of the latent loss.

Evaluation

We evaluate both the generative model that is extracted from the decoder as well as the full autoencoder that can be used for regression.

We begin by presenting implementation details (Section 3.4.1), the datasets employed (Section 3.4.2), and the proposed evaluation protocol that analizes the quality of the generative model (Section 3.4.3). In Section 3.4.4 we show results using these metrics over alternative latent weights and model dimensions, as well as comparisons to state-of-the art methods that learn multilinear 3D

CHAPTER 3. A MULTILINEAR AUTOENCODER FOR 3D FACE MODEL LEARNING FROM LARGE DATASETS

face models from incomplete data. Section 3.4.5 evaluates next the multilinear autoencoder and its ability to register raw scans into the new model. Finally, Section 3.4.6 showcases a few applications of the multilinear autoencoder.

Implementation Details

To pre-train the encoder and to learn the generative model during finetuning we use the AdaDelta algorithm [START_REF] Zeiler | Adadelta: an adaptive learning rate method[END_REF], with parameters as provided in the paper. We use a mini-batch size of 64, a learning rate of 0.01 for pre-training and a learning rate of 1 for training the autoencoder. The encoder was pre-trained for 100 epochs and the autoencoder was fine-tuned for 200 epochs. Unless otherwise specified, we use λ = 1 in Equation 3.10, and set the dimensions of identity and expression spaces to 89 and 25 respectively, which is the maximum allowed by the initial tensor (see Section 3.

Datasets

Training data for initialization We use BU-3DFE [START_REF] Yin | A 3d facial expression database for facial behavior research[END_REF]] and Bosphorus [START_REF] Savran | Bosphorus database for 3d face analysis[END_REF] datasets for initialization, as these come with manually annotated landmarks that simplify pre-processing. The data is registered using Optimal Step NICP [Amberg et al., 2007], initialized with Laplacian deformation using the provided landmarks. In the case of BU-3DFE we register against the "raw" version of the scans, switching to the post-processed version provided by the authors only in the cases where this failed. In total we registered 2499 scans from BU-3DFE and 2698 from Bosphorus. To initialize the decoder we run HOSVD [De Lathauwer et al., 2000a] on a data tensor built from BU-3DFE, which provides 100 identities performing 25 expressions each: the seven prototypical expressions2 , with each non-neutral expression in four levels of intensity. We consider each intensity as a distinct expression and remove subjects that belong to the testing set, resulting in a training data tensor of size 3 * 10057 × 89 × 25 3 . The CNN encoder is pretrained with both BU-3DFE and Bosphorus. To augment the training data we randomly rotate each face by an angle θ ∈ [-10 • ; 10 • ] in yaw, pitch or roll axes, and apply a random scale in [0.95; 1.05]. Furthermore, we use both the registered data and the corresponding raw 3D scans, for which the registered versions allow to recover ground truth vertex correspondences for training. This augmentation allows the CNN encoder to learn richer feature extractors, as the raw scans contain larger geometric errors, holes and extra parts such as hair and the neck. This results in a set of 99, 000 heightmap images for pretraining.

Training data for model optimization We demonstrate the capabilities of the multilinear autoencoder (MAE) trained on two different datasets. A first MAE is learned from static data, using the combined Bosphorus and BU-3DFE databases, for a total of 4500 meshes. We will refer to this MAE as Bu3+Bosph. We use seven labels from BU-3DFE which correspond to the highest intensity of each expression; the lower intensities are left unlabeled. For Bosphorus we label the seven prototypical expressions as well as the action units, with the exception of action units 43 and 44 which are not correctly captured by the registration. The second MAE is learned by combining the previous with the dynamic database D3DFACS [START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF] using the publicly available registrations of [START_REF] Li | Learning a model of facial shape and expression from 4D scans[END_REF] 4 . Although there is redundancy in consecutive frames, this allows to test a scenario where MAE is trained on a considerably larger training set. We will refer to this as Bu3+Bosph+D3D. The dataset is sparsely labeled by considering the first three frames of each sequence as the neutral expression, and the five frames located around the middle as peak frames, which are assigned the facial action unit of the sequence (establishing a semantic correspondence with the expressions in Bosphorus). In total, Bu3+Bosph+D3D is trained from 49811 scans, an order of magnitude larger than the training sets used in previous methods [Bolkart andWuhrer, 2016, Wang et al., 2017].

Test data

We leave 10 subjects out from BU-3DFE and 10 from Bosphorus, and test both Bu3+Bosph and Bu3+Bosph+D3D on these. The testing subjects were selected among those whose registrations were of good quality (manually verified), while keeping a balance between male/female subjects as well as different ethnicities. 5

Evaluation Protocol

We measure the quality of the generative models using the metrics generalization and specificity [START_REF] Davies | Statistical models of shape: Optimisation and evaluation[END_REF]. Generalization measures the ability of the model to adapt to unseen data, and is evaluated by projecting test data into the model space and calculating the reconstruction error. To provide a common framework for comparisons, this is implemented by iteratively fixing one space and finding the optimal coefficients for the other one [START_REF] Vlasic | Face transfer with multilinear models[END_REF]. We ignore border vertices during evaluation as these contain noise due to the registration process. Specificity measures whether only valid members of the shape class are modeled, or in other words, the model's suitability for 4. We registered our template to one of the frames using Amberg et al. [2007] and transferred the correspondences to the rest of the dataset.

5. We use the following identities: F0007, F0013, F0043, F0045, F0056, M0012, M0015, M0027, M0037, M0038 for BU-3DFE, bs003, bs024, bs032, bs038, bs081, bs086, bs090, bs092, bs095, bs101 for Bosphorus.

CHAPTER 3. A MULTILINEAR AUTOENCODER FOR 3D FACE MODEL LEARNING FROM LARGE DATASETS

generating synthetic data. To evaluate specificity, we assume the data to follow independent normal distributions in identity and expression spaces and sample 1000 faces. For each randomly drawn sample we measure its mean vertex distance to all elements in the training data and keep the minimum value; specificity is defined as the average of this process over all synthetically generated faces. To compute the normal distribution we consider the sample mean and standard deviation based on the training data. We account for an imbalanced number of labels by first grouping the coefficients by label, summarizing each group by its medoid, and computing the normal distribution based on per-group values.

There is no standard metric to evaluate decoupling, and hence we propose here a protocol that was adapted from Ghosh et al. [2017]. We first train an external classifier to recognize the seven prototypical expressions (anger, happiness, disgust, sadness, fear, surprise and neutral) given an input image with a rendered mesh6 . The evaluation proceeds as follows. We obtain identity and expression weights for each sample in the testing split of BU-3DFE by iterative registration. We regularize this process with a Tikhonov regularization term,

d id k=1 w 2 (k) -w2 (k) σ 2 (k) 2 (3.11)
for identity and similarly defined for expression, where σ 2 (k) denotes the standard deviation for identity coefficients. We further regularize by simultaneously registering all the expressions of a same identity (i.e.we solve for a unique identity weight w 2 while fixing several expression weights w

(i)
3 in a single system), and again ignore the border vertices due to noisy registrations. Once the identity and expression coefficients were recovered, we transfer the seven expressions of one test identity to all the other identities, and repeat the process for each subject in the testing set. Expression transfer is performed by replacing the expression weight w 3 with that of the current source face. Finally, we let the classifier measure whether the known transferred expression was preserved, and report the average accuracy of the classifier.

Generative Model Evaluation

This section shows results on the quality of the learned generative model under different configurations, as well as comparisons to classic tensor decomposition and two state-of-the-art methods on multilinear model learning of 3D faces from incomplete data. better decoupling of the spaces, but it appears to saturate at one point. An illustration of the effect of λ on the Bu3-Bosph model is shown in Figure 3.2. All selected models produce plausible synthetic faces, but there is a clear decrease in the quality of the transfers when the value of λ is too low. Interestingly, larger values of λ also appear to improve generalization, which suggests that the latent loss is acting as a regularizer that can help to better reconstruct unseen data. We select λ = 1 for the following experiments.

Influence of the latent loss

Compactness Ideally a model should perform well using a small number of parameters. We thus evaluate generalization, specificity and expression transfer under different latent dimensions, varying either the identity or the expression space. For this experiment we re-use the pretrained encoder from the previous evaluation (trained on dimensions 89 -25 for identity and expression) by simply removing the corresponding rows in the last linear layer of the encoder. The results can be found in Table 3.3. As expected, lower dimensions increase the error in generalization, although this is more significant when varying the identity space. On the other hand, the expression transfer capacity, as captured by the current evaluation, does not appear to vary much among the different dimensions. A change can be observed under 20 dimensions for the identity space (last row in Table 3.3). This could be explained by the fact that Euclidean distances between vectors, as required by Equation 3.7, are more significant when the dimensionality is lower. On the other hand, a correct expression transfer depends also on correctly recovering the coefficients and it is thus LEARNING FROM LARGE DATASETS that the proposed training indeed improves the initial model, even when no additional data is used. We further compare against HOOI since it shares our goal of enhancing an initial model provided by HOSVD, which is achieved through an iterative approach. We observe from Table 3.4 that all metrics are improved compared to both tensor decomposition methods. This includes the expression transfer capacity thanks to the addition of the latent loss, with a correct classification value that is almost 10% higher.

Effect of training data

The main benefit of our approach is the ability to train with large datasets that do not necessarily form a complete data tensor. Hence, we show in the bottom of Table 3.4 the improvements that can be attained by comparing against different training data sizes. While MAE-bu3, HOSVD and HOOI were trained on 2225 samples, MAE-bu3-bosph was trained on 4500 and MAE-bu3-bosph-D3D on 49811 scans. We can see from Table 3.4 that the ability to generalize to unseen data is greatly improved, as well as the specificity values, showing the benefit of leveraging all available training data. Table 3.5 -Comparison between state-of-the-art and the MAE decoder, in terms of median generalization error (mm), specificity error (mm), and percentage of correct classifications for expression transfer.

dimensions. Table 3.5 shows the results obtained. We can see that our method outperforms the other two in terms of specificity and expression transfer. Figure 3.3 shows an example of expression transfer results for the three methods. Note that while RMM and MAE achieve visually plausible results, Wang et al. [2017] gives noisy faces that do not preserve identities, as their tensor decomposition does not yield a good decoupling of the different modes.

Multilinear Autoencoder Evaluation

We now evaluate the multilinear autoencoder, including the encoder that can efficiently regress into the learned multilinear model. We start by dis- Table 3.6 -Generalization value (mm) under different fitting methods: using only the encoder; using the iterative approach with average identity and expression weights for initialization; and using the iterative approach with encoder initialization.

cussing the computation times of the method, and afterwards consider the capacity of the model to register unseen data. Generalization with the autoencoder For a fair comparison, all generalization values presented in the previous section were obtained by iterative fitting initialized with the mean identity or expression vector. We show in Table 3.6 two alternatives for this that leverage the encoder: by using the encoder for registration, and by using the encoder as initialization for the iterative method.

Computation times

We can see from Table 3.6 that the generalization value using the encoder alone is significantly higher than the rest. On the other hand this can be done very efficiently when performed on the GPU. Thus, we leverage the encoder for efficiently initializing the iterative process, achieving an even better generalization value as shown in the bottom row of Table 3.6.

Registration of raw scans

We evaluate the reconstructions of the test set obtained by regressing with the multilinear autoencoder using the original raw scan images. We consider the initial registered versions of the scans as groundtruth even though this might not be exact, since the registrations were manually verified to be globally correct. This gives a median per-vertex Euclidean error of 3.86mm for Bu3+Bosph, and a median per-vertex Euclidean error of 3.85mm for Bu3+Bosph+D3D. Figure 3.4 shows one example of raw scan registration (from a different dataset). Even though the error is relatively high, we observe that the outputs are in general visually close to the expected identity and expression, and could be used as initializations for optimization-based refinements.

Applications

We finish this section by showcasing two possible applications of the multilinear autoencoder.

Identity recognition from 3D mesh sequences Face recognition from 3D or 4D data has received increasing attention thanks to both a wider availability of depth data, as well as the potential it holds in overcoming the inherent limitations of 2D images. We test the ability of MAE to perform recognition as follows. Given an input mesh sequence, we use the encoder on each frame to recover identity and expression coefficients. We then measure the Euclidean distance between the recovered identity coefficient and each of the coefficients in a training database, keeping the identity label of the closest sample. Finally, we set as identity label of the sequence the label that was guessed by the majority of the frames.

We base the evaluation on the protocol used in Sun et al. [2010] and Alashkar et al. [2016]. In particular, we take 60 identities from the BU-4DFE dataset [START_REF] Yin | A high-resolution 3d dynamic facial expression database[END_REF] and perform two experiments: Expression Dependent (ED) and Expression Independent (EI). For the ED experiment we split each mesh sequence in half, using the first half as training database and the second half for testing. For the EI experiment we use one expression sequence for training and the rest of the expressions for testing, repeating for each of the six expressions. In both cases we report the percentage of correct classifications.

We show results in Tables 3.7 and 3.8, where we also compare to the values reported by [START_REF] Sun | Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis[END_REF] and Alashkar et al. [2016]. Our simple recognition method achieves the best results for the ED experiment, as well as competitive results on the EI experiment. Note that unlike these methods our approach is very efficient, as it requires only one pass of the encoder and an L2 distance calculation against the database, where both can be done efficiently on the GPU.

Expression Synthesis on Raw Data

The multilinear autoencoder can also be used to plausibly deform 3D facial scans, e.g.for automatic creation of a blendshape rig. We propose to this end the following method. Given an input raw scan in neutral expression, we first register our template in order to obtain identity and expression coefficients of the model. Here we registered in particular using Amberg et al. [2007] and then projected into the model using the iterative approach, initialized with the known neutral expression coefficient. We next recover the target expression coefficients by registering multiple expressions from a different identity. In this case we regularize both with Equation 3.11 and by fitting a unique identity weight to multiple expressions. We then combine the source identity coefficient with the target expression coefficients, thus obtaining deformed templates that perform the required expressions. Finally, we deform the original mesh based on the positions of the deformed template and with Laplacian regularization. Point-to-point correspondences between the template and the scan can be easily obtained thanks to the initial registration. An example of results obtained using this method on a subject from Bosphorus can be found in Figure 3.5, where different action units where transferred to a source mesh (taken from the testing set). For comparison, we show in the top row of

Conclusion

In this chapter we demonstrated that it is possible to obtain an expressive multilinear model from large and diverse datasets, by leveraging a novel architecture that we call Multilinear Autoencoder. The proposed approach is capable of making better use of all available data, learning a generative model that can better decouple the latent space, and an encoder that can perform fast regression into this model from raw, unregistered scans. Throughout the experimental evaluation we showed that the Multilinear Autoencoder outperforms current state-of-the-art methods that learn multilinear models from incomplete data, particularly in terms of decoupling the spaces. We believe this work opens up possibilities for learning rich generative 3D face models from large training sets, which in turn can enhance numerous applications including recognition and animation. The next chapter will present one of these possible applications: the registration of large datasets of 3D scans.

The proposed method has a few limitations that are worth mentioning. A first limitation lies in the need to initialize the decoder with a complete data tensor, requiring at least a subset of the data to be capable of being assembled as such. It is clear that removing this would further simplify the requirements on the training set. Moreover, it would also eliminate the restrictions on the latent space dimensions (see Section 3.3.3), which are bounded by the size of this subset. As mentioned, a random initialization of the decoder did not yield good results in our experiments, yet better initialization approaches can still be explored.

Another disadvantage is related to the choice of dimensionality of the different spaces. Not only this choice is bounded by the size of the initial tensor, but there is also no principled approach for selecting the appropriate model dimensions. Unlike PCA or HOSVD where the percentage of retained vari-ance can guide model selection, here it must be done by ad-hoc procedures, e.g.parameter sweep as in Table 3.3. It is not clear yet how model selection can be performed for this type of deep-learning based models, but a more principled approach would certainly be desirable, as compactness is an important aspect.

A final limitation is the large number of parameters involved in the model, a problem that comes from the use of a multilinear model itself. The amount of entries in the core tensor is typically very large and grows exponentially with each new factor that is added. In our implementation, the number of trainable parameters in the decoder is an order of magnitude larger than those in the convolutional encoder. This also results in large disk space usage for the trained model, on the order of hundreds of megabytes.

Most of these limitations will be addressed by considering a novel strategy for modeling decoupled spaces in Chapter 5.

Large-Scale Registration of Faces in Motion

Registration is an essential step in the process of learning a model such as the one presented in the previous chapter. Given two or more scans of a 3D face (not necessarily of the same subject) this technique ensures that anatomically corresponding points are consistently identified, such that the vertex located at e.g.the tip of the nose will always be found at that location1 . Without this step, the different 3D scans have no coherent structure and their common patterns cannot be studied.

We focus here on the registration of spatiotemporal data, i.e.sequences of 3D face scans, otherwise called 4D data. The interest in the context of this work is two-fold. First, this allows to parameterize not only the individual facial shapes but also their temporal evolution, expanding the scope and performance of automatic facial analysis systems such as expression recognition [START_REF] Yu | Perception-driven facial expression synthesis[END_REF]. Second, motion acquisitions allow to capture a larger range of expressions including spontaneous ones, which would be much harder to elicit in a static capture. This can give access to a rich and diverse source of information for building generative models of the 3D face, and registration is a fundamental first step towards this goal.

When dealing with motion sequences one could in principle apply a purely spatial registration algorithm to each frame, e.g. [Amberg et al., 2007, Salazar et al., 2014], yielding a static face pose parametrization that is agnostic to time information. Yet, in the case of faces in motion, the registration can account for the temporal aspects through a spatiotemporal parametrization. The interest here is to better capture face deformations with a temporal tracking, where static registrations provide only coarse and noisy motion information. Spatiotemporal registration is however more complex than its static counterpart, since tracking robustly and reliably is still challenging in practice.

The process of collecting 4D scans is expensive and time-consuming. Yet, as mentioned before, multiple research groups have captured and released large 4D face databases throughout the last decade, e.g. [START_REF] Yin | A high-resolution 3d dynamic facial expression database[END_REF], Cosker et al.

[2011], Zhang et al. [2014]. We aim here to harness such source of information by registering the large corpus into a single parameterization. This in turn poses additional challenges to the registration process. Not only the amount of data is considerably larger, but the capture systems and acquisition protocols differ from one dataset to the other. Face registration methods in this context should be able to handle datasets in a fully automatic way, as manual intervention is not feasible for thousands of frames, in addition to being robust to different types of noise resulting from different acquisition scenarios. Furthermore, they must be efficient in order to process thousands of frames in a reasonable time.

The work presented in this chapter addresses the aforementioned objectives by proposing a novel method to register spatiotemporal 3D face data. The approach is based purely on geometry to allow leveraging any available temporal 3D face scan, even when the associated RGB images are not provided e.g.for privacy reasons. We do not require pre-determined landmarks as input, which are more challenging to obtain for 3D data, thus removing a possible source of error. The main innovation here is the use of a spatiotemporal model as opposed to a purely static one, which combined with a regression-based approach allows to exploit the spatial and temporal coherence of the data in an efficient manner. The use of such model enables registrations that both fix identities over temporal sequences and regularize observed motions to prevent high-frequency flickering. The approach presents the following advantages: it can register multiple datasets into a single representation; it does not require color information as in e.g. Cosker [2011] show that the proposed approach can achieve similar or better results in terms of vertex-to-scan error and in terms of semantic parametrization, while remaining either more general in terms of requirements of the datasets, or more efficient in terms of computational times.

Related Work

Numerous works have studied the registration of static 3D face scans, and an overview can be found in Section 2.2. While a static method can be applied independently to each frame of a motion sequence, this is known to lead to artifacts including high-frequency jitter. We focus therefore on methods that take advantage of the temporal redundancy captured by 4D data. A related line of research that has recently received considerable attention is the reconstruction of 4D facial motion based on monocular 2D video, e.g. [START_REF] Cao | Real-time high-fidelity facial performance capture[END_REF], Garrido et al., 2016a]. These works solve an underconstrained reconstruction instead of a 3D registration as addressed in this work; the interested reader is referred to the survey of [START_REF] Zollhöfer | State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications[END_REF]. In the following we discuss strategies for the registration of 4D face data.

Registration of 4D face data

Initial geometry-based methods used a coarse-to-fine approach combined with Free Form Deformations [START_REF] Wang | High resolution acquisition, learning and transfer of dynamic 3-d facial expressions[END_REF], or through harmonic [START_REF] Wang | High resolution tracking of non-rigid motion of densely sampled 3d data using harmonic maps[END_REF] or conformal [START_REF] Sun | Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis[END_REF] maps that reduce the problem to 2D registration. For expression recognition, Fang et al. [2012] performed pairwise registration of consecutive frames using an Annotated Face Model (AFM) [START_REF] Kakadiaris | Three-dimensional face recognition in the presence of facial expressions: An annotated deformable model approach[END_REF], where temporal information was exploited by initializing with the result of the previous frame. More recent alternatives, used mainly for high-quality acquisition setups, follow two main lines. The first performs registration in texture space by computing correspondences between sparse landmarks predicted using an Active Appearance Model (AAM), which are densified using thin-plate spline deformations [START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF], Cheng et al., 2017a]. The method of Cosker et al. [2011] achieves inter-sequence correspondence by registering each frame towards a manually selected neutral expression, and intra-sequence correspondence by registering these neutral frames to a template. To better handle texture variations, Cheng et al. [2017a] extend the previous by using sessionand-subject specific AAM, and non-rigid ICP [Amberg et al., 2007] between manually selected neutral frames. Since these methods operate on color information, they require careful acquisition setups with controlled lighting conditions, as e.g.moving shadows can lead to inaccuracies.

The second line of work takes advantage of low dimensional parametric shape spaces learned from large databases of static 3D face scans and used as prior during registration. Most related to our work, multilinear models of identity and expression [Bolkart and Wuhrer, 2015a] and a linear articulated model with expressions [START_REF] Li | Learning a model of facial shape and expression from 4D scans[END_REF] have been used for this purpose. These works achieve registrations of relatively high accuracy and report running times of 30 seconds to 2 minutes per frame. Although the overall facial shape is recovered, fine-scale details such as wrinkles are not modeled. Our work shares this property, as well as the robustness and accuracy of these methods while allowing for a gain in efficiency.

Joint registration and reconstruction

Performance capture is concerned with the recovery of both the 3D shape and its temporally coherent deformations, and it is hence also related to this work. In this context, several authors use optical flow to recover a consistent geometry from passive [START_REF] Bradley | High resolution passive facial performance capture[END_REF][START_REF] Beeler | High-quality passive facial performance capture using anchor frames[END_REF] or active [START_REF] Zhang | Spacetime faces: Highresolution capture for modeling and animation[END_REF]] systems using synchronized multi-view 2D videos. Bradley et al.

[2010] jointly solve for registration and reconstruction by sequentially tracking the initial frame. Optical flow with sequential tracking is known to be prone to drift (i.e.the accumulation of tracking errors), and thus [START_REF] Beeler | High-quality passive facial performance capture using anchor frames[END_REF] propose instead to do optical flow on sub-sequences defined by automatically selected key-frames. Results are of very high quality and achieve pore-level details. Non-sequential tracking has also been explored, by using a minimum spanning tree [START_REF] Klaudiny | High-detail 3d capture and non-sequential alignment of facial performance[END_REF], a performance flow graph [START_REF] Fyffe | Driving highresolution facial scans with video performance capture[END_REF], or by independent optical flow between a template and each frame [START_REF] Fyffe | Multi-view stereo on consistent face topology[END_REF]. All of these methods require a dense setup of synchronized video cameras. [START_REF] Valgaerts | Lightweight binocular facial performance capture under uncontrolled lighting[END_REF] simplify these requirements by introducing a method that achieves results of similar quality from a single pair of stereo cameras, combining sequential scene flow to compute the global registration with shading-based refinement to compute fine-scale details. More recently, [START_REF] Wu | Deep incremental learning for efficient high-fidelity face tracking[END_REF] proposed an incremental approach in which a personspecific neural network is used for initialization and gradually improved as more frames are registered. These methods achieve temporally coherent results which are of high quality and include fine-scale details. However, they are limited to specific acquisition setups as the input to the methods are synchronized and calibrated 2D videos. In this work, we consider the more general problem of registering the geometry of 4D face scans without the need for reliable color information.

Method

We aim here at registering a large number of sequences of 3D face scans, displaying a varied range of identities and emotions. Each of these sequences may contain many frames, and each frame a large number of vertices, making the problem high-dimensional and difficult to optimize. Furthermore, datasets captured with different acquisition setups present different levels of noise, missing data and occlusions, hence the naive application of a frame-by-frame template fitting approach is prone to failure. To keep the method as general as possible we do not assume availability of either landmark or color information, Figure 4.1 -Overview of the proposed spatiotemporal registration approach. The input to the system is a sequence of 3D scans showing a 3D face in motion, and the output is a registered version of the sequence encoded in a compact representation. The registration is initialized by a frame-by-frame automatic regression approach that gives identity and expression coefficients of a multilinear model. The identity coefficients are then combined in order to build a spatiotemporal model of the sequence, and an iterative process alternates between projecting into this model and refining the geometry. which allows to register data coming from sources for which privacy is a concern.

In order to process large datasets we seek a strategy that is centered around robustness and efficiency. To this end, we follow a model-based approach as was previously considered by e.g. Amberg et al. [2017] in the temporal case. But unlike these, we propose instead to use a spatiotemporal model, combining a shape space that regularizes the spatial information with a temporal space that regularizes the trajectories of each vertex, thus capitalizing on the redundancies present both in space and time. This allows not only for more accurate registrations, but also faster computational times and a very compact representation of the output.

A second step towards robustness and efficiency is taken by the use of a regression-based initialization. This builds on the work presented in the previous chapter by leveraging the multilinear encoder for a fast initialization of the spatial component of the model. The regression is done independently on each frame and outputs for each time step the identity and expression coefficients of a static multilinear model. Per-frame identity coefficients are then combined into a single identity weight and used to build the shape basis of the spatiotemporal model. The algorithm proceeds by alternating between fitting the current estimation to the possibly noisy correspondences, and projecting the result back into the spatiotemporal model. Figure 4.1 summarizes the process.

We will begin by describing the spatiotemporal model in Section 4.2.1, while the details of the algorithm are presented in Section 4.2.2.

Spatiotemporal Model

The model used in this approach is an extension of the multilinear model introduced in Section 2.3.3 and Chapter 3. Recall that, given x ∈ R 3n a vector of coordinates associated with the n vertices of a registered mesh, M ∈ R 3n×d id ×d exp a core tensor, and w id ∈ R d id , w exp ∈ R d exp the identity and expression coefficients respectively, the multilinear model relates these to the 3D face by:

x ≈ x + M × 2 w id × 3 w exp , (4.1)
where x is the mean face over the model's training data, and × i denotes mode-i multiplication.

When the data is a sequence of F 3D faces in correspondence, [x 1 , . . . , x F ], x i ∈ R 3n , one could encode it within the multilinear model by using a unique identity representation w id plus an array of expression weights [w 1 exp , . . . , w F exp ]. This is the approach followed for example by Bolkart and Wuhrer [2015a], and while it gives a relatively compact representation, it has a few drawbacks. First, unless a prior is imposed on the curve, the expression weights can take any form which in practice results in flickering of the reconstructed vertices. Second, the formulation does not take into account the high temporal regularity that each vertex exhibits, resulting in redundancy of the representation.

As originally proposed by Akhter et al. [2012], a bilinear model can be built that leverages both spatial and temporal redundancies. To this end, the sequence is organized into a matrix S ∈ R F×3n containing each frame in a row, S = [x 1 , . . . , x F ] T , or equivalently each vertex trajectory in a column. Let B ∈ R 3n×d s be a matrix with the shape basis vectors in its columns, encoding each frame into a space of dimension d s . Similarly, let Θ ∈ R F×d t be a matrix with the temporal basis vectors in its columns, encoding the trajectory of each vertex into a space of dimension d t . Then the sequence matrix S can be decomposed as S ≈ ΘCB T , (

where C ∈ R d t ×d s is a matrix of spatiotemporal coefficients that compactly encode S. The dimensions d s and d t allow to trade off the compactness of the representation and the approximation error of the input sequence.

We can easily incorporate this model into the multilinear framework. Given the unique identity coefficients w id , multiplying it with the core tensor results in a shape matrix for that particular subject:

x ≈ x + (M × 2 w id ) × 3 w exp (4.3) = x + w T exp (M × 2 w id ) (3) (4.4) = x + w T exp B T (4.5)
where X (3) denotes mode-3 matricization of a tensor X (see [START_REF] Kolda | Tensor decompositions and applications[END_REF]). We can thus obtain the shape matrix B of the spatiotemporal model by simply multiplying M × 2 w id , assuming we know w id .

METHOD

51

For the temporal basis Θ we follow Akhter et al. [2012] and fix it to the Discrete Cosine Transform (DCT), since this approaches the optimal PCA-learned basis when the data is generated from a stationary first-order Markov process, and was empirically demonstrated to hold for sparse facial data in Akhter et al. [2012]. Note that the dimensions d t of the temporal basis need to be chosen carefully, as a very low-dimensional space will not allow high-frequency trajectories and will flatten the original motion. In this work we set d t to be a factor of the sequence length (i.e.d t = F/k for some constant k), which allows a certain degree of independence from the sampling rate if we assume that the motions are of approximately the same speed. This approach worked well for the experiments shown in this chapter where this property holds, but a more accurate method for model selection should be investigated. We leave this as future work.

Registration

As depicted in Figure 4.1, we proceed in two major steps. First, we perform an initialization independently on each frame that robustly and efficiently regresses each scan against identity and expression coefficients of the multilinear model. The resulting meshes correctly capture the general structure of the motion and shape but are still overly smooth and, because each frame is treated independently, exhibit high-frequency jitter. To remedy this we use the multilinear face model to build the spatiotemporal model described in Section 4.2.1. The second step makes use of this model to iteratively improve the initial approximations, regularizing the motion of the vertices and turning the problem into a much lower-dimensional one compared to a frame-wise formulation. We now describe each of these steps in more detail.

Frame-wise Initialization

Given a sequence of observations [o 1 , . . . , o F ] consisting of F frames, we register each frame independently by regressing identity and expression coefficients using the encoder part of the multilinear autoencoder presented in Chapter 3. We thus obtain a sequence of identity and expression weights that represent the face motion:

W id = [w 1 id , . . . , w F id ]
, and (4.6)

W exp = [w 1 exp , . . . , w F exp ]. (4.7)
Even though the network was trained with raw scans presenting different types of noise, the quality of the registration degrades when the input data differs in form and orientation from the original training data. To ensure results of high quality, the input 3D scan is therefore pre-processed as follows. We first detect the nose tip by training a neural network for this task on depth data, using the same architecture and training data as in the previous chapter (Section 3.4.2)2 . We next crop the face a radius of 100mm around this point, and perform a coarse frontalization step so that it approximately looks towards the z-direction. For frontalization, we consider the direction of the normals of each vertex, which gives a distribution of orientations over the sphere that sample a semi-sphere for a cropped face. The directional mean of this distribution gives a coarse approximation of where the face is "looking at", and we align this to the directional mean of the model's mean face. This makes the weak assumption that the face is not upside-down and works well as long as the cropped face does not contain too many holes or extra parts. Note that only a coarse alignment is required here since the autoencoder was trained to have some degree of robustness to pose variation in the input image. As a result, this pre-processing step can be replaced with any other method that will produce a cropped face and a rough frontalization.

The resulting multilinear model representations [x 1 , . . . , x F ], obtained by reconstructing the faces using Equation 4.1 with the coefficients from Equations 4.6 and 4.7, are in the coordinate system in which the multilinear model was learned. For further refinement of these approximations they need to be compared to the original scans. To align the observations [o 1 , . . . , o F ] to the model coordinate system we take advantage of the depth images generated during regression to find initial correspondences. In particular, we consider the depth image of the cropped and frontalized scan, and the depth image of the registered mesh, and establish preliminary correspondences by assigning pixels at the same location. This correspondence is used to rigidly transform o i to x i (i = 1, . . . , F). We then perform a few iterations of regular ICP alignment 3 . Once each frame is aligned, we discard the cropped version and go back to the original raw scan; this allows to remove the quality of the crop as possible source of error in the subsequent steps.

Given the identity coefficients of Equation 4.6 we next proceed to build the model outlined in Section 4.2.1. Unlike the original formulation of Akhter et al. [2012] in which both the shape basis B and the model coefficients C need to optimized, we leverage the multilinear model to obtain a person-specific spatiotemporal representation. Specifically, we summarize the regressed identity coefficients W id into a unique coefficient w id for the entire sequence, given that the identity of the subject is fixed for any given motion. We compute w id as a mean over the regressed results, w id = W id , and create the shape basis B by multiplying the core tensor with w id as B T = M × 2 w id . The temporal basis Θ is fixed to the DCT basis according to the specified dimensions d t .

Iterative Refinement

Up to this point we have computed, for each frame i, spatially aligned observations o i along with registered faces x i that approximate the geometry of o i . By iterating between refining the geometry of x i to match o i , and projecting into the spatiotemporal model, we further improve the geometric approximation of the registrations, obtaining temporally smooth results that can be represented compactly.

Geometric refinement

To improve the quality of the approximation x i of o i we non-rigidly deform the registrations to the scans. The following discussion omits the frame index i to simplify notation. The registration x is warped to o by optimizing for displacements δ x of the vertices of x along their normal directions with Laplacian regularization. In particular, we solve for

min {δ x 1 ,...,δ x n } α n j=1 w j ||v x j + δ x j n x j -p x j || 2 + β n j=1 ||L(δ x j )|| 2 , ( 4.8)
where v x j ∈ R 3 is a vertex in x, p x j ∈ R 3 the closest point to v x j in o, n x j ∈ R 3 the normal vector of v x j , L the cotangent discretization of the Laplace-Beltrami operator [START_REF] Meyer | Discrete differentialgeometry operators for triangulated 2-manifolds[END_REF], and w j , α, β are scalar weights. We discard closest points whose Euclidean distance is greater than 5mm and whose deviation in normal vector is greater than 45 • by setting w j to zero. This formulation can be efficiently minimized by solving a linear system of equations.

Spatiotemporal sequence projection

For each iteration, we gather the approximations x i obtained in the previous step into a sequence matrix S, and use Equation 4.2 to compute C by solving

BC T = S T Θ. (4.9)
This can be performed efficiently, as B is fixed and can be factorized once for all the iterations.

Final refinement

The two previous steps are iterated a few times until convergence. To obtain more detailed results we complete the iterative process with a final geometric refinement step. This allows to leave the bounds of the multilinear model, thereby providing more accurate approximations of o i . To prevent artifacts we use a stronger regularization weight β in the last geometric refinement. However, this loses both the compactness of the representation and the motion regularization. To rectify this we project the trajectory of the displacements δ x i into a second DCT basis (of possibly different dimensionality d ′ t ), obtaining a displacement coefficient vector d i ∈ R d ′ t for each of the n vertices, with d ′ t << F. We thus retain compactness while allowing for more detailed registrations, as well as preventing flickering in the final trajectory of the vertices.

Sequence Representation

After registration we can compactly store the results given the multilinear model. For each registered motion the following information suffices to reconstruct the sequence [x 1 , . . . , x F ]: (1) the identity coefficient w id ∈ R d id that compactly encodes the shape matrix B; (2) the spatiotemporal coefficients C ∈ R d t ×d s ; (3) the dimensions of the temporal bases d t and d ′ t ; and (4) the m displacement coefficients {d 1 , . . . ,

d m }, d i ∈ R d ′ t .
This significantly reduces the storage requirements of large datasets (e.g.from 9.1GB to less than 1MB in the example from Figure 4.5), while still retaining a reasonable level of detail.

Evaluation

We validate the approach on D3DFACS [START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF], BU-4DFE [START_REF] Yin | A high-resolution 3d dynamic facial expression database[END_REF]] and BP4D-Spontaneous [START_REF] Zhang | Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database[END_REF] datasets. D3DFACS contains 519 sequences of 10 subjects performing different types of facial action units, while BU-4DFE contains 101 subjects with 6 sequences each performing the six prototypical emotions; in both cases the average sequence length is around 100 frames and the meshes contain around 30K and 35K vertices respectively. BP4D-Spontaneous contains 328 sequences with 41 subjects performing 8 tasks each, which were designed to elicit spontaneous emotions. The average sequence length is around 1100 frames and the average number of vertices is around 37K. In the following we provide both qualitative and quantitative evaluations over these.

Implementation details

The code was implemented in C++ using Eigen3 [START_REF] Guennebaud | [END_REF] and libigl [START_REF] Jacobson | libigl: A simple C++ geometry processing library[END_REF]. We use the autoencoder from Chapter 3 that was trained on BU-3DFE [START_REF] Yin | A 3d facial expression database for facial behavior research[END_REF] and Bosphorus [START_REF] Savran | Bosphorus database for 3d face analysis[END_REF] datasets for 500 epochs 4 . The dimensions of identity and expression spaces are set to 65 and 20, and the dimension of the temporal basis is set to F/5 (where F is the number of frames). We set d ′ t = 5, and unless otherwise specified, we fix the number of iterations to 5. For Equation 4.8 we set β = 1, α = 0.9 during iterations and α = 0.8 for the final step. Since BU-4DFE contains noisier scans, we set α = 0.5 during iterations and α = 0.2 for the final step to avoid overfitting. The template mesh has 5996 vertices and is depicted in Figure 4.4.

Qualitative results

Quantitative results

We evaluate the quality of the registrations with two commonly used metrics: median per-vertex error towards the input scan, and landmark distances. The median per-vertex error is taken across all registered frames in the dataset, and shows how close the registrations are to the real scans. We also evaluate semantic accuracy by manually placing landmarks on five key-frames over 10 randomly selected sequences of D3DFACS, and measuring the Euclidean distances between these and the landmarks defined over the template. We use in particular 11 landmarks, which can be visualized in Fig 4 .4. The chosen key-frames sample the sequence by taking the first and last frame, the peak frame, and two intermediate ones.

We evaluate the stability of the motion by using a compactness measure (see [START_REF] Davies | Statistical models of shape: Optimisation and evaluation[END_REF]) as follows. For each sequence, we align the frames using generalized Procrustes analysis, perform PCA, and measure the amount of variabilty captured by each principal component. If the registrations exhibit high-frequency jitter, we expect to see less variability retained by the first principal components, as the variations coming from flickering vertices would have to be encoded by higher-order principal components. To summarize this over the entire dataset we determine the mean variability obtained as a function of the percentage of principal components considered.

Figure 4.6 shows a cumulative plot of the median per-vertex error on D3DFACS obtained after the initial regression, after 1, 3, 5 and 10 iterations of spatiotemporal registration, and for the final result. We can see that the iterative process improves the initial regressions in terms of surface fit. Furthermore, Table 4.1 shows the mean landmark error over the 11 landmarks for the final results obtained after 1, 3, 5 and 10 iterations. Despite being a landmark-free registration method, the method obtains a good semantic accuracy that is improved with each iteration.

We evaluate the benefits of the temporal regularization by comparing the full model with a static version of our framework. For this, instead of projecting onto the spatiotemporal model we independently project each frame onto the shape basis B, and measure the results in terms of vertex error and compactness. Figure 4.7 shows cumulative plots obtained for these registrations. Note that while using a spatiotemporal model achieves similar accuracies in terms of vertex error, the compactness of each sequence improves with the spatiotemporal model, implying less high-frequency jitter with the latter. This results can also be qualitatively assessed in the accompanying video.

Finally, we show the ability of the method to track long videos by registering the sequences from BP4D-Sponteanous, many of which consist of more than 1000 frames. We obtain a mean vertex error of 0.33mm over all registered sequences and frames. Figure 4.5 further shows the median per-vertex error for each frame on one example. This error stays between 0.1 and 0.4mm and does not increase with time, suggesting that no drift is occuring. This is expected, as the regression step is performed independently on each frame.

With respect to the running times, we report a mean per-frame processing time of 578ms on the D3DFACS dataset, 637ms on BU-4DFE and 399ms for BP4D, for five iterations in all cases. Computation times were measured on an Intel Xeon 3.30GHz with NVidia GeForce GTX 1080 GPU.

Comparisons

We compare our method to the previous works of Bolkart Bolkart and Wuhrer Bolkart and Wuhrer [2015a] also register motion sequences in a fully-automatic manner by using a multilinear model and geometric information only. We compare to this method on 497 sequences from [START_REF] Li | Learning a model of facial shape and expression from 4D scans[END_REF] was used to register D3DFACS, and thus we compare our results over this dataset. For a fair comparison we crop their full-head model so that it contains only the face, to be similar to our registrations. We obtain a mean vertex error of 0.13 mm for our method, and 0.33mm for Li et al. [2017]. In Figure 4.8b we show the cumulative plots for the median per-vertex distance for both methods. They demonstrate that our approach achieves higher accuracy while reducing both the running time [START_REF] Li | Learning a model of facial shape and expression from 4D scans[END_REF] reported 155 seconds per-frame) and the requirements on the dataset. We also compare with respect to landmark errors; results can be found in Table 4.1. Our approach achieves better semantic accuracy with a single iteration, even though it requires no pre-determined landmarks to guide the process, confirming that our registrations faithfully preserve the anatomic semantics. Figure 4.9b shows a qualitative comparison.

Li et al. The method of

Comparisons on efficiency

The efficiency of our method comes from both the regression step and the spatiotemporal model optimization. The regression is essential to get a good starting point that is already close to a local minimum, and it can be performed efficiently on the GPU thanks to the heightmap representation. Furthermore, due to the spatiotemporal model we need to optimize for much less parameters, while still remaining in a global sequence formulation. In particular, on each iteration we need to solve for the matrix C which is of size d s × d t , with d s = d exp . In our implementation d t = F/5 and thus we optimize for d exp F/5 parameters, reducing by a factor of 5 compared to a frame-by-frame formulation. The method of Bolkart and Wuhrer [2015a] optimizes for the parameters of each frame, which amounts to d id + Fd exp variables to be solved. The main data term on the method of Li et al. [2017] optimizes for shape and expression parameters plus per-joint pose parameters of an articulated model on a frame-by-frame basis, increasing the complexity. Moreover, each frame is initialized from the previous one and thus it cannot be parallelized. As for UV-based methods such as [START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF], the computational complexity depends on the number of pixels of the image; while these are usually more efficient than 3D-based ones, we have shown that we can achieve similar accuracy, while remaining more general with respect to the acquisition setup.

Limitations

The regression-based initialization allows the method to be robust to noise in the input data, but it comes with drawbacks. In particular, the use of a depth map implies that the method is not rotation-invariant, and thus a proper pre-processing is needed to ensure that the face is "looking front". Although this does not require accurate pose detection (the network was trained with data showing ±30 • of pose variation), the output will be more accurate the closer the input is to a frontal pose, and this in turn affects the final result. Furthermore, our choice of initialization can sometimes be a source of failure, particularly with the nose tip detection; see e.g.Figure 4.10. When this step fails all subsequent steps fail too, since regressions are inaccurate and initial correspondences cannot be found. In our experiments, this resulted in erroneous registration of some of the frames in BU-4DFE and BP4D datasets. On the other hand, since we are dealing with motion data, unsuccessful frames that are isolated can be ignored without resulting in failure of the entire sequence. In our implementation we automatically detect failed frames after ICP diverges, and this is fixed by interpolating pose and shape parameters using correct neighbouring frames. With this simple approach all sequences from BU-4DFE and 95% from BP4D were registered (no errors were found during registration of D3DFACS).

Another intrinsic limitation comes from the restricted scope of our trained model. Particularly for expressions that are far from this scope, the framework will provide only a coarse approximation and even the final refinement step can fail to compensate. A related problem, already mentioned in Section 4.3.4, is the simplistic approach for temporal model selection, in which the dimensions are only dependent on the number of frames. Even though this worked well for most of the sequences registered here, a few of the sequences from BP4D include speech, which occurs faster than pre-defined expressions. In our experiments the chosen temporal resolution is not sufficient for this type of motion, and some of the visemes get smoothed out.

Conclusion

We described in this chapter a novel method for the automatic registration of large datasets of 3D face scans in motion. Having access to a fully automatic, efficient and accurate registration approach not only enables the study of both spatial and temporal patterns of the face, but also allows to do so at a larger scale. This in turn allows to benefit from the efforts of multiple researchers that captured and made publicly available different aspects of the 3D face in motion. The technique proposed in this chapter holds several properties that are appealing in this context: it is fully automatic, robust to different noise characteristics, has minimum requirements on the input scans and as such it is not limited to a specific capture system, it is efficient, and yields compact representations of the data. We successfully registered in Section 4.3 three standard datasets, without losing accuracy and with significantly better time performances than competing methods. The approach shows how the use of a global spatiotemporal model -as opposed to a purely static shape model, or a sequential motion prior-can benefit the task of 4D registration. It also indicates how a regression-based approach can help to achieve a robust initialization, despite its limited accuracy against more classic methods.

There are several aspects that should be considered in a future work. First, as mentioned in Section 4.2.1, the approach we used for temporal model selection is quite simplistic and a more accurate strategy, capable of handling motions of variable speed, would be more valuable. In particular, we observed that speech-related motions -which are performed at higher speed than predefined expressions or action units-can be easily "washed out" due to the low dimensional temporal space. Setting higher dimensions for the temporal space on the other hand results in high-frequency jitter, and thus a better approach -e.g.an automatic selection of the best temporal dimension, or a semi-local model covering a certain time window-should be explored. An alternative that is also worth exploring is to consider a different type of temporal model, e.g.a data-driven one.

Another interesting extension would be to consider a refinement not only of the expression and geometry factors, but also the identity which is used to build the shape basis B. Recall that this is constructed purely from the regression results -a bad identity initialization could result in poor registrations. Although we did not observe such problems, it is clear that simultaneously refining the shape would further benefit the accuracy of the technique.

A final extension should devote particular attention to the most expressive parts of the face: the mouth and the eyes. These areas have very distinct motion, geometry and appearance, and are key for transmitting emotions. Our current method is not capable of distinguishing closed from open eyes, or capturing subtle eyelid motion, and as previously mentioned the temporal dimensions are sometimes insufficient to correctly capture speech. A local treatment of these (as considered in e.g. Garrido et al. [2016b], [START_REF] Bermano | Detailed spatio-temporal reconstruction of eyelids[END_REF]) should be considered for more accurate results.

A Decoupled 3D Facial Shape Model by Adversarial Learning

The previous chapter provided access to a large dataset of registered 3D faces. With this at hand, we revisit the problem of building decoupled models that was addressed in Chapter 3, this time in light of more advanced modeling techniques.

We are interested here in building generative models that can capture the space of realistic three-dimensional faces, while also differentiating the various factors that influence the generation of this shape, e.g.the individual identity or the expression. As mentioned, these decoupled models offer an independent parameterization to each of the sources of variation while at the same time modeling the interactions that occur among them. While the interest is typically to disentangle identity from expression, other factors may come to play too. For instance, a model that can disentangle shape, expression and viseme1 can have applications in visual 3D speech synthesis, enabling systems that generalize to multiple identities and can control the emotion in which a viseme is performed.

The identity and expression subpaces are typically modeled as two independent linear factors which are additively combined [Amberg et al., 2008]. While simple and effective for inference applications, these models can produce artifacts when transferring expressions among very different facial shapes. Another commonly used alternative for decoupling the latent space is the multilinear model, and we have seen in Chapter 3 how it can be learned from a large dataset of 3D facial scans. Yet, there are several challenges that remain.

First, while scalability in terms of size of the training set was already addressed in this thesis, multilinear models are still not scalable in terms of number of factors. The size of the core tensor grows exponentially with each new dimension, a property that is shared by the Multilinear Autoencoder of Chapter 3. Furthermore, acquiring the data becomes harder with each new factor: a model that decouples for example identity, expression and viseme would require the capture of multiple subjects each performing all the expressions in all of the visemes; a high cost both in time and money. Multilinear Autoencoders also suffer from this, as they still require a subset of the data to be assembled as a tensor for initialization.

A second downside is the assumption of linearity. During motion the face undergoes complex deformations that experts believe cannot be correctly captured by a restricted linear space [START_REF] Pighin | Facial motion retargeting[END_REF][START_REF] Cosker | Perception of linear and nonlinear motion properties using a facs validated 3d facial model[END_REF][START_REF] Trutoiu | Spatial and temporal linearities in posed and spontaneous smiles[END_REF]. To relax the linear assumption in modeling 3D faces, deep generative models with autoencoder architectures have recently been proposed. They demonstrate benefits in modeling geometric details [START_REF] Bagautdinov | Modeling Facial Geometry using Compositional VAEs[END_REF][START_REF] Tran | Towards high-fidelity nonlinear 3d face morphable model[END_REF], joint appearance and shape [START_REF] Lombardi | Deep appearance models for face rendering[END_REF][START_REF] Zhou | Dense 3d face decoding over 2500fps: Joint texture & shape convolutional mesh decoders[END_REF] and non-linear deformations present in extreme facial expressions [START_REF] Ranjan | Generating 3d faces using convolutional mesh autoencoders[END_REF]]. Yet, none of these approaches are capable of decoupling the factors of variation. 2 A final drawback, this time more specific to the technique presented in Chapter 3, relates to the loss function that was used to enforce decoupling while training the network. The loss in Equation 3.7 induces an error solely based on the structure of the latent space. Yet, we believe that the perception of the output should also be taken into acount if we want to model subtler changes. For example, given two meshes with a same expression code, they should be perceived by a human observer as having the same expression even if the geometric displacements are significantly different, and this should be valid not only for e.g.the coarse "smile" expression label, but also for all the subtle variations that exist. Ideally, the latent loss should focus too on this aspect.

We present here an altogether different technique for learning a decoupled 3D face model, and investigate the use of Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF] for this task. GANs are well suited to our problem since the loss function is designed to evaluate the output and not the structure of the latent space, while performing a non-linear transformation between the latent variables and the resulting mesh. Furthermore, the proposed approach is scalable in terms of the number of factors to be modeled, requiring only sparsely labeled data and a new discriminator for each additional factor.

A key challenge is how to best represent the data to enable stable training of GANs. While current deep learning techniques have shown impressive results in the image domain, extending these to 3D data is not straightforward. We propose here a novel 3D-2D architecture in which a multilayer perceptron (MLP) generates the 3D face shape given a latent code, while a regular convolutional network is used as a 2D discriminator. This is allowed by an intermediate geometry mapping layer that transforms a 3D surface mesh into a geometry image encoding the mesh vertex locations.

To effectively decouple the factors of variation we build on auxiliary classifiers [START_REF] Odena | Conditional Image Synthesis with Auxiliary Classifier GANs[END_REF] whose task is to correctly guess the label associated with each factor (e.g."happy" expression), and introduce a loss on the classifier features for unlabeled samples. Comparisons with recent approaches based on autoencoder architectures [START_REF] Fernández Abrevaya | Multilinear autoencoder for 3d face model learning[END_REF][START_REF] Ranjan | Generating 3d faces using convolutional mesh autoencoders[END_REF] demonstrate that the proposed model can better decouple identity and expression, and exhibit more variability in the generated data.

In summary, this chapters contributes: 2. A concurrent work was published that tackles this, see [START_REF] Jiang | Disentangled representation learning for 3d face shape[END_REF].

A new generative 3D face model that captures non-linear deformations

due to expression, as well as the relationship between identity and expression subspaces.

2. A novel 3D-2D architecture that allows to generate 3D meshes while leveraging the discriminative power of CNNs, by introducing a geometry mapping layer that acts as bridge between the two domains.

3.

A training scheme that enables to effectively decouple the factors of variation, leading to significant improvements with respect to the state of the art.

Related Work

We focus in the following on closely related deep learning works for 3D face modeling and disentanglement. A more detailed review on classic data-driven models such as the 3D Morphable Model (3DMM) can be found in Section 2.3.

Autoencoders for 3D faces

Recent works leverage deep learning methods to overcome the limitations of (multi-)linear models. [START_REF] Ranjan | Generating 3d faces using convolutional mesh autoencoders[END_REF] proposed an autoencoder architecture that learns a single global model of the 3D face, and as such the different factors cannot be decoupled directly. However, an extension called DeepFLAME was proposed that combines a linear model of identity [START_REF] Li | Learning a model of facial shape and expression from 4D scans[END_REF] with the autoencoder trained on expression displacements. While expressions are modeled non-linearly, the relationship between identity and expression is not addressed explicitly. In Chapter 3 we developed the multilinear autoencoder (MAE) [START_REF] Fernández Abrevaya | Multilinear autoencoder for 3d face model learning[END_REF] in which the decoder is a multilinear tensor structure. While the relationship between the two spaces is accounted for, transferring expressions still presents artifacts. We compare our proposed approach to DeepFLAME and MAE, as they achieve state-of-the-art results on decoupling identity and expression variations. [START_REF] Bagautdinov | Modeling Facial Geometry using Compositional VAEs[END_REF] proposed a multiscale model of 3D faces at different levels of geometric detail. Two recent works [Tran et al., 2018, Tewari et al., 2018] use autoencoders to learn a global or corrective morphable model of 3D faces and their appearance based on 2D training data. However, none of these methods allow to disentangle factors of variation in the latent space. Unlike the aforementioned works, we investigate the use of GANs to learn a decoupled model of the 3D face.

GANs for 3D faces Some recent works have proposed to combine a 3DMM with an appearance model obtained by adversarial learning. Slossberg et al.

[2018] train a GAN on aligned facial textures and combine this with a linear 3DMM to generate realistic synthetic data. [START_REF] Gecer | Synthesizing coupled 3d face modalities by trunk-branch generative adversarial networks[END_REF] train a similar model and show that GANs can be used as a texture prior for accurate fitting to 2D images. [START_REF] Deng | Uv-gan: Adversarial facial uv map completion for pose-invariant face recognition[END_REF] fit a 3DMM to images and use a GAN to complete the missing parts of the resulting UV map. All of these methods rely on linear 3DMMs, and hence to shape spaces limited in expressiveness. While the focus is on improving the appearance, we follow a different objective with a generative shape model that decouples identities and expressions.

To the best of our knowledge, by the time of publication the only work that learned 3D facial shape variations using a GAN was Shamai et al. [2019], which is an extension of [START_REF] Slossberg | High quality facial surface and texture synthesis via generative adversarial networks[END_REF]. The authors proposed to learn identity variations by training a GAN on geometry images, but unlike our work they do not model the non-linear variations due to expression nor the correlation between identity and expression, since the main focus is on appearance. Recently, [START_REF] Moschoglou | 3dfacegan: Adversarial nets for 3d face representation, generation, and translation[END_REF] combined an autoencoder architecture with a GAN adversarial loss to learn shape variations, where geometry is again encoded in a UV map. Unlike these works, we propose to generate the shapes directly in the 3D domain, and use geometry images only for discrimination.

Two other methods learn to enhance an input 3D face geometry with photometric information using a GAN. Given a texture map and a coarse mesh, [START_REF] Huynh | Mesoscopic facial geometry inference using deep neural networks[END_REF] augment the latter with fine scale details, and given an input image and a base mesh, [START_REF] Yamaguchi | High-fidelity facial reflectance and geometry inference from an unconstrained image[END_REF] infer detailed geometry and high quality reflectance. Both works require the conditioning of an input, and unlike us they do not build a generative 3D face model.

Generative models with disentangled representations

The problem of learning disentangled representations has received considerable attention in the machine learning community, see e.g. [Bengio et al., 2013, van In their original form, GANs are unable to explicitly disentangle latent factors according to known features or attributes. Numerous works have been proposed that modify certain factors of an input image using GANs conditioned on an image and control labels, e.g. [START_REF] Usman | Puppetgan: Cross-domain image manipulation by demonstration[END_REF]. They all require explicit conditioning on a key factor (e.g.expression, rotation, lighting) as well as identity in the form of an input image, whereas we aim here to learn the latent spaces implicitly. To our knowledge only a few works decouple without conditioning on an input shape. [START_REF] Mathieu | Disentangling factors of variation in deep representation using adversarial training[END_REF] combine an encoder-decoder generator and a reconstruction loss with swapped latent vectors to disentangle identity, but only experimented with very low resolution Figure 5.1 -Proposed architecture. A MLP generates the 3D coordinates of the mesh, while discrimination occurs in 2D space thanks to the geometry mapping layer. Identity and expression codes z id , z exp are used to control the generator, and classification losses are added to decouple between the two. A feature loss is introduced to ensure consistency over features with fixed identities or expressions.

images. [START_REF] Donahue | Semantically decomposing the latent spaces of generative adversarial networks[END_REF] decouples by classifying pairs with a common identity. Neither of these are symmetrical with respect to the two factors to disentangle as they focus on preserving identity only. We propose here an alternative that succeeds in decoupling latent codes into a constant number of separate factors.

Background

Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF] are based on a minimax game, in which a discriminator D and a generator G are optimized for competing goals. The discriminator is tasked with learning the difference between real and fake samples, while the generator is trained to maximize the mistakes of the discriminator. At convergence, G approximates the real data distribution. Training involves the optimization of the following:

min G max D L GAN = E x∼p data [log D(x)] + E z∼p z [log(1 -D(G(z)))],
(

where p data denotes the distribution of the training set, and p z denotes the prior distribution for G, typically N (0, I ).

GANs have been shown to be very challenging to train with the original formulation and prone to low diversity in the generated samples. To address this, Arjovsky et al. [2017] propose to minimize instead an approximation of the Earth Mover's distance between generated and real data distributions, which is the strategy we adopt in this work:

L GAN = E x∼p data [D(x)] -E z∼p z [D(G(z))].
(5.2)

In particular we use the extension of [START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF] which uses a gradient penalty in order to enforce that D is 1-Lipschitz. When labels are available, using them has proven to be beneficial for GAN performance. [START_REF] Odena | Conditional Image Synthesis with Auxiliary Classifier GANs[END_REF] proposed the Auxiliary Classifier GANs (AC-GAN), in which D is augmented so that it outputs the probability of an image belonging to a pre-defined class label c ∼ p c . In this case, the loss function for G and D is extended with:

L real C = E x∼p data ,c∼p c [log P(C = c|x)], (5.3)
L f ake C = E z∼p z ,c∼p c [log P(C = c|G(z, c))].
(5.4)

In order to evaluate if a model is correctly decoupling, we need to be able to distinguish whether two identites or expressions sharing the same latent code are perceptually similar. Thus, our work builds on the idea of auxiliary classifiers in order to learn a decoupling of the shape variations into factors, as will be explained in the next section.

Method

We consider as input a dataset of registered and rigidly aligned 3D facial meshes, where each mesh is defined by (V , F ), the set of 3D vertices V ∈ R 3×n v and the set of triangular faces F ∈ N 3×n f that connect the vertices. Our goal is to build an expressive model that can decouple the representation based on known factors of variation. In contrast to classical approaches in which a reconstruction error is optimized, we rely instead on the adversarial loss enabled by a convolutional discriminator. To this end, we introduce an architecture in which a geometry mapping layer serves as bridge between the generated 3D mesh and the 2D domain, for which convolutional layers can be applied (Section 5.3.1). To learn a decoupled parameterization, we build on the idea of Auxiliary Classifiers and introduce a feature loss to further improve the results (Section 5.3.3). We will consider here a model that decouples between identity and expression, however the principle can be easily extended to more factors.

Geometry Mapping Layer

While deep learning can be efficiently used on regularly sampled signals, such as 2D pixel grids, applying it to 3D surfaces is more challenging due to their irregular structure. In this work we propose to generate the 3D coordinates of the mesh using a multilayer perceptron, while the discriminative aspects are handled in the 2D image domain. This allows to benefit from efficient and well established architectures that have been proven to behave adequately under adversarial training, while still generating the 3D shape in its natural domain.

In particular, a 2D representation of a mesh can be achieved through a UV parameterization φ : V → D that associates each vertex v k ∈ V with a coordinate (u, v) k in the unit square domain D. Continuous images can be obtained by interpolating the (x, y, z) vertex values according to the 2D barycentric coordinates, and storing them in the image channels. Borrowing the term from [START_REF] Gu | Geometry images[END_REF], we call this a geometry image (see Figure 5.2a).

Note that although our method could generate geometry images instead of 3D meshes, this would introduce an unnecessary additional reconstruction step that is likely to cause information loss and artifacts in the final meshes, as illustrated in Figure 5.2b. This is due to the fact that a single planar unfolding of a mesh may create distortions such as triangle flipping [START_REF] Sheffer | Mesh parameterization methods and their applications[END_REF], and a many-to-one mapping may be obtained even with a bijective parameterization due to the finite size of images. In addition, as elaborated in [START_REF] Gu | Geometry images[END_REF], unless border vertices are preassigned to distinct pixels which can be challenging for large meshes, sampling these locations results in erroneous interpolations. Generating 3D point coordinates instead allows to avoid reconstruction artifacts, and to apply common mesh regularization techniques that simplify and improve the learning process. We use geometry images only as the representation for the discriminative component that evaluates the 3D generator through CNNs.

The mapping layer operates as follows. Given a mesh made of vertices V = {v k /k = 1..n v }, a target image size n × n, and a pre-computed UV parameterization φ, we build two images I U , I V of dimension n × n, and three images I v 1 , I v 2 and I v 3 of dimension n × n × 3 each. For each pixel (i, j), we consider the φ-projected mesh triangle ( v1 , v2 , v3 ) containing it. The barycentric abscissa and ordinate of pixel (i, j) in triangle ( v1 , v2 , v3 ) are then stored in images I U and I V respectively, and the original face vertex coordinates v 1 , v 2 and v 3 are stored in images I v 1 , I v 2 and I v 3 . The mapping layer computes the output geometry image I as:

I = I U * I v 1 + I V * I v 2 + (1 -I U -I V ) * I v 3 ,
(5.5)

where * denotes element-wise multiplication and 1 ∈ R n×n is the matrix of ones. Since this layer simply performs indexing and linear combinations on the elements of V using the predefined parameters in I U and I V , all operations are differentiable and the gradients can be back-propagated from the discriminated image to the generated mesh. Convolving the UV map obtained through the geometry mapping layer corresponds to convolving the original mesh, such that the convolutional kernel covers the surface in a possibly un-even manner, as illustrated in Figure 5.3. This process allows to take advantage of the efficiency of regular grids while still generating shapes in 3D space, retaining knowledge of the surface connectivity.

Figure 5.3 -Illustration of mesh convolutions using the geometry mapping layer. The top row represents a UV map with a convolutional kernel being applied, while the bottom row shows the effect on the original mesh. The geometry mapping layer allows to backpropagate results from the image representation to the generated mesh.

Architecture

Figure 5.1 depicts our proposed architecture. The generator consists of two fully connected layers that map the latent code z to a vector of size 3n v containing the stacked 3D coordinates of displacements from a reference face mesh. The output vertex positions are passed through the mapping layer to generate a geometry image of size n × n, which is then processed by the discriminator in order to classify whether the generated mesh is real or fake. We also consider auxiliary classifiers for the discriminator, denoted as C id and C exp . The design of D shows two main differences with respect to the original AC-GAN. First, instead of classifying only one type of label, we use here classifiers for both identity and expression. This favors decoupling, since the classification of one factor is independent of the choice of parametrization for the other factors. Second, we provide distinct convolutional layers for the real/fake, identity and expression blocks. This is motivated by the observation that the 5.3. METHOD features required to classify identities and expressions are not necessarily the same.

Decoupled Model Learning

We rely on the discriminator not only to generate realistic faces, but also to decouple the factors of variation. For this, we optimize D such that it maximizes

L D = L GAN + λ C (L ID + L EXP ).
(5.6)

Here, L GAN denotes the standard adversarial loss (see Equation 5.2), and L ID , L EXP the classification losses measured against the labels provided with the dataset and weighted by scalar λ C . These losses are defined similarly to Equation 5.3 as:

L ID = E x∼p data ,c∼p id c [log P(C = c|x)], L EXP = E x∼p data ,c∼p exp c [log P(C = c|x)], (5.7)
where p id c and p exp c denote the distribution of identity and expression labels, respectively. We ignore the sample contribution in the classification loss if it is not labeled.

The generator G takes as input a random vector z = {z id , z exp , z noise }, which is the concatenation of the identity code z id ∼ p id , the expression code z exp ∼ p exp and a random noise z noise ∼ p noise . It produces the location of n v displacement vectors from a reference mesh, and is trained by minimizing:

L G = λ 1 L GAN -λ 2 L id CL + L exp CL +λ 3 L id FEAT + L exp FEAT + λ 4 L reg , (5.8)
where L GAN is the standard GAN loss (Equation 5.2); L id CL and L exp CL are classification losses; L id FEAT and L exp FEAT are feature losses that aim to further increase the decoupling of the factors; L reg is a regularizer; and λ 1 , λ 2 , λ 3 , λ 4 are weights for the different loss terms. We explain each of these in the following.

Classification Loss

In addition to the adversarial loss, the generator is trained to classify its samples with the correct labels by maximizing:

L id CL = E z∼p z ,c∼p id c [log P(C = c|G(z))], L exp CL = E z∼p z ,c∼p exp c [log P(C = c|G(z))].
(5.9)

In order to generate data belonging to a specific class, we sample one identity/expression code z id , z expr for each label and fix it throughout the training; this becomes the input for G each time the classification loss must be evaluated. We denote the set of fixed codes for identity and expression as T id and T exp respectively.

Feature Loss

The classification loss is limited to codes in T id /T exp , which have associated labels. We found that better decoupling results can be obtained if we include a loss on the classifier features. We measure this by generating samples in pairs which share the same identity or expression vector, and measuring the error as:

L id FEAT = 2 N z id 1 -cos(f 1,z id , f 2,z id ) , (5.10)
L exp FEAT = 2 N z exp 1 -cos(f 1,z exp , f 2,z exp ) . (5.11)
Here, N is the batch size, and f i,z id = f G(z id , z exp,i , z noise,i ) are feature vectors obtained by inputting the sample G(z id , z exp,i , z noise,i ) through the classifier C id and extracting the features from the second to last layer. That is, given two inputs which were generated with the same identity vector, L id FEAT enforces that their feature vectors in the identity classifier are also aligned. The definition is analogous for f i,z exp with C exp .

To enable training with both classification and feature loss, for each batch iteration we alternate between the sampling of labeled identity codes z id ∈ T id with unlabeled expression codes z exp ∼ p exp , and the sampling of unlabeled identity codes z id ∼ p id with labeled expression codes z exp ∈ T exp . The classification is evaluated for the labeled factor only, while the feature loss is used for unlabeled codes, and the alternation allows to better cover the identity and expression sub-spaces during training.

Regularization Generating a 3D mesh allows us to reason explicitly at the surface level and define high order loss functions using the mesh connectivity. In particular, we enforce spatial consistency over the generated faces by minimizing the following term on the output displacements v = G(z):

L reg = ||Lv|| 2 2 , ( 5.12)
where L is the cotangent discretization of the Laplace-Beltrami operator [START_REF] Meyer | Discrete differentialgeometry operators for triangulated 2-manifolds[END_REF].

Evaluation

We provide in this section results obtained with the proposed framework, which demonstrate its benefits particularly in decoupling. We first clarify our set-up with implementation details in Section 5.4.1 and the datasets used in 5.4.2. We explain in Section 5.4.3 the proposed metrics for the evaluation of a 3D face model, and introduce a new measure for analyzing the diversity of the generated samples. In Section 5.4.4 we perform ablation studies to verify that all the components are necessary to effectively train an expressive model. Finally, in Section 5.4.5 we compare our results to state-of-the-art 3D face models that can decouple the latent space, and show that our approach outperforms with respect to decoupling and diversity.

Implementation Details

We set the weights to λ C = 0.1 (Equation 5.6), λ 1 = λ 2 = 1, λ 3 = 0.5 and λ 4 = 100 (Equation 5.8). The classification losses are further weighted to account for imbalanced labels [START_REF] King | Logistic regression in rare events data[END_REF]. For the generator, we use two fully connected layers with an intermediate representation of size 512 and ReLU non-linearity. For the discriminator we use a variant of DC-GAN [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF], with the first two convolutional blocks shared between C real , C id and C expr , while the remaining are duplicated for each module (more details can be found in Appendix A.1). The models were trained for 200 epochs using ADAM optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with β 1 = 0.9 and β 2 = 0.999, a learning rate of 0.0002 and a batch size of 64. During training we add instance noise [START_REF] Sønderby | Amortised map inference for image super-resolution[END_REF] with σ = 0.1 to the input of D. The discriminator is trained for 3 iterations each time we train the generator. The models take around 2 hours to train on a NVidia GeForce GTX 1080 GPU.

The template mesh contains 22129 vertices. We pre-compute the UV map φ using harmonic parameterization [START_REF] Eck | Multiresolution analysis of arbitrary meshes[END_REF], setting the outer boundary face vertices to a unit square to ensure full usage of the image domain. We generate geometry images of size 64 × 64; we experimented with other image sizes but the best decoupling results were obtained with this resolution. The dimensions for (z id , z exp , z noise ) are set to (65, 15, 5) to facilitate comparison with Fernández Abrevaya et al. [2018], and the feature vectors used in Equations 5.10 and 5.11 are of size 2048.

Datasets

All models were trained using a combination of four publicly available 3D face datasets. In particular, we use two datasets containing static 3D scans of multiple subjects: BU-3DFE [START_REF] Yin | A 3d facial expression database for facial behavior research[END_REF] and Bosphorus [START_REF] Savran | Bosphorus database for 3d face analysis[END_REF], and combine these with two datasets of 3D motion sequences of multiple subjects: BP4D-Spontaneous [START_REF] Zhang | Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database[END_REF] and BU-4DFE [START_REF] Yin | A high-resolution 3d dynamic facial expression database[END_REF]. The static datasets provide variability of identities, while the motion datasets provide variability of expressions and a larger number of training samples. We registered BU-3DFE and Bosphorus with a template fitting approach [START_REF] Salazar | Fully automatic expressioninvariant face correspondence[END_REF], and the motion datasets with the spatiotemporal approach introduced in Chapter 4.

The final dataset contains 30559 registered 3D faces and was obtained by subsampling the motion sequences. We provide identity labels for all meshes, while the expression labels are limited to the seven basic emotional expressions, which appear in both static datasets. For BU-4DFE, expression labels are assigned to three frames per sequence: the neutral expression to the first and last frame, and the labeled expression of the sequence to the peak frame. For BP4D, one neutral frame is manually labeled per subject (this is a requirement for comparison to [START_REF] Ranjan | Generating 3d faces using convolutional mesh autoencoders[END_REF]). Overall, due to the use of motion data, only 7% of it is assigned expression labels.

Evaluation Metrics

We evaluate the models in terms of diversity of the generated samples, decoupling of identity and expression spaces, and specificity to the 3D facial shape. We believe it is necessary to simultaneously consider all the metrics, as they provide complementary information on the model. For instance, a good decoupling value can be obtained when the diversity is poor, since small variations facilitate the classification of samples as "same". Conversely, a large diversity value can be obtained when decoupling is poor, since the identities/expressions sharing the same code can yield very different shapes. We detail these in the following.

Diversity

We consider it important to measure the diversity of the 3D face shapes generated by a model, particularly with GANs that are known to be prone to mode collapse. To the best of our knowledge, this has not yet been considered in the context of 3D face models and we propose therefore to evaluate as follows. We sample p pairs of randomly generated meshes and compute the mean vertex distance among the pairs; diversity is then defined as the average of distances over the p pairs. We expect here to see higher values for more diverse models. We evaluate on three sets of sampled pairs:

(1) among pairs chosen randomly (global diversity), (2) among pairs that share the same identity code (identity diversity) and (3) among pairs that share the same expression code (expression diversity). For all cases we evaluate on 10000 pairs. For comparison, the training set is also evaluated on these three metrics by leveraging the labels.

Decoupling To evaluate decoupling in both identity and expression spaces we follow the protocol proposed in [START_REF] Donahue | Semantically decomposing the latent spaces of generative adversarial networks[END_REF]. In particular, we first train two networks, one for identity and one for expression, that transform an image representation of the mesh to an n-dimensional vector using triplet loss [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF], where n = 128 in our experiments. The trained networks allow to measure whether two meshes share the same identity or expression by checking whether the distance between their embeddings is below a threshold τ.

To measure identity decoupling, we generate n random faces x i = G(z i id , z i exp , z i noise ), and for each random face we fix the identity code and sample m faces Y (x i ) = {G(z i id , z j exp , z j noise ), j = 1..m}. We then use the embedding networks to evaluate whether the original faces x i and their corresponding samples in Y (x i ) correspond to the same identity, and report the percentage of times the pairs were classified as "same". We proceed analogously for expression decoupling. We set n = 100, m = 100, τ = 0.14 for identity and τ = 0.226 for expression; more implementation details are given in Appendix A.2. Specificity Specificity is a metric commonly used for the evaluation of statistical shape models [START_REF] Davies | Statistical models of shape: Optimisation and evaluation[END_REF] and whose goal is to quantify whether all the generated samples belong to the original shape class, faces in our case. For this, n samples are randomly drawn from the model and for each the mean vertex distance to each member of the training set is measured, keeping the minimum value. The metric then reports the mean of the n values. We use here n = 1000.

Ablation Tests

We start by demonstrating that each of the proposed components is necessary to obtain state-of-the-art results according the metrics previously defined. To this end, we compare our approach against the following alternatives: (1) without mesh regularization (Equation 5.12); (2) with identity classification only; (3) with expression classification only; and (4) without feature loss (Equations 5.10 and 5.11).

Table 5.1 gives the evaluation metrics for each of these options, and Figure 5.4 provides qualitative examples. From the results we observe that: (1) The mesh regularization is crucial to generate samples that are realistic facial shapes. This is reflected by a very large value in specificity as well as low diversity, due to the fact that the model never converged to realistic faces (see Figure 5 Considering classification in only one factor significantly reduces the capacity of the model to preserve semantic properties in the other factor, as indicated by the very low decoupling values obtained in the corresponding rows. This justifies the use of classifiers for each of the factors. (3) Without the feature loss the model can still achieve good results, but both expression decoupling and diversity are lower than with the full model and the inclusion of the feature loss improves expression classification by almost 10%. Note that decoupling the expression space is significantly more challenging than identity, as the provided labels are very sparse. This effect is illustrated on Figure 5.4c, where models with the same expression code can lead to faces with slightly different expressions. Our approach provides more coherent faces, as shown in Figure 5.4d.

Comparisons

We compare the proposed approach against state-of-the-art generative 3D face models. Our goal is to build a decoupled latent space, and thus we focus the comparison to works that either enforce this explicitly [START_REF] Fernández Abrevaya | Multilinear autoencoder for 3d face model learning[END_REF], or combine a model trained on expressions with a linear space of identities [START_REF] Ranjan | Generating 3d faces using convolutional mesh autoencoders[END_REF], Amberg et al., 2008]. We train all models using the same dimensions (65 for identity and 20 for expression).

The model proposed in Fernández Abrevaya et al. [2018], called MAE in the following, was trained with the same dataset and the same label information (Section 5.4.2) for 200 epochs, with the default parameters given in the paper. We initialize the encoder and the decoder from the publicly available models. alternative [START_REF] Li | Learning a model of facial shape and expression from 4D scans[END_REF], which we also train with the same dataset. This results in a PCA model built from 299 identities and an autoencoder trained on 30330 displacements from the corresponding neutral face. For the identity space we manually selected one neutral frame for each sequence in BP4D-Spontaneous, as this dataset does not provide labels. The model was trained using the publicly available code for 200 epochs. We also trained an additive linear model as described in Amberg et al. [2008] using our dataset, and the same neutral/expression separation selected for CoMA (see above). We refer to this model as 3DMM.

Model quality

We show quantitative results with respect to decoupling, diversity and specificity in the bottom of Table 5.1. Note that the proposed approach significantly outperforms the others in terms of expression decoupling, which is more challenging than identity due to the sparse labeling. This is shown qualitatively in Figure 5.5, where we transferred expressions by simply exchanging the latent code z exp . We can see here that the expression is well preserved by our model.

With respect to identity decoupling the four methods perform similarly well, with 3DMM achieving the highest value. Note that, in the case of MAE, the large decoupling value is combined with the lowest diversity in identity (Div-Id), which suggests limited generative capabilities.

Our model outperforms all methods in terms of diversity.

||ṽ i (z) -v i || 2 2 + λ||z|| 2 2 , ( 5.13)
where v i are the 3D locations of the p key-points in the testing set, ṽi (z) are the corresponding key-points in the face model generated with code z, and λ the regularization weight. We optimize using a gradient descent approach [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] starting from a randomly sampled code z. Note that this is a challenging scenario since the training set does not contain such expressions, and the correspondences are very sparse. We compare our results with those obtained with 3DMM, MAE and CoMA, using the same optimization for all methods. We measure the reconstruction error against the ground-truth surface and report the RMSE. Quantitative results can be found in Table 5.2 for different regularization weights λ. Our method outperforms in all cases, including without regularization (λ = 0). We found that our model can produce reasonable faces in most cases, while MAE and CoMA easily produce un-realistic faces when the regularization is not strong enough. Qualitative examples can be seen in Figure 5.9a.

Extension to other factors

One of the benefits of our framework lies in its ability to easily extend to other factors of variation. As an illustration, we trained a model that decouples identity, expression and viseme (the visual counterpart of a phoneme). The results can be found in We trained the model using the audiovisual 3D dataset of Fanelli et al. [2010], which contains 14 subjects performing 40 speech sequences in neutral and "expressive" mode. We assign phoneme labels using the Montreal Forced Aligner tool [START_REF] Mcauliffe | Montreal forced aligner: Trainable text-speech alignment using kaldi[END_REF] with the provided audio, which are mapped to visemes following [START_REF] Neti | Audio visual speech recognition[END_REF]. For expression, we manually labeled 699 frames with the aid of the provided expression ratings of each sequence. This resulted in a database with 100% labeled identites, 68% labeled visemes, and 3% labeled expressions. We set the latent dimensions to (50, 50, 50, 5) for identity, expression, viseme and noise, respectively.

Note that this is a simplified model of speech, since the temporal information is not taken into account. Yet, we can see in Figure 5.6 that a decoupling between the aspects affected by phoneme production, and those affected by expressions such as happiness or surprise can be easily distinguished by our framework. Note for example the change in eye expression and the subtle mouth movements that occur to accommodate the viseme under different expressions. It is also worth noting that these results were obtained with fully automatic labels for viseme, and very sparse manual labels for expression, thus simplifying the efforts required to annotate the dataset. Unlike the identity and expression factors, which are intuitively easier to separate, the viseme and expression factors are more intertwined and decoupling them is very challenging even for a human annotator. In spite of this, our results show that we can reasonably decouple the three factors.

Latent Space Manipulation

Thanks to the decoupling of identity and expression spaces, we can synthesize new expressions by simple manipulation of the latent space. We show here two possibilities for this.

Given a source mesh obtained with G(z src id , z src expr , z src noise ) and a target mesh obtained with G(z Finally, we show that the latent space is smooth with an example of interpolation and extrapolation in Figure 5.8.

Source

Target Replaced Added Target Replaced Added Figure 5.7 -Example of expression space manipulation. In gray a source mesh and a target mesh. In purple the result of (1) replacing the expression code of the target with that of the source (replaced), and (2) adding the source and target expression codes (added).

Conclusion

We explored in this work the use of adversarial training for learning decoupled 3D facial models. Our results show that purely discriminative losses are well suited for the decoupling task, achieving state-of-the-art performance in terms of decoupling and diversity of the generated samples. Although the expressiveness of the model remains limited by the diversity of the training data and the accuracy of its labels, we show that adversarial learning has strong potential in building performant 3D facial models.

Our framework is enabled by a novel method for processing 3D data using deep learning approaches, namely the 3D-2D architecture. This architecture allows to benefit from advances in 2D convolutional networks while still generating the data in the 3D domain. The results of this work suggest that such 3D-2D approach is a viable alternative to other mesh processing frameworks, e.g. [START_REF] Verma | Feastnet: Feature-steered graph convolutions for 3d shape analysis[END_REF]. Exploring its true capacity in light of other tasks is an interesting direction that we leave for future work.

GANs have shown an impressive ability to retain fine-scale details in 2D facial images. Unfortunately, due to the lack of proper training data the present work does not answer the question of whether similar progress can be obtained in 3D through the proposed approach. Yet, we believe that it should be possible given the proper data and enough network capacity, and leave this too as a future direction. The previous chapters dealt with the global properties of the face: how to model and track the geometry of the 3D shape and motion, as well as the interactions between the identity and expression subspaces. In this final chapter we consider a slightly different but related problem: how to estimate local geometric details. To this end, we depart from the global shape models used until now and build instead a prior that can simultaneously encode information about the facial normals and natural face images. The goal here is to estimate accurate surface normals from images in-the-wild, which can be used to enhance coarser estimations from parametric models as illustrated in Figure 6.1. 3D reconstruction of the human face is a long-standing problem in computer vision, with a wide range of applications including biometrics, forensics, animation, gaming, and human digitalization. In many of these applications monocular inputs are considered in order to limit the acquisition constraints, Figure 6.1 -Our model predicts accurate normals from a single input image that can be used to enhance a coarse geometry (e.g. PRN [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF]).

CHAPTER 6. ESTIMATING 3D FACE NORMALS FROM NATURAL IMAGES

hence enabling uncontrolled environments as well as efficient information usage for e.g.facial telecommunication and entertainment. Although significant progress has been recently made by the scientific community, recovering detailed 3D face models given only single images is still an open problem.

Monocular face reconstruction is in essence an ill-posed problem which requires strong prior knowledge. Assuming a simple shading model, seminal shape-from-shading (SfS) approaches [START_REF] Horn | Shape from shading[END_REF]Brooks, 1989, Zhang et al., 1999] were estimating shape normals by considering local pixel intensity variations. Fine scale surface details can be recovered using this strategy, however the applicability to in-the-wild images is limited by the simplified image formation model that is assumed. Later on, a more global strategy was proposed with parametric face models, such as the ones considered in the previous chapters. They allow fitting a template face controlled by only a few coefficients, resulting hence in improved robustness. While being widely adopted, parametric models are inherently restricted in expressiveness and have difficulties in recovering small surface details, as a consequence of their low dimensional representation. Recently, deep learning methods that exploit large-scale face image datasets have been investigated with the aim of better generalization. While most works in this category are trained to estimate the coefficients of a parametric model [START_REF] Tewari | MoFa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction[END_REF], 2018[START_REF] Genova | Unsupervised training for 3d morphable model regression[END_REF], Kim et al., 2018a[START_REF] Sanyal | Learning to regress 3D face shape and expression from an image without 3D supervision[END_REF], a few other approaches infer directly perpixel depth [Sela et al., 2017], UV position maps [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF] or surface normals [START_REF] Trigeorgis | Face normals" in-the-wild" using fully convolutional networks[END_REF], Sengupta et al., 2018].

As observed in previous work [Smith et al., 2019, Zhang and[START_REF] Zhang | Deep depth completion of a single rgb-d image[END_REF], regressing depth information alone can lead to suboptimal results, especially detail-wise, as the inherent scale ambiguity with single images can make convergence difficult for neural networks. On the other hand, the estimation of normals appears to be an easier task for such networks, given that normals are strongly correlated to pixel intensities and depend mostly on local information, a fact already exploited by SfS techniques. Still, only a few approaches have been proposed in this line for facial images [START_REF] Shu | Neural face editing with intrinsic image disentangling[END_REF], Sengupta et al., 2018], mostly due to the limited available ground-truth data. We propose here a method that overcomes this limitation and can leverage all data available through the use of cross-modal learning. Our experiments demonstrate that this strategy can estimate more accurate and sharper facial surface normals from single images.

The proposed approach recovers accurate normals corresponding to the facial region within an RGB image, with the goal of enhancing an existing coarse reconstruction, [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF] in our experiments. We cast the problem as a color-to-normal image translation, which can be in principle solved by combining an image encoder E I with a normal decoder D N as in Trigeorgis et al.

[2017], and including skip connections between E I and D N [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] in order to transfer details from the image domain to the normals domain. However, training such a network can prove difficult unless a large dataset of image/normal pairs, that ideally contains images in-the-wild, is available. In practice few such datasets are currently publicly available, e.g. [START_REF] Zafeiriou | The photoface database[END_REF], which were moreover captured under controlled conditions. To improve generalization, we propose to augment the architecture with a normal encoder E N and an image decoder D I , where all encoders/decoders share the same latent space. This augmented architecture provides additional constraints on the latent space with the auto-encoded image-to-image and normal-to-normal branches, effectively building a prior over both realistic facial shapes and realistic facial images. In order to keep advantage of the skip connections between E I and D N , while avoiding the resulting bonded connections between E N with D N that hamper the architecture, we introduce the deactivable skip connections. This allows skip connections to be turned on and off during training according to the type of data.

To summarize, we contribute in this chapter:

1. A framework that leverages cross-modal learning for the estimation of normals from a single face image in-the-wild.

2. The introduction of the deactivable skip connection.

3. An extensive evaluation that shows that our approach outperforms state-of-the-art methods on the Photoface [START_REF] Zafeiriou | The photoface database[END_REF] and Florence [START_REF] Bagdanov | The florence 2d/3d hybrid face dataset[END_REF] datasets, with up to nearly 10% improvements in angular error on the Florence dataset, as well as visually compelling reconstructions.

Related Work

We focus the discussion below on methods that consider 3D face reconstruction, or normal estimation, given single RGB images.

Reconstruction with Parametric Models 3D reconstruction from a single image is ill-posed and many methods resort therefore to strong priors with parametric face models such as blendshape [START_REF] Garrido | Reconstructing detailed dynamic face geometry from monocular video[END_REF][START_REF] Cao | Real-time high-fidelity facial performance capture[END_REF][START_REF] Thomas | Augmented blendshapes for real-time simultaneous 3d head modeling and facial motion capture[END_REF] or statistical models, typically the 3D Morphable Model (3DMM) [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF]. These models are commonly used within an analysis-by-synthesis optimization [START_REF] Romdhani | Estimating 3d shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior[END_REF] or the full image decomposition [START_REF] Shu | Neural face editing with intrinsic image disentangling[END_REF], Sengupta et al., 2018, Kim et al., 2018b]. This strategy has proven robustness, however it is constrained by the parametric representation that offers limited expressiveness and fails in recovering fine scale details. IMAGES In order to improve the quality of the reconstructions several works have proposed to add medium-scale correctives on top of the parametric model [START_REF] Li | Realtime facial animation with on-the-fly correctives[END_REF], Garrido et al., 2016a, Tewari et al., 2018], to train a local wrinkle regressor [START_REF] Cao | Real-time high-fidelity facial performance capture[END_REF], or to learn deep non-linear 3DMMs [START_REF] Tran | Towards high-fidelity nonlinear 3d face morphable model[END_REF][START_REF] Zhou | Dense 3d face decoding over 2500fps: Joint texture & shape convolutional mesh decoders[END_REF]] that can capture higher-frequency details. Our method also enables to enhance a face prediction through the estimation of more accurate normals.

Normal Estimation with Shape from Shading

Shape-from-shading (SfS) [START_REF] Horn | Shape from shading[END_REF]Brooks, 1989, Zhang et al., 1999] is a well-studied technique that aims at recovering detailed 3D surfaces from a single image based on shading cues. It estimates surface normals using the image irradiance equation, as well as illumination model parameters when these are unknown. SfS is inherently limited by the simplified image formation model assumed but has inspired numerous works that build on the correlation between pixel intensity and normals, either explicitly or implicitly. For instance, a few works on faces combined SfS with a data-driven model, e.g. [START_REF] Smith | Recovering facial shape using a statistical model of surface normal direction[END_REF], Kemelmacher-Shlizerman and Basri, 2010, Snape and Zafeiriou, 2014], which helps to avoid some of the limitations such as ill-posedeness and ambiguities e.g. [START_REF] Belhumeur | The bas-relief ambiguity[END_REF] [START_REF] Eigen | Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture[END_REF] simultaneously regress depth, normal and semantic segmentation using a multi-scale approach. [START_REF] Zhang | Deep depth completion of a single rgb-d image[END_REF] predict surface normal and occlusion boundaries to later optimize for depth completion; a similar direction was followed by [START_REF] Qiu | Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image[END_REF] for outdoor scenes. [START_REF] Trigeorgis | Face normals" in-the-wild" using fully convolutional networks[END_REF] estimate facial normals with a supervised approach trained on synthetic data. Our approach differs from the aforementioned methods with a new architecture that enables cross-modal learning, hence improving performances in monocular 3D face normal estimation.

Geometry Enhancement using Deep Networks Methods have been proposed

Method

We propose to predict face normals from a single color image using a deep convolutional encoder-decoder network. A natural solution to this purpose is to combine an image encoder E I with a normal decoder D N , as in e.g. [START_REF] Trigeorgis | Face normals" in-the-wild" using fully convolutional networks[END_REF]. However training such an architecture requires pairs of normal and color images in correspondence. Although a few public datasets are available that contain high-quality 3D or normal ground-truth information for faces, for instance ICT-3DRFE [START_REF] Stratou | Effect of illumination on automatic expression recognition: a novel 3d relightable facial database[END_REF] or Photoface [START_REF] Zafeiriou | The photoface database[END_REF], they were obtained under controlled conditions and do not, therefore, really cover the distribution of images in-the-wild. On the other hand, numerous large datasets of natural images are publicly available, for example CelebA [START_REF] Liu | Deep learning face attributes in the wild[END_REF] and AffectNet [START_REF] Mollahosseini | Affectnet: A database for facial expression, valence, and arousal computing in the wild[END_REF]], yet without the associated accurate and detailed ground-truth normal values. Whereas other works have approached this by augmenting the training corpus with synthetic groundtruth [START_REF] Trigeorgis | Face normals" in-the-wild" using fully convolutional networks[END_REF], Sengupta et al., 2018], we propose instead a method based on cross-modal learning that can leverage all available data, even unpaired.

Cross-modal Architecture

As depicted in Figure 6.2, we use two encoder/decoder networks, one for images E I /D I and one for normals E N /D N , sharing the same latent space. This architecture is trained with image-to-image, normal-to-normal and image-tonormal supervision simultaneously in order to obtain a robust and rich latent representation. To this purpose, we exploit paired images of normal and color information on faces, as available from [START_REF] Stratou | Effect of illumination on automatic expression recognition: a novel 3d relightable facial database[END_REF], Zafeiriou et al., 2011], in addition to individual images of either color or normal information, from e.g.CelebA-HQ [START_REF] Karras | Progressive growing of gans for improved quality, stability, and variation[END_REF] and BJUT-3D [bju, 2005]. To improve the overall performance we augment this architecture with long skip connections between E I and D N , as it favors the transfer of details between the image and normal domains, and since it has been shown to significantly increase performance in several image translation tasks e.g. [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF]. In practice we use a U-Net+ResNet [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF][START_REF] He | Deep residual learning for image recognition[END_REF] architecture that combines the benefits of both short and long skip connections.

Training such an architecture end-to-end raises an obstacle: the skip connections from E I to D N (E I → D N ), which are based on concatenating feature maps, impose by construction to also have skip connections between the encoder and decoder of the normal modality, i.e. E N → D N . This is counterproductive in practice: by setting skip connections within the same modality, it is in fact easier for the normal autoencoder to transfer features from the earliest layers of its encoder to the last layers of its decoder through the skip connection, thus depriving the deeper layers of any meaningful gradients during training. Not only will this fail to improve the latent face representation, but it will also alter the coefficients of the normal decoder for the image-to-normal inference task.

For this reason, we introduce the deactivable skip connections as shown in Figure 6.3 and detailed in Section 6.2.2. This allows us to train the framework end-to-end by setting long connections solely between E I and D N , thus learning a rich latent space that encodes facial features from both color and normal images while profiting from all available data.

Deactivable Skip Connections

As mentioned earlier, skip connections are well suited to our problem as they allow sharing of low-level information at multiple scales while still preserving the general structure. In the implementation of standard skip con- ) and the encoder features at layer i, F i E , where n is the total number of layers (see Figure 6.3a).

Let m(F i E I ) be the number of feature maps at the i th layer of E I . The proposed architecture (Figure 6.2) requires to set connections from the image encoder E I to the normal decoder D N , and as a consequence, each layer features F n-i D N of D N are expected to always have an additional m(F i E I ) channels. In order to gain generalization over each domain, both the color and the normal images can be auto-encoded during training. However, since the concatenation is expected during training on the decoder D N side, features of the normal encoder E N must be concatenated as well, which as discussed is detrimental to our model.

The deactivable skip connections are designed such that, during training, the transfer of feature maps from encoders to decoders can be selectively activated or deactivated. Compared to a decoder equipped with standard skip connections, the processed features f (F n-i-1 D ) of our decoder include m(F i E ) extra channels (light blue in Figure 6.3b). During a normal-to-normal pass, the skip connections are deactivated and the (ni) th layer features of the 92 CHAPTER 6. ESTIMATING 3D FACE NORMALS FROM NATURAL IMAGES normal decoder correspond to the processed previous layer features e.g.F n-i D = f (F n-i-1 D ). During an image-to-normal pass, the skip connection is activated: we first perform an element-wise max-pooling between the i th layer features of the encoder F i E and the last m(F i E ) channels of the processed (ni -1) th layer features of the decoder f (F n-i-1 D ), as illustrated in Figure 6.3b. The result is stacked back with the remaining of the processed previous layer features thus forming the final (n-i) th decoder layer features F n-i D . Doing so allows to transfer the information from encoder to decoder without degrading performances when the transfer operation does not occur, as when auto-encoding normals.

Training

We train the framework end-to-end using both supervised and unsupervised data, where the latter includes individual image and normal datasets.

During training, the skip connections are deactivated when doing a normalto-normal pass. For the supervised case, and for unsupervised normals, the loss function is the cosine distance between the output and the ground-truth, which in our experiments gave better results than the L1/L2 norm:

L nrm (N , N ) = 1 - 1 |N | (i,j) N (i, j) ⊤ • N (i, j) ||N (i, j)|| 2 || N (i, j)|| 2 , ( 6.1)
where N (i, j) and N (i, j) are the normal vectors at pixel (i, j) in the ground-truth and output normal images N and N respectively, and |N | is the number of pixels in N . For unsupervised image data we use the L2 loss:

L img (I , Î) = ||I -Î || 2 2 , (6.2)
where Î is the output color image and I the ground-truth. In both cases, the loss is applied only on facial regions segmented using masks obtained as explained in Section 6.3.1.

In practice, as we can only perform a training iteration for one input modality at a time, either an input batch of images or normals, we train our model as follows: when loading a batch of images with image/normal ground-truth pairs, we perform a normal-to-normal iteration first, followed by an image-tonormal plus image-to-image iteration, where both losses in the latter iteration are summed with equal weights. When loading a batch of images only, we perform an image-to-image iteration. Finally, with a batch of normals only, we naturally proceed with a normal-to-normal iteration alone.

Evaluation

We report below on the accuracy of the normals estimated with our approach on standard datasets [START_REF] Zafeiriou | The photoface database[END_REF][START_REF] Bagdanov | The florence 2d/3d hybrid face dataset[END_REF]. We compare against state-of-the-art methods on normal estimation and 3D reconstruction, and show significant improvements in terms of normal prediction accuracy. This is supported by compelling reconstructions of images in-the-wild from 300-W [START_REF] Sagonas | 300 faces in-the-wild challenge: The first facial landmark localization challenge[END_REF], as can be seen in Figures 6.4 and 6.5.

Following previous work [START_REF] Sengupta | Sfsnet: Learning shape, reflectance and illuminance of facesin the wild[END_REF], Trigeorgis et al., 2017], we use as metric the mean angular error between the output and the ground-truth normals, as well as percentage of pixels within the facial region with an angular error of less than 20 • , 25 • and 30 • . For qualitative comparisons we show both the output normal map, as well as the mesh results obtained by enhancing the output of PRN [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF] using normal mapping [START_REF] Cohen | Appearance-preserving simplification[END_REF]]: we append the predicted normals to the the PRN mesh thus rendering enhanced geometric shading.

Implementation Details

The framework was implemented in PyTorch [Paszke et al., 2019], and all experiments were run on a GTX TITAN Black. The networks were trained for 40 epochs using ADAM solver [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with a learning rate of 10 -4 . We use a ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF] architecture and set five skip connections, one at the output of the initial layer and the rest at the output of each of the four residual blocks. Each mini-batch during training consists of data of the same type, i.e.images only, normals only or image-normal pairs, as this worked best for us empirically.

Similar to prior work, input images are crops of fixed size around the face. We extract 2D keypoints with a face detector [START_REF] King | Dlib-ml: A machine learning toolkit[END_REF] and create masks on the facial region by finding the tightest square of edge size l around the convex hull of the points. The images are then cropped with a square patch of size 1.2 × l centered at the same 2D location as the previously detected box, and subsequently resized to 256 × 256.

Datasets

Our training set comprises multiple datasets: ICT-3DRFE [START_REF] Stratou | Effect of illumination on automatic expression recognition: a novel 3d relightable facial database[END_REF] and Photoface [START_REF] Zafeiriou | The photoface database[END_REF] which provide image/normal pairs, CelebA-HQ [START_REF] Karras | Progressive growing of gans for improved quality, stability, and variation[END_REF] which only contains 2D images, and BJUT-3D [bju, 2005], which consists of high-quality 3D scans.

We generated 8625 image/normal pairs from ICT-3DRFE by randomly rotating the 345 3D models and relighting them using the provided albedos. We sampled random rotation axes and angles in [-π/4, π/4], random lighting directions with positive z, and random intensities in [0, 2]. For Photoface, following the setting in [START_REF] Trigeorgis | Face normals" in-the-wild" using fully convolutional networks[END_REF], Sengupta et al., 2018], we randomly selected a training subset of 353 people resulting in 9478 image/normal pairs. We also generated 5000 high resolution facial images from CelebA-HQ, which is used to train the image-to-image branch exclusively. In addition, we render 3000 normal images from the 500 scans of BJUT-3D, rotated with random axes and angles in [-π/4, π/4]. We only render normal images from this dataset as the original scan color images are not provided.

Comparisons

We compare our results to methods that explicitly recover surface normals, either for facial images (SfSNet [START_REF] Sengupta | Sfsnet: Learning shape, reflectance and illuminance of facesin the wild[END_REF], NiW [START_REF] Trigeorgis | Face normals" in-the-wild" using fully convolutional networks[END_REF]) or for general scenes (Marr Rev [START_REF] Bansal | Marr revisited: 2d-3d alignment via surface normal prediction[END_REF], UberNet [START_REF] Kokkinos | Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory[END_REF]). We also compare against state-of-the-art approaches for 3D face reconstruction, namely the classic 3DMM fitting method used in Trigeorgis et al. Quantitative results can be found in Table 6.1 for Photoface and Table 6.2 for Florence datasets. We show results of our method both with (Ours-ft) and without (Ours) fine-tuning of the training split of Photoface in the upper and lower parts of Table 6 The proposed approach shows the best values both in mean angular error and percentage under 20 • , 25 • and 30 • degrees, only outperformed by 3DMM on errors under 30 • . As noted by the authors in [START_REF] Trigeorgis | Face normals" in-the-wild" using fully convolutional networks[END_REF], 3DMM fitting performs well under 30 • because of the coarseness of the model and the keypoint supervision, but its performance on tighter angles drops drastically as it lacks precision. We found that, although Sela et al. [2017] and Tran et al. [2018] usually provide seemingly detailed reconstructions, the actual normals of these methods lack accuracy as witnessed by their numbers.

Our good performance is also confirmed by qualitative comparisons over images in-the-wild in various head poses and under arbitrary lighting conditions as can be seen in Figures 6.4 and 6.5. For comparisons with mesh results (Figure 6.5), we show for both our approach and SfSNet [START_REF] Sengupta | Sfsnet: Learning shape, reflectance and illuminance of facesin the wild[END_REF] the normal mapping over the same base mesh, obtained using PRN [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF], and we refer to these as Ours+PRN and SfSNet+PRN respectively. We show our meshes from two views to illustrate that the output is not optimized for a particular viewpoint, a known limitation with SfS. Compared to SfSNet we recover much more refined details that significantly enhance the base mesh. Compared to Extreme [Tran et al., 2018] our approach does not include unnecessary additional noise. As observed by other authors, Pix2Vertex [Sela et al., 2017] cannot handle difficult poses or illuminations, and sometimes simply fails to converge. Both PRN and 3DDFA [START_REF] Zhu | Face alignment in full pose range: A 3d total solution[END_REF] can correctly recover the general structure of the face, although their goal was not to recover surface details as we do.

We believe our improved results are due to the fact that we do not rely on a parametric model for training data generation, as was done in e.g. [START_REF] Sengupta | Sfsnet: Learning shape, reflectance and illuminance of facesin the wild[END_REF], as well as the strongly regularized latent space that is learned through the two encoder/decoder networks, in addition to the skip connections that can transfer the necessary details.

Ablation

We evaluate here the influence of the proposed architectural components. In particular, we compare against the alternatives shown in Figure 6.6: our model without skip connections (Figure 6.6b), without the normal encoder E N (Figure 6.6c), and without both the normal encoder E N and image decoder D I (Figure 6.6d), i.e.a basic encoder-decoder architecture. Since there is no need in the last two cases for deactivable skip connections we use standard ones. We show quantitative results in Table 6.7, and qualitative examples in Figure 6.8. Our final model outperforms the alternatives both quantitatively and quali-tatively which validates the proposed cross-modal architecture design, and the benefit of the introduced deactivable skip connections.

For example, we can see in the geometric shape of the eyelids in the first row of Figure 6.8 and the shading in the second row that our final model gets the best from each of the alternatives. Our correct global shape estimate is comparable to that of the cross-modal model without skip connections, although the latter is smoother and clearly lacks details. Additionally we can see that removing the image decoder D I and normal encoder E N (i.e.a standard encoder-decoder with skip connections) gives poor results for images in-the-wild, due to the domain gap between training and evaluation. This can be visualized particularly in the artifacts appearing on the third and fourth examples, or the inaccurate shadings of the second example. Finally, our fine details are comparable to those of the model with skip connections but without the normal encoder E N , which in turn has a reduced ability to represent the shape accurately, since it has not learned an additional prior on the geometric aspects of the face.

Low-cost depth enhancement

We can use our model to enhance the appearance of the noisy depth data coming from low-cost RGB-D sensors, e.g. Kinect. We show an example of this using the FaceWarehouse dataset [START_REF] Cao | 3d shape regression for real-time facial animation[END_REF], where we use the accompanying RGB image to predict normals with our method, and append these normals to the raw depth image pixel-wise using normal mapping, thus rendering enhanced geometric shading. In Figure 6.9 we show the RGB images in the first row, the raw depth in the second, and the same depth enhanced with our model's predictions in the last one. The ability to recover accurate normals allows to enhance the depth appearance significantly.

Limitations

The proposed method still has limitations, some of which are shown in Figure 6.10. These belong to extreme situations that represent outliers to the training data, including faces in very severe lighting/shades (Figure 6.10a,6.10b), occlusion (Figures 6.10c,6.10d), very low quality images (Figure 6.10e) and unusual facial textures (Figure 6.10f).

Conclusion

This chapter presented a novel deep learning based approach for the estimation of facial normals in-the-wild. The proposed method is centered on a new architecture that combines the robustness of cross-modal learning and the detail transfer ability of skip connections, enabled thanks to the new deactivable skip connections. By leveraging both paired and unpaired data of image and normal modalities during training, we learn a strong prior knowledge on the distribution of both natural images and facial shape in the form of surface Compared to classic SfS approaches, we achieve accurate estimations even under hard conditions imposed by natural images; and we can further do it efficiently thanks to the use of a neural network. Compared to methods based on parametric approaches, we are able to recover much finer details as a result of the skip connections and our novel cross-modal architecture that can leverage all available data. Through ablation studies and comparisons to other approaches, this chapter also confirms that (1) normal estimation is a task well suited for convolutional networks, and (2) training exclusively on real images and high-quality scans is highly benefitial, which was again allowed by the use of cross-modal learning and the deactivable skip connections.

Among the limitations of our work are the inability to properly handle occlusions (as it is mostly a local method) and to recover finer-details, e.g.porelevel details, which are directions that will be tackled in future work. Moreover, unlike parametric models our results are not temporally coherent, and thus cannot be used to study motion-related aspects without a registration step. We believe however that this framework can be leveraged to enhance generative models such as the ones presented in previous chapters. An interesting future direction would be to harness this in order to learn fine-scale details that correlate with both identity and expression. Considering that the proposed architecture is rather generic, a final future direction will be to investigate its use on other tasks that exhibit similar data conditions. This thesis presented novel methods for learning data-driven models of the 3D facial shape from large-scale datasets. There are many aspects of the face geometry that are interesting and challenging to explore. We focused here on the following aspects: (1) how to build decoupled models that can capture the interaction between the shape and motion components; (2) how to place large datasets of 3D faces in motion into a single parameterization; and (3) how to obtain finer details by focusing on the problem of surface normal recovery from natural images. In each of these methods we investigated whether better performances could be obtained by working on large-scale datasets. We circumvented the difficulty of acquiring a large number of 3D scans by profiting from the numerous publicly available sources, and proposed techniques designed to handle the challenges that come with such data while harnessing the underlying information.

To conclude this work we summarize next the main contributions of this thesis, as well as directions for future work.

Summary of Contributions

The main contributions of this thesis were the following: Chapter 3 proposed the multilinear autoencoder, a novel method for building multilinear models that does not require complete data tensors for training. The approach was based on a new architecture that enabled the use of the deep learning optimization machinery to refine an initial tensor model. This allowed to better encode all available training data, thus demonstrating that expressive multilinear models can be learned from large-scale sources. Additionally, we proposed a loss function on the latent space which allowed to retain the decoupling capabilities of multilinear models, and even improve them when compared to competing methods. Chapter 4 contributed a spatiotemporal registration approach for 3D faces in motion, designed to automatically process datasets coming from multiple acquisition systems. Through the combination of a spatiotemporal model that globally handles entire motion sequences, and a regression-based approach that can efficiently and robustly initialize the registration process, we demonstrated accurate performances while remaining both efficient and scalable. The method was tested on three publicly available datasets showing different types of CHAPTER 7. CONCLUSIONS motion including spontaneous ones, and combined allowed us to register more than 300, 000 facial frames.

Chapter 5 explored the use of adversarial learning for decoupled models, contributing an alternative technique to linear and multilinear models that achieved significantly better decoupling results. We enabled the use of adversarial learning for 3D faces by proposing the geometry mapping layer, which acts as bridge between the 3D generator and 2D discriminators. This strategy allowed to leverage recent progress in generative adversarial learning while still generating three-dimensional faces in their natural domain. Our purely discriminative loss functions demonstrated significantly better decoupling capabilities, and were able to capture subtle expression differences in the latent space.

Since there is no standard evaluation protocol, both Chapters 3 and 5 further proposed metrics to assess the degree of decoupling of generative models based on external classifiers. Chapter 5 also proposed a diversity metric, which was not considered before in the evaluation of 3D face models.

Finally, Chapter 6 introduced a new approach for the problem of estimating facial normals from images in-the-wild. Improved performances were obtained through the use of a novel cross-modal learning technique that enabled training exclusively from high quality data, whether paired or not. This was achieved by a novel module that we called deactivable skip connections, which allowed to integrate both the auto-encoded and image-to-normal branches within a same architecture, while still transferring the local details from the input image to the output surface. We showed how this strategy can learn a rich latent space of both natural images and surface normals that enabled accurate reconstructions, as well as state-of-the-art results and visually compelling enhancements in challenging cases.

Each of these methods have a few drawbacks that were discussed by the end of each chapter. We summarize here what we consider to be the main points:

• For methods that learn a global latent space such as those in Chapters 3 and 5, it remains a question what is the optimal dimension of each space, and how to properly choose these dimensions. This was also a problem in the spatiotemporal model used in Chapter 4, where a low number of temporal coefficients can lead to overly-smoothed motions. The problem is shared by many of the recent deep-learning based techniques that operate as "black-box" machines. A principled way of choosing the optimal dimensionality such that it balances model expressiveness against compactness is certainly desirable, particularly for telecommunication systems.

• The registration method in Chapter 4 globally considers the entire facial surface and motion, but a more local treatment of certain attributes

FUTURE WORK

105

of the face might be worth exploring. This is valid both in terms of space and time. In terms of time, some motions occur faster than others, and a unique temporal space for the entire sequence might not be able to capture higher frequencies. A more local approach, e.g.a sliding window with variable dimensionality, could be necessary for accurate results in different types of motion. In terms of space, areas such as the mouth and eyes are key for transmitting emotions, and a faithful recovery of their shape is essential for applications such as animation and re-targeting. Specific optimization terms for these areas, as it was considered in e.g. [START_REF] Bermano | Detailed spatio-temporal reconstruction of eyelids[END_REF], Garrido et al. [2016a], could further improve the results of Chapter 4.

• The approach for surface normal estimation in Chapter 6 does not yield temporally coherent results, and thus it cannot be used to study how finer details evolve over time. Furthermore, because of the local formulation the results are not robust to occlussions. To address this, a combination of global models (like those considered in previous chapters) and local approaches such as the one presented here should be considered.

More general directions for future work are discussed in the next section.

Future work

The models presented here were learned from publicly available sources that allowed to consider datasets of larger scale than most related work. Yet, these are still expensive and time consuming to capture, and can only be acquired under controlled setups which limit the range of motions that can be studied. Learning high-quality models from cheaper acquisition devices such as RGB or RGB-D sensors would not only give access to a larger corpus of training data, but will also allow to model human behaviour that cannot be recorded in controlled scenarios. In the case of faces, this means modeling truly spontaneous expressions and micro-expressions, as well as their relationship with the environment. First works that learn 3D models purely from RGB images have begun to appear, e.g. [START_REF] Tran | Nonlinear 3d face morphable model[END_REF], [START_REF] Tran | Towards high-fidelity nonlinear 3d face morphable model[END_REF], but are still limited in the amount of variations they can capture. Extending the directions considered in this thesis such that equivalent or improved results can be obtained using less constrained data is an exciting avenue for future work.

For decoupled models there are two interesting extensions that were not addressed here. First is the unsupervised discovery of the latent factors: instead of training with labeled data as considered in this thesis, it would be benefitial to explore the case were the labels corresponding to each of the factors are unknown. This would allow for example to build the identity-expressionviseme model of Figure 5.6 without the need to provide any manual annotation, CHAPTER 7. CONCLUSIONS which is very time consuming and prone to errors. While numerous work have explored this with 2D applications in mind (e.g. [START_REF] Chen | InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets[END_REF]), the direction is mostly unexplored in the case of 3D data.

Another extension is the fine-grained encoding of semantics in the latent space. The decoupling considered here is concerned with the independent manipulation of identity and expression, but there is no mechanism to control for example the size of the nose or how closed the eyes are. Having access to semantically-based parameters that can modify for example the intensity of an action unit (as it is commonly done in the film industry through the blendshape parameterization) would allow for widespread adoption of more complex models like the one in Chapter 5. Furthermore, such models could be use for inferring semantic information from e.g.2D images, which can in turn be useful for fine-grained recognition and manipulation.

The models of Chapters 3 and 5 are global models that encode the entire face, while the approach of Chapter 6 considers mostly local information. There is a trade-off for each: using global models gives robustness to different tasks by providing a strong knowledge of what a face should look like and how the different spaces interact with each other, but it is not capable of capturing details. On the other hand, the local approach of Chapter 6 allows to recover finer details 1 , but the results are neither robust nor temporally coherent. A future direction in terms of modeling will be to explore a combination of these two, in order to benefit from the best of each. Some of the technical contributions of this thesis can be applied to other problems and it would be interesting to explore their capacity for this. This is the case of the geometry mapping layer of Chapter 5 and the deactivable skip connections of Chapter 6. For example, the geometry mapping layer can be useful for general tasks on 3D shapes such as correspondence and classification, while the deactivable skip connections can be leveraged on other applications that both involve multiple modalities and can profit from the use of skip connections.

Figure 2

 2 Figure 2.1 -Example of facial mesh. From left to right: raw scan (from Savran et al. [2008]), template mesh (adapted from Alexander et al. [2010]), registered mesh.

  and Wuhrer [2015b], Zhang et al. [2016]. Finally, a line of work focuses on the case where the face is undergoing a certain motion, a problem usually referred to as tracking. Examples of this include Beeler et al. [2011], Bolkart and Wuhrer [2015a], Li et al. [2017].

  al., 2017a, Zhu et al., 2017, Tewari et al., 2017, Genova et al., 2018, Bas and Smith, 2019], and newer versions have been recently proposed [Gerig et al., 2018]. Other publicly available alternatives include the multi-resolution model of Huber et al. [2016], the models of Brunton et al. [2014b] and the full-head models of Dai et al. [2017] and Li et al. [2017].

  Deep neural networks have experimentally been shown to summarize large groups of data and automatically extract only the relevant features for a large variety of problems, providing an efficient structure for the optimization of large datasets. This motivates the use of deep learning as a scalable and robust alternative for training a multilinear model. Initial works that used CNN frameworks to recover the 3D shape from a single photograph include Zhu et al. [2016], Richardson et al. [2016, 2017], Tran et al. [2017a], Güler et al. [2017], Sela et al. [2017]. This was predominantly achieved by supervised regression towards the coefficients of a 3DMM, restricting the accuracy of the solution due to the use of synthetic data. The work of Tewari et al. [2017] was the first to frame the generative model as decoder of a neural network. This allowed for end-to-end self-supervised training using an analysis-by-synthesis loss function. Subsequent methods focused on improving the loss formulation, such as Genova et al. [2018], Sengupta et al. [2018], Sanyal et al. [2019]. Unlike the present work, none of these model-based decoders were trained or improved during the CNN optimization. By the time of publication, only the work of Laine et al. [2017] observed that optimizing an initial PCA model within a deep learning task can yield better results than a fixed model. The work on this chapter follows a similar path, yet unlike Laine et al.

  [2018] learned a corrective space for a fixed identity model, Tewari et al. [2019] learned the full identity space, and Tran et al. [2018, 2019] learned both identity and expression models directly from images.

Figure 3 .

 3 Figure 3.1 -Multilinear Autoencoder architecture. The encoder takes as input a 3D mesh, which is rendered into a heightmap, processed by a deep CNN, and transformed into a latent representation by the fully-connected layers. The decoder splits the latent representation according to the specified factors and performs a multilinear transformation in order to get the output mesh. Both encoder and decoder are optimized during training.

  3.3). The framework was implemented in PyTorch v1.0.1 [Paszke et al., 2019], and the experiments were run using a NVidia GeForce GTX 1080 GPU. For the facial mesh template we used a cropped version of the publicly available Digital Emily [Alexander et al., 2010], which consisted of n = 10057 vertices (see Figure 2.1).

  Comparison to standard tensor decompositionWe compare here to standard tensor decomposition methods, namely higher-order SVD (HOSVD) [De Lathauwer et al., 2000a] and higher-order orthogonal iteration (HOOI) [De Lathauwer et al., 2000b]. Both methods require complete data tensors to perform a Tucker decomposition. For a fair comparison MAE is both pre-trained and trained using BU-3DFE alone, with distinct labels assigned to the different expression intensities. We refer to this version as "MAE -bu3".Results are shown in the top three rows of Table3.4. Note that HOSVD is the method we use for initialization of the decoder, and thus we show here (a) λ = 0.1 (b) λ = 1

Figure 3 .

 3 Figure 3.2 -Influence of the latent loss on expression transfer. From left to right: input mesh, transferred expressons: angry, disgust, fear, happy, sad, surprise. Lower values of λ can sometimes fail to properly decouple the latent space, and hence transferring expressions does not preserve semantics.

  Figure 3.3 -Qualitative comparison between state-of-the-art and MAE. From left to right: original scan, transferred expressions: angry, disgust, fear, happy, sad, surprise.

  Computing the core tensor using HOSVD for 89 -25 dimensions requires on average 3 seconds. Pre-training the encoder takes about 40ms per mini-batch and ∼ 3 minutes per epoch for our data (including dataloading time). Fine-tuning the Bu3+Bosph model takes about 13 seconds on average per epoch, while fine-tuning Bu3+Bosph+D3D takes around 2 minutes per epoch. Generating each depth image takes ∼ 20ms for the registered data. Once the training is finished, regressing from a single raw scan to 3D vertices requires around 250ms for a batch of size 64.

Figure 3 . 4 -

 34 Figure 3.4 -Registration of raw scans using the encoder. Top: input scans. Bottom: registered results.

Figure 3 .Figure 3

 33 Figure 3.5 -Expression synthesis on raw data. Top: ground-truth scans, bottom: our results.

  For real-time expression transfer, Weise et al. [2009] introduced a system based on non-rigid Iterative Closest Point (ICP), from which a person-specific blendshape model was built and used to sequentially track sequences of the same actor in real-time. Follow-up work [Weise et al., 2011] improved on this by using color cues and a probabilistic animation prior which can handle noisier input from an RGB-D camera. The methods of Li et al. [2013], Bouaziz et al. [2013] further removed the need for calibration by updating an initial blendshape model on-the-fly. Other real-time tracking approaches from RGB-D video include Zollhöfer et al. [2014] that deform a template using an asrigid-as-possible prior, and Thies et al. [2015] that track blendshape weights through an analysis-by-synthesis framework. Further improvements on this line of work included robustness to occlusions and pose [Hsieh et al., 2015], detailed blendshape models through the use of displacement maps [Thomas and Taniguchi, 2016], eye-gaze control [Thies et al., 2018a,b], and full head and upper body tracking [Thies et al., 2018a].

  [2008], Schneider and Eisert [2009], Cheng et al. [2017b] in the static case, or Bolkart and Wuhrer [2015a], Li et al.

Figure 4 .

 4 Figure 4.3 shows an example of the results obtained on each of the steps of the method: regression, spatiotemporal registration, and final refinement.Figure 4.2 shows a few more examples of registrations obtained on D3DFACS, BU-4DFE and BP4D-Spontaneous. They illustrate that accurate cross-dataset registrations can be obtained, while still being robust to different types of noise in the data.

Figure 4 .

 4 Figure 4.3 shows an example of the results obtained on each of the steps of the method: regression, spatiotemporal registration, and final refinement.Figure 4.2 shows a few more examples of registrations obtained on D3DFACS, BU-4DFE and BP4D-Spontaneous. They illustrate that accurate cross-dataset registrations can be obtained, while still being robust to different types of noise in the data.

Figure 4 .

 4 Figure 4.2 -Registration examples on (from top to bottom): D3DFACS, BU-4DFE and BP4D datasets.

Figure 4 . 3 -Figure 4 . 4 -

 4344 Figure 4.3 -Results for the successive steps (left to right): raw scan, regression, 1 iteration, 5 iterations, final.

Figure 4 . 5 -

 45 Figure 4.5 -Median per-vertex error for each frame of a long sequence in BP4D.

Figure 4 . 6 -

 46 Figure 4.6 -Cumulative plot of median per-vertex error over D3DFACS (46028 frames) for: regression results, spatiotemporal registration (1, 3, 5 and 10 iterations) and final refinement.

Figure 4 . 7 -

 47 Figure 4.7 -Comparison between our method and a static version, in terms of vertex error and sequence compactness.

Figure 4 .

 4 8a shows cumulative plots of the median pervertex error on all the registered sequences in BU-4DFE, comparing Bolkart and Wuhrer [2015a] to our registration without and with the final refinement step, since Bolkart and Wuhrer [2015a] has no refinement step.

Figure 4 .

 4 9a further shows a qualitative comparison over a challenging example. Results reveal similar accuracy for both methods without the refinement step, whereas Bolkart and Wuhrer [2015a] requires around 30 seconds per-frame to process.

  (a) Bolkart and Wuhrer [2015a] on 49757 frames (b) Li et al. [2017] on 46028 frames (c) Cosker et al. [2011] on 292 frames

Figure 4 .

 4 Figure 4.8 -Comparisons to Bolkart and Wuhrer [2015a] on BU-4DFE, Li et al. [2017] on D3DFACS, and Cosker et al. [2011] on a subset of D3DFACS. Cumulative plots for median per-vertex error.

  (a) Bolkart and Wuhrer [2015a] (b) Li et al. [2017] (c) Cosker et al. [2011]

Figure 4 .Figure 4 . 10 -

 4410 Figure 4.9 -Qualitative comparisons. From left to right: original scan, compared method, our result.

  Steenkiste et al., 2019]. When full label supervision is available, bilinear [Tenenbaum and Freeman, 2000] and multi-linear [Vasilescu and Terzopoulos, 2002, Vlasic et al., 2005] models were initially proposed to disentangle known factors of variation. More recently, Reed et al. [2014] extended a Restricted Boltzmann Machine by clamping parts of the hidden units assigned to a specific factor, and Dosovitskiy et al. [2015], Kulkarni et al. [2015] trained deep neural networks to generate 2D projections of 3D objects from high-level descriptions. Weaker forms of labeling were considered in Reed et al. [2015], Mathieu et al. [2016], Jha et al. [2018], and fully unsupervised approaches were proposed by Chen et al. [2016], Higgins et al. [2017], Kim and Mnih [2018]. Because of the lack of supervision these methods cannot control which factors are encoded.

  Figure5.2 -While a GAN could be used to generate geometry images, recovering the mesh from them is prone to artifacts, e.g. erroneous boundary interpolations (red) and precision loss (blue) in 5.2b. In this work we generate instead the 3D mesh, while geometry images are used only for discrimination.

  (a) W/o mesh reg. (b) W/o expr. classif. (c) W/o feature loss (d) Proposed

Figure 5 . 4 -

 54 Figure 5.4 -Qualitative results for alternative approaches. From left to right: randomly generated samples (dark gray), random samples with a same expression code (light gray), random samples with a same identity code (purple).

  The model proposed in Ranjan et al. [2018], called CoMA in the following, does not explicitly favor decoupling and thus we use the DeepFLAME Source Target CoMA MAE Ours (a) Transferring expression to a target face (b) Sampling novel identities from the transferred expression.

Figure 5 . 5 -

 55 Figure 5.5 -Qualitative comparison in terms of expression transfer. Top: expression code z expr transferred to a target identity. Bottom: using z expr from the source in the top row, we sample novel identities (left to right: CoMA, MAE, ours).

Figure 5

 5 Figure 5.6 -Example of decoupling between identity, expression and viseme.

  we generate new expressions for the target mesh by either 1. Replacing the expression with that of the source: G(z target id , z src expr , z target noise ) 2. Adding the expression vectors: G(z target id , z src expr + z target expr , z target noise ) Results can be seen in Figure 5.7. In particular, note how adding the latent vectors results in plausible expressions which preserve the semantics of both sources.

Figure 5

 5 Figure 5.8 -From top to bottom: interpolation (purple) and extrapolation (gray) of expression code, identity code, and the full latent.

  Figure 5.9 -Reconstruction of sparse data

Figure 6

 6 Figure 6.2 -Overview of the proposed approach. Our cross-modal architecture allows exploitation of paired and unpaired image/normal data for imageto-normal translation (red), by means of further image-to-image (green) and normal-to-normal (blue) regularizations during training. The deactivable skip connections allow to transfer details from the image encoder E I to the normal decoder D N without having to link the normal encoder E N to the normal decoder D N .

  Figure 6.3 -Instead of concatenating the encoder features (red) and decoder features (blue), as with standard skip connections, we fuse the encoder features with part of the decoder features (light blue), to be able to deactivate this operation when needed.

Figure 6 . 4 -

 64 Figure 6.4 -Qualitative comparisons on normals in the 300-W dataset

Figure 6

 6 Figure 6.5 -Qualitative comparisons on geometries in the 300-W dataset. (a) Input, (b) Ours+PRN, (c) SfSNet+PRN, (d) PRN, (e) Extreme, (f) Pix2Vertex, (g) 3DDFA

  .1 respectively. The same is done with SfSNet. The error values on Photoface for the methods of Sengupta et al. [2018], Trigeorgis et al. [2017], Sela et al. [2017], Bansal et al. [2016] and Kokkinos [2017] are as reported in Sengupta et al. [2018], and we use the publicly available implementations of Tran et al. [2018], Zhu et al. [2017] and Feng et al. [2018] for the others. For the Florence dataset we use the publicly available implementations. Note that, to be able to evaluate the per-pixel normal accuracy, we can only compare to 3D reconstruction methods whose output is aligned with the image. For a fair comparison, all methods were given facial images of size 256 × 256 as input, resized if necessary.

  Figure 6.6 -Architectures for the ablation test: (a) our proposed architecture, (b) without skip connections, (c) without the normal encoder and (d) without the normal encoder and the image decoder.

  w/o E N D I (d) w/o E N (e) w/o skip co.

Figure 6 . 8 -

 68 Figure 6.8 -Qualitative comparisons between architectures: (b) our proposed architecture, (c) without the normal encoder and the image decoder, (d) without the normal encoder, and (e) without skip connections.

Figure 6

 6 Figure 6.9 -Raw Kinect depth enhancement using our normals on the Facewarehouse dataset [Cao et al., 2013].

Figure 6

 6 Figure 6.10 -Failure cases

  WUHRER, S., BOYER, E. Multilinear Autoencoder for 3D Face Model Learning. IEEE Winter Conference on Applications of Computer Vision (WACV). (2018)

	Chapter 4:

• FERNANDEZ ABREVAYA, V., WUHRER, S., BOYER, E. Spatiotemporal Modeling for Efficient Registration of Dynamic 3D Faces. IEEE International Conference on 3D Vision (3DV). (2018) Chapter 5: • FERNANDEZ ABREVAYA, V., BOUKHAYMA, A., WUHRER, S., BOYER, E. A Decoupled 3D Facial Shape Model by Adversarial Training. IEEE International Conference on Computer Vision (ICCV). (2019) Chapter 6: • FERNANDEZ ABREVAYA, V., BOUKHAYMA, A., TORR, P.H.S., BOYER, E. Cross-modal Deep Face Normals with Deactivable Skip Connections.

  Ma et al. [2007], Zivanov et al. [2009], Ghosh et al. [2011], Seck et al. [2016].

  Example works include Passalis et al. [2011], Creusot et al. [2013], Bolkart and Wuhrer [2015a],

  , Aldrian and Smith, 2013, Lewis et al., 2014a, Patel and Smith, 2016, Thies et al., 2016].

  Decarlo and Metaxas [2000], Tena et al. [2011], Neumann et al. [2013], Wu et al. [2016], Cheng et al. [2017b]. While these can generalize better, they usually cannot capture the co-articulation effects that occur between different parts of the face, e.g.the simultaneous movement of the eyes and mouth during a smile. A few non-linear models were recently proposed, such as Li et al. [2017] which includes jaw articulation and corrective blendshapes, or Ranjan et al. [2018], Lombardi et al. [2018], Tran et al. [2019] that used deep neural networks to model the expression space.

	al., 2016, Cong et al., 2016, Kozlov et al., 2017,
	Ichim et al., 2017, Barrielle and Stoiber, 2019].
	In the computer vision community several works proposed extensions to
	3DMM that handle expressive faces. Blanz et al. [2003] encoded expressions as
	displacements from the corresponding neutral face and built a PCA model from
	35 scanned expressions of a single subject. This idea was extended to several
	subjects in the work of Amberg et al. [2008], while Yang et al. [2011] built
	instead one PCA model per-expression. Local expression models have also
	been considered, e.g.

  2.6)where w id ∈ R d id , w exp ∈ R d exp are the latent vectors for identity and expression respectively, f id : R d id -→ R 3n is a function that decodes the vertices of the CHAPTER 2. BACKGROUND neutral face, and f exp : R d expr -→ R 3n decodes displacement vectors from a neutral expression. In the case of a linear model, typically f id (w id ) = x + B id w id with B id a PCA basis for the identity space, and f exp (w exp ) = B exp w exp with

	B exp a PCA or a blendshape basis for expression. This is the approach followed
	for example by Amberg et al. [2008], Garrido et al. [2016a], Thies et al. [2016],
	Tewari et al. [2017], Liu et al. [2018], Kim et al. [2018a], Tewari et al. [2019].
	A handful of works have also modeled either the identity or the expression
	functions as nonlinear, such as Ranjan et al. [2018] where the PCA identity
	model of Flame [Li et al., 2017] is combined with a deep mesh autoencoder for
	expressions, or Tran et al. [2018, 2019], Li et al. [2020] that train a deep neural
	network for each space.

  ], as displacement maps [Golovinskiy et al., 2006, Thomas and Taniguchi, 2016, Tran et al., 2019], or as normal maps [Ichim et al., 2015, Lattas et al., 2020]. At the coarsest level, corrective layers can be added to a parametric model to more closely match the input data and capture person-specific idiosyncrasies. To keep the model tractable the correctives can too be encoded in a low-dimensional space, such as the linear spectral basis of Bouaziz et al. [2013], Garrido et al. [2016a] or the automatically learned corrective function of Tewari et al. [2018].

Table 3 .

 3 We first measure how different values of λ affect the output model, both for Bu3+Bosph and Bu3+Bosph+D3D . Results are shown in Tables 3.1 and 3.2. As expected, greater values of λ result in progressively 1 -Influence of the latent loss on the Bu3+Bosph model. Median generalization error (mm), specificity error (mm) and percentage of correct classifications after expression transfer. Best values in bold.

			Bu3+Bosph
	λ	Generalization Specificity Expression
	0.1	1.06	3.33	32.54
	1	1.04	3.47	58.41
	10	1.09	3.41	57.30
			Bu3+Bosph+D3D
	λ	Generalization Specificity Expression
	0.1	1.00	3.38	47.46
	1	1.00	3.61	50.16
	10	0.99	3.32	45.87

Table 3 .

 3

2 -Influence of the latent loss on the Bu3+Bosph+D3D model. Median generalization error (mm), specificity error (mm) and percentage of correct classifications after expression transfer. Best values in bold.

Table 3 .

 3 3 -Influence of latent size for identity and expression spaces: median generalization error (mm), specificity error (mm) and percentage of correct classifications after expression transfer.

related to how well the model generalizes, which might explain why the value sometimes fluctuates.

Table 3 .

 3 4 -Comparison against standard tensor decomposition methods, and influence of training data size. The top three rows show comparisons against classic tensor decomposition methods, using the same training dataset. The bottom rows show improvements obtained when training with larger datasets that cannot be assembled as a tensor. In terms of median generalization error (mm), specificity error (mm), and percentage of correct classifications for expression transfer.

Comparison to state-of-the-art Finally

  , we compare to two closely related works that learn multilinear models of 3D faces from incomplete data tensors: RMM[START_REF] Bolkart | A robust multilinear model learning framework for 3d faces[END_REF] and[START_REF] Wang | Learning the multilinear structure of visual data[END_REF]. We run RMM on our own registration using the publicly available code with default parameters. We use the same subset from BU-3DFE and Bosphorus as in the published model since this was already proven to work correctly for RMM, except we remove the testing identities that were used for the previous experiments (see Section 3.4.2). In particular, we use 184 identites from BU-3DFE and Bosphorus, and 7 expressions from BU-3DFE (with the highest intensities) plus 23 expressions from Bosphorus. The latent dimensions are set to 23 and 6 for identity and expression respectively, as in the published RMM model. We build a model using this setting for the method of Wang et al.[2017] with code provided by the authors, and train MAE on this data and with same

	CHAPTER 3. A MULTILINEAR AUTOENCODER FOR 3D FACE MODEL
		LEARNING FROM LARGE DATASETS
	Method	Generalization Specificity Expression
	Bolkart and Wuhrer [2016]	1.71	3.44	56.51
	Wang et al. [2017]	1.53	3.57	13.65
	MAE	2.17	3.42	61.59

Table 3 .

 3 7 -Identity recognition from 3D mesh sequences (ED experiment): comparison against Sun et al. [2010], Alashkar et al. [2016] and approaches using static 3D data reported in Sun et al. [2010]. Alashkar et al. [2016] 85.20 87.70 83.49 83.36 84.86 80.49 84.13 MAE 87.00 90.33 86.67 86.67 87.33 86.00 87.33

	Method					Recognition Rate (%)
	LLE on static 3D (reported in [Sun et al., 2010])		82.34
	PCA on static 3D (reported in [Sun et al., 2010])		80.78
	LDA on static 3D (reported in [Sun et al., 2010])		91.37
	Sun et al. [2010]						97.47
	Alashkar et al. [2016]				100.00
	MAE					100.00
	Method	AN	DI	FE	HA	SA	SU	Avg.
	Sun et al. [2010]	94.12 94.09 94.45 94.52 93.87 95.02 94.37

Table 3 .

 3

8 -Identity recognition from 3D mesh sequences (EI experiment): Recognition rate (percentage) for each of the training expressions: angry (AN), disgust (DI), fear (FE) happy (HA), sad (SA), surprise (SU), and comparisons to Sun et al. [2010], Alashkar et al. [2016].

  [Alashkar et al., 2016, Fang et al., 2012, Sandbach et al., 2011], pain detection [Zhang et al., 2015] and realistic expression synthesis

  et al. [2011], Cheng et al. [2017a], Fyffe et al. [2017], and is robust to occlusions by construction; it runs an order of magnitude faster than recently proposed methods based on parametric face models [Bolkart and Wuhrer, 2015a, Li et al., 2017] while achieving comparable accuracy; and provides compact representations of the results. The method is evaluated qualitatively and quantitatively on three publicly available datasets, namely D3DFACS [Cosker et al., 2011], BU-4DFE [Yin et al., 2008] and BP4D-Spontaneous [Zhang et al., 2014], demonstrating it can efficiently obtain accurate registrations as well as compact representations. Comparisons to Bolkart and Wuhrer [2015a], Li et al. [2017] and Cosker et al.

  and Wuhrer [2015a], Li et al. [2017] and Cosker et al. [2011] using registrations provided by

	It. 1	It. 3	It. 5 It. 10	Li et al. [2017]
	Mean error 2.92 2.77 2.69	2.65	3.13

Table 4 .

 4

1 -Mean landmark error in mm., for 1, 3, 5 and 10 iterations, and comparison with Li et al. [2017], over 10 selected sequences of the D3DFACS dataset.

  Cosker et al.Finally, we compare to the work of[START_REF] Cosker | A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling[END_REF]. Comparisons are done over 3 sequences of D3DFACS that were provided by the authors. We obtain a mean error of 0.13mm for our method and 0.18 mm for Cosker et al.[2011]. Figure4.8c shows the results in terms of median pervertex error. They demonstrate similar accuracy although our method is more general as it does not require a controlled capture setup. Figure4.9c shows a qualitative comparison.

  Tran et al. [2017b], Pumarola et al. [2018], Shen et al. [2018], Zhao et al. [2018],

  Combined with ADVERSARIAL LEARNINGReconstruction of Sparse DataWe test here model generalization when reconstructing partial face data given sparse constraints. To this purpose, we use the dataset provided by Ranjan et al.[2018], which contains 12 subjects performing 12 extreme expressions. We take the middle frame of each sequence and manually label 85 landmarks (see Figure5.9b), resulting in a testing set of 144 subjects. The face model is fitted by minimizing:

	Method		λ = 0 λ = 0.01 λ = 10
	3DMM [Amberg et al., 2008]	6.62	4.64	2.46
	MAE [Fernández Abrevaya et al., 2018] 4.46	4.06	2.78
	CoMA [Ranjan et al., 2018]	3.05	3.02	2.83
	Ours		2.62	2.55	2.42
	Table 5.2 -Reconstruction of sparse data under different regularization weights
	(RMSE, in mm).
	a specificity value that is among the best, this implies that it has learned to
	generate significant variations that remain valid facial shapes.
		p
	arg min
	z	i=1

  , Huber et al., 2016, Egger et al., 2018, Booth et al., 2018, Gecer et al., 2019] or, more recently, using deep learning to regress model parameters [Richardson et al., 2016, 2017, Tewari et al., 2017, Tran et al., 2017a, Genova et al., 2018, Feng et al., 2018, Kim et al., 2018a, Tewari et al., 2019, Sanyal et al., 2019], or alternatively to regress other face information using 3DMM training data, for instance volumetric information [Jackson et al., 2017], UV position map [Feng et al., 2018], normal map [Trigeorgis et al., 2017], depth map [Sela et al., 2017],

Normal Estimation with Deep Networks Closely

  . The recent works of[START_REF] Shu | Neural face editing with intrinsic image disentangling[END_REF] and[START_REF] Sengupta | Sfsnet: Learning shape, reflectance and illuminance of facesin the wild[END_REF] use deep neural networks to decompose in-the-wild facial images into surface normals, albedo and shading, assuming Lambertian reflectance and using a semi-supervised learning approach inspired by SfS. Our work follows a similar direction and estimates the normal information from a single image, but unlikeShu et al. [2017] and Sengupta et al. [2018] we do not rely on an image formation model and let instead the network learn such transformation from real data. related to our work are methods that recover surface normals from an image using deep neural networks, e.g. [Wang et al., 2015, Eigen and Fergus, 2015, Yoon et al., 2016, Kokkinos, 2017, Bednarik et al., 2018, Qi et al., 2018, Zhang and Funkhouser, 2018, Qiu et al., 2019, Du et al., 2019, Smith et al., 2019, Alldieck et al., 2019]. Yoon et al. [2016] and Bansal et al. [2016] focus on the normal prediction task in order to recover detailed surfaces.

For anatomical shapes, scale is usually preserved.

The Facial Action Coding System is a description of the visible facial movements, classifying each into Action Units (AUs) which describe a contraction or relaxation of a specific muscle.

An introduction to multilinear models is provided in Section

2.3.3.

Neutral, Angry, Disgust, Fear, Happy, Sad, Surprise.

We used 89 out of 100 identities: ten subjects were left out for testing, and one subject was left out of the initial tensor as one of the expressions did not register succesfully.

We render normal maps as we found this to work better for recognition. The classifier is trained using BU-3DFE and Bosphorus which provide the necessary labels.

An introduction is provided in Section

2.2.

We use the bu-bosph version which has ground-truth nose tip locations. The input is the same heightmap used for regressing multilinear coefficients. Note that nose tip detection on a heightmap image is a relatively easy task.

We accelerate the nearest point search using the libigl implementation of the axis-aligned bounding box (AABB) data structure[START_REF] Jacobson | libigl: A simple C++ geometry processing library[END_REF]. We stop when the difference between iterations is small, or a maximum of 30 iterations is reached.

We use in fact the version that was originally published in Fernández Abrevaya et al.[2018].

The visual counterpart of a phoneme (a unit of sound).

CHAPTER 6. ESTIMATING 3D FACE NORMALS FROM NATURAL IMAGES

For evaluation purposes we use the remaining testing subset of Photoface, which consists of 100 subjects not seen during training and 1489 image/normal pairs. This subset challenges the reconstruction with very severe lighting conditions. Following the work of [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF], we create an additional evaluation set by rendering 530 color and normal facial images from the 53 3D models of the Florence dataset [START_REF] Bagdanov | The florence 2d/3d hybrid face dataset[END_REF], rotated with random axes and angles in [-π/4, π/4]. This allows to evaluate on a completely unseen dataset. Finally, we use the 300-W dataset [START_REF] Sagonas | 300 faces in-the-wild challenge: The first facial landmark localization challenge[END_REF] of 2D face images to assess qualitative performances in-the-wild. Note that for both training and testing, we limited ourselves to 3D face datasets of high quality and details.

Bibliography

The bjut-3d large-scale chinese face database.

A Appendix

A.1 Chapter 5 -Architecture Figure A.1 shows the architecture for the Generator and Discriminator (the latter with the classification branches). Here, d id , d exp and d noise are the dimensions for identity, expression and noise, respectively; n id is the number of distinct labels for identity, and n exp the number of distinct labels for expression. We use Leaky ReLU with a slope of 0.2.

A.2 Chapter 5 -Decoupling Evaluation

We train the embedding networks using a Resnet-18 architecture with input images of size 224 × 224. The images contain the orthographic projection of the facial mesh, and the values in the RGB channels encode the normal direction of each vertex, as we found this to give better results than the UV images. The networks were trained using the datasets described in Section 5.4.2 with the provided labels. The threshold is selected such that it maximizes the accuracy on the validation set, while keeping the False Acceptance Rate (FAR) below 10%. We build the validation set by randomly choosing an equal number of positive and negative pairs from the testing split. We choose 0.14 as threshold for identity, which achieves 98.66% accuracy and a FAR of 1.21%.