Prof Liming Chen

Prof Huang Di

Liqun Liu

Shiwei Li

Haoyu Zehua Fu

Keywords: face pose editing, 3D face reconstruction, facial image de-occlusion

Recently, with the development of deep Convolutional Neural Networks (CNNs) and large-scale datasets, face recognition (FR) has made remarkable progress. However, recognizing faces in large poses and under heavy occlusion remains a vital challenge due to the unbalanced training data. Thanks to Generative Adversarial Neural networks (GANs), synthesizing photorealistic multi-view faces and unveiling heavily occluded face images becomes feasible, which significantly facilitates FR and has a wide range of applications in entertainment and art fields. This thesis provides an in-depth study of GAN-based face image synthesis and its application to FR. The current facial image synthesizing methods have featured two main research lines, i.e., 2D-based and 3D reconstruction-based. Our works cover both of them.

For 2D-based face pose editing, current methods primarily focus on modeling the identity preserving ability but are less able to preserve the image style properly, which refers to the color, brightness, saturation, etc. This thesis proposes a novel two-stage approach to solve the style in-consistency problem, where face pose manipulation is cast into pixel sampling and face inpainting. With pixels sampled directly from the input image, the face editing result faithfully keeps the identity and the image style.

For traditional 3D face reconstruction, due to the linear and low-dimensional nature of the 3D Morphable Model (3DMM), the reconstructed textures hardly capture high-frequency details, resulting in blurred textures that are far from satisfactory. Some recent 3D face reconstruction methods have also leveraged adversarial training to improve the texture quality. However, these methods either rely on scarce, non-public 3D face data or complex and costly optimization approach. This thesis proposes a high-fidelity texture generation method, which predicts the global texture of the 3D face from a single input face image. The training is based on the pseudo ground truth blended by the 3DMM and input face textures. Multiple partial UV map discriminators are leveraged to handle the imperfect artifacts in the pseudo ground truth.

In terms of face de-occlusion, we propose a Segmentation-Reconstruction-Guided face de-occlusion GAN, consisting of three parts, a 3DMM parameter regression module N θ , a face segmentation module N S , and an image generation module N G .

With the texture prior provided by N θ and the occluded parts indicated by N S , N G can faithfully recover the missing textures. The proposed method outperforms the state-of-the-art methods quantitatively and qualitatively.

Résumé

Récemment, avec le développement des "Convolutional Neural Networks"(CNNs) et des ensembles de données à grande échelle, la reconnaissance des visages (RF) a fait des progrès remarquables. Cependant, la reconnaissance de visages dans de grandes poses et sous forte occlusion reste un défi vital en raison du déséquilibre des données d'entraînement. Grâce aux "Generative Adversarial Networks" (GANs), il est possible de synthétiser des visages multi-vues photoréalistes et de dévoiler les images de visages fortement occlus, ce qui facilite considérablement la RF et offre un (1) Given an input face image, the UV sampler predicts its face mask m and sampling map S att , based on which samples an incomplete UV map U V spl [START_REF] Bulat | How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)[END_REF] The UV generator will further complete the sampled UV map and output the U V pred .

(3) With an off-the-shelf 3DMM regressor, we predict the shape and texture of the input face image, which is used for getting the ground truth of the U V spl : U V gt and the pseudo ground truth of the U V pred :

U V bl . ( 4

Introduction

With the advent of deep Convolutional Neural Networks (CNNs) [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] and the construction of large-scale databases, computer vision has made tremendous progress in recent years. Various computer vision-related applications have profoundly affected our lives, among which the most widely used is the ubiquitous face recognition (FR) system. As we all know, for CNN-based FR, the performance of a model is mainly influenced by two factors: one is the algorithm, and the other is the data: On the algorithm side, a series of models and metrics have been proposed, making the current FR models smaller, faster and more accurate; on the data side, researchers and companies have built face datasets in the millions or even tens of millions of volumes. However, it is not enough to increase face data in terms of quantity, as the uneven distribution and lack of diversity in the dataset lead to a decrease in accuracy under some extreme conditions, e.g., recognizing faces across large poses and under heavy occlusions. To address these issues, researchers typically use methods such as constructing more balanced and diverse datasets, augmenting training data with geometric transformations/randomly synthesized occlusions, and designing more efficient loss functions and network structures that decouple poses/occlusions from facial features.

Recently, the development of face image synthesizing provides new solutions to such problems. For example, the technique can be used in both the training and testing phases in large-pose face recognition. In the training phase, one can synthesize face images with extreme poses to make the model robust to the view angle; In the testing phase, the profile images can be transformed into frontal and then recognized. In addition, since the human face is the main organ for recognizing people, expressing/understanding emotions, and communicating in our daily lives, synthetic photo-realistic face images have broad application scenarios, including media and entertainment industries, art, etc. However, psychological [7,8] and anatomical [START_REF] Liu | Perception of face parts and face configurations: an fmri study[END_REF][START_REF] Rossion | A network of occipito-temporal face-sensitive areas besides the right middle fusiform gyrus is necessary for normal face processing[END_REF]11] studies have demonstrated that humans have a special perceptual mechanism for faces and that minor imperfections in face images can attract human attention and even bring about huge differences in perception, making face image synthesizing a challenging task.

This thesis focuses on high-fidelity face image synthesizing and its application to FR. Specifically, we cover three sub-domains, 1. face pose editing, 2. texture generation for 3D face reconstruction, 3. face image de-occlusion.

Face Pose Editing

Face pose editing aims to change the view angle of an input face image while keeping its original identity unchanged. Research in this area can be divided into three categories: traditional 3D reconstruction-based methods, GAN-based 2D image generation methods, and the combination of both.

3D Reconstruction-based Methods

Before the widespread use of deep learning, the dominant approaches have focused on 3D reconstruction-based methods. [START_REF] Asthana | Fully automatic pose-invariant face recognition via 3d pose normalization[END_REF] uses 68 facial landmarks to aligns an average 3D face model with the input face image. The face image is then projected onto the aligned model and rotated to present the frontal view. Figure 1.1 shows its process. The method has two obvious weaknesses: firstly, it is unreliable to fit all face images with an average face model, regardless of the expression and the identity, and secondly, it cannot handle the self-occluded textures, resulting in significant artifacts in the frontalized face images. [12] also employs an average 3D face to perform face frontalization. They propose a soft-symmetry strategy to fill the invisible part of the profile image with pixels from its visible symmetric region.

Despite the visual improvements, this method still fails to produce photo-realistic face images. A more commonly used 3D face model, in addition to the average 3D face above, is the 3D Morphable Model (3DMM) [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF]. It has been widely used in Figure 1.1: Overview of 3D pose normalization of [START_REF] Asthana | Fully automatic pose-invariant face recognition via 3d pose normalization[END_REF] 3D face-related researches since its appearance. The model is a vector basis of the shape and texture learned from a set of 3D face scans. One can get the weights of those vectors, as well as the lighting conditions, by optimizing reconstruction losses of the 2D face images [START_REF]Face recognition based on fitting a 3d morphable model[END_REF]. With the reconstructed 3D face, face images at any target pose can be obtained by 3D geometrical transformation and 2D projection.

Nevertheless, the 3DMM is constructed by a small number of face scans under well-controlled conditions, limiting its diversity to identity, race, age, gender, etc.

Besides, due to the linear and low-dimensional nature of the model, it can hardly capture high-frequency details. In summary, the traditional 3D reconstruction-based methods mainly face the following difficulties:

• The shape is not accurate enough.

• The synthesized textures are not photo-realistic, either sampled from the input image or reconstructed from the texture model.

• The optimization process is complex and costly.

GAN-based Methods

Thanks to the breakthroughs of Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF], numerous GAN-based face pose editing approaches have been proposed [START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF]17,[START_REF] Hu | Pose-guided photorealistic face rotation[END_REF][START_REF] Qian | Unsupervised face normalization with extreme pose and expression in the wild[END_REF], significantly improving the visual realism of the synthesized images. Typically, these methods contain an encoder and a decoder and follow an adversarial training strat-egy. DR-GAN [START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF] encodes face images to the feature space, then feeds it to the decoder together with a one-hot pose label. The generator will generate the face image with the indicated pose. A multi-task discriminator is employed to supervise both the identity preservation ability and the pose editing ability of the generator.

However, without the supervison of pixel-level loss, DR-GAN's reults are of low quality. TP-GAN [17] introduces a dual-path generator that focuses on local and global transformations, respectively, to frontalize the input face images. CAPG-GAN [START_REF] Hu | Pose-guided photorealistic face rotation[END_REF] proposes a couple-agent discriminator to distinguish generated pairs from ground-truth pairs, one agent for pose manipulation and the other for identity consistency; the introduction of 2D landmark heat maps as pose representation makes pose editing more flexible than DR-GAN or TP-GAN. FNM [START_REF] Qian | Unsupervised face normalization with extreme pose and expression in the wild[END_REF] makes the following improvements comparing to TP-GAN: 1. replacing the local path generator by multiple local and global discriminators, 2. applying pixel-wise loss only on frontal inputs (since their corresponding outputs should remain as they are), avoiding the need for paired training data. Most of the above methods are trained on controlled datasets, such as Multi-PIE [START_REF] Gross | Multi-pie[END_REF], and focus on the identity preserving ability during pose synthesis, leading to stylistic inconsistencies between the output and the input.

3D Reconstruction & GAN-based Methods

It is natural to think of incorporating the 3D prior in GAN-based methods. FF-GAN [START_REF] Yin | Towards large-pose face frontalization in the wild[END_REF] made the first attempt. Besides the traditional encoder-decoder structure, it trains a 3DMM regression model which regresses the 3D coefficients of the input face, providing additional information for the generation. However, it is unclear how much this implicit provision of 3D information helps the generation, and the quality of the generated images is low. In comparison, 3D-PIM [START_REF] Zhao | 3d-aided deep pose-invariant face recognition[END_REF] explicitly leverages the 3D shape to guide the face frontalization. It is actually a combination of the traditional 3D face reconstruction-based method and TP-GAN. The 3D module generates texture-deficient frontalized faces, while TP-GAN serves to refine them to be photo-realistic. The explicit 3D rotation allows the 3D-PIM to converge faster and requires less training data than pure 2D GAN-based approaches. HF-PIM [START_REF] Cao | Learning a high fidelity pose invariant model for high-resolution face frontalization[END_REF] predicts a UV texture map and a sampling map from the input image. The former is a flattened global facial texture, while the latter stores the coordinates of pixels in the UV map, establishing the correlation between the pixels in the target image and the pixels in the UV map. Although it claims that it does not employ 3DMM to present shape or texture information, its method is equivalent to 3DMM because the ground truth of its sampling map derives from the 3DMM reconstruction results.

Most data-driven face pose editing methods rely on paired training data, which is relatively scarce compared to ordinary face datasets. [5] solves the problem by introducing a rotate and render strategy: Given a face Rd a at pose P a , they rotate and render it to an arbitrary pose P b to get Rd b (Notice that this step is similar to [START_REF] Asthana | Fully automatic pose-invariant face recognition via 3d pose normalization[END_REF]), the artifacts in Rd b don't matter; then, they rotate and render Rd b back to P a to get Rd a . Comparing to Rd a , the missing textures in Rd a are due to the self-occlusion from P b to P a , which is equivalent to convert a real face image from P b to P a . So they only need to train the generator to reconstruct Rd a from Rd a . In general, all GAN-related methods in this section and the previous section perform better than traditional 3D reconstruction-based methods, but they suffer from more or less the following problems:

• The lack of sufficient paired training data [17,[START_REF] Hu | Pose-guided photorealistic face rotation[END_REF][START_REF] Cao | Learning a high fidelity pose invariant model for high-resolution face frontalization[END_REF][START_REF] Yin | Towards large-pose face frontalization in the wild[END_REF].

• Style inconsistency between the input and output [17,[START_REF] Hu | Pose-guided photorealistic face rotation[END_REF][START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF][START_REF] Qian | Unsupervised face normalization with extreme pose and expression in the wild[END_REF].

• Texture inconsistency between outputs of different view angles [START_REF] Hu | Pose-guided photorealistic face rotation[END_REF]5].

Texture Generation for 3D Face Reconstruction

3D face reconstruction has two main goals, shape and texture. Recently, significant advances have been made in single image-based 3D face reconstruction. However, most of the previous works focus on predicting accurate and fine-grained 3D shapes, relatively little work has focused on generating high-fidelity face textures. The existing 3D face texture generation techniques can be divided into three categories:

Texture model-based 3DMM has become an essential foundation for 3D reconstruction since its advent. Earlier approaches regress the 3DMM parameters by solving a non-linear optimization problem [START_REF] Richardson | 3d face reconstruction by learning from synthetic data[END_REF][START_REF] Booth | 3d face morphable models" in-the-wild[END_REF], which is often slow and costly.

With the development of Convolutional Neural Networks, recent studies tend to predict the parameters using learning-based methods [START_REF] Richardson | Learning detailed face reconstruction from a single image[END_REF][START_REF] Guo | Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images[END_REF]3]. Nevertheless, as discussed in section 1.1.1, due to the linear, low-dimensional nature of the model, textures generated by the 3DMM model cannot capture fine details of the face image.

Image generation-based GANs provide a powerful tool for generating photorealistic images. Some recent 3D face reconstruction methods have leveraged GAN-based methods to improve the quality of the generated textures [28,4,[START_REF] Lee | Uncertainty-aware mesh decoder for high fidelity 3d face reconstruction[END_REF].

However, those approaches are highly dependent on large 3D face databases. [28] is trained on a synthetic 3D face database [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], leading to artifacts and blurring generation. The training of [4,[START_REF] Lee | Uncertainty-aware mesh decoder for high fidelity 3d face reconstruction[END_REF] is based on a large UV map dataset, which is not publicly available.

GAN optimization-based

The traditional yet most powerful GANs are trained to synthesize images from noise vectors [START_REF] Karras | Progressive growing of gans for improved quality, stability, and variation[END_REF][START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF][START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]. To leverage the power of a pre-trained GAN, a series of works are established on inverting the image back to a GAN's latent space using optimization-based approaches [START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF][START_REF] Ma | Invertibility of convolutional generative networks from partial measurements[END_REF][START_REF]Invertibility of convolutional generative networks from partial measurements[END_REF][START_REF] Zhu | Generative visual manipulation on the natural image manifold[END_REF]. Similar methods are used to generate the UV map of a face image [START_REF] Gecer | Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction[END_REF][START_REF] Lee | Styleuv: Diverse and highfidelity uv map generative model[END_REF]. First, they train a generator that converts noise vectors into UV maps. Then they directly optimize the latent code to minimize the reconstruction error between the input face image and the image rendered by the generated UV map. Instead of training a UV map generator, [START_REF] Gecer | Ostec: One-shot texture completion[END_REF] first rotates the input image in 3D and optimizes the latent code of the pre-trained StyleGAN to fill in the missing textures, then stitches textures of different view angles by alpha blending to form the final UV map. By far, the optimization-based methods can yield the most realistic face UV maps. Nevertheless, they are usually complex and time-consuming, e.g., GANFIT [START_REF] Gecer | Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction[END_REF] takes 30 seconds to generate the UV map of an input face, while OSTeC [START_REF] Gecer | Ostec: One-shot texture completion[END_REF] takes up to 5 minutes.

In summary, among the current texture generation methods for 3D face reconstruction, those based on texture models cannot yield high-fidelity results due to the model's simplicity; those based on image generation rely heavily on large training dataset; those based on optimization are time-consuming and require a high computational cost.

Face Image De-occlusion

The study of face images is a popular area in computer vision with vast application scenarios. In the past decade, deep learning has boosted many impressive face-related techniques, such as face recognition, expression recognition, 3D face reconstruction, and facial landmark detection. However, faces obtained in unconstrained environments are often occluded by various objects, e.g., masks, glasses, hands, scarfs, etc., leading to the degraded performance of the techniques above.

Therefore, faithfully recovering face images from occlusions is an important research direction.

Before widely used deep learning-based methods, the dominant approaches were based on matching and copying patches from known regions to the invisible regions.

A representative one is PatchMatch [START_REF] Barnes | Patchmatch: A randomized correspondence algorithm for structural image editing[END_REF], which recursively searches the nearest neighbor textures to fill in the holes. Such copy-and-paste-based methods work well in recovering simple low-frequency textures and obtaining smooth images; however, they cannot recover high-level textures with complex structures, for example, missing nose/mouth in occluded face images. SSDA [START_REF] Xie | Image denoising and inpainting with deep neural networks[END_REF] proposes a deep auto-encoder to remove specific noises and occlusions in the image, pioneered the use of deep neural networks for denoising and inpainting images. Its impressive performance led to the prosperity of deep learning-based image inpainting methods. Inspired by the GAN-based image generation method, Context Encoder [START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF] introduces adversarial training to image inpainting for the first time. Since then, the combination of auto-encoder and discriminator has been adopted as the basic model structure for image inpainting tasks. To make the model concentrate more on the missing parts [START_REF] Li | Generative face completion[END_REF][START_REF] Yu | Generative image inpainting with contextual attention[END_REF] leverage two discriminators: the global discriminator takes the whole image as input, while the local discriminator takes only the small region around the missing part. However, such a design is only suitable for a single rectangular(or even square) hole and cannot be applied to images with irregular and arbitrarily distributed holes. To properly handle free-form masks, [START_REF] Liu | Image inpainting for irregular holes using partial convolutions[END_REF] propose a Partial Convolutional Layer, comprising a masked and re-normalized convolution operation followed by a mask-update step. Precisely, they fuse mask information into the forward path, and each convolution operation considers only the features corresponding to the visible positions. [START_REF] Yu | Free-form image inpainting with gated convolution[END_REF] pointed out that each neuron's receptive field expands with the forward propagation, making the corresponding mask inaccurate. Furthermore, it is not reasonable to "hard-gate" features in all channels equally. Therefore, they propose a gated convolution module, which learns a soft mask for the features in different layers. Due to the ineffectiveness of CNNs in modeling long-term correlations between distant textures and the hole regions, the inpainting results often have boundary artifacts and look unreal. [START_REF] Yu | Generative image inpainting with contextual attention[END_REF] propose a Contextual Attention module, which utilizes the image features as a convolution kernel, forcing the distant textures interact with each other.

The common problem of the above methods is that they all require manually masking out the occlusions when applied to the face de-occlusion task. ByeGlass-GAN [START_REF] Lee | Byeglassesgan: Identity preserving eyeglasses removal for face images[END_REF] jointly trains an eyeglass-face segmentation decoder, which shares the same encoder with the face decoder. The implicit segmentation hints extracted by the encoder, rather than the manual mask, guide the face decoder to remove the eyeglasses. [START_REF] Cai | Semi-supervised natural face de-occlusion[END_REF] propose a semi-supervised face de-occlusion method. Benefiting from an elaborated forward process, the model automatically learns to predict occlusions without ground truth. [START_REF] Yuan | Face de-occlusion using 3d morphable model and generative adversarial network[END_REF] use the 3D face reconstruction results as priors and feed them into the generator along with the occluded faces. Since the 3D reconstruction results are occlusion-free, implicitly provide the occlusion information, their model directly de-occludes the face image without relying on the segmentation task. [START_REF] Dong | Occlusion-aware gan for face deocclusion in the wild[END_REF] proposed a two-stage GAN, where the first GAN aims to reconstruct the occlusion part solely, and the second GAN takes the result of the first GAN as a hint making the input image occlusion-free. Although the above methods do not require manual labeling, they all need synthetic occluded face images for training. Thus the type and amount of synthetic occlusions are crucial.

In summary, there are two factors that determine the effect of face de-occlusion, one is how to detect the occlusions' position and the other is how to generate realistic faces. Existing algorithms either require manually labeled occlusion regions or a large amount of simulated data for training, and the variety of occlusions are often too limited, resulting in poor generalization ability.

Contributions

The contributions of this thesis are listed as follows:

• In terms of face pose editing, we propose a novel "Pixel Attention Sampling" module, casting the problem into pixel sampling and image in-painting, explicitly preserving more style information of the input image and reducing the reliance on paired training data.

• For 3D face texture generation, a novel weakly supervised learning strategy is proposed, bypassing the need for scarce 3D data and the costly optimization approach.

• A segmentation-reconstruction-guided facial image de-occlusion framework is proposed, which aggregates face segmentation, 3D face reconstruction, and face generation. The method outperforms the SOTAs qualitatively and quantitatively.

Outline

The rest of this thesis is organized as follows:

• Chapter 2 -Literature Review
We review all related work in this chapter, including the fundamentals of GANs and the techniques we use in our research.

• Chapter 3 -Pixel Sampling for Style Preserving Face Pose Editing

This chapter focuses on preserving the style information while performing pose editing of 2D facial images. A novel two-stage face pose editing method is proposed, a Pixel Attention Sampling module is designed.

• Chapter 4 -Texture Generation for 3D Face Reconstruction This chapter introduces a weakly-supervised texture generation method for 3D face reconstruction. A large pseudo texture dataset is generated based on Poisson Blending. A multi-head sampler module is designed to 1) extract the in-complete UV map from face images, 2) mask out the occlusions.

• Chapter 5 -Facial Image De-occlusion GAN This chapter proposes a segmentation-reconstruction-guided facial image deocclusion method. A novel occluded face image dataset is proposed, with both the ground truth occlusion and the synthesized occlusion. An occlusion robust face reconstruction module and a face segmentation module are effectively trained based on the dataset. A novel facial de-occlusion module which can handle arbitrary occlusions is proposed.

• Chapter 6 -Conclusions and Future Work

The work of this paper is summarized, and the next step of work is proposed.

Chapter 2

Literature Review

This thesis focuses on synthesizing photo-realistic face images, where the most crucial factor determining the generation quality is the model's design. Therefore, this chapter first introduces several representative generative models, including Variational AutoEncoder(VAE) [START_REF] Kingma | Auto-encoding variational bayes[END_REF], Generative Adversarial Networks(GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF]. And then focus on several essential modifications to the traditional GANs, including Wasserstein GAN (WGAN), Wasserstein GAN with gradient penalty (WGAN-GP), spectrally normalized GAN (SN-GAN), Self-Attention GAN (SAGAN), followed by the evaluation metrics of the generation quality, i.e., Fréchet Inception Distance (FID), Inception Score(IS). Finally, we summarize the chapter.

Generative models

Machine learning models can be roughly classified into two categories: the discriminative model and the generative model. Given a set of data X and their corresponding labels Y , the discriminative model captures the probability of Y conditioned by X, P (Y |X); the generative model captures the joint distribution of X and Y , P (X, Y ).

As shown in Figure 2, the discriminative model predicts data categories by learning the boundary between categories, whereas the generative model generates data indistinguishable from the real data by modeling the data distribution. Generative models tackle a more difficult task than discriminative models since they have to consider all aspects of the data in the real world. In contrast, the discriminative models only need to concentrate on the differences between classes. For example, it is much more challenging to train a model to generate photo-realistic cat/dog

Variational Autoencoder

Variational Autoencoder (VAE) [START_REF] Kingma | Auto-encoding variational bayes[END_REF] combines the deep neural networks methods with statistical learning to approximate the real data distribution by simple distributions. Like the traditional autoencoder [START_REF] Ng | Sparse autoencoder[END_REF][START_REF] Vincent | Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion[END_REF], it is composed of an encoder, which encodes the data into the latent space, and a decoder, aiming to reconstruct the input from the latent code. The difference is that VAE maps the inputs to distributions (regularized by a predefined simple distribution, e.g., standard multivariate normal distribution) rather than isolated points, and the decoder's inputs are sampled from the encoded distributions, Figure 2.2 illustrates the difference between the two models. Therefore, the VAE's hidden space has the following properties:

1. Continuity, two similar codes will decode to similar data; 2. Completeness, any code in the hidden space can be mapped to the data space. Mathematically, the

p(x i ) = p(x i |z; θ)p(z)dz, (2.1)
where x i is sampled from the real data, z is the latent code sampled from a known distribution p(z), and θ represents the parameter of the decoder. However, the maximization of Equation 2.1 is intractable. Therefore, VAE seeks to maximize the Evidence Lower Bound (ELBO) of logp(x i ), which is its core idea. The encoder aims to approximate the posterior distribution of p(z|x i ), denoted as q φ (z|x i ),

where φ is its parameter. Thus, the Kullback-Leibler (KL) divergence between two distributions is given by,

D KL (q φ (z|x i ) p(z|x i )) = -q φ (z|x i )log p(z|x i ) q φ (z|x i ) dz > 0, (2.2)
By Bayes' rule and rules of logarithms, Equation 2.2 is further derived as,

-q φ (z|x i )log p(z|x i ) q φ (z|x i ) dz > 0, -q φ (z|x i )log p θ (x i |z)p(z) q φ (z|x i )p(x i ) dz > 0 -q φ (z|x i )log p(z) q φ (z|x i ) dz -q φ (z|x i )logp θ (x i |z)dz+ q φ (z|x i )logp(x i )dz > 0 D KL (q φ (z|x i ) p(z)) -E z∼q φ (z|x i ) [logp θ (x i |z)] + logp(x i ) > 0 logp(x i ) > E z∼q φ (z|x i ) [logp θ (x i |z)] -D KL (q φ (z|x i ) p(z)), (2.3)
The right hand side of Equation 2.3 is the optimization objective of VAE, i.e., the ELBO of the log-likelihood. It has two terms, the first term measures the likelihood of the reconstructed data, and the second term regularizes the posterior distribution by the predefined distribution. The implication of the first term is as follows: we first encode x i to a specific latent distribution q φ (z|x i ), then we sample z from this distribution ∼ q φ (z|x i ), next we decode z back to a new distribution in data space, the likelihood of x i under this distribution is denoted as P θ (x i |z), the higher the P θ (x i |z), the better the reconstruction ability of the model. Thus, the first term of the ELBO serves as the reconstruction error like the traditional autoencoder. More specifically, suppose

p θ (x i |z) = N (f θ (z), σ 2 I), where x i ∈ R k , p θ (x i |z) = 1 (2π) k σ k exp - x -f θ (z) 2 2σ 2 (2.4)
the first term drives as:

E x∼q φ (z|x i ) [logp θ (x i |z)] = - 1 2σ 2 E x∼q φ (z|x i ) [ x -f θ (z) 2 ] + C, (2.5)
where C is a constant. Maximizing the first term of Equation 2.3 is equivalent to minimizing x -f θ (z) 2 . Although p θ (x i |z) is not necessarily normally distributed, this does not change the fact that the first term represents the reconstruction ability of the model. In practice, we can use L 1 loss, L 2 loss, or cross-entropy loss to guide the training.

All that remains is to derive the closed-form of the second term of Equation 2.3, D KL (q φ (z|x i ) p(z)). p(z) is predefined as N (0, I) to facilitate computation and to allow good properties of the hidden space (complete, continuous, easy to sample).

q φ (z|x i ) is also restricted to be a normal distribution N (µ, Σ), where µ, Σ are outputs of the encoder, and Σ is diagonal. We deduce the KL divergence of two multivariate normal distributions below.

By definition, the distribution function of a multivariate normal distribution is as,

N (x; µ, Σ) = 1 (2π) k |Σ| exp - 1 2 (x -µ) T Σ -1 (x -µ) , (2.6)
where µ is its mean and Σ is its covariance matrix.

The KL divergence between two distributions p(x) and q(x) is given by,

D KL (p(x)||q(x)) = p(x) • log p(x) q(x) dx = p(x) (logp(x) -logq(x)) dx = E x∼p [logp(x) -logq(x)].
(2.7)

Let p(x) = N (x; µ 1 , Σ 1 ), q(x) = N (x; µ 2 , Σ 2 ), we get, logp(x) -logq(x) =log 1 (2π) k |Σ 1 | exp - 1 2 (x -µ 1 ) T Σ -1 1 (x -µ 1 ) - log 1 (2π) k |Σ 2 | exp - 1 2 (x -µ 2 ) T Σ -1 2 (x -µ 2 ) = 1 2 log |Σ 2 | |Σ 1 | + 1 2 (x -µ 2 ) T Σ -1 2 (x -µ 2 ) -(x -µ 1 ) T Σ -1 1 (x -µ 1 ) .
(2.8) Thus,

D KL (p(x)||q(x)) = E x∼p [logp(x) -logq(x)] =E x∼p 1 2 log |Σ 2 | |Σ 1 | + 1 2 (x -µ 2 ) T Σ -1 2 (x -µ 2 ) -(x -µ 1 ) T Σ -1 1 (x -µ 1 ) = 1 2 log |Σ 2 | |Σ 1 | + E x∼p (x -µ 2 ) T Σ -1 2 (x -µ 2 ) - E x∼p (x -µ 1 ) T Σ -1 1 (x -µ 1 )
(2.9)

For E x∼p (x -µ 1 ) T Σ -1 1 (x -µ 1 )
, we apply the trace trick 1 ,

E x∼p (x -µ 1 ) T Σ -1 1 (x -µ 1 ) =E x∼p tr (x -µ 1 ) T Σ -1 1 (x -µ 1 ) =E x∼p tr (x -µ 1 )(x -µ 1 ) T Σ -1 1 , (2.10
)

1 s = v T W v = tr(v T W v) = tr(vv T W )
the linearity of the trace operator further gives,

E x∼p tr (x -µ 1 )(x -µ 1 ) T Σ -1 1 =tr E x∼p (x -µ 1 )(x -µ 1 ) T Σ -1 1 =tr Σ 1 Σ -1 1 =tr(I d ).
(2.11)

For E x∼p (x -µ 2 ) T Σ -1 2 (x -µ 2 )
, we need to introduce µ 1 to facilitate the derivation, since x ∼ p = N (x; µ 1 , Σ 1 ),

E x∼p (x -µ 2 ) T Σ -1 2 (x -µ 2 ) =E x∼p ((x -µ 1 ) + (µ 1 -µ 2 )) T Σ -1 2 ((x -µ 1 ) + (µ 1 -µ 2 )) =E x∼p (x -µ 1 ) T Σ -1 2 (x -µ 1 ) + (x -µ 1 ) T Σ -1 2 (µ 1 -µ 2 )+ (µ 1 -µ 2 ) T Σ -1 2 (x -µ 1 ) + (µ 1 -µ 2 ) T Σ -1 2 (µ 1 -µ 2 )
(2.12)

where µ 1 -µ 2 is a fixed vector, thus the expectation of the second and the third term above is zero. We apply the trace trick again for the first term and get,

E x∼p (x -µ 2 ) T Σ -1 2 (x -µ 2 ) =tr(Σ 1 Σ -1 2 ) + (µ 1 -µ 2 )Σ -1 2 (µ 1 -µ 2 ).
(2.13)

Taking Equation 2.11, Equation 2.13 into Equation 2.9 gives,

D KL (p(x) q(x)) = 1 2 log |Σ 2 | |Σ 1 | + tr(Σ 1 Σ - 2 1) + (µ 1 -µ 2 ) T Σ -1 2 (µ 1 -µ 2 ) -tr(I d ) .
(2.14)

Recall that q φ (z|x i ) ∼ N (µ, Σ), where Σ is diagonal,

Σ =         σ 2 1 0 0 • • • 0 0 σ 2 2 0 • • • 0 . . . . . . . . . . . . . . . 0 0 0 • • • σ 2 k        
and q(z) ∼ N (0, I), thus,

D KL (q φ (z|x i ) p(z)) = 1 2 k j=1 (-logσ 2 i + σ 2 i -1 + µ 2 i ). (2.15)
The ELBO in Equation 2.3 is therefore given by,

logp(x i ) > E z∼q φ (z|x i ) [logp θ (x i |z)] - 1 2 k j=1 (-logσ 2 i + σ 2 i -1 + µ 2 i ), (2.16)
where σ 2 i is the i-th element on the diagonal of matrix Σ, and µ i is the i-th element of the mean µ.

Maximizing the ELBO is equivalent to minimize the following loss function,

L(x i ) = -E z∼q φ (z|x i ) [logp θ (x i |z)] + 1 2 k j=1 (σ 2 i + µ 2 i -logσ 2 i -1). (2.17)
As discussed before, E z∼q φ (z|x i ) [logp θ (x i |z)] measures the reconstruction ability, thus could be replaced by typical reconstruction losses, e.g., L 1 loss,

L(x i ) = xi -x i 1 + C 2 k j=1 (σ 2 i + µ 2 i -logσ 2 i -1), (2.18)
where xi denotes the reconstruction result corresponding to x i and C is a hyperparameter that balances the regularization and the reconstruction errors.

As mentioned before, the input z to the decoder follows the distribution N (µ, Σ),

where µ and Σ are the outputs of the encoder. However, when we train the model, sampling directly from the above distribution causes the back-propagation gradient to be blocked, making it impossible for the encoder parameters to be updated, so the reparameterization trick is used here. We sample a random variable ε from N (0, I), and transform it to have the mean µ and covariance matrix Σ, Sample minibatch of m samples x B = {x 1 , ..., x m } from the training data.

z = Σ • ε + µ. ( 2

3:

Encode the minibatch:

µ B , Σ B = q φ (x B ) 4:
Sample noise for the minibatch:

ε B ∼ N (0, I) 5:
Reparameterize the noise:

z B = Σ B • ε B + µ B 6:
Reconstruct the minibatch from the reparameterized parameters:

xB = p θ (z B ) 7:
Compute the loss:

L(x B ) = x B -xB 1 + C k j=1 (Σ 2 Bj + µ 2 Bj -log(Σ 2 Bj ) -1) 8:
Update the parameters φ and θ by the gradients of the loss:

θ k+1 := θ k + ∇ θ L(x B ) φ k+1 := φ k + ∇ φ L(x B ) 9: end for (a) (b)

Generative Adaversarial Networks

The idea of Generative Adversarial Networks (GAN) [START_REF] Goodfellow | Generative adversarial nets[END_REF] originates from the zerosum game in game theory. As shown in Figure 2.4, it consists of two parts, a

Generator Discriminator Real Fake? Similar to the decoder of the VAE, the generator maps random hidden variable z ∼ N (0, I) to the data space to generate fake data x. The discriminator is a binary classifier predicting whether its input is real.

Mathematically, D and G are trained in an adversarial paradigm with the following value function:

min G max D V (D, G) = E x∼p data (x) [logD(x)] + E z∼pz(z) [log (1 -D(G(z)))], (2.20)
where p data (x) and p z (z) represent the real data distribution and the random noise distribution, respectively.

The discriminator has two kinds of inputs, the real data x ∼ p data (x) and the fake data G(z) generated by the generator from the noise z ∼ p z (z). It is trained by the Cross-Entropy loss:

L D = - 1 m m i=1 [logD(x i ) + log (1 -D(G(z i )))] , (2.21)
where m denotes the batch size.

The generator takes the noise z ∼ p z (z) as input and generates the fake data G(z), which is expected to trick the discriminator. Thus, its loss function is as,

L G = 1 m m i=1 log (1 -D(G(z i ))) . (2.22)
However, the generated data is very unrealistic at the beginning of the training, such that the discriminator can easily identify them correctly. Therefore, D(G(z i ))

tends to be zero. The derivative of Equation 2.22 with respect to the generator parameters θ G is as:

dL G (z i ) dθ G = - 1 1 -D(G(z i )) • dD(G(z i )) dG(z i ) • dG(z i ) dθ G . (2.23)
On the right side of the above equation, the first term tends to 1, while the second term tends to 0 (because D(G(z i )) is roughly a constant function whose derivative is 0), causing the overall derivative to tend to 0. Therefore, in practice we usually use the following modified version of the generator loss:

L G = - 1 m m i=1 log(D(G(z i ))), (2.24)
whose derivative with respect to θ G is as:

dL G (z i ) dθ G = - 1 D(G(z i )) • dD(G(z i )) dG(z i ) • dG(z i ) dθ G . (2.25)
The above equation is similar to Equation 2.24, except for the first term, which tends to infinity rather than 1. Thus the derivative calculated by Equation 2 for k steps do

3:

Sample minibatch of m samples {x 1 , ..., x m } from real data p data (x).

4:

Sample minibatch of m noise samples {z 1 , ..., z m } from noise prior p z (z).

5:

Update the discriminator parameters:

θ k+1 D := θ k D + ∇ θ D 1 m m i=1 [logD(x i ) + log (1 -D(G(z i )))] 6:
end for

7:

Sample minibatch of m noise samples {z 1 , ..., z m } from noise prior p z (z).

8:

Update the generator parameters:

θ k+1 G := θ k G + ∇ θ G 1 m m i=1 log(D(G(z i )))
9: end for

Modification of GANs

Problems of the traditional GAN

GANs have faced two main problems since their appearance: 1. training instability, which requires careful balancing of the generator and the discriminator during training; and 2. mode collapse, which leads to a lack of diversity in the generated samples. In this section, we describe it in detail.

As discussed previously, the discriminator (D) minimizes

L D = -E x∼Pr [logD(x)] -E x∼Pg [log(1 -D(x))], (2.26)
where P r is the real data distribution, and P g is the generated data distribution.

The generator (G) minimizes either

L G = E x∼Pg [log(1 -D(x))] (2.27)
or

L G = E x∼Pg [-logD(x)].
(2.28)

However, they both have flaws.

• Equation 2.27 suffers from the vanishing gradients problem: the better the discriminator is trained, the more severe the gradient vanishing.

• Equation 2.28 is equivalent to an unreasonable distance metric, resulting in unstable gradient and mode collapse.

We will prove them separately in the following.

By letting the derivative of Equation 2.26 with respect to D(x) be zero, the optimal discriminator is derived as:

D * (x) = P r (x) P r (x) + P g (x)
.

(2.29)

Figure 2.5: Example of two probability distributions that do not overlap.

To facilitate the derivation, we add a generator-independent term to Equation 2.27 (thus not changing the optimization objective):

E x∼Pr [log(D(x))],
and substitute the discriminator with the optimal one in Equation 2. [START_REF] Lee | Uncertainty-aware mesh decoder for high fidelity 3d face reconstruction[END_REF].

L G = E x∼Pg [log(1 -D(x))] + E x∼Pr [logD(x)] = E x∼Pg log P g (x) 1 2 [P r (x) + P g (x)] + E x∼Pr log P r (x) 1 2 [P r (x) + P g (x)] -2log2 = KL(P g 1 2 (P r + P g )) + KL(P r 1 2 (P r + P g )) -2log2 (2.30)
According to the definition of the JS divergence, we have

L G = 2JSD(P r P g ) -2log2. (2.31)
Therefore, minimizing the generator loss of Equation 2.27 is equivalent to minimizing the JS divergence of P r and P g , which could be discretized as,

JSD(P r (x) P g (x)) = 1 2 P r (x)log 2P r (x) P r (x) + P g (x) + 1 2 P g (x)log 2P g (x) P r (x) + P g (x) = 1 2 P r (x)log P r (x) P r (x) + P g (x) + P g (x)log P g (x) P r (x) + P g (x)
+ log2.

(2.32) P r and P g barely overlap at the beginning of training(as shown in Figure 2.52 , P r /(P r + P g ) → 1, P g /(P r + P g ) → 1 ), which makes Equation 2.32 converge to a constant log2 under the optimal discriminator, further resulting in the vanishing gradients of the generator. Thus, when we use Equation 2.27 as the generator loss, it is crucial to balance the training of the generator and the discriminator to avoid the vanishing gradients problem and update the generator efficiently.

Next, we concentrate on the second type of generator loss shown in Equation 2.28. We start by representing KL(P g P r ) using the optimal discriminator of Equation 2.29,

KL(P g P r ) = E x∼Pg log P g (x) P r (x) = E x∼Pg log P g (x)/(P r (x) + P g (x)) P r (x)/(P r (x) + P g (x)) = E x∼Pg log 1 -D * (x) D * (x) = E x∼Pg log[1 -D * (x)] -E x∼Pg logD * (x).
(2.33)

Based on Equation 2.30 and Equation 2.31, we have

E x∼Pg [log(1 -D * (x))] + E x∼Pr [logD * (x)] = 2JSD(P r P g ) -2log2. (2.34)
Combining Equation 2.34 with Equation 2.33 yields

E x∼Pg [-logD * (x)] = KL(P g P r ) -E x∼Pg log[1 -D * (x)] = KL(P g P r ) -2JSD(P r P g ) + E x∼Pr [logD * (x)] + 2log2, (2.35)
where the last two terms are independent to the generator. Thus the second type of generator loss is equivalent to KL(P g P r ) -2JSD(P r P g ).

The above transformation reveals that the optimization objective defined by Equation 2.28 contains two mutually exclusive terms: the KL(P g P r ) and the -2JSD(P r P g ). The former aims to reduce the distance between P g and P r , but the latter increases this distance, resulting in gradient instability during training.

Whats more, KL(P g P r ) = P g logP g /P r dx is not symmetric:

• If P r (x) → 1 and P g (x) → 0, x tends to be real instance that cannot be generated by the generator. This case contributes very little to the loss function. • If P r (x) → 0 and P g (x) → 1, x tends to be fake. The corresponding KL divergence will be ∞.

This asymmetry criterion causes the generator to prefer to produce limited kinds of "safe" samples, resulting in insufficient generating diversity, i.e., mode collapse.

Wasserstein GAN

Wasserstein GAN [56] successfully addresses the problems described in the previous section. Instead of the JS/KL divergence used above, it leverages the 1-Wasserstein distance (Earth-Mover distance, EMD)

W[P r , P g ] = inf γ∈Π(Pr,Pg) γ(x, y) x -y dxdy = inf γ∈Π(Pr,Pg) E (x,y)∼γ [ x -y ] (2.36)
where Π(P r , P g ) denotes the set of all joint distributions γ(x, y) whose marginals are respectively P r and P g , i.e., γ(x, y)dy = P r (x), γ(x, y)dx = P g (y).

(2.37)

Intuitively, if we consider P r and P g as two piles of earth, Π(P r , P g ) is a set of transport plans that transform P r into P g , the EMD is the minimum "cost" corresponding to the optimal transportion. Wasserstein distance is superior to the KL/JS divergences in that it quantifies the distance of two distributions that do not overlap at all. Take two 2D distributions in Figure 2.7 as an example: P 1 and P 2 are one-dimensional distributions with the same shape, placed at different locations along the y-axis of the 3D space at a distance of θ . Obviously,

KL(P 1 P 2 ) =      +∞, if θ = 0 0, otherwise (2.38
)

JSD(P 1 P 2 ) =      log2, if θ = 0 0, otherwise (2.39)
W(P 1 , P 2 ) = inf γ∈Π(P 1 ,P 2 ) E (x,y)∼γ [ x -y ] = θ. (2.40)
The JS/KL divergences are not continuous, taking either the maximum or minimum value. In comparison, the Wasserstein distance is smooth and correctly measures the distance between the two distributions. If we were to optimize θ using gradient descent, the first two could not provide an effective gradient at all, but the Wasserstein distance could. While training GAN to map low-dimensional noise vectors to the high-dimensional image space, the generated distribution has a high probability of not overlapping with the real distribution, thus making the KL/JS divergences ineffective optimization targets.

The infimum in the definition of the Wasserstein distance (Equation 2.36) is highly intractable, making it difficult to serve as a loss function. Thanks to the Kantorovich-Rubinstein duality [START_REF] Villani | Optimal transport: old and new[END_REF], it has the following equivalent form:

W(P r , P g ) = 1 K sup f L ≤K E x∼Pr [f (x)] -E x∼Pg [f (x)], (2.41)
where f L ≤ K denotes the space of K-Lipschitz continuous functions f : X → R.

Given two metric spaces (X, d x ) and (Y, d Y ), where d x denotes the metric on the set X and d Y is the metric on the set Y , a function f :

X → Y is called Lipschitz continuous if there exists K ∈ R, K ≤ 0 such that ∀x 1 , x 2 ∈ X, d Y (f (x 1 ), f (x 2 )) ≤ Kd x (x 1 , x 2 ), (2.42)
K is called the Lipschitz constant of the function f . Simply, if f : R → R, the above requirement is equivalent to restricting the absolute value of the derivative of f not exceeding K.

Equation 2.41 searches function f L ≤ K that maximizes E x∼Pr [f (x)] - E x∼Pg [f (x)].
If we use w to parametrize a family of functions {f w } w∈W that are all K-Lipschitz for some K, the optimization problem is converted to

KW(P r , P g ) = max w∈W E x∼Pr [f w (x)] -E x∼Pg [f w (x)]. (2.43)
In practice, f w (x) could be approximated by a neural network due to its powerful fitting capabilities. In order for the neural network to be Lipschitz continuous,

[56] simply clamps the network's parameters to a fixed range W = [-c, c], i.e.

[-0.01, 0.01] in the paper, after each gradient update. In this case, ∂fw ∂x will not exceed a certain value, the Lipshcitz continuity condition is satisfied.Thus we can approximate the Wasserstein distance by maximizing

L = E x∼Pr [f w (x)] -E x∼Pg [f w (x)], (2.44)
where f w is the neural network paramatrized by w ∈ [-c, c]. The Lipschitz constant K is omitted as it is positive and only affects the magnitude of the gradient without changing its direction.

Equation 2.44 is exactly the objective function of the discriminator. It is easy to observe that its goal is to train the discriminator to maximize the distinction between the real and the generated distribution. In traditional GANs, the discriminator outputs a real number between [0, 1], indicating the probability that a sample is real.

In comparison, the discriminator of the Wasserstein GAN (WGAN) approximates the Wasserstein distance of two distributions, which is not limited to [0, 1] anymore.

Thus the last layer of the W-GAN is no longer the Sigmoid layer.

The generator maps a noise vector z ∼ p z to the data space G(z) ∼ P g . And its goal is to minimize approximated Wasserstein distance between the generated distribution P g and the real distribution P r . The overall objective function of W-GAN is as

min θ max w E x∼Pr [D(x; w)] -E z∼pz [D(G(z; θ); w)], (2.45)
where w and θ are the discriminator and generator parameters, respectively.

By replacing the KL/JS divergences with Wasserstein distance, WGAN has the following achievements:

• It solves the problem of instability of adversarial training, and one no longer needs to carefully balance the training process of the generator and the discriminator.

• It solves the problem of mode collapse and generates images with a high degree of diversity.

• The Wasserstein distance could be used as a criterion indicating the training process; the smaller the value, the better the GAN is trained.

The training process of WGAN is described in Algorithm 3.

Algorithm 3

Training algorithm of WGAN. α = 5e -5 , c = 0.01, n critic = 5.

Require: α, the learning rate; c, the clipping parameter; m, the batch size; n critic , the number of iterations of the discriminator per generator iteration. Require: Initial parameters of D and G: w 0 , θ 0 .

1: while θ has not converged do 2:

for t = 1, ..., n critic do 3:
Sample a batch of real data {x (i) } m i=1 ∼ P r .

4:

Sample a batch of latent variables {z (i) } m i=1 ∼ p(z).

5:

g w ← ∇ w [ 1 m m i=1 f w (x (i) ) -1 m m i=1 f w (g θ (z (i) ))] 6: w ← w + α • RMSProp(w, g w ) 7: w ← clip(w, -c, c) 8:
end for

9:

Sample a batch of latent variables {z (i) } ∼ p(z).

10: According to the properties of the Lipschitz continuous functions, if D :

g θ ← -∇ θ 1 m m i=1 f w (g θ (z (i) )) 11: θ ← θ -α • RMSProp(θ,
X → R is K-Lipschitz continuous, we have, ∇ x D(x) p ≤ K.
(2.46)

Given p = 2 and K = 1, we can say that the L2 norm of the gradient of a 1-Lipschitz continuous function does not exceed 1. Therefore WGAN-GP [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] imposes

L = -E x∼Pr [D(x)] + E x∼Pg [D(x)] + λE x∼P x [( ∇ xD(x) 2 -1) 2 ], (2.47)
where P x is the distribution of the entire sample space. Due to the curse of dimensionality, it is challenging to estimate the expectation over the high-dimensional sample space. Therefore, the proposed method samples data by randomly interpolating the real and generated pairs,

x r ∼ P r ,x g ∼ P g , ε ∼ Uniform[0, 1], x = εx r + (1 -ε)x g .
(2.48)

The left side of Figure 2.8 shows that the gradients of the WGAN-GP remain stable as the training progresses and do not vanish or explode, as is the case with WGAN. The right side of Figure 2.8 shows that the weights learned by WGAN-GP are more diverse, unlike WGAN, where the weights are stacked at the extremes of the clipping interval. The training of WGAN-GP is described in Algorithm 4.

Algorithm 4 WGAN with gradient penalty. λ = 10, n critic = 5, α = 1e -4 , β 1 = 0, β 2 = 0.9. Require: λ, gradient penalty coefficient; m, the batch size; α, β 1 , β 2 , Adam hyperparameters; n critic , the number of iterations of the discriminator per generator iteration; Require: Initial parameters of D and G: w 0 , θ 0 .

1: while θ has not converged do 2:

for t = 1, ..., n critic do 3:

for i = 1, ..., m do 4: Sample x ∼ P r , latent variable z ∼ p(z), a random number ε ∼ U [0, 1]. 5:
x ← G θ (z)

6:

x ← εx + (1 -ε)x 7:

L (i) ← D w (x) -D w (x) + λ( ∇ xD w (x) 2 -1) 2 8:
end for

9: w ← Adam(∇ w 1 m m i=1 L (i) , w, α, β 1 , β 2 ) 10:
end for 11:

Sample a batch of latent variables {z (i) } ∼ p(z).

12:

g θ ← -∇ θ 1 m m i=1 f w (g θ (z (i) )) 13: θ ← Adam(∇ θ 1 m m i=1 -D w (G θ (z)), θ, α, β 1 , β 2 ) 14: end while

Spectral Normalization

As discussed in the previous section, the Lipschitz continuity is a favorable property that allows the discriminator to bypass the unreasonable probability-based loss functions, i.e., KL divergence or JS divergence. Besides, the Lipschitz continuity restricts the upper bound of the gradient norm, thus making the training more stable. To ensure Lipschitz continuity of the discriminator, WGAN uses weight clipping, while WGAN-GP imposes a gradient penalty term in the discriminator loss. For the same purpose, spectrally normalized GAN (SN-GAN) [59] applies the spectral normalization to the weights of the layers in the discriminator, making the discriminator 1-Lipschitz. We present it in detail in this section. Unlike WGAN-GP, which restricts the overall discriminator gradient, SN-GAN focuses on the weights of each layer and makes the whole network Lipschitz continuous by normalizing the weights corresponding to each layer.

In the case of discriminators D : I → R, where I denotes the data space. If D is K-Lipschitz continuous, then ∀x, y, ∈ I,

D(x) -D(y) ≤ K x -y , (2.49)
where • denotes the L 2 norm. Since the linear and convolutional layers are essentially matrix multiplication, we first study the Lipschitz continuity of the matrix.

If a linear function is K-Lipschitz at the origin, then it is K-Lipschitz everywhere.

As matrix transformations are linear, a matrix

A : R n → R m is K-Lipschitz if and only if Ax ≤ K x , ∀x ∈ R n . (2.50)
This is equivalent to

Ax, Ax ≤ K 2 x, x , ∀x ∈ I, (2.51)
further equivalent to

(A T A -K 2 )x, x ≤ 0, ∀x ∈ I. (2.52)
As A T A is positive semidefinite, all of its eigenvalues are nonnegative. The orthonormal basis constructed by its eigenvectors is denoted as {ν 1 , ν 2 , ..., ν n }, correspondingly, the eigenvalues are denoted as λ 1 , λ 2 , ..., λ n . Thus, ∀x ∈ I could be decomposed as

x = x 1 • ν 1 + x 2 • ν 2 + ... + x n ν n . (2.53)
Substituting the decomposed x back to Equation 2.52 yields

(A T A -K 2 )x, x = (A T A -K 2 ) n i=1 x i v i , n j=1 x j v j = n i=1 n j=1 x i x j (A T A -K 2 )v i , v j = n i=1 (λ i -K 2 )x 2 i ≤ 0.
(2.54)

Since λ i is nonnegative, to make Equation 2.54 hold for ∀x ∈ R n , we must have

K 2 ≥ max(λ i ). Suppose λ 1 is the largest eigenvalue of A T A, we get K ≥ √ λ 1 ,
where √ λ 1 is also called the spectral norm of A. Thus, the matrix A : R n → R m is K-Lipschitz and the minimum of K is √ λ 1 . To make A further 1-Lipschitz, it suffices to divide √ λ 1 over all elements of A.

For composite function f •g, we have f

• g L ≤ f L g L . A multi-layer neural
network is exactly the composition of matrix multiplication and nonlinear activation functions. The commonly used activation functions, e.g., ReLU, LeakyReLU, Sigmoid, are all 1-Lipschitz and do not affect the overall Lipschitz norm of the model.

Thus, to make the discriminator 1-Lipschitz, we only need to ensure the convolution/linear layers to be 1-Lipschitz, which can be done by dividing the weights by their spectral norms. The remaining task is to efficiently calculate the spectral norm of the weight in the convolution/linear layer.

SN-GAN uses power iteration to estimate the spectral norm of a matrix (layer).

Suppose A is a full rank matrix of n × n. Its eigenvalues are λ 1 , λ 2 , ..., λ n (in descending order), and the corresponding normalized eigenvectors are ν 1 , ν 2 , ..., ν n .

∀x ∈ R n , x = 0, we have x = x 1 • ν 1 + x 2 • ν 2 + ... + x n • ν n , thus Ax = A(x 1 • ν 1 + x 2 • ν 2 + ... + x n • ν n ) = x 1 (Aν 1 ) + x 2 (Aν 2 ) + ... + x n (Aν n ) = x 1 (λ 1 ν 1 ) + x 2 (λ 2 ν 2 ) + ... + x n (λ n ν n ).
(2.55)

We iterate k times of Equation 2.55 and get

A k x = x 1 (λ k 1 ν 1 ) + x 2 (λ k 2 ν 2 ) + ... + x n (λ k n ν n ) = λ k 1 x 1 ν 1 + x 2 λ 2 λ 1 k ν 2 + ... + x n λ n λ 1 k ν n (2.56)
Given λ 1 > λ 2 > ... > λ n (the case where two eigenvalues are equal is not considered as it is too rare for randomly initialized weights), we have

lim k→+∞ λ i λ 1 k = 0, for i > 1.
(2.57)

Thus

lim k→+∞ A k x = λ k 1 x 1 ν 1 . (2.58)
where x 1 and λ 1 are scalars and ν 1 is a vector. That is, ∀x ∈ R n , x = 0, after multiplying sufficient times of A, we get the scaled dominant eigenvector of A. Due to the high dimensionality of W , computing W T W explicitly is computationally intensive. The power iteration in the paper is formulated as

ṽ := W T ũ W T ũ 2 (2.59a) ũ := W ṽ W ṽ 2 (2.59b)
where ũ is initialized by a random vector. Substituting Equation 2.59b into Equation 2.59a yields a fractional expression with a numerator of W T W ṽ. The normalization operation in each step ensures ṽ to be a unit vector while making the algorithm numerically stable.

Lastly, we left multiply both sides of Equation 2.59b by ũT ,

ũT ũ = ũW ṽ W ṽ 2 1 = ũT W ṽ ṽT (W T W ṽ) 1 = ũT W ṽ λ 1 ṽT ṽ 1 = ũT W ṽ √ λ 1 (2.60)
Thus, the spectral norm is Compared to other methods, SN-GAN exhibits better generation performance as well as better robustness to different training settings. In addition, the spectral normalization is computationally efficient and easy to be integrated into other methods, thus widely used in GAN-based applications, including SA-GAN [START_REF] Zhang | Self-attention generative adversarial networks[END_REF], Big-GANs [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF], StyleGAN [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF], etc. Due to its excellent properties, we integrate this module into all GAN-related models in the subsequent chapters.

λ 1 = ũT W ṽ. ( 2

Self-Attention GAN

Due to the limited kernel size of the convolution layer, each convolution operation can only cover a small area around a pixel and cannot capture the long-term dependencies of different image regions. Traditional convolutional neural networks leverage stacked convolution and pooling layers to mitigate this problem. However, such an approach is neither straightforward nor efficient, and there is no guarantee that feature dependencies are well captured after passing through stacked layers.

However, such an approach is neither straightforward nor efficient, and there is no guarantee that feature dependencies are well captured after passing through stacked layers. Specifically for the image generation task, traditional network structures have difficulty learning structural patterns that occur consistently in certain classes (e.g., dogs have well-defined independent four legs that can neither be more nor less).

Contextual Attention [START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF] reshapes the image features to a convolutional kernel to force the distant textures to interact with each other. However, the method is designed only for image completion and has little follow-up work. Self-Attention GAN (SAGAN) successfully applies the self-attention mechanism [START_REF] Wang | Non-local neural networks[END_REF] to image generation, pioneered the self-attention-based generation methods [START_REF] Cao | Gcnet: Non-local networks meet squeeze-excitation networks and beyond[END_REF][START_REF] Lei | Face sketch-to-photo transformation with multiscale self-attention gan[END_REF]. The self-attention mechanism is also widely used in natural language processing [START_REF] Vaswani | Attention is all you need[END_REF], but this is beyond the scope of this thesis. We focus on SAGAN with the overall structure shown in where W represents the kernel's weight.

The attention map β is a normalized weight sized N ×N for each feature location,

β j,i = exp(s ij ) N i=1 exp(s i j)
, where s = f (x) T g(x), (2.62)

β j,i indicates the influence of the features on the i th location to the j th location.

f (x), g(x) ∈ R N ×C are two convolution operations calculated by f (x) = W f x, g(x) = W g x,
where W f and W g are the matrixed 1 × 1 sized kernels' weights, respectively. The input feature itself is also mapped to a new feature by a 1 × 1 convolution with channels unchanged,

h(x) = W h x. (2.63)
Based on the output of the Equation 2.62 and Equation 2.63, the output of the SA layer is as

o = γh(x)β + x, (2.64)
where γ is also a learned parameter, and is initialized by 0. The intuition behind this design is to allow the attention map to gradually exerts its influence as the training progresses.

Evaluation Metrics of GAN

Inception Score

To evaluate the performance of a generative model, we need to focus on two aspects:

1) whether the generated images are photorealistic and 2) whether the generated images are diverse (Some generative models can only simulate the distribution of a subset of the training data, which are caught in the so-called mode collapse).

Inception Score (IS) [START_REF] Salimans | Improved techniques for training gans[END_REF] considers these two aspects in the following way:

• Reality: Feed generated image x to the Inception V3 Network pre-trained on the ImageNet dataset. The model will output a 1000-dimensional vector with each dimension representing the probability of x being to one class. Intuitively, a realistic image should have a high probability on one class and low probability on others. Formally, the entropy of p(y|x) should be low.

• Diversity: If the generation is sufficiently diverse, the generated distribution should be even across all categories. Assuming that we generate 10 000 images, then ideally, ten images should be generated in each of the 1000 categories. Formally, the marginal distribution of the generated images across all categories p(y) should be high.

A good generator should have a sharp p(y|x) and a smooth p(y), thus the KL divergence of the two distributions should be high. Therefore, the IS is formulated as

IS(G) = exp E x∼pg D KL (p(y|x) p(y) , (2.65)
where p(y) is approximated by the empirical distribution p(y),

p(y) = 1 N N i=1 p(y|x (i) ), (2.66)
where N is the number of synthesized image. And the KL divergence is approximated by

D KL (p(y|x) p(y)) = 1 N N i=N p(y|x (i) )log p(y|x (i) ) p(y) . (2.67)
We take the log of Equation 2.65 and make a simple derivation: (2.68)

log(IS(G)) = E x∼pg D KL (p(y|x) p(y)) = x p g (x) i p(y = i|x)log p(y = i|x) p(y = i) = x∼pg,
The effectiveness of IS lies in its ability to recover good estimates of 1) p(y), the marginal class distribution across the generated images, and of 2) p(y|x), the conditional class distribution for generated images x.

However, when the GAN is trained on datasets different from the ImageNet, estimating the above two distributions is problematic. [START_REF] Barratt | A note on the inception score[END_REF] calculated the marginal distribution of CIFAR-10 [67] using Equation 2.66, and find that the top 10 predicted classes are obscured and not relevant to their true categories, which means that p(y) is not a good estimation to p(y). As Equation 2.68 shows, the IS uses E x∼pg p(y|x)logp(y|x) to measure the generation quality, [START_REF] Barratt | A note on the inception score[END_REF] thus calculated this term for CIFAR-10 images, images of random pixels, and images in the test set of ImageNet, and the results are 4.664 bits, 6.512 bits, and 1.97 bits, respectively, meaning that CIFAR-10 images are closer to random pixels than to images of Ima-geNet under this metric, which is not reasonable.

The root cause of the above problem is that IS only considers the generated images and ignores the real data in its formula, i.e., IS cannot reflect the distance between the real and the synthesized data. The basis for IS to judge the reality of the data is from the training set of Inception V3: ImageNet. Thus, in the "worldview"

of Inception V3, any image that does not look like ImageNet is not real.

In conclusion, the Inception Sore is not a good measure of the GANs' perfor-mance. Most GAN-related literature has now abandoned this metric in favor of the more reasonable FID score [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF].

FID score

The Fréchet Inception Distance (FID) was proposed by [START_REF] Dowson | The fréchet distance between multivariate normal distributions[END_REF] which is originally used to measure the distance between two multivariate normal distributions. [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF] use it to measure the distance between the generated and the real image distributions.

As is known, pre-trained neural networks can extract the high-level information of an image, which reflects the essence of the image to some extent. Therefore, [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF] calculates the FID based on the features extracted by the pre-trained Inception V3

network. The FID is formulated as

FID(p g , p r ) = µ r -µ g 2 + Tr(Σ r + Σ g -2(Σ r Σ g ) 1/2 ), (2.69)
where µ _ , Σ _ denote the mean and covariance matrix of the features extracted from the real and synthetic data. • The calculation involves both the generated data and the real data, which is more reasonable than IS. It can be understood that IS measures the generation quality by comparing the generated data with the ImageNet data. In contrast, FID compares the generated data with the training data and is, therefore, more reasonable.

Summary

This chapter mainly focuses on the theoretical underpinnings of the various techniques applied later in the thesis.

Chapter 3

Pixel Sampling for Style Preserving Face Pose Editing To fight the trade-off incurred by the low-dimensional restriction in the feature space, we seek solutions from the high-dimensional embeddings. But to make the condition label not ignored by the decoder, the encoding dimension should not be for the extreme shape changes. Therefore, how to enable face pose editing in the high dimensional feature space is the main problem to be solved.

Introduction

To tackle the challenge above, we present a novel two-stage method and a module named "Pixel Attention Sampling" (PAS) in this chapter. Inspired by the fact that face images of different view angles also share a large number of similar pixels as highlighted by the optical illusion of face images [START_REF]Combine side and front portrait shots to create optical illusion | photoshop cc[END_REF][START_REF]Photoshop tutorial: How to make a bi-directional, optical illusion[END_REF] in Figure 3. In summary, our main contributions are as follows:

• A novel two-stage face pose editing method is proposed, which casts the task of face pose manipulation as face inpainting, thereby enabling it fully utilize the fine details of the given input image by exploiting high-dimensional embedding.

• A new "Pixel Attention Sampling" module is designed, which effectively resolves the conflict between the identity and style preserving.

• The 3D facial landmarks is introduced to represent face poses for the first time, resulting in more flexible pose editing than using the discrete one-hot pose label or ambiguous 2D facial landmarks.

• The proposed method demonstrates competitive performance in comparison with the current state-of-the-art, both qualitatively and quantitatively.

Related Work

Generative Adversarial Network (GAN)

Traditional GAN is composed of a generator and a discriminator. Due to its simplicity and promising effect, most of the recent GAN based algorithms make use of this technique, including SA-GAN [START_REF] Zhang | Self-attention generative adversarial networks[END_REF], BigGANs [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF], StyleGAN [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF],

etc. In our method, SN-GAN is also adopted in the structure.

Image-to-Image Translation

The combination of auto-encoder with discriminator has achieved impressive results in image-to-image translation [START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF][START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF][START_REF] Zhang | Age progression/regression by conditional adversarial autoencoder[END_REF][START_REF] Pumarola | Ganimation: Anatomically-aware facial animation from a single image[END_REF]. In multi-domain image translation tasks, the domain information is provided either to the bottleneck layer of the autoencoder [START_REF] Zhang | Age progression/regression by conditional adversarial autoencoder[END_REF][START_REF] He | Attgan: Facial attribute editing by only changing what you want[END_REF][START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF], or to the entry of the encoder/generator [START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF][START_REF] Pumarola | Ganimation: Anatomically-aware facial animation from a single image[END_REF][START_REF] Hu | Pose-guided photorealistic face rotation[END_REF], by simply concatenating the domain label with the features or input images. Conditional batch normalization [START_REF] Vries | Modulating early visual processing by language[END_REF] and conditional instance normalization (CIN) [START_REF] Dumoulin | A learned representation for artistic style[END_REF] provide another way of introducing the conditional label in addition to concatenation, via predicting the affine parameters of the normalized feature map (either by batch normalization or by instance normalization) from the input label. Here, the CIN technique is exploited in our decoder to avoid the operation of duplicating the label.

In multi-domain image translation, the discriminator is used to not only estimate the image quality, but also control the target domain of the generated image. Our approach borrows the idea of projection discriminator [START_REF] Miyato | cgans with projection discriminator[END_REF], which introduces the reality score and the inner product of the embedded label with the features of the input data.

Face Pose Manipulation

The existing methods can be roughly divided into two categories: 3D reconstruction based, and simple 2D based.

For the 3D based models, DA-GAN [START_REF] Zhao | Dual-agent gans for photorealistic and identity preserving profile face synthesis[END_REF] uses a predefined 3D face model to produce the synthesized faces with arbitrary poses, and the dual agents serve to keep the identity information stable and improve the realism, Feng et al. [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF] train a model to regress the UV map from a single 2D image directly, which records the 3D shape information. Tran et al. [START_REF] Tran | On learning 3d face morphable model from in-the-wild images[END_REF] proposes a framework to learn a nonlinear 3DMM model from a large set of unconstrained face images. FF-GAN [START_REF] Yin | Towards large-pose face frontalization in the wild[END_REF] incorporates 3DMM [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF] into the GAN based structure, where the 3DMM coefficients provide the low-frequency information, while the input image injects high-frequency local information.

For 2D based models, DR-GAN [START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF] learns a disentangled representation of face identity with the supervision of an auxiliary face classifier of the discriminator. TP-GAN [17] employs a two-pathway architecture to preserve both global and local texture information separately, and generates the frontalized face images.

With the guidance of 2D facial landmarks, CAPG-GAN [START_REF] Hu | Pose-guided photorealistic face rotation[END_REF] is able to generate faces of arbitrary poses, where the couple-agent discriminator distinguishes the generated face/landmark pairs and profile/front pairs from ground-truth pairs, such design enables the algorithm generate face images of target poses while keeping the identity unchanged. CR-GAN [START_REF] Tian | Cr-gan: learning complete representations for multi-view generation[END_REF] trains the generator to produce face images directly from the noises, together with the training of pose manipulation, maintaining the completeness of the learned embedding space. FNM [START_REF] Qian | Unsupervised face normalization with extreme pose and expression in the wild[END_REF] employs unsupervised training and synthesizes normalized face images of Multi-PIE [START_REF] Gross | Multi-pie[END_REF] style. Most of the above methods only focus on modeling the identity preserving ability, whereas they generally ignore the image style preserving ability, such as color, facial expression, lightning, etc. Although it is claimed that the synthesized frontal face images improve the face verification accuracy, the generated face images are visually far from the input images, thus greatly limits their further usage scenarios other than face recognition. [START_REF] Zhang | Face frontalization using an appearance-flow-based convolutional neural network[END_REF] frontalizes the face image by predicting the pixel displacement.

However, it's hard to extend to the arbitrary face pose editing problem due to the time consuming SIFT feature extraction.

Method

The goal of our method is to keep not only the identity but also the image style during face pose manipulation. We first define several notations: (I, J) denotes paired face images in the training set, where I is the source image, and J is the target one. The 3D facial landmarks are denoted as ldmk I and ldmk J , which could be detected by an off-the-shelf 2D\3D facial landmark detector [START_REF] Bulat | How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)[END_REF]. I tf represents the input image after similarity transformation. To guide the training, the landmark based segmentation maps of I tf and J are also required, which we denote as I seg_tf and J seg . These notations are visualized in Figure 3.3.

Our approach is composed of three major steps: preprocessing, pixel attention sampling, and image inpainting. They are described in detail subsequently.

Preprocessing

Given the fact that human faces are roughly left-right symmetrical, thus a face at an arbitrary pose always has at least one side fully exposed to the camera. This preprocessing step aims to align the fully exposed side of face I to that of a target face J.

The inputs of this step are the input face image I, its 3D facial landmarks ldmk I , and the landmarks ldmk J of the image J at a target pose. We first find the fully ex- From Figure 3.3(g), we observe that I tf and J indeed share pixel-level similarities.

Finally, with the 2D facial landmarks of I and the transformation matrix obtained above, we obtain the 2D facial landmarks of I tf , as well as the landmark based segmentation map I seg_tf . Besides, to guide the training process of the PAS module, the segmentation map of the target image J seg is also prepared at this stage.

Pixel Attention Sampling

The previous preprocessing step delivers the input face image with the larger side aligned to the target pose. Despite the fact that the transformed input face image I tf and the face at the target pose J share many similarities in terms of texture, there still exist great gaps between them, from the global shape to the finer details of textures. Therefore, our goal at this stage is to preserve and fine-tune their similar face regions while eliminating the major differences. This is achieved by a novel pixel sampling based module, which we call Pixel Attention Sampling module (PAS), since the process of sampling mainly "focuses" on bridging the gaps. Specifically, given the transformed image I tf and the target pose ldmk J , PAS generates a two-channel coordinate sampling map of the same size as I tf . The first channel holds the abscissa while the second one for the ordinate. Each pixel location of the map is registered a coordinate, indicating which input pixel of I tf that location will sample from. Note, the original pixel indices are converted into decimal coordinates ranging from -1 to 1, for the purpose of gradient backpropagation, and the final sampling is achieved by interpolating the adjacent pixels. Our sampling map is similar to the one used in the spatial transformer network [91]. The difference lies in that the one in [91] is determined by a 2D affine transform matrix, with only six parameters, whereas our sampling map is directly predicted by the neural network, resulting in height × width × 2 parameters in total. The PAS module is composed of two parts i.e., the image embedder and the sampler. The embedder consists of stacked convolution layers, conditional instance normalization [START_REF] Dumoulin | A learned representation for artistic style[END_REF] layers (CIN), and self-attention [START_REF] Zhang | Self-attention generative adversarial networks[END_REF] layers (SA). The CIN layers incorporates the 3D facial landmarks of the target pose to guide the embedding, and the SA layers enable the embedder focus more on the global structure of the face.The embedder finally outputs a 512-dimensional feature vector, which is further fed into the sampler to generate the sampling maps. The sampler is composed of fully connected layers and ReLU layers. After applying the obtained sampling map to the transformed input face image I tf and its corresponding segmentation map I seg_tf , we could obtain the intermediate face image at the target pose, denoted as Ĵfake , and its corresponding fake segmentation map, denoted as J seg_f ake . In order to maintain the reconstruction ability of the module, the original image I and its corresponding 3D landmarks ldmk I are also fed into the PAS module and the reconstructed output Îrecon is achieved.

The training process of the PAS is guided by the following losses:

Pixel-wise loss between Ĵfake and J, Îrecon and I, which is commonly used in the image-to-image translation algorithms. It can be formulated as:

L pix = L 1 ( Ĵfake , J) + 0.1 • L 1 ( Îrecon , I) (3.1)
where

L 1 (I, J) = 1 W HC W,H,C x,y,c=1 |I(x, y, c) -J(x, y, c)| (3.2)
Since it does not take much effort to learn an identity mapping, we set the weight of the reconstruction loss to 0.1, which makes the PAS module concentrate much more on the pose manipulation task.

Segmentation loss between J seg_f ake and J seg . Based on the assumption that if the sampled image Ĵfake is close to the target image J, the segmentation map J seg_f ake should be close to the target segmentation map J seg as well. We therefore introduce a segmentation-related loss so as to push Ĵfake close to J. To facilitate the training converge, the segmentation loss is used as a complement to the aforementioned pixel-wise loss L pix . Here, we make use of the Dice loss [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF], which has been widely exploited in image segmentation tasks, and it can be formulated as:

L seg = N c=1 1 - 2
x,y J c seg (x, y) • J c seg_f ake (x, y)

x,y J c seg (x, y) +

x,y J c seg_f ake (x, y)

(3.3)
where c represents the different classes of facial attributes. Since each pixel location (x, y) of the segmentation map is represented by a c-dimensional one-hot vector, the fraction in Equation 3.3 is thus a simple intersection over union. The benefit of the adopted loss function is that it is independent to the amount of pixels of different classes.

Perceptual loss [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] between Ĵfake and J. Perceptual loss is significant to preserve the identity information and high-level semantic features of the face images.

We follow the work of [START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF] and employ the pre-trained VGG-Face [START_REF] Parkhi | Deep face recognition[END_REF] network to extract the features:

L per = V GG loss ( Ĵfake , J) (3.4) with V GG loss (I, J) = i |V GG F ace (I) i -V GG F ace (J) i | (3.5)
where i is the layer index of the pre-trained model and i ∈ {3,8,15,22,29}, which are the last convolutional layer of each feature map scale.

Total variation loss. Total variation [START_REF] Mahendran | Understanding deep image representations by inverting them[END_REF] loss has been widely used in GAN based algorithms for its powerful ability of reducing the noises and smoothing the generated results. In our PAS module, there inevitably exist obvious noises, since the resultant face image is pixel-wise sampled from the transformed input face image.

Therefore, the TV loss is incorporated:

L tv = T V ( Ĵfake ) + T V ( Îrecon ) (3.6)
where

T V (I) = W -1,H,C
x,y,c=1

|I(x + 1, y, c) -I(x, y, c)| 2 + W,H-1,C x,y,c=1 |I(x, y + 1, c) -I(x, y, c)| 2 (3.7)
The overall training loss of the PAS module is a sum of the above losses: Thanks to the PAS module, we achieve a face image whose facial attributes have been aligned to the target pose location, with the original identity and style characteristics well preserved. It should be noted that, as the sampling is accomplished by interpolating adjacent pixels, it only modifies the location of the pixels within a small area around them, the PAS module is thus not able to sample for instance the left eye from the right one or the opposite. As a result, the sampled face images possibly contain artifacts, holes and noises, as illustrated in Figure 3.5. In order to further improve the generated image quality, image inpainting is introduced subsequently.

L sampler = L pix + L seg + L per + L tv (3.8)

Image Inpainting

The image inpainting stage is to restore the holes and remove the noises and artifacts on the intermediate faces generated by PAS, and finally generate photo-realistic face images. To accomplish this goal, we introduce a Conditional Adversarial Auto-Encoder, where the discriminator is implemented by a projection discriminator [START_REF] Miyato | cgans with projection discriminator[END_REF],

and the auto-encoder is based on the U-Net structure [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. We also make use of CIN layer to merge the information provided by 3D facial landmarks, the identity features, and the image features, thereby making the generated face image in desired pose and shape. More precisely, the inputs of the encoder are the images generated by PAS together with their target poses, i.e., Ĵfake with ldmk J for the task of pose manipulation, and Îrecon with ldmk I for the task of reconstruction. To well preserve the face identity, the decoder is conditioned by the high level feature extracted by the pre-trained LightCNN [START_REF] Wu | A light cnn for deep face representation with noisy labels[END_REF] model, where the parameters of the fully connected layer is fine-tuned during training. The outputs of the auto-encoder are denoted as J f ake and I recon , whose ground truths are J and I, respectively. To further improve the model's generalization ability, unpaired face images could also be exploited to supplement the training set. This is achieved by feeding the network with the partially occluded face images S occ and their 3D landmarks, and expect the network to output S recon restoring the original S. For the discriminator, we feed all the generated images, including J f ake , I recon and S recon , as fake samples, while their corresponding ground truth as the genuine ones, with dis real and dis f ake as output, respectively.

The loss function of the inpainting network is composed of four parts:

Pixel-wise loss, formulated as:

L pix = L 1 (J f ake , J) + λ • L 1 (I recon , I) + L 1 (S recon , S) (3.9)
where L 1 is defined in equation 3.2.

Perceptual loss to capture the semantic similarity:

L per = V GG loss (J f ake , J) + V GG loss (S recon , S)

(3.10)
where V GG loss is defined in equation 3.5. We do not include (I recon , I) here, because compared to image reconstruction task, image inpainting and pose manipulation are more likely to lose the identity consistency.

Identity loss to maintain the identity-related characteristics stable. We use the pre-trained LightCNN [START_REF] Wu | A light cnn for deep face representation with noisy labels[END_REF] to extract the identity feature of the synthesized image and the target image, and minimize the L 1 loss of them:

L id = 1 N N i=1 |F (J f ake ) i -F (J) i | (3.11)
Adversarial loss to guarantee the generated image quality:

L adv = -dis f ake (3.12)
Besides, we also incorporate the total variation loss to reduce the spike artifacts.

The overall training loss of the generator of the image inpainting network is a sum of the aforementioned losses:

L gen = L pix + L per + L id + L tv + L adv (3.13)
Following the work of [START_REF] Lim | Geometric gan[END_REF], the discriminator loss is defined as:

L dis = max(1 -dis real , 0) + max(1 + dis f ake , 0) (3.14)

Experiments

Given an input face image, the proposed method aims to manipulate its pose while keeping the identity unchanged along with its style. Correspondingly, we evaluate it in two aspects: the style-conserving skill and the identity-preserving ability during face pose editing. In this section, we present the training details first, then the qualitative analysis for face style conserving, followed by the quantitative results for identity preserving. Ablation studies are also carried out to highlight the effectiveness of the proposed PAS module.

Training details

The training is based on four databases: Multi-PIE [START_REF] Gross | Multi-pie[END_REF], 300W-LP [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], CAS-PEAL-R1 [START_REF] Gao | The cas-peal large-scale chinese face database and baseline evaluations[END_REF], and CelebA [START_REF] Liu | Deep learning face attributes in the wild[END_REF]. Multi-PIE has four sessions with face images under 13 poses and 20 illuminations. We follow Setting 1 of TP-GAN [17] and train the proposed algorithm on the first 150 subjects of session 1, then test on the remained 99 subjects. 300W-LP contains large-pose face images synthesized from 300W [START_REF] Sagonas | 300 faces in-thewild challenge: The first facial landmark localization challenge[END_REF].

After manually filtering out the low-quality images, we have 40,159 images from 2,815 subjects in total. CAS-PEAL-R1 contains 1,040 subjects. For each subject, gray-scale images across 21 different poses are included. CelebA is a large-scale face attributes dataset with more than 200K celebrity images in it.

During the training process, we use the occluded face images as input, and the effectiveness of the proposed method. A further qualitative comparison of our method and CR-GAN [START_REF] Tian | Cr-gan: learning complete representations for multi-view generation[END_REF], DR-GAN [START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF] and FNM [START_REF] Qian | Unsupervised face normalization with extreme pose and expression in the wild[END_REF] are demonstrated in Fig- Quantitative evaluations are further performed. We calculate the FID score of the above models, on the frontalized large pose face images from CelebA, the results are shown in Table 3.1, indicating that the proposed method generates face images with styles closer to the input face images.

Style-conserving validation

Identity-preserving ability evaluation

There are 249 subjects in Session 1 of Multi-PIE. Following the Setting 1 of TP-GAN, we use the first 150 subjects for training, and the remaining 99 subjects for testing. The identity preserving ability is evaluated by Rank-1 recognition rate.

The face with frontal view and normal illumination in the testing set compose the gallery, and the rest non-frontal images are used as probe.

The evaluation is conducted based on the features extracted by the pre-trained Light-CNN model. We directly extract the features of the probe images as baseline.

For the proposed method, we first frontalize the probe face images, based on which their face representations are extracted. As can be seen from Table 3.2, the proposed method achieves similar or even better Rank-1 recognition rate in comparison with the baseline and state-of-the-art algorithms when the rotation angle is smaller than 60 • . For larger rotation angles (≥ 60 • ), the proposed algorithm drastically outperforms the baseline, whereas it does not perform as well as the SOTA algorithms. There exist two possible reasons: 1) The face images of extreme poses share relatively less pixels with the face images of front view, thus the pixels sampled by the PAS module are not sufficient enough for the following inpainting stage, and

2) most of the SOTA algorithms normalize the face images into a consistent style, where the information irrelevant to identity is filtered out, in contrast, our method preserves relatively more style information.

Ablation Study

To highlight the effectiveness of the PAS module, the ablation study is conducted by removing it and training the U-Net based conditional adversarial auto-encoder directly. For the sake of fair comparison, we apply the same preprocessing pipeline (i.e., align the larger side of the input image to match the target pose) and train the model with the same number of iterations.

Conclusion

In this work, we first carefully analyze the trade-off between the style-preserving ability and the identity-preserving ability of the existing 2D based pose manipulation methods. Based on the observation that face images in different poses share a large number of pixels, we propose a novel pose editing method and a sophisticatedly designed PAS module. The method selectively samples pixels from the input face and adjust their relative locations with the PAS module, so that the recovered face editing result match the target pose and faithfully keeps the original identity and style information unchanged. In this way, we convert the pose manipulation problem to a image inpainting problem, and further make the best of the finer details in the original face images to obtain convincing pose editing results. We also utilize 3D facial landmarks to represent the face pose, which is more precise and flexible comparing to the one-hot labels and the 2D facial landmarks adopted in previous studies. Extensive experiments validate that the proposed pose editing approach preserves the style information of the input images better than the existing methods.

Chapter 4

Texture Generation for 3D Face Texture model-based Since the 3D Morphable Model (3DMM) [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF] was proposed, it has been widely used in 3D face reconstruction. The model is a vector basis of the shape and texture learned from a set of 3D face scans. Earlier approaches regress the 3DMM parameters by solving a non-linear optimization problem [START_REF] Richardson | 3d face reconstruction by learning from synthetic data[END_REF][START_REF] Richardson | Learning detailed face reconstruction from a single image[END_REF][START_REF] Booth | 3d face morphable models" in-the-wild[END_REF], which is often slow and costly. With the development of Convolutional Neural Networks, recent studies tend to predict the parameters using learning-based methods [START_REF] Richardson | Learning detailed face reconstruction from a single image[END_REF][START_REF] Guo | Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images[END_REF]3]. However, the 3DMM is constructed by a small Image generation-based Generative Adversarial Network (GAN) [START_REF] Goodfellow | Generative adversarial nets[END_REF] provides a powerful tool for generating photorealistic images. Since its appearance, numerous image generation methods with stunning results have been proposed. Thanks to various large databases and the highly structured geometry of the human face, 2D face image generation is one of the most prosperous areas [START_REF] Cai | Fcsr-gan: End-to-end learning for joint face completion and super-resolution[END_REF][START_REF] Huang | Multimodal unsupervised image-to-image translation[END_REF][START_REF] Pumarola | Ganimation: Anatomically-aware facial animation from a single image[END_REF][START_REF] Portenier | Faceshop: Deep sketch-based face image editing[END_REF][START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF][START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF][START_REF] Karras | Progressive growing of gans for improved quality, stability, and variation[END_REF]. Influenced by this trend, some recent 3D face reconstruction methods have also leveraged adversarial training to improve the texture quality [28,4,[START_REF] Lee | Uncertainty-aware mesh decoder for high fidelity 3d face reconstruction[END_REF]. However, such an image generation approach is highly dependent on a large 3D face database. [28] is trained on a synthesized 3D face database [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], where originally self-occluded textures are obtained by simple interpolation of visible parts, resulting in imperfect generation. [4,[START_REF] Lee | Uncertainty-aware mesh decoder for high fidelity 3d face reconstruction[END_REF] are trained on a large UV map dataset, which is not publicly available.

GAN optimization-based

The traditional yet most powerful GANs are trained to synthesize images from noise vectors [START_REF] Karras | Progressive growing of gans for improved quality, stability, and variation[END_REF][START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF][START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]. To leverage the power of a pre-trained GAN, a series of works are established on inverting the image back to a GAN's latent space using optimization-based approaches [START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF][START_REF] Ma | Invertibility of convolutional generative networks from partial measurements[END_REF][START_REF]Invertibility of convolutional generative networks from partial measurements[END_REF][START_REF] Zhu | Generative visual manipulation on the natural image manifold[END_REF]. Similar methods are used to generate the UV map of a face image [START_REF] Gecer | Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction[END_REF][START_REF] Lee | Styleuv: Diverse and highfidelity uv map generative model[END_REF]. First, they train a generator that converts noise vectors into UV maps. Then they directly optimize the latent code to minimize the reconstruction error between the input face image and the image rendered by the generated UV map. Instead of training a UV map generator, [START_REF] Gecer | Ostec: One-shot texture completion[END_REF] first rotates the input image in 3D and optimizes the latent code of the pre-trained StyleGAN to fill in the missing textures, then stitches textures of different view angles by alpha blending to form the final UV map. By far, the optimization-based methods can yield the most realistic face UV maps. Nevertheless, they are usually complex and time-consuming, e.g., GANFIT [START_REF] Gecer | Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction[END_REF] takes 30 seconds to generate the UV map of an input face, while OSTeC [START_REF] Gecer | Ostec: One-shot texture completion[END_REF] takes up to 5 minutes.

Besides generating a global face texture, we note that a series of pure 2D image generation methods can also synthesize face images of different view angles [START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF]5,[START_REF] Hu | Pose-guided photorealistic face rotation[END_REF]. However, the generation consistency is poor due to the absence of global consistency constraints and a priori knowledge of the 3D shape.

In summary, among the current texture generation methods for 3D face reconstruction, those based on texture models cannot yield high-fidelity results due to the model's simplicity; those based on image generation rely heavily on large training dataset; those based on optimization are time-consuming and require a high computational cost.

To this end, we propose a novel image-to-image translation model that converts the input face image into its corresponding UV map. The proposed method is image generation-based, therefore much faster than optimization-based methods. We use the pseudo UV map for training, bypassing the dependency on the real UV map dataset. Thanks to multiple partial UV discriminators, we can use cropped parts of incomplete UV maps (acquired using the data pre-processing method provided in [4]) for training to improve the generation quality. Our contributions are as follows:

• A novel image generation-based UV map prediction framework is proposed.

The generated results are comparable to the optimization-based method but much faster.

• With the proposed UV sampler module, the visible face textures can be directly mapped to the UV space, forming an incomplete UV map. No 3D information (shape, occlusion) is required during the inference stage. Therefore our model can be stitched seamlessly with any 3D shape reconstruction models.

• The training is doesn't rely on the real UV map dataset, and the design of multiple discriminators can compensate well for the imperfect ground truth.

• The proposed method outperforms the state-of-the-art methods, both qualitatively and quantitatively.

Related Work

3D shape reconstruction From earlier optimization-based methods to CNN prediction-based methods, acquiring accurate 3D face shape becomes easier and faster, bringing powerful tools and significant opportunities for face-related tasks.

Our training process relies on 3D shape reconstruction of a given face, where numerous 3D shape fitting methods are applicable. In this paper, we adopt an off-the-shelf model [3] as our shape re-constructor, which is the current SOTA 3DMM-based method. The model will predict its corresponding pose and 3DMM shape/texture parameters with a single face image as input.

UV map generation There exist mainly two texture representation methods for 3D models, vertex-based and UV map-based. The vertex-based representation is very intuitive, where each vertex has a color, and the interpolation of those colors generates the texture of the 3D surface. However, such representation flattens the texture into a linear vector, destroys the spatial relationship of texture patches, thus prevents it from leveraging powerful CNN-based methods. The UV map-based representation unwraps the 3D texture into a 2D space. Briefly, each 3D vertex's color is mapped to its corresponding location of a 2D image, and adjacent vertices are mapped to adjacent regions so that the positional relationships between vertices are well preserved. [4] first sample the color of visible 3D vertices from the input face image, then map them to UV space to get the incomplete UV map, in which the generative model will further complete the missing parts. However, their method is highly dependent on the precise 3D shape and ground truth UV maps.

In contrast, our method does not need the UV map data for training or 3D shape for inference. [28] propose a non-linear 3DMM, where the predicted texture takes the UV map-based representation. Nevertheless, their UV map generator's input is a low-dimensional encoding of the input image, resulting in an loss of detail of the predicted UV map. In addition, their model is trained on linear 3DMM synthesized images [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], where artifacts caused by self-occlusion appear frequently. Unlike [28],

our model is trained on real face images, and the coding keeps a large dimension across the forward path, making the generated UV map photorealistic.

Differentiable renderer To obtain the gradient of the loss function and thus train the network, a differentiable renderer is widely used in 3D face-related algorithms [START_REF] Richardson | Learning detailed face reconstruction from a single image[END_REF][START_REF] Guo | Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images[END_REF]28,3]. Briefly, a renderer is composed of a rasterizer and a shader.

The rasterizer applies depth-buffering to select the mesh triangles corresponding to each pixel, and the shader computes the pixel colors as follows:

c = w 0 c 0 + w 1 c 1 + w 2 c 2 (4.1)
where c i is the color of the i th vertex of the mesh triangle the pixel resides in, w i is the barycentric coordinate of the pixel in the triangle. During backward propagation, the gradients are passed from each pixel to the vertices: illustrates the above steps. Obviously, such a method is tedious and relies on an accurate 3D shape fitting. Since the UV map contains only the texture information of a 3D surface, is it really necessary to fit the exact 3D shape before getting the UV map? We do not think so. In fact, the only purpose of the 3D shape is to establish a one-to-one relationship between the pixel in the 2D face image and the pixel in the UV map, so why not learn such a mapping relationship in a data-driven manner? To achieve such a goal, we designed the UV sampler, a CNN-based model that maps the face image's pixels directly to the UV map.

dL dc i = dL dc dc dc i = dL dc w i (4.2)
The model has three parts, i.e., the feature extractor, the segmentation head, and the sampler head. Similar to most generative models, the feature extractor is composed of stacked residual blocks [START_REF] He | Deep residual learning for image recognition[END_REF]. Spectral normalization [59] is applied to each convolution layer to stabilize the training. With this module, 2D feature maps of different scales and a 512-dimensional vector are extracted from the input image.

The 2D feature maps are fed into the FPN structured [START_REF] Lin | Feature pyramid networks for object detection[END_REF] segmentation head and output an attention mask m. Besides, the 1D feature vector is fed into the sampler head, a stack of fully connected layers interspersed with ReLU activations. The sampler head's output is reshaped as S att ∈ R B×256×256×2 , which is the attention sampling map, where B is the batch size, 256 is the height/width of the UV map, and the last two channels hold the normalized abscissa and ordinate of the pixel in the input image to sample. Based on S att and m, differentiable sampling [91] is applied to the masked input image I, and an incomplete UV map U V spl is finally obtained.

To train the model, we use the above-mentioned traditional method to generate the ground truth (incomplete) UV map, U V gt . One issue to note is that the 3D shape fitting is not completely accurate. The projected 3D vertices sometimes could lie on the background part of the face image, resulting in wrong vertices' colors and further wrong U V gt . Moreover, since the surface normal of the face edge is almost parallel to the image plane, numerous projected vertices are piled up in the narrow edge region, aggravating the UV map's inaccuracy. The occlusions such as eye glasses and hairs also cause the wrong UV texture. To mitigate this problem, we first generate a binary mask for the face region based on a pre-trained face segmentation model, then erode the mask's edge to ensure that the region inside of which must be the face, the generation of U V gt only takes into account the vertices that fall inside the mask. Although this would result in a loss of texture near the edge, it is worth sacrificing the unimportant edges to ensure the accuracy of U V gt .

The training is guided by the following loss function:

L spl = U V spl -U V gt 1 + S att -S gt 1 + L seg (m, m gt ) + λT V ( U V spl ) (4.3)
where S att and m are the outputs of the UV sampler, U V spl is the sampled UV map based on them. S gt is the ground truth sampling map, which is obtained by mapping the normalized x,y coordinates of the visible 3D vertices into the UV space, i.e., UV position map [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF]. L seg (m, m gt ) is the binary cross-entropy loss of the predicted face mask.

L seg = -[m gt logm + (1 -m gt )log(1 -m)] (4.4)
T V ( U V spl ) is the total variation loss [START_REF] Mahendran | Understanding deep image representations by inverting them[END_REF] of the predicted UV map, which is powerful in smoothing the noises of the generated UV map.

T V ( U V spl ) = W -1,H,C
x,y,c=1

U V spl (x + 1, y, c) -U V spl (x, y, c) 2 + W,H-1,C x,y,c=1 U V spl (x, y + 1, c) -U V spl (x, y , c) 2 (4.5)
Thanks to the UV sampler, an incomplete UV map could be sampled directly from the input image, bypassing a series of complex and expensive steps of traditional methods, including 3D shape fitting, visible vertices determination, UV map rendering, etc.

UV Map Inpainting

With the UV sampler described above, we can sample an incomplete UV map from a face image. The next task is to fill the missing parts with textures consistent with the sampled parts. This is an image inpainting problem, which has been extensively studied. However, most image inpainting methods are trained on paired images, meaning the ground truth image is uniquely determined. In contrast, in our case, the ground truth is not available. This section studies how to train a UV map inpainting model without the supervision of the ground truth. Briefly, our approach is to generate a pseudo ground truth UV map to assist the training. Then, we work with multiple discriminators to make the generated images as photorealistic as possible.

Pseudo UV Map Generation

Generating the pseudo UV map consists of three steps: 1) incomplete ground truth UV map generation, 2) 3DMM texture fitting, 3) seamless image blending. The first step has been described in detail in the previous section. For the second step, we use directly the BFM [START_REF] Paysan | A 3d face model for pose and illumination invariant face recognition[END_REF] texture parameter predicted by [3]. The UV map representation of the reconstructed BFM texture is denoted as U V bf m . Obviously, due to the linear, low-dimensional nature of the BFM model, U V bf m is far from reality, as can be seen in Figure 4. 4(d). Therefore, we move to the third step: seamless image blending.

The U V gt obtained in the first step can faithfully restore the input image( (face edge excluded)), but it is incomplete due to self-occlusion. While the U V bf m obtained in the second step is complete, but it is only a rough approximation in the 3DMM solution space. Thanks to Poisson image editing [START_REF] Pérez | Poisson image editing[END_REF], we can seamlessly blend the two results by solving the following Poisson equation with Dirichlet boundary conditions:

∆f = div v over Ω, with f | ∂Ω = f * | ∂Ω (4.6)
where in our case, Ω denotes the domain of real textures in U V gt , f is the texture to be modified in the U V bf m , v is the gradient of the texture in U V gt , f * denote the texture of U V bf m outside the Ω. We also leverage the texture of the visible region to fill its missing symmetric region. That is, we do two times Poisson Blending, the first time blends the U V gt to the U V bf m , the second time blends the flipped U V gt to its symmetric missing parts, as illustrated in

Multiple discriminators

The training of the UV generator follows an adversarial paradigm; therefore, a large amount of data from the target domain is essential. However, the pseudo UV map U bl generated above is not very reliable. Its quality depends on the accuracy of U V bf m , the texture area of U V gt , and the accuracy of the 3D shape. We only use the pseudo UV map to calculate the reconstruction loss, which is a rough guide to the generator's output. Although the complete UV map data is not available, we might as well collect a bunch of partial UV maps using the traditional method, i.e., for UV maps generated from frontal face images, the central region, denoted as U V ctr , is accurate, and for UV maps generated from profile face images, the visible half side, U V sd is precise. Note that the partial UV maps collected in this way are not paired with U V pred , so they are only used for adversarial loss, thus indirectly force the U V pred lying in the real domain.

We design two partial UV map discriminators, one for the half side, the other for the center region. Together with the masked face discriminator, the system has three discriminators in total, as shown in Adversarial loss Given an output of the UV sampler, U V spl , the generator will predict a global UV map, U V pred . Three UV patches can be cropped from U V pred , namely U V ctr , U V lef t , U V right . Due to UV map's symmetry, the latter two can be put together and denoted as U V sd . With the U V pred and the 3D shape/pose parameters predicted by the model of [3], a reconstructed face image I R 0 could be rendered. By changing the pose parameter, we can get a face image in a different view angle, denoted as I R 1 . So far, we have three types of fake data: U V sd , U V ctr , and I R 0,1 , each of which corresponds to real data represented as U V sd , U V ctr , and I m , where I m is the input face image with occlusions/background masked.

The adversarial loss is thus formulated as:

L adv = E x [logD(x)] + E x[log(1 -D(x))] (4.7)
where

(x, x, D) ∈ {(U V ctr , U V ctr , D ctr ), (U V sd , U V sd , D sd ) ({I m , I R 0,1 } m gt , D f ace )}
Reconstruction loss The reconstruction loss consists of two terms, the UV reconstruction loss and the face reconstruction loss.

L rec = U V pred -U V bl 1 + I R 0 m gt -I m 1 (4.8)
Symmetry loss Since the UV map of the face is left-right symmetrical, we design the symmetry loss to help the model learn this property.

L sym = U V pred -F lipLR(U V pred ) 1 (4.9)
Identity loss Since the pose is arbitrary, the ground truth of I R 1 is not available. Thus we use the pre-trained FaceNet [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF] to extract the identity feature of I R 1 and I m , and minimize their L 1 distance.

L id = F(I R 1 ) -F(I m ) 1 (4.10)
TV loss TV loss of Equation 4.5 is also applied to U V pred . The total loss function is as follows:

L = L rec + λ 1 L adv + λ 2 L sym + λ 3 L id + λ 4 T V (4.11)

Experiments

The proposed method can faithfully convert the input face image to its corresponding UV map. To demonstrate the conversion ability, we qualitatively compare the 3D reconstruction results with the current state-of-the-art methods, both 2D-based and 3D-based. A quantitative evaluation is also presented.

Implementation details

Our training is based on two datasets:, CelebA-HQ [START_REF] Karras | Progressive growing of gans for improved quality, stability, and variation[END_REF], and FFHQ [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF]. Face images are pre-aligned with landmarks detected by [START_REF] Bulat | How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)[END_REF]. The input image size is 256 × 256, and the predicted UV map is the same size as the input. As for the ground-truth face mask, we first train a stand-alone face segmentation model, using the attribute mask of the CelebA-HQ and our manually labeled occlusions (eyeglasses, hands, etc.). Then we use this model to detect the face masks of the training data, and use them as the ground truth. We set the learning rate to 1e -4 and use Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] optimizer with betas of [0.5, 0.999], the batch size is set to 6. We first pre-train the

Qualitative results

We use the predicted UV maps to render 3D shapes. By changing the pose parameters, images of different view angles are generated. For the qualitative evaluation, as a usual convention, we take the same inputs as others and paste the generated results after them. Figure 4.5 compares our frontalization results with 2D-based face pose editing methods, including TP-GAN [START_REF] Huang | Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis[END_REF], CAPG-GAN [START_REF] Hu | Pose-guided photorealistic face rotation[END_REF], HF-PIM [START_REF] Cao | Learning a high fidelity pose invariant model for high-resolution face frontalization[END_REF],

FNM [START_REF] Qian | Unsupervised face normalization with extreme pose and expression in the wild[END_REF] and Zhouel al. [5]. As shown in Figure 4.5, TP-GAN doesn't convert the pose well, and the third face image it generates is obviously left-skewed. Furthermore, the images generated by TP-GAN, CAPG-GAN, and FNM have large color [4] and [5] Data augmentation As in many previous works, we use our proposed method to synthesize face images for face data augmentation and evaluate the performance of the model trained on the augmented dataset to demonstrate the merits of our approach. The experiment is based on the CASIA [START_REF] Yi | Learning face representation from scratch[END_REF] dataset. Due to the low resolution of the images in CASIA, we retrained a model with an input/output resolution of 128 × 128. Moreover, we remove the segmentation head in the UV sampler to increase the diversity of the augmented images. For each image with less than 30 • yaw angle, we randomly increase its yaw angle from 15 • to 60 • and get a synthesized images. Our basic training settings are the same as [5], with

ResNet18 [START_REF] He | Deep residual learning for image recognition[END_REF] for the backbone and ArcFace [START_REF] Deng | Arcface: Additive angular margin loss for deep face recognition[END_REF] for the loss function. Results are shown in Table 4.1. Since UV-GAN [4] uses ResNet27 as its backbone, which is deeper than ours and [5], it is not surprising that it achieves the highest accuracy.

Although we take the same settings as We conduct the experiments on the AFLW2000-3D [START_REF] Zhu | Face alignment across large poses: A 3d solution[END_REF], which contains 2000 face images with ground truth shape parameters.

For the reconstruction ability evaluation, we calculate the L1 loss and the structural similarity [START_REF] Wang | Multiscale structural similarity for image quality assessment[END_REF] of the reconstructed face images. As can be seen from Table 4.2, our method outperforms others in both these metrics.

As for the identity-preserving ability, the evaluation is conducted by features extracted by the pre-trained LightCNN-29 v2 [START_REF] Wu | A light cnn for deep face representation with noisy labels[END_REF] model. We calculate the cosine similarity of the features corresponding to the input images and the reconstructed/frontalized images. Results are shown in the two rightmost columns of

Ablation Study

UV sampler Due to the inaccurate 3D shape, textures in the edge areas of the incomplete UV maps are not reliable, especially when the face has a large yaw an-

Application

An obvious application of our method is the profiling and frontalization of face images. Chapter 5

Facial Image De-occlusion GAN

Introduction

Thanks to deep Convolutional Neural Networks (CNNs), massive training data, and the widespread use of cameras, face-related techniques have successfully found many applications in our daily life, covering the fields of entertainment, media, art, security, etc. However, face images in the wild may be occluded by various objects, which leads to loss of information and undesirable noise, further resulting in degraded performance of algorithms for face analysis. To address this issue, researchers typically combine methods such as augmenting training data with synthesized occlusions [START_REF] Lv | Data augmentation for face recognition[END_REF][START_REF] Trigueros | Enhancing convolutional neural networks for face recognition with occlusion maps and batch triplet loss[END_REF], designing more sophisticated metrics and network structures [START_REF] Xia | Face occlusion detection based on multitask convolution neural network[END_REF][START_REF] Wan | Occlusion robust face recognition based on mask learning[END_REF][START_REF] Song | Occlusion robust face recognition based on mask learning with pairwise differential siamese network[END_REF][START_REF] Ben Fredj | Face recognition in unconstrained environment with cnn[END_REF], exploiting elaborated training strategies [START_REF] He | Multiscale representation for partial face recognition under near infrared illumination[END_REF][START_REF] Wu | Robust facial landmark detection under significant head poses and occlusion[END_REF][START_REF] He | Dynamic feature learning for partial face recognition[END_REF].

Despite their effectiveness, most of them are task-specific and cannot be migrated to other face-related tasks. A relatively more general solution is to de-occlude the face image before passing it to the downstream tasks.

Before the widespread use of deep learning-based methods, the dominant approaches were based on matching and copying visible patches to the missing parts.

A representative one is PatchMatch [START_REF] Barnes | Patchmatch: A randomized correspondence algorithm for structural image editing[END_REF], which recursively searches the nearest neighbor textures to fill in the holes. Such copy-and-past-based methods work well in recovering simple low-frequency textures and obtain smooth results; however, they cannot recover high-level textures with complex structures, e.g., eyes, nose, mouth. [START_REF] Turk | Eigenfaces for recognition[END_REF][START_REF] Wu | Automatic eyeglasses removal from face images[END_REF][START_REF] Park | Glasses removal from facial image using recursive error compensation[END_REF][START_REF] Wang | Reconstruction of partially occluded face by fast recursive pca[END_REF] leveraged Principal Component Analysis (PCA) to recover high-level semantic face textures. Nevertheless, such methods are based on the assumption that face images are in a linear space. Even for images acquired under constrained conditions, the results are blurry and far from satisfactory. As pointed out by [START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF], the deep autoencoder is a non-linear generalization of the PCA. Therefore, it is reasonable to replace the PCA part of the above methods with an autoencoder. SSDA [START_REF] Xie | Image denoising and inpainting with deep neural networks[END_REF] proposed a deep sparse autoencoder to remove Gaussian noises and superimposed text from images, pioneered the use of deep neural networks in this area. Influenced by the great success of Generative Adversarial Networks (GAN) [START_REF] Goodfellow | Generative adversarial nets[END_REF] in image generation, Context Encoder [START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF] introduced adversarial training to image inpainting for the first time. Its impressive performance led to the prosperity of GAN-based image inpainting methods. Since then, the combination of autoencoder and discriminator has been adopted as the basic model structure for image inpainting tasks [START_REF] Li | Generative face completion[END_REF][START_REF] Yu | Generative image inpainting with contextual attention[END_REF][START_REF] Liu | Image inpainting for irregular holes using partial convolutions[END_REF][START_REF] Yu | Free-form image inpainting with gated convolution[END_REF]. Despite the remarkable progress so far achieved, current methods mainly focus on filling the holes with visually plausible textures and neglect the detection of imperfectness. When applied to the face de-occlusion task, images must be accompanied by manually labeled masks. One strategy to address this problem is to train the model to detect the occlusions au-tomatically. Due to the lack of well-labeled paired training data, existing methods mainly train on the synthetic occluded face images [START_REF] Zhao | Robust lstm-autoencoders for face de-occlusion in the wild[END_REF][START_REF] Cai | Semi-supervised natural face de-occlusion[END_REF][START_REF] Dong | Occlusion-aware gan for face deocclusion in the wild[END_REF][START_REF] Yuan | Face de-occlusion using 3d morphable model and generative adversarial network[END_REF][START_REF] Lee | Byeglassesgan: Identity preserving eyeglasses removal for face images[END_REF][START_REF] Hu | Unsupervised eyeglasses removal in the wild[END_REF]. Unfortunately, these methods either generate low-resolution results or only tackle specific occlusion types. We attribute this to the limited variety of synthetic occlusions.

Real occlusions have different shapes, textures, and blurred boundaries, making it infeasible for synthesis-based methods to cover all possibilities of occlusion. In addition, collecting completely occlusion-free face images as ground truth is not easy because the forehead part of the face is often occluded by bangs, which is perhaps why none of the existing methods consider hair as a source of occlusion.

Given the limitations of existing methods, this paper explores how to enable neural networks to remove arbitrary kinds of face occlusions. Our main contributions are summarized as follows:

• We propose a novel face de-occlusion framework robust to all types of occlusions, regardless of their shapes and textures.

• We build a large occlusion dataset with extensive manually labeled occlusions from real face images.

• The proposed method outperforms the state-of-the-art baselines quantitatively and qualitatively.

Related Work

Mask-Dependent Face Inpainting

Image inpainting aims to recover the missing textures of the input image. Recently, the autoencoder-GAN-based models have achieved impressive results. Benefiting from such structure, Context Encoder [START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF] successfully predicts the 64 × 64 missing part of 128 × 128 images. To make the model further concentrate on the missing parts, [START_REF] Yu | Generative image inpainting with contextual attention[END_REF][START_REF] Li | Generative face completion[END_REF] leverage two discriminators: a global discriminator, which takes the whole image as input, and a local discriminator, which only takes the small region around the missing part. However, such a design is only suitable for a single rectangular(or even square) hole and cannot be applied to images with irregular and arbitrarily distributed holes. To properly handle free-form masks, [START_REF] Liu | Image inpainting for irregular holes using partial convolutions[END_REF] proposed a Partial Convolutional Layer, comprising a masked and re-normalized convolution operation followed by a mask-update step. [START_REF] Yu | Free-form image inpainting with gated convolution[END_REF] proposed a Gated Convolution module, which learns a soft mask for the features in different layers. However, due to the ineffectiveness of CNNs in modeling long-range correlations between distant textures and the hole regions, the inpainting results often have boundary artifacts and look unreal. [START_REF] Yu | Generative image inpainting with contextual attention[END_REF] proposed a Contextual Attention module, which utilizes the image features as a convolution kernel, forcing the distant textures to interact with each other; this could be simply achieved by attention. [START_REF] Zhang | Self-attention generative adversarial networks[END_REF] proposed a Self Attention module, which allows long-range dependency modeling of features;

since then the attention mechanism has been widely applied in image generation tasks [START_REF] Cao | Gcnet: Non-local networks meet squeeze-excitation networks and beyond[END_REF][START_REF] Mejjati | Unsupervised attention-guided image to image translation[END_REF][START_REF] Lei | Face sketch-to-photo transformation with multiscale self-attention gan[END_REF][START_REF] Zheng | Pluralistic image completion[END_REF]. To make the generated image faithfully recover the facial topology, [START_REF] Yang | Generative landmark guided face inpainting[END_REF] leverages facial landmarks as a shape prior. [START_REF] Zheng | Pluralistic image completion[END_REF] proposed a two-path probabilistic framework to generate pluralistic inpainting results. The common problem of the above methods is that they cannot detect the occluded parts automatically and rely on manually specified masks, limiting their usage scenarios.

Mask Free Face De-occlusion

Some methods are devoted to getting rid of masks' dependence and making the face de-occlusion process fully automated. They are typically trained on face pairs with synthesized occlusions. [START_REF] Zhao | Robust lstm-autoencoders for face de-occlusion in the wild[END_REF] proposed an LSTM-Autoencoder to gradually substitute the occlusions with facial textures. Benefiting from an elaborated forward pass, [START_REF] Cai | Semi-supervised natural face de-occlusion[END_REF] proposed a semi-supervised face de-occlusion method, which is trained to predict the occlusion mask without the supervision of ground-truth. [START_REF] Dong | Occlusion-aware gan for face deocclusion in the wild[END_REF] proposed a two-stage GAN, where the first GAN aims to reconstruct the occlusion part solely, and the second GAN takes the result of the first GAN as a hint making the input image occlusion-free. Due to the limited variety of occlusions in the training set, the above methods can only generate images in 128 × 128 resolution with noticeable artifacts, restricting their application to face recognition. Given the complexity of occlusion types, a relatively simple task would be to deal with only specific occlusion: [START_REF] Lee | Byeglassesgan: Identity preserving eyeglasses removal for face images[END_REF][START_REF] Hu | Unsupervised eyeglasses removal in the wild[END_REF] for eyeglasses and [START_REF] Din | A novel gan-based network for unmasking of masked face[END_REF] for masks. Besides being limited by the occlusion variety, as they lack face topology information, the results are often far from satisfactory, especially on large-angled face images.

3D-Guided Face De-occlusion

The work closest to ours is FD3D [START_REF] Yuan | Face de-occlusion using 3d morphable model and generative adversarial network[END_REF], where we both use 3D Morphable Model [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF] (3DMM)-based reconstruction results as guidance. FD3D feeds the synthetic occluded face and the 3D reconstructed one to an adversarial auto-encoder and expects the output to be occlusion-free. Our method surpasses theirs in the following aspects: 1) They inpaint directly on the occluded face despite the challenge of converting various occlusion textures to face textures; in contrast, our method explicitly removes occlusions with predicted masks before inpainting, bypassing the disturbance of the occlusion textures. 2) FD3D entirely relies on supervised learning, which requires occlusion-free images to serve as the ground truth, while our method can be trained on initially occluded images. 3) They synthesize images with fixed occlusions, namely, cup, glasses, hand, mask, and scarf, which have 99 variations in total; by detecting the face mask instead of innumerable types of occlusions, our method can handle all types of occlusion.

Proposed Method

Overview

.2 illustrates the overview of our method. Given that covering all possible occlusions is impossible, we consider the opposite direction. We employ a face segmentation module N S to predict the face mask M f , which is much easier than predicting the mask of all kinds of occlusions. Then, we use the 3D reconstruction module N R to predict the prior 3DMM-based texture and the corresponding mask for the whole face, denoted as I m and M m , respectively. The mask of the occlusions M o can then be calculated as: The overall framework contains three modules, N S , N R , and N G , for face segmentation, 3D reconstruction, and image inpainting. We present the implementation details of each module separately in the following.

M o = M m -M m M f , (5.1)

Face Segmentation

Extracting face regions from occluded face images is a simple, valuable, but never seriously addressed problem. As a result, the recent 3D face reconstruction method [3] requires this functionality still needs to resort to a Naive Bayesian classifier. We present our solution to the problem in this section.

Data Preparation. We build our training data based on CelebAMask-HQ [START_REF] Lee | Maskgan: Towards diverse and interactive facial image manipulation[END_REF], which has 30,000 high-resolution face images, each with a segmentation we can easily get the face region by gathering the labels of eyes, nose, lips, mouth, and skin. Unfortunately, the segmentation mask in the dataset does not consider the occlusions, e.g., hands, accessories, microphones, which are all misclassified as skin. In addition, the segmentation masks in the dataset do not distinguish between sunglasses and spectacles; however, in our task, we should not discard the skin and eyes under the transparent lenses of the spectacles. To solve the above problems, we manually picked out the occluded face images with incorrect segmentation masks and labeled more than 1700 occlusions from them. We also collected about 300 occlusions from Google to get a more balanced occlusion type (e.g., masks, scarfs).

In addition, we collected more than 800 texture patches covering every type we can think of, which will substitute for the original occlusion textures during training to produce more occlusion variations. Those occlusions are utilized for data augmentation, as illustrated in Figure 5.3 (a) and the following equation.

M gt = M f (1 -M o ), I = I f (1 -M o ) + I o M o .
(5.2) landmark detector as required in other methods. The loss is calculated as :

L ldmk = 1 n pt npt i=1 ω n qi -q i 2 , (5.8)
where n pt denotes the number of landmarks, qi denotes the predicted 3D coordinates of the i-th facial landmark. The weights ω n are set to 20 for the nose and inner mouth points and 1 for others.

As c gt provides sufficient regularity, we discard the complex regularization terms in [3]. The overall loss is the weighted sum of the above losses:

L = L coef + λ pix L pix + λ id L id + λ ldmk L ldmk , (5.9)
where λ pix = 1.92, λ id = 0.2, λ ldmk = 1.6e -3 .

Face Inpainting

With the N S and N R described above, we get the following information of an oc- The generator N G is constructed from stacked gated residual blocks [START_REF] Yu | Free-form image inpainting with gated convolution[END_REF] and follows the classical encoder-decoder structure; a self-attention [START_REF] Zhang | Self-attention generative adversarial networks[END_REF] module is applied to the bottleneck features. As shown in Figure 5.6, the input is composed of three parts: the occlusion mask M o , the face image with noised occlusion I n , and the reconstructed face I m . The output Î is the de-occluded image. A VGG [START_REF] Simonyan | Very deep convolutional networks for largescale image recognition[END_REF] shaped discriminator is further employed to increase the photorealism to Î. We randomly superimpose occlusions to the training data and train N G to recover the missing textures. One issue to note is that the input image itself always contains occlusions, such as bangs, and we do not have ground truth for these parts.

So we use Poisson Blending [START_REF] Pérez | Poisson image editing[END_REF] to seamlessly merge I f and I m and get I p , which provides weak supervision for the inpainting task. The training is guided by the following losses:

Pixel-wise face loss.

L pix = 1 M Î M -I f M 1 , (5.10)
where M = M m M f , limiting the loss to be calculated on the face region solely.

This loss serves for recovering textures occluded by the synthetic occlusion and cannot recover the initially occluded textures.

SSIM loss. We use I p to guide the generation of initially occluded textures.

Although the Poisson blending result is visually pleasing, it changes the color of I f , so we cannot simply apply L 1 or L 2 loss. Instead, we leverage the Structural Similarity loss (SSIM) [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], emphasizing the structural level discrepancy:

L sm = -1 Mm SSIM ( Î Mm , I p Mm ), (5.11)
where SSIM stands for the similarity mapping function; Mm is the eroded M m , eliminating the edge effects in the similarity map. To further mitigate the impact of inaccurate color of I p and make the loss focus on the missing textures, we apply the OHEM again as in Equation 5.4.

Background loss. Areas other than the face should remain unchanged:

L bg = 1 M bg Î M bg -I M bg , (5.12)
where M bg denotes the background mask, calculated by 1 -M m . We also erode M b g to alleviate the edge effects.

Identity loss. The generated image should have the same identity as the original image, so we also leverage Equation 5.7 to add feature level identity constraints.

TV loss. To penalize the noises in Î, we adopt the total variation loss [START_REF] Mahendran | Understanding deep image representations by inverting them[END_REF].

L tv = 1 W HC ∇ x Î 2 + ∇ y Î 2 , (5.13)
where W , H, C are the width, height, and the number of channels, respectively, ∇_ calculates the image gradient along a direction.

Adversarial loss. To further make Î photorealistic, we employ the adversarial loss:

L adv = -E Î [log(D( Î))], (5.14)
where D(•) denotes the output of the discriminator, which is the term to be maximized by the generator.

The global objective function of the generator can be summarized as follows:

L =λ pix L pix + λ sm L sm + λ bg L bg + λ id L id + λ tv L tv + λ adv L adv , (5.15)
where the weights are empirically set to λ pix = 10, λ sm = 5, λ bg = 5, λ id = 0.2, λ tv = 0.1, λ adv = 0.01.

Discriminator loss. We use the BCE loss to train the discriminator, aiming to distinguish Î from real images:

L adv = E I [log(D(I))] + E Î [log(1 -D( Î))], (5.16)
where I is the real image with less occlusion, selected from CelebA-HQ and FFHQ according to the overlap rate of its corresponding M m and M f .

Experiments

The proposed method can effectively remove the occlusions from face images. To demonstrate the power of our model, we qualitatively compare our results with state-of-the-art methods, including those for face de-occlusion and for image inpainting. Quantitative experiments are conducted in two aspects as well, namely reconstruction ability and identity recovery ability.

Implementation Details

We follow an incremental training strategy: 1) We train the segmentation module N S . 2) we train the reconstruction module N R based on N S . 3) We train the inpainting module N G based on the result of the first two. The training is performed on two public datasets, CelebAMask-HQ [START_REF] Lee | Maskgan: Towards diverse and interactive facial image manipulation[END_REF] and FFHQ [START_REF] Karras | Progressive growing of gans for improved quality, stability, and variation[END_REF], and our manually labeled/collected occlusions and textures. All the images are aligned with facial landmarks detected by [START_REF] Bulat | How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)[END_REF] and cropped to 256 × 256.

The training of N S consists of two rounds: In the first round, we train on 300,000 images of CelebAMask-HQ with our manually labeled occlusions to obtain a coarse model. In the second round, we apply the coarse model to both CelebAMask-HQ and FFHQ [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF], select 500 hard examples from the results, relabel and add them to the occlusion dataset to retrain a more accurate model. Each round is trained for 30 epochs with a batch size of 16 and a learning rate of 1e -4 . The training takes about two hours on two Nvidia GTX 1080 GPUs. The 3D reconstruction module and the face inpainting modules are trained on 100,000 images of CelebAMask-HQ and FFHQ, with an initial learning rate of 1e -4 . For the reconstruction module N R , we train 50 epochs with a batch size of 16, and for the inpainting module N G , we train 80 epochs with a batch size of 4. The learning rates of both modules are dropped halfway through the training with ratios of 0.2 and 0.1, respectively. All three modules are optimized using Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with a weight decay of 0. Betas are One issue to note is that when FD3D compares with other methods in their paper, the images are not pre-aligned according to the rules of the corresponding method, resulting in questionable inferior results.

As shown in Figure 5.7, our method generally outperforms FD3D and other image inpainting methods, especially on face images with large yaw angles. We argue that the image inpainting methods are purely based on statistical learning, thus highly dependent on the training data distribution. When a few face images with large yaw angles occur, these methods record dramatic performance degradation.

Although FD3D does not require a manual mask for the de-occlusion, they apparently can only handle limited types of occlusions: the bangs in the third row and the hat in the fourth row are not identified as occlusions. In addition, FD3D produces low-quality face images with blurred boundaries and unreal textures, as evidenced in the images from the second to the last row. We believe this is due to the following reasons: 1) they do not explicitly detect the occlusions and mask them out, causing the generator to only cope with the occlusions it has ever seen while failing to deocclude arbitrary occlusions; 2) they use a very coarse 3D reconstruction method, which cannot provide correct and effective face prior to the generator.

Quantitative Results

This section evaluates the proposed method in two aspects: the reconstruction ability and the identity recovery ability. We synthesize 1000 sunglasses-occluded face images and use DeepFill-v2, PICNet, LaFIn, and our method to recover the initial images. Since the image inpainting methods only synthesize the missing region, the reconstruction ability is evaluated only in that region, with the following metrics: L1 loss, SSIM [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF] score, and PSNR score. The identity recovery ability is evaluated by the cosine similarity of the features extracted by the pre-trained ArcFace [START_REF] Deng | Arcface: Additive angular margin loss for deep face recognition[END_REF].

Results are reported in Table 5.1.

As can be seen, our method outperforms the state-of-the-art image inpainting methods across all listed metrics. It performs particularly well in recovering the identity, much exceeding its comparators. We attribute this mainly to the occlusionrobust face reconstruction module.

Ablation Study

The three modules in our framework are interdependent, making it impossible to remove one for ablation study. Instead, we mainly focus on the SSIM loss of the inpainting module, as the reconstruction and identity losses are conventional prac- As Table 4.2 shows, without the SSIM loss, the model degrades in all aspects of metrics (nevertheless, benefiting from the 3D prior, it still outperforms others).

Moreover, the initially occluded textures are now only supervised by the adversarial loss; thus, the results are not guaranteed to be occlusion-free. As shown in Fig-

Discussion

Limitation.The proposed method focuses on removing the occlusions within the 3D reconstructed face mask. For occlusions spanning the face and the background, it can only rigidly remove the parts above the face without producing a smooth transition, creating an unrealistic scene where the face seems to hover above the occlusions, as Figure 5.9 shows. We have tried to create a gap between the face area and the background through mask erosion, and the gap is solely supervised by the adversarial loss during training. This trick alleviates the boundary effect to some extent but still cannot cope with large-sized occlusions.

Social impacts. Our method helps pre-process face images to avoid the negative impact of occlusions such as hair, glasses, hands, etc., on downstream tasks (e.g., fine-grained 3D face reconstruction, face recognition). Despite the benefits it brings, it also risks violating human privacy. To quantify this risk, we further conduct experiments to analyze the identity recovery ability across different types of occlusions. The above observation proves that the method is controllable for privacy violations.

People's identities can be safely protected when they simultaneously wear sunglasses and masks.

Additional results

Conclusion

This work proposes a segmentation-3D reconstruction-guided facial image deocclusion method that automatically removes all types of occlusions. We first analyze the limitations of current face de-occlusion methods: they either require a manually labeled mask or can only handle a limited number of occlusion types, which mainly stems from the vast diversity of possible occlusions. Our key innovation is bypassing the segmentation of infinite occlusions and instead segmenting the face regions, which is much easier. We manually labeled a large face occlusion Conclusions and Future Work

Conclusion

We focused on GAN-based face image generation and its application to face recognition. Specifically, we proposed several novel methods for 1) face pose editing of 2D facial images, 2) texture generation for 3D face reconstruction, 3) facial image de-occlusion. The methods employed 2D and 3D-based techniques in computer vision and computer graphics, including image generation, segmentation, and 3D face reconstruction.

Chapter 3 first analyzed the problem of the existing encoder-decoder-based facepose editing methods: they primarily focus on modeling the identity preservation ability during pose synthesis but are less able to preserve the image style properly, which refers to the color, brightness, saturation, etc., resulting in a style discrepancy between the output image and the input image. Then, it proposed a two-stage framework that converts the problem into pixel-sampling and image inpainting. In the first stage, the pixel attention sampling module directly samples pixels from the input image, thus explicitly preserving more style information. In the second stage, the missing textures are further completed, and the noises are removed. Thanks to the pixel sampling module and the high-dimensional embedding in the inpainting module, the image style is well preserved during face pose editing.

Although pure 2D-based methods could generate face images of different view angles, the generation consistency is poor due to the lack of global consistency constraints and prior knowledge of the 3D shape. Chapter 4 solved the problem by generating high-fidelity global facial textures for 3D face reconstruction. The proposed method also benefited from a pixel-sampling module similar to Chapter3 and exploited a two-stage framework. In the first stage, the sampling module directly predicts an incomplete global face texture (UV map) from the input image without relying on 3D shapes as traditional methods. In the second stage, the incomplete UV map is completed by a texture generator supervised by the pseudo UV map and multiple partial UV discriminators, bypassing the need for scarce UV datasets and the costly optimization approach. The generated textures are of high-fidelity and could be used for data augmentation in face recognition tasks to improve the robustness of the model to view angles.

In addition to the view angle, another factor that harms face recognition accuracy is the occlusion. Chapter 5 tackled this problem by proposing a facial image de-occlusion framework. We trained a practical face mask prediction module and an occlusion-robust face reconstruction module, with extensive manually labeled occlusions from real images. Any occlusions could be masked out by collaboratively exploiting the two modules, despite their shapes and textures. The inpainting module takes the occlusion masks and the 3D reconstruction results as prior and generates plausible non-occluded face images. The proposed method far exceeds the state-of-the-arts qualitatively and quantitatively.

Our work advanced incrementally. In the beginning, we only worked with pure 2D images, without incorporating 3D shape information. Chapter 4 focused only on the texture for 3D face reconstruction; the shape part still relied on an off-the-shelf model. While chapter 5 trained all texture and shape modules from scratch. Our work has spanned from 2D to 3D, from low resolution to high resolution, from a single model to a combination of multiple models, from following the state-of-the-art methods to significantly exceeding them.

Future Work

The 3D reconstruction method used in this work is based on the 3D Morphable Model [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF], a low-dimensional linear model that cannot capture high-frequency details of shapes and textures. Chapter 4 solved the problem of unreal textures by training a texture generator; however, generating fine-grained 3D face shapes still

6 3 2. 1 29 2. 9 35 3. 1 48 3. 4 49 3. 5 65 xii 4 . 3

 312993514844956543 Conclusions and Future Work 6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliography ix List of Figures 1.1 Overview of 3D pose normalization of [1] . . . . . . . . . . . . . . . Difference between the discriminative models and the generative model 12 2.2 Difference between the AE (a) and the VAE (b) . . . . . . . . . . . . 13 2.3 2D encoding of MNIST by VE (a) and VAE (b) . . . . . . . . . . . . 18 2.4 Structure of GAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.5 Example of two probability distributions that do not overlap. . . . . 22 2.6 Illustration of the EMD. The different color blocks on the left are transported to the position on the right of the same color. . . . . . . 24 2.7 Two 2D distributions that do not overlap. . . . . . . . . . . . . . . . 25 2.8 Comparison of weight clipping and gradient penalty. . . . . . . . . . Illustration of the Self-Attention layer. ⊗ denotes matrix multiplication. Illustration of the trade-off between identity preserving and style preserving. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.2 (a) Example of front/profile optical illusion. Indicating that face images in different view angles still share pixel-level similarities. (b) The ambiguity of representing 3D face pose by 2D landmarks. The two faces above have almost the same landmark distribution, but are in different poses.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3 (a) and (e) are the source image and the target one, respectively, where the corresponding landmarks and the bounding boxes of their bigger side are shown to illustrate our aligning strategy. (b) and (f) are the 3D facial landmarks detected by [2]. (c) shows the aligned image I tf . (g) is the alpha blend of I tf and J, illustrating that the target image shares pixel-level similarities with the source image. (d) and (h) are the segmentation maps of I tf and J transformed from their 2D facial landmarks. . . . . . . . . . . . . . . . . . . . . . . . . Structure of the proposed Pixel Attention Sampling (PAS) module. The result of PAS. The first row shows the guiding face images at target poses, the first image in the second row is the input face image, and the remaining images are synthesized faces based on the landmarks of the guiding face images. We can see that the pixels are sampled and adjusted to the target pose. The noises and holes will be removed or filled at the next image inpainting stage. . . . . . . . 52 3.6 The final result of our approach. The first row shows the guidance images. The input images are in the first column, and the simple reconstructed images are in the second column. The rest images are the pose editing results based on the landmarks of the exemplars. . 55 3.7 (a) From top to bottom shows the input images, results of CR-GAN, DR-GAN, FNM and our method. (b) From left to right are the input images and the generated images with both yaw angles and pitch angles changed. (c) From left to right are the input images, half-face aligned images, frontalized images w/o PAS module, results of the proposed algorithm, and the ground-truth images. . . . . . . . . . . 56 4.1 Results of the proposed method. The left column shows the input images. Images on the right are synthesized using the predicted UVmap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.2 The traditional method for incomplete UV map generation. Which is used for generating the target output of the UV sampler. . . . . . Overview of our approach.

)

  The predicted UV map is used to render face images of different poses: I R 0 and I R 1 , which are further fed into a face discriminator. (5) U V pred is cropped to the side part U V sd and center part U V ctr , fed into their corresponding discriminators. . . . . . . . . . . 4.4 (a) The input image. (b) U V gt . (c) The face reconstructed from U V bf m . (d) U V bf m . (e) The face reconstructed from U V bl . (f ) U V bl . (g) The face in (e) under different view angle. (h) The texture marked in blue is used to fill the missing texture in its symmetric area (marked in red). . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Frontalization results comparing with 2D-based face pose editing methods. Zoom-in for a better view. . . . . . . . . . . . . . . . . . . 4.6 Qualitative comparison with other state-of-the art 3D reconstruction methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Ablation study of the UV sampler. The UV sampler not only makes the first stage independent of the 3D shape, but also helps to generate more accurate textures in the second stage. . . . . . . . . . . . . . . 4.8 Ablation study of partial discriminators. . . . . . . . . . . . . . . . 4.9 Ablation study of the segmentation head. . . . . . . . . . . . . . . . 4.10 Face profiling results . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11 Face frontalization results . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Results of the proposed method. The first row and the third row are input images, the second row and the last row shows the de-occlusion results. Zoom in for a better view. . . . . . . . . . . . . . . . . . . . xiii 5.2 Overview of the proposed method, which does not include the discriminator part for space limitation . . . . . . . . . . . . . . . . . . . 86 5.3 (a) Data augmentation method. I f : face image, M f : face mask of I f , I o : occlusion, M o : occlusion mask, I: occluded image, M gt : face mask of I. (b) Structure of the segmentation model N S , M : predicted face mask. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.4 Overview of the 3D reconstruction module. I: original image, I o : occluded image, ĉ: predicted 3D parameters, Î: reconstructed face, M m : mask of Î, M f : face mask of I, I f : I M f . . . . . . . . . . . 89 5.5 The first row shows the input images; the second row and the last row are reconstructed faces by [3] and ours, respectively. . . . . . . . 90 5.6 Overview of the inpainting module. . . . . . . . . . . . . . . . . . . 92 5.7 De-occlusion results compared with state-of-the-art methods. Input images are aligned and cropped according to the rules of each method, and we re-paste the outputs back to facilitate comparison. . . . . . . 95 5.8 Ablation study of the SSIM loss. . . . . . . . . . . . . . . . . . . . . 97 5.9 Limitation of the proposed method. . . . . . . . . . . . . . . . . . . . 98 5.10 Identity recovery ability for different types of occlusion. . . . . . . . 99 5.11 De-occlusion results. . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 xiv Chapter 1

Figure 2 . 1 :

 21 Figure 2.1: Difference between the discriminative models and the generative model

Figure 2 . 2 :

 22 Figure 2.2: Difference between the AE (a) and the VAE (b)

Figure 2 . 3 :Figure 2 .

 232 Figure 2.3: 2D encoding of MNIST by VE (a) and VAE (b)

Figure 2 . 4 :

 24 Figure 2.4: Structure of GAN

Algorithm 2

 2 .25 is much bigger than that of Equation 2.24. The training of GAN follows Algorithm 2: Training algorithm of GAN Input: training dataset {x i }; G and D; D update step k per G update Output: Parameters of G and D: θ G and θ D 1: for number of training iterations do 2:

Figure 2 . 6 :

 26 Figure 2.6: Illustration of the EMD. The different color blocks on the left are transported to the position on the right of the same color.

Figure 2 . 6 3

 26 illustrates how a pile of earth can be transported into another shape and position, with the transport plan represented by different colors.

Figure 2 .

 2 Figure 2.7: Two 2D distributions that do not overlap.

  g θ ) 12: end while Despite the success of WGAN, weight clipping in not a good way to ensure the Lipschitz continuity of the discriminator. It suffers from two main problems: • The hyper-parameter of the weight clipping should be carefully tuned. Due to the cumulative effect across multiple model layers, a too large or too small clipping parameter will lead to exploding or vanishing gradients problem, respectively. As shown in Figure 2.8 4 on the left. • As shown in the top right of Figure 2.8, weight clipping pushes the weights towards the extremes of the clipping interval, largely restricting the neural network's power.

Figure 2 .

 2 Figure 2.8: Comparison of weight clipping and gradient penalty.

Figure 2 .9 5 .

 25 Figure 2.9 5 .Suppose the input feature to the Self-Attention (SA) layer is as x ∈ R B×C×H×W , where B is the batch size, C is the number of channels, and H × W are the height and width of the feature map. To simplify the notation, we assume B = 1 and omit this term in the equations, i.e. x ∈ R C×H×W .

Figure 2 . 9 :

 29 Figure 2.9: Illustration of the Self-Attention layer. ⊗ denotes matrix multiplication.

  y p(x, y)log p(x, y) p(y)p(x) , y)logp(y|x)y p(y)logp(y) = E x∼pg p(y|x)logp(y|x)y p(y)logp(y)

FID

  only uses Inception V3 as a feature extractor, thus does not rely on an image's predicted class probability distribution. It has the following advantages over IS: • The training set of the generated model can be different from the training set of Inception V3.

Face

  pose editing aims to change the pose of an input face image while keeping its original identity unchanged. It has many potential applications, e.g., face recognition, movie industry and entertainment. The current state-of-the-art has featured two main research lines in this field, i.e., 3D reconstruction-based, and simple 2D based.For 3D reconstruction-based approaches, face pose editing is achieved by either mapping the 2D face images to 3D face models with fixed or regressed parameters[START_REF] Liu | Joint face alignment and 3d face reconstruction with application to face recognition[END_REF][START_REF] Zhao | Dual-agent gans for photorealistic and identity preserving profile face synthesis[END_REF][START_REF] Yin | Towards large-pose face frontalization in the wild[END_REF] or directly regressing the UV map[4,[START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF] of the input face. The advantage of such models is that pose control is not demanding. With the reconstructed 3D face, face images at any target pose can be obtained by 3D geometrical transformation and 2D projection. However, regressing either the parameters of predefined 3D models or the UV map requires large amounts of high-quality training data. Moreover, due to the restriction of the predefined model and the missing texture of extreme poses, fine details of the images are ignored. As a result, the faces generated by these approaches are generally not photo-realistic enough and require further refinements[START_REF] Gecer | Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3d morphable model[END_REF].Thanks to the development of Generative Adversarial Networks (GAN)[START_REF] Goodfellow | Generative adversarial nets[END_REF], a number of GAN based 2D approaches to face pose editing have been proposed in recent years. GAN has achieved great success in face image inpainting and facial attribute editing[START_REF] Cai | Fcsr-gan: End-to-end learning for joint face completion and super-resolution[END_REF][START_REF] Huang | Multimodal unsupervised image-to-image translation[END_REF][START_REF] Pumarola | Ganimation: Anatomically-aware facial animation from a single image[END_REF][START_REF] Portenier | Faceshop: Deep sketch-based face image editing[END_REF][START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF]. However, the existing methods are generally only capable of editing the subtle attributes or local regions of the image, whereas the global structure remains almost unchanged. Regarding face pose manipulation, when changing the view angle from side to front, not only the local texture but also the global shape of the face image dramatically changes. Despite these difficulties, there still exist significant efforts tackling this problem[START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF] 17,[START_REF] Hu | Pose-guided photorealistic face rotation[END_REF][START_REF] Tian | Cr-gan: learning complete representations for multi-view generation[END_REF][START_REF] Qian | Unsupervised face normalization with extreme pose and expression in the wild[END_REF]. Most of the methods are implemented by an encoder-decoder structured network, with a bottleneck layer in the middle, where the faces are first encoded into a low dimensional feature vector, and then decoded into the image space conditioned by the pose information, e.g., CR-GAN[START_REF] Tian | Cr-gan: learning complete representations for multi-view generation[END_REF], DR-GAN[START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF]. However, there exists an intrinsic trade-off between the image style conserving capability and the identity preserving ability in the compact deep feature space, i.e., it is hard to model the expertise of both the face identity and other image properties, such as lightning condition, saturation, background color, etc.To highlight the aforementioned dilemma that commonly incurs in current 2D based methods, we remove the face classification branch of DR-GAN[START_REF] Tran | Disentangled representation learning gan for pose-invariant face recognition[END_REF] (with the latent feature dimensionality of 320) and train the model only with the adversarial loss and the reconstruction loss. In this case, an adversarial auto-encoder (AE) is achieved, where the reconstruction loss aims to efficiently preserve the style of the input image, and the adversarial loss enforces the generated images photo-realistic.

Figure 3 .

 3 Figure 3.1 illustrates the input images (the first row) and the results obtained by the adversarial auto-encoder (second row) and DR-GAN (third row), respectively. As can be seen, the auto-encoder properly preserves the style of the input image, but it fails maintaining the identities. The reconstructed faces by DR-GAN successfully catch the identity characteristics of the input images, whereas the output ones are distorted and present obvious artifacts. If it is even painful for the model to faithfully rebuild the given input face in terms of both style and the identity without any pose manipulation, how can we further expect it to preserve them after changing the pose?

Figure 3 . 1 :

 31 Figure 3.1: Illustration of the trade-off between identity preserving and style preserving.

Figure 3 . 2 :

 32 Figure 3.2: (a) Example of front/profile optical illusion. Indicating that face images in different view angles still share pixel-level similarities. (b) The ambiguity of representing 3D face pose by 2D landmarks. The two faces above have almost the same landmark distribution, but are in different poses..

  The training follows an adversarial paradigm. To overcome the problems of unstable gradient and mode collapse, Wasserstein GAN (WGAN)[56] proposes the earth move distance as metric in the discriminator's loss function. To enforce the Lipschitz constraint of the discriminator, SN-GAN [59] applies spectral normalization to the weight parameters.

Figure 3 . 3 :

 33 Figure 3.3: (a) and (e) are the source image and the target one, respectively, where the corresponding landmarks and the bounding boxes of their bigger side are shown to illustrate our aligning strategy. (b) and (f) are the 3D facial landmarks detected by [2]. (c) shows the aligned image I tf . (g) is the alpha blend of I tf and J, illustrating that the target image shares pixel-level similarities with the source image. (d) and (h) are the segmentation maps of I tf and J transformed from their 2D facial landmarks.

Figure 3 . 4 :

 34 Figure 3.4: Structure of the proposed Pixel Attention Sampling (PAS) module.

Figure 3 . 4 depicts

 34 the corresponding diagram.

Figure 3 . 5 :

 35 Figure 3.5: The result of PAS. The first row shows the guiding face images at target poses, the first image in the second row is the input face image, and the remaining images are synthesized faces based on the landmarks of the guiding face images. We can see that the pixels are sampled and adjusted to the target pose. The noises and holes will be removed or filled at the next image inpainting stage.

Figure 3 . 6 :

 36 Figure 3.6: The final result of our approach. The first row shows the guidance images. The input images are in the first column, and the simple reconstructed images are in the second column. The rest images are the pose editing results based on the landmarks of the exemplars.

Figure 3

 3 Figure 3.7: (a) From top to bottom shows the input images, results of CR-GAN, DR-GAN, FNM and our method. (b) From left to right are the input images and the generated images with both yaw angles and pitch angles changed. (c) From left to right are the input images, half-face aligned images, frontalized images w/o PAS module, results of the proposed algorithm, and the ground-truth images.

ure 3 .

 3 7(a). As can be seen, our results are more visually convincing and the styles are closer to the input images compared to DR-GAN, and the identities are better preserved than CR-GAN. As for FNM, the generated image style is more similar to the training set, where the lighting and expressions are normalized, and the color has been changed, by contrast, our results better preserve those characteristics of the input face images. Moreover, the proposed approach changes the face poses in three degrees of freedom, resulting in more flexible pose editing results than merely controlling the yaw angles as previous methods. Figure3.7(b) shows the results of editing both pitch and yaw angles of input face images (leftmost).

Figure 3 .

 3 7(c) shows the results. As can be seen, the synthesized images without PAS are blurred. More specifically, in the second row and the fourth row, the mouths are not well aligned, and the unexpected edges of the aligned input images are not well removed. The results indicate that it is indeed difficult for the single U-Net based model to change the original patterns of the input image thus results in undesired artifacts.

Reconstruction

4. 1

 1 Introduction3D face reconstruction is an important yet challenging domain in computer vision, aiming to faithfully restore the shape and texture of a face from one or more face images. It has a wide range of applications, such as face recognition, face editing, face animation, and other artistic and entertainment fields. Recently, there has been a surge of interest in single-image based 3D face reconstruction[3,[START_REF] Guo | Towards fast, accurate and stable 3d dense face alignment[END_REF][START_REF] Zhu | Face alignment in full pose range: A 3d total solution[END_REF][START_REF] Richardson | Learning detailed face reconstruction from a single image[END_REF][START_REF] Guo | Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images[END_REF] 3,[START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF][START_REF] Bai | Deep facial non-rigid multiview stereo[END_REF] 28]. While most previous work has been devoted to predicting more accurate and detailed 3D shapes, not much work has focused on generating photorealistic face textures. However, studies[START_REF] Masi | Face-specific data augmentation for unconstrained face recognition[END_REF] 12] have shown that the texture plays a more significant role than that of the shape in face recognition tasks. Thus we can never ignore the importance of the texture in 3D face reconstruction.Existing 3D face texture generation methods can be broadly classified into three categories: texture model-based, image generation-based, and GAN optimizationbased.

Figure 4 . 1 :

 41 Figure 4.1: Results of the proposed method. The left column shows the input images. Images on the right are synthesized using the predicted UV-map.

Figure 4 . 2 :

 42 Figure 4.2: The traditional method for incomplete UV map generation. Which is used for generating the target output of the UV sampler.

Figure 4 . 3 :

 43 Figure 4.3: Overview of our approach. (1) Given an input face image, the UV sampler predicts its face mask m and sampling map S att , based on which samples an incomplete UV map U V spl[START_REF] Bulat | How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)[END_REF] The UV generator will further complete the sampled UV map and output the U V pred . (3) With an off-the-shelf 3DMM regressor, we predict the shape and texture of the input face image, which is used for getting the ground truth of the U V spl : U V gt and the pseudo ground truth of the U V pred : U V bl . (4) The predicted UV map is used to render face images of different poses: I R 0 and I R 1 , which are further fed into a face discriminator. (5) U V pred is cropped to the side part U V sd and center part U V ctr , fed into their corresponding discriminators.

Figure 4 . 4 :

 44 Figure 4.4: (a) The input image. (b) U V gt . (c) The face reconstructed from U V bf m . (d) U V bf m . (e) The face reconstructed from U V bl . (f ) U V bl . (g) The face in (e) under different view angle. (h) The texture marked in blue is used to fill the missing texture in its symmetric area (marked in red).

Figure 4 .

 4 4(h). The final blending result is denoted as U V bl , as in Figure 4.4(f), both Figure 4.4(e) and Figure 4.4(g) are generated from it, which is far more photorealistic than the BFM reconstruction result in Figure 4.4(c).

Figure 4 . 3 .

 43 The training is guided by the following losses.

Figure 4 . 5 :

 45 Figure 4.5: Frontalization results comparing with 2D-based face pose editing methods. Zoom-in for a better view.

Figure 4 . 6 :

 46 Figure 4.6: Qualitative comparison with other state-of-the art 3D reconstruction methods.

Figure 4 . 8 :

 48 Figure 4.8: Ablation study of partial discriminators.

Figure 4 . 9 :

 49 Figure 4.9: Ablation study of the segmentation head.

Figure 4 . 10 :

 410 Figure 4.10: Face profiling results

Figure 4 .

 4 Figure 4.11: Face frontalization results

Figure 5 . 1 :

 51 Figure 5.1: Results of the proposed method. The first row and the third row are input images, the second row and the last row shows the de-occlusion results. Zoom in for a better view.

Figure 5 . 2 :

 52 Figure 5.2: Overview of the proposed method, which does not include the discriminator part for space limitation

ResNet18Figure 5 . 3 :

 53 Figure 5.3: (a) Data augmentation method. I f : face image, M f : face mask of I f , I o : occlusion, M o : occlusion mask, I: occluded image, M gt : face mask of I. (b) Structure of the segmentation model N S , M : predicted face mask.

Figure 5 .

 5 Figure 5.5 compares our results with those of[3], demonstrating the effectiveness of the proposed training strategy in improving the occlusion-robustness of the model. A quantitative comparison is also performed: we use[3] and our model to reconstruct 1000 face images with synthetic occlusions, the L pix of our model is 0.153, which is much less than 0.177 of[3].

  cluded face image: face mask M f , face I f , reconstructed face I m , reconstructed face mask M m . Based on M f and M m , Equation 5.1 further calculates the occlusion mask M o . The goal of this section is to restore the missing textures indicated by M o .

Figure 5 . 6 :

 56 Figure 5.6: Overview of the inpainting module.

Figure 5 . 8 :

 58 Figure 5.8: Ablation study of the SSIM loss.

Figure 5 . 9 :

 59 Figure 5.9: Limitation of the proposed method.

ure 5 .

 5 8, the shadows on the occlusion edges are difficult to eliminate with mere adversarial loss.

Figure 5 . 10 :

 510 Figure 5.10: Identity recovery ability for different types of occlusion.

Figure 5 .

 5 Figure5.10 shows that the proposed method performs well on sunglasses-occluded faces; the performance decreases for the mask-type occlusion; and when combining the mask with sunglasses, the method can no longer recover the face identity.

Figure 5 .

 5 Figure 5.11 are the de-occlusion results of the initially occluded images in the FFHQ[START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF] dataset.

Figure 5 .

 5 Figure 5.11: De-occlusion results.

Table 3 .

 3

	1: FID score of frontalized face images (lower is better)
	CR-GAN DR-GAN FNM ours
	204	122	150	105

Table 3 .

 3 2: Rank-1 recognition rates (%) across views under Setting 1.Method ±90 • ±75 • ±60 • ±45 • ±30 • ±15 •

	HPN [103]	29.82 47.57 61.24 72.77 78.26 84.23
	c-CNN [104]	47.26 60.7	74.4	89	94.1	97.0
	TP-GAN [17]	64.0	84.1	92.9	98.6	99.9	99.8
	PIM [105]	75.0	91.2	97.7	98.3	99.4	99.9
	CAPG-GAN [18] 77.1	87.4	93.7	98.3	99.4	99.9
	FNM [19]	55.8	81.3	93.7	98.2	99.5	99.9
	Light CNN [97]	2.6	10.5	32.7	71.2	95.1	99.8
	Ours	45.5	78.7	90.0	99.6	99.9	100

Table 4 . 1 :

 41 Comparison of the face augmentation ability with UV-GAN

	4.4.3 Quantitative results
	Training Data	Method	ACC(%) AUC(%)
	CASIA(baseline) Zhou et al. 98.77	99.90
	CASIA+rot	Zhou et al. 98.95	99.91
	CASIA(baseline) UV-GAN	99.02	-
	CASIA+augUV UV-GAN	99.22	-
	CASIA(baseline) ours	98.75	99.88
	CASIA+augUV ours	98.98	99.90

Table 4 .

 4 Zhou et al., we achieve a slightly lower baseline due to numerous differences in training details (learning rate, batch size, optimizer, etc.). However, by training on the augmented dataset, our model exceeds their accuracy, and our AUC closes the gap with them, demonstrating the efficiency of our data augmentation ability. 2: Pixel-wise reconstruction and the identity-preserving ability on AFLW2000-3D, non-facial areas of all images are masked out for fair comparison.Face reconstruction We evaluate the proposed method in two aspects: the pixel-wise reconstruction ability and the identity-preserving ability. As most previous works are not open-sourced, we only compare with Deng et al.[3] and Zhou et al.[5], SOTA methods based on 3DMM and 2D face image generation, respectively.

		Reconstruction	Recognition
	Method	L 1	SSIM Recon Front
	Deng et al. 0.064	0.698	0.554	0.501
	Zhou et al. 0.069	0.613	0.780	0.675
	Ours	0.021 0.913 0.862 0.684

Table 4 .

 4 2. An interesting thing to notice is that, although Zhou et al. is inferior to the 3DMM-based model in terms of reconstruction loss, they are more capable of preserving the face identity. However, our proposed method achieves the best performance in both aspects.

Table 5 . 1 :

 51 Comparison of the proposed method with state-of-the-art image inpainting methods.

	Method	L 1 ↓	SSIM ↑ PSNR↑	ID↑
	LaFIn	0.067	0.607	27.944 0.639
	PICNet	0.0654	0.604	28.137 0.615
	DeepFill	0.065	0.604	28.105 0.659
	w/o SSIM 0.065	0.619	28.491 0.665
	Ours	0.062 0.623 28.902 0.690

large éventail d'applications dans les domaines du divertissement et de l'art. Cette thèse fournit une étude approfondie de la synthèse d'images de visages basée sur les GAN et de son application à la RF. Les méthodes actuelles de synthèse d'images faciales présentent deux axes de recherche principaux, à savoir les méthodes basées sur la 2D et celles basées sur la reconstruction 3D. Nos travaux couvrent les deux. Pour l'édition de la pose du visage en 2D, les méthodes actuelles se concentrent principalement sur la modélisation de la capacité de préservation de l'identité mais sont moins capables de préserver correctement le style de l'image, qui fait référence à la couleur, la luminosité, la saturation, etc. Cette thèse propose une nouvelle approche en deux étapes pour résoudre le problème d'incohérence de style, où la manipulation de la pose du visage est divisée en échantillonnage de pixels et en peinture du visage. Avec des pixels échantillonnés directement à partir de l'image d'entrée, le résultat de l'édition du visage conserve fidèlement l'identité et le style de l'image. Pour la reconstruction traditionnelle de visages en 3D, en raison de la nature linéaire et basse dimensionnelle du modèle morphable 3D (3DMM), les textures reconstruites capturent à peine les détails à haute fréquence, ce qui donne des textures floues qui sont loin d'être satisfaisantes. Certaines méthodes récentes de reconstruction de visages en 3D ont également exploité l'entraînement contradictoire pour améliorer la qualité de la texture. Cependant, ces méthodes s'appuient soit sur des données de visage 3D rares et non publiques, soit sur une approche d'optimisation complexe et coûteuse. Cette thèse propose une méthode de génération de texture haute-fidélité, qui prédit la texture globale du visage 3D à partir d'une seule image de visage en entrée. L'apprentissage est basé sur la pseudo vérité de terrain mélangée par le 3DMM et les textures du visage d'entrée. De multiples discriminateurs de cartes UV partielles sont utilisés pour gérer les artefacts imparfaits de la pseudo-vérité terrain. En termes de désocclusion de visage, nous proposons un GAN de désocclusion de visage basé sur sengmentation et reconstruction, composé de trois parties, un module de régression des paramètres 3DMM N θ , un module de segmentation des occlusions N S , et un module de génération d'images N G . Avec la texture préalable fournie par N θ et les parties occluses indiquées par N S , N G peut récupérer fidèlement les textures manquantes. La méthode proposée surpasse quantitativement et qualitativement les méthodes de l'état de l'art. Mots clés: synthèse de pose de visage, reconstruction de visage tridimensionnelle, dé-occlusion d'images faciales.

Figure take from https://blog.csdn.net/Invokar/article/details/88917214

Figure taken from https://vincentherrmann.github.io/blog/wasserstein/

Figure taken from[START_REF] Gulrajani | Improved training of wasserstein gans[END_REF]

Figure taken from[START_REF] Zhang | Self-attention generative adversarial networks[END_REF]

Acknowledgements

where L is the loss function. Since c i is sampled from the output of the texture generator, i.e., the UV map, the gradients could be further backpropagated. In our project, we adopt the off-the-shelf differentiable renderer of PyTorch3D [START_REF] Ravi | Accelerating 3d deep learning with pytorch3d[END_REF].

Pixel attention sampling To get the UV map of visible parts, UV-GAN [4] first fits a 3DMM to the input image, then use the vertices' projected 2D coordinates to sample their corresponding colors, and the incomplete UV map is further generated. However, their method relies on accurate 3D shape fitting and facial landmark detection. Furthermore, such a method does not have a mechanism to deal with face occlusions(hands, hair, eyeglasses, etc.). Inspired by [START_REF] Yin | Pixel sampling for style preserving face pose editing[END_REF], we apply a pixel attention sampling (PAS) module to sample the incomplete UV map from the input image directly. Thanks to this module, the inference process is free from 3D shape or facial landmarks. Besides, different from [START_REF] Yin | Pixel sampling for style preserving face pose editing[END_REF], where input images require landmark-based pre-alignment due to the arbitrary target poses. The target output, i.e., the UV map, is highly structured, so neither spatial transformation to the input image nor the target pose condition is demanded.

Proposed Method

The goal of our method is to predict the face UV map from a single face image. As illustrated in Figure 4.3, the proposed model consists of two parts: a UV attention sampling module (UV sampler) and a UV map inpainting module (UV generator).

During the inference process, the UV sampler will sample the pixels from the input image to generate an incomplete UV map, and then the UV generator will further complete the semi-finished UV map. We describe the details of each component as follows.

UV Attention Sampling

The UV map is a two-dimensional representation of the global texture of a 3D object. Due to self-occlusion, it is an ill-posed problem to get the UV map from a single image. This section studies how to generate an incomplete UV map that contains only visible textures of the input face image. As a comparison, we recall gle. As shown in Figure 4.7, the incomplete UV map generated by the UV sampler discards the inaccurate parts, forcing the subsequent UV generator to work on optimizing this region. The incomplete UV map generated by the traditional method, on the other hand, has preserved the inaccurate edges, causing the subsequent UV generator to lack the urge to optimize the edge area. In consequence, the generated UV map is blurred and full of noise. However, the textures corresponding to the input face image's occluded parts will still suffer from blurring and artifacts. The 4th and 5th row of Figure 4.8 shows the problems caused by the absence of partial discriminators.

Segmentation head During pixel sampling, the segmentation head of the UV sampler could mask out all face occlusions (glasses, hands, scarf, etc.) to avoid the occlusions appear in the predicted UV map. Figure 4.9 illustrates its effect.

Model Structure. N S follows the classical U-Net structure [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], with ResNet18 [START_REF] He | Deep residual learning for image recognition[END_REF] as encoder, while the decoder is a stack of convolutional blocks corresponding to each stage of the encoder. Each decoder block takes features from its previous block and its symmetric encoder stage through skip-connection. Since the target mask is binary, a single channel output for the decoder suffices.

Losses. The training is guided by the Dice loss [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF] and the Binary Cross-Entropy loss (BCE):

where W and H represent the width and height of the mask, respectively. Further, we apply the "online hard example mining" (OHEM) [START_REF] Shrivastava | Training region-based object detectors with online hard example mining[END_REF] strategy to the BCE loss to make the training focus on hard examples, thus achieving more effective and efficient training.

The proposed segmentation module achieves a precision of 0.981 and an IoU score of 0.954 on the validation dataset, which is acceptable considering the inaccurate edge region of the ground truth. By random checking the results, we surprisingly find that the predicted face region is often more accurate than the ground truth.

We use this module to extract the face regions of the images in CelebA-HQ and FFHQ, which greatly facilitates the training of the other two modules, i.e., the reconstruction module N R and the inpainting module N G .

3D Face Reconstruction

The reconstruction module is adapted from [3], the state-of-the-art 3D reconstruction method with a ResNet50 [START_REF] He | Deep residual learning for image recognition[END_REF] as its backbone. Given a face image, it predicts a vector c ∈ R 239 , containing 6 translation and rotation parameters; 144 shape and 80 texture coefficients of the 3DMM; and nine illumination coefficients of the Spherical Harmonics [START_REF] Ramamoorthi | An efficient representation for irradiance environment maps[END_REF][START_REF]A signal-processing framework for inverse rendering[END_REF] model. With vector c, the face is reconstructed and further rendered to the image through a differentiable renderer. Losses. Since we already have c gt , the most straightforward option is to use it as supervision; the loss is as: where ĉ is the predicted coefficient, and N is its dimension.

Training solely with Equation 5.5 will result in inaccurate and non-discriminative results. Since it equally optimizes all the coefficients. However, different parts of ĉ obviously have different impacts on the reconstruction result, e.g., the poses play a more critical role than the illumination coefficients. Therefore, we also leverage the pixel-wise and the perceptual [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] losses to reduce the discrepancy between the reconstructed face Î and the real face I f :

where M is the overlap of the original face mask M f and the reconstruction mask M m ; F(•) denotes the feature embedding function of a pre-trained face recognition model, and we use Arcface [START_REF] Deng | Arcface: Additive angular margin loss for deep face recognition[END_REF] here. We only extract the feature of I f , rather than the entire image I, to avoid the noises introduced by the non-facial patterns.

To accelerate the training, we also use the landmark loss. Thanks to c gt , we have ground truth 3D facial landmarks q, thus eliminating the need for a 2D facial

Qualitative Results

The method closest to ours is FD3D [START_REF] Yuan | Face de-occlusion using 3d morphable model and generative adversarial network[END_REF]; we both leverage 3D face reconstruction for face de-occlusion. Unfortunately, they do not release their code, so we use images from their paper to conduct the qualitative evaluation. Figure 5.7 compares our method with FD3D and several other publicly available image inpainting methods, including DeepFill-v2 [START_REF] Yu | Free-form image inpainting with gated convolution[END_REF], PConv [START_REF] Liu | Image inpainting for irregular holes using partial convolutions[END_REF], PICNet [START_REF] Zheng | Pluralistic free-from image completion[END_REF], and LaFIn [START_REF] Yang | Generative landmark guided face inpainting[END_REF]. The FD3D results are taken from their paper, while the rest are generated from their official implementations with models pre-trained on face images. We provide those inpainting methods with manually labeled occlusion masks for a fair comparison.

work collaboratively to mask the occlusions and provide beneficial priors to guide the subsequent face inpainting module. Qualitative and quantitative evaluations demonstrate the superiority of the proposed method.

remains unsolved. There are two categories of research addressing this problem:

one is to construct larger 3D face datasets as training data, which are costly and involve privacy and legal issues; the other is to follow a analysis-by-synthesis training strategy: the model render the reconstructed 3D face into image space and uses the reconstruction loss to guide the generation of fine-grained shapes. However, such a method could be corrupted by the occlusions (e.g., bangs, eyeglasses) in the input face image. In the future, we plan to use the method of chapter 5 to de-occlude the face image before applying the above method, thus eliminating the negative influences of the occlusions.