Ugo Tanielian Invité

Keywords: GANs, generative models, adversarial training, image editing

The past years have seen a great progress of deep generative models, including Generative Adversarial Networks (GANs). Notably, they can synthesize high-resolution images, sometimes indistinguishable from real images. Deep generative models were also at the root of empirical successes such as music generation or molecular discovery. However, we lack a fundamental understanding of the capabilities and limitations of deep generative models. In this thesis, we first characterize a model misspecification of generative models with connected output distributions, such as GANs or normalizing flows. Indeed, such models can not perfectly fit a target distribution composed of several disconnected modes. We analyse theoretically their best achievable performance in the setting of disconnected target distribution, and which geometrical structure can allow them to achieve best performance. Moreover, we propose methods that improve the performance of GANs by making them more amenable for disconnected data modelling.

In the second part of the thesis, we aim to improve image editing techniques thanks to deep generative models. First, we leverage a pre-trained unconditional generative model and show that it can perform a wide range of image editing tasks without re-training. Second, we build on adversarial learning to improve virtual try-on models, which consist in replacing the clothing item on an image of a person.

Goals

The goal of this thesis is to understand and develop the capabilities of deep generative models for natural image generation and manipulation. Generative modelling has applications in several areas of machine learning, such as semi-supervised learning or adversarial robustness.

Generative modelling aims at learning the distribution of data by having access to a discrete set of samples.

Adversarial Networks (GANs). In section 3.3, we propose a new analysis of the latent space of deep generative models. We prove that there exists an optimal geometric structure that partitions the Gaussian latent space of generative models. Moreover, we demonstrate experimentally that the appearance of this structure is positively correlated with the performance of such models, and that it can be enforced. In section 3.4, we derive a learning-based method that models a rejection mechanism in the latent space of pre-trained GANs. Interestingly, it allows to model disconnectedness in the generated distribution.

Leveraging deep generative models for image manipulation. Image manipulation encompasses a wide range of tasks, from scribble-based editing to image denoising or virtual try-on. It has exciting applications, such as creating intelligent virtual assistants for artists or developing tools for e-retailers that increase client engagement. However, it suffers from inherent difficulties: 1) their objective might be ill-defined, and tough to define; 2) the ideal data for these tasks might be complex and costly to collect. In section 4.2 we leverage a mask-based training objective and show that it allows to learn a single generative model able to tackle various image editing tasks, like image inpainting and image composition. In chapter 4.3, we approach the virtual try-on task with a teacher-student strategy, augmented with adversarial training, that allows to train the student model on a ideal synthetic dataset created by the teacher model.

Motivations

The development of deep learning methods for natural image generation and manipulation is motivated by a wide range of applications, from molecular generation in biology [START_REF] Prykhodko | A de novo molecular generation method using latent vector based generative adversarial network[END_REF] to designing SDE solvers in physics [START_REF] Yang | Physics-informed generative adversarial networks for stochastic differential equations[END_REF]. We present some of the main applications below.

Arts and advertising. The creation and editing of visual content is a highly valued skill, particularly in digital advertising, where visually appealing content is key. However, the process of creating product catalogues can be both expensive and complex, particularly when dealing with large and heavy furniture items. Furthermore, image editing demands extensive human expertise and knowledge of professional software. Therefore, the development of smart and automatic image editing techniques would greatly benefit marketing teams by expediting the process of transforming advertising ideas into visually appealing content. Additionally, automatic image editing has the potential to enhance advertising personalization. For instance, in Figure 1.1, a virtual try-on system is showcased, which would enable e-shoppers to try on clothing items on a picture of themselves. Intelligent virtual assistants have the potential to transform the way artists work, revolutionizing the creation process of artworks through either pure generation or human-guided iterative procedures. This innovation has already sparked significant interest, with AI-designed artworks selling for high prices in 2018. Recently, large text-to-image generation models, such as Parti [START_REF] Yu | Scaling autoregressive models for content-rich text-to-image generation[END_REF] and Stable Diffusion [START_REF] Rombach | High-resolution image synthesis with latent diffusion models[END_REF], have achieved impressive performances, as demonstrated in Figure 1.2. Such models have been leveraged by some artists to create generated animation movies. These developments suggest that the potential of intelligent virtual assistants is vast and that they could accelerate the art creation process. 2023) introduce a method to improve the generated images from text-to-image models, specifically Stable Diffusion in this instance. The process involves generating an image from a given text using Stable Diffusion and allowing the user to refine the generated image by modifying the prompt. This can be achieved by adjusting the intensity of a descriptor (such as crowded or fluffy), adding new text, or modifying existing words. can enhance the capabilities of deep learning models with minimal labeling by learning the underlying data structure without accessing any labels. For instance, [START_REF] Li | Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations[END_REF] employed deep generative models to generate large-scale datasets with pixel-wise annotations of objects in images. However, the mechanisms that support the separation of data modes in the latent and feature space of deep generative models are not fully understood. Therefore, we argue that a better comprehension of these mechanisms could enable even more effective use of deep generative models in this context.

Context

Enhancing adversarial robustness with deep generative models. Deep learning models used for discriminative tasks such as image classification or image segmentation have a wellknown lack of robustness. For instance, adversarial attacks are a highly studied phenomenon in which the output of a classifier is modified by adding a small, imperceptible amount of noise to an image. Surprisingly, a deep learning-based classifier can even be fooled by modifying a single pixel of an image. Fortunately, there have been several proposed defense algorithms that leverage deep generative models. The intuition behind this approach is that adversarial attacks use the low-dimensional manifold structure of the data and send adversarial points to unseen portions of the data space. By utilizing deep generative models, we can project the attacked images back to the data manifold. For example, [START_REF] Song | Pixeldefend: Leveraging generative models to understand and defend against adversarial examples[END_REF] use neural density estimators (PixelCNN), while [START_REF] Samangouei | Defense-GAN: Protecting classifiers against adversarial attacks using generative models[END_REF] leverage Generative Adversarial Networks to improve the robustness of deep learning-based classifiers.

Context

In recent years, the field of deep learning has undergone rapid progress, particularly in the development of deep generative models. GANs were invented in 2014 [START_REF] Goodfellow | Generative adversarial nets[END_REF], and represent a significant breakthrough in the application of deep neural networks for learning probability distributions and synthesizing images. Initially, these models were only used for grayscale 32 × 32 images. A few years later, [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF] successfully extended GANs to class-conditional synthesis on ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF] and [START_REF] Karras | Progressive growing of GANs for improved quality, stability, and variation[END_REF]Karras et al. ( , 2019a) ) scaled GANs to high-resolution 1024 × 1024 images using multi-scale approaches and specific architectural design.

This thesis seeks to address and overcome the limitations of deep generative models, specifically Generative Adversarial Networks (GANs), on multi-modal target distributions. Additionally, we aim to employ these models to derive smarter algorithms for image editing. Several factors have contributed to these developments. First, the availability of large-scale datasets in computer vision has been increasing. Second, Graphics Processing Units (GPUs), used for computing deep learning experiments, have become more efficient and easier to use. This progress in performance is known as Huang's law, which posits that GPUs are advancing at a faster rate than standard Central Processing Units (CPUs). At Criteo AI Lab, we have access to a cluster with nodes consisting of two NVIDIA Tesla V100 GPUs with 16GB of RAM. For larger experiments, we also have access to a large machine with eight NVIDIA Tesla V100 GPUs, each with 32GB of RAM. Third, open-source libraries such as PyTorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF] or Tensorflow [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF] have greatly facilitated the prototyping of deep learning models. In this thesis, PyTorch was utilized for the majority of the experimental work. Finally, the progress of deep generative models has been driven by both fundamental and applied research, including the development of new layers, architectures, or optimization methods. The code for recent methods is often open-sourced, along with the pre-trained model weights. All of these factors have facilitated rapid progress in deep generative models and have yielded impressive results, such as large-scale text-to-image synthesis (refer to Figure 1.2).

Challenges

When tackling the goals presented above (learning image generation and editing with deep neural networks), we face two main challenges.

Multi-modality and high diversity of natural image distributions. Looking from a historical perspective, initial computer vision datasets consisted of only a small number of classes with limited intra-class variability, such as black-and-white digits [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. However, benchmark datasets have grown increasingly complex over time, with current large-scale datasets often containing more than a thousand classes, such as ImageNet with 21k classes [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF]. Consequently, such datasets exhibit high variability and diversity in the underlying and unknown target distribution. However, when working with such complex distributions, deep generative models may have two potential drawbacks. These models may occasionally overlook some modes, resulting in the well-known phenomenon of mode collapse, or they may generate low-quality data points that are located between data modes.

Lack of data for learning-based image editing. The success of deep learning models arise from well-defined training objectives optimized over large sets of training data. However, in the case of image editing tasks, such ideal combination of datasets and proper training objectives are often nonexistent. For instance, consider the task of scribble-based image editing, where a user edits an image with scribbles and the objective of the model is to produce a realistic final image while taking the user's scribbles into account. The ideal dataset for this task would comprise input images with scribbles and their corresponding edited versions. Unfortunately, constructing such a dataset would require a significant investment of time and effort from professional graphic designers. As a result, a supervised learning approach to this problem becomes infeasible. Instead, in this thesis, we investigate the use of pre-trained generative models to project images with scribbles onto the manifold of real images, thus offering a viable solution.

Thesis outline and contributions

We present the thesis outline along with the contributions:

Chapter 2: Related work. we provide a comprehensive overview of the various approaches for deep generative modelling. Specifically, we discuss the different families of models, including Generative Adversarial Networks, Autoregressive Models, and Score-Based Models. Additionally, we present standard techniques for manipulating images using deep neural networks.

Chapter 3: Learning multi-modal distributions with deep generative models. In this chapter, we focus on the challenge of learning multi-modal distributions with push-forward generative models such as GANs and VAEs. We make three distinct contributions in this area:

• Section 3.2: learning disconnected manifolds: a no GANs' land. We formalize the fundamental tradeoff that GANs face when learning multi-modal data, and propose a novel solution to this problem. Indeed, since GANs have a connected output distribution, they either fit all modes of the data and generate low-quality points, either ignore all but one mode and generate only good quality points. Thus, we give an upper-bound on the maximum attainable precision of GANs. This upper-bound depends on the Lipschitz constant of the generator, on the number of modes in the data and on the minimal distance between modes. Moreover, building on our theoretical analysis, we propose the 'Jacobian-Based Truncation' method for GANs. This approach involves rejecting samples with the highest Jacobian Frobenius Norm of the generator. This resulted in the following publication:

Ugo Tanielian, Thibaut Issenhuth, Elvis Dohmatob, and Jérémie Mary. Learning disconnected manifolds: a no gan's land. (2020) In International Conference on Machine Learning (ICML).

• Section 3.3: unveiling the latent space geometry of push-forward generative models.

We derive a new theoretical analysis on the problem of learning disconnected manifolds with deep generative models formulated as push-forward models of a Gaussian latent distribution (such as GANs or VAEs). We study the role of the latent space geometry on the performance of such push-forward generative models. Notably, we prove a sufficient condition for their optimality: generators that structure their latent space with a specific geometry called 'simplicial cluster' are optimal. We show that this also verified experimentally with GANs. The more GANs cluster the data modes in linear regions, the better their performance. Additionally, we propose a truncation method that enforces a 'simplicial cluster' structure and improves performance of GANs. This work resulted in the following publication:

Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, David Picard. Unveiling the latentspace geometry of push-forward generative models. ( 2023) International Conference on Machine Learning (ICML).

• Section 3.4: latent reweighting for GANs. Drawing from the preceding analysis, we present a novel approach for enhancing the quality of pre-trained GANs through a learning-based method. Specifically, we introduce an additional network that predicts importance weights from latent vectors, with the objective function of this network being adversarial. Our approach offers several advantages, as latent importance weights can be assigned zero values, allowing the generated distribution to model disconnected distributions effectively. Moreover, as the re-sampling mechanism occurs in the latent space, our approach is computationally efficient and can be rapidly executed at inference time. This work resulted in the following publication:

Thibaut Issenhuth, Ugo Tanielian, David Picard, Jérémie Mary. Latent reweighting, an almost free improvement for GANs. ( 2022) IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

Chapter 4: image editing with deep neural networks. This chapter focuses on the development of deep learning methods that can perform image editing tasks without requiring direct supervision. We present two contributions. The first concerns a generalist model for image editing, and the second is designed for image-based virtual try-on:

• Section 4.2: EdiBERT: a generative model for image editing. The first section of this chapter introduces a novel approach that enables multiple image editing tasks to be performed using a single trained model. To achieve this, we utilize a training objective that shares similarities with most editing tasks -generating a highly realistic image from a low-quality input. Our proposed approach, named EdiBERT, is a bi-directional transformer that re-samples image patches conditioned on the entire input image. Trained with a single generic objective, EdiBERT outperforms other multi-task editing methods, such as GANs' inversion. This method is particularly well-suited for tasks that lack large-scale conditional datasets, such as image composition or scribble-based editing. This chapter is based on the following publication:

Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, David Picard. EdiBERT: a generative model for image editing. (2022) Transactions on Machine Learning Research (TMLR).

• Section 4.3: a parser-free virtual try-on. The virtual try-on task poses a significant challenge due to the lack of an ideal dataset, resulting in a complex pipeline comprising multiple neural networks for human parsing, pose estimation, image warping, and image composition. The pipeline is computationally intensive, with pre-processing steps (human parsing and pose estimation) adversely impacting the final image quality since it masks important information in the input image. To address these limitations, we propose an adversarial teacher-student paradigm that eliminates the pre-processing steps during inference. Our approach reduces the inference time for virtual try-on significantly while improving the image quality.

Thibaut Issenhuth, Clément Calauzènes, and Jérémie Mary. Do not mask what you do not need to mask: a parser-free virtual try-on. (2020) In European Conference on Computer Vision (ECCV).

Chapter 5: Conclusion. We summarize the contributions of this thesis and outline future directions of research, such as exploring the neural collapse phenomenon in deep generative models or investigating the latent space properties of score-based models.

Chapter 2

Related work

In this chapter, we present and discuss the recent advances in deep generative modelling, before reviewing the use of deep neural networks for image editing. This part assumes basic knowledge on machine learning and deep learning, which can be found for example in [START_REF] Goodfellow | Deep learning[END_REF].

Deep generative modelling

In this section, we provide an overview of the various families of deep generative models employed in this thesis, highlighting their inherent challenges and limitations with regards to the questions that we adress in this thesis. For an in-depth and comprehensive presentation of deep generative models, interested readers can refer to the works of [START_REF] Murphy | Probabilistic machine learning: Advanced topics[END_REF] or [START_REF] Tomczak | Deep generative modeling[END_REF].

A deep generative model is a parametric family of probability distribution p θ , where θ ∈ Θ are the parameters of a neural network. It is trained to fit a target distribution µ ⋆ which is only accessible through an empirical distribution µ ⋆ n , i.e. a dataset of training samples X = (x 1 , . . . , x n ) where x i ∈ R D and n is the number of samples in the dataset. During training, it is optimized to minimize a distance (or divergence) D between the empirical distribution and the modelled probability distribution p θ :

θ ⋆ = min θ ∈Θ D(µ ⋆ n , p θ ).
(2.1.1)

The choice of D varies depending on the family of deep generative model. For instance, Wasserstein GANs employ a Wasserstein distance as D (Arjovsky and Bottou, 2017;[START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF], while auto-regressive models rely on a Kullback-Leibler (KL) divergence. Fig. 2.1 The evolution of images generated by GANs, from their invention in 2014 to recent developments in 2020. Source: (Zhang et al., 2021a, Figure 2.1.7).

Another notable difference between deep generative models is the way they model probability distributions. Push-forward generative models, whose denomination was introduced by [START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF], are a broad class of deep generative models that learn a push-forward mapping from a simple latent distribution γ, typically a Gaussian or Uniform distribution, to the modeled distribution, denoted by p θ = G θ ♯γ. The generator G θ is a neural network with parameters θ , and ♯ represents the push-forward operator. GANs, VAEs, and normalizing flows are all examples of push-forward generative models, while score-based and diffusion models are classified as indirect push-forward generative models.

In contrast, auto-regressive models do not belong to the push-forward generative model class. Auto-regressive models utilize the decomposition p θ (x) = ∏ i p θ (x i |x ⩽i ), where x ∈ R d . Sampling in auto-regressive models is performed iteratively by sampling the 1-dimensional conditional distributions with a multinomial distribution. Unlike push-forward generative models, auto-regressive models do not possess a latent distribution γ. The lack of latent space makes them challenging to reuse for purposes other than sampling, such as inverse problems with generative models.

Families of deep generative models

Generative adversarial networks

GANs, proposed in [START_REF] Goodfellow | Generative adversarial nets[END_REF], were the first family of deep generative models able to sample high-quality image even at high-resolution of 1024 × 1024 [START_REF] Karras | Progressive growing of GANs for improved quality, stability, and variation[END_REF]. On Figure 2.1, we can observe the fast progress of the quality of GANs' generated images.

GANs are based on an adversarial game between a generator network G θ : R d → R D and a discriminator network D φ : R D → R. The discriminator tries to distinguish real from fake images, while the generator tries to fool the discriminator. More formally, GANs rely on a minimax optimization procedure:

min θ ∈Θ max φ ∈Φ E x∼µ ⋆ n ,z∼γ [ f 1 D φ (x) + f 2 D φ (G θ (z) ] (2.1.2)
where f 1 (x) = log(x) and f 2 (x) = 1log(x). Under the assumptions of an optimal discriminator, this procedure is shown to minimize the Jenson-Shannon divergence between the empirical distribution and the generator's distribution. Surprisingly, [START_REF] Goodfellow | Generative adversarial nets[END_REF] found that instead of minimizing f 2 D φ (G θ (z)) in the generator step, minimizingf 1 D φ (G θ (z)) produced better results and reduced mode collapse. This approach is known as the non-saturating loss.

One of the primary challenges researchers faced was stabilizing the GANs training procedure. One significant observation was that, using this formulation, training the discriminator to optimality results in a discriminator with 100% accuracy and zero gradients for the generator (Arjovsky and Bottou, 2017). This led to the development of Wasserstein GANs (Arjovsky et al., 2017), where f 1 (x) = f 2 (x) = x and discriminators are restricted to 1-Lipschitz functions. Initially, enforcing the discriminator to be 1-Lipschitz was achieved by clipping weights. Later, gradient penalty [START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF], spectral normalization [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF], and adversarial regularization [START_REF] Terjék | Adversarial lipschitz regularization[END_REF] were introduced. However, it was later discovered that Wasserstein GANs could not converge, even on simple one-dimensional cases [START_REF] Mescheder | Which training methods for gans do actually converge[END_REF]. The standard formulation of GANs with gradient regularization on the discriminator leads to a better-behaved optimization procedure. Overall, gradient regularization of the discriminator was critical to stabilizing the GANs' training.

A second crucial challenge is the learning of multi-modal distributions. In the seminal paper of GANs [START_REF] Goodfellow | Generative adversarial nets[END_REF], the mode collapse issue was already mentioned. On the opposite, when there is no mode collapse and all target modes are represented, another problem arises: images are sampled in-between modes and they are unrealistic. Indeed, as brought to the fore by [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF], there is most often no realistic interpolation between two modes of image datasets, e.g. between a car and a dog. This is a density misspecification problem [START_REF] Roth | Stabilizing training of generative adversarial networks through regularization[END_REF]: in the case of multi-modal and disconnected target distribution, there exists no parameters θ ∈ Θ such that µ θ = µ ⋆ . A solution is to use mixture distributions as latent space [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF], or mixture of generators [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF]. However, this introduce additional hyper-parameters and can thus complicate the training procedure. Another line of work considers rejection mechanisms from pre-trained GANs, leveraging the discriminator scores [START_REF] Azadi | Discriminator rejection sampling[END_REF][START_REF] Tanaka | Discriminator optimal transport[END_REF] or the generator's curvature [START_REF] Arvanitidis | Latent space oddity: on the curvature of deep generative models[END_REF][START_REF] Humayun | Polarity sampling: Quality and diversity control of pre-trained generative networks via singular values[END_REF].

Variational auto-encoders

Variational auto-encoders (VAEs) [START_REF] Kingma | Auto-encoding variational bayes[END_REF][START_REF] Kingma | An introduction to variational autoencoders[END_REF] are another family of push-forward generative model, based on a likelihood training objective. However, training a latent variable model with a likelihood objective is not direct. Indeed, the modelled probability p θ requires an intractable computation: p θ (x) = z p θ (x, z)dz. The solution proposed by [START_REF] Kingma | Auto-encoding variational bayes[END_REF] is to use variational inference, which resorts to approximating the posterior p θ (z|x) with a stochastic encoder network q φ (z|x). The loglikelihood can then be written:

log p θ (x) = E q φ (z|x) [log p θ (x|z)] -D KL q φ (z|x)| p(z) + D KL q φ (z|x)| p θ (z|x) (2.1.3)
where D KL is the KL-divergence term.

Since the true posterior p θ (z|x) is unknown, this formulation is still not tractable. However, the last term which involves p θ (x|z) is positive and can be dropped. This gives the Evidence Lower Bound (ELBO), which is the objective function of VAEs: log

p θ (x) ⩾ L ELBO (x; θ , φ ) = E q φ (z|x) [log p θ (x|z)] -D KL q φ (z|x)| p(z) .
Furthermore, choosing p(z) and q φ (z|x) to be Gaussian distributions allows to get a closedform expression of the KL-divergence term. Most often, p(z) is a standard Normal distribution with a zero mean vector and identity covariance matrix. The encoder network thus predicts the mean vector µ q φ (x) and diagonal covariance matrix σ q φ (x) of a Gaussian distribution. Since q φ is a Gaussian distribution, a final challenge is to back-propagate the reconstruction term E q φ (z|x) [log p θ (x|z)] to the encoder. This is achieved via the reparametrization trick, which consists in sampling from ε ∼ N(0, I) and computing x ′ = σ q φ (x) ⊙ ε + µ q φ (x), where ⊙ is an element-wise product.

Variational auto-encoders have achieved good image synthesis quality, as we observe in Figure 2.2, thanks to hierarchical approaches [START_REF] Ranganath | Hierarchical variational models[END_REF][START_REF] Kingma | Improved variational inference with inverse autoregressive flow[END_REF][START_REF] Vahdat | Nvae: A deep hierarchical variational autoencoder[END_REF]. The idea behind hierarchical variational models is to separate latent variables into different groups, so that the prior distribution has more expressiveness. However, variational model continue to lag behind GANs in terms of sampling quality.

Finally, alternative regularization methods have been proposed, such as the use of a discriminator to enforce a specific prior distribution on the encoded data points [START_REF] Tolstikhin | Wasserstein auto-encoders[END_REF][START_REF] Makhzani | Adversarial autoencoders[END_REF]. Interestingly, this method allows to use a deterministic encoder, and the regularization term is applied to the aggregate posterior rather than individual samples.

Auto-regressive models

Auto-regressive models are a family of models based on the maximization of the data loglikelihood. An auto-regressive model adopts a sequence modelling approach, and models probability distribution with the following decomposition:

p θ (x) = ∏ i p θ (x i |x <i ) (2.1.4)
This assumes to pre-define an order on the data. Although this decomposition makes sense for natural language processing, where sentences have a natural order, it is however less natural on images, where pixels reside on a two-dimensional plane. Generally, a simple 'raster-scan' order is adopted, where pixels are processed sequentially row by row, moving from left to right and from the top of the image to the bottom.

A seminal work of autoregressive models for images was PixelCNN ( Van den Oord et al., 2016;Van Den Oord et al., 2016), which proposed a deep auto-regressive neural network for images. Notably, they proposed masked convolutions, allowing each pixel to only access information from the previous ones. However, a limit of auto-regressive model for images is the large number of pixels, e.g. 10 6 for an image of resolution 1024 × 1024. To make auto-regressive models more efficient, [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] derive a two-stage approach. In the first stage, an auto-encoder compresses images into sequences of discrete elements. In the second stage, an auto-regressive model learns the distribution of the sequences of discrete elements. This lead to high-quality results with the adoption of the transformer architecture [START_REF] Vaswani | Attention is all you need[END_REF] as an auto-regressive model [START_REF] Esser | Taming transformers for high-resolution image synthesis[END_REF]. This approach is shown in Figure 2.3. A great advantage of such approaches is their very stable training, which allows an easier scaling of models with overparametrized models and large-scale datasets.

A line of research has focused on getting rid of the arbitrary ordering of sequences, following the path of BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] where the objective is to reconstruct masked sequences. Notably, MaskGIT [START_REF] Chang | Maskgit: Masked generative image transformer[END_REF] achieved state-of-the-art synthesis results by training a transformer to reconstruct masked sequences, and generating samples with a parallel decoding scheme. This was then extended to a large text-to-image model [START_REF] Chang | Muse: Text-to-image generation via masked generative transformers[END_REF].

Diffusion and score-based models

Diffusion models [START_REF] Sohl-Dickstein | Deep unsupervised learning using nonequilibrium thermodynamics[END_REF] are based on an iterative process that maps data to noise, and consists in learning the reverse process with a neural network. In the first years, they received less attention than other deep generative models. However, recent work by [START_REF] Ho | Denoising diffusion probabilistic models[END_REF] has revived interest in diffusion models by drawing a fundamental connection between score-based models [START_REF] Song | Generative modeling by estimating gradients of the data distribution[END_REF][START_REF] Song | Score-based generative modeling through stochastic differential equations[END_REF] and diffusion models. The first score-based generative model [START_REF] Song | Generative modeling by estimating gradients of the data distribution[END_REF] builds upon two key ingredients: learning the score s(x) = ∇ x log p(x) of a distribution µ ⋆ with denoising autoencoders [START_REF] Vincent | A connection between score matching and denoising autoencoders[END_REF], and Langevin sampling.

First, [START_REF] Vincent | A connection between score matching and denoising autoencoders[END_REF] showed that the following simple denoising objective allows to learn the score of a distribution:

L(θ ) = E x∼µ ⋆ n , x∼q σ (x) [∥s θ ( x) - x -x σ 2 ∥ 2 2 ] (2.1.5)
where s θ is a neural network with parameters θ , x are samples from the target distribution, and x are noisy samples x = x + ε with ε ∼ N (0, σ 2 I).

Second, Langevin sampling allows to draw samples from a given distribution µ ⋆ when having access to ∇ x log µ ⋆ (x). It is a Monte-Carlo Markov Chain (MCMC) composed of the following update steps:

x t+1 = x t + ε∇ x log µ ⋆ (x) + √ 2εz t
, where z t is a standard Gaussian random variable.

Score-based models, introduced in Song and Ermon ( 2019) and further developed in [START_REF] Song | Score-based generative modeling through stochastic differential equations[END_REF], rely on a hierarchy of Gaussian noise variances σ 1 < • • • < σ L to generate highquality images. An efficient training strategy is to use a single neural network conditioned to the noise level s θ (x, σ i ). For sampling, an annealing Langevin strategy is proposed, which runs Langevin MCMC with the highest noise level σ L and gradually reduces the noise level until the lowest level σ 1 is reached. Building upon these ideas, [START_REF] Song | Score-based generative modeling through stochastic differential equations[END_REF] propose using an infinite number of noise scales, effectively reversing a continuous-time stochastic process. This approach allows for new numerical stochastic differential equation solvers to be derived [START_REF] Song | Score-based generative modeling through stochastic differential equations[END_REF][START_REF] Karras | Elucidating the design space of diffusionbased generative models[END_REF].

In conjunction with architectural improvements and large-scale training, score-based and diffusion models have achieved state-of-the-art image quality in unconditional [START_REF] Karras | Elucidating the design space of diffusionbased generative models[END_REF][START_REF] Dhariwal | Diffusion models beat gans on image synthesis[END_REF]) and text-to-image models [START_REF] Saharia | Photorealistic text-to-image diffusion models with deep language understanding[END_REF]. This can be observed in Figure 2.4. The superiority of these models over GANs is not yet fully understood, although it seems that the training objective (L2 loss) contributes to their stability, ease of optimization, and scalability. A notable feature of score-based and diffusion models is their iterative sampling procedure, which involves L forward passes on the score network to generate one image. This could contribute to their greater expressive power compared to models that rely on a single forward pass.

Evaluating deep generative models

The evaluation of deep generative models is a critical yet often neglected problem. The machine learning pipeline relies heavily on optimizing metrics, whether for learning the model's parameters or estimating the optimal hyperparameters. However, many generative models are trained with a differentiable loss function that does not directly reflect their ultimate objectives, namely distribution fitting and sample quality. Even when log-likelihood evaluation is possible, it may not necessarily correlate with other objectives such as sample quality, as noted by [START_REF] Theis | A note on the evaluation of generative models[END_REF]. Also, it is worth noting that the value of deep generative models lies in their ability to generalize, which is essential for generating unseen samples or testing the likelihood of unseen data.

Standard metrics and their flaws. The initial efforts to evaluate deep generative models, particularly GANs which do not have likelihood estimates, led to the development of the Inception Score (IS) [START_REF] Salimans | Improved techniques for training GANs[END_REF] and the Fréchet Inception Distance (FID) [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF]. IS leverages a pre-trained Inception classifier to compute the probability of classes per sample. It is minimized when samples are diverse and the sample-wise class distributions are peaked. On the other hand, the FID also utilizes the Inception Net as a feature extractor but only compares target and generated distributions, assuming they are Gaussian, using the Fréchet Distance. However, FID has been shown to have several limitations, such as the false assumption of Gaussianity for multimodal datasets [START_REF] Luzi | Evaluating generative networks using gaussian mixtures of image features[END_REF] and the potential for misleadingly high scores when there is a high intra-class variation and poor coverage by the generative model [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF].

Metrics based on support estimation. To solve this issue, another line of work developed precision and recall metrics for generative models. These metrics have gained traction due to their clarity and interpretability. Intuitively, precision measures the quality of generated samples, while recall measures the mode coverage of the generated samples. The first precision and recall metric was proposed by [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF]. However, the algorithm had limitations, relying on clustering and discrete probability comparison, which does not account for situations with intra-cluster variability. To address this issue, [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF] developed an improved precision and recall metric by directly approximating the real and generated supports with k-nearest neighbor spheres around samples. Precision and recall can then be defined per sample. Precise samples are generated sampled that fall within the approximated support of data samples. Figure 2.5 provides an illustration of this metric. However, [START_REF] Naeem | Reliable fidelity and diversity metrics for generative models[END_REF] found that the improved precision and recall metrics are sensitive to outliers and proposed density and coverage metrics. What about generalization? While the aforementioned methods provide ways to assess the quality of generative models, they fail to evaluate their generalization abilities. Suppose we compute the test-FID or the test-Precision and Recall, which would compare generated samples to test samples. A good score obtained using these metrics does not indicate good generalization abilities since using the training set (instead of generated samples) also yields a great score. To address this issue, [START_REF] Alaa | How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models[END_REF] proposed a new metric called authenticity that measures the proportion of generated points that are very similar to training data, thereby providing a heuristic evaluation of a model's generalization abilities.

Image editing with deep neural networks

In this section, we provide an overview of modern methods for image editing using deep neural networks. First, we discuss neural network architectures that are particularly well-suited for image editing tasks. Next, we describe training techniques for image editing tasks that can be classified into two categories: supervised and unsupervised.

To begin, let us define image editing more precisely. Image editing involves mapping a given source image I s to a realistic image I r , given some user-defined conditioning C. The function that we aim to learn is of the form f θ (I s ,C), which can be stochastic or deterministic depending on the type of I s and C. The conditioning C can take various forms, including image-based and text-based conditioning.

Image-based conditioning encompasses a wide range of tasks, such as image inpainting or scribble-based editing. Additionally, image-based virtual try-on, which involves inserting a clothing item onto an image of a person, can also be classified as image editing with imagebased conditioning.

Text-based conditioning involves editing an image based on a natural language prompt. This task was introduced in Figure 1.2 in the Introduction. With the advent of large-scale text-to-image generation models, we can expect image editing with text-based conditioning to achieve high levels of quality in the coming months or years.

Specific neural architectures for image editing

In this part, we describe two neural network architectures that are well suited for image editing tasks, namely U-Nets [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] and spatial transformers [START_REF] Jaderberg | Spatial transformer networks[END_REF]. Some tasks, such as virtual try-on, are based on the combination of these two architectures.

U-Nets

U-Nets, first introduced by [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], were initially designed for image segmentation but have proven to be particularly effective for image-to-image operations due to two key properties. Firstly, they incorporate multi-scale processing of the image, and secondly, they conserve high-resolution information. The first part of a U-Net consists of a series of convolutional blocks C i , which are intertwined with down-sampling operations D. Given an input image x, the hidden layers x 1 = C 1 (x), x 2 = C 2 (D(x 1 )), and x 3 = C 3 (D(x 2 )) are defined. The second part of the network involves convolutional blocks intertwined with up-sampling operations U, and concatenation of image tensors using the cat() function. The U-Net combines low-resolution and high-resolution information by employing the operation x 4 = C 4 cat x 2 ;U(x 3 ) , which is sequentially performed until reaching the input resolution of the image x.

U-Nets have proven to be crucial for the success of several deep image editing models, including CycleGAN [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] and virtual try-on models [START_REF] Han | Viton: An image-based virtual try-on network[END_REF]Wang et al., 2018a). The primary advantage of U-Nets is their strong inductive bias towards preserving finegrained, high-resolution details in images. This feature is particularly important for maintaining the quality of the source image, which is critical in image editing.

Spatial transformers for image warping

The Spatial Transformer Network [START_REF] Jaderberg | Spatial transformer networks[END_REF] is a module designed to enable differentiable image sampling. It allows spatial image manipulation, such as affine transformation, in a differentiable manner. Moreover, the parameters of the spatial transform are predicted by the neural network. Its primary purpose is to aid neural networks in learning invariance to complex geometric transformations, including translation, rotation, scaling, and warping. Originally Fig. 2.6 The spatial transformer module allows to spatially transform feature maps in a differentiable manner. From an input feature map U, the module predicts the parameters θ of a spatial transform, which is then transformed into a sampling grid T θ (G), where G is the regular base sampling grid. T θ (G) is then used to re-sample pixels from the input feature map U, which gives the output feature map V . Figure from [START_REF] Jaderberg | Spatial transformer networks[END_REF].

included in image classification and recognition networks, Spatial Transformer Networks have proven effective in enhancing the robustness of such networks to geometrical variations. An illustration of the Spatial Transformer Network module is presented in Figure 2.6. An important development of the Spatial Transformer Network concerns tasks that require attention mechanisms, such as geometric matching. In this context, Spatial Transformer Networks are learned to align two different images [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF].

For image editing, spatial transformer networks are particularly useful in tasks requiring object insertion. For example, in human pose transfer, it is used to warp the person features towards the target pose [START_REF] Dong | Soft-gated warping-gan for pose-guided person image synthesis[END_REF]. In the virtual try-on task, spatial transformer networks are leveraged to deform the clothing item before inserting it on the person image (Wang et al., 2018a).

Supervised approaches

In the following, we present various supervised techniques for training deep neural networks in image editing tasks. The common thread among these methods is the reliance on extensive labelled datasets, which both facilitate their success and constrain their applicability. We will see that, among these methods, there are different levels of supervision required, from fully paired to unpaired datasets. These approaches benefit from the extensive history of supervised learning with deep neural networks, which has led to the refinement of crucial components such as loss functions, architectures, and initialization. However, their effectiveness is limited by the unavailability or high cost of data collection for some image editing tasks.

Regression-based methods

One of the most direct approaches for learning image editing with deep neural networks is through regression-based methods. To illustrate this, let us consider a dataset consisting of a set of source images, X = (I 1 s , . . . , I N s ), and their corresponding target images, Y = (I 1 t , . . . , I N t ). A neural network, denoted as f θ , is trained on this dataset to minimize the following loss function:

L = 1 N N ∑ i=1 D f θ (I i s ), I i t (2.2.1)
Here, D is a function that measures the distance between images, and can be specified as L1 or L2 loss. Recent studies have shown that using perceptual loss [START_REF] Zhang | The unreasonable effectiveness of deep features as a perceptual metric[END_REF] can result in better-guided image editing or synthesis models. Perceptual loss involves computing the L2 distance in the feature space of a pre-trained deep neural network, typically the VGG network [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF].

As previously mentioned, this setting is often limited due to the requirement of collecting pairs of source and target images, (I i s , I i t ), which can be a challenging and time-consuming process. To address the issue of limited data, synthetic data creation can be employed in some cases. For instance, a synthetic dataset can be combined with a supervised regressionbased objective to achieve virtual try-on tasks [START_REF] Han | Viton: An image-based virtual try-on network[END_REF] or scribble-based editing [START_REF] Liu | Deflocnet: Deep image editing via flexible low-level controls[END_REF]. However, creating synthetic data typically requires careful design and significant engineering effort.

Cycle-based methods

Suppose we have two datasets, one consisting of apples and the other of oranges, and we wish to learn an image editing model that can transform an apple to an orange, and vice versa. Unfortunately, we do not have access to explicit pairs of source and target images that demonstrate these transformations, making it impossible to use a regression objective. One solution is to use cycle-based methods [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF], which leverage adversarial training and cycle-consistency to learn the mapping between two unpaired datasets.

The idea of cycle-based methods is illustrated in Figure 2.7, and has been applied to domain adaptation [START_REF] Hoffman | Cycada: Cycle-consistent adversarial domain adaptation[END_REF], unsupervised object insertion [START_REF] Zhan | Spatial fusion gan for image synthesis[END_REF], and virtual try-on tasks [START_REF] Ge | Disentangled cycle consistency for highly-realistic virtual try-on[END_REF].

Conditional distribution modelling

If we have access to pairs of data and the target function f ⋆ (I s ,C) is stochastic, the most successful approach is typically to model the conditional distribution p θ (I t |I s ,C), where I t is Fig. 2.7 The Cycle GAN approach consists in combining an adversarial objective with a cycle-consistency constraint to learn a mapping between unpaired datasets. The adversarial loss enforces G to map an image I s from domain X to an image with properties of domain Y , while F maps from X to Y . The cycle-consistency constraint enforces the mapping to keep information about the original image I s . Figure from [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF].

the target image, I s is the source image, and C is the conditioning. Standard deep generative modeling techniques, such as GANs or auto-regressive models, can be used to model conditional distributions. Depending on the approach and base architecture, incorporating conditions into the modeled distribution p θ may require more or less adaptation. For example, when using auto-regressive transformers, the conditioning can be simply concatenated to the input sequence. In contrast, when using GANs and a style-based generator [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF], a more careful adaptation of the generator and discriminator is often necessary. For instance, Co-Modulated GANs [START_REF] Zhao | Large scale image completion via co-modulated generative adversarial networks[END_REF] present an adaptation of StyleGAN for image inpainting.

This approach can be very effective for tasks that fulfill two conditions: 1) the ease of data collection; and 2) the stochastic nature of the target function f ⋆ (I s ,C). The image inpainting task satisfies both of these conditions, and state-of-the-art results have been achieved through conditional distribution modelling. Some approaches rely on conditional GANs, such as Co-Modulated GANs [START_REF] Zhao | Large scale image completion via co-modulated generative adversarial networks[END_REF]. Others use conditional auto-regressive models, as proposed in [START_REF] Peng | Generating diverse structure for image inpainting with hierarchical vq-vae[END_REF]. More recently, some approaches have been based on diffusion models [START_REF] Lugmayr | Repaint: Inpainting using denoising diffusion probabilistic models[END_REF].

Unsupervised approaches: leveraging pre-trained generative models

In unsupervised approaches, a prevalent strategy is to rely on pre-trained generative models and adopt an inverse problem viewpoint. The fundamental assumption is that the source images I s in most editing tasks are perturbed and noisy. For example, in the image inpainting task, I s corresponds to a realistic image I that has been distorted by zeroing out an area defined by a binary mask m. Consequently, we have I s = I ⊙ m, where ⊙ denotes the element-wise product.

The underlying idea can be cast under the Maximum A Posteriori view:

I t = arg max x p(x| I s ) = arg max x p(I s |x)p(x) (2.2.2)
where the prior distribution p(x) is estimated with a deep generative model, and the conditional distribution p(I s |x) is approximated with a distance between the two images x and I s .

In GANs, the inversion of edited images I s into the latent space of a pre-trained GAN can be accomplished by minimizing the distance D between the generated image G θ (z) and the edited image I s :

I t = G θ (z ⋆ ) where z ⋆ = arg min z D(G θ (z), I s ) (2.2.3)
The optimal latent code z ⋆ can be obtained through optimization or by using an encoder network that maps images to their corresponding latent codes. These two methods can be combined, as proposed in the seminal work of GANs' inversion by [START_REF] Zhu | Generative visual manipulation on the natural image manifold[END_REF]. This generic approach has been adapted to new generator architectures like StyleGAN, as demonstrated in subsequent works. For instance, [START_REF] Abdal | Image2stylegan: How to embed images into the stylegan latent space[END_REF][START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF] Recently, the concept of inverting generative models has also proven to be effective in the context of diffusion and score-based models. The work by [START_REF] Kawar | Denoising diffusion restoration models[END_REF] has demonstrated the efficacy of pre-trained diffusion probabilistic models in solving inverse problems, such as deblurring or inpainting. In a similar vein, [START_REF] Meng | SDEdit: Guided image synthesis and editing with stochastic differential equations[END_REF] has shown that pre-trained generative models can facilitate image editing tasks. By judiciously controlling the noise that is added to the image, these models can convert sketches into realistic images, perform image inpainting or scribble-based editing using mask guided Langevin dynamics. Despite the success of diffusion models in solving inverse problems, their long sampling procedure is a significant limitation, requiring many forward passes on a neural network. [START_REF] Chung | Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction[END_REF] have endeavored to reduce the number of iterations required in inverse problems, which remains a challenging task. Finally, [START_REF] Couairon | Diffedit: Diffusion-based semantic image editing with mask guidance[END_REF] propose a text-based editing method based on diffusion models. Interestingly, it removes the burden of giving a mask as input to the editing algorithm by automatically generating a mask.

Attribute editing after inversion. Another approach for image editing is to combine image inversion with latent space traversal. The first step involves a standard inversion procedure, which can be formulated as follows:

Ĩs = G θ (z ⋆ ) where z ⋆ = arg min z D(G θ (z), I s ) (2.2.4)
Here, Ĩs is an approximation of the source image I s generated by the GAN. The primary objective of this procedure is to derive z ⋆ , which corresponds to the latent vector associated with Ĩs . This vector z ⋆ can then be utilized to perform attribute editing by following specific directions in the latent space. Previous studies have demonstrated that the latent space of GANs contains interpretable and disentangled control directions [START_REF] Voynov | Unsupervised discovery of interpretable directions in the gan latent space[END_REF][START_REF] Härkönen | Ganspace: Discovering interpretable gan controls[END_REF]. Consequently, the inversion process can be integrated with attribute modifications in GANs' latent space, leading to straightforward image attribute editing procedures [START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF]. Figure 2.9 illustrates the outcomes of such a procedure. Finally, [START_REF] Grechka | Magecally invert images for realistic editing[END_REF] design a method that provides a tradeoff between reconstruction quality and editability of the latent vector.

The task of attribute editing can be facilitated through the natural language interaction between a user and an editing system. Notably, recent research has proposed utilizing the CLIP model [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF], which evaluates the alignment between text and image, in combination with text-to-image diffusion models [START_REF] Kim | Diffusionclip: Text-guided diffusion models for robust image manipulation[END_REF], or with unconditional StyleGAN [START_REF] Patashnik | Styleclip: Text-driven manipulation of stylegan imagery[END_REF]. Interestingly, [START_REF] Couairon | Flexit: Towards flexible semantic image translation[END_REF] encodes and manipulates images in the latent space of CLIP. These approaches have demonstrated promising results in enabling users to control and manipulate various attributes of an image, such as changing its background or altering its facial expressions, by providing textual descriptions.

Chapter 3

Learning multi-modal distributions with deep generative models

Introduction

In this chapter, our focus is on the learning of multi-modal target distributions with deep generative models, specifically, with a class of deep generative models called "push-forward generative models". These models encompass a broad range of deep generative models, including GANs, VAEs, and normalizing flows, but exclude auto-regressive models. Pushforward generative models represent a distribution p θ as the push-forward of a latent distribution γ by a neural network G θ , which is denoted as p θ = G θ ♯γ. In this chapter, we demonstrate that push-forward generative models are susceptible to misspecifications when learning multi-modal target distributions. When employing standard design choices of latent distribution and neural network, the hypothesis family of generative distributions p θ exclusively comprises connected distributions. Consequently, when attempting to cover all modes of the target distribution, the generative distribution necessarily samples in between the target modes, which raises several fundamental questions. For instance, what is the best achievable performance for push-forward generative models? How can we optimize the structure of the latent space such that low-quality points are present only in small proportion? And how can we develop principled rejection mechanisms that enhance pre-trained models?

In Section 3.2, we present an upper bound on the precision of push-forward generative models when the target distribution is made of disconnected modes. This result is derived from the Gaussian isoperimetric inequality, which states that among all sets with a given Gaussian measure, half-spaces have the minimal Gaussian perimeter or Gaussian boundary measure. Furthermore, we leverage this analysis to develop a truncation method that enhances pre-trained GANs by removing samples where the generator has a high Jacobian Frobenius norm.

In Section 3.3, we expand upon our analysis and demonstrate the existence of an optimal geometry for the latent space. This optimal structure, referred to as a "simplicial cluster", is based on a recent mathematical breakthrough by [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF] that solves the Gaussian isoperimetric problem for partitions with more than two subsets. This result enables us to demonstrate that the simplicial cluster structure is also optimal for push-forward generative models learning disconnected target distributions. Furthermore, we propose a way to enforce this structure in the latent space of GANs and show that it leads to improved performance.

In Section 3.4, we present a novel method for learning a rejection mechanism in pre-trained GANs. This method is based on adversarial learning of importance weights. Modelling importance weights requires respecting certain constraints on the neural network's output. To enforce these constraints, we introduce regularization terms to the objective function. We provide two algorithms for sampling from these importance weights, which can be combined. We conduct experiments to demonstrate that this approach outperforms other learning-based post-processing methods for pre-trained GANs.

Learning disconnected manifolds: a no GAN's land

Abstract. Typical architectures of push-forward generative models make use of a unimodal latent/input distribution transformed by a continuous generator. This includes GANs, VAEs and normalizing flows. Consequently, the modeled distribution always has connected support which is cumbersome when learning a disconnected set of manifolds. We formalize this problem by establishing a "no free lunch" theorem for the disconnected manifold learning stating an upper-bound on the precision of the targeted distribution. This is done by building on the necessary existence of a low-quality region where the generator continuously samples data between two disconnected modes. Finally, we derive a rejection sampling method based on the norm of generator's Jacobian and show its efficiency on several generators including BigGAN.

Introduction

Push-forward generative models such as GANs [START_REF] Goodfellow | Generative adversarial nets[END_REF] provide a very effective tool for the unsupervised learning of complex probability distributions. For example, [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF] generate very realistic human faces while [START_REF] Yu | Seqgan: Sequence generative adversarial nets with policy gradient[END_REF] match state-ofthe-art text corpora generation. Despite some early theoretical results on the stability of GANs (Arjovsky and Bottou, 2017) and on their approximation and asymptotic properties [START_REF] Biau | Some theoretical properties of GANs[END_REF], their training remains challenging. More specifically, GANs raise a mystery formalized by [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF]: how can they fit disconnected manifolds when they are trained to continuously transform a unimodal latent distribution? While this question remains widely open, we will show that studying it can lead to some improvements in the sampling quality of GANs. Indeed, training a GAN with the objective of continuously transforming samples from an unimodal distribution into a disconnected requires balancing between two caveats. On one hand, the generator could just ignore all modes but one, producing a very limited variety of high quality samples: this is an extreme case of the well known mode collapse (Arjovsky and Bottou, 2017). On the other hand, the generator could cover the different modes of the target distribution and necessarily generates samples out of the real data manifold as previously explained by [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF].

As brought to the fore by [START_REF] Roth | Stabilizing training of generative adversarial networks through regularization[END_REF], there is a density mis-specification between the true distribution and the model distribution. Indeed, one cannot find parameters such that the model density function is arbitrarily close to the true distribution. To solve this issue, many empirical works have proposed to over-parameterize the generative distributions, as for instance, using a mixture of generators to better fit the different target modes. [START_REF] Tolstikhin | Adagan: Boosting generative models[END_REF] rely on boosting while [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] force each generator to target different sub-manifolds thanks to a criterion based on mutual information. Another direction is to add complexity in the latent space using a mixture of Gaussian distributions [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF].

To better visualize this phenomenon, we consider a simple 2D motivational example where the real data lies on two disconnected manifolds. Empirically, when learning the distribution, GANs split the Gaussian latent space into two modes, as highlighted by the separation line in red in Figure 3.1a. More importantly, each sample drawn inside this red area in Figure 3.1a is then mapped in the output space in between the two modes (see Figure 3.1b). For the quantitative evaluation of the presence of out-of-manifold samples, a natural metric is the Precision-Recall (PR) proposed by [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF] and its improved version (Improved PR) [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF]. A first contribution of this section is to formally link them. Then, taking advantage of these metrics, we lower bound the measure of this out-of-manifold region and formalize the impossibility of learning disconnected manifolds with standard GANs. We also extend this observation to the multi-class generation case and show that the volume of off-manifold areas increases with the number of covered manifolds. In the limit, this increase drives the precision to zero.

To solve this issue and increase the precision of GANs, we argue that it is possible to remove out-of-manifold samples using a truncation method. Building on the work of [START_REF] Arvanitidis | Latent space oddity: on the curvature of deep generative models[END_REF] who define a Riemaniann metric that significantly improves clustering in the latent space, our truncation method is based on information conveyed by the Jacobian's norm of the generator. We empirically show that this rejection sampling scheme enables us to better fit disconnected manifolds without over-parametrizing neither the generative class of functions nor the latent distribution. Finally, in a very large high dimensional setting, we discuss the advantages of our rejection method and compare it to the truncation trick introduced by [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF].

In a nutshell, our contributions are the following:

• We discuss evaluation of GANs and formally link the PR measure [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF] and its Improved PR version [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF].

• We upper bound the precision of GANs with Gaussian latent distribution and formalize an impossibility result for disconnected manifolds learning.

• Using toy datasets, we illustrate the behavior of GANs when learning disconnected manifolds and derive a new truncation method based on the Jacobian's Frobenius norm of the generator. We confirm its empirical performance on state-of-the-art models and datasets.

Related work

Fighting mode collapse. [START_REF] Goodfellow | Generative adversarial nets[END_REF] were the first to raise the problem of mode collapse in the learning of disconnected manifolds with GANs. They observed that when the generator is trained too long without updating the discriminator, the output distribution collapses to a few modes reducing the diversity of the samples. To tackle this issue, [START_REF] Salimans | Improved techniques for training GANs[END_REF]; [START_REF] Lin | Pacgan: The power of two samples in generative adversarial networks[END_REF] suggested feeding several samples to the discriminator. [START_REF] Srivastava | Veegan: Reducing mode collapse in gans using implicit variational learning[END_REF] proposed the use of a reconstructor network, mapping the data to the latent space to increase diversity. In a different direction, Arjovsky and Bottou (2017) showed that training GANs using the original formulation [START_REF] Goodfellow | Generative adversarial nets[END_REF] leads to instability or vanishing gradients. To solve this issue, they proposed a Wasserstein GAN architecture Arjovsky et al. (2017) where they restrict the class of discriminative functions to 1-Lipschitz functions using weight clipping. Pointing to issues with this clipping, [START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF]; [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF] proposed relaxed ways to enforce the Lipschitzness of the discriminator, either by using a gradient penalty or a spectral normalization. Albeit not exactly approximating the Wasserstein's distance [START_REF] Petzka | On the regularization of Wasserstein GANs[END_REF], both implementations lead to good empirical results, significantly reducing mode collapse. Building on all of these works, we will further assume that generators are now able to cover most of the modes of the target distribution, leaving us the problem of out-of-manifold samples (a.k.a. low-quality pictures).

Generation of disconnected manifolds. When learning complex manifolds in high dimensional spaces using deep generative models, [START_REF] Fefferman | Testing the manifold hypothesis[END_REF] highlighted the importance of understanding the underlying geometry. More precisely, the learning of disconnected manifold requires the introduction of disconnectedness in the model. [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF] used a multi-modal entry distribution, making the latent space disconnected, and showed better coverage when data is limited and diverse. Alternatively, [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] studied the learning of a mixture of generators. Using a mutual information term, they encourage each generator to focus on a different submanifold so that the mixture covers the whole support. This idea of using an ensemble of generators is also present in the work of [START_REF] Tolstikhin | Adagan: Boosting generative models[END_REF] and [START_REF] Zhong | Rethinking generative mode coverage: A pointwise guaranteed approach[END_REF], though they were primarily interested in the reduction of mode collapse.

In this section, we propose a truncation method to separate the latent space into several disjoint areas. It is a way to learn disconnected manifolds without relying on the previously introduced over-parameterization techniques. As our proposal can be applied without retraining the whole architecture, we can use it successfully on very larges nets. Close to this idea, [START_REF] Azadi | Discriminator rejection sampling[END_REF] introduced a rejection strategy based on the output of the discriminator. However, this rejection sampling scheme requires the discriminator to be trained with a classification loss while our proposition can be applied to any generative models.

Evaluating GANs. The evaluation of generative models is an active area of research. Some of the proposed metrics only measure the quality of the generated samples such as the Inception score [START_REF] Salimans | Improved techniques for training GANs[END_REF] while others define distances between probability distributions. This is the case of the Frechet Inception distance [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF], the Wasserstein distance Arjovsky et al. (2017) or kernel-based metrics [START_REF] Gretton | A kernel two-sample test[END_REF]. The other main caveat for evaluating GANs lies in the fact that one does not have access to the true density nor the model density, prohibiting the use of any density based metrics. To solve this issue, the use of a third network that acts as an objective referee is common. For instance, the Inception score uses outputs from InceptionNet while the Fréchet Inception Distance compares statistics of InceptionNet activations. Since our work focuses on out-of-manifold samples, a natural measure is the PR measure [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF] and its Improved PR version [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF], extensively discussed in the next section.

In the following, alongside precise definitions, we exhibit an upper bound on the precision of GANs with high recall (i.e. no mode collapse) and present a new truncation method.

Our approach

We start with a formal description of the framework of GANs and the relevant metrics. We later show a "no free lunch" theorem proving the necessary existence of an area in the latent space that generates out-of-manifold samples. We name this region the no GAN's land since any data point sampled from this area will be in the frontier in between two different modes. We claim that dealing with it requires special care. Finally, we propose a rejection sampling procedure to avoid points out of the true manifold.

Notations

In the original setting of Generative Adversarial Networks (GANs), one tries to generate data that are "similar" to samples collected from some unknown probability measure µ ⋆ . To do so, we use a parametric family of generative distribution where each distribution is the push-forward measure of a latent distribution γ and a continuous function modeled by a neural network.

Assumption 1 (γ Gaussian). The latent distribution γ is a standard multivariate Gaussian.

Note that for any distribution µ, S µ refers to its support. Assumption 1 is common for GANs as in many practical applications, the latent variable defined on a low dimensional space R d is either a multivariate Gaussian, either a uniform distribution on a compact.

The measure µ ⋆ is defined on a subset E of R D (potentially a highly dimensional space), equipped with the norm ∥ • ∥. The generator has the form of a parameterized class of functions from R d (a space with a much lower dimension) to E, say G = {G θ : θ ∈ Θ }, where Θ ⊆ R p is the set of parameters describing the model. Each function G θ thus takes input from a ddimensional space variable Z (Z is associated with probability distribution γ) and outputs "fake" observations with distribution µ θ . Thus, the class of probability measures P = {µ θ : θ ∈ Θ } is the natural class of distributions associated with the generator, and the objective of GANs is to find inside this class of candidates the one that generates the most realistic samples, closest to the ones collected from the unknown distribution µ ⋆ .

Assumption 2. Let L > 0. The generator G θ takes the form of a neural network whose Lipchitz constant is smaller than L, i.e. for all (z, z ′ ), we have

∥G θ (z ′ ) -G θ (z)∥ ⩽ L∥z -z ′ ∥.
This is a reasonable assumption, since [START_REF] Virmaux | Lipschitz regularity of deep neural networks: analysis and efficient estimation[END_REF] present an algorithm that upper-bounds the Lipschitz constant of deep neural networks. Initially, 1-Lipschitzness was enforced only for the discriminator by clipping the weigths Arjovsky et al. (2017), adding a gradient penalty [START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF]; [START_REF] Roth | Stabilizing training of generative adversarial networks through regularization[END_REF]; [START_REF] Petzka | On the regularization of Wasserstein GANs[END_REF], or penalizing the spectral norms [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF]. Nowadays, state-of-the-art architectures for large scale generators such as SAGAN Zhang et al. (2019) and BigGAN Brock et al. (2019) also make use of spectral normalization for the generator.

Evaluating GANs with Precision and Recall

When learning disconnected manifolds, [START_REF] Srivastava | Veegan: Reducing mode collapse in gans using implicit variational learning[END_REF] proved the need of measuring simultaneously the quality of the samples generated and the mode collapse. [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF] proposed the use of a PR metric to measure the quality of GANs. The key intuition is that precision should quantify how much of the fake distribution can be generated by the true distribution while recall measures how much of the true distribution can be re-constructed by the model distribution. More formally, it is defined as follows:

Definition 3.2.1. [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF] Let X,Y be two random variables. For α, β ∈ (0, 1], X is said to have an attainable precision α at recall β w.r.t. Y if there exists probability distributions

µ, ν X , ν Y such that Y = β µ + (1 -β )ν Y and X = α µ + (1 -α)ν X .
The component ν Y denotes the part of Y that is "missed" by X, whereas, ν X denotes the "noise" part of X. We denote ᾱ (respectively β ) the maximum attainable precision (respectively recall). Th. 1 of [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF] states:

X S Y = ᾱ and Y S X = β .
Improved PR metric. [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF] highlighted an important drawback of the PR metric proposed by [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF]: it cannot correctly interpret situations when a large numbers of samples are packed together. To better understand this situation, consider a case where the generator slightly collapses on a specific data point, i.e. there exists x ∈ E, µ θ (x) > 0. We show in Appendix A.1.1 that if µ ⋆ is a non-atomic probability measure and µ θ is highly precise (i.e. α = 1), then the recall β must be 0.

To solve these issues, [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF] proposed an Improved Precision-Recall (Improved PR) metric built on a nonparametric estimation of support of densities. Definition 3.2.2. [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF] Let X,Y be two random variables and D X , D Y two finite sample datasets such that D X ∼ X n and D Y ∼ Y n . For any x ∈ D X (respectively for any y ∈ D Y ), we consider (x (1) , . . . , x (n-1) ), the re-ordening of elements in D X \ x given their euclidean distance with x. For any k ∈ N and x ∈ D X , the precision α n k (x) of point x is defined as:

α n k (x) = 1 ⇐⇒ ∃y ∈ D Y , ∥x -y∥ ⩽ ∥y (k) -y∥.
Similarly, the recall β n k (y) of any given y ∈ D Y is:

β n k (y) = 1 ⇐⇒ ∃x ∈ D X , ∥y -x∥ ⩽ ∥x (k) -x∥.
Improved precision (respectively recall) are defined as the average over D X (respectively D Y ) as follows:

α n k = 1 n ∑ x i ∈D X α n k (x i ) β n k = 1 n ∑ y i ∈D Y β n k (y i ).
A first contribution is to formalize the link between PR and Improved PR with the following theorem:

Theorem 3.2.1. Let X,Y two random variables with probability distributions µ and ν. Assume that both µ and ν are associated with uniformly continuous probability density functions f µ and f ν . Besides, there exists constants a 1 > 0, a 2 > 0 such that for all x ∈ E we have a

1 < f µ ⋆ (x) ⩽ a 2 and a 1 < f µ θ (x) ⩽ a 2 for some c > 0. Also, (k, n) are such that k log(n) → +∞ and k n → 0. Then, α n k → ᾱ in probability and β n k → β in probability.
This theorem, whose proof is delayed to Appendix A.1.2, underlines the nature of the Improved PR metric: the metric compares the supports of the modeled probability distribution µ θ and of the true distribution µ ⋆ . This means that Improved PR is a tuple made of both maximum attainable precision ᾱ and recall β (e.g. Theorem 1 of [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF]). As Improved PR is shown to have a better performance evaluating GANs sample quality, we use this metric for both the following theoretical results and experiments.

Learning disconnected manifolds

In this section, we aim to stress the difficulties of learning disconnected manifolds with standard GANs architectures. To begin with, we recall the following lemma.

Lemma 3.2.1. Assume that Assumptions 1 and 2 are satisfied. Then, for any θ ∈ Θ , the support S µ θ is connected.

There is consequently a discrepancy between the connectedness of S µ θ and the disconnectedness of S µ ⋆ . In the case where the manifold lays on two disconnected components, our next theorem exhibit a no free lunch theorem: Theorem 3.2.2. ("No free lunch" theorem) Assume that Assumptions 1 and 2 are satisfied. Assume also that true distribution µ ⋆ lays on two equally measured disconnected manifolds distant from a distance D > 0. Then, any estimator µ θ that samples equally in both modes must have a precision ᾱ such that ᾱ

+ D √ 2πL e -Φ -1 ( ᾱ 2 ) 2 2 ⩽ 1, where Φ is the c.d.f. of a standard normal distribution. Besides, if ᾱ ⩾ 3/4, ᾱ ≲ 1 -2 π W ( D 2 4L 2 )
where W is the Lambert W function.

The proof of this theorem is delayed to Appendix A.1.3. It is mainly based on the Gaussian isoperimetric inequality [START_REF] Borell | The brunn-minkowski inequality in gauss space[END_REF][START_REF] Sudakov | Extremal properties of half-spaces for spherically invariant measures[END_REF] that states that among all sets of given Gaussian measure in any finite dimensional Euclidean space, half-spaces have the minimal Gaussian boundary measure. If in Figure 3.1, the generator has thus learned the optimal separation, it is not known clear how to enforce such geometrical properties in the latent space. We will show, in the next section, a method that indeed enforces a well-structured latent space.

In real world applications, when the number of distinct sub-manifolds increases, we expect the volume of these boundaries to increase with respect to the number of different classes covered by the modeled distribution µ θ . Going in this direction, we better formalize this situation, and show an extended "no free lunch theorem" by expliciting an upper-bound of the precision ᾱ in this broader framework.

Assumption 3. The true distribution µ ⋆ lays on M equally-measured disconnected components at least distant from some constant D > 0. This is likely to be true for datasets made of symbol designed to be highly distinguishable (e.g. digits in the MNIST dataset). In very high dimension, this assumption also holds for complex classes of objects appearing in many different contexts (e.g. the bubble class in ImageNet, see Appendix).

To better apprehend the next theorem, note A m the pre-image in the latent space of mode m and A r m its r-enlargement:

A r m := {z ∈ R d | dist(z, A m ) ≤ r}, r > 0.
Theorem 3.2.3. (Generalized "no free lunch" theorem) Assume that Assumptions 1, 2, and 3 are satisfied, and that the pre-image enlargements A ε m , with ε = D 2L , form a partition of the latent space with equally measured elements.

Then, any estimator µ θ with recall β > 1 M must have a precision ᾱ at most 1+x 2

x 2 e -1 2 ε 2 e -εx where x = Φ -1 (1 -1 β M ) and Φ is the c.d.f. of a standard normal distribution.

Theorem 3.2.3, whose proof is delayed to Appendix A.1.4, states a lower-bound the measure of samples mapped out of the true manifold. We expect our bound to be loose since no theoretical results are known, to the best of our knowledge, on the geometry of the separation that minimizes the boundary between different classes (when M ⩾ 3). Finding this optimal cut would be an extension of the honeycomb theorem [START_REF] Hales | The honeycomb conjecture[END_REF]. In Appendix A.1.4.2 we give a more technical statement of Theorem 3.2.3 without assuming equality of measure of the sets A ε m .

The idea of the proof is to consider the border of an individual cell with the rest of the partition. It is clear that at least half of the frontier will be inside this specific cell. Then, to get to the final result, we sum the measures of the frontiers contained inside all of the different cells. Remark that our analysis is fine enough to keep a dependency in M which translates into a maximum precision that goes to zero when M goes to the infinity and all the modes are covered. More precisely, in this scenario where all pre-images have equal measures in the latent space, one can derive the following bound, when the recall β is kept fixed and M increases:

ᾱ M→∞ ⩽ e -1 2 ε 2 e -ε √ 2 log( β M) where ε = D 2L . (3.2.1)
For a fixed generator, this equation illustrates that the precision ᾱ decreases when either the distance D (equivalently ε) or the number of classes M increases. For a given ε, ᾱ converges to 0 with a speed O( 1

( β M) √ 2ε
). To better illustrate this asymptotic result, we provide results from a 2D synthetic setting. In this toy dataset, we control both the number M of disconnected manifolds and the distance D.

Jacobian-based truncation (JBT) method

The analysis of the deformation of the latent space offers a grasp on the behavior of GANs. For instance, [START_REF] Arvanitidis | Latent space oddity: on the curvature of deep generative models[END_REF] propose a distance accounting for the distortions made by the generator. For any pair of points (z 1 , z 2 ) ∼ Z 2 , the distance is defined as the length of the geodesic

d(z 1 , z 2 ) = [0,1] ∥J G θ (γ t ) dγ t
dt ∥dt where γ is the geodesic parameterized by t ∈ [0, 1] and J G θ (z) denotes the Jacobian matrix of the generator at point z. Authors have shown that the use of this distance in the latent space improves clustering and interpretability. We make a similar observation that the generator's Jacobian Frobenius norm provides meaningful information.

Indeed, the frontiers highlighted in Figures 3.2a, 3.2b, and 3.2c correspond to areas of low precision mapped out of the true manifold: this is the no GAN's land. We argue that when learning disconnected manifolds, the generator tries to minimize the number of samples that do not belong to the support of the true distribution and that this can only be done by making paths steeper in the no GAN's land. Consequently, data points G θ (z) with high Jacobian Frobenius norm (JFN) are more likely to be outside the true manifold. To improve the precision of generative models, we thus define a new truncation method by removing points with highest JFN.

However, note that computing the generators's JFN is expensive to compute for neural networks, since being defined as follows,

∥J G θ (z)∥ 2 F = m ∑ i=1 n ∑ j=1 ∂ G θ (z) i ∂ z j 2 ,
it requires a number of backward passes equal to the output dimension. To make our truncation method tractable, we use a stochastic approximation of the Jacobian Frobenius norm based on the following result from [START_REF] Rifai | Higher order contractive auto-encoder[END_REF]:

∥J G θ (z)∥ 2 = lim N→∞ σ →0 1 N N ∑ ε i 1 σ 2 ∥G θ (z + ε i ) -G θ (z)∥ 2 ,
where ε i ∼∼ N (0, σ 2 I and I is the identity matrix of dimension d. The variance σ of the noise and the number of samples are used as hyper-parameters. In practice, σ in [1e-4; 1e-2] and N = 10 give consistent results.

Based on the preceding analysis, we propose a new Jacobian-based truncation (JBT) method that rejects a certain ratio of the generated points with highest JFN. This truncation ratio is considered as an hyper-parameter for the model. We show in our experiments that our JBT can be used to to detect samples outside the real data manifold and that it consequently improves the precision of the generated distribution as measured by the Improved PR metric.

Experiments

In the following, we show that our truncation method, JBT, can significantly improve the performances of generative models on several models, metrics and datasets. Furthermore, we compare JBT with over-parametrization techniques specifically designed for disconnected manifold learning. We show that our truncation method reaches or surpasses their performance, while it has the benefit of not modifying the training process of GANs nor using a mixture of generators, which is computationally expensive. Finally, we confirm the efficiency of our method by applying it on top of BigGAN [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF].

Except for BigGAN, for all our experiments, we use Wasserstein GAN with gradient penalty [START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF], called WGAN for conciseness. We give in Appendix A.3 the full details of our experimental setting. The use of WGAN is motivated by the fact that it was shown to stabilize the training and significantly reduce mode collapse (Arjovsky and Bottou, 2017). However, we want to emphasise that our method can be plugged on top of any generative model fitting disconnected components.

Evaluation metrics

To measure performances of GANs when dealing with low dimensional applications -as with synthetic datasets -we equip our space with the standard Euclidean distance. However, for high dimensional applications such as image generation, [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]; [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF] have shown that embedding images into a feature space with a pre-trained convolutional classifier provides more semantic information. In this setting, we consequently use the euclidean distance between the images' embeddings from a classifier. For a pair of images (a, b), we define the distance d(a, b) as d(a, b) = ∥φ (a)φ (b)∥ 2 where φ is a pre-softmax layer of a supervised classifier, trained specifically on each dataset. Doing so, they will more easily separate images sampled from the true distribution µ ⋆ from the ones sampled by the distribution µ θ .

We compare performances using Improved PR [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF]. We also report the Marginal Precision which is the precision of newly added samples when increasing the ratio of kept samples. Besides, for completeness, we report FID [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF] and recall precise definitions in Appendix A.2.2. Note that FID was not computed with InceptionNet, but a classifier pre-trained on each dataset.

Synthetic dataset

We first consider the true distribution to be a 2D Gaussian mixture of 9 components. Both the generator and the discriminator are modeled with feed-forward neural networks.

Interestingly, the generator tries to minimize the sampling of off-manifolds data during training until its JFN gets saturated (see Appendix A.2.3). One way to reduce the number of off-manifold samples is to use JBT. Indeed, off-manifold data points progressively disappear when being more and more selective, as illustrated in Figure 3.3c. We quantitatively confirm that our truncation method (JBT) improves the precision. On Figure 3.3d, we observe that keeping the 70% of lowest JFN samples leads to an almost perfect precision of the support of the generated distribution. Thus, off-manifold samples are in the 30% samples with highest JFN.

Image datasets

We further study JBT on three different datasets: MNIST (LeCun et al., 1998), FashionMNIST [START_REF] Xiao | Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms[END_REF] and CIFAR10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF]. Following [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] implementation, we use a standard CNN architecture for MNIST and FashionMNIST while training a ResNet-based model for CIFAR10 [START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF].

Figure 3.4 highlights that JBT also works on high dimensional datasets as the marginal precision plummets for high truncation ratios. Furthermore, when looking at samples ranked by increasing order of their JFN, we notice that samples with highest JFN are standing in-between manifolds. For example, those are ambiguous digits resembling both a "0" and a "6" or shoes with unrealistic shapes. To further assess the efficiency of our truncation method, we also compare its performances with two state-of-the-art over-parameterization techniques that were designed for disconnected manifold learning. First, [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF] propose DeliGAN, a reparametrization trick to transform the unimodal Gaussian latent distribution into a mixture. The different mixture components are later learnt by gradient descent. For fairness, the re-parametrization trick is used on top of WGAN. Second, [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] define DMLGAN, a mixture of generators to better learn disconnected manifolds. In this architecture, each generator is encouraged to target a different submanifold by enforcing high mutual information between generated samples and generator's ids. Keep in mind that for DeliGAN (respectively DMLGAN), the optimal number of components (respectively generators) is not known and is a hyper-parameter of the model that has to be cross-validated.

The results of the comparison are presented in Table 3.1. In both datasets, JBT 80 % outperforms DeliGAN and DMLGAN in terms of precision while keeping a reasonnable recall. This confirms our claim that over-parameterization techniques are unnecessary. As noticed 2019)'s truncation trick and our truncation method (JBT), on three ImageNet classes generated by BigGAN. We show better results on complex and disconnected classes (e.g. bubble). Reported confidence intervals are 97% confidence intervals. On the second row, generated samples ordered by their JFN (left to right, top to bottom). We observe a concentration of off-manifold samples for images on the bottom row, confirming the soundness of JBT.

by [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF], we also observe that FID does not correlate properly with the Improved PR metric. Based on the Frechet distance, only a distance between multivariate Gaussians, we argue that FID is not suited for disconnected manifold learning as it approximates distributions with unimodal ones and looses many information.

Spurious samples rejections on BigGAN

Thanks to the simplicity of JBT, we can also apply it on top of any trained generative model.

In this subsection, we use JBT to improve the precision of a pre-trained BigGAN model [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF], which generates class-conditionned ImageNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] samples. The class-conditioning lowers the problem of off-manifold samples, since it reduces the disconnectedness in the output distribution. However, we argue that the issue can still exist on high-dimensional natural images, in particular complex classes can still be multi-modal (e.g. the bubble class). The bottom row in Figure 3.5 shows a random set of 128 images for three different classes ranked by their JFN in ascending order (left to right, top to bottom). We observe a clear concentration of spurious samples on the bottom row images.

To better assess the Jacobian based truncation method, we compare it with the truncation trick from [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]. This truncation trick aims to reduce the variance of the latent space distribution using truncated Gaussians. While easy and effective, this truncation has some issues: it requires to complexify the loss to enforce orthogonality in weight matrices of the network. Moreover, as explained by [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF] "only 16% of models are amenable to truncation, compared to 60% when trained with Orthogonal Regularization". For fairness of comparison, the pre-trained network we use is optimized for their truncation method. On the opposite, JBT is simpler to apply since 100% of the tested models were amenable to the proposed truncation.

Results of this comparison are shown in the upper row of Figure 3.5. Our method can outperform their truncation trick on difficult classes with high intra-class variation, e.g. bubble and house finch. This confirms our claim that JBT can detect outliers within a class. However, one can note that their trick is particularly well suited for simpler unimodal classes, e.g. parachute and reaches high precision levels.

Conclusion

In this section, we provide insights on the learning of disconnected manifolds with push-forward generative models. Our analysis shows the existence of an off-manifold area with low precision. We empirically show on several datasets and GAN models that we can detect these areas and remove samples located in between two modes thanks to a newly proposed truncation method.

However, we do not have yet a clear idea of the geometrical structure of a generator's latent space. When there are two modes in the target distribution, it is clear that the best way to split the latent space is to use half-spaces (according to the standard Gaussian isoperimetric inequality). But when there are more than two modes, what is an optimal geometrical structure? In the next section, we provide an explicit characterization of an optimal latent space geometry. Moreover, we investigate if this structure is indeed learned by generative models, and if it is possible to enforce it.

Unveiling the latent space geometry of push-forward generative models

Abstract. Many deep generative models are defined as a push-forward of a Gaussian measure by a continuous generator, such as Generative Adversarial Networks (GANs) or Variational Auto-Encoders (VAEs). This work explores the latent space of such deep generative models. A key issue with these models is their tendency to output samples outside of the support of the target distribution when learning disconnected distributions. We investigate the relationship between the performance of these models and the geometry of their latent space. Building on recent developments in geometric measure theory, we prove a sufficient condition for optimality in the case where the dimension of the latent space is larger than the number of modes. Through experiments on GANs, we demonstrate the validity of our theoretical results and gain new insights into the latent space geometry of these models. Additionally, we propose a truncation method that enforces a simplicial cluster structure in the latent space and improves the performance of GANs.

Introduction

models. Since the modeled distribution is connected, some areas of its support are necessarily mapped outside the true data distribution. However, when covering several modes of a disconnected distribution, generators still try to minimize the numbers of samples lying outside the true modes (e.g. the purple area on the right of Figure 3.6). In other words, generators aim at minimizing the measure of the existing borders between the modes in the latent space. Considering a Gaussian latent space, finding such minimizers is closely linked to Gaussian isoperimetric inequalities [START_REF] Ledoux | Isoperimetry and gaussian analysis[END_REF] where the goal is to derive the partitions that split a Gaussian space with minimal Gaussian-weighted perimeters. Most notably, a recent result [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF] shows that, as long as the number of components m in the partition and the number of dimensions d of the Gaussian space are such that m ≤ d + 1, the optimal partition is a 'simplicial cluster': a Voronoi diagram with equidistant seeds, see left of Figure 3.6 for m = 3 and d = 3.

In this section, we demonstrate the effectiveness of applying simplicial clusters to the latent space of push-forward generative models. We show both experimentally and theoretically that generators with a latent space structured as a simplicial cluster minimize the occurrence of out-of-distribution generated samples. Using the precision metric [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF][START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF], we show that generators with a simplicial cluster latent space achieve optimal precision levels and provide both an upper and a lower bound on their precision. Our experiments reveal that GANs with higher performances tend to organize their latent space as simplicial clusters. More importantly, we illustrate that enforcing this 'simplicial structure' with a truncation method can boost GANs' performance. Interestingly, simplicial clusters are highly similar to the 'simplex Equiangular Tight Frames' observed in the last-layer features of deep classification networks [START_REF] Papyan | Prevalence of neural collapse during the terminal phase of deep learning training[END_REF]. This study stresses that they also naturally emerge in deep push-forward generative models. Our contributions are the following:

• We are the first to build on the latest results from Gaussian isoperimetric inequalities by [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF] in the study and understanding of push-forward generative models.

• We present a new theoretical analysis, providing both an upper bound on the precision of push-forward generative models. We demonstrate that generators with a latent space organized as a simplicial cluster have an optimal precision, with lower bounds that decrease in √ m log m, where m is the number of modes.

• Experimentally, we verify that GANs tend to structure their latent space as simplicial clusters' by exploring two properties of the latent space: linear separability and convexity of classes. Also, we analyse the impact of latent space dimension on GANs, and reveal a positive correlation between GANs' performance and latent space geometry. • Finally, we show that enforcing a simplicial structure into GANs' latent space can boost their performance and outperforms other boosting methods.

Related Work

Notation

Data. We consider a target distribution µ ⋆ defined on a Euclidean space R D , which may be a high-dimensional space, and equipped with the Euclidean norm ∥ • ∥. We use S µ to represent the support of any distribution µ.

Push-forward generative models. We consider the set of L-Lipschitz continuous functions, denoted as G L , from the latent space R d to the high-dimensional space R D . The primary goal of each generator in this set is to produce realistic samples. The distribution in the latent space, defined on R d , is assumed to be Gaussian and is represented as γ. For each generator G ∈ G L , we associate the push-forward distribution (or image distribution) of γ by G, and denote it G♯γ, where ♯ denotes the push-forward operator. In the context of generative models, each distribution G♯γ is now a candidate distribution to represent µ ⋆ .

The Lipschitzness assumption on G L is reasonnable: Virmaux and Scaman (2018) have shown the lipschitzness of deep neural networks, and have developed an algorithm that can upper-bound their Lipschitz constant. While deep neural networks can have high Lipschitz constants, it is possible to constrain this in practice by techniques such as clipping the neural network's parameters (Arjovsky et al., 2017), penalizing the discriminative functions' gradient [START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF][START_REF] Kodali | On convergence and stability of GANs[END_REF][START_REF] Wei | Improving the improved training of wasserstein gans: A consistency term and its dual effect[END_REF][START_REF] Zhou | Lipschitz generative adversarial nets[END_REF], or penalizing the spectral norms [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF]. Large-scale generators such as SAGAN [START_REF] Zhang | Self-attention generative adversarial networks[END_REF] and BigGAN [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]) also make use of spectral normalization for the generator.

Generative models and disconnected distributions

The phenomenon of misspecification in continuous generative models, while primarily studied in the context of GANs, is also relevant to other families such as VAEs or normalizing flows [START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF]. This issue has been investigated both experimentally [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] and theoretically [START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF]. The problem stems from a fundamental tradeoff: continuous generators can either cover all modes, resulting in out-of-manifold samples, or generate only high-quality samples, neglecting some modes. To address this, various methods have been proposed, such as training disconnected distributions [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF][START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] or deriving rejection mechanisms from pre-trained generators [START_REF] Azadi | Discriminator rejection sampling[END_REF][START_REF] Humayun | Polarity sampling: Quality and diversity control of pre-trained generative networks via singular values[END_REF].

Empirical studies have provided valuable insights into the structure of the latent space of generative models. For example, [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF] demonstrate that binary attributes are linearly separable in the Gaussian latent space and even more separable in an intermediate latent space. Similarly, [START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF] find that face attributes are separated by hyperplanes in the latent space. [START_REF] Arvanitidis | Latent space oddity: on the curvature of deep generative models[END_REF] and Chen et al. (2018a) view the latent space of generative models with a Riemannian perspective.

While these findings provide valuable insights into the latent space structure of generative models, they may not be sufficient for a comprehensive understanding of the latent space geometry. For instance, In section 3.2, we stress the relevance of this problem by showing that the precision of GANs can converge to 0 when the number of modes or the distance between them increases. In this section, we take a step towards a deeper understanding of the behavior of push-forward generative models and reveal an optimal latent space configuration when the number of modes m and the dimension of the latent space d are such that m ≤ d + 1.

Evaluating generative models

When learning disconnected manifolds, [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF] illustrated the need for measures that simultaneously evaluate both the quality (Precision), and the diversity (Recall) of the generated samples. However, [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF] pointed out an important limitation of the PR metric: it cannot accurately interpret situations when large numbers of samples are packed together. They propose an Improved PR metric based on the non-parametric estimation of manifolds to correct this.

Improved PR metric. Informally, for a generator G, precision (α G ) quantifies the proportion of generated samples that can be approximated with true samples, while recall (β G ) measures the proportion of true samples that can be approximated with generated ones. Applying this to GANs, using the target distribution µ ⋆ and modeled distribution G♯γ, the Improved PR metric was shown, by Theorem 3.2.1, to be asymptotically equivalent to:

α n G → n→∞ α G = G♯γ S µ ⋆ and β n G → n→∞ β G = µ ⋆ S G♯γ ,
where S µ ⋆ denotes the support of µ ⋆ and n is the number of samples. However, [START_REF] Naeem | Reliable fidelity and diversity metrics for generative models[END_REF] have shown that the Improved PR metric [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF] is sensitive to outlier samples of both the target and the generated distribution. To correct this and fix the overestimation of the manifold around real outliers, [START_REF] Naeem | Reliable fidelity and diversity metrics for generative models[END_REF] propose the Density/Coverage metric.

Density/Coverage. Instead of counting how many fake samples belong to a real sample neighborhood, density counts how many real sample neighborhoods contain a generated sample. On the other hand, coverage counts the number of real sample neighborhoods that contain at least one fake sample.

In the next analysis both theoretical and experimental, we use both notions of precision and density defined above.

Simplicial Structure in Push-Forward Generative Models

The goal is to gain a deeper understanding of the latent space of push-forward generative models and identify which ones possess the highest precision under certain conditions. As previously mentioned, push-forward generative models map a unimodal Gaussian distribution γ through a Lipschitz-continuous function, represented by a generator G. As a result, the modeled generative distribution G♯γ necessarily has a connected support.

In cases where the target distribution µ ⋆ contains disconnected manifolds, generators have to generate fake data points that fall outside of the true manifold. This prompts the question: given that a generator samples data points from each of the distinct modes, what is the maximum precision that it can achieve? To begin with, let's consider a target distribution µ ⋆ composed of m disconnected modes.

Assumption 4 (Disconnected manifolds). The target distribution µ ⋆ consists of m disconnected spheres S i , i ∈ [1, m] of equal measure (with centers X i and radius r i ). Additionally, the spheres satisfy the two following properties:

• Small individual radius: each radius r i satisfies

r i < min j ∥X i -X j ∥ 2 . (3.3.1)
• Each distance ∥X i -X j ∥ satisfies:

min k∈[1,m],k̸ =i, j ∥(X i + X j )/2 -X k ∥ > ∥X i -X j ∥ 2 . (3.3.2)
We believe that the assumption of disconnectedness is a reasonable one, particularly for multi-class datasets such as MNIST LeCun et al. (1998), CIFAR10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], or STL10 [START_REF] Coates | An analysis of single-layer networks in unsupervised feature learning[END_REF]. To validate this property, we run a pre-trained CLIP [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] on the dataset, identify a certain number of clusters using a K-means algorithm, and further test the disconnectedness of these modes by training a linear classifier. The accuracy on these datasets is 98.1% on MNIST, 93.9% on CIFAR10, and 92.7% on STL10.

The second point in (3.3.2) has a direct impact on the location of the data points X 1 , . . . , X m . Specifically, it implies that each cell in the Voronoi diagram with seeds X 1 , . . . , X m shares a side with all the other cells. In other words, the dual graph of this Voronoi diagram is complete. This assumption, which is further discussed with specific examples in Figure 3.7, can be justified by the concentration of distances in high-dimensional spaces: all the modes are roughly at equal distance [START_REF] Beyer | When is "nearest neighbor" meaningful?[END_REF][START_REF] Aggarwal | On the surprising behavior of distance metrics in high dimensional space[END_REF]. Furthermore, a recent work by [START_REF] Papyan | Prevalence of neural collapse during the terminal phase of deep learning training[END_REF] has shown that embeddings of deep neural networks trained for classification tend to collapse around means that are equidistant and maximally equiangular to one another. By using these embedded representations to measure distance, the target distribution would thus easily satisfy Assumption 4. Projected GANs [START_REF] Sauer | Projected gans converge faster[END_REF] is really close to this idea as the authors show the effectiveness of leveraging a pre-trained classifier when training GANs: instead of directly discriminating images, the discriminator is trained on features extracted from the classifier.

Throughout the rest of the section, we define the set of well-balanced generators as those mapping an equal number of data points to each mode of the data distribution: Definition 3.3.1. A generator G is well-balanced if for all spheres, we have G♯γ(S 1 ) = . . . = G♯γ(S m ).

Considering well-balanced generators is reasonable as many empirical improvements such as WGAN-GP [START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF] or BigBiGAN [START_REF] Donahue | Large scale adversarial representation learning[END_REF] have significantly reduced mode collapse. GANs generate diverse output distributions on datasets such as CIFAR10, CIFAR100, and ImageNet. To validate the use of well-balanced generators, we conducted a small experiment and evaluated the proportion of each class generated by GANs on MNIST and CIFAR10. On MNIST, the minimum proportion of a class is 9.2 and the maximum 10.9, while on CIFAR10 it is 8.3 and 11.9 (in %). The variance-to-mean ratio is equal to 0.03 for MNIST and 0.22 for CIFAR10.

Precision and the associated partition

Now that we have defined the prerequisites for both the data and the model, we propose to establish a connection between the latent space partition and the precision of a generator. We create a link between the set of generators from R d to R D and the set of partitions in the latent space. Specifically, for each given partition in R d , there exists a set of associated generators defined as follows: Definition 3.3.2. For a given partition A = A 1 , . . . , A m on R d , we say that G is associated to

A if: for all i ∈ [1, m], for all z ∈ A i , i = arg min j∈[1,m] ∥G(z) -X j ∥.
Each given generator G is associated with a unique partition A in R d . The geometry of the associated partition A plays a key role in explaining the behavior and performance of the generator G. We are interested in maximizing the precision of generative models. Points in the intersection of two cells A i ∩ A j , (i, j) ∈ [1, m] 2 are equidistant from X i and X j and thus do not belong to any of these modes (since bot r i and r j < ∥X i -X j ∥/2 according to Assumption 4). Additionally, due to the generator's Lipschitzness, there is a small neighborhood around the boundary such that any points in this neighborhood are mapped out of the target manifold. This region in the latent space thus reduces the precision. For a given ε > 0, we now define the epsilon-boundary of the partition A as follows.

Definition 3.3.3. For a given partition A = {A 1 , . . . , A m } of R d and a given ε ∈ R ⋆ + , we denote ∂ ε A the ε-boundary of A , defined as follows.

∂ ε A = m i=1 ∪ j̸ =i A j ε \ ∪ j̸ =i A j ,
where A ε corresponds to the ε-extension of set A. The following lemma makes the connection between the precision of a generator α G and its associated partition A .

Lemma 3.3.1. Assume that Assumption 4 is satisfied and A be a partition in R d . Then, any generator G ∈ G L associated with A verifies:

α G ⩽ 1 -γ(∂ ε min A ).
(3.3.3)

where ε min = min i, j ∥X i -X j ∥/L.
Interestingly, this result holds independently of the partition A . It highlights that the geometry of the partition gives an upper-bound on the precision of the generator. Consequently, to properly determine this bound on the precision levels of generative models, one might be interested in determining the measure of this epsilon-boundary ∂ ε A . By using the result from Lemma 3.3.1, we can derive an upper-bound on the precision that depends on D, L and m: Corollary 3.3.1. Assume that Assumption 4 is satisfied, m ⩽ d + 1. Then, there exists L with L ⩾ D log(m), such that for any well-balanced generator G ∈ G L :

α G ⩽ 1 -ε min log m e -3/2 (3.3.4)
where ε min = min i, j ∥X i -X j ∥/L. In particular, the result in (3.3.4) gives an interesting insight when training GANs on a finite number of modes. Theorem 3.2.3 showed a similar result but for the asymptotic case when the number of modes increases:

α G m→∞ ⩽ e -1 8 ε 2 min e -ε min √ log(m)/2 . (3.3.5)

Optimality for push-forward generative models

To exhibit generative models with optimal precision levels, one must look at partitions with the smallest epsilon-boundary measures γ(∂ ε A ). We argue that this is tightly connected to the theoretical field of Gaussian isoperimetric inequalities. Isoperimetric inequalities link the measure of sets with their perimeters. More specifically, these inequalities highlight minimizers of the perimeter for a fixed measure, e.g. the sphere in an euclidean space with a given Lebesgue measure. In the Gaussian space, [START_REF] Borell | The brunn-minkowski inequality in gauss space[END_REF] and [START_REF] Sudakov | Extremal properties of half-spaces for spherically invariant measures[END_REF] show that in a finite-dimensional case, among all sets of a given measure, half-spaces have a minimum Gaussian perimeter. More formally, for any Borel set A in R d and a half-space H, if we have

γ(A) ⩾ γ(H), then γ(A ε ) ⩾ γ(H ε ) for any ε > 0, where A ε denotes the ε-extension of A.
The Gaussian multi-bubble conjecture was formulated when looking for a way to partition the Gaussian space in m parts, with the least-weighted boundary. It was recently proved by [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF] who showed that the best way to split a Gaussian space R d in m clusters of equal measure, with 2 ⩽ m ⩽ d + 1, is by using 'simplicial clusters' obtained as the Voronoi cells of m equidistant points in R d . Convex geometry theory tells us that each cell is a convex cone, whose borders are hyperplanes going through the origin of R d . We note A ⋆ any partition corresponding to this optimal configuration, see Figure 3.6 for m = 3.

In the following theorem, we apply this result to the understanding of GANs. We make the connection between optimal generators (when m ⩽ d + 1) in levels of precision and the partition A ⋆ derived in [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF].

Theorem 3.3.1 (Optimality of generators with simplicial cluster latent space.). Assume that Assumption 4 is satisfied and m ⩽ d +1. For any δ > 0, there exists C large enough (independent of δ ) and L ⩾ D √ m π log(Cm), and a well-balanced generator G ⋆ ∈ G L associated with A ⋆ such that for any other well-balanced generator G ∈ G L , we have:

α G ⋆ ⩾ α G -δ (3.3.6) Moreover, if m ⩽ d, noting ε max = max i, j ∥X i -X j ∥/L: α G ⋆ ⩾ 1 -ε max m log(Cm), (3.3.7)
Theorem 3.3.1 shows that when L is large enough, the bound in (3.3.4) is almost tight, and thus that the given generator based on the simplicial partition A ⋆ is almost optimal. However, it is not clear whether those are the only generators with optimal precision. The proof is delayed in Appendix B.1.

What if Assumption 4 is not verified? This assumption is needed for the definition of a well-balanced generator associated with A ⋆ as in Theorem 3.3.1. As shown in Figure 3.7, the latent space configuration obtained by the GANs for 3 almost equidistant points (1st row) and 3 almost aligned data points (2nd row). We see that in the later case, the Voronoi partition of the target data points does not verify Assumption 4, and the optimal latent structure is not known. We observe in this specific case that it is made of two parallel hyperplanes, much different from A ⋆ defined by [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF]

(1st row).

What if the dimension m > d + 1? The position of the different spheres could be such that Assumption 4 is no longer valid. Second, since the result from [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF] does not hold, the optimal partition of the Gaussian space in m equal cells is unknown. In this generalized context, GANs could hint at the optimal partition geometry. The left column shows the modes (X 1 , X 2 , X 3 ) from the target distribution and the generated points (small blue dots). In the middle, we plot the Voronoi diagram generated from (X 1 , X 2 , X 3 ). On the right column, we show the boundaries in the GANs latent space with heatmaps of the norm of the gradient of the generator. In the first row, when the data satisfies Assumption 4, GANs achieve the optimal configuration. However, when the data modes do not satisfy this assumption, as seen in the second row, this is no longer the case.

gives some insights on how to divide the Gaussian space into m equitable areas with least Gaussian-weighted perimeter.

What if the modes do not have equal measure? The fact that each mode has equal measure in the target distribution might not be verified for unbalanced datasets. First, the optimality of simplicial clusters holds because the multi-bubble theorem is still valid. However, the lowerbound (Equation 3.3.7) does not hold. Additionally, the upper-bound from Corollary 3.3.1 can be relaxed. Consider w 1 , . . . , w m ∈ R m the weights of the different modes, and w min = min i w i , the upper-bound becomes:

α G ⩽ 1 -mε min w min log(1/w min ) e -3/2 .
We observe that this upper-bound might not be tight anymore since it depends on the minimum of the weights w min .

Improving generative models

Our proposed theoretical analysis offers valuable insights into the optimal structure of the latent space for push-forward generative models. We demonstrate that by leveraging this structure, it is possible to design GANs with improved performance. To achieve this, we enforce a simplicial cluster structure in the latent space of GANs during training using a novel rejection sampling procedure called simplicial cluster truncation that can be combined with a mutual-information loss. Note that modifying the latent space distribution of other generative models, such as VAEs or score-based models, is a more complex task.

Simplicial cluster truncation.

Let us denote a simplicial cluster [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF] as (u 1 , . . . , u m ) | u i ∈ R d . The rejection sampling procedure, based on Theorem 3.3.1, involves sampling a latent vector z from γ and accepting it if max i∈ [1,...,m] (z • u i ) > τ, where both τ and m are considered as hyper-parameters. This defines a new latent space distribution where the density is high near the unit vectors u i , i ∈ [1, m]. As a result, the boundaries of the simplicial cluster, which are points with high distances to the centers of Voronoi cells, are rejected. The threshold parameter τ determines the ε value. With this method, the boundaries between different modes are never sampled, leading to a disconnected latent space. This approach can improve the learning of disconnected manifolds by injecting disconnectedness into the modeled generative distribution. Additionally, the use of a geometrical structure that is particularly well suited to separate several modes [START_REF] Papyan | Prevalence of neural collapse during the terminal phase of deep learning training[END_REF] enhances the performance.

Mutual-information loss. The rejection sampling procedure might not be sufficient for the generator to properly use the different clusters of its latent space. To encourage the simplicial cluster structure, we also optimize the mutual information between generated samples and the [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF]. The loss is applied at the beginning of the training and is then dropped.

Experiments

In the following experiments, we validate our theoretical analysis and derive insights for GANs trained on toy and image datasets. We verify: 1) that the latent space geometry of GANs has similar properties than simplicial clusters; 2) that increasing the latent space dimension (d + 1 > m) can help improve GANs, as highlighted in the theoretical section; 3) that GANs' performance is correlated with their latent space geometry; 4) that the proposed simplicial cluster truncation method is effective and boost GANs' performance.

In the following experiments, we train WGANs with gradient penalty (Arjovsky et al., 2017;[START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF]. For mixture of Gaussians, generator and discriminator are MLP networks. For MNIST, the generator and discriminator are standard convolutional architectures. On CIFAR-10, CIFAR-100 , and STL-10, we use either a Resnet-based [START_REF] He | Deep residual learning for image recognition[END_REF] convolutional architecture with self-modulation in the generator [START_REF] Chen | On self modulation for generative adversarial networks[END_REF], either the transformer-based architecture from [START_REF] Jiang | Transgan: Two pure transformers can make one strong gan, and that can scale up[END_REF]. To evaluate the performance of GANs, we use both the precision [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF], the FID [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF], and the density/coverage [START_REF] Naeem | Reliable fidelity and diversity metrics for generative models[END_REF]. We use a dataset-specific classifier to extract image features on MNIST, and InceptionNet pre-trained on ImageNet for CIFAR-10, CIFAR-100 and STL-10. Implementation details are given in Appendix B.2 and code is provided in Supplementary Material.

Linear separability and convexity

According to [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF], the optimal configuration in the latent space is obtained as the Voronoi cells of m equidistant points in R d , if m ≤ d + 1. This means that if GANs reach this optimal configuration, each of the cells must be convex polytopes and have the following properties: 1) the boundaries of a cell are flat; 2) each cell is convex. To investigate this, we use a labeled dataset and assess whether a simple linear model (e.g., multinomial logistic regression) can map latents to labels. If the cells in the latent space are bounded by hyperplanes, then, using the hyperplane separation theorem, the linear model is expected to be a good predictor of a generated sample's label.

We use a standard multi-class labeled dataset. G θ is a pre-trained generator and C φ is a pre-trained classifier considered as an oracle. Using G θ and C φ , we construct a dataset of latent vectors z ∈ R d and their associated labels y = C φ (G θ (z)). On CIFAR-10/100, similarly to [START_REF] Razavi | Generating diverse high-fidelity images with vq-vae-2[END_REF], only data points with a confidence threshold of 0.7 or higher are accepted. This dataset is later split into 100k training points and 10k test points. We use multinomial logistic regression to learn the mapping from latent vectors z to their labels y. We can see in Table 3.2, that the LogReg Accuracy reaches high levels: 90% on MNIST and 70% on CIFAR-10. For the Convex accuracy, we draw two random latent vectors z 0 and z 1 that belong to the same class, and check whether linear interpolations in the latent space also belong to the same class, that is

C φ (G θ (z 0 )) = C φ (G θ (z 0 )) = λ ×C φ (G θ (z 0 )) + (1 -λ ) ×C φ (G θ (z 1 )) for λ ∈ [0, 1]
. Interestingly, we see in Table 3.2 a correlation between the Logreg and Convex accuracy and the precision metric: the more the latent space behaves like a simplicial cluster, the higher the precision. For a qualitative evaluation, we show this phenomenon in Figure 3.9 and stress that linear interpolations conserve the image class.

Impact of the latent space dimension

To evaluate the impact of the latent space dimension, we train GANs with latent space dimension ranging from 2 to 128 on several datasets. In Figure 3.10, we exhibit two phases in the performance of GANs when changing the number of latent dimensions. For a fixed architecture, and a given dataset, we observe the existence of an optimal latent space dimension d ⋆ . When d < d ⋆ the precision or density of the model falls significantly. Interestingly, when d > d ⋆ , the precision becomes constant: overparameterizing the model does not bring a significant improvement. As expected, we observe in Figure 3.10 that the maximum precision/density depends on the complexity of the dataset and its number of modes: the more complex the dataset, the lower the precision. This is also coherent with our theoretical results from both Corollary 3.3.1 and Theorem 3.3.1. An interesting problem was also brought to the fore by [START_REF] Roth | Stabilizing training of generative adversarial networks through regularization[END_REF]. When training GANs two different issues can arise: 1) dimensional misspecification where the true and modeled distributions do not have density functions w.r.t. the same base measure, and 2) density misspecification, where GANs try to fit a disconnected manifold with a unimodal distribution. To isolate the density misspecification studied in this section, we train a conditional GAN with a low-dimensional latent space R d (e.g. R 5 in our setting), so that the dimension of the generated manifold is at most 5. We later collect a dataset of synthetic generated samples Synthetic CIFAR-10, and train unconditional GANs with varying latent space dimensions. Figure 3.10 shows that GANs converge to the same limits for Precision and Density on Synthetic CIFAR-10 and CIFAR-10. This shows that the performance is more impacted by the density misspecification (trying to fit a disconnected target distribution with a connected one) rather than the dimensional misspecification.

The latent geometry and GANs' performance

We investigate the relationship between the performance of GANs and their latent space geometry. To do so, we train many generators with different capacities (increasing width), and study how it impacts both the latent geometry and the performance. The results in Figure 3.11 reveal a strong positive correlation between the performance of GANs and the linearity/convexity of the latent space: the better the GANs perform, the more linearly separable and convex the latent space is. Indeed, the Pearson correlation between Precision and LogReg Accuracy is 0.98 on CIFAR-10, and 0.94 on CIFAR-100. Interestingly, overparametrization was known to help push-forward generative models in their optimization procedure [START_REF] Balaji | Understanding over-parameterization in generative adversarial networks[END_REF] and in increasing their Lipschitz constant [START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF]. We demonstrate here that it can help GANs in reaching an optimal latent space structure, resulting in improved performance.

Impact of the simplicial truncation method

Finally, we aim to improve GANs performance by using our theoretical results (Theorem 3.3.1). This is done by truncating the latent Gaussian distribution, as discussed in Section 3.3.4, so Table 3.3 Improving GANs with simplicial cluster latent space. JBT stands for the Jacobianbased truncation from section 3.2; DeliG. for latent space with mixture of Gaussians [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF]; simp. for our proposed truncation method with simplicial cluster. These results demonstrate that generators with a simplicial cluster latent space consistently outperform the baseline generator in Precision/Density, and most of the times outperforms other boosting methods (DeliGAN and JBT). that the generator structures its latent space with a simplicial cluster geometry. Note that the rejection threshold used at inference time can be higher than the one used at training time, since we have observed that higher rejection thresholds can help us increase both the precision and density of the models. The results in Table 3.3 demonstrate that the use of this truncation method can improve the density and precision of GANs, without lowering the coverage nor the FID. This simplicial-based truncation has thus proved to be effective at removing off-manifold samples and can help improve push-forward generative models.

Conclusion

In conclusion, this section takes a step towards a better understanding of push-forward generative models. When the latent space dimension is large enough, we prove the existence of an optimal latent space geometry, called 'simplicial clusters'. Through experiments, we demonstrate that generative models with sufficient capacity tend to conform to this optimal geometry and also that enforcing this latent structure can improve GANs' performance. Our analysis has potential to drive further research on generative models with both theoretical and practical implications, such as developing new models that favor the emergence of such clusters in both latent and feature spaces. Similarly to what has been done in classification [START_REF] Papyan | Prevalence of neural collapse during the terminal phase of deep learning training[END_REF], studying thoroughly the feature space of deep generative models is also an open question.

While our theoretical analysis demonstrates the existence of optimal generators, we were unable to prove their uniqueness. This limitation is associated with identifying partitions with the lowest ε-boundary measures in the Gaussian space, which is a challenging and unresolved problem in geometric measure theory. This could be a topic for future work.

Another challenge is to learn principled truncation method for pre-trained GANs. Indeed, in section 3.2, we use a heuristic method to filter generated samples. In this section, the truncation is based on the intuition that the generator naturally exploits the latent space structure. However, we have no guarantee that our truncation method will improve the performance of the generative model. In some cases, . In the next section, we investigate learning-based approaches to improve pre-trained GANs. Specifically, we propose a method that relies on an adversarial mechanism to learn importance weights in the latent space of a pre-trained GAN.

Latent reweighting, an almost free improvement for

GANs

Abstract. Standard formulations of GANs, where a continuous function deforms a connected latent space, have been shown to be misspecified when fitting different classes of images. In particular, the generator will necessarily sample some low-quality images in between the classes. Rather than modifying the architecture, a line of works aims at improving the sampling quality from pre-trained generators at the expense of increased computational cost. Building on this, we introduce an additional network to predict latent importance weights and two associated sampling methods to avoid the poorest samples. This idea has several advantages: 1) it provides a way to inject disconnectedness into any GAN architecture, 2) since the rejection happens in the latent space, it avoids going through both the generator and the discriminator, saving computation time, 3) this importance weights formulation provides a principled way to reduce the Wasserstein's distance to the target distribution. We demonstrate the effectiveness of our method on several datasets, both synthetic and high-dimensional.

Introduction

GANs [START_REF] Goodfellow | Generative adversarial nets[END_REF] are an effective way to learn complex and high-dimensional distributions, leading to state-of-the-art models for image synthesis in both unconditional [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF] and conditional settings [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]. However, it is well-known that a single generator with an unimodal latent variable cannot recover a distribution composed of disconnected sub-manifolds [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF]. This leads to a common problem for practitioners: the existence of very low-quality samples when covering different modes. This is formalized in section 3.2 and 3.3, where we provide impossibility theorems on the learning of disconnected manifolds with standard formulations of GANs. Fitting a disconnected target distribution requires an additional mechanism inserting disconnectedness in the modeled distribution. A first solution is to add some expressivity to the model: [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] propose to train a mixture of generators, while [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF] make use of a multi-modal latent distribution.

A second line of research relies heavily on a variety of Monte-Carlo algorithms, such as Rejection Sampling [START_REF] Azadi | Discriminator rejection sampling[END_REF] or Metropolis-Hastings [START_REF] Turner | Metropolis-hastings generative adversarial networks[END_REF]. Monte-Carlo methods aim at sampling from a target distribution, while having only access to samples generated from a proposal distribution. Using the previously learned generative distribution as a proposal distribution, this idea was successfully applied to GANs. However, one of the main drawbacks is that Monte-Carlo algorithms only guarantee to sample from the target distribution LatentRS LatentGA Fig. 3.12 Overview of the proposed method. GANs tend to produce poor images for unlucky draws of the latent variable (top row, left). We introduce importance weights w ϕ (z) in the latent space that allow us to use rejection sampling and accept a given latent variable z with probability P a (z) ∝ w ϕ (z) (LatentRS, top row), or to perform a simple gradient ascent over the importance weight (LatentGA, bottom row), leading to better images. Both strategies can be combined for improved image quality. Images generated with StyleGAN2 trained on LSUN Church.

under strong assumptions. First, we need access to the density ratios between the proposal and target distributions or equivalently to a perfect discriminator [START_REF] Azadi | Discriminator rejection sampling[END_REF]. Second, these methods are efficient only if the support of the proposal distribution fully covers the one of the target distribution. This is unlikely to be the case when dealing with high-dimensional datasets (Arjovsky and Bottou, 2017).

To tackle this issue, we propose a novel method aiming at reducing the Wasserstein distance between the previously trained generative model and the target distribution. This is done via the adversarial training of a third network that learns importance weights in the latent space. Note that this network does not aim at increasing the support of the proposal distribution but at re-weighting the latent distribution, under a Wasserstein criterion. Thus, these importance weights define a new distribution in the latent space, from which we propose to sample with two complementary methods: latent rejection sampling (latentRS) and latent gradient ascent (latentGA). To better understand our approach, we illustrate its efficiency with simple examples. On the top of the Figure 3.12, we show samples coming from a pre-trained StyleGAN2 [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF] and their respective acceptance probability (latentRS). At the bottom, we exhibit a sequence of generated images while following a gradient ascent on the learned importance weights (latentGA).

Our contributions are the following:

• We propose a novel approach that trains a neural network to directly modify the latent space of a GAN. This provides a principled way to reduce the Wasserstein distance to the target distribution.

• We show how to sample from this new generative model with different methods: latent Rejection Sampling (latentRS), latent Gradient Ascent (latentGA), and latentRS+GA, a method that leverages the complementarity between the two previous solutions.

• We run a large empirical comparison between our proposed methods and previous approaches on a variety of datasets and distributions. We empirically show that all of our proposed solutions significantly reduce the computational cost of inference. More interestingly, our solutions propose a wide span of performances ranging from latentRS, optimizing speed, that matches state-of-the-art almost for free (computational cost divided by 15) and latentRS+GA (computational cost divided by 3) that outperforms previous approaches.

Notation. Before moving to the related work section, we shortly present the notation needed in the section. The goal of the generator is to generate data points that are "similar" to samples collected from some target probability measure µ ⋆ . The measure µ ⋆ is defined on a potentially high-dimensional space R D , equipped with the euclidean norm ∥ • ∥. We call µ n the empirical measure. To approach µ ⋆ , we use a parametric family of generative distribution, where each distribution is the push-forward measure of a latent distribution Z and a continuous function modeled by a neural network. In most applications, the random variable Z defined on a low-dimensional space R d is either a multivariate Gaussian distribution or uniform distribution. The generator is a parameterized class of functions from R d to R D , say G = {G θ : θ ∈ Θ }, where Θ ⊆ R p is the set of parameters describing the model. Each function G θ takes input from Z and outputs "fake" observations with distribution µ θ = G θ ♯Z. On the other hand, the discriminator is described by a family of functions from R D to R, say D = {D α : α ∈ Λ }, Λ ⊆ R Q . Finally, for any given distribution µ, we note S µ its support. [START_REF] Goodfellow | Generative adversarial nets[END_REF] already stated that when training vanilla GANs, the generator could ignore modes of the target distribution: this is called mode collapse. A significant step towards understanding this phenomenon was made by Arjovsky and Bottou (2017) who explained that the standard formulation of GANs leads to vanishing or unstable gradients. The authors proposed the Wasserstein GANs (WGANs) architecture (Arjovsky et al., 2017) where, in particular, discriminative functions are restricted to the class of 1-Lipschitz functions. WGANs aim at solving the following:

Related Work

sup α∈A inf θ ∈Θ E x∼µ ⋆ D α (x) -E z∼γ D α (G θ (z)) (3.4.1)
3.4.2.1 Learning disconnected manifolds with GANs: training and evaluation

The broader drawback of standard GANs is that, since any modeled distribution is the pushforward of a unimodal distribution by a continuous transformation, it has a connected support. This means that when the generator covers multiple disconnected modes of the target distribution, it necessarily generates samples out of the real data manifold [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF]. Consequently, any thorough evaluation of GANs should assess simultaneously both the quality and the variety of the generated samples. To solve this issue, [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF] and [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF] propose a Precision/Recall metric that aims at measuring both the mode dropping and the mode inventing. The precision refers to the portion of generated points that belongs to the target manifold, while the recall measures how much of the target distribution can be reconstructed by the model distribution.

Building on this metric, we highlighted the trade-off property of GANs deriving upperbounds on the precision of standard GANs in section 3.2 and 3.3. To solve this problem, a common direction of research consists in over-parameterizing the generative model. [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] enforce diversity by using a mixture of generators, while [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF] suggests that a mixture of Gaussians in the latent space is efficient to learn diverse and limited data. Similarly, [START_REF] Balaji | Robust optimal transport with applications in generative modeling and domain adaptation[END_REF] propose importance weights that aim at robustifying the training of GANs and make it less sensitive to the target distribution's outliers.

Improving the quality of GANs post-training

Another line of research consists in improving the sampling quality of pre-trained GANs. [START_REF] Tanaka | Discriminator optimal transport[END_REF] designed Discriminator Optimal Transport (DOT), a gradient ascent driven by a Wasserstein discriminator to improve every single sample. Similarly, [START_REF] Che | Your gan is secretly an energy-based model and you should use discriminator driven latent sampling[END_REF] follow a discriminator-driven Langevin dynamic.

Another well-studied possibility would be to use Monte-Carlo (MC) methods [START_REF] Robert | Monte Carlo statistical methods[END_REF]. Following this path, [START_REF] Azadi | Discriminator rejection sampling[END_REF] were the first to use a rejection sampling method to improve the quality of the proposal distribution µ θ . The authors use the fact that the optimal vanilla discriminator trained with binary cross-entropy is equal to µ ⋆ /(µ ⋆ + µ θ ). Thus, a parametric discriminator D α : R D → [0, 1] can be used to approximate the density ratios r α as follows:

r α (x) := µ ⋆ (x) µ θ (x) = D α (x) 1 -D α (x) . (3.4.2)
This density ratio can then be plugged in the Rejection Sampling (RS) algorithm. Doing so, it can be shown that sampling from µ θ and accepting samples probabilistically is equivalent to sample from the target distribution µ ⋆ . The acceptance probability of a given sample x is P a (x) = r α (x) k . This is valid as long as there is a constant k ∈ R + such that µ ⋆ (x) ≤ kµ θ (x) for all x.

Turner et al. ( 2019) use similar density ratios and derive MH-GAN, by using the independent Metropolis-Hasting algorithm [START_REF] Hastings | Monte carlo sampling methods using markov chains and their applications[END_REF]. Finally, Grover et al. ( 2019) use these density ratios r α as importance weights and perform discrete sampling relying on the Sampling-Importance-Resampling (SIR) algorithm [START_REF] Rubin | Using the sir algorithm to simulate posterior distribution[END_REF]. Given X 1 , . . . , X n ∼ µ n θ , we have:

µ SIR θ ,α (X i ) = r α (X i ) n ∑ j=1 r α (X j ) . (3.4.3)
Note that these algorithms all rely on similar density ratios and differ by the acceptancerejection scheme chosen. Interestingly, in RS, the acceptance rate is not controlled, but we are guaranteed to sample from µ ⋆ . Conversely, with SIR and MH, the acceptance rate is a chosen parameter, but we are sampling from an approximation of the target distribution.

Drawbacks of density-ratio-based methods

Even though these methods have the advantage of being straightforward, they suffer from one main drawback. In practice, because both the target and the proposal manifold do not have full dimension in R D [START_REF] Fefferman | Testing the manifold hypothesis[END_REF], (Arjovsky and Bottou, 2017, Lemma 3) show that it is highly likely that µ θ (S µ θ S µ ⋆ ) = 0 and µ ⋆ (S µ θ S µ ⋆ ) = 0. Consequently, when dealing with high-dimensional datasets, the proposal distribution µ θ and the target distribution µ ⋆ might intersect on a null set. Thus, one would have r α (x) = 0 almost everywhere on S µ θ . In this setting, the assumptions of MC methods are broken, and these algorithms will not allow sampling from µ ⋆ . This is shown in Figure 3.13. In order to correct this drawback, our method proposes to avoid the computation of density ratios from a classifier and to directly learn how to re-weight the proposal distribution. Our proposed scheme aims at minimizing the Wasserstein distance to the empirical measure while controlling the range of these importance weights.

Adversarial Learning of Latent Importance weights

Similar to previous works, our method aims at improving the performance of a generative model, post-training. We assume the existence of a WGAN model (G θ , D α ) pre-trained using (3.4.1).

The pushforward generative distribution µ θ is assumed to be an imperfect approximation of the target distribution. The goal is now to learn how to redistribute the mass of the modeled distribution so that it best fits the target distribution.

Definition of the method

To improve the sampling quality of our pre-trained GANs, we propose to learn an importance weight function that directly learns how to avoid low-quality images and focus on very realistic ones. More formally, we over-parameterize the class of generative distributions and define a parametric class Ω = {w ϕ , ϕ ∈ Φ} of importance weight functions. Each function w ϕ associates importance weights to latent space variables and is defined from R d to R + . For a given latent space distribution γ and a network w ϕ , a new measure γ ϕ is defined on R d :

for all z ∈ R d , dγ ϕ (z) = w ϕ (z)dγ(z) (3.4.4)
Using this formulation, we can prove the following lemma:

Lemma 3.4.1. Assume that E γ w ϕ = 1, then the measure γ ϕ is a probability distribution defined on R d .

Consequently, we now propose a new modeled generative distribution µ ϕ θ , the pushforward distribution µ ϕ θ = G θ ♯γ ϕ . The objective is to find the optimal importance weights w ϕ that minimizes the Wasserstein distance between the true distribution µ ⋆ and the new class of generative distributions. The proposed method can thus be seen as minimizing the Wasserstein distance to the target distribution, over an increased class of generative distributions. Denoting by Lip 1 the set of 1-Lipschitz real-valued functions on R D , i.e.,

Lip 1 = f : R D → R : | f (x) -f (y)| ∥x -y∥ ⩽ 1, (x ̸ = y) ∈ (R D ) 2 ,
we want, given a pre-trained model µ θ , to solve:

arg min ϕ∈Φ W (µ ⋆ , µ ϕ θ ) = arg min w ϕ ∈Ω sup D∈Lip 1 E µ ⋆ D -E µ ϕ θ D = arg min w ϕ ∈Ω sup D∈Lip 1 E µ ⋆ D -E µ θ w ϕ D
The network w ϕ , parameterized using a feed-forward neural network, thus learns how to redistribute the mass of µ θ such that µ ϕ θ is closer to µ ⋆ in terms of Wasserstein distance. Similarly to the WGANs training, the discriminator D α approximates the Wasserstein distance. D α and w ϕ are trained adversarially, whilst keeping the weights of G θ frozen, using the following optimization scheme:

inf ϕ∈Φ sup α∈Λ E x∼µ ⋆ D α (x) -E z∼Z w ϕ (z) × D α (G θ (z)) (3.4.5)
Note that our formulation can also be plugged on top of any objective function used for GANs.

Optimization procedure

However, as in the field of counterfactual estimation, a naive optimization of importance weights by gradient descent can lead to trivial solutions.

1. First, if for example, the Wasserstein critic D α outputs negative values for any generated sample, the network w ϕ could simply learn to avoid the dataset and output 0 everywhere (Swaminathan and Joachims, 2015a).

2. Second, another problem comes from the fact that (3.4.5) can be minimized not only by putting large importance weights w ϕ (z) on the examples with high likelihoods D α (G(z)) but also by maximizing the sum of the weights: this is the propensity overfitting (Swaminathan and Joachims, 2015b).

3. For the objective defined in (3.4.5) to be a valid Wasserstein distance minimization scheme, the measure µ ϕ θ must be a probability distribution, i.e. E γ w ϕ = 1.

To tackle this, we first add a penalty term in the loss to enforce the expectation of the importance weights to be close to 1. This is similar to the self-normalization proposed by [START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF]. However, one still has to cope with the setting where the distribution γ ϕ collapses to discrete data points:

Theorem 3.4.1. Given a pre-trained generative distribution µ θ absolutely continuous with respect to the Lebesgue measure on R D . Let Φ be the non-parametric class of continuous functions satisfying E γ w ϕ = 1. We have that:

W (µ n , 1 n n ∑ i=1 δ ( Xi )) ⩽ inf ϕ∈Φ W (µ n , µ ϕ θ )
where δ refers to the Dirac probability distribution and Xi = arg min

x∈S µ θ ∥x -X i ∥.
For clarity, the proof is delayed in Appendix. Intuitively, this theorem shows that the best way to approximate the empirical measure µ n would be by considering a mixture of Diracs with each mode being the projection of a training data point on the support of the learned manifold S µ θ . The network w ϕ could thus be tempted to approximate this mixture of Diracs defined in Theorem 3.4.1 and collapse on some specific latent data points. This could lead to an increased time complexity at inference (see (Azadi et al., 2019, Section 3)). More importantly, this would mean a mode collapse and a lack of diversity in the generated samples.

To avoid such cases where small areas of z have really high w ϕ (z) values (mode collapse), we enforce a soft-clipping on the weights [START_REF] Bottou | Counterfactual Reasoning and Learning Systems: the example of Computational Advertising[END_REF][START_REF] Grover | Bias correction of learned generative models using likelihood-free importance weighting[END_REF]. Note that this constraint on w ϕ (z) could also be implemented with a bounded activation function on the final layer, such as a re-scaled sigmoid or tanh activation. Finally, we get the following objective function for the network w ϕ :

sup ϕ∈Φ E z∼Z w ϕ (z) D α (G θ (z)) -∆ discriminator reward -λ 1 E z∼Z w ϕ (z) -1 2 self-normalization -λ 2 E z∼Z max 0, (w ϕ (z) -m) 2 soft-clipping , (3.4.6)
where ∆ = min z∼Z D α (G(z)). λ 1 , λ 2 , and m are hyper-parameters (values displayed in Appendix). For more details, we refer the reader to Algorithm 1.

Sampling from the latent importance weights

Given a pre-trained generator G θ and an importance network w ϕ , we now present the three proposed sampling algorithms associated with our model:

1) Latent Rejection Sampling (latentRS, Algorithm 2). The first proposed method aims at sampling from the newly learned latent distribution γ ϕ defined in (3.4.4). Since the learned importance weights are capped by m defined in (3.4.6), this setting fits in the Rejection Sampling (RS) algorithm [START_REF] Robert | Monte Carlo statistical methods[END_REF]. Any sample z ∼ γ is now accepted with probability P a (z) = w ϕ (z)/m. Interestingly, by actively capping the importance weights as it is done in counterfactual estimation [START_REF] Bottou | Counterfactual Reasoning and Learning Systems: the example of Computational Advertising[END_REF][START_REF] Faury | Distributionally robust counterfactual risk minimization[END_REF], one controls the acceptance rates P a (z) of the rejection sampling algorithm:

E γ P a (z) = R d w ϕ (z) m dγ(z) = 1 m .
2) Latent Gradient Ascent (latentGA). Inspired from (Tanaka, 2019, Algorithm 2), we propose a second method, latentGA, where we perform gradient ascent in the latent space (see the algorithm in Appendix). For any given sample in the latent space, we follow the path maximizing the learned importance weights. This method is denoted latentGA. Note that the learning rate and the number of updates used for this method are hyper-parameters that need to be tuned.

{z i } b i=1 ∼ Z ; 6 EMD ← 1 b ∑ b i=1 D α (x i ) -w ϕ (z i )D α (G θ (z i )); 7 GP ← Gradient-Penalty(D α , x, G θ (z)); 8 grad α ← ∇ α (-EMD + GP) ; 9 Update α with grad α ; end Sample {z i } b i=1 ∼ Z ; ∆ ← min i [D α (G θ (z i ))] ; EMD ← 1 b ∑ b i=1 w(z i )[D α (G θ (z i )) -∆ ]; R norm ← ([ 1 b ∑ b i=1 w(z i )] -1) 2 ; R clip ← 1 b ∑ b i=1 max(0, w ϕ (z i ) -m) 2 ; grad ϕ ← ∇ ϕ (EMD + λ 1 R norm + λ 2 R
3) Combining latentRS with Gradient Ascent (latent RS+GA, see Appendix). Finally, we propose to combine sequentially both methods. In a first step, we avoid low-quality samples with latentRS. Then, we use latentGA to further improve the remaining generated samples. See algorithm in Appendix.

Advantages of the proposed approach

We now discuss two advantages of our method compared to previous density-ratio-based Monte-Carlo methods.

Computational cost. By using sampling algorithms in the latent space, we avoid going through both the generator and the discriminator, leading to a significant computational speedup. This is of particular interest when dealing with high-dimensional spaces, since we do not need to pass through deep CNNs generator and discriminator [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF]. In the next experimental section, we observe a computational cost decreased by a factor of 10.

Monte-Carlo methods do not properly work when the support S µ θ does not fully cover the support S µ ⋆ . To better illustrate this claim, we consider a simple 2D motivational example where the real data lies on four disconnected manifolds. We start with a proposal distribution (in blue) that does not fully recover the target distribution (Figure 3.13a). In this setting, we see in Figure 3.13b that the discriminator's density-ratio-based methods [START_REF] Azadi | Discriminator rejection sampling[END_REF] avoids half of the proposal distribution, while our proposed method learns a very different re-weighting (see Figure 3.13c). This illustration is important since (Arjovsky and Bottou, 2017, Theorem 2.2) have shown that in high-dimension the intersection S µ ⋆ S µ θ is likely to be a negligible set under µ θ . Knowing that S µ θ does not fully recover S µ ⋆ , there is thus no theoretical guarantee that using a sampling algorithm will improve the estimation of µ ⋆ . On the opposite, our method looks for the optimal re-weighting of µ θ under a well-defined criterion: the Wasserstein distance. This results in a better fit of the real data distribution (see next section). (d) Heatmap of the w ϕ in the latent space (in the blue areas, w ϕ =0). Fig. 3.13 Synthetic experiment mimicking the setting of GANs in high-dimension, where data and generated manifolds are close but do not perfectly intersect. While DRS only selects the intersection of manifolds and ignores the rest, the latent importance weights define a rejection mechanism that minimizes the Wasserstein distance. For conciseness, WGAN stands for WGAN-GP.

Experiments

In this section, we illustrate the efficiency of the proposed methods, latentRS, latentGA, and latentRS+GA on both synthetic and natural image datasets. On image generation tasks, we empirically stress that latentRS slightly surpasses density-ratio-based methods with respect to the Earth Mover's distance while reducing the time complexity by a factor of around 10. The use of latentGA also gives interesting experimental visualizations and improves image quality. More importantly, when combined, we show that latenRS+GA surpasses the concurrent methods, while still being less computationally intensive. Finally, we show results with different models such as Progressive GAN [START_REF] Karras | Progressive growing of GANs for improved quality, stability, and variation[END_REF] and StyleGAN2 [START_REF] Karras | Analyzing and improving the image quality of stylegan[END_REF].

Evaluation metrics

To measure the performances of GANs when dealing with low-dimensional applications, we equip our space with the standard Euclidean distance. However, for the case of image generation, we follow [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]; [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF] and consider the euclidean distance between embeddings of a pre-trained network, that convey more semantic information. Thus, for a pair of images (a, b), we define the distance d(a, b) as d(a, b) = ∥φ (a)φ (b)∥ 2 where φ is a pre-softmax layer of a supervised classifier. On MNIST and F-MNIST, the classifier is pre-trained on the given dataset. On CelebA and LSUN Church, we use VGG-16 pre-trained on ImageNet.

To begin with, we report the FID [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF]. We also compare the performance of the different methods with the Precision/Recall (PR) metric [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF]. It is a more robust version of the Precision/Recall metric, which was first applied in the context [START_REF] Sajjadi | Assessing generative models via precision and recall[END_REF]. Finally, we approximate the Wasserstein distance using the Earth Mover's Distance (EMD) between generated and real data points. This measure is particularly suited to the study of WGANs, since it is linked to their objective function. Letting X = {x 1 , . . . , x n } and Y = {y 1 , . . . , y n } be two collections of n data points and S be the set of permutations of [1, n], the Earth Mover's distance between X and Y is defined by:

EMD(X,Y ) = min σ ∈S n ∑ i=1 ∥x i -y σ i ∥

Synthetic datasets

To begin the experimental study, we test our method on 2D synthetic datasets in the same setting as Tanaka (2019). Table 3.4 compares the latentRS method with previous approaches on the Swiss roll dataset and on a mixture of 25 Gaussians. We see that the network w ϕ efficiently redistributes the pre-trained distribution µ θ since EMD(µ n , µ ϕ θ ) is significantly smaller than EMD(µ n , µ θ ).

Image datasets

Implementation of baselines. We now compare latentRS, latentGA, and latentRS+GA with previous works leveraging discriminator's information on high-dimensional data. In particular, we implemented a wide set of post-processing methods for GANs: DRS [START_REF] Azadi | Discriminator rejection sampling[END_REF], MH-GAN [START_REF] Turner | Metropolis-hastings generative adversarial networks[END_REF], SIR [START_REF] Grover | Bias correction of learned generative models using likelihood-free importance weighting[END_REF] and DOT [START_REF] Tanaka | Discriminator optimal transport[END_REF]. DRS, MH-GAN and SIR use the same density ratios, and we did not see significant differences between those three methods in our experiments. Consequently, for the following experiments, we compare our algorithms to SIR and DOT. For SIR, we take the discriminator at the end of the adversarial training, fine-tune it with the binary cross-entropy loss and select the best Prec.

Rec [START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF], spectral normalization [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF] during fine-tuning and do not include an explicit mechanism to calibrate the classifier. Description of datasets and neural architectures. We first consider two well-known image datasets that are MNIST (LeCun et al., 1998) and FashionMNIST (F-MNIST). We follow [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] and use a standard CNN architecture composed of a sequence of blocks made of 3x3 convolution layer and ReLU activations with nearest neighbor upsampling. For these datasets, the discriminator is trained using the hinge loss with gradient penalty (Hinge-GP). Finally, the architecture used for the network w ϕ is very simple: an MLP with 4 fully-connected layers and ReLU activation (with a width = 4 × d).

CelebA [START_REF] Liu | Deep learning face attributes in the wild[END_REF] is a large-scale dataset of faces covering a variety of poses. We use a pre-trained model of Progressive GAN [START_REF] Karras | Progressive growing of GANs for improved quality, stability, and variation[END_REF]) at 128x128 resolution. The discriminator is trained using a Wasserstein loss with gradient-penalty. Also, the architecture used for the network w ϕ is really standard: a 5 hidden-layer MLP with a width of the same size than the latent space dimension.

LSUN Church [START_REF] Yu | Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop[END_REF] is a dataset of church images with a lot of variety. We use a pre-trained model of StyleGAN2 [START_REF] Karras | Analyzing and improving the image quality of stylegan[END_REF]) at 256x256 resolution. Similarly to the CelebA dataset, the discriminator is trained using a Wasserstein loss with gradient-penalty. Also, the architecture used for the network w ϕ is a 3 hidden-layer MLP with width equal to the latent space dimension. Note that the StyleGAN architecture already contains an 8-layer MLP network M θ : R d → R d that transforms a latent space variable to an intermediate latent variable [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF]. We consequently leverage this pre-trained M θ and train the network w ϕ on top of it.

Results. The main results of this comparison are shown in Table 3.5 and Figure 3.14. On all studied datasets, our latentRS+GA outperforms every other method on the EMD with lower computational cost. Interestingly, latentRS achieves good performance on FID while being more than 15 times faster. Figure 3.14 is particularly interesting since it gives a good visualization of the trade-off between computational cost and quality of the generated samples. On this experiment ran on CelebA and LSUN, we observe that latentRS+GA can achieve a significantly better precision than both SIR and DOT while being much faster. Interestingly, even though these datasets are high-dimensional, contain only one-class, and w ϕ has a low capacity, our proposed methods still produce interesting results.

To visualize the efficiency of the proposed method, Figure 3.15 shows generated samples along with their acceptance probabilities. As expected, we observe that higher acceptance probabilities correlate with higher quality images. Figure 3.16 stresses how generated images improve when performing latent gradient ascent on the importance weights. Finally, we provide more qualitative results and details on the experiments in supplementary material.

P a (z) = 0.00P a (z) = 0.05P a (z) = 0.68P a (z) = 0.73P a (z) = 0.00P a (z) = 0.08P a (z) = 0.57P a (z) = 0.73 P a (z) = 0.00P a (z) = 0.04P a (z) = 0.38P a (z) = 0.69P a (z) = 0.00P a (z) = 0.06P a (z) = 0.58P a (z) = 0.69 Fig. 3.15 Images drawn from the generative model and their acceptance probabilities with the latentRS algorithm, given by the network w ϕ . As expected, the quality of images correlates with higher acceptance rates on all datasets: MNIST, F-MNIST, CelebA, and LSUN. Fig. 3.16 Gradient ascent on latent importance weights (latentGA): the quality is gradually improved as we move to larger importance weights. Each image is generated only for visualization, and one can run this gradient ascent directly in the latent space using w ϕ . Interestingly, this gradient ascent only involves a simple MLP network which is computationally cheap.

Conclusion

This section deals with improving the quality of pre-trained GANs. Conversely to concurrent methods which leverage the discriminator at inference time, we propose to train adversarially a neural network which learns importance weights in the latent space of GANs. These latent importance weights are then used with two complementary sampling methods: latentRS and latentGA. We experimentally show that this latent reweighting consistently enhances the quality of the pre-trained model. When these two methods are combined in latentRS+GA, it surpasses concurrent post-training methods while being less computationally intensive.

Chapter 4

Image editing with deep neural networks

Introduction

Designing learning algorithms for image editing using deep neural networks is a challenging task due to the scarcity of target data. Ideally, supervised learning would be the optimal approach to address image editing. This would enable the use of a standard pipeline for supervised deep learning, starting with the collection of a labeled dataset. In image editing tasks, the input data, denoted as X, is usually composed of a source image, I s , and a conditioning variable, C, where X = [I s ,C]. Meanwhile, the target data represents the image to be produced, denoted as I t . However, collecting labeled datasets for image editing tasks is often not feasible, as it can be prohibitively expensive. For instance, in the virtual try-on task discussed in Section 4.3, I s denotes the source image of the person, C denotes the clothing item image, and I t represents the image of the person in the exact pose as I s , wearing the clothing item from C.

In this chapter, we present two original contributions enabling image editing tasks to be performed without complete supervision. In Section 4.2, we demonstrate the ability to address various image editing tasks, such as local denoising, image inpainting, image compositing, or scribble-based editing, using a single training objective and model per dataset. This training objective, inspired by BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], a widely-used unsupervised pretraining objective for large language models, involves predicting masked sequences. We demonstrate its superior performance in image editing compared to other generic methods such as GANs' inversions.

In Section 4.3, we leverage adversarial training and a teacher-student distillation mechanism to enhance image-based virtual try-on algorithms. The conventional pipeline for virtual try-on entails supervised training, where the input image I s is a masked image I s = M(I t ), with M as a masking operator that hides the clothing item. We demonstrate that this standard pipeline, coupled with adversarial training, can guide a student model that takes unmasked images I s as input, thereby outperforming the teacher model and achieving state-of-the-art performance at the time of publication.

Introduction

Denoising

Completion Compositing Scribble-edit Crossover Significant progress in image generation has been made in the past few years, thanks notably to Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF]. For example, the StyleGAN architecture [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF][START_REF] Karras | Analyzing and improving the image quality of stylegan[END_REF]) yields state-of-the-art results in datadriven unconditional generative image modeling. Empirical studies have also shown the usefulness of GANs' architecture when it comes to image manipulation. By following specific directions in the latent space, one can modify an image attribute such as gender, age, the pose of a person [START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF], or the angle [START_REF] Jahanian | On the "steerability" of generative adversarial networks[END_REF]. However, since the whole picture is generated from a Gaussian vector, changing some undesired elements while keeping the others frozen is difficult. To solve this problem, edition algorithms involving optimization procedures have been proposed [START_REF] Abdal | Image2stylegan: How to embed images into the stylegan latent space[END_REF][START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF] but with one main caveat: the results are not convincing when manipulating complex visuals [START_REF] Niemeyer | Giraffe: Representing scenes as compositional generative neural feature fields[END_REF] (cf. experimental section for visual results).

Independently, [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] propose VQVAE, a promising latent representation by training an encoder/decoder using a discrete latent space. The authors demonstrate the possibility to embed images in sequences of discrete tokens borrowing ideas from vector quantization (VQ), paving the way for the generation of images with autoregressive transformer models [START_REF] Ramesh | Zero-shot text-to-image generation[END_REF][START_REF] Esser | Taming transformers for high-resolution image synthesis[END_REF]. Building on this litterature, we argue that one of the benefits of this representation is that each token in the sequence is mostly coding for a localized patch of pixels (see section 4.2.3.4), thus opening the possibility for an efficient localized latent edition.

Aiming to build a unified approach for image manipulation, we propose a method that leverages both the spatial property of the discrete vector-quantized representation and the use of model that performs attention on the whole image. To do so, we train a bi-directionnal transformer network based on ideas from the language model BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], naming EdiBERT the resulting model. During training, EdiBERT tries to recover the original tokens of a perturbed sequence through a bidirectional attention schema. In computer vision, this approach has mainly been studied in the context of self-supervised representation learning [START_REF] Bao | BEit: BERT pre-training of image transformers[END_REF][START_REF] He | Masked autoencoders are scalable vision learners[END_REF]. We advocate that training a single model using this generic objective provides a sounded way to obtain a model able to tackle several editing tasks. Finally, to practically handle these tasks, we also derived two different sampling algorithms: one dedicated for image denoising and editing, and a second one for inpainting.

To better visualize the abilities of EdiBERT after a single training, we show in Figure 4.1 that the same model can now be used in many different image manipulation tasks such as denoising, inpainting (or completion), compositing, and scribble-based editing.

To sum up, our contributions are the following:

+ We analyze the VQ latent representations and illustrate their spatial properties, and show how to improve the reconstruction capabilities of VQGAN, using a post-processing procedure that better recovers the pixel content outside of the edited region.

+ We show how to derive two different sampling algorithms from a single bidirectional transformers: one for the task of image denoising where the locations of the edits are unknown, and a second one for inpainting or image compositing where the mask specifying the area to edit is known.

+ Finally, we show that using this generic simple training algorithm along with its companion post-processing allow us to achieve competitive results on various image manipulation tasks.

Related work

We start by introducing transformer models for image generation. Then, we motivate the use of the VQ representation and bidirectional models for image manipulation.

Autoregressive image generation

The use of autoregressive transformers in the field of generative modeling [START_REF] Parmar | Image transformer[END_REF] has been made possible by two simultaneous research branches. First, the extensive deployment of attention mechanisms such as non-local means algorithms [START_REF] Buades | A non-local algorithm for image denoising[END_REF], non-local neural networks [START_REF] Wang | Non-local neural networks[END_REF], and also attention layers in GANs [START_REF] Zhang | Self-attention generative adversarial networks[END_REF][START_REF] Hudson | Generative adversarial transformers[END_REF]. Second, the development of both classifiers and generative models sequentially inferring pixels via autoregressive convolutional networks such as PixelCNN (Van Den Oord et al., 2016;Van den Oord et al., 2016). The self-attention mechanism [START_REF] Vaswani | Attention is all you need[END_REF], which now become ubiquitous in computer vision, is quickly recalled here: a sequence X ∈ R L×d , where L is length of the sequence, is mapped by a position-wise linear layer to a query

Q ∈ R L×d k , a key K ∈ R L×d k and a value V ∈ R L×d v . The self-attention layer is then: attn(Q, K,V ) = softmax( QK t √ d k )V ∈ R L×d v (4.2.1)
If autoregressive transformers allow a principled log-likelihood estimation of the data, attention layers have a complexity scaling with the square of the sequence length, a clear bottleneck to scale to high-resolution images. To reduce the size of these sequences, [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] proposed the use of discrete representation. In this framework, an encoder E, a decoder D, and a codebook/dictionary Z are learned simultaneously to represent images with a single sequence of tokens. [START_REF] Esser | Taming transformers for high-resolution image synthesis[END_REF] later trained an autoregressive model on these token sequences, stressing that high-capacity transformers can generate realistic high-resolution images. The framework consists of three steps:

1. Training simultaneously a set of encoder/decoder/codebook (E, D, Z), by combining reconstruction, commitment and adversarial losses. The reconstruction loss is a perceptual distance [START_REF] Zhang | The unreasonable effectiveness of deep features as a perceptual metric[END_REF]. The commitment loss [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] pushes the codebook towards the output of the encoder using a quantization loss. The adversarial loss is the Vanilla GANs loss defined in [START_REF] Goodfellow | Generative adversarial nets[END_REF]. The training objective becomes :

E ⋆ , D ⋆ , Z ⋆ = arg min E,D,Z [L rec. (E, D, Z) + L commit. (E, Z) + λ L adv. ({E, D, Z})]. ( 4

Bidirectional attention

The main property of autoregressive models is that they only perform attention on previous tokens, making them inadequate when dealing with image manipulation (Esser et al., 2021a). Some works alleviate this bias in different ways. [START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF] [START_REF] Hoogeboom | Argmax flows and multinomial diffusion: Learning categorical distributions[END_REF] in the discrete latent space of VQGAN. Each generated sequence is conditioned on the previous one and performs attention to the whole image. However, this method is computationally heavy since it requires making N × L inferences, where N is the number of generated sequences and L is the number of tokens in the sequence. A more efficient way to perform bidirectional attention for image generation has been proposed in MaskGIT [START_REF] Chang | Maskgit: Masked generative image transformer[END_REF]. MaskGIT consists of training with a BERT-like objective [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] on sequences randomly perturbed with [MASK] tokens, and generating images with a parallel decoding scheme. Similarly, [START_REF] Zhang | M6-ufc: Unifying multi-modal controls for conditional image synthesis[END_REF] propose to use a masking-based strategy to perform conditional image editing with bidirectional attention mechanisms. However, they still require specific conditional data to learn their model editing model. We argue that by performing bidirectional attention over all the tokens and learning with a denoising objective (tokens perturbed by randomization instead of [MASK] tokens), it is possible to train a single model tackling many editing tasks.

Unifying image manipulation

Initially, image manipulation methods were implemented without any trainable parameters. Image completion was tackled using nearest-neighbor techniques along with a large dataset of scenes [START_REF] Hays | Scene completion using millions of photographs[END_REF]. As to image insertion, blending methods were widely used, such as the Laplacian pyramids [START_REF] Burt | The laplacian pyramid as a compact image code[END_REF]. In recent years, image manipulation has benefited from the advances of deep generative models. A first line of work has consisted of gathering datasets of corrupted and target images to train conditional generative models.

By doing so, one can therefore learn a mapping from any corrupted image to a real one. For example, [START_REF] Liu | Deflocnet: Deep image editing via flexible low-level controls[END_REF] proposes an encoder-decoder architecture for sketch-guided image inpainting. However, in all cases, a dataset with both types of images is required, therefore limiting the applicability.

To avoid this dependency, a second idea -known as GAN inversion methods -leverages pre-trained unconditional GANs. They work by projecting edited images on the manifold of real images learned by the pre-trained GAN. It can be solved either by optimization [START_REF] Abdal | Image2stylegan: How to embed images into the stylegan latent space[END_REF][START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF][START_REF] Daras | Intermediate layer optimization for inverse problems using deep generative models[END_REF], or with an encoder mapping to the latent space [START_REF] Chai | Using latent space regression to analyze and leverage compositionality in gans[END_REF][START_REF] Richardson | Encoding in style: a stylegan encoder for image-to-image translation[END_REF][START_REF] Tov | Designing an encoder for stylegan image manipulation[END_REF]. Pros of these GAN-based methods are that one benefits from the outstanding properties of StyleGan, state-of-the-art in image generation. However, these methods rely on a task-specific loss function that needs to be defined and optimized. More recently, another line of research is based on the development of score-based models [START_REF] Song | Score-based generative modeling through stochastic differential equations[END_REF]: [START_REF] Meng | SDEdit: Guided image synthesis and editing with stochastic differential equations[END_REF] use Langevin dynamics for image edition, and (Esser et al., 2021a) combine discrete diffusion models [START_REF] Hoogeboom | Argmax flows and multinomial diffusion: Learning categorical distributions[END_REF][START_REF] Austin | Structured denoising diffusion models in discrete state-spaces[END_REF] with the discrete vector-quantized representations from VQGANs.

Motivating EdiBERT for image editing

This section gives a global description of the proposed EdiBERT model. We start with notations before describing the different steps leading to the BERT-based edition.

Discrete auto-encoder VQGAN

Let I be an image with width w, a height h, and a number c of channels. I thus belongs to R h×w×c . Let (E, D, Z) be respectively the encoder, decoder, and codebook defined in VQVAE and VQGAN [START_REF] Van Den Oord | Neural discrete representation learning[END_REF][START_REF] Esser | Taming transformers for high-resolution image synthesis[END_REF]. The codebook Z consists of a finite number of tokens with fixed vectors in an embedding space: Z = {t 1 , . . . ,t N } with t k ∈ R d and N being the cardinality of the codebook.

For any given image I, the encoder E outputs a vector E(I) ∈ R L×d , which is then quantized and reshaped into a sequence s of length L as follows:

s = (arg min z∈Z ∥E(I) 1 -z∥, . . . , arg min z∈Z ∥E(I) L -z∥)) = Q Z (E(I)), (4.2.3)
where E(I) l = E(I) l,: ∈ R d is the feature vector of E(I) at position l, and Q Z refers to the quantization operation using the codebook Z. Recall that, after the quantization step, one gets a sequence composed of L codebook elements, thus s ∈ Z L . After we feed the codebook embeddings to the decoder D, the reconstructed image becomes Î = D(Q Z (E(I))).

Let's note D, the available image dataset. From a pre-trained encoder E and codebook Z, one can transform the image dataset D into a dataset of token-sequences D S := {Q Z (E(I)), I ∈ D}. When learning transformers on sequences of tokens, the practitioner is directly working with D S .

Learning sequences with autoregressive models

The following sections aim at motivating the training objective for the EdiBERT model. To begin with, let p θ be a transformer model parameterized with Θ trained on D S . For each position i in s, we note p i θ (.|s), the modeled distribution of tokens conditionally to s. When training an autoregressive transformer on the discrete sequences of tokens D S [START_REF] Esser | Taming transformers for high-resolution image synthesis[END_REF], one needs to compute the likelihood p θ (s) of each given sequence s = (s 1 , . . . , s L ) ∈ D S as follows: (4.2.5)

p θ (s) = L ∏ i=1 p i θ (s i |s <i ), with s <i = (s 1 , . . . , s i-1 ). ( 4
Limitations of the model. If this setting is well suited for unconditional image generation, it is ill-posed for image manipulation tasks, as shown by Esser et al. (2021a). In the case of scribble-based editing, or inpainting, one wants to resample tokens conditionally to the whole image, so that the model has all the information at its disposal.

A unique training objective for EdiBERT.

Let us define the training objective for EdiBERT. For a sequence s = (s 1 , ..., s L ), a function ϕ randomly selects a subset of k indices {ϕ 1 , ..., ϕ k } where ϕ k < L. At each selected position ϕ i , a perturbation is applied on the token s ϕ i . We attribute a random token with probability p, or keep the same token with probability 1p. Consequently, the perturbed token sϕ i becomes:

sϕ i = U(Z) with probability p, sϕ i = s ϕ i with probability 1 -p,
where U(Z) refers to the uniform distribution on the space of tokens Z. Similarly to [START_REF] Bao | BEit: BERT pre-training of image transformers[END_REF], the sampling function ϕ is defined with a 2D selection strategy, and the positions are selected by drawing random 2D rectangles, see in Figure 4.2. Contrarily to [START_REF] Bao | BEit: BERT pre-training of image transformers[END_REF] and [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], we only use random tokens from the codebook but no [MASK] tokens. We argue this setting corresponds more to the cases of denoising and editing, where tokens have to be sampled conditionally to an entire perturbed sequence.

Let us now call s = (s 1 , . . . , sϕ 1 , . . . , sϕ k , . . . , s L ) the perturbed sequence, and Ds = { s, s ∈ D} the perturbed dataset. The training of EdiBERT optimizes the following objective :

arg max θ ∈Θ E s∈ Ds 1 k k ∑ i=1 log p i θ (s ϕ i | s). (4.2.6)
Contrary to (4.2.5), we note that the objective in (4.2.6) does not require a causal left-to-right attention. Instead, the attention can be performed over the whole input sequence.

Sampling from EdiBERT: [START_REF] Wang | Bert has a mouth, and it must speak: Bert as a markov random field language model[END_REF] show that it is possible to generate realistic samples with a BERT model starting with random initialization. However, compared with standard autoregressive language models, the authors stress that BERT generations are more diverse but of slightly worse quality. Building on the findings of [START_REF] Wang | Bert has a mouth, and it must speak: Bert as a markov random field language model[END_REF], we do not aim to use BERT for pure unconditional sequence generation but rather improve an already existing sequence of tokens. In our defined EdiBERT model, for any given position i ∈ s, a token will be sampled according to the multinomial distribution p i θ (.|s).

On the locality of Vector Quantization encoding

In this section, we argue that one of the main advantages of EdiBERT comes from the VQ latent space proposed by [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] where each image is encoded in a discrete sequence of tokens. In this section, we illustrate with simple visualizations the property of this VQGAN encoding. We explore the spatial correspondence between the position of the token in the sequence and a set of pixels for the encoded image. We aim at answering the following question: do local modifications of the image lead to local modifications of the latent representation and vice versa?

Modifying the image. To answer this question, images are voluntarily perturbed with grey masks (i -→ i m ). Then, we encode the two images, quantize their representation using a pretrained codebook, and plot the distance between the two latent representations:

∥Q Z (E(i)) - Q Z (E(i m ))∥ 2 2 .
The results are shown in the first row in Figure 4.3. Due to the large receptive field of the encoder, tokens can be influenced by distant parts of the image: the down-sampled Modifying the latent space via the image Modifying the image via the latent space Fig. 4.3 Each token in the sequence is tied to a small spatial area in the decoded image. In the 1st row: we voluntarily perturb images and display the variations among the tokens in the latent space. The heatmaps represent the distance (red is high) between the tokens of the original image and the tokens of the perturbed image. In the 2nd row: we stress how collages of images can easily be done with this discrete latent representation: third and fourth images are generated by the decoder from a latent space collage. mask does not recover all of the modified tokens. However, tokens that are largely modified are either inside, or very close to the down-sampled mask.

Modifying the latent space. To understand the correspondence between tokens and pixels, we stress how one can easily manipulate images using the discrete latent space. In Figure 4.3, we show that cutting a specific area of a source image to insert it in a different location of another image is possible only by replacing the corresponding tokens in both sequences. This spatial correspondence between VQGANs' latent space and the image is interesting for localized image editing tasks, i.e. tasks that require modifying only a subset of pixels without altering the other ones.

On the reconstruction capabilities of Vector Quantization encoding

A limit of the framework resides in the use of the vector quantization operation and the induced loss of information. Indeed, we observe in Figure 4.4 that VQGAN struggles to reconstruct highfrequency details, for example complex backgrounds on FFHQ dataset [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF]. To improve the reconstruction capabilities of VQGANs, we propose a simple optimization procedure over the latent space vectors. On the left, we see that both the L1 and perceptual loss (LPIPS) between original and reconstructed images significantly decrease when optimizing LPIPS over the latent vectors of VQGAN. This may be a consequence of a higher number of dimensions spanned by the latent vectors (on the right), after the optimization (allowing for more complex reconstructions).

The objective is to find the latent vectors that minimize the LPIPS [START_REF] Zhang | The unreasonable effectiveness of deep features as a perceptual metric[END_REF]) between the target image and the decoded reconstruction. We initialize the procedure from the output of the encoder E(I), and optimize the objective with gradient descent. Figure 4.4 shows how this procedure improves the inversion capabilities of VQGAN to make it better than GAN inversion methods [START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF]. A potential explanation of the limited reconstruction capabilities of VQGAN is displayed in Figure 4.5: the latent vectors of the codebook might suffer from a very low rank. The optimization procedure seems to solve this since the latent vectors span much more dimensions of the embedding space after a few hundred optimization steps. Fig. 4.6 Image denoising with EdiBERT: the color in the 4 different heatmaps is proportional to the negative likelihood of the token. Tokens with a lower likelihood appear in red in the heatmap and have a higher probability of being sampled and edited. Consequently, conditional distributions output by EdiBERT are an efficient tool to detect anomalies and artifacts in the image.

measures fidelity while coverage measures diversity. Finally, the FID [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF] quantifies the distance between generated and target distributions. Moreover, we perform a user study on FFHQ image compositing. More details and quantitative results on LSUN Bedroom are presented in Appendix.

Localized image denoising

Image denoising aims to improve the quality of a pre-generated image or improve a locally perturbed one. The model has to work without information on the localization of the perturbations. This means we need to find and replace the perturbed tokens with more likely ones to recover a realistic image. Thus, given a sequence s = (s 1 , . . . , s L ), we want to:

1. Detect the tokens that do not fit properly in the sequence s.

2. Change them for new tokens increasing the likelihood of the new sequence.

We desire a significantly more likely sequence with as few as possible token amendments. To do so, we measure the likelihood of each token s i based on the whole sequence s, aiming to compute p(s i |s), and replace the least-probable tokens considering them independently. That is, we propose to use the conditional probability output by the model in order to detect and sample the less likely odd tokens. Some examples of image denoising are presented in Figure 4.6, and we observe that EdiBERT is able to detect artifacts and replace them with more likely tokens. The full algorithm is given in the Algorithm 3.

Ref. image

EdiBERT ⋆ LC I2SG † + + Com-GAN Fig. 4.7 Image inpainting comparisons on FFHQ. EdiBERT performs better than inversion methods such as LC [START_REF] Chai | Using latent space regression to analyze and leverage compositionality in gans[END_REF] and I2SG [START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF]. Note that Com-GAN [START_REF] Zhu | In-domain gan inversion for real image editing[END_REF] is specialized for image inpainting and was trained on pair datasets (masked image, target image), it can not perform other image editing tasks. In this setting, we have access to a masked image i m ∈ R h×w×c along with the location of the binary mask m ∈ R h×w . i m has been obtained by masking an image i ∈ R h×w×c as follows: i m = i ⊙ m with ⊙ pointwise multiplication. The goal of image inpainting is to generate an image î that is both realistic (high density) and conserves non-masked parts, that is

î ⊙ (1 -m) = i ⊙ (1 -m).
Among the different tasks in image manipulation, image inpainting stands out. Indeed, when masking a specific area of an image, one shall not consider the pixels within the mask to recover the target image. The image inpainting task thus requires specific care to reach a state-of-the-art performance; this is why we added five different elements to our approach, and validated these elements with visual results in Figure 4.8.

1. Randomization: to erase the mask influence, the tokens within the mask are given random values.

2. Dilation of the mask: as shown in Figure 4.3, some tokens outside of the down-sampled mask in the latent space are also impacted by the mask on the image. Modifying only tokens inside the down-sampled mask might not be enough and could lead to images with irregularities on the borders. As a solution, we apply a dilation on the down-sampled mask and show in Figure 4.8 that it helps better blend the target image's completion since the boundaries are removed.

3. Spiral ordering: since there is no pre-defined ordering of positions in EdiBERT, one can look for an optimal sampling of positions. We argue that by sampling positions randomly, one does not fully leverage the spatial location of the mask. Instead, we propose a spiral ordering that goes from the border to the inside of the mask. Qualitative and quantitative results in Figure 4.8 and Table 4.2 confirm the advantage of this ordering.

4. Periodic image collage: to preserve fidelity to the original image, we periodically perform a collage between the masked image and the decoded image. We observed in Figure 4.8, that without this collage trick, the reconstruction can diverge too much from the input image.

5. Online optimization on latent sequences: to improve fidelity to the masked image i m , the final stage of the algorithm consists in an optimization procedure on the latent sequence s ∈ R h×w×d . The objective function is defined as:

L = L p (D(s) -i m ) ⊙ m + L p (D(s) -D(s 0 )) ⊙ (1 -m) (4.2.7)
where L p is a perceptual loss [START_REF] Zhang | The unreasonable effectiveness of deep features as a perceptual metric[END_REF], and s 0 is the initial sequence from EdiBERT. Intuitively, the first term enforces the decoded image to get closer to the masked image i m , while the second term is a regularization enforcing the decoded image to stay similar to the completion proposed by the transformer's likelihood. We illustrate in Figure 4.5 and Figure 4.8 that the optimization leads to a better-preserved source image.

Analyzing the results: we see from Table 4.1 and Figure 4.7 that the specialized method com-GAN [START_REF] Zhao | Large scale image completion via co-modulated generative adversarial networks[END_REF] outperforms non-specialized methods on image inpainting. This was expected since it is the only method that has been trained specifically for this task. Note that the trained model co-mod GAN cannot be used in any other image manipulation task.

Compared with the non-specialized method, EdiBERT always provides better fidelity to the source image (lower Masked L1) and realism (best FID and top-2 density). An ablation study is available in Table 4.2 and validates our choices. Finally, more details regarding the sampling algorithm for the task of inpainting are given in Appendix.

Image composition

In this setting, we have access to a non-realistically edited image i e ∈ R h×w×c . The edited image i e is obtained by a composition between a source image i s ∈ R h×w×c and a target image i t ∈ R h×w×c . The target image can be a user-drawn scribble or another real image in the case of image compositing. Besides, pixels are edited inside a binary mask m ∈ R h×w , which indicates the areas modified by the user. Thus, the final edited image is computed pointwise as: Image composition aims to transform an edited image i e to make it more realistic and faithful without limiting the changes outside the mask. We note the source image i s outside the mask and the edits of the target image i m for the inserted elements in the edition mask. Three tasks fall under this umbrella: scribble-based editing, image compositing, and image crossovers.

i e = i s ⊙ m + i t ⊙ (1 -m). (4.2.8) Masked EdiBERT (a) (b) (c) (d) (e)
Results of image compositing on FFHQ are presented in Table 4.1 and Figure 4.9. EdiBERT always has the lowest masked L1. We also present the results from a user study in Table 4.1. 30 users were shown 40 original and edited images, along with four results (EdiBERT and baselines). They were asked which one is preferable, accounting for both fidelity and realism. The survey shows that on average, users prefer EdiBERT over competing approaches. We give more visual results along with the detailed answers of the user study in Appendix.

Discussions

EdiBERT is a bidirectional transformers model that can tackle multiple editing tasks from one single training. One of the key elements of the proposed method is that it does not require having access to paired datasets (source, target), or unpaired image datasets. This property shows how flexible EdiBERT is and why it can be easily applied to different tasks. Overall, the proposed framework is simple and tractable: 1) train a VQGAN [START_REF] Esser | Taming transformers for high-resolution image synthesis[END_REF], 2) train an EdiBERT model following the objective defined in (4.2.6). Interestingly, for simple applications, one can directly train EdiBERT based on the representations output by the VQGAN pre-trained on ImageNet. However, for more complex data or when dealing with multiple domains, one might have to train a specialized codebook, which requires a large auto-encoder and a lot of data. Another EdiBERT's drawback is related to the core interest of image editing. Since the tokens are predominantly localized, EdiBERT is perfectly suited for small manipulations that only require amending a few numbers of tokens.

Source Composite EdiBERT (⋆) ID-GAN I2SG † ++ Source Composite EdiBERT (⋆) ID-GAN I2SG
Fig. 4.9 Scribble-based editing and image compositing: comparison with ID-GAN [START_REF] Zhu | In-domain gan inversion for real image editing[END_REF] and I2SG [START_REF] Abdal | Image2stylegan: How to embed images into the stylegan latent space[END_REF]. EdiBERT preserves better the fidelity to the source image while being also able to fit the inserted object properly. This confirms the quantitative results in Table 4.1, EdiBERT seems to be leading in both fidelity and realism.

However, some manipulations such as zooms or rotations require changing large areas of the source image. In these cases, modifying a large number of tokens might be more demanding.

Conclusion

In this section, we demonstrated the possibility to perform several editing tasks by using the same pre-trained model. The proposed framework is simple and aims at making a step towards a unified model able to do any conceivable manipulation task on images. An exciting direction of research would be to extend the editing capabilities of EdiBERT to global transformations (e.g. zoom, rotation).

A parser-free virtual try-on

Abstract. The 2D virtual try-on task has recently attracted a great interest from the research community, for its direct potential applications in online shopping as well as for its inherent and non-addressed scientific challenges. This task requires fitting an in-shop cloth image on the image of a person, which is highly challenging because it involves cloth warping, image compositing, and synthesizing. Casting virtual try-on into a supervised task faces a difficulty: available datasets are composed of pairs of pictures (cloth, person wearing the cloth). Thus, we have no access to ground-truth when the cloth on the person changes. State-of-the-art models solve this by masking the cloth information on the person with both a human parser and a pose estimator. Then, image synthesis modules are trained to reconstruct the person image from the masked person image and the cloth image. This procedure has several caveats: firstly, human parsers are prone to errors; secondly, it is a costly pre-processing step, which also has to be applied at inference time; finally, it makes the task harder than it is since the mask covers information that should be kept such as hands or accessories. In this section, we propose a novel student-teacher paradigm where the teacher is trained in the standard way (reconstruction) before guiding the student to focus on the initial task (changing the cloth). The student additionally learns from an adversarial loss, which pushes it to follow the distribution of the real images. Consequently, the student exploits information that is masked to the teacher. A student trained without the adversarial loss would not use this information. Also, getting rid of both human parser and pose estimator at inference time allows obtaining a real-time virtual try-on.

Introduction

A photo-realistic virtual try-on system would provide a significant improvement for online shopping. Whether used to create catalogs of new products or to propose an immersive environment for shoppers, it could impact e-commerce and open the door for automated image-editing possibilities. Earlier work addresses this challenge using 3D measurements and model-based methods [START_REF] Guan | Drape: Dressing any person[END_REF][START_REF] Hahn | Subspace clothing simulation using adaptive bases[END_REF][START_REF] Pons-Moll | Clothcap: Seamless 4d clothing capture and retargeting[END_REF]. However, these are, by nature, computationally intensive and require expensive material, which would not be acceptable at scale for shops. Recent works aim to leverage deep generative models to tackle the virtual try-on problem [START_REF] Dong | Towards multi-pose guided virtual try-on network[END_REF][START_REF] Han | Viton: An image-based virtual try-on network[END_REF][START_REF] Jetchev | The conditional analogy gan: Swapping fashion articles on people images[END_REF]Wang et al., 2018a). CAGAN [START_REF] Jetchev | The conditional analogy gan: Swapping fashion articles on people images[END_REF]) is a U-net based Cycle-GAN [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] approach. However, this method fails to generate realistic results since such networks cannot handle large spatial deformations. In VITON [START_REF] Han | Viton: An image-based virtual try-on network[END_REF], the authors recast the virtual try-on as a supervised task. They propose to use a human parser and a pose estimator to mask the cloth in the person image and construct an agnostic person representation p ⋆ . The human parser allows segmenting the upper-body and the cloth, while the pose estimator locates the keypoints (i.e. shoulders, wrists, etc.) of the person. Then, with p ⋆ and the image of the original cloth c on a white background, they train a model in a fully supervised fashion to reconstruct p. Namely, they propose a coarse-to-fine synthesis strategy with shape context matching algorithm [START_REF] Belongie | Shape matching and object recognition using shape contexts[END_REF] to warp the cloth on the target person. To improve this model, CP-VTON (Wang et al., 2018a) incorporates a convolutional geometric matcher [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF], which learns geometric deformations (i.e. thin-plate spline transform [START_REF] Bookstein | Principal warps: Thin-plate splines and the decomposition of deformations[END_REF]) that align the cloth with the person. State-of-the-art models are based on the supervised formulation of the virtual try-on task, which has some drawbacks. Human parsers and pose estimators are trained on other datasets and thus fail in some situations (see Figure 4.10, two first rows). Retraining them on fashion datasets would require similar labels of semantic segmentation or unsupervised domain adaptation methods. Even though they would still be imperfect. Moreover, for a virtual try-on, one wants to preserve person's attributes like handbags or jewels. When constructing p ⋆ , these person's attributes are masked and can not be preserved, such as the partially masked handbag on the third row of Figure 4.10. Finally, the human parsing and pose estimation are the wall clock bottleneck of the pipeline.

In our work, we distill [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] the standard pipeline of virtual try-on composed of human parser, pose estimator, and synthesis modules in the synthesis modules. Namely, we train a student synthesizer with the outputs of a pre-trained standard virtual try-on pipeline.

To force the student to use information that is masked to the teacher, we also train the student with an adversarial loss. The distillation process allows us to remove the need for human parsing and pose estimation at inference time, which improves image quality and speeds up the computations from 6FPS to 77FPS. In Figure 4.10, we show visual results of a baseline CP-VTON, our teacher model T-WUTON and our student model S-WUTON. Since S-WUTON does not rely on human parsing, it is robust to parsing errors and preserves a person's attributes such as fingers or handbags.

Additionally, to build an efficient teacher model, we propose an improved architecture for virtual try-on, a Warping U-Net for a Virtual Try-On (WUTON). Our architecture is composed of two modules: a convolutional geometric matcher [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF] and a U-net generator with a siamese encoder, where the former warps the feature maps of the latter. The architecture is trained end-to-end, which leads to high-quality synthesized images.

We demonstrate the benefit of our method with several experiments on a virtual try-on dataset, with quantitative and visual results, and a user study.

Problem statement and related work

Given the 2D images p ∈ R h×w×3 of a person and c ∈ R h×w×3 of a clothing item, we want to generate the image p ∈ R h×w×3 where a person p wears the cloth c. The task can be separated in two parts : the geometric deformation T required to align c with p, and the refinement that fits the aligned cloth c = T (c) on p. These two sub-tasks can be modelled with learnable neural networks, i.e. spatial transformers networks ST N [START_REF] Jaderberg | Spatial transformer networks[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF] that output parameters θ = ST N(p, c) of geometric deformations, and conditional generative networks G that give p = G(p, c, θ ).

Because it would be costly to construct a dataset with {(p, c), p} triplets, previous works [START_REF] Han | Viton: An image-based virtual try-on network[END_REF]Wang et al., 2018a) propose to use an agnostic person representation p ⋆ ∈ R h×w×c where the clothing items in p are hidden but identity and shape of the persons are preserved. p ⋆ is built with pre-trained human parsers and pose estimators : p ⋆ = h(p). These use subspace methods to accelerate physics-based simulations and generate realistic wrinkles. ClothCap [START_REF] Pons-Moll | Clothcap: Seamless 4d clothing capture and retargeting[END_REF] aligns a 3D cloth-template to each frame of a sequence of 3D scans of a person in motion. However, the use of 3D scans is expensive and thus not doable for online users.

The task we are interested in is the one introduced in CAGAN (Jetchev and Bergmann, 2017) and further studied by VITON [START_REF] Han | Viton: An image-based virtual try-on network[END_REF] and CP-VTON (Wang et al., 2018a), which we defined in the problem statement. In CAGAN, the auhtors propose a cycle-GAN approach that requires three images as input: the reference person, the cloth worn by the person and the target in-shop cloth. Thus, it limits its practical uses. To facilitate the task, VITON introduces the supervised formulation of the virtual try-on, as described above. Their pipeline separates the task in sub-tasks: constructing the agnostic person representation (i.e. mask the area to replace but preserve body shape), warping the cloth and compositing the final image. Based on the agnostic person representation p ⋆ and the cloth image c, the VITON model performs a generative composition between the warped cloth and a coarse result. The warping is done with a non-parametric geometric transform [START_REF] Belongie | Shape matching and object recognition using shape contexts[END_REF]. To improve this model, CP-VTON incorporates a learnable geometric matcher ST N [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF]. The ST N is trained to align c on p with a L1 loss on paired images. However, the L1 loss is overwhelmed with the white background and the solid color parts of clothes. Thus, it faces difficulties to align patterns and to preserve inner structure of the cloth. In VTNFP [START_REF] Yu | Vtnfp: An image-based virtual try-on network with body and clothing feature preservation[END_REF] and ClothFlow [START_REF] Han | Clothflow: A flow-based model for clothed person generation[END_REF], a module generating the new human parsing is added. It allows to better preserve body parts and edges, but at an increased computational cost. Moreover, ClothFlow [START_REF] Han | Clothflow: A flow-based model for clothed person generation[END_REF] replaces the TPS warping by a dense flow from the target cloth to the person. All these recent works [START_REF] Han | Viton: An image-based virtual try-on network[END_REF]Wang et al., 2018a;[START_REF] Han | Clothflow: A flow-based model for clothed person generation[END_REF][START_REF] Yu | Vtnfp: An image-based virtual try-on network with body and clothing feature preservation[END_REF] rely on pre-trained human parser and pose estimator.

Recent work MG-VTON [START_REF] Dong | Towards multi-pose guided virtual try-on network[END_REF] extends the task to a multi-pose virtual try-on system, where they also change the pose of the reference person. Similarly to [START_REF] Dong | Soft-gated warping-gan for pose-guided person image synthesis[END_REF][START_REF] Yu | Vtnfp: An image-based virtual try-on network with body and clothing feature preservation[END_REF][START_REF] Dong | Towards multi-pose guided virtual try-on network[END_REF], they add a module generating the new human parsing, based on input and target pose information.

Our approach

Our task is to build a virtual try-on system that is able to fit a given in-shop cloth on a reference person. In this work, we build a virtual try-on that does not rely on a human parser nor a pose estimator for inference. To do so, we use a teacher-student approach to distill the standard virtual try-on pipeline composed of human parser, pose estimator, and synthesis module in the synthesis module.

In Section 4.3.3.1, we detail the architecture of our synthesis module WUTON. It is trainable end-to-end and composed of two existing modules: a convolutional geometric matcher ST N [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF] and a U-net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] with siamese encoder whose skip connections from the cloth encoder to the decoder are deformed by ST N. We then explain its training procedure in the standard supervised setting, which gives the teacher T-WUTON.

We finally explain our distillation process. Once the first generative model is trained, the pipeline {h, T-WUTON} becomes a teacher model for a student model S-WUTON by constructing synthetic triplets {(p, c), p}. These serve to supervise the training of S-WUTON, which hence does not need a human parser to pre-process the image and construct the agnostic person representation. Importantly, S-WUTON also learns from an adversarial loss so it does not only follow the teacher's distribution and it can learn to preserve a person's attributes. Our warping U-net is composed of two connected modules, as shown in Figure 4.11. The first one is a convolutional geometric matcher, which has a similar architecture as [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF]Wang et al., 2018a). It outputs the parameters θ of a geometric transformation, a TPS transform in our case. This geometric transformation aligns the in-shop cloth image with the reference person. However, in contrast to previous work [START_REF] Dong | Towards multi-pose guided virtual try-on network[END_REF][START_REF] Han | Viton: An image-based virtual try-on network[END_REF]Wang et al., 2018a), we use the geometric transformation on the feature maps of the generator rather than at a pixel-level. Thus, we learn to deform the feature maps that pass through the skip connections of the second module, a U-net generator which synthesizes the output image p.

The architecture of the convolutional geometric matcher is taken from CP-VTON (Wang et al., 2018a), which reuses the generic geometric matcher from [START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF]. It is composed of two feature extractors F 1 and F 2 , which are standard convolutional neural networks. The local vectors of feature maps F 1 (c) and F 2 (p ⋆ ) are then L2-normalized and a correlation map C is computed as follows:

C i jk = F 1 i, j (c) • F 2 m,n (p ⋆ ) (4.3.3)
where k is the index for the position (m, n). This correlation map captures dependencies between distant locations of the two feature maps, which is useful to align the two images. C is the input of a regression network, which outputs the parameters θ and allows to perform the geometric transformation T θ . We use TPS transformations [START_REF] Bookstein | Principal warps: Thin-plate splines and the decomposition of deformations[END_REF], which generate smooth sampling grids given control points. Each scale of the U-net is transformed with the same parameters θ . The input of the U-net generator is also the tuple of pictures (p ⋆ , c). Since these two images are not spatially aligned, we cannot simply concatenate them and feed a standard U-net. To alleviate this, we use two different encoders E 1 and E 2 processing each image independently and with non-shared parameters. Then, the feature maps of the in-shop cloth E 1 (c) are transformed at each scale i: E i 1 (c) = T θ (E i 1 (c)). Then, the feature maps of the two encoders are concatenated and feed the decoder at each scale. With aligned feature maps, the generator is able to compose them and to produce realistic results. Feature maps warping was also proposed in [START_REF] Dong | Soft-gated warping-gan for pose-guided person image synthesis[END_REF]; [START_REF] Siarohin | Deformable gans for posebased human image generation[END_REF]. We use instance normalization in the U-net generator, which is more effective than batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] for image generation [START_REF] Ulyanov | Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis[END_REF].

Training procedure of the teacher model

We will now detail the training procedure of T-WUTON, i.e. the data representation and the different loss functions of the teacher model.

While previous works use a rich person representation with more than 20 channels representing human pose, body shape and the RGB image of the head, we only mask the upper-body of the reference person. Our agnostic person representation p ⋆ is thus a 3-channel RGB image with a masked area. We compute the upper-body mask from pose and body parsing information provided by a pre-trained neural network from [START_REF] Liang | Look into person: Joint body parsing & pose estimation network and a new benchmark[END_REF]. Precisely, we mask the areas corresponding to the arms, the upper-body cloth and a bounding box around the neck.

Using The cloth worn by the person c a,p allows us to guide directly the geometric matcher with a L 1 loss:

L warp = ∥T θ a (c a ) -c a,p ∥ 1 (4.3.4)
The image p a of the reference person provides a supervision for the whole pipeline. Similarly to CP-VTON (Wang et al., 2018a), we use two different losses to guide the generation of the final image pa , the pixel-level L 1 loss ∥ pap a ∥ 1 and the perceptual loss [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF]. We focus on L 1 losses since they are known to generate less blur than L 2 for image generation [START_REF] Zhao | Loss functions for image restoration with neural networks[END_REF]. The latter consists of using the features extracted with a pre-trained neural network, VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] in our case. Specifically, our perceptual loss is:

L perceptual = 5 ∑ i=1 ∥φ i ( pa ) -φ i (p a )∥ 1 (4.3.5)
where φ i (I) are the feature maps of an image I extracted at the i-th layer of the VGG network. Furthermore, we exploit adversarial training to train the network to fit c b on the same agnostic person representation p ⋆ a , which is extracted from a person wearing c a . This is only feasible with an adversarial loss, since there is no available ground-truth for this pair (p ⋆ a , c b ). Thus, we feed the discriminator with the synthesized image pb and real images of persons from the dataset. This adversarial loss is also back-propagated to the convolutional geometric matcher, which allows to generate much more realistic spatial transformations. We use the relativistic adversarial loss (Jolicoeur-Martineau, 2019) with gradient-penalty (Arjovsky et al., 2017;[START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF], which trains the discriminator to predict relative realness of real images compared to synthesized ones. Finally, we optimize with Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] the

L perceptual = 5 ∑ i=1 ∥φ i ( pb ) -φ i ( pb )∥ 1 (4.3.8) L 1 = ∥ pb -pb ∥ 1 (4.3.9)
Finally, the total loss of the student model is:

L = λ w L warp + λ p L perceptual + λ L 1 L 1 + λ adv L adv (4.3.10)
The adversarial loss L adv is independant from T-WUTON. Here, we also use the relativistic loss with gradient penalty on the discriminator. The real data consists of images of persons from the dataset p a , and the fake data corresponds to the synthesized images pb . Notice that without the adversarial loss, it would be a standard teacher-student setting, where the student is only guided by the outputs of the teacher. In our case, the discriminator (i.e. L adv ) helps S-WUTON to be close to the real data distribution, and not only to the teacher's distribution. As shown by the ablation study in Section 4.3.4.6, it is an important component and is necessary to make S-WUTON exploit the components that are masked from T-WUTON (e.g. hands).

Experiments and analysis

We first describe the dataset. We then compare our approach with CP-VTON (Wang et al., 2018a), a current state-of-the-art for the virtual try-on task. We present visual and quantitative results proving that S-WUTON achieves state-of-the-art results, and that the distillation process allows to improve image quality. We show that this stands for several metrics, and with a user study. We then provide a comparison of the runtime of virtual try-on algorithms on a Tesla NVIDIA V100 GPU. The teacher-student distillation allows to decrease the runtime by an order of magnitude. Finally, we outline the importance of the adversarial loss in our teacher-student setting.

We also show some visual comparisons with recent work VTNFP [START_REF] Yu | Vtnfp: An image-based virtual try-on network with body and clothing feature preservation[END_REF]. Images are taken from their paper. However, since their model is not available, we could not compute the other metrics. We provide more visual comparisons with VTNFP and ClothFlow [START_REF] Han | Clothflow: A flow-based model for clothed person generation[END_REF]

Dataset

For copyright issues, we can not use the dataset from VITON [START_REF] Han | Viton: An image-based virtual try-on network[END_REF] and CP-VTON (Wang et al., 2018a). Instead, we leverage the Image-based Multi-pose Virtual try-on dataset. This dataset contains 35,687/13,524 person/cloth images at 256x192 resolution. 4175 pairs are kept for test so the cloth was not seen during training. A random shuffle of these pairs produces the unpaired person/cloth images. For each in-shop cloth image, there are multiple images of a model wearing the given cloth from different views and in different poses. We remove images tagged as back images since the in-shop cloth image is only from the front. We process the images with a neural human parser and pose estimator, specifically the joint body parsing and pose estimation network [START_REF] Liang | Look into person: Joint body parsing & pose estimation network and a new benchmark[END_REF].

Visual results

Visual results of our method and CP-VTON are shown in Figure 4.13. On the left side, images are computed from models trained on MG-VTON dataset, with p ⋆ t-wuton representation for T-WUTON and CP-VTON for fairness. On the right side, images are taken from VTNFP paper [START_REF] Yu | Vtnfp: An image-based virtual try-on network with body and clothing feature preservation[END_REF]. There, CP-VTON and VTNFP were trained on the original dataset from VITON, and CP-VTON uses p ⋆ cp-vton . More images from S-WUTON are provided in Figure 4.10 and Figure 4.15.

CP-VTON has trouble to realistically deform and render complex patterns like stripes or flowers. Control points of the T θ transform are visible and lead to unrealistic curves and deformations on the clothes. Also, the edges of cloth patterns and body contours are blurred.

Firstly, our proposed T-WUTON architecture allows to improve the baseline CP-VTON. Indeed, our method generates spatial transformations of a much higher visual quality, which is specifically visible for stripes (1st row). It is able to preserve complex visual patterns of clothes and produces sharper images than CP-VTON and VTNFP on the edges. Secondly, we can observe the importance of our distillation process with the visual results from S-WUTON.

Since it has a non-masked image as input, it is able to preserve body details, especially the hands. Moreover, as shown in Figure 4.10, S-WUTON is robust to a bad parsing and preserves a person's attributes that are important for the virtual try-on task.

Generally, our method generates results of high visual quality while preserving the characteristics of the target cloth and of the person. However, VTNFP can surpass S-WUTON when models are crossing arms (4th row, right side), which is sometimes a failure case of our method. Note that this is not general, since on (3rd row, right side) and (4th row, left side) in Figure 4.13 and on the two last columns in Figure 4.14, models are crossing arms and S-WUTON manages to nicely compose the arms with the occluded cloth.

Quantitative results

Table 4.3 Quantitative results on paired setting (LPIPS and SSIM) and on unpaired setting (IS and FID). ± reports std. dev. T-WUTON and S-WUTON are our proposed models. The two last lines (methods with ⋆ ) are the results presented in ACGPN (Yang et al., 2020a). However, it has to be taken carefully since the experiments are conducted on another dataset. To further evaluate our method, we use four different metrics. Two are designed for the paired setting, that does not allow us to evaluate S-WUTON (because the input image is not masked), and one is for the unpaired setting. The first one for the paired setting is the linear perceptual image patch similarity (LPIPS) developed in [START_REF] Zhang | The unreasonable effectiveness of deep features as a perceptual metric[END_REF], a state-of-the-art metric for comparing pairs of images. It is very similar to the perceptual loss we use in training (see Section 4.3.3.2) since the idea is to use the feature maps extracted by a pre-trained neural network to quantify the perceptual difference between two images. Different from the basic perceptual loss, they first unit-normalize each layer in the channel dimension and then learn a rescaling that match human perception.

Method

Such as previous works, we also use the structural similarity (SSIM) [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF] in the paired setting, inception score (IS) [START_REF] Salimans | Improved techniques for training GANs[END_REF] and Fréchet Inception Distance (FID) [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF] in the unpaired setting. CP-VTON (A) (Wang et al., 2018a) refers to CP-VTON trained with their agnostic person representation p ⋆ cp-vton (20 channels with RGB image of head and shape/pose information), while CP-VTON (B) refers to CP-VTON trained with p ⋆ t-wuton . Results are reported in Table 4.3.

User study

We perform A/B tests on 7 users. Each one has to vote 100 times between CP-VTON and S-WUTON synthesized images, given reference person and target cloth. The user is asked to choose for the most realistic image, that preserves both person and target cloth details. The selected 100 images are a random subset of the test set in the unpaired setting. This subset is sampled for each user and is thus different for each user. There is no time limit for the users. Let us denote p the probability that an image from S-WUTON is preferable to an image from CP-VTON. The users choose our method 88% of the time. In terms of statistical significance, it means that we can say p > 0.85 with a confidence level of 98.7%.

ClothFlow and VTNFP also performed user studies where they compare to CP-VTON. The authors respectively report that users prefer their method 81.2% and 77.4% of the time. Note that the experiment was not performed in the same setting (dataset, number of users, number of pictures per user).

Runtime analysis

In Table 4.4, we compare the runtime of our method to CP-VTON, ClothFlow and VTNFP. Note that the running times are estimated on a NVIDIA V100 GPU. For the human parsing and pose estimation networks, we use state-of-the-art models from [START_REF] Liang | Look into person: Joint body parsing & pose estimation network and a new benchmark[END_REF]. These are based on shared neural backbones for the two tasks, which accelerates the computations. The try-on architecture of T-WUTON and S-WUTON is slightly slower than that of CP-VTON, due to the non-shared encoder and the warping at each scale of the U-Net. However, with S-WUTON we remove the wall clock bottleneck of virtual try-on system, which is the human parsing and pose estimation. Doing so, we decrease by an order of magnitude the runtime of virtual try-on algorithms, from 6FPS to 77FPS.

We include comparisons with VTNFP and ClothFlow in the Table 4.4. Indeed, both models use human parsing and pose estimation. For VTNFP, they add a module on top of CP-VTON architecture, so their try-on architecture takes at least 9ms per image. For ClothFlow, the use of human parser and pose estimator gives a lower bound on the total runtime. We show the impact of the adversarial loss on S-WUTON.

We train a variant student model S-WUTON without the adversarial loss. We provide a comparison of synthesized images in Figure 4.14, and IS and FID scores in Table 4.5.

The adversarial loss on the student model is a constraint to make the student model closer to the real data distribution and to not only follow the teacher's distribution. Without the adversarial loss, the student model does not preserve person's attributes, even though they are not masked.

Conclusion

In this work, we propose a teacher-student setting to distill the standard virtual try-on pipeline and refocus on the initial task: changing the cloth of a non-masked person. This leads to a significant computational speed-up and largely improves image quality. Importantly, this allows to preserve person's attributes such as hands or accessories, which is necessary for a virtual try-on. Chapter 5

Conclusion

Summary of contributions

In this thesis, we contribute to a better understanding of the behavior of deep generative models on disconnected distributions, which are prevalent in real-world datasets containing various classes and modes. This research setting is crucial for advancing deep generative models. Chapter 3 formalizes a fundamental limitation of some generative models, such as GANs and VAEs, that generate samples outside the target modes because of the continuous generated distribution and disconnected target distribution. Based on our analysis, we propose a truncation method that improves sampling quality. Second, we demonstrate that when dealing with disconnected distribution, there exists an optimal geometry of the latent space which minimizes the proportion of generated samples lying outside the target modes. This geometry is characterized by linearity and convexity of the different regions and can be enforced in GANs to enhance their performance. Finally, to keep on reducing these off-manifold generated samples, we define a rejection mechanism from pre-trained generators. To do so, we train an MLP network to predict importance weights in the latent space. This method can be used on top of any pre-trained generator and helps select latent space areas corresponding to high-quality samples. Our findings highlight the importance of considering the multi-modal nature of data to design methods that improve the sampling quality of deep generative models.

The second research axis focuses on developing generative models for image editing purposes with a view to reducing the need for supervised data, which is often expensive to collect. Our work demonstrates that the use of deep generative models can be effective in this direction. In section 4.2, we propose a transformer-based generative model that handles various editing tasks such as local denoising, image inpainting, and scribble-based editing. The model is trained with a self-supervised objective of predicting randomized tokens, enabling it to learn how to modify parts of an image conditionally to the whole image. In section 4.3, we address the image-based virtual try-on task and propose a teacher-student approach with adversarial learning to create an ideal synthetic dataset with unmasked input images. We show that this approach outperforms the suboptimal method of relying on pre-trained neural networks for human segmentation and human parsing.

Furthermore, the questions and contributions of this thesis bring interesting future research questions. We detail some of them below.

Future work

The approaches and methods developed in this thesis can shed light on recent developments in deep learning and deep generative modelling. This opens the way for future research. We describe below some interesting ideas that would be interesting follow-ups:

Disconnected latent space versus disconnected function modelling. In this thesis, we have operated under the assumption that neural networks are Lipschitz and continuous functions. To improve the sampling quality of generative models, we focused on designing the latent space and derived methods to make it non-connected, such as through different types of truncation. However, an alternative approach would be to model disconnected functions directly. This raises the question of what design choices can be made to model disconnected functions. One promising direction for investigation is the use of Sparse Mixture-of-Experts (SMoE) models, which have achieved impressive performance on large-scale discriminative tasks in Natural Language Processing [START_REF] Shazeer | Outrageously large neural networks: The sparsely-gated mixture-of-experts layer[END_REF][START_REF] Fedus | Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity[END_REF] and, more recently, in Computer Vision [START_REF] Riquelme | Scaling vision with sparse mixture of experts[END_REF]. Some theoretical advances have suggested that their disconnected nature may be responsible for their strong performance on datasets with multiple clusters [START_REF] Chen | Towards understanding the mixture-ofexperts layer in deep learning[END_REF]. Exploring the use of SMoE layers for push-forward generative models, both theoretically and experimentally, would be of great interest, as it would enable these models to design disconnected distributions and overcome issues of misspecification. This opens up exciting possibilities for future research in generative modeling.

Latent space of score-based models. In this work, particularly in Section 3.3, we have demonstrated the structured nature of GANs' latent space. When learning multimodal data, different modes are clustered in linear regions of the latent space, and linear interpolations between generated samples follow a reasonably smooth path on the data manifold. In contrast, in score-based models, the interpolations are much more chaotic, and the latent space has less structure. This raises a fundamental question: why do score-based models achieve better performance than GANs? Several hypotheses can be proposed. First, the iterative generation procedure of score-based models could make their Lipschitz constant greater than that of GANs. In fact, most attempts to distill these models show that performance decreases as the number of generation steps decreases [START_REF] Salimans | Progressive distillation for fast sampling of diffusion models[END_REF]. Second, the latent space has the same dimension as the data space, making it more difficult to structure due to common issues in high-dimensional problems. Finally, the objective function, which is a simple L2 loss, may be more stable and have a better optimization landscape, leading to better-trained networks. Further investigation is required to fully understand the reasons behind the superior performance of score-based models over GANs.

Neural collapse in deep generative models. In Section 3.3, we establish a connection between the neural collapse phenomenon and the latent space of push-forward generative models. The neural collapse denotes the geometric structure that emerges in the final layer of deep classification networks upon achieving zero training loss. Notably, [START_REF] Papyan | Prevalence of neural collapse during the terminal phase of deep learning training[END_REF] have demonstrated that features in the ante-penultimate layer tend to converge towards class means, which are arranged in a regular simplex configuration featuring equidistant and maximum equiangular relationships. If we consider the class means as the seeds of a Voronoi partition, they would form a simplicial cluster that represents the optimal partition of the latent space derived in Section 3.3.

In our work, we limit the analysis of this geometrical structure to the latent space of deep generative models. However, if this structure arises in the latent space, it is very likely to be propagated through the layers of the generator. Is the generator further separating the modes in its feature space? This seems like a reasonable hyptohesis. First, this would explain methods using feature space of generators as a feature extractor for tasks such as image segmentation. Second, it could allow to exploit the feature space geometrical structure to design rejection mechanisms. We have some preliminary evidence for this. When learning latent importance weights in Section 3.4, we used the intermediate feature space of StyleGAN and it gave good performance for predicting sample quality. However, as we go through the generator's layers, up-sampling layers increase the dimensionality of the features, and we approach the data manifold which often has a highly non-linear structure. Thus, this study would raise some technical challenges.

Lemma A.1.2. Let µ be a probability distribution associated with a uniformly continuous probability density function f µ . Assume that there exists constants a 1 > 0, a 2 > 0 such that for all x ∈ E, we have a

1 < f µ (x) ⩽ a 2 .
x∈E ∥ f ϕ(n) n (x) -f µ (x)∥ → 0. where f ϕ(n) n (x) = ϕ(n) nV d ∥x-x ϕ(n) ∥ d with V d being the volume of the unit ball in R d . Let ε > 0 such that ε < a 1 /2.
There exists N ∈ N such that for all n ⩾ N, we have, almost surely, for all x ∈ E:

a 1 -ε ⩽ f ϕ(n) n (x) ⩽ a 2 + ε a 1 -ε ⩽ ϕ(n) nV d ∥x -x ϕ(n) ∥ d ⩽ a 2 + ε
Consequently, for all n ⩾ N, for all x ∈ E almost surely:

∥x -x ϕ(n) ∥ ⩽ ϕ(n) nV d (a 1 -ε) 1/d Thus, sup x∈E ∥x -x ϕ(n) ∥ → 0 a.s.. Also, almost surely n∥x -x ϕ(n) ∥ d ⩾ ϕ(n) V d (a 2 + ε) Thus, inf x∈E ∥x -x ϕ(n) ∥ → ∞ a.s..
Lemma A.1.3. Let µ, ν be two probability distributions associated with uniformly continuous probability density functions f µ and f ν . Assume that there exists constants a 1 > 0, a 2 > 0 such that for all x ∈ E, we have a 1 < f µ (x) ⩽ a 2 and a 1 < f ν ⩽ a 2 . Also, let D µ , D ν be datasets sampled from ν n , µ n . If µ is an estimator for ν, then

(i) for all x ∈ D µ , α n ϕ(n) (x) → n→∞ 1 supp(ν) (x) in proba. (ii) for all y ∈ D ν , β n ϕ(n) (y) → n→∞ 1 supp(µ) (x) in proba.
Proof. We will only show the result for (i), since a similar proof holds for (ii). Thus, we want to show that

for all x ∈ D µ , α n ϕ(n) (x) → n→∞ 1 supp(ν) (x) a. s.
First, let's assume that x / ∈ S ν . Biau and Devroye (2015, Lemma 2.2) have shown that

lim n→∞ ∥x (ϕ(n)) -x∥ = inf{∥x -y∥ | y ∈ S ν } a.s.
As S ν is a closed set -e.g. [START_REF] Kallenberg | Foundations of modern probability[END_REF] -we have

lim n→∞ ∥x -x (ϕ(n)) ∥ > 0 a.s.
and for all y ∈ D ν , lim n→∞ ∥yy (ϕ(n)) ∥ = 0 a.s.

Thus, lim n→∞ α n ϕ(n) (x) = 0 a.s.. Now, let's assume that x ∈ S ν . Using Definition 3.2.2, the precision of a given data point x can be rewritten as follows:

α n ϕ(n) (x) = 1 ⇐⇒ ∃y ∈ D ν , x ∈ B(y, ∥y -y (ϕ(n)) ∥)
Using notation from (A.1.1), we note

R min = min y∈ ∥y -y (ϕ(n)) ∥, R max = max y∈E ∥y -y (ϕ(n)) ∥.
It is clear that :

y∈D ν B(y, R min ) ⊆ S n ν ⊆ y∈D ν B(y, R max ), (A.1.2)
where S n ν = y∈D ν B(y, ∥yy (ϕ(n)) ∥)).

Besides, combining Lemma A.1.2 with Devroye and Wise (1980, Theorem 1), we have that:

ν(S ν ∆ y∈D ν B(y, R min )) -→ n→0 0 in proba. ν(S ν ∆ y∈D ν B(y, R max )) -→ n→0 0 in proba.
where ∆ here refers to the symmetric difference.

Thus, using (A.1.2), it is now clear that, µ(S ν ∆ S n ν ) → 0 in probability. Finally, given x ∈ S µ , we have µ

(x ∈ S n ν ) = ν(α n ϕ(n) (x) = 1) → 1 in probability.
We can now finish the proof for Theorem 3.2.1. Recall that ᾱ = µ S ν and similarly, β = ν S µ .

Proof. We have that

|α n ϕ(n) -ᾱ| = | 1 n ∑ x i ∈D µ α n ϕ(n) (x i ) - E 1 x∈S ν µ(dx)| Then, |α n ϕ(n) -ᾱ| = | 1 n ∑ x i ∈D µ (α n ϕ(n) (x i ) -1 x i ∈S ν ) + 1 n ∑ x i ∈D µ 1 x i ∈S ν - E 1 x∈S ν µ(dx) | = |E x i ∼µ n (α n ϕ(n) (x i ) -1 x i ∈S ν ) (A.1.3) + E µ n 1 S ν -E µ 1 S ν | (A.1.4)
where µ n is the empirical distribution of µ. As µ n converges weakly to µ almost surely (e.g. Dudley (2004, Theorem 11.4.1)) and since 1 x∈S ν is bounded, we can bound (A.1.4) as follows:

lim n→∞ E x∼µ n 1 x∈supp(µ) -E x∼µ 1 x∈supp(µ) = 0 a. s.
Now, to bound (A.1.3), we use the fact that for any x ∈ D µ , the random variable α n ϕ(n) (x) converges to 1 x∈S ν in probability (Lemma A.1.3) and that for all x ∈ D µ , both α n ϕ(n) (x) ⩽ 1 and 1 x∈S ν ⩽ 1. Consequently, using results from the weak law for triangular arrays, we have that

lim n→∞ 1 n ∑ x i ∈D µ (α n ϕ(n) (x i ) -1 x i ∈S ν ) = 0 in proba. Finally, |α n ϕ(n) -ᾱ| → n→∞ 0 in proba.,
which proves the result. The same proof works for lim

k→∞ β n k = β . γ (G -1 θ (M 1 )) ε + γ (G -1 θ (M 2 )) ε ⩾ 2Φ Φ -1 ( α 2 ) + ε (using Theorem 1.3 from Ledoux (1996)) ⩾ α + 2ε √ 2π e -Φ -1 ( α 2 ) 2 /2 (since Φ -1 ( α 2 ) + ε < 0 and Φ convex on ] -∞, 0])
Thus, we have that

α + 2ε √ 2π e -Φ -1 ( α 2 ) 2 /2 ⩽ 1
Thus, by noting

α ⋆ = sup{α ∈ [0, 1] | α + 2ε √ 2π e -Φ -1 ( α 2 ) 2 2 ⩽ 1},
we have our result.

For α ⩾ 3/4. By noting α = 1x, we have

Φ -1 ( α 2 ) = √ 2πx 2 + O(x 3 )
And, e

-Φ -1 ( α 2 ) 2 2 = e -πx 2 4 + O(e -x 4
)

Thus, 1 -x + 2ε √ 2π e -πx 2 4 + O(e -x 4 ) ⩽ 1 ⇐⇒ x ⩾ 2ε √ 2π e -πx 2 4 + O(e -x 4 ) =⇒ x ⩾ 2 π W (ε 2 )
where W is the product log function. Thus, α ⩽ 1 -2 π W (ε 2 ).

As an example, in the case where ε = 1, we have that W (1) ≈ 0.5671, x > 0.4525 and α < 0.5475.

A.1.4 Proof of Theorem 3.2.3

A.1.4.1 Equitable setting

This result is a consequence of Theorem A.1.1 that we will assume true in this section.

We consider that the unknown true distribution µ ⋆ lays on M disjoint manifolds of equal measure. As specified in Section 3.2.3, the latent distribution γ is a multivariate Gaussian defined on R d . For each k ∈ [1, M], we consider in the latent space, the pre-images A k .

It is clear that A 1 , . . . , A M are pairwise disjoint Borel subsets of R d . We denote M, the number of classes covered by the estimator µ θ , such that for all i ∈ [1, M], we have γ(A i ) > 0. We know that M ⩾ M β > 1.

For each i ∈ [1, M], we denote A ε i , the ε-enlargement of A i . For any pair (i, j) it is clear that A ε i A ε j = 0 where ε = D 2L (D being the minimum distance between two sub-manifolds and L being the Lipschitz constant of the generator).

As assumed, we know that A ε i , i ∈ [1, M] partition the latent space in equal measure, consequently, we assume that

n ∑ i=1 γ(A ε i ) = 1 and γ(A 1 ) = . . . = γ(A M) = 1/ M (A.1.5) Thus, we have that ᾱ = M ∑ i=1 γ(A ε i ) = 1 -γ(∆ -ε (A ε 1 , . . . , A ε M))
Using Theorem A.1.1, we have

γ(∆ -ε (A ε 1 , . . . , A ε n )) ⩾ 1 - 1 + x 2 x 2 e -1 2 ε 2 e -εx Thus, ᾱ ⩽ 1 + y 2 y 2 e -1 2 ε 2 e -εy
where y

= Φ -1 1 -max k∈[ M] γ(A ε k ) = Φ -1 ( M-1 M ) and Φ(t) = t -∞ exp(-t 2 /2) √ 2π
ds.

Knowing that M ⩾ β M we have that

Φ -1 (1 - 1 M ) ⩾ Φ -1 (1 - 1 β M )
We conclude by saying that the function x → 1+x 2

x 2 e -εx is decreasing for x > 0. Thus, A.1.6) where

ᾱ ⩽ 1 + y 2 y 2 e -1 2 ε 2 e -εy (
y = Φ -1 (1 -1 β M ) and Φ(t) = t -∞ exp(-t 2 /2) √ 2π
ds.

and let A 1 , . . . , A K be K ≥ 2 pairwise disjoint Borel subsets of R d whose union has unit (i.e full) Gaussian measure ∑ K k=1 w k = 1, where w k := γ(A k ). Such a collection {A 1 , . . . , A K } will be called an (w 1 , . . . , w K )-partition of standard d-dimensional Gaussian space (R d , γ).

For each k ∈

[[K]], define the compliment A -k := ∪ k ′ ̸ =k A k ′ , and let ∂ -ε A k := {z ∈ A k | dist(z, A -k ) ≤ ε} be the inner ε-boundary of A k , i.e the points of A k which are within distance ε of some other A k ′ . For every (k, k ′ ) ∈ [[K]] 2 with k ′ ̸ = k, it is an easy exercise to show that ∂ -ε A k ∩ ∂ -ε A k ′ = / 0 (A.1.8) ∂ -ε A k ∩ A -k = / 0 A ε -k = ∂ -ε A k ∪ A -k Now, let ∆ -ε (A 1 , . . . , A K ) := ∪ K k=1 ∂ -ε
A k be the union of all the inner ε-boundaries. This is ∆ -ε (A 1 , . . . , A K ) the set of points of ∪ K k=1 A k which are on the boundary between some two distinct A k and A k ′ . We want to find a lower bound in the measure γ(∆ -ε (A 1 , . . . , A K )).

Theorem A.1.1. Given K ≥ 4 and w 1 , . . . , w K ∈ (0, 1/4] such that ∑ K k=1 w k = 1, we have the bound:

inf A 1 ,...,A K γ(∆ -ε (A 1 , . . . , A K )) ≥ 1 - 1 + x 2 x 2 e -1 2 ε 2 e -εx
where the infinimum is taken over all (w 1 , . . . , w k )-partitions of standard Gaussian space (R d , γ), and x :

= Φ -1 1 -max k∈[[M]] w k .
Proof. By (A.1.8), we have the formula

γ(∆ -ε (A 1 , . . . , A K )) = K ∑ k=1 γ(∂ -ε A k ) (A.1.9) = K ∑ k=1 γ(A ε -k ) -γ(A -k ). (A.1.10) Let w -k := γ(A -k ) = 1 -w k , and assume w -k ≥ 3/4, i.e w k ≤ 1/4, for all k ∈ [[K]].
For example, this condition holds in the equitable scenario where w k = 1/K for all k. Now, by standard Gaussian Isoperimetric Inequality (see [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF] for example), one has .1.11) Using the bound x 1+x 2 ϕ(x) < 1 -Φ(x) < 1 x ϕ(x) ∀x > 0 where ϕ is the density of the standard Gaussian law. We can further find that

γ(A ε -k ) ≥ Φ(Φ -1 (γ(A -k ) + ε) = Φ(Φ -1 (1 -w k ) + ε). ( A
Φ(Φ -1 (1 -w k ) + ε) ≥ 1 -w k 1 + Φ -1 (1 -w k ) 2 Φ -1 (1 -w k ) 2 × e -1 2 ε 2 e -εΦ -1 (1-w k ) ≥ 1 -w k 1 + x 2 x 2 e -1 2 ε 2 e -εx > 0 (A.1.12) (since the function x → 1 + x 2
x 2 e -εx is decreasing for x > 0)

where

x := min k∈[[K]] Φ -1 (1 -w k ) = Φ -1 1 -max k∈[[K]] w k ≥ Φ -1 (3/4) > 0.
67. Combining (A.1.9), (A.1.11), and (A.1.12) yields the following

γ(∆ -ε (A 1 , . . . , A K )) ≥ K ∑ k=1 1 -w k 1 + x 2 x 2 e -1 2 ε 2 e -εx -(1 -w k )) = K ∑ k=1 1 - 1 + x 2 x 2 e -1 2 ε 2 e -εx w k = 1 - 1 + x 2 x 2 e -1 2 ε 2 e -εx ,

Asymptotic analysis

In the limit, it is easy to check that in the case where max k∈[[K]] w k -→ 0, we have that x -→ ∞. In this setting, we thus have 1+x 2 x 2 -→ 1 and can now derive the following bound: inf

A 1 ,...,A K γ(∆ -ε (A 1 , . . . , A K )) max k∈[[K]] w k →0 -→ 1 -e -1 2 ε 2 e -εx .
Equitable scenario In the equitable scenario where w k = 1/K for all k, we have inf

A 1 ,...,A K γ(∆ -ε (A 1 , . . . , A K )) ⩾ 1 - 1 + x 2 x 2 e -1 2 ε 2 e -εx
where x = Φ -1 (1 -1/K). When K ≥ 8 we have:

Φ -1 (1 -1/K) ⩾ 2 log K (q(K) 2 -1) √ 2πq(K) 3 (A.1.13)
where q(K) = 2 log( √ 2πK).

Consequently, we have when K → ∞, the following behavior:

γ(∆ -ε (A 1 , . . . , A K )) K→∞ ⩽ 1 -e -1 2 ε 2 e -ε √ 2 log(K) (A.1.14)
Proof of the inequality (A.1.13). Set p := 1/K. First, for any x > 0, we have the following upper:

∞ x e -y 2 /2 dy = ∞ x y y e -y 2 /2 dy ≤ 1 x ∞ x ye -y 2 /2 dy = e -x 2 /2
x .

For a lower bound:

∞ x e -y 2 /2 dy = ∞ x y y e -y 2 /2 dy = e -x 2 /2 x -∞ x 1 y 2 e -y 2 /2 dy and ∞ x 1 y 2 e -y 2 /2 dy = ∞ x y y 3 e -y 2 /2 dy ≤ e -x 2 /2 x 3
and combining these gives

∞ x e -y 2 /2 dy ≥ 1 x - 1 x 3 e -x 2 /2 . Thus 1 √ 2π 1 x - 1 x 3 e -x 2 /2 ≤ 1 -Φ(x) ≤ 1 √ 2π 1 x e -x 2 /2 , from where 1 √ 2π 1 Φ -1 (1 -p) - 1 Φ -1 (1 -p) 3 e -Φ -1 (1-p) 2 /2 (A.1.15) ≤ p ≤ 1 √ 2π 1 Φ -1 (1 -p) e -Φ -1 (1-p) 2 /2 (A.1.16) Using (A.1.16), when Φ -1 (1 -p) ≥ 1 (that is p ⩽ 0.15 or equivalently K ≥ 8), we have the following upper bound Φ -1 (1 -p) ⩽ q(p) where q(p) := 2 log( √ 2π/p). Then, injecting q(p) in (A.1.15): 1 √ 2π 1 q(p) - 1 q(p) 3 e -Φ -1 (1-p) 2 /2 ≤ p.
Now when q(p) ≥ 1 you have:

e -Φ -1 (1-p) 2 /2 ≤ √ 2π pq(p) 3 q(p) 2 -1 and Φ -1 (1 -p) ≥ 2 log q(p) 2 -1 √ 2π pq(p) 3 .
There is one additional requirement on p which is simply that the argument of the log should be ≥ 1 i.e. q(p) 2 -1 ≥ √ 2π pq(p) 3 , which is true as soon as K ≥ 8.

A.2 Complementary experiments

A.2.1 Visualization of Theorem 3.2.3

To further understand and illustrate Theorem 3.2.3, we propose in Figure A.1, an interesting visualization where we plot the manifold learned by a WGANs architecture and its corresponding latent space configuration. As expected, we observe that when the number of distinct modes increase, the number of data generated out of the manifolds increase too. • the Frechet Inception distance: first proposed by [START_REF] Dowson | The fréchet distance between multivariate normal distributions[END_REF], the Frechet distance was applied in the setting of GANs by [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF]. This distance between mutlivariate Gaussians compares statistic of generated samples to real samples as follows

FID = ∥ν ⋆ -ν θ ∥ 2 + Tr Σ ⋆ + Σ θ + 2(Σ ⋆ Σ θ ) 1 2
where X ⋆ = N (ν ⋆ , Σ ⋆ ) and X θ = N (ν θ , Σ θ ) are the activations of a pre-softmax layer. However, when dealing with disconnected manifolds, we argue that this distance is not well suited as it approximates the distributions with unimodal one, thus loosing many information.

The choice of such metrics is motivated by the fact that metrics measuring the performances of GANs should not rely on relative densities but should rather be point sets based metrics.

A.2.3 Saturation of a MLP neural network

In Section 3.2.4.2, we claim that the generator reduces the sampling of off-manifold data points up to a saturation point. Figure A.2 below provides a visualization of this phenomenon. In this synthetic case, we learn a 9-component mixture of Gaussians using simple GANs architecture (both the generator and the discriminator are MLP with two hidden layers). The minimal distance between two modes is set to 9. We clearly see in Figure A.2d that the precision saturates around 80%.

A.2.4 More results and visualizations on MNIST/F-MNIST/CIFAR10

Additionally to those in Section 3.2. Let L, D be such that L ⩾ D log(m). Let's prove that for any well-balanced generator G ∈ G L , we have:

α G ⩽ 1 -ε min log m e -3/2 .
Using the method from Schechtman (2012), we have the measure of the border of cell i:

γ ∪ j̸ =i A j ε \ ∪ j̸ =i A j ⩾ 1 √ 2π t+ε t e -s 2 /2 ds, ⩾ ε √ 2π e -(t+ε) 2 /2 , ⩾ ε √ log m m e -εt-ε 2 /2 , ⩾ ε √ log m m e -ε √ log m-ε 2 /2 .
In first line, t is such that 1 √ 2π

∞ t e -s 2 /2 ds = 1/m. In third line, we use

√ log m ≤ t ≤ √ 2 log m). Thus: γ(∂ ε min A ) = m ∑ i=1 γ ∪ j̸ =i A j ε \ ∪ j̸ =i A j ⩾ ε min log m e -ε min √ log m-ε 2 min /2 .
Thus, we have

α G ⩽ 1 -γ(∂ ε min A ), ⩽ 1 -ε min log m e -ε min √ log m-ε 2 min /2 . Moreover, using ε min = D L and L ⩾ D √ log m, so we get ε min √ log m ⩽ 1: α G ⩽ 1 -ε min log m e -3/2 .
Proof of Corollary 3.3.1 with w i (non-equal measure of modes). Let L, D be such that L ⩾ D log(m). Let's prove that for any well-balanced generator G ∈ G L , we have:

α G ⩽ 1 -mw min ε min log 1/w min e -3/2 .
Using the method from Schechtman (2012), we have the measure of the border of cell i:

γ ∪ j̸ =i A j ε \ ∪ j̸ =i A j ⩾ 1 √ 2π t+ε t e -s 2 /2 ds, ⩾ ε √ 2π e -(t+ε) 2 /2 , ⩾ w min ε log 1/w min e -εt-ε 2 /2 , ⩾ w min ε log 1/w min e -ε √ log 1/w min -ε 2 /2 .
In first line, t is such that 1 √ 2π

∞ t e -s 2 /2 ds = w min . In third line, we use log 1/w min ≤ t ≤ 2 log 1/w min ).

Thus:

γ(∂ ε min A ) = m ∑ i=1 γ ∪ j̸ =i A j ε \ ∪ j̸ =i A j ⩾ mw min ε min log 1/w min e -ε min √ log 1/w min -ε 2 min /2 .
Thus, we have

α G ⩽ 1 -γ(∂ ε min A ), ⩽ 1 -mw min ε min log 1/w min e -ε min √ log 1/w min -ε 2 min /2 .
Moreover, using ε min = D L and L ⩾ D log 1/w min , so we get ε min log 1/w min ⩽ 1:

α G ⩽ 1mε min w min log 1/w min e -3/2 .

B.1.3 Proof of Theorem 3.3.1

Let µ ⋆ be the target distribution. We know that µ ⋆ lays on m disconnected components contained in spheres S i , i ∈ [1, m]. We note M i , i ∈ [1, m] the centers, and r i the radius of each sphere. We also assume that the spheres verify Assumption 4. For each pair (i, j) ∈ [1, m] 2 , we define X i j ∈ S i and X ji ∈ S j the points verifying X i j = arg min We consider the optimal partition A ⋆ in the Gaussian latent space. For each given latent point z ∈ R d , we define:

N z = { j ∈ [1, m], z ∈ A ε j }.
We then distinguish two different cases:

A 2 A 3 A 1 ∂ ε A 3
Latent space Output space

M 2 M 3 M 1 Fig. B
.1 An optimal generator maps a 2D latent space to a 2D output space with three modes (M 1 , M 2 , M 3 ). The latent space has an optimal 'simplicial cluster' geometry. In the latent space, all the ε-boundaries intersect each other in the gray circle, which is mapped in the output space in the convex hull of the three modes.

1. |N z | = 1: the point z belongs to the interior of a single cell, z ∈ A -ε i .

2. |N z | ⩾ 2: the point z is in the neighborhood of at least two different cells.

Interestingly, a point can only belong at most to the interior of one cell, but it can be at the intersection of several boundaries. We are now ready to define the optimal generator. Proof that G ⋆ ε is well-balanced. We recall that a generator is well-balanced if we have G♯γ(M 1 ) = . . . = G♯γ(M m ). By construction (B.1.1), we have that for any i ∈ [1, m]

∥G ⋆ ε (z) -X i ∥ = ∥ ∑ k̸ =i w k (X k -X i )∥, = D × (1 -w i ).
So, for any z ∈ A i , we have that Thus G ⋆ ε is associated with the optimal partition A ⋆ , . Besides, for a given radius r of the different modes, since everything is symmetrical, we have that γ({z ∈ R d , ∥G(z) -X 1 ∥ ⩽ r} = . . . = γ({z ∈ R d , ∥G(z) -X m ∥ ⩽ r}. Thus, the generator is well-balanced.

Showing that G ⋆ ε ⋆ is in G L .
α i = d(z, (A ε i ) ∁ ) ∑ j∈A z d(z, (A ε j ) ∁ )
and

β i = d(z ′ , (A ε i ) ∁ ) ∑ j∈A z d(z ′ , (A ε j ) ∁ )
We have that

∥G(z) -G(z ′ )∥ = ∥(1 -∑ i̸ =1 α i )X 1 -(1 -∑ i̸ =1 β i )X 1 + ∑ i̸ =1 α i X i -∑ i̸ =1 β i X i ∥ = ∥ ∑ i̸ =1 (α i -β i )(X 1 -X i )∥ ⩽ max (i, j)∈[1,m] 2 ∥X i -X j ∥ ∥α -β ∥, ⩽ max (i, j)∈[1,m] 2 ∥X i -X j ∥ ∥h(z) -h(z ′ )∥,
where h is the function from R d → R m defined as:

h(z) = ( d(z, (A ε 1 ) ∁ ) ∑ i∈A z d(z, (A ε i ) ∁ ) , . . . , d(z, (A ε m ) ∁ ) ∑ i∈A z d(z, (A ε i ) ∁ )
).

We can write h = f • g with f the function defined from R We have that f is a √ m-Lipschitz functions (given that z → d(z, (A ε m ) ∁ ) is 1-Lipschitz). Besides, we know that outside the ball B ε/2 (0), the function g is (2/ε)-Lipschitz. Since it is clear that for every point z such that |N z | ⩾ 2, we have that | f (z)| ⩾ ε/2. Finally, the function h is 2 √ m ε -Lipschitz. Thus, we have that:

∥G ⋆ ε (z) -G ⋆ ε (z ′ )∥ ⩽ 2D √ m ε ∥z -z ′ ∥,
with D = max i, j ∥X i -X j ∥, (i, j) ∈ [1, m] 2 , i ̸ = j. Now, by noting ε max = D L , and considering ε ⋆ = 2 √ m ε max , we have:

∥G ⋆ ε ⋆ (z) -G ⋆ ε ⋆ (z ′ )∥ ⩽ L∥z -z ′ ∥.
Now, consider two latent vectors z, z ′ in the same cell A -ε i . There exists i ∈ [1, m], and a pairs ( j, j ′ ) ∈ [1, m] 2 (note that j could be equal to j ′ ) such that G(z) = X i, j and G(z ′ ) = X i, j ′ . Using a similar reasoning as before, we can show that:

∥G ⋆ ε ⋆ (z) -G ⋆ ε ⋆ (z ′ )∥ ⩽ L∥z -z ′ ∥, with D = 2 max i∈[1,m] r i .
We can now conclude on the Lipschitzness of G ⋆ on R d .

Proving that: for m ⩽ d + 1, for any δ > 0, if L is large enough, then, for any well-balanced G ∈ G L , we have α G ⋆ ε max ⩾ α Gδ . Let G be a well-balanced generator and A the partition associated with G. Let us first define the gaussian boundary measure P γ of a partition A of R d . For partitions with smooth boundaries, it coincides with the (d -1)-dimensional gaussian measure of the boundary, defined as follows:

P γ (A ) = lim inf ε→0 γ(∂ ε A ) -γ(A ) 2/πε
Moreover, for sets with smooth boundaries, we have from Federer (2014, Theorem 3.2.29):

lim inf ε→0 γ(∂ ε A ) -γ(A ) 2/πε = lim ε→0 γ(∂ ε A ) -γ(A ) 2/πε
Let us denote A ⋆ , the optimal partition defined in [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF], based on simplicial clusters. A ⋆ is a standard partition where γ(A ⋆ 1 ) = . . . = γ(A ⋆ m ) for all i, and ∑ i γ(A i ) = 1. By the multi-bubble theorem [START_REF] Milman | The gaussian double-bubble and multi-bubble conjectures[END_REF], simplicial clusters (such as A ⋆ ) are the unique minimizers of the gaussian isoperimetric problem, thus:

P γ (A ⋆ ) ⩽ P γ (A ) lim ε→0 γ(∂ ε A ⋆ ) ε ⩽ lim ε→0 γ(∂ ε A ) ε L A ⩽ L A ⋆ where L A = lim ε→0 γ(∂ ε A ⋆ ) ε and L A ⋆ = lim ε→0 γ(∂ ε A ⋆ ) ε
. Then, for any δ > 0, there exists ε ′ > 0 such that, for any ε < ε ′ ,

| γ(∂ ε A ⋆ ) ε -L A ⋆ | < δ , | γ(∂ ε A ) ε -L A | < δ and L A ⋆ ⩽ L A
Thus, for any δ > 0, there exists ε ′ > 0 such that, for any ε < ε ′ ,

γ(∂ ε A ⋆ ) ⩽ γ(∂ ε A ) + 2δ ε (B.1.2)
Besides, we know that

α G ⩽ 1 -γ(∂ ε min A )
Consequently, we have that:

α G ⩽ 1 -γ(∂ ε min A )
⩽ 1γ(∂ ε min A ⋆ ) + 2δ ε min using (B.1.2). Now, by construction of G ⋆ ε max , we have that

α G ⋆ ε max ⩾ 1 -γ(∂ ε max A ⋆ ).
Consequently,

α G ⩽ 1 -γ(∂ ε min A ⋆ ) + 2δ ε max + γ(∂ ε max A ⋆ ) -γ(∂ ε max A ⋆ ) ⩽ α G ⋆ ε + 2δ ε max + γ(∂ ε max A ⋆ ) -γ(∂ ε min A ⋆ ) ⩽ α G ⋆ ε + 2δ ε max + γ(∂ ε max A ⋆ ) -2L A ⋆ ε max -γ(∂ ε min A ⋆ ) + 2L A ⋆ ε min + 2L A ⋆ (ε max -ε min ) ⩽ α G ⋆ ε + 4δ ε max + 2L A ⋆ ε max , ⩽ α G ⋆ ε + ε max (4δ + 2L A ⋆ ).
We conclude by choosing L big enough such that ε max is strictly smaller than

δ 4δ +2L A ⋆ .
Proving the lower-bound 3.3.7 of Theorem 3.3.1. Let's consider G ε ⋆ defined using (B.1.1) and ε ⋆ = 2 √ mε max . The precision of G ⋆ ε ⋆ is thus such that:

α G ⋆ ε ⋆ ⩾ 1 -γ(∂ ε ⋆ A ).
However, since ∂ ε A ⊂ n i=1 A ε i , we have that for any ε:

γ(∂ ε A ) ⩽ n ∑ i=1 γ(A ε i ).
Using results from Schechtman (2012, Proposition 1), when m ≤ d, there exists C large enough, such that

γ(A ε ⋆ i ) ⩽ ε ⋆ m π log(Cm) .
Thus, we have

α G ⋆ ε ⋆ ⩾ 1 -ε ⋆ π log(Cm), To have α G ⋆
ε max ⩾ 0, we must have ⋆ ⩽ 1/ π log(Cm). This is the case since we have

⋆ = 2D √ m/L and L ⩾ D √ m π log(Cm),
where D = max i, j ∥X i -X j ∥. For TransGAN [START_REF] Jiang | Transgan: Two pure transformers can make one strong gan, and that can scale up[END_REF], we follow the implementation from the authors available at https://github.com/VITA-Group/TransGAN. TransGAN is trained with a WGAN-GP loss, 4 discriminator steps for 1 generator step, and Adam optimizer with a learning rate of 10 -4 .

Evaluation. For evaluation metrics, we follow the setting proposed by the authors. For FID [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF], we use 50k real images and 50k fake images. For precision, recall, density and coverage [START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF][START_REF] Naeem | Reliable fidelity and diversity metrics for generative models[END_REF], we use 10k real images and 10k fake images with nearest-k= 5.

GPUs. For all datasets, the training of GANs was run on NVIDIA Tesla V100 GPUs (16 GB). The training of ResNet GANs for 100k steps on CIFAR takes around 30 hours. For TransGAN models, the training is done for 250k steps on two NVIDIA Tesla V100 GPUs, which takes around 35 × 2 = 70 GPU hours.

B.2.3 Details on simplicial truncation method (Details for Section 3.3. 5.4) We provide here more details about our truncation method. First, the rejection sampling in the latent space R d of GANs procedure is the following:

• Define hyper-parameters threshold τ, number of clusters N, latent space dimension d.

• Initialize N equidistant points in {(u 0 , . . . , u N ) | u i ∈ R d }. This can be done easily when N ≤ d.

• When sampling latent vectors z ∈ R d , compute a softmax over the negative distances between z and u i : p i (z) = e -d(z,u i )

∑ j e -d(z,u j ) . Then, z is selected if max i p i (z) > τ.

Second, we add a classification loss to encourage the generator to use this latent structure. This loss is motivated by the need to maximize mutual information between the latent cluster and the modes of the generator [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF], and can be written as:

L c = -E z∼γ [ln q φ i(z)|G θ (z) ]
where q φ is parametrized by a second classification head added to the discriminator; i(z) = arg max During inference, if the generator has properly learned to use the different clusters of the latent space, we observe that augmenting the threshold τ leads to an increased density and precision.

We present full results in Table B. To illustrate our theoretical results, we propose to vary the number of modes of the data distribution. On real-world data, the number of modes is set but usually unknown, and removing/adding classes as a proxy for modes usually does not give insightful results since some classes can be much more complex than others. We thus use a synthetic setting, where we can easily control both the number of modes and their complexity. Figure B.3 stresses that as the number of modes increase, the precision decrease. Interestingly, using large latent space dimension can relieve the problem, even if the latent space dimension is clearly below that of the target. Recall the two problems that arise when training GANs: i) dimensional misspecification where the true and modeled distributions do not have density functions w.r.t. the same base measure, and ii) density misspecification, where GANs try to fit a disconnected manifold with a unimodal disitribution. From the results we conclude that:

• With very low latent space dimensions, both problems i) and ii) have to be addressed and this leads to poor precision as the number of modes increases.

• With larger latent space dimensions, the problem i) is less of a burden even when there is a clear dimensional misspecification and thus the GANs' performance is more tied to problem ii). Crossovers. We generate 2500 crossovers from random pairs of images, on both FFHQ and LSUN Bedroom.

Editing/Compositing. We create small datasets of 100 images from the test set of EdiBERT for FFHQ scribble-based editing, FFHQ compositing and LSUN Bedroom compositing. A user study on FFHQ compositing is presented in main paper with statistically significant number of votes. We also provide some metrics in D.3. Because of the small size of the dataset, we only report masked L1 and density. For density, the support of the real distribution is estimated with 2500 real points, and density is averaged over the individual density of the 100 generated images. [START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF] 0.0763 22.1 1.25 0.91 LC [START_REF] Chai | Using latent space regression to analyze and leverage compositionality in gans[END_REF] 0.1027 27.9 1.12 0.84 EdiBERT (⋆) 0.0290 13.8 1.16 0.98 Co-mod. GAN [START_REF] Zhao | Large scale image completion via co-modulated generative adversarial networks[END_REF] 0.0128 4.7 1.24 0.99 FFHQ: free-form masks I2SG++ [START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF] 0.0440 22.3 0.92 0.89 I2SG † ++ [START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF] 0.0435 21.1 1.17 0.91 LC [START_REF] Chai | Using latent space regression to analyze and leverage compositionality in gans[END_REF] 0.0620 27.9 [START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF] 0.0777 1.11 ID-GAN [START_REF] Zhu | In-domain gan inversion for real image editing[END_REF] 0.0461 0.79 LC [START_REF] Chai | Using latent space regression to analyze and leverage compositionality in gans[END_REF] 0.1016 1.14 EdiBERT (⋆) 0.0281 0.96 FFHQ compositing I2SG++ [START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF] 0.0851 0.77 I2SG † ++ [START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF] 0.0866 1.07 ID-GAN [START_REF] Zhu | In-domain gan inversion for real image editing[END_REF] 0.0570 0.75 LC [START_REF] Chai | Using latent space regression to analyze and leverage compositionality in gans[END_REF] 0 Hardware. We use a NVIDIA Tesla V100 with 16GB of RAM. The training takes around 2 days for T-WUTON, and around 3 days for S-WUTON. For inference, S-WUTON processes ∼77 frames per second.

E.2 More visual examples on the importance of distillation

In Figure E.1, we show more visual results proving the soundness of our teacher-student approach. Visually, our student model solves two kinds of problems: it is robust to human parser errors; it preserves important information that is masked to the standard virtual try-ons (hands, skin, handbags). the not end-to-end variant show that these metrics are less suited to the virtual try-on task than LPIPS.

The adversarial loss generates sharper images and improves the contrast. This is confirmed by the LPIPS, IS and FID metrics in Table E.1 and with visual results in Figure E.2. With the unpaired adversarial setting, the system better handles large variations between the shape of the cloth worn by the person and the shape of the new cloth. On metrics in the paired setting (LPIPS and SSIM), the best model is the variant using adversarial loss on paired data, which is logical. However, visual investigation suggests that the unpaired adversarial loss is better in the real use case of our work (see Figure E.2).

5 1. 4 6 1. 5

 5465 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thesis outline and contributions . . . . . . . . . . . . . . . . . . . . . . 7

Fig. 1 . 1

 11 Fig. 1.1 Virtual try-on: this task consists in warping a clothing image on the image of a person. It has the potential to improve the user experience in e-shopping, since it would allow users to try clothing items on a picture of themselves.

Fig. 1

 1 Fig. 1.2 Text-to-image generation and editing: Hertz et al. (2023) introduce a method to improve the generated images from text-to-image models, specifically Stable Diffusion in this instance. The process involves generating an image from a given text using Stable Diffusion and allowing the user to refine the generated image by modifying the prompt. This can be achieved by adjusting the intensity of a descriptor (such as crowded or fluffy), adding new text, or modifying existing words.

Fig. 1 . 3

 13 Fig. 1.3 Reducing the need for labels: Li et al. (2022) exploit deep generative models to construct large-scale datasets with pixel-wise annotations for image segmentation. During training, the model has only access to a few images with pixel-wise segmentation masks.

Fig. 2

 2 Fig. 2.2 Image generations from a variational auto-encoder with a deep and hierarchical architecture. Figure from Vahdat and Kautz (2020).

Fig. 2

 2 Fig. 2.3 Two-stage approach with discrete auto-encoder and auto-regressive models, parameterized with a transformer in this case. Source: Esser et al. (2021b).

Fig. 2 . 4

 24 Fig. 2.4 From left to right, the text-conditioning of the generated images are: "A braing riding a rocketship heading towards the moon"; "A robot couple fine dining with Eiffel Tower in the background"; "A small cactus wearing a straw hat and neon sunglasses in the Sahara desert."; "A photo of a Corgi dog riding a bike on Time Squares. It is wearing sunglasses and a beach hat.". Source: https://imagen.research.google/.

Fig. 2 . 5 (

 25 Fig. 2.5 (Left) Illustration of two distributions, where the blue one P r is the target distribution and the red one P g is the generated one. (Middle) Precision corresponds to the proportion of generated points that fall in the approximated support of the target distribution. (Right) Recall corresponds to the proportion of real points that fall in the approximated support of the generated distribution. Figure from Kynkäänniemi et al. (2019).

Fig. 2 . 8

 28 Fig. 2.8 Inverting images with scribbles in the latent space of GANs allow to recover realistic images that still takes into account the user-input (scribbles). Figure from Abdal et al. (2020).

Fig. 2

 2 Fig. 2.9 The latent space of GANs allow image editing along the main factors of variation in a disentangled manner. Figure from Shen et al. (2020).

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2 Learning disconnected manifolds: a no GAN's land . . . . . . . . . . . 29 3.3 Unveiling the latent space geometry of push-forward generative models 45 3.4 Latent reweighting, an almost free improvement for GANs . . . . . . . 62

  (a) Heatmap of the generator's Jacobian norm. White circles: quantiles of the latent distribution N (0, I). (b) Green: target distribution. Coloured dots: generated samples colored w.r.t. the Jacobian Norm using same heatmap than (a).

Fig. 3

 3 Fig. 3.1 Learning disconnected manifolds leads to the apparition of an area with high gradients and data sampled in between modes.

  (a) WGAN 4 classes: visualisation of ∥J G (z)∥ F . (b) WGAN 9 classes: visualisation of ∥J G (z)∥ F . (c) WGAN 25 classes: visualisation of ∥J G (z)∥ F . (d) Precision w.r.t. D (mode distance) and M (classes).

Fig. 3

 3 Fig. 3.2 Illustration of Theorem 3.2.3. If the number of classes M → ∞ or the distance D → ∞, then the precision ᾱ → 0. We provide in appendix heatmaps for more values of M.

  Figure 3.2 clearly corroborates (3.2.1) as we can easily get the maximum precision close to 0 (M = 25, D = 27).

Fig. 3 . 3

 33 Fig. 3.3 Mixture of 9 Gaussians in green, generated points in blue. Our truncation method (JBT) removes least precise data points as marginal precision plummets.

Fig. 3 . 4

 34 Fig. 3.4 For high levels of kept samples, the marginal precision plummets of newly added samples, underlining the efficiency of our truncation method (JBT). Reported confidence intervals are 97% confidence intervals. On the second row, generated samples ordered by their JFN (left to right, top to bottom). In the last row, the data points generated are blurrier and outside the true manifold.

Fig. 3 . 5

 35 Fig. 3.5 On the first row, per-class precision-recall curves comparing Brock et al. (2019)'s truncation trick and our truncation method (JBT), on three ImageNet classes generated by BigGAN. We show better results on complex and disconnected classes (e.g. bubble). Reported confidence intervals are 97% confidence intervals. On the second row, generated samples ordered by their JFN (left to right, top to bottom). We observe a concentration of off-manifold samples for images on the bottom row, confirming the soundness of JBT.

Fig. 3 . 6

 36 Fig. 3.6 Illustration of the capability of GANs to discover an optimal geometry of the latent space. On the left, the propeller shape represents a partition of 3D Gaussian space with the smallest Gaussian-weighted perimeter (Figure from[START_REF] Heilman | Solution of the propeller conjecture in r3[END_REF]). On the right, we show the 3D Gaussian latent space of a GAN trained on three classes of MNIST. Each area colored in blue, green, or red corresponds to samples in one of the three classes. Using a pre-trained classifier, we highlight in purple the samples with low-confidence, and observe that the partition reached by the GAN (right) is close to optimality (left), as the latent space partition is similar to the intersection of the propeller on a sphere.

  Figure 3.8 stresses examples when training GANs from R 2 to R m with m equidistant modes. This

Fig. 3 . 7

 37 Fig. 3.7 Illustration of the impact of the geometry of data modes on the latent space of GANs. The left column shows the modes (X 1 , X 2 , X 3 ) from the target distribution and the generated points (small blue dots). In the middle, we plot the Voronoi diagram generated from (X 1 , X 2 , X 3 ). On the right column, we show the boundaries in the GANs latent space with heatmaps of the norm of the gradient of the generator. In the first row, when the data satisfies Assumption 4, GANs achieve the optimal configuration. However, when the data modes do not satisfy this assumption, as seen in the second row, this is no longer the case.

Fig. 3 . 8

 38 Fig. 3.8 Extension of the multi-bubble conjecture when m > d + 1. We depict the partition of the R 2 latent space of a GAN that maps to m equidistant points in R m , with m = 4, 6, 12. Each colored cell maps to a distinct data point in R m .

Fig. 3

 3 Fig. 3.9 Visualization of the convexity of classes in the latent space of GANs trained on CIFAR-10. The plot shows that latent linear interpolations within a class preserve the class label.

Fig. 3 .

 3 Fig. 3.10 Performance of GANs with regard to the number of modes and latent space dimensions.As the number of modes and latent space dimension increases, we observe an improvement in Precision (left) and Density (right), with a saturation point beyond a certain threshold.

Fig. 3 .

 3 Fig. 3.11 Study of the correlation between GANs' performance and their latent space geometry. This is done by increasing the width of the generator (w ∈ {32, 64, 128, 256, 512}) in a fixed training setting on the CIFAR-10 and CIFAR-100 datasets. The results reveal a positive correlation between GANs' performance (measured by Precision and Density) and the linear separability and convexity of their latent space (measured by LogReg and Convex Accuracy). Confidence intervals are computed on 10 checkpoints of a training.

Algorithm 1 : 5

 15 Adversarial learning of w ϕ 1 Require: Data µ n , Prior Z, Gen. G θ , Disc. D α , number of D α updates n d , soft-clipping param. m, regularization weights λ 1 and λ 2 , batch size b; 2 while ϕ has not converged do 3 for i = 0, ..., n d do 4 Sample real data {x i } b i=1 ∼ µ n ; Sample latent vectors

  clip ) ; Update ϕ with grad ϕ ; end Algorithm 2: LatentRS 1 Requires: Prior Z, Gen. G θ , Importance weight network w ϕ , maximum importance weight m

  (a) Synthetic WGAN: real samples in green and fake ones in blue.(b) MC method optimizing for a precision criterion[START_REF] Azadi | Discriminator rejection sampling[END_REF].(c) Optimizing for Wasserstein criterion with latentRS (ours ⋆ ).

Fig. 3 .

 3 Fig. 3.14 Visualization of the trade-off between the time spent to generate an image and its average precision. Interestingly, latentRS+GA has the best Pareto front. Left: ProGAN trained on CelebA. Right: StyleGan2 trained on LSUN Church.

Contents 4 .

 4 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.2 EdiBERT: a generative model for image editing . . . . . . . . . . . . . 81 4.3 A parser-free virtual try-on . . . . . . . . . . . . . . . . . . . . . . . . . 100

Fig. 4 .

 4 Fig. 4.1 Using a single and straightforward training, EdiBERT can tackle a wide variety of different tasks in image editing. In this image, the top row is the input, while the second and third rows are different samples from EdiBERT, showing realism, consistency, and variety.

  .2.2) 2. Training an autoregressive transformer to maximize the log-likelihood of the encoded sequences. 3. At inference, sampling a sequence with the transformer and decoding it with the decoder D. This vector-quantized representation was later improved by Yu et al. (2021a) and used by Yu et al. (2022) to create PARTI, a state-of-the-art text-to-image generative model. Interestingly, our work EdiBERT builds on top of the first step of VQGAN, and also requires the training of the triplet (E, D, Z) following (4.2.2).

Fig. 4 .

 4 Fig. 4.2 The 2D selection and randomization strategy for the training of our bidirectional transformer: EdiBERT is trained on sequences where localized patch of tokens have been perturbed.

Fig. 4 . 4

 44 Fig. 4.4 Comparison of reconstruction capabilities of VQGAN + optimization to two GANs inversion methods such as Id-GAN (Zhu et al., 2020) and I2SG † ++[START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF]. Averaged LPIPS are computed on the validation set FFHQ.

Fig. 4 . 5

 45 Fig. 4.5 Analysis of reconstruction capabilities of VQGAN. On the left, we see that both the L1 and perceptual loss (LPIPS) between original and reconstructed images significantly decrease when optimizing LPIPS over the latent vectors of VQGAN. This may be a consequence of a higher number of dimensions spanned by the latent vectors (on the right), after the optimization (allowing for more complex reconstructions).

Fig. 4 . 8

 48 Fig. 4.8 Ablation study for inpainting. Components removed are (a) optimization, (b) dilation, (c) randomization, (d) collage, (e) spiraling (random order instead). Optimization improves fidelity to the source image, while the other components help increase image quality.

Fig. 4 .

 4 Fig. 4.10 Typical failure cases of the human parser. On the two first rows, it does not segment the person properly. On the third row, it masks the handbag which we would like to preserve in a virtual try-on. CP-VTON and our T-WUTON, which rely on the parsing information, are not robust to a bad parsing. However, the student model S-WUTON which is distilled from the human parser, pose estimator and T-WUTON, can preserve the person's attributes and does not rely on the parsing information.

Fig. 4 .

 4 Fig. 4.11 The teacher T-WUTON : our proposed end-to-end warping U-net architecture. Dotted arrows correspond to the forward pass only performed during training. Green arrows are the human parser, red ones are the loss functions. The geometric transforms share the same parameters but do not operate on the same spaces. The different training procedure for paired and unpaired pictures is explained in Section 4.3.3.2.

  the dataset from Dong et al. (2019), we have pairs of in-shop cloth image c a and a person wearing the same cloth p a . Using a human parser and a human pose estimator, we generate p ⋆ a . From the parsing information, we can also isolate the cloth on the image p a and get c a,p , the cloth worn by the reference person. Moreover, we get the image of another in-shop cloth c b . The inputs of our network are the two tuples (p ⋆ a , c a ) and (p ⋆ a , c b ). The outputs are respectively ( pa , θ a ) and ( pb , θ b ).

Fig. 4 .

 4 Fig. 4.12 S-WUTON: our training scheme allowing to remove the need for a human parser at inference time. We use human parser and pre-trained T-WUTON to generate synthetic ground-truth for a student model S-WUTON.

Fig. 4 .

 4 Fig. 4.13 On the left side, comparison of our method with CP-VTON (Wang et al., 2018a). For fairness, the two methods are trained on the same dataset and on the same agnostic person representation p ⋆ . More examples are provided in supplementary material. On the right side, comparison with recent work VTNFP. Except for S-WUTON's column, images are taken from their paper.

4. 3 . 4 . 6

 346 Impact of the adversarial loss in the teacher-student setting

Fig. 4 .

 4 Fig. 4.14 Visual comparison of the student model with and without the adversarial loss. Interestingly, the student model without the adversarial loss can not exploit information that is masked to the teacher, e.g. arms and hands.

Fig. 4 .

 4 Fig. 4.15 Our student model S-WUTON generates high-quality images and preserves both person's and cloth's attributes.

  is useful to evaluate the closeness of two different supports from a metric space, but is sensitive to outliers because of the max operation. It has been recently used for theoretical purposes by[START_REF] Pandeva | Mmgan: Generative adversarial networks for multi-modal distributions[END_REF].

Fig. A. 2

 2 Fig. A.2 Learning 9 disconnected manifolds with a standard GANs architecture.

Fig. A. 3

 3 Fig. A.3 Visualization of our truncation method (JBT) on three real-world datasets. From top to bottom: MNIST, F-MNIST and CIFAR-10. Left column: examples of data points selected by our JBT with a truncation ratio of 90% (we thus removed the 10% highest gradients). Right column: examples of data points removed by our JBT with a truncation ratio of 90% (these are the 10% highest gradients data points).

Fig. A. 4

 4 Fig. A.4 For high levels of kept samples, the marginal precision plummets of newly added samples, underlining the efficiency of our truncation method (JBT). Reported confidence intervals are 97% confidence intervals. On the second row, generated samples ordered by their JFN (left to right, top to bottom). In the last row, the data points generated are blurrier and outside the true manifold.

A. 2 . 5

 25 More results on BigGAN and ImageNetIn Figure A.5, we show images from the Bubble class of ImageNet. It supports our claim of manifold disconectedness, even within a class, and outlines the importance of studying the learning of disconnected manifolds in generative models. Then, in Figure A.6 and Figure A.7, we give more exemples from BigGAN 128x128 class-conditionned generator. We plot in the same format than in 3.2.4.4. Specifically, for different classes, we plot 128 images ranked by JFN. Here again, we see a concentration of off-manifold samples on the last row, proving the efficiency of our method. Example of classes responding particularly well to our ranking are House Finch c, Monnarch Butterfly e or Wood rabbit c. For each class, we also show an histogram of JFN based on 1024 samples. It shows that the JFN is a good indicator of the complexity of the class. For example, classes such as Cornet (see Figure A.7e) or Football helmet (see Figure A.7a) are very diverse and disconnected, resulting in high JFNs.

Fig. A. 5

 5 Fig. A.5 Images from the Bubble class of ImageNet showing that the class is complex and slightly multimodal.

Fig. A. 6

 6 Fig. A.6 Images from BigGAN class-conditional generator, along with an histogram of classspecific Jacobian Frobenius Norms.

  First, we setG(z) = X i, j for all z ∈ {z ∈ R d , |N z | = 2, z ∈ A -ε i ∩ A ε i ∩ A ε j where N z = {i, j}}.Second, we define the generator in the interior of the cells, i.e. N z = {i}. For each z ∈ A -ε i and for a given unit vector u ∈ R d , we assume that the generator is constant along the parametric linez = k × u, k ∈ R.Finally, we define the generator when z does not belong to the interior of any cell, i.e.|N z | ⩾ 2: j (z) X i, j 1 j∈N z 1 i∈N z (B.1.1)wherew i, j (z) = d(z, (A ε i ) ∁ ) ∑ i∈[1,m] ∑ j̸ =i d(z, (A ε j ) ∁ ) 1 j∈N z 1 i∈Nz where d(z, A) = min a∈A ∥z -a∥. An illustration of the optimal generator is given in Figure B.1. When z belongs to the intersection of two ε-boundaries, G ε (z) is a simple linear combination of 2 points. It is only when |N z | ⩾ 3 that more complex samples are generated. A simple illustration of G ⋆ ε for d = 2 and m = 3 is given in Figure B.1. Interestingly, one can also show that the image of G ⋆ ε is equal to the convex hull of the Diracs X i , i ∈ [1, m]. In particular, there exists a particularly interesting neighborhood ν of 0 where G ⋆ ε (ν) is equal to the whole convex hull of the points X i , i ∈ [1, m].

i

  = arg min j∈[1,m] w j = arg min j∈[1,m] ∥G(z)-X j ∥

  It is clear that when|N z | = 1, we have that G ⋆ ε (z) is a L-Lipshitz continuous function. Now, assume that |N z | ⩾ 2. Consider z, z ′ such that N z = N ′ z . Let α = (α 1 , .. . , α m ) and β = (β 1 , . . . , β m ) be two vectors, both in R m , such that for all i ∈ [1, m]:

  d → R m by f (z) = d(z, (A ε 1 ) ∁ ), . . . , d(z, (A ε k ) ∁ ) ,and g the function defined on R m \ {0} by g(z) = z ∥z∥ 1

  i p i (z) is the index of the latent cluster of the sample. This loss is added during training, at each step of generator's and discriminator's training, during the first 20 epochs. It is then dropped, since we noticed that it harms the GANs performance if it is kept until the end of the training. Training hyper-parameters: for N = 10, we use a latent dimension of d = 64 and training threshold of τ = 0.135; for N = 100, we use d = 128 and τ = 0.08.

  4 and Figure B.2.

Fig. B. 2

 2 Fig. B.2 Density/Coverage curves comparing TransGAN and boosting methods for multi-modal datasets and different threshold ratios. Our simplicial truncation method (TransGAN + simp.) consistently outperforms the TransGAN and TransGAN + DeliGAN baselines.

Fig. B. 3

 3 Fig. B.3 Training on a mixture of Gausians in R 100 with varying number of modes and varying latent space dimension. The bigger the number of modes, the lower the precision. Increasing the latent space dimension helps up to a limit depending on the number of modes.

Fig. C. 1

 1 Fig. C.1 Gradient ascent on latent importance weights (latentGA), on StyleGAN2 trained on LSUN Church.

Fig. C. 2

 2 Fig. C.2 Gradient ascent on latent importance weights (latentGA), on StyleGAN2 trained on LSUN Church.

Fig. C. 3

 3 Fig. C.3 Gradient ascent on latent importance weights (latentGA), on Progressive GAN trained on CelebA.

Fig. D. 1 4

 14 Fig. D.1 Image compositing and scribble-based editing on FFHQ. ID-GAN refers to, while I2SG †++ refers to Abdal et al. (2020) with the backbone from Karras et al. (2020a).

  Fig. E.1 Visual results proving the importance of the student-teacher approach. It is robust to parsing errors and preserves person's attributes such as arms, hands, and handbags.

  use an optimization procedure over an intermediate feature space of StyleGAN, while Richardson et al. (2021) and Tov et al. (2021) design a specific encoder architecture that maps images into the intermediate features of StyleGAN. Figure 2.8 illustrates an example of GANs inversion.

Table 3

 3

	MNIST	Prec.	Rec.	FID
	WGAN	91.2±0.3 93.7±0.5 24.3±0.3
	WGAN JBT 90% 92.5±0.5 92.9±0.3 26.9±0.5
	WGAN JBT 80% 93.3±0.3 91.8±0.4 33.1±0.3
	W-Deligan	89.0±0.6 93.6±0.3 31.7±0.5
	DMLGAN	93.4±0.2 92.3±0.2 16.8±0.4
	F-MNIST
	WGAN	86.3±0.4 88.2±0.2 259.7±3.5
	WGAN JBT 90% 88.6±0.6 86.6±0.5 257.4±3.0
	WGAN JBT 80% 89.8±0.4 84.9±0.5 396.2±6.4
	W-Deligan	88.5±0.3 85.3±0.6 310.9±3.1
	DMLGAN	87.4±0.3 88.1±0.4 253.0±2.8

.1 JBT x% means we keep the x% samples with lowest Jacobian norm. Our truncation method (JBT) matches over-parameterization techniques. ± is 97% confidence interval.

Table 3 .

 3 2 Validation of linear separability (LogReg Acc.) and convexity (Convex Acc.) in GAN latent spaces. The results align with the predictions of Corollary 3.3.1, where a linearly separable and convex structure of the latent space indicates a high precision. The architecture Transformer refers to the TransGAN model from[START_REF] Jiang | Transgan: Two pure transformers can make one strong gan, and that can scale up[END_REF]. The supervised classifiers used as oracles haves test-accuracies of 80.2% on CIFAR-10 and 61.8% on CIFAR-100.

	Dataset	Architecture Latent dim Precision (↑) LogReg Acc. (↑) Convex Acc. (↑)
	100 Gauss.	MLP	100	75.5	78.5	87.2
	MNIST	CNN	64	93.2	90.4	98.7
	CIFAR-10	ResNet	64	66.8	65.3	75.2
	CIFAR-10	Transformer	256	72.8	70.7	84.3
	CIFAR-100	ResNet	64	64.3	30.5	42.1
	CIFAR-100 Transformer	64	64.2	26.5	39.2
	corresponding cluster

Table 3 .

 3 [START_REF]2 Correlation between latent space geometry and GANs' performance (Details for Section 3.3.5.3) We present the full results of this study in Table B.3. Table B.3 Correlation between GANs' performance and their latent space geometry. Increasing the capacity of GANs tend to structure their latent space in simplicial clusters (better LogReg accuracy) and improve their performance on precision, density and coverage. Confidence intervals are computed on several sets of generated/training points from a given generator[END_REF] Comparison of latentRS with concurrent methods on two synthetic datasets in the same setting as DOT[START_REF] Tanaka | Discriminator optimal transport[END_REF]. Our method enables a consistent gain in EMD, surpassing other methods on Swiss Roll and slightly behind DOT on Mixture of 25 Gaussians. For conciseness, WGAN stands for WGAN-GP.

		EMD	EMD
		Swiss Roll 25 Gaussians
	WGAN	0.030±0.002 0.044±0.001
	WGAN: DRS	0.036±0.004 0.038±0.002
	WGAN: SIR	0.037±0.003 0.041±0.001
	WGAN: DOT	0.029±0.003 0.035±0.002
	WGAN: latentRS (⋆) 0.025±0.002 0.036±0.001
	of GANs by

Table 3 .

 3

			.	EMD	FID	Inference
		(↑)	(↑)	(↓)	(↓)	(ms)
	CelebA 128x128
	ProGAN	74.2±0.9 60.7±1.4 25.4±0.1 11.30±0.02	3.6
	ProGAN: SIR	79.5±0.4 57.3±1.0 24.9±0.2 12.01±0.04	49.0
	ProGAN: DOT	81.3±1.0 52.9±1.4 25.0±0.1 11.01±0.03	67.6
	ProGAN: latentRS (⋆)	80.4±0.9 55.7±1.0 24.7±0.1 10.77±0.04	4.5
	ProGAN: latentRS+GA (⋆)	83.3±1.0 52.7±0.9 24.5±0.1 10.75±0.04	20.5
	LSUN Church 256x256
	StyleGAN2	55.6±1.2 62.4±1.1 23.6±0.1 6.91±0.02	11.7
	StyleGAN2: SIR	60.5±1.4 58.1±1.3 23.4±0.1 7.36±0.01	130.0
	StyleGAN2: DOT	67.4±1.4 48.3±1.0 23.1±0.1 6.85±0.02	196.7
	StyleGAN2: latentRS (⋆)	63.3±0.7 57.7±1.0 23.1±0.1 6.31±0.02	16.2
	StyleGAN2: latentRS+GA (⋆) 72.6±1.1 43.2±1.3 22.6±0.1 6.27±0.03	43.2

5 latentRS+GA is the best performer, and latentRS matches SOTA with a significantly reduced inference cost (by an order of at least 10). ± is 97% confidence interval. Inference refers to the time in milliseconds needed to compute one image on a NVIDIA V100 GPU. model in terms of EMD. Overall, we explicitly follow the framework used by Azadi et al. (2019); Grover et al. (2019): we keep the gradient penalty

  learn an autoregressive model on random permutations of the ordering.[START_REF] Cao | The image local autoregressive transformer[END_REF] propose a model where missing tokens are inferred autoregressively, conditionally to the set of kept tokens. Similarly,[START_REF] Wan | High-fidelity pluralistic image completion with transformers[END_REF] use an auto-regressive procedure conditioned on the masked image, while[START_REF] Yu | Diverse image inpainting with bidirectional and autoregressive transformers[END_REF] use BERT training with [MASK] tokens and Gibbs sampling. If this setting is ideal for tasks with masked tokens such as inpainting, it makes it ill-posed for scribbleediting and insertion without existing paired datasets. On the opposite, our EdiBERT tackles all tasks without any need for supervision. Finally,Esser et al. (2021a) train ImageBART, a multinomial diffusion process

  .2.4)

								2	3	16	4			2	3	16	4
						Encoder		25	33	7	15	2D selection	19	14	7	15
						+ quantization	1	6	31	24	+ randomization	1	8	31	24
								39	35	5	48			39	35	5	48
												Flattening
	2	3	16	4	19	14	7	15	1	8	31	24	39	35	5	48
						Bidirectional transformer
					?	?			?	?

  Algorithm 3: Image denoising 1 Requires: Sequence (s 1 , . . . , s L ), BERT model p θ , number of iterations T ; Sample p ∼ (p 1 , . . . , p l ) (less likely position);

	2 for iterations in [0,T] do
	3	Compute p i = logit(-p i θ (s i |s)), ∀i ∈ [1, L] ;
	5	Sample t ∈ Z ∼ p p θ (•|s) ;
	6	Insert sampled token: s i ← t ;
	7 end
	8 Image ← Decoder(s);
		Result: Image
	4.2.4.2 Image inpainting

4

Table 4 .

 4 1 Image inpainting and compositing on FFHQ 256 × 256. Com-GAN is a model specific for image inpainting, ID-GAN handles several editing tasks but not inpainting, while other methods handle both. I2SG † ++ refers to[START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF] with a better GAN backbone(Karras et al., 2020a), LC to[START_REF] Chai | Using latent space regression to analyze and leverage compositionality in gans[END_REF], Com-GAN to[START_REF] Zhu | In-domain gan inversion for real image editing[END_REF], ID-GAN to[START_REF] Zhu | In-domain gan inversion for real image editing[END_REF].

			Inpainting			Compositing
		Masked L1 ↓FID ↓Dens. ↑Cover. ↑Masked L1 ↓Dens. ↑User study ↑
	I2SG++	0.0767	23.6 0.99	0.88	0.0851	0.77	-
	I2SG † ++	0.0763	22.1 1.25	0.91	0.0866	1.07	8.3%
	LC	0.1027	27.9 1.12	0.84	0.1116	1.00	14.8%
	EdiBERT ⋆	0.0290	13.8 1.16	0.98	0.0307	0.94	61.2%
	Com-GAN	0.0086	10.3 1.42	1.00	-	-	-
	ID-GAN	-	-	-	-	0.0570	0.75	15.7%

Table 4 .

 4 2 Inpainting: Ablation study on the components of EdiBERT sampling algorithm. EdiBERT (1st row) shows the best tradeoff between fidelity (masked L1) and quality (FID, density/coverage).

	Ordering Optim-Random-Collage Dilation Masked FID Density Coverage
		ization ization			L1 ↓	↓	↑	↑
	Spiral	✓	✓	✓	✓	0.0201 19.4 1.14	0.96
	Random	✓	✓	✓	✓	0.0206 20.7 1.13	0.95
	Spiral	X	✓	✓	✓	0.0299 20.3	1.20	0.94
	Spiral	✓	X	✓	✓	0.0198 20.5 1.26	0.92
	Spiral	✓	✓	X	✓	0.0252 19.9 1.11	0.95
	Spiral	✓	✓	✓	X	0.0197 23.3 0.96	0.91

  in supplementary material.

	Ref.	Target	CP-	T-WUT-	S-WUTON	Ref.	Target	CP-	VTNFP	S-WUTON
	person	cloth	VTON	ON(ours)	(ours)	person	cloth	VTON		(ours)

Table 4 .

 4 [START_REF]2 Correlation between latent space geometry and GANs' performance (Details for Section 3.3.5.3) We present the full results of this study in Table B.3. Table B.3 Correlation between GANs' performance and their latent space geometry. Increasing the capacity of GANs tend to structure their latent space in simplicial clusters (better LogReg accuracy) and improve their performance on precision, density and coverage. Confidence intervals are computed on several sets of generated/training points from a given generator[END_REF] Comparison of runtime of state-of-the-art architectures for virtual try-on. The time is computed on a NVIDIA Tesla V100 GPU.

		CP-VTON VTNFP ClothFlow T-WUTON S-WUTON
	Parsing + pose 168ms	168ms	168ms	168ms	0ms
	Try-on	9ms	>9ms	>0ms	13ms	13ms
	Total	177ms >177ms >168ms	181ms	13ms

Table 4 .

 4

		5 Comparison of IS and
	FID scores of S-WUTON and S-
	WUTON without the adversarial
	loss.
		λ adv = 0 λ adv = 1
	IS	2.912	3.154
	FID 12.620	7.927

  4.3, we provide in Figure A.3 and Table A.1 supplementary results for MNIST, F-MNIST and CIFAR-10 datasets.

Samples after 5k training steps. Samples after 50k training steps. Samples after 100k training steps. Evolution of the precision ᾱ during training.

Table A

 A

.1 Scores on MNIST and Fashion-MNIST. JFN stands for Jacobian Frobenius norm. ± is 97% confidence interval.

  Table B.2 GANs training details on CIFAR datasets. BN stands for batch-normalization.

					Conditional
	Operation	Kernel Strides Feature Maps BN	Activation
	Generator G(z)
	z ∼ N(0, Id)			128
	Fully Connected			4 × 4 × 128 -
	ResBlock	[3 × 3] × 21 × 1 4 × 4 × 128 Y	ReLU
	Nearest Up Sample			8 × 8 × 128 -
	ResBlock	[3 × 3] × 21 × 1 8 × 8 × 128 Y	ReLU
	Nearest Up Sample			16 × 16 × 128-
	ResBlock	[3 × 3] × 21 × 1 16 × 16 × 128Y	ReLU
	Nearest Up Sample			32 × 32 × 128-
	Convolution	3 × 3	1 × 1 32 × 32 × 3 -	Tanh
	Discriminator D(x)			32 × 32 × 3
	ResBlock	[3 × 3] × 21 × 1 32 × 32 × 256-	ReLU
	AvgPool	2 × 2	1 × 1 16 × 16 × 256-
	ResBlock	[3 × 3] × 21 × 1 16 × 16 × 256-	ReLU
	AvgPool	2 × 2	1 × 1 8 × 8 × 256 -
	ResBlock	[3 × 3] × 21 × 1 8 × 8 × 256 -	ReLU
	ResBlock	[3 × 3] × 21 × 1 8 × 8 × 256 -	ReLU
	Mean spatial pooling	-	-	256	-
	Fully Connected			1	-	-
	Batch size	256
	Gradient Penalty weight	10
	Learning Rate Discriminator1 × 10 -4
	Learning Rate Generator	5 × 10 -5
	Discriminator steps	2
	Optimizer	Adam	β 1 = 0.β 2 = 0.999

Table B .

 B 4 Density/Coverage curves comparing TransGAN and boosting methods for multimodal datasets and different threshold ratios. Our simplicial truncation method (TransGAN + simp.) consistently outperforms the TransGAN and TransGAN + DeliGAN baselines. ± 0.1 72.8 ± 0.862.6 ± 0.779.3 ± 0.979.3 ± 1.2 TransGAN + 90% JBT 8.7 ± 0.1 73.0 ± 0.661.9 ± 0.883.5 ± 1.780.0 ± 1.6 TransGAN + 80% JBT 8.8 ± 0.1 73.3 ± 0.861.2 ± 1.085.7 ± 2.881.1 ± 0.6 TransGAN + DeliGAN N=10 9.8 ± 0.1 74.6 ± 0.858.6 ± 0.993.2 ± 2.880.0 ± 0.6 TransGAN + lin. N=10 (0.23,0.23) 9.2 ± 0.1 73.1 ± 1.161.9 ± 1.278.1 ± 2.779.4 ± 0.9 (0.23,0.29) 9.2 ± 0.1 73.7 ± 0.861.5 ± 1.283.4 ± 3.379.8 ± 0.5 (0.23,0.31) 9.3 ± 0.1 74.0 ± 0.561.0 ± 0.786.1 ± 1.881.3 ± 0.7 (0.23,0.4) 9.8 ± 0.1 75.1 ± 0.859.8 ± 1.289.5 ± 1.680.5 ± 1.2 TransGAN + simp. N=10 (0.135,0.135) 9.0 ± 0.1 72.9 ± 0.561.8 ± 0.982.7 ± 1.980.4 ± 0.7 (0.135,0.14) 9.0 ± 0.1 74.2 ± 1.560.7 ± 1.088.5 ± 3.181.3 ± 1.4 (0.135,0.1445) 9.3 ± 0.1 75.3 ± 0.658.8 ± 0.898.6 ± 1.382.9 ± 0.5 TransGAN 15.2 ± 0.164.2 ± 0.563.1 ± 0.953.4 ± 1.366.0 ± 1.1 TransGAN + 90% JBT 15.1 ± 0.264.8 ± 1.062.9 ± 1.353.6 ± 2.466.2 ± 1.4 TransGAN + 80% JBT 14.8 ± 0.265.4 ± 1.761.7 ± 1.255.0 ± 4.065.6 ± 2.3 TransGAN Deligan 10 15.9 ± 0.263.5 ± 0.862.2 ± 0.752.6 ± 1.364.4 ± 0.6 TransGAN DeliGAN 100 15.3 ± 0.164.2 ± 0.561.9 ± 0.952.6 ± 0.665.9 ± 0.8 ± 0.165.1 ± 0.662.3 ± 0.555.6 ± 0.667.1 ± 0.5 (0.135, 0.14) 15.1 ± 0.164.8 ± 0.261.1 ± 0.555.3 ± 1.366.8 ± 1.1 (0.135,0.1445) 15.1 ± 0.165.6 ± 1.361.5 ± 0.856.3 ± 1.566.4 ± 1.4 TransGAN 10.5 ± 0.175.7 ± 0.660.1 ± 0.887.5 ± 1.983.0 ± 0.2 TransGAN + 90% JBT 10.5 ± 0.176.9 ± 0.758.8 ± 0.591.9 ± 1.982.1 ± 0.8 TransGAN + 80% JBT 11.0 ± 0.178.1 ± 0.357.6 ± 1.399.3 ± 2.883.8 ± 0.7 TransGAN DeliGAN 10 12.1 ± 0.174.2 ± 1.260.2 ± 0.581.5 ± 1.579.6 ± 0.8 TransGAN DeliGAN 100 10.5 ± 0.276.0 ± 0.560.2 ± 1.685.5 ± 2.881.5 ± 1.4 ± 0.177.8 ± 0.659.8 ± 0.894.1 ± 0.983.5 ± 0.8

	Dataset	Model	FID	Prec	Rec	Dens.	Cov.
	CIFAR-10 8.9 CIFAR-100 TransGAN
		TransGAN + simp. N=10
	(0.135,0.135) 15.1 STL-10 (32x32)
		TransGAN + simp. N=100
		(0.08,0.08)	10.1 ± 0.176.5 ± 0.960.2 ± 0.890.0 ± 1.783.0 ± 0.5
		(0.08,0.15)	10.0 ± 0.176.9 ± 0.859.9 ± 0.691.4 ± 1.183.8 ± 0.3
		(0.08,0.20)	10.0

  Table D.1 Image inpainting. Masked L1 ↓ FID ↓ Density ↑ Coverage ↑

	FFHQ: rect. masks
	I2SG++ (Abdal et al., 2020)	0.0767	23.6	0.99	0.88
	I2SG † ++

Table D .

 D [START_REF]2 Correlation between latent space geometry and GANs' performance (Details for Section 3.3.5.3) We present the full results of this study in Table B.3. Table B.3 Correlation between GANs' performance and their latent space geometry. Increasing the capacity of GANs tend to structure their latent space in simplicial clusters (better LogReg accuracy) and improve their performance on precision, density and coverage. Confidence intervals are computed on several sets of generated/training points from a given generator[END_REF] Detailed results of the user study. Each line corresponds to an image, with the associated number of votes per method.

	EdiBERT	ID-GAN	LC	I2SG †++
		(Zhu et al., 2020) (Chai et al., 2021) (Abdal et al., 2019)
	17	5	7
	15	1	8
	22	4	1
	19	4	5
	22	0	4
	6	7	8
	21	1	5
	23	1	4
	20	5	5
	11	13	6
	27	0	3
	12	3	3	12
	16	4	6
	25	2	1
	18	8	1
	8	13	9
	26	0	4
	7	0	21
	14	9	1
	27	0	1
	11	19	0
	14	9	4
	16	14	0
	21	1	3
	8	2	18
	19	3	3
	22	7	0
	23	2	1
	18	0	2	10
	27	2	1
	22	2	1
	24	0	5
	3	25	2
	28	0	2
	24	0	6
	27	0	3
	27	1	2
	22	7	1
	9	15	6
	14	0	14
	Total 735: 61.25%	189: 15.75%	177: 14.75%	99: 0.0825%

Table E .

 E 1 Ablation studies on T-WUTON. Quantitative metrics on paired setting (LPIPS and SSIM) and on unpaired setting (IS and FID). For LPIPS and FID, the lower is the better. For SSIM and IS, the higher is the better. ± reports std. dev.

	Method	T-WUTON	W/o L adv	Paired L adv	Not end-to-end
	Paired L adv			✓
	Unpaired L adv	✓			✓
	End-to-end	✓	✓	✓
	LPIPS	0.101 ± 0.047 0.107 ± 0.049 0.099 ± 0.046 0.112 ± 0.053
	SSIM	0.799 ± 0.089 0.799 ± 0.088 0.800 ± 0.089 0.799 ± 0.089
	IS	3.114 ± 0.118 2.729 ± 0.091 3.004 ± 0.091 3.102 ± 0.077
	FID	9.877	13.020	8.298	11.125

GANs[START_REF] Goodfellow | Generative adversarial nets[END_REF] and VAEs[START_REF] Kingma | Auto-encoding variational bayes[END_REF] have shown great capacities to generate photorealistic images[START_REF] Karras | Alias-free generative adversarial networks[END_REF][START_REF] Vahdat | Nvae: A deep hierarchical variational autoencoder[END_REF]. These two models are also helpful for diverse tasks such as image editing[START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF][START_REF] Wu | Stylespace analysis: Disentangled controls for stylegan image generation[END_REF] or unsupervised image segmentation[START_REF] Abdal | Labels4free: Unsupervised segmentation using stylegan[END_REF][START_REF] Zoran | Parts: Unsupervised segmentation with slots, attention and independence maximization[END_REF]. GANs and VAEs rely on learning a Lipschitz-continuous transformation from a low dimensional Gaussian space. As such, they have been described as push-forward generative models[START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF]. According to the same taxonomy, score-based models can be defined as indirect push-forward generative models since they result from the composition of a large number of transformations and are trained with an auxiliary denoising objective.The present section aims at making a step towards a better understanding of push-forward generative models such as GANs. In particular, the goal is to shed light on the latent space of these architectures, and to stress how it impacts the performance of both GANs and VAEs. If empirical studies such as[START_REF] Donahue | Large scale adversarial representation learning[END_REF] have suggested the emergence of simple geometrical structure in the latent space of GANs, there is still a poor theoretical understanding of how generators organize their latent space.To better understand the latent space of generative models, the setting of disconnected distributions learning is enlightening. Experimental and theoretical works[START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF][START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF] have shown a fundamental limitation of push-forward generative

4.2 EdiBERT: a generative model for image editingAbstract Advances in computer vision are pushing the limits of image manipulation, with generative models sampling highly-realistic detailed images on various tasks. However, a specialized model is often developed and trained for each specific task, even though many image edition tasks share similarities. In denoising, inpainting, or image compositing, one always aims at generating a realistic image from a low-quality one. In this section, we aim at making a step towards a unified approach for image editing. To do so, we propose Ed-iBERT, a bidirectional transformer that re-samples image patches conditionally to a given image. Using one generic objective, we show that the model resulting from a single training matches state-of-the-art GANs inversion on several tasks: image denoising, image completion, and image composition. We also provide several insights on the latent space of vectorquantized auto-encoders, such as locality and reconstruction capacities. The code is available at https://github.com/EdiBERT4ImageManipulation/EdiBERT.

'Black swan' class.

C.7 Qualitative results of latentGA.

Remerciements

Although it eases the training of G, h is a bottleneck in the virtual try-on pipeline. We will show that we can train a student model with synthetic triplets {(p, c), p}, where p comes from our pre-trained teacher generative model in Eq. 4.3.1. This allows to remove the need for h at inference time for the student model:

where G s and ST N s are the student modules and p the generated image.

Conditional image generation. Generative models for image synthesis have shown impressive results with adversarial training [START_REF] Goodfellow | Generative adversarial nets[END_REF]. Combined with deep networks [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF], this approach has been extended to conditional image generation in [START_REF] Mirza | Conditional generative adversarial nets[END_REF] and performs increasingly well on a wide range of tasks. However, as noted by Alami [START_REF] Alami Mejjati | Unsupervised attention-guided image-to-image translation[END_REF], these models cannot handle large spatial deformations and fail to modify the shape of objects, which is necessary for a virtual try-on.

Appearance transfer. Close to the virtual try-on task, some research focus on human appearance transfer. Given two images of different persons, the goal is to transfer the appearance of a part of the person A on the person B. Approaches using pose and appearance disentanglement [START_REF] Lorenz | Unsupervised part-based disentangling of object shape and appearance[END_REF][START_REF] Ma | Disentangled person image generation[END_REF] fit this task but others are specifically designed for it. SwapNet [START_REF] Raj | Swapnet: Garment transfer in single view images[END_REF]) is a dual path network which generates a new human parsing of the reference person and region of interest pooling to transfer the texture. In [START_REF] Wu | M2e-try on net: Fashion from model to everyone[END_REF], the method relies on DensePose information [START_REF] Güler | Densepose: Dense human pose estimation in the wild[END_REF], which provides a 3D surface estimation of a human body, to perform a warping and align the two persons. The transfer is then done with segmentation masks and refinement networks. However, the warping relies on matching source and target pose, which is not feasible for the virtual try-on task.

Virtual try-on. Most of the approaches for a virtual try-on system come from computer graphics and rely on 3D measurements or representations. Drape [START_REF] Guan | Drape: Dressing any person[END_REF]) learns a deformation model to render clothes on 3D bodies of different shapes. [START_REF] Hahn | Subspace clothing simulation using adaptive bases[END_REF] Appendix A

A.1 Technical results

A.1.1 Highlighting drawbacks of the Precision/Recall metric Lemma A.1.1. Assume that the modeled distribution µ θ slightly collapses on a specific data point, i.e. there exists x ∈ E, µ θ (x) > 0. Assume also that µ ⋆ is a continuous probability measure and that µ θ has a recall β = 1. Then the precision must be such that α = 0.

Proof. Using Definition 3.2.1, we have that there exists µ such that

Thus, 0 = µ ⋆ (x) ⩾ α µ(x) = α µ θ (x). Which implies that α = 0.

A.1.2 Proof of Theorem 3.2.1

The proof of Theorem 3.2.1 relies on theoretical results from non-parametric estimation of the supports of probability distribution studied by [START_REF] Devroye | Detection of abnormal behavior via nonparametric estimation of the support[END_REF].

For the following proofs, we will require the following notation: let ϕ be a strictly monotonous function be such that lim

the open ball centered in x and of radius r. For a given probability distribution µ, S µ refers to its support. We recall that for any x in a dataset D, x (k) denotes its k nearest neighbor in D. Finally, for a given probability distribution µ and a dataset D µ sampled from µ n , we note R min and R max the following:

In the following lemma, we show asymptotic behaviours for both R min and R max .

For further analysis, when M → ∞, refer to subsection A.1.5 and note using the result in (A.1.14) that one obtains the desired upper-bound on ᾱ

A.1.4.2 More general setting

As done previously, we denote M, the number of classes covered by the estimator µ θ , such that for all i ∈ [1, M], we have γ(A i ) > 0. We still assume that M > 1. However, we now relax the previous assumption made in (A.1.5) and assume the milder assumption that there exists

In this setting, it is clear that A 1 , . . . , A M, A ∁ is a a partition of R d under the measure γ. Using, result from Theorem A.1.1, we have

ds. Finally, we have that

In the case where γ(A ∁ ) = 0, we find a result similar to (A.1.6).

A.1.5 Lower-bounding boundaries of partitions in a Gaussian space Notations and preliminaries Given ε ≥ 0 and a subset A of euclidean space A.3 Supplementary details

A.3 Supplementary details

We now provide the different network's architecture used and their corresponding hyperparameters. For DeliGan, we use the same architecture and simply add 50 Gaussians for the reparametrization trick. For DMLGAN, we re-use the architecture of the authors.

where

Proof by contradiction. Assume a generator G such that there exists z ∈ ∂ ε min A and i ∈ [1, m] such that G(z) ∈ M i . Since G is associated with A , we have using Definition 3.3.2, that there exists z ′ and j ∈ [1, m], j ̸ = i such that ∥zz ′ ∥ < ε min /2 and j = arg min k∈ [1,m] ∥G(z ′ ) -M k ∥. Thus, we have:

This contradicts G being in G A L . Training. We use the Wasserstein loss with gradient-penalty on interpolations of fake and real data. At each iteration, the discriminator is trained 2 steps and the generator 1 step with Adam optimizer. The batch size is 256. The learning rate of the discriminator is two times larger [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF], i.e. 5 × 10 -5 for the generator and 1 × 10 -4 for the discriminator. GANs are trained for 80k steps on MNIST and for 100k steps on CIFAR datasets. Architectures of generator and discriminator are described in Table B.1 and Table B.2.

Appendix C

C.1 Proof of Lemma 3.4.1

Let's prove that E γ ϕ = 1. We have that:

by assumption. Consequently, the measure γ ϕ is a well-defined probability distribution on R d .

C.2 Proof of Theorem 3.4.1

It is clear that the network w ϕ is a density function with respect to the distribution γ defined on R d . Consequently, the measure µ Consequently, we have:

(where

where δ refers to the Dirac probability distribution. Finally, when taking the infinimum over all continuous functions ϕ, we have that:

C.3 Evaluation details

Precision recall metric. For the precision-recall metric, we use the algorithm from [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF]. Namely, when comparing the set of real data points (x 1 , ..., x n ) with the set of fake data points (y 1 , ..., y n ):

A point x i has a recall r(x i ) = 1 if there exists y j , such that ∥x iy j ∥ ≤ ∥y jy j (k)∥, where y j (k) is the k-nearest neighbor of n. Finally, the recall is the average of individual recall:

A point y i has a precision p(y i ) = 1 if there exists x j , such that ∥y ix j ∥ ≤ ∥x jx j (k)∥, where x j (k) is the k-nearest neighbor of n. Finally, the precision is the average of individual precision: 1 n ∑ i p(x i ). Parameters. For all datasets, we use k = 3 (3rd nearest neighbor). For MNIST and F-MNIST, we use a set of n = 2048 points. For CelebA and LSUN Church, we use a set of n = 1024 points. This is also valid for the EMD. For FID, we use the standard protocol with n = 50000 points and Inception Net. We run 10 evaluations of each metric (each evaluation is done with a different set of random points), report the average and the 97% confidence interval by considering that we have 10 i.i.d. samples from a normal distribution.

C.4 Sampling algorithms: latentRS, latentGA, and laten-tRS+GA

We present here the three sampling algorithms associated with our importance weight function w ϕ . See Section C.5 for details on the hyper-parameters used in latentGA and latentRS+GA. SIR [START_REF] Grover | Bias correction of learned generative models using likelihood-free importance weighting[END_REF]: Model selection: we fine-tune with a binary cross-entropy loss the discriminator from the end of the adversarial training and select the best model in terms of EMD. We tested with/without regularizing the discriminator during the fine-tuning (with gradient penalty or spectral normalization). Without regularization, the performance drops fast. Best results are obtained by regularizing the discriminator, thus we report these results.

We use then use Sampling-Importance-Resampling algorithm. In SIR, we sample N points from the generator, compute their importance weights according density ratios, and accept one of them (each point is accepted with a probability proportional to its importance weight). The hyper-parameter of SIR algorithm is N. Results for grid search on N are shown below in Table C.2 and Table C.3. In Table 3.5, results are shown with N = 10.

DOT Tanaka (2019): Model selection: we fine-tune with the WGAN-GP loss the discriminator from the end of the adversarial training and select the best model in terms of EMD, when running DOT. We perform a projected gradient descent as described in [START_REF] Tanaka | Discriminator optimal transport[END_REF] with SGD. Hyper-parameters are the number of steps N steps and the step size ε. We made the following grid search: N steps = [2, 5, 10, 50] and ε = [0.01, 0.05, 0.1]. Results for grid search on N steps are shown below in Table C.2 and Table C.3. In Table 3.5, results are shown with N steps = 10 and ε = 0.05 or ε = 0.01 depending on the dataset (we select the best one).

Training of w ϕ : For MNIST and F-MNIST, we use the same hyper-parameters: λ 1 = 10, λ 2 = 3 and m = 3. w ϕ is a standard MLP with 4 hidden layers, each having 400 nodes (4x dimension of latent space), and relu activation. The output layer is 1-dimensional and with a relu activation. The learning rate of the discriminator is 4 * 10 -4 , the learning rate of w ϕ is 10 -4 . The two networks are optimized with Adam algorithm, where we set β = (0.5, 0.5). We use 1 step of importance weight optimization for 1 step of discriminator optimization.

For Progressive GAN on CelebA (128x128), we use: λ 1 = 20, λ 2 = 5 and m = 3. w ϕ is a standard MLP with 4 hidden layers, each having 512 nodes (1x dimension of latent space), and leaky-relu activation (0.2 of negative slope). The output layer is 1-dimensional and with relu activation. Since we do not have the pre-trained discriminator, we first train a WGAN-GP discriminator between ProGAN and CelebA images for 500 steps, and then start the adversarial training of w ϕ . The learning rate of the discriminator is 10 -4 , the learning rate of w ϕ is 10 -5 . The two networks are optimized with Adam algorithm, where we set β = (0., 0.999). During optimization, we perform iteratively 3 w ϕ updates and 1 discriminator's updates.

For StyleGAN2 on LSUN Church (256x256), we use: λ 1 = 30, λ 2 = 5 and m = 2. w ϕ is a standard MLP with 3 hidden layers, each having 512 nodes (1x dimension of latent space), and leaky-relu activation (0.2 of negative slope). The output layer is 1-dimensional and with a relu activation. Since we do not have the pre-trained discriminator, we first train a WGAN-GP discriminator between StyleGAN2 and LSUN Church images for 500 steps, and then start the adversarial training of w ϕ . The learning rate of the discriminator is 10 -4 , the learning rate of w ϕ is 10 -5 . The two networks are optimized with Adam algorithm, where we set β = (0., 0.999). During optimization, we perform 3 w ϕ updates for 1 discriminator's updates.

LatentRS: Once the network w ϕ is trained (see above), there is no hyper-parameter for latentRS algorithm.

LatentGA and latentRS+GA: We use the same neural network than in LatentRS. The hyper-parameters for this method are similar to DOT: number of steps of gradient ascent N steps and step size ε. With the model selected on LRS, we make the following grid search: 5,10,50] and ε = [0.01, 0.05, 0.1]. Best results were obtained with ε = 0.05 on all datasets. Results for grid search on N steps are shown below in Table C.2 and Table C.3. In Table 3.5, results are shown with N steps = 10 and ε = 0.05.

C.6 Comparisons with concurrent methods on synthetic and real-world datasets

In this section, we provide more quantitative results: a comparison of SIR, DOT, SIR, LatentRS and latentRS+GA on MNIST and F-MNIST in Table C.1; an ablation study on the impact of number of points (respectively gradient ascent steps) in SIR (respectively DOT, latentGA and latentRS+GA), on ProGAN trained on CelebA in Table C.2 and StyleGAN2 trained on Lsun Church in Table C.3.

Table C.1 latentRS+GA is the best performer and latentRS matches SOTA with a significantly reduced inference cost (by an order of at least 10). FID was computed using the same datasetspecific classifier used for the Precision/Recall metric. ± is 97% confidence interval. Inference refers to the time in milliseconds needed to compute one image on a NVIDIA V100 GPU.

Appendix D D.1 Implementation details

The code for the implementation of EdiBERT is available on GitHub at the following link https://github.com/EdiBERT4ImageManipulation/EdiBERT.

A pre-trained model on FFHQ is available on a linked Google Drive. Notebooks to showcase the model have also been developped.

D.1.1 Training hyper-parameters

We use the same architecture than [START_REF] Esser | Taming transformers for high-resolution image synthesis[END_REF] for both VQGAN and transformer. On LSUN Bedroom and FFHQ, we use a codebook size of 1024. For the transformer, we use a model with 32 layers of width 1024.

To train the transformer with 2D masking strategy, we generate random rectangles before flattening Q(E(I)). The height of rectangles is drawn uniformly from [0.2 × h, 0.5 × h]. Similarly, the width of rectangles is drawn uniformly from [0.2 × w, 0.5 × w]. In our experiments, since we work at resolution 256 × 256 and follow the downsampling factor of 4 from [START_REF] Esser | Taming transformers for high-resolution image synthesis[END_REF], we have h = w = 256/16 = 16.

Tokens outside the rectangle are used as input, to give context to the transformer, but not for back-propagation. Tokens inside the rectangle are used for back-propagation. p rand = 90% of tokens inside the mask are put to random tokens, while p same = 1p rand = 10% are given their initial value. Although we did not perform a large hyper-parameter study on this parameter, we feel it is an important one. The lower p rand , the more the learned distributions p i θ (.|s) will be biased towards the observed token s i . However, setting p rand = 1 leads to a model that diverges too fast from the observed sequence. We set the number of epochs to 2, collage frequency to 4 per epoch, top-k sampling to 100, dilation to 1, and number of optimization steps to 200. We apply a gaussian filter on the mask for the periodic image collage.

Additionally, we use these two implementation details. 1) We use two latent masks: the latent down-sampled mask latent_mask 1 , and the dilated mask latent_mask 2 , obtained by a dilation of latent_mask 1 . The randomization is done with latent_mask 1 , such that no information from the unmasked parts of the image is erased. However, the selection of positions that are re-sampled by EdiBERT is done with latent_mask 2 . 2) At the second epoch, we randomize the token value, at the position that is being replaced. This is only done for image inpainting.

D.1.2.2 Image compositing.

We set the number of epochs to 2, collage frequency to 4 per epoch, top-k sampling to 100, dilation to 1, and number of optimization steps to 200. We apply a gaussian filter on the mask for the periodic image collage. Contrarily to inpainting, we do not randomize such that EdiBERT samples stay closer to the original sequence.

The full algorithm is presented below in Algorithm 7.

D.2 Additional experimental results

We give additional comparisons on FFHQ and LSUN Bedroom, for the following tasks: image inpainting in Inpainting. We use 2500 images. On FFHQ, we provide results for free-form masks and rectangular masks. The height of rectangular masks is drawn uniformly from [0.4 × h, 0.6 × h] with h = 256, and similarly for the width. For non-rectangular masks generations, we follow the procedure of [START_REF] Chai | Using latent space regression to analyze and leverage compositionality in gans[END_REF]: we draw a binary mask at low-resolution 6 × 6 and uspsample it to 256 × 256 with bilinear interpolation.

The ablation study in Table 4.2 of main paper is performed on free-form masks. Results in

D.5 Survey on FFHQ image compositing

The survey was presented as a Google Form with 40 questions. For each question, the user was shown 6 images: Source, Composite, EdiBERT, ID-GAN [START_REF] Zhu | In-domain gan inversion for real image editing[END_REF], I2SG †++ [START_REF] Abdal | Image2stylegan: How to embed images into the stylegan latent space[END_REF] based on pre-trained network from Karras et al. (2020a), LC [START_REF] Chai | Using latent space regression to analyze and leverage compositionality in gans[END_REF]. The different generated images were referred as Algorithm 1, ..., Algorithm 4. The user was asked to vote for its preferred generated image, by taking into account realism and fidelity criterions. The user had no time limit for the poll. 30 users answered our poll. We provide the detailed answers for each image in Table D. [START_REF]2 Correlation between latent space geometry and GANs' performance (Details for Section 3.3.5.3) We present the full results of this study in Table B.3. Table B.3 Correlation between GANs' performance and their latent space geometry. Increasing the capacity of GANs tend to structure their latent space in simplicial clusters (better LogReg accuracy) and improve their performance on precision, density and coverage. Confidence intervals are computed on several sets of generated/training points from a given generator[END_REF].

Appendix E E.1 Implementation details

Convolutional geometric matcher. To extract the feature maps, we apply five times one standard convolution layer followed by a 2-strided convolution layer which downsamples the maps. The depth of the feature maps at each scale is (16,32,64,128,256). The correlation map is then computed and feeds a regression network composed of two 2-strided convolution layers, two standard convolution layers and one final fully connected layer predicting a vector θ ∈ R 50 . We use batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] and relu activation. The parameters of the two feature maps extractors are not shared. Siamese U-net generator. We use the same encoder architecture as in the convolutional geometric matcher, but we store the feature maps at each scale. The decoder has an architecture symmetric to the encoder. There are five times one standard convolution layer followed by a 2-strided deconvolution layer which upsamples the feature maps. After a deconvolution, the feature maps are concatenated with the feature maps passed through the skip connections. In the generator, we use instance normalization, which shows better results for image and texture generation [START_REF] Ulyanov | Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis[END_REF], with relu activation.

Discriminator. We adopt the fully convolutional discriminator from Pix2Pix Isola et al. ( 2017), but with five downsampling layers instead of three in the original version. Each of it is composed of: 2-strided convolution, batch normalization, leaky relu, 1-strided convolution, batch normalization, leaky relu.

Adversarial loss. We use the relativistic formulation of the adversarial loss Jolicoeur-Martineau (2019). In this formulation, the discriminator is trained to predict that real images are more real than synthesized ones, rather than trained to predict that real images are real and synthesized images are synthesized. We also use gradient penalty on the discriminator.

Optimization. We use the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with β 1 = 0.5, β 2 = 0.999, a learning rate of 10e -3 and a batch size of 8. Also, we use λ p = λ L1 = λ w = λ adv = 1. E.1 show the importance of our end-to-end learning of geometric deformations. When the geometric matcher only benefits from L warp , it only learns to align c with the masked area in p ⋆ . However, it does not preserve the inner structure of the cloth. Back-propagating the loss computed on the synthesized images p alleviates this issue. The quantitative results of IS and SSIM scores on

E.3 Ablation studies on T-WUTON