Issenhuth Thibaut

Tanielian Ugo

Mary Jérémie

Picard David

David Picard

Unveiling

Thibaut Issenhuth

Ugo Tanielian

Jérémie Mary

Unveiling the Latent Space Geometry of Push-Forward Generative Models

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

GANs (Goodfellow et al., 2014) and VAEs [START_REF] Diederik | Auto-encoding variational {Bayes}[END_REF] have shown great capacities to generate photorealistic images [START_REF] Karras | Aliasfree generative adversarial networks[END_REF][START_REF] Vahdat | Nvae: A deep hierarchical variational autoencoder[END_REF]. These two models are also helpful for diverse tasks such as image editing [START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF][START_REF] Wu | Stylespace analysis: Disentangled controls for stylegan image generation[END_REF] or unsupervised image segmentation [START_REF] Abdal | Labels4free: Unsupervised segmentation using stylegan[END_REF][START_REF] Daniel Zoran | Parts: Unsupervised segmentation with slots, attention and independence maximization[END_REF]. GANs and VAEs rely on learning a Lipschitzcontinuous transformation from a low dimensional Gaussian space. As such, they have been described as push-forward generative models [START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF]. According to the same taxonomy, score-based models can be defined as indirect push-forward generative models since they result from the composition of a large number of transformations and are trained with an auxiliary denoising objective.

The present paper aims at making a step towards a better understanding of push-forward generative models such as GANs. In particular, the goal is to shed light on the latent space of these architectures, and to stress how it impacts the performance of both GANs and VAEs. If empirical studies such as [START_REF] Donahue | Large scale adversarial representation learning[END_REF] have suggested the emergence of simple geometrical structure in the latent space of GANs, there is still a poor theoretical understanding of how generators organize their latent space. We would like to highlight that although our theoretical results apply to all pushforward generative models, our primary emphasis in the experiment section is on GANs.

To better understand the latent space of generative models, the setting of disconnected distributions learning is enlightening. Experimental and theoretical works [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF][START_REF] Tanielian | Learning disconnected manifolds: a no gan's land[END_REF][START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF] have shown a fundamental limitation of push-forward generative models. Since the modeled distribution is connected, some areas of its support are necessarily mapped outside the true data distribution. However, when covering several modes of a disconnected distribution, generators still try to minimize the numbers of samples lying outside the true modes (e.g. the purple area on the right of Figure 1). In other words, generators aim at minimizing the measure of the existing borders between the modes in the latent space. Considering a Gaussian latent space, finding such minimizers is closely linked to Gaussian isoperimetric inequalities [START_REF] Ledoux | Isoperimetry and gaussian analysis[END_REF] where the goal is to derive the partitions that split a Gaussian space with minimal Gaussian-weighted perimeters. Most notably, a recent result [START_REF] Milman | The gaussian doublebubble and multi-bubble conjectures[END_REF] shows that, as long as the number of components m in the partition and the number of dimensions d of the Gaussian space are such that m ≤ d + 1, the optimal partition is a 'simplicial cluster': a Voronoi diagram with equidistant seeds, see left of Figure 1 for m = 3 and d = 3.

In this paper, we demonstrate the effectiveness of applying simplicial clusters to the latent space of push-forward generative models. We show both experimentally and theoretically that generators with a latent space structured as a simplicial cluster minimize the occurrence of out-of-distribution generated samples. Using the precision metric (Sajjadi et al.,Figure 1. Illustration of the capability of GANs to discover an optimal geometry of the latent space. On the left, the propeller shape represents a partition of 3D Gaussian space with the smallest Gaussian-weighted perimeter (Figure from [START_REF] Heilman | Solution of the propeller conjecture in R 3[END_REF]). On the right, we show the 3D Gaussian latent space of a GAN trained on three classes of MNIST. Each area colored in blue, green, or red corresponds to samples in one of the three classes. Using a pre-trained classifier, we highlight in purple the samples with low-confidence, and observe that the partition reached by the GAN (right) is close to optimality (left), as the latent space partition is similar to the intersection of the propeller on a sphere. 2018; Kynkäänniemi et al., 2019), we show that generators with a simplicial cluster latent space achieve optimal precision levels and provide both an upper and a lower bound on their precision. Our experiments reveal that GANs with higher performances tend to organize their latent space as simplicial clusters. More importantly, we illustrate that enforcing this 'simplicial structure' with a truncation method can boost GANs' performance. Interestingly, simplicial clusters are highly similar to the 'simplex Equiangular Tight Frames' observed in the last-layer features of deep classification networks [START_REF] Vardan Papyan | Prevalence of neural collapse during the terminal phase of deep learning training[END_REF]. This study stresses that they also naturally emerge in deep push-forward generative models. Our contributions are the following:

• We are the first to build on the latest results from Gaussian isoperimetric inequalities by [START_REF] Milman | The gaussian doublebubble and multi-bubble conjectures[END_REF] in the study and understanding of push-forward generative models.

• We present a new theoretical analysis, providing both an upper bound on the precision of push-forward generative models. We demonstrate that generators with a latent space organized as a simplicial cluster have an optimal precision, with lower bounds that decrease in √ m log m, where m is the number of modes.

• Experimentally, we verify that GANs tend to structure their latent space as simplicial clusters' by exploring two properties of the latent space: linear separability and convexity of classes. Also, we analyse the impact of latent space dimension on GANs, and reveal a positive correlation between GANs' performance and latent space geometry.

• Finally, we show that enforcing a simplicial structure into GANs' latent space can boost their performance and outperforms other boosting methods.

Related Work

Notation

Data. We consider a target distribution µ defined on a Euclidean space R D , which may be a high-dimensional space, and equipped with the Euclidean norm • . We use S µ to represent the support of any distribution µ.

Push-forward generative models. We consider the set of L-Lipschitz continuous functions, denoted as G L , from the latent space R d to the high-dimensional space R D . The primary goal of each generator in this set is to produce realistic samples. The distribution in the latent space, defined on R d , is assumed to be Gaussian and is represented as γ.

For each generator G ∈ G L , we associate the push-forward distribution (or image distribution) of γ by G, and denote it G γ, where denotes the push-forward operator. In the context of generative models, each distribution G γ is now a candidate distribution to represent µ .

The Lipschitzness assumption on G L is reasonnable: Virmaux and Scaman (2018) have shown the lipschitzness of deep neural networks, and have developed an algorithm that can upper-bound their Lipschitz constant. While deep neural networks can have high Lipschitz constants, it is possible to constrain this in practice by techniques such as clipping the neural network's parameters [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF], penalizing the discriminative functions' gradient (Gulrajani et al., 2017;[START_REF] Kodali | On convergence and stability of GANs[END_REF][START_REF] Wei | Improving the improved training of Wasserstein GANs: A consistency term and its dual effect[END_REF][START_REF] Zhou | Lipschitz generative adversarial nets[END_REF], or penalizing the spectral norms [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF]. Large-scale generators such as SAGAN [START_REF] Zhang | Self-attention generative adversarial networks[END_REF] and BigGAN [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]) also make use of spectral normalization for the generator.

Generative models and disconnected distributions

The phenomenon of misspecification in continuous generative models, while primarily studied in the context of GANs, is also relevant to other families such as VAEs or normalizing flows [START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF]. This issue has been investigated both experimentally [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] and theoretically [START_REF] Tanielian | Learning disconnected manifolds: a no gan's land[END_REF][START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF]. The problem stems from a fundamental trade-off: continuous generators can either cover all modes, resulting in out-of-manifold samples, or generate only high-quality samples, neglecting some modes. To address this, various methods have been proposed, such as training disconnected distributions [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF][START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF] or deriving rejection mechanisms from pre-trained generators [START_REF] Azadi | Discriminator rejection sampling[END_REF][START_REF] Tanielian | Learning disconnected manifolds: a no gan's land[END_REF][START_REF] Imtiaz Humayun | Polarity sampling: Quality and diversity control of pre-trained generative networks via singular values[END_REF].

Empirical studies have provided valuable insights into the structure of the latent space of generative models. For example, [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF] demonstrate that binary attributes are linearly separable in the Gaussian latent space and even more separable in an intermediate latent space. Similarly, [START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF] find that face attributes are separated by hyperplanes in the latent space. [START_REF] Arvanitidis | Latent space oddity: on the curvature of deep generative models[END_REF] and Chen et al. (2018a) view the latent space of generative models with a Riemannian perspective.

While these findings provide valuable insights into the latent space structure of generative models, they may not be sufficient for a comprehensive understanding of the latent space geometry. For instance, [START_REF] Tanielian | Learning disconnected manifolds: a no gan's land[END_REF] stress the relevance of this problem by showing that the precision of GANs can converge to 0 when the number of modes or the distance between them increases. In this paper, we take a step towards a deeper understanding of the behavior of push-forward generative models and reveal an optimal latent space configuration when the number of modes m and the dimension of the latent space d are such that m ≤ d + 1.

Evaluating generative models

When learning disconnected manifolds, Sajjadi et al. (2018) illustrated the need for measures that simultaneously evaluate both the quality (Precision), and the diversity (Recall) of the generated samples. However, Kynkäänniemi et al. (2019) pointed out an important limitation of the PR metric: it cannot accurately interpret situations when large numbers of samples are packed together. They propose an Improved PR metric based on the non-parametric estimation of manifolds to correct this.

Improved PR metric. Informally, for a generator G, precision (α G ) quantifies the proportion of generated samples that can be approximated with true samples, while recall (β G ) measures the proportion of true samples that can be approximated with generated ones. Applying this to GANs, using the target distribution µ and modeled distribution G γ, the Improved PR metric was shown, by Tanielian et al. (2020, Theorem 1), to be asymptotically equivalent to:

α n G → n→∞ α G = G γ S µ and β n G → n→∞ β G = µ S G γ ,
where S µ denotes the support of µ and n is the number of samples. However, [START_REF] Naeem | we use 10k real images and 10k fake images with nearest-k= 5[END_REF] have shown that the Improved PR metric (Kynkäänniemi et al., 2019) is sensitive to outlier samples of both the target and the generated distribution. To correct this and fix the overestimation of the manifold around real outliers, [START_REF] Naeem | we use 10k real images and 10k fake images with nearest-k= 5[END_REF] propose the Density/Coverage metric.

Density/Coverage. Instead of counting how many fake samples belong to a real sample neighborhood, density counts how many real sample neighborhoods contain a generated sample. On the other hand, coverage counts the number of real sample neighborhoods that contain at least one fake sample.

In the next analysis both theoretical and experimental, we use both notions of precision and density defined above.

Simplicial Structure in Push-Forward Generative Models

The goal is to gain a deeper understanding of the latent space of push-forward generative models and identify which ones possess the highest precision under certain conditions. As previously mentioned, push-forward generative models map a unimodal Gaussian distribution γ through a Lipschitzcontinuous function, represented by a generator G. As a result, the modeled generative distribution G γ necessarily has a connected support.

In cases where the target distribution µ contains disconnected manifolds, generators have to generate fake data points that fall outside of the true manifold. This prompts the question: given that a generator samples data points from each of the distinct modes, what is the maximum precision that it can achieve? To begin with, let's consider a target distribution µ composed of m disconnected modes.

Assumption 3.1 (Disconnected manifolds). The target distribution µ consists of m disconnected spheres S i , i ∈ [1, m] of equal measure (with centers X i and radius r i ). Additionally, the spheres satisfy the two following properties:

• Small individual radius: each radius r i satisfies

r i < min j X i -X j 2 .
(1)

• Each distance X i -X j satisfies:

min k∈[1,m],k =i,j (X i + X j )/2 -X k > X i -X j 2 .
(2)

We believe that the assumption of disconnectedness is a reasonable one, particularly for multi-class datasets such as MNIST (LeCun et al., 1998), CIFAR10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], or STL10 [START_REF] Coates | An analysis of single-layer networks in unsupervised feature learning[END_REF]. To validate this property, we run a pre-trained CLIP [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] on the dataset, identify a certain number of clusters using a Kmeans algorithm, and further test the disconnectedness of these modes by training a linear classifier. The accuracy on these datasets is 98.1% on MNIST, 93.9% on CIFAR10, and 92.7% on STL10.

The second point in (2) has a direct impact on the location of the data points X 1 , . . . , X m . Specifically, it implies that each cell in the Voronoi diagram with seeds X 1 , . . . , X m shares a side with all the other cells. In other words, the dual graph of this Voronoi diagram is complete. This assumption, which is further discussed with specific examples in Figure 2, can be justified by the concentration of distances in high-dimensional spaces: all the modes are roughly at equal distance [START_REF] Beyer | When is "nearest neighbor" meaningful?[END_REF][START_REF] Charu | On the surprising behavior of distance metrics in high dimensional space[END_REF]. Furthermore, a recent work by Papyan et al. ( 2020) has shown that embeddings of deep neural networks trained for classification tend to collapse around means that are equidistant and maximally equiangular to one another. By using these embedded representations to measure distance, the target distribution would thus easily satisfy Assumption 3.1. Projected GANs [START_REF] Sauer | Projected gans converge faster[END_REF] is really close to this idea as the authors show the effectiveness of leveraging a pre-trained classifier when training GANs: instead of directly discriminating images, the discriminator is trained on features extracted from the classifier.

Throughout the rest of the paper, we define the set of wellbalanced generators as those mapping an equal number of data points to each mode of the data distribution:

Definition 3.2. A generator G is well-balanced if for all spheres, we have G γ(S 1 ) = . . . = G γ(S m ).
Considering well-balanced generators is reasonable as many empirical improvements such as WGAN-GP (Gulrajani et al., 2017) or BigBiGAN [START_REF] Donahue | Large scale adversarial representation learning[END_REF] have significantly reduced mode collapse. GANs generate diverse output distributions on datasets such as CIFAR10, CIFAR100, and ImageNet. To validate the use of wellbalanced generators, we conducted a small experiment and evaluated the proportion of each class generated by GANs on MNIST and CIFAR10. On MNIST, the minimum proportion of a class is 9.2 and the maximum 10.9, while on CIFAR10 it is 8.3 and 11.9 (in %). The variance-to-mean ratio is equal to 0.03 for MNIST and 0.22 for CIFAR10.

Precision and the associated partition

Now that we have defined the prerequisites for both the data and the model, we propose to establish a connection between the latent space partition and the precision of a generator. We create a link between the set of generators from R d to R D and the set of partitions in the latent space. Specifically, for each given partition in R d , there exists a set of associated generators defined as follows:

Definition 3.3. For a given partition A = A 1 , . . . , A m on R d , we say that G is associated to A if: for all i ∈ [1, m], for all z ∈ A i , i = arg min j∈[1,m] G(z) -X j .
Each given generator G is associated with a unique partition A in R d . The geometry of the associated partition A plays a key role in explaining the behavior and performance of the generator G. We are interested in maximizing the precision of generative models. Points in the intersection of two cells

A i ∩ A j , (i, j) ∈ [1, m] 2
are equidistant from X i and X j and thus do not belong to any of these modes (since bot r i and r j < X i -X j /2 according to Assumption 3.1). Additionally, due to the generator's Lipschitzness, there is a small neighborhood around the boundary such that any points in this neighborhood are mapped out of the target manifold. This region in the latent space thus reduces the precision. For a given ε > 0, we now define the epsilonboundary of the partition A as follows.

Definition 3.4. For a given partition A = {A 1 , . . . , A m } of R d and a given ε ∈ R + , we denote ∂ ε A the ε-boundary of A, defined as follows.

∂ ε A = m i=1 ∪ j =i A j ε \ ∪ j =i A j ,
where A ε corresponds to the ε-extension of set A. The following lemma makes the connection between the precision of a generator α G and its associated partition A.

Lemma 3.5. Assume that Assumption 3.1 is satisfied and A be a partition in R d . Then, any generator G ∈ G L associated with A verifies:

α G 1 -γ(∂ εmin A). (3)
where

ε min = min i,j X i -X j /L.
Interestingly, this result holds independently of the partition A. It highlights that the geometry of the partition gives an upper-bound on the precision of the generator. Consequently, to properly determine this bound on the precision levels of generative models, one might be interested in determining the measure of this epsilon-boundary ∂ ε A. By using the result from Lemma 3.5, we can derive an upper-bound on the precision that depends on D, L and m:

Corollary 3.6. Assume that Assumption 3.1 is satisfied, m d + 1. Then, there exists L with L D log(m), such that for any well-balanced generator G ∈ G L :

α G 1 -ε min log m e -3/2 (4)
where ε min = min i,j X i -X j /L. In particular, the result in (4) gives an interesting insight when training GANs on a finite number of modes. Tanielian et al. (2020, Theorem 3) showed a similar result but for the asymptotic case when the number of modes increases:

α G m→∞ e -1 8 ε 2 min e -εmin √ log(m)/2 .
(5)

Optimality for push-forward generative models

To exhibit generative models with optimal precision levels, one must look at partitions with the smallest epsilonboundary measures γ(∂ ε A). We argue that this is tightly connected to the theoretical field of Gaussian isoperimetric inequalities. Isoperimetric inequalities link the measure of sets with their perimeters. More specifically, these inequalities highlight minimizers of the perimeter for a fixed measure, e.g. the sphere in an euclidean space with a given Lebesgue measure. In the Gaussian space, [START_REF] Borell | The brunn-minkowski inequality in gauss space[END_REF] and [START_REF] Vladimir | Extremal properties of half-spaces for spherically invariant measures[END_REF] show that in a finitedimensional case, among all sets of a given measure, halfspaces have a minimum Gaussian perimeter. More formally, for any Borel set A in R d and a half-space H, if we have γ(A) γ(H), then γ(A ε ) γ(H ε ) for any ε > 0, where A ε denotes the ε-extension of A.

The Gaussian multi-bubble conjecture was formulated when looking for a way to partition the Gaussian space in m parts, with the least-weighted boundary. It was recently proved by [START_REF] Milman | The gaussian doublebubble and multi-bubble conjectures[END_REF] who showed that the best way to split a Gaussian space R d in m clusters of equal measure, with 2 m d + 1, is by using 'simplicial clusters' obtained as the Voronoi cells of m equidistant points in R d . Convex geometry theory tells us that each cell is a convex cone, whose borders are hyperplanes going through the origin of R d . We note A any partition corresponding to this optimal configuration, see Figure 1 for m = 3.

In the following theorem, we apply this result to the understanding of GANs. We make the connection between optimal generators (when m d + 1) in levels of precision and the partition A derived in [START_REF] Milman | The gaussian doublebubble and multi-bubble conjectures[END_REF]. Theorem 3.7 (Optimality of generators with simplicial cluster latent space.). Assume that Assumption 3.1 is satisfied and m d + 1. For any δ > 0, there exists C large enough (independent of δ) and L D √ m π log(Cm), and a well-balanced generator G ∈ G L associated with A such that for any other well-balanced generator G ∈ G L , we have:

α G α G -δ (6) Moreover, if m d, noting ε max = max i,j X i -X j /L: α G 1 -ε max m log(Cm), (7)
Theorem 3.7 shows that when L is large enough, the bound in ( 4) is almost tight, and thus that the given generator based on the simplicial partition A is almost optimal. However, it is not clear whether those are the only generators with optimal precision. The proof is delayed in Appendix A.

What if Assumption 3.1 is not verified? This assumption is needed for the definition of a well-balanced generator associated with A as in Theorem 3.7. As shown in Figure 2, the latent space configuration obtained by the GANs for 3 almost equidistant points (1st row) and 3 almost aligned data points (2nd row). We see that in the later case, the Voronoi partition of the target data points does not verify Assumption 3.1, and the optimal latent structure is not known. We observe in this specific case that it is made of two parallel hyperplanes, much different from A defined by Milman and Neeman (2022) (1st row).

Figure 2. Illustration of the impact of the geometry of data modes on the latent space of GANs. The left column shows the modes (X1, X2, X3) from the target distribution and the generated points (small blue dots). In the middle, we plot the Voronoi diagram generated from (X1, X2, X3). On the right column, we show the boundaries in the GANs latent space with heatmaps of the norm of the gradient of the generator. In the first row, when the data satisfies Assumption 3.1, GANs achieve the optimal configuration. However, when the data modes do not satisfy this assumption, as seen in the second row, this is no longer the case.

What if the dimension m > d + 1? The position of the different spheres could be such that Assumption 3.1 is no longer valid. Second, since the result from [START_REF] Milman | The gaussian doublebubble and multi-bubble conjectures[END_REF] does not hold, the optimal partition of the Gaussian space in m equal cells is unknown. In this generalized context, GANs could hint at the optimal partition geometry. Figure 3 stresses examples when training GANs from R 2 to R m with m equidistant modes. This gives some insights on how to divide the Gaussian space into m equitable areas with least Gaussian-weighted perimeter. What if the modes do not have equal measure? The fact that each mode has equal measure in the target distribution might not be verified for unbalanced datasets. First, the optimality of simplicial clusters holds because the multi-bubble theorem is still valid. However, the lower-bound (Equation 7) does not hold. Additionally, the upper-bound from Corollary 3.6 can be relaxed. Consider w 1 , . . . , w m ∈ R m the weights of the different modes, and w min = min i w i , the upper-bound becomes:

α G 1 -mε min w min log(1/w min ) e -3/2 .
We observe that this upper-bound might not be tight anymore since it depends on the minimum of the weights w min .

Improving generative models

Our proposed theoretical analysis offers valuable insights into the optimal structure of the latent space for pushforward generative models. We demonstrate that by leveraging this structure, it is possible to design GANs with improved performance. To achieve this, we enforce a simplicial cluster structure in the latent space of GANs during training using a novel rejection sampling procedure called simplicial cluster truncation that can be combined with a mutual-information loss. Note that modifying the latent space distribution of other generative models, such as VAEs or score-based models, is a more complex task.

Simplicial cluster truncation. Let us denote a simplicial cluster [START_REF] Milman | The gaussian doublebubble and multi-bubble conjectures[END_REF] as Mutual-information loss. The rejection sampling procedure might not be sufficient for the generator to properly use the different clusters of its latent space. To encourage the simplicial cluster structure, we also optimize the mutual information between generated samples and the corresponding cluster [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF]. The loss is applied at the beginning of the training and is then dropped.

(u 1 , . . . , u m ) | u i ∈ R d .

Experiments

In the following experiments1 , we validate our theoretical analysis and derive insights for GANs trained on toy and image datasets. We also run a small experiment on VAEs. We verify: 1) that the latent space geometry of GANs has similar properties than simplicial clusters; 2) that increasing the latent space dimension (d + 1 > m) can help improve GANs, as highlighted in the theoretical section; 3) that GANs' performance is correlated with their latent space geometry; 4) that the proposed simplicial cluster truncation method is effective and boost GANs' performance.

In the following experiments, we train WGANs with gradient penalty [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF]Gulrajani et al., 2017).

For mixture of Gaussians, generator and discriminator are MLP networks. For MNIST, the generator and discriminator are standard convolutional architectures. On CIFAR-10, CIFAR-100 , and STL-10, we use either a Resnetbased [START_REF] He | Deep residual learning for image recognition[END_REF] convolutional architecture with selfmodulation in the generator (Chen et al., 2018b), either the transformer-based architecture from [START_REF] Jiang | Transgan: Two pure transformers can make one strong gan, and that can scale up[END_REF].

To evaluate the performance of GANs, we use both the precision (Kynkäänniemi et al., 2019), the FID [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF], and the density/coverage [START_REF] Naeem | we use 10k real images and 10k fake images with nearest-k= 5[END_REF]. We use a dataset-specific classifier to extract image features on MNIST, and InceptionNet pre-trained on ImageNet for CIFAR-10, CIFAR-100 and STL-10. Implementation details are given in Appendix B and code is provided in Supplementary Material.

Linear separability and convexity

According to [START_REF] Milman | The gaussian doublebubble and multi-bubble conjectures[END_REF], the optimal configuration in the latent space is obtained as the Voronoi cells of m equidistant points in R d , if m ≤ d+1. This means that if GANs reach this optimal configuration, each of the cells must be convex polytopes and have the following properties: 1) the boundaries of a cell are flat; 2) each cell is convex. To investigate this, we use a labeled dataset and assess whether a simple linear model (e.g., multinomial logistic regression) can map latents to labels. If the cells in the latent space are bounded by hyperplanes, then, using the hyperplane separation theorem, the linear model is expected to be a good predictor of a generated sample's label.

We use a standard multi-class labeled dataset. G θ is a pretrained generator and C φ is a pre-trained classifier considered as an oracle. Using G θ and C φ , we construct a dataset of latent vectors z ∈ R d and their associated labels y = C φ (G θ (z)). On CIFAR-10/100, similarly to [START_REF] Razavi | Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems[END_REF], only data points with a confidence threshold of 0.7 or higher are accepted. 100k training points and 10k test points. We use multinomial logistic regression to learn the mapping from latent vectors z to their labels y. We can see in Table 1, that the LogReg Accuracy reaches high levels: 90% on MNIST and 70% on CIFAR-10. For the Convex accuracy, we draw two random latent vectors z 0 and z 1 that belong to the same class, and check whether linear interpolations in the latent space also belong to the same class, that is

C φ (G θ (z 0 )) = C φ (G θ (z 0 )) = λ × C φ (G θ (z 0 )) + (1 -λ) × C φ (G θ (z 1 )) for λ ∈ [0, 1]
. Interestingly, we see in Table 1 a correlation between the Logreg and Convex accuracy and the precision metric: the more the latent space behaves like a simplicial cluster, the higher the precision. For a qualitative evaluation, we show this phenomenon in Figure 4 and stress that linear interpolations conserve the image class.

In Table 2, we demonstrate that VAEs also tend to structure their latent space in linear regions. In this experiment, we use a slightly different setting: instead of drawing random latent vectors, we directly use the dataset samples and encode them with the VAE's encoder to test LogReg and Convex accuracies.

Impact of the latent space dimension

To evaluate the impact of the latent space dimension, we train GANs with latent space dimension ranging from 2 to 128 on several datasets. In Figure 5, we exhibit two phases in the performance of GANs when changing the number of latent dimensions. For a fixed architecture, and a given dataset, we observe the existence of an optimal latent space dimension d . When d < d the precision or density of the model falls significantly. Interestingly, when d > d , the precision becomes constant: overparameterizing the model does not bring a significant improvement. As expected, we observe in Figure 5 that the maximum precision/density depends on the complexity of the dataset and its number of modes: the more complex the dataset, the lower the precision. This is also coherent with our theoretical results from both Corollary 3.6 and Theorem 3.7. An interesting problem was also brought to the fore by Roth et al. (2017). When training GANs two different issues can arise: 1) dimensional misspecification where the true and modeled distributions do not have density functions w.r.t. the same base measure, and 2) density misspecification, where GANs try to fit a disconnected manifold with a unimodal distribution. To isolate the density misspecification studied in this paper, we train a conditional GAN with a low-dimensional latent space R d (e.g. R 5 in our setting), so that the dimension of the generated manifold is at most 5. We later collect a dataset of synthetic generated samples Synthetic CIFAR-10, and train unconditional GANs with varying latent space dimensions. Figure 5 shows that GANs converge to the same limits for Precision and Density on Synthetic CIFAR-10 and CIFAR-10. This shows that the performance is more impacted by the density misspecification (trying to fit a disconnected target distribution with a connected one) rather than the dimensional misspecification.

The latent geometry and GANs' performance

We investigate the relationship between the performance of GANs and their latent space geometry. To do so, we train many generators with different capacities (increasing width), and study how it impacts both the latent geometry and the performance. The results in Figure 6 reveal a strong positive correlation between the performance of GANs and the linearity/convexity of the latent space: the better the GANs perform, the more linearly separable and convex the latent space is. Indeed, the Pearson correlation between Precision and LogReg Accuracy is 0.98 on CIFAR-10, and 0.94 on CIFAR-100. Interestingly, overparametrization was known to help push-forward generative models in their optimization procedure [START_REF] Balaji | Understanding overparameterization in generative adversarial networks[END_REF] and in increasing their Lipschitz constant [START_REF] Salmona | Can push-forward generative models fit multimodal distributions?[END_REF]. We demonstrate here that it can help GANs in reaching an optimal latent space structure, resulting in improved performance.

Impact of the simplicial truncation method

Finally, we aim to improve GANs performance by using our theoretical results (Theorem 3.7). This is done by truncating the latent Gaussian distribution, as discussed in Section 3.3, so that the generator structures its latent space with a simplicial cluster geometry. Note that the rejection threshold used at inference time can be higher than the one used at training time, since we have observed that higher rejection thresholds can help us increase both the precision and density of the models. The results in Table 3 demonstrate that the use of this truncation method can improve the density and precision of GANs, without lowering the coverage nor the FID. This simplicial-based truncation has thus proved to be effective at removing off-manifold samples and can help improve push-forward generative models. Table 3. Improving GANs with simplicial cluster latent space. JBT stands for the Jacobian-based truncation [START_REF] Tanielian | Learning disconnected manifolds: a no gan's land[END_REF]; DeliG. for latent space with mixture of Gaussians [START_REF] Gurumurthy | Deligan: Generative adversarial networks for diverse and limited data[END_REF]; simp. for our proposed truncation method with simplicial cluster. These results demonstrate that generators with a simplicial cluster latent space consistently outperform the baseline generator in Precision/Density, and most of the times outperforms other boosting methods (DeliGAN and JBT).

Conclusion

In conclusion, this paper takes a step towards a better understanding of push-forward generative models. When the latent space dimension is large enough, we prove the exis-tence of an optimal latent space geometry, called 'simplicial clusters'. Through experiments, we demonstrate that generative models with sufficient capacity tend to conform to this optimal geometry and also that enforcing this latent structure can improve GANs' performance. Our analysis has potential to drive further research on generative models with both theoretical and practical implications, such as developing new models that favor the emergence of such clusters in both latent and feature spaces. Similarly to what has been done in classification [START_REF] Vardan Papyan | Prevalence of neural collapse during the terminal phase of deep learning training[END_REF], studying thorougly the feature space of deep generative models is also an open question.

Limitations. While our theoretical analysis demonstrates the existence of optimal generators, we were unable to prove their uniqueness. This limitation is associated with identifying partitions with the lowest ε-boundary measures in the Gaussian space, which is a challenging and unresolved problem in geometric measure theory.

Potential negative societal impacts. This work may increase potential negative impacts of deep generative models, such as deepfakes [START_REF] Fallis | The epistemic threat of deepfakes[END_REF].

A. Technical results: proofs

A.1. Proof of Lemma 3.5

We want to show that generator G ∈ G A L is such that α G 1 -γ(∂ εmin A), where

∂ εmin A = m i=1 ∪ j =i A εmin \ ∪ j =i A j .
Proof by contradiction.

Assume a generator G such that there exists z ∈ ∂ εmin A and i ∈ [1, m] such that G(z) ∈ M i . Since G is associated with A, we have using Definition 3.3, that there exists z and j ∈ [1, m], j = i such that z -z < ε min /2 and j = arg min

k∈[1,m]
G(z ) -M k . Thus, we have:

G(z) -G(z ) d(G(z ), M i ), d(M i , M i )/2, D min /2.
And,

G(z) -G(z ) z -z > D min /ε min , > L.
This contradicts G being in G A L .

A.2. Proof of Corollary 3.6.

Let L, D be such that L D log(m). Let's prove that for any well-balanced generator G ∈ G L , we have:

α G 1 -ε min log m e -3/2 .
Using the method from Schechtman (2012), we have the measure of the border of cell i:

γ ∪ j =i A j ε \ ∪ j =i A j 1 √ 2π t+ε t e -s 2 /2 ds, where t is such that 1 √ 2π ∞ t e -s 2 /2 ds = 1/m, ε √ 2π e -(t+ε) 2 /2 , ε √ log m m e -εt-ε 2 /2 (using log m ≤ t ≤ 2 log m), ε √ log m m e -ε √ log m-ε 2 /2 .
Thus:

γ(∂ εmin A) = m i=1 γ ∪ j =i A j ε \ ∪ j =i A j ε min log m e -εmin √ log m-ε 2 min /2 .
Thus, we have

α G 1 -γ(∂ εmin A), 1 -ε min log m e -εmin √ log m-ε 2 min /2 .
Moreover, using ε min = D L and L D √ log m, so we get ε min √ log m 1:

α G 1 -ε min log m e -3/2 .
Proof of Corollary 3.6 with w i (non-equal measure of modes). Let L, D be such that L D log(m). Let's prove that for any well-balanced generator G ∈ G L , we have:

α G 1 -mw min ε min log min e -3/2 .
Using the method from Schechtman (2012), we have the measure of the border of cell i:

γ ∪ j =i A j ε \ ∪ j =i A j 1 √ 2π t+ε t e -s 2 /2 ds, where t is such that 1 √ 2π ∞ t e -s 2 /2 ds = w min , ε √ 2π e -(t+ε) 2 /2 ,
w min ε log 1/w min e -εt-ε 2 /2 (using log 1/w min ≤ t ≤ 2 log 1/w min ),

w min ε log 1/w min e -ε √ log 1/wmin-ε 2 /2 .
Thus:

γ(∂ εmin A) = m i=1 γ ∪ j =i A j ε \ ∪ j =i A j mw min ε min log 1/w min e -εmin √ log 1/wmin-ε 2 min /2 .
Thus, we have

α G 1 -γ(∂ εmin A), 1 -mw min ε min log 1/w min e -εmin √ log 1/wmin-ε 2 min /2 .
Moreover, using ε min = D L and L D log 1/w min , so we get ε min log 1/w min 1:

α G 1 -mε min w min log 1/w min e -3/2 .

A.3. Proof of Theorem 3.7

Let µ be the target distribution. We know that µ lays on m disconnected components contained in spheres S i , i ∈ [1, m]. We note M i , i ∈ [1, m] the centers, and r i the radius of each sphere. We also assume that the spheres verify Assumption 3.1.

For each pair (i, j) ∈ [1, m] 2 , we define X ij ∈ S i and X ji ∈ S j the points verifying We consider the optimal partition A in the Gaussian latent space. For each given latent point z ∈ R d , we define:

X ij =
N z = {j ∈ [1, m], z ∈ A ε j }.
We then distinguish two different cases:

1. |N z | = 1: the point z belongs to the interior of a single cell, z ∈ A -ε i .

2. |N z | 2: the point z is in the neighborhood of at least two different cells.

Interestingly, a point can only belong at most to the interior of one cell, but it can be at the intersection of several boundaries. We are now ready to define the optimal generator.

First, we set

G(z) = X i,j for all z ∈ {z ∈ R d , |N z | = 2, z ∈ A -ε i ∩ A ε i ∩ A ε j where N z = {i, j}}. A 2 A 3 A 1 ∂ ε A 3 Latent space Output space M 2 M 3 M 1 Figure 7.
An optimal generator maps a 2D latent space to a 2D output space with three modes (M1, M2, M3). The latent space has an optimal 'simplicial cluster' geometry. In the latent space, all the -boundaries intersect each other in the gray circle, which is mapped in the output space in the convex hull of the three modes.

Second, we define the generator in the interior of the cells, i.e. N z = {i}. For each z ∈ A -ε i and for a given unit vector u ∈ R d , we assume that the generator is constant along the parametric line z = k × u, k ∈ R.

Finally, we define the generator when z does not belong to the interior of any cell, i.e. |N z | 2:

G ε (z) = i∈[1,m] j =i w i,j (z) X i,j 1 j∈Nz 1 i∈Nz where w i,j (z) = d(z, (A ε i ) ) i∈[1,m] j =i d(z, (A ε j ) ) 1 j∈Nz 1 i∈Nz (8)
where d(z, A) = min a∈A z -a . An illustration of the optimal generator is given in Figure 7. When z belongs to the intersection of two ε-boundaries, G ε (z) is a simple linear combination of 2 points. It is only when |N z | 3 that more complex samples are generated. A simple illustration of G ε for d = 2 and m = 3 is given in Figure 7. Interestingly, one can also show that the image of G ε is equal to the convex hull of the Diracs X i , i ∈ [1, m]. In particular, there exists a particularly interesting neighborhood ν of 0 where G ε (ν) is equal to the whole convex hull of the points

X i , i ∈ [1, m].
Proof that G ε is well-balanced. We recall that a generator is well-balanced if we have G γ(M 1 ) = . . . = G γ(M m ).

By construction (8), we have that for any

i ∈ [1, m] G ε (z) -X i = k =i w k (X k -X i ) , = D × (1 -w i ).
So, for any z ∈ A i , we have that

i = arg min j∈[1,m] wj = arg min j∈[1,m] G(z)-Xj
Thus G ε is associated with the optimal partition A , .

Besides, for a given radius r of the different modes, since everything is symmetrical, we have that γ

({z ∈ R d , G(z)-X 1 r} = . . . = γ({z ∈ R d , G(z) -X m r}.
Thus, the generator is well-balanced.

Showing that G ε is in G L . It is clear that when |N z | = 1, we have that G ε (z) is a L-Lipshitz continuous function. Now, assume that |N z | 2.
Consider z, z such that N z = N z . Let α = (α 1 , . . . , α m ) and β = (β 1 , . . . , β m ) be two vectors, both in R m , such that for all i ∈ [1, m]:

α i = d(z, (A ε i ) ) j∈Az d(z, (A ε j ) )
and

β i = d(z , (A ε i ) ) j∈Az d(z , (A ε j ) )
We have that

G(z) -G(z ) = (1 - i =1 α i )X 1 -(1 - i =1 β i )X 1 + i =1 α i X i - i =1 β i X i = i =1 (α i -β i )(X 1 -X i ) max (i,j)∈[1,m] 2 X i -X j α -β , max (i,j)∈[1,m] 2 X i -X j h(z) -h(z ) ,
where h is the function from R d → R m defined as:

h(z) = ( d(z, (A ε 1 ) ) i∈Az d(z, (A ε i ) ) , . . . , d(z, (A ε m ) ) i∈Az d(z, (A ε i ) )
).

We can write h

= f • g with f the function defined from R d → R m by f (z) = d(z, (A ε 1 ) ), . . . , d(z, (A ε k ) ) ,
and g the function defined on R m \ {0} by g

(z) = z z 1 We have that f is a √ m-Lipschitz functions (given that z → d(z, (A ε m ) ) is 1-Lipschitz).
Besides, we know that outside the ball B ε/2 (0), the function g is (2/ε)-Lipschitz. Since it is clear that for every point z such that |N z | 2, we have that |f (z)| ε/2. Finally, the function h is 2 √ m ε -Lipschitz. Thus, we have that:

G ε (z) -G ε (z ) 2D √ m ε z -z , with D = max i,j X i -X j , (i, j) ∈ [1, m] 2 , i = j.
Now, by noting ε max = D L , and considering ε = 2 √ m ε max , we have:

G ε (z) -G ε (z ) L z -z .
Now, consider two latent vectors z, z in the same cell A -ε i . There exists i ∈ [1, m], and a pairs (j, j ) ∈ [1, m] 2 (note that j could be equal to j ) such that G(z) = X i,j and G(z ) = X i,j . Using a similar reasoning as before, we can show that:

G ε (z) -G ε (z ) L z -z , with D = 2 max i∈[1,m] r i .
We can now conclude on the Lipschitzness of G on R d .

Proving that: for m d + 1, for any δ > 0, if L is large enough, then, for any well-balanced G ∈ G L , we have α G εmax α G -δ. Let G be a well-balanced generator and A the partition associated with G. Let us first define the gaussian boundary measure P γ of a partition A of R d . For partitions with smooth boundaries, it coincides with the (d -1)-dimensional gaussian measure of the boundary, defined as follows:

P γ (A) = lim inf ε→0 γ(∂ ε A) -γ(A)

2/πε

Moreover, for sets with smooth boundaries, we have from Federer (1969, Theorem 3.2.29):

lim inf ε→0 γ(∂ ε A) -γ(A) 2/πε = lim ε→0 γ(∂ ε A) -γ(A)

2/πε

Let us denote A , the optimal partition defined in [START_REF] Milman | The gaussian doublebubble and multi-bubble conjectures[END_REF], based on simplicial clusters. A is a standard partition where γ(A 1 ) = . . . = γ(A m ) for all i, and i γ(A i ) = 1. By the multi-bubble theorem [START_REF] Milman | The gaussian doublebubble and multi-bubble conjectures[END_REF], simplicial clusters (such as A ) are the unique minimizers of the gaussian isoperimetric problem, thus:

P γ (A ) P γ (A) lim ε→0 γ(∂ ε A ) ε lim ε→0 γ(∂ ε A) ε L A L A where L A = lim ε→0 γ(∂ ε A ) ε and L A = lim ε→0 γ(∂ ε A ) ε .
Then, for any δ > 0, there exists ε > 0 such that, for any ε < ε ,

| γ(∂ ε A ) ε -L A | < δ , | γ(∂ ε A) ε -L A | < δ and L A L A
Thus, for any δ > 0, there exists ε > 0 such that, for any ε < ε ,

γ(∂ ε A ) γ(∂ ε A) + 2δε (9)
Besides, we know that α G 1 -γ(∂ εmin A)

Consequently, we have that:

α G 1 -γ(∂ εmin A)
1 -γ(∂ εmin A ) + 2δε min using (9). Now, by construction of G εmax , we have that

α G εmax 1 -γ(∂ εmax A ).
Consequently,

α G 1 -γ(∂ εmin A ) + 2δε max + γ(∂ εmax A ) -γ(∂ εmax A ) α G ε + 2δε max + γ(∂ εmax A ) -γ(∂ εmin A ) α G ε + 2δε max + γ(∂ εmax A ) -2L A ε max -γ(∂ εmin A ) + 2L A ε min + 2L A (ε max -ε min ) α G ε + 4δε max + 2L A ε max , α G ε + ε max (4δ + 2L A ).
We conclude by choosing L big enough such that ε max is strictly smaller than

δ 4δ+2L A .
Proving the lower-bound 7 of Theorem 3.7. Let's consider G ε defined using (8) and ε = 2

√ mε max . The precision of G ε is thus such that: α G ε 1 -γ(∂ ε A). However, since ∂ ε A ⊂ n i=1 A ε i , we have that for any ε: γ(∂ ε A) n i=1 γ(A ε i ).
Using results from Schechtman (2012, Proposition 1), when m ≤ d, there exists C large enough, such that GPUs. For all datasets, the training of GANs was run on NVIDIA Tesla V100 GPUs (16 GB). The training of ResNet GANs for 100k steps on CIFAR takes around 30 hours. For TransGAN models, the training is done for 250k steps on two NVIDIA Tesla V100 GPUs, which takes around 35 × 2 = 70 GPU hours.

γ(A ε i ) ε m π log(Cm) .
× 4 × 128 - ResBlock [3 × 3] × 2 1 × 1 4 × 4 × 128 Y ReLU Nearest Up Sample 8 × 8 × 128 - ResBlock [3 × 3] × 2 1 × 1 8 × 8 × 128 Y ReLU Nearest Up Sample 16 × 16 × 128 - ResBlock [3 × 3] × 2 1 × 1 16 × 16 × 128 Y ReLU Nearest Up Sample 32 × 32 × 128 - Convolution 3 × 3 1 × 1 32 × 32 × 3 - Tanh Discriminator D(x) 32 × 32 × 3 ResBlock [3 × 3] × 2 1 × 1 32 × 32 × 256 - ReLU AvgPool 2 × 2 1 × 1 16 × 16 × 256 - ResBlock [3 × 3] × 2
B.3. Details on simplicial truncation method (Details for Section 4.4)

We provide here more details about our truncation method. First, the rejection sampling in the latent space R d of GANs procedure is the following:

• Define hyper-parameters threshold τ , number of clusters N , latent space dimension d.

• Initialize N equidistant points in {(u 0 , . . . , u N ) | u i ∈ R d }. This can be done easily when N ≤ d.

• When sampling latent vectors z ∈ R d , compute a softmax over the negative distances between z and u i : p i (z) = e -d(z,u i ) j e -d(z,u j ) . Then, z is selected if max i p i (z) > τ .

Second, we add a classification loss to encourage the generator to use this latent structure. This loss is motivated by the need to maximize mutual information between the latent cluster and the modes of the generator [START_REF] Khayatkhoei | Disconnected manifold learning for generative adversarial networks[END_REF], and can be written as:

L c = -E z∼γ [ln q φ i(z)|G θ (z) ]
where q φ is parametrized by a second classification head added to the discriminator; i(z) = arg max During inference, if the generator has properly learned to use the different clusters of the latent space, we observe that augmenting the threshold τ leads to an increased density and precision.

We present full results in Table 7 and Figure 8. To illustrate our theoretical results, we propose to vary the number of modes of the data distribution. On real-world data, the number of modes is set but usually unknown, and removing/adding classes as a proxy for modes usually does not give insightful results since some classes can be much more complex than others. We thus use a synthetic setting, where we can easily control both the number of modes and their complexity. Figure 9 stresses that as the number of modes increase, the precision decrease. Interestingly, using large latent space dimension can relieve the problem, even if the latent space dimension is clearly below that of the target. Recall the two problems that arise when training GANs: i) dimensional misspecification where the true and modeled distributions do not have density functions w.r.t. the same base measure, and ii) density misspecification, where GANs try to fit a disconnected manifold with a unimodal disitribution. From the results we conclude that:

• With very low latent space dimensions, both problems i) and ii) have to be addressed and this leads to poor precision as the number of modes increases.

• With larger latent space dimensions, the problem i) is less of a burden even when there is a clear dimensional misspecification and thus the GANs' performance is more tied to problem ii).

Figure 9. Training on a mixture of Gausians in R 100 with varying number of modes and varying latent space dimension. The bigger the number of modes, the lower the precision. Increasing the latent space dimension helps up to a limit depending on the number of modes.

Figure 3 .

 3 Figure 3. Extension of the multi-bubble conjecture when m > d + 1. We depict the partition of the R 2 latent space of a GAN that maps to m equidistant points in R m , with m = 4, 6, 12. Each colored cell maps to a distinct data point in R m .

  ) and convexity(Convex Acc.) in GAN latent spaces. The results align with the predictions of Corollary 3.6, where a linearly separable and convex structure of the latent space indicates a high precision. The architecture Transformer refers to the TransGAN model from[START_REF] Jiang | Transgan: Two pure transformers can make one strong gan, and that can scale up[END_REF]. The supervised classifiers used as oracles haves test-accuracies of 80.2% on CIFAR-10 and 61.8% on CIFAR-

Figure 4 .

 4 Figure 4. Visualization of the convexity of classes in the latent space of GANs trained on CIFAR-10. The plot shows that latent linear interpolations within a class preserve the class label.

Figure 5 .

 5 Figure 5. Performance of GANs with regard to the number of modes and latent space dimensions. As the number of modes and latent space dimension increases, we observe an improvement in Precision (left) and Density (right), with a saturation point beyond a certain threshold.

Figure 6 .

 6 Figure 6. Study of the correlation between GANs' performance and their latent space geometry. This is done by increasing the width of the generator (w ∈ {32, 64, 128, 256, 512}) in a fixed training setting on the CIFAR-10 and CIFAR-100 datasets. The results reveal a positive correlation between GANs' performance (measured by Precision and Density) and the linear separability and convexity of their latent space (measured by LogReg and Convex Accuracy). Confidence intervals are computed on 10 checkpoints of a training.

  arg min x∈Si d(x, S j ) and X ji = arg min x∈Sj d(x, S i ).

  i p i (z) is the index of the latent cluster of the sample. This loss is added during training, at each step of generator's and discriminator's training, during the first 20 epochs. It is then dropped, since we noticed that it harms the GANs performance if it is kept until the end of the training. Training hyper-parameters: for N = 10, we use a latent dimension of d = 64 and training threshold of τ = 0.135; for N = 100, we use d = 128 and τ = 0.08.

Figure 8 .

 8 Figure 8. Density/Coverage curves comparing TransGAN and boosting methods for multi-modal datasets and different threshold ratios. Our simplicial truncation method (TransGAN + simp.) consistently outperforms the TransGAN and TransGAN + DeliGAN baselines.

  As a result, the boundaries of the simplicial cluster, which are points with high distances to the centers of Voronoi cells, are rejected. The threshold parameter τ determines the ε value. With this method, the boundaries between different modes are never sampled, leading to a disconnected latent space. This approach can improve the learning of disconnected manifolds by injecting disconnectedness into the modeled generative distribution. Additionally, the use of a geometrical structure that is particularly well suited to separate several modes[START_REF] Vardan Papyan | Prevalence of neural collapse during the terminal phase of deep learning training[END_REF] enhances the performance.

The rejection sampling procedure, based on Theorem 3.7, involves sampling a latent vector z from γ and accepting it if max i∈[1,...,m] (z • u i ) > τ , where both τ and m are considered as hyper-parameters. This defines a new latent space distribution where the density is high near the unit vectors u i , i ∈ [1, m].

Table 1 .

 1 Validation of linear separability (LogReg Acc.

	This dataset is later split into

Table 2 .

 2 Validation

of linear separability (LogReg Acc.) and convexity (Convex Acc.) in VAE latent spaces. The supervised classifiers used as oracles haves test-accuracies of 80.2% on CIFAR-10.

Table 5 .

 5 GANs training details on CIFAR datasets. BN stands for batch-normalization.

	Conditional

Our code is open-source and can be found there: https://github.com/thibautissenhuth/unveiling latent geometry.

Acknowledgements

We would like to thank Jean-Yves Franceschi for helpful discussions and a thorough review of the paper.

Thus, we have

To have α G εmax 0, we must have 1/ π log(Cm). This is the case since we have

where D = max i,j X i -X j .

B. Experiments

B.1. Implementation details First, let us note that we share our code in Supplementary Material for reproducibility.

Training. We use the Wasserstein loss with gradient-penalty on interpolations of fake and real data. At each iteration, the discriminator is trained 2 steps and the generator 1 step with Adam optimizer. The batch size is 256. The learning rate of the discriminator is two times larger [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF], i.e. 5 × 10 -5 for the generator and 1 × 10 -4 for the discriminator. GANs are trained for 80k steps on MNIST and for 100k steps on CIFAR datasets. Architectures of generator and discriminator are described in Table 4 andTable 5.

B.2. Correlation between latent space geometry and GANs' performance (Details for Section 4.3)

We present the full results of this study in Table 6. TransGAN + simp. N=100, (0.08,0.08) 10.1 ± 0.1 76.5 ± 0.9 60.2 ± 0.8 90.0 ± 1.7 83.0 ± 0.5 (0.08,0.15) 10.0 ± 0.1 76.9 ± 0.8 59.9 ± 0.6 91.4 ± 1.1 83.8 ± 0.3 (0.08,0.20) 10.0 ± 0.1 77.8 ± 0.6 59.8 ± 0.8 94.1 ± 0.9 83.5 ± 0.8