Clément Chadebec
email: clement.chadebec@inria.fr

Louis J Vincent
email: louis.vincent@inria.fr

Stéphanie Allassonnière
email: stephanie.allassonniere@inria.fr

Pythae: Unifying Generative Autoencoders in Python A Benchmarking Use Case

In recent years, deep generative models have attracted increasing interest due to their capacity to model complex distributions. Among those models, variational autoencoders have gained popularity as they have proven both to be computationally efficient and yield impressive results in multiple fields. Following this breakthrough, extensive research has been done in order to improve the original publication, resulting in a variety of different VAE models in response to different tasks. In this paper we present Pythae, a versatile open-source Python library providing both a unified implementation and a dedicated framework allowing straightforward, reproducible and reliable use of generative autoencoder models. We then propose to use this library to perform a case study benchmark where we present and compare 19 generative autoencoder models representative of some of the main improvements on downstream tasks such as image reconstruction, generation, classification, clustering and interpolation. The open-source library can be found at https://github.com/clementchadebec/benchmark_VAE.

Introduction

Over the past few years, generative models have proven to be a promising approach for modelling datasets with complex inherent distributions such a natural images. Among those, Variational AutoEncoders (VAE) [START_REF] Diederik | Auto-encoding variational bayes[END_REF][START_REF] Jimenez Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] have gained popularity due to their computational efficiency and scalability, leading to many applications such as speech modelling [START_REF] Blaauw | Modeling and transforming speech using variational autoencoders[END_REF], clustering [START_REF] Dilokthanakul | Deep unsupervised clustering with gaussian mixture variational autoencoders[END_REF][START_REF] Yang | Deep clustering by gaussian mixture variational autoencoders with graph embedding[END_REF], data augmentation [START_REF] Chadebec | Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder[END_REF] or image generation [START_REF] Razavi | Generating diverse high-fidelity images with vq-vae-2[END_REF]. Similarly to autoencoders, these models encourage good reconstruction of an observed input data from a latent representation, but they further assume latent vectors to be random variables involved in the generation process of the observed data. This imposes a latent structure wherein latent variables are driven to follow a prior distribution that can then be used to generate new data. Since this breakthrough, various contributions have been made to enrich the original VAE scheme through new generating strategies [START_REF] Dilokthanakul | Deep unsupervised clustering with gaussian mixture variational autoencoders[END_REF][START_REF] Tomczak | Vae with a vampprior[END_REF][START_REF] Ghosh | From variational to deterministic autoencoders[END_REF][START_REF] Bauer | Resampled priors for variational autoencoders[END_REF][START_REF] Pang | Learning latent space energy-based prior model[END_REF], reconstruction objectives [START_REF] Boesen | Autoencoding beyond pixels using a learned similarity metric[END_REF][START_REF] Snell | Learning to generate images with perceptual similarity metrics[END_REF] and more adapted latent representations [START_REF] Higgins | beta-VAE: Learning basic visual concepts with a constrained variational framework[END_REF][START_REF] Kim | Disentangling by factorising[END_REF][START_REF] Van Den | Neural discrete representation learning[END_REF][START_REF] Arvanitidis | Latent space oddity: On the curvature of deep generative models[END_REF][START_REF] Chadebec | Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder[END_REF] to cite a few. A drawback of VAEs is that due to the intractability of the log-likelihood objective function, VAEs have to resort to optimizing a lower bound on the true objective as a proxy, which has proven to be a major limitation of the model [START_REF] Burda | Importance weighted autoencoders[END_REF][START_REF] Alexander A Alemi | Deep variational information bottleneck[END_REF][START_REF] Higgins | beta-VAE: Learning basic visual concepts with a constrained variational framework[END_REF][START_REF] Cremer | Inference suboptimality in variational autoencoders[END_REF][START_REF] Zhang | Advances in variational inference[END_REF]. Hence, extensive research has been proposed to improve this bound through richer distributions [START_REF] Salimans | Markov chain monte carlo and variational inference: Bridging the gap[END_REF][START_REF] Rezende | Variational inference with normalizing flows[END_REF][START_REF] Durk | Improved variational inference with inverse autoregressive flow[END_REF][START_REF] Anthony L Caterini | Hamiltonian variational auto-encoder[END_REF]. More recently, it has been shown that autoencoders can be turned into generative approaches through latent density estimation [START_REF] Ghosh | From variational to deterministic autoencoders[END_REF], extending the concept of Generative AutoEncoders (GAE) to a more general class of autoencoder models.

Nonetheless, most of this research has been done in parallel across disjoint sub-fields of research and to the best of our knowledge little to no work has been done on homogenising and integrating these distinct methods in a common framework. Moreover, for many of the aforementioned publications, implementations may not be available or maintained, therefore requiring time-consuming re-implementation. This induces a strong bottleneck for research to move forward in this field and makes reproducibility challenging, which calls for the need of a unified generative autoencoder framework. To address this issue we introduce Pythae (Python AutoEncoder), a versatile open source Python library for generative autoencoders providing unified implementations of common methods, along with a reproducible framework allowing for easy model training, data generation and experiment tracking. We then propose to illustrate the usefulness of the proposed library on a benchmark case study of 19 generative autoencoder methods on classical image datasets. We consider five different downstream tasks: image reconstruction and generation, latent vector classification and clustering, and image interpolation on three well known imaging datasets.

Variational autoencoders

In this section, we recall the original VAE setting and present some of the main improvements that were proposed to enhance the model.

Background

Given x ∈ R D , a set of observed variables deriving from an unknown distribution p(x), a VAE assumes that there exists z ∈ R d such that z is a latent representation of x. The generation process of x thus decomposes as

p θ (x) = Z p θ (x|z)p z (z)dz , (1)
where p z is the prior distribution on the latent space R d . The distribution p θ (x|z) is referred to as the decoder and is modelled with a simple parametric distribution whose parameters are given by a neural network. Since the true posterior p θ (z|x) is most of the time intractable due to the integral in Eq. ( 1) recourse to Variational Inference [START_REF] Michael I Jordan | An introduction to variational methods for graphical models[END_REF] is needed and a variational distribution q ϕ (z|x) which we refer to as the decoder is introduced. The approximate posterior q ϕ is again taken as a simple parametric distribution whose parameters are also modelled by a neural network. This allows to define an unbiased estimate p θ of the marginal distribution p θ (x) using importance sampling with q ϕ (z|x) i.e. p θ (x) = p θ (x|z)pz(z) q ϕ (z|x)

and E z∼q ϕ p θ = p θ . Applying Jensen's inequality leads to a lower bound on the likelihood given in Eq. [START_REF] Alexander A Alemi | Deep variational information bottleneck[END_REF]:

log E z∼q ϕ p θ (x) p θ (x) ≥ E z∼q ϕ log p θ (x) = E z∼q ϕ [log p θ (x|z)] reconstruction -D KL q ϕ (z|x)||p z (z) regularisation , (2)
where D KL (p||q) is the Kullback-Leibler divergence between distributions p and q. This bound is referred to as the Evidence Lower Bound (ELBO) [START_REF] Diederik | Auto-encoding variational bayes[END_REF] and is used as the training objective to maximize in the traditional VAE scheme. It can be interpreted as a two terms objective [START_REF] Ghosh | From variational to deterministic autoencoders[END_REF] where the reconstruction loss forces the output of the decoder to be close to the original input x, while the regularisation loss forces the posterior distribution q ϕ (z|x) outputted by the encoder to be close to the prior distribution p z (z). Under standard VAE assumption, the prior distribution is a multivariate standard Gaussian p z = N (0, I d ), the approximate posterior is set to q ϕ (z|x) = N z µ(x), Σ(x) where µ(x), Σ(x) are outputs of the encoder network. p θ (x|z) is chosen depending on the modelling of the data (e.g. N (µ θ (z), σ • I D ), σ > 0) with µ θ given by the decoder network). Building on the breakthrough of VAEs, several papers have proposed improvements to the model.

Enhancing the model

Improving the prior It has been shown that the role of the prior distribution p z is crucial in the good performance of the VAE [START_REF] Matthew | Elbo surgery: yet another way to carve up the variational evidence lower bound[END_REF] and choosing a family of overly simplistic priors can lead to over-regularization [START_REF] Connor | Variational autoencoder with learned latent structure[END_REF] and poor reconstruction performance [START_REF] Dai | Diagnosing and Enhancing VAE Models[END_REF]. Hence, multi-modal priors [START_REF] Nalisnick | Approximate inference for deep latent gaussian mixtures[END_REF][START_REF] Dilokthanakul | Deep unsupervised clustering with gaussian mixture variational autoencoders[END_REF][START_REF] Tomczak | Vae with a vampprior[END_REF] were proposed, followed by hierarchical latent variable models [START_REF] Casper Kaae Sønderby | Ladder variational autoencoder[END_REF][START_REF] Klushyn | Learning Hierarchical Priors in VAEs[END_REF] and prior learning based approaches [START_REF] Chen | Variational lossy autoencoder[END_REF][START_REF] Aneja | NCP-VAE: Variational autoencoders with noise contrastive priors[END_REF] to address the poor expressiveness of the prior distribution and model richer generative distributions. Considering a specific geometry of the latent space also led to alternative priors taking into account geometrical aspects of the latent space [START_REF] Tim R Davidson | Hyperspherical variational auto-encoders[END_REF][START_REF] Falorsi | Explorations in homeomorphic variational auto-encoding[END_REF][START_REF] Mathieu | Continuous hierarchical representations with poincaré variational auto-encoders[END_REF][START_REF] Arvanitidis | Latent space oddity: On the curvature of deep generative models[END_REF][START_REF] Chen | Metrics for deep generative models[END_REF][START_REF] Shao | The riemannian geometry of deep generative models[END_REF][START_REF] Kalatzis | Variational autoencoders with riemannian brownian motion priors[END_REF][START_REF] Chadebec | Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder[END_REF]. Another interesting approach proposed for instance in [START_REF] Van Den | Neural discrete representation learning[END_REF][START_REF] Ghosh | From variational to deterministic autoencoders[END_REF] consists in using density estimation post training with another distribution or normalising flows [START_REF] Rezende | Variational inference with normalizing flows[END_REF] on the learnt latent codes to generate.

Towards a better lower bound Another major axis of improvement of the VAE model has been to tighten the gap between the ELBO objective and the true log probability [START_REF] Burda | Importance weighted autoencoders[END_REF][START_REF] Alexander A Alemi | Deep variational information bottleneck[END_REF][START_REF] Higgins | beta-VAE: Learning basic visual concepts with a constrained variational framework[END_REF][START_REF] Cremer | Inference suboptimality in variational autoencoders[END_REF][START_REF] Zhang | Advances in variational inference[END_REF]. The ELBO objective can indeed be rewritten as the difference between the true log probability and a KL divergence between the approximate posterior and the true posterior

L ELBO (x) = log p θ (x) -D KL q ϕ (z|x)||p(z|x) . (3)
Hence, if one wants to make the ELBO gap tighter, particular attention should be paid to the choice in the approximate posterior q ϕ (z|x). In the original model, q ϕ (z|x) is chosen as a simple distribution for tractability of the ELBO in Eq. ( 2). However, several approaches have been proposed to extend the choice of q ϕ to a wider class of distributions using MCMC sampling [START_REF] Salimans | Markov chain monte carlo and variational inference: Bridging the gap[END_REF] or normalising flows [START_REF] Rezende | Variational inference with normalizing flows[END_REF]. For instance, Kingma et al. [START_REF] Durk | Improved variational inference with inverse autoregressive flow[END_REF] improve upon the works of [START_REF] Rezende | Variational inference with normalizing flows[END_REF] with inverse auto-regressive normalising flow (IAF), a new type of normalizing flow that better scales to high-dimensional latent spaces. With this objective in mind a Hamiltonian VAE aimed at targeting the true posterior during training with a Hamiltonian Monte Carlo [START_REF] Radford | Hamiltonian importance sampling[END_REF] inspired scheme was proposed [START_REF] Anthony L Caterini | Hamiltonian variational auto-encoder[END_REF] and extended to Riemannian latent spaces in [START_REF] Chadebec | Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder[END_REF].

Encouraging disentanglement One key interest of VAEs is to learn a more representative and interpretable latent representation of the input data. In that regard, several approaches have been proposed encouraging a better disentanglement of the features in the latent space. Higgins et al. [START_REF] Higgins | beta-VAE: Learning basic visual concepts with a constrained variational framework[END_REF] first argue that increasing the weight of the KL divergence term in the ELBO loss enforces a higher disentanglement of the latent features as the posterior probability is forced to match a multivariate normal standard Gaussian. Several other methods were developed following this idea by for instance gradually increasing the proximity between the posterior and the prior [START_REF] Christopher | Understanding disentangling in β-vae[END_REF], augmenting the VAE objective with a penalty that encourages factorial representation of the marginal distributions [START_REF] Kim | Disentangling by factorising[END_REF] or enforcing an added penalty on the total correlation favouring disentanglement [START_REF] Ricky Tq Chen | Isolating sources of disentanglement in variational autoencoders[END_REF].

Amending the distance between distributions It can be stressed that the reconstruction term E z∼q ϕ (z|x) [log p θ (x|z)] in eq. ( 2) has a crucial role in the reconstruction and that its choice should be dependent of the application. For instance, methods such as using a discriminator [START_REF] Boesen | Autoencoding beyond pixels using a learned similarity metric[END_REF] or using a deterministic differentiable loss function [START_REF] Snell | Learning to generate images with perceptual similarity metrics[END_REF] acting as a distance between the input data and its reconstruction were also proposed. The second term in the ELBO measures the distance between the approximate posterior and the prior distribution through the KL divergence and it has however been argued that other distances between probability distributions could be used instead. Hence, approaches using a GAN to distinguish samples from the posterior from samples from the prior distribution [START_REF] Alireza | Adversarial autoencoders[END_REF] or methods based on optimal transport were also proposed [START_REF] Tolstikhin | Wasserstein auto-encoders[END_REF][START_REF] Zhao | Infovae: Information maximizing variational autoencoders[END_REF].

The Pythae library

Why Pythae ? To the best of our knowledge, there exists no framework combining and unyfing implementations of state-of-the-art Generative AutoEncoder (GAE) methods. This induces both a strong brake for reproducible research and democratisation of the models since implementations might be difficult to adapt to other use-cases, no longer maintained, or completely unavailable.

Project vision

Starting from this observation, we created Pythae, an open-source python library inspired from [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF][START_REF] Wolf | Transformers: State-of-the-art natural language processing[END_REF] providing unified implementations of generative autoencoding methods, allowing for easy use and training of GAE models. This provides on-the-shelf ready-to-use implementations directly applicable to any use case. This library is distributed under the Apache2.0 licence. Pythae is unit-tested ensuring code quality and continuous development with a code coverage of 97% as of release 0.1. To make it as accessible as possible, the library has an online documentation 3 and is also explained and illustrated through tutorials available either on a local machine or on the Google Colab platform [START_REF] Bisong | Google Colaboratory[END_REF]. This library is intended to evolve with the addition of new models through time to enrich the existing model base. Furthermore, beyond providing unified implementations, we believe that Pythae can also provide an open model testing environment driven by the community where people can contribute by adding their own models, and by doing so encourage reproducible research and the gathering of proposed methods at a same location.

Code structure Pythae was thought for easy model training and data generation, while striving for simplicity with a quick and user-friendly model selection and configuration. The backbone of the library is the module pythae.models in which all the autoencoder models are implemented. Each model implementation is accompanied with a configuration dataclass containing any hyperparameters relative to the model and allowing easing configuration loading and saving from json files. All the models are implemented using a common API allowing for a seamless integration with pythae.trainers (for training) and pythae.samplers (for generation) along with a simplified usage as illustrated in Fig. 1. In particular, Pythae provides pipelines allowing to train an autoencoder model or to generate new data with only a few lines of code. It mainly relies on the Pytorch [START_REF] Paszke | Automatic differentiation in pytorch[END_REF] framework and in its basic usage only essential hyper-parameter configurations and data (arrays or tensors) are needed to launch a model training or generation. More advanced options allowing further flexibility such as defining custom encoder and decoder neural nets are also available and can be found in the documentation and tutorials. Moreover, the library also integrates an easy-to-use experiments tracking tool (wandb) [START_REF] Biewald | Experiment tracking with weights and biases[END_REF] allowing to monitor runs launched with Pythae and compare them through a graphic interface. It can adapt to various types of data through the use of different already-implemented or user provided encoder and decoder neural-network architectures. In addition, Pythae also provides several ways to generate new data through different popular sampling methods in the pythae.samplers module. We detail some aspects of the library in Appendix. A. We now propose a case study where we use this library to perform a benchmark comparison of implemented models on image reconstruction and generation, latent vector classification and clustering, and image interpolation on 3 datasets and 2 distinct autoencoding architectures.

Benchmark setting

In this section, we present the setting of the benchmark. Comprehensive results for all the experiments are available through the monitoring tool [START_REF] Biewald | Experiment tracking with weights and biases[END_REF] used in Pythae to allow complete transparency. We also intend to share the trained models upon acceptance.

The data To perform the different tasks presented in this paper, 3 classical image datasets are considered: MNIST [START_REF] Lecun | The MNIST database of handwritten digits[END_REF], CIFAR10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] and CELEBA [START_REF] Liu | Deep learning face attributes in the wild[END_REF]. These datasets are publicly available, widely used for generative model related papers and have well known associated metrics in the literature. Each dataset is split into a train set, a validation set and a test set. For MNIST and CIFAR10 the validation set is composed of the last 10k images extracted from the official train set and the test set corresponds to the official one. For CELEBA, we use the official train/val/test split.

The models We propose to compare 19 generative autoencoder models representative of the improvements proposed in the literature and presented in Sec. 2.2. Descriptions and explanations of each implemented model can be found in Appendix. D.4. We use as baseline an Autoencoder (AE) and a Variational Autoencoder (VAE). To assess the influence of a more expressive prior, we propose using a VAE with VAMP prior (VAMP) [START_REF] Tomczak | Vae with a vampprior[END_REF] and regularised autoencoders with either a gradient penalty (RAE-GP) or a L2 penalty on the weights of the decoder (RAE-L2) that use ex-post density estimation [START_REF] Ghosh | From variational to deterministic autoencoders[END_REF]. To represent models trying to reach a better lower bound, we choose a Importance Weighted Autoencoder (IWAE) [START_REF] Burda | Importance weighted autoencoders[END_REF] and VAEs adding either simple linear normalising flows (VAE-lin-NF) [START_REF] Rezende | Variational inference with normalizing flows[END_REF] or using IAF (VAE-IAF) [START_REF] Durk | Improved variational inference with inverse autoregressive flow[END_REF]. For disentanglement-based models, we select a β-VAE [START_REF] Higgins | beta-VAE: Learning basic visual concepts with a constrained variational framework[END_REF], a FactorVAE [START_REF] Kim | Disentangling by factorising[END_REF] and a β-TC VAE [START_REF] Ricky Tq Chen | Isolating sources of disentanglement in variational autoencoders[END_REF]. To stress the influence of the distance used between distributions we add a Wasserstein Autoencoder (WAE) [START_REF] Tolstikhin | Wasserstein auto-encoders[END_REF] and an InfoVAE [START_REF] Zhao | Infovae: Information maximizing variational autoencoders[END_REF] with either Inverse Multi-Quadratic (IMQ) or a Radial Basis Function kernel (RBF) together with an Adversarial Autoencoder (AAE) [START_REF] Alireza | Adversarial autoencoders[END_REF], a VAEGAN [START_REF] Boesen | Autoencoding beyond pixels using a learned similarity metric[END_REF] and a VAE using structural similarity metric for reconstruction (MSSSIM-VAE) [START_REF] Snell | Learning to generate images with perceptual similarity metrics[END_REF]. Finally, we add a VQVAE [START_REF] Van Den | Neural discrete representation learning[END_REF] since having a discrete latent space has proven to yield promising results. Models implemented in Pythae requiring too much training time or more intricate hyper-parameter tuning were excluded from the benchmarks.

In the following, we will distinguish AE-based (autoencoder-based) methods (AE, RAE, WAE and VQVAE) from the other variational-based methods.

Training paradigm Each of the aforementioned models is equipped with the same neural network architecture for both the encoder and decoder leading to a comparable number of parameters 4 . For each task, 10 different configurations are considered for each model, allowing a simple exploration of the models' hyper-parameters, leading to 10 trained models for each dataset and each neural network type (ConvNet or ResNet) leading to a total of 1140 models 5 . It is important to note that the hyper-parameter exploration is not exhaustive and models sensitive to hyper-parameter tuning may have better performances with a more extensive parameter search. The sets of hyper-parameters explored are detailed in Appendix. D.4 for each model.

Experiments

In this section, we present the main results observed on 5 downstream tasks.

Fixed latent dimension

In this first part, latent dimensions are set to 16, 256 and 64 for the MNIST, CIFAR10 and CELEBA datasets respectively. See results per model and across the 10 configurations specified in Appendix. D.4 to assess the influence of the parameters on the tasks.

Task 1: Image reconstruction For each model reconstruction error is evaluated by selecting the configuration minimising the Mean Square Error (MSE) on the validation set, while results are shown on the test set 6 . We show in Table . 1 the MSE and Frechet Inception Distance7 (FID) [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF] of the reconstructions from this model on the test set. It is important to note that using the MSE as a metric places models using different reconstruction losses (VAEGAN and MSSSIM-VAE) at a disadvantage. As expected, the autoencoder-based models seem to perform best for the reconstruction task. Nonetheless, this experiment also shows the interest of adding regularisation to the autoencoder since improvements over the AE (RAE-GP, RAE-L2) achieve better performance than the regular AE. Moreover, β-VAE type models demonstrate their versatility as small enough β values can lead to less regularisation, therefore favouring a better reconstruction.

Task 2: Image generation We consider an image generation task with the trained models. In this experiment, we also explore different ways of sampling new data, either 1) using a simple distribution chosen as N (0, I d ) and corresponding to the prior for variational approaches (N ); 2) fitting a 10 components mixture of Gaussian in the latent space post training as proposed in [START_REF] Ghosh | From variational to deterministic autoencoders[END_REF] (GMM), 3) fitting a normalising flow taken as a Masked Autoregressive Flow (MAF) [START_REF] Papamakarios | Masked autoregressive flow for density estimation[END_REF] or 4) fitting a VAE in a similar fashion as [START_REF] Dai | Diagnosing and Enhancing VAE Models[END_REF]. For the MAF, two-layer MADE [START_REF] Mathieu Germain | Made: Masked autoencoder for distribution estimation[END_REF] are used. For each sampler, we select the models achieving the lowest FID on the validation set and compute the Inception Score8 (IS) [START_REF] Salimans | Improved techniques for training gans[END_REF] and the FID on the test set 9 . The main results are shown in Table . 3 for the normal and GMM sampler (see Appendix. D.2 for the other sampling schemes). One of the key findings of this experiment is that performing ex-post density (therefore not using the standard Gaussian prior) for the variational approach tends to almost always lead to better generation metrics even when a simple 10-components mixture of Gaussian is used. Interestingly, we note that when a more advanced density estimation model such as a MAF is used, results appear equivalent to those of the GMM (see Appendix. D.2). This may be due to the simplicity of the database we used and in consequence of the distribution of the latent codes that can be approximated well enough with a GMM. It should nonetheless be noted that the number of components in the GMM remains a key parameter which was set to the number of classes for MNIST and CIFAR10 since it is known, however too high a value may lead to overfitting while a low one may lead to worse results. For this task the adversarial approach proposed in the VAEGAN clearly surpasses peers on MNIST and CELEBA with impressive results when compared to other models. Nonetheless, fitting such a model may be challenging due to the adversarial approach and results can be considerably affected if parameters are not chosen correctly (see Appendix. D.4).

Task 3: Classification To measure the meaningfulness of the learned latent representations we perform a simple classification task with a single layer classifier as proposed in [START_REF] Coates | An analysis of single-layer networks in unsupervised feature learning[END_REF]. The rationale behind this is that if a GAE succeeds in learning a disentangled latent representation a simple linear classifier should perform well [START_REF] Berthelot | Understanding and improving interpolation in autoencoders via an adversarial regularizer[END_REF]. A single layer classifier is trained in a supervised manner on the latent embeddings of MNIST and CIFAR10. The train/val/test split used is the same as for the autoencoder training. For each model configuration, we perform 20 runs of the classifier on the latent embeddings and define the best hyper-parameter configuration as the one achieving the highest mean accuracy on these 20 runs on the validation set. We report the mean accuracy on the test set across the 20 runs for the selected configuration in Table . 2 (left) 10 . As expected, models explicitly encouraging disentanglement in the latent space such as the β-VAE and β-TC VAE achieve better classification when compared to a standard VAE. Nonetheless, AE-based models seem again the best suited for such a task since variational approaches tend to enforce a continuous space, consequently bringing latent representations of different classes closer to each other.

Task 4: Clustering As a complement to the previous task, performing clustering directly in the latent space of the trained autoencoders can give insights on the quality of the latent representation. Indeed, a well defined latent space will maintain the separation of the classes inherent to the datasets, leading to easy and stable k-means performances. To do so, we propose to fit 100 separate runs of the k-means algorithm and we show the mean accuracy obtained on the train embeddings in Table . 2 (right) 11 . This experiment allows us to explore and measure the clusterability of the generated latent spaces [START_REF] Berthelot | Understanding and improving interpolation in autoencoders via an adversarial regularizer[END_REF]. To measure accuracy we assign the label of the most prevalent class to each cluster. The conclusions of this experiment are slightly different from the previous one since models targeting disentanglement seem to be equalled by the original VAE. Interestingly, adversarial approaches and other alternatives to the standard VAE KL regularisation method seem to achieve the best results.

Task 5: Interpolation Finally, we propose to assess the ability of the model to perform meaningful interpolations. For this task, we consider a starting and ending image in the test set of MNIST and CIFAR10 and perform a linear interpolation in the generated latent spaces between the two encoded images. We show in Appendix. B the decoding along the interpolation curves. For this task, no metric was found relevant since the notion of "good" interpolation can be disputable. Nonetheless, the obtained interpolated images can be reconstructed and qualitatively evaluated. For this task, variational approaches were found to obtain better results as the inherent structure the posterior distribution imposes in the latent space results in a "smoother" transition from one image to another when compared to autoencoders that mainly superpose images, especially in higher dimensional latent spaces.

Varying latent dimension

An important parameter of autoencoder models which is too often neglected in the literature is the dimension of the latent space. We now propose to keep the same configurations as previously but re-evaluate Tasks 1 to 5 with the latent space varying in the range [START_REF] Coates | An analysis of single-layer networks in unsupervised feature learning[END_REF][START_REF] Diederik | Adam: A method for stochastic optimization[END_REF][START_REF] Zhao | Infovae: Information maximizing variational autoencoders[END_REF]128,256,512]. Results are shown in Fig. 2 for MNIST and a ConvNet12 (see Appendix. B and D for CIFAR, ResNet and interpolations). For the generation task, we select the sampler with lowest FID on the validation set.

Assessing the influence of the latent dimension A clear difference in behaviour is exhibited between variational-based and AE-based methods. For each given task, AEs share a common trend with respect to the evolution of the latent dimension: a common optimal latent dimension is found for each task, but differs drastically among different tasks (e.g 512 for reconstruction, 16 for generation, either 16 or 512 for classification and 512 for clustering with the MNIST dataset). This suggests the existence of a common intra-group optimal latent space dimension for a given task. In addition, we observe that β-VAE type methods (with the right hyper-parameter choice) can exhibit similar behaviours to AE models. The same observation can be made for variational-based methods, where it is interesting to note that although lower performances are achieved, the optimal latent dimension varies less with respect to the choice of the task. Therefore, a latent dimension of 16 to 32 appears to be the optimal choice for all 4 Tasks on the MNIST dataset, and 32 to 128 on the CIFAR10 dataset. It should be noted that unsupervised tasks such as clustering of the latent representation of the CIFAR10 dataset are hard and models are expected to perform poorly leading to less interpretable results.

V A E V A M P I W A E V A E _ L in N F V A E _ I A F B e t a V A E B e t a T C V A E F a c t o r V A E I N F O V A E -im q I N F O V A E -r b f A d v e r s a r ia l_ A E V A E G A N A E W A E -im q W A E -r b f V Q V A E R A E _ L 2 R A E _ G P 0.

Conclusion

In this paper, we introduced Pythae, a new open-source Python library unifying common and state-of-the-art Generative AutoEncoder (GAE) implementations, allowing reliable and reproducible model training, data generation and experiment tracking. This library was designed as an open model testing environment driven by the community, wherein peers are encouraged to contribute by adding their own models, and by doing so favour reproducible research and accessibility to readyto-use GAE models. As an illustration of the capabilities of Pythae, we perform a benchmarking of 19 generative autoencoder models on 5 downstream tasks (image reconstruction, generation, classification, clustering and interpolation) leading to some key findings. We show that performing ex-post density estimation in the latent space to balance the potential poor expressiveness of the prior distribution (GMM sampling) tends to almost always improve image quality generation. Interestingly, using more complex density estimators than a GMM did not improve results. We also show that AEbased generative methods achieve the best results on most common tasks (reconstruction, generation, classification). However, they tend to be more sensitive to the choice of the latent dimension and don't scale well to high dimension on more complex tasks such as interpolation. Finally we show

A Example of Pythae use

In this section we illustrate through simple examples how to use Pythae pipelines. The library is documented 13 and also available on pypi 14 allowing a wider use and easier integration in other codes. All of the implementations proposed in the library are adaptations of the official code when available and allowed by the licence. If not, the method is re-implemented.

B Interpolations

In this section, we show the interpolations obtained on the three considered datasets. For each model, we select both a starting image and an ending image from the test set and perform a linear interpolation between the corresponding embeddings in the learned latent space. We then show the decoded trajectory all along the interpolation line. For this task, we use the model configuration that obtained the lowest FID on the validation set with a GMM sampler from the generation task. We show the resulting interpolations for latent spaces of dimension 16 and 256 for MNIST, 32 and 256 for CIFAR10 and 64 for CELEBA. As mentioned in the paper, for this complex task, variational approaches tend to outperform the AE-based methods. This is well illustrated on MNIST with a latent space of dimension 256 since all the AE-based approaches eventually superpose the starting and ending image, making the interpolation visually irrelevant. Impressively, the regularisation imposed by the variational approaches prevents such undesirable behaviours from occurring. This adds to the observation made in Sec. 4.2.2 of the paper where we note some robustness to the latent dimension for the variational methods. Nonetheless, as stated in the paper this regularisation can also degrade image reconstruction, leading to very blurry interpolations, as illustrated on Fig. 5.

C Detailed experiments set-up

We detail here the main experimental set-up and implementation choices made in the benchmark. We let the reader refer to the code available online for specific implementation aspects

The data To perform the benchmaks presented in the paper, we select 3 classical free-to-use image datasets: MNIST [START_REF] Lecun | The MNIST database of handwritten digits[END_REF], CIFAR10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] and CELEBA [START_REF] Liu | Deep learning face attributes in the wild[END_REF]. These datasets are publicly available, widely used for generative model related papers and have well known associated metrics in the literature. Each dataset is split into a train set, a validation set and a test set. For MNIST and CIFAR10 the validation set is composed of the last 10k images extracted from the official train set and the test set corresponds to the official one. For CELEBA, we use the official train/val/test split.

Training paradigm We equip each model used in the benchmark with the same neural network architecture for both the encoder and decoder, taken as a ConvNet and ResNet (architectures given in Tables. 5 and 6) leading to a comparable number of parameters 15 . For the 19 considered models, due to computational limitations, 10 different configurations are considered, allowing a simple exploration of the models' hyper-parameters. The sets of hyper-parameters explored are detailed in Appendix. D.4 for each model. The models are then trained on MNIST and CIFAR10 for 100 epochs, a starting learning rate of 1e -4 and batch size of 100 with Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF]. A scheduler reducing the learning rate by half if the validation loss does not improve for 10 epochs is also used. For CELEBA, we use the same setting but we train the models for 50 epochs with a starting learning rate of 1e Sampling paradigm for the MAF and VAE samplers For the Masked Autoregressive Flow sampler used for sampling we use a 3-layer MADE [START_REF] Mathieu Germain | Made: Masked autoencoder for distribution estimation[END_REF] with 128 hidden units and ReLU activation for each layer and stack 2 blocks of MAF to create the flow. For the masked layers, the mask is made sequentially and the ordering is reversed between each MADE. For this normalising flow we consider a starting distribution given by a standard Gaussian. For the auxiliary VAE sampling method proposed in [START_REF] Dai | Diagnosing and Enhancing VAE Models[END_REF], we consider a simple VAE with a Multi Layer Perceptron (MLP) encoder and decoder, with 2 hidden layers composed of 1024 units and ReLU activation. Both samplers are fitted with 200 epochs using the train and evaluation embeddings coming from the trained autoencoder models. A learning rate of 1e -4 , a scheduler decreasing the learning rate by half if the validation loss does not improve for 10 epochs and a batch size of 100 are used for these samplers.

D Additional results

D.1 Effect of the latent dimension on the 4 tasks with the CIFAR10 database

Analogously to the results shown in the paper on the MNIST dataset for the 4 chosen tasks (reconstruction, generation, classification and clustering), Fig. 9 shows the impact the choice of the latent space dimension has on the performances of the models on the CIFAR10 dataset, whose image arguably have a greater intrinsic latent dimension than images of the MNIST dataset. Similarly to MNIST, two distinct groups appear: the AE-based methods and variational methods. Again, for all tasks but clustering, variational based methods demonstrate good robustness properties with respect to the dimension of the latent space when compared to AE approaches.

V A E V A M P I W A E V A E _ L in N F V A E _ I A F B e t a V A E B e t a T C V A E F a c t o r V A E I N F O V A E -im q I N F O V A E -r b f A d v e r s a r ia l_ A E V A E G A N A E W A E -im q W A E -r b f V Q V A E R A E _ L 2 R A E

D.2 Complete generation table

In Table . 7 are presented the full results obtained for generation i.e. including the MAF and 2-stage VAE sampler [START_REF] Dai | Diagnosing and Enhancing VAE Models[END_REF]. As mentioned in the paper, it is interesting to note that fitting a GMM instead of using the prior for the variational-based approaches seems to often allow a better image generation since it allows a better prospecting of the learned latent space of each model. Interestingly, it seems that fitting more complex density estimators such as a normalising flow (MAF sampler) or another VAE (2-stage sampler) does not improve the generation results when compared to the GMM for those datasets.

D.3 Further interesting results

Generated samples In addition to quantitative metrics, we also provide in Fig. 10 and Fig. 11 some samples coming from the different models using either a N (0, I d ) or fitting a GMM with 10 components on MNIST and CELEBA. This allows to visually differentiate the quality of the different sampling methods. Sampler ablation study Fig. 12 shows the same results as Table . 7 but under a different prism.

In this plot, we show the influence each sampler has on the generation quality for all the models considered in this study. Note that sampling using a N (0, I d ) for an AE, RAE or VQVAE is far from being optimal since those models do not enforce explicitly the latent variables to follow this distribution. As mentioned in the paper, this experiment shows that using more complex density estimators such as a GMM or a normalising flow almost always improves the generation metric. Neural network architecture ablation study As explained in the paper and in Appendix. C, we consider two different neural architectures for the encoder and decoder of each model: a ConvNet (convolutional neural network) and a ResNet (residual neural network). Fig. 13 shows the influence the choice of the neural architecture has on the ability of the model to perform the 4 tasks presented in the paper. The results are computed for each model on MNIST and a latent dimension of 16.

V A E V A M P IW A E V A E _ L in N F V A E _ IA F B e t a V A E B e t a T C V A E F a c t o r V A E IN F O V A E -im q IN F O V A E -r b f A d v e r s a r ia l_ A E V A E G A N A E W A E -im q W A E -r b f V Q V A E R A E _ L 2 R A E _ G P
The ConvNet architecture has approximately 20 times more parameters than the ResNet in such conditions. We select the best configuration for each model and each task on the validation set and report the results on the test set. Unsuprisingly, we see in Fig. 13 that the ConvNet architecture, more adapted to capture features intrinsic to images, leads to the best performances for reconstruction and generation. Interestingly, the ResNet outperforms the ConvNet for the classification and clustering tasks, meaning that in addition to the network complexity, its structure can play a major role in the representation learned by the models.

Training time Fig. 14 shows the training times required for each model for both network architectures on the MNIST dataset. For each model, we show the results obtained with the configuration giving the best performances on the generation task with fixed latent dimension 16. It is interesting to note that although VAEGAN outperforms other models on the generation task, it is at the price of a higher computational time. This is due to the discriminator network (a convolutional neural net) that is called several times during training and takes images as inputs. It should be noted that methods applying normalising flows to the posterior (VAE-lin-NF and VAE-IAF) maintain a reasonable training time, as the flows were chosen for their scalability. V

V A E V A M P I W A E V A E _ L in N F V A E _ I A F B e t a V A E B e t a T C V A E F a c t o r V A E I N F O V A E -im q I N F O V A E -r b f A d v e r s a r ia l_ A E V A E G A N A E W A E -im q W A E -r b f V Q V A E R A E _ L 2 R A E _ G P 0.
A E V A M P I W A E V A E _ L in N F V A E _ I A F B e t a V A E B e t a T C V A E F a c t o r V A E I N F O V A E -im q I N F O V A E -r b f A d v e r s a r ia l_ A E V A E G A N A E W A E -im q W A E -r b f V Q V A E R A E _ L 2 R A E _ G P 10 1

D.4 Configurations and results by models

In this section we briefly explain each model considered in the benchmark, and show the evolution of performances on the 4 tasks and the training speed with respect to the choice of the hyper-parameters. For all 4 tasks we consider the MNIST dataset and a fixed latent space of dimension 16, as well as the Normal Gaussian sampler (if applicable) and the convolutional network architecture. For each model, 10 configuration runs with different hyper-parameters were tested. It should be noted that this configuration search was done empirically and is not exhaustive, therefore models with multiple hyper-parameters or that are sensitive to the choice of hyper-parameters will tend to have sub-optimal configuration choices. Although hyper-parameter choices are dependant on both the auto-encoder architecture and the dataset, it is interesting to note the relative evolution of the performances on the different tasks and the training time induced by different hyper-parameter choices.

Notations In order to better underline the differences between different models and for clarity purposes, we set the following unified notations:

• X = {x 1 , . . . , x N } ∈ X N the input dataset
• x ∈ X an observation from the dataset, and z ∈ Z = R d its corresponding latent vector • x the reconstruction of x by the auto-encoder model • p z (z) the prior distribution, with p z ≡ N (0, I d ) under standard VAE assumption • q ϕ (z|x) the approximate posterior distribution, modelled by the encoder. Kingma and Welling [START_REF] Diederik | Auto-encoding variational bayes[END_REF] set

q ϕ (z|x) ≡ N µ ϕ (x), Σ ϕ (x))
where Σ ϕ (x) = diag[σ ϕ (x)] and µ ϕ (x), σ ϕ (x) ∈ R 2×d are outputs of the encoder network. The sampling process z ∼ q ϕ (z|x) is therefore performed by sampling ε ∼ N (0, We further recall that Kingma and Welling [START_REF] Diederik | Auto-encoding variational bayes[END_REF] use the unbiased estimate p θ (x) of p θ (x) defined as

I d ) and setting z = µ ϕ (x) + Σ ϕ (x) 1/2 • ε (re-parametrization trick). • p θ (x|z) the distribution of x given z • p θ (x) = Z p θ (x|z)p z (z)dz the marginal distribution of x • q ϕ (z) = 1 N N i=1 q ϕ (z|x i )
p θ (x) = p θ (x|z)p z (z) q ϕ (z|x)
to derive the standard Evidence Lower Bound (ELBO) of the log probability log p θ (x) which we wish to maximize:

L ELBO = E x∼p θ (x) L ELBO (x) with L ELBO (x) = E z∼q ϕ [log p θ (x|z)] reconstruction -D KL q ϕ (z|x)||p z (z) regularisation
The reconstruction loss is maximized when x is close to x, thus encouraging a good reconstruction of the input x, while the regularisation term is maximized when q ϕ (z|x) is close to p z (z), encouraging the posterior distribution to follow the chosen prior distribution.

The integration over p θ (x) is approximated by the empirical distribution of the training dataset, and the negative ELBO function acts as a loss function to minimize for the encoder and decoder networks.

VAE with a VampPrior (VAMP) Starting from the observation that a standard Gaussian prior may be too simplistic, Tomczak and Welling [START_REF] Tomczak | Vae with a vampprior[END_REF] proposes a less restrictive prior: the Variational Mixture of Posteriors (VAMP). A VAE with a VAMP prior aims at relaxing the posterior constraint by replacing the conventional normal prior with a multimodal aggregated posterior given by:

p z (z) = 1 K K k=1 q ϕ (z|u k ) ,
where u k are pseudo-inputs living in the data space X learned through back-propagation and acting as anchor points for the prior distribution. For the VAMP VAE implementation, we use the same architecture as the authors' implementation for the network generating the pseudo-inputs: a MLP with a single layer and Tanh activation. Importance Weighted Autoencoder (IWAE) Burda et al. [START_REF] Burda | Importance weighted autoencoders[END_REF] introduce an alternative lower bound to maximize, derived from importance weighting where the new unbiased estimate p θ (x) of the marginal distribution p θ (x) is computed with L samples z 1 , . . . , z L ∼ q ϕ (z|x) :

Results by configuration

p θ (x) = 1 L L i=1 p θ (x|z i )p z (z i ) q ϕ (z i |x) .
This estimate induces a new lower bound of the true marginal distribution p θ (x) using Jensen's inequality:

L IWAE (x) := E z1,...,z L ∼q(z|x) log 1 L L i=1 p θ (x|z i )p z (z i ) q ϕ (z i |x) ≤ log E z1,...,z L ∼q(z|x) p θ (x) p θ (x)
.

As the number of samples L increases, L IWAE (x) becomes closer to log p θ (x), therefore providing a tighter bound on the true objective. Note that when L = 1 we recover the original VAE framework.

As expected the reconstruction quality increases with the number of samples. Nonetheless, we note that increasing the number of samples has a significant impact on the computation of a single training step, therefore leading to a much slower training process. Variational Inference with Normalizing Flows (VAE-lin-NF) In order to model a more complex family of approximate posterior distributions, Rezende and Mohamed [START_REF] Rezende | Variational inference with normalizing flows[END_REF] propose to use a succession of normalising flows to transform the simple distribution q ϕ (z|x), allowing it to model more complex behaviours. In practice, after having sampled z 0 ∼ q ϕ (z|x), z 0 is passed through a chain of K invertible smooth mappings from the latent space R d to itself:

Results by configuration

z K = f K • • • • • f 2 • f 1 (z 0 ) .
The modified latent vector z K is then used as input z for the decoder network. In their paper, the authors propose to use two types of transformations: planar and radial flows.

f planar (z) = z + uh(w ⊤ z + b) ; f radial (z) = z + βg(α, r)(z -z 0 ) ,
where h is a smooth non-linearity with tractable derivatives and g(α, r) = 1 α+r . The parameters are such that r = ∥z -z 0 ∥, u, w, z 0 ∈ R d , α ∈ R + and b, β ∈ R. We can easily compute the resulting density q given by log q(z K ) = log q ϕ (z 0 |x) -

K k=1 log det ∂f k ∂z .
This makes the ELBO tractable and optimisation possible. Variational Inference with Inverse Autoregressive Flow (VAE-IAF) Kingma et al. [START_REF] Durk | Improved variational inference with inverse autoregressive flow[END_REF] improve upon the works of [START_REF] Rezende | Variational inference with normalizing flows[END_REF] with a new type of normalising flow that better scales to high-dimensional latent spaces. The main idea is again to apply several transformations to a sample from a simple distribution in order to model richer distributions. Starting from z 0 ∼ q ϕ (z|x), the proposed IAF flow consists in applying consecutively the following transformation

Results by configuration

z k = µ k + σ k • z k-1 ,
where µ k and σ k are the outputs of an autoregressive neural network taking z k-1 as input. Inspired from the original paper, to implement one Inverse Autoregressive Flow we use MADE [START_REF] Mathieu Germain | Made: Masked autoencoder for distribution estimation[END_REF] and stack multiple IAF together to create a richer flow. The MADE mask is made sequentially for the masked autoencoders and the ordering is reversed after each MADE. [START_REF] Higgins | beta-VAE: Learning basic visual concepts with a constrained variational framework[END_REF] argue that increasing the weight of the KL divergence term in the ELBO loss enforces a stronger disentanglement of the latent features as the posterior probability is forced to match a multivariate standard Gaussian. They propose to add a hyper-parameter β in the ELBO leading to the following objective to maximise:

Results by configuration

L β-VAE (x) = E z∼q ϕ [log p θ (x|z)] -βD KL q ϕ (z|x)||p z (z) .
Although the original publication specifies β > 1 to encourage a better disentanglement, a smaller value of β can be used to relax the regularisation constraint of the VAE. Therefore, for this model we consider a range of values for β from 1e -3 to 1e 3 .

As expected, we see in Fig. 19 a trade-off appearing between reconstruction and generation. Indeed, a very small β will tend to less regularise the model since the latent variables will no longer be driven to follow the prior, favouring a better reconstruction. On the other hand, a higher value for β will constrain the model, leading to a better generation quality. Moreover, as can be seen in Fig. 19, too high a value of β will lead to over-regularisation, resulting in poor performances on all evaluated tasks. [START_REF] Ricky Tq Chen | Isolating sources of disentanglement in variational autoencoders[END_REF] extend on the ideas of Higgins et al. [START_REF] Higgins | beta-VAE: Learning basic visual concepts with a constrained variational framework[END_REF] by rewriting and re-weighting specific terms in the ELBO loss with multiple hyperparameters. The authors note that the KLdivergence term of the ELBO loss can be rewritten as

Results by configuration

E x∼p θ D KL q ϕ (z|x)||p z (z) = I(x, z) Mutual information + D KL q ϕ (z)|| d j=1 q ϕ (z j ) TC-loss + d j=1 D KL q ϕ (z j )||p z (z j )
Dimension-wise KL

• The mutual information term corresponds to the amount of information shared by x and its latent representation z. It is claimed that maximising the mutual information encourages better disentanglement and a more compact representation of the data. • The TC-loss corresponds to the total correlation between the latent distribution and its fully disentangled version, maximising it enforces the dimensions of the latent vector to be uncorrelated. • Maximising the dimension-wise KL prevents the marginal distribution of each latent dimension from diverging too far from the prior Gaussian distribution

The authors therefore propose to replace the classical regularisation term with the more general term

L reg := αI(x, n) + βD KL q ϕ (z)|| j q ϕ (z j ) + γ j D KL q ϕ (z j )||p z (z j ) .
Similarly to the authors, we set α = γ = 1 and only perform a search on the parameter β. Fig. 20 shows a reconstruction-generation trade-off similar to the β-VAE model Factor VAE Kim and Mnih [START_REF] Kim | Disentangling by factorising[END_REF] augment the VAE objective with a penalty that encourages factorial representation of the marginal distributions, enforcing a stronger disentangling of the latent space.

Results by configuration

Noting that a high β value in β-VAE ELBO loss encourages disentanglement at the expense of reconstruction quality, FactorVAE proposes a new lower bound of the log likelihood with an added disentanglement term:

L FactorVAE (x) := L ELBO (x) -γD KL q ϕ (z)||q ϕ (z) , with qϕ (z) := d j=1 q ϕ (z j )
The distribution of representations q ϕ (z) = 1 N N i=1 q ϕ (z|x i ) of the entire dataset is therefore forced to be close to its fully-disentangled equivalent qϕ (z) while leaving the ELBO loss as it is. They further propose to approximate the KL divergence with a discriminator network D that is trained jointly to the VAE:

D KL q(z)||q(z) ≈ E qz(z) log D(z) 1 -D(z)
As suggested in the authors's paper, the discriminator is set as a MLP composed of 6 layers each with 1000 hidden units and LeakyReLU activation. [START_REF] Zhao | Infovae: Information maximizing variational autoencoders[END_REF] note that the traditional VAE ELBO objective can lead to both inaccurate amortized inference and VAE models that tend to ignore most of the latent variables, therefore not fully taking advantage of the modelling capacities of the VAE scheme and learning less meaningful latent representations. In order to counteract these two issues, they propose to rewrite and re-weight the ELBO objective in order to counterbalance the imbalance between the distribution in the data space and the latent space, and add a mutual information term between x and z to encourage a stronger dependency between the two variables, preventing the model from ignoring the latent encoding. One can re-write the ELBO loss in order to explicit the KL divergence between the marginalised posterior and the prior

Results by configuration

L ELBO (x) := -D KL [q ϕ (z)||p z (z)] -E z∼pz D KL [q ϕ (x|z)||p θ (x|z)] .
Introducing an additional mutual information term I q (x; z) and extending the objective function to use any given divergence D between probability measures instead of the KL objective, the authors propose a new objective defined as

L InfoVAE (x) := -λD[q ϕ (z)||p z (z)] -E z∼pz D KL [q ϕ (x|z)||p θ (x|z)] + αI q (x; z)
where λ and α are hyperparameters. In our experiments, α is set to 0 as recommended in the paper in the case where p θ (x|z) is a simple distribution. D is chosen as the Maximum Mean Discrepancy (MMD) [START_REF] Gretton | A kernel two-sample test[END_REF], defined as

MMD k (p λ (z), q ϕ (z)) = || Z k(z, .)dp λ (z) - Z k(z, .)dq ϕ (z)|| H k with k : Z × Z → R
a positive-definite kernel and its associated RKHS H k . We choose to differentiate 2 cases in the benchmark: one with a Radial Basis Function (RBF) kernel, the other with the Inverse MultiQuadratic (IMQ) kernel as proposed in [START_REF] Tolstikhin | Wasserstein auto-encoders[END_REF] where the kernel is given by k(x, y) = The authors underline that choosing λ > 0, α = 1 -λ and D = D KL , we recover the β-VAE model [START_REF] Higgins | beta-VAE: Learning basic visual concepts with a constrained variational framework[END_REF], while choosing α = λ = 1 and setting D as the Jensen Shannon divergence we recover the Adversarial AE model [START_REF] Alireza | Adversarial autoencoders[END_REF]. [START_REF] Alireza | Adversarial autoencoders[END_REF] propose to use a GAN-like approach by replacing the regularisation induced by the KL divergence with a discriminator network D trained to differentiate between samples from the prior and samples from the posterior distribution. The encoder network therefore acts as a generator network, leading to the following objective

Results by configuration

L AAE (x) = E z∼q ϕ (z|x) [log p θ (x|z)] + αL GAN ,
with L GAN the standard GAN loss defined by

L GAN = E z∼pz(z) log(1 -D(z))) + E x∼p θ E z∼q ϕ (z|x) [log D(z)] .
For the Adversarial Autoencoder implementation, we use a MLP neural network for the discriminator composed of a single hidden layer with 256 units and ReLU activation.

We observe a similar trade-off between reconstruction and generation quality as observed with β-VAE type models, as the α term acts like the β term, balancing between regularisation and reconstruction. [START_REF] Snell | Learning to generate images with perceptual similarity metrics[END_REF] propose an extension of the ELBO loss to a more general case where any deterministic reconstruction loss ∆(x, x) can be used by replacing the probabilistic decoder p θ with a deterministic equivalent f θ such that the reconstruction x of x given z ∼ q ϕ (z|x) is defined as x = f θ (z). The modified ELBO objective is thus defined as L EL-VAE (x) = ∆(x, x) -βD KL q ϕ (z|x)||p(z) , with β ≤ 1. As suggested in the original paper we use a multi scale variant of the single scale SSIM [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF]: the Multi Scale Structural Similarity Metric (MS-SSIM) [START_REF] Wang | Multiscale structural similarity for image quality assessment[END_REF]. VAE-GAN Larsen et al. [START_REF] Boesen | Autoencoding beyond pixels using a learned similarity metric[END_REF] use a GAN like approach by training a discriminator to distinguish real data from reconstructed data. In addition, the discriminator learns to distinguish between real data and data generated by sampling from the prior distribution in the latent space.

Results by configuration

Results by configuration

Noting that intermediate layers of a discriminative network trained to differentiate real from generated data can act as data-specific features, the authors propose to replace the reconstruction loss of the ELBO with a Gaussian log-likelihood between outputs of intermediate layers of a discriminative network D:

L VAE-GAN = E z∼q θ (z|x) log N (D l (x)|D l (x), I) reconstruction -D KL q ϕ (z|x)||p z (z) regularisation -L GAN ,
where D l is the output of the l th layer of the discriminator D, chosen to be representative of abstract intermediate features learned by the discriminator, and L GAN is the standard GAN objective defined as

L GAN = log D(x) 1 -D(x gen )
, where x gen is generated using z ∼ p z (z). As encouraged by the authors, we add a hyper-parameter α to the reconstruction loss for the decoder only, such that a higher value of α will encourage better reconstruction abilities with respect to the features extracted at the l th layer of the discriminator network, whereas a smaller value will encourage fooling the discriminator, therefore favouring regularisation toward the prior distribution. For the VAEGAN implementation, we use a discriminator whose architecture is similar to the model's encoder given in Table . 5. For MNIST and CIFAR we remove the BatchNorm layer and change the activation of layer 2 to Tanh instead of ReLU. For CELEBA, the BatchNorm layer is kept and the activation of layer 2 is also changed to Tanh. For all datasets, the output size of the last linear layer is set to 1 instead of d and followed by a Sigmoid activation. Wasserstein Autoencoder (WAE) Tolstikhin et al. [START_REF] Tolstikhin | Wasserstein auto-encoders[END_REF] generalise the VAE objective by replacing both terms in the ELBO: similarly to [START_REF] Snell | Learning to generate images with perceptual similarity metrics[END_REF] (EL-VAE), the reconstruction loss is replaced by any measurable cost function ∆, and the standard KL divergence is substituted with any arbitrary divergence D between two distributions, leading to the following objective function E q ϕ (z|x) [∆(x, x)] + λD z (p z (z), q ϕ (z)) , with λ a hyper-parameter. The authors propose two different penalties for D z :

Results by configuration

1. GAN-based: WAE-GAN An adversarial discriminatory network D(z, z ′ ) is trained jointly to separate the "true" points sampled from the prior p z (z) from the "fake" ones sampled from q ϕ (z|x), similarly to [START_REF] Alireza | Adversarial autoencoders[END_REF] (Adversarial AE).

MMD-based

The Maximum Mean Discrepancy is used as a distance between the prior and the posterior distribution. This is the case considered in the benchmark. We choose to differentiate 2 cases in the benchmark: one with a Radial Basis Function (RBF) kernel, the other with the Inverse MultiQuadratic (IMQ) kernel as proposed in [START_REF] Tolstikhin | Wasserstein auto-encoders[END_REF] where the kernel is given by k(x, y) = [START_REF] Van Den | Neural discrete representation learning[END_REF] propose to use a discrete space. Therefore, the latent embedding space is defined as a R K×D vector space of K different D dimensional embedding vectors E = {e 1 , . . . , e K } which are learned and updated at each iteration.

Results by configuration

Given an embedding size d and an input x, the output of the encoder z e (x) is of size R d×D . Each of its d elements is then assigned to the closest embedding vector resulting in an embedded encoding z q (x) ∈ E d such that z q (x) j = e l where l = argmin 1≤l≤d ||(z e (x)) j -e l || 2 for j ∈ [1, d]. Since the argmin operation is not differentiable, learning of the embeddings and regularisation of the latent space is done by introducing the stopgradient operator sg in the training objective: L VQ-VAE (x) := log p(x|z q (x)) + α||sg[z e (x)] -e|| 2 2 + β||z e (x) -sg[e]|| 2 2 . For the VQVAE implementation we use the Exponential Moving Average update as proposed in [START_REF] Van Den | Neural discrete representation learning[END_REF] to replace the term ||sg[z e (x)] -e|| 2 2 in the loss. Thus, we consider only two hyper-parameters in the search: the size of the dictionary of embeddings K and the regularisation factor β. (σ ϕ (x) i -log σ ϕ (x) i ) .

Results by configuration

Arguing that the regularisation of the VAE model is done through a noise injection mechanism by sampling from the approximate posterior distribution z ∼ N (µ ϕ , diag(σ ϕ )), the authors propose to replace this stochastic regularisation with an explicit regularisation term, leading to the following deterministic objective:

L RAE = ||x -x|| 2 2 + β 2 ||z|| 2 2 + λL REG , (4)
where L REG is an explicit regularisation. They propose to use either

• a L2 loss on the weights of the decoder (RAE-L2), which amounts to applying weight decay on the parameters of the decoder. • a gradient penalty on the output of the decoder (RAE-GP), which amounts to applying a L2 norm on the gradient of the output of the decoder.

Results by configuration

Figure 1 :

 1 Figure 1: Pythae library diagram

Figure 2 :

 2 Figure 2: From top to bottom: Evolution of the reconstruction MSE, generation FID, classification accuracy and clustering accuracy with respect to the latent space dimension on the MNIST dataset.

3 . 4 . 5 .

 345 Training A model training can then be launched by simply using the built-in training pipeline in which only the training/evaluation data need to be specified. arrays or tensors eval_data=your_eval_data) # arrays or tensors13 https://pythae.readthedocs.io/en/latest/?badge=latest 14 https://pypi.org/project/pythae/ Model reloading The weights and configuration of the trained model can be reloaded using the AutoModel instance proposed in Pythae.from pythae.models import AutoModel my_trained_vae = AutoModel.load_from_folder('path/to/trained_model') Data generation A data generation pipeline can be instantiated similarly to a model training. The pipeline can then be called with any relevant arguments such as the number of samples to generate or the training and evaluation data that may be needed to fit the sampler. from pythae.samplers import GaussianMixtureSamplerConfig from pythae.pipelines import GenerationPipeline # Define your sampler configuration

Figure 3 :Figure 4 :Figure 5 :Figure 6 :Figure 7 :Figure 8 :

 345678 Figure 3: Interpolations on MNIST with the same starting and ending images for latent spaces of dimension 16 and 256. For each model we select the configuration achieving the lowest FID on the generation task on the validation set with a GMM sampler.

Figure 9 :

 9 Figure 9: From top to bottom: Evolution of the reconstruction MSE, generation FID, classification accuracy and clustering accuracy with respect to the latent space dimension on the CIFAR dataset.

Figure 10 :Figure 11 :

 1011 Figure 10: Generated samples on MNIST for a latent space of dimension 16 and ConvNet architecture.For each model, we select the configuration achieving the lowest FID on the validation set.

Figure 12 :

 12 Figure12: Evolution of the FID for the generation task depending on the sampler, for a ConvNet, the MNIST dataset and a latent dimension of 16. For each sampler and model, we select the configuration achieving the lowest FID on the validation set.

Figure 13 :

 13 Figure 13: Evolution of the metrics for the 4 tasks depending on the network type on the MNIST dataset and a latent dimension of 16.

10 2Figure 14 :

 1014 Figure 14: Total training time for the models trained on the MNIST dataset with latent dimension 16 with the best performance on the generation task.

  the aggregated posterior integrated over the training set. • D KL the Kullback-Leibler divergence

Figure 15 :

 15 Figure 15: Results on VAMP

Figure 16 :

 16 Figure 16: Results on IWAE

Figure 17 :

 17 Figure 17: Results on VAE-lin-NF

Figure 19 :

 19 Figure 19: Results on β-VAE

Figure 20 :

 20 Figure 20: Results on β-TC-VAE

2 2

 2 with s ∈ [0.1, 0.2, 0.5, 1, 2, 5, 10] and C = 2 • d • σ 2 , d being the dimension of the latent space and σ a parameter part of the hyper-parameter search.

Figure 24 :

 24 Figure 24: Results on Adversarial AE

Figure 25 :

 25 Figure 25: Results on MSSSIM-VAE

Figure 26 :

 26 Figure 26: Results on VAEGAN

2 2

 2 with s ∈ [0.1, 0.2, 0.5, 1, 2, 5, 10] and C = 2 • d • σ 2 , d being the dimension of the latent space and σ a parameter part of the hyper-parameter search. As proposed by the authors, for this model we choose to use a deterministic encoder meaning that q ϕ (z|x) = δ µ ϕ (x) .

Figure 27 :Figure 28 :

 2728 Figure 27: Results on WAE-RBF

Figure 29 :

 29 Figure 29: Results on VQVAE

Figure 30 :

 30 Figure 30: Results on RAE-L2

Table 1 :

 1 Mean Squared Error(10 -3 ) and FID (lower is better) computed with 10k samples on the test set. For each model, the best configuration is the one achieving the lowest MSE on the validation set.

				ConvNet					ResNet
	Model	MNIST (16)	CIFAR10 (256)	CELEBA (64)	MNIST (16)	CIFAR10 (256)	CELEBA (64)
		MSE ↓	FID ↓	MSE ↓	FID ↓	MSE ↓	FID ↓	MSE ↓	FID ↓	MSE ↓	FID ↓	MSE ↓	FID ↓
	VAE	16.85	30.71	16.24	218.66	9.83	49.22	17.24	36.06	16.33	176.63	10.59	58.75
	VAMP	24.17	44.95	17.45	221.40	10.81	51.64	17.11	37.58	16.87	177.03	11.50	60.89
	IWAE	14.14	34.28	16.19	237.14	9.47	50.00	15.79	38.74	16.02	183.37	10.14	60.18
	VAE-lin-NF	16.75	31.14	16.57	221.39	9.90	49.84	17.23	36.74	16.59	177.08	10.68	58.73
	VAE-IAF	16.71	30.64	16.33	223.65	9.87	50.05	17.05	35.98	16.39	177.05	10.63	58.41
	β-VAE	5.61	10.55	3.60	50.55	7.28	46.96	5.87	15.81	2.40	55.67	7.78	51.59
	β-TC VAE	6.78	14.11	5.06	53.49	7.65	50.82	7.12	18.44	4.05	66.89	8.08	52.70
	Factor VAE	17.27	30.39	16.41	224.3	10.16	53.61	18.13	37.97	16.55	176.8	10.93	59.46
	InfoVAE-IMQ	16.65	30.62	16.19	216.44	9.81	50.51	17.17	37.33	16.32	173.79	10.63	58.04
	InfoVAE-RBF	16.59	30.63	16.23	217.52	9.85	50.14	17.01	37.04	16.32	175.37	10.64	58.68
	AAE	5.59	10.87	2.60	40.66	7.25	50.22	5.98	17.01	2.33	55.93	7.76	50.97
	MSSSIM-VAE	32.60	37.91	39.42	276.70	35.60	124.52	33.67	40.25	39.61	254.34	35.43	119.92
	VAEGAN	15.49	5.54	31.40	289.35	8.91	86.58	23.25	11.35	30.22	300.07	9.32	86.32
	AE	5.47	11.61	2.82	41.98	7.03	51.08	6.13	13.74	2.34	55.43	7.74	50.54
	WAE-IMQ	5.55	11.29	2.81	41.79	7.04	52.11	5.78	16.21	2.34	56.55	7.74	50.50
	WAE-RBF	5.53	11.34	2.82	42.21	7.03	51.43	5.80	16.14	2.34	56.00	7.74	51.38
	VQVAE	5.59	11.02	2.84	44.60	7.06	52.27	6.00	15.27	2.34	55.84	7.73	50.29
	RAE-L2	5.24	15.37	2.25	49.28	6.90	53.98	5.76	17.27	2.35	57.85	7.74	51.07
	RAE-GP	5.31	12.08	2.81	41.15	7.06	51.85	5.83	15.69	2.34	56.71	7.76	51.36

Table 2 :

 2 Left

			Classification			Clustering
	Model	ConvNet	ResNet	ConvNet	ResNet
		MNIST	CIFAR10	MNIST	CIFAR10	MNIST	CIFAR10	MNIST	CIFAR10
	VAE	86.75 (0.05)	32.61 (0.03)	86.80 (0.03)	32.37 (0.03)	69.71 (2.01)	17.18 (0.68)	74.21 (0.97)	18.12 (0.74)
	VAMP	92.17 (0.02)	33.46 (0.17)	92.58 (0.04)	33.03 (0.22)	67.26 (1.25)	24.03 (0.24)	72.48 (0.96)	23.35 (0.11)
	IWAE	87.96 (0.04)	31.86 (0.04)	88.18 (0.03)	32.26 (0.04)	63.93 (1.73)	19.55 (0.67)	73.66 (2.30)	18.44 (0.86)
	VAE-lin-NF	86.04 (0.04)	31.57 (0.02)	85.85 (0.05)	31.74 (0.03)	65.48 (2.76)	17.09 (0.64)	68.80 (3.65)	18.74 (0.68)
	VAE-IAF	88.32 (0.02)	33.52 (0.02)	87.91 (0.02)	32.41 (0.02)	75.31 (1.69)	17.81 (0.73)	76.11 (2.15)	18.42 (0.66)
	β-VAE	93.67 (0.02)	45.15 (0.03)	94.10 (0.03)	40.01 (0.07)	73.13 (1.64)	24.64 (0.90)	76.54 (1.60)	24.78 (0.73)
	β-TC VAE	90.96 (0.02)	45.40 (0.05)	91.91 (0.02)	42.17 (0.07)	65.68 (0.91)	24.14 (0.65)	68.98 (2.67)	25.57 (0.61)
	Factor VAE	86.08 (0.06)	31.38 (0.04)	83.44 (0.05)	31.76 (0.04)	51.02 (1.73)	15.77 (0.60)	60.79 (2.06)	17.56 (0.68)
	InfoVAE-IMQ	86.33 (0.04)	32.48 (0.02)	86.31 (0.06)	32.10 (0.05)	68.17 (2.34)	16.65 (0.80)	71.31 (2.62)	18.10 (0.79)
	InfoVAE-RBF	85.94 (0.03)	32.50 (0.03)	86.12 (0.04)	31.67 (0.03)	66.02 (1.14)	16.22 (0.69)	71.93 (1.91)	18.61 (0.67)
	AAE	93.28 (0.03)	43.93 (0.07)	94.31 (0.03)	40.62 (0.11)	74.19 (3.22)	24.72 (0.75)	80.41 (2.09)	24.76 (0.53)
	MSSSIM-VAE	78.30 (0.03)	20.26 (0.06)	76.54 (0.03)	20.24 (0.04)	49.33 (1.32)	11.70 (0.19)	48.58 (1.37)	11.70 (0.17)
	VAEGAN	92.34 (0.02)	26.56 (0.04)	90.31 (0.03)	29.90 (0.03)	77.29 (1.19)	17.20 (0.45)	79.67 (0.90)	22.23 (0.44)
	AE	93.81 (0.02)	42.15 (0.07)	94.26 (0.03)	40.47 (0.13)	73.55 (0.60)	23.19 (0.52)	77.30 (0.84)	23.18 (0.37)
	WAE-IMQ	93.60 (0.02)	45.89 (0.07)	94.62 (0.03)	41.35 (0.03)	72.33 (2.92)	23.81 (0.61)	78.46 (3.48)	25.09 (0.82)
	WAE-RBF	93.72 (0.02)	43.38 (0.08)	94.51 (0.02)	40.63 (0.08)	74.20 (1.94)	23.70 (0.71)	77.33 (1.92)	24.66 (0.63)
	VQVAE	93.45 (0.02)	42.89 (0.07)	94.63 (0.04)	40.40 (0.09)	72.61 (0.40)	23.85 (0.48)	76.68 (2.36)	23.68 (0.37)
	RAE-L2	94.75 (0.01)	42.76 (0.08)	94.43 (0.03)	40.22 (0.05)	74.07 (0.36)	23.77 (0.54)	78.66 (0.29)	24.84 (0.73)
	RAE-GP	94.10 (0.02)	43.66 (0.07)	94.45 (0.02)	40.93 (0.14)	72.88 (0.52)	24.84 (0.53)	77.66 (1.29)	23.86 (0.32)

: Mean test accuracy of a single layer classifier on the embedding obtained in the latent spaces of each model average on 20 runs. Right: Mean accuracy of 100 k-means fitted on the training embeddings coming from the autoencoders.

Table 3 :

 3 Inception Score (higher is better) and FID (lower is better) computed with 10k samples on the test set. For each model and sampler we report the results obtained by the model achieving the lowest FID score on the validation set.

					ConvNet					ResNet
	Model	Sampler	MNIST	CIFAR10	CELEBA	MNIST	CIFAR10	CELEBA
			FID ↓	IS ↑	FID	IS	FID	IS ↑	FID ↓	IS ↑	FID	IS	FID	IS
	VAE	N GMM	28.5 26.9	2.1 2.1	241.0 235.9	2.2 2.3	54.8 52.4	1.9 1.9	31.3 32.3	2.0 2.1	181.7 179.7	2.5 2.5	66.6 63.0	1.6 1.7
	VAMP	VAMP	64.2	2.0	329.0	1.5	56.0	1.9	34.5	2.1	181.9	2.5	67.2	1.6
	IWAE	N GMM	29.0 28.4	2.1 2.1	245.3 241.2	2.1 2.1	55.7 52.7	1.9 1.9	32.4 34.4	2.0 2.1	191.2 188.8	2.4 2.4	67.6 64.1	1.6 1.7
	VAE-lin NF	N GMM	29.3 28.4	2.1 2.1	240.3 237.0	2.1 2.2	56.5 53.3	1.9 1.9	32.5 33.1	2.0 2.1	185.5 183.1	2.4 2.5	67.1 62.8	1.6 1.7
	VAE-IAF	N GMM	27.5 27.0	2.1 2.1	236.0 235.4	2.2 2.2	55.4 53.6	1.9 1.9	30.6 32.2	2.0 2.1	183.6 180.8	2.5 2.5	66.2 62.7	1.6 1.7
	β-VAE	N GMM	21.4 9.2	2.1 2.2	115.4 92.2	3.6 3.9	56.1 51.7	1.9 1.9	19.1 11.4	2.0 2.1	124.9 112.6	3.4 3.6	65.9 59.3	1.6 1.7
	β-TC VAE	N GMM	21.3 11.6	2.1 2.2	116.6 89.3	2.8 4.1	55.7 51.8	1.8 1.9	20.7 13.3	2.0 2.1	125.8 106.5	3.4 3.7	65.9 59.3	1.6 1.7
	FactorVAE	N GMM	27.0 26.9	2.1 2.1	236.5 234.0	2.2 2.2	53.8 52.4	1.9 2.0	31.0 32.7	2.0 2.1	185.4 184.4	2.5 2.5	66.4 63.3	1.7 1.7
	InfoVAE -RBF	N GMM	27.5 26.7	2.1 2.1	235.2 230.4	2.1 2.2	55.5 52.7	1.9 1.9	31.1 32.3	2.0 2.1	182.8 179.5	2.5 2.5	66.5 62.8	1.6 1.7
	InfoVAE -IMQ	N GMM	28.3 27.7	2.1 2.1	233.8 231.9	2.2 2.2	56.7 53.7	1.9 1.9	31.0 32.8	2.0 2.1	182.4 180.7	2.5 2.6	66.4 62.3	1.6 1.7
	AAE	N GMM	16.8 9.3	2.2 2.2	139.9 92.1	2.6 3.8	59.9 53.9	1.8 2.0	19.1 11.1	2.1 2.1	164.9 118.5	2.4 3.5	64.8 58.7	1.7 1.8
	MSSSIM-VAE	N GMM	26.7 27.2	2.2 2.2	279.9 279.7	1.7 1.7	124.3 124.3	1.3 1.3	28.0 28.8	2.1 2.1	254.2 253.1	1.7 1.7	119.0 119.2	1.3 1.3
	VAEGAN	N GMM	8.7 6.3	2.2 2.2	199.5 197.5	2.2 2.1	39.7 35.6	1.9 1.8	12.8 6.5	2.2 2.2	198.7 188.2	2.2 2.6	122.8 84.3	2.0 1.7
	AE	N GMM	26.7 9.3	2.1 2.2	201.3 97.3	2.1 3.6	327.7 55.4	1.0 2.0	221.8 11.0	1.3 2.1	210.1 120.7	2.1 3.4	275.0 57.4	2.9 1.8
	WAE -RBF	N GMM	21.2 9.2	2.2 2.2	175.1 97.1	2.0 3.6	332.6 55.0	1.0 2.0	21.2 11.2	2.1 2.1	170.2 120.3	2.3 3.4	69.4 58.3	1.6 1.7
	WAE -IMQ	N GMM	18.9 8.6	2.2 2.2	164.4 96.5	2.2 3.6	64.6 51.7	1.7 2.0	20.3 11.2	2.1 2.1	150.7 119.0	2.5 3.5	67.1 57.7	1.6 1.8
	VQVAE	N	28.2	2.0	152.2	2.0	306.9	1.0	170.7	1.6	195.7	1.9	140.3	2.2
		GMM	9.1	2.2	95.2	3.7	51.6	2.0	10.7	2.1	120.1	3.4	57.9	1.8
	RAE-L2	N	25.0	2.0	156.1	2.6	86.1	2.8	63.3	2.2	170.9	2.2	168.7	3.1
		GMM	9.1	2.2	85.3	3.9	55.2	1.9	11.5	2.1	122.5	3.4	58.3	1.8
	RAE -GP	N	27.1	2.1	196.8	2.1	86.1	2.4	61.5	2.2	229.1	2.0	201.9	3.1
		GMM	9.7	2.2	96.3	3.7	52.5	1.9	11.4	2.1	123.3	3.4	59.0	1.8

that the choice of the latent dimension exhibits noteworthy distinguishable trends between 2 distinct groups: AE-based and variational-based methods.

  Table. 4 lists all the implemented models as of June 2022. 1. Training configuration Before launching a model training, one must specify the training configuration that should be used. This can be done easily by instantiating a BaseTrain-erConfig instance taking as input all the hyper-parameters related to the training (number of training epochs, learning rate to apply...). See the full documentation for additional arguments that can be passed to the BaseTrainerConfig.

	from pythae.trainers import BaseTrainerConfig
	# Set up the model configuration
	my_training_config = BaseTrainerConfig(
	output_dir='my_model',
	num_epochs=50,
	learning_rate=1e-3,
	batch_size=200)

2. Model configuration Similarly to the TrainerConfig, the model can then be instantiated with the model configuration specifying any hyper-parameters relevant to the model. Note that each model has its own configuration with specific hyper-parameters. See the online documentation for more details.

Table 5 :

 5 Neural network architecture used for the convolutional networks.

		MNIST	CIFAR10	CELEBA
	Encoder	(1, 28, 28)	(3, 32, 32)	(3, 64, 64)
	Layer 1	Conv(128, 4, 2), BN, ReLU	Conv(128, 4, 2), BN, ReLU	Conv(128, 4, 2), BN, ReLU
	Layer 2	Conv(256, 4, 2), BN, ReLU	Conv(256, 4, 2), BN, ReLU	Conv(256, 4, 2), BN, ReLU
	Layer 3	Conv(512, 4, 2), BN, ReLU	Conv(512, 4, 2), BN, ReLU	Conv(512, 4, 2), BN, ReLU
	Layer 4	Conv(1024, 4, 2), BN, ReLU	Conv(1024, 4, 2), BN, ReLU	Conv(1024, 4, 2), BN, ReLU
	Layer 5	Linear(1024, latent_dim)*	Linear(4096, latent_dim)*	Linear(16384, latent_dim)*
	Decoder
	Layer 1	Linear(latent_dim, 16384)	Linear(latent_dim, 65536)	Linear(latent_dim, 65536)
	Layer 2	ConvT(512, 3, 2), BN, ReLU	ConvT(512, 4, 2), BN, ReLU	ConvT(512, 5, 2), BN, ReLU
	Layer 3	ConvT(256, 3, 2), BN, ReLU	ConvT(256, 4, 2), BN, ReLU	ConvT(256, 5, 2), BN, ReLU
	Layer 4	Conv(1, 3, 2), Sigmoid	Conv(3, 4, 1), Sigmoid	ConvT(128, 5, 2), BN, ReLU
	Layer 5	-	-	ConvT(3, 5, 1), Sigmoid
	*Doubled for VAE-based models

Table 7 :

 7 Inception Score (higher is better) and FID (lower is better) computed with 10k samples on the test set. For each model and sampler we report the results obtained by the model achieving the lowest FID score on the validation set.

					ConvNet					ResNet
	Model	Sampler	MNIST	CIFAR10	CELEBA	MNIST	CIFAR10	CELEBA
			FID ↓	IS ↑	FID	IS	FID	IS ↑	FID ↓	IS ↑	FID	IS	FID	IS
		N	28.5	2.1	241.0	2.2	54.8	1.9	31.3	2.0	181.7	2.5	66.6	1.6
	VAE	GMM VAE	26.9 40.3	2.1 2.0	235.9 337.5	2.3 1.7	52.4 70.9	1.9 1.6	32.3 48.7	2.1 1.8	179.7 358.0	2.5 1.3	63.0 76.4	1.7 1.4
		MAF	26.8	2.1	239.5	2.2	52.5	2.0	31.0	2.1	181.5	2.5	62.9	1.7
	VAMP	VAMP	64.2	2.0	329.0	1.5	56.0	1.9	34.5	2.1	181.9	2.5	67.2	1.6
		N	29.0	2.1	245.3	2.1	55.7	1.9	32.4	2.0	191.2	2.4	67.6	1.6
	IWAE	GMM VAE	28.4 42.4	2.1 2.0	241.2 346.6	2.1 1.5	52.7 74.3	1.9 1.5	34.4 50.1	2.1 1.9	188.8 364.8	2.4 1.2	64.1 76.4	1.7 1.4
		MAF	28.1	2.1	243.4	2.1	52.7	1.9	32.5	2.1	190.4	2.4	64.3	1.7
		N	29.3	2.1	240.3	2.1	56.5	1.9	32.5	2.0	185.5	2.4	67.1	1.6
	VAE-lin NF	GMM VAE	28.4 40.1	2.1 2.0	237.0 311.0	2.2 1.6	53.3 71.1	1.9 1.6	33.1 49.7	2.1 1.9	183.1 296.2	2.5 1.7	62.8 75.6	1.7 1.4
		MAF	27.7	2.1	239.1	2.1	53.4	2.0	32.4	2.0	184.2	2.5	62.7	1.7
		N	27.5	2.1	236.0	2.2	55.4	1.9	30.6	2.0	183.6	2.5	66.2	1.6
	VAE-IAF	GMM VAE	27.0 39.4	2.1 2.0	235.4 330.5	2.2 1.1	53.6 73.0	1.9 1.5	32.2 44.8	2.1 1.9	180.8 322.7	2.5 1.5	62.7 76.7	1.7 1.4
		MAF	26.9	2.1	236.8	2.2	53.6	1.9	30.6	2.1	182.5	2.5	63.0	1.7
		N	21.4	2.1	115.4	3.6	56.1	1.9	19.1	2.0	124.9	3.4	65.9	1.6
	β-VAE	GMM VAE	9.2 14.0	2.2 2.2	92.2 139.6	3.9 3.6	51.7 55.0	1.9 1.9	11.4 20.3	2.1 2.1	112.6 152.5	3.6 3.5	59.3 61.5	1.7 1.7
		MAF	9.5	2.2	100.9	3.5	51.5	2.0	12.0	2.1	120.0	3.6	59.7	1.8
		N	21.3	2.1	116.6	2.8	55.7	1.8	20.7	2.0	125.8	3.4	65.9	1.6
	β-TC VAE	GMM VAE	11.6 18.4	2.2 2.2	89.3 127.9	4.1 4.2	51.8 59.7	1.9 1.8	13.3 28.3	2.1 2.0	106.5 164.0	3.7 3.3	59.3 66.4	1.7 1.5
		MAF	12.0	2.2	95.6	3.6	52.2	1.9	13.7	2.1	116.6	3.4	60.1	1.7
		N	27.0	2.1	236.5	2.2	53.8	1.9	31.0	2.0	185.4	2.5	66.4	1.7
	FactorVAE	GMM VAE	26.9 41.2	2.1 1.9	234.0 338.3	2.2 1.5	52.4 75.0	2.0 1.5	32.7 54.7	2.1 1.8	184.4 316.2	2.5 1.3	63.3 77.7	1.7 1.4
		MAF	26.7	2.2	236.7	2.2	52.7	1.9	32.8	2.1	185.8	2.5	63.4	1.7
		N	27.5	2.1	235.2	2.1	55.5	1.9	31.1	2.0	182.8	2.5	66.5	1.6
	InfoVAE -RBF	GMM VAE	26.7 39.7	2.1 2.0	230.4 327.2	2.2 1.5	52.7 73.7	1.9 1.5	32.3 50.6	2.1 1.9	179.5 363.4	2.5 1.2	62.8 75.8	1.7 1.4
		MAF	25.9	2.1	233.5	2.2	52.2	2.0	30.5	2.1	181.3	2.5	62.7	1.7
		N	28.3	2.1	233.8	2.2	56.7	1.9	31.0	2.0	182.4	2.5	66.4	1.6
	InfoVAE -IMQ	GMM VAE	27.7 40.4	2.1 1.9	231.9 323.8	2.2 1.6	53.7 73.7	1.9 1.5	32.8 49.9	2.1 1.9	180.7 341.8	2.6 1.8	62.3 75.7	1.7 1.4
		MAF	27.2	2.1	232.3	2.1	53.8	2.0	30.6	2.1	182.5	2.5	62.6	1.7
		N	16.8	2.2	139.9	2.6	59.9	1.8	19.1	2.1	164.9	2.4	64.8	1.7
	AAE	GMM VAE	9.3 13.4	2.2 2.2	92.1 144.0	3.8 3.4	53.9 58.2	2.0 1.8	11.1 15.1	2.1 2.1	118.5 145.2	3.5 3.6	58.7 59.0	1.8 1.7
		MAF	9.3	2.2	101.1	3.2	53.8	2.0	11.9	2.1	133.6	3.1	59.2	1.8
		N	26.7	2.2	279.9	1.7	124.3	1.3	28.0	2.1	254.2	1.7	119.0	1.3
	MSSSIM-VAE	GMM VAE	27.2 51.2	2.2 1.9	279.7 355.5	1.7 1.1	124.3 137.9	1.3 1.2	28.8 51.6	2.1 1.9	253.1 372.1	1.7 1.1	119.2 136.5	1.3 1.2
		MAF	26.9	2.2	279.8	1.7	124.0	1.3	27.5	2.1	254.1	1.7	119.5	1.3
		N	8.7	2.2	199.5	2.2	39.7	1.9	12.8	2.2	198.7	2.2	122.8	2.0
	VAEGAN	GMM VAE	6.3 11.2	2.2 2.1	197.5 310.9	2.1 2.0	35.6 54.5	1.8 1.6	6.5 9.2	2.2 2.1	188.2 272.7	2.6 2.0	84.3 88.8	1.7 1.6
		MAF	6.9	2.3	199.0	2.1	36.7	1.8	6.6	2.2	191.9	2.5	84.8	1.7
		N	26.7	2.1	201.3	2.1	327.7	1.0	221.8	1.3	210.1	2.1	275.0	2.9
	AE	GMM	9.3	2.2	97.3	3.6	55.4	2.0	11.0	2.1	120.7	3.4	57.4	1.8
		MAF	9.9	2.2	108.3	3.1	55.7	2.0	12.0	2.1	136.5	3.0	58.3	1.8
		N	21.2	2.2	175.1	2.0	332.6	1.0	21.2	2.1	170.2	2.3	69.4	1.6
	WAE -RBF	GMM	9.2	2.2	97.1	3.6	55.0	2.0	11.2	2.1	120.3	3.4	58.3	1.7
		MAF	9.8	2.2	108.2	3.1	56.0	2.0	11.8	2.2	135.3	3.0	58.3	1.8
		N	18.9	2.2	164.4	2.2	64.6	1.7	20.3	2.1	150.7	2.5	67.1	1.6
	WAE -IMQ	GMM	8.6	2.2	96.5	3.6	51.7	2.0	11.2	2.1	119.0	3.5	57.7	1.8
		MAF	9.5	2.2	107.8	3.1	51.6	2.0	11.8	2.1	130.2	3.0	58.7	1.7
		N (0, 1)	28.2	2.0	152.2	2.0	306.9	1.0	170.7	1.6	195.7	1.9	140.3	2.2
	VQVAE	GMM	9.1	2.2	95.2	3.7	51.6	2.0	10.7	2.1	120.1	3.4	57.9	1.8
		MAF	9.6	2.2	104.7	3.2	52.3	1.9	11.7	2.2	136.8	3.0	57.9	1.8
		N	25.0	2.0	156.1	2.6	86.1	2.8	63.3	2.2	170.9	2.2	168.7	3.1
	RAE-L2	GMM	9.1	2.2	85.3	3.9	55.2	1.9	11.5	2.1	122.5	3.4	58.3	1.8
		MAF	9.5	2.2	93.4	3.5	55.2	2.0	12.3	2.2	136.6	3.0	59.1	1.7
		N	27.1	2.1	196.8	2.1	86.1	2.4	61.5	2.2	229.1	2.0	201.9	3.1
	RAE -GP	GMM	9.7	2.2	96.3	3.7	52.5	1.9	11.4	2.1	123.3	3.4	59.0	1.8
		MAF	9.7	2.2	106.3	3.2	52.5	1.9	12.2	2.2	139.4	3.0	59.5	1.8

Table 8 :

 8 VAMP configurations

			Config			1	2	3	4	5	6	7	8	9	10
			Number of pseudo-inputs (K)	10	20	30	500	100	150	200	250	300	500
	Classification_accuracy	0.84 0.86 0.88 0.90 0.92					VAMP_MNIST_convnet_16				0.66 0.60 0.62 0.64	Clustering_accuracy
		0.02550
	Train time (minutes) Reconstruction_mse	100 200 300 0.02425 0.02450 0.02475 0.02500 0.02525	1	2	3	4		5 run configuration number 6	7	8	9	10	10 2 7 × 10 1 8 × 10 1 9 × 10 1	Generation_fid

Table 9 :

 9 IWAE configurations

			Config		1	2	3	4	5	6	7	8	9	10
			Number of samples (L)	2	3	4	5	6	7	8	9	10	12
						IWAE_MNIST_convnet_16
	Classification_accuracy	0.860 0.865 0.870 0.875 0.880
	Reconstruction_mse	0.0145 0.0150 0.0155
		1	2	3	4	5 run configuration number 6	7		8	9	10

Table 10 :

 10 VAE-lin-NF configurations

	Config		1	2	3	4	5	6	7	8	9	10
	Flow sequence	PPPPP	RRRRR	PRPRP	10P	15P	20P	30P	PRPRPRPRPR	PPP	PRPPRPPPPR
	'P' stands for planar flow -'R' stands for radial flow
	Classification_accuracy	0.5 0.6 0.7 0.8

Table 11 :

 11 VAE-IAF configurations

		Config			1	2	3	4	5	6	7	8	9	10
		hidden size in MADE	32.0	32.0	32.0	32.0	32.0	32.0	32.0	32.0	64.0	128.0
		number hidden units in MADE	2.0	2.0	2.0	2.0	2.0	2.0	4.0	6.0	2.0	2.0
	Classification_accuracy	number of IAF blocks 10.0 0.85 1.0 2.0 5.0 VAE_IAF_MNIST_convnet_16 0.86 0.87 0.88					0.74 0.64 0.66 0.68 0.70 0.72	Clustering_accuracy
		0.0171												2.95 × 10 1
	Train time (minutes) Reconstruction_mse	0.0167 0.0168 0.0169 0.0170 85 90	1	2	3	4	5 run configuration number 6	7	8	9	10	2.75 × 10 1 2.8 × 10 1 2.85 × 10 1 2.9 × 10 1	Generation_fid
					Figure 18: Results on VAE-IAF

Table 12 :

 12 β-VAE configurations

		Config	1	2	3	4	5	6	7	8	9	10
		β		1e -3	1e -2	1e -1	0.5	2	5	10	20	1e 2	1e 3
						BetaVAE_MNIST_convnet_16
	Classification_accuracy	0.2 0.4 0.6 0.8
		0.07
		0.06
	Reconstruction_mse	0.01 0.02 0.03 0.04 0.05
		1	2	3	4	5 run configuration number 6	7	8	9	10

Table 13 :

 13 β-TC-VAE configurations

	Config	1	2	3	4	5	6	7	8	9	10
	β	1e -3	1e -2	1e -1	0.5	1	2	5	10	50	1e 2

Table 14 :

 14 FactorVAE configurations

				Config	1	2	3	4	5	6	7	8	9	10
				γ	1	2	5	10	15	20	30	40	50	100
		0.850					FactorVAE_MNIST_convnet_16			0.5
	Classification_accuracy	0.725 0.750 0.775 0.800 0.825												0.2 0.3 0.4	Clustering_accuracy
		0.024												3.1 × 10 1
	Train time (minutes) Reconstruction_mse	0.018 0.020 0.022 252 254 256	1	2	3	4		5 run configuration number 6	7	8	9	10	2.7 × 10 1 2.8 × 10 1 2.9 × 10 1 3 × 10 1	Generation_fid
					Figure 21: Results on FactorVAE

Table 15 :

 15 InfoVAE configurations

	Config	1	2	3	4	5	6	7	8	9	10
	kernel bandwidth -σ	1e -2	1e -1	0.5	1	1	1	1	1	2	5
	λ	10	10	10	1e -2	1e -1	10	100	100	10	10

Generation_fid

Figure 23: Results on InfoVAE-IMQ Adversarial AE (AAE) Makhzani

Table 16 :

 16 AAE configurations

		Config	1	2	3	4	5	6	7	8	9	10
		α	1e -3	1e -2	1e -1	0.25	0.5	0.75	0.9	0.95	0.99	0.999
		0.935			Adversarial_AE_MNIST_convnet_16
	Classification_accuracy	0.905 0.910 0.915 0.920 0.925 0.930
		0.012
	Reconstruction_mse	0.006 0.008 0.010
		1	2	3	4	5 run configuration number 6	7	8	9	10

Table 17 :

 17 MSSSIM-VAE configurations

		Config			1	2	3	4	5	6	7	8	9	10
		β			1e -2	1e -2	1e -2	1e -1	1e -1	1e -1	1	1	1	1
		window size in MSSSIM	3	5	11	5	3	11	11	5	3	15
		0.8				MSSSIM_VAE_MNIST_convnet_16			0.5
	Classification_accuracy	0.2 0.4 0.6										0.1 0.2 0.3 0.4	Clustering_accuracy
		0.10
	Reconstruction_mse	0.04 0.06 0.08										10 2	Generation_fid
	Train time (minutes)	85 90 95	1	2	3	4	5 run configuration number 6	7	8	9	10

Table 18 :

 18 VAEGAN configurations

	Config	1	2	3	4	5	6	7	8	9	10
	α	0.3	0.5	0.7	0.8	0.8	0.8	0.9	0.9	0.99	0.999
	reconstruction layer (l)	3	3	3	3	2	4	3	3	3	3

Table 19 :

 19 WAE configurations

		Config		1	2	3	4	5	6	7	8	9	10
		kernel bandwidth -σ	1e -2	1e -1	0.5	1	1	1	1	1	2	5
		λ			1	1	1	1e -2	1e -1	1	10	100	1	1
						WAE_MMD-rbf_MNIST_convnet_16
		0.9375											0.74
	Classification_accuracy	0.9250 0.9275 0.9300 0.9325 0.9350											0.71 0.72 0.73	Clustering_accuracy
	Reconstruction_mse	0.005550 0.005575 0.005600 0.005625 0.005650 0.005675											2.2 × 10 1 2.3 × 10 1 2.4 × 10 1 2.5 × 10 1 2.9 × 10 1 2.8 × 10 1 2.7 × 10 1 2.6 × 10 1	Generation_fid
	Train time (minutes)	0.005525 85 90	1	2	3	4	5 run configuration number 6	7		8	9	10	2.1 × 10 1

Table 20 :

 20 VQVAE configurations

		Config	1	2	3	4	5	6	7	8	9	10
		K	128	256	512	512	512	512	512	512	1024	2948
		β	0.25	0.25	0.9	0.1	0.5	0.25	0.75	0.25	0.25	0.25
	Classification_accuracy	0.9275 0.9300 0.9325 0.9350 0.9375 0.9400

Table 21 :

 21 RAE configurations

		Config	1	2	3	4	5	6	7	8	9	10
		β	1e -6	1e -4	1e -3	1e -3	1e -3	1e -3	1e -3	1e -2	1e -1	1
		λ	1e -3	1e -3	1e -6	1e -4	1e -2	1e -1	1	1e -3	1e -3	1e -3
		0.95				RAE_L2_MNIST_convnet_16
	Classification_accuracy	0.90 0.91 0.92 0.93 0.94
	Reconstruction_mse	0.006 0.007 0.008
		1	2	3	4	5 run configuration number 6	7	8	9	10

The full documentation can be found at https://pythae.readthedocs.io/en/latest/.

Some models may actually have additional parameters in their intrinsic structure e.g. a VQVAE learns a dictionary of embeddings, a VAMP learns the pseudo-inputs, a VAE-IAF learns the auto-regressive flows. Nonetheless, since we work on images, the number of parameters remains in the same order of magnitude.

The training setting (curves, configs ...) can be found at https://wandb.ai/benchmark_team/ trainings while detailed experimental set-up is available in Appendix. C

See the whole results at https://wandb.ai/benchmark_team/reconstructions

We used the implementation of https://github.com/bioinf-jku/TTUR

We used the implementation of https://github.com/openai/improved-gan

See the whole results at https://wandb.ai/benchmark_team/generations

See the whole results at https://wandb.ai/benchmark_team/classifications

See the whole results at https://wandb.ai/benchmark_team/clustering

MSSSIM-VAE was removed from this plot for visualisation purposes.

Some models may actually have additional parameters in their intrinsic structure e.g. a VQVAE learns a dictionary of embeddings, a VAMP learns the pseudo-inputs, a VAE-IAF learns the auto-regressive flows. Nonetheless, since we work on images, the number of parameters remains in the same order of magnitude.

Acknowledgments and Disclosure of Funding

The research leading to these results has received funding from the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute) and reference ANR-10-IAIHU-06 (Agence Nationale de la Recherche-10-IA Institut Hospitalo-Universitaire-6). This work was granted access to the HPC resources of IDRIS under the allocation AD011013517 made by GENCI (Grand Equipement National de Calcul Intensif).