Abstract

Many methods for statistical inference and generative modeling rely on a probability divergence to eectively compare two probability distributions. This divergence should be carefully chosen as its theoretical properties and practical implications strongly aect the performance of the associated algorithm. In that context, the Wasserstein distance, which emerges from optimal transport (OT) theory, has been an interesting choice due to its theoretical guarantees. However, it suers from important computational and statistical limitations, which have severly hindered its use to problems with large amounts of high-dimensional data.

In recent years, several workarounds have been proposed to alleviate these issues thus enable the use of OT in machine learning (ML) applications. In particular, the Sliced-Wasserstein distance (SW) is an alternative OT metric, which compares two distributions by computing the expected Wasserstein distance between their one-dimensional linear projections. SW has been increasingly popular since it can oer signicant computational advantages over the Wasserstein distance, especially on large-scale problems, and has been successfully applied in many practical tasks, such as classication, Bayesian inference, and implicit generative modeling. Nevertheless, there has been little work regarding the theoretical guarantees of such SW-based methods. This thesis further explores the use of the Sliced-Wasserstein distance in modern statistical and ML problems, with a twofold objective: on the one hand, provide new theoretical insights to understand in depth the empirical behavior of existing SW-based algorithms; on the other hand, design novel tools inspired by SW to extend its applicability and oer increased scalability.

We rst focus on the estimators obtained by minimizing SW, which form the basis of several recently proposed generative modeling methods. We prove a set of asymptotic properties of these estimators, as well as a central limit theorem which characterizes their asymptotic distribution and exhibits a dimension-free convergence rate. We also develop a novel likelihood-free inference technique, SW-ABC, by incorporating SW in the Approximate Bayesian Computation framework. We prove asymptotical guarantees on the convergence of the posterior distribution returned by SW-ABC, and illustrate the advantages of our algorithm in practice, on synthetic data and an image denoising problem.

By denition, SW is an expectation over random projections, which is intractable in general and commonly estimated with a simple Monte Carlo procedure. This approximation induces an error that can potentially degrade the performance of SW-based algorithms on high-dimensional settings. To overcome this issue, we introduce the Generalized Sliced-Wasserstein distances (GSW), which extends the denition of SW by considering nonlinear projections of the distributions. We study the metric axioms of GSW using the theory of the Radon transform, and show that GSW can yield better results than SW on generative modeling applications.

We then adopt another perspective to address the issues due to the Monte Carlo approximation: we leverage the concentration of measure phenomenon, which states under mild assumptions that one-dimensional linear projections of a high-dimensional random vector are approximately Gaussian. Based on this result, we develop a simple deterministic approximation for SW, which can lead to a signicant computational time reduction over Monte Carlo. We derive nonasymptotical guarantees for our methodology under a weak dependence condition, validate them on synthetic experiments, and illustrate the proposed approximation on image generation.

Inspired by the growing success of SW, new instances of sliced probability divergences (SPDs) have been deployed in statistical and ML applications, but their theoretical implications have not been well established. To bridge this gap, we introduce the rst general denition of SPDs and investigate their statistical and topological properties. Our theoretical analysis sheds light to the consequences of slicing: in particular, we prove that the sample complexity of any SPD does not depend on the data dimension, but can be impacted by an additional error term due to the Monte Carlo approximation. We then apply our general results to specic SPDs in order to gain a better understanding of them.

Résumé

De nombreuses méthodes d'inférence statistique et de modélisation générative reposent sur une divergence pour comparer de façon pertinente deux distributions de probabilité. Cette divergence doit être choisie avec soin puisque ses propriétés théoriques et répercussions pratiques aectent fortement les performances de l'algorithme en question. Dans ce contexte, la distance de Wasserstein, qui découle de la théorie du transport optimal, est un choix intéressant de par ses garanties théoriques. Cependant, elle présente d'importantes limites computationnelle et statistique qui l'empêchent de traiter ecacement de grandes quantités de données à haute dimension.

Plusieurs méthodes visant à atténuer ces problèmes ont été proposées ces dernières années, permettant ainsi l'utilisation du transport optimal pour l'apprentissage automatique. En particulier, la distance de Sliced-Wasserstein (SW) est une métrique alternative qui compare deux distributions en calculant la distance de Wasserstein moyenne entre leurs projections linéaires unidimensionnelles. SW est de plus en plus utilisée en raison de sa capacité à fournir des avantages calculatoires signicatifs par rapport à la distance de Wasserstein, surtout pour les problèmes à grande échelle, et a fait ses preuves dans de nombreuses tâches pratiques, telles que la classication, l'inférence bayésienne et la modélisation générative implicite. Néanmoins, peu de travaux ont étudié les garanties théoriques des méthodes basées sur SW. Cette thèse explore plus en profondeur l'utilisation de SW pour des problèmes modernes de statistique et d'apprentissage automatique, avec un double objectif : d'une part, apporter de nouvelles connaissances théoriques permettant une compréhension approfondie des algorithmes basés sur SW; d'autre part, concevoir de nouveaux outils inspirés de SW an d'élargir son champ d'applications et orir une meilleure scalabilité.

Nous nous focalisons d'abord sur les estimateurs obtenus en minimisant SW, qui constituent la base de plusieurs méthodes de modélisation générative. Nous prouvons un ensemble de propriétés asymptotiques pour ces estimateurs, ainsi qu'un théorème central limite qui caractérise leur distribution asymptotique avec un taux de convergence indépendent de la dimension des données. Nous développons également une nouvelle technique d'inférence qui n'utilise pas la vraisemblance, appelée SW-ABC, en incorporant SW dans un algorithme de type Approximate Bayesian Computation. Nous prouvons des garanties asymptotiques sur la convergence de la distribution a posteriori calculée par SW-ABC, et illustrons les avantages de notre algorithme en pratique, sur des données synthétiques et un problème de débruitage d'image.

Par dénition, SW est une espérance sur des projections aléatoires, qui est dicile à calculer en général et couramment estimée par une méthode de Monte Carlo simple. Cette approximation entraîne une erreur susceptible de dégrader les performances des algorithmes basés sur SW en grande dimension. An de pallier à ce problème, nous introduisons les distances de Sliced-Wasserstein généralisées (SWG), en étendant la dénition des SW de façon à y inclure les projections non linéaires des distributions. Nous étudions dans quelle mesure SWG vérient les axiomes d'une distance en utilisant la théorie autour de la transformation de Radon, et montrons que SWG peut fournir de meilleurs résultats que SW dans des applications de modélisation générative.

Nous adoptons ensuite une autre perspective pour résoudre les problèmes dus à l'estimation par Monte Carlo : nous tirons parti du phénomène de concentration de mesure qui arme, selon des hypothèses modérées, que les projections linéaires unidimensionnelles d'une variable aléatoire à grande dimension suivent une loi presque gaussienne. A partir de ce résultat, nous proposons une nouvelle formule simple et déterministe pour calculer SW bien plus rapidement qu'avec Monte Carlo. Nous prouvons des garanties non-asymptotiques pour notre méthodologie sous une condition de dépendance faible, les validons sur des expériences synthétiques, puis utilisons notre nouvelle approximation pour la génération d'images.

Motivées par le succès grandissant de SW, de nouvelles divergences sliced (DS) ont été déployées pour des applications statistiques et d'apprentissage, mais leurs implications théoriques restent méconnues. Nous introduisons alors la première dénition des DS et étudions leurs propriétés statistiques et topologiques. Notre analyse théorique met en lumière les conséquences du slicing : en particulier, nous prouvons que l'erreur d'approximation de tout DS par des échantillons ne dépend pas de la dimension des données, mais peut être aectée par l'approximation par Monte Carlo. Nous appliquons ensuite nos résultats à des DS spéciques pour mieux les comprendre.

dernières années. Roland, je te suis avant tout très reconnaissante d'avoir accepté de diriger ma thèse et d'avoir veillé à ce qu'elle se déroule dans les meilleures conditions possibles du début à la n. Merci également pour ton soutien et tous les retours que tu as pris le temps de me donner dès que j'avais besoin d'un avis, que ce soit pour ma recherche ou certaines formalités administratives. Alain, merci de m'avoir poussée à être la plus minutieuse possible dans mes preuves mathématiques, mes expériences numériques, mais aussi mes notations avec le fameux chier def.tex. J'admire sincèrement ta rigueur scientique, dont les exigences m'ont permis de sortir de ma zone de confort et d'améliorer mon esprit critique. Je tiens également à te remercier d'avoir été aussi présent et impliqué, notamment en période de rush : ta patience et tes conseils m'ont considérablement aidée à apprivoiser mon stress à l'approche de certaines deadlines. Umut, I am forever grateful to you for oering me the opportunity to pursue a PhD under your supervision. Thank you for teaching me so much about conducting a research project and for introducing me to many great people. Thank you, also, for sharing my obsession with details and my love for awkward jokes. More importantly, thank you for encouraging and pushing me, because you knew I could do it long before I did, while caring about my feelings. I couldn't have dreamt of a better mentor.

Merci à mes rapporteurs de thèse, Gabriel Peyré et Nicolas Courty, d'avoir pris le temps de relire attentivement mon manuscrit. I sincerely thank Julie Delon, Laetitia Chapel, Marco Cuturi and Justin Solomon, for accepting to be part of the jury. It has been an honor to defend my thesis in front of all of you, whose work and passion I greatly admire. Merci Marco d'avoir également accepté, un an et demi plus tôt, de compter parmi les membres du jury pour ma soutenance à mi-parcours ; et bien avant cela, de m'avoir accompagnée dans la procédure pour partir en stage de master au Japon, et de m'y avoir fait découvrir le transport optimal.

Many thanks to all the people I have been very fortunate to collaborate with during my PhD: Soheil Kolouri, Gustavo K. Rohde, Valentin De Bortoli, Lénaïc Chizat, Shahin Shahrampour, Tudor Manole, Ruben Ohana and Pierre E. Jacob. Valentin, je garde un bon souvenir de notre collaboration, qui nous a été particulièrement fructueuse. J'espère que nous aurons d'autres occasions de travailler ensemble à l'avenir. Soheil, I feel lucky to have worked with you on several research projects. Your immense enthusiasm and creativity made our collaboration even more enjoyable and have been a great source of inspiration for me. Pierre, merci d'avoir inspiré et relu avec intérêt mes premiers articles de thèse, puis d'avoir accepté de faire partie du jury de ma soutenance à mi-parcours.

Je te suis très reconnaissance pour nos discussions autour de la recherche académique, ainsi que pour tes conseils qui m'ont poussée à me dépasser. Collaborer avec toi fut une belle manière de conclure ma thèse, et j'espère que nous pourrons un jour creuser les 7 autres idées que nous avons eues.

Merci à tous mes collègues du laboratoire LTCI de Télécom Paris, qui ont participé à créer une ambiance de travail conviviale, et à la chaire DSAIDIS d'avoir nancé mon contrat doctoral. J'ai une pensée particulière à tous les membres et visiteurs du bureau B412 et de son annexe, grâce auxquels j'ai pu démarrer ma thèse dans un environnement amical et bienveillant. Merci à Amaury et Nidham pour ces soirées passées autour d'une IPA à discuter de la thèse sans ltre je suis heureuse de vous avoir rencontrés et de vous compter parmi mes amis.

Je tiens à exprimer ma gratitude envers Claire Boyer et Julie Josse pour m'avoir oert l'opportunité de travailler avec elles après mon doctorat.

Enn, je souhaite adresser mes sincères remerciements à Romain Laroche, avec qui j'ai écrit mon tout premier article scientique à Montréal, quelques mois avant de commencer ma thèse à Paris. Merci Romain de m'avoir redonné conance en ma capacité à faire de la recherche.

J'aimerais remercier mes amis, à qui je n'ai pas forcément parlé de ma thèse en détail, mais qui y ont contribué en vivant avec moi des moments de joie et d'insouciance.

Merci Karen et Anthony, les voisins de la rue Barrault chez qui je pouvais me réfugier pour déguster des tagliatelles au citron, discuter plus ou moins savamment de psychologie et rire (puis me battre avec Edward) pendant une partie de jeu de société.

Merci à mes amis de l'Ensimag, sur lesquels je pouvais compter pour aller prendre des bières et décompresser : ne perdez jamais le goût de la fête. Maxime, mon acolyte emo/indie, merci de partager ma passion pour les concerts, et donc aussi mon désespoir quand tout s'est arrêté. J'ai hâte de retourner au Primavera avec toi et le reste de la team pour se laisser à nouveau emporter. Dr. Bentriou et Dr. Mezghani, je vous remercie de m'avoir montré la voie, même si Dr. Mezghani perdait souvent la sienne le dimanche matin. Merci d'avoir partagé votre sagesse et de me faire autant rire.

Bien entendu, je n'oublie pas d'où je viens. Merci à Chloé, mon amie d'enfance également docteure, qui comprenait donc parfaitement ce que je traversais. Tant que la tradition Bookmas persiste, j'ai conance en l'avenir. Edward, merci d'être présent dans ma vie depuis aussi longtemps et de toujours répondre à l'appel pour en fêter les grandes étapes. Je ne me lasserai pas de tes gâteaux, de nos débats sur la société ni de nos blagues, surtout quand elles ne font rire que nous.

Merci Guillaume pour tout le réconfort que tu as su m'apporter, même dans les périodes diciles. Je me sens extrêmement chanceuse d'avoir rencontré un être aussi sensible et lumineux que toi.

Manon, je n'oublierai pas toutes ces soirées pendant lesquelles nous avons pu vivre des instants parfois absurdes, souvent hilarants : Paris était une fête avec toi. Merci pour ta présence rassurante, ton écoute et tes analyses pertinentes qui m'ont permis plus d'une fois d'ouvrir les yeux.

Merci Canelle de n'avoir jamais raté une occasion de me hyper et de m'avoir communiqué ta belle énergie quand j'en avais besoin. J'attends avec impatience le prochain volet de nos aventures sur la route, mais en attendant, continuons de nous retrouver à Paris ou à Nantes pour écouter des podcasts et débattre du sens de la vie tout en buvant des tisanes aux plantes mystérieuses. Mes derniers remerciements s'adressent à ma famille, en particulier à ma s÷ur, mon frère, mon père et ma mère. Mes accomplissements n'auraient jamais été possibles sans les sacrices et la générosité de mes parents : merci d'avoir tout entrepris, quitte à vous oublier vous-mêmes, pour favoriser mon épanouissement et me permettre de faire de longues études. Et bravo maman pour ton troisième doctorat.

Contents Summary of Notations

Throughout this thesis, we consider a probability space (Ω, F, P) with associated expectation operator E, on which all the random variables are dened.

B(Y)

Borel set of Y P(Y)

Set of probability distributions supported on Y P p (Y)

Set of distributions of P(Y) with nite p'th moment Fourier transform of the probability distribution µ f µ

Push-forward measure of the probability measure µ by the measurable function f x ∼ µ

x is a sample drawn from the probability distribution µ

x Euclidean norm of the vector x x, y

Euclidean inner-product between the vectors x and y card(X)

Cardinal of the set X diam(X) Diameter of the compact set X Chapter 1

Introduction

The signicant breakthroughs achieved in the eld of machine learning (ML) over the last decades have, in turn, raised many technical questions. In particular, building a successful machine learning algorithm remains an important problem, which is even more challenging given the evergrowing collection of data in various forms as well as the limits of available computational resources: the success of an algorithm is not only measured by the accuracy of the returned results, but also by its computational requirements and processing speed. As a result, many eorts have been made in machine learning research to improve existing methods so that, for example, they produce more accurate results, require less hyperparameters tuning, or execute faster, especially on large datasets.

The performance of machine learning algorithms depends on many design elements taking into account dierent factors, such as the formalism of the problem, the nature of data, and the computational resources at hand. This thesis focuses on a crucial component for many algorithms, that is the probability divergence used to compare two distributions. To further clarify our motivation, we consider a concrete problem, namely generative modeling, and illustrate the importance of having an appropriate divergence within this framework.

1.1

Motivation: Probability Divergences and Generative

Models

The goal of generative modeling is to reproduce new data points, by learning a probabilistic model that best describes how the input dataset is generated. Specically, generative models receive as input a set of n ∈ N * observations, generally assumed to be independent and identically distributed (i.i.d.) samples from an unknown probability distribution µ , and aim at learning µ . Many dierent techniques can be employed to reach this objective, thus dening a variety of generative models.

A common practice consists in tting µ with a parametric distribution: one denes a statistical model, M = {µ θ : θ ∈ Θ}, and nds the optimal parameters θ ∈ Θ such that µ θ ∈ M yields the most accurate approximation of µ . For instance, Gaussian mixture models (GMMs) propose to t µ with a mixture of K ∈ N * Gaussian distributions (Figure 1.1). The parameters θ then refer to the mean and covariance matrix of each Gaussian, and mixing coecients {π k } K k=1 , where π k is the probability for the observations {x i } n i=1 to be sampled from the k'th Gaussian distribution. While GMMs dene simple and powerful generative models with many applications, their performance can be limited by the fact that a mixture of Gaussians does not necessarily describe well the structure of the observations.

To address the drawbacks of such explicit generative models, e.g. GMMs, the implicit generative modeling (IGM) methodology builds on the recent advances in deep learning and has attracted considerable attention within the ML community. The strategy, illustrated in Figure 1.2, consists in rst choosing a distribution ζ from which it is easy to sample (e.g., a standard Gaussian distribution), then apply nonlinear dierentiable operators which dene a (deep) neural network with parameters θ, denoted by T θ . The transformed distribution then corresponds to µ θ . Once the neural network is trained, it can map z ∼ ζ, received as input, to new data points. The most common methods include generative adversial networks (GANs, [START_REF] Goodfellow | Generative adversarial nets[END_REF]) and auto-encoders [START_REF] Kingma | Auto-encoding variational bayes[END_REF]. Despite their ability to generate samples of high quality, such approaches inherit from the drawbacks of deep learning: designing and training a neural network can be highly dicult and time-consuming. More precisely, dening an implicit generative model amounts to choosing, for example, the nature of the nonlinear operators, the number of hidden layers, the latent distribution ζ or its objective function. To this day, why some architectures perform better than others is still an open question. On the other hand, the performance of implicit generative models also depends on how the similarity between µ θ ∈ M and µ is measured. We expand on this aspect in the remainder of this section, since the general intention of this thesis is to study a specic distance between probability distibutions. As we described above, IGM denes a methodology to address the general problem of density tting: the goal is to nd a parametric distribution µ θ that best approximates the true underlying distribution µ . In practical terms, this task requires having a tool that eectively measures how close one distribution is to another. Probability divergences are suited for the job, since they compute a certain notion of distance between two distributions. More formally, let X be a Polish space, i.e. a topological space that is separable and completely metrizable, and denote by P(X) the class of probability distributions supported on X; we will use the notation D : P(X) × P(X) → R + ∪ {+∞} to refer to a generic divergence used to compare any two distributions in P(X). The objective of density tting is a minimum distance estimation problem, written as min θ∈Θ D(µ θ , µ ) .

In implicit generative modeling, µ θ in (1.1) corresponds to the output of a deep neural network. Since there are several options available for the choice of D, which will be presented later on, the question becomes, what probability divergence should one use in their generative models? .

As a rst answer, we provide general informal guidelines by enumerating some of the main desirable properties for D.

(P1) D is a proper distance function, i.e. it veries the metric axioms.

(P2) D satises a certain notion of continuity, which can be described as follows: if a sequence of distributions (µ k ) k∈N gets closer to µ as k grows to innity, then D(µ k , µ) shrinks to 0.

(P3) θ → D(µ θ , µ ) is dierentiable and has a unique minimizer.

(P4) D(µ, ν) can be eectively estimated from samples drawn from µ and ν.

We will give the formal statement for the metric axioms (P1) and the continuity (P2) in Section 2.1. (P4) is motivated by the fact that in most problems in data sciences, one has access to nite sets of observations instead of the underlying probability distributions.

Therefore, any distribution ξ is only accessible through an empirical approximation, which usually corresponds to a discrete measure ξn computed over the sequence of n ∈ N * random variables {Y i } n i=1 i.i.d. from ξ, given by

ξn = 1 n n i=1 δ Y i ,
with δ Y being the Dirac measure with mass on the point Y . In that case, the chosen probability divergence D must have specic practical features in order to appropriately handle the empirical approximations, which will be discussed in Section 2.1.

Overall, properties (P1) to (P4) reect that D should be a suciently regular and computationally practical divergence, so that the associated generative model is easy to use, somewhat robust and able to capture relevant information regarding the geometry of the problem.

Classical Probability Divergences

We now present traditional choices of probability divergences and explain their implications in terms of the criteria listed in the previous section. If D satises (P2), we say that D is weakly continuous and report the associated mathematical denition in Section 2.1.

f -divergences. These divergences were introduced in [START_REF] Rényi | On Measures of Entropy and Information[END_REF] and also referred to as ϕ-divergences, Ciszár divergences [START_REF] Ciszár | Information-type measures of dierence of probability distributions and indirect observation[END_REF] or Ali-Silvey distances [START_REF] Ali | A General Class of Coecients of Divergence of One Distribution from Another[END_REF]. They include widely known instances which have played a crucial role in various areas such as probability theory, statistics and information theory. For example, the χ 2 -divergence has commonly been used for adaptive importance sampling [START_REF] Cornebise | Adaptive methods for sequential importance sampling with application to state space models[END_REF]; the Kullback-Leibler divergence (KL) is related to the notions of mutual information and relative entropy [Cover and Thomas, 2006, Section 2.3] and is the core component of variational inference [START_REF] Blei | Variational Inference: A Review for Statisticians[END_REF] and more recently, of variational auto-encoders [START_REF] Kingma | Auto-encoding variational bayes[END_REF].

In the context of generative modeling, f -divergences suer from important drawbacks. First, they do not metrize weak convergence, which might severely aect the quality of the comparisons made. This is illustrated by the fact that KL evaluated between µ and ν is innite when the two distributions µ and ν are supported on domains that do not overlap. On the other hand, approximating a f -divergence, denoted by D f , from a set of samples is not computationally simple: the straightforward estimator D f (μ n , νn ) obtained by plugging the empirical measures in place of µ and ν does not converge to D f (µ, ν) as n increases in general. Several workarounds have then been developed, but they solve provably hard problems, induce a sample complexity with slow convergence rates, or rely on strong structural assumptions: see the discussion in [Rubenstein et al., 2019].

Integral Probability Metrics (IPMs). Introduced by [START_REF] Müller | Integral Probability Metrics and Their Generating Classes of Functions[END_REF], the class of IPMs provides important advantages over f -divergences for generative modeling: they satisfy almost all metric axioms, are weakly continuous under mild assumptions, and can easily be estimated from the available samples. More details on these aspects are provided in Section 2.2.

A popular example of IPMs is given by the Maximum Mean Discrepancy (MMD, [START_REF] Gretton | A Kernel Two-Sample Test[END_REF]), whose attractive analytical and computational properties make it an interesting probability divergence for many applications, especially statistical hypothesis testing [START_REF] Gretton | A Kernel Two-Sample Test[END_REF] and generative modeling [START_REF] Li | Generative Moment Matching Networks[END_REF][START_REF] Dziugaite | Training Generative Neural Networks via Maximum Mean Discrepancy Optimization[END_REF][START_REF] Sutherland | Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy[END_REF][START_REF] Bi«kowski | Demystifying MMD GANs[END_REF][START_REF] Arbel | Maximum Mean Discrepancy Gradient Flow[END_REF]. However, the training and performance of these generative models are highly sensitive to the kernel function dening MMD and its parametrization. For example, the Gaussian RBF kernel is a very traditional choice, but its bandwith parameter determines the statistical eciency of the associated MMD and is not easy to tune appropriately [START_REF] Li | Generative Moment Matching Networks[END_REF].

Besides, its derivatives decay exponentially which cause important stability issues when training deep neural networks, as discussed in [START_REF] Arbel | Maximum Mean Discrepancy Gradient Flow[END_REF].

A second important instance of IPMs corresponds to the Wasserstein distance of order 1 [Villani, 2008, Theorem 5.10]. This metric also falls into the category of optimal transport metrics, which is presented next.

Optimal transport divergences. Optimal transport (OT) is a mathematical theory on a specic optimization problem, which has been extensively studied and leveraged in various applied elds such as economics, combinatorial optimization and more recently, data sciences. Specic formulations of the OT problem dene a family of powerful probability divergences: the Wasserstein distances. These metrics, which will be formally presented in Section 2.4, are able to capture key information for comparing two distributions µ and ν, since they rely on a cost function that carries relevant geometric properties of the supports of µ and ν. If the cost function c is the p'th power of the Euclidean distance, i.e. for any x, y ∈ R d , c : (x, y) → xy p with p ∈ [1, +∞), then the resulting Wasserstein distance is known as the Wasserstein distance of order p and denoted by W p . This divergence satisfy all metric axioms and is weakly continuous when evaluated on the space of probability distributions with nite p'th moment [Villani, 2008, Chapter 6]. Therefore, the Wasserstein distance of order p can eectively compare any two distributions even when their supports do not overlap, as opposed to f -divergences, and does not require tedious hyperparameter tuning, unlike MMD. However, it is well known that Wasserstein distances suer from important computational and statistical limitations, especially in high dimensions. Indeed, their computation is expensive, apart in some special settings presented in Section 2.4.1, for example when comparing two probability distributions supported on R. Let us assume the following typical scenario in machine learning: one would like to compare two distributions µ and ν supported on R d , but only observe n samples drawn from them. In general, Wasserstein distances are not analytically available but can be estimated with approximate solvers, which tend to have a super-cubic cost in practice and executes in O(n 3 log(n)) in the worst case. This implies that a single evaluation of Wasserstein distances is computationally demanding, especially on large-scale datasets.

Besides, the Wasserstein distance computed from the empirical approximations μn and νn has been shown to converge to the true value, i.e. the Wasserstein distance between µ and ν, with a rate in O(n -1/d ): see our discussion in Section 2.4.2. Therefore, for high values of the ambient dimension d, the estimates computed from n samples are reasonably accurate provided that n is suciently large, which induces an important computational complexity according to the previous paragraph.

Hence, the computational and statistical limitations of Wasserstein distances have considerably prevented their application in data sciences for a long time, in particular in generative modeling, which was studied solely from a theoretical perspective [START_REF] Bassetti | On minimum Kantorovich distance estimators[END_REF]. Nevertheless, in recent years, number of studies have introduced various methods to alleviate the practical issues of Wasserstein distances and thus managed to expand the domain of applicability of OT. These techniques dene the active research topic of Computational Optimal Transport [START_REF] Peyré | Computational Optimal Transport: With Applications to Data Science[END_REF], and their development was made possible by the progress in optimization and large-scale machine learning. For instance, Wasserstein distances have recently been incorporated within the IGM framework, leading to the formulation of novel eective models [START_REF] Arjovsky | Wasserstein Generative Adversarial Networks[END_REF], Bousquet et al., 2017[START_REF] Gulrajani | Improved training of Wasserstein GANs[END_REF][START_REF] Tolstikhin | Wasserstein Auto-Encoders[END_REF]. [START_REF] Kolouri | Optimal mass transport: Signal processing and machine-learning applications[END_REF]).

The eld of computational optimal transport also gave rise to the formulation of alternative metrics to the Wasserstein distance, such as Sinkhorn divergences, which emerge from the regularization of the OT problem [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF], and the Sliced-Wasserstein distance. We dedicate the next section to the latter distance, as it constitutes the main focus of this thesis, and will discuss the former in Section 2.5.

1.3

Focus on the Sliced-Wasserstein Distance

While regularized OT and Sinkhorn divergences have strongly help OT theory gain immense interest from the machine learning community, another alternative to classical OT has become increasingly popular in the last few years: the Sliced-Wasserstein distance (SW).

In this section, we explain the denition of SW and its practical implications; the formal statements will be given in Section 2.6. We then discuss related work to review the existing theoretical properties and applications of SW, as well as motivate why this metric is the central object of this thesis. In what follows, SW p with p ∈ [1, +∞) denotes the Sliced-Wasserstein of order p (Denition 2.9).

SW was rst introduced by [START_REF] Rabin | Wasserstein Barycenter and Its Application to Texture Mixing[END_REF] and [START_REF] Bonneel | Sliced and Radon Wasserstein barycenters of measures[END_REF] as a practical alternative to the Wasserstein distance to speed up the computation of barycenters of measures. The idea behind SW is to oer computational eciency by leveraging the analytical form of the Wasserstein distance between univariate distributions: SW compares two multivariate probability distributions by rst, obtaining a family of representations in R for each distribution through projections, then computing the expected value of the Wasserstein distance between these univariate representations. We illustrate the Sliced-Wasserstein distance in Figure 1.3 and provide its denition in the caption of that gure.

The expectation that denes SW does not admit an analytical expression in general and is thus commonly estimated with a simple Monte Carlo average based on L ∈ N * samples. In other words, computing SW amounts to solving a nite number of onedimensional OT problems, which can conveniently be done in closed-form. This approximation technique has a complexity in O Ldn + Ln log(n) , thus can provide signicant computational benets over Wasserstein distances.

Many methods building on the Sliced-Wasserstein distance have been developed to address various applied problems in a computationally ecient way, such as in machine learning, imaging and statistics. Barycenters of measures based on SW have successfully been used on several image processing tasks, i.e. color transfer, texture synthesis and texture mixing [START_REF] Rabin | Wasserstein Barycenter and Its Application to Texture Mixing[END_REF][START_REF] Bonneel | Sliced and Radon Wasserstein barycenters of measures[END_REF]. Very recently, these barycenters have been extended to the case where SW compares more than two measures [START_REF] Cohen | Sliced Multi-Marginal Optimal Transport[END_REF].

Novel kernel functions based on SW have also been proposed, and their benets have been illustrated in topological data analysis and pattern recognition tasks, e.g., classication and clustering [START_REF] Kolouri | Sliced-Wasserstein Kernels for Probability distributions[END_REF][START_REF] Carrière | Sliced Wasserstein Kernel for Persistence Diagrams[END_REF].

Lastly, an important body of literature is dedicated to applying SW in generative modeling applications: in [START_REF] Karras | Progressive growing of GANs for improved quality, stability, and variation[END_REF], SW serves as a score to evaluate the performance of GANs; [START_REF] Kolouri | Sliced Wasserstein distance for Learning Gaussian Mixture Models[END_REF] build a novel GMM which learns the parameters of Gaussian distributions by minimizing SW; nally, SW forms the basis of several new IGM models, including auto-encoders and GANs [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF][START_REF] Liutkus | Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diusions[END_REF][START_REF] Wu | Sliced Wasserstein Generative Models[END_REF], Kolouri et al., 2019b[START_REF] Dai | Sliced Iterative Normalizing Flows[END_REF].

On the theoretical side, SW has been shown to satisfy the metric axioms [Bonnotte, 2013, Proposition 5.1.2], is always bounded above by the Wasserstein distance [Bonnotte, 2013, Proposition 5.1.3], and both metrics are equivalent when computed between compactly supported probability measures [Bonnotte, 2013, Theorem 5.1.5]. The equivalence between the Wasserstein distance and SW has further been studied in a very recent study [START_REF] Bayraktar | Strong equivalence between metrics of Wasserstein type[END_REF]. A mathematical analysis of gradient ows based on SW is provided in [Bonnotte, 2013, Chapter 5] and has been used to derive the theoretical guarantees of the IGM methodology proposed in [START_REF] Liutkus | Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diusions[END_REF]. Finally, recent work observed the statistical benets of SW in practice, and consequently, derived a concentration inequality to bound |SW 2 (μ n , νn ) -SW 2 (µ, ν)| with high probability for the specic case where µ, ν are Gaussian [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF].

Hence, the Sliced-Wasserstein distance has established itself as a powerful practical metric for data sciences, due to its connection to OT, computational eciency and exibility. Besides, SW shares certain properties with the Wasserstein distance, while providing statistical advantages on very specic settings. The contributions to the theoretical analysis of SW are nevertheless limited, which contrasts with the availability of numerous empirical studies. This imbalance forms the starting point of this thesis.

Outline and Contributions

Computational optimal transport has become a very active and popular eld within the machine learning community in recent years, and the growing literature on the subject has demonstrated that SW is a practical tool with broad application. Motivated by the reported empirical success of SW and its increasing popularity, this thesis further studies this metric, with particular focus on its theoretical guarantees, relevance to approximate inference and generative modeling, and extensions. The broad objective is to conduct a thorough analysis of the theoretical and empirical implications of SW, which helps unlock its full potential on modern machine learning problems. More precisely, our contributions revolve around the following research directions.

Objective 1. While the empirical performance of SW has been analyzed on a wide range of practical tasks, there has been little work regarding its theoretical guarantees.

This implies that most SW-based methods are not suciently theoretically grounded.

Our rst goal is then to bridge this gap by investigating the theoretical properties of SW and their repercussions in practice.

Objective 2. Since the literature on SW is quite recent, we believe that there are other applied problems where this metric nds relevance. We thus explore how SW can be used to design novel methods for tasks that have not been addressed in prior work, such as Bayesian inference.

Objective 3. For the sake of providing a balanced analysis, we examine the known major limitation of SW, caused by its Monte Carlo approximation: the computational eciency induced by this method is oset by an approximation error, which can be important depending on the problem at hand. We further illustrate this issue and develop new techniques to mitigate it, by leveraging existing tools from other elds.

Objective 4. Our last objective aims at better understanding an essential component of SW, that is the slicing operation: we step back from the OT paradigm and study from a theoretical perspective the consequences of slicing any divergence other than the Wasserstein distance. This analysis will allow us to broaden the reach of SW, as well as sharpen its theoretical analysis.

We now present an overview of the organization of this thesis: for each chapter, we explain the motivation, present related work, and summarize our key ndings which are in line with the aforementioned objectives.

Chapter 3: Asymptotic Guarantees for Learning Generative Models with the Sliced-Wasserstein Distance

Let us consider the minimum distance estimation (MDE, Wolfowitz [1957], Basu et al.

[2011]) problem, formally dened as

θn = argmin θ∈Θ D(μ n , µ θ ) , (1.1)
where D denotes a divergence between probability measures, Θ is the parameter space, µ θ is a probability measure indexed by θ ∈ Θ, and μn = n -1 n i=1 δ Y i is the empirical distribution of a set of i.i.d. observations {Y i } n i=1 . MDE has been extremely useful in statistical inference to infer the parameters of a distribution whose analytical expression is unknown [START_REF] Basu | Statistical Inference: The Minimum Distance Approach[END_REF], and have inspired the formulation of several IGM strategies. In that context, optimal transport metrics have become increasingly popular due to their attractive theoretical properties: the minimum Wasserstein estimator [START_REF] Bassetti | On minimum Kantorovich distance estimators[END_REF][START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF], obtained by replacing D in (1.1) with W p , forms the basis of popular IGM algorithms [START_REF] Arjovsky | Wasserstein Generative Adversarial Networks[END_REF][START_REF] Genevay | GAN and VAE from an optimal transport point of view[END_REF][START_REF] Tolstikhin | Wasserstein Auto-Encoders[END_REF]. Motivated by its practical success, the theoretical properties of this estimator have been studied [START_REF] Bousquet | From optimal transport to generative modeling: the VEGAN cookbook[END_REF][START_REF] Liu | Approximation and convergence properties of generative adversarial learning[END_REF] and very recently, [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF] have derived a set of asymptotic properties, including the asymptotic distribution of the estimator when the distributions are supported on R.

Due to the computational limitations induced by W p , the computational complexity of minimum Wasserstein estimators rapidly becomes excessive with the increasing problem dimension. To avoid this problem, several practical alternatives to the Wasserstein distance have been proposed, and in particular, the Sliced-Wasserstein distance has been an increasingly popular metric, including in IGM [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF][START_REF] Liutkus | Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diusions[END_REF][START_REF] Wu | Sliced Wasserstein Generative Models[END_REF], Kolouri et al., 2019b]. However, since the theoretical properties of these estimators had not been established, the techniques based on them are not sufciently theoretically grounded. Summary of our contributions in Chapter 3. To further motivate the use of SW in statistical inference, we investigate the asymptotic properties of the minimum Sliced-Wasserstein estimators, which are obtained by replacing D in (1.1) with SW p . Our theoretical contributions are summarized below.

1. We prove that convergence under SW implies weak convergence of probability measures on general domains.

2. We show, under some assumptions, that minimum SW estimators exist, are measurable, and are consistent in the sense that as the number of observations n increases, in well-specied models, the estimates will converge to the parameters that generated the observed dataset. Similar consistency guarantees hold for misspecied models.

3. We nally derive a central limit theorem which characterizes the asymptotic distribution of minimum SW estimators, and establishes a convergence rate of √ n for any nite dimension.

Our work is inspired by [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF] and the adaptation of their techniques was made possible by the identication of novel properties regarding the topology induced by SW: for example, we established for the rst time that convergence in SW implies weak convergence of probability measures, which generalizes the results given in [START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF]. Besides, our CLT is stronger than the analogous one derived in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF] for minimum Wasserstein estimators, since ours is not restricted to one-dimensional data, and our convergence rate in √ n is valid for any dimension.

Our theoretical ndings are supported with experiments that we conducted on synthetic and real data. We rst consider a classical statistical inference problem, where the statistical models are characterized by a Gaussian or a multidimensional α-stable distribution. In both models, the experiments validate our consistency and CLT results. We also illustrate that, as expected, the minimum SW estimators have signicantly better computational properties as compared to the minimum Wasserstein estimators, especially on high-dimensional problems. Finally, we consider the deep generative model in [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF] and conduct an empirical analysis on the MNIST dataset, which conrms the consistent behavior of minimum SW estimators in IGM applications.

Chapter 4: Approximate Bayesian Computation with the Sliced-Wasserstein Distance

We consider the problem of estimating the posterior distribution of some model parameters θ ∈ Θ given n data points y 1:n = (y 1 , . . . , y n ). By the Bayes' theorem, this distribution has a closed-form expression which depends on the likelihood π(y 1:n |θ). For many statistical models of interest, π(y 1:n |θ) cannot be numerically evaluated in a reasonable amount of time, which prevents the application of classical likelihood-based approximate inference methods.

Nevertheless, in various settings, it is possible to generate data from the likelihood given any parameter value θ. This generative setting gave rise to the popular likelihoodfree framework for approximate inference, called Approximate Bayesian Computation [START_REF] Tavaré | Inferring Coalescence Times From DNA Sequence Data[END_REF][START_REF] Beaumont | Approximate Bayesian Computation in Population Genetics[END_REF], which has proven useful in various practical applications, e.g. in ecology [START_REF] Wood | Statistical inference for noisy nonlinear ecological dynamic systems[END_REF] and biology [START_REF] Tanaka | Using Approximate Bayesian Computation to Estimate Tuberculosis Transmission Parameters From Genotype Data[END_REF]. ABC approximates the exact posterior π(θ|y 1:n ) from the parameter values for which the synthetic data z 1:m generated from the likelihood are close enough to the observations y 1:n .

Closeness is usually measured with a discrepancy measure between the two datasets reduced to some summary statistics (e.g., empirical mean or empirical covariance). The quality of the approximate posterior distribution highly depends on these summaries, and nding relevant statistics is a non-trivial and tedious task.

Recently, discrepancy measures that view data sets as empirical probability distributions to eschew the construction of summary statistics have been proposed for ABC.

Examples include the Kullback-Leibler divergence [START_REF] Jiang | Approximate Bayesian Computation with Kullback-Leibler Divergence as Data Discrepancy[END_REF], maximum mean discrepancy [START_REF] Park | K2-ABC: Approximate Bayesian Computation with Kernel Embeddings[END_REF], and Wasserstein distance (WABC, [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF]).

While the Wasserstein distance seems like a relevant choice of discrepancy in that context thanks to its strong theoretical properties, its computational and statistical issues can strongly aect the performance of WABC applied to high-dimensional data.

Summary of our contributions for Chapter 4. Motivated by the computational eciency of SW and its successful performance in generative settings, we develop a novel framework for likelihood-free approximate Bayesian inference, which estimates the posterior by retaining the parameter values for which

SW p (μ n , νm ) ≤ ε , (1.2)
where μn denotes the empirical distributions of the observations y 1:n , νm is the empirical distribution of the samples z 1:n drawn from the likelihood, and ε > 0 is a tolerance threshold. This results in a novel ABC method, called Sliced-Wasserstein ABC (SW-ABC), which does not require choosing summary statistics. Besides, SW-ABC is more ecient than WABC on high-dimensional settings, thanks to the computational and statistical benets of SW.

We show that SW-ABC comes with guarantees on the convergence of the resulting posterior, under two asymptotic regimes.

1. On the one hand, we prove that for a xed set of observations y 1:n , the posterior approximated by SW-ABC converges to the true posterior as ε goes to 0, under specic assumptions on the density used to generate synthetic data.

2. On the other hand, we study the SW-ABC posterior when the value of ε is kept xed and the number of observations grows. We show that, as n goes to +∞, the approximate posterior converges to the prior distribution on θ restricted to a specic subset of Θ that depends on ε.

We then illustrate on a synthetical problem the superior empirical performance of SW-ABC against existing ABC techniques based on other divergences. We also demonstrate the exibility and computational advantages of our methodology by designing a novel algorithm for image denoising, which corresponds to a combination of SW-ABC with a widely used technique for this task, Non-Local Means [START_REF] Buades | A non-local algorithm for image denoising[END_REF].

Chapter 5: Generalized Sliced Wasserstein Distances

By denition, the Sliced-Wasserstein distance measures the dissimilarity between µ, ν ∈ P(R d ) by comparing linear projections of µ and ν along all possible directions on S d-1 . These projections correspond to the push-forward measures u µ and u ν for any u ∈ S d-1 , and are actually closely related to the Radon transform [Rabin et al., 2012, Proposition 6], which is widely used in tomography [START_REF] Radon | Uber die bestimmug von funktionen durch ihre integralwerte laengs geweisser mannigfaltigkeiten[END_REF][START_REF] Helgason | The Radon transform on Rn[END_REF]. SW does not admit an analytical formula in general and is thus commonly estimated by Monte Carlo: in practice, one approximates the expectation over u ∼ σ in (2.20) with an average over a nite number of directions {u l } L l=1 , where for l ∈ {1, . . . , L}, u l ∼ σ.

Previous empirical studies have reported that this Monte Carlo strategy might degrade the performance of SW-based algorithms on high-dimensional settings because of the induced approximation error, and propose to change the nature of the projections to overcome this problem. For instance, in [START_REF] Rowland | Orthogonal Estimation of Wasserstein Distances[END_REF][START_REF] Wu | Sliced Wasserstein Generative Models[END_REF], SW is estimated with a Monte Carlo average based on a nite number of orthogonal projection directions. An alternative OT metric called the maximum Sliced-Wasserstein distance is introduced in [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF], and extends SW by replacing the expectation with a maximum operator so that one retains the most informative projection direction. The information returned by a direction u ∈ S d-1 is measured by W p (u µ, u ν): the larger this Wasserstein distance, the more informative u. [START_REF] Paty | Subspace Robust Wasserstein distances[END_REF] generalizes this idea by considering k-dimensional projections of µ, ν with k ∈ {1, . . . , d}: the goal is then to nd the most informative subspace on which µ and ν are being projected. While these methods reduce the computational cost by requiring a lower number of projections, they incur an additional cost due to the resolution of a non-convex optimization problem over manifolds.

Summary of our contributions in Chapter 5. In this chapter, we take an alternative route to alleviate the ineciencies caused by the Monte Carlo approximation of SW, by assuming that the linear nature of the projections might not guarantee an ecient evaluation of SW. Indeed, in very high-dimensional settings, the data often lives in a thin manifold, so the number of randomly chosen linear projections required to capture the structure of the data distribution grows very quickly.

We introduce a novel class of divergences between probability distributions, called generalized Sliced-Wasserstein distances (GSW). GSW are dened as SW, except they compute dierent types of projections: to compare µ and ν, one chooses a function g u : R d → R dened for any u ∈ R q (q ∈ N * ), and collect the pushforward measures (g u ) µ, (g u ) ν for any u ∈ R q . GSW is then dened as the expected value of the Wasserstein distance between these one-dimensional representations. As the name suggests, GSW generalizes the concept behind SW: when g u = u , its denition boils down to the Sliced-Wasserstein distance. We also use these non-linear projections to extend the maximum Sliced-Wasserstein distance and introduce the maximum generalized Sliced-Wasserstein distances (max-GSW).

Analogously to (max-)SW, our denition of (max-)GSW is connected with the Radon transform: by using the theory of the generalized Radon transform (GRT, [START_REF] Beylkin | The inversion problem and applications of the generalized Radon transform[END_REF]), we identify some regularity conditions g u needs to satisfy to ensure that the resulting generalized distance is well-dened. We also prove that GSW and max-GSW verify all metric axioms if and only if the GRT they rely on is injective; otherwise, they are pseudo-metrics. This result helps us identify useful instances of g u that guarantee the injectivity of the associated GRT.

Then, we demonstrate that GSW and max-GSW can outperform SW and max-SW in several generative modeling applications, with both synthetic and real data: the inherent non-linearity of the one-dimensional representations used in (max-)GSW seems to capture the complex structure of high-dimensional distributions with much less projections.

Besides, to ensure an automatic tuning of g u , we propose to dene it as a neural network.

This scheme brings practical advantages and an interesting perspective on adversarial generative modeling, showing that such algorithms contain an implicit stage for learning projections with dierent cost functions than ours.

Chapter 6: Fast Approximation of the Sliced-Wasserstein Distance Using Concentration of Random Projections

The motivations of this chapter are the same as in Chapter 5: although SW has been shown to oer important computational and theoretical advantages over the Wasserstein distance, its practical performance can be limited by the error induced by the commonly used Monte Carlo approximation. To ensure that this approximation error is reasonably small, one might need to choose a large number of projections L, which inevitably increases the computational complexity of SW. We thus aim at developing an alternative method to overcome this issue.

We adopt a dierent perspective to approximate SW by leveraging concentration results on random projections: under relatively mild conditions, the typical distribution of low-dimensional projections of high-dimensional random variables is close to some Gaussian law [Sudakov, 1978, Diaconis and[START_REF] Diaconis | Asymptotics of Graphical Projection Pursuit[END_REF]. This result has recently been illustrated with a bound in terms of the Wasserstein distance [Reeves, 2017, Theorem 1]: let {X i } d i=1 be a sequence of real random variables with distribution µ d , such that X 1 , . . . , X d are independent with nite fourth-order moments; then,

E[W 2 2 (u µ, N µ ) 2 ]
goes to zero as d increases, where N µ denotes a univariate Gaussian distribution whose variance depends on µ d , and the expectation is taken with respect to a Gaussian variable u. This result has very recently been used to bound the maximum-sliced distance between any probability measure and its Gaussian approximation [START_REF] Goldt | The Gaussian equivalence of generative models for learning with shallow neural networks[END_REF].

Summary of our contributions in Chapter 6. We develop a novel technique that approximates SW with a simple deterministic formula, which builds on [Reeves, 2017, Theorem 1]. As opposed to Monte Carlo, our methodology does not rely on a nite set of random projections, thus eliminates the need of tuning the hyperparameter L and can lead to a signicant computational time reduction.

The formulation of our approximate SW is supported by the following ndings.

1. We dene an alternative SW whose projection directions are drawn from the same Gaussian distribution as in [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF], instead of uniformly on S d-1 . We establish its relation with the original SW, and in particular, we prove that the two distances are equal when p = 2.

2. We use this alternative SW and [Reeves, 2017, Theorem 1] to bound the absolute dierence between SW applied to any two probability measures µ d , ν d on R d and the Wasserstein distance between the univariate Gaussians N µ d , N ν d . We explain why the mean parameters of µ d and ν d should necessarily be zero for the absolute dierence to decrease as d grows.

3. We show that the requirement on the mean parameters is not a practical problem, by proving the following result: SW between µ d , ν d can be equivalently written as the sum of the dierence between their means and the SW between the centered versions of µ d , ν d .

Based on the aforementioned results, we introduce a novel estimate of SW, dened as

SW 2 2 (µ d , ν d ) = 1 d m µ d -m ν d 2 + W 2 2 (N μd , N νd ) , (1.3)
where for ξ ∈ {µ d , ν d }, m ξ is the mean parameter of ξ and ξ denotes the centered version of ξ. Since the Wasserstein distance between Gaussian distributions admits a closed-form solution, (1.3) is very easy to compute, and faster than the Monte Carlo estimate obtained with a large number of projections.

We derive nonasymptotical guarantees on the error induced by our approach: we dene a weak dependence condition that is weaker than the one in [START_REF] Doukhan | Probability and moment inequalities for sums of weakly dependent random variables, with applications[END_REF], which is a notion commonly used in statistics, and prove that under this

condition, |SW 2 (µ d , ν d ) -SW 2 (µ d , ν d )| goes to zero as d grows to +∞.
The nonasymptotical guarantees of our approximate SW, as well as its computational eciency, are then validated with experiments conducted on synthetic data. We nally leverage our theoretical insights to design a novel adversarial framework for a typical generative modeling problem, namely image generation. As compared to generative models based on SW estimated with Monte Carlo, our framework produces images of higher quality with further computational benets.

Chapter 7: Statistical and Topological Properties of Sliced Probability Divergences

Recent years have witnessed the formulation of novel sliced probability divergences (SPDs), such as Sliced Gromov-Wasserstein [START_REF] Vayer | NeurIPS 2019 -Thirty-third Conference on Neural Information Processing Systems[END_REF] or Sliced-Cramér [Kolouri et al., 2020a], due to the success of the Sliced-Wasserstein distance in generative modeling. We identify two reasons why slicing a divergence is benecial. First, some probability divergences are only dened to compare measures supported on one-dimensional spaces, for instance the Cramér distance [START_REF] Cramér | On the composition of elementary errors[END_REF]. The slicing operation thus extends these divergences to multivariate distributions [START_REF] Knop | Cramer-Wold Auto-Encoder[END_REF], Kolouri et al., 2020a]. Then, slicing leverages the computational advantages available in one dimension to dene divergences achieving computational eciency on multivariate settings [START_REF] Rabin | Wasserstein Barycenter and Its Application to Texture Mixing[END_REF][START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF][START_REF] Paty | Subspace Robust Wasserstein distances[END_REF], Kolouri et al., 2019b[START_REF] Vayer | NeurIPS 2019 -Thirty-third Conference on Neural Information Processing Systems[END_REF].

Even though various sliced divergences have successfully been deployed in practical applications, their theoretical properties have not yet been well understood. Indeed, the literature on such divergences has largely been devoted to the study of SW. Besides, some properties of SW have only been characterized for specic settings, in particular its statistical benets observed in practice [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF][START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF].

Summary of our contributions in Chapter 7. We conduct a theoretical analysis on sliced probability divergences, in order to bridge the gap between theory and practice and gain insight on what the slicing operation itself is bringing. To this end, we formulate the rst general denition of the class of SPDs, which includes existing instances in the literature and enables us to adopt a general point of view. Specically, we consider a generic base divergence ∆ between one-dimensional probability measures, and dene its sliced version, denoted by S∆, which operates on multivariate settings.

We rst prove several results on the topology induced by S∆, whose statements can be summarized as follows.

1. If ∆ is a metric, so is S∆. In other words, slicing preserves the metric properties.

2. If the convergence in ∆ implies the weak convergence of measures (or conversely), then slicing preserves this property, i.e. the convergence in S∆ implies the weak convergence of measures (or conversely).

3. If ∆ is an integral probability metric, ∆ and S∆ are strongly equivalent under specic sucient conditions which we identify.

We also study the statistical properties of S∆.

4. We show that the sample complexity of S∆ is proportional to the sample complexity of ∆ for one-dimensional measures. Therefore, the sample complexity of any SPDs does not depend on the dimension d.

5. We also derive a bound on the error made when estimating divergences with Monte Carlo, which is the most common practice: we prove that this approximation scheme induces an additional variance term in the complexity of computing the sliced divergence.

1.5. List of Publications 29 Hence, our results demonstrate that while SPDs can oer important statistical benets thanks to the dimension-free rate in their sample complexity, the Monte Carlo strategy induces an error that might aect the overall complexity of computing SPDs in practice, especially on high-dimensional settings. Our results conrm the recent empirical observations reported in [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF], which motivated Chapters 5 and 6, and provide a better understanding for them.

We demonstrate the applicability of our general theoretical results by applying them to specic instances of SPDs. For example, we establish a novel result on the topology induced by the Sliced-Cramér distance. We also derive a sample complexity result for SW which has never been shown before, under dierent assumptions on the measures to be compared.

We then introduce the sliced version of Sinkhorn divergences and demonstrate its statistical and computational advantages. Indeed, by combining our general results with recent work [Genevay et al., 2019, Mena andNiles-Weed, 2019], we derive the sample complexity of Sliced-Sinkhorn divergences, and obtain rates which, as opposed the sample complexity of Sinkhorn divergences, do not depend on d nor on the regularization parameter ε. We also show that this sliced divergence improves the worst-case computational complexity bounds of Sinkhorn divergences in R d .

Finally, we support our theory by conducting numerical experiments on synthetic and real data: we consider examples of sliced divergences (Sliced-MMD, Sliced-Wasserstein distance, Sliced-Sinkhorn divergences) and provide an empirical analysis of their topological, statistical or computational properties, which agrees with our theoretical results.

List of Publications

The contributions that we described in Section 1.4 led to the following publications. Chapter 2

Technical Background

This chapter provides a set of technical results to specify the notions addressed in Chapter 1, which will be useful for the remainder of the thesis. We rst describe some of the theoretical properties that help evaluate the advantages of a probability divergence when used in generative modeling applications, specically its metric axioms, metrization of weak convergence and sample complexity.

Then, we examine these properties for specic instances of probability divergences, namely the general class of Integral Probability Metrics, with a special focus on the Maximum Mean Discrepancy, and the Wasserstein distance. To properly introduce this latter metric, we give some background on optimal transport theory.

Finally, we explain the computational and statistical limitations of the Wasserstein distance to motivate the use of alternatives, including Sinkhorn divergences and the Sliced-Wasserstein distance. We dene these two divergences, which rely on fundamentally dierent approaches, and present their known benets over classical optimal transport metrics.

General Properties on Probability Divergences

In this section, we give the formal statement of desirable properties for a probability divergence in the context of generative modeling, starting with the metric axioms.

Denition 2.1 (Metric axioms). Let X be a Polish space and consider a divergence D on the space of probability distributions P(X). D is a metric if it takes nite values, i.e. D : P(X) × P(X) → R + , and satises the following axioms.

1. Symmetry: For any µ, ν ∈ P(X), D(µ, ν) = D(ν, µ).

Triangle inequality:

For any µ, ν, ξ ∈ P(X), D(µ, ν) ≤ D(µ, ξ) + D(ξ, ν).

Identity of indiscernibles:

For any µ, ν ∈ P(X), D(µ, ν) = 0 if and only if µ = ν.

If D only veries some of these axioms, then it is said to be a pseudo-metric.

To clarify the property of continuity described in (P2) (page 17), we dene the weak convergence of measures, an important topological notion that characterizes a certain type of convergence for sequences of probability distributions [Billingsley, 1999, Problem 1.11, Chapter 1]. Denition 2.2 (Weak convergence). Let X be a Polish space, (µ k ) k∈N a sequence of probability measures supported on X, and µ ∈ P(X). We say that µ k converges weakly to a probability measure µ on X, and write (µ k ) k∈N w -→ µ (or µ k w -→ µ), if for any continuous and bounded function

f : X → R, lim k→+∞ X f dµ k = X f dµ .
Another type of convergence for probability distributions can be characterized through a probability divergence: consider a Polish space X and D : P(X) × P(X) → R + ∪ {+∞}; using the same notations as in Denition 2.2, we say that (µ k ) k∈N converges to µ under

D if lim k→∞ D(µ k , µ) = 0.
Depending on the choice of D and X, convergence under D can imply weak convergence, and conversely. If weak convergence implies convergence under D, i.e. property (P2) (page 17) is true, then we say that D is weakly continuous. If the reverse implication also holds, i.e. convergence under D is equivalent to weak convergence, and D is additionally a metric, then D is said to metrize the weak convergence in P(X).

Finally, property (P4) (page 17) is directly related to the statistical eciency of a divergence. Indeed, consider that one has access to sets of samples drawn from unknown or intractable distributions, which is typically the case in data sciences. Then, from a practical point of view, a probability divergence D is useful in that context if it is able to appropriately handle the empirical approximations. Above all, this means that the evaluation of D from samples, or equivalently from empirical measures, must be easy to implement. Then, the error induced by this empirical approximation should be suciently small: given two unknown distributions µ, ν ∈ P(X) and their respective empirical instantiations μn , νn , the deviation |D(μ n , νn ) -D(µ, ν)| must shrink to 0 as n increases with a reasonably fast convergence rate. This rate is commonly referred to as the sample complexity of D.

Integral Probability Metrics

We now study the family of Integral Probability Metrics, introduced by [START_REF] Müller | Integral Probability Metrics and Their Generating Classes of Functions[END_REF]

P F (Y) = {µ ∈ P(Y) : ∀f ∈ F, Y |f (y)| dµ(y) < +∞} .
The Integral Probability Metric associated with F, denoted by γ F , is dened for any µ, ν ∈ P F (Y) as

γ F (µ, ν) = sup f ∈F Y f (y)d(µ -ν)(y) .
(2.1)

If µ or ν does not belong to P F (Y), we set γ F (µ, ν) = +∞.

IPMs oer several theoretical guarantees which suggest that their deployment in generative modeling applications is relevant. First, they are pseudo-metrics [START_REF] Sriperumbudur | On Integral Probability Metrics, φ-Divergences and Binary Classication[END_REF]: they are non-negative, symmetric, verify the triangle inequality and for any µ ∈ P F (Y), γ F (µ, µ) = 0. Besides, γ F metrizes weak convergence, provided that the span of F is dense in the space of continous and bounded functions on Y endowed with the supremum norm [Ambrosio et al., 2005, Section 5

γ F (μ n , νn ) = sup f ∈F 1 n n i=1 f (x i ) -f (y i ) .
(2.2)

Then, γ F (μ n , νn ) converges to γ F (µ, ν) as n grows to +∞, under some mild assumptions on µ, ν and their empirical instances [Sriperumbudur et al., 2012, Lemma 3.1]. Nevertheless, for arbitrary F, the empirical estimate γ F (μ n , νn ) is not always simple to compute nor converges to γ F (µ, ν) suciently fast. An important example of IPMs which admits a consistent and statistically ecient empirical estimator is the Maximum Mean Discrepancy (MMD, [START_REF] Gretton | A Kernel Two-Sample Test[END_REF]), which is dened below.

Denition 2.4 (Maximum Mean Discrepancy). Let H be a reproducing kernel Hilbert space (RKHS) for real-valued functions on a measurable space Y, and F be the unit ball in H. Then, γ F (Denition 2.3) denes the MMD in RKHS: for any µ, ν ∈ P F (Y),

MMD(µ, ν; F) = sup f ∈F, i.e. f :Y→R, f H ≤1 Y f (y)d(µ -ν)(y) (2.3)
By [Gretton et al., 2012, Lemma 6], the above denition can equivalently be written in terms of the kernel k associated to H,

MMD 2 (µ, ν; F) = Y×Y k(x, x )d(µ ⊗ µ)(x, x ) + Y×Y k(y, y )d(ν ⊗ ν)(y, y ) -2 Y×Y k(x, y)d(µ ⊗ ν)(x, y) (2.4)
MMD has been shown to metrize weak convergence when Y is a compact space or Y = R d , under specic conditions on its dening kernel k [START_REF] Sriperumbudur | Hilbert Space Embeddings and Metrics on Probability Measures[END_REF]. One of the most popular kernels is the Gaussian (RBF) kernel, dened for any x, y ∈ R d as k(x, y) = exp -x-y 2 2σ 2

, where σ > 0 is called the bandwidth parameter. According to the aforementioned general result, MMD based on the Gaussian kernel metrizes weak convergence on compact spaces.

When F is the unit ball of an RKHS, the solution of (2.2) is unique and available in closed form, thus denes an empirical estimator for MMD that is easy to implement [Sriperumbudur et al., 2012, Theorem 2.4]. This estimate, denoted by MMD(•, •; F), is consistent and exhibits a rate that does not depend on the data dimension d: by [Sriperumbudur et al., 2012, Corollary 3.5],

MMD(μ n , νn ; F) -MMD(µ, ν; F) = O(n -1/2 ) .

(2.5)

The analytical properties of MMD have been especially useful for statistical hypothesis testing [Gretton et al., 2008, Fukumizu et al., 2008, Gretton et al., 2012] and generative modeling [START_REF] Li | Generative Moment Matching Networks[END_REF][START_REF] Dziugaite | Training Generative Neural Networks via Maximum Mean Discrepancy Optimization[END_REF][START_REF] Sutherland | Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy[END_REF][START_REF] Bi«kowski | Demystifying MMD GANs[END_REF][START_REF] Arbel | Maximum Mean Discrepancy Gradient Flow[END_REF]. However, these studies also demonstrate that the theoretical guarantees and practical implications induced by MMD are strongly aected by the choice of its kernel function. For instance, the constant in the sample complexity (2.5) depends on sup x∈Y k(x, x) [Sriperumbudur et al., 2012, Corollary 3.5]. The inuence of the kernel function on the training and performance of MMD-based IGM methods has not been fully understood, and the study of this aspect is an active research topic: while [START_REF] Arbel | Maximum Mean Discrepancy Gradient Flow[END_REF] provided some answers by identifying the settings where the gradient estimators used in MMD GANs are unbiased, it is still unclear why some kernel choices yields better results than others in their experiments.

We conclude this section by presenting another important instance of IPMs: when

F = {f : Y → R : f Lip ≤ 1}, where f Lip = sup x,y∈Y,x =y |f (x) -f (y)| x -y , (2.6)
γ F is then known as the Wasserstein distance of order 1 [Villani, 2008, Theorem 5.10]. This metric is directly related to the eld of optimal transport theory, as we explain in the next sections.

Optimal Transport

We provide the basics of optimal transport theory in order to give a better understanding of the roots of the Wasserstein distance.

OT was rst formulated by Monge [1781] and denes a mathematical formalism which, intuitively, aims at nding a way to move the probability mass from one distribution to another with least eort. This eort is quantied by means of a cost function, which operates as follows: denote by µ ∈ P(X) the source distribution and by ν ∈ P(Y) the target distribution, where X and Y are two Polish spaces. Then, c : X×Y → R + ∪{∞} denes the function that returns for any (x, y) ∈ X × Y the cost of transporting x to y. This cost function is typically chosen as a distance on X × Y, so that it evaluates how far x and y are from each other: the smaller c(x, y), the closer x is to y, so the least eort.

On the other hand, the idea of transporting one distribution to another is described by the notion of push-forward. To illustrate how this mathematical operator works, we consider the specic case of discrete measures: let f : X → Y be a continuous map, and ξ ∈ P(X) supported over n points, i.e. ξ = n -1 n i=1 δ x i . Then, the push-forward operator associated to f , denoted by f , takes ξ as input and returns a probability distribution on Y characterized by

f ξ = 1 n n i=1 δ f (x i ) .
In other words, f moves ξ ∈ P(X) towards a new distribution on Y, by applying f to its support points {x i } n i=1 . This operation can be generalized to continuous distributions: in this case, f moves each elementary mass of the input distribution. We give the formal denition of the push-forward operator in Denition 2.5. Denition 2.5 (Push-forward operator). Let X, Y be two Polish spaces, f : X → Y a continuous map, and ξ ∈ P(X). We denote by f : P(X) → P(Y) the push-forward operator associated to f . The evaluation of f at ξ yields a probability distribution supported on Y, denoted by f ξ and characterized by the following two properties, which are equivalent.

For any measurable set

A in Y, f ξ(A) = ξ(f -1 (A)) where f -1 (A) = {x ∈ X : f (x) ∈ A} .

For any

g ∈ C(Y), Y h(y)df ξ(y) = X g • f (x)dξ(x) .
f ξ can be referred to as the push-forward measure of ξ by f .

The OT problem as formulated by Monge consists in nding the map T : X → Y that transports the source distribution µ ∈ P(X) to its target ν ∈ P(Y) (via the push-forward operator T ) with minimal total cost (measured by the cost function c). Formally, the corresponding optimization problem is given by min T X c x, T (x) dµ(x) s.t. T µ = ν .

(2.7)

Monge's formulation leads to a nonconvex optimization problem (2.7), which boils down to a combinatorial assignment problem in the discrete case [Peyré and Cuturi, 2019, Section 2.2]. Such problems are dicult to solve in general, and feasible solutions do not always exist.

To overcome the issues induced by the resolution of Monge's OT problem, Kantorovich [1942] introduced a relaxed version of (2.7) such that the transportation becomes probabilistic: the probability mass of any source point can be split into smaller masses which are then assigned to dierent target points, whereas the Monge problem performs a one-to-one assignment. To allow for mass splitting, Kantorovich proposes to use couplings instead of deterministic transport maps, i.e. the feasible set of solutions now corresponds to the set of joint distributions on X×Y whose rst and second marginals are given by µ and ν respectively. The condition on the marginals reects the conservation of total probability mass when carried from µ to ν.

Π(µ, ν) = π ∈ P(X × Y) : for any measurable sets A ⊂ X, B ⊂ Y, π(A × Y) = µ(A), π(X × B) = ν(B) .
The solution to (2.8) has been shown to always exist, provided that the following conditions are satised.

(i) X and Y are compact metric spaces, and the cost function c is continuous [Santambrogio, 2015, Theorem 1.4] or lower semi-continuous and bounded from below [Santambrogio, 2015, Theorem 1.5],

(ii) or alternatively, X and Y are Polish spaces and c is lower semi-continuous [Santambrogio, 2015, Theorem 1.7].

These theoretical guarantees provides a signicant advantage over the Monge problem, which can be ill-posed. Furthermore, Kantorovich's formulation led to the denition of powerful probability divergences, namely Wasserstein distances.

Wasserstein Distances

Wasserstein distances dene a class of probability divergences which compare two distributions by solving a transport problem, and have gradually become a useful tool for many other applications than OT theory. In this section, we introduce the mathematical denition of Wasserstein distances, specify some of their key theoretical properties and present two special settings where they are easily computable. Then, we explain the main limitations of these OT metrics, which makes them impractical when deployed in modern machine learning applications.

Denition and elementary properties

We start by dening the Wasserstein distance of order p between any two distributions.

In what follows, we will assume for ease of reading that the compared distributions are supported on the same space, i.e. X = Y.

Denition 2.6 (Wasserstein distances). Let X be a Polish space equipped with a distance ρ, and p ∈ [1, +∞). The Wasserstein distance of order p is dened for any µ, ν ∈ P(X) as

W p (µ, ν) = inf π∈Π(µ,ν) X×X ρ(x, y) p dπ(x, y) 1/p .
(2.9)

One can easily see from Denition 2.6 that Wasserstein distances correspond to specic instances of the Kantorovich transport problem recalled in (2.8). Hence, comparing two distributions µ and ν via the Wasserstein distance amounts to solving a transport problem where the source and target distributions are given by µ and ν; or conversely, by ν and µ, since this order has no importance in Kantorovich's formulation.

By denition, Wasserstein distances leverage the information captured by the cost function c on the geometry of the supports of µ and ν in order to optimally move the probability mass from µ to ν. The comparisons made via this OT metric are then conceptually more powerful than with traditional divergences, e.g. f -divergences which perform pointwise comparisons of the probability mass [START_REF] Rényi | On Measures of Entropy and Information[END_REF].

The benets of Wasserstein distances are further conrmed by the following theoretical results: denote by P p (X) the set of probability measures on X with nite p'th moment, i.e.

P p (X) = µ ∈ P(X) :

X ρ(x 0 , x) p dµ(x) < +∞, for some x 0 ∈ X .

Then, W p is a metric on P p (X) [Villani, 2008, Chapter 6] which metrizes the weak convergence, i.e. the weak convergence of probability measures supported on P p (X) is equivalent to convergence under W p [Villani, 2008, Theorem 6.9]. These properties explain why the Wasserstein distance can eectively compare any two distributions, even when their supports do not overlap, as opposed to f -divergences, e.g. KL, and some instances of IPMs: see [Arjovsky et al., 2017, Example 1] for an illustration of this advantage.

There are some special cases where computing the Wasserstein distance, i.e. solving the corresponding Kantorovich problem, is easy and reasonably cheap. We present two of these settings, which provide closed-form solutions and are of signicant important in this thesis.

Gaussian distributions. Denote by N (m, Σ) the Gaussian distribution on R d with mean m ∈ R d and covariance matrix Σ ∈ R d×d symmetric positive-denite. The Wasserstein distance of order 2 between two Gaussians, also known as the Wasserstein-Bures metric [START_REF] Dowson | The Fréchet distance between multivariate normal distributions[END_REF], is given by

W 2 2 {N (m 1 , Σ 1 ), N (m 2 , Σ 2 )} = m 1 -m 2 2 + Tr Σ 1 + Σ 2 -2 Σ 1/2 1 Σ 2 Σ 1/2 1 1/2 , (2.10)
where Tr denotes the trace operator.

Besides, if Σ 1 Σ 2 = Σ 2 Σ 1 , (2.10) can be written in the following simpler form

W 2 2 {N (m 1 , Σ 1 ), N (m 2 , Σ 2 )} = m 1 -m 2 2 + Σ 1/2 1 -Σ 1/2 2 2 F ,
(2.11) where

• F denotes the Frobenius norm.

Univariate distributions. Consider µ, ν ∈ P p (R), and denote by F -1 µ and F -1 ν the quantile functions of µ and ν respectively. By [Rachev and Rüschendorf, 1998, Theorem 3.1.2.(a)],

W p p (µ, ν) = 1 0 F -1 µ (t) -F -1 ν (t) p dt .
(2.12)

The analytical formula in (2.12) can be eciently approximated by replacing the integral with a Monte Carlo estimate. The rst approximation scheme we consider is given by,

W p p (µ, ν) ≈ 1 K K k=1 F -1 µ (t k ) -F -1 ν (t k ) p , (2.13)
where {t k } K k=1 are uniform and independent samples from [0, 1] and for ξ ∈ {µ, ν}, F -1 ξ is a linear interpolation of F -1 ξ which denotes either the exact quantile function of ξ if ξ is discrete, or an approximation by a Monte Carlo procedure. This last option is justied by the Glivenko-Cantelli theorem [START_REF] Loève | Probability Theory I[END_REF].

The second approximation is given by,

W p p (µ, ν) ≈ 1 K K k=1 s k -F -1 ν ( Fµ (s k )) p , (2.14)
where {s k } K i=1 are uniform and independent samples from µ and for ξ ∈ {µ, ν}, Fξ (resp. F -1 ξ ) is a linear interpolation of Fξ (resp. F -1 ξ ) which denotes either the exact cumulative distribution function (resp. quantile function) of ξ if ξ is discrete or an approximation by a Monte Carlo procedure.

Let us nally mention the convenient case of univariate discrete measures: if µ = n -1 n i=1 δ x i and ν = n -1 n i=1 δ y i , where {x i } n i=1 , {y i } n i=1 are two sets of n ∈ N * observations taking values in R, then (2.12) can simply be calculated by sorting {x i } n i=1 and {y i } n i=1 . In this case,

W p p (µ, ν) = 1 n n i=1 |x (i) -y (i) | p , (2.15)
where x (1) ≤ . . . ≤ x (n) and y (1) ≤ . . . ≤ y (n) .

However, apart from favorable singular cases such as the ones discussed above, computing Wasserstein distances entails several issues in practice, which are especially problematic in data sciences. We elaborate on these practical limitations in the next section.

Practical aspects and limitations

When evaluated between two multivariate empirical distributions, the Wasserstein distance admits a discrete formulation which can be rewritten as a linear program [Peyré and Cuturi, 2019, Section 3.1] and the solution is not analytically available in general.

Standard solvers from linear programming and combinatorial optimization (e.g., the simplex method) are then particularly relevant in that context, since they can be used to compute W p (μ n , νn ) [Peyré and Cuturi, 2019, Chapter 3]. While they participated in the spread of OT, these methods tend to have a super-cubic cost in practice, and their worst-case computational complexity scales in O(n 3 log(n)). Therefore, Wasserstein distances usually require important computational resources when deployed in applications that deal with medium-to-large volumes of data, such as machine learning problems.

Additionally, several prior studies have demonstrated the statistical limitations of the Wasserstein distance by deriving its sample complexity: the convergence rate of

W p (μ n , νn ) to W p (µ, ν) is in O(n -1/d
). This result was rst established by Dudley [1969] for p = 1 and compactly supported measures, and has been extended and sharpened in subsequent work [START_REF] Dereich | Constructive quantization: Approximation by empirical measures[END_REF][START_REF] Boissard | On the mean speed of convergence of empirical and occupation measures in Wasserstein distance[END_REF][START_REF] Fournier | On the rate of convergence in Wasserstein distance of the empirical measure[END_REF]. Their contribution show that in general, the convergence rate of W p (µ, μn ) to zero (consequently, of W p (μ n , νn ) to W p (µ, ν), by the triangle inequality) degrades exponentially in the ambient dimension d, meaning that for high data dimensions, n must be very large for W p (μ n , νn ) to yield an accurate approximation of W p (µ, ν). This requirement might be unrealistic or too restrictive, since increasing n inevitably increases the complexity of computing W p (μ n , νn ), let alone that it might be dicult to collect suciently many samples in some practical settings. We note however that a recent study drew a more optimistic conclusion by considering measures that are intrinsically lower-dimensional: in this specic case, the rate can be reasonably fast as it depends on that intrinsic dimension [START_REF] Weed | Sharp asymptotic and nite-sample rates of convergence of empirical measures in Wasserstein distance[END_REF].

Because of these computational and statistical limitations, deploying the Wasserstein distance to address practical tasks in data sciences can result in highly inecient algorithms. Nevertheless, these issues have in turn paved the way for novel research directions within the machine learning community, which taken altogether, dene a whole new eld called computational optimal transport (COT).

The next two sections are precisely dedicated to the main classes of probability divergences that have emerged from COT: Sinkhorn divergences, which stem from regularized optimal transport, and the Sliced-Wasserstein distance, which is the central object of this thesis. Thanks to their favorable computational and statistical properties, these divergences serve as practical alternatives to the Wasserstein distance and have inspired the design of novel ecient techniques in data sciences.

Regularized Optimal Transport, Sinkhorn Divergences

Popularized by [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF], regularized optimal transport is one of the main methodologies that have allowed the application of OT in high-dimensional applied problems.

Its guiding principle consists in adding a penalty term to the Kantorovich problem in order to approximate its solution. This penalty can be any strictly convex function, and this choice of regularization leads to dierent practical consequences [Peyré and Cuturi, 2019, Remark 4.10]. We will only describe the most common practice, which is entropic penalization and thereby denes the following regularized version of (2.9), This entropic regularization provides a number of desirable properties and has thereupon contributed signicantly to the renewed interest in OT for data sciences, e.g. in supervised learning [START_REF] Frogner | Learning with a Wasserstein loss[END_REF], computer vision [START_REF] Solomon | Convolutional Wasserstein Distances: Ecient Optimal Transportation on Geometric Domains[END_REF], domain adaptation [START_REF] Courty | Optimal Transport for Domain Adaptation[END_REF][START_REF] Redko | Optimal Transport for Multi-source Domain Adaptation under Target Shift[END_REF], dictionary or embeddings learning [START_REF] Schmitz | Wasserstein dictionary learning: Optimal transport-based unsupervised nonlinear dictionary learning[END_REF][START_REF] Frogner | Learning Embeddings into Entropic Wasserstein Spaces[END_REF] and generative modeling [START_REF] Genevay | Learning Generative Models with Sinkhorn Divergences[END_REF].

W p,ε (µ, ν) = min π∈Π(µ,ν) X×X ρ(x,
The rst major advantage is that the solutions of the Kantorovich problem can be eciently approximated via this regularization. Indeed, when µ and ν are two discrete distributions based on n points, the penalized problem in (2.16) can be eciently solved by applying an iterative numerical solver called Sinkhorn's algorithm [START_REF] Franklin | On the scaling of multidimensional matrices[END_REF], where each iteration consists in matrix-vector products. Such operations induce a complexity in O(n 2 ), which can further be improved for example by implementing them on GPU since they can be carried out in parallel, or by leveraging the structure of the cost function: see [Peyré and Cuturi, 2019, Section 4.3] for the explanation of such acceleration techniques. Therefore, Sinkhorn's algorithm executes considerably faster than the approximate solvers used in classical OT, and has been shown to converge to the solution of the unregularized Kantorovich problem with an

accuracy of δ in O(n 2 c 2 ∞ log(n)δ -2
) operations [START_REF] Dvurechensky | Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithm[END_REF]. Several algorithms for regularized optimal transport have then been developed and can achieve improved convergence rates or a superior empirical performance, e.g. a better computational eciency [START_REF] Altschuler | Near-Linear Time Approximation Algorithms for Optimal Transport via Sinkhorn Iteration[END_REF][START_REF] Dvurechensky | Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithm[END_REF][START_REF] Seguy | Large-Scale Optimal Transport and Mapping Estimation[END_REF][START_REF] Abid | Greedy stochastic algorithms for entropy-regularized optimal transport problems[END_REF][START_REF] Lin | On Ecient Optimal Transport: An Analysis of Greedy and Accelerated Mirror Descent Algorithms[END_REF].

On the other hand, the entropic regularization of OT yields an alternative divergence to the Wasserstein distance that is better suited for data sciences: W p,ε (µ, ν) is convex as a function of (µ, ν) for ε ≥ 0, and is always smooth provided that ε > 0, with a gradient known in closed-form [START_REF] Feydy | Interpolating between Optimal Transport and MMD using Sinkhorn Divergences[END_REF]. Therefore, adding an entropic penalty has been especially useful in dening a reliable loss function which, as opposed to the unregularized Wasserstein distance, can eciently be dierentiated in general: a detailed discussion on this aspect accompanied with concrete examples is provided in [Peyré and Cuturi, 2019, Section 9.1].

Nevertheless, this divergence does not verify all metric axioms since the entropic penalty term introduces the following bias: for any distribution µ, W p,ε (µ, µ) = 0.

The family of Sinkhorn divergences has then been introduced to x this problem, while inheriting from the benets of entropic regularized OT.

Denition 2.7 (Sinkhorn divergences). Let X be a Polish space equipped with a distance ρ. Let p ∈ [1, +∞) and ε > 0. The Sinkhorn divergence is dened for any µ, ν ∈ P(X)

as W p,ε (µ, ν) = W p,ε (µ, ν) - 1 2 W p,ε (µ, µ) + W p,ε (ν, ν) , (2.18)
where W p,ε is the regularized OT cost given by (2.16).

Sinkhorn divergences have been proved to be smooth and convex, similarly to W p,ε , but they are also symmetric positive denite and metrize the weak convergence [START_REF] Feydy | Interpolating between Optimal Transport and MMD using Sinkhorn Divergences[END_REF]. Regarding their practical implications, computing W p,ε is not much more expensive than for W p,ε , and can be done eciently on GPU [Feydy et al., 2019, Section 3].

Another important feature of Sinkhorn divergences is that they interpolate between the Wasserstein distance (when ε → 0) and MMD (when ε → +∞) [START_REF] Ramdas | On Wasserstein Two-Sample Testing and Related Families of Nonparametric Tests[END_REF]. Hence, Sinkhorn divergences achieve a trade-o between these two divergences, which is clearly reected in their sample complexity recalled below.

Theorem 2.8 (Theorem 3 in [START_REF] Genevay | Sample Complexity of Sinkhorn Divergences[END_REF]). Let X be a compact set of R d with diameter denoted by D X , and consider µ, ν ∈ P(X). The sample complexity of the Sinkhorn divergence is given by

E W p,ε (μ n , νn ) -W p,ε (µ, ν) ≤ K d,D X 1 + ε -d/2 exp k D X ε n -1/2 , (2.19)
where K d,D X is a constant that depends on d and D X , and k D X is a constant that depends on D X .

Theorem 2.8 has very recently been rened to the case where µ, ν are subgaussian [Mena and Niles-Weed, 2019]. In both [START_REF] Genevay | Sample Complexity of Sinkhorn Divergences[END_REF] and [Mena and Niles-Weed, 2019],

the convergence rate is consistent with the aforementioned interpolation property: when ε goes to innity, it scales in n -1/2 and is thus comparable to the MMD case (2.5); when ε shrinks to 0, the rate gets signicantly slower and, analogously to the Wasserstein distance, further degrades as the dimension d increases.

Sliced-Wasserstein Distance

Another important class of alternative divergence emerging for computational optimal transport relies on the use of low-dimensional projections of probability distributions.

This line of work was initiated by [START_REF] Rabin | Wasserstein Barycenter and Its Application to Texture Mixing[END_REF] and [START_REF] Bonneel | Sliced and Radon Wasserstein barycenters of measures[END_REF], which designed the Sliced-Wasserstein distance in order to speed up the computation of Wasserstein barycenters. We give the formal denition of SW in Denition 2.9, which we illustrate in Figure 1.3 (page 20).

Denition 2.9 (Sliced-Wasserstein distance). Let X ⊂ R d be a Polish space endowed with the Euclidean distance and denote by S d-1 = θ ∈ R d : θ = 1 the unit sphere in R d . For any u ∈ S d-1 , denote by u : X → R the linear form given by u (x) = u, x . Let Algorithm 1: Monte Carlo approximation of SW Input: Two sets of observations {x i } n i=1 and {y i } n i=1 , number of projection directions L, order p. SW = 0 for l = 1, . . . , L do Sample: u l ∼ σ for i = 1, . . . , n do Project:

x i = u l , x i , y i = u l , y i Sort: x (1) ≤ x (2) ≤ • • • ≤ x (n) , y (1) ≤ y (2) ≤ • • • ≤ y (n) SW = SW + (1/n) n i=1 |x (i) -y (i) | p SW = (SW/L) 1/p return SW p ∈ [1, +∞).
The Sliced-Wasserstein distance of order p is dened for any µ, ν ∈ P p (X) as

SW p p (µ, ν) = S d-1 W p p (u µ, u ν)dσ(u) , (2.20)
where σ is the uniform distribution on S d-1 , and for any u ∈ S d-1 , u = (u ) denotes the push-forward operator associated to u .

In other words, SW measures the dissimilarity between µ, ν ∈ P p (X) by computing

E[W p p (u µ, u ν)]
, where E is taken with respect to u uniformly distributed on S d-1 . The push-forward measures u µ and u ν are univariate since they correspond to projections of µ and ν along the direction u ∈ S d-1 . In particular, when ξ ∈ {µ, ν} is approximated by the empirical measure ξn = (1/n) n i=1 δ x i , where {x i } n i=1 are i.i.d. samples from ξ,

then u ξn = 1 n n i=1 δ u,x i .
This means that empirical approximations of u µ and u ν can simply be obtained by projecting the available samples from µ and ν along u.

Besides, the expectation that denes SW can easily be approximated with a standard

Monte Carlo method: one draws L ∈ N * samples i.i.d. from σ, denoted by {u l } L l=1 , and approximates SW (2.20) with

SW p p,L (µ, ν) = 1 L L l=1 W p p u l µ, u l ν , (2.21)
where for l ∈ {1, . . . , L}, u l = {(u l ) } is the push-forward operator associated to (u l ) , the linear form introduced in Denition 2.9. The Monte Carlo estimate of SW thus requires solving nitely many OT problems in R, which is convenient given our discussion in Section 2.4.1. This approximation method is summarized in Algorithm 1, and has a complexity in O Ldn + Ln log(n) due to the projecting and sorting operations.

The computational benets of SW over the Wasserstein distance have encouraged its use in various practical applications, thus making it an increasingly popular divergence over the last few years. We present hereafter the main contributions on the theoretical properties of SW.

First, [Bonnotte, 2013, Chapter 5] provides a collection of useful results on SW, including the following: SW is a distance on P p (R d ) [Bonnotte, 2013, Proposition 5.1.2], and its relation with the Wasserstein distance of order p has been rigorously established as follows.

Theorem 2.10 (Proposition 5.1.3 and Theorem 5.1.5 of [START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF]). Let µ, ν ∈ P p (R d ). There exists a positive constant c d,p ≤ 1 depending on d and p such that

SW p p (µ, ν) ≤ c d,p W p p (µ, ν) . (2.22)
Besides, denote by

B d (0, R) = x ∈ R d : x < R the open ball in R d of radius R > 0 centered around 0 ∈ R d
, and assume that µ and ν are supported on B d (0, R). Then, there exists a constant C d,p > 0 such that

W p p (µ, ν) ≤ C d,p c d,p R p-1/(d+1) SW p (µ, ν) 1/(d+1) (2.23)
Very recently, [START_REF] Bayraktar | Strong equivalence between metrics of Wasserstein type[END_REF] conducted a thorough theoretical analysis on the equivalence between the Wasserstein distance and SW, which nicely complements Theorem 2.10. Their results also concern a variant of SW, the maximum Sliced-Wasserstein distance [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF], which we described in Section 1.4 (page 25) and dene in Chapter 5. To conclude, let us mention that the complete review of the literature on SW provided in Sections 1.3 and 1.4 conrms the relevance of this metric in data sciences, but also emphasizes a severe lack of theoretical insights. As explained in Section 1.4, one of the objectives of this thesis is then to bridge this gap between theory and practice: in particular, the next chapter aims at making SW-based generative models more theoretically grounded, by analyzing the asymptotic properties of the estimators obtained by minimizing SW.

Chapter 3

Asymptotic Guarantees for

Learning Generative Models with the Sliced-Wasserstein Distance

This chapter is based on [START_REF] Nadjahi | Asymptotic Guarantees for Learning Generative Models with the Sliced-Wasserstein Distance[END_REF].

Minimum expected distance estimation (MEDE) algorithms have been widely used for probabilistic models with intractable likelihood functions, and have become increasingly popular due to their use in implicit generative modeling. Emerging from computational optimal transport, the Sliced-Wasserstein distance has become a popular choice in MEDE thanks to its simplicity and computational benets. While several studies have reported empirical success on generative modeling with SW, the theoretical properties of such estimators have not yet been established.

In this chapter, we investigate the asymptotic properties of estimators that are obtained by minimizing SW. We rst show that convergence in SW implies weak convergence of probability measures in general Wasserstein spaces. Then we show that estimators obtained by minimizing SW (and also an approximate version of SW) are asymptotically consistent. We nally prove a central limit theorem, which characterizes the asymptotic distribution of the estimators and establish a convergence rate of

√ n,

where n denotes the number of observed data points. We illustrate the validity of our theory on both synthetic data and neural networks.

Introduction

Minimum distance estimation (MDE) is a generalization of maximum-likelihood inference, where the goal is to minimize a distance between the empirical distribution of a

set of i.i.d. observations Y 1:n = (Y 1 , . . . , Y n
) and a family of distributions indexed by a parameter θ. The problem is formally dened as follows [Wolfowitz, 1957[START_REF] Basu | Statistical Inference: The Minimum Distance Approach[END_REF],

θn = argmin θ∈Θ D(μ n , µ θ ) , (3.1)
where D denotes a distance (or a divergence in general) between probability measures, µ θ denotes a probability measure indexed by θ, Θ denotes the parameter space, and

μn = 1 n n i=1 δ Y i (3.2)
denotes the empirical measure of Y 1:n . When D is chosen as the Kullback-Leibler divergence, this formulation coincides with the maximum likelihood estimation (MLE, Basu et al. [2011]).

While MDE provides a fruitful framework for statistical inference, when working with generative models, solving the optimization problem in (3.1) might be intractable since it might be impossible to evaluate the probability density function associated with µ θ . Nevertheless, in various settings, even if the density is not available, one can still generate samples from the distribution µ θ , and such samples turn out to be useful for making inference. More precisely, under such settings, a natural alternative to (3.1) is the minimum expected distance estimator [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF], which is dened as

θn,m = argmin θ∈Θ E [D(μ n , μθ,m )|Y 1:n ] , (3.3)
where

μθ,m = 1 m m i=1 δ Z i (3.4)
denotes the empirical distribution of Z 1:m , that is a sequence of i.i.d. random variables with distribution µ θ . This algorithmic framework has computationally favorable properties since one can replace the expectation with a simple Monte Carlo average in practical applications.

In the context of MDE, distances that are based on optimal transport have become increasingly popular due to their computational and theoretical properties [START_REF] Arjovsky | Wasserstein Generative Adversarial Networks[END_REF][START_REF] Tolstikhin | Wasserstein Auto-Encoders[END_REF][START_REF] Genevay | Learning Generative Models with Sinkhorn Divergences[END_REF][START_REF] Patrini | Sinkhorn autoencoders[END_REF][START_REF] Adler | Banach Wasserstein GAN[END_REF]. For instance, if we replace the distance D in (3.3) with the Wasserstein distance, we obtain the minimum expected Wasserstein estimator [START_REF] Bassetti | On minimum Kantorovich distance estimators[END_REF][START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF]. In the classical statistical inference setting, the typical use of such an estimator is to infer the parameters of a measure whose density does not admit an analytical closed-form formula [START_REF] Basu | Statistical Inference: The Minimum Distance Approach[END_REF]. On the other hand, in the implicit generative modeling (IGM) setting, this estimator forms the basis of two popular IGM strategies: Wasserstein generative adversarial networks [START_REF] Arjovsky | Wasserstein Generative Adversarial Networks[END_REF] and Wasserstein auto-encoders [START_REF] Tolstikhin | Wasserstein Auto-Encoders[END_REF]. These methods are related to each other according to [START_REF] Genevay | GAN and VAE from an optimal transport point of view[END_REF], and fall within the IGM framework explained in Section 1.1: the goal is to nd the best parametric transport map T θ , such that T θ transforms a simple distribution µ (e.g., standard Gaussian or uniform) to a potentially complicated data distribution μn , by minimizing the Wasserstein distance between the transported distribution µ θ = T θ µ and μn . In practice, θ is typically chosen as a neural network, for which it is often impossible to evaluate the induced density µ θ . However, one can easily generate samples from µ θ by rst generating a sample from µ and then applying T θ to that sample, making minimum expected distance estimation (3.3) feasible for this setting. Motivated by its practical success, the theoretical properties of this estimator have been recently taken under investigation [START_REF] Bousquet | From optimal transport to generative modeling: the VEGAN cookbook[END_REF][START_REF] Liu | Approximation and convergence properties of generative adversarial learning[END_REF] and very recently, [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF] have established the consistency (for the general setting) and the asymptotic distribution (for one dimensional setting) of this estimator.

Even though estimation with the Wasserstein distance has served as a fertile ground for many generative modeling applications, except for the case when the measures are supported on R 1 , the computational complexity of minimum Wasserstein estimators rapidly becomes excessive with the increasing problem dimension, and developing accurate and ecient approximations is a highly non-trivial task. Therefore, there have been several attempts to use more practical alternatives to the Wasserstein distance [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF][START_REF] Genevay | Learning Generative Models with Sinkhorn Divergences[END_REF], and in this context, the Sliced-Wasserstein distance has been an increasingly popular alternative to the Wasserstein distance. While several studies have reported empirical success on generative modeling with SW, the theoretical properties of such estimators have not yet been fully established, as we discussed in Section 1.3.

In this chapter, we investigate the asymptotic properties of estimators given in (3.1) and(3.3) when D is replaced with SW p . We rst prove that convergence in SW implies weak convergence of probability measures dened on general domains, which generalizes the results given in [START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF]. Then, by using similar techniques to the ones in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF], we show that the estimators dened by (3.1) and (3.3) are consistent, meaning that as the number of observations n increases the estimates will get closer to the data-generating parameters. We nally prove a central limit theorem (CLT) in the multidimensional setting, which characterizes the asymptotic distribution of these estimators and establishes a convergence rate of √ n. The CLT that we prove is stronger than the one derived in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF] in the sense that it is not restricted to the one-dimensional setting, as opposed to [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF].

We support our theory with experiments that are conducted on both synthetic and real data. We rst consider a more classical statistical inference setting, where we consider a Gaussian model and a multidimensional α-stable model whose density is not available in closed-form. In both models, the experiments validate our consistency and CLT results. We further observe that, especially for high-dimensional problems, the estimators obtained by minimizing SW have signicantly better computational properties when compared to the ones obtained by minimizing the Wasserstein distance, as expected. In the IGM setting, we consider the neural network-based generative modeling algorithm proposed in [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF] to show that our results also hold in the real data setting as well.

Asymptotic Guarantees for Minimum Sliced-Wasserstein

Estimators

We rst clarify the mathematical formalism for the problem of parameter inference in purely generative models. Let (Y k ) k∈N be a sequence of random variables associated with observations, where each observation takes value in Y ⊂ R d . We assume that these observations are i.i.d. from µ ∈ P(Y), and we consider the statistical model

M = {µ θ ∈ P(Y), θ ∈ Θ}, where Θ ⊂ R d θ is the parametric space. For all θ ∈ Θ, we can generate i.i.d. samples (Z k ) k∈N * ∈ Y N * from µ θ , but the associated likelihood is numerically intractable. The empirical approximation of µ θ based on m ∈ N * samples is then given by μθ,m = m -1 m i=1 δ Z i .
Throughout this chapter, we assume that the following conditions hold:

(C1) Y, endowed with the Euclidean distance ρ, is a Polish space, (C2) Θ, endowed with the distance ρ Θ , is a Polish space, (C3) Θ is a σ-compact space, i.e. the union of countably many compact subspaces, (C4) Parameters are identiable, i.e. µ θ = µ θ implies θ = θ .

We endow P(Y) with the Lévy-Prokhorov distance d P , which metrizes the weak convergence by [Billingsley, 1999, Theorem 6.8] since Y is assumed to be a Polish space.

We dene the minimum Sliced-Wasserstein estimator (MSWE) of order p as the estimator obtained by plugging SW p in place of D in (3.1). Similarly, we dene the minimum expected Sliced-Wasserstein estimator (MESWE) of order p as the estimator obtained by plugging SW p in place of D in (3.3). In the rest of the chapter, MSWE and MESWE will be denoted by θn and θn,m respectively.

In what follows, we present the asymptotic properties that we derived for MSWE and MESWE, namely their existence, consistency and measurability. We also formulate a CLT that characterizes the asymptotic distribution of MSWE and establishes a convergence rate for any dimension. All the proofs for these results are provided in Sections 3.5.3 to 3.5.7.

Note that since the Sliced-Wasserstein distance is dened as an average of onedimensional Wasserstein distances, some proofs are inevitably similar to the proofs done in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF]. However, the adaptation of these techniques to the SW case is made possible by the identication of novel properties regarding the topology induced by the SW distance, which we establish for the rst time in this study: see Sections 3.2.1 and 3.5.1.

Topology induced by the Sliced-Wasserstein distance

We begin this section by a useful result which we believe is interesting on its own and implies that the topology induced by SW p on P p (R d ) is ner than the weak topology induced by the Lévy-Prokhorov metric d P .

Theorem 3.1. Let p ∈ [1, +∞). The convergence in SW p implies the weak convergence in

P(R d ), i.e. if (µ k ) k∈N is a sequence of measures in P p (R d ) satisfying lim k→+∞ SW p (µ k , µ) = 0 with µ ∈ P p (R d ), then (µ k ) k∈N w -→ µ.
The property that convergence in SW p implies weak convergence has already been proven in [START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF] for compact domains only. While the implication of weak convergence is one of the most crucial requirements that a distance metric should satisfy, to the best of our knowledge, this implication has not been proved for general domains before. In [START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF], the main proof technique was based on showing that SW p is equivalent to W p in compact domains, whereas we follow a dierent path and use the Lévy characterization: see Section 3.5.2 for the detailed proof.

Existence and consistency of MSWE and MESWE

In our next set of results, we will show that both MSWE and MESWE are consistent, in the sense that, when the number of observations n increases, the estimators will converge to a parameter θ that minimizes the ideal problem θ → SW p (µ , µ θ ). Before we make this argument more precise, let us rst present the assumptions that will imply our results.

A1. The map θ → µ θ is continuous from (Θ, ρ Θ ) to (P(Y), d P ), i.e. for any sequence

(θ n ) n∈N in Θ satisfying lim n→+∞ ρ Θ (θ n , θ) = 0, then (µ θn ) n∈N w -→ µ θ .
A 2. The data-generating process is such that lim n→+∞ SW p (μ n , µ ) = 0, P-almost surely.

A3. There exists > 0, such that setting = inf θ∈Θ SW p (µ , µ θ ), the set

Θ = {θ ∈ Θ : SW p (µ , µ θ ) ≤ + } is bounded.
These assumptions are mostly related to the identiability of the statistical model and the regularity of the data generating process. They are arguably mild assumptions, analogous to those that have already been considered in the literature [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF]. Note that, without Theorem 3.1, the formulation and use of A2 in our proofs would not be possible. In the next result, we establish the consistency of MSWE.

Theorem 3.2 (Existence and consistency of MSWE). Assume A1, A2 and A3. There

exists E ∈ F with P(E) = 1 such that, for all ω ∈ E, lim n→+∞ inf θ∈Θ SW p (μ n (ω), µ θ ) = inf θ∈Θ SW p (µ , µ θ ), and (3.5) lim sup n→+∞ argmin θ∈Θ SW p (μ n (ω), µ θ ) ⊂ argmin θ∈Θ SW p (µ , µ θ ) , (3.6)
where μn is dened by (3.2).

Besides, for all ω ∈ E, there exists n(ω) such that, for all n ≥ n(ω), the set argmin θ∈Θ SW p (μ n (ω), µ θ ) is non-empty.

Our proof technique is similar to the one given in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF]. This result shows that, when the number of observations goes to innity, the estimate θn will con- verge to a global minimizer of the problem min θ∈Θ SW p (µ , µ θ ). In our next result, we prove a similar property for MESWEs as min(m, n) goes to innity. In order to increase clarity, and without loss of generality, in this setting, we consider m as a function of n such that lim n→+∞ m(n) = +∞. Now, we derive an analogous version of Theorem 3.2 for MESWE. For this result, we need to introduce another continuity assumption.

A4. If lim

n→+∞ ρ Θ (θ n , θ) = 0, then lim n→+∞ E [SW p (µ θn , μθn,n )|Y 1:n ] = 0.
The next theorem establishes the consistency of MESWE.

Theorem 3.3 (Existence and consistency of MESWE). Assume A1, A2, A3 and A4.

Let (m(n)) n∈N * be an increasing sequence satisfying lim n→+∞ m(n) = +∞. There exists a set E ⊂ Ω with P(E) = 1 such that, for all w ∈ E,

lim n→+∞ inf θ∈Θ E SW p (μ n , μθ,m(n) ) Y 1:n = inf θ∈Θ SW p (µ , µ θ ), and (3.7) lim sup n→+∞ argmin θ∈Θ E SW p (μ n , μθ,m(n) ) Y 1:n ⊂ argmin θ∈Θ SW p (µ , µ θ ) , (3.8)
where μn and μθ,m(n) are dened by (3.2) and (3.4) respectively.

Besides, for all ω ∈ E, there exists n(ω) such that, for all n ≥ n(ω), the set

argmin θ∈Θ E SW p (μ n , μθ,m(n) )|Y 1:n is non-empty.
Similar to Theorem 3.2, this theorem shows that, when the number of observations goes to innity, the estimator obtained with the expected distance will converge to a global minimizer.

Convergence of MESWE to MSWE

Since in practical applications, we can only use a nite number of generated samples Z 1:m , we analyze the case where the observations Y 1:n are kept xed while the number of generated samples increases, i.e. m → +∞. We show in this scenario that MESWE converges to MSWE, assuming the latter exists. Before deriving this result, we formulate a technical assumption below.

A5. For some > 0 and n = inf θ∈Θ SW p (μ n , µ θ ), the set

Θ ,n = {θ ∈ Θ : SW p (μ n , µ θ ) ≤ n + } is bounded almost surely.
Theorem 3.4 (MESWE converges to MSWE as m → +∞). Assume A1, A4 and A5.

Then,

lim m→+∞ inf θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] = inf θ∈Θ SW p (μ n , µ θ ) (3.9) lim sup m→+∞ argmin θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] ⊂ argmin θ∈Θ SW p (μ n , µ θ ) (3.10) There exists m * such that, for any m ≥ m * , argmin θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] is a non- empty set.
This result shows that MESWE would be indeed promising in practice, as one get can more accurate estimations by increasing m.

Measurability of MSWE and MESWE

The measurability of the MSWE and MESWE follows from the application of [Brown and Purves, 1973, Corollary 1], also used in [START_REF] Bassetti | On minimum Kantorovich distance estimators[END_REF][START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF],

and which we recall in Theorem 3.18. Theorem 3.5 (Measurability of the MSWE). Assume A1. For any n ≥ 1 and > 0, there exists a Borel measurable function θn, : Ω → Θ that satises: for any

ω ∈ Ω, θn, (ω) ∈    argmin θ∈Θ SW p (μ n (ω), µ θ ), if this set is non-empty, {θ ∈ Θ : SW p (μ n (ω), µ θ ) ≤ + }, otherwise.
where = inf θ∈Θ SW p (µ , µ θ ).

We prove an analogous result to Theorem 3.5 in Section 3.5.6, which establishes the measurability of MESWE.

Rate of convergence and the asymptotic distribution

In our last set of theoretical results, we investigate the asymptotic distribution of MSWE and we establish a rate of convergence. We now suppose that we are in the well-specied setting, i.e. there exists θ in the interior of Θ such that µ θ = µ , and we consider additional assumptions, as stated below.

For any u ∈ S d-1 and t ∈ R, we dene

F θ (u, t) = Y 1 (-∞,t] ( u, y )dµ θ (y). Note that for any u ∈ S d-1 , F θ (u, •) is the cumulative distribution function (CDF) associated to the measure u µ θ .
A6. For all > 0, there exists δ > 0 such that

inf θ∈Θ: ρ Θ (θ,θ )≥ SW 1 (µ θ , µ θ ) > δ .
Let L 1 (S d-1 ×R, σ ⊗Leb 1 ) denote the class of functions that are absolutely integrable on the domain S d-1 × R w.r.t. the product measure σ ⊗ Leb 1 , where Leb 1 denotes the Lebesgue measure on R.

A7. Assume that there exists a measurable function

D = (D ,1 , . . . , D ,d θ ) : S d-1 × R → R d θ such that for each i = 1, . . . , d θ , D ,i ∈ L 1 (S d-1 × R, σ ⊗ Leb 1 ) and S d-1 R |F θ (u, t) -F θ (u, t) -θ -θ , D (u, t) | dtdσ(u) = (ρ Θ (θ, θ )) , where : R + → R + satises lim t→0 (t) = 0. Besides, {D ,i } d θ i=1 are linearly independent in L 1 (S d-1 × R, σ ⊗ Leb 1 ). For any u ∈ S d-1 , and t ∈ R, dene Fn (u, t) = n -1 card {i ∈ {1, . . . , n} : u, Y i ≤ t} ,
where card denotes the cardinality of a set, and for any u ∈ S d-1 , Fn (u, •) is the CDF associated to the measure u μn .

A8. There exists a random element G :

S d-1 × R → R such that the stochastic process √ n( Fn -F θ ) converges weakly in L 1 (S d-1 × R, σ ⊗ Leb 1 ) to G .
Under mild assumptions on the tails of u µ for any u ∈ S d-1 , we believe that one can prove that A8 holds in general by extending [Dede, 2009, Proposition 3.5] and [del Barrio et al., 1999, Theorem 2.1a].

We can now formulate our central limit theorem based on these assumptions.

Theorem 3.6. Assume A1, A2, A3, A6, A7 and A8. Then, the asymptotic distribution of the goodness-of-t statistic is given by,

√ n inf θ∈Θ SW 1 (μ n , µ θ ) w -→ inf θ∈Θ S d-1 R |G (u, t) -θ, D (u, t) | dtdσ(u)
as n → +∞, where μn is dened by (3.2).

Theorem 3.7. Assume A1, A2, A3, A6, A7 and A8. Suppose also that the random

map θ → S d-1 R |G (u, t) -θ, D (u, t) | dtdσ(u) has a unique inmum almost surely. Then, MSWE with p = 1 satises, √ n( θn -θ ) w -→ argmin θ∈Θ S d-1 R |G (u, t) -θ, D (u, t) | dtdσ(u)
as n → +∞, where θn is dened by (3.1) with SW 1 in place of D.

These results show that the estimator and the associated goodness-of-t statistics will converge to a random variable in distribution, where the rate of convergence is √ n.

Note that G is dened as a random element (see A8), therefore we cannot claim that the convergence in distribution derived in Theorem 3.6 and 3.7 implies the convergence in probability.

-2 -1 0 1 2 √ n( σ 2 n -σ 2 ) 0.
This CLT is also inspired by [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF], where they identied the asymptotic distribution associated to the minimum Wasserstein estimator. However, since W p admits an analytical form only when d = 1, their result is restricted to the scalar case, and in their conclusion, [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF] conjecture that the rate of the minimum Wasserstein estimators would depend negatively on the dimension of the observation space. On the contrary, since SW p is dened in terms of one-dimensional W p distances, we circumvent the curse of dimensionality and our result holds for any nite dimension. While the perceived computational burden has created a pessimism in the machine learning community about the use of Wasserstein-based methods in large dimensional settings, which motivated the rise of regularized optimal transport, we believe that our ndings provide another interesting counter-example to this conception.

Experiments

We conduct experiments on synthetic and real data to empirically conrm our theorems.

We explain in Section 3.5.8 the optimization methods used to nd the estimators. Specifically, we can use stochastic iterative optimization algorithm (e.g., stochastic gradient descent). Note that, since we calculate (expected) SW with Monte Carlo approximations over a nite set of projections (and a nite number of `generated datasets' ), MSWE and MESWE fall into the category of doubly stochastic algorithms. Our experiments on synthetic data actually show that using only one random projection and one randomly generated dataset at each iteration of the optimization process is enough to illustrate our theorems. We provide the code to reproduce the experiments 1 .

Multivariate Gaussian distributions

We consider the task of estimating the parameters of a 10-dimensional Gaussian distribution using our SW estimators: we are interested in the model

M = N (m, σ 2 I) : m ∈ R 10 , σ 2 > 0 ,
and we draw i.i.d. observations with (m , σ 2 ) = (0, 1). The advantage of this simple setting is that the density of the generated data has a closed-form expression, which makes MSWE tractable.

We empirically verify our central limit theorem: for dierent values of n, we compute 500 times MSWE of order 1 using one random projection, then we estimate the density of σ2

n with a kernel density estimator. Figure 3.1 shows the distributions cen- tered and rescaled by

√ n for each n, and conrms the convergence rate that we derived (Theorem 3.7).

To illustrate the consistency property in Theorem 3.2, we approximate MSWE of order 2 for dierent numbers of observed data n using one random projection and we report for each n the mean squared error between the estimate mean and variance and the data-generating parameters (m , σ 2 ). We proceed the same way to study the consistency of MESWE (Theorem 3.3), which we approximate using one random projection and one 1 See our GitHub repository: https://github.com/kimiandj/min_swe.

Multivariate elliptically contoured stable distributions

We focus on parameter inference for a subclass of multivariate stable distributions, called elliptically contoured stable distributions and denoted by EαS c [START_REF] Nolan | Multivariate elliptically contoured stable distributions: theory and estimation[END_REF]. Stable distributions refer to a family of heavy-tailed probability distributions that generalize Gaussian laws and appear as the limit distributions in the generalized central limit theorem [START_REF] Samorodnitsky | Stable Non-Gaussian Random Processes: Stochastic Models with Innite Variance[END_REF]]. These distributions have many attractive theoretical properties and have been proven useful in modeling nancial [START_REF] Mandelbrot | Fractals and Scaling in Finance: Discontinuity, Concentration, Risk. Selecta[END_REF] data or audio signals [START_REF] Im³ekli | Alpha-stable matrix factorization[END_REF][START_REF] Leglaive | Alpha-stable multichannel audio source separation[END_REF]. While special univariate cases include Gaussian, Lévy and Cauchy distributions, the density of stable distributions has no general analytic form, which restricts their practical application, especially for the multivariate case.

If Y ∈ R d ∼ EαS c (Σ, m), then its joint characteristic function is dened for any t ∈ R d as E exp(it T Y ) = exp -(t T Σt) α/2 + it T m ,
where Σ is a positive denite matrix (akin to a correlation matrix), m ∈ R d is a location vector (equal to the mean if it exists) and α ∈ (0, 2) controls the thickness of the tail. Even though their densities cannot be evaluated easily, it is straightforward to sample from EαS c [START_REF] Nolan | Multivariate elliptically contoured stable distributions: theory and estimation[END_REF], and we explain the sampling method in Section 3.5.8.

M = EαS c (I, m) : m ∈ R d .
The Wasserstein distance on multivariate data is either computed exactly by solving the linear program, or approximated by solving a regularized version of this problem with Sinkhorn's algorithm (Section 2.5). The MESWE is approximated using 10 random projections and 10 sets of generated samples. Then, following the approach in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF], we use the gradient-free optimization method Nelder-Mead [Nelder and Mead, 1965] to minimize the Wasserstein and SW distances. We report on Figure 3.3a the mean squared error between each estimate and m , as well as their average computational time for dierent values of dimension d. We see that MESWE provides the same Mean-squared error between the training (test) loss for (n, m) ∈ (1, 1), (100, 20), (1000, 40), (10 000, 60) and the training (test) loss for (n, m) = (60 000, 200) on MNIST using the SW generator. We trained for 20 000 iterations with the ADAM optimizer [START_REF] Kingma | A method for stochastic optimization[END_REF]. quality of estimation as its Wasserstein-based counterparts while considerably reducing the computational time, especially in higher dimensions.

We focus on this model in R 10 and we illustrate the consistency of the MESWE mn,m , approximated with one random projection and one generated dataset, the same way as for our previous Gaussian model: see Figure 3.3b. To conrm the convergence of mn,m to the MSWE mn , we x n = 100 observations and we compute the mean squared error between the two approximate estimators (using one random projection and one generated dataset) for dierent values of m (Figure 3.3c). Note that the MSWE is approximated with the MESWE obtained for a large enough value of m: mn ≈ mn,10 000 .

High-dimensional real data using GANs

Finally, we run experiments on image generation using the Sliced-Wasserstein Generator (SWG), an alternative GAN formulation based on the minimization of the SW distance [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF]. The goal is to optimize the neural network parameters such that the generated images are close to the observed ones. [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF] proposes to minimize the SW distance between µ θ and the real data distribution over θ as the generator objective, and train on MESWE in practice.

For our experiments, we design a neural network with the fully-connected conguration given in [Deshpande et al., 2018, Appendix D]

Conclusion

The Sliced-Wasserstein distance has been an attractive metric choice for learning in generative models, where the densities cannot be computed directly. In this chapter, we investigated the asymptotic properties of estimators that are obtained by minimizing SW and the expected SW. We showed that (i) convergence in SW implies weak convergence of probability measures in general Wasserstein spaces, (ii) the estimators are consistent, and (iii) the estimators converge to a random variable in distribution with a rate of √ n.

We validated our mathematical results on both synthetic data and neural networks.

We would like to point out that, in all of our experiments, the random projections used in the Monte Carlo estimate of SW were picked uniformly on S d-1 : see Section 3.5.8 for more details. The sampling on S d-1 directly impacts the quality of the resulting approximation of SW, and might induce variance in practice when learning generative models: this aspect is addressed in the next chapters, specically in Chapters 5 to 7. On the theoretical side, studying the asymptotic properties of SW-based estimators obtained with a nite number of projections is an interesting question (e.g., their behavior might depend on the sampling method or the number of projections used). We leave this study for future research.

3.5

Appendix: Postponed Proofs and Experimental Details

Preliminary theoretical results

We rst recall the denition of epi-convergence and gather technical results regarding the lower semi-continuity of (expected) Sliced-Wasserstein distances, which will be needed in our proofs.

Denition 3.8 (Epi-convergence). Let Θ be a metric space and f : Θ → R. Consider a sequence (f k ) k∈N of functions from Θ to R. We say that the sequence (f k ) k∈N epiconverges to a function f : Θ → R, and write

(f k ) k∈N e - → f , if for each θ ∈ Θ, lim inf k→∞ f k (θ k ) ≥ f (θ) for every sequence (θ k ) n∈N s.t. lim k→+∞ θ k = θ ,
and lim sup k→∞ f k (θ k ) ≤ f (θ) for a sequence (θ k ) n∈N s.t. lim k→+∞ θ k = θ .
An equivalent and useful characterization of epi-convergence is given in [Rockafellar et al., 2009, Proposition 7.29], which we paraphrase in Proposition 3.10 after recalling the denition of lower semi-continuous functions.

Denition 3.9 (Lower semi-continuity). Let Θ be a metric space and f : Θ → R. We say that f is lower semi-continuous (l.s.c.) on Θ if for any θ 0 ∈ Θ, lim inf

θ→θ 0 f (θ) ≥ f (θ 0 )
Proposition 3.10 (Characterization of epi-convergence via minimization, Proposition 7.29 of [START_REF] Rockafellar | Variational Analysis. Grundlehren der mathematischen Wissenschaften[END_REF]). Let Θ be a metric space and f : Θ → R be a l.s.c. function. The sequence

(f k ) k∈N , with f k : Θ → R for any n ∈ N, epi-converges to f if and only if (a) lim inf k→∞ inf θ∈K f k (θ) ≥ inf θ∈K f (θ) for every compact set K ⊂ Θ ; (b) lim sup k→∞ inf θ∈O f k (θ) ≤ inf θ∈O f (θ)
for every open set O ⊂ Θ.

[ Rockafellar et al., 2009, Theorem 7.31], paraphrased below, gives asymptotic properties for the inmum and argmin of epiconvergent functions and will be useful to prove the existence and consistency of our estimators.

Theorem 3.11 (Inf and argmin in epiconvergence, Theorem 7.31 of [START_REF] Rockafellar | Variational Analysis. Grundlehren der mathematischen Wissenschaften[END_REF]). Let Θ be a metric space, f : Θ → R be a l.s.c. function and (f k ) k∈N be a sequence with

f k : Θ → R for any n ∈ N. Suppose (f k ) k∈N e - → f with -∞ < inf θ∈Θ f (θ) < ∞. (a) It holds lim k→∞ inf θ∈Θ f k (θ) = inf θ∈Θ f (θ)
if and only if for every η > 0 there exists a compact set K ⊂ Θ and N ∈ N such for any k ≥ N ,

inf θ∈K f k (θ) ≤ inf θ∈Θ f k (θ) + η .
(b) In addition, lim sup k→∞ argmin θ∈Θ f k (θ) ⊂ argmin θ∈Θ f (θ).

P p (Y) such that µ k w -→ µ and ν k w -→ ν. We show that for any u ∈ S d-1 , W p p (u µ, u ν) ≤ lim inf k→+∞ W p p (u µ k , u ν k ) .
(3.11) Indeed, if (3.11) holds, then the proof is completed using the denition of the Sliced-Wasserstein distance and Fatou's Lemma. Let u ∈ S d-1 . For any k ∈ N, let γ k ∈ P(R × R) be an optimal transference plan between u µ k and u ν k for the Wasserstein distance of order p, which exists by [Villani, 2008, Theorem 4.1], i.e. b). Note that by [Villani, 2008, Lemma 4.4] and Prokhorov's Theorem, (γ k ) k∈N is sequentially compact in P(R × R) for the topology associated with the weak convergence. Now, consider a subsequence (γ φ 1 (k) ) k∈N where φ 1 :

W p p (u µ k , u ν k ) = R×R |a -b| dγ k (a,
N → N is increasing such that lim k→+∞ R×R |a -b| p dγ φ 1 (k) (a, b) = lim k→+∞ W p p (u µ φ 1 (k) , u ν φ 1 (k) ) = lim inf k→+∞ W p p (u µ k , u ν k ) .
(3.12)

Since (γ k ) k∈N is sequentially compact, (γ φ 1 (k) ) k∈N is sequentially compact as well, so there exists an increasing function φ 2 : N → N and a probability distribution γ ∈ P(R × R) such that (γ φ 2 (φ 1 (k)) ) k∈N converges weakly to γ. Then, we obtain by (3.12),

R×R a -b p dγ(a, b) = lim k→+∞ R×R a -b p dγ φ 2 (φ 1 (k)) (a, b) = lim inf k→+∞ W p p (u µ k , u ν k ) .
If we show that γ ∈ Γ(u µ, u ν), it will conclude the proof of (3.11) by denition of the Wasserstein distance (Denition 2.6). But for any continuous and bounded function

f : R → R, since for any k ∈ N, γ k ∈ Γ(µ k , ν k )
, and (µ k ) k∈N , (ν k ) k∈N converge weakly to µ and ν respectively, we have

R×R f (a)dγ(a, b) = lim k→+∞ R×R f (a)dγ φ 2 (φ 1 (k)) (a, b) = lim k→+∞ R f (a)du µ φ 2 (φ 1 (k)) (a) = R f (a)du µ(a) ,
and similarly,

R×R f (b)dγ(a, b) = R f (b)du ν(a) .
This shows that γ ∈ Γ(u µ, u ν) and therefore, (3.11) is true. We conclude by applying Fatou's Lemma.

By a direct application of Lemma 3.12, we obtain the following corollary.

Corollary 3.13. Assume A1. Then, (µ, θ) → SW p (µ, µ θ ) is lower semi-continuous in

P p (Y) × Θ.
Next, we establish analogous properties for the expected SW distance.

Lemma 3.14

d P (ζ 1 , ζ 2 ) = inf { > 0 : for any A ∈ B(Y), ζ 1 (A) ≤ ζ 2 (A ) + and ζ 2 (A) ≤ ζ 1 (A ) + } ,
where B(Y) is the Borel σ-eld of (Y, ρ) and for any A ∈ B(Y), A = {x ∈ Y : ρ(x, y) < for any y ∈ A}, we get

d P (ν k,m , νm ) ≤ 2 max i∈{1,...,m} ρ(x i k , x i ) ,
and therefore lim k→+∞ d P (ν k,m , νm ) = 0, so that, (ν k,m ) k∈N weakly converges to νm . Finally, we have that μk,m = (1/m) m i=1 δ Xi k , weakly converges to μm = (1/m) m i=1 δ Xi P-almost surely and we obtain the nal result using the lower semi-continuity of the Sliced-Wasserstein distance derived in Lemma 3.12 and Fatou's lemma,

Ẽ [SW p (ν, μm )] ≤ Ẽ lim inf i→∞ SW p (ν i , μm,i ) ≤ lim inf i→∞ Ẽ [{SW p (ν i , μm,i )] ,
where Ẽ is the expectation corresponding to P.

The following corollary is a direct consequence of Lemma 3.14.

Corollary 3.15. Assume A1. Then, Then, there exists an increasing function φ : N → N such that the subsequence (µ φ(k) ) k∈N converges weakly to µ.

(ν, θ) → E[SW p (ν, μθ,m )|Y 1:n ] is lower semi- continuous on P(Y) × Θ.
Proof. By denition, we have that

lim k→∞ S d-1 W 1 (u µ k , u µ)dσ(u) = 0 .
Therefore, by [Bogachev, 2007, Theorem 2.2.5], there exists an increasing mapping φ :

N → N such that for σ-almost every (σ-a.e.) u ∈ S d-1 , lim k→∞ W 1 (u µ φ(k) , u µ) = 0 .
By [Villani, 2008, Theorem 6.9], it implies that for σ-a.e. u ∈ S d-1 ,

(u µ φ(k) ) k∈N w -→ u µ .
Lévy's characterization [Kallenberg, 1997, Theorem 4.3] gives that, for σ-a.e. u ∈ S d-1 and any s ∈ R,

lim k→∞ Φ u µ φ(k) (s) = Φ u µ (s) ,
where, for any distribution ν ∈ P(R p ), Φ ν denotes the characteristic function of ν and is dened for any v ∈ R p as

Φ ν (v) = R p e i v,w dν(w) .
Then, we can conclude that for Lebesgue-almost every z ∈ R d ,

lim k→∞ Φ µ φ(k) (z) = Φ µ (z) .
(3.13)

We can now show that (µ φ(k) ) k∈N w -→ µ, i.e. for any f :

R d → R continuous with compact support, lim n→∞ R d f (z)dµ n (z) = R d f (z)dµ(z) .
(3.14)

Let f : R d → R be a continuous function with compact support and σ > 0. Consider the function f σ dened for any x ∈ R d as

f σ (x) = (2πσ 2 ) -d/2 R d f (x -z) exp - z 2 2σ 2 dLeb d (z) = f * g σ (x) ,
where * denotes the convolution product and g σ is the density of the d-dimensional Gaussian with zero mean and covariance matrix σ 2 I d . We rst show that (3.14) holds with f σ in place of f . Since for any z ∈ R d ,

E [exp(i G, z )] = exp i m, z + z 2 2σ 2
if G is a d-dimensional Gaussian random variable with zero mean and covariance matrix (1/σ 2 ) I d , then by Fubini's theorem, we get for any k ∈ N,

R d f σ (z)dµ φ(k) (z) = R d R d f (w)g σ (z -w)dwdµ φ(k) (z) = R d R d f (w)(2πσ 2 ) -d/2 R d e i z-w,x g 1/σ (x)dxdwdµ φ(k) (z) = R d R d (2πσ 2 ) -d/2 f (w)e -i w,x g 1/σ (x)Φ µ φ(k) (x)dxdw = (2πσ 2 ) -d/2 R d F[f ](x)g 1/σ (x)Φ µ φ(k) (x)dx , (3.15)
where

F[f ](x) = R d f (w)e i w,
x dw denotes the Fourier transform of f , which exists since f is assumed to have a compact support. In an analogous manner, we prove that

R d f σ (z)dµ(z) = (2πσ 2 ) -d/2 R d F[f ](x)g 1/σ (x)Φ µ (x)dx . (3.16) Now, using that F[f ] is bounded by R d |f (w)|dw < +∞ since f has compact support, we obtain that, for any k ∈ N and x ∈ R d , F[f ](x)g 1/σ (x)Φ µ φ(k) (x) ≤ g 1/σ (x) R d |f (w)|dw .
lim k→∞ R d (2πσ 2 ) -d/2 F[f ](x)g 1/σ (x)Φ µ φ(k) (x)dx = R d (2πσ 2 ) -d/2 F[f ](x)g 1/σ (x)Φ µ (x)dx lim k→∞ R d f σ (z)dµ φ(k) (z) = R d f σ (z)dµ(z) .
(3.17)

We can now complete the proof of (3.14). For any σ > 0, we have

R d f (z)dµ φ(k) (z) - R d f (z)dµ(z) ≤ 2 sup z∈R d |f (z) -f σ (z)| + R d f σ (z)dµ φ(k) (z) - R d f σ (z)dµ(z) .
Therefore, by (3.17), for any σ > 0, we get

lim sup k→+∞ R d f (z)dµ φ(k) (z) - R d f (z)dµ(z) ≤ 2 sup z∈R d |f (z) -f σ (z)| .
Finally, [Folland, 1999, Theorem 8.14-b] implies that

lim σ→0 sup z∈R d |f σ (z) -f (z)| = 0 ,
which concludes the proof.

Proof of Theorem 3.1. Now, assume that

lim k→∞ SW p (µ k , µ) = 0 (3.18)
and that (µ k ) k∈N does not converge weakly to µ. Therefore, lim k→∞ d P (µ k , µ) = 0, where d P denotes the Lévy-Prokhorov metric, and there exists > 0 and a subsequence (µ ψ(k) ) k∈N with ψ : N → N increasing, such that for any k ∈ N,

d P (µ ψ(k) , µ) > (3.19)
In addition, by Hölder's inequality, we know that W

1 (µ k , µ) ≤ W p (µ k , µ), thus SW 1 (µ k , µ) ≤ SW p (µ k , µ
), and by (3.18), lim k→∞ SW 1 (µ ψ(k) , µ) = 0. Then, according to Lemma 3.16, there exists a subsequence (µ φ(ψ(k)) ) k∈N with φ :

N → N increasing, such that µ φ(ψ(k)) w -→ µ which is equivalent to lim k→∞ d P (µ φ(ψ(k)) , µ) = 0, thus contradicts (3.19). We conclude that (3.18) implies (µ k ) k∈N w -→ µ.

Proof of Theorem 3.2

This result is proved analogously to [Bernton et al., 2019, Theorem 2.1]. The key step is to show that the function θ → SW p (μ n , µ θ ) epi-converges to θ → SW p (µ , µ θ ) P-almost surely, and then apply [Rockafellar et al., 2009, Theorem 7.31], recalled in Theorem 3.11.

Proof of Theorem 3.2. First, by A1 and Corollary 3.13, the map θ → SW p (µ, µ θ ) is l.s.c. on Θ for any µ ∈ P p (Y). Therefore, by A3, there exists θ ∈ Θ such that SW p (µ , µ θ ) = , and the set Θ is non-empty as it contains θ , closed by lower semi-continuity of θ → SW p (µ , µ θ ), and bounded. Θ is thus compact, and we conclude again by lower semi-continuity that the set argmin θ∈Θ SW p (µ , µ θ ) is non-empty [Aliprantis et al., 1999, Theorem 2.43].

Consider the event given by A2, E ∈ F such that P(E) = 1 and for any ω ∈ E, lim n→∞ SW p (μ n (ω), µ ) = 0. Then, we prove that θ → SW p (μ n , µ θ ) epi-converges to θ → SW p (µ , µ θ ) P-almost surely using the characterization in [Rockafellar et al., 2009, Proposition 7.29], i.e. we verify that, for any ω ∈ E, the two conditions below hold: for every compact set K ⊂ Θ and every open set O ⊂ Θ, lim inf

n→∞ inf θ∈K SW p (μ n (ω), µ θ ) ≥ inf θ∈K SW p (µ , µ θ ) lim sup n→∞ inf θ∈O SW p (μ n (ω), µ θ ) ≤ inf θ∈O SW p (µ , µ θ ) . (3.20) We x ω in E. Let K ⊂ Θ be a compact set. By lower semi-continuity of θ → SW p (μ n (ω), µ θ ), there exists θ n = θ n (ω) ∈ K such that for any n ∈ N, inf θ∈K SW p (μ n (ω), µ θ ) = SW p (μ n (ω), µ θn ) .
(3.21)

We consider the subsequence (μ φ(n) ) n∈N where φ :

N → N is increasing such that SW p (μ φ(n) (ω), µ θ φ(n) ) converges to lim inf n→∞ SW p (μ n (ω), µ θn ), which is equal to lim inf n→∞ inf θ∈K SW p (μ n (ω), µ θ )
by (3.21). Since K is compact, there also exists an increasing function ψ :

N → N such that, for θ ∈ K, lim n→∞ ρ Θ (θ ψ(φ(n)) , θ) = 0. Therefore, lim inf n→∞ inf θ∈K SW p (μ n (ω), µ θ ) = lim n→∞ SW p (μ φ(n) (ω), µ θ φ(n) ) = lim n→∞ SW p (μ ψ(φ(n)) (ω), µ θ ψ(φ(n)) ) = lim inf n→∞ SW p (μ ψ(φ(n)) (ω), µ θ ψ(φ(n)) )
≥ SW p (µ , µθ)

(3.22) ≥ inf θ∈K SW p (µ , µ θ ) ,
(θ n ) n∈N in O such that the sequence {SW p (µ , µ θn )} n∈N converges to inf θ∈O SW p (µ , µ θ ) as n → +∞, and for n ∈ N, inf θ∈O SW p (μ n (ω), µ θ ) ≤ SW p (μ n (ω), µ θn ), by denition of the inmum. Then, lim sup n→∞ inf θ∈O SW p (μ n (ω), µ θ ) ≤ lim sup n→∞ SW p (μ n (ω), µ θn ) ≤ lim sup n→∞ SW p (μ n (ω), µ ) + SW p (µ , µ θn ) (by the triangle inequality) ≤ lim sup n→∞ SW p (µ , µ θn ) (by A2) = inf θ∈O SW p (µ , µ θ ) (by denition of (θ n ) n∈N ) .
This shows that the second condition in (3.20) holds, and hence, the sequence of func-

tions θ → SW p (μ n (ω), µ θ ) epi-converges to θ → SW p (µ , µ θ ).
Next, we apply [Rockafellar et al., 2009, Theorem 7.31]. First, by [Rockafellar et al., 2009, Theorem 7.31(b)], (3.6) immediately follows from the epi-convergence of θ → SW p (μ n (ω), µ θ ) to θ → SW p (µ , µ θ ). We then show that [Rockafellar et al., 2009, Theorem 7.31(a)] can be applied by proving that, for any η > 0, there exists a compact set B ⊂ Θ and N ∈ N such that, for all n ≥ N ,

inf θ∈B SW p (μ n (ω), µ θ ) ≤ inf θ∈Θ SW p (μ n (ω), µ θ ) + η . (3.23)
In fact, we simply show that there exists a compact set B ⊂ Θ and N ∈ N such that, for all n ≥ N , inf

θ∈B SW p (μ n (ω), µ θ ) = inf θ∈Θ SW p (μ n (ω), µ θ ) .
On the one hand, the second condition in (3.20) gives us lim sup

n→∞ inf θ∈Θ SW p (μ n (ω), µ θ ) ≤ inf θ∈Θ SW p (µ , µ θ ) = .
We deduce that there exists n /4 (ω) such that, for n ≥ n /4 (ω),

inf θ∈Θ SW p (μ n (ω), µ θ ) ≤ + /4 , where is given by A3. As n ≥ n /4 (ω), the set Θ /2 = {θ ∈ Θ : SW p (μ n (ω), µ θ ) ≤ + 2 } is non-empty since it contains θ * dened as SW p (μ n (ω), µ θ * ) = inf θ∈Θ SW p (μ n (ω), µ θ ) .
On the other hand, by A2, there exists n /2 (ω) such that, for n ≥ n /2 (ω), SW p (μ n (ω), µ ) ≤ 2 .

(3.24)

Let n ≥ n * (ω) = max{n /4 (ω), n /2 (ω)} and θ ∈ Θ /2 . By the triangle inequality,

SW p (µ , µ θ ) ≤ SW p (μ n (ω), µ ) + SW p (μ n (ω), µ θ ) ≤ + (since θ ∈ Θ /2 and by (3.24))
This means that, when n ≥ n * (ω), Θ /2 ⊂ Θ , and since inf θ∈Θ SW p (μ n (ω),

µ θ ) is attained in Θ /2 , we have inf θ∈Θ SW p (μ n (ω), µ θ ) = inf θ∈Θ SW p (μ n (ω), µ θ ) . (3.25)
As shown in the rst part of the proof Θ is compact and then by [Rockafellar et al., 2009, Theorem 7.31(a)], (3.5) is a direct consequence of (3.23)-(3.25) and the epi-convergence

of θ → SW p (μ n (ω), µ θ ) to θ → SW p (µ , µ θ ).
Finally, by the same arguments used in this proof for argmin θ∈Θ SW p (µ , µ θ ), the set argmin θ∈Θ SW p (μ n (ω), µ θ ) is non-empty for n ≥ n * (ω).

Proof of Theorem 3.3

This result is proved analogously to [Bernton et al., 2019, Theorem 2.4]. The key step is to show that the function θ

→ E[SW p (μ n , μθ,m(n) )|Y 1:n ] epi-converges to θ → E[SW p (µ , µ θ )|Y 1:n ],
and then apply [Rockafellar et al., 2009, Theorem 7.31], which we recall in Theorem 3.11.

Proof of Theorem 3.3. Since we assume A1 and A3, we can apply the same reasoning as in the proof of Theorem 3.2 to show that argmin θ∈Θ SW p (µ , µ θ ) is a non-empty set.

Next, consider the event given by A2, E ∈ F such that P(E) = 1 and for any ω ∈ E, lim n→∞ SW p (μ n (ω), µ ) = 0. Then, we prove that θ → E[SW p (μ n , μθ,m(n) )|Y 1:n ] epiconverges to θ → SW p (µ , µ θ ) P-almost surely using the characterization of [Rockafellar et al., 2009, Proposition 7.29], i.e. we verify that, for any ω ∈ E, the two conditions below hold: for every compact set K ⊂ Θ and for every open set O ⊂ Θ,

lim inf n→+∞ inf θ∈K E SW p (μ n (ω), μθ,m(n) ) Y 1:n ≥ inf θ∈K SW p (µ , µ θ ) lim sup n→+∞ inf θ∈O E SW p (μ n (ω), μθ,m(n) ) Y 1:n ≤ inf θ∈O SW p (µ , µ θ ) (3.26) We x ω in E. Let K ⊂ Θ be a compact set. By A1 and Corollary 3.15, the mapping θ → E[SW p (μ n (ω), μθ,m(n) )|Y 1:n ] is l.s.c., so there exists θ n = θ n (ω) ∈ K such that for any n ∈ N, inf θ∈K E SW p (μ n (ω), μθ,m(n) ) Y 1:n = E SW p (μ n (ω), μθn,m(n) ) Y 1:n .
We consider the subsequence (μ φ(n) ) n∈N where φ :

N → N is increasing such that E[SW p (μ φ(n) (ω), μθ φ(n) ,m(φ(n)) )|Y 1:n ] converges to lim inf n→∞ E[SW p (μ n (ω), μθ n,m(n) )|Y 1:n ] = lim inf n→∞ inf θ∈K E[SW p (μ n (ω), μθ,m(n) )|Y 1:n ] .
Since K is compact, there also exists an increasing function ψ :

N → N such that, for any θ ∈ K, lim n→∞ ρ Θ (θ ψ(φ(n)) , θ) = 0. Therefore, lim inf n→∞ inf θ∈K E SW p (μ n (ω), μθ,m(n) ) Y 1:n = lim n→∞ E SW p (μ φ(n) (ω), μθ φ(n) ,m(φ(n)) ) Y 1:n = lim n→∞ E SW p (μ ψ(φ(n)) (ω), μθ ψ(φ(n)) ,m(ψ(φ(n))) ) Y 1:n = lim inf n→∞ E SW p (μ ψ(φ(n)) (ω), μθ ψ(φ(n)) ,m(ψ(φ(n))) ) Y 1:n ≥ lim inf n→∞ SW p (μ ψ(φ(n)) (ω), µ θ ψ(φ(n)) ) -E SW p (µ θ ψ(φ(n)) , μθ ψ(φ(n)) ,m(ψ(φ(n))) ) Y 1:n (3.27) ≥ lim inf n→∞ SW p (μ ψ(φ(n)) (ω), µ θ ψ(φ(n)) ) -lim sup n→∞ E SW p (µ θ ψ(φ(n)) , μθ ψ(φ(n)) ,m(ψ(φ(n))) ) Y 1:n ≥ SW p (µ , µθ) (3.28) ≥ inf θ∈K SW p (µ ,
E[SW p (µ θ ψ(φ(n)) , μθ ψ(φ(n)) ,m(ψ(φ(n))) )|Y 1:n ] = 0 .
We conclude that the rst condition in (3.26) holds. Now, we x O ⊂ Θ open. By denition of the inmum, there exists a sequence (θ n ) n∈N in O such that {SW p (µ , µ θn )} n∈N converges to inf θ∈O SW p (µ , µ θ ) as n → +∞, and

inf θ∈O E SW p (μ n (ω), μθ,m(n) ) Y 1:n ≤ E SW p (μ n (ω), μθn,m(n) ) Y 1:n for any n ∈ N. Then, lim sup n→∞ inf θ∈O E SW p (μ n (ω), μθ,m(n) ) Y 1:n ≤ lim sup n→∞ E SW p (μ n (ω), μθn,m(n) ) Y 1:n ≤ lim sup n→∞ SW p (μ n (ω), µ ) + SW p (µ , µ θn ) + E SW p (µ θn , μθn,m(n) ) Y 1:n (3.29) = lim sup n→∞ SW p (µ , µ θn ) (by A2 and A4) = inf θ∈O SW p (µ , µ θ ) (by denition of (θ n ) n∈N )
where (3.29) follows from the triangle inequality. This shows that the second condi-

tion in (3.26) holds, hence θ → E SW p (μ n (ω), μθ,m(n) ) Y 1:n epi-converges to θ → SW p (µ , µ θ ).
We now apply [Rockafellar et al., 2009, Theorem 7.31]. First, by [Rockafellar et al., 2009, Theorem 7.31(b)], (3.8) immediately follows from the epi-convergence of θ → E SW p (μ n (ω), μθ,m(n) ) Y 1:n to θ → SW p (µ , µ θ ). Next, we show that [Rockafellar et al., 2009, Theorem 7.31(a)] holds by nding, for any η > 0, a compact set B ⊂ Θ and N ∈ N such that, for all n ≥ N ,

inf θ∈B E SW p (μ n (ω), μθ,m(n) ) Y 1:n ≤ inf θ∈Θ E SW p (μ n (ω), μθ,m(n) ) Y 1:n + η .
In fact, we simply show that there exists a compact set B ⊂ Θ and N ∈ N such that, for all n ≥ N ,

inf θ∈B E SW p (μ n (ω), μθ,m(n) ) Y 1:n = inf θ∈Θ E SW p (μ n (ω), μθ,m(n) ) Y 1:n .
On the one hand, the second condition in (3.26) gives us lim sup

n→∞ inf θ∈Θ E SW p (μ n (ω), μθ,m(n) ) Y 1:n ≤ inf θ∈Θ SW p (µ , µ θ ) = .
We deduce that there exists n /6 (ω) such that, for n ≥ n /6 (ω),

inf θ∈Θ E SW p (μ n (ω), μθ,m(n) ) Y 1:n ≤ + 6 , with from A3. When n ≥ n /6 (ω), Θ /3 = {θ ∈ Θ : E[SW p (μ n (ω), μθ,m(n) )|Y 1:n ] ≤ + 3 } is a non-empty set as it contains θ * dened as E SW p (μ n (ω), μθ * ,m(n) ) Y 1:n = inf θ∈Θ E SW p (μ n (ω), μθ,m(n) ) Y 1:n .
On the other hand, by A2, there exists n /3 (ω) such that, for n ≥ n /3 (ω), SW p (μ n (ω), µ ) ≤ 3 .

(3.30)

Finally, by A4, there exists n /3 (ω) such that, for n ≥ n /3 (ω),

E SW p (µ θ , μθ,m(n) ) Y 1:n ≤ 3 . (3.31)
Let n ≥ n * (ω) = max{n /6 (ω), n /3 (ω), n /3 (ω)} and θ ∈ Θ /3 . By the triangle inequality, This means that, when n ≥ n * (ω), Θ /3 ⊂ Θ with Θ as dened in A3, and since

SW p (µ , µ θ ) ≤ SW p (μ n (ω), µ ) + E SW p (μ n (ω), μθ,m(n) ) Y 1:n + E SW p (µ θ , μθ,m(n) ) Y 1:n ≤ + (since θ ∈ Θ /3
inf θ∈Θ E SW p (μ n (ω), μθ,m(n) ) Y 1:n is attained in Θ /3 , we obtain inf θ∈Θ E SW p (μ n (ω), μθ,m(n) ) Y 1:n = inf θ∈Θ E SW p (μ n (ω), μθ,m(n) ) Y 1:n .
(3.32)

By [Rockafellar et al., 2009, Theorem 7.31(a)], (3.7) is a direct consequence of (3.32) and

the epi-convergence of θ → E SW p (μ n (ω), μθ,m(n) ) Y 1:n to θ → SW p (µ , µ θ ).
Finally, by the same arguments used in this proof for argmin θ∈Θ SW p (µ , µ θ ), the set argmin θ∈Θ E SW p (μ n (ω), μθ,m(n) ) Y 1:n is non-empty for n ≥ n * (ω).

Proof of Theorem 3.4

Here again, the result follows from applying [Rockafellar et al., 2009, Theorem 7.31] paraphrased in Theorem 3.11.

Proof of Theorem 3.4. First, by A1 and Corollary 3.13, the map θ → SW p (μ n , µ θ ) is l.s.c. on Θ. Therefore, there exists θ n ∈ Θ such that SW p (μ n , µ θn ) = n . The set Θ ,n with the from A5 is non-empty as it contains θ n , closed by lower semi-continuity of θ → SW p (μ n , µ θ ), and bounded. Θ ,n is thus compact, and we conclude again by lower semi-continuity that the set argmin θ∈Θ SW p (μ n , µ θ ) is non-empty [Aliprantis et al., 1999, Theorem 2.43].

Then, we prove that θ → E [SW p (μ n , μθ,m )|Y 1:n ] epi-converges to θ → SW p (μ n , µ θ ) as m → ∞ using the characterization in [Rockafellar et al., 2009, Proposition 7.29], i.e. we verify that for every compact set K ⊂ Θ and every open set O ⊂ Θ, lim inf

m→∞ inf θ∈K E [SW p (μ n , μθ,m )|Y 1:n ] ≥ inf θ∈K SW p (μ n , µ θ ) lim sup m→∞ inf θ∈O E [SW p (μ n , μθ,m )|Y 1:n ] ≤ inf θ∈O SW p (μ n , µ θ ) . (3.33)
Let K ⊂ Θ be a compact set. By A1 and Corollary 3.15, for any m ∈ N, the map θ → E[SW p (μ n , μθ,m )|Y 1:n ] is l.s.c., so there exists θ m ∈ K such that

inf θ∈K E[SW p (μ n , μθ,m )|Y 1:n ] = E[SW p (μ n , μθm,m )|Y 1:n ] .
We consider the subsequence {μ θ φ(m) ,φ(m) } m∈N where φ :

N → N is increasing such that E[SW p (μ n , μθ φ(m) ,φ(m) )|Y 1:n ] converges to lim inf m→∞ E[SW p (μ n , μθm,m )|Y 1:n ] = lim inf m→∞ inf θ∈K E[SW p (μ n , μθ,m )|Y 1:n ] .
Since K is compact, there also exists an increasing function ψ :

N → N such that for θ ∈ K, lim m→∞ ρ Θ (θ ψ(φ(m)) , θ) = 0. Therefore, lim inf m→∞ inf θ∈K E [SW p (μ n , μθ,m )|Y 1:n ] = lim m→∞ E SW p (μ n , μθ φ(m) ,φ(m) ) Y 1:n = lim m→∞ E SW p (μ n , μθ ψ(φ(m)) ,ψ(φ(m)) ) Y 1:n = lim inf m→∞ E SW p (μ n , μθ ψ(φ(m)) ,ψ(φ(m)) ) Y 1:n ≥ lim inf m→∞ [SW p (μ n , µ θ ψ(φ(m)) ) -E SW p (µ θ ψ(φ(m)) , μθ ψ(φ(m)) ,ψ(φ(m)) ) Y 1:n ] (3.34) ≥ lim inf m→∞ SW p (μ n , µ θ ψ(φ(m)) ) -lim sup m→∞ E SW p (µ θ ψ(φ(m)) , μθ ψ(φ(m)) ,ψ(φ(m)) ) Y 1:n ≥ SW p (μ n , µθ) (3.35) ≥ inf θ∈K SW p (μ n , µ θ )
where (3.34) results from the triangle inequality and (3.

≤ lim sup m→∞ E [SW p (μ n , μθm,m )|Y 1:n ] ≤ lim sup m→∞ [SW p (μ n , µ θm ) + E [SW p (µ θm , μθm,m )|Y 1:n ]] (3.36) ≤ lim sup m→∞ SW p (μ n , µ θm ) (by A4) = inf θ∈O SW p (μ n , µ θ ) (by denition of (θ m ) m∈N )
where (3.36) results from applying the triangle inequality. This shows that the second condition in (3.33) holds, hence the sequence of functions θ

→ E [SW p (μ n , μθ,m )|Y 1:n ] epi-converges to θ → SW p (μ n , µ θ ).
Now, we apply [Rockafellar et al., 2009, Theorem 7.31]. By [Rockafellar et al., 2009, Theorem 7.31(b)], (3.10) immediately follows from the epi-convergence of θ → E [SW p (μ n , μθ,m )|Y 1:n ] to θ → SW p (μ n , µ θ ). Next, we show that [Rockafellar et al., 2009, Theorem 7.31(a)] holds by nding for any η > 0 a compact set B ⊂ Θ and N ∈ N such that, for all n ≥ N ,

inf θ∈B E [SW p (μ n , μθ,m )|Y 1:n ] ≤ inf θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] + η .
In fact, we simply show that there exists a compact set B ⊂ Θ and N ∈ N such that inf θ∈B E [SW p (μ n , μθ,m )|Y 1:n ] = inf θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] for all n ≥ N . On one hand, the second condition in (3.33) gives us lim sup

m→∞ inf θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] ≤ inf θ∈Θ SW p (μ n , µ θ ) = n .
We deduce that there exists m /4 such that, for m ≥ m /4 ,

inf θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] ≤ n + 4 , (3.37) with from A5. When m ≥ m /4 , Θ /2 = {θ ∈ Θ : E[SW p (μ n , μθ,m )|Y 1:n ] ≤ n + 2 } is a non-empty set, as it contains θ * dened as E[SW p (μ n , μθ * ,m )|Y 1:n ] = inf θ∈Θ E[SW p (μ n , μθ,m )|Y 1:n ] .
On the other hand, by A4, there exists m /2 such that, for m ≥ m /2 ,

E [SW p (µ θ , μθ,m )|Y 1:n ] ≤ 2 . (3.38) Let θ ∈ Θ /2 and m ≥ m * = max{m /4 , m /2 }. By the triangle inequality, SW p (μ n , µ θ ) ≤ E [SW p (μ n , μθ,m )|Y 1:n ] + E [SW p (µ θ , μθ,m )|Y 1:n ] ≤ n + (since θ ∈ Θ /2 and by (3.38)) Therefore, when m ≥ m * , Θ /2 ⊂ Θ ,n , and since inf θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] is at- tained in Θ /2 , inf θ∈Θ ,n E [SW p (μ n , μθ,m )|Y 1:n ] = inf θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] .
(3.39)

By [Rockafellar et al., 2009, Theorem 7.31(a)], (3.9) is a direct consequence of (3.39) and the epiconvergence of θ → E [SW p (μ n (ω), μθ,m )|Y 1:n ] to θ → SW p (μ n , µ θ ).

Finally, by the same arguments used in this proof for argmin θ∈Θ SW p (μ n , µ θ ), the set argmin θ∈Θ E [SW p (μ n , μθ,m )|Y 1:n ] is non-empty for m ≥ m * .

Proof of Theorem 3.5

Let us rst formally establish measurability for MESWE.

Theorem 3.17 (Measurability of the MESWE). Assume A1. For any n ≥ 1, m ≥ 1 and > 0, there exists a Borel measurable function θn,m, : Ω → Θ that satises for any

ω ∈ Ω, θn,m, (ω) ∈    argmin θ∈Θ E [SW p (μ n (ω), μθ,m )|Y 1:n ] , if this set is non-empty, θ ∈ Θ : E [SW p (μ n (ω), μθ,m )|Y 1:n ] ≤ * + } , otherwise, where * = inf θ∈Θ E[SW p (μ n (ω), μθ,m )|Y 1:n ].
We prove the measurability of MSWE and MESWE by verifying the conditions of [Brown and Purves, 1973, Corollary 1], which are recalled below.

Theorem 3.18 (Corollary 1 in Brown and Purves [1973]). Let U, V be Polish spaces and f be a real-valued Borel measurable function dened on a Borel subset D of U × V. We denote by proj(D) the set dened as proj(D) = {u : there exists v ∈ V, (u, v) ∈ D} .

Suppose that for each u ∈ proj(D), the section D u = {v ∈ V, (u, v) ∈ D} is σ-compact and f (u, •) is lower semi-continuous with respect to the relative topology on D u . Then, 1. The sets proj(D) and

I = {u ∈ proj(D), for some v ∈ D u , f (u, v) = inf f u } are Borel 2.
For each > 0, there is a Borel measurable function φ satisfying, for u ∈ proj(D),

f (u, φ (u)) = inf Du f u , if u ∈ I, ≤ + inf Du f u , if u / ∈ I, and inf Du f u = -∞ ≤ --1 , if u / ∈ I, and inf Du f u = -∞ .
Proof of Theorems 3.5 and 3.17. We start by proving Theorem 3.5. The empirical measure μn (ω) depends on ω ∈ Ω only through y = (y 1 , . . . , y n ) ∈ Y n , so we can consider it as a function on Y n rather than on Ω. We introduce D = Y n × Θ. Since Y is Polish, Y n (n ∈ N * ) endowed with the product topology is Polish. For any y ∈ Y n , the set D y = {θ ∈ Θ, (y, θ) ∈ D} = Θ is assumed to be σ-compact.

The map y → μn (y) is continuous for the weak topology (see the proof of Lemma 3.14), as well as the map θ → µ θ according to A1. We deduce by Corollary 3.13 that the map (µ, θ) → SW p (µ, µ θ ) is l.s.c. for the weak topology. Since the composition of a lower semi-continuous function with a continuous function is l.s.c., the map (y, θ) → SW p (μ n (y), µ θ ) is l.s.c. for the weak topology, thus measurable and for any y ∈ Y n , θ → SW p (μ n (y), µ θ ) is l.s.c. on Θ. A direct application of Theorem 3.18 nalizes the proof. Theorem 3.17 can be proved via the same methodology: we verify that we can apply Theorem 3.18 using Corollary 3.15 instead of Corollary 3.13.

3.5.7 Proof of Theorems 3.6 and 3.7

The proof of Theorem 3.6 and Theorem 3.7 consists in showing that the conditions of [Pollard, 1980, Theorems 4.2] and [Pollard, 1980, Theorem 7.2] respectively are satised: conditions (i), (ii) and (iii) follow from A6, A7 and A8.

Additional details on Section 3.3

Sampling schemes. We rst explain the methods that we used to generate i.i.d. samples from the uniform distribution on S d-1 (required for the Monte Carlo estimate of SW (2.21)) and multivariate elliptically contoured stable distributions (for Section 3.3.2).

• Uniform sampling on the sphere. To sample from S d-1 , we form the ddimensional vector s by drawing each of its d components from the standard normal distribution N (0, 1) and we normalize it, i.e. s = s s 2 , so that s lies on the sphere.

• Sampling from multivariate elliptically contoured stable distributions.

We recall that if Y ∈ R d is α-stable and elliptically contoured, i.e. Y ∼ EαS c (Σ, m), then its joint characteristic function is dened as, for any t ∈ R d ,

E[exp(it T Y )] = exp -(t T Σt) α/2 + it T m , (3.40)
where Σ is a positive denite matrix (akin to a correlation matrix), m ∈ R d is a location vector (equal to the mean if it exists) and α ∈ (0, 2) controls the thickness of the tail. Elliptically contoured stable distributions are scale mixtures of multivariate Gaussian distributions [Samorodnitsky and Taqqu, 1994, Proposition 2.5.2], whose densities are intractable, but can easily be simulated [START_REF] Nolan | Multivariate elliptically contoured stable distributions: theory and estimation[END_REF]:

let A ∼ S α/2 (β, γ, δ) be a one-dimensional positive (α/2)-stable random variable with β = 1, γ = 2 cos( πα 4 ) 2/α and δ = 0, and G ∼ N (0, Σ). Then,

Y = √ AG + m has (3.40) as characteristic function.
Optimization methods. By denition, computing the M(E)SWE and implies minimizing the (expected) Sliced-Wasserstein distance over the set of parameters, which is in general computationnally intractable. We then resort to numerical methods in our experiments to approximate these two estimators, as we detail below.

• Multivariate Gaussian distributions. We derive the explicit gradient expressions of the approximate SW 2 2 distance with respect to the mean and scale param- eters m and σ 2 , and we use the ADAM stochastic optimization method with the default parameter settings suggested in [START_REF] Kingma | A method for stochastic optimization[END_REF]. For the MSWE, we use (2.14) to approximate the one-dimensional Wasserstein distance, and we evaluate directly the Gaussian density of the generated samples, utilizing the fact that the projection of a Gaussian of parameters (m, σ 2 I) along u ∈ S d-1 is a 1D normal distribution of parameters ( u, m , σ 2 u, u ). In this case, the gradient of the approximate SW 2 2 between µ = N (m, σ 2 I) and the empirical distribution associated to n samples drawn by N (m , σ 2 I), denoted by ν, is given by,

∇ m SW 2 2 (µ, ν) =(1/ card(U) card(S)) u∈U,s∈S s -F -1 u ν ( Fu µ (s)) 2 N (s; u, m , σ 2 u 2 ) s -u, m σ 2 u 2 u , ∇ σ 2 SW 2 2 (µ, ν) =(1/ card(U) card(S)) u∈U,s∈S s -F -1 u ν ( Fu µ (s)) 2 N (s; u, m , σ 2 u 2 ) 1 2σ 2 (s -u, m ) 2 σ 2 u 2 -1 ,
where U ⊂ S d-1 is a nite set of random projections picked uniformly on S d-1 , S is a nite subset in R, and for any s ∈ S, N (s; u, m , σ 2 u 2 ) denotes the density function of the Gaussian of parameters ( u, m , σ 2 u 2 ) evaluated at s.

For the MESWE, we use (2.13) and evaluate the empirical distribution of generated samples instead of their normal density. Therefore, the gradient of the approximate SW 2 2 between the empirical distributions corresponding to one generated dataset of m samples drawn from N (µ, σ 2 I) and n samples drawn from N (µ , σ 2 I), respectively denoted by μ and ν, is obtained with,

∇ m SW 2 2 (μ, ν) = -2 card(U).K u∈U K k=1 F -1 u μ(t k ) -F -1 u ν (t k ) u , ∇ σ 2 SW 2 2 (μ, ν) = 1 card(U).K u∈U K k=1 F -1 u μ(t k ) -F -1 u ν (t k ) u, m -F -1 u μ(t k ) σ 2 .
• Multivariate elliptically contoured stable distributions. When comparing MESWE to MEWE, we approximate these estimators using the derivative-free optimization method Nelder-Mead (Nelder and Mead [1965], implemented in Scipy),

following the approach in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF].

When illustrating the theoretical properties of MESWE, we proceed in the same way as for the multivariate Gaussian experiment: we compute the explicit gradient expression of the approximate SW 2 2 distance with respect to the location parameter m, and we use the ADAM stochastic optimization method with the default settings. Equation (3.41) gives the formula of the gradient of the approximate SW 2 2 between the empirical distributions of one generated dataset of m samples drawn from EαS c (I, m) and n samples drawn from EαS c (I, m ), respectively denoted by μ and ν, with respect to m.

∇ m SW 2 2 (μ, ν) = -2 card(U).K u∈U K k=1 F -1 u μ(t k ) -F -1 u ν (t k ) u .
(3.41)

• High-dimensional real data using GANs. We use the ADAM optimizer provided by TensorFlow GPU.

Computing infrastructure. In this chapter, we propose a new ABC technique, called Sliced-Wasserstein ABC and relying on the Sliced-Wasserstein distance, which has better computational and statistical properties. We derive two theoretical results showing the asymptotical consistency of our approach, and we illustrate its advantages on synthetic data and an image denoising task.

Introduction

Consider the problem of estimating the posterior distribution of some model parameters θ ∈ R d θ given n data points y 1:n ∈ Y n . This distribution has a closed-form expression

given by the Bayes' theorem up to a multiplicative constant, π(θ|y

1:n ) ∝ π(y 1:n |θ)π(θ) .
For many statistical models of interest, the likelihood π(y 1:n |θ) cannot be numerically evaluated in a reasonable amount of time, which prevents the application of classical likelihood-based approximate inference methods. Nevertheless, in various settings, even if the associated likelihood is numerically intractable, one can still generate synthetic data given any model parameter value. This generative setting gave rise to an alternative framework of likelihood-free inference techniques. Among them, Approximate Bayesian Computation methods [START_REF] Tavaré | Inferring Coalescence Times From DNA Sequence Data[END_REF][START_REF] Beaumont | Approximate Bayesian Computation in Population Genetics[END_REF] have become a popular choice and have proven useful in various practical applications, e.g.

[ [START_REF] Peters | Bayesian inference, Monte Carlo sampling and operational risk[END_REF][START_REF] Tanaka | Using Approximate Bayesian Computation to Estimate Tuberculosis Transmission Parameters From Genotype Data[END_REF][START_REF] Wood | Statistical inference for noisy nonlinear ecological dynamic systems[END_REF]. The core idea of ABC is to bypass calculation of the likelihood by using simulations: the exact posterior is approximated by retaining the parameter values for which the synthetic data are close enough to the observations. Closeness is usually measured with a discrepancy measure between the two datasets reduced to some `summary statistics' (e.g., empirical mean or empirical covariance). While summaries allow a practical and ecient implementation of ABC, especially in high-dimensional data spaces, the quality of the approximate posterior distribution highly depends on them and constructing sucient statistics is a non-trivial task. Summary statistics can be designed by hand using expert knowledge, which can be tedious especially in real-world applications, or in an automated way, for instance see [START_REF] Fearnhead | Constructing summary statistics for approximate Bayesian computation: semi-automatic approximate Bayesian computation[END_REF].

Recently, discrepancy measures that view data sets as empirical probability distributions to eschew the construction of summary statistics have been proposed for ABC.

Examples include the Kullback-Leibler divergence [START_REF] Jiang | Approximate Bayesian Computation with Kullback-Leibler Divergence as Data Discrepancy[END_REF], maximum mean discrepancy [START_REF] Park | K2-ABC: Approximate Bayesian Computation with Kernel Embeddings[END_REF], and Wasserstein distance [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF]]. As we discussed in Chapters 1 and 2, this latter distance emerging from optimal transport theory has attracted abundant attention in statistics and machine learning, due to its strong theoretical properties and applications on many domains. In particular, we recall that it has the ability of making meaningful comparisons even between probability measures with non-overlapping supports, unlike KL. However, the computational complexity of the Wasserstein distance rapidly becomes a challenge when the dimension of the observations is large, and several numerical methods have been proposed during the past few years to speed-up this computation. In particular, Wasserstein-ABC (WABC, [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF]) introduces a dierent computational approach to those presented in Chapter 2: the Wasserstein distance is estimated with a novel approximation based on the Hilbert space-lling curve and termed the Hilbert distance, which is computationally ecient but accurate for small dimensions only. Besides, under a general setting, the Wasserstein distance suers from a curse of dimensionality in the sense that the error made when approximating it from samples grows exponentially fast with the data space dimension (Section 2.4.2). These computational and statistical issues can strongly aect the performance of WABC applied to high-dimensional data.

Building on the computational eciency of SW and its successful performance in generative settings, as demonstrated by prior studies (reviewed in Section 1.3) and our previous chapter, we develop a novel ABC framework that uses SW as the data discrepancy measure. This denes a likelihood-free method which does not require choosing summary statistics and is ecient even with high-dimensional observations, thus overcoming the limitations of WABC. We derive asymptotical guarantees on the convergence of the resulting ABC posterior, and we illustrate the superior empirical performance of our methodology by applying it on a synthetical problem and an image denoising task.

Background on Approximate Bayesian Computation

In this chapter, we consider the same purely generative modeling framework as in Chapter 3, so we keep the formalism and notations presented in Section 3.1. Additionally, we assume that conditions (C1), (C2) and (C4) hold. However, instead of using minimum distance estimation to perform parameter inference in such models, we focus on another class of approximate inference methods, called Approximation Bayesian Computation Algorithm 2: Vanilla ABC.

Input: observations y 1:n , number of iterations T , data discrepancy measure D, summary statistics s, tolerance threshold ε > 0. for t = 1, . . . , T do repeat θ ∼ π(•) and z 1:m ∼ µ θ i.i.d. until D s(y 1:n ), s(z 1:m ) ≤ ε; θ (t) = θ return θ (1) , . . . , θ (T ) algorithms.

ABC methods are used to approximate the posterior distribution in generative models when the likelihood is numerically intractable but easy to sample from. The basic and simplest ABC algorithm is an acceptance-rejection method [START_REF] Tavaré | Inferring Coalescence Times From DNA Sequence Data[END_REF], which iteratively draws a candidate parameter θ from a prior distribution π, and `synthetic data' z 1:m = (z i ) m i=1 from µ θ , and keeps θ if z 1:m is close enough to the observations y 1:n = (y i ) n i=1 . Specically, the acceptance rule is

D s(y 1:n ), s(z 1:m ) ≤ ε , (4.1)
where D is a data discrepancy measure taking non-negative values, ε is a tolerance threshold, and s : n∈N * Y n → R ds with small d s is a summary statistics. The algorithm is summarized in Algorithm 2 and returns samples of θ that are distributed from:

π ε y 1:n (θ) = π(θ) Y m 1{D s(y 1:n ), s(z 1:m ) ≤ ε}dµ θ (z 1:m ) Θ dπ(θ) Y m 1{D s(y 1:n ), s(z 1:m ) ≤ ε}dµ θ (z 1:m ) (4.2)
The choice of s(•) directly impacts the quality of the resulting approximate posterior: if the statistics are sucient statistics, π ε y 1:n (θ) converges to the true posterior π(θ|y 1:n ) as ε → 0, otherwise, the limiting distribution is at best π(θ|s(y 1:n )) [START_REF] Sisson | Overview of Approximate Bayesian Computation[END_REF], Frazier et al., 2018]. Wasserstein-ABC has then been proposed to avoid this loss of information.

Wasserstein distance and ABC. Wasserstein-ABC [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF] is a variant of ABC (2) that uses W p , p ∈ [1, +∞) between the empirical distributions of the observed and synthetic data, in place of the discrepancy measure D between summaries.

To make this method scalable to any dataset size, [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF] introduces a new approximation of the Wasserstein distance, called the Hilbert distance, which extends the idea behind the computation of W p in 1D to higher dimensions, by sorting samples according to their projection obtained via the Hilbert space-lling curve. This alternative can be computed in O(n log(n)), but yields accurate approximations only for low dimensions, as emphasized in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF]. The same work also uses a second approximation, the swapping distance, based on an iterative greedy swapping algorithm. However, each iteration requires n 2 operations, and there is no guarantee of convergence to W p .

4.3

Sliced-Wasserstein ABC

Given the computational and statistical issues caused by the Wasserstein distance, we state that the ABC framework can benet from using an alternative computational OT metric, especially on high-dimensional settings. We consider the Sliced-Wasserstein distance based on the computational eciency of its Monte Carlo estimate, as well as its statistical advantages over the Wasserstein distance and the Hilbert and swapping approximations.

We illustrate the latter statistical aspect on the task of estimating the scaling factor of the covariance matrix in a multivariate Normal model, as in the supplementary material of [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF]. For any σ > 0, denote by µ σ the d-dimensional Gaussian distribution with zero-mean and covariance matrix σ 2 I d . Observations are assumed i.i.d. from µ σ with σ 2 = 4, and we draw the same number of i.i.d. data from µ σ for 100 values of σ 2 equispaced between 0.1 and 9. We then compute W 2 and SW 2 between the empirical distributions of the samples, and the swapping and Hilbert approximations presented in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF], for d ∈ {2, 10, 100} and 1000 observations. We know that W 2 between two Gaussian measures has an analytical formula, which boils down in our setting to

W 2 2 (µ σ , µ σ ) = d(σ -σ) 2 , (4.3)
and we approximate the exact SW using a Monte Carlo approximation of

SW 2 2 (µ σ , µ σ ) = W 2 2 (µ σ , µ σ ) S d-1 u T u dσ(u) , (4.4)
This formula (4.4) follows from Denition 2.9 and (4.3). We approximate KL with the estimator proposed for KL-based ABC (KL-ABC, [START_REF] Jiang | Approximate Bayesian Computation with Kullback-Leibler Divergence as Data Discrepancy[END_REF]).

Figure 4.1 shows the distances plotted against σ 2 for each d. When the dimension increases, we observe that (i) as pointed out in [START_REF] Bernton | On parameter estimation with the Wasserstein distance[END_REF], the quality of the approximation of empirical Wasserstein returned by Hilbert and swapping rapidly deteriorates, and (ii) SW, approximated using densities or samples, is the only approximate metric that attains its minimum at σ 2 . This curse of dimensionality can be a limiting factor for the performance of WABC and KL-ABC in high dimensions.

Motivated by the practical success of SW regardless of the dimension value in the previous experiment, we propose a variant of ABC based on SW, referred to as Sliced-Wasserstein ABC (SW-ABC). Our method is similar to WABC in the sense that it compares empirical distributions, but instead of W p , we choose the discrepancy measure to be SW p , p ∈ [1, +∞). The usage of SW allows the method to scale better to the data size and dimension. The resulting posterior distribution, called the SW-ABC posterior, is thus dened in (4.2) with D replaced by SW p .

Theoretical Study

In this section, we analyze the asymptotic behavior of the SW-ABC posterior under two dierent regimes. Our rst result concerns the situation where the observations y 1:n are xed, and ε goes to zero. We prove that the SW-ABC posterior is asymptotically consistent in the sense that it converges to the true posterior, under specic assumptions on the density used to generate synthetic data.

Proposition 4.1. Let p ∈ [1, +∞). Suppose that µ θ has a density f θ w.r.t. the Lebesgue measure such that f θ is continuous and there exists N Θ ⊂ Θ satisfying π(N Θ ) = 0 and

sup θ∈Θ\N Θ f θ (y 1:n ) < ∞ .
In addition, assume that there exists ε > 0 such that,

sup θ∈Θ\N Θ sup z 1:m ∈A ε f θ (z 1:m ) < ∞ ,
where A ε = {z 1:m : SW p (y 1:n , z 1:m ) ≤ ε}. Then, with y 1:n xed, the SW-ABC posterior converges to the true posterior as ε goes to 0, in the sense that, for any measurable

B ⊂ Θ, lim ε→0 π ε y 1:n (B) = π(B|y 1:n ) ,
where π ε y 1:n is dened by (4.2).

The proof of Proposition 4.1 is provided in Section 4.7 and consists in applying [Bernton et al., 2019, Proposition 3.1].

Next, we study the limiting SW-ABC posterior when the value of ε is xed and the number of observations increases, i.e. n → ∞. We suppose that the size m of the synthetic dataset grows to αn with α > 0, such that m can be written as a function of n, m(n), satisfying lim n→∞ m(n) = ∞. We show that, under this setting and appropriate additional conditions, the resulting approximate posterior converges to the prior distribution on θ restricted to the region {θ ∈ Θ : SW p (µ θ , µ θ ) ≤ ε}.

Proposition 4.2. Let p ∈ [1, +∞), ε > 0 and (m(n)) n∈N * be an increasing sequence satisfying lim n→∞ m(n)/n = α, for α > 0. Assume that the statistical model M Θ is well specied, i.e. there exists θ ∈ Θ such that µ = µ θ , and that almost surely the following holds.

lim n→∞ SW p (μ n , μθ,m(n) ) = SW p (µ θ , µ θ ) , (4.5)
where μn , μθ,m(n) denote the empirical distributions of the observations y 1:n and synthetic data z 1:m(n) respectively. Then, the SW-ABC posterior converges to the restriction of the prior π on the region {θ ∈ Θ : SW p (µ θ , µ θ ) ≤ ε} as n → ∞, i.e. for any θ ∈ Θ, Each strategy uses 1000 particles and are run for 3 hours max. First row shows ABC and true posteriors of σ 2 , second row reports W 1 -distance to true posterior vs. time.

lim n→∞ π ε y 1:n (θ) = π(θ|SW p (µ θ , µ θ ) ≤ ε) ∝ π(θ)1{SW p (µ θ , µ θ ) ≤ ε} .
SW is approximated with its MC estimate over 100 random projections.

Proposition 4.2 follows from the application of [Jiang et al., 2018, Theorem 1] to SW p and the required conditions. Note that condition (4.5) is a mild assumption, e.g. is fullled if Y is compact and separable: in this case, for any ν ∈ P p (Y) and its empirical instantiation νn , lim n→∞ W p (ν, νn ) = 0 ν-almost surely [START_REF] Weed | Sharp asymptotic and nite-sample rates of convergence of empirical measures in Wasserstein distance[END_REF], then lim n→∞ SW p (ν, νn ) = 0 ν-almost surely [Bonnotte, 2013, Proposition 5.1.3], and (4.5)

follows by applying the triangle inequality.

Experiments

Synthetic experiments

As a rst set of experiments, we investigate the performance of SW-ABC on a synthetical setting where the posterior distribution is analytically available. We consider n = 100 observations (y i ) n i=1 i.i.d. from a d-dimensional Gaussian N (m , σ 2 I d ), with m ∼ N (0, I d ) and σ 2 = 4. The parameter θ is σ 2 for which the prior distribution is assigned to be an inverse gamma distribution IG(1, 1). Therefore, the posterior distribution of σ 2 given (y i ) n i=1 and m is an inverse gamma distribution as well, whose parameters are given by

IG 1 + nd 2 , 1 + 1 2 n i=1 y i -m 2 .
We compare SW-ABC against ABC using the Euclidean distance between sample variances (Euclidean-ABC), WABC with the Hilbert distance, WABC with the swapping distance and KL-ABC. Each ABC approximation was obtained using the sequential Monte Carlo sampler-based ABC method [Toni et al., 2009], which is computationally more ecient than vanilla ABC (Algorithm 2) and implemented in the package pyABC [START_REF] Klinger | pyABC: distributed, likelihood-free inference[END_REF]. We provide the code to reproduce our empirical results 1 .

1 See our GitHub repository: https://github.com/kimiandj/slicedwass_abc relies on statistics that are sucient in our setting. WABC performs poorly with highdimensional observations, contrary to SW-ABC, which approximates well the posterior for each dimension value and is as fast.

Application to image denoising

We now evaluate our approach on a real application, namely image denoising. We consider a widely used algorithm for this task, the Non-Local Means algorithm (NL-means, [START_REF] Buades | A non-local algorithm for image denoising[END_REF]), and we present a novel variant of it derived from SW-ABC.

We formally dene the denoising problem as follows. Let u ∈ R M ×N , denote a clean gray-level image. We observe a corrupted version of this image, v = u + w, where w is some noise in R M ×N . The goal is to restore u from v. We focus on denoising methods that consider `patch-based representations' of images, e.g. NL-means. Specically, let r ∈ N be a patch size and I = {1, . . . , M } × {1, . . . , N } the set of pixel positions. For i ∈ I, u (i) denotes the pixel value at position i in image u , and P i is a (2r + 1) × (2r + 1) window in v centered at i: for k ∈ {-r, . . . , r} 2 , P i (k) = v(i + k), where v is extended to Z 2 by periodicity. Let D ⊂ I be a dictionary of positions, and φ : I → D such that, for i ∈ I, φ(i) = argmin j∈D P i -P j 2 , i.e. φ(i) is the position in D of the most similar patch to P i . For j ∈ D, an estimator of P j is given by Pj = E π(i|(P k ) k∈φ -1 (j) )π(l) [P i+l ], π being the uniform distribution on φ -1 (j).

In practice, it is approximated with a Monte Carlo scheme,

Pj ≈ (T n) -1 T t=1 S s=1 P i (t) +l (s) , (4.6)
where i (t) ∼ π(i (t) |(P k ) k∈φ -1 (j) ), l (s) ∼ π(l), and i, l are mutually independent. Finally, we construct an estimate û of u as follows: for any i ∈ I,

û(i) = k, k-i ∞≤r Pφ(k) (i -k) (2r + 1) -2 .
The classical NL-means estimator corresponds to the case where D = I (thus φ = Id)

and for any i ∈ I and P ∈ R (2r+1)×(2r+1) , π(i, P ) ∝ 1 W (i)e -P -P i 2 /(2σ 2 ) , where W is a search window.

In our work, we assume that the likelihood π(P |i) is not available, but we observe for j ∈ D, (P k ) m =1 (k ∈ φ -1 (j)) i.i.d. from π(•|i). By replacing π(i|(P k ) m =1 ) in (4.6) by the SW-ABC posterior, we obtain the proposed denoising method, called the SW-ABC NL-means algorithm. We provide the Python implementation of our algorithm 2 . We compare our approach with the classical NL-means. We consider one of the standard image denoising datasets [START_REF] Fan | Brief review of image denoising techniques[END_REF], called CBSD68 [START_REF] Martin | A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[END_REF] 2 See https://vdeborto.github.io/publication/sw_abc σ = 10 σ = 20 σ = 30 σ = 50 255 2 N M .

In our experiments, we use a dictionary of 1000 patches picked uniformly at random, we set T = S = m = 10, r = 3, W = {-10, . . . , 10} 2 , ε = (2r + 1) 2 , and we compute SW with a MC scheme over L = 100 projections. We report the average PSNR values for dierent values of the noise level σ in Table 4.1. We observe that for small σ, NL-means provides more accurate results, whereas when σ becomes larger SW-ABC outperforms NL-means, thanks to the patch representation and the use of SW.

On the other hand, another important advantage of SW-ABC becomes prominent in the computation time: the proposed approach takes ≈ 6s on a standard laptop computer per image whereas the classical NL-means algorithm takes ≈ 30s. Indeed, the computational complexity of SW-ABC NL-means is upper-bounded by card(D)T S C SW , where C SW = Lm log(m) is the cost of computing SW, and for the naïve implementation of NL-means, it is given by N M card(W )(2r + 1) 2 . We can observe that SW-ABC has a lower computational complexity since card(D) N M in practice. We note that the computation time of NL-means can be improved by certain acceleration techniques, which can be directly used to improve the speed of SW-ABC NL-means as well.

Finally, in Figure 4.3, we illustrate the performance of SW-ABC on two 512 × 512

images for visual inspection. The results show that the injected noise is successfully removed by the proposed approach.

Conclusion

In this chapter, we explored other applications where the Sliced-Wasserstein distance can be useful, and proposed a novel ABC method, SW-ABC, based on this metric. We derived asymptotic guarantees for the convergence of the SW-ABC posterior to the true posterior under dierent regimes, and we evaluated our approach on a synthetical and an image denoising problem. Our results showed that SW-ABC provides an accurate approximation of the posterior, even with high-dimensional data spaces and a small number of samples. This amounts to verify that:

(i) For any y 1:n and z 1:m , with respective empirical distributions μn and μθ,m , SW p (μ n , μθ,m ) = 0 if and only if μn = μθ,m .

(ii) SW p is continuous in the sense that, if (z k 1:m ) k∈N converges to z 1:m in the metric ρ, then, for any empirical distribution μn , lim k→∞ SW p (μ n , μk θ,m ) = SW p (μ n , μθ,m ), where μk θ,m is the empirical measure of z k 1:m .

Condition (i) follows from the fact that SW p is a distance [Bonnotte, 2013, Proposition 5.1.2]. Now, let y ∈ Y and ψ : Y → R be a continuous function such that for any y ∈ Y, |ψ(y)| ≤ K 1 + ρ(y , y) p with K ∈ R. Since (z k 1:m ) k∈N converges to z 1:m in the metric ρ and ψ is continuous, we get that lim k→∞ ψ dμ k θ,m = ψ dμ θ,m . This implies that μk θ,m weakly converges to μθ,m in P p (Y) [Villani, 2008, Denition 6.7], which, by [Villani, 2008, Theorem 6.8], is equivalent to lim k→∞ W p (μ k θ,m , μθ,m ) = 0. By applying the triangle inequality and [Bonnotte, 2013, Proposition 5.1.3], there exists C ≥ 0 such that, for any empirical measure μn ,

|SW p (μ n , μk θ,m ) -SW p (μ n , μθ,m )| ≤ SW p (μ k θ,m , μθ,m ) ≤ C 1/p W p (μ k θ,m , μθ,m ) .
We conclude that lim k→∞ SW p (μ n , μk θ,m ) = SW p (μ n , μθ,m ), making condition (ii) applicable.

Chapter 5

Generalized Sliced Wasserstein

Distances

This chapter is based on [Kolouri et al., 2019a].

The Wasserstein distance and its practical alternatives have recently attracted a lot of attention from the machine learning community. The Sliced-Wasserstein distance, specically, was shown to have similar theoretical properties to the Wasserstein distance, while being much simpler to compute thanks to its Monte Carlo approximation. SW has therefore been used in various applications, including generative modeling and general supervised/unsupervised learning, but its performance might suer from the error induced by the Monte Carlo estimation. This limitation has thus motivated the formulation of alternatives, such as the maximum Sliced-Wasserstein distance (max-SW).

In this chapter, we propose another method to address this issue, by dening a novel family of probability divergences that extends the idea behind SW. We rst clarify the mathematical connection between SW and the Radon transform, then leverage the generalized Radon transform to formulate the class of generalized Sliced-Wasserstein distances (GSW). We also formulate a generalization of max-SW, called the maximum generalized Sliced-Wasserstein distances (max-GSW). We identify some conditions on the generalized Radon transform under which GSW and max-GSW satisfy the metric axioms. Finally, we compare the empirical performance of the proposed distances on dierent implicit generative modeling problems to illustrate their advantages over SW.

Introduction

To compare two probability distributions µ and ν supported on R d with the Sliced-Wasserstein distance, one needs to collect linear projections of µ and ν along all possible directions on S d-1 , which is done by computing the push-forward measures θ µ and θ ν for any θ ∈ S d-1 . As we will detail in the next section, these push-forward measures are closely related to the Radon transform [Rabin et al., 2012, Proposition 6], which is widely used in tomography [START_REF] Radon | Uber die bestimmug von funktionen durch ihre integralwerte laengs geweisser mannigfaltigkeiten[END_REF][START_REF] Helgason | The Radon transform on Rn[END_REF]. In practice, unless an analytical formula is known, the integral that denes SW (Denition 2.9) is usually approximated with a Monte Carlo strategy, which computes an average over a nite number of directions uniformly picked at random on S d-1 . Intuitively, the linear nature of these projections does not guarantee an ecient approximation of the Sliced-Wasserstein distance: since in very high-dimensional settings, the data often lives in a thin manifold, one might have to sample a very large number of directions to eectively capture the structure of the data distribution, which can be very expensive according to our discussion on the complexity of SW in Section 2.6. Reducing the number of required projections would thus result in a signicant performance improvement.

To alleviate the ineciencies caused by the linear projections, several attempts have recently been made: linear projections can be combined with orthogonal coupling in Monte Carlo estimation to increase computational eciency and estimation quality [START_REF] Rowland | Orthogonal Estimation of Wasserstein Distances[END_REF][START_REF] Wu | Sliced Wasserstein Generative Models[END_REF]. Alternatively, [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF] extended SW to the maximum Sliced-Wasserstein distance, where the integral over S d-1 in (2.20) is replaced by a maximum operator so that one retains the most informative projection direction. In this context, the information returned by a direction θ ∈ S d-1 is measured by W p (θ µ, θ ν): the larger this Wasserstein distance, the more informative θ. This idea is also reected in another study [START_REF] Paty | Subspace Robust Wasserstein distances[END_REF], which considers k-dimensional projections of µ, ν with k ∈ {1, . . . , d}, and aims at nding the most informative subspace on which µ and ν are being projected. While these methods reduce the computational cost by requiring a lower number of projections, they incur an additional cost due to the resolution of a non-convex optimization problem over manifolds.

In this chapter, we address the computational limitations of the Sliced-Wasserstein distance by taking an alternative route: we allow the projections of the compared distributions µ and ν to be non-linear. More precisely, we use the theory of the generalized Radon transform [START_REF] Beylkin | The inversion problem and applications of the generalized Radon transform[END_REF] to extend the denition of SW to an entire class of probability divergences, which we call generalized Sliced-Wasserstein distances (GSW).

We then show that, similar to [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF], we can formulate a metric that relies on the most informative projection instead of innitely many projections. We aptly call this distance the maximum generalized Sliced-Wasserstein distance (max-GSW). We prove that replacing the linear projections with non-linear projections can still yield a valid metric on the space of probability distributions: we identify general conditions under which GSW and max-GSW satisfy the metric axioms recalled in Denition 2.1.

As instances of non-linear projections, we rst investigate projections with polynomial kernels, which meet all the conditions that we identied. However, we observe that the memory complexity required by such projections has a combinatorial growth with respect to the dimension of the problem, which hinders their applications to modern ML problems, such as IGM. This motivates us to consider a neural-network-based projection scheme, where we observe that fully connected or convolutional networks with leaky ReLU activations fulll the crucial conditions for the resulting GSW to be a pseudometric.

Due to their inherent non-linearity, GSW and max-GSW are expected to capture the complex structure of high-dimensional distributions by using much less projections.

Besides, the use of deep learning techniques additionally allows the projections to be data-adaptive. For these reasons, we expect our metrics to reduce the iteration complexity in a signicant amount. We verify this intuition in our experiments, where we illustrate the superior performance of the proposed generalized distances in IGM problems, with both synthetic and real data.

Background on the Radon transform

We review in this section the Radon transform and explain how it enables the denition of SW and max-SW, accordingly to [START_REF] Rabin | Wasserstein Barycenter and Its Application to Texture Mixing[END_REF]. Then, we present an extension of that transform, namely the generalized Radon transform, which is a fundamental tool in this chapter.

Denition of the Radon transform

Before formally dening the Radon transform, we present some useful notations. De- The Radon transform, introduced in [START_REF] Radon | Uber die bestimmug von funktionen durch ihre integralwerte laengs geweisser mannigfaltigkeiten[END_REF] and denoted by R, maps a function in L 1 (R d , Leb d ) to the innite set of its integrals over the hyperplanes of R d , i.e. R :

note by L 1 (R d , Leb d ) = f : R d → R s.t. R d |f (x)| dLeb d (x) < ∞ the
L 1 (R d , Leb d ) → L 1 (S d-1 × R, σ ⊗ Leb 1 )
, and is dened as follows.

Denition 5.1 (Radon transform). Let I ∈ L 1 (R d , Leb d ). The Radon transform associated to I is a function dened for any (t, θ) ∈ R × S d-1 as

RI(t, θ) = R d I(x)δ(t -x, θ )dx .
(5.1) Note that by denition, each hyperplane in R d can be written as (5.3)

H t,θ = x ∈ R d : x, θ = t ,
The Radon transform has been shown to be invertible, which allows to recover a function I out of its projections along hyperplanes, RI [START_REF] Natterer | The mathematics of computerized tomography[END_REF][START_REF] Helgason | The Radon transform on Rn[END_REF]. In particular, the ltered back-projection method denes the inverse formula of the transform when I ∈ L 1 (R 2 , Leb 2 ) and has been extensively used for image reconstruction, for example in tomographic imaging [START_REF] Deans | The Radon Transform and Some of Its Applications[END_REF].

On the other hand, Denition 5.1 can be extended so that the Radon transform applies to measures instead of functions in L 1 (R d , Leb d ). The formal statement [Bonneel et al., 2015, Denition 6] is recalled below. Denition 5.2 (Radon transform of measures). Denote by C 0 (A) the space of continuous functions on a set A that tend to 0 at innity. Let µ ∈ P(R d ). The Radon transform associated to µ, denoted by Rµ, is dened through the following characterization: for any

f ∈ C 0 (R × S d-1 ), R×S d-1 f (t, θ)d(Rµ)(t, θ) = R d (R * f )(x)dµ(x) , (5.4) where R * : C 0 (R × S d-1 ) → C 0 (R d ) is the back-projection operator dened as R * f (x) = S d-1
f ( x, θ , θ)dθ .

(5.5) Hence, Rµ : P(R d ) → P(R × S d-1 ).

Note that the Radon transform of measures can actually be dened on the general space of Radon measures supported on R d , instead of the subset of probability measures P(R d ): the reason why we consider µ ∈ P(R d ) in Denition 5.2 is that we aim at clarifying the link between the Sliced-Wasserstein distance and the Radon transform.

Link between Radon transform and Sliced-Wasserstein distance

We rst present an important result originally established in [Bonneel et al., 2015, Proposition 6], which shows that Radon transforms of measures are equivalent to specic push-forward measures.

Proposition 5.3. Let µ ∈ P(R d ). Then, for any θ ∈ S d-1 ,

Rµ(•, θ) = θ µ , (5.6)
where θ denotes the push-forward operator (Denition 2.5) associated to the linear form θ (x) = θ, x .

Therefore, the one-dimensional representations of µ, ν ∈ P p (R d ) computed by the Sliced-Wasserstein distance, and respectively denoted by θ µ and θ ν for θ ∈ S d-1 , are actually obtained via the Radon transform: by Proposition 5.3, the denition of SW of order p ∈ [1, +∞) between µ and ν (Denition 2.9) can equivalently be formulated as

SW p p (µ, ν) = S d-1
W p p Rµ(•, θ), Rν(•, θ) dσ(θ) .

(5.7)

In practice, SW p (µ, ν) is approximated with a simple Monte Carlo scheme (Section 2.6), which may lead to underestimating the actual dissimilarity between µ and ν, especially when these two distributions are supported on a high-dimensional space. To further illustrate this phenomenon, let us consider µ = N (0, I d ) and ν = N (m, I d ) with m ∈ S d-1 . Then, their projected representations are univariate Gaussians, given for any θ ∈ S d-1 by, Rµ(•, θ) = N (0, 1) , and Rν(•, θ) = N ( θ, m , 1) .

(5.8) It is therefore clear that W 2 (Rµ(•, θ), Rν(•, θ)) achieves its maximum value when θ = m, and is equal to zero when θ is orthogonal to m. On the other hand, an application of Hoeding's inequality gives the following concentration inequality for any m ∈ S d-1 ,

P(| θ, m | ≤ ε) ≥ 1 -2e -dε 2 2 ,
(5.9) which implies that for a high dimension d, θ ∼ σ is likely to be nearly orthogonal to m, so W 2 (Rµ(•, θ), Rν(•, θ)) is almost null with high probability.

To remedy the inaccuracies caused by the Monte Carlo estimation, one can pick projection directions that return discriminant information between µ and ν, instead of uniformly sampling them on S d-1 . This idea was for instance used in [START_REF] Rowland | Orthogonal Estimation of Wasserstein Distances[END_REF][START_REF] Wu | Sliced Wasserstein Generative Models[END_REF], where the Monte Carlo samples {θ l } L l=1 forms a set of orthogonal vectors, or in [Deshpande et al., 2018, Section 3.2], where implements a GAN to nd discriminant projections.

A similarly avored but less heuristic approach consists in using the maximum Sliced-Wasserstein distance (max-SW), an alternative OT metric dened in [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF] as follows.

Denition 5.4 (Maximum Sliced-Wasserstein distance). Let p ∈ [1, +∞). The maximum Sliced-Wasserstein distance of order p is dened for µ, ν ∈ P p (R d ) as

maxSW p (µ, ν) = max θ∈S d-1 W p (θ µ, θ ν) ,
(5.10) where for any θ ∈ S d-1 , θ = (θ ) denotes the push-forward operator associated to the linear form θ : R d → R given by θ (x) = θ, x . By Proposition 5.3, (5.10) is equivalent to

maxSW p (µ, ν) = max θ∈S d-1 W p (Rµ(•, θ), Rν(•, θ)) .
(5.11) Since W p is a distance (Section 2.4.1), one can show that maxSW p is also a distance: we will prove in Section 5.3.1 that the metric axioms hold for the class of maximum Generalized Sliced-Wasserstein distances, which contains max-SW as a special case.

Generalized Radon transform

The generalized Radon transform (GRT) extends the original idea of the classical Radon transform presented in Section 5.2.1 from integration over hyperplanes of R d to integration over hypersurfaces, i.e. (d -1)-dimensional manifolds [START_REF] Beylkin | The inversion problem and applications of the generalized Radon transform[END_REF][START_REF] Denisyuk | Inversion of the generalized Radon transform[END_REF][START_REF] Ehrenpreis | The universality of the Radon transform[END_REF][START_REF] Gel'fand | Dierential forms and integral geometry[END_REF][START_REF] Kuchment | Generalized transforms of Radon type and their applications[END_REF][START_REF] Homan | Injectivity and stability for a generic class of generalized Radon transforms[END_REF].

According to (5.2), any hyperplane of R d can alternatively be interpreted as a level set of the function g ∈ R d × S d-1 → R given by g(x, θ) = x, θ . Therefore, to generalize the Radon transform, one can simply consider another function g, which is then referred to as the dening function and characterized as follows.

Denition 5.5 (Dening function). Consider a function g : X × (R q \{0}) → R, where X ⊂ R d and q ∈ N * . We say that g is a dening function if it satises the four conditions below.

(D1) g is a real-valued C ∞ function on X × (R n \{0}) (D2) g is homogeneous of degree 1 with respect to its second variable, i.e.

∀(x, θ) ∈ X × (R q \{0}), ∀λ ∈ R, g(x, λθ) = λg(x, θ) .
(D3) g is non-degenerate in the sense that

∀(x, θ) ∈ X × (R q \{0}), ∂g ∂x (x, θ) = 0 .
(D4) The mixed Hessian of g is strictly positive, i.e.

Det

∂ 2 g ∂x i ∂θ j i,j > 0 . Then, given a dening function g, the generalized Radon transform associated to g and applied to I ∈ L 1 (R d , Leb d ) integrates I over the hypersurfaces characterized by the level sets of g, i.e.

Ht,θ = {x ∈ X : g(x, θ) = t} .

(5.12) Denition 5.6 (Generalized Radon transform). Consider a dening function g :

X × (R q \{0}) → R, with X ⊂ R d and q ∈ N * . Let I ∈ L 1 (R d , Leb d ).
The generalized Radon transform of I based on g is a function dened for any (t, θ) ∈ X × (R q \{0}) as GI(t, θ; g) = X I(x)δ(tg(x, θ))dx .

(5.13)

According to Denition 5.6, the standard Radon transform (Denition 5.1) is indeed a special case of the GRT, as it is obtained with g(x, θ) = x, θ for (x, θ) ∈ R d ×S d-1 . GRT has various applications, including thermoacoustic tomography, where the hypersurfaces correspond to spheres, and electrical impedance tomography, which requires integration over hyperbolic surfaces.

One can also use GRT to project high-dimensional distributions along hypersurfaces and obtain one-dimensional representations: Denition 5.6 can be extended so that GRT receives probability distributions as input, analogously to Denition 5.2. In that case, the resulting projections of µ ∈ P(R d ) acquired via the GRT based on a dening function g correspond to (g θ ) µ for any θ ∈ R q \{0}, where (g θ ) is the push-forward operator of g θ , dened for any x ∈ R d as g θ (x) = g(x, θ).

Generalized Sliced-Wasserstein Distances

We propose in this chapter to extend the denition of the Sliced-Wasserstein distance to formulate new optimal transport metrics, which we call Generalized Sliced-Wasserstein distances. These are obtained using the same procedure as for SW, except that the onedimensional representations are acquired through nonlinear projections via the generalized Radon transform. We also extend the concept of max-SW to the class of maximum generalized Sliced-Wasserstein distances (max-GSW).

In this section, we formally dene (maximum) Generalized Sliced-Wasserstein distances and establish the conditions under which they satisfy the metric axioms. We then provide examples of dening functions such that these conditions are met, and present an alternative implementation of GSW inspired by neural networks.

Denition and theoretical properties

Following the denition of SW in terms of the Radon transform (5.7), we dene Generalized Sliced-Wasserstein distances using the generalized Radon transform as follows.

Denition 5.7 (Generalized Sliced-Wasserstein distances). Consider a dening function g : X × (R q \{0}) → R, with X ⊂ R d and q ∈ N * . Let p ∈ [1, +∞). The Generalized Sliced-Wasserstein distance of order p based on g is dened for any µ, ν ∈ P p (R d ) as

GSW p p (µ, ν) = R q \{0}
W p p Gµ(•, θ; g), Gν(•, θ; g) dσ q (θ) ,

(5.14)

where σ q denotes the uniform distribution on R q \{0}.

We also formulate the maximum Generalized Sliced-Wasserstein distance, which generalizes the maximum Sliced-Wasserstein distance dened in (5.11).

Denition 5.8 (Maximum Generalized Sliced-Wasserstein distances). Consider a dening function g : X × (R q \{0}) → R, with X ⊂ R d and q ∈ N * . Let p ∈ [1, +∞). The Maximum Generalized Sliced-Wasserstein distance of order p based on g is dened for any µ, ν ∈ P p (R d ) as maxGSW p (µ, ν) = max θ∈Ω θ W p Gµ(•, θ; g), Gν(•, θ; g) ,

(5.15)

where for any θ ∈ S d-1 , Ω θ ⊂ R q \{0} is a compact set of feasible parameters for θ → g(•, θ).

We point out that max-GSW must rely on a compact set Ω θ so that (5.15) is not illdened. For instance, for the specic case of max-SW where g(•, θ) = θ (•), we have Ω θ = S d-1 .

In the next proposition, we show that GSW and max-GSW are, indeed, distances on P p (R d ) if and only if the underlying GRT is injective. The proof of this result is given in Section 5.6.

Proposition 5.9. Let p ∈ [1, +∞). The Generalized Sliced-Wasserstein (or maximum Generalized Sliced-Wasserstein) distance of order p based on the dening function g satises all metric axioms if and only if the generalized Radon transform associated to g is injective.

According to Proposition 5.9, the injectivity of GRT is sucient and necessary for GSW to be a metric. In this respect, our result brings a dierent perspective on [Bonnotte, 2013, Proposition 5.1.2]: since the standard Radon transform is injective, SW is indeed a distance.

Remark 5.10. If the chosen generalized Radon transform is not injective, then we can only say that the resulting GSW and max-GSW are pseudo-metrics: they still satisfy nonnegativity, symmetry, the triangle inequality, and GSW p (µ, µ) = 0, maxGSW p (µ, µ) = 0.

Injectivity of the generalized Radon transform

We have shown in Proposition 5.9 that the injectivity of the GRT is crucial for the resulting GSW and max-GSW to be distances between probability measures. We now enumerate some of the known dening functions that lead to injective GRTs.

The investigation of the sucient and necessary conditions guaranteeing the injectivity of GRTs is a long-standing topic [START_REF] Beylkin | The inversion problem and applications of the generalized Radon transform[END_REF][START_REF] Homan | Injectivity and stability for a generic class of generalized Radon transforms[END_REF][START_REF] Uhlmann | Inside out: inverse problems and applications[END_REF][START_REF] Ehrenpreis | The universality of the Radon transform[END_REF]. The circular dening function, supported on R d × S d-1 and given by g(x, θ) = xrθ

(5.16) with r ∈ R+, provides an injective GRT [START_REF] Kuchment | Generalized transforms of Radon type and their applications[END_REF]. Homogeneous polynomials with an odd degree also yield an injective GRT [START_REF] Rouvière | Nonlinear Radon and Fourier Transforms[END_REF], and are dened as

g(x, θ) = |α|=m θ α x α ,
(5.17)

where we use the multi-index notation α = (α 1 , . . . , α dα ) ∈ N dα , |α| = dα i=1 α i , and x α = dα i=1 x α i i . The summation in (5.17) iterates over all possible multi-indices α, such that |α| = m, where m denotes the degree of the polynomial and θ α ∈ R. The parameter set for homogeneous polynomials is then set to Ω θ = S dα-1 . We can observe that choosing m = 1 reduces to the linear case g(x, θ) = x, θ , since the set of multi-indices with |α| = 1 becomes {(α 1 , . . . , α d ) : α i = 1 for a single i ∈ N * , 1 ≤ i ≤ d, and α j = 0, ∀j = i} , and contains d elements.

Algorithm 3: Approximation of GSW Input: Two sets of observations {x i } n i=1 and {y i } n i=1 , number of projection directions L, order p, dening function g supported on X × (R q \{0}). GSW = 0 for l = 1, . . . , L do Sample: θ l ∼ σ q for i = 1, . . . , n do Project:

x i = g(x i , θ l ), y i = g(y i , θ l ) Sort: x (1) ≤ x (2) ≤ • • • ≤ x (n) , y (1) ≤ y (2) ≤ • • • ≤ y (n) GSW = GSW + (1/n) n i=1 |x (i) -y (i) | p GSW = (GSW/L) 1/p return GSW
While the polynomial dening functions form an interesting alternative to linear projections, their memory complexity d α grows exponentially with the dimension of the data and the degree of the polynomial, hence deteriorates their potential in modern machine learning problems. As a remedy, inspired by the current success of neural networks, a natural task in our context would be to come up with a neural network, which would yield a valid GSW or max-GSW, when used as the dening function in the GRT.

As a neural network-based dening function, we propose a multi-layer fully connected network with leaky ReLU activations. Under this specic network architecture, one can easily show that the corresponding dening function satises (D1) to (D4) on (X\{0}) × (R q \{0}), where X ⊂ R d and q ∈ N * is the number of parameters of the network. On the other hand, proving the injectivity of the associated GRT is highly nontrivial, so the GSW associated with this particular dening function is a pseudo-metric, as we discussed in Remark 5.10. However, as illustrated in Section 5.4, this neural networkbased dening function still performs well in practice, and the non-dierentiability of the leaky ReLU function at 0 does not seem to be a big issue in practice.

With a neural network as the dening function, minimizing max-GSW between two distributions is analogical to adversarial learning, where the adversary network's goal is to distinguish the two distributions. In the max-GSW case, the adversary network, i.e. the dening function, seeks optimal parameters that maximize the GSW distance between the input distributions.

Numerical Implementation and Experiments

We conduct several experiments to compare the empirical performance of GSW and max-SW for dierent choices of dening functions. To this end, we rst explain how we compute GSW and max-GSW in practice.

Implementation of generalized Sliced-Wasserstein distances

Consider µ, ν ∈ P p (R d ), which we wish to compare using GSW. In most machine learning applications, we do not have access to the distributions, but to two sets of n ∈ N * i.i.d. samples from µ and ν, which are respectively denoted by {x i } n i=1 and {y j } n j=1 .

This implies that one can only compute GSW p (μ n , νn ), which is an approximation Algorithm 4: Approximation of max-GSW Input: Two sets of observations {x i } n i=1 and {y i } n i=1 , number of projection directions L, order p, dening function g, maximum number of iterations T , step size ρ. Randomly initialize θ 0 ∈ Ω θ for t = 0, . . . , T -1 do for i = 1, . . . , n do Project:

x i = g(x i , θ t ), y i = g(y i , θ t ) Sort: x (1) ≤ x (2) ≤ • • • ≤ x (n) , y (1) ≤ y (2) ≤ • • • ≤ y (n) W = (1/n) n i=1 |x (i) -y (i) | p Perform one gradient ascent step: θ = θ l + ρ∇ θ W Project θ on Ω θ : θ l+1 = proj Ω θ (θ ) for i = 1, . . . , n do Project: x i = g(x i , θ T ), y i = g(y i , θ T ) Sort: x (1) ≤ x (2) ≤ • • • ≤ x (n) , y (1) ≤ y (2) ≤ • • • ≤ y (n) mGSW = (1/n) n i=1 |x (i) -y (i) | p mGSW = (mGSW) 1/p return mGSW
of GSW p (µ, ν). Besides, the integral in (5.14) is generally intractable, hence will be estimated.

Similarly to SW, we will use a simple Monte Carlo scheme and the analytical expression of the Wasserstein distance between univariate distributions (2.15) to compute the following estimate of GSW p (μ n , νn ),

GSW p (μ n , νn ) = 1 Ln L l=1 n i=1 |g(x (i) , θ l ) -g(y (i) , θ l )| p 1/p , (5.18)
where for any sequence of vectors {z i } n i=1 and θ ∈ R q \{0}, {g(z (i) , θ)} n i=1 is the sorted sequence of projections, i.e. g(z (1) , θ) ≤ g(z (2) , θ) ≤ • • • ≤ g(z (n) , θ). The procedure to approximate GSW is summarized in Algorithm 3.

To compute maxGSW p (μ n , νn ), we employ a numerical optimization method similar to the expectation-maximization (EM) algorithm, which repeats the following: (a)

given θ ∈ Ω θ , {g(x i , θ)} n i=1 and {g(y i , θ)} n i=1 are sorted to compute the one-dimensional Wasserstein distance, once again according to (2.15), (b) θ is updated with a projected gradient ascent step. Once the convergence is reached (for example, by setting a maximum number of iterations), the algorithm returns an nearly-optimal projection direction θ * , which is then used to approximate maxGSW p (μ n , νn ) as follows

max GSW p (μ n , νn ) = 1 n n i=1 |g(x (i) , θ * ) -g(y (i) , θ * )| p .
The whole procedure is summarized in Algorithm 4. Note that the gradient with respect to θ is computed via automatic dierentiation, and the gradient ascent can be replaced with any iterative optimization method, such as Adam [START_REF] Kingma | A method for stochastic optimization[END_REF].

Our EM-like method nds the optimal θ by optimizing the actual Wasserstein distance between the projected distributions, as opposed to the heuristic approaches pro- posed in [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF][START_REF] Kolouri | Sliced Wasserstein distance for Learning Gaussian Mixture Models[END_REF], where the pseudo-optimal slice is found with perceptrons or penalized linear discriminant analysis [START_REF] Wang | Penalized Fisher discriminant analysis and its application to image-based morphometry[END_REF].

Experiments

Now that we have claried the numerical implementation of (max-)GSW, we present the experiments that we conducted to evaluate the performance of our metrics in generative modeling applications. Our empirical results can be reproduced with our open source code 1 .

To study the eects of the dening function on the practical performance of GSW and max-GSW, we consider the following ows problem min µ D(µ, ν) ,

( 5.19) where D denotes an instance of (max-)GSW, ν is a target distribution and µ is the source distribution. The solution of (5.19) is approximated using the following iterative optimization scheme: rst, µ is initialized as the empirical distribution associated to i.i.d. observations from N (0, I d ); then, these observations are updated by performing gradient descent on D(µ, ν).

The target ν corresponds to the distribution of i.i.d. samples from one of these four well-known distributions: 25-Gaussians, 8-Gaussians, Swiss Roll and Circle. We 1 See our GitHub repository: https://github.com/kimiandj/gsw compare dierent instances of GSW and max-GSW, characterized by dierent choices of dening functions, namely linear (in that case, GSW and max-GSW are equivalent to SW and max-SW respectively), homogeneous polynomials of degree 3 and 5, and the neural networks described in Section 5.3.2 with 1 to 3 hidden layers. We used the exact same optimization scheme for all instances, and kept only L = 1 projection when approximating GSW with (5.18). In order to easily compare the results produced by the dierent ows, we computed the Wasserstein distance of order 2 between µ t and ν (by solving the corresponding linear program), where µ t denotes the source distribution at iteration t of the optimization procedure. We repeated each experiment 100 times and report the average Wasserstein distance (computed with the POT implementation [START_REF] Flamary | POT: Python Optimal Transport[END_REF]) for all target datasets in Figure 5.2. We also show in that same gure a snapshot of µ 100 and ν for all datasets.

We observe that (i) max-GSW outperforms GSW, of course at the cost of an additional optimization, and (ii) while the choice of the dening function g(•, θ) is datadependent, one can see that the homogeneous polynomials are often among the top performers for all datasets. Specically, SW is always outperformed by GSW with polynomial projections (Poly 3 and Poly 5 in Figure 5.2) and by all the variants of max-GSW. Besides, max-SW is consistently outperformed by max-GSW based on neural networks. The only variant of GSW that is outperformed by SW is GSW with a neural network-based dening function, which was expected because of the inherent complexity of approximating the integral over a very large domain (5.14) with a simple Monte Carlo average. Similarly to max-SW, max-GSW replaces sampling with optimization to circumvent this issue.

We then move to more realistic datasets, by running the same experiment for the MNIST dataset [LeCun and Cortes, 2010]: we solve (5.19), where µ is initialized to the distribution of 100 random images of dimension 784, and ν is associated to the training set of MNIST. Given the high-dimensional nature of the problem, we cannot use the homogeneous polynomials due to memory constraints caused by the combinatorial growth of the coecients, as discussed in Section 5.3.2. Therefore, we only compare max-SW against max-GSW whose dening function is a 3-layer neural network. We report the Wasserstein distance of order 2 between each µ t (the 100 images) and ν (the training set of MNIST) in Figure 5.3, where t is the number of training epochs. We observe that with the proposed generalized approach, the error is decreasing signicantly faster when compared to max-SW. We also show the generated images and observe that max-GSW produces crisper images than max-SW.

Max-GSW Max-SW

Finally, we considered a larger dataset, namely CelebA [START_REF] Liu | Deep Learning Face Attributes in the Wild[END_REF]. Since the dimension is very large, we ran a pre-trained auto-encoder to nd a latent space of dimension 256 for the dataset, then solved the ows problem on this lower-dimensional space. We compared max-SW against max-GSW based on a 3-layer neural network, by measuring the Wasserstein distance of order 2 between the target and optimized source distributions in the latent space. Figure 5.4 shows the results of this experiment: max-GSW nds a better solution than max-SW in fewer iterations, but each iteration takes more time because of the neural network training. We also report the generated images in Figure 5.4 and observe that the quality of the images produced by max-GSW is slightly better. Hence, although max-GSW seems like an interesting alternative to max-SW as it produces better results, the practitionner should also be careful about the fact that it has important computational implications and might execute more slowly.

Conclusion

We introduced a novel family of probability divergences, which extends the concept behind the Sliced-Wasserstein and maximum Sliced-Wasserstein distances: while SW and max-SW measure the dissimilarity between two distributions by comparing their projections on hyperplanes, we propose to compare projections on hypersurfaces instead. The resulting divergences, called the Generalized Sliced-Wasserstein and maximum Sliced-Wasserstein distances, are characterized through a general version of the Radon transform, whose standard version was originally used to dene SW. We proved that GSW and max-GSW satisfy all metric axioms if and only if the generalized Radon transform they are based on is injective. We then explained how to implement GSW and max-GSW between any two empirical measures, and demonstrated the superior performance of our generalized divergences over SW in several generative modeling applications.

5.6

Appendix: Proof of Proposition 5.9

The proof of Proposition 5.9 consists in verifying that GSW and max-GSW satisfy all the metric axioms and relies on the fact that W p is a metric.

Proof of Proposition 5.9. Let p ∈ [1, +∞). Consider a dening function g : X×(R q \{0}) → R, with X ⊂ R d and q ∈ N * . We denote by Ω θ a compact set of feasible parameters for g(•, θ).

Non-negativity. Since the Wasserstein distance is a distance and is thus non-negative, we can easily prove that GSW and max-GSW distances satisfy non-negativity as well.

Indeed, consider any dening function g. By using the denition of GSW and max-GSW, and the fact that W p (µ , ν ) ≥ 0 for any two probability distributions µ , ν , we obtain for any µ, ν ∈ P p (R d ),

GSW p (µ, ν) = R q \{0}
W p p Gµ(•, θ; g), Gν(•, θ; g) dσ q (θ)

1 p ≥ R q \{0}
(0) p dσ q (θ)

1 p = 0 , and, maxGSW p (µ, ν) = max θ∈Ω θ W p Gµ(•, θ; g), Gν(•, θ; g) = W p Gµ(•, θ * ; g), Gν(•, θ * ; g) ≥ 0 ,
where θ * = argmax θ∈Ω θ W p (Gµ(•, θ; g), Gν(•, θ; g)).

Symmetry. Since the Wasserstein distance is symmetric, we have for any two distributions µ , ν , W p (µ , ν ) = W p (ν , µ ). In particular, we can write for all θ ∈ R q \{0}, W p (Gµ(•, θ; g), Gν(•, θ; g)) = W p (Gν(•, θ; g), Gµ(•, θ; g)) ,

(5.20) and max

θ∈Ω θ W p (Gµ(•, θ; g), Gν(•, θ; g)) = max θ∈Ω θ W p (Gν(•, θ; g), Gµ(•, θ; g)) .
(5.21)

The symmetry of GSW and max-GSW directly follows from (5.20) and (5.21) respectively.

Triange inequality. We now prove that GSW and max-GSW satisfy the triangle inequality. Let µ 1 , µ 2 and µ 3 in P p (R d ). Since the Wasserstein distance satises the triangle inequality, the following holds for any θ ∈ R q \{0}.

W p (Gµ 1 (•, θ; g), Gµ 3 (•, θ; g)) ≤ W p (Gµ 1 (•, θ; g), Gµ 2 (•, θ; g)) + W p (Gµ 2 (•, θ; g), Gµ 3 (•, θ; g)) .
Therefore, we can write

GSW p (µ 1 , µ 3 ) = R q \{0}
W p p (Gµ 1 (•, θ; g), Gµ 3 (•, θ; g))dσ q (θ)

1 p ≤ R q \{0} W p (Gµ 1 (•, θ; g), Gµ 2 (•, θ; g)) + W p (Gµ 2 (•, θ; g), Gµ 3 (•, θ; g)) p dσ q (θ) 1 p ≤ R q \{0}
W p p (Gµ 1 (•, θ; g), Gµ 2 (•, θ; g))dσ q (θ)

1 p + R q \{0} W p p (Gµ 2 (•, θ; g), Gµ 3 (•, θ; g))dσ q (θ) 1 p (5.22)
where (5.22) follows from the application of the Minkowski inequality in L p (R q \{0}, σ q ).

We conclude that GSW satises the triangle inequality.

Now, denote by θ

* = argmax θ∈Ω θ W p (Gµ 1 (•, θ; g), Gµ 3 (•, θ; g)); then, maxGSW p (µ 1 , µ 3 ) = max θ∈Ω θ W p (Gµ 1 (•, θ; g), Gµ 3 (•, θ; g)) = W p (Gµ 1 (•, θ * ; g), Gµ 3 (•, θ * ; g)) ≤ W p (Gµ 1 (•, θ * ; g), Gµ 2 (•, θ * ; g)) + W p (Gµ 2 (•, θ * ; g), Gµ 3 (•, θ * ; g)) ≤ max θ∈Ω θ W p (Gµ 1 (•, θ; g), Gµ 2 (•, θ; g)) + max θ∈Ω θ W p (Gµ 2 (•, θ; g), Gµ 3 (•, θ; g)) ≤ maxGSW p (µ 1 , µ 2 ) + maxGSW p (µ 2 , µ 3 ) ,
hence, max-GSW also satises the triangle inequality.

Identity of indiscernibles. Since for any distribution µ , W p (µ , µ ) = 0, then by denition of GSW and max-GSW, for any µ ∈ P p (R d ),

GSW p (µ, µ) = 0, and maxGSW p (µ, µ) = 0 .

(5.23)

Now, assume for µ, ν ∈ P p (R d ), GSW p (µ, ν) = 0 and maxGSW p (µ, ν) = 0. Both statements are equivalent to Gµ(•, θ; g) = Gν(•, θ; g) for almost all θ ∈ R q \{0}. Therefore, GSW and max-GSW satisfy the identity of indiscernibles if and only if Gµ(•, θ; g) = Gν(•, θ; g) implies µ = ν, i.e. the GRT is injective. Introduction

The Sliced-Wasserstein distance is a practical alternative optimal transport metric, as it exploits the analytical form of the Wasserstein distance between univariate distributions.

As a reminder, its denition, which is formally given in Denition 2.9, reads as follows:

consider two random variables X and Y in R d with respective distributions µ and ν, and denote by θ µ, θ ν the univariate distributions of the projections of X, Y along θ ∈ R d ; SW then compares µ and ν by computing E[W p p (θ µ, θ ν)], where the expectation E is taken with respect to θ uniformly distributed on the unit sphere, and W p is the Wasserstein distance of order p ≥ 1 (Denition 2.6).

In practice, this expectation is typically estimated by Monte Carlo: one uniformly draws L projection directions {θ l } L l=1 and approximates SW with L -1 L l=1 W p p θ l µ, θ l ν .

97

Since the Wasserstein distance between univariate distributions can easily be computed in closed form, this scheme leads to signicant computational benets as compared to the Wasserstein distance, provided that L is not chosen too large. As we discussed in previous chapters, SW has been successfully applied in several practical tasks, and has been shown to oer nice theoretical properties as well. For instance, even though the sample complexity of Wasserstein grows exponentially with the data dimension (Section 2.4.2), the sample complexity of SW does not depend on the dimension [Nadjahi et al., 2020b].

This latter study, which will be presented in detail in Chapter 7, also demonstrated with a theoretical error bound that the quality of the Monte Carlo estimate of SW depends on the number of projections and the variance of the one-dimensional Wasserstein distances (Theorem 7.9). In other words, to ensure that the induced approximation error is reasonably small, one might need to choose a large value for L, which inevitably increases the computational complexity of SW. Alternative approaches have been proposed to overcome this issue, and mainly consist in picking more informative projection directions: e.g., SW based on orthogonal projections [START_REF] Wu | Sliced Wasserstein Generative Models[END_REF], Meng et al., 2019], maximum SW [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF], generalized SW distances (Chapter 5) and distributional SW distances [START_REF] Nguyen | Distributional Sliced-Wasserstein and Applications to Generative Modeling[END_REF].

In this chapter, we adopt a dierent perspective and leverage concentration results on random projections to approximate SW: previous work showed that, under relatively mild conditions, the typical distribution of low-dimensional projections of high-dimensional random variables is close to some Gaussian law [Sudakov, 1978, Diaconis and[START_REF] Diaconis | Asymptotics of Graphical Projection Pursuit[END_REF]. Recently, this phenomenon has been illustrated with a bound in terms of the Wasserstein distance [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF]: let {X i } d i=1 be a sequence of real random variables with distribution µ d , such that X 1 , . . . , X d are independent with nite fourth-order moments; then, E[W 2 2 (θ µ d , N µ d ) 2 ] goes to zero as d increases, where N µ d is a univariate Gaussian distribution whose variance depends on µ d and the expectation is taken with respect to a Gaussian variable θ. This result has very recently been used to bound the maximum-sliced distance between any probability measure and its Gaussian approximation [START_REF] Goldt | The Gaussian equivalence of generative models for learning with shallow neural networks[END_REF]. In our work, we use it to design a novel technique that estimates SW with a simple deterministic formula. As opposed to Monte Carlo, our method does not depend on a nite set of random projections, therefore it eliminates the need of tuning the hyperparameter L and can lead to a signicant computational time reduction. Besides, our proposal is quite dierent from the aforementioned variants of SW which consist in selecting informative projection directions: these alternatives are dened as optimization problems whose resolution is challenging (e.g., [Nguyen et al., 2021, Section 3.2]) and are then computed by nding an approximate solution. This incurs an additional computational cost and estimation error, while our method directly approximates SW (thus, does not dene an alternative distance) via simple deterministic operations, does not rely on any hyperparameters, and comes with theoretical guarantees on its induced error.

The important steps to formulate our approximate SW are summarized as follows.

We rst dene an alternative SW whose projection directions are drawn from the same Gaussian distribution as in [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF], instead of uniformly on S d-1 , and establish its relation with the original SW. By combining this property with [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF]orem 1], we bound the absolute dierence between SW applied to any two probability measures µ d , ν d on R d , and the Wasserstein distance between the univariate Gaussians N µ d , N ν d . Then, we explain why the mean parameters of µ d and ν d should be zero for the approximation error to decrease as d grows. Nevertheless, we show that it is not a limiting factor, by exploiting the following decomposition of SW: SW between µ d , ν d can be equivalently written as the sum of the dierence between their means and the SW between the centered versions of µ d , ν d .

Our approach then consists in estimating SW between the centered versions with the Wasserstein term between Gaussian approximations to meet the zero-means condition, and recover SW between the original measures via the aforementioned property. Since the Wasserstein distance between Gaussian distributions admits a closed-form solution, our approximate SW is very easy to compute, and faster than the Monte Carlo estimate obtained with a large number of projections. We derive nonasymptotical guarantees on the error induced by our approach. Specically, we dene a weak dependence condition under which the error is shown to go to zero with increasing d. Our theoretical results are then validated with experiments conducted on synthetic data. Finally, we leverage our theoretical insights to design a novel adversarial framework for a typical generative modeling problem in machine learning, and illustrate its advantages in terms of accuracy and computational time, over generative models based on the Monte Carlo estimate of SW. Our empirical results can be reproduced with our open source code 1 .

Background on Central Limit Theorems for Random

Projections

There is a rich literature on the typical behavior of one-dimensional random projections of high-dimensional vectors. To be more specic, let (θ i ) i∈N * be i.i.d. standard onedimensional Gaussian random variables and (X i ) i∈N * be a sequence of one-dimensional random variables. Denote for any d ∈ N * , θ 1:

d = {θ i } d i=1 and X 1:d = {X i } d i=1 .
Sev- eral central limits theorems ensure that, under relatively mild conditions, the sequence

of distributions of d -1/2 θ 1:d , X 1:d ∈ R given θ 1:d ∈ R d converges in distribution to a
Gaussian random variable in probability. This line of work goes back to [Sudakov, 1978, Diaconis and[START_REF] Diaconis | Asymptotics of Graphical Projection Pursuit[END_REF], whose contributions has then been sharpened and generalized in [START_REF] Hall | On almost Linearity of Low Dimensional Projections from High Dimensional Data[END_REF][START_REF] Weizsäcker | Sudakov's typical marginals, random linear functionals and a conditional central limit theorem[END_REF][START_REF] Anttila | The Central Limit Problem for Convex Bodies[END_REF][START_REF] Bobkov | On concentration of distributions of random weighted sums[END_REF][START_REF] Klartag | A central limit theorem for convex sets[END_REF][START_REF] Meckes | Approximation of Projections of Random Vectors[END_REF][START_REF] Dümbgen | On low-dimensional projections of highdimensional distributions[END_REF][START_REF] Leeb | On the conditional distributions of low-dimensional projections from highdimensional data[END_REF].

In particular, a recent study [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF] gives a quantitative version of this phenomenon. More precisely, denote for any d ∈ N * by µ X d , the distribution of X 1:d (i.e., the joint distribution of X 1 , X 2 , . . . , X d ), and γ d the zero-mean Gaussian distribution with covariance matrix (1/d)I d . Assume that for any d ∈ N * , µ X d ∈ P 2 (R d ). Then, [Reeves, 2017, Theorem 1] shows that there exists a universal constant C ≥ 0 such that

R d W 2 2 θ µ X d , N 0, d -1 m 2 (µ X d ) dγ d (θ) ≤ CΞ d (µ X d ) , with (6.1) Ξ d (µ X d ) = d -1 {α(µ X d ) + m 2 (µ X d )β 1 (µ X d ) 1/2 + m 2 (µ X d ) 1/5 β 2 (µ X d ) 4/5 } , (6.2) m 2 (µ X d ) = E X 1:d 2 , (6.3) α(µ X d ) = E X 1:d 2 -m 2 (µ X d ) , (6.4) β q (µ X d ) = E 1 q [ X 1:d , X 1:d q ] , (6.5)
where q ∈ {1, 2} and (X i ) i∈N * is an independent copy of (X i ) i∈N * . A formal statement of this result is also given for completeness in Section 6.6.1. We develop a novel method to approximate the Sliced-Wasserstein distance of order 2, by extending the bound in (6.1) and deriving novel properties for SW. We then derive nonasymptotical guarantees of the corresponding approximation error, which ensure that our estimate is accurate for high-dimensional data under a weak dependence condition.

Sliced-Wasserstein distance with Gaussian projections

First, to enable the use of (6.1) for the analysis of SW, we introduce a variant of SW p (Denition 2.9) whose projections are drawn from the Gaussian distribution considered in (6.1), instead of uniformly on the sphere. The Sliced-Wasserstein distance of order p ∈ [1, +∞) based on Gaussian projections is dened for any µ, ν

∈ P p (R d ) as SW p p (µ, ν) = R d W p p (θ µ, θ ν)dγ d (θ) .
(6.6)

In the next proposition, we establish a simple mathematical relation between traditional SW and the newly introduced one: we prove that SW p is equal to SW p up to a proportionality constant that only depends on the data dimension d and the order p.

(R d ), SW p (µ, ν) = 2 d 1/2 Γ(d/2 + p/2) Γ(d/2) 1/p SW p (µ, ν) ,
where Γ is the Gamma function.

Since (6.1) only applies to the Wasserstein distance of order 2, we will focus on SW of that same order in the rest of the chapter. In this case, SW with Gaussian projections is equal to the original SW. Indeed, we can show that the constant (2/d) 1/2 {Γ(d/2 + p/2) / Γ(d/2)} 1/p dened in Proposition 6.1 is equal to 1 when p = 2, by using the property Γ(d/2 + 1) = (d/2)Γ(d/2).

Approximate Sliced-Wasserstein distance

Our next result is an easy consequence of (6.1) and Proposition 6.1, and shows that, for

any µ d , ν d ∈ P 2 (R d ), the dierence between W 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))} and SW 2 (µ d , ν d ) in absolute value is bounded from above by Ξ d (µ d ) + Ξ d (ν d ) (6.2).
Theorem 6.2. There exists a universal constant C > 0 such that for any

µ d , ν d ∈ P 2 (R d ), SW 2 (µ d , ν d ) -W 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))} ≤ C Ξ d (µ d ) + Ξ d (ν d ) 1/2 , (6.7)
where, for

ξ d ∈ {µ d , ν d }, Ξ d (ξ d )
and m 2 (ξ d ) are dened in (6.2) and ( 6.3) respectively.

Since W 2 {N (0,

d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))
} has a closed-form solution given by (2.10), it provides a computationally ecient approximation of SW 2 (µ d , ν d ) whose accuracy is quantied by Theorem 6.2. Next, we identify settings where this approximation is accurate, by analyzing the error Ξ d (µ d ) + Ξ d (ν d ).

Our rst observation is that µ d and ν d should have zero means for the error to go to zero as d → +∞, and we develop a novel approximation of SW that takes into account this constraint. Going back to the denition of Ξ

d (µ X d ) in (6.2), setting Xi = X i -E[X i ] and X i = X i -E[X i ], we get m 2 (µ X d ) = E[ X1:d 2 ] + E[X 1:d ] 2 (6.8) β 2 2 (µ X d ) = E X1:d , X 1:d 2 + 4E E[X 1:d ], X1:d 2 + E[X 1:d ] 4 .
(6.9)

By Equations (6.8) and (6.9), since in practice the norm of the mean E[X

SW 2 2 (µ d , ν d ) = SW 2 2 (μ d , νd ) + 1 d m µ d -m ν d 2 . (6.10) Based on Proposition 6.3, instead of approximating SW 2 (µ d , ν d ) directly with the term W 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))}, we propose estimating SW 2 (μ d , νd ) with W 2 {N (0, d -1 m 2 (μ d )), N (0, d -1 m 2 (ν d
))} and then using (6.10). This strategy yields our nal approximation of SW, which is dened for any µ d , ν

d ∈ P 2 (R d ) as SW 2 2 (µ d , ν d ) = W 2 2 {N (0, d -1 m 2 (μ d )), N (0, d -1 m 2 (ν d ))} + 1 d m µ d -m ν d 2 , (6.11)
where for ξ d ∈ {μ d , νd }, m 2 (ξ d ) is dened in (6.3). Note that (6.11) can be simplied since by (2.10),

W 2 2 {N (0, d -1 m 2 (μ d )), N (0, d -1 m 2 (ν d ))} = d -1 (m 2 (μ d ) 1/2 -m 2 (ν d ) 1/2 ) 2 .
Besides, if µ d and ν d are both supported on a nite set of points, SW 2 (µ d , ν d ) has a closed-form expression: given

ξ d = n -1 n j=1 δ x (j) ∈ P 2 (R d ) with x (j) ∈ R d for j ∈ {1, . . . , n}, we then have m ξ d = 1 n n j=1 x (j) , and m 2 (ξ d ) = 1 n n j=1 x (j) 2 .
The associated computational complexity is therefore in O(dn), which constitutes a signicant benet of our methodology over the traditional Monte Carlo estimation. Indeed, as explained in Section 2.6, computing SW p,L (2.21) between two empirical distributions amounts to projecting sets of n observations in R d along L directions, and sorting the projected data. The resulting computational complexity is O(Ldn + Ln log n), meaning that the Monte Carlo estimate is more expensive when d, n and L increase, and it is often unclear how L should be chosen in order to control the approximation error.

Hence, we introduced an alternative technique to estimate SW which does not rely on a nite set of random projections, as opposed to the commonly used Monte Carlo technique. Our approach thus eliminates the need for practitioners to tune the number of projections L, but also to sort the projected data. As a consequence, it is more ecient to compute SW 2 (µ d , ν d ) than SW 2,L (µ d , ν d ) for large L. We illustrate this latter point with empirical results in Section 6.4.

Error analysis under weak dependence

We have discussed in Section 6.3.2 why centering the data is important to ensure that the approximation error goes to zero with increasing d. Next, we introduce a weak dependence condition under which the error is guaranteed to decrease as d increases.

We rst consider a setting mentioned in [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF] where d) , ⊗ denoting the tensor product of measures, and µ (j) , ν (j) ∈ P 4 (R) for j ∈ {1, . . . , d}. We prove in this case that W 2 {N (0,

µ d = µ (1) ⊗• • •⊗µ (d) and ν d = ν (1) ⊗ • • • ⊗ ν (
d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))} converges to SW 2 (µ d , ν d ) at a rate of d -1/8
. This result is reported in Section 6.6.5, and can be interpreted as an extension of [Reeves, 2017, Corollary 3] for SW.

We emphasize that the assumptions of this rst setting severely restrict the scope of application of our approximation method: in several statistical and machine learning tasks, the random variables of interest {X i } d i=1 are not independent from each other (e.g. for image data, each X i typically represents the value of a pixel at a certain position, thus depends on the neighboring pixels). Therefore, we relax this independence condition by considering a concept of weak dependence inspired by [START_REF] Doukhan | Probability and moment inequalities for sums of weakly dependent random variables, with applications[END_REF] and properly dened in Denition 6.4. Denition 6.4. Let (X j ) j∈N * be a stationary sequence of one-dimensional random variables with mean zero, i.e. X i and X j have the same distribution for any i, j ∈ N * and E[X 1 ] = 0. We say that (X j ) j∈N * is fourth-order weakly dependent if there exist some constant K ≥ 0 and a nonincreasing sequence of real coecients {ρ(n)} n∈N such that, for any

i, j ∈ N * , i ≤ j, |Cov(X 2 i , X 2 j )| ≤ Kρ(j -i) , |Cov(X i , X j )| ≤ Kρ(j -i) .
(6.12)

In addition, the sequence {ρ

(n)} n∈N satises +∞ n=0 ρ(n) ≤ ρ ∞ < +∞.
Intuitively, in practical applications, the weak dependence condition would essentially require the components of the observations not to exhibit strong correlations; yet, they are allowed to depend on each other. Furthermore, since our weak dependence condition is weaker than the one introduced in [START_REF] Doukhan | Probability and moment inequalities for sums of weakly dependent random variables, with applications[END_REF] Theorem 1], it is satised by the various examples of models described in [Doukhan and Neumann, 2007, Section 5]. We present some of them below, to illustrate Denition 6.4 more clearly.

1) Gaussian processes and associated processes [Doukhan and Louhichi, 1999, Section 3.1], provided that they are stationary.

2) Bernoulli shifts: X t = H(ε t , . . . , ε t-r ) for t ∈ N * , where H : R r+1 → R is a measurable function and (ε i ) i∈N * is a sequence of i.i.d. real random variables. A simple example of such process is given by moving-average models.

3) Autoregressive models, dened as X t = f (X t-1 , . . . , X t-r ) + ε t for t ∈ N * , where (ε i ) i∈N * a sequence of i.i.d. real random variables with E |ε 1 | < ∞, and

|f (u 1 , . . . , u r ) -f (v 1 , . . . , v r )| ≤ r i=1 a i |u i -v i |
for some a 1 , . . . , a r ≥ 0 such that r i=1 a i 1/r < 1.

We then consider a sequence of fourth-order weakly dependent random variables (X j ) j∈N * , and prove that Ξ d (µ X d ) goes to zero as d → ∞, with a rate of convergence depending on {ρ(n)} n∈N . This result is given in Section 6.6.6, and helps us rene Theorem 6.2 under this weak dependence condition: the next corollary establishes that the error approaches 0 at a rate of d -1/8 . Corollary 6.5. Let (X j ) j∈N * and (Y j ) j∈N * be sequences of random variables which are fourth-order weakly dependent. Set for any

d ∈ N * , X 1:d = {X j } d j=1 and Y 1:d = {Y j } d j=1
, and denote by µ d , ν d the distributions of X 1:d , Y 1:d respectively. Then, there exists a universal constant C > 0 such that

SW 2 (µ d , ν d ) -W 2 (N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))) ≤ Cd -1/8 .
Hence, by replacing the independence condition of the rst setting with weak dependence, we broaden the scope of application whilst guaranteeing that the approximation error goes to zero as d increases. We nally note that in these two settings, the data are required to have zero mean, which is automatically veried with our approximation method since we estimate SW between the centered distributions: see (6.11).

Experiments

Synthetic experiments

The goal of these experiments is to illustrate our theoretical results derived in Section 6.3.

In each setting, we generate two sets of d-dimensional samples, denoted by {x (j) } n j=1 and {y (j) } n j=1 with n = 10 4 and x (j) , y (j) ∈ R d for j ∈ {1, . . . , n}. We then approximate SW between their empirical distributions in P 2 (R d ), given by µ d = n -1 n j=1 δ x (j) and ν d = n -1 n j=1 δ y (j) . First, we analyze the consequences of centering data. Here, {x (j) } n j=1 and {y (j) } n j=1 are n independent samples from Gaussian or Gamma distributions: see Section 6.6.7 for more details. We compute

|W 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))} -SW 2 (µ d , ν d )|
on the one hand, and

|W 2 {N (0, d -1 m 2 (μ d )), N (0, d -1 m 2 (ν d ))} -SW 2 (μ d , νd )|
on the other hand. In the Gaussian case, the exact value of SW 2 is known (6.73), while for the Gamma distributions, it is approximated with Monte Carlo based on 2 × 10 4 random projections. Figures 6.1a and 6.1b show that the error goes to zero as d increases if the data are centered. This conrms our analysis provided in Section 6.3.2 about the 10 1 10 2 10 3 dimension 10 0 6 × 10 1 approximation error Gaussian Gamma inuence of the mean, and in Section 6.3.3 on sequences of independent random variables.

Next, we consider autoregressive processes of order one (AR(1)). An AR(1) process is dened as X 1 = ε 1 and, for t ∈ N * , X t = αX t-1 + ε t , where α ∈ [0, 1] and (ε i ) i∈N * is an i.i.d. sequence of real random variables with E[ε 1 ] = 0 and nite second-order moment. If α < 1, the process has a stationary distribution and (X j ) j∈N * satises the weak dependence condition in its stationary regime [START_REF] Doukhan | The notion of Ψ-weak dependence and its applications to bootstrapping time series[END_REF]. In practice, we generate a sample by using this recursion formula for 10 4 + d steps, and keeping the last d samples. The discarded samples correspond to a burn-in phase which helps reaching the stationary solution of the process. We generate {x (j) } n j=1 and {y (j) } n j=1 using the same distribution for the noise (either a Gaussian or Student's tdistribution, as described in Section 6.6.7). This means that both datasets come from the same distribution, thus the exact value of SW 2 is zero. We plot on Figures 6.1c and 6.1d the approximation error according to d ∈ [10, 10 3 ] for dierent values of α. The error converges to zero with increasing d, which is consistent with Corollary 6.5. Note that Figure 6.1 exhibits rate of convergence that are better than the one in d -1/8 derived in Section 6.3.3: in Figure 6.1b, the slope is approximately -0.45 (Gaussian) and -0.7 (Gamma), and in Figures 6.1c and 6.1d, it is on average -0.35. This suggests that our theoretical bounds might be improved, and we further investigate this aspect for the Gaussian case: we consider the case where {x (j) } n j=1 , {y (j) } n j=1 are n independent samples from Gaussian distributions with diagonal covariance matrices, and we prove

that E|W 2 {N (0, d -1 m 2 (μ d )), N (0, d -1 m 2 (ν d ))} -SW 2 (μ d , νd
)| goes to 0 as dn → +∞ with a convergence rate in d -1/2 n -1/2 . We provide the complete statement and formal proof in Section 6.6.7 (Proposition 6.9). This result is consistent with Figure 6.1b, and is a rst encouraging step towards the following research direction: we can study if our proofs and the ones in [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF] can be rened when assuming additional structure on the distributions (e.g., sub-Gaussian and sub-exponential), in order to identify the settings under which our current bounds are tight or can be improved.

Finally, we compare our approximation scheme against the standard Monte Carlo estimation, in terms of accuracy and computation time. We use the same setting as in Figure 6.1b, where the n samples are independently drawn from Gamma distributions. We compute SW 2 (µ d , ν d ) (6.11) and SW 2,L (µ d , ν d ) (2.21) with L ∈ {100, 1000, 5000}, and we compare each approximation with SW 2,2×10 4 (µ d , ν d ), which we consider as the exact value of SW. Figure 6.2 reports the approximation error and computation time of each scheme for d ∈ [10, 10 4 ], and shows that our method is more accurate and faster than Monte Carlo. In particular, when d = 10 3 , the average computation time of our technique is 0.02s, while the second best approximation (Monte Carlo with L = 5000)

takes more than 8s. Besides, we observe that Monte Carlo is very sensitive to the hyperparameters, since it loses accuracy when L decreases and gets slower as L and d increase. This observation is consistent with the computational complexity of SW 2,L recalled at the end of Section 6.3.2. On the other hand, our approximation scheme is extremely ecient even for large d and n, since it is based on a simple deterministic formula which does not require projecting and sorting data along random directions.

Image generation

Finally, we leverage our theoretical insights to design a novel method for a typical generative modeling application. The problem consists in tuning a neural network that takes as input k-dimensional samples from a reference distribution (e.g., uniform or Gaussian), to generate images of dimension d > k. During the training phase, the parameters of the network are updated by iteratively minimizing a dissimilarity measure between the dataset to t and the generated images.

In [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF], the dissimilarity measure is Monte Carlo SW approximated with 10 4 random projections, and the resulting generative model is called the Sliced-Wasserstein generator (SWG). This model performs well on moderately highdimensional image datasets (e.g., 28 × 28 for MNIST images [START_REF] Lecun | MNIST handwritten digit database[END_REF]). However, for very large dimensions (e.g., 64 × 64 × 3 for the CelebA dataset [START_REF] Liu | Deep Learning Face Attributes in the Wild[END_REF]), Monte Carlo SW requires more than 10 4 random projections to capture relevant information, which leads to very expensive training iterations and potential memory issues.

To oer better scalability, SWG can be augmented with a discriminator network [Deshpande et al., 2018, Section 3.2] that aims at nding a lower-dimensional space in which the two projected datasets are clearly distinguishable. The intuition behind this heuristic is that the more distinct the two datasets are from each other, the fewer projection directions Monte Carlo SW requires to provide useful information. The training then consists in optimizing the generator's and discriminator's objective functions in an alternating fashion.

Our novel approach builds on SWG and modies the saddle-point problem in [Deshpande et al., 2018, Section 3.2]: motivated by the gain in accuracy and time illustrated in Figure 6.2 on high-dimensional datasets, we propose to replace Monte Carlo SW with our approximate SW (6.11) in the generator's objective; then, to make sure that our approximation is accurate, we regularize the discriminator's objective:

max ψ L(ψ) + λ 1 Cov[d ψ (X)] 2 F + λ 1 Cov[d ψ (g φ (Z))] 2 F (6.13) + λ 2 E d ψ (X) -2 + λ 2 E d ψ (g φ (Z)) -2 (6.14)
where L is the discriminator's loss used in SWG, g φ and d ψ are the generator's last layer and the discriminator's penultimate layer respectively (parameterized by φ, ψ), X and Z are the random variables corresponding to the images to t and the generator's input, Cov denotes the covariance matrix, • F the Frobenius norm, and λ 1 , λ 2 ≥ 0. The regularization in (6.13) enforces the weak dependence condition (Corollary 6.5), while (6.14) prevents the network to converge to d ψ = 0. We call this generative adversarial network regularized deterministic SWG (reg-det-SWG).

To investigate the consequences of (i) regularizing the discriminator, and (ii) replacing the Monte Carlo SW with our approximation, we design another model, called regularized SWG (reg-SWG): similarly to SWG, the generator minimizes SW 2,10 4 , but the discriminator's objective is regularized as in (6.13), (6.14).

We then compare reg-det-SWG against SWG and reg-SWG, by training the models on MNIST and CelebA and measuring their respective training time and Fréchet Inception Distances (FID, [START_REF] Heusel | GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium[END_REF]): see Table 6.1. We used the same network architectures for all methods, and tuned (λ 1 , λ 2 ) via cross-validation: more details on the experimental setup are given in Section 6.6.8. First, we observe that the regularized models produce images of higher quality, since reg-SWG and reg-det-SWG return lower FID values than SWG. The FID of reg-SWG and reg-det-SWG are close for both datasets, thus the two models seem to yield similar performances. Hence, we report in Figure 6.3 the images generated by SWG and reg-det-SWG only.

The training process is more expensive when regularizing the discriminator: the average running time per epoch is higher for the regularized models. We also observe that Note that the models were trained using PyTorch, thus Monte Carlo SW benets from a GPU-accelerated implementation of the sorting operation (with the function torch.sort). We also reported the computation times when models are trained on CPU.

In this case, computing SW 2 takes at most less than 3s per epoch, whereas the Monte Carlo estimation executes in several minutes (e.g., approximately 45min on CelebA).

As a result, the total training time is almost the same for reg-det-SWG and SWG on CelebA, and the lowest for reg-det-SWG on MNIST. Our approximation method then fosters the development of models to speed up existing machine learning algorithms on CPU, which is useful when powerful hardware resources are not available, or when their use is deliberately avoided for environmental purposes.

Conclusion

We presented a novel method to approximate the Sliced-Wasserstein distance of order 2, which relies on the concentration-of-measure phenomenon for random projections.

The resulting method computes SW with simple deterministic operations, which are computationally ecient even on high-dimensional settings and do not require any hyperparameters. We proved nonasymptotical guarantees showing that, under a weak dependence condition, the approximation error goes to zero as the dimension increases.

Our theoretical ndings are then illustrated with experiments on synthetic datasets.

Motivated by the computational eciency and accuracy of our approximate SW, we nally designed a novel approach for image generation that leverages our theoretical insights. As compared to generative models based on SW estimated with Monte Carlo, our framework produces images of higher quality with further computational benets. This encourages the use of our approximate SW on other algorithms that rely on Monte Carlo SW, e.g. autoencoders [Kolouri et al., 2019b] or normalizing ows [START_REF] Dai | Sliced Iterative Normalizing Flows[END_REF]. The weak dependence condition can be inappropriate to describe the underlying geometry of real data in ML applications, and in that case, approximating SW with our method seems inadequate. To overcome this problem, we encourage practitioners to resort to models where real data are represented by features that can be made weakly dependent. This strategy has proven successful in our image generation experiment: the reg-det-SWG model uses our approximation to compare two sets of features (instead of the raw images) whose covariance matrices are regularized to enforce weak dependence.

Since many ML techniques make use of features and regularizers, we believe that our methodology is not restrictive and can then be applied to other standard problems than image generation. Besides, our weak dependence condition in Denition 6.4 is weaker than the one in [START_REF] Doukhan | Probability and moment inequalities for sums of weakly dependent random variables, with applications[END_REF], which is a notion commonly used in statistics.

Our empirical results on synthetic data show that the approximation error goes to zero with a faster convergence rate than the one we proved. Then, the main current limitation of our framework is that our theoretical convergence rate in d -1/8 might be slower than necessary. We proved that the overall approximation error is upper-bounded by a term in d -1/2 when comparing Gaussians with diagonal covariance matrices, and the improvement of our error bounds for other specic distributions is left for future work. On the other hand, the extension of our methodology to variants of SW is another challenging future research direction. To the best of our knowledge, the literature on the concentration of measure phenomenon focuses on linear random projections, therefore the derivation of deterministic approximations for SW based on nonlinear projections seems highly nontrivial. A more promising direction would be to generalize our approach to SW based on k-dimensional linear projection by leveraging the bound in [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF] Theorem 1] for k > 1.

6.6

Appendix: Postponed Proofs and Experimental Details 6.6.1 Conditional central limit theorem for Gaussian projections

We give the formal statement of the result presented in Section 6.2, corresponding to [Reeves, 2017, Theorem 1] for the special case of one-dimensional projections.

Theorem 6.6 ([Reeves, 2017, Theorem 1]). There exists a constant C such that for any

µ ∈ P 2 (R d ), R d W 2 2 θ µ, N 0, d -1 m 2 (µ) dγ d (θ) (6.15) ≤ Cd -1 α(µ) + m 2 (µ)β 1 (µ) 1/2 + m 2 (µ) 1/5 β 2 (µ) 4/5 , (6.16)
where

m 2 (µ) = R d
x 2 dµ(x) , (6.17)

α(µ) = R d x 2 -m 2 (µ) dµ(x) , (6.18) β q (µ) = R d ×R d x, x q d(µ ⊗ µ)(x, x ) 1 q , ( 6.19)
with q ∈ {1, 2}.

6.6.2 Proof of Proposition 6.1

Proof of Proposition 6.1. Let θ ∈ R d and write θ = r θ, r ≥ 0 and θ ∈ S d-1 . Then, we get

W p p (θ µ, θ ν) = W p p (r θ) µ, (r θ) ν (6.20) = 1 0 F ← (r θ) µ (t) -F ← (r θ) ν (t) p dt , (6.21)
where (6.21) results from (2.12): F μ and F ← μ denote the cumulative distribution and quantile function respectively, of a one-dimensional probability measure μ, i.e. F μ(s) = μ((-∞, s]) and F ← μ (t) = inf{s ∈ R : F μ(s ) ≥ t} for s ∈ R and t ∈ [0, 1]. For any r > 0 and θ ∈ S d-1 , we get

F (r θ) µ (s) = (r θ) µ {(-∞, s]} (6.22) = θ µ {(-∞, s/r]} = Fθ µ (s/r) , (6.23)
which easily implies that F ← (r θ) µ (t) = rF ← θ µ (t). Therefore, using this property in (6.21), we obtain,

W p p (θ µ, θ ν) = 1 0 rF ← θ µ (t) -rF ← θ ν (t) p dt (6.24) = r p W p p ( θ µ, θ ν) .
(6.25)

By applying a d-spherical change of variables in the denition of SW p (6.6) and plugging (6.25), [START_REF] Huber | Gamma Function Derivation of n-Sphere Volumes[END_REF], and by denition of SW (2.20),

SW p p (µ, ν) = R + S d-1 r p W p p θ µ, θ ν (2π) -d 2 d d 2 e -d 2 r θ 2 r d-1 d θdr (6.26) = (2π) -d 2 d d 2 R + r p+d-1 e -d 2 r 2 S d-1 W p p θ µ, θ ν d θ dr . (6.27) Since the surface area of S d-1 is equal to 2π d 2 Γ(d/2) -1 [
S d-1 W p p θ µ, θ ν d θ = 2π d 2 Γ(d/2) -1 SW p p (µ, ν) .
Besides, by applying the change of variables t = (d/2) 1/2 r,

R + r p+d-1 e -d 2 r 2 dr = 2 (p+d)/2 d -(p+d)/2 R + t p+d-1 e -t 2 dt = 2 (p+d)/2-1 d -(p+d)/2 Γ (d + p)/2 .
We nally obtain,

SW p p (µ, ν) = 2 d p/2 Γ d/2 + p/2 Γ(d/2) SW p p (µ, ν) .
(6.28) 6.6.3 Proof of Theorem 6.2

Proof of Theorem 6.2. By the triangle inequality, for any θ ∈ R d ,

W 2 (θ µ d , θ ν d ) -W 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))} (6.29) ≤ W 2 {θ µ d , N (0, d -1 m 2 (µ d ))} + W 2 {θ ν d , N (0, d -1 m 2 (ν d ))} (6.30)
Therefore, taking the integral with respect to γ d ,

R d W 2 (θ µ d , θ ν d ) -W 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))} 2 dγ d (θ) (6.31) ≤ R d W 2 {θ µ d , N (0, d -1 m 2 (µ d ))} + W 2 {θ ν d , N (0, d -1 m 2 (ν d ))} 2 dγ d (θ) (6.32) ≤ 2 R d W 2 2 {θ µ d , N (0, d -1 m 2 (µ d ))}dγ d (θ) + R d W 2 2 {θ ν d , N (0, d -1 m 2 (ν d ))}dγ d (θ) , (6.33)
where (6.33) follows from (a + b) 2 ≤ 2(a 2 + b 2 ). Then, we apply Theorem 6.6 to bound (6.33), and we conclude there exists a universal constant C > 0 such that

R d W 2 (θ µ d , θ ν d ) -W 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))} 2 dγ d (θ) (6.34) ≤ C Ξ d (µ d ) + Ξ d (ν d ) (6.35) Using | a -b | ≤ a -b in L 2 (R d , γ d ) gives R d W 2 2 (θ µ d , θ ν d )dγ d (θ) 1/2 - R d W 2 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))}dγ d (θ) 1/2 (6.36) ≤ R d W 2 (θ µ d , θ ν d ) -W 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))} 2 dγ d (θ) 1/2 (6.37) ≤ C 1/2 Ξ d (µ d ) + Ξ d (ν d ) 1/2 (6.38)
By (6.6) and Proposition 6.1,

R d W 2 2 (θ µ d , θ ν d )dγ d (θ) = SW 2 2 (µ d , ν d ) = SW 2 2 (µ d , ν d ) .
We then obtain the nal result by rewritting (6.36) as

SW 2 (µ d , ν d ) -W 2 {N (0, d -1 m 2 (µ d )), N (0, d -1 m 2 (ν d ))} .
6.6.4 Proof of Proposition 6.3

Proof of Proposition 6.3. This result follows from an analogous translation property of the Wasserstein distance: by [Peyré and Cuturi, 2019, Remark 2.19], W 2 can factor out translations; in particular, for any ξ, ξ ∈ P 2 (R d ) with respective means m ξ , m ξ and centered versions ξ, ξ ,

W 2 2 (ξ, ξ ) = W 2 2 ( ξ, ξ ) + m ξ -m ξ 2 .
(6.39) By using (6.39) in the denition of SW of order 2 (2.20), we obtain for any µ d , ν

d ∈ P 2 (R d ), SW 2 2 (µ d , ν d ) = S d-1 W 2 2 (θ μd , θ νd )dσ(θ) + S d-1 |m θ µ d -m θ ν d | 2 dσ(θ) (6.40) = SW 2 2 (μ d , νd ) + S d-1 |m θ µ d -m θ ν d | 2 dσ(θ) . (6.41)
By the properties of push-forward measures, m θ ξ = θ, m ξ for any θ ∈ S d-1 and ξ ∈ P 2 (R d ). The second term of (6.41) can thus be reformulated as (6.45) where the last equation results from

S d-1 |m θ µ d -m θ ν d | 2 dσ(θ) (6.42) = S d-1 | θ, m µ d -m ν d | 2 dσ(θ) (6.43) = (m µ d -m ν d ) S d-1 θθ dσ(θ) (m µ d -m ν d ) (6.44) = (1/d) m µ d -m ν d 2 ,
S d-1 θθ dσ(θ) = (1/d)I d .
The nal result is obtained by incorporating (6.45) in (6.41).

Error analysis under independence

This section gives a detailed analysis of the error bound under the rst setting discussed in Section 6.3.3: we consider sequences of independent random variables which have zero means and nite fourth-order moments, and we derive an upper bound for Ξ d in the next proposition.

Proposition 6.7. Let (X j ) j∈N * be a sequence of independent random variables with zero means and E[X 4 j ] < +∞ for j ∈ N * . Set for any d ∈ N * , X 1:d = {X j } d j=1 and let µ d be the distribution of X 1:d . Then, we have

Ξ d (µ d ) ≤ d -1/2 max 1≤j≤d Var[X 2 j ] 1/2 + d -1/4 + d -2/5 max 1≤j≤d Var[X j ] . (6.46)
Proof of Proposition 6.7. Given the denition of Ξ d (µ d ) (6.2), the proof consists in bounding m 2 (µ d ), α(µ d ) and β q (µ d ) for q ∈ {1, 2}.

Since for any j ∈ {1, . . . , d}, E[X j ] = 0, then Var X

j = E[X 2 j ] and m 2 (µ d ) = d j=1 E[X 2 j ] = d j=1 Var[X j ] ≤ d max 1≤j≤d Var[X j ] (6.47)
To bound α(µ d ), we rst use the CauchySchwarz inequality.

α(µ d ) ≤ R d x 1:d 2 -m 2 (µ d ) 2 dµ d (x 1:d ) 1/2 (6.48) Besides, R d x 1:d 2 -m 2 (µ d ) 2 dµ d (x 1:d ) = Var X 1:d 2
, and since the d components of X 1:d are assumed to be pairwise independent, Var X 1:d

2 = d j=1 Var X 2 j . We conclude that α(µ d ) ≤   d j=1 Var X 2 j   1/2 ≤ d max 1≤j≤d Var[X 2 j ] 1/2 . (6.49)
Finally, we bound β q (µ d ) for q ∈ {1, 2} by bounding β 2 (µ d ) then using the fact that β 1 (µ d ) ≤ β 2 (µ d ) by the CauchySchwarz inequality. Denote by X 1:d an independent copy of X 1:d .

X 1:d , X 1:d 2 = d j=1 X j X j 2 = d j=1 X j 2 X j 2 + 2 i<j X i X i X j X j .
(6.50)

Since X 1:d and X 1:d are independent on one hand, and they both are sequences of d independent random variables with zero means on the other hand, we have

R d ×R d ( x 1:d , x 1:d ) 2 d(µ d ⊗ µ d )(x 1:d , x 1:d ) (6.51) = d j=1 E X j 2 E X j 2 = d j=1 E X 2 j 2 = d j=1
Var X j 2 .

(6.52)

Therefore, β 2 (µ d ) ≤ ( d j=1 Var[X j ] 2 ) 1/2 ≤ (d max 1≤j≤d Var[X j ] 2 ) 1/2 .
m 2 (µ d ) ≤ d max 1≤j≤d Var[X j ], (6.53)
α(µ d ) ≤ d 1/2 ( max 1≤j≤d Var[X 2 j ]) 1/2 , (6.54) β 1 (µ d ), β 2 (µ d ) ≤ d 1/2 max 1≤j≤d Var[X j ] . (6.55)
The nal result is obtained by bounding Ξ(µ d ) using (6.53), (6.54) and (6.55).

Note that the setting considered in Proposition 6.7 was mentioned in [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF] to illustrate the conditions of [Reeves, 2017, Corollary 3]. We derived an explicit upper bound of Ξ d under this setting for completeness, showing that Ξ d (µ d ) goes to zero as d → ∞, which we can then use to rene the convergence rate in Theorem 6.2, as we explained in Section 6.3.3.

Error analysis under weak dependence

We now analyze the error under the weak dependence condition introduced in Denition 6.4. Specically, the proposition below gives the formal statement of the result mentioned before Corollary 6.5: we consider a sequence of fourth-order weakly dependent random variables, and we prove that Ξ(µ d ) goes to zero as d → ∞, with a convergence rate that depends on {ρ(n)} n∈N * . Proposition 6.8. Let (X j ) j∈N * be a sequence of random variables which is fourth-order weakly dependent. Set for any d ∈ N * , X 1:d = {X j } d j=1 and denote by µ d the distribution of X 1:d . Then, there exists a universal constant C > 0 such that

Ξ d (µ d ) ≤ C d -1/2 ρ(0) + 2ρ ∞ 1/2 + d -1/4 ρ(0) 1/2 ρ(0) 2 + 2ρ ∞ max 1≤k≤d-1 ρ(k) 1/4 + d -2/5 ρ(0) 1/5 ρ(0) 2 + 2ρ ∞ max 1≤k≤d-1 ρ(k) 2/5 . (6.56)
Proof of Proposition 6.8. We proceed as in the proof of Proposition 6.7, i.e. by bounding m 2 (µ d ), α(µ d ) and β 2 (µ d ).

Since (X j ) j∈N * is assumed to be fourth-order weakly dependent, then by Deni- tion 6.4, there exist some constant K ≥ 0 and a nonincreasing sequence of real coecients {ρ(n)} n∈N such that, for any

1 ≤ i ≤ j ≤ d, |Cov(X 2 i , X 2 j )| ≤ Kρ(j -i), |Cov(X i , X j )| ≤ Kρ(j -i) .
(6.57)

First, using the same arguments as in (6.47), we have m 2 (µ d ) = d j=1 Var[X j ]. We then use the second inequality in (6.57) to bound m 2 (µ d ) as follows.

m 2 (µ d ) = d j=1
Cov(X j , X j ) ≤ dKρ(0) (6.58) Regarding α(µ d ), we use the CauchySchwarz inequality again (6.48) but in this setting, the right-hand side features non-zero covariance terms:

R d x 1:d 2 -m 2 (µ d ) 2 dµ d (x 1:d ) = Var X 1:d 2 (6.59) = d j=1 Var X 2 j + 2 i<j Cov X 2 i , X 2 j .
(6.60)

By using the rst inequality in (6.57), we get for any d ∈ N * ,

d j=1 Var X 2 j = d j=1
Cov(X 2 j , X 2 j ) ≤ Kdρ(0) ,

(6.61) i<j Cov X 2 i , X 2 j ≤ i<j |Cov X 2 i , X 2 j | ≤ K i<j ρ(j -i) (6.62) ≤ K d-1 n=1 (d -n)ρ(n) (6.63) ≤ Kd d-1 n=1 ρ(n) ≤ Kdρ ∞ (6.64)
where (6.63) results from the change of variable n = ji. Besides, by Denition 6.4, {ρ(n)} n∈N is a nonincreasing sequence satisfying +∞ n=0 ρ(n) ≤ ρ ∞ < +∞, hence (6.64). We conclude that for any d ∈ N * ,

α(µ d ) ≤ d 1/2 K 1/2 ρ(0) + 2ρ ∞ 1/2 .
(6.65)

Let us now bound β 2 (µ d ). First, for any d ∈ N * ,

R d ×R d ( x 1:d , x 1:d ) 2 d(µ d ⊗ µ d )(x 1:d , x 1:d ) (6.66) = d j=1 E X j 2 E X j 2 + 2 i<j E X i X j E X i X j (6.67) = d j=1 E X j 2 2 + 2 i<j E X i X j 2 (6.68) = d j=1 Var X j 2 + 2 i<j Cov(X i , X j ) 2 , (6.69)
where we used E[X i ] = 0 for any i ≥ 1. To bound (6.69), we apply the second inequality in (6.57), and adapt the arguments used to prove (6.61) and (6.63), .

d j=1 Var X j 2 ≤ K 2 dρ(0) 2 (6.70) i<j Cov(X i , X j ) 2 ≤ K 2 d d-1 n=1 ρ(n) 2 ≤ K 2 dρ ∞ max 1≤n≤d-1 ρ(n) (6.71) Since +∞ n=0 ρ(n) ≤ ρ ∞ < ∞, {ρ (n)
} n∈N converges to 0 as n → +∞ and is thus bounded, so max 1≤n≤d-1 ρ(n) < ∞. We then use (6.70) and (6.71) in the denition of β 2 (µ d ), and β 1 (µ d ) ≤ β 2 (µ d ), to derive the upper-bound below for any d ∈ N * . The n generated samples of X 1:d and Y 1:d are respectively denoted by {x (j) } n j=1 , {y (j) } n j=1 , with x (j) , y (j) ∈ R d for j ∈ {1, . . . , n}. We approximate SW of order 2 between the empirical distributions of {x (j) } n j=1 and {y (j) } n j=1 , given by μd,n = n -1 n j=1 δ x (j) and νd,n = n -1 n j=1 δ y (j) respectively. Note that in Section 6.4, these two distributions were denoted by µ d , ν d instead of μd,n , νd,n , to simplify the notation.

β 1 (µ d ), β 2 (µ d ) ≤ d 1/2 K ρ(0) 2 + 2ρ ∞ max
We rst consider the setting described in Section 6.6.5, where µ d) with µ (j) , ν (j) ∈ P 4 (R) for j ∈ {1, . . . , d}. This means that {X j } d j=1 and {Y j } d j=1 are two sequences of d independent random variables. For each j ∈ {1, . . . , d}, µ (j) (or ν (j) ) refers to a Gaussian or a Gamma distribution, centered or not, as we explain hereafter.

d = µ (1) ⊗ • • • ⊗ µ (d) and ν d = ν (1) ⊗ • • • ⊗ ν (
Gaussian distributions (Figure 6.1a). Here, for j ∈ {1, . . . , d}, we have µ (j) = N (m

(j) 1 } d j=1 , m 2 = {m (j) 2 } d j=1 ∈ R d .
We prove that the SW of order 2 between such Gaussian distributions admits a closed-form expression: for any m 1 , m 2 ∈ R d and σ 2 1 , σ 2 2 > 0,

SW 2 2 {N (m 1 , σ 2 1 I d ), N (m 2 , σ 2 2 I d )} = 1 d m 1 -m 2 2 + (σ 1 -σ 2 ) 2 (6.73)
Proof of (6.73). First, given the properties of ane transformations of Gaussian random variables, we know that for any θ ∈ S d-1 , m ∈ R d and Σ ∈ R d×d symmetric positivedenite, θ N (m, Σ) is the univariate Gaussian distribution N ( θ, m , θ Σθ). Using this property in the denition of SW (2.20) and the fact that θ = 1 for θ ∈ S d-1 ,

SW 2 2 {N (m 1 , σ 2 1 I d ), N (m 2 , σ 2 2 I d )} = S d-1 W 2 2 {N ( θ, m 1 , σ 2 1 ), N ( θ, m 2 , σ 2 2 )}dσ(θ) (6.74) = S d-1 θ, m 1 -m 2 2 + (σ 1 -σ 2 ) 2 dσ(θ) , (6.75)
where (6.75) results from the closed-form solution of the Wasserstein distance of order 2 between Gaussian distributions (2.10). Besides, by denition of the Euclidean innerproduct, for any θ ∈

S d-1 , θ, m 1 -m 2 2 = θ (m 1 -m 2 ) 2 = (m 1 -m 2 ) θθ (m 1 -m 2 ) .
(6.76)

We can thus rewrite (6.75) to obtain

SW 2 2 {N (m 1 , σ 2 1 I d ), N (m 2 , σ 2 2 I d )} = (m 1 -m 2 ) S d-1 θθ dσ(θ) (m 1 -m 2 ) + (σ 1 -σ 2 ) 2 .
(6.77)

We conclude by using the fact that

S d-1 θθ dσ(θ) = (1/d)I d .
Gamma distributions (Figure 6.1a). Denote by Γ(k, s) the Gamma distribution with shape parameter k > 0 and scale s > 0. For j ∈ {1, . . . , d}, µ (j) = Γ(k

(j) 1 , s 1 ) and ν (j) = Γ(k (j) 2 , s 2 ), where k (j) 1 (respectively, k (j)
2 ) is drawn from the uniform distribution over [1,5) (respectively, over [5, 10)), s 1 = 2 and s 2 = 3.

Centered (Gaussian or Gamma) distributions (Figures 6.1b and 6.2). We rst generate {x (j) } n j=1 , {y (j) } n j=1 using the Gaussian (or Gamma) distributions described in the two paragraphs above. Then, we center the data: for j ∈ {1, . . . , n}, x(j) = x (j)n -1 n i=1 x (i) and ȳ(j) = y (j)n -1 n i=1 y (i) . The two distributions that we compare with SW, referred to as μd , νd in Section 6.4, correspond to the empirical distri- butions of the centered datasets {x (j) } n j=1 , {ȳ (j) } n j=1 , which can be denoted by μd,n and νd,n .

We prove in the next proposition that our theoretical bounds derived in Section 6.6.5 can be improved for centered Gaussian distributions: in this setting, the expected approximation error is upper-bounded by a term in d -1/2 , which is consistent with the slope observed in Figure 6

E SW 2 (μ d , νd ) -W 2 {N (0, d -1 m 2 (μ d,n )), N (0, d -1 m 2 (ν d,n ))} ≤ σ 1 + σ 2 (2dn) 1/2 + O 1 dn ,
where E is the expectation with respect to {X (j)

1:d } n j=1 and {Y (j) 1:d } n j=1 , and m 2 (μ d,n ), m 2 (ν d,n ) are dened in (6.3): m 2 (μ d,n ) = n -1 n j=1 X (j) 1:d -X1:d 2 , m 2 (ν d,n ) = n -1 n j=1 Y (j) 1:d - Ȳ1:d 2 .
Proof of Proposition 6.9. Given the closed-form expressions in (6.73) and (2.10), we have

E SW 2 (μ d , νd ) -W 2 {N (0, d -1 m 2 (μ d,n )), N (0, d -1 m 2 (ν d,n ))} = E |σ 1 -σ 2 | -|d -1/2 m 2 (μ d,n ) 1/2 -d -1/2 m 2 (ν d,n ) 1/2 | ≤ E σ 1 -σ 2 -d -1/2 m 2 (μ d,n ) 1/2 + d -1/2 m 2 (ν d,n ) 1/2 (6.79) ≤ E σ 1 -d -1/2 m 2 (μ d,n ) 1/2 + E σ 2 -d -1/2 m 2 (ν d,n ) 1/2 .
(6.80)

where (6.79) results from applying the reverse triangle inequality, and (6.80) follows from the triangle inequality and the linearity of the expectation.

The nal result follows from bounding from above the two terms in (6.80), i.e. E σ 1 -

d -1/2 m 2 (μ d,n ) 1/2 and E σ 2 -d -1/2 m 2 (ν d,n ) 1/2 . First, by the CauchySchwarz inequality, E σ 1 -d -1/2 m 2 (μ d,n ) 1/2 ≤ E (σ 1 -d -1/2 m 2 (μ d,n ) 1/2 ) 2 1/2 , (6.81) with E (σ 1 -d -1/2 m 2 (μ d,n ) 1/2 ) 2 = σ 2 1 -2σ 1 d -1/2 E[m 2 (μ d,n ) 1/2 ] + E[d -1 m 2 (μ d,n )] . (6.82)
Consider the random variable dened as Z = dn i=1 (X i -X) 2 /σ 2 1 , where {X i } dn i=1 are i.i.d. from N (0, σ 2 1 ) and X = (dn) -1 dn i=1 X i . Then, by Cochran's theorem, Z is distributed from the chi distribution with dn -1 degrees of freedom. This implies that,

E[d -1 m 2 (μ d,n )] = σ 2 1 dn -1 dn , E[Z] = √ 2 Γ(dn/2) Γ((dn -1)/2) = √ dn -1 1 - 1 4dn + O 1 (dn) 2 .
Hence, (6.82) boils down to

E (σ 1 -d -1/2 m 2 (μ d,n ) 1/2 ) 2 = σ 2 1 2 - 1 dn -2 1 - 1 dn 1/2 1 - 1 4dn + O 1 (dn) 2 .
(6.83)

Besides, we know that

1 - 1 dn 1/2 = 1 - 1 2dn
+ O 1 (dn) 2 , (6.84) so we can write (6.83) as

E (σ 1 -d -1/2 m 2 (μ d,n ) 1/2 ) 2 = σ 2 1 2dn + O 1 (dn) 2 .
(6.85) By plugging (6.85) in (6.81), we conclude that

E σ 1 -d -1/2 m 2 (μ d,n ) 1/2 ≤ σ 1 (2dn) 1/2 + O 1 dn .
(6.86)

We can use the same reasoning to prove that

E σ 2 -d -1/2 m 2 (ν d,n ) 1/2 ≤ σ 2 (2dn) 1/2 + O 1 dn , (6.87)
and we use (6.86) and (6.87) to bound (6.80), which concludes the proof.

We move on to the explanation of autoregressive processes: let (X j ) j∈N * be an autoregressive process of order 1 dened as X 1 = ε 1 and for t ∈ N * , t > 1, X t = αX t-1 + ε t , where α ∈ [0, 1) and (ε j ) j∈N * is a sequence of i.i.d. real random variables such that E[ε 1 ] = 0 and E[ε 2 1 ] < ∞. For d ∈ N * and B = 10 4 , we generate n realizations of {X j } B+d j=B+1 ∈ R d using the aforementionned recursion. This gives us our rst dataset {x (j) } n j=1 , where x (j) ∈ R d for j ∈ {1, . . . , n}. Note that the rst B steps of the process are discarded in order to reach its stationary regime (which exists since |α| < 1), and thus meet the weak dependence condition [START_REF] Doukhan | The notion of Ψ-weak dependence and its applications to bootstrapping time series[END_REF]. We repeat the same procedure to obtain the second dataset, {y (j) } n j=1 . Since the two datasets are generated using the same AR(1) model, µ d and ν d are the same distribution, so the exact value of SW is zero.

We conducted our experiments on two types of AR(1) processes, which dier from the distribution used to draw n i.i.d. samples of {ε j } B+d j=1 . The two settings are specied below.

Gaussian noise (Figure 6.1c). For j ∈ {1, . . . , B + d}, ε j ∼ N (0, 1).

Student's t noise (Figure 6.1d). Denote by t(r) the Student's t distribution with r > 0 degrees of freedom. For j ∈ {1, . . . , B + d}, ε j ∼ t(10).

Finally, we specify that the experiment comparing the computation time of our methodology against Monte Carlo estimation (Figure 6.2) was conducted on a dailyuse laptop equipped with 8 × Intel Core i7-8650U CPU @ 1.90GHz, 16GB of RAM. high-dimensional spaces, but can eciently be computed in the univariate case (e.g., the Wasserstein distance (2.12)). The slicing operation then leverages these advantages originally available in one dimension to dene divergences achieving computational efciency on multivariate settings [START_REF] Rabin | Wasserstein Barycenter and Its Application to Texture Mixing[END_REF][START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF][START_REF] Paty | Subspace Robust Wasserstein distances[END_REF], Kolouri et al., 2019b[START_REF] Vayer | NeurIPS 2019 -Thirty-third Conference on Neural Information Processing Systems[END_REF].

Even though various sliced divergences have successfully been deployed in practical applications, their theoretical properties have not yet been well understood, and existing results are largely restricted to the specic case of the Sliced-Wasserstein distance.

Besides, some properties of SW have only been characterized for specic settings, in particular its statistical benets observed in practice [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF][START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF]. In this chapter, we aim to bridge this gap by investigating the theoretical properties of sliced probability divergences from a general point of view: since such divergences are all characterized via the same slicing operation, we explore in depth the topological and statistical implications of this operation. Specically, we consider a generic base divergence ∆ between one-dimensional probability measures, and dene its sliced version, denoted by S∆, which operates on multivariate settings.

We rst establish several topological properties of S∆. Thanks to our general approach, our ndings can directly be applied to any instance of sliced divergence, including those motivated by the two aforementioned purposes. Specically, we show that slicing preserves the metric properties: if ∆ is a metric, so is S∆ (Proposition 7.1). We then focus on ner topological properties of S∆ and show in Theorem 7.2 that, if the convergence in ∆ implies the weak convergence of measures (or conversely), then slicing preserves this property, i.e. the convergence in S∆ implies the weak convergence of measures (or conversely). We also consider the case when ∆ is an integral probability metric (Denition 2.3) and identify sucient conditions for S∆ to be upper-bounded by ∆, which implies that S∆ induces a weaker topology (Theorem 7.4). Similarly, we identify sucient conditions such that ∆ and S∆ are strongly equivalent (Corollary 7.6), meaning that ∆ is upper-and lower-bounded by S∆.

Then, we derive the following statistical properties of S∆: we prove that the sample complexity of S∆ is proportional to the sample complexity of ∆ for one-dimensional measures, and does not depend on the dimension d (Theorems 7.7,7.8). This property explains why any S∆ motivated by the second purpose oers statistical benets when the original divergence suers from the curse of dimensionality. However, this comes with a caveat: we show that, if one approximates the expectation over the random projections that appears in S∆ with a Monte Carlo average, which is the most common practice, then an additional variance term appears in the sample complexity and can limit the performance of S∆ in high dimensions (Theorem 7.9). Our results agree with the recent empirical observations reported in [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF], which motivated Chapters 5 and 6, and provide a better understanding for them.

We illustrate all our theoretical ndings on various examples, which demonstrate their applicability. In particular, our general topological analysis allows us to establish a novel result for the Sliced-Cramér distance. We also derive a sample complexity result for SW which has never been shown before, under dierent assumptions on the measures to be compared. We then consider Sinkhorn divergences (Denition 2.7), whose sample complexity is known to have an exponential dependence on the dimension d and regularization parameter ε (Theorem 2.8), and introduce its sliced version, referred to as the Sliced-Sinkhorn divergence. We prove that this new divergence has several merits: we derive its sample complexity by combining our general results with recent work [Genevay et al., 2019, Mena andNiles-Weed, 2019], and obtain rates that do not depend 7.2. Sliced Probability Divergences 123 on d nor on ε. We also show that this divergence improves the worst-case computational complexity bounds of Sinkhorn divergences in R d . Finally, we support our theory with numerical experiments on synthetic and real data.

Sliced Probability Divergences

In this section, we dene the family of Sliced Probability Divergences, then we present our theoretical contributions regarding their topological and statistical properties. We provide all the proofs in Sections 7.6.1 and 7.6.3.

Denition

Consider a base divergence ∆ : P(R) × P(R) → R + ∪ {∞} which measures the dissimilarity between two probability measures on R, and let p ∈ [1, ∞). We dene the Sliced Probability Divergence of order p associated to ∆, denoted by S∆ p , for µ, ν ∈ P(R d ) as

S∆ p p (µ, ν) = S d-1
∆ p (θ µ, θ ν)dσ(θ) .

(7.1)

We assume that θ → ∆ p (θ µ, θ ν) is measurable so that (7.1) is well-dened. This can easily be checked if (µ , ν ) → ∆(µ , ν ) is continuous for the weak topology on P(R), since this implies θ → ∆ p (θ µ, θ ν) is continuous.

In practice, since the integration over S d-1 in (7.1) does not admit an analytical form in general, it is approximated with a simple Monte Carlo scheme (e.g., Section 2.6, [START_REF] Vayer | NeurIPS 2019 -Thirty-third Conference on Neural Information Processing Systems[END_REF], Kolouri et al. [2020a]). The Monte Carlo estimate of S∆ p obtained with L random projection directions is dened as

S∆ p p,L (µ, ν) = 1 L L l=1 ∆ p θ l µ, θ l ν , (7.2)
with {θ l } L l=1 i.i.d. from σ and θ l (x) = θ l , x . Since each term of the sum in (7.2) can be computed independently from each other, the approximation of SPDs can be carried out in parallel, which constitutes a nice practical feature. Recent work [START_REF] Paty | Subspace Robust Wasserstein distances[END_REF]Cuturi, 2019, Deshpande et al., 2019] has shown that sampling many projection directions uniformly on the sphere might not be the best strategy, in the sense that some directions can be more helpful than others to discriminate the two distributions at hand. However, the Monte Carlo estimate based on uniform sampling (7.2) is the most common method used in practice to approximate sliced divergences, hence we focus on this approximation throughout the rest of the chapter.

Topological properties

We provide several results to describe the topology induced by SPDs, given the properties of base divergences. We rst relate in Proposition 7.1 the metric properties of ∆ and S∆ p , p ∈ [1, ∞).

Proposition 7.1. (i) If ∆ is non-negative (or symmetric), then S∆ p is non-negative (symmetric resp.).

(ii) If ∆ satises for µ , ν ∈ P(R), ∆(µ , ν ) = 0 if and only if µ = ν , then for µ, ν ∈ P(R d ), S∆ p (µ, ν) = 0 if and only if µ = ν.

Furthermore, suppose that Sγ G,p is bounded. Then, for any p ∈ [1, +∞), there exists C p > 0 such that γ G (µ, ν) ≤ C p Sγ G,p (µ, ν) 1/(d+1) .

(7.3)

One can show that the exponent 1/(d + 1) in (7.3) is intrinsic to slicing, hence cannot be avoided. By combining Theorems 7.4 and 7.5, we nally establish a strong equivalence result below, which implies that the convergence of probability measures in Sγ G,p is equivalent to the convergence in γ G .

Corollary 7.6. Let µ, ν ∈ P(R d ), with support included in B d (0, R), and let

G ⊂ M b (R d ).
Assume that the conditions of Theorems 7.4 and 7.5 are satised. Then, for any p ∈ [1, +∞), there exists C p ≥ 0 independent of µ, ν such that d+1) .

Sγ G,p (µ, ν) ≤ γ G (µ, ν) ≤ C p Sγ G,p (µ, ν) 1/(
Our analysis on IPMs builds on [Bonnotte, 2013, Chapter 5.1], which contains analogous results for the Sliced-Wasserstein distance only. The novelty of Theorems 7.4 and 7.5 is the identication of the relationships between the function classes F,F and G,G, which might provide a useful guideline for practitioners interested in slicing any IPM, and cannot be directly obtained from [START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF]. We further illustrate these relations in Section 7.6.2 for classical instances of IPMs.

Statistical properties

In most practical applications, we have at hand nite sets of samples drawn from unknown underlying distributions. An important question is then the bound of the error made when approximating a divergence with nitely many samples: given S∆ p and any µ, ν ∈ P(R d ), our goal is to quantify the sample complexity of S∆ p , i.e. the convergence rate of S∆ p (μ n , νn ) to S∆ p (µ, ν) according to n. We show in Theorem 7.7 that the sample complexity of any SPD is proportional to the sample complexity of the base divergence, and more importantly, does not depend on d.

Theorem 7.7. Let p ∈ [1, ∞). Suppose that ∆ p admits the following sample complexity: for any µ , ν ∈ P(R) with respective empirical measures μ n , ν n ,

E ∆ p (µ , ν ) -∆ p (μ n , ν n ) ≤ β(p, n) .
Then, for any µ, ν ∈ P(R d ) with respective empirical measures μn , νn , the sample complexity of S∆ p is given by

E S∆ p p (µ, ν) -S∆ p p (μ n , νn ) ≤ β(p, n) .
If ∆ is a bounded pseudo-metric and we have a direct control over the convergence rate of empirical measures in ∆, we can further derive the following result.

Theorem 7.8. Let p ∈ [1, ∞). Assume that for any µ ∈ P(R) with empirical measure μ n , E|∆ p (μ n , µ )| ≤ α(p, n) . Then, for any µ ∈ P(R d ) with empirical measure μn , we have

E S∆ p p (μ n , µ) ≤ α(p, n) .
Besides, if ∆ is non-negative, symmetric, and satises the triangle inequality, then

E |S∆ p (µ, ν) -S∆ p (μ n , νn )| ≤ 2 α(p, n) 1/p .
So far, our results show that slicing preserves some useful topological properties of the base divergence, and additionally oers a dimension-free sample complexity. On the other hand, slicing results in less discriminant divergences, as we mentioned for IPMs (Theorem 7.4), and in such a case, the improvement in the sample complexity might be less signicant. More analysis is required to understand the potential reduction in the discriminative power, and we leave it out of scope of this study.

In practice, SPDs also induce an approximation error due to the Monte Carlo estimate (7.2). We use the term projection complexity to refer to the convergence rate of S∆ p,L to S∆ p as a function of the number of projections L.

Theorem 7.9. Let p ∈ [1, ∞) and µ, ν ∈ P(R d ). Then, the error made with the Monte Carlo estimation of S∆ p can be bounded as follows

E S∆ p p,L (µ, ν) -S∆ p p (µ, ν) 2 ≤ L -1 S d-1
∆ p (θ µ, θ ν) -S∆ p p (µ, ν) 2 dσ(θ) .

(7.4) By denition of S∆ p p (µ, ν), Theorem 7.9 illustrates that the quality of the Monte Carlo estimates is impacted by the number of projections as well as the variance of the evaluations of the base divergence. This behavior has previously been empirically observed in dierent scenarios [Deshpande et al., 2019, Paty andCuturi, 2019], and paved the way for the max-sliced distances and our methodologies in Chapters 5 and 6.

We now leverage Theorems 7.7 and 7.9 to derive the overall complexity of sliced divergences, i.e. the convergence rate of S∆ p (μ n , νn ) to S∆ p (µ, ν).

Corollary 7.10. Let p ∈ [1, ∞) and µ, ν ∈ P(R d ). Denote by μn (respectively, νn ) the empirical distribution computed over a sequence of i.i.d. random variables X 1:n = {X k } n k=1 from µ (resp., Y 1:n = {Y k } n k=1 from ν). Assume ∆ p admits the following sample complexity: for any µ , ν ∈ P(R) and empirical instantiations μ n , ν n ,

E[ ∆ p (µ , ν ) -∆ p (μ n , ν n ) ] ≤ β(p, n) .
Then,

E | S∆ p p,L (μ n , νn ) -S∆ p p (µ, ν)| ≤ β(p, n) + L -1/2 S d-1 E ∆ p (θ μn , θ νn ) -S∆ p p (μ n , νn ) 2 dσ(θ) 1/2
, where S∆ p p,L (μ n , νn ) is dened by (7.2), and E is the expectation w.r.t. X 1:n , Y 1:n and {θ l } L l=1 i.i.d. from the uniform distribution on S d-1 .

Hence, the overall complexity S∆ p,L (μ n , νn ) -S∆ p (µ, ν) is bounded by the sum of the sample and the projection complexities. This result is useful as it helps understanding the behavior of sliced divergences in most practical applications, where S∆ p (µ, ν) is approximated using nite sets of samples drawn from µ and ν along with Monte Carlo estimates.

Applications

We already referred to Section 7.6.2, which contains applications of Theorems 7.4 and 7.5

to classical instances of IPMs in order to clarify the assumptions of these theorems. In this section, we apply the rest of our topological and statistical results to specic sliced divergences and present the interesting properties that we obtained. Our goal is to further illustrate the signicance of the general theoretical analysis that we conducted in the previous section. In particular, we will introduce a novel divergence based on Sinkhorn divergences, and provide theoretical results that emphasize its statistical and computational advantages. The associated proofs are given in Sections 7.6.4 to 7.6.6.

Topology induced by the Sliced-Cramér distance

First, Theorem 7.2 can be applied to various base divergences (e.g., see those listed in [Gibbs and Su, 2002, Theorem 6]) and foster interesting applications. In particular, we focus on the Cramér distance [START_REF] Cramér | On the composition of elementary errors[END_REF] and its sliced version [START_REF] Knop | Cramer-Wold Auto-Encoder[END_REF], Kolouri et al., 2020a], whose denitions are recalled in Denitions 7.11 and 7.12 respectively.

Denition 7.11 (Cramér distance). Let p ∈ [1, ∞) and µ, ν ∈ P(R). Denote by F µ , F ν the cumulative distribution functions of µ, ν respectively. The Cramér distance of order p between µ and ν is dened by

C p p (µ, ν) = R |F µ (t) -F ν (t)| p dt . (7.5)
By [START_REF] Dedecker | The empirical distribution function for dependent variables: asymptotic and nonasymptotic results in L p[END_REF], Lemma 1], the Cramér distance can be written as an IPM.

Denition 7.12 (Sliced-Cramér distance). Let p ∈ [1, ∞) and µ, ν ∈ P(R d ). The Sliced-Cramér distance of order p between µ and ν is dened by

SC p p (µ, ν) = S d-1
C p p (θ µ, θ ν)dσ(θ) .

(7.6)

We establish theoretical guarantees which, to the best of our knowledge, have not been proved before: we show that convergence under the Sliced-Cramér distance implies weak convergence in P(R d ), and the converse is true for measures supported on a compact space.

Corollary 7.13. Let p ∈ [1, ∞). For any sequence (µ k ) k∈N in P(R d ) and µ ∈ P(R d ), lim k→∞ SC p µ k , µ = 0 implies (µ k ) k∈N converges weakly to µ.

Besides, if (µ k ) k∈N and µ are supported on a compact space K ⊂ R d , then the converse implication holds, meaning that the convergence under SC p is equivalent to the weak convergence in P(K).

Theorem 7.2 also applies to the broader class of Sliced-IPMs, assuming a density property for the space of functions associated with the base IPM. We provide the formal statements and proofs of these results in Section 7.6.4.

Sample complexity of the Sliced-Wasserstein distance

Then, we derive the sample complexity of SW p under dierent moment conditions.

While previous works have illustrated the statistical benets of SW, our next corollary establishes a novel result.

Corollary 7.14. Let p ∈ [1, ∞), q > p, and µ, ν ∈ P q (R d ) with corresponding empirical measures μn , νn . We use the notation

M 1/q q (µ, ν) = M 1/q q (µ) + M 1/q q (ν) ,
where M q (ζ) refers to the moment of order q of ζ ∈ P q (R d ). Then, there exists a constant C p,q depending on p, q such that

E SW p (μ n , νn ) -SW p (µ, ν) ≤ C 1/p p,q M 1/q q (µ, ν)        n -1/(2p) if q > 2p, n -1/(2p) log(n) 1/p if q = 2p,
n -(q-p)/(pq) if q ∈ (p, 2p).

(7.7)

Corollary 7.14 completes the literature on the sample complexity of SW: [START_REF] Deshpande | Max-Sliced Wasserstein distance and its use for GANs[END_REF] derived the sample complexity for Gaussian distributions only and [START_REF] Manole | Minimax Condence Intervals for the Sliced Wasserstein Distance[END_REF] provided condence intervals which partially cover our result.

Sliced-Sinkhorn divergences

We now introduce a new family of probability divergences obtained by slicing the regularized OT cost and Sinkhorn divergences, and called Sliced-Sinkhorn divergences (SSD):

for p ∈ [1, ∞), ε ≥ 0 and µ, ν ∈ P p (R d ), SW p,ε (µ, ν) = S d-1 W p,ε (θ µ, θ ν) dσ(θ), SW p,ε (µ, ν) = S d-1 W p,ε (θ µ, θ ν) dσ(θ) (7.8)
We show that these divergences enjoy interesting statistical and computational properties. For clarity purposes, our results are only presented for SW p,ε , but also apply for SW p,ε . Since W p,ε is not an IPM, we rst derive a topological property analogous to Theorem 7.4.

Theorem 7.15. Let p ∈ [1, ∞) and ε ≥ 0. For any µ, ν ∈ P p (R d ),

SW p,ε (µ, ν) ≤ W p,ε (µ, ν) .
Next, we show that on compact domains, while the sample complexity of regularized OT exponentially worsens as ε decreases [Genevay et al., 2019, Theorem 3], the sample complexity of SSD does not depend on ε.

Theorem 7.16. Let X be a compact subset of R d , p ∈ [1, ∞) and µ, ν ∈ P(X), with respective empirical instanciations μn , νn . Then, there exists a constant C(µ, ν) that depends on the moments of µ and ν, such that

E SW p,ε (μ n , νn ) -SW p,ε (µ, ν) ≤ diam(X)C(µ, ν)n -1/2 .
In practice, we approximate SSD by using (7.2). The estimator corresponds to randomly picking a nite set of directions and solving, for each direction, a regularized OT problem in R. To obtain solutions associated to the regularized Wasserstein cost, a method which is now standard is the Sinkhorn's algorithm: more details are given in Section 2.5 and at the end of Section 7.6.6. In particular, if we use the squared Euclidean ground cost and consider the empirical measures μn , νn on R d associated to the observations (x i ) n i=1 , (y j ) n j=1 respectively, computing W p,ε (μ n , νn ) has a worst-case convergence rate that depends on C(μ n , νn ) = max i,j∈{1,...,n}

x iy j 2 ε .

See also [START_REF] Altschuler | Near-Linear Time Approximation Algorithms for Optimal Transport via Sinkhorn Iteration[END_REF] for a sublinear rate with a better constant, still depending on this quantity. The rate for W p,ε (θ μn , θ νn ), with θ ∈ S d-1 , then depends on C(θ μn , θ νn ) = max i,j∈{1,...,n} θ, x iy j 2 /ε.

We show in Proposition 7.17 that with high probability, C(θ μn , θ νn ) is smaller than C(μ n , νn ) by a factor of d at least, unless n grows super-polynomially with d. Our result, combined with the parallel computation of (7.2), implies that slicing the regularized OT may lead to signicant computational benets.

Proposition 7.17. Let (x i ) n i=1 be a set of vectors in R d such that max i,j

x i -x j 2 2 ≤ R 2 ,
and θ chosen uniformly at random on S d-1 . Then for δ ∈ (0, 1], it holds with probability 1δ,

max i,j | θ, x i -x j | 2 ≤ 2R 2 d log( √ 2πn 2 /δ) .
Finally, we note that an advantage of the Sinkhorn divergence over the Wasserstein distance is that the former is always dierentiable [START_REF] Feydy | Interpolating between Optimal Transport and MMD using Sinkhorn Divergences[END_REF], Proposition 2] while the latter is not. This property, which is crucial in dierential programming pipelines, suggests that SSD is potentially better-behaved than SW in tasks such as generative modeling. We leave its analysis to future work.

Experiments

We present the numerical experiments that we conducted to illustrate our theoretical ndings, and we provide the code to reproduce them 1 . We also provide additional em- pirical results in Section 7.6.7.

We rst verify that IPMs and Sinkhorn divergences are bounded below by their sliced versions, as demonstrated in Theorems 7.4 and 7.15 respectively. Consider n = 1000 observations i.i.d. from N (0, σ 2 I d ), with σ 2 = 4. We generate n i.i.d. samples from N (0, σ 2 I d ) for σ 2 varying between 0.1 and 9. We compute MMD between the empirical distributions of the observations and the generated datasets, as well as the Wasserstein distance of order 1 and normalized Sinkhorn divergence (7.8) with order 1 and ε = 1.

We used a Gaussian kernel for MMD combined with the heuristic proposed in [START_REF] Gretton | A Kernel Two-Sample Test[END_REF], which sets the kernel width to be the median distance over the aggregated data, and we approximated this discrepancy with the biased estimator in [Gretton et al., 1 See our GitHub repository: https://github.com/kimiandj/sliced_div The curves for Wasserstein, Sinkhorn and MMD are above their respective sliced version's ones, as predicted by our theoretical bounds. This gure also illustrates the statistical benets induced by slicing: all sliced divergences attain their minimum at σ , while Wasserstein and Sinkhorn fail at this. This observation is in line with [START_REF] Bellemare | The Cramer distance as a solution to biased Wasserstein gradients[END_REF], where the authors showed that both the minimum point and gradients of the Wasserstein distance have a bias, which can be prominent unless n is large enough.

MMD performs well in this task, and this can be explained by its dimension-free sample complexity. In that sense, Sliced-MMD acts more as a sanity-check of our theory, rather than a practical proposal.

The next experiments aim at illustrating our statistical properties. We rst analyze the convergence rate of the Monte Carlo estimates (Theorem 7.9) in a synthetical setting.

We consider two sets of 500 samples i.i.d. from the d-dimensional Gaussian distribution N (0, I d ), and we approximate SW 2 between the empirical distributions with a Monte Carlo scheme that uses a high number of projections L = 10 000. Then, we compute the Monte Carlo estimate SW 2,L obtained with L < L random projections. Figure 7.2a shows the absolute dierence of SW 2,L and SW 2,L against L, for dierent values of dimension d. We observe that the Monte Carlo error indeed shrinks to zero when we increase the number of projections, with a convergence rate of order L -1/2 . Then, we illustrate the sample complexity of Sliced-Wasserstein and Sliced-Sinkhorn (Corollary 7.14 and Theorem 7.16, respectively). We consider two sets of n samples i.i.d. from N (0, I d ), and we compute W 2 and W 2,ε and their sliced versions approxi- mated with 100 random projections. We analyze the convergence rate for dierent n and dimensions d. We also study the inuence of the regularization parameter ε for Sinkhorn divergences. Figure 7.2b reports the Wasserstein and Sliced-Wasserstein distances vs. n, for d between 2 and 100. We observe that, as opposed to W 2 , the convergence rate of SW 2 does not depend on the dimension, therefore SW 2 converges faster than W 2 when the dimension increases. As predicted by the theory, Sliced-Sinkhorn oers more robustness than Sinkhorn: its convergence rate does not depend on the dimension nor on the regularization coecient.

To illustrate Proposition 7.17, we plot on Figure 7.3c the number of iterations when the convergence of Sinkhorn's algorithm is reached, as a function of d. For Sliced-Sinkhorn, this number is an average over the number of projections used in the approximation.

Our experiment emphasizes the computational advantages of Sliced-Sinkhorn, since its number of iterations remains the same with the increasing dimension, while it grows exponentially for Sinkhorn.

Our last experiment operates on real data and is motivated by the two-sample testing problem [START_REF] Gretton | A Kernel Two-Sample Test[END_REF], whose goal is to determine whether two sets of samples were generated from the same distribution or not. This is useful for various applications, including data integration, where we wish to understand that two datasets were drawn from the same distribution in order to merge them. In this context, we run the following experiment: for dierent values of n, we randomly select two subsets of n samples from the same dataset, and we compute the Wasserstein and Sliced-Wasserstein distances (of order 2) between the empirical distributions, as well as the Sinkhorn and Sliced-Sinkhorn divergences (ε = 1). The sliced divergences are approximated with 10 random projections. We use the MNIST [LeCun and Cortes, 2010] and CIFAR-10 [Krizhevsky, 2009, Chapter 3] datasets, and we report the divergences against n, and the mean execution time for the computation of Sinkhorn and Sliced-Sinkhorn, on Figure 7.4. The sliced divergences perform the best, in the sense that they need less samples to converge to zero. Besides, Sliced-Sinkhorn is faster than Sinkhorn in terms of execution time (which was expected, given our discussion above Proposition 7.17), and the dierence is even more visible for a high number of samples. For example, for n = 2500 on MNIST or All plots have a log-log scale.

n = 1000 on CIFAR-10, Sliced-Sinkhorn is almost 130 times faster than for Sinkhorn on average.

Conclusion

In this study, we considered sliced probability divergences, which have been increasingly popular in machine learning applications. We derived theoretical results about their induced topology as well as their statistical eciency in terms of number of samples and projections, and we empirically illustrated our ndings on dierent setups. Specically, we proved that the preserved topology and dimension-free sample complexity are intrinsic to slicing. Since this was unclear in the previous literature, which combined slicing with a specic distance, our unied treatment of these results brings insight to the properties of particular instances used in practice.

The gains in statistical eciency could be explained by an ability of slicing to overlook irrelevant characteristics of the distributions. (ii) Assume that ∆ satises the identity of indiscernibles, i.e. for µ , ν ∈ P(R), ∆(µ , ν ) = 0 if and only if µ = ν . For any µ ∈ P(R d ) and θ ∈ S d-1 , ∆(θ µ, θ µ) = 0, therefore S∆ p (µ, µ) = 0 by its denition (7.1). Now, consider µ, ν ∈ P(R d ) such that S∆ p (µ, ν) = 0. Then, by the denition of S∆ p (7.1), we have ∆(θ µ, θ ν) = 0 for σ-almost every (σ-a.e.) θ ∈ S d-1 , therefore θ µ = θ ν for σ-a.e. θ ∈ S d-1 . Next, we use the same technique as in [Bonnotte, 2013, Proposition 5.1.2]: for any measure ξ ∈ P(R s ) (s ≥ 1), F[ξ] denotes the Fourier transform of ξ and is dened as, for any w ∈ R s ,

F[ξ](w) =

R s e -i w,x dξ(x) .

(7.9) Then, by using (7.10) and the property of push-forward measures, we have for any t ∈ R and θ ∈ S d-1 ,

F[θ µ](t) = R e -itu dθ µ(u) = R d e -it θ,x dµ(x) = F[µ](tθ) . (7.10) Since for σ-a.e. θ ∈ S d-1 , θ µ = θ ν thus F[θ µ] = F[θ ν], we obtain F[µ] = F[ν]
. By the injectivity of the Fourier transform, we conclude that µ = ν.

(iii) Suppose ∆ is a metric. Based on the previous results, to show that S∆ p is a metric, all we need to prove here is that it veries the triangle inequality. Let µ, ν, ξ ∈ P(R d ). Using that ∆ satises the triangle inequality and the Minkowski inequality in L p (S d-1 , σ), we get with µ ∈ P(R d ), and assume that the convergence in ∆ implies the weak convergence in P(R). Then, there exists an increasing function φ : N → N such that the subsequence (µ φ(k) ) k∈N converges weakly to µ.

S∆ p (µ, ν) = S d-1 ∆ p θ µ, θ ν dσ(θ) 1/p ≤ S d-1 ∆ θ µ, θ ξ + ∆ θ ξ, θ ν p dσ(θ) 1/p ≤ S d-1 ∆ p θ µ, θ ξ dσ(θ) 1/p + S d-1 ∆ p θ ξ, θ ν dσ(θ) 1/p ≤ S∆ p (µ, ξ) + S∆ p (ξ, ν) .
Proof. We assume that lim k→∞ S∆ 1 (µ k , µ) = 0, i.e.:

lim k→∞ S d-1 ∆(θ µ k , θ µ)dσ(θ) = 0 (7.12)
By [Bogachev, 2007, Theorem 2.2.5], (7.12) implies that, there exists an increasing function φ : N → N such that for σ-a.e. θ ∈ S d-1 , lim k→∞ ∆(θ µ φ(k) , θ µ) = 0 .

Since ∆ is assumed to imply weak convergence in P(R), then, for σ-a.e. θ ∈ S d-1 , (θ µ φ(k) ) k∈N converges weakly to θ µ. By Lévy's characterization [Kallenberg, 1997, Theorem 4.3], we have for σ-a.e. θ ∈ S d-1 and any s ∈ R,

lim k→∞ Φ θ µ φ(k) (s) = Φ θ µ (s) , (7.13)
where Φ ν is the characteristic function of ν ∈ P(R s ) (s ≥ 1) and is dened for any v ∈ R s as, Φ ν (v) = R s e i v,w dν(w). Therefore, for Lebesgue (Leb)-almost every z ∈ R d ,

lim k→∞ Φ µ φ(k) (z) = Φ µ (z) . (7.14)
We now use (7.14) to show that (µ φ(k) ) k∈N converges weakly to µ. By [Billingsley, 1999, Problem 1.11, Chapter 1], this boils down to proving that, for any f : R d → R continuous with compact support,

lim k→∞ R d f (z)dµ φ(k) (z) = R d f (z)dµ(z) .
x ∈ R d , f σ (x) = (2πσ 2 ) -d/2 R d f (x -z) exp -z 2 /(2σ 2 ) dz = f * g σ (x) , (7.16)
where * is the convolution product, and g σ is the density of the d-dimensional Gaussian with zero mean and covariance matrix σ 2 I d . First, we prove that (7.15) holds with f σ in place of f . The characteristic function associated to a d-dimensional Gaussian random variable G with zero mean and covariance matrix (1/σ 2 )I d is given by: for any z ∈ R d , E e i z,G = e -z 2 /(2σ 2 ) . By plugging this in the denition of f σ and using Fubini's theorem, we obtain for any k ∈ N,

R d f σ (z)dµ φ(k) (z) = R d R d f (w)g σ (z -w)dwdµ φ(k) (z) = (2πσ 2 ) -d/2 R d R d f (w) R d e i z-w,x g 1/σ (x)dxdwdµ φ(k) (z) = (2πσ 2 ) -d/2 R d R d f (w)e -i w,x g 1/σ (x)Φ µ φ(k) (x)dxdw = (2πσ 2 ) -d/2 R d F[f ](x)g 1/σ (x)Φ µ φ(k) (x)dx , (7.17) where F[f ](x) = R d f (w)e -i w,
x dw is the Fourier transform of f . Since the support of f is assumed to be compact, F[f ] exists and is bounded by

R d |f (w)| dw < +∞, therefore, for any k ∈ N and x ∈ R d , F[f ](x)g 1/σ (x)Φ µ φ(k) (x) ≤ g 1/σ (x) R d |f (w)|dw . (7.18)
We can prove with similar techniques that (7.17) holds with µ in place of µ φ(k) , i.e.

R d f σ (z)dµ(z) = (2πσ 2 ) -d/2 R d F[f ](x)g 1/σ (x)Φ µ (x)
lim k→∞ (2πσ 2 ) -d/2 R d F[f ](x)g 1/σ (x)Φ µ φ(k) (x)dx = (2πσ 2 ) -d/2 R d F[f ](x)g 1/σ (x)Φ µ (x)dx , i.e., lim k→∞ R d f σ (z)dµ φ(k) (z) = R d f σ (z)dµ(z) . (7.20)
We can now prove (7.15): for any σ > 0,

R d f (z)dµ φ(k) (z) - R d f (z)dµ(z) (7.21) ≤ 2 sup z∈R d |f (z) -f σ (z)| + R d f σ (z)dµ φ(k) (z) - R d f σ (z)dµ(z) .
(7.22) By (7.20), we deduce that for any σ > 0,

lim sup k→+∞ R d f (z)dµ φ(k) (z) - R d f (z)dµ(z) ≤ 2 sup z∈R d |f (z) -f σ (z)| , (7.23)
and since lim σ→0 sup z∈R d |f (z)-f σ (z)| = 0 [Folland, 1999, Theorem 8.14-b], we conclude that (µ φ(k) ) k∈N converges weakly to µ.

We can now prove Theorem 7.2.

Proof of Theorem 7.2. Let p ∈ [1, ∞) and (µ k ) k∈N be a sequence of probability measures in P(R d ).

First, suppose (µ k ) k∈N converges weakly to µ ∈ P(R d ). By the continuous mapping theorem, since for any θ ∈ S d-1 , θ is a bounded linear form thus continuous, then (θ µ k ) k∈N converges weakly to θ µ. Therefore, according to our assumption on ∆, for any θ ∈ S d-1 , lim k→∞ ∆(θ µ k , θ µ) = 0 .

(7.24)

Besides, ∆ is assumed to be non-negative and bounded. Hence, there exists M > 0 such that, for any k ∈ N, ∆ p (θ µ k , θ µ) ≤ M .

(7.25) Using (7.24), (7.25) and the bounded convergence theorem, we obtain

lim k→∞ S∆ p p (µ k , µ) = lim k→∞ S d-1 ∆ p (θ µ k , θ µ)dσ(θ) = S d-1 0 p dσ(θ) = 0 . (7.26)
Since the mapping t → t 1/p is continuous on R+ (and can be applied to S∆ p p , which is non-negative by the non-negativity of ∆ and Proposition 7.1), then (7.26) implies lim k→∞ S∆ p (µ k , µ) = 0.

Now, let us prove the other implication, i.e. lim k→∞ S∆ p µ k , µ = 0 implies the weak convergence of (µ k ) k∈N to µ, given the assumptions on ∆. This result is a generalization of Theorem 3.1, and is proved analogously, using Lemma 7.18: consider (µ k ) k∈N and µ in P(R d ) such that lim k→∞ S∆ p (µ k , µ) = 0 , (7.27) and suppose (µ k ) k∈N does not converge weakly to µ. Then, lim k→∞ d P (µ k , µ) = 0, where d P is the Lévy-Prokhorov metric, i.e. there exists > 0 and a subsequence (µ ψ(k) ) k∈N with ψ : N → N increasing, such that for any k ∈ N, d P (µ ψ(k) , µ) > .

(7.28)

On the other hand, an application of Hölder's inequality on S d-1 gives for any µ, ν in P(R d ), S∆ 1 (µ, ν) ≤ S∆ p (µ, ν) .

(7.29)

Then, by (7.27), lim k→∞ S∆ 1 (µ ψ(k) , µ) = 0. Since we assume the convergence in ∆ implies the weak convergence in P(R), Lemma 7.18 gives us: there exists a subsequence (µ φ(ψ(k)) ) k∈N with φ : N → N increasing such that (µ φ(ψ(k)) ) k∈N converges weakly to µ.

This is equivalent to

lim k→∞ d P (µ φ(ψ(k)) , µ) = 0 ,
which contradicts (7.28). We conclude that (7.27) implies the weak convergence of

(µ k ) k∈N to µ. Proof of Theorem 7.4. Let p ∈ [1, ∞) and µ, ν ∈ P(R d ). (Sγ F,p ) p (µ, ν) = S d-1 γ p F (θ µ, θ ν)dσ(θ) (7.30) = S d-1 sup f ∈ F R f (t) d(θ µ -θ ν)(t) p dσ(θ) (7.31) = S d-1 R f * (t)d(θ µ -θ ν)(t) p dσ(θ) (7.32) = S d-1 R d f * θ (x) d(µ -ν)(x) p dσ(θ) , (7.33) with f * = argmax f ∈ F R f (t)dθ µ(t) - R f (t)dθ ν(t)
, which is assumed to exist. Note that (7.33) results from applying the property of push-forward measures.

By denition of F, for any θ ∈ S d-1 , there exists f

* θ ∈ F such that f * θ = f * •θ . Therefore, we obtain (Sγ F,p ) p (µ, ν) = S d-1 R d f * θ (x)d(µ -ν)(x) p dσ(θ) (7.34) ≤ S d-1 sup f ∈F R d f (x)d(µ -ν)(x) p dσ(θ) (7.35) = γ p F (µ, ν) S d-1 dσ(θ) = γ p F (µ, ν) , ( 7
x ∈ R s by, F[f ](x) = R s f (w)e -i w,x dw. Let g ∈ G.
By the isometry properties of the Fourier transform and the denition of ϕ λ , we have (7.37) where µ λ = µ * ϕ λ and ν λ = ν * ϕ λ . By representing w with its polar coordinates

R d g(x)d(µ λ -ν λ )(x) = R d F[g](w) {F[µ](w) -F[ν](w)} F[ϕ](λw)dw ,
(r, θ) ∈ [0, ∞) × S d-1 , we obtain R d g(x)d(µ λ -ν λ )(x) = S d-1 ∞ 0 F[g](rθ) {F[µ](rθ) -F[ν](rθ)} F[ϕ](λr)r d-1 drdσ(θ) . Since g is a real function, F[g] is an even function, hence R d g(x)d(µ λ -ν λ )(x) (7.38) = 1 2 S d-1 R F[g](rθ) {F[µ](rθ) -F[ν](rθ)} F[ϕ](λr) |r| d-1 drdσ(θ) (7.39) = 1 2 S d-1 R F[g](rθ) F[θ µ](r) -F[θ ν](r) F[ϕ](λr) |r| d-1 drdσ(θ) (7.40) = 1 2 S d-1 R R -R F[g](rθ)e -iru d(θ µ -θ ν)(u)F[ϕ](λr) |r| d-1 drdσ(θ) (7.41) = 1 2 S d-1 R R d R -R g(x)e -ir(u+ θ,x ) d(θ µ -θ ν)(u) F[ϕ](λr) |r| d-1 dxdrdσ(θ) , (7.42)
where (7.40) follows from (7.10), (7.41) results from the denition of the Fourier transform and the fact that u ∈ [-R, R], and in the last line, we used the denition of the Fourier transform and Fubini's theorem. By making the change of variables x → xuθ, we obtain

R d g(x)d(µ λ -ν λ )(x) (7.43) = 1 2 S d-1 R R d R -R g(x -uθ)e -ir θ,x d(θ µ -θ ν)(u)F[ϕ](λr) |r| d-1 dxdrdσ(θ) .
(7.44)

Since we assumed supp(µ), supp(ν) are included in B d (0, R), then supp(µ λ ), supp(µ λ ) are in B d (0, R+λ), and the domain of x → g(x-uθ) must be contained in B d (0, 2R+λ). By Fubini's theorem and the denition of G, we have

R d g(x)d(µ λ -ν λ )(x) (7.45) ≤ 1 2 R B d (0,2R+λ) S d-1 R -R g(x -uθ)d(θ µ -θ ν)(u)e -ir θ,x F[ϕ](λr) |r| d-1 dσ(θ)dxdr (7.46) ≤ 1 2 R B d (0,2R+λ) S d-1 γ G (θ µ, θ ν) e -ir θ,x F[ϕ](λr) |r| d-1 dσ(θ)dxdr (7.47) ≤ C(2R + λ) d S d-1 γ G (θ µ, θ ν)dσ(θ) R λ -d F[ϕ](r) |r| d-1 dr (7.48) ≤ C(2R + λ) d λ -d S d-1 γ p G (θ µ, θ ν)dσ(θ) 1/p R F[ϕ](r)|r| d-1 dr (7.49) ≤ C 1 (2R + λ) d λ -d Sγ G,p (µ, ν) , (7.50)
where in (7.48), C > 0 and does not depend on µ and ν, (7.49) results from applying Hölder's inequality on S d-1 if p > 1, and in (7.50

), C 1 = C R F[ϕ](r)|r| d-1 dr.
By using the denition of γ G and (7.50), we obtain

γ G (µ λ , ν λ ) = sup g∈G R d g(x)d(µ λ -ν λ )(x) ≤ C 1 (2R + λ) d λ -d Sγ G,p (µ, ν) .
(7.51)

We now relate γ G (µ λ , ν λ ) with γ G (µ, ν). We start with the following estimate

R d g(x)d(µ -ν)(x) -γ G (µ λ , ν λ ) (7.52) ≤ R d g(x)d(µ -ν)(x) - R d g(x)d(µ λ -ν λ )(x) (7.53) ≤ R d g(x) -(ϕ λ * g)(x) dµ(x) + R d g(x) -(ϕ λ * g)(x) dν(x) (7.54) (7.55)
Since we assumed any g ∈ G is L-Lipschitz continuous, we can bound the integrand in (7.54) as follows: for

x ∈ R d , g(x) -(ϕ λ * g)(x) = λ -d R d g(x) -g(y) ϕ (x -y)/λ dy (7.56) ≤ λ -d R d g(x) -g(y) ϕ (x -y)/λ dy (7.57) ≤ Lλ -d+1 R d x -y λ -1 ϕ (x -y)/λ dy (7.58) ≤ Lλ -d+1 R d u λ -1 ϕ u/λ du ≤ Lλ z ϕ(z)dz .
(7.59) Hence, by denoting by M 1 (ϕ) the moment of order 1 of ϕ, (7.54) is bounded by

R d g(x)d(µ -ν)(x) -γ G (µ λ , ν λ ) ≤ 2LM 1 (ϕ)λ . (7.60)
Taking the supremum of both sides over G gives us γ G (µ, ν)γ G (µ λ , ν λ ) ≤ 2LM 1 (ϕ)λ .

(7.61) By combining the above inequality with (7.51), we get

γ G (µ, ν) ≤ C 1 (2R + λ) d λ -d Sγ G,p (µ, ν) + 2LM 1 (ϕ)λ (7.62) ≤ C 2 λ (2R + λ) d λ -(d+1) Sγ G,p (µ, ν) + 1 , (7.63) with C 2 satisfying C 2 ≥ C 1 and C 2 ≥ 2LM 1 (ϕ). Finally, by choosing λ = R d/(d+1) Sγ G,p (µ, ν) 1/(d+1) ,
and using the hypothesis that Sγ G,p is bounded, we obtain

γ G (µ, ν) ≤ C 2 R d/(d+1) Sγ G,p (µ, ν) 1/(d+1) (2R + λ) d R -d + 1 (7.64) ≤ C p Sγ G,p (µ, ν) 1/(d+1) , (7.65)
for some C p > 0, as desired. This concludes the proof.

Proof of Corollary 7.6. The desired result is obtained as a direct application of Theorems 7.4 and 7.5.

7.6.2 Application of Theorems 7.4 and 7.5

In order to illustrate their assumptions, we apply Theorems 7.4 and 7.5 to well-known instances of IPMs given below. Note that some of these IPMs were already presented in Section 2.2: we recall their denitions in this chapter for completeness.

(1) Wasserstein distance of order 1. By the Monge Kantorovich duality theorem [Villani, 2008, Theorem 5.10], when F = {f : Y → R : f Lip ≤ 1}, where f Lip = sup x,y∈Y,x =y {|f (x)f (y)| / xy }, γ F is the Wasserstein distance of order 1, denoted by W 1 .

(2) Maximum mean discrepancy. Let H be a reproducing kernel Hilbert space (RKHS) for real-valued functions on Y, and F be the unit ball in H. Then, γ F denes the MMD in RKHS [Gretton et al., 2012, Section 2].

(3) Total variation distance. (TV) By choosing [Douc et al., 2018, Proposition D.2.4].

F = {f : Y → R : f ∞ ≤ 1}, with f ∞ = sup x∈Y |f (x)|, γ F corresponds to TV
Informally, the condition on the function classes in Theorem 7.4 requires that F and F should be linked to each other in the way that F should be large enough to contain the composition of all elements of F with all possible linear forms θ for θ ∈ S d-1 .

Let us illustrate this condition by considering the Wasserstein distance of order 1. In this case, F is the set of 1-Lipschitz functions from R d to R, and F is the set of 1-Lipschitz functions from R to R. Then, the condition on F boils down to showing that the composition of any f ∈ F with any linear projection θ results in a 1-Lipschitz function in R d , which is simply true since f is 1-Lipschitz and θ = 1 for all θ ∈ S d-1 .

In the next three corollaries, we formally prove that Theorem 7.4 holds for the Wasserstein distance of order 1 W 1 , total variation distance TV and maximum mean discrepancy MMD. We denote by SW 1 , STV p and SMMD p the respective sliced versions of these IPMs with order p ∈ [1, ∞).

Corollary 7.19. Let p ∈ [1, ∞). For any µ, ν ∈ P 1 (R d ), SW 1 (µ, ν) ≤ W 1 (µ, ν). Proof. Choose F = { f : R → R : f Lip ≤ 1}, where f Lip = sup x,y∈R d ,x =y f (x) -f (y) / x -y . Let f : R d → R such that f = f • θ with f ∈ F, θ ∈ S d-1
. Then, by using the Cauchy-Schwarz inequality and the denition of F, we have for any

x, y ∈ R d , |f (x) -f (y)| = f θ (x) -f θ (y) ≤ θ, x -y ≤ θ x -y ≤ x -y . (7.66) Therefore, f ∈ F = {f : R d → R : f Lip ≤ 1}. Corollary 7.
19 follows from the application of Theorem 7.4 along with the denition of W 1 .

Note that Corollary 7.19 is not a new result: the fact that SW p is bounded above by W p for p ∈ [1, ∞) was established in [Bonnotte, 2013, Proposition 5.1.3]. While their result is proved using the primal formulation of the OT problem, we used the dual formulation available for p = 1 to illustrate the applicability of Theorem 7.4. Our result is thus consistent with the existing results in the literature.

Corollary 7.20. Let p ∈ [1, ∞). For any µ, ν ∈ P(R d ),

STV p (µ, ν) ≤ TV(µ, ν) . (7.67) Proof. Choose F = f : R → R, f ∞ ≤ 1 , and let f : R d → R such that f = f • θ with f ∈ F, θ ∈ S d-1 . Then, f ∞ = f • θ ∞ = sup x∈R d f θ (x) ≤ sup t∈R f (t) = f ∞ ≤ 1 , (7.68) hence, f ∈ F = f : R d → R : f ∞ ≤ 1 .
We obtain the nal result by using Theorem 7.4 and the denition of TV.

Corollary 7.21. Let F ⊂ M b (R) be the unit ball of the RKHS with reproducing kernel k, and k be the positive denite kernel such that for any In particular, this property holds for, of the linear kernel in R

x i , x j ∈ R d , k(x i , x j ) = S d-1 k θ (x i ), θ (x j ) dσ(θ) .
d : k(x i , x j ) = x i x j /d for x i , x j ∈ R d and d ≥ d. Then, for any n ∈ N, v 1 , . . . , v n ∈ R and x 1 , . . . , x n ∈ R d , n i=1 n j=1 v i v j {k(x i , x j ) -k(x i , x j )} (7.77) = n i=1 n j=1 v i v j S d-1 θ(x i )θ(x j )dσ(θ) -x i x j /d (7.78) = n i=1 n j=1 v i v j x i S d-1 θθ dσ(θ) x j -x i x j /d (7.79) = n i=1 n j=1 v i v j x i x j 1/d -1/d ≥ 0 , (7.80)
where (7.80) results from n i=1 n j=1 v i v j x i x j ≥ 0 (the linear kernel is positive denite) and d ≥ d. We conclude that (7.76) holds with F dened as the unit ball of the RKHS associated with the linear kernel k(t i , t j ) = t i t j for t i , t j ∈ R, and F being the unit ball of the RKHS associated with the rescaled linear kernel k(x i , x j ) = x i x j /d for x i , x j ∈ R d and d ≥ d.

We focus now on RBF kernels: let h ≥ 0 and choose k(t i , t j ) = e -|t i -t j | 2 /h for t i , t j ∈ R, and k(x i , x j ) = e -x i -x j 2 /h for x i , x j ∈ R d . We have for any (7.84) where M (a, c, κ) stands for the conuent hypergeometric function evaluated at a, c, κ ∈ R, and appears in the normalizing constant of the multivariate Watson distribution: see [Sra, 2016, Section 2.3] for more details.

x i , x j ∈ R d , k(x i , x j ) = S d-1 k θ(x i ), θ(x j ) dσ(θ) = S d-1 e -|θ x i -θ x j | 2 /h dσ(θ) (7.81) = S d-1 e -|θ (x i -x j )| 2 /h dσ(θ) (7.82) = S d-1 e (-x i -x j 2 /h)(θ (x i -x j )/ x i -x j ) 2 dσ(θ) (7.83) = M 1 2 , d 2 , - x i -x j 2 h ,
M satises the following property,

M 1 2 , d 2 , - x i -x j 2 h = e -x i -x j 2 /h M d -1 2 , d 2 , x i -x j 2 h .
(7.85) Since x ix j 2 /h ≥ 0 and κ → M (•, •, κ) is increasing, we have

M d -1 2 , d 2 , x i -x j 2 h ≥ M d -1 2 , d 2 , 0 = M 1 2 , d 2
, 0 = 1 . where the last line follows from (7.86) and n i=1 n j=1 v i v j e -x i -x j 2 /h ≥ 0 (RBF kernels are positive denite). We conclude that k -k is positive denite, hence (7.76) holds for RBF kernels.

As with Theorem 7.4, Theorem 7.5 assumes that the function classes G and G are linked to each other and suciently regular. The condition on G is veried with W 1 (simply by denition) and MMD (provided that the reproducing kernel is Lipschitzcontinuous, which holds on compact spaces for classical choices of kernels), but not with TV.

On the other hand, the second condition requires G to be large enough to contain any possible slice g(xuθ) for any g ∈ G. α(p, n)dσ(θ) = α(p, n) .

(7.98)

Additionally, if we assume that ∆ satises non-negativity, symmetry and the triangle inequality, then S∆ p also veries these three properties by Proposition 7.1, and we can derive its sample complexity: for any µ, ν in P(R d ) with respective empirical measures μn , νn , the triangle inequality give us |S∆ p (µ, ν) -S∆ p (μ n , νn )| ≤ S∆ p (μ n , µ) + S∆ p (ν n , ν) Proof of Theorem 7.9. Let p ∈ [1, ∞) and µ, ν ∈ P(R d ). We recall that S∆ p,L (µ, ν) denotes the approximation of S∆ p (µ, ν) obtained with a Monte Carlo scheme that uniformly picks L projection directions on S d-1 (cf. Equation (7.2)).

By using Hölder's inequality and the results on the moments of the Monte Carlo estimation error, we obtain ∆ p (θ µ, θ ν) -S∆ p p (µ, ν) 2 dσ(θ)

is the variance of ∆ p (θ µ, θ ν) with respect to θ ∼ σ. where Var is the variance w.r.t. X 1:n , Y 1:n and θ (which is distributed according to the uniform distribution on S d-1 and independent of X 1:n , Y 1:n ).

7.6.4 Proofs for Section 7.3.1

As discussed in Section 7.3.1, we can use the general result in Theorem 7.2 to establish novel topological properties for specic sliced probability divergences, for example the Sliced-Cramér distance (whose denition is recalled in Denition 7.12) and the broader class of Sliced-IPMs. We present our results and proofs for these examples below.

Proof of Corollary 7.13. Let p ∈ [1, ∞). By Hölder's inequality, for any µ , ν ∈ P(R), By (7.114), this implies lim k→∞ C 1 (µ k , µ ) = 0. Since the Cramér distance of order 1 is equivalent to the Wasserstein distance of order 1, then by [Villani, 2008, Theorem 6.8],

the convergence of (µ k ) k∈N to µ under C p implies (µ k ) k∈N converges weakly to µ in P(R). By Theorem 7.2, we conclude that the convergence under SC p implies the weak convergence in P(R d ).

We now show the second part of the statement. This result partly follows from slight modications of the techniques we used in the proof of Theorem 7.2.

Consider a compact space K ⊂ R and a sequence (µ k ) k∈N in P(K ). Suppose that (µ k ) k∈N converges weakly to µ ∈ P(K ). Since F µ is non-decreasing, it is almost everywhere continuous w.r.t. to the Lebesgue convergence, and using the Portmanteau theorem, we get that for Leb-almost every t ∈ R, lim k→∞ F µ k (t) = F µ (t). Besides, for any k ∈ N and t ∈ K , |F µ k (t)| ≤ 1, and since K is compact, K 1 p dt 1/p < ∞. By the dominated convergence theorem in L p -spaces, we conclude that (7.115) in other words, the weak convergence of measures in P(K ), where K is a compact subspace of R, implies the convergence under C p . Now, consider a compact space K ⊂ R d and a sequence (µ k ) k∈N in P(K) which converges weakly to µ ∈ P(K). For any θ ∈ S d-1 , dene K θ = { θ, x : x ∈ K} , which is a compact subset of R (since it is the image of K by a continuous function) with diam(K θ ) ≤ diam(K) (by the Cauchy-Schwarz inequality). The sequence of push-forward measures (θ µ k ) k∈N is in P(K θ ) and, by the continuous mapping theorem, converges weakly to θ µ ∈ P(K θ ). Therefore, by (7.115), for any θ ∈ S d-1 , lim k→∞ C p (θ µ k , θ µ) = 0 . Suppose that the space spanned by F is dense in the space of continous functions for • ∞ . Then, the convergence under the Sliced Integral Probability Metric of order p associated with F, Sγ F,p , implies the weak convergence in P(R d ). Besides, if γ F is bounded, the converge implication holds, i.e. the weak convergence in P(R d ) implies the convergence under Sγ F,p .

lim k→∞ K |F µ k (t) -F µ (t)| p dt 1/p = 0 ,
Proof. By construction of F and [Ambrosio et al., 2005, Section 5.1], γ F metrizes the weak convergence in P(R), i.e. the weak convergence in P(R) is equivalent to the convergence of measures under γ F . The properties of Sγ F,p , p ∈ [1, ∞) result from the application of Theorem 7.2.

Remark 7.24. The boundedness assumption for γ F is achieved if we additionally suppose that F is a uniformly bounded family of functions in M(R), which is a mild assumption. 7.6.5 Proof of Corollary 7.14 Lemma 7.25. Let p ∈ [1, ∞) and µ ∈ P(R) with empirical distribution μ n . Suppose there exists q > p such that the moment of order q of µ , dened as M q (µ ) = R |t| q dµ (t), is bounded above by K < ∞. Then, there exists a constant C p,q depending on p, q such that

E W p p (μ n , µ ) ≤ C p,q K        n -1/2 if q > 2p,
n -1/2 log(n) if q = 2p, n -(q-p)/q if q ∈ (p, 2p).

(7.118)

Proof. This immediately results from [Fournier and Guillin, 2015, Theorem 1].

Proof of Corollary 7.14. We rst recall that, for any ξ ∈ P(R s ) (s ≥ 1) and θ ∈ S d-1 , the moment of order k > 0 of θ ξ is lower than the one associated with ξ. Indeed, by using the property of push-forward measures, the Cauchy-Schwarz inequality, and θ ≤ 1, we have

M k (θ ξ) = R |t| k dθ ξ(t) = R d | θ, x | k dξ(x) ≤ R d
x k dξ(x) = M k (ξ) . (7.119) Now, let p ∈ [1, ∞) and µ ∈ P q (R d ) (q > p) with empirical distribution μn . Then, by (7.119), for any θ ∈ S d-1 , M q (θ µ) ≤ M q (µ) < ∞, and we can apply Lemma 7.25 and Theorem 7.8 to derive the convergence rate under SW p : there exists a constant C p,q such that, E SW p p (μ n , µ) ≤ C p,q M p/q q (µ)

      
n -1/2 if q > 2p, n -1/2 log(n) if q = 2p, n -(q-p)/q if q ∈ (p, 2p).

(7.120)

Besides, since W p is a metric, we can apply Theorem 7.8 to derive the sample complexity of SW p . Consider µ, ν ∈ P q (R d ) with q > p, with respective empirical measures μn , νn . Then, starting from (7.101) and using the convergence rate derived in (7.120), we obtain the desired result as follows ≤ C 1/p p,q M 1/q q (µ) + M 1/q q (ν)

      
n -1/(2p) if q > 2p, n -1/(2p) log(n) 1/p if q = 2p, n -(q-p)/(pq) if q ∈ (p, 2p). where C(R) denotes the set of continuous real functions, and φ ε (t, s, x, y) = t + sεe (t+s-x-y p )/ε . Consider for any θ ∈ S d-1 , ũ θ , ṽ θ as the functions attaining the maximum in (7.125), which exist by [Genevay et al., 2019, Theorem 4 (7.129) By Proposition 7.1, since W p,ε is non-negative, so is SW p,ε , and we can apply t → t 1/p on both sides of (7.129) to obtain the nal result.

We move on to the proof of Theorem 7.16, which requires preliminary technical results.

Proposition 7.26. Let X be a compact subset of R, and µ , ν ∈ P( X) with respective empirical instantiations μ n , ν n . Let p ∈ [1, ∞) and ε ≥ 0. Then, E µ ⊗ν φ ε ũ( X), ṽ( Ỹ ), X, Ỹ E μ n ⊗ν φ ε ũ( X), ṽ( Ỹ ), X, Ỹ , where φ ε (t, s, x, y) = t + sεe (t+s-x-y 2 /2)/ε . By [Genevay et al., 2019, Proposition 1], the Sinkhorn potentials (ũ, ṽ) are Lipschitz continuous with Lipschitz constant diam( X) < ∞. Therefore, by denoting by Lip diam( X) (R) the space of diam( X)-Lipschitz continuous functions dened on R, (7.131) and (7.132) can be rewritten with the maximization over Lip diam( X) (R).

W p,ε (μ n , ν n ) -W p,ε (µ , ν ) ≤ 2 diam( X) W 1 (µ , μ n ) + W 1 (ν , ν n ) .
We can now use [Mena and Niles-Weed, 2019, Proposition 2] to bound the absolute dierence of W p,ε (µ , ν ) and W p,ε (μ n , ν ). We provide the detailed proof below for completeness. By [Mena and Niles-Weed, 2019, Proposition 6, Appendix A], there exist smooth potentials (ũ , ṽ ) attaining the maximum in (7.131) such that, for all x, ỹ ∈ R, R φ ε (ũ (x), ṽ (ỹ), x, ỹ)dν (ỹ) = 1 µ -almost surely, (7.133) R φ ε (ũ (x), ṽ (ỹ), x, ỹ)dµ (x) = 1 ν -almost surely .

(7.134)

Analogously, there exist smooth optimal potentials (ũ n , ṽ n ) for (7.132) satisfying (7.133) and (7.134) where ũ , ṽ and µ are replaced by ũ n , ṽ n and μ n respectively.

The optimality of these potentials give us E µ ⊗ν φ ε (ũ n ( X), ṽ n ( Ỹ ), X, Ỹ ) -E μ n ⊗ν φ ε (ũ n ( X), ṽ n ( Ỹ ), X, Ỹ ) (7.135) ≤ E µ ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ ) -E μ n ⊗ν φ ε (ũ n ( X), ṽ n ( Ỹ ), X, Ỹ ) (7.136) ≤ E µ ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ ) -E μ n ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ ) . = E µ ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ ) -E μ n ⊗ν φ ε (ũ n ( X), ṽ n ( Ỹ ), X, Ỹ ) (7.139) ≤ E µ ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ ) -E μ n ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ ) (7.140) + E µ ⊗ν φ ε (ũ n ( X), ṽ n ( Ỹ ), X, Ỹ ) -E μ n ⊗ν φ ε (ũ n ( X), ṽ n ( Ỹ ), X, Ỹ ) . (7.141)

We bound each term of the sum in (7.141) as follows E µ ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ ) -E μ n ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ )

(7.142) = R ũ (x)d(µ -μ n )(x) -ε R R
e (ũ (x)+ṽ (ỹ)-|x-ỹ| 2 /2)/ε dν (ỹ)d(µμ n )(x) 7.144) where (7.144) results from (7.133). Since for any f ∈ Lip L (R) with L > 0, f /L ∈ Lip 1 (R),

(7.143) = R ũ (x)d(µ -μ n )(x) ≤ sup ũ∈Lip diam( X) (R) R ũ(x)d(µ -μ n )(x) , (
(7.144) can be bounded as follows E µ ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ ) -E μ n ⊗ν φ ε (ũ ( X), ṽ ( Ỹ ), X, Ỹ ) where (7.146) follows from the dual formulation of the Wasserstein distance of order 1 [Villani, 2008, Theorem 5.10].

We show with an analogous proof that E µ ⊗ν φ ε (ũ n ( X), ṽ n ( Ỹ ), X, Ỹ ) -E μ n ⊗ν φ ε (ũ n ( X), ṽ n ( Ỹ ), X, Ỹ ) (7.147)

≤ diam( X)W 1 (µ , μ n ) , (7.148) which leads to the conclusion that W p,ε (µ , ν ) -W p,ε (μ n , ν ) ≤ 2 diam( X)W 1 (µ , μ n ) . ≤ 2 diam( X) W 1 (µ , μ n ) + W 1 (ν , ν n ) .

(7.152)

Corollary 7.27. Let X be a compact subset of R, and µ , ν ∈ P( X). Denote by μ n , ν n their respective empirical instantiations. Let p ∈ [1, ∞) and ε ≥ 0. Then, E W p,ε (μ n , ν n ) -W p,ε (µ , ν ) ≤ 2 diam( X)C q M 1/q q (µ ) + M 1/q q (ν ) n -1/2 , (7.153)

where q > 2, C q < ∞ is a constant that depends on q, and M q (µ ), M q (ν ) are the moments of order q of µ , ν respectively.

Proof. We apply Proposition 7.26 and take the expectation of (7.130) with respect to X1:n ∼ μ n and Ỹ1:n ∼ ν n E W p,ε (μ n , ν n ) -W p,ε (µ , ν ) ≤ 2 diam( X)E W 1 (µ , μ n ) + W 1 (ν , ν n ) . (7.154)

Since µ and ν are both supported on a compact space, they have innitely many nite moments. We can then bound (7.154) using the convergence rate of empirical measures in W 1 , recalled in Lemma 7.25. This concludes the proof.

Proof of Theorem 7.16. Let p ∈ [1, ∞) and ε ≥ 0. Consider µ, ν ∈ P(X) with X ⊂ R d compact, and denote by μn , νn their respective empirical distributions. Let θ ∈ S d-1 and dene X θ = { θ, x : x ∈ X}. X θ is compact (since X is compact and θ is continuous) and veries diam(X θ ) ≤ diam(X) (by the Cauchy-Schwarz inequality). Besides, by (7.119), for any k > 0, M k (θ µ) ≤ M k (µ) and M k (θ ν) ≤ M k (ν). By Corollary 7.27, there exists C q < ∞ which depends on q > 2 such that, E W p,ε (θ μn , θ νn ) -W p,ε (θ µ, θ ν) (7.155) ≤ 2 diam(X)C q M 1/q q (µ) + M 1/q q (ν) n -1/2 .

(7.156)

The sample complexity of SW p,ε is nally obtained by applying Theorem 7.7.

Finally, in addition to the background elements given in Section 2.5, we recall an important result regarding the convergence of Sinkhorn's algorithm, which will be useful for the proof of Proposition 7.17. techniques which rely on dierent divergences. Besides, we leveraged our methodology to design a novel image denoising algorithm, which demonstrates the exibility of SW-ABC.

Limitations of SW and solutions. This paragraph provides answers to objective 3.

As illustrated in previous empirical studies, the common practice that consists in approximating SW with a standard Monte Carlo estimate (over L ∈ N * random projections) can be inecient in high-dimensional settings and might thus degrade the performance of SW-based generative models. We support this observation with our theoretical ndings in Chapter 7: we showed that the overall complexity of computing SW (and more generally, sliced divergences, as explained in objective 4) depends on the sample complexity but also on an error due to the Monte Carlo approximation, which depends on L and an additional variance term.

We then proposed two dierent approaches to overcome the limitations induced by the Monte Carlo estimates of SW. On the one hand, the literature on concentration of random projections, combined with the analytical expression of the Wasserstein distance between Gaussian distributions, helped us formulate a novel technique to compute SW with simple deterministic operations (Chapter 6). The returned SW estimates are guaranteed to converge to the exact value of SW, provided that the compared distributions satisfy a weak dependence condition. Besides, our approximation method can be used to improve the performance of generative models that are based on traditional Monte Carlo estimates.

On the other hand, we dened the class of Generalized Sliced-Wasserstein distances in Chapter 5 and illustrated their ability to outperform SW on generative modeling applications. These novel metrics bring an interesting perspective on adversarial generative modeling, showing that such algorithms contain an implicit stage for learning projections with dierent cost functions than ours.

Theoretical analysis of slicing. This paragraph provides answers to objective 4.

In order to understand what the slicing operation itself is bringing, we introduced in Chapter 7 the rst general denition for sliced divergences, and derived their topological and statistical properties. Our results show that slicing leads to a dimension-free sample complexity, while carrying out useful topological properties of the base divergence, e.g., metric axioms and metrizing weak convergence. If the focus is on sustaining such topological properties, then the improvement in the convergence rate is meaningful and circumvents the curse of dimensionality but in practice, this rate is impacted by the Monte Carlo approximation.

Future Research Directions

Eventually, we hope that our contributions provide useful insights for practitioners in terms of designing new methodologies based on the Sliced-Wasserstein distance and its variants, as well as obtaining a better understanding of these tools. This thesis also opens up new prospects which motivate future research directions, notably on the approximation of sliced divergences and the analysis of GSW, as we describe below.

Improve the estimation of sliced divergences. As we argued in Chapters 5 to 7, the Monte Carlo strategy, widely used to compute sliced probability divergences in practice, is not ideal given the induced approximation error. Our alternative SW estimate, Abstract : Many methods for statistical inference and generative modeling rely on a probability divergence to effectively compare two probability distributions. The Wasserstein distance, which emerges from optimal transport, has been an interesting choice, but suffers from computational and statistical limitations on large-scale settings. Several alternatives have then been proposed, including the Sliced-Wasserstein distance (SW), a metric that has been increasingly used in practice due to its computational benefits. However, there is little work regarding its theoretical properties. This thesis further explores the use of SW in modern statistical and machine learning problems, with a twofold objective: 1) provide new theoretical insights to understand in depth SW-based algorithms, and 2) design novel tools inspired by SW to improve its applicability and scalability. We first prove a set of asymptotic properties on the estimators obtained by minimizing SW, as well as a central limit theorem whose convergence rate is dimension-free. We also design a novel likelihood-free approximate inference method based on SW, which is theoretically grounded and scales well with the data size and dimension. Given that SW is commonly estimated with a simple Monte Carlo scheme, we then propose two approaches to alleviate the inefficiencies caused by the induced approximation error: on the one hand, we extend the definition of SW to introduce the Generalized Sliced-Wasserstein distances and illustrate their advantages on generative modeling applications; on the other hand, we leverage concentration of measure results to formulate a new deterministic approximation for SW, which is computationally more efficient than the usual Monte Carlo technique and has nonasymptotical guarantees under a weak dependence condition. Finally, we define the general class of sliced probability divergences and investigate their topological and statistical properties; in particular, we establish that the sample complexity of any sliced divergence does not depend on the problem dimension.

Institut Polytechnique de Paris 91120 Palaiseau, France

  Dirac measure with mass on y Leb d Lebesgue measure on R d N (m, Σ) Gaussian distribution with mean m and covariance matrix Σ µ ⊗ ν Product measure of two probability distributions µ and ν μn Empirical distribution computed over n ∈ N * samples i.i.d. from the probability distribution µ, μn = n -1 n i=1 δ x i F µ Cumulative distribution function of the probability distribution µ F -1 µ Quantile function of the probability distribution µ F[µ]

Figure 1 . 1 :

 11 Figure 1.1: Illustration of Gaussian mixture models: the goal is to t the true unknown distribution of the observations, µ , with a mixture of K = 7 Gaussian distributions, µ θ .

Figure

  Figure courtesy of Soheil Kolouri (adapted from one of his presentations).

Figure 1 . 2 :

 12 Figure 1.2: Illustration of implicit generative modeling: ζ is a distribution easy to sample from (e.g., a standard Gaussian distribution), which is then mapped to a more complicated distribution µ θ by applying the parametric map T θ (e.g., a deep neural network). The goal is to t the true unknown distribution of the observations, µ , with µ θ . Figure courtesy of Soheil Kolouri (adapted from one of his presentations).

Figure 1 . 3 :

 13 Figure 1.3: Illustration of the Sliced-Wasserstein distance of order p ∈ [1, +∞). The two distributions to compare, µ and ν, are projected along any direction u on the unit sphere S d-1 . This gives the univariate distributions denoted by u µ and u ν, which are then compared with the Wasserstein distance of order p. SW is nally dened as E[W p p (u µ, u ν)] where the expectation is computed over u uniformly distributed on S d-1 . Figure courtesy of Soheil Kolouri (adapted from[START_REF] Kolouri | Optimal mass transport: Signal processing and machine-learning applications[END_REF]).

Finally, only a

  few studies have investigated the sample complexity of SW to justify the statistical eciency of SW-based methods. If µ, ν are two isotropic Gaussian distributions supported on R d , then |SW 2 (μ n , νn ) -SW 2 (µ, ν)| can be bounded with high probability using the concentration inequality derived in[Deshpande et al., 2019, Claim 1], which shows that SW 2 oers a polynomial sample complexity. A very recent study derived two upper bounds on E[SW p (μ n , µ)], which demonstrate that the rate in the sample complexity of SW p cannot be worse than n -1/2p and can achieve n -1/2 , under specic assumptions on the regularity of µ as measured by a certain functional[Manole et al., 2019, Proposition 1].

Figure 3 . 1 :

 31 Figure 3.1: Probability density estimates of the MSWE σ2 n of order 1, centered and

  generated dataset z 1:m of size m = n for dierent values of n. We also verify the convergence of MESWE to MSWE (Theorem 3.4): we compute these estimators on a xed set of n = 2000 observations for dierent m, and we measure the error between them for each m. Results are shown in Figure3.2. We see that our estimators indeed converge to (m , σ 2 ) as the number of observations increases(Figures 3.2a, 3.2b), and on a xed observed dataset, MESWE converges to MSWE as we generate more samples (Figure 3.2c).

  Figure 3.3: Min. SW estimation for the location parameter of multivariate elliptically contoured stable distributions. Figure 3.3a compares the quality of the estimation provided by SW and Wasserstein-based estimators as well as their average computational time, for dierent values of dimension d.Figure 3.3b and Figure 3.3c illustrate, for d = 10, the consistency of MESWE mn,m and its convergence to the MSWE mn . Re-

  Figure 3.3b and Figure 3.3c illustrate, for d = 10, the consistency of MESWE mn,m and its convergence to the MSWE mn . Re- sults are averaged over 100 runs, the shaded area represent the standard deviation.

  Figure 3.4:

  and we use the MNIST dataset [LeCun and Cortes, 2010], made of 60 000 training images and 10 000 test images of size 28 × 28. Our training objective is MESWE of order 2 approximated with 20 random projections and 20 dierent generated datasets. We study the consistent behavior of the MESWE by training the neural network on dierent sizes n of training data and dierent numbers m of generated samples, and by comparing the nal training loss and test loss to the ones obtained when learning on the whole training dataset (n = 60 000) and m = 200. Results are averaged over 10 runs and shown on Figure 3.4, where the shaded areas correspond to the standard deviation over the runs. We observe that our results conrm Theorem 3.3.

  3.5.2 Proof of Theorem 3.1 Lemma 3.16. Let (µ k ) k∈N be a sequence of probability measures on R d and µ a measure in R d such that lim k→∞ SW 1 (µ k , µ) = 0 .

  By (3.13), (3.15), (3.16) and Lebesgue's Dominated Convergence Theorem, we obtain

  where (3.22) is obtained by lower semi-continuity since μψ(φ(n)) (ω) w -→ µ by A2 and Theorem 3.1, and µ θ ψ(φ(n)) w -→ µθ by A1. We conclude that the rst condition in (3.20) holds. Now, we x O ⊂ Θ open. There exists a sequence

  µ θ ) where (3.27) follows from the triangle inequality, and (3.28) is obtained on one hand by lower semi-continuity since μψ(φ(n)) (ω) w -→ µ by A2 and Theorem 3.1 and µ θ ψ(φ(n)) w -→ µθ by A1, and on the other hand by A4 which gives lim sup n→∞

  and by (3.30) and (3.31))

  35) is obtained by A4 on one hand and by lower semi-continuity on the other hand since µ θ ψ(φ(n)) w -→ µθ by A1. We conclude that the rst condition in (3.33) holds. Now, we x O ⊂ Θ open. There exists a sequence (θ m ) m∈N in O such that the sequence {SW p (μ n , μθm,m )} m∈N converges to inf θ∈O SW p (μ n , μθ,m ), and for any m ∈ N, inf θ∈O E [SW p (μ n , μθ,m )|Y 1:n ] ≤ E [SW p (μ n , µ θm,m )|Y 1:n ], by denition of the inmum. p (μ n , μθ,m )|Y 1:n ]

  an approximate posterior distribution by nding parameters for which the simulated data are close to the observations in terms of summary statistics. These statistics are dened beforehand and might induce a loss of information, which has been shown to deteriorate the quality of the approximation. To overcome this problem, a Wasserstein-based ABC technique has recently been proposed, and compares the datasets via the Wasserstein distance between their empirical distributions, but does not scale well to the dimension or the number of samples.

Figure 4 . 1 :

 41 Figure 4.1: Comparison of OT distances and KL between data generated from ddimensional Gaussian distributions µ σ vs. µ σ , σ 2 = 4, with 1000 i.i.d draws. SW is approximated with 100 random projections.

Figure 4 . 2 :

 42 Figure 4.2: Comparison of SMC-ABC strategies in the multivariate Gaussians problem.

Figure 4 .

 4 Figure 4.2 reports for d ∈ {2, 10, 20}, the resulting ABC posteriors and W 1 to the true posterior (computed with the POT package[START_REF] Flamary | POT: Python Optimal Transport[END_REF]) vs. time. Due to the poor performance of the estimator of KL between two empirical distributions proposed in[START_REF] Jiang | Approximate Bayesian Computation with Kullback-Leibler Divergence as Data Discrepancy[END_REF] (see Figure4.1), KL-ABC fails at approximating well the posterior in these experiments. Hence, we excluded those results from Figure4.2 for clarity. Euclidean-ABC provides the most accurate approximation, as expected since it

Figure 4

 4 Figure 4.3: Visualization of the results. For each couple, the left one is the noisy image (σ = 20) and the right one is the output of SW-ABC.

  class of functions that are absolutely integrable on R d w.r.t. the Lebesgue measure on R d , Leb d , and L 1 (S d-1 × R, σ ⊗ Leb 1 ) the class of absolutely integrable functions on the domain S d-1 × R w.r.t. σ ⊗ Leb 1 .

  t ∈ R and θ ∈ S d-1 . Therfore, for a xed θ ∈ S d-1 , RI(•, θ) : R → R integrates the input function I over all hyperplanes in R d that are orthogonal to θ, and (5.1) can be rewritten as RI(t, θ) = H t,θ I(x)dx .

Figure 5 . 1 :

 51 Figure 5.1: Illustration of the application of the standard or generalized Radon transform for the Half Moons distribution.

  Figure 5.1 illustrates the application of dierent Radon transforms on the Half Moons dataset and demonstrates that the output projections highly depend on the dening function. This encourages us to investigate the consequences of using the generalized Radon transform instead of the standard one in the denition of SW (5.7).

  Figure 5.2: Evaluation of the performance of GSW on the ows experiment when using dierent dening functions and four synthetic datasets as the target. Linear and Max Linear refer to SW and max-SW respectively.

Figure 5 . 3 :

 53 Figure 5.3: Comparison of max-GSW and max-SW on the ows experiment applied to the MNIST dataset.

Figure 5 . 4 :

 54 Figure 5.4: Comparison of max-GSW and max-SW on the ows experiment applied to the CelebA dataset.

  Figure 6.1: Analysis of the approximation according to the dimension: in Figures 6.1a

  and 6.1b, data have independent components; in Figures 6.1c and 6.1d, they are stationary AR(1) processes. Errors are averaged over 100 runs and reported on log-log scale with their 10th-90th percentiles.

Figure 6

 6 Figure 6.2: Comparison of dierent methods to approximate SW, according to their accuracy (left) and computation time (right). The datasets contain n samples of dimension d independently drawn from Gamma distributions, with d ∈ [10 1 , 10 3 ] and n = 10 4 . Results are averaged over 100 runs.

  Figure 6.3: Images generated after training on MNIST (top row) and CelebA (bottom row). For each model, the images are associated with the lowest FID obtained over 5 runs.

  for synthetic experiments We explain in more details the setup for the synthetic experiments discussed in Section 6.4, specically the procedure to generate data. For d ∈ N * , we generate n = 10 4 i.i.d. realizations of two random variables in R d , denoted by X 1:d = {X j } d j=1 and Y 1:d = {Y j } d j=1 and respectively distributed from µ d , ν d ∈ P 2 (R d ).

  i.i.d. samples from N (1, 1), σ 2 1 = 1 and σ 2 2 = 10. Therefore, µ d = N (m 1 , I d ) and ν d = N (m 2 , 10 I d ), where I d denotes the identity matrix of size d, and m 1 = {m

  .1b. Proposition 6.9. For d ∈ N * , letµ d = N (m 1 , σ 2 1 I d ) and ν d = N (m 2 , σ 2 2 I d ),and denote by μd , νd their centered versions, i.e. μd = N (0, σ 2 1 I d ) and νd = N (0, σ 2 2 I d ). Consider the empirical distributions μd,n , νd,n given byμd,n = (1/n) } n j=1) is a sequence of n random variables i.i.d. from µ d (respectively, from ν d ), X1:d = n -1 n j=1 X

Figure 7

 7 Figure 7.1: (Sliced-)Divergences between two sets of 1000 samples in R 10 i.i.d. from N (0, 4I) and N (0, σ 2 I), for varying σ 2 .

  Figure 7.2: (Sliced-)Wasserstein distances of order 2 between two sets of n samples generated from N (0, I d ) for dierent d, on log-log scale. Results are averaged over 100 runs, and the shaded areas correspond to the 10th-90th percentiles.

  Figures 7.3a and 7.3b show Sinkhorn and Sliced-Sinkhorn divergences vs. n, and respectively study the inuence of d and ε on the convergence rate.

  Figure 7.3: (Sliced-)Sinkhorn divergences between two sets of n samples generated from N (0, I d ) for dierent values of n, dimension d, and regularization coecient ε. Results are averaged over 100 runs, and the shaded areas correspond to the 10th-90th percentiles.

  of Theorem 7.2, we start by proving Lemma 7.18 below, which extends Lemma 3.16 to the more general class of Sliced Probability Divergences. Lemma 7.18. Consider (µ k ) k∈N a sequence in P(R d ) satisfying lim k→∞ S∆ 1 (µ k , µ) = 0 ,

  (7.15) Consider σ > 0 and a continuous function f : R d → R with compact support. We introduce the function f σ dened as: for any

  .36) which completes the proof. Proof of Theorem 7.5. We start by upper bounding the distance between two regularized measures. Denote by supp(ζ) the support of the function ζ. Let ϕ : R → R * + be a smooth and even function verifying supp(ϕ) ⊂ [-1, 1] and R ϕ(t)dLeb 1 (t) = 1. Dene ϕ λ (x) = λ -d ϕ( x /λ)/A(S d-1 ) , with A(S d-1 ) denoting the surface area of the d-dimensional unit sphere,i.e. A(S d-1 ) = 2π d/2 /Γ(d/2) , where Γ is the gamma function. Denote by F[f ] the Fourier transform of any function f dened on R s (s ≥ 1), given for any

  F ⊂ M b (R d ) as the unit ball of the RKHS whose reproducing kernel k satises k -k is positive denite. Then, for any p ∈ [1, ∞) and µ, ν ∈ P(R d ), SMMD p (µ, ν; F) ≤ MMD(µ, ν; F) , (7.70) where MMD(•, • ; F ) and SMMD p (•, • ; F ) respectively denote the MMD and the Sliced-MMD of order p in the RKHS whose unit ball is F .

  using (7.84) and (7.85), we obtain: for any n ∈ N, v 1 , . . . , v n ∈ R and x 1 , . . . , x n ∈ R d , j {k(x i , x j ) -k(x i , x j )} j e -x i -x j 2 /h M d -

∆E

  7.7. Let p ∈ [1, ∞) and µ, ν in P(R d ) with respective empirical measures μn , νn . By using the denition of S∆ p , the triangle inequality and the assumption on the sample complexity of ∆ p , we haveE S∆ p p (µ, ν) -S∆ p p (μ n , νn ) = E S d-1 ∆ p (θ µ, θ ν) -∆ p (θ μn , θ νn ) dσ(θ) p (θ µ, θ ν) -∆ p (θ μn , θ νn ) dσ(θ) ∆ p (θ µ, θ ν) -∆ p (θ μn , θ νn ) dσ(θ) of Theorem 7.8. Let p ∈ [1, ∞) and µ ∈ P(R d ) with corresponding empirical measure μn . By using the denition of S∆ p , the triangle inequality and the assumed convergence rate of empirical measures in ∆ p , we obtain the convergence rate in S∆ p as follows E S∆ p p (μ n , µ) = E S d-1∆ p (θ μn , θ µ)dσ(θ)

  expectation of (7.99) with respect to μn , νn , we obtainE |S∆ p (µ, ν) -S∆ p (μ n , νn )| ≤ E |S∆ p (μ n , µ)| + E |S∆ p (ν n , ν)| (7.100) ≤ E S∆ p p (μ n , µ) 1/p + E S∆ p p (ν n , ν) p, n) 1/p + α(p, n) 1/p = 2α(p, n)1/p , (7.102) where (7.101) results from applying Hölder's inequality on S d-1 if p > 1, and (7.102) follows from the convergence rate result in (7.98).

  ν) = S d-1 ∆ p (θ µ, θ ν)dσ(θ)by denition, the quantity given by, S d-1

  µ , ν ) ≤ C p (µ , ν ) .

  (µ k ) k∈N in P(R) and µ ∈ P(R) such that lim k→∞ C p (µ k , µ ) = 0 .

  any µ, ν ∈ P(R d ) with support in K, and θ ∈ S d-1 , C p (θ ν, θ µ) = R |F ν (t) -F µ (t)| p dt =

  K θ |F ν (t) -F µ (t)| p dt ≤ 2 p diam(K θ ) ≤ 2 p diam(K) .(7.117) By (7.116) and the dominated convergence theorem, lim k→∞ SC p (µ k , µ) = 0. Corollary 7.23. Let p ∈ [1, ∞) and F ⊂ M b (R).

E

  [|SW p (µ, ν) -SW p (μ n , νn )|]

  7.15. Let p ∈ [1, ∞) and ε ≥ 0. We use the reformulation of W p,ε as the maximum of an expectation, as given in[Genevay et al., 2016, Proposition 2.1],

E

  in the supplementary document]. We obtainSW p p,ε (µ, ν) = S d-1 E θ µ⊗θ ν φ ε ũ θ ( X), ṽ θ ( Ỹ ), X, Ỹ µ⊗ν φ ε ũ θ • θ (X), ṽ θ • θ (Y ), X, Y p dσ(θ) .

  w ∈ C(R) and θ ∈ S d-1 , w • θ ∈ C(R d ), we can bound (7.127) as follows SW p p,ε (µ, ν) ≤ S d-1 max u,v∈C(R d ) E µ⊗ν φ ε u(X), v(Y ),

  p ∈ [1, ∞), ε ≥ 0 and X ⊂ R compact. Consider µ , ν ∈ P( X) with respective empirical distributions μ n , ν n . We rst express the regularized OT cost as the maximum of an expectation[Genevay et al., 2016, Proposition 2.1] W p,ε (µ , ν ) = max ũ,ṽ∈C(R)

  ε (µ , ν ) -W p,ε (μ n , ν ) (7.138)

  d(θ µθ μn )(x) = diam( X)W 1 (µ , μ n ) ,(7.146)

  triangle inequality and (7.149), we obtain the nal resultW p,ε (μ n , ν n ) -W p,ε (µ , ν ) (7.150) ≤ W p,ε (µ , ν ) -W p,ε (μ n , ν ) + W p,ε (μ n , ν ) -W p,ε (μ n , ν n ) (7.151)

Figure 7 . 5 :

 75 Figure 7.5: Illustration of Corollary 7.14: Wasserstein and Sliced-Wasserstein distances of order 2 between two sets of n samples generated from N (0, I d ) vs. n, for dierent d, on log-log scale. SW 2 is approximated with L random projections, L ∈ {1, 10, 1000}.Results are averaged over 100 runs, and the shaded areas correspond to the 10th-90th percentiles. Figure7.2b shows the results for L = 100.

Titre:

  La distance de Sliced-Wasserstein pour l'Apprentissage Automatique à Grande Echelle : Th éorie, M éthodologie et Extensions Mots cl és : Apprentissage Automatique, Transport Optimal, Mod élisation G én érative R ésum é : De nombreuses m éthodes d'inf érence statistique et de mod élisation g én érative ont recours à une divergence pour pouvoir comparer de fac ¸on pertinente deux distributions de probabilit é. La distance de Wasserstein, qui d écoule du transport optimal, est un choix int éressant, mais souffre de limites computationnelle et statistique à grande échelle. Plusieurs alternatives ont alors ét é propos ées, notamment la distance de Sliced-Wasserstein (SW), une m étrique de plus en plus utilis ée en pratique en raison de ses avantages computationnels. Cependant, peu de travaux ont analys é ses propri ét és th éoriques. Cette th èse examine plus en profondeur l'utilisation de SW pour des probl èmes modernes de statistique et d'apprentissage automatique, avec un double objectif : 1) apporter de nouvelles connaissances th éoriques permettant une compr éhension approfondie des algorithmes bas és sur SW, et 2) concevoir de nouveaux outils inspir és de SW afin d'am éliorer son application et sa scalabilit é. Nous prouvons d'abord un ensemble de propri ét és asymptotiques sur les estimateurs obtenus en minimisant SW, ainsi qu'un th éor ème central limite dont le taux de convergence est ind épendant de la dimension. Nous d éveloppons également une nouvelle technique d'inf érence bas ée sur SW qui n'utilise pas la vraisemblance, offre des garanties th éoriques et s'adapte bien à la taille et à la dimension des donn ées. Etant donn é que SW est couramment estim ée par une simple m éthode de Monte Carlo, nous proposons ensuite deux approches pour att énuer les inefficacit és dues à l'erreur d'approximation : d'une part, nous étendons la d éfinition de SW pour introduire les distances de Sliced-Wasserstein g én éralis ées, et illustrons leurs avantages sur des applications de mod élisation g én érative ; d'autre part, nous tirons parti des r ésultats de concentration de la mesure pour formuler une nouvelle approximation d éterministe de SW, qui est plus efficace à calculer que la technique de Monte Carlo et pr ésente des garanties non asymptotiques sous une condition de d épendance faible. Enfin, nous d éfinissons la classe g én érale de divergences "sliced" et étudions leurs propri ét és topologiques et statistiques ; en particulier, nous prouvons que l'erreur d'approximation de toute divergence sliced par des échantillons ne d épend pas de la dimension du probl ème. Title : Sliced-Wasserstein Distance for Large-Scale Machine Learning: Theory, Methodology and Extensions Keywords : Machine Learning, Optimal Transport, Generative Modeling

  Figure 3.2: Min. SW estimation on Gaussians in R 10 . Figure 3.2a and Figure 3.2b show the mean squared error between (m , σ 2 ) = (0, 1) and MSWE ( mn , σ2n ) (resp. MESWE ( mn,n , σ2 n,n )) for n from 10 to 10 000, illustrating Theorems 3.2 and 3.3. Figure3.2c shows the error between ( mn , σ2

	0.15					0.15
	0.05 0.10 -m 2 mn 2					0.05 0.10 -m 2 mn,n 2					0.0002 0.0004 0.0006 -mn * 2 mn * ,m 2
	0.00	10 1	10 2	10 3	10 4	0.00	10 1	10 2	10 3	10 4	0.0000	10 1	10 2	10 3	10 4
			number of observations n				number of observations n			number of generated samples m
	0.100										0.008
	0.025 0.050 0.075 -σ 2 2 σ 2 n 2					0.1 -σ 2 2 σ 2 n,n 2 0.2					0.002 0.004 -σ 2 σ 2 n * ,m n 0.006 * 2 2
	0.000					0.0					0.000
		10 1	10 2	10 3	10 4		10 1	10 2	10 3	10 4		10 1	10 2	10 3	10 4
			number of observations n				number of observations n			number of generated samples m
		(a) MSWE vs. n		(b) MESWE vs. n = m	(c) MESWE vs. m

n ) and ( mn,m , σ2 n,m ) for n = 2000 observations and m from 10 to 10 000, to illustrate Theorem 3.4. Results are averaged over 100 runs, the shaded areas represent the standard deviation.

  Then, the map (ν, µ) → E [SW p (ν, μm )] is lower semi-continuous on P p (Y) × P p (Y) endowed with the topology of weak convergence.Proof. We consider two sequences (µ k ) k∈N and (ν k ) k∈N of probability measures in Y, such that (µ k ) k∈N w -→ µ and (ν k ) k∈N w -→ ν, and we x m ∈ N * . Xm ) dened on Ω such that for any k ∈ N and i ∈ {1, . . . , m}, Xi k has distribution µ k , Xi has distribution µ and ( X1 k , . . . , Xm k ) k∈N * converges to ( X1 , . . . , Xm ), P-almost surely. We then show that the sequence of (random) empirical distributions (μ k,m ) P metrizes the weak convergence by[Billingsley, 1999, Theorem 6.8], we only need to show that lim k→+∞ d P (ν k,m , νm ) = 0. More precisely, since for any probability measure ζ 1 and ζ 2 ,

	By Skorokhod's representation theorem, there exists a probability space ( Ω, F, P), a sequence of random variables ( X1
	k∈N dened i=1 δ Xi , P-almost surely. , weakly converges to μm = (1/m) m Note that it is sucient to show that for any deterministic sequence (x 1 by μk,m = (1/m) m i=1 δ Xi k k , . . . , x m k ) k∈N

(Lower semi-continuity of ESW p ). Let p ∈ [1, ∞) and m ∈ N * . Denote for any µ ∈ P p (Y), μm = (1/m) m i=1 δ Z i , where Z 1:m are i.i.d. samples from µ. k , . . . , Xm k ) k∈N and a random variable ( X1 , . . . , * which converges to (x 1 , . . . , x m ), i.e. lim k→+∞ max i∈{1,...,m} ρ(x i k , x i ) = 0, then the sequence of empirical distributions (ν k,m ) k∈N dened by νk,m = (1/m) m i=1 δ x i k , weakly converges to νm = (1/m) m i=1 δ x i . Since the Lévy-Prokhorov metric d

  1:d ] is expected to increase linearly with d 1/2 at least, so are m 2 (µ X d ) and β 2 (µ X d ) as functions of d. As a consequence, Ξ d (µ X d ) cannot be shown to converge to 0 as d → ∞ in this setting, but only to be bounded. However, if the data are centered, the norm of the mean is zero, thus Ξ d (µ X d ) might be decreasing. Therefore, we derive a convenient formula to compute SW 2 (µ d , ν d ) from SW 2 (μ d , νd ) where for any ξ d ∈ P 2 (R d ), ξd is the centered version of ξ d , i.e. the push-forward measure of ξ d by x → xm ξ d with m ξ d = R d y dξ d (y). This result is the last ingredient to formulate our approximation of SW. Proposition 6.3. Let µ d , ν d ∈ P 2 (R d ) with respective means m µ d , m ν d . Then, the Sliced-Wasserstein distance of order 2 can be decomposed as

  Table 6.1: Results obtained after training generative models on MNIST and CelebA, averaged over 5 runs. FID are reported with their standard deviation (the lower FID, the better). T SW denotes the average time per epoch for approximating SW. T tot is the average running time per epoch.reg-det-SWG is faster than reg-SWG, which is consistent with the fact that our approximation method is faster than Monte Carlo on high-dimensional settings. To further illustrate this point, we reported the average time spent in computing the generative loss per epoch, i.e. SW 2,10 4 for SWG and reg-SWG, and SW 2 for reg-det-SWG: see column T SW in Table6.1. On GPU, reg-det-SWG is at least 15 times faster than SWG and reg-SWG on MNIST, and 6 times faster on CelebA.

	Dataset Model	FID	T SW (s/epoch)	T tot (s/epoch)
				GPU CPU	GPU	CPU
	MNIST CelebA	SWG Reg-SWG Reg-det-SWG 15.72 ± 0.57 0.07 22.41 ± 2.34 1.3 15.53 ± 0.88 1.1 SWG 31.04 ± 2.78 10.1 Reg-SWG 24.14 ± 0.48 10.0 Reg-det-SWG 23.65 ± 0.93 1.3	1.4 ×10 2 4.5 1.1 ×10 2 6.5 0.2 5.3 2.7 ×10 3 3.9 ×10 2 1.6 ×10 4 2.7 ×10 2 3.0 ×10 2 1.5 ×10 2 2.7 ×10 3 4.4 ×10 2 2.0 ×10 4 2.6 4.2 ×10 2 1.7 ×10 4

  Since X 1:d has nite second and fourth-order moments,

	max 1≤j≤d	Var[X j ], max 1≤j≤d	Var[X 2 j ] < ∞ ,
	and we get

  Proof of Proposition 7.1. (i) The fact that S∆ p is non-negative (or symmetric) if ∆ is, immediately follows from the denition of S∆ p (7.1).

		10 1				Sinkhorn
					10 3	Sliced-Sinkhorn
	divergence	10 -2 10 -1 10 0	Sinkhorn	execution time (s)	10 0 10 1 10 2
			Sliced-Sink.
		10 -3	Wasserstein		10 -1
			Sliced-Wass.
		10 1	10 2	10 3	10 1	10 2	10 3
			number of samples			number of samples
				Results on MNIST
						Sinkhorn
		10 1				Sliced-Sinkhorn
					10 2
	divergence	10 -2 10 -1 10 0	Sinkhorn	execution time (s)	10 0 10 1
			Sliced-Sink.
			Wasserstein
		10 -3	Sliced-Wass.		10 -1
		10 1	10 2	10 3	10 1	10 2	10 3
			number of samples			number of samples
				Results on CIFAR-10
	Figure 7.4: (Sliced-)Wasserstein and (Sliced-)Sinkhorn (ε = 1) between two random
	subsets of n samples of real datasets (top: MNIST, bottom: CIFAR-10), for dierent
	values of n. Results are averaged over 10 runs, and the shaded areas correspond to the
	10th-90th percentiles. All plots have a log-log scale.
	7.6	Appendix: Postponed proofs and Additional Empirical
			Results
	7.6.1 Proofs for Section 7.2.2

An interesting question for future work is then to understand precisely what geometrical features are well preserved by the slicing operation.

  dx .

	Using (7.14), (7.17), (7.19) and Lebesgue's Dominated Convergence Theorem, we obtain:
	(7.19)

  Proof of Corollary 7.10. Let p ∈ [1, ∞), µ, ν ∈ P(R d ) and the respective empirical distributions μn , νn . By the triangle inequality, (μ n , νn )-S∆ p p (µ, ν)| ≤ | S∆ p p,L (μ n , νn ) -S∆ p p (μ n , νn )| + |S∆ p p (μ n , νn ) -S∆ p p (µ, ν)| . (μ n , νn ) -S∆ p p (μ n , νn )| X 1:n , Y 1:n ] (μ n , νn ) -S∆ p p (μ n , νn )| X 1:n , Y 1:n ∆ p (θ μn , θ νn ) -S∆ p p (μ n , νn ) 2 dσ(θ) E[(∆ p (θ μn , θ νn ) -S∆ p p (μ n , νn )) 2 ]dσ(θ) ,which appears in Corollary 7.10, is equal to E[Var{∆ p (θ μn , θ νn )|X 1:n , Y 1:n }],

	p | S∆ p,L Therefore, by linearity of expectation, we have
	E | S∆ p p,L (μ n , νn ) -S∆ p p (µ, ν)| p p,L (7.106) (7.105) ≤ E E[| S∆ + E |S∆ p p (μ n , νn ) -S∆ p p (µ, ν)| . (7.107)
	We bound (7.106): by Theorem 7.9, we have
	p p,L (7.108) E | S∆ S d-1 1/2 ≤ L -1/2 . (7.109)
	By taking the expectation then using Jensen's inequality, we get
	E E | S∆
			1/2
			(7.111)
	≤ L -1/2 E 1/2	S d-1	∆ (7.113)
	Combining (7.112) and (7.113) in (7.106) and (7.107) completes the proof.
	Remark 7.22. Note that by Fubini's theorem,
	S d-1

p p,L (μ n , νn ) -S∆ p p (μ n , νn )| X 1:n , Y 1:n (7.110) ≤ L -1/2 E S d-1 ∆ p (θ μn , θ νn ) -S∆ p p (μ n , νn ) 2 dσ(θ) p (θ μn , θ νn ) -S∆ p p (μ n , νn )

2 dσ(θ) . (7.112) Next, we bound (7.107): by the sample complexity assumption for ∆ p and Theorem 7.7, we have E |S∆ p p (μ n , νn ) -S∆ p p (µ, ν)| ≤ β(p, n) .

Sinkhorn, d = 2 Sliced-Sinkhorn, d = 2 Sinkhorn, d = 10 Sliced-Sinkhorn, d = 10 Sinkhorn, d = 20 Sliced-Sinkhorn, d = 20 Sinkhorn, d = 50 Sliced-Sinkhorn, d = 50 Sinkhorn, d = 100 Sliced-Sinkhorn, d = 100

Remerciements

En premier lieu, je souhaite remercier mes directeurs de thèse, Roland, Alain et Umut.

Je mesure pleinement ma chance d'avoir pu travailler avec chacun d'entre vous ces trois

Projections

This chapter is based on [START_REF] Nadjahi | Fast Approximation of the Sliced-Wasserstein Distance Using Concentration of Random Projections[END_REF].

The Sliced-Wasserstein distance is being increasingly used in machine learning applications as an alternative to the Wasserstein distance and oers signicant computational and statistical benets. Since it is dened as an expectation over random projections, SW is commonly approximated by Monte Carlo. We adopt a new perspective to approximate SW by making use of the concentration of measure phenomenon: under mild assumptions, one-dimensional projections of a high-dimensional random vector are approximately Gaussian. Based on this observation, we develop a simple deterministic approximation for SW. Our method does not require sampling a number of random projections, and is therefore both accurate and easy to use compared to the usual Monte Carlo approximation. We derive nonasymptotical guarantees for our approach, and show that the approximation error goes to zero as the dimension increases, under a weak dependence condition on the data distribution. We validate our theoretical ndings on synthetic datasets, and illustrate the proposed approximation on a generative modeling problem.

6.1 6.6.8 Experimental details for image generation Architecture. For each model (SWG, reg-SWG or reg-det-SWG), we used the architectures described in [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF]: the Conv & Deconv generator and discriminator in [Deshpande et al., 2018, Section D] for MNIST, and DCGAN [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF] with layernorm for both the generator and discriminator for CelebA.

Data preprocessing. For MNIST, we do not apply any specic preprocessing. For CelebA, each image is cropped at the center and resized to 140 × 140 (using the notation width × height, both in pixels), then resized to 64 × 64.

Optimization. For each model, we used the same optimization routine as in [START_REF] Deshpande | Generative modeling using the sliced Wasserstein distance[END_REF]: one training iteration consists in performing one update for the generator then one update for the discriminator, both with the default setting of Adam [START_REF] Kingma | A method for stochastic optimization[END_REF] (i.e. β 1 = 0.9, β 2 = 0.999, ε = 10 -8 ). The values of other important hyperparameters are given in Regularization parameters. For reg-SWG and reg-det-SWG, we tuned the regularization coecients (λ 1 , λ 2 ) via cross-validation: we trained the models for λ 1 ∈ {10 -3 , 10 -2 , 10 -1 , 1} and λ 2 ∈ {0, 10 -3 , 10 -2 , 10 -1 , 1}, and selected the tuple that minimizes the average FID over 5 runs.

Computing infrastructure. The FID and computation times on GPU reported in Table 6.1 (columns FID, T SW , GPU and T tot , GPU) were obtained by training each model on a computer cluster equipped with 3 GPUs (NVIDIA Tesla V100-PCIE-32GB and 2 × NVIDIA Tesla V100-PCIE-16GB) for CelebA, and with 1 GPU (NVIDIA GP100GL, Tesla P100 PCIe 16GB) for MNIST.

To obtain the computation times on CPU (Table 6.1, columns T SW , CPU and T tot , CPU), we used a workstation equipped with 24 × Intel Xeon CPU E5-2620 v3 @ 2.40GHz.

Chapter 7

Statistical and Topological

Properties of Sliced Probability

Divergences

This chapter is based on [Nadjahi et al., 2020b].

The idea of slicing divergences has been proven to be successful when comparing two probability measures in various machine learning applications including generative modeling, and consists in computing the expected value of a base divergence between one-dimensional random projections of the two measures. However, the topological, statistical, and computational consequences of this technique have not yet been wellestablished.

In this chapter, we aim at bridging this gap and derive various theoretical properties of sliced probability divergences. First, we show that slicing preserves the metric axioms and the weak continuity of the divergence, implying that the sliced divergence will share similar topological properties. We then precise the results in the case where the base divergence belongs to the class of integral probability metrics. On the other hand, we establish that, under mild conditions, the sample complexity of a sliced divergence does not depend on the problem dimension. We nally apply our general results to several base divergences, and illustrate our theory on both synthetic and real data experiments.

Introduction

Most inference methods in implicit generative modeling rely on the use of a particular divergence in order to be able to discriminate probability distributions. Recent advances in this eld have illustrated that the choice of this divergence is of crucial importance since it can lead to very dierent practical and theoretical properties (Chapter 1). In this context, sliced probability divergences, such as Sliced-Wasserstein or Sliced-Cramér [Kolouri et al., 2020a], have become increasingly popular.

This slicing strategy has been essentially motivated by two main purposes. The rst purpose is that some probability divergences are only dened to compare measures supported on one-dimensional spaces (e.g., Cramér distance, Cramér [1928]); hence, the slicing operation allows the use of such divergences to multivariate distributions [START_REF] Knop | Cramer-Wold Auto-Encoder[END_REF], Kolouri et al., 2020a]. The second purpose arises when the computational complexity of a divergence becomes excessive when comparing measures on 121 (iii) If ∆ is a metric, then S∆ p is a metric.

Next, we extend our result stated in Theorem 3.1, which showed that the convergence under SW implies the weak convergence of probability measures: we prove that this property holds for the general class of SPDs, but also that the converse implication is true, provided that ∆ is weakly continuous. We refer to Section 2.1 for the denitions of convergence under a probability divergence, weak convergence of probability measures, and weak continuity.

Theorem 7.2. Let p ∈ [1, ∞) and ∆ be a non-negative base divergence.

(i) If the convergence under ∆ implies the weak convergence in P(R), then the convergence under S∆ p implies the weak convergence in P(R d ).

(ii) If ∆ is bounded and the weak convergence in P(R) implies the convergence under ∆, then the weak convergence in P(R d ) implies the convergence under S∆ p .

We now focus on IPMs and formally dene Sliced-IPMs, before providing ner topological results. We introduce the following notations. Let X be a closed and measurable subset of R d . M(X) denotes the set of real-valued measurable functions on X, M b (X) is the set of bounded functions of M(X), and

The Sliced Integral Probability Metric of order p associated with F, denoted by Sγ F,p , is dened for any µ, ν ∈ P(R d ) as

Since γ F is a pseudo-metric, Sγ F,p is a pseudo-metric as well by Proposition 7.1. We now identify some regularity conditions on the function classes F and F such that we are able to show that Sliced-IPMs can be bounded above and below by IPMs. Note that for the next results, we will assume that the supremum in (2.1) is attained. This property is for example veried for W 1 and MMD, by [START_REF] Villani | Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften[END_REF] and [START_REF] Gretton | A Kernel Two-Sample Test[END_REF] respectively.

Theorem

and assume that

Then, for any p ∈ [1, ∞) and µ, ν ∈ P(R d ), Sγ F,p (µ, ν) ≤ γ F (µ, ν).

Theorem 7.4 states that Sγ F,p induces a weaker topology, which is computationally benecial as argued in [START_REF] Arjovsky | Wasserstein Generative Adversarial Networks[END_REF], but also indicates that Sγ F,p comes with less discriminative power, which can be restrictive for hypothesis testing applications such as in [START_REF] Gretton | A Kernel Two-Sample Test[END_REF].

We now derive a lower-bound on compact domains.

Theorem 7.5. Let µ, ν ∈ P(R d ), with support included in

and suppose that there exists L ≥ 0 such that for any g ∈ G, g is L-Lipschitz continuous. Consider a class of functions G satisfying

(i) Linear kernels: k(t i , t j ) = t i t j for t i , t j ∈ R, and k(x i , x j ) = x i x j /d for x i , x j ∈ R and d ≥ d.

(ii) Radial basis function (RBF) kernels: let h ≥ 0, k(t i , t j ) = e -|t i -t j | 2 /h for t i , t j ∈ R, and k(x i , x j ) = e -x i -x j 2 /h for x i , x j ∈ R d .

Proof. Dene F as the unit ball of an RKHS whose reproducing kernel is denoted by k.

Then, any f ∈ F satises

where n ∈ N * , α 1 , . . . , α n ∈ R and t 1 , . . . , t n ∈ R.

The integration of (7.72) over S d-1 gives us

Since k is positive denite, so is k. By the Moore-Aronszajn theorem, there exists a unique RKHS with reproducing kernel k. Therefore, (7.74) means that f is in the unit ball of the RKHS associated with k.

Additionally, consider a positive denite kernel k : R d × R d → R such that k -k is positive denite on R d . In other words, the following holds for any n ∈ N, v 1 , . . . , v n ∈ R and x 1 , . . . ,

Then, by (7.74), we obtain

Therefore, any f dened as f = f • θ with f ∈ F and θ ∈ S d-1 is in the unit ball of the RKHS associated with k, which we denote by F. By using Theorem 7.4 and the denition of MMD, we obtain the desired result: for any p ∈ [1, ∞) and µ, ν ∈ P(R d ), SMMD p (µ, ν; F) ≤ MMD(µ, ν; F) . Next, we show that this result holds for two popular choices of kernels. First, we choose k as the linear kernel: k(t i , t j ) = t i t j for t i , t j ∈ R. Dene k as a rescaled version Sinkhorn's algorithm refers to an iterative procedure which operates on empirical distributions as follows: consider a cost matrix C between two sets of n samples, and dene the matrix K such that, for 1 ≤ i, j ≤ n,

and initialize b (0) = 1 ∈ R n ; then, compute for > 1,

where ./ stands for the entry-wise division. This denes a sequence γ

which converges to a solution of the regularized OT problem (2.16) at a linear rate. The convergence rate of Sinkhorn's algorithm is recalled in Theorem 7.28. For an extended discussion on this result, we refer to [Peyré and Cuturi, 2019, Section 4.2].

Theorem 7.28 [START_REF] Franklin | On the scaling of multidimensional matrices[END_REF]). The iterates a ( ) and b ( ) of Sinkhorn's algorithm converge linearly for the Hilbert metric at a rate 1-tanh(τ (K)/4), with τ (K) = log max i,j,i ,j

. In particular, for the squared-norm cost, i.e. K ij = exp(-x i -

Proof of Proposition 7.17. For i, j ∈ {1, . . . , n}, f i,j : θ ∈ S d-1 → 1 R θ, x ix j is 1-Lipschitz and has median 0 for θ uniformly distributed on the unit sphere. Thus, by concentration of measure on the sphere [Wainwright, 2019, Example 3.12], it holds for ε > 0,

Taking a union bound over the n(n -1) pairs of indices and setting τ = (Rε) 2 , it follows

Hence, for any δ > 0, it holds with probability 1 -

). This argument was suggested to us by an anonymous reviewer at the NeurIPS 2020 conference.

Additional empirical results

In this section, we provide additional results obtained for the synthetical experiments illustrating the sample complexity of Sliced-Wasserstein and Sliced-Sinkhorn divergences:

we produce gures analogously to Figures 7.2b, 7.3a and 7.3b, with dierent hyperparameter values.

Chapter 8 Conclusion

Summary

This thesis builds on the recent success of optimal transport tools for machine learning models, and is specically focused on the increasingly popular Sliced-Wasserstein distance. Our general purpose was to provide a more comprehensive understanding of this metric by investigating its theoretical and practical implications, especially when applied for parameter inference in generative models. Some of our results enabled us to identify the limitations caused by the estimation of SW and overcome them by designing novel methodologies. We summarize our main contributions below, accordingly to the objectives listed in Section 1.4 (page 22).

Theoretical properties of SW. This paragraph provides answers to objective 1. We established several theoretical results that aim at shedding light on the empirical behavior of SW as reported in prior work. Specically, in Chapter 3, we studied the estimators obtained by minimizing SW over a parametric space and proved some of their asymptotic properties. Besides ensuring theoretical consistency to existing SW-based generative models, our results allowed us to derive new topological properties, namely the lower semi-continuity of SW and the fact that convergence under SW implies weak convergence.

Then, we demonstrate that SW is able to mitigate the statistical limitations of the Wasserstein distance in high-dimensional settings. The central limit theorem proved in Chapter 3 and characterizing the asymptotic distribution of minimum SW estimators exhibits a convergence rate of √ n, where n is the number of observations. This dimensionfree rate as well as the empirical comparison in Chapter 4 (Figure 4.1, page 74) are rst evidence that SW oers important statistical benets over the Wasserstein distance. This is further conrmed by the sample complexity of SW derived in Chapter 7, which does not depend on the dimension, as opposed to the sample complexity of the Wasserstein distance which can grow exponentially in dimension.

Finally, our theoretical results in Chapter 7 also explain why the statistical eciency is actually balanced with the fact that SW is dened as an integral over S d-1 . We elaborate on this aspect later, as an answer to objective 3.

New methodology for approximate inference. This paragraph provides answers to objective 2. We expanded the applicability of SW by developing a likelihood-free approximation inference technique based on this metric and called SW-ABC (Chapter 4).

We showed that SW-ABC comes with convergence guarantees under dierent asymptotic regimes and oers a superior empirical performance as compared to existing ABC 155 which we developed in Chapter 6 to address this aspect, provides important computational advantages over Monte Carlo, but still leaves room for improvement. First, the convergence rate of our approximate to the true SW seems slow according to our nonasymptotical guarantees in Section 6.3.3 (page 102), while it is reasonably fast in our experiments. To bridge this gap between theory and practice, we can study if our proofs and the ones in [START_REF] Reeves | Conditional central limit theorems for Gaussian projections[END_REF] can be rened when assuming additional structure on the distributions, e.g. sub-Gaussian and sub-exponential: this will help identify the settings under which our current bounds are tight or can be improved.

Furthermore, since our methodology in Chapter 6 applies to the Sliced-Wasserstein distance only, another crucial question is how to eectively approximate the larger class of sliced divergences introduced in Chapter 7. One idea would be to resort to alternative Monte Carlo algorithms, such as sequential Monte Carlo samplers [Chopin, 2002[START_REF] Del Moral | Sequential Monte Carlo Samplers[END_REF], in order to develop a technique that improves the convergence rate or the variance of the traditional Monte Carlo estimates.

In-depth analysis of GSW. Our work on Generalized Sliced-Wasserstein distances in Chapter 5 have since then inspired other follow-up studies, including [START_REF] Nguyen | Distributional Sliced-Wasserstein and Applications to Generative Modeling[END_REF][START_REF] Chen | N. Chopin. A sequential particle lter method for static models[END_REF][START_REF] Naderializadeh | Set Representation Learning with Generalized Sliced-Wasserstein Embeddings[END_REF], and pave the way for a deeper theoretical investigation. Indeed, our experiments showed that the choice of the dening function g is data-dependent and highly impacts on the performance of the associated GSW, and the question when and why do certain choices of g perform well in practice?

has not yet been elucidated. To address this matter, we can study to what extent the theoretical analysis in [START_REF] Nadjahi | Asymptotic Guarantees for Learning Generative Models with the Sliced-Wasserstein Distance[END_REF][Nadjahi et al., , 2020b] can be generalized to GSW: the topological and statistical properties of GSW might highly depend on the types of onedimensional representations used. This requires an analysis of the generalized Radon transform and would help identifying the advantages of some representations against others.

On the other hand, we can explore whether a connection can be established between kernel functions and GSW (see [Kolouri et al., 2020b] as our preliminary study), and between SW for manifolds and GSW. One hypothesis would be that, with appropriate representations, GSW might be equivalent to dening SW on manifolds. To verify this assumption, a starting point would be to formulate a denition of SW on compact Riemannian manifolds, for example by leveraging existing characterizations of the Wasserstein distance on such spaces [START_REF] Rabin | Transportation Distances on the Circle[END_REF]. Apart from understanding better GSW, such a distance would be very useful in practice: the Sliced-Wasserstein distance has only been dened for data living on R d , although there are various elds where the training data are supported on Riemannian manifolds, e.g., in bioinformatics [START_REF] Mardia | Directional Statistics in Protein Bioinformatics[END_REF], neurology [START_REF] Kaufman | Spline-based non-parametric regression for periodic functions and its application to directional tuning of neurons[END_REF], life sciences [Ameijeiras-Alonso et al., 2018] and text mining [START_REF] Banerjee | Clustering on the Unit Hypersphere using von Mises-Fisher Distributions[END_REF].