Adam M Oberman

Michèle Sebag

Merci À Jérémy

Axel Stéphanie

Clément Et

Bharath

Devis Jean-Christophe Sylvain

Kilian Fatras

Younes Zine

Rémi Flamary

Rémi Gribonval

Nicolas Courty

8×8×128 Gatedconv

Relu Gatedconv

Relu Gateddeconv

Deep learning and optimal transport : learning

Keywords: Entropic regularization: formulation and practical interest 2, 4, Optimal Transport extensions

Ce manuscrit contient tous les travaux de ma thèse qui ont été réalisés de Novembre 2018 à Février 2021. Avant de clôturer ce bien beau chapitre, je tiens à remercier toutes les personnes qui m'ont permis de l'écrire.

Je tiens tout d'abord à remercier mes directeurs de thèse, les professeurs Rémi Flamary et Nicolas Courty, pour les différentes directions de recherches qu'ils m'ont suggérés pendant ces trois années. J'aimerais aussi les remercier de m'avoir permis de travailler étroitement avec l'équipe Panama de l'IRISA et l'INRIA Rennes, cela m'a permis de m'épanouir professionnellement et personnellement pendant ces trois années.

I would like to thank Marco Cuturi and Cristian Wolf for their report and the time they took to read my PhD manuscript. I also want to thank

Résumé (Français)

La communauté d'intelligence artificielle a été capable de créer des algorithmes pour certaines tâches non triviales dont la performance est comparable ou supérieure aux performances humaines. Bien que non triviales, ces tâches peuvent être résolues avec la puissance de calcul des ordinateurs en suivant une liste formelle de règles. Par exemple en 1997, Deep Blue, un super calculateur capable de jouer aux échecs a battu le champion du monde d'échecs: le russe Garry Kasparov. Ce fut la première victoire d'un ordinateur face à un champion du monde d'échecs. Deep Blue était basé sur un algorithme de force brute qui explorait un nombre maximum de coups dans un temps donné et choisissait le meilleur d'entre eux. Cependant, nonobstant ces succès, certaines tâches qui sont naturelles pour les êtres humains, telles que la classification d'objets ou la compréhension de conversations, restent compliquées à résoudre numériquement. Ces problèmes reposent sur des concepts intuitifs et sont compliqués à formaliser. C'est pourquoi une méthode similaire à Deep Blue ne peut pas capturer toutes les complexités cachées. Ces différents concepts nous amènent à considérer une nouvelle approche pour résoudre ces tâches: l'apprentissage machine.

Les différences et forces des algorithmes d'apprentissage machine par rapport aux autres algorithmes sont leurs capacités à apprendre sans l'aide d'opérateurs humains. En faisant des erreurs sur des exemples d'entraînement, les algorithmes s'adaptent afin de se rapprocher de la réponse désirée. Ils sont ensuite capable de faire des prédictions sur des situations qu'ils n'ont pas vu lors de leur entraînement. Cela leur donne l'unique capacité d'effectuer des tâches qui sont naturelles pour des êtres humains. Les potentielles applications de l'apprentissage machine sont nombreuses et le domaine a déjà été appliqué aux voitures autonomes, à la détection de cancers ou à la traduction. Récemment, une classe spécifique d'algorithme d'apprentissage machine a poussé les performances encore plus loin. Cette classe d'algorithme est appelée apprentissage profond.

Les modèles d'apprentissage profond sont un cas particulier d'apprentissage machine. Les différences résident dans leurs constructions. Ce sont des réseaux de neurones artificiels qui sont composés de nombreuses couches, elles mêmes composées de nombreux neurones. Cela amène ces architectures à posséder un grand nombre de neurones ce qui augmente le nombre de calculs à effectuer afin de les entraîner. Leur entraînement n'a été possible qu'avec l'arrivée de cartes graphiques puissantes ce qui explique leur avènement récent. Cependant, le succès de l'apprentissage profond est aussi dû aux développements récents de domaines mathématiques, qui ont aidé les réseaux de neurones à résoudre les problèmes désirés. Cette thèse porte sur l'étude des différentes interactions de l'apprentissage profond avec l'un de ces domaines appelé transport optimal.

Apprentissage profond et transport optimal. Les premiers architectures de réseaux de neurones peuvent être tracées au milieu du vingtième siècle [Rosenblatt 1958] et les premières architectures de réseaux profonds, ainsi que leur entraînement, trente ans plus tard [Rumelhart 1986[START_REF] Lecun | [END_REF], Lecun 1998]. Bien que ces architectures ont été inventés il y a une trentaine d'années, les chercheurs ont fait face aux limites de calculs des ordinateurs à cause de la taille des réseaux. En 2012, sans se décourager face à ces limites, les chercheurs ont réussi à avoir de meilleurs résultats avec des modèles d'apprentissage profond qu'avec des algorithmes d'apprentissage machine sur un problème concret de classification. Cette année là, le réseau de neurones AlexNet [START_REF] Krizhevsky | [END_REF]], entraîné sur une tâche de classification avec des gradients stochastiques et des cartes graphiques, a atteint la plus haute performance jamais réalisé sur la compétition ImageNet [Deng 2009]. Ce premier succès retentissant sur un problème de classification, couplé avec l'amélioration des cartes graphiques, a amené la communauté scientifique à reconsidérer et à développer le domaine de l'apprentissage profond.

Depuis ce succès, l'apprentissage profond est devenu la classe d'algorithme la plus performante pour réaliser des décisions basées sur des données. Pour pousser les performances de l'apprentissage profond encore plus loin, de nombreux travaux de recherche ont été conduits afin de trouver de nouvelles architectures de réseaux de neurones. Par exemple en classification, l'architecture ResNet s'est montrée particulièrement performante afin de réduire certains problèmes d'entraînement [START_REF] He | [END_REF]]. En segmentation, les chercheurs ont développé l'architecture U-net [Ronneberger 2015]. Une autre approche qui a permis d'améliorer significativement les performances des modèles d'apprentissage est l'utilisation des minis lots couplée avec un algorithme d'optimisation. En effet, les réseaux de neurones sont entraînés avec des algorithmes d'optimisation de premier ordre comme par exemple la descente de gradient, malgré le fait que ces réseaux ne soient pas convexes. Dans un régime de données massives, utiliser la descente de gradient signifierait le calcul d'un gradient à chaque itération de la descente, ce qui coûte cher à calculer. C'est pourquoi afin d'accélérer le calcul d'une itération, la communauté s'est tournée vers l'optimisation stochastique et l'algorithme de gradient stochastique (SGD). Le prix à payer d'utiliser SGD est une convergence plus lente nécessitant un plus grand nombre d'itérations de l'algorithme. Cependant, son coût global reste inférieur car ses itérations sont moins coûteuses. Plusieurs variantes de SGD sont apparues telles qu'Adam [Kingma 2015] ou RMSprop [Tieleman 2012a]. Il est donc naturel de s'assurer de la convergence de ces algorithmes stochastiques vers une bonne solution lorsqu'une nouvelle fonction de coût est utilisée avec les réseaux de neurones.

Suivant ces développements fondamentaux, l'apprentissage profond est aujourd'hui état de l'art sur bon nombre de domaines à la fois en apprentissage supervisé et non supervisé, tels qu'en adaptation de domaine [START_REF] Ganin | [END_REF]], en segmentation [Ronneberger 2015] et en modèles génératifs [Brock 2019]. Son utilisation s'étend aussi à d'autres domaines comme la physique [Schütt 2017] ou la biologie [START_REF] Jumper | [END_REF]]. L'amélioration rapide de la compréhension de l'apprentissage profond est aussi possible grâce aux multiples ressources libres. Nous souhaiterions mentionner à ce titre les librairies PyTorch [START_REF] Paszke | [END_REF]] et TensorFlow [START_REF] Abadi | [END_REF]].

Cependant, il est important de mentionner que les réseaux de neurones profonds ne sont pas les seuls moteurs derrière les succès de l'apprentissage profond. Par exemple, les problèmes d'adaptation de domaine et les modèles génératifs nécessitent de comparer des ensembles de données entre eux. Sur ces problèmes, les réseaux de neurones transforment les données sur lesquelles ils sont appliqués en nouvelles données. Mathématiquement, cela correspond à créer une mesure image et donc à de nouveaux ensembles de données. Ainsi, résoudre ces problèmes nécessite une notion de distance entre des ensembles de données. Pour définir une telle distance, nous devons tout d'abord correctement représenter les données sur lesquelles nous travaillons. Données d'entraînement comme mesure empirique. Afin de représenter les données mathématiquement, nous choisissons le cadre des mesures de probabilité. Les mesures de probabilité empiriques prennent la forme d'une somme de Diracs α n = n i=1 a i δ xi , où ∀i ∈ [1, n], a i > 0 et n i=1 a i = 1. Cette représentation est appelée la représentation Lagrangienne et elle représente l'ensemble des données comme un seul objet mathématique. Comme nous supposerons que les données sont tirées de manière indépendante et identiquement distribuées, nous choisissons des poids uniformes pour la mesure α n , i.e., ∀i ∈ [1, n], a i = 1 n . Pour des applications spécifiques, comme dans le Chapitre 6, il est possible de considérer d'autres poids.

Ainsi, la distance mathématiques dont nous avons besoin est une distance entre mesures de probabilité. Il est donc possible de recourir aux divergences de Csiszar [Csiszar 1975]. Dans cette thèse, nous nous concentrons sur le transport optimal pour comparer ces distributions. Le transport optimal peut être vu comme une méthode générique pour créer une distance entre des ensembles de points à partir d'une distance entre deux points. Contrairement aux divergences précédemment mentionnées, il considère des aspects géométriques pour comparer les différentes distributions. C'est donc naturellement que le transport optimal a été utilisé en apprentissage profond. Plus surprenant, l'apprentissage profond a été utilisé afin d'estimer le transport optimal continu.

Transport optimal et apprentissage profond. L'origine du transport optimal remonte au 18ème siècle dans les travaux de Gaspard Monge [START_REF] Gaspard Monge | Mémoire sur la théorie des déblais et des remblais[END_REF]]. Bien que le transport optimal a trouvé de nombreuses applications en mathématiques théoriques et appliquées [Santambrogio 2015, Villani 2009], il a aussi rencontré un certain succès en apprentissage machine [START_REF] Peyré | [END_REF]]. Parmi ces succès, nous pouvons noter son utilisation en adaptation de domaine en transportant les données sources vers le domaine cible [Courty 2017a] ou bien en alignant les données plongées dans les réseaux de neurones [START_REF] Damodaran | [END_REF]]. De plus, nous pouvons aussi mentionner son utilisation comme fonction de coût pour des problèmes de classification multi-étiquettes, où le transport a été utilisé entre les prédictions des modèles d'apprentissage et les étiquettes des données [Frogner 2015]. Cependant, l'application la plus célèbre du transport optimal en apprentissage profond est son utilisation avec les réseaux génératifs adversaires [Goodfellow 2014b] où il a été utilisé en tant que fonction dans Wasserstein GAN [START_REF] Arjovsky | [END_REF]]. Nous terminons par les différentes applications en biologie [START_REF] Schiebinger | [END_REF]], en astrophysique [START_REF] Frisch | [END_REF]] ou en traitement du signal [START_REF] Kolouri | [END_REF]].

Afin de comprendre ses succès et d'améliorer les performances des méthodes basées sur le transport optimal, les chercheurs ont étudié plusieurs aspects du transport optimal. Le premier aspect est l'estimation statistique du transport car en apprentissage profond, les données utilisées vivent en grandes dimensions. Plusieurs recherches ont été faites pour étudier le taux de convergence en population. Le cas du transport optimal exact a été étudié dans [Dudley 1969, Weed 2019] et sa variante entropique dans [START_REF] Genevay | [END_REF], Mena 2019]. Concernant le calcul du transport optimal, celui-ci est bien trop élevé pour que le transport soit utilisé sur des jeux de données à large échelle. C'est pourquoi beaucoup de recherches ont été faites afin de réduire ce coût de calcul. Avec une régularisation entropique, le transport peut être efficacement calculé sur des cartes graphiques avec l'algorithme de Sinkhorn [Cuturi 2013]. De plus, la complexité algorithmique de l'algorithme de Sinkhorn a été étudiée dans [START_REF] Altschuler | [END_REF]]. D'autres stratégies basées sur une approximation du transport avec des minis lots ont été développées [START_REF] Genevay | [END_REF][START_REF] Damodaran | [END_REF][START_REF] Salimans | [END_REF]] mais un formalisme rigoureux était manquant.

Enfin, des variantes du transport optimal ont trouvé de nombreuses applications. Par exemple, la distance de Gromov-Wasserstein permet de comparer des mesures de probabilités qui vivent dans différents espaces métriques. Elle a été appliquée pour des modèles génératifs [Mémoli 2011[START_REF] Bunne | [END_REF]] ou sur des graphes [Vayer 2019a]. Des variantes avec des contraintes relaxées ont été utilisées afin de rendre le transport robuste aux données aberrantes [START_REF] Chapel | Partial Optimal Transport with Applications on Positive-Unlabeled Learning[END_REF][START_REF] Balaji | [END_REF].

Apprentissage profond et transport optimal. L'apprentissage profond peut-être utilisé pour approximer la formulation continue du transport optimal. Comme les mesures de probabilité sont inconnues dans beaucoup d'applications, nous nous reposons sur des mesures empiriques des mesures inconnues. Ainsi nous ne pouvons estimer que la version discrète du transport optimal. Les potentielles duaux du transport optimal peuvent être approximer avec un réseau de neurones comme dans [START_REF] Arjovsky | [END_REF]], ou nous pouvons utiliser un réseau de neurones pour approximer une transformation entre les domaines.

L'objectif de cette thèse est d'améliorer l'état des connaissances sur :

• Comment le transport optimal peut-être utilisé en apprentissage profond pour de nouvelles applications ?

• Est-ce que l'approximations du transport optimal par minis lots définit un problème de transport?

Les contributions de cette thèse

Cette thèse a débuté en Novembre 2018 et contient des contributions sur les interactions du transport optimal et de l'apprentissage profond. Les différentes contributions peuvent être séparées en deux parties. Dans la première partie, nous développons l'utilisation du transport optimal en tant que fonction de coût pour l'apprentissage profond.

• Dans notre travail initial [Fatras 2021a], nous étudions le problème des étiquettes corrompues. Cela correspond au problème d'apprentissage supervisé où une proportion des étiquettes est corrompue, ce qui diminue les performances de généralisation des modèles d'apprentissage profond [START_REF] Zhang | [END_REF]].

Pour résoudre ce problème, nous proposons d'utiliser le transport optimal, entre les étiquettes et les prédictions, comme fonction de coût dans l'entraînement virtuel adversaire (VAT) [Miyato 2018a].

VAT promeut une uniformité locale de la prédiction du classifier en pénalisant les changements brutaux de la classification pour de faibles perturbations des données. La pénalisation est "isotrope" entre toutes les classes, ce qui peut ne pas être optimal lorsque deux classes sont similaires. Incorporer le transport optimal dans VAT permet de moduler la force de la régularisation en fonction de la similarité sémantique des classes. Intuitivement, nous voulons des frontières complexes lorsque les classes sont similaires et des frontières lisses lorsque les classes ne le sont pas. Ce comportement est encouragé en choisissant de nouveaux coûts de transport comme discuté dans le Chapitre 5.

• Dans un second travail, nous avons utilisé le transport optimal afin de générer des données qui sont incorrectement classifiées pour un classifier donné [Burnel 2021]. Nous utilisons WGAN, une variante de réseaux adversaires génératifs [Goodfellow 2014b] basée sur le transport optimal, qui est capable de générer des données réalistes [START_REF] Arjovsky | [END_REF], Gulrajani 2017]. A partir du classifier fixé et des données d'entraînement, nous avons développé plusieurs stratégies pour créer de nouvelles mesures de probabilité pour les données d'entraînement. Ces nouvelles mesures donnent plus d'importance aux exemples incorrectement classifiés par le classifier. Ensuite, nous donnons cette nouvelle distribution au WGAN afin de générer des données incorrectement classifiées.

Les publications associées à ces contributions sont rassemblées dans la liste suivante:

• [Burnel 2021] Jean-Christophe Burnel, Kilian Fatras, Rémi Flamary and Nicolas Courty. Generating natural adversarial Remote Sensing Images. In: IEEE Transactions on Geoscience and Remote Sensing (TGRS), 2021.

• [Fatras 2021a] Kilian Fatras, Bharath Damodaran, Sylvain Lobry, Rémi Flamary, Devis Tuia and Nicolas Courty. Wasserstein Adversarial Regularization on label noise. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.

• [START_REF] Burnel | [END_REF]] Jean-Christophe Burnel, Kilian Fatras, Nicolas Courty. Generating natural adversarial hyperspectral examples with a modified Wasserstein GAN. In: C&ESAR 2020.

Dans la seconde partie, nous étudions comment une approximation usuelle et performante du transport optimal amène à des questions ouvertes sur ce dernier. Afin de réduire la complexité algorithmique dans de nombreuses applications d'apprentissage profond, le transport est calculé entre des minis lots des données, tirés uniformément aléatoirement. Les réseaux profonds sont ensuite entraînés avec SGD. La stratégie des minis lots a aussi été utilisé avec le transport optimal de manière fructueuse [START_REF] Genevay | [END_REF][START_REF] Muzellec | [END_REF][START_REF] Damodaran | [END_REF]]. Cependant, calculer le transport optimal entre des minis lots ne correspond pas à calculer le transport entre les mesures de probabilité entières. C'est pourquoi nous présentons une étude théorique du transport optimal entre minis lots en tant que nouvelle fonction de coût.

• Dans nos travaux [Fatras 2021c, Fatras 2021b], nous proposons d'abord un formalisme rigoureux de l'approximation par des minis lots. Nous construisons un estimateur non biaisé qui respecte les marginales des mesures de probabilités. Nous donnons aussi une solution en forme close pour des données 1D. Elle illustre les effets de l'approximation par minis lots sur le plan de transport. Nous démontrons aussi que les minis lots construisent des connexions non optimales. Nous proposons ensuite une nouvelle fonction de coût, basée sur le transport entre minis lots, qui corrige quelques pertes causées par l'approximation. Nous étudions ensuite théoriquement les propriétés statistiques et d'optimisation du transport optimal, afin de faire un premier pas dans la compréhension des différents succès en apprentissage profond.

• Dans un travail successif, nous étudions aussi quelques limites de l'approximation des minis lots [Fatras 2021b]. Nous montrons empiriquement que les connections non optimales peuvent diminuer les performances des réseaux de neurones sur des applications spécifiques. Ces connexions sont des conséquences du tirage des minis lots ainsi que des contraintes sur les marginales. En effet, au niveau des minis lots, le tirage des données peut créer des données aberrantes qui seront transportées. Afin de remédier à ce problème, nous proposons d'utiliser une variante du transport à contraintes relaxées appelée transport optimal non balancé avec les minis lots. Nous montrons ensuite sur différentes expériences que notre méthode surpasse l'approximation classique.

Ces contributions ont été publiées dans plusieurs publications:

• [Fatras 2021b] Kilian Fatras, Thibault Séjourné, Nicolas Courty and Rémi Flamary. Unbalanced minibatch Optimal Transport; applications to Domain Adaptation. In: International Conference on Machine Learning (ICML), 2021.

• [Fatras 2021c] Kilian Fatras, Younes Zine, Szymon Majewski, Rémi Flamary, Rémi Gribonval and Nicolas Courty. In: arXiv:2101.01792 (Under Review)

• [Fatras 2020b] Kilian Fatras, Younes Zine, Rémi Flamary, Rémi Gribonval and Nicolas Courty.

Learning with minibatch Wasserstein: asymptotic and gradient properties. In: the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.

Chapitre 4 nous amène à la seconde partie de ce manuscrit. Il présente les différents concepts d'exemples adversaires en apprentissage profond. Nous donnons une définition générale ainsi que les stratégies basiques afin de générer de tels exemples. Nous expliquons ensuite comment ces exemples peuvent pirater des modèles pré-entraîner et les différents concepts derrière ces attaques pirates. Finalement, nous montrons comment ces exemples peuvent être utilisés pour définir une nouvelle stratégie d'apprentissage appelée entraînement adversaire virtuel (VAT), pour des problèmes de classification semi-supervisé. Cet algorithme promeut une prédiction locale uniforme autour de chaque donnée d'entraînement. Il le fait en pénalisant de large changement dans la classification.

Chapitre 5 présente notre première contribution. Il étudie le problème de classification avec des labels corrompus. Basé sur l'entraînement adversaire virtuel, nous utilisons le transport optimal dans cette régularisation qui permet de lisser les frontières entre les classes très différentes, comme le fait VAT. Cependant, le transport optimal nous permet aussi d'obtenir des frontières complexes entre les classes proches les unes des autres. Nous évaluons ensuite notre méthode sur des jeux de données standards et concrets.

Chapitre 6 présente une nouvelle distribution de probabilité qui permet de générer des exemples incorrectement classifiés par un classifier pré-entraîné. Après avoir décrit plusieurs stratégies de pondération des poids, nous créons empiriquement une nouvelle distribution basée sur une fonction softmax . Nous utilisons ensuite un modèle génératif afin de générer ces données incorrectement classifiées. Afin de montrer la pertinence de cette approche, nous évaluons les exemples générés afin de pirater des classifiers entraînés sur des données satellites.

Chapitre 7 ouvre la troisième et dernière partie de cette thèse. Nous présentons l'approximation du transport optimal, qui est étudié théoriquement et empiriquement dans cette troisième partie. Afin d'étudier cette approximation théorique, nous rappelons quelques résultats avancés de statistiques et d'optimisation du transport optimal. Nous présentons les différentielles généralisées de Clarke qui définissent une notion de régularité pour les fonctions localement Lipschitz [Clarke 1990]. Nous terminons ce chapitre par une courte introduction aux U-statistiques qui apparaissent dans le formalisme des minis lots [Hoeffding 1948[START_REF] Hoeffding | Probability Inequalities for Sums of Bounded Random Variables[END_REF].

Chapitre 8 est dédié à la définition rigoureuse du formalisme qui permet d'exprimer le transport optimal par minis lots. Nous commençons tout d'abord par définir un formalisme dans le cas de mesures empiriques et nous illustrons les conséquences sur le plan de transport. Nous prouvons aussi qu'une solution en forme close existe lorsque les données vivent en 1D. Ensuite, nous définissons un formalisme général pour toutes distributions et nous montrons que le transport optimal par minis lots n'est pas une distance. Cela nous amène à définir une nouvelle fonction qui corrige certaines pertes du transport par minis lots. Nous étudions la positivité de cette fonction de coût ainsi que ces performances statistiques. Après, nous présentons des bornes de déviation entre les estimateurs que nous avons créés et leurs espérances pour différentes hypothèses sur les mesures de probabilité. Nous terminons ce chapitre avec des expériences sur des modèles génératifs et sur du transfert de couleur.

Chapitre 9 termine les contributions de cette thèse. Nous discutons les différentes limites du transport optimal par minis lots à cause du tirage des minis lots ainsi que de la formulation originale du transport optimal. Comme le transport a des contraintes sur les marginales, il est sensible aux données aberrantes.

i Or nous montrons que le tirage des minis lots peut créer artificiellement de telles données aberrantes. Afin de réduire ces sensibilités, nous proposons de remplacer le transport optimal entre les minis lots par une variante qui a des marginales relaxées telle que le transport optimal non balancé. Nous démontrons empiriquement que le transport optimal non balancé entre les minis lots a de meilleures performances que le transport optimal original entre les minis lots sur des expériences de flots de gradients et d'adaptation de domaine.

Chapitre 10 conclut cette thèse et discute les potentielles directions pour les futures travaux sur les interactions entre le transport optimal et l'apprentissage profond.

Appendice A rassemble les différentes preuves des résultats qui ont été annoncés dans les chapitres précédents.

vi

Contents vii

Notations

Linear algebra a, A all vectors in R d and matrices are written in bold. The coordinates will be written A i,j for matrices and a i for vectors without bold.

. , , a norm (depends on the context) and an inner product p denotes the standard . p norm tr, det is the trace operator for matrices i.e. tr(P) = i,j P i,j and the determinant operator

. F is the Frobenius norm for matrices i.e. P F = tr(P T P)

., . F is the inner product for matrices i.e. P, Q F = tr(Q T P) element-wise division operator for two vectors, i.e. The community of artificial intelligence has been able to design human-level performance algorithms for non trivial tasks. While being non trivial, these tasks can be solved using list of formal rules and the calculus power of computers. For instance in 1997, Deep Blue, a supercalculus computer which played chess, was the first computer to beat a chess world champion, the Russian Garry Kasparov. It was based on a brute force algorithm which explored the maximum possible number of moves in a certain time and chose the best one among them. However, despite these successes, some challenging tasks that are natural to human beings, like classifying different objects or understanding conversations, remained hard to solve. These problems rely on intuitive concepts which are hard to formalize and a method like the one used by Deep Blue can not catch all the hidden complexity. We thus need to rely on another approach to perform these tasks, which is called Machine Learning.

u v = ( ui vi ) i M m (
The difference and strength of machine learning algorithms lie in their ability to learn from data, without requiring human operators. By making mistakes on training samples, the algorithms are able to adapt in order to match the desired outcome. They are then capable of making predictions on situations they have not seen during their training. It gives them the unique ability to achieve repetitive tasks which are natural to human beings. The potential applications of machine learning are wide and the domain has already been applied for autonomous vehicles, cancer detection or language translation. Recently, a specific class of machine learning algorithms pushed the performances of the domain even further by improving almost all applications. This class of algorithms is called Deep Learning.

Deep learning models are a particular instance of machine learning algorithms. They are artificial neural networks which are composed of many layers with many neurons. This leads to a very high number of neurons and training these networks is computationally challenging. It has only be possible thanks to a new class of hardware called Graphics Processing Units (GPUs). These networks allow a more complex representation of training samples and thus they can make more complex predictions. However, deep learning's recent successes are also due to the development of recent mathematical fields, which helped the neural networks to achieve the desired tasks. This thesis is about studying the different interactions of deep learning with one of these mathematical areas called Optimal Transport (OT).

Deep Learning and Optimal Transport

The rise of Deep Learning. The first neural network architecture can be traced back to the middle of the twentieth century [Rosenblatt 1958] and the first deep neural network architectures, as well as their training procedure, thirty years latter [Rumelhart 1986[START_REF] Lecun | [END_REF], Lecun 1998]. In spite of their early design, researchers faced computational limits due to the size of the networks. Undeterred by these limits, in 2012, researchers made a first breakthrough on a real-world problem using these architectures. That year, the heavy neural network AlexNet [START_REF] Krizhevsky | [END_REF]], trained on a classification task with stochastic gradient descent (SGD) and GPUs [Robbins 1951a], achieved by a large margin the highest performance ever on the ImageNet competition [Deng 2009]. This first success in classification problems, coupled with the computational improvements resulting from high performing GPUs, led the research community to reconsider and develop the area of Deep Learning.

Since this success, deep learning has arisen as the current most competitive method to make datadriven decisions. To improve its performances even further, a lot of work has been done to find relevant architectures depending on the problem. For instance in computer vision and classification, the ResNet architecture empirically eased the training of deep neural networks [START_REF] He | [END_REF]]. In segmentation, researchers developed the U-net architecture [Ronneberger 2015]. Another successful approach that improved deep learning results was the use of minibatch training, coupled with a performing optimizer. Indeed, neural networks are trained using gradient-based algorithms, such as gradient descent, despite the fact that they are not convex. In a big data regime, using gradient descent would mean to compute a full gradient at each iteration. Thus to accelerate the training of neural networks, the community has relied on stochastic optimization [START_REF] Bottou | Large-scale machine learning with stochastic gradient descent[END_REF]]. Stochastic optimization only needs to compute a gradient of a minibatch of data. The price to pay for using SGD is a slower convergence. However, the overall performance is faster as one SGD iteration is a lot cheaper than one iteration of gradient descent. Several SGD variants were published to fasten the training of neural networks or improve their results such as Adam [Kingma 2015] or RMSprop [Tieleman 2012a]. It is thus a natural question to investigate the convergence of SGD towards a good solution when a new loss function is used with neural networks.

Following these corner-stone developments, deep learning has reached state-of-the-art results on many supervised and unsupervised problems, such as for instance domain adaptation [START_REF] Ganin | [END_REF]], segmentation [Ronneberger 2015] and generative modelling [Brock 2019]. Its use also expanded to other research domains such as physics [Schütt 2017] and biology [START_REF] Jumper | [END_REF]]. The rapidly growing knowledge of deep learning is also possible thanks to the open-source mindset in the field. We highlight several deep learning Python open-source libraries such as PyTorch [START_REF] Paszke | [END_REF] or TensorFlow [START_REF] Abadi | [END_REF]], as well as the JMLR open-source journal edition, which encourages the creation of open-source libraries.

Nonetheless, it is important to mention that deep neural networks were not the only reason behind some recent machine learning successes. For instance, domain adaptation and generative modelling are applications which require to compare sets of samples. A deep neural network is a transformation applied on samples that mathematically defines a pushforward operator. When samples are fed to the network, it leads to new sets of "embedded" samples, which correspond to pushforward distributions, that we want to compare. Thus some mathematical notion of distance between sets of samples is needed. In order to define such a distance, we first need to correctly represent sets of training samples as data distributions.

Training data as empirical measure. We consider the framework of probability measures, where we associate an empirical probability measure to the collection of training data (x i ) n i=1 . The empirical measure takes the form of a sum of Diracs α n = n i=1 a i δ xi , where ∀i ∈ [1, n], a i > 0 and n i=1 a i = 1. This

Deep Learning and Optimal Transport 3

representation is called the Lagrangian representation and it represents a set of data as one mathematical object. As we suppose that the data are independent and identically distributed, we choose the weights as uniform, i.e., ∀i ∈ [1, n], a i = 1 n . For specific applications, as presented in Chapter 6, it is possible to reweight this distribution.

Thus the mathematical notion of distance we need is a distance between probability measures. It is possible to rely on divergences like Maximum Mean Discrepancies [START_REF] Gretton | [END_REF]] or Csiszar divergences [Csiszar 1975]. In this thesis, we focus on optimal transport to compare distributions. Optimal transport can be seen as a generic method to lift a distance between two points to a distance between sets of points. Unlike the methods mentioned previously, it considers geometric notions to compare probability measures. It is thus natural that optimal transport was used in deep learning to measure the discrepancy between probability measures. On the other hand, deep learning was also used to estimate the continuous setting of optimal transport.

Optimal Transport for Deep Learning. The origin of optimal transport traces back to the 18th century in the work of Gaspard Monge [START_REF] Gaspard Monge | Mémoire sur la théorie des déblais et des remblais[END_REF]]. While optimal transport has found numerous applications in different areas of theoretical and applied mathematics [Santambrogio 2015, Villani 2009], it has also found many successes in machine learning [START_REF] Peyré | [END_REF]]. Among them, we can highlight its use in domain adaptation, where it reached state-of-the-art results, either by transporting source data into the target domain [Courty 2017a] or by aligning embeddings of domains [START_REF] Damodaran | [END_REF]]. Furthermore we can also mention its use as a loss function for multi-label classification [Frogner 2015], where optimal transport was defined between labels and took into account the closeness of classes. Yet, the most famous application of optimal transport in deep learning is probably in Generative Adversarial Networks (GANs), where it was used as a loss function in the so-called Wasserstein GAN [START_REF] Arjovsky | [END_REF]]. Following GANs, an optimal transport variant of AutoEncoders has also been designed [START_REF] Tolstikhin | Wasserstein Auto-Encoders[END_REF]]. We finish our overview by mentioning the different applications in biology [START_REF] Schiebinger | [END_REF]], astrophysics [START_REF] Frisch | [END_REF] or signal processing [START_REF] Kolouri | [END_REF]].

In order to understand its successes and to improve its performances in deep learning, researchers studied several aspects of optimal transport. The first aspect is the statistical estimation of optimal transport, as in deep learning we mostly deal with high dimensional data. Several research have been done to study its convergence rate in population, known as sample complexity. The exact optimal transport case was studied in [Dudley 1969,Weed 2019] and the entropic-regularized OT case in [START_REF] Genevay | [END_REF],Mena 2019]. On the other hand, the computational complexity of optimal transport makes it prohibitive to use on large scale datasets. Thus, a lot of research has been done to accelerate the computation of optimal transport. Using an entropic regularization, optimal transport can be efficiently computed on GPUs with the Sinkhorn algorithm [Cuturi 2013]. Furthermore, the algorithmic complexity of the Sinkhorn algorithm was provided in [START_REF] Altschuler | [END_REF]]. Other strategies based on a minibatch approximation were developed in order to reduce the computational complexity [START_REF] Genevay | [END_REF][START_REF] Damodaran | [END_REF][START_REF] Salimans | [END_REF]], but a rigorous formalism was lacking.

On another note, variants of optimal transport has found numerous applications. For example, the Gromov-Wasserstein distance allows to compare probability measures which lie in different metric spaces. It has been applied in generative modelling [Mémoli 2011[START_REF] Bunne | [END_REF] or graphs [Vayer 2019a]. Relaxed marginal variants of optimal transport have been used in many applications to make the transport robust to outliers [START_REF] Chapel | Partial Optimal Transport with Applications on Positive-Unlabeled Learning[END_REF][START_REF] Balaji | [END_REF]].

Chapter 1. Introduction

Deep Learning for Optimal Transport. Deep learning can be used to approximate the continuous formulation of optimal transport. As the probability measures are unknown in many applications, we rely on empirical estimations of these measures. Thus we can only estimate a discrete optimal transport cost. The dual potential of optimal transport can be approximated with a neural network, as done in [START_REF] Arjovsky | [END_REF], or we can also use a neural network to approximate a mapping between domains [Seguy 2018].

The purpose of this thesis is to bring new knowledge on:

• How can optimal transport be used in deep learning for new applications ?

• Do optimal transport approximations used in deep learning still define a transport problem ?

Contributions of the thesis

This thesis, started in November 2018, makes some contributions on the use of optimal transport in deep learning. The different contributions can be divided in two distinct parts. In the first part, we develop the use of optimal transport as a loss function for deep learning.

• In the initial work [Fatras 2021a], we study the label noise problem. It corresponds to a supervised learning setting where a proportion of labels are corrupted, which hurts generalization of deep neural networks [START_REF] Zhang | [END_REF]]. To solve this problem, we propose to use optimal transport, between labels, as a loss function in the well known perturbation-based regularization, the Virtual Adversarial Training (VAT) [Miyato 2018a]. VAT promotes a uniform local prediction of the classifier by penalizing large local changes in the classification. The penalization is "isotropic" between all classes, which might not be optimal when two classes are similar. Incorporating optimal transport in VAT allows us to modulate the regularization strength with respect to the semantic similarity of classes. Intuitively, we want to have complex classifier boundaries for classes which are similar to each other and smooth boundaries for classes which are not. This behaviour is encouraged with well-chosen new ground costs as discussed in Chapter 5.

• In a second work, we use optimal transport to generate samples which are misclassified for a given classifier [Burnel 2021]. We rely on WGAN, an optimal transport variant of generative adversarial networks [Goodfellow 2014b], which has been used successfully to generate highly realistic data [START_REF] Arjovsky | [END_REF], Gulrajani 2017]. Given the classifier and training data, we develop several weighting strategies to create new probability measures for training data, where misclassified data receive bigger weights than correctly classified data. Then, we feed the new probability measure to the WGAN in order to generate misclassified data.

The publications associated to these contributions are gathered in the following list:

• [Burnel 2021] Jean-Christophe Burnel, Kilian Fatras, Rémi Flamary and Nicolas Courty. Generating natural adversarial Remote Sensing Images. In: IEEE Transactions on Geoscience and Remote Sensing (TGRS), 2021.

• [Fatras 2021a In the second part, we study how a usual and successful deep learning approximation can lead to open theoretical questions in optimal transport. In order to reduce the computational complexity in many deep learning applications, the loss function is computed between minibatches of data, drawn uniformly at random. The deep neural network is then trained with SGD. The minibatch strategy has also been used with optimal transport and showed impressive results [START_REF] Genevay | [END_REF][START_REF] Muzellec | [END_REF][START_REF] Damodaran | [END_REF]]. However, computing optimal transport between minibatches of data does not correspond to computing optimal transport between the full input measures. Thus we present a theoretical study of the optimal transport minibatch approximation.

• In our works [Fatras 2020b, Fatras 2021c], we first provide a rigorous formalism of the minibatch approximation. We build unbiased empirical estimators which respect the marginals of input measures. We also provide a closed-form solution when data lie in 1D, which illustrates the effect of the minibatch approximation on the transport plan, and we empirically show that minibatch optimal transport creates non-optimal connections. We then propose a new loss function, based on minibatch OT, to correct some loss of distance properties, due to the minibatch approximation. We then theoretically study the statistical and optimization properties of minibatch optimal transport, to make a first step in understanding its successes in deep learning.

• In a successive work, we also study some limits of the minibatch approximation [Fatras 2021b]. We empirically show that the non-optimal connections can harm performances of neural networks on specific applications. These connections are consequences of the minibatch sampling and the marginal constraints. Indeed, at the minibatch level, the sampling can create outliers which are transported.

To mitigate this effect, we propose to use a relaxed marginal OT variant, called Unbalanced Optimal Transport (UOT), at the minibatch level. We then show on different experiments that our method successfully outperforms minibatch OT.

These contributions have led to several publications which are gathered in the list below:

• [Fatras 2021b] Kilian Fatras, Thibault During his PhD, the author also contributed to the Python Optimal Transport open-source library which was acknowledged by the community through a publication. Finally, he also had a collaboration with Fabian Pedregosa on stochastic optimization algorithms for complex penalties. In the next section, we provide a detailed presentation of the different parts and chapters constituting this thesis.

Organization of the manuscript

The thesis is divided in three parts corresponding to the different contributions made by the author. The first part reviews the basic concepts of numerical optimal transport and its use in deep learning. The second part discusses the first contributions on optimal transport as a loss function in deep learning. After a short introduction about adversarial examples, we present our perturbation-based regularization, coupled with optimal transport, for label noise problems. We then present our reweighting strategies of empirical distributions in order to generate data which are misclassified by a given classifier. Finally, the third and final part is focused on studying the minibatch approximation of optimal transport. We provide a rigorous formalism of this approximation and then, empirically study some of its limits. We now detail the organization of the manuscript.

Chapter 2 is written to bring basic mathematical and numerical knowledge of optimal transport to the reader. It presents the fundamental concepts behind the Kantorovich formulation as well as different variants used in this manuscript. Among all the different variants, we give a particular focus on entropicregularized optimal transport, unbalanced optimal transport and the Gromov-Wasserstein distance. We also present basic solvers of specific optimal transport variants. A reader familiar with the concepts of numerical optimal transport may skip this part, although it defines crucial notations that will be used along the thesis.

Chapter 3 reviews the use of optimal transport as a loss function in deep learning. More specifically, it is split in two parts. The first part gathers the different classification problems such as supervised learning, semi-supervised learning, learning with label noise and domain adaptation. We then review the literature on classification with an optimal transport loss. The second part is dedicated to study generative modelling problems with optimal transport. We first discuss the standard generative model methods and then how optimal transport has been used to improve these models. A reader familiar with standard deep learning applications might skip this chapter but it gathers basic concepts and algorithms that will be used in the different experiments presented in this manuscript.

Organization of the manuscript 7

Chapter 5 presents the first contribution of this manuscript. It studies the classification problem of learning with noisy labels. It is a setting where a fraction of the labels are corrupted. Based on the Virtual Adversarial Training, we use an optimal transport regularization to force the neural network to smoothen its decision boundary between classes that are very different from each other, as VAT does. However, optimal transport allows us to keep complex boundaries between classes close to each other. We then evaluate our method on standard and real-world label noise benchmarks.

Chapter 6 aims at introducing a new reweighted empirical distribution in order to generate data which are misclassified by a pre-trained classifier. After defining several strategies to reweight the different samples, we empirically design a new empirical distribution based on a softmax function. We then use a generative model to generate misclassified data. To show the relevance of this approach, our reweighted distribution is applied on real-world remote sensing data to fool pre-trained classifiers.

Chapter 7 opens the third and final part of this manuscript. We present the minibatch approximation of optimal transport, which is studied theoretically and empirically in this third part. In order to study this approximation theoretically, we review more advanced statistical and optimization results of optimal transport. We introduce the Clarke generalized gradients, which define a notion of regularity for locally Lipschitz functions [Clarke 1990]. We finish with the review of U-statistics [Hoeffding 1948[START_REF] Hoeffding | Probability Inequalities for Sums of Bounded Random Variables[END_REF] which arises in the sampling formalism of minibatches.

Chapter 8 is dedicated to define a rigorous formalism of minibatch optimal transport. We start by defining a formalism for uniform empirical measures and illustrate the consequences on the transport plan. We also highlight a closed-form formula when data lie in 1D. Next, we define a general formalism for any discrete distribution. Afterwards, we review the metric properties of minibatch optimal transport which leads us to define a new loss function to correct some loss of metric properties. We also study the positivness of this loss function. We then present deviation bounds and optimization properties for different hypotheses on probability measures. We finish with extensive experiments in generative modelling and color transfer.

Chapter 9 closes the contributions of the manuscript. We discuss the different limits of minibatch optimal transport due to the minibatch sampling and the OT formulation. As OT has marginal constraints, it is sensitive to outlier data and the minibatch sampling artificially creates such outliers. To mitigate these limits, we propose to replace optimal transport by a variant with relaxed marginals known as the Unbalanced Optimal Transport. We then empirically show that unbalanced minibatch optimal transport outperforms minibatch optimal transport on gradient flows and domain adaptation experiments.

Chapter 10 concludes this thesis and reviews the potential directions for future works on the interactions of optimal transport and deep learning.

Appendix A gathers the proof of the different results from the precedent chapters.

Part I

Optimal Transport and Machine Learning

Chapter 2 In this chapter, we rigorously define the notion of optimal transport. We first detail the probability measure framework which is ubiquitous in machine learning. We then review the optimal transport definition, its metric properties and its computational and memory complexities. Afterwards, we introduce the entropic-regularized optimal transport and review the benefits and consequences of this formulation. The presentation of two optimal transport variants will close this chapter.

Introduction to computational Optimal Transport

Motivations: dealing with measures

Machine Learning (ML) models aim at learning the best models that achieve a given goal from training data. The training data depend on the problem considered. In computer vision, data are images or videos, while in natural language processing, data are sentences or texts. In this manuscript, we mostly consider the area of computer vision and we focus on images in particular. Specifically, the images represent daily objects, such as cars or planes, human faces or remote sensing images. The tasks we want the models to achieve can be split in two categories. The first category is a classification task, where we want a model

Chapter 2. Introduction to computational Optimal Transport

to predict the class of a given image. The second category is image generation, where we want a model to generate images which would look like training data. Machine learning models are trained to achieve these desired tasks by minimizing a certain quantity. These quantities are estimated using probability distributions, which is a framework that allows to represent a set of data as one mathematical object. In this section, we define formally the probability distributions in the discrete and continuous cases. We then show how it appears in the two machine learning problems mentioned above. We finish this section by discussing how probability distributions are estimated in practice when they are unknown.

Definitions of probability distributions.

We now give a formal definition of probability measures and we start with the discrete case. We consider n samples from the data space X as {x 1 , • • • , x n }. Let Σ n denote the probability simplex of order n, i.e., the set of positive vectors which sum to 1, namely

Σ n = {a ∈ R + : n i=1 a i = 1}.
(2.1)

In order to avoid degeneracy issues, we suppose that all a i > 0. A discrete probability measure is defined as follows.

Definition 1. Given a tuple of weights a ∈ Σ n and locations

{x 1 , • • • , x n } ∈ X , a discrete probability measure reads α = n i=1 a i δ xi , (2.2) Where (δ xi ) i∈[1,n] is a Dirac at the position x i . The set of Diracs {δ x1 , • • • , δ xn } is called the support of the measure α.
The probability measure framework also allows to handle the case where we have an infinite number of samples, that we refer to the continuous case. This occurs in generative modelling where we can generate an infinite number of data. In this setting, the probability measure definition needs to be adapted. Consider the set of Radon measures M(X ) on the data space X which is equipped with a distance d.

We denote the set of probability measures on X as M 1 + (X ). We say that α ∈ M 1 + (X ) is a probability measure if it can be integrated against any continuous function f on X with compact support, formally ˆX f (x)dα(x) ∈ R and ˆX dα(x) = 1.

We now give concrete examples on why the measure framework is adapted for the tasks we consider.

Motivating examples.

In generative modelling, the purpose is to generate training-like data. A parametrized model denoted β θ where θ is the parametrization vector, i.e., θ → β θ , is designed to generate images. To make them look like training data, we want to minimize the distance between the generated data and the training data with respect to parameters θ. We achieve this goal by minimizing the distance between the corresponding probability measures of generated data β θ and training data α. In order to compute the dissimilarities between measures, it is common to rely on a contrast function or divergence

L : M 1 + (X ) × M 1 + (X ) → R + .
Thus, the goal is to find the optimal θ * which minimizes the distance L between the measures β θ and α, i.e. θ * = argmin θ L(α, β θ ).

The purpose of classification is to predict the class of a given image. In the supervised learning setting, we have access to an image x i and its label y i , which describes the class of the given image P (y i |x = x i ).

The label is a one-hot vector, where all coordinates are set to 0 except the one which represents the class

Kantorovich formulation of Optimal Transport

13

of x, which is set to 1. It is thus a probability vector, i.e., y i ∈ Σ n C where n C is the number of classes. To classify data, we wish to learn a function f : X → Σ n C which describes the relationship between a sample x and its label y. The function is learnt by minimizing the errors between its predictions and the correct labels, which are probability distributions with the same support. The error takes the form of a distance between these probability measures. It is typically calculated with the cross-entropy.

Definition 2. Let a and b be two probability vectors. Then the cross-entropy loss reads

L CE (a, b) = - n i=1 a i log(b i ).
(2.3)

We give a more detailed description of supervised learning in Section 3.1.1 as well as some of its variants.

Empirical estimation. Ideally, machine learning models would be trained on the true probability distribution of data α. However in practice, the distribution α is unknown and we have access to independent and identically distributed (i.i.d.) random samples drawn from α instead. We can thus estimate the measure α using a discrete probability measure, which support is the set of empirical samples, and the weights a ∈ Σ n are uniform. Formally, we define the empirical measure of α as

α n = 1 n n i=1 δ xi .
It is known that α n converges to α as the number of data n grows to infinity and we report to Section 7.2.1 for a longer discussion.

For instance in the supervised learning setting of classification, the data and their labels are sampled from an unknown joint measure P X ×Y . So its empirical counter-part is defined as

α n = 1 n n i=1 δ (xi,yi) . (2.4)
where δ (xi,yi) is a Dirac on the joint space of data and labels X × Y. Hence approximating the real joint measure P X ×Y is an important problem in order to get the best possible function with minimum prediction errors.

As probability measures are at the heart of many machine learning problems, finding a suitable distance to measure the discrepancy between them is a fundamental problem. Optimal transport has become a key tool to achieve this goal and is the topic of the next sections.

Kantorovich formulation of Optimal Transport

We start by giving an intuition about optimal transport, which measures the distance between two probability measures by taking into account the data space geometry. Intuitively it minimizes the displacement cost of a measure to another one. In this section, we introduce the formal optimal transport framework between arbitrary measures. We review the primal formulation, which is a linear program, and then we review its corresponding dual formulation. Afterwards, we study the metric properties of optimal transport. Next, we illustrate the particular case when data lie in one dimension, where a closed-form solution of optimal transport is available. Finally, we discuss the computational and memory complexities of solving the OT problem.

Primal and dual Optimal Transport formulations

We start by defining the primal formulation in the case of two continuous measures. Optimal transport measures a distance between two probability measures (α,

β) ∈ M 1 + (X ) × M 1 + (Y) by considering a ground
Chapter 2. Introduction to computational Optimal Transport cost c : X × Y → R + between the spaces X and Y. It looks for a probabilistic coupling to move a measure to the other. A probabilistic coupling π is a joint probability measure with marginals α and β, formally:

Definition 3. Let α ∈ M 1 + (X ), β ∈ M 1 + (Y)
be two probability measures. A probabilistic coupling π is a joint probability such that its marginals are equal to α and β. We denote π ∈ U(α, β) with

U(α, β) = π ∈ M 1 + (X , Y) : P X #π = α, P Y #π = β ,
where P X #π (resp. P Y #π) is the marginalization of π over X (resp. Y). P X (resp. P Y ) denotes the projection over the space X (resp. Y), and # denotes the pushforward operator.

The pushforward operator can be defined as follows.

Definition 4. The pushforward operator for a continuous map T : X → Y and for any measurable set B ⊂ Y is defined as:

β(B) = α {x ∈ X : T (x) ∈ B} .
Note that the notion of pushforward operator is important to formalize the image generation problem. We are now ready to state the Kantorovich formulation of optimal transport between continuous measures.

Definition 5. Let α ∈ M 1 + (X ), β ∈ M 1 + (Y)
be two probability measures and π a probabilistic coupling. The Kantorovich optimal transport formulation between probability distributions α and β is defined as

L(α, β, c) = min π∈U (α,β) ˆX ×Y c(x, y)dπ(x, y). (2.5)
It is called the Kantorovich problem [Kantorovich 1942] and we refer in this thesis to this problem as the continuous optimal transport formulation. It can be defined between measures of the same mass, but as our applications are machine learning tasks, measures are mostly probability measures. The ground cost represents the space geometry and influences the optimal connections between measures. Solving some machine learning problems requires to design a specific ground cost as it has been done in domain adaptation [Courty 2017b] or generative modelling [START_REF] Genevay | [END_REF]]. To assure the existence of optimal transport plans, we make the following hypotheses: unless said otherwise, X and Y are subsets of R d , where d is the space dimension. Regarding the ground cost, it can only be a positive and semi-continuous function on X × Y but for the sake of simplicity, we suppose that it is a positive continuous function on the space X × Y. We refer the reader to [Santambrogio 2015, Chapter 1] for a detailed review about the existence of optimal couplings for the Kantorovich problem. Note that the use of "min" instead of "inf" is justified in the formulation as the probabilistic coupling π = α ⊗ β ∈ U(α, β) is always admissible. We can also define the optimal transport problem for discrete measures.

Remark 1. Let X = {x 1 , • • • , x n } and Y = {y 1 , • • • , y n } be two data n-tuples. Let α = n i=1 a i δ xi and β = n i=1 b i δ yi be
two discrete probability measures with weights a and b respectively. Let C ∈ R n×n + be a ground cost matrix between the support of α and β, i.e., positions X and Y , which elements are C i,j = c(x i , y j ), ∀(i, j) ∈ n 2 . The Kantorovich optimal transport discrete formulation between probability distributions α and β is defined as

L(α, β, c) = L(a, b, C) = min Π∈U (a,b) Π, C F , (2.6)
where •, • F denotes the Frobenious product and where

U (a, b) = Π ∈ R n×n + : Π1 n = a, Π T 1 n = b .
When we do not deal with sum of Diracs but with a given ground cost C and random probability vectors a and b, we can also define an optimal transport problem L(a, b, C). The continuous and discrete cases are illustrated in Figure 2.1. The last case of interest is the semidiscrete case, where one of the measures is discrete while the other is continuous. We report to the next section for a detailed discussion about this particular case.

Kantorovich formulation of Optimal

When using optimal transport to compare probability measures with one measure being parametrized by a vector θ, i.e., θ * = argmin θ L(α, β θ , c), the corresponding estimator is usually found in the literature under the name of Minimum Kantorovich Estimator [START_REF] Bassetti | [END_REF][START_REF] Peyré | [END_REF]].

Kantorovich duality

The Kantorovich problem (2.5) is the minimization of a linear problem with respect to the transport plan π. As such it is a constrained convex minimization problem. It can be naturally transformed in a dual problem called Kantorovich dual, which is a constrained concave maximization problem. We start this section by defining the c-transform which is at the heart of many duality theorems.

Definition 6 (c-transform). Let R = R ∪ {±∞}, X , Y be subsets of R d and f : X → R be a function. We define its c-transform as the function f c : Y → R:

f c (y) = inf x∈X c(x, y) -f (x) (2.7)
and the c-transform of a function g : Y → R as the function g c : X → R:

g c(x) = inf y∈Y c(x, y) -g(y).
(2.8)

Fonctions that can be written as f c or g c are called respectively c-concave or c-concave functions.

One important property of the c-transform

is that ∀(x, y) ∈ X × Y, f (x) + f c (y) ≤ c(x, y).
The c-transform is a generalization of Fenchel-Legendre conjugate, where the Fenchel-Legendre conjugate of a function G is defined as:

∀y ∈ R d , F (y) = sup x∈R d x, y -G(x).
It is a convex function for any function G. The following fundamental proposition makes a connections between the continuous primal and dual formulations.

Proposition 1 (Dual for arbitrary measures). The Kantorovich problem admits the dual

L(α, β, c) = sup (f,g)∈R(c)
ˆX f (x)dα(x) + ˆY g(y)dβ(y), (2.9)

Chapter 2. Introduction to computational Optimal Transport

where the set of admissible dual potentials is :

R(c) = {(f, g) ∈ C(X ) × C(Y) : ∀(x, y) ∈ X × Y, f (x) + g(y) ≤ c(x, y)}.
The continuous functions f, g are called the Kantorovich potentials. The following remark highlights the dual formulation in case of discrete measures.

Remark 2 (Dual for discrete measures). The dual optimal transport problem in the discrete setting is

L(a, b, C) = max (f ,g)∈R(C) f , a + g, b ,
(2.10)

where the set of admissible dual potentials is :

R(C) = {(f , g) ∈ R n × R n : ∀i, j ∈ n 2 , f i + g j ≤ C i,j }.
The equality between discrete formulations holds thanks to the strong duality for linear programs [Bertsimas 1997] and the proof can be found in [Peyré 2019, Section 2.5].

Kantorovich -Rubinstein W 1 formulation

Using the c-transform, one can make the dual problem dependant on only one dual potential. It is useful in the context of semidiscrete optimal transport, where one of the measure is discrete while the other is continuous. Formally, we have:

L(α, β, c) = sup (f,g)∈R(c) ˆX f (x)dα(x) + ˆY g(y)dβ(y), = sup f ∈L 1 (α) ˆX f (x)dα(x) + ˆY f c (y)dβ(y), = sup g∈L 1 (β)
ˆX g c(x)dα(x) + ˆY g(y)dβ(y).

(2.11)

This formulation based on c-transform leads to another appealing formulation called the Kantorovich -Rubinstein W 1 dual formulation. When we transport two measures in the metric space (X , d) with the cost c = d, the dual formulation can be rewritten as a supremum over 1-Lipschitz dual potential. A full discussion can be found in [Peyré 2019, Chapter 6.]. We have then:

L(α, β, d) = sup g∈Lip1(X ) ˆX g(x)dα(x) -ˆX g(y)dβ(y) (2.12)
This formulation is at the heart of the Wasserstein GAN [START_REF] Arjovsky | [END_REF]] used for generative modeling, where the ground cost is the usual Euclidean distance. Note that the cost appears in the Lipschitz property.

After defining the primal and dual formulations, we state the optimal transport metric properties and its assumptions.

Metric Properties

Optimal transport is a metric between measures when certain properties are satisfied by the ground cost.

It can be seen as a canonical method to lift a ground distance between points to a distance between measures. We first recall the metric definition.

Definition 7 (Metric definition).

A metric on a space Z is a function f : Z × Z → R + which verifies the following axioms ∀x, y, z ∈ Z:

i d(x, y) = 0 ⇐⇒ x = y (separability axiom) ii d(x, y) = d(y, x) (symmetry) iii d(x, y) ≤ d(x, z) + d(z, y) (triangle inequality)
When the metric space is a space of measures, the elements x, y, z are measures and the axioms remain the same. We now state under what conditions the optimal transport cost is a metric.

Theorem 2.2.1. We assume X = Y and that it is equipped with a distance denoted d. Let p ∈ [1, +∞[, then for the ground cost c = d p , the optimal transport cost

W p (α, β, d) = L(α, β, c p ) 1/p (2.13)
is indeed a metric, called the p-Wasserstein distance, between probability measures α and β.

In addition to these metric properties, some statistical properties are discussed in Section 7.2. The metric properties of optimal transport make it a good candidate for comparing probability measures in ML. After defining the formalism and the optimal transport metric properties, we now give a specific case where the optimal transport has a closed-form solution.

1D data closed-form

The optimal transport problem does not have a closed-form solution in general. However when data lie on the real line R, a closed-form solution can be expressed for general probability measures on R. The 1D closed-form is well known in the OT literature and is at the heart of an OT variant called the sliced Wasserstein distance [START_REF] Rabin | [END_REF][START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF]], which we detail in the next paragraph. The closed-form solution of linear optimal transport is given in the next result and discussed in more details in [Santambrogio 2015, Theorem 2.9] and [Peyré 2019, Remark 2.30].

Theorem 2.2.2 (Closed-form expression on the real-line). Assume that X = R, α, β ∈ M 1 + (R). Let F α be the cumulative distribution function:

∀t ∈ R, F α (t) = α(] -∞, t]) (2.14)
and F -1 α its pseudo inverse, namely:

∀x ∈ [0, 1], F -1 α (x) = inf{t ∈ R |F α (t) ≥ x}. (2.15) If c(x, y) = h(y -x)
where h is strictly convex then the Kantorovich problem (2.5) has a unique optimal transport plan given by π

* = (F -1 α × F -1 β )#L [0,1] where L [0,1] is the Lebesgue measure restricted to [0, 1]. Moreover if α is atomless π * is supported on the map T (x) = F -1 β (F α (x)), i.e. π * = (id × T )#α.
If h is only convex then the optimal transport plan π * is still optimal but not necessarily unique. In the case of non uniform weights and different number of atoms α = n1 i=1 a i δ xi , β = n2 i=1 b i δ yi , the 1D closed-form sends as much mass it can between x 1 and y 1 and then add the remaining mass to y 2 . The operation is repeated until there is no mass left.

As for two discrete uniform probability measures with the same number of atoms and sorted supports, the optimal transport plan is a scaled identity matrix, it can be solved by a sort algorithm in O(n log(n)) operations.

Sliced Wasserstein distance: taking advantage of the 1D closed-form

This appealing property led to the creation of the so-called sliced Wasserstein distance. It consists in averaging the Wasserstein distance between projected data in 1D over all the directions in space. Formally it is defined as follows (2.16) where S d-1 is the sphere of dimension d and f θ , g θ the projected data in 1D. In practice, it is estimated with a Monte-Carlo method. It was first introduce in [START_REF] Rabin | [END_REF][START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF]] and then used in the context of generative models [START_REF] Kolouri | [END_REF], Kolouri 2018, Kolouri 2019a[START_REF] Liutkus | [END_REF]. However in the general case, the computational complexity of OT is much higher than O(n log(n)) and is discussed in the next section.

SW p (µ f , µ g , c) p = ˆSd-1 W p (µ f θ , µ g θ , c) p dθ,

Computational and memory complexities

For discrete probability measures, the Kantorovich formulation (2.6) is a linear program and can be solved using standard linear program solvers [Dantzig 1997]. One can then use the network simplex algorithm to solve the Kantorovich dual formulation. This algorithm has a computational complexity of O(n 3 log(n)) where n is the number of samples in the input measures. We report interested readers to [Peyré 2019, section 3.] for a more detailed discussion on how to solve the OT linear program. In a more simple case, we can make a connection between the Kantorovich formulation and the assignment problem.

Assignment problem

We consider the special case of two discrete probability measures with the same number of atoms. For any permutation σ ∈ Perm(n), we write the corresponding transport plan Π σ which

Entropic regularization: formulation and practical interest

19

is equal to Π σ i,j = 1 n if j = σ(i) 0 otherwise
. In this case, minimizing the assignment problem reads

min σ Π σ , C F = min σ 1 n i C i,σi .
While this formulation is greater or equal than the Kantorovich problem, one can show that for uniform weights, the two formulations are equal as proved in [Peyré 2019, Proposition 2.1]. The assignment problem can be solved using the Auction algorithm in O(n 3 ).

Memory complexity

The discrete primal formulation of optimal transport requires to store the ground cost C and the optimal transport plan. Thus its memory complexity is of order O(n 2 ). Regarding the dual formulation, it could be computed by storing the data in memory to compute the constraints, as well as the dual potentials and the probability vectors. Thus its memory complexity is of order O(nd) which is smaller than the primal formulation for big data scenario.

In this section, we defined the Kantorovich problem and reviewed its primal and dual formulations. We then discussed the different cases it defines a metric between probability distributions. Finally, we provided a closed-form solution when data lie in 1D and discussed its computational and memory complexities. In the next section, we regularize the Kantorovich problem with an entropic regularization.

Entropic regularization: formulation and practical interest

In this section, we discuss the introduction of the entropic-regularized optimal transport. It is defined by adding an entropic regularization penalty on the coupling to the original Kantorovich problem. This regularized loss has several important advantages with respect to original OT. For instance, it can be efficiently computed on GPUs using a 5 line algorithm. Furthermore, the resulting formulation is smooth with respect to input histogram weights and to the ground cost. Also, it can be differentiated using automatic differentiation. We present the entropic-regularized optimal transport definition, its equivalent dual formulation, the available solvers and the consequences on the optimal solutions. We finish by introducing a divergence based on entropic-regularized OT.

Definition and basic properties

The Kantorovich formulation has several interests but has also some downsides. The first problem is the non uniqueness of the transport plan which makes OT non differentiable w.r.t. the ground cost c, as discussed in Section 7.2.2. The second is its cubical computational complexity. Using an entropic regularization on the coupling helps to mitigate these two problems. This idea was first used in [Schrodinger 1931] and it was used for OT in [Wilson 1969]. It was then reintroduced recently in the OT literature in [Cuturi 2013] with the rise of GPUs. The entropic-regularized optimal transport problem between continuous probability measures with a ground cost c is defined as

L ε (α, β, c) = min π∈U (α,β) X ×Y c(x, y)dπ(x, y) + εH(π|ξ), (2.17) with H(π|ξ) = ˆX ×Y log dπ(x, y) dα(x)dβ(y) dπ(x, y) - ˆX ×Y dπ(x, y) + ˆX ×Y dα(x)dβ(y),
where ξ = α ⊗ β and ε ≥ 0 is the regularization coefficient. We denote it as the entropic OT loss.

Chapter 2. Introduction to computational Optimal Transport

Remark 4. In the discrete measures case the problem becomes:

L ε (a, b, C) = min Π∈U (a,b) Π, C + εH(Π|a ⊗ b), with H(Π|a ⊗ b) = i,j Π i,j log( Π i,j a i b j ) -Π i,j + a i b j . (2.18)
Adding this penalty leads to several interesting outcomes. First of all, the problem is now strongly convex w.r.t. the coupling π and remains linear in the cost c. Indeed for discrete measures, if one computes the Hessian of the Kullback-Leibler divergence w.r.t. the transport plan, one gets

∂ 2 H ∂Π 2 i,j (Π|ξ) = 1/Π i,j
, which is a matrix with positive entries. Hence the problem is now strongly convex and as such, has a unique solution. Regarding the dual of the entropic-regularized OT problem we have the following results.

Proposition 2. The dual of entropic-regularized OT reads

L ε (α, β, c) = sup (f,g)∈C(X )×C(Y) ˆX f (x)dα(x) + ˆY g(y)dβ(y) -ε ˆX ×Y exp( -c(x, y) + f (x) + g(y) ε )dα(x)dβ(y), (2.19)
Following the same logic as before, we can express the dual formulation between discrete probability measures.

Remark 5 (Dual for discrete measures).

L ε (a, b, C) = max (f ,g)∈R n ×R n f , a + g, b -ε exp( f ε ), exp( -C ε ) exp( g ε ) .
(2.20)

The proof of this proposition can be found in [Peyré 2019, Proposition 4.4]. The equality between the primal and the dual formulations is proven by the general duality theorem (Karush-Kuhn-Tucker conditions and Lagrangian formulation, see [Bertsekas 1997, Chapter 5]). Furthermore, expressing the Lagrangian of the dual formulation leads to the expression of the optimal transport plan. Proposition 3. The optimal transport plan of the discrete entropic OT has the form: (2.21) where K = e -C ε .

Π i,j = e f i ε e -C i,j ε e g j ε = u i K i,j v j ,
See [Peyré 2019, Proposition 4.3] for more details. The form of the optimal solution can be further simplified. It also highlights a simple algorithm to solve the entropic-regularized OT problem, which is the topic of the next subsection.

Solving entropic Optimal Transport

In this section, we detail the possible solvers of entropic-regularized optimal transport. We first detail why the Sinkhorn algorithm can be used in this context and then we focus on stochastic solvers.

Sinkhorn algorithm

The optimal transport plan of entropic-regularized OT has the matrix multiplication form Π = diag(u)K diag(v), thanks to Proposition 3. Furthermore, the transport plan needs to verify the marginal constraints, thus it reads

Π1 n = diag(u)K diag(v)1 n = u Kv = a. The marginal constraint on b equivalently gives Π 1 n = v K u1 n = b.
2: K = exp(-C/ε) 3: Define at random (u, v) ∈ R n + × R m + 4: while (u, v) did not converge do 5: u = a (Kv) 6: v = b (K u)
7: end while 8: return u, v numerical analysis community as the matrix scaling problem and it can be solved using the Sinkhorn algorithm [Sinkhorn 1967]. The Sinkhorn algorithm is detailed in Algorithm 1 and it can be interpreted as a block coordinate ascent on the dual problem. A practical feature of the Sinkhorn algorithm is that it can be efficiently solved on GPUs as noted in [Cuturi 2013], allowing a fast parallel computation.

Regarding its complexities, let η = 4 log(n) ε , then the Sinkhorn algorithm produces an optimal solution Π * such that C,

Π * F ≤ L ε (a, b, C) + η after O( C 3 ∞ log(n)η -3
) iterations [START_REF] Altschuler | [END_REF]]. A full discussion on the Sinkhorn algorithm with its analogies, convergence results and complexities can be found in [Peyré 2019, section 4.].

The Sinkhorn algorithm has seen many variants published in the recent years. As numerical stability appear for small regularization value ε, a stabilized version was developed in [Chizat 2018a] which is based on log-sum-exp iterations and is mathematically equivalent to the original version. Several published papers have focused on studying the Sinkhorn algorithm complexity [START_REF] Altschuler | [END_REF], Pham 2020]. To accelerate the computation of the solution, one can rely on a greedy strategy [START_REF] Altschuler | [END_REF], Abid 2018], or on reducing the problem by finding negligible components [Alaya 2019]. Other strategies such as Bregman projection operators [START_REF] Thibault | [END_REF]] or gradient descent acceleration [Dvurechensky 2018] are possible. Regarding its use in machine learning, as the entropic formulation is a smooth formulation w.r.t. the cost or the input histograms, a Sinkhorn AutoDiff formulation can be developed as done in [START_REF] Genevay | [END_REF]]. To speed up computations, they used a fixed number of Sinkhorn iterations. Finally, the entropic formulation can also be solved using stochastic algorithms.

Stochastic solvers for entropic-regularized OT The entropic OT formulation can be expressed as an expectation over data, allowing the use of stochastic algorithms to solve the OT problem. The idea is to express the entropic-regularized OT formulation using the c-transform. Suppose ε > 0, then using the c-transform, the dual OT program reads sup

f,g∈C(R d )×C(R d ) E x∼α,y∼β [F ε (f (x), g(y))], (s-D) sup g∈C(R d ) E x∼α [G ε (x, g)],
(s-SD)

where

F ε (f (x), g(y)) = f (x) + g(y) -εe f (x)+g(y)-c(x,y) ε and G ε (x, g) = ´Y g(y)dβ(y) - ε log( ´e 1 ε (g(y)-c(x,y)) dβ(y)) -ε.
Those smoothed problems are called the smooth dual (s-D) and the smooth semi-dual (s-SD) problems. The two problems are unconstrained maximization problems of an expectation and the idea is to use Stochastic gradients descent [Robbins 1951a] (SGD) or reduced variance stochastic (RVS) algorithms [Schmidt 2017a, Defazio 2014[START_REF][END_REF][START_REF] Nguyen | [END_REF] solution. These equations were first used in [Genevay 2016] and the authors compared the different stochastic algorithms to compute the solutions. We first discuss the discrete case.

When both measures α =

F ε (α i , β j )a i b j , (s-Ddis) max β∈R m n i=1 G ε (x i , β)a i . (s-SDdis)
The stochastic gradients computation complexity of (s-SDdis) is O(n). RVS algorithms can be used to solve the (s-SDdis) problem and have a converge rate of O(t -1 ) but at the price of a full gradient computation [START_REF][END_REF]] or the storage of past stochastic gradients [Schmidt 2017a, Defazio 2014].

When the extra cost of RSV algorithms can not be handled, one can rely on SGD with a convergence rate of O(t -1 2 ). A time comparison between the computation of original OT and entropic-regularized OT with the Sinkhorn algorithm or stochastic algorithms can be found in Figure 2.3. It shows that the stochastic formulation is slower than the Sinkhorn algorithm or the network simplex algorithm for a small number of data but faster in the Big Data regime. These poor performances in the small data regime might come from a non efficient Python loop in the stochastic solvers from the POT library [START_REF] Flamary | [END_REF]], which would decrease the time performances. Regarding (s-Ddis), its stochastic gradient computation cost is of order O(m 2 ), where m is the minibatch size. It has been solved using directly stochastic gradients between small minibatches in [Seguy 2018]. Due to this small gradient computation cost, it was proven to converge faster than SGD on the (s-SDdis) formulation [Seguy 2018]. Now suppose that one of the measures is continuous, another particular advantage of the smooth semi-dual formulation (s-SDdis) is that it can tackle the semi-discrete OT case, which is the formalism of generative models. Unfortunately RVS algorithms can not be used anymore as one of the measures is continuous and we need to rely on SGD. Thus this approach is not scalable in the Big Data regime.

Finally, the continuous OT case has also been studied in several works. [Genevay 2016] represented the dual variables with kernel expansions, [Seguy 2018] parametrized the dual variables with a neural network. Stochastic solvers for exact OT In the context of unregularized OT, the W 1 Kantorovich-Rubinstein formulation could be used as done in [START_REF] Arjovsky | [END_REF]]. The challenge of the W 1 formulation is to ensure that the potential is indeed 1-Lipschitz. The solutions are either based on weight clipping [START_REF] Arjovsky | [END_REF]] or making the gradient norm less or equal to 1 [Gulrajani 2017].

We have shown many desirable complexities of entropic-regularized OT. However, it is not the panacea and we develop its limits in the next section.

Entropic bias: solution far from original Optimal Transport solution

Adding the entropic regularization leads to several advantages, but it also changes the original problem and we discuss the consequences on the transported samples and the metric properties in this section.

The optimal transport plan of the entropic-regularized OT problem is not the plan with minimum transport cost anymore. Thus, using the entropic regularization breaks the separability axiom of distances and L ε (α, α, c) = 0. This is defined in the literature as the entropic bias [START_REF] Feydy | [END_REF]]. The greater the regularization coefficient ε is, the more uniform the transport plan gets as illustrated in Figure 2.4. Thus the optimal barycentring mapping of the entropic-regularized OT is no longer the target measure but a shrunk measure, whose support is smaller than the one of the target measure. The size of the barycentric mapping support depends on the coefficient ε. We now formally state the limit properties of entropic-regularized OT depending on the regularization parameter ε with the following proposition. Proposition 4. When the regularization coefficient ε converges to 0, the entropic-regularized optimal transport problem converges to the original optimal transport problem. Furthermore, when the regularization coefficient converges to ∞, the transport plan converges to the product measure a ⊗ b, i.e.,

ε → 0 =⇒ L ε (a, b, C) → L(a, b, C), (2.22) ε → ∞ =⇒ Π ε → a ⊗ b.
(2.23)

The proof of the above proposition can be found in [Peyré 2019, Proposition 4.1] when the OT problem is regularized with the entropy function. The case for the Kullback-Leibler regularization follows the same Chapter 2. Introduction to computational Optimal Transport steps. To mitigate this issue and recover the target solution, a debiased loss has been introduced in the literature and it is the topic of the next section.

Different OT regularization

The regularization of optimal transport has several flavors. In this section, we detail some other useful variants but which are out of the scope of this manuscript. In domain adaptation, some well-chosen regularizations of optimal transport were designed to ensure a special behaviour on connections and the transport plan. A l p l 1 norm regularization was proposed in [START_REF] Courty | [END_REF]] in order to concentrate the transport information on elements of the same class. In [Courty 2017a], the authors introduced a group-lasso regularization to promote the mass transfer from source data with the same labels to a given target data. They also introduced a Laplacian regularization which aims at preserving the data structure -approximated by a graph -during transport. Those problems were solved using a general conditional gradient. A squared 2-norm and a group-lasso regularizations of optimal transport were studied in [START_REF] Blondel | [END_REF]], leading to sparse and group-sparse transportation plans contrary to the entropic-regularized optimal transport plan.

Sinkhorn divergence, a debiased entropic Optimal Transport approach

Despite the appealing advantages of the entropic-regularized OT, the associated entropic bias might lead to poor performances in practice [START_REF] Feydy | [END_REF]]. To mitigate this issue, a debiased loss has been introduced, called the Sinkhorn Divergence.

The idea is simple, if the entropic-regularized OT L ε (α, β, c) is not zero for ε > 0 and α = β, then we just need to remove the non-zero bias for both measures α and β. The Sinkhorn Divergence is defined as follows:

S ε (α, β, c) = L ε (α, β, c) - 1 2 (L ε (α, α, c) + L ε (β, β, c)).
(2.24)

The discrete case is straightforward by replacing the entropic-regularized OT terms by their discrete counterparts. It is clear that we recover S ε (α, α, c) = 0 by construction of the loss. It can still be computed with the same order of computational complexity as the entropic-regularized OT, by using the Sinkhorn algorithm on each term for instance. The Sinkhorn divergence has been proven to interpolate between OT and Maximum Mean Discrepancy (MMD). MMD are integral probability metrics which consist of the integration of a positive kernel k over a Reproducing Kernel Hilbert Space (RKHS) X [START_REF] Gretton | [END_REF]] and they are defined as

α -β 2 k = E α⊗α [k (x, x )] + E β⊗β [k (y, y )] -2E α⊗β [k(x, y)].
(2.25)

We have the following proposition regarding Sinkhorn divergence's interpolation behaviour.

Proposition 5. Let α and β be probability measures and c a ground cost.

lim ε→0 S ε (α, β, c) = L(α, β, c) and lim ε→+∞ S ε (α, β, c) = 1 2 α -β 2 -c . (2.26)
The proof of this proposition can be found in [START_REF] Ramdas | [END_REF]]. The entropic-regularized OT is a positive cost as the sum of two positive functions. But this might not be the case for the Sinkhorn Divergence as we remove two positive terms. Fortunately, its has been proven in [START_REF] Feydy | [END_REF], Theorem 1] that the Sinkhorn Divergence is a convex, smooth, positive loss function that metrizes the convergence in law. We state their theorem.

Theorem 2.3.1 (Theorem 1 [START_REF] Feydy | [END_REF]].). Let X be a compact metric space with a Lipschitz cost function c(x, y) that induces, for ε > 0, a positive universal kernel k ε (x, y) = exp(-c(x, y)/ε).

Then, S ε defines a symmetric positive definite, smooth loss function that is convex in each of its input variables. It also metrizes the convergence in law: for all probability Radon measures α and

β ∈ M(X ) + 1 . 0 = S ε (α, α, c) ≤ S ε (α, β, c), (2.27) α = β ⇐⇒ S ε (α, β, c) = 0, (2.28) α n α ⇐⇒ S ε (α n , α, c) 0. (2.29)
This theorem ensures that the unique minimizer of S ε (α, β, c) = 0 is α = β on contrary to entropicregularized OT. It is a smoothed function and it can be approximated with an AutoDiff strategy. The Sinkhorn divergence has been used successfully in the context of generative models [START_REF] Genevay | [END_REF]] or gradient flows [START_REF] Feydy | [END_REF]] for instance. An important aspect of the Sinkhorn Divergence is its sample complexity which interpolates between OT and MMD, it is a concept that we detail in Section 7.2.1.

Optimal Transport extensions

In this section we review several other variants of optimal transport. The first variant is called the Gromov-Wasserstein distance and can compare measures which lie in different metric spaces. We present it as we investigate the theoretical justifications of the Gromov-Wasserstein distance computed between minibatches as presented in Chapter 8. The second variant is the unbalanced optimal transport. UOT allows to compare probability distributions which do not share the same mass by relaxing the marginal constraints on the coupling. This formulation is also robust to data outliers, i.e., it is not affected negatively by these outliers. The Unbalanced OT variant is introduced because it mitigates some downsides of optimal transport computed between minibatches as detailed in Chapter 9. There exist other variants of interests such as the multi-marginal optimal transport [START_REF] Peyré | [END_REF]] or factored coupling OT [Forrow 2019], but we do not present them as they are out of the scope of this manuscript.

Comparing measures lying in different spaces with Gromov-Wasserstein

Classical OT distances are defined when a relevant ground cost between the measures is defined. Unfortunately, the case where the source and the target data live in different metric spaces, such as R 2 and R 3 , does not fall into this framework. In this section, we review the Gromov-Wasserstein (GW) distance [Mémoli 2011], which can compare measures lying in different metric spaces. It has been investigated in the past few years and relies on comparing intra-domain distances. The general setting corresponds to computing couplings between metric measure spaces (X , c X , α) and (Y, c Y , β), where (c X , c Y ) are distances, while α and β are measures on their respective spaces. One defines the Gromov-Wasserstein distance as:

GW p ((α, c X ), (β, c Y )) = min π∈U (α,β) ˆX 2 ×Y 2 |c X (x, x ) -c Y (y, y )| p dπ(x, y)dπ(x , y ) 1/p .
(2.30)

In the discrete case the ground cost takes the form of a tensor C i,j,k,l = |C i,k -C j,l |, where C is the ground cost between the source data and C the ground cost between the target data. We can interpret the GW distance as follows: the coupling tends to associate samples that share common relations with

Chapter 2. Introduction to computational Optimal Transport

the other samples in their respective metric spaces. The Gromov-Wasserstein distance is not convex in the transport plan, but it is linear in the tensor cost. However, the Gromov-Wasserstein distance is challenging to compute, as a non convex quadratic program which is NP hard must be solved [Peyré 2016]. To address this issue from another perspective, one can realign the spaces X and Y using a global transformation before using the classical Wasserstein distance [START_REF] Alvarez-Melis | [END_REF]. In this paper, the authors cast the problem as a joint optimization over transport couplings and transformations chosen from a flexible class of invariances. Furthermore, an entropic variant of Gromov-Wasserstein [Peyré 2016] has been proposed to reduce its computational complexity following the same framework defined in Section 2.3. A Sinkhorn-like algorithm can also be adapted to the entropic-regularized Gromov-Wasserstein problem. A Sinkhorn Gromov-Wasserstein divergence has been introduced in [START_REF] Bunne | [END_REF]], but its positivity remains an open question. More recently, a sliced variant has been introduced in [Vayer 2019b] in the case of the specific squared Euclidean ground cost. Finally, a tree variant was proposed to accelerate the computation of Gromov-Wasserstein in [START_REF][END_REF]]. We now review the metric properties of the Gromov-Wasserstein distance.

Metric properties Gromov-Wasserstein GW defines a distance between different metric measure spaces under some hypotheses on the measure spaces. Let us first define isometries: Definition 8 (Isometry). Let (X , d X ) and (Y, d Y ) be two metric spaces. An isometry is a sujective map ϕ : X → Y that preserves the distances:

∀x, x ∈ X , d Y (ϕ(x), ϕ(x )) = d X (x, x ).
(2.31)

Thanks to the metric properties of d X , an isometry is bijective. Indeed for ϕ(x) = ϕ(x ) we have d Y (ϕ(x), ϕ(x )) = 0 = d X (x, x ) and hence x = x . By construction, the inverse map ϕ -1 is an isometry as well. The isometry concept can be extended between measure spaces.

Definition 9 (Strong isomorphism). Let (R d , d R d , α), (R q , d R q , β) be two measure spaces. We say that (R d , d R d , α) is strongly isomorphic to (R q , d R q , β) if there exists a bijection ϕ : supp(α) → supp(β) such that:

i ϕ is an isometry, i.e. d R q (ϕ(x), ϕ(x )) = d R d (x, x ) for x, x ∈ supp(α) 2 .
ii ϕ pushes α forward to β, i.e. ϕ#α = β.

However, the notion of strong isomophism relies on the notion of metric. This, in turn, means that metrics should be used in the GW distance definition. That is why we introduce the concept of weak isomorphism in order to consider other ground costs.

Definition 10 (Weak isomorphism). Let (R d , c R d , α), (R q , c R q , β) be two measure spaces. We say that (R d , c R d , α) is weakly isomorphic to (R q , c R q , β) if there exists (R q , c R q , ζ), with supp(ζ) = R q and maps ϕ 0 : R q → R d ,ϕ 1 : R q → R q such that: i c R q (z, z ) = c R d (ϕ 0 (x), ϕ 0 (x )) = c R q (ϕ 1 (x), ϕ 1 (x )) for z, z ∈ (R q ) 2 . ii ϕ 0 #ζ = α and ϕ 1 #ζ = β.
The isometry notion allows us to compare two different metric spaces. We are now ready to enumerate the Gromov-Wasserstein metric properties.

Theorem 2.4.1 (Metric properties of GW). Let (R d , d R d , α), (R q , d R q , β) be two measure spaces. The Gromov-Wasserstein is symmetric, positive and satisfies the triangle inequality. More precisely, let (R d , d R d , α), (R q , d R q , β), (R q , d R q , ζ) be measure spaces, we have:

GW p ((α, d R d ), (β, d R q )) ≤ GW p ((α, d R d ), (ζ, d R q )) + GW p ((ζ, d R q ), (β, d R q )).
(2.32)

The separability axiom is verified under the following condition:

i GW p ((α, d R d ), (β, d R q )) = 0 if and only if (R d , d R d , α) and (R q , d R q , β) are strongly isomorphic. ii GW p ((α, d R d ), (β, d R q )) = 0 if and only if (R d , d R d , α) and (R q , d R q , β) are weakly isomorphic.
iii More generally, for any p ≥ 1, GW p ((α,

d p R d ), (β, d p R q )) = 0 if and only if (R d , d R d , α) and (R q , d R q , β) are strongly isomorphic.
The positivity is clear by the GW definition. The symmetry and triangle inequality proof can be found in [Chowdhury 2019, Theorem 16]. For (ii), the proof can be found in [Sturm 2012, Lemma 1.10]. For (iii), the proof is available in [Chowdhury 2019, Theorem 18]. We now state the distance theorem for Gromov-Wasserstein.

Theorem 2.4.2 (GW is a distance). GW p is a distance on R d , . d , α ∈ P p (R d ); d ∈ N quotiented by the strong isomorphisms, where . d is a norm on R d .
The Gromov-Wasserstein distance allows to compare probability measures which lie in non comparable metric spaces. In the next section, we detail an OT variant which can compare measures with different masses.

Unbalanced Optimal Transport: relaxation of mass constraints

Due the mass conservation constraint, optimal transport can not compare measures with different mass. Unbalanced OT is a generalization of 'classical' OT that relaxes the conservation of mass constraints by allowing the system to either transport or create and destroy mass. In this section, we discuss this loss and its advantages over the original optimal transport formulation.

The unbalanced considered optimal transport formulation is built upon [Liero 2017, Frogner 2015] which replaces the 'hard' marginal constraints of OT by 'soft' penalties using Csiszàr divergences. A dynamical formulation has been investigated in [Chizat 2017, Chizat 2018b] but is out of the scope of this manuscript. There exists other extensions of the static formulations of OT. A famous one is partial OT which consists in transporting a fixed budget of mass [Figalli 2010a] or to move mass in and out of the system at a fixed cost [Figalli 2010b]. Another line of work proposes to optimize over various sets of Lipschitz functions [Hanin 1992[START_REF] Piccoli | [END_REF], Schmitzer 2017]. One can also replace Csiszàr divergences by integral probability metrics [Nath 2020].

To define this unbalanced variant, consider a convex, positive, lower-semicontinuous function φ such that φ(1) = 0. Define φ ∞ = lim x→+∞ φ(x)/x that we suppose strictly positive. Csiszàr divergences D φ are measures of discrepancy that compare pointwise ratios of mass using a penalty φ and are defined as D φ (x, y) = yi =0 y i φ xi yi + φ ∞ yi=0 x i . Total Variation and Kullback-Leibler divergences are particular instances of such divergences. Consider two positive measures α, β ∈ M + (X ). The Unbalanced

OT τ φ (α, β, c) = min π∈M+(X 2 ) ˆcdπ + τ (D φ (π 1 α) + D φ (π 2 β)), (2.33)
where π is the transport plan, π 1 and π 2 the plan's marginals and τ the marginal penalization. Note that the marginals of π are no longer equal to (α, β) in general. Regarding the metric properties we have the following theorem.

Theorem 2.4.3. Consider the square Euclidean ground cost c(x, y) = xy 2 and the Csiszàr divergence

D φ = KL, then the Unbalanced OT cost OT τ φ (α, β, c) 1 2 is the Gaussian-Hellinger distance, which is a distance on M + 1 (R d ).
Proof. The proof can be found in [Liero 2017].

Finally, note that an unbalanced variant of the Gromov-Wasserstein distance was proposed and studied in [START_REF] Séjourné | The Unbalanced Gromov Wasserstein Distance: Conic Formulation and Relaxation[END_REF]]. In this paper, the authors defined a distance and an upper-bounding relaxation which is more tractable. Both of them allow the comparison of metric spaces equipped with arbitrary positive measures up to isometries.

Entropic regularization Straightforwardly, it is possible to define an entropic extension of unbalanced optimal transport. It is defined as

OT τ,ε φ (α, β, c) = min π∈M+(X 2 ) ˆcdπ + εKL(π|α ⊗ β) + τ (D φ (π 1 α) + D φ (π 2 β)).
(2.34)

The formulation considered is computable via a generalized Sinkhorn algorithm [Chizat 2018a, Séjourné 2019] which is proven to converge and detailed in Algorithm 2. A recent analysis of the Sinkhorn algorithm for D φ = KL proved a convergence rate of Õ(n 2 /ε) [Pham 2020]. For these reasons, we rely in this manuscript on the unbalanced optimal transport with Kullback-Leibler divergence as our function φ and the entropic regularization. We illustrate the effect on the transport plans in Figure 2.5. We can see that when the parameter τ is high we indeed recover entropic-regularized OT, but when this parameter is low, the intensity of the connections declines.

Recently, an Unbalanced Sinkhorn Divergence was proposed in [Séjourné 2019]. It was proven to be a generalization of the original Sinkhorn Divergence. Furthermore it keeps the same properties, i.e., when e -C/ε is a positive definite kernel, it is a convex, symmetric, positive definite loss function which metrizes the convergence in law [Séjourné 2019]. The Unbalanced Sinkhorn Divergence reads where m α = ´dα and m β = ´dβ. The main difference is the introduction of the mass terms which are necessary to ensure positiveness and to recover a divergence. Those terms are equal and cancel each other in the case of balanced OT. Thus the Unbalanced Sinkhorn Divergence allows to mitigate between accelerated computations and conservation of key theoretical guarantees of the unbalanced formulation.

S τ,ε φ (α, β, c) = OT τ,ε φ (α, β, c) - 1 2 OT τ,ε φ (α, α, c) - 1 2 OT τ,ε φ (β, β, c) + ε 2 (m α -m β ) 2 , ( 2
Discussion A downside of the unbalanced formulation is that for small values τ , the unbalanced OT cost is very small because too much of the mass is destroyed. This leads to a transport plan with a very small norm, and thus to gradients with a small norm which might increase the training time of neural network in domain adaptation [Fatras 2021b] for instance. This will be discussed in Chapter 9.

Conclusion

We introduced the concept of optimal transport in this chapter. We reviewed its primal definition, which corresponds to a linear program in the discrete case, and the corresponding dual formulation. Afterwards, we reviewed the 1D optimal transport closed-form solution. We then discussed the computational and memory complexities of OT. Next, we presented the entropic-regularized optimal transport variant, its impact on the OT problem and some of its solvers. To recover the separability axiom from the entropic regularization, we introduced the Sinkhorn Divergence which merges the best of entropic-regularized OT and exact OT. Finally, we discussed two other optimal transport variants of interests: the Gromov-Wasserstein distance which compares probability measures in different metric spaces and the unbalanced optimal transport variant which relaxes the marginal constraints and can compare measures with different mass. In the next section, we present more formally the supervised learning setting as well as the adversarial formulation and the connections with optimal transport.

Classification using neural networks

In this section, we start by presenting how optimal transport can be used in a context of classification with neural networks. We present formally the supervised learning problem for multi-class and multilabel classification. Then we review how OT can be used in a supervised learning setting. We then present the domain adaptation problem, which consists of transferring knowledge from a source dataset to a target dataset, and how optimal transport is a well suited tool to achieve this transfer. After discussing classification problems, we review basic methods to generate highly realistic data. We start with autoencoders and follow with generative adversarial networks. We also discuss how optimal transport was used to solve these problems.

Supervised learning and multi-label classification

Supervised learning

We start with the problem of supervised learning. In supervised learning, we want to find a function f ∈ F that describes the relationship between a random feature vector x and a random target vector y, which follow the joint measure P(X , Y). To this end, we first define a loss function L that penalizes the differences between the prediction f (x) and the target y, for instance an image x and the given class encoded in a label y. Then, we define the average of the loss function L over the data measure P, also known as the expected risk: R(f ) = ˆL(f (x), y)dP(x, y). (3.1) Unfortunately the joint measure P(X , Y) is unknown and we have instead access to empirical samples drawn from this joint measure. Formally, let the i.i.d. data and their label be denoted as

Chapter 3. Optimal Transport in Deep Learning

(x 1 , y 1 ), • • • , (x n , y n ). We thus approximate the expected risk with the empirical measure, called empirical risk. We then minimize this empirical risk known as empirical risk minimization (ERM) [Vapnik 1998]. It is defined as

argmin θ 1 n n i=1 L(f θ (x i ), y i ), (3.2)
where f θ is the classifier parametrized by a vector θ such as a deep neural network. We consider a multi-class problem, i.e., y

= e c = [0, • • • , 0, 1, 0, • • • , 0],
where 1 is at the c-th position. y is a "one-hot" vector. In general, the last function of a neural network is a softmax activation function whose output is a probability vector. The hypothesis of multi-class supervised learning is that one image has a single element to classify. However, real-world images have several classes. For instance autonomous vehicles need to deal with a rich environment composed of cars, traffic signals, pedestrians and many other important elements. To correctly classify all these elements, the supervised learning paradigm needs to be adapted.

It is called multi-label classification and it is the topic of the next section.

Multi-label classification and Optimal Transport

In this section we review the multi-label classification problem and then, how we can use optimal transport losses to tackle this problem.

Multi-label classification

In multi-label classification, the label is not a "one-hot" vector anymore and thus not a probability vector. In this problem, we can encode several information in the label and the formalism is the following. Each element of the label corresponds to a class which potentially appears in the image. Thus an image with a dog, a human and a frisbee would have three "1" for each element in the label. So the neural network outputs now a vector where its elements are all probabilities for the different classes to be in the image. In order to get a probability classification for each output element, the last activation function of a neural network is usually a sigmoid function [Kang 2006]. The function f is still learnt through empirical risk minimization and the contrast loss is usually a Kullback-Leibler divergence [Long 2015a]. However the usual loss L for classification penalizes the error in an isotropic manner, i.e., they give the same importance to all errors between all classes independently. This behaviour does not reflect real life impact of errors, for instance confusing a cat and a dog is a bigger mistake than confusing two breeds of dogs. In the next paragraph, we discuss how optimal transport can be used in ERM to take into account the class similarities.

Multi-label Optimal Transport loss Let f θ : X → (P) n C be a map, where P denotes a probability and where n C is the number of classes. In multi-label classification, the function f θ (x) and the label y are measures of mass f θ (x) 1 and y 1 respectively. Thus we can use unbalanced optimal transport cost as our function L to measure the dissimilarities between the label and the model output as the mass f θ (x) 1 and y 1 are generally different. The strength of optimal transport losses is that it can take into account class similarities contrary to other losses. The similarities are encoded in the ground cost C. Formally, the UOT cost can be computed as h(f θ (x), y, C), where h is the entropic-regularized unbalanced optimal transport cost [Frogner 2015] as introduced in Section 2.4.2. Note that it is a smooth function w.r.t. the histograms, thus it has well defined gradients to train the model f θ with stochastic gradient descent [START_REF] Bottou | Large-scale machine learning with stochastic gradient descent[END_REF]]. The difficulty is to choose an informative ground cost. In [Frogner 2015], authors chose to rely on an Euclidean distance between the Word2Vec class representation [START_REF] Mikolov | [END_REF]]. This choice allows to measure the similarities between the classes. Using the UOT loss in ERM improved the performances of the trained models on standard real-world datasets. Straightforwardly, the unbalanced optimal transport cost could be used in a multi-class learning problem as detailed in the previous section. In the next section, we discuss a problem where we want to classify unlabelled data using labeled data of two different domains.

Domain adaptation

In this section, we introduce the problem of Domain Adaptation (DA). It consists of transferring knowledge from a source dataset to a target dataset by using the labeled source data to classify the unlabeled target data [Pan 2010]. We follow the settings of unsupervised DA where we have a labeled source domain dataset D s = (x s i , y s i ) n i=1 , x i ∈ R d and an unlabelled target domain dataset D t = (x t j ) n j=1 , x j ∈ R d . Other variants exist such as semi-supervised domain adaptation, where a few labels are available in the target domain and supervised domain adaptation where all labels in the target domain are available. There are also extensions of domain adaptation called multi-source domain adaptation or multi-target domain adaptation where we have respectively access to several source or target datasets. We give an illustration of the Office-Home dataset [START_REF] Venkateswara | [END_REF]] with different domains, see Figure 6.12.

In vanilla domain adaptation setting, both domains share the same label space Y s = Y t . Other variants can be considered such as partial domain adaptation, where extra classes in the source domain are present but not in the target domain Y t ⊂ Y s and open set domain adaptation, where extra classes are in the target domain but not in the source domain Y s ⊂ Y t . Those two variants need dedicated solutions that we discuss latter. Usually, at least one of the two following assumptions is made in most domain adaptation methods:

• Class imbalanced: Label distributions are different in the two domains (P s (y) = P t (y)), but the conditional distributions of the samples with respect to the labels are the same (P s (x s |y) = P t (x t |y)).

Open set and partial DA settings are special cases of class imbalanced;

• Covariate shift: Conditional distributions of the labels with respect to the data are equal P s (y|x s ) = P t (y|x t ). However, data distributions in the two domains are supposed to be different P s (x s ) = P t (x t ).

Based on the covariate shift hypothesis, [START_REF] Sugiyama | [END_REF]] used an importance re-weighting strategy to learn a classifier, however it requires a large enough overlapping of distributions. Kernel alignment has also been proposed [START_REF] Zhang | [END_REF]]. Other works learn a feature map T such that the new representations Chapter 3. Optimal Transport in Deep Learning of input data are matching, P s (T (x)) = P t (T (x)) [START_REF] Gong | [END_REF]]. Other approaches consider different class of maps such as projection [START_REF] Baktashmotlagh | Unsupervised Domain Adaptation by Domain Invariant Projection[END_REF], Gong 2012[START_REF] Fernando | [END_REF], Long 2014[START_REF] Gong | [END_REF]] and affine transform [START_REF] Zhang | [END_REF]]. We can also use different divergences between the probability measures of source and target data [START_REF] Si | [END_REF], Long 2014[START_REF] Gong | [END_REF]], but unfortunately these divergences require that the measures share the same support.

Another line of works is the alignment strategy in feature spaces and it can be summarized as follows; the neural network is used as a feature extractor and we align the extracted data features when data share the same class. The alignment of data spaces can be done with weight sharing [Sun 2016], reconstruction [Aljundi 2016], backpropagation [START_REF] Ganin | [END_REF]] or by adding Maximum Mean Discrepancy and association-based losses between source and target layers [Long 2015b, Haeusser 2017]. The alignment can also be achieved by an adversarial formulation which pushes CNN to be able to discriminate whether a sample comes from the source or the target domain [START_REF] Luo | [END_REF][START_REF] Tzeng | [END_REF][START_REF] Ganin | [END_REF][START_REF] Long | [END_REF][START_REF][END_REF]. Finally, the most recent works extend this adversarial logic to the use of generative models [START_REF] Liu | [END_REF][START_REF] Sankaranarayanan | [END_REF][START_REF] Murez | [END_REF], they use a class-conditioning or cycle consistency term to learn the discriminative embedding, such that embedded similar images in both domains are projected close by in the latent space.

In parallel of the empirical methods, several works were dedicated to make an analysis of the alignment strategy. They developed a bound on the target domain error rate by a source error rate plus an estimable term reflecting a certain distance between domains, called the alignment term, plus a non estimable term that is assumed to be small when adaptation is possible [START_REF] Zhang | [END_REF], Germain 2013]. [START_REF][END_REF] considered the bounds for 0-1 loss in binary classification setting by introducing a divergence term that takes into account the complexity of the hypothesis space, and to a case where the hypothesis case is a RKHS [Cortes 2014]. In the next section, we discuss how optimal transport has been successful at solving the domain adaptation problem.

Optimal Transport in Domain Adaptation

Optimal transport has been successfully used on the domain adaptation problem. Earlier work [START_REF] Courty | [END_REF], Courty 2017a] used optimal transport to find a coupling between the source and the target domains, then they transported the source data and their labels into the target domain with a barycentric mapping from the coupling. Finally they learnt a classifier on the transported samples in the target domain. This strategy is illustrated in Figure 3.2. Authors studied a great variety of regularized optimal transport to create optimal connections between samples. As we discussed in Section 2.3.3, a l p l 1 norm regularization was proposed in [START_REF] Courty | [END_REF]] in order to concentrate the transport information on elements of the same class. In [Courty 2017a], a group-lasso regularization was used to promote the mass transfer from source data with the same labels to a given target data. They also introduce a Laplacian regularization which aims at preserving the data structure -approximated by a graph -during transport. On the theoretical side, [Redko 2017] justified that the Wasserstein metric can be used as a divergence measure between distributions to obtain generalization guarantees.

We now detail more recent approaches. jdot is a method where optimal transport was used to find a coupling on the joint data and label distributions [Courty 2017b]. The ground cost incorporated a term on the data features and on the labels. Thus for two losses d and L, two data x i , y i and x j and a classifier f θ , the cost is of the form c(x i , y i , x j , f θ (x j )) = ηd(x i , x j ) + L(y i , f θ (x j )). Intuitively, matching close source and target samples with similar labels has a smaller cost than with different labels. η is a positive parameter which balances the metric in the feature space and the label loss. When η is set to +∞, we recover the initial setting from [Courty 2017a]. Finally, authors [Courty 2017b] also provided a generalization bound to justify their method theoretically.

Based on this approach, deepjdot [START_REF] Damodaran | [END_REF]] used a neural network g ψ as feature extractor, and used it instead of the data in the ground cost. Hence, the ground cost was c(g ψ (x i ), y i , g ψ (x j ), f θ (g ψ (x j ))) = ηd(g ψ (x i ), g ψ (x j )) + L(y i , f θ (g ψ (x j ))). The feature extractor g ψ and the classifier f θ are then trained together to match the source and the target measures. The full deepjdot loss and architecture is illustrated in Figure 3.3. It is important to note that the coupling was computed between minibatches of source and target data to keep a small computational complexity. This approximation is used in several problems and is the topic of Part III. This strategy reached state-of-the-art results. Other methods based on deepjdot, which focused on the coupling, were published [Xu 2020a, Fatras 2021b] and other methods were designed to handle the case of open set DA [Xu 2020b] and partial DA [Fatras 2021b].

Several other alignment strategies based on optimal transport were developed [Mengxue 2020]. Authors built an attention-aware transport distance, which can be viewed as the prediction feedback of the iteratively learned classifier, to measure the domain discrepancy. Another work learnt the transport cost for the specific task of domain adaptation [START_REF] Kerdoncuff | [END_REF]]. Finally, optimal transport can be coupled with adversarial learning to learn a domain invariant feature representation. [Shen 2018] uses a neural network, denoted by the domain critic, to estimate empirical Wasserstein distance between the source and target samples and optimizes the feature extractor network to minimize the estimated Wasserstein distance in an adversarial manner. And more recently, [Dhouib 2020b] introduced an OT-based adversarial approach to encourage large margin separation.

In this section, we have reviewed the domain adaptation problem as well as the different designed methods to solve it. We also discussed the specific case where optimal transport has been used to solve this problem. In the next section, we do not consider a classification problem but a data generation problem.

Chapter 3. Optimal Transport in Deep Learning

Generating high realistic data with Deep Learning

In this section, we review two main methods to generate highly realistic data. The first one is the autoencoder method. It aims to transform inputs into latent vectors and then, transform these same latent vectors as their corresponding inputs, with the least possible amount of distortion. We will review the basic idea and how optimal transport has been useful to solve it. Then we will review the Generative Adversarial Networks. Following an adversarial training, we will show how deep neural networks can generate data in an unsupervised way and how optimal transport has also been coupled with Generative Adversarial Networks.

AutoEncoder: Encoding data as a latent space

AutoEncoder [Rumelhart 1986] has become a standard tool to generate data because it has seen many improvements recently. It is composed of two distinct parts. An encoder whose the purpose is to encode an input into a latent vector which lies in a latent space. The second element is a decoder which aims to decode the latent vector into the original input. The encoder and the decoder are trained in order to minimize the reconstruction loss between the output and the original input. Then the latent space and the decoder are used to generate new data. Unfortunately, when the encoder is trained in this manner, it does not give a useful representation and sampling from the latent space becomes hard [Bengio 2013].

To overcome these difficulties, Variational Autoencoders (VAEs) [START_REF] Kingma | [END_REF]] minimizes a variational upper bound on the Kullback-Leibler divergence. The upper bound is a reconstruction loss between the output and the input coupled with a regularization. Formally, we denote the measure associated to real and training data as P r ∈ M + 1 (X ) (a positive Radon measure). We denote a latent variable model P G , specified by the prior distribution P z on the latent space Z, and the decoder P G (X|Z), which models the conditional likelihood. For two random variables X, Z, we denote the encoder as Q(Z|X) which approximates the posterior distribution of the decoder. The regularization is E Pr D KL (Q(Z, X), P z ) and it captures how distinct the image by the encoder of each training set is from the prior.

The total loss of VAEs is then: where Q is the set of, possibly restricted, conditional probability distributions. The decoder P G (X|Z) is parametrized by a neural network and can have any form as long as its density p G (x|z) can be computed and differentiated with respect to G's parameters. A standard choice is a Gaussian distribution

D VAE (P r , P G ) = inf Q(Z,X)∈Q E Pr -E Q(Z,X) log(p G (X|Z)) + D KL (Q(Z, X), P z ) , ( 3.3)
P G (X|Z) = N ((X; G(Z)), σ 2 • I). If Q is the set of all conditional probability distributions Q(Z, X),
the VAE loss coincides with the negative marginal log-likelihood D VAE (P r , P G ) = -E Pr log(P G (X)).

However to make the Kullback-Leibler divergence (3.3) tractable in closed-form, [START_REF] Kingma | [END_REF]] uses a standard normal distribution P z and restricts Q to Gaussian distributions. That is why VAE is minimizing an upper bound on the KL-divergence D KL (P r , P G ). To reduce the gap between the Kullback-Leibler divergence and VAE's loss, one possibility is to consider a larger set of encoder function Q as done in [START_REF] Mescheder | [END_REF][START_REF] Makhzani | [END_REF]]. The encoder is learnt using Generative Adversarial Networks and their adversarial learning formulation. We detail this class of generative models in the next section.

Optimal Transport meets AutoEncoders

As the purpose of autoencoders is to reduce the discrepancy between the measures P r and P G , [Tolstikhin 2018] proposed to rely on optimal transport. Following the above VAE framework, [START_REF] Tolstikhin | Wasserstein Auto-Encoders[END_REF]] focused on deterministic decoders G(X|Z), i.e., P G = G P z . In this case, the Wasserstein distance can be rewritten as

W (P G , P z , c) = inf Q(Z|X):Qz=Pz E X∼Pr E Z∼Q(Z|X) c(X, G(Z)) , ( 3.4)
where

Q z = Q(Z|X) P r = E X∼PrQ(Z|X)
. The full theorem and its proof can be found in [Tolstikhin 2018, Theorem 1]. This theorem allows us to optimize over random encoders Q(Z|X) instead of optimizing over all couplings between the input measures. We relax the problem by removing the constraint on Q z and adding a penalty instead. Thus the full WAE loss is

D WAE (P G , P z , c) = inf Q(Z|X)∈Q E X∼Pr E Z∼Q(Z|X) c(X, G(Z)) + ηD Z (Q z , P z ), (3.5)
where Q is any nonparametric set of probabilistic encoder and η the penalty constant. Similarly to VAEs, the encoder and the decoder are parametrized by a neural network. [START_REF] Tolstikhin | Wasserstein Auto-Encoders[END_REF]] proposed two different regularizations D Z . The first one is called WAE-GAN where the divergence is equal to the Jensen-Shannon divergence D Z = D JS . The divergence is estimated using a discriminator and adversarial training as done in Generative Adversarial Networks. The other considered penalty is a MMD loss (see Section 2.3) and it is called WAE-MMD. MMD can be estimated with U-statistics as detailed in Section 7.3 and thus can be optimized with SGD. A strength of WAE on VAE is that the encoder is a deterministic mapping. We illustrate the difference between WAE and VAE in Figure 3.4 and we gathered some samples of the three methods in Figure 3.5. Several other autoencoder variants of WAE were published in [START_REF] Patrini | [END_REF], Kolouri 2019b].

In this section, we described the autoencoder as a generative model. We formulate the Variational AutoEncoder and then the Wasserstein AutoEncoder which takes an optimal transport loss to measure the distance between measures. In the next section, we describe another class of generative models called

Generating high realistic data with Deep Learning 39

Generative Adversarial Networks.

Generate realistic data with Generative Adversarial Networks

In this section, we describe a popular method to generate data, the Generative Adversarial Networks (GANs). After describing the initial formulation, we review a more recent formulation based on optimal transport. We close this section by discussing specific applications of GANs.

A two player game formulation

Generative Adversarial Networks [Goodfellow 2014a] have become a popular unsupervised method for data generation. Their objective is to learn a model that can generate data which are similar to a given training dataset. It has initially been designed as a two player game between two DNNs, a generator and a discriminator. The discriminator tries to predict if an image is real or generated and the generator tries to fool the discriminator with its generated images. More formerly, learning a GAN corresponds to minimizing a divergence between the measures of generated data and training data. The choice of this distance is fundamental and has led to several variants of GANs [START_REF] Arjovsky | [END_REF], Gulrajani 2017, Miyato 2018b[START_REF] Nowozin | [END_REF], Li 2017a[START_REF] Genevay | [END_REF]. Formally, let the measure associated to real and training data be denoted P r ∈ M + 1 (X ) and the generated data distribution as

P G ∈ M + 1 (X )
. G is a deterministic generator, classically expressed as a neural network, that takes as input a random vector z ∼ N (0 q , I q ) to generate a sample G(z) ∼ P G , where q is the dimension of this random variable. We generally assume that q d, with d the dimensionality of the training data. Then, we want to reduce the distance with the real data measure P r . P z is usually a Gaussian or a uniform distribution. GANs improve the generated data quality by minimizing the Jensen-Shannon divergence between the measures P G and P r : min

ψ max φ E x∼Pr [log D φ (x)] -E z∼Pz [log(1 -D φ (G ψ (z)))]. (3.6)
However as the true measures P G and P r are unknown, we only consider their empirical measures from the available true data and the generated ones. Several variants were considered to control the label/class of generated data [START_REF] Mirza | [END_REF][START_REF] Odena | [END_REF]].

Wasserstein GAN: Optimal Transport meets GANs

The Wasserstein GAN [START_REF] Arjovsky | [END_REF]] relies on a different divergence than the Jensen-Shannon divergence.

It uses the Wasserstein distance to measure the discrepancy between the generated measure and the data measure. Unfortunately, the Wasserstein distance is intractable when one of the measure is continuous as it is the case for the generated measure P G ψ, . Hence, [START_REF] Arjovsky | [END_REF] proposed to approximate the Kantorovich-Rubinstein duality defined in Section 2.2.1. The difficulty of this formulation is to satisfy the Lipschitz property of the map f . Thus we approximate f with a neural network, namely

W 1 (P r , P G ψ, ) = min ψ max φ∈Φ E x∼Pr [D φ (x)] -E z∼Pz [D φ (G ψ (z))], (3.7)
where D φ is the dual potential and is within the set of 1-Lipschitz functions parameterized by weights φ, denoted Φ. Analogous to GANs, we still call D the "discriminator" although it is actually a real-valued function. In order to respect the 1-Lipschitz constraint, [START_REF] Arjovsky | [END_REF]] used a weight clipping trick on the discriminator's weights during the optimization procedure. However this practice leads to unstability during optimization and poor minima. Another approach was proposed in [Gulrajani 2017], involving a gradient penalty that enforces the norm of the gradient to be equal to one, thus promoting D 1-Lipschitz. Formally it is defined as

Chapter 3. Optimal Transport in Deep Learning

min ψ max φ E x∼Pr [D φ (x)] -E z∼Pz [D φ (G ψ (z))] + λE x∼P x [(max{0, ∇D φ ( x) 2 -1}) 2 ], (3.8)
where P x is the measure of samples along the straight lines between a pair of points from P r and P G and λ the regularization parameter. In practice, the true measure P r is unknown and only an empirical counterpart P r with n i.i.d. samples from P r is available.

Other variants of GANs based on optimal transport have been published in the literature [START_REF] Genevay | [END_REF][START_REF] Salimans | [END_REF],Fatras 2021c]. These methods relied on primal OT by using other ground cost [START_REF] Salimans | [END_REF] or by using extracted features learned in an adversarial manner [START_REF] Genevay | [END_REF], Fatras 2021c]. We discuss in more details these methods in Section 7.1.

Other applications of GANs

In this section, we discuss the different applications of GANs. When GANs were introduced they were first applied to images. Since then, large scale GAN training in order to get high resolution and diverse images was achieved in Big GAN [Brock 2019] as illustrated in Figure 3.6. GANs also allow to tackle the super resolution task and it was studied in [START_REF] Ledig | [END_REF]] and illustrated in Figure 3.7. In order to control the attributes of the generated images, a new generator architecture was proposed in Style GAN [Karras 2019]. Other work on image inpainting use GANs to reconstruct images [Liu 2018] or even modify them with some given labels [START_REF][END_REF]]. GAU-GAN [START_REF] Park | [END_REF]] proposed to synthesize an image solely based on some given labels. Recently; a line of work focused on image-to-image translation problem with GANs [Isola 2017] or on video-to-video [Wang 2018b]. The goal was to learn a mapping function from a source input to a corresponding realistic target output that precisely depicts the content of the source input. The input can also be matched to several outputs as done in [Choi 2018b]. When the corresponding output of the input is not available, CycleGAN can be used to achieve the input-to-output translation problem in an unsupervised manner [START_REF] Zhu | [END_REF][START_REF] Bansal | [END_REF]. A similar approach was built for motion transfer in order to create videos of people dancing known as Every Body Dance [START_REF][END_REF]]. This strategy was also applied to domain adaptation [START_REF] Murez | [END_REF]].

Conclusion

In this chapter, we have introduced the supervised learning setting for multi-class and multi-label problems. We have presented how optimal transport can be used to tackle the supervised learning problem, with a ground cost which encodes the class similarities. We then review the domain adaptation problem, it consists of classifying unlabeled data in a target domain using labeled data from a source domain. Optimal transport is used to align the domains of source and target data. In order to align the domains, recent methods used neural networks as feature extractors.

Then we reviewed two standard methods to generate highly realistic data. We first introduce the autoencoders which tries to reconstruct a given image from a latent space with the minimum of distortion. It takes the form of minimizing a divergence between two distributions, a setting particularly well suited for optimal transport, which was studied in Wasserstein autoencoder. Finally, we presented generative adversarial neural networks. Following an adversarial training between a generator and a discriminator, we showed that it also minimizes a divergence between measures. Hence, an optimal transport based variant, called Wasserstein GAN, was used in place of the original divergence. We reviewed the different advantages of this method and its implementation.

In the next part, we introduce the notion of adversarial examples and how it can be used to attack and to train a neural network for classification tasks. We then introduce the first two contributions of this

Adversarial examples: attacking a classifier

Adversarial examples: an unexpected weakness

In this section, we present the concept of adversarial attacks. An adversarial example is an example which was designed in order to fool a pre-trained classifier. The notion of adversarial examples for neural networks was first discussed in [START_REF] Szegedy | [END_REF]. They observed that perturbed images with small modifications can lead to a large change in the classification and to incorrect class predictions. These examples pose security issues as one could for instance fool an autonomous vehicle by using stickers or paint on traffic signals in order to make the car fail to interpret the signals. Adversarial examples thus create new lines of research to make the neural networks robust to such attacks or to use this concept for training purposes.

Attack and defense lines of research have studied the adversarial robustness of neural networks, i.e. the sensitivity of neural networks against these adversarial attacks. The attack line of research aims at creating adversarial examples for a pre-trained classifier. It also studies whether the adversarial examples can be transferred to other unseen pre-trained classifiers. To attack a classifier, we can have different hypotheses regarding the knowledge of the pre-trained classifier [START_REF] Goodfellow | [END_REF], Song 2018]. In order to make the neural network more robust to adversarial examples, efficient methods include specific training procedures with better adversarial resistance [Zhang 2018a[START_REF] Papernot | Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks[END_REF], modified training data [START_REF] Goodfellow | [END_REF][START_REF] Szegedy | [END_REF]] or the input samples before feeding them to classifiers [Hendrycks 2017[START_REF] Samangouei | [END_REF]].

Generating adversarial examples

Generating adversarial examples with the classifier's gradients. Turning to the existing literature on adversarial examples, there exists two main strategies to generate such examples. The first strategy is to add a small perturbation to images that are correctly classified, i.e., x adv = x + r a , r a ≤ ρ. The small perturbation, which can be computed from the classifier gradients, fools the classifier into making the wrong prediction. Intuitively, the gradient leads toward the direction with the most variation in the prediction. This is the main idea of the Fast Gradient Sign Method (FGSM) algorithm [START_REF] Goodfellow | [END_REF]] and it is illustrated in Figure 4.1. Formally, let θ be the parameters of our neural network f θ , the true class probability q(x) and a divergence D. As q(x) is unknown, we use the available label y instead. We look for the perturbation that leads to the largest change in the classification

r a = argmax r, r ≤ρ D(y, f θ (x + r)). (4.1)
Furthermore, estimating r a is untractable but we can approximate it with a linear approximation of D with respect to r in (4.1). Thus the adversarial example is generated as

x a = x + ρ sign(∇ x L(f θ (x), y))
and the gradient is computed using backpropagation. One problem of this approach is that one needs to know the full classifier architecture to compute the gradients. Another problem is that even a small perturbation can harm the image quality and lead to images that are not similar to the original data anymore. Such images can easily be detected as adversarial images [START_REF] Feinman | [END_REF], Pang 2018].

Generating adversarial examples with a GAN. The second paradigm does not require access to the classifier. It looks for adversarial examples only by having access to the classifier's prediction f θ (x). Most of the techniques using the second strategy try to find adversarial examples using GANs [Goodfellow 2014a]. For instance, to generate adversarial examples without requiring the classifier's gradients, [START_REF] Zhao | [END_REF]] uses a GAN and an invertor. The generator is a function which maps a latent space to the true data space. On the other hand, the invertor is an inverse function which maps the true data space to the latent space and they use it to look for adversarial examples. They add small perturbations to the latent vector z * which represents a data x * until they find an adversarial example. This method allows them to have very realistic adversarial examples without using the classifier's gradients. In [Song 2018] the authors use an

Virtual Adversarial Training for semi-supervised learning

47

AC-GAN [START_REF] Odena | [END_REF]] to model the data distribution with the assumption that an ideal model could generate all the sets of legitimate data. With such a model they search in the latent space for all the adversarial examples. Their search is done by minimizing the confidence score of a classifier while having the auxiliary classifier still predicting the correct class. So the adversarial examples look like true images and are adversarial for the attacked classifier.

Concepts of adversarial attacks

We gather now the different concepts related to adversarial attacks and some definitions. We rely on notations used in [START_REF] Goodfellow | [END_REF][START_REF] Zhao | [END_REF], Song 2018, Brown 2018]. We define by y pred = f θ (x) the operation of getting the vector in the simplex of prediction (for every classes) y pred of the datum x with respect to f θ . Furthermore for a classifier f θ , the probability to belong to the correct class y of a sample

x is denoted as f y θ (x). Without loss of generality, we identify here X to a set of normalized images with d pixels I {[0, 1] d }. Depending on the application, we note that X can be different (e.g. spectral data).

N I ⊆ I stands for the subset of all natural images. By natural images, we mean an image which is likely to be drawn from the empirical distribution of training images P r , therefore related to a given training set of images. Assume that we dispose of an oracle o, such that o(x) is the ground truth of the considered classification or prediction problem. In an attack scenario, we are given a classifier f θ that one wants to fool. This can be understood in several ways. Before, we precise two different cases: A white box attack is an attack where an attacker has complete access to the classifier's structure and parameters θ (e.g. when we have access to the neural network gradients). Oppositely, a black box attack is an attack where the only knowledge is the outputs {y pred } for some inputs {x}. Note that we suppose in the following that we have access to a fixed set of pairs {x, y pred } but we cannot query the classifier on new samples (since we could compute approximated gradients similarly to FGSM [START_REF] Goodfellow | [END_REF]. We now give a detailed description about misclassified example categories.

As described in the above section, the idea of perturbation-based example is to add a small perturbation to an input in order to create an adversarial example. Most of the time; the perturbation is limited by a factor ρ. The perturbed example is an adversarial examples if f θ (x) = f θ (x adv ). Formally, it can be written as

{x adv ∈ I | ∃x test ∈ I, x adv -x test < ρ ∧ o(x adv ) = o(x test ) = f θ (x test ) = f θ (x adv )}.
Natural examples are misclassified data-like images. It can be misclassified training or testing data for instance. Some perturbation-based examples are also natural examples. Formally, it can be written as

{x adv ∈ N I | o(x adv ) = f θ (x adv )}.
The last category is unrestricted examples. In this context there are both a pre-trained classifier and an oracle. Unrestricted examples are samples where the oracle and the classifier predict different result. Formally, it can be written as

{x adv ∈ I | o(x adv ) = f θ (x adv )}.
We are now ready to describe different attacks against the pre-trained classifier, i.e., the different strategies to fool the classifier. Untargeted attack stands for an attack from any adversarial example.

Targeted attack acknowledges an attack with an adversarial example from a source label, which is classified as a target label,

y target = f θ (x adv ) = y source = o(x adv ).
In the next section, we study how one can use adversarial examples to define a new training procedure for neural networks.

Virtual Adversarial Training for semi-supervised learning

In this section, we describe the semi-supervised learning problem, a variant of supervised learning, and we detail a state-of-the-art algorithm to solve it based on adversarial examples and semi-supervised learning

Semi-supervised learning

Semi-supervised learning is a classification task where some data have labels (x 1 , y 1 ), • • • , (x l , y l ) while the remaining data are unlabelled x l+1 , • • • , x n such that the number of labelled data is usually small, i.e., l n. This is a challenging problem because neural networks are known to need a large amount of data to achieve good generalization [START_REF] Zhang | [END_REF]]. Hence learning neural network classifiers in semisupervised learning is a difficult problem. To overcome the lack of labelled data, several works relied on using a relationship graph between the labeled and the unlabeled data to spread the label information [START_REF] Belkin | Semi-Supervised Learning on Riemannian Manifolds[END_REF]]. A more recent approach uses input perturbations with deep neural networks [Zhang 2018a, Berthelot 2019, Miyato 2018a]. The method we describe in the next section, called Virtual Adversarial Training (VAT) [Miyato 2018a], is one of these methods. They rely on the semi-supervised learning assumptions [START_REF] Chapelle | [END_REF], Section 1.2],

1. Data which are close to each other are more likely to share the same label.

2. Data tend to form clusters, and data in the same cluster are more likely to share a label.

VAT is built on the semi-supervised hypotheses to classify unlabeled data using labeled data. It promotes a local uniformity of the classifier in a ball around each input, thus data close to each other should share the same label as well as data inside a cluster. It is the topic of the next section.

Virtual Adversarial Training: the power of approximations

In this section, we introduce the Virtual Adversarial Training. We start by presenting the adversarial training framework and then how we use it to define a regularization for semi-supervised learning.

Adversarial Training. To make the model f θ more robust to adversarial examples, one can use the generated adversarial examples from the FGSM method to define a new loss function called adversarial training [START_REF] Goodfellow | [END_REF]]. For a sample x with corresponding label y, the adversarial training framework aims at minimizing a measure of discrepancy between the true class probability, q(x), and the prediction of a model f θ for a slightly perturbed version of x, f θ (x + r a ). We discussed the computation of r a in Section 4.1. In other words, adversarial training tries to minimize the change in classification in a ball around each input. It takes the form of the following regularization,

L AT (x, θ) = D(q(x), f θ (x + r a )), (4.2)
where r a = argmax r, r ≤ρ

D(q(x), f θ (x + r)).
In Eq. ( 4.2), D(p 1 , p 2 ) measures the discrepancy between the probability vectors p 1 and p 2 thanks to the divergence D. For instance, D can be the Kullback-Leibler divergence. ρ represents the radius of the local ball-shaped neighbourhood, where we seek the adversarial sample.

Approximating the true measure. The true class probability q(x) is unknown and intractable in practice. In semi-supervised learning, most of samples do not have a label y so we can not rely on it. That is why, [Miyato 2018a] proposed to use a 'virtual' label, i.e., the current label estimate from the classifier f θ (x). This approximation makes sense as at the end of the training, the current estimate should be close to the true label when the number of data is large. Another strength of relying on f θ (x) is that the quantity is defined even for unlabelled data. Since in VAT f θ (x) is taken as a label, it becomes a constant that we refer to as fθ (x), leading to the following regularization term,

L VAT (x, f θ ) = D( fθ (x), f θ (x + r a )),
where r a = argmax r, r ≤ρ D( fθ (x), f θ (x + r)). (4.3) Using f θ (x) instead of q(x) promotes a local uniformity in the predictions around each input. Thus, L VAT can be seen as a measure (negative) of the local smoothness, or also as an estimation of the local Lispschitz constant in the ρ neighborhood of x with respect to the metric D, hence a measure of complexity of the function. The full regularization term of VAT is the expectation of L VAT over all samples,

R VAT (X, f θ ) = 1 N N i=1 L VAT (x i , f θ ). (4.4)
By penalizing large local changes in classification, the classifier should have a smooth local prediction, thus giving the same label to data close to each other. This local spread of information should also spread among the whole cluster, thanks to the semi-supervised hypothesis.

Adversarial samples computation.

VAT requires an efficient computation of adversarial samples.

One could use the gradient with respect to the input but because of differentiability, it vanishes in r = 0. When we approximate D in r = 0 through the second order Taylor expansion, we have

D( fθ (x), f θ (x + r)) ∼ r=0 1 2 r t H r r. (4.5)
However, computing the Hessian H r with respect to the input is costly. Instead we use an iterative method to estimate the dominant Hessian's eigenvector that represents the direction in which the classification function will change the most. We use the power iteration method [Golub 2000, Chapter 6]. The algorithm starts from a random normalized vector d and computes the gradient w.r.t ξd as follows:

d ←-(∇ d D( fθ (x), f θ (x + ξd)) -∇ d D( fθ (x), f θ (x)))/ξ,
where ξ be a small real number. Since the second term is zero, we update the value of d with the calculated gradient ∇ d D( fθ (x), f θ (x + ξd))/ξ. This operation is repeated t max times, but the literature suggests that only one iteration is sufficient to achieve state-of-the-art results [Miyato 2018a]. In practice, the value of ξ is set to 10 -6 .

Once the adversarial direction d is defined, one can obtain the adversarial example with r a = ρd/ d 2 , by projecting onto the sphere of radius ρ. The adversarial samples follow the direction where the classification function has the biggest variation w.r.t D.

Conclusion

We have presented a weakness of deep neural networks when they are trained in a classical manner for classification problems: the adversarial examples. Adversarial examples are malicious data, which look like training data, but are misclassified for a given pre-trained classifier. Thus they can be used to attack the pre-trained classifier. We discussed the standard strategies to generate such examples, by input modification or by using GANs, and we reviewed the different concepts for attacking classifiers. We then showed how the adversarial examples can be used to design a new training algorithm for neural networks called adversarial training. Following this training strategy, virtual adversarial training was designed to train a model in semi-supervised learning, a setting where only a few data have a label. For a given input, VAT uses the current neural network prediction as a label and penalizes large changes in the prediction for a perturbed version of the input.

In the next chapters, we introduce two contributions. The first contribution is an extension of the VAT algorithm which uses an optimal transport loss in a learning with noisy labels scenario. We review the label noise problems and the different lines of research to solve it. We then introduce our second contribution and finally we evaluate it on different benchmarks. The second chapter is dedicated to the generation of natural examples, training images which are misclassified, using GANs. We introduce a simple and efficient reweighting strategy in order to give more importance to misclassified samples during the training of GANs.

Chapter 5 In this chapter, we present the problem of learning with noisy labels problem and different lines of research to solve this problem. We then present how adversarial training can be used in the context of noisy labels. Next, we present our main contribution: using a Wasserstein loss in the virtual adversarial training to promote local smoothness between very different classes and complex boundaries between classes similar to each other. We also present the selection of an insightful ground cost. Afterwards, we extensively evaluate our proposed method on different classical learning with noisy labels benchmarks. These contributions have been published in IEEE Transactions on Pattern Analysis and Machine Intelligence [Fatras 2021a].

Wasserstein Adversarial Regularization for learning with noisy labels

Learning with noisy labels

In this section, we first present the label noise problem, we also discuss the type of noise we consider for labels and the related work.

Definition and setting

The learning with noisy labels setting shares a lot of similarities with supervised learning. Like the supervised learning setting, we suppose that for each data x, we have access to its corresponding label y 52 Chapter 5. Wasserstein Adversarial Regularization for learning with noisy labels which encodes its class. The notable difference is that a portion of the label y are corrupted and encodes the wrong class of x. For instance, an image of a bird whose label is a plane. Such labels are called noisy labels. It is a common problem in practice since annotating large datasets is a challenging and costly task, which is practically impossible to do perfectly for every task at hand. It is then most likely that datasets will contain incorrectly labeled data, which induces noise in those datasets and can hamper learning. Furthermore, the probability of facing this problem increases when the dataset contains several fine grained classes that are difficult to distinguish [Schroff 2011[START_REF] Krause | [END_REF][START_REF] Dubey | [END_REF]]. These noisy labels are problematic because as pointed out in [START_REF] Zhang | [END_REF]], deep convolutional neural networks have huge memorization abilities and can learn very complex functions. That is why training a neural network with noisy labels can lead to poor generalization [START_REF] Arpit | [END_REF], Wang 2018a, Choi 2018a]. We now discuss the choice of noise setting we consider.

The label noise problem arises whenever some elements y do not match the real class of x. Several scenarii exist: in the symmetric label noise, labels can be flipped uniformly across all classes, whereas in the asymmetric label noise, labels y in the training set can be flipped with higher probability toward specific classes. We note that the first scenario, while thoroughly studied in the literature, is highly improbable in real situations: for example, it is more likely that an annotator mislabels two breeds of dogs than a dog and a car. Hence, noise in labels provided by human annotators is not symmetric since annotators make mistakes depending on class similarities [START_REF] Misra | [END_REF]]. In the next section, we describe the different lines of research to mitigate the influence of these noisy labels.

Related work on label noise

The label noise problem has been considered mainly in three ways in recent literature.

First are data cleaning methods: [START_REF] Brodley | [END_REF]] uses a set of learning algorithms to create classifiers that serve as noise filters for the training data. [Vahdat 2017, Xiao 2015, Li 2017b] learn relations between noisy and clean labels before estimating new labels for training. In [START_REF] Lee | [END_REF]], few human who verified labels were necessary to detect noisy labels and adapt learning. Authors in [START_REF] Natarajan | [END_REF]] use a simple weighted surrogate loss to mitigate the influence of noisy labels. In [START_REF] Jiang | [END_REF][START_REF] Ren | [END_REF], the methods rely either on a curriculum or on meta-gradient updates to re-weight training sets and downweight samples with noisy labels.

Second are Transition probability-based methods: [Liu 2014[START_REF] Menon | [END_REF], Sukhbaatar 2014[START_REF] Patrini | [END_REF][START_REF] Hendrycks | [END_REF]] estimate a probability for each label to be flipped to another class and use these estimations to build a noise transition matrix. In [Sukhbaatar 2014], the authors add an extra linear layer to the softmax in order to learn the noise transition matrix itself, while [START_REF] Hendrycks | [END_REF]] uses a small set of clean validation data to estimate it. [START_REF] Patrini | [END_REF]] proposes a forward/backward loss correction method, which exploits the noise transition matrix to correct the loss function itself.

Third are regularization-based methods:

In [START_REF] Reed | [END_REF], Ma 2018], the authors use a mixture between the noisy labels and network predictions. In [Tanaka 2018a, Kun 2019a], the regularization is achieved by alternatively optimizing the network parameters and estimating the true labels while the authors of [START_REF] Han | [END_REF][START_REF] Yu | [END_REF], Hongxin 2020] propose peer networks feedbacking each other about predictions for the noisy labels. [START_REF] Song | [END_REF] proposes to replace noisy labels in the mini-batch by the consistent network predictions during training, while [START_REF][END_REF] proposes noisy cross-validation to identify samples that have correct labels. In [START_REF] Wang | [END_REF], Zhang 2018b[START_REF] Ghosh | [END_REF], robust loss functions are proposed to overcome limitations of cross entropy loss function. Our proposed method follows this strategy where the regularization promotes local smoothness to fight the label noise influence and it is the topic of the next 5.2. Wasserstein Adversarial Regularization for label noise 53 section.

Wasserstein Adversarial Regularization for label noise

In this section, we present how we can use the Virtual Adversarial Training algorithm to tackle the label noise problem but discuss some limits. We then introduce an optimal transport based extension of VAT which corrects these limits.

Adversarial regularization: smoothing local predictions

To prevent a classifier to overfit on noisy labels, we would like to regularize its decision function in areas where the local uniformity of training labels is broken, i.e., when some labels are under represented in specific data area. To achieve such desired local uniformity, robust optimization can be used. This amounts to enforce that predicted labels are uniform in a local neighborhood U i of data point x i . This changes the total loss function in the following way:

arg min θ N i=1 max x u i ∈Ui L CE (f θ (x u i ), y i ) (5.1)
Because the robust optimization problem [START_REF] Caramanis | [END_REF]] in (5.1) is hard to solve exactly, adversarial training [START_REF] Goodfellow | [END_REF][START_REF] Shaham | [END_REF] was proposed as a possible surrogate. Instead of solving the max inner problem, it suggests to replace it by enforcing uniformity in the direction of maximum perturbation, called the adversarial direction as presented in Section

L tot (x i , y i , f θ ) = L CE (f θ (x i ), y i ) + ηR VAT (x i , f θ ), (5.2)
Where like in VAT, f θ (x i ) is the neural network prediction of the input and f θ (x i + r a ) is the neural network prediction of the adversarial input. A sound choice for D can be the Kullback-Leibler (KL) divergence. R VAT promotes local uniformity in the predictions without using the potentially noisy label y i : therefore, it reduces the influence of noisy labels, since it is computed from the prediction f θ (x i ) that can be correct when the true label is not. Our regularization shares strong similarities with Virtual Adversarial Training (VAT) [Miyato 2018a], at the notable exception that we do not consider a semi-supervised learning problem and that we regularize on the labeled training positions x i , where VAT is applied on unlabeled samples. Thus we call this framework AR which stands for Adversarial Regularization. It can be shown that this regularization acts as a label smoothing technique:

Proposition 6. Let D be the Kullback-Leibler divergence. Let γ = η η+1 ∈ [0, 1[. Let y a = f θ (x + r a
) be the predicted adversarial label. Let H be the entropy. The regularized learning problem L tot (x, y, f θ ) in (5.2) is equivalent to :

L tot (x, y, f θ ) ≡ L CE (f θ (x), (1 -γ)y + γy a ) -γH(y a ).
Proof. The total learning loss for one sample (x, y) is:

L tot (x, y, f θ ) = L CE (f θ (x), y) + ηR AR (x, f θ ).
(5.3)

Chapter 5. Wasserstein Adversarial Regularization for learning with noisy labels

We write the R AR regularization as:

R AR (x, f θ ) = D KL (f θ (x + r a ), f θ (x)),
where r a = argmax r, r ≤ρ

D KL (f θ (x + r), f θ (x)).
(5.4)

We have that:

D KL (f θ (x + r a ), f θ (x)) = c f θ (x + r a ) (c) log f θ (x + r a ) (c) f θ (x) (c) = c f θ (x + r a ) (c) log f θ (x + r a ) (c) - c f θ (x + r a ) (c) log f θ (x) (c) = - c f θ (x + r a ) (c) log f θ (x) (c) -H(f θ (x + r a )). (5.5)
where H is the entropy function. Consequently, the total loss can be rewritten as:

L tot (x, y, f θ ) = - c (y (c) + ηf θ (x + r a ) (c) ) log f θ (x) (c) -ηH(f θ (x + r a )) (5.6) Here η ∈ R + . Let η = ε 1-ε with ε ∈ [0, 1[. We have the following equivalence: (1 -ε)L tot (x, y, f θ ) = - c ((1 -ε)y (c) + εf θ (x + r a ) (c) ) log f θ (x) (c) -εH(f θ (x + r a )) = L CE (f θ (x), (1 -ε)y + εf θ (x + r a )) Interpolated label -ε H(f θ (x + r a ))
adversarial label entropy (5.7) This leads to the following interpretation: instead of learning over the exact label or over a mix between the exact label and the network prediction, we learn over an interpolation between the data y i and the adversarial label y a i , while maximizing the entropy of the adversarial label (i.e. blurring the boundaries of the classifier). Note that label smoothing techniques are known to create cluster of classes in the penultimate layer of the neural network [START_REF] Müller | [END_REF]], thus leading to appealing behaviour in the presence of noisy labels. Related developments can be found in adversarial label smoothing (ALS) [START_REF] Shafahi | [END_REF][START_REF] Goibert | [END_REF], which aims at providing robustness against adversarial attacks.

Yet, one of the major limit of this approach is that the regularization is conducted with the same magnitude between all classes without considering potential class similarities. As a consequence, a strong regularization can remove the label noise, but also hinder the ability of the classifier to separate similar classes where a complex boundary is needed. In the next section we present how we overcome this issue. We propose to replace D by a geometry-aware divergence taking into account the specific relationships between the classes for the classifier to not overfit on noisy labels when the classes are significantly different and to have complex boundaries when they are similar. Taking these relations into account avoid overfitting on noisy labels between dissimilar class, while allowing complex boundaries between classes with high similarities.

Wasserstein adversarial regularization to consider class similarities

To make the divergence aware of specific relationships between classes, we replace the "isotropic" divergence D with an optimal transport cost computed in the labels space. We name our proposed method Wasserstein

Wasserstein Adversarial Regularization for label noise 55

Adversarial Regularization. In a related way, [Frogner 2015] used the Wasserstein distance as a loss in a learning system between the output of the model for multi-label learning as detailed in Section 3.1.1. In this setting, OT can be computed because in multi-class problem, the classifier output and the label are probability measures. Note that the computational complexity of exact OT scales with respect to the number of classes instead of the number of data as main OT applications, i.e., the complexity is

O(n 3 C log(n C )).
In classical learning with noisy labels benchmarks, the number of classes can go up to 1000 classes. The interest of optimal transport is to take into consideration the geometry of the label space. We define the proposed regularization term R WAR as follows:

R WAR (x i ) = L ε (f θ (x i + r a i ), f θ (x i ), C) with r a i = argmax ri, ri ≤ρ L ε (f θ (x i + r a i ), f θ (x i ), C).
(5.8)

In practice, we will use the solution of the sharp entropic variant of the optimal transport problem [Cuturi 2013[START_REF][END_REF]]:

L ε sharp (f θ (x i + r a i ), f θ (x i ), C) = Π * ε , C with Π * ε = argmin Π∈U (f θ (xi+r a i ),f θ (xi)) Π, C -εH(Π)
Using this regularized version has several advantages similar to the entropic-regularized OT that we recall: i) it lowers the computational complexity to near-quadratic [START_REF] Altschuler | [END_REF]], ii) when ε > 0 , it turns the problem into a strongly convex one, which makes the OT loss differentiable with respect to the input measures [Peyré 2019, proposition 4.6], iii) it allows to vectorize the computation of all Wasserstein distances in a batch, which is particularly appealing for training deep neural nets and finally iv) it is a better approximation than the entropic-regularized OT [START_REF][END_REF]]. Based on [START_REF] Genevay | [END_REF]], we use the AutoDiff framework, which approximates the derivative of this regularization with a fixed number of iterations of the Sinkhorn algorithm. We recall that using a large ε results in smoothing the loss and speeds up the Sinkhorn algorithm, but drives the solution away from the true OT solution. As such, we recommend to use a small ε to stay as close as possible to the true OT loss. Note that for all the experiments in the main task, we have used same regularization parameter (ε = 0.05) with 20 Sinkhorn iterations, to illustrate that, while the regularization parameter is important, it is not too sensitive across datasets. To summarize, for each data in a batch, we compute the OT between the input data network prediction and the adversarial input data network prediction. Hence the OT complexity is of order

O(m × n s × n 2 C )
, where m is the batch size, n s the computation budget and n C the number of classes. We compare the computation time of our method in the experimental section. The full model update using WAR can be found in Algorithm 3.

Choice of ground cost. The ground cost C reflects the geometry of the label space. It bridges the gap between AR and WAR. An uninformative 0-1 ground cost, i.e. 0 over the diagonal and 1 everywhere else, would give the total variation (TV) loss (Remark 2.26 in [START_REF] Peyré | [END_REF]), which could also be used as D in the AR framework. Below, we refer to this special case as WAR 0-1 . To define a C matrix encoding class relationships, multiple choices are possible. Relying on expert knowledge, one could set it manually, but this becomes unpractical when a large number of classes is present, and this knowledge is not always available, or even prone to errors. We propose two variants to estimate the ground cost:

• In absence of prior information about the nature of the source of labelling errors, we first propose to rely on semantic distances based on word embeddings such as word2vec [START_REF] Mikolov | [END_REF]

r = ξ * d/ d 2 6: d ←-∇ r L ε ( fθt (x), f θt (x + r), C)/ξ 7: end for 8: r a = ρd/ d 2 9: L WAR (x i , f θt ) = L ε ( fθt (x), f θt (x + r a ), C) 10: return θ t+1 = θ t -γ∇ θt L CE (f θt (x i ), y i ) + ηL WAR (x i , f θt )
between classes are then defined via Euclidean distances in the embedding space, as proposed in [Frogner 2015]. Finally, as our method requires large values of the cost between similar classes, we apply the function e -δ (where δ is the Euclidean distance between the two class names) element-wise and set the diagonal of C to 0. We denote it WAR w2v ;

• Our second ground cost computation relies on the distance between mean of embedded data. We use a pre-trained neural network (details given in experimental section) to embed data and calculate the distance between mean of each class. The rationale is that two close classes, in terms of their mean distances, should be harder to distinguish, and as such the method should allow for a complex boundary to discriminate them. We denote it WAR embed .

While both options are only approximations of the difficulty to discriminate two classes, we show in the experimental section that they nonetheless provide better results than the uninformative 0-1 cost and random standard normal ground cost matrices. Note that we considered pre-calculated and static ground costs. Updating dynamically the ground cost might be an interesting research direction as future work.

Other application dependent options could be designed with respect to the problem at hand. We finally note that estimating the ground cost C could be very interesting but, despite some recent progress in metric learning for OT [START_REF] Cuturi | [END_REF], Huang 2016[START_REF] Li | [END_REF], adapting them for WAR is a difficult task, and it would be prone to overfitting due to the presence of label noise in the data.

Function smoothness and ground metric. Now we discuss how the proposed regularization term regularizes the model f θ with a smoothness controlled by the ground metric C. It is not possible to extend the label smoothing in Proposition 6 to WAR because the OT cost does not admit a close form solution, but we can still show how R WAR promotes label smoothness. To this end, we look at the regularization term L ε sharp ( fθ (x), f θ (x + r), C) for a given sample x and a pre-computed r. We can prove the following proposition:

Proposition 7. Minimizing R WAR with a symmetric cost C such that C i,i = 0, ∀i is equivalent to minimizing an upper bound of a weighted total variation (TV) norm between f θ (x) and f θ (x + r). (5.9) where c k = min i,i =k c k,i is the minimal off-diagonal cost for row k of C and c = min k c k is a global minimum out of the diagonal.

cT V (f θ (x), f θ (x + r)) ≤ k c k |f θ (x) k -f θ (x + r) k | ≤ L ε sharp (f θ (x), f θ (x + r), C),

Wasserstein Adversarial Regularization for label noise 57

Proof. We aim to prove the following relations for two probability measures α and β that

cT V (α, β) ≤ i c i |a i -b i | ≤ L ε sharp (α, β, c).
It is well known and obvious from [Cuturi 2013[START_REF][END_REF]] that the optimal OT matrix of regularized OT, Π ε , leads to a larger OT loss than the exact OT solution Π . This means that

W C (α, β) = Π , C ≤ Π ε , C = L ε sharp (α, β, c) (5.10)
and the relation is strict when ε > 0. Now if we suppose that the cost matrix is symmetric and C i,i = 0 and C i,j > 0 when i = j then solving OT means that the maximum amount of mass on the diagonal of Π since it leads to a 0 cost. Under constraints U (α, β) this maximum amount is equal to Π i,i = min(a i , b i ), ∀i. This implies that for a given row i in Π the amount of mass not on the diagonal row i is j =i Π i,j = max(a i -b i , 0) because of the left marginal constraint in U . Note that a similar result can be expressed with the column j such that the mass not on the diagonal of column j is i =j Π i,j = max(b j -a j , 0). This obviously means that for a given column/row index k we have

i =k Π i,k + Π k,i = |a k -b k |. Let's write A k = i≥k,j≥k T i,j C i,j . We have that W C (α, β) = A 1 . Now we remark that A k -A k+1 ≥ c k |a k -b k |.
We can write that

A 1 = A 1 - k=2 A k + k=2 A k . Since A N = 0 because C N,N = 0, it turns out that A 1 = k=1 (A k -A k+1 ).
Lower bounding every elements of the sum by the previous minoration gives that: (5.11) which gives the desired results.

A 1 = k=1 (A k -A k+1 ) ≥ k=1 c k |a k -b k | ≥ cT V (α, β),
By minimizing the proposed R WAR regularization with r belonging in a small ball around x, we actually minimize a local approximation of the Lipschitz constant of f θ . This has the effect of smoothingout the model around x and makes it more robust to label noise. One can see the effect of the cost matrix in the center term of (5.9), where the values in the ground metric correspond to a weighting of a total variation, hence controlling the effect of the regularization and the adversarial direction r during adversarial computation. Interestingly, the Wasserstein distance can be bounded both below (c) and above (c) by Total Variation and weighted total variation similarly to the equation above. Finally, in practice we minimize the expectation of the OT loss, which means that we will penalize areas of high density similarly to a regularization with the Sobolev norm (i.e. penalizing the expected norm of the model gradient [Mroueh 2018]), while keeping a finer control of the class relations, since we use the ground loss C that promotes anisotropy. After demonstrating and discussing the theoretical advantages of our method, we illustrate it on a toy experiment.

Illustration of the effect of R WAR We illustrate AR and WAR in a simple toy 3-classes classification problem using the Scikit-learn library [Pedregosa 2011a] with noise in regularization. From left to right, we compare the effect of training with the cross entropy alone (CCE, no regularization), R WAR with ε = 0.1 and ε = 0.05, as well as R AR with TV (that is the same as R WAR with 0 -1 loss), KL (as used in VAT [Miyato 2018a]) and L2 divergences as D. For the classification problem, we generated two close classes (in orange and red), as well as a third (in black), which is far from the others. Then, we introduced noisy labels (of the black class) in the region of the red class.

On this toy example, CCE overfits the noisy black labels, yet is able to distinguish the red and orange classes. The R AR regularizers, being class agnostic, correct for the noisy black labels in the bottom part, but smooth the complex decision function between the orange and red classes. On the contrary, R WAR uses a different cost per pair of classes, illustrated in the top left panel of Figure 5.1: the smallest cost is set between the red and black classes, which has the effect of promoting adversarial examples in that direction. This cost / smoothing relation is due to the fact that our problem is a minimization of the OT loss: in other words, the higher the cost between the classes, the less the binary decision boundary will be smoothed. Finally, the effect of the global ε parameter can also be appreciated in the classification results: while using ε = 0.05, the smoothing of the loss is decreased and the final decision boundary between the mixed classes keeps all its complexity.

Numerical experiments on learning with noisy labels and openset noise

We evaluate the proposed approach WAR on both image classification and semantic segmentation tasks. We first showcase the performance of WAR on a series of image classification benchmarks (Section 5.3.1), and then consider two real-world cases: first, the classification of clothing images from online shopping websites (Section 5.3.2) and then the semantic segmentation of land use in sub-decimeter resolution aerial images (Section 5.3.3). We also evaluated our method in the context of openset noise which consists to have image whom labels are not encoded, for instance non digit images in the MNIST dataset (Section 5.3.4). Following the community evaluation strategies, we used WAR on four different architectures depending on

Numerical experiments on learning with noisy labels and openset noise 59

benchmarks. On each benchmark, WAR outperformed state-of-the-art competitors showing the robustness of our methods on the used architectures. The code can be found here * . Datasets and noisy labels simulations We consider three image classification benchmark datasets: Fashion-MNIST (F-M) [START_REF][END_REF]], and CIFAR-10 / CIFAR-100 [Krizhevsky 2009]. Fashion-MNIST consists of 60 000 gray scale images of size 28 × 28 with 10 classes. CIFAR-10 and CIFAR-100 consist of 50 000 color images of size 32 × 32 covering 10 and 100 classes, respectively. Each dataset also contains 10 000 test images with balanced classes. Since we want to evaluate robustness to noisy labels, we simulated label noise in the training data only. For all datasets, we introduced 0%, 20% and 40% of noise in the labels. We considered only asymmetric noise, a class-conditional label noise where each label y i in the training set is flipped into y j with probability P i,j . As described above, asymmetric noise is more common in real-world scenarios than symmetric noise, where the labels are flipped uniformly over all the classes. For CIFAR-10 and CIFAR-100, we follow the asymmetric noise simulation setting by [START_REF] Patrini | [END_REF], where class labels are swapped only among similar classes with probability p (i.e. the noise level). For Fashion-MNIST, we visually inspected the similarity between classes on a t-SNE plot of the activations of the model trained on clean data; we then swapped labels between overlapping classes (→: one-directional swap, ↔ mutual Model Similarly to other works [START_REF] Han | [END_REF], Miyato 2018a], we used a 9 layer CNN for the three image classification benchmark datasets: Fashion-MNIST, CIFAR10, and CIFAR100 as shown in Table 5.3.

Image classification on simulated benchmark datasets

Between each layer we use a batch norm layer, a drop-out layer and a leaky-relu activation function with slope of 0.01. We use the Adam optimizer for all our networks with an initial learning rate of 0.001 with coefficient (β 1 , β 2 ) = (0.9, 0.999) and with mini-batch size of 256. The learning rate is divided by 10 after epochs 20 and 40 for Fashion-MNIST (60 epochs in total), after epochs 40 and 80 for CIFAR-10 (120 epochs in total), and after epochs 80 and 120 for CIFAR-100 (150 epochs in total). While training WAR, we set η = 0 for 15 epochs for faster convergence, as we observed that the network does not overfit on noisy labels at early stages of training. The input images are scaled between [-1, 1] for Fashion-MNIST, and mean subtracted for the CIFAR10, and CIFAR100 datasets before feeding into the network. The proposed method WAR, AR, and cross entropy loss functions are implemented in PyTorch, and for the JoCoR † , Pencil ‡ , Co-teaching § , Co-teaching+ ¶ , SL ‖ method we used the PyTorch code provided by the authors. For the rest of the state-of-the-art methods (dimensionality driven learning * * , forward, backward loss correction and robust loss functions † † : unhinged and boot strapping) the experiments are conducted using the Keras code provided by respective authors. We used similar layer initialization for all the methods in Pytorch and Keras.

For WAR w2v , we set the hyper-parameters η= 10, ε= 0.05, and ρ= 0.005 for all the datasets. The hyper-parameters of the baselines are set according to their original papers. The noise transition matrix for the Forward and Backward method is estimated from the model trained with cross entropy [START_REF] Patrini | [END_REF]].

Results

Classification accuracies are reported in Table 5.1. Results show that WAR w2v consistently outperforms the competitors by large margins, across noise levels and datasets. In particular, WAR w2v achieved improvements of 2-3% points on fashion-MNIST/CIFAR-10, and 15% on CIFAR-100 at the highest noise level. This demonstrates that the inclusion of class geometric information during training mitigates the effect of over-fitting to noisy labels. On the most noisy datasets, the most competitive method with WAR w2v is Pencil, however Pencil tends to decrease the performance when the dataset has a smaller percentage of noisy labels which is not the case of our method. Besides WAR w2v and Pencil, JoCoR, Unhinged, Co-Teaching, and SL also performed well. The Forward and Backward method performed slightly better than CCE, which is most likely due to the burden in accurately estimating the noise transition matrix. It is noted that Co-Teaching uses true noise estimate, and the accuracy might drop if the noise ratio is estimated directly from the noisy data. Furthermore, performance of Co-Teaching+ ‡ ‡ is surprising lower than the one of Co-Teaching on two datasets, in contrast to the observations in [START_REF] Yu | [END_REF]]. From our experiments, we observed that Co-Teaching+ underperforms when the noise is class-dependent and the model considered has a wide capacity.

Importance of encoding class relationships to better assess the significance of including class relationships and to study informative cost matrices, we compare WAR in three settings: 1) WAR 0-1 : a version of WAR with an uninformative 0-1 ground cost (the cost matrix is a matrix of ones, except for the diagonal); 2) WAR w2v : the version of WAR with the ground cost defined by word2vec embedding (as in all other experiments in this chapter); and 3) WAR embed : a version of WAR with a ground cost defined by similarity across classes obtained by off the shelf CNNs.

WAR embed is computed as follows: After embedding our data with a pre-trained ResNet18 [START_REF] He | [END_REF]], we compute each class centroid (which is an imperfect statistic, as belonging to one class is determined from potentially noisy labels), then we compute the distance between them. Finally, we apply the same exponential trick as for WAR w2v . The intuition behind WAR embed is the following: when classes are close (similar) in a relevant embedding, the noisy labels will lead to closer class centroids. Here the pretrained neural network can be seen as generic feature extractor used to assess class similarity only, similarly to [START_REF] Damodaran | [END_REF][START_REF] Damodaran | [END_REF]. Note that for Fashion-MNIST, we computed the centroids directly in the original data space. We also compared the performance of our ground costs against 20 standard normal cost matrices drawn randomly: in all cases, our proposed ground costs were ranked at the first and second positions, showing the relevance to have a good crafted ground cost. Finally, we also compare these results against AR using KL divergence. We used η=5 and ρ = 0.005 (similarly to WAR approaches), and followed the WAR training procedure. and shows that WAR w2v is consistently better than AR and WAR 0-1 (except in one case), and outperformed AR significantly by a 2-3% margin at the highest noise level. When looking at the two encodings of the C matrix, WAR embed and WAR w2v results are on par most of the time with the word2vec encoding showing better performances on Fashion-MNIST and CIFAR-100 and the CNN embedding being slightly better on CIFAR-10. These results show that both embeddings are senseful and provide good priors against the label noise. We found the word2vec embedding a good choice in our experiments, especially because they can be applied widely according to the semantics of the classes, while WAR embed needs access to a meaningful pre-trained model, which can be complicated depending on the problem (e.g. for semantic segmentation or other less mainstream tasks than image classification). Regarding computation time, all experiments were performed on a single GPU GTX TITAN. For one epoch on the CIFAR10 dataset, WAR embed took 187 seconds, AR took 165 seconds and the usual cross-entropy loss took 57 seconds. We recall that in WAR the quadratic complexity of the Wasserstein distance [START_REF] Altschuler | [END_REF]] only applies to the number of classes and stays of linear complexity w.r.t. the number of samples, hence the effect of WAR to the computational burden is relatively small. Thus the replacement of the KL divergence by the Wasserstein distance is of the same order of complexity. To demonstrate this point, we trained WAR and AR on the imagenet32 dataset with clean labels and the 9 layer CNN network. This dataset has 1000 classes and more than 1.2M 32 × 32 images. On average per epoch, the AR method took 1409 seconds, while WAR method took 1712 seconds, so it is only 20% more computationally expensive.

Sensitivity analysis of η:

We conducted an experimental study to analysis the sensitivity of the trade-off parameter (η) between the cross entropy and the adversarial regularization term on CIFAR-10 dataset with 40% noise level. The experimental results with different values of η are shown in Table 5.4 and the result reveals that as η increases, AR, WAR embed and WAR w2v are robust to the label noise. However for the higher η, AR and WAR embed do over-smoothing and decreases the classification accuracy. On the other hand, WAR w2v increases the accuracy as η increases. This behaviour shows the capability of our proposed method WAR w2v to preserve the discrimination capability between similar classes. It is noted that WAR w2v points the gradient direction towards the low cost classes, as a result it does not over smooths between the conflicting classes, thus maintaining the discrimination ability. Furthermore, WAR 0-1 with uninformative ground cost did not provide better results, and it is mostly similar with different values of η. This observation reinstates need of having meaningful ground cost to capture the relationship between the classes in the dataset, and to guide gradient direction with respect to the ground cost.

In this section, we evaluated our method on standard benchmarks where we simulated the label noise ourselves. We also reviewed the proof of concept on the importance to choose a good ground cost and made a sensitivity analysis. In the next section, we evaluate our method on a real-world dataset which contains label noise.

Image classification on real-world noisy label benchmark datasets

Dataset In this section, we demonstrate the robustness of WAR w2v on a large scale real-world noisy label dataset, Clothing1M [Xiao 2015]. The Clothing1M dataset contains 1 million images of clothing obtained from online shopping websites and has 14 classes. The labels have been obtained from text surrounding the images and are thus extremely noisy. The overall accuracy of the labels has been estimated to ≈61.54%.

The dataset also contains additional manually refined clean data for training (50k samples), validation (14k) and testing (10k). We did not use the clean training and validation data in this work, as WAR w2v assumes clean labels are unavailable (which is generally a more realistic assumption). Nonetheless, we report the results obtained using the model from the epoch with the lowest validation error, calculated on the clean validation set (last column of Table 5.5) to show how close to it our unsupervised strategy can get and to compare with recent methods (e.g. [Tanaka 2018b]). Clean test data was only used to evaluate the performance of the different approaches when learning with label noise.

Experimental setup and Results

Similar to [START_REF] Patrini | [END_REF][START_REF] Wang | [END_REF], we used ResNet-50 pre-trained on ImageNet for a fair comparison between methods. For WAR w2v , the hyperparameters are similar to those of the previous experiment, except ρ = 0.5. Regarding the training procedure for the Clothing1M dataset we give the following details. Data pre-processing includes resizing the image to 256 x 256, center cropping a 224 x 224 patch from the resized image, and performing mean subtraction. We used a batch size of 64 and learning rate of 0.002 to update the network with Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9, and weight decay of 0.001. The learning rate is divided by 10 after 5 epochs (10 epochs in total). We set ρ = 0.5 and divided by 10 (after 5 epochs) when the training loss is not decreasing. For [Kun 2019b], the first stage was for three epochs with step size of 2e -3 , the second for 6 epochs with step size 2e -4 , and the last stage was for 6 epochs with step size 2e -5 . For JoCoR [Wei 2020], we got better results by setting the learning rates as follows: we use the step size 2e -3 for the three first epochs, then we divided it by 10 for 6 epochs and finally used 2e -5 for the remaining 6 epochs.

For fairness of comparison, we only selected the methods which are similar to ours (robust loss functions), and which have the similar training methods (optimizer, learning rate, batchsize, epochs) and architectures. We compared WAR w2v against the competitors from [Patrini 2017, Wang 2019], and also reproduced the CCE accuracy by our own experiments. Results are reported in Table 5.5 and our method achieved the highest performance compared to all the baselines. Note that Forward uses a mix of noisy and clean labels to estimate the noise transition matrix, while JOF and WAR w2v ( †) use clean validation labels only to report the test accuracy with respect to the best validation score. This shows the competitive advantage of our method.

In this section we evaluated WAR on a real-world clothing images where it achieved state-of-the-art results. In the next section, we applied our method on semantic segmentation with real-world remote sensing data.

Chapter 5. Wasserstein Adversarial Regularization for learning with noisy labels

Semantic segmentation of aerial images

Datasets and noisy labels simulations In this experiment we consider the task of assigning every pixel of an aerial image to an urban land use category. We use a widely used remote sensing benchmark, the ISPRS Vaihingen semantic labeling dataset § § . The data consist of 33 tiles (of varying sizes, for a total of 168 287 871 pixels) acquired by an aircraft at the ground resolution of 9cm. The images are true orthophotos with three spectral channels (near infrared, red, green). A digital surface model (DSM) and a normalized digital surface model (nDSM) are also available, making the input space 5-dimensional. Among the 33 tiles, we used the initial data split (11 tiles for training, 5 for validation and 17 for testing). As ground truth, six land cover classes (impervious surfaces, building, low vegetation, tree, car, background/clutter) are densely annotated.

We simulated label noise by swapping labels at the object level rather than flipping single pixels. An object is the connected component of pixels sharing the same label. We also focused on plausible labeling errors: for instance, a car could be mislabeled to an impervious surface, but not to a building or a tree. § § http://www2.isprs.org/commissions/comm3/wg4/2d-sem-label-vaihingen.html Following this methodology, a third of the connected components had the label flipped. An example of the corrupted label data is shown in Figure 5.2.

Numerical experiments on learning with noisy labels and openset noise 65

Model

We used a U-Net architecture [Ronneberger 2015], modified to take the 5 channels input data as inputs. The network was trained for 300 epochs (with 90 • , 180 • or 270 • rotations and vertical or horizontal flips as data augmentation) using the Adam optimizer with an initial learning rate of 10 -4 and coefficients (β 1 , β 2 ) = (0.9, 0.999). After 10 epochs, the learning rate is set to 10 -5 . Furthermore, we predict on the full image using overlapping patches (200 pixels overlap) averaged according to a Gaussian kernel centered in the middle of the patch (σ = 1). This architecture has been shown to perform well on the semantic segmentation task in general, and on this dataset in particular [START_REF] Xu | [END_REF]]. Using this methodology, we obtain an overall accuracy on the clean data of 83.89%, which is close to the state-of-the-art for this dataset.

Results

We compare WAR w2v with standard CCE, Bootsoft, Co-Teaching and AR. The results, computed on the full test ground truth (including boundaries) and averaged over 2 runs, are reported in Table 5.6. Note that the classes are unbalanced and, for most of them, the F1-score is improved using WAR w2v , except for the dominant class (impervious surfaces). This leads to a much higher average F1-score using WAR w2v (compared to its competitors), while the overall accuracy is only slightly decreased compared to Bootsoft. This behavior can be seen in the maps shown in Figure 5.3. We can see in the close-ups that Bootsoft performs poorly in detecting the cars, which are often confused with generic impervious surfaces.

We applied our method for semantic segmentation on remote sensing data. In the next section, we make a final evaluation on image classification with open set noise.

Image classification with open set noisy labels

As a final benchmark, following [Wang 2018c[START_REF] Yu | [END_REF] we evaluate our proposed method on realistic open set noisy labels scenarios. The open set noisy datasets are created by replacing some training images with out of domain images, while keeping the labels and number of images per class unchanged. In our experiments, we simulated open set noisy datasets following [Wang 2018c[START_REF] Yu | [END_REF] for CIFAR 10 dataset by replacing images with others coming either from the SVHN or ImageNet32 (images of size 32×32) [START_REF] Oord | [END_REF]] datasets. Experimental set-up and Results. For the openset noise experiments, we used a 6 layers CNN with one fully connected layer following [Wang 2018c[START_REF] Yu | [END_REF]] (see Table 5.8). Between each convolutional layer we used a batch normalization layer and a momentum of 0.1. The images are scaled in the range [-1, 1] and the networks are trained by SGD with learning rate 0.01, weight decay 10 -4 and momentum 0.9. The learning rate is divided by 10 after 40 and 80 epochs (100 in total). Table 5.7 reports the classification accuracy on the CIFAR-10 dataset with 40% of openset noise. The experimental setting is similar to [Wang 2018c] in order to ensure a fair comparison with the state-of-the-art methods. The scores for the Iterative and Forward method are from [Wang 2018c], while for other competing methods we report our replicated scores. Our method outperformed the competitors on both openset noisy datasets. Our method is significantly better than all competitors for openset samples from SVHN, while for ImageNet32 openset samples we are slightly better than Co-teaching, and significantly better than remaining methods. Note that our method does not need to reject openset samples on contrary to competitors.

Chapter 5. Wasserstein Adversarial Regularization for learning with noisy labels

Conclusion

In this chapter, we introduced a regularization based on optimal transport in order to learn in a noisy label setting. Noisy labels often occur in a context of supervised learning where labels of given images do not encode their classes correctly. These noisy labels impact negatively the learning of neural networks as they have huge memorization abilities [START_REF] Zhang | [END_REF]]. We show that using the VAT regularization is effective to counter-balance the label noise influence on the neural networks. However when two classes are similar, we want to modulate the regularization to keep complex boundaries. We achieved this by using optimal transport as our divergence in the regularization and it is called Wasserstein Adversarial Training. To encode the class similarities, we used the distance between the word2vec class representations or the distance between embedded data of different classes. We have evaluated our methods on standard benchmarks, two real-world datasets and in an open set setting.

Chapter 6

Generating natural adversarial Remote Sensing Images Contents In this chapter, we present how to adapt a Wasserstein GAN in order to generate natural examples of a given pre-trained classifier. The considered setup is the following: our intent is to generate untargeted natural adversarial examples in a black box scenario, which is detailed in Section 4.1.3. We first review how GANs were used to generate remote sensing images, in what purposes and how they can be used to generate adversarial examples. We then present our main contribution: a reweighted discrete measure fed to a Wasserstein GAN to generate natural examples. The reweighted measure gives bigger weights to misclassified samples by the given classifier. We present several reweighting strategies and discuss their pros and cons. We finish this chapter by presenting the performance of our reweighted measure to generate adversarial examples on real-world remote sensing data. These contributions have been published to IEEE Transactions on Geoscience and Remote Sensing [Burnel 2021].

GANs for remote sensing images

Generative modelling for remote sensing data has been investigated in several references. GANs were used to generate such data in [START_REF] Audebert | [END_REF]]. They used a conditional GAN architecture, where they take as inputs a hyperspectral class and a random noise, to control the class of generated data. They showed that the generated spectra quality is genuine-looking and physically plausible. Furthermore, they experimentally validated that the generated samples can be used for data augmentation strategy to improve classification accuracy. The generation of super resolved remote sensing data was investigated Chapter 6. Generating natural adversarial Remote Sensing Images in [START_REF] Jiang | [END_REF]]. To this end; they relied on two main subnetworks combined with an adversarial learning strategy. GANs have also been used to learn unsupervised representations as done in [Lin 2017]. They used the generator to generate data-like images and the discriminator was used as a feature extractor for classification purposes. [START_REF] Duan | [END_REF]] also used GANs for learning representation by incorporating a non-local layer into their architecture. GANs have also been used in remote sensing for domain adaptation on hetereogenous data [START_REF] Voreiter | [END_REF]], which might be seen as transfer learning with different type of data.

In the next section we introduce our main contribution. We propose to reweight the empirical measure in order to generate adversarial data. We then discuss the different reweighting strategies and their pros and cons.

Adversarial Reweighting WGAN

We recall the notations. As we have only access to n samples x ∈ X of an unknown measure α, we mostly deal with underlying probability measures. The measure associated to real and training data is denoted

P r ∈ M + 1 (X )
and the generated data distribution as

P G ∈ M + 1 (X )
. G is a deterministic generator, classically expressed as a neural network, that takes as input a random vector z ∼ N (0 q , I q ), where q is the dimension of this random variable. We generally assume that q d, with d the dimensionality of the training data.

Adversarial Reweighting WGAN

To generate data similar to the training data, we use an empirical distribution with uniform weight P r in the classical GAN training, i.e P r = 1/n n i δ xi . It is customary for empirical distributions to suppose that the samples are drawn i.i.d., from the underlying distribution. One solution to change the GAN goal is to change the empirical distribution P r . Instead of encouraging the generator to generate data close to the true data, our method, called ARWGAN, encourages the generator to generate realistic data that make the classifier's prediction fail. A natural solution to reach this goal is to give a bigger weight in the distribution to true misclassified data than to those that are correctly classified. The resulting weighted distribution, denoted P a r , must have the following form: P a r = n i=1 p(x i )δ xi , where:

n i=1 p(x i ) = 1.
The GAN is trained to minimize the Wasserstein distance between the generator distribution P G and the reweighted true distribution P a r . Finally, our new loss function is of the form:

min ψ max φ E x∼ P a r [D φ (x)] -E z∼Pz [D φ (G ψ (z))] + λE x∼ P x [(max{0, ∇D φ ( x) 2 -1}) 2 ]. (6.1)
Intuitively, the generator is a map between the latent space and the adversarial data area of the true distribution. To the best of our knowledge, it is the first time the distribution is modified for generative purpose. Now we describe and review the impact of different reweighting methods.

Several flavors of reweighting

Several possibilities exist to modify the empirical distribution of the true data. We give a list of the different weighting strategies for p(x). We start by the considering only misclassified data.

Soft weighting

Another weighting approach is to use the prediction score from the classifier. Such approach could take into account examples which are correctly classified but with a low confidence. We refer to those samples as soft adversarial examples. In order to consider the soft adversarial examples, we can use a softmax function. It is defined as:

S(f y θ (x), w) exp(w * [f y θ (x) -f y θ (x) max ]) K i=0 exp(w * [f y θ (x i ) -f y θ (x) max ]) , ( 6.2)
where f y θ (x) is the prediction to belong to the correct class, f y θ (x) max is the maximal probability among the batch of data to belong to the correct class and w is a temperature coefficient that controls the entropy of the resulting distribution. However as the objective is to generate adversarial data, we consider 1 -f y θ (x) rather than f y θ (x). The soft weighting strategy can be expressed as p soft (x) S(1 -f y θ (x), w). Figure 1.(c) is an example of weighting using the softmax function with a temperature w = 5. Note that in the case of w = 0, we recover the original WGAN and when w tends to infinity, we recover the hard weighting strategy.

Weight clipping. In order to make the softmax strategy more robust to potential outliers in the dataset, we propose to apply a clipping function to all prediction vectors f θ (x). We denote the threshold κ and clip the prediction vector f θ (x) to this maximum value κ. The clipping function is applied to the prediction vectors before the softmax strategy. In our experiments, we selected a threshold of 75%. Putting all together, the weighted distribution is :

p clip (x) S(min(κ, (1 -f y θ (x)), w). (6.3)
Example of weights. To demonstrate the relevance of the different reweighting methods, we use a toy dataset to illustrate the softmax approach. See Figure 1. (a) to (f), where we represent data from a certain class with points cloud, the classifier's decision boundary with a black line, so that points under the line are correctly classified and points over the line are misclassified. When we get closer and beyond the classification line, the point clouds become bigger with a bigger weight. However, this might be problematic when the training data is tainted by outliers. For instance, in the presence of few misclassified data with high inaccurate predictions and a majority of misclassified data with medium inaccurate prediction, the former get a disproportionately large weight as shown in Figure 1.(e). The effect of the threshold can be seen in Figure 1.(f). where the outlier now has a similar weight to the other misclassified samples.

Minibatch weighting

We numerically found that computing the reweighting directly on the full distributions gives numerical instabilities when batches with small weighted data are selected. A possible solution to this problem is to compute the reweighting strategy on batches from the distributions. In practice, we draw samples uniformly at random from the training images and make a reweighted distribution on this batch. We denote this distribution as P a r ⊗m . The loss function becomes an expectation over mini-batches:

min ψ max φ E x∼P a r ⊗m [D φ (x)] -E z∼Pz [D φ (G ψ (z))] + λE x∼P x [(max{0, ∇D φ ( x) 2 -1}) 2 ], (6.4)
where m is the minibatch size. As we have detailed our different reweighting strategies, we evaluate them on different scenarios in the next section.

Numerical experiments on generative modelling

In this section we describe the different experiments that were designed to evaluate ARWGAN on simulated and real data. Section 6.3.1 shows a simple example where ARWGAN is trained to fool a classifier on synthetic 2D data. The first experiment on real-life data, provided in Section 6.3.2, aims at fooling a classifier trained on hyperspectral images. In Section 6.3.2, we study the transferability of our adversarial generators across different classifiers. In Section 6.3.3 we focus on segmentation, and train ARWGAN models to modify realistically an image in order to fool the segmentation of a given model. This is done by modifying IRRGB images through a patch in order to make them adversarial. Finally in Section 6.3.4, we measure the capacity of ARWGAN to generate data which can fool the state-of-the-art detector YOLOV3. The code can be found here * .

For all experiments we use a RMSProp optimizer [Tieleman 2012b], with a learning rate of 0.00005 for the generator and 0.0001 for the critic. Experiments were done on a single RTX 2080 Ti GPU.

Adversarial generator for 2D data classification

The first experiment illustrates ARWGAN against a pre-trained classifier on the well known two moons toy dataset [Pedregosa 2011b]. We consider 4000 training samples. On this toy task, we want to generate data which belong to class 1 but are classified as belonging to class 2, i.e., the white point clouds in Figure 6.2. The classifier used is a pre-trained dense 2-layer with a classification accuracy of 92%. The classifier boundary is represented as a black line. The generator and discriminator share the same dense 3 hidden layers architecture of size 128, 128 and 64 respectively. The input noise dimension z ∼ P z is 10. The batch size is 256 and the networks is trained for 600 epochs. We then train both a WGAN and an ARWGAN to see in which zone they generate class 1 data. To visualize the generation area, 1500 samples are generated and a kernel density estimation of these data is performed. For ARWGAN, the temperature w is set to 20 and the clipping value κ is 0.5. We see that WGAN generates data all over the class 1. On the other hand, ARWGAN generates a majority of misclassified data and some correctly classified but close to the classifier boundaries. This illustrates the effectiveness of our method to focus on generating adversarial data for a pre-trained classifier. We now evaluate its performance on real-world remote sensing images.

Adversarial generator for hyperspectral data classification

In this subsection, we investigate the effectiveness of ARWGAN to fool a classifier trained on real-world hyperspectral data. Unlike traditional RGB images, hyperspectral imagery divides the color spectrum in multiple contiguous bands that can outreach the visible spectrum. In those images a pixel is represented by a spectrum, which can be used among others to identify the materials observed in this pixel. We consider two different hyperspectral images with two different classifiers. We first train a pixel classifier that takes a spectrum as input and outputs a material probability vector. We aim to fool the classifiers by generating spectra which are misclassified. In order to evaluate the pertinence of ARWGAN's generated data, we first check if the adversarial data are genuine-looking like the targeted class and then the percentage of correctly classified adversarial data. We evaluate in the first paragraph ARWGAN generated data against a Support vector Machine classifier trained on the Pavia University dataset. In a second paragraph, we consider a convolutional neural network on the Houston dataset. And in a third paragraph, we evaluate the transfer of adversarial examples on unseen classifiers.

Support Vector Machines on Pavia University

We first test our method against a classical SVM [START_REF] Boser | [END_REF][START_REF] Hearst | [END_REF]] classifier trained on the Pavia University hyperspectral dataset. The Pavia dataset has 9 classes, composed of hyperspectral images with 103 bands of size 610×340px, (see Figure 6.3). The SVM has an accuracy score of 84% and a κ = 0.79, see the confusion matrix for more details (Figure 6.4) spatially disjoint splitting with 1/3 of data for training, as shown in Figure 6.3.

Experimental setting. For this experiment, we focused on generating meadows spectra. We choose 1-dimensional convolution layers for both WGAN's generator and critic as they keep coherence between close spectral bands. The architecture used is detailed in Figure 6.5. We use the whole dataset to train ARWGAN (which does not require labels), with a batch size of 64, λ of 10 and w of 20. Regarding the classifier, its accuracy performance is 84% for the meadows spectra class.

Results. We first check if the generator is able to generate adversarial data that are close to the training data. We consider 1000 adversarial generated spectra. A majority is classified as Bare Soil (80%) and the remaining mostly as Trees (17%). With Figure 6.6 we present a comparison between Meadows spectra, our adversarial Meadows spectra used to reconstruct an image close to the original and the two most predicted classes: Bare Soil and Trees. Our generated spectra are very close to the targeted spectra and the two other spectra appear to be very different. This means that we successfully fool the classifier while having visually valid generated data. We also conduct a comparison with the method from [Song 2018] using the same network architecture and training procedure for comparison purpose. We reconstructed Pavia image from 1000 generated adversarial spectra for both methods in Figure 6.7. We see that while our generated adversarial spectra look like the targeted data it is not the case for generated data from [Song 2018] showing the success of our strategy.

We then check the accuracy of the pre-trained SVM classifier on the generated data. Results are gathered in Table 6.1. We can observe that 35% of ARWGAN's spectra are correctly classified while it goes up to 79% for WGAN's spectra, which confirms that ARWGAN generates a larger proportion of misclassified data.

We have conducted a sensitivity analysis of our temperature parameter w without using any ceiling and gathered the results in Table 6.2. As expected our method tends to generate more adversarial data for bigger temperature. Indeed, for a temperature w = 1, ARWGAN generates a few more adversarial example than WGAN, but when w is equal to 5, the generated adversarial example proportion doubles. For further understanding we show the result of using Hard weighting for the Pavia class painted metal sheets, and compare it with a soft weighting strategy. If we look at adversarial proportion for this class then the Hard weighting is able to generate 81% of adversarial example, while Soft weighting have a smaller rate with 18.6% and a WGANs have a rate of only 0.6%. However we illustrate in Figure 6.8 the limit of the Hard weighting strategy, that is overfitting with just a few modes of generated examples very similar to those in the dataset, while soft weighting is able to generate more diverse examples.

Deep neural networks on DFC2018

We now test ARWGAN on the DFC2018 dataset [grs ] [Xu 2019], which was acquired over the University of Houston campus and its neighbourhoods. The hyperspectral data cover a 380-1050 nm spectral range with 48 bands at a 1-m Ground Sampling Distance. Here classes do not only cover urban classes (buildings, cars, railways, etc.) but it also covers vegetation classes (healthy or stressed grass, artificial turf, etc.

). An overview of the dataset is given with Figure 6.9, where we only take 3 of the 48 bands for visualization purpose, and the ground truth labels are shown in the same figure. The considered classifier is based 76 Chapter 6. Generating natural adversarial Remote Sensing Images Experimental setting. For this experiment, we investigate different adversarial class spectra: adversarial healthy grass spectra, adversarial car spectra and adversarial cross-walk spectra. For the WGAN architecture, we choose 1-dimensional convolution layers for both the generator and the critic as done in the previous experiments. The architecture is detailed in Figure 6.10. We use the whole dataset with a batch size of 64, λ is set to 20 and in this experiment w is set to 20. Regarding the classifier performance, its accuracy on the whole dataset is 82% for the healthy grass spectra, 39% on the Cars class, and only 5% on the Crosswalks class. This allows us to discuss the quality of adversarial generated data according to the classifier performance on a specific class with different accuracies.

Results.

We create an adversarial image by generating adversarial spectra for all the pixels of the class we attack. A visualization of the adversarial image for healthy grass can be viewed in Figure 6.11 in false colors, this is the image we refer to in this section. The pre-trained classifier has an overall accuracy of 82% on real healthy grass spectra. On adversarial generated spectra, its performance decreases by 50% as shown in the top row of We now check the adversarial quality of generated spectra by comparing their means and standard deviations to the original data. Figure 6.12 gathers all spectra statistics for adversarial healthy grass generation. We compare the target class against the generated adversarial examples from ARWGAN, the generated adversarial examples from [Song 2018] and the two most predicted classes by the classifier (evergreen trees and stressed grass classes). We see that the ARWGAN adversarial spectra statistics match the targeted class better than the predicted class and state-of-the-art adversarial data generation. This confirms the competitiveness of our method to generate data similar to the training data.

We now investigate the performance of ARWGAN to generate adversarial cross-walk and cars spectra. The spectra statistics evaluation can be found in Figure 6.13. We compare the target class, ARWGAN generated data, the state-of-the-art adversarial data and the most predicted class by the classifier. We can see that ARWGAN generated data stays very similar to the target class. Regarding the classifier accuracy, we see in Table 6.3 that the classification accuracy of ARWGAN generated spectra is smaller than for real data, it decreases by 20% for the car class and 4% for the cross-walk class. We also see that there is a correlation between the classifier performance and the number of adversarial examples per batch. For high performing classifier, the number of misclassified data is really low and the GANs have few misclassified examples to be trained on.

Qualitative comparison with state-of-the-art. We now compare more deeply ARWGAN with [Song 2018] † . We adapted their method to use the same architectures considered here. We used untargeted attacks, which means that we want our spectra to be misclassified but we do not want it to be classified as a specific label. The results are visible with Figure 6.12,6.13. Regarding the Healthy grass class, the adversarial spectrum of [Song 2018] exhibits a lot of noise and does not fit the target class as well as ARWGAN generated spectra. Similarly for the Crosswalks class, their algorithm was not able to produce

Cross-model transferability: generalizing adversarial examples to unseen classifier

In this part we consider the question of transferring generated adversarial examples for a pre-trained classifier to attack an unseen classifier. In other words, we want to measure the ability of generated adversarial examples to fool a classifier that was not used during training.

Experimental setting. For this experiment we used two different neural networks, whose details are shown in Table 6.4, and an SVM classifier. While the two neural networks are built upon the same blocks (Dense and Convolutional layers), we can see that they do not share the same performances on the Healthy Grass class, meaning that we can expect that they learned different features. Classifier A has an accuracy of 80% on the Healthy grass class while the accuracy of classifier B is only of 60% on this class. Note that classifier B has a better overall accuracy. Classifier C is a SVM with a 52% overall accuracy and a 75% accuracy on Healthy Grass class. is a strong transfer of adversarial examples across classifiers. In the case of generators trained against Classifier A, we generate 77.8% of adversarial examples for all classifiers. This shows that our method can be used to attack unseen classifiers.

Results

In this section, we showed that ARWGAN can successfully generate hyperspectral data which genuinely look like training data and that are adversarial. In the next section, we demonstrate that ARWGAN can modify training images in order to make them adversarial.

Adversarial segmentation with modified mask images

GANs are able to generate images but they can also be used to modify existing images. In this segmentation experiment, we now focus on this last ability to modify realistically training data in order to fool the segmentation of a pre-trained model. In order to modify an image we rely on a mask denoted M , in the shape of a square at the center of an image. For an image to be both adversarial and realistic, the generator makes slight modifications inside the mask. Our images have pixels which are composed of Red-Green-Blue-InfraRed channels. To evaluate the performance of ARWGAN on this modification task, we compare the predicted segmentation of the original and the modified data and we want the original data to have a better segmentation. Furthermore we check that the modified images are still realistic by using human perceptual evaluations.

Numerical experiments on generative modelling 81

Dataset. We consider two datasets. First the Potsdam dataset [pot ] which is a dataset composed of 38 6000×6000 pixels patches over the city of Potsdam, Germany. The patches are true orthophotos with four channels red, green, blue, infrared and have a GSD of 5-cm, making it possible to see clearly objects such as cars. We split the 38 images as follows: 28 images are used as training set and the remaining 10 images as test set. In total, we have 9138 cars in the dataset and from the patches, small images of cars of size 128 by 128 were extracted using center of mass of individual cars on patches. Regarding the neural network, we use a segmentation network (SegNet) [Badrinarayanan 2017] as a pre-trained model. Its architecture is provided in Figure 6.14 and its performance on real data is available as a confusion matrix in Figure 6.15.

The second dataset we considered is the Vaihigen dataset [vai ]. The main differences with the Potsdam datasets are that images are RGB and have a GSD of 9-cm. We used 23 images as training set and the remaining 10 images as test set.

Experimentation. The car segmentation is done within a 64 by 64 mask in the center of the images. For the generator we used a U-Net [Ronneberger 2015] that takes as inputs both the image and the mask in addition to a latent vector z, then it outputs a single new image as shown in Figure 6.16. The generator works as follows: inputs are first compressed in latent information through convolutional and pool layers, then the resulting latent vector is decoded with transposed convolutional layers to give a single new image modified only inside the mask. Our modification task can be viewed as image inpainting and following this analogy, we designed a network close to [START_REF][END_REF]] which showed impressive results using dilated convolutions [START_REF] Yu | [END_REF]]. In this experiments, the modified loss function is:

L 2 (P a r ⊗m , P z , M , ψ, φ, θ) = E (x,y)∼P a r ⊗m D φ (x) -E z∼Pz [D φ (G ψ (x, M , z))] + λE x∼P x [(max{0, ∇D φ ( x) 2 -1}) 2 ] + η 1 m i=0 ((1 -M ) * (G(x, M , z) i -x i )) 2 -η 2 m i=0 (y * M ) log[(1 -f θ (x)) * M ]. (6.5)
We add a regularization term to the generator's loss in (6.5), to ensure that only the mask M is modified by penalizing the squared difference between unmasked original image and unmasked modified image and we also add a term to favor adversarial segmentation. The latter modification of the loss function, penalizes the GAN when it modified other pixels than the car pixels. Both of these terms are weighted by constants η 1 and η 2 , but taking η 1 = η 2 = 1 in the experiments gave meaningful results. We now give more details about the training procedure. The critic D φ is composed of 4 blocks of 3 convolutions each. Each block starts with a stride 2 convolution and each convolution has a kernel of size 3. Finally, we use a dense layer to get a scalar from the critic. We used a batch size of 32 and λ is set to 50.

Results

. Some examples from the test set are gathered in Figure 6.17. We see in the first row two different cases where ARWGAN performed well. In the first case few modifications of the image led to huge difference in segmentation, the differences for the RES column are in red if the car is not segmented anymore or in green if the car is better segmented.

In the second case we needed much more modification to erase the car from the segmentation. The second row shows two failure cases: a heavy modification of the input which results in a poor image quality and a modification which leads to a better segmentation, thus failing as an adversary. Note that these failure cases do not happen often. This can be seen in the perceptual evaluation, see Table 6.18, that we designed to investigate if the generated results are convincing and adversarial. We did not add a comparison with any method as, to the best of our knowledge, there is no existing method providing region based generation of natural adversarial examples.

We illustrate in Figure 6.18 our method on Vaihingen dataset. While our method can give nice results, we have noticed an unstability during training, suggestng that it might be important to validate η 2 in some applications with long training.

Perceptual evaluation. To assess the quality of our generated patches we conduct a perceptual evaluation where we compare the ability of both trained classifier and humans to classify adversarial samples. The classifier performance is evaluated using the test set, for both original and mask modified images. And as we can see with the first row of Table 6.6, the classifier looses 28 points of accuracy when it is evaluated on data from ARWGAN. However those results are only meaningful if those are real natural adversarial example as we defined it in the first section, meaning that the generated patches are indeed cars. To this end; we conducted a perceptual evaluation similar to [Song 2018], with the assumption that if a human can detect a car then its segmentation is trivial. We conducted the perceptual evaluation by first taking randomly 50 images where we had cars and 50 images where there were no cars, and then we applied our method to the 50 images with cars, letting us with 150 images in total. We have developed a simple test where 12 images are presented to humans. Among these images, 5 are ground truth cars, 5 were modified using our method and 2 do not have cars in them. In this test you must indicate whether you see a car or not and associate this decision with a confidence level. This test was carried out by 74 persons and the results are gathered in the second and third rows of Table 6.6. Interestingly humans loose 8 points of accuracy, however their confidence remains of the same order: Moderately confident. When we compare the results, we see a 36% drop of performance for the classifier against 9% for Humans, meaning that our method affected far more the classifier than the humans and that our method produces convincing adversarial examples. After demonstrating how to use ARWGAN to modify a training example to make it adversarial, we evaluate it against a state-of-the-art detector.

Adversarial car images for YOLOV3 detector

The last experiment consists in evaluating ARWGAN's performance for generating cars and fooling state-of-the-art detectors. Instead of semantic segmentation where each pixel has a label, the classifier output is bounding boxes surrounding an object. In this experiment, we solely focus on the cars class object, thus the classification task can be summarized as the detection of a car in images. We evaluate the percentage of misclassified data in the generated ARWGAN and WGAN data.

Dataset. We use the same Potsdam dataset [pot ] than in previous experiment. We transformed the Potsdam segmentation images to different images with bounding boxes around cars. To train the classifier we used the same train set as the patch modification experiment. However to train the GAN, we used both training and testing sets as done in our first experiment, which allows to increase the number of training data.

Experimentation. The selected detector is a YoloV3 [START_REF] Redmon | [END_REF]] that achieves an overall (Train and Test) accuracy score of 99.5% with an objectness threshold of 0.75. This task seems very easy for our classifier as we do not consider the Intersection Over Union. The generator architecture we consider contains three convolutional layers and one dense layer (details in Figure 6.19). We set λ to 50. We consider two different methods, we first train a WGAN and we evaluate its ability to generate adversarial data. Then we used the WGAN generator as initialization for our method ARWGAN, and evaluate the number of generated adversarial data. The critic in this experiment is the same as the previous experiment.

Results. state-of-the-art classifiers have high accuracy and only have a few natural adversarial examples making it hard to train our method. Nevertheless, using a pre-trained WGAN generator as initialization for our generator leads to a high number of generated adversarial data with good image quality. Example of natural adversarial car images can be found in Figure 6.20. We see that our generated adversarial images have a better quality than natural adversarial images from the ground truth, called adv. GT, or WGAN generated images. As seen in Table 6.7 the considered WGAN generates only 2.6% adversarial data while our methods improve its score by more than 5 points (more than 3 times more examples are adversarial), showing that even in extreme scenarios, our method is still able to generate adversarial data. Moreover Figure 6.20 shows that despite having few good looking natural adversarial examples in the ground truth, our method manages to generate better looking images.

Conclusion 87

Adv. generation rate std WGAN 2.6% ±0.94% Our method 7.8% ±2.5% As expected, the method of [Song 2018] failed to generate adversarial examples on this example, as the associated auxiliary classifier was always surpassed by YOLOV3 in terms of car detection. The auxiliary classifier has to detect if there is a car in the generated examples, which in practice led YoloV3 to also detect the presence of a car.

Conclusion

In this chapter, we introduced a novel method to generate adversarial data for a given pre-trained classifier. The purpose is to reweight the initial uniform measure and to give larger weights to misclassified data. Thus the WGAN is trained with the reweighted empirical measure and is encouraged to generate misclassified samples. We proposed several reweighting strategies. After describing the limits of a hard weighting strategy, we proposed a soft strategy based on a softmax function. We then extensively evaluated our method on remote sensing data. We started by generating hyperspectral data and demonstrated the realistic quality of the generated samples. They were both misclassified and statistically close to the Chapter 6. Generating natural adversarial Remote Sensing Images targeted class. Furthermore, they were not similar to data of the predicted class. We also evaluated the transfer of adversarial examples between seen and unseen classifiers and we show that a good proportion of ARWGAN adversarial examples can attack unseen classifiers. We then proposed to modify a given training remote sensing image in order to make it adversarial with a WGAN. Using a centered mask on the center of the image, ARWGAN modified what was in the mask in order to make the segmentation fail. Finally, we trained our method to fool a state-of-the-art detector on cars.

Unfortunately, this method can not be straightforwardly adapted to other OT based GANs. The reason is that other OT based GANs rely on the computation of optimal transport between minibatches of uniform empirical measures. This approximation is the topic of the final part of this thesis. When optimal transport is used as a loss function to train a neural network, it has to be computed at each iteration of the optimization procedure. In a big data scenario, the number of samples n is very large. Thus, the cubical computational complexity in n of exact optimal transport makes it prohibitive in practice. To use OT in practice, we can possibly rely on some OT variants. The entropic-regularized OT reduces the cubical complexity to quadratic, which is still expensive. Other strategies were developed to reach complexities close to a linear complexity. Multi-scale OT strategy is one of them [START_REF] Gerber | [END_REF]]. The multiscale decomposition yields a sequence of optimal transport problems, that are solved in a top-to-bottom fashion from the coarsest to the finest scale. It is approximately linear in time and memory in the number of nodes. Another line of research used ground costs computed between well chosen positive features, it reduces the cost of a Sinkhorn iteration to a linear computation [START_REF] Scetbon | [END_REF]]. We can also build low rank transport plan as done in [Forrow 2019[START_REF] Scetbon | [END_REF].

Part III

Theory and applications of Minibatch Optimal Transport

In this chapter, we discuss another practical solution for large-scale OT. We introduce the concept of optimal transport between minibatches. We detail how it is computed and the empirical benefits of this strategy. Despite strong computational gains, a formal study of the minibatch OT formulation is lacking. In this manuscript, based on the literature regarding the statistical properties of OT and the sampling strategy, we aim at providing a theoretical study of minibatch OT. That is why, in Section 7.2, we discuss the optimal transport statistical and smoothness properties. And then, we finish this chapter by presenting the concept of U-statistics, which is at the heart of our study on minibatch optimal transport. 92 Chapter 7. Optimal Transport statistical properties and subsampling

Large scale Optimal Transport through minibatches computation

In this section, we review how the use of minibatch with optimal transport appeared.

Empirical interests

The Kantorovich-Rubinstein duality turns the original marginal equality constraints into inequality constraints, which can be hard to optimize. These inequalities are generally formulated as specific relations between the two dual potentials, known as the c-transform. In the notable case of Euclidean distance as a ground cost, however, these constraints translate more simply in constraint over the Lipschitzness of the potential. Nonetheless, practical approximations are still difficult to obtain and are still a source of active research [START_REF] Arjovsky | [END_REF], Gulrajani 2017].

Thus, we might want to compute the primal or dual OT with the given ground cost. Unfortunately, we showed in Section 2.3 that the computational complexity of optimal transport is of order O(n 3 log(n)) where n is the number of samples and that the memory complexity is O(n 2 ). The latter is due to the storage of the ground cost and the coupling, which are square matrices of size n. Thus in the big data scenario, practitioners relied on a common technique in deep learning, a minibatch computation of optimal transport. The idea is simple, instead of estimating the optimal transport cost between the full empirical measures, we only estimate the cost between a subset of them. It takes the form of drawing m samples at random in both the source and the target measures, associating a uniform weight to each of them, and then computing the OT cost between these samples only. This strategy gives a computational complexity of O(m 3 log(m)) if one uses the original OT cost and a memory cost of O(m 2 ) with m n.

The minibatch strategy has been successful on several machine learning problems. It has been popular in generative modelling to design variants of the Wasserstein GANs. For instance, [START_REF] Genevay | [END_REF]] developed a GAN based on the AutoDiff Sinkhorn Divergence. After taking two minibatches of training data and generated data, the authors used a feature extractor D φ before applying the squared Euclidean distance as ground cost, i.e., for x, y ∈ R d , c(x, y) = D φ (x)-D φ (y) 2 . Then they optimize the feature extractor and the generator as a min max problem on minibatches. In a concurrent work, [START_REF] Salimans | [END_REF]] developed a MMD loss based on a minibatch computation of the entropic-regularized OT. On another note, generative models based on the sliced Wasserstein distance [START_REF] Kolouri | [END_REF], Kolouri 2018, Kolouri 2019a[START_REF] Liutkus | [END_REF] fall into the minibatch framework where they compute the SWD between minibatches of input distributions.

Another popular example is domain adaptation. We recall that a fundamental technique to transfer the knowledge between domains is to use a domain alignment loss. Based on a feature extractor, the idea is to align the extracted data features of the two domains when data share the same class. The alignment strategy can be achieved by optimal transport on the joint domains of data and labels as done in [START_REF] Damodaran | [END_REF], Xu 2020a, Fatras 2021b]. The ground cost incorporates a term on the data features and on the labels. However, computing the optimal transport cost between the full distributions at each iteration of the optimization procedure is too costly. Thus practitioners successfully relied on an optimal transport minibatch computation. We nonetheless prove in Section 9 that the minibatch procedure creates negative transfer and we propose a strategy based on unbalanced optimal transport to alleviate this problem.

Minibatch OT has also been used in missing data imputation [START_REF] Muzellec | [END_REF]] or to compute an online OT variant [START_REF] Mensch | [END_REF]]. This approximation was successful in practice even if only subsets of input measures were considered. We now present key statistical elements of OT that we use to justify the empirical success of the minibatch strategy.

Optimal Transport statistical and smoothness properties 93

Optimal Transport statistical and smoothness properties

In this section, we discuss the statistical and smoothness properties of optimal transport which are important in the context of machine learning. We start by discussing the approximation of optimal transport between continuous measures by their empirical counter-parts. We then discuss the empirical estimator bias w.r.t. the continuous optimal transport. Finally, we discuss the smoothness of optimal transport w.r.t. the ground cost, which is at the heart of modern optimization strategy.

Empirical estimation of Optimal Transport Approximation with empirical measures: the curse of dimensionality

In machine learning, the probability measures α, β ∈ M 1 + (X ) are unknown and we only have access to the empirical counter-parts α n , β n . That is why in order to estimate the optimal transport cost L(α, β), we compute directly the OT cost between the empirical measures L(α n , β n ). This approximation is valid because our empirical estimators metricize the weakly convergence, i.e., L(α n , β n ) n→+∞ L(α, β). Indeed the empirical distribution weakly converges towards the true distribution as the number of samples grows, α n n→+∞ α (resp. β n n→+∞ β) and the original optimal transport cost, the Sinkhorn divergence and the unbalanced variants metrize the convergence in law [START_REF] Peyré | [END_REF][START_REF] Feydy | [END_REF], Séjourné 2019]. While the Gromov-Wasserstein distance does not metricize the weak convergence, it still verifies the property β), this is straightforward using [Mémoli 2011, Theorem 5.1 (c)] and Wasserstein distance's weak convergence. As the convergence is ensured between the two terms, we want to answer the important question of the convergence rate. This rate is often called the sample complexity. A quick convergence rate means that the quantity L(α, β) is easy to estimate empirically. We now give the sample complexity of optimal transport and the entropic-regularized OT and compare them to the sample complexity of MMD [START_REF] Gretton | [END_REF]].

α n n→+∞ α =⇒ GW (α n , β n ) n→+∞ GW (α,
Theorem 7.2.1 ( [Dudley 1969]). For X = R d and measures supported on bounded domains, for d > 2 and 1 < p < +∞ we have

E(|W p (α n , β n ) -W p (α, β)|) = O(n -1/d ). (7.1)
The expectation is taken with respect to the samples (x i , y i ) 1≤i≤n . This rate is tight in R d if one of the measures has a Lebesgue density. The rate can be refined if the measures are supported on low dimension subdomains of dimension s and the rate becomes O(n -1/s ) as shown in [Weed 2019]. Unfortunately, this sample complexity is really slow for high dimensional data and it is known as the curse of dimension. The higher the ambient dimension is, the slower the convergence of empirical estimators as we need an exponential number of samples to estimate the optimal transport cost. In comparison, MMD has a sample complexity independent from the dimension and of order O(n -1/2 ). This makes MMD loss easier to estimate with empirical measures.

As the Sinkhorn Divergence interpolates between optimal transport and MMD, it is a natural question to determine its sample complexity. It is provided in the following theorem: The entropic-regularized OT's sample complexity interpolates between the optimal transport and the MMD samples complexities. Thus it can be more robust to the curse of dimensionality depending on the regularization coefficient value. Straightforwardly, the Sinkhorn divergence has a similar sample complexity as the sum of three entropic-regularized OT terms. This highlights the practical interest of using the Sinkhorn divergence when data lie in high dimension instead of non-regularized optimal transport. The Sinkhorn divergence sample complexity has been improved with weaker assumptions in [Mena 2019] such as unbounded measures. They also introduced a central limit theorem of entropic-regularized OT which can be applied to estimate the entropy of a random variable corrupted by a Gaussian noise. After reviewing the sample complexity of empirical estimators, we finish this section by stating concentration bounds of original OT and entropic-regularized OT estimators around their mean.

E(|L ε (α n , β n ) -L ε (α, β)|) = O e κ ε √ n 1 + 1 ε d/2 , ( 7

Concentration around the mean of Optimal Transport

One of the main questions in statistical applications is to understand the concentration of a given estimator around the unknown parameter of interest. In our case, we want to measure the deviation and the converge speed of our estimators to their expectation as the number of samples grows. A first result come from [Weed 2019, Proposition 20]: Proposition 8. Consider a compact space X , for all n ≥ 0 and 1 ≤ p < +∞, we have with probability 1 -δ:

|W p p (α, α n ) -EW p p (α, α n )| ≤ 1 n log( 2 δ ) (7.3)
Note that this concentration result is only for the p-Wasserstein distance to the power p. For the entropic-regularized OT, we have: Proposition 9. Consider a compact space X a compact subspace of R d , a C ∞ and L-Lipschitz cost, for all n ≥ 1 and 1 ≤ p < +∞, we have with probability 1 -δ:

|L ε (α, α n ) -EL ε (α, α n )| ≤ κ 1 √ n + κ 2 2 log(1/δ) n . (7.4)
With κ 1 , κ 2 constants detailed in [START_REF] Genevay | [END_REF]].

For concentration bounds, the convergence speeds are similar between the p-Wasserstein distance to the power p and the entropic-regularized OT. These results are interesting because the concentration bounds do not depend on the ambient dimension d contrary to the sample complexities. This means that the estimator converges to its expectation at a rate of O(n -1 2 ), where n is the number of samples. In Section 8.3, we introduce concentration bounds for any optimal transport costs computed between minibatches. After reviewing the sample complexity and the concentration properties of the optimal transport estimator and its entropic-regularized counter-part, we now review its bias with respect to the loss L(α, β).

Optimal Transport statistical and smoothness properties 95

Biased empirical estimator

We saw that optimal transport suffers from the curse of dimension which penalizes its statistical properties. Unfortunately another weakness occurs when dealing with empirical measures. The optimal transport cost between empirical measures is a biased estimator of optimal transport between continuous measures, i.e., E αn,βn (L(α n , β n , C)) = L(α, β, c). This surprising result was first discussed in [Bellemare 2017] and authors proposed a specific integral probability metric [START_REF] Gretton | [END_REF]] to solve this problem. It was studied in more details in [START_REF] Bińkowski | [END_REF], where authors showed that this bias also exists for all integral probability metrics. The Sinkhorn Divergence and the entropic-regularized OT also suffer from such a bias. In this manuscript, we show that the minibatch procedure does not suffer from a biased estimator but at the price of losing the metric properties, see Section 8.2.1 for more details.

In machine learning, we deal with empirical measures and we have discussed the statistical properties of the optimal transport in this context. In the next section, we discuss another important problem of interest in machine learning, the optimization properties of optimal transport in a data fitting problem.

Minimizing Optimal Transport with respect to a parameter θ

In this section, we review the optimal transport optimization properties when the Wasserstein distance is used to fit a model to empirical data. Given discrete samples (x i ) n i=1 ∈ R d from an unknown distribution α, we want to fit a parametric measure θ → β θ ∈ M + 1 (R d ) to α using an optimal transport cost and the empirical samples. Let Υ be a contrast function, we thus look for the solution of

θ = arg min θ∈θ Υ(α n , β θ ), (7.5)
where α n is the empirical distribution of sample (x i ) n i=1 . When the contrast function is chosen to be the Wasserstein distance, the above optimization problem is known as Minimal Wasserstein estimation [Bernton 2019]. Learning many generative models can also be framed as solving (7.5) [START_REF] Genevay | [END_REF]]. To find the optimal θ which minimizes the Kantorovich estimator, it is standard to rely on a gradient descent procedure [START_REF] Genevay | [END_REF]].

To ensure that minimizing the above equation leads to the correct minimum of Υ(α, β θ ), we need for the estimator Υ(α n , β θ ) to be an unbiased estimator of Υ(α, β θ ) with respect to the draw of the data and that we can permute the expectation on the data and the gradient with respect to θ, i.e., E αn,βn ∇ θ OT (α, β θ ) = ∇ θ E αn,βn OT (α, β θ ) [START_REF] Bottou | [END_REF], Robbins 1951b]. We saw in the previous section that the first condition is not met as the Wasserstein estimator is a biased estimator of the Wasserstein distance between continuous distributions. However, minimizing a biased estimator can still lead to good performances in practice. We now study whether the second condition is met by optimal transport or not.

Regularity with respect to the ground cost C

To prove that the exchange between the expectation and the gradient is possible, a common argument is to rely on the differentiation lemma.

Lemma 7.2.1 (Differentiation lemma). Let V be a nontrivial open set in R p and let P be a probability distribution on R d × R d . Define a map F : R d × R d × V → R with the following properties:

• For any θ ∈ V, E P [|F(x, y, θ)|] < ∞. • For P-almost all (x, y) ∈ R d × R d , the map V → R, θ → F(x, y, θ) is differentiable.

Chapter 7. Optimal Transport statistical properties and subsampling

• There exists a P-integrable function ϕ :

R d × R d → R such that |∂ θ F(x, y, θ)| ≤ ϕ(x, y) for all θ ∈ V .
Then, for any θ ∈ V , E P [|∂ θ F(x, y, θ)|] < ∞ and the function θ → E P [F(x, y, θ)] is differentiable with differential:

E P ∂ θ [F(x, y, θ)] = ∂ θ E P [F(x, y, θ)].
(7.6)

The proof can be found in [Klenke 2008, Theorem 6.28]. The optimal transport loss uses the parameters θ in the ground cost, formally we have

θ → y θ → C(x, y θ ) → L(α n , β n , C(X, Y θ )
). Thus we need to assure the differentiability of OT with respect to the the ground cost C as the probability vectors a and b remain constant. To study this regularity, we use the Danskin theorem [Bertsekas 1973, proposition B.25] which gives the regularity of a max problem. It is stated as follow:

Proposition 10 (Danskin Theorem). Let Z ⊂ R d be a compact set and let φ :

R d × Z → R be continuous such that φ(., z) : R d → R is convex for each z ∈ Z. Then 1. The function f (x) = max z∈Z φ(x, z) (7.7)
is convex and has directional derivatives given by f (x, y) = max z∈Z(x) φ(x, z; y) where φ(x, z; y) is the directional derivatives of φ(., z) at x in direction of y and Z(x) is the set of maximum points in (7.7), i.e.,

Z(x) = { z ∈ Z|φ(x, z) = max z∈Z φ(x, z)}. (7.8)
In particular, if Z(x) is a singleton and φ(., z) is differentiable at x, then f is differentiable at x and ∇ x f (x) = ∇ x φ(x, z).

If φ(., z)

is differentiable for all z ∈ Z, and ∇ x φ(x, .) is continuous on Z for each x, then the subdifferential of f (x) is given by:

∂f (x) = conv{∇ x φ(x, z)|z ∈ Z(x)}, ∀x ∈ R d , (7.9)
where conv indicates the convex hull operation.

If we apply the Danskin Theorem to discrete optimal transport, the function φ corresponds to optimal transport, x corresponds to the ground cost C and z corresponds to the transport plan Π. The set of transport plan U (α, β) is indeed a compact set of measures [Santambrogio 2015, Theorems 1.4, 1.5, 1.7]. Regarding the continuity of optimal transport, the Frobenius product C, Π is obviously continuous and it is linear in C and Π. As -φ is a linear function of C, it is a convex function of C. Unfortunately, original optimal transport does not always have a unique solution as illustrated in Figure 7.1, thus we only have access to subgradients. Regarding the entropic-regularized OT, the Kullback-Leibler divergence is also a continuous function as the product and sum of continuous functions (we recall that their probability vectors a and b have full support, i.e., ∀i, a i > 0). As the entropic-regularized OT is strongly convex, it has a unique solution. Thus it is differentiable with respect to the ground cost C contrary to original OT. So the differentiation lemma works only for entropic-regularized OT. To justify the convergence of a SGD strategy with original OT, we rely on a concept of generalized gradients developed in the next section. It allows us to exchange generalized gradients and expectations.

Clarke regularity

As the original optimal transport is not differentiable, we can not apply standard first order optimization methods for smooth functions. That is why instead we introduce the Clarke regularity (see [Clarke 1990] for a full survey). A fundamental problem of nonsmooth analysis is to define a generalization of the notion of derivative, that preserves some fundamental properties of the classical notion of derivative and allows for a construction of calculus for nondifferentiable functions. Subdifferentials of convex functions are a well known example of such a generalization. While extremely useful and important in convex analysis and optimization, their obvious drawback is that they are defined only for convex functions. Clarke generalized derivatives (also called Clarke subdifferentials) are well defined for all locally Lipschitz functions. We remind the reader that a function f : X → R, where X is an open set in R d , is called locally Lipschitz at x ∈ X if there exists an open neighbourhood U of x, such that f is Lipschitz on U . Such a function is called globally Lipschitz, if it is locally Lipschitz at all points of its domain. We first introduce Clarke generalized directional derivatives: Definition 11. Let f : X → R be a locally Lipschitz function defined on an open set X ⊂ R d . Clarke generalized directional derivative of f at x in direction v is denoted by f • (x; v) and defined by:

f • (x; v) = lim sup h↓0,y→x f (y + hv) -f (y) h .
For a given x, the function f • (x; v) is finite, positively homogeneous, and a subadditive function of v. We refer to Proposition 2.1.1 [Clarke 1990] for the proof of those properties. Positive homogenity and subadditivity imply, that f • (x; v) is a convex function of v. Clarke's subdifferential at x is defined as the subdifferential of f • (x; •) at the origin. Definition 12. Let f : X → R be a locally Lipschitz function. The Clarke generalized gradient, also called Clarke subdifferential is a set valued map defined by:

∂f (x) := {ξ : f • (x; v) ≥ ξ, v for all v ∈ R d }.
We use the symbol ∂f for the Clarke subdifferential, that is commonly used for subdifferentials of convex functions. This is no accident and will not be a cause of confusion, as when a convex function is Chapter 7. Optimal Transport statistical properties and subsampling locally Lipschitz at x, for example when x is in the interior of its domain, then the two notions coincide (see Proposition 2.2.7 [Clarke 1990]). In fact, Clarke generalized gradients share a number of important properties with subdifferentials of convex functions. Namely for any locally Lipsthiz function f defined on an open set X, ∂f (x) is a closed, compact set for all x ∈ X (Proposition 2.1.2(a) [Clarke 1990]) and the multifunction ∂f has a closed graph (Proposition 2.1.5(b) [Clarke 1990]).

For such defined generalization of derivative, we have a well defined calculus, including Mean Value Theorem (Theorem 2.3.7 [Clarke 1990]), chain rule (Theorem 2.3.9, Theorem 2. 3.10 [Clarke 1990]) as well as formulas for generalized derivatives of sums, products and quotients. A problematic aspect of this calculus is that it usually only gives one-sided inclusions in the general case, similarly to how for two convex functions f, g, we always have the following inclusion:

∂(f + g) ⊆ ∂f + ∂g,
while in general the reverse inclusion doesn't always hold. A classical condition on the locally Lipschitz function f under which those relations often become exact is regularity.

Definition 13. (Definition 2.3.4, [Clarke 1990]) A locally Lipschitz function f is said to be regular at x provided that the following two conditions are satisfied:

1. For all v, the usual one-sided directional derivative f (x; v) exists, 2. For all v we have:

f (x; v) = f • (x; v).
We will also say that a function f is minus regular, if -f is regular. An intuitive geometric interpretation of regularity is that a function is regular if it doesn't have "upwards dashes" in its graph. For example x → |x| is regular, since the dash is downwards, but x → -|x| is not. Nevertheless, the function x → -|x| behaves equally well as the absolute value function from the perspective of calculus of generalized derivatives since it is minus regular, see Remark 2.3.5 [Clarke 1990]. Finally, we present the most important result which justifies the use of Clarke gradients: the generalization of the differentiation lemma to Clarke regular functions.

Theorem 7.2.3 (Generalized gradients of integral functionals). Let (T, T , µ) be a positive measurable space. Suppose that U is an open subset of a separable Banach space Θ, and that we are given a family of functions f t : U → R satisfying the following conditions:

• For each θ ∈ U , the map t → f t (θ) is measurable. • For some k(•) ∈ L 1 (T, R), for all θ 1 and θ 2 in U , and t in T , one has |f t (θ 1 ) -f t (θ 2 )| ≤ k(t) θ 1 -θ 2 .
Write f (θ) = ´T f t (θ)µ(dt) and suppose f is defined at some point θ in U . Then f is defined and Lipschitz in U . Then we have

∂f (θ) = ∂ ˆT f t (θ)µ(dt) ⊂ ˆT ∂f t (θ)µ(dt).
(7.10)

If in addition each f t (•) is regular at θ, then f is regular at θ and the right hand side expression in (7.10) is an equality.

U-statistics: generalized mean 99

For the proofs of chain rule and generalized gradient of sum (or expectation) becoming exact for regular functions, we refer once again to [Clarke 1990]. We show that optimal transport is Clarke regular with respect to the ground cost C in Section 8.3.2.

In the next section, we finally introduce a key element of the rigorous study of minibatch OT.

U-statistics: generalized mean

In this section, we introduce the concept of U-statistics and related quantities. A U-statistic is an average of a kernel h over m-tuples taken from an n-tuple X. Thus they are a good formalism to study minibatch OT that allow us to derive statistical and optimization properties. We start by defining what U-statistics are. We then review their use in machine learning and we finish by discussing their concentration bounds.

U-statistics definition and application in Machine Learning

In this section, we present U-statistics which are estimators of specific quantities. The theory of U-statistics can be drawn back to [Hoeffding 1948[START_REF] Hoeffding | Probability Inequalities for Sums of Bounded Random Variables[END_REF]]. We start by giving a formal definition.

Definition 14 (Complete U-statistics). Consider two sequences of i.i.

d random variables X = {x 1 , • • • , x n } ∼ α ⊗n and Y = {y 1 , • • • , y n } ∼ β ⊗n (n ≥ 2) taking values in a measurable space (T, T ). Assume h : X ⊗m × Y ⊗m → R is a X ⊗m × Y ⊗m measurable function, with 2 ≤ m ≤ n, and also that it is a permutation symmetric map, meaning that h(x 1 , • • • , x m , y 1 , • • • , y m ) = h(x σ1 , • • • , x σm , y σ1 , • • • , y σm )
for any m-tuples of data and permutation σ. A two samples U-statistics of order m is defined as: (7.11) where P m is the set of m-tuples without replacement and X(I) (resp Y (J)) represents the data whose indices correspond to the indices in I (resp. J).

U n,m = (n -m)! n! 2 I,J∈P m h(X(I), Y (J)),
It can be interpreted as a generalized mean over m-tuples of data. We have presented above the two samples U-statistics which results will be applied to optimal transport, but one sample and more general U-statistics can be defined. Regarding the expectation of this quantity, U n,m is clearly an unbiased estimator of

E x1,••• ,xm,y1,••• ,ym h(x 1 , • • • , x m , y 1 , • • • , y m )
and one of its properties is that it has the minimum variance among all unbiased estimators. A full review of U-statistics properties can be found in [J Lee 2019]. The combinatorial number of terms makes this quantity hard to use in practice. In order to remedy to this problem, we present a sub-sample quantity called incomplete U-statistics. Incomplete U-statistics are defined as following.

Definition 15 (Incomplete U-statistics). Consider the notations and hypotheses from Definition 14. Pick two integers k > 0 and 0 < m ≤ n. Then, we define the incomplete estimator:

U k n,m (a, b) := 1 k (I,J)∈D k h(X(I), Y (J)), (7.12)
where D k is a set of k pairs of m-tuples uniformly drawn independently.

They allow a smaller computational cost than their corresponding complete estimator. We finish the U-statistics definition by discussing a related quantity of interest called V-statistics. V-statistics are related to U-statistics with the notable difference that they average the kernel h over m-tuples of indices 100 Chapter 7. Optimal Transport statistical properties and subsampling with possibly repeated indices, i.e., I ∈ n m . So they are not necessarily unbiased estimators due to the repetition of samples. They are defined as

V n,m = 1 n m 2 I,J∈ n m h(X(I), Y (J)). (7.13)

Applications in Machine Learning

U-statistics have been used in various machine learning problems as a performance criterion. We present two examples where U-statistics are present: clustering and metric learning. More detailed examples can be found in [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF]]. We then finish this section by an application of U-statistics for comparing probability distributions.

Clustering. Consider a distance d : X × X → R + . Clustering is an unsupervised learning task that consists in partitioning a set of data points x 1 , • • • , x n in a feature space X into a finite collection of subgroups depending on their closeness with respect to d, in other words data points in the same subgroup should be closer to each other than to those lying in other subgroups. We thus want to evaluate the quality of a partition P of X with respect to the clustering of an i.i.d. empirical data x 1 , • • • , x n drawn from α. This quality can be assessed through the within cluster point scatter, which takes the form of a one sample U-statistic of order 2 [START_REF] Clémençon | Ranking and Empirical Minimization of U-statistics[END_REF]]:

L clustering (P) = 2 n(n -1) i<j d(x i , x j ). C∈P I{(x i , x j ) ∈ C}, (7.14)
The U-statistic kernel is h(x i , x j ) = d(x i , x j ). C∈P I{(x i , x j ) ∈ C}.

Metric Learning. Choosing the appropriate metric for a given application is a fundamental problem.

For instance in domain adaptation as we detailed above, a joint metric on the extracted feature and the label spaces gave state-of-the-art results [Courty 2017b]. In the context of supervised learning, the goal of metric learning is to find a metric under which data with the same label are close to each other and those with different labels are far away [Bellet 2015, Kulis 2013b]. Consider an i.i.d. empirical data

(x 1 , y 1 ), • • • , (x n , y n ) ∈ X 1 = X × {1, • • • , j} the empirical pairwise classification performance of a distance d : X 1 × X 1 → R + can be evaluated by: L metric (d) = 6 n(n -1)(n -2) i<j<k I{d(x i , x j ) < d(x i , x k ), y i = y j = y k }, (7.15)
which is a one sample U-statistic of degree 3 with a kernel h((x, y), (x , y ), (x , y )) = I{d(x, x ) < d(x, x ), y = y = y }.

U-statistics as divergence between probability measures. We recall that a MMD loss with a positive kernel k is defined as:

α -β 2 k = E α⊗α [k (x, x )] + E β⊗β [k (y, y )] -2E α⊗β [k(x, y)]. (7.16)
These above expectations can be estimated using empirical samples X ∼ α ⊗n , Y ∼ β ⊗n and with the estimator

U 2 MMD (X, Y ) = 1 n(n -1) n i=1,i =j k(x i , x j ) + 1 n (n -1) n i=1,i =j k(y i , y j ) - 2 nn n i=1 n j=1
k(x i , y j ), (7.17)

U-statistics: generalized mean 101

which is composed of two one sample U-statistics of degree 2 and one sample average. When we have the same number of empirical data n, we can estimate it with the estimator

U 2 MMD (X, Y ) = 1 n(n -1) n i =j h(z i , z j ), (7.18)
which is a one sample U-statistic and where the U-statistic kernel is equal to h(z i , z j ) = k(x i , x j ) + k(y i , y j ) + k(x i , y j ) + k(x j , y i ). We can also estimate the MMD loss using V-statistics, which leads to a biased loss:

V 2 MMD (X, Y ) = 1 n(n -1) n i,j=1 k(x i , x j ) + 1 n (n -1) n i,j=1 k(y i , y j ) - 2 nn n i=1 n j=1 k(x i , y j ), (7.19)
Note that in the V-statistics, the index i can be equal to the index j. Intuitively we expect the empirical test statistic, whether biased or unbiased, to be small if α = β, and large if the distributions are far apart. U-statistics have appealing concentration bounds and thus, their use is an important element for the statistical study of MMD losses. We present the main U-statistics concentration results in the next section.

U-statistics concentration bounds

In the case of U-statistics, we want to aim at estimating the concentration of U n,m around EU n,m . It is the goal of this section to give the main standard result. We start by recalling the famous Hoeffding lemma: Lemma 7.3.1 (Hoeffding's Lemma). Let the real random variable x ∈ [a, b] and denote Ex = µ. Then for all s ∈ R: E e s(x-µ) ≤ e s 2 (b-a) 2 /8 .

(7.20)

The proof can be found in [START_REF] Hoeffding | Probability Inequalities for Sums of Bounded Random Variables[END_REF]]. This quantity gives us an upper bound on the momentgenerating function of any bounded random variable minus its mean value. The goal is now to apply this lemma to U-statistics to get an Hoeffding inequality type bound and by doing so, we get the following result:

Lemma 7.3.2. Let δ ∈ [0, 1] and m ≥ 1 be fixed. Suppose that the kernel h is bounded, i.e., 0 ≤ h ∞ ≤ M h . We have a concentration bound between U n,m and EU n,m depending on the number of empirical i.i.d. data X ∼ α ⊗n and Y ∼ β ⊗n , with probability 1 -δ:

|U n,m -EU n,m | ≤ M h log(2/δ) 2 n/m (7.21)
Proof. We fix r = n/m . Let 0 ≤ k ≤ r -1, we define the set

I k := {km + 1, • • • , km + m}.
Then we define the function ζ as : We can see that

ζ(x 1 , • • • , x n , y 1 , • • • , y n ) = 1 r r-1 k=0 ζ k (X(I k ), Y (I k )) = 1 r r-1 k=0 h(X(I k ), Y (I k )). ( 7
ζ(x 1 , • • • , x n , y 1 , • • • , y n
) is a sum of independent random variables. In the summation below, σ x or σ y denotes a generic permutation of {1, • • • , n}. We compute :

r σx,σy ζ(x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ) = r(n -m)! 2 I,J∈I h(X(I), Y (J)) (7.23) = r(n -m)! 2 n! (n -m)! 2 U n,m (7.24) Thus we have U n,m = 1 n! 2 σx,σy ζ(x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) )
, which is a sum of independent random variables. Thus, we now highlight the quantity U n,m -EU n,m for the different sums we have. As our data are i.i.d., we get:

U n,m -EU n,m = 1 n! 2 σx,σy ζ(x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ) -Eζ(x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ) = 1 n! 2 σx,σy 1 r r-1 k=0 ζ k (X(I k σx ), Y (I k σy )) -Eζ k (X(I k σx ), Y (I k σy )) .
Where I k σx (resp. I k σy ) represents the I k indices after applying the permutation σ x (resp. σ y ). In what follows, we abbreviate the summation of ζ over the permutations σ x , σ y as σ ζ σ . Thus, for t > 0, we have:

P (U n,m -EU n,m ≥ t) ≤ e -λt E(e λ(Un,m-EUn,m) ) = e -λt E e λ 1 n! 2 σ ζσ-Eζσ ≤ e -λt 1 n! 2 σ E e λζσ-Eζσ ≤ e -λt max σ E e λζσ-Eζσ ,
where in the first and second inequalities, we used Markov's and Jensen's inequalities respectively. Furthermore, for any ζ σ0 , we have from Lemma 7.3.1 along with the bounded kernel assumptions

E e λζσ 0 -Eζσ 0 = r-1 k=0 E e λ r ζ k -Eζ k ≤ r-1 k=0 E e λ 2 8r 2 M 2 h = exp λ 2 8r M 2 h (7.25) Thus P (U n,m -EU n,m ≥ t) ≤ exp -λt + λ 2 8r M 2 h .
Optimizing over λ ∈ R + gives the desired result.

The proof for the one sample U-statistics is the same. The exact same result is true for the V-statistics and the proof can be found in [Hoeffding 1963, section 5.C]. Thus we can see that for a fixed m and δ, the difference between the complete estimator and the expected quantity decreases at a rate of O( m n ). This concentration result is at the heart of several of our contributions that we detail in the next chapter.

Conclusion

In this chapter, we presented the empirical interest of computing the optimal transport problem between minibatches. We discussed the practical advantages as well as the different domains where it was used. Then we presented several optimal transport properties that are at the foundation of our theoretical analysis of minibatch OT. We reviewed the optimal transport sample complexities, the concentration bounds, the estimator bias and the regularity with respect to the ground cost c. Finally, we presented the U-statistics and we gave their concentration bounds.

Conclusion 103

Several works were published to justify the minibatch paradigm. [Bernton 2019] studied statistical inference, performed by the minimization over the parameter space of the Wasserstein distance between model distributions and the empirical distribution of the data. They showed that for generative models, the minimizers of the minibatch loss converge to the true minimizer when the minibatch size and the number of data increase. [Sommerfeld 2019] considered another approach. They show that approximating OT with minibatch OT is a good approximation loss and they exhibit a deviation bound between the two quantities. In this manuscript, we propose a study of the use of minibatch OT as a loss function and we gather all our contributions in the two next Chapters 8 and 9. We propose a rigorous formalism for all empirical measures, which assures that all the mass is transported. Based on the formalism, we give a closed-form in the case where samples lie in 1D and we illustrate the non optimal connections created by the minibatch approximation on the resulting transport plan. We then study the loss properties of minibatch OT and we show that minibatch OT is not a distance. We introduce a new loss function to recover some distance properties and study its positiveness. We evaluated our different theoretical contributions on different generative models and color transfer experiments. Finally, the different illustrations highlighted that the non optimal connections could hurt the generalization of deep neural networks. We finally propose to use a robust OT variant at the minibatch level to mitigate these connections and investigate the performances on domain adaptation experiments.

Chapter 8 In this chapter, our goal is to study the minibatch approximation of optimal transport as a loss function. We introduce a rigorous formalism and provide a theoretical study of minibatch optimal transport. In the first section, we formalize the problem, develop empirical estimators of minibatch OT for uniform probability measures and give a closed-form solution in the 1D case. Following the uniform case, we extend empirical estimators to non uniform probability vectors. We then introduce a debiased minibatch OT cost inspired by the Sinkhorn Divergence and review its positivity. We then study the concentration bounds of minibatch OT and its optimization properties. Finally, we evaluate our debiased loss functions and minibatch OT on several experiments. These contributions have been published in the 23rd International Conference on Artificial Intelligence and Statistics [Fatras 2020b] and a longer version is currently submitted to SIAM Journal on Mathematics of Data Science [Fatras 2021c].

Minibatch Optimal Transport

Chapter 8. Minibatch Optimal Transport

Notations Before starting, we recall some notations that will be used in this chapter. Suppose we have access to n data x ∈ R d . We first assign a fixed index to each data and then get a n-tuple of data X, i.e., X = (x 1 , x 2 , • • • , x n ). This assignation allows us to draw minibatches of data, and moreover, permutations of assigned indices would not change any result.

As each element inside the data m-tuple has an index, it is then possible to characterize a m-tuple of data with a corresponding m-tuple of indices. A generic element of indices I = (i 1 , . . . , i m ) ∈ n m is called an index m-tuple. We recall that we denote by P m the set of m-tuples without replacement. For an index m-tuple I = (i 1 , . . . , i m ), X(I) = (x i1 , . . . , x im ) is the corresponding data m-tuple, and vice-versa, any data m-tuple can be written X(I) for some index m-tuple I.

We finish by defining extra notations. The characteristic function of the set A, which is equal to 1 if i ∈ A and 0 otherwise, is denoted 1 A (i). With a slight abuse of notation we write i ∈ I if the index i appears in the m-tuple I. We also write 1 I (i) for tuples of indices. Regarding the sum over the elements of

I = (i 1 , • • • , i m ), we denote it as i∈I f (i) = m k=1 f (i k )
and similarly for the product over the elements

Π i∈I f (i) = Π m k=1 f (i k ).

Expectation of Optimal Transport over minibatches

Throughout the minimization of a data fitting problem, the optimal transport cost can be computed between minibatches of input empirical measures. This takes the form of 2 m-tuples of samples drawn uniformly at random from the source and the target measures, to associate them a uniform weight 1/m and to compute the optimal transport between these sub-measures. It is likely that the selected data change over each iteration and this leads to minimizing the expectation of optimal transport between minibatches of input measures. Formally, consider the minibatches (X, Y ) drawn from α ⊗m ⊗ β ⊗m , we minimize

E m h = E (X,Y )∼α ⊗m ⊗β ⊗m [h(u m , u m , C m (X, Y ))], (8.1)
where h is an OT cost and u m is a uniform vector of size m. The expectation over minibatches is fundamentally different with respect to the original OT problem between the full input measures. In this section, we start by building a rigorous formalism in the case of empirical measures, we also prove a closed-form solution for 1D data of the minibatch OT problem and give 1D and 2D illustrations. Following the formalism in the uniform case, we extend it to general discrete measures.

Empirical estimators in the uniform case

As we have only access to the empirical measures α n and β n , we need to define relevant estimators of E m h . Relevant estimators would correctly estimate E m h but also transport the entire input measures u m . We start with defining a ground cost and for simplicity we consider the Euclidean distance, but our formalism can be adapted to any ground cost. We consider:

C m,p : (X, Y ) ∈ (R m×d ) 2 → C m,p (X, Y ) = x i -y j p 2 1≤i,j≤m ∈ M m (R), (8.2)
The power p is useful when we consider the p-Wasserstein distance. The expectation of the OT cost over minibatches E m h can be estimated using U-statistics as presented in Section 7.3. Thus we define the following estimator.

Expectation of Optimal Transport over minibatches 107

Definition 16 (Minibatch OT). Let C = C n,p (X, Y ) be a square matrix of size n. Given an OT loss h ∈ {L, W p , L ε , S ε } and an integer m ≤ n, we define the following quantity:

h m C (X, Y ) := (n -m)! 2 n! 2 I,J∈P m h(u m , u m , C I,J ) (8.3)
where for I, J two m-tuples, C (I,J) is the matrix extracted from C by keeping the rows and columns corresponding to I and J respectively.

A similar estimator can be built for Gromov-Wasserstein and we report the reader to its general form in Section 8.1.3. We omit C when clear from context. While it is easier to get an upper bound of minibatch OT and statistical results with the ground cost C n,p (X, Y ), which is a square matrix of size n, in practice we only need to compute C (I,J) as it is equal to C m,p (X(I), Y (J)) and it reduces the memory cost. These quantities represent an average of OT costs h over minibatches of size m. It is clear that these estimators are two sample U-statistics of order 2m. When one of the measure is continuous while the other is discrete, we have a semi-discrete variant of minibatch OT:

h m C (X, β) := (n -m)! n! I∈P m E Y ∼β ⊗m h(u m , u m , C(X(I), Y )) (8.4)
It can be seen as a one sample U-statistic of order m.

Remark 6.

We recall that all OT costs are symmetric. This means that for any permutation I of a given tuple

I, we have h(u m , u m , C I,J ) = h(u m , u m , C I ,J ).
h m,k C (X, Y ) := 1 k (I,J)∈D k h(u m , u m , C (I,J) ) (8.6)
where D k is a set of k pairs of m-tuples drawn uniformly at random.

The subsample quantity for exact optimal transport has then a computational complexity of O(km 3 log(m)). The value of k depends on the application and is discussed in more details in the experimental section. We finish this paragraph with the following remark:

Remark 7. In (8.3) and (8.6), the dependence in the measure supports is implicit through the Euclidean ground cost

C, i.e., (X, Y ) → C(X, Y ) → h(u, u, C(X, Y )).
The transport plan is interesting to see the connections between samples, that is why we build similar estimators.

Chapter 8. Minibatch Optimal Transport

Transport plan estimators Classical OT losses such as L or its entropic variant L ε are directly associated with a transport plan between measures. We now propose to similarly define a transport plan associated to the proposed minibatch OT losses when it is possible. The main idea is that for each pair of samples x i and y j , one can average the connections provided by all possible "transport plan between minibatches", with the following definition.

Definition 18 (minibatch transport plan). We denote by Opt(h, C, u, u) the set of all optimal transport plans for h an OT cost, cost matrix C and a marginal u. Let u m ∈ R m be a discrete uniform probability vector. For each pair of index m-tuples I = (i 1 , . . . , i m ) and J = (j 1 , . . . , j m ) from 1, n m , consider

C := C I,J the m × m matrix with entries C k = C i k ,j
and denote by Π m I,J an arbitrary element of Opt(h, C, u m , u m ). It can be lifted to an n × n matrix where all entries are zero except those indexed in I × J:

Π I,J = Q I Π m I,J Q J (8.7)
where Q I and Q J are m × n matrices defined entrywise as

(Q I ) ki = δ i k ,i , 1 ≤ k ≤ m, 1 ≤ i ≤ n (8.8) (Q J ) j = δ j ,j , 1 ≤ ≤ m, 1 ≤ j ≤ n. (8.9)
Each row of these matrices is a Dirac vector, hence they satisfy

Q I 1 n = 1 m and Q J 1 n = 1 m .
We also define the averaged minibatch transport matrix which takes into account all possible minibatch couples.

Definition 19 (Averaged minibatch transport matrix). Consider as in Definition 16 an OT kernel h and 1 ≤ m ≤ n. Given data n-tuples X, Y , consider for each pair of m-tuples I, J, the uniform vector of size m, u m , and let Π I,J be defined as in Definition 18. The averaged minibatch transport matrix is

Π h,m (X, Y ) = (n -m)! 2 n! 2 I,J∈P m Π I,J (8.10)
Similarly to incomplete minibatch OT estimator, we can define incomplete minibatch transport plan estimator. Let two integers k ≥ 1 and 1 ≤ m ≤ n: (8.11) where Π I,J is the lifted n × n OT plan between minibatches.

Π h,m,k (X, Y ) := 1 k (I,J)∈D k Π I,J ,
When there are no possible confusion, we omit the inputs X, Y . The average in the above definition can be expressed as a finite weighted sum of Π I,J . It is therefore well defined for an arbitrary choice of optimal transport plans Π I,J , and we do not need to concern ourselves with the measurability of selection of optimal transport plans. The same will be true whenever an average of optimal transport plans will be taken in the rest of the manuscript, since all results concerning such averages will be nonasymptotic. We will therefore avoid further mentioning this issue, for the sake of brevity. Note also that the Sinkhorn divergence involves three terms, hence three transport plans, which explains why we do not attempt to define an associated averaged minibatch transport matrix.

This construction is consistent with the construction of minibatch OT. Indeed, for an exact OT loss L or a p-Wasserstein distance to the power p, W

h m C (X, Y ) ≥ h(u n , u n , C
). The proof can be found in Appendix A.1.2. The fact that Π h,m (X, Y ) is an admissible transportation plan means that even though it is not optimal, we still transport data similarly to OT. Finally, note that because incomplete U-statistics are not U-statistics in general, the incomplete minibatch transport plan estimator does not define a transport plan between the full measures in general, i.e., their marginals are not equal to uniform probability vectors.

Illustration examples on 1D ans 2D data

In this section, we illustrate the minibatch OT framework on two simple cases. The first case is when data lie in 1D. Thanks to the 1D OT close-formed, detailed in Section 2.2.3, we can express a closed-form for minibatch OT and we illustrate the minibatch OT plan for several values of minibatches. We then illustrate the transport plan for 2D toy data.

Closed-form solution for 1D data. We recall that the solution of the 1D OT problem can be computed with a sorting algorithm which gives an appealing O(n log(n)) complexity compare to the initial O(n 3 log(n)) of a network simplex solver. We assume (without loss of generality) that the points are ordered on the R line. In such a case, we can compute the 1D Wasserstein 1 distance with cost c(x, y) = |x -y| as: W (u, u) = 1 n n i=1 |x i -y i | and the OT matrix is simply an identity matrix scaled by 1 n . Based on the symmetry definition of minibatch OT (8.5) and after a short combinatorial calculus (given in appendix), the 1D minibatch transport plan coefficient between sorted samples (Π h,m ) j,k can be computed as:

(Π h,m ) j,k = 1 m n m -2 imax i=imin j -1 i -1 k -1 i -1 n -j m -i n -k m -i
where i min = max(0, m -n + j, m -n + k) and i max = min(j, k). i min and i max represent the sorting constraints.

We show on the first row of Figure 8.1 the minibatch OT plans Π h,m with n = 20 samples for different values of the minibatch size m. On the second row of the figure a plot of the measures in several rows of Π h,m , to illustrate the number of connections. We give the OT plans for entropic and quadratic regularized OT between full measures for comparison purpose. It is clear from the figure that the OT matrix densifies when m decreases, which is a similar effect to entropic regularization. Regarding the quadratic regularization, the spread of mass is more localized and preserves the sparsity as discussed in [START_REF] Blondel | [END_REF]].

While the entropic regularization spreads the mass in a similar manner for all samples, minibatch OT concentrates the mass at the extremities. Note that the minibatch OT plan solution is for ordered samples and does not depend on the position of the samples once ordered, as opposed to the regularized OT methods. This will be better illustrated in the next example.

Finally, a closed-form is also available in the case of drawing with replacement. We provide it in appendix. Unfortunately, its computational complexity makes it hard to use in practice.

Minibatch Wasserstein in 2D

We illustrate several OT plans between two empirical measures of 10 2D-samples each in Figure 8.2. We consider the MBOT transport plan for several batch sizes, the entropic and quadratic regularized OT between full measures. We use two 2D empirical measures (point cloud) where the samples have a cluster structure and the samples are sorted w.r.t. their cluster. We first discuss the sampling without replacement. We can see from the OT plans in the first row of the figure that the cluster structure is more or less recovered with the regularization effect of the minibatches (and also regularized OT). On the second row one can see the effect of the geometry of the samples on the spread of mass. Similarly to 1D, for Minibatch OT, samples on the border of the simplex cannot spread as much mass as those in the center and have darker rows. This effect is less visible on regularized OT.

The defined formalism supposed we had uniform probability vectors for the full measure and we considered the probability vectors as uniform at the minibatch level. However as justified by Proposition 11 the above formalism would output a transport plan with uniform marginals. Thus in the case of non uniform probability vectors such as in ARWGAN (see Chapter 6), the formalism would not give the

Expectation of Optimal Transport over minibatches 111

correct marginals and a GAN using minibatch OT would generate data similar to all training data. We also supposed that we had a sampling of data without replacement to form a minibatch, which does not correspond to the GAN sampling. In the next section, we extend our formalism to non uniform weights and to other sampling.

Empirical estimators in the general case

In this section, we generalize the minibatch OT formalism that we defined in Section 8.1.1. We aim at defining a formalism which can consider different samplings than sampling without replacement and other probability vectors than the uniform.

We recall that discrete probability measures can either be represented as a sum of Diracs i a i δ xi or with a probability vector and the support of measures (a, X). We chose the latter as it is easier to define formal mathematical objects when we consider discrete probability measures and for consistency with [START_REF] Peyré | [END_REF]]. Indeed sum of Diracs are equal for different indices assignations, i.e.,

n k=1 a i k δ xi k = n k=1 a k δ x k ,
the result of selecting an element with a given index from the minibatch would depend on the order of Diracs. One can define a discrete probability measure from a probability vector a ∈ Σ n and locations X in a canonical way by α n = n i=1 a i δ xi (see remark 2.1 [START_REF] Peyré | [END_REF]]), and we will often implicitly use this assignment throughout the rest of the chapter (see remark 2.11 [START_REF] Peyré | [END_REF]).

To define minibatch OT losses for general probability vectors a, b, a first ingredient is a reweighting function w that takes as inputs a discrete probability a ∈ Σ n and an m-tuple of indices I = (i 1 , . . . , i m ) and outputs a discrete probability vector a I = w(a, I) ∈ Σ m . A second ingredient is a parametric family of distributions {P a : a ∈ Σ n } such that for each a ∈ Σ n , P a is a probability distribution over m-tuples I of indices. The law on probability tuples assures that we have a weighted average of OT kernels and its combination with a suited reweighting function assures all samples are transported. Those ingredients are needed to get unbiased estimator of minibatch OT. Formally, we need: Definition 20 (Reweighting and probability functions). A reweighting function is a map w of the form :

w : (a, I) ∈ Σ n × n m → Σ m (8.12)
A probability function is a map P of the form : (8.13) where P n m is the set of probability distributions over the set of m-tuples I of indices n m .

P : a ∈ Σ n → P a ∈ P n m ,
We are now ready to give a formal definition of minibatch OT losses. The idea is to compute the expectation of the OT kernels over minibatches I, furthermore we need the reweighting functions to assure that the OT kernels has probability vectors as inputs.

Definition 21 (Minibatch Wasserstein). Let C = C n,p (X, Y ) be a matrix of size n × n. Given a kernel h ∈ {L, W p , W p p , L ε , S ε }, two reweighting functions w 1 , w 2 and two probability functions P 1 , P 2 as in (8.12) and (8.13) respectively, we define the minibatch OT loss h m w1,w2,P 1 ,P 2 for any a, b ∈ Σ n by : (8.14) where for I, J two m-tuples, C (I,J) is the matrix extracted from C by keeping the rows and columns corresponding to I and J respectively. Regarding the incomplete estimator, we have:

h m w1,w2,P 1 ,P 2 ,C (a, b) := E I∼P 1 a E J∼P 2 b h w 1 (a, I), w 2 (b, J), C (I,J) ,
h m,k w,P (a, b) := 1 k (I,J)∈D k h w(a, I), w(b, J), C (I,J) (8.15)
where D k is a set of k pairs of m-tuples drawn independently from the joint distribution P a ⊗ P b .

Similarly we have for Gromov-Wasserstein:

Definition 22 (Minibatch Gromov-Wasserstein). Let w 1 , w 2 , P 1 , P 2 be as in Definition 21. We define for two ground costs C 1 = C n,p (X, X) and C 2 = C n,p (Y , Y ). The minibatch Gromov-Wasserstein is defined as: (8.16) where C 1 (I,I) (resp. C 2 (J,J) ) is the matrix extracted from C 1 (resp. C 2 ) by keeping the rows and columns corresponding to I and I (resp. J and J).

GW m w1,w2,P 1 ,P 2 ,C 1 ,C 2 (a, b) := E I∼P 1 a E J∼P 2 b GW w 1 (a, I), w 2 (b, J), C 1 I,I , C 2 J,J ,
Like in the uniform probability vectors case, the dependence of minibatch OT in the ground cost C will often be omitted when there is no possible confusion. When the reweighting functions and the probability laws on tuples are the same (equal to w and P respectively), we use the following shorthand notations for the above losses : h m w,P . With a slight abuse of notation, we also use the notation h m w,P for the GW loss. Regarding the incomplete estimator, the notable difference is that now the minibatches are drawn according to P a ⊗ P b .

The loss h(a, b) corresponds to an averaged optimal transport distance between sub-probability distributions of input probability distributions a and b. The minibatch OT losses define weighted Ustatistics and V-statistics [J Lee 2019] where the weights depend on the input probability vectors a, b and on the laws over index m-tuple P a , P b . This connection turns out to be central to get quantitative statistical results. Concrete versions of these minibatch OT losses are obtained by specifying its ingredients h, w, and P . We now give a few examples of some reweighting functions and families of distributions. For instance, consider four X = (x 1 , x 2 , x 3 , x 4 ) data with weights a = ( 1 4 , 1 8 , 1 8 , 1 2 ), if one picks the batch I = (2, 4), the reweighting functions give w W (a, I) = [ 1 5 , 4 5 ] and w U (a,

I) = [ 1 2 , 1 2 ].
In the case of a uniform discrete probability u ∈ Σ n , the two reweighting functions are identical.

Regarding the parametric law on indices, which gives the probability to pick a given batch of samples, we focus on two constructions depending whether sampling is done with or without replacement. Indeed in practice, most of work use a sampling without replacement, and it is easy to design this case with our formalism. We first consider sampling with replacement. (8.17) Now we give an example of drawing without replacement. The idea is to give a zero probability to pick a batch with repeated indices.

Expectation of Optimal Transport over minibatches 113

Example 4 (Drawing indices "without replacement"). Given a discrete probability distribution a ∈ Σ n , it is also possible to draw distinct indices i ∈ n , 1 ≤ ≤ m, by defining P W a (I) = 0 if the m-tuple I has repeated indices, otherwise

P W a (I) = 1 m (n -m)! (n -1)! i∈I a i . (8.18)
With a uniform discrete probability, a i = 1 n , 1 ≤ i ≤ n, this law corresponds to drawing the m-tuples without repeated indices I uniformly at random among all possible m-tuples without repeated indices, i.e., drawing the m indices i p without replacement. By abuse of language, we will sometimes refer to this law as a draw "without replacement" even for non uniform a.

Note that the minibatch OT between uniform probability vectors in Definition 16 can be recovered with the sampling P W a and the reweighting functions w W , w U .

Minibatch transport plan.

Regarding the transport plan, we can also generalize the formulation (8.10). where Π I,J is the lifted n × n OT plan between minibatches. For brevity this is simply denoted Π h,m w,P when w = w 1 = w 2 and P = P a = P b are clear from context. We can also generalize the incomplete transport plan as folllows:

Definition 23 (Averaged minibatch transport matrix). Consider as in

Π h,m,k w,P (a, b) := 1 k (I,J)∈D k Π I,J . (8.20)
While the n × n matrix defined in (8.19) is candidate to be transport plan between a and b, we need to check if it is indeed admissible, i.e., if it has the right marginals. This is why it is a priori only called an averaged minibatch transport matrix.

Proposition 12. If the reweighting function w and the parametric distribution on m-tuples P c satisfy the following admissibility condition

E I∼Pc Q I w(c, I) = c, ∀c ∈ Σ n (8.21)
Then with the notations of Definition 19, the averaged minibatch transport matrix Π Lemma 8.1.1 (Admissibility). The uniform reweighting function w U and the parametric law "with replacement" P U satisfy the admissibility condition (8.21). The admissibility condition also holds for the parametric law without replacement P W with the normalized reweighting function w W . In contrast for w U , P W when a is not uniform, the resulting OT matrix is not a transportation plan. ) with law P U a and reweighting function w U stands for the minibatch Wasserstein loss (respectively minibatch OT plan) over the m-tuples I.

We now justify empirically that our general formalism can deal with non uniform probability vectors. To this end, we consider generating misclassified data for a pre-trained classifier using the reweighted distribution from ARWGAN (see Chapter 6). To generate data, we use a GAN with minibatch OT loss as loss function. The ground cost is the squared Euclidean cost and we use either the uniform or the general formalism h m U . We follow the experimental setting from Section 6.3.1 and the results can be found in Figure 8.3. It shows that our uniform formalism generates data following the empirical data while our general formalism generates data which are misclassified by the pre-trained classifier.

Drawing data with or without replacement

Our general flexible formalism allows us to define several minibatch strategies by playing with the probability law on tuples. The laws can be also different between the source and the target distributions. (u, u) (top) and the shape of the distributions on the rows of the OT plans (bottom). We consider the exact optimal transport cost as h.

In particular, as given in examples, the cases of drawing with or without replacement. An estimator based on sampling without replacement is the most common practice when we have access to n samples. While this drawing has been investigated for minibatch OT losses, the case with replacement, which appears in the GANs formalism, remains an open question that we aim at answering. The minibatch OT losses represent a weighted sum of Wasserstein distance over batches of size m. In the case of sampling without replacement, they are generalized unbiased U-statistics while with a sampling with replacement, we get generalized biased V-statistics [J Lee 2019, [START_REF] Gretton | [END_REF]]. Precisely, they are two sample U-statistics or V-statistics of order 2m (see [J Lee 2019]) and h is a U-statistic kernel. Interestingly, similar biased and unbiased estimators have been designed to estimate MMD [START_REF] Gretton | [END_REF]].

Finally an important parameter is the value of the minibatch size m. In the case of sampling without replacement, we remark that the minibatch procedure allows us to interpolate between OT, when m = n and averaged pairwise distance, when m = 1. This property is not shared by the sampling with replacement. Indeed when m = n, it does not correspond to original OT due to the repetition of data. It only converges to the true OT when n → ∞. This effect is illustrated in Figure 8.4. We consider the same setting as in Figure 8.2 but with 5 empirical data. On each column we show the transport plan and the shape of connection between samples. We can see that the estimator Π h,m U (u, u) has always a denser plan, i.e. a bigger number of connections, than the estimator Π h,m W (u, u). In particular, when m = n = 5, we get the optimal transport plan with Π h,m W (u, u) while we do not recover it with Π h,m U (u, u) due to the fact that samples can be repeated. Now that we have rigorously defined how we can build minibatch Wasserstein losses between empirical measures, we study its loss properties.

Debiased minibatch Optimal Transport: a Sinkhorn divergence approach

In this section, we introduce a new OT cost based on minibatch OT. It follows the form of the Sinkhorn Divergence. We start by motivating this loss function by reviewing the distance properties of minibatch OT. After, we introduce our loss function to recover some loss of minibatch OT and finally we review its positivity.

Chapter 8. Minibatch Optimal Transport

Metric properties: a fundamental difference

The optimal transport cost can define a distance between probability distributions with some specific ground cost. It is of interest to see if these properties remain true for minibatch OT. Answering this question is the purpose of this section and we start with the following proposition.

Proposition 13 (Estimator properties). The minibatch OT losses enjoy the following properties:

• The losses are symmetric • The losses are not distances Proof. We give the proof that minibatch OT losses are not distances. Consider a uniform probability vector and random 3-data tuple X = (x 1 , x 2 , x 3 ) with distinct vectors. As h m w,P is a weighted sum of positive terms, it is equal to 0 if and only if each of its term is 0. But consider the minibatch term I 1 = (i 1 , i 2 ) and I 2 = (i 1 , i 3 ), then obviously h(w(u, I 1 ), w(u, I 2 ), C(X(I 1 ), X(I 2 ))) = 0 as x 2 = x 3 , finishing the proof.

The symmetry of the losses is inherited from the optimal transport problem which is itself symmetric. The loss of the separability distance axiom means that for data fitting problems, the final solution will not match the target distribution. The axiom is recovered for minibatches without replacement when m = n as we recover the original OT formulation.

We defined minibatch OT losses and reviewed their basic properties. In what follows, we propose an elegant formulation which fixes this loss.

Debiasing minibatch Wasserstein losses

As we have shown before, the minibatch OT losses are not distances, for general probability vectors and data n-tuple X, h m w,P (b, b) > 0. This leads to an undesirable situation when one uses it for learning purposes as the final solution is not the target distribution but a shrunk version of it. Hence, we would like to debias the losses to get h Definition 24 (Debiased Minibatch Wasserstein estimators). Let C = C n,p . Consider 1 ≤ m ≤ n be an integer and h be the OT cost L, the entropic loss L ε , the Sinkhorn divergence S ε , or the Gromov-Wasserstein distance GW for some ground cost c(x, y), we define the following quantities:

Λ h,m w,P,C(X,Y ) (a, b) := h m w,P,C(X,Y ) (a, b) - 1 2 h m w,P,C(X,X) (a, a) + h m w,P,C(Y ,Y ) (b, b) , (8.23)
That we note when it is clear of context Λ h,m w,P and its incomplete counter part:

Λ h,m,k w,P,C(X,Y ) (a, b) := h m,k w,P,C(X,Y ) (a, b) - 1 2 h m,k w,P,C(X,X) (a, a) + h m,k w,P,C(Y ,Y ) (b, b) . (8.24)
Remark 8. We keep making the slight abuse of notation to consider all OT kernels with h, but we explicit the loss Λ for a Gromov-Wasserstein loss. We note the ground cost C n,p,1 and C n,p,2 as C 1 and C 2 for sake of readability. With the GW kernel, the loss Λ is equal to: It is straight forward to see that Λ h,m w,P (b, b) = 0. A similar loss has been proposed in [START_REF] Salimans | [END_REF]] as a generalized energy distance using the entropic Wasserstein distance as metric. While their loss debiased the minibatch bias, it still had a bias from the entropic regularization. They then relied on the energy distance properties to argue positiveness. The downside of this loss is that it needs to rely on a metric to be positive, however the entropic regularized optimal transport is not a metric between probability distributions as W ε (a, a) = 0.

Λ h,m w,P,C 1 (X,X),C 2 (Y ,Y ) (a, b) := h m w,P,C 1 (X,X),C 2 (Y ,Y ) (a, b) - 1 2 h m w,P,C 1 (X,X),C 1 (X,X) (a, a) + h m w,P,C 2 (Y ,Y ),C 2 (Y ,Y ) (b, b) , ( 8
We bring insights to this debiased loss and compare its differences to the minibatch OT losses both mathematically and empirically. We use our loss Λ h,m w,P with the Wasserstein distance and Sinkhorn divergence because they respect the distance separability axiom. Unfortunately, we prove that even if we consider the Wasserstein distance, this loss is not positive and we will give counter examples.

Positivity: a negative counter example

The loss function Λ h,m w,P is composed of three terms of the form of h The loss function might not be always positive for particular case but in practice, we always had a positive loss and it performed better than the biased minibatch OT losses. Furthermore, while we have been able to find counter examples for p ≥ 2, we have not found any counter example for W 1 . Hence we conjecture that Λ W1 might be a positive loss function, and the proof is left as future work, as we did not succeed to formally proving it.

Learning with minibatch Optimal Transport: concentration bounds and gradients

In this section, we present different deviation bounds for minibatch OT. We study the case where input probability measures have compact supports or are sub-Gaussian probability measures. We also study the case where we have a bounded cost. We then review the optimization properties of minibatch OT and see if it can be minimized with modern optimization techniques.

Concentration bounds between estimators and their expectations

We want to find similar results to the U-statistics concentration bounds (see Theorem 7.3.2) for our estimator h m,k w,P (a, b) and its expectation E h m,k w,P (a, b). We will give a maximal deviation bound for several scenarios. For bounded measures, we will prove that we have a Hoeffding-type inequality. Then we relax the boundness condition on probability measures to give a concentration bound for sub-Gaussian probability measures. Finally, we give a deviation bound in the case of a bounded cost and general probability measures.

In this context, the probability vectors (a (n) ) and (b (n) ) are sequences which depend on the number of data n. More precisely (a (n) ) and (b (n) ) are sequences of vectors such that for each n ∈ N, a (n) , b (n) ∈ Σ n , we denote the space of these sequences as (a (n) ), (b (n) ) ∈ Σ. The sequences of probability vectors (a (n) ) and (b (n) ) can not be taken arbitrarily if we want to guarantee convergence. Indeed, they appear in the expectations over the index m-tuples in h m w,P and we need to bound them to apply Hoeffding's lemma. Hence we rely on local constraints that the probability vectors a and b must verify.

Definition 25 (Local averages conditions). Let (a (n) ) ∈ Σ and two integers n, m ∈ N * such as n ≥ m. We say that (a (n) ) verifies the local mean condition if there exists a constant D > 0 and γ ∈ (0, 1] such that for any n ∈ N * and I ⊂ n with |I| = m we have:

1 m i∈I a (n) i ≤ D n γ . (8.26)
We write that (a n) ) is said to verify the local geometric mean condition if there exists a constant D > 0 and γ > 0 such that for any n ∈ N * and I ∈ n m we have

(n) ) satisfies Loc A (m, γ, D)) (or Loc A (m, γ)) when the constant D is implicit). (ii) Analogously, (a (
Π i∈I a (n) i 1 m ≤ D n γ . (8.27)
We write that (a

(n) ) verifies Loc G (m, γ, D)) (or Loc G (m, γ)) when the constant D is implicit).
Intuitively, these conditions require that the product or the sum of any m elements of (a (n) ) do not get too big. A straight forward example is the uniform vector u (n) which respects Loc A (m, 1, 1) for the local mean condition and Loc G (m, 1, 1) for the local product condition. Thus Eq.(8.26) naturally extends and quantifies the fact that a sequence has uniformly controlled m-averages. We also observe that for any generic sequence (a (n) ) in Σ verifies Loc A (m, 0, 1 m ). Regarding the local product condition, Eq.(8.27) extends the fact that a sequence has uniformly controlled m-products. We illustrate the local constraints on the simplex in Figure 8.6 with python ternary [START_REF] Harper | marcharper/python-ternary: New Features and Bug Fixes[END_REF]]. We have the following result about the local constraints:

(i) If (a (n) ) verifies Loc A (m, γ, D) or Loc G (m, γ, D) then γ ≤ 1. (ii) If (a (n) ) is Loc A (m, γ, D) then (a (n) ) is Loc G (m, γ, D).
Bounded data. For bounded data, we show that in order to obtain reasonable convergence properties of the estimators h m w,P (a (n) , b (n) ) we need to ensure that the sequences (a (n) ) and (b (n) ) verify the local condition with enough decay, e.g. (a (n) ), (b (n) ) are Loc A (m, γ) or Loc G (m, γ) for a γ sufficiently close to 1. Theorem 8.3.1 (Maximal deviation bound for compactly supported distributions). Let δ ∈ (0, 1), k 1 an integer and m 1 be a fixed integer. Let C = C n,p be as in (8.2). Consider two compactly supported distributions α, β, two n-tuples of empirical i.i.d. data X ∼ α ⊗n , Y ∼ β ⊗n and a kernel h ∈ {W p , W p p , L ε , S ε , GW }. Let the reweighting function w and the probability law over m-tuple P be as in Examples 1,2,[START_REF]6 Samples generated by BigGAN model at 512×512 resolution[END_REF][START_REF]1 Example of adversarial example for a GoogLeNet[END_REF] Let the sequences of probability vectors (a

(n) ) ∈ Σ and (b (n) ) ∈ Σ satisfy Loc A (m, γ, D) and let D > 0 and γ ∈ ( 3 4 , 1].
We have a deviation bound for the sampling without replacement between h m,k w,P (a (n) , b (n) ) and Eh m w,P (a (n) , b (n) ) depending on the number of empirical data n and the number of batches k:

P   | h m,k W (a (n) , b (n) ) -Eh m W (a (n) , b (n) )| ≥ M 2 D 2 m 1 2 n 2(γ-3 4 ) 2 log( 2 δ ) + 2 log( 2 δ ) k   ≤ δ, (8.28)
where M is a constant depending on the diameters of distribution supports. And for the sampling with replacement, let the sequences of probability vectors (a (n) ), (b (n) ) verify Loc G (m, γ, D) for some 120 Chapter 8. Minibatch Optimal Transport γ ∈ (1 -1 4m , 1] and D > 0, we have:

P   | h m,k U (a (n) , b (n) ) -Eh m U (a (n) , b (n) )| ≥ M 2 D 2m m 1 2 n 2m( 1 4m -1+γ) 2 log( 2 δ ) + 2 log( 2 δ ) k   ≤ δ.
(8.29) Remark 9. In the case of uniform measures, we recover the sampling without replacement bounds of Section 8.1.1 for both sampling with or without replacement:

P | h m,k W (u (n) , u (n) ) -Eh m W (u (n) , u (n) )| ≥ M 2 2 m n log(2/δ) + 2 log(2/δ) k ≤ δ, P | h m,k U (u (n) , u (n) ) -Eh m U (u (n) , u (n) )| ≥ M 2 2 m n log(2/δ) + 2 log(2/δ) k ≤ δ.
The proof is based on the U-statistics concentration bound proof [START_REF] Hoeffding | Probability Inequalities for Sums of Bounded Random Variables[END_REF]] and can be found in Appendix A.1.2 with the proof of constant M . The proof idea is to rewrite our minibatch OT losses as a sum of independent terms and then to apply Hoeffding's lemma to the rewritten sum. The local constraints were necessary for a generalization of the concentration bounds to non uniform probability vectors a (n) and b (n) . These concentration bounds are also valid for our debiased minibatch OT loss as it is composed of three terms of the form h. Furthermore, it is possible to extend this concentration inequality with an expectation over the batch couples and empirical data.

Corollary 1. With the same hypothesis and notations as in Theorem 8.3.1. The following inequality holds:

E[| h m,k W (a (n) , b (n) ) -Eh m W (a (n) , b (n) )|] 20 • M max 2 √ 2D 2 m 1 2 n 2(γ-3 4 ) , 2 k , (8.30) E[| h m,k U (a (n) , b (n) ) -Eh m U (a (n) , b (n) )|] 20 • M max 2 √ 2D 2m m 1 2 n 1/2-2m+2mγ , 2 k . (8.31)
And for our debiased minibatch OT loss:

E[| Λ h,m,k W (a (n) , b (n) ) -E Λ h,m,k W (a (n) , b (n) )|] 40 • M max 2 √ 2D 2 m 1 2 n 2(γ-3 4 ) , 2 k , (8.32) E[| Λ h,m,k U (a (n) , b (n) ) -E Λ h,m,k U (a (n) , b (n) )|] 40 • M max 2 √ 2D 2m m 1 2 n 1/2-2m+2mγ , 2 k . (8.33)
This deviation bound shows that if we increase the number of data n and batches k while keeping the minibatch size fixed, we get closer to the expectation. Remarkably for all OT kernel h, the bound does not depend on the dimension of X , which is an appealing property when data lie in high dimensional space. A similar property was proven but only for W p p (see proposition 20, [Weed 2019]). Another nice property of the bounds above is that for a fixed minibatch size m, if one chooses k proportional to the number of samples, the convergence of h m,k w,P to its mean is in O(n -1/2 ) for a O(n) computational complexity. Now let us consider a small experiments. To illustrate the dependence to the dimension, we consider 2 empirical data n-tuple, X and Y ∼ α ⊗n , where α is the uniform distribution on the unit cube [0, 1] d , and compute Λ h (u, u) as a function of n. For a first experiment, we fix the batch size m = 128 and we consider several values of dimension d. For a second experiment, we now fix the dimension d and consider several batch sizes. Both experiments highlight no dependence of Λ h (u, u) to the dimension. To the best of our knowledge, it is the first time that a loss using the exact Wasserstein distance has no dependence on the dimension, making it a good candidate for learning problems.

We gave concentration bounds in the bounded data case and now we extend these results to the unbounded data case.

Unbounded data. We supposed in the previous results that the distributions have a bounded support. We can relax this condition by supposing they have light tails, i.e., they are sub-Gaussian. We consider the Euclidean norm ( . 2 ) and give a formal definition: Definition 26 (sub-Gaussian random vectors). A random vector x ∈ R d is sub-Gaussian, if there exists σ ∈ R so that:

Ee v,x-Ex ≤ e y 2 σ 2 2 , ∀y ∈ R d
The proof uses a related class of sub-Gaussian random vectors and a discussion of the difference is available in appendix. In the case of sub-Gaussian data, we can not rely on the Hoeffding inequality anymore as the data are not bounded. However we are able to get a similar concentration inequality. Hereafter we write A γ B for A, B > 0 if there exists a large constant τ = τ (γ) > 0 such that A ≤ τ B.

Theorem 8.3.2 (Concentration inequality sub-Gaussian data). Let the cost C = C n,p be defined as in (8.2). Let (x i ) 1≤i≤n and (y i ) 1≤i≤n be two i.i.d. sequences of random vectors such that

x 1 ∈ normSG(ρ x , σ 2 x ) and y 1 ∈ normSG(ρ y , σ 2 y ) with σ x , σ y > 0 and ρ x , ρ y ∈ R d . Let us introduce σ := min(σ x , σ y ) ρ := ρ x -ρ y 2
Let the sequence probability vectors (a (n) ), (b (n) ) verify Loc A (m, γ, D) for some γ ∈ ( 3 4 , 1] and D > 0. We assume that n verifies the following condition:

n ≥ τ (m, σ, ρ, D, p).
(8.34)

Chapter 8. Minibatch Optimal Transport

Consider m 1 be a fixed integer and a kernel h ∈ {W p , W ε , S ε }. Let the reweighting function w W and the probability law over m-tuple P W be as in Examples 2 and 4. Then we have the following concentration bound for the sampling without replacement:

P h m W (a (n) , b (n) ) -Eh m W (a (n) , b (n) ) ≥ (2 3p+4 m) 1 2 σ p D 2 • log(4n) p+1 2 n 2(γ-3 4 ) ≤ 4n -1 2 p , ( 8.35)
The proof can be found in Appendix A.1.3. It uses a truncation argument where we split data which lie in a compact and data which do not. We can remark the following facts about our bounds in the unbounded case:

• The decay and the constants in (8.35) are artificial consequences of the constants chosen in the proof.

• The condition (8.34) seems to be necessary. In the bounded case of Theorem 8.3.1 such a condition was not needed.

• We loose a log(n) factor between (8.35) and the deviation bound between the complete estimator h and its mean from the compactly supported data case. We report to a longer discussion on this comparison in Appendix A.1.3.

Bounded cost.

We finish this section by discussing the case where we have a bounded cost. In this case, it is possible to get deviation bounds of the form of Theorem 8.3.1 for any distribution. This case is not unrealistic because it is easy to to bound a cost. For instance, consider an unbounded ground cost c(x, y), then you can consider the ground cost c(x, y) = c(x,y) 1+c(x,y) as a new ground cost which is bounded. Furthermore if c is a distance, then c is also a distance. Thus we have: Theorem 8.3.3 (Maximal deviation bound for bounded cost). Let δ ∈ (0, 1), k 1 an integer and m 1 be a fixed integer. Consider two distributions α, β possibly unbounded, two n-tuples of empirical data X ∼ α ⊗n , Y ∼ β ⊗n and a kernel h ∈ {W p , W p p , L ε , S ε , GW } with a bounded ground cost C. Let the reweighting function w and the probability law over m-tuple P be as in Examples 1,2,[START_REF]6 Samples generated by BigGAN model at 512×512 resolution[END_REF][START_REF]1 Example of adversarial example for a GoogLeNet[END_REF] Let the sequences of probability vectors (a (n) ) ∈ Σ and (b (n) ) ∈ Σ satisfy Loc A (m, γ, D) and let D > 0 and γ ∈ ( 3 4 , 1]. We have a deviation bound for the sampling without replacement between h m,k w,P (a (n) , b (n) ) and Eh m w,P (a (n) , b (n) ) depending on the number of empirical data n and the number of batches k:

P   | h m,k W (a (n) , b (n) ) -Eh m W (a (n) , b (n) )| ≥ M 2 D 2 m 1 2 n 2(γ-3 4 ) 2 log( 2 δ ) + 2 log( 2 δ ) k   ≤ δ, (8.36)
where M is an upper bound on the ground cost. And for the sampling with replacement, let the sequences of probability vectors (a (n) ), (b (n) ) verify Loc G (m, γ, D) for some γ ∈ (1 -1 4m , 1] and D > 0, we have:

P   | h m,k U (a (n) , b (n) ) -Eh m U (a (n) , b (n) )| ≥ M 2 D 2m m 1 2 n 2m( 1 4m -1+γ) 2 log( 2 δ ) + 2 log( 2 δ ) k   ≤ δ.
(8.37)

Learning with minibatch Optimal Transport: concentration bounds and gradients 123

After studying concentration bounds for the minibatch OT loss in the bounded, unbounded data and bounded cost cases, we give similar bounds for the minibatch OT plan.

Minibatch Transport plan.

As discussed before an interesting output of Minibatch Wasserstein is the minibatch OT plan Π h,m w,P , but since it is hard to compute in practice, we instead use Π h,m,k w,P and we investigate the error on the marginal constraints. In what follows, we denote Π (i) the i-th row of matrix Π and 1 ∈ R n the vector whose entries are all equal to 1. Theorem 8.3.4 (Distance to marginals). Let δ ∈ (0, 1), two integers m ≤ n and consider two sequences of probability vectors (a

(n) ), (b (n) ) ∈ Σ. Let a ground cost C = C m,p be as in (8.2) for some p ≥ 1. Consider an OT kernel h ∈ {W p , W p p , W ε , S ε , GW }.
. Suppose now that the probability law over m-tuples P and the reweighting function w, as defined in (8.12) and (8.13), satisfy the admissibility condition (8.21). For all integers k 1 and all integers 1 i n, we have:

P Π h,m,k w,P (a (n) , b (n) ) (i) 1 -a (n) i ≥ 2 log(2/δ) k ≤ δ (8.38)
The proof uses the convergence of Π h,m,k w,P to Π h,m w,P and the fact that Π h,m w,P is a transport plan and respects the marginals. It is far easier to get this result as we always have bounded transport plan. Let us now study the convergence of stochastic algorithm when minimizing MBOT.

Unbiased gradients for stochastic optimization

Thanks to the statistical properties of our estimator, we now know that minibatch OT losses concentrate well around their expectation. We now study their behaviour with modern optimization techniques. We consider the same data fitting problem as in Section 7.2.2. We recall that we aim at finding the minimum θ = arg min θ∈θ h(α n , β θ , c), (8.39) where α n is an empirical measure, β θ a parametrized measure, h an OT loss and c a given ground cost. One way to compute the estimator θ given by (8.39), is to use a stochastic solver for semidiscrete optimal transport [Peyré 2019, Chapter 5]. This strategy is unfortunately not efficient in practice [Genevay 2016, Seguy 2018]. A common alternative approach is to use stochastic gradient descent which were computed between minibatches sampled from α, β θ [START_REF] Genevay | [END_REF][START_REF] Salimans | [END_REF]]. We want to prove that the SGD algorithm would converge towards the correct θ. Unfortunately, OT is not differentiable with respect to the ground cost C as shown in Section 7.2.2. We thus rely on Clarke gradients defined in Section 7.2.2. The full formal statement and proof of those theorems can be found in Appendix A.1.5.

Theorem 8.3.5. Let a, b ∈ Σ m . Let X be a R dm -valued random variable, and {Y θ } a family of R dm -valued random variables defined on the same probability space, indexed by θ ∈ Θ, where

Θ ⊂ R q is open. Assume that θ → Y θ is C 1 . Denote C = C m,p for some p ≥ 1 and let h ∈ {L, L ε }. Then the function θ → -h(a, b, C(X, Y θ )
) is Clarke regular and for all 1 ≤ i ≤ q we have:

∂ θi h(a, b, C(X, Y θ )) = co{-tr(P • D T ) • (∇ θi Y ) :P ∈ Π(h, C(X, Y θ ), a, b), (8.40) D ∈ R m,m , D j,k ∈ ∂ Y C j,k (X, Y θ )}
where ∂ θi is the Clare subdifferential with respect to θ i , ∂ Y C j,k is the subdifferential of the cell C j,k of the cost matrix with respect to Y , co denotes the closed convex hull and Opt(h, C, a, b) is defined in Definition 18. For h = GW , when the cost matrix is differentiable (that is p > 1), the function

-h(a, b, C(X, X), C(Y θ , Y θ )
) is also regular, and an analogous formula holds.

Theorem 8.3.6. Let a, b, X, Y be as in Theorem 8.3.5, and assume in addition that the random variables X, {Y θ } θ∈Θ have finite p-moments. For h ∈ {L, L ε }, under an additional integrability assumption, we have:

∂ θ E [h(a, b, C(X, Y θ ))] = E [∂ θ h(a, b, C(X, Y θ ))] . (8.41)
with both expectation being finite. Furthermore the function θ

→ -E [h(a, b, C(X, Y θ ))
] is also Clarke regular. An analogous result holds for h = GW , given that the cost is differentiable (that is p > 1) and random variables X, {Y θ } have finite 2p-moments.

Remark 10. If the cost matrix in Theorem 8.3.5 is differentiable with respect to Y (that is for p > 1) and h = W ε , then all the Clarke derivatives in (8.40), (8.41) are sets consisting of one element, which is the gradient of respective functions. In that case we may deduce for h = S ε a formula for the gradient from Theorem 8.3.5 and an interchange of expectation and integration from Theorem 8.3.6.

Suppose that in the above theorem the random variable X is distributed according to α ⊗m , each random variable Y θ is distributed according to β ⊗m and X is independent of family of variables {Y θ } θ∈Θ . Then Theorem 8.3.5 implies that it is easy to compute unbiased stochastic gradients of a Minibatch OT loss, defined as follows:

Definition 27 (Minibatch OT). Let α, β ∈ P p (R n ) be two measures on an Euclidean space with finite p-moments, for p ≥ 1. Chose an integer m ∈ N and let h ∈ {L, L ε , S ε }. Given the ground cost the ground cost C m,p defined in Eq.( 8.2), we define the following quantity:

E m h (a, b, α, β) := E (X,Y )∼α ⊗m ⊗β ⊗m h m w,P,C m,p (X,Y ) (a, b) (8.42)
for any a, b ∈ R n . We define an analogous quantity For h = GW . Assuming that α, β ∈ P 2p (R n ), we denote

E m h (a, b, α, β) := E (X,Y )∼α ⊗m ⊗β ⊗m h m w,P,C m,p (X,X),C m,p (Y ,Y ) (a, b) (8.43) for any a, b ∈ R n .
The fact that the above is well defined follows trivially from the assumption that measures α, β have finite p-moments (or finite 2p-moments for h = GW ) and a standard bound (A.92) used in the proof of Theorem 8.3.6. In fact, the finiteness of (8.42) and (8.43) is shown in that proof. We finish this section by noting, that Theorem 8.3.6 implies that if we use the Minibatch OT loss with h ∈ {L, L ε , GW } (or h = S ε for p > 1) as a contrast function in (7.5), then the objective function is minus Clarke regular. In this case, it is known that SGD with decreasing step sizes converges almost surely to the set of critical points of Clarke generalized derivative [START_REF] Davis | [END_REF]], [START_REF] Majewski | [END_REF]].

Accelerating stochastic methods Other solvers than SGD have been discussed to optimize Ustatistics in [START_REF] Papa | [END_REF]]. It is not realistic in our problem to rely on variance reduced algorithms such as SAG [Schmidt 2017b]. The later needs to store the past gradients, which leads to high memory cost, while the former needs to compute the full gradients on a regular basis, which leads to a high computation cost. However as we have a huge number of terms, one might hope to accelerate its convergence by considering a smaller number of data but it leads to poorer results at convergence.

Numerical experiments on generative modelling, color transfer and minibatch

Gromov-Wasserstein 125

Numerical experiments on generative modelling, color transfer and minibatch Gromov-Wasserstein

After presenting the formalism of minibatch OT, studied its statistical and optimization properties and defining a new loss function, we now explore different applications of our methods. To compare the minibatch OT losses and their debiased counter parts, we set three qualitative experiments and two quantitative ones. The first experiment is a gradient flow between male and female images and the second is a Monge map estimation between male and female images. The quantitative experiment consists in learning a GAN where we investigate the inception score of several minibatch OT losses. Our fourth experiment is a color transfer experiment, we also investigate the sparsity degree of the resulting minibatch OT plan. Finally, our two last experiments are dedicated to the minibatch Gromov-Wasserstein loss where we investigate the inherited properties from the Gromov-Wasserstein distance. As our experiments are learning scenarios, we have uniform measures and consider the reweighting function w U , regarding the probability laws on tuples, we investigate both P W and P U . Note that w U and P W check the admissibility condition from Proposition (12). Finally, experiments were computed on a single GTX Titan GPU.

Gradient flows between human faces with minibatch Optimal Transport

The first experiment we conducted is a gradient flow of a source distribution towards a target distribution. It corresponds to the nonparametric setting of a data fitting experiments such as GANs. For two given probability vectors a and b, and support X, Y associated to b, the goal of gradient flows is to model a support x t associated to a which at each iteration follows the loss gradient x t → h(a, b, C(X t , Y )). This experiment has been investigated in [START_REF] Liutkus | [END_REF], Peyré 2015]. We apply it between male and female images from the celebA dataset [Liu 2015] where we seek a natural evolution along iterations. CelebA is a large-scale face attribute dataset with 202,599 face images, 5 landmark locations, and 40 binary attribute annotations per image. We only considered 5000 male images and 5000 female images. We build the training dataset by cropping and scaling the aligned images to 64 x 64 pixels. Following the procedure in [START_REF] Feydy | [END_REF]], the gradient flow algorithm uses an Euler scheme and we start from an initial distribution at time t = 0. At each iteration we numerically integrate the ordinary differential equation: ,b). As our losses take probability vectors as inputs, we need to correct an inherent scaling when we calculate the gradient. The scaling comes from the sample weights a i , which is equal to 1/m in our case. To correct the scaling, we apply a re-scaling to the gradient equal to m. Finally, for a n-tuple of data X we integrate:

Ẋ(t) = -∇ x Λ h,m,k w,P,C(X(t),Y ) (a
Ẋ(t) = -m∇ x h m,k w,P,C(X(t),Y ) (a, b) - 1 2 h m,k w,P,C(X(t),X(t)) (a, a) + h m,k w,P,C(Y ,Y ) (b, b) . (8.44)
We conducted gradient flow experiments for both minibatch OT loss and debiased minibatch OT loss with the Wasserstein distance as OT kernel and probability laws on m-tuples P W and P U . However, as our images lie in high dimension, the Euclidean ground cost is not meaningful anymore, that is why we followed the experiments of [START_REF] Liutkus | [END_REF] where they considered gradient flows in the latent space of a pre-trained AutoEncoder. We considered a pre-trained DFC-VAE [START_REF] Hou | [END_REF]] with 64 × 64 image and perform gradient flow in the encoder's latent space. In addition of the typical [START_REF] Kingma | [END_REF]] loss, DFC-VAE considers the difference between features of the input image and the reconstructed image through a pre-trained neural network. In our case, we considered a pre-trained VGG-19 network and the

Generative adversarial networks on Cifar-10 with minibatch Optimal Transport

In this experiment, we want to learn a GAN using our new loss function which is a debiased version of minibatch SD. We recall that the objective of a GAN is to train a neural network G θ that can generate realistic data which are close to real data X. For the ground cost of the Wasserstein distance, we could rely on an Euclidean cost between images. Unfortunately, using an Euclidean cost on high dimensional images generates blurred versions of the real images [START_REF] Aggarwal | [END_REF][START_REF] Wang | [END_REF],Kulis 2013a]. Hence, we will learn adversarially a critic network f ϕ which extracts meaningful feature vectors for input images. Then we will apply the Euclidean distance between the encoded generated data f ϕ (G θ (z j )) and encoded real data f ϕ (x i ). Other methods relied on a feature extractor such as MMD GAN [Li 2017a] of Sinkhorn GAN [START_REF] Genevay | [END_REF]]. We can summarize our learning problem as the following:

min θ max ϕ EΛ h,m w,P,Cϕ(X,G θ (Z)) (u, u), (8.46) where C ϕ (x i , y j ) def. = f ϕ (x i ) -f ϕ (y j ) 2 and f ϕ : X → R p .

Numerical experiments on generative modelling, color transfer and minibatch Gromov-Wasserstein 129

Generator Critic INPUTS: 100 INPUTS: 3 × 32 × 32 Conv2D nc=256 k=4 stride=1, BN, ReLU Conv2D nc=64 k=4 stride=2, LReLU(slope=0.2) Conv2D nc=128 k=4 stride=2, BN, ReLU Conv2D nc=128 k=4 stride=2, LReLU(slope=0.2) Conv2D nc=64 k=4 stride=2, BN, ReLU Conv2D nc=256 k=4 stride=2, LReLU(slope=0.2) Conv2D nc=3 k=4 stride=2, TanH Conv2D nc=100 k=4 stride=1, LReLU(slope=0.2) Where

G θ (Z) = {G θ (z 1 ), • • • , G θ (z m )} and u ∈ R m .
We train GAN for image generation of CIFAR-10 data [Krizhevsky ]. The number of data is 50K of size 32 × 32. Regarding the implementation detail, we consider the same setting as [Li 2017a[START_REF] Genevay | [END_REF]]. The input noise is of dimension 100. The generator and the critic have 4 convolution layers (full detail in Table 8.1). We clip the parameters of the critic in order to have a Lipschitz constant bounded by 1 as done in [Li 2017a[START_REF] Genevay | [END_REF]]. The batch size m we considered is 64 and we set the number of batch couple to k = 1 for each SGD update. The optimizer we used is RMSProp [Tieleman 2012a] with a learning rate of 5.10 -5 . Regarding the entropic regularization parameter for the Sinkhorn divergence, we set it in {10, 100, 1000}. We update the discriminator 5 times before one update of the generator. The code can be found here * .

We compare our method to 4 different methods: WGAN-GP [Gulrajani 2017], Sinkhorn GAN, OT-GAN [START_REF] Salimans | [END_REF]] and MMD GAN. Regarding Sinkhorn GAN we use a batch size of 256 as done in their work. We also compared our GAN to the effective WGAN-GP, we considered the same architecture as above but we used the hyperparameters described in their paper [Gulrajani 2017]. Finally for OT-GAN [START_REF] Salimans | [END_REF]], we used an entropic regularization parameter set to 500 and for fair comparison with other methods, we set the batch size to 256. In their paper authors used batch size of 8000 images to get a more stable training, however this method is not reproducible in our setting with a single GPU. We report Inception scores in Table 8.2. As we can see, the debiased minibatch Sinkhorn divergence gives the best Inception score showing the relevance of this new loss function. Comparing to the typical Sinkhorn GAN, the debiased strategy increases the inception score by 1 point. Furthermore, it seems that regularizing the problem with the entropic regularization helps to get better performance as already suggested in previous work [START_REF] Genevay | [END_REF]]. We also report in Figure 8.11 some generated examples from MBSD, UMBSD and WGAN models and we can see that UMBSD leads to slightly more detailed samples than MBSD and more realistic than WGAN-GP.

Large scale color transfer between images

The purpose of color transfer is to transform the color of a source image so that it follows the color of a target image. Optimal transport is a well known method to solve this problem and has been studied before in [START_REF] Ferradans | [END_REF][START_REF] Blondel | [END_REF]. Images are represented by point clouds in the RGB color space. Then by calculating the transport plan between the two point clouds, we get a transfer color mapping by using a barycentric projection. As the number of pixels might be huge, previous work selected a subset of pixels using k-means clusters for each point cloud. This strategy allows to make the problem memory tractable but looses some information to the quantification. With MB optimal transport, we can compute a barycentric mapping for all pixels in the image by incrementally updating the full transported vector at * https://github.com/kilianFatras/unbiased_minibatch_sinkhorn_GAN

SD =100

SD , = 100 WGAN-GP each minibtach. When one selects a source indices m-tuple I 1 and a target m-tuple I 2 , she just needs to update the transported vector between the considered minibatches as

Q I1 Y s = I2 Π m I1,I2 Q I2 X t
, with matrix Q I1 and Q I2 defined as in Definition 18. Indeed, the incremental computation can be rewritten as: (8.47) when we use the incomplete MBOT plan Π

Y s = n s Π W 2 2 ,m,k W (u, u)X t ,
W 2 2 ,m,k W
. To the best of our knowledge, it is the first time that a barycentric mapping algorithm has been scaled up to 1M pixel images. About the required memory for experiments, the memory cost to store data is O(n). The minibatch OT calculus requires O(m 2 ) because we need to store the ground cost and the OT plan. The marginal experiment requires O(n), as we just need to average the marginals of the plan. Finally, the memory cost is O(n) while exact OT would be

O(n 2 ).
The source image has (943000, 3) RGB dimension and the target image has RGB dimension (933314, 3). For this experiments, we compare the results between the minibatch framework with the Wasserstein distance for several m and k. We used batch of size 10, 100 and 1000. We selected k so as to obtain a good visual quality and observed that a smaller k was needed when using large minibatches. Also

Numerical experiments on generative modelling, color transfer and minibatch Gromov-Wasserstein 131

Image 1

Image 2 m=10, K=10000000 m=100, K=2000000 m=1000, K=600000 note that performing MB optimal transport can be done in parallel and can be greatly speed-up on multi-CPU architectures. The code can be found here † . One can see in Figure 8.12 the color transfer (in both directions) provided with our method. We can see that the diversity of colors falls when the batch size is too small as the entropic solver would do for a large regularization parameter. However, even for 1M pixels, a batch size of 1000 is enough to keep a good diversity of colors.

From now on for speed constraints, we consider a selected subset of 1000 pixels using k-means clusters for each point cloud. We reproduced empirically the results of Theorem 8.3.4 about the marginal errors, as shown in Figure 8.13. We recover the O(k -1/2 ) convergence rate on the marginal with a constant depending on the batch size m.

As we stated above, minibatch Wasserstein loss increases the number of connections similarly to regularized OT variants. Hence, we want to conduct a sparsity experiment of the minibatch Wasserstein transport plan and we report it for several settings. We considered batch sizes of 50, 100, 200, 350 and 500 and computed the sparsity of the incomplete minibatch OT plan with respect to several number of minibatches k. The results are gathered in Figure 8.13. We see that as m gets smaller, the degree of sparsity decreases and that the sparsity reaches a limit as the number of minibatches increases. Intuitively, it is expected as when m gets smaller, the number of connections increases. The results can be justified with the following facts. When the minibatch size between the source and target batches is the same and with uniform weights, then m coefficients of the transport matrix will be non null for the exact Wasserstein distance. As we draw k batch couples, such as k.m < n and if we suppose that the batches define a disjoint union of the samples X and Y , then we have at most km coefficients of Π

Minibatch Gromov-Wasserstein rotation and translation invariance

The Gromov-Wasserstein distance has the nice properties to be rotational and translation invariant, so in this section we study if the minibatch Gromov-Wasserstein loss (MBGW) shares the same properties. As shown in the previous section, our statistical results can be extended to the Gromov-Wasserstein distance. We start with a spiral experiment where we compute the value of the MBGW loss for several rotations of the spirals. Then, we aim at checking if the MBGW loss is able to recover the motion of a galloping horse on a dataset containing a sequence of shapes [Solomon 2016].

Rotational invariance. Our first result shows the stability of rotation and translation invariances with minibatches. We have the following results: Proposition 14 (Invariance). The minibatch Gromov-Wasserstein GW m w1,w2,P 1 ,P 2 ,C 1 ,C 2 is rotation and translation invariant.

Proof. Let a and b be two probability vectors with support X and Y respectively. Consider now the support Y which is a rotation and a translation of Y . Consider three ground costs C 1 = C(X, X),

C 2 = C(Y , Y ) and C 3 = C(Y , Y ).
For fixed minibatches I and J, as Y is a translation and rotations of Y , we have:

GW w 1 (a, I), w 2 (b, J), C 1 I,I , C 2 J,J = GW w 1 (a, I), w 2 (b, J), C 1 I,I , C 3 J,J ,
summing over all minibatch couples finishes the proof.

Empirically, distances which are rotation invariant return a constant when comparing rotated distributions. To support the proposition, we consider a small spiral experiment for different rotations of the target distribution. We follow the procedure in [Vayer 2019b]. The source and the target distributions are spirals taken from the scikit-learn spiral dataset [Pedregosa 2011b]. We compute Gromov-Wasserstein distance and the MBGW loss on n = 300 samples. We report in Figure 8.14 the average values of the GW and MBGW losses for a varying angle and we can see that it is in practice invariant to rotation. From the Figure 8.14, one recovers that the MBGW loss returns a constant, depending on the minibatch size m, and hence is rotation invariant.

Meshes comparison

In the context of computer graphics, Gromov-Wasserstein distance can be used to measure similarities between two meshes [Peyré 2016, Solomon 2015, Vayer 2019b]. It can also be used for shape matching, search, exploration or organization of databases. As minibatch GW loss and its debiased counter parts are not distances, we want to know if they are meaningful for use in a context of meshes comparison. From a time series of 45 meshes representing the motion of a galloping horse, we compute a multidimensional scaling (MDS) of the pairwise distances with minibatch GW losses, that allows plotting each mesh as a 2D point. Each horse mesh is composed of approximately 9, 000 vertices. The results can be found in Figure 8.15. As one can observe in Figure 8.15, the cyclical nature of this motion is successfully recovered in this 2D plot for both MBGW loss and its debiased counter parts.

Running time comparison Our last experiment is the time computation of minibatch Gromov-Wasserstein. We compare it to Gromov-Wasserstein distance, the entropic regularized Gromov-Wasserstein, the Sliced Gromov-Wasserstein and its rotational invariant variant [Vayer 2019b]. Unfortunately, the Sliced variant can only be computed for square Euclidean ground cost unlike the MBGW and is not rotational invariant. We calculate these distances between two 100-D random measures of n ∈ 10 2 , ..., 10 4 points. For the minibatch Gromov-Wasserstein we consider two settings. The first setting is with a fixed number of minibatch couples (k = 5000) and the second is a linear setting where k grows linearly according to n (k = n 10 ). The latter is due to our concentrations bounds which decreases linearly in the number of samples if we consider a number of minibatch couples proportional to the number of samples (see Theorem 8.3.1). We use the Python Optimal Transport (POT) toolbox to compute GW distance on CPU. For entropic-GW we use the POT implementation with a regularization parameter of ε = 0.01. We were not able to compute transport plan for a bigger number of data than 10 4 for both GW and its entropic variant, with respect to our limited memory and time budget.

We see that MBGW enjoys a constant time computation. The sliced Gromov-Wasserstein and its rotational variant grow in O(n log(n)) making it slower than the minibatch GW for large scale dataset.

Regarding GW and its entropic counter part, we see that for 10e 4 points, the MBGW is 100 time faster than GW.

Discussion: non-optimal connections consequences

In this chapter, we introduced a rigorous formalism of minibatch optimal transport. We first designed an estimator in the simple case of uniform empirical measures. Following this definition, we expressed a closed-form formula when data lie in 1D and illustrate the consequences on the transport plan. We showed that minibatch OT creates a bigger of connections between distributions as illustrated in Figure 8.1. From the formalism for uniform probability vectors, we extended the formalism for general probability vectors 8.5. Discussion: non-optimal connections consequences 135 as well as different sampling. We show the effectiveness of this general formalism to generate misclassified data for a pre-trained classifier as in ARWGAN from Chapter 6. Then we reviewed the basic metric properties and we introduced a new loss function which respects the separability axiom. Unfortunately, this loss is not always positive but always outperformed minibatch OT in practice. We also studied the deviation bounds of our estimators towards their expectation and their optimization properties. We indeed showed that we can optimize them using SGD. Finally, we extensively evaluated minibatch OT losses for generative modelling and color transfer. We also showed that minibatch Gromov-Wasserstein enjoys many properties of the Gromov-Wasserstein distance.

As illustrated in Figures 8.1 and 8.2, the number of connections increases with minibatch OT. Theses non optimal connections can connect two samples of different clusters and this is similar to a regularization effect. Furthermore, the intensity of these connections tend to be uniform as the minibatch size decreases. In the next chapter, we show that these non optimal connections can lead to undesirable effects for specific applications and we propose a simple manner to mitigate them.

Chapter 9 In this chapter, we discuss and prove some limits of the minibatch OT framework due to the marginal constraints and the minibatch sampling which create non-optimal connections. We then give empirical and theoretical evidences that using unbalanced optimal transport at the minibatch level can help mitigating the aforementioned non optimal connections. We show that the deviation bounds and the optimization properties, proved in the last chapter, remain true for unbalanced minibatch OT despite the fact that the UOT transport plans are possibly unbounded. Finally, we extensively evaluate our method on gradient flow, domain adaptation and partial domain adaptation experiments and show that our method improves the results from a state-of-the-art algorithm based on minibatch OT. These contributions have been published to the The Thirty-eighth International Conference on Machine Learning [Fatras 2021b].

Unbalanced Minibatch Optimal Transport

Robustness to outliers and sampling

In this section, we discuss the limits and sensitivities of minibatch OT. We then justify that using a relaxed marginal OT variant can help mitigating these weaknesses. Finally, we theoretically show that unbalanced optimal transport is robust to outliers contrary to exact optimal transport. Limits of minibatch OT. First, we discuss the limitations of combining balanced OT with the minibatch framework. OT is sensitive to the geometry of distributions. When those distributions are tainted by outliers, OT is forced to transport them due to the marginal constraints, inducing an undesirable extra transportation cost. Minibatch OT averages several OT terms related to subsamples of the original distributions, thus sharing this sensitivity. The problem is even worse as two minibatches do not necessarily Chapter 9. Unbalanced Minibatch Optimal Transport share samples that would lie in the support of the full OT plan, hence forced to match samples that could be, at the level of a minibatch, considered as outliers. Take as an example two distributions with clustered samples. While in the full OT plan clusters can be matched exactly, those clusters are likely to appear as imbalanced in the minibatches, especially if the size of the minibatch is small and does not respect the statistics of the original distribution. Due to the marginal constraints, samples from one cluster are likely to be matched to unrelated clusters, as depicted in Figure 8.2. This explains why in practice previous works relied on large minibatches to mitigate this sampling effect [START_REF] Damodaran | [END_REF]]. To overcome this issue, we propose the natural solution of relaxing the marginal constraints at the minibatch level. The expected outcome is twofold: i) mitigating the effect of subsampling in the minibatch strategy and ii) providing a natural and scalable robust optimal transport computation strategy at the global level. In the following paragraph, we present some robust OT formulations.

Relaxed marginals: Unbalanced and Robust Optimal Transport variants. Unbalanced optimal transport is empirically known to be more robust to outliers than OT as it does not need to meet the marginals [Chizat 2017, Liero 2017]. Indeed, if an outlier is too expensive to move, UOT would not transport it. Several other formulations make optimal transport robust for practical and statistical reasons. Partial OT can be adapted for partial matchings problem with applications for positive-unlabeled learning [START_REF] Chapel | Partial Optimal Transport with Applications on Positive-Unlabeled Learning[END_REF]. A line of work proposes 'distributionnally robust' models, where models are trained in a Wasserstein ball around the empirical distribution in the space of probabilities [START_REF] Esfahani | [END_REF], Kuhn 2019]. In a similar approach, several variants relax the OT marginal constraints with a ball constraint, and consider several penalties such as integral probability metrics [Nath 2020], total variation or Csiszàr divergences for outlier detection [START_REF] Mukherjee | [END_REF][START_REF] Balaji | [END_REF]. Such relaxations allow to derive statistical guarantees w.r.t. noise and outliers. Another idea to ensure robustness consists in learning the cost adversarially, and is formulated as a max-min problem where the cost is modeled by an Euclidean embedding [START_REF] Genevay | [END_REF]], a compact space of matrices [Dhouib 2020a] or a projection on a lower dimensional subspace [START_REF] Paty | [END_REF]].

We chose to rely on unbalanced optimal transport with an entropic regularization at the minibatch level to design a new loss function that we called Unbalanced Minibatch Optimal Transport. The entropicregularized unbalanced formulation of optimal transport can be easily computed with a Sinkhorn algorithm as discussed in Section 2.4.2. We discuss in the following some theoretical considerations to support Unbalanced OT's robustness.

Theoretical analysis: impact of an outlier. We start by examining the impact of an outlier in the behaviors of OT and UOT. The following lemma illustrates the relations between those two quantities. Equation (9.2) shows that when z gets further from the supports of (α, β), the OT loss increases. However for UOT the upper bound (9.1) tends to saturate as z gets further away. What remains is the UOT loss between distributions whose outliers are removed, with a cost of removing the outlier proportional to its mass.

We illustrate Lemma 9.1.1 with a toy example in Figure 9.1. We consider a probability distribution α tainted with an outlier (green dot) to get a target probability distribution α = 1 n+1 (n.α + δ z ). We then move away the outlier from α's support, as shown with the green arrow, and we calculate several OT costs. The minibatch size is set to m = 5 and the total cost is the average of k OT costs between those minibatches. We see that OT variants are not robust to the outlier as their loss increases along the outlier displacement unlike UOT variants which reach a plateau as predicted by Lemma 9.1.1. Each computation is done 5 times to show that variance is lower for bigger k and that UMBOT has a lower variance than MBOT for k = 30 and k = 500. We consider now an example in 2D, akin to Figure 8.2, where our goal is to illustrate the OT plan between two empirical distributions of 10 samples in Figure 9.2. We use two 2D empirical distributions where the samples belong to a certain cluster depending on a related class (color information). The source data are equally distributed between classes while the target data have different proportions, 3 samples belong to the red class while 7 samples belong to the green class. Different proportions between domains are ubiquitous for real-world data. We compare unbalanced minibatch OT, minibatch OT, entropic OT and UOT. For UOT, the divergence D φ equals to KL divergence and for the minibatch variant, the minibatch size is m = 2. We can see from the OT plans in the first row of the figure that the cluster structure is more or less recovered. However OT and minibatch OT tend to connect samples from different classes. This configuration would lead, for instance, to negative transfer in a context of domain adaptation applications, i.e., matching of samples between different domains. This is less true for UOT, where the pairings between different classes is diminished and tend to disappear when we reduce the penalty τ . In the next section, we show that under reasonable hypotheses, the main theoretical results from Chapter 8 remain true.

Statistical and optimization properties 141

of batches k. With probability at least 1 -δ on the draw of X, Y and D k we have:

| h m,k (X, Y ) -E m h | ≤ M   log( 2 δ ) 2 n m + 2 log( 2 δ ) k   ,
where M is the UOT upper bound. Furthermore, for h = OT τ,ε φ , let M ∞ Π be the maximum mass of minibatch transport plans. For all k 1, all 1 i n, with probability at least 1 -δ on the draw of X, Y and D k we have:

| Π h,m,k (X, Y ) (i) 1 n -Π h,m (X, Y ) (i) 1 n | M ∞ Π 2 log(2/δ) k ,
where we denote by Π (i) the i-th row of matrix Π and by 1 n ∈ R n the vector whose entries are all equal to 1.

The rate m n is the same as in Chapter 8.3.1. The main difference is the upper bound M which bounds UOT. Note that the bound does not depend on the dimension of X unlike original unbalanced OT [Séjourné 2019]. Regarding OT plans, the constant M Π represents the maximum mass of minibatch transport plans which would be 1 for OT. In the next section, we study the convergence of SGD using Clarke gradients when the contrast function is unbalanced minibatch OT.

Unbiased Clarke gradients

In Section 8.3.2, we proved that minibatch OT does not suffer from biased gradients contrary to optimal transport. In this section we show that this property can be generalized for minibatch UOT, including unregularized UOT. Like in Section 8.3.2, we also achieve this point by relying on Clarke regularity. It is not immediate because UOT is not differentiable as we do not have a unique optimal transport plan when ε = 0. We have the following theorem.

Theorem 9.2.2. Let X, { Ŷθ } θ∈Θ be two m-tuples of random vectors compactly supported, h ∈ {OT τ,ε , S τ,ε } and C m a C 1 cost. Under an additional integrability assumption, we have:

∂ θ E[h(u, u, C m ( X, Ŷθ ))] = E[∂ θ h(u, u, C m ( X, Ŷθ ))],
with both expectation being finite. Furthermore the function θ → -E[h(u, u, C m ( X, Ŷθ ))] is also Clarke regular.

To prove this theorem, we used a similar proof which was used for proving the minibatch OT case. We also needed to prove that UOT is Lipschitz in the ground cost C, which is possible because the transport plan is bounded by Lemma 9.2.1. Theorem 9.2.2 implies that if we use the Minibatch UOT loss with h ∈ {OT τ,ε φ , S τ,ε φ } as a loss function, then the minus objective function is Clarke regular. Furthermore, Stochastic gradient with decreasing step sizes converges almost surely to the set of critical points of Clarke generalized derivative [START_REF] Davis | [END_REF][START_REF] Majewski | [END_REF]]. As a consequence, it is justified to use SGD with minibatch UOT.

Lazy gradients. We finish this section with a phenomenon that we call lazy gradients. We show that for small values of τ , the norm of the partial gradient with respect to the parameters is small as well, which leads to slow training phase in practice. We first express the partial derivative of unbalanced minibatch 142 Chapter 9. Unbalanced Minibatch Optimal Transport OT. For h = OT τ,ε φ and for all 1 ≤ i ≤ q we have:

∂ θi h(u, u, C(X, Y θ )) = co{-Π • D • (∇ θi Y ) :Π ∈ Opt h (X, Y ), (9.3) D ∈ R m,m , D j,k = ∇ Y C j,k (X, Y θ )}.
Where ∂ θi is the Clarke subdifferential with respect to θ i , ∇ Y C j,k is the differential of the cell C j,k of the cost matrix with respect to Y , Opt h (X, Y θ ) is the set of optimal transport plan and co denotes the closed convex hull. Note that when ε > 0 the set Opt h (X, Y θ ) is reduced to a singleton, the notation co is then superfluous and it reduces to

∂ θi h(u, u, C(X, Y θ )) = -Π • D • (∇ θi Y ), (9.4)
where Π is the optimal plan. As we are in finite dimension, all norms are equivalent and we can consider

• to be a submultiplicative norm. We can then bound the partial gradient as:

∂ θi h(u, u, C(X, Y θ )) = -Π • D • (∇ θi Y ) ≤ Π D ∇ θi Y . (9.5)
So for small values of τ , the mass of the optimal transport plan Π is small. Hence the gradient norm is small as well which leads to slow down the training phase. Thus a trade-off is needed on the value of τ in order to have transport plan with big enough mass and which do not transport outliers. Further analysis can be made by using the closed-form solution of entropic-regularized UOT between Gaussian distributions [Janati 2020] and we left such analysis as future work.

Numerical experiments on gradient flows and domain adaptation

In this section, we illustrate the practical behavior of unbalanced minibatch OT for gradient flow and for domain adaptation experiments. We relied on the POT package [START_REF] Flamary | [END_REF]] to compute the exact OT solver or the entropic UOT loss and the Geomloss package [START_REF] Feydy | [END_REF]] for the Unbalanced Sinkhorn divergence. The experiments were designed in PyTorch [START_REF] Paszke | [END_REF]] and all the code can be found here * .

Unbalanced MiniBatch OT gradient flow: a qualitative example

We consider the same experiments as in Section 8.4.1 but with toy 2D data. Consider a given target distribution α, we recall that the purpose of gradient flows is to model a distribution β(t) which at each iteration follows the gradient direction minimizing the loss β t → h(α, β t ) [Peyré 2015[START_REF] Liutkus | [END_REF].

For α and β(0) we generate 10000 2D points divided in 2 imbalanced clusters with number of samples in each cluster provided in Figure 9.3. We consider the (unbalanced) sinkhorn divergence, a squared Euclidean cost, a learning rate of 0.02, 5000 iterations, m equals 64 or 128 and k = 1. We show the gradient flow of the upper clusters to the lower clusters in Figure 9.3. From the experiment, we can see that the minibatch OT is not robust to imbalanced classes on the contrary to the minibatch UOT. Indeed there are data from the upper left cluster which converge to the down right cluster and we can also see an overlap between the classes. Due to OT marginal constraints, the loss forces to transport all data in the batch which results in breaking the target shapes. This is not the case for minibatch UOT, which better respects the shape of target distributions.

JUMBOT: a new approach for domain adaptation

In this section, we evaluate Unbalanced minibatch OT in domain adaptation. Based on the state-of-theart algorithm deepjdot, we replace OT by UOT at the minibatch level. We consider vanilla domain adaptation and partial domain adaptation experiments.

Domain adaptation.

JUMBOT. To solve the domain adaptation problem, we use an alignment strategy with a cross entropy on the source domain and a transfer term. The transfer term of our method is based on [START_REF] Damodaran | [END_REF]] and aims at finding a joint distribution map between the source and the target distributions by taking into account a term on a neural network embedding space and on the label space. Formally, let g θ be an embedding function where the input is mapped into the latent space and f λ which maps the latent space to the label space on the target domain. The embedding space is in our experiment the penultimate layer of a neural network. For a given minibatch, embedding g θ and classification map f λ , the transfer term is:

hm C θ,λ ((X s , Y s ), (X t , f λ (g θ (X t )))), with (9.6) (C θ,λ ) i,j = η 1 g θ (x s i ) -g θ (x t j ) 2 2 + η 2 L(y s i , f λ (g θ (x t j ))),
where L(., .) is the cross-entropy loss and η 1 , η 2 are positive constants. Basically, this specific transportation cost combines a distance between the representation of the data through the neural network g θ and a loss function between the associated labels [Courty 2017b]. Taking k = 1 led to state-of-the-art results.

The Csiszàr divergence φ is the Kullback-Leibler divergence KL. We also add a cross entropy term on the source data. Hence our optimization problem is: and SVHN to MNIST (S →M). MNIST [LeCun 2010] contains 60,000 images of handwritten digits, M-MNIST contains the 60,000 MNIST images with color patches [START_REF] Ganin | [END_REF]] and USPS [Hull 1994] contains 7,291 digit images. Street View House Numbers (SVHN) [START_REF] Netzer | [END_REF]] consists of 73, 257 images with digits and numbers in natural scenes. We report the evaluation results on the test target datasets.

min θ,λ i L(f λ (g θ (x s i )), y s i ) + η 3 h m,k C θ,λ ((X s , Y s ), (X t , f λ (g θ (X t )))). ( 9
Office-Home [START_REF] Venkateswara | [END_REF]] is a difficult dataset for unsupervised domain adaptation (UDA), it has 15,500 images from four different domains: Artistic images (A), Clip Art (C), Product images (P) and Real-World (R). For each domain, the dataset contains images of 65 object categories that are common in office and home scenarios. We evaluate all methods in 12 adaptation scenarios. VisDA-2017 [START_REF] Peng | [END_REF]] is a large-scale dataset for UDA from simulation to real. VisDA contains 152,397 synthetic images as the source domain and 55,388 real-world images as the target domain. 12 object categories are shared by these two domains. Following [START_REF] Long | [END_REF][START_REF][END_REF], we evaluate all methods on VisDA validation set.

Setup. First note that for all datasets, jumbot uses a stratified sampling on source minibatches as done in deepjdot [START_REF] Damodaran | [END_REF]]. Stratified sampling means that each class has the same number of samples in the minibatches. This is a realistic setting as labels are available in the source dataset.

For Digits datasets, we used the 9 CNN layers architecture and the 1 dense layer classification proposed in [START_REF] Damodaran | [END_REF]]. We trained our neural network on the source domain during 10 epochs before applying jumbot. We used Adam optimier with a learning rate of 2e -4 with a minibatch size of 500. Regarding competitors, we use the official implementations with the considered architecture and training procedure.

For Office-Home and VisDA, we employed ResNet-50 as generator. ResNet-50 is pretrained on ImageNet and our discriminator consists of two fully connected layers with dropout, which is the same as previous works [START_REF] Ganin | [END_REF][START_REF] Long | [END_REF][START_REF][END_REF]]. As we train the classifier from scratch, we set its learning rates to be 10 times that of the generator. We train the model with Stochastic Gradient Descent optimizer with a momentum of 0.9. We schedule the learning rate with the strategy in [START_REF] Ganin | [END_REF]], it is adjusted by χ p = χ0 (1+µq) ν , where q is the training progress linearly changing from 0 to 1, χ 0 = 0.01, µ = 10, ν = 0.75.

We compare jumbot against recent domain adaptation papers, namely dann [START_REF] Ganin | [END_REF]], cdane [START_REF] Long | [END_REF]], alda [START_REF][END_REF]], deepjdot [START_REF] Damodaran | [END_REF]] and rot [START_REF] Balaji | [END_REF]] on all considered datasets. We reproduced their scores and contrary to these papers we do not report the best classification on the test along the iterations but at the end of training, which explains why there might be a difference between reported results and reproduced results. We sincerely believe that the evaluation shall only be done at the end of training as labels are not available in the target domain. But we also report the maximum accuracy along epochs for the Office-Home DA task in is above all of the competitors by a safe margin of 3%.

For Office-Home, we made 10000 iterations with a batch size of 65 and for VisDA, we made 10000 iterations with a batch size of 72. For fair comparison we used our minibatch size and number of iterations to evaluate competitors. The hyperparameters used in our experiments are as follows η 1 = 0.1, η 2 = 0.1, η 3 = 1, τ = 1, ε = 0.1 for the digits and for Office-Home datasets η 1 = 0.01, η 2 = 0.5, η 3 = 1, τ = 0.5, ε = 0.01. For VisDA, η 1 = 0.005, η 2 = 1, η 3 = 1, ε = 0.01 and τ was set to 0.3.

Results.

The results on digit datasets can be found in Table 9.1 where (*) denotes reproduced results. We conducted each experiment three times and report their average results and variance. For fair comparisons, we only resize and normalize the image without data augmentation. We see that our method performs best with a margin of at least 1.5 points. Furthermore, we see an important 4% increase of the performance compared to deepjdot. A deeper analysis of this difference is considered in the next paragraph. Office-Home results are gathered in Table 9.2 and VisDA are reported in Table 9.4. For fair comparison with previous work, we used a similar data pre-processing and we used the ten-crop technique [START_REF] Long | [END_REF][START_REF][END_REF] for testing our methods. Experiments in [START_REF] Balaji | [END_REF]] follow a different setup explaining the difference of performance between their reported score and our reproduced score.

jumbot achieves the best accuracy on average and on 11 of the 12 scenarios on the Office-Home dataset and achieves the best accuracy on VisDA at the end of training with a margin of 4% and 2% respectively.

Ablation. The main difference between jumbot and deepjdot is the use of a different OT solver.

jumbot uses entropic regularized unbalanced OT and deepjdot uses original OT. We have conducted an ablation study in all our domain adaptation experiments to measure the effect of the entropic regularization and the unbalanced formulation. The use of entropic regularized OT in place of original OT in deepjdot is denoted e-deepjdot. We can see that for the digits experiments, the entropic regularization helps to get better performances of 0.6% on the DA task USPS → MNIST and more than 2% on SVHN → MNIST. For the last digit task MNIST → M-MNIST, the grid search on ε needed to be reduced due to some numerical instabilities. Overall, the performances of e-deepjdot are still lower than the entropic regularized Unbalanced OT of jumbot by more than 1%. A similar performance gain was observed on the VisDA dataset, where e-deepjdot was 1.2% higher than deepjdot but still more than 3% below jumbot. On the Office-Home experiments, the entropic regularization alone did not get better results. Analysis. In this paragraph, we study the difference of behavior between deepjdot and jumbot. Along jumbot's training on the DA task USPS to MNIST, we measured the percentage of mass between data with different labels at each iteration. In average along training about 7% of deepjdot connections are between data with different labels while this percentage decreases to 0.7% for jumbot as shown in Figure 9.6. So deepjdot transfers wrong labels to target data which will decrease the overall accuracy.

We also plot a TSNE embedding of our method and deepjdot (see Figure 9.4), we can see that there are some overlaps between clusters for deepjdot unlike our method. This is probably due to the minibatch smoothing effect which would tend to bring clusters of different classes closer.

We also provide a sensitivity analysis to the batch size, ε and τ parameters. All results are gathered in Figure 9.5. When τ is too small, jumbot creates negative transfer because of the entropic regularization. When τ increases, we see that jumbot accuracy increases and it reaches its maximum around τ = 1. However when τ is too high, we recover entropic OT.

When ε varies for jumbot and the entropic variant of deepjdot, we see that entropy helps getting slightly better results. However when the entropic regularization coefficient is too big, the accuracy falls. We conjecture that entropic regularized OT regularizes the neural network because the target prediction is matched to a smoothed source label (see a similar discussion in [START_REF] Damodaran | [END_REF]). And it is well known that label smoothing creates class clusters in the penultimate layer of the neural network [START_REF] Müller | [END_REF]].

We now discuss the minibatch size. While jumbot has a constant accuracy along all batch size, the deepjdot accuracy falls of 4% for SVHN → MNIST and 6% for USPS → MNIST. The benefits of our method over deepjdot are twofold, it is more robust to small batch sizes and it is performant for small computation budget unlike deepjdot.

Finally, we provide a classification accuracy along training which demonstrates the network overfitting with deepjdot and not with jumbot, the results are gathered in Figure 9.7. One can see that deepjdot starts overfitting from epoch 30 on each class. There are some classes which are more affected by overfitting than others. The accuracy on each class is reduced of several points. This behaviour is not shared with our method jumbot. Indeed it is more stable, it does not show any sign of overfitting and it has a higher accuracy. This shows the relevance of using our method jumbot.

Partial domain adaptation.

Partial DA. Finally we consider the Partial DA (PDA) application. In PDA, the target labels are a subset of the source labels, i.e., Y t ⊂ Y s . Samples belonging to these missing classes become outliers which can produce negative transfer. We want to investigate the robustness of our method in such an extreme scenario. We evaluate our method on partial Office-Home, where we follow [Cao 2018] to select the first 25 categories (in alphabetic order) in each domain as a partial target domain.

Set up. We compare our method against state-of-the-art PDA methods: pada [Cao 2018], etn [START_REF] Cao | [END_REF]] and ba3us [START_REF] Jian | [END_REF]]. For fair comparison we followed the experimental setting of pada, etn and ba3us. We considered a neural network architecture and a training procedure similar as in the domain adapation experiments which also corresponds to the setting in [START_REF] Jian | [END_REF]]. Our hyperparameters are set as follows : τ = 0.06, η 1 = 0.003, η 2 = 0.75 and finally η 3 was set to 10. Regarding training procedure, we made 5000 iterations with a batch size of 65 and for optimization procedure, we used the same as in [START_REF] Jian | [END_REF]]. We do not use the ten crop technique to evaluate our model on the test set as we were not able to reproduce the results from ent and pada. Furthermore, we do not know if the reported results ent and pada were evulated at the end of optimization or during training, but our reported scores are above their scores by at least 5% on average.

Results.

The final performances are gathered in the lower part of Table 9.2. We can see that jumbot is state-of-the-art on 9 out of the 12 domain adaptation tasks and is on average 2% above competitors. Finally, we also evaluate deepjdot [START_REF] Damodaran | [END_REF]] and its entropic variant e-deepjdot on the PDA task to compare the performances of regularized unbalanced OT against regularized and unregularized OT. e-deepjdot gives similar results to deepjdot and they both produce negative transfer. jumbot is on average 15% higher on this problem, showing the clear advantages of our strategy in the presence of unbalanced classes.

Conclusion 149 9.4 Conclusion

In this chapter, we presented some weaknesses of minibatch optimal transport. Due to the marginal constraints, minibatch OT, like OT, is sensitive to outliers. Furthermore, two minibatches do not necessarily share samples that would lie in the support of the full OT plan. Hence it would force OT to match samples that could be, at the level of a minibatch, considered as outliers. To mitigate these issues, we proposed to use unbalanced OT at the minibatch level. We proved that contrary to OT, UOT is robust to these outliers. Then we also proved that the deviation bounds and optimization properties showed in the previous chapter remain true for unbalanced minibatch OT. Finally we demonstrated the relevance of our method on gradient flows, domain adaptation and partial domain adaptation experiments. We show that by using unbalanced minibatch OT in deepjdot, nonetheless it outperforms the original deepjdot based on minibatch OT but also outperforms all recent state-of-the-art domain adaptation algorithms.

Chapter 10

Conclusion

Overview of the contributions

In this thesis, we developed new optimal transport based methods for deep learning and we theoretically and empirically studied a commonly used optimal transport approximation in deep learning.

Optimal Transport for label noise. We first focused on the label noise problem, which is a supervised learning setting where a proportion of labels are corrupted. Based on the intuition that label noise can be mitigated if the predicted classification is uniform among clusters of data, we used an optimal transport loss in the Virtual Adversarial Algorithm. The VAT algorithm promotes a local uniform prediction by penalizing large local changes in the prediction thanks to a divergence. We replaced the divergence with optimal transport and crafted different ground costs in order to change the geometry of the regularization. These ground costs allowed us to have complex boundaries between similar classes and smooth boundaries between non similar classes. One ground cost uses semantic distances based on word embeddings such as word2vec, as similarly done in [Frogner 2015] and another uses the distance between embedded data. We then showed the relevance of using our method on standard and real-world datasets including remote sensing images. We also showed the effectivness of our methods on open-set noise, where images that do not correspond to given classes are in the dataset.

Generating natural misclassified data. Our second contribution uses generative adversarial networks to generate misclassified data. Based on a pre-trained classifier, where we have access to the output, and to training data, we developed a new weighting strategy to define a new probability measure for training data. The weighting strategy is based on a softmax function and gives bigger weights to misclassified data than correctly classified data. We then give this probability measure to Wasserstein GAN [START_REF] Arjovsky | [END_REF], Gulrajani 2017], an optimal transport variant of GAN, in order to generate misclassified data. We evaluated empirically our method on different remote sensing data, where we showed that the generated data were similar to training data and misclassified by the pre-trained classifier.

Chapter 10. Conclusion

We also demonstrated that a large proportion of misclassified data for the pre-trained classifier are also misclassified for other classifiers, showing the transfer property of our method. Finally, we also showed our method can modify training data to make them misclassified and that we can fool state-of-the-art detector.

Theoretical and empirical study of minibatch Optimal Transport. After using optimal transport as a loss function in deep learning, we studied a standard optimal transport approximation. To fasten training of deep learning, loss functions are commonly computed between minibatches of training data. While it is a competitive method in practice [START_REF] Genevay | [END_REF][START_REF] Damodaran | [END_REF], a theoretical explanation and study were lacking. We thus developed a rigorous formalism that works for general probability measures. After defining new empirical estimators, we found a closed-form solution for 1D data. We also reviewed the metric properties, and in order to fix the separability axiom loss, we introduced a new loss function based on minibatch OT. Then we reviewed deviation bounds between our estimators and their expectations, which is the quantity we aim to minimize. We also proved the convergence of SGD in a data fitting scenario when the contrast loss is MBOT. We showed that minibatch OT can solve large-scale color transfer and kept a high diversity of color. Finally, we empirically showed the practical success of our new loss function over minibatch OT on generating data.

Mitigating non-optimal connections. Our last contribution studied the practical consequences of the non-optimal connections in a classification scenario. The standard strategy to solve the domain adaptation problem is to align the embeddings of source and target data which share the same class. Unfortunately, minibatch OT creates non-optimal connections between the domains due to the minibatch sampling and the OT marginal constraints. Thus, the non-optimal connections align domains of data which have different classes. In order to alleviate these connections, we proposed to rely on unbalanced optimal transport. We showed that this variant is indeed robust to outliers unlike original optimal transport. We also showed that our theoretical contributions to minibatch OT remained true when unbalanced optimal transport was used at the minibatch level. Finally, we empirically demonstrated the success of our method on domain adaptation experiments including in extreme imbalanced scenario. We also showed that the performance of unbalanced minibatch optimal transport remains constant when the minibatch size decreases contrary to exact optimal transport.

Perspectives of future works 10.2.1 Perspectives on our contributions

The different contributions developed in this manuscript lead to several open questions.

Perspectives on the Wasserstein Adversarial Regularization. An important question to improve WAR is to study new ground costs. The two ground costs introduced for our regularizations have been empirically crafted and thus might be crude approximations. New ground costs could be for instance derived from auxiliary tasks that do not involve labels, or by inspecting the confusion matrix score, obtained on the noisy validation set, along the learning process. A dynamic update of the class matrix is also a relevant research avenue to be studied in the case the noise statistics of the data change over time. Recently, several works were published were authors tried to learn the ground cost for a given machine learning problem [START_REF] Cuturi | [END_REF], Heitz 2020]. We believe that this strategy can also be relevant 10.2. Perspectives of future works 153 for WAR. Finally, we recall that WAR is based on the VAT algorithm, which was designed to solve the semi-supervised learning setting. While preliminary results showed that WAR do not outperform VAT on this task, it might be due to non-insightful ground cost. We also foresee an application for imbalanced classes where the ground cost could allow to have complex boundaries for imbalanced classes and smooth boundaries for others. Finally, WAR can also be relevant on other tasks were perturbation based regularization were successful.

Perspectives on minibatch OT. The recent work on minibatch OT suggests new open questions. On the theoretical side, new concentration bounds could be derived for minibatch OT using the Talagrand concentration inequality [Talagrand 1995]. A recent work [START_REF] Nguyen | [END_REF]] introduced a new minibatch OT cost based on a different sampling. Instead of averaging the optimal transport cost between minibatches, they see the minibatches as their new measures to transport. They create a ground cost which measures the pairwise distance of minibatches. They proved that their loss is a metric between probability measures and that their transport plan is empirically more sparse. However, it is not proven that they indeed transport all the input masses. While appealing, this strategy remains heavy in practice as you need to compute a lot of minibatch distances as a pre-processing step in order to get your ground cost. The empirical gain in deep learning is also not clear and needs to be demonstrated.

Perspectives on Unbalanced Optimal Transport. The use of unbalanced optimal transport at the minibatch level to improve classification score in domain adaptation, highlighted some limits of UOT. In practice, when the marginal coefficient τ is too small, the mass of the optimal transport plan is small as well. This leads to gradients with possibly small norms which slow down the learning process. Studying empirically and theoretically this effect is an important element for the use of UOT in deep learning. This subject could be investigated with closed-form solutions of unbalanced OT, as done for the Gaussian case in [Janati 2020]. In order to get a transport plan with a bigger mass, we could also learn adversarially the probability measures of our minibatches, similarly to [START_REF] Balaji | [END_REF]]. Intuitively, we might want to give small weights to transport data with a high cost and bigger weights to data which are cheap to transport.

Perspectives on Optimal Transport in Machine Learning

Several new Machine Learning applications were recently published and we make a small overview of domains where optimal transport has been or can be applied.

Normalizing Flow. Among them, we note that optimal transport has been applied to normalizing flow. A normalizing flow is an invertible map between an arbitrary probability distribution and a standard normal distribution. The flow can be used for density estimation, statistical inference or generative modelling. Several variants of normalizing flow rely on the dynamical formulation of optimal transport [START_REF] Onken | [END_REF], Finlay 2020], on the sliced Wasserstein distance formulation [Dai 2021] or the Brenier theorem [START_REF] Huang | [END_REF], where the optimal transport map is the gradient of a convex function for the squared euclidean distance [START_REF] Peyré | [END_REF]]. Among all these variants, a minibatch computation of the primal might bring interesting elements such a more stable training as for GANs [START_REF] Genevay | [END_REF]].

Out-of-distribution sample. Deep neural networks have a tendency to make spurious correlations that do not hold outside a specific training distribution. For instance, [Beery 2018] trained a neural network to classify camels from cows. Most of the training pictures of cows had green pastures, while Chapter 10. Conclusion most pictures of camels had a desert landscape. Unfortunately, the neural network associated a green pasture to the cows and incorrectly classified cows on a beach. The main method to solve this problem is the Invariant Risk Minimization [START_REF] Arjovsky | [END_REF]. It aims at building a classifier that performs well across many unseen environments. As optimal transport has been successful in Domain Adaptation to create cluster of data between domains [START_REF] Damodaran | [END_REF], Fatras 2021b]. It would be interesting to see if a similar approach could lead to create cluster of training data and improvements on classifying OOD samples.

• Appendix A.1.5 details the optimization proofs. We prove that we can exchange Clarke gradients over parameters and expectations, which justifies the convergence of SGD when we use MBOT as a contrast function. Notably, this proof includes the Wasserstein distance and the Gromov-Wasserstein distance.

• Appendix A.1.6 discusses the 1D case. We detail the calculus of the 1D minibatch Wasserstein distance closed-form.

A.1.1 Formalism

In this appendix, we show results which justify our formalism and then, we show how we can upper bound our minibatch OT loss. We first show that Example 4 defines a probability law on m-tuples without replacement.

Example 4 (Drawing indices "without replacement"). Given a discrete probability distribution a ∈ Σ n , it is also possible to draw distinct indices i ∈ n , 1 ≤ ≤ m, by defining P W a (I) = 0 if the m-tuple I has repeated indices, otherwise

P W a (I) = 1 m (n -m)! (n -1)! i∈I a i . (A.1)
Denote by P m the set of all m-tuples without repeated elements. Let us check, that equation (A.1) defines a probability distribution on P m . Observe that n i=1 a i = 1 and that for each .3) This shows that I∈P m P W a (I) = 1.

1 ≤ i ≤ n {I ∈ P m : i ∈ I} = {I ∈ P m : n ∈ I} = {I = (i 1 , . . . , i m ) ∈ P m : i 1 = n} + . . . + {I = (i 1 , . . . , i m ) ∈ P m : i m = n} = m • {I = (i 1 , . . . , i m ) ∈ P m : i m = n}. (A.2) Since {I = (i 1 , . . . , i m ) ∈ P m : i m = n} is the number of (m -1)-tuples without repeated indices of 1, n -1 , (n -1)!/(n -m)!, it follows that m(n -1)! (n -m)! • I∈P m P W a (I) = I∈P m i∈I a i = n i=1 a i • {I ∈ P m : i ∈ I} = m • (n -1)! (n -m)! . ( A
We now prove that if Equation (8.21) is respected, then the minibatch OT matrix defines a transport plan. We also prove that for a W p p kernel, minibatch W p p , i.e., W p p , is an upper bound of W p p .

Proposition 12. If the reweighting function w and the parametric distribution on m-tuples P c satisfy the following admissibility condition

E I∼Pc Q I w(c, I) = c, ∀c ∈ Σ n (A.4)
Then with the notations of Definition 19, the averaged minibatch transport matrix Π h w,P is an admissible transport plan between the discrete probabilities a, b ∈ Σ n in the sense that Π

n = E I,J Q I Π m I,J Q J 1 n = E I,J Q I Π m I,J 1 m = E I,J Q I a I = E I Q I a I = E I∼Pa Q I w(a, I) = a.
The proof that 1 n Π h w,P = b is similar. This establishes that Π h w,P ∈ U (a, b) is an admissible transport plan between the discrete probabilities a and b.

We now prove (A.5) for the Wasserstein distance h = W p p (ε = 0, 1 ≤ p < ∞). Since Π h w,P is an admissible transport plan we have:

h(a, b) = min Π∈U (a,b) Π, C ≤ Π h w,P , C .
Further, by definition of the average minibatch transport plan Π h w,P , and observing that the matrices

Q I , Q J from Definition 23 are such that C I,J = Q I CQ J , we obtain Π h w,P , C = E I,J Π I,J , C = E I,J Π I,J , C = E I,J Q I Π m I,J Q J , C = E I,J Π m I,J , C I,J .
Now observe that by definition of the minibatch transport plans Π m I,J (cf Definition 23) we have,

Π m I,J , C I,J = h w 1 (a, I), w 2 (b, J), C (I,J) .
For the Wasserstein distance h = W p p , combining all of the above we obtain h(a, b) ≤ Π h w,P , C = E I,J Π m I,J , C I,J = E I,J h w 1 (a, I), w 2 (b, J), C (I,J) .

We prove some associations of reweighting functions and parametric laws on m-tuple which respects the marginal constraints (A.4). Lemma 8.1.1 (Admissibility). The uniform reweighting function w U and the parametric law "with replacement" P U satisfy the admissibility condition. The admissibility condition also holds for the parametric law without replacement P W with the normalized reweighting function w W . In contrast for w U , P W when a is not uniform, the resulting OT matrix is not a transportation plan.

Proof. Consider first w W and draws with the probability law P W . This law only allows to draw m-tuples without repeated entries. Since the probability of drawing a tuple without repeated indices such that j∈I a j = 0 is zero, without loss of generality we consider a draw I such that j∈I a j > 0 as a

i > 0 with 1 ≤ i ≤ n. Given 1 ≤ i ≤ n, we distinguish several cases: if i / ∈ I then Q I w W (a, I) = 0; otherwise there exists 1 ≤ k ≤ m such that i = i k , hence (Q I w W (a, I)) i = w W k (a, I) = a i k m p=1 a ip = a i j∈I a j . As a result E I∼P W a (Q I w W (a, I)) i = E I∼P W a a i j∈I a j 1 I (i) = a i E I∼P W a 1 I (i) j∈I a j . Annex A. Appendix
If a i = 0 the right hand side equals a i . Assuming now a i > 0, we have j∈I a j > 0 for each I that contains i, and we prove that E I∼P W a 1 I (i) j∈I aj = 1. Indeed, by definition of P W we have

E I∼P W a 1 I (i) j∈I a j = I∈P m P W a (I) 1 I (i) j∈I a j = I∈P m ,I i P W a (I) 1 j∈I a j = I∈P m ,I i (n -m)! m(n -1)! = (n -m)! m(n -1)! • {I ∈ P m , i ∈ I} = 1.
Where the last equality is from (A.2). We can conclude that E I∼P W a Q I w W (a, I) = a for every a. To show that admissibility does not hold with w U and P W , we similarly obtain

E I∼P W a (Q I w U (a, I)) i = E I∼P W a 1 m 1 I (i) = 1 m E I∼P W a 1 I (i).
When a is not uniform, by the pigeonhole principle there is an index i such that a i > 1/m. Since the right hand side above cannot exceed 1/m, we conclude that E I∼P W a (Q I w U (a, I)) = a. Consider now the pair (w U , P U ). For an m-tuple I = (i 1 , . . . , i m ) we denote m j = m j (I) the multiplicity of index 1 ≤ j ≤ n and observe that m

1 + • • • + m n = m, and Π j∈I a j = Π n k=1 a m k k . Vice-versa, given integers (m 1 , . . . , m n ) such that m 1 + • • • + m n = m there are m!/(m 1 ! • • • m n !) m-tuples I with the corresponding multiplicity. Given 1 ≤ i ≤ n, reasoning as above we obtain E I∼P U a (Q I w U (a, I)) i = E I∼P U a m i m 1 I (i) = I∈ n m P U a (I) m i m 1 I (i) = I∈ n m (Π j∈I a j ) m i m 1 I (i) = m1+•••+mn=m m! m 1 ! • • • m n ! (Π n k=1 a m k k ) m i m 1(m i ≥ 1) = m1+•••+mn=m mi≥1 m! m 1 ! • • • m i ! • • • m n ! m i m Π n k=1 a m k k = a i m 1 +•••+m n =m-1 (m -1)! m 1 ! • • • m i ! • • • m n ! Π n k=1 a m k k = a i n i=1 a i m-1 = a i .
In the last line, we used Newton's multinomial theorem and the fact that n i=1 a i = 1.

Upper bound. We now give a upper bound of minibatch optimal transport. We have access to empirical data and the distance between each data can be bounded by the maximum distance between data, i.e., for two random data x and y, we have :

x -y p 2 2 max 1 i,j n x i -y j p 2 . Lemma A.1.1 (Upper bounds on OT kernels). Let X = (x 1 , • • • , x n ), Y = (y 1 , • • • , y n ) be two n-tuple
of vectors in R d and C the ground cost matrix. Let a and b be two probability vectors, w 1 and w 2 be two reweighting functions and let (I, J) be two m tuples. Then, we have the following bounds for kernel OT

h ∈ {L, W p , W p p , L ε , S ε }: h w 1 (a, I), w 2 (b, J), C (I,J) (X, Y ) 2 max 1 i,j n x i -y j p 2 . (A.6) And for h = GW , let C 1 = C(X, X) and C 2 = C(Y , Y ). Then, h w 1 (a, I), w 2 (b, J), C 1 (I,I) , C 2 (J,J) max 1 i,j,k,l n C 1 i,k -C 2 j,l p 2 .
(A.7)

A.1. Proofs of Chapter 8 161

where M = τ (diam(supp(α)) ∪ diam(supp(β))). And for the sampling with replacement, let the sequence probability vectors (a (n) ), (b (n) ) verify Loc G (m, γ, D) for some γ ∈ (1 -1 4m , 1] and D > 0. We have the following concentration bound .14) where M = τ (diam(supp(α)) ∪ diam(supp(β))).

P h m U (a (n) , b (n) ) -Eh m U (a (n) , b (n) ) ≥ 2M D 2m m 1 2 n 2m(γ-1+ 1 4m ) 2 log(2/δ) ≤ δ, ( A
Proof. The proof is inspired by the two-sample U-statistic proof from [Hoeffding 1963, section 5]. We start with the sampling without replacement case.

Sampling without replacement :

We first consider the case of Example 4, i.e, when the law P is given by (8.18). The proof is based on two-sample U-statistic Hoeffding inequalities and we give it for h ∈ {W p , W ε , S ε } as the GW p follows the same principle. The goal is to rewrite h m W as a sum of terms where each term is a sum of independent random variables. Let (a (n) ) and (b (n) ) be as in the above. To ease the notations, the dependence in n will be implicit.

We fix r = n/m . Let 0 ≤ k ≤ r -1, we define the set

I k := {km + 1, • • • , km + m}.
Then we define the function V as :

V (x 1 , • • • , x n , y 1 , • • • , y n ) = 1 r r-1 k=0 P W a (I k )P W b (I k )h w W (a, I k ), w W (b, I k ), C (I k ,I k ) (A.15)
We recall the implicit dependence in X and Y in the right-hand-side of (A.15) through the ground cost C. In the summation below, σ x or σ y denotes a generic permutation of {1, • • • , n}. We compute :

1 n! 2 σx,σy V (x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ) (A.16) = (n -m)! 2 n! 2 I∈P m J∈P m P W a (I)P W a (J)h w W (a, I), w W (b, J), C (I,J) (A.17) = (n -m)! 2 n! 2 h m W (a, b) (A.18)
Finally, we have :

h m W (a, b) = 1 n! 2 σx,σy V (x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ), (A.19)
where the function V is defined by

V = n! (n -m)! 2 V Let us define T (x 1 , • • • , x n , y 1 , • • • , y n ) = V (x 1 , • • • , x n , y 1 , • • • , y n ) -E V (x 1 , • • • , x n , y 1 , • • • , y n ) , We have h m W (a, b) -E h m W (a, b) = 1 n! 2 σx,σy T (x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ) (A.20) Annex A. Appendix
Note that T may be rewritten as a sum as in (A.15) with h w W (a,

I k ), w W (b, I k ), C (I k ,I k ) replaced by h w W (a, I k ), w W (b, I k ), C (I k ,I k ) -E h w W (a, I k ), w W (b, I k ), C (I k ,I k ) for each k.
More precisely, we write T

(x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ) = 1 r r-1 k=0 T σx,σy k
for σ x , σ y two permutations of n . Here, T σx,σy k are independent and centered random variables such that

T σx,σy k ≤ 2M 1 m (n -m)! (n -1)! 2 n! (n -m)! 2 (k+1)m i=km+1 a σx(i) (k+1)m i=km+1 b σy(i) ≤ 2M D 2 n 2(1-γ) (A.21) thanks to the Loc A (m, γ, D)-condition and to the constant M = τ (diam(supp(α)) ∪ diam(supp(β))).
In what follows we write T σ = T (x σx(i) ) i , (y σy(i) ) i for simplicity. From (A.20), we get

P h m W (a, b) -E h m W (a, b) ≥ t ≤ e -λt E e λ(h m W (a,b)-E[h m W (a,b)]) = e -λt E e λ 1 n! 2 σ Tσ ≤ e -λt 1 (n!) 2 σ E e λTσ ≤ e -λt max σ E e λTσ ,
where in the first and second inequalities, we used Markov's and Jensen's inequalities respectively. Furthermore, for any T = T σ we have from Lemma 7.3.1 along with (A.21),

E e λT = r-1 k=0 E e λ r T k ≤ r-1 k=0 E e 2λ 2 r 2 M 2 D 4 n 4(1-γ) = exp 2λ 2 M 2 D 4 mn 3-4γ (A.22) Hence, P h m W (a, b) -E h m W (a, b) ≥ t ≤ exp -λt + 2λ 2 M 2 D 4 m • n 3-4γ
Optimizing the latter over λ ∈ R + and following a similar reasoning for -

h m W (a, b) -E h m W (a, b) gives (A.13).
The GW p proof follows the same reasoning but differs in the definition of V which would be equal to:

V (x 1 , • • • , x n , y 1 , • • • , y n ) = 1 r r-1 k=0 P W a (I k )P W b (I k )GW w W (a, I k ), w W (b, I k ), C 1 (I k ,I k ) , C 2 (I k ,I k ) , with ground costs C 1 = C n,p (X, X) and C 2 = C n,p (Y , Y ).

Sampling with replacement

Let us now consider the sampling with replacement P U with the reweighting function w U . We follow the same procedure as in section 5.C from [START_REF] Hoeffding | Probability Inequalities for Sums of Bounded Random Variables[END_REF]]. For the sake of simplicity, we abbreviate w U (a, I) as u ∈ Σ m and give the proof for h ∈ {W p , W ε , S ε } as the GW case can be deduced from it. In this case, it is possible to rewrite h m U as a sum over m-tuples without replacement, i.e.,

h m U (a, b) = I1∈ n m ,I 2 ∈ n m P U a (I 1 )P U b (I 2 )h u, u, C (I1,I2) (A.23) = n m (n -m)! n! 2 I1∈P m ,I2∈P m h (I 1 , I 2 ). (A.24)
Where h is a weighted arithmetic mean of certain values of h. Let us take an example with m = 2, we consider (i 1 , i 2 ) ∈ P 2 and (j 1 , j 2 ) ∈ P 2 we have:

h ((i 1 , i 2 ), (j 1 , j 2 )) = n -1 n 2 P U a ((i 1 , i 2 ))P U b ((j 1 , j 2 ))h(u, u, C (i1,i2),(j1,j2) ) + n -1 n 2 P U a ((i 1 , i 1 ))P U b ((j 1 , j 2 ))h(u, u, C (i1,i1),(j1,j2) ) + n -1 n 2 P U a ((i 1 , i 2 ))P U b ((j 1 , j 1 ))h(u, u, C (i1,i2),(j1,j1) ) + 1 n 2 P U a ((i 1 , i 1 ))P U b ((j 1 , j 1 ))h(u, u, C (i1,i1),(j1,j1) )
More examples for V-statistics can be found in (section 5.C, [START_REF] Hoeffding | Probability Inequalities for Sums of Bounded Random Variables[END_REF]). Following the above example, we see that we have the bounds : 0 ≤ h ≤ max I1 max I2 P U a (I 1 )P U b (I 2 )M . From now, the proof is like the sampling without replacement proof and we only give the main differences.

We fix r = n/m . Let 0 ≤ k ≤ r -1, we define the set

I k := {km + 1, • • • , km + m}.
Then we define the function V as :

V (x 1 , • • • , x n , y 1 , • • • , y n ) = 1 r r-1 k=0 h I k , I k (A.25)
We recall the implicit dependence in X and Y in the right-hand-side of (A.25) through the ground costs C. In the summation below, σ x or σ y denotes a generic permutation of {1, • • • , n}. We compute :

( n m n! ) 2 σx,σy V (x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ) (A.26) = (n m ) 2 (n -m)! 2 n! 2 I 1 ∈P m I 2 ∈P m h I 1 , I 2 (A.27) = h m U (a, b) (A.28)
Finally, we have :

h m U (a, b) = 1 n! 2 σx,σy V (x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ), (A.29)
where the function V is defined by

V = (n m ) 2 V . Let us define T (x 1 , • • • , x n , y 1 , • • • , y n ) = V (x 1 , • • • , x n , y 1 , • • • , y n ) -E V (x 1 , • • • , x n , y 1 , • • • , y n ) , We have h m U (a, b) -E h m U (a, b) = 1 n! 2 σx,σy T (x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ) (A.30)
Note that T may be rewritten as a sum as in (A.25) with h u, u,

C (I k ,I k ) replaced by h u, u, C (I k ,I k ) - E h u, u, C (I k ,I k ) for each k. More precisely, we write T (x σx(1) , • • • , x σx(n) , y σy(1) , • • • , y σy(n) ) = 1 r r-1 k=0 T σx,σy k
for σ x , σ y two permutations of n . Here, T σx,σy k are independent and centered random variables such that

|T σx,σy k | ≤ 2M n 2m max I1 P U a (I 1 ) max I2 P U b (I 2 ) ≤ 2M D 2m n 2m(1-γ) (A.31) Annex A. Appendix With M = τ (diam(supp(α)) ∪ diam(supp(β))
) and where the second inequality uses Definition 8.27. In what follows we write T σ = T (x σx(i) ) i , (y σy(i) ) i for simplicity. From (A.30), we get

P h m U (a, b) -E h m U (a, b) ≥ t ≤ e -λt max σ E e λTσ ,
Furthermore, for any T = T σ we have from Lemma 7.3.1 along with (A.31),

E e λT ≤ r-1 k=0 E e 2 λ 2 r 2 M 2 D 4 = exp 2 λ 2 r M 2 D 4m n 4m(1-γ) (A.32) Hence, P h m U (a, b) -E h m U (a, b) ≥ t ≤ exp -λt + 2λ 2 M 2 D 4m mn 4m-1-4mγ
Optimizing the latter over λ ∈ R + and following a similar reasoning for - .14) gives the desired results.

h m U (a, b) -E h m U (a, b) gives (A
The sampling with replacement bounds show that when the minibatch size m gets bigger, a and b must have a γ close to 1. Now that we have a deviation between the complete estimator and its mean, we focus on the approximation of the complete estimator with its incomplete counter part.

Deviation between the incomplete and complete estimator h.

We are now ready to give the second and last ingredient of our proof: a deviation between the complete and the incomplete estimators. And in order to prove it, we rely on the Hoeffding inequality.

Lemma A.1.2 (Hoeffding's inequality). Let X 1 , . . . , X n be independent random variables such that X i takes its values in [a i , b i ] almost surely for all i ≤ n. Let the random variable

S = n i=1 (X i -EX i ) .
Then for every t > 0, we have:

P {S ≥ t} ≤ exp - 2t 2 n i=1 (b i -a i ) 2 (A.33)
The next reformulation of the incomplete estimator is useful to prove deviation bounds between the complete and the incomplete estimators. See Lemma A.1.

P | h m,k w,P (a (n) , b (n) ) -h m w,P (a (n) , b (n) )| ≥ M 2 log(2/δ) k ≤ δ, (A.34) where M = τ (diam(supp(α)) ∪ diam(supp(β)))
Proof. Thanks to Remark 11 we have

h m,k w,P (a (n) , b (n) ) -h m w,P (a (n) , b (n) ) = 1 k k =1 ω l where ω l = I,J∈ n m b P a (n) ,P b (n) (I, J)-P a (n) (I)P b (n) (J) h w(a (n) , I), w(b (n) , J), C (I,J) . Conditioned upon X = (x 1 , • • • , x n ) and Y = (y 1 , • • • , y n )
, the variables ω l are independent, centered and bounded by 2M with M = τ (diam(supp(α)) ∪ diam(supp(β))) thanks to Lemma A.6. Using Hoeffding's inequality yields

P(| h m,k w,P (a (n) , b (n) ) -h m w,P (a (n) , b (n) )| > ε) = E[P(| h m,k w,P (a (n) , b (n) ) -h m w,P (a (n) , b (n) )| > ε|X, Y )] = E[P(| 1 k k l=1 ω l )| > ε|X, Y )] E[2e -kε 2 2M 2 ] = 2e -kε 2 2M 2
which concludes the proof.

Proof of Theorem 8.3.1.

We have now the three ingredients to prove Theorem 8. (n) , b (n) ) and Eh m w,P (a (n) , b (n) ) depending on the number of empirical data n and the number of batches k: .35) where M = τ (diam(supp(α)) ∪ diam(supp(β))). And for the sampling with replacement, let the sequences of probability vectors (a (n) ), (b (n) ) verify Loc G (m, γ, D) for some γ ∈ (1 -1 4m , 1] and D > 0.

P   | h k W (a (n) , b (n) ) -Eh m W (a (n) , b (n) )| ≥ M 2 D 2 m 1 2 n 2(γ-3 4 ) 2 log( 2 δ ) + 2 log( 2 δ ) k   ≤ δ. ( A
P   | h k U (a (n) , b (n) ) -Eh m U (a (n) , b (n) )| ≥ M 2 D 2m m 1 2 n 2m(γ-1+ 1 4m ) 2 log( 2 δ ) + 2 log( 2 δ ) k   ≤ δ, (A.36)
Proof. With the triangle inequality, we have:

| h m,k w,P (a (n) , b (n) ) -Eh m w,P (a (n) , b (n) )| ≤ | h m,k w,P (a (n) , b (n) ) -h m w,P (a (n) , b (n) )| + |h m w,P (a (n) , b (n) ) -Eh m w,P (a (n) , b (n) )| (A.
E[| h k W (a (n) , b (n) ) -Eh m W (a (n) , b (n) )|] 20 • M max 2 √ 2D 2 m 1 2 n 2(γ-3 4 ) , 2 k (A.38) E[| h k U (a (n) , b (n) ) -Eh m U (a (n) , b (n) )|] 20 • M max 2 √ 2D 2m m 1 2 n 2m(γ-1+ 1 4m ) , 2 k (A.39)
Proof. Once again, we give the proof for the sampling without replacement and the proof for the sampling with replacement follows the same steps. The proof for the debiased minibatch is straight forward as we have three terms of the form n) ). Let us recall the formula : for a real random variable X. If X 0 then E[X] = ´∞ 0 P(X > λ)dλ. We denote by X the random variable

h k W (a (n) , b (n) ) -Eh m W (a (n) , b (
| h k W (a (n) , b (n) ) -Eh m W (a (n) , b (n) )|.
The last Theorem 8.3.1 writes

P X > C log( 2 δ ) δ where C := 2M max(2 √ 2D 2 m 1 2 n 2(γ-3 4
) , 2 k ). We can rewrite it as

P(X > λ) exp (- λ 2 C 2 )
Thus, using the formula above:

E[X] ˆ∞ 0 exp (- λ 2 C 2 )dλ = C ˆ∞ 0 exp(-u 2 )du 10C
as announced.

Bounded cost.

In this paragraph we also prove the case for bounded cost.

Theorem 8. For simplicity, we assume that the maximum in (A.53) is attained at σ = Id. Observe the following equality

1 A δ = n i=1 1 B(ρx,δ) (X i )1 B(ρy,δ) (Y i ) (A.54)
We write T = T Id and insert the indicator function 1 B(ρx,δ) c (X 1 ) using (A.54) and (A.52),

E e λT 1 A δ = E exp λ r r-1 k=1 T k 1 A δ = E 1 B(ρx,δ) c (X 1 ) + E exp λ r r-1 k=1 T k 1 A δ 1 B(ρx,δ) (X 1 ) .
Hence, by repeating this procedure we find, using the tail estimate on the variables

X k , Y k (1 ≤ k ≤ n): E e λT Id 1 A δ ≤ 4n exp - δ 2 2σ 2 + E exp λ r r-1 k=1 T k 1 A δ . (A.55)
The first term in the right on side is the same as in (A.48) so this loss is acceptable. In the following,

we estimate E exp λ r r-1 k=1 T k 1 A δ . We denote, for each 0 ≤ k ≤ r -1, by A δ (k) the set of data from X(I k ) which belong to the ball of radius δ around their mean:

A δ (k) := i∈I k { x i -ρ x 2 ≤ δ} ∩ { y i -ρ y 2 ≤ δ},
and the set of the remaining n -rm data:

A δ (r) := n i=rm+1 { x i -ρ x 2 ≤ δ} ∩ { y i -ρ y 2 ≤ δ}.
Using the independence properties of the T k 's recalled above we get

E exp λ r r-1 k=1 T k 1 A δ = E r-1 k=1 exp λ r T k 1 A δ (k) 1 A δ (r) = r-1 k=1 E exp λ r T k 1 A δ (k) E 1 A δ (r) ≤ r-1 k=1 E exp λ r T k 1 A δ (k) . (A.56) Hence, it suffices to estimate E exp λ r T k 1 A δ (k) for a fixed 0 ≤ k ≤ r -1. Let 0 ≤ k ≤ r -1.
From Lemma A.6, the Loc A (m, γ, D) property of our variables, we have for ω ∈ A δ (k):

T ω k (x i ) i∈I k , (y i ) i∈I k ≤ 2(ρ + δ) p D 2 n 2(1-γ) =: M δ . (A.57)
Unfortunately, we can not use Lemma 7.3.1 on T ω k because we need to have a bounded random variable. To overcome this issue we introduce:

T δ k := T k 1 |T k |≤M δ + M δ 1 |T k |>M δ . (A.58) Annex A. Appendix
which is bounded by M δ . Note that by construction the following equality holds

E exp λ r T k 1 A δ (k) = E exp λ r T δ k 1 A δ (k) .
Thus we can compute using Lemma 7.3.1:

E exp λ r T k 1 A δ (k) = E exp λ r (T δ k -E T δ k ) 1 A δ (k) exp λ r E T δ k ≤ exp λ 2 M 2 δ 8r 2 exp λ r E T δ k . (A.59)
Hence, taking the products in (A.59) for 0 ≤ k ≤ r, we have

(A.56) ≤ r-1 k=0 exp λ 2 M 2 δ 8r 2 • r-1 k=0 exp λ r E T δ k . (A.60)
From Lemma A.1.4 proven below, we have

E T δ k ≤ n 1-2γ+ 1 5 (A.61)
Thus, combining (A.60) and (A.61) we get by optimizing in λ the inequalities,

(A.53) ≤ exp λ 2 mM 2 δ 4n -λ(t -n 1-2γ • n 1 5 ) ≤ exp - n mM 2 δ (t -n 1-2γ • n 1 5 ) 2 ≤ exp - n 4mM 2 δ t 2 , = exp - n 4(γ-3 Annex A. Appendix
Proof. We write

E T δ k = E T δ k 1 A δ (k) + E T δ k 1 A δ (k) c =: I + II (A.75)
From (A.58) we observe using the definition of M δ in (A.57), the union bound and the inequality e -x = e -1 5 x • e -4 5 x ≤ τ x p • e -4 5 x for x, p ≥ 1 and some constant τ = τ (p).

|II| ≤ M δ • P A δ (k) c ≤ 2(ρ + δ) p D 2 n 2(1-γ) • 4m exp - δ 2 2σ 2 ≤ τ (D, m, σ, p) • δ p n 2(1-γ) • τ (p) δ 2p exp - 4 5 • δ 2 2σ 2 ≤ τ (D, m, σ, p) • 1 log(n) p 2 n 2(1-γ) • n -4 5 ≤ 1 2 n 1-2γ+ 1 5 . (A.76)
In the above, we used the assumptions (A.51) and (A.43) in the last two inequalities. We now estimate the contribution of I. Recalling the definition of T δ k in (A.58) we observe the following using the definition of M δ in (A.57) and the fact that T k is a mean-zero random variable,

I = E T k 1 |T k |≤M δ 1 A δ (k) = E T k 1 A δ (k) = -E T k 1 A δ (k) c . (A.77)
Note that the the indicator function 1 A δ (k) c can be expressed as the superposition of at most 4 m functions of the form

f (J 1 , J 2 ) := k1∈J1 1 B(ρx,δ) c (x k1 ) • k2∈J2 1 B(ρy,δ) c (y k2 ), (A.78) where J 1 , J 2 ⊂ I k with |J 1 | + |J 2 | ≥ 1.
Consider a positive real numbers r and a vector z, we denote an annulus around the vector z as A(z, r), i.e., x ∈

A(z, r) if r ≤ x -z 2 ≤ 2r. For i ∈ {1, 2} let t i = |J i | and write J i = {j 1 (i), • • • , j ti (i)}.
By decomposing each indicator function in (A.78) dyadically we get from the (mutual) independence of the variables {x i , y j : (i, j) ∈ J 1 × J 2 } and Lemma A.1.1, .80) where max = max (j 1 (1)), • • • (j t1 (1)), (j 1 (2)), • • • (j t2 (2) . Hence, the sub-Gaussian assumption on the family {x i , y j : (i, j) ∈ J 1 × J 2 , } yields using again the inequality e -x = e -1 5 x • e -4 5 x ≤ C x p • e -4

E T k f (J 1 , J 2 ) = = E T k t1 i1=1 (ji 1 (1))≥0 1 A(ρx,2 (j i 1 (1)) δ) (x ji 1 (1) ) × t2 i2=1 (ji 2 (2))≥0 1 A(ρy,2 (j i 2 (2)) δ) (y ji 2 (2) ) (A.79) = (j1(1)),••• (jt 1 (1))≥0 (j1(2)),••• (jt 2 (2))≥0 E T k 1≤i1≤t1 1≤i2≤t2 1 A(ρx,2 (j i 1 (1)) δ) (x ji 1 (1) )1 A(ρy,2 (j i 2 (2)) δ) (y ji 2 (2) ) ≤ (j1(1)),••• (jt 1 (1))≥0 (j1(2)),••• (jt 2 (2))≥0 D 2 n 2(1-γ) (ρ + 2 max +1 δ) p 1≤i1≤t1 1≤i2≤t2 E 1 A(ρx,2 (j i 1 (1)) δ) (x ji 1 (1) ) E 1 A(ρy,2 (j i 2 (2)) δ) (y ji 2 (2) ) , ( A

A.1.4 Distance to marginals

In this appendix, we give the details of the proof of Theorem 8. 3.4. In what follows, we denote by Π (i) the i-th row of matrix Π. Let us denote by 1 ∈ R n the vector whose entries are all equal to 1.

Theorem 8.3.4 (Distance to marginals). Let δ ∈ (0, 1), two integers m ≤ n and consider two sequences of probability vectors (a (n) ), (b (n) ) ∈ Σ. Let a ground cost C = C m,p for some p ≥ 1 as in 8.2. Consider an OT kernel h ∈ {W p , W p p , W ε , S ε , GW }. Suppose now that the probability law over m-tuples P and the reweighting function w, as defined in (8.12) and (8.13), satisfy the admissibility condition (8.21). For all integers k 1 and all integers 1 i n, we have:

P Π h,k w,P (a (n) , b (n) ) (i) 1 -a (n) i ≥ 2 log(2/δ) k ≤ δ (A.84)
Proof. Let us recall that thanks to the admissibility condition (8.21), Π h w,P is a transport plan between the input probability vectors a (n) and b (n) and hence, it verifies the marginal constraints, i.e (Π h w,P (a (n)

, b (n) )) i × 1 = a i . Thanks to Remark 11 we have Π h,k w,P (a (n) , b (n) ) (i) 1 = 1 k k =1 ω where ω = I,J∈( n m ) 2 n j=1 (Π I,J ) i,j b P a (n) ,P b (n) (I, J). Conditioned upon X = (x 1 , • • • , x n ) and Y = (y 1 , • • • , y n ),
(a (n) , b (n) ) i 1] = Π h w,P (a (n) , b (n) ) i 1. Using Hoeffding's inequality yields P(| Π h,k w,P (a (n) , b (n) ) i 1 -Π h w,P (a (n) , b (n) ) i 1)| > ε) = E[P(| 1 k k p=1 ω p -E[ 1 k k p=1 ω p ])| > ε|X, Y )] 2e -2kε 2
which concludes the proof.

A.1.5 Optimization

This appendix is dedicated to providing the full statements and proofs of Theorem 8. Proof. We start with the case h ∈ {L, L ε }. The function Y → C j,k (X, Y ) is equal to x j -y k p 2 . It is therefore convex, and thus Clarke regular by Proposition 2. 3.6(b) [Clarke 1990]. Since θ → Y θ is C 1 , from Theorem 2. 3.10 [Clarke 1990] it follows that θ → C m,p j,k (X, Y θ ) is Clarke regular, and:

∂ θi C j,k (X, Y θ ) = {D j,k • ∇ θi Y θ : D j,k ∈ ∂ Y C j,k (X, Y θ )}.
Note, that the set of admissible transport plans for any marginals a, b ∈ Σ m is compact. Furthermore, the transport cost for a given plan P is a linear function of cost matrix C. Therefore, from Danskin's Theorem (Proposition B.25 [Bertsekas 1997]) it follows that for h ∈ {L, L ε } the function C → -h(a, b, C) is convex, and it's subderivative is equal to Opt(h, C, a, b). Therefore from Theorem 2.3.9(i) and Proposition 2.3.1 (for s = 1) in [Clarke 1990] it follows that θ → -h(a, b, C(X, Y θ )) is Clarke regular for h ∈ {L, L ε }, and that (A.85) holds. Assume now that h = GW and that p > 1. The proof is analogous. In this case, the function Y → C j1,k1,j2,k2 (X, Y ) is differentiable. Therefore the function θ → C j1,k1,j2,k2 (X, Y θ ) is differentiable. Again the set of admissible transport plans is compact and for a given transport plan, the transport cost is a linear function of the four dimensional tensor C j1,k1,j2,k2 . Therefore, using Danskin's Theorem (Proposition B.25 [Bertsekas 1997]), as well as Theorem 2.3.10 and Proposition 2.3.1 in [Clarke 1990] we get that -h(a, b, C(X, X), C(Y θ , Y θ )) is Clarke regular and the formula (A.86) holds. ))] will follow from Theorem 2.7.2 and Remark 2.3.5 [Clarke 1990], once we establish that the expectation on the left hand side is finite. This follows trivially from the standard bound:

xy p ≤ 2 p-1 ( x p + y p ) (A.92)

and the assumption that X, Y θ have finite p-moments. The same argument applies to the case when h = GW . The GW cost depends on the four-dimensional tensor C j1,k1,j2,k2 defined in the proof of Theorem 8.3.5 in a Lipshitz manner, since it's supdifferential is bounded. Again, the thesis for h = GW will follow from Theorem 2.7.2 and Remark 2.3.5 [Clarke 1990], once we establish that the expectation on the left hand side of (A.91) is finite. This follows from applying the bound (A.92) twice and the assumption of finite 2p-moments.

A.1.6 Minibatch OT closed-form solution for 1D data

We now give the closed-form combinatorial calculus for the 1D data. We start by sorting all the data and give to each of them an index which represents their position after the sorting phase. Then we select and sort all the minibatches. x j can not be at a position superior to its index j inside a batch. For a fixed x j , a simple combinatorial arguments tells you that there are C i xj sets where x j is at the i-th position:

C m,n i,xj = j -1 i -1 n -j m -i (A.93)
Suppose that x j is transported to a y k points in the target mini batch. Then, they both share the same positions i in their respective minibatch. As there are several i where x j is transported to y k , we sum over all those possible positions. Hence our current transport matrix coefficient Π j,k can be calculated as : .94) Where i min = max(0, m -n + j, m -n + k) and i max = min(j, k). i min and i max represent the sorting constraints. Furthermore, as we have uniform weight histograms, we will transport a mass of 1 m and averaged it by the total number of transport. So finally, our transport matrix coefficient Π j,k are:

Π j,k = imax i=imin C m,n i,xj C m,n i,y k (A
Π j,k = 1 m n m 2 imax i=imin C m,n i,xj C m,n i,y k (A.95)
The sampling with replacement case is much more complex and highly computationally costly. Following the same strategy as above, we sort all minibatches. In this case, x i might appears several times more or less than y j and we need to take that into account. We denote m i the number of repetitions of the i-th element and the summation |vecm| = m k=1 m k . We denote N i,j m1,m2 the number of times that x i and y j share the same position in their respective minibatches m 1 and m 2 , after sorting. For the coefficient π i,j , we have :

π i,j = 1 m 1 mn m-1 2 |vecms|=m |vecmt|=m N i,j
ms,mt (A.96)

A.2 Proofs of Chapter 9

While many proofs are similar to the proofs of Chapter 8, we recall them for readers interested only for the uniform measure case. We start with the basic properties of minibatch OT proof in Section A.2.1. Then we prove several UOT properties in Section A.2.2. The UOT properties are at the heart of the main statistical and optimization proofs presented in Section A.2.3.

A.2.1 Basic properties

Proposition 16 (Positivity, symmetry and bias). The minibatch UOT are positive and symmetric losses. However, they are not definites, i.e., h m (X, X) > 0 for non trivial X, τ > 0 and 1 < m < n.

Proof. The first two properties are inherited from the classical UOT cost. Consider a uniform probability vector and random 3-data tuple X = (x 1 , x 2 , x 3 ) with distinct vectors (x i ) 1≤i≤3 . As h m is an average of positive terms, it is equal to 0 if and only if each of its term is 0. But consider the minibatch term I 1 = (i 1 , i 2 ) and I 2 = (i 1 , i 3 ), then obviously h(u, u, C(X(I 1 ), X(I 2 ))) = 0 as x 2 = x 3 , where X(I 1 ) denotes the data minibatch corresponding to indices in I 1 .

We now give the proof for our claim "A simple combinatorial argument assures that the sum of u m over all m-tuples I gives u n ."

Proposition 17 (Averaged distributions). Let u m be a uniform vector of size m. The average over m-tuples I ∈ P m for a given index of u m is equal to ma n , i.e., ∀i ∈ 1, n , I∈P m (u m ) i = (u n ) i = ma n .

Proof. We recall that P m denotes the set of all m-tuples without repeated elements. Let us check we recover the initial weights (u n ) i = ma n . Observe that

A.2.2 Unbalanced Optimal Transport properties

In this appendix we prove the UOT properties necessary to prove the minibatch statistical and optimization theorems. We start with the robustness property and we follow with upper bound, bounded transport plan and Lipschitz properties. We recall the definition of Csiszàr divergences. Consider a convex, positive, lower-semicontinuous function such that φ(1) = 0. Define its recession constant as φ ∞ = lim x→+∞ φ(x)/x. The Csiszàr It allows to generalize OT programs. We retrieve common penalties such as Total Variation and Kullback-Leibler divergence by respectively taking φ(x) = |x -1| and φ(x) = (x log xx + 1). We provide a generalized definition of all OT programs as OT τ,ε φ (a, b, C) = min Where F denotes the UOT energy.

Robustness

We start by showing the robustness properties Lemma 9.1.1 that we split in two different lemmas. Lemma 9.1.1.a shows that the UOT cost is robust to an outlier while Lemma 9.1.1.b shows that OT is not robust to an outlier.

Lemma 9.1.1.a. Take (µ, ν) two probability measures with compact support, and z outside of ν's support.

Recall the Gaussian-Hellinger distance [Liero 2017] between two positive measures as Proof. Write π the optimal plan for OT τ,0 φ (µ, ν). We consider a suboptimal plan for OT τ,0 φ (μ, ν) of the form If ||Π t || 1 = m Π → +∞, then L(m Π ) → +∞ if ε > 0 and L(m Π ) → min C ij + 2φ ∞ > 0 otherwise. In both cases, as t → ∞ and ||Π t || 1 = m Π → +∞, we are supposed to approach the infimum but its lower bound goes to +∞, which contradicts the fact that the optimal OT cost is finite. More precisely, there exists a large enough value M such that for m Π > M , the lower bound is superior to the upper bound M m a m b + τ m a φ(m b ) + τ m b φ(m a ) and thus necessarily not optimal. Furthermore, M depends on (m a , m b , M ) since 0 ≤ C ≤ M . Thus, there exists M > 0 and some t 0 such that for t ≥ t 0 any plan approaching the optimum statisfies ||Π t || 1 ≤ M . The sequence (Π t ) t is in a finite dimensional, bounded, and closed set, i.e. a compact set. One can extract a converging subsequence whose limit is a plan attaining the minimum. Thus any optimal plan is necessarily in a compact set.

GH τ (µ, ν) = inf
To generalize to local compactness, we consider δ > 0 and a neighbourhood U of C such that for any C ∈ U one has 0 ≤ C ≤ max C + δ. Reusing the above proof yields the existence of M such that for any C ∈ U , any plan approaching the optimum satisfies ||Π t || 1 ≤ M , but this time M depends on (m a , m b , max C + δ), which is independent of C in its neighbourhood.

As each element Π of Opt(h, C, a, b) is bounded by a constant M , Opt(h, C, a, b) is a compact space of M + (X ). We denote the maximal constant M which bounds all elements of Opt(h, C, a, b) as M Π . We

Proof of Theorem 9.2.1

We now give the details of the proof of Theorem 9.2.1. We separate Theorem 9.2.1 in two sub theorems: Theorem 9.2.1.a and Theorem 9.2.1.b. In the Theorem 9.2.1.a, we show the deviation bound between h m,k and E h and in Theorem 9.2.1.b, we show the deviation bound between Π h,m,k and Π h,m . For Theorem 9.2.1.a, we rely on two lemmas. The first lemma bounds the deviation between the complete estimator h m and its expectation E h . We denote the floor function as x which returns the biggest integer smaller than x.

Lemma A.2. The second lemma bounds the deviation between the incomplete estimator h m,k and the complete estimator h m .

Lemma A.2.4 (Deviation bound). Let δ ∈ (0, 1), three integers k ≥ 1 and m ≤ n be fixed, and two compactly supported distributions α, β. Consider two n-tuples X ∼ α ⊗n and Y ∼ β ⊗n and a kernel h ∈ {OT τ,ε φ , S τ,ε φ } with the squared Euclidean distance as ground cost. We have a deviation bound between h m,k (X, Y ) and h m (X, Y ) depending on the number of batches k: where ∂ θi is the Clarke subdifferential with respect to θ i , ∇ Y C j,k is the differential of the cell C j,k of the cost matrix with respect to Y , Opt(h, C(X, Y θ ), u, u) is the set of optimal transport plan and co denotes the closed convex hull. Note that when ε > 0 the set Opt(h, C(X, Y θ ), u, u) is reduced to a singleton, and the notation co is superfluous.

| h m,k (X, Y ) -h m (X, Y )| M h
Proof. We start with the regularity of θ → -OT τ,ε φ (u, u, C(X, Y θ )). To prove the Clarke regularity of this map, we rely on a chain rule argument. Consider the function Y → C j,k (X, Y ), it is Clarke regular because it is C 1 . Since θ → Y θ is C 1 , it follows by the chain rule that θ → C j,k (X, Y θ ) is C 1 and thus Clarke regular. The Unbalanced OT cost OT τ,ε φ is a minimization of an energy which is linear in C, and it is thus concave in C, hence -OT τ,ε φ is Clarke regular by convexity. Therefore from Theorem 2.3.9(i) and Proposition 2.3.1 (for s = 1) in [Clarke 1990] it follows that θ → -OT τ,ε φ (u, u, C(X, Y θ )) is Clarke regular.

We now furnish the gradients associated to θ → -OT τ,ε φ (u, u, C(X, Y θ )). By chain rule, the gradient of θ → C j,k (X, Y θ ) reads

∇ θi C j,k (X, Y θ ) = ∇ Y C j,k (X, Y θ ) • ∇ θi Y θ .
We now deal with the gradient of the map C → OT τ,ε φ (u, u, C) by verifying the assumptions of Danskin's theorem [Clarke 1975, Theorem 2.1]. We use in particular the remark below [Clarke 1975, Theorem 2.1] which states that the hypothesis on the map are verified if the map is u.s.c in both variables (Π, C) and convex in C. We recall that Opt(h, C(X, Y ), u, u) is a compact and a convex set, thanks to Lemma A.2.1 and Lemma A.2.2. Furthermore, the energy associated to h = OT τ,ε φ is concave in the cost C and l.s.c in (Π, C) [Liero 2017, Lemma 3.9]. From [Clarke 1975, Theorem 2.1] it follows that the subderivatives of the convex function C → -h(u, u, C) are equal to Opt(h, C(X, Y ), u, u), due to the energy's linearity in C. Thus combining the formulas of the Danskin theorem with the Chain rule yields Equation (A.125). When ε > 0 the set Opt(h, C(X, Y θ ), u, u) is reduced to a singleton, and the notation co is superfluous.

We now give the proof for the regularity of the map θ → S τ,ε φ (u, u, C(X, Y θ )) with ε > 0 as when ε = 0, we get the unregularized UOT treated in the above paragraph. We recall that S τ,ε φ is the summation of three terms of the form OT τ,ε φ . For ε > 0 and each term of the sum, the set of optimal plans Opt(OT τ,ε φ , C(X, Y ), u, u) is reduced to a unique element and the differential (A.125) is also a singleton, thus OT τ,ε φ is differentiable. Then S τ,ε φ is differentiable as a difference of differentiable functions. Furthermore S τ,ε φ is also Clarke regular as a difference of differentiable functions.

We finally prove Theorem 9.2.2. Le TO est en particulier adapté à la mesure de distance entre distributions de probabilité, en calculant le coût minimal pour déplacer une distribution vers une autre. Sa force consiste en l'utilisation de la géométrie des espaces de probabilité grâce aux différents coûts.

Cette thèse explore les deux faces de l'interaction entre transport optimal et apprentissage profond. D'abord, nous travaillerons sur comment le TO peut définir des fonctions de coût pertinentes pour les réseaux de neurones. Nous nous concentrerons sur la définition d'une régularisation basée sur le TO pour l'apprentissage en présence de labels corrompus et sur l'utilisation du TO pour la génération de données mal classifiées pour un réseau pré-entrainé. Nous étudierons ensuite le TO au travers de son utilisation dans l'apprentissage profond avec un focus particulier sur l'utilisation de sous lots. Nous analyserons alors les gains et pertes tant théoriques que pratiques de cette méthode.

Title: Optimal Transport and Deep Learning : Learning from one another Keywords: Optimal Transport, Minibatch, Noisy Labels, Generative Adversarial Network Abstract: Deep learning (DL) models are artificial neural networks and they have arisen as the current most competitive method to make data-driven decisions. In classification, these networks have a more complex representation of data and thus they make more complex predictions. However, DL's recent successes are also due to the development of some mathematical fields : this thesis is about studying the different interactions of DL with one of these fields called Optimal Transport (OT).

To measure the distance between probability distributions, one can rely on the OT theory. It defines a measure through the minimal displacement cost of a distribution to another. Its strength is to use the space geometry with a given ground cost on the data space. Several DL methods are built upon this theory.

This thesis proposes to explore two faces of the interaction between OT and DL. We will first focus on how OT can define meaningful cost functions for neural networks. We will focus on how to define an OT based regularization for learning with noisy labels and how we can use OT to generate misclassified data for a pre-trained classifier. We will then explore what can be learnt about OT from DL applications, with a focus on the minibatch approximation of OT. We will answer what are the gains and downsides of the minibatch formulation both in theory and practice.

4 1. 3

 43 R) set of square (real) matrices of size m Measure theory M(X ) the set of Borel finite signed measures on a space X M 1 + (X ) the set of probability measures on a space X α, β probability measures α ⊗ β the product measure of two probability measures α, β, i.e., α⊗β(A×B) = α(A)β(B) α ⊗m m-product measure α supp(α) the support of α δ x the Dirac measure on x, i.e. δ x (y) = 0 if y = x else 1 # the push forward operator. U(α, β) the set of couplings of two continuous probability measures α, β L(α, β, c) the Kantorovitch cost between two probability measures α, β W p (α, β, c) the p-Wasserstein distance between two probability measures α, β GW p (α, β, c X , c Y ) the p-Gromov-Wasserstein distance between two probability measures probability vectors in R n + (or histograms with n bins), i.e. Σ n = {α ∈ R n + | n i=1 a i = 1} a ∈ Σ n probability vector u ∈ Σ n uniform probability vector Σ set of all sequences of probability vectors viii Contents Functions C(X ), C(X , Y) the set of continuous functions form X to R (resp. from X to Y) C b (X ), C b (X , Y) the set of continuous and bounded functions form X to R (resp. from X to Y) C p (X ) the set of functions of class C p . Lip k (X ) the set of Lipschitz functions on X with Lipschitz constant k L p (α) the set of p-integrable functions with respect to a measure α, i.e. f ∈ L p (α) if ´|f | p dα < +∞ ∇ of all index m-tuples P m set of all index m-tuples without replacement P m,o set of all ordered index m-tuples without replacement i∈I f (i) sum over all elements of tuple I Π i∈I f (i) product over all elements of tuple I Learning and Optimal Transport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Contributions of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Organization of the manuscript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Chapter 4

 4 leads us to the second part of this manuscript. It introduces the concept of adversarial examples in deep learning. We give a general definition and the different basic strategies to generate such examples. We then explain how these adversarial examples can attack a pre-trained classifier and the different concepts associated to these attacks. Finally, we show how these adversarial examples can be used to define a new training algorithm, called Virtual Adversarial Training (VAT), for semi-supervised problems. This algorithm promotes a uniform prediction of the classification around each input. It does so by penalizing large changes in the classification.

  Figure 2.1: Optimal transport illustration between 2D measures. The left image represents the continuous optimal transport setting. The right image represents the discrete optimal transport setting. The black lines represent the optimal connections.

Remark 3 .

 3 Suppose the probability measures α and β are discrete uniform measures with the same number of Diracs, i.e., α = 1n n i=1 δ xi , β = 1 n n i=1 δ yi .Then if the measure supports (x i ) i∈ n and (y i ) i∈ n are respectively sorted on R, the identity scaled by a uniform weight is an optimal transport plan. This case is illustrated in the Figure2.2.

Figure 2

 2 Figure 2.2: Illustration of OT between 1D uniform probability measures with sorted supports. The number of samples is set to 10. The optimal transport plan is the identity scaled by 1 n .

Figure 2 . 4 :

 24 Figure 2.4: Optimal transport variants between 2D measures. For the entropic-regularized OT variants, the ε coefficients are set to 0.05 and 0.5.

3 :+ 4 :

 34 Inputs : histograms a and b, cost matrix C, regularization coefficient ε, penalisation coefficient τ2: K = exp(-C/ε) Define at random (u, v) ∈ R n + × R m while (u, v) did not converge doOptimal Transport (UOT) program between measures and cost c is defined as

Figure 2 . 5 :

 25 Figure 2.5: Optimal transport illustration between 2D measures. From left to right, we have an illustration of original OT, entropic-regularized OT and entropic-regularized unbalanced OT for two parameters τ .

Figure 3 . 1 :

 31 Figure 3.1: Illustration of the Office Home dataset.

Figure 3 . 2 :

 32 Figure 3.2: Illustration of [Courty 2017a] method. (left) dataset for training, i.e. source domain and target domain. Note that a classifier estimated on the training examples clearly does not fit the target data. (middle) a data dependent transportation map is estimated and used to transport the training samples onto the target domain. Note that this transformation is usually not linear. (right) the transported labeled samples are used for estimating a classifier in the target domain. Courtesy of [Courty 2017a].

Figure 3 . 3 :

 33 Figure 3.3: Overview of the deepjdot method. While the structure of the feature extractor g and the classifier f are shared by both domains, they are represented twice to distinguish between the two domains. Both the latent representations and labels are used to compute per batch a coupling matrix Π that is used in the global loss function. Courtesy of [Damodaran 2018].

Figure 3 . 4 :

 34 Figure 3.4: Both VAE and WAE minimize two terms: the reconstruction cost and the regularizer penalizing discrepancy between Pz and distribution induced by the encoder Q. VAE forces Q(Z|X = x) to match Pz for all the different input examples x drawn from Pr. This is illustrated on picture (a), where every single red ball is forced to match Pz depicted as the white shape. Red balls start intersecting, which leads to problems with reconstruction. In contrast, WAE forces the continuous mixture QZ := ´Q(Z|X)dPr to match Pz, as depicted with the green ball in picture (b). As a result latent codes of different examples get a chance to stay far away from each other, promoting a better reconstruction. Courtesy of [Tolstikhin 2018].

Chapter 3 .Figure 3 . 5 :

 335 Figure 3.5: VAE (left column), WAE-MMD (middle column), and WAE-GAN (right column) trained on CelebA dataset. In "test reconstructions" odd rows correspond to the real test points. Courtesy of [Tolstikhin 2018].

Figure 3 .

 3 6 gathers some highly realistic GAN-generated examples. The GAN formulation takes the form of a min-max optimization problem. It has thus bring a lot of attention to this underlooked optimization problem and several work were published to justify convergence or empirical methods to fasten convergence [Gidel 2019a, Gidel 2019b].

Figure 3 . 6 :

 36 Figure 3.6: Samples generated by BigGAN model at 512×512 resolution. Courtesy of [Brock 2019].

Figure 3 .

 3 Figure 3.7: Super resolution samples examples from different technique. From left to right : bicubic interpolation, deep residual network optimized for MSE, deep residual generative adversarial network optimized for a loss more sensitive to human perception, original HR image. Courtesy of [Ledig 2017]

  examples: an unexpected weakness . . . . . . . . . . . . . . . . . . . . . . . 45 4.1.2 Generating adversarial examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.1.3 Concepts of adversarial attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.2 Virtual Adversarial Training for semi-supervised learning . . . . . . . . . . . . . . . . . . . 47 4.2.1 Semi-supervised learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.2.2 Virtual Adversarial Training: the power of approximations . . . . . . . . . . . . . . . . we review adversarial examples in deep learning. We first give a definition and general algorithms to generate such examples and detail how we can attack a classifier with such examples. We then describe the semi-supervised learning problem and how we can design a new training strategy based on adversarial examples for semi-supervised learning.

Figure 4 .

 4 Figure 4.1: Example of adversarial example for a GoogLeNet [Szegedy 2014] trained on ImageNet. J(θ, x, y) is the loss function to train the neural network. Courtesy of [Goodfellow 2015].

Chapter 4 .

 4 Adversarial examples in Deep Learninghypotheses.

Contents 5 . 1 5 . 3

 5153 Learning with noisy labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Definition and setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Related work on label noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Wasserstein Adversarial Regularization for label noise . . . . . . . . . . . . . . . . . . . . . 5.2.1 Adversarial regularization: smoothing local predictions . . . . . . . . . . . . . . . . . . . 5.2.2 Wasserstein adversarial regularization to consider class similarities . . . . . . . . . . . . . Numerical experiments on learning with noisy labels and openset noise . . . . . . . . . . . 5.3.1 Image classification on simulated benchmark datasets . . . . . . . . . . . . . . . . . . . . 5.3.2 Image classification on real-world noisy label benchmark datasets . . . . . . . . . . . . . . 5.3.3 Semantic segmentation of aerial images . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.4 Image classification with open set noisy labels . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 5

 5 Figure 5.1: Illustration of the regularization geometry for different losses in the adversarial training. (Top) Regularization values on the simplex of class probabilities. Each corner stands for a class. All losses are computed with respect to a prediction represented as the green x. Colors are as follows: white is zero while darker is bigger. In the case of WAR, the ground cost C is given on the left. (Down) Classification boundaries when using these losses for regularization. The unregularized classifier (CCE) is given on the left.

  swap): DRESS → T-SHIRT/TOP, COAT ↔ SHIRT, SANDAL → SNEAKER, SHIRT → PULLOVER, ANKLE BOOT → SNEAKER. Baselines We compared the proposed WAR with an informative C matrix based on the word2vec embedding (WAR w2v ) against several state-of-the-art methods: Unhinged [Rooyen 2015], Bootstrapping [Reed 2015], Forward and Backward loss correction [Patrini 2017], Dimensionality driven learning (D2L) [Ma 2018], Co-Teaching [Han 2018], Co-Teaching+ [Yu 2019], Symmetric cross entropy (SL) [Wang 2019], Pencil [Kun 2019b] and finally JoCoR [Wei 2020]. We then compare WAR w2v with WAR embed , AR and WAR 0-1 . Finally, as a baseline for all the considered methods we also included a categorical cross entropy (CCE) loss function. All the methods shared the same architecture and training procedures, as detailed below. * https://github.com/bbdamodaran/WAR 60 Chapter 5. Wasserstein Adversarial Regularization for learning with noisy labels

Figure 5 .

 5 Figure 5.2: Comparison of the original and the corrupted ground truths for the semantic segmentation experiment.

Figure 5 . 3 :

 53 Figure 5.3: Semantic segmentation maps obtained on the test set of the ISPRS Vaihingen dataset (tile #12 of the original data). The top row shows the full image, and the second row shows a close-up of the area delineated in orange.

Figure 6 . 1 :

 61 Figure 6.1: Illustration of different reweighting strategies for adversarial data generation. (a) is the standard uniform weight between data. (b) is a hard weighting where we only consider misclassified data. (c), (d) and (e) are softmax weighting strategy for different temperatures. (f) is the combination of softmax and clipping strategies.

Figure 6

 6 Figure 6.2: Adversarial data generation with WGAN (left) and ARWGAN (right) on two moons dataset. The black line is the classifier boundary. The kernel density estimation of generated data is in red.

Figure 6 . 3 :

 63 Figure 6.3: [Best viewed in color] Pavia University false RGB (left) and ground truth used for training (center) and testing (right).

Figure 6 . 5 :

 65 Figure 6.4: SVM Classifier confusion matrix on Pavia dataset

Figure 6 . 6 :

 66 Figure 6.6: [Best viewed in color] Comparison between several class spectra means against meadows spectra from Pavia dataset. All means are reported centered around the mean spectrum of meadows for better visualization. The spectra means are denoted in plain line and the standard deviations are in dotted lines

Figure 6 .

 6 Figure 6.7: [Best viewed in color] Comparison with state-of-the-art[Song 2018] class spectra means against meadows spectra. All means are reported centered around the mean spectrum of meadows for better visualization. The spectra means are denoted in plain line and the standard deviations are in dotted lines

Figure 6 .

 6 Figure 6.8: Comparison between hard and soft weighting spectra for painted metal sheets class on Pavia dataset. We plot 100 spectra for both weighting strategies.

Figure 6 . 9 :

 69 Figure 6.9: Data from the DFC2018 dataset. Example of false RGB image (left) and the ground truth (right).

Figure 6 .

 6 Figure 6.11: [Best viewed in color] Healthy grass visualization. (Top) False RBG image built from original spectra (left) and from adversarial spectra (right). (Bottom) Classifier prediction on the original spectra (left) and adversarial spectra (right).

Figure 6 .

 6 Figure 6.13: [Best viewed in color] Comparison between several class spectra means against car and cross-walk spectra. All means are reported centered around the mean spectrum of car or cross-walk for better visualization. The means are in plain and the standard deviations are in dotted lines.

Figure 6 .

 6 Figure 6.14: Segmentation network for detection on Potsdam dataset

Figure 6 .

 6 Figure 6.16: Mask modification generator architecture on Potsdam and Vaihingen datasets

Figure 6 .

 6 Figure 6.17: Results for patches modification, first row show examples where our method worked, second row show failure cases, and the third row is showing the difference between the two other rows, for the labels we only showed the difference with respect to ground truth of the car class.

Figure 6 .

 6 Figure 6.18: Results on Vaihingen dataset. For each example, original (top) and adversarial (bottom) images with car heatmaps. Our method reduces the classifier's probability.

Figure 6 .

 6 Figure 6.19: Car generator architecture trained on Potsdam dataset [pot ]

Figure 6 .

 6 Figure 6.20: Adversarial car images for the YoloV3 detector. Left column are GT adversarial examples, middle column are WGAN adversarial examples and right column are ARWGAN adversarial examples

  scale Optimal Transport through minibatches computation . . . . . . . . . . . . . . 92 7.2 Optimal Transport statistical and smoothness properties . . . . . . . . . . . . . . . . . . . 93 7.2.1 Empirical estimation of Optimal Transport . . . . . . . . . . . . . . . . . . . . . . . . . . 93 7.2.2 Minimizing Optimal Transport with respect to a parameter θ . . . . . . . . . . . . . . statistics definition and application in Machine Learning . . . . . . . . . . . . . . . . . 99 7.3.2 U-statistics concentration bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Theorem 7.2.2 (Theorem 3, [Genevay 2019]). Consider two measures α and β on X and Y, two 94 Chapter 7. Optimal Transport statistical properties and subsampling subset domains of R d , with a C ∞ , L-lipschitz cost c. One has:

  .2) where κ = 2L|X |+ c ∞ and the constants only depend on |X |, |Y|, d and c (k) ∞ for k = 0, • • • , [d/2]. c (k) denotes the k-th differential and |X | denotes the diameter of the space X .

Figure 7

 7 Figure 7.1: Optimal transport plan illustration between 2D measures. The left image represents the original optimal transport plans. The right image represents the unique entropic-regularized optimal transport plan.

Contents 8 . 1

 81 Expectation of Optimal Transport over minibatches . . . . . . . . . . . . . . . . . . . . . . 8.1.1 Empirical estimators in the uniform case . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.2 Illustration examples on 1D ans 2D data . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 8.1.3 Empirical estimators in the general case . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Debiased minibatch Optimal Transport: a Sinkhorn divergence approach . . . . . . . . . . 8.2.1 Metric properties: a fundamental difference . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 Debiasing minibatch Wasserstein losses . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.3 Positivity: a negative counter example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Learning with minibatch Optimal Transport: concentration bounds and gradients . . . . . 8.3.1 Concentration bounds between estimators and their expectations . . . . . . . . . . . . . . 8.3.2 Unbiased gradients for stochastic optimization . . . . . . . . . . . . . . . . . . . . . . . . flows between human faces with minibatch Optimal Transport . . . . . . . . . . 8.4.2 Mapping estimation between human faces with minibatch Optimal Transport . . . . . . . 126 8.4.3 Generative adversarial networks on Cifar-10 with minibatch Optimal Transport . . . . . . 128 8.4.4 Large scale color transfer between images . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.5 Minibatch Gromov-Wasserstein rotation and translation invariance . . . . . . . . . . . . .

  The symmetry property allows us to rewrite the minibatch OT definition. Let P o,m be the set of ordered m-tuples without replacement, we can rewrite (8.3) as: Y ) is an expectation over the combinatorial number of all possible pairs of m-tuples I, J, it is often estimated by drawing only k such pairs of m-tuples uniformly, called subsample quantity.Definition 17 (Minibatch subsampling). Consider the notations from Definition 16. Pick two integers k > 0 and 0 < m ≤ n. Then, we define the incomplete estimator:

Figure 8 Figure 8

 88 Figure 8.1: Several OT plans between measures with n = 20 samples in 1D. The first row shows the minibatch OT plans Π h,m (u, u) for different values of m, the second row provides the shape of the measures on the rows of Π h,m (u, u). h is the exact OT. The two last columns correspond to classical entropic and quadratic regularized OT.

Example 1 (

 1 Uniform reweighting function). The uniform reweighting function w U is independent of the input discrete probability a. It is defined coordinatewise for any m-tuple I = (i 1 , . . . , i m ) by w U k (a, I) = 1 m , 1 ≤ k ≤ m and yields to a uniform probability vector in Σ m . Example 2 (Normalized reweighting function). The normalized reweighting function w W normalizes the restriction of the input discrete probability a to the support of I, to ensure it remains a discrete probability. It is defined coordinatewise for any m-tuple I = (i 1 , . . . , i m ) by w W k (a, I) = k ≤ m, which is again a probability vector even if entries in I are repeated.

Example 3 (

 3 Drawing indices with replacement). Drawing i ∈ n , 1 ≤ ≤ m i.i.d. (with replacement) from the discrete probability distribution a ∈ Σ n yields the law on indices P U a (I) = Π i∈I a i .

  Definition 21 an OT kernel h, two reweighting functions w 1 , w 2 and a family of probability distributions {P a : a ∈ Σ n } over index m-tuples from 1, n , where 1 ≤ m ≤ n. Given discrete probabilities a, b ∈ Σ n and data tuples X, Y , consider for each pair of m-tuples I, J the discrete probabilities a I = w 1 (a, I) ∈ Σ m , b J = w 2 (b, J) ∈ Σ m . Let Π I,J be defined as in Definition 18 but Π m I,J is an optimal transport plan with marginals a I , b J . The averaged minibatch transport matrix is Π h,m w1,w2,Pa,P b (a, b) = E I∼Pa,J∼P b Π I,J (8.19)

Figure 8 . 3 :

 83 Figure 8.3: Difference between generated 2D data from different MBOTGAN. We used the uniform formalism (left) and the general formalism with replacement h m U (right). We use the minibatch OT loss as a loss function for GANs. The black line represents the classifier boundary decision. The hyperparameters for the reweighted distributions are set to κ = 0.5 and w = 20. We trained the generator for 30000 iterations.

  Empirical justification. We introduce some notations for the aforementioned Examples 1, 2, 3, 4. When referring to these 4 examples, we define two estimators. h m W (respectively Π h,m W ) with law P W a and reweighting function w W stands for the minibatch Wasserstein loss (respectively minibatch OT plan) over the m-tuples without repetitions. And h m U (respectively Π h,m U

Figure 8 . 4 :

 84 Figure 8.4: Difference between transport plan estimators with 2D distributions and n = 5 samples. Each column gives the OT plan Π h,m W (u, u) or Π h,m U

  m w,P (b, b) = 0. We debias the minibatch OT losses by following the same idea as the Sinkhorn divergence, we remove half of each self term h m w,P (a, a) and h m w,P (b, b).

Figure 8 . 5 :

 85 Figure 8.5: Positivity counter example. (Left) source and target distributions for a given perturbation. (Middle and right) Comparison of different estimator values for ΛW 1 and ΛW 2 with an Euclidean ground cost between the distributions. The red line is the y-axis equals to 0.

  us consider 8 points on the unit circle equally distributed. Then let us add a perturbation as a rotation to each point position, where the rotation vary from 0 to π. The perturbed distribution becomes our target distribution. When computing the quantity Λ Wp (u, u), with p ≥ 2 and an Euclidean ground cost, it can return a negative value. We give the variations of the debiased minibatch OT losses in function of the pertubartion in Figure8.5 for both the estimators h m W and h m U .

Figure 8 . 6 :

 86 Figure 8.6: LocA and LocG local constraints illustrations on the simplex with m = 2 and n = 3.

  Figure 8.7: Λ h,m,k W (a, b) as a function of n in log-log space. Here (a, b) are two probability vectors associated to X and Y ∼ α ⊗n , where α is the uniform distribution on the unit cube [0, 1] d . (Left) Λ h,m,k W (a, b) is tested for several values of d ∈ {2, 7, 10} and a fix m = 128 or (right) Λ h,m,k W (a, b) is tested for several values of m ∈ {64, 128, 256} and a fix d = 7. The experiments were run 5 times and the shaded bar corresponds to the 20% and 80% percentiles.

Figure 8

 8 Figure 8.10: Map learning between 5000 male source images and 5000 female target images. The batch size m is set to 128 and the number of batch couple k is set to 1. (First row) Source data. (Second and third rows) Respectively minibatch Wasserstein without replacement and with replacement (r) mapping on the CelebA dataset in a DFC-VAE latent space. (Fourth and fifth rows) Respectively unbiased minibatch Wasserstein without and with replacement (r) mapping on the CelebA dataset in a DFC-VAE latent space.

Figure 8 .

 8 Figure 8.11: Generated samples with the same latent vectors from different GANs.

Figure 8 .

 8 Figure 8.12: Color transfer between full images for different batch size and number of batches. (Top) color transfert from image 1 to image 2. (Bottom) color transfer from image 2 to image 1.

Figure 8

 8 Figure 8.13: (left) L1 error on both marginals (log-log scale). We selected 1000 points from original images and computed the error on marginals for several m and k (log-log scale). (Right) Sparsity of incomplete minibatch OT plan Π W 2 2 ,m,k W (u, u). We selected 1000 points from original images and computed the sparsity of Π W 2 2 ,m,k W (u, u) for several k and m.

Figure 8 Figure 8

 88 Figure 8.15: MDS on the galloping horse animation with MB Gromov-Wasserstein loss and its debiased variant. Each sample in this Figure corresponds to a mesh and is colored by the corresponding time iteration. One can see that the cyclical nature of the motion is recovered.

Contents 9 . 1

 91 Robustness to outliers and sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 9.2 Statistical and optimization properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 9.2.1 Deviation bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 9.2.2 Unbiased Clarke gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 9.3 Numerical experiments on gradient flows and domain adaptation . . . . . . . . . . . . . . 142 9.3.1 Unbalanced MiniBatch OT gradient flow: a qualitative example . . . . . . . . . . . . . . 142 9.3.2 JUMBOT: a new approach for domain adaptation . . . . . . . . . . . . . . . . . . . . . . 143 9.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

Figure 9

 9 Figure 9.1: Several OT costs between 2D distributions with n = 10 samples and m = 5. Target distribution is equal to the source distribution tainted with a moving outlier (green dot). The shaded area represent the variance of subsample MBOT on 5 run.

9. 3 .

 3 Figure 9.3: (Best viewed in colors) Minibatch UOT gradient flow on a 2D dataset. Source data and target data are divided in 2 imbalanced clusters, source left (red) and target right (green) shapes have 6400 samples while source right (purple) and target left (orange) shapes have 3600 samples. The batch size m is set to {64, 128} and the number of minibatch k is set to 1, meaning that the explicit Euler integration step is conducted for each batch. Results are computed with the (unbalanced) minibatch Sinkhorn divergence losses.

Figure 9

 9 Figure 9.4: T-SNE embeddings of 10000 test samples for MNIST (source) and MNIST-M(target) for deepjdot and our method. It shows the ability of the methods to discriminate classes (samples are colored w.r.t. their classes).

Figure 9 . 5 :

 95 Figure 9.5: (Best viewed in colors) deepjdot and jumbot sensitivity analysis. We report the classification accuracy of deepjdot and jumbot on the DA tasks USPS → MNIST and SVHN → MNIST for several hyperparameter variations. We consider the marginal coeffifient τ , the entropic coefficient ε and the batch size m.

Figure 9 . 6 :Figure 9

 969 Figure 9.6: Percentage of mass between data with different labels for jdot and jumbot during the USPS to MNIST DA task.

  of future works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 10.2.1 Perspectives on our contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 10.2.2 Perspectives on Optimal Transport in Machine Learning . . . . . . . . . . . . . . . . . . 153

  1 n = a and 1 n Π h w,P = b . Considering the Wasserstein kernel h = W p p , the minibatch loss defined in Definition (21), as the associated coupling Π h w,P is not the optimal coupling of the full OT problem, it satisfies h m w,P (a, b) = Π h w,P , C F ≥ h(a, b). (a, b) an averaged minibatch transport plan. Proof. By the definition of Π m h and the properties Q J 1 n = 1 m and Π m I,J 1 m = a I = w(a, I) we have Π h w,P 1

b

  3 and Theorem 8.3.4 below. Remark 11. Let m, n ≥ m and k be positive integers. Let b a,b (I, J)) I,J∈ n m ,1≤ ≤k be a sequence of mutually independent Bernoulli variables of parameter P a (I)P b (J) such that b Pa,P b (I, J) =    1 if (I, J) has been selected in the -th draw Pa,P b (I, J)h w(a, I), w(b, J), C (I,J) Π h,k w,P (a, b) = 1 k k =1 I,J∈ n m b Pa,P b (I, J)Π I,J . ready to bound the deviation between the two quantities and the following lemma gives us the wanted deviation: Lemma A.1.3 (Deviation bound). Consider two compactly supported distributions α, β, two n-tuples of empirical data X ∼ α ⊗n , Y ∼ β ⊗n . Let (a (n) ), (b (n) ) ∈ Σ be two sequences of probability vectors, let δ ∈ (0, 1) and an integer k 1. Consider a reweighting function w, a probability law over m-tuple P as in Examples 1, 2, 3, 4 and an OT kernel h ∈ {W p , W ε , S ε , GW p } with a ground cost C as in 8.2. We have a deviation bound between h m,k w,P (a, b) and h m w,P (a, b) depending on the number of minibatches k.

5 .175

 5 Let a, b ∈ Σ m . Let X be a R dm -valued random variable, and {Y θ } a family of R dmvalued random variables defined on the same probability space, indexed by θ ∈ Θ, whereΘ ⊂ R q is open. Assume that θ → Y θ is C 1 . Denote C = C m,p for some p ≥ 1 and let h ∈ {L, L ε }. Then the function θ → -h(a, b, C(X, Y θ )) is Clarke regular and for all 1 ≤ i ≤ q we have:∂ θi h(a, b, C(X, Y θ )) = co{-tr(P • D T ) • (∇ θi Y ) :P ∈ Opt(h, C(X, Y θ ), a, b), (A.85) D ∈ R m,m , D j,k ∈ ∂ Y C j,k (X, Y θ )}where ∂ θi is the Clare subdifferential with respect to θ i , ∂ Y C j,k is the subdifferential of the cell C j,k of the cost matrix with respect to Y , co denotes the closed convex hull and Opt(h, C, a, b) is defined in Definition 18.For h = GW and p > 1, the function -h(a, b, C(X, X), C(Y θ , Y θ )) is also Clarke regular, and we have:∂ θi h(a, b, C(X, X), C(Y θ , Y θ )) = {-m j1,j2,k1,k2=1 D j1,k1,j2,k2 P j1,j2 P k1,k2 • (∇ θi Y ) : (A.86) P ∈ Opt(h, C(X, X), C(Y θ , Y θ ), a, b) (A.87) D ∈(R m ) 4 , D j1,k1,j2,k2 = ∇ Y C j1,k1,j2,k2 (X, Y θ )}where C j1,k1,j2,k2 = C j1,k1 (X, X) -C j2,k2 (Y θ , Y θ ) p .

Theorem 8. 3 . 6 .

 36 Let a, b, X, Y , C be as in theorem 8.3.5, h ∈ {L, L ε }, and assume in addition that the random variables X, {Y θ } θ∈Θ have finite p-moments. If for all θ ∈ Θ there exists an open neighbourhood U , θ ∈ U ⊂ Θ, and a random variable K U : Ω → R with finite expected value, such thatC(X(ω), Y θ1 (ω)) -C(X(ω), Y θ2 (ω)) ≤ K U (ω) θ 1 -θ 2 (A.88)then we have∂ θ E [h(a, b, C(X, Y θ ))] = E [∂ θ h(a, b, C(X, Y θ ))] . (A.89)with both expectation being finite. Furthermore the function θ→ -E [h(a, b, C(X, Y θ ))] is also Clarke regular.For h = GW , assume that p > 1 and that random variables X, {Y θ } have finite 2p-moments. Assume also that for each θ ∈ Θ there exists an open neighbourhood U , θ ∈ U ⊂ Θ, and a random variableK U : Ω → R with finite expected value, such that C(X(ω), Y θ1 (ω)) -C(X(ω), Y θ2 (ω)) ≤ K U (ω) θ 1 -θ 2 (A.90) where C(X, Y ) = C(X, X) -C(Y , Y ) p . Then we have ∂ θ E [h(a, b, C(X, X), C(Y θ , Y θ ))] = E [∂ θ h(a, b, C(X, X), C(Y θ , Y θ ))] . (A.91)with both expectation being finite. Furthermore the function θ→ -E [h(a, b, C(X, X), C(Y θ , Y θ ))] is also Clarke regular. Annex A. Appendix Proof. We start with the case h ∈ {L, L ε }. Suppose that U ⊂ Θ is open and K U is a function for which (A.88) is satisfied. Then the same bound is also satisfied for the function h(a, b, C(X(ω), Y θ (ω))), since the function C → h(a, b, C) is 1-Lipshitz. Hence, given the regularity of -h(a, b, C(X, Y θ )), the interchange (A.89) and regularity of θ → -E[h(a, b, C(X, Y θ

n

  i=1 a i = m a and that for each 1 ≤ i ≤ n{I ∈ P m : i ∈ I} = {I ∈ P m : n ∈ I} = {I = (i 1 , . . . , i m ) ∈ P m : i 1 = n} + . . . + {I = (i 1 , . . . , i m ) ∈ P m : i m = n} = m • {I = (i 1 , . . . , i m ) ∈ P m : i m = n}.(A.97)Since {I = (i 1 , . . . , i m ) ∈ P m : i m = n} is the number of (m -1)-tuples without repeated indices of 1, n -1 , (n -1)!/(n -m)!, it follows that (n -

  + τ D φ (Π1 n |a) + τ D φ (Π T 1 n |b) + εKL(Π|a ⊗ b).

  π≥0 ˆC(x, y)dπ(x, y) + τ KL(π 1 |µ) + τ KL(π 2 |ν). For ζ ∈ [0, 1], write μ = ζµ+(1-ζ)δ z a measure perturbed by a Dirac outlier. Write m(z) = ´C(z, y)dν(y). One hasGH τ (μ, ν) ≤ ζ GH τ (µ, ν) + 2τ (1 -ζ)(1 -e -m(z)/2τ ).(A.100)In particular, with the notation OT τ,0 KL it readsOT τ,0 KL (μ, ν, C) ≤ ζ OT τ,0 KL (µ, ν, C) + 2τ (1 -ζ)(1 -e -m(z)/2τ).

  π = ζπ + (1 -ζ)κδ z ⊗ ν,where κ is mass parameter which will be optimized after. Note that the marginals of the plan π are π1 = ζπ 1 + (1 -ζ)κδ z and π2 = ζπ 2 + (1 -ζ)κν. Note that KL is jointly convex, thus one hasKL(π 1 |μ) ≤ ζKL(π 1 |µ) + (1 -ζ)KL(κδ z |δ z ), KL(π 2 |μ) ≤ ζKL(π 2 |ν) + (1 -ζ)KL(κν|ν).Thus a convex inequality yieldsOT τ,0 φ (μ, ν) ≤ ζ ˆ||x -y||dπ(x, y) + τ KL(π 1 |µ) + τ KL(π 2 |ν) + (1 -ζ) κm(z) + τ KL(κδ z |δ z ) + τ KL(κν|ν) .Annex A. AppendixProof. As ab , C < +∞ is finite, one can bound the ground cost C as 0 C i,j M . Consider the OT kernel h ∈ {OT τ,ε φ } for any ε ≥ 0. Let us consider the transport plan Π = ab = (a i b j ) (with respect to the cost matrix C). Because all terms are positive, we have:|h| ab , C + εKL(ab |ab ) + τ D φ (ab )1 n |a + τ D φ (ba )1 n |b , M i,j a i b j + τ m a φ(m b ) + τ m b φ(m a ), M m a m b + τ m a φ(m b ) + τ m b φ(m a ). (A.105)Defining M h a,b as the last upper bound finishes the proof. The case h = S τ,ε φ , is the sum of three terms of the form OT τ,ε φ Thus the sumM h a,b + 1 2 M h a,a + 1 2 M h b,b is an upper bound of S ε .We now bound the UOT plan.Lemma 9.2.1.b (locally compact optimal transport plan). Assume that ab , C < +∞. Consider regularized or unregularized UOT with entropy φ and penalty D φ such that one has φ ∞ > 0. Then there exists an open neigbhourhood U around C, and a compact set K, such that the set of optimal transport plan for any C ∈ U is in K, i.e., Opt(h, C, a, b) ⊂ K. Furthermore, if all costs are uniformly bounded such that 0 ≤ C ≤ M < ∞, then the compact K can be taken global, i.e. independent of C.Proof. We identify the mass of a positive measure with its L1 norm, i.e. m a = a i = ||a|| 1 . We first consider the case where 0 ≤ C ≤ M < ∞. The OT cost is finite because the plan π = ab is suboptimal and yields OT τ,ε φ (a, b, C) ≤ M m a m b + τ m a φ(m b ) + τ m b φ(m a ) < +∞. Take a sequence Π t approaching the infimum. Note that thanks to the Jensen inequality, one has D φ (x, y) ≥ m y φ(m x /m y ) (see [Liero 2017]). Write m Π = Π t,ij . One has for any t Π t , C + τ D φ (Π t,1 1 n |a) + τ D φ (Π t,2 1 n |b) + εKL(Π t |ab ), ≥ m Π min C ij + τ m b , ≥ m Π L(m Π ).

  least 1 -δ and where M h u,u is an upper bound defined in Lemma 9.2.1.a.Proof. First note that h m,k (X, Y ) is a subsample quantity of h m (X, Y ). Let us consider the sequence of random variables ((b l (I, J) (I,J)∈P m ) 1 l k such that b l (I, J) is equal to 1 if (I, J) has been selected at the l-th draw and 0 otherwise. By construction of h m,k , the aforementioned sequence is an i.i.d sequence of random vectors and the b l (I, J) are Bernoulli random variables of parameter 1/|Γ|. We then haveh m,k (X, Y ) -h m (X, ω l = (I,J)∈P m (b l (I, J) -1 |Γ| )h(I, J). Conditioned upon X = (x 1 , • • • , x n ) and Y = (y 1 , • • • , y n ),the variables ω l are independent, centered and bounded by 2M h u,u thanks to Lemma 9.2.1.a. Using

3

 3 Let u be a uniform probability vector and let X, Y , C be as inLemma A.2.5, h ∈ {OT τ,ε φ , S τ,ε φ }, and assume in addition that the random variables X, {Y θ } θ∈Θ are compactly supported. If for all θ ∈ Θ there exists an open neighbourhood U , θ ∈ U ⊂ Θ, and a random variable K U : Ω → R with finite expected value, such thatC(X(ω), Y θ1 (ω)) -C(X(ω), Y θ2 (ω)) ≤ K U (ω) θ 1 -θ 2 , (A.126)then we have∂ θ E [h(u, u, C(X, Y θ ))] = E [∂ θ h(u, u, C(X, Y θ ))] . (A.127) with both expectation being finite. Furthermore the function θ → -E [h(u, u, C(X, Y θ ))] is alsoClarke regular. transport illustration between 2D measures. The left image represents the continuous optimal transport setting. The right image represents the discrete optimal transport setting. The black lines represent the optimal connections. . . . . . . . . . . . . . . . . . .2.2Illustration of OT between 1D uniform probability measures with sorted supports. The number of samples is set to 10. The optimal transport plan is the identity scaled by 1 n . . . 2.3 Time comparison of optimal transport variants between 2D measures with growing support.The figure shows that the entropic-regularized OT with the Sinkhorn algorithm or the stochastic algorithms are faster to compute than original OT for a large number of samples. We used the solvers from the POT library[START_REF] Flamary | [END_REF]]. . . . . . . . . . . . . . . . . . .2.4Optimal transport variants between 2D measures. For the entropic-regularized OT variants, the ε coefficients are set to 0.05 and 0.5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Optimal transport illustration between 2D measures. From left to right, we have an illustration of original OT, entropic-regularized OT and entropic-regularized unbalanced OT for two parameters τ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Illustration of the Office Home dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Illustration of [Courty 2017a] method. (left) dataset for training, i.e. source domain and target domain. Note that a classifier estimated on the training examples clearly does not fit the target data. (middle) a data dependent transportation map is estimated and used to transport the training samples onto the target domain. Note that this transformation is usually not linear. (right) the transported labeled samples are used for estimating a classifier in the target domain. Courtesy of [Courty 2017a]. . . . . . . . . . . . . . . . . . 3.3 Overview of the deepjdot method. While the structure of the feature extractor g and the classifier f are shared by both domains, they are represented twice to distinguish between the two domains. Both the latent representations and labels are used to compute per batch a coupling matrix Π that is used in the global loss function. Courtesy of [Damodaran 2018]. 3.4 Both VAE and WAE minimize two terms: the reconstruction cost and the regularizer penalizing discrepancy between P z and distribution induced by the encoder Q. VAE forces Q(Z|X = x) to match P z for all the different input examples x drawn from P r . This is illustrated on picture (a), where every single red ball is forced to match P z depicted as the white shape. Red balls start intersecting, which leads to problems with reconstruction. In contrast, WAE forces the continuous mixture Q Z := ´Q(Z|X)dP r to match P z , as depicted with the green ball in picture (b). As a result latent codes of different examples get a chance to stay far away from each other, promoting a better reconstruction. Courtesy of [Tolstikhin 2018]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of Figures 6.9 Data from the DFC2018 dataset. Example of false RGB image (left) and the ground truth (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.10 Generator (Top) and critic (Bottom) 3 convolutional layer architectures to generate adversarial hyperspectral DFC2018 data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.11 [Best viewed in color] Healthy grass visualization. (Top) False RBG image built from original spectra (left) and from adversarial spectra (right). (Bottom) Classifier prediction on the original spectra (left) and adversarial spectra (right). . . . . . . . . . . . . . . . . . 6.12 [Best viewed in color] Comparison between several class spectra means against healthy grass spectra from DFC2018 dataset. All means are reported centered around the mean spectrum of healthy grass for better visualization. The spectra means are denoted in plain line and the standard deviations are in dotted lines. . . . . . . . . . . . . . . . . . . . . . 6.13 [Best viewed in color] Comparison between several class spectra means against car and cross-walk spectra. All means are reported centered around the mean spectrum of car or cross-walk for better visualization. The means are in plain and the standard deviations are in dotted lines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.14 Segmentation network for detection on Potsdam dataset . . . . . . . . . . . . . . . . . . . 6.15 Segmentation network confusion matrix on Potsdam dataset for the considered U-net. . . 6.16 Mask modification generator architecture on Potsdam and Vaihingen datasets . . . . . . . 6.17 Results for patches modification, first row show examples where our method worked, second row show failure cases, and the third row is showing the difference between the two other rows, for the labels we only showed the difference with respect to ground truth of the car class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.18 Results on Vaihingen dataset. For each example, original (top) and adversarial (bottom) images with car heatmaps. Our method reduces the classifier's probability. . . . . . . . . 6.19 Car generator architecture trained on Potsdam dataset [pot ] . . . . . . . . . . . . . . . . 6.20 Adversarial car images for the YoloV3 detector. Left column are GT adversarial examples, middle column are WGAN adversarial examples and right column are ARWGAN adversarial examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7.1 Optimal transport plan illustration between 2D measures. The left image represents the original optimal transport plans. The right image represents the unique entropic-regularized optimal transport plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Several OT plans between measures with n = 20 samples in 1D. The first row shows the minibatch OT plans Π h,m (u, u) for different values of m, the second row provides the shape of the measures on the rows of Π h,m (u, u). h is the exact OT. The two last columns correspond to classical entropic and quadratic regularized OT. . . . . . . . . . . . . . . . 8.2 Several OT plans between 2D measures with n = 10 samples. The first row shows the minibatch OT plans Π h,m (u, u), where h is the exact optimal transport cost, for different values of m. The second row provide a 2D visualization of where the mass is transported between the 2D positions of the sample. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Difference between generated 2D data from different MBOTGAN. We used the uniform formalism (left) and the general formalism with replacement h m U (right). We use the minibatch OT loss as a loss function for GANs. The black line represents the classifier boundary decision. The hyperparameters for the reweighted distributions are set to κ = 0.5 and w = 20. We trained the generator for 30000 iterations. . . . . . . . . . . . . . . . . . 114 8.4 Difference between transport plan estimators with 2D distributions and n = 5 samples. Each column gives the OT plan Π u) (top) and the shape of the distributions on the rows of the OT plans (bottom). We consider the exact optimal transport cost as h. 115 8.5 Positivity counter example. (Left) source and target distributions for a given perturbation. (Middle and right) Comparison of different estimator values for Λ W1 and Λ W2 with an Euclidean ground cost between the distributions. The red line is the y-axis equals to 0. . 117 8.6 Loc A and Loc G local constraints illustrations on the simplex with m = 2 and n = 3. . . . . 119 8.7 Λ h,m,k W (a, b) as a function of n in log-log space. Here (a, b) are two probability vectors associated to X and Y ∼ α ⊗n , where α is the uniform distribution on the unit cube [0, 1] d . (Left) Λ h,m,k W (a, b) is tested for several values of d ∈ {2, 7, 10} and a fix m = 128 or (right) Λ h,m,k W (a, b) is tested for several values of m ∈ {64, 128, 256} and a fix d = 7. The experiments were run 5 times and the shaded bar corresponds to the 20% and 80% percentiles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 8.8 minibatch OT gradient flow on the CelebA dataset in a DFC-VAE latent space. Source data are 5000 male images while target data are 5000 female images. The batch size m is set to 200 and the number of minibatch k is set to 10. (r) means that the probability law on m-tuple is P U otherwise it is P W . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 8.9 Unbiased minibatch Wasserstein gradient flow on the CelebA dataset in a DFC-VAE latent space. Source data are 5000 male images while target data are 5000 female images. The batch size m is set to 200 and the number of minibatch k is set to 10. (r) means that the probability law on m-tuple is P U otherwise it is P W . . . . . . . . . . . . . . . . . . . . . . . 127 8.10 Map learning between 5000 male source images and 5000 female target images. The batch size m is set to 128 and the number of batch couple k is set to 1. (First row) Source data. (Second and third rows) Respectively minibatch Wasserstein without replacement and with replacement (r) mapping on the CelebA dataset in a DFC-VAE latent space. (Fourth and fifth rows) Respectively unbiased minibatch Wasserstein without and with replacement (r) mapping on the CelebA dataset in a DFC-VAE latent space. . . . . . . . . . . . . . . . . 128 8.11 Generated samples with the same latent vectors from different GANs. . . . . . . . . . . . 130 8.12 Color transfer between full images for different batch size and number of batches. (Top) color transfert from image 1 to image 2. (Bottom) color transfer from image 2 to image 1. 131 8.13 (left) L1 error on both marginals (log-log scale). We selected 1000 points from original images and computed the error on marginals for several m and k (log-log scale). (Right) Sparsity of incomplete minibatch OT plan Π u) for several k and m. . . . . . 132 8.14 Average value of MBGW and GW losses as a function of rotation angle on 2D spirals. Colored areas correspond to the 20% and 80% percentiles. Experiments were run 10 times. 133 8.15 MDS on the galloping horse animation with MB Gromov-Wasserstein loss and its debiased variant. Each sample in this Figure corresponds to a mesh and is colored by the corresponding time iteration. One can see that the cyclical nature of the motion is recovered. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 comparison between MBGW, SGW, RISGW, GW, entropic-GW between two 100-D random distributions with varying number of points from 0 to 10 4 in log-log scale. The time includes the calculation of the pair-to-pair distances. . . . . . . . . . . . . . . . 134 9.1 Several OT costs between 2D distributions with n = 10 samples and m = 5. Target distribution is equal to the source distribution tainted with a moving outlier (green dot). The shaded area represent the variance of subsample MBOT on 5 run. . . . . . . . . . . . 139 9.2 Several OT plans, normalized by their maximum value, between 2D distributions with n = 10 samples. The first row shows the minibatch OT plans Π h,m for different values of m and different OT kernels, the second row provides an equivalent geometric interpretation of the OT plans, where the mass transportation is depicted as connections between samples.140 9.3 (Best viewed in colors) Minibatch UOT gradient flow on a 2D dataset. Source data and target data are divided in 2 imbalanced clusters, source left (red) and target right (green) shapes have 6400 samples while source right (purple) and target left (orange) shapes have 3600 samples. The batch size m is set to {64, 128} and the number of minibatch k is set to 1, meaning that the explicit Euler integration step is conducted for each batch. Results are computed with the (unbalanced) minibatch Sinkhorn divergence losses. . . . . . . . . . . . 143 9.4 T-SNE embeddings of 10000 test samples for MNIST (source) and MNIST-M(target) for deepjdot and our method. It shows the ability of the methods to discriminate classes (samples are colored w.r.t. their classes). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 9.5 (Best viewed in colors) deepjdot and jumbot sensitivity analysis. We report the classification accuracy of deepjdot and jumbot on the DA tasks USPS → MNIST and SVHN → MNIST for several hyperparameter variations. We consider the marginal coeffifient τ , the entropic coefficient ε and the batch size m. . . . . . . . . . . . . . . . . . . . . . . . . 147 9.6 Percentage of mass between data with different labels for jdot and jumbot during the USPS to MNIST DA task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 9.7 (Best viewed in colors) deepjdot and jumbot class accuracies along training. We report the class accuracies along training of deepjdot and jumbot on the DA task MNIST → M-MNIST for optimal hyper-parameters. Each color represents a different class. . . . . . 148List of Tables 5.1 Test accuracy (%) of different models on Fashion-MNIST (F-M), Cifar-10, and Cifar-100 datasets with varying noise rates (0% -40%). The mean accuracies and standard deviations averaged over the last 10 epochs of three runs are reported, and the best results are highlighted in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Comparison of variants of WAR with AR with varying noise rates (0% -40%). The mean accuracies and standard deviations averaged over the last 10 epochs of three runs are reported, and the best results are highlighted in bold. . . . . . . . . . . . . . . . . . . . 5.3 CNN models used in our learning with noisy label experiments on Fashion-MNIST, CIFAR-10 and CIFAR-100. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Test accuracy (in %) of adversarial regularization methods: AR, WAR 0-1 , WAR w2v and WAR embed with different η values on CIFAR-10 dataset with 40% noise level. The average accuracies and standard deviations over last 10 epochs are reported for one run. . . . . . . 5.5 Test accuracy of different models on Clothing1M dataset with ResNet-50. (*) refers to results reproduced by us. ( †) means WAR w2v result at the epoch showing the best validation accuracy assessed on the clean validation set. . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Per class F1 scores, average F1 score and overall accuracy (%) on the test set of Vaihingen. The best results (on the noisy dataset) are highlighted in bold. . . . . . . . . . . . . . . . 5.7 Test accuracy on CIFAR10 dataset with 40% openset samples from SVHN and ImageNet32. 5.8 CNN models used in our open set noisy label experiments on CIFAR-10 with openset noise from SVHN and ImageNet32. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Classification accuracy comparison between sprectra from Full Pavia image, classical GAN reconstruction, and ARWGAN reconstruction (10 runs), lower is better . . . . . . . . . . 6.2 Sensitivity analysis of the temperature parameter w for generated meadows from Pavia dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Classification accuracy for the pre-trained classifier over ARWGAN generated spectra and real DFC2018 data (10 runs). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Details on the three different classifiers used to test the generalisation of our generator, while having similar overall accuracy we can see by looking at healthy grass accuracy that they do not learned the same features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Results for cross model transferability. Each column corresponds to a different generator trained against a classifier and each row corresponds to a test on a classifier or a combination of them. Results are the percentage of adversarial examples for 1000 generated samples (over 10 runs). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Results of perceptual evaluation, we can observe that out method lower the accuracy on both dataset while retaining a good accuracy from human perception. . . . . . . . . . . . Titre : Transport optimal et apprentissage profond : apprendre l'un de l'autre Mot clés : Transport Optimal, Sous-lots, labels corrompus, Réseaux adversaires génératifs Résumé : Les modèles d'apprentissage profond sont des réseaux de neurones artificiels et sont très compétitifs dans le cadre de problèmes décisionnels. En classification, ces réseaux permettent par exemple une représentation plus complexe des données et donc des décisions plus pertinentes elles aussi. Cependant, l'essor de ces réseaux est aussi lié au développement de plusieurs champs des mathématiques : cette thèse porte sur l'interaction des réseaux de neurones avec l'un de ces champs appelé Transport Optimal (TO).

6 Chapter 1. Introduction •

  [START_REF] Flamary | [END_REF]] RémiFlamary, et al. POT: Python Optimal Transport. In: Journal of Machine Learning Research (JMLR) -Open Source Software, 2021.

	• [Pedregosa 2019] Fabian Pedregosa, Kilian Fatras and Mattia Casotto. Proximal Splitting Meets
	Variance Reduction. In: the 22nd International Conference on Artificial Intelligence and Statistics
	(AISTATS), 2019.

2.3. Entropic regularization: formulation and practical interest 21 Algorithm 1 Sinkhorn Algorithm

  This problem is known in the

1: Inputs : histograms a and b, cost matrix C, regularization coefficient ε

to compute their Chapter 2. Introduction to computational Optimal Transport

  Time comparison of optimal transport variants between 2D measures with growing support. The figure shows that the entropic-regularized OT with the Sinkhorn algorithm or the stochastic algorithms are faster to compute than original OT for a large number of samples. We used the solvers from the POT library[START_REF] Flamary | [END_REF]].

			Time for different OT solvers	OT between 2D distributions
		10 1	emd time Sinkhorn time, = 0.01 SAG time, = 0.1 SAG time, = 0.01 Sinkhorn time, = 0.1	Source samples Target samples
	Time (s)	10 1 10 0
		10 2
		10 2	10 3 Number of samples (n)	10 4
	Figure 2.3:

Chapter 3 Optimal Transport in Deep Learning Contents 3.1 Classification using neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

  Domain adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	3.1.1 Supervised learning and multi-label classification . . . . . . . . . . . . . . . . . . . . . . . 31
	3.1.2

33 3.2 Generating high realistic data with Deep Learning . . . . . . . . . . . . . . . . . . . . . . . 36

	3.2.1 AutoEncoder: Encoding data as a latent space . . . . . . . . . . . . . . . . . . . . . . . 36
	3.2.2 Generate realistic data with Generative Adversarial Networks . . . . . . . . . . . . . . . . 39
	3

.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

56 Chapter 5. Wasserstein Adversarial Regularization for learning with noisy labels Algorithm 3

  Wasserstein Adversarial Training (WAT) model update

]. Similarities 1: Inputs: x, y, ρ, ξ, η, ε, θ t , γ, k max , ground cost C 2: Select sample (x, y) 3: Random d ∈ R d 4: for k=1, • • • , k max do 5:

Table 5 .

 5 1: Test accuracy (%) of different models on Fashion-MNIST (F-M), Cifar-10, and Cifar-100 datasets with varying noise rates (0% -40%). The mean accuracies and standard deviations averaged over the last 10 epochs of three runs are reported, and the best results are highlighted in bold.

	Data / noise	CCE	Backward	Forward	Unhinge	Bootsoft	CoTeaching	CoTeaching+	D2L	SL	Pencil	JoCoR	WARw2v
				[Patrini 2017] [Patrini 2017] [Rooyen 2015] [Reed 2015] [Han 2018] [Yu 2019]	[Ma 2018]	[Wang 2019] [Kun 2019b] [Wei 2020]	Ours
		0% 94.69±0.11 94.86±0.04	94.81±0.04 95.12 ± 0.03 94.79±0.02 94.28±0.04 93.62±0.01 94.47±0.02	94.18±0.04 93.19 ± 0.23 94.60 ± 0.04 94.70±0.02
	F-M	20% 89.02±0.47 88.84±0.10	91.03±0.12	90.04± 0.08	88.17±0.11 91.24±0.06 92.26±0.02 89.12±0.15	93.36±0.03 92.50 ± 0.09 91.01 ± 0.08 93.37±0.08
		40% 78.85±0.56 81.74±0.08	82.85 ±0.2	78.32 ±0.15	73.84±0.28 86.83±0.10 86.15±0.03 78.98±0.25	86.83±0.07 90.17 ± 0.03 84.86 ± 0.20 90.41±0.02
		0% 91.76±0.04 91.63 ± 0.04 91.59 ± 0.03 92.27 ± 0.04 91.67 ± 0.03 90.12 ±0.04 88.47±0.14 91.29±0.02	90.48±0.05 87.90 ± 0.39 91.94 ± 0.02 91.88±0.31
	Cifar-10	20% 85.26±0.09 84.67 ± 0.1	85.70 ± 0.08 87.09 ± 0.05 85.35 ± 0.8 86.19 ±0.07 82.97±0.25 86.64 ±0.12 87.51±0.06 87.09 ± 0.25 87.93 ± 0.09 89.12±0.48
		40% 76.23±0.15 73.49 ± 0.14 75.10 ± 0.15	77.94 ± 0.1	74.32 ± 0.2 80.87±0.09 72.65±0.10 73.12 ±0.43 76.87±0.12 84.48 ± 1.04 82.07 ± 0.11 84.76±0.25
		0% 68.60±0.09 69.53±0.07	70.12±0.07	70.54±0.06	69.81±0.04 65.42±0.06 58.93±0.14 70.93 ±0.02 68.27±0.06 63.32 ± 0.24 69.29 ± 0.26 68.16±0.18
	Cifar-100	20% 58.81±0.10 59.23±0.08	59.54±0.05	61.06±0.06	58.97±0.08 56.55±0.08 44.88±0.14 60.90±0.03	58.41±0.08 59.05 ± 0.34 60.43 ± 0.07 62.72±0.16
		40% 42.45±0.12 43.02±0.09	42.17±0.1	42.87±0.07	41.73±0.08 42.73±0.08 29.94±0.34 42.61±0.04	40.97±0.12 45.70 ± 0.67 42.26 ± 0.45 58.86±0.21
	Avg,. rank	7.7	6.9	6.2	4.2	8.1	7.1	10.1	6.7	7.0	6.2	5.0	2.5
	noise only	8.8	8.2	7.1	5.6	10.0	5.8	9.5	7.3	6.0	3.5	5.0	1.0

Table 5 .

 5 2: Comparison of variants of WAR with AR with varying noise rates (0% -40%). The mean accuracies and standard deviations averaged over the last 10 epochs of three runs are reported, and the best results are highlighted in bold.

	Methods		Fashion-MNIST			CIFAR-10			CIFAR-100
		0%	20%	40%	0%	20%	40%	0%	20%	40%
	AR	94.81±0.09	93.10±0.14	89.74±0.10	91.49±0.07	88.91±0.09	81.98±0.25	67.83±0.10	65.44±0.11	55.75±0.14
	WAR 0-1	94.60±0.03	90.99±0.07	86.03±0.20	90.94±0.12	86.12±0.21	74.15±0.34	65.78±0.15	60.56±0.14	51.00±0.31
	WAR w2v	94.70±0.02	93.37±0.08 90.41±0.02	91.88±0.31	89.12±0.48	84.76±0.25	68.16±0.18	62.72±0.16	58.86±0.21
	WAR embed	94.63±0.09	93.25±0.15	90.20±0.36	91.88±0.12 89.93±0.02 85.08±0.32	66.58±0.21	62.82±0.76	55.79±0.25

5.3. Numerical experiments on learning with noisy labels and openset noise 61

	Fashion-MNIST	CIFAR-10	CIFAR-100
	28×28×1	32×32×3	32×32 ×3
		3×3 conv, 128 LReLU
		3×3 conv, 128 LReLU
		3×3 conv, 128 LReLU
	2×2 max-pool, stride 2
		dropout, p=0.25
		3×3 conv, 256 LReLU
		3×3 conv, 256 LReLU
		3×3 conv, 256 LReLU
	2×2 max-pool, stride 2
		dropout, p=0.25
		3×3 conv, 512 LReLU
		3×3 conv, 256 LReLU
		3×3 conv, 128 LReLU
		avg-pool
	dense 128 → 10 dense 128 → 10 dense 128 → 100

Table 5 .

 5 3: CNN models used in our learning with noisy label experiments on Fashion-MNIST, CIFAR-10 and CIFAR-100.

Table 5 .

 5 2 reports the performance of WAR w2v , WAR embed , WAR 0-1 and AR, ‡ ‡ We used the code provided by the authors: https://github.com/xingruiyu/coteaching_plus

62 Chapter 5. Wasserstein Adversarial Regularization for learning with noisy labels

	η	AR	WAR 0-1	WAR w2v	WAR embed
	0.5 77.49±0.18 76.80±0.22 77.05±0.36 76.96 ± 0.15
	1	77.25±0.25 76.35±0.21 76.90±0.37 77.00 ± 0.31
	5	81.37±0.21 74.76±0.15 80.16±0.36 83.13 ± 0.12
	10 76.84±0.84 74.14±0.16 84.76±0.25 85.08 ± 0.32
	20 57.36±0.10 75.58±0.18 86.73±0.20 80.36 ± 0.22

Table 5 .

 5 4: Test accuracy (in %) of adversarial regularization methods: AR, WAR0-1, WARw2v and WAR embed with different η values on CIFAR-10 dataset with 40% noise level. The average accuracies and standard deviations over last 10 epochs are reported for one run.

Table 5 .

 5 5: Test accuracy of different models on Clothing1M dataset with ResNet-50. (*) refers to results reproduced by us. ( †) means WARw2v result at the epoch showing the best validation accuracy assessed on the clean validation set.

					Unsupervised					Using yval
	Method	CCE	CCE	bootsoft	D2L	GCE	SL	JoCoR WARw2v	Forward	JOF	Pencil WARw2v
		[Wang 2019] (*) [Rooyen 2015] [Ma 2018] [Zhang 2018b] [Wang 2019] (*)	Unsup. [Patrini 2017] [Tanaka 2018b] (*)	( †)
	Acc.	68.80	68.65	68.94	69.47	69.75	71.02	69.78	71.61	69.84	72.16	69.66 72.20

Table 5 .

 5 6: Per class F1 scores, average F1 score and overall accuracy (%) on the test set of Vaihingen. The best results (on the noisy dataset) are highlighted in bold.

	Class	CCE CCE Bootsoft CoTeaching AR WARw2v
	Training set	Clean		Noisy
	Buildings	90.29 75.06	88.34	75.1	81.6 89.04
	Cars	58.91 14.21	10.98	26.6	21.6 25.78
	Imperv. surf.	85.76 62.20 82.66	76.9	70.9 79.01
	Low vegetation 76.32 25.92	61.40	57.4	57.8 71.56
	Trees	84.72 70.89	80.49	77.5	78.9 82.92
	Average F1	79.20 49.65	64.77	63.6	62.2 69.66
	Total Accuracy 83.89 63.95 78.87	74.6	77.1 78.43

Table 5 .

 5 7: Test accuracy on CIFAR10 dataset with 40% openset samples from SVHN and ImageNet32.

	Dataset	CCE Forward Iterative Co-teaching Co-teaching+ WAR w2v
	CIFAR-10+ 40% SVHN	67.45	56.70	77.73	80.95	77.53	82.03
	CIFAR-10+ 40% ImageNet32 66.69	66.77	79.38	80.34	75.47	80.61
		CIFAR-10+SVHN CIFAR-10+ImageNet32
		32×32×3		32×32 ×3
		3×3 conv, 64 ReLU
		3×3 conv, 64 ReLU
		2×2 max-pool, stride 2
		3×3 conv, 128 ReLU
		3×3 conv, 128 ReLU
		2×2 max-pool, stride 2
		3×3 conv, 196 ReLU
		3×3 conv, 16 ReLU
		2×2 max-pool, stride 2
			dense 256 → 10

Table 5

 5

.8: CNN models used in our open set noisy label experiments on CIFAR-10 with openset noise from SVHN and ImageNet32.

6.2. Adversarial Reweighting WGAN 71 Hard weighting

  An intuitive way to generate adversarial examples is to only consider misclassified data, where a constant weight is given to misclassified data and 0 for correctly classified data. Let N m be the number of misclassified data, then the misclassified data have normalized weights equal to 1/N m . The downside of this weighting strategy is that few data are left for training in the context of a very accurate classifier. This is problematic as GANs are data hungry which means that this method is not efficient to generate

	adversarial examples. To summary we have p hard (x) =	0 Nm 1	if x is correctly classified by f θ otherwise	.

Chapter 6. Generating natural adversarial Remote Sensing Images

  . Performances are competitive as we consider a

	False RGB	Train set	Test set
	Asphalt Meadows	Gravel Trees	Painted metal sheets Bare Soil	Bitumen Self-Blocking Bricks	Shadows

Table 6 .

 6 1: Classification accuracy comparison between sprectra from Full Pavia image, classical GAN reconstruction, and ARWGAN reconstruction (10 runs), lower is better

					Accuracy	Std
		Real meadows		0.89	/
		GAN meadows		0.79	0.004
		ARWGAN meadows	0.35	0.01
	spectral mean -meadows speactra	0.2 0.1 0.0 0.1 0.2 0.3 0.4	Adv. Meadows Meadows Trees Bare Soil
		0	20	40	bands	60	80	100

Table 6 .

 6

	w	0	1	5	10	20	35	50	100
	adv. proportion 25.5% 29.2% 50.4% 55.3% 57.7% 55.0% 61.6% 60.4%
	result observed in recent reviews [Audebert 2019].

2: Sensitivity analysis of the temperature parameter w for generated meadows from Pavia dataset

Table 6 .3. Chapter 6. Generating natural adversarial Remote Sensing Images

 6

	1	1 2 8	3 8 4	64	6	32	1 2 16	2 4	1	4 8
	1
	1	1	16	2 4	32	1 2	64	6	3 8 4	1	1
							1
	Generator: 110,769 parameters		Critic: 22,561 parameters
	INPUTS: 128×1						INPUTS: 48×1
	Dense 128 → 384, ReLU				Conv1D k=3 stride=2, 24×16 ReLU
	Reshape 384 → 6×64				Conv1D k=3 stride=2, 12×32 ReLU
	Conv1D k=3 stride=2, 12×32 ReLU		Conv1D k=3 stride=2, 6×64 ReLU
	Conv1D k=3 stride=2, 24×16 ReLU		Reshape 6×64 → 384
	Conv1D k=5 stride=2, 48×1 TanH		Dense 384 → 1

Figure 6.10: Generator (Top) and critic (Bottom) 3 convolutional layer architectures to generate adversarial hyperspectral DFC2018 data.

Table 6 . 3

 63

		Mean	Std	Real data
	Healthy Grass	0.34	0.06	0.82
	Car	0.19	0.05	0.39
	Crosswalks	0.01	0.009	0.05

: Classification accuracy for the pre-trained classifier over ARWGAN generated spectra and real DFC2018 data (10 runs).

spectra that fit the real spectra unlike ARWGAN. The method from

[Song 2018

], while being able to handle all classes, have greater difficulties that ARGWAN on underrepresented classes even if the classifier has a poor accuracy on these classes.

.

  For this experiment we look at the ratio of adversarial examples generated given classifier A, B or C. All the results are visible in Table 6.5. ARWGAN generates around 90% of adversarial examples against classifiers A, B and C when they are seen during training. From the results, we can see that there

Chapter 6. Generating natural adversarial Remote Sensing Images

  Figure 6.12: [Best viewed in color] Comparison between several class spectra means against healthy grass spectra from DFC2018 dataset. All means are reported centered around the mean spectrum of healthy grass for better visualization. The spectra means are denoted in plain line and the standard deviations are in dotted lines.

				0.100	Comparison of several spectra means against healthy grass spectra
			spectral mean -healthy grass mean	0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075	Healthy grass Adv. Healthy grass Evergreen trees Stressed grass Adv. Song et al.
		0.10 0.15 0.20	Comparison against cars spectra	Comparison against crosswalks spectra Crosswalks Adv. Crosswalks Residential buildings Adv. Song et al.
	spectral means	0.05 0.00 0.05
		0.10
		0.15	Cars Adversarial Cars Non-residential buildings
		0.20

Table 6 . 4 :

 64 Details on the three different classifiers used to test the generalisation of our generator, while having similar overall accuracy we can see by looking at healthy grass accuracy that they do not learned the same features.

		Num. Param.	Overall Accuracy	Healthy Grass Accuracy	Convs Layers	FCs Layers
	Classifier A	728,532	60%	80%	4	3
	Classifier B	186,132	62%	68%	2	2
	Classifier C	7,168,080	52%	75%	/	/

Table 6 .

 6 5: Results for cross model transferability. Each column corresponds to a different generator trained against a classifier and each row corresponds to a test on a classifier or a combination of them. Results are the percentage of adversarial examples for 1000 generated samples (over 10 runs).

	Test	Train	Classifier A	Classifier B	Classifier C
	Classifier A	89.68%, ±0.85% 78.78%, ±1.28% 73.03%, ±1.10%
	Classifier B	93.68%, ±0.68% 89.32%, ±0.44% 83.39%, ±1.18%
	Classifier C	84.05%, ±0.74% 81.50%, ±1.30% 90.70%, ±0.68%
	A & B	88.27%, ±0.79% 77.59%, ±1.43% 71.73%, ±1.12%
	A & C	81.58%, ±0.94% 76.26%, ±1.05% 79.23%, ±1.34%
	B & C	78.36%, ±0.99% 68.15%, ±1.47% 69.57%, ±1.38%
	all		77.80%, ±0.97% 67.52%, ±1.54% 68.71%, ±1.41%

Table 6 . 6

 66

									0.90
		Imp_Surf	0.9	0.019	0.051	0.016	0.0067	0.0064
									0.75
		Building	0.26	0.64	0.054	0.002	0.013	0.036
									0.60
		Low_Veg	0.11	0.03	0.77	0.083	0.0039	0.0053
	True label	Tree	0.048	0.007	0.4	0.54	0.0029	0.0013	0.45
									0.30
		Car	0.23	0.047	0.0043	0.0027	0.7	0.0094
									0.15
		Clutter	0.51	0.12	0.14	0.034	0.12	0.075
			Imp_Surf	Building	Low_Veg Predicted label Tree	Car	Clutter
	Classifier Potsdam		0.762		0.481		/
	Classifier Vaihingen		0.758		0.453		/
	Human Potsdam		0.918		0.835		0.945
	Mean confidence		2.524		2.240		2.229

Figure 6.15: Segmentation network confusion matrix on Potsdam dataset for the considered U-net.

Original image Modified image Image w/o cars : Results of perceptual evaluation, we can observe that out method lower the accuracy on both dataset while retaining a good accuracy from human perception.

Table 6 .

 6

7: Adversarial generation rate for our pre-trained classifier over generated data from different methods.

Expectation of Optimal Transport over minibatches 109 Π

  h,m (X, Y ), C . Thus as Π h,m (X, Y ) is an average of connections, we want to known if all source and target mass are entirely transported, i.e., we want to check that Π h,m (X, Y ) is an admissible transport plan between uniform probability vectors u n . We have the following proposition

	Proposition 11. The transportation plan Π h,m (X, Y ) is an admissible transportation plan between the
	full input measures u

p p , it is straight forward to check that h m C (X, Y ) = 8.1. n , u n . Furthermore, for an exact OT loss kernel h or p-Wasserstein distance to the power p we have :

Table 8

 8

	.1: Generator (left) and critic (right) 4 convolutional layer architectures used in our experiments to generate
	CIFAR10 data.

Table 8 .

 8

2: Inception Scores on CIFAR10 for several GAN variants trained with a batch size of 64. Biggest score is in bold.

8.4. Numerical experiments on generative modelling, color transfer and minibatch Gromov-Wasserstein 133

  Figure 8.14: Average value of MBGW and GW losses as a function of rotation angle on 2D spirals. Colored areas correspond to the 20% and 80% percentiles. Experiments were run 10 times.

	Spiral datasets		MBGW values increasing rotation with k=1000
	Source Target (rot=0) Target (rot= /4) Target (rot= /2)	Value	0.035 0.030 0.015 0.020 0.025	GW MBGW, m=10 MBGW, m=50 MBGW, m=100
			0.010
			0.005
			0	/8 Rotation angle (radian) /4 3 /8	/2

  .7)Our method is called jumbot and stands for Joint Unbalanced MiniBatch OT. It is related to deepjdot[Courty 2017b[START_REF] Damodaran | [END_REF] at the notable exceptions that we use minibatch UOT, which can also handle partial domain adaptation as suggested by our experiments.Datasets. We start with digits datasets. Following the evaluation protocol of[START_REF] Damodaran | [END_REF]] we experiment on three adaptation scenarios: USPS to MNIST (U →M), MNIST to M-MNIST (M →MM), Table 9.1: Summary table of DA results on digit datasets. Experiments were run three times. (*) denotes the reproduced methods.

		Chapter 9. Unbalanced Minibatch Optimal Transport
	Methods/DA	U → M	M → MM	S → M	Avg
	dann(*)	92.2 ± 0.3 96.1 ± 0.6 88.7 ± 1.2 92.3
	cdan-e(*)	99.2 ± 0.1 95.0 ± 3.4 90.9 ± 4.8 95.0
	alda(*)	97.0 ± 1.4 96.4 ± 0.3 96.1 ± 0.1 96.5
	deepjdot(*) 96.4 ± 0.3 91.8 ± 0.2 95.4 ± 0.1 94.5
	e-deepjdot(*) 97.1 ± 0.3 94.2 ± 0.1 97.6 ± 0.1 96.3
	jumbot	98.2 ± 0.1 97.0 ± 0.3 98.9 ± 0.1 98.0

Table 9

 9

	.3 and it shows that our method

. Numerical experiments on gradient flows and domain adaptation 145

		Method	A-C	A-P	A-R	C-A	C-P	C-R	P-A	P-C	P-R	R-A	R-C	R-P	avg
		resnet-50	34.9	50.0	58.0	37.4	41.9	46.2	38.5	31.2	60.4	53.9	41.2	59.9	46.1
		dann (*)	44.3	59.8	69.8	48.0	58.3	63.0	49.7	42.7	70.6	64.0	51.7	78.3	58.3
	da	cdan-e(*) deepjdot (*)	52.5 50.7	71.4 68.6	76.1 74.4	59.7 59.9	69.9 65.8	71.5 68.1	58.7 55.2	50.3 46.3	77.5 73.8	70.5 66.0	57.9 54.9	83.5 78.3	66.6 63.5
		e-deepjdot (*) 50.6	68.9	74.4	59.3	65.1	69.0	56.2	46.5	74.5	65.1	54.7	78.1	63.5
		alda (*)	52.2	69.3	76.4	58.7	68.2	71.1	57.4	49.6	76.8	70.6	57.3	82.5	65.8
		rot (*)	47.2	71.8	76.4	58.6	68.1	70.2	56.5	45.0	75.8	69.4	52.1	80.6	64.3
		jumbot	55.2	75.5	80.8	65.5	74.4	74.9	65.2	52.7	79.2	73.0	59.9	83.4	70.0
		resnet-50	46.3	67.5	75.9	59.1	59.9	62.7	58.2	41.8	74.9	67.4	48.2	74.2	61.4
		deepjdot(*)	48.2	66.2	76.6	56.1	57.8	64.5	58.3	42.7	73.5	65.7	48.2	73.7	60.9
	pda	e-deepjdot(*) pada	47.6 51.9	67.0 67.0	77.3 78.7	57.1 52.2	57.9 53.8	65.4 59.0	58.1 52.6	41.3 43.2	74.4 78.8	66.4 73.7	47.7 56.6	75.1 77.1	61.3 62.1
		etn		59.2	77.0	79.5	62.9	65.7	75.0	68.3	55.4	84.4	75.7	57.7	84.5	70.4
		ba3us(*)	56.7	76.0	84.8	73.9	67.8	83.7	72.7	56.5	84.9	77.8	64.5	83.8	73.6
		jumbot	62.7	77.5	84.4	76.0	73.3	80.5	74.7	60.8	85.1	80.2	66.5	83.9	75.5
	Table 9.2: Summary table of DA and Partial DA results on Office-Home (ResNet-50). (*) denotes the reproduced
	methods.
	Method	A-C A-P	A-R	C-A	C-P	C-R	P-A	P-C	P-R	R-A	R-C	R-P	avg
	resnet-50 34.9 50.0	58.0	37.4	41.9	46.2	38.5	31.2	60.4	53.9	41.2	59.9	46.1
	dann(*)	46.2 65.2	73.0	54.0	61.0	65.2	52.0	43.6	72.0	64.7	52.3	79.2	60.7
	cdan-e(*) 52.8 71.4	76.1	59.7	70.6	71.5	59.8	50.8	77.7	71.4	58.1	83.5	67.0
	alda(*)	53.7 70.1	76.4	60.2	72.6	71.5	56.8	51.9	77.1	70.2	56.3	82.1	66.6
	rot(*)	47.2 70.8	77.6	61.3	69.9	72.0	55.4	41.4	77.6	69.9	50.4	81.5	64.6
	deepjdot(*) 53.4 71.7	77.2	62.8	70.2	71.4	60.2	50.2	77.1	67.7	56.5	80.7	66.6
	jumbot	55.3 75.5 80.8 65.5 74.4 74.9 65.4 52.7 79.3 74.2 59.9	83.4	70.1
	Table 9.3: Vanilla domain adaptation experiments on Office-Home dataset with maximum classification along
	training iterations. (ResNet50)

Table 9 .

 9 4: Summary table of DA results on VisDA datasets. (*) denotes the reproduced methods.

  ⊗n and a kernel h ∈ {W p , W p p , W ε , S ε , GW p } with a ground cost C as in 8.2. Let the reweighting function w and the probability law over m-tuple P be as inExamples 1, 2,[START_REF]6 Samples generated by BigGAN model at 512×512 resolution[END_REF][START_REF]1 Example of adversarial example for a GoogLeNet[END_REF] Let the sequences of probability vectors (a (n) ) ∈ Σ and (b (n) ) ∈ Σ satisfy Loc A(m, γ, D)

	and let D > 0 and γ ∈ ( 3 4 , 1]. We have a deviation bound for the sampling without replacement between h m,k w,P (a

3.1 : Theorem 8.3.1 (Maximal deviation bound for compactly supported distributions). Let δ ∈ (0, 1), k 1 an integer and m 1 be a fixed integer. Consider two compactly supported distributions α, β, two n-tuples of empirical data X ∼ α ⊗n , Y ∼ β

  37)Thanks to Lemma A.1.3 and 15 we get the desired results.Corollary.It is also possible to have a bound on the expectation over the batch couples and the data.

	Corollary 8.31. With the same hypothesis and notations as in Theorem 8.3.1. The following inequality
	holds:

Proofs of Chapter 8 169

3.3 (Maximal deviation bound for bounded cost). Let δ ∈ (0, 1), k 1 an integer and m 1 be a fixed integer. Consider two distributions α, β possibly unbounded, two n-tuples of empirical data X ∼ α ⊗n , Y ∼ β ⊗n and a kernel h ∈ {W p , W p p , L ε , S ε , GW } with a ground cost C as in 8.2. Let A.1.

  3 (U-statistics concentration bound). Let δ ∈ (0, 1), three integers k ≥ 1 and m ≤ n be fixed, and two compactly supported distributions α, β. Consider two n-tuples X ∼ α ⊗n and Y ∼ β ⊗n and a kernel h ∈ {OT τ,ε φ , S τ,ε φ } with the squared Euclidean distance as ground cost. We have a concentration bound between h m (X, Y ) and the expectation over minibatches E h depending on the number of empirical As α, β are compactly supported by hypothesis, the UOT loss is bounded thanks to Lemma 9.2.1.a. Thus, we can apply Theorem 7.3.2 to our U-statistic and get the claimed bound.

	data n:
	|h	u,u	log(2/δ) 2 n/m	,	(A.110)

m (X, Y ) -E h | ≤ M h with probability at least 1 -δ and where M h u,u is an upper bound defined in Lemma 9.2.1.a. Proof. h m (X, Y ) is a two-sample U-statistic of order 2m and E h is its expectation as X and Y are i.i.d. random variables. h m (X, Y ) is a sum of dependant variables and it is possible to rewrite h m (X, Y ) as a sum of independent random variables.

Deep learning and optimal transport : learning from one anotherKilian Fatras 2021 Deep learning and optimal transport : learning from one anotherKilian Fatras 2021

* https://github.com/PythonOT/ARWGAN

† following their online implementation : https://github.com/ermongroup/generative_adversary

* https://github.com/kilianFatras/JUMBOT

Remerciements

t = 0 t = 1/6 t = 2/6 t = 3/6 t = 4/6 t = 5/6 t = 1 t = 0 (r) t = 1/6 (r) t = 2/6 (r) t = 3/6 (r) t = 4/6 (r) t = 5/6 (r) t = 1 (r) t = 0 t = 1/6 t = 2/6 t = 3/6 t = 4/6 t = 5/6 t = 1 t = 0 (r) t = 1/6 (r) t = 2/6 (r) t = 3/6 (r) t = 4/6 (r) t = 5/6 (r) t = 1 (r) t = 0 t = 1/6 t = 2/6 t = 3/6 t = 4/6 t = 5/6 t = 1 t = 0 (r) t = 1/6 (r) t = 2/6 (r) t = 3/6 (r) t = 4/6 (r) t = 5/6 (r) t = 1 (r) layers 1-2-3. We trained the DFC-VAE with a batch size of 64 for 5 epochs over the training dataset and use Adam method for optimization with initial learning rate of 0.0005, see [START_REF] Hou | [END_REF]] for more details.

With the feature extraction, we are able to improve the quality of final distribution's images.

The minibatch OT loss produces blurred images at the end of the flow as shown in Figure 8.8, especially at the back of the image where all details are lost. This is due to the fact that the minibatch OT shrinks the distribution. On the contrary, the debiased minibatch OT reported in Figure 8.9 produced images with high background details, quality and coherence with respect to the original background. Moreover, the evolution seems more natural between the source and the target distributions.

Mapping estimation between human faces with minibatch Optimal Transport

While the previous application focused on updating samples, the second application is a continuous mapping estimation between source and target distributions that will allow transforming new samples that are not in the original training data. The map is parametrized by a neural network f ϕ : X → R p between the source data and the target data. The objective is to minimize the loss:

w,P,C(fϕ(X),Y ) (u, u), (8.45)

Numerical experiments on generative modelling, color transfer and minibatch

Gromov-Wasserstein 127 t = 0 t = 1/6 t = 2/6 t = 3/6 t = 4/6 t = 5/6 t = 1 t = 0 (r) t = 1/6 (r) t = 2/6 (r) t = 3/6 (r) t = 4/6 (r) t = 5/6 (r) t = 1 (r) t = 0 t = 1/6 t = 2/6 t = 3/6 t = 4/6 t = 5/6 t = 1 t = 0 (r) t = 1/6 (r) t = 2/6 (r) t = 3/6 (r) t = 4/6 (r) t = 5/6 (r) t = 1 (r) t = 0 t = 1/6 t = 2/6 t = 3/6 t = 4/6 t = 5/6 t = 1 t = 0 (r) t = 1/6 (r) t = 2/6 (r) t = 3/6 (r) t = 4/6 (r) t = 5/6 (r) t = 1 (r) Where

We apply this problem on the celebA dataset [Liu 2015]. We considered 5000 male and 5000 female images. The image size is 64 × 64. The goal is to learn how to transform a male image into a female one. Unfortunately, in order to avoid blurry images, we once again relied on the latent space, of dimension 100, of a pre-trained DFC-VAE [START_REF] Hou | [END_REF]]. We used the same setting as described in the Gradient Flow section. We performed the training in the latent space and then we decoded the transformed samples. We consider a 4 dense layer neural network with ReLu activation function (100 → 1024 → 1024 → 512 → 100). The minibatch size m is set to 128, and we used the Adam optimizer [Kingma 2015] with a step size of 1e -4 and the coefficients β 1 = 0 and β 2 = 0.9.

We conducted the experiments for minibatch Wasserstein loss and for the debiased loss Λ h . We spotted once again that the transformed samples with the minibatch Wasserstein losses are blurred (Figure 8.10). However, the results with the unbiased minibatch Wasserstein loss are more diverse and more realistic. It shows the effectiveness of the unbiased loss to debiased the minibatch Wasserstein losses (Figure 8.10). It is interesting to note that the estimated mapping are quite different on some images between losses which use a sampling with or without replacement.

Statistical and optimization properties

For sake of simplicity and due to the experimental setting where we have empirical measures, we consider the uniform formalism of minibatch OT developed in Section 8.1.1. In this section, we show that the deviation bounds on the transport plan and the Unbalanced minibatch OT loss remain true, despite the fact that the mass of the transport plans are possibly unbounded. We demonstrate that under reasonable hypotheses, the UOT cost can be bounded and the associated set of optimal transport plans is compact.

Then we demonstrate that under the same hypotheses, we can exchange Clarke gradients and expectations to justify SGD's convergence.

Deviation bounds

To generalize the concentration bounds from Section 8.3.1, we first need to assure that our estimators are bounded. From now on, we suppose that the measures α and β are compactly supported. Our first lemma intends to show that the UOT cost is finite and that the optimal transport plan is bounded.

Lemma 9.2.1 (Bounded UOT and optimal transport plan). Let C be a ground cost and a, b two positive

φ and assume ε > 0 or φ ∞ > 0. Then h(a, b, C) is finite and the set of optimal transport plan is a compact set.

It is straightforward to prove boundedness of h = S τ,ε φ from Lemma 9.2.1. We can now turn to establish deviation bounds for both incomplete estimators h m,k and Π h,m,k . Theorem 9.2.1 (Maximal deviation bound). Let δ ∈ (0, 1), three integers k ≥ 1 and m ≤ n be fixed. Consider two n-tuples X ∼ α ⊗n and Y ∼ β ⊗n and a kernel h ∈ {OT τ,ε φ , S In this appendix, we gather the different proofs of the results we claimed in the previous chapter. In Appendix A.1 we gather the proofs of the different results from Chapter 8. In Appendix A.2 we gather the proofs of results from Chapter 9.

A.1 Proofs of Chapter 8

Outline. This Appendix is organized as follows:

• Appendix A.1.1 gives the proofs of our general minibatch OT distances formalism. In particular, it proves under what conditions the minibatch OT matrix is a minibatch OT plan.

• Appendix A.1.2 provides the proofs of concentration bounds for compactly supported distributions. We generalize the U-statistic proof to know under what conditions our estimator is close to its mean.

• Appendix A.1.3 provides the proofs of concentration bounds for sub-Gaussian distributions. Based on the compactly supported case, we use a truncation argument to provide a more general concentration bound for unbounded distributions.

• Appendix A.1.4 gives the proofs of concentration bounds for the minibatch OT plan. We provide a concentration bound of our incomplete minibatch OT plan around the input marginals.

A.1. Proofs of Chapter 8 159

Proof. We start with the case h = W ε for ε ≥ 0. Note that with our choice of cost matrix C = (C i,j ) 1≤i,j≤n one has 0 C i,j 2 max 1 i,j n x i -y j p 2 . Denote the optimal transport plan between I and J as Π = (Π i,j ) (with respect to the cost matrix C I,J ), consider the transport plan w 1 (a, I) ⊗ w 2 (b, J), we directly have:

As the second term is equal to zero in first inequality's right hand side expression. The extension to h = S ε is direct as it is a weighted sum of three terms of the form W ε . Lastly, a similar argument gives the desired bound for the Gromov-Wasserstein distance.

m-tuples I, J, one can write:

Finally, in the case of data lying in a compact, the quantity 2 max

In the next section, we use the upper bound on optimal transport to show a deviation bound between the empirical estimators of MBOT and their mean value in the case of compactly supported measures.

A.1.2 Concentration theorem (compactly supported measures)

In this section, we consider two compactly supported probability distributions α, β. In what follows, we are interested in concentration bounds with m ∈ N * fixed. For n ∈ N * and n ≥ m, we denote by u the element of Σ n such that

We will also often omit the dependence of various quantities (the minibatch procedure h, the reweighting function w etc.) in the asymptotic parameter n. The purpose of this appendix is to prove Theorem 8.3.1. The appendix is structured as follows. In a first section, we prove the deviation between the complete estimator h and its mean. Afterwards, we provide the deviation between the complete estimator h and its incomplete counter part h in a second section. The third section gathers all previous propositions and lemmas to prove Theorem 8.3.1 and corollaries.

Deviation between the complete estimator h and its mean.

We focus on the first ingredient of our proof: the deviation between the complete estimator h and its mean. This proof is based on the U-statistics concentration inequality proof but needs to be adapted due to the non-uniform probability vectors a and b.

From now on, the probability vectors (a (n) ) and (b (n) ) are sequences which depend on the number of data n. More precisely (a (n) ) n∈N and (b (n) ) n∈N are sequences of vectors of size n such that for each n ∈ N, a (n) , b (n) ∈ Σ n , we denote the space of these sequences as (a (n) ), (b (n) ) ∈ Σ. The sequence of probability vectors (a (n) ) and (b (n) ) can not be taken arbitrarily if we want to guarantee convergence. Hence we rely on local constraints that we defined in Chapter 8. We recall them: Definition 25 (Local averages conditions). Let (a (n) ) ∈ Σ and two integers n, m ∈ N * such as n ≥ m. (i) We say that (a (n) ) satisfies the local arithmetic mean condition if there exists a constant D > 0 and γ > 0 such that for any n ∈ N and I ∈ n m we have

We write that (a

) is said to verify the local geometric mean condition if there exists a constant D > 0 and γ > 0 such that for any n ∈ N * and I ∈ n m we have

We write that (a

We record the following properties of the Loc A and Loc G conditions.

Lemma 8.3.1. Let m ∈ N * , γ > 0 and D > 0. Let (a (n) ) ∈ Σ be a sequence of probability vectors. The following statements hold:

Proof. We first prove (i). Let (a

Hence, we find by summing (A.12) for 1 ≤ ≤ n

Lastly, (ii) follows from the arithmetic-geometric mean inequality.

We show that in order to obtain concentration properties of the estimators h m w,P (a (n) , b (n) ) we need to ensure that the sequences (a (n) ) and (b (n) ) verify the local condition with enough decay, e.g. (a

Hereafter, we denote by τ an absolute (and possibly large) constant. We also writes τ = τ (γ) to denote constants which depend on some parameter γ.

Proposition 15 (Generalized U-statistics concentration bound). Let δ ∈ (0, 1) and m ≥ 1 be a fixed integer. Consider two compactly supported distributions α, β, two n-tuples of empirical data X ∼ α ⊗n , Y ∼ β ⊗n and a kernel h ∈ {W p , W ε , S ε , GW p } with a ground cost C as in 8.2. Let the reweighting function w and the probability law P over m-tuple be as in Examples 1, 2, 3, 4. Let (a (n) ), (b

, 1] and D > 0. We have the following concentration bound for the sampling without replacement We have a deviation bound for the sampling without replacement between h m,k w,P (a (n) , b (n) ) and Eh m w,P (a (n) , b (n) ) depending on the number of empirical data n and the number of batches k:

where M is an upper bound on the ground cost. And for the sampling with replacement, let the sequences of probability vectors (a (n) ), (b (n) ) verify Loc G (m, γ, D) for some γ ∈ (1 -1 4m , 1] and D > 0, we have:

Proof. The proof is the same as for Theorem 8.3.1. The only difference is that we do not need to have compactly supported data to bound the OT kernel in (A.22), as the OT kernel is bounded by assumption on the ground cost.

A.1.3 Concentration theorem (sub-Gaussian)

In this section we relax the assumption of bounded data and give a proof for Theorem 8.3.2. The proof for the sub-Gaussian case rely on a truncation argument between data that lie in some compact data which do not. We start by recalling the sub-Gaussian data definition:

Definition 28 (sub-Gaussian random vectors). A random vector x ∈ R d is sub-Gaussian, if there exists σ ∈ R so that:

We write the class of sub-Gaussian random vectors as SG(σ). For the sake of simplicity, we consider the following class of random vectors: Definition 29 (Norm sub-Gaussian data [Jin 2019]). Let x ∈ R d be a random vector and ρ x = Ex. We say that x ∈ normSG(ρ x , σ 2 ) for some σ > 0 if the following inequality holds

Norm sub-Gaussian random vectors are a generalization of both sub-Gaussian random vectors and norm bounded random vectors. They show tighter concentration bounds than sub-Gaussian random vectors. We also have the following inclusion: SG(σ/ √ d) ⊂ normSG(σ) ⊂ SG(σ), see [Jin 2019] for a detailed review of the connections between these two random vector classes.

Theorem 8.3.2 (Concentration inequality sub-Gaussian data). Let the cost C = C n,p be defined as in (8.2). Let (x i ) 1≤i≤n and (y i ) 1≤i≤n be two i.i.d. sequences of random vectors such that Consider m 1 be a fixed integer and a kernel h ∈ {W p , W ε , S ε }. Let the reweighting function w W and the probability law over m-tuple P W be as in Examples 2 and 4. Then we have the following concentration bound for the sampling without replacement:

Proof. Let us fix ε > 0 to be chosen later. We use the following notations for δ > 0

We estimate,

First, by the union bound, we have

Next, we claim that

assuming the following conditions together with (A.43),

Let us show how (A.49) comes from a slight modification of the (proof) of (A.13). Using the same notations as in the proof of (A.13) we have (in place of (A.20)):

and

We emphasize that for each 1 ≤ k ≤ r -1, T k depends only on the set

Moreover, the variables (T k ) k are mutually independent. As in the proof of Proposition 15 is suffices to estimate P (∆1 A δ ≥ t) up to a factor 2. We have,

Setting δ = δ n such that Ξ 1 = Ξ 2 we find that δ n satisfies the equation given by:

4 p+2 mD 4 ε 2 , (A.69) which implies

We are interested in getting an upper bound of our quantity. Let us assume that we have:

If (A.71) holds then we have

and hence

showing that (A.70) is essentially sharp. We now investigate under which condition (A.71) holds. From (A.70) the condition (A.71) holds if we have

It suffices to have,

We thus choose the parameter e as follows

n 4(γ-3 4 ) .

(A.74)

Note that the condition (A.74) implies that (A.50) is verified for n large enough (i.e. under (A.43)). If (A.74) holds, then the condition (A.73) is verified and hence (A.72) holds. We now compute,

In the second inequality, we used the fact that ρ is upper bounded by 2 max +1 δ due to the hypotheses (A.51) and (A.43). In the last inequality we used that 1 ≤ t 1 + t 2 ≤ 2m. Hence, from (A.78) and (A.81) we estimate

for n as in (A.43). Combining (A.81) and (A.76) with (A.75) yields the desired result.

Now that we have bound the deviation between the complete estimator and its expectation, let us bound the deviation between the complete estimator and its incomplete counter part.

Remark 12. Theorem 8.3.2 holds when the distributions are compactly supported. Suppose we have probability sequences (a (n) ) and (b (n) ). Setting δ = n -1 2 p in (A.13) gives:

Hence we essentially lose a log(n) p 2 factor in comparison with (A.44).

We now discuss the difference between the deviation bounds of the estimator h and its mean in the bounded and unbounded data cases.

Remark 13. The proof of Lemma A.1.3 also yields

for sub-Gaussian data x 1 , y 1 , • • • , x n , y n and any δ > 0.

A.2. Proofs of Chapter 9 179

We optimize now the upper bound w.r.t. κ. Both KL terms are equal to φ(κ) = κ log κ -κ + 1, thus differentiating w.r.t. κ yields

Reusing this expression of κ in the upper bound yields Equation (A.100).

Lemma 9.1.1.b. Take (µ, ν) two probability measures with compact support, and z outside of ν's support. Define the Wasserstein distance between two probabilities as

measure perturbed by a Dirac outlier. Write (f, g) the optimal dual potentials of W(µ, ν), and y * a point in ν's support. One has

In particular, with the notation L it reads

Proof. We consider a suboptimal pair ( f , g) of potentials for L(μ, ν). On the support of (µ, ν) we take the optimal potentials pair (f, g) for (µ, ν), i.e. f = f and g = g. We need to extend f at z. To do so we take the c-transform of g, i.e.

where the infimum is attained at some y * since ν has compact support. the pair ( f , g) is suboptimal, thus

Hence the resulted given by Equation (A.102).

UOT properties

Now let us present results which will be useful for concentration bounds. A key element is to have a bounded plan and a finite UOT cost in order to derive a Hoeffding-type bound. We start this appendix by proving Lemma 9.2.1. We split it in two, Lemma 9.2.1.a proves that the UOT cost is finite and provides an upper bound while Lemma 9.2.1.b proves that the UOT plans exist and belong to a compact set. now prove that the set of optimal transport plan is convex , which will be useful for the optimization section.

Lemma A.2.1 (optimal transport plan convexity). Consider regularized or unregularized UOT with entropy φ and penalty D φ . The set of all optimal transport plans Opt(h, C, a, b) is a convex set.

Proof. It is a general property of convex analysis. Take a convex function f and two points (x, y) that both attain the minimum over a convex set E. Write z = tx + (1 -t)y for t ∈ [0, 1]. By convexity and suboptimality of z one has min

Thus z is also optimal, hence the set of minimizers is convex. The losses OT τ,ε φ fall under this setting.

Finally, we provide a final result about UOT cost which is also useful for the optimization properties.

Lemma A.2.2 (UOT is Lipschitz in the cost C). The map C → h(u, u, C) is locally Lipshitz. Furthermore, if the costs are uniformly bounded (0 ≤ C ≤ M ) then the loss is globally Lipschitz.

Proof. We recall that h(u, u, C) = min

Let C 1 and C 2 be two ground costs. Let Π 1 and Π 2 be the optimal solutions of h(u, u, C 1 ) and h(u, u, C 2 ), i.e., h(u, u, C 1 ) = F(Π 1 , C 1 ). Then we have:

Thus we have

Where the last inequality uses the Cauchy-Schwarz inequality. Following the same logic we get a bound for minus the left hand term

A.2.3 Statistical and optimization proofs

We consider a positive, symmetric, definite and C 1 ground cost and without loss of generality, we consider our ground cost to be the squared Euclidean distance. We recall our definitions and hypotheses. As the distributions α and β are compactly supported, there exists a constant M > 0 such that for any 1 i, j n, c(x i , y j ) M with M := diam(Supp(α) ∪ Supp(β)) 2 . We also furthermore suppose that the input masses m a and m b of positive vectors are strictly positive and finite, i.e., 0 < m a < ∞. These hypotheses assure us that the UOT cost is finite and that the UOT plan is bounded.

Hoeffding's inequality yields (A.115) which concludes the proof.

We are now ready to prove Theorem 9.2.1.a.

Theorem 9.2.1.a (Maximal deviation bound). Let δ ∈ (0, 1), three integers k ≥ 1 and m ≤ n be fixed and two compactly supported distributions α, β. Consider two n-tuples X ∼ α ⊗n and Y ∼ β ⊗n and a kernel h ∈ {OT τ,ε φ , S τ,ε φ } with the squared Euclidean distance as ground cost. We have a maximal deviation bound between h m,k (X, Y ) and the expectation over minibatches E h depending on the number of empirical data n and the number of batches k .116) with probability at least 1δ and where M h u,u is an upper bound defined in Lemma 9.2.1.a.

Proof. Thanks to Lemma A.2.4 and A.2.3 we get .118) with probability at least 1 -

We now give the details of the proof of Theorem 9.2.1.b. In what follows, we denote by Π (i) the i-th row of matrix Π. Let us denote by 1 ∈ R n the vector whose entries are all equal to 1. Theorem 9.2.1.b (Distance to marginals). Let δ ∈ (0, 1), two integers m ≤ n be fixed. Consider two n-tuples X ∼ α ⊗n and Y ∼ β ⊗n and the kernel h = OT τ,ε φ with the squared Euclidean distance as ground cost. For all integer k ≥ 1, all 1 i n, with probability at least 1 -δ on the draw of X, Y and D k we have

where M ∞ Π denotes an upper bound of all minibatch UOT plan.

Proof. Let us consider the sequence of random variables ((b p (I, J) (I,J)∈Γ ) 1 p k such that b p (I, J) is equal to 1 if (I, J) has been selected at the p-th draw and 0 otherwise. By construction of Π h,m,k (X, Y ), the aforementioned sequence is an i.i.d sequence of random vectors and the b p (I, J) are bernoulli random variables of parameter 1/|Γ|. We then have:

, the random vectors ω p are independent, and thanks to Lemma 9.2.1.b, they are bounded by a constant M Π which is the maximum mass of all optimal minibatch unbalanced plan in Opt(h, C(X(I), Y (J)), u m , u m ). We denote the maximum upper bound M Π of all minibatch UOT plan as M ∞ Π . Moreover, one can observe that

which concludes the proof.

Note that the unbalanced Sinkhorn divergence S τ,ε φ involves three terms of the form OT τ,ε φ , hence three transport plans, which explains why we do not attempt to define an associated averaged minibatch transport matrix.

Proof of Theorem 9.2.2

To prove the exchange of gradients and expectations over minibatches we rely on Clarke differential. We need to use this non-smooth analysis tool as unregularized UOT is not differentiable. It is not differentiable because the set of optimal solutions might not be a singleton. Clarke differential are generalized gradients for locally Lipschitz function and non necessarily convex. A similar strategy was developed in Section A.1. The key element of this appendix is to rewrite the original UOT problem OT τ,ε φ as:

Where Opt(OT τ,ε φ , C, a, b) is a compact set of the set of measures M + (X ). The compact set is a key element for using Danskin like theorem (Proposition B.25 [Bertsekas 1997]).

We start by recalling a basic proposition for Clarke regular function:

Proposition 18. A C 1 or convex map is Clarke regular.

Proof. see Proposition 2.3.6 [Clarke 1990]

We first give a lemma which gives the Clarke regularity of the UOT cost with respect to a parametrized random vector. Lemma A.2.5. Let u be a uniform probability vector. Let X be a R dm -valued random variable, and {Y θ } a family of R dm -valued random variables defined on the same probability space, indexed by θ ∈ Θ, where Θ ⊂ R q is open. Assume that θ → Y θ is C 1 . Consider a C 1 cost C and let h ∈ {OT τ,ε φ , S τ,ε φ }. Then the function θ → -h(u, u, C(X, Y θ )) is Clarke regular. Furthermore, for h = OT τ,ε φ and for all 1 ≤ i ≤ q we have: ))] will follow from Theorem 2.7.2 and Remark 2.3.5 [Clarke 1990], once we establish that the expectation on the left hand side is finite. This is direct as we suppose we have compactly supported distributions and C is a C 1 cost. Indeed consider the function which is equal to M h u,u on the distributions's support and which is set to 0 everywhere else. Taking the expectation on this function is finite as M h u,u is finite.

List of Tables