Hadrien Thibault

Puisqu'il est ainsi permis de plumer plus librement, de mettre rigueur de côté, pour des remerciements : Merci à mes encadrant.e.s: Laetita, Nicolas, Romain; les officiel.le.s. Vous m'avez permis de jouer aux mathématiques et à l'ordinateur avec une rare liberté, je pense. C'était excitant, gratifiant, inspirant et je garderai de votre confiance un souvenir certain. Merci Rémi, l'officieux mais tout autant présent. La seule chose égalant de fréquence tes enthousiasmes sont tes conjectures, souvent justes, je crois. Je m'excuse par retard pour mes nombreuses exothermies avortées, qui ont parsemé ces trois années. Au-delà de tout ce qui a permis ce manuscrit, merci aux moments d'ailleurs et de pas loin, aux discussions, aux rires. Cela me manquera. Merci aussi à Nicolas Klutchnikoff, tu m'as sauvé la mise dans des situations plutôt importantes; nos discussions de 100 ans avec Romain devant un tableau à Villejean resteront. Merci Ievgen, j'espère que l'on pourra continuer à jouer au Transport Optimal en dessous de la Loire. Merci à Obelix, ce fut un plaisir de passer ces trois années à vos côtés.

Je remercie également les rapporteurs, Gabriel Peyré et Marc Sebban, ainsi que les membres du jury, Julie Delon, Filippo Santambrogio, Pierre Vandergheynst, chercheur.e.s dont j'admire les travaux. Merci d'avoir pris le temps de vous intéresser aux miens.

Thank you Caglayan, we followed each other through space and time, and I am so happy that you were on this trip. I am confident that we will share a şerefe in Turkey one day. Until then I believe we can still have some Yec'hed mat in Rennes.

Merci Mathilde, Maëlle,

Résumé (Français) Introduction

"Ainsi, l'on voit dans les Sciences, tantôt des théories brillantes, mais longtemps inutiles, devenir tout à coup le fondement des applications les plus importantes, et tantôt des applications très simples en apparence, faire naître l'idée de théories abstraites dont on n'avait pas encore le besoin, diriger vers les théories des travaux des Géomètres, et leur ouvrir une carrière nouvelle." C'est ainsi que Nicolas De Condorcet [De Condorcet 1781] au 18ème siècle a introduit les travaux de Gaspard Monge [START_REF] Gaspard Monge | Mémoire sur la théorie des déblais et des remblais[END_REF]] qui sont au coeur de la théorie du transport optimal. Comment déplacer des masses d'un endroit à un autre de sorte à minimiser l'effort global de déplacement ? Condorcet avait raison : cette "idée simple" a maturé au fil des ans en une élégante théorie au croisement des mathématiques et de l'optimisation et est aujourd'hui au centre de nombreuses applications en machine learning.

D'une manière générale, l'intérêt du transport optimal réside à la fois dans sa capacité à fournir des relations, des correspondances, entre des ensembles de points et dans le fait qu'il induise une notion géométrique de distance entre des distributions de probabilité (voir Figure 1). Ces deux propriétés se sont révélées très utiles pour un large éventail de tâches qui sont, pour n'en citer que quelquesunes, le recalage d'images [START_REF] Haker | [END_REF]], la recherche d'image par contenu [Rubner 1998], l'adaptation de domaine [Courty 2017], le traitement du signal [START_REF] Kolouri | [END_REF]], l'apprentissage non supervisé [Arjovsky 2017[START_REF] Genevay | [END_REF]], l'apprentissage supervisé et semi-supervisé [Frogner 2015, Solomon 2014], le traitement automatique du langage [START_REF] Kusner | [END_REF]], l'équité en machine learning [Gordaliza 2019], ou encore en biologie [START_REF] Schiebinger | [END_REF]] ou en astrophysique [START_REF] Frisch | [END_REF]].

Malgré ses nombreuses propriétés, le problème de transport optimal reste difficile à résoudre en pratique et il est connu pour souffrir de problèmes de scalabilité qui empêchent son utilisation sur des données volumineuses, omniprésentes en machine learning. L'émergence du transport optimal a été grandement favorisée par de récentes avancées dans le domaine de l'optimisation [Cuturi 2013, Altschuler 2017[START_REF] Genevay | [END_REF]].

De plus, dans sa formulation originelle, le transport optimal reste assez limité aux applications où il existe un "moyen direct" pour comparer les points, appelés samples, provenant des distributions. Il est donc souvent limité aux cas où les samples font partie du même espace métrique, qui est la plupart du temps un espace euclidien. Cette limitation empêche notamment l'utilisation du transport optimal pour une variété de tâches dans lesquelles il existe une information de structure supplémentaire sur les données, qui ne peut généralement pas être décrite par des espaces euclidiens. On peut citer par exemple le cas où les samples sont décrits par des graphes, des arbres ou des séries temporelles. Cette limitation empêche également son utilisation lorsque les samples se trouvent dans des espaces métriques différents, potentiellement non liés, ou lorsqu'une notion de distance entre les samples ne peut pas être facilement définie. Tous ces cas font partie de ce qu'on appellera par la suite le cas de figure incomparable. Une solution intéressante se trouve dans l'élégante théorie de la distance de Gromov-Wasserstein [Memoli 2011] qui ne nécessite pas la comparaison des échantillons entre les distributions. Cependant, cette distance est connue pour être encore plus difficile à résoudre que le transport optimal classique. L'objectif de cette thèse est d'aider à surmonter ces différents obstacles en:

(i) Définissant de nouveaux problèmes de transport optimal sur des espaces incomparables et en particulier pour des données structurées.

(ii) Réduisant l'écart entre la compréhension théorique du transport optimal classique et celle du problème de Gromov-Wasserstein.

Données structurées et espaces incomparables en machine learning

Avant d'entrer dans le détail des contributions de cette thèse il est important de rendre explicite la notion de structure et d'espaces incomparables. Une première approche est de considérer une information structurelle comme étant l'élément d'information qui encode les relations spécifiques qui existent entre les composants d'un objet. Cette définition peut être mise en relation avec le concept de relationnal reasoning [START_REF] Battaglia | [END_REF]] dans lequel des entités (ou des éléments ayant des attributs tels que l'intensité d'un signal) coexistent avec certaines relations ou propriétés entre eux. Des tels cas de données structurées apparaissent naturellement lorsque la structure est explicite. Par exemple, dans le contexte des graphes, les arêtes sont représentatives de la structure, de sorte que chaque attribut du graphe (généralement un vecteur de R d ) peut être lié à d'autres par les arêtes entre les noeuds. Ces objets apparaissent notamment pour modéliser des composés chimiques ou des molécules [START_REF] Kriege | [END_REF]], la connectivité du cerveau [Ktena 2017] ou encore les réseaux sociaux [START_REF] Yanardag | [END_REF]]. Cette famille générique de données structurées comprend également les arbres [START_REF] William | Optimal algorithms for comparing trees with labeled leaves[END_REF]] ou encore les séries temporelles dont les valeurs sont corrélées dans le temps, de sorte leur comparaison nécessite de prendre en compte dans la modélisation la structure temporelle, la direction du temps.

Les informations de structure des données en machine learning peuvent apparaître de manière plus subtile voire même implicitement. Elles peuvent se matérialiser lorsqu'on construit une prior ou un biais structurel sur la représentation des données. Par exemple, dans le contexte de l'apprentissage profond, de nombreuses architectures exploitent l'équivariance à une transformation pour améliorer la généralisation des modèles. Les réseaux de neurones convolutifs (CNN) sont un excellent exemple de cette prior structurelle: un CNN est équivariant aux translation, i.e. si nous translatons l'entrée du réseau de neurone, la sortie des convolutions sera également translatée. Ce biais structurel est connu pour révéler une certaine hiérarchie spatiale sur les pixels utile dans de nombreuses applications [START_REF] Wang | [END_REF],Chen 2018]. D'autres travaux ont étudié la conception de couches avec des équivariances à d'autres transformations telles que les permutations, rotations ou réflexions [START_REF] Kondor | [END_REF][START_REF] Cohen | [END_REF][START_REF] Gens | [END_REF].

Contrairement aux méthodes "end-to-end" telles que les réseaux de neurones, d'autres approches plus "hand-engineering", basées sur la segmentation des images, permettent de dévoiler une certaine structure utile sur les images [START_REF] Bach | [END_REF][START_REF] Shi | [END_REF]. Les structures implicites sont également au coeur de nombreux outils de traitement du langage naturel (NLP) utilisés pour trouver de bonnes représentations vectorielles des mots [Mikolov 2013b, Mikolov 2013a[START_REF] Pennington | [END_REF], pour la reconnaissance vocale [Hinton 2012] ou plus généralement pour l'apprentissage de séquences [Sutskever 2014]. Dans ces cas, les biais structurels passent soit par l'utilisation de variables latentes ou celle de probabilités conditionnelles. Lorsqu'ils sont disponibles, les labels ou les classes induisent également une structure implicite sur les features des données. Par exemple, en adaptation de domaines, on peut souhaiter que les samples du domaine source ayant le même label soient appariés de manière cohérente dans la même région de l'espace cible, et ainsi éviter qu'ils ne soient divisés en des emplacements trop éloignés [Courty 2017, Alvarez-Melis 2018b]. La tendance récente dans la communauté de machine learning des réseaux de neurones pour les graphes (GNN) [START_REF] Wu | [END_REF]] est l'un des nombreux exemples soulignant l'importance des données structurées de nos jours.

Alors que les exemples précédents considèrent les données structurées comme des entrées du processus d'apprentissage, la prédominance des données structurées en machine learning se manifeste également dans de nombreux travaux où les données structurées sont des sorties. On peut citer par exemple le domaine de la prédiction structurée dans lequel on veut apprendre à produire des objets structurés tels que des séquences, des arbres ou des assignements [Taskar 2005[START_REF] Lafferty | [END_REF], Collins 2002, Blondel 2020[START_REF] Mensch | [END_REF], Korba 2018].

En bref, la notion de structure en machine learning est omniprésente et apparaît aussi souvent qu'il y a une information supplémentaire sur les objets qui va au-delà de leurs représentations caractéristiques, de leurs features. Comme le montrent de nombreux contextes de machine learning tels que les modèles graphiques [Pearl 1986[START_REF] Pearl | Judea Pearl. Causality: Models, reasoning and inference[END_REF]], l'apprentissage par renforcement [Dvzeroski 2001] ou les modèles bayésiens non-paramétriques [Hjort 2010], considérer les objets comme une composition complexe d'entités avec certaines interactions est particulièrement utile, afin d'apprendre à partir de petites quantités de données.

La notion précédente de données structurées peut se voir comme étant un cas particulier de données définies sur des espaces incomparables. Dans cette situation, chaque sample possède une "caractéristique" qui lui est propre et qui peut ne pas être partagée avec les autres samples. Par exemple, lorsque l'on considère un ensemble de données composé de plusieurs graphes, la structure d'un graphe n'est généralement pas partagée avec les autres graphes. Cette notion, volontairement large, englobe également le cas où les données proviennent de sources hétérogènes. Un exemple particulier de ce problème est l'adaptation de domaines hétérogènes [START_REF] Yeh | [END_REF][START_REF] Zhou | [END_REF][START_REF] Wang | [END_REF]] qui vise à exploiter les connaissances provenant de domaines sources hétérogènes pour améliorer les performances d'apprentissage dans un domaine cible avec, potentiellement, différents espaces pour les features entre les domaines source et cible. Le cas des datasets MNIST/USPS [LeCun 2010, Hull 1994] illustre assez bien cette situation : sur la base de la connaissance d'images de chiffre de taille 28 × 28 (i.e. des vecteurs de R 784 ) de MNIST, comment construire e.g. un classifieur qui fonctionne sur des images de chiffres 16 × 16 (i.e. des vecteurs de R 256 ) de USPS ? Il va sans dire que ce problème se pose souvent dans tous les domaines du machine learning : il est courant que les données soient recueillies à partir de sources diverses et hétérogènes et les méthodes qui s'appuient sur cette diversité présentent souvent un grand intérêt.

Le transport optimal pour le machine learning La question centrale qui se pose souvent en machine learning est la suivante : comment représenter les données et comment les comparer ? Le cadre des distributions de probabilités apporte certaines réponses à cette question en associant une mesure de probabilité à une collection de samples qui forment un dataset (x i ) i∈ [[n]] . La représentation Lagragienne du dataset résulte en une mesure de probabilité discrète n i=1 a i δ xi dans laquelle on associe à chaque point x i un dirac δ xi (x) = 1 si x = x i sinon 0 ainsi qu'un poids a i ≥ 0 tel que (a i ) i∈ [[n]] est un vecteur de probabilité qui satisfait n i=1 a i = 1. Lorsqu'aucune information concernant l'importance relative des samples dans le dataset n'est disponible les poids peuvent être choisis uniformes de sorte que a i = 1 n . De même une représentation Eulerienne peut être construite via la distribution de probabilité N i=1 a i δ xi dans laquelle (x i ) i∈ [[N ]] est une grille régulière qui discrétise l'espace. Cette formulation est équivalente à construire un histogramme sur nos données (voir Figure 2). Ces points de vue sur les données préconisent de trouver un moyen approprié de comparer leur représentation sous forme de distributions de probabilité et, à ce titre, la question de trouver des outils pour les comparer est au coeur de nombreux algorithmes de machine learning.

Bien qu'il existe diverses divergences telles que φ-divergences [Csiszar 1975] ou Maximum Mean Discrepancies (MMD) [START_REF] Gretton | [END_REF]], la richesse du transport optimal réside dans sa capacité à intégrer la géométrie de l'espace sous-jacent dans sa formulation et à prêter attention aux relations, aux correspondances des échantillons au sein de leurs représentations respectives. Pour souligner en bref l'avantage de représenter les données par des histogrammes/distributions de probabilités couplé à l'utilisation du transport optimal, nous pouvons citer par exemple [START_REF] Rubner | [END_REF]] pour la recherche d'images par contenu ou [START_REF] Kusner | [END_REF]] pour le traitement du langage naturel. À ce stade, des questions naturelles se posent : cette représentation théorique des données est-elle utilisable lorsque leur nature est intrinsèquement structurée, ou lorsque les samples se trouvent dans des espaces incomparables ? Dans ce cas, comment pouvons-nous représenter les données sous forme de distributions de probabilités ? Dans quelle mesure cette représentation est-elle valable ? Le transport optimal est-il toujours applicable et, si ce n'est pas le cas, comment comparer ces distributions de probabilités ? L'objectif de cette thèse, entre autres, est de fournir des réponses à ces questions.

Contributions

Cette thèse couvre la majeure partie des travaux de l'auteur et se concentre sur un seul axe de recherche qui est Le transport optimal sur des espaces incomparables. Des travaux supplémentaires [Vayer 2020a] sur les séries temporelles sur des espaces incomparables, qui ne sont pas basés sur le transport optimal, ne sont pas inclus dans cette thèse. Le lecteur intéressé peut cependant trouver les détails dans la bibliographie.

Chapitre 2

Ce chapitre présente les résultats fondamentaux de la théorie classique du transport optimal et résume/illustre ses différentes formulations ainsi que quelques solveurs numériques connus. La philosophie de ce chapitre est de fournir un aperçu de haut niveau du transport optimal, tant en théorie qu'en pratique. Ce chapitre se conclut sur la théorie de Gromov-Wasserstein qui est au coeur de la thèse. Un lecteur familier avec les concepts de base du transport optimal peut passer cette partie bien qu'elle contienne des concepts et des notations essentiels qui seront abordés tout au long de la thèse.

Chapitre 3

Ce chapitre est consacré au transport optimal pour les données structurées, et notamment dans le contexte des graphes. Il est basé sur les travaux des articles [Vayer 2019a] et [Vayer 2020b] et fournit des réponses à la question de la définition d'un cadre mathématique pour le transport optimal dans le cas de données structurées. Nous fournissons un cadre général, basé sur la notion de Fused Gromov-Wasserstein qui définit une distance de transport optimal entre des objets structurés tels que des graphes labelés non dirigés.

En résumé, nous considérons les graphes labelés non orientés comme des tuples de la forme G = (V, E, f , s ) où (V, E) est l'ensemble des noeuds et des arêtes du graphe. f : V → Ω f est une fonction qui associe à chaque noeud v i ∈ V un feature a i def = f (v i ) dans un espace métrique (Ω f , d). Nous appelons information de feature l'ensemble de tous les features (a i ) i du graphe. De même, s : V → Ω s associe à un noeud v i du graphe un point x i def = s (v i ) appartenant à un espace métrique (Ω s , C) spécifique à chaque graphe. C : Ω s × Ω s → R est une application symétrique qui vise à mesurer la similarité entre les noeuds du graphe. Cependant, contrairement à l'espace des features, Ω s est implicite et, en pratique, il suffit de connaître la mesure de similarité C. Avec un léger abus de notation, C est utilisé pour désigner à la fois la mesure de similarité de la structure et la matrice qui encode cette similarité entre les paires de noeuds du graphe (C(i, k) = C(x i , x k )) i,k . Selon le contexte, C peut soit encoder les informations de voisinage des noeuds, les informations des arrêtes du graphe, ou plus généralement, il peut modéliser une distance entre } } } Figure 3: (à gauche) Graphe labelé avec (ai)i l'information de feature, (xi)i l'information de structure et (hi)i un vecteur de poids qui mesure l'importance relative des noeuds. (à droite) Les données structurées sont entièrement décrites par une mesure de probabilité µ sur l'espace produit des features et de la structure, avec respectivement des marginales µX et µA sur la structure et les features.

les noeuds telle que la distance shortest-path. Nous désignons par information de structure l'ensemble de tous les points de structure (x i ) i du graphe.

Nous proposons d'enrichir le graphe décrit précédemment avec un vecteur de poids qui a pour but d'encoder l'importance relative des noeuds du graphe. Pour ce faire, si nous supposons que le graphe a n noeuds, nous associons aux noeuds des poids (h i ) i ∈ Σ n . Par cette procédure, nous obtenons la notion de données structurées comme un tuple S = (G, h G ) où G est un graphe labelé et h G est une fonction qui associe un poids à chaque noeud. Cette définition permet au graphe d'être représenté par une mesure de probabilité µ = n i=1 h i δ (xi,ai) sur l'espace produit feature/structure qui décrit l'ensemble de la donnée structurée (voir Figure 3). Considérons à présent deux données structurées µ = n i=1 h i δ (xi,ai) et ν = m i=1 g j δ (yj ,bj ) , où h ∈ Σ n et g ∈ Σ m sont des histogrammes. Nous notons M AB = (d(a i , b j )) i,j la matrice n × m de distance entre les features et C 1 , C 2 , les matrices de structure des graphes.

Nous définissons une nouvelle distance de transport optimal appelée la distance de Fused Gromov-Wasserstein. Elle est donnée pour un paramètre α ∈ [0, 1] par :

F GW (C 1 , C 2 , h, g) = min π∈Π(h,g) i,j,k,l (1 -α)d(a i , b j ) q + α|C 1 (i, k) -C 2 (j, l)| q π i,j π k,l (1)
Nous prouvons que cette fonction définit bien une distance entre les graphes labelés (Theorem 3.3.1) et qu'elle permet aussi de donner naissance à une notion de barycentre, de moyenne, de graphes basée sur la moyenne de Fréchet (Section 3.3.2). Nous établissons un algorithme pour calculer ces différents objets (Section 3.3.3) qui nous permet en particulier de résoudre aussi le problème de Gromov-Wasserstein classique, et nous prouvons qu'elle s'avère utile dans de nombreux scénarii de machine learning sur les graphes comme la classification, la simplification de graphes ou encore le clustering de graphes (Section 3.4). Nous concluons cette partie en élargissant la définition précédente au cas de données structurées continues (Section 3.5) où nous montrons que FGW possède des propriétés de distance similaires et définit de plus une géodésique sur l'espace des données structurées.

Chapitre 4

Ce Chapitre vise à combler l'écart entre la théorie classique du transport optimal et la théorie de Gromov-Wasserstein. Il s'ouvre sur le cas particulier des distributions 1D dont l'étude est basée sur les travaux de l'article [Vayer 2019b]. Nous établissons la première closed-form de Gromov-Wasserstein dans le cas des mesures discrètes sur la droite réelle et du coût euclidien au carré. Nous prouvons que celle-ci peut être calculée rapidement avec une complexité en O(n log(n)). En particulier nous montrons qu'un couplage optimal pour Gromov-Wasserstein est trouvé en considérant soit le couplage diagonal ou le couplage anti-diagonal lorsque les points sont triés. Nous proposons, en utilisant cette closed-form, une nouvelle divergence appelée Sliced Gromov-Wasserstein, à la manière de Sliced Wasserstein. Nous établissons ses propriétés (Theorem 4.1.3) et l'utilisons dans des scénarii tels que la comparaison de meshes 3D et les réseaux de neurones génératifs.

Une deuxième partie plus prospective se concentre sur la théorie de Gromov-Wasserstein pour les espaces euclidiens en la liant à la théorie classique du transport optimal. Nous abordons notamment la question de la régularité des plans de transport de Gromov-Wasserstein. Nous donnons des conditions nécessaires sous lesquelles nous pouvons prouver que le couplage optimal de Gromov-Wasserstein est supporté par une fonction déterministe, de la même manière que le couplage optimal pour le cas de Wasserstein avec un coût quadratique entre des mesures régulières (théorème de Brenier [Brenier 1991]). Pour cela nous considérons les cas où les mesures de distance ou similarité dans chaque espace c X , c Y sont définies par les produits scalaires ou par des coûts quadratiques. Nous montrons que résoudre GW équivaut à résoudre conjointement un problème de transport linéaire et un problème d'alignement. Ainsi, la régularité des plans optimaux de GW peut être étudiée au travers de formulations équivalentes, plus simples à analyser. Cela nous permet également de construire des solutions algorithmiques pour GW sur des espaces euclidiens. En résumé :

(i) Dans la Section 4.2.2 nous considérons le cas où c X , c Y sont définis par des produits scalaires dans chaque espace. A condition que la mesure de probabilité source soit régulière par rapport à la mesure de Lebesgue, nous donnons une condition suffisante pour l'existence d'un plan de transport optimal déterministe, i.e. supporté par une fonction déterministe T . Nous montrons que cette fonction est de la forme ∇u • P où u est une fonction convexe et P est une application linéaire qui peut être considérée comme une transformation globale "recalant" les mesures de probabilité dans le même espace (Theorem 4.2.1). Nous utilisons cette formulation pour montrer que la distance GW entre les mesures de probabilité 1D admet une solution closed-form. Plus précisément, nous montrons que le couplage optimal est déterminé par les fonctions de distribution cumulative et anticumulative de la distribution source (Theorem 4.2.4).

(ii) Dans la Section 4.2.3 nous considérons c X , c Y définis comme étant le carré des distances euclidiennes dans chaque espace. Nous montrons que le problème équivaut à une maximisation d'une fonction convexe sur Π(µ, ν). Nous utilisons la dualité Fenchel-Legendre dans l'espace des mesures pour en déduire un problème équivalent à celui de . Nous l'analysons plus en profondeur et montrons que la régularité des plans de transport optimaux est plus compliquée à établir que dans le cas précédent.

(iii) Dans la Section 4.2.4 nous utilisons les formulations précédentes pour obtenir des solutions numériques efficaces pour le problème GW basées sur des BCD. Nous montrons que ces algorithmes se comparent favorablement par rapport aux solveurs standards tels que le gradient conditionnel ou avec la régularisation entropique.

MNIST USPS USPS samples MNIST samples

s matrix between samples USPS colored coded pixels MNIST pixels through v MNIST pixels through entropic v (iv) Nous concluons par la Section 4.2.5 en considérant le problème Gromov-Monge dans les espaces euclidiens, qui est l'équivalent du problème de Monge du transport linéaire dans le contexte de Gromov-Wasserstein. Nous discutons du cas particulier de Gromov-Monge entre les mesures gaussiennes et nous montrons que ce problème admet une closed-form quand on se limite aux push-forward linéaires (Theorem 4.2.6). Nous donnons des interprétations géométriques de ce résultat et nous comparons le push-forward optimal avec la celui de la théorie du transport optimal classique dans le cas des mesures gaussiennes.

Chapitre 5

Ce chapitre présente un nouveau cadre théorique pour comparer des mesures de probabilité sur des espaces incomparables, à savoir le problème de co-transport optimal. Contrairement à l'approche de Gromov-Wasserstein, cette approche permet d'optimiser simultanément deux couplages entre les samples et les features des données. Ce chapitre fournit une analyse théorique approfondie de ce cadre et, du point de vue des applications, il aborde le problème de l'adaptation des domaines hétérogènes et du co-clustering. Ce chapitre est basé sur l'article [START_REF] Redko | [END_REF]]. Plus précisément nous considérons deux datasets quelconques X = [x 1 , . . . , x n ] T ∈ R n×d et X = [x 1 , . . . , x n ] T ∈ R n ×d , avec en général n = n et d = d . Les lignes sont appelées samples et les colonnes features. Nous associons aux samples (x i ) i∈ [[n]] et (x i ) i∈ [[n ]] des poids w = [w 1 , . . . , w n ] ∈ Σ n et w = [w 1 , . . . , w n ] ∈ Σ n . De la même manière on associe aux features des poids v ∈ Σ d et v ∈ Σ d . Le problème de Co-transport optimal est défini par: COOT (X, X , w, w , v, v ) = min

π s ∈Π(w,w ) π v ∈Π(v,v ) i,j,k,l L(X i,k , X j,l )π s i,j π v k,l (2)
Pour illustrer cette définition, nous résolvons le problème d'optimisation entre deux datasets classiques : MNIST et USPS. Ils contiennent des images de différentes résolution (les images de USPS sont de taille 16 × 16 et MNIST 28 × 28) qui appartiennent aux mêmes classes (chiffres entre 0 et 9). En outre, les chiffres sont également différemment centrés comme l'illustrent les exemples de la partie gauche de la Figure 4. Cela signifie que sans prétraitement, les images ne se trouvent pas dans le même espace topologique et donc ne peuvent pas être comparées directement à l'aide des distances conventionnelles. Les images représentent les samples tandis que chaque pixel agit comme un feature conduisant à 256 et 784 features i pour USPS et MNIST Le résultat de la résolution du problème est reporté sur la Figure 4. Dans la partie centrale gauche, nous fournissons le couplage optimal π s entre les samples, i.e. les différentes images, triés par classe. Le couplage π v , à son tour, décrit les relations entre les features, i.e. les pixels, dans les deux domaines. Pour le visualiser, nous codons en couleur les pixels de l'image USPS source et utilisons π v pour transporter les couleurs sur une image MNIST cible de sorte que ses pixels soient définis comme des combinaisons convexes de couleurs de la première avec des coefficients donnés par π v . Les résultats correspondants sont présentés dans la partie droite de la Figure 4.

Nous montrons que cette nouvelle formulation inclut Gromov-Wasserstein comme cas particulier (Proposition 5.5.2), et qu'elle présente l'avantage de travailler directement sur les données brutes sans avoir à calculer, stocker et choisir les mesures de similarité. De plus, COOT fournit deux couplages interprétables entre les features et les samples. Nous montrons que COOT définit une notion de distance entre les datasets (Proposition 5.3.1) et nous établissons une procédure d'optimisation basée sur la résolution de transports linéaires (Section 5.4). Sur le plan pratique, nous apportons la preuve de l'utilité de COOT pour le machine learning notamment en adaptation de domaines hétérogènes (Section 5.6.1) et pour le co-clustering (Section 5.6.2).

vi

Contents

Linear algebra a, A

all vectors in R d and matrices are written in bold. The coordinates will be written A i,j or A(i, j) for matrices and a i for vectors without bold. [[n]] the subset {1, • • • , n} of N I n , J n the n × n identity matrix and anti-identity matrix . , , a norm (depends on the context) and an inner product p Denotes the standard . p norm ⊗ K is the Kronecker product of matrices, i.e. for two matrices A ∈ R n×m , B p×q , A ⊗ K B ∈ R np×mq defined by (A i,j B) i,j ⊗ the the tensor-matrix multiplication, i.e. for a tensor L = (L i,j,k,l ) and a matrix A, L ⊗ A is the matrix k,l L i,j,k,l A k,l i,j

. tr, det is the trace operator for matrices i.e. tr(P) = i,j P i,j and the determinant operator

. F is the Frobenius norm for matrices i.e. P F = tr(P T P)

., . F is the inner product for matrices i.e. P, Q F = tr(Q T P) rank(A), ker(A)

The rank and the kernel of a matrix A DS is the set of doubly-stochastic matrices, i.e.

DS = {X ∈ R n×n |X1 n = 1 n , X T 1 n = 1 n , X ≥ 0} Π n
the set of permutation matrices, i.e. X = (x ij

) (i,j)∈[[n]] 2 ∈ Π n if x ij ∈ {0, 1} and n i=1 x ij = n j=1 x ij = 1. O(p)
the subset of R p×p of all orthogonal matrices.

V p (R q ) is the Stiefel manifold, i.e. the set of all orthonormal p-frames in R q or equivalently V p (R q ) = {B ∈ R q×p |B T B = I p }.

S n the set of all permutations of [[n]]. Element-wise division operator for two vectors, i.e. u v = ( ui vi ) i Σ n the set of probability vectors in R n + (or histograms with n bins), i.e.

Σ n = {a ∈ R n + | n i=1 a i = 1}
Measure theory P(X ) the set of probability measures on a space X M(X ) the set of Borel finite signed measures on a space X L the Lebesgue measure on R or R n depending on the context supp (µ) the support of µ (see definition 2.1.4) δ x the dirac measure on x, i.e. δ x (y) = 0 if y = x else 1 # the push forward operator.

µ ⊗ ν the product measure of two probability measures µ, ν, i.e., µ⊗ν(A×B) =

µ(A)ν(B) (id × T )#µ
Is the measure µ ⊗ (T #µ) Π (µ, ν) the set of couplings of two probability measures µ, ν (see definition 2.1.1)

M c (µ, ν)
The Monge cost between two probability measures µ, ν (see definition MP)

T c (µ, ν) The Kantorovitch cost between two probability measures µ, ν (see definition KP)

W p (µ, ν)
The p-Wasserstein distance between two probability measures µ, ν GW p (µ, ν)

The p-Gromov-Wasserstein distance between two probability measures µ, ν F µ the Fourier transform of the probability measure µ i.e. F µ (s) = ´e-2iπ s,x dµ(x)

Chapter 1

Introduction

Moving probability measures: a least effort problem

"Thus, we see in science, sometimes brilliant but for a long time useless theories, suddenly becoming the basis of the most important applications, and sometimes seemingly very simple applications, giving rise to the idea of abstract theories that are not yet needed, directing towards the theories of the Surveyors' work, and opening up a new career for them" 1 . That is how Nicolas De Condorcet [De Condorcet 1781] in the 18th century introduced the work of Gaspard Monge [START_REF] Gaspard Monge | Mémoire sur la théorie des déblais et des remblais[END_REF]] which is at the core of the Optimal Transport (OT) theory. How to move some masses from one location to another so as to minimize the overall effort? Condorcet was right: this "simple idea" has evolved over the years to become a theory at the crossroads of mathematics/optimization and is today at the center of many machine learning applications.

Broadly speaking the interest of Optimal Transport lies in both its ability to provide correspondences between sets of points and its ability to induce a geometric notion of distance between probability distributions (see Figure 1.1) Both have proved to be very useful for a wide range of tasks that are, to name a few, image registration [START_REF] Haker | [END_REF]], image retrieval [Rubner 1998], domain adaptation [Courty 2017], signal processing [START_REF] Kolouri | [END_REF]], unsupervised learning [Arjovsky 2017[START_REF] Genevay | [END_REF]], supervised and semi-supervised learning [Frogner 2015, Solomon 2014], natural language processing [START_REF] Kusner | [END_REF]], fairness [Gordaliza 2019], in biology [START_REF] Schiebinger | [END_REF]] or in astrophysics [START_REF] Frisch | [END_REF]].

Despite its many properties, the optimal transport problem remains difficult to solve in practice and is known to suffer from scalability problems that prevent its use on large, ubiquitous data in machine learning. The emergence of optimal transport in the machine learning community has been greatly favoured by recent achievements on the optimization side [Cuturi 2013, Altschuler 2017[START_REF] Genevay | [END_REF]] which tend to circumvent the heavy computational complexity of solving OT problems.

1 Original quote: "Ainsi, l'on voit dans les Sciences, tantôt des théories brillantes, mais longtemps inutiles, devenir tout à coup le fondement des applications les plus importantes, et tantôt des applications très simples en apparence, faire naître l'idée de théories abstraites dont on n'avait pas encore le besoin, diriger vers les théories des travaux des Géomètres, et leur ouvrir une carrière nouvelle." OT distance between two probability distributions in red and blue. The further away the mean of the red probability distribution is from the blue one the higher is the OT distance.

Moreover optimal transport, in its early formulation, is quite restricted to applications where there exists a direct way of comparing the samples of the data. Its applicability is thus often limited to the case where the samples are part of a common ground metric space, that is most of the time Euclidean. This limitation prevents its use for a variety of machine learning tasks where there is an additional structural information on the data which can not usually be described in the Euclidean setting e.g. when the samples are described by graphs, trees or time series. It also prevents the use of optimal transport when the samples lie in different, seemingly not related, metric spaces or when a meaningful notion of distance between the samples can not be easily defined. All of these instances can be framed into the incomparable setting, that is when the samples lie on incomparable spaces. An interesting remedy in this situation can be found in the theory of the Gromov-Wasserstein distance [Memoli 2011] which does not require the comparison of the samples across the distributions. However, it is well-known to be arduous to solve and to suffer from tedious scalability issues. The purpose of this thesis is to help overcoming these obstacles by: (i) Defining new Optimal Transport frameworks for incomparable spaces and especially for structured data.

(ii) Reducing the theoretical gap between classical Optimal Transport and the Gromov-Wasserstein theory in Euclidean spaces and in particular derive scalable and applicable formulations.

Structured Data and Incomparable Spaces in Machine Learning

Before going into the details of this thesis, it is important to make explicit what is behind the notion of structure through the manuscript. As a starter we can see the structural information as the piece of information which encodes the specific relationships that exist among the components of an object. This definition can be related with the concept of relational reasoning [START_REF] Battaglia | [END_REF]] where some entities (or elements with attributes such as the intensity of a signal) coexist with some relations or properties between them.

Natural instances of such structured data arise when the structure is explicit. For example, in a graph context, edges are representative of this notion so that each attribute of the graph (typically R d vectors) may be linked to some others through the edges between the nodes. Notable examples are found in chemical compounds or molecules modeling [START_REF] Kriege | [END_REF]], brain connectivity [Ktena 2017], or social networks [START_REF] Yanardag | [END_REF]]. This generic family of structured data also encompasses trees [START_REF] William | Optimal algorithms for comparing trees with labeled leaves[END_REF] or time series where the signals' values are correlated through time so that comparing times series requires one to take the direction of time into account.

Structure information of data in machine learning can also be more subtle or even implicit. For example it can appear where one build structural priors, or inductive bias [START_REF] Battaglia | [END_REF]], on the objects representation. For instance in the context of deep learning, many successful architectures exploit the equivariance to a symmetry transformation to improve generalization. The convolutional neural network (CNN) is a prime example of this "built-in" inductive bias which satisfies translation equivariance, i.e. if we translate the input, the output of the convolutions will also be translated. This inductive bias is known to reveal useful spatial hierarchy, or structure, on the pixels [START_REF] Wang | [END_REF], Chen 2018] and other works have studied designing layers with equivariances to other transformations such as permutation, rotation, reflection [START_REF] Kondor | [END_REF][START_REF] Cohen | [END_REF][START_REF] Gens | [END_REF]. Unlike end-to-end methods other more "hand-engineering" approaches based e.g. on images segmentation can leverage some structure in images that can further be usefully exploited [START_REF] Bach | [END_REF][START_REF] Shi | [END_REF]]. Implicit structure is also at the core of many natural language processing (NLP) tools used to find good word representations [Mikolov 2013b, Mikolov 2013a[START_REF] Pennington | [END_REF]. It is also the main ingredient of speech recognition [Hinton 2012] or sequence learning [Sutskever 2014]. In these cases structural assumptions on the sequences of words are made whether by the mean of latent variables or via conditional probabilities. When available, labels or classes also induce an implicit structure on the feature space of the data. For instance in Domain Adaptation one may desire the source samples with the same label to be matched consistently within the same region of the target space preventing them from being split into disjointed far locations [Courty 2017, Alvarez-Melis 2018b]. The recent trend of graph neural networks (GNN) [START_REF] Wu | [END_REF]] in the machine learning community is one of the many examples emphasizing that structured data remains an important and challenging setting nowadays. While previous instances consider structured data as inputs of the learning process the prevalence of this notion in machine learning also arises in another line of works where it is an output. This setting is e.g. considered in the structured prediction approach where one wants to learn to produce structured object such as sequences, trees or assignments [Taskar 2005[START_REF] Lafferty | [END_REF], Collins 2002, Blondel 2020[START_REF] Mensch | [END_REF], Korba 2018].

In short, the notion of structure in machine learning is omnipresent and appears as often as there is an additional information about the objects that goes beyond their feature representations. As shown in many contexts in machine learning such as graphical models [Pearl 1986[START_REF] Pearl | Judea Pearl. Causality: Models, reasoning and inference[END_REF], relational reinforcement learning [Dvzeroski 2001] or Bayesian nonparametrics [Hjort 2010], considering objects as a complex composition of entities together with some interactions is crucial in order to learn from small amounts of data.

The previous notion of structure data can be seen as a special case of data defined on incomparable spaces. Informally in this situation, each sample has its own "characteristic" that may not be shared with the other samples. For instance when considering a dataset of multiple graphs (each graph being a data point) the structure of one graph is usually not shared among the other graphs. This notion, purposely broad, encompasses also the case where data come from heterogeneous sources. As a particular instance of this problem Heterogeneous Domain Adaptation [START_REF] Yeh | [END_REF][START_REF] Zhou | [END_REF][START_REF] Wang | [END_REF]] aims to exploit knowledge from heterogeneous source domains to improve the learning performance in a target domain with, potentially, different feature spaces between the source and target domains. The MNIST/USPS [LeCun 2010, Hull 1994] case is a prime example of this situation: based on the knowledge of 28 × 28 digit images (i.e. R 784 vectors) from MNIST how to build e.g. a classifier that works well on 16 × 16 digit images (i.e. R 256 vectors) of USPS? Needless to say this problem often arises in all the fields of machine learning: it is common that the data are gathered from heterogeneous sources in practice and methods that build upon this diversity are often of high interest.

Motivating Optimal Transport

The central question that often arises in machine learning is: how to represent data and how to compare them? The framework of probability distributions provides an answer to this query by associating a probability measure to a collection of samples that forms a dataset (x i ) i∈ [[n]] . The Lagragian representation of the dataset results in a discrete probability measure n i=1 a i δ xi in which one associates to each point x i a dirac δ xi (x) = 1 if x = x i otherwise 0 as well as a weight a i ≥ 0 such that (a i ) i∈ [[n]] is a probability vector which satisfies n i=1 a i = 1. When no information about the relative importance of the samples in the dataset is available the weights can be chosen as uniform so that a i = 1 n . Similarly a Eulerian representation can be constructed via the probability distribution

N i=1 a i δ xi in which (x i ) i∈[[N ]
] is a regular grid on the space. This formulation produces an histogram of our data (see Figure 2). This point of view on the data advocates finding an appropriate way of comparing their representation as probability distributions and, as such, the question of finding adequate measures of "how far" are two probability distributions is at the core of many machine learning algorithms. Although various divergences exist such as φ-divergences [Csiszar 1975] or Maximum Mean Discrepancies (MMD) [START_REF] Gretton | [END_REF]], the richness of optimal transport lie in its ability to incorporate the geometry of the underlying space in its formulation and to pay attention to the relations, the correspondences of the samples within their respective representations. To highlight in short the benefit of representing data through probability distributions coupled with OT we can cite [START_REF] Rubner | [END_REF]] for image retrieval or [START_REF] Kusner | [END_REF]] for natural language processing. At this point natural questions arise: is this framework applicable when the nature of the data is inherently structured or when the different data points lie in incomparable spaces? In this case how can we represent data as probability distributions? To what extent this representation is valuable? Is the Optimal transport framework still applicable, and, if not, how do we compare these probability distributions? The purpose of this thesis, inter alia, is to give some answers to these questions.

Outline of the Thesis

This thesis covers mostly all the author's work conducted and focus on a single line of research that is Optimal Transport on Incomparable Spaces. Additional line of works [Vayer 2020a] on time series on incomparable spaces, which is not based on optimal transport, is not included in this thesis but the interested reader can find the details in the bibliography. The rest of the thesis is hinged so that all chapters can be read separately in any order except for Chapter 2 that provides all the mathematical background and tools used in the other chapters.

Chapter 2 sets up the mathematical and numerical background of optimal transport. It presents the fundamental results of classical optimal transport theory and summarizes/illustrates its different formulations as well as some well-known solvers. The philosophy of this chapter is to provide a high-level overview of OT both in theory and practice. This chapter concludes with the Gromov-Wasserstein theory which is at the core of thesis. A reader familiar with the basic concepts of optimal transport may skip this part although it contains crucial concepts and notations that will be discussed throughout the thesis.

Chapter 3 is dedicated to optimal transport for structured data, and especially in the context of graphs. It is based on the works of the articles [Vayer 2019a] and [Vayer 2020b] and gives some answers to the question of defining a mathematical framework for optimal transport in the case of structured data. A general framework for this setting is given, based on the Fused Gromov-Wasserstein distance that defines a OT distance between structured data such as undirected graphs, and applied on real world graph data applications.

Chapter 4 aims at bridging the gap between the classical optimal transport theory and the Gromov-Wasserstein theory. The chapter opens with the special case of 1D distributions which results in the Sliced Gromov-Wasserstein formulation based on the works in [Vayer 2019b]. A second more prospective part focuses on the Gromov-Wasserstein theory for Euclidean spaces and connects with the classical optimal transport theory by questioning the regularity of Gromov-Wasserstein optimal transport plans.

Chapter 5 presents a new framework for comparing probability measures on incomparable spaces, namely the CO-Optimal transportation problem. Contrary to the Gromov-Wasserstein approach this approach simultaneously optimizes two transport maps between both samples and features of the data. This chapter provides a thorough theoretical analysis of this framework and, from an application side, this work tackles the problem of Heterogeneous Domain Adaptation and co-clustering/data summarization. This chapter is based on the article [START_REF] Redko | [END_REF]].

Chapter 2 Optimal transport is a long-standing mathematical problem whose theory has matured over the years. A good gateway for this theory can be found in [Santambrogio 2015]. A more mathematical oriented overview can be found in [Villani 2008] while the most complete document about numerical aspects of OT can be found in [START_REF] Peyré | [END_REF]]. The objective of this chapter to present in short the main results of the "classical" OT theory both mathematically and numerically. We will discuss in the last part of this chapter the theory related to the Gromov-Wasserstein transportation problem for which we refer the reader to [Memoli 2011, Sturm 2012] for its foundation.

Generality about optimal transport

Linear Optimal Transport theory

The Monge problem The OT problem has been historically introduced by Gaspard Monge [START_REF] Gaspard Monge | Mémoire sur la théorie des déblais et des remblais[END_REF]] and can be described as the following "least effort problem": given two probability distributions µ and ν how do we transfer all the probability mass of µ onto ν so that the overall effort of transferring this mass is minimized? Originally the idea was to move dirt (déblais) to one place to another (remblais) in the most efficient way.

For properly defining this problem we need to define the notions of transfer and effort: the former can be expressed through the notion of push-forward and the latter through the notion of cost. More precisely and given two Polish spaces X , Y 1 and two probability measures µ ∈ P(X ), ν ∈ P(Y) a cost is a function c : X × Y → R + ∪ {+∞} which values c(x, y) aim at measuring how far is x ∈ X from y ∈ Y and quantifies somehow the "effort" of moving x forward to y. The push-forward 2 of a probability measure µ through a function T : X → Y is defined as the probability measure T #µ ∈ P(Y) which satisfies equivalently one of the two following conditions:

(i) T #µ(A) = µ(T -1 (A)) = µ({x ∈ X | T (x) ∈ A}) for every measurable set A (ii) ´Y φ(y)d(T #µ)(y) = ´X φ • T (x)dµ(x) for every measurable function φ.
These conditions simply state that we transform, or push, the probability measure µ thanks to T so as to create another probability measure on Y. When we consider a discrete probability distribution

µ = n i=1 a i δ xi the push-forward measure T #µ is simply defined by T #µ = n i=1 a i δ T (xi) .
As described at the beginning, one wants to move the source distribution µ forward to the target distribution ν. This translates mathematically as finding a map T which satisfies T #µ = ν. When we consider the Euclidean setting and when the probability measures have densities f, g with respect to the Lebesgue measure by the change of variable formula the push-forward condition writes:

g(T (x)) det(DT (x)) = f (x) (2.1)
where DT stands for the Jacobian of T . Among all these possible push-forwards, OT aims at finding the map T which minimizes the total cost of having moved µ forward to ν that is ´X c(x, T (x))dµ(x). Overall the problem of Monge (MP) can be formulated as the following non-convex optimization problem:

M c (µ, ν) = inf T #µ=ν ˆX c(x, T (x))dµ(x). (MP)
In general finding such optimal map T of the (MP) problem is quite difficult to solve since the solution may not be unique and may not even exists (see Figure 2.1). Even in the regular setting where µ, ν have densities, equation (2.1) is highly non-linear in T which is one of the major difficulty preventing from an easy analysis of the Monge Problem. As such the Monge problem remained an open question for many years and results about the existence and unicity of the optimal Monge map were limited to special cases until the works of Brenier [Brenier 1991] which implications will be detailed after.

Kantorovitch formulation

Major breakthroughs in the OT theory were made possible thanks to Kantorovitch [Kantorovich 1942] who proposes a relaxation of the (MP) problem. The key idea is to consider a probabilistic mapping instead of a deterministic map T to push the source measure forward to the target one. In the Kantorovitch formulation it is allowed to split the mass of the probability measures into pieces and transport them towards several targets points. This translates mathematically by replacing the push-forward of a measure by a probabilistic coupling: Definition 2.1.1 (Couplings). Let µ ∈ P(X ), ν ∈ P(Y). A coupling π of µ and ν is a probability distribution on X × Y such that both marginals of π are respectively µ and ν. More precisely π is part of the following set:

Π(µ, ν) = {π ∈ P(X × Y) |A ⊂ X , B ⊂ Y measurable π(A × Y) = µ(A) ; π(X × B) = ν(B)} (2.2)
1 A Polish space X is a separable completely metrizable topological space. 2 T is often called a map or Monge map in the OT literature Remark 2.1.1. Unlike the set of push-forwards of µ forward to ν the set of couplings of two probability measures is always non-empty as the product measure µ ⊗ ν is in Π(µ, ν).

An important illustration of the former definition is when µ and ν are discrete probability measures. This situation will be omnipresent thorough the manuscript we propose to detail the notations in the following example: [[m]] ∈ Σ m are probability vectors which belong to the following probability simplex:

Example 2.1.1 (The case of discrete probability measures). Let µ = n i=1 a i δ xi , ν = m i=1 b j δ yj be discrete probability measures where x i ∈ X , y j ∈ Y and a = (a i ) i∈[[n]] ∈ Σ n , b = (b j ) j∈
Σ n def = {a ∈ R n + | n i=1 a i = 1}. (2.3)
We will use interchangeably the term histogram or probability vector for an element a ∈ Σ n . In this case a coupling π is a matrix of the following set:

Π(a, b) = {π ∈ R n×m + |π1 m = a ; π T 1 n = b} = {π ∈ R n×m + |∀(i, j) ∈ [[n]] × [[m]], m j=1 π ij = a i ; n i=1 π ij = b j } (2.4)
Using the coupling instead of a deterministic map allows defining the OT problem for a very large class of probability measures under very mild assumptions. More precisely let X , Y be Polish spaces and Memo 2.1.1 (Lower semi-continuity). On a metric space (X , d) a function f : X → R ∪ {+∞} is said to be lower semi-continuous (l.s.c.) if for every sequence x n → x we have f (x) ≤ lim inf f (x n ). Such functions have the following properties:

• If (f k ) k is a sequence of l.s.c. functions on X then f = sup k f k is l.s.c.
• If f is l.s.c. and bounded from below then there exists a sequence of continuous and bounded functions (f k ) k∈N converging increasingly to f. We can also suppose that each f k is k-Lipsichtz. The resulting cost, potentially infinite without further assumptions, corresponds to the minimal cost of moving µ forward to ν by splitting their masses and transporting forward the pieces according to the transport plan π. The good news about this formulation is that the infimum is always well defined providing that the cost is positive and lower semi-continuous (actually it suffices that c is bounded from below see [Santambrogio 2015, Theorem 1.7]). The problem KP is defined regardless of the nature of the probability distributions: they can be both discrete or continuous (see Figure 2.3). The problem appears to be linear in π so that we will denote (KP) as the linear transportation problem.

Relying on the Kantorovitch formulation (KP) appears to be very useful in order to find a solution of the Monge problem (MP). Indeed a push forward T #µ = ν induces a coupling π = (id × T )#µ 3 then it is easy to verify that T c (µ, ν) ≤ M c (µ, ν). To find the converse inequality it suffices to find an optimal solution π * of (KP) which is of the form π * = (id × T * )#µ where T * #µ = ν. In this case we would have proven that both problems are equal and that T * is optimal for (MP). In other words if there is an optimal coupling supported on a deterministic function then both (KP) and (MP) are equivalent. We will see in next sections cases where we can ensure that necessarily the optimal coupling is of this form.

Example 2.1.2. The famous bakery analogy of Villani's book [Villani 2003] provides a simple illustration of the linear OT problem. Suppose that someone is in charge of the distribution of bread from bakeries

3 (id × T )#µ is the measure µ ⊗ T #µ or equivalently d(id × T )#µ(x, y) = dµ(x)d(T #µ(y))
Figure 2.3: Different OT problems between (left) two discrete probability distributions, the coupling π is a matrix (center) one continuous and one discrete probability distribution, this is called the semi-discrete OT problem (right) two continuous distributions. Inspired from [START_REF] Peyré | [END_REF] to cafés in Manhattan. The bakeries are located at some points y and the cafés at some points x distant from each other by c(x, y). At 8 a.m sharp all the bread from the bakeries has to be transferred to the cafés in order for the citizens of Manhattan to have a good day. The company in charge of the distribution wants to route the breads from bakeries to cafés the cheapest way possible. This problem can be recast into a linear OT problem. Considering two distributions µ="all available breads in bakeries" and ν="all the demands in bread of cafés", the company seeks for a transport plan π such that dπ(x, y) is the amount of breads transferred from bakery y to café x. The best transport plan minimizes the overall cost of moving all the breads from bakeries to cafés which is ´c(x, y)dπ(x, y) (see Figure 2.2)

Wasserstein distance

The most notable scenario in many OT applications is when X = Y = Ω where (Ω, d) is a Polish space, e.g. an Euclidean space. In this case there is a natural way of defining the cost c since the space Ω is already endowed with a notion of distance between the points. In this situation we can define the so-called p-Wasserstein distance

for p ∈ [1, +∞[ as W p (µ, ν) = (T d p (µ, ν)) 1 p or precisely: W p (µ, ν) = inf π∈Π(µ,ν) ˆΩ×Ω d p (x, y)dπ(x, y) 1 p (2.5)
The name is not misleading: this function satisfies all the axioms of distance on the space of probability distributions with bounded p-moments as stated in the next theorem (see [Villani 2008, Definition 6.4]):

Theorem 2.1.1 (The Wasserstein distance is a distance). Let (Ω, d) be a Polish space, p ∈ [1, +∞[ and:

P p (Ω) def = {µ ∈ P(Ω) | ˆΩ d(x 0 , x) p dµ(x) < +∞} (2.6) with x 0 ∈ Ω arbitrary. Let µ, ν ∈ P p (Ω). Then W p (µ, ν) < +∞. Moreover: (i) W p (µ, ν) = W p (ν, µ) (symmetry) (ii) W p (µ, ν) = 0 ⇐⇒ µ = ν (identity of indiscernibles) (iii) Let ζ ∈ P p (Ω) 2 then W p (µ, ν) ≤ W p (µ, ζ) + W p (ζ, ν) (triangle inequality)
Example 2.1.3. One direct example of Wasserstein distance is between two diracs supported on x,y. In this case W p (δ x , δ y ) = d(x, y) which is quite the intuitive behavior, that is the more the dirac are located far from each others the larger their Wasserstein distance is.

The distance property renders W p a powerful tool for comparing probability measures. Another valuable feature of the former distance is that it gives a characterization of the weak convergence of probability measure. Informally a sequence of probability measures gets as close as possible to a probability measure µ if the Wasserstein distance tends to zero. The convergence is based on the following definition: Definition 2.1.2 (Weak-convergence). Let (µ n ) n∈N be a sequence of probability measures on X a Polish space. We say that (µ n ) n∈N converges weakly to µ in X if for all continuous and bounded functions

f : X → R: ˆX f dµ n → ˆX f dµ (2.7)
The Wasserstein distance metrizes the weak convergence of probability measures, in other words (µ n ) n∈N converges weakly to µ if and only if W p (µ n , µ) → 0 [Villani 2003, Theorem 6.9]. Note that other distances can be proposed to metrize the space of probability measures e.g. the Lévy-Prokhorov distance, but the richness of W lies in its ability to incorporate a lot of the geometry of the underlying space through the distance d. Consequently Wasserstein spaces (P p (Ω), W p ) are very large and many metric spaces can be embed into Wasserstein spaces with low distortion [START_REF] Bourgain | The metrical interpretation of superreflexivity in banach spaces[END_REF][START_REF] Andoni | [END_REF][START_REF] Frogner | [END_REF]].

The main theorem of the linear Optimal Transport theory

A fundamental result of the linear OT theory is the cyclical monotonicity property of its optimal transport plans. Basically it illustrates that an optimal transport plan can not be improved locally and more importantly that is also sufficient for being a global optimal transport plan. Consequently it characterizes the set of optimal couplings using the notion of c-concave functions which appears to be very useful for defining the notions of duality and for solving the Monge problem (MP) by relying on the Kantorovitch relaxation (KP). As such the cyclical monotonicity is maybe the main ingredient of linear OT. This section aims at presenting in short both this result and its consequences.

Definition 2.1.3 (Cyclically Monotone Set). Let c : X × Y →] -∞, +∞] be a real valued function on arbitrary sets X , Y. A set Γ ⊂ X × Y is said to be cyclically monotone if for (x i , y i ) N i=1 ∈ Γ N and σ a permutation of [1, .., N ] N i=1 c(x i , y i ) ≤ N i=1 c(x i , y σ(i) )
(2.8)

We will call c-CM such sets.

c-CM sets is an important notion in OT theory since it characterizes optimal transport plans for well behaved costs. We consider the following definition: Definition 2.1.4 (Support). Let (X , d) be a Polish space and µ ∈ P(X ). The support of µ is defined as the smallest closed set F such that µ(F ) = 1 or equivalently:

supp(µ) = {x ∈ X |∀ε > 0, µ(B(x, ε)) > 0}
(2.9)

Informally the support of a distribution is where the distribution "lives", i.e. where it is not zero. In the discrete case π is a matrix and the support is found in the indices (i, j) such that π ij > 0. The following theorem states that the support of an optimal coupling is actually a c-CM set:

Theorem 2.1.2 (Theorem 1.38 in [Santambrogio 2015]). Let c : X × Y → [0, +∞[ continuous and µ ∈ P(X ), ν ∈ P(Y) with T c (µ, ν) < +∞. If a coupling π ∈ Π(µ, ν) is optimal for (KP) then supp(π) is a c-CM set.
Interestingly enough the c-CM sets are characterized by specific functions based on the notion of c-transforms. This property gives another way of computing optimal transport plans based on deterministic functions.

Definition 2.1.5 (c-transforms). Let X , Y be Polish spaces and ψ : X → R be a function. We define its c-transform as the function ψ c : Y → R:

ψ c (y) = inf x∈X c(x, y) -ψ(x)
(2.10)

and the c-transform of a function φ : Y → R as the function φ c : X → R:

φ c(x) = inf y∈Y c(x, y) -φ(y) (2.11)
Fonctions that can be written as ψ c or φ c are called respectively c-concave or c-concave functions.

Remark 2.1.2. The c-transform is a a generalization of the Legendre transform that is well-known in convex analysis [Rockafellar 1970]. More precisely for function u : R d → R its Legendre transform is defined as u * (y) = sup

x∈R d y, x -u(x) (see Memo 2.1.2).
The c-transform corresponds to this notion by considering c(x, y) = x, y (up to the change of sign). Another special case deserves attention that is when c(x, y) = 1 2 xy 2 . Consider ψ : R d → R ∪ {+∞}, then the function ψ is c-concave if and only if the function u : x → 1 2 x 2 -ψ(x) is convex and lower semi-continuous and the Legendre transform of u is the function x → 1 2 x 2 -ψ c (x) (see [Santambrogio 2015, Proposition 1.21]). When X = Y and c is symetric both notions are equivalent and in this case we will drop this distinction. One important property about c-transform is that it satisfies:

∀x, y ∈ X × Y, ψ(x) + ψ c (y) ≤ c(x, y)
(2.12)

The case of equality of (2.12) is attained on special subsets of X × Y that are precisely the c-CM sets as stated in the next theorem:

Memo 2.1.2 (Convex analysis). For any function u : R d → R its convex conjugate or Legendre transform is defined by u * (y) = sup

x∈R d y, x -u(x). The subdiffenrential of u, denoted as ∂u, is defined for x ∈ R d as ∂u(x) = {y ∈ R d | ∀z ∈ R d u(x)-u(z) ≥ y, z-x } which reduces to ∂u(x) = {∇u(x)}
when u is differentiable at x. When u is convex differentiable the convex conjugate has the following important properties (see [Rockafellar 1970, Theorem 23.5]):

i u(x) + u * (y) ≥ x, y (Fenchel-Young inequality) ii u(x) + u * (y) = x, y ⇐⇒ y ∈ ∂u(x)
iii In particular u(x) + u * (∇u(x)) = x, ∇u(x)

Theorem 2.1.3 (Theorem 1.37 in [Santambrogio 2015]).

If Γ = ∅ is a c-CM set in X × Y and c : X × Y → R, then there exists a c-concave function ψ : X → R ∪ {-∞} such that: Γ ⊂ {(x, y) ∈ X × Y : ψ(x) + ψ c (y) = c(x, y)} (2.13)
To summarize, the support of optimal couplings are necessarily c-CM sets and these c-CM sets are also characterized by functions using the notion of c-transform. The fundamental theorem of optimal transport states that all these results are in fact equivalent:

Theorem 2.1.4 (Fundamental theorem of linear OT). Let X , Y be Polish spaces, µ ∈ P(X ), ν ∈ P(Y) and c : X × Y → [0, +∞[ lower-semi continuous such that T c (µ, ν) < ∞. Let π ∈ Π(µ, ν) then the following conditions are equivalent: (i) π is optimal for (KP) (ii) The support of π is c-cyclically monotone (iii) There exists a measurable c-concave function ψ such that ψ(x) + ψ c (y) = c(x, y) π a.e.
Proof. We will give a sketch of proof, for completeness the reader can refer to Theorem 5.10 in [Villani 2008]. Theorems 2.1.2 and 2.1.3 already proved that (i) =⇒ (ii) =⇒ (iii) in the case where c is continuous. To pass from continuity to lower semi-continuity we can consider a sequence (c k ) k of costs that converges increasingly to c and observe that [Santambrogio 2015, Lemma 1.41]:

lim k→+∞ inf π∈Π(µ,ν) ˆck (x, y)dπ(x, y) = inf π∈Π(µ,ν) ˆc(x, y)dπ(x, y)
and, using some subtleties, this can prove (i) =⇒ (ii) =⇒ (iii) when c is lower semi-continuous. For the converse we can easily prove that (iii) =⇒ (i). By hypothesis ´c(x, y)dπ(x, y) = ´ψ(x)dµ(x) + ´ψc (y)dν(y). However for any other coupling π we have ´c(x, y)dπ (x, y) ≥ ´ψ(x)dµ(x) + ´ψc (y)dν(y) by relation (2.12) and so ´´c(x, y)dπ (x, y) ≥ ´c(x, y)dπ(x, y) so that π is optimal. Technical details are hidden here for proving the measurability and integrability of ψ, ψ c .

The main theorem of OT for duality A first implication of the fundamental theorem is related to a duality principle which is a widely used property in linear programming (see Section 2.1.4 for more details). This property can be extended in full generality in the context of OT as stated in the next theorem: Theorem 2.1.5 (Duality theorem). Let X , Y be Polish spaces, µ ∈ P(X ), ν ∈ P(Y) and c : X × Y → [0, +∞] be lower semi-continuous (l.s.c.) such that T c (µ, ν) < +∞ then strong duality holds. More precisely the dual problem:

sup φ,ψ∈C b (X )×C b (Y) ∀(x,y)∈X ×Y, φ(x)+ψ(y)≤c(x,y) ˆX φ(x)dµ(x) + ˆY ψ(y)dν(y) (DKP)
leads to the same optimum as the (KP) problem. Equivalently:

T c (µ, ν) = sup φ,ψ∈Φc ˆX φ(x)dµ(x) + ˆY ψ(y)dν(y) (2.14)
where Φ c is the set of continuous bounded functions which verifies:

∀x, y ∈ X × Y, φ(x) + ψ(y) ≤ c(x, y) (2.15)
This result holds when Φ c is replaced by Φ c (µ, ν) the set of integrable functions which satisfies (2.15).

Proof. For completeness we will give here an idea of the proof, the interested reader can refer to Theorem 5.10 in [Villani 2008] for more details. If φ, ψ ∈ Φ c (µ, ν) and π ∈ Π(µ, ν) then by hypothesis:

ˆφ(x)dµ(x) + ˆψ(y)dν(y) = ˆφ(x) + ψ(y)dπ(x, y) ≤ ˆc(x, y)dπ(x, y) (2.16) Which implies that sup φ,ψ∈Φc(µ,ν) ´X φ(x)dµ(x)+ ´Y ψ(y)dν(y) ≤ T c (µ, ν).
To show the converse inequality we will use the cyclical monotonicity properties of optimal transport plans. Let π * be an optimal coupling for the (KP) problem. Using Theorem 2.1.4 we know that there exists a c-concave function ψ such that

ψ(x) + ψ c (y) = c(x, y) for all x, y ∈ X × Y. In this way: ˆc(x, y)dπ * (x, y) = ˆψ(x)dµ(x) + ˆψc (y)dν(y) ≤ sup φ,ψ∈Φc ˆX φ(x)dµ(x) + ˆY ψ(y)dν(y) (2.17)
Last inequality stems from the property (2.12) of c-transform since (ψ, ψ c ) ∈ Φ c . If c is continuous and bounded then so are ψ, ψ c so last inequality is valid. If c is only l.s.c. then we can show that there is a sequence (c k ) k bounded and n-Lipschitz such that c = sup k c k . A limit argument suffices to conclude for this case.

The functions φ, ψ are usually called Kantorovitch potentials and play an important role in OT problems. Given two admissible potentials φ, ψ i.e. that satisfy φ(x) + ψ(y) ≤ c(x, y) we can always cook up a pair of "better" potentials using the c-transform. Indeed, due to (2.12), one can check that the pairs (φ, φ c ), (ψ, ψ c) are also admissible potentials and improve the objective function. It turns out that after one iteration of this procedure we can not improve the potentials anymore. Based on this remark we can also write the duality as the maximization over one single potential which is the semi-dual formulation:

sup φ c-concave ˆX φ(x)dµ(x) + ˆY φ c (y)dν(y) (2.18)
Example 2.1.4. When c = d is a distance on some space Ω then there is a tight connection between c-transform and 1-Lipschitz functions. Indeed suppose that φ is a 1-Lipschitz function, then for x, y

∈ Ω 2 , φ(y) ≤ φ(x) + d(x, y) so that φ(y) = inf x∈Ω φ(x) + d(x, y) = (-φ) d (y) which proves that the d-transform of -φ is φ.
The converse is also true so that the semi-dual formulation can be written:

sup φ∈Lip1(Ω) ˆΩ φ(x)dµ(x) -ˆΩ φ(y)dν(y) (2.19)
This formulation is very useful in practice in the context of generative modeling [Arjovsky 2017] (see Section 2.1.5).

The main theorem of OT for the Monge problem

From a theoretical perspective one fundamental question that arises is the regularity of these potentials and, with some assumptions, we can use them to solve the Monge problem (MP) based on the Kantorovitch relaxation (KP). We have the following result:

Proposition 2.1.1 (Proposition 1.15 in [Santambrogio 2015]). Let X = Y = Ω ⊂ R d and c ∈ C 1 . In the following φ denotes a Kantorovitch potential. If (x 0 , y 0 ) ∈ supp(π * ) then:

∇φ(x 0 ) = ∇ x c(x 0 , y 0 ) (2.20) provided that φ is differentiable at x 0 .
This proposition suggests the following strategy in order to find an optimal coupling: (1) Ensure that φ is differentiable µ a.e. This can be guaranteed when µ is absolutely continuous with respect to the Lebesgue measure and when φ is regular enough, such as Lipschitz.

(2) Deduce from previous proposition that π is characterized by a deterministic function that is the map associating y 0 to each x 0 . The idea here is to "inverse" ∇c in (2.20) and deduce from (x 0 , y 0 ) that y 0 is uniquely determined from x 0 . This can be done using some regularity assumptions on c and the spaces X , Y. When conditions (1) and ( 2) are satisfied we can deduce that the optimal coupling is unique since it was constructed using φ and c only.

The step (2) can be verified e.g. when we have the following condition:

Definition 2.1.6 (Twist condition). For Ω ⊂ R d we say that c : Ω × Ω → R satisfies the Twist condition whenever c is differentiable w.r.t. x at every point, and the map y → ∇ x c (x 0 , y) is injective for every x 0 .

When working on Euclidean domains and when the cost c ∈ C 2 this condition corresponds to det ∂ 2 c ∂yi∂xj = 0. The squared Euclidean cost is an important example of costs which satisfies the Twist condition and leads to the celebrated Brenier theorem [Brenier 1991]: Theorem 2.1.6 (Brenier). Let Ω = R p , c(x, y) = 1 2 xy 2 2 and µ ∈ P(R p ) absolutely continuous with respect to the Lebesgue measure and ν

∈ P(R p ) with ´ x 2 2 dµ(x) < +∞,
´ y 2 2 dν(y) < +∞. The optimal transport plan π * of (KP) is unique and supported on the gradient of a convex function. More precisely it can be written as π * = (id × T )#µ where T = ∇φ and φ : R d → R ∪ {+∞} is convex and finite almost everywhere.

Moreover T is the unique solution of (MP). If T is another optimal solution then T = T µ a.e.

This result can be generalized to costs c(x, y) = h(y -x) with h strictly convex and in this case T can be written as T (x) = x -(∇h) -1 ∇φ(x) where φ is a c-concave function (see e.g. [Gangbo 1996]).

The regularity of potential functions φ, ψ and its consequences for Optimal Transport problems is a long-standing line of research. Other more general hypothesis on the cost function c than the Twist condition can be built thanks to Ma, Trudinger and Wang who found a key assumption on the cost c that requires fourth-order condition on the cost functions [Ma 2005]. The resulting MTW conditions turned out to be sufficient to prove the regularity of the Kantorovitch potentials. We refer the reader to [Figalli 2010] for a survey on this topic.

Special cases: 1D transportation and transport between Gaussians

Two important special cases will be considered in this manuscript, namely the cases where µ and ν are probability distributions on R or when they are Gaussians distributions. Theses cases are wellknown in linear OT for having closed-form solutions which are given in the next results (respectively [Santambrogio 2015, Theorem 2.9] and [Peyré 2019, Remark 2.30]). Theorem 2.1.7 (Closed-form expression on the real-line). Asssume that Ω = R, µ, ν ∈ P(R). Let F µ be the cumulative distribution function:

∀t ∈ R, F µ (t) = µ(] -∞, t]) (2.21)
and F -1 µ its pseudo inverse, namely:

∀x ∈ [0, 1], F -1 µ (x) = inf{t ∈ R |F µ (t) ≥ x} (2.22) If c(x, y) = h(y -x)
where h is stricly convex then (KP) has a unique solution given by π

* = (F -1 µ × F -1 ν )#L [0,1] where L [0,1] is the Lebesgue measure restricted to [0, 1]. Moreover if µ is atomless π * is supported on T mon (x) = F -1 ν (F µ (x)), i.e. π * = (id × T mon )#µ.
If h is only convex then π * is still optimal but uniqueness can not be guaranteed.

This theorem states that it suffices to sort the support of the distributions in order to recover the optimal coupling (see Figure 2.4). In the special case where µ

= 1 n n i=1 δ xi , ν = 1 n n i=1 δ yi this corresponds to sort x 1 ≤ • • • ≤ x n , y 1 ≤ • • • ≤ y n
and to associate x 1 with y 1 , x 2 with y 2 and so on. In the case µ = n i=1 a i δ xi , ν = m i=1 b j δ yj the previous theorem states that, after sorting the points, the optimal mapping is obtained by putting as much mass as possible from x 1 to y 1 and to add the remaining mass to y 2 . This procedure is repeated until there is no more mass left. This corresponds to a monotone rearrangement π ij , π i j > 0 then x i ≤ x i implies that y j ≤ y j . Overall the Wasserstein distance in 1D can be solved using simple sorts. This result is the main ingredient of the sliced-Wasserstein distance (see Section 2.1.5).

Another special case arises when the probability measures are Gaussian. This is a well known result in the literature of OT geometry [Givens 1984, McCann 1997, Takatsu 2011] which is recalled in the following theorem:

Theorem 2.1.8 (Closed form expression for Gaussians). Let µ = N (m µ , Σ µ ), ν = N (m ν , Σ ν ) and suppose that c(x, y) = h(y -x) with h striclty convex. Let T : x → m ν + A(xm ν ) where:

A = Σ -1/2 µ (Σ 1/2 µ Σ ν Σ 1/2 µ ) 1 2 Σ -1/2 µ (2.23)
then T is the unique optimal solution of (MP) and π * = (id × T )#µ is the unique optimal solution of (KP).

In particular when c(x, y) = xy 2 is the Euclidean distance on R d the 2-Wasserstein distance is given by:

W 2 2 (µ, ν) = m µ -m ν 2 2 + B(Σ µ , Σ ν ) 2 (2.24)
where

B(Σ µ , Σ ν ) = tr Σ µ + Σ ν -2(Σ 1/2 µ Σ ν Σ 1/2 µ ) 1 2
is the Bures metric [Bures 1969].

Interestingly enough the problem of computing OT between Gaussian measures draws connections with the general case. Indeed for µ, ν ∈ P 2 (R d ) and c(x, y) = xy 2 the optimal map T defined in Theorem 2.1.8 is actually the optimal Monge map of (MP) when restricted to the class of linear Monge map [Flamary 2019, Proposition 1]. Figure 2.5 illustrates the behavior of this map for two discrete probability measures on R 2 .

Note that a generalization of the previous result exists for elliptical distributions which are somehow generalizations of Gaussian densities. In this case the W 2 admits also a closed-form (see [START_REF] Muzellec | [END_REF]).

Some statistical aspects of OT

In most of machine learning applications we do not have access to the true distributions µ, ν but only to samples from these distributions. As such a natural question arises: can we infer from this samples good estimates of OT objects such as couplings or OT distances? One particular question is how well can we estimate the Wasserstein distance by relying only on samples of the distribution? If we consider a probability distribution µ ∈ P(R d ) and an empirical distribution µ n = 1 n n i=1 δ xi where x i ∼ µ are iid samples does µ n is a good proxy for µ? Unfortunately the sample complexity of the estimation of the Wasserstein distance is exponential in the dimension of the ambient space. More precisely E[W p (µ n , µ)] = O(n -1 d ) so that the Wasserstein distance suffers from the curse of dimensionality [Dudley 1969, Weed 2017]. It was shown in [Weed 2017] that this result can be refined to O(n -1 p ) where p is the intrinsic dimension of the , x2, x3, x4 to y1, y2, y3, y4 can be described whether by a permutation σ or using assignment matrix πσ. In this example both correspond to the permutation σ:

1 → 3,2 → 2,3 → 4, 4 → 1
data but, generally, this is a major bottleneck for the use of OT in high-dimensional machine learning problems. Previous analysis can be extended to the infinite dimensional setting as analysed in [Lei 2020].

The problem of estimating the optimal coupling by relying on small batches of µ when it is discrete was further analyzed in [START_REF] Fatras | [END_REF]].

To circumvent this limitation some robust projection formulations have been proposed [START_REF] Niles | Estimation of Wasserstein distances in the Spiked Transport Model[END_REF], Lin 2020] as well as strategies such as gaussian-smoothing [Goldfeld 2020] or based on wavelet estimator [Weed 2019a]. The entropic regularization presented in the next section in also one of the tool that facilitates the estimation of W p for high-dimensional settings. For more details about statistical aspects of OT we refer the reader to [Weed 2019b].

A quick numerical tour: solving Optimal Transport

In this section we consider the problem of computing OT between discrete probability measures µ = n i=1 a i δ xi , ν = m i=1 b j δ yj . The problem can be solved in many ways and we aim here at giving a brief summary these possibilities. We denote by C = (c ij ) i∈ [[n]],j∈ [[m]] the matrix of all pair-to-pair costs between the samples x i , y j , i.e. c ij = c(x i , y j ) for all i ∈ [[n]], j ∈ [[m]]. In the discrete case the underlying problem reads:

min π∈Π(a,b) C, π F = min π∈Π(a,b) ij c ij π ij .
(2.25)

As described previously the problem is linear in π, in this way the discrete case corresponds to a linear program (LP) [Dantzig 1997]. Before discussing potential algorithms for solving equation (2.25) we detail one important special case.

Assignment problems Suppose that m = n. In this case we can look for an assignment of the points, that is a one-to-one correspondence between the points x i , y i . This translates mathematically by looking at the permutation σ ∈ S n of the points or at the permutation matrix

π σ =    1 n , if j = σ(i) 0, otherwise such that
the overall cost is minimized (see Figure 2 .6). In this situation one aims at solving:

min σ∈Sn 1 n n i=1 c i,σ(i) = min σ∈Sn C, π σ F (2.26)
This problem is well known in the literature as the linear assignment problem (see e.g. [Burkard 1999]). It is worth pointing that, in this case, it exactly corresponds to the Monge problem (MP) in the discrete case. Interestingly enough when the weights of the OT problem (2.25) are set as uniform i.e. a = b = 1 n 1 n Algorithm 1 North-West corner rule 1: a, b, i, j = 1 2: while i <= n, j <= m do 3:

π ij = min{a i , b j } // Send as many units as possible form i to j 4:

a i = a i -π ij // Adjust the supply 5: b j = b j -π ij // Adjust the demand 6:
If a i = 0, i = i + 1, if b j = 0, j = j + 1 7: end while both problems (2.25) and (2.26) are equivalent. More precisely by combining the fundamental theorem of linear programming [START_REF] Bertsimas | [END_REF]], which states that the minimum of a linear program is reached at an extremal point of the polyhedron, and Birkhoff's theorem [START_REF] Birkhoff | [END_REF]], which states that the extremal points of Π( 1 n 1 n , 1 n 1 n ) is the set of permutation matrices, we can conclude that the optimal map of (2.25) is reached at π σ which is optimal for (2.26).

Algorithmic solutions

To solve the OT problem (2.25) in general one can rely on classical algorithms for solving (LP) [Dantzig 1997]. We make here a brief overview of possible numerical solutions and we refer the reader to Section 3 in [START_REF] Peyré | [END_REF]] for more details.

As seen in Theorem 2.14 the OT problem can be solved using duality which reads in the discrete case:

max α∈R n ,β∈R m ∀(i,j)∈[[n]]×[[m]],αi+βj ≤cij α T a + β T b (2.27)
where α, β denotes the Kantorovitch potentials. Thanks to the fundamental Theorem 2.1.4 an optimal solution π * of the primal problem is found when α * i + β * j = c ij for π * ij > 0 where α * , β * are solutions of the dual problem. Using this remark we can solve (2.25) by relying on the Network Simplex algorithm which philosophy is to find feasible solutions (α, β) such that α i + β j = c ij whenever π ij > 0 (in this case we say that π and (α, β) are complementary w.r.t. C). The complexity of this algorithm is O(n 3 log(n)) when m = n. The special case of uniform weights for assignment problems can be solved using the Auction algorithm which has a cubic complexity O(n 3 ).

Special cases: Monge property

The case where C has special structure deserves attention. In particular when C satisfies the following Monge property [Burkard 1996]:

∀(i, j) ∈ [[n]] × [[m]], c i,j + c i+1,j+1 ≤ c i+1,j + c i,j+1
(2.28) which can be tested in O(mn) operations. This property has some interesting historical background. It is actually based on the original observation of Monge who states that if quantity must be transported from locations x 1 ,y 1 to locations x 2 ,y 2 then the route from x 1 and the route from y 1 must not intersect: better not to cross the paths!4 In this case the simple North-West corner rule (see Algorithm 1) produces an optimal solution in O(n + m). Special cases: 1D probability distributions As seen in Section 2.1.2 the case of 1D probability distributions can be solved efficiently using simple sorts when C is e.g. a squared Euclidean distance matrix. The complexity of computing the Wasserstein distance is O(n log(n)) when n = m and weights are uniform and in general it suffices to compute the two cumulative distribution functions which is

O(n log(n) + m log(m)).
Special cases: Gaussian distributions When µ and ν are Gaussian distributions (and with a Euclidean cost) the OT problem is also quite easy to solve. In the discrete case, when relying on samples from µ, ν, and using the empirical version of the means and covariances, finding the optimal solution has a O((n + m)d 2 + d 3 ) complexity.

Although previous special cases exist solving the OT problem in general remains costly. The next section presents a regularization scheme that tends to lower this computational complexity and was one of the major breakthrough in OT past years.

Entropic regularization

The idea of penalizing the entropy of the joint coupling can be traced back to Schrödinger [Schrodinger 1931] and its use for linear OT to Wilson [Wilson 1969], yet it was made popular quite recently in the OT community [Cuturi 2013]. The entropic regularization has multiple virtues in practice: 1) it turns the optimal transport problem into a strongly-convex minimization problem which solution is unique 2) solving an entropic regularized OT only involves simple iterations of matrix-vectors products which can be plugged easily into modern differentiable frameworks 3) it can be accelerated on GPU and can solve in parallel several OT problems 4) it has many desirable properties for high-dimensional problems statistically speaking.

The entropy term for a coupling π reads as:

H(π) = - ij (log(π ij ) -1)π ij (2.29)
which corresponding entropic regularized OT problem:

Chapter 2. Generality about optimal transport

Algorithm 2 Sinkhorn-Knopp Algorithm for entropic transport (i-1) // Update right scaling 5: end for 6: return π * = Diag(u)K Diag(v)

1: a, b, C, ε > 0, u (0) , v (0) = 1, K = exp(-C ε ) 2: for i = 1, . . . , n it do 3: u (i) = a K v (i-1) // Update left scaling 4: v (i) = b Ku
T ε c (µ, ν) = min π∈Π(a,b) C, π F -εH(π) (ε-KP)
Interestingly enough the optimal cost of (KP) can be obtained as ε → 0, i.e. lim ε→0 T ε c (µ, ν) = T c (µ, ν) [Peyré 2019, Propositon 4.1]. As a side effect, the entropic term tends to blur the optimal coupling so that more points are associated compared to the sparse optimal solution of the original problem. In other words entropy forces the solution to have a spread support. In the limit setting where ε → +∞ all points are coupled together such that lim ε→+∞ π * ε = ab T where π * ε denotes the optimal coupling of (ε-KP) (see Figure 2.7). Note that the entropy regularization can also be defined when the probability measures are not discrete and in this case reads H(π) = ´X ×Y (log( dπ(x,y) dµ(x)dν(y) ) -1)dπ(x, y).

Sinkhorn-Knopp and Bregman projections

A simple analytic solution of (ε-KP) can be found using the lagragian duality as expressed in the following proposition [Peyré 2019, Proposition 4.3]:

Proposition 2.1.2. Problem (ε-KP) has a unique solution of the form π * = Diag(u)K Diag(v) with K = e -C ε and u, v ∈ R n + × R n + .
As written in [START_REF] Sinkhorn | [END_REF]] there is a unique solution of the form π * = Diag(u)K Diag(v) with marginals a, b providing that K is positive definite. Moreover it can be recovered based on the Sinkhorn-Knopp Matrix scaling algorithm that relies on matrices multiplications by alternatively updating u and v in order for π * to have the prescribed marginals (see in Algorithm 2). When n = m and by setting τ = 4 log(n) ε the Sinkhorn algorithm produces an optimal solution π * such that C,

π * F ≤ T c (µ, ν) + τ after O( C 3 ∞ log(n)τ -3
) iterations [Altschuler 2017]. In particular this implies that a τ -approximate solution of the original unregularized problem can be computed in O(n 2 log(n)τ -3 ) time.

From a practical point of view the Sinkhorn's algorithm suffers from stability issues when ε → 0 as the kernel K vanishes rapidly which results in divisions by 0 during the algorithms' iterations. To avoid such underflows for small value of ε [Schmitzer 2016] suggest a log-sum-exp stabilization trick whose iterations turn to be mathematically equivalent to the original iterations.

This problem is also a special case of a Kullback-Leiber minimization problem where one wants to find a coupling matrix π * the closest possible to a kernel K in the sense of the Kullback-Leiber geometry. More precisely (ε-KP) is equal to:

KL(π|K)

(2.30)

where

K = e -C ε and KL(π, K) = ij π ij log( πij Kij ) -π ij + K ij
is the Kullback-Leiber divergence between π and K. Reformulating OT problems as a minimization of a Kullback-Leiber divergence allows the use of the machinery of Bregman projections in order to find a solution and to analyse the convergence [Benamou 2015]. This formulation is particularly interesting for solving multi-marginals OT problems [Nenna 2016], regularized OT barycenter [Benamou 2015, Bigot 2019b] and the Gromov-Wasserstein problem [START_REF] Peyré | [END_REF]] in short.

Sinkhorn divergences

One drawback of entropic regularized OT is that it induces a bias T ε c (µ, µ) = 0 which can be problematic for learning using T ε c . In [START_REF] Genevay | [END_REF]] authors propose to correct this bias by considering the so-called Sinkhorn divergence:

SD c,ε (µ, ν) = T ε c (µ, ν) - 1 2 T ε c (µ, µ) - 1 2 T ε c (ν, ν).
(2.31)

This divergence enjoys many valuable properties. First it defines a symmetric positive definite smooth function on the space of probability measures that is convex in both µ, ν and that metrizes the weak convergence of probability measures [START_REF] Feydy | [END_REF]]. Second it interpolates, through ε, between Wasserstein distance and Kernel norms (MMD) allowing finding a trade-off between both. Finally it is more suited for high-dimensional problems where the estimation of the Wasserstein distance is known to suffer from the curse of dimensionality (the sample complexity if

O(n -1 d ) as explained in Section 2.1.3) whereas the sample complexity of SD c,ε is O(ε -d 2 n -1 2 ) [Genevay 2019].
Stochastic Optimal Transport: going large scale. The regularization of linear OT allows deriving stochastic formulations that are useful in practice to handle large scale datasets. This setting was considered in [START_REF] Genevay | [END_REF], Seguy 2018] where authors rely on the dual formulation (2.14) or the semi-dual formulation (2.18) in the regularized case. More precisely for µ, ν ∈ P(R d ) and ε > 0 the regularized dual (resp. semi-dual) boils down to solve the following unconstrained maximization problems:

sup φ,ψ∈C(R d )×C(R d ) E x∼µ,y∼ν [F ε (φ(x), ψ(y))] (s-D) sup ψ∈C(R d ) E x∼µ [H ε (x, ψ)] (s-SD)
where F ε (φ(x), ψ(y)) = φ(x) + ψ(y) -εe φ(x)+ψ(y)-c(x,y) ε and H ε (x, ψ) = ´Y ψ(y)dν(y)ε log( ´e 1 ε (ψ(y)-c(x,y)) dν(y)) -ε when entropic regularization is used. Since the problem is recast in the form of an unconstrained maximization of an expectation, the idea is to use stochastic gradients tools such as Stochastic Gradient Descent (SGD), or Stochastic Averaged Gradient (SAG) to compute a solution of (s-D),(s-SD). When both µ = n i=1 a i δ xi ,ν = m j=1 b j δ yj are discrete (SGD) or (SAG) are directly applicable to maximize the following finite sums:

max α,β∈R n ×R m n,m i=1,j=1 F ε (α i , β j )a i b j (s-Ddis) max β∈R m n i=1 H ε (x i , β)a i (s-SDdis)
In [START_REF] Genevay | [END_REF]] authors propose to use (SAG) to compute (s-SDdis) which operates at each iteration by sampling a point x k from µ then to compute the gradient of H ε (x k , β) corresponding to that sample while keeping in memory a copy of past gradients. This approach costs O(n) per iteration due to the computation of the gradient and converges to a solution within O(k -1 ) iterations. In contrast in [Seguy 2018] propose to solve (s-Ddis) by applying an (SGD) on mini-batches of both µ, ν which comes with a O(p 2 ) cost per iteration where p is the mini-batch size and also converges in O(k -1 ). In the continuous setting the problem is infinite dimensional so that it can not be solved using (SGD) anymore. In [START_REF] Genevay | [END_REF] authors propose to represent the dual variables as kernel expansions while in [Seguy 2018] the dual variables are parametrized by a neural network. Another line of works rely on the unregularized problem and on the special case of W 1 . In this case the duality reads sup φ E x∼µ,y∼ν

[φ(x) -φ(y)] where the maximization is done over all 1-Lipschitz function φ. In [Arjovsky 2017] authors tackled this problem in the context of generative modelling. They parametrized µ, ν using a neural network and used the same (SGD)+mini-batch procedure resulting on a O(p) cost per iteration. Their approach however relies on a weight clipping of the (NN) weights in-between gradient updates to enforce the Lipschitz constraint which lead to optimization difficulties [Gulrajani 2017]. Note all approaches comes at the price of biasing the optimal coupling due to the mini-batch sampling. This effect was further analyzed in [START_REF] Fatras | [END_REF]].

Other formulations

Apart from entropic-regularized OT there are a lot of other methods for approximating OT. One of them relies on the closed-form expression of OT for probability distributions over the real line resulting on the so-called Sliced Wasserstein distance (SW) [START_REF] Rabin | [END_REF]]. Considering µ, ν ∈ P(R d ) the key idea is to randomly select lines in R d , to project the measures into these lines and to compute the resulting 1D-Wasserstein distance which can be done using simple sorts as seen previously. The sliced-Wasserstein distance is the average of all these 1D-Wasserstein distances over all drawn lines. More precisely:

Definition 2.1.7 (Sliced Wasserstein distance). Let λ d-1 be the uniform measure on S d-1 . For θ ∈ S d-1 we note P θ the projection on θ, i.e. P θ (x) = x, θ . Let µ, ν ∈ P(R d ) 2 . The Sliced Wasserstein distance between µ and ν is defined as:

SW p p (µ, ν) = ˆSd-1 W p p (P θ #µ, P θ #ν)dλ d-1 (θ) (2.32)
where the Wasserstein distance is defined with the standard Euclidean distance on R d .

SW enjoys several interesting properties. First SW 2 induces a similar topology than W 2 : it defines a distance on P p (R d ) [START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF]] that metrizes the weak convergence [START_REF] Nadjahi | [END_REF]] and which is equivalent to the Wasserstein distance for measures with compact supports [START_REF] Nadjahi | [END_REF][START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF]. Second it defines a positive definite kernel e -γSW 2 2 (µ,ν) for γ > 0 over the space of probability distributions that can be easily plugged into an SVM [START_REF] Kolouri | [END_REF]]. This contrasts with the Wasserstein distance W 2 which is not Hilbertian and consequently does not define a positive definite kernel (see Section 8.3 in [START_REF] Peyré | [END_REF]). In terms of sample complexity SW is known to be dimension independant [START_REF] Nadjahi | [END_REF]] such as O(n -1/2 ) when p = 2 [Lin 2020[START_REF] Nadjahi | [END_REF]] and better samples complexities can be found by projecting on subspaces of dimension k > 1 instead of random lines [Lin 2020, Paty 2019], yet raising tractability issues as the sorting trick is no longer valid. Moreover, as a side effect of its definition, SW is unable to find the correspondences between the samples of the distributions as it does not provide an optimal transport map π which is valuable for certain application such as domain adaptation [Courty 2017].

From a practical side estimating SW requires the calculation of an integral over the hypersphere which can be done using a simple Monte-Carlo scheme. Hence for discrete probability measures with n atoms the overall complexity of computing SW is O(Ln log(n)) where L is the number of projection directions on S d-1 . The quality of the Monte Carlo estimates is impacted by the number of projections as well as the variance of the evaluations of the Wasserstein distance has pointed out empirically in [Kolouri 2019a[START_REF] Deshpande | [END_REF]] and more formally in [START_REF] Nadjahi | [END_REF]]. The low computational complexity of SW makes it very attractive for a number of scenarii such as in deep learning for generative modeling [Deshpande 2018[START_REF] Deshpande | [END_REF],

for barycenter computation [START_REF] Bonneel | [END_REF]] or topological data analysis [Carrière 2017] to name a few. This "projection" idea was further developed and improved in several works which proposed to project on k-dimensional subspaces [Paty 2019], to use non-linear projection [Kolouri 2019a] or to generalize SW for the unbalanced setting [Bonneel 2019].

Many other interesting formulations can be derived from the original OT formulation. Since they are not considered in this manuscript we just give a brief overview here. In [START_REF] Ferradans | [END_REF]] author propose to regularize the linear OT with a quadratic term, resulting on a quadratic regularized OT. For regular grids [Solomon 2015] define a Wasserstein distance that can be computed efficiently in O(n 2 log(n)) using convolutions. Another line of works consider an unbalanced setting where the source probability measure is partially transferred to the target probability measure resulting on the unbalanced formulation [Chizat 2017]. A case of particular interest is when the target probability measure is discrete and the source continuous, namely the semi-discrete OT. It founds many applications in practice and can be tackled using Laguerre cells [Lévy 2018]. Finally the multi-marginal OT aims at solving an linear OT problem where there are many target/source probability measures and one optimal coupling for transporting them all [Nenna 2016].

Wasserstein barycenter

The Wasserstein distance is also an interesting tool in order to compute a notion of barycenter of probability distributions. In an Euclidean setting the traditional barycenter of points (x i ) i∈ [[m]] can be computed by solving inf x∈R d n i=1 λ i xx i 2 2 where λ i ≥ 0 and i λ i = 1. The barycenter vector is then given by x = n i=1 λ i x i . This can be generalized to arbitrary metric spaces (X , d) using the so-called Fréchet (or Karcher) mean [Karcher 2014]:

inf x∈X n i=1 λ i d(x, x i ) p
(2.33)

for p ∈ N. The problem (2.33) motivates the use of Wasserstein barycenter by considering the metric space (P p (Ω), W p ). Generally (2.33) is non-convex and difficult to solve for arbitrary metric space, however in the case of the Wasserstein distance the situation is somehow easier since it can be formulated as a convex problem for which existence can be proved and efficient numerical solvers exist. For a set of input probability measures (ν i ) i∈ [[n]] ∈ P(Ω) the Wasserstein barycenter reads as the following variational problem:

inf

µ∈P(Ω) n i=1 λ i T c (µ, ν i ).
(2.34)

The barycentric formulation finds many applications in machine learning such in Bayesian inference [Srivastava 2015], fairness [Gordaliza 2019], in image processing for texture synthesis and mixing [START_REF] Rabin | [END_REF] or in neuroimaging [START_REF] Gramfort | Fast Optimal Transport Averaging of Neuroimaging Data[END_REF]] to name a few. As proven in [START_REF] Agueh | Barycenters in the Wasserstein space[END_REF]] in the context of W 2 2

for Ω = R d this problem is convex and when one of the input measure has a density the barycenter is well-defined and unique. Even though there exist special cases (see Section 9.2 in [START_REF] Peyré | [END_REF]) in practice finding a solution in the general setting is difficult. In the following we detail one solution for the scenario where the input measures are discrete. More formally let (ν i ) n i=1 be discrete probability measures with weights b i ∈ Σ ni and that are supported on

Y i = (y i q ) q∈[[ni]] ∈ R ni×d for each i ∈ [[n]].
π, C XYi F = min a∈Σ k ,X∈R k×d ∀i∈[[n]],πi∈Π(a,bi) n i=1 λ i π i , C XYi F (2.35)
where C XYi ∈ R k×ni is the matrix defined by all pair to pair costs between the points of the barycenter and ν i , i.e. C XYi = (c(x p , y i q )) p,q∈ [[k]]× [[ni]] . In [START_REF] Cuturi | [END_REF]] author propose to solve (2.35) using Block Coordinate Descent (BCD) that alternates between minimizing w.r.t. a, X and π i while keeping others fixed:

(i) The minimization w.r.t. all π i with a, X fixed involves solving n OT problems which can be done using algorithms described in Section 2.1.4.

(ii) The minimization w.r.t. X with a, π i fixed can be performed in closed-form in the case Ω = R d and c(x, y) = xy 2 2 [Cuturi 2014, Equation 8]:

X = Diag 1 a n i=1 λ i π i Y i (2.36) (iii)
The minimization w.r.t. the weight a with X, π i fixed relies on the optimal dual variables of all OT sub-problems of step (i) and applies a projected subgradient minimization w.r.t. a as described in Algortihm 1 in [START_REF] Cuturi | [END_REF]].

These three steps are repeated until convergence of X and a. The major bottleneck of this approach is its computational complexity which is driven by the calculation of many OT problems. When the support X is fixed and by denoting C XYi = C i the problem reduces to:

min a∈Σ k ∀i∈[[n]],πi∈Π(a,bi) n i=1 λ i π i , C i (2.37)
which is an (LP) with kn 2 + n variables and 2N n constraints. Note that first order methods such as subgradient descent on the dual have been proposed in [Carlier 2015] to solve (2.37) but in general its scale forbids the use generic solvers even for medium scale problems. These remarks advocate for the use of entropic regularized OT to obtain fast and smooth approximations of the original barycenter problem as given by: min

a∈Σ k ∀i∈[[n]],πi∈Π(a,bi) n i=1 λ i π i , C i F -εH(π i ) (2.38)
The resulting problem is a smooth convex minimization problem, which can be tackled using gradient descent [START_REF] Cuturi | [END_REF]] or with descent method on the semi-dual [START_REF] Cuturi | [END_REF]]. Another possibility is to rewrite (2.38) as a the following weighted KL minimization problem [Benamou 2015]:

min (πi)i ∀i∈[[n]],π T i 1=bi π11=•••=πn1 n i=1 λ i εKL(π i |K i ) (2.39)
where K i = e -C i ε . In this formulation the barycenter a is encoded in the row marginals of all the couplings π i such that a = π 1 1 = • • • = π n 1. It is shown in [Benamou 2015] that this problem can also be solved using a generalized Sinkhorn algorithm which involves iterative projections. As such the entropic regularization is quite suited for the barycenter problem and was further analyzed for the general case of continuous probability measures in [Bigot 2019a, Bigot 2019b]. Note that other methods have been proposed which rely e.g. on the sliced Wasserstein formulation [START_REF] Bonneel | [END_REF]], unbalanced formulation [Chizat 2017] or on convolutions for geometric domains [Solomon 2015].

The case n = 2: McCann interpolant One special case deserves attention that is when n = 2 and in the case Ω = R d equipped with . 2 . This setting corresponds to the so-called McCann interpolant [McCann 1997] where one wants to find:

inf µ∈P(R d ) (1 -t)W 2 2 (µ, ν 1 ) + tW 2 2 (µ, ν 2 ) (2.40)
with t ∈ [0, 1] and ν 1 is regular with respect to the Lebesgue measure. Using Brenier theorem we know that there exists a unique push-forward such that T #ν 1 = ν 2 . In this case the barycenter is unique and obtained with µ t = ((1 -t)id + tT )#ν 1 . In practice when the probability measures ν 1 , ν 2 are discrete with respectively n and m atoms this interpolant can be computed by

µ t = m,n i=1,j=1 π * ij δ (1-t)xi+tyj
where π * is an optimal coupling between ν 1 , ν 2 .

The Gromov-Wasserstein problem

Problem statement

Despite its valuable properties the linear OT problem faces the challenging problem of probability measures whose supports lie in incomparable spaces, that is to say when X , Y are not part of a common ground metric space. For example when µ ∈ P(R 3 ), ν ∈ P(R 2 ) the definition of a meaningful cost c : R 3 ×R 2 → R + is not straightforward. In particular in this setting we can not define a distance between x, y ∈ R 3 × R 2 so that the Wasserstein distance can no longer be defined. Moreover the Wasserstein distance is not invariant to important families of invariants, such translations or rotations or more generally isometries which is an important flaw of linear OT for certain applications such as shape matching.

The Gromov-Wasserstein (GW) framework is an elegant remedy for this situation. It is built upon a quadratic Optimal Transport problem, as opposed to a linear one for the linear OT problem, and, informally its optimal value quantifies the metric distortion when transporting points from one space to another. This section aims at presenting the GW problem, its fundamental metric properties as well as numerical solvers. We refer the reader to [Sturm 2012, Memoli 2011, Chowdhury 2019a] for further readings.

We consider two polish spaces (X , d X ), (Y, d Y ). Let c X : X ×X → R and c Y : Y ×Y → R be continuous measurable functions and µ ∈ P(X ), ν ∈ P(Y) be probability measures on X , Y. The Gromov-Wasserstein (GW ) problem aims at finding: in depth in [Sturm 2012]. Another possibility is to consider triplets (X , c X , µ) where c X is a integrable function, this notion refers to measure networks and was studied in [Chowdhury 2019a].

GW p (c X , c Y , µ, ν) = inf π∈Π(µ,ν) ˆX ×Y ˆX ×Y c X (x, x ) -c Y (y,
The GW objective is constructed so that if an optimal coupling π maps x to y and x to y then the couple (x, x ) should be "as similar" in X as (y, y ) in Y. When c X , c Y are distances it implies that x, x are as close in X as y, y in Y. In this work we consider a general setting where c X , c Y are continuous and X , Y are Polish spaces and we will detail the two previous settings.

As for the linear OT problem the equation (2.41) always admits a solution. To show that we define L(x, x , y, y ) = c X (x, x ) -c Y (y, y ) . If Π(µ, ν) is compact and the functionnal π → ´´Ldπdπ is l.s.c. for the weak-convergence, Weierstrass theorem (see Memo 2.2.1) proves that the infimum will be attained at some optimal coupling. The first condition is a well-known result in OT theory provided that X , Y are Polish spaces [Santambrogio 2015, Theorem 1.7]. For the lower semi-continuity w.r.t. the weak-convergence we can show that it suffices that L be itself l.s.c. using the following lemma:

Lemma 2.2.1. Let Ω be a Polish space. If f : Ω × Ω → R + ∪ {+∞} is lower semi-continuous, then the functional J : P(Ω) → R ∪ {+∞} with J(µ) = ´´f (w, w )dµ(w)dµ(w ) is l.s.c. for the weak convergence of measures.

Proof. Since f is l.s.c. and bounded from below by 0 we can consider (f k ) k a sequence of continuous and bounded functions converging increasingly to f (see e.g [Santambrogio 2015]). By the monotone convergence

theorem J k (µ) → J(µ) def = sup k J k (µ) = sup k ´´f k dµdµ.
Moreover every J k is continuous for the weak convergence. Using theorem 2.8 [Billingsley 1999] on the Polish space Ω × Ω we know that if µ n converges weakly to µ then the product measure µ n ⊗ µ n converges weakly to µ ⊗ µ. In this way lim n→∞ J k (µ n ) = J k (µ) since f k are continuous and bounded. In particular every J k is l.s.c. We can conclude that J is l.s.c. as the supremum of l.s.c. functionals on the metric space of (P(Ω), δ) (see e.g. [Santambrogio 2015]).

Here we equipped P(Ω) with a metric δ as e.g.

δ(µ, ν) = ∞ k=1 2 -k | ´Ω f k dµ -´Ω f k dν| (see remark 5.11 in [Ambrosio 2005]).
Memo 2.2.1 (Weierstrass theorem). The Weierstrass theorem states that if f : X → R ∪ +∞ is l.s.c. and X is compact then there exists x * = inf x∈X f (x) (see box 1.1 in [Santambrogio 2015]).

Overall since L is l.s.c. due to the continuity of c X , c Y we can apply Lemma 2.2.1 with Ω = X × Y and conclude that π → ´´Ldπdπ is l.s.c. for the weak-convergence and by the means of Weierstrass theorem equation (2.41) always admits a minimizer.

Remark 2.2.1. As a consequence of our formulation the resulting GW cost main be infinite. A simple condition to remedy this possibility would be ´´

c X (x, x ) -c Y (y, y ) p d(µ ⊗ µ)(x, x )d(ν ⊗ ν)(y, y ) < ∞.
Since: GW for measures on Euclidean spaces One special case will be consider in Chapter 4, that is when X = R p , Y = R q and µ ∈ P(R p ), ν ∈ P(R q ) with p not necessarily equal to q. We can consider the GW problem with the standard Euclidean distances for c X , c Y on respectively R p , R q . This setting illustrates the invariance property of the GW problem w.r.t. rotations and translations. More precisely let O ∈ O(p) and x 0 ∈ R p associated with T (x) = Ox + x 0 . Then the GW problem is invariant by

c X (x, x ) -c Y (y, y ) p ≤ (|c X (x, x )| + |c Y (y, y )|) p * ≤ 2 p-1 (|c X (x, x )| p + |c Y (y, y )| p ) (2.42) if c X , c Y are p-integrable functions, i.e. c X ∈ L p (µ ⊗ µ) and c Y ∈ L p (ν ⊗ ν) then
T that is GW p p (T #µ, ν) = GW p p (µ, ν) (same applies for ν).
To see that we simply used for all x, x c X (T (x),

T (x )) = Ox + x 0 -Ox -x 0 2 = O(x -x ) 2 = x -x 2 since O ∈ O(p)
. This property will be generalized for any metric space by considering the notion of isometry and contrasts with the Wasserstein distance which is not invariant neither to translation or rotation of the support of one probability measure.

Example 2.2.1. As a first illustration of the GW problem we consider two discrete probability measures µ ∈ P(R 2 ), ν ∈ P(R 3 ) following respectively a spiral in R 2 , R 3 and composed of a mixture of Gaussian distributions. c X , c Y are defined by the Euclidean distances between the points. We compute an optimal coupling of the GW problem using the FW solver presented in Chapter 3 (see Section 2.2.3 for more details). The result is depicted in Figure 2.9. Memo 2.2.2 (Hölder's inequality). Let (X , µ) be a measurable space and (f, g) ∈ L p (µ) × L q (µ) with p, q > 0 verifying 1 p + 1 q = 1. The Hölder's inequality states:

ˆ|f g|dµ ≤ ( ˆ|f | p dµ) 1 p ( ˆ|g| q dµ) 1 q (2.43)
As a corollary for q ≥ 1 we have:

∀x, y ∈ R + , (x + y) q ≤ 2 q-1 (x q + y q ).
(2.44)

Indeed, if q > 1: (x + y) q = ( 1 2 q-1 ) 1 q x ( 1 2 q-1 ) 1 q + ( 1 2 q-1 ) 1 q y ( 1 2 q-1 ) 1 q q ≤ ( 1 2 q-1 ) 1 q-1 + ( 1 2 q-1 ) 1 q-1 q-1 x q 1 2 q-1 + y q 1 2 q-1 = x q 1 2 q-1 + y q 1 2 q-1 .
Last inequality is a consequence of Hölder's inequality. The result remains valid for q = 1.

Figure 2.9: GW problem between two discrete probability measures µ ∈ P(R 2 ), ν ∈ P(R 3 ). The optimal coupling π is depicted in dashed lines. It associates points so as to minimize the distortion between all pair-to-pair distances within the support of each measures.

Properties of GW

One of the main property of the GW problem is that it allows for comparing probability measures whose supports dwell in different, potentially non-related, spaces by defining a notion of equivalence of two probability distributions in this case. This is made possible thanks to the concepts of isometry and isomorphism.

Definition 2.2.1 (Isometry). Let (X , d X ) and (Y, d Y ) be two metric spaces. An isometry is a sujective map φ : X → Y that preserves the distances:

∀x, x ∈ X, d Y (φ(x), φ(x )) = d X (x, x ).
(2.45)

An isometry is necessarily bijective, since for φ

(x) = φ(x ) we have d Y (φ(x), φ(x )) = 0 = d X (x, x
) and hence x = x (in the same way φ -1 is also a isometry). When it exists, X and Y share the same "size" and any statement about X which can be expressed through its distance is transported to Y by the isometry φ.

Example 2.2.2. Let us consider the two following graphs whose discrete metric spaces are obtained as shortest path between the vertices (see corresponding graphs in

Figure 2.10).     x1 x2 x3 x4     ,     0 1 1 1 1 0 1 2 1 1 0 2 1 2 2 0     d X (xi,xj ) and     y1 y2 y3 y4     ,     0 1 1 1 1 0 2 2 1 2 0 1 1 2 1 0     d Y (yi,yj ) . x 2 x 3 x 4 x 1 y 2 y 3 y 1 y 4 Figure 2
.10: Two isometric metric spaces. Distances between the nodes are given by the shortest path, and the weight of each edge is equal to 1.

x

1 x 2 y 1 y 2 1 4 3 4 1 2 1 2
Figure 2.11: Two isometric but not isomorphic spaces.

These spaces are isometric since the surjective map φ such that φ(

x 1 ) = y 1 , φ(x 2 ) = y 3 , φ(x 3 ) = y 4 , φ(x 4 ) = y 2 verifies equation (2.45).
Another natural and straightforward example is two point clouds rotated from each other. More precisely if we consider (

x i ) i∈[[n]] , (y i ) i∈[[n]] where x i , y i ∈ R p × R p equipped with the Euclidean norm . 2 . Suppose that there exists a orthogonal matrix O ∈ O(p) such that y i = Ox i for all i ∈ [[n]
] (with a slight abuse of notations we identify the matrix with its linear application). Then for all (i, j) ∈ [[n]] 2 we have:

y i -y j 2 = Ox i -Ox j 2 = O(x i -x j ) 2 = x i -x j 2 (2.46) since O ∈ O(p) so that X = (x i ) i∈[[n]] and Y = (y i ) i∈[[n]] are isometric.
This notion can be enriched in order to take into account the measures, which results in the notion of strong isomorphism:

Definition 2.2.2 (Strong isomorphism). Let (X , d X ), (Y, d Y ) be Polish spaces and µ ∈ P(X ), ν ∈ P(Y). We say that (X , d X , µ) is strongly isomorphic to (Y, d Y , ν) if there exists a bijection φ : supp(µ) → supp(ν) such that: i φ is an isometry, i.e. d Y (φ(x), φ(x )) = d X (x, x ) for x, x ∈ supp(µ) 2 ii φ pushes µ forward to ν, i.e. φ#µ = ν
When it is clear from the context we will simply say that µ is strongly isomorphic to ν when previous conditions are satisfied.

Example 2.2.3. Let us consider two mm-spaces

(X = {x 1 , x 2 }, d X = {1}, µ = { 1 2 , 1 2 }) and (Y = {y 1 , y 2 }, d Y = {1}, ν = { 1 4 , 3 4
}) as depicted in Figure 2.11. These spaces are isometric but not isomorphic as there exists no measure preserving map which pushes µ forward to ν Another notion of isomorphism deserves attention especially when c X , c Y are not distances. In this case we will consider the following weak isomorphism property:

Definition 2.2.3 (Weak isomorphism). Let X , Y be Polish spaces and µ ∈ P(X ), ν ∈ P(Y). We say that (X , c X , µ) is weakly isomorphic to (Y, c Y , ν) if there exists (Z, c Z , m), with supp(m) = Z and maps φ 0 : Z → X ,φ 1 : Z → Y such that: i c Z (z, z ) = c X (φ 0 (x), φ 0 (x )) = c Y (φ 1 (x), φ 1 (x )) for z, z ∈ Z 2 ii φ 0 #m = µ and φ 1 #m = ν
When it is clear from the context we will simply say that µ is weakly isomorphic to ν when previous conditions are satisfied.

The weak isomorphism brings to light a kind of "tripod structure" in which the isomorphism is defined trough a third space Z. In fact both notions are equivalent when c X , c Y are distances as stated in the next proposition [Sturm 2012, lemma 1.10]:

Proposition 2.2.1. The spaces (X , d X , µ) and (Y, d Y , ν) are strongly isomorphic if and only if (X , d X , µ) and (Y, d Y , ν) are weakly isomorphic.
However the weak-isomorphism property has its own interest when working with arbitrary similarity measures c X , c Y . The following theorem is fundamental for GW and aims to unify the metric properties of GW given in [Sturm 2012, Chowdhury 2019a]. It proves that GW defines a metric w.r.t. the isomorphism notions:

Theorem 2.2.1 (Metric properties of GW ). In the following (X , d X ), (Y, d Y ) are Polish spaces and µ ∈ P(X ), ν ∈ P(Y) i GW p is symmetric, positive and satisfies the triangle inequality. More precisely for (X , c X , µ),

(Y, c Y , ν), (Z, c Z , m) we have: GW p (c X , c Y , µ, ν) ≤ GW p (c X , c Z , µ, m) + GW p (c Z , c Y , m, ν) (2.47) ii GW p (d X , d Y , µ, ν) = 0 if and only if (X , d X , µ) and (Y, d Y , ν) are strongly isomorphic. iii GW p (c X , c Y , µ, ν) = 0 if and only if (X , c X , µ) and (Y, c Y , ν) are weakly isomorphic.
iv More generally, for any q ≥ 1, GW p (d q X , d q Y , µ, ν) = 0 if and only if (X , d X , µ) and (Y, d Y , ν) are strongly isomorphic.

Proof. For the first point (i) positiveness is straightforward. For the triangle inequality and symmetry see [Chowdhury 2019a, Theorem 16]. For (ii) see [Sturm 2012, Lemma 1.10] and for (iii) see [Chowdhury 2019a, Theorem 18]. Note in the proof that for (iii) the result is still valid even if c X , c Y are not p-integrable. For (iv) see [Sturm 2012, Lemma 9.2].

This theorem can endow the space of all spaces of the form (X , c X , µ) with a distance defined by GW which, however, requires the finiteness of GW . More precisely: Definition 2.2.4. Let X be a Polish space, µ ∈ P(X ) and c X : X → R be measurable. We define the size size p of X , given c X and µ by size p (X , c X , µ) = ( ´cX (x, x ) p dµ(x)dµ(x ))

1 p
We define X p be the space of all metric measure spaces with finite L p -size, i.e X p = {(X , d X , µ)|size p (X , d X , µ) < +∞} where (X , d X ) is a Polish space and µ ∈ P(X ).

We define also N p be the space of all network measure spaces [Chowdhury 2019a] with finite L psize, i.e. N p = {(X , c X , µ)|size p (X , c X , µ) < +∞} where X is a Polish space, µ ∈ P(X ) and c X a continuous measurable function.

The function size p quantifies somehow an average diameter of X given a probability measure and a function c X . Using this notion and Theorem 2.2.1 we now state the main theorem about the metric properties of GW : Theorem 2.2.2 (GW is a distance). GW p is a distance on X p quotiented by the strong isomorphisms. GW p is a distance on N p quotiented by the weak isomorphisms.

This theorem has a lot of implications. It endows the space of all metric (network) measure spaces with a topology, a geometric structure, induced by Gromov-Wasserstein and, as such, allows the use of a wide family of geometric tools and a notion of convergence of metric measure spaces. Moreover it indicates that GW is well suited for comparing objects with respect to a large class of invariants that are for instance rotations, translations or permutations. This property is important e.g. for shape comparison where the orientation of a shape does not define its nature or for graphs where any permutation of the nodes result in the same graph. It is sometimes valuable to have a notion of distance which is insensitive to these transformations so as to focus properly on what matters rather than to encode the invariance (see Remark 2.2.2). Finally if GW vanishes it implies necessarily that the objects are isomorphic which is interesting for detecting such cases.

Note also that GW is deeply connected to the Gromov-Hausdorff distance [START_REF] Gromov | [END_REF]] that aims at measuring how far are (X , d X ) and (Y, d Y ) from being isometric and can be used for studying the convergence of metric spaces [START_REF] Burago | [END_REF]]. However computing this distance results in a highly non-convex optimization problem whose global solution is untractable. As shown in [Memoli 2011] the introduction of measures turns out to "smooth" the definition of the Gromov-Hausdorff distance and results in the Gromov-Wasserstein distance.

Remark 2.2.2 (Implicit or explicit encoding of the invariances). Let µ, ν ∈ P(R d ) × P(R d ) and c : [START_REF] Alvarez-Melis | [END_REF] authors propose the following OT problem:

R d × R d → R a cost.
InvOT (µ, ν) = min π∈Π(µ,ν) min f ∈F ˆc(x, f (y))dπ(x, y) (2.48)
where F is a class of functions from R d to R d aiming at encoding a global transformation of the features. For example F can be defined as O(d) the set of orthogonal transformations or any linear transformation with bounded Schatten norm (see [START_REF] Alvarez-Melis | [END_REF] or Chapter 4 for more details). When considering

F = O(d)
the resulting InvOT becomes invariant by rotation of the support of the target measure. It could be interesting in a setting where one wants to match two distributions modulo a rotation as e.g. in unsupervised word translation where word embedding algorithms are known to produce vectors intrinsically invariant to angle [START_REF] Alvarez-Melis | [END_REF][START_REF] Grave | Unsupervised Alignment of Embeddings with Wasserstein Procrustes[END_REF]. This approach can be put into perspective with the GW distance as, in the GW case, one makes the implicit assumption, or prior, that the invariances are the isometric transformations of the data, whereas in the InvOT approach one makes the prior that we know somehow which class of invariances is of interest for the problem and we encode it into the loss. Both approach are relevant and related (see Chapter 4): if we have another prior than isometric transformations it is maybe more suited to encode it directly via F and if we know that isometries are relevant we can directly build upon GW .

Geodesics and GW interpolation

The space of all mm-spaces X p endow with GW has also a nice geodesic structure which is important in order to derive dynamic formulations and gradient flows [START_REF][END_REF]]. Informally in X p we can connect any two points (that are mm-spaces) with a curve that somehow represents the shortest path connecting these points. More precisely given two mm-spaces

(X , d X , µ), (Y, d Y , ν) the curve t ∈ [0, 1] → (X × Y, d t , π * )
where π * is an optimal coupling of the GW problem between µ and ν and:

∀(x, y), (x , y ) ∈ (X × Y) 2 , d t ((x, y), (x , y )) = (1 -t)d X (x, x ) + td Y (y, y ) (2.49)
is a geodesic [Sturm 2012]. However computing this geodesic is often intractable in practice since it implies the calculation of the cartesian product X × Y. One can rely instead on the barycenter formulation defined in Section 2.2.4.

Solving GW

In this section we describe some numerical solutions to the GW problem. In the following µ

= n i=1 a i δ xi ∈ P(X ), ν = m j=1 b j δ yj ∈ P(Y) are discrete probability measures over respectively (X , d X ), (Y, d Y ). We note also C 1 , C 2 the matrices of pair-to-pair distances inside each space, i.e. ∀(i, k) ∈ [[n]] 2 , C 1 (i, k) = d X (x i , x k ) and ∀(j, l) ∈ [[m]] 2 , C 2 (j, l) = d Y (y j , y l ).
The GW problem aims at solving:

GW p p (C 1 , C 2 , a, b) = min π∈Π(a,b) i,j,k,l |C 1 (i, k) -C 2 (j, l)| p π i,j π k,l = min π∈Π(a,b) L(C 1 , C 2 ) p ⊗ π, π F (2.50)
where we define

L(C 1 , C 2 ) as the tensor L(C 1 , C 2 ) = (|C 1 (i, k) -C 2 (j, l)|) i,j,k,l and ⊗ is the the tensor- matrix multiplication, i.e. for a tensor L = (L i,j,k,l ), L ⊗ π is the matrix k,l L i,j,k,l π k,l i,j
.

The optimization problem (2.50) is a non-convex Quadratic Program (QP) which is NP-hard in general [START_REF] Loiola | [END_REF]] and notoriously hard to approximate. When p = 2, i.e. when L = |.| 2 equation (2.50) can be recast as: [START_REF] Peyré | [END_REF]). In standard QP form this problem reads also:

min π∈Π(a,b) tr(c C1,C2 π T ) -2tr(C 1 πC 2 π T ) (2.51) where c C1,C2 = (C 1 ) 2 a1 T m + 1 n b T (C 2 ) 2 (see Proposition 1 in
min π∈Π(a,b) c T x(π) + 1 2 x(π) T Qx(π) (2.52) where x(π) = vec(π), c = vec(c C1,C2 ) and Q = -4C 1 ⊗ K C 2 with ⊗ K the Kronecker product of matrices defined for two arbitrary matrices A ∈ R n×m , B p×q as A ⊗ K B ∈ R np×mq with A ⊗ K B = (A i,j B) i,j (see Memo 2.2.3). Equation (2.52
) is a non-convex QP as the Hessian Q is not positive semi-definite in general (its eigenvalues are the products of the eigeinvalues of C 1 , C 2 ).

Relations with Quadratic Assignment Problem and Graph Matching

The GW problem is very related to the so called Quadratic Assignment Problem (QAP). This problem was first introduced by Koopmans and Beckmann [Koopmans 1957] to model a plant location problem and plays today many roles in optimization. Given two matrices A = (a i,j

) (i,j)∈[[n]] 2 and B = (b i,j ) (i,j)∈[[n]
] 2 the standard form for the QAP reads:

min σ∈Sn n i=1,j=1 a σ(i)σ(j) b ij (2.53)
The QAP can be understood as a facility location problem: given n facilities and n locations, one wants to assign each facility to a location with a ij the flow of material moving from facility i to facility j and b ij the distance from facility i to facility j. In this context the cost of simultaneously locating facility σ(i) to location i and facility σ(j) to location j is a σ(i)σ(j) b ij . In this model one wants to find the assignment that minimizes the overall cost of locating each facility. This problem was considered for example in [Elshafei 1977] for locating hospital departments so as to minimize the total distance traveled by patients but it also covers a large variety of applications such as scheduling [START_REF] Geoffrion | [END_REF]], parallel and distributed computing [Bokhari 1981] or balancing of turbine runners [Laporte 1988]. For a comprehensive survey on this topic we refer to [START_REF] Çela | The quadratic assignment problem: Theory and algorithms[END_REF][Çela , Loiola 2007]]. Unfortunately the QAP is NP-Hard in general and only few special cases are known to be solvable in polynomial time. That is the case for example when matrices A and B have simple known structures, such as a diagonal structure and Toeplitz or separability properties such as a i,j = α i α j [Çela 2018, Çela 2011, Çela 2015]). The question of finding a polynomial time algorithm that solves the QAP when A and B satisfy the Monge and the anti-Monge properties are, to the best of our knowledge, still open [Çela 2013].

The QAP is intrinsically linked with the graph matching problem whose literature is also extensive (see e.g. [Berg 2005[START_REF] Lyzinski | [END_REF][START_REF] Zaslavskiy | [END_REF][START_REF] Maron | [END_REF][START_REF] Caetano | [END_REF]). The graph matching problem refers to optimization problems where the goal is to match edge affinities of two graphs that are represented by symmetric matrices A = (a i,j

) (i,j)∈[[n]] 2 and B = (b i,j ) (i,j)∈[[n]] 2 .
A common approach for these types of problem is to attempt to solve:

min X∈Πn AX -XB 2 F (2.54) where Π n is the set of permutation matrices, i.e. X = (x ij ) (i,j)∈[[n]] 2 ∈ Π n if x ij ∈ {0, 1} and n i=1 x ij = n j=1 x ij = 1. By noticing that AX -XB 2 F = AX 2 F + XB 2 F -2tr(AXBX T ) and that AX 2 F = tr(X T A T AX) = tr(XX T A T A) = tr(A T A) = A 2 F the problem (2.54) is equivalent to: min X∈Πn -tr(AXBX T ) (2.55)
As such the graph matching problem is a QAP and consequently is NP-Hard in general. A way of finding a approximate solution is to consider a relaxation of the constraints by replacing Π n by its convexhull, namely the set of doubly stochastic matrices [Dym 2017[START_REF] Bernard | [END_REF][START_REF] Schellewald | [END_REF]).

DS = {X ∈ R n×n |X1 n = 1 n , X T 1 n = 1 n , X ≥ 0} (see
The previous discussion can relate the GW problem with the graph matching one. Indeed when we consider two discrete probability measures with the same number of atoms and with uniform weights (i.e. a = b = 1 n /n) then GW is equivalent to the relaxation of the graph matching problem with affinity matrices C 1 , C 2 . To see this it suffices to notice that, in this case, Π(a, b) is the set of doubly stochastic matrices (modulo a factor n which has no impact). We will see in Chapter 4 that the QAP and graph matching point of views can be quite enlightening for deriving properties of the GW distance.

Entropic regularization

In [START_REF] Peyré | [END_REF], Solomon 2016] authors propose to solve (2.50) using entropic regularization which results in the following optimization problem:

min π∈Π(a,b) L(C 1 , C 2 ) p ⊗ π, π F -εH(π).
(2.56) This is a non-convex optimization which was tackled using projected gradient descent using the geometry of the KL divergence for both the gradient step and the projection step [START_REF] Peyré | [END_REF]]. More precisely by denoting E ε (π), the loss in (2.56) the iterations of this algorithm read:

π ← Proj KL Π(a,b) (π * e -τ ∇Eε(π) ) (2.57) Algorithm 3 Solving Entropic-regularized GW 1: a, b, C 1 , C 2 , ε > 0 2: Initalize π = ab T , u (0) , v (0) = 1 3: for i = 1, . . . , n 1 it do 4:
Compute the gradient of the GW loss

C = 2L(C 1 , C 2 ) p ⊗ π 5: Set K = exp(-C ε ) 6:
// Do Sinkhorn-Knopp Algorithm 2 7:

for i = 1, . . . , n 2 it do 8: u (i) = a K v (i-1) // Update left scaling 9: v (i) = b Ku (i-1) //
return π * = Diag(u)K Diag(v)
where τ > 0 is the step size of the gradient descent and * denotes elementwise (Hadamard) matrix multiplication. The projection operator is defined as the result of the minimization problem:

Proj KL Π(a,b) (K) = arg min π∈Π(a,b) KL(π|K) (2.58)
As shown in Section 2.1.4 the projection can be solved using the efficient Sinkhorn-Knopp Algorithm (see Algorithm 2). The gradient ∇E ε (π) can be calculated as 2L(C 1 , C 2 ) p ⊗ π + ε log(π), and, as noted in [START_REF] Peyré | [END_REF]], the special case where the step size τ is defined as τ = 1 ε the iterations (2.57) boil down to solving an entropic regularized linear OT problem with ground cost 2L(C 1 , C 2 ) p ⊗ π. Overall the procedure to solve GW with entropic regularization is a projected gradient procedure where the projection step can be solved using an entropic linear OT as described in Algorithm 3. Note that, as for the linear OT case, the resulting optimal coupling is not sparse since the entropy term tends to blur the solution. Moreover one usually wants to have a gradient step τ which is not "too big" in order to ensure the convergence of the algorithm. Since the gradient step τ is inversely proportional to the regularization parameter ε this comes at the price of blurring the resulting optimal solution. In practice, there is a trade-off between regularization and convergence of the algorithm (see [START_REF] Peyré | [END_REF]] and experiments of Chapter 4).

Memo 2.2.3 (Kronecker product and vec operator).

Let A ∈ R n×m , B p×q be two arbitrary matrices.

The vec operator converts the matrix into a column vector. It is defined as vec

(A) ∈ R nm×1 = (A 1,1 , . . . , A n,1 , A 1,2 , . . . , A n,2 , . . . , A n,m ) T .

The Kornecker product of two matrices result in a block

matrix A ⊗ K B ∈ R np×mq defined by A ⊗ K B = (A i,j B) i,j
. These two operators satisfy the following properties (see [Petersen 2012]):

• vec(AB) = (I ⊗ K A)vec(B) = (B T ⊗ K I)vec(A) • (A ⊗ K B) T = (A T ⊗ K B T ) • (A ⊗ K B)(B ⊗ K A) = (AB ⊗ K BA) • tr(A T B) = vec(A) T vec(B)
When p = 2 the previous problem reduces to the softassign quadratic assignement problem. In the special case where the problem is convex the convergence of the previous scheme was analyzed in [Rangarajan 1997b[START_REF] Rangarajan | [END_REF]] but, with arbitrary matrices C 1 , C 2 there is no known results to the best of our knowledge.

Computing a lower-bound Originally (2.50) was tackled by computing a lower bound (called the TLB) in [Memoli 2011]. More precisely, authors propose to solve the following problem:

min π 1 ∈Π(a,b) k,l   min π 2 ∈Π(a,b) i,j |C 1 (i, k) -C 2 (j, l)| p π 2 i,j   π 1 k,l (2.59)
This problem is actually a "Wasserstein of Wasserstein distances": it is equivalent to solve an OT problem which ground cost results itself on OT problems between the 1D empirical distributions of the lines of

C 1 , C 2 . More precisely by considering µ k = i a i δ C1(i,k) ∈ P(R) and ν l = j b j δ C2(j,l) ∈ P(R) (2.59) is equivalent to: min π∈Π(a,b) k,l W p p (µ k , ν l )π k,l (2.60)
where the Wasserstein distance is computed using |.| as ground cost. The advantage of this formulation is that (2.60) only involves linear OT problems that can be solved using tools presented in Section 2.1.4. This idea is based on the local distribution of distances and was also successfully applied in computer graphics for 3D shape comparison [START_REF] Gelfand | [END_REF], Memoli 2011].

Computational complexities

One major bottleneck for computing GW is first the calculation of the big tensor (|C [START_REF] Peyré | [END_REF] show that, in the case p = 2, one can rely on the separability of L which results in a O(n 2 m + m 2 n) complexity. A second bottleneck is the complexity of finding an optimal solution which is driven by the algorithmic method. The convergence of Algorithm 3 for the entropic-regularized GW problem is still not well-understood and slow in practice as shown for example in Chapter 4. We will present in Chapter 3 an algorithm based on Frank-Wolfe (FW) to find a sparse local optimal solution with a cubic complexity. Using the FW properties we will show that this algorithm also converges to a local stationary point with a O( 1 √ t ) rate. Regarding the lower bound computation one can rely on the sorting strategies for 1D distributions to compute an inner Wasserstein distance

1 (i, k) -C 2 (j, l)| p ) i,j,k,l which is O(n 2 m 2 ) is general. By noticing that it suffices instead to compute L(C 1 , C 2 ) p ⊗ π authors in
W p p (µ i , ν j ) in O(n log(n) + m log(m)) hence a O(n 2 m log(n) + m 2 n log(m))
) complexity for all pairs. Then finding a solution π has the same complexity as computing a linear OT problem and one can rely on entropic regularization to reach a quadratic time complexity. More recently, in [START_REF] Sato | [END_REF]] authors propose to fix the outer optimal transport plan of the lower bound to ab T which results in a divergence that can be computed in O((n 2 + m 2 ) log(nm)) using a sweep line strategy.

Illustration of previous solvers

In order to give a simple illustration of the different solvers presented above we consider two unlabeled graphs with the same number of nodes (n = m = 20) and with 4 communities. Each node of the graph lies in the implicit metric space defined by the shortest-path distance inside the graph so that C 1 is the shortest path distance matrix between each node (same for C 2 ). We consider uniform weights, i.e. a = b = 1 n 1 n and p = 2 for GW. We solve the GW problem by relying on (1) The FW algorithm defined in Chapter 3, (2) The entropic regularized GW problem with ε = 8e -4, (3) The lower bound approach TLB. The behavior of the three approaches is depicted in Figure 2.12 Figure 2.12: Illustration of the optimal coupling for the GW problem between two graphs described by their shortest-path. (left) FW solver of Chapter 3 (middle) Entropic regularized GW (Algorithm 3) (right) Lower bound approach TLB. The optimal coupling π * is depicted in dashed lines. The darker the stronger.

Gromov-Wasserstein Barycenter

In the same vein as the Wasserstein barycenter we can build upon the Gromov-Wasserstein distance a notion of barycenter. This setting was first tackled in [START_REF] Peyré | [END_REF]] where authors consider the problem of computing the barycenter of a family of discrete probability measures over different metric spaces w.r.t. the Gromov-Wasserstein geometry. More formally let (C i , b i ) i∈ [[n]] be this family where C i is a arbitrary matrix and b i is a probability vector. C i can be chosen to be a distance matrix or more generally any similarity matrix, such as a kernel matrix encoding a notion of similarity between the points inside each distribution as in [Chowdhury 2019a]. When the weights of the barycenter are given and fixed to a ∈ Σ k with k ∈ N * the GW barycenter problem aims at finding:

min C∈R k×k n i=1 λ i GW p p (C, C i , a, b i ) = min C∈R k×k ∀i∈[[n]],πi∈Π(a,bi) n i=1 λ i L(C, C i ) p ⊗ π, π F (2.61)
with λ i ≥ 0 and n i=1 λ i = 1. In [START_REF] Peyré | [END_REF]] authors consider this problem based on the entropic regularized version of GW. They propose to solve (2.61) by relying on a BCD procedure which alternates between solving n GW problems with C fixed and by finding C with all π i fixed. The latter is given in closed-form when p = 2 by [Peyré 2016, Equation 14]:

C = 1 aa T n i=1 λ i π T i C i π i (2.62)
where the division is made element-wise. Solving the n GW problems can be done using Algorithm 3 as in [START_REF] Peyré | [END_REF]] or with the GW or the lower bound approach as presented in the previous section. This approach was further generalized in [Chowdhury 2019b] where authors leverage the Riemannian geometry of Gromov-Wasserstein space to treat e.g. the case where C i are not necessarily symmetric.

Illustration

To illustrate the GW barycenter we consider a simple dataset consisting in 4 2D-shapes from the apple class of the MPEG-7 computer vision database. Each shape is associated with a discrete probability measure with uniform weights. The matrices C i are simply the Euclidean distances between the points in each shape. We compute the barycenter using the previous discussion with both the FW solver for solving the GW problems and using the entropic regularized GW (ε = 1e -3) with Algorithm 3. We compute a MDS on the C matrix obtained by the barycenter procedure to recover 2D points. Results are depicted in Figure 2.13.

Applications of GW

The GW problem is well suited for comparing heterogeneous data while being invariant to the isometries of the data. As such it has first received attention for shapes comparison [Memoli 2011, Solomon 2016] or in computer vision [Schmitzer 2013] where it is often valuable to compare objects without any assumption on their orientation. GW was further exploited to handle unstructured geometric data such as point clouds or meshes in [Ezuz 2017] where authors use GW to learn regular 2D grids that faithfully represent the 3D meshes while being applicable for standards CNN architectures.

More recently GW has been the subject of much attention in the graph community as a graph matching tool [Xu 2019b, Xu 2019a[START_REF] Fey | Deep Graph Matching Consensus[END_REF] or for graphs representation [START_REF] Kwon | [END_REF]] (see Chapter 4 for more details). It also proves its usefulness in cellular biology thanks to its ability for aligning heterogeneous types of single-cell measurements [START_REF] Demetci | [END_REF]]. Closer to the machine learning community GW has been applied in Domain Adaptation (DA) in the complex settings of Unsupervised DA [START_REF] Xia | [END_REF]] and Heterogeneous (DA) [START_REF][END_REF]] (see Chapter 5 for more details), in generative modeling on incomparable spaces i.e. when the data generated do not share the same Euclidean space as the source data [Bunne 2019], or for cross-lingual correspondences of word embeddings [Alvarez-Melis 2018a].

Conclusion

The optimal transport framework is a powerful tool for comparing probability distributions by relying on both Wasserstein and Gromov-Wasserstein distances. The linear OT theory is a well-studied problem both from the theoretical and numerical side. In contrast a lot of questions remain unanswered for the Gromov-Wasserstein theory and interesting connections with the Wasserstein theory can be made. From a theoretical perspective there is no known result yet about the regularity of the optimal transport plans.

As for the practical side the GW problem remains very costly to solve and difficult to approximate. The purpose of the next chapters is, inter alia, to work towards these directions and to address the following questions:

i Is there any favorable cases where Optimal Transport plans of the GW are supported on a Monge map as in the Brenier theorem? (see Chapter 4)

ii Can we find some special cases where GW admits a closed-form solution such as the 1D or the Gaussian cases? If so, can we derive scalable and useful formulations from these cases? (see Chapter 4)

iii Is the GW framework suited for the structured data setting? How does it behave for concrete structured data problems such as graph applications? (see Chapter 3)

iv More generally can we derive other formulations than GW that are maybe more useful for data on incomparable spaces? (see Chapter 5)

Chapter 3

Optimal Transport for structured data

Perhaps as you went along you did learn something.

-Ernest Hemingway, The Sun Also Rises

Summary of the contributions

This chapter is based on the papers [Vayer 2019a, Vayer 2020b] and considers the problem of computing distances between structured objects such as undirected graphs, seen as probability distributions in a specific metric space.

We consider a new transportation distance (i.e. that minimizes a total cost of transporting probability masses) that unveils the geometric nature of the structured objects space. Unlike Wasserstein or Gromov-Wasserstein metrics that focus solely and respectively on features (by considering a metric in the feature space) or structure (by seeing structure as a metric space), our new distance exploits jointly both information, and is consequently called Fused Gromov-Wasserstein (FGW). After discussing its properties and computational aspects, we show results on a graph classification task, where our method outperforms both graph kernels and deep graph convolutional networks. Exploiting further on the metric properties of FGW, interesting geometric objects such as Fréchet means or barycenters of graphs are illustrated and discussed in a clustering context. We provide in a second part the mathematical framework for this distance in the continuous setting, prove its metric, geodesic and interpolation properties and provide a concentration result for the convergence of finite samples.

Introduction

There is a longstanding line of research on learning from structured data, i.e. objects that are a combination of a feature and structural information (see for example [START_REF] Bakir | [END_REF][START_REF] Battaglia | [END_REF]). As immediate instances, graph data are usually ensembles of nodes with attributes (typically R d vectors) linked by some specific relation. Notable examples are found in chemical compounds or molecules modeling [START_REF] Kriege | [END_REF]], brain connectivity [Ktena 2017], or social networks [START_REF] Yanardag | [END_REF]]. This generic family of objects also encompasses time series [Cuturi 2017], trees [START_REF] William | Optimal algorithms for comparing trees with labeled leaves[END_REF] or even images [START_REF] Bach | [END_REF]].

Being able to leverage on both feature and structural information in a learning task is a tedious task, that requires the association in some ways of those two pieces of information in order to capture the similarity between the structured data. Several kernels have been designed to perform this task [START_REF] Shervashidze | [END_REF][START_REF] Vishwanathan | [END_REF]]. As a good representative of those methods, the Weisfeiler-Lehman kernel [START_REF] Vishwanathan | [END_REF]] captures in each node a notion of vicinity by aggregating, in the sense of the topology of the graph, the surrounding features. Recent advances in graph convolutional networks [Bronstein 2017[START_REF] Kipf | [END_REF],Defferrard 2016[START_REF] Wu | [END_REF] allows learning end-to-end the best combination of features by relying on parametric convolutions on the graph, i.e. learnable linear combinations of features. In the end, and in order to compare two graphs that might have different number of nodes and connections, those two categories of methods build a new representation for every graph that shares the same space, and that is amenable to classification.

Contrasting with those previous methods, we suggest in this chapter to see graphs as probability distributions, embedded in a specific metric space. We propose to define a specific notion of distance between these probability distributions, that can be used in most of the classical machine learning approaches. Beyond its mathematical properties, disposing of a distance between structured data, provided it is meaningful, is desirable in many ways: i) it can then be plugged into distance-based machine learning algorithms such as k-nn or t-SNE ii) its quality is not dependent on the learning set size, and iii) it allows considering interesting quantities such as geodesic interpolation or barycenters.

Yet, defining this distance is not a trivial task. While features can always be compared using a standard metric, such as Euclidean distances, comparing structures requires a notion of similarity which can be found via the notion of isometry, since the graph nodes are not ordered (we define later on which cases two graphs are considered identical). We use the notion of transportation distance to compare two graphs represented as probability distributions. Optimal transport have inspired a number of recent breakthroughs in machine learning (e.g. [START_REF] Huang | [END_REF], Courty 2017, Arjovsky 2017]) because of its capacity to compare empirical distributions, and also the recent advances in solving the underlying problem [START_REF] Peyré | [END_REF]]. Yet, the natural formulation of OT cannot leverage the structural information of objects since it only relies on a cost function that compares their feature representations.

However, some modifications over OT formulation have been proposed in order to compare structural information of objects. Following the pioneering work by Mémoli [Memoli 2011], Peyré et al. [START_REF] Peyré | [END_REF]] propose a way of comparing two distance matrices that can be seen as representations of some objects' structures. They use the Gromov-Wasserstein distance (see Chapter 2) capable of comparing two distributions even if they do not lie in the same ground space and apply it to compute barycenter of molecular shapes. Even though this approach has wide applications, it only encodes the intrinsic structural information in the transportation problem. To the best of our knowledge, the problem of including both structural and feature information in a unified OT formulation remains largely under-addressed.

OT distances that include both features and structures. Recent approaches tend to incorporate some structure information as a regularization of the OT problem. For example in [Alvarez-Melis 2018b] and [Courty 2017], authors constrain transport maps to favor some assignments in certain groups. These approaches require a known and simple structure such as class clusters to work but do not generalize well to more general structural information. In their work [START_REF] Thorpe | [END_REF]], propose an OT distance that combines both a Lagrangian formulation of a signal and its temporal structural information. They define a metric, called Transportation L p distance, that can be seen as a distance over the coupled space of time and feature. They apply it for signal analysis and show that combining both structure and feature tends to better capture the signal information. Yet, for their approach to work, the structure and feature information should lie in the same ambiant space, which is not a valid assumption for more general problems such as similarity between graphs. In [START_REF] Nikolentzos | Matching Node Embeddings for Graph Similarity[END_REF]], authors propose a graph similarity measure for discrete labeled graph with OT. Using the eigenvector decomposition of the adjency matrix, which captures graph connectivities, nodes of a graph are first embedded in a new space, then a ground metric based on the distance in both this embedding and the labels is used to compute a Wasserstein distance serving as a graph similarity measure.

Contributions. After defining structured data as discrete probability measures (Section 3.2), we propose a new framework, namely F GW , capable of taking into account both structure and feature information into the optimal transport problem. The framework can compare any usual structured machine learning data even if the feature and structure information dwell in spaces of different dimensions, allowing the comparison of undirected labeled graphs. It is based on a distance that embeds a trade-off parameter which allows balancing the importance of the features and the structure. We provide a conditional-gradient algorithm for computing F GW (Section 3.3), and we evaluate it (Section 3.4) on both synthetic and real-world graph datasets on various tasks. We show that F GW is particularly useful for both supervised and unsupervised learning on graphs.

Among the contributions of this Chapter the numerical solution presented in Section 3.3 can also be used to compute the Gromov-Wasserstein distance. To the best of our knowledge this is the first optimization scheme for GW that does not require entropic regularization and which results in a sparse optimal solution.

We also define and illustrate a notion of labeled graph barycenters using F GW (Section 3.3.2), based on the Fréchet mean, and apply it for clustering and coarsening of graphs problems.

In a last part (Section 3.5) we generalize the definition of structured data to compact metric spaces. We } } } Figure 3.1: (left) Labeled graph with (ai)i its feature information, (xi)i its structure information and a probability vector (hi)i that measures the relative importance of the vertices. (right) Associated structured data which is entirely described by a fully supported probability measure µ over the product space of feature and structure, with marginals µX and µA on the structure and the features respectively.

present the theoretical foundations of our framework in this general setting and states the mathematical properties of F GW . Notably, we show that it is a metric in the space of structured objects with respect to an intuitive equivalence relation between structured objects, we give a concentration result for the convergence of finite samples, and we study its interpolation and geodesic properties.

Structured data as probability measures

In this chapter, we focus on comparing structured data which combine a feature and a structure information. In order to give a good intuition about the method we first consider the discrete setting which corresponds to labeled graphs. More formally, we consider undirected labeled graphs as tuples of the form G = (V, E, f , s ) where (V, E) are the set of vertices and edges of the graph. f : V → Ω f is a labelling function which associates each vertex v i ∈ V with a feature a i def = f (v i ) in some feature metric space (Ω f , d). We will denote by feature information the set of all the features (a i ) i of the graph.

Similarly, s : V → Ω s maps a vertex v i from the graph to its structure representation x i def = s (v i ) in some structure space (Ω s , C) specific to each graph. C : Ω s × Ω s → R is a symmetric application which aims at measuring the similarity between the nodes in the graph. Unlike the feature space however, Ω s is implicit and in practice, knowing the similarity measure C will be sufficient. With a slight abuse of notation, C will be used in the following to denote both the structure similarity measure and the matrix that encodes this similarity between pairs of nodes in the graph (C(i, k) = C(x i , x k )) i,k . Depending on the context, C can either encode the neighborhood information of the nodes, the edge information of the graph or more generally it can model a distance between the nodes such as the shortest path distance or the harmonic distance [START_REF] Verma | [END_REF]]. When C is a metric, such as the shortest-path distance, we naturally endow the structure with the metric space (Ω s , C). We will denote by structure information the set of all the structure embeddings (x i ) i of the graph.

We propose to enrich the previously described graph with a probability vector which serves the purpose of signaling the relative importance of the vertices in the graph. To do so, if we assume that the graph has n vertices, we equip those vertices with weights (h i ) i ∈ Σ n . Through this procedure, we derive the notion of structured data as a tuple S = (G, h G ) where G is a graph as described previously and h G is a Figure 3.2: F GW loss Eq for a coupling π depends on both a similarity between each feature of each node of each graph (d(ai, bj))i,j and between all intra-graph structure similarities (|C1(xi,

x k ) -C2(xj, x l )|) i,j,k,l .
function that associates a weight to each vertex. This definition allows the graph to be represented by a fully supported probability measure over the product space feature/structure µ = n i=1 h i δ (xi,ai) which describes the entire structured data (see Figure 3.1). When all the weights are equal (i.e. h i = 1 n ), so all vertices have the same relative importance, the structured data holds the exact same information as its graph. However, weights can be used to encode some a priori information. For instance on segmented images, one can construct a graph using the spatial neighborhood of the segmented zones, the features can be taken as the average color in the zone, and the weights as the ratio of image pixels in the zone.

Fused Gromov-Wasserstein approach for structured data

We aim at defining a distance between two graphs G 1 and G 2 , described respectively by their probability measure µ = n i=1 h i δ (xi,ai) and ν = m i=1 g j δ (yj ,bj ) , where h ∈ Σ n and g ∈ Σ m are probability vectors. Without loss of generality we suppose (x i , a i ) = (x j , a j ) for i = j (same for y j and b j ). We recall that Π(h, g) the set of all admissible couplings between h and g. To that extent, the matrix π ∈ Π(h, g) describes a probabilistic matching of the nodes of the two graphs. We note M AB = (d(a i , b j )) i,j the n × m matrix standing for the distance between the features. The structure matrices are denoted C 1 and C 2 , and µ X and µ A (resp. ν Y and ν B ) are representative of the marginals of µ (resp. ν) w.r.t. the structure and feature respectively (see Figure 3.1). We also define the similarity between the structures by measuring the similarity between all pairwise distances within each graph thanks to the 4-dimensional tensor L(C 1 , C 2 ):

L(C 1 , C 2 ) = (L i,j,k,l (C 1 , C 2 )) i,j,k,l = (|C 1 (i, k) -C 2 (j, l)|) i,j,k,l .

Fused Gromov-Wasserstein distance

We define a novel Optimal Transport discrepancy called the Fused Gromov-Wasserstein distance. It is defined for a trade-off parameter α ∈ [0, 1] as

F GW (M AB , C 1 , C 2 , h, g) = min π∈Π(h,g) E q (M AB , C 1 , C 2 , π) (3.1)
where

E q (M AB , C 1 , C 2 , π) = (1 -α)M q AB + αL(C 1 , C 2 ) q ⊗ π, π F = i,j,k,l (1 -α)d(a i , b j ) q + α|C 1 (i, k) -C 2 (j, l)| q π i,j π k,l (3.2)
The F GW distance looks for the coupling π between the vertices of the graph that minimizes the cost E q which is a linear combinaison of a cost d(a i , b j ) of transporting one feature a i to a feature b j and a cost |C 1 (i, k) -C 2 (j, l)| of transporting pairs of nodes in each structure (see Figure 3.2). As such, the optimal coupling tends to associate pairs of feature and structure points with similar distances within each structure pair and with similar features. α acts as a trade-off parameter between the cost of the structures represented by L(C 1 , C 2 ) and the cost on the features M AB . In this way, the convex combination of both terms leads to the use of both information in one formalism resulting on a single map π which "moves" the mass from one joint probability measure forawrd to the other. As an important feature of F GW , by relying on a sum of (inter-and intra-)vertex-to-vertex distances, it can handle structured data with continuous attributed or discrete labeled nodes (thanks to the definition of d) and can also be computed even if the graphs have different number of nodes. This new distance is called the F GW distance as it acts as a generalization of the Wasserstein and Gromov-Wasserstein distances. Indeed as α tends to zero, the F GW distance recovers the Wasserstein distance between the features W q (µ A , ν B ) q and as α tends to one, we recover the Gromov-Wasserstein distance GW q (µ X , ν Y ) q between the structures (see Proposition 3.5.2 of Section 3.5.5).

More importantly F GW enjoys metric properties on labeled graphs as stated in the following theorem:

Theorem 3.3.1 (F GW defines a metric for q = 1 and a semi-metric for q > 1). If q = 1, and if C 1 , C 2 are distance matrices such as shortest-path matrices then F GW defines a metric over the space of structured data quotiented by the measure preserving isometries that are also feature preserving. More precisely, F GW satisfies the triangle inequality and vanishes iff n = m and there exists a permutation

σ ∈ S n such that: ∀i ∈ [[n]], h i = g σ(i) (3.3) ∀i ∈ [[n]], a i = b σ(i) (3.4) ∀i, k ∈ [[n]] 2 , C 1 (i, k) = C 2 (σ(i), σ(k)) (3.5)
If q > 1, the triangle inequality is relaxed by a factor 2 q-1 such that F GW defines a semi-metric.

This results is a direct consequence of Theorem 3.5.1 in Section 3.5.2 where F GW is defined for general metric spaces. The resulting permutation σ preserves the weight of each node (equation (3.3)), the features (equation (3.4)) and the pairwise structure relation between the nodes (equation (3.5)). For example, comparing two graphs with uniform weights on the vertices and with shortest-path structure matrices, the F GW distance vanishes iff the graphs have the same number of vertices and there exists a one-to-one mapping between the vertices of the graphs which preserves both the shortest-paths and the features. More informally, in this case graphs have vertices with the same labels connected by the same edges, and thus F GW can be used to determine if two graphs are isomorphic [West 2000].

The metric F GW is fully unsupervised and can be used in a wide set of applications such as k-nearestneighbors, distance-substitution kernels, pseudo-Euclidean embeddings, or representative-set methods. Arguably, such a distance also allows for a fine interpretation of the similarity (through the optimal mapping π), contrary to end-to-end learning machines such as neural networks.

Fused Gromov-Wasserstein barycenter

OT barycenters have many desirable properties and applications (see Chapter 2 for more details), yet no formulation can leverage both structural and feature information in the barycenter computation. In this section, we consider the F GW distance to define a barycenter of a set of structured data as a Fréchet mean. We look for the structured data µ that minimizes the sum of (weighted) F GW distances within a given set of structured data (µ

k ) k∈[[K]] associated with structure matrices (C k ) k∈[[K]] , features (B k ) k∈[[K]] and base histograms (h k ) k∈[[K]] .
For simplicity, we assume that the histogram h associated to the barycenter is known and fixed; in other words, we set the number of vertices N and the weight associated to each of them.

In this context, for a fixed N ∈ N and (λ k ) k such that K k=1 λ k = 1 , we aim to find the set of features A = (a i ) i and the structure matrix C of the barycenter that minimize the following equation:

min µ K k=1 λ k F GW q,α (µ, µ k ) = min C∈R N ×N , A∈R N ×d ∀k∈[[K]],π k ∈Π(h,h k ) K k=1 λ k E q (M AB k , C, C k , π k ) (3.6)
Note that this problem is jointly convex w.r.t. C and A but not w.r.t. π k . We discuss the proposed algorithm to solve this problem in the next section. Interestingly enough, one can derive several variants of this problem, where the features or the structure matrices of the barycenter can be fixed. Solving the related simpler optimization problem extends straightforwardly. We give examples of such barycenters both in the experimental section where we solve a graph based k-means problem.

Optimization and algorithmic solution

In this section we discuss the numerical optimization problem for computing the F GW distance between discrete distributions.

Solving the Quadratic Optimization problem. Equation (3.1) is clearly a quadratic problem w.r.t. π which is NP-hard in general [START_REF] Loiola | [END_REF]]. However finding a solution in practice can be done quite efficiently. We propose here a method based on the Frank-Wolfe algorithm [Jaggi 2013] (aka Conditional Gradient). When considering q = 2 the F GW computation problem can be re-written as finding π * such that:

π * = arg min π∈Π(h,g) vec(π) T Q(α)vec(π) + vec(D(α)) T vec(π) (3.7) where Q = -2αC 2 ⊗ K C 1 and D(α) = (1 -α)M AB .
⊗ K denotes the Kronecker product of two matrices, vec the column-stacking operator. With such form, the resulting optimal map can be seen as a quadratic regularized map from initial Wasserstein [START_REF] Ferradans | [END_REF][START_REF] Flamary | [END_REF]]. However, unlike these approaches, we have a quadratic but provably non convex term. The gradient G that arises from equation (3.1) can be expressed with the following partial derivative w.r.t. π:

G(π) = (1 -α)M q AB + 2αL(C 1 , C 2 ) q ⊗ π (3.8)
Note that despite the apparent O(m 2 n 2 ) complexity of computing the tensor product L(C 1 , C 2 ) q ⊗ π given π, one can simplify the sum to complexity O(mn 2 + m 2 n) [START_REF] Peyré | [END_REF]] operations when q = 2. Solving a large scale QP with a classical solver can be computationally expensive. In [START_REF] Ferradans | [END_REF]], authors propose a solver for a graph regularized optimal transport problem whose resulting optimization problem is also a QP. We can then directly use their conditional gradient scheme to solve our optimization problem as presented in Algorithm 4. It only needs at each iteration to compute the gradient in equation (3.8) and to solve a linear OT problem with classical solvers (see Chapter 2 for more details). The line-search part is a constrained minimization of a second degree polynomial function and, as such, admits a closed form expression written in Algorithm 5. While the problem is non convex, conditional gradient is Algorithm 4 Conditional Gradient (CG) for F GW 1: π (0) ← hg 2: for i = 1, . . . , do 3: G ← Gradient from equation (3.8) w.r.t. π (i-1) 4: π(i) ← Solve OT with ground loss G 5: τ (i) ← Line-search for loss (3.1) with τ ∈ (0, 1) using Alg. 5 6:

π (i) ← (1 -τ (i) )π (i-1) + τ (i) π(i)
7: end for Algorithm 5 Line-search for CG (q = 2) 1: c C1,C2 from equation ( 6) in [START_REF] Peyré | [END_REF]]

2: a = -2α C 1 π(i) C 2 , π(i) 3: b= (1 -α)M AB + αc C1,C2 , π(i) F -2α C 1 π(i) C 2 , π (i-1) F + C 1 π (i-1) C 2 , π(i) F 4: if a > 0 then 5: τ (i) ← min(1, max(0, -b 2a )) 6: else 7: τ (i) ← 1 if a + b < 0 else τ (i) ← 0 8: end if known to converge to a local stationary point with a O( 1 √ t ) rate [Lacoste-Julien 2016]
. More precisely we note g (i) the Frank-Wolfe gap at iteration i defined by:

g (i) = max π∈Π(h,g) π -π (i) , -G i F (3.9)
We note also |||.||| the dual norm for tensors: |||L||| = sup A F =1 L ⊗ A F where L is a 4-dimensional tensor and A a matrix. Then using this dual norm we have that the gradient G(.) is |||2αL(C 1 , C 2 ) q |||-Lipschitz:

∀(π 1 , π 2 ) ∈ Π(h, g) 2 , G(π 1 ) -G(π 2 ) F = 2αL(C 1 , C 2 ) q ⊗ (π 1 -π 2 ) F ≤ 4α 2 |||L(C 1 , C 2 ) q ||| π 1 -π 2 F (3.10)
We note also diam . F (Π(h, g)) 2 the diameter of Π(h, g) (see [Jaggi 2013]):

diam . F (Π(h, g)) 2 = max (π1,π2)∈Π(h,g) 2 π 1 -π 2 2 F (3.11)
Then by using Theorem 1 in [Lacoste-Julien 2016] then minimal gap encountered by the iterates during the algorithm after t iterations satisfies:

min 0≤i≤t g (i) ≤ max{2h 0 , C} √ t + 1 (3.12)
where

C = 4α 2 |||L(C 1 , C 2 ) q ||| diam . F (Π(h, g)) 2
and h 0 is is the initial global suboptimality.

Solving the barycenter problem with Block Coordinate Descent (BCD).

We propose to minimize equation (3.6) using a BCD algorithm, i.e. iteratively minimizing with respect to the couplings π k , to the metric C and the feature vector A. The minimization of this problem w.r.t. (π k ) k∈ [[K]] is equivalent to compute S independent Fused Gromov-Wasserstein distances as discussed above. We suppose that the feature space is Ω f = (R d , .

W = 0 F GW > 0 GW = 0
2 2 ) and we consider q = 2. Minimization w.r.t. C in this case has a closed form (see Proposition 4 in [START_REF] Peyré | [END_REF]] and Chapter 2) :

C ← 1 hh T K k=1 λ k π T k C k π k (3.13)
where the division is computed elementwise and h is the histogram of the barycenter as discussed in section 3.3.2. Minimization w.r.t. A can be computed with [START_REF] Cuturi | [END_REF], Equation 8]:

A ← K k=1 λ k B k π T k diag( 1 h ) (3.14)

Experimental results

We illustrate in this section the behavior of our method on synthetic and real datasets. The algorithm presented in the previous section have been implemented in the Python Optimal Transport toolbox [Flamary 2017].

Illustration of FGW on trees

We construct two trees as illustrated in Figure 3.3, where the 1D node features are shown with colors (in red, features belong to [0, 1] and in blue in [9,10]). The structure similarity matrices C 1 and C 2 are the shortest-paths between the nodes. Both trees have the same individual structure and the same features up to a permutation. However when combining both informations the trees are not the same, as they do not have the same labels at the same place. Figure 3.3 illustrates the behavior of the F GW distance when the trade-off parameter α changes. The left part recovers the Wasserstein distance between the features (α = 0): red nodes are coupled to red ones and the blue nodes to the blue ones. For an alpha close to 1 (right), we recover the Gromov-Wasserstein distance between the structures of the trees: all couples of points are coupled to another couple of points, without taking into account the features. Both approaches fail in discriminating the two trees. Finally, for an intermediate α in F GW (center), the bottom and first level structure are preserved as well as the feature matching (red on red and blue on blue), resulting on a positive distance. Note that F GW preserves also the substructures of the trees through its coupling.

Illustration of FGW on 1D distributions

Figure 3.4 illustrates the differences between Wasserstein, Gromov-Wasserstein and Fused Gromov-Wasserstein couplings π * on 1D distributions. In this example both the feature and structure are 1-dimensional, that is (x i , y j ) ∈ R 2 and (a i , b j ) ∈ R 2 (Figure 3.4 left). The feature space (vertical axis) denotes two clusters among the elements of both objects illustrated in the OT matrix M AB , the structure space (horizontal axis) denotes a noisy temporal sequence along the indexes illustrated in the matrices C 1 and C 2 (Figure 3.4 center). Wasserstein respects the clustering but forgets the temporal structure, Gromov-Wasserstein respects the structure but do not take the clustering into account. Only FGW retrieves a transport matrix respecting both feature and structure.

Illustration of FGW on simple images

We extract a 28 × 28 image from the MNIST dataset and generate a second one through translation or mirroring of the digit in the original image. We use pixel gray levels as the features, and the structure is defined as the city-block distance on the pixel coordinate grid. We use equal weights for all the pixels in the image. Figure 3.5 shows the different couplings obtained when considering either the features only, the structure only or both information. F GW aligns the pixels of the digits, recovering the correct order of the pixels, while both Wassertein and Gromov-Wasserstein distances fail at providing a meaningful transportation map. Note that in the Wasserstein and Gromov-Wasserstein case, the distances are equal to 0, whereas F GW manages to spot that the two images are different. Also note that, in the F GW sense, the original digit and its mirrored version are also equivalent as there exists an isometry between their structure spaces, making F GW invariant to rotations or flips in the structure space in this case.

Illustration of FGW on time series data

One of the main assets of F GW is that it can be used on a wide class of structured data such as graphs and also time series. We consider here 25 monodimensional time series composed of two humps in [0, 1] For readibility issues, only the couplings starting from non white pixels on the left picture are depicted.

with random uniform height between 0 and 1. Signals are distributed according to two classes translated from each other with a fixed gap. The F GW distance is computed by considering M AB as the Euclidean distance between the features of the signals (here the value of the signal in each point) and C 1 and C 2 as the euclidean distance between timestamps.

A 2D embedding is computed from a F GW distance matrix between a number of examples in this dataset with multidimensional scaling (MDS) in Figure 3. 6 (top). One can clearly see that the representation with a reasonable α value in the center is the most discriminant one. This can be better understood by looking as the OT matrices between the classes. Figure 3.6 (bottom) illustrates the behavior of F GW on one pair of examples when going from Wasserstein to Gromov-Wasserstein. The black line depicts the matching provided by the transport matrix and one can clearly see that while Wasserstein on the left assigns samples completely independently to their temporal position, the Gromov-Wasserstein on the right tends to align perfectly the samples (note that it could have reversed exactly the alignment with the same loss) but discards the values in the signal. Only the true F GW in the center finds a transport matrix that both respects the time sequences and aligns similar values in the signals.

Graph-structured data classification

We now use F GW on real-world dataset where we study its behavior on a graph classification task. More precisely we address the question of training a classifier for graph data and evaluate the FGW distance used in a kernel with SVM.

Datasets

We consider 12 widely used benchmark datasets divided into 3 groups. BZR, COX2 [START_REF] Sutherland | [END_REF], PROTEINS, ENZYMES [START_REF] Borgwardt | [END_REF]], CUNEIFORM [Kriege 2018] and SYNTHETIC [START_REF] Feragen | [END_REF]] are vector attributed graphs. MUTAG [START_REF] Debnath | [END_REF]], PTC-MR [START_REF] Kriege | [END_REF]] and NCI1 [Wale 2008] contain graphs with discrete attributes derived from small molecules. IMDB-B, IMDB-M [START_REF] Yanardag | [END_REF]] contain unlabeled graphs derived from social networks. All datasets are available in [START_REF] Kersting | [END_REF]]. Experimental setup Regarding the feature distance matrix M AB between node features, when dealing with real valued vector attributed graphs, we consider the 2 distance between the labels of the vertices.

Increasing value of

In the case of graphs with discrete attributes, we consider two settings: in the first one, we keep the original labels (denoted as raw); we also consider a Weisfeiler-Lehman labeling (denoted as wl) by concatenating the labels of the neighbors. A vector of size h is created by repeating this procedure h times [START_REF] Vishwanathan | [END_REF][START_REF] Kriege | [END_REF]]. In both cases, we compute the feature distance matrix by using

d(a i , b j ) = H k=0 δ(τ (a k i ), τ (b k j ))
where δ(x, y) = 1 if x = y else δ(x, y) = 0 and τ (a k i ) denotes the concatenated label at iteration k (for k = 0 original labels are used). Regarding the structure distances C, they are computed by considering a shortest path distance between the vertices.

For the classification task, we run a SVM using the indefinite kernel matrix e -γF GW which is seen as a noisy observation of the true positive semidefinite kernel [Luss 2007]. We compare classification accuracies with the following state-of-the-art graph kernel methods: (SPK) denotes the shortest path kernel [START_REF] Borgwardt | [END_REF]], (RWK) the random walk kernel [Gärtner 2003], (WLK) the Weisfeler Lehman kernel [START_REF] Vishwanathan | [END_REF]], (GK) the graphlet count kernel [Shervashidze 2009]. For real valued vector attributes, we consider the HOPPER kernel (HOPPERK) [START_REF] Feragen | [END_REF]] and the propagation kernel (PROPAK) [START_REF] Neumann | [END_REF]]. We build upon the GraKel library [Siglidis 2018] to construct the kernels and C-SVM to perform the classification. We also compare F GW with the PATCHY-SAN framework for CNN on graphs (PSCN) [START_REF] Niepert | Learning Convolutional Neural Networks for Graphs[END_REF]] building on our own implementation of the method1 .

To compare the methods, most papers about graph classification usually perform a nested cross validation (using 9 folds for training, 1 for testing, and reporting the average accuracy of this experiment repeated 10 times) and report accuracies of the other methods taken from the original papers. However, these comparisons are not fair because of the high variance on most datasets w.r.t. the folds chosen for training and testing. This is why, in our experiments, the nested cross validation is performed on the same folds for training and testing for all methods. In the result tables 3.1, 3.2 and 3.3 we add a (*) when the best score does not yield to a significative improvement (based on a Wilcoxon signed rank test on the test scores) compared to the second best one. Note that, because of their small sizes, we repeat the experiments 50 times for MUTAG and PTC-MR datasets. For all methods using SVM, we cross validate the parameter C ∈ {10 -7 , 10 -6 , ..., 10 7 }. The range of the WL parameter h is {0, 1..., 10}, and we also compute this kernel with h fixed at 2, 4. The decay factor λ for RWK {10 -6 , 10 -5 ..., 10 -2 }, for the GK kernel we set the graphlet size κ = 3 and cross validate the precision level ε and the confidence δ as in the original paper [Shervashidze 2009]. The t max parameter for PROPAK is chosen within {1, 3, 5, 8, 10, 15, 20}. For PSCN, we choose the normalized betweenness centrality as labeling procedure and cross validate the batch size in {10, 15, ..., 35} and number of epochs in {10, 20, ..., 100}. Finally for F GW , γ is cross validated within {2 -10 , 2 -9 , ..., 2 10 } and α is cross validated via a logspace search in [0, 0.5] and symmetrically [0.5, 1] (15 values are drawn).

Results and discussion

Vector attributed graphs. The average accuracies reported in Table 3.1 show that FGW is a clear state-of-the-art method and performs best on 4 out of 6 datasets with performances in the error bars of the best methods on the other two datasets. Results for CUNEIFORM are significantly below those from the original paper [Kriege 2018] which can be explained by the fact that the method in this paper uses a graph convolutional approach specially designed for this dataset and that the experimental setting is different. In comparison, the other competitive methods are less consistent as they exhibit some good performances on some datasets only.

Discrete labeled graphs.

We first note in Table 3.2 that F GW using WL attributes outperforms all competitive methods, including F GW with raw features. Indeed, the WL attributes allow encoding more finely the neighborood of the vertices by stacking their attributes, whereas FGW with raw features only consider the shortest path distance between vertices, not their sequence of labels. This result calls for using meaningful feature and/or structure matrices in the FGW definition, that can be dataset-dependant, in order to enhance the performances. We also note that F GW with WL attributes outperforms the WL kernel method, highlighting the benefit of an optimal transport-based distance over a kernel-based similarity. Surprisingly results of PSCN are significantly lower than those from the original paper. We believe that it comes from the difference between the folds assignment for training and testing, which suggests that PSCN is difficult to tune. Non-attributed graphs. The particular case of the GW distance for graph classification is also illustrated on social datasets, that contain no labels on the vertices. Accuracies reported in Table 3.3 show that it greatly outperforms SPK and GK graph kernel methods.

Comparison between F GW , W and GW During the validation step, the optimal value of α was consistently selected inside the ]0, 1[ interval, excluding 0 and 1, suggesting that both structure and feature pieces of information are necessary (details are given in Section 6.1.1).

Graph barycenter and compression

In this section we use the barycentric formulation of F GW using the Fréchet mean formulation in equation (3.6) in two settings. The first one is the computation of barycenter of several toy labeled graphs and the second the compression of one graph into a smaller graph, also known as coarsening [Loukas 2019].

Graph barycenter

In this part, we use F GW to compute barycenter of toy graphs. In a first example, we generate graphs following either a circle or 8 symbol with 1D features following a sine and linear variation respectively. For each example, the number of nodes is drawn randomly between 10 and 25, Gaussian noise is added to the features and a small noise is applied to the structure (some connections are randomly added). An example graph with no noise is provided for each class in the first column of Figure

Noiseless graph Barycenter

Noiseless graph

Noisy graphs samples 3.7. One can see from there that the circle class has a feature varying smoothly (sine) along the graph but the 8 has a sharp feature change at its center (so that low pass filtering would loose some information). Some examples of the generated graphs are provided in the 2nd-to-7th columns of Figure 3.7. We compute the F GW barycenter containing 10 samples using the shortest path distance between the nodes as the structural information and the distance induced by the Euclidean norm . 2 for the features. Note that the iterations of the barycenter defined in equation (3.13) result in a dense C matrix and to visualize properly the graph barycenter we need a adjacency matrix. We propose a simple heuristic procedure to recover an adjacency matrix for the graphs' barycenter based on a thresholding of the matrix C. Given a threshold t the matrix thresh t (C) is given by 1 if C ij <= t and 0 elsewhere. The threshold t is tuned so as to minimize the Frobenius norm between the original C matrix and the shortest path matrix constructed after thresholding C. More precisely if SP denotes the algorithm which takes as input an adjacency matrix and outputs a shortest-path matrix then the threshold is given:

arg min t∈R+ C -SP (thresh t (C)) 2 F (3.15)
The idea behind equation (3.15) is that C represents somehow the shortest-path matrix of a graph so that we want the adjacency matrix thresh t (C) to give a shortest-path matrix as close as possible to C. Unfortunately equation (3.15) is not differentiable with respect to t. We use a simple brute force strategy to find a suitable threshold t by looking at arg min t∈{t1,...,t L } C -SP (thresh t (C)) 2 F where t 1 , . . . , t L are drawn from R + .

Graph with communities

Approximate Graph Clustering with transport matrix

Graph with bimodal communities Approximate Graph

Clustering with transport matrix Resulting barycenters are showed in Figure 3.7 for n = 15 and n = 7 nodes. First, one can see that the barycenters are denoised both in the feature space and the structure space. Also note that the sharp change at the center of the 8 class is conserved in the barycenters which is a nice result compared to other divergences that tend to smooth-out their barycenters ( 2 for instance). Finally, note that by selecting the number of nodes in the barycenter one can compress the graph or estimate a "high resolution" representation from all the samples. To the best of our knowledge, no other method can compute such graph barycenters. Finally, note that F GW is interpretable because the resulting OT matrix provides correspondence between the nodes from the samples and those from the barycenter.

Graph compression

In the second experiment, we evaluate the ability of FGW to perform graph approximation and compression on a Stochastic Block Model graph [Wang 1987[START_REF] Nowicki | [END_REF]]. The question is to see if estimating an approximated graph can recover the relation between the blocks and perform simultaneously a community clustering on the original graph (using the coupling matrix π). We generate two community graphs illustrated in the left column of Figure 3.8. The coarsened graph is obtained by solving the Fréchet mean formulation with k = 1. More precisely given an original graph µ 0 described by its features B 0 and its structure C 0 we look for a graph of N nodes, with N smaller than the number of nodes of the original graph, which solves:

min µ F GW q,α (µ, µ 0 ) = min C∈R N ×N , A∈R N ×n ,π E q (M AB0 , C, C 0 , π) (3.16)
The results are depicted in Figure 3.8. We can see that the relation between the blocks is sparse and has a "linear" structure, the example in the first line has features that follow the blocks (noisy but similar in each block) whereas the example in the second line has two modes per block. The first graph approximation (top line) is done with N = 4 nodes and we can recover both the blocks in the graph and the average feature on each blocks (colors on the nodes). The second problem is more complex due to the two modes per block but one can see that when approximating the graph with N = 8 nodes we recover both the structure between the blocks and the sub-clusters in each block, which illustrates the strength of F GW that encodes both features and structures.

Unsupervised learning: graphs clustering

In the last experiment, we evaluate the ability of F GW to perform a clustering of multiple graphs and to retrieve meaningful barycenters of such clusters. To do so, we generate a dataset of 4 groups of community graphs. Each graph follows a simple Stochastic Block Model [Wang 1987[START_REF] Nowicki | [END_REF]] and the groups are defined w.r.t. the number of communities inside each graph and the distribution of their labels. The dataset is composed of 40 graphs (10 graphs per group) and the number of nodes of each graph is drawn randomly from {20, 30, ..., 50} as illustrated in Figure 3.9. We perform a k-means clustering using the F GW barycenter defined in equation (3.6) as the centroid of the groups and the F GW distance for the cluster assignment. We fix the number of nodes of each centroid to 30. We perform a thresholding on the pairwise similarity matrix C of the centroid at the end in order to obtain an adjacency matrix for visualization purposes. The threshold value is empirically chosen with the procedure described in the previous section. The evolution of the barycenters along the iterations is reported in Figure 3.9. We can see that these centroids recover community structures and feature distributions that are representative of their cluster content. On this example, note that the clustering recovers perfectly the known groups in the dataset. To the best of our knowledge, there exists no other method able to perform a clustering of graphs and to retrieve the average graph in each cluster without having to solve a pre-image problem.

Other applications of Fused Gromov-Wasserstein

Since its introduction the Fused Gromov-Wasserstein distance was also used in various contexts where structured data are involved. It has found applications in molecular biology for the analysis of Single-cell RNA (scRNA-seq) sequencing data in [Cang 2020] where authors use F GW as a building block to recover spatial properties of scRNA-sequation In machine learning F GW was used for learning structured autoencoders [START_REF] Xu | [END_REF]], to study the continuity of Graph Neural Network [START_REF] Béthune | [END_REF]] or on domain adaptation tasks on functional Near-Infrared Spectroscopy (fNIRS) data [START_REF] Lyu | [END_REF]]. The Fused Gromov-Wasserstein approach was further analysed in [Barbe 2020] where authors propose to improve the F GW distance using a smoothing strategy on the features of the graphs. They propose to incorporate a diffusion kernel on the features which result in a more robust similarity measures of labeled graphs. Authors tackle domain adaptation tasks between labeled graphs where the label information is only available in a target domain. F GW was also used on point clouds in [Puy 2020] for the estimation of scene flows i.e. of the 3D motion of points at the surface of objects in a scene.

FGW in the continuous setting

The previous graph representation for objects with a finite number of points/vertices extends naturally to the continuous setting. The purpose of this section is to generalize F GW to general probability distributions and to state some of its mathematical properties. We consider the following definition of structured objects:

Definition 3.5.1 (Structured objects). A structured object over a metric space (Ω, d) is a triplet (X × Ω, d X , µ)
, where: (X , d X ) is a metric space and µ is a probability measure over X × Ω. (Ω, d) is denoted as the feature space, such that d : Ω × Ω → R + is the distance in the feature space and (X , d X ) the structure space, such that d X : X × X → R + is the distance in the structure space. We will note µ X and µ A the structure and feature marginals of µ.

Definition 3.5.2 (Space of structured objects). We note X the set of all metric spaces. The space of all structured objects over (Ω, d) will be written as S(Ω) and is defined by all the triplets (X × Ω, d X , µ) where (X , d X ) ∈ X and µ ∈ P(X × Ω). To avoid finiteness issues we define for p ∈ N * the space S p (Ω) ⊂ S(Ω),

(X × Ω, d X , µ) ∈ S p (Ω) if: ˆΩ d(a, a 0 ) p dµ A (a) < +∞ (3.17)
(the finiteness of this integral does not depend on the choice of a 0 )

ˆX ×X d X (x, x ) p dµ X (x)dµ X (x ) < +∞. (3.18)
For the sake of simplicity, and when it is clear from the context, we will sometimes denote only by µ the whole structured object. In the same way as the discrete case we will note µ X and µ A the structure and feature marginals of µ. We recall that those marginals encode very partial information since they focus only on independent feature distributions or only on the structure. This definition encompasses the discrete setting discussed in above. More precisely let us consider a labeled graph of n nodes with features A = (a i ) n i=1 with a i ∈ Ω and X = (x i ) n i=1 the structure representation of the nodes. Let (h i ) n i=1 be an histogram, then the probability measure µ = n i=1 h i δ (xi,ai) defines structured object in the sense of Definition 3.5.1 since it lies in P(X × Ω). In this case, an example of µ, µ X and µ A is provided in Figure 3.1.

Note that the set of structured objects is quite general and allows also considering discrete probability measures of the form µ = p,q i,j=1 h i,j δ (xi,aj ) with p, q possibly different than n. We propose to focus on a particular type of structured objects, namely the generalized labeled graphs as described in the following definition: Definition 3.5.3 (Generalized labeled graph). We call generalized labeled graph a structured object

(X × Ω, d X , µ) ∈ S p (Ω) such that µ can be expressed as µ = (id × f )#µ X where f : X → Ω is surjective and pushes µ X forward to µ A , i.e. f #µ X = µ A .
This definition implies that there exists a function f which associates a feature a = f (x) to a structure point x ∈ X and, since f is surjective, one structure point can not have two different features. The labeled graph described by µ = n i=1 h i δ (xi,ai) is a particular instance of a generalized labeled graph in which f is defined by f (x i ) = a i .

Comparing structured objects

We now aim to define a notion of equivalence between two structured objects (X × Ω, d X , µ) and (Y × Ω, d Y , ν). We note in the following ν Y , ν B the marginals of ν. Intuitively, two structured objects are the same if they share the same feature information, if their structure information are lookalike and if the probability measures are corresponding in some sense. In this section, we present mathematical tools for individual comparison of the elements of structured objects. For completeness we recall here some useful mathematical tools defined in Chapter 2 for comparing the elements of structured objects and we refer the reader to this chapter for more details. Definition 3.5.4 (Isometry). Let (X , d X ) and (Y, d Y ) be two metric spaces. An isometry is a sujective map φ : X → Y that preserves the distances:

∀(x, x ) ∈ X 2 , d Y (φ(x), φ(x )) = d X (x, x ) (3.19)
We refer the reader to section 2.2.2 for wider explanations about isometry. The previous map φ can be used in order to compare the structure information of two structured objects. When the metric spaces are enriched with a probability measure they define a measurable metric spaces also called mm-spaces (see section 2.2.2). In this case the notion of strong isomorphism can be used for comparing mm-spaces:

Definition 3.5.5 (Strong isomorphism). Two mm-spaces (X , d X , µ X ), (Y, d Y , µ Y ) are strongly isomorphic if there exists an isometry φ : supp(µ X ) → supp(ν Y ) which pushes µ X forward to µ Y , i.e. φ#µ X = µ Y .
In this case we say that φ is measure preserving.

All this considered, we can now define a notion of equivalence between structured objects: Definition 3.5.6 ((II)-Strong isomorphism of structured objects.). Two structured objects are said to be (II)-strongly isomorphic if there exists an isomorphism I : supp(µ X ) → supp(ν Y ) between the structures such that φ = (I, id) is bijective between supp(µ) and supp(ν) and measure preserving. More precisely φ satisfies the following properties:

P.1 φ#µ = ν.

P.2 The function φ statisfies:

∀(x, a) ∈ supp(µ), φ(x, a) = (I(x), a).

P.3 The function

I : supp(µ X ) → supp(ν Y ) is surjective, satisfies I#µ X = ν Y and: ∀x, x ∈ supp(µ X ) 2 , d X (x, x ) = d Y (I(x), I(x )).
Remark 3.5.1. It is easy to check that the (II)-strong isomorphism defines an equivalence relation over S p (Ω). Moreover the function φ described in this definition can be seen as a feature, structure and measure preserving function. Indeed from P.1 φ is measure preserving. Moreover (X , d X , µ X ) and (Y, d Y , ν Y ) are isomorphic through I. Finally using P.1 and P.2 we have that µ A = ν B so that the feature information is also preserved.

To illustrate this definition, we consider a simple example of two structured objects in the discrete case:

Example 3.5.1. Let two structured objects defined by:

     (x 1 , a 1 ) (x 2 , a 2 ) (x 3 , a 3 ) (x 4 , a 4 )      xi,ai ,      0 1 1 1 1 0 1 2 1 1 0 2 1 2 2 0      d X (xi,xj ) ,      1 /4 1 /4 1 /4 1 /4      hi and      (y 1 , b 1 ) (y 2 , b 2 ) (y 3 , b 3 ) (y 4 , b 4 )      yi,bi ,     0 1 1 1 1 0 2 2 1 2 0 1 1 2 1 0     d Y (yi,yj ) ,      1 /4 1 /4 1 /4 1 /4      h i
with for i, a i = b i and for i = j, a i = a j (see Figure 3.10). The two structured objects have isometric structures and same features individually but they are not (II)-strongly isomorphic. One possible map

φ = (φ 1 , φ 2 ) : X × Ω → Y × Ω such that φ 1 leads to an isometry is φ(x 1 , a 1 ) = (y 1 , b 1 ), φ(x 2 , a 2 ) = (y 3 , b 3 ), φ(x 3 , a 3 ) = (y 4 , b 4 ), φ(x 4 , a 4 ) = (y 2 , b 2
). Yet this map does not satisfy φ 2 (x, .) = id for any x since φ(x 2 , a 2 ) = (y 3 , b 3 ) and a 2 = b 3 . The other possible functions such that φ 1 leads to an isometry are simply permutations of this example, yet it is easy to check that none of them verifies P.2 (for example with

φ(x 2 , a 2 ) = (y 4 , b 4 )).
We generalize the definition (3.1) of Fused Gromov-Wasserstein (F GW ) to the continuous setting as follows:

Definition 3.5.7. The Fused-Gromov-Wasserstein distance is defined for α ∈ [0, 1] and p, q ≥ 1 as:

F GW α,p,q (µ, ν) = inf π∈Π(µ,ν) E p,q,α (π) 1 p (3.20)
where:

E p,q,α (π) = ˆˆ (1 -α)d(a, b) q + α|d X (x, x ) -d Y (y, y )| q p dπ((x, a), (y, b))dπ((x , a ), (y , b ))
We will write in the following L(x, y, x , y

) = |d X (x, x ) -d Y (y, y )|. (x 2 , a 2 ) (x 3 , a 3 ) (x 4 , a 4 ) (x 1 , a 1 ) (y 2 , b 2 ) (y 3 , b 3 ) (y 1 , b 1 ) (y 4 , b 4 ) Figure 3
.10: Two structured objects with isometric structures and identical features that are not (II)-strongly isomorphic. The color of the nodes represent the node feature and each edge represents a distance of 1 between the connected nodes. The feature space Ω is the common space for all features. The two metric spaces (X , dX ) and (Y, dY ) represent the structures of the two structured objects, the similarity between all pair-to-pair distances of the structure points is measured by L(x, y, x , y ). µ and ν are the joint measures on the structure space and the feature space.

Note that this definition is coherent with the definition given in equation (3.1) when p = 1. For brevity we will simply note F GW instead for F GW α,p,q when it is clear from the context. Many desirable properties arise from this definition. Among them, one can define a topology over the space of structured objects using the F GW distance to compare structured objects, in the same philosophy as for Wasserstein and Gromov-Wasserstein distances. The definition also implies that F GW acts as a generalization of both Wasserstein and Gromov-Wasserstein distances, with F GW achieving an interpolation between these two distances. More remarkably, F GW distance also realizes geodesic properties over the space of structured objects, allowing the definition of gradient flows. Before reviewing all these properties, we first compare F GW with GW and W (by assuming for now that F GW exists, which will be shown later in Theorem 3.5.1). Proposition 3.5.1 (Comparison between F GW , GW and W .). With previous notations:

• The following inequalities hold:

F GW α,p,q (µ, ν) ≥ (1 -α)W pq (µ A , ν B ) q (3.21) F GW α,p,q (µ, ν) ≥ αGW pq (µ X , ν Y ) q (3.
F GW α,p,1 (µ, ν)(µ, ν) ≤ 2W p (µ, ν). (3.23)
Proof of this proposition can be found in Section 6.1.2. In particular, following this proposition, when the F GW distance vanishes then both GW and W distances vanish so that the structure and the feature of the structure object are individually "the same" (with respect to their corresponding equivalence relation). However the converse is not necessarily true as shown further in Section 3.5.3.

In the following we establish some mathematical properties of the F GW distance. The first result relates to the existence of the F GW distance and the topology of the space of structured objects. We prove that the F GW distance is indeed a distance regarding the equivalence relation between structured objects as defined in Defintion 3.5.6, allowing us to derive a topology on S(Ω).

Topology of the structured object space

The F GW distance has the following properties:

Theorem 3.5.1 (Metric properties). Let p, q ≥ 1, α ∈]0, 1[ and (µ, ν) ∈ S pq (Ω) × S pq (Ω). The func- tional π → E p,q,α (π) always achieves an infimum π * in Π(µ, ν) s.t. F GW α,p,q (µ, ν) = E p,q,α (π * ) < +∞. Moreover:
• F GW α,p,q is symmetric and, for q = 1, satisfies the triangle inequality. For q ≥ 2, the triangular inequality is relaxed by a factor 2 q-1 .

• For α ∈]0, 1[, F GW α,p,q (µ, ν) = 0 if an only if there exists a bijective function φ = (φ 1 , φ 2 ) : supp(µ) → supp(ν) such that:

φ#µ = ν (3.24) ∀(x, a) ∈ supp(µ) , φ 2 (x, a) = a (3.25) ∀(x, a), (x , a ) ∈ supp(µ) 2 , d X (x, x ) = d Y (φ 1 (x, a), φ 1 (x , a )) (3.26) • If (µ, ν) are generalized labeled graphs then F GW α,p,q (µ, ν) = 0 if and only if (X × Ω, d X , µ) and (Y × Ω, d Y , ν) are (II)-strongly isomorphic.
Proof of this theorem can be found in Section 6.1.3. The identity of indiscernibles is the most delicate part to prove and is based on using the Gromov-Wasserstein distance between the spaces X × Ω and Y × Ω. The previous theorem states that F GW is a distance over the space of generalized labeled graphs endowed with the strong isomorphism as equivalence relation defined in Definition 3.5.6. More generally for any structured objects the equivalence relation is given by equations (3.24), (3.25) and (3.26). Informally, invariants of the F GW are structured objects that have both the same structure and the same features in the same place. Despite the fact that q = 1 leads to a proper metric the case q = 2 can be computed more efficiently using a separability trick from [START_REF] Peyré | [END_REF]] as seen in Section 3.3.3. Remark 3.5.2. Note that the previous theorem actually proves Theorem 3.3.1. Indeed if we consider µ = n i=1 h i δ (xi,ai) and ν = m j=1 g j δ (yj ,bj ) describing two labeled graphs as discussed in the previous section. Then µ and ν are generalized labeled graph. Using Theorem 3.5.1 µ and ν are (II)-strongly isomorphic if and only if there exists a bijection between the supports satisfies (P.1), (P.2) and (P.3). Since the supports are discrete this is equivalent to the condition n = m and there exists a permutation σ ∈ S n which satisfies the conditions of Theorem 3.3.1. The triangle inequality property of Theorem 3.3.1 derives directly from the triangle inequality of Theorem 3.5.1.

There are some special cases where W and GW can be adapted to structured objects and can be used also to compare them. These cases result in different notions of equivalence as described in the following discussion.

Can we adapt W and GW distances for structured objects?

Despite the appealing properties of both Wasserstein and Gromov-Wasserstein distances, they fail at comparing structured objects by focusing only on the feature and structure marginals respectively. However, with some hypotheses, one could adapt these distances for structured objects. (y,b) and apply the Wasserstein distance. In this case, when the Wasserstein distance vanishes then the structured objects are equal in the sense µ = ν which implies that µ and ν are de facto (II)-strongly isomorphic. This approach is very related with the one discussed in [START_REF] Thorpe | [END_REF]] where authors define the Transportation L p distance for signal analysis purposes. Their approach can be viewed as a transport between two joint measures:

µ(X × Ω) = L({(x, f (x)) | x ∈ X ⊂ Z = R d | f (x) ∈ Ω ⊂ R m }) (3.27) ν(Y × Ω) = L({(y, g(y)) | y ∈ Y ⊂ Z = R d | g(y) ∈ Ω ⊂ R m }) (3.28)
for functions f, g : Z → R m representative of the signal values and L the Lebesgue measure. The distance for the transport is defined as d((x, f (x)), (y, g(y))) = 1 α xy p p + f (x) -g(y) p p for α > 0 and • p the l p norm. In this case f (x) and g(y) can be interpreted as encoding the feature information of the signal while x, y encode its structure information. This approach is interesting but cannot be used on structured objects such as graphs that will not share a common structure embedding space.

Adapting Gromov-Wasserstein

The Gromov-Wasserstein distance can also be adapted to structured objects by considering the distances (1 -β)d X ⊕ βd and (1 -β)d Y ⊕ βd within each space X × Ω and Y × Ω respectively and β ∈]0, 1[. When the resulting GW distance vanishes, structured objects are strongly isomorphic with respect to (1 -β)d X ⊕ βd and (1 -β)d Y ⊕ βd. However the (II)-strong isomorphism is stronger than this notion since the strong isomorphism allows for "permuting the labels" but not the (II)-strong isomorphism. More precisely we have the following lemma:

Lemma 3.5.1. Let (X × Ω, d X , µ), (Y × Ω, d Y , ν) be two structured objects and β ∈]0, 1[. If (X × Ω, d X , µ) and (Y × Ω, d Y , ν) are (II)-strongly isomorphic then (X × Ω, (1 -β)d X ⊕ βd, µ) and (Y × Ω, (1 -β)d Y ⊕ βd, ν
) are strongly isomorphic. However the converse is not true in general.

Proof. To see this, if we consider φ as defined in Theorem 3.5.1, then for (x, a), (x , b) ∈ supp(µ) 2 we have

d X (x, x ) = d Y (I(x), I(x ))
. In this way:

(1 -β)d X (x, x ) + βd(a, b) = (1 -β)d Y (I(x), I(x )) + βd(a, b) (3.29)
which can be rewritten as:

(1 -β)d ⊕ βd X ((x, a), (x , b)) = (1 -β)d ⊕ βd Y (φ(x, a), φ(x , b)) (3.30)
and so φ is an isometry with respect to (1 -β)d ⊕ βd X and (1 -β)d ⊕ βd Y . Since φ is also measure preserving and surjective (X × Ω, (1 -β)d X ⊕ βd, µ) and (Y × Ω, (1 -β)d Y ⊕ βd, ν) are strongly isomorphic. However the converse is not necessarily true as it is easy to cook up an example with the same structure but with permuted labels so that objects are strongly isomorphic but not (II)-strongly isomorphic. For example in the tree example depicted in Figure 3.10, the structures are isometric and the distances between the features within each space are the same between each structured objects so that (X ×Ω, (1-β)d X ⊕βd, µ) and (Y × Ω, (1 -β)d Y ⊕ βd, ν) are strongly isomorphic, yet not (II)-strongly isomorphic as shown in the example since F GW > 0.

Convergence of finite samples

The metric properties of F GW naturally endow the structured object space with a notion of convergence as described in the next definition: Definition 3.5.8 (Convergence of structured objects.). Let (X n × Ω, d Xn , µ n ) n∈N be a sequence of structured objects. It converges to (X × Ω, d X , µ) in the Fused Gromov-Wasserstein sense if:

lim n→∞ F GW α,p,1 (µ n , µ) = 0 (3.31)
We consider in this definition only the case q = 1 as it gives a proper metric (with q > 1 the triangle inequality is relaxed by a factor 2 q-1 ). Using Prop. 3.5.1, it is straightforward to see that if the sequence converges in the F GW sense, both the features and the structure converge respectively in the Wasserstein and Gromov-Wasserstein sense.

An interesting question arises from this definition. If we consider a structured object (X × Ω, d X , µ) and if we sample the joint distribution so as to consider ({(

x i , a i )} i∈{1,..,n} , d X , µ n ) n∈N with µ n = 1 n n i=1 δ xi,ai
where (x i , a i ) ∈ X × Ω are sampled from µ. Does this sequence converges to (X × Ω, d X , µ) in the F GW sense and how fast is the convergence? This question can be answered thanks to a notion of "size" of a probability measure. For the sake of conciseness we will not present exhaustively the theory but the reader can refer to [Weed 2017] for more details. Given a measure µ on Ω we denote as dim * p (µ) its upper Wasserstein dimension. It coincides with the intuitive notion of "dimension" when the measure is sufficiently well behaved. For example, for any absolutely continuous measure µ with respect to the Lebesgue measure on

[0, 1] d , we have dim * p (µ) = d for any p ∈ [1, d 2 ]
.Using this definition and the results in [Weed 2017], we can answer the question of convergence of finite sample in the following proposition (proof can be found in Section 6.1.4): Theorem 3.5.2 (Convergence of finite samples and a concentration inequality). With previous notations. Let p ≥ 1. We have:

lim n→∞ F GW α,p,1 (µ n , µ) = 0 (3.32)
Moreover, suppose that s > d * p (µ). Then there exists a constant C that does not depend on n such that:

E[F GW α,p,1 (µ n , µ)] ≤ Cn -1 s . (3.33)
The expectation is taken over the i.i.d samples (x i , a i ). A particular case of this inequality is when α = 1 so that we can use the result above to derive a concentration result for the Gromov-Wasserstein distance. More precisely, if ν n = 1 n i δ xi denotes the empirical measure of ν ∈ P(X ) and if

s > d * p (ν) we have: E[GW p (ν n , ν)] ≤ C n -1 s . (3.34)
This result is a simple application of the convergence of finite sample properties of the Wasserstein distance, since in this case µ n and µ are part of the same ground space so that equation (3.33) derives naturally from equation (3.23) and the properties of Wasserstein. In contrast to the Wasserstein distance case this inequality is not necessarily sharp and future work will be dedicated to the study of its tightness.

Interpolation properties between W and GW distances

F GW distance is a generalization of both Wasserstein and Gromov-Wasserstein distances in the sense that it achieves an interpolation between them. More precisely, we have the following theorem: Proposition 3.5.2 (Interpolation properties.). As α tends to zero, one recovers the Wasserstein distance between the features information and as α goes to one, one recovers the Gromov-Wasserstein distance between the structure information:

lim α→0 F GW α,p,q (µ, ν) = (W pq (µ A , ν B )) q (3.35) lim α→1 F GW α,p,q (µ, ν) = (GW pq (µ X , ν Y )) q (3.36)
Proof of this proposition can be found in Section 6.1.5. This result shows that F GW can revert to one of the other distances and thus acts as a generalization of Wasserstein and Gromov-Wasserstein distances, as claimed in the discrete case section.

Geodesic properties

One desirable property in OT is the underlying geodesics defined by the mass transfer between two probability distributions. These properties are useful in order to define dynamic formulation of OT problems. This dynamic point of view is inspired by fluid dynamics and finds its origin in the Wasserstein context with [START_REF] Benamou | [END_REF]]. Various applications in machine learning can be derived from this formulation: interpolation along geodesic paths was used in computer graphics for color or illumination interpolations [Bonneel 2011]. More recently, [START_REF] Chizat | [END_REF]] used Wasserstein gradient flows in an optimization context, deriving global minima results for non-convex particles gradient descent. In [Zhang 2018] authors used Wasserstein gradient flows in the context of reinforcement learning for policy optimization.

The main idea of this dynamic formulation is to describe the optimal transport problem between two measures as a curve in the space of measures minimizing its total length. We first describe some generality about geodesic spaces and recall classical results for dynamic formulation in both Wasserstein and Gromov-Wasserstein contexts. In a second part, we derive new geodesic properties in the F GW context.

Geodesic spaces Let (X , d X ) be a metric space and x, y two points in X . We say that a curve w : [0, 1] → X joining the endpoints x and y (i.e. with w(0) = x and w(1

) = y) is a constant speed geodesic if it satisfies d X (w(t), w(s)) ≤ |t -s|d(w(0), w(1)) = |t -s|d X (x, y) for t, s ∈ [0, 1]. Moreover, if (X , d X
) is a length space (i.e. if the distance between two points of X is equal to the infimum of the lengths of the curves connecting these two points) then the converse is also true and a constant speed geodesic satisfies d X (w(t), w(s)) = |t -s|d X (x, y). It is easy to compute distances along such curve as they are directly embedded into R.

In the Wasserstein context, if the ground space is a complete separable, locally compact length space and if the endpoints of the geodesic are given, then there exists a geodesic curve. Moreover, if the transport between the endpoints is unique then there is a unique displacement interpolation between the endpoints (see Corollary 7.22 and 7.23 in [Villani 2008]). For example, if the ground space is R d and the distance between the points is measured via the . 2 norm, then the geodesics exist and are uniquely determined (note that this can be generalized to costs of the form c(x, y) = h(y -x) where h is strictly convex). In the Gromov-Wasserstein context, there always exists constant speed geodesics as long as the endpoints are given. These geodesics are unique modulo strong isomorphisms (see [Sturm 2012]).

The F GW case In this paragraph, we suppose that Ω = R d . We are interested in finding a geodesic curve in the space of structured objects i.e. a constant speed curve of structured objects joining two structured objects. As for Wasserstein and Gromov-Wasserstein, the structured object space endowed with the Fused Gromov-Wasserstein distance maintains some geodesic properties. The following result proves the existence of such a geodesic and characterizes it:

Theorem 3.5.3 (Constant speed geodesic.). Let p ≥ 1 and (X × Ω, d X , µ 0 ) and (Y × Ω, d Y , µ 1 ) in S p (R d ).
Let π * be an optimal coupling for the Fused Gromov-Wasserstein distance between µ 0 , µ 1 and t ∈ [0, 1]. We equip R d with m norm for m ≥ 1.

We define

η t : X × Ω × Y × Ω → X × Y × Ω such that: ∀(x, a), (y, b) ∈ X × Ω × Y × Ω, η t (x, a, y, b) = (x, y, (1 -t)a + tb) (3.37)
Then:

(X × Y × Ω, (1 -t)d X ⊕ td Y , µ t = η t #π * ) t∈[0,1] (3.38) is a constant speed geodesic connecting (X × Ω, d X , µ 0 ) and (Y × Ω, d Y , µ 1 ) in the metric space S p (R d ), F GW α,p,1 .
Proof of the previous theorem can be found in Section 6.1.6. In a sense this result combines the geodesics in the Wasserstein space and in the space of all mm-spaces since it suffices to interpolate the distances in the structure space and the features to construct a geodesic. The main interest is that it defines the minimum path between two structured objects. For example, considering two discrete structured objects represented by the measures µ 0 = n i=1 h i δ (xi,ai) and µ 1 = m j=1 g j δ (yj ,bj ) , the interpolation path is given for t ∈ [0, 1] by the measure µ t = n i=1 m j=1 π * (i, j)δ (xi,yj ,(1-t)ai+tbj ) where π * is an optimal coupling for the F GW distance. However this geodesic is difficult to handle in practice since it requires the computation of the cartesian product X 0 × X 1 . The Fréchet mean defined in Section 3.3.2 seems to be more suited in practice. The proper definition and properties of velocity fields associated to this geodesic is postponed to further works.

Discussion and conclusion

Countless problems in machine learning involve structured data, usually stressed in light of the graph formalism. We consider here labeled graphs enriched by an histogram, which naturally leads to represent structured data as probability measures in the joint space of their features and structures. Widely known for their ability to meaningfully compare probability measures, transportation distances are generalized in this chapter so as to be suited in the context of structured data, motivating the so-called Fused Gromov-Wasserstein distance. We theoretically prove that it defines indeed a distance on structured data, and consequently on graphs of arbitrary sizes. F GW provides a natural framework for analysis of labeled graphs as we demonstrate on classification, where it reaches and surpasses most of the time the state-of-the-art performances, and in graph-based k-means where we develop a novel approach to represent the clusters centroids using a barycentric formulation of F GW . We believe that this metric can have a significant impact on challenging graph signal analysis problems.

While we considered a unique measure of distance between nodes in the graph structure (shortest path), other choices could be made with respect to the problem at hand, or eventually learned in an end-to-end manner. The same applies to the distance between features. We also envision a potential use of this distance in deep learning applications where a distance between graph is needed (such as graph auto-encoders). Another line of work will also try to lower the computational complexity of the underlying optimization problem to ensure better scalability to very large graphs.

Chapter 4

The Gromov-Wasserstein problem in Euclidean spaces

"Is it all right if I go out there?" "Sure,"

Summary of the contributions

This chapter is based on the paper [Vayer 2019b] and addresses the problem of GW in Euclidean spaces. Recently used in various machine learning contexts, the Gromov-Wasserstein distance allows for comparing distributions which supports do not necessarily lie in the same metric space. However, this optimal transport distance requires solving a complex non convex quadratic program which is most of the time very costly both in time and memory. Contrary to GW , the Wasserstein distance enjoys several properties (e.g. duality) that permit large scale optimization. Among those, the solution of W on the real line, that only requires sorting discrete samples in 1D, allows defining the Sliced Wasserstein (SW ) distance. This first part of this chapter presents a new divergence based on GW akin to SW . More precisely the contributions of are the following:

• We derive the first closed form solution for GW when dealing with discrete 1D distributions, based on a new result for the related quadratic assignment problem (Theorem 4.1.1 and Theorem 4.1.2).

• Based on this result we define a novel OT discrepancy which can deal with large scale distributions via a slicing approach and we show how it relates to the GW distance while being O(n log(n)) to compute.

• We illustrate the behavior of this so called Sliced Gromov-Wasserstein (SGW ) discrepancy in experiments where we demonstrate its ability to tackle similar problems as GW while being several order of magnitudes faster to compute.

The second part of this chapter is more prospective and tackle the problem of probability distributions which supports lie on Euclidean spaces with, potentially, different dimensions. This part investigate the regularity of GW optimal transport plan in the cases of inner product similarities and Euclidean distances. The contributions of this part are, in summary:

• We show that the GW problem in Euclidean spaces is equivalent to jointly solve a linear transportation problem and a "alignment" problem (Theorem 4.2.1 and Theorem 4.2.5).

• We give necessary conditions under which a GW optimal transport plan is supported on a deterministic function (Theorem 4.2.3 and Proposition 4.2.4). This allows to derive a closed-form expression for GW with inner product similarities between 1D probability distributions (not necessarily discrete, see Theorem 4.2.4).

• We study the Gromov-Monge problem in Euclidean spaces, and in particular the linear Gromov-Monge problem for which we exhibit a closed-form expression between Gaussian distributions (Theorem 4.2.6).

Sliced Gromov-Wasserstein

Introduction

As described in Chapter 2 the linear optimal transport problem aims at defining ways to compare probability distributions, through e.g. the Wasserstein distance. It has proved to be very useful for a wide range of machine learning tasks including generative modelling (Wasserstein GANs [Arjovsky 2017]), domain adaptation [Courty 2017] or supervised embeddings for classification purposes [START_REF] Huang | [END_REF]]. However one limitation of this approach is that it implicitly assumes aligned distributions, i.e. that lie in the same metric space or at least between spaces where a meaningful distance across domains can be computed. From another perspective, the Gromov-Wasserstein distance benefits from more flexibility when it comes to the more challenging scenario where heterogeneous distributions are involved, i.e. distributions which supports do not necessarily lie on the same metric space. It only requires modelling the topological or relational aspects of the distributions within each domain in order to compare them. As such, it has recently received a high interest in the machine learning community, solving learning tasks such as heterogenous domain adaptation [START_REF][END_REF]], deep metric alignment [Ezuz 2017], graph classification (see Chapter 3 for more details) or generative modelling [Bunne 2019].

OT is known to be a computationally difficult problem: the Wasserstein distance involves a linear program that most of the time prevents its use to settings with more than a few tens of thousands of points. For medium to large scale problems, some methods relying e.g. on entropic regularization or dual formulation (as seen in Chapter 2) have been investigated in the past years. Among them, one builds upon the mono-dimensional case where computing the Wasserstein distance can be trivially solved in O(n log n) by sorting points in order and pairing them from left to right. While this 1D case has a limited interest per se, it is one of the main ingredients of the Sliced Wasserstein distance [START_REF] Rabin | [END_REF]]: high-dimensional data are linearly projected into sets of mono-dimensional distributions, the sliced Wasserstein distance being the average of the Wasserstein distances between all projected measures. This framework provides an efficient algorithm that can handle millions of points and has similar properties to the Wasserstein distance [START_REF] Bonnotte | Unidimensional and Evolution Methods for Optimal Transportation[END_REF]]. As such, it has attracted attention and has been successfully used in various tasks such as barycenter computation [START_REF] Bonneel | [END_REF]], classification [START_REF] Kolouri | [END_REF]] or generative modeling [Kolouri 2019b, Deshpande 2018[START_REF] Liutkus | [END_REF], Wu 2019].

Regarding GW , the optimization problem is a non-convex quadratic program, with a prohibitive computational cost for problems with more than a few thousands of points: the number of terms grows quadratically with the number of samples and one cannot rely on a dual formulation as for Wasserstein. However several approaches have been proposed to tackle its computation. Initially approximated by a linear lower bound, GW was thereafter estimated through an entropy regularized version that can be efficiently computed by iterating Sinkhorn projections (see Chapter 2) or using a conditional gradient scheme relying on linear program OT solvers (see Chapter 3). However, all these methods are still too costly for large scale scenarii. In this section, we propose a new formulation related to GW that lowers its computational cost. To that extent, we derive a novel OT discrepancy called Sliced Gromov-Wasserstein (SGW ). It is similar in spirit to the Sliced Wasserstein distance as it relies on the exact computation of 1D GW distances of distributions projected onto random directions. We notably provide the first 1D closed-form solution of the GW problem by proving a new result about the Quadratic Assignment Problem for matrices that are squared euclidean distances of real numbers. Computation of SGW for discrete distributions of n points is O(L n log(n)), where L is the number of sampled directions. This complexity is the same as the Sliced-Wasserstein distance and is even lower than computing the value of GW which is O(n 3 ) for a known coupling (once the optimization problem solved) in the general case [START_REF] Peyré | [END_REF]]. Experimental validation shows that SGW retains various properties of GW while being much cheaper to compute, allowing its use in difficult large scale settings such as large mesh matching or generative adversarial networks.

From 1D GW to Sliced Gromov-Wasserstein

We first provide and prove a solution for an 1D Quadratic Assignement Problem with a quasilinear time complexity of O(n log(n)). This new special case of the QAP is shown to be equivalent to the hard assignment version of GW , called the Gromov-Monge (GM ) problem, with squared Euclidean cost for distributions lying on the real line. We also show that, in this context, solving GM is equivalent to solving GW . We derive a new discrepancy named Sliced Gromov-Wasserstein (SGW ) that relies on these findings for efficient computation.

Solving a Quadratic Assignment Problem in 1D

In Koopmans-Beckmann form [Koopmans 1957] a QAP takes as input two n × n matrices A = (a ij ), B = (b ij ). The goal is to find a permutation σ ∈ S n , the set of all permutations of [[n]], which minimizes the objective function n i,j=1 ai,jb σ(i),σ(j) . In full generality this problem is NP-hard (see Section 2.2.3 for more details). The following theorem is a new result about QAP and states that it can be solved in polynomial time when A and B are squared Euclidean distance matrices of sorted real numbers:

Theorem 4.1.1 (A new special case for the Quadratic Assignment Problem). For real numbers

x 1 < • • • < x n and y 1 < • • • < y n , min σ∈Sn i,j -(x i -x j ) 2 (y σ(i) -y σ(j) ) 2 (4.1)
is achieved either by the identity permutation σ(i) = i (Id) or the anti-identity permutation σ(i) = n + 1 -i (anti -Id). In other words:

∃σ ∈ {Id, anti -Id}, σ ∈ arg min σ∈Sn i,j -(x i -x j ) 2 (y σ(i) -y σ(j) ) 2 (4.2)
To the best of our knowledge, this result is new. It states that if one wants to find the best one-to-one correspondence of real numbers such that their pairwise distances are best conserved, it suffices to sort the points and check whether the identity has a better cost than the anti-identity. Proof of this theorem can be found in Section 6.2.1. We postulate that this result also holds for a ij = |x i -x j | k and b ij = -|y i -y j | k with any k ≥ 1 but leave this study for future works.

Gromov-Wasserstein distance on the real line

When n = m and a i = b j = 1 n , one can look for the hard assignment version of the GW distance resulting in the Gromov-Monge problem [Mémoli 2018] associated with the following GM distance:

GM 2 (c X , c Y , µ, ν) = min σ∈Sn 1 n 2 i,j c X (x i , x j ) -c Y (y σ(i) , y σ(j) ) 2 (4.3)
where σ ∈ S n is a one-to-one mapping [[n]] → [[n]]. Interestingly when the permutation σ is known, the computation of the cost is O(n 2 ) which is far better than O(n 3 ) for the general GW case. It is easy to see that this problem is equivalent to minimizing n i,j=1 a i,j b σ(i),σ(j) with a ij = c X (x i , x j ) and b ij = -c Y (y σ(i) , y σ(j) ). Indeed we have:

i,j c X (x i , x j ) -c Y (y σ(i) , y σ(j) ) 2 = i,j c X (x i , x j ) 2 + i,j c Y (y σ(i) , y σ(j) ) 2 -2 i,j c X (x i , x j )c Y (y σ(i) , y σ(j) ) = i,j c X (x i , x j ) 2 + i,j c Y (y i , y j ) 2 -2 i,j c X (x i , x j )c Y (y σ(i) , y σ(j) )
So that only the term -2 i,j c X (x i , x j )c Y (y σ(i) , y σ(j) ) depends on σ. Thus, when squared Euclidean costs are used for distributions lying on the real line, Theorem 4.1.1 exactly recovers the solution of the GM problem defined in equation (4.3). As matter of consequence, Theorem 4.1.1 provides an efficient way of solving the Gromov-Monge problem. Moreover, this theorem also allows finding a closed-form for the GW distance. Indeed, some recent advances in graph matching state that, under some conditions on A and B, the assignment problem is equivalent to its soft-assignment counterpart [START_REF] Maron | [END_REF]]. This way, using both Theorem 4.1.1 and [START_REF] Maron | [END_REF]], one can find a solvable case for the GW distance as stated in the following theorem: Theorem 4.1.2 (Equivalence between GW and GM for discrete measures). Let µ ∈ P(R p ), ν ∈ P(R q ) be discrete probability measures with same number of atoms and uniform weights, i.e. µ =

1 n n i=1 δ xi , ν = 1 n n i=1 δ yi with x i ∈ R p , y i ∈ R q . For x ∈ R p we note x 2,p = p i=1 |x i | 2 the 2 norm on R p (same for R q ). Let c X (x, x ) = x -x 2 2,p , c Y (y, y ) = y -y 2 2,q . Then: GW 2 (c X , c Y , µ, ν) = GM 2 (c X , c Y , µ, ν) (4.4) Moreover when p = q = 1, i.e. c X (x, x ) = c Y (x, x ) = |x -x | 2 , and x 1 < • • • < x n and y 1 < • • • < y n
the optimal values are achieved by considering either the identity or the anti-identity permutation.

A detailed proof is provided in Section 6.2.2. Note also that, while both possible solutions for problem (4.3) can be computed in O(n log(n)), finding the best one requires the computation of the cost which seems, at first sight, to have a O(n 2 ) complexity. However, under the hypotheses of squared Euclidean distances, the cost can be computed in O(n). Indeed, in this case, one can develop the sum in equation ( 4.3) to compute it in O(n) operations using binomial expansion (see details in Section 6.2.3) so that the overall complexity of finding the best assignment and computing the cost is O(n log(n)) which is the same complexity as the Wasserstein for 1D distributions.

Sliced Gromov-Wasserstein discrepancy Theorem 4.1.2 can be put in perspective with the Wasserstein distance for 1D distributions which is achieved by the identity permutation when points are sorted [START_REF] Peyré | [END_REF]]. As explained in Chapter 2, this result was used to approximate the Wasserstein distance between measures of R q using the so called Sliced Wasserstein (SW) distance [START_REF] Bonneel | [END_REF]]. The main idea is to project the points of the measures on lines of R q where computing a Wasserstein distance is easy since it only involves a simple sort and to average these distances. In the same philosophy we build upon Theorem 4.1.2 to define a "sliced" version of the GW distance. In the following, we consider µ ∈ P(R p ), ν ∈ P(R q ) be probability distributions (not necessarily discrete).

Let S q-1 = {θ ∈ R q | θ 2 = 1} be the q-dimensional hypersphere and λ q-1 the uniform measure on S q-1 . For θ we note P θ the projection on θ, i.e. P θ (x) = x, θ . For a linear map ∆ ∈ R q×p (identified with slight abuses of notation by its corresponding matrix), we define the Sliced Gromov-Wasserstein (SGW) discrepancy as follows:

SGW ∆ (µ, ν) = E θ∼λq-1 [GW 2 2 (P θ #µ ∆ , P θ #ν)] = ˆSq-1 GW 2 2 (d 2 , P θ #µ ∆ , P θ #ν)dλ q-1 (θ) (4.5)
where µ ∆ = ∆#µ ∈ P(R q ). The function ∆ acts as a mapping for a point in R p of the measure µ onto R q . When p = q and when we consider ∆ as the identity map we simply write SGW (µ, ν) instead of SGW Ip (µ, ν). When p < q, one straightforward choice is ∆ = ∆ pad the "uplifting" operator which pads each point of the measure with zeros: ∆ pad (x) = (x 1 , . . . , x p , 0, . . . , 0 q-p ). The procedure is illustrated in Fig In general fixing ∆ implies that some properties of GW , such as the rotational invariance, are lost. Consequently, we also propose a variant of SGW that does not depends on the choice of ∆ called Rotation Invariant SGW (RISGW ) and expressed for p ≥ q as the following:

RISGW (µ, ν) = min ∆∈Vp(R q ) SGW ∆ (µ, ν).
(4.6)

y 1 y 2 y 3 y 4 x 1 x 4 x 2 x 3 P θ #ν P θ #(∆#µ) for θ ∈ S q-1 y θ 1 y θ 2 y θ 3 y θ 4 x θ 1 x θ 2 x θ 3 x θ 4 Figure 4
.1: Example in dimension p = 2 and q = 3 (left) that are projected on the line (right). The solution for this projection is the anti-diagonal coupling.

We propose to minimize SGW ∆ with respect to ∆ in the Stiefel manifold V p (R q ) [Absil 2009] which is defined as

V p (R q ) = {∆ ∈ R q×p |∆ T ∆ = I p }.
It can be seen as finding an optimal projector of the measure µ [Paty 2019[START_REF] Deshpande | [END_REF]]. This formulation comes at the cost of an additional optimization step but allows recovering one key property of GW. When p = q this encompasses for e.g. all rotations of the space, making RISGW invariant by rotation. Interestingly enough, SGW holds various properties of the GW distance as summarized in the following theorem:

Theorem 4.1.3 (Properties of SGW ).

• For all ∆, SGW ∆ and RISGW are translation invariant. RISGW is also rotational invariant when p = q, more precisely if Q ∈ O(p) is an orthogonal matrix, RISGW (Q#µ, ν) = RISGW (µ, ν) (same for any Q ∈ O(q) applied on ν).

• SGW and RISGW are pseudo-distances on P(R p ), i.e. they are symmetric, satisfy the triangle inequality and SGW (µ, µ) = RISGW (µ, µ) = 0 .

• Let µ, ν ∈ P(R p ) × P(R p ) be probability distribution with compact supports. If SGW (µ, ν) = 0 then µ and ν are isomorphic for the distance induced by the 1 norm on R p , i.e. d(x, x ) =

p i=1 |x i -x i | for (x, x ) ∈ R p × R p .
In particular this implies:

SGW (µ, ν) = 0 =⇒ GW 2 (d, d, µ, ν) = 0 (4.7)
(with a slight abuse of notation we identify the matrix Q by its linear application). A proof of this theorem can be found in Section 6.2.4. This theorem states that if SGW vanishes then measures must be isomorphic, as it is the case for GW . It states also that RISGW holds most of the properties of GW in term of invariants.

Remark 4.1.1. The ∆ map can also be used in the context of the Sliced Wasserstein distance so as to define SW ∆ (µ, ν), RISW (µ, ν) for µ, ν ∈ P(R p ) × P(R q ) with p = q. Please note that from a purely computational point of view, complexities of these discrepancies are the same as SGW and RISGW when µ and ν are discrete measures with the same number of atoms n = m, and uniform weights. Also, unlike SGW and RISGW , these discrepancies are not translation invariant. This approach was studied in [Lai 2014] for the case p = q in the context of point cloud registration. More details are given in Section 6.2.5. Algorithm 6 Sliced Gromov-Wasserstein for discrete measures 1: p < q, µ = 1 n n i=1 δ xi ∈ P(R p ) and ν = 1 n n i=1 δ yj ∈ P(R q ) 2: ∀i, x i ← ∆(x i ), sample uniformly (θ l ) l=1,...,L ∈ S q-1 3: for l = 1, . . . , L do 4:

Sort ( x i , θ l ) i and ( y j , θ l ) j in increasing order 5:

Solve (4.3) for reals ( x i , θ l ) i and ( y j , θ l ) j , σ θ l (Anti-Id or Id is a solution) 6: end for 7: return 1

n 2 L L l=1 n i,k=1 xi-x k , θ l 2 -y σ θ l (i) -y σ θ l (k) , θ l 2 2
Computational aspects In the following µ, ν are discrete measures with the same number of atoms n = m, and uniform weights, i.e. µ = 1 n n i=1 δ xi , ν = 1 n n i=1 δ yi with x i ∈ R p , y i ∈ R q so that we can apply Theorem 4.1.2. Similarly to Sliced Wasserstein, SGW can be approximated by replacing the integral by a finite sum over randomly drawn directions. In practice we compute SGW as the average of GW 2 2 projected on L directions θ. While the sum in (4.5) can be implemented with libraries such as Pykeops [Charlier 2018], Theorem 4.1.2 shows that computing (4.5) is achieved by an O(n log(n)) sorting of the projected samples and by finding the optimal permutation which is either the identity or the anti identity. Moreover computing the cost is O(n) for each projection as explained previously. Thus the overall complexity of computing SGW with L projections is O(Ln(p+q)+Ln log(n)+Ln) = O(Ln(p+q +log(n))) when taking into account the cost of projections. The pseudo-code for SGW is presented in Algorithm 6 Note that these computations can be efficiently implemented in parallel on GPUs with modern toolkits such as Pytorch [START_REF] Paszke | [END_REF]].

The complexity of solving RISGW is higher but one can rely on efficient algorithms for optimizing on the Stiefel manifold [Absil 2009] that have been implemented in several toolboxes [Townsend 2016, Meghwanshi 2018]. Note that each iteration in a manifold gradient decent requires the solution of SGW , that can be computed and differentiated efficiently with the frameworks described above. Moreover, the optimization over the Stiefel manifold does not depend on the number of points but only on the dimension d of the problem so that overall complexity is n iter (Ln(d + log(n)) + d 3 ), which is affordable for small d. In practice, we observed in the numerical experiments that RISGW converges in few iterations (the order of 10).

Experimental results

The goal of this section is to validate SGW and its rotational invariant on both quantitative (execution time) and qualitative sides. All the experiments were conducted on a standard computer equipped with a NVIDIA Titan X GPU.

SGW and RISGW on spiral dataset

As a first example, we use the spiral dataset from sklearn toolbox and compute GW , SGW and RISGW on n = 100 samples with L = 20 sampled lines for different rotations of the target distribution. The optimization of ∆ on the Stiefel manifold is performed using Pymanopt [Townsend 2016] with automatic differentiation with autograd [Maclaurin 2015]. Some examples of empirical distributions are available in Figure 4.2 (left). The mean value of GW , SGW and RISGW are reported on Figure 4.2 (right) where we can see that RISGW is invariant to rotation as GW whereas SGW with ∆ = I p is clearly not.

Runtimes comparison

We perform a comparison between runtimes of SGW , GW and its entropic counterpart [Solomon 2016]. We calculate these distances between two 2D random measures of n ∈ {1e2, ..., 1e6} points. For SGW , the number of projections L is taken from {50, 200}. We use the Python Optimal Transport (POT) toolbox [START_REF] Flamary | [END_REF]] to compute GW distance on CPU. For entropic-GW we use the Pytorch GPU implementation from [Bunne 2019] that uses the log-stabilized Sinkhorn algorithm [Schmitzer 2016] with a regularization parameter ε = 100. For SGW , we implemented both a Numpy implementation and a Pytorch implementation running on GPU. Figure 4.3 illustrates the results. SGW is the only method which scales w.r.t. the number of samples and allows computation for n > 10 4 . While entropic-GW uses GPU, it is still slow because the gradient step size in the algorithm is inversely proportional to the regularization parameter [START_REF] Peyré | [END_REF]] which highly curtails the convergence of the method. On CPU, SGW is two orders of magnitude faster than GW . On GPU, SGW is five orders of magnitude faster than GW and four orders of magnitude faster than entropic GW . Still the slope of both GW implementations are surprisingly good, probably due to their maximum iteration stopping criteria. In this experiment we were able to compute SGW between 10 6 points in 1s. Finally note that we recover exactly a quasi-linear slope, corresponding to the O(n log(n)) complexity for SGW .

Meshes comparison

In the context of computer graphics, GW can be used to quantify the correspondances between two meshes. A direct interest is found in shape retrieval, search, exploration or organization of databases. In order to recover experimentally some of the desired properties of the GW distance, we reproduce an experiment originally conducted in [Rustamov 2013] and presented in [Solomon 2016] with the use of entropic-GW .

From a given time series of 45 meshes representing a galloping horse, the goal is to conduct a multidimensional scaling (MDS) of the pairwise distances, computed with SGW between the meshes, that allows ploting each mesh as a 2D point. As one can observe in Figure 4.4, the cyclical nature of this motion is recovered in this 2D plot, as already illustrated in [Solomon 2016] with the GW distance. Each horse mesh is composed of approximately 9, 000 vertices. The average time for computing one distance is around 30 minutes using the POT implementation, which makes the computation of the full pairwise distance matrix impractical (as already mentioned in [Solomon 2016]). In contrast, our method only requires 25 minutes to compute the full distance matrix, with an average of 1.5s per mesh pair, using our CPU implementation. This clearly highlights the benefits of our method in this case.

SGW as a generative adversarial network (GAN) loss In a recent paper [Bunne 2019], Bunne and colleagues propose a new variant of GAN between incomparable spaces, i.e. of different dimensions.

In contrast with classical divergences such as Wasserstein, they suggest to capture the intrinsic relations between the samples of the target probability distribution by using GW as a loss for learning. More formally, this translates into the following optimization problem over a desired generator G:

G * = arg min GW 2 2 (c X , c G(Z) , µ, ν G ), (4.8)
where Z is a random noise following a prescribed low-dimensional distribution (typically Gaussian), G(Z) performs the uplifting of Z in the desired dimensional space, and c G(Z) is the corresponding metric. µ and ν G correspond respectively to the target and generated distributions, that we might want to align in the sense of GW . Following the same idea, and the fact that sliced variants of the Wasserstein distance have been successfully used in the context of GAN [Deshpande 2018], we propose to use SGW instead of GW as a loss for learning G. As a proof of concept, we reproduce the simple toy example of [Bunne 2019].

Those examples consist in generating 2D or 3D distributions from target distributions either in 2D or 3D spaces (Figure 4.5 and Figure 4.6). These distributions are formed by 3, 000 samples. We do not use their adversarial metric learning as it might confuse the objectives of this experiment and as it is not required for these low dimensional problems [Bunne 2019]. The generator G is designed as a simple multilayer perceptron with 2 hidden layers of respectively 256 and 128 units with ReLu activation functions, and one final layer with 2 or 3 output neurons (with linear activation) as output, depending on the experiment. The Adam optimizer is used, with a learning rate of 2.10 -4 and β 1 = 0.5, β 2 = 0.99. The convergence to a visually acceptable solution takes a few hundred epochs. Contrary to [Bunne 2019], we directly back-propagate through our loss, without having to explicit a coupling matrix and resorting to the envelope Theorem. Compared to [Bunne 2019] and the use of entropic-GW , the time per epoch is more than one order of magnitude faster, as expected from previous experiment.

Discussion and conclusion

In this section we establish a new result about Quadratic Assignment Problem when matrices are squared euclidean distances on the real line, and use it to state a closed-form expression for GW between monodimensional measures. Building upon this result we define a new similarity measure, called the Sliced Gromov-Wasserstein and a variant Rotation-invariant SGW and prove that both conserve various properties of the GW distance while being cheaper to compute and applicable in a large-scale setting.

Notably SGW can be computed in 1 second for distributions with 1 million samples each. This paves the way for novel promising machine learning applications of optimal transport between metric spaces. Yet, several questions are raised in this work. Notably, our method perfectly fits the case when the two distributions are given empirically through samples embedded in an Hilbertian space, that allows for projection on the real line. This is the case in most of the machine learning applications that use the Gromov-Wasserstein distance. However, when only distances between samples are available, the projection operation can not be carried anymore, while the computation of GW is still possible. One can argue that it is possible to embed either isometrically those distances into a Hilbertian space, or at least with a low distortion, and then apply the presented technique. Our future line of work considers this option, as well as a possible direct reasoning on the distance matrix. For example, one should be able to consider geodesic paths (in a graph for instance) as the equivalent appropriate geometric object related to the line. This constitutes the direct follow-up of this work, as well as a better understanding of the accuracy of the estimated discrepancy with respect to the ambiant dimension and the projections number.

Regularity & formulations of GW problems in Euclidean spaces

Introduction

In the previous part we built upon the special case of 1D probability discrete measures. We consider in this section general probability measures µ ∈ P(R p ) and ν ∈ P(R q ) supported on Euclidean spaces R p and R q with (possibly) p = q. The corresponding inner products are denoted by x, x p (resp. y, y q ) for vectors in R p (resp. R q ) associated with the Euclidean norms which are denoted both by . 2 to avoid overloading notations.

We tackle in this section the problem of the regularity of the optimal transport plans of GW 2 in the Euclidean setting. More precisely we consider the following problem: Problem 1. Let µ ∈ P(R p ), ν ∈ P(R q ). Can we find an deterministic transport map to the GW 2 problem? More precisely does the following statement hold? ∃T : R p → R q such that T #µ = ν and γ T = (id × T )#µ is optimal for the GW 2 problem:

inf π∈Π(µ,ν) ˆˆ|c X (x, x ) -c Y (y, y )| 2 dπ(x, y)dπ(x , y ) (4.9)
The Euclidean setting is motivated by the linear transportation theory where the first regularity solution for Optimal Transport was proved by Brenier for probability measures in R p (see Chapter 2). In this case we recall that the optimal transport plan γ T between two probability measure (µ, ν) ∈ P(R p ) × P(R p ), with µ "well behaved" and with cost c(x, y) = xy 2 2 , is unique and supported by a map T such that γ T = (id × T )#µ. The purpose of this section is to show that the Euclidean setting is also quite suited for the GW case. The problem of regularity of GW optimal transport plans was first addressed by Sturm in his seminal work about GW [Sturm 2012, Challenges 3.6]. More precisely Sturm asks the following question: are there some "nice" spaces in which we are able to prove Brenier's like results for GW ? We give in this section some partial answers to this query by considering the two cases where c X , c Y are defined by the inner products or by the squared Euclidean distances in each space.

The main results of this section is to derive equivalent formulations of the GW in these two cases. More precisely we show that solving GW is equivalent to jointly solve a linear transportation problem and a "alignment" problem. As such the regularity of GW optimal plans can be observed in the light of these "dual problems". As another consequence it allows also to derive algorithmic solutions for the GW problems in Euclidean spaces based on simple Block Coordinate Descent procedures. This section is organized as follow:

(i) In Section 4.2.2 we consider the case where c X , c Y are defined by the inner products in each space.

Providing that the source probability measure is regular with respect to the Lebesgue measure we give a sufficient condition for the existence of a deterministic optimal transport plan, i.e. supported on a deterministic function T . We show that this function is of the form ∇u • P where u is a convex function and P is a linear application which can be seen as a global transformation "realigning" the probability measures in the same space. We use this formulation to show that the GW distance between 1D probability measures admits a closed-form solution. More precisely we show that the optimal coupling is determined by the cumulative and the anti-cumulative distribution functions of the source distribution. We further discuss the difference between the linear OT problem W 2 and the GW problem when the target measure is a perturbed version of the source measure.

(ii) In Section 4.2.3 we consider c X , c Y as the squared Euclidean distances in each space. We show that this setting is equivalent to a maximization of a convex function on Π(µ, ν). We use the Fenchel-Legendre duality in the space of measures to derive a problem equivalent to that of Gromov-Wasserstein. We further analyse it and show that the regularity of optimal transport plans is more complicated to state than in the previous case.

(iii) In Section 4.2.4 we use the previous formulations to derive efficient numerical solutions for the GW problem based on Block Coordinate Descent. We show that these procedures compare favourably with respect to standard solvers such as Conditional Gradient (see Chapter 3) or with entropic regularization (see Chapter 2).

(iv) We conclude in Section 4.2.5 by considering the Gromov-Monge problem in Euclidean spaces, which is the exact counterpart of the Monge problem of linear transportation but in the Gromov-Wasserstein context. We discuss the special case of the Gromov-Monge between Gaussian measures and we show that this problem admits a closed-form solution when restricting to linear push-forward. We give geometric interpretations of this result and compare the optimal push-forward with the standard optimal map of linear OT theory in the case of Gaussian measures (see Chapter 2).

This section is more prospective and somehow opens more doors than it closes. We hope that it will paves the path for further interesting works on this topic and believe that it could help for bridging the gap between the understanding of the linear OT problem and the Gromov-Wasserstein theory. We have chosen to include in the main text some proofs that are reasonably long and that we consider interesting for the overall understanding of the section. The other proofs, which require more space, are postponed to Section 6.2.

The inner product case

To encompass the two cases described in the introduction we consider the following lemma (a proof can be found in Section 6.2.6): Lemma 4.2.1. For a coupling π ∈ Π(µ, ν) we note:

J 2 (c X , c Y , π) def = ˆX ×X ˆY×Y |c X (x, x ) -c Y (y, y )| 2 dπ(x, y)dπ(x , y ) (4.10)
the GW 2 loss. Suppose that there exist scalars a, b, c such that c X (x, x ) = a x 2 2 + b x 2 2 + c x, x p and c Y (y, y ) = a y 2 2 + b y 2 2 + c y, y q . Then:

J 2 (c X , c Y , π) = C µ,ν -2Z(π) (4.11)
where

C µ,ν = ´c2 X dµdµ + ´c2 Y dνdν -4ab
´ x 2 2 y 2 2 dµ(x)dν(y) and:

Z(π) = (a 2 + b 2 ) ˆ x 2 2 y 2 2 dπ(x, y) + c 2 ˆyx T dπ(x, y) 2 F + (a + b)c ˆ x 2 2 E Y ∼ν [Y ], y q + y 2 2 E X∼µ [X], x p dπ(x, y) (4.12)
In this section we study the GW problem with c X = x, x p and c Y = y, y q . This corresponds to a, b = 0 and c = 1 case of Lemma 4.2.1. With a small abuse of notation we will denote by P ∈ R q×p both the linear application P : R p → R q and its associated matrix. Moreover we will often make no distinction between a vector x ∈ R p and the matrix associated to x ∈ R p×1 such that x T ∈ R 1×p . The next theorem gives a equivalent formulation of GW 2 in this context:

Theorem 4.2.1 (Equivalence of GW for the inner product case). Let µ ∈ P(R p ), ν ∈ P(R q ) with ´ x 4 2 dµ(x) < +∞,

´ y 4 2 dν(y) < +∞. Suppose without loss of generality that p ≥ q and let:

F p,q def = {P ∈ R q×p | P F = √ p} (4.13)
Then problems:

inf π∈Π(µ,ν) ˆˆ x, x p -y, y q 2 dπ(x, y)dπ(x , y ) (innerGW) sup π∈Π(µ,ν) sup P∈Fp,q ˆ Px, y q dπ(x, y) (MaxOT)
are equivalent. In other words, π * ∈ Π(µ, ν) is an optimal solution of (innerGW) if and only if π * is an optimal solution of (MaxOT).

Remark 4.2.1. The condition ´ x 4 2 dµ(x) < +∞,

´ y 4 2 dν(y) < +∞ suffices to prove that both (innerGW) and (MaxOT) are finite and that (MaxOT) admits an optimal solution π * ∈ Π(µ, ν) (we postponed this study to Lemma 6.2.7 in Section 6.2.7).

This theorem gives another interesting formulation of the Gromov-Wasserstein problem. It proves that

GW is equivalent to a linear OT problem combined with an "alignment" of the measures µ and ν on the same space using a linear application P. The set F p,q can be regarded as the set of matrices with fixed Schatten 2 norms, that is P ∈ F p,q if σ(P) 2 = √ p where σ(P) is a vector containing the singular values of P. When p = q any orthogonal matrix O ∈ O(p) is in F p,q since O F = tr(O T O) = tr(I p ) = √ p. More generally when p > q any matrix in the Stiefel manifold ∆ ∈ V p (R q ) is an element of F p,q since ∆ T ∆ = I p . Interestingly enough, the problem (MaxOT) can be related to the work of Alvarez and coauthors [Alvarez-Melis 2019] where they proposed a linear optimal transport problem which takes into account a latent global transformation of the measures. More precisely they consider two probability measures (µ, ν) ∈ P(R p ) × R p (i.e. p = q) and propose two minimize the following problem: where in (*) we used x T P T Px ∈ R, in (**) that the trace is invariant by cyclical permutation and in (***) the linearity of the trace. Finally we used that tr(P T P) = P 2 F = p by hypothesis. However note that in general both problems may differ since the µ-whitened property does not hold in general.

InvOT (µ, ν) = min π∈Π(µ,ν) min P F = √ p ˆ Px -y 2 2 dπ(x,
Theorem 4.2.1 is based on the following generalization of the Frobenius norm duality to the continuous setting: Lemma 4.2.2. For any µ ∈ P(R p ), ν ∈ P(R q ) and π ∈ Π(µ, ν). Then:

sup P F = √ p ˆ Px, y q dπ(x, y) = √ p ˆyx T dπ(x, y) F (4.17)
This supremum is achieved for where in (*) we used that y T Px ∈ R, in (**) we used the cyclical permutation invariance of the trace and in (***) its linearity. Hence sup

P * = √ p ´yx T dπ(x,y) F ´yx T dπ(x,
P F = √ p
´ Px, y q dπ(x, y) = sup

P F = √ p
P, ´yx T dπ(x, y) F . We note V π = ´yx T dπ(x, y). We want to solve:

sup

P F = √ p P, V π F (4.18)
Let P such that P F = √ p. Then by Cauchy-Schwartz (see Memo 4.2.1) P, V π F ≤ P F V π F = √ p V π F . Hence, sup

P F = √ p P, V π F ≤ √ p V π F .
Conversely, take P * = √ p Vπ Vπ F . Then P * F = √ p so sup

P F = √ p P, V π F ≥ P * , V π F = √ p Vπ Vπ F , V π F = √ p V π F which concludes the proof.
Combining Lemma 4.2.1 and Lemma 4.2.2 actually proves Theorem 4.2.1. Indeed using Lemma 4.2.1 we see that (innerGW) is equivalent to maximizing Z(π) = ´yx T dπ(x, y) 2 F over the couplings π ∈ Π(µ, ν) since the other terms are constant. In this way it is equivalent to maximize ´yx T dπ(x, y) F which is equivalent by Lemma 4.2.2 to maximize sup

P∈Fp,q

´ Px, y q dπ(x, y) w.r.t. π.

Regularity of (innerGW) OT plans Theorem 4.2.1 proves that it is equivalent to study the problem (MaxOT) for studying the regularity of GW optimal transport plans. Interestingly enough the problem (MaxOT) echoes the linear transportation problem sup π∈Π(µ,ν)

´ x, y p dπ(x, y) when µ, ν ∈ P(R p ) × P(R p ) which is widely studied in the literature an can be tackled using tools from convexity analysis such as the Legendre transform. The following result due to McCann is particularly useful in this case: Theorem 4.2.2 ( [McCann 1995]). Let µ ∈ P(R p ), ν ∈ P(R p ). Suppose that µ is absolutely continuous with respect to the Lebesgue measure, then there exists a convex function u : R p → R whose gradient ∇u pushes µ forward to ν, i.e.∇u#µ = ν. Moreover ∇u is unique µ a.e.

By noticing that, for all x, y ∈ R p × R p , x, y p ≤ u * (x) + u(y) where u * is the Legendre transform of the convex function u the result of McCann proves that the map ∇u defines an optimal coupling γ = (id × ∇u)#µ for the problem sup π∈Π(µ,ν)

´ x, y p dπ(x, y) between (µ, ν) ∈ P(R p ) × P(R p ). Indeed for any coupling π ∈ Π(µ, ν): ´ x, y p dγ(x, y) for any coupling π which proves that γ is optimal. The idea is to use the same reasoning to find an optimal solution of (MaxOT). In order to invoke McCann's theorem we will need the regularity of the probability measure P#µ for P ∈ F p,q : Memo 4.2.1 (Cauchy-Schwartz inequality). Let Ω be a vector space associated with an inner product , which defines a norm . through x = x, x for x ∈ Ω. The Cauchy-Schwartz inequality reads:

∀x, y ∈ Ω 2 , | x, y | ≤ x y (4.19)
Proposition 4.2.1. Let µ ∈ P(R p ) regular with respect to the Lebesgue measure in R p and a linear map l : R p → R q associated with a matrix L ∈ R q×p

• If p < q (we go from lower to higher dimension), then l#µ ∈ R q is not regular with respect to the Lebesgue measure on R q .

• If p ≥ q (we go from higher to lower dimension), then l#µ ∈ R q is regular with respect to the Lebesgue measure on R q if and only if the linear map l is surjective, that is rank(L) = q.

Proof. We have l#µ(l(R p )) = µ(l -1 (l(R p )) = µ(R p ) = 1 so that l#µ gives measure 1 to the image of l.

However for the first point the image of l is a strict linear subspace of R q and therefore has Lebesgue measure zero. Using Radon-Nykodym theorem this implies that l#µ can not have a density with respect to the Lebesgue on R q . For the second point, suppose that l is surjective. Then rank(L) = q which implies that LL T is invertible so that det(LL T ) = 0. We define J = det(LL T ). Let g be the density of µ with respect to the Lebesgue measure in R p . Then by the coarea formula the density of l#µ with respect to the Lebesgue measure on R q is:

h(y) = ˆl-1 (y) g(x) J dV l -1 (y) (x) (4.22)
where denotes dV l -1 (y) (x) the volume element. Conversely if l is not surjective then rank(L) < q. Then the image of l is a strict linear subspace of R q with Lebesgue measure zero and therefore l#µ can not have a density with respect to the Lebesgue on R q .

In the light of previous results we can give the following sufficient condition so that (innerGW) problem admits an optimal transport plan supported on a deterministic function:

Theorem 4.2.3. Let µ ∈ P(R p ), ν ∈ P(R q ) with ´ x 4 2 dµ(x) < +∞,
´ y 4 2 dν(y) < +∞. Suppose that p ≥ q and that µ is regular with respect to the Lebesgue measure in R p . Suppose that there exists (π * , P * ) an optimal solution of (MaxOT) with P * surjective.

Then there exists u : R q → R convex such that ∇u • P * pushes µ forward to ν. Moreover the coupling γ = (id × ∇u • P * )#µ is optimal for (innerGW). In particular problem 1 holds.

Proof. Let (π * , P * ) be maximizers of (MaxOT) with P * surjective. Using Proposition 4.2.1 we know that P * #µ is regular with respect to the Lebesgue measure on R q . Using Theorem 4.2.2 there exists u : R q → R convex such that ∇u#P * #µ = ν or equivalently ∇u • P * pushes µ forward to ν. Moreover we have:

ˆ P * x, y d dπ * (x, y) 1 ≤ ˆ(u(P * x) + u * (y)) dπ * (x, y) = ˆu(P * x)dµ(x) + ˆu * (y)dν(y) 2 = ˆu(P * x)dµ(x) + ˆu * (y)d(∇u • P * #µ)(y)
= ˆu(P * x)dµ(x) + ˆu * (∇u(P * x))dµ(x)

3 = ˆ P * x, ∇u(P * x) q dµ(x)
where in (1) we used that u is convex which implies u(Px) + u * (y) ≥ Px, y q by Fenchel-Young inequality, in (2) we used ∇u • P * pushes µ forward to ν, in (3) we used that for any x and convex function

u * (x) + u(x) = x, ∇u(x) .
If we define T = ∇u • P * and γ = (id × T )#µ ∈ Π(µ, ν) we can deduce from (3) that:

sup π∈Π(µ,ν)
sup P∈Fp,q ˆ Px, y q dπ(x, y) ≤ ˆ P * x, y q dγ(x, y)

By suboptimality the converse inequality is also true (since γ = (id × T )#µ ∈ Π(µ, ν)). In this way we have:

sup π∈Π(µ,ν)
sup P∈Fp,q ˆ Px, y q dπ(x, y = ˆ P * x, y q dγ(x, y)

Overall the couple (γ, P * ) is optimal for the problem (MaxOT) and γ is optimal for (innerGW) using Theorem 4.2.1 which concludes the proof.

The last result indicates that Monge map of the form ∇u • P where P is a linear application may be of interest to study optimal solutions of (innerGW). When considering p = q and µ, ν ∈ P(R p ) × P(R p ) looking at such maps can generate the optimal Monge map ∇u of linear optimal transport problem sup π∈Π(µ,ν)

´ x, y p dπ(x, y) when P = I p . Note that a couple (u, P) which satisfies both ∇u • P pushes µ onto ν and γ = (id × ∇u • P) is optimal for (innerGW) is not guaranteed to be unique. This difference should be put in perspective with the theory of linear transportation with ground cost c(x, y) = xy 2 2 . Indeed in this context if one finds a mapping T = ∇u with any convex function u then Brenier's theorem states that this mapping is optimal since there is a unique Monge map satisfying this property. This unicity result is particularly interesting e.g. in order to prove that the linear transport between Gaussian measure admits a closed-form expression [Takatsu 2011]. However in our context the map ∇u • P may fail to be unique as shown in the following example: Example 4.2.1. Consider p = q, a source measure µ ∈ P(R p ) and a target measure ν ∈ P(R p ) whose support is invariant by rotation, i.e. such that O#ν = ν for any O ∈ O(p). We can consider e.g. any isotropic Gaussian measure N (m ν , σ 2 I p ). Since O preserves the angles then the problem (innerGW) is invariant by O which implies that any optimal map defined in Theorem 4.2.3 will fail to be unique. More precisely let suppose that ∇u • P pushes µ forward to ν and that γ = (id × ∇u • P) is optimal for (innerGW). Then for any O ∈ O(p), O T • ∇u • P also pushes µ forward to ν since ∇u • P#µ = ν = O#ν by rotational invariance of the support of ν. This implies that O T • ∇u • P#µ = ν. Moreover the map γ = (id × O T • ∇u • P) is also optimal with the same cost as γ. Indeed:

ˆˆ x, x p -y, y p 2 dγ (x, y)dγ (x , y ) = ˆˆ x, x p -O T ∇u(Px), O T ∇u(Px ) p 2 dµ(x)dµ(x ) * = ˆˆ x, x p -∇u(Px), ∇u(Px ) p 2 dµ(x)dµ(x ) = ˆˆ x, x p -y, y p 2 dγ(x, y)dγ(x , y )
where in (*) we used that O ∈ O(p).

The condition of Theorem 4.2.3 seems reasonable and not too restrictive: it suffices that one optimal solution of (MaxOT) where P * is surjective exists in order to prove that an optimal coupling of innerGW is supported by a deterministic map. We believe that some simple assumption can be made on µ, ν in order to meet this condition. In particular this is satisfied when µ, ν are 1D distributions as detailed below.

Application of Theorem 4.2.3 for 1D probability distributions

The sufficient condition given in Theorem 4.2.3 can be used to derived a closed-form expression for the problem (innerGW) between 1D probability distributions. Let us consider (µ, ν) ∈ P(R) × P(R) and F µ and F ν be the cumulative distribution functions of µ and ν and F -1 µ , F -1 ν its pseudo inverses (see Chapter 2). We suppose that µ is regular with respect to the Lebesgue measure in R.

In this case the linear application P in (MaxOT) reduces to a scalar p ∈ R:

sup π∈Π(µ,ν) sup |p|=1 ˆ(px).y dπ(x, y) (4.23)
In this way an optimal any optimal solution (π * , p * ) of (4.23) satisfies p * ∈ {-1, 1} so that p defines a surjective linear application. Then by applying Theorem 4.2.3 there exists u : R → R, convex such that u • p * pushes µ forward to ν and that γ = (id × u • p * )#µ is optimal for (innerGW). In other words there exists f = u non-decreasing such that γ = (id × f • p * )#µ is optimal for (innerGW). However we known from linear transport theory that there is a unique non-decreasing map T asc : R → R such that T asc #µ = ν and it is given by T asc (x) = F -1 ν (F µ (x)) (see theorem 2.5 in [Santambrogio 2015]). This proves that if p * = 1 then f • p * = T asc so that γ = (id × T asc )#µ is optimal for (innerGW). If p * = -1 then we have to consider the "anti" cumulative distribution function. Indeed in this case f • p * is non-increasing and pushes µ forward to ν which is equivalent to say that f is non-decreasing and pushes μ forward to ν where "dμ(x) = dµ(-x)". This discussion leads to the following result: Theorem 4.2.4 (Closed Form expression for (innerGW) between 1D distributions). Let (µ, ν) ∈ P(R) × P(R) with µ regular with respect to the Lebesgue measure. Let F µ (x) = µ(] -∞, x]) be the cumulative distribution function and F µ (x) = µ(] -x, +∞]) be the anti-cumulative distribution function. Let T asc : R → R defined by T asc (x) = F -1 ν (F µ (x)) and T desc : R → R defined by

T desc (x) = F -1 ν (F µ (x))
. Then an optimal solution of (innerGW) is achieved by the map γ = (id × T asc )#µ or by the map γ = (id × T desc )#µ. Theorem 4.2.4 proves that it suffices to compute the CDF or the anti-CDF of the distribution to recover an optimal coupling. It can be put in light of the results of Section 4.1 where we proved that for discrete probability measures with uniform weights and same number of atoms an optimal coupling for the GW problem with squared Euclidean distances can be found in the diagonal or the anti-diagonal coupling when samples are sorted. As such the previous theorem is stronger since it can be applied for general 1D probability distribution when considering inner product similarities. Can we find other examples of optimal couplings for (innerGW), for example when the dimension is larger than 1? The next discussion gives another example which answers this question.

Construction of an optimal couple (u, P) As seen in the previous discussion the couples (u, P) where u is convex so that ∇u • P pushes µ forward to ν may lead to an optimal map w.r.t. the Gromov-Wasserstein distance with inner product similarities. In this part we wish to give an example of such couple (u, P) which leads to an optimal map. Moreover this discussion will also highlight another difference between the linear OT problem and the Gromov-Wasserstein problem. It is known that when µ ∈ P(R p ) and when u : R p → R is convex and differentiable µ a.e then the optimal transport plan for the linear OT problem inf π∈Π(µ,ν)

´ xy 2 2 dπ(x, y) between µ and ν def = ∇u#µ is given by γ = (id × ∇u)#µ. In other words if one perturbs the source measure µ with a transformation which is the gradient of a convex function then the cheapest way (in terms of W 2 ) of moving the source measure forward to the target is by the means of this transformation. To see this we can do the same reasoning as in (4.20) by noticing that inf π∈Π(µ,ν)

´ xy 2 2 dπ(x, y) is equivalent to sup π∈Π(µ,ν)

´ x, y p dπ(x, y) (see also [Santambrogio 2015, Theorem 1.48]). In our case however the situation is a little bit different as detailed in the next proposition: Proposition 4.2.2. Let µ ∈ P(R p ) with

´ x 4 2 dµ(x) < +∞. Let u : R p → R be a convex function. We consider:

sup

Q∈Fp,q ˆu(Qx)dµ(x) (E u )
Let P ∈ F p,q be a solution to (E u ) with ´ ∇u(Px) 4 2 dµ(x) < +∞ then γ = (id × ∇u • P) #µ is an optimal solution of (innerGW) between µ and ν def = ∇u • P#µ.

Proof. Let (π * , P * ) be maximizers of (MaxOT). We have:

ˆ P * x, y q dπ * (x, y) 1 ≤ ˆ(u(P * x) + u * (y)) dπ * (x, y) = ˆu(P * x)dµ(x) + ˆu * (y)dν(y) 2 = ˆu(P * x)dµ(x) + ˆu * (y)d(∇u • P#µ)(y) = ˆu(P * x)dµ(x) + ˆu * (∇u(Px))dµ(x) ≤ sup Q∈Fp,q ˆu(Qx)dµ(x) + ˆu * (∇u(Px))dµ(x) 3 = ˆu(Px)dµ(x) + ˆu * (∇u(Px))dµ(x) 4 = ˆ Px, ∇u(Px) q dµ(x)
where in (1) we used that u is convex, in (2) we used (∇u • P)#µ = ν, in(3) we used that P maximizes sup Q∈Fp,q ´u(Qx)dµ(x) and in (4) we used that for any x and convex function u * (x) + u(x) = x, ∇u(x) .

We can deduce from (4) that:

sup π∈Π(µ,ν)
sup P∈Fp,q ˆ Px, y q dπ(x, y) ≤ ˆ Px, y q dγ(x, y)

By suboptimality the converse inequality is also true so that (γ, P) is an optimal solution of (MaxOT) and consequently γ is an optimal solution of (innerGW) using Theorem 4.2.1. This results states that if one perturbs a source measure µ ∈ P(R p ) with a map ∇u • P with the condition that P : R p → R q achieves sup Q∈Fp,q ´u(Qx)dµ(x) then the cheapest way w.r.t. (innerGW) of moving µ forward to ∇u • P#µ is by the means of the transformation ∇u • P. To illustrate this proposition we can look at simple convex transformations u(x) = 1 2 x T Ux where U is symmetric positive semi-definite (which means that the transformation ∇u is linear). In this case the following proposition exhibits a couple (u, P) which leads to an optimal map for (innerGW): Proposition 4.2.3 (An optimal couple (u, P)). Let µ ∈ P(R p ). We note Σ µ = ´xx T dµ(x). Let u : R p → R convex defined by u(x) = 1 2 x T Ux where U is symmetric positive semi-definite. Let v, w be the eigenvectors corresponding to the largest eigenvalue of respectively Σ µ and U and P = √ p vw T ∈ F p,p

The coupling γ = (id × ∇u • P) is optimal for (innerGW) between µ and ν def = ∇u • P#µ

Proof. In this case the problem sup Q∈Fp,p ´u(Qx)dµ(x) reduces to sup

Q 2 F =p = ´xT Q T UQxdµ(x) = sup Q 2 F =p tr(Q T UQΣ µ )
where Σ µ = ´xx T dµ(x). By vectorizing the matrix Q it is equivalent to: max

q∈R p 2 q 2 2 =p q T M u,µ q (4.24)
where M u = U ⊗ K Σ µ with ⊗ K the Kronecker product of matrices. M u,µ is symmetric positive semi-definite. We can rewrite this problem with q = √ p q:

max qT q=1 qT M u,µ q (4.25)
Which is a maximization of a Rayleigh quotient problem. It is well known that a solution of this problem is found at any eigenvector associated to the largest eigenvalue of M u,µ (see e.g. [Anstreicher 1998]). However the eigenvalues of M u,µ are given by all the products of the eigenvalues of Σ µ and U (see e.g. [START_REF] Horn | [END_REF]).

Since they are all positive the largest eigenvalue of M u,µ is found at the largest eigenvalue of Σ µ and U with corresponding eigenvector v, w and the optimal q is v, w T . This implies that the optimal Q is Q = √ p vw T . We can apply Proposition 4.2.2 to conclude.

The squared Euclidean case

The GW problem is usually considered with distances, as it provides a metric with respect to strong isomorphisms. The goal of this section is to study the case c X (x,

x ) = x -x 2 2 , c Y (y, y ) = y -y 2 2
with µ ∈ P(R p ), ν ∈ P(R q ). As for the inner product case we can prove that this problem is equivalent to another linear OT problem parametrized by a linear application. More precisely:

Theorem 4.2.5. Let X and Y be compact subset of respectively R p and R q . Let µ ∈ P(X ), ν ∈ P(Y).

Assume without loss of generality that E X∼µ [X] = 0 and E Y ∼ν [Y ] = 0. Then problems:

inf π∈Π(µ,ν) ˆ( x -x 2 2 -y -y 2 2 ) 2 dπ(x, y)dπ(x , y ) (sqGW)
and sup

π∈Π(µ,ν) sup P∈R q×p ˆ( Px, y q + x 2 2 y 2 2 )dπ(x, y) - 1 8 P 2 F (dual-sqGW)
are equivalent.

To prove the previous theorem we will rely on the calculus of Lemma 4.2.1 and the observation that the cost J 2 (c X , c Y , π) is invariant by translation of the support of the measures so that they can be centered without loss of generaly. This implies that the term

´ x 2 2 E Y ∼ν [Y ], y q + y 2 2 E X∼µ [X],
x p dπ(x, y) in Lemma 4.2.1 vanishes. More presicely we have the following result: Lemma 4.2.3. Let X and Y be compact subset of respectively R p and R q . Let µ ∈ P(X ), ν ∈ P(Y). We can assume without loss of generality that E X∼µ [X] = 0 and E Y ∼ν [Y ] = 0. In this case (sqGW) is equivalent to:

sup π∈Π(µ,ν) ˆ x 2 2 y 2 2 dπ(x, y) + 2 ˆyx T dπ(x, y) 2 F (4.26)
A proof of this lemma can be found in Section 6.2.8. In the following we will note F (π) = ´

x 2 2 y 2 2 dπ(x, y) + 2 ´yx T dπ(x, y) 2 F . To prove Theorem 4.2.5 the idea is to observe that this problem is a maximization of a convex function of π. We can use standard convex analysis tools such that the Fenchel-Moreau duality to derive the Fenchel dual of F (π) as detailed below.

Duality in the space of measure We suppose in the following that X , Y are general compact spaces. In this case the dual space of C(X × Y) is M(X × Y) (see e.g. Memo 1.3 in [Santambrogio 2015]). We recall the main definitions of the Legendre-Fenchel tansform: Definition 4.2.1 (Legendre-Fenchel Transformation). Let X , Y be compact Hausdorff spaces. For a function F : M(X × Y) → R ∪ {+∞} we define its convex conjugate F * : C(X × Y) → R ∪ {+∞} and its bi-conjugate F * * : M(X × Y) → R ∪ {+∞} as:

F * (h) = sup π∈M(X ×Y) ˆh(x, y)dπ(x, y) -F (π) F * * (π) = sup h∈C(X ×Y) ˆh(x, y)dπ(x, y) -F * (h) (4.27)
One remarkable property of the convex conjugate F * is that it is always l.s.c and convex because h → ´hdπ -F (π) is an affine function which is always convex and continuous. In this way F * is the pointwise supremum of continuous linear functions which is convex and l.s.c. We denote by dom(F ) = {π|F (π) < +∞} the domain of F . A function is called proper if dom(F ) = ∅. A fundamental result in convex analysis is the Fenchel-Moreau theorem states that the bi-conjugate of a proper and l.s.c convex function equals to the original function (see e.g. [START_REF] Lai | [END_REF]). In other words when F is convex and well-behaved one can rely on the bi-conjugate to study the original function. In our context this implies that F (π) = F * * (π) for all π ∈ M(X × Y). To compute F * * (π) we will need a notion of derivative in the space of measures as described in the next definition:

Definition 4.2.2 (Fréchet differentiable). Let X , Y be compact Hausdorff spaces. A function F : M(X × Y) → R is is Fréchet differentiable at π if there exists ∇F (π) ∈ C(X × Y) such that for any ε ∈ M(X × Y)
as t → 0:

F (π + tε) = F (π) + t ˆ∇F (π)dε + o(t) (4.28)
Application to the Gromov-Wasserstein problem Let X ⊂ R p , Y ⊂ R q be compact spaces and µ ∈ P(X ), ν ∈ P(Y). Since X , Y are compact and the norms are continuous the GW distance is finite and the function F defined in (4.26) is proper, convex and l.s.c. By application of the Fenchel-Moreau theorem we have:

sup π∈Π(µ,ν) F (π) = sup π∈Π(µ,ν) F * * (π) = sup π∈Π(µ,ν) sup h∈C(X ×Y) ˆh(x, y)dπ(x, y) -F * (h) (4.29)
In the following we denote by V π = ´yx T dπ(x, y). We can prove that we can solve the dual problem by parametrizing h by a linear application P ∈ R q×p as h(x, y) = Px, y q + x 2 2 y 2 2 . Using this form the problem becomes much simpler as we only need to optimize on a finite dimensional space instead of maximizing over all continuous function. Lemma 4.2.4. If π * is a solution of the primal problem sup π∈Π(µ,ν) F (π) then there exists P ∈ R q×p and h * ∈ C(X × Y) of the form h * (x, y) = Px, y q + x 2 2 y 2 2 such that (π * , h * ) is a solution of the dual problem (4.26). Moreover when h * is in such form we have F * (h * ) = 1 8 P 2 F .

A proof can be found in Section 6.2.9. The previous Lemma 4.2.4 can be used to prove Theorem 4.2.5. Indeed with previous notations h can be written in the form h(x, y) = Px, y q + x 2 2 y 2 2 . Plugging the calculus of the conjugate F * (h) = 1 8 P 2 F into (4.29) gives the desired result.

Regularity of (sqGW) optimal plans

The Fenchel dual problem is more difficult to analyse than the dual problem of the inner product case. Indeed the cost c : (x, y) → Px, y q + x 2 2 y 2 2 hardly relates to a "standard" linear OT problem. Especially it does not statifies the Twist condition (see Chapter 2) in general and an approach with convex tools is trickier due to the term x 2 2 y 2 2 . We will show in the following that if ∇u • P is "not too far" from an isometry then it defines an optimal coupling. We note Γ 0 (R, R) the set of derivable convex functions from R to R and we define the following set: Definition 4.2.3. We define H(µ, ν) the set of push-forward between µ and ν defined by:

H(µ, ν) = {T #µ = ν |∃f ∈ Γ 0 (R, R) , T (x) 2 2 = f ( x 2 2 ) µ a.e} (4.30)
When p = q this set encompasses all the linear push forward of the form T = c O where c > 0, O ∈ O(p). It turns out that this condition is also sufficient when considering only linear push-forward (see Lemma 6.2.11).

We have the following sufficient condition if we look at Monge map of the form ∇u • P with u convex and and P ∈ R q×p : Proposition 4.2.4. Let X and Y be compact subset of respectively R p and R q with p ≥ q. Let µ ∈ P(X ), ν ∈ P(Y). Assume that µ is regular w.r.t. the Lebesgue measure on R p and that E X∼µ [X] = 0 and

E Y ∼ν [Y ] = 0 without loss of generality.
Let (π * , P * ) be optimal solution of (dual-sqGW). If P * is surjective there exists u : R q → R convex such that ∇u • P * pushes µ forward to ν. Moreover if ∇u • P * ∈ H(µ, ν) then the coupling γ = (id × ∇u • P * )#µ is optimal for (sqGW).

Proof. Let (P * , π * ) be optimal solution of (dual-sqGW) with P * surjective. We have seen previously that P * #µ is regular with respect to the Lebesgue measure on R q using Proposition 4.2.1 such that there exists u : R q → R convex such that ∇u • P * pushes µ forward to ν. Moreover:

ˆ P * x, y q + x 2 2 y 2 2 dπ * (x, y) 1 ≤ ˆ(u(P * x) + u * (y)) dπ * (x, y) + ˆ x 2 2 y 2 2 dπ * (x, y) = ˆu(P * x)dµ(x) + ˆu * (y)dν(y) + ˆ x 2 2 y 2 2 dπ * (x, y) 2 = ˆu(P * x)dµ(x) + ˆu * (y)d(∇u • P * #µ)(y) + ˆ x 2 2 y 2 2 dπ * (x, y) = ˆu(P * x)dµ(x) + ˆu * (∇u(P * x))dµ(x) + ˆ x 2 2 y 2 2 dπ * (x, y) 3 = ˆ P * x, ∇u(P * x) q dµ(x) + ˆ x 2 2 y 2 2 dπ * (x, y)
In (1) we used convexity of u, in (2) we used that ∇u • P * is a push-forward and in (3) we used u * (∇u(x)) + u(x) = x, ∇u(x) . Now suppose that ∇u • P * ∈ H(µ, ν). Then there exists

f ∈ Γ 0 (R, R) , T (x) 2 2 = f ( x 2 2 ) µ a.e. Moreover we have ´ x 2 2 y 2 2 dπ * (x, y) ≤ ´f ( x 2 2 ) + f * ( y 2
2 )dπ * (x, y) by Young's inequality (see Memo 4.2.2). In this way: ˆ P * x, y q + x 2 2 y 2 2 dπ * (x, y) ≤ ˆ P * x, ∇u(P * x) q dµ(x) + ˆf * ( x 2 2 )dµ(x) + ˆf ( y 2 2 )dν(y)

5 = ˆ P * x, ∇u(P * x) q dµ(x) + ˆf * ( ∇u(P * x) 2 2 ) + f ( x 2 2 )dµ(x) 6 = ˆ P * x, ∇u(P * x) q dµ(x) + ˆ ∇u(P * x) 2 2 y 2 2 dµ(x) 7 = ˆ P * x, y q + x 2 2 y 2 2 dγ(x, y)
In ( 5) we used that ∇u • P * is a push-forward. In ( 6) we used that f satisfies ∇u( 7) γ is defined as γ = (id × ∇u • P * )#µ. Overall (P * , γ) is optimal for (dual-sqGW). Using Theorem 4.2.5 this proves that γ is optimal for (sqGW) which concludes the proof.

P * x) 2 2 ∈ ∂f ( x 2 2 ) = f ( x 2 2 ) by definition of H(µ, ν) which implies f * ( ∇u(P * x) 2 2 ) + f ( x 2 2 ) = ∇u(P * x) 2 2 x 2 2 (see Memo 2.1.2). In (
The condition ∇u • P ∈ H(µ, ν) is quite strong compared to the conditions of Theorem 4.2.3. For example in the case where ∇u is linear only orthogonal transformations are admissible (modulo a scaling). We believe that this result can be improved and we leave this study for further works. In general, and without any furter assumption on µ and ν, we postulate that there might be degenerate cases in which µ is regular but there is no deterministic optimal couplings.

Optimization and numerical experiments

In this section we provide numerical solutions for Gromov-Wasserstein problems in Euclidean spaces. We will rely on the equivalent formulation defined in Theorem 4.2.1 and Theorem 4.2.5. We consider two discrete probability measures µ min

= 1 n n i=1 a i δ xi ∈ P(R p ) and ν = m i=1 b j δ yj ∈ P(R q ) with a ∈ Σ n , b ∈ Σ m . Numerical solution for (innerGW) We note X = (x i ) n i=1 ∈ R n×p ,Y = (y j ) m j=1 ∈ R m×q .
P F = √ p -XP T Y T , π F (4.33)
The resulting problem is convex w.r.t. P and π (but not jointly convex). We propose to solve equation (4.33) using Block Coordinate Descent (BCD) which alternates between minimizing w.r.t. P and π.

Memo 4.2.2 (Young's Inequality). Let a, b ∈ R + × R + and p, q real numbers greater than 1 with

1 p + 1 q = 1. Then: ab ≤ a p + b q (4.31)
More generally if f is a convex function and f * is its Legendre transform then:

ab ≤ f (a) + f * (b) (4.32)
which is a consequence of the celebrated Fenchel-Young inequality (see Memo 2.1.2)

Algorithm 7 Gromov-Wasserstein with inner products

1: Require µ = 1 n n i=1 a i δ xi ∈ P(R p ) and ν = m i=1 b j δ yj ∈ P(R q ) 2: Set X = (x i ) n i=1 ∈ R n×p ,Y = (y j ) m j=1 ∈ R m×q 3: Initialize π = ab T . 4: while not converged do 5: P ← √ d Y T π T X F Y T π T X // (
P ← √ d Y T π T X F Y T π T X (4.34)
This procedure is presented in Algorithm 7. The complexity is driven by the linear OT problem which is O(n 3 log(n)) when n = m. The complexity for computing P at each iteration is O(mn(q + p) + pq) with O(mn(q + p)) for the computing Y T π T X and O(pq) for Y T π T X F . The overall complexity when

n = m is then O(n 3 log(n) + n 2 (q + p) + pq).
-XP T Y T , π F + 1 8 P 2 F def = min P∈R q×p G(P) (4.36)
which is a unconstrained QP which can be solved in closed-form. Indeed the gradient reads ∇G(P) = -Y T π T X + 1 4 P and by first order condition a solution can be found at ∇G(P) = 0. In this way the iteration for P are:

P ← 4Y T π T X (4.37)
This procedure is presented in Algorithm 8. Computing all the terms x (i) 2 2 y (j) 2 2 where x (i) is the i-th row of X has a naive complexity of O(np + mq + mn) and one needs also to compute 4Y T π T X at a O(mn(q + p)) price. The overall complexity is then

O(n 3 log(n) + (p + q)n 2 + n 2 + n(p + q)) = O(n 3 log(n) + (p + q)n 2 ) when n = m.

Runtimes comparison

We perform a comparison between runtimes of GW using different algorithmic solutions. We consider the Gromov-Wasserstein problem between 10 realizations of one source 2D random probability measures of n ∈ {10, ..., 30000} points and one target 3D random measures with m = 100 Algorithm 8 Gromov-Wasserstein with squared Euclidean distances points. We compute the Gromov-Wasserstein distance using squared Euclidean distances as c X , c Y . We compare the timings of the Frank-Wolfe algorithm (see Chapter 3), the entropic regularized approach with ε ∈ {5, 10, 100} (see Chapter 2), and the BCD approach Algorithm 8 using the same initialization of π = ab T for all methods. The result is depicted in Figure 4.7. Please note that the timings are calculated without taking into account the time needed for computing the matrices C 1 , C 2 or X, Y but are only based on the loops of the different algorithms. The entropic-GW could not be computed with ε ≤ 5 due to overflows in the Sinkhorn algorithm. Another difficulty of computing entropic-GW is the fact that the regularization parameter ε is inversely proportional to the gradient step [START_REF] Peyré | [END_REF]]. Thus only high value of ε are computable in reasonable time and for ε < 5 we observe that the convergence is very slow even on this simple example. This often implies that only very blurred solutions can be computed which explains the low variance of entropic-GW with ε = 100. We can see in Figure 4.7 that the BCD approach is a little bit faster than the FW approach, suggesting that the BCD may converge faster to a local minimum than FW, which is, of course, data dependant. Overall both methods are equally rapid on this example.

1: Require µ = 1 n n i=1 a i δ xi ∈ P(R p ) and ν = m i=1 b j δ yj ∈ P(R q ) 2: Subtract the mean: x i ← x i -1 n k x k , y j ← y j -1 m k y k . 3: Set X = (x i ) n i=1 ∈ R n×p ,Y = (y j ) m j=1 ∈ R m×q , x = Diag(XX T ) ∈ R n , y = Diag(YY T ) ∈ R

Costs comparison

In this experiment we compared the ability of the BCD approach to find a better solution than the FW approach. We consider the Gromov-Wasserstein problem with both inner product similarities and squared Euclidean distances. We compute 200 distances using both algorithms where each distance is calculated by: (1) We draw n ∈ {500, 2000} samples from two 10 dimensional normal distributions (2) We associate to these points random weights (a, b) ∈ Σ n × Σ n (3) We initialize the algorithms with the same random coupling matrix π. The initialization is computed by sampling a random matrix with positive entries and by scaling it with the Sinkhorn algorithm in order to have the prescribed marginals a, b. Results are depicted in Figure 4.8. We plot the GW cost obtained by BCD approach vs the cost obtained by the FW approach after convergence of each algorithm. As seen in Figure 4.8 there is no strong differences between FW approach and the BCD approach when squared Euclidean distances are considered (left part of the figure): the F W algorithm finds a better solution in 51% of the cases when n = 2000 and 56% for n = 500. Surprisingly both algorithms seem to lead to the same solution when inner product similarities are used (right part of the figure). Indeed in 99% of the cases the costs are identical up to 1e -7.

Conclusion on the experiments

As seen in these experiments, there are no significant differences between the FW approach and the BCD procedure. It is not surprising for the runtimes comparison since both methods have a theoretical cubic complexity. The cost comparison experiment may also suggest that both BCD procedure and FW are equivalent in the case of inner product similarities, that is the iterations of the BCD are the same than the iterations of the FW. Further studies could be conducted to examine the high-dimensional setting where we postulate that the BCD approach may lead to better results that the FW procedure, i.e. may converge faster a produce a better solution.

The Gromov-Monge problem in Euclidean spaces

As seen in Section 4.1.2 the Gromov-Wasserstein problem is also very related the Gromov-Monge problem [Mémoli 2018] which is defined in Euclidean spaces for µ ∈ P(R p ), ν ∈ P(R q ) as:

Definition 4.2.4 (Gromov-Monge). The Gromov-Monge problem aims at finding:

GM 2 2 (µ, ν) = inf T #µ=ν J(T ) = inf T #µ=ν ˆ x -x 2 2 -T (x) -T (x ) 2 2 2 dµ(x)dµ(x ) (GM)
With the understanding that GM 2 (µ, ν) = +∞ when {T :

R p → R q |T #µ = ν} = ∅
The problem (GM) is the exact counterpart for the Gromov-Wasserstein distance of the Monge problem for the Wasserstein distance. If we note GW 2 (µ, ν) the Gromov-Wasserstein distance with squared Euclidean costs then we have GW 2 (µ, ν) ≤ GM 2 (µ, ν) but in general both problems are not equivalent (see [Mémoli 2018]). Note that when problem 1 holds, i.e. when the Gromov-Wasserstein problem admits an optimal transport plan supported on a deterministic map then both (GM) and (sqGW) are equivalent. Moreover we have seen previously in Theorem 4.1.2 that when µ and ν are discrete probability measures with the same number of atoms and uniform weights then (GM) is also equivalent to the Gromov-Wasserstein problem (sqGW) so that that GW 2 (µ, ν) = GM 2 (µ, ν).

We propose to study further the Gromov-Monge problem in this section. Especially we consider the special case of Gaussian measures µ = N (m ν , Σ ν ), ν = N (m µ , Σ µ ). It is motivated by the linear OT theory where, when p = q, there is a close form solution for W 2 . In this case the optimal Monge map is linear and given by [Takatsu 2011] T : x → m ν + A(xm ν ) where:

A = Σ -1/2 µ (Σ 1/2 µ Σ ν Σ 1/2 µ ) 1 2 Σ -1/2 µ (4.38)
Can we derive the same type of result for the Gromov-Monge geometry? We will prove that when restricted to linear push-forward and in the special case p = q the problem admits also a close form expression. In this way we consider the following linear Gromov-Monge problem:

LGM

2 2 (µ, ν) = inf T #µ=ν T is linear J(T ) = inf T #µ=ν T is linear ˆ x -x 2 2 -T (x) -T (x ) 2 2 2 dµ(x)dµ(x ) (LGM)
We recall that V p (R q ) is the Stiefel manifod defined by

V p (R q ) = {B ∈ R q×p |B T B = I p }.
The main result of this section is the following theorem:

Theorem 4.2.6. Let µ = N (0, Σ ν ) ∈ P(R p ), ν = N (0, Σ µ ) ∈ P(R q ) centered without loss of generality. Let Σ µ = V µ D µ V µ , Σ ν = V ν D ν V ν
be the diagonalizations of the covariance matrices such that eigenvalues of D µ and D ν are ordered nondecreasing. When p = q we have:

LGM An illustration of the map A is given in Figure .4.9 where we compute LGM between two 2D empirical distributions, that we consider Gaussian in first approximation. The optimal map A gives a different behaviour than the Wasserstein map A W which seems to better grasp the transformation of the samples. It is somehow natural since the Gromov-Monge cost is less rigid than the Wasserstein one and only forces the samples to be isometrically distributed on average. However note that the target samples are arbitrary rotated for the case of Gromov-Monge since LGM is invariant by rotation of the samples.

2 2 (µ, ν) = 4(tr(Σ µ ) -tr(Σ ν )) 2 + 8(tr(Σ µ Σ µ ) + tr(Σ ν Σ ν )) + 16 min B∈Vp(R q ) -tr(D µ B D ν B) (

Solving the problem when p = q

As described in Theorem 4.2.6 the situation is more delicate when p = q, i.e. when the Gaussian measures are not part of the same ground Euclidean space. In this case there is no close form anymore and one needs to solve the following Quadratic optimization with Orthogonality Constraints (QPOC) problem: min

B∈Vp(R q ) -tr(B D ν BD µ ) def = min B∈Vp(R q ) F (B) (QPOC)
This problem is a particular special case of optimizing a smooth function over the Stiefel manifold, which is non-convex in general but for which various methods have been proposed over the years (see e.g [Jiang 2014[START_REF][END_REF][START_REF] Abrudan | [END_REF], Absil 2009]). A standard approach for solving these types of problems is to leverage the Riemannian structure of V p (R q ) to use a gradient descent method on the manifold. Informally, the idea is to start with a arbitrary point B 0 ∈ V p (R q ) then iteratively move in a search direction D(B 0 ) defined by a tangent vector while staying on V p (R q ) until a critical point is found. In general this procedure require to compute the so-called exponential map which is quite difficult. A cheaper alternative can be found into a retraction map which approximates the exponential map (see e.g. [START_REF] Liu | [END_REF]] and references therein). We propose to solve (QPOC) using the Geoopt library [Kochurov 2020]. We illustrate the resulting optimal map in Figure 4.10 where we compute the LGM problem between two empirical distributions. The source distribution is 3 dimensional while the target distribution is 2 dimensional. We observe that the optimal Gromov-Monge map manages to capture the overall transformation of the source distribution.

Conclusion: perspectives and open questions

In our opinion the previous results raise interesting questions and pave the path for a deeper understanding of the Gromov-Wasserstein and Gromov-Monge geometry in Euclidean spaces. The works on the regularity of optimal couplings are, to the best of our knowledge, the first work on this subject and we believe that these results could be improved in order to bridge the theoretical gap between Wasserstein and Gromov-Wasserstein.

Gromov-Wasserstein in Euclidean spaces

In the inner product case we have seen that among the set of solutions of (MaxOT) the couples with P * surjective are of particular interest. For example is it possible under some mild conditions on µ, ν to prove that there always exists an optimal couple (π * , P * ) of (MaxOT) such that P * is surjective? If this result holds then it would imply that, if the source measure is regular, the Gromov-Wasserstein problem with inner products is equivalent to its Monge counterpart:

inf

T #µ=ν ˆ( x, x p -T (x), T (x ) q ) 2 dµ(x)dµ(x ) (4.43)
The equivalence between Gromov-Monge and Gromov-Wasserstein problems would be a nice generalization of the Brenier's theorem in the case of the Gromov-Wasserstein geometry. Related to this problem the unicity of the couples (u, P) is also an interesting further work. More precisely can we find suitable conditions on µ, ν so that there is a unique ∇u • P which pushes µ forward to ν optimally? When the support of the measures are not "too symmetric" it seems reasonable that all the solutions are somehow unique modulo a class of isometries such as rotations. As described in this section the Brenier's theorem states that there is a unique optimal push forward in the case of Wasserstein geometry. This property allows to derive close form expression for the Gaussian case and we believe that similar reasoning could be made if the unicity (up to some rotations) holds for (MaxOT). Finally can we extend the previous results about the inner case for kernel similarities? More precisely when c X (x, x ) = k X (x, x ) defines a kernel (same for c Y ) we know from the kernel trick that there exists an inner product space V X and a feature map φ :

X → V X such that c X (x, x ) = φ(x), φ(x ) V X .
In this case can we find similar results regarding the regularity of the optimal transport plans of the Gromov-Wasserstein distance?

The squared Euclidean distance case seems to be more delicate to handle as it does not echo to classical OT costs. We believe that the dual problem could be used to find a close form expression for 1D discrete probability measures with possibly different number of atoms and non-uniform weights, as done for the inner product case. This would be an interesting stronger result than the case considered in Section 4.1.2. We postulate that, in this case, an optimal coupling is also given by a monotone rearrangement which is increasing or decreasing. For general probability measures assessing the regularity of optimal transport plans seems more complicated. The sufficient condition of Proposition 4.2.4 may be improved with a necessary condition in order to characterize optimal couplings under some hypothesis on µ, ν. Another interesting approach for the squared Euclidean distance case would be to consider optimal couplings when the target measure is a small perturbation of the source measure. More precisely we believe the following holds:

Problem 2. Let µ ∈ P(R p ). Let ν = (id + εu)#µ ∈ P(R p ) with u : R p → R p be a small perturbation of µ. Then the map γ = (id × T )#µ where T = id + εu is optimal for:

inf π∈Π(µ,ν) ˆ x -x 2 2 -y -y 2 2 2 dπ(x, y)dπ(x , y ) (4.44)
Chapter 5

CO-Optimal Transport

"Tu as tort de lire les journaux; ça te congestionne."

-André Gide, Les Faux-Monnayeurs

Introduction

The problem of comparing two sets of samples arises in many fields in machine learning, such as manifold alignment [Cui 2014], image registration [START_REF] Haker | [END_REF]], unsupervised word and sentence translation [Rapp 1995] among others. When correspondences between the sets are known a priori, one can align them with a global transformation of the features, e.g, with the widely used Procrustes analysis [START_REF] Gower | [END_REF], Goodall 1991]. For unknown correspondences, other popular alternatives to this method include correspondence free manifold alignment procedure [Wang 2009], soft assignment coupled with a Procrustes matching [Rangarajan 1997a] or Iterative closest point and its variants for 3D shapes [START_REF] Besl | [END_REF][START_REF][END_REF]]. When one models the considered sets of samples as empirical probability distributions, the optimal transport framework provides a solution to find, without supervision, a softcorrespondence map between them given by an optimal coupling. OT-based approaches have been used with success in numerous applications such as embeddings' alignments [Alvarez-Melis 2019, Grave 2019] and Domain Adaptation (DA) [Courty 2017] to name a few. However, one important limit of using OT for such tasks is that the two sets are assumed to lie in the same space so that the cost between samples across them can be computed. This major drawback does not allow OT to handle correspondences' estimation across heterogeneous spaces, preventing its application in problems such as, for instance, heterogeneous DA (HDA). To circumvent this restriction, one may rely on the Gromov-Wasserstein distance [Memoli 2011]: a non-convex quadratic OT problem that finds the correspondences between two sets of samples based on their pairwise intra-domain similarity (or distance) matrices. Such an approach was successfully applied to sets of samples that do not lie in the same Euclidean space, e.g for shapes [Solomon 2016], word embeddings [Alvarez-Melis 2018a] and HDA [START_REF][END_REF]] mentioned previously. One important limit of GW is that it finds the samples' correspondences but discards the relations between the features by considering pairwise similarities only. Another line of works [Alvarez-Melis 2019, Grave 2019] considers the problem of matching sets of points with respect to a global transformations of the features, usually modeled by a linear transformation such as a rotation. These approches differ from the work proposed here where we consider instead a probabilistic coupling of the features as described below.

In this Chapter, we propose a novel OT approach called CO-Optimal transport (COOT) that simultaneously infers the correspondences between the samples and the features of two arbitrary sets. Our new formulation includes GW as a special case, and has an extra-advantage of working with raw data directly without needing to compute, store and choose computationally demanding similarity measures

CO-Optimal transport optimization problem

required for the latter. Moreover, COOT provides a meaningful mapping between both instances and features across the two datasets thus having the virtue of being interpretable. We thoroughly analyze the proposed problem, derive an optimization procedure for it and highlight several insightful links to other approaches. On the practical side, we provide evidence of its versatility in machine learning by putting forward two applications in HDA and co-clustering where our approach achieves state-of-the-art results.

The rest of this chapter is organized as follows. We introduce the COOT problem in Section 5.2, states its mathematical properties in Section 5.3 and give an optimization routine for solving it efficiently in Section 5.4. In Section 5.5, we show how COOT is related to other OT-based distances and recover efficient solvers for some of them in particular cases. Finally, in Section 5.6.1 and Section 5.6.2, we present an experimental study providing highly competitive results in HDA and co-clustering compared to several baselines.

CO-Optimal transport optimization problem

We consider two datasets represented by matrices

X = [x 1 , . . . , x n ] T ∈ R n×d and X = [x 1 , . . . , x n ] T ∈ R n ×d
, where in general we assume that n = n and d = d . In what follows, the rows of the datasets are denoted as samples and its columns as features. We endow the samples (x i ) i∈ [[n]] and (x i ) i∈ [[n ]] with weights w = [w 1 , . . . , w n ] ∈ Σ n and w = [w 1 , . . . , w n ] ∈ Σ n that both lie in the simplex so as to define empirical distributions supported on (x i ) i∈ [[n]] and (x i ) i∈ [[n ]] . In addition to these distributions, we similarly associate weights given by vectors v ∈ Σ d and v ∈ Σ d with features. Note that when no additional information is available about the data, all the weights' vectors can be set as uniform.

We define the CO-Optimal Transport problem as follows:

min π s ∈Π(w,w ) π v ∈Π(v,v ) i,j,k,l L(X i,k , X j,l )π s i,j π v k,l = min π s ∈Π(w,w ) π v ∈Π(v,v ) L(X, X ) ⊗ π s , π v F (COOT)
where L : R × R → R + is a divergence measure between 1D variables, L(X, X ) is the d × d × n × n tensor of all pairwise divergences between the elements of X and X , and Π(•, •) is the set of linear transport constraints.

Note that problem (COOT) seeks for a simultaneous transport π s between samples and a transport π v between features across distributions. In the following, we write COOT(X, X , w, w , v, v ) (or COOT(X, X ) when it is clear from the context) to denote the objective value of the optimization problem (COOT).

Entropic regularization Equation (COOT) can be also extended to the entropic regularized case favoured in the OT community for remedying the heavy computation burden of OT and reducing its sample complexity [Cuturi 2013, Altschuler 2017[START_REF] Genevay | [END_REF]]. This leads to the following problem:

min π s ∈Π(w,w ) π v ∈Π(v,v ) L(X, X ) ⊗ π s , π v F + Ω(π s , π v ) (5.1)
where for ε 1 , ε 2 > 0, the regularization term writes as Ω(π s , π

v ) = ε 1 H(π s |ww T ) + ε 2 H(π v |vv T ) with H(π s |ww T ) = i,j log( π s i,j wiw j
)π s i,j being the relative entropy. Note that similarly to OT [Cuturi 2013] and GW [START_REF] Peyré | [END_REF]], adding the regularization term can lead to a more robust estimation of the transport matrices but prevents them from being sparse.

MNIST USPS USPS samples MNIST samples

s matrix between samples USPS colored coded pixels MNIST pixels through v MNIST pixels through entropic v

Illustration of COOT

In order to illustrate our proposed COOT method and to explain the intuition behind it, we solve the optimization problem (COOT) using the algorithm described in Section 5.4 between two classical digit recognition datasets: MNIST and USPS. We choose these particular datasets for our illustration as they contain images of different resolutions (USPS is 16×16 and MNIST is 28×28) that belong to the same classes (digits between 0 and 9). Additionally, the digits are also slightly differently centered as illustrated on the examples in the left part of Figure 5.1. Altogether, this means that without specific pre-processing, the images do not lie in the same topological space and thus cannot be compared directly using conventional distances. We randomly select 300 images per class in each dataset, normalize magnitudes of pixels to [0, 1] and consider digit images as samples while each pixel acts as a feature leading to 256 and 784 features for USPS and MNIST respectively. We use uniform weights for w, w and normalize average values of each pixel for v, v in order to discard non-informative ones that are always equal to 0.

The result of solving problem (COOT) is reported in Figure 5.1. In the center-left part, we provide the coupling π s between the samples, i.e the different images, sorted by class and observe that 67% of mappings occur between the samples from the same class as indicated by block diagonal structure of the coupling matrix. The coupling π v , in its turn, describes the relations between the features, i.e the pixels, in both domains. To visualize it, we color-code the pixels of the source USPS image and use π v to transport the colors on a target MNIST image so that its pixels are defined as convex combinations of colors from the former with coefficients given by π v . The corresponding results are shown in the right part of Figure 5.1 for both the original COOT and its entropic regularized counterpart. From these two images, we can observe that colored pixels appear only in the central areas and exhibit a strong spatial coherency despite the fact that the geometric structure of the image is totally unknown to the optimization problem, as each pixel is treated as an independent feature. COOT has recovered a meaningful spatial transformation between the two datasets in a completely unsupervised way, different from trivial rescaling of images that one may expect when aligning USPS digits occupying the full image space and MNIST digits lying in the middle of it.

For further evidence Figures 5.2 and 5.3 illustrate different images of both datasets obtained by transporting pixels from USPS (resp. MNIST) to MNIST (resp. USPS) using the optimal coupling π v . Notably the case USPS → MNIST show that transporting the pixel through π v leads to a better spatial coherency than a simple rescaling of the image.

Properties of COOT

COOT as a billinear program COOT is a special case of a Quadratic Program (QP) with linear constraints called a Bilinear Program (BP). More precisely, it is an indefinite BP problem [START_REF] Gallo | [END_REF]]. It was proved (e.g in in [START_REF] Pardalos | [END_REF][START_REF] Horst | [END_REF]) that there exists an optimal solution lying on extremal points of the polytopes Π(w, w ) and Π(v, v ). When n = n , d = d and weights w [START_REF] Birkhoff | [END_REF]] states that the set of extremal points of Π( 1n n , 1n n ) and Π(

= w = 1n n , v = v = 1 d d are uniform, Birkhoff's theorem
1 d d , 1 d d )
are the set of permutation matrices so that there exists an optimal solution (π s * , π v * ) which transport maps are supported on two permutations σ s * , σ v * ∈ S n × S d . The BP problem is also related to the Bilinear Assignment Problem (BAP) where π s and π v are searched in the set of permutation matrices. The latter was shown to be NP-hard if d = O( r √ n) for fixed r and solvable in polynomial time if d = O( log(n)) [START_REF] Custic | [END_REF]]. In this case, we look for the best permutations of the rows and columns of our datasets that lead to the smallest cost. COOT provides a tight convex relaxation of the BAP by 1) relaxing the constraint set of permutations into the convex set of doubly stochastic matrices and 2) ensuring that two problems are equivalent, i.e., one can always find a pair of permutations that minimizes (COOT), as explained in the paragraph above.

Finding a meaningful similarity measure between datasets is useful in many machine learning tasks as pointed out, e.g in [Alvarez-Melis 2020]. Interestingly enough, COOT induces a notion of distance between datasets X and X . More precisely it vanishes iff they are the same up to a permutation of rows Algorithm 9 BCD for COOT

1: π s (0) ← ww T , π v (0) ← vv T , k ← 0 2: while k < maxIt and err > 0 do 3: π s (k) ← arg min π s ∈Π(w,w ) L(X, X ) ⊗ π v (k-1)
, π s F // linear OT problem on the samples 4:

π v (k) ← arg min π v ∈Π(v,v ) L(X, X ) ⊗ π s (k-1) , π v F // linear OT problem on the features 5: err ← ||π v (k-1) -π v (k) || F 6:
k ← k + 1 7: end while and columns as established below:

Proposition 5.3.1 (COOT is a distance). Suppose L = | • | p , p ≥ 1, n = n , d = d
and that the weights w, w , v, v are uniform. Then COOT(X, X ) = 0 iff there exists a permutation of the samples

σ 1 ∈ S n and of the features σ 2 ∈ S d , s.t, ∀i, k X i,k = X σ1(i),σ2(k)
. Moreover, it is symmetric and satisfies the triangular inequality as long as L satisfies the triangle inequality, i.e., COOT(X, X ) ≤ COOT(X, X ) + COOT(X , X ).

Note that in the general case when n = n , d = d , positivity and triangle inequality still hold but COOT(X, X ) > 0. The proof can be found in Section 6.3. Interestingly, our result generalizes the metric property proved in [Faliszewski 2019] for the election isomorphism problem with this latter result being valid only for the BAP case (for a discussion on the connection between COOT and the work of [Faliszewski 2019], see Section 6.3.7). Finally, we note that this metric property means that COOT can be used as a divergence in a large number of potential applications as, for instance, in generative learning [Bunne 2019].

Optimization algorithm and complexity

Even though solving COOT exactly is NP-hard, in practice computing a solution can be done rather efficiently. To this end, we propose to use Block Coordinate Descent (BCD) that consists in iteratively solving the problem for π s or π v with the other kept fixed. Interestingly, this boils down to solving at each step a linear OT problem that requires O(n 3 log(n)) operations with a network simplex algorithm as detailed in the pseudo-code given in Algorithm 9. This approach, also known as the "mountain climbing procedure" [Konno 1976a] in the BP literature, was proved to decrease the loss at each iteration and so to converge within a finite number of iterations [START_REF] Horst | [END_REF]]. We also note that at each iteration one needs to compute the equivalent cost matrix L(X, X ) ⊗ π (•) which has a complexity of O(ndn d ). However, one can reduce it using Proposition 1 from [START_REF] Peyré | [END_REF]] for the case when L is the squared Euclidean distance | • | 2 or the Kullback-Leibler divergence. In this case, the overall computational complexity becomes O(min{(n + n )dd + n 2 n; (d + d )nn + d 2 d}). We refer the interested reader to Section 6.3.2 for further details.

Finally, we can use the same BCD procedure for the entropic regularized version of COOT (5.1) where each iteration an entropic regularized OT problem can be solved efficiently using Sinkhorn's algorithm [Cuturi 2013] with several possible improvements [Altschuler 2017[START_REF] Altschuler | [END_REF], Alaya 2019]. Note that this procedure can be easily adapted in the same way to include unbalanced OT problems [Chizat 2017] as well.

Relation with other OT distances

Gromov-Wasserstein

The COOT problem is defined for arbitrary matrices X ∈ R n×d , X ∈ R n ×d and so can be readily used to compare pairwise similarity matrices between the samples C = (c(x i ,

x j ) i,j ) ∈ R n×n , C = (c (x k , x l )) k,l ∈
R n ×n for some c, c . To avoid redundancy, we use the term "similarity" for both similarity and distance functions in what follows. This situation arises in applications dealing with relational data, e.g, in a graph context (see Chapter 3) or e.g. in deep metric alignment [Ezuz 2017]. These problems have been successfully tackled using the Gromov-Wasserstein distance (see Chapter 2). We recall that given C ∈ R n×n and C ∈ R n ×n , the GW distance is defined by:

GW (C, C , w, w ) = min π s ∈Π(w,w ) L(C, C ) ⊗ π s , π s F (5.2)
As suggested by the similar objective functions and constraints, GW and COOT are linked in multiple ways. Below, we explicit the link between GW and COOT using a reduction of a concave QP to an associated BP problem established in [Konno 1976b] and show that they are equivalent when working with squared Euclidean distance matrices C ∈ R n×n , C ∈ R n ×n or with inner product similarities (see Chapter 4). More precisely, this latter equivalence follows from [Konno 1976b] where it was shown that a concave QP can be solved by a reduction to an associated BP problem:

Theorem 5.5.1 (Adapted from [Konno 1976b]). If Q is a negative definite matrix then problems:

min x f (x) = c T x + 1 2 x T Qx s.t. Ax = b, x≥ 0 (5.3) min x,y g(x, y) = 1 2 c T x + 1 2 c T y + 1 2 x T Qy s.t.
Ax = b, Ay = b, x, y≥ 0 (5.4) are equivalent. More precisely, if x * is an optimal solution for (5.3), then (x * , x * ) is a solution for (5.4) and if (x * , y * ) is optimal for (5.4), then both x * and y * are optimal for (5.3).

Using this principle one can link GW with the COOT problem when working on intra domain similarity matrices C ∈ R n×n , C ∈ R n ×n thanks to the next proposition:

Proposition 5.5.1. Let L = | • | 2 and suppose that C ∈ R n×n , C ∈ R n ×n are squared Euclidean distance matrices such that C = x1 T n + 1 n x T -2XX T , C = x 1 T n + 1 n x T -2X X T with x = diag(XX T ), x = diag(X X T ). Then,

the GW problem can be written as a concave quadratic program (QP) which Hessian reads

Q = -4 * XX T ⊗ K X X T . If C ∈ R n×n , C ∈ R n ×n are inner products similarities, i.e. such that C = XX T , C = X X T then the GW is also a concave quadratic program (QP) which Hessian reads Q = -2 * XX T ⊗ K X X T .
When working with arbitrary similarity matrices, COOT provides a lower-bound for GW and using Proposition 5.5.1 we can prove that both problems become equivalent for the cases of squared Euclidean distances and inner product similarities.

Proposition 5.5.2. Let C ∈ R n×n , C ∈ R n ×n be any symmetric matrices, then: COOT(C, C , w, w , w, w ) ≤ GW (C, C , w, w ).

Algorithm 10 DC Algorithm for solving COOT and GW with squared Euclidean matrices or inner product similarities 1: π (0) ← ww T , k ← 0 2: while k < maxIt and err > 0 do 3:

π (k) ← arg min π∈Π(w,w ) L(C, C ) ⊗ π (k-1) , π F // linear OT problem 4: err ← ||π (k-1) -π (k) || F 5: k ← k + 1 6: end while 7: return π (k) for GW and (π (k) , π (k) ) for COOT
The converse is also true under the hypothesis of Proposition 5.5.1. In this case, if (π s * , π v * ) is an optimal solution of (COOT), then both π s * , π v * are solutions of (5.2). Conversely, if π s * is an optimal solution of (5.2), then (π s * , π s * ) is an optimal solution for (COOT) .

Equivalence of algorithms Under the hypothesis of Proposition 5.5.1 we know that there exists an optimal solution for the COOT problem of the form (π * , π * ), where π * is an optimal solution of the GW problem. This gives a conceptually very simple fixed-point procedure where one only iterates over one coupling in order to compute a optimal solution of GW as described in Algorithm 10. Interestingly enough, the iterations of the fixed point method are exactly equivalent to the Frank Wolfe procedure described in Chapter 3, since, in the concave setting, the line search step can be fixed to 1 [START_REF] Maron | [END_REF]] (see Section 6.3.6 for more details). Also note that the steps of Algorithm 10 are iterations of Difference of Convex Algorithm (DCA) [Tao 2005, Yuille 2003] where the concave function is approximated at each iteration by its linear majorization. When applying the same procedure for entropic regularized COOT, the resulting DCA also recovers exactly the projected gradients iterations proposed in [START_REF] Peyré | [END_REF]] for solving the entropic regularized version of GW.

We would like to stress out that COOT is much more than a generalization of GW and that is for multiple reasons. First, it can be used on raw data without requiring to choose or compute the similarity matrices, that can be costly, for instance, when dealing with shortest path distances in graphs, and to store them (O(n 2 + n 2 ) overhead). Second, it can take into account additional information given by feature weights v, v and provides an interpretable mapping between them across two heterogeneous datasets. Finally, contrary to GW, COOT is not invariant neither to feature rotations nor to the change of signs leading to a more informative samples' coupling when compared to GW in some applications. One such example is given in the previous MNIST-USPS transfer task (Figure 5.1) for which the coupling matrix obtained via GW (given in Figure 5.4) exhibits important flaws in respecting class memberships when aligning samples.

Relation with Invariant OT and hierarchical approaches

In [Alvarez-Melis 2019], authors consider a scenario where the OT problem is used to align measures supported on sets of points for which meaningful pairwise distances are hard or impossible to calculate. This may happen, for instance, due to some latent transformation that have been applied to the features. The main underlying idea of their approach is to find an assignment of the points and to calculate a transformation to match the features. More precisely, for two datasets X, X , with same feature space R d , the corresponding objective function is:

InvOT (X, X ) = min π∈Π(w,w ) min P∈F d M P , π F (5.5)
where

M P (i, j) = x i -Px j 2 2 and F d is a space of matrices F d = {P ∈ R d×d | ||P|| F = √ d}.
As noted by the authors in Lemma 4.3, equation (5.5) can be related to the GW problem when C, C are calculated using inner product similarities and when X is w -whitened, i.e X T diag(w )X = I d . In this case, author show that GW and InvOT are equivalent, namely a solution of GW is a solution of InvOT and conversely. Since the GW problem with cosine similarities is actually concave we have proven COOT and GW are also equivalent in this case which proves the following proposition: Proposition 5.5.3. Using previous notations, L = | • | 2 , d = d , and inner product similarities C = XX T , C = X X T . Suppose that X is w -whitened, i.e X T diag(w )X = I d . Then, InvOT (X, X ), COOT(C, C ) and GW (C, C ) are equivalent, namely any optimal coupling of one of this problem is a solution to others problems.

Another way of proving this result is to consider the Theorem 4.2.1 of Chapter 4 where we proved that the Gromov-Wasserstein distance, when considering inner product similarities, is equivalent to the problem:

min π∈Π(w,w ) min P∈F d n,n i=1,j=1 x i , Px j π ij (MaxOT)
When X is w -whitened we can check easily by developping the terms in M P (i, j) = x i -Px j 2 2 that InvOT is equivalent to (MaxOT) (see Chapter 4). InvOT was further used as a building block for aligning clustered datasets in [START_REF] Lee | [END_REF] where the authors applied it as a divergence measure between the clusters, thus leading to an approach different from ours. Finally, in [Yurochkin 2019] the authors proposed a hierarchical OT distance as an OT problem with costs defined based on precomputed Wasserstein distances but with no global features' mapping, contrary to COOT that optimises two couplings of the features and the samples simultaneously.

Election isomorphism

The election isomorphism problem mentioned earlier has recently been introduced in [Faliszewski 2019] to compare two elections given by preference orders for candidates of voters. The authors express their problem quite similarly to COOT and also seek for correspondences between voters and candidates across two elections where the preferences of each voter are known (which is unrealistic in modern democracies). They focus on the setting where both elections have exactly the same number of voters n = n and candidates d = d and search for an optimal permutation via a Linear Integer Program. It is interesting to see that their problem consisting in aligning voters using the Spearman distance is actually equivalent to solving COOT with L = | • | on voters preferences. To that extent, COOT is a more general approach as it is applicable for general loss functions L, contrary to the Spearman distance used in [Faliszewski 2019], and generalizes to the cases where n = n and m = m . A more detailed comparison is given in Section 6.3.7.

Experiments

In the next section, we highlight two possible applications of COOT in a machine learning context: HDA and co-clustering. We consider these two particular tasks because 1) OT-based methods are considered as a strong baseline in DA; 2) COOT is a natural match for co-clustering as it allows for soft assignments of data samples and features to co-clusters.

Heterogeneous domain adaptation

In a classification context, the problem of domain adaptation (DA) arises whenever one has to perform classification on a set of data X t = {x t i } Nt i=1 (usually called the target domain) but has only few or no labelled data associated. Given a source domain X s = {x s i } Ns i=1 with associated labels Y s = {y s i } Ns i=1 , one would like to leverage on this knowledge to train a classifier in the target domain. Unfortunately, direct use of the source information usually leads to poor results because of the discrepancy between source and target distributions. Among others, several works, e.g. [Courty 2017], use OT to perform this adaptation. However, in the case where the data do not belong to the same metric space (X s ∈ R Ns×d and X t ∈ R Nt×d with d = d ), the problem is getting harder as domains probability distributions can not be anymore compared or aligned in a straightforward way. This instance of the DA problem, known as Heterogeneous Domain Adaptation (HDA), has received less attention in the literature, partly due to the lack of appropriate divergence measures that can be used in such context. State-of-the-art HDA methods include Canonical Correlation Analysis [START_REF] Yeh | [END_REF]] and its kernelized version and a more recent approach based on the Gromov-Wasserstein discrepancy [START_REF][END_REF]]. Usually, one considers a semi-supervised variant of the problem, where one has access to a small number n t of labelled samples per class in the target domain, because the unsupervised problem (n t = 0) is much more difficult. We investigate here the use of COOT for both semi-supervised HDA, where one has access to a small number n t of labelled samples per class in the target domain and unsupervised HDA with n t = 0.

Solving HDA with COOT In order to solve the HDA problem, we compute COOT(X s , X t ) between the two domains and use the π s matrix providing a transport/correspondence between samples (as illustrated in Figure 5.1) to estimate the labels in the target domain via label propagation [Redko 2019]. Assuming uniform sample weights and one-hot encoded labels, a class prediction Ŷt in the target domain samples can be obtained by computing Ŷt = π s Y s . When labelled target samples are available, we further prevent source samples to be mapped to target samples from a different class by adding a high cost in the cost matrix for every such source sample as suggested in [Courty 2017, Section 4.2

Datasets

We choose to test our method on the classical Caltech-Office dataset [Saenko 2010], which is dedicated to object recognition in images from several domains. Those domains exhibit variability in term of presence/absence of background, lightning conditions, image quality, that as such induce distribution shifts between the domains. Among the available domains, we select the following three: Amazon (A), the Caltech-256 image collection (C) and Webcam (W). Ten overlapping classes between the domains are used and two different deep feature representations of image in each domain are obtained using two different neural networks, namely: the Decaf [START_REF] Donahue | [END_REF]] and GoogleNet [START_REF] Szegedy | [END_REF]] neural network architectures. In both cases, we extract the image representations as the activations of the last fully-connected layer, yielding respectively sparse 4096 and 1024 dimensional vectors. The heterogeneity comes from these two very different representations.

Competing methods and experimental settings

We evaluate COOT on Amazon (A), Caltech-256 (C) and Webcam (W) domains from Caltech-Office dataset [Saenko 2010] with 10 overlapping classes between the domains and two different deep feature representations obtained for images from each domain using the Decaf [START_REF] Donahue | [END_REF]] and GoogleNet [START_REF] Szegedy | [END_REF]] neural network architectures. In both cases, we extract the image representations as the activations of the last fully-connected layer, yielding respectively sparse 4096 and 1024 dimensional vectors. The heterogeneity comes from these two very different representations. We consider 4 baselines: CCA, its kernalized version KCCA [START_REF] Yeh | [END_REF]] with a Gaussian kernel which width parameter is set to the inverse of the dimension of the input vector, EGW representing the entropic version of GW and SGW [START_REF][END_REF]] that incorporates labelled target data into two regularization terms. For EGW and SGW, the entropic regularization term was set to 0.1, and the two other regularization hyperparameters for the semi-supervised case to λ = 10 -5 and γ = 10 -2 as done in [START_REF][END_REF]]. We use COOT with entropic regularization on the feature mapping, with parameter ε 2 = 1 in all experiments. For all OT methods, we use label propagation to obtain target labels as the maximum entry of Ŷt in each row. For all non-OT methods, classification was conducted with a k-nn classifier with k = 3. We run the experiment in a semi-supervised setting with n t = 3, i.e., 3 samples per class were labelled in the target domain. The baseline score is the result of classification by only considering labelled samples in the target domain as the training set. For each pair of domains, we selected 20 samples per class to form the learning sets. We run this random selection process 10 times and consider the mean accuracy of the different runs as a performance measure.

Results

We first provide in Table 5.1 the results for the semi-supervised case where we perform adaptation from Decaf to GoogleNet features. Note that we report the results in the opposite direction in Section 6.3.8 for n t ∈ {0, 1, 3, 5}. From it, we see that COOT surpasses all the other state-of-the-art methods in terms of mean accuracy. This result is confirmed by a p-value lower than 0.001 on a pairwise method comparison with COOT in a Wilcoxon signed rank test. SGW provides the second best result, while CCA and EGW have a less than average performance. Finally, KCCA performs better than the two latter methods, but still fails most of the time to surpass the no-adaptation baseline score given by a classifier learned on the available labelled target data. Results for the unsupervised case can be found in Table 5.2. This setting is rarely considered in the literature as unsupervised HDA is regarded as a very difficult problem. In this table, we do not provide scores for the no-adaptation baseline and SGW, as they require labelled data.

As one can expect, most of the methods fail in obtaining good classification accuracies in this setting, despite having access to discriminant feature representations. Yet, COOT succeeds in providing a meaningful mapping in some cases. The overall superior performance of COOT highlights its strengths and underlines the limits of other HDA methods. First, COOT does not depend on approximating empirical quantities from the data, contrary to CCA and KCCA that rely on the estimation of the cross-covariance matrix that is known to be flawed for high-dimensional data with few samples [Song 2016]. Second, COOT takes into account the features of the raw data that are more informative than the pairwise distances used in EGW. Finally, COOT avoids the sign invariance issue discussed previously that hinders GW's capability to recover classes without supervision as illustrated for the MNIST-USPS problem before.

Co-clustering and data summarization

While clustering methods present an important discovery tool for data analysis, one of their main limations is to completely discard the potential relationships that may exist between the features that describe the data samples. For instance, in recommendation systems, where each user is described in terms of his or her preferences for some product, clustering algorithms may benefit from the knowledge about the correlation between different products revealing their probability of being recommended to the same users. This idea is the cornerstone of co-clustering [Hartigan 1972] where the goal is to perform clustering of both samples and features simultaneously. More precisely given a data matrix X ∈ R n×d and the number of samples (rows) and features (columns) clusters denoted by g ≤ n and m ≤ d, respectively, we seek to find X c ∈ R g×m that summarizes X in the best way possible.

COOT-clustering

We look for X c which is as close as possible to the original X w.r.t. COOT by solving:

min Xc COOT(X, X c ) = min π s ,π v ,Xc L(X, X c ) ⊗ π s , π v F (5.6)
with potentially entropic regularization. More precisely, we set w, w , v, v as uniform, initialize X c with random values and apply the BCD algorithm over (π s , π v , X c ) by alternating between the following steps:

1. Obtain π s and π v by solving COOT(X, X c ) 2. Set X c to gmπ s Xπ v .

This second step of the procedure is a least-square estimation when L = |•| 2 and corresponds to minimizing the COOT objective w.r.t.. X c . In practice, we observed that few iterations of this procedure are enough to ensure the convergence. Once solved, we use the soft assignments provided by coupling matrices π s ∈ R n×g , π v ∈ R d×m to partition data points and features to clusters by taking the index of the maximum element in each row of π s and π v , respectively.

Simulated data

We follow [START_REF] Laclau | Coclustering through Optimal Transport[END_REF]] where four scenarios with different number of co-clusters, degrees of separation and sizes were considered (for details, see the supplementary materials). We choose to evaluate COOT on simulated data as it provides us with the ground-truth for feature clusters that are often unavailable for real-world data sets. As in [START_REF] Laclau | Coclustering through Optimal Transport[END_REF]], we use the same co-clustering baselines including ITCC [START_REF] Dhillon | [END_REF]], Double K-Means (DKM) [START_REF] Rocci | [END_REF]], Orthogonal Nonnegative Matrix Tri-Factorizations (ONTMF) [START_REF] Ding | [END_REF]], the Gaussian Latent Block Models (GLBM) [START_REF] Nadif | Algorithms for Model-based Block Gaussian Clustering[END_REF]] and Residual Bayesian Co-Clustering (RBC) [START_REF] Shan | [END_REF]] as well as the K-means and NMF run on both modes of the data matrix, as clustering baseline. The performance of all methods is measured using the co-clustering error (CCE) defined as follows [START_REF] Patrikainen | [END_REF]]: CCE((z, w), (ẑ, ŵ)) = e(z, ẑ) + e(w, ŵ) -e(z, ẑ) × e(w, ŵ) (5.7)

where ẑ and ŵ are the partitions of samples and features estimated by the algorithm; z and w are the true partitions and e(z, ẑ) (resp. e(w, ŵ)) denotes the error rate, i.e., the proportion of misclassified instances (resp. features). For all configurations, we generate 100 data sets and present the mean and standard deviation of the CCE over all sets for all baselines in Table 5.4. Based on these results, we see that our algorithm outperforms all the other baselines on D1, D2 and D4 data sets, while being behind (CCOT-GW) proposed by [START_REF] Laclau | Coclustering through Optimal Transport[END_REF]] on D3. This result is rather strong as our method relies on the original data matrix, while (CCOT-GW) relies on its kernel representation and thus benefits from the non-linear information captured by it. Finally, we note that while both competing methods rely on OT, they remain very different as (CCOT-GW) approach is based on detecting the positions and the number of jumps in the scaling vectors of GW entropic regularized solution, while our method relies on coupling matrices to obtain the partitions.

Olivetti Face dataset As a first application of COOT for the co-clustering problem on real data, we propose to run the algorithm on the well known Olivetti faces dataset [START_REF] Ferdinando | Parameterisation of a stochastic model for human face identification[END_REF]].

We take 400 images normalized between 0 and 1 and run our algorithm with g = 9 image clusters and m = 40 feature (pixel) clusters. As before, we consider the empirical distributions supported on images and features, respectively. The resulting reconstructed image's clusters are given in Figure 5.5 and the pixel clusters are illustrated in its rightmost part. We can see that despite the high variability in the data set, we still manage to recover detailed centroids, whereas L2-based clustering such as standard NMF or k-means based on 2 norm cost function are known to provide blurry estimates in this case. Finally, as in the MNIST-USPS example, COOT recovers spatially localized pixel clusters with no prior information about the pixel relations.

MovieLens

We now evaluate our approach on the benchmark MovieLens-100K1 data set that provides 100,000 user-movie ratings, on a scale of one to five, collected from 943 users on 1682 movies. The Promise, The (Versprechen, Das) (1994) Rear Window (1954) To Cross the Rubicon (1991) Usual Suspects, The (1995) Daens (1992)

Table 5.5: Top 5 of movies in clusters M1 and M20. Average rating of the top 5 rated movies in M1 is 4.42, while for the M20 it is 1.

main goal of our algorithm here is to summarize the initial data matrix so that X c reveals the blocks (co-clusters) of movies and users that share similar tastes. We set the number of user and film clusters to g = 10 and m = 20, respectively as in [START_REF] Banerjee | [END_REF]].

The obtained results provide the first movie cluster consisting of films with high ratings (3.92 on average), while the last movie cluster includes movies with very low ratings (1.92 on average). Among those, we show the 5 best/worst rated movies in those two clusters in Table 5.5. Overall, our algorithm manages to find a coherent co-clustering structure in MovieLens-100K and obtains results similar to those provided in [START_REF] Laclau | Coclustering through Optimal Transport[END_REF][START_REF] Banerjee | [END_REF].

Conclusion

In this chapter, we presented a novel variant of the optimal transport problem which aims at comparing distributions supported in different spaces. To this end, two optimal transport maps, one acting on the sample space, and the other on the feature space, are optimized to connect the two heterogenous distributions. We show that this novel problem has connections with bilinear assignment and provide algorithms to solve it. We demonstrate its usefulness and versatility on two difficult machine learning problems: heterogeneous domain adaptation and co-clustering/data summarization, where promising results were obtained.

Numerous follow-up of this work are expected. Beyond the potential applications of the method in various contexts, such as e.g. statistical matching, data analysis or even losses in deep learning settings, one immediate and intriguing question lies into the generalization of this framework in the continuous setting, and the potential connections to duality theory. This might lead to stochastic optimization schemes enabling large scale solvers for this problem.

Proofs and additional results of Chapter 3

This section contains all the proofs of the claims and additional results of the Chapter 3. In the following we will denote by H q the Wasserstein loss and by J q the Gromov-Wasserstein loss. More precisely, with notation of Chapter 3: ((x , a ), (y , b )) ( 6.3)

H q (π) = ˆd(a, b) q dπ(a, b) (6.1) J q (d X , d Y , π) = ˆˆL(x, y, x , y) q dπ(x, y)dπ(x , y ) = ˆˆ|d X (x, x ) -d Y (y, y )| q dπ(x, y)dπ(x , y ) (6.2) E p,q,α (π) = ˆˆ (1 -α)d(a, b) q + α|d X (x, x ) -d Y (y, y )| q p dπ((x, a), (y, b))dπ
We note P i #π the projection on the marginal i of π.

Additional results -Comparison with W and GW

Cross validation results

During the nested cross validation, we divided the dataset into 10 and use 9 folds for training, where α is chosen within [0, 1] via a 10-CV cross-validation, 1 fold for testing, with the best value of α (with the best average accuracy on the 10-CV) previously selected. The experiment is repeated 10 times for each dataset except for MUTAG and PTC where it is repeated 50 times. Table 6.1 and 6.2 report the average number of time α was chose within ]0, ...1[ without 0 and 1 corresponding to the Wasserstein and Gromov-Wasserstein distances respectively. Results suggests that both structure and feature pieces of information are necessary as α is consistently selected inside ]0, ...1[ except for PTC and COX2. Nested CV results We report in tables 6.4 and 6.3 the average classification accuracies of the nested classification procedure by taking W and GW instead of F GW (i.e by taking α = 0, 1). Best result for each dataset is in bold. A (*) is added when best score does not yield to a significative improvement compared to the second best score. The significance is based on a Wilcoxon signed rank test between the best method and the second one.

Results illustrates that F GW encompasses the two cases of W and GW , as scores of F GW are usually greater or equal on every dataset than scores of both W and GW and when it is not the case the difference is not statistically significant.

Timings

In this paragraph we provide some timings for the discrete attributed datasets. Table 6.5 displays the average timing for computing F GW between two pair of graphs. We recall the proposition:

Proposition. Comparison between F GW , GW and W .

• The following inequalities hold: F GW α,p,1 (µ, ν)(µ, ν) ≤ 2W p (µ, ν). (6.6) Proof. For the two inequalities (6.4) and ( 6.5) let π be an optimal coupling for the Fused Gromov-Wasserstein distance between µ and ν. Clearly:

F GW α,p,q (µ, ν) ≥ (1 -α)W pq (µ A , ν B ) q (6.4) F GW α,p,q (µ, ν) ≥ αGW pq (µ X , ν Y ) q (6.
F GW α,p,q (µ, ν) = (X ×Ω×Y×Ω) 2
(1 -α)d(a, b) q + αL(x, y, x , y ) q p dπ((x, a), (y, b))dπ ((x , a ), (y , b ))

1 p ≥ X ×Ω×Y×Ω (1 -α) p d(a, b) pq dπ((x, a), (y, b)) 1 p = (1 -α) Ω ×Ω d(a, b) pq dP 2,4 #π(a, b) 1 p
Since π ∈ Π(µ, ν) the coupling P 2,4 #π is in Π(µ A , ν B ). So by suboptimality:

F GW α,p,q (µ, ν)≥(1 -α)(W pq (µ A , ν B )) q
which proves equation (6.4). Same reasoning is used for equation (6.5).

For the last inequality (6.6) let π ∈ Π(µ, ν) be any admissible coupling. By suboptimality: (*) is the triangle inequality of d Z and (**) Minkowski inequality. Since this inequality is true for any admissible coupling π we can apply it with the optimal coupling for the Wasserstein distance defined in the proposition and the claim follows.

F GWα,p,1(µ, ν) ≤ (X×Ω×Y×Ω) 2 (1-α)d(a, b) + α|dZ (x, x ) -dZ (y, y )| p dπ((x,

Proof of Theorem 3.5.1 -Metric properties of FGW

We recall the theorem: Theorem (Metric properties). Let p, q ≥ 1, α ∈]0, 1[ and (µ, ν) ∈ S pq (Ω) × S pq (Ω). The functional π → E p,q,α (π) always achieves an infimum π * in Π(µ, ν) s.t. F GW α,p,q (µ, ν) = E p,q,α (π * ) < +∞. Moreover:

• F GW α,p,q is symmetric and, for q = 1, satisfies the triangle inequality. For q ≥ 2, the triangular inequality is relaxed by a factor 2 q-1 .

• For α ∈]0, 1[, F GW α,p,q (µ, ν) = 0 if an only if there exists a bijective function φ = (φ 1 , φ 2 ) : supp(µ) → supp(ν) such that:

φ#µ = ν ∀(x, a) ∈ supp(µ) , φ 2 (x, a) = a ∀(x, a), (x , a ) ∈ supp(µ) 2 , d X (x, x ) = d Y (φ 1 (x, a), φ 1 (x , a ))
• If (µ, ν) are generalized labeled graphs then F GW α,p,q (µ, ν) = 0 if and only if (X × Ω, d X , µ) and

(Y × Ω, d Y , ν) are (II)-strongly isomorphic.
We propose to prove the theorem point by point: first the existence, then the the triangle inequality statement and finally the equality relation. Proposition 6.1.1 (Existence of an optimal coupling for the F GW distance.). For p, q ≥ 1, π → E p,q,α (π) always achieves a infimum π * in Π(µ, ν) such that F GW α,p,q (µ, ν) = E p,q,α (π * ) < +∞.

Proof. Since X × Ω and Y × Ω are Polish spaces we known that Π(µ, ν) ⊂ P(X × Ω × Y × Ω) is compact (Theorem 1.7 in [Santambrogio 2015]), so by applying Weierstrass theorem we can conclude that the infimum is attained at some π * ∈ Π(µ, ν) if π → E p,q,α (π) is l.s.c.

We will use Lemma 2.2.1 to prove that the functionnal is l. where in (*) we used equation (2.44) in Memo 2.2.2 and in (**) that µ, ν are in S pq (Ω).

Proposition 6.1.2 (Symmetry and triangle inequality.). F GW α,p,q is symmetric and for q = 1 satisfies the triangle inequality. For q ≥ 2 the triangular inequality is relaxed by a factor 2 q-1

To prove this result we will use the following lemma:

Lemma 6.1.1. Let (X × Ω, d X , µ), (Y × Ω, d Y , β), (Z × Ω, d Z , ν) ∈ S(Ω) 3 . For (x, a), (x , a ) ∈ (X × Ω) 2 , (y, b), (y , b ) ∈ (Y × Ω) 2 and (z, c), (z , c ) ∈ (Z × Ω) 2
we have:

L(x, z, x , z ) q ≤ 2 q-1 (L(x, y, x , y ) q + L(y, z, y , z ) q ) (6.8) d(a, c) q ≤ 2 q-1 (d(a, b) q + d(b, c) q ) (6.9)

Proof. Direct consequence of equation (2.44) in Memo 2.2.2 and triangle inequalities of d, d X , d Y , d Z .

Proof of Proposition 6.1.2. To prove the triangle inequality of F GW α,p,q distance for arbitrary measures we will use the Gluing lemma which stresses the existence of couplings with a prescribed structure. Let

(X × Ω, d X , µ), (Y × Ω, d Y , β), (Z × Ω, d Z , ν) ∈ S(Ω) 3 .
Let π 1 ∈ Π(µ, β) and π 2 ∈ Π(β, ν) be optimal transportation plans for the Fused Gromov-Wasserstein distance between µ, β and β, ν respectively. By the Gluing Lemma (see [Villani 2008] and Lemma 5.3.2 in [START_REF][END_REF]) there exists a probability measure π ∈ P (X × Ω) × (Y × Ω) × (Z × Ω) with marginals π 1 on (X × Ω) × (Y × Ω) and π 2 on (Y × Ω) × (Z × Ω). Let π 3 be the marginal of π on (X × Ω) × (Z × Ω). By construction π 3 ∈ Π(µ, ν). So by suboptimality of π 3 : F GWα,p,q(dX , dZ , µ, ν) ≤ (X×Ω×Z×Ω) 2

(1 -α)d(a, c) q + αL(x, z, x , z ) q p dπ3((x, a), (z, c))dπ3 ((x , a ), (z , c ))

1 p = (X×Ω×Y×Ω×Z×Ω) 2
(1 -α)d(a, c) q + αL(x, z, x , z ) q p dπ((x, a), (y, b), (z, c))dπ ((x , a ), (y , b ), (z , c ))

1 p ( * ) ≤ 2 q-1 (X×Ω×Y×Ω×Z×Ω) 2 (1 -α)d(a, b) q + (1 -α)d(b, c
) q + αL(x, y, x , y ) q + αL(y, z, y , z ) q p dπ((x, a), (y, b), (z, c))dπ ((x , a ), (y , b ), (z , c ))

1 p ( * * ) ≤ 2 q-1 (X×Ω×Y×Ω×Z×Ω) 2
(1 -α)d(a, b) q + αL(x, y, x , y ) q p dπ((x, a), (y, b), (z, c))dπ ((x , a ), (y , b ), (z , c ))

1 p + (X×Ω×Y×Ω×Z×Ω) 2
(1 -α)d(b, c) q + αL(y, z, y , z ) q p dπ((x, a), (y, b), (z, c))dπ ((x , a ), (y , b ), (z , c ))

1 p = 2 q-1 (X×Ω×Y×Ω) 2
(1 -α)d(a, b) q + αL(x, y, x , y ) q p dπ1((x, a), (y, b))dπ1 ((x , a ), (y , b ))

1 p + (Y×Ω×Z×Ω) 2
(1 -α)d(b, c) q + αL(y, z, y , z ) q p dπ2((y, b), (z, c)) dπ2 ((y , b ), (z , c ))

1 p = 2 q-1 (F GWα,p,q(µ, β)+F GWα,p,q(β, ν))

with (*) comes from (6.8) and (6.9) and (**) is Minkowski inequality. So when q = 1, F GW α,p,q satisfies the triangle inequality and when q > 1, F GW α,p,q satisfies a relaxed triangle inequality so that it defines a semi-metric as described previously.

Proposition 6.1.3 (Equality relation.). For α ∈]0, 1[, F GW α,p,q (µ, ν) = 0 if an only if there exists a bijective function φ = (φ 1 , φ 2 ) : supp(µ) → supp(ν) such that:

φ#µ = ν (6.10) ∀(x, a) ∈ supp(µ) , φ 2 (x, a) = a (6.11) ∀(x, a), (x , a ) ∈ supp(µ) 2 , d X (x, x ) = d Y (φ 1 (x, a), φ 1 (x , a )) (6.12)
Moreover if (µ, ν) are generalized labeled graphs then F GW α,p,q (µ, ν) = 0 if and only if (X × Ω, d X , µ) and (Y × Ω, d Y , ν) are (II)-strongly isomorphic.

Proof. For the first point, let us assume that there exists a function φ verifying (6.10), (6.11) and (6.12).

We consider the map π = (I d × φ)#µ ∈ Π(µ, ν). We note φ = (φ 1 , φ 2 ). Then:

E p,q,α (π) = (X ×Ω×Y×Ω) 2
(1 -α)d(a, b) q + αL((x, y, x , y ) q p dπ((x, a), (y, b))dπ ((x , a )

, (y , b )) = (X ×Ω) 2
(1 -α)d(a, φ 2 (x, a)) q + αL (x, φ 1 (x, a), x , φ 1 (x , a )

q p dµ(x, a)dµ(x , a ) = (X ×Ω) 2 (1 -α)d(a, φ 2 (x, a)) q + α|d X (x, x ) -d Y (φ 1 (x, a), φ 1 (x , a ))| q p dµ(x, a)dµ(x , a ) = 0 (6.13)
Conversely suppose that F GW α,p,q (µ, ν) = 0. To prove the existence of a map φ : supp(µ) → supp(ν) verifying (6.10), (6.11) and ( 6.12) we will use the Gromov-Wasserstein properties. We are looking for a vanishing Gromov-Wassersein distance between the spaces X × Ω and Y × Ω equipped with our two measures µ and ν.

More precisely, we define for (x, a), (y, b), (x , a ), (y , b ) ∈ (X × Ω × Y × Ω) 2 and β ∈]0, 1[:

d X ×Ω (x, a), (x , a ) = (1 -β)d X (x, x ) + βd(a,
∀π ∈ Π(µ, ν), ∀n ≥ 1, J n (d X ×Ω , d Y×Ω , π) ≤ (1 -β)J n (d X , d Y , π) + βM n (π) (6.14)
Now let π * be an optimal coupling for F GW α,p,q between µ and ν. By hypothesis F GW α,p,q (µ, ν) = 0 so that: (6.15) and: In particular π * is concentrated on {(x, y) = φ(x, y)} or equivalently π * = (I d × φ)#µ. Injecting π * in (6.16) leads to:

J qp (d X , d Y , π * ) = 0
H qp (π * ) = 0. ( 6
H qp (π * ) = (X ×Ω×Y×Ω) d(a, b) qp dπ * ((x, a), (y, b)) = X ×Ω d(a, φ 2 (x, a)) qp dµ(x, a) = 0 (6.19)
Which implies:

∀(x, a) ∈ supp(µ) , φ 2 (x, a) = a (6.20)
Moreover, using the equality (6.18) we can conclude that:

∀(x, a)(x , a ) ∈ supp(µ) 2 , d X (x, x ) = d Y (φ 1 (x, a), φ 1 (x , a )) (6.21)
In this way f verifies all the properties (6.10), (6.11),(6.12). Moreover suppose that µ and ν are generalized labeled graphs. In this case there exists f : X → Ω surjective such that µ = (id × f )#µ X . Then (6.21) implies that:

∀(x, x ) ∈ supp(µ X ) 2 , d X (x, x ) = d Y (φ 1 (x, f (x)), φ 1 (x , f (x ))) (6.22)
We define I : supp(µ X ) → supp(µ Y ) such that I(x) = φ 1 (x, f (x)). Then we have by ( 6 Overall φ satisfies all P.1, P.2 and P.3 if µ and ν are generalized labeled graphs. The converse is also true using the reasoning in (6.13).

Proof of Theorem 3.5.2 -Convergence and concentration inequality.

We recall the theorem:

Theorem. Convergence of finite samples and a concentration inequality Let p ≥ 1. We have:

lim n→∞ F GW α,p,1 (µ n , µ) = 0
Moreover, suppose that s > d * p (µ). Then there exists a constant C that does not depend on n such that:

E[F GW α,p,1 (µ n , µ)] ≤ Cn -1 s .
The expectation is taken over the i.i.d samples (x i , a i ). A particular case of this inequality is when α = 1 so that we can use the result above to derive a concentration result for the Gromov-Wasserstein distance. More precisely, if ν n = 1 n i δ xi denotes the empirical measure of ν ∈ P(X ) and if s > d * p (ν) we have:

E[GW p (ν n , ν)] ≤ C n -1 s .
Proof. The proof of the convergence in F GW derives directly from the weak convergence of the empirical measure and Lemma 2.2.1. Moreover, since µ n and µ are both in the same ground space, we have:

F GW α,p,1 (µ n , µ) ≤ 2W p (µ n , µ) =⇒ E[F GW α,p,1 (µ n , µ)] ≤ 2E[W p (µ n , µ)].
We can directly apply theorem 1 in [Weed 2017] to state the inequality.

Proof of proposition 3.5.2 -Interpolation properties between GW and W

We recall the proposition:

Proposition (Interpolation properties.). As α tends to zero, one recovers the Wasserstein distance between the features information and as α goes to one, one recovers the Gromov-Wasserstein distance between the structure information:

lim α→0 F GW α,p,q (µ, ν) = (W pq (µ A , ν B )) q lim α→1 F GW α,p,q (µ, ν) = (GW pq (µ X , ν Y )) q
Proof. Let π OT ∈ Π(µ A , ν B ) be an optimal coupling for the pq-Wasserstein distance between µ A and ν B . We can use the same Gluing lemma (lemma 5.3.2 in [START_REF][END_REF]) to construct: Let α ≥ 0 and π α optimal plan for the Fused Gromov-Wasserstein distance between µ and ν.

ρ ∈ P( µ X × π OT Ω × ν Ω × Y) such that ρ ∈ Π(µ,
We can deduce that: , a, b, y)dρ(x , a , b , y ).

F GWα,p,q(µ, ν) p -(1 -α) p Wpq(µA, νB) pq = (X×Ω×Y×Ω) 2 (1 -α)d(a, b) q + αL(x, y, x , y ) q p dπα((x, a), (y, b))dπα((x , a ), (y , b )) - Ω×Ω (1 -α) p d(a, b) pq dπOT (a, b) ( * ) ≤ (X×Ω×Y×Ω) 2 (1 -α)d(a, b) q + αL(x, y, x , y ) q p dρ(x, a, b, y)dρ(x , a , b , y ) - X×Ω×Y×Ω (1 -α) p d(a, b) pq dρ(x, a, b, y) = (1 -α) p (X×Ω×Y×Ω) 2 d(a, b) pq dρ(x, a, b, y)dρ(x , a , b , y ) -(1 -α) p X×Ω×Y×Ω d(a, b) pq dρ(x, a, b, y) + p-1 k=0 p k (1 -α) k α p-k (X×Ω×Y×Ω) 2 d(a, b) qk L(x, y, x , y ) q(p-k) dρ(x, a, b, y)dρ(x , a , b , y ) = p-1 k=0 p k (1 -α) k α p-k (X×Ω×Y×Ω) 2 d(a, b) qk L(x, y, x , y ) q(p-k) dρ(x
We note

H k = (X ×Ω×Y×Ω) 2
d(a, b) qk L(x, y, x , y ) q(p-k) dρ(x, a, b, y)dρ(x , a , b , y ).

Using (6.4) we have shown that:

(

1 -α)(W pq (µ A , ν B )) q ≤ F GW α,p,q (µ, ν) ≤ (1 -α) p (W pq (µ A , ν B )) pq + p-1 k=0 p k (1 -α) k α p-k H k 1 p So lim α→0 F GW α,p,q (µ, ν) = (W pq (µ A , ν B )) q .
For the case α → 1 we rather consider π GW ∈ Π(µ X , ν Y ) an optimal coupling for the pq-Gromov-Wasserstein distance between µ X and ν Y and we construct

γ ∈ P( µ Ω × π GW X × ν Y × Ω)
such that γ ∈ Π(µ, ν) and P 2,3 #ρ = π GW . In the same way as previous reasoning we can derive:

α(GW pq (µ X , ν Y )) q ≤ F GW α,p,q (µ, ν) ≤ α p (GW pq (µ X , ν Y )) pq + p-1 k=0 p k (1 -α) p-k α k J k )
1 p (6.24) with J k = ´d(a, b) q(p-k) L(x, y, x , y ) qk dρ (x, a, b, y)dρ(x , a , b , y ). In this way lim α→1 F GW α,p,q (µ, ν) = (GW pq (µ X , ν Y )) q .

6.1.6 Proof of Theorem 3.5.3 -Constant speed geodesic.

We recall the theorem:

Theorem (Constant speed geodesic.). Let p ≥ 1 and (X × Ω, d X , µ 0 ) and (Y × Ω, d Y , µ 1 ) in S p (R d ). Let π * be an optimal coupling for the Fused Gromov-Wasserstein distance between µ 0 , µ 1 and t ∈ [0, 1]. We equip R d with m norm for m ≥ 1.

We define η t :

X × Ω × Y × Ω → X × Y × Ω such that: ∀(x, a), (y, b) ∈ X × Ω × Y × Ω, η t (x, a, y, b) = (x, y, (1 -t)a + tb) Then: (X × Y × Ω, (1 -t)d X ⊕ td Y , µ t = η t #π * ) t∈[0,1] is a constant speed geodesic connecting (X × Ω, d X , µ 0 ) and (Y × Ω, d Y , µ 1 ) in the metric space S p (R d ), F GW α,p,1 .
Proof. We note

S t = (X × Y × Ω, d t , µ t = η t #π * ) t∈[0,1] where d t = (1 -t)d X ⊕ td Y .
Let . be any m norm for m ≥ 1. It suffices to prove:

F GW α,p,1 (µ t , µ s ) ≤ |t -s|F GW α,p,1 (µ 0 , µ 1 ) (6.25)
To do so we consider ∆

t s ∈ P(X × Y × Ω × X × Y × Ω) defined by ∆ t s = (η t × η s )#π * ∈ Π(µ t ,
= (X×Ω×Y×Ω) 2 (1 -α) (1 -t)a + tb-(1-s)a-sb + α|(1 -t)dX (x, x )+tdY (y, y ) -(1 -s)dX (x, x ) + sdY (y, y )| p dπ * (x, a, y, b)dπ * (x , a , y , b ) = |t -s| p (X×Ω×Y×Ω) 2 (1 -α) a -b + α|dX (x, x ) -dY (y, y )| p dπ * (x, a, y, b)dπ * (x , a , y , b )
So F GW α,p,1 (µ t , µ s ) ≤ |t -s|F GW α,p,1 (d 0 , d 1 , µ 0 , µ 1 ).

Proofs and additional results of Chapter 4

This Chapter contains all the proofs of the Chapter 4. We recall the following notations:

J 2 (c X , c Y , π) def = ˆX ×X ˆY×Y |c X (x, x ) -c Y (y, y )| 2 dπ(x, y)dπ(x , y ) J(T ) def = ˆ x -x 2 2 -T (x) -T (x ) 2 2 2 dµ(x)dµ(x )
LGM 2 2 (µ, ν)

def = inf T #µ=ν T is linear J(T ) = inf T #µ=ν T is linear ˆ x -x 2 2 -T (x) -T (x ) 2 2 2 dµ(x)dµ(x )
6.2.1 Proof of Theorem 4.1.1 -New special of the QAP

We recall the theorem:

Theorem (A new special case for the Quadratic Assignment Problem). For real numbers

x 1 < • • • < x n and y 1 < • • • < y n , min σ∈Sn i,j -(x i -x j ) 2 (y σ(i) -y σ(j) ) 2 (6.27)
is achieved either by the identity permutation σ(i) = i (Id) or the anti-identity permutation σ(i) = n + 1 -i (anti -Id). In other words:

∃σ ∈ {Id, anti -Id}, σ ∈ arg min σ∈Sn i,j -(x i -x j ) 2 (y σ(i) -y σ(j) ) 2 (6.28) Let us note I = {x, y ∈ R n × R n |x 1 < x 2 < • • • < x n , y 1 < y 2 < • • • < y n }.
We consider for x, y ∈ I:

max σ∈Sn Z(x, y, σ) = max σ∈Sn i,j (x i -x j ) 2 (y σ(i) -y σ(j) ) 2 (6.29)
The original problem is equivalent to maximizing Z(x, y, σ) over S n . Given x, y ∈ I, we define

X def = i x i and Y def = i y i . Then: max σ∈Sn Z(x, y, σ) = max σ∈Sn i,j (x i -x j ) 2 (y σ(i) -y σ(j) ) 2 = max σ∈Sn i,j (x 2 i + x 2 j )(y 2 σ(i) + y 2 σ(j) ) -2 i,j x i x j (y 2 σ(i) + y 2 σ(j) ) -2 i,j y σ(i) y σ(j) (x 2 i + x 2 j )
+ 4

i,j

x i x j y σ(i) y σ(j)

= max

σ∈Sn 2n i x 2 i y 2 σ(i) -2 i,j x i x j (y 2 σ(i) + y 2 σ(j) ) -2 i,j y σ(i) y σ(j) (x 2 i + x 2 j ) + 4 i,j x i x j y σ(i) y σ(j) + 2( i x 2 i )( i y 2 i ) = max σ∈Sn 2n i x 2 i y 2 σ(i) -4X i x i y 2 σ(i) -4Y i x 2 i y σ(i) + 4 i,j x i x j y σ(i) y σ(j) + 2( i x 2 i )( i y 2 i ) ( * ) = Cte + 2 max σ∈Sn i nx 2 i y 2 σ(i) -2 i (Xx i y 2 σ(i) + Y x 2 i y σ(i) ) + 2( i x i y σ(i) ) 2
where in (*) we defined Cte def = 2( i x 2 i )( i y 2 i ) the term that does not depend on σ. Overall we have:

∀x, y ∈ I, argmax σ∈Sn Z(x, y, σ) = argmax σ∈Sn i nx 2 i y 2 σ(i) -2 i (Xx i y 2 σ(i) +Y x 2 i y σ(i) )+2( i x i y σ(i) ) 2 (6.30)
Since Z is invariant by translation of x, y we can suppose without loss of generality that X = Y = 0. We consider the set

D = {x, y ∈ R n × R n |x 1 < x 2 < • • • < x n , y 1 < y 2 < • • • < y n ,
i x i = j y j = 0}. We want to find for x, y ∈ D:

max σ∈Sn n i x 2 i y 2 σ(i) + 2 i x i y σ(i) 2 def = max σ∈Sn g(x, y, σ) (QAP)
We have the following result: Lemma 6.2.1. Let x, y ∈ D and consider the problem:

max π∈DS ijkl (x 2 i y 2 j + 2x i y j x k y l )π ij π kl (QP)
where DS is the set of doubly stochastic matrices. Then (QP) and (QAP) are equivalent. More precisely if σ * is an optimal solution of (QAP) then π σ * defined by π σ * (i, j) = 1 if j = σ * (i) else 0 for all

(i, j) ∈ [[n]]
2 is an optimal solution of (QP) and if π * is an optimal solution of (QP) then it is supported on a permutation σ * which is an optimal solution of (QAP).

Proof. The problem (QAP) can be rewritten as:

max P ij ∈{0,1} ∀j i P ij =1 ∀i j P ij =1 n ij x 2 i y 2 j Pij + 2 i,j xiyjPij 2 = max P ij ∈{0,1} ∀j i P ij =1 ∀i j P ij =1 n ij x 2 i y 2 j Pij + 2 ijkl xix k yjy l PijP kl * = max P ij ∈{0,1} ∀j i P ij =1 ∀i j P ij =1 ijkl x 2 i y 2 j PijP kl + 2 ijkl xiyjx k y l PijP kl = max P ij ∈{0,1} ∀j i P ij =1 ∀i j P ij =1 ijkl (x 2 i y 2 j + 2xiyjx k y l )PijP kl (6.31)
In (*) we used k,l P k,l = n. We consider the following relaxation of (6.31) as:

max π∈DS ijkl (x 2 i y 2 j + 2x i y j x k y l )π ij π kl (6.32)
which is a maximization of a convex function. More precislely it is quadratic programming problem which Hessian is positive semi-definite xx T ⊗ K yy T . Since the problem is a maximization of a convex function an optimal solution π * of (QP) lies necassarily in the extremal points of DS [Rockafellar 1970] such that both (QP) and (QAP) are equivalent: if π * is an optimal solution it is necessarily supported on a σ * ∈ S n such that σ * is an optimal solution of (QAP) and if σ * ∈ S n is an optimal solution of (QAP) then π * defined by π

* ij = 1 if j = σ * (i) else 0 for all (i, j) ∈ [[n]]
2 is an optimal solution of (QP). Lemma 6.2.2. Let x, y ∈ D. For σ ∈ S n we note C(x, y, σ) = i x i y σ(i) . Let π * an optimal solution of (QP) with σ * the permutation associated to π * . If C(x, y, σ * ) > 0 then π * = I n is the identiy and if C(x, y, σ * ) < 0 then π * = J n is the anti-identity.

To prove this result we will rely on the following theorem which gives necessary conditions for being an optimal solution of (QP):

Theorem (Theorem 1.12 in [Murty 1988]). Consider the following (QP):

min x f (x) = cx + x T Qx s.t. Ax = b, x ≥ 0 (6.33)
Then if x * is an optimal solution of (6.33) it is an optimal solution of the following (LP):

min x f (x) = (c + x T * Q)x s.t. Ax = b, x ≥ 0 (6.34)
Proof. Of Lemma 6.2.2. Applying Theorem 6.2.1 in our case gives that if π * is a solution of (QP) it necessarily a solution of the following (LP):

max π∈DS ijkl (x 2 i y 2 j + 2x i y j x k y l )π * ij π kl = n ij x 2 i y 2 j π * ij + max π∈DS 2( ij x i y j π * ij )( kl x k y l π kl ) (6.35)
Since π * is supported on a permutation σ * this gives:

n i x 2 i y 2 σ * (i) + max π∈DS C(x, y, σ * ) kl x k y l π kl (LP)
where C(x, y, σ * ) = 2 i x i y σ * (i) .

• If C(x, y, σ * ) > 0 then this (LP) has a unique solution which is the identity π * = I n . This is a consequence of the Rearrangement Inequality (see Memo 6.2.1) which states that for all permutations i x i y σ(i) < i x i y i (since x i and y j are distinct). Using the fact that an optimal solution of (LP) is supported on a permutation concludes.

• If C(x, y, σ * ) < 0 then the anti-identity is the unique optimum with the same reasoning since i x i y n+1-i < i x i y σ(i) for all permutation because of Rearrangement Inequality.

Using both results we can prove the following proposition which is the main ingredient to prove Theorem 4.1.1: Proposition 6.2.1. Let x, y ∈ D and σ * a solution of (QAP) i.e. σ * ∈ arg max σ∈Sn g(x, y, σ). For σ ∈ S n we note C(x, y, σ) = i x i y σ(i) .

If C(x, y, σ * ) > 0 then σ * is the identiy permutation σ * (i) = i and if C(x, y, σ * ) < 0 then σ * is the anti-identity permutation σ *

(i) = n + 1 -i for all i ∈ [[n]].
Proof. Let σ * be an optimal solution of (QAP) and π * defined by π * ij = 1 if j = σ * (i) else 0. By Lemma 6.2.1 we know that π * is an optimal solution of (QP). Consider the case C(x, y, σ * ) > 0. Suppose that σ * is not the identity, then π * = I n which is not possible by Lemma 6.2.2 since π * is an optimal solution of (QP). Same applies for C(x, y, σ * ) < 0 and the anti-identity.

To state that (QAP) admits the identity or the anti-identity as optimal permutations we will rely on the previous proposition and on the continuity of the loss g: Lemma 6.2.3 (Continuity of g). Let x, y ∈ D fixed. There exists ε x,y > 0 such that for all h < ε x,y we have:

arg max σ∈Sn g(x + h, y, σ) ⊂ arg max σ∈Sn g(x, y, σ) (6.37) Memo 6.2.1 (Rearrangement Inequality). Let x 1 ≤ • • • ≤ x n , y 1 ≤ • • • ≤ y n then we have: ∀σ ∈ S n , i x i y n+1-i ≤ i x i y σ(i) ≤ i x i y i (6.36)
If the numbers are different then the lower bound (resp upper bound) is attained only for the permutation which reverses the order (resp for the identiy permutation)

Proof. Let x, y ∈ D, σ * ∈ arg max σ∈Sn g(x, y, σ) and τ any permutation in S n such that τ / ∈ arg max σ∈Sn g(x, y, σ). Then we have g(x, y, σ * ) > g(x, y, τ ). Let η = g(x, y, σ * ) -g(x, y, τ ) > 0. For all permutation β we have that g(., y, β) is continuous. In this way: (β, σ, τ ). By (6.38) applied to σ * and τ : Using the previous lemma we can now prove the following result:

∀β ∈ S n , ∃ε x,y (β, σ * , τ ) > 0, ∀ h < ε x,y (β, σ * , τ ), |g(x + h, y, β) -g(x, y, β)| < η 4 (6.38) Let h ∈ R n such that h < min (β,σ,τ )∈(Sn) 3 ε x,y
g(x + h, y, σ * ) -g(x + h, y, τ ) = g(x + h, y, σ * ) -g(x, y, σ * ) + g(x, y, σ * ) -g(x, y, τ ) + g(x, y, τ ) -g(x + h, y, τ ) > - η 4 + η - η 4 = η 2 > 0 ( 6
(x i + ε 0 (i)) = i x i + ζ -ζ = 0 and x 1 + ε 0 (1) < • • • < x N + ε 0 (N ) since x 1 + ζ < x 2 -ζ.
Let σ * ε0 ∈ arg max σ∈Sn g(x + ε 0 , y, σ). By Lemma 6.2.3 we have σ * ε0 ∈ arg max σ∈Sn g(x, y, σ).

Moreover we have

C(x + ε 0 , y, σ * ε0 ) = i x i y σ * ε 0 (i) + ζ(y σ * ε 0 (0) -y σ * ε 0 (1) ). • If i x i y σ * ε 0 (i) = 0 then C(x + ε 0 , y, σ * ε0 ) = ζ(y σ * ε 0 (0) -y σ * ε 0 ( 1
) ) = 0 since all y i are distinct. We can apply Proposition 6.2.1 with x + ε 0 , y ∈ D to conclude that σ * ε0 is wether the identity or the anti-identity.

• If i x i y σ * ε 0 (i) = 0 then σ * ε0 ∈ arg max σ∈Sn g(x, y, σ) and C(x, y, σ * ε0 ) = 0 so by Proposition 6.2.1 with x, y ∈ D we can conclude that σ * ε0 is wether the identity or the anti-identity.

Corollary 6.2.1 (Theorem 4.1.1 is valid). Let x, y ∈ D. The identity or the anti-identity is an optimal solution of (QAP)

Proof. Let x, y ∈ D. We consider ε 0 defined in Lemma 6.2.4 and σ * ε0 ∈ arg max σ∈Sn g(x + ε 0 , y, σ). Then by Lemma 6.2.4 σ * ε0 is wether the identity or the anti-identity. Moreover by Lemma 6.2.4 σ * ε0 ∈ arg max σ∈Sn g(x, y, σ) so it is an optimal solution of (QAP). This concludes that the identity or the anti-identity is an optimal solution of (QAP) which proves Theorem 4.1.1.

Proof of Theorem 4.1.2 -Equivalence between GM and GW in the discrete case

This paragraph aims at proving the equivalence between GM and GW . We recall the theorem:

Theorem (Equivalence between GW and GM for discrete measures). Let µ ∈ P(R p ), ν ∈ P(R q ) be discrete probability measures with same number of atoms and uniform weights, i.e. µ =

1 n n i=1 δ xi , ν = 1 n n i=1 δ yi with x i ∈ R p , y i ∈ R q . For x ∈ R p we note x 2,p = p i=1 |x i | 2 the 2 norm on R p (same for R q ). Let c X (x, x ) = x -x 2 2,p , c Y (y, y ) = y -y 2 2,q . Then: GW 2 (c X , c Y , µ, ν) = GM 2 (c X , c Y , µ, ν) (6.41) Moreover if p = q = 1, i.e. c X (x, x ) = c Y (x, x ) = |x -x | 2 for x, x ∈ R, and if x 1 < • • • < x n and y 1 < • • • < y n the
optimal values are achieved by considering either the identity or the anti-identity permutation.

Proof. The proof is essentially based on theoretical results from [START_REF] Maron | [END_REF]] and on Theorem 4.1.1. In [START_REF] Maron | [END_REF]] authors consider the minimizing energy problem min X∈Πn -tr(BX T AX) where Π n the set of permutation matrices. In fact, the GM problem defined in this chapter is equivalent to min X∈Πn -tr(BX T AX) by considering A = ( x i -x j 2 2,p ) i,j and B = ( y i -y j 2 2,q ) i,j . To tackle this problem authors propose to minimize -tr(BX T AX) over the set of doubly stochastic matrices (which is the convex-hull of Π n ):

DS = {X ∈ R n×n s.t. X1 n = X T 1 n = 1 n , X ≥ 0}
Minimizing -tr(BX T AX) over DS is equivalent to solving the GW distance when a i = b j = 1 n . The paper proves that when both A and B are conditionally positive (or negative) definite of order 1 then the relaxation leads to the same optimum so that the minimum over DS is the same as the minimum over Π n [Maron 2018, Theorem 1]. Yet A and B defined previously satisfy this property (see examples under Definition 2 in [START_REF] Maron | [END_REF]) and so GW and GM coincide.

Moreover when p = q = 1 and when the sample are sorted we can apply Theorem 4.1.1 to prove that an optimal permutation of the GM problem is found whether at the identity or the anti-identity permutation which concludes the proof.

Computing GW in the 1d case

We recall the result:

Lemma. The GM 2 and GW 2 costs in 1D with same numbers of atoms and uniform weights can be computed in O(n).

Proof. As seen in Theorem 4.1.2 finding the optimal permutation σ * can be found in O(n log(n)). Moreover the final costs can be written using binomial expansion: 6.42) which can be computed in O(n) operations.

i,j (x i -x j ) 2 -(y σ * (i) -y σ * (j) ) 2 2 = 2n i x 4 i -8 i x 3 i k x k + 6( i x 2 i ) 2 + 2n i y 4 i -8 i y 3 i k y k + 6( i y 2 i ) 2 -4( i x i ) 2 ( k y k ) 2 -4n i x 2 i y 2 σ * (i) + 8 i (( k x k )x i y 2 σ * (i) + ( k y k )x 2 i y σ * (i) ) -8( i x i y σ * (i) ) 2 (

Proof of Theorem 4.1.3 -Properties of SGW

We recall the theorem:

Theorem (Properties of SGW ).

• For all ∆, SGW ∆ and RISGW are translation invariant.

RISGW is also rotational invariant when p = q, more precisely if

Q ∈ O(p) is an orthogonal matrix, RISGW (Q#µ, ν) = RISGW (µ, ν) (same for any Q ∈ O(q) applied on ν).
• SGW and RISGW are pseudo-distances on P(R p ), i.e. they are symmetric, satisfy the triangle inequality and SGW (µ, µ) = RISGW (µ, µ) = 0 .

• Let µ, ν ∈ P(R p ) × P(R p ) be probability distributions with compact supports. If SGW (µ, ν) = 0 then µ and ν are isomorphic for the distance induced by the 1 norm on R p , i.e. d(x, x ) = p i=1 |x i -x i | for (x, x ) ∈ R p × R p . In particular this implies:

SGW (µ, ν) = 0 =⇒ GW 2 (d, d, µ, ν) = 0 (6.43)
The invariance by translation is clear since the costs are invariant by translation of the support of the measures. The pseudo-distances properties are straightforward thanks to the properties of GW . For the invariance by rotation if p = q then V p (R p ) is bijective with O(p) so for Q ∈ O(p):

RISGW (Q#µ, ν) = min ∆∈Vp(R p ) SGW ∆ (Q#µ, ν) = min ∆∈O(p) SGW ∆ (Q#µ, ν) = min ∆∈O(p) E θ∼λq-1 [GW (d 2 , P θ #(∆Q#µ), P θ #ν)] = min ∆ ∈O(p) E θ∼λq-1 [GW (d 2 , P θ #∆ #µ, P θ #ν)] = RISGW (µ, ν) (6.44)
On the other side for ν a change of formula on theta gives the result. For the case SGW = 0 =⇒ GW = 0 it will be a consequence of the following theorem: Theorem 6.2.1. Let µ, ν ∈ P(R p ) × P(R p ) be probability distributions such that µ, ν have compact supports. If for almost all θ ∈ S p-1 , P θ #µ and P θ #ν are isomoprhic then µ and ν are isomoprhic. In other words if for almost all θ ∈ S p-1 we have:

∃T θ : supp(P θ #µ) ⊂ R → supp(P θ #ν) ⊂ R, surjective s.t. T θ #(P θ #µ) = P θ #ν ∀x, x ∈ supp(P θ #µ), |T θ (x) -T θ (x )| = |x -x | (6.45)
Then there exists a measure preserving isometry f between supp(µ) and supp(ν). More precisely we have f #µ = ν and: 6.46) To prove this theorem we will exhibit the isometry. This result can be put in light of Cramer-Wold theorem [START_REF] Cramér | Some Theorems on Distribution Functions[END_REF]] which states that a probability measure is uniquely determined by the totality of its one-dimensional projections. Equivalently, if we consider two probability measures so that the one-dimensional measures resulting from the projections over all the hypersphere are equal then the measures are equal. The equality relation is replaced in our theorem by the isomoprhism relation.

∀x, x ∈ supp(µ), f (x) -f (x ) 1 = x -x 1 (
The proof is divided into four parts. In the first one, we construct an "almost orthogonal" basis on which measures are isomorphic. Building upon this result we define a sequence of functions from supp(µ) to supp(ν) and show that it has a convergent subsequence. We conclude by proving that the limit of the subsequence is actually a good candidate for being the isometry we are looking for. In the following . 1 denotes the 1 norm, . 2 denotes the 2 norm and p ≥ 2. We recall that F µ is the Fourier transform of µ.

We consider the following Q θ property for θ ∈ S p-1 :

∃T θ : supp(P θ #µ) ⊂ R → supp(P θ #ν) ⊂ R, surjective s.t. T θ #(P θ #µ) = P θ #ν ∀x, x ∈ supp(P θ #µ), |T θ (x) -T θ (x )| = |x -x | (Q θ )
Informally if we have the Q θ property for θ ∈ S p-1 it implies that µ and ν are isomorphic on the 1D line given by the projection w.r.t. θ. We have the following lemma: Lemma 6.2.5. Let µ, ν ∈ P(R p ) × P(R p ) and suppose that Q θ holds for almost all θ ∈ S p-1 . Let n > p -1. There exists a basis (e 1 (n), ..., e p (n)) of R p part of the following spaces:

B n p def = {(θ 1 , ..., θ p ) ∈ (S p-1 ) p s.t. | θ i , θ j | < 1 n } (6.47) and Q def = {(θ 1 , ..., θ p ) ∈ (S p-1 ) p s.t. ∀i ∈ {1, ..., p}, Q θi } (6.48)
Proof. We want to construct a basis (e 1 , ..., e p ) as orthogonal as possible such that for all i we have Q ei . We note λ ⊗p p-1 the product measure λ p-1 ⊗ ... ⊗ λ p-1 where λ p-1 is the uniform measure on the sphere.

B n

p is an open set as inverse image by a continuous function of an open set. Then λ ⊗p p-1 (B n p ) > 0. Moreover, since for almost all θ ∈ S p-1 we have Q θ then λ ⊗p p-1 (Q) > 0 and so λ ⊗p p-1 (B n p ∩ Q) > 0. In this way we can consider (e 1 (n), ..., e p (n)) ∈ B n p ∩Q. If n > p-1 the Gram matrix of (e 1 (n), ..., e p (n)) is strictly diagonal dominant, thus invertible, such that (e 1 (n), ..., e p (n)) is a basis. Note that we can not consider directly an orthogonal basis since the set of all orthogonal basis has measure zero.

We now express all the vectors and inner products in this new almost orthogonal basis as expressed in the following lemma: Lemma 6.2.6. Let n > p -1 and a basis (e 1 (n), ..., e p (n)) as defined in Lemma 6.2.5. Then all x ∈ R p can be written as:

x = p i=1 [ x, e i (n) + R(x, e i (n))]e i (n) (6.49) where |R(x, e i (n))| = o( 1 n ). Moreover for all (x, y) ∈ R p × R p : x, y = p i=1
x, e i (n) y, e i (n) + R(x, y) (6.50) where

| R(x, y)| = o( 1 n ).
Proof. In the following x i denotes the i-th coordinate of a vector x in the standard basis, i.e. a vector writes x = (x 1 , . . . , x p ). For x ∈ R p , we can write in the new basis

x = p i=1 [ x, e i (n) + R(x, e i (n))]e i with R(x, e i (n)) def = x i -x, e i (n) . We have also |R(x, e i (n))| = o( 1 n ). Indeed, x = p i=1
x i e i =⇒ ∀j, x, e j = p i=1

x i e i , e j =⇒ x j -x, e j = i =j

x i e i , e j =⇒ |R(x, e j (n))| = | i =j x i e i , e j | =⇒ |R(x, e j (n))| ≤ 1 n i =j |x i |
Also in the same way for x, y ∈ R p × R p we can rewrite their inner product:

x, y = p i=1

x, e i (n) y, e i (n) + R(x, y) ( x, e i (n) y, e i (n)

= i =j
x, e i (n) y, e i (n) e j (n), e i (n) + i,j

x, e i (n) R(y, e j (n)) e j (n), e i (n)

+ i,j y, e j (n) R(x, e i (n)) e j (n), e i (n) + i,j R(x, e j (n))R(y, e i (n)) e j (n), e i (n)
and with the same calculus than for R we have

| R(x, y)| = o( 1 n ).
Proposition 6.2.2. Let µ, ν ∈ P(R p ) × P(R p ) and suppose that Q θ holds for almost all θ ∈ S p-1 and that ν has compact support. There exists a sequence (f n ) n∈N from supp(µ) to supp(ν) uniformly bounded which satisfies:

∀n ∈ N, ∀x, x ∈ supp(µ) 2 , f n (x) -f n (x ) 1 -x -x 1 = o( 1 n ) (6.52) ∀n ∈ N, ∀s ∈ R p , |F fn#µ (s) -F ν (s)| = o( 1 n ) (6.53)
Proof. In the following x i denotes the i-th coordinate of a vector x in the standard basis, i.e. a vector writes x = (x 1 , . . . , x p ). We define:

∀n > p -1, ∀x ∈ supp(µ), f n (x) = (T e1(n) ( x, e 1 (n) ), ..., T ep(n) ( x, e p (n) )) (6.54)
where (e k (n)) k∈ [[p]] is the almost orthogonal basis define in Lemma 6.2.5, and T e k (n) is defined from (Q θ ) since we have Q e k (n) for all k. It is clear from the definition that f n (x) ∈ supp(ν). Moreover for

x, x ∈ supp(µ):

f n (x) -f n (x ) 1 = p k=1 |T e k (n) ( x, e k (n) ) -T e k (n) ( x , e k (n) )| ( * ) = p k=1 | x, e k (n) -x , e k (n) | = p k=1 | x -x , e k (n) |
where in (*) we used that T e k (n) is an isometry since we have Q e k (n) and x, e k (n) ∈ supp(P e k (n) #µ) (idem for x ). In this way:

f n (x) -f n (x ) 1 -x -x 1 = p k=1 | x -x , e k (n) | -|x k -x k | ≤ p k=1 | x -x , e k (n) | -|x k -x k | * ≤ p k=1 | x -x , e k (n) -(x k -x k )| = p k=1 |R(x -x , e k (n))| = o( 1 n )
where in (*) the second triangular inequality ||x| -|y|| ≤ |x -y|. Hence: This implies ´e-2iπt.T e k (n) ( e k (n),x ) dµ(x) = ´e-2iπt e k (n),y dν(y). So by applying this results for t = s k we have: ˆe-2iπs k T e k (n) ( x,e k (n) ) dµ(x) = ˆe-2iπs k e k (n),y dν(y) (6.56)

f n (x) -f n (x ) 1 -x -x 1 = o( 1 n ) ( 6
Combining both results:

F fn#µ (s) = ˆe-2iπ p k=1
s k e k (n),y dν(y) (6.57)

We can now bound |F fn#µ (s) -F ν (s)| as:

|F fn#µ (s) -F ν (s)| = |F fn#µ (s) -ˆe-2iπ s,y dν(y)| * = |F fn#µ (s) -ˆe-2iπ[ p k=1 s,e k (n) e k (n),y + R(s,y)] dν(y)| * * = | ˆe-2iπ p k=1 s k e k (n),y dν(y) -ˆe-2iπ R(s,y) e -2iπ p k=1 s,e k (n) e k (n),y dν(y)|
where in (*) we used the expression in the new base of the inner product s, y seen in Lemma 6.2.6, in (**) we used (6.57). By injecting the expression of s k w.r.t. the new base we have:

|F fn#µ (s) -F ν (s)| ≤ | ˆe-2iπ p k=1 ( s,e k (n) +R(s,e k (n))) e k (n),y dν(y) -ˆe-2iπ R(s,y) e -2iπ p k=1 s,e k (n) e k (n),y dν(y)| = ˆe-2iπ p k=1 s,e k (n) e k (n),y (e -2iπ p k=1 R(s,e k (n)) e k (n),y -e -2iπ R(s,y) )dν(y) ≤ ˆ|e -2iπ p k=1 R(s,e k (n)) e k (n),y -e -2iπ R(s,y) |dν(y) = ˆ|e -2iπ R(s,y) (e -2iπ( p k=1 R(s,e k (n)) e k (n),y -R(s,y)) -1)|dν(y) ≤ ˆ|e -2iπ( p k=1 R(s,e k (n)) e k (n),y -R(s,y)) -1|dν(y) = ˆ|2ie -iπ( p k=1 R(s,e k (n)) e k (n),y -R(s,y)) sin(π( p k=1 R(s, e k (n)) e k (n), y -R(s, y))|dν(y) ≤ ˆ| sin(π( p k=1 R(s, e k (n)) e k (n), y -R(s, y))|dν(y) ≤ π ˆ( p k=1 |R(s, e k (n)) e k (n), y | + | R(s, y)|)dν(y) * = o( 1 n ) (6.58) in (*) the fact that each term is o( 1 n ).
In this way:

|F fn#µ (s) -F ν (s)| = o( 1 n ) (6.59)
Moreover (f n ) n>p-1 is also uniformly bounded. To see that we consider x ∈ supp(µ). We have that for all k ∈ [[p]] T e k (n) ( x, e k (n) ) ∈ supp(P e k (n) #ν) by definition of T e k (n) . So there exists a

y 0 (x, n, k) ∈ supp(ν) such that T e k (n) ( x, e k (n) ) = y 0 (x, n, k), e k (n) . In this way |T e k (n) ( x, e k (n) )| = | y 0 (x, n, k), e k (n) | ≤ y 0 (x, n, k) 2 e k (n) 2 by Cauchy-Swartz. Moreover e k (n) 2 < 1 n ≤ 1 p-1 ≤ 1 and since ν has compact support then there is a constant M ν we have y 0 (x, n, k) 2 ≤ M ν So we have for n ∈ N, x ∈ supp(µ), f n (x) 2 2 = p k=1 |T e k (n) ( x, e k (n) )| 2 ≤ pM ν
Since on R p all norms are equivalent this suffices to state the existence of a constant C such that

∀x ∈ R p , n ∈ N, f n (x) 1 ≤ C so that (f n ) n∈N is uniformly bounded. Reindexing (f n ) n>p-1
gives the desired result.

We can now prove Theorem 6.2.1.

Proof of Theorem 6.2.1. We consider the sequence (f n ) n∈N defined in Proposition 6.2.2. We will show that (f n ) n∈N is equicontinuous. Let ε > 0, using (6.52) there exists a N ∈ N such that we have for all

x, x ∈ supp(µ):

f n (x) -f n (x ) 1 ≤ ε + x -x 1 for all n ≥ N (6.60) Now let δ < ε. Suppose that x -x 1 < δ then f n (x) -f n (x ) 1 < ε + δ < 2ε for all n ≥ N (6.61)
Without loss of generality we can reindex (f n ) n∈N for n large enough (n ≥ N ) so that (f n ) n∈N is equicontinuous with the previous argument.

Since (f n ) n∈N is a uniformly bounded and equicontinuous sequence from the support of µ which is compact to R p we can apply Arzela-Ascoli theorem (see Memo 6.2.2) which states that (f n ) n∈N has a uniformly convergent subsequence. We denote by (f φ(n) ) n this sequence. We have

f φ(n) u → n→∞ f this sequence.
Moreover equation (6.53) states that for all s ∈ R p , F fn#µ (s) → n→∞ F ν (s). In this way (F fn#µ (s)) n∈N is a convergent real valued sequence, so every adherence value goes to the same limit, hence

F f φ(n) #µ (s) → n→∞ F ν (s).
Moreover the function f is a measure preserving isometry from supp(µ) to supp(ν). Indeed let ε 1 > 0, s ∈ R p , there exists from previous statements N

0 , N 1 ∈ N such that for n ≥ N 0 , |F f φ(n) #µ (s)-F ν (s)| < ε 1 and n ≥ N 1 , |F f φ(n) #µ (s) -F f #µ (s)| < ε 1 . Let n ≥ max(N 0 , N 1 ) |F f #µ (s) -F ν (s)| ≤ |F f φ(n) #µ (s) -F ν (s)| + |F f φ(n) #µ (s) -F f #µ (s)| < 2ε 1
As this result holds for any ε 1 > 0 we have F f #µ (s) = F ν (s) and by injectivity of the Fourrier transform f #µ = ν such that f is measure preserving.

In the same way for any x, x ∈ supp(µ), ε 2 > 0 and n large enough:

f (x) -f (x ) 1 -x -x 1 ≤ f φ(n) (x) -f φ(n) (x ) 1 -f (x) -f (x ) 1 + f φ(n) (x) -f φ(n) (x ) 1 -x -x 1 < 2ε 2 using f φ(n) u → n→∞
f and (6.52). As this result holds true for any ε 2 > 0 we have f (x) -f (x ) = xx for any x, x ∈ supp(µ) which concludes.

• A family F ⊂ C(X , R p ) is bounded means if there exists a positive constant M < ∞ such that f (x) ≤ M for all x ∈ X and f ∈ F • A family F ⊂ C(X, R p
) is equicontinuous means if for every ε > 0 there exists δ > 0 (which depends only on ε) such that for x, y ∈ X :

d(x, y) < δ ⇒ f (x) -f (y) < ε ∀f ∈ F (6.62) If (f n ) n∈N is a sequence in C(X, R p
) that is bounded and equicontinuous then the Arzela-Ascoli states that it has a uniformly convergent subsequence (see Theorem 7.25 [Rudin 1976]) . (x, x ) ∈ R p × R p . In particular this implies:

SGW (µ, ν) = 0 =⇒ GW 2 (d, d, µ, ν) = 0 (6.63)
Proof. If SGW (µ, ν) = 0 then using the Gromov-Wasserstein properties it implies that for almost all θ ∈ S p-1 the projected measures are isomorphic. Moreover since µ, ν have compact support, it is bounded and we can directly apply Theorem 6.2.1 to state the existence of a measure preserving application f as defined in Theorem 6.2.1. We consider the coupling π = (id × f )#µ ∈ Π(µ, ν) since f #µ = ν. Then we have:

ˆˆ|d(x, x ) -d(y, y )| 2 dπ(x, y)dπ(x , y ) = ˆˆ|d(x, x ) -d(f (x), f (x ))| 2 dµ(x)dµ(x ) = ˆˆ| x -x 1 -f (x) -f (x ) 1 | 2 dµ(x)dµ(x ) = 0
Since f is an isometry. This directly implies that GW 2 (d, d, µ, ν) = 0.

Additional results -SW ∆ and RISW

Analogously to SGW we can define for the Sliced-Wasserstein distance SW ∆ (µ, ν) for µ, ν ∈ P(R p )×P(R q ) with p = q and its rotational invariant counterpart as:

SW ∆ (µ, ν) = ˆSq-1 SW (P θ #µ ∆ , P θ #ν)dλ q-1 (θ) RISW (µ, ν) = min ∆∈Vq(R p ) SW ∆ (µ, ν) (6.64)
where SW is the Sliced-Wasserstein distance. The complexity for computing SW ∆ between two discrete probability measures with n atoms and uniform weights is O(Ln(p + q + log(n))) which is exactly the same complexity as SGW ∆ . With these formulations, we can perform the same experiment as for RISGW on the spiral dataset. The optimisation on the Stiefel manifold is performed using Pymanopt as for SGW . Results are reported in Figure 6.1. As one can see, RISW is rotational invariant on average whereas SW is not. One can also note that, due to the sampling process of the spiral dataset, the variance is quite large. This can be explained by the fact that, unlike SGW , the Sliced-Wasserstein may realign the distributions without taking the rotation into account.

Proof of Lemma 4.2.1 -Reductions of the GW costs for inner products and squared Euclidean distance matrices

We recall the lemma:

Lemma. Suppose that there exist scalars a, b, c such that c X (x, x ) = a x 2 2 + b x 2 2 + c x, x p and c Y (y, y ) = a y 2 2 + b y 2 2 + c y, y q . Then:

J 2 (c X , c Y , π) = C µ,ν -2Z(π) (6.65) where C µ,ν = ´c2 X dµdµ + ´c2 Y dνdν -4ab
´ x 2 2 y 2 2 dµ(x)dν(y) and:

Z(π) = (a 2 + b 2 ) ˆ x 2 2 y 2 2 dπ(x, y) + c 2 ˆyx T dπ(x, y) 2 F + (a + b)c ˆ x 2 2 E Y ∼ν [Y ], y q + y 2 2 E X∼µ [X], x p dπ(x, y) (6.66) Proof. Let π ∈ Π(µ, ν). We have J 2 (c X , c Y , π) = ´c2 X dµdµ + ´c2 Y dνdν -2 ´cX c Y dπdπ. In this way: ˆcX c Y dπdπ = ˆ(a x 2 2 + b x 2 2 + c x, x p )(a y 2 2 + b y 2 2 + c y, y q )dπ(x, y)dπ(x , y ) = ˆ (a 2 x 2 2 y 2 2 + ab x 2 2 y 2 2 + ac x 2 2 y, y q ) + (ab x 2 2 y 2 2 + b 2 x 2 2 y 2 2 + bc x 2 2 y, y q )
+ (ca x, x p y 2 2 + cb x, x p y 2 2 + c 2 x, x p y, y q ) dπ(x, y)dπ(x , y ) where in (*) we used that x T x ∈ R so it is equal to its transpose.

= (a 2 + b 2 ) ˆ x 2 2 y 2 2 dπ(x, y) + 2ab ˆ x 2 2 y 2 2 dµ(x)dν(y) + c 2 ˆ x,

Proof of the existence and finiteness (innerGW) and (MaxOT)

In this section we prove that (innerGW) and (MaxOT) are finite and that (MaxOT) always admits a maximizer. More precisely: Lemma 6.2.7. Let µ ∈ P(R p ), ν ∈ P(R q ) with ´ x 4 2 dµ(x) < +∞,

´ y 4 2 dν(y) < +∞. Both (innerGW) and (MaxOT) are finite.

Moreover the set F p,q is a compact subset of R q×p . The functional π → sup P∈Fp,q

´ Px, y q dπ(x, y)

is continuous for the weak convergence of measure. In particular problem (MaxOT) admits an optimal solution π * ∈ Π(µ, ν)

Proof. In this case ´ x 2 2 dµ(x) < +∞,

´ y 2 2 dν(y) < +∞ by Hölder's inequality and for all (x, x , y, y ) ∈ X 2 × Y 2 x, x p -y, y q 2 ≤ 2( x, x p 2 + y, y q 2 ) ≤ 2( x 2 2 x 2 2 + y 2 2 y 2 2 ) by Cauchy-Swartz. In particular this implies: inf π∈Π(µ,ν)

ˆˆ

x, x p -y, y q 2 dπ(x, y)dπ(x , y ) ≤ ˆˆ x, x p -y, y q 2 dµ(x)dµ(x )dν(y)dν(y )

≤ 2( ˆ x 2 2 dµ(x) ˆ x 2 2 dµ(x ) + ˆ y 2 2 dν(x) ˆ y 2 2 dν(y )) < +∞ Moreover for any π ∈ Π(µ, ν), P ∈ F p,q : ˆ Px, y q dπ(x, y) = ˆ P, yx T F dπ(x, y) ≤ ˆ P 2 F xy T 2 F dπ(x, y) ≤ p ˆ xy T 2 F dπ(x, y) = p ˆtr(yx T xy T )dπ(x, y) = p ˆtr(x T xy T y)dπ(x, y) = p ˆtr( x 2 2 y 2 2 )dπ(x, y) = p ˆ x 2 2 y 2 2 dπ(x, y) * ≤ p 2 ( ˆ x 4 2 dµ(x) + ˆ y 4 2 dν(y)) < +∞
where in (*) we used Young's inequality (recalled in Memo 4.2.2).

For the compacity, using Borel Lebesgue theorem it suffices to show that F p,q is closed and bounded. It is clearly bounded by √ p and closed as the pre-image of the closed set {0} by the continuous application P → √ p -P F . We note f : Π(µ, ν) × F p,q → R the function f (π, P) = -

´ Px, y q dπ(x, y). For any π ∈ Π(µ, ν), f (π, .) is continuous. Indeed suppose that P m → F P where → F denotes the convergence in Frobenius norm, Memo 6.2.3. Let X , Y be topological spaces with Y compact. If f : X × Y → R is continuous then g : x → inf y f (x, y) is well defined and continuous.

Proof. Note first that g(x) > -∞, since for every x ∈ X , f (x, .) : Y → R is continuous on a compact space so it is bounded. To prove the continuity it suffices to show that g

-1 (] -∞, a[) and g -1 (]b, +∞[) are open. For the former, if we note π X : X × Y → X the canonical projection then g -1 (] -∞, a[) = π X • f -1 (] -∞, a[). By continuity of f we can conclude that g -1 (] -∞, a[) is open. For the latter first observe that g(x) > b =⇒ ∀y, f (x, y) > b which means that g(x) > b =⇒ ∀y, (x, y) ∈ f -1 (]b, +∞[).
In particular for x ∈ g -1 (]b, +∞[) and y ∈ Y there exists a neighborhood U (x,y) × V (x,y) contained in f -1 (]b, +∞[). Since Y is compact there exists a finite subset {(x, y i )} of all the neighborhoods U (x,y) × V (x,y) which cover all of {x} × Y. Overall:

{x} × Y ⊂ (∩ k i=1 U (x,yi) ) × Y ⊂ f -1 (]b, +∞[) (6.67) Hence g -1 (]b, +∞[) = ∪ x∈g -1 (]b,+∞[) ∩ k i=1 U (x,yi)
) which is open so g is continuous.

then:

| ˆ P m x, y q dπ(x, y) -ˆ Px, y q dπ(x, y)| ≤ ˆ| (P m -P)x, y q |dπ(x, y)

≤ P m -P 2 F ( ˆ x 2 2 y 2 2 dπ(x, y)) ≤ 1 2 P m -P 2 F ( ˆ x 4 2 dµ(x) + ˆ y 4 2 dν(y)) → m→+∞ 0
In this way, since F p,q is compact, g : π → inf P∈F f (π, P) is well defined and continuous for the weak convergence of measure (see Memo 6.2.3). Since it is continuous and Π(µ, ν) is compact we can applied the Weirstrass' theorem to state that inf π∈Π(µ,ν) g(π) exists which concludes the proof.

Proof of Lemma 4.2.3 -Reduction of the GW cost

We recall the lemma:

Lemma. Let X and Y be compact subset of respectively R p and R q . Let µ ∈ P(X ), ν ∈ P(Y). We can assume without loss of generality that

E X∼µ [X] = 0 and E Y ∼ν [Y ] = 0. In this case (sqGW) is equivalent to: sup π∈Π(µ,ν) ˆ x 2 2 y 2 2 dπ(x, y) + 2 ˆyx T dπ(x, y) 2 F Indeed equation (sqGW) corresponds to the case a = 1, b = 1, c = -2 of Lemma 4.2.1 and so is equivalent to: sup π∈Π(µ,ν) 2 ˆ x 2 2 y 2 2 dπ(x, y) + 4 ˆxy T dπ(x, y) 2 F -4 ˆ x 2 2 E Y ∼ν [Y ], y d + y 2 2 E X∼µ [X], x d dπ(x, y) (6.68)
We can reduce this problem using the translation invariance of the GW cost since x + t -(x + t) 2 2 = xx 2 2 for all x, x (same for y). More precisely we can rely on the following lemmas: Lemma 6.2.8. Let f : R p → R p and f : R q → R q Borel and µ

∈ P(R p ), ν ∈ P(R q ) Then Π (f #µ, f #ν) = {(f × f )#π|π ∈ Π(µ, ν)}
Proof. This is a straightforward extension of the Lemma 6 in [Paty 2019].

Based on previous Lemma 6.2.8 we have: Lemma 6.2.9 (Translation Invariance). Let µ ∈ P(R p ), ν ∈ P(R q ) and c

X (x, x ) = x -x 2 2 , c Y (y, y ) = y -y 2 2 . Let t, t ∈ R p × R q and f t (x) = x + t, f t = y + t be translations. Then: sup π∈Π(µ,ν) J 2 (c X , c Y , π) = sup π∈Π(ft#µ,f t #ν) J 2 (c X , c Y , π) (6.69)
Proof. Using Lemma 6.2.8 we have:

sup π∈Π(ft#µ,f t #ν) J 2 (c X , c Y , π) = sup π∈Π(µ,ν) J 2 (c X , c Y , (f t × f t )#π) * = sup π∈Π(µ,ν) J 2 (c X , c Y , π)
where in (*) we used the invariance of the cost J 2 (c X , c Y , π) with respect to translations.

Using this property we can assume without loss of generality µ and ν centered, i.e, E X∼µ [X] = 0 and

E Y ∼ν [Y ] = 0.
By plugging this condition into Z(π) of Lemma 4.2.1 we have the desired result. Lemma 6.2.12. Let µ = N (0, Σ ν ) ∈ P(R p ), ν = N (0, Σ µ ) ∈ P(R q ) centered without loss of generality. For a linear push-forward T #µ = ν we note T (x) = Ax. The Gromov-Monge problem is equivalent to: 6.77) where:

min A∈R q×p AΣµA =Σν M, A A F (
M = E x,x ∼µ [-2(x x).xx -2(x x ).xx -4(x x ).
x x ] (6.78)

Proof. We have:

min

T #µ=ν T linear J(T ) = min T #µ=ν T linear E x,x ∼µ [ x -x 2 2 -T (x) -T (x ) 2 2 2 ] = min T #µ=ν T linear E x,x ∼µ [ x -x 4 ] + E x,x ∼µ [ T (x) -T (x ) 4 2 ] -2 E x,x ∼µ [ x -x 2 2 T (x) -T (x ) 2 2 ] = E x,x ∼µ [ x -x 4 2 ] + E y,y ∼ν [ y -y 4 2 ] + 2 min T #µ=ν T linear J 1 (T ) With J 1 (T ) def = -E x,x ∼µ [ x -x 2 2 T (x) -T (x ) 2 2 ]
. In this way the problem is equivalent to minimizing J 1 (T ). Since T #µ = ν:

J 1 (T ) = -E x,x ∼µ [ x -x 2 2 T (x) -T (x ) 2 2 ] = -E x,x ∼µ [( x 2 2 + x 2 2 -2 x, x p )( T (x) 2 2 + T (x ) 2 -2 T (x), T (x ) q )] = -E x,x ∼µ [ x 2 2 T (x) 2 2 + x 2 2 T (x ) 2 2 ]-E x,x ∼µ [ x 2 2 T (x) 2 2 + x 2 2 T (x ) 2 2 ] +2 E x,x ∼µ [ x, x T (x) 2 2 ] + 2 E x,x ∼µ [ x, x p T (x ) 2 2 ] -4 E x,x ∼µ [ x, x p T (x), T (x ) q ] +2 E x,x ∼µ [ T (x), T (x ) q x 2 2 ] + 2 E x,x ∼µ [ T (x), T (x ) q x 2 2 ] = -2 E x∼µ [ x 2 2 T (x) 2 2 ]-2 E x,x ∼µ [ x 2 2 T (x) 2 2 ]+4 E x,x ∼µ [ x, x p T (x) 2 2 ] -4 E x,x ∼µ [ x, x p T (x), T (x ) q ]+4 E x,x ∼µ [ T (x), T (x ) q x 2 2 ]
Since T is linear it can be written in the form T (x) = Ax. The push-forward constraint in this case reads [START_REF] Flamary | [END_REF]]:

AΣ µ A = Σ ν (6.79)
When A is symmetric positive definite this equation admits a unique solution which is the optimal linear map for the Wasserstein problem. However here we have no result about the regularity of A. Plugging T (x) = Ax into J 1 (T ) and writing the push-forward condition gives the equivalent problem:

min A∈R q×p ,AΣµA =Σν J 2 (A) (6.80)
where

J 2 (A) = -2 E x∼µ [x xx A Ax]-2 E x,x ∼µ [x x x A Ax]+4 E x,x ∼µ [x x x A Ax] (6.81) -4 E x,x ∼µ [x x x A Ax ]+4 E x,x ∼µ [x A Ax x x] (6.82)
Using the property of the trace of a matrix and linearity of the inner product, one can reformulate J 2 as J 2 (A) = M, A A F (6.83) where

M = E x,x ∼µ [-2(x x).xx -2(x x ).xx +4(x x ).xx -4(x x ).x x +4(x x).x x ] (6.84)
Indeed for all terms we used the following reasoning: 6.85) where in (*) we used that x xx A Ax ∈ R, in (**) that x x ∈ R and in (***) the cyclical permutation invariance of tr. Moreover E

E x∼µ [x xx A Ax] * = E x∼µ [tr(x xx A Ax)] * * = E x∼µ [x xtr(x A Ax)] * * * = E x∼µ [x xtr(xx A A)] = E x∼µ [tr(x xxx A A)] = tr( E x∼µ [x xxx A A]) = tr( E x∼µ [x xxx ]A A) = E x∼µ [x xxx ], A A F (
x,x ∼µ [(x x).x x ], E

x,x ∼µ (x x ).xx ] = 0 since Gaussians are centered. Hence:

M = E x,x ∼µ [-2(x x).xx -2(x x ).xx -4(x x ).x x ] (6.86)
We can go further on this calculus by using Isserlis' theorem recalled here:

Theorem (Isserlis [Isserlis 1918]). If (X 1 , X 2 , X 3 , X 4 ) is a zero-mean multivariate normal random vector then:

E[X 1 X 2 X 3 X 4 ] = E[X 1 X 2 ]E[X 3 X 4 ] + E[X 1 X 3 ]E[X 2 X 4 ] + E[X 1 X 4 ]E[X 2 X 3 ] (6.87)
Applying this theorem to M in (6.77) gives the following lemma: Lemma 6.2.13. Let µ = N (0, Σ ν ) ∈ P(R p ), ν = N (0, Σ µ ) ∈ P(R q ) centered without loss of generality. Then:

min A∈R q×p AΣµA =Σν M, A A F = min A∈R q×p AΣµA =Σν -4tr(Σ µ ).Σ µ -8Σ µ Σ µ , A A F (6.88)
Proof. We will use Isserlis' theorem to calculate M in (6.77). We have for i, j: 6.89) In (*) we used Isserlis' theorem. Hence E x∼µ [(x x).xx ] = tr(Σ µ ).Σ µ + 2Σ µ Σ µ . In the same way:

( E x∼µ [(x x).xx ]) i,j = E x [ k x k x k x i x j ] = k E x [x k x k x i x j ] * = k E x [x k x k ]E x [x i x j ] + E x [x k x i ]E x [x k x j ] + E x [x k x j ]E x [x k x i ] = k Σ k,k µ Σ i,j µ + Σ k,i µ Σ k,j µ + Σ k,j µ Σ k,i µ = k Σ k,k µ Σ i,j µ + 2Σ k,i µ Σ k,j µ = Σ i,j µ tr(Σ µ ) + 2(Σ µ Σ µ ) i,j (
( E x,x [(x x ).xx ]) i,j = E x,x [ k x k x k x i x j ] = k E x,x [x k x k x i x j ] = k E x,x [x k x k ] E x,x [x i x j ] + E x,x [x k x i ] E x,x [x k x j ] + E x,x [x k x j ] E x,x [x k x i ] * = k Σ k,k µ Σ i,j µ = tr(Σ µ )Σ i,j µ (6.90)
In (*) we used the independence of x and x and the fact that Gaussians are centered. Finally,

( E x,x [(x x ).x x ]) i,j = E x,x [ k x k x k x i x j ] = k E x,x [x k x k x i x j ] = k E x,x [x k x k ] E x,x [x i x j ] + E x,x [x k x i ] E x,x [x k x j ] + E x,x [x k x j ]E x [x k x i ] = k Σ k,i µ Σ k,j µ = (Σ µ Σ µ ) i,j (6.91)
Overall we want to solve the following optimization problem: min

Proof of Proposition 5.3.1 -COOT is a distance

We recall the proposition:

Proposition (COOT is a distance). Suppose L = | • | p , p ≥ 1, n = n , d = d
and that the weights w, w , v, v are uniform. Then COOT(X, X ) = 0 iff there exists a permutation of the samples σ 1 ∈ S n and of the features σ 2 ∈ S d , s.t, ∀i, k X i,k = X σ1(i),σ2(k) . Moreover, it is symmetric and satisfies the triangular inequality as long as L satisfies the triangle inequality, i.e., COOT(X, X ) ≤ COOT(X, X ) + COOT(X , X ).

Proof. The symmetry follows from the definition of COOT. To prove the triangle inequality of COOT for arbitrary measures, we will use the gluing lemma (see [Villani 2008]) which states the existence of couplings with a prescribed structure. Let X ∈ R n×d , X ∈ R n ×d , X ∈ R n ×d associated with

w ∈ Σ n , v ∈ Σ d , w ∈ Σ n , v ∈ Σ d , w ∈ Σ n , v ∈ Σ d .
Without loss of generality, we can suppose in the proof that all weights are different from zeros (otherwise we can consider wi = w i if w i > 0 and wi = 1 if w i = 0 see proof of Proposition 2.2 in [START_REF] Peyré | [END_REF]]) Let (π s 1 , π v 1 ) and (π s 2 , π v 2 ) be two couples of optimal solutions for the COOT problems associated with COOT(X, X , w, w , v, v ) and COOT(X , X , w , w , v , v ) respectively.

We define:

S 1 = π s 1 diag 1 w π s 2 , S 2 = π v 1 diag 1 v π v 2
Then, it is easy to check that S 1 ∈ Π(w, w ) and S 2 ∈ Π(v, v ) (see e.g Proposition 2.2 in [START_REF] Peyré | [END_REF]). We now show the following: COOT(X, X , w, w , v, v ) * ≤ L(X, X ) ⊗ S 1 , S 2 = L(X, X )

⊗ [π s 1 diag( 1 w )π s 2 ], [π v 1 diag( 1 v )π v 2 ] * * ≤ [L(X, X ) + L(X , X )] ⊗ [π s 1 diag( 1 w )π s 2 ], [π v 1 diag( 1 v )π v 2 ] = L(X, X ) ⊗ [π s 1 diag( 1 w )π s 2 ], [π v 1 diag( 1 v )π v 2 ] + L(X , X ) ⊗ [π s 1 diag( 1 w )π s 2 ], [π v 1 diag( 1 v )π v 2 ] ,
where in (*) we used the suboptimality of S 1 , S 2 and in (**) the fact that L satisfies the triangle inequality. Now note that: where in (*) we used: Overall, from the definition of π s 1 , π v 1 and π s 2 , π v 2 we have:

L(X, X ) ⊗ [π s 1 diag( 1 w )π s 2 ], [π v 1 diag( 1 v )π v 2 ] + L(X , X ) ⊗ [π s 1 diag( 1 w )π s 2 ], [π v 1 diag( 1 v )π v 2 ] = i,
COOT(X, X , w, w , v, v ) ≤ COOT(X, X , w, w , v, v ) + COOT(X , X , w , w , v , v ).

For the identity of indiscernibles, suppose that n = n , d = d and that the weights w, w , v, v are uniform. Suppose that there exists a permutation of the samples σ 1 ∈ S n and of the features σ 2 ∈ S d , (k) . We define the couplings π s , π v supported on the graphs of the permutations σ 1 , σ 2 respectively, i.e π s = (id × σ 1 ) and π v = (id × σ 2 ). These couplings have the prescribed marginals and lead to a zero cost hence are optimal.

s.t ∀i, k ∈ [[n]] × [[d]], X i,k = X σ1(i),σ2
Conversely, as described in the chapter, there always exists an optimal solution of (COOT) which lies on extremal points of the polytopes Π(w, w ) and Π(v, v ). When n = n , d = d and uniform weights are used, Birkhoff's theorem [START_REF] Birkhoff | [END_REF]] states that the set of extremal points of Π( 1n n , 1n n ) and Π( 1 d d , 1 d d ) are the set of permutation matrices so there exists an optimal solution (π s

Complexity of computing the value of COOT

We recall the result: Proof. As mentionned in [START_REF] Peyré | [END_REF]], if L can be written as L(a, b) = f (a) + f (b) -h 1 (a)h 2 (b) then we have that L(X, X ) ⊗ π s = C X,X -h 1 (X)π s h 2 (X ) T , where C X,X = Xw1 T n +1 n w T X T so that the latter can be computed in O(ndd +n dd ) = O((n+n )dd ). To compute the final cost, we must also calculate the inner product with π v that can be done in O(n 2 n) making the complexity of L(X, X ) ⊗ π s , π v equal to O((n + n )dd + n 2 n).

Finally, as the cost is symmetric w.r.t π s , π v , we obtain the overall complexity of O(min{(n + n )dd + n 2 n; (d + d )nn + d 2 d}).

Proofs of theorem 5.5.1 -Equivalence between QAP and BAP

As pointed in [Konno 1976b], we can relate the solutions of a QAP and a BAP. We will prove the equivalent following result (maximization version of theorem 5.5.1):

Theorem. If Q is a positive semi-definite matrix, then problems:

max x f (x) = c T x + 1 2 x T Qx s.t.
Ax = b, x ≥ 0 (6.101) max x,y g(x, y) = 1 2 c T x + 1 2 c T y + 1 2 x T Qy s.t. Ax = b, Ay = b, x, y ≥ 0 (6.102) are equivalent. More precisely, if x * is an optimal solution for (6.101), then (x * , x * ) is a solution for (6.102) and if (x * , y * ) is optimal for (6.102), then both x * and y * are optimal for (6.101).

Proof. This proof follows the proof of Theorem 2.2 in [Konno 1976b]. Let z * be optimal for (6.101) and (x * , y * ) be optimal for (6.102). Then, by definition, for all x satisfying the constraints of (6.101), f (z * ) ≥ f (x). In particular, f (z * ) ≥ f (x * ) = g(x * , x * ) and f (z * ) ≥ f (y * ) = g(y * , y * ). Also, g(x * , y * ) ≥ max By adding these inequalities we obtain:

(x * -y * ) T Q(x * -y * ) ≤ 0.

Since Q is positive semi-definite, this implies that Q(x * -y * ) = 0. So, using previous inequalities, we have c T (x * -y * ) = 0, hence g(x * , y * ) = g(x * , x * ) = g(y * , y * ) as required.

Note also that this result holds when we add a constant term to the cost function. We can rewrite this equation as stated in the proposition using the vec operator.

Using a standard QP form c T x + 1 2 xQ x T with c = -4vec(M) and Q = -4XX T ⊗ K X X T we see that the Hessian is negative semi-definite as the opposite of a Kronecker product of positive semi-definite matrices XX T and X X T .

The inner product case is the same calculus, only the constant term changes and M = 0. Note that both calculus were also made in Chapter 4, and precisely in lemma 4.2.1.

Proof of Proposition 5.5.2 -Equality between COOT and GW in the concave regime

We recall the proposition:

Proposition. Let C ∈ R n×n , C ∈ R n ×n be any symmetric matrices, then:

COOT(C, C , w, w , w, w ) ≤ GW (C, C , w, w ).

The converse is also true under the hypothesis of Proposition 5.5.1. In this case, if (π s * , π v * ) is an optimal solution of COOT, then both π s * , π v * are solutions of GW . Conversely, if π s * is an optimal solution of GW , then (π s * , π s * ) is an optimal solution for COOT.

Proof. The inequality follows from the fact that any optimal solution of the GW problem is an admissible solution for the COOT problem, hence the inequality is true by suboptimality of this optimal solution. For the equality part, by following the same calculus as in the proof of lemma 6.3.1, we can verify that:

COOT(C, C , w, w , w, w ) = min π s ∈Π(w,w )

-2vec(M) T vec(π s )

-2vec(M) T vec(π v ) -8vec(π s ) T Qvec(π v ) + Cte, with M, Q as defined in lemma 6.3.1. Since Q is negative semi-definite, we can apply Theorem 5.5.1 to prove that both problems are equivalent and lead to the same cost and that every optimal solution of GW is an optimal solution of COOT and vice versa. Same applies for the inner product case.

Equivalence of DC algorithm and Frank-Wolfe algorithm for GW

We recall the result:

Proposition. When X = C, X = C are squared Euclidean distance matrices or inner product similarities the iterations of Algorithm 10 are the same as the iteration of the FW procedure defined in Chapter 3 for solving GW (provided that the initialization is the same).

Proof. Using Proposition 5.5.2, we know that when X = C, X = C are squared Euclidean distance matrices or inner product similarities, then there is an optimal solution of the form (π * , π * ). In this case, we can set π s (k) = π v (k) during the iterations of Algorithm 9 to obtain an optimal solution for both COOT and GW. This reduces to Algorithm 10 that corresponds to a DC algorithm where the quadratic form is replaced by its linear upper bound.

Below, we prove that this DC algorithm for solving GW problems is equivalent to the Frank-Wolfe The cases when L = |•| 2 and C, C are squared Euclidean distance matrices or inner product similarities have interesting implications in practice, since in this case the resulting GW problem is a concave QP (as explained in this Chapter and shown in Lemma 6.3.1). In [START_REF] Maron | [END_REF]], the authors investigated the solution to QP with conditionally concave energies using a FW algorithm and showed that in this case the line-search step of the FW is always 1. Moreover, as shown in Proposition 6.3.1, the GW problem can be written as a concave QP with concave energy and is minimizing a fortiori a conditionally concave energy. Consequently, the line-search step of the FW algorithm proposed in Chapter 3 and described in Algorithm.11 always leads to an optimal line-search step of 1. In this case, the Algorithm.11 is equivalent to Algorithm.12 goven below, since τ (k) = 1 for all k.

Algorithm 12 FW Algorithm for GW with squared Euclidean distance matrices or inner product similarities (k-1) , which gives the same OT solution as for the OT problem in step 3 of Algorithm 10, we can conclude that the iterations of both algorithms are equivalent.

Additional results -Relation with election isomorphism problem

This section shows that COOT approach can be used to solve the election isomorphism problem defined in [Faliszewski 2019] as follows: let E = (C, V ) and E = (C , V ) be two elections, where C = {c 1 , . . . , c m } (resp. C ) denotes a set of candidates and V = (v 1 , . . . , v n ) (resp. V ) denotes a set of voters, where each voter v i has a preference order, also denoted by v i . The two elections E = (C, V ) and E = (C , V ), where abs C = abs C , V = (v 1 , . . . , v n ), and V = (v 1 , . . . , v n ), are said to be isomorphic if there exists a bijection σ : C → C and a permutation ν ∈ S n such that σ(v i ) = v ν(i) for all i ∈ As shown in the chapter, there exists an optimal solution for COOT(X, X ) given by permutation matrices as solutions of the Monge-Kantorovich problems for uniform distributions supported on the same number of elements. Then, one may show that the solution of the two problems coincide modulo a multiplicative factor, i.e., π s * = 1 n N * and π v * = 1 m M * are optimal since abs C = abs C and abs V = abs V . For π s * (the same reasoning holds for π v * as well), we have that

(π s * ) ij = 1 n , j = ν * i 0, otherwise.
where ν * i is a permutation of voters in the two sets. The only difference between the two solutions π s * and N * thus stems from marginal constraints (6.104). To conclude, we note that COOT is a more general approach as it is applicable for general loss functions L, contrary to the Spearman distance used in [Faliszewski 2019], and generalizes to the cases where n = n and m = m .

Additional results -Complementary results for the HDA experiment

Here, we present the results for the heterogeneous domain adaptation experiment not included in section 5.6.1. Table 6.6 follows the same experimental protocol as in the chapter but shows the two cases where n t = 1 and n t = 5. Table 6.7 and Table 6.8 contain the results for the adaptation from GoogleNet to Decaf features, in a semi-supervised and unsupervised scenarios, respectively Overall, the results are coherent with those from the chapter: in both settings, when n t = 5, one can see that the performance differences between SGW and COOT is rather significant.

Overview of the contributions

This thesis presents a set of optimal transport tools for dealing with probability distributions on incomparable spaces, or equivalently probability distributions whose supports do not lie in a common metric space. We explained how this problem occurs e.g. when one needs to consider some structural knowledge about the data or when the data come from heterogeneous sources. As a first instance of probability distributions on incomparable spaces we studied the setting of structured data such as labeled graphs, times series or any "relational" data whose structure can be modelled through a notion of cost or similarity. We showed how to describe them as a probability distributions and how to compare them using the so-called Fused Gromov-Wasserstein distance which builds upon Wasserstein and the Gromov-Wasserstein distances. This new optimal transport distance was successfully applied in a graph context where it finds applications for the classification, clustering or summarization of labeled graphs.

As the main building block of F GW , the Gromov-Wasserstein distance is a central notion of this thesis. We attempted to bridge the gap between the understanding of the Wasserstein distance and GW where we consider the special case of Euclidean spaces. This setting allows us to derive a sliced approach, based on the first closed-form expression for GW between 1D probability distributions. We called it Sliced Gromov-Wasserstein, akin to the Sliced Wasserstein distance which has recently found many applications in machine learning. The Euclidean setting reveals to be also a good starting point for analysing the regularity of GW optimal transport plans and, as such, we partially answered the following question: can we find guarantees on the probability measures so that an optimal transport plan for GW can be expressed through a deterministic function? This question, central in linear optimal transport theory, can be tackled with the celebrated Brenier's theorem in the context of the Wasserstein distance but was still quite under-addressed when dealing with GW .

Although GW is a powerful tool for comparing probability distributions on incomparable spaces it is limited in its ability to find correspondences between the features of the samples of these distributions. differ or in order to retrieve the invariants of GW . The max-sliced approach could be a interesting remedy for this situation. We can for example draw one line in each space without requiring any map ∆ but with the price of losing the divergence property (i.e. max -SGW (µ, µ) could be different from zero). Another potential direction would be to consider many lines in each space and then find their correspondences by using e.g. Wasserstein which could define a proper metric w.r.t. isomorphisms but which will come with a cubic complexity. Interestingly enough this approach would exactly be the COOT distance but with replacing the matching of the feature by a sliced Wasserstein. In any case there are room for improvements in the way of defining SGW when dimensions of the support of the probability measures differ.

Regarding the theoretical aspects of SGW another interesting line of research, in our opinion, would be to draw connections with the statistical properties of the Sliced Wasserstein distance which is known to behave well in terms of convergence of finite samples. We believe that similar type of studies for the Sliced Gromov-Wasserstein could be promising.

From a computational OT point of view the CO-Optimal Transport framework seems also to be quite suited for large scale datasets. Indeed since the BCD procedure used for solving COOT is based on alternating linear optimal transport problems another line of works could be to rely on the dual or semi dual formulations of linear OT, used for large scale OT, in order to compute COOT for large scale datasets whose samples are in incomparable spaces. Finally from a theoretical point of view a continuous formulation of COOT could also be of interest for studying the properties of GW , and especially the regularity of its optimal plans. Indeed we have proven that an optimal solution for COOT can be found via permutation matrices, whose continuous counterparts are deterministic push-forwards. As such, can we find a natural extension for COOT which preserves this property in the continuous setting? Also if the problem is concave, such as when GW is considered with squared Euclidean distances, can we preserve the result stating that COOT and GW are equivalent so that an optimal solution of the first is optimal solution for the second? In this way can we conclude that there exists an optimal map for GW that is supported on a Monge map when the problem is concave? We believe, enthusiastically, that these questions worth investigating.

How other fields can contribute to the frameworks developed in this thesis? One of the most challenging improvement of the work developed in this thesis is maybe to lighten the computational complexity of calculating F GW which is driven by the complexity of the GW distance. In this form, the computational complexity F GW renders this framework limited to small to medium scale scenarii and, in particular, is inapplicable for very large graphs. Moreover one can not hope to directly rely on the works about GW on Euclidean spaces in order to derive tractable formulations since the structures of labeled graphs are usually highly non-Euclidean. However it worth pointing out that approximating C 1 , C 2 with Euclidean matrices can be done in several ways [START_REF] Borg | [END_REF], Glunt 1990[START_REF] Alfakih | [END_REF], Liberti 2014]. We believe that a kind of no free lunch theorem applies here: if we faithfully model the structure of the data it is likely to result in a more precise notion of distance between structured data but also in a more challenging optimization problem than if we approximate it for better scalability. Needless to say, since GW is at the backbone of our method, any further improvements in computational efficiency for non convex Quadratic programs, and by way of consequence for GW , would directly benefit to F GW . In this way progress in graph matching could definitely contribute to derive efficient algorithms for solving F GW as both are intrinsically related. Another interesting study would be regarding the design of the matrices C 1 , C 2 representative of the structures for F GW . We considered in this thesis that these matrices are given, such as shortest-path matrices, yet, one could argue that this is a strong prior on the structures of the graphs and that no perfect choice arises in practice. The problem of finding "good" structure matrices is very related with the metric learning field [START_REF] Bellet | [END_REF]] and further appealing works for F GW could be to go into this direction and to add C 1 , C 2 into the learning process. Related to this problem we believe that very insightful connections between F GW and the field of signal graph processing [Ortega 2018] can be drawn and that both can benefit from each other. As one example: can we build upon the spectral tools from signal graph processing in order to find adequate measures of structure C 1 , C 2 of the graphs? How do the above-mentioned optimal transport tools can, or can not, be used for machine learning?

As mentioned throughout this thesis, having a adequate measure for comparing data lying on incomparable spaces can be useful for a wide range of machine learning tasks. For applications involving structured data we proposed the F GW distance based on optimal transport. Note, however, that many ways of comparing structured data have been proposed in the literature: from the design of dedicated kernels (see [START_REF] Kriege | [END_REF]] for a comprehensive survey on this topic) to other choices of distances between graphs [Bento 2019, Wills 2020] including the popular graph-edit distance [START_REF] Willett | [END_REF][START_REF][END_REF]]. More recently, end-to-end approaches [Li 2019[START_REF] Bai | [END_REF][START_REF] Riba | Learning Graph Distances with Message Passing Neural Networks[END_REF][START_REF] Sun | Graph Neural Networks for Graph Search[END_REF]] attempt to learn a similarity measure function between graphs based on graph neural networks, i.e. to learn a neural network based function that takes two graphs as input and outputs the desired similarity. The question of finding a "good" similarity measure for structured data is far from being closed and is naturally dependent of the application. The choice of F GW can be motivated by its metric properties which allow detecting whether two graphs are isomorphic and, as such, could be used as an alternative to the graph-edit distance. However, despite its appealing properties, one could question the optimal transport framework on which this method is based. As described in this thesis the coupling matrices are used to find a probabilistic matching between all the nodes of two graphs. Conversely, and depending on the application, one might want to match only a small portion of the nodes of two graphs which is not possible using the F GW framework which considers global matchings. In this way, in a scenario where only the local structure is important, the F GW machinery appears to be quite disproportionate, and some kernel methods which are built on local structures may be more suited. The use of the F GW distance for structured data such as time series is also questionable. Indeed the set of couplings itself is constrained so that a matching "back in time" is made possible. On the contrary, it may be more interesting to force that the points which are matched to the target sequence at time t can only depend on the source sequence up to time t as done e.g. using Dynamic Time Warping distances [Cuturi 2017[START_REF] Sakoe | [END_REF]. To remedy to this problem the recent casual optimal transport framework [Lassalle 2018, Veraguas 2016[START_REF] Acciaio | [END_REF], Zalashko 2017] may be more suited for these type of problems. Apart from these limitations the F GW framework could be useful for other applications than these mentioned in this thesis. As a first instance it could be used as a way for learning graph neural networks (GNN). Given a set of graphs we could for example minimize a triplet loss [START_REF] Chechik | [END_REF]] or a graph similarity score [START_REF] Bai | [END_REF]] which impose that the distances in the embedding of the GNN are close to the F GW distances so as to force the GNN to produce similar embeddings for similar graphs. Moreover the properties of the Fréchet barycenter with F GW could be elaborated, especially the projection on a smaller graph using F GW . A perspective on this topic would be to draw connections with graph signal processing and more standard coarsening procedures (see [Loukas 2019] and references therein) by questioning the ability of the F GW projection to reduce the graph without altering too much its spectral properties.

Finally the results of this thesis suggests that COOT may be suited for Heterogeneous Domain Adaptation. An interesting further work on this topic would be to see if we can confirm these empirical results with theoretical guarantees such as bounds for Heterogeneous Domain Adaptation by relying on the COOT framework. For example can we prove that the adaptation is easier when the datasets are close from each other w.r.t. the COOT distance than if they are far from each other? To the best of our knowledge the question of finding bounds for HDA problems is still quite under-addressed [START_REF] Zhou | [END_REF]] and COOT may lead to promising further works on this topic.

All these considered we hope that the works proposed in this thesis will pave the path for positive and interesting studies on various and interdisciplinary topics in machine learning and that it will also contribute to the richness of the optimal transport theory, which is far from being extinguished. Résumé : Le Transport Optimal est une théorie permettant de définir des notions géométriques de distance entre des distributions de probabilité et de trouver des correspondances, des relations, entre des ensembles de points. De cette théorie, à la frontière entre les mathématiques et l'optimisation, découle de nombreuses applications en machine learning. Cette thèse propose d'étudier le scénario, complexe, dans lequel les différentes données appartiennent à des espaces incomparables. En particulier nous abordons les questions suivantes : comment définir et appliquer le transport optimal entre des graphes, entre des données structurées ? Comment l'adapter lorsque les données sont variées et ne font pas partie d'un même espace mé-trique ? Cette thèse propose un ensemble d'outils de Transport Optimal pour ces différents cas. Un important volet est notamment consacré à l'étude de la distance de Gromov-Wasserstein dont les propriétés permettent de définir d'intéressants problèmes de transport sur des espaces incomparables. Plus largement, nous analysons les propriétés mathématiques des différents outils proposés, nous établissons des solutions algorithmiques pour les calculer et nous étudions leur applicabilité dans de nombreux scenarii de machine learning qui couvrent, notamment, la classification, la simplification, le partitionnement de données structurées, ainsi que l'adaptation de domaines hétérogènes.

Title: A contribution to Optimal Transport on incomparable spaces

Keywords: Optimal Transport, Structured Data, Wasserstein and Gromov-Wasserstein Abstract: Optimal Transport is a theory that allows to define geometrical notions of distance between probability distributions and to find correspondences, relationships, between sets of points. Many machine learning applications are derived from this theory, at the frontier between mathematics and optimization. This thesis proposes to study the complex scenario in which the different data belong to incomparable spaces. In particular we address the following questions: how to define and apply the optimal transport between graphs, between structured data? How can it be adapted when the data are varied and not embedded in the same metric space? This thesis proposes a set of Optimal Transport tools for these different cases. An important part is notably devoted to the study of the Gromov-Wasserstein distance whose properties allow to define interesting transport problems on incomparable spaces. More broadly, we analyze the mathematical properties of the various proposed tools, we establish algorithmic solutions to compute them and we study their applicability in numerous machine learning scenarii which cover, in particular, classification, simplification, partitioning of structured data, as well as heterogeneous domain adaptation.

Figure 1 :

 1 Figure 1: Le Transport Optimal fournit des outils pour trouver des correspondances, relations, entre des ensembles de points et donne une notion géométrique de distance entre des distributions de probabilité. (à gauche) Deux ensembles de points en 2D. Les correspondances trouvées par l'OT sont illustrées en pointillées. (à droite) Les distances définies par l'OT entre deux distributions de probabilité bleue et rouge. Plus la moyenne de la distribution bleue est éloignée de la rouge, plus la distance est grande.

Figure 2 :

 2 Figure 2: On associe à un dataset (xi) i∈[[n]] une distribution de probabilité le décrivant entièrement. Dans cet exemple xi ∈ R 2 . (à gauche) Formulation Lagragienne (ou nuage de points): n i=1 aiδx i est une mesure de probabilité discrète où (ai) i∈[[n]] est un vecteur de probabilité, i.e. ai ≥ 0 et n i=1 ai = 1 et δ est la mesure de dirac i.e. δx i (x) = 1 si x = xi et 0 sinon. (à droite) Formulation Euleriennne (histogrammes): on associe une mesure de probabilité N i=1 aiδ xi où (xi) i∈[[N ]] est une grille régulière qui discrétise l'espace R 2 et (ai) i∈[[N ]] est un vecteur de probabilité.

Figure 4 :

 4 Figure 4: Illustration de COOT entre les datasets MNIST et USPS. (gauche) samples des datasets MNIST et USPS ; (centre gauche) Matrice de transport π s entre les samples, triés par classe ; (centre) Image USPS avec les pixels colorés en respectant leur position 2D ; (centre droite) couleurs transportées sur image MNIST en utilisant π v , les pixels noirs correspondent aux pixels MNIST non informatifs (toujours à 0); (droite) couleurs transportées sur image MNIST en utilisant π v avec régularisation entropique.

  Data and Incomparable Spaces in Machine Learning . . . . . . . . . . . . . . . 2 1.3 Motivating Optimal Transport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4 Outline of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

  Figure 1.1: Optimal Transport provides tools for finding correspondences between set of point and gives a geometric notion of distance between probability distributions. (left) Two sets of points in 2D. The correspondences found by OT is depicted in dashed line. (right) OT distance between two probability distributions in red and blue. The further away the mean of the red probability distribution is from the blue one the higher is the OT distance.

Figure 1

 1 Figure 1.2: We associate to a dataset (xi) i∈[[n]] a probability measure describing the dataset. In this example xi ∈ R 2 . (left) Lagragian formulation (points clouds): n i=1 aiδx i is a discrete probability measure where (ai) i∈[[n]] is a probability vector, i.e. ai ≥ 0 and n i=1 ai = 1 and δ is the dirac measure δx i (x) = 1 if x = xi else 0. (right) Eulerian formulation (histograms): N i=1 aiδ xi where (xi) i∈[[N ]] is a regular grid on R 2 and (ai) i∈[[N ]] is a probability vector.

Figure 2

 2 Figure 2.1: Push-forward between two discrete probability measures µ and ν in three scenarii. (left) µ is supported on x1,x2 with corresponding weights 1 2 , 1 2 and ν on y1 with weight 1. The only possible push-forward T is T (x1) = y1 and T (x2) = y1. (center) In this situation there is no push-forward of µ onto ν because no function can satisfies T (x1) = y1 and T (x1) = y2 when y1 = y2. The problem of Monge admits no solution in this case.(right) All points are equidistant form each other that is c(xi, yj) = 1 for i, j ∈ [[2]] 2 . In this case the solution of problem MP is not unique it may associate x1 whether with y1 or y2 with the same overall cost (same with x2).

Figure 2

 2 Figure 2.2: The bakery/cafés problem in Manhattan based on example 2.1.2. The bakeries (resp cafés) are in blue (resp red) and the dots' size denote the amount of bread disposable (resp needed). Lines represent the amount of bread transferred dπ(x, y): the wider the larger. The optimal transport plan is depicted in the Figure and represents the cheapest transport plan so that all breads are transferred from bakeries to cafés.

Figure 2

 2 Figure2.4: OT for 1D probability measures can be computed using simple sorts. (left) Optimal coupling between two discrete probability measures with uniform weights. When the points are sorted it associates first point of the source with the first point of the target and so on. (right) Generic case: the optimal coupling associates horizontally the points w.r.t. the cumulative distributions of the probability measures. In this example x is associated with y.

Figure 2 . 5 :

 25 Figure 2.5: Linear displacement interpolation between two discrete probability measure µ, ν on R 2 using the linear map T defined in 2.1.8. The figure shows the displacement interpolant which is a probability measure is defined by ((1 -t)id + T )#µ for t ∈ [0, 1] [McCann 1997] (see Section 2.1.6 for more details)

Figure 2 . 6 :

 26 Figure 2.6: An assignment of x1, x2, x3, x4 to y1, y2, y3, y4 can be described whether by a permutation σ or using assignment matrix πσ. In this example both correspond to the permutation σ:1 → 3,2 → 2,3 → 4, 4 → 1

  Figure 2.7: Effect of the entropic regularization parameter ε on the optimal coupling π * ε between two 1D probability distributions. As ε increases the coupling tends to blur and converges to the marginals' product coupling.

Figure 2 .

 2 Figure 2.13: GW barycenter of 4 apple shapes in blue. The barycenters are depicted in the right side of the Figure in red. The first barycenter uses the FW algorithm for solving the GW problems and the second is computed using entropic regularization solved with Algorithm 3. Note that the barycenter are arbitrarily rotated due to the MDS procedure.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.2 Structured data as probability measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3 Fused Gromov-Wasserstein approach for structured data . . . . . . . . . . . . . . . . . . . 45 3.3.1 Fused Gromov-Wasserstein distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.3.2 Fused Gromov-Wasserstein barycenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.3.3 Optimization and algorithmic solution . . . . . . . . . . . . . . . . . . . . . . . . . . of FGW on trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.4.2 Illustration of FGW on 1D distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.4.3 Illustration of FGW on simple images . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.4.4 Illustration of FGW on time series data . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.4.5 Graph-structured data classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.4.6 Graph barycenter and compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.4.7 Unsupervised learning: graphs clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.4.8 Other applications of Fused Gromov-Wasserstein . . . . . . . . . . . . . . . . . . . . . . 57 3.5 FGW in the continuous setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.5.1 Comparing structured objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 3.5.2 Topology of the structured object space . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.5.3 Can we adapt W and GW distances for structured objects? . . . . . . . . . . . . . . . . 63 3.5.4 Convergence of finite samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.5.5 Interpolation properties between W and GW distances . . . . . . . . . . . . . . . . . . . 65 3.5.6 Geodesic properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 3 :

 33 Figure 3.3: Example of F GW , GW and W on synthetic trees. Dark grey color represents a non null πi,j value between two nodes i and j. (left) the W distance between the features with α = 0, (middle) F GW (right) the GW between the structures α = 1.

  Figure 3.4: Illustration of the difference between W , GW and F GW couplings. (left) empirical distributions µ with 20 samples and ν with 30 samples which color is proportional to their index. (middle) Cost matrices in the feature (MAB) and structure domains (C1, C2) with similar samples in white. (right) Solution for all methods. Dark blue indicates a non zero coefficient of the transportation map. Feature distances are large between points laying on the diagonal of MAB such that Wasserstein maps is anti-diagonal but unstructured. Fused Gromov-Wasserstein incorporates both feature and structure maps in a single transport map.

Figure 3

 3 Figure 3.5: Couplings obtained when considering (top left) the features only, where we have W = 0 (top right) the structure only, with GW = 0 (bottom left and right) both the features and the structure, with F GW0.1,2.For readibility issues, only the couplings starting from non white pixels on the left picture are depicted.

Figure 3 . 6 :

 36 Figure 3.6: Behavior of trade-off parameter α on a toy time series classification problem. α is increasing from left (α = 0 : Wasserstein distance) to right (α = 1 : Gromov-Wasserstein distance). (top row) 2D-embedding is computed from the set of pairwise distances between samples with MDS (bottom row) illustration of couplings between two sample time series from opposite classes.

Figure 3 . 7 :

 37 Figure 3.7: Illustration of F GW graph barycenter. The first column illustrates the original settings with the noiseless graphs, and columns 2 to 5 are the noisy samples that constitute the datasets. Column 6 show the barycenters for each setting, with different number of nodes. Blue nodes indicates a feature value close to -1, yellow nodes close to 1.

Figure 3 . 8 :

 38 Figure 3.8: Example of community clustering on graphs using F GW . (top) Community clustering with 4 communities and uniform features per cluster. (bottom) Community clustering with 4 communities and bimodal features per cluster (and two nodes per cluster in the approximate graph).

Figure 3

 3 Figure 3.9: (left) Examples from the clustering dataset, color indicates the labels. (right) Evolution of the centroids of each cluster in the k-means clustering, from the random initialization until convergence to the final centroid.

Figure 3 .

 3 Figure 3.11: Illustration of definition 3.5.7. The figure shows two structured objects (X ×Ω, dX , µ) and (Y ×Ω, dY , µ).The feature space Ω is the common space for all features. The two metric spaces (X , dX ) and (Y, dY ) represent the structures of the two structured objects, the similarity between all pair-to-pair distances of the structure points is measured by L(x, y, x , y ). µ and ν are the joint measures on the structure space and the feature space.

  22) • Let us suppose that the structure spaces (X , d X ),(Y, d Y ) are part of a single ground space (Z, d Z ) (i.e. X , Y ⊂ Z and d X = d Y = d Z ). We consider the Wasserstein distance between µ and ν for the distance on Z × Ω : d((x, a), (y, b)) = (1 -α)d(a, b) + αd Z (x, y). Then:

Adapting

  Wasserstein: common structure space If the structure spaces (X , d X ) and (Y, d Y ) are part of a same ground space (Z, d Z ) one can build a distance d = d Z ⊕ d between couples (x, a) and

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 From 1D GW to Sliced Gromov-Wasserstein . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.3 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.4 Discussion and conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Regularity & formulations of GW problems in Euclidean spaces . . . . . . . . . . . . . . . 4.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 The inner product case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 The squared Euclidean case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.4 Optimization and numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.5 The Gromov-Monge problem in Euclidean spaces . . . . . . . . . . . . . . . . . . . . . . 4.3 Conclusion: perspectives and open questions . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 2 :

 42 Figure 4.2: Illustration of SGW , RISGW and GW on spiral dataset for varying rotations on discrete 2D spiral dataset. (left) Examples of spiral distributions for source and target with different rotations. (right) Average value of SGW , GW and RISGW with L = 20 as a function of rotation angle of the target. Colored areas correspond to the 20% and 80% percentiles.

Figure 4

 4 Figure 4.3: Runtimes comparison between SGW , GW , entropic-GW between two 2D random distributions with varying number of points from 0 to 10 6 in log-log scale. The time includes the calculation of the pair-to-pair distances.

Figure 4 . 4 :

 44 Figure 4.4: Each sample in this Figure corresponds to a mesh and is colored by the corresponding time iteration. One can see that the cyclical nature of the motion is recovered.

Figure 4 . 5 :

 45 Figure 4.5: Using SGW in a GAN loss. First image shows the loss value along epochs. The next 4 images are produced by sampling the generated distribution (3, 000 samples, plotted as a continuous density map). Last image shows the target 3D distribution.

Figure 4 . 6 :

 46 Figure 4.6: Using SGW in a GAN loss. The three rows depicts three different examples. First row is 2D (Generator) to 2D (Target) , Second 3D to 2D. First column is initialization, second one is at 100 Epochs, third one at 1000. Last column depicts the target distribution.

=

  ˆtr(x T P T Px)dµ(x) * * = ˆtr(P T Pxx T )dµ(x) * * * = tr(P T P ˆxx T dµ(x)) = tr(P T P) = P 2 F = p (4.16)

  q dπ(x, y) = ˆyT Pxdπ(x, y) * = ˆtr(y T Px)dπ(x, y) * * = ˆtr(Pxy T )dπ(x, y) * * * = tr(P ˆxy T dπ(x, y)) = ˆyx T dπ(x, y), P F

  ˆ x, y p dπ(x, y) ≤ ˆu * (x) + u(y)dπ(x, y) = ˆu * (x)dµ(x) + ˆu(y)dν(y) (4.20) Using that ∇u pushes µ forward to ν implies: ˆ x, y p dπ(x, y) ≤ ˆu * (x)dµ(x) + ˆu(∇u(x))dµ(x) convex function u * (x) + u(x) = x, ∇u(x) . Overall ´ x, y p dπ(x, y) ≤

  As seen in Theorem 4.2.1 computing (innerGW) can be achieved by solving sup π∈Π(a,b) sup P F = √ p XP T Y T , π F or equivalently: min π∈Π(a,b)

  Numerical solution for (sqGW) We note x = Diag(XX T ) ∈ R n , y = Diag(YY T ) ∈ R m . As seen in Theorem 4.2.5 computing (sqGW) can be achieved by first substracting the mean of the measures and then by solving: min π∈Π(a,b)

m 4 :Figure 4

 44 Figure4.7: Runtimes comparison of GW using the BCD approach Algorithm 8. (GW BCD), the Frank-Wolfe approach (GW FW) and entropic-GW between two random distributions whose source points vary from 10 to 30000 in log-log scale. The time does not include the calculation of the pair-to-pair distances but only the time of the different loops. The variance is computed among the different 10 realizations.

Figure 4 . 8 :

 48 Figure 4.8: Comparison of the FW solver and the BCD solver for computing the Gromov-Wasserstein distance. (left) cX , cY are squared Euclidean distances. (right) cX , cY are inner products. The x-axis is the GW cost obtained by the BCD approach and the y-axis the GW cost obtained with the FW algorithm for respectively (top) n = 2000 samples (bottom) n = 500 samples in each distribution. The dashed line represents the diagonal y = x.

Figure 4 . 9 :

 49 Figure 4.9: Example of linear Gromov-Monge mapping estimation between empirical distributions. (left) 2D source and target distributions. (middle) resulting linear mapping of Wasserstein AW . (right) resulting linear mapping using the linear Gromov-Monge mapping A. Note that in this case the mapped samples are arbitrary rotated.

Figure 4

 4 Figure 4.10: Example of linear Gromov-Monge mapping estimation between two empirical distributions. (left) 3D source and 2D target distributions. (right) resulting linear mapping using the linear Gromov-Monge mapping A.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 5.2 CO-Optimal transport optimization problem . . . . . . . . . . . . . . . . . . . . . . . . . . 103 5.2.1 Illustration of COOT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . -Wasserstein . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.5.2 Relation with Invariant OT and hierarchical approaches . . . . . . . . . . . . . . . . . . . 108 5.5.3 Election isomorphism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . domain adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.6.2 Co-clustering and data summarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Figure 5 . 1 :

 51 Figure 5.1: Illustration of COOT between MNIST and USPS datasets. (left) samples from MNIST and USPS data sets; (center left) Transport matrix π s between samples sorted by class; (center) USPS image with pixels colored w.r.t.. their 2D position; (center right) transported colors on MNIST image using π v , black pixels correspond to non-informative MNIST pixels always at 0; (right) transported colors on MNIST image using π v with entropic regularization.

  Figure 5.2: Linear mapping from USPS to MNIST using π v . (First row) Original USPS samples, (Second row) Samples resized to target resolution, (Third row) Samples mapped using π v , (Fourth row) Samples mapped using π v with entropic regularization.

Figure 5

 5 Figure 5.4: Comparison between the coupling matrices obtained via GW and COOT on MNIST-USPS.

Figure 5

 5 Figure 5.5: Co-clustering with COOT on the Olivetti faces dataset. (left) Example images from the dataset, (center) centroids estimated by COOT (right) clustering of the pixels estimated by COOT where each color represents a cluster.

  1994) Police Story 4: Project S (Chao ji ji hua) (1993) Schindler's List (1993) Eye of Vichy, The (Oeil de Vichy, L') (1993) Casablanca(1942)

5 )•

 5 Let us suppose that the structure spaces (X , d X ),(Y, d Y ) are part of a single ground space (Z, d Z ) (i.e. X , Y ⊂ Z and d X = d Y = d Z ). We consider the Wasserstein distance between µ and ν for the distance on Z × Ω : d((x, a), (y, b)) = (1 -α)d(a, b) + αd Z (x, y). Then:

  a), (y, b))dπ((x , a ), (y , b )) α)d(a, b) + αdZ (x, y) + αdZ (x , y ) p dπ((x, a), (y, b))dπ((x , a ), (y , b )) α)d(a, b) + αdZ (x, y) + (1-α)d(a , b ) + αdZ (x , y ) p dπ((x, a), (y, b))dπ((x , a ), (y , b )) α)d(a, b) + αdZ (x, y) p dπ((x, a), (y, b)) 1 p

  s.c. on Π(µ, ν). If we consider W = X × Ω × Y × Ω which is a a metric space endowed with the distance d X ⊗ d ⊗ d Y ⊗ d and f ((w = (x, a, y, b), w = (x , a , y , b )) = ((1 -α)d(a, b) q + αL(x, y, x , y ) q ) p then f is l.s.c. by continuity of d, d X and d Y . With the previous reasoning we can conclude that the infimum is attained. Finally finiteness come from: α)d(a, b) q + αL(x, y, x , y ) q p dπ((x, a), (y, b)) dπ((x , a ), (y , b )) * ≤ ˆ2p-1 (1 -α)d(a, b) qp dµ A (a)dν(b) + ˆ2p-1 αL(x, y, x , y ) qp dµ X (x)dµ X (x )dν Y (y )dν Y (y )

  a ) and d Y×Ω (y, b), (y , b ) = (1 -β)d Y (y, y ) + βd(b, b )We will prove that d GW,p (d X ×Ω , d Y×Ω , µ, ν) = 0. To show that we will bound the Gromov cost with the metrics d X ×Ω , d Y×Ω by the Gromov cost with the metrics d X , d Y and a Wasserstein cost.Let π ∈ Π(µ, ν) be any admissible transportation plan. Then for n ≥ 1:Jn(d X ×Ω , d Y×Ω , π) def = (X×Ω×Y×Ω) 2 L(x,y, x , y ) n dπ((x, a), (y, b))dπ((x , a ), (y , b )) = (X×Ω×Y×Ω) 2 |(1 -β)(dX (x, x ) -dY (y, y )) + β(d(a, a ) -d(b, b ))| n dπ((x, a), (y, b))dπ((x , a ), (y , b )) ≤ (X×Ω×Y×Ω) 2 (1 -β)|dX (x, x ) -dY (y, y )| n dπ((x, a), (y, b)) dπ((x , a ), (y , b )) + (X×Ω×Y×Ω) 2 β|d(a, a ) -d(b, b )| n dπ((x, a), (y, b))dπ((x , a ), (y , b )) using Jensen inequality with convexity of t → t n and subadditivity of |.| . We note ( * ) the first term above and ( * * ) the second term above. By the triangle inequality property of d we have: a, b) + d(a , b ) n dπ((x, a), (y, b)) dπ((x , a ), (y , b )) def = βM n (π) such that we have shown:

  b) qp dπ * ((x, a), (y, b))=0 which implies that d is zero π * a.e. so that (X ×Ω×Y×Ω) d(a, b) m dπ * ((x, a), (y, b)) = 0 for any m ∈ N * . In this way:M qp (π * ) = β b) h d(a , b ) qp-h dπ * ((x, a), (y, b))dπ * ((x , a ), (y , b )) , b) h dπ * ((x, a), (y, b)) , b ) qp-h dπ * ((x , a ), (y , b )) = 0Using equation (6.14) we have shownJ qp (d X ×Ω , d Y×Ω , π * ) = 0 which implies that d GW,p (d X ×Ω , d Y×Ω , µ, ν) = 0 for the coupling π * .Thanks to the Gromov-Wasserstein properties (see Chapter 2) this states the existence of an isometry between supp(µ) and supp(ν). So there exists a surjective function φ = (φ 1 , φ 2 ) : supp(µ) → supp(ν) which verifies P.1 and: ∀((x, a), (x , a )) ∈ (supp(µ)) 2 , d X ×Ω((x, a), (x , a )) = d Y×Ω (φ(x, a), φ(x , a ))(6.17) or equivalently:∀((x, a), (x , a )) ∈ (supp(µ)) 2 , (1-β)dX (x, x )+βd(a, a ) = (1-β)dY (φ1(x, a), φ1(x , a ))+βd(φ2(x, a), φ2(x , a ))(6.18)

  .22) d X (x, x ) = d Y (I(x), I(x )) for (x, x ) ∈ supp(µ X ) 2 . Overall we have φ(x, a) = (I(x), a) for all (x, a) ∈ supp(µ). Also since φ#µ = ν we have I#µ X = ν Y .Moreover I is a surjective function. Indeed let y ∈ supp(ν Y ). Let b ∈ supp(ν B ) such that (y, b) ∈ supp(ν). By surjectivity of φ there exists (x, a) ∈ supp(µ) such that (y, b) = φ(x, a) = (I(x), a) so that y = I(x).

  ν) and P 2,3 #ρ = π OT . Moreover we have: Ω×Ω d(a, b) pq dπ OT (a, b) = X ×Ω×Ω×Y d(a, b) pq dρ(x, a, b, y) (6.23)

Lemma 6 . 2 . 4 .

 624 Let x, y ∈ D fixed. There exists ε 0 ∈ R n such that: arg max σ∈Sn g(x + ε 0 , y, σ) ⊂ arg max σ∈Sn g(x, y, σ) arg max σ∈Sn g(x + ε 0 , y, σ) ⊂ {Id, anti -Id} (6.40) Proof. Let x, y ∈ D. We consider ε 0 = (ζ, -ζ, 0, . . . , 0) with ζ > 0 small enough to ensure ζ < x2-x1 2 and ε 0 < ε x,y defined in Lemma 6.2.3. We have x + ε 0 , y ∈ D since i

  .55) Moreover we have by definition of the Fourier transform, for s ∈ R P ,F fn#µ (s) = ˆe-2iπ s,fn(x) dµ(x) = ˆe-2iπ p k=1 s k T e k (n) ( x,e k (n) ) dµ(x)Moreover using (Q θ ) we have F T e k (n) #(P e k (n) #µ) (t) = F P e k (n) #ν (t) for all k ∈ {1, ..., p}, and any real t ∈ R.

Figure 6 . 1 :

 61 Figure 6.1: Illustration of SW , RISW on spiral datasets for varying rotations on discrete 2D spiral datasets. (left) Examples of spiral distributions for source and target with different rotations. (right) Average value of SW and RISW with L = 20 as a function of rotation angle of the target. Colored areas correspond to the 20% and 80% percentiles.

  x p y, y q dπ(x, y)dπ(x , y ) + (a + b)c ˆ x 2 2 ˆydπ(x, y), y q dπ(x, y) + (a + b)c ˆ y 2 2 ˆxdπ(x, y), x p dπ(x, y) Moreover: ˆ x, x p y, y q dπ(x, y)dπ(x , y ) = ˆxT x y T y dπ(x, y)dπ(x , y ) = ˆtr(x T x y T y )dπ(x, y)dπ(x , y ) = ˆtr(y x T x y T )dπ(x, y)dπ(x , y ) * = ˆtr(y x T xy T )dπ(x, y)dπ(x , y ) = tr(( ˆy x T dπ(x , y )( ˆxy T dπ(x, y)))) = ˆxy T dπ(x, y) 2 F = ˆyx T dπ(x, y) 2 F

Lemma.

  The overall computational complexity of computing the value of COOT whenL = |.| 2 is O(min{(n + n )dd + n 2 n; (d + d )nn + d 2 d}).

  x,x s.t Ax=b,x≥0 g(x, x) = f (z * ). To prove the theorem, it suffices to prove thatf (y * ) = f (x * ) = g(x * , y * ) (6.103) since, in this case, g(x * , y * ) = f (x * ) ≥ f (z * ) and g(x * , y * ) = f (y * ) ≥ f (z * ).Let us prove (6.103). Since (x * , y * ) is optimal, we have:0 ≤ g(x * , y * ) -g(x * , x * ) = 1 2 c T (y * -x * ) + 1 2 x * T Q(y * -x * ) 0 ≤ g(x * , y * ) -g(y * , y * ) = 1 2 c T (x * -y * ) + 1 2 y * T Q(x * -y * ).

6. 3 . 4

 34 Proofs of Proposition 5.5.1 -Concavity of GWWe recall the proposition:Proposition. Let L = | • | 2 and suppose that C ∈ R n×n , C ∈ R n ×n are squared Euclidean distance matrices such that C = x1 T n + 1 n x T -2XX T , C = x 1 T n + 1 n x T -2X X T with x = diag(XX T ), x = diag(X X T ).Then, the GW problem can be written as a concave quadratic program (QP) which Hessian readsQ = -4 * XX T ⊗ K X X T . If C ∈ R n×n , C ∈ R n ×n are inner products similarities, i.e. such that C = XX T , C = X X T then the GW is also a concave quadratic program (QP) which Hessian reads Q = -2 * XX T ⊗ K X X T .This result is a consequence of the following lemma. Lemma 6.3.1. When C, C are squared Euclidean distance matrices as defined previously, the GW problem can be formulated as:GW (C, C , w, w ) = min π s ∈Π(w,w ) -4vec(M) T vec(π s ) -8vec(π s ) T Qvec(π s ) + Cte with M = xx T -2xw T X X T -2XX T wx T and Q = XX T ⊗ K X X T , w j -4w T xw T xWhen C, C are inner product similarities as defined in Proposition 5.5.1, the GW problem can be formulated as:GW (C, C , w, w ) = min π s ∈Π(w,w ) -4vec(π s ) T Qvec(π s ) + Cte Proof. Using the results in [Peyré 2016] for L = | • | 2 , we have L(C, C ) ⊗ π s = c C,C -2Cπ s C with c C,C = (C) 2 w1 T n + 1 n w T (C ) 2 , where (C) 2 = (C 2 i,j) is applied element-wise. We now have that:Cπ s C , π s = tr π sT (x1 T n + 1 n x T -2XX T )π s (x 1 T n + 1 n x T -2X X T ) = tr (π sT x1 T n + w x T -2π sT XX T )(π s x 1 T n + wx T -2π s X X T ) = tr π sT xw T x 1 T n + π sT xx T -2π sT xw T X X T + w x T π s x 1 T n + w x T wx T -2w x T π s X X T -2π sT XX T π s x 1 T n -2π sT XX T wx T + 4π sT XX T π s X X T * = tr π sT xw T (x 1 T n + 1 n x T ) + π sT xx T + w x T wx T -2π sT xw T X X T -2w x T π s X X T -2π sT XX T π s x 1 T n -2π sT XX T wx T + 4π sT XX T π s X X T ,where in (*) we used:tr(w x T π s x 1 T n ) = tr(x 1 T n w x T π s ) = tr(π sT xw T 1 n x T ).Moreover, since:tr(π sT XX T π s x 1 T n ) = tr(1 T n π sT XX T π s x ) = tr(w T XX T π s x ) = tr(π sT XX T wx T )and tr(w x T π s X X T ) = tr(π sT xw T X X T ), we can simplify the last expression to obtain:Cπ s C , π s = tr π sT xw T (x 1 T n + 1 n x T ) + π sT xx T + w x T wx T -4π sT xw T X X T -4π sT XX T wx T + 4π sT XX T π s X X T .Finally, we have thatCπ s C , π s = tr π sT xw T x 1 T n + π sT xw T 1 n x T + π sT xx T + w x T wx T -4π sT xw T X X T -4π sT XX T wx T + 4π sT XX T π s X X T = tr 2w x T wx T + 2π sT xx T -4π sT xw T X X T -4π sT XX T wx T + 4π sT XX T π s X X T = 2w T xw T x + 2 xx T -2xw T X X T -2XX T wx T , π s + 4tr(π sT XX T π s X X T ).The term 2w T xw T x is constant since it does not depend on the coupling. Also, we can verify that c C,C does not depend on π s as follows: -2Cπ s C , π s = Cte -4 xx T -2xw T X X T -2X T Xwx T , π s -8tr(π sT XX T π s X X T ).

  (FW) based algorithm presented in Chapter 3 and recalled in Algorithm 11 when L = | • | 2 and for squared Euclidean distance matrices C , C . Algorithm 11 FW Algorithm for GW (see Chapter 3) 1: Input: maxIt, thd 2: π (0) ← ww 3: while k < maxIt and err > thd do 4: G ← Gradient from equation (5.2) w.r.t. π s (k-1) 5:πs (k) ← OT (w, w , G) 6: z k (τ ) ← π s (k-1) + τ ( πs (k) -π s (k-1) ) for τ ∈ (0← (1 -τ (k) )π s (k-1) + τ (k) πs (k)

1:(

  Input: maxIt, thd 2: π (0) ← ww 3: while k < maxIt and err > thd do 4: G ← Gradient from equation (5.2) w.r.t. π s noticing that in the step 3 of Algorithm 12 the gradient of (5.2) w.r.t π s (k-1) is 2L(C, C ) ⊗ π s

  [n]. The authors further propose a distance underlying this problem defined as follows:d-ID(E, E ) = min ν∈Sn min σ∈Π(C,C ) n i=1 d σ(v i ), v ν(i) ,where S n denotes the set of all permutations over {1, . . . , n}, Π(C, C ) is a set of bijections and d is an arbitrary distance between preference orders. The authors of[Faliszewski 2019] compute d-ID(E, E ) in practice by expressing it as the following Integer Linear Programming problem over the tensorP ijkl = M ij N kl where M ∈ R m×m , N ∈ R n×n min P,N,M i,j,k,l P k,l,i,j |pos vi (c k ) -pos v j (c l )| s.t. (N1 n ) k = 1, ∀k, (N 1 n ) l = 1, ∀l (6.104) (M1 m ) i = 1, ∀i, (M 1 m ) j = 1, ∀j P kl ≤ N k,l , P i,j,k,l ≤ M i,j , ∀i, j, k, l i,k P i,j,k,l = 1, ∀j, l (6.105)where pos vi (c k ) denotes the position of candidate c k in the preference order of voter v i . Let us now define two matrices X and X such that X i,k = pos vi (c k ) and X j,l = pos v j (c l ) and denote by π s * , π v * a minimizer of COOT(X, X , 1 n /n, 1 n /n, 1 m /m, 1 m /m) with L = | • | and by N * , M * the minimizers of problem (6.104), respectively.

Contents 7 . 1

 71 Overview of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 7.2 Perspective for further works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

Titre:

  Une contribution au Transport Optimal sur des espaces incomparables Mot clés : Transport Optimal, Données Structurées, Wassertein et Gromov-Wasserstein

Contents 2.1 Linear Optimal Transport theory

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1 The main theorem of the linear Optimal Transport theory . . . . . . . . . . . . . . . . . . 12 2.1.2 Special cases: 1D transportation and transport between Gaussians . . . . . . . . . . . . . 16 2.1.3 Some statistical aspects of OT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.1.4 A quick numerical tour: solving Optimal Transport . . . . . . . . . . . . . . . . . . . . . 19 2.1.5 Other formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.1.6 Wasserstein barycenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.2 The Gromov-Wasserstein problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.2.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.2.2 Properties of GW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.2.3 Solving GW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.2.4 Gromov-Wasserstein Barycenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Chapter 2. Generality about optimal transport min a∈Σ k ,X∈R k×d n i=1 λ i min π∈Π(a,bi)

  Instead of looking at all possible discrete probability measures we can search a k atoms probability measure i.e. of the form μ = k p=1 a p δ xp where X = (x p ) p∈[[k]] ∈ R k×d and a ∈ Σ k . Overall the resulting problem is:

  When it is clear form the context we will simply note GW p (µ, ν) instead of GW p (c X , c Y , µ, ν). Since X , Y are already endowed with a natural metric one choice would be to consider c X = d X and c Y = d Y . This setting brings in light the notion of metric measure spaces, as triplets of the form (X , d X , µ) where (X , d X ) is a complete separable metric space and µ is a Borel probability measure on X . It was studied

	y ) for p ∈ N * (see Figure 2.8). GW depends on the choice of similarities c X , c Y between points in X and p dπ(x, y)dπ(x , y )) 1 p (2.41) not necessarily share a common metric. It is built upon the similarities cX , cY within each space and on a measure Y. Figure 2.8: The GW problem considers two probability measures µ ∈ P(X ), ν ∈ P(Y) over two spaces that do of the distortion between each pair of points cX (x, x ) -cY (y, y ) .

  Update right scaling

	10:	end for
	11: end for
	12:

Table 3 .

 3 1: Average classification accuracy on the graph datasets with vector attributes.

	Vector attributes BZR	COX2	CUNEIFORM ENZYMES PROTEIN	SYNTHETIC
	FGW	85.12±4.15* 77.23±4.86 76.67±7.04	71.00±6.76	74.55±2.74 100.00±0.00
	HOPPERK	84.15±5.26	79.57±3.46 32.59±8.73	45.33±4.00	71.96±3.22 90.67±4.67
	PROPAK	79.51±5.02	77.66±3.95 12.59±6.67	71.67±5.63* 61.34±4.38 64.67±6.70
	PSCN k=10	80.00±4.47	71.70±3.57 25.19±7.73	26.67±4.77	67.95±11.28 100.00±0.00
	PSCN k=5	82.20±4.23	71.91±3.40 24.81±7.23	27.33±4.16	71.79±3.39 100.00±0.00

Table 3 .

 3 2: Average classification accuracy on the graph datasets with discrete attributes.

	Discrete attr. MUTAG	NCI1	PTC-MR
	FGW raw	83.26±10.30 72.82±1.46 55.71±6.74
	FGW wl h=2	86.42±7.81 85.82±1.16 63.20±7.68
	FGW wl h=4	88.42±5.67 86.42±1.63 65.31±7.90
	GK k=3	82.42±8.40 60.78±2.48 56.46±8.03
	RWK	79.47±8.17 58.63±2.44 55.09±7.34
	SPK	82.95±8.19 74.26±1.53 60.05±7.39
	WLK	86.21±8.48 85.77±1.07 62.86±7.23
	WLK h=2	86.21±8.15 81.85±2.28 61.60±8.14
	WLK h=4	83.68±9.13 85.13±1.61 62.17±7.80
	PSCN k=10	83.47±10.26 70.65±2.58 58.34±7.71
	PSCN k=5	83.05±10.80 69.85±1.79 55.37±8.28
	Without attribute IMDB-B	IMDB-M
	GW		63.80±3.49 48.00±3.22
	GK k=3		56.00±3.61 41.13±4.68
	SPK		55.80±2.93 38.93±5.12

Table 3.3: Average classification accuracy on the graph datasets with no attributes.

  Thomas Hudson had told him. "But it s rugged from now on until spring and spring isn't easy." "I want it to be rugged," Roger had said. "I am going to start new again." "How many time it is now you've started new?" "Too many," Roger had said. "And you don't have to rub it in." -Ernest Hemingway, Islands in the Stream

  Interestingly enough the minimization of π with P fixed is a linear transportation problem with ground cost -XP T Y T which can be computed using standard solvers (see Chapter 2).The minimization w.r.t. P with π fixed reads sup P F = √ p XP T Y T , π F and has a closed-form solution based on Lemma 4.2.2. More precisely it reads:

	7: end while
	8: return (π, P)

maximize (MaxOT) w.r.t. P) 6: π ← arg min π∈Π(a,b) -XP T Y T , π F // (maximize (MaxOT) w.r.t. π ∼ linear OT)

Table 5

 5 ].

	Domains No-adaptation baseline	CCA	KCCA	EGW	SGW	COOT
	C→W	69.12±4.82	11.47±3.78 66.76±4.40 11.35±1.93 78.88±3.90 83.47±2.60
	W→C	83.00±3.95	19.59±7.71 76.76±4.70 11.00±1.05 92.41±2.18 93.65±1.80
	W→W	82.18±3.63	14.76±3.15 78.94±3.94 10.18±1.64 93.12±3.14 93.94±1.84
	W→A	84.29±3.35	17.00±12.41 78.94±6.13 7.24±2.78 93.41±2.18 94.71±1.49
	A→C	83.71±1.82	15.29±3.88 76.35±4.07 9.82±1.37 80.53±6.80 89.53±2.34
	A→W	81.88±3.69	12.59±2.92 81.41±3.93 12.65±1.21 87.18±5.23 92.06±1.73
	A→A	84.18±3.45	13.88±2.88 80.65±3.03 14.29±4.23 82.76±6.63 92.12±1.79
	C→C	67.47±3.72	13.59±4.33 60.76±4.38 11.71±1.91 77.59±4.90 83.35±2.31
	C→A	66.18±4.47	13.71±6.15 63.35±4.32 11.82±2.58 75.94±5.58 82.41±2.79
	Mean	78.00±7.43	14.65±2.29 73.77±7.47 11.12±1.86 84.65±6.62 89.47±4.74
	p-value	<.001	<.001	<.001	<.001	<.001	-

.1: Semi-supervised HDA for nt = 3 from Decaf to GoogleNet task.

Table 5 .

 5 2: Unsupervised HDA for nt = 0 from Decaf to GoogleNet task.

	Domains	CCA	KCCA	EGW	COOT
	C→W	14.20±8.60 21.30±15.64 10.55±1.97 25.50±11.76
	W→C	13.35±3.70	18.60±9.44 10.60±0.94 35.40±14.61
	W→W	10.95±2.36	13.25±6.34 10.25±2.26 37.10±14.57
	W→A	14.25±8.14 23.00±22.95 9.50±2.47 34.25±13.03
	A→C	11.40±3.23	11.50±9.23 11.35±1.38 17.40±8.86
	A→W	19.65±17.85 28.35±26.13 11.60±1.30 30.95±18.19
	A→A	11.75±1.82	14.20±4.78 13.10±2.35 42.85±17.65
	C→C	12.00±4.69	14.95±6.79 12.90±1.46 42.85±18.44
	C→A	15.35±6.30 23.35±17.61 12.95±2.63 33.25±15.93
	Mean	13.66±2.55	18.72±5.33 11.42±1.24 33.28±7.61
	p-value	<.001	<.001	<.001	-

Table 5 .

 5 Table 5.3 below summarizes the characteristics of the simulated data sets used in our experiment. .003 .042 ± .037 .025 ± .048 .082 ± .063 .021 ± .011 .021 ± .001 .017 ± .045 .018 ± .013 .004 ± .002 0 D2 .072 ± .044 .083 ± .063 .038 ± .000 .052 ± .065 .032 ± .041 .047 ± .042 .039 ± .052 .023 ± .036 .011 ± .056 .047 .121 ± .075 .102 ± .071 .093 ± .032 .079 ± .031 0.068 ± 0.04

	Data set	K-means	NMF	DKM	Tri-NMF	Algorithms GLBM ITCC	RBC	CCOT	CCOT-GW	COOT
	D1	.018 ± .009 ± 0.04
	D3	-	-	.310 ± .000	-	.262 ± .022 .241 ± .031	-	.031 ± .027 .008 ± .001	.04 ± .05
	D4	.126 ± .038	-	.145 ± .082	-	.115 ±

Table 6 .

 6 1: Percentage of α chosen in ]0, ..., 1[ compared to {0, 1} for discrete labeled graphs

	Discrete attr. MUTAG NCI1 PTC
	FGW raw sp	100%	100% 98%
	FGW wl h=2 sp 100%	100% 88%
	FGW wl h=4 sp 100%	100% 88%

Table 6 .

 6 2: Percentage of α chosen in ]0, ..., 1[ compared to {0, 1} for vector attributed graphs

	Vector attributes BZR COX2 CUNEIFORM ENZYMES PROTEIN SYNTHETIC
	FGW sp	100 % 90%	100%	100%	100%	100%

Table 6 .

 6 3: Average classification accuracy on the graph datasets with discrete attributes.

	Discrete attr. MUTAG	NCI1	PTC-MR
	FGW raw sp	83.26±10.30 72.82±1.46	55.71±6.74
	FGW wl h=2 sp 86.42±7.81 85.82±1.16	63.20±7.68
	FGW wl h=4 sp 88.42±5.67 86.42±1.63* 65.31±7.90
	W raw sp	79.36±3.49 70.5±4.63	54.79±5.76
	W wl h=2 sp	87.78±8.64 85.83±1.75	63.90±7.66
	W wl h=4 sp	87.15±8.23 86.42±1.64	66.28±6.95*
	GW sp	82.73±9.59 73.40±2.80	54.45± 6.89

Table 6 .

 6 4: Average classification accuracy on the graph datasets with vector attributes.

	Vector attributes BZR	COX2	CUNEIFORM ENZYMES PROTEIN	SYNTHETIC
	FGW sp	85.12±4.15	77.23±4.86* 76.67±7.04	71.00±6.76	74.55±2.74	100.00±0.00
	W	85.36±4.87* 77.23±3.16	61.48±10.23	71.16±6.32* 75.98± 1.97* 34.07±11.33
	GW sp	82.92±6.72	77.65±5.88	50.66±8.91	23.66±3.63	71.96± 2.40	41.66±4.28

Table 6 .

 6 5: Average timings for the computation of F GW between two pairs of graph

	Discrete attr. MUTAG NCI1 PTC-MR
	FGW	2.5 ms	7.3 ms 3.7 ms
	6.1.

2 Proof of Proposition 3.5.1 -Comparison between FGW, GW and W

  leads to a striclty better cost. Overall we have proven that for any permutation τ , if τ /

		∈ arg max σ∈Sn g(x, y, σ) and
	h <	min (β,σ,τ )∈(Sn) 3 ε x,y (β, σ, τ ) then τ / ∈ arg max

.39) So that g(x + h, y, σ * ) > g(x + h, y, τ ) and in this way τ / ∈ arg max σ∈Sn g(x + h, y, σ) because σ * σ∈Sn g(x + h, y, σ) which proves that arg max σ∈Sn g(x + h, y, σ) ⊂ arg max σ∈Sn g(x, y, σ).

Corollary 6.2.2. Let

µ, ν ∈ P(R p ) × P(R p ) with compact support. If SGW (µ, ν) = 0 then µ and ν are isomorphic for the distance induced by the 1 norm on R p , i.e. d(x, x ) = p i=1 |x i -x i | for Memo 6.2.2. Let (X , d) be a compact metric space and . a norm on R p . We say taht:

Table 6 . 6 :

 66 84±4.27 19.95±23.40 38.16±19.30 12.32±1.56 75.84±7.37 89.42±4.24 A→A 42.68±8.36 15.21±7.36 38.26±16.99 13.63±2.93 75.53±6.25 91.84±2.48 C→C 28.58±7.40 18.37±17.81 35.11±17.96 11.05±1.63 61.21±8.43 78.11±5.77 C→A 31.63±4.25 15.11±5.10 33.84±9.10 11.84±1.67 66.26±7.95 82.11±2.58 Semi-supervised Heterogeneous Domain Adaptation results for adaptation from Decaf to GoogleNet representations with different values of nt. 95±3.89 15.32±8.18 40.16±17.54 13.21±3.49 87.89±4.03 90.68±7.54 A→A 36.89±4.73 13.84±2.47 34.84±10.44 13.16±1.56 89.79±3.93 94.68±2.21 W→W 32.05±4.63 19.89±11.82 36.26±21.98 10.00±2.59 84.21±4.55 90.42±2.66 W→C 32.68±5.56 21.53±21.01 33.79±22.72 11.47±3.03 86.26±3.41 89.53±1.92 A→W 33.84±4.75 16.00±7.74 39.32±18.94 11.00±4.01 87.21±3.67 91.53±5.85 C→W 32.32±7.76 15.58±7.72 34.05±15.96 12.89±2.52 81.84±3.51 84.84±5.71

				Decaf → GoogleNet
	Domains	Baseline	CCA	KCCA	EGW	SGW	COOT
				n t = 1
	C→W	30.47±6.90 13.37±7.23 29.21±13.14 10.21±1.31 66.95±7.61 77.74±4.80
	W→C	26.53±7.75 16.26±5.18 40.68±12.02 10.11±0.84 80.16±4.78 87.89±2.65
	W→W	30.63±7.78 13.42±1.38	36.74±8.38	8.68±2.36	78.32±5.86 89.11±2.78
	W→A	30.21±7.51 12.47±2.99	39.11±6.85	9.42±2.90	80.00±3.24 89.05±2.84
	A→C	41.89±6.59 12.79±2.95	28.84±6.24	9.89±1.17	72.00±8.91 84.21±3.92
	A→W 39.Mean 33.61±5.77 15.22±2.44	35.55±3.98 10.80±1.47 72.92±6.37 85.50±4.89
				n t = 5
	C→W	74.27±5.53 14.53±7.37	73.27±4.99 11.40±1.13 84.00±3.99 85.53±2.67
	W→C	90.27±2.67 21.13±6.85	85.00±3.44 10.60±1.05 95.20±2.84 94.53±1.83
	W→W	90.93±2.50 15.80±3.27	90.67±2.95	9.80±2.60 95.40±2.47 94.93±2.70
	W→A	90.47±2.92 16.67±4.85	87.93±2.47	9.80±2.68	95.40±1.53 95.80±2.15
	A→C	88.33±2.33 15.73±4.64	83.13±2.84 10.40±1.89 84.47±5.81 91.47±1.45
	A→W	88.40±3.17 13.60±6.25	87.27±2.82 11.87±2.40 87.87±4.66 93.00±1.96
	A→A	86.20±3.08 14.07±2.93	87.00±3.48 14.07±1.65 89.80±2.58 92.20±1.69
	C→C	75.93±4.83 13.13±2.98	70.47±3.45 11.13±1.52 85.73±3.54 84.60±2.32
	C→A	73.47±3.62 15.47±6.50	74.13±5.42 11.20±2.47 85.07±3.26 87.20±1.78
	Mean	84.25±7.01 15.57±2.25	82.10±7.03 11.14±1.23 89.21±4.64 91.03±3.97

Table 6 .

 6 7: Semi-supervised Heterogeneous Domain Adaptation results for adaptation from GoogleNet to Decaf representations with different values of nt.

			GoogleNet → Decaf
	Domains	CCA	KCCA	EGW	COOT
	C→A	11.30±4.04	14.60±8.12	8.20±2.69 25.10±11.52
	C→C	13.35±4.32 17.75±10.16 11.90±2.99 37.20±14.07
	W→A	14.55±10.68 25.05±24.73 14.55±2.05 39.75±17.29
	A→C	13.80±6.51 20.70±17.94 16.00±2.44 30.25±18.71
	A→A	16.90±10.45 28.95±30.62 12.70±1.79 41.65±16.66
	W→W	14.50±6.72 24.05±19.35 9.55±1.77	36.85±9.20
	W→C	13.15±4.98	14.80±8.79 11.40±2.65 30.95±17.18
	A→W	10.85±4.62 14.40±12.36 12.70±2.99 40.85±16.21
	C→W	18.25±14.02 25.90±25.40 11.30±3.87 34.05±13.82
	Mean	14.07±2.25	20.69±5.22 12.03±2.23 35.18±5.24

Table 6 .8: Unsupervised Heterogeneous Domain Adaptation results

 6 for adaptation from GoogleNet to Decaf representations.ConclusionLe soleil est noyé. -C'est le soir -dans le port Le navire bercé sur ses câbles, s'endort -Tristan Corbière, Les Amours jaunes

	Chapter 7

The original quote by Monge is[START_REF] Gaspard Monge | Mémoire sur la théorie des déblais et des remblais[END_REF]]: "Lorsque le transport du deblai se fait de manière que la somme des produits des molécules par l'espace parcouru est un minimum, les routes de deux points quelconques A & B, ne doivent plus se couper entre leurs extrémités, car la somme Ab + Ba, des routes qui se coupent, est toujours plus grande que la somme Aa + Bb, de celles qui ne se coupent pas."

https://github.com/tvayer/PSCN

https://grouplens.org/datasets/movielens/100k/

Acknowledgements Chapter 7. Conclusion

This limitation originates from the method itself which discards the feature information by focusing only on pair-to-pair distance matrices. To circumvent this constraint we proposed a novel optimal transport distance which both finds the correspondences between the samples and the features of the distributions. This work is based on the CO-Optimal transport framework which computes two optimal transport plans directly on the raw data unlike GW which requires pre-computed pair-to-pair distance or similarity matrices. We showed that it is particularly suited for problems such as Heterogeneous Domain Adaptation and Co-clustering. In the light of the previous results we drew interesting connections between COOT and the GW distance the latter being a special case of the former in the concave regime and with data described by distance or similarity matrices.

Chapter 4. The Gromov-Wasserstein problem in Euclidean spaces

When p = q, an optimal linear Monge map is given by T (x) = Ax where:

so that:

LGM 2 2 (µ, ν) = 4(tr(Σ µ ) -tr(Σ ν )) 2 + 8(tr(Σ µ Σ µ ) + tr(Σ ν Σ ν )) -16tr(D µ D ν ) (4.41)

Sketch of proof. We can show that when considering only linear push-forward the problem (GM) can be recast as a Orthogonality constrained Quadratic Program (QPOC) which, when p = q, admits a close form. This is made possible thanks to the Gaussian assumption which allows to compute the 4-th order moments of the distributions using Isserlis theorem [Isserlis 1918] which prove that 4-th order moments can be computed using the 2-nd order ones. We give the full proof in Section 6.2.11.

Geometric interpretations of

LGM Interestingly enough the optimal linear map can be related, inter alia, to the optimal linear map of the classical Monge problem. In the following we consider p = q so that we have using Theorem 4.2.6:

LGM 2 2 (µ, ν) = 4(tr(Σ µ ) -tr(Σ ν )) 2 + 8(tr(Σ µ Σ µ ) + tr(Σ ν Σ ν )) -16tr(D µ D ν ) (4.42)

This implies that an optimal way for transferring the masses so that, on average, the pair-to-pair distances are preserved is by the means of the identity mapping.

If there is a scaling factor between the covariances Σ µ = kΣ ν with k > 0 then the optimal map scales uniformly the first measure to preserve the distances so that A = 1

LGM 2 (µ, ν) is minimal, equal to zero, when k = 1 which corresponds to the previous case. When k increases above 1 the distance increases quadratically and when k ∈]0, 1] the distances decreases as k goes to 1.

• Rotation. Another interesting case is when we rotate the samples of the distribution and we compute the linear Gromov-Monge between the original distribution and its rotated counterpart. This case corresponds to Σ ν = OΣ µ O T where O ∈ O(p) and we can check easily that LGM 2 (µ, ν) = 0 with an optimal map given the rotation. This behavior is intuitive since the Gromov-Monge problem with linear map is invariant by rotations.

• Commuting covariances. Finally when the covariances commute i.e. Σ µ Σ ν = Σ ν Σ µ we can relate with the linear transportation. In this situation both are simultaneously diagonalizable and eigenspaces coincide. We recall that the optimal linear map for the Wasserstein distance with

-1/2 µ when covariances are commuting. Moreover since matrices share the same eigenspaces then we can take V µ = V ν for the linear Gromov-Monge. In this way A reduces to A = Σ 1/2 ν Σ -1/2 µ = A W . This proves that when the covariances commute the optimal map of Wasserstein is an optimal map for the linear Gromov-Monge.

Summary of the contributions

This chapter is based on the paper [START_REF] Redko | [END_REF]] and addresses the problem of optimal transport on incomparable spaces. The original formulation of the optimal transport problem relies on the existence of a cost function between the samples of the two distributions, which makes it impractical for comparing data distributions supported on different topological spaces. To circumvent this limitation, we propose a novel OT problem, named COOT for CO-Optimal Transport, that aims to simultaneously optimize two transport maps between both samples and features. This is different from other approaches that either discard the individual features by focusing on pairwise distances (e.g. Gromov-Wasserstein) or need to model explicitly the relations between the features. COOT leads to interpretable correspondences between both samples and feature representations and holds metric properties. We provide a thorough theoretical analysis of our framework and establish rich connections with the Gromov-Wasserstein distance. We demonstrate its versatility with two machine learning applications in heterogeneous domain adaptation and co-clustering/data summarization, where COOT leads to performance improvements over the competing state-of-the-art methods.

Chapter 6

Proofs of claims and additional results

Contents

Proof of Lemma 4.2.4

We recall the lemma:

To prove this result we will need the following calculus: Lemma 6.2.10. With previous notations, the Fréchet derivative of F reads:

Proof.

The calulus in Lemma 6.2.10) implies that ∇F (π * ) = (x, y) → 4 V π * x, y q + x 2 2 y 2 2 . Setting P = 4V π * ∈ R q×p concludes for the first point.

For the second point we have by definition F * (h) = sup π∈M(X ×Y) ´h(x, y)dπ(x, y) -F (π). We note

by Lemma 6.2.10. Using the fact that h is parametrized by a linear application we have ∇G(π) = (x, y) → (P -4V π )x, y q . Then for (x, y) ∈ X × Y: ∇G(π)(x, y) = 0 ⇐⇒ (P -4V π )x, y q = 0 (6.72)

We write P = i λ i v i u T i the SVD of P. We note γ P ∈ M(X × Y) the measure γ P = 1 4 i λ i δ (ui,vi) (note that we do not need that γ P be a probability measure). Then ´yx T dγ P (x, y) = 1 4 i λ i v i u T i = 1 4 P. By previous calculus ∇G(γ P ) = 0 so that γ P satisfies the first order condition and is a solution to sup π∈M(X ×Y) ´h(x, y)dπ(x, y) -F (π). Overall:

Characterization of H(µ, ν) for linear push-forward

We have the following result: Lemma 6.2.11. Let µ, ν ∈ P(R p ) × P(R p ). If T is a linear symmetric push-forward of µ to ν then:

where O ∈ O(p) is an orthogonal matrix and λ > 0.

Proof. We note T (x) = Ax. Then we have Ax 2 2 = x T A T Ax, we can write A T A = PDP T where D diagonal and P orthogonal, then it is equivalent to find a g = f such that:

2 ) (6.75)

(with β i ≥ 0 and we note v = P T x). If there is i such that β i = 0 (the first one without loss of generality) then for all t ≥ 0 we can take v = ( √ t, 0, .., 0) then g( v 2 2 ) = g(t) = 0 which is not possible unless A = 0. In this way β i > 0 for all i. Let e i be one eigenvector of A T A then

= λ for all i. So A T A = λ 0 I p . Overall it implies that A can be written as A = λO with O an orthogonal matrix. Conversely if T = λO then

We can take the function f (t) = 1 2 c 2 t 2 which is convex and continuous from R to R.

Proof of Theorem 4.2.6 -Closed form expression of the linear Gromov-Monge problem between Gaussian measures

We recall the theorem:

be the diagonalizations of the covariance matrices such that eigenvalues of D µ and D ν are ordered nondecreasing. When p = q we have:

LGM 2 2 (µ, ν) = 4(tr(Σ µ ) -tr(Σ ν )) 2 + 8(tr(Σ µ Σ µ ) + tr(Σ ν Σ ν )) + 16 min

-tr(D µ B D ν B)

When p = q, an optimal linear Monge map is given by T (x) = Ax where:

LGM

In order to prove Theorem 4.2.6 we will rely on the the following result: Proposition 6.2.3 (Proposition 3.1 in [Anstreicher 1998]). Consider Σ 1 , Σ 2 ∈ R p×p ×R p×p two symmetric matrices and the following (QQP): min

where the eigenvalues of D 1 are ordered nonincreasing the eigenvalues in D 2 are ordered nondecreasing. An optimal solution of (6.76) is found at

2 and the optimal value of (6.76) is tr(D 1 D 2 ).

We will prove that when considering only linear push-forward we can recast the Gromov-Monge problem in the form of Proposition 6.2.3. We have the following result: 6.92) where:

We can now prove the main result:

Of Theorem 4.2.6. With previous notations and calculus, we consider the following change of variable:

then the pushforward equality becomes:

Indeed:

With this change of variable the criterion becomes:

Overall we have the following optimization problem: min

Note that for the final cost we need also to compute:

Overall we have:

Since the covariances are symmetric it gives (4.39). We can use Proposition 6.2.3 to solve min B,BB =Iq -tr(D µ B D ν B). Indeed -D µ is already diagonal which values are nonincreasing and D ν is already diagonal which values are nondecreasing. In this way when p = q min B,BB =Iq -tr(D µ B D ν B) can be solved in close form with B = I q which corresponds to

Proofs and additional results of Chapter 5

This section contains all the proofs of the claims and additional results of the Chapter 5. We recall the notations of the Chapter. Two datasets are represented by matrices X = [x 1 , . . . , x n ] T ∈ R n×d and X = [x 1 , . . . , x n ] T ∈ R n ×d . The rows of the datasets are denoted as samples and their columns as features. Let µ = n i=1 w i δ xi and µ = n i=1 w i δ x i be two empirical distributions related to the samples, where x i ∈ R d and x i ∈ R d . We refer in the following to w = [w 1 , . . . , w n ] and w = [w 1 , . . . , w n ] as to sample weights vectors that both lie in the simplex (w ∈ ∆ n and w ∈ ∆ n ). In addition to them, we also introduce weights for the features that are stored on vectors v ∈ ∆ d and v ∈ ∆ d . Finally, we let vec denote the column-stacking operator.

Perspective for further works

There are many possible extensions and improvements of the works developed here. To conclude we discuss potential limitations and further works of the methods proposed in this thesis from the OT perspective to the machine learning point of view.

What are the improvements and limitations of the different results of this thesis from the OT perspective? The work about GW on Euclidean spaces suggests, in our opinion, interesting further works. The slicing approach for GW finds e.g. natural extensions inspired by the Wasserstein distance. An immediate one is the max-sliced approach [START_REF] Deshpande | [END_REF] where only the projection which maximizes the loss is drawn. This setting is directly applicable in our case without changing the theoretical properties of SGW stated in Theorem 4.1.3. Moreover, one limitation of SGW in the formulation that we proposed is the map ∆ which has to be chosen or optimized when one wants to compute SGW when dimensions