domain: math info

text:
cinq heures par nuit, tout ça pour aller à l'escalade le plus souvent possible se casser les chevilles afin d'être enfin coincés au bureau pour être productifs. J'en profite également pour remercier les différents professeurs qui m'ont donné tout au long de ma scolarité le goût des sciences et des mathématiques. Je pense particulièrement à mesdames Kleinman et Lowenfeld.

Pour relacher la pression de ces années ou pour partager les nombreux moments très heureux qu'elles ont contenus, je ne peux pas ne pas remercier tous mes amis, et particulièrement le filtre, avec qui la thèse ou le premier emploi ont été vécus un peu en commun. Merci à Séginus, pour son humour acéré, à Jean pour les soirées à discuter sans fin en errant dans Paris, à Cyril pour les jeux de mots abyssaux et pour sa confiance infinie en la recherche, à Rémy pour ce savant mélange de palabre, sport, maths, cuisine et bled qui a animé cette conversation dans ses heures les plus sombres. Merci à David pour les discussions de cuisine, pour les ballades dans Paris et en montagne, et à Marc pour sa capacité à passer de la barbe sérieuse et impassible aux rires aux éclats en une fraction de seconde. Merci à mes deux collocs, Thibaut et Simon, au lessives du soir et aux gateaux de ramadan en rentrant, a la salle du puzzle, a Aïcha écoute moi, aux barbecues a 100 personnes, aux moelleux au chocolat et aux tartes à la rhubarbe. Comme dit Rémy, la composante connexe s'agrandit : cela ne fait que rendre les choses encore plus amusantes.

Un immense merci à Juliette et Arielle d'avoir partagé avec moi le goût du thé dans un couloir ou cela sentait plutôt la bière. Le bruit et les odeurs de Chirac s'appliquait bien plus dans ce couloir de normaliens qu'à n'importe quel quartier de Paris. Nous avons survécu ensemble. Merci à Juliette de me permettre de mesurer combien la thèse en sciences est un luxe par rapport à celle de lettres, et à Arielle de me rappeler qu'à l'extérieur, la Start-up Nation ! Une dédicace particulière à Joseph, Julie, Louis, Quentin, Timothée, avec qui j'ai beaucoup partagé de choses malgré, pour certains, de bien grandes distances.

Le plus dur reste pourtant à faire : remercier ma famille. Il est évident que je ne serai pas la sans mes parents qui m'ont donné le goût des études et des maths. Votre soutien constant depuis toujours, et en particulier pendant mes années de prépa, m'ont fait passé cette période de la meilleure façon possible, et c'est grâce à vous que je n'ai jamais hésité à travailler ce qui me plaisait le plus, sans trop de prises de tête. Je ne serais pas non plus là sans mes soeurs, avec qui j'ai passé beaucoup de déjeuners ou de séance d'escalade sans leur adresser la parole car n'arrivant pas à me séparer d'un de mes problèmes dans ma tête. Votre joie de vivre, et vos recadrages subtils ont été un soutien précieux. Je sais, j'ai oublié de remercier crapou. Raby, je pense que si je rentre toujours au maximum le midi chez moi pour déjeuner, c'est en grande partie grâce à toi : merci d'avoir fait de la maison un endroit où l'on a toujours voulu rentrer le plus vite possible.

Je ne suis pas sûr que j'aurais persévéré en recherche, et encore moins autant aimé cela, sans la joie et la confiance que Juliette m'a apportées durant ces quatre années. Partager avec toi les moments forts, un peu durs et joyeux, a été et sera toujours inestimable. Ces années de thèse seront toujours indissociables de ces premières années tous les deux.

Et enfin, merci à Bérénice, qui au delà de l'immense joie qu'on a tous les jours de te voir grandir, a su faire en sorte que je sois le plus efficace possible pour préparer ma soutenance.

Il me paraitrait impensable de commencer ce manuscrit sans remercier les nombreuses personnes sans qui je n'aurais sûrement ni fait de thèse, ni autant aimé ces quatre années. Tout d'abord Francis et Alessandro, qui ont su à la fois me recruter à un moment ou je n'avais pas les idées très claires, me guider au début puis me laisser mariner pendant des mois. Votre bienveillance tout au long de ce processus et particulièrement pendant les différents passages difficiles de ces années m'a été très précieuse. Merci aussi d'avoir été aussi flexibles et de m'avoir permis de laisser de coté le travail pendant quelques mois pour organiser les quelques événements personnels qui ont aussi marqué ces années de thèse. Merci enfin de m'avoir fait découvrir autant de sujets, et d'avoir partagé avec moi votre curiosité sur des thèmes parfois oubliés mais toujours passionnants. Vous m'avez appris a travailler en m'amusant. Et merci d'avoir créé et entretenu cette petite bulle de bonne humeur qu'est Sierra; je pense qu'elle vous doit beaucoup.

Je souhaites aussi remercier mes deux rapporteurs, Gilles Blanchard et Jean-Bernard Lasserre, qui ont accepté de relire ce pavé. Vos retour et vos commentaires, ainsi que vos travaux, ont été et restent pour moi une grande source d'inspiration. Merci également aux autres membres de mon jury, Aymeric Dieuleveut, Claire Boyer, Antonin Chambolle et Yohann de Castro d'avoir accepté d'assister, tous en présentiel, à ma soutenance.

Je voudrais également remercier tout particulièrement le LAAS, Jean-Bernard Lasserre, Didier Henrion, Victor Magron, de m'avoir permis de découvrir le monde merveilleux de l'optimisation polynomiale, et de ses nombreuses applications, et de m'avoir invité à leurs séminaires et conférences. Un grand merci également à Rémi Gribonval et Yohann de Castro, qui, en m'invitant mi-mai à Lyon, m'ont donné l'entrain et la bouffée d'air frais (et de discussions intéressantes) salvateurs au milieu de la rédaction de thèse.

Ensuite, je souhaiterais remercier l'Inria de nous offrir un cadre de recherche extraordinaire. Un merci particulier aux équipes de communication et médiation, qui m'ont permis de profiter de la recherche différemment, et de découvrir des personnes à l'énergie incroyable. Prendre quelques après midi pour aller parler de sciences à des collégiens en prenant des cafés avec Hélène et Anais et d'autres doctorants sont une expérience que je conseille à tout nouveau doctorant.

Le bureau C406 est passé par différentes phases sociales au cours de mes quatre ans et demi de thèse. Merci a Loucas d'en avoir fait un lieu de débats, intellectuels ou non (en atteste le jeu qui traine encore sur mon bureau), de passage, de bonne humeur, à Raphaël d'en avoir fait un haut lieu du golf parisien, de l'humour et de la sieste post prandiale, a Yann d'en avoir fait un lieu de recherche de vacances en vélo, de git commit et de partage d'experience de vie très intense, et a Céline d'y avoir apporté enfin la baisse de caféine qu'il fallait pour qu'on dorme tous plus de 1 Chapter 1 Ce résumé en langue francaise a pour but d'introduire les principaux résultats obtenus durant cette thèse. Dans un premier temps l'outil principal sur lequel nous avons travaillé : les espaces à noyau reproduisant (Sec. 1.1). Nous présenterons ensuite les trois parties de la thèse une par une, essayant de présenter les résultats principaux. Cette introduction en français est entèrement construite sur les différentes introductions du manuscrit anglais, il est donc normal d'y trouver des similarités (voire des équivalences).

Résumé en français

Introduction aux méthodes à noyau 1.Apprentissage supervisé

Rappelons qu'en mathématiques appliquées, il est courant de vouloir apprendre, ou approcher une fonction à valeurs réelles (ou vectorielle) f : X → R sur un espace X . Dans le contexte de l'apprentissage supervisé [START_REF] Hastie | The Elements of Statistical Learning[END_REF], l'objectif est de prédire une sortie Y ∈ R à partir d'une entrée X ∈ X , modélisées comme une paire de variables aléatoires (X, Y ) de distribution inconnue. Le problème de trouver la meilleure fonction de prédiction de X à Y se formalise comme le problème suivant :

f * ∈ arg min f ∈M(X ,R) R(f ) := E [ (f (X), Y )]]
(1.1) où : R × R → R + est une fonction de perte qui compare la prédiction Y = f (X) à la valeur Y , et où M(X , R) est l'ensemble des fonctions mesurables de X dans R. La fonction R est appelée risque en généralisation, et nous supposons pour des questions de simplicité que le minimum de cette fonction est atteint parmi les fonctions mesurables de X dans R, et nous noterons f * un de ses minimiseurs.

10 CHAPTER 1. R ÉSUM É EN FRANC ¸AIS En apprentissage supervisé, l'objectif est d'apprendre une bonne approximation de f * à partir d'exemples de n couples entrée-sortie :

(x 1 , y 1 ), ..., (x n , y n ) ∈ X × R.

Pour des questions statistiques et numériques, lorsque l'on cherche à approcher une telle fonction, il est important de définir un modèle H de fonctions dans lequel un algoritme va effectivement chercher à trouver f . Plus précisément, l'espace H doit satisfaire les conditions suivantes.

a) H doit être suffisamment grand pour bien approcher le problème "idéal" (si le risque R atteint son minimum à un certain f * , nous pouvons par exemple exiger que f * soit dans H); b) il doit être adapté aux données, en étant suffisamment petit ou régularisé pour ne pas surapprendre; c) il doit conduire à un problème d'optimisation (rapidement) résolvable numériquement.

L'outil principal de cette thèse sont les espaces à noyaux reproduisants (ENR), qui sont une classe de tels espaces H.

Modèles linéaires. L'une des classes de fonctions les plus utilisées en apprentissage et en mathématiques appliquées est la classe des modèles linéaires paramétriques, (voir Sec. 2.1.1). Un modèle linéaire est caractérisé par une représentation φ : X → R p qui représente l'espace X par p caractéristiques φ 1 , ..., φ p . Les φ i peuvent être des coordonnées de x, des moments, des résultats de certains filtres ou convolutions appliqués à un signal. Cette représentation peut elle-même être apprise (c'est typiquement le rôle des encodeurs en apprentissage profond), ou conçue en fonction du problème à résoudre. Le modèle linéaire associé à la représentation est l'ensemble des fonctions définies comme suit :

H = f θ : X → R | f θ (x) = θ, φ(x) R p .
(1.2)

Ces modèles présentent de nombreux avantages du point de vue de la modélisation. Ils sont généralement interprétables (le coefficient associé à une caractéristique étant une mesure de son importance), ils héritent de la structure euclidienne de R p (ainsi que de toute autre structure normée définie sur R p ), et préservent la convexité des fonctions de perte convexes, ce qui permet d'exploiter l'ensemble des techniques d'optimisation convexe.

Cependant, ces modèles sont paramétriques, et sont donc assez limités : la dimension p est finie, et ils ne peuvent pas s'adapter à la taille des données (en effet, on doit considérer un modèle plus grand lorsque la taille des données augmente). De plus, le choix de la représentation n'est parfois pas évident, surtout dans les cas où l'on a peu de connaissances a priori sur la fonction f (prenons par exemple le problème de l'apprentissage d'une fonction f où la seule information dont on dispose est qu'elle est régulière).

Dans cette section, nous allons présenter un type de modèle pour les fonctions appelé ENR. Ce modèle généralise les modèles linéaires à des espaces de dimension potentiellement infinie, ce qui conduit à de meilleures propriétés d'approximation, et se concentre non pas sur la conception de la représentation mais plutôt sur celle du noyau, qui est une mesure de similarité entre les points dans X . De plus, les bonnes propriétés des modèles linéaires sont toujours vérifiées, ce qui fait de ce type de modèle un outil très intéressant pour l'apprentissage statistique et les mathématiques appliquées en général. Dans cette section, nous définissons les différents objets fondamentaux en relation avec les espace à noyau reproduisant (ENR), à travers trois points de vue différents. Définissons formellement un noyau reproduisant d'un espace de Hilbert : [START_REF] Aronszajn | Theory of reproducing kernels[END_REF][START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF][START_REF] Shawe | Kernel Methods for Pattern Analysis[END_REF].

Definition 1.1 (noyau reproduisant). Soit H un espace séparable de Hilbert de fonctions à valeurs réelles définies sur un ensemble X . Une fonction k : X × X → R est un noyau reproduisant de H si :

• les fonctions k x : x ∈ X → k(x, x ) appartiennent toutes à H ;

• la propriété de reproduction est satisfaite avec les k x :

∀x ∈ X , ∀f ∈ H, f (x) = f, k x H (1.3)
Si un espace de Hilbert de fonctions possède un noyau reproduisant, alors on peut montrer que ce noyau reproduisant est unique (étant donné H, on peut parler du noyau reproduisant de s, s'il en possède un). Inversement, si H et H ont le même noyau reproduisant k, alors ils sont égaux.

Ainsi, si H existe, H est appelé l 'espace de Hilbert à noyau reproduisant avec noyau k. Nous présentons maintenant trois constructions de ENR, en partant de perspectives différentes. La plus importante en aprentissage est sans doute la deuxième.

Construction par une repésentation. La première façon de construire un ENR consiste à adopter le point de vue des modèles linéaires et à partir d'une représentation. Supposons que l'on nous donne une représentation φ : X → H qui représente X dans un espace de Hilbert. Comme dans le cas du modèle linéaire, nous pouvons définir l'espace de fonctions associé :

H := f : X → R : ∃θ ∈ H, f (x) = θ, φ(x) H .

(1.4)

Nous allons maintenant montrer que H est bien un espace de Hilbert à noyau reproduisant, avec le noyau k(x, x ) := φ(x), φ(x ) H . Notons que ce noyau est défini positif (d.p.), c'està-dire a) symétrique, i.e. , k(x, x ) = k(x , x), et b) pour tout n ∈ N et tout (x i ) ∈ X n , la matrice (k(x i , x j )) 1≤i,j≤n est semi-définie positive (cela vient du fait que cette matrice est une matrice de Gram). k(x, x ) est la mesure naturelle de similarité entre les points impliqués par la représentation. Nous désignons par f θ les fonctions θ, φ(x) H .

Soit H 0 := {θ ∈ H : ∀x ∈ X , θ, φ(x) H = {θ ∈ H : f θ = 0} est l'ensemble des vecteurs θ ∈ H qui représentent la fonction f = 0, de sorte que f θ = f θ i. i.f. θθ ∈ H 0 (en un sens, H 0 représente les θ inutiles en termes d'évaluations de fonctions). Il existe un isomorphisme linéaire naturel entre l'orthogonal de H 0 et H. Notez que H 0 peut lui-même être vu comme l'orthogonal de l'ensemble {φ(x) : x ∈ X } et que l'orthogonal de H 0 est donc H φ := span({φ(x) : x ∈ X }).

(•) = v i , φ(•) H , i ∈ {1, 2}, on a f 1 , f 2 H = P H φ v 1 , P H φ v 2 H et non f 1 , f 2 H = v 1 , v 2 H .
En pratique, il est généralement difficile de construire une représentation de dimension infinie. C'est pourquoi, contrairement aux modèles linéaires, le point de vue sur les ENR adopté en aprentissage est généralement centré sur le noyau k, qui peut être utilisée pour construire l'espace H.

Construction à partir du noyau . Dans ce cas, nous considérons les ENR du point de vue du noyau. Un noyau sur un espace X est simplement une fonction k : X × X → R.

Les noyaux jouent un rôle essentiel dans de nombreuses branches des mathématiques, équations aux dérivées partielles, analyse harmonique, théorie des probabilités. Dans la plupart de ces branches, les noyaux interviennent par l'intermédiaire d'un opérateur noyau sur un espace de mesures M sur X . Cet opérateur est généralement défini comme une convolution avec le noyau k, c'est-à-dire (T k µ)(x) = X k(x, x ) µ(dx ) pour des mesures µ dans M . Ce point de vue sera mentionné dans Sec. 1.1.4. Cependant, en aprentissage, le cadre est un peu moins abstrait.

En aprentissage, le noyau est considéré comme une mesure de similarité. Ce point de vue donne des moyens naturels de concevoir des noyaux pour certaines tâches spécifiques, dans lesquelles une bonne mesure de similarité peut être conçue (ce qui peut être plus facile que de concevoir une représentation). C'est le cas pour les séquences biologiques par exemple, où les noyaux ont été utilisés pour comparer des séquences d'ADN et de protéines (voir Jaakkola, Diekhans, and Haussler (1999) et les travaux ultérieurs). Nous donnons des exemples de noyaux dans la section Sec. 2.2.2.

. Dans la mesure où nous souhaitons que le noyau k soit une mesure de similarité entre deux éléments de X , un bon modèle linéaire basé sur les caractéristiques serait un modèle où la représentation φ : X → H (où H est un espace de Hilbert) reflète cette mesure de similarité : φ(x), φ(x ) H = k(x, x ). C'est exactement le rôle d'un ENR H avec un noyau k comme décrit ci-dessus. La pierre angulaire des méthodes à noyau est que nous pouvons effectivement trouver un tel espace H, étant donné tout noyau d.p. (voir la définition dans Eq. (1.7)).

Fonctions de base. Comme nous l'avons vu dans le paragraphe précédent, si H est un ENR avec un noyau k, il doit nécessairement être la fermeture de l'espace span{k x : x ∈ X } pour la bonne structure de Hilbert (c'est-à-dire qu'il doit satisfaire k x , k x = k(x, x )). Par conséquent, pour construire un ENR associé à k, on commence par considérer l'espace des fonctions élémentaires définies par le noyau.

H 0 = span({k(x, •) : x ∈ X }) = f : X → R | f (x) = n i=1 α i k(x i , x), n ∈ N, (x i ) ∈ X n , (α i ) ∈ R n .
(1.5)

Forme bilinéaire Cet espace peut être équipé d'une forme bilinéaire définie par le noyau (afin de (1.6) pour tout n, (x i ), (α i ) et m, (x j ), (β j ). La condition pour que cette forme bilinéaire soit un produit scalaire est exactement que le noyau k soit défini positivement, comme défini dans le paragraphe précédent, i.e., que k soit symétrique (k(x, x ) = k(x , x) pour tout x, x ), et que

1.1. INTRODUCTION AUX M ÉTHODES À NOYAU 13 satisfaire k x , k x H 0 = k(x, x )) : n i=1 α i k(x i , •), m j=1 β j k(x j , •) H 0 = n i=1 m j=1 α i k(x i , x j )β j ,
∀n ∈ N, ∀(x i ) ∈ X n , ∀(α i ) ∈ R n , n i,j=1
α i k(x i , x j )α j ≥ 0.

(1.7)

Construction de H. Sous l'hypothèse que k est d.p., l'espace H 0 est appelé un espace pré-Hilbert, c'est-à-dire un espace vectoriel équipé d'un produit scalaire. À partir de H 0 , nous pouvons construire l'espace H comme la complétion de H 0 par rapport à la norme définie par le produit scalaire Eq. (1.6). Puisque dans ce cas, H 0 = H, il s'agit exactement de l'espace de Hilbert à noyau reproduisant associé à k. Ce résultat est dû à [START_REF] Aronszajn | Theory of reproducing kernels[END_REF] qui l'attribue à [START_REF] Moore | Abstract for "On properly positive Hermitian matrices[END_REF].

Theorem 1.1 [START_REF] Aronszajn | Theory of reproducing kernels[END_REF]). Soit k un noyau d.p. sur un espace X . Alors il existe un ENR unique H avec pour noyau reproducteur k.

Définition comme un espace de Hilbert de fonctions avec la propriété de reproduction. La dernière façon de définir un ENR est de passer par l'espace de fonctions lui-même. Supposons que l'on vous donne un espace de Hilbert de fonctions H. Il s'agit d'un ENR si et seulement si les morphismes d'évaluation f ∈ H → f (x) sont continus. En effet, dans ce cas, par le théorème de représentation de Riesz, il existe un élément k x ∈ H tel que f (x) = f, k x H pour tout f ∈ H. Ainsi, H est un ENR avec le noyau k(x, x ) := k x , k x H .

Ce point de vue permet de comprendre directement que certains espaces de Hilbert classiques de fonctions sont des ENRs. C'est le cas des espaces de Sobolev W s 2 (Ω) pour Ω ⊂ R d , qui sont des espaces de Hilbert à noyau reproducteur pour s > d/2. Cependant, trouver une expression pour le noyau est souvent compliqué. Dans le cas où Ω = [0, 1], il est connu pour s = 1 où k(x, y) = max(x, y). Cependant, l'expression pour s général est beaucoup plus complexe (voir [START_REF] Wahba | Spline Models for Observational Data[END_REF]). Dans Sec. 2.2.2, nous donnerons des exemples de tels noyaux. Par ailleurs, ce point de vue souligne également que les ENR sont particulièrement adaptés lorsque la fonction f est connue via des évaluations de fonctions. En un sens, un ENR contient des "diracs", i.e., des éléments k x dont les propriétés sont similaires à celles des diracs dans L 2 qui satisfont f, δ x L 2 (R d ) = f (x), mais qui se trouvent dans l'espace lui-même.

Noyaux en aprentissage . Dans cette section, nous avons vu trois façons de considérer les ENR et de les construire : une par le biais d'une représentation à valeur dans Hilbert, une par le biais d'un noyau d.p. sur X , et une en regardant directement l'espace des fonctions H. Les deux dernières constituent les deux principales tendances de la littérature utilisant les ENR.

L'approche centrée sur les espaces de fonctions utilise le noyau k comme outil pour décrire et caractériser l'espace de fonctions H, en particulier à travers l'opérateur noyau T k défini ci-dessus. Cette approche a été utilisée pour décrire certains espaces en analyse harmonique [START_REF] Zaremba | L'équation biharmonique et une class remarquable de functions fondamentales harmoniques[END_REF].

CHAPTER 1. R ÉSUM É EN FRANC ¸AIS En revanche, l'approche centrée sur le noyau d.p. lui-même a été développée successivement par [START_REF] Mercer | Functions of positive and negative type, and their connection with the theory of integral equations[END_REF], [START_REF] Moore | Abstract for "On properly positive Hermitian matrices[END_REF][START_REF] Hastings | of Memoirs of the American Philosophical Society[END_REF] et [START_REF] Aronszajn | Theory of reproducing kernels[END_REF].

En aprentissage, nous utilisons ces deux approches. Nous nous intéressons à l'espace fonctionnel sous-jacent du point de vue de la modélisation et des statistiques. En effet, l'espace de fonctions H est l'espace des fonctions de test, c'est-à-dire l'espace où nous cherchons une solution à notre problème fonctionnel. De plus, la structure de Hilbert naturelle sur H et en particulier sa norme aura un rôle clé dans la régularisation. Nous caractériserons donc les erreurs statistiques par des quantités qui illustrent la relation entre notre espace modèle de fonctions H et la vraie fonction cible f * (si elle existe).

Les noyaux pour la minimisation du risque empirique

Dans cette section, nous décrivons comment les ENR sont adaptés dans le contexte de l'apprentissage. Rappelons que notre problème "idéal" est le problème de minimisation du risque attendu, que l'on prendra ici sous la la forme

R * = inf f ∈F R(f ) = E [ Z (f )],
(1.8) où F est une classe idéale de fonction (pensez-y comme la plus grande classe de fonctions sur laquelle le risque est défini), Z une variable aléatoire sur un espace Z avec une distribution ρ représentant les données, et est une fonction de perte. Nous désignons par f * tout minimiseur de Eq. (1.8) s'il existe. Dans cette section, nous supposerons que a) F est une classe de fonctions d'un espace X à R, et b) que nous avons une application π : Z → X telle que z (f ) est de la forme (z, f (π(z))), que nous désignerons parfois par z (f (π(z))) . Cette hypothèse plutôt abstraite indique simplement que la perte de f au point de données z ne dépend de f qu'à travers la évaluation de f en un point x = π(z) donné par les données (nous supposons en effet que π est connu). Définissons X = π(Z) ; l'expression du risque devient R(f

) = E [ Z (f (X))].
Pour rendre les choses plus claires, prenons le cadre de l'apprentissage supervisé, où l'objectif est de prédire une sortie Y ∈ R à partir d'une entrée X ∈ X . La variable aléatoire des données est donc Z = (X, Y ) ∈ Z = X × R, et le risque est généralement de la forme E (X,Y ) (f (X)) ; dans ce cas, l'application π est simplement la projection sur la première coordonnée. Dans le cas de la régression des moindres carrés, nous avions z (f ) = f (x) -y 2 , et donc cela correspond à z (t) = 1 2 ty 2 . Pour la régression logistique, la fonction de perte a été définie comme z (f ) = log(1 + exp(-yf (x))), ce qui correspond à z (t) = log(1 + exp(-yt)).

Le fait que la fonction de perte ne dépende que des évaluations de fonctions nous permet de reformuler a) le problème de minimisation du risque attendu sur H, et b) le problème de minimisation du risque empirique (régularisé) sur H de telle sorte que f n'apparaisse que dans un produit scalaire avec un élément k x . En effet, la minimisation du risque espéré sur H peut s'écrire

R H = inf f ∈H E [ Z ( f, k X H )].
(1.9)

Nous désignerons par f H un minimiseur de Eq. (1.9) s'il existe. Comme nous l'expliquons plus en détail dans l'introduction générale de ce manuscrit, le problème nous allons effectivement résoudre afin d'approcher f H est le problème dit de minimisation du risque empirique (MRE):

1.1. INTRODUCTION AUX M ÉTHODES À NOYAU 15 f n,λ = arg min f ∈H 1 n n i=1 z i ( f, k x i H ) + λ 2 f 2 H , x i = π(z i ),
(1.10) où z 1 , ..., z n sont les points de données, les x i = π(z i ) sont les points d'évaluation de f associés aux z i , et le terme de régularisation λ 2 f 2 H est appelé la régularisation de Tikhonov. Ce terme de régularisation est le plus standard, mais notez qu'il existe d'autres procédures de régularisation, que nous n'utiliserons pas dans cette thèse : [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]. Comme les ENR sont généralement de dimension infinie, une régularisation est généralement nécessaire, comme expliqué dans Eq. (2.9) et détaillé dans Sec. 1.1.4. L'un des théorèmes fondamentaux des méthodes à noyau est le suivant.

Theorem 1.2 (Théorème des représentants, [START_REF] Cucker | On the mathematical foundations of learning[END_REF]). Soit L : R n → R∪{+∞} une fonction semi-continue inférieurement minorée, soit x 1 , .., x n ∈ X , et H un ENR. Si le problème

min f ∈H L(f (x 1 ), ..., f (x n )) + λ 2 f 2 H , (1.11)
a une solution, alors il a une solution de la forme

f (x) = n i=1 α i k(x i , x), α ∈ R n .
(1.12)

De plus, une telle solution existe toujours lorsque λ > 0.

En d'autres termes, tout problème de la forme Eq. (1.11), c'est-à-dire qui ne dépend de f que par ses évaluations en x 1 , ..., x n , a un minimiseur dans H n = span({k x 1 , ..., k xn }). Ceci est facilement prouvé en décomposant f sur H n et son orthogonal. Il y a deux conséquences importantes de ce théorème. La première est que le problème de minimisation du risque empirique Eq. (1.10) peut être formulé comme un problème en dimension n :

α ∈ arg min α∈R n 1 n n i=1 z i (e i Kα) + λ 2 α Kα, (1.13)
où K = (k(x i , x j )) 1≤i,j≤n ∈ R n×n est la matrice du noyau associée aux points de données (x i ). La deuxième conséquence est que la fonction résultante f n,λ ne peut être exprimée qu'à l'aide du noyau : f n,λ (x) = n i=1 α i k(x i , x). En résumé, les principaux avantages des noyaux dans ce contexte sont les suivants : a) bien que l'ENR puisse être de dimension infinie, le problème de minimisation du risque empirique qui en résulte est de dimension finie, et b) les problèmes et les fonctions résultantes peuvent être décrits en utilisant uniquement le noyau, en omettant la définition abstraite de l'ENR. Cependant, notez que le théorème du représentant (Theorem 1.2) n'est vrai que si le risque empirique ne dépend de la fonction f que par le biais des évaluations. Cela n'inclut pas les cas où des contraintes sur f peuvent être ajoutées. Cela peut parfois être utile, par exemple lors de l'apprentissage de densités de probabilité (comme dans chapters 6 and 7), où une contrainte naturelle peut être que la fonction soit égale à un. Cependant, nous pouvons montrer que sous certaines conditions, chercher f dans H n peut être suffisant statistiquement (voir chapter 7). Dans cette thèse, nous allons essayer de nous éloigner un peu du paradigme du théorème du représentant, et tenter de montrer que les méthodes à noyaux sont performantes non pas parce que H n contient le minimiseur du risque empirique, mais parce que lorsqu'on a n points de données, H n est une assez CHAPTER 1. R ÉSUM É EN FRANC ¸AIS bonne approximation de H. Dans les deux sections suivantes, nous discuterons brièvement des propriétés satistiques du minimiseur du risque empirique f n,λ , et des propriétés d'optimisation de la résolution du problème Eq. (1.13).

1.1.4 Un point de vue statistique sur la MRE avec des noyaux Exemple : régression des moindres carrés. Dans les deux prochaines sections, et dans le ??, nous utiliserons le problème de régression des moindres carrés par noyau comme exemple. Dans ce cas, le risque en espérance ou risque en généralisation est R ls (f ) = 1 2 E f (X) -Y 2 . De plus, nous supposerons que Y ∈ L 2 ce qui montre qu'il existe un prédicteur optimal f * qui est dans L 2 (X , ρ X ). Dans le cas des moindres carrés, il existe une solution sous forme close au problème de minimisation du risque empirique régularisé Eq. (1.13) étant donné les points de données (x i , y i ) et λ > 0 :

f n,λ = n i=1 α i k(•, x i ) α = (K + λnI) -1 y, y = (y i ) ∈ R n .
(1.14)

En effet, les ENR étant généralement de dimension infinie, on peut montrer que l'erreur d'approximation R H -R * est nulle pour certains ENR et certaines pertes. Ces résultats sont généralement appelés résultats de "universalité" ; ils visent à montrer que sous certaines conditions sur le ENR H (ou le noyau k), H peut approximer toute fonction f ∈ F, c'est-à-dire est dense dans F pour une certaine classe de fonctions F. Bien sûr, "approcher toute fonction" doit être défini, et de nombreuses définitions peuvent être données : [START_REF] Micchelli | Universal kernels[END_REF][START_REF] Bharath | Universality, characteristic kernels and rkhs embedding of measures[END_REF]. Si X est un espace de Hausdorff localement compact par exemple (c'est le cas de tous les espaces X que nous considérons), un type d'universalité est la L p -universalité : H est dense dans L p (X , ρ X ) pour toute mesure de probabilité de Borel donnée ρ X (habituellement la loi de X dans le cadre supervisé), où L p (X , ρ X ) est équipé de la norme L p . Un autre type d'universalité, l'universalité cc, est le fait que H est dense dans l'ensemble C(X ) des fonctions continues sur X pour la convergence compacte (i.e. , pour tout K ⊂ X compact et toute fonction f ∈ C(X ), il existe une suite f n ∈ H de fonctions telle que sup x∈K f n (x) -f (x) -→ n→∞ 0). Par exemple, dans le problème des moindres carrés, l'erreur d'approximation est nulle dès que le noyau est L 2 -universel pour la mesure ρ X . On peut prouver que les noyaux gaussiens et de Sobolev sont L p -universels et cc-universels par exemple : [START_REF] Micchelli | Universal kernels[END_REF][START_REF] Bharath | Universality, characteristic kernels and rkhs embedding of measures[END_REF].

Remark 1. L'erreur d'approximation étant nulle ne signifie pas que f * , s'il existe, est dans H. Une erreur d'approximation nulle signifie simplement qu'il existe une suite f n de fonctions dans H telle que R(f n ) → R(f * ). En effet, si H est l'espace de Sobolev ou d'ordre s, il n'est bien sûr pas vrai que toute fonction dans L 2 (R d , ρ X ) pour une mesure de probabilité donnée ρ X est également dans W s 2 (R d ). Dans la discussion suivante et tout au long de cette thèse, nous n'analyserons que l'erreur d'estimation, et nous ferons des hypothèses qui garantissent toujours que R H = R * , faisant ainsi de l'erreur d'estimation l'excès de risque réel.

INTRODUCTION AUX M ÉTHODES À NOYAU

Compromis biais-variance dans les méthodes à noyau . Lorsque l'espace H est grand, nous modulons sa capacité en utilisant une certaine forme de régularisation. Dans cette thèse, nous utiliserons la régularisation de Tikhonov λ 2 f H car elle ne dépend que de la norme du noyau, et possède de bonnes propriétés d'optimisation (elle rendra l'objectif fortement convexe, voir Sec. 2.1.3). Nous mentionnons d'autres régularisations dans chapter 3.

Dans le cas des méthodes à noyau, la décomposition traditionnelle approximation-estimation peut être remplacée par une décomposition dite biais-variance de l'erreur d'estimation, de la forme

R( f n,λ ) -R H ≤ Bias(λ) + Variance(λ, n),
(1.15) où le terme de biais ne dépend que de λ et peut être vu comme un analogue de l'erreur d'approximation pour l'espace "H régularisé avec λ", et le terme de variance est un analogue de l'erreur d'estimation, et quantifie l'impact statistique d'avoir accès à la vraie distribution uniquement à travers les n points de données.

Plus précisément, dans le cas de la régularisation de Tikhonov, si f λ est un minimiseur du risque attendu régularisé R(f ) + λ 2 f H (qui existe dès que R est semi-continue inférieure et bornée inférieurement comme ce sera le cas pour toutes les pertes considérées), alors on peut voir le terme de biais comme le risque excédentaire de f λ : Bias(λ) ≈ R(f λ ) -R H . Ce terme de biais peut être quantifié par la régularité de la solution optimale f * par rapport à l'ENR choisi H. Si f * est dans H par exemple, on peut montrer que le terme de biais se comporte comme Bias(λ) = O 0 (λ). Si f * n'est pas dans H, alors le terme de biais convergera plus lentement vers 0. Ce comportement est plus précisément quantifié dans le contexte des moindres carrés, brièvement décrit ci-dessous.

Pour le terme de variance, la forme typique que nous recherchons est Variance(λ, n) = d λ n , où d λ est une notion de dimension intrinsèque de l'espace H régularisé avec λ. Ce type de bornes est appelé bornes "sans dimension", non pas parce qu'aucune notion de dimension n'apparaît, mais parce que la dimension de l'espace H n'apparaît pas, ce qui est nécessaire si l'on veut quantifier la variance sur un espace de Hilbert de dimension infinie.

Dans chapter 4, nous établissons de telles décompositions biais-variance dans le cas où la perte satisfait une certaine propriété appelée auto-concordance généralisée. Cette décomposition a déjà été bien étudiée dans la littérature dans le cadre des moindres carrés.

Le cas des moindres carrés. Dans le cas de la régression des moindres carrés, de telles décompositions biais-variance ont été obtenues afin d'étudier les taux de convergence optimaux [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF][START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]. Dans le cadre de la régularisation de Tikhonov, nous pouvons prouver les décompositions biais-variance suivantes avec une probabilité élevée (un énoncé plus formel peut être trouvé dans chapter 3) :

R( f n,λ ) -R H ≤ C     b λ biais + d λ n variance     ,
(1.16)

CHAPTER 1. R ÉSUM É EN FRANC ¸AIS où b λ est le terme de biais et d λ est ce qu'on appelle la dimension effective de H [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF] et C est une constante dépendant de la probabilité avec laquelle nous voulons que cette affirmation soit vérifiée (puisque les points d'échantillonnage z 1 , ..., z n sont aléatoires). De manière informelle, la dimension effective est une mesure de la taille de H par rapport à L 2 (X , ρ X ), où nous savons que vit la fonction optimale f ρ (nous soulignons le lien entre l'optimum et la mesure dans cette section en utilisant cette notation).

Pour comprendre le comportement du terme de biais ainsi que la définition de la dimension effective, nous devons introduire un peu plus de notions liées aux espaces à noyau. Supposons que le noyau k soit continu et borné (i.e., k(x, x) ≤ κ 2 ), et que X soit séparable. Soit ρ la distribution des données, ρ X sa marginale sur X et supposons que Y ∈ L 2 (X × R, ρ), c'est-à-dire est de carré intégrable.

Comprendre la dimension effective. Puisque le noyau est borné, les fonctions dans H appartiennent également à L 2 (X , ρ X ) ( f L 2 ≤ κ f ) et l'application S : H → L 2 (X , ρ X ) est une application linéaire continue entre espaces de Hilbert. On définit l'opérateur de covariance :

Σf = S * Sf = X f (x )k x ρ X (dx ) such that f, Σg = f, g L 2 (X ,ρ X ) .
(1.17)

Il s'agit d'un opérateur semi-défini positif symétrique de trace finie (donc compact), qui décrit la relation entre la norme de H et la norme de L 2 , qui est celle adaptée au problème des moindres carrés. Comme Σ est un opérateur de trace finie, il peut être décomposé en utilisant le théorème spectral comme i∈I λ i φ i ⊗ H φ i où les λ i sont des valeurs positives triées par ordre décroissant, I est de la forme {1, ..., k} si Σ est de rang fini ou N (ce qui est le cas le plus courant), et φ i sont les vecteurs propres renormalisés correspondants. L'espace H peut alors être décomposé comme H = span({φ i : i ∈ I}) ⊥ ⊕ Ker(Σ), où Ker(Σ) est l'ensemble de toutes les fonctions dans H telles que f L 2 (X ,ρ X ) = 0 (ceci est possible si le support de f est disjoint du support de la mesure ρ X ). Notez que le spectre de Σ donne beaucoup d'informations quantitatives sur la taille de H par rapport à L 2 . Si le spectre est fini, cela signifie que seul un sous-ensemble de dimension finie de H est réellement significatif pour approximer une fonction en norme L 2 . Plus généralement, une grande λ i sera associée à un vecteur propre capturant beaucoup d'informations sur L 2 , alors que les vecteurs propres associés à de petites λ i ne captureront presque aucune information sur la structure L 2 . La vitesse à laquelle les valeurs propres convergent vers zéro est donc un bon indicateur de la taille de l'espace H par rapport à l'espace idéal L 2 (X , ρ X ). C'est exactement ce que mesure la dimension effective d λ , qui est définie comme suit

d λ = Tr((Σ + λI) -1 Σ) = i∈I λ i λ + λ i , λ > 0.
(1.18)

La dimension effective d λ est une quantité telle que d λ -→ λ→0 +∞ dès que le spectre est infini ; ce qui est intéressant est la vitesse à laquelle cette quantité diverge. Ceci est lié à la décroissance des valeurs propres λ i . Par exemple, si H est l'ENR associé au noyau de Sobolev k s (voir Sec. 2.2.2), et si la mesure ρ X est comparable à la mesure de Lebesgue (il existe une constante K telle que ρ X ≤ KΛ où Λ est la mesure de Lebesgue), la décroissance des valeurs propres est de l'ordre de λ i = O(i -2s/d ), ce qui implique que le d λ ≤ Cλ -d/(2s) . Cela confirme l'intuition selon laquelle plus l'espace est petit (plus le s est grand), plus d λ est petit.

Comprendre le terme de biais. Le terme de biais est un terme qui quantifie la régularité de f ρ par rapport à l'ENR H. On a b λ = o λ→0 (1), et la vitesse de convergence de b λ vers zéro est 1.1. INTRODUCTION AUX M ÉTHODES À NOYAU 19 caractérisée par la régularité de f ρ . Si f ρ ∈ H, on a b λ = O(λ). Si f ρ est très régulier, il tend encore plus vite vers zéro. Pour définir formellement ce que signifie la régularité de f ρ par rapport à H, définissons l'opérateur noyau associé à k : T k : L 2 (X , ρ X ) → L 2 (X , ρ X ) tel que ∀g ∈ L 2 (X , ρ X ), (T k g)(x) = X k(x, x )g(x )ρ X (dx ).

(1.19)

L'opérateur T k est lié à l'opérateur de covariance par l'application S qui injecte H dans L 2 (X , ρ X ) ; on a Σ = S * S et T k = SS * . En particulier, T k = i∈I λ i ψ i ⊗ L 2 ψ i où les λ i sont les mêmes valeurs propres que celles de Σ et les ψ i sont les vecteurs propres normalisés correspondants, qui sont liés aux vecteurs propres de Σ, vus comme des éléments de L 2 , par la relation ψ i = φ i / √ λ i . Les ψ i forment une base orthonormée de Ker(T k ) ⊥ ⊂ L 2 (X , ρ X ). De plus, par le théorème de Mercer (voir Carmeli, De Vito, and Toigo (2005) et [START_REF] Dieuleveut | Nonparametric stochastic approximation with large step-sizes[END_REF] pour une extension), on peut montrer que H = range(T

1/2 k ) ⊕ H 0 où H 0 = {f ∈ H : f 2
L 2 (X ,ρ X ) = 0}. Notons que formellement, L 2 (X , ρ X ) = range(T 0 k ). En général, nous dirons que f ∈ L 2 (X , ρ X ) est r régulière si elle appartient à l'ensemble H r := range(T r/2 k ) pour r ≥ 0. Nous avons H 0 = L 2 (X , ρ X ), H 1 ⊂ H, et en général H r ⊂ H pour r ≥ 1. Lorsque r augmente, la régularité des éléments de H r augmente. Faire l'hypothèse que f ρ ∈ H r est un bon moyen de quantifier la régularité de f ρ (qui est r = 0 a priori ). Le biais b λ peut être contrôlé par la régularité, et comme prévu, plus la fonction est régulière, plus vite b λ tend vers zéro :

b λ ≤ C λ r si f ρ ∈ H r , (1.20)
pour une constante C dépendant de la norme de f ρ dans H r . Remarquons que cette condition peut également être exprimée comme une condition sur les coefficients f i de la décomposition de f ρ sur la base propre ψ i par rapport au λ i , i.e., f ρ ∈ H r i.i.f i∈I f 2 i /λ r i < +∞. Example 1.1 (Mesure uniforme et espaces de Sobolev). Dans le cas du noyau de Sobolev, et si ρ X est la mesure uniforme sur X , et si H s désigne l'ENR associé à k s , f ρ ∈ H r s signifie simplement que f ρ ∈ W sr 2 (X ), ce qui correspond exactement à l'interpolation entre les espaces de Sobolev.

Conclusion.

Dans le cas des moindres carrés, nous avons une façon très fine de quantifier le décomposition biais-variance, et une bonne compréhension du rôle de la régularité de f ρ ainsi que de celui de la dimension effective de H dans la performance du minimiseur du risque empirique. Cela peut nous aider à acquérir une certaine intuition sur les raisons pour lesquelles un problème d'apprentissage est difficile ou facile, ou sur ce qu'il faut faire pour réduire sa dimension (voir part I).

Un point de vue d'optimisation sur la MRE avec noyaux

Les méthodes à noyaux présentent à la fois de grands avantages et inconvénients potentiels en termes d'optimisation pour trouver le minimiseur du risque empirique f n,λ . Rappelons qu'en utilisant le théorème des représentants, trouver le minimiseur du rsique empirique est équivalent à la résolution de Eq. (1.13) rappelée ici :

α ∈ arg min α∈R n 1 n n i=1 z i (e i Kα) + λ 2 α Kα.
CHAPTER 1. R ÉSUM É EN FRANC ¸AIS Les principaux avantages sont que le problème hérite de la convexité des fonctions de perte z (•) ainsi que du caractère lisse de z dès que le noyau est borné. De plus, si nous reparamétrons le problème en fixant β = K 1/2 α, le problème est λ-fortement convexe en β, ce qui permet d'appliquer des méthodes rapides du premier ordre. Les inconvénients sont que la matrice du noyau peut être énorme (de taille n × n), ce qui est prohibitif pour les grands ensembles de données à la fois en termes de mémoire et de coût de calcul (car le calcul d'un gradient stochastique coûterait O(n) et un gradient complet O(n 2 )). De plus, la reparamétrisation avec β telle que α = K 1/2 β est à la fois nécessaire (car la matrice du noyau est très mal conditionnée) et coûteuse (O(n 3 ) en raison de l'inversion de la matrice, cela ne peut être entièrement parallélisée sur des GPUs). Même dans ce cas, les λ qui semblent être optimaux après l'évaluation du modèle sont très petits dans de nombreux cas (voir le cas des données de Higgs dans chapter 5). Cela signifie que le problème sera très mal conditionné, même après la reparamétrisation, et que les méthodes classiques du premier ordre seront lentes.

Heureusement, ces dernières années, des moyens ont été développés pour surmonter ces difficultés, principalement dans le contexte de la régression linéaire (des moindres carrés). Le premier domaine principal d'amélioration a été la réduction de la dimension, c'est-à-dire la réduction de la dimension du problème d'optimisation à un certain m n. Il existe deux méthodes principales pour y parvenir. La première consiste à sous-échantillonner les points x1 , ..., xm de x 1 , ..., x n et à rechercher une solution au problème MRE limitée à l'ensemble des fonctions de la forme f (x) = m j=1 αj k(x j , x) (cette méthode est appelée Nyström ou sous-échantillonnage en colonnes, voir [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]). L'autre consiste à disposer d'un noyau dont l'expression est de la forme k(x, x ) = E ω∼p [φ(x, ω)φ(x , ω)], qui, en tant qu'espérance de représentations aléatoires (voir ?? et la discussion suivante sur le noyau gaussien) ; dans ce cas, on approxime la matrice du noyau par une approximation de rang faible de la forme K = RR où R = 1 √ m (φ(x i , ω j )) 1≤i≤n 1≤j≤m où les ω j sont tirés de p. Nous détaillerons les propriétés de ces méthodes dans chapter 3 ; elles permettent de réduire la dimension de n à m d'ordre d λ , sans perdre aucune performance statistique pour le minimiseur de risque empirique. Le deuxième domaine d'amélioration concerne les algorithmes. En utilisant la méthode du gradient conjugué détaillée dans Eq. (2.19) dans proposition 2.1 ainsi qu'une technique de pré-conditionnement (pour plus de détails, voir chapter 3), [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF] ont accéléré la résolution de la MRE à dimension réduite à une complexité d'ordre O(nd λ ), sans payer le prix du mauvais conditionnement. Ces méthodes ont été implémentées dans une bibliothèque par Meanti, Carratino, Rosasco, and Rudi (2020) et permettent de traiter des milliards de points, en utilisant notamment des sous-programmes développés par [START_REF] Charlier | Kernel operations on the gpu, with autodiff, without memory overflows[END_REF] pour calculer les évaluations du noyau k(x, x ) de manière rapide.

Dans part I et plus particulièrement dans chapter 5, ces techniques sont étendues pour d'autres pertes que la perte carrée. Ces techniques ont depuis été implémentées dans une bibliothèque par Meanti, Carratino, Rosasco, and Rudi (2020), avec les techniques rapides des moindres carrés.

Dans ces deux travaux, nous étudions le problème classique de minimisation du risque (espérance du risque) par la résolution du problème de minimisation du risque empirique régularisé, sur un espace de Hilbert à noyau reproduisant H avec un noyau k (voir la section précédente). L'objectif de ces deux travaux était d'étendre les décomposition précis entre biais et variance, les taux et les algorithmes rapides déjà connus pour les moindres carrés aux pertes autoconcordantes généralisées. Il s'agit d'une classe de pertes intéressante car elle contient la perte de régression logistique x,y (f ) = log(1 + exp(-yf (x))), et la perte de régression logistique multiclasse (voir Sec. 2.1.1 pour plus de détails).

Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] commencent par étudier l'excès de risque du minimiseur de risque empirique régularisé f n,λ . Nous montrons que, comme dans le cas des moindres carrés, il existe ue décomposition biais-variance de la forme

R( f n,λ ) -R H ≤ C log 1 δ b λ + d λ n ,
avec une probabilité au moins 1 -δ, (1.21) où le terme de biais et la dimension effective sont définis comme des quantités qui correspondent à celles définies dans le cas des moindres carrés (où essentiellement, on remplace l'opérateur de covariance par la matrice hessienne du risque attendu à l'optimum). En particulier, cela nous permet d'obtenir des bornes supérieures sur les taux de convergence.

Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] développent un algorithme pratique pour résoudre la régression logistique à noyau à grande échelle, qui est maintenant utilisé dans la bibliothèque créée par Meanti, Carratino, Rosasco, and Rudi (2020). Il est basé sur trois contributions.

• La première est de montrer que, comme dans le cas des moindres carrés, la MRE peut être restreinte à un problème de dimension beaucoup plus petite grâce à l'échantillonnage de Nyström (nous réduisons la dimension de n à m ≈ d λ ).

• Le second est de concevoir un algorithme du second ordre globalement convergent pour les pertes autoconcordantes généralisées, dont la complexité est d'ordre O(nd λ ) (en omettant les facteurs logarithmiques). De manière similaire aux méthodes de points intérieurs, cette méthode utilise un schéma décroissant de λ (λ 0 , ..., λ T ) pour optimiser de manière itérative le risque empirique régularisé pour λ t+1 en partant de l'estimation précédente f n,λt . L'optimisation à chaque étape est effectuée à l'aide d'une méthode de Newton approchée, qui permet de calculer approximativement les étapes de Newton en utilisant une approximation de la Hessienne.

• Le troisième est de prouver que le résultat de l'algorithme atteint la même erreur statistique que celle de l'estimateur MRE complet, i.e., il satisfait une borne de la forme Eq. (1.21).

Un point crucial est que cette méthode est indépendante du conditionnement du problème (en négligeant les facteurs logarithmiques). Nous illustrons ce fait sur des ensembles de données réels où le paramètre de régularisation optimal λ doit être très petit.

Dans cette section, nous présentons les principaux résultats de nos contributions sur l'extension des taux rapides et des algorithmes existants pour le problème des moindres carrés à une classe plus large de fonctions convexes, la classe des fonctions ACG.

Le but de cette section est de présenter nos résultats dans un cadre cohérent et unifié, ainsi qu'une idée des contributions techniques. Dans les articles originaux, dont le verbatim peut être trouvé dans chapters 4 and 5, les notations sont légèrement différentes, et les hypothèses sont souvent plus faibles. Ces articles contiennent toutes les preuves et les énoncés précis. Nous ferons référence aux résultats que nous invoquons, même si les notations peuvent changer.

Afin de présenter les résultats, le plan sera le suivant. Nous commencerons dans Sec. 1.2.3 par présenter les résultats statistiques obtenus dans Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] sur la performance non-asymptotique de l'estimateur régularisé MRE, qui reflète les résultats dans le cadre des moindres carrés, présentés plus en détail dans Sec. 3.1.1. Nous continuerons dans Sec. 1.2.4 à montrer comment il est possible de réduire la dimension du problème de dimension finie résultant, de la même manière que dans le cas des moindres carrés (bien que les preuves soient un peu plus impliquées). Ces résultats ont été prouvés par Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]. Dans Sec. 1.2.5, nous mentionnons très succintement l'une des principales contributions de Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] qui est un schéma du second ordre globalement convergent pour les fonctions ACG qui est à peu près indépendant du conditionnement du problème. Nous montrons que, comme l'algorithme de FALKON dans le cas des moindres carrés, il est possible de calculer efficacement un estimateur, en utilisant le schéma précédent, avec une faible complexité temporelle et une précision statistique optimale. Ceci a également été établi par Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF].

Hypothèses

Nous utilisons toutes les notations pour les fonctions et les Hessiennes (H(f ) pour la Hessienne en f ) introduites dans ?? (celles-ci sont très simples). L'objectif est de minimiser le risque R(f ) = E [ Z (f (X))] à partir d'échantillons z i de Z. Soit ρ la distribution de Z. Comme dans le cas des moindres carrés, nous supposons que les échantillons z 1 , ..., z n sont des échantillons i.i.d. de Z (voir Assumption 2.1). De plus, nous supposons qu'il existe f ρ ∈ H tel que f ρ minimise le risque attendu (voir Assumption 3.1). Nous supposerons que le noyau est borné par 1, bien que les résultats soient valables dès que le noyau est borné (voir Assumption 3.3). Nous supposerons également que Assumption 3.7 est satisfaite, c'est-à-dire que les fonctions z (•) sont R-ACG pour tout z ∈ Z (nous ne détaillons pas cette hypothèse ici par soucis de simplicité).

Nous avons également besoin d'une hypothèse plus technique. Tout d'abord, pour toute distribution ρ de Z, définissons

b 1 (f ) = | Z (f (X))| L ∞ (ρ) , b 2 (f ) = | Z (f (X))| L ∞ (ρ) .
(1.22)

Nous supposerons que ces quantités sont bornées supérieurement à l'optimum f ρ .

(f ρ ) ≤ b 1 , b 2 (f ρ ) ≤ b 2 .
(1.23) L'hypothèse sur le gradient remplace l'hypothèse sur le bruit dans le cadre des moindres carrés. En effet, si est la perte associée à la perte des moindres carrés, l'hypothèse sur le gradient est simplement |Y -f ρ (X)| ≤ b 1 , ce qui implique en particulier l'hypothèse de bruit avec σ, M = b 1 .

Notez que nous utiliserons les notations abrégées H, H λ , H n,λ pour désigner les quantités H(f ρ ), H λ (f ρ ), et H n,λ (f ρ ).

Exemples

Nous montrons deux exemples où l'hypothèse ACG autoconcordante généralisée Assumption 3.7 est satisfaite : la régression logistique et la perte au carré.

Example 1.2. Pour la régression logistique, nous avons z (t) = log(1 + exp(-yt). Si nous supposons que Z = X × [-1, 1] (ou plus généralement que les y sont bornés), est satisfaite avec R = 2. De plus, les z sont tous 1 lipschitz, et on a :

∀f ∈ H, b 1 (f ) ≤ 1, b 2 (f ) ≤ 1/4. (1.24)
Example 1.3. Pour une régression par les moindres carrés, on a z (t) = |t -y| 2 . Dans ce cas, ?? est satisfaite pour tout R positif. De plus, on a, pour toute distribution ρ pour Z,

∀f dansH, b 1 (f ) = |Y -f (X)| L ∞ (ρ) , b 2 (f ) ≤ 1.
(1.25)

Décomposition biais-variance

Dans ce contexte, la décomposition du biais et de la variance se fait par l'introduction de deux quantités clés, qui sont les analogues des quantités b λ et σ 2 d λ dans le cadre des moindres carrés.

• Pour tout λ > 0, le biais b λ est défini comme suit

b λ := ν(R λ , f ρ ) 2 = ∇R λ (f ρ ) 2 H -1 λ = λ 2 f ρ 2 H -1 λ .
(1.26)

• Pour toute λ > 0, la dimension effective df λ est définie comme suit

df λ := E H -1/2 λ Z (f ρ (X))k X 2 = Tr(H -1/2 λ GH -1/2 λ ), G = E Z (f ρ (X)) 2 k X ⊗ k X (1.27)
Dans le cadre des moindres carrés, nous avons H = Σ et

G = E (Y -f ρ (X)) 2 k X ⊗ k X .
(1.28)

La définition du biais b λ dans Eq. (1.26) correspond donc exactement à la définition donnée dans ?? dans le cas des moindres carrés. La définition de la dimension effective est légèrement CHAPTER 1. R ÉSUM É EN FRANC ¸AIS différente, et peut être reliée à la dimension effective d λ introduite dans le contexte des moindres carrés comme suit :

df λ ≤ σ 2 d λ ,
(1.29) où σ 2 est une limite supérieure de la variance du bruit E (Yf ρ (X)) 2 |X (voir ??). Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] prouvent la décomposition biais-variance suivante (nous désignons par a ∨ b le maximum entre a et b), qui est le résultat principal de leur article, et est rapporté ici dans le Théorème principal 1.

Main theorem 1: Marteau-Ferey et al. ( 2019), Theorem 4.

Il existe des constantes explicites n 0 , n 1 et C bias , C var telles que pour tout n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ b 2 , lorsque

n ≥ n 0 b 2 λ log b 2 λδ , n ≥ n 1 df λ ∨ q 2 r 2 λ log 2 δ , b λ ≤ r λ /2,
avec q 2 = b 2 1 /b 2 , alors avec une probabilité d'au moins 1 -2δ, il est vrai que

R( f n,λ ) -R(f ρ ) ≤ C bias b λ + C var df λ ∨ q 2 n log 2 δ .
(1.30)

Taux de convergence supérieurs

En suivant ce qui est fait dans Sec. (i) Condition de source : il existe h ∈ H tel que f ρ = H r-1/2 h et h H ≤ R. C'est exactement la condition de source dans le cadre des moindres carrés, mais adaptée au cas où la métrique n'est plus la métrique L 2 , mais la métrique associée à la fonction de perte .

(ii) Condition de capacité : les valeurs propres généralisées λ i de la paire (G, H λ ) (c'est-à-dire les valeurs propres de

H -1/2 λ GH -1/2 λ ) satisfont n i=1 λ i λ i + λ = Tr(H -1/2 λ GH -1/2 λ ) ≤ Q 2 λ -1/b . (1.31)
En particulier, ceci est satisfait dès que le λ i ≤ β i b pour un certain b > 1.

(iii) Borne sur les gradients et les Hessiennes : Assumption 1.1 est satisfaite avec b 1 , b 2 .

Sur cette classe de fonctions, les taux de convergence supérieurs non-asymptotiques suivants sont dérivés du théorème principal 1, qui a été prouvé à l'origine sous la forme Cor. 4.4.

TAUX ET ALGORITHMES RAPIDES POUR LES FONCTIONS AUTO-CONCORDANTES G ÉN ÉR

Main corollary 1: [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], Corollaire 3

Fixons b 1 , b 2 > 0, 1 ≥ r > 1/2, Q > 0,R > 0, et b ≥ 1 et soit M = M(R, r, Q, b, b 1 , b 2 ). Si nous définissons λ = 256Q 2 R 2 n 1 2br+1
,

(1.32) alors pour tout δ ∈ (0, 1/2], avec une probabilité d'au moins 1 -2δ,

∀ρ ∈ M, R( f n,λ ) -R(f ρ ) ≤ 8 R 2 256Q 2 R 2 n 2br 2br+1 log 2 δ , (1.33)
à condition que n ≥ N et où N est défini dans ??, dépend de tous les paramètres du modèle M ainsi que de δ, (en particulier, elle dépend polynomialement de log 1 δ ).

Ce résultat peut être adapté pour ressembler à Cor. 3.1 (en particulier, où la relation entre n et log 1 δ est plus explicite), ce qui, nous le supposons, pourrait alors conduire à des taux supérieurs minimax. Cependant, ceci n'est pas fait dans le travail de Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]. Notez que nous avons supposé une condition de source plus forte que la simple hypothèse que le modèle soit bien spécifiée, i.e., nous avons supposé r > 1/2. Dans le cas où r = 1/2, certains taux peuvent être obtenus, mais ils dépendent fortement des paramètres du problème, par une constante exponentielle e R fρ . Ceci est dû au fait que nous ne pouvons pas garantir la condition b λ ≤ r λ /2 dans ce contexte. Pour plus de détails, nous renvoyons à chapter 4 pour plus d'éléments sur cette question.

Comme expliqué dans le cas de la régression des moindres carrés, la résolution du problème de minimisation du risque empirique régularisé peut être coûteuse en termes de calcul. La première étape pour obtenir un meilleur algorithme consiste à réduire la dimension du problème grâce à l'une des techniques de réduction de dimension présentées dans Sec. 3.1.2. [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] considèrent le cas où la dimension est réduite par l'échantillonnage de m points Nyström x 1 , ..., x m soit uniformément, soit en utilisant des "scores" approximatifs associés à la matrice noyau K = (k(x i , x j )) dansR n×n . Rappelons que dans le dernier cas, la technique consiste à sous-échantillonner les indices i 1 , ..., i m de {1, ..., n} en utilisant une bonne approximation du vecteur de probabilité p i = l i (t)/ i l i (t), où l i (t) = e i K(K + ntI) -1 e i (pour plus de détails sur les scores, voir Eq. (3.42) ou Sec. 5.D .3). Les points de Nyström sont alors définis comme suit : x j = x i j . Lors de l'échantillonnage de Nyström, H est approximé par

H m =    x → m j=1 α j k(x, x j ) : α ∈ R m    , (3.35)
Cet ensemble peut être vu simplement comme un sous-espace de dimension m de H (il s'agit simplement de span(k x j : 1 ≤ j ≤ m)) avec la norme induite : m j=1 α j k(•, x j ), m j=1 β j k(x, x j ) Hm = α K mm β où K mm est la matrice du noyau associée au x j . Le noyau associé à l'espace H peut être exprimé comme k m (x, x ) = k m (x) K † mm k m (x) où k m (x) = (k( x j , x)) 1≤i≤m ∈ R m et † désigne le pseudo-inverse de Moore-Penrose. L'estimateur à dimension réduite est alors calculé comme suit Le théorème suivant présente un décomposition entre le biais et la variance dans le cas où la dimension est réduite en utilisant le sous-échantillonnage de Nyström. Il s'agit d'une réécriture d'un théorème de [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] qui peut être trouvé dans Theorem 5.6.

TAUX ET ALGORITHMES RAPIDES POUR LES FONCTIONS AUTO-CONCORDANTES G ÉN ÉR

Main theorem 2: Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] Il existe des constantes explicites n 0 , n 1 , m 0 , m 1 , C, et C bias , C var ainsi qu'une constante λ 0 qui dépend de b 2 telles que la proposition suivante soit vraie. Soit n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ λ 0 , et supposons

n ≥ n 0 1 + b 2 λ log 1 + b 2 λδ , n ≥ n 1 df λ ∨ q 2 r 2 λ log 2 δ , C b λ ≤ r λ /2,
avec q 2 = b 2 1 /b 2 . Supposons que la réduction de dimension ait été appliquée en utilisant le souséchantillonnage de Nyström, où les échantillons sont obtenus avec l'une des valeurs de m et des techniques de sous-échantillonnage suivantes.

(i) m ≥ m 0 1+b 2 λ log 8(1+b 2 ) λδ en utilisant un échantillonage uniforme. (ii) m ≥ m 1 d λ/(1+b 2 ) log 8(1+b 2 ) λδ en utilisant des t-scores approximés avec t = λ/(1 + b 2 ). Avec probabilité 1 -δ, il vient

R( f n,λ,m ) -R(f ρ ) ≤ C bias b λ + C var df λ ∨ q 2 n log 2 δ , (1.37) et R f n,λ,m ≤ R f ρ + 10.
La preuve de cette proposition est faite dans chapter 5. Pour intégrer le fait que nous réduisons la dimension, la quantité clé à controler est H 1/2 (I -P m ) où P m est la projection orthogonale sur H m (le sous-ensemble de H défini par les points de Nyström). Nous montrons que l'approximation de H par H m est statistiquement optimale dès que H 1/2 (I -P m ) ≤ c √ λ pour une certaine constante c (voir Sec. 5.H .3 pour plus de détails). Il y a ici une subtilité dans le cas où l'on utilise un échantillonnage avec des scores. En effet, l'objectif idéal est d'approximer l'opérateur H avec le moins de points de Nystrom possible. Cependant, nous ne pouvons calculer aucun type d'approximation des scores de l'opérateur empirique associé H n puisque f ρ n'est pas connu ( H n = H n (f ρ )). Au lieu de cela, nous savons comment approximer les scores de la covariance empirique Σ associée à l'opérateur de covariance Σ, et puisque H b 2 Σ, nous pouvons utiliser ceci pour obtenir une garantie √ b 2 Σ 1/2 (I -P m ) ≤ c √ λ en utilisant ces scores, ce qui impliquera la limite désirée. C'est pourquoi au point (ii) du théorème principal 2, la dimension effective de l'opérateur de covariance d λ/(1+b 2 ) apparaît.

Les résultats précédents conduisent au taux de convergence par au dessus informel suivant, qui est une reformulation d'un sous-cas de Theorem 5.7.

CHAPTER 1. R ÉSUM É EN FRANC ¸AIS Main corollary 2: Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], Theorem 7

Fixons b 1 , b 2 > 0, r > 1/2, Q > 0,R > 0, et b ≥ 1 et soit M = M(R, r, Q, b, b 1 , b 2 ). Let λ = Q 2 R 2 n 1 2br+1
,

(1.38) et soit δ ∈ (0, 1/2]. Supposons que la réduction de dimension ait été appliquée en utilisant le sous-échantillonnage de Nyström, où les échantillons sont obtenus avec l'une des valeurs de m et des techniques de sous-échantillonnage suivantes (i) m ≥ m 0 1+b 2 λ log 8(1+b 2 ) λδ en utilisant un échantillonage uniforme. (ii) m ≥ m 1 d λ/(1+b 2 ) log 8(1+b 2 ) λδ en utilisant des t-scores avec t = λ/(1 + b 2 ). Avec probabilité au moins 1 -2δ,

∀ρ ∈ M, R( f n,λ,m ) -R(f ρ ) ≤ C R 2 Q 2 R 2 n 2br 2br+1 log 2 δ , (1.39)
où C est une constante explicite, à condition que n ≥ N et où N peut être caractérisé à l'aide de la preuve de Theorem 5.7, dépend de tous les paramètres du modèle M ainsi que de δ, (en particulier, elle dépend de façon polynomiale de log 1 δ ).

Notez que le résultat ci-dessus peut être rendu formel, mais pour des raisons de lisibilité, nous l'avons laissé sous cette forme. Notez que contrairement au cas des moindres carrés, nous ne donnons pas explicitement l'ordre de m en fonction de n. Cela pourrait être fait en supposant une autre diminution de la valeur propre pour la dimension effective de la matrice de covariance dans les paramètres du modèle. Cependant, pour des raisons de lisibilité, nous garderons cette hypothèse implicite.

Algorithme

Pour finir, notons que dans le théroème principal 10, nous détaillons un algorithme pour calculer une bonne approximation de f n,λ,m . En utilisant cet algorithme, nous obenons le théorème suivant.

1.3. MOD ÈLES SEMI-D ÉFINIS POSITIFS : TH ÉORIE ET APPLICATIONS

29

Main theorem 3: Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], Proposition 14

Il existe des constantes explicites n 0 , n 1 , m 0 , m 1 , C, et C bias , C var ainsi qu'une constante λ 0 qui dépend de b 2 telles que l'énoncé suivant est vrai. Soit ε > 0, n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ λ 0 et supposons

n ≥ n 0 1 + b 2 λ log 1 + b 2 λδ , n ≥ n 1 df λ ∨ q 2 r 2 λ log 2 δ , C( b λ + √ ε) ≤ r λ /2,
avec q 2 = b 2 1 /b 2 . Supposons que la réduction de dimension ait été appliquée à l'aide du sous-échantillonnage de Nyström en utilisant des scores de levier approximatifs associés à la covariance empirique avec t = λ/(1

+ b 2 ) et m ≥ m 1 d λ/(1+b 2 ) log 8(1 + b 2 ) λδ .
Supposons que nous effectuons le schéma de globalisation en utilisant les paramètres donnés dans le théorème principal 10 avec une précision de ε, et que nous calculons les étapes approximatives de Newton en utilisant une méthode d'esquisse comme celle de [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF], afin d'obtenir f n,λ,m . Avec probabilité au moins 1 -δ, nous avons

R( f n,λ,m ) -R(f ρ ) ≤ C bias b λ + C var df λ ∨ q 2 n log 2 δ + ε, (1.40)
et la complexité en temps pour calculer f n,λ,m est d'ordre

(nm log n + mq 2 + q 3 ) R f ρ log R ∇F (0) λ + log 2 (1 + (λ -1 /R 2 )) m, q = O d λ/b 2 log b 2 λδ ,
(1.41)

Le "vrai" résultat est légèrement plus complexe. En particulier, il définit la dimension effective pour toutes les Hessiennes croisées, et relie statistiquement les dimensions effectives empiriques aux dimensions idéales. Ce théorème peut être étendu pour obtenir des taux de convergence supérieurs comme les corollaires précédents (voir le théorème 7 de Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]).

La principale conclusion de ce résultat est que nous pouvons calculer un estimateur statistiquement optimal en un temps grossièrement d'ordre O(nd λ + d 3 λ ) (nous cachons les constantes logarithmiques). En particulier, dans le cas bien spécifié, Eq. (1.41) montre que la complexité est indépendante du conditionnement du problème. Il s'agit d'une propriété très souhaitable dans les problèmes de classification où nous sommes amenés à considérer de très petites valeurs de λ, comme dans les deux exemples dans Fig. 3.1 tirés du travail de Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], où nous comparons notre méthode avec un concurrent du premier ordre, qui a une dépendance moins bonne en le conditionnement.

Modèles semi-définis positifs : théorie et applications

Cette partie est consacrée au développement d'un modèle pour les fonctions positives que nous appelons les modèles PSD. Le premier article définit et analyse les principales propriétés CHAPTER 1. R ÉSUM É EN FRANC ¸AIS de ces modèles, tandis que le second applique un tel modèle pour concevoir un algorithme d'échantillonnage i.i.d. à partir d'une fonction de densité non normalisée. Ces deux articles ont été publiés.

• Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Non-parametric Dans chapter 6, nous décrivons le travail de Marteau-Ferey, [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. Nous considérons une classe de modèles à sorties non négatives, qui présentent les mêmes propriétés que les modèles linéaires et les méthodes à noyaux. Ce modèle consiste à considérer des fonctions paramétrées par un opérateur PSD sur un espace de Hilbert H sur lequel l'espace d'entrée X est cartographié par φ : X → H avec la notation des modèles linéaires et x ∈ X → k x ∈ H dans la notation ENR :

{f A : A 0} f A (x) = φ(x) Aφ(x) ou f A (x) = k x , Ak x H , x ∈ X .
(1.42)

Nous appelons ces modèles modèles PSD. Comme ce modèle est lui-même linéaire, il peut directement être utilisé de la même manière pour résoudre les problèmes de MRE. Eqs. (1.10) and (1.13). Nous établissons un théorème des représentants similaire à Theorem 1.2 pour nos modèles dans le contexte de la minimisation du risque empirique, et nous fournissons une formulation duale convexe en dimension finie du problème d'apprentissage, qui ne dépend que des exemples d'entraînement. Nous prouvons également que ce modèle possède de bonnes propriétés d'approximation, et nous appliquons la méthode aux problèmes d'estimation de densité, de régression avec des erreurs hétéroscédastiques gaussiennes, et de régression de plusieurs quantiles simultanément. Nous établissons les algorithmes d'apprentissage correspondants pour la formulation duale convexe, et les comparons aux techniques standard utilisées pour les problèmes spécifiques sur quelques simulations de référence.

Dans le chapitre chapter 7, nous décrivons le travail de Marteau-Ferey, Bach, and Rudi (2022a). Ce travail applique le cadre de modélisation ci-dessus au problème de l'échantillonnage de n échantillons i.i.d. à partir d'une distribution dont la densité est connue jusqu'à une constante grâce aux évaluations de fonctions. Contrairement à la plupart des méthodes existantes dans la littérature telles que l'échantillonnage par rejet ou les méthodes de Monte-Carlo par chaîne de Markov [START_REF] Gelman | Bayesian Data Analysis[END_REF][START_REF] Jun | Monte Carlo Strategies in Scientific Computing[END_REF][START_REF] Lelièvre | Free Energy Computations[END_REF][START_REF] Robert | Monte Carlo statistical methods[END_REF], nous proposons une procédure en deux étapes en modélisant d'abord cette densité à l'aide d'un modèle semi-défini positif, puis en échantillonnant à partir de ce modèle PSD à l'aide d'un algorithme adapté. En particulier, nous utilisons des modèles PSD avec le noyau gaussien défini dans Sec. 2.2.2 (nous appellerons ces modèles des modèles PSD gaussiens), c'est-à-dire que nous approchons la densité cible par une fonction de la forme

f A (x) = n i,j=1
A ij k σ (x i , x)k σ (x j , x), A 0, k σ (x, x ) = exp(-xx 2 /(2σ 2 )).

(1.43)

La modélisation des distributions de probabilité avec des modèles PSD gaussiens a été développée par [START_REF] Rudi | Psd representations for effective probability models[END_REF]. Dans ce travail, nous continuons à explorer les bonnes propriétés de ce modèle pour les distributions de probabilité, et nous établissons un algorithme facile à implémenter qui peut générer un nombre arbitraire d'échantillons i.i.d. à partir d'un modèle PSD gaussien donné, avec une précision donnée.

Nous montrons ensuite que nous pouvons échantillonner un nombre arbitraire d'échantillons i.i.d. à partir d'une distribution de probabilité cible qui est suffisamment régulière, avec une précision donnée. L'algorithme consiste à (a) approximer la densité non normalisée p via un modèle PSD, en utilisant des évaluations de p, et (b) extraire des échantillons i.i.d. du modèle PSD. Nous montrons que pour des densités suffisamment régulières, le modèle PSD résultant est concis et évite la malédiction de la dimension : pour atteindre l'erreur ε, le modèle PSD gaussien requiert un nombre de paramètres et un nombre d'évaluations de p qui sont de l'ordre de ε -2-d/β , où d est la dimension de l'espace et β est l'ordre de différentiabilité de la densité.

Sommes de carrés de fonctions et optimisation globale

Cette dernière partie est consacrée à l'utilisation des modèles PSD dans le contexte de l'optimisation globale. Le premier article développe une méthode d'optimisation globale de fonctions régulières qui tire parti de la régularité pour briser la malédiction de la dimensionnalité. Le deuxième article est plus théorique et fournit des conditions suffisantes pour que les fonctions régulières soient décomposées en somme de carrés de fonctions régulières. Cette propriété est cruciale pour notre schéma d'optimisation globale. Ces deux articles ont été soumis et sont en cours de révision.

• Alessandro Rudi, Ulysse Marteau-Ferey, and Francis Bach. Finding global minima via kernel approximations, 2020. URL https://arxiv.org/abs/2012.11978

• Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Second order conditions to decompose smooth functions as sums of squares, 2022b. URL https://arxiv.org/abs/ 2202.13729

Rudi, Marteau-Ferey, and Bach (2020) décrivent un cadre et un algorithme permettant de minimiser des fonctions f de la classe C r sur un sous-ensemble ouvert X de R d . Pour ce faire, nous formulons le problème min x∈X f (x) comme un problème convexe sup c

subject to c ∈ R, g = fc, g ≥ 0, (2.25) où les contraintes fonctionnelles fc = g et g ≥ 0 doivent être vérifiées pour tous les x ∈ X . De manière similaire au cas des polynomes [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF], nous renforçons la contrainte g ≥ 0 en demandant à g d'être un modèle PSD de la forme :

g A (x) = k x , Ak x Hs , A 0, Tr(A) < ∞, (1.44)

CHAPTER 1. R ÉSUM É EN FRANC ¸AIS où H s est l'espace de Sobolev sur X équipé du noyau de Sobolev k s . ?? est donc renforcé comme suit sup c

subject to c ∈ R, f (x) -c = k x , Ak x Hs , x ∈ X , A 0, Tr(A) < ∞.

(1.45)

Pour résoudre un problème de dimension finie, et en supposant que nous ayons accès à f par les valeurs de la fonction, nous utilisons le problème de substitution suivant qui remplace la contrainte fc = g A par la même contrainte évaluée en n échantillons x 1 , ..., x n : (1.46) où nous avons ajouté un terme de régularisation supplémentaire λ Tr(A) pour éviter le surapprentissage dû à la quantité finie de données. Notez que ce problème n'est ni un renforcement ni une relaxation du problème initial, ce qui est l'une des principales différences avec l'optimisation polynomiale.

sup c -λ Tr(A) subject to c ∈ R, f (x i ) -c = k x i , Ak x i Hs , x ∈ X , A 0,
Rudi, Marteau-Ferey, and Bach (2020) montrent que le SDP de Eq. (1.46) peut être résolu avec une erreur de ε en utilisant une méthode de Newton en temps O(n 3.5 log 1 ε ). De plus, nous prouvons qu'en utilisant le noyau de Sobolev k s avec s = r -3, l'estimation obtenue c n,λ du minimum f * satisfait approximativement c n,λ -f * ≤ C r,d n -r/d+1/2+3/d , λ = n -r/d+1/2 , (1.47) où r est la régularité de f , C r,d est une constante qui dépend de la dimension d, de f , et de sa régularité r, mais pas de n. L'intérêt de ce résultat est de montrer que cet algorithme brise effectivement la malédiction de la dimension dans les taux d'optimisation globale, en tirant parti de la régularité pour obtenir une convergence d'ordre n -r/d (par opposition aux taux lents typiques d'ordre n -1/d ). Notez cependant que les constantes peuvent encore dépendre exponentiellement en d. [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] fournissent également des moyens de calculer le minimiseur global.

La principale hypothèse pour que Eq. (1.47) soit vraie est que Eq. (1.45) ait une solution, c'est-à-dire qu'il existe un opérateur de trace finie A tel que f (x) -f * = k x , Ak x Hs pour tous les x ∈ X . Une condition suffisante pour que cela soit vrai est le fait que ff * soit une somme finie de carrés de fonctions f i ∈ C s (X ). Dans Rudi, Marteau-Ferey, and Bach (2020), nous supposons que f n'a que des minima stricts (c'est-à-dire que les hessiennes aux minima sont définies positives), et que f reste très éloigné de son minimum à la frontière de X , c'est-à-dire , que {f -f * ≤ δ} soit compact dans X pour un certain δ > 0.

Marteau-Ferey, Bach, and Rudi (2022b) fournissent une hypothèse pour qu'une fonction f ∈ C s+2 soit décomposée comme une somme de carrés de fonctions de classe C s qui généralise celle cidessus, et permet de traiter les fonctions qui sont définies sur des variétés, et qui ont des variétés de zéros. La condition que nous trouvons est que la Hessienne de f doit être défini positivement orthogonalement aux variétés de zéros. C'est un cas important pour les situations où les fonctions f ont de "grands" ensembles de zéros, comme en transport optimal (voir Vacher, [START_REF] Vacher | A Dimension-free Computational Upper-bound for Smooth Optimal Transport Estimation[END_REF]). Ceci est lié dans l'esprit aux différents Positivstellensätzen en optimisation polynomiale, pour garantir la convergence des algorithmes.

Pour présenter l'article de Rudi, Marteau-Ferey, and Bach (2020), nous adoptons une approche différente de l'article, dont le verbatim peut être trouvé dans chapter 9. Nous commençons par présenter la méthode de manière générale, sans garanties, dans Sec. 1.4.1, avant de l'appliquer dans le cadre considéré par [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] où des garanties peuvent être obtenues, dans Sec. 1.4.2.

Notations. Dans cette section, nous allons considérer un ENR H sur un ensemble X associé au noyau k, que nous supposerons séparable (i.e., il existe une suite dense au plus dénombrable dans H). Nous désignons par S(H) l'ensemble des opérateurs symétriques bornés sur H. En particulier, dans cette section, nous considérons l'ensemble S 1 (H) des opérateurs symétriques de trace finie sur H [START_REF] Weidmann | Linear Operators in Hilbert Spaces[END_REF] équipé de la norme trace (notez que nous pourrions prendre d'autres normes spectrales, telles que la norme de Hilber-Schmidt, comme expliqué dans chapter 6). Nous désignerons par S 1,+ (H), ou simplement S + (H) (la norme considérée sera toujours la norme de trace dans cette section), l'ensemble des opérateurs symétriques postivie semi-définis de classe trace sur H.

Une méthode générique fondée sur les noyaux

Dans cette section, nous présentons les principaux ingrédients de la méthode développée par [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. Nous présentons ces ingrédients en toute généralité, sans supposer que X = R d par exemple, mais sans fournir aucune garantie à ce stade.

La méthode repose sur trois étapes principales.

(i) Comme dans le cas polynomial, nous commençons par considérer un noyau équivalent de la relaxation moment-SOS.

(ii) Nous considérons ensuite une discrétisation régularisée de cette relaxation, dans l'esprit de la minimisation du risque empirique.

(iii) Enfin, en utilisant le théorème des représentants, nous résolvons efficacement cette version discrétisée régularisée.

Relaxations primales et duales

La première étape consiste à considérer les relaxations primale et duale suivantes du problème de mesure Eq. ( 8.2) and ??. Il est plus facile de voir les choses dans le dual : on remplace la contrainte de positivité fc ≥ 0 par la contrainte que fc soit un modèle PSD introduit dans part II. Ceci est fait dans Eq. ( 9.3) dans le travail de Rudi, Marteau-Ferey, and Bach (2020).

Définissons

SOS H 0 = {f : X → R : f (x) = k x , Ak x H , x ∈ X , A ∈ S + (H), rank(A) < ∞} , SOS H = {f : X → R : f (x) = k x , Ak x H , x ∈ X , A ∈ S + (H)} .
(1.48)

Notons que, comme expliqué dans chapter 6, SOS H 0 est simplement l'ensemble des sommes finies des carrés des fonctions dans H, alors que SOS H est sa complétion pour la norme de trace, définie comme f 1 = min{Tr(A) : f (x) = k x , Ak x H , x ∈ X , A ∈ S + (H)}. En renforçant la contrainte fc ≥ 0 par fc ∈ SOS H , on obtient le renforcement suivant du problème dual :

CHAPTER 1. R ÉSUM É EN FRANC ¸AIS ρ * H = sup c subject to f (x) -c = g(x), x ∈ X , g ∈ SOS H
(1.49)

Cela correspond à la relaxation suivante du problème primal sur les mesures :

ρ H = inf X f (x)µ(dx) subject to µ ∈ M (X ), X µ(dx) = 1, X k x ⊗ H k x µ(dx) 0,
(1.50) où la contrainte de positivité de la mesure est relaxée par la contrainte PSD X k x ⊗ H k x µ(dx) 0 sur les mesures signées.

Comme ces problèmes sont des relaxations/renforcements du problème original, nous avons les garanties suivantes:

ρ * H ≤ ρ H ≤ f * = inf x∈X f (x).
(1.51)

Note sur les polynômes. Comme nous le verrons dans Sec. 8.3 , la relaxation présentée ci-dessus correspond exactement aux relaxations moment-SOS sans contrainte polynomiale, si l'on considère l'ENR H = R r [x] (la structure ENR peut être facilement obtenue, voir Sec. 8.3 ).

Note sur les moments. Il n'est pas aussi évident de définir le problème des moments comme dans le cas polynomial, puisque nous n'avons rien supposé sur f à ce stade. Il est possible de formuler le problème des moments si l'on suppose que f appartient à un certain espace, comme nous le verrons dans Sec. 8.3 . Pour le reste de cette section, nous nous en tiendrons au problème de mesure en tant que problème primal, sans le simplifier en problème de moment comme c'est le cas dans le cadre polynomial.

Note sur les bornes. Comme nous le verrons en appliquant cette méthode avec le noyau de Sobolev dans Sec. 1.4.2, dans le cadre ENR de dimension infinie (non paramétrique), il est plus facile de montrer que Eq. (1.49) sont tendues, c'est-à-dire , ρ H = ρ * H = f * . Cependant, comme nous le verrons dans la section suivante, il sera crucial d'avoir plus que ce type de garanties, et d'avoir une solution optimale dans le cas dual Eq. (1.49) (i. e., l'existence de c * , g * qui résolvent le problème et atteignent le supremum), afin d'obtenir de bonnes propriétés de la méthode, puisque nous ne résolvons pas directement Eq. (1.49), mais l'approchons. Cela correspond à une hypothèse bien spécifiée dans ??, car nous adopterons la même approche "MRE régularisée" pour résoudre le problème Eq. (8.32).

Évaluer la contrainte sur un nombre fini de points Bien que ces relaxations primale et duale soient représentables par des matrices PSD, elles ne sont pas de dimension finie, du fait que H est potentiellement de dimension infinie. Dans un deuxième temps, nous adoptons donc la même stratégie que lors de la minimisation du risque empirique, et au lieu de garantir que l'égalité dans Eq. (1.49) est valable en tout point de X , nous restreignons cette contrainte à un nombre fini de points. Nous nous donnons une séquence (x i ) 1≤i≤n ∈ X n de points tests, correspondant à un sous-ensemble X n = {x i : 1 ≤ i ≤ n} ⊂ X de taille n, et nous résolvons le problème régularisé suivant correspondant à Eq. (1.49) :

ρ * n,λ = sup c -λ g 1 subject to f (x i ) -c = g(x i ), 1 ≤ i ≤ n, g ∈ SOS H , (1.52)
que nous pouvons reformuler comme le problème suivant sur les opérateurs semi-définis de trace finie :

ρ * n,λ = sup c -λ Tr(A) subject to f (x i ) -c = k x i , Ak x i H , 1 ≤ i ≤ n, A ∈ S + (H).
(1.53)

Cela correspond à la réécriture suivante du problème primal sur les mesures, où les mesures sont restreintes pour avoir un support dans X n :

ρ n,λ = inf Xn f (x)µ(dx) subject to µ ∈ M (X n ), Xn µ(dx) = 1 Xn k x ⊗ H k x µ(dx) + λI H 0.
(1.54)

En écrivant une mesure signée à support dans X n comme µ = n i=1 α i δ x i , nous obtenons le dual suivant de Eq. (1.53)

ρ n,λ = inf n i=1 α i f (x i ) subject to (α i ) ∈ R n , n i=1 α i = 1 n i=1 α i k x i ⊗ H k x i + λI H 0.
(1.55) Différence avec les hiérarchies SOS. Nous voulons souligner que c'est cette modification du problème Eq. (1.49) qui rend cette méthode fondamentalement différente de la méthode polynomiale. En effet, en passant de Eq. (1.49) à Eq. (1.52) les phénomènes suivants apparaissent.

(i). Prenons λ = 0. La restriction à l'ensemble X n rend le problème primal plus difficile (nous restreignons l'ensemble des mesures aux sommes de diracs sur X n ) et le problème dual plus facile (puisque nous n'appliquons la contrainte d'égalité que sur un sous-ensemble fini de X ). Ainsi, l'opération de restriction à X n est un renforcement du primal, et une relaxation du dual : ρ n,0 ≥ ρ H et ρ * n,0 ≥ ρ * H . Dans le cas où nous n'avons pas de garanties a priori sur les problèmes Eqs. (1.49) and (1.50), nous ne pouvons rien dire sur ρ * n,0 , ρ n par rapport à f * , comme nous avions ρ H , ρ * H ≤ f * . Même dans le cas où l'on peut montrer que Eqs. (1.49) and (1.50) sont tendus a priori, si la classe de fonctions SOS H est trop grande (et ce sera généralement le cas dès que H est de dimension infinie, par exemple dans le cadre du noyau de Sobolev de chapter 9), le problème reste compliqué car nous avons besoin de régularisation. En effet, dans le cas où l'espace SOS H est grand, il est possible qu'il puisse interpoler toute fonction fc avec c ≤ min 1≤i≤n f (x i ). Dans ce cas, nous avons ρ * n,0 = min 1≤i≤n f (x i ), ce qui est une borne supérieure triviale sur le minimum et peut être obtenu sans une méthode aussi élaborée.

CHAPTER 1. R ÉSUM É EN FRANC ¸AIS (ii). Dès que H et donc SOS H est trop grand, une régularisation est nécessaire pour éviter le surapprentissage ou l'interpolation. Comme l'espace SOS H est doté d'une norme naturelle donnée par la norme de la trace, il est naturel de régulariser le problème dual (avec la contrainte SOS) avec cette norme. Ceci correspond à une forme de renforcement du dual et de relaxation du primal, et nous avons donc ρ n,λ ≤ ρ n,0 et ρ * n,λ ≤ ρ * n,0 . Puisque les bornes vont dans le sens inverse des bornes entre les problèmes discrétisés et non discrétisés, il n'y a aucun moyen, même lorsque le problème SOS original est tendu, de savoir si ρ * n,λ , ρ n,λ sont des bornes inférieures ou supérieures pour f * . Cependant, comme c'est le cas dans le cadre de la MRE., l'interaction entre n et λ permet d'obtenir des bornes supérieures asymptotiques entre ρ * H et ρ * n,λ sous certaines conditions, comme nous le verrons dans Sec. 1.4.2.

Algorithme

En regardant Eq. (1.52), nous voyons que nous pouvons appliquer proposition 6.3 dans chapter 6, i.e., le théorème des représentants pour les modèles PSD. En suivant le raisonnement dans chapter 9, définissons Φ i la i-ième colonne d'une matrice telle que Φ Φ = K, où K est la matrice noyau K = (k(x i , x j )) 1≤i,j≤n . Le problème peut être écrit en dual sous la forme

ρ * n,λ = sup c -λ Tr(A) subject to f (x i ) -c = Φ i AΦ i , 1 ≤ i ≤ n, A 0, (1.56)
et le primal :

ρ n,λ = inf n i=1 α i f (x i ) subject to (α i ) ∈ R n , n i=1 α i = 1, n i=1 α i Φ i Φ i + λI R n 0.
(1.57)

Ces deux problèmes sont des programmes semi-définis linéaires dans le cône S + (R n ). Si, par exemple, nous résolvons la formulation primale en utilisant des méthodes de points intérieurs, nous pouvons obtenir une ε-solution en temps O(n 3.5 log 1 ε ). Ces méthodes sont détaillées dans chapter 9, dans algorithm 6 and Sec. 9.6 .

Remarque sur l'extraction. Contrairement au cas polynomial, où une stabilisation du rang était une condition suffisante pour trouver un minimiseur, ce n'est pas directement une condition suffisante pour extraire un minimiseur associé à l'estimation ρ n,λ ou ρ * n,λ du minimum. Une façon de procéder si X est équipé d'une métrique d X est de chercher x * comme solution au problème suivant :

x * = arg min x∈X n i=1 α i d X (x, x i ) 2 = X d X (x, x ) 2 µ n (dx ) = X ×X d X (x 1 , x 2 ) 2 µ n (dx 1 )δ x (dx 2 ),
(1.58)

SOMMES DE CARR ÉS DE FONCTIONS ET OPTIMISATION GLOBALE

37

Où δ x est juste un dirac dans X , et µ n = n i=1 α i δ x i est la mesure signée avec support dans X n qui est la solution à Eq. (1.54). En d'autres termes, on cherche x * tel que la mesure de dirac δ x * soit aussi proche que possible en norme de Wasserstein-2 [START_REF] Santambrogio | Optimal Transport for Applied Mathematicians[END_REF] de la mesure signée µ n . Dans le cas où X = R d , la solution de Eq. (1.58) est simplement x * = n i=1 α i x i . La méthode choisie par [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] est essentiellement celle-ci. Notez que pour que cette stratégie soit bien motivée, il est important que le minimiseur soit unique.

1.4.2 Application de cette méthode à la minimisation de fonctions régulières dans l'espace euclidien

Dans cette section, nous expliquons comment [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] appliquent la méthode décrite dans Sec. 1.4.1 dans le cadre de la minimisation d'une fonction f appartenant à C r b (Ω), où Ω est un domaine ouvert de R d , et où C r b (Ω) désigne l'ensemble des fonctions r fois continuement différentiables avec des dérivées bornées.

L'objectif de ce travail, d'un point de vue général, est de tirer parti de la régularité dans l'optimisation globale, même si celle-ci est effectuée en utilisant uniquement les valeurs des fonctions. Étant donné ε > 0, nous souhaitons trouver une telle approximation avec le plus petit nombre d'appels à la fonctiof (i.e., le plus petit nombre possible de x i , 1 ≤ i ≤ n dans Sec. 1.4.1), et avec des garanties dans le pire des cas sur toutes les fonctions f dans une certaine classe pertinente de fonctions F ⊂ C r b (Ω). Nous voulons que la méthode qui renvoit x satisfasse :

Problème et pire des cas

sup f ∈F f (x) -min x∈Ω f (x) ≤ ε.
(1.59) [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF] prouve que lorsqu'on a accès à n valeurs de fonction, le problème d'optimiser f est aussi difficile que le problème d'approcher f uniformément (voir l'introduction et la section 1.3 de [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF] et Fig. 1.1 pour une représentation visuelle). Dans le cas où f ∈ C r b (Ω), la section 1.3.9 de Novak (2006) montre que le problème d'optimisation est effectivement aussi dur que l'approximation, et que n =∝ ε -d/r points sont nécessaires pour obtenir une approximation ε (les constantes de proportionnalité peuvent dépendre exponentiellement de d). L'objectif des travaux de Rudi, Marteau-Ferey, and Bach (2020) est d'égaler ces limites, c'est-à-dire d'exposer un algorithme qui ne nécessite que n ∝ ε -d/r points d'évaluation pour obtenir une erreur de ε.

Nous disons qu'un tel algorithme brise la malédiction de la dimension de l'exposant en utilisant la régularité dans le sens où le nombre d'échantillons nécessaires dans le pire des cas n'est pas n ∝ ε -d (comme c'est le cas dans le cas de Lischitz et des algorithmes de base), mais d'ordre ε -d/r . Notez que la malédiction de la dimensiondénote parfois le fait que nous avons besoin de beaucoup plus de régularité pour optimiser en haute dimension : dans ce cas, bien sûr, nous ne brisons pas cette malédiction spécifique, mais ajoutons plutôt à son énoncé.

CHAPTER 1. R ÉSUM É EN FRANC ¸AIS x f (x) f (x) -ε f (x) + ε f (x)
Figure 1.1: Optimization is as hard as approximation.

Hypothèses

Le travail de Rudi, Marteau-Ferey, and Bach (2020) est effectué dans le cadre suivant. Supposons que Ω est un sous-ensemble ouvert borné de R d , que f ∈ C r b (Ω) pour quelque r ≥ 0, et que l'hypothèse suivante est satisfaite.

Assumption 1.2. Il existe ε > 0 tq {x ∈ X : f (x) -f * ≤ ε} est compact.

Assumption 1.2 indique qu'à proximité de l'optimum, les ensembles de niveaux sont compacts. Cela implique a) l'existence d'au moins un minimiseur, et b) qu'il n'y a pas de minimiseur près de la frontière.

Enfin, [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] prennent le noyau de Sobolev k s avec s > d/2 (voir Eq. (2.38)) pour appliquer la méthode présentée dans Sec. 8.2 .1. Notez que ce choix de noyau est utile en raison du lien direct avec la différentiabilité, et rend l'analyse claire. Cependant, comme on peut le voir dans chapter 7, on peut en fait approximer un tel noyau en utilisant un noyau gaussien, avec lequel les calculs sont plus faciles.

Sous ces hypothèses, nous pouvons facilement prouver que Eq. (1.49) résout en fait le problème de minimisation, i.e., ρ * H = ρ H = f * .

(1.60) Il s'agit d'une différence majeure avec le cadre polynomial, où l'équivalences des problèmes SOS et normaux Eq. (8.43) ne se produisait que si ff * était une somme de carrés de polynômes. Dans notre contexte, l'espace SOS H est suffisamment grand pour contenir toute approximation proche de ff * . Cependant, cela ne nous donne aucune information sur ρ n,λ , le résultat de la méthode de Rudi, Marteau-Ferey, and Bach (2020).

De manière similaire au cadre bien spécifié présenté dans part I, afin d'obtenir des garanties sur ρ n,λ , il est utile que le problème soit bien spécifié, i. e., que les solutions de Eqs. (1.49) and (8.33) soient atteintes en un certain point µ * pour le primal ou (c * , g * ) pour le dual.

Garanties sur les relaxations

La première étape du travail de Rudi, Marteau-Ferey, and Bach (2020) est donc de montrer que sous certaines hypothèses supplémentaires, il existe une solution exacte de Eq. (1.49). Pour ce faire, on fait l'hypothèse suivante sur l'ensemble des minimiseurs.

∀x ∈ Ω, f (x) -f * = k x , A * k x H .
(1.61)

En particulier, cela montre que Eq. (1.49) a une solution (c * , g * ) avec

c * = f * et g * = k (•) , A * k (•) H .
La preuve de ce théorème est basée sur une décomposition locale comme une somme de carrés autour de chaque minimum global, avant de coller les décompositions ensemble en utilisant des lemmes de collage standard de géométrie différentielle. En utilisant le fait que la Hessienne est définie positive, nous effectuons une décomposition exacte de type Taylor au deuxième ordre autour de chaque minimum global : c'est pourquoi nous perdons un exposant de 2 dans les limites, i.e., r ≥ s + 2. Cette approche est généralisée dans Marteau-Ferey, Bach, and Rudi (2022b) afin de passer aux fonctions qui ont un ensemble continu de zéros, et qui sont définies sur une variété (voir Sec. 1.4.3 pour plus de détails).

Garanties après restriction à un nombre fini de points et algorithme quasi-optimal La deuxième étape de la méthode présentée par [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] résultant, les auteurs utilisent la distance de remplissage h Xn,Ω de X n dans Ω, définie comme suit

h Xn,Ω = sup x∈Ω inf x i ∈Xn x -x i R d . (1.62)
Le théorème simplifié suivant borne la performance de ρ * n,λ , qui est la solution de Eq. (1.52). Les détails peuvent être trouvés dans Theorem 9.5 et a été prouvé à l'origine comme Théorème 5. par [START_REF] Rudi | Finding global minima via kernel approximations[END_REF].

-f * = g * , on a |ρ * n,λ -f * | ≤ C r ,d |f | Ω,r h r Xn,Ω + λ g 1 , (1.63) dès que h Xn,Ω ≤ h R,r , λ ≥ C r ,d h r Xn,Ω .
La preuve de ce théorème repose sur les limites obtenues par l'analyse des données dispersées [START_REF] Wendland | Scattered Data Approximation[END_REF], et sur le fait que H s s'emboîte dans C r (Ω) pour tout r < sd/2.

En supposant que Ω est une boule de rayon R, et que nous pouvons générer des échantillons i.i.d. de la mesure uniforme sur la boule, nous pouvons choisir les points x i au hasard avec la garantie suivante sur la distance de remplissage.

Lemma 1.1 (Lemma 9.4). Soit X n = {x 1 , . . . , x n } des points indépendants échantillonnés à partir de la distribution uniforme sur Ω. Il existe une constante n d dépendant uniquement de d telle que pour tout δ > 0, si n ≥ n d log 2 δ , alors avec une probabilité d'au moins 1 -δ :

h Xn,Ω ≤ 11R n -1 d (log 2 d n δ ) 1/d . (1.64)
Notons que ?? est un résultat plus général pour les ensembles qui vont au-delà de la simple boule de rayon R. Dans le cas d'un hypercube, on pourrait simplement prendre une grille X n , et on aurait h Xn ≤ Cn -1/d où n est le nombre de points dans la grille.

Nous énonçons maintenant Theorem 9.6, dans le cas où nous échantillonnons n points de la mesure uniforme lorsque Ω est une boule de rayon R.

Main theorem 6: [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]

∈ (0, 1). Si n ≥ n 0 log 2 δ et λ satisfait à λ ≥ C 0 n -r d (log 2 d n δ ) r d , (1.65) alors, avec une probabilité d'au moins 1 -δ, | ρ * n,λ -f * | ≤ 3 λ g * + |f | Ω,r .
(1.66)

En particulier, en supposant que r > d/2 + 2 et en prenant s = r -2 et r = r -3 -d/2 et en prenant λ n comme dans la borne inférieure de ??, on a des constantes n 1 , C 1 ne dépendant que de r, d telles que dès que n ≥ n 1 log 2 δ , on a

| ρ * n,λ -f * | ≤ C 1 g * + |f | Ω,r n -(r/d-3/d-1/2) (log 2 d n δ ) r/d-2/d-1/2 . (1.67)
Ce résultat montre qu'avec le bon choix de régularisation, si f ∈ C r b (Ω) et satisfait les hypothèses supplémentaires Assumptions 1.2 and 1.3, la méthode présentée dans Sec. 1.4.1 atteint une erreur d'ordre n -r/d+3/d+1/2 jusqu'à des termes logarithmiques et constants (qui ne dépendent pas de n). Ceci est à comparer à la limite dans le pire des cas obtenue par [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF], qui était de n -r/d . Il nous manque un terme 3/d + 1/2, ce qui est dû au fait que a) notre décomposition en somme de carrés réduit la régularité, et que b) nous utilisons le fait que C s b (Ω) ⊂ W s 2 (Ω), ce qui est très grossier. Cependant, ce résultat montre encore que dans l'esprit, dès que la régularité est de l'ordre de la dimension, on ne paie pas la dimensionnalité dans le taux (i.e, approximativement,

| ρ * n,λ -f * | ≤ Cn -1/2
), bien que la dimension soit encore présente dans les constantes. Pour terminer cette partie sur les garanties des problèmes régularisés discrétisés, notons que les résultats originaux (voir chapter 9), s'attachent également à fournir des bornes explicites pour les constantes des théorèmes ci-dessus. C'est pourquoi il existe une condition particulière sur le domaine Ω, (bien que cette condition soit plus générale que celle d'être une boule de rayon R, voir Assumption 9.1). Le même raisonnement pourrait être appliqué sur un domaine satisfaisant une condition de cône uniforme, comme dans le chapitre 4 de [START_REF] Adams | Sobolev Spaces[END_REF], et étendre formellement ces résultats à une classe plus large de domaines, bien que garder la trace des constantes conduirait à des calculs longs et pénibles.

Extraction

Jusqu'à présent, nous avons seulement parlé de l'approximation d'un minimum de f , mais pas d'un minimiseur. Pour pouvoir approximer un minimiseur, il faut utiliser [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] Eq. (1.58), et définir x = n i=1 α i x i , où les α i sont les variables primales optimales. Ceci est justifié formellement dans Sec. 9.7 , en faisant les hypothèses que (i) il existe un unique minimiseur x * de f ;

(ii) il existe une parabole de la forme ν xx * 2 avec ν suffisamment petit pour que cette parabole soit une borne inférieure de f . Sous ces hypothèses, [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] fournissent un algorithme permettant de retrouver à la fois une approximation du minimum et du minimiseur, en utilisant le même principe que celui décrit ci-dessus, et avec des garanties de convergence de la forme ν x n,λ -x * 2 ≤ C n -d/r+3/d+1/2 pour un choix approprié de λ (voir Theorem 9.8).

Passage à des fonctions avec des ensembles continus de zéros sur des variétés

La méthode présentée dans Sec. 1.4.1 était assez générale, et n'avait pas du tout besoin que X soit un domaine de R d pour être appliquée. Cependant, l'analyse que nous avons effectuée dans cette section repose de manière cruciale sur certaines propriétés de ENR sur R d , qui sont nécessaires pour prouver que le problème Eq. (1.49) admet une solution.

Le but du travail de Marteau-Ferey, Bach, and Rudi (2022b) est d'étendre ces résultats qui reposent sur le fait que certaines fonctions régulières positives peuvent être décomposées en sommes de carrés de fonctions régulières. Il est motivé par deux grandes lignes d'applications de cette méthode que nous souhaitons explorer.

• La première était de pouvoir décomposer des fonctions dont l'ensemble des minimiseurs n'est pas discret (comme l'implique Assumption 1.3), mais qui peuvent avoir certaines propriétés de régularité, comme être une variété. Cela a été motivé par des applications au transport optimal, où de tels ensembles de minimiseurs apparaissent naturellement (voir Vacher, Muzellec, Rudi, Bach, and Vialard (2021)).

• La seconde était de pouvoir passer de R d à n'importe quelle variété M , afin d'appliquer cet algorithme pour minimiser des fonctions sur des variétés.

Les idées haut niveau du travail de Rudi, Marteau-Ferey, and Bach (2020) Main theorem 7: Marteau-Ferey, Bach, and Rudi (2022b), Théorème 3.9

Soit f une fonction non négative de classe C p pour p ≥ 2 et soit Z l'ensemble des zéros de f . Si Z est une sous-variété compacte de M de classe C 1 tel que In many fields of applied mathematics, from machine learning to physics, we aim to find a certain mathematical object o in a set of mathematical objects O, which will help us tackle a certain task. This object can be a function f : X → Y, a point in a set x ∈ R d with particular properties, a trajectory (x t ) t∈T , or even a measure µ on a space X . In many cases, this object can be thought of as the solution to an "ideal" optmization problem : we have a functional F : O → R, and wish to find an object o * such that

∀x 0 ∈ Z, ∀h ∈ T x 0 M \ T x 0 Z, H(x 0 )[h, h] > 0, (1.68) alors il existe un nombre fini de fonctions f 1 , ..., f N ∈ C p-2 (M ) telles que ∀x ∈ M, f (x) = N i=1 f i (x) 2 . (1.69)
F (o * ) = inf o∈O F (o), (2.1)
where "ideal" functional F incorporates everything there is to know about the problem. In practice however, we cannot solve Eq. (2.1), because a) we only have part of the information about the ideal problem at hand, b) we have to be able to perform computations on a computer (on which we cannot necessarily represent the ideal object o ∈ O or the function F ), and c) we have only a finite computational budget, in terms of time and memory, and therefore have to use a finite-time algorithm. Instead of solving the ideal problem Eq. (2.1), we therefore solve a surrogate problem

ô = Alg F , O , F (ô) ≈ inf o∈ O F ( o) (2.2)
where Alg is an algorithm which computes an approximation of the solution of the surrogate problem in finite time on a computer. The function F is usually called the surrogate function and incorporates a priori information on the objective F , the set O is called a model for O and incorporates a priori information (like shape constraints or regularity of the target), and is representable in a concise way on a computer, and the optimization algorithm Alg is taylored to the surrogate problem, and has a given time and space complexity.

As we already see in this simple setting, algorithms, models, surrogate problems are interconnected.

Understanding the quality of the returned object ô, measured by the quantity R(ô) = F (ô)inf o∈O F (o), requires an understanding of all these components and how they relate. Breaking down the roles of these different parts is crucial to understand the limits of each one. In this thesis, in parts I to III, we will try to look at all these aspects, to fully grasp the applied mathematics problem at hand. We will formulate the ideal problem Eq. (2.1), then present a model as well as 44 CHAPTER 2. GENERAL INTRODUCTION a surrogate problem for the function (this can be done in different steps, as in part I), before designing an algorithm whose complexity we can control. We hope that the reader will be able to see and put together these different steps, and that it will expose the interactions between these different phases, how they can limit or help each other out.

This introduction is structured into three parts. In Sec. 2.1, we will introduce the main applied mathematical setting of this thesis. We will almost completely focus on the statistical machine learning setting which is the setting of parts I and II. In particular, we will present a framework for constructing a surrogate problem from an ideal statistical problem, by assuming that instead of knowing the entire distribution (which would be the ideal case), we only have access to it through samples. We will also present basic optimization algorithms, to approximately solve the surrogate problem. We will end this section by briefly presenting a simplified setting of part III.

In Sec. 2.2, we will present the main classes of models which we will explore in this thesis, linear models and kernel methods. The goal is to center the attention of the reader on the importance of modelling, and how it interacts with both statistics and optimization questions.

Finally, in Sec. 2.3, we will give a brief outline of the thesis, and of the content of each part.

Machine learning and a little bit of global optimization

In this section, we will present some backround on the problems we worked on in this thesis. Most of these problems in parts I and II are formulated in the context of statistical machine learning. That is why in Secs. 2.1.1 to 2.1.3, we will present the general machine learning framework and context in terms of ideal problems (expected risk minimization), surrogate problems (empirical risk minimization) and optimization algorithms. Instead, in Sec. 2.1.4, we briefly present these three phases in the context of part III.

In parts I and II, we will deal with problems of the form Eq. (2.1) in the context of machine learning. In this context, the object to find (or to learn) is usually a function f : X → Y from an input space X to an output space Y.

The specificity of machine learning is that one of the main (if not the main) a priori knowledge that we use to learn the function f is data : we are given a finite set (z i ) 1≤i≤n ∈ Z n of examples z i living in a set Z which help us to learn f . For comprehensive introductions to machine learning, one can refer to [START_REF] Shalev | Understanding Machine Learning: From Theory to Algorithms[END_REF]; [START_REF] Hastie | The Elements of Statistical Learning[END_REF].

In general, the complexity of a machine learning problem can be roughly summarized by two key quantities.

• d the dimension of the input space X of the function f (d if X is a subset of R d for instance, or other notions of dimensions such as the one proposed by [START_REF] Vapnik | The nature of statistical learning theory[END_REF]);

• n the number of data points available to learn the function f .

In most modern machine learning tasks, the scale is large, meaning that n is large, and the dimensionality d of the data points is also large. While these differences are not set in stone, the large scale / large dimensional setting of machine learning has driven the development for new methods adapted to these specificities, compared to the traditional signal processing world. We do not mention anything on Y here. This is because, in this thesis, we will mostly take Y to be equal to R, a subset of R, or {-1, 1} in the case of a classification predictor. The case where Y is large of infinite dimensional can also be of great importance, but we do not treat it here.

In the literature, we distinguish two main kinds of machine learning tasks.

Supervised learning. [START_REF] Hastie | The Elements of Statistical Learning[END_REF][START_REF] Vapnik | The nature of statistical learning theory[END_REF] In this setting, we want to predict an output y ∈ Y from an input x ∈ X . For example, we might want to predict whether a patient will survive or not (Y is therefore binary) given its medical record and treatment, encoded in X , whether a particle detected at the CERN through measurements is a Higgs Boson or not [START_REF] Baldi | Searching for exotic particles in high-energy physics with deep learning[END_REF], or what price should be set given customer data. In that case, the data usually takes the form z i = (x i , y i ) of examples of input output pairs, the point being that from examples of on inputs-outputs, we can learn a function f which will perform well on new data points x. In supervised learning, a distinction is often made between classification, where Y is finite (as in the Higgs Boson case), and regression, where one wishes to learn a continuous variable Y (as in the sales case). In part I, we handle problems which are typical supervised learning problems, such as logistic regression.

Unsupervised learning. [START_REF] Shawe | Kernel Methods for Pattern Analysis[END_REF][START_REF] Hastie | The Elements of Statistical Learning[END_REF] In this setting, the goal is to directly extract some meaningful information from the data. Examples of unsupervised learning tasks include the learning of a generative model, i.e., given examples x 1 , ..., x n sampled from a certain distribution ρ, create a sampler which samples from a distribution ρ close enought to ρ, the learning of the mean of a distribution through samples, the extraction of meaningful low dimensional features from high dimensional data, clustering, etc. In part II, we present a model which can tackle certain unsupervised learning tasks, such as the task of generating samples.

Whether it be in a supervised or unsupervised setting, most machine learning problems boil down to learning a function f : X → Y from a finite amount of seen data, in order to use it on other unseen data. In order to formalize the notion of seen and unseen data, as well as to understand how one may use data to infer information, the complexity of the full problem is often encapsulated into a probability distribution, to which we have limited access through the data : this is the statistical learning theory framework.

Statistical learning framework

In the statistical learning setting, the data points z 1 , ..., z n are assumed to be realisations of a random variable Z on Z (therefore equipped with a σ-algebra), with ρ the associated probability measure on Z. Implicitly, the statistical learning setting assumes that all the information on the data can be contained in the random variable Z. One of the very standard assumptions is that the data points z 1 , ..., z n are independent and identically distributed (i.i.d.) samples from Z.

Assumption 2.1 (iid samples). The samples z 1 , ..., z n are i.i.d. samples from Z.

We will make this assumption in this thesis. In practice, this assumption may not be satisfied. In particular, when dealing with time series, the independence assumption may not be true (think of a Markov chain). Moreover, in some settings, the data is collected from multiple sources, which induces a difference in distribution depending on the source. This is the case when dealing with medical data coming from different medical centers for instance, where images are acquired using different tools; the distribution of the images depends on the acquisition device, and hence the data used are not identically distributed.

Supervised learning. In the particular case of supervised learning, the data usually takes the form of input-output pairs, i.e., the samples (x i , y i ) are assumed to be drawn from a joint distribution Z = (X, Y ). Denote with ρ X the probability measure of X and assume that ρ can be decomposed as dρ = ρ(dy|x)dρ X for a certain ρ(•|•) : S(Y) × X → [0, 1] where S(Y) denotes the σ-algebra of Y and p(Y|x) = 1, ρ X -almost everywhere (a.e.). This is simply the formal definition of conditional probabilities. There are two sources of randomness in the supervised learning setting : a) a randomness of the input data, from which we draw the x i , and b) a randomness of the output data conditioned on the input, i.e., the fact that y i is drawn from ρ(dy|x). If Y = f (X) a.e., then Y is actually deterministic knowing x and ρ(dy|x) = δ f (x) is simply a dirac for almost every x. In this case, we assume that Y can fully be recovered from X. However, it is typically not the case, because X does not contain enough information to predict Y , or because the observations are noisy.

Generalization error. Assume first that we are in the supervised learning setting, and that the data follows the distribution ρ of (X, Y ). The goal is to find a good predictor f : X → Y, such that f (X) is a good approximation of Y . A natural space to look for this predictor is the set M(X , Y) of measurable functions from X to Y. To measure the quality of a prediction f (x) compared to y ∈ Y, we define a loss function a priori. A loss function will be a map : (X × Y) × M(X , Y) → R + , such that ((x, y), f ) quantifies the error performed when approximating y by f (x). The risk or generalization error of a function f is then defined as R(f

) := E [ ((X, Y ), f )], f ∈ M(X , Y), (2.3)
where the expectation is taken over ρ. We also define R * := inf f ∈M(X ,Y) R(f ) to be the smallest possible risk. If this optimum is reached for a certain function f * (note that there can be none, one or more than one), f * is called an optimal predictor. The excess risk of a function f is defined as the quantity R(f ) -R * . Going back to the introduction of this chapter, finding f * is therefore our ideal problem Eq. (2.1).

More generally, going beyond the previous setting, we will consider losses of the form : Z × F → R + , where Z is the data space (equipped with the random variable Z), F is the space of functions (predictors, probability distributions, probabity samplers or even single points) suited for the problem. We assume (•, f ) is measurable for all f ∈ F and is the loss function. As before, (z, f ) will measure how well f performs on data point z. We will also use the notation z (f ) in the place of (z, f ). Note that in the supervised learning setting described above, we would define z (f ) = (x,y) (f ) and F would be M(X , Y) a priori. Note that this formulation allows some flexibility : the output space of f might not be Y in the supervised setting (this is the case in certain classification problems). The general "ideal" goal is therefore to solve inf

f ∈F R(f ), R(f ) := E [ (Z, f )],
(2.4)

and to find f with small excess risk R(f ) -R * .

Losses. As can been seen in Eq. (2.4), the choice of the loss defines what we consider to be a good inference from data and must therefore be chosen according to the problem at hand. However, as we will see in Sec. 2.1.3, the choice of the loss also impacts optimization procedures to solve the surrogate problem and must therefore be chosen with this in mind as well. We now present different settings and the associated typical losses.

Regression and square loss. In the setting of regression, that is in supervised learning where Y takes values in a sub-interval of R or more generally a subset of R k , one often uses the square loss : x,y (f ) = l(y, f (x)) = 1 2 yf (x) 2 . It is the most standard loss for regression and solving Eq. (2.4) is called the least squares regression problem. Note that as soon as Y ∈ L 2 (which seems a natural hypothesis in order for the risk to be defined), there is a solution to the ideal problem and it is simply the projection of Y onto L 2 (X , ρ X ) seen as a closed linear subspace of L 2 (X × Y, ρ), also called the conditional expectation of Y given X, or the Bayes predictor. It is defined as

f * (X) = E [Y |X] , f * ∈ arg min f ∈M(X ,R) R(f ) = 1 2 E Y -f (X) 2 .
One can decompose the variable Y = f * (X) + ε, where ε is called the noise and is orthogonal to the input variable X, in the sense that E [ε|X] = 0. Note that R * = E ε 2 . In part I, and more specifically in Sec. 3.1, we will go into much greater detail in the analysis of least-squares regression, from a statistical but also computational perspective, and give the necessary references.

In particular, we will see that the regularity of the Bayes predictor, as well as the hypotheses on the noise ε play a crucial role in this analysis.

Binary classification.

In binary classification, we wish to predict a binary variable Y which has values in Y = {-1, 1} (this choice is arbitrary). The natural loss for such problems would be the 0 -1 loss, that is the loss x,y (f ) := 1 f (x) =y which penalizes f by 1 if the prediction is incorrect. However, while this loss is ideal from the point of view of the task, it neither smooth nor convex, and can be hard to optimize (see Sec. 2.1.3 for more details). Moreover, it is sometimes easier to parametrize functions f with vector valued outputs, and to allow f to take its values in R and not strictly {-1, 1}. This is easily remedied by allowing f to be real-valued and predict the output according to its sign sgn(f (x)). When f is linearly parametrized, f is called a separating hyperplane, as {f (x) = 0} defines the boundary between the two classes.

With our choice of Y, the 0 -1 loss can simply be expressed as 1 yf (x)<0 even in the case where f is real-valued. The typical losses used in binary classification are of the form ϕ(yf (x)), where ϕ is a surrogate for 1 t<0 . They include the Hinge loss ϕ(t) = max(0, 1 -t) which is the convex relaxation of the indicator function, and is convex but is not smooth, and the logistic loss ϕ(t) = log(1 + e -t ), which is smooth and convex. In this thesis, we will not discuss the hinge loss or Support Vector Machines in general. However, they are central losses and algorithms in the classification setting; a good reference can be found in the work by [START_REF] Steinwart | Support Vector Machines[END_REF]. On the other hand, logistic regression will be at the center of part I. In Fig. 2.1, we show these three different loss functions (the corresponding ϕ) to illustrate their different convexity and smoothness properties.

Multi-class classification. In multi-class classification, we wish to predict a variable whose values lie in a finite set {0, 1, ..., K} where K ≥ 1. Of course, the 0 -1 loss is still the ideal loss, and can also be defined, and has the same issues than in binary classification in terms of smoothness and optimization difficulty. However, if ρ can be decomposed as a conditional law ρ(dy|x)dρ X , the conditional law ρ(dy|x) is necessarily a multinomial law whose conditional probability can be represented by a vector (p 0 (x), p 1 (x), ..., p K (x)) on the simplex (p y ≥ 0 and y p y = 1). One of the typical ways to proceed is to use multiclass logistic regression, that is to look for functions f y : X → R such that p y (x) = ρ(y|x) ∝ exp(-f y (x)). The loss of f = (f y ) y∈Y is then defined as

x,y (f ) = -log e -fy(x) K i=0 e -f i (x)
. Note that f : X → R Y is not a predictor per say; however, given f , one defines the associated predictor f simply by predicting f (x) := arg min y∈Y f y (x), that is the y with highest inferred probability, proportional to e -fy(x) . We will deal with these losses in more detail in part I, for certain classes of models.

CHAPTER 2. GENERAL INTRODUCTION

In the case where K = 1 (binary classification), this is equivalent to logistic regression by taking

Y = {-1, 1} and f (x) = f -1 (x) -f 1 (x).
Density estimation. This time, in a non-supervised setting, the goal is to perform density estimation, i.e., try to approximate a density p on a space X with respect to a base measure dν through i.i.d. samples x 1 , ..., x n from p dν. In that case, the model F of functions should contain functions which are non-negative and sum to one with respect to dν. This is a typical case where one uses the negative log-likelihood loss:

x (f ) = -log(f (x)).
Such an approach is actually quite general and comes from statistics, as we will see in the following point. We use this loss in chapter 6 in part II to learn densities.

Log-likelihood. In classical statistics, it is common that the space F parametrizes a set of probability densities on Z with respect to a base measure dν, denoted with

P F = {p f (dν) ∈ M 1 (Z) : f ∈ F}, where M 1 (Z)
is the set of probability measures on Z. Given samples from the unknown distribution ρ ∈ M 1 (Z), it is classical to look for p f ∈ P F (or equivalently for f ∈ F) which maximizes the expected log likelihood :

max f ∈F L(f ) := E [log(p f (Z))] = KL(p f ||ν),
which can be seen as a Kullback-Leibler divergence between p f and ν (for more details, see Sec. 3.1 in chapter 3). In machine learning, we adopt the point of view of minimizing a loss, and instead of maximizing the log-likelihood, we usually minimize the negative log-likelihood. Note that logistic regression, as presented above, or multi-class logistic regression is equivalent to this for conditional likelihoods p(y|x).

Least-squares regression can also be seen as a maximization of the log-likelighood, for the model

P F := {ρ ∈ M 1 (X × Y), dρ = exp(-y -f (x) 2 /2)(dρ X ⊗ dy)}
, that is we are looking for a conditional distribution of Y |x as a Gaussian centered at f (x).

Constructing a surrogate problem in statistical learning : empirical risk minimization

As explained in Sec. 2.1.1, we have access to the distribution of Z only through samples z 1 , ..., z n , and are therefore unable to directly solve Eq. (2.4). Instead, one can approximate this error by replacing ρ with the empirical distribution constructed from samples ρ := 1 n n i=1 δ z i and define the so-called empirical risk

R n (f ) := E ρ [ Z (f )] = 1 n n i=1 z i (f ), f ∈ F. (2.5)
For example, in the case of the square loss, the empirical risk can be written

R n (f ) = 1 n n i=1 f (x i ) -y i 2 , f ∈ F. (2.6)
One could then think that simply minimizing the empirical risk over the class F would lead a small excess risk. However, there are two limiting factors to this approach : a) the space F can be so complex that it is not representable on a computer and b) the space F can be too large. If we go back to the example of supervised learning where F = M(X , Y), in the case of the empirical square loss Eq. (2.6), any interpolating function minimizes the empirical risk (and some of them could be wild). Note that in this introduction, F simply denotes a large set of functions on which the loss function is defined, and is typically given by the problem : M(X , Y) for supervised learning in general, L 2 (X , ρ X ) for least-squares regression (since we know the minimizer to be in that space), the set of measurable functions which sum to one in the case of density estimation.

It is therefore important to minimize the empirical risk over a smaller space H ⊂ F of functions adapted to the problem and to the amount of data we have at hand. A first surrogate problem in this spirit is therefore the following empirical risk minimization (e.r.m.) problem on H:

find f n ∈ H such that f n ∈ arg min f ∈H R n (f ) = 1 n n i=1 z i (f ).
(2.7)

As explained above, the choice of H is crucial. It must satisfy the following properties :

• the functions in H need to be representable on a computer;

• the e.r.m. problem Eq. (2.7) needs to be approximately solved by an optimizer;

• H must be small enough so that the solutions are not ill-behaved (in other words, H must incorporate enough a priori information compared to the amount of data);

• H must be large enough to approximate F well.

CHAPTER 2. GENERAL INTRODUCTION

A classical yet powerful model for classes F of real-valued functions is to consider H in the form

H = {f | f (x) = θ φ(x)} for a given feature map φ : X → R p .
H is called a linear model with feature map φ, and p denotes the number of parameters of the model. In that case, it is easy to see that f ∈ H can be identified to the corresponding θ ∈ R p and is therefore representable on a computer as a vector of size p. For example, one may think of polynomials of degree at most p -1 on R, which is a p dimensional set of functions parametrized by its coefficients θ where φ(x) = (1, x, ..., x p-1 ).

The size of H is determined both by the complexity of the feature map φ and the number p of parameters. Sec. 2.2 will be centered around a certain class of spaces H used in this thesis : the infinite dimensional counterpart of linear models, reproducing kernel Hilbert spaces (RKHS). We will also consider more complex models H of functions, based on these linear models, but which allow to approximate classes of functions F which are not necessarily real-valued in part II and part III.

In the following discussion, we will decompose the excess risk in order to mathematically show the different trade-offs between model capacity and number of samples.

Approximation and estimation error. Recall in this setting that F denotes a large "ideal" space where we expect to find f * without assumptions, and that H is a smaller approximation of F (this notation is not standard; the notation F is sometimes used for the model). Let R H := inf f ∈H R(f ) be the the best achievable performance for functions in H and recall that the best achievable performance is defined as R * = inf f ∈F R(f ). We can decompose the excess risk of an empirical risk minimizer f n into two terms :

R( f n ) -R * = R( f n ) -R H estimation error + R H -R * approximation error
.

(2.8)

These two errors are often studied differently and have different meanings.

• The approximation error is the error one makes when modelling F with H. This error decreases when H becomes larger.

• The estimation error is intrinsic to H; it is the error that one makes when approximating ρ with the empirical distribution ρ, given the estimation space H. For a fixed n, this error becomes larger as H increases.

In Fig. 2.2 we plot the typical evolution of the error when the size of H increases. We see that the typical evolution for the excess risk is to first decrease (a regime called underfitting : H is too small), before going back up again, because H is too large compared to the number of data points : this is called overfitting.

In Fig. 2.3, we further illustrate this underfitting-overfitting phenomenon in the context of polynomials. We take X to be uniform on [0, 1] and Y = f * (X) + ε where f * is a polynomial of degree 6 and ε is a Gaussian noise. We learn f ∈ H through least-squares regression with n = 20 samples, where we take H = R k [X] for increasing values of k to model increasing H.

The plot illustrates the underfitting phenomenon for small values of k (we see that a line or a degree 3 polynomial is not enough to approximate f * well) , and the overfitting phenomenon for large values : in particular, f interpolates the points (x i , y i ), i.e., passes through all points. The graph on the bottom-right corner plots the error as a function of the degree k; we see the same U -shaped tendency described theoretically in Fig. Remark 2 (train/test). In practice, to evaluate the risk of a function f , learnt using n data points (throught e.r.m. for example), one uses a test set z n+1 , ..., z n+m of additional data (which are not in the training set) and uses the empirical loss on this additional data set to approximate the exptected risk. One can also have multiple test sets in order to obtain error bars on the risk, or perform K-fold cross validation. For more details, see [START_REF] Hastie | The Elements of Statistical Learning[END_REF].

Regularization. In order to avoid overfitting, it is necessary to reduce the size of the space H. Another method consists in penalizing functions in H which are too wild, i.e., solve

find f n,λ ∈ H such that f n,λ ∈ arg min f ∈H R n (f ) + λΩ(f ), (2.9)
where Ω is a penalty function, i.e., function controlling the complexity of f . λ is therefore a hyper-parameter which controls the penalization, and hence the effective size of the space (we say that it is a hyper-parameter because it is not optimized on during training but in the model selection phase). This will be made formal in part I and chapter 3. Note that restricting the functions set from F to H is actually already a form of penalization with the penalty Ω(f

) = ι H , where ι H (f ) = 0 if f ∈ H +∞ otherwise .
When F = {f θ | θ ∈ Θ} can be identified to a subset of Θ ⊂ R p , p ∈ N (as in the case of linear models), typical penalties include The first can be used as soon as H has a Euclidean or Hilbert space structure. Using this regularization with the square loss is called ridge regression. It will be at the center of part I, as we will use this regularization to control the size of the space H. It is also beneficial from an optimization point of view, as it induces strong convexity (see Sec. 2.1.3). In Fig. 2.4, we perform the same task as in Fig. 2.3, but instead of making the degree vary, we fix the degree k = 20 and perform Ridge Regression for different values of λ. We see that λ really behaves like a size of H parameter, and that we can balance the fact that H is too large with some regularization.

Ω(f θ ) = θ 2 2 = p i=1 |θ i | 2 or Ω(f θ ) = θ 1 = p i=1 |θ i |. =ℝ1[X] f* f n samples =ℝ4[X] f* f n samples =ℝ7[X] f* f n samples =ℝ10[X] f* f n samples =ℝ13[X] f* f n samples =ℝ16[X] f* f n samples =ℝ19[X] f* f n samples =ℝ22[X]
The second is often used as a way of find a solution θ * with a small number of non-zero coefficients. Each coefficient usually corresponds to a feature, which is activated if the coefficient is non-zero; these methods are called LASSO methods [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]; [START_REF] Hastie | The Elements of Statistical Learning[END_REF]. We will briefly use these methods in chapter 6 and chapter 9.

Bounding the approximation error. Controling this error is usually done by assuming something on the distribution ρ of the data. For example, given a loss and F, we can consider the set of distributions M(H)

:= {ρ ∈ M (Z) : ∃f H ∈ H, R(f H ) = R * } ,
that is the set of distributions ρ such that there is an optimal predictor in H. Assuming that ρ ∈ M(H) is usually called well-specified assumption, which implies R H = R * and hence a 0 approximation error. Other less stringent assumptions can help control the size of the approximation error.

If H is chosen to be "dense" in the set F for a metric linked to the loss (for example the L 2 distance in the case of least-squares regression, see part I, chapter 3), the approximation error can be shown to be zero (this is called a universality property in the context of linear models, see Sec. 2.2). When the approximation error is zero, one usually has to add some regularization, as the space of functions is very large. In that case, another error plays the role of the approximation error : it is called the bias. This is the approach taken in part I.

Bounding the estimation error. There are two main classes of methods in order to bound the estimation error.

Rademacher complexities. Consider the case of empirical risk minimization (without regularization). One of the way to go, which we will not adopt in this thesis (except for one result in chapter 6 in part II to show statistical properties of the models we introduce), can be to bound the approximation error in the following way :

R( f n ) -R(f H ) = R( f n ) -R n ( f n ) + R n ( f n ) -R n (f H ) ≤0 + R n (f H ) -R(f H ) ≤ 2 sup f ∈H R n (f ) -R(f ) ,
that is we bound the estimation error by a uniform bound between R and R n . In empirical process theory, tools are developed in order to bound this uniform deviation (see [START_REF] Talagrand | Sharper Bounds for Gaussian and Empirical Processes[END_REF] for a classical approach), and rely on the study of the tail of certain Gaussian processes. One of the main tools more recently formalized are Rademacher complexities, such as

Rad z 1 ,...,zn (H) = E σ sup f ∈F n i=1 σ i z i (f )) , Rad n (H) = E z 1 ,...,zn∼ρ ⊗n [Rad z 1 ,...,zn (H)] ,
where the σ i are i.i.d. Bernoulli variables such that P(σ i = ±1) = 1/2. A Rademacher complexity is a measure of size of the set H with respect to the samples and the loss. For example, a CHAPTER 2. GENERAL INTRODUCTION symmetrization argument shows that the uniform bound sup f ∈H R n (f ) -R(f ) is bounded in expectation by Rad z 1 ,...,zn (H), which bounds the expected estimation errors in the following way :

E R( f n ) -R(f H ) ≤ Rad n (H),
where the expectation is taken with respect to the observed data z 1 , ..., z n , which are i.i.d. samples from ρ. These techniques often provide the baseline to prove so-called slow rates of convergence, that is rates of the form C/ √ n for a constant C (see for example [START_REF] Mohri | Foundations of Machine Learning[END_REF], the Rademacher complexity obtained in chapter 6, or the discussion at the beginning of chapter 4). They are very general, and cover many loss functions. They can also be used to prove faster rates (when dealing with regularization for example) by defining localized Rademacher complexities [START_REF] Peter | Local rademacher complexities[END_REF][START_REF] Sridharan | Fast rates for regularized objectives[END_REF].

Minimax upper and lower rates of convergence. Another point of view on bounding the estimation error is to study so-called "rates". Here, the setting is a bit more involved. Let M be a class of probability distributions ρ on Z, which is our hypothesis space (we assume that the data measure ρ is in M). M can be seen as a hypothesis on the regularity of the expected risk minimization problem. Formally, an n-estimator f n in H is a function which to n data points (z 1 , ..., z n ) ∈ Z n maps a function in F (also denoted f n by abuse of notation, for example the empirical risk minimizer); an estimator is therefore a random quantity in terms of the data samples z 1 , ..., z n . Given the samples and a distribution ρ ∈ M, the performance of the estimator

f n is R( f n ) -R * = E ρ Z ( f n ) -inf f ∈H E ρ [ Z (f )].
Informally an upper/lower/optimal rate (r n ) n∈N is a sequence of positive numbers such that the performance of the best possible estimator f n is upper/lower/upper-and-lower bounded by a positive constant factor of r n . Reformulating this, we ask that r -1 n R( f n ) -R * is upper/lower/upper-and-lower bounded by a positive constant as n goes to infinity. Since r -1 n R( f n ) -R * is a random quantity in the data points z 1 , ..., z n , we define the performance of the the estimator f n on the distribution ρ relatively to r n as an expectation of the form

E ρ ⊗n ψ R( fn)-R * rn
where ψ is an non-decreasing function. Choices for ψ depend on the use case, but typical choices include | • | p , p ≥ 1 to have bounds in L p norm [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] but also indicators of the form 1 t>τ , τ > 0 to have bounds in probability [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF][START_REF] Alexandre | Introduction to Nonparametric Estimation[END_REF]. The performance of the best possibles estimator f n is therefore the quantity

MinMax(ψ, H, M, n, r n ) := inf fn sup ρ∈M E ρ ⊗n ψ R( f n ) -R * r n .
It is called the minimax performance at n for the rate r n . We say that (r n ) is :

• a minimax upper rate if lim sup n→∞ MinMax(ψ, H, M, n, r n ) < ∞; • a minimax lower rate if lim inf n→∞ MinMax(ψ, H, M, n, r n ) > 0;
• a minimax optimal rate if it is both an upper and a lower rate.

In this thesis, we will mostly focus on upper bounds and in particular non-asymptotic upper bounds. Indeed, we will study specific estimators f n , and typically show bounds of the form

P ρ ⊗n R( f n ) -R * r n > τ ≤ Ce -cτ α ,
for a given loss function , a given set H of functions and a given model M of data distributions, and where C and c are explicit and do not depend on n or ρ but only on M, H, . These bounds actually show that (R( f n ) -R * )/r n has an exponential tail. Note that this type of bounds can actually lead to minimax upper rates (see for example [START_REF] Alexandre | Introduction to Nonparametric Estimation[END_REF]; [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]). In general, the strategy to find an upper bound is usually to build an estimator which matches that bound. Such non-asymptotic upper bounds and rates can be found in every chapter of the thesis (except perhaps chapter 10); they will not always be formalized in this way, but the spirit is the same.

Finding lower bounds, on the other hand, is a different exercise altogether. As proposed by Tsybakov ( 2008), the strategy is usually, to build a finite sequence of densities ρ 1 , ..., ρ m which are far in terms of risk, but close statistically, so that f n will be unable to approach all of them well simultaneously. For slightly more details, see chapter 3.

In chapter 3, we will detail minimax optimal rates in the setting of least-squares regression (loss ) and linear methods (model H) for different classes of measures M.

Concluding remarks on surrogate problems. In this section Sec. 2.1.2, we have seen that solving an empirical risk minimization problem can be a good way to find a good estimator while having only access to n samples from the data distribution. However, the excess risk of an empirical risk minimizer depends on the size of H, and is caracterized by a trade-off between the approximation error, i.e., the capacity of H to approximate all functions of interest, which increases with its size, and the estimation error i.e., the capacity of the empirical risk minimizer to generalize well enough. We have also seen that regularization can be a good way of controlling the capacity of H which does not require changing the space, but rather penalizing functions already in the space. Finally, we have developed the two main ways of bounding the estimation error, and explained how the results obtained in this thesis, non-asymptotic upper rates, compare with the minimax point of view.

Note that the framework presented here does not contain all forms of machine learning methods or losses. We mentioned two examples, but there are of course many others. First, it happens that the regularization term in e.r.m. is data-dependent. This is the case for example in ranking problems. This leads to what is called non-decomposable losses, that is losses which cannot be written as n i=1 z i (f ) + Ω(θ) i.e., as a finite sum where each term depends only on one data point plus a penalty, but with sums involving cross terms of the data (Kar, Narasimhan, and Jain, 2014). Second, while the framework for studying e.r.m. here relies on the definition of a space H of candidate functions, it happens that one uses this space H to build "learners" (as in random forests, see [START_REF] Hastie | The Elements of Statistical Learning[END_REF]), and then aggregates these learners into a predictor, which is therefore not in H anymore. This is called improper estimation (see [START_REF] Hastie | The Elements of Statistical Learning[END_REF]; for the definition of such estimator on the logistic regression problem, which we treat in part I, see [START_REF] Mourtada | An improper estimator with optimal excess risk in misspecified density estimation and logistic regression[END_REF]).

Finally, the classical approximation/estimation trade-off point of view is sometimes not enough to explain the whole story. Indeed, certain over-parametrized models generalize well, i.e., yield good estimators, even when they interpolate the data. This phenomenon is called double descent, and is graphically shown in Fig. 2.5. Recently, theory has made progress in understanding certain situations where double descent does or does not happen. For more details on this subject, one can refer to [START_REF] Belkin | Reconciling modern machinelearning practice and the classical bias–variance trade-off[END_REF]; [START_REF] Mei | The generalization error of random features regression: Precise asymptotics and double descent curve[END_REF].

In all this discussion, we have yet to address a key issue : how do we effectively solve an empirical risk minimization problem such as Eqs. (2.7) and (2.9) ? This is the role of optimization methods, i.e., the development of algorithms to optimize functions. In the next section, we will give a quick introduction to the different optimization settings and methods we will consider in this thesis, and which are used in machine learning in general.

Optimization

Recall that we have started from an ideal problem such as expected risk minimization in Eq. (2.4), which we then transformed into a surrogate problem such as the empirical risk minimization problem Eqs. (2.7) and (2.9) due to the fact that we have only partial access to the data distribution. However, we have yet to address the actual minimization of Eqs. (2.7) and (2.9). This is the role of optimization algorithms. In this section, we adopt the standard notations of optimization; f will not denote the target predictor but the function to optimize, and d will denote the dimension of the set on which we optimize and not the dimension of the input (data) space.

Very generally, an optimization algorithm aims at finding an approximate solution to inf θ∈Θ f (θ), where Θ ⊂ R d is a set which can be encoded in a computer. Optimization is a vast field which reflects the wide variety of problems which it can handle. Optimization algorithms may vary according to the way we access the function f , its structure, its regularity or convexity properties, the structure of the set Θ (un-constrained if Θ = R d ), the precision we want for the returned solution of the algorithm, the time and space complexity available, the means of computation (GPU versus CPU).

INTRODUCTION TO MACHINE LEARNING

57

The goal of this section is to provide an overview of optimization techniques used or mentioned in this thesis in parts I and II. We will place ourselves in the setting where the function f is a sum of a decomposable function and a regularization term (of the type Eq. (2.9)), and is convex. Hence, unless stated otherwise, we will assume that the optimization problem we want to solve is of the form

min θ∈Θ f (θ) = 1 n n i=1 i (θ) + λΩ(θ), (2.10)
where Θ is a convex set of a vector space and the i and Ω are convex in θ. In most cases, we will have i = z i coming from an empirical problem. The constraint set Θ will be either the whole of R d or the positive semidefinite cone of symmetric matrices A ∈ R d×d subject to A 0 (i.e., A is symmetric and x Ax ≥ 0 for all x). Some linear equality constraints will sometimes be added in part II and part III when dealing with probability densities, i.e., a constraint of the form θ a = b.

We will assume that we have access to the i , their gradients and sometimes their second order derivatives. We will usually assume easy access to the gradients ∇ i (θ), with cost O(d) in time. Note that this means the access to a full gradient will cost at least O(nd) in time. The penalty Ω will be discussed in each case, but we will mostly use the classical square norm penalty Ω(θ) = 1 2 θ 2 , the gradient of which is simply θ and hence is directly accessible, and the Hessian of which is the identity matrix.

Machine learning setting. In the machine learning setting, [START_REF] Bottou | The trade-offs of large scale learning[END_REF] have summarized the specific optimization requirements, which are a bit different than those of the classical optimization community (although since then, the communities have become closer). They highlight two specificities of machine learning problems.

(i) Since the function we wish to minimize is the "ideal" excess risk Eq. (2.4) and not the surrogate Eq. (2.9), we do not need to optimize the surrogate with a better precision than the approximation error. This differs from the classical optimization setting, where the goal is often to have a very precise approximation of the minimum. Thus, using a "bad" optimization may very well do the trick, as it will reach the precision of the approximation error, which is all that is needed for our purposes.

(ii) Complexity matters : indeed, using a rough but fast algorithm in terms of computations per iteration (like a first order method) is likely to yield better results than a precise but slow algorithm. This is due to the fact that since the dimension d and the number of data points n are often huge (and hence the complexity of computing function values, gradients, Hessians), one must limit the number of such computations and leverage hardware architectures, namely GPUs, which perform small computations in a fast parallel way (as opposed to CPU, which can perform large computations such as full gradients of f in Eq. (2.10), but only sequentially).

We will apply the first principle in order to design optimization methods whose goal is to achieve an error of the same order as that of the approximation error, in particular in part I, in chapter 5.

The second principle has often been interpreted as a way of saying that first order methods, that is methods relying only on gradient or stochastic gradients (for more details, see below), are the best methods for machine learning. While this is often true, we develop a different approach in chapter 5, based on second order information (that is the Hessian of F ), showing that there can indeed be efficient second-order methods for machine learning in certain contexts.

First order methods : the optmization workhorse in machine learning and dependence on the condition number. Here, we briefly present the basic first order optimization techniques. These methods will be a strong baseline for our algoritms developed in part I. In particular, we want to highlight that the speed of convergence of these algorithms depends strongly on the so-called condition number of the problem, defined below. In part I, we will argue that certain classical machine learning problems have very high condition number, and that first order methods are not necessarily the best solution in that setting. Moreover, we will implement techniques to reduce the condition number of linear systems, after which one of these first order methods can be used.

Assume here that the regularization is Ω(θ) = 1 2 θ 2 , and that the optimization problem is unconstrained, i.e., Θ = R d . Note that we can write f as a finite sum in the form (2.11) and that the cost in time and memory of computing a gradient of f i is still O(d). A first order method is a method which relies only on gradient computations; they are the most common methods in machine learning. In particular, the fact that gradients can be computed as fast as function values in a neural network [START_REF] Hastie | The Elements of Statistical Learning[END_REF][START_REF] Paszke | Automatic differentiation in pytorch[END_REF] makes them methods of choice in modern machine learning.

f (θ) = 1 n n i=1 f i (θ), f i (θ) = i (θ) + λ 2 θ 2 ,
The most well-known method in optimization is gradient descent (GD) : the idea is simply to evaluate the gradient at time t and to go in the opposite direction -γ t ∇f (θ t ) (the decreasing direction) with a certain stepsize γ t > 0 which quantifies how conservative we are with respect to that gradient information (we can take a large stepsize if we know the direction is stable). Given a starting point θ 0 , gradient descent simply implements the recursion

θ t+1 = θ t -γ t ∇f (θ t ), γ t > 0.
(2.12)

Note that the cost of each iteration here is a priori O(nd) in time and O(d) in memory. In machine learning, as n and d are very large, the computation of a full gradient is quite expensive.

To mitigate this problem, another first order algorithm which is used is stochastic gradient descent (SGD), i.e.,

θ t+1 = θ t -γ t ∇f it (θ), γ t > 0, (2.13)
where i t is selected uniformly at random in {1, ..., n} at each iteration. The cost of an iteration here is therefore only O(d) and not O(nd) since we only access one gradient of the f i , which we use as a gross approximation of the full gradient. When comparing GD to SGD, one therefore has to keep in mind that n iterations of SGD (called one epoch) is equivalent to one iteration of GD in term of complexity.

SGD is actually part of a much broader class of method which date back to the 50s, [START_REF] Robbins | A Stochastic Approximation Method[END_REF]. A stochastic gradient descent can be implemented as soon as we have an unbiased estimator of the gradient at each timestep. Here we just use the finite sum structure to see it as an expectation and artificially give it a random structure. Note however that in the statistical learning framework, the f i are random, as they come from i.i.d. samples z i ∼ Z,

INTRODUCTION TO MACHINE LEARNING

59

and that stochastic gradient descent taking i t = t for n iterations is actually stochastic gradient descent performed on the expected risk minimization problem Eq. (2.4).

The analysis of these methods can be done in many settings, but one is of particular interest to us in this thesis. To do so, we need to make some assumptions on the functions f i . We say that a function g defined on a convex domain Θ is L-smooth if its gradient is L-Lipschitz, i.e., if

∀(θ, θ ) ∈ Θ 2 , ∇g(θ) -∇g(θ ) ≤ L θ -θ . (2.14)
We say that g is µ-strongly-convex if

∀(θ, θ ) ∈ Θ 2 , ∇g(θ) -∇g(θ ), θ -θ ≥ µ θ -θ 2 .
(2.15)

We denote with F L,µ the class of functions which are L-smooth and mu-strongly-convex. Informally, being L-smooth means being locally upper bounded by a parabola of magnitude L everywhere, while being mu-strongly-convex means being lower bounded by a parabola of magnitude µ everywhere. The prototypical example of a function

f ∈ F L,µ is a quadratic of the form f (θ) = 1 2 θ Σθ -b θ + constant, Σ = Σ, µI Σ LI, (2.16)
whose minimization is equivalent to finding θ * = Σ -1 b. Note that this form of function naturally appears when solving the least-squares problem with a linear model, as we will see in part I and chapter 3, and that first order methods allow to bypass the inversion operation which is very costly when d is large. Note that if the i in Eq. (2.10) are all L-smooth, then the f i all belong to F L+λ,λ . Moreover, if the f i belong to F L,µ , then F = 1 n n i=1 f i also belongs to F L,µ . One of the key quantities which characterizes how these first-order optimization methods perform is the so-called condition number, usually defined as the ratio κ = L µ . Note that κ > 1 as µ ≤ L. In the case of a quadratic Eq. (2.16), the condition number (also denoted with κ 2 (Σ) in the literature) is simply the ratio between the largest and smallest eigenvalues of Σ.

Assuming the f i are in F L,µ , the best known algorithms in term of convergence are more involved than the simple GD/SGD updates Eq. (2.12) and Eq. (2.13). In the case of gradient descent, [START_REF] Nesterov | A method for solving the convex programming problem with convergence rate o(1/k 2)[END_REF] proposes an optimal acceleration (i.e., an acceleration of gradient descent) for the class F L,µ , where θ t is updated using a two-step procedure with a so-called "momentum step". Linear convergence (that is showing that f (θ t ) -f (θ * ) ≤ ρ t (f (θ 0 ) -f (θ * )) for ρ < 1) can be obtained for standard and Nesterov accelerated gradient descent when assuming that f ∈ F L,µ . Moreover, in the case where f is a quadratic Eq. (2.16), that is when solving a linear system with a positive definite matrix Σ, there exists a more elaborate first order iterative method named conjugate gradient (see chapter 10 of [START_REF] Golub | Matrix Computations[END_REF] and Theorem 10.2.6), which reaches the same accelerated convergence bounds as that of Nesterov but in a more efficient way. This method will be an important sub-routine in chapter 5.

Proposition 2.1 (Rates of convergence of deterministic gradient descents). If f ∈ F L,µ and reaches its minimum at θ * , and γ t = γ := 1 L , the sequence of gradient descent iterates Eq. (2.12) initizalized at θ 0 satisfies :

∀t ∈ N, f (θ t ) -f (θ * ) ≤ (1 -1/κ) t (f (θ 0 ) -f (θ * ))
(2.17)

This means that achieving an error ratio of order can be done with t = O(κ log 1 ε ) iterations, i.e., a complexity of order O(ndκ log 1 ε ) in our context of minimizing Eq. (2.10). Using instead CHAPTER 2. GENERAL INTRODUCTION Nesterov acceleration (see Nesterov (1983)), a better convergence rate (where essentially κ is replaced by √ κ) can be obtained :

∀t ∈ N, f (θ t ) -f (θ * ) ≤ 1 + κ 2 1 -1/ √ κ t (f (θ 0 ) -f (θ * )) (2.18)
This means that achieving an error ratio of order can be done with t = O( √ κ log κ ε ) iterations, i.e., a complexity of order O(nd √ κ log κ ε ) in our context of minimizing Eq. (2.10). Finally, when the function f is a quadratic function, the conjugate gradient method iterates θ t (which are constructed only from function evaluations and gradients of f ) satisfy

∀t ∈ N, f (θ t ) -f (θ * ) ≤ 4 √ κ -1 √ κ + 1 2t (f (θ 0 ) -f (θ * )). (2.19)
For stochastic gradient descent, the situation is a bit more complex; proving linear convergence in the strong convexity and smooth setting proves to be hard, and such rates are proved for algorithms which are more involved such as SAG (Roux, Schmidt, and Bach, 2012), SVRG [START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF] and SAGA [START_REF] Defazio | SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives[END_REF]. Evaluating the error of θ t is usually done through a linear combination of

1 µ (E [f (θ t )] -f (θ * )) and E θ t -θ * 2 (instead of simply f (θ t ) -f (θ * )
in the "deterministic" setting, since θ t is now random). For these methods, linear rates can be achieved : O((n + κ) log n ε ) stochastic gradient iterations are necessary to reach an error ratio of order . Acceleration methods exist in this setting as well, such as Allen-Zhu (2017); [START_REF] Zhou | A simple stochastic variance reduced algorithm with fast convergence rates[END_REF] for SVRG and Defazio (2016); [START_REF] Zhou | Direct acceleration of saga using sampled negative momentum[END_REF] for SAGA. They lead to a a complexity of order O((n + √ nκ) log 1 ) in terms of stochastic gradient iterations, which leads to a total time complexity of order O(d(n + √ nκ) log 1 ).

We see that in spirit, the fastest first order methods to solve Eq. (2.10) under the smooth and strongly convex assumptions are accelerated stochastic methods, which achieve an error ratio of order ε in time O(d(n + √ nκ) log 1 ε ). We see that this error crucially depends on κ when the problem is ill-conditioned, i.e., when the regularization λ is small. This will motivate the introduction of a new second order algorithm in chapter 5 in part I.

Second order methods : suited for machine learning ? In this thesis, we will have need of second order methods in two quite different settings. The first, in chapter 5, will be to produce a machine learning method which is not limited by the condition number, as in the case of first order methods. The second, in chapters 7 and 9 will be when dealing with problems with conic constraints, where Θ is a positive semidefinite cone of matrices. In this paragraph, we briefly introduce these methods and explain both their promises and why they are less popular in the machine learning setting. In the next paragraph, we will briefly mention how these methods are used to deal with conic constraints.

The dependence of first order methods on the condition number is due to the fact that we do not take into account that gradients can evolve differently in different directions (i.e., the method does not adapt to the curvature of f ). Second order methods aim to solve this problem. The prototypical second order algorithm is called Newton method, and follows the update rule of the type

θ t+1 = θ t -γ t ∆ t , ∆ t = ∇ 2 f (θ t ) -1 ∇f (θ t ),
(2.20)

INTRODUCTION TO MACHINE LEARNING

61

where γ t > 0 is a stepsize and ∆ t is called the Newton step, a renormalized gradient step. Informally, in directions where the gradient h is stable, h ∇ 2 f (θ)h will be small, and the newton method will renormalize that direction as if multiplying by large stepsize (∇ 2 f (θ) -1 h will be large). On the other hand, in directions where the gradient changes quickly, the Newton method will renormalize that direction as if multplying by a small stepsize. This is illustrated by the fact that in the case of a quadratic Eq. (2.16), the Newton method converges in one step with γ 0 = 1.

Under certain assumptions on the function f , i.e., self-concordance (see [START_REF] Nesterov | Lectures on Convex Optimization[END_REF]; [START_REF] Boyd | Convex optimization[END_REF] and Sec. 3.2 for details), Newton methods can be shown to have a two regimes convergence : a first slow regime, where the loss typically decreases by a constant at every iteration, and a second "quadratic" regime the convergence is quadratic, i.e., of the form ρ ρ -t for ρ < 1. The number of iterations needed to get a relative error is therefore O(log log 1 ε + C) where C is a constant linked to the first regime. Crucially, in the case of self-concordant functions, C does not depend on the conditioning of the problem and is a fixed constant. In other words, the quadratic convergence happens in a large region around the minimum.

However, in machine learning, and in particular in the context of finite sums in Eq. (2.10), it is almost impossible to know if the target function satisfies the self-concordant property or not (the expectation of self-concordant functions is not self-concordant in general). The classical analysis of Newton methods for such problems rely on other assumptions, and in particular on the generalized self-concordant assumption [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF]) (see Sec. 3.2 for precise definitions). However, these methods and analysis suffer from two key issues.

• The superlinear convergence happens in a small region around the optimum of order θθ * 2 ≤ λ. But getting in that region can take as many steps as that of gradient descent a priori (thus depending on the condition number). Moreover, in machine learning, a great precision is often not needed as it can be the case in classical optimization (recall the discussion at the beginning of Sec. 2.1.3 from [START_REF] Bottou | The trade-offs of large scale learning[END_REF]), and often reaching precision λ is already enough.

• Computing one Newton step is computationally expensive : indeed, the solving of the linear problem; the computing of the Hessian at a given point usually takes time of order O(nd 2 ), and computing its inverse takes time O(d 3 ). This is prohibitive both in terms of storage capacity and of time complexity.

To summarize, Newton methods seem to miss the mark in machine learning : they have a large per-iteration complexity, and are fast only after reaching a certain precision which is already good enough for machine learning. However, in part I and in particular in chapter 5, we show that a) we can leverage the fast convergence globally, and b) that we can drastically reduce the cost of iterations using the structure of Eq. (2.10) and pre-conditioning techniques.

Note that there exists other second-order type methods named Quasi-Newton methods, such as BFGS, which consist in sketching the inverse Hessian little by little at each step rather than computing the entire Hessian and inverting it. However, they are not studied in this thesis. Details on those methods can be found in [START_REF] Nocedal | Numerical Optimization[END_REF].

Handling non-smoothness with proximal methods. In part II and in particular in chapter 6, we study problems for which the methods described above, relying on smoothness, cannot be readily applied.

CHAPTER 2. GENERAL INTRODUCTION

Problem 1. The first is a problem coming from the solving of Eq. (2.10) in the case of positive semidefinite matrices, i.e., min

A∈R d×d , A 0 n i=1 i (A) + nλΩ(A) (2.21)
where the regularization Ω(A) is a so-called p-schatten norm, that is Ω(A) = σ i (A) p p = d i=1 |σ i (A)| p where σ i (A) denotes the eigenvalues of A counted with multiplicity. In particular, when p = 1, Ω is the nuclear norm, which is non-smooth. In this first problem, we assume the i are L-smooth, and hence the problem is of the form min

A∈S d (R) L(A) + λnΩ + (A) where L is L-smooth and Ω + (A) = ι A 0 Ω(A) is non smooth.
Problem 2. The second problem comes from a dual formulation of Eq. (2.21) in the case where the penalty Ω(A) is simply the Frobenius norm A 2 F = Tr(AA ). In this case, the dual problem has the form sup

α∈R n -n i=1 l * i (α) -1 2λn Ω * (α)
where Ω * is smooth and where the l * i are Fenchel conjugates (see [START_REF] Boyd | Convex optimization[END_REF]) of functions which can be logarithms, and hence are not smooth (for more details, see chapter 6, as well as [START_REF] Parikh | Proximal algorithms[END_REF]). Thus, the problem is equivalent to a problem of the form min α∈R n L * (α) + Ω * (α), where Ω * is smooth and L * is non smooth.

In both these cases, we aim to minimize the function f (θ) = g(θ)+h(θ) where h is not smooth and not even differentiable, and g is L-smooth. There are first order algorithm analog to (accelerated) gradient descent algorithms to solve these problems, called proximal algorithms. These algorithms are based on two basic requirements :

• we have access to gradients ∇g(θ);

• we are able to easily compute the proximal operator of λh for all λ > 0 at every point θ 0 :

∀θ 0 ∈ R d , prox λh (x) = arg min θ∈R d h(θ) + 1 2λ θ -θ 0 2 . (2.22)
Proximal operators are readily available in many cases [START_REF] Bach | Optimization with Sparsity-Inducing Penalties[END_REF][START_REF] Parikh | Proximal algorithms[END_REF], and in particular in those we will consider in chapter 6. That is the case for instance for the 1 norm θ 1 = d i=1 |θ i |, where the proximal operator is called the soft thresholding operator :

prox λ • 1 (θ 0 ) = sgn(θ 0 ) × max(|θ 0 | -λ, 0),
where all operations are considered coordinate-wise. The main idea behind proximal methods is that if one is at a point θ 0 , then by L smoothness, we have an upper bound

f (θ) + g(θ) ≤ f (θ 0 ) + θ -θ 0 , ∇f (θ 0 ) + L 2 θ -θ 0 2 + g(θ).
(2.23)

One can minimize this upper bound in closed form : indeed, if we denote by p L (θ 0 ) this minimizer, we have p L (θ 0 ) = prox h/L (θ 0 -1 L ∇g(θ 0 )). We see that p L (θ 0 ) is very close to a gradient descent update Eq. (2.12) with step-size 1/L. In the case where h = 0, this is in fact exactly gradient descent. In the case where f is L-smooth, proximal methods therefore extend the rates of convergence of gradient descent for L-smooth functions to composite losses with the addition of a potentially non-smooth term in the loss. The equivalent algorithm to gradient descent is dubbed ISTA (iterative shrinkage threshold algorithm) : it is defined through the recursion [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]). [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] also develop the equivalent of acceleration in the proximal setting, dubbed Fast ISTA (or FISTA), which satisfies f

θ t+1 = p L (θ t ) 2.1. INTRODUCTION TO MACHINE LEARNING 63 and satisfies f (θ t ) -f (θ * ) ≤ L θ 0 -θ * 2t (see Theorem 3.1 in
(θ t ) -f (θ * ) ≤ 2L θ 0 -θ * (t+1) 2
(see Theorem 4.4). It is this method we will use in chapter 6. Note that one cannot leverage strong convexity of f to have linear convergence : these methods will not be adapted to the classical machine learning setting, but rather for more moderate sizes of n and d (as the number of iterations needed will be too high to run for large scale, large dimensional problems).

Handling constraints and interior point methods. For now, we have presented methods for unconstrained convex optimization problems, that is we have always assumed that Eq. (2.10) is solved on the whole of R d . However, in parts II and III, we use models which are parametrized by a positive semidefinite matrix, and sometimes even impose constraints that certain functions sum to one in order to recover probability densities. The goal of this section is to very briefly present the spirit of such methods, in particular in the case of semidefinite programs (SDPs). For a more complete introduction, we refer to [START_REF] Boyd | Convex optimization[END_REF], chapters 10 and 11, as well as to [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF]. Moreover, note that linear equality constraints of the form Aθ = b can often easily be dealt with by either a) going to the dual, where the constraints disappear, b) using a projected method like projected gradient descent, which projects each gradient step on the affine space defined by the linear equalities and c) incorporating them in a Newton method which can be easily adapted to equality constraints. In the few cases where we have to deal with equality constraints in this thesis (mainly in chapters 6 and 9), these are not hard to handle, in the sense that they do not add to the complexity of the problem. The constraints which are hard to deal with in this thesis are the PSD constraints, i.e., handling problems of the form min A 0 f (A) where A is a d × d PSD matrix. Note that we do not treat feasability conditions here as we will always be able to exhibit a feasible solution.

Interior point methods. Interior point methods deal with problems of the form min θ 0 f (θ) by adding a self-concordant barrier, that is a function 1 t φ(θ) where φ has domain θ 0 (that is φ explodes at the boundary). In a sense, adding this barrier forces the problem to stay in the set θ 0. The idea is then to construct a sequence of solutions θ(t i ) for increasing values of t i in order to approximate the real problem.

In the context of solving semidefinite programs, of the form min f (A) subject to A ∈ S(R p ), A 0 (it is possible to add additional equality constraints), the classical logarithmic barrier used is the log determinant barrier, i.e., φ(A) = -log det(A), which is self-concordant and the domain of which is the set of positive definite matrices. The idea of interior point methods is to iteratively solve the problem

A(t) := arg min f (A) -1 t log det(A),
(2.24) using a Newton method for a finite sequence t 1 , ..., t K for increasing values t i . Indeed, the idea is that as t increases, A(t) -→ t→+∞ A * , where A * denotes an optimal solution. The sequence

A i := A(t i
) is called the central path, and the method is called interior point method because by definition, A i is a strictly feasible point (i.e., it lies in A i 0 and satisfies the equality constraints if there are some). Under a certain self-concordant hypothesis on f (which will be satisfied when handling linear or quadratic objectives f in parts II and III), as well as a strict feasibility condition, it can be shown that to achieve precision , one roughly needs K = √ d log 1 ε interior points [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF][START_REF] Tuncel | Potential Reduction and Primal-Dual Methods[END_REF].

Special case of global optimization

In this section, we briefly present the setting of part III in terms of ideal, surrogate problems and optimization algorithm. In part III we present methods which deal with global optimization of functions with SDPs. Such methods have been developed for a long time for polynomials (see for example [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF] and all the references in chapter 8), and we develop such methods for regular functions. Here, we present the very general setting and principle of such methods in the unconstrained optimization case, while keeping in mind two cases which we will see in part III.

Polynomial case. In this case, we consider the optimization of a polynomial function f of degree at most 2r over R d . We denote with R 2r [x 1 , ..., x d ] the set of such polynomial functions. We assume we have access to its coefficients f α with |α| ≤ 2r, where α ∈ N d and |α|

= d i=1 α i . We define R n : f ∈ R 2r [x 1 , ..., x d ] → R n where n = d+2r d
the bijective linear map which identifies the polynomial function f to its coefficients (f α ) |α|≤2r .

Regular function case

In this case, we consider the optimization of a regular function f of class C r on a domain U with access to n function values at points x 1 , ..., x n . We define

R n : f ∈ C r (U ) → (f (x 1 ), ..., f (x n )) ∈ R n the linear map which evaluates f at points x 1 , ..., x n .
Ideal problem. Let f be a function defined on a space X belonging to a vector space F containing constant functions. The global optimization problem min x∈X f (x) can be formulated as a convex optimization problem of the form sup c

subject to c ∈ R, g = f -c, g ≥ 0, (2.25)
which is simply stating that the minimum of a function is the maximum lower bound of that function. This will be our ideal problem Eq. (2.1). In the polynomial case, F = R 2r [x 1 , ..., x d ] while in the regular function case, F = C r (U ).

Surrogate problems. The idea of methods presented in part III is to solve a surrogate problem for Eq. (2.25) by modelling the two constraints g ≥ 0 and g = fc which are constraints which have to hold for all x ∈ X (and hence imply an infinite number of constraints for our two problems). We therefore proceed in two steps.

Step 1. We model the set of functions g ≥ 0 with a finite-dimensional model of functions

G + = {g A : ∀x ∈ X , g A (x) = φ(x) Aφ(x)}, parametrized by PSD matrix A ∈ S(R p ), A 0 where φ : x ∈ X → R p is a feature map.
The PSD constraint on A guarantees that the functions g A are non-negative. We can see the set G + as the set of sum of squares of functions of the form h v (x) = v φ(x) for v ∈ R p (this is a direct application of the spectral theorem). In certain cases, we will use a penalty Ω(A) on the set G + .

In the case of polynomials, we consider the map of monomials of degree less or equal to r :

φ(x) = (x α ) |α|≤r ∈ R p for p = r+d d . In this case, g A (x) = |α|,|β|≤r A α,β
x α+β is a non-negative polynomial of degree at most 2r which is a sum of squares of polynomials of degree at most r (see chapter 8). In that case, no penalty Ω is considered.

INTRODUCTION TO MACHINE LEARNING

65

In the case of regular functions, we will consider models of where the feature map φ is defined through a kernel function k : X × X → R: φ(x) = (k(x, xj )) 1≤j≤p ∈ R p (see chapters 6, 7 and 9). G + therefore represents the set of sums of squares of functions of the form p j=1 α j k(x, xj ). We will also consider the penalty Ω(A) = Tr(KA) on G + , where K is a PSD matrix (for more details, see chapter 6.)

Step 2. Replacing with the model for non-negative functions g, the equality constraint becomes fc = g A . This equality also needs to hold pointwise for all x ∈ X , which can be infinite.

In the case of polynomials, as f, c and g A are polynomials of degree at most 2r, this constraint is actually a n-dimensional constraint on the coefficients of the polynomials. Abstractly, it can be written R n (fc) = Rn (A) where R n is the map defined above which maps a polynomial function to its coefficients, and Rn is the linear map which maps A ∈ R p×p to the coefficients of the polynomial function g A (x) which are also in R n .

In the case of regular functions, we have access to f only through n function evaluations. What we do in chapter 9 is the following. Instead of enforcing the constraint g A = fc everywhere, we enforce it only at the points x i . The constraint is therefore approximated by the n equality constraints g A (x i ) = f (x i ) -c for 1 ≤ i ≤ n. Once again, this can abstractly be written in the form R n (fc) = Rn (A) where R n is the evaluation function defined above, and Rn is the linear map A → (g A (x i )) 1≤i≤n .

Combining these two approximations of the constraints, we build the following surrogate problem.

sup c -Ω(A) subject to R n (f -c) = Rn (A), A 0, A ∈ S(R p ), c ∈ R, (2.26)
Linking the result of Eq. (2.26) with the result of the ideal problem Eq. (2.25) (that is the minimum f * of f ) can be challenging. In the context of polynomials, it can be shown to be a lower bound [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF]. In the context of functions, certain rates can be obtained when the penalty is the trace norm and under certain additional conditions (see chapter 9). However, from an optimization standpoint, the surrogate problem can be solved using an interior point method as described above for the primal or the dual problem. In this thesis, we will study global optimization through SDPs.

Optimization. As Eq. (2.26) is a semidefinite program which is often linear or quadratic, we can readily apply interior point methods to solve the SDP. However, interior point methods are conditioned by the existence of a feasible point, which is not always the case for this problem (in particular, in the polynomial setting). That is why we usually consider its dual problem, called the moment problem in the context of polynomials, where it is often easier to find a feasible point.

inf λ R n (f ) + Ω * (B -R * n λ) subject to λ ∈ R n , B 0, λ R n 1 = 1, (2.27)
where Ω * is the Fenchel conjugate of Ω. Again, this problem can readily be solved with interior point methods as soon as there is a feasible point, using K = √ p log 1 interior points obtained via the damped Newton method applied to Eq. (2.27). For more details on the polynomial setting, we refer to chapter 5 of [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF]. For methods based on function evaluations, we refer to chapter 9. The cost of performing the Newton method depends in particular on the fenchel conjugate of the penalty Ω * , and is detailed in both these particular cases.

Reproducing kernel Hilbert spaces

In Secs. 2.1.2 and 2.1.4, we have seen that an important choice when building a surrogate problem is the choice of model H (or the model G + in the case of global optimization). For this section, let us take the machine learning notations introduced in Secs. 2.1.1 to 2.1.3, and consider the task of learning a real-valued function f : X → R on a space X . Recall that the ideal requirements for H are that a) it must be large enough to approximate the "ideal" problem well (for example, if the expected risk R reaches its minimum at a certain f * , we could require that f * be in H), b) it must be adapted to the data at hand, being small or regularized enough not to overfit, and c) it must lead to a (rapidly) solvable optmization problem.

Linear models. One of the most widely used classes of functions in machine learning and applied mathemathics is the class of parametric linear models, mentioned in the beginning of Sec. 2.1.1. A linear model is characterized by a feature map φ : X → R p which represents the space X through p features φ 1 , ..., φ p . The φ i 's can be coordinates of x, moments, results of certain filters or convolutions applied to a signal. This feature map can itself be learnt (this is typically the role of encoders in deep learning), or designed according to the problem at hand. The linear model associated to the feature map is the set of functions defined as :

H = f θ : X → R | f θ (x) = θ, φ(x) R p .
(2.28)

These models have many advantages from a modelling perspective. They are usually interpretable (the coefficient associated to a feature being a measure of its importance), they inherit the Euclidean structure of R p (as well as any other normed structure defined on R p ), and preserve the convexity of convex loss functions, allowing to leverage the entire set of optimization techniques in that setting as described in Sec. 2.1.3.

However, these models are parametric, and therefore are quite limited : the dimension p is finite, and they cannot adapt to any number of data points (indeed, one has to consider a larger model as the number of data points gets larger). Moreover, the choice of the feature map is sometimes not obvious, especially in cases where there is little a priori knowledge on the function f (take for example the problem of learning a function f where the only information we have is that it is regular).

In this section, we will present a type of model for functions called reproducing kernel Hilbert spaces (RKHS). This model generalizes linear models to potentially infinite dimensional spaces, leading to better approximation properties, and is focused not on the design of the feature map but rather on that of the kernel, which is a measure of similarity between points in X . Moreover, the nice properties of linear models still hold, making this type of model a very interesting tool for machine learning and applied mathematics in general.

This section will be organized as follows. In Sec. 2.2.1, we will define what a RKHS is, and provide different points of view on that definition. In Sec. 2.2.2, we will provide examples of kernels and RKHS, some of which will be used in this thesis, and illustrate the strength of the kernel approach for approximation. In Sec. 2.2.3, we show how RKHSs are particularly adapted in the 2.2. REPRODUCING KERNEL HILBERT SPACES 67 empirical risk minimization setting, and how to design finite dimensional surrogate problems for the expected risk minimization problem even when the underlying space is infinite dimensional. In Sec. 2.2.4, we will discuss the statistical properties of kernel methods. Finally, in Sec. 2.2.5 we will briefly discuss the problems and perspectives of effectively solving the optimization problem (more details will be given in part I).

Definition and constructions of reproducing kernel Hilbert spaces

In this section, we define the different fundamental objects in relation to reproducing kernel Hilbert spaces (RKHS), through three different points of view. Let us formally define a reproducing kernel of a Hilbert space [START_REF] Aronszajn | Theory of reproducing kernels[END_REF][START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF][START_REF] Shawe | Kernel Methods for Pattern Analysis[END_REF].

Definition 2.1 (reproducing kernel). Let H be a Hilbert separable space of real-valued functions defined on a set X . A function k : X × X → R is a reproducing kernel of H if :

• the functions k x : x ∈ X → k(x, x ) all belong to H;

• the reproducing property is satisfied with the k x :

∀x ∈ X , ∀f ∈ H, f (x) = f, k x H (2.29)
If a Hilbert space of functions has a reproducing kernel, then that reproducing kernel can be shown to be unique (given H, we can talk about its reproducing kernel, if it has one). Conversely, if H and H have the same reproducing kernel k, then they are equal. Thus, if H exists, H is called the reproducing kernel Hilbert space with kernel k. We now present three constructions of RKHS, starting from different perspectives. The most important in machine learning is arguably the second one.

Construction through a feature map. The first way to build a RKHS is to take the point of view of linear models and start from a feature map. Assume that we are given a feature map φ : X → H which represents X into a Hilbert space. As in the linear model case, we can define the associated space of functions :

H := f : X → R : ∃θ ∈ H, f (x) = θ, φ(x) H .
(2.30)

We are now going to show that H is indeed a reproducing kernel Hilbert space, with kernel k(x, x ) := φ(x), φ(x ) H . Note that this kernel is positive definite (p.d.), that is a) symmetric, i.e., k(x, x ) = k(x , x), and b) for any n ∈ N and any (x i ) ∈ X n , the matrix (k(x i , x j )) 1≤i,j≤n is positive semidefinite (this comes from the fact that this matrix is a Gram matrix). k(x, x ) is the natural measure of similarity between points implied by the feature map. We still denote with f θ the functions θ, φ(x) H .

Let H 0 := {θ ∈ H : ∀x ∈ X , θ, φ(x) H = 0} = {θ ∈ H : f θ = 0} be the set of vectors θ ∈ H which represent the function f = 0, so that f θ = f θ i.i.f. θθ ∈ H 0 (in a sense, H 0 represent the useless θ in terms of function evaluations). The there is a natural linear isomorphism between the orthogonal of H 0 and H. Note that H 0 can itself be seen as the orthogonal of the set {φ(x) : x ∈ X } and that the orthogonal of H 0 is therefore H φ := span({φ(x) : x ∈ X }). Thus, there is a natural isomorphism between H φ and H, and H can therefore be naturally equipped with the Hilbert structure of H φ . The fact that k is the reproducing kernel of H then follows.

CHAPTER 2. GENERAL INTRODUCTION

Note that the isomorphism between H and H φ directly shows that H = span({k x : x ∈ X }). In fact, the map x ∈ X → k x ∈ H is a way of representing X in the Hilbert space H (it is a feature map). The reason for which we distinguish H and H is that the space H is a space of functions on X while H is not (it can be larger). Indeed, any Hilbert space on which an embedding φ : X → H exists contains a subspace isometric to H, H φ . Moreover, the inner product of H, while inherited from that of H, is not exactly the same. Indeed, if

f i (•) = v i , φ(•) H , i ∈ {1, 2}, we have f 1 , f 2 H = P H φ v 1 , P H φ v 2 H and not f 1 , f 2 H = v 1 , v 2 H .
In practice, it is usually hard to construct an infinite dimensional feature map. That is why, contrary to linear models, the point of view of reproducing kernel Hilbert spaces taken in machine learning is usually centered around the kernel function k, which can be used to construct the space H.

Construction from the kernel function. In this case, we view the RKHS from the point of view of the kernel. A kernel function on a space X is simply a function k : X × X → R.

Kernels play an essential role in many branches of mathematics, partial differential equations, harmonic analysis, probability theory. In most of these branches, kernels intervene through a kernel operator on a space of measures M on X . This operator is usually defined as a convolution with the kernel k, that is (T k µ)(x) = X k(x, x ) µ(dx ) for measures µ in M . This point of view will be mentioned in Sec. 2.2.4. However, in machine learning, the setting is somewhat less abstract.

In machine learning, the kernel is seen as a similarity measure. This point of view gives natural ways to design kernels for certain specific tasks, in which a good similarity measure can be designed (which can be easier than designing a feature map). This is the case for biological sequences for instance, where kernels have been used to compare DNA and protein sequences (see [START_REF] Jaakkola | Using the fisher kernel method to detect remote protein homologies[END_REF] and subsequent work). We will give examples of kernels in the next section Sec. 2.2.2.

Sine we would like the kernel k to be a similarity measure between two elements of X , a good feature based linear model would be a model where the feature map φ : X → H (where H is a Hilbert space) reflects this similarity measure : φ(x), φ(x ) H = k(x, x ). This is exactly the role of a RKHS H with kernel k as described above. The cornerstone of kernel methods is that we can indeed find such a space H, given any p.d. kernel (see definition in Eq. (2.33)).

Basic functions. As we saw in the previous paragraph, if H is a RKHS with kernel k, it must necessarily be the closure of the space span{k x : x ∈ X } for the right Hilbert structure (that is is must satisfy k x , k x = k(x, x )). Therefore, to build a RKHS associated to k, we start by considering the space of elementary functions defined by the kernel.

H 0 = span({k(x, •) : x ∈ X }) = f : X → R | f (x) = n i=1 α i k(x i , x), n ∈ N, (x i ) ∈ X n , (α i ) ∈ R n . (2.31)
Bilinear form This space can be equipped with a bilinear form defined by the kernel (in order to 2.2. REPRODUCING KERNEL HILBERT SPACES (2.32) for any n, (x i ), (α i ) and m, (x j ), (β j ). The condition for this bilinear form to be a scalar product is exactly that the kernel k be positive definite, as defined in the previous paragraph, i.e., k is symmetric (k(x, x ) = k(x , x) for all x, x ), and

69 satisfy k x , k x H 0 = k(x, x )) : n i=1 α i k(x i , •), m j=1 β j k(x j , •) H 0 = n i=1 m j=1 α i k(x i , x j )β j ,
∀n ∈ N, ∀(x i ) ∈ X n , ∀(α i ) ∈ R n , n i,j=1 α i k(x i , x j )α j ≥ 0. (2.33)
Construction of H. Under the assumption that k is p.d., the space H 0 is called a pre-Hilbert space, that is a vector space equipped with a scalar product. From H 0 , we can construct the space H as the completion of H 0 with respect to the norm defined by the scalar product Eq. (2.32). Since in that case, H 0 = H, it is exactly the reproducing kernel Hilbert space associated to k. This result is due to [START_REF] Aronszajn | Theory of reproducing kernels[END_REF] who attributes it to [START_REF] Moore | Abstract for "On properly positive Hermitian matrices[END_REF].

Theorem 2.1 [START_REF] Aronszajn | Theory of reproducing kernels[END_REF] This point of view allows to directly understand that certain classical Hilbert spaces of functions are RKHSs. This is the case of Sobolev spaces W s 2 (Ω) for Ω ⊂ R d , which are reproducing kernel Hilbert spaces for s > d/2. However, finding an expression for the kernel is often complicated. In the case where Ω = [0, 1], it is known for s = 1 where k(x, y) = max(x, y). However, the expression for general s is much more complex (see [START_REF] Wahba | Spline Models for Observational Data[END_REF]). In Sec. 2.2.2, we will give examples of such kernels. Moreover, this point of view also emphasizes that RKHS are particularly adapted when the function f is known via function evaluations. In a sense, a reproducing kernel Hilbert space contains "diracs", i.e., elements k x with porperties which are similar to those of diracs in L 2 which satisfy f, δ x L 2 (R d ) = f (x), but which lie in the space itself.

Kernels in machine learning. In this section, we have seen three ways of looking at RKHS and constructing them : one through a Hilbert valued feature map, one through a p.d. kernel on X , and one by looking directly at the space of functions H. The last two constitute the two main trends in the literature using RKHS.

The approach centered around function spaces uses the kernel k as a tool to describe and characterize the space of function H, in particular through the kernel operator T k defined above. This approach has been used to describe certain spaces in harmonic analysis [START_REF] Zaremba | L'équation biharmonique et une class remarquable de functions fondamentales harmoniques[END_REF].

On the other hand, the approach centered around the p.d. kernel itself has been developed successively by [START_REF] Mercer | Functions of positive and negative type, and their connection with the theory of integral equations[END_REF], [START_REF] Moore | Abstract for "On properly positive Hermitian matrices[END_REF][START_REF] Hastings | of Memoirs of the American Philosophical Society[END_REF] and [START_REF] Aronszajn | Theory of reproducing kernels[END_REF].

In machine learning, we use both of these approaches. We are interested in the underlying function space from a modelling and statistics perspective. Indeed, the function space H is the space of test functions, that is the space where we look for a solution to our surrogate problem Eq. (2.7). Moreover, the natural Hilbert structure on H and in particular its norm will have a key role in regularization. We will therefore caracterize statistical errors with quantities which illustrate the relationship between our model space of functions H and the true target function f * (if it exists). However, the kernel perspective is also of great importance to actually solve Eq. (2.7). Indeed, as will be shown in the next sections, there are large classes of algorithms and machine learning problems of the form Eq. (2.7) which can be solved using only dot products of feature maps in the case of linear models, and can therefore be adapted to the kernel setting by replacing dot products with the kernel and otherwise ignore the underlying Hilbert space. This is essentially because many problems depend on the function f only through function evaluations on the data f (x i ) (see Sec. 2.2.3). Moreover, as the basic functions of the RKHS are linear combinations of kernel functions centered at different points, these can be used to approximate any function in the RKHS with given degree of precision. This will lead to compressions of functions in H in finite dimension, which will be of paramount importance to control the complexity of optimization algorithms (see Sec. 2.2.5 and chapters 7 and 9).

Examples of kernels

In this section, we briefly present and discuss examples of kernels, some of which we will use in the rest of this thesis. We start with a bit of nomenclature, by defining the following standard classes of kernels.

• Translation invariant kernels are defined on an abelian group X and are of the form k(x, x ) = v(xx ) for a certain function v : X → R;

• Radial basis function kernels (RBF kernels) are defined on a space X equipped with a semi-distance d X and are of the form k(x, x ) = v(d X (x, x )) for a certain function v : R + → R.

• Zonal kernels are defined on a space X with a scalar product • , • X and are of the form v( x, x X ) for a function v : R → R.

In a sense, all these different types of kernels respect an underlying structure of X : group structure, metric structure, Euclidean structure. One type of kernel which will be central to the different articles regrouped in this thesis are translation invariant kernels on X = R d .

Translation invariant kernels on R d . Let k(x, x ) = v(xx ) be a translation invariant p.d. kernel on R d , and assume that v ∈ L 1 (R d ). The RKHS defined by k can be conveniently defined through the Fourier transform.

Recall that given a function f ∈ L 1 (R d ), the fourier transform of f is defined as

f (ξ) = R d f (x)e -i ξ,x R d dx.
(2.34)

Note that since v ∈ L 1 (R d ) and k is p.d., the fourier transform of v is nonnegative and bounded by v L 1 (R d ) . Since the Fourier transform defines an isometry (up to a constant factor) from

L 1 (R d ) ∩ L 2 (R d ) to L 2 (R d
) equipped with the L 2 distance, it can actually be extended to the whole of L 2 (R d ) and hence is defined on L 2 (R d ) as well. As soon as v ∈ L 1 (R d ) and defines a p.d. kernel k, the RKHS associated to k can be characterized using the fourier transform in the following way (see for example [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF]).

H = f ∈ L 2 (R d ) R d | f (ξ)| 2 v(ξ) dξ < ∞ ; ∀f, g ∈ H, f, g H = 1 (2π) d R d f (ξ) g(ξ) v(ξ) dξ.
(2.35)

This clearly shows that the regularity of a function in H is linked to the regularity of the kernel function (as the regularity of f is related to the decay of f ). If k and hence v is very regular, v decays very fast, and hence f must also decay very fast to be in H. Note that if k is also radial (i.e., v(t) = ṽ( t )), then the Fourier transform of v is also radial. We will give examples of such kernels in the next paragraph.

Some generic kernels. In this paragraph, we present some widely used kernels, some of which we will use in parts II and III as they caracterize important regularity properties, or have nice computational properties.

Linear and polynomial kernels are defined as k(x, x ) = x, x X and k(x, x ) = x, x m , m ∈ N respectively on any Euclidean space X . The associated RKHS to the linear kernel is simply the set of linear functions on X while the set of functions associated to the polynomial kernel is simply the set of m homogeneous polynomial functions on X . These RKHS are finite dimensional, and of dimension m-1 d-1 , where d is the dimension of X and m the exponent of the polynomial kernel.

Gaussian kernels, or also Gaussian RBF kernels, are the radial, translation invariant kernels on R d , parametrized by a bandwidth σ, and defined as

∀x, x ∈ R d , k σ (x, x ) = exp(-x -x 2 /(2σ 2 )).
(2.36)

It is one of the most used kernels if no a priori is known or when trying to approximate a regular functions, because it is easy to implement, and because of the freedom given by the choice of the parameter σ. We use it in almost all the applications we perform in this thesis, as well as in the developments in chapter 7, in which its computational properties play a key role. Using Eq. (2.35), the RKHS H σ associated to the Gaussian kernel has norm

f 2 Hσ = σ √ π d R d | f (ξ)| 2 exp(σ 2 ξ 2 /2)dξ.
(2.37)

This shows that the functions belonging to H σ are extremely regular, as their Fourier transform must decay exponentially fast. Moreover, the parameter σ controls the speed of this decay; the larger the σ, the more regular the functions. In chapter 7, we show that choosing the parameter σ in a good way (i.e., in a way adapted to the number of samples) is statistically equivalent to using a Sobolev kernel of given regularity, defined below.

Sobolev kernels, also called Matérn kernels, are defined by [START_REF] Wendland | Scattered Data Approximation[END_REF], and are radial, translation invariant kernels on open sets Ω of R d . They are defined for s > d/2 as :

∀x, x ∈ Ω, k s (x, x ) = c s x -x s-d/2 K s-d/2 ( x -x ), (2.38)
where K is the Bessel function of the second kind (see Wendland, 2004, chapter 5.10) and c s is a normalizing constant (for more details, see chapter 9). In the case where Ω has Lispchitz continuous boundary, the associated RKHS H s can be shown to be equivalent to the Sobolev space W s 2 (Ω) of functions whose derivatives up to order s are square integrable, and its norm is equivalent to the classical norm on these spaces, defined as :

f 2 W s 2 (Ω) = |α|≤s Ω |∂ α f (x)| 2 dx.
(2.39)

Note that when Ω = R d , k s is of the form k s (x, x ) = v s (xx ), and the Fourier transform of v s is given by (1 + |ξ| 2 ) -s . This leads to the following expression for the RKHS norm of H s associated to k s :

f 2 Hs = (2π) -d R d | f (ξ)| 2 (1 + |ξ| 2 ) s dξ, f ∈ H s .
(2.40)

One particular case of Sobolev kernels is the exponential or Laplace kernel, which is the Sobolev kernel for s = d/2 + 1/2 and is equal to exp(-xx ). These kernels are very useful to characterize the regularity of a function in term of derivatives (indeed, the parameter s characterizes the degree of regularity of the functions in the RKHS). This is very useful from a theoretical point of view, to gain intuition on how regularity affects the properties of the learning problem. These kernels will be central examples to understand part I, and the cornerstone kernel in chapter 9, where we use it to leverage the regularity of a function to perform fast global optimization.

Advantages and disadvantages of using "off-the-shelf kernels". Using these "off-the-shelf" kernels has many advantages. From a practical perspective, they are easy to implement and use, and in the case of the Gaussian kernel, allow some tuning to the data by selecting the bandwidth σ, which is interpretable : it is the typical range of action of a data point. From a theoretical perspective, they are also very well known and characterized, and in the Gaussian and Sobolev kernel case, possess a very nice approximation property called universality : they can essentially approximate any function on R d (for more details, see Sec. 2.2.4 and Micchelli, Xu, and Zhang (2006);[START_REF] Bharath | Universality, characteristic kernels and rkhs embedding of measures[END_REF]). However, their level of generality is also their weakness. These kernels are isotropic and therefore can fail to identify manifold structures in high dimensions (on which it is believed the data is usually concentrated). Moreover, as we see with the Gaussian kernel, there is a single scale of interaction given by σ. This can be a problem when multiple scales are needed to understand data (in image processing for instance; that is the role of convolutional layers).

Kernel engineering. In the previous paragraph, we have seen examples of generic kernels, defined mostly on X = R d , which can be used in an "off-the-shelf" way. However, one of the main advantages of kernels is the fact that one can build a kernel suited to a specific task, even on data which is non vectorial (we count images in this setting as the vector does not reflect the geometrical structure of the image). Moreover, this can even be necessary to capture the structure of vectorial data, when it is anisotropic or multi-scale. In this paragraph, we thereofore briefly introduce different ways and examples of building kernels. The basic ways of constructing kernels are mentioned by [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF] and we therefore refer to it for the basic operations (product of kernels, compositions).

Non-vectorial data. It is impossible to list kernels for non-vectorial data as they are intrinsically taylored to the situation at hand. However, we can mention a few examples of such situations : they have been used on biological data [START_REF] Jaakkola | Using the fisher kernel method to detect remote protein homologies[END_REF], on images [START_REF] Harchaoui | Image classification with segmentation graph kernels[END_REF], on graphs in biology [START_REF] Borgwardt | Graph kernels: State-of-the-art and future challenges[END_REF]) and many others.

Building kernels using multiple features. A classical way of designing kernels is, given a potentially infinite number of uniformly bounded features ψ i : X → R, to define the associated Mercer kernel, i.e. k(x, x ) =

i∈I w i ψ i (x)ψ i (x ).
(2.41)

where the w i are non-negative weights which are summable. Note that this exactly corresponds to defining a feature map φ : x ∈ X → ( √ w i ψ i (x)) i∈I ∈ 2 (I). The ψ i are usually designed either as meaningful features or basis functions, which have been developed in many branches such as PDEs or signal processing. One can think for example of wavelet basis or cosine basis cos(ω i x).

Generalizing this, one can consider kernels of the form

k(x, x ) = i∈I w i k i (x, x ), (2.42)
for bounded kernels k i . For example, [START_REF] Opfer | Multiscale kernels[END_REF] considers a basis function ϕ(x) which is rescaled by a factor δ i to form the kernel k i (x, x ) = δ -d i ϕ((xx )/δ i ); they correspond to different scales of interactions between data points. In general, these models can be used a) to model the interactions at different scales (as for images) using kernels k i adapted to different scales δ i , and b) to learn the kernel by learning the weights w i [START_REF] Francis R Bach | Multiple kernel learning, conic duality, and the smo algorithm[END_REF].

Note that Eq. (2.41) can be generalized (up to rescaling) to define random feature kernels [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF] , i.e., kernels of the form

k(x, x ) = E ω∼µ φ(x, ω)φ(x , ω) = Ω φ(x, ω)φ(x , ω)µ(dω).
(2.43)

These form of kernels play a crucial role in dimension reduction techniques as we will see in Sec. 2.2.5 and chapter 3. For example, the Gaussian kernel can be seen as a random feature kernel :

k σ (x, x ) = E ω,b cos(ωx + b) cos(ωx + b) ω ∼ N (0, 1/σI d ), b ∼ Uniform(0, 2π). (2.44)
Summary. In this section, we gave examples of kernels and RKHS used in practice and references to how to build them. In particular, we described the most commonly used "off-theshelf" kernels as well as the induced RKHS, discussed their limitations, and gave certain directions as to how people in the community have tackled these problems and designed problem-specific kernels.

Kernels for empirical risk minimization

In this section, we describe how RKHS are adapted in the context of learning, using the framework from Sec. 2.1. Recall that our "ideal" problem is the expected risk minimization problem, of the form

CHAPTER 2. GENERAL INTRODUCTION R * = inf f ∈F R(f ) = E [ Z (f )], (2.45)
where F is an ideal class of function (think of it as the largest class of functions on which the risk is defined), Z a random variable on a space Z with distribution ρ representing the data, and is a loss function. We denote with f * any minimizer of Eq. (2.45) it it exists. In this section, we will assume that a) F is a class of functions from a space X to R, and b) that we have a map π : Z → X such that z (f ) is of the form (z, f (π(z))), which we will sometimes denote with z (f (π(z))) . This rather abstract assumption simply states that the loss of f at data point z depends on f only through the evalution of f at a point x = π(z) given by the data (indeed we assume π to be known). Define X = π(Z); the expression of the risk becomes

R(f ) = E [ Z (f (X))].
Remark 3. We want to emphasize the difference in notations which might be confusing. In the general case introduced in Sec. 2.1 and which we will again use in chapter 4, the loss function is a map : Z × F → R and (z, f ) evaluates the performance of f at z, while here, the map is a map : Z × R → R which evaluates the performance of the function value of f at z which is f (x) ∈ R at x = π(z). The fact that the loss only depends on f through function values is key for Theorem 2.2.

To make things clearer, take the supervised learning setting, where the goal is to predict an output Y ∈ R from an input X ∈ X . The data random variable is therefore Z = (X, Y ) ∈ Z = X ×R, and the risk is usually of the form E (X,Y ) (f (X)) ; in that case, the map π is simply the projection on the first coordinate. In the least-squares regression case, we had z (f ) = f (x) -y 2 , and hence this corresponds to z (t) = 1 2 ty 2 . For logistic regression, the loss function was defined as z (f ) = log(1 + exp(-yf (x))), which corresponds to z (t) = log(1 + exp(-yt)).

The fact that the loss function depends only on function evaluations allows us to reformulate a) the problem of minimizing the expected risk on H, and b) the problem of minimizing the (regularized) empirical risk on H in a way that f appears only in a scalar product with an element k x . Indeed, the expected risk minimization on H can be written

R H = inf f ∈H E [ Z ( f, k X H )].
(2.46)

We will denote with f H a minimizer of Eq. (2.46) if it exists. As seen in Eq. (2.9) in Sec. 2.1.2, the surrogate problem which we will effectively solve is the so-called empirical risk minimization problem :

f n,λ = arg min f ∈H 1 n n i=1 z i ( f, k x i H ) + λ 2 f 2 H , x i = π(z i ), (2.47)
where z 1 , ..., z n are the data points, the x i = π(z i ) are the evaluation points of f associated to the z i , and the regularization term λ 2 f 2 H is called the Tikhonov regularization. This regularization term is the most standard, but note that other regularization procedures exists, which we will not use in this thesis [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]. As RKHS are usually infinite dimensional, a regularization is usually necessary, as explained in Eq. (2.9) and detailed in Sec. 2.2.4. One of the cornerstone theorems in kernel methods is the following.

Theorem 2.2 (Representer theorem, [START_REF] Cucker | On the mathematical foundations of learning[END_REF]). Let L : R n → R ∪ {+∞} be a lower semi-continuous loss function which is bounded below and let x 1 , .., x n ∈ X , and H a RKHS. If the problem

min f ∈H L(f (x 1 ), ..., f (x n )) + λ 2 f 2 H , (2.48)
has a solution, then there exists a solution of the form

f (x) = n i=1 α i k(x i , x), α ∈ R n .
(2.49)

Moreover such a solution always exists for λ > 0.

In other words, any problem of the form Eq. (2.48), that is which depends on f only through its evaluations at x 1 , . where K = (k(x i , x j )) 1≤i,j≤n ∈ R n×n it the kernel matrix associated to the data points (x i ). The second consequence is that the resulting function f n,λ can be expressed only using the kernel :

f n,λ (x) = n i=1 α i k(x i , x).
To summarize, the main advantages of kernels in this setting is the fact that a) although RKHS can be infinite dimensional, the resulting empirical risk minimization problem is finite-dimensional, and b) the problems and resulting functions can be described using only the kernel function, omitting the abstract definition of the RKHS. However, note that the representer theorem (Theorem 2.2) only holds if the empirical risk depends on the function f only through evaluations. This does not include cases where constraints on f may be added. This can sometimes be useful, for example when learning probability densities (as in chapters 6 and 7), where a natural constraint can be that the function sums to one. However, we can show that under certain conditions, looking for f in H n can be sufficient statistically (see chapter 7). In this thesis, we will try to move a bit aside from the representer theorem paradigm, and try to show that kernel methods perform well not because H n contains the empirical risk minimizer, but because when one has n data points, H n is a good enough approximation of H. In the two next sections, we will briefly discuss the satistical properties of the empirical risk minimizer f n,λ , and the optimization properties of solving the problem Eq. (2.50).

A statistics point of view on e.r.m. with kernels

Running example : least-squares. In the next two sections, and in chapter 3, we will use the kernel least-squares regression problem as a running example. Recall from Sec. 2.1 that in that case, the expected risk is R ls (f ) = 1 2 E f (X) -Y 2 . Moreover, we will assume that Y ∈ L 2 which shows that there exists an optimal predictor f * which is in L 2 (X , ρ X ). Moreover, in this case, there is a closed form solution to the regularized empirical risk minimization problem Eq. (2.50) given data points (x i , y i ) and λ > 0 :

f n,λ = n i=1 α i k(•, x i ) α = (K + λnI) -1 y, y = (y i ) ∈ R n . (2.51) CHAPTER 2. GENERAL INTRODUCTION
Handling the approximation error : universality. Recall from Sec. 2.1.2 that the standard way of decomposing the excess risk of the empirical risk mimizer f n,λ is to separate approximation and estimation error : R(

f n,λ ) -R * = (R H -R * ) + (R( f n,λ ) -R H ).
In kernel methods, as we will see in the next paragraph, one usually only looks at the estimation error.

Indeed, since RKHSs are usually infinite dimensional, the approximation error R H -R * can be shown to be zero for certain RKHSs and certain losses. These results are usually called universality results; they aim to show that under certain conditions on the RKHS H (or the kernel k), H can approximate any function f ∈ F, i.e. is dense in F for a certain class of functions F. Of course, "approximating any function" has to be defined, and many definitions can be given [START_REF] Micchelli | Universal kernels[END_REF][START_REF] Bharath | Universality, characteristic kernels and rkhs embedding of measures[END_REF]. If X is a locally compact Hausdorff space for example (this is the case of all spaces X we consider), one type of universality is the L p -universality : H is dense in L p (X , ρ X ) for any given Borel probability measure ρ X (usually the law of X in the supervised setting), where L p (X , ρ X ) is equipped with the L p norm. Another type of universality, cc -universality, is the fact that H is dense in the set C(X ) of continuous functions on X for the compact convergence (i.e., for any compact K ⊂ X and any function f ∈ C(X ), there exists a sequence f n ∈ H of functions such that sup x∈K f n (x) -f (x) -→ n→∞ 0). For instance, in the least-squares problem, the approximation error is zero as soon as the kernel is L 2 -universal for the measure ρ X . The Gaussian and Sobolev kernels can be proved to be L p -universal and cc-universal for instance [START_REF] Micchelli | Universal kernels[END_REF][START_REF] Bharath | Universality, characteristic kernels and rkhs embedding of measures[END_REF].

Remark 4. The approximation error being zero does not mean that f * , if it exists, is in H. Zero approximation error simply means that there exists a sequence

f n of functions in H such that R(f n ) → R(f * ).
Indeed, if H is the Sobolev space or order s, it is of course not true that every function in L 2 (R d , ρ X ) for a given probability measure ρ X is also in W s 2 (R d ). In the following discussion and throughout this thesis, we will only analyse the estimation error, and make hypothesis which always ensure that R H = R * , therefore making the estimation error the actual excess risk.

Bias variance trade-offs in kernel methods. As explained in Sec. 2.1.2, when the space H is large, we modulate its capacity using some form of regularization. In this thesis we will use the Tikhonov regularization λ 2 f H as it depends only on the kernel norm, and has good optimization properties (it will make the objective strongly convex, see Sec. 2.1.3). We mention other regularizations in chapter 3.

In the case of kernel methods, the traditional approximation-estimation decomposition can be replaced by a so-called bias-variance decomposition of the estimation error, of the form

R( f n,λ ) -R H ≤ Bias(λ) + Variance(λ, n), (2.52)
where the bias term depends only on λ and can be seen as an analog of the approximation error for the space "H regularized with λ", and the variance term is an analog of the estimation error, and quantifies the statistical impact of having access to the true distribution only through the n data points.

More precisely, in the case of Tikhonov regularization, if f λ is a minimizer of the regularized expected risk R(f ) + λ 2 f H (which exists as soon as R is lower semi-continuous and lower bounded as will be the case for all losses considered), then one can see the bias term as the excess risk of f λ : Bias(λ) ≈ R(f λ ) -R H . This bias term can be quantified by the regularity of the optimal solution f * with respect to the chosen RKHS H. If f * is in H for example, the bias term can be shown to behave as Bias(λ) = O 0 (λ). If f * is not in H, then the bias term will converge more slowly towards 0. This behavior is more precisely quantified in the context of least squares, briefly described below.

For the variance term, the typical form we will be looking for is Variance(λ, n) = d λ n , where d λ is a notion of intrinsic dimension of the space H regularized with λ. This type of bounds is called "dimensionless" bounds, not because no notion of dimension appear, but because the dimension of the space H does not appear, which is necessary if one is to quantify the variance on an infinite dimensional Hilbert space.

In chapter 4, we establish such bias-variance decompositions in the case where the loss satisfies a certain property called generalized self-concordance. This decomposition has already been well studied in the literature in the least-squares setting.

The case of least squares. In the case of least-squares regression, such bias variance decompositions have been obtained in order to study optimal rates of convergence [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF][START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]. In the setting of Tikhonov regularization, we can prove the following bias variance decompositions in high probability (a more formal statement can be found in chapter 3) :

R( f n,λ ) -R H ≤ C     b λ bias + d λ n variance     , (2.53)
where b λ is the bias term and d λ is the so-called effective dimension of H [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF] and C is a constant depending on the probability with which we want this statement to hold (as the sample points z 1 , ..., z n are random). Informally, the effective dimension is a measure of the size of H with respect to L 2 (X , ρ X ), where we know the optimal function f ρ lives (we emphasize the link of the optimum with the measure in this section using this notation).

To understand the behavior of the bias term as well as the definition of the effective dimension, we need to introduce a bit more kernel machinery. Assume that the kernel k is continuous and bounded (i.e., k(x, x) ≤ κ 2 ), and that X is separable. Let ρ be the data distribution, ρ X be its marginal on X and assume that Y ∈ L 2 (X × R, ρ), that is is square integrable.

Understanding the effective dimension. By boundedness of the kernel, functions in H also belong to L 2 (X , ρ X ) ( f L 2 ≤ κ f ) and that the S : H → L 2 (X , ρ X ) is a linear continuous map between Hilbert spaces. Define the covariace operator :

Σf = S * Sf = X f (x )k x ρ X (dx ) such that f, Σg = f, g L 2 (X ,ρ X ) .
(2.54)

It is a trace-class, symmetric positive semidefinite operator (hence compact), and describes the relationship between the H-norm and the L 2 -norm, which is the one adapted to the leastsquares problem. As Σ is a trace-class operator, it can be decomposed using the spectral theorem as i∈I λ i φ i ⊗ H φ i where the λ i are positive values sorted in decreasing order, I is either of the form {1, ..., k} if Σ is finite rank or N (which is the most common case), and CHAPTER 2. GENERAL INTRODUCTION φ i are corresponding renormalized eigenvectors. The space H can then be decomposed as

H = span({φ i : i ∈ I}) ⊥ ⊕ Ker(Σ)
, where Ker(Σ) is the set of all functions in H such that f L 2 (X ,ρ X ) = 0 (this is possible if the support of f is disjoint from the support of the measure ρ X ). Note that the spectrum of Σ gives a lot of quantitative information about the size of H with respec to L 2 . If the spectrum is finite, this means that only a finite dimensional subset of H is actually meaningful to approximate a function in L 2 norm. More generally, a large λ i will be associated to an eigenvector capturing a lot of L 2 information, whereas eigenvectors associated to small λ i will capture almost no information about the L 2 structure. The speed at which the eigenvalues converge to zero is therefore a good indicator of the size of the space H with respect to the ideal space L 2 (X , ρ X ). This is exactly what the effective dimension d λ measures, and is defined as

d λ = Tr((Σ + λI) -1 Σ) = i∈I λ i λ + λ i , λ > 0.
(2.55)

The effective dimension d λ is a quantity such that d λ -→ λ→0 +∞ as soon as the spectrum is infinite;

what is interesting is the speed at wich this quantity goes to infinity. This is linked to the eigenvalue decay λ i . For example, if H is a the RKHS associated to the Sobolev kernel k s (see Sec. 2.2.2), and if the measure ρ X is comparable to the Lebesgue measure (there exists a constant K such that ρ X ≤ KΛ where Λ is the Lebesgue measure), the eigenvalue decay is of the order (2s) . This confirms the intuition that the smaller the space (the larger the s) the smaller d λ gets.

λ i = O(i -2s/d ), which implies that the d λ ≤ Cλ -d/
Understanding the bias term. The bias term is a term which quantifies the regularity of f ρ with respect to the RKHS H. We have b λ = o λ→0 (1) and the speed of convergence of b λ towards zero is characterized by the regularity of f ρ . If f ρ ∈ H, we have b λ = O(λ). If f ρ is very regular, it goes even faster towards zero. To formally define what the regularity of f ρ with respect to H means, define the kernel operator associated to k :

T k : L 2 (X , ρ X ) → L 2 (X , ρ X ) such that ∀g ∈ L 2 (X , ρ X ), (T k g)(x) = X k(x, x )g(x )ρ X (dx ).
(2.56)

The operator T k is linked to the covariace operator through the map S which injects H in L 2 (X , ρ X ); we have Σ = S * S and T k = SS * . In particular, T k = i∈I λ i ψ i ⊗ L 2 ψ i where the λ i are the same eigenvalues as those of Σ and the ψ i are the corresponding normalized eigenvectors, which are related to the eigenvectors of Σ, seen as elements of L 2 , by the relation

ψ i = φ i / √ λ i . The ψ i form an orthonormal basis of Ker(T k ) ⊥ ⊂ L 2 (X , ρ X )
. Moreover, by the Mercer theorem (see [START_REF] Carmeli | Reproducing kernel Hilbert spaces and Mercer theorem[END_REF] and [START_REF] Dieuleveut | Nonparametric stochastic approximation with large step-sizes[END_REF] for an extension), it can be shown that H = range(T

1/2 k ) ⊕ H 0 where H 0 = {f ∈ H : f 2 L 2 (X ,ρ X ) = 0}. Note that formally, L 2 (X , ρ X ) = range(T 0 k ).
In general, we will say that f ∈ L 2 (X , ρ X ) is r regular if it belongs to the set H r := range(T r/2 k ) for r ≥ 0. We have H 0 = L 2 (X , ρ X ), H 1 ⊂ H, and in general H r ⊂ H for r ≥ 1. As r increases, the regularity of the elements of H r increases. Making the assumption that f ρ ∈ H r is a good way of quantifying the regularity of f ρ (which is r = 0 a priori ). The bias b λ can be controlled by the regularity, and as expected, the more regular the function, the faster b λ goes to zero :

b λ ≤ C λ r if f ρ ∈ H r ,
(2.57)

REPRODUCING KERNEL HILBERT SPACES

79

for a constant C depending on the norm of f ρ in H r . Note that this condition can also be expressed as a condition on the coefficients f i of the decomposition of f ρ over the eigenbasis ψ i with respect to the λ i , i.e., f ρ ∈ H r i.i.f i∈I f 2 i /λ r i < +∞. Example 2.1 (Uniform measure and Sobolev spaces). In the Sobolev kernel case, and if ρ X is the uniform measure over X , and if H s denotes the RKHS associated to k s , f ρ ∈ H r s simply means that f ρ ∈ W sr 2 (X ), which corresponds exactly to the interpolation between Sobolev spaces. Conclusion. In the least-squares case, we have a very fine way of quantifying the bias-variance trade-off, and a good understanding of the role of the regularity of f ρ as well as that of the effective dimension of H in the performance of the empirical risk minimizer. This can help us gain some intuition on why a learning problem is hard or easy, or what to do to reduce its dimension (see part I).

An optimization point of view on e.r.m. with kernels

Kernel methods have both great potential advantages and drawbacks in terms of optimization to find the empirical risk minimizer f n,λ . Recall that using the representer theorem, finding the e.r.m. minimizer is equivalent to solving Eq. (2.50) recalled here for clarity :

α ∈ arg min α∈R n 1 n n i=1 z i (e i Kα) + λ 2 α Kα.
The main advantages are that the problem inherits convexity from the loss functions l z (•) as well as the smoothness of l z as soon as the kernel is bounded. Moreover, if we reparametrize the problem setting β = K 1/2 α, the problem is λ-strongly convex in β, which allows to apply fast first order methods. The disadvantages are that the kernel matrix can be huge (of size n × n), which is prohibitive for large data sets both in terms of memory and computational cost (as computing one stochastic gradient would cost O(n) and a full gradient O(n 2 )). Moreover, reparametrizing with β such that α = K 1/2 β is both necessary (because the kernel matrix is very ill-conditioned) and costly (O(n 3 ) due to the matrix inversion and cannot be fully parallelized on GPUs). Even then, the λ which appear to be optimal after evaluation of the model are very small in many cases (see the case of the Higgs data set in chapter 5). This means that the problem will be very ill-conditioned even after reparametrization, and that the classical first order methods will be slow.

Thankfully, in recent years, ways have been developed to overcome these challenges mainly in the context of least-squares regression. The first main area of improvement has been dimension reduction; i.e. reducing the dimension of the optimization problem to a certain m n. There are two main ways of doing so. One method is to sub-samples points x1 , ..., xm from x 1 , ..., x n and looks for a solution to the e.r.m. problem restricted to the set of functions of the form f (x) = m j=1 αj k(x j , x) (this is called Nyström or column sub-sampling, see [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]). The other is to have a kernel whose expression is of the form k(x, x ) = E ω∼p [φ(x, ω)φ(x , ω)], that as an expectation of random features (see Eq. (2.43) and the following discussion on the Gaussian kernel); in that case, one approximates the kernel matrix with a low rank approximation of the form

K = RR where R = 1 √ m (φ(x i , ω j )) 1≤i≤n 1≤j≤m
where the ω j are drawn from p. We will detail the properties of these methods in chapter 3; they allow to reduce the dimension from n to m of order d λ , without losing any statistical performance for the empirical risk minimizer. The second area of improvement has been algorithms. Using the conjugate gradient method Eq. (2.19) in proposition 2.1 along with a pre-conditioning technique (for more details, see chapter 3), [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF] have sped up the solving of the dimension-reduced e.r.m. to a complexity of order roughly O(nd λ ), without paying the price of ill-conditioning. These methods have been implemented in a library by Meanti, Carratino, Rosasco, and Rudi (2020) and allow to handle billions of points, using in particular subroutines developed by [START_REF] Charlier | Kernel operations on the gpu, with autodiff, without memory overflows[END_REF] to compute kernel evaluations k(x, x ) in a fast way.

In part I and more specifically in chapter 5, we extend these techniques for other losses than the square loss. These techniques have since been implemented in a library by Meanti, Carratino, Rosasco, and Rudi (2020), along with the fast least-squares techniques.

Outline of the thesis

In this section, we give a very broad outline of the thesis and its contributions; these will be made precise in the introduction of each part. The thesis is organised in three main parts, which each corresponds to two published or submitted articles on a specific topic. Note that this outline may have some redundencies with the introductions of these different works.

Part I : fast rates and algorithms for logistic regression

This first part is based on the two articles by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]; [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]. In these two works, we study the classical problem of expected risk minimization Eq. (2.4) through the solving of the regularized empirical risk minimization problem Eq. (2.7), on a reproducing kernel Hilbert space H with kernel k (see Sec. 2.2). The goal of these two works was to extend the precise bias-variance trade-offs, fast rates and fast algorithms already known for least-squares to generalized self-concordant losses. This is an interesting class of losses as it contains the logistic regression loss x,y (f ) = log(1 + exp(-yf (x))), and the multiclass logistic regression loss (see Sec. 2.1.1). [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] start by studying the excess risk of the regularized empirical risk minimizer f n,λ . We show that as in the least-squares case, there is a bias-variance trade-off of the form

R( f n,λ ) -R H ≤ C log 1 δ b λ + d λ n , with probability at least 1 -δ, (2.58)
where the bias term and the effective dimension are defined as meaningful quantities which match those defined in the least-squares case (where essentially, one replaces the covariace operator with the Hessian matrix of the expected risk at the optimum). In particular, this allows us to essentially derive upper rates of convergence.

Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] develop a practical algorithm to solve large-scale kernel logistic regression, which is now used in the library created by Meanti, Carratino, Rosasco, and Rudi (2020). It is based on three contributions.

• The first is to show that as in the least-squares case, the e.r.m. problem can be restricted to a much smaller dimensional problem through Nyström sampling (we reduce the dimension from n to m ≈ d λ ).

• The second is to design a globally convergent second-order algorithm for generalized selfconcordant losses, whose complexity is of order O(nd λ ) up to log factors. Similarly to interior point methods, this method uses a decreasing λ scheme (λ 0 , ..., λ T ) to iteratively optimize the regularized empirical risk for λ t+1 starting at the previous estimation f n,λt . The optimization at each step is performed using an approximate Newton method, which allows to approximately compute Newton steps using Hessian sketching.

• The third is to prove that the result of the algorithm achieves the same statistical error as that of the full e.r.m. estimator, i.e., it satisfies a bound of the form Eq. (2.58).

Crucially, this method is independent of the conditioning of the problem (up to log factors). We illustrate this fact on real-life data sets where the optimal regularization parameter λ has to be taken very small.

Part II : introducing non-parametric models for non-negative functions, and applications to sampling

This part is based on the two articles by Marteau-Ferey, Bach, andRudi (2020, 2022a). In this part, we develop a model for non-negative functions based on reproducing kernel Hilbert spaces, and apply this model to sample from probability distributions given their un-normalized density function, breaking the curse of dimensionality with regularity.

In chapter 6, we describe the work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. We consider a class of models with non-negative outputs, which exhibit the same properties as linear models and kernel methods. This model is to consider functions parametrized by a PSD operator on a Hilbert space H on which the input space X is mapped through φ : X → H with linear models notations and x ∈ X → k x ∈ H in RKHS notation :

{f A : A 0} f A (x) = φ(x) Aφ(x) or f A (x) = k x , Ak x H , x ∈ X .
(2.59)

We call these models PSD models. As this model is itself linear, it can directly be used in the same way to solve e.r.m. problems Eqs. (2.47) and (2.50). We derive a representer theorem similar to Theorem 2.2 for our models in the context of empirical risk minimization, and provide a convex finite-dimensional dual formulation of the learning problem, depending only on the training examples. We also prove that this model has good approximation properties, and apply the method to the problems of density estimation, regression with Gaussian heteroscedastic errors, and multiple quantile regression. We derive the corresponding learning algorithms for convex dual formulation, and compare it with standard techniques used for the specific problems on a few reference simulations.

In chapter 7, we describe the work by Marteau-Ferey, Bach, and Rudi (2022a). This work applies the modelling framework above to the problem of sampling n i.i.d. samples from a distribution whose density is known up to a constant through function evaluations. Contrary to most of the existing methods in the literature such as rejection sampling or Monte-Carlo Markov chain methods [START_REF] Gelman | Bayesian Data Analysis[END_REF][START_REF] Jun | Monte Carlo Strategies in Scientific Computing[END_REF][START_REF] Lelièvre | Free Energy Computations[END_REF][START_REF] Robert | Monte Carlo statistical methods[END_REF], we propose a two-step procedure by first modelling this density using a positive semidefinite model, and then sampling from this PSD model using an adapted algorithm. In particular, we use PSD models with the Gaussian kernel defined in Sec. 2.2.2 (we CHAPTER 2. GENERAL INTRODUCTION will call these models Gaussian PSD models), that is we approximate the target density with a function of the form

f A (x) = n i,j=1 A ij k σ (x i , x)k σ (x j , x), A 0, k σ (x, x ) = exp(-x -x 2 /(2σ 2 )).
(2.60) modelling probability distributions with Gaussian PSD models has been developed by [START_REF] Rudi | Psd representations for effective probability models[END_REF]. In this work, we continue to explore the good properties of this model for probability distributions, and derive an algorithm that is easy to implement and which can generate an arbitrary number of i.i.d. samples from a given Gaussian PSD model, with any given precision.

We then show that we can sample an arbitrary number of i.i.d. samples from a target probability distribution that is regular enough, with any given precision. The algorithm consists in (a) approximating the un-normalized density p via a PSD model, using evaluations of p, and (b) extracting i.i.d. samples from the PSD model. We show that for sufficiently regular densities the resulting PSD model is concise and avoids the curse of dimensionality : to achieve error ε, the Gaussian PSD model requires a number of parameters and a number of evaluations of p that are in the order ε -2-d/β , where d is the dimension of the space and β is the order of differentiability of the density. (2.25) where the functional constraints fc = g and g ≥ 0 have to hold pointwise for all x ∈ X . Similarly to the poynomial sum of squares setting [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF], we tighten the constraint g ≥ 0 by asking g to be a PSD model of the form :

2.3.3

subject to c ∈ R, g = f -c, g ≥ 0,
g A (x) = k x , Ak x Hs , A 0, Tr(A) < ∞, (2.61)
where H s is the Sobolev space on X equipped with the Sobolev kernel k s . Eq. (2.25) is thus tightened as sup c

subject to c ∈ R, f (x) -c = k x , Ak x Hs , x ∈ X , A 0, Tr(A) < ∞.
(2.62)

In order to solve a finite dimensional problem, and assuming we have access to f through function values, we use the following surrogate problem which replaces the constraint fc = g A by the same constraint evaluated at n sample points x 1 , ..., x n :

sup c -λ Tr(A) subject to c ∈ R, f (x i ) -c = k x i , Ak x i Hs , x ∈ X , A 0, (2.63)
where we have added an extra regularization term λ Tr(A) to avoid overfitting due to the finite amount of data. Note that this problem is neither a tightening or a relaxation of the initial problem, which is one of the main differences with polynomial optimization. [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] show that the SDP in Eq. (2.63) can be solved with ε error using a Newton method in time O(n 3.5 log 1 ε ). Moreover, we prove that using the Sobolev kernel k s with s = r -3, the obtained estimation c n,λ of the minimium f * satisfies roughly

c n,λ -f * ≤ C r,d n -r/d+1/2+3/d , λ = n -r/d+1/2 , (2.64)
where r is the regularity of f , C r,d is a constant which depends on the dimension d, on f , and on its regularity r, but not on n. The point of this result is to show that this algorithm indeed breaks the curse of dimensionality in the rates for global optimization, leveraging the regularity to obtain a convergence of order n -r/d (as opposed to typical slow rates of order n -1/d ). Note however that the constants may still depend exponentially in d. [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] also provide ways of computing the global minimizer.

The main assumption in order for Eq. (2.64) to hold is that Eq. ( 2.62) has a solution, i.e. that there exists a trace-class operator A such that f (x) -f * = k x , Ak x Hs for all x ∈ X .

A sufficient condition for this to hold is the fact that ff * is a finite sum of squares of functions f i ∈ C s (X ). In [START_REF] Rudi | Finding global minima via kernel approximations[END_REF], we assume that f has only strict minima (i.e. the Hessian at the minima is positive definite), and that f stays far away from its minimum at the boundary of X , i.e., that {f -f * ≤ δ} is compact in X for a certain δ > 0.

Marteau-Ferey, Bach, and Rudi (2022b) provide an assumption for a function f ∈ C s+2 to be decomposed as a sum of squares of functions of class C s which generalizes the one above, and allows to deal with functions which are defined on manifolds, and which have manifolds of zeros.

The condition we find is that the Hessian of f has to be positive definite orthogonally to the manifolds of zeros. This is an important case for situations where functions f have "large" sets of zeros, such as in optimal transport (see [START_REF] Vacher | A Dimension-free Computational Upper-bound for Smooth Optimal Transport Estimation[END_REF]). This is linked in spirit to the different Positivstellensätzen in polynomial optimization, to guarantee convergence of algorithms.

Part I

Fast rates and algorithms for generalized self-concordant losses In this chapter, we present work by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]; [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]. These two articles are then reported as verbatim, without modification from their original version, in chapter 4 and chapter 5 (these articles have been peer-reviewed and published). As explained in Sec. 2.3.1, our ideal goal is to study the classical problem of expected risk minimization Eq. (2.4) through the solving of the regularized empirical risk minimization problem Eq. (2.7), on a reproducing kernel Hilbert space H with kernel k (see Sec. 2.2). The goal of these two works is to extend the precise bias-variance trade-offs, fast rates and fast algorithms already known for least-squares to a broader class of loss functions including logistic regression and multiclass logistic regression (see Sec. 2.1.1).

We will slightly simplify and reformulate the different problems we consider from the original works from Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]; [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] in order to keep uniform notations between introductions, and to make the statements clearer. More formally, we will consider the setting where we are learning a function f : X → R (note that the results hold for R p valued functions and kernels in the reference articles), where X is a Borel space, through i.i.d. samples z 1 , ..., z n ∼ Z ∈ Z. The points z 1 , ..., z n define n evaluation points x i = π(z i ) which follow the law of X = π(Z) (we assume π to be known). We also assume the loss function is of the form z (t), z ∈ Z, t ∈ R ( z (f (x)) measures the cost of predicting f (x) at z). In that case, the expected risk minimization problem takes the form :

find f ρ such that R(f ρ ) = min f ∈M (X ,R) R(f ), R(f ) = E Z∼ρ [ Z (f (X))] , (3.1)
where M (X , R) is the set of measurable functions from X to R, and ρ is the probability distribution of Z. We highlight the relation of f ρ (if it exists) with the distribution ρ. As in the 90 CHAPTER 3. BACKROUND AND MAIN RESULTS introduction, ρ X will denote the density of X = π(Z) if Z has density ρ. As seen in Eq. (2.47) in Sec. 2.2.3, the regularized empirical risk minimizer can be defined as

f n,λ = arg min f ∈H 1 n n i=1 z i ( f, k x i H ) + λ 2 f 2 H , x i = π(z i ).
(2.47)

To start with, we will make the following three assumptions.

Assumption 3.1 (well-specified). For the data distribution ρ, f ρ exists and belongs to the space H.

This assumption implies in particular that the approximation error is zero. This is a strong assumption, but makes the whole statistical analysis much easier. It is commonly made in the reference works on statistics for least-squares, such as the work by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]. [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF] make a slightly different assumption, which is weaker in the case where the set H is not L 2 universal (it asks only that the projection of f ρ on the closure of H in L 2 (X , ρ X ) be in H), but which, in the case of the Gaussian or Sobolev kernels, corresponds exactly to Assumption 3.1. Moreover, this does not change the analysis much, and we believe that in general, analysis using Assumption 3.1 can easily be adapted to this projected setting (with a slightly different control of the noise term in Assumption 3.4).

Assumption 3.2 (convexity). The loss function is convex, i.e., z (•) is convex for all z ∈ Z.

This assumption is a weaker version than the one we will actually make later, the generalized self-concordance assumption. However, it already allows to guarantee that for any λ > 0, the empirical risk minimizer exists and is unique; f n,λ is therefore well-defined.

Assumption 3.3 (bounded kernel). The kernel k is bounded by 1, i.e., k(x, x) ≤ 1 for all x ∈ X .

This last assumption could easily be replaced by the condition that k(x, x) ≤ κ 2 for some positive constant κ, by considering the modified kernel k/(κ 2 ) which defines the same RKHS with norm 1 κ • H . For simplicity, we therefore assume that κ = 1. It is satisfied by the Gaussian and Sobolev kernels (see Sec. 2.2.2), which will be the ones we will use the most in this thesis.

In this chapter, we start by presenting the backround which leads to the statistical and optimization results of this part. In particular, in Sec. 3.1, we continue the work started in Secs. 2.2.4 and 2.2.5 and recall the main results for least-squares regression. These results concern biasvariance trade-offs, optimal rates of convergence in terms of statistics, dimension reduction (which is also statistical), and optimization. After presenting this vast literature on square-loss, Sec. 3.2 introduces the main tool which has allowed us to go from the square-loss setting to a broader class of functions : the notion of generalized self-concordance [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF]. In Sec. 3.3, we present the main results of this part, using the same notations as that of the introduction.

Note on slow rates

If the loss z (t) is convex and uniformly B-Lipschitz (i.e. z (•) is B-lipschitz for all z ∈ Z), which is the case for logistic regression or the support vector machines, there are very general non-asymptotic bounds for the excess risk by [START_REF] Sridharan | Fast rates for regularized objectives[END_REF], of the form :

R( f n,λ ) ≤ C B 2 log 2 δ λn + λ f ρ 2 H , (3.2)
3.1. FAST RATES FOR LEAST-SQUARES 91 with probability at least 1 -δ, where C is an explicit constant (it does not depend on any parameters). The bound above is a bias-variance decomposition. Indeed, B 2 /(λn) is a variance term which depends on the sample size n but not on the optimal predictor f ρ , and λ f ρ 2 H is a bias term, which depends on the optimal predictor but not on the sample size n. All our bounds will have this form but with smaller quantities (but asking for more assumptions). Without further assumptions, in Eq. (3.2), λ is taken proportional to 1/ √ n, and we get the usual optimal slow rate in excess risk of O(1/ √ n) associated with such a general set-up (see, e.g., [START_REF] Cesa-Bianchi | On the complexity of learning with kernels[END_REF].

Fast rates and algorithms for least-squares regression

In this section, we present a brief overview of the results which exists in the case of least-squares regression. In that case, Z = X × R and that is the case where (x,y) (t) = 1 2 |t -y| 2 , as in Sec. 2.2.4.

In Sec. 3.1.1, we present the statistical aspects of least-squares regression, the bias variance trade offs for regularized empirical risk minimization, classical classes of distributions and associated upper and lower rates. We will also try to give hints as to the proof techniques and different quantities to control, as a similar methodology will be applied to prove oour results.

These bias-variance trade-offs are important not only for the purely statistical part, but will diffuse throughout the section as they provide a precision target for the optimziation routine (which does not need to go beyond the statistical precision), and also, through a notion of effective dimension, the dimension we must aim for when reducing the dimension of the empirical risk minimization problem.

In Sec. 3.1.2, we will describe the two main ways of performing dimension reduction on the empirical risk minimization problem Eq. (2.47) developped for least-squares.

Finally, in Sec. 3.1.3, we will present work by [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF], which develops a fast algorithm to solve this dimension reduced-problem, and which will be a sub-routine in our main work.

Statistics and rates for least squares

Let us first introduce some key linear operators in the context of least-squares regression. Some of them have already been introduced in Sec. 2.2.4.

Covariance operator. Recall from the definition of the operator S : f ∈ H → f ∈ L 2 (X , ρ) where S is just the injection from H to L 2 (X , ρ X ), and the definition of the covariance operator

Σ = S * S = X k x ⊗ k x dρ X , Σf = X f (x)k x dρ X , f ∈ H, (2.54)
where for any u, v in a Hilbert space H, u⊗v denotes the operator defined by (u⊗v)w = v, w H u.

We can express the excess risk of a function f using only this operator in the case of the square loss (which is the quantity we wish to minimize) : (3.4) and the regularized empirical risk can be written as 1 2 f, ( Σ n + λI)f H -f, S * n y plus a constant term. The regularized empirical risk minimizer can therefore be expressed as

R(f ) -R(f ρ ) = 1 2 Σ 1/2 (f -f ρ ) 2 H . ( 3
Σ n = S * n S n = 1 n n i=1 k x i ⊗ k x i ,
f n,λ = ( Σ n + λI) -1 S * n y where y = (y i ) ∈ R n .
Note that one of the great advantages of the least-squares setting is that we have closed form of the solutions using these linear operators. One of the main tools in order to generalize those results will be to be able to use such operators near the optimal solution.

Finer bias-variance trade offs

One of the quantities which has to be controlled is the noise, i.e. the quantity Yf ρ (X). One way to do so is that the noise, conditionally on X, be uniformly sub-gaussian (where by uniformly, we mean that ε|{X = x} is sub gaussian with parameters which are uniform on x ∈ X ). This assumption can be found as Hypothesis 2. in the work by [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF], and as Assumption 2.9 in the work by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF].

Assumption 3.4 (noise distribution). We make the following assumption on the noise ε = Yf ρ (X) :

E [|ε| m |X] ≤ 1 2 m!σ 2 M m-2 ρ X a.s. , m ≥ 2. (3.5)
As explained in Sec. 2.2.4, the bias-variance trade off in the least-square setting is expressed as a function of two main "kernel" quantities.

Definition 3.1 (Definition of the bias and the effective dimension). We define the two following key quantities :

• the effective dimension, defined as

d λ = Tr((Σ + λI) -1 Σ); (3.6)
• the bias, defined as

b λ = λ 2 (Σ + λ) -1/2 f ρ 2 H = Σ 1/2 (f λ -f ρ ) 2 H , (3.7)
where f λ = (Σ + λ) -1 Σf ρ is the minimizer of the regularized expected risk.

The following non-asymptotic bias-variance result can be obtained with these quantities (see [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF], Proposition 5.8 or Caponnetto and De Vito (2007), Theorem 4).

Theorem 3.1 (bias-variance trade-off for least squares). There exists two explicit constants C 1 , C 2 such that for any δ ∈ (0, 1] and λ ∈

(0, 1], if n ≥ C 1 max(1,d λ ) λ
log 2 4 δ , then the following bound holds with probability at least 1 -δ :

Σ 1/2 ( f n,λ -f ρ ) H ≤ C 2 log(4/δ) b λ + M λ 1/2 n + σ 2 d λ n . (3.8)
This trade off can be obtained by using the following main ingredients. Recall that using the closed forms, it holds

Σ 1/2 ( f n,λ -f ρ ) H = Σ 1/2 (( Σ n + λI) -1 S * n y -f ρ ) H . (3.9)
Equivalence of the empirical and expected covariance. The first step is to guarantee that the regularized covariance and empirical covariance are equivalent, i.e., 1 2 Σ n + λI Σ + λI 2 Σ n + λI . This can be obtained with classical "dimensionless" concentration bounds [START_REF] Tropp | User-friendly tail bounds for sums of random matrices[END_REF] with probability 1-δ as soon as n ≥ C 1 max(1,d λ ) λ log 2 2 δ (hence this condition in the theorem). This equivalence allows to change from (Σ+λI) factors to ( Σ n +λI) factors by only multiplying by constants : in particular (Σ + λI

) -1/2 ( Σ n + λI) 1/2 ≤ √ 2 and (Σ + λI) 1/2 ( Σ n + λI) -1/2 ≤ √ 2.
With this, we can decompose the error Eq. (3.9) into two main terms (here C denotes an explicit constant which does not depend on any problem parameters) :

Σ 1/2 ( f n,λ -f ρ ) H ≤ C b λ + (Σ + λI) -1/2 ( S * n y -Σf ρ ) (3.10)
Concentration of the second term. The second main step is to concentrate the second term. For example, this is done by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]. The way of doing this is simply to note that the second term can be written as

1 n n i=1 ζ i -E [ζ] H , ζ = (Σ + λI) -1 Y k X . (3.11)
One can then use classical concentration bounds on sub-gaussian variables, by showing that ζ is sub-gaussian with parameters σ 2 d λ and M/ √ λ using Eq. (3.5) (see [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF], proposition 5.2 for more details). This shows that with probability 1 -δ, the right hand side of Eq. (3.10) is bounded by a term of the form C log

2 δ M λ 1/2 n + σ 2 d λ n
. Performing a union bound on the probabilistic results yields Theorem 3.1.

In chapter 4, we extend Theorem 3.1 using slightly different assumptions in the context of generalized self-concordant losses (in particular, for simplicity, we assume bounded noise as the noise is not as easily defined). These results are reported in Sec. 3.3.

Minimax upper bounds for least-squares. Recall from Sec. 2.1.2 that in order to define minimax rates, we have to define a class of test measures M for which we get the minimax rate for all measures ρ ∈ M. To get upper minimax rates, we define the class M < (R, r, β, b, σ, M ) to be the set of Borel probability measures ρ on X × R such that Eq. (3.5) holds with σ and M , and the following assumptions hold for R, r, β, b. Assumption 3.5 (Source condition). f ρ satisfies a source condition of the type seen in Sec. 2.2.4 (see definitions before Eq. (2.57)) with r ≥ 1/2, i.e., there exists h ∈ H such that f ρ = Σ r-1/2 h and h H ≤ R. This corresponds to the hypothesis of f ρ being in H 2r = Im(T r k ) mentioned in Sec. 2.2.4. Assumption 3.6 (Capacity condition). The eigenvalues λ i of Σ satisfy λ i ≤ β i b , that is they decrease at least polynomially with order b. This shows that the effective dimension will explode as

d λ ≤ Q 2 b,β λ -1/b
, where Q b,β depends on b, β. In the literature (Rudi and Rosasco, 2017;[START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF][START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], the capacity condition is sometimes directly formulated as

d λ ≤ Q 2 λ -γ , expressed in terms of Q, γ instead of β, b. Let Θ = {(R, σ, M ) ∈ R 3
++ } where R ++ is the set of positive real numbers. For fixed values of r, β, b, we denote with M < θ the class M < (R, r, β, b, σ, M ). Indeed, the minimax upper rates which can be obtained here are uniform over θ when the other parameters are fixed

We can now informally formulate the minimax rates obtained in this setting (it will be made formal in Theorem 3.2) : if we set

λ n,θ = min   σ 2 R 2 n b 2br+1 , 1   , a n,θ = R 2 σ 2 R 2 n 2br 2br+1 , (3.12) then it holds : R( f n,λ n,θ ) -R(f ρ ) ≤ a n,θ in the minimax sense on M θ . (3.13)
To get an intuition of why that is the case, note that if we substitute the value of λ by that of λ n,θ given in Eq. (3.12), we obtain a bound of the following form.

Corollary 3.1 (Blanchard and Mücke (2018), Cor. 5.9).

Let M > 0, β > 0, R > 0, r ∈ [1/2, 1],
b > 1 and σ > 0. There exists constants C β,b,r , C β,b,r,σ,R and n β,b,r,σ,R,M depending on the subscripted parameters such that for any n ≥ n β,b,r,σ,R,M , any δ ∈ (0, 1], it holds

R( f n,λ n,θ ) -R(f ρ ) a n,θ ≤ C β,b,r log 2 4 δ (3.14) provided log δ -1 ≤ C β,b,r,σ,R n b(r-1/2) 2br+1
In the works by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]; Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] which can be found in chapters 4 and 5, results of the type Cor. 3.1 are presented as fast rates. While using the same methodology as that of [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF]; [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF], we could get minimax upper bounds, this is not formally done.

Let us now state such upper bounds and sketch very roughly the technique to go from Cor. 3.1 to minimax rates. We cite here a particular case of the result by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF], Theorem 3.4.

Theorem 3.2 (Minimax upper rates). Let r ∈ [1/2, 1], β > 0, b > 1 be fixed, and assume λ n,θ and a n,θ are defined by Eq. (3.12) . For all ψ(•) = | • | p for p > 0, the following minimax rates of convergence hold with rate a n,θ :

sup θ∈Θ lim sup n→+∞ sup ρ∈M < θ E ρ ⊗n ψ R( f n,λ n,θ ) -R(f ρ ) r n,θ < ∞ (3.15)
This shows that a n,θ is a minimax upper rate of convergence for all L p norms and uniformly in θ.

FAST RATES FOR LEAST-SQUARES

95

The rates obtained by [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF] are a bit different, and quantify the probability that

R( f n,λ n,θ )-R(fρ) a n,θ
> τ for all τ .

High level ideas to obtain minimax rates from Cor. 3.1. Reformulate Cor. 3.1 as the following : there exists constants C, C β,b,r , C β,b,r,σ,R such that for n ≥ n β,b,r,σ,R,M and all τ ∈ [0, τ 0 (n)],

P R( f n,λ n,θ ) -R(f ρ ) a n,θ > τ ≤ C exp(-C β,b,r √ τ ), where τ 0 (n) = C β,b,r,σ,R n 2br-b 2br+1 . (3.16)
This almost gives the desired bound : indeed, without the condition that τ ≤ τ 0 (n), we would simply integrate the bound using the fact that for any differentiable ψ on R + and Borel measurable random variable T , it holds E [ψ(T )] = +∞ τ =0 ψ (τ )P(T > τ ) dτ . To handle the condition that τ ≤ τ 0 (n), [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] combine this bound which holds for all small enough τ with another bound which is rougher but which works for all τ > 0. For more details, we refer directly to the original work.

Saturation effect and spectral regularization. Note that in Cor. 3.1 and Theorem 3.2, we have imposed the condition r ≤ 1. This is due to the fact that there is a saturation effect with the Tikhonov regularization. Indeed, when using Tikhonov regularization, we cannot obtain better bias decrease than b λ ≤ Cλ max(r,1) , and hence we do not leverage regularity beyond r = 1. It is possible to do so, using techniques from spectral filtering [START_REF] Gerfo | Spectral algorithms for supervised learning[END_REF]; [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]. Spectral filtering consists in regularizing with a spectral regularization function g : (t, λ) ∈ [0, 1] × R → R, which defines a regularization f n,λ = g( Σ n , λ) S * n y where g(•, λ) is applied spectrally. Tikhonov regularization corresponds to g(t, λ) = 1 t+λ , but other regularizations exists (such as iterated Tikhonov, King and [START_REF] King | Approximation of generalized inverses by iterated regularization[END_REF]) which have a better qualification (for a precise definition see [START_REF] Gerfo | Spectral algorithms for supervised learning[END_REF]; [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]; [START_REF] Beugnot | Beyond tikhonov: faster learning with self-concordant losses, via iterative regularization[END_REF]). Under the assumption that the spectral filter g has better qualification than r, one can go beyond the previous setting and bound the bias term by Cλ r , and obtain the same minimax upper bounds as the ones stated above.

Note that the work by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] goes beyond the setting presented above, taking into account many spectral regularizations, as well as obtaining minimax rates for other metrics that the excess risk, and taking into account the inverse problem setting.

Minimax lower bounds. The results obtained in this thesis do not concern minimax lower bounds : indeed, the methodology for obtaining such bounds is very different from the one to obtain upper bounds, and does not involve constructing an estimator (like the regularized e.r.m.). In this paragraph, we give a few elements in order to understand the definitions and high level strategies in order to obtain lower bounds for least squares. Throughout this paragraph, we refer to the works by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]; [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF] who obtain minimax lower bounds which essentially match those of Theorem 3.2, and hence shows that the rate a θ,n defined above is optimal.

Classes. Define the class M(R, r, ν, σ, M ) (or M(ν) for short) to be the set of Borel probability measures ρ on X × R such that the following conditions hold.

CHAPTER 3. BACKROUND AND MAIN RESULTS

(i) Marginal ρ X = ν.

(ii) Source condition f ρ satisfies a source condition of the type seen in Sec. 2.2.4 (see definitions before Eq. (2.57)) with r ≥ 1/2, i.e., there exists h ∈ H such that f ρ = Σ r-1/2 h and h H ≤ R.

(iii) Assumption on the noise. Eq. (3.5) holds with σ and M .

Note that M < (R, r, b, β, σ, M ) defined for the minimax upper bounds is just the union of all M(R, r, ν, σ, M ) over the ν ∈ P < (b, β), i.e., such that the capacity condition holds : the eigenvalues λ i of Σ (whis is defined by ν) satisfy λ i ≤ β i b . To obtain minimax lower bounds, [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] instead consider the set P > (b, α) of probability distributions on X such that the capacity is lower bounded : the eigenvalues λ i of Σ satisfy λ i ≥ α i b . Moreover, they add a technical condition in order to obtain strong minimax lower rates (see the definition of P > strong theirein). This technical assumption is satisfied as soon as ν ∈ P < (b, β) for some β > α. The class studied for minimax lower bounds is the union M > (R, r, b, α, σ, M ) = ν∈P > (b,α) M(R, r, ν, σ, M ). In the same way as in the upper bound case, we consider the parameter θ ∈ Θ = {(R, σ, M ) ∈ R 3 ++ : σ ≤ M } as being free, i.e. M θ (ν) resp. M > θ denote the previous models for fixed ν, r resp. r, α, b. As explained in Sec. 2.1.2, in order to obtain minimax lower bounds, we must lower bound the following quantity asymptotically :

MinMax(ψ, H, n, θ, M > θ , a n,θ ) := inf fn sup ρ∈M > θ E ρ ⊗n ψ R( f n ) -R(f ρ ) a n,θ , (3.17)
where the f n are minimized on the set of estimators f n : Z n → H. [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF], prove the following theorem on minimax lower rates for the L p norm.

Theorem 3.3 [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]). Let r ≥ 0, α > 0, b > 1 be fixed, and assume a n,θ is defined by Eq. (3.12) . For all ψ(•) = | • | p for p > 0, the following minimax rates of convergence hold with rate a n,θ :

inf θ∈Θ lim inf n→+∞ MinMax(ψ, H, n, θ, M > θ , a n,θ ) > 0 (3.18)
This shows that a n,θ is a minimax lower rate of convergence for all L p norms and uniformly in θ.

We now very briefly describe the proof techniques in order to get such rates. Fix α, b, r.

Reduction to a single ν The first step is to note that since

M > (R, r, b, α, σ, M ) = ν∈P > (b,α) M(R, r, ν, σ, M ), (3.19)
one can decompose the minimax along the ν ∈ P > (b, α) to get a lower bound :

MinMax(ψ, H, n, θ, M > θ , a n,θ ) ≥ sup ν∈P > (b,α) MinMax(ψ, H, n, θ, M θ (ν), a n,θ ). (3.20)
Reduction to a bound in probability over a finite set. Note that for a fixed ν ∈ P > (b, α), the risk R( f n ) -R(f ρ ) can be expressed as a squared distance on H defined by ν :

R( f n ) -R(f ρ ) = d ν ( f n , f ρ ) 2 d ν (f, f ) := Σ 1/2 ν (f -f ) H , ( 3
P ρ ⊗n R( f n ) -R(f ρ ) > ε , ≥ ψ(ε 2 /a n,θ ) inf fn sup ρ∈M θ (ν) P ρ ⊗n d ν ( f n , f ρ ) > ε .
(3.22)

Applying the Tskybakov method. We now concentrate on bounding the term inf fn sup ρ∈M θ (ν)

P ρ ⊗n d ν ( f n , f ρ ) > ε .
This term can be bounded using a generic method presented by [START_REF] Alexandre | Introduction to Nonparametric Estimation[END_REF]

P ρ ⊗n d ν ( f n , f ρ ) > ε ≥ inf fn sup 1≤i≤N P ρ ⊗n i d ν ( f n , f ρ i ) > ε (3.23)
for well-chosen N and ρ 0 , ..., ρ N (which have to be adapted to θ, ε and n in this setting). These measures ρ 0 , ..., ρ N have to be designed as being 2ε-separated for the distance d ν , and have "small" KL-divergence KL(ρ|ρ ) = E ρ log dρ dρ between them. This basically says that the ρ i will be statistically indistinguishable while being far from each other in d ν distance, and hence the best estimator will not be able to choose between them with precision better than ε. More formally, we can set the following requirements, which are adapted from Thm. 2. 5 by Tsybakov (2008) wihch is itself based on Fano's lemma, in the present setting.

Proposition 3.1 (Tsybakov method). Fix θ ∈ Θ. Let ε > 0. Assume that there exists N ε > 2, ω ε ≥ 0 and ρ 0 , ..., ρ Nε ∈ M θ (ν) such that

(i) For any 0 ≤ i < j ≤ N ε , d ν (f ρ i f ρ j ) ≥ 2ε ;
(ii) For any j = 1, ..., N ε , ρ j is absolutely continuous with respect to ρ 0 , and

1 N ε N i=1 KL(ρ j |ρ 0 ) ≤ ω ε log(N ε ). (3.24)
Then for all n such that 0

≤ nω ε ≤ 1/8, it holds inf fn sup ρ∈M θ (ν) P ρ ⊗n d ν ( f n , f ρ ) > ε ≥ inf fn sup 0≤i≤Nε P ρ ⊗n i d ν ( f n , f ρ i ) > ε ≥ √ N ε 1 + √ N ε 1 -2nω ε - 2nω ε log(N ε ) ≥ 1 4 3 2 - 1 log(2) :=c > 0.
(3.25)

CHAPTER 3. BACKROUND AND MAIN RESULTS

Constructing the ρ i to apply Tsybakov's method. The crux of the problem is therefore to design the ρ 0 , ..., ρ Nε in order to apply the above proposition. Both Caponnetto and De Vito (2007); [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] proceed in essentially the same manner. They design the ρ i in the form ρ i (dx, dy) = ρ f i (dy|x)ν(dx) where ρ f i (dy|x) = N (f i (x), σ 2 ), that is ρ i is gaussian gaussian conditionally on x. It is clear the f ρ i = f i , and a bound on the KL-divergence can easily be obtained in therms of the distance d ν (f i , f j ) and σ. Based on a technical lemma from [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF], Proposition 6.4 by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] give a constant C α,b,r and ε 0 depending on α, b, r, σ, M, R such that for all ε < ε 0 , one can construct ρ i , 1 ≤ i ≤ N ε satisfying conditions (i) and (ii) with ω ε = C α,b,r σ -2 R 2 (εR -1 ) 2+1/br .

Bringing everything together. The final step is setting

ε n = D α,b,r R σ 2 R 2 n br 2br+1
, where D α,b,r is set to satisfy nω ε ≤ 1/8, and such that ε n ≤ ε 0 , as soon as n ≥ n 0 where n 0 depends on σ, R, M, r, β, b. We can then go back up the chain of inequalities : combining Eq. (3.25) and Eq. (3.22) as well as Eq. (3.12) for the definition of a n,θ , it holds that for all n ≥ n 0 ,

MinMax(ψ, H, n, θ, M θ (ν), a n,θ ) ≥ ψ(D 2 α,b,r )c.
Theorem 3.3 then immediately follows from Eq. (3.20) and going to the limit.

Note that the work by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] goes beyond the setting presented above, dealing with other distances d ν as well as problems coming from inverse problems.

Brief recap. In Sec. 3.1.1, we give the main statistical trade-offs, as well as upper and lower minimax rates for least squares, as well as the high level methodology to prove these results. We will use similar a very similar methodology in Sec. 3.3 to prove our bias variance trade offs and upper rates (not exactly minimax but which could be made into minimax upper rates). These upper rates, in the least squares setting, are actually optimal for a well-chosen class of functions, as corresponding lower bounds can be proven in the well specified setting (that is when f ρ ∈ H).

Dimension reduction with the effective dimension

In Sec. 3.1.1, we have presented the classical statistical rates, and bias variance trade offs for the regularized empirical risk minimizer f λ . Recall that a priori, for λ > 0, f λ exists and is the unique solution to

f n,λ = arg min f ∈H 1 n n i=1 |f (x i ) -y i | 2 + λ f 2 H .
(2.47)

The representer theorem (Theorem 2.2) allows to guarantee that f n,λ automatically belongs to the set H n = span({k x i : 1 ≤ i ≤ n}) which is n dimensional and can be parametrized by

α ∈ R n -→ n i=1 α i k x i ∈ H n ,
f n,λ,m = arg min f ∈Hm 1 n n i=1 |f (x i ) -y i | 2 + λ f 2 H , (3.26)
where we index the result with the dimension m, we want to guarantee that R(

f n,λ,m ) -R(f ρ ) ≤ C(R( f n,λ ) -R(f ρ ))
, ideally with m n. Such results are usually proved in two forms. The first, form is to ask that as soon as m ≥ m λ , the bias-variance decomposition of Theorem 3.1 holds not only for f n,λ but also for f n,λ,m . We therefore have to design m λ in order for the following decomposition to hold.

Theorem 3.4 (Prototypical dimension reduced bias-variance trade off). There exists two explicit constants C 1 , C 2 such that for any δ ∈ (0, 1] and λ ∈

(0, 1], if n ≥ C 1 max(1,d λ ) λ log 2 4
δ and m ≥ m λ then the following bound holds with probability at least 1 -δ : There exists a constant C such that for all θ ∈ Θ, there exist n 0 , n 1 depending on θ such that for any n ≥ n 0 , any δ ∈ (0, 1], and any m ≥ m θ,n , it holds

Σ 1/2 ( f n,λ,m -f ρ ) H ≤ C 2 log(4/δ) b λ + M λ 1/2 n + σ 2 d λ n . ( 3
R( f n,λ n,θ ,m n,θ ) -R(f ρ ) ≤ C log 2 4 δ R 2 σ 2 R 2 n 2br 2br+1 , ρ ∈ M < θ , (3.28) provided log δ -1 ≤ (n/n 1 ) b(r-1/2)
2br+1 and where λ n,θ is given by Eq. (3.12).

This second form of result is usually very useful to get an idea on how hard it is to approximate H with H m depending on the number of samples n and the hardness of the problem (parametrized by the class M < θ . Typically, the ideal dimesnion would be m

n,θ = d λ n,θ = C β,b,r σ 2 R 2 n -1/(2br+1)
, which is, in a sense, the true dimension of the space H regularized with λ n,θ . However, this is not always possible, depending on the way we construct the space H m .

We will detail two ways of constructing spaces H m in the literature : random features and Nyström projections.

CHAPTER 3. BACKROUND AND MAIN RESULTS

Random features

As explained in Sec. 2.2.5 random features are based on approximating the RKHS H defined by a kernel whose expression is of the form k(x, x ) = E ω∼π [φ(x, ω)φ(x , ω)], that as an expectation of random features (see Eq. (2.43) and the following discussion on the Gaussian kernel). For the sake of simplicity, we will assume in this section that the features are bounded by 1; in particular, this implies that the kernel is bounded by 1.

The idea of using random features to approximate the RKHS H is to approximate it with the set

H m :=    x → 1 √ m m j=1 α j φ(x, ω j ) : α ∈ R m    , (3.29)
where the ω j are m i.i.d. samples from π. H m can be equipped with the RKHS structure given by 1

√ ,m m j=1 α j φ(•, ω j ), 1 √ m m j=1 β j φ(•, ω j ) Hm = m i=1 α j β j , associated to the kernel k m (x, x ) = 1 m m j=1 φ(x, ω j )φ(x , ω j ).
This rather abstract formulation can also simply be seen as replacing the kernel matrix K in Eq. (2.50) with a low rank approximation K ≈ Φ Φ where Φ = 1 √ m (φ(x i , ω j )) i,j ∈ R n×m . Indeed, using the parametrization in Eq. (3.29), Eq. (3.26) becomes min

α∈R m 1 n Φ α -y 2 R n + λ α 2 , (3.30)
whose solution is (ΦΦ + nλI) -1 Φy.

In Rudi and Rosasco (2017), theorems of the form Theorems 3.4 and 3.5 are proved and give lower bounds on m in order to obtain the same performance as that of the classical empirical risk minimizer. For example, Theorem 2. from Rudi and Rosasco (2017) can be adapted to show the following.

Theorem 3.6 (Rates for random features). Fix r ∈ [1/2, 1], β > 0, b > 1. There exists C and m 0 such that for all θ ∈ Θ, there exists n 0 , n 1 depending on θ such that for any n ≥ n 0 , for any

δ ∈ (0, 1] if m ≥ m θ,n log 108n δ where m θ,n = m 0 σ 2 n R 2 b+2r-1 2br+1 (3.31) and log δ -1 ≤ (n/n 1 ) b(r-1/2)
2br+1 , it holds with probability at least 1 -δ :

R( f n,λ θ,n ,m ) -R(f ρ ) ≤ CR 2 σ 2 n R 2 2br 2br+1
.

(3.32)

Note that in the situation where b ≈ 1 and r ≈ 1/2 (that is the loosest assumptions), typically m ≈ √ n features are needed, reducing the dimension by a significant exponent. Note that this result is proved from a more complex bias-variance decomposition such as that of Theorem 3.4, in Theorem 5 by Rudi and Rosasco (2017).

FAST RATES FOR LEAST-SQUARES

101

Note that this result does not match the "ideal" dimension m w would get as m n,θ = d λ n,θ . This will be explained in part in the following discussion, which details the high level techniques and principles to obtain Theorem 3.6.

Sketch of the techniques involved. In order to compare the space H m with the space H, one compares the associated kernel operators on L 2 (X ρ X ), defined for a generic kernel k by Eq. (2.56) as (T k g)(x) = X k(x, x )g(x )ρ(dx ). Let T m be a short-hand for T km and T for T k .

There are main two terms depending on m and which have to be controlled in order to get the same precision with H m as with H. They are linked to two guarantees which must be satisfied.

(i) The fact that the metric of H m and H are essentially the same, at fixed λ. This can be formalized as the operators T + λI and T m + λI being equivalent.

(ii) The fact that we are approximating a projected problem leads to a satisfactory solution, i.e. controlling the term (T m + λI) -1 T m f ρ -(T + λI) -1 T f ρ , which is the difference in performance between the regularized expected minimizer on H m and H.

These terms can be controlled using the key quantity

F ∞ (λ) = sup ω∈supp(π) (T + λI) -1/2 φ(•, ω) 2 L 2 (X ,ρ X ) . (3.33)
Using concentration bounds, it is possible to show that (i) holds as soon as m ≥ F ∞ (λ) up to logarithmic terms. Contronlling (ii) is a bit more involved, and the lower bound for m to guarantee that a bias-variance decomposition of the form Theorem 3.4 holds depends on F ∞ (λ) but also on the effective dimension d λ as well as the bias term.

Reducing the dimension by adapting the sampling to the problem. Note that s λ (ω) = (T + λI) -1/2 φ(•, ω) 2 L 2 (X ,ρ X ) measures the score of ω, that is the contribution of ω to the approximation of T by T m . F ∞ is therefore the highest possible score, and imposing m ≥ F ∞ (λ) essentially says that to lower bound m, we pay the price of having to see the ω with the highest score. To obtain a lower bound on m which is less conservative, it is natural to sample not from π, but from a density which favors ω with high scores, and therefore is adapted to the problem.

Ideally, we would like to use reweighted features and sample ω from the distribution

π λ (dω) = s λ (ω)π(dω) d λ , (3.34)
which is a probability density (d λ is the renormalization term). If we use this measure to sample the ω, it is possible to show that F ∞ = d λ , and hence that we can better adapt to the effective dimension, i.e., we would obtain m θ,n = m 0 σ 2 n/R 2 2br-b+1 2br+1 , which is lower than the one obtained before as b > 1, but still not as good as the ideal d λ θ,n .

Of course, we do not have access to this ideal measure π λ , as it is defined using problem dependent quantities. However, Rudi and Rosasco (2017) believe it is possible to approximate this measure well enough in order to obtain the same rates, even though no formal result has been published on the subject yet.

CHAPTER 3. BACKROUND AND MAIN RESULTS

Nyström sampling

In the case of Nyström sampling, the set H is approximated with a set of the form

H m =    x → m j=1 α j k(x, x j ) : α ∈ R m    , (3.35)
where x 1 , ..., x m ∈ X . Usually, the Nyström points are sampled from from the x 1 , ..., x n , as we will see below. This set can simply be seen as a m dimensional subspace of H with the induced norm :

m j=1 α j k(•, x j ), m j=1 β j k(x, x j ) Hm = α K mm β
where K mm is the kernel matrix associated to the x j . The kernel associated to the space H can be k

m (x, x ) = k m (x) K † mm k m (x)
where k m (x) = (k( x j , x)) 1≤i≤m ∈ R m and † denotes the Moore-Penrose pseudoinverse. Using the parametrization of H m in Eq. (3.35), the problem Eq. (3.26) becomes:

min α∈R n 1 n K n,m α -y + λα K mm α, K n,m = (k(x i , x j )) 1≤i≤n 1≤j≤m
.

(3.36)

Eq. (3.36) shows quite clearly why Nyström is also sometimes called column subsampling. Indeed, if the x 1 , ..., x m are sampled from the x 1 , ..., x n , the matrix K n,m consists in subsampled columns of the full matrix K, and therefore Eq. (3.36) is a subsampled version of Eq. (2.50). Note in particular that if we take the Nyström points to be x 1 , .., x n , we recover the standard empirical risk minimization problem. The following reparametrization is therefore also valid in that setting.

A much better way parametrizing is to write K mm = T T using a cholesky decomposition, where T is upper triangular matrix. One then notess that defining φ

(x) = T -k m (x) ∈ R m , φ is a feature map which defines the kernel k m : k m (x, x ) = φ(x) φ(x ). Hence, H m = {α φ(•) : α ∈ R m }.
If Φ is the matrix whose columns are the φ(x i ), that is Φ = K n,m T -1 , the problem Eq. (3.26) can be formulated as min

α∈R m 1 n Φ α -y 2 R n + λ α 2 . (3.37)
This is exactly the form of Eq. (3.30). Of course, this comes at the price of computing and saving a cholesky factor T, which is O(m 3 ) in time and O(m 2 ) in memory.

In [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF], theorems of the form Theorems 3.4 and 3.5 are proved and give lower bounds on m in order to obtain the same performance as that of the classical empirical risk minimizer. For example, Theorem 1. from [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF] can be adapted to show the following in the case where the points x 1 , ..., x m are sampled uniformly from ρ X (to obtain such points, we sample m points uniformly without replacement from x 1 , ..., x n ).

Theorem 3.7 (Rates for uniform Nyström).

Fix r ∈ [1/2, 1], β > 0, b > 1.
There exists C and m 0 such that for all θ ∈ Θ, there exists n 0 , n 1 depending on θ such that for any n ≥ n 0 , for any

δ ∈ (0, 1] if m ≥ m θ,n log 108n δ where m θ,n = m 0 σ 2 n R 2 b 2br+1 (3.38) 3.1. FAST RATES FOR LEAST-SQUARES 103 and log δ -1 ≤ (n/n 1 ) b(r-1/2)
2br+1 , if the m Nyström points are sampled uniformly from ρ X (dx), it holds with probability at least 1 -δ :

R( f n,λ θ,n ,m ) -R(f ρ ) ≤ CR 2 σ 2 n R 2 2br 2br+1
.

(3.39)

Note that in the situation where b ≈ 1 and r ≈ 1/2 (that is the loosest assumptions), typically m ≈ √ n features are needed, reducing the dimension by a significant exponent.

Note that as in the random features case, this result does not match the "ideal" dimension m w would get as m n,θ = d λ n,θ . In the case of Nyström points, this can be corrected. This will be explained in part in the following discussion.

High level techniques. The analysis is somewhat simpler in the setting of Nyström points, as the H m are sub-spaces of H can can be characterized by the associated orthogonal projection P m from H onto H m . Therefore, there is no need to go to L 2 (X , ρ X ), and operators are therefore only compared as operators on H. Let Σ m = P m ΣP m be the covariance operator of H m (but defined on the whole of H for convenience through the projection).

There are main two terms depending on m and which have to be controlled in order to get the same precision with H m as with H. They are linked to two guarantees which must be satisfied.

(i) The fact that the metric of H m and H are essentially the same, at fixed λ. This can be formalized as the operators Σ m + λI and Σ + λI being equivalent.

(ii) The fact that we are approximating a projected problem leads to a satisfactory solution, i.e. controlling the term Σ

1/2 (Σ m + λI) -1 Σ m f ρ -(Σ + λI) -1
Σf ρ , which is the difference in performance between the retularized expected minmimaxer on H m and H.

Note that both these terms can actually be controlled using only the projection, through the quantity Σ 1/2 (I -P m ) , which quantifies the information Σ contained that we omit when projecting on H m .

In the same way as for random features, these terms can be controlled using the key quantity

F ∞ (λ) = sup x∈X dρ(x) dπ(x) (Σ + λI) -1 k x 2 H , (3.40)
where π denotes the measure from which we sample the points x. Using concentration bounds, it is possible to show that (i) and (ii) are well behaved as soon as m ≥ F ∞ (λ) up to logarithmic terms, and that in that case, we recover a bound of the form Theorem 3.4.

Reducing the dimension by adapting the sampling to the problem. In this setting, we can also define the score or leverage score s λ of a sample :

s λ (x) = (Σ + λI) -1/2 k x 2 .
It the contribution of x to the approximation of Σ + λI by Σ m + λI. F ∞ is therefore the highest possible score, and imposing m ≥ F ∞ (λ) essentially says that to lower bound m, we pay the price of having to see the x with the highest score. To obtain a lower bound on m which is less conservative, it is natural to sample not from ρ X , but from a density which favors x with high scores, and therefore is adapted to the problem.

CHAPTER 3. BACKROUND AND MAIN RESULTS

Ideally, we would like to sample x from the distribution

π λ (dx) = s λ (x)ρ X (dx) d λ , (3.41)
which is a probability density (d λ is the renormalization term). If we use this measure to sample the x, it is possible to show that F ∞ = d λ , and hence that we can better adapt to the effective dimension, i.e., we would obtain m θ,n = m 0

σ 2 n R 2 1 2br+1
, which is lower than the one obtained before and matches the ideal d λ θ,n .

Of course, we do not have access to this ideal measure π λ , as it is defined using problem dependent quantities. However, [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]; [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF] show that it is possible to approximate samples from that ideal distribution by computing approximate leverage scores. The strategy is to sub-sample indices i 1 , ..., i m from {1, ..., n} according to a probability vector p i in order to obtain Nyström centers x j = x i j . The probability p i of sampling index i is set to be proportional to the leverage score :

l i (t) = e i (K + tnI) -1 Ke i = ( Σ n + tI) -1/2 k x i 2 , p i = l i (t) d n,t , d n,t = n i=1 l i (t). (3.42)
and x is therfore sampled from the distribution

π n,t (dx) = s n,t (x) d n,t ρ(dx), s n,t (x) = ( Σ n + tI) -1/2 k x 2 , ρ = n i=1 δ x i . (3.43)
It is therefore possible to show that under assumptions on n and λ, sampling points from this distributionfor t ≈ λ actually yields the same performance as sampling points from the ideal distribution in Eq. (3.41). Of course, as the point of reducing dimension is not to compute these leverage scores exactly (this would cost O(n 3 )), [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF] develop a technique to compute good approximation of these scores in just min(nd 2 λ , d 2 λ /λ). A rapid introduction to the use of leverage scores can be found in Sec. 5.D .3. Summary. To summarize using either random feature techniques or Nyström points, it is possible to reduce the dimension to m n while keeping the same statistical properties. In the case of Nyström sub-sampling, we can even reduce that dimension to d λ , the "true" dimension of the space H regularized with λ, and this with a good computational cost. One last step remains to be done in order to actually compute a good estimator : the actual solving of problems Eqs. (3.30) and (3.37).

Fast algorithms

As we have seen in Eqs. (3.30) and (3.37), the dimension-reduced empiricical risk minimization problem are equivalent to the solving of a problem of the form :

α n,λ,m = min α∈R m 1 n Φ α -y 2 + λ α 2 R m , f n,λ,n (x) = α n,λ,m φ(x), (3.44)
where Φ ∈ R m×n and whose solution is (ΦΦ + nλI) -1 Φy. The definition of Φ and φ from the have been given in Sec. 3.1.2 for the Nyström approach and the random features approach. Recall that the Nyström approach includes the standard e.r.m., where we just take the Nyström points to be x 1 , ..., x n . Note that Φ = 1/ √ m(φ(x i , ω j )) in the random features case, and Φ = T -K n,m in the Nystrom case, where T is an upper triangular matrix.

Computing α n,λ,m by directly solving the system (ΦΦ + nλI)α = Φy and inverting the matrix (ΦΦ + nλI) is prohibitive. Suprisingly, this is not because of the matrix inversion, whose cost is of order O(m 3 ), but because computing the matrix (ΦΦ + nλI) costs O(nm 2 ) because of the matrix product (and the inversion of the triangular matrix in the Nystrom case).

However, computing matrix vector products of the form Φa for a ∈ R n and Φ α for α ∈ R m has a cost of O(nm + m 2 ) = O(nm) since m ≤ n. This suggest the use of an iterative method, such as gradient descent or conjugate gradient descent (see proposition 2.1), the second one being specifically adapted to the setting of solving linear systems, and which consist in solving iteratively the associated optimization problem

α n,λ,m = arg min α∈R m F (α) = 1 2 α (ΦΦ + nλI)α -α B -1 Φy, (3.45)
However, as we have seen in proposition 2.1, the number of gradient steps needed depends on the conditioning of the system, which will be of order 1 λ in this case. This can be solved by applying a pre-condtitionning method, which we describe below.

Preconditioning. The idea of preconditioning is to find a matrix B called preconditioner such that κ -1/2 BB (ΦΦ + nλI) κ 1/2 BB for some fixed κ ≥ 1, to define α = B β and to solve the problem in β, which becomes well conditioned. The condition to do this is that B be fast to compute, and that solving linear systems with B be easy to compute as well. We can then solve the system (3.46) which can also be expressed as the convex minimization problem

B -1 (ΦΦ + nλI)B -β n,λ,m = B -1 Φy,
β n,λ,m = arg min β∈R m 1 2 β B -1 (ΦΦ + nλI)B -β -β B -1 Φy, (3.47)
which is well conditioned, with conditioning κ (it is √ κ smooth and 1/ √ κ strongly-convex). One can therefore readily apply either the conjugate gradient or gradient descent algorithm (in order for the method to be iterative). The number of gradient or conjugate gradient steps needed is of order log 1 ε in order to achieve precision ε (as the condition number is now a fixed κ which disappears from the bound). The precision which must be achieved is of the order of the statistical error.

Pre-conditioned gradient descent seen as an approximate Newton step. An interesting remark is that pre-conditioned gradient descent is actually closely related to a second order method. Indeed, note that if we apply standard gradient descent, i.e.,

β t+1 = β t -1 √ κ B -1 (ΦΦ + nλI)B -β t -B -1 Φy , then the corresponding equation for the variable α is just α t+1 = α t -∆ t where ∆ t = H -1 ((ΦΦ + nλI)α t -Φy) and H = √ κBB . As ∆ t = H -1 ∇F (α t ) and ∇ 2 F (α t ) H κ∇ 2 F (α t )
, this is an approximate Newton step. In Sec. 3.3 and in particular in Sec. 3.3.3, we will use these methods in order to minimize a broader class of convex functions.

CHAPTER 3. BACKROUND AND MAIN RESULTS

Ways of finding pre-conditioners

In order to find a good pre-conditioner for (ΦΦ + nλI) exploiting the particular structure of that matrix, it is common to proceed in two steps.

(i) Find a matrix Φ ∈ R m×q with q ≤ m such that :

1/ √ κ(ΦΦ + nλI) Φ Φ + nλI √ κ(ΦΦ + nλI).
(3.48)

(ii) Compute B as a lower triangular cholesky factor : BB = Φ Φ + nλI. Finding this cholesky factor has cost O(min(m, q) 3 ) in time and O(m 2 ) in memory.

Such a matrix Φ can be obtained in two main ways, and the complexity of obtaining such a matrix depends on the effective dimesion

d λ (Φ) = Tr((ΦΦ + nλI) -1 ΦΦ ) ≤ min(m, n).
Note that if n and m satisfy the assumptions needed for the non asymptotic bias variance trade offs of the type Theorem 3.4, the effective dimension d λ (Φ) actually matches the effective dimension of the statistical problem d λ with high probability.

Column subsampling. This is exactly the same thing as Nyström sampling, i.e. we approximate

Φ = (φ 1 |...|φ n ) with Φ = (φ i 1 |...|φ iq ) diag(1/ √ p i k ) ∈ R m×q
where the selected columns (i k ) 1≤k≤q are q i.i.d. samples from {1, .., , n} with distribution given by the probability vector (p i ) 1≤i≤n .

In the literature, two main types of column subsampling exist, and satisfy the Eq. (3.48) with probability at least 1 -δ for δ ∈ (0, 1].

(a) Subsampling uniformly from the colums [START_REF] Roosta | Sub-sampled Newton methods[END_REF][START_REF] Rudi | Less is more: Nyström computational regularization[END_REF], where q = O(1/(λκ) log 1 λδ ). This leads to a total complexity of O(m 3 + 1/λ 2 ) in time to compute the precondioner, up to logarithmic terms and considering κ is fixed.

(b) Subsampling with approximate leverage scores [START_REF] Roosta | Sub-sampled Newton methods[END_REF][START_REF] Alaoui | Fast randomized kernel ridge regression with statistical guarantees[END_REF][START_REF] Rudi | Less is more: Nyström computational regularization[END_REF], where the the samples are sampled using a probability vector (p i ) which are an approximation of the probability vector proportional to the the true leverage scores (e i (Φ Φ + nλI) -1 Φ Φe i ) 1≤i≤n . Such an approximation can be computed in time O(min(n, 1/λ)d λ (Φ) 2 ) [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF]. In that case, taking q = O(κ -1 d λ (Φ) log 1 λδ ) is sufficient to guarantee Eq. (3.48).

Sketching. In this case, we approximate Φ by Φ = ΦΩ where Ω ∈ R n×q is a random matrix, and where the product ΦΩ can be computed in time O(nm log n) using fast Fourier or Hadamard transforms as is done by [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF]. In the work by [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF], it is shown that it is sufficient to take q = O(κ -1 d λ (Φ) log 1 λδ ) in order to guarantee Eq. (3.48) with probability at least 1 -δ.

To summarize, it is possible to find a good preconditioner in time O(nm + m 3 + md λ (Φ) 2 ) up to logarithmic terms if we use sketching, or O(m 3 + md λ (Φ) 2 + min(n, 1/λ)d λ (Φ) 2 ) up to logaritmic terms if we use column subsampling with approximate leverage scores.

An algorithm with optimal statistical guarantees and a low complexity

In the work by [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF], a complete methodology to obtain an estimator with optimal guarantees but with low computational complexity is presented. This 3.2. GENERALIZED SELF-CONCORDANCE 107 methodology uses the following ingredients.

• The dimension reduction phase is done with m Nyström points, which are subsampled from the (x i ) using a probability vector (p i ) 1≤i≤n computed from approximate leverage scores.

• Solving Eq. (3.37) is done with t steps of a conjugate gradient descent, and with a preconditioner which is computed using column subsampling using the same indices as that of the m Nyström points (and hence associated with the same probability vector (p i )).

Denote with f n,λ,m,t the function in H m obtained after t iterations of conjugate gradient descent using the preconditioner described above. This estimator has been dubbed the "FALKON" estimator by [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF]. The following non-asymptotic upper rates can be obtained for this estimator,which a) matches the rates obtained for the regularized empirical risk minimizer, and b) is computable in finite time with a complexity of order O(nd λ + d 3 λ ). Theorem 3.8 is a relatively informal rewriting of Theorem 5. by [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF] for the probability classes described in Sec. 3.1.1.

Theorem 3.8 (Informal rates for FALKON). Fix r ∈ [1/2, 1], β > 0, b > 1.
There exists C and m 0 such that for all θ ∈ Θ, there exists n 0 , n 1 , t 0 ∈ N depending on θ such that for any n ≥ n 0 , for any δ

∈ (0, 1] if m ≥ m θ,n log n δ where m θ,n = m 0 σ 2 n R 2 1 2br+1 , (3.49) if log δ -1 ≤ (n/n 1 ) b(r-1/2)
2br+1 , and if t ≥ t 0 log(n), it holds with probability at least 1 -δ :

R( f n,λ θ,n ,m,t ) -R(f ρ ) ≤ CR 2 σ 2 n R 2 2br 2br+1
.

(3.50)

Moreover, if m = m θ,n and t = t 0 log(n), the time complexity of computing f n,λ θ,n ,m,t is of order O(n 2br+2 2br+1
), where the O notations hides constants depending on the class parameters, and logarithmic terms in n and 1/δ. This corresponds to a complexity of order O(nd λ θ,n ) up to logarithmic terms.

Going beyond the quadratic case with (generalized) self-concordance

In order to extend the result of Sec. 3.1, we will need tools which allow to go from quadratic loss functions to more general convex losses. The core idea is to be able to locally approximate the losses by their second-order taylor expansion, that is if F is defined on a Hilbert pace H and is twice differentialble,

F (w) = F (w 0 ) + ∇F (w 0 ) (w -w 0 ) + 1 2 w -w 0 2 H(w 0 ) + o w-w 0 →0 ( w -w 0 2 ), (3.51)
where H(w) is the Hessian of F at w. However, in general, such an approximation is not strong enough. In order to generalize the results from least-squares, a precise control on the evolution of the Hessians is needed (since in the least-squares case, the Hessian stays constant). In particular,

CHAPTER 3. BACKROUND AND MAIN RESULTS

this control has to encompass the entire spectrum of the Hessian : one must find regions where the relative evolution of the Hessian is small. We have two main types of results to generalize from the least-squares setting : a) optimization results, which are independent from the condition number, and b) satistical results, to show that a bias-variance decomposition is still possible. The main difficulty here is that there is no closed-form solution for the estimators f n,λ ; they are only known as minimizers of a certain problem. Moreover, in order to have optimization algorithms which are independent from the condition number, one has to resort to second-order type methods.

Extending the optimization properties. To extend the optimization algorithms deployed in the least-square case, it is natural to think of a Newton method, as the full Newton methods formally minimizes the quadratic approximation of F at each step. Moreover, as it is invariant by reparametrization, it is independent of the conditioning of the problem.

Recall that a Newton method is of the form

w t+1 = w t -α t ∆(F, w t ) with ∆(F, w t ) = H(w t ) † ∇F (w t )
is the Newton step and α t is a step size and is set to 1 in the case of a full Newton method. In the least-squares case, the Hessian never varies, and this makes the Newton method actually just a one step method. But if the Hessian changes rapidly between two close points, (even in terms of small eigenvalues), the Newton steps between two close points may be significantly different. In general, there is no fast convergence rate for Newton methods even in the L-lipschitz µ-strongly convex case.

In the literature, another assumption is made, which allows to control the evolution of the entire Hessian spectrum locally, and which leads to very fast convergence of the Newton method, which is called self-concordance [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF].

Extending the statistical rates. Recall that we consider the problem of minimizing the expected risk

R(f ) = E [ Z (f (X))
] for a loss function z (t), and that we would like to study the performance of the regularized empirical risk minimizer f n,λ . Let R λ be the regularized expected risk and f λ its minimizer. In the case where z is a log-likelihood, as explained by [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF], the local asymptotic normality theory for maximum likelihood estimation can show for instance that for any λ > 0,

√ nH 1/2 λ ( f n,λ -f λ ) -→ n→∞ N (0, H -1/2 λ G λ H -1/2 λ ) (3.52)
where

G λ = E [( Z (f λ (X))k X + λf λ ) ⊗ Z (f λ (X))k X + λf λ )], H λ is the Hessian of R λ at f λ ,
and the convergence happens in distribution.

These results are usually proven using the second order delta method, which relies on the second order development of the function R λ as in Eq. (3.51). However, they are asymptotic. Since in the non-parametric (kernel) setting, there is a trade off between the regularization parameter and n, a non-asymptotic result is necessary to understand the interaction between the two and derive upper rates of convergence.

Note that while the result Eq. (3.52) is not sufficient, it gives a good intuition on what the variance term in a bias-variance decomposition should look like. In order to establish finer non-asymptotic bounds, we need to be able to measure upper bound the distance between f n,λ and f λ with an expression which can be computed in closed form, knowing that the minimizer themselves are not accessible as such. This will be the purpose of the localization lemmas, introduced by Bach (2010); [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF] and which we generalize to the random design, infinite dimensional setting in the work by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]. We will further discuss these lemmas in Sec. 3.2.3.

(Generalized)-self concordance and control of the Hessian

In this section, we present two ways of controlling the variations of the Hessians which can be found in the literature. While the first, called self-concordance, is arguably the best in terms of optimization, it is not obvious to show that our ideal loss function R satisfies this assumption, based on assumptions on the loss z (f (x)); in a sense, it is a priori more adapted to optimization than statistical learning. The second, called generalized self-concordance, is more adapted to the statistical learning setting, and is satisfied by classical losses in machine learning. However, it is less performant from an optimization point of view, as the evolution of the Hessian is bounded less tightly.

In this section, we will denote with F : H → R functions defined on a Hilbert space H. We will use the notation • M to denote M 1/2 • for a positive semi-definite operator M on H. We will denote with H(w) the Hessian of F at point w ∈ H.

Self concordance

The notion of self-concordance has been defined and used by [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF] in order to design Newton based second-order methods. Standard barrier functions, such as the log-barrier or the log determinant in the context of semidefinite programming, satisfy this assumption, making it the key to the success of interior point methods (see works by [START_REF] Boyd | Convex optimization[END_REF]; [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF]; [START_REF] Nesterov | Lectures on Convex Optimization[END_REF] as well as the description of interior point methods in Sec. 2.1.3).

Definition 3.2 (self-concordance).

A function F defined on a domain of H is said to be self-concordant if it is thrice differentiable and

∀h ∈ H, D 3 F (w)[h, h, h] ≤ 2 D 2 F (w)[h, h] 3/2 , (3.53)
where D k F is simply the k-th diffential of F and is a symmetric k form on H.

This assumption implies the following control on the second derivatives which can be found as Eq. ( 9.46) by [START_REF] Boyd | Convex optimization[END_REF] :

∀h ∈ H, 1 (1 + t h H(w 0 ) ) 2 h 2 H(w 0 ) ≤ h 2 H(w 0 +th) ≤ 1 (1 -t h H(w 0 ) ) 2 h 2 H(w 0 ) , (3.54)
this being valid for all t, h, w 0 such that t < h -1 H(w 0 ) . This bound allows to control the Hessian values close to w 0 only using the metric • H(w 0 ) , and can be used to derive the different localization results we will see in Sec. 3.2.2. However, this self-concordance notion is less adapted to the statistical setting. Indeed, if

F 1 , ..., F K are self concordant, then K 1 K K i=1 F i is self-concordant (and not 1 K K i=1 F i ). This is a problem in the setting we consider since in general, even if the functions f ∈ H → z (f (x)) are all self-concordant, the expected risk R(f ) = E [ Z (f (X))] is not necessarily self-concordant; it is hard to know if a problem is self-concordant based
only on the loss function. Note that more involved assumptions can be made to guarantee this property, as in the work by [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF].

CHAPTER 3. BACKROUND AND MAIN RESULTS

Generalized self-concordance

The fact that self-concordant functions do not interact well with expectations motivated the following generalized self-concordance or pseudo self-concordance assumption (GSC), made by [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF] in order to analyse empirical risk minimization for logistic regression. Note that the logistic loss is not self-concordant, but satisfies a different property controlling the variations of its Hessians. Definition 3.3 (generalized self-concordance). A function F defined on a domain of H is said to be R-generalized self-concordant if for any w in the domain of H, it holds

∀h 1 , h 2 ∈ H, D 3 F (w)[h 1 , h 1 , h 2 ] ≤ R h 2 H D 2 F (w)[h 1 , h 1 ].
(3.55)

Moreover, we define r(F ) = inf w∈dom(F ) λ(w)/R, where λ(w) is the smallest eigenvalue of H(w).

Note that in order for r(F ) to be positive, the function F must be strongly convex, and r(F ) = √ λ/R where λ is the highest strong convexity constant of F . The quantity r(F ) will be refered to as the Dikin radius of the function F . Remark 5. The term "generalized self-concordance" can be a bit confusing as a self-concordant function is not necessarily generalized self-concordant. However, all self-concordant functions with bounded Hessians are generalized self-concordant.

Note that a more involved definition of generalized self-concordance is given in chapters 4 and 5.

In this introduction, we will keep to this definition for simplicity. Note that if

F 1 , ..., F K are R generalized self concordant, then 1 K K i=1 F i is also R generalized self-concordant.
More generally, expectations of GSC functions are GSC, which is adapted to our setting. Example 4.1 provide examples of functions in machine learning which are generalized self-concordant, among which the logistic regression function for outputs in {-1, 1} with a RKHS associated to the kernel

k: f ∈ H → x,y (f (x)) is GSC with R = 2 k(x, x).
In the case of generalized self-concordance, the following bound can be obtained on the Hessians and are proved by [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF].

e -tR h H • H(w 0 ) • H(w 0 +th) e tR h H • H(w 0 ) . (3.56)
This bound is not as good as the one in Eq. (3.54) as it depends on the norm of h in H which, in a sense, is not the right norm (the right one being the norm h H(w 0 ) ).

In chapters 4 and 5, we make an analysis analoguous to that of least squares but for generalized self-concordance losses, i.e., when the z (•) are self-concordant with the same constant.

In what follows, we will recall the main properties of self-concordant and generalized selfconcordant functions in terms of optimization and statistics, and relate them to the problem we will handle in Sec. 3.3.

Optimization and Newton methods

In this section, we will show how the Newton method behaves under the self-concordant (SC) and the generalized self-concordant (GSC) assumption. Define the Newton decrement ν(F, w) of F at w :

ν(F, w) 2 = ∆(F, w) 2 H(w) = ∇F (w) H(w) † ∇F (w) = ∇F (w) 2 H(w) -1 , (3.57)
with a slight abuse of notation for the last case (when the Hessian is not invertible). The Newton decrement is a key quantity to analyse the Newton method for SC and GSC functions, and was used for these analysis by [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF]. Indeed, it has the nice property of depending only on the function F at w. Moreover, it is possible to show that when the Newton decrement is small enough, then it is equivalent to the quantities ww * H(w * ) and F (w) -F (w * ), which are the quantities we would like to minimize but to which we have no access since w * is unknown. More formally, we define the Dikin ellipsoids of F as the sets :

• D(F, c) = {w ∈ H : ν(F, w) ≤ c} if F is self-concordant; • D(F, c) = {w ∈ H : ν(F, w) ≤ c r(F )} if F is generalized self concordant.
These sets have been introduced by [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF]; Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]; [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] in different ways which are essentially equivalent to this one. The following proposition shows that as soon as the Newton decrement is small enough, i.e.,, as soon as w belongs to a certain Dikin ellipsoid, the Newton decrement is actually equivalent to the distance to the optimum. This proposition can easily be deduced from the standard Taylor expansion bounds for SC and GSC functions, as can be found in the work by [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF] for SC functions and propositions 4.4 and 4.5 in chapter 4 for the GSC case (note that these last results are themselves inspired from results by [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF]; [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF]).

Proposition 3.2 (localization).

There exists positive constants c 1 , c 2 , c 3 , c 4 such that if there exists w 0 ∈ D(F, c 0 ) where c 0 = 1/2 in the GSC case and c 0 = 1/4 in the SC case, then a minimizer w * of F exists, and for any w ∈ D(F, c 0 ), it holds

ν(F, w) 2 ≤ c 1 w -w * 2 H(w) ≤ c 2 w -w * 2 H(w * ) ≤ c 3 (F (w) -F (w * )) ≤ c 4 ν(F, w) 2 .
(3.58)

The result stated in proposition 3.2 is called a localization result since it shows that one can localize the minimizer w * with only local information at w (i.e., its Newton decrement) as soon as the Newton decrement is small enough.

The previous results shows that if we are able to control the decrease of the Newton decrement, then we are able to control the distance to the optimum. The analysis of the full Newton method relies precisely on this fact, and shows that as soon as w is in the right Dikin ellipsoid, then the Newton decrement of w -∆(F, w) can be controlled by the one of w. This result can be found as Proposition 2 in the work by [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF] for the GSC case. The formulas for the SC case are well known from the work by [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF] and are also recalled in section 1. of the work by [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF].

Theorem 3.9 (behavior of the full Newton method). If F is self concordant, then as soon as w ∈ D(F, c 0 ) for c 0 = 1/4, F has a global minimizer w * and it holds :

ν(F, w -∆(F, w)) ≤ ν(F, w) 2 . (3.59)
If F is generalized self concordant, then as soon as w ∈ D(F, c 0 ) for c 0 = 1/2, F has a global minimizer w * and it holds :

ν(F, w -∆(F, w)) r(F ) ≤ ν(F, w) r(F ) 2 . (3.60) CHAPTER 3.

BACKROUND AND MAIN RESULTS

The results of Theorem 3.9 directly imply that a full Newton method w t+1 = w t -∆(F, w t ) starting at w 0 ∈ D(F, t 0 ) converges quadratically, i.e., w t ∈ D(F, t 2 t 0 ).

Note that this fast convergence only happens "near" the optimum, that is for small Newton decrements. If one wishes to apply a Newton method starting from any point w 0 ∈ H, one cannot apply a full Newton method directly. This is not an issue in the context of SC functions. Indeed, one can apply a damped Newton method w t+1 = w t -α t ∆(F, w t ), where the stepsize α t is suitably chosen (either by a line search or defined by the Newton decremement, see the original method by [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF] or the book by [START_REF] Boyd | Convex optimization[END_REF] for the line-search version). This first phase can be shown to converge in a finite number of steps which is independent from the function F to minimize and is proporitional to the quantity F (w 0 ) -F (w * ) (in particular, it does not depend on the conditioning of the Hessians of F ). In a sense, for SC functions, the Dikin ellipsoid is large, and reaching it is not complicated.

In the context of GSC functions however, this becomes an issue. First, choosing the stepsize α t is more involved, and does not lead to a constant decrease of F (w t ) -F (w * ) as in the SC case (see the results by [START_REF] Sun | Generalized Self-Concordant Functions: A Recipe for Newton-Type Methods[END_REF]). Moreover, the Dikin ellipsoid is smaller, since r(F ) depends on the conditioning of the problem through the strong convexity constant of F .

Roughly, one has to reach a precision of order F (w t ) -F (w * ) ≈ λ R 2 in order to be in the Dikin ellipsoid, where λ is the strong convexity constant (this is to be compared to F (w t ) -F (w * ) ≈ 1 16 in the SC case). That is why the Newton method is said to be a local method for GSC functions, i.e., it converges fast locally. In Sec. 3.3.3 and chapter 5, we will present a global scheme to minimize GSC functions. Summary. To summarize, we have defined classes of functions, SC and GSC functions, which allow a precise local control of the Hessians. We have seen that SC and GSC functions can be locally optimized using a full Newton method. These results are based on two main ingredients : a) as soon as a point w 0 is in the Dikin ellipsoid of F with certain radius t 0 , the Newton decrement characterizes the distance of w 0 to the optimum, and b) in this same Dikin ellipsoid, the full Newton methods induces a quadratic decrease of the Newton decrement and hence of the distance to the optimum in function values. However, while a global scheme can be derived for SC functions, it is not the case for GSC functions, and reaching the Dikin ellipsoid is a priori a challenge.

Remarks. Note that a unifying framework to analyze Newton methods for SC and GSC functions (and other classes of functions) has been provided by [START_REF] Sun | Generalized Self-Concordant Functions: A Recipe for Newton-Type Methods[END_REF]. In particular, they provide a choice of stepsize in the first phase of the Newton method for the GSC case. However, this does not lead to a constant decrease in function values, as is the case for SC functions.

Moreover, [START_REF] Sun | Generalized Self-Concordant Functions: A Recipe for Newton-Type Methods[END_REF] also consider the case where the Newton method is not applied exactly, but using a relative approximation of the Newton step. More formally, they consider the case where the Newton step ∆(F, w t ) is approximated by ∆ t such that ∆(F, w t ) -∆ t H(wt) ≤ ρν(F, w t ) for a fixed ρ < 1. This can be quite useful in practice, as it allows to avoid solving the linear system to compute the Newton step. Using this approximate Newton step instead of the full Newton step still leads to a fast convergence of the Newton method in the Dikin ellipsoid, but at the cost of losing the quadratic convergence for a linear convergence. We will detail this in chapter 5 as well as in Sec. 3.3.3, where we derive a globally convergent algorithm for GSC functions based on approximate Newton methods.

Statistics

In this section, we show how the control brought by generalized self-concordance can help derive statistical bounds, by allowing to localize the empirical risk minimizer using a Newton decrement, on which wecan then apply standard concentration bounds. More specifically, in this section, we will show how the reguarlized empirical risk minimizer can be localized in proposition 3.3, paving the way to the use of standard concentration bounds to bound the variance term in a bias variance decomposition. We will the use these bounds in Sec. 3.3.1 to prove a bias-variance decomposition in the setting of GSC functions.

In spirit, the same type of bounds have been developed by [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF]; [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF]. However, they are made for a different setting than ours, and therefore slightly differ from the method we will expose here. In particular, Bach (2010) considers the fixed design setting, while the work by [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF] does not include the regularization parameter, which is crucial in the non-parametric setting.

Let us once again consider the problem of minimizing the expected riskt R(f

) = E [ Z (f (X))]
over the set H, whose Hessian at f we denote with H(f ). We will make the following assumption on the loss function.

Assumption 3.7 (GSC assumption). For all z ∈ Z, the function t → z (t) is R generalized self-concordant.
As in the previous sections, we will also assume that the kernel is bounded by 1 (see Assumption 3.3). We adopt the following notations for the different regularized functions and Hessian of functions.

• We denote with R λ = R + λ 2 • 2 H the regularized expected risk, and denote with H λ (f ) its Hessian at f , and with f λ its minimizer (which exists by strong convexity as soon as λ > 0). R λ is R GSC and λ strongly convex.

• We also denote with R n,λ the regularized empirical risk, and denote with H n,λ (f ) its Hessian at point f . Recall that as soon as λ > 0, it R n,λ has a unique minimizer f n,λ . R n,λ is also R GSC and λ strongly convex.

• Finally, we denote with r λ the quantity √ λ/R. It is a lower bound for the Dikin radius of R λ and R n,λ : we have r

(R λ ), r( R n,λ ) ≥ r λ . Bounding the Newton decrement of a function f ν( R n,λ , f ) = ∇ R n,λ (f ) H -1 n,λ (f ) ≤ H -1/2 n,λ (f )H λ 1/2 (f ) ∇ R n,λ (f ) H -1 λ (f ) , (3.61)
we can show the following localization bound for the empirical risk minimizer by using proposition 3.2.

Proposition 3.3 (localization of the the regularized e.r.m.). There exists an explicit constant c such that for any

f ∈ H, if ∇ R n,λ (f ) H λ -1 (f ) ≤ r λ 4 , H -1/2 n,λ (f )H λ 1/2 (f ) ≤ 2, (3.62)
then the following holds :

f -f n,λ H λ (f ) ≤ c ∇ R n,λ (f ) 2 H λ -1 (f ) (3.63) 114 CHAPTER 3.

BACKROUND AND MAIN RESULTS

To obtain a bound on the variance term in the bias variance decomposition for GSC functions, we will apply proposition 3.3 to f λ , which paves the way to bound f λ -f n,λ H λ (f λ ) . This term will account for the variance term in our bias-variance trade off (see Sec. 3.3.1 as well as the sketch of proof in Sec. 4.6). Indeed, it is possible to guarantee

H -1/2 n,λ (f )H λ 1/2 (f ) 2 ≤ 2 using
standard concentration bounds for operators, and to bound ∇ R n,λ (f λ ) H λ -1 (f λ ) as we can see it as

1 n n i=1 ξ i where ξ i are i.i.d. samples from ξ = H λ (f λ ) -1/2 ( Z (f λ (X))k X + λf λ )
, which has zero mean. This term can therefore also be concentrated using standard concentration bounds for random vectors in a Hilbert space.

Main results and contributions of this part

In this section, we present the main results of our contributions on extending the fast rates and algorithms existing for the least squares problem to a broader class of convex functions, the class of GSC functions.

The goal of this section is to present our results in a coherent, unified framework, as well as an idea of the technical contributions. In the original articles, whose verbatim can be found in chapters 4 and 5, the notations are slightly different, and the assumptions are often weaker. These articles contain all the proofs and precise statements, and have been peer-reviewed. We will refer to the results we invoke from these articles in the introduction, even though the notations may vary.

In order to present the results, the outline will be the following. We will start in Sec. 3.3.1 by presenting the statistical results obtained in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] on the non-asymptotic performance of the regularized e.r.m. estimator, which mirrors the results in the least squares setting, presented in Sec. 3.1.1. We will continue in Sec. 3.3.2 to show how it is possible to reduce the dimension of the resulting finite dimensional problem, in the same way as in the least-squares case (although the proofs are a bit more involved). These results have been proved by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]. In Sec. 3.3.3, we present one of the main contributions by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] which is a globally convergent second order scheme for GSC functions which is roughly independent of the conditioning of the problem. Finally, in Sec. 3.3.4, we show that, as the FALKON algorithm in the least-squares case, it is possible to effectively compute an estimator, using the previous scheme, with low time complexity and optimal statistical precision. This has also been derived by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF].

Assumptions

We use all notations for functions and Hessians introduced in Sec. 3.2.3. The goal is to minimize the expected risk R(f

) = E [ Z (f (X))] from samples z i from Z. Let ρ be the distribution of Z.
As in the least squares case, we make the assumption that the samples z 1 , ..., z n are i.i.d. samples from Z (see Assumption 2.1). Moreover, we assume there exists f ρ ∈ H such that f ρ minimizes the expected risk (see Assumption 3.1). We will assume that the kernel is bounded by 1, although the results hold as soon as the kernel is bounded (see Assumption 3.3). We will also suppose that Assumption 3.7 is satisfied, that is that the functions z (•) are R GSC for all z ∈ Z.

We also need a more technical assumption. First, for any distribution ρ of Z, define

b 1 (f ) = | Z (f (X))| L ∞ (ρ) , b 2 (f ) = | Z (f (X))| L ∞ (ρ) .
(3.64)

MAIN RESULTS AND CONTRIBUTIONS OF THIS PART 115

We will assume that these quantities is bounded above at the optimum f ρ .

Assumption 3.8 (Bounded gradients and Hessians). There exists finite constants b

1 , b 2 such that b 1 (f ρ ) ≤ b 1 , b 2 (f ρ ) ≤ b 2 . (3.65)
The assumption on the gradient replaces the noise assumption in the least squares setting. Indeed, if is the loss associated to the least-squares loss, the assumption on the gradient is just |Y -f ρ (X)| ≤ b 1 , which in particular implies the noise assumption with σ, M = b 1 .

Note that we will use the shorthand notations H, H λ , H n,λ to denote the quantities H(f ρ ), H λ (f ρ ), and H n,λ (f ρ ).

Examples

We recall two examples where the GSC assumption Assumption 3.7 is satisfied : logistic regression, and the square loss. Other examples can be found in Example 4.1.

Example 3.1. For logistic regression, we have z (t) = log(1 + exp(-yt). If we assume that Z = X × [-1, 1] (or more generally that the y are bounded), Assumption 3.7 is satisfied with R = 2. Moreover, the z are all 1 lipschitz, and we have :

∀f ∈ H, b 1 (f ) ≤ 1, b 2 (f ) ≤ 1/4. (3.66)
Example 3.2. For least-squares regression, we have z (t) = |t-y| 2 . In this case, Assumption 3.7 is satisfied for any positive R. Moreover, we have, for any distribution ρ for Z,

∀f ∈ H, b 1 (f ) = |Y -f (X)| L ∞ (ρ) , b 2 (f ) ≤ 1.
(3.67)

Statistics of the regularized empirical risk minimizer for GSC functions

In this section, we present the results obtained by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] on regularized empirical risk minimization for GSC functions. We start by showing that, as in the least-squares case, there is a bias-variance trade-off of the form

R( f n,λ ) -R H ≤ C log 2 1 δ b λ + d λ n , with probability at least 1 -δ, (2.58)
where the bias term and the effective dimension are defined as meaningful quantities which match those defined in the least-squares case, described below. We then present rates of convergence for classes of distributions ρ, as in the least-squares case, before presenting a high level description of the proof of the bias variance trade off.

In [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], great attention has been put to obtain bounds with explicit constants. While we do not report them here, these can be found in the referenced theorems.

Bias-variance decomposition

In this context, the descritption of the bias and variance tade off is done through the introduction of two key quantities, which are the analogs of the quantities b λ and σ 2 d λ in the least-squares setting.

CHAPTER 3. BACKROUND AND MAIN RESULTS

• For any λ > 0, the bias b λ is defined as

b λ := ν(R λ , f ρ ) 2 = ∇R λ (f ρ ) 2 H -1 λ = λ 2 f ρ 2 H -1 λ .
(3.68)

• For any λ > 0, the effective dimension df λ is defined as

df λ := E H -1/2 λ Z (f ρ (X))k X 2 = Tr(H -1/2 λ GH -1/2 λ ), G = E Z (f ρ (X)) 2 k X ⊗ k X (3.69)
In the least squares setting, we have H = Σ and

G = E (Y -f ρ (X)) 2 k X ⊗ k X . (3.70)
The definition of the bias b λ in Eq. (3.68) therefore exactly matches the definition given in Sec. 3.1.1 in the least-squares case. The definition of the effective dimension is slightly different, and can be related to the effective dimension d λ introduced in the context of least-squares as follows :

df λ ≤ σ 2 d λ , (3.71)
where σ 2 is an upper bound on the variance of the noise [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] prove the following bias variance decomposition (we denote with a ∨ b the maximum between a and b), which is the main result of their paper, and is reported here in Theorem 4.4.

E (Y -f ρ (X)) 2 |X (see Assumption 3.4). Marteau-Ferey,
Main theorem 8: Marteau-Ferey et al. ( 2019), Theorem 4.

There exists explicit constants n 0 , n 1 and C bias , C var such that for any n ∈ N, δ

∈ (0, 1/2], 0 < λ ≤ b 2 , whenever n ≥ n 0 b 2 λ log b 2 λδ , n ≥ n 1 df λ ∨ q 2 r 2 λ log 2 δ , b λ ≤ r λ /2, with q 2 = b 2 1 /b 2 , then with probability at least 1 -2δ, it holds R( f n,λ ) -R(f ρ ) ≤ C bias b λ + C var df λ ∨ q 2 n log 2 δ . (3.72)

Upper rates of convergence

Following what is done in Sec. 3.1.1, we define classes of test measures M for which we get precise statistical rates. Formally, for R,

Q, b 1 , b 2 > 0, r ∈ [1/2, 1] and b ≥ 1, we will define the class of test measures M(R, r, Q, b, b 1 , b 2 )
as the set of measures ρ such that the following hold.

(i) Source condition : there exists h ∈ H such that f ρ = H r-1/2 h and h H ≤ R. This is exactly the source condition in the least-squares setting, but adapted to the case where the metric is not the L 2 metric anymore, but the metric associated to the loss function .

(ii) Capacity condition : the generalized eigenvalues λ i of the pair (G, H λ ) (that is the eigenvalues of

H -1/2 λ GH -1/2 λ ) satisfy n i=1 λ i λ i + λ = Tr(H -1/2 λ GH -1/2 λ ) ≤ Q 2 λ -1/b . (3.73)
In particular, this is satisfied as soon as the λ i ≤ β i b for some b > 1.

(iii) Boundedness of gradients and Hessians : Assumption 3.8 is satisfied with b 1 , b 2 .

On this class of functions, the following non-asymptotic upper rates of convergence are derived from Main theorem 8, which was originally proved as Cor. 4.4.

Main corollary 3: Marteau-Ferey et al. ( 2019), Corollary 3 andb ≥ 1 andlet

Fix b 1 , b 2 > 0, 1 ≥ r > 1/2, Q > 0,R > 0,
M = M(R, r, Q, b, b 1 , b 2 ). If we set λ = 256Q 2 R 2 n 1 2br+1 , (3.74)
then for any δ ∈ (0, 1/2], with probability at least 1 -2δ,

∀ρ ∈ M, R( f n,λ ) -R(f ρ ) ≤ 8 R 2 256Q 2 R 2 n 2br 2br+1 log 2 δ , (3.75)
provided n ≥ N and where N is defined in Eq. ( 4.46), depends on all the parameters of the model M as well as δ, (in particular, it depends polynomially on log 1 δ ).

This result can be adapted to resemble Cor. 3.1 (in particular, where the relationship between n and log 1 δ is more explicit), which we assume could then lead to minimax upper rates. However, this is not done in the work by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]. Note that we have assmumed a stronger source condition than the simple well-specified assumption, i.e., we have assumed r > 1/2. In the case where r = 1/2, some rates can be obtained, but have a worst depedence on the problem parameters, through an exponential constant e R fρ . This is due to the fact that we cannot guarantee the condition b λ ≤ r λ /2 in that setting. For more details, we refer to chapter 4 for more elements on this question.

Proof techniques

We refer to Sec. 4.6 for a sketch of the proof.

Reducing the dimension while keeping optimal rates

As explained in the least-squares regression case, solving the regularized empirical risk minimization problem can be computationally expensive. The first step to have a better algorithm is to reduce dimension of the problem through one of the dimension reduction techniques presented in Sec. 3.1.2. Marteau-Ferey, Bach, and Rudi (2019) consider the case where the dimension is reduced by sampling m Nyström points x 1 , ..., x m either uniformly or using approximate leverage scores associated to the kernel matrix K = (k(x i , x j )) ∈ R n×n . Recall that in the last case, the technique is to sub-sample indices i 1 , ..., i m from {1, ..., n} using a good approximation of the probability vector p i = l i (t)/ i l i (t), where l i (t) = e i K(K + ntI) -1 e i (for more details on leverage scores, see Eq. (3.42) or Sec. 5.D .3). The Nyström points are then defined as x j = x i j . When performing Nyström sampling, H is approximated by

CHAPTER 3. BACKROUND AND MAIN RESULTS H m =    x → m j=1 α j k(x, x j ) : α ∈ R m    , (3.35)
This set can simply be seen as a m dimensional subspace of H (it is simply span(k

x j : 1 ≤ j ≤ m))
with the induced norm :

m j=1 α j k(•, x j ), m j=1 β j k(x, x j ) Hm = α K mm β
where K mm is the kernel matrix associated to the x j . The kernel associated to the space H can be expressed as

k m (x, x ) = k m (x) K † mm k m (x)
where k m (x) = (k( x j , x)) 1≤i≤m ∈ R m and † denotes the Moore-Penrose pseudoinverse. The dimension reduced estimator is then computed as

f n,λ,m = arg min f ∈Hm 1 n n i=1 z i (f (x i )) + λ 2 f 2 Hm (3.76)
Using the parametrization of H m in Eq. (3.35), the problem Eq. (3.76) becomes

α n,λ,m = arg min α∈R m 1 n n i=1 z i (e i K n,m α) + λ 2 α K mm α, K n,m = (k(x i , x j )) 1≤i≤n 1≤j≤m , (3.77)
and the minimizer of Eq. (3.76) can be expressed as f n,λ,m = k(x) α n,λ,m . As in the least-squares case, a much better way parametrizing is to write K mm = T T using a cholesky decomposition, where T is upper triangular matrix. One then notes that defining φ

(x) = T -k m (x) ∈ R m , φ is a feature map which defines the kernel k m : k m (x, x ) = φ(x) φ(x ). Hence, H m = {α φ(•) : α ∈ R m }. If Φ is the matrix whose columns are the Φ i = φ(x i ), that is Φ = K n,m T -1
, the problem Eq. (3.76) can be formulated as

α n,λ,m = arg min α∈R m 1 n n i=1 z i (Φ i α) + λ 2 α 2 , (3.78)
and the minimizer of Eq. (3.76) can be expressed as

f n,λ,m = k(x) (T -1 α n,λ,m
). Of course, this comes at the price of computing and saving a cholesky factor T, which is O(m 3 ) in time and O(m 2 ) in memory.

The following theorem presents a bias variance trade-off in the case where the dimension is reduced using Nyström subsampling. This is a rewriting of a theorem by Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] which can be found in Theorem 5.6.

Main theorem 9: Marteau-Ferey, Bach, and Rudi (2019)

There exists explicit constants n 0 , n 1 , m 0 , m 1 , C, and C bias , C var as well as a constant λ 0 depending on b 2 such that for any n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ λ 0 , whenever

n ≥ n 0 1 + b 2 λ log 1 + b 2 λδ , n ≥ n 1 df λ ∨ q 2 r 2 λ log 2 δ , C b λ ≤ r λ /2, with q 2 = b 2 1 /b 2 .
Assume that dimension reduction has been applied using Nyström subsampling, where the samples are obtained with either one of the following values of m and subsampling techniques.

(i) m ≥ m 0 1+b 2 λ log 8(1+b 2 ) λδ using uniform sampling. (ii) m ≥ m 1 d λ/(1+b 2 ) log 8(1+b 2 ) λδ using approximate leverage scores with t = λ/(1 + b 2 ). With probability at least 1 -δ, it holds R( f n,λ,m ) -R(f ρ ) ≤ C bias b λ + C var df λ ∨ q 2 n log 2 δ , (3.79) and R f n,λ,m ≤ R f ρ + 10.
The proof of this proposition is done in chapter 5. To incorporate the fact that we reduce the dimension, the key quantity to bound is H 1/2 (I -P m ) where P m is the orthogonal projection on H m (the subset of H defined by the Nyström points). We show that approximating H with H m is statistically optimal as soon as H 1/2 (I -P m ) ≤ c √ λ for a certain constant c (see Sec. 5.H .3 for more details). There is a subtlety here in the case where sampling with leverage scores is used. Indeed, the ideal goal is to approximate the operator H with as few Nystrom points as possible. However, we cannot compute any kind of approximation of the leverage scores of the associated empirical operator H n since f ρ is not known ( H n = H n (f ρ )). Instead, we know how to approximate leverage scores of the empirical covariance Σ associated to the covariance operator Σ, and since H b 2 Σ, we can use this to obtain a guarantee √ b 2 Σ 1/2 (I -P m ) ≤ c √ λ using these leverage scores, which will imply the desired bound. That is why in point (ii) of Main theorem 9, the effective dimension of the covariance operator d λ/(1+b 2 ) appears.

The previous results leads to the following informal upper rate of convergence, which is a reformulation of a sub-case of Theorem 5.7.

Main corollary 4: Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]

, Theorem 7 Fix b 1 , b 2 > 0, r > 1/2, Q > 0,R > 0, and b ≥ 1 and let M = M(R, r, Q, b, b 1 , b 2 ). Let λ = Q 2 R 2 n 1 2br+1 , (3.80)
and let δ ∈ (0, 1/2]. Assume that dimension reduction has been applied using Nyström subsampling, where the samples are obtained with either one of the following values of m and subsampling techniques.

(i) m ≥ m 0 1+b 2 λ log 8(1+b 2 ) λδ using uniform sampling. (ii) m ≥ m 1 d λ/(1+b 2 ) log 8(1+b 2 ) λδ using approximate leverage scores with t = λ/(1 + b 2 ). With probability at least 1 -2δ, it holds ∀ρ ∈ M, R( f n,λ,m ) -R(f ρ ) ≤ C R 2 Q 2 R 2 n 2br 2br+1 log 2 δ , (3.81)
where C is an explicit constant, provided n ≥ N and where N can be characterized using the proof of Theorem 5.7, depends on all the parameters of the model M as well as δ, (in particular, it depends polynomially on log 1 δ ).

Note that the above result can be made formal, but for the sake of readability, we have left it in this form. Note that contrary to the least squares case, we do not explicitely give the order of m as a function of n. This could be done by assuming another eigenvalue decrease for the effective dimension of the covariance matrix in the model parameters. However, for the sake of readability, we will keep this implicit.

A globally convergent optimization algorithm for GSC functions

Recall from Sec. 3.2.2 that Newton methods converge very fast in the vicinity of the optimum, that is when they are initialized at a point w 0 which lies in a certain Dikin ellipsoid (see Theorem 3.9). The problem is that this region of fast convergence is not directly accessible, and can be quite small. The purpose of this section is to present an algorithm introduced by Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], whose purpose is to enter this small region.

More formally, let F be a R GSC function defined on a Hilbert space H. We define F λ to be the λ-regularizer version of F , i.e.,

F λ = F + λ 2 • 2 H . Let r λ := √ λ/R.
We denote with D(F λ , c) the modified Dikin ellipsoid :

D(F λ , c) := {w ∈ H : ν(F λ , w) ≤ c r λ }. (3.82)
Note that by definition of r(F λ ), we have r λ ≤ r(F λ ) and hence D(F λ , c) ⊂ D(F λ , c). Moreover, we will denote with H(w), H λ (w) the Hessians of F , F λ at w, and with w λ the minimizer of F λ for λ > 0.

The aim of the section is to present an algorithm which minimizes F λ up to ε error : we want to find x such that ν(F λ , x) 2 , F (x) -F λ (x λ ) ≤ ε. To do so, we will proceed in two steps.

• We will start by showing a variant of Theorem 3.9 which shows that we can actually perform an approximate Newton method rather than an exact Newton method in the modified

MAIN RESULTS AND CONTRIBUTIONS OF THIS PART 121

Dikin ellipsoid, where the price to pay is a linear rather than quadratic convergence towards the optimum. This will be useful when solving the dimension reduced problem Eq. (3.78), where computing an exact Newton step would be too costly.

• We then explain the core of the algorithm by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] to reach the Dikin ellipsoid of F λ . Similarly to interior point methods, this method is based on approximately minimizing F λt for a decreasing sequence (λ t ) 0≤t≤T where λ T = λ. It is based on the fact that the approximation w λt of the minimizer w λt will be in a modified Dikin ellipsoid of F λ t+1 with good properties, thus guaranteeing that we can perform a fast minimization of F λ t+1 starting from w λt using an approximate Newton method. Crucially Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] show that under certain conditions, this method is independent of the conditioning of the problem (up to log factors).

At the end of this section, we will also show how approximate Newton steps can be computed in the setting where F λ is given by Eq. (3.78), our dimension reduced problem.

The rest of this section is greatly inspired from certain sections in the work by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], and in particular Secs. 5.2 , 5.3 , 5.B and 5.C .

Approximate Newton methods for GSC losses

In Theorem 3.9, we saw the behavior of a full Newton method inside the Dikin ellipsoid D(F λ , c) for c ≤ 1/2, and a fortiori in the modified Dikin ellipsoid D(F λ , c) for c ≤ 1/2. However, computing the full Newton step ∆(F λ , w) at point w ∈ H requires the exact solving of a linear system, which can be very expensive when the dimension of H is large. A natural idea is to approximate the Newton iteration, leading to approximate Newton methods (ANM), which take the form

w t+1 = w t -∆(F λ , w t ), ∆(F λ , w t ) ≈ ∆(F λ , w t ), (3.83)
and the symbol ≈ will be formally defined. [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] generically consider any technique to compute approximate Newton steps ∆(F λ , w) which are ρ-relative approximation of ∆(F λ , w), defined as follows [START_REF] Deuflhard | Newton Methods for Nonlinear Problems: Affine Invariance and Adaptive Algorithms[END_REF].

Definition 3.4 (relative approximation). Let ρ < 1, let A be an invertible positive definite Hermitian operator on H and b in H. A ρ-relative approximations of z

* = A -1 b is an element z satisfying z -z * A ≤ ρ z * A .
We denote with LinApprox(A, b, ρ) the set of all ρ-relative approximations of z * = A -1 b.

A ρ-relative approximation of the full Newton step of F λ at w will therefore be any element ∆(F λ , w) ∈ LinApprox(H λ (w), ∇F λ (w), ρ). Moreover, we will say that w is the result of k steps of an approximate Newton method of parameter ρ < 1 starting at w 0 , and write w ∈ ANM ρ (F, w 0 , k), if there exists w 1 , ..., w k such that w = w k and for any t ∈ {1, ..., k}, we have

w t-1 -w t ∈ LinApprox(H λ (w t-1 ), ∇F λ (w t-1 ), ρ).
The following result, which can be found in Lemma 5.2 and proved in Lemma 5.11 in Sec. 5.B .3, shows that when w 0 ∈ D(F λ , c) for a sufficiently small c, the convergence of the approximate Newton method is linear and does not depend on the condition number of the problem, as was the case for full Newton methods. Note that this type of result is not new, and had already been obtained, if not formalized in exactly the same way, in a work by [START_REF] Sun | Generalized Self-Concordant Functions: A Recipe for Newton-Type Methods[END_REF].

Proposition 3.4 (behavior of the approximate Newton method). Let c 0 = 1/7 and ρ 0 = 1/7. For any w ∈ D(F λ , c 0 ), it holds :

1 4 ν 2 (F λ , w) ≤ F λ (w) -F λ (w λ ) ≤ ν 2 (F λ , w). (3.84) As soon as ∆(F λ , w) ∈ LinApprox(H λ (w), ∇F λ (w), ρ) for ρ ≤ ρ 0 , it holds ν(F λ , w -∆(F λ , w)) ≤ 1 2 ν(F λ , w). (3.85) Thus, if w 0 ∈ D(F λ , c 0 ), then any w ∈ ANM ρ 0 (F λ , w 0 , k) satisfies ν(F λ , w) ≤ 2 -k ν(F λ , w 0 ).

Globally convergent scheme for ANM algorithms on GSC functions

We are now ready to introduce the globally convergent scheme proposed by Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]. Note that in the literature, some other globalization schemes arrive to regions of interest by first-order methods or back-tracking schemes Agarwal, Bullins, and Hazan (2017); A. [START_REF] Murat | Convergence rates of sub-sampled Newton methods[END_REF]. However such approaches require a number of steps that is usually proportional to L/λ, making them depend on the condition number of the problem.

Instead, our algorithm is based on the previous observation that as soon as w ∈ D(F λ , c 0 ) (with say c 0 = 1/7), t steps of ANM converge as fast as 2 -t : it is therefore interesting to apply ANM only in Dikin ellipsoids. Our idea is to start from a very large regularization parameter λ 0 , such that we are sure that w 0 = 0 is in the convergence region D(F λ 0 , w 0 ), and perform some steps of an approximate Newton methods to obtain an approximation w 1 of w λ 0 such that the solution enters in the Dikin ellipsoid of F λ 1 , for a certain λ 1 < λ 0 , and to iterate this procedure until we enter the convergence region of F λ K where λ K = λ. Formally, we propose the following globalization scheme, where the function q(•, •) will help define the next λ k from the previous one.

Proposed Globalization Scheme

Phase I: Getting in the Dikin ellispoid of F λ Inputs :

w 0 ∈ H, λ 0 > 0, t ∈ N, T ∈ N, ρ < 1 and a function q : H × R + → [0, 1). k ← 0 While λ k > λ w k+1 ← w ∈ ANM ρ (F λ k , w k , t) λ k+1 ← max(q(w k+1 , λ k )λ k , λ) k ← k + 1
Phase II: reach a certain precision starting from inside the Dikin ellipsoid

Output : w ∈ ANM ρ (F λ , w k , T )
The main ingredient to guarantee that the scheme will work is the following lemma (see Lemma 5.13 in Sec. 5.C .1 for a proof).

Lemma 3.1. Let c < 1 and w ∈ H. Let q c (w) = 1 - 2 3(1+R w /c) . ∀q ∈ [q c (w), 1), ∀λ > 0, w ∈ D(F λ , c/3) =⇒ w ∈ D(F qλ , c).
(3.86)

This will allow to show the loop invariant

w k ∈ D(F λ k , c). Indeed assume that w k-1 ∈ D(F λ k-1 , c). Then ν(F λ k-1 , w k-1 ) ≤ c λ k-1 /R.
= ANM ρ (F λ k-1 , w k-1 , t), by proposition 3.4, w k ∈ D(F λ k-1 , c/4). Setting q = q(w k ), this implies that w k ∈ D(F qλ k-1 , c) = D(F λ k , c
), by Lemma 3.1. Now we are ready to state our main theorem of this section.

Main theorem 10: Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], Theorem 1.

Let > 0. If the proposed globalization scheme is performed with the following input parameters :

• λ 0 = 7R ∇F (0) and w 0 = 0 for the initialization points;

• ρ ≤ 1/7 and t = 2 for the approximate Newton methods in phase I;

• q(w, λ) = q 1/7 (w) in order to update the value of λ;

• T = log 2 1 ∨ (λ -1 /R 2 )
in the last approximate Newton method in phase II. The proposed scheme with these parameters finishes, and if K denotes the number of passes through the Phase I loop, it holds

F λ (w) -F λ (w λ ) ≤ ν(F λ , w) 2 ≤ , K ≤ (3 + 11R w λ ) log(7R ∇F (0) /λ) .
Note that the theorem above (proven in Sec. 5.C .3) guarantees a solution with error with K steps of ANM each performing 2 iterations of approximate linear system solving, plus a final step of ANM which performs T iterations of approximate linear system solving.

Remarks. The proposed method does not depend on the condition number of the problem, but on the term R w λ which can be in the order of R/ √ λ in the worst case, but is usually way smaller. For example, it is possible to prove that this term is bounded by an absolute constant not depending on λ, if at least one minimizer of F exists. Note that in the statistical setting, i.e., when solving Eq. (3.78), then w λ is bounded by a term coming from the statistical problem, as can be seen in Main theorem 9.

In proposition 5.7, Marteau-Ferey, Bach, and Rudi (2019) show a variant of this adaptive method which can leverage the regularity of the solution with respect to the Hessian, i.e., depending on the smaller quantity R √ λ w λ H -1 λ (w λ ) instead of R w λ . Finally note that it is possible to use q k = q fixed for all the iterations and way smaller than the one in Theorem 5.1, depending on some regularity properties of H (see proposition 5.8 in Sec. 5.C .2).

Methods for computing approximate Newton steps in the setting of decomposable losses

In the previous section, our algorithm to compute an ε approximation of the minimizer of F λ relies crucially on the capacity to perform approximate Newton methods for F λ with λ ≥ λ. In this section, we will present different methods to compute such approximate Newton steps. They all approximate the true Newton step ∆(F λ , w) with ∆(F λ , w) of the form H λ (w) -1 ∇F λ (w), where H λ (w) is an approximation of the Hessian H λ (w) satisfying

1 √ κ H λ (w) H λ (w) √ κH λ (w), (3.87)
for some κ ≥ 1. In that case, it is easy to prove that ∆(F λ , w) ∈ LinApprox(H λ (w), ∇F λ (w), ρ)

with ρ = 1 -1/ √ κ.
Assume now that F is defined on a finite dimensional Hilbert space R m , and is of the form 124 CHAPTER 3. BACKROUND AND MAIN RESULTS

F (w) = 1 n n i=1 i (Φ i w), (3.88)
where the Φ i are bounded by one, and the i are

R-GSC. Let b 2 (w) = sup i (2)
i (Φ i w). The Hessian of F at w is in the form :

∇ 2 F (w) = 1 n Ψ w Ψ w , Ψ w = Φ diag (2) ı (Φ i w) ∈ R m×n . (3.89)
Note that in Sec. 3.1.3, we introduced sketching and subsampling methods which exactly compute

Ψ w ∈ R m×q such that H λ = 1 n Ψ w Ψ w + λI satisfies Eq. (3.87) with probability at least 1 -δ as soon as q ≥ d λ (Ψ w ) log b 2 (w)
λδ , where

d λ (Ψ w ) = Tr(H λ (w) -1 H(w)) ≤ d λ/b 2 (w) (Φ).
If, for example, we apply the sketching method by [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF] to perform the approximate Newton steps (with κ such that ρ ≤ 1/7), the total complexity of global method can be bounded by

O (nm log n + mq 2 + q 3 ) R w λ log R ∇F (0) λ + log 2 (1 + (λ -1 /R 2 )) in time, q = O d λ/b 2 log b 2 λδ , (3.90)
where b 2 = sup w∈Γ λ b 2 (w), is the supremum of the bounds on the second derivatives of the i over the path of regularized minimizers Γ λ = {w λ : λ ≥ λ}, which included in the ball centered at 0 and of radius w λ .

Statistical bounds for the whole algorithm

Note that the dimension reduced problem can be cast exactly as a problem of the form Eq. (3.88).

Recall that in that case, f n,λ,m = f α n,λ,m where f α = α φ(x), for φ(x) defined just before Eq. (3.78), and where α n,λ,m is defined by

α n,λ,m = arg min α∈R m F λ (α), F (α) = 1 n n i=1 z i (Φ i α), (3.78)
and which can be optimized using the global scheme, with approximate Newton steps computed using one of the techniques explained in Sec. 3.1.3. We now state an informal result, which gives the spirit of the results by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], Theorem 4.4, or in Sec. 5.D .6 on the complexity and statistical performance of the algorithm. For the sake of simplicity, assume here that the

(2)
z are all bounded by a constant b 2 . Note that this result is the analog of the result obtained in the least-squares case for FALKON.

Main theorem 11: Marteau-Ferey, Bach, and Rudi (2019), Proposition 14

There exists explicit constants n 0 , n 1 , m 0 , m 1 , C, and C bias , C var as well as a constant λ 0 depending on b 2 such that the following hold. Let ε > 0, n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ λ 0 and assume

n ≥ n 0 1 + b 2 λ log 1 + b 2 λδ , n ≥ n 1 df λ ∨ q 2 r 2 λ log 2 δ , C( b λ + √ ε) ≤ r λ /2, with q 2 = b 2 1 /b 2 .
Assume that dimension reduction has been applied using Nyström subsampling using approximate leverage scoresassociated with the empirical covariance with t = λ/(1 + b 2 ) and

m ≥ m 1 d λ/(1+b 2 ) log 8(1 + b 2 ) λδ .
Assume we perform the globalization scheme using the parameters given in Main theorem Main theorem 10 with precision ε, and that we compute approximate Newton steps using a sketching method as the one by [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF], in order to obtain f n,λ,m . With probability at least 1 -δ, it holds

R( f n,λ,m ) -R(f ρ ) ≤ C bias b λ + C var df λ ∨ q 2 n log 2 δ + ε, (3.91)
and the time complexity necessary to compute f n,λ,m is of order

(nm log n + mq 2 + q 3 ) R f ρ log R ∇F (0) λ + log 2 (1 + (λ -1 /R 2 )) m, q = O d λ/b 2 log b 2 λδ , (3.92)
The "real" result is slightly more complex. In particular, it defines the effective dimension for all the Hessians that are crossed, and relates the empirical effective dimensions with the ideal ones statistically. This theorem can be extended to obtain upper rates of convergence as the previous corollaries (see Theorem 7 by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]).

The main takeaway of this result is that we can compute a statistically optimal estimator in time roughly of order O(nd λ + d 3 λ ) (we hide logarithmic constants). In particular, in the well-specified case, Eq. (3.92) shows that the complexity is independent from the condition number of the problem. This is a very desirable property in classification problems where we are brought to consider very small values of λ, such as in the two examples in Fig. 3.1 taken from the work by [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], where we compare our method with a first order competitor, which has a worse dependance on the condition number.

Introduction

Regularized empirical risk minimization remains a cornerstone of statistics and supervised learning, from the early days of linear regression [START_REF] Hoerl | Ridge regression iterative estimation of the biasing parameter[END_REF] and neural networks [START_REF] Geman | Neural networks and the bias/variance dilemma[END_REF], then to spline smoothing [START_REF] Wahba | Spline Models for Observational Data[END_REF]) and more generally kernel-based methods [START_REF] Shawe | Kernel Methods for Pattern Analysis[END_REF]. While the regularization by the squared Euclidean norm is applied very widely, the statistical analysis of the resulting learning methods is still not complete.

The main goal of this paper is to provide a sharp non-asymptotic analysis of regularized empirical risk minimization (ERM), or more generally regularized M -estimation, that is estimators obtained as the unique solution of

min θ∈H 1 n n i=1 z i (θ) + λ 2 θ 2 , (4.1)
where H is a Hilbert space (possibily infinite-dimensional) and z (θ) is the convex loss associated with an observation z and the estimator θ ∈ H. We assume that the observations z i , i = 1, . . . , n are independent and identically distributed, and that the minimum of the associated unregularized expected risk L(θ) is attained at a certain θ ∈ H.

In this paper, we focus on dimension-independent results (thus ultimately extending the analysis in the finite-dimensional setting from [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF]. For this class of problems, two main classes of problems have been studied, depending on the regularity assumptions on the loss.

Convex Lipschitz-continuous losses (with respect to the parameter θ), such as for logistic regression or the support vector machine, lead to general non-asymptotic bounds for the excess risk of the form [START_REF] Sridharan | Fast rates for regularized objectives[END_REF]:

B 2 λn + λ θ 2 , (4.2)
where B is a uniform upper bound on the Lipschitz constant for all losses θ → z (θ). The bound above already has a form that takes into account two separate terms: a variance term B 2 /(λn) which depends on the sample size n but not on the optimal predictor θ , and a bias term λ θ 2 which depends on the optimal predictor but not on the sample size n. All our bounds will have this form but with smaller quantities (but asking fore more assumptions). Without further assumptions, in Eq. (4.2), λ is taken proportional to 1/ √ n, and we get the usual optimal slow rate in excess risk of O(1/ √ n) associated with such a general set-up (see, e.g., [START_REF] Cesa-Bianchi | On the complexity of learning with kernels[END_REF].

For the specific case of quadratic losses of the form

z (θ) = 1 2 (y -θ • Φ(x)) 2
, where z = (x, y), and y ∈ R and Φ(x) ∈ H, the situation is much richer. Without further assumptions, the same rate O(1/ √ n) is achieved, but stronger assumptions lead to faster rates [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF]. In particular, the decay of the eigenvalues of the Hessian E Φ(x) ⊗ Φ(x) (often called the capacity condition) leads to an improved variance term, while the finiteness of some bounds on θ for norms other than the plain Hilbertian norms θ (often called the source condition) leads to an improved bias term. Both of these assumptions lead to faster rates than O(1/ √ n)

for the excess risk, with the proper choice of the regularization parameter λ. For least-squares, these rates are then optimal and provide a better understanding of properties of the problem that influence the generalization capabilities of regularized ERM (see, e.g. [START_REF] Smale | Learning theory estimates via integral operators and their approximations[END_REF][START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF][START_REF] Steinwart | Optimal rates for regularized least squares regression[END_REF][START_REF] Fischer | Sobolev norm learning rates for regularized least-squares algorithm[END_REF][START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF].

INTRODUCTION 129

Our main goal in this paper is to bridge the gap between Lipschitz-continuous and quadratic losses by improving on slow rates for general classes of losses beyond least-squares. We first note that: (a) there has to be an extra regularity assumption because of lower bounds [START_REF] Cesa-Bianchi | On the complexity of learning with kernels[END_REF], and (b) asymptotically, we should obtain bounds that approach the local quadratic approximation of z (θ) around θ with the same optimal behavior as for plain least-squares.

Several frameworks are available for such an extension with extra assumptions on the losses, such as "exp-concavity" [START_REF] Koren | Fast rates for exp-concave empirical risk minimization[END_REF][START_REF] Nishant | Fast rates with high probability in exp-concave statistical learning[END_REF], strong convexity (Van de Geer, 2008) or a generalized notion of self-concordance [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF][START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF]. In this paper, we focus on self-concordance, which links the second and third order derivatives of the loss. This notion is quite general and corresponds to widely used losses in machine learning, and does not suffer from constants which can be exponential in problem parameters (e.g., θ ) when applied to generalized linear models like logistic regression. See Sec. 4.1.1 for a comparison to related work.

With this self-concordance assumption, we will show that our problem behaves like a quadratic problem corresponding to the local approximation around θ , in a totally non-asymptotic way, which is the core technical contribution of this paper. As we have already mentioned, this phenomenon is naturally expected in the asymptotic regime, but is hard to capture in the non-asymptotic setting without constants which explode exponentially with the problem parameters.

The paper is organized as follows: in Sec. 4.2, we present our main assumptions and informal results, as well as our bias-variance decomposition. In order to introduce precise results gradually, we start in Sec. 4.3 with a result similar to Eq. ( 4.2) for our set-up to show that we recover with a simple argument the result from [START_REF] Sridharan | Fast rates for regularized objectives[END_REF], which itself applies more generally. Then, in Sec. 4.4 we introduce the source condition allowing for a better control of the bias. Finally, in Sec. 4.5, we detail the capacity condition leading to an improved variance term, which, together with the improved bias leads to fast rates (which are optimal for least-squares).

Related work

Fast rates for empirical risk minimization. Rates faster than O(1/ √ n) can be obtained with a variety of added assumptions, such as some form of strong convexity [START_REF] Sridharan | Fast rates for regularized objectives[END_REF][START_REF] Boucheron | A high-dimensional Wilks phenomenon[END_REF], noise conditions for classification [START_REF] Steinwart | Fast rates for support vector machines using gaussian kernels[END_REF], or extra conditions on the loss, such as self-concordance [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF] or exp-concavity [START_REF] Koren | Fast rates for exp-concave empirical risk minimization[END_REF][START_REF] Nishant | Fast rates with high probability in exp-concave statistical learning[END_REF], whose partial goal is to avoid exponential constants. Note that Bach (2010) already considers logistic regression with Hilbert spaces, but only for well-specified models and a fixed design, and without the sharp and simpler results that we obtain in this paper.

Avoiding exponential constants for logistic regression. The problem of exponential constants (i.e., leading factors in the rates scaling as e RD where D is the radius of the optimal predictor, and R the radius of the design) is long known. In fact, [START_REF] Hazan | Logistic regression: tight bounds for stochastic and online optimization[END_REF] showed a lower bound, explicitly constructing an adversarial distribution (i.e., an ill-specified model) for which the problem manifests in the finite-sample regime with n = O(e RD ). Various attempts to address this problem are found in the literature. For example, Ostrovskii and Bach (2018, App. C) prove the optimal d/n rate in the non-regularized d-dimensional setting but, multiplied with the curvature parameter ρ which is at worst exponential but is shown to grow at most as (RD) 3/2 in the case of Gaussian design. Another approach is due to [START_REF] Foster | Logistic regression: The importance of being improper[END_REF]: they establish "1-mixability" of the logistic loss, then apply Vovk's aggregating algorithm in the online setting, and then proceed via online-to-batch conversion. While this result allows to obtain the fast O(d/n) rate (and its counterparts in the nonparametric setting) without exponential constants, the resulting algorithm is improper (i.e., the canonical parameter η = Φ(x) • θ , see below, is estimated by a non-linear functional of Φ(x)).

A closely related approach is to use the notion of exp-concavity instead of mixability [START_REF] Rakhlin | Online nonparametric regression with general loss functions[END_REF][START_REF] Koren | Fast rates for exp-concave empirical risk minimization[END_REF][START_REF] Nishant | Fast rates with high probability in exp-concave statistical learning[END_REF]. The two close notions are summarized in the so-called central condition (due to Van Erven et al. ( 2015)) which fully characterizes when the fast O(d/n) rates (up to log factors and in high probability) are available for improper algorithms. However, when proper learning algorithms are concerned, this analysis requires η-mixability (or η-exp-concavity) of the overall loss z (θ) for which the η parameter scales with the radius of the set of predictors. This scaling is exponential for the logistic loss, leading to exponential constants.

Main Assumptions and Results

Let Z be a Polish space and Z be a random variable on Z with distribution ρ. Let H be a separable (non-necessarily finite-dimensional) Hilbert space, with norm • , and let : Z ×H → R be a loss function, we denote by z (•) the function (z, •). Our goal is to minimize the expected risk with respect to θ ∈ H:

inf θ∈H L(θ) = E [ Z (θ)] .
Given (z i ) n i=1 ∈ Z n , we will consider the following estimator based on regularized empirical risk minimization given λ > 0 (note that the minimizer is unique in this case):

θ λ = arg min θ∈H 1 n n i=1 z i (θ) + λ 2 θ 2 ,
where we assume the following.

Assumption 4.1 (i.i.d. data). The samples (z i ) 1≤i≤n are independently and identically distributed according to ρ.

The goal of this work is to provide upper bounds in high probability for the so-called excess risk

L( θ λ ) -inf θ∈H L(θ),
and thus to provide a general framework to measure the quality of the estimator θ λ . Algorithms for obtaining such estimators have been extensively studied, in both finite-dimensional regimes, where a direct optimization over θ is performed, typically by gradient descent or stochastic versions thereof (see, e.g., [START_REF] Bottou | The trade-offs of large scale learning[END_REF][START_REF] Shalev-Shwartz | Pegasos: Primal estimated sub-gradient solver for SVM[END_REF] and infinite-dimensional regimes, where kernel-based methods are traditionally used (see, e.g., [START_REF] Sathiya Keerthi | A fast dual algorithm for kernel logistic regression[END_REF][START_REF] Gerfo | Spectral algorithms for supervised learning[END_REF][START_REF] Dieuleveut | Nonparametric stochastic approximation with large step-sizes[END_REF][START_REF] Tu | Large scale kernel learning using block coordinate descent[END_REF]Rudi, Carratino, and Rosasco, 2017, and references therein).

Example 4.1 (Supervised learning). Although formulated as a general M -estimation problem (see, e.g., [START_REF] Lehmann | Theory of Point Estimation[END_REF], our main motivation comes from supervised learning, with Z = X × Y where X is the data space and Y the target space. We will consider, as examples, losses with both real-valued outputs but also the multivariate case. For learning real-valued outputs, consider we have a bounded representation of the input space Φ : X → H (potentially implicit when using kernel-based methods, [START_REF] Aronszajn | Theory of reproducing kernels[END_REF]. We will provide bounds for the following losses.

MAIN ASSUMPTIONS AND RESULTS

131

• The square loss z (θ) = 1 2 (yθ • Φ(x)) 2 , which is not Lipschitz-continuous.

• The Huber losses z [START_REF] Frank | Robust statistics: the approach based on influence functions[END_REF], which are Lipschitz-continuous.

(θ) = ψ(y -θ • Φ(x)) where ψ(t) = √ 1 + t 2 -1 or ψ(t) = log e t +e -t 2 (
• The logistic loss z (θ) = log(1 + e -yθ•Φ(x) ) commonly used in binary classification where y ∈ {-1, 1}, which is Lipschitz-continuous.

Our framework goes beyond real-valued outputs, and can be applied to all generalized linear models (GLM) [START_REF] Mccullagh | Generalized Linear Models[END_REF], including softmax regression: we consider a representation function Φ : X × Y → H and an a priori measure µ on Y. The loss we consider in this case is

z (θ) = -θ • Φ(x, y) + log Y exp (θ • Φ(x, y )) dµ(y ),
which corresponds to the negative conditional log-likelihood when modelling y given x by the distribution p(y|x, θ)

∼ exp(θ•Φ(x,y))
Y exp(θ•Φ(x,y ))dµ(y ) dµ(y). Our framework applies to all of these generalized linear models with almost surely bounded features Φ(x, y), such as conditional random fields [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting and labeling sequence data[END_REF].

We can now introduce the main technical assumption on the loss . Assumption 4.2 (Generalized self-concordance). For any z ∈ Z, the function z (•) is convex and three times differentiable. Moreover, there exists a set ϕ(z) ⊂ H such that it holds :

∀θ ∈ H, ∀h, k ∈ H, ∇ 3 z (θ)[k, h, h] ≤ sup g∈ϕ(z) |k • g| ∇ 2 z (θ)[h, h].
This is a generalization of the assumptions introduced by Bach ( 2010), by allowing a varying term sup g∈ϕ(z) |k • g| instead of a uniform bound proportional to k . This is crucial for the fast rates we want to show.

Example 4.2 (Checking assumptions). For the losses in Example 4.1, this condition is satisfied with the following corresponding set-function ϕ.

• For the square loss [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF]. For the logistic loss z (θ) = log(1 + e -yθ•Φ(x) ), we have ϕ(z) = {yΦ(x)} (here, ϕ(z) is reduced to a point).

z (θ) = 1 2 (y -θ • Φ(x)) 2 , ϕ(z) = {0}. • For the Huber losses z (θ) = ψ(y -θ • Φ(x)), if ψ(t) = √ 1 + t 2 -1, then ϕ(z) = {3Φ(x)} and if ψ(t) = log e t +e -t 2 , then ϕ(z) = {2Φ(x)}
• For generalized linear models, ∇ 3 z (θ) is a third-order cumulant, and thus

∇ 3 z (θ)[k, h, h] ≤ E p(y|x,θ) |k • Φ(x, y) -k • E p(y |x,θ) Φ(x, y )| • |h • Φ(x, y) -h • E p(y |x,θ) Φ(x, y )| 2 ≤ 2 sup y∈Y |k • Φ(x, y)| ∇ 2 z (θ)[h, h]. Therefore ϕ(z) = {2Φ(x, y
), y ∈ Y} (which is not a singleton). Moreover we require the following two technical assumptions to guarantee that L(θ) and its first and second derivatives are well defined for any θ ∈ H.

Assumption 4.3 (Boundedness).

There exists R ≥ 0 such that sup g∈ϕ(Z) g ≤ R almost surely. The assumptions above are usually easy to check in practice. In particular, if the support of ρ is bounded, the mappings z → z (0), ∇ z (0), Tr(∇ 2 z (0)) are continuous, and ϕ is uniformly bounded on bounded sets, then they hold. The main regularity assumption we make on our statistical problems follows.

Assumption 4.5 (Existence of a minimizer). There exists θ ∈ H such that L(θ ) = inf θ∈H L(θ).

CHAPTER 4. BEYOND LEAST SQUARES While Assumption 4.3 is standard in the analysis of such models [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF][START_REF] Sridharan | Fast rates for regularized objectives[END_REF][START_REF] Steinwart | Optimal rates for regularized least squares regression[END_REF][START_REF] Bach | Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression[END_REF], Assumption 4.5 imposes that the model is "well-specified", that is, for supervised learning situations from Example 4.1, we have chosen a rich enough representation Φ. It is possible to study the non-realizable case in our setting by requiring additional technical assumptions (see [START_REF] Steinwart | Optimal rates for regularized least squares regression[END_REF] or discussion after (4.6)), but this is out of scope of this paper. Note that our well-specified assumption (for logistic regression for simplicity of arguments) is weaker than requiring f (x) = E [Y |X] being equal to θ • Φ(x). We can now introduce the main definitions allowing our bias-variance decomposition.

Definition 4.1 (Hessian, Bias, Degrees of freedom). Let L λ (θ) = L(θ) + λ 2 θ 2 ; define the expected Hessian H(θ), the regularized Hessian H λ (θ), the bias Bias λ and the degrees of freedom df λ as:

H(θ) = E ∇ 2 Z (θ) , and H λ (θ) = H(θ) + λI, (4.3) Bias λ = H λ (θ ) -1/2 ∇L λ (θ ) , (4.4
)

df λ = E H λ (θ ) -1/2 ∇ Z (θ ) 2 . (4.5)
Note that the bias and degrees of freedom only depend on the optimum θ ∈ H and not on the minimizer θ λ of the regularized expected risk. Moreover, the degrees of freedom df λ correspond to the usual Fisher information term commonly seen in the asymptotic analysis of M -estimation ( [START_REF] Van Der | Asymptotic Statistics[END_REF][START_REF] Lehmann | Theory of Point Estimation[END_REF], and correspond to the usual quantities introduced in the analysis of least-squares [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF]. Indeed, in the least-squares case, we recover exactly

Bias λ = λ C -1/2 λ θ and df λ = Tr(CC -1 λ ), where C is the covariance operator C = E [Φ(x) ⊗ Φ(x)] and C λ = C + λI.
Our results will rely on the quadratic approximation of the losses around θ . Borrowing tools from the analysis of Newton's method [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF], this will only be possible in the vicinity of θ . The proper notion of vicinity is the so-called radius of the Dikin ellipsoid, which we define as follows:

r λ (θ) such that 1/r λ (θ) = sup z∈supp(ρ) sup g∈ϕ(z) H λ -1/2 (θ)g . (4.6)
Our most refined bounds will depend whether the bias term is small enough compared to r λ (θ ). We believe that in the non realizable setting, the results we obtain would still hold when the bias term is smaller than the Dikin radius, although one would have to modify the definitions to incorporate the fact that θ is not in H. The following informal result summarizes all of our results.

Theorem 4.1 (General bound, informal). Let n ∈ N, δ ∈ (0, 1/2], λ > 0. Under Assumptions 4.1 to 4.5, whenever n ≥ C 0 R 2 df λ log 2 δ λ ,
then with probability at least 1 -2δ, it holds

L( θ λ ) -L(θ ) ≤ C bias Bias 2 λ + C var df λ log 2 δ n ,
where C 0 , C bias and C var are either universal or depend only on R θ . Theorem 4.4 andCor. 4.3 Table 4.1: Summary of convergence rates, without constants except λ, for source condition (Asm. 4.6): θ ∈ Im(H(θ ) r ), r ∈ (0, 1/2], capacity condition (Asm. 4.7):

df λ = O(λ -1/α ), α ≥ 1 .
This mimics a usual bias-variance decomposition, with a bias term Bias 2 λ and a variance term proportional to df λ /n. In particular in the rest of the paper we quantify the constants and the rates under various regularity assumptions, and specify the good choices of the regularization parameter λ. In Table 4.1, we summarize the different assumptions and corresponding rates.

Slow convergence rates

Here we bound the quantity of interest without any regularity assumption (e.g., source of capacity condition) beyond some boundedness assumptions on the learning problem. We consider the various bounds on the derivatives of the loss :

B 1 (θ) = sup z∈supp(ρ) ∇ z (θ) , B 2 (θ) = sup z∈supp(ρ) Tr(∇ 2 z (θ)), B 1 = sup θ ≤ θ B 1 (θ), B 2 = sup θ ≤ θ B 2 (θ).
Example 4.3 (Bounded derivatives). In all the losses considered above, assume the feature representation (Φ(x) for the Huber losses and the square loss, yΦ(x) for the logistic loss, and Φ(x, y) for GLMs) is bounded by R. Then the losses considered above apart from the square loss are Lipschitz-continuous and B 1 is uniformly bounded by R. For these losses, B 2 is also uniformly bounded by R2 . Using Example 4.2, one can take R to be equal to a constant times R (1/2 and 1/3 for the respective Huber losses, 1 for logistic regression and 1/2 for canonical GLMs). For the square loss (where R = 0 because the third-order derivative is zero), B 2 ≤ R2 and B 1 ≤ R y ∞ + R2 θ , where y ∞ is an almost sure bound on the output y.

Theorem 4.2 (Basic result). Let n ∈ N and 0 < λ ≤ B 2 . Let δ ∈ (0, 1/2]. If n ≥ 512 θ 2 R 2 ∨ 1 log 2 δ , n ≥ 24 B 2 λ log 8B 2 λδ , n ≥ 256 R 2 B 2 1 λ 2 log 2 δ ,
then with probability at least 1 -2δ,

L( θ λ ) -L(θ ) ≤ 84 B 2 1 λn log 2 δ + 2λ θ 2 . (4.7)
This result shown in Sec. 4.C .3 as a consequence of Theorem 4.6 (also see the proof sketch in Sec. 4.6) matches the one obtained with Lipschitz-continuous losses [START_REF] Sridharan | Fast rates for regularized objectives[END_REF] and the one for least-squares when assuming the existence of θ [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF]. The following corollary (proved as Theorem 4.8 in Sec. 4.E ) gives the bound optimized in λ, with explicit rates.

Corollary 4.1 (Basic Rates). Let δ ∈ (0, 1/2]. Under Assumptions 4.1 to 4.5, when n ≥ N, λ = C 0 log(2/δ)/n, then with probability at least 1 -2δ,

L( θ λ ) -L(θ ) ≤ C 1 n -1/2 log 1/2 2 δ .
with

C 0 = 16B 1 max(1, R), C 1 = 48B 1 max(1, R) max(1, θ 2
) and with N defined in Eq. (4.41) and satisfying N=O(poly(B 1 , B 2 , R θ )) where poly denotes a polynomial function of the inputs.

Both bias and variance terms are of order O(1/ √ n) and we recover up to constants terms the result of [START_REF] Sridharan | Fast rates for regularized objectives[END_REF]. In the next section, we will improve both bias and variance terms to obtain faster rates.

Faster Rates with Source Conditions

Here we provide a more refined bound, where we introduce a source condition on θ allowing to improve the bias term and to achieve learning rates as fast as O(n -2/3 ). We first define the localized versions of B 1 , B 2 :

B 1 = B 1 (θ ), B 2 = B 2 (θ ),
and recall the definition of the bias

Bias λ = H λ (θ ) -1/2 ∇L λ (θ ) . (4.8)
Note that since θ is the minimizer of L, we have ∇L(θ ) = 0, so that ∇L λ (θ ) = ∇L(θ ) + λθ = λθ , and Bias λ = λ H λ (θ ) -1/2 θ . This characterization is always bounded by λ θ 2 , but allows a finer control of the regularity of θ , leading to improved rates compared to Sec. 4.3.

Note that in the least-squares case, we recover exactly the bias of ridge regression

Bias λ = λ C -1/2 λ θ , where C is the covariance operator C = E [Φ(x) ⊗ Φ(x)].
Using self-concordance, we will relate quantities at θ to quantities at θ λ using:

t λ = sup z∈supp(ρ) sup g∈ϕ(z) |(θ λ -θ ) • g|.
The following theorem, proved in Sec. 4.D .4, relates Bias λ to the excess risk.

Theorem 4.3 (Decomposition with refined bias).

Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Whenever n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , n ≥ 2 (B 1 R) 2 λ 2 log 2 δ ,
then with probability at least 1 -2δ, it holds

L( θ λ ) -L(θ ) ≤ C bias Bias 2 λ + C var (B 1 ) 2 λn log 2 δ , (4.9)
where

1 ≤ e t λ /2 , 1 ≤ 2304e 4t λ (1/2 ∨ R θ ), 2 ≤ 256e 2t λ , C bias ≤ 6e 2t λ , C var ≤ 256e 3t λ .
It turns out that the radius of the Dikin ellipsoid r λ (θ ) defined in Eq. (4.6) provides the sufficient control over the constants above: when the bias is of the same order of the radius of the Dikin ellipsoid, the quantities C bias , C var , 1 , 2 become universal constants instead of depending exponentially on R θ , as shown by the lemma below, proved in Lemma 4.4 in Sec. 4.D .

Lemma 4.1. When Bias λ ≤ r λ (θ ) 2 then t λ ≤ log 2 else t λ ≤ 2R θ .
Interestingly, regularity of θ , like the source condition below, can induce this effect, allowing a better dependence on λ for the bias term.

Assumption 4.6 (Source condition). There exists r ∈ (0, 1/2] and v ∈ H such that θ = H(θ ) r v.

In particular we denote by L := v . Assumption 4.6 is commonly made in least-squares regression [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF][START_REF] Steinwart | Optimal rates for regularized least squares regression[END_REF][START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] and is equivalent to requiring that, when expressing θ with respect to the eigenbasis 4.5. RATES FOR SOURCE AND CAPACITY 135 of H(θ ), i.e., θ = j∈N α j u j , where λ j , u j is the eigendecomposition of H(θ ), and α j = θ • u j , then α j decays as λ r j . In particular, with this assumption, defining

β j = v • u j , Bias 2 λ = λ 2 j α 2 j λ j + λ = λ 2 j λ 2r j β 2 j λ j + λ ≤ λ 2 sup j λ 2r j λ j + λ j β 2 j ≤ λ 1+2r v 2 .
Note moreover that H(θ ) B 2 C, meaning that the usual sufficient conditions leading to the source conditions for least-squares also apply here. For example, for logistic regression, if the log-odds ratio is smooth enough, then it is in H. So, when H corresponds to a Sobolev space of smoothness m and the marginal of ρ on the input space is a density bounded away from 0 and infinity with bounded support, then the source condition corresponds essentially to requiring θ to be (1 + 2r)m-times differentiable (see discussion after Thm. 9 of Steinwart, Hush, and Scovel, 2009, for more details). A precise example can be found in Sec. 4.1 of Pillaud-Vivien, [START_REF] Pillaud-Vivien | Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes[END_REF].

In conclusion, the effect of additional regularity for θ as Assumption 4.6, has two beneficial effects: (a) on one side it allows to obtain faster rates as shown in the next corollary, (b) as mentioned before, somewhat surprisingly, it reduces the constants to universal, since it allows the bias to go to zero faster than the Dikin radius (indeed, the squared radius r 2 λ (θ ) is always larger than λ/R 2 , which is strictly larger than λ 1+2r v 2 if r > 0 and λ small enough). This is why we do not the get exponential constants imposed by [START_REF] Hazan | Logistic regression: tight bounds for stochastic and online optimization[END_REF].

Corollary 4.2 (Rates with source condition). Let δ ∈ (0, 1/2]. Under Assumptions 4.1 to 4.5 and Assumption 4.6, whenever n ≥ N and λ = (C 0 /n) 1/(2+2r) , then with probability at least 1 -2δ,

L( θ λ ) -L(θ ) ≤ C 1 n -1+2r 2+2r log 2 δ , with C 0 = 256 (B 1 /L) 2 , C 1 = 8 (256) γ ((B 1 ) γ L 1-γ ) 2 , γ = 1+2r
2+2r and with N defined in Eq. (4.48) and satisfying N = O(poly(B 1 , B 2 , L, R, log(1/δ))).

The corollary above, derived in Sec. 4.F , is obtained by minimizing in λ the r.h.s. side of Eq. (4.9) in Theorem 4.3, and considering that when θ satisfies the source condition, then Bias λ ≤ λ 1+2r L, while the variance is still of the form 1/(λn). When r is close to 0, the rate 1/ √ n is recovered.

When instead the target function is more regular, implying r = 1/2, a rate of n -2/3 is achieved. Two considerations are in order: (a) the obtained rate is the same as least-squares and minimax optimal [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF][START_REF] Steinwart | Optimal rates for regularized least squares regression[END_REF][START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF], (b) the fact that regularized ERM is adaptive to the regularity of the function up to r = 1/2 is a byproduct of Tikhonov regularization as already shown for the least-squares case by [START_REF] Gerfo | Spectral algorithms for supervised learning[END_REF]. Using different regularization techniques may remove the limit r = 1/2.

Fast Rates with both Source and Capacity Conditions

In this section, we consider improved results with a finer control of the effective dimension df λ (often called degrees of freedom), which, together with the source condition allows to achieve rates as fast as 1/n:

df λ = E H λ (θ ) -1/2 ∇ Z (θ ) 2 ,
As mentioned earlier this definition of df λ corresponds to the usual asymptotic term in Mestimation. Moreover, in the case of least-squares, it corresponds to the standard notion of effective dimension df λ = Tr(CC -1 λ ) [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF]Blanchard and Mücke, CHAPTER 4. BEYOND LEAST SQUARES 2018). Note that by definition, we always have df λ B 1 2 /λ, but we can have in general a much finer control. For example, for least-squares, df λ = O(λ -1/α ) if the eigenvalues of the covariance operator C decay as λ j (C) = O(j -α ), for α ≥ 1. Moreover note that since C is trace-class, by Asm. 4.3, the eigenvalues form a summable sequence and so C satisfies λ j (C) = O(j -α ) with α always larger than 1.

Example 4.4 (Generalized linear models). For generalized linear models, an extra assumption makes the degrees of freedom particularly simple: if the probabilistic model is well-specified, that is, there exists θ such that almost surely, p(y|x

) = p(y|x, θ ) = exp(θ •Φ(x,y))
Y exp(θ •Φ(x,y ))dµ(y ) , then from the usual Bartlett identities [START_REF] Bartlett | Approximate confidence intervals[END_REF] relating the expected squared derivatives and Hessians, we have

E [∇ z (θ ) ⊗ ∇ z (θ )] = H(θ ), leading to df λ = Tr(H λ (θ ) -1 H(θ )).
As we have seen in the previous example there are interesting problems for which df λ = Tr(H(θ ) + λI) -1 H(θ )). Since we have H(θ )

B 2 C, df λ still enjoys a polynomial decay depending on the eigenvalue decay of C as observed for least-squares. In the finite-dimensional setting where H is of dimension d, note that in this case, df λ is always bounded by d. Now we are ready to state our result in the most general form, proved in Sec. 4.D .4.

Theorem 4.4 (General bound). Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Whenever n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , n ≥ 2 df λ ∨ (Q ) 2 r λ (θ ) 2 log 2 δ , with (Q ) 2 = B 1 2 /B 2 , then with probability at least 1 -2δ, it holds L( θ λ ) -L(θ ) ≤ C bias Bias 2 λ + C var df λ ∨ (Q ) 2 n log 2 δ , (4.10)
where, C bias , C var , 1 ≤ 414, 1 , 2 ≤ 5184 when Bias λ ≤ r λ (θ )/2;

otherwise C bias , C var , 1 ≤ 256e 6R θ , 1 , 2 ≤ 2304(1 + R θ ) 2 e 8R θ .
As shown in the theorem above, the variance term depends on df λ /n, implying that, when df λ has a better dependence in λ than 1/λ, it is possible to achieve faster rates. We quantify this with the following assumption.

Assumption 4.7 (Capacity condition). There exists α > 0 and Q ≥ 0 such that df λ ≤ Qλ -1/α . Assumption 4.7 is standard in the context of least-squares, [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF] and in many interesting settings is implied by the eigenvalue decay order of H(θ ), or C as discussed above. In the following corollary we quantify the effect of df λ in the learning rates.

Corollary 4.3. Let δ ∈ (0, 1/2]. Under Assumptions 4.1 to 4.5, Assumption 4.6 and Assumption 4.7, when n ≥ N and λ = (C 0 /n) α/(1+α(1+2r)) , then with probability at least 1 -2δ,

L( θ λ ) -L(θ ) ≤ C 1 n - α(1+2r) 1+α(1+2r) log 2 δ , with C 0 = 256(Q/L) 2 , C 1 = 8(256) γ (Q γ L 1-γ ) 2 , γ = α(1+2r)
1+α(1+2r) and N defined in Eq. (4.48) and satisfying

N = O(poly(B 1 , B 2 , L, Q, R, log(1/δ))).
The result above is derived in Cor. 4.4 in Sec. 4.F and is obtained by bounding Bias λ with λ 1+2r L due to the source condition, and df λ with λ -1/α due to the capacity condition and then optimizing the r.h.s. of Eq. (4.10) in λ. Note that (a) the learning rate under the considered assumptions is the same as least-squares and minimax optimal [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF], and (b) when α = 1 the same rate of Cor. 4.2 is achieved, which can be as fast as n -2/3 , otherwise, when α 1, we achieve a learning rate in the order of 1/n, for λ = n -1/(1+2r) .

4.6. SKETCH OF THE PROOF 137

Sketch of the proof

In this section we will use the notation v A := A 1/2 v , with v ∈ H and A a bounded positive semi-definite operator on H. Here we prove that the excess risk decomposes using the bias term Bias λ defined in Eq. (4.8) and a variance term V λ , where V λ is defined as

V λ := ∇ L λ (θ λ ) H -1 λ (θ λ ) , with L λ (•) = 1 n n i=1 z i (•) + λ 2 • 2 ,
which in turn is a random variable that concentrate in high probability to df λ /n.

Required tools. To proceed with the proof we need two main tools. The first is a result on the equivalence of norms of the empirical Hessian

H λ (θ) = ∇ 2 L λ (θ) w.r.t. the true Hessian H λ (θ) = ∇ 2 L λ (θ) for λ > 0 and θ ∈ H.
The result is proven in Lemma 4.6 of Sec. 4.D .3, using Bernstein inequalities for Hermitian operators [START_REF] Tropp | User-friendly tail bounds for sums of random matrices[END_REF], and essentially states that for δ ∈ (0, 1], whenever n ≥ 24B 2 (θ) λ log 8B 2 (θ) λδ , then with probability 1 -δ, it holds

• H λ (θ) ≤ 2 • H λ (θ) , • H -1 λ (θ) ≤ 2 • H -1 λ (θ) . (4.11)
The second result is about localization properties induced by generalized self-concordance on the risk. We express the result with respect to a generic probability µ (we will use it with µ = ρ and

µ = 1 n n i=1 δ z i ).
Let µ be a probability distribution with support contained in the support of ρ.

Denote by L µ (θ) the risk L µ (θ) = E z∼µ [ z (θ)] and by L µ,λ (θ) = L µ (θ) + λ 2 θ 2 (then L µ,λ = L λ when µ = ρ, or L λ when µ = 1 n n i=1 δ z i ).
Proposition 4.1. Under Assumptions 4.2 to 4.4, the following holds: (a) L µ,λ (θ), ∇L µ,λ (θ), H µ,λ (θ) are defined for all θ ∈ H, λ ≥ 0, (b) for all λ > 0, there exists a unique θ µ,λ ∈ H minimizing L µ,λ over H, and (c) for all λ > 0 and θ ∈ H, H µ,λ (θ) e t(θ-θ µ,λ ) H µ,λ (θ µ,λ ), (4.12)

L µ,λ (θ) -L µ,λ (θ µ,λ ) ≤ ψ(t(θ -θ µ,λ )) θ -θ µ,λ 2 H µ,λ (θ µ,λ ) , (4.13) φ(t(θ -θ µ,λ )) θ -θ µ,λ H µ,λ (θ) ≤ ∇L µ,λ (θ) H -1 µ,λ (θ) , (4.14)
(d) Eqs. (4.12) and (4.13) hold also for λ = 0, provided that θ µ,0 exists. Here. φ(t) = (1 -e -t )/t and ψ(t) = (e tt -1)/t 2 .

The result above is proved in Sec. 4.B .1 and is essentially an extension of results by [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF] applied to L µ,λ under Assumptions 4.2 to 4.4.

Sketch of the proof. Now we are ready to decompose the excess risk using our bias and variance terms. In particular we will sketch the decomposition without studying the terms that lead to constants terms. For the complete proof of the decomposition see Theorem 4.7 in Sec. 4.D .1. Since θ exists by Assumption 4.5, using Eq. (4.13), applied with µ = ρ and λ = 0, we have

L(θ) -L(θ ) ≤ ψ(t(θ -θ )) θ -θ 2 H(θ ) for any θ ∈ H. By setting θ = θ λ , we obtain L( θ λ ) -L(θ ) ≤ ψ(t( θ λ -θ )) θ λ -θ 2 H(θ ) .
The term ψ(t( θ λ -θ )) will become a constant. For the sake of simplicity, in this sketch of proof we will not deal with it nor with other terms of the form t(•) leading to constants. On the other hand, the term θ λ -θ 2 H(θ ) will yield our bias and variance terms. Using the fact that H(θ ) H(θ ) + λI =: H λ (θ ), by adding and subtracting θ λ , we have

CHAPTER 4. BEYOND LEAST SQUARES θ λ -θ H(θ ) ≤ θ λ -θ H λ (θ ) ≤ θ λ -θ H λ (θ ) + θ λ -θ λ H λ (θ ) , so L( θ λ ) -L(θ ) ≤ const. × ( θ λ -θ 2 H λ (θ ) + θ λ -θ λ H λ (θ ) ) 2
. By applying Eq. (4.12) with µ = ρ and θ = θ , we have H λ (θ ) e t λ H λ (θ λ ) and so we further bound

θ λ -θ λ H λ (θ ) with e t λ /2 θ λ -θ λ H λ (θ λ ) obtaining L( θ λ ) -L(θ ) ≤ const. × ( θ λ -θ H λ (θ ) + e t λ /2 θ λ -θ λ H λ (θ λ ) ) 2 .
The term θ λ -θ H λ (θ ) will lead to the bias terms, while the term θ λ -θ λ H λ (θ λ ) will lead to the variance term.

Bounding the bias terms. Recall the definition of bias Bias λ = ∇L λ (θ ) H -1 λ (θ ) and of the constant t λ := t(θθ λ ). We bound θθ λ H λ (θ ) by applying Eq. (4.14) with µ = ρ and

θ = θ θ -θ λ H λ (θ ) ≤ 1/φ(t λ ) ∇L λ (θ ) H -1 λ (θ ) = 1/φ(t λ ) Bias λ .
Bounding the variance terms. To bound the term θ λ -θ λ H λ (θ λ ) , we assume n large enough to apply Eq. (4.11) in high probability. Thus, we obtain

θ λ -θ λ H λ (θ λ ) ≤ 2 θ λ -θ λ H λ (θ λ ) .
Applying Eq. (4.14

) with µ = 1 n n i=1 δ z i and θ = θ λ , since L µ,λ = L λ for the given choice of µ, θ λ -θ λ H λ (θ λ ) ≤ ∇ L λ (θ λ ) H -1 λ (θ λ ) / φ(t(θ λ -θ λ )
), and applying Eq. (4.11) in high probability again, we obtain

∇ L λ (θ λ ) H -1 λ (θ λ ) ≤ 2 ∇ L λ (θ λ ) H λ -1 (θ λ ) .
Bias-variance decomposition. A technical part of the proof relates

∇ L λ (θ λ ) H λ -1 (θ λ ) with ∇ L λ (θ ) H λ -1 (θ ) =: V λ ,
by many applications of Prop. 4.1. Here we assume it is done, obtaining

L( θ λ ) -L(θ ) ≤ const. × (Bias 2 λ + V 2 λ ). From V λ to df λ /n. By construction, ∇ L λ (θ λ ) = 1 n n i=1 ζ i , with ζ i := ∇ z i (θ λ ) + λθ λ . Moreover since the z i 's are i.i.d. samples from ρ, E [ζ i ] = ∇L λ (θ λ ). Finally since θ λ is the minimizer of L λ , ∇L λ (θ λ ) = 0. Thus ∇ L λ (θ λ )
is the average of n i.i.d. zero-mean random vectors, and so the variance of V λ is exactly

E V 2 λ = 1 n E H -1/2 λ (θ )∇ Z (θ ) 2 = df λ n .
Finally, by using Bernstein inequality for random vectors (e.g., Yurinsky, 1995, Thm. 3.3.4), we bound V λ roughly with df λ log(2/δ)/n in high probability.

Conclusion

In this paper we have presented non-asymptotic bounds with faster rates than O(1/ √ n), for regularized empirical risk minimization with self-concordant losses such as the logistic loss. It would be interesting to extend our work to algorithms used to minimize the empirical risk, in particular stochastic gradient descent or Newton's method.

4.A Setting, definitions, assumptions

Let Z be a Polish space and Z a random variable on Z whith law ρ. Let H be a separable (non-necessarily finite) Hilbert space and let : Z × H → R be a loss function; we denote by

z (•) the function (z, •). Our goal is to solve inf θ∈H L(θ), with L(θ) = E [ Z (θ)] .
Given (z i ) n i=1 we will consider the following estimator

θ λ = arg min θ∈H L λ (θ), with L λ (θ) := 1 n n i=1 z i (θ) + λ 2 θ 2 .
The goal of this work is to give upper bounds in high probability to the so called excess risk

L( θ λ ) -inf θ∈H L(θ).
In the rest of this introduction we will introduce the basic assumptions required to make θ λ and the excess risk well defined, and we will introduce basic objects that are needed for the proofs.

CHAPTER 4. BEYOND LEAST SQUARES

First we introduce some notation we will use in the rest of the appendix: let λ ≥ 0, θ ∈ H and A be a bounded positive semidefinite Hermitian operator on H, we denote by I, the identity operator and

f A := A 1/2 f , (4.15) A λ := A + λI, (4.16) λ z (θ) := z (θ) + λ 2 θ 2 , (4.17) L λ (θ) := L(θ) + λ 2 θ 2 . (4.18)
Now we recall the assumptions we require on the loss function , ρ, (z i ) 1≤i≤n .

Assumption 4.1 (i.i.d. data). The samples (z i ) 1≤i≤n are independently and identically distributed according to ρ.

Assumption 4.8 (Generalized self-concordance). The mapping z → z (θ) is measurable for all θ ∈ H and for any z ∈ Z, the function z is convex and three times differentiable. Moreover, there exists a set ϕ(z) ⊂ H such that it holds:

∀θ ∈ H, ∀h, k ∈ H, ∇ 3 z (θ)[k, h, h] ≤ sup g∈ϕ(z) |k • g| ∇ 2 z (θ)[h, h].
Assumption 4.3 (Boundedness). There exists R ≥ 0 such that sup g∈ϕ(Z) g ≤ R almost surely. Introduce the following definitions. Definition 4.2. Let λ > 0, θ ∈ H. We introduce

B 1 (θ) = sup z∈supp(ρ) ∇ z (θ) , B 2 (θ) = sup z∈supp(ρ) Tr ∇ 2 z (θ) . (4.19) H(θ) = E ∇ 2 Z (θ) , H λ (θ) = H(θ) + λI. (4.20) θ λ = arg min θ∈H L λ (θ). ( 4
(θ), B 2 (θ), L(θ), ∇L(θ), H(θ), θ λ exist for any θ ∈ H, λ > 0. Moreover ∇L = E [∇ Z (θ)], H(θ) = ∇ 2 L(θ) and H(θ) is trace class.
Proof. We start by proving, using the assumptions, that B 2 , B 1 and θ → sup z∈supp(ρ) | z (θ)| are all locally bounded (see Lemmas 4.11 to 4.13). This allows us to show that z (θ), ∇ z (θ) and Tr(∇ 2 z (θ)) are uniformly integrable on any ball of finite radius. The fact that θ λ exists is due to the strong convexity of the function L λ .

Proposition 4.3. Under Assumptions 4.1, 4.4 and 4.8, when λ > 0, θ λ exists and is unique.

Proof. By Assumption 4.1 we know that z 1 , . . . , z n are in the support of ρ. Thus, by Assumption 4.4, 1 n n i=1 z i is finite valued in 0. Since 1 n n i=1 z i is convex three times differentiable as a sum of such functions, it is real-valued on H and hence L λ is real-valued on H; by strong convexity, θ λ exists and is unique.

4.B . GENERALIZED SELF-CONCORDANT LOSSES 141

Recall that we also make the following regularity assumption. Assumption 4.5 (Existence of a minimizer). There exists θ ∈ H such that L(θ ) = inf θ∈H L(θ).

Finally we conclude with the following definitions that will be used later.

Bias λ = ∇L λ (θ ) H λ -1 (θ ) , (4.22
)

Var λ = H 1/2 λ (θ λ ) H -1/2 λ (θ λ ) 2 ∇ L λ (θ λ ) H -1 λ (θ λ ) , (4.23
)

df λ = E ∇ Z (θ ) 2 H λ -1 (θ ) , (4.24) t λ = t(θ -θ λ ), (4.25) r λ (θ) such that 1/r λ (θ) = sup z∈supp(ρ) sup g∈ϕ(z) g H λ -1 (θ) . (4.26)

4.B Preliminary results on self concordant losses

In this section, we show how our definition/assumption of self concordance (see Assumption 4.8) enables a fine control on the excess risk. In particular, we clearly relate the difference in function values to the quadratic approximation at the optimum as well as the renormalized gradient. We start by presenting a general bounds in Sec. 4.B .1 before applying them to the problem of localizing the optimum Sec. 4.B .2.

4.B .1 Basic results on self-concordance

In this section, as in the rest of the appendix, we are under the conditions of Assumption 4.8. In this section only, we give ourselves a probability measure µ on Z. We will apply the results of this section to µ = ρ, ρ, δ z , where ρ = 1 n n i=1 δ z i and z is sampled from ρ. First of all, let us introduce the following notation. For any probability measure µ on Z and any θ ∈ H, define

• R µ = sup z∈supp(µ) sup g∈ϕ(z) g , • t µ (θ) = sup z∈supp(µ) sup g∈ϕ(z) |θ • g| .
In order to be able to define L µ (θ) = E µ [ z (θ)] and to derive under the expectation, we assume that Assumptions 4.3 and 4.4 are satisfied for µ (replace ρ by µ in the assumption).

Since µ and satisfy Assumptions 4.3, 4.4 and 4.8, proposition 4.8 ensures that we can define

L µ (θ) = E µ [ z (θ)] and L µ,λ (θ) = L µ (θ) + λ 2 θ 2
, as well as their respective Hessians H µ (θ) and H µ,λ (θ).

The following result is greatly inspired from results in [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF] on generalized self concordant losses, and their refinement in [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF]. However, while Eqs. (4.27), (4.29) and (4.30) appear more or less explicitly, Eq. (4.28) provides an easier way to deal with certain bounds afterwards and was not used in this form before.

Proposition 4.4 (using the self-concordance of ). Let θ 0 , θ 1 ∈ H and λ ≥ 0. Assume that ( z ) z and µ satisfy Assumptions 4.3, 4.4 and 4.8. We have the following inequalities:

• Bounds on Hessians H µ,λ (θ 1 ) exp (t µ (θ 1 -θ 0 )) H µ,λ (θ 0 ).
(4.27)

• Bounds on gradients (if λ > 0)

φ (t µ (θ 1 -θ 0 )) θ 1 -θ 0 H µ,λ (θ 0 ) ≤ ∇L µ,λ (θ 1 ) -∇L µ,λ (θ 0 ) H -1 µ,λ (θ 0 ) , (4.28) ∇L µ,λ (θ 1 ) -∇L µ,λ (θ 0 ) H -1 µ,λ (θ 0 ) ≤ φ (t µ (θ 1 -θ 0 )) θ 1 -θ 0 H µ,λ (θ 0 ) , (4.29)
where φ(t) = (e t -1)/t and φ(t) = (1 -e -t )/t.

• Bounds on function values

L µ,λ (θ 1 ) -L µ,λ (θ 0 ) -∇L µ,λ (θ 0 )(θ 1 -θ 0 ) ≤ ψ (t µ (θ 1 -θ 0 )) θ 1 -θ 0 2 H µ,λ (θ 0 ) , (4.30)
where ψ(t) = (e tt -1)/t 2 .

Proof. First of all, note that for any µ and λ, given θ ∈ H and k, h ∈ H,

∇ 3 L µ,λ (θ)[h, k, k] = E z∼µ ∇ 3 λ z (θ)[h, k, k] ≤ E z∼µ ∇ 3 z (θ)[h, k, k] ≤ E z∼µ sup g∈ϕ(z) |h • g| ∇ 2 z (θ)[k, k] ≤ t µ (h) E z∼µ ∇ 2 z (θ)[k, k] = t µ (h) ∇ 2 L µ (θ)[k, k].
This yields the following fundamental inequality :

∇ 3 L µ,λ (θ)[h, k, k] ≤ t µ (h) ∇ 2 L µ,λ (θ)[k, k]. (4.31)
We now define, for any t ∈ R, θ t := θ 0 + t(θ 1 -θ 0 ).

Point 1. For the first inequality, let h ∈ H be a fixed vector, and consider the function

ϕ : t ∈ R → ∇ 2 L µ,λ (θ t )[h, h]. Since ϕ (t) = ∇ 3 L µ,λ (θ t )[θ 1 -θ 0 , h, h],
using Eq. (4.31), we get that ϕ (t) ≤ t µ (θ 1 -θ 0 ) ϕ(t). Using Lemma 4.10, we directly find that ϕ(1) ≤ exp(t µ (θ 1 -θ 0 ))ϕ(0), which, rewriting the definition of ϕ, yields

∇ 2 L µ,λ (θ 1 )[h, h] ≤ exp(t µ (θ 1 -θ 0 ))∇ 2 L µ,λ (θ 0 )[h, h].
This being true for any direction h, we have (4.27).

Point 2. To prove Eq. (4.28), let us look at the quantity (θ

1 -θ 0 ) • (∇L µ,λ (θ 1 ) -∇L µ,λ (θ 0 )). Since ∇L µ,λ (θ 1 ) -∇L µ,λ (θ 0 ) = 1 0 ∇ 2 L µ,λ (θ t )(θ 1 -θ 0 )dt, we have (θ 1 -θ 0 ) • (∇L µ,λ (θ 1 ) -∇L µ,λ (θ 0 )) = 1 0 ∇ 2 L µ,λ (θ t )[θ 1 -θ 0 , θ 1 -θ 0 ]dt.
Applying Eq. (4.27) to θ 0 and θ t and the reverse, we find that

∀t ∈ [0, 1], e -tt µ (θ 1 -θ 0 ) ∇ 2 L µ,λ (θ 0 ) ∇ 2 L µ,λ (θ t ).
Hence, integrating the previous equation, we have

(θ 1 -θ 0 ) • (∇L µ,λ (θ 1 ) -∇L µ,λ (θ 0 )) ≥ φ (t µ (θ 1 -θ 0 )) θ 1 -θ 0 2 H µ,λ (θ 0 ) . Finally, bounding (θ 1 -θ 0 )•(∇L µ,λ (θ 1 ) -∇L µ,λ (θ 0 )) by θ 1 -θ 0 H µ,λ (θ 0 ) ∇L µ,λ (θ 1 )-∇L µ,λ (θ 0 ) H -1 µ,λ (θ 0 )
, and simplifying by θ 1 -θ 0 H µ,λ (θ 0 ) , we obtain Eq. (4.28).

Point 3. To prove Eq. (4.29), first write

∇L µ,λ (θ 1 ) -∇L µ,λ (θ 0 ) H -1 µ,λ (θ 0 ) = 1 0 H -1/2 µ,λ (θ 0 )H µ,λ (θ t )(θ 1 -θ 0 )dt = 1 0 H -1/2 µ,λ (θ 0 )H µ,λ (θ t )H -1/2 µ,λ (θ 0 ) H 1/2 µ,λ (θ 0 )(θ 1 -θ 0 )dt ≤ 1 0 H -1/2 µ,λ (θ 0 )H µ,λ (θ t )H -1/2 µ,λ (θ 0 ) dt θ 1 -θ 0 H µ,λ (θ 0 ) .
Then apply Eq. (4.27) to have

∀t ∈ [0, 1], H µ,λ (θ t ) e tt µ (θ 1 -θ 0 ) H µ,λ (θ 0 ). This implies ∀t ∈ [0, 1], H -1/2 µ,λ (θ 0 )H µ,λ (θ t )H -1/2 µ,λ (θ 0 ) e tt µ (θ 1 -θ 0 ) I. And hence in particular ∀t ∈ [0, 1], H -1/2 µ,λ (θ 0 )H µ,λ (θ t )H -1/2 µ,λ (θ 0 ) ≤ e tt µ (θ 1 -θ 0 ) .
Finally, integrating this, we get (θt) ≤ e t t µ (θ 1 -θ 0 ) ϕ (0). Then using the fact that ϕ(0), ϕ (0) = 0 and integrating this inequality two times, we get the result.

1 0 H -1/2 µ,λ (θ 0 )H µ,λ (θ t )H -1/2 µ,λ (θ 0 ) dt ≤ φ (t µ (θ 1 -θ 0 )) . Thus Eq. (4.29) is proved. Point 4. To prove Eq. (4.30), define ∀t ∈ R, ϕ(t) = L µ,λ (θ t ) -L µ,θ (θ 0 ) -t ∇L µ,λ (θ 0 )(θ 1 -θ 0 ) We have ϕ (t) = θ 1 -θ 0 2 H µ,λ
Proof. of proposition 4.1. First note that since the support of µ is included in the support of ρ, Assumption 4.3 and Assumption 4.4 also hold for µ. Hence, since Assumptions 4.2 to 4.4 are satisfied, by proposition 4.8, L µ,λ , ∇L µ,λ and ∇ 2 L µ,λ are well-defined.

Assuming the existence of a minimizer θ µ,λ of L µ,λ , the reported equations are the same than those of proposition 4.4 when taking θ 1 = θ and θ 0 = θ µ,λ , with the fact that t µ (v) ≤ t(v) for any v ∈ H since the support of µ is a subset of the support of ρ, and ∇L µ,λ (θ µ,λ ) = 0. Note that since L µ,λ is defined on H, if λ > 0, then θ µ,λ always exists and is unique by strong convexity.

4.B .2 Localization properties for t λ

The aim of this section is to localize the optima θ λ and θ λ using the re-normalized gradient. This type of result is inspired by Proposition 2 of [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF] or Proposition 3.5 of [START_REF] Ostrovskii | Finite-sample analysis of M-estimators using selfconcordance[END_REF]. However, their proof is based on a slightly different result, namely Eq. (4.28), and its formulation is slightly different. Indeed, while the two propositions mentioned above concentrate on performing a quadratic approximation directly, we bound the term that could have been too large in that quadratic approximation. Proposition 4.5 (localisation). Let θ ∈ H, then the following holds

∇L λ (θ) H -1 λ (θ) ≤ r λ (θ) 2 =⇒ t(θ -θ λ ) = t λ ≤ log 2, (4.32) ∇ L λ (θ) H λ -1 (θ) H -1/2 λ (θ)H λ 1/2 (θ) 2 ≤ r λ (θ) 2 =⇒ t(θ -θ λ ) ≤ log 2. (4.33)
Proof. To prove Eq. ( 4.32), we first write

t(θ -θ λ ) = sup z∈supp(ρ) sup g∈ϕ(z) |(θ -θ λ ) • g| ≤ θ -θ λ H λ (θ) sup z∈supp(ρ) sup g∈ϕ(z) g H -1 λ (θ) .
Now we use Eq. (4.14) to bound θθ λ H λ (θ) , and putting things together, we get

t(θ -θ λ )φ (t(θ -θ λ )) ≤ ∇L λ (θ) H -1 λ (θ) r λ (θ) .
Using the fact that tφ(t) = 1 -e -t is an increasing function, we see that if tφ(t) ≤ 1/2, then t ≤ log 2 hence the result.

To prove Eq. (4.33), we use the same reasoning. First, we bound

t(θ-θ λ ) = sup z∈supp(ρ) sup g∈ϕ(z) (θ -θ λ ) • g ≤ θ-θ λ H λ (θ) H -1/2 λ (θ)H λ 1/2 (θ) sup z∈supp(ρ) sup g∈ϕ(z) g H -1 λ (θ) .
Now using Eq. (4.14) to the function L λ , we get

t(θ -θ λ )φ t ρ(θ -θ λ ) ≤ ∇ L λ (θ) H -1 λ (θ) H -1/2 λ (θ)H λ 1/2 (θ) 1 r λ (θ)
.

Now using the fact that t ρ(θθ λ ) ≤ t(θθ λ ) and that φ is a decreasing function, and that

∇ L λ (θ) H -1 λ (θ) ≤ H -1/2 λ (θ)H λ 1/2 (θ) ∇ L λ (θ) H λ -1 (θ) , this yields t(θ -θ λ )φ t(θ -θ λ ) ≤ ∇ L λ (θ) H λ -1 (θ) H -1/2 λ (θ)H λ 1/2 (θ) 2 1 r λ (θ) .
We conclude using the same argument as before.

4.C Main result, simplified

In this section, we perform a simplified analysis in the case where we assume nothing on Bias λ more than just the fact that θ exists. In this section we assume that z and ρ satisfy Assumptions 4.3 to 4.5 and 4.8.

Definition 4.4 (Definition of B 1 , B 2 and df λ ). Under assumptions Assumptions 4.3 to 4.5 and 4.8, the following quantities are well-defined and real-valued.

B 1 = sup θ ≤ θ B 1 (θ) B 2 = sup θ ≤ θ B 2 (θ), df λ = E ∇ z (θ λ ) 2 H λ -1 (θ λ ) .
Proposition 4.6. The quantities in definition 4.4 are finite and moreover

df λ ≤ B 2 1 λ .
Proof. These are well defined thanks to Lemmas 4.11 and 4.12.

Definition 4.5 (Constants). In this section, we will use the following constants.

K var = 1 + ψ(log 2) φ(log 2) 2 ≤ 4, = 2 √ 2 1 + 1 2 √ 3 ≤ 4, C bias = 1 + K var 8 ≤ 2, C var = 2K var 2 ≤ 84.

4.C .1 Analytic results

Theorem 4.5 (Analytic decomposition). For any λ > 0 and n

∈ N, if R √ λ Var λ ≤ 1 2 , L( θ λ ) -L(θ ) ≤ K var Var 2 λ + λ θ 2 , (4.34)
where K var is defined in definition 4.5.

Proof. First decompose the excess risk of θ λ in the following way:

L( θ λ ) -L(θ ) = L λ ( θ λ ) -L λ (θ λ ) variance + L(θ λ ) -L(θ ) bias + λ 2 θ λ 2 -θ λ 2 mixed .
1) Variance term: For the variance term, use Eq. (4.13)

L λ ( θ λ ) -L λ (θ λ ) ≤ ψ t(θ λ -θ λ ) θ λ -θ λ 2 H λ (θ λ ) .
2) Bias term: For the bias term, note that since θ λ ≤ θ ,

L(θ λ ) -L(θ ) = L λ (θ λ ) -L λ (θ ) + λ 2 θ 2 - λ 2 θ λ 2 ≤ λ 2 θ 2 .
3) Mixed term: For the mixed term, since

θ λ H λ (θ λ ) -1 ≤ H λ (θ λ ) -1/2 θ λ ≤ λ -1/2 θ λ ≤ λ -1/2 θ , we have λ 2 θ λ 2 -θ λ 2 = λ 2 θ λ -θ λ • θ λ + θ λ ≤ λ 2 θ λ -θ λ H λ (θ λ ) θ λ -θ λ H λ (θ λ ) -1 + 2 θ λ H λ (θ λ ) -1 ≤ 1 2 θ λ -θ λ 2 H λ (θ λ ) + √ λ θ θ λ -θ λ H λ (θ λ ) ≤ θ λ -θ λ 2 H λ (θ λ ) + λ 2 θ 2 .
where we get the last inequality by using ab ≤ a 2 2 + b 2 2 . 4) Putting things together

L( θ λ ) -L(θ ) ≤ 1 + ψ t(θ λ -θ λ ) θ λ -θ λ 2 H λ (θ λ ) + λ θ 2 .
By using Eq. (4.14) we have

θ λ -θ λ H λ (θ λ ) ≤ H λ 1/2 (θ λ ) H -1/2 λ (θ λ ) θ λ -θ λ H λ (θ λ ) ≤ 1 φ t ρ (θ λ -θ λ ) H λ 1/2 (θ λ ) H -1/2 λ (θ λ ) ∇ L λ (θ λ ) H -1 λ (θ λ ) .
Note that by multiplying and dividing for H λ 1/2 (θ λ ),

∇ L λ (θ λ ) H -1 λ (θ λ ) = H -1/2 λ (θ λ )∇ L λ (θ λ ) = H -1/2 λ (θ λ )H λ -1/2 (θ λ )H λ 1/2 (θ λ )∇ L λ (θ λ ) ≤ H -1/2 λ (θ λ )H λ -1/2 (θ λ ) H λ 1/2 (θ λ )∇ L λ (θ λ ) = H -1/2 λ (θ λ )H λ -1/2 (θ λ ) ∇ L λ (θ λ ) H λ -1 (θ λ ) .
Then,

θ λ -θ λ H λ (θ λ ) ≤ 1 φ t(θ λ -θ λ ) H λ 1/2 (θ λ ) H -1/2 λ (θ λ ) 2 ∇ L λ (θ λ ) H λ -1 (θ λ ) = 1 φ t(θ λ -θ λ ) Var λ .
Now we know that using Eq. (4.33), if

Var λ ≤ r λ (θ λ )
2 , then t(θ λ -θ λ ) ≤ log 2, which yields the following bound:

L( θ λ ) -L(θ ) ≤ (1 + ψ (log 2)) φ(log 2) 2 Var λ + λ θ 2 .
Finally, we can bound 1 r λ (θ λ ) ≤ R λ 1/2 to have the final form of the proposition.

4.C .2 Probabilistic results

Lemma 4.2

(bounding ∇ L λ (θ λ ) H λ (θ λ ) ). Let n ∈ N, λ > 0 and δ ∈ (0, 1]. For k ≥ 1, if n ≥ 24 B 2 λ log 2 δ , n ≥ k 2 2 log 2 δ ,
then with probability at least 1 -δ,

H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤ /2 df λ ∨ (B 2 1 /B 2 ) log 2 δ n + 2 k √ λ θ
where is defined in definition 4.5.

Proof. 1) First use Bernstein inequality for random vectors (e.g. Thm. 3.3.4 of [START_REF] Yurinsky | Sums and Gaussian vectors[END_REF]: for any n ∈ N and δ ∈ (0, 1], with probability at least 1 -δ, we have

H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤ 2M log 2 δ n + σ 2 log 2 δ n ,
where

M = sup z∈supp(ρ) ∇ λ z (θ λ ) H λ -1 (θ λ ) and σ = E ∇ λ z (θ λ ) 2 H λ -1 (θ λ ) 1/2
.

2) Using the fact that ∇ λ z (θ λ ) = ∇ z (θ λ ) + λθ λ , we bound M as follows:

M = sup z∈supp(ρ) ∇ λ z (θ λ ) H λ (θ λ ) ≤ sup z∈supp(ρ) ∇ z (θ λ ) H λ (θ λ ) + λ θ λ H λ -1 (θ λ ) ≤ B 1 √ λ + √ λ θ ,
where in the last inequality, we use the fact that

θ λ H λ -1 (θ λ ) ≤ 1 √ λ θ λ ≤ 1 √ λ θ . Similarly, we bound σ σ ≤ E ∇ z (θ λ ) 2 H λ -1 (θ λ ) 1/2 + λ θ λ H λ -1 (θ λ ) ≤ df 1/2 λ + √ λ θ .
3) Injecting these bounds in the concentration inequality,

H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤ 2B 2 log 2 δ λn 2(B 2 1 /B 2 ) log 2 δ n + 2df λ log 2 δ n + √ λ θ   2 log 2 δ n + 2 log 2 δ n   ,
where we have decomposed

2B 2 1 log 2 δ √ λn = 2B 2 log 2 δ λn 2(B 2 1 /B 2 ) log 2 δ n
for the first term. Reordering the terms, this yields

H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤   1 + 2B 2 log 2 δ λn   2df λ ∨ B 2 1 /B 2 log 2 δ n + √ λ θ   2 log 2 δ n + 2 log 2 δ n   . 4) Now assuming that n ≥ 24 B 2 λ log 2 δ , n ≥ k 2 2 log 2 δ ,
this yields

H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤ 1 + 1 2 √ 3 2df λ ∨ (B 2 1 /B 2 ) log 2 δ n + 2 k √ λ θ .
Combining the two previous lemmas, we get:

Lemma 4.3 (Bounding Var λ ). Let n ∈ N and 0 < λ ≤ B 2 . Let δ ∈ (0, 1]. If for k ≥ 1 n ≥ 24 B 2 λ log 8B 2 λδ , n ≥ 2k 2 log 2 δ ,
then with probability at least 1 -2δ,

Var λ ≤ df λ ∨ (B 2 1 /B 2 ) log 2 δ n + 4 k √ λ θ ,
where is a constant defined in definition 4.5.

Proof. Recall that

Var λ = H λ 1/2 (θ λ ) H -1/2 λ (θ λ ) 2 ∇ L λ (θ λ ) H -1 λ (θ λ )
. Using Lemma 4.6, under the conditions of this lemma, we have

H λ 1/2 (θ λ ) H -1/2 λ (θ λ ) 2 ≤ 2. Combining this with the bound for ∇ L λ (θ λ ) H -1
λ (θ λ ) obtained in Lemma 4.2, we get the result (the probability 1 -2δ comes from the fact that we perform a union bound).

4.C .3 Final result

< λ ≤ B 2 . Let δ ∈ (0, 1]. If n ≥ 512 θ 2 R 2 ∨ 1 log 2 δ , n ≥ 24 B 2 λ log 8B 2 λδ , n ≥ 16 2 R 2 df λ ∨ (B 2 1 /B 2 ) λ log 2 δ ,
then with probability at least 1 -2δ,

L( θ λ ) -L(θ ) ≤ C var df λ ∨ (B 2 1 /B 2 ) n log 2 δ + C bias λ θ 2 ,
where , C bias , C var are defined in definition 4.5.

Proof. 1) Recall the analytical decomposition in Theorem 4.5. For any λ > 0 and n ∈ N, if

R √ λ Var λ ≤ 1 2 , L( θ λ ) -L(θ ) ≤ K var Var 2 λ + λ θ 2 ,
where K var is defined in definition 4.5.

2) Now apply Lemma 4.3 for a given k ≥ 1. If

n ≥ 24 B 2 λ log 8B 2 λδ , n ≥ 2k 2 log 2 δ ,
then with probability at least 1 -2δ,

Var λ ≤ df λ ∨ (B 2 1 /B 2 ) log 2 δ n + 4 k √ λ θ ,
where is a constant defined in definition 4.5.

4.D . MAIN RESULT, REFINED ANALYSIS 149

In order to satisfy the condition to have the analytical decomposition, namely

R √ λ Var λ ≤ 1 2 , it is therefore sufficient to have R df λ ∨ (B 2 1 /B 2 ) log 2 δ λn ≤ 1 4 , 4 k R θ ≤ 1 4 .
3) Thus, if we choose k = 16(R θ ∨ 1), we have both k ≥ 1 and the second condition in the previous equation. Moreover, the condition n ≥ 2k 2 log 2 δ becomes n ≥ 512(R 2 θ 2 ∨ 1) log 2 δ . Hence, under the conditions of this theorem, we can apply the analytical decomposition :

L( θ λ ) -L(θ ) ≤ K var Var 2 λ + λ θ 2 ≤ 2K var 2 df λ ∨ (B 2 1 /B 2 ) log 2 δ n + 1 + K var 32 k 2 λ θ 2 .
In the last inequality, we have used (a + b) 2 ≤ 2a 2 + 2b 2 to separate the terms coming from Var 2 λ . Finally, using the fact that k ≥ 16 and hence that 32 k 2 ≤ 1 8 , we get the constants in the theorem.

Proof. of Theorem 4.2 Since ∀λ > 0, df λ ≤ B 2 1 λ , and since λ ≤ B 2 , df λ ∨ B 2 1 /B 2 ≤ B 2 1
λ . From definition 4.5, we get that ≤ 4, C bias ≤ 2, C var ≤ 84. Thus, we can use these bounds in Theorem 4.6 to obtain the result.

4.D Main result, refined analysis

In subsection Sec. 4.D .1 we split the excess risk in terms of bias and variance, that will be controlled in Sec. 4.D .3, the final result is Theorem 4.4 in Sec. 4.D .4, while in Sec. 4.F a version with explicit dependence in λ, n is reported.

Constants First, we introduce three constants that will be crucial for the final bound.

Definition 4.6.

B 1 = B 1 (θ ), B 2 = B 2 (θ ), Q * = B 1 / B 2 .
In the following sections, we also will use the following functions of t λ and t λ which we will treat as constants (see proposition 4.7).

Definition 4.7.

K bias (t λ ) = 2 ψ(t λ + log 2) φ(t λ ) 2 ≤ 2e 3t λ , K var (t λ ) = 2 ψ(t λ + log 2)e t λ φ(log 2) 2 ≤ 8e 2t λ , 1 (t λ ) = e t λ /2 , 2 (t λ ) = e t λ /2 1 + e t λ ≤ 2e 3t λ /2 C bias = ψ(t λ + log 2) 2 φ(t λ ) + e t λ φ(log 2) 2 ≤ 6e 2t λ , C var = 64ψ(t λ + log 2)e 2t λ φ(log 2) 2 ≤ 256e 3t λ 1 = 576 2 1 2 2 (1/2 ∨ t λ ) 2 ≤ 2304e 4t λ ( t λ ∨ 1/2) 2 , 2 = 256 4 1 ≤ 256e 2t λ .
Note that theses functions are all increasing in t λ and t λ , and are lower bounded by strictly positive constants.

For the second bounds, we use the fact that ψ(t) ≤ e t 2 and 1/φ(t) ≤ e t to bound all the quantities using only exponentials of t λ .

A priori, these constants will depend on λ. However, we can always bound t λ and t λ in the following way.

Lemma 4.4. Recall the definitions of t λ := t(θ λ -θ ) and t λ := Bias λ r λ (θ ) . We have the following cases.

• If t λ ≤ 1 2 , then t λ ≤ log 2, • else, t λ ≤ R θ and t λ ≤ 2R θ .
Proof. The first point is a direct application of Eq. (4.32). One can obtain the second by noting that t(θ λ -θ ) ≤ R θ λ -θ . Since θ λ ≤ θ , we have the bound on t λ . For the bound on t λ , since Bias λ ≤ √ λ θ and 1 r λ (θ ) ≤ R √ λ , we have the wanted bound.

Hence, we can always bound the constants in definition 4.7 by constants independant of λ.

Proposition 4.7. If t λ ≤ 1/2, then t λ ≤ log 2 and

K bias (t λ ) ≤ 4, K var (t λ ) ≤ 7, 1 (t λ ) ≤ 2, 2 (t λ ) ≤ 5 1 (t λ , t λ ) ≤ 5184, 2 (t λ ) ≤ 1024, C bias ≤ 6, C var ≤ 414.
Else,

K bias (t λ ) ≤ 2e 6R θ , K var (t λ ) ≤ 8e 4R θ , 1 (t λ ) ≤ e R θ , 2 (t λ ) ≤ 2e 3R θ , 1 (t λ , t λ ) ≤ 2304(R θ ) 2 e 8R θ , 2 (t λ ) ≤ 256e 4R θ , C bias ≤ 6e 4R θ , C var ≤ 256e 6R θ .
Proof. For the first bound, we use the fact that t λ ≤ log 2 and plug that in the expressions above as these functions are increasing in t λ . We compute them numerically from the definition.

For the second set of bounds, we simply inject the bounds for t λ and t λ in the second bounds of definition 4.7.

4.D .1 Analytic decomposition of the risk

In this section, we make use of self-concordance to control certain quantities required to control the variance, with respect to our main quantities Bias λ , r λ and df λ . The excess risk has been already decomposed in Sec. 4.6.

Theorem 4.7 (Analytic decomposition). Let λ > 0 and K bias and K var be the increasing functions of t λ described in Eq. (4.37). When Var λ ≤ r λ (θ λ )/2, then

L( θ λ ) -L(θ ) ≤ K bias (t λ ) Bias 2 λ + K var (t λ ) Var 2 λ . (4.35) Moreover K bias (t λ ), K var (t λ ) ≤ 7 if Bias λ ≤ 1 2 r λ (θ ), otherwise K bias (t λ ), K var (t λ ) ≤ 8e 6 θ
R (see proposition 4.7 in Sec. 4.D for more precise bounds).

Proof. Since θ exists by Assumption 4.5, using Eq. (4.13), applied with µ = ρ and λ = 0, we have

L(θ) -L(θ ) ≤ ψ(t(θ -θ )) θ -θ 2 H(θ )
, for any θ ∈ H. By setting θ = θ λ , we obtain

L( θ λ ) -L(θ ) ≤ ψ(t( θ λ -θ )) θ λ -θ 2 H(θ ) .
Using the fact that H(θ ) H(θ ) + λI =: H λ (θ ), by adding and subtracting θ λ , we have

θ λ -θ H(θ ) ≤ θ λ -θ H λ (θ ) ≤ θ λ -θ H λ (θ ) + θ λ -θ λ H λ (θ ) ,
and

analogously since t(•) is a (semi)norm, t( θ λ -θ ) ≤ t λ + t( θ λ -θ ), so L( θ λ ) -L(θ ) ≤ ψ(t λ + t( θ λ -θ λ )) ( θ λ -θ H λ (θ ) + θ λ -θ λ H λ (θ ) ) 2 .
By applying Eq. (4.12) with µ = ρ and θ = θ , we have H λ (θ ) e t λ H λ (θ λ ) and so

L( θ λ ) -L(θ ) ≤ ψ(t λ + t( θ λ -θ λ )) ( θ λ -θ H λ (θ ) + e t λ /2 θ λ -θ λ H λ (θ λ ) ) 2 . (4.36)
The terms t λ and θ λ -θ H λ (θ ) are related to the bias terms, while the terms t( θ λ -θ λ ) and θ λ -θ λ H λ (θ λ ) are related to the variance term.

Bounding the bias terms. Recall the definition of the bias Bias λ = ∇L λ (θ ) H -1 λ (θ ) . We bound t λ = t(θ λ -θ ), by Lemma 4.1 and the term θθ λ H λ (θ ) by applying Eq. (4.14

) with µ = ρ and θ = θ θ -θ λ H λ (θ ) ≤ 1/φ(t λ ) ∇L λ (θ ) H -1 λ (θ ) = 1/φ(t λ ) Bias λ .
Bounding the variance terms. First we bound the term θ λ -θ λ H λ (θ λ ) := H λ (θ λ ) 1/2 ( θ λθ λ ) , by multiplying and dividing for H λ (θ λ ) -1/2 , we have

θ λ -θ λ H λ (θ λ ) = H λ (θ λ ) 1/2 H λ (θ λ ) -1/2 H λ (θ λ ) 1/2 ( θ λ -θ λ ) ≤ H λ (θ λ ) 1/2 H λ (θ λ ) -1/2 θ λ -θ λ H λ (θ λ ) .
Applying Eq. (4.14) with µ = 1 n n i=1 δ z i and θ = θ λ , since L µ,λ = L λ for the given choice of µ, we have

θ λ -θ λ H λ (θ λ ) ≤ ∇ L λ (θ λ ) H -1 λ (θ λ ) / φ(t(θ λ -θ λ )) and since ∇ L λ (θ λ ) H -1 λ (θ λ ) := H -1/2 λ (θ λ )∇ L λ (θ λ )
, by multiplying and dividing by H λ (θ λ ), we have:

H -1/2 λ (θ λ )∇ L λ (θ λ ) = H -1/2 λ (θ λ )H λ (θ λ ) 1/2 H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤ H -1/2 λ (θ λ )H λ (θ λ ) 1/2 ∇ L λ (θ λ ) H λ -1 (θ λ ) . Then θ λ -θ λ H λ (θ λ ) ≤ 1 φ(t(θ λ -θ λ )) H 1/2 λ (θ λ )H λ -1/2 (θ λ ) 2 ∇ L λ (θ λ ) H -1 λ (θ λ ) = Var λ φ(t(θ λ -θ λ ))
.

To conclude this part of the proof we need to bound t( θ λ -θ λ ). Since we require Var λ /r λ (θ λ ) ≤ 1/2, by proposition 4.5 we have t( θ λ -θ λ ) ≤ log 2.

CHAPTER 4. BEYOND LEAST SQUARES

Gathering the terms. By gathering the results of the previous paragraphs

L( θ λ ) -L(θ ) ≤ ψ(t λ + log 2) ( 1/φ(t λ ) Bias λ + e t λ /2 /φ(log 2) Var λ ) 2
Using the fact that (a + b) 2 ≤ 2a 2 + 2b 2 , we have the desired result, with

K bias (t λ ) = 2ψ(t λ + log 2)/φ(t λ ) 2 , K var (t λ ) = 2ψ(t λ + log 2)e t λ /φ(log 2) 2 . (4.37)
which are bounded in definition 4.7 and proposition 4.7 of Sec. 4.D .

4.D .2 Analytic bounds for terms related to the variance

In this lemma, we aim to control the essential supremum and the variance of the random vector H λ -1/2 (θ λ )∇ λ z (θ λ ) relating it to quantities at θ . The results will be used to control the variance via Bernstein concentration inequalities, so we are going to control its essential supremum and its variance.

Lemma 4.5 (Control of H λ (θ λ ) -1/2 ∇ λ z (θ λ ))
. For any 0 < λ ≤ B 2 , we have 1. A bound on the essential supremum:

sup z∈supp(ρ) ∇ z (θ λ ) H λ -1 (θ λ ) ≤ 1 B 1 √ λ + 2 2 B 2 λ Bias λ .

A bound on the variance

E ∇ λ z (θ λ ) 2 H λ -1 (θ λ ) 1/2 ≤ 1 df λ + √ 2 2 B 2 λ Bias λ ,
where 1 , 2 are increasing functions of t λ :

1 (t λ ) = e t λ /2 2 (t λ ) = e t λ /2 (1 + e t λ ). Proof. Start by noting that if λ ≤ B 2 , then sup z∈supp(ρ) H λ -1/2 (θ )∇ 2 λ z (θ ) 1/2 2 ≤ 1 + B 2 λ ≤ 2 B 2
λ . Moreover, note that for any vector h ∈ H, multiplying and dividing by

∇ 2 z (θ ) 1/2 , h H λ -1 (θ ) := H λ -1/2 (θ ) h = H λ -1/2 (θ ) ∇ 2 z (θ ) 1/2 ∇ 2 z (θ ) -1/2 h ≤ H λ -1/2 (θ ) ∇ 2 z (θ ) 1/2 ∇ 2 z (θ ) -1/2 h ≤ 2B 2 λ ∇ 2 z (θ ) -1/2 h = 2B 2 λ h ∇ 2 z (θ ) -1 ,
where the last bound is mentioned at the beginning of the proof. Similarly, we can show

h ∇ 2 z (θ ) ≤ 2B 2 λ h H λ (θ ) , h H λ -1 (θ ) ≤ 2B 2 λ h ∇ 2 z (θ ) -1 . (4.38)
Essential supremum. Let z ∈ supp(ρ). First note that using Eq. (4.27), we have

∇ λ z (θ λ ) H λ -1 (θ λ ) ≤ e t λ /2 ∇ λ z (θ λ ) H λ -1 (θ ) . Now bound ∇ λ z (θ λ ) H λ -1 (θ ) ≤ ∇ λ z (θ λ ) -∇ λ z (θ ) H λ -1 (θ ) + ∇ λ z (θ ) H λ -1 (θ ) .
Since ∇ λ z (θ ) = ∇ z (θ ) + λθ , the last term is bounded by

Bias λ + sup z∈supp(ρ) ∇ z (θ ) H λ -1 (θ ) ≤ Bias λ + B 1 √ λ .
For the first term, start by using Eq. (4.38).

∇ λ z (θ λ ) -∇ λ z (θ ) H λ -1 (θ ) ≤ 2B 2 λ ∇ λ z (θ λ ) -∇ λ z (θ ) ∇ 2 λ z (θ ) -1 .
Using Eq. (4.29) on λ z , we find

∇ λ z (θ λ ) -∇ λ z (θ ) ∇ 2 λ z (θ ) -1 ≤ φ(t λ ) θ λ -θ ∇ 2 λ z (θ )
. Applying once again Eq. (4.38), we bound

θ λ -θ ∇ 2 λ z (θ ) ≤ 2B 2 λ θ λ -θ H λ (θ ) .
Finally, using Eq. (4.28) on L λ , we get

θ λ -θ H λ (θ ) ≤ 1 φ(t λ ) Bias λ .
Hence, putting things together, we get

∇ λ z (θ λ ) -∇ λ z (θ ) H λ -1 (θ ) ≤ 2B 2 λ φ(t λ ) φ(t λ ) Bias λ = 2B 2 λ e t λ Bias λ .
We the combine all our different computation to get the bound.

Variance. We start by using Eq. (4.27) to show that

E ∇ λ z (θ λ ) 2 H λ -1 (θ λ ) 1/2 ≤ e t λ /2 E ∇ λ z (θ λ ) 2 H λ -1 (θ ) 1/2 .
Then we use the triangle inequality

E ∇ λ z (θ λ ) 2 H λ -1 (θ ) 1/2 ≤ E ∇ λ z (θ λ ) -∇ λ z (θ ) 2 H λ -1 (θ ) 1/2 + E ∇ λ z (θ ) 2 H λ -1 (θ ) 1/2 .
We can easily bound the last term on the right hand side by Bias λ + df λ . For the first term, we proceed as in the previous case to obtain

∀z ∈ supp(ρ), ∇ λ z (θ λ ) -∇ λ z (θ ) H λ -1 (θ ) ≤ 2B 2 λ φ(t λ ) θ λ -θ ∇ 2 λ z (θ ) .
Now taking the expectancy of this inequality squared,

E ∇ λ z (θ λ ) -∇ λ z (θ ) 2 H λ -1 (θ ) 1/2 ≤ 2B 2 λ φ(t λ )E θ λ -θ 2 ∇ 2 λ z (θ ) 1/2 = 2B 2 λ φ(t λ ) θ λ -θ H λ (θ ) ,
where the last equality comes from E ∇ 2 λ Z (θ ) = H λ (θ ). Now applying Eq. (4.28) to L λ , we obtain

θ λ -θ H λ (θ ) ≤ 1 φ(t λ ) Bias λ .
Regrouping all these bounds, we obtain

E ∇ λ z (θ λ ) -∇ λ z (θ ) 2 H λ -1 (θ ) 1/2 ≤ e t λ 2B 2 λ Bias λ .
Hence the final bound is proved, regrouping all our computations.

4.D .3 Concentration lemmas

Here we concentrate in high probability the quantities obtained in the analytical decomposition. Details on the proof technique are given in Sec. 4.6 of the paper.

Lemma 4.6 (Equivalence of empirical and expected Hessian). Let θ ∈ H and n ∈ N. For any

δ ∈ (0, 1], λ > 0, if n ≥ 24 B 2 (θ) λ log 8B 2 (θ) λδ , (4.39)
then with probability at least 1 -δ: H λ (θ) 2 H λ (θ), or equivalently

H λ 1/2 (θ) H -1/2 λ (θ) 2 ≤ 2.
Proof. By Remark 9 and the definition of B 2 (θ), the condition we require on n is sufficient to apply proposition 4.10, in particular Eq. (4.51), to H λ (θ), H λ (θ), for t = 1/2, which provides the desired result.

Lemma 4.7 (Concentration of the empirical gradient).

Let n ∈ N, δ ∈ (0, 1], 0 < λ ≤ B 2 . For any k ≥ 4, if n ≥ k 2 2 2 B 2
λ log 2 δ , then with probability at least 1 -δ, we have

∇ L λ (θ λ ) H λ -1 (θ λ ) ≤ 2 √ 3 k Bias λ + 2 1 df λ ∨ (Q ) 2 log 2 δ n . (4.40)
Here, 1 , 2 are defined in Lemma 4.5 in Sec. 4.D and

(Q ) 2 = (B 1 ) 2 /B 2 . Proof. 1) First let us concentrate H λ (θ λ ) -1/2 ∇ L λ (θ λ ) using a Bernstein-type inequality.
We can see H λ (θ λ ) -1/2 ∇ L λ (θ λ ) as the mean of n i.i.d. random variables distributed from the law of the vector H λ (θ λ ) -1/2 ∇ z (θ λ ).

As we have shown in Lemma 4.5, the essential supremum and variance of this vector is bounded, then we can use Bernstein inequality for random vectors (e.g. Thm. 3.3.4 of [START_REF] Yurinsky | Sums and Gaussian vectors[END_REF]: for any λ > 0, any n ∈ N and δ ∈ (0, 1], with probability at least 1 -δ, we have

H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤ 2M log 2 δ n + σ 2 log 2 δ n , where M = sup z∈supp(ρ) ∇ z (θ λ ) H λ -1 (θ λ ) and σ = E ∇ z (θ λ ) 2 H λ -1 (θ λ ) 1/2
.

2) Using the bounds obtained in Lemma 4.5,

M ≤ 1 B 1 √ λ + 2 2 B 2 λ Bias λ , σ ≤ 1 df λ + √ 2 2 B 2 √ λ Bias λ .
3) Injecting these in the Bernstein inequality,

H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤ 2 1 B 1 / √ λ + 2 2 (B 2 /λ)Bias λ log 2 δ n + 1 df 1/2 λ + √ 2 2 B 2 /λBias λ 2 log 2 δ n =   4 2 B 2 log 2 δ λn + 4 2 2 B 2 log 2 δ λn   Bias λ + 2 2 1 df λ log 2 δ n + 2 B 2 log 2 δ λn 2 2 1 (B 1 ) 2 /B 2 log 2 δ n .
In the last inequality, we have regrouped the terms with a factor Bias λ and we have separated the first term of the decomposition in the following way :

2 1 B 1 log 2 δ √ λn = 2 B 2 log 2 δ λn 2 2 1 (Q ) 2 log 2 δ n .
Hence, we can bound the second line of the last inequality:

2 2 1 df λ log 2 δ n + 2B 2 log 2 δ λn 2 2 1 (Q ) 2 log 2 δ n ≤   1 + 2B 2 log 2 δ λn   2 2 1 df λ ∨ (Q ) 2 log 2 δ n . Thus, if we assume that n ≥ k 2 2 2 B 2 λ log 2 δ , H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤ 4 k 2 + 2 k Bias λ + 1 + √ 2 k 2 2 1 df λ ∨ (B 1 ) 2 /B 2 log 2 δ n .
In particular, for k ≥ 4,

H λ (θ λ ) -1/2 ∇ L λ (θ λ ) ≤ 3 k Bias λ + 2 1 df λ ∨ (Q ) 2 log 2 δ n . Lemma 4.8 (control of Var λ ). Let n ∈ N, δ ∈ (0, 1] and 0 < λ ≤ B 2 . Assume that for a certain k ≥ 5, n ≥ k 2 2 2 B 2 λ log 8 2 1 B 2 λδ .
Then with probability at least 1 -2δ, we have

Var λ ≤ 6 k Bias λ + 4 1 df λ ∨ (Q ) 2 log 2 δ n .
Here, 1 , 2 are defined in Lemma 4.5

Proof.

• First we apply Lemma 4.

6 to θ = θ λ . Since B 2 (θ λ ) ≤ e t λ B 2 = 2 1 B 2 , we see that the condition n ≥ 24 B 2 (θ λ ) λ log 8B 2 (θ λ ) λδ is satisfied if n ≥ 24 2 1 B 2 λ log 8 2 1 B 2 λδ .
Because k ≥ 5 and 2 ≥ 1 , and we see that the assumption of this lemma imply the conditions above and hence Lemma 4.6 is satisfied. In particular,

H λ (θ λ ) 1/2 H λ (θ λ ) -1/2 2 ≤ 2.
• Note that the condition of this proposition also imply the conditions of Lemma 4.7, because λ ≤ B 2 and 1 ≥ 1 imply

2 1 B 2 λδ ≥ 1 δ .

4.D .4 Final results

First, we find conditions on n such that the hypothesis Var λ ≤

r λ (θ λ ) 2 is satisfied. Lemma 4.9. Let n ∈ N, δ ∈ (0, 1], 0 < λ ≤ B 2 and n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , n ≥ 2 df λ ∨ (Q ) 2 r λ (θ ) 2 log 2 δ ,
then with probability at least 1 -2δ

Var λ r λ (θ λ ) ≤ 1 Var λ r λ (θ ) ≤ 1 2 ,
where 1 , 1 , 2 are constants defined in definition 4.7.

Proof. Recall that t λ = Bias λ r λ (θ ) . Using Lemma 4.8, we see that under the conditions of this lemma, we have

1 Var λ r λ (θ ) ≤ 6 1 Bias λ k r λ (θ ) + 4 2 1 df λ ∨ (Q ) 2 log 2 δ n r λ (θ ) 2 .
Thus, taking k = 24 1 (1/2 ∨ t λ ) and n ≥ 256 4 1 df λ ∨Q r λ (θ ) 2 log 2 δ , both terms in the sum are bounded by 1/4 hence the result.

4.E . EXPLICIT BOUNDS FOR THE SIMPLIFIED CASE 157

Note that here, we have defined

1 = 576 2 1 2 2 (1/2 ∨ t λ ) 2 , 2 = 256 4 1 ,
hence the constants in the definition above.

Proof. of Theorem 4.4 First we recall that 1 , 2 , 1 , C bias and C var are defined in definition 4.7, and bounded in proposition 4.7.

First note that, given the requirements on n, by Lemma 4.9, we have Var λ ≤ r λ (θ λ ) 2 with probability at least 1 -2δ. Thus, we are in a position to apply Theorem 4.7 :

L( θ λ ) -L(θ ) ≤ K bias Bias 2 λ + K var Var 2 λ ,
with K bias , K var defined in the proof of the theorem. Note that in the proof of Lemma 4.9, we have taken k = 24 1 (1/2 ∨ t λ ) ≥ 12. Hence, using Lemma 4.8, we find

Var λ ≤ 1 2 Bias λ + 4 1 df λ ∨ (Q ) 2 log 2 δ n .
Hence,

Var 2 λ ≤ 1 2 Bias 2 λ + 32 2 1 df λ ∨ (Q ) 2 log 2 δ n ,
which yields the wanted result with C bias = K bias + 1 2 K var and C var = 32 2 1 K var .

Proof. of Theorem 4.3

We get this theorem as a corollary of Theorem 4.4. Indeed,

∀λ ≤ B 2 , df λ ∨ (Q ) * ≤ (B 1 ) 2
λ , hence the result.

4.E Explicit bounds for the simplified case

In this section, assume that Assumptions 4.1, 4.3 to 4.5 and 4.8 hold.

Define the following constant N :

N = 36A 2 log 2 6A 2 1 δ ∨ 256 1 A 2 log 2 δ ∨ 512 θ 2 R 2 ∨ 1 log 2 δ , (4.41)
where

A = B 2 B 1 .
We have the following slow rates theorem.

Theorem 4.8 (Quantitative slow rates result).

Let n ∈ N. Let δ ∈ (0, 1]. Setting λ = 16((R ∨ 1)B 1 ) 1 √ n log 1/2 2 δ , if n ≥ N , with probability at least 1 -2δ, L( θ λ ) -L(θ ) ≤ 48 max(R, 1) max( θ 2 , 1)B 1 1 √ n log 1/2 2 δ , (4.42) and N = O poly(B 1 , B 2 , R θ ) is given explicitly in Eq. (4.41).
Here, poly denotes a certain rational function of the inputs.

Proof. Note that

df λ ≤ B 2 1 λ . Hence, if λ ≤ B 2 , then df λ ∨ (B 2 1 /B 2 ) ≤ B 2 1 λ . 1) Let us reformulate Theorem 4.6. Let n ∈ N and 0 < λ ≤ B 2 . Let δ ∈ (0, 1]. If n ≥ 512 θ 2 R 2 ∨ 1 log 2 δ , n ≥ 24 B 2 λ log 8B 2 λδ , n ≥ 16 2 R 2 B 2 1 λ 2 log 2 δ ,
then with probability at least 1 -2δ,

L( θ λ ) -L(θ ) ≤ C var B 2 1 λn log 2 δ + C bias λ θ 2 ,
where , C bias , C var are defined in definition 4.5.

2)

Now setting λ = 16RB 1 log 1/2 2 δ 1 n 1/2 , we see that the inequality n ≥ 16 2 R 2 B 2 1 λ 2 log 2 δ is automatically satisfied since ≤ 4. Hence, if n ≥ 512 θ 2 R 2 ∨ 1 log 2 δ , n ≥ 24 B 2 λ log 8B 2 λδ , 0 < λ ≤ B 2 , then L( θ λ ) -L(θ ) ≤ C var 256 1 R 2 λ + C bias λ θ 2 ≤ C var 256 + C bias max( 1 R 2 , θ 2 )λ.
Since by definition 4.5, C var ≤ 84 and C bias ≤ 2, we get

L( θ λ ) -L(θ ) ≤ 3 max( 1 R 2 , θ 2 )λ.
3) Having our fixed λ = 16

B 1 R log 1/2 2 δ n 1/2
, let us look for conditions for which

n ≥ 512 θ 2 R 2 ∨ 1 log 2 δ , n ≥ 24 B 2 λ log 8B 2 λδ , 0 < λ ≤ B 2 ,
are satisfied.

To deal with n ≥ 24 B 2 λ log 8B 2 λδ , bound

B 2 λ ≤ 1 16 B 2 RB 1 log 1/2 2 δ n 1/2 ≤ 1 8 B 2 RB 1 n 1/2 , 4.F . EXPLICIT BOUNDS FOR THE REFINED CASE 159
where we have used the fact that log 1/2 2 δ ≥ 1 2 . apply Lemma 4.14 with

a 1 = 3, a 2 = 1, A = B 2 RB 1
to get the following condition:

n ≥ 4a 2 1 A 2 log 2 2a 1 a 2 A 2 δ ,
which we express as

n ≥ 36A 2 log 2 6A 2 1 δ .
To deal with the bound λ < B 2 , we need only apply the definition to obtain

n ≥ 256 R 2 B 2 1 B 2 2 log 2 δ .
Thus, we can concentrate all these bounds as n ≥ N where

N = 36A 2 log 2 6A 2 1 δ ∨ 256 1 A 2 log 2 δ ∨ 512 θ 2 R 2 ∨ 1 log 2 δ ,
where

A = B 2 RB 1 . 4) Since R is only an upper bound, we can replace R by R ∨ 1. In this case, we see that A ≤ B 2 B 1 and max( 1 R∨1 , (R ∨ 1) θ 2 ) ≤ (R ∨ 1)( θ ∨ 1) 2 hence the final bounds.

4.F Explicit bounds for the refined case

In this part, we continue to assume Assumptions 4.1, 4.3 to 4.5 and 4.8. We present a classification of distributions ρ and show that we can achieve better rates than the classical slow rates.

Definition 4.8 (class of distributions). Let α ∈ [1, +∞] and r ∈ [0, 1/2].
We denote with P α,r the set of probability distributions ρ such that there exists L, Q ≥ 0,

• Bias λ ≤ L λ 1+2r 2 • df λ ≤ Q 2 λ -1/α ,
where this holds for any 0 < λ ≤ 1. For simplicity, if α = +∞, we assume that Q ≥ Q .

Note that given our assumptions, we always have

ρ ∈ P 1,0 , L = θ , Q = B 1 . (4.43)
We also define

λ 1 = Q Q 2α ∧ 1, (4.44) such that ∀λ ≤ λ 1 , df λ ∨ (Q ) 2 ≤ Q 2 λ 1/α . CHAPTER 4. BEYOND LEAST SQUARES

Interpretation of the classes

• The bias term Bias λ characterizes the regularity of the objective θ . In a sense, if r is big, then this means θ is very regular and will be easier to estimate. The following results reformulates this intuition.

Remark 6 (source condition). Assume there exists 0 ≤ r ≤ 1/2 and v ∈ H such that

P H(θ ) θ = H(θ ) r v.
Then we have

∀λ > 0, Bias λ ≤ L λ 1+2r 2 , L = H(θ ) -r θ .
• The effective dimension df λ characterizes the size of the space H with respect to the problem. The higher α, the smaller the space. If H is finite dimensional for instance, α = +∞.

We will give explicit bounds for the performance of θ λ depending on which class ρ belongs to, i.e., as a function of α, r.

Well -behaved problems r λ (θ ) has a limiting role. However, as soon as we have some sort of regularity, this role is no longer limiting, i.e. this quantity does not appear in the final rates and the constants in these rates have no dependence on the problem. This motivates the following definition.

We say that a problem is well behaved if the following equation holds.

∀δ ∈ (0, 1 2 ], ∃λ 0 (δ) ∈ (0, 1], ∀0 < λ ≤ λ 0 (δ), Lλ 1/2+r r λ (θ ) log 2 δ ≤ 1 2 . (4.45)
Remark 7 (well-behaved problems). Note that Eq. (4.45) is satisfied if one of the following holds.

• If R = 0, then the condition holds for λ 0 = 1.

• If r > 0, then the condition holds for λ 0 = (2LR log 2 δ ) -1/r ∧ 1.

• If there exists µ ∈ [0, 1) and

F ≥ 0 such that r λ (θ ) ≥ 1 F λ µ/2 , then this holds for λ 0 = (2RF log 2 δ ) -2/(1-µ+2r) ∧ 1. Moreover, if Eq. (4.45) is satisfied, than for any λ ≤ λ 0 , t λ ≤ log 2.
Note that the first possible condition corresponds to the case where the loss functions are quadratic in θ (if the loss is the square loss for instance). The second condition corresponds to having a strict source condition, i.e. something strictly better than just θ ∈ H. Finally, the third condition corresponds to the fact that the radius r λ decreases slower than the original bound of r λ ≥ λ 1/2 R , and hence it is not limiting. Note that a priori, using only the assumptions, our problems do not satisfy Eq. (4.45) (see Eq. (4.43), and the fact that r λ ≥

4.F .1 Quantitative bounds

In this section, for any given pair (α, r) characterizing the regularity and size of the problem, we associate

β = 1 1 + 2r + 1/α , γ = α(1 + 2r) α(1 + 2r) + 1 .
In what follows, we define

N = 256Q 2 L 2 (B 2 ∧ λ 0 ∧ λ 1 ) -1/β ∨ 1296 1 1 -β A log 5184 1 1 -β A 2 1 δ 1/(1-β) , (4.46) where A = B 2 L 2β
Q 2β , λ 0 is given by Eq. (4.45) and λ 1 is given by Eq. (4.44) :

λ 1 = Q 2α (Q ) 2α
. Theorem 4.9 (Quantitative results when Eq. (4.45) is satisfied and α < ∞ or r > 0). Let ρ ∈ P α,r and that we have either α < ∞ or r > 0. Let δ ∈ (0, 1 2 ]. If Eq. (4.45) is satisfied, and

n ≥ N, λ = 256 Q L 2 1 n β ,
then with probability at least 1 -2δ,

L( θ λ ) -L(θ ) ≤ 8 (256) γ Q γ L 1-γ 2 1 n γ log 2 δ ,
where N is defined in Eq. (4.46).

Proof. Using the definition of λ 1 , as soon as λ ≤ λ 1 we have

df λ ∨ (Q ) 2 ≤ Q 2 λ -1/α .
Let us formulate Theorem 4.4 using the fact that ρ ∈ P α,r .

Let δ ∈ (0, 1], 0 < λ ≤ B 2 ∧ λ 1 and n ∈ N such that n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , n ≥ 2 Q 2 λ 1/α r λ (θ ) 2 log 2 δ ,
then with probability at least 1 -2δ

L( θ λ ) -L(θ ) ≤ C bias L 2 λ 1+2r + C var Q 2 λ 1/α n log 2 δ ,
where C bias , C var are defined in definition 4.7. Now let us distinguish the two cases of our theorem.

Assume that ρ satisfies Eq. (4.45) . In this case the proof proceeds as follows. Note that as soon as λ ≤ λ 0 , we have Bias λ r λ (θ ) ≤ 1 2 and hence the bounds in proposition 4.7 apply. 1) First, we find a simple condition to guarantee

r λ (θ ) 2 λ 1/α ≥ 2 Q 2 1 n log 2 δ .
Using the fact that Eq. (4.45) is satisfied, we see that if λ ≤ λ 0 , then r λ ≥ 2Lλ 1/2+r log 2 δ . Hence, this condition is satisfied if

λ ≤ λ 0 , 4L 2 λ 1+2r+1/α ≥ 2 Q 2 1 n .
2) Now fix C λ = 256 ≥ 2 /4 (see proposition 4.7) and fix

λ 1+2r+1/α = C λ Q 2 L 2 1 n ⇐⇒ λ = C λ Q 2 L 2 1 n β .
where

β = 1/(1 + 2r + 1/λ) ∈ [1/2, 1).
Using our restatement of Theorem 4.4, we have that with probability at least 1 -2δ,

L( θ λ ) -L(θ ) ≤ C bias + 1 C λ C var log 2 δ L 2 λ 1+2r ≤ K log 2 δ L 2 λ 1+2r ,
where we have set

K = C bias + 1 256 C var ≤ 8 (see proposition 4.7). This result holds provided 0 < λ ≤ B 2 ∧ λ 0 ∧ λ 1 , n ≥ 1 B 2 λ log 8 2 1 B 2 λδ . (4.47)
Indeed, we have shown in the previous point that since

C λ ≥ 2 4 , r λ (θ ) 2 λ 1/α ≥ 2 Q 2 1 n log 2 δ .
3) Let us now work to guarantee the conditions in Eq. (4.47). First, to guarantee n ≥ 1

B 2 λ log 8 2 1 B 2 λδ , bound B 2 λ = B 2 L 2β n β C β λ Q 2β log β 2 δ ≤ 2 C β λ B 2 L 2β Q 2β n β .
Then apply Lemma 4.15 with

a 1 = 2 1 C β λ , a 2 = 16 2 1 C β λ , A = B 2 L 2β Q 2β .
Since β ≥ 1/2, using the bounds in proposition 4.7, we find a 1 ≤ 648 and a 2 ≤ 4, hence the following sufficient condition:

n ≥ 1296 1 1 -β A log 5184 1 1 -β A 2 1 δ 1/(1-β)
.

Then, to guarantee the condition

λ ≤ B 2 ∧ λ 0 ∧ λ 1 , we simply need n ≥ 256Q 2 L 2 (B 2 ∧ λ 0 ∧ λ 1 ) -1/β . Hence, defining N = 256Q 2 L 2 (B 2 ∧ λ 0 ∧ λ 1 ) -1/β ∨ 1296 1 1 -β A log 5184 1 1 -β A 2 1 δ 1/(1-β)
,

where A = B 2 L 2β
Q 2β , we see that as soon as n ≥ N , Eq. (4.47) holds.

4.G . ADDITIONAL LEMMAS 163

We now state the following corollary, for r > 0. We define N in the following way:

N = 256Q 2 L 2 (B 2 ∧ λ 0 ∧ λ 1 ) -1/β ∨ 1296 1 1 -β A log 5184 1 1 -β A 2 1 δ 1/(1-β) (4.48) where A = B 2 L 2β Q 2β , λ 0 = (2LR log 2 δ ) -1/r ∧ 1 and λ 1 = Q 2α (Q ) 2α
. Corollary 4.4. Assume ρ ∈ P α,r with r > 0. Let δ ∈ (0, 0.5] and n ≥ N , where N is defined in Eq. (4.48). For

λ = 256 Q L 2 1 n β , with probability at least 1 -2δ, L( θ λ ) -L(θ ) ≤ 8 (256) γ Q γ L 1-γ 2 1 n γ log 2 δ , Moreover, N = O poly B 1 , B 2 , L, Q, R, log 1 δ ,

4.G Additional lemmas

4.G .1 Self-concordance, sufficient conditions to define L and related quantities

In this section, we will consider an arbitrary probability measure µ on Z. We assume that z satisfies Assumption 4.8 with a certain given function ϕ. Recall that R µ = sup z∈supp(µ) sup g∈ϕ(z) g . In this section, we will also assume that R µ < ∞.

Lemma 4.10 (Gronwall lemma). Let ϕ : R → R be a differentiable function such that

∀t ∈ R, ϕ (t) ≤ Cϕ(t). Then ∀(t 0 , t 1 ) ∈ R 2 , ϕ(t 1 ) ≤ e C|t 1 -t 0 | ϕ(t 0 ).
Lemma 4.11. Assume that there exists θ 0 such that sup z∈supp(µ)

Tr ∇ 2 z (θ 0 ) < ∞ • sup z∈supp(µ) Tr ∇ 2 z (θ) < ∞ for any θ ∈ H;
• For any given radius T > 0, and any θ 0 ≤ T , we have

∀ θ ≤ T, ∀z ∈ Z, Tr ∇ 2 z (θ) ≤ exp (2R µ T ) Tr ∇ 2 z (θ 0 ) < ∞.
Proof. Let z ∈ supp(µ) be fixed. Using the same reasoning as in the proof of Eq. (4.27), we can show

∀θ 0 , θ 1 ∈ H, ∇ 2 z (θ 1 ) exp sup g∈ϕ(z) |g • (θ 1 -θ 0 )| ∇ 2 z (θ 0 ) exp (R µ θ 1 -θ 0 ) ∇ 2 z (θ 0 )
Where we have used the fact that R µ = sup z∈supp(µ) sup g∈ϕ(z) g < ∞ Thus, in particular

∀z ∈ supp(µ), ∀θ 0 , θ 1 ∈ H, Tr ∇ 2 z (θ 1 ) ≤ exp (R µ θ 1 -θ 0 ) Tr ∇ 2 z (θ 0 ) ,
which leads to the desired bounds.

Lemma 4.12. Assume that there exists θ 0 such that

sup z∈supp(µ) Tr ∇ 2 z (θ 0 ) < ∞, sup z∈supp(µ) ∇ z (θ 0 ) < ∞.
Then

• sup z∈supp(µ) ∇ z (θ) < ∞ for any θ ∈ H
• For any T > 0 and any θ 0 , θ ≤ T, z ∈ supp(µ),

∇ z (θ) ≤ ∇ z (θ 0 ) + 2T Tr ∇ 2 z (θ 0 ) + 4R µ ψ(2R µ T ) Tr ∇ 2 z (θ 0 ) R 2 .
Proof. Fix z ∈ Z, θ 0 , θ 1 ∈ H and h ∈ H. Let us look at the function

f : t ∈ [0, 1] → ∇ z (θ t ) -∇ z (θ 0 ) -t∇ 2 z (θ 0 )(θ 1 -θ 0 ) • h. We have f (t) = ∇ 3 z (θ t )[θ 1 -θ 0 , θ 1 -θ 0 , h].
By the self-concordant assumption, we have

f (t) ≤ sup g∈ϕ(z) |g • h|∇ 2 z (θ t )[θ 1 -θ 0 , θ 1 -θ 0 ] ≤ sup g∈ϕ(z) |g • h| exp(t sup g∈ϕ(z) |g • θ 1 -θ 0 |) θ 1 -θ 0 2 ∇ 2 z (θ 0 ) . Integrating this knowing f (0) = f (0) = 0 yields |f (1)| ≤ sup g∈ϕ(z) |g • h| ψ( sup g∈ϕ(z) |g • (θ 1 -θ 0 )|) θ 1 -θ 0 2 ∇ 2 z (θ 0 ) .
Hence :

∇ z (θ 1 )-∇ z (θ 0 ) ≤ ∇ 2 z (θ 0 ) θ 1 -θ 0 + ϕ(z) ψ( sup g∈ϕ(z) |g •(θ 1 -θ 0 )|) ∇ 2 z (θ 0 ) θ 1 -θ 0 2
where ψ(t) = (e tt -1)/t 2 . Then, noting that ∇ 2 z (θ) ≤ Tr(∇ 2 z (θ)), we have proved our lemma.

Lemma 4.13. Assume that there exists θ 0 such that

sup z∈supp(µ) Tr ∇ 2 z (θ 0 ) < ∞, sup z∈supp(µ) ∇ z (θ 0 ) < ∞, sup z∈supp(µ) | z (θ 0 )| < ∞. Then • For any θ ∈ H, sup z∈supp(µ) | z (θ)| < ∞ • For any θ 0 ∈ H, T ≥ θ 0 , θ ≤ T, z ∈ supp(µ), we have: | z (θ)| ≤ | z (θ 0 )| + 2 ∇ z (θ 0 ) T + ψ(2R µ T ) Tr(∇ 2 z (θ 0 )) T 2 .
Proof. Proceeding as in the proof of Eq. (4.30), we get

∀z ∈ Z, ∀θ 0 , θ 1 ∈ H, 0 ≤ z (θ 1 )-z (θ 0 )-∇ z (θ 0 )(θ 1 -θ 0 ) ≤ ψ( sup g∈ϕ(z) |g•(θ 1 -θ 0 )|) θ 1 -θ 0 2 ∇ 2 z (θ 0 )
where ψ(t) = (e tt -1)/t 2 .

To conclude, we give the following result.

| z (θ)|, sup z∈supp(ρ) ∇ z (θ) , sup z∈supp(ρ) Tr ∇ 2 z (θ) < ∞.
Proof. We combine the results given in Lemmas 4.11 to 4.13.

4.G .2 Bernstein inequalities for operators

We start by proposing a slight modification of Proposition 6 in (Rudi and Rosasco, 2017). First we need to introduce the following quantitity and some notation for Hermitian operators. We denote by is the partial order between positive semidefinite Hermitian operators. Let A, B be bounded Hermitian operators on H,

A B ⇐⇒ v • (Av) ≤ v • (Bv), ∀v ∈ H ⇐⇒ B -A is positive semidefinite.
Let q be a random positive semi-definite operator and let Q := E [q], denote by F ∞ (λ) the function of λ defined as

F ∞ (λ) := ess sup Tr Q -1/2 λ qQ -1/2 λ ,
where ess sup is the essential support of q.

Remark 8. Note that if Tr(q) ≤ c 0 , for a c 0 > 0 almost surely, then

F ∞ (λ) ≤ c 0 /λ. Vice versa, if F ∞ (λ 0 ) < ∞ for a given λ 0 > 0, then Tr(q) ≤ ( Q + λ 0 )F ∞ (λ 0 ) almost surely, moreover F ∞ (λ) < Q +λ 0 Q +λ F(λ 0 )
for any λ > 0. Proposition 4.9 (Prop. 6 of (Rudi and Rosasco, 2017)). Let q 1 , ..., q n be identically distributed random positive semi-definite operators on a separable Hilbert space H such that the q are trace class and

Q = E [q]. Let Q n = 1 n n i=1 q i and take 0 < λ ≤ Q and assume F ∞ (λ) < ∞.
For any δ > 0, the following holds with probability at least 1 -δ:

Q -1/2 λ (Q -Q n ) Q -1/2 λ ≤ 2β(1 + F ∞ (λ)) 3n + 2βF ∞ (λ) n , β = log 8F ∞ (λ) δ
Proof. Use Proposition 3 of (Rudi and Rosasco, 2017) and proceed as in the proof of Proposition 6 of (Rudi and Rosasco, 2017) except that we bound Tr(Q

-1 λ Q) ≤ F ∞ (λ) instead of bounding Tr(Q -1 λ Q) ≤ Tr(Q)
λ , we find this result.

Here we slightly extend the results of Prop. 8 and Prop. 6 of (Rudi and Rosasco, 2017), to extend the range of λ for which the result on the partial order between operators holds, from 0 < λ < Q to λ > 0.

Proposition 4.10 (Prop. 8 together with Prop. 6 of (Rudi and Rosasco, 2017)). Let q 1 , ..., q n be identically distributed random positive semi-definite operators on a separable Hilbert space H such that the q are trace class and

Q = E [q]. Let Q n = 1 n n i=1 q i . Let any δ ∈ (0, 1], t > 0, 0 < λ ≤ Q and assume F ∞ (λ) < ∞, when n ≥ 8F ∞ (λ) log 8F ∞ (λ) δ 1 4t 2 + 1 t (4.49)
then the following holds with probability at least 1 -δ:

Q -1/2 λ (Q -Q n ) Q -1/2 λ ≤ t. (4.50)
Moreover let λ > 0, δ ∈ (0, 1] and Eq. (4.49) is satisfied for t ≤ 1/2, then the following holds with probability at least 1 -δ,

Q λ 2Q n,λ , ⇐⇒ Q -1/2 n,λ Q 1/2 λ 2 ≤ 2. (4.51) Finally, let λ > 0, δ ∈ (0, 1], Eq. (4.49) is satisfied for t ≤ 1/2 and n ≥ 16 c 2 0 Q 2 log 2 δ ,
with c 0 = ess sup Tr(q), then the following holds with probability at least 1 -δ,

Q n,λ 3 2 Q λ , ⇐⇒ Q 1/2 n,λ Q -1/2 λ 2 ≤ 3/2. (4.52)
Proof. Point 1) Let δ ∈ (0, 1] and 0 < λ ≤ Q. Using proposition 4.9, we have that with probability at least 1 -δ,

Q -1/2 λ (Q -Q n ) Q -1/2 λ ≤ 2β(1 + F ∞ (λ)) 3n + 2βF ∞ (λ) n , β = log 8F ∞ (λ) δ . Now note that if λ ≤ Q , we have 1 2 ≤ Q Q + λ = Q -1 λ Q ≤ Tr Q -1 λ Q ≤ F ∞ (λ).
Thus we can bound 1 + F ∞ (λ) ≤ 3F ∞ (λ), and we rewrite the previous bound

Q -1/2 λ (Q -Q n ) Q -1/2 λ ≤ 2βF ∞ (λ) n + 2βF ∞ (λ) n , β = log 8F ∞ (λ) δ . Point 2) Now let t > 0, δ ∈ (0, 1] and 0 < λ ≤ Q . If n ≥ 8F ∞ (λ)β 1 4t 2 + 1 t , then Q -1/2 λ (Q -Q n ) Q -1/2 λ ≤ t.
Indeed, assume we want to find n 0 > 0 for which for all n ≥ n 0

, A n + B n ≤ 1 2 where A, B ≥ 0. setting x = √ n, this is equivalent to finding x 0 such that ∀x ≥ x 0 , x 2 2 - √ Bx -A ≥ 0. A sufficient condition for this is that x ≥ √ B + √ B + 2A. Thus, since A, B ≥ 0, the condition x ≥ 2 √ B + 2A is sufficient, hence the condition n ≥ 4(B + 2A
). Then we apply this to the following A and B to obtain the condition.

A = βF ∞ (λ) t , B = βF ∞ (λ) 2t 2 .
Point 3) When λ > Q , the result is obtained noting that

Q 1/2 λ Q -1/2 n,λ 2 ≤ Q + λ λ = 1 + Q λ ≤ 2.
When, on the other hand 0 < λ ≤ Q , the final result is obtained by applying Prop. 6 and Prop. 8 of (Rudi and Rosasco, 2017), or equivalently applying Eq. (4.50), with t = 1/2, for which the following holds with probability 1 -δ:

Q -1/2 λ (Q -Q n ) Q -1/2 λ
≤ t and noting that,

Q 1/2 λ Q -1/2 n,λ 2 ≤ 1 1 -Q -1/2 λ (Q -Q n ) Q -1/2 λ ≤ 2.
To conclude this point, we recall that, given two Hermitian operators A, B and t > 0, the inequality A tB is equivalent to B -1/2 AB -1/2 tI, when B is invertible. Since B -1/2 AB -1/2 and tI are commutative, then

B -1/2 AB -1/2 tI is equivalent to v • (B -1/2 AB -1/2 v) ≤ t v 2 for any v ∈ H, which in turn is equivalent to B -1/2 AB -1/2 ≤ t. So A 1/2 B -1/2 2 ≤ t ⇐⇒ A tB.
Point 4) First note that

Q -1/2 λ Q 1/2 n,λ 2 ≤ 1 + Q -1/2 λ (Q -Q n ) Q -1/2 λ . (4.53)
When 0 < λ ≤ Q , by applying Eq. (4.50) with t = 1/2, we have with probability 1 -δ:

Q -1/2 λ (Q -Q n ) Q -1/2 λ
≤ t, moreover by Eq. (4.53) we have

Q -1/2 λ Q 1/2 n,λ 2 ≤ 1 + t ≤ 3/2.
When instead λ > Q , we consider the following decomposition

Q -1/2 λ (Q -Q n ) Q -1/2 λ ≤ 1 λ Q -Q n ≤ 1 λ Q -Q n HS ,
where we denote by • HS , the Hilbert-Schmidt norm (i.e. A 2 HS = Tr(A * A)) and Q -Q n HS is well defined since both Q, Q n are trace class. Now since the space of Hilbert-Schmidt operators on a separable Hilbert space is itself a separable Hilbert space and q are bounded almost surely by c 0 := ess sup Tr(q), we can concentrate Q -Q n HS via Bernstein inequality for random vectors (e.g. Thm. 3.3.4 of [START_REF] Yurinsky | Sums and Gaussian vectors[END_REF], obtaining with probability at least 1 -δ

Q -Q n HS ≤ 2c 0 log 2 δ n + 2c 2 0 log 2 δ n ≤ Q /2,
where the last step is due to the fact that we require n ≥ 16c 2 0 (log 2 δ )/ Q 2 , and the fact that by construction Q ≤ B. Then,

Q -1/2 λ Q 1/2 n,λ 2 ≤ 1 + Q 2λ ≤ 3/2.
The final result on is obtained as for Point 5.

CHAPTER 4. BEYOND LEAST SQUARES Remark 9. Let Tr(q) ≤ c 0 almost surely, for a c 0 > 0. Then F ∞ (λ) ≤ c 0 /λ. So Eq. (4.49) is satisfied when

n ≥ 8c 0 λ log 8c 0 λ δ 1 4t 2 + 1 t , since F ∞ (λ) ≤ c 0 /λ

4.G .3 Last technical lemmas

Lemma 4.14. Let a 1 , a 2 , A ≥ 0 and δ > 0.

If

n ≥ 4a 2 1 A 2 log 2 2a 1 a 2 A 2 δ , then n ≥ a 1 An 1/2 log a 2 An 1/2 δ . Proof. Indeed, note that n ≥ a 1 An 1/2 log a 2 An 1/2 δ ⇐⇒ a 1 A n 1/2 log a 2 An 1/2 δ ≤ 1. Now use the fact that for A, B ≥ 0, k ≥ 2A log(2AB) implies A k log(Bk) ≤ 1. Indeed, log(Bk) = log(2AB) + log Bk 2AB = log(2AB) + log k 2A ≤ log(2AB) + k 2A .
Hence, multiplying by A k , we get the result.

We apply this to A = a 1 A, B = a 2 A δ and k = n 1/2 to get the bound.

Lemma 4.15. Let a 1 , a 2 , A ≥ 0 and δ > 0.

Let p ∈ [ 1 2 , 1). If n 1-p ≥ 2 1 1 -p a 1 A log 2a 1 (a 2 ∨ 1) 1 1 -p A 2 1 δ , then n ≥ a 1 An p log a 2 An p δ . Proof. 1) Let C 1 , C 2 ≥ 0, and p ∈ [0, 1). Then n ≥ C 1 n p log(C 2 n p ) ⇐⇒ C 1 p 1-p n 1-p log C (1-p)/p 2 n 1-p ≤ 1. Now use the fact that for A, B ≥ 0, k ≥ 2A log(2AB) implies A k log(Bk) ≤ 1 (see proof of Lemma 4.14). Thus, n 1-p ≥ 2C 1 p 1-p log 2C 1 p 1-p C (1-p)/p 2 is a sufficient condition. 2) Now taking C 1 = a 1 A and C 2 = a 2 A δ , we find that n 1-p ≥ 2 p 1 -p a 1 A log 2a 1 a (1-p)/p 2 p 1 -p A 1/p ( 1 δ ) (1-p)/p .
Since 0.5 ≤ p ≤ 1, we see that 1-p p ≤ 1 and 1 p ≤ 2 and thus we get our final sufficient condition.

n 1-p ≥ 2 1 1 -p a 1 A log 2a 1 (a 2 ∨ 1) 1 1 -p A 2 1 δ .

Introduction

Minimization algorithms constitute a crucial algorithmic part of many machine learning methods, with algorithms available for a variety of situations [START_REF] Bottou | Optimization methods for large-scale machine learning[END_REF]. In this paper, we focus on finite sum problems of the form

min x∈H f λ (x) = f (x) + λ 2 x 2 , with f (x) = 1 n n i=1 f i (x),
where H is a Euclidean or a Hilbert space, and each function is convex and smooth. The running-time of minimization algorithms classically depends on the number of functions n, the explicit (for Euclidean spaces) or implicit (for Hilbert spaces) dimension d of the search space, and the condition number of the problem, which is upper bounded by κ = L/λ, where L characterizes the smoothness of the functions f i , and λ the regularization parameter.

In the last few years, there has been a strong focus on problems with large n and d, leading to first-order (i.e., gradient-based) stochastic algorithms, culminating in a sequence of linearly convergent algorithms whose running time is favorable in n and d, but scale at best in √ κ Defazio, Bach, and Lacoste-Julien (2014); [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]; [START_REF] Defazio | A simple practical accelerated method for finite sums[END_REF]; Allen-Zhu (2017). However, modern problems lead to objective functions with very large condition numbers, i.e., in many learning problems, the regularization parameter that is optimal for test predictive performance may be so small that the scaling above in √ κ is not practical anymore (see examples in Sect. 5.5 ).

These ill-conditioned problems are good candidates for second-order methods (i.e., that use the Hessians of the objective functions) such as Newton method. These methods are traditionally discarded within machine learning for several reasons: (1) they are usually adapted to high precision results which are not necessary for generalization to unseen data for machine learning problems [START_REF] Bottou | The trade-offs of large scale learning[END_REF], (2) computing the Newton step ∆ λ (x) = ∇ 2 f λ (x) -1 ∇f λ (x) requires to form the Hessian and solve the associated linear system, leading to complexity which is at least quadratic in d, and thus prohibitive for large d, and (3) the global convergence properties are not applicable, unless the function is very special, i.e., self-concordant [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF] (which includes only few classical learning problems), so they often are only shown to converge in a small area around the optimal x.

In this paper, we argue that the three reasons above for not using Newton method can be circumvented to obtain competitive algorithms: (1) high absolute precisions are indeed not needed for machine learning, but faced with strongly ill-conditioned problems, even a lowprecision solution requires second-order schemes;

(2) many approximate Newton steps have been designed for approximating the solution of the associated large linear system A. Erdogdu and Montanari (2015); Roosta-Khorasani and Mahoney (2019); Pilanci and Wainwright (2017); [START_REF] Bollapragada | Exact and inexact subsampled newton methods for optimization[END_REF];

(3) we propose a novel second-order method which is globally convergent and which is based on performing approximate Newton methods for a certain class of so-called generalized self-concordant functions which includes logistic regression [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF]. For these functions, the conditioning of the problem is also characterized by a more local quantity: κ = R 2 /λ, where R characterizes the local evolution of Hessians. This leads to second-order algorithms which are competitive with first-order algorithms for well-conditioned problems, while being superior for ill-conditioned problems which are common in practice.

Contributions. We make the following contributions:

(a) We build a global second-order method for the minimization of f λ , which relies only on computing approximate Newton steps of the functions f µ , µ ≥ λ. The number of such steps will be of order O(c log κ + log 1 ) where is the desired precision, and c is an explicit constant. In the parametric setting (H = R d ), c can be as bad as √ κ in the worst-case but much smaller in theory and practice. Moreover in the non-parametric/kernel machine learning setting (H infinite dimensional), c does not depend on the local condition number κ .

(b) Together with the appropriate quadratic solver to compute approximate Newton steps, we obtain an algorithm with the same scaling as regular first-order methods but with an improved behavior, in particular in ill-conditioned problems. Indeed, this algorithm matches the performance of the best quadratic solvers but covers any generalized self-concordant function, up to logarithmic terms.

(c) In the non-parametric/kernel machine learning setting we provide an explicit algorithm combining the previous scheme with Nyström projections techniques. We prove that it achieves optimal generalization bounds with O(ndf λ ) in time and O(df 2 λ ) in memory, where n is the number of observations and df λ is the associated degrees of freedom. In particular, this is the first large-scale algorithm to solve logistic and softmax regression in the non-parametric setting with large condition numbers and theoretical guarantees.

.1 Comparison to related work

We consider two cases for H and the functions f i that are common in machine learning: H = R d with linear (in the parameter) models with explicit feature maps, and H infinite-dimensional, corresponding in machine learning to learning with kernels [START_REF] Shawe | Kernel Methods for Pattern Analysis[END_REF]. Moreover in this section we first consider the quadratic case, for example the squared loss in machine learning (i.e., f i (x) = 1 2 (x z i -y i ) 2 for some z i ∈ H, y i ∈ R). We first need to introduce the Hessian of the problem, for any λ > 0, define

H(x) := ∇ 2 f (x), H λ (x) := ∇ 2 f λ (x) = H(x) + λI,
in particular we denote by H (and analogously H λ ) the Hessian at optimum (which in case of squared loss corresponds to the covariance matrix of the inputs).

Quadratic problems and H = R d (ridge regression). The problem then consists in solving a (ill-conditioned) positive semi-definite symmetric linear system of dimension d × d. Methods based on randomized linear algebra, sketching and suitable subsampling Drineas, Mahoney, Muthukrishnan, and Sarlós (2011); Drineas, Magdon-Ismail, Mahoney, and Woodruff (2012); [START_REF] Boutsidis | Improved matrix algorithms via the subsampled randomized hadamard transform[END_REF] are able to find the solution with precision in time that is O((nd + min(n, d) 3 ) log(L/λ )), so essentially independently of the condition number, because of the logarithmic complexity in λ.

Quadratic problems and H infinite-dimensional (kernel ridge regression).

Here the problem corresponds to solving a (ill-conditioned) infinite-dimensional linear system in a reproducing kernel Hilbert space [START_REF] Shawe | Kernel Methods for Pattern Analysis[END_REF]. Since however the sum defining f is finite, the problem can be projected on a subspace of dimension at most n [START_REF] Aronszajn | Theory of reproducing kernels[END_REF], leading to a linear system of dimension n × n. Solving it with the techniques above would lead to a complexity of the order O(n 2 ), which is not feasible on massive learning problems (e.g., n ≈ 10 7 ). Interestingly these problems are usually approximately low-rank, with the rank represented by the so called effective-dimension df λ [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF], counting essentially the eigenvalues of the problem larger than λ,

df λ = Tr(HH -1 λ ). (5.1)
Note that df λ is bounded by min{n, L/λ} and in many cases df λ min(n, L/λ). Using suitable projection techniques, like Nyström Williams and Seeger (2001) or random features [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF] it is possible to further reduce the problem to dimension df λ , for a total cost to find the solution of O(ndf 2 λ ). Finally recent methods [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF], combining suitable projection methods with refined preconditioning techniques, are able to find the solution with precision compatible with the optimal statistical learning error [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF] in time that is O(ndf λ log(L/λ)), so being essentially independent of the condition number of the problem.

Convex problems and explicit features (logistic regression).

When the loss function is self-concordant it is possible to leverage the fast techniques for linear systems in approximate Newton algorithms [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF] (see more in Sec. 5.2 ), to achieve the solution in essentially O(nd + min(n, d) 3 ) time, modulo logarithmic terms. However only few loss functions of interest are self-concordant, in particular the widely used logistic and soft-max losses are not self-concordant, but generalized-self-concordant [START_REF] Bach | Self-concordant analysis for logistic regression[END_REF]. In such cases we need to use (accelerated/stochastic) first order optimization methods to enter in the quadratic convergence region of Newton methods [START_REF] Agarwal | Second-order stochastic optimization for machine learning in linear time[END_REF], which leads to a solution in O(dn + d nL/λ + min(n, d) 3 ) time, which does not present any improvement on a simple accelerated first-order method. Globally convergent second-order methods have also been proposed to solve such problems Karimireddy, Stich, and Jaggi (2018), but the number of Newton steps needed being bounded only by L/λ, they lead to a solution in O(L/λ (nd + min(n, d) 3 )). With λ that could be as small as 10 -12 in modern machine learning problems, this makes both these kind of approaches expensive from a computational viewpoint for ill-conditioned problems. For such problems, with our new global second-order scheme, the algorithm we propose achieves instead a complexity of essentially O((nd + min(n, d) 3 ) log(R 2 /λ )) (see Theorem 5.1).

Convex problems and H infinite-dimensional (kernel logistic regression). Analogously to the case above, it is not possible to use Newton methods profitably as global optimizers on losses that are not self-concordant as we see in Sec. 5.3 . In such cases by combining projecting techniques developped in Sec. 5.4 and accelerated first-order optimization methods, it is possible to find a solution in O(ndf λ + df λ nL/λ) time. This can still be prohibitive in the very small regularization scenario, since it strongly depends on the condition number L/λ. In Sec. 5.4 we suitably combine our optimization algorithm with projection techniques achieving optimal statistical learning error Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]

in essentially O(ndf λ log(R 2 /λ)).
First-order algorithms for finite sums. In dimension d, accelerated algorithms for stronglyconvex smooth (not necessarily self-concordant) finite sums, such as K-SVRG Allen-Zhu (2017), have a running time proportional O((n + nL/λ)d). This can be improved with preconditioning to O((n + dL/λ)d) for large n Agarwal, Bullins, and [START_REF] Agarwal | Second-order stochastic optimization for machine learning in linear time[END_REF]. Quasi-Newton methods can also be used [START_REF] Gower | Accelerated stochastic matrix inversion: general theory and speeding up BFGS rules for faster second-order optimization[END_REF], but typically without the guarantees that we provide in this paper (which are logarithmic in the condition number in natural scenarios).

Background: Newton methods and generalized self concordance

In this section we start by recalling the definition of generalized self concordant functions and motivate it with examples. We then recall basic facts about Newton and approximate Newton methods, and present existing techniques to efficiently compute approximate Newton steps. We start by introducing the definition of generalized self-concordance, that here is an extension of the one in Bach (2010).

Definition 5.1 (generalized self-concordant (GSC) function). Let H be a Hilbert space. We say that f is a generalized self-concordant function on G ⊂ H, when G is a bounded subset of H and f is a convex and three times differentiable mapping on H such that

∀x ∈ H, ∀h, k ∈ H, ∇ (3) f (x)[h, k, k] ≤ sup g∈G |g • h| ∇ 2 f (x)[k, k].
We will usually denote by R the quantity sup g∈G g < ∞ and often omit G when it is clear from the context (for simplicity think of G as the ball in H centered in zero and with radius R > 0, then sup g∈G |g • h| = R h ). The globally convergent second-order scheme we present in Sec. 5.3 is specific to losses which satisfy this generalized self-concordance property. The following loss functions, which are widely used in machine learning, are generalized-self-concordant, and motivate this work.

Example 5.1 (Application to finite-sum minimization). The following loss functions are generalized self-concordant functions, but not self-concordant: (a) Logistic regression:

f i (x) = log(1 + exp(-y i w i x)), where x, w i ∈ R d and y i ∈ {-1, 1}. (b) Softmax regression: f i (x) = log k j=1 exp(x j w i ) -x y i w i
, where now x ∈ R d×k and y i ∈ {1, . . . , k} and x j denotes the j-th column of x. (c) Generalized linear models with bounded features (see details in Bach (2014, Sec. 2.1)), which include conditional random fields [START_REF] Sutton | An introduction to conditional random fields[END_REF]. (d) Robust regression: f i (x) = ϕ(y i -w i x) with ϕ(u) = log(e u + e -u ).

Note that these losses are not self-concordant in the sense of [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF]. Moreover, even if the losses f i are self-concordant, the objective function f is not necessarily self-concordant, making any attempt to prove the self-concordance of the objective function f almost impossible.

Newton method (NM). Given x 0 ∈ H, the Newton method consists in doing the following update:

x t+1 = x t -∆ λ (x t ), ∆ λ (x t ) := H -1 λ (x t )∇f λ (x t ). (5.
2)

The quantity ∆ λ (x) := H -1 λ (x)∇f λ (x) is called the Newton step at point x, and x -∆ λ (x) is the minimizer of the second order approximation of f λ around x. Newton methods enjoy the following key property: if x 0 is close enough to the optimum, the convergence to the optimum is quadratic and the number of iterations required to a given precision is independent of the condition number of the problem [START_REF] Boyd | Convex optimization[END_REF].

However Newton methods have two main limitations: (a) the region of quadratic convergence can be quite small and reaching the region can be computationally expensive, since it is usually done via first order methods [START_REF] Agarwal | Second-order stochastic optimization for machine learning in linear time[END_REF] that converge linearly depending on the condition number of the problem, (b) the cost of computing the Hessian can be really expensive when n, d are large, and also (c) the cost of computing ∆ λ (x t ) can be really prohibitive. In the rest of the section we recall some ways to deal with (b) and (c). Our main result of Sec. 5.3 is to provide globalization scheme for the Newton method to tackle problem (a), which is easily integrable with approximate techniques to deal with (b) ans (c), to make second-order technique competitive.

Approximate Newton methods (ANM) and approximate solutions to linear systems.

Computing exactly the Newton increment ∆ λ (x t ), which corresponds essentially to the solution of a linear system, can be too expensive when n, d are large. A natural idea is to approximate the Newton iteration, leading to approximate Newton methods,

x t+1 = x t -∆ λ (x t ), ∆ λ ≈ ∆ λ (x t ). (5.3)
In this paper, more generally we consider any technique to compute ∆ λ (x t ) that provides a relative approximation Deuflhard (2011) of ∆ λ (x t ) defined as follows.

Definition 5.2 (relative approximation). Let ρ < 1, let A be an invertible positive definite Hermitian operator on H and b in H. We denote by LinApprox(A, b, ρ) the set of all ρ-relative

approximations of z * = A -1 b, i.e., LinApprox(A, b, ρ) = {z ∈ H | z -z * A ≤ ρ z * A }.
Sketching and subsampling for approximate Newton methods. Many techniques for approximating linear systems have been used to compute ∆ λ , in particular sketching of the Hessian matrix via fast transforms and subsampling (see [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF]; [START_REF] Bollapragada | Exact and inexact subsampled newton methods for optimization[END_REF]; [START_REF] Agarwal | Second-order stochastic optimization for machine learning in linear time[END_REF] and references therein). Assuming for simplicity that f i = i (w i x), with i : R → R and w i ∈ H, it holds:

H(x) = 1 n n i=1 (2) i (w i x)w i w i = V x V x , (5.4) with V x ∈ R n×d = D x W , where D x ∈ R n×n is a diagonal matrix defined as (D x ) ii = ( (2)
i (w i x)) 1/2 and W ∈ R n×d defined as W = (w 1 , . . . , w n ) .

Both sketching and subsampling methods approximate

z * = H λ (x) -1 ∇f λ (x) with z = H λ (x) -1 ∇f λ (x), in particular, in the case of subsampling H(x) = Q j=1 p j w i j w i j where Q min(n, d), (p j ) n j=1
are suitable weights and (i j ) Q j=1 are indices selected at random from {1, . . . , n} with suitable probabilities. Sketching methods instead use H(x) = V x V x , with V x = ΩV x with Ω ∈ R Q×n a structured matrix such that computing V x has a cost in the order of O(nd log n); to this end usually Ω is based on fast Fourier or Hadamard transforms [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF]. Note that essentially all the techniques used in approximate Newton methods guarantee relative approximation. In particular the following results can be found in the literature (see Lemmas 5.28 and 5.29 in Sec. 5.I and Pilanci and Wainwright (2017), Lemma 2 for more details).

λ (x), b, ρ), in O(Q 2 d + Q 3 + c) time, O(Q 2 + d) space:
(a) Subsampling with uniform sampling (see [START_REF] Roosta | Sub-sampled Newton methods[END_REF]; [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]), where Q = O(ρ -2 a/λ log 1 λδ ) and c = O(1). (b) Subsampling with approximate leverage scores Roosta-Khorasani and Mahoney (2019); [START_REF] Alaoui | Fast randomized kernel ridge regression with statistical guarantees[END_REF]; [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]), where [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF]. Note that d f λ ≤ min(n, d).

Q = O(ρ -2 d f λ log 1/λδ), c = O(min(n, a/λ) d f λ 2 ) and d f λ = Tr(W W (W W + λ/aI) -1 ) Rudi,
(c) Sketching with fast Hadamard transform [START_REF] Pilanci | Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence[END_REF], where

Q = O(ρ -2 d f λ log a/λδ), c = O(nd log n).

Globally convergent scheme for ANM algorithms on GSC functions

The algorithm is based on the observation that when f λ is generalized self concordant, there exists a region where t steps of ANM converge as fast as 2 -t . Our idea is to start from a very large regularization parameter λ 0 , such that we are sure that x 0 is in the convergence region and perform some steps of ANM such that the solution enters in the convergence region of f λ 1 , with λ 1 = qλ 0 with q < 1, and to iterate this procedure until we enter the convergence region of f λ . First we define the region of interest and characterize the behavior of NM and ANM in the region, then we analyze the globalization scheme.

Preliminary results: the Dikin ellipsoid. We consider the following region that we prove to be contained in the region of quadratic convergence for the Newton method and that will be useful to build the globalization scheme. Let c, R > 0 and f λ be generalized self-concordant with coefficient R, we call Dikin ellipsoid and denote by D λ (c) the region

D λ (c) := x | ν λ (x) ≤ c √ λ/R , with ν λ (x) := ∇f λ (x) H -1 λ (x) ,
where ν λ (x) is usually called the Newton decrement and x A stands for A 1/2 x .

Lemma 5.2. Let λ > 0, c ≤ 1/7, let f λ be generalized self-concordant and x ∈ D λ (c). Then it holds:

1 4 ν λ (x) 2 ≤ f λ (x) -f λ (x λ ) ≤ ν λ (x) 2
. Moreover Newton method starting from x 0 has quadratic convergence, i.e., let x t be obtained via t ∈ N steps of Newton method in Eq. (5.2), then ν λ (x t ) ≤ 2 -(2 t -1) ν λ (x 0 ). Finally, approximate Newton methods starting from x 0 have a linear convergence rate, i.e., let x t given by Eq. (5.3), with ∆ t ∈ LinApprox(H λ (x t ), ∇f λ (x t ), ρ) and ρ ≤ 1/7, then ν λ (x t ) ≤ 2 -t ν λ (x 0 ). This result is proved in Lemma 5.11 in Sec. 5.B .3. The crucial aspect of the result above is that when x 0 ∈ D λ (c), the convergence of the approximate Newton method is linear and does not depend on the condition number of the problem. However D λ (c) itself can be very small depending on √ λ/R. In the next subsection we see how to enter in D λ (c) in an efficient way.

Entering the Dikin ellipsoid using a second-order scheme. The lemma above shows that D λ (c) is a good region where to use the approximate Newton algorithm on GSC functions. However the region itself is quite small, since it depends on √ λ/R. Some other globalization schemes arrive to regions of interest by first-order methods or back-tracking schemes [START_REF] Agarwal | Second-order stochastic optimization for machine learning in linear time[END_REF]; A. [START_REF] Murat | Convergence rates of sub-sampled Newton methods[END_REF]. However such approaches require a number of steps that is usually proportional to L/λ making them non-beneficial in machine learning contexts. Here instead we consider the following simple scheme where ANM ρ (f λ , x, t) is the result of a ρ-relative approximate Newton method performing t steps of optimization starting from x.

The main ingredient to guarantee the scheme to work is the following lemma (see Lemma 5.13 in Sec. 5.C .1 for a proof).

CHAPTER 5. SECOND ORDER STRIKES BACK Lemma 5.3. Let µ > 0, c < 1 and x ∈ H. Let s = 1 + R x /c, then for q ∈ [1 -2/(3s), 1)

D µ (c/3) ⊆ D qµ (c).
Now we are ready to show that we can guarantee the loop invariant

x k ∈ D µ k (c). Indeed assume that x k-1 ∈ D µ k-1 (c). Then ν µ k-1 (x k-1 ) ≤ c √ µ k-1 /R
. By taking t = 2, ρ = 1/7, and performing Lemma 5.3. Now we are ready to state our main theorem of this section.

x k = ANM ρ (f µ k-1 , x k-1 , t), by Lemma 5.2, ν µ k-1 (x k ) ≤ 1/4ν µ k-1 (x k-1 ) ≤ c/4 √ µ k-1 /R, i.e., x k ∈ D µ k-1 (c/4). If q k is large enough, this implies that x k ∈ D q k µ k-1 (c) = D µ k (c), by

Proposed Globalization Scheme

Phase I: Getting in the Dikin ellispoid of

f λ Start with x 0 ∈ H, µ 0 > 0, t, T ∈ N and (q k ) k∈N ∈ (0, 1]. For k ∈ N x k+1 ← ANM ρ (f µ k , x k , t) µ k+1 ← q k+1 µ k Stop when µ k+1 < λ and set x last ← x k .
Phase II: reach a certain precision starting from inside the Dikin ellipsoid Return x ← ANM ρ (f λ , x last , T ) Fully adaptive method. The scheme presented above converges with the following parameters.

Theorem 5.1. Let > 0. Set µ 0 = 7R ∇f (0) , x 0 = 0, and perform the globalization scheme above for ρ ≤ 1/7, t = 2, and

q k = 1/3+7R x k 1+7R x k , T = log 2 1 ∨ (λ -1 /R 2 ) .
Then denoting by K the number of steps performed in the Phase I, it holds:

f λ ( x) -f λ (x λ ) ≤ , K ≤ (3 + 11R x λ ) log(7R ∇f (0) /λ) .
Note that the theorem above (proven in Sec. 5.C .3) guarantees a solution with error with K steps of ANM each performing 2 iterations of approximate linear system solving, plus a final step of ANM which performs T iterations of approximate linear system solving. In case of

f i (x) = i (w i x), with i : R → R, w i ∈ H with (2)
i ≤ a, for a > 0, the final runtime cost of the proposed scheme to achieve precision , when combined with of the methods for approximate linear system solving from Lemma 5.1 (i.e. sketching), is O(Q 2 + d) in memory and

O (nd log n + dQ 2 + Q 3 ) R x λ log R λ + log λ R in time, Q = O d f λ log 1 λδ ,
where d f λ , defined in Lemma 5.1, measures the effective dimension of the correlation matrix W W with W = (w 1 , . . . , w n ) ∈ R n×d , corresponding essentially to the number of eigenvalues of W W larger than λ/a. In particular note that d f λ ≤ min(n, d, rank(W ), ab 2 /λ), with b := max i w i , and usually way smaller than such quantities.

Remark 10. The proposed method does not depend on the condition number of the problem L/λ, but on the term R x λ which can be in the order of R/ √ λ in the worst case, but usually way smaller. For example, it is possible to prove that this term is bounded by an absolute constant not depending on λ, if at least one minimum for f exists. In the appendix (see proposition 5.7), we show a variant of this adaptive method which can leverage the regularity of the solution with respect to the Hessian, i.e., depending on the smaller quantity R √ λ x λ H -1 λ (x λ ) instead of R x λ .

. APPLICATION TO KERNEL METHODS

Finally note that it is possible to use q k = q fixed for all the iterations and way smaller than the one in Theorem 5.1, depending on some regularity properties of H (see proposition 5.8 in Sec. 5.C .2).

Application to the non-parametric setting: Kernel methods

In supervised learning the goal is to predict well on future data, given the observed training dataset. Let X be the input space and Y ⊆ R p be the output space. We consider a probability distribution P over X × Y generating the data and the goal is to estimate g * : X → Y solving the problem

g * = arg min g:X →Y L(g), L(g) = E[ (g(x), y)], (5.5)
for a given loss function : Y × Y → R. Note that P is not known, and accessible only via the dataset (x i , y i ) n i=1 , with n ∈ N, independently sampled from P . A prototypical estimator for g * is the regularized minimizer of the empirical risk L(g) = 1 n n i=1 (g(x i ), y i ) over a suitable space of functions G. Given φ : X → H a common choice is to select G as the set of linear functions of φ(x), that is, G = {w φ(•) | w ∈ H}. Then the regularized minimizer of L, denoted by g λ , corresponds to

g λ (x) = w λ φ(x), w λ = arg min w∈H 1 n n i=1 f i (w) + λ w 2 , f i (w) = (w φ(x i ), y i ).
(5.6)

Learning theory guarantees how fast g λ converges to the best possible estimator g * with respect to the number of observed examples, in terms of the so called excess risk L( g λ ) -L(g * ). The following theorem recovers the minimax optimal learning rates for squared loss and extend them to any generalized self-concordant loss function.

Note on df λ . In this section, we always denote with df λ the effective dimension of the problem in Eq. (5.5). When the loss belongs to the family of generalized linear models (see Example 5.1) and if the model is well-specified, then df λ is defined exactly as in Eq. ( 5.1) otherwise we need a more refined definition (see [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] or Eq. (5.30) in Sec. 5.D ).

Theorem 5.2 (from Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], Thm. 4). Let λ > 0, δ ∈ (0, 1]. Let be generalized self-concordant with parameter R > 0 and sup x∈X φ(x) ≤ C < ∞.

Assume that there exists g * minimizing L. Then there exists c 0 not depending on n, λ, δ, df λ , C, g * , such that if df λ /n, b λ ≤ λ 1/2 /R, and n ≥ C/λ log(δ -1 C/λ) the following holds with probability 1 -δ:

L( g λ ) -L(g * ) ≤ c 0 df λ n + b 2 λ log(1/δ), b λ := λ g * H -1 λ (g * ) .
(5.7)

Under standard regularity assumptions of the learning problems Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], i.e., (a) the capacity condition σ j (H(g * )) ≤ Cj -α , for α ≥ 1, C > 0 (i.e., a decay of eigenvalues σ j (H(g * )) of the Hessian at the optimum), and (b) the source condition g * = H(g * ) r v, with v ∈ H and r > 0 (i.e., the control of the optimal g * for a specific Hessian-dependent norm), df λ ≤ C λ -1/α and b 2 λ ≤ C λ 1+2r , leading to the following optimal learning rate,

L( g λ ) -L(g * ) ≤ c 1 n -1+2rα 1+α+2rα log(1/δ), when λ = n -α 1+α+2rα .
(5.8)

CHAPTER 5. SECOND ORDER STRIKES BACK

Now we propose an algorithmic scheme to compute efficiently an approximation of g λ that achieves the same optimal learning rates. First we need to introduce the technique we are going to use.

Nyström projection. It consists in suitably selecting {x 1 , . . . , xM } ⊂ {x 1 , . . . , x n }, with M n and computing ḡM,λ , i.e., the solution of Eq. ( 5.6) over H M = span{φ(x 1 ), . . . , φ(x M )} instead of H. In this case the problem can be reformulated as a problem in

R M as ḡM,λ = ᾱ M,λ T -1 v(x), ᾱM,λ = arg min α∈R M fλ (α), f (α) = 1 n n i=1 fi (α) + λ α 2 , (5.9)
where fi

(α) = (v(x i ) T -1 α, y i ) and v(x) ∈ R M , v(x) = (k(x, x1 ), . . . , k(x, xM )) with k(x, x ) = φ(x) φ(x )
the associated positive-definite kernel Shawe-Taylor and Cristianini ( 2004), while T is the upper triangular matrix such that

K = T T, with K ∈ R M ×M with K ij = k(x i , xj ).
In the next theorem we characterize the sufficient M to achieve minimax optimal rates, for two standard techniques of choosing the Nyström points {x 1 , . . . , xM }.

Theorem 5.3 (Optimal rates for learning with Nyström). Let λ > 0, δ ∈ (0, 1]. Assume the conditions of Theorem 5.2. Then the excess risk of ḡM,λ is bounded with prob. 1 -2δ as in Eq. (5.7) (with c 1 ∝ c 1 ), when (1) Uniform Nyström method Rudi, Camoriano, and Rosasco (2015); Rudi, Carratino, and Rosasco ( 2017) is used and M ≥ C 1 /λ log(C 2 /λδ).

(2) Approximate leverage score method Alaoui and Mahoney (2015); Rudi, Camoriano, and Rosasco (2015); [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF] is used and M ≥ C 3 df λ log(C 4 /λδ).

Here C, C 1 , C 2 , C 4 do not depend on λ, n, M, df λ , δ.

Theorem 5.3 generalizes results for learning with Nyström and squared loss [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF], to GSC losses. It is proved in Theorem 5.6,in Sec. 5.D .4. As in [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF], Theorem 5.3 shows that Nyström is a valid technique for dimensionality reduction. Indeed it is essentially possible to project the learning problem on a subspace H M of dimension M = O(c/λ) or even as small as M = O(df λ ) and still achieve the optimal rates of Theorem 5.2. Now we are ready to introduce our algorithm.

Proposed algorithm. The algorithm conceptually consists in (a) performing a projection step with Nyström, and (b) solving the resulting optimization problem with the globalization scheme proposed in Sec. 5.3 based on ANM in Eq. ( 5.3). In particular, we want to avoid to apply explicitly T -1 to each v(x i ) in Eq. (5.9), which would require O(nM 2 ) time. Then we will use the following approximation technique based only on matrix vector products, so we can just apply T -1 to α at each iteration, with a total cost proportional only to O(nM + M 2 ) per iteration. Given α, ∇ fλ (α), we approximate z * = Hλ (α) -1 ∇ fλ (α), where Hλ is the Hessian of fλ (α), with z defined as z = prec-conj-grad t ( Hλ (α), ∇ fλ (α)), where prec-conj-grad t corresponds to performing t steps of preconditioned conjugate gradient [START_REF] Golub | Matrix Computations[END_REF] with preconditioner computed using a subsampling approach for the Hessian among the ones presented in Sec. 5.2 , in the paragraph starting with Eq. (5.4). The pseudocode for the whole procedure is presented in Alg. 1, Sec. 5.E . This technique of approximate linear system solving has been studied in [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF] in the context of empirical risk minimization for squared loss. Lemma 5.4 (Rudi, Carratino, and Rosasco (2017)). Let λ > 0, α, b ∈ R M . The previous method, applied with t = O(log 1/ρ), outputs an element of LinApprox( Hλ (α), b, ρ), with probability 1 -δ with complexity O((nM

+ M 2 Q + M 3 + c)t) in time and O(M 2 + n) in space, with Q = O(C 1 /λ log(C 1 /λδ)), c = O(1) if uniform sub-sampling is used or Q = O(C 2 df λ log(C 1 /λδ)), c = O(df 2 λ min(n, 1 λ ))
if sub-sampling with leverage scores is used [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF].

A more complete version of this lemma is shown in proposition 5.12 in Sec. 5.D .5. We conclude this section with a result proving the learning properties of the proposed algorithm.

Theorem 5.4 (Optimal rates for the proposed algorithms). Let λ > 0 and < λ/R 2 . Under the hypotheses of Theorem 5.3, if we set M as in Theorem 5.3, Q as in Lemma 5.4 and setting the globalization scheme as in Theorem 5.1, then the proposed algorithm (Alg. 1, Sec. 5.E ) finishes in a finite number of newton steps N ns = O(R g * log(C/λ) + log(C/ )) and returns a predictor

g Q,M,λ of the form g Q,M,λ = α T -1 v(x)
. With probability at least 1 -δ, this predictor satisfies:

L(g Q,M,λ ) -L(g * ) ≤ c 0 df λ n + b 2 λ + log(1/δ), b λ := λ g * H -1 λ (g * ) .
(5.10)

The theorem above (see proposition 5.14, Sec. 5.D .6 for exacts quantifications) shows that the proposed algorithm is able to achieve the same learning rates of plain empirical risk minimization as in Theorem 5.2. The total complexity of the procedure, including the cost of computing the preconditioner, the selection of the Nyström points via approximate leverage scores and also the computation of the leverage scores Rudi, Calandriello, Carratino, and Rosasco ( 2018) is then

O R g * log(R 2 /λ) n df λ log(Cλ -1 δ -1 ) c X + + df 3 λ log 3 (Cλ -1 δ -1 ) + min(n, C/λ) df 2 λ in time and O(df 2 λ log 2 (Cλ -1 δ -1
)) in space, where c X is the cost of computing the inner product k(x, x ) (in the kernel setting assumed when the input space

X is X = R p it is c = O(p)).
As noted in [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF], under the standard regularity assumptions on the learning problem seen above, df 2 λ ≤ df λ /λ ≤ n when the optimal λ is chosen. So the total computational complexity is

O R log(R 2 /λ) log 3 (Cλ -1 δ -1 ) g * • n • df λ • c X in time, O(df 2 λ • log 2 (Cλ -1 δ -1 )) in space.
First note, the fact that due to the statistical properties of the problem the complexity does not depend even implicitly on C/λ, but only on log(C/λ), so the algorithm runs in essentially O(ndf λ ), compared to O(df λ nC/λ) of the accelerated first-order methods we develop in Sec. 5.F and the O(ndf λ C/λ) of other Newton schemes (see Sec. 5.1 .1). To our knowledge, this is the first algorithm to achieve optimal statistical learning rates for generalized self-concordant losses and with complexity only O(ndf λ ). This generalizes similar results for squared loss Rudi, Carratino, and Rosasco (2017); Rudi, Calandriello, Carratino, and Rosasco (2018).

Experiments

The code necessary to reproduce the following experiments is available on GitHub at https: //github.com/umarteau/Newton-Method-for-GSC-losses-.

We compared the performances of our algorithm for kernel logistic regression on two large scale classification datasets (n ≈ 10 7 ), Higgs and Susy, pre-processed as in [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF]. We implemented the algorithm in pytorch and performed the computations on 1 Tesla P100-PCIE-16GB GPU. For Susy (n = 5 × 10 6 , p = 18): we used Gaussian kernel with k(x, x ) = e -x-x 2 /(2σ 2 ) , with σ = 5, which we obtained through a grid search (in [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF], σ = 4 is taken for the ridge regression); M = 10 4 Nyström centers and a subsampling Q = M for the preconditioner, both obtained with uniform sampling. Analogously for Higgs (n = 1.1 × 10 7 , p = 28): , we used a Gaussian kernel with σ = 5 and M = 2.5 × 10 4 and Q = M , using again uniform sampling. To find reasonable λ for supervised learning applications, we cross-validated λ finding the minimum test error at λ = 10 -10 for Susy and λ = 10 -9 for Higgs (see Figs. 5.2 and 5.3 in Sec. 5.F ) for such values our algorithm and the competitor achieve an error of 19.5% on the test set for Susy, comparable to the state of the art (19. 6% Rudi, Carratino, and Rosasco (2017)) and analogously for Higgs (see Sec. 5.F ). We then used such λ's as regularization parameters and compared our algorithm with a well known accelerated stochastic gradient technique Katyusha SVRG (K-SVRG) Allen-Zhu (2017), tailored to our problem using mini batches. In Fig. 5.1 we show the convergence of the training loss and classification error with respect to the number of passes on the data, of our algorithm compared to K-SVRG. It is possible to note our algorithm is order of magnitude faster in achieving convergence, validating empirically the fact that the proposed algorithm scales as O(ndf λ ) in learning settings, while accelerated first order methods go as O((n + nL/λ)df λ ). Moreover, as mentioned in the introduction, this highlights the fact that precise optimization is necessary to achieve a good performance in terms of test error. Finally, note that since a pass on the data is much more expensive for K-SVRG than for our second order method (see Sec. 5.F for details), the difference in computing time between the second order scheme and K-SVRG is even more in favour of our second order method (see Figs. 5.4 and 5.5 in Sec. 5.F ).

Organization of the Appendix

5.A . Main results on generalized self-concordant functions

Notations, definitions and basic results concerning generalized self-concordant functions.

5.B . Results on approximate Newton methods

In this section, the interaction between the notion of Dikin ellipsoid, approximate Newton methods and generalized self-concordant functions is studied. The results needed in the main paper are all concentrated in Sec. 5.B .3. In particular the results in Lemma 5.2 are proven in a more general form in Lemma 5.11.

5.C . Proof of bounds for the globalization scheme

In this section, we leverage the results of the previous two sections to analyze the globalization scheme.

5.C .1. Main technical lemmas

We start by proving the result on the inclusion of Dikin ellipsoids (Lemma 5.3).

5.C .2. Proof of main theorems

In particular, a general version of Theorem 5.1 is proven. Moreover Remark 10 is proven in proposition 5.7, while the fixed scheme to choose (q k ) k∈N is proven in proposition 5.8.

5.C .3. Proof of Thm. 1

Finally, we prove the properties of the globalization schemes presented in Theorem 5.1.

5.D . Non-parametric learning with generalized self-concordant functions

In this section, some basic results about non-parametric learning with generalized selfconcordant functions are recalled and the main results of Sec. 5.4 are proven.

5.D .1. General setting and assumptions, statistical result for regularized ERM.

More details about the generalization properties of empirical risk minimization as well as the optimal rates in Theorem 5.2 are recalled.

5.D .2.

Reducing the dimension: projecting on a subspace using Nyström subsampling.

5.D .3. Sub-sampling techniques.

The basics of uniform sub-sampling and sub-sampling with approximate leverage scores are recalled.

5.D .4. Selecting the M Nyström points

Theorem 5.3 is proven in a more general version in Theorem 5.6.

5.D .5 Performing the globalization scheme to approximate β M,λ

A general scheme is proposed to solve the projected problem approximately using the globalization scheme.

5.D .5. Performing approximate Newton steps

We start by describing the way of computing approximate Newton steps. A generalized version of Lemma 5.4 is proven in proposition 5.12.

5.D .5. Applying the globalization scheme to control ν M,λ (β)

We then completely analyse the approximating of β M,λ from an optimization point of view (see proposition 5.13).

5.D .6. Final algorithm and results

Finally, the proof of Theorem 5.4 is provided, using the results of the previous subsections.

5.E . Algorithm

In this section, the pseudocode for the algorithm presented in Sec. 5.4 and analyzed in Theorem 5.7 is provided.

5.F . Experiments

In this section, more details about the experiments are provided.

5.G . Solving a projected problem to reduce dimension

In this section, more details about the problem of randomized projections are provided.

5.G .2. Relating the projected to the original problem

In particular, results to relate the ERM with the projected ERM in terms of excess risk are provided for generalized self-concordant functions.

5.H . Relations between statistical problems and empirical problem.

In this section, we provide results to relate excess expected risk with excess empirical risk for generalized self-concordant functions.

5.I . Multiplicative approximations for Hermitian operators

In this section, some general analytic results on multiplicative approximations for Hermitian operators are derived. Moreover they are used to provide a simplified proof for the results in Lemma 5.1. See in particular Lemmas 5.28 and 5.29 and Pilanci and Wainwright (2017), Lemma 2.

5.A Main results on generalized self-concordant functions

In this section, we start by introducing a few notations. We define the key notion of generalized self-concordance in Sec. 5.A .1, and present the main results concerning generalized self-concordant functions. In Sec. 5.A .2, we describe how generalized self-concordance behaves with respect to an expectation or to certain relaxations.

Notations Let λ ≥ 0 and A be a bounded positive semidefinite Hermitian operator on H. We denote with I the identity operator, and

x A := A 1/2 x , (5.11)
A λ := A + λI.

(5.12)

Let f be a twice differentiable convex function on a Hilbert space H. We adopt the following notation for the Hessian of f :

∀x ∈ H, H f (x) := ∇ 2 f (x) ∈ L(H).
For any λ > 0, we define the λ-regularization of f :

f λ := f + λ 2 • 2 .
f λ is λ-strongly convex and has a unique minimizer which we denote with x f,λ . Moreover, define

∀x ∈ H, H f,λ (x) := ∇ 2 f λ (x) = H f (x) + λI, ν f,λ (x) := ∇f λ (x) H -1 f,λ (x) .
The quantity ν f,λ (x) is called the Newton decrement at point x and will play a significant role.

When the function f is clear from the context, we will omit the subscripts with f and use H, H λ , ν λ ....

5.A .1 Definitions and results on generalized self-concordant functions

In this section, we introduce the main definitions and results for self-concordant functions. These results are mainly the same as in appendix B of Marteau-Ferey, Ostrovskii, Bach, and Rudi (2019).

Definition 5.3 (generalized self-concordant function). Let H be a Hilbert space. Formally, a generalized self-concordant function on H is a couple (f, G) where:

i G is a bounded subset of H; we will usually denote G or R the quantity sup g∈G g < ∞;

ii f is a convex and three times differentiable mapping on H such that

∀x ∈ H, ∀h, k ∈ H, ∇ (3) f (x)[h, k, k] ≤ sup g∈G |g • h| ∇ 2 f (x)[k, k].
To make notations lighter, we will often omit G from the notations and simply say that f stands both for the mapping and the couple (f, G).

Definition 5.4 (Definitions). Let f be a generalized self-concordant function. We define the following quantities.

• ∀h ∈ H, t f (h) := sup g∈G |h • g|;

• ∀x ∈ H, ∀λ > 0, r f,λ (x) := 1 sup g∈G g H -1 f,λ (x)
;

• ∀c ≥ 0, ∀λ > 0, D f,λ (c) := {x : ν f,λ (x) ≤ cr f,λ (x)}.
We also define the following functions:

ψ(t) = e t -t -1 t 2 , φ(t) = 1 -e -t t , φ(t) = e t -1 t .
(5.13) Note that ψ, φ are increasing functions and that φ is a decreasing function. Moreover, φ(t) φ(t) = e t . Once again, if f is clear, we will often omit the reference to f in the quantities above, keeping only t, r λ , D λ ...

CHAPTER 5. SECOND ORDER STRIKES BACK

We condense results obtained in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] under a slightly different form. The proofs, however, are exactly the same.

While in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], only the regularized case is dealt with, the proof techniques are exactly the same to obtain proposition 5.1. proposition 5.2 is proved explicitly in Proposition 4 of [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] and Lemma 5.5 is proved in Proposition 5. Omitting the subscript f , we get the following results.

Proposition 5.1 (Bounds for the non-regularized function f ). Let f be a generalized selfconcordant function. Then the following bounds hold (we omit f in the subscripts):

∀x ∈ H, ∀h ∈ H, e -t(h) H(x) H(x + h) e t(h) H(x), (5.14) ∀x, h ∈ H, ∀λ > 0, ∇f (x + h) -∇f (x) H -1 λ (x) ≤ φ(t(h)) h H λ (x) , (5.15) ∀x, h ∈ H, ψ(-t(h)) h 2 H(x) ≤ f (x + h) -f (x) -∇f (x).h ≤ ψ(t(h)) h 2 H(x) .
(5.16)

We get the analoguous bounds in the regularized case.

Proposition 5.2 (Bounds for the regularized function f λ ). Let f be a generalized self-concordant function and λ > 0 be a regularizer. Then the following bounds hold: x) .

∀x, h ∈ H, e -t(h) H λ (x) H λ (x + h) e t(h) H λ (x), (5.17) ∀x, h ∈ H, φ(t(h)) h H λ (x) ≤ ∇f λ (x + h) -∇f λ (x) H -1 λ (x) ≤ φ(t(h)) h H λ (x) , (5.18) ∀x, h ∈ H, ψ(-t(h)) h 2 H λ (x) ≤ f λ (x + h) -f λ (x) -∇f λ (x).h ≤ ψ(t(h)) h 2 H λ (
(5.19)

Corollary 5.1. Let f be a G generalized self-concordant function and λ > 0 be a regularizer, and x λ the unique minimizer of f λ . Then the following bounds hold for any x ∈ H:

φ(t(x -x λ )) x -x λ H λ (x) ≤ ∇f λ (x) H -1 λ (x) ν λ (x) ≤ φ(t(x -x λ )) x -x λ H λ (x) , (5.20) ψ(-t(x -x λ )) x -x λ 2 H λ (x λ ) ≤ f λ (x) -f λ (x λ ) ≤ ψ(t(x -x λ )) x -x λ 2 H λ (x λ ) . (5.21)
Moreover, the following localization lemma holds.

Lemma 5.5 (localization). Let λ > 0 be fixed. If

ν λ (x) r λ (x) < 1, then t(x -x λ ) ≤ -log 1 - ν λ (x) r λ (x) . ( 5

.22)

In particular, this shows:

∀c < 1, ∀λ > 0, x ∈ D λ (c) =⇒ t(x -x λ ) ≤ -log(1 -c).

5.A . GENERALIZED SELF-CONCORDANCE

187

We now state a Lemma which shows that the difference to the optimum in function values is equivalent to the squared newton decrement in a small Dikin ellipsoid. We will use this result in the main paper.

Lemma 5.6 (Equivalence of norms). Let λ > 0 and x ∈ D λ ( 1 7 ). Then the following holds:

1 4 ν λ (x) 2 ≤ f λ (x) -f λ (x λ ) ≤ ν λ (x) 2 .
Proof. Apply Lemma 5.5 knowing x ∈ D λ ( 1 7 ) to get t(xx λ ) ≤ log(7/6). Then apply Eq. ( 5.19) and Eq. ( 5.18) to get:

f λ (x) -f λ (x λ ) ≤ ψ(t(x -x λ )) x -x λ 2 H λ (x λ ) ≤ e t(x-x λ ) ψ(t(x -x λ )) x -x λ 2 H λ (x) ≤ e t(x-x λ ) ψ(t(x -x λ )) φ(t(x -x λ )) 2 ν λ (x) 2 .
Replacing with the bound above, we get

∀λ > 0, ∀x ∈ D λ ( 1 7 ), f λ (x) -f λ (x λ ) ≤ ν λ (x) 2 .
For the lower bound, proceed in exactly the same way.

5.A .2 Comparison between generalized self-concordant functions

The following result is straightforward.

Lemma 5.7 (Comparison between generalized self-concordant functions). Let G 1 ⊂ G 2 ⊂ H be two bounded subsets. If (f, G 1 ) is generalized self-concordant, then (f, G 2 ) is also generalized self-concordant. Moreover, ∀x ∈ H, ∀λ > 0, r (f,G 1 ),λ (x) ≥ r (f,G 2 ),λ (x).
In particular, we will often use the following fact. If (f, G) is generalized self-concordant, and G is bounded by R, then (f, B H (R)) is also generalized self-concordant. Moreover,

r (f,B H (R)),λ (x) = λ + λ min (H f (x)) R ≥ √ λ R .
We now state a result which shows that, given a family of generalized self-concordant functions, the expectancy of that family is also generalized self-concordant. This can be seen as a reformulation of Proposition 2 of Marteau-Ferey, Ostrovskii, Bach, and Rudi (2019).

Proposition 5.3 (Expectation). Let Z be a polish space equipped with its Borel sigma-algebra, and H be a Hilbert space. Let ((f z , G z )) z∈Z be a family of generalized self-concordant functions such that the mapping (z, x) → f z (x) is measurable.

Assume we are given a random variable Z on Z, whose support we denote with supp(Z), such that

• the random variables f Z (0) , ∇f Z (0) , Tr(∇ 2 f Z (0)) are are bounded;

• G := z∈supp(Z) G z is a bounded subset of H.
Then the mapping f :

x ∈ H → E [f Z (x)] is well defined, (f, G) is generalized self-concordant,
and we can differentiate under the expectation.

Corollary 5.2. Let n ∈ N and (f i , G i ) 1≤i≤n be a family of generalized self-concordant functions. Define

f (x) = 1 n n i=1 f i (x), G = n i=1 G i .
Then (f, G) is generalized self-concordant.

5.B . APPROXIMATE NEWTON METHODS

189

5.B Results on approximate Newton methods

In this section, we assume we are given a generalized self-concordant function f in the sense of Sec. 5.A . As f will be fixed throughout this part, we will omit it from the notations. Recall the definitions from definition 5.4:

ν λ (x) := ∇f λ (x) H -1 λ (x) , 1 r λ (x) := sup g∈G g H -1 λ (x) , D λ (c) := x : ν λ (x) r λ (x) ≤ c .
Define the following quantities:

• the true Newton step at point x for the λ-regularized problem:

∆ λ (x) := H -1 λ (x)∇f λ (x).
• the renormalized Newton decrement ν λ (x):

ν λ (x) := ν λ (x) r λ (x) .
Moreover, note that a direct application of Eq. ( 5.17) yields the following equation which relates the radii at different points:

∀λ > 0, ∀x ∈ H, ∀h ∈ H, e -t(h) r λ (x) ≤ r λ (x + h) ≤ e t(h) r λ (x).
(5.23)

In this appendix, we develop a complete analysis of so-called approximate Newton methods in the case of generalized self-concordant losses. By "approximate Newton method", we mean that instead of performing the classical update x t+1 = x t -∆ λ (x t ), we perform an update of the form x t+1 = x t -∆ t where ∆ t is an approximation of the real Newton step. We will characterize this approximation by measuring its distance to the real Newton step using two parameters ρ and

0 : ∆ t -∆ λ (x t ) ≤ ρν λ (x t ) + 0 .
We start by presenting a few technical results in Sec. 5.B .1. We continue by proving that an approximate Newton method has linear convergence guarantees in the right Dikin ellipsoid in Sec. 5.B .2. In Sec. 5.B .3, we adapt these results to a certain way of computing approximate Newton steps, which will be the one we use in the core of the paper. In Sec. 5.B .4, we mention ways to reduce the computational burden of these methods by showing that since all Hessians are equivalent in Dikin ellipsoids, one can actually sketch the Hessian at one given point in that ellipsoid instead of re-sketching it at each Newton step. For the sake of simplicity, this is not mentioned in the core paper, but works very well in practice.

5.B .1 Main technical results

We start with a technical decomposition of the Newton decrement at point x -∆ for a given ∆ ∈ H.

Lemma 5.8 (Technical decomposition). Let λ > 0, x ∈ H be fixed. Assume we perform a step of the form x -∆ for a certain ∆ ∈ H. Define

δ := ∆ -∆ λ (x) H λ (x) , δ := δ r λ (x)
.

The following holds:

ν λ (x -∆) ≤ e ν λ (x)+ δ ψ( ν λ (x) + δ)( ν λ (x) + δ) 2 + δ ; (5.24) ν λ (x -∆ λ (x)) ≤ e ν λ (x)+ δ ψ( ν λ (x) + δ)( ν λ (x) + δ)(ν λ (x) + δ) + δ .
(5.25)

Proof. Note that by definition, ∇f λ (x) = H λ (x)∆ λ (x). Hence

∇f λ (x -∆) H -1 λ (x) = ∇f λ (x -∆) -∇f λ (x) + H λ (x)∆ λ (x) H -1 λ (x) ≤ ∇f λ (x -∆) -∇f λ (x) + H λ (x) ∆ H -1 λ (x) + H λ (x)(∆ λ (x) -∆) H -1 λ (x) = 1 0 [H λ (x -s ∆) -H λ (x)] ∆ds H -1 λ (x) + δ ≤ 1 0 H -1/2 λ (x)H λ (x -s ∆)H -1/2 λ (x) -I ds ∆ H λ (x) + δ.
Now using Eq. ( 5.17), one has

H -1/2 λ (x)H λ (x -s ∆)H -1/2 λ (x) -I ≤ e st( ∆) -1, whose integral on s is ψ(t( ∆))t( ∆)
where ψ is defined in definition 5.4. Morever, bounding

∆ H λ (x) ≤ ∆ -∆ λ (x) H λ (x) + ∆ λ (x) H λ (x) = δ + ν λ (x), it holds ∇f λ (x -∆) H -1 λ (x) ≤ ψ(t( ∆))t( ∆) (ν λ (x) + δ) + δ.
1. Now note that using Eq. ( 5.17), it holds: ν λ (x -∆) ≤ e t( ∆)/2 ∇f λ (x -∆) H -1 λ (x) and hence:

ν λ (x -∆) ≤ e t( ∆)/2 ψ(t( ∆))t( ∆) (ν λ (x) + δ) + δ .
(5.26)

2.

Moreover, using Eq. ( 5.23),

ν λ (x -∆) ≤ e t( ∆) ψ(t( ∆))t( ∆) ( ν λ (x) + δ) + δ . (5.27) Noting that t( ∆) ≤ ∆ H λ (x) r λ (x) ≤ ν λ (x) + δ,
and bounding Eq. ( 5.26) simply by taking e t( ∆)/2 ≤ e t( ∆) , we get the two bounds in the lemma.

5.B . APPROXIMATE NEWTON METHODS

191

We now place ourselves in the case where we are given an approximation of the Newton step of the following form. Assume λ and x are fixed, and that we approximate ∆ λ (x) with ∆ such that there exists ρ ≥ 0 and 0 ≥ 0 such that it holds:

∆ -∆ λ (x) H λ (x) ≤ ρν λ (x) + 0 .
We define/prove the three different following regimes.

Lemma 5.9 (3 regimes). Let x ∈ D λ 1 7 and λ > 0 be fixed. Let

0 ≤ ρ ≤ 1 7 , 0 ≥ 0 s.t. ε0 := 0 r λ (x) ≤ 1 21 .
Let ∆ be an approximation of the Newton steps satisfying ∆ -∆ λ (x) H λ (x) ≤ ρν λ (x) + 0 . The three following regimes appear.

• If ν λ (x) ≥ ρ and ν λ (x) 2 ≥ ε0 , then we are in the quadratic regime, i.e.

10 ν λ (x -∆ λ (x)) 3 ≤ 10 ν λ (x) 3 2 , ν λ (x -∆ λ (x)) ≤ 10 3 ν λ (x)ν λ (x).
• If ρ ≥ ν λ (x) and ρ ν λ (x) ≥ 0 , then we are in the linear regime, i.e.

10 3 ν λ (x -∆ λ (x)) ≤ 10ρ 3 10 3 ν λ (x) , ν λ (x -∆ λ (x)) ≤ 10 3 ν λ (x)ν λ (x).
• If 0 ≥ ν λ (x) 2 , ρ ν λ (x), then the maximal precision of the approximation is reached, and it holds:

ν λ (x -∆ λ (x)) ≤ 3 0 ≤ 1 7 , ν λ (x -∆ λ (x)) ≤ 3 0 .
Proof. Using the previous lemma,

ν λ (x -∆ λ (x)) ≤ e (1+ρ) ν λ (x)+ 0 ψ((1 + ρ) ν λ (x) + 0 )((1 + ρ) ν λ (x) + 0 ) 2 + ρ ν λ (x) + 0 ≤ 1 ( ν λ (x), ρ, 0 ) ν λ (x) 2 + 2 ( ν λ (x), ρ, 0 ) ρ ν λ (x) + 3 ( ν λ (x), ρ, 0 ) 0 , and
ν λ (x -∆ λ (x)) ≤ 1 ( ν λ (x), ρ, 0 ) ν λ (x)ν λ (x) + 2 ( ν λ (x), ρ, 0 ) ρν λ (x) + 3 ( ν λ (x), ρ, 0 ) 0 ,
where the following defintions are used:

1 ( ν, ρ, 0 ) := e (1+ρ) ν+ 0 ψ((1 + ρ) ν + 0 )(1 + ρ) 2 , 2 ( ν, ρ, 0 ) := e (1+ρ) ν+ 0 , 3 ( ν, ρ, 0 ) := e (1+ρ) ν+ 0 [2ψ((1 + ρ) ν + 0 )(1 + ρ) ν + 1] . Now assume 0 ≤ 1 21 , ν λ (x), ρ ≤ 1 7 .
Replacing these values in the functions above bounds 1 , 2 and 3 , and using the case distinction, we get the result.

5.B .2 General analysis of an approximate Newton method

The following proposition describes the behavior of an approximate newton method where ρ and 0 are fixed a priori. Proposition 5.4 (General approximate Newton scheme results). Let c ≤ 1 7 be fixed and x 0 ∈ D λ (c) be a given starting point. Let ρ ≤ 1 7 and 0 such that 0 ≤ c 4 r λ (x 0 ). Define the following approximate Newton scheme:

∀t ≥ 0, x t+1 = x t -∆ t , ∆ t -∆ λ (x t ) H λ (xt) ≤ ρν λ (x t ) + 0 .
The following guarantees hold.

• ∀t ≥ 0, x t ∈ D λ (c).

• Let t c = log 2 log 2 3 10ρ + 1. ∀t ≤ t c , 10 ν λ (x t ) 3 ≤ max 12 0 r λ (x 0 ) , 2 -2 t , ∀t ≥ t c , 10 ν λ (x t ) 3 ≤ max 12 0 r λ (x 0 ) , 10ρ 3 t-tc+1
.

• We can bound the relative decrease for both the Newton decrement and the renormalized Newton decrement:

∀t ≤ t c , ν λ (x t ) ≤ max 3 0 , 1 2 2 t -1 ν λ (x 0 ) , ν λ (x t ) ≤ max 18 0 5r λ (x 0 ) , 1 2 2 t -1 ν λ (x 0 ) . ∀t ≥ t c , ν λ (x t ) ≤ max 3 0 , 10ρ 3 t-tc+1 ν λ (x 0 ) , ν λ (x t ) ≤ max 18 0 5r λ (x 0 ) , 10ρ 3 t-tc+1 ν λ (x 0 ) .
Proof. Start by noting, using Eq. ( 5.23),

∀x ∈ D λ 1 7 , ε ≤ r λ (x) 21 , 6 7 r λ (x 0 ) ≤ r λ (x) ≤ 7 6 r λ (x 0 ).
(5.28)

In particular, this holds for any x ∈ D λ (c), c ≤ 1 7 . Thus,

∀c ≤ 1 7 , ∀x 0 ∈ D λ (c), 0 r λ (x 0 ) ≤ c 4 =⇒ ∀x ∈ D λ (c), 0 r λ (x) ≤ c 3 .
1. Proving the first point is simple by induction. Indeed, assume ν λ (x t ) ≤ c. We can apply Lemma 5.9 since the conditions on ε and ρ guarantee that the conditions of this lemma are satisfied.

If we are in either the linear or quadratic regime, the fact that 10ρ 3 , 10 ν λ (xt)

3

≤ 10 21 show that ν λ (x t+1 ) ≤ 10 21 ν λ (x t ) ≤ c. If we are in the last case, ν λ (x t+1 ) ≤ 3 0 r λ (xt) ≤ c.

2. Let us prove the second bullet point by induction. Start by assuming the property holds at t. By the previous point, the hypothesis of Lemma 5.9 are satisfied at x t with ρ and ε. Assume we are in the limiting case; we easily show that in this case,

10 ν λ (x t+1 ) 3 ≤ 10 3 3 0 r λ (x t ) ≤ 35 0 3r λ (x 0 )
.

Here, the last inequality comes from Eq. ( 5.28). If we are not in the limiting case, let us distinguish between the two following cases.

If t ≤ t c -1, 10 ν λ (x t+1 ) 3 ≤ 10 ν λ (x t ) 3 max 10 ν λ (x t ) 3 , 10ρ 3
≤ max 35 0 3r λ (x 0 ) , 10 ν λ (x t ) 3 max 1 2 2 t , 10ρ 3
,
where the last inequality comes from using the induction hypothesis and the fact that 10 ν λ (xt) 3 ≤ 1.

Using once again the induction hypotheses and the fact that t ≤ log 2 log 2 3 10ρ which implies

10ρ 3 ≤ 1 2 2 t
, we finally get

10 ν λ (x t+1 ) 3 ≤ max 35 0 3r λ (x 0 ) , 1 2
2 t+1
.

The fact that the second property holds for t = t c is trivial Now consider the case where t ≥ t c . Using the same technique as before but noting that in this case

10 ν λ (x t ) 3 ≤ max 35 0 3r λ (x 0 ) , 10ρ 3 t-tc+1 ≤ max 35 0 3r λ (x 0 ) , 10ρ 3
,
We easily use Lemma 5.9 to reach the desired conclusion.

3. Let t < t c . Then using Lemma 5.9:

∀s ≤ t, ν λ (x s+1 ) ≤ max 3 0 , max( 10ρ 3 , 10 ν λ (x s ) 3 )ν λ (x s ) .
Using the fact that for any s ≤ t, 10 ν λ (xs) 3 ≤ max( 35 0 3r λ (x 0 ) , 1 2 2 s

):

∀s ≤ t, ν λ (x s+1 ) ≤ max 3 0 , 35 0 3 ν λ (x s ) r λ (x 0 ) , max( 10ρ 3 , 1 2 2 s
)ν λ (x s ) .

194

CHAPTER 5. SECOND ORDER STRIKES BACK Now using the fact that for any s ≤ t, ν λ (x s ) ≤ 1 7 , we see that ν λ (xs) r λ (x 0 ) ≤ 7 6 ν λ (x s ) ≤ 1 6 and hence

35 0 3 ν λ (xs) r λ (x 0 ) ≤ 3 0 . Moreover, since s ≤ t < t c , max( 10ρ 3 , 1 2 2 s ) = 1 2 2 s
. Thus:

∀s ≤ t, ν λ (x s+1 ) ≤ max 3 0 , 1 2 2 s ν λ (x s ) .
Combining these results yields:

ν λ (x t+1 ) ≤ max 3 0 , 1 2 2 t+1 -1 ν λ (x 0 ) .
This shows the first equation, that is:

∀t ≤ t c , ν λ (x t ) ≤ max 3 0 , 1 2 2 t -1 ν λ (x 0 ) .
The case for t ≥ t c is completely analogous. We can also reproduce the same proof to get the same bounds for ν, since the bounds in Lemma 5.9 are the same for both.

5.B .3 Main results in the paper

In the main paper, we mention two types of Newton method. First, we present a result of convergence on the full Newton method:

Lemma 5.10 (Quadratic convergence of the full Newton method). Let c ≤ 1 7 and x 0 ∈ D λ (c).

Define

x t+1 = x t -∆ λ (x t ).

Then this scheme converges quadratically, i.e.: 1) .

∀t ∈ N, ν λ (x t ) ν λ (x 0 ) , ν λ (x t ) ν λ (x 0 ) ≤ 2 -(2 t -
Thus :

• ∀t ∈ N, x t ∈ D λ (c). • For any c ≤ c then ∀t ≥ log 2 1 + log 2 c c , x t ∈ D λ (c). • For any ε > 0, ∀t ≥ log 2 1 + log 2 ν λ (x 0 ) √ ε , ν λ (x t ) ≤ √ ε, f λ (x) -f λ (x λ ) ≤ ε.
• If we perform the Newton method and return the first x t such that ν λ (x t ) ≤ √ ε, then the number of Newton steps computations is at most

1 + log 2 1 + log 2 ν λ (x 0 ) √ ε .
Proof. A full Newton method is an approximate Newton method where ρ, 0 = 0. Thus apply proposition 5.4; note that in this case t c = +∞. The last point shows that if c ≤ 1 7 , and if we perform the Newton method with a full Newton step, then

∀t ≥ 0, ν λ (x t ) ≤ 2 -(2 t -1) ν λ (x 0 ), ν λ (x t ) ≤ 2 -(2 t -1) ν λ (x 0 ).
This shows the quadratic convergence, and the first two points directly follow. For the third point, the result for ν λ (x t ) directly follows from the previous equation, and the one on function values is a direct consequence of Lemma 5.6 and the fact that x t ∈ D λ (1/7).

For the last point, note that ν t (x t ) = ∇f λ (x t ) • ∆ λ (x t ) is accessible. Moreover, the bound on t is given in the point before, and since one has to compute ∆ λ (x s ) for 0 ≤ s ≤ t, there are at most t + 1 computations.

In the main paper, we compute approximate Newton steps by considering methods which naturally yield only a relative error ρ and no absolute error 0 . Indeed, we take the following notation.

Approximate solutions to linear problems. Let A be a positive definite Hermitian operator on H, b in H, and a wanted relative precision ρ.

We say that x is a ρ-relative approximation to the linear problem Ax = b and write x ∈ LinApprox(A, b, ρ) if the following holds:

A -1 b -x A ≤ ρ b A -1 = ρ A -1 b A . Note that if x ∈ LinApprox(A, b, ρ) for ρ < 1, then (1 -ρ) b A -1 ≤ x • b ≤ (1 + ρ) b A -1 .
The following lemma shows that if, instead of computing the exact Newton step, we compute a relative approximation of the Newton step belonging to LinApprox(H λ (x), ∇f λ (x), ρ) for a given ρ < 1, then one has linear convergence. Moreover, we show that we can still perform a method which automatically stops. Proposition 5.5 (relative approximate Newton method). Let λ > 0, ρ ≤ 1 7 , c ≤ 1 7 and a starting point x 0 ∈ D λ (c). Assume we perform the following Newton scheme:

∀t ≥ 0, x t+1 = x t -∆ t , ∆ t ∈ LinApprox(H λ (x t ), ∇f λ (x t ), ρ).
Then the scheme converges linearly, i.e.

∀t ∈ N, ν λ (x t ) ν λ (x 0 ) , ν λ (x t ) ν λ (x 0 ) ≤ 2 -t .
Thus,

• ∀t ∈ N, x t ∈ D λ (c). • For any c ≤ c then ∀t ≥ log 2 c c , x t ∈ D λ (c).
• For any ε > 0, ∀t ≥ log 2

ν λ (x 0 ) √ ε , ν λ (x t ) ≤ √ ε, f λ (x) -f λ (x λ ) ≤ ε
• If the method is performed and returns the first x t such that x t • ∆ t ≤ 6 7 ε, then at most 2 + log 2 4 3

ν λ (x 0 ) √ ε
approximate Newton steps computations have been performed, and

ν λ (x t ) ≤ √ ε, f λ (x) -f λ (x λ ) ≤ ε.
Proof. Apply proposition 5.4 with 0 = 0 and ρ = 1 7 , since if ρ ≤ 1 7 , then a fortiori the approximation satisfies the condition for ρ = 1 7 . The last point clearly states that

∀t ∈ N, ν λ (x t ) ν λ (x 0 ) , ν λ (x t ) ν λ (x 0 ) ≤ 10 21 t ≤ 2 -t .

196

CHAPTER 5. SECOND ORDER STRIKES BACK From this, using Lemma 5.6 for the third point, the first three points are easily proven.

For the last point, note that since ∆ t ∈ LinApprox(H λ (x t ), ∇f λ (x t ), ρ), the following holds:

∇f λ (x t ) • ∆ t = ν λ (x t ) 2 + ∇f λ (x t ) • ∆ t -H -1 λ (x t )∇f λ (x t ) . Now bound |∇f λ (x t ) • ∆ t -H -1 λ (x t )∇f λ (x t ) | ≤ ν λ (x t ) ∆ t -H -1 λ (x t )∇f λ (x t ) H λ (xt) ≤ ρν λ (x t ) 2 . Thus: (1 -ρ)ν λ (x t ) 2 ≤ ∇f λ (x t ) • ∆ t ≤ (1 + ρ)ν λ (x t ) 2 . Since ρ = 1 7 , we see that if ∇f λ (x t ) • ∆ t ≤ 6 7 ε, then ν λ (x t ) 2 ≤ ε.
Moreover, since we stop at the first t where ∇f λ (x t ) • ∆ t ≤ 6 7 ε, then if t denotes the time at which we stop,

6 7 ε < ∇f λ (x t-1 ) • ∆ t-1 ≤ 8 7 ν λ (x t-1 ) 2 =⇒ ν λ (x t-1 ) 2 ≥ 3 4 ε. Since ν λ (x t-1 ) 2 ≤ 2 -2(t-1) ν λ (x 0 ) 2 , this implies in turn that t -1 ≤ log 2 4 3 ν λ (x 0 ) √ ε . Thus, necessarily, t ≤ 1 + log 2 4 3 ν λ (x 0 ) √ ε
, and since we compute approximate Newton steps for s = 0, ..., t, we finally have that the number of approximate Newton steps is bounded by 2 + log 2 4 3

ν λ (x 0 ) √ ε .
Last but not least, we summarize all these theorem in the following simple result.

Lemma 5.11. Let λ > 0, c ≤ 1/7, let f λ be generalized self-concordant and x ∈ D λ (c). It holds:

1 4 ν λ (x) 2 ≤ f λ (x) -f λ (x λ ) ≤ ν λ (x) 2
. Moreover, the full Newton method starting from x 0 has quadratic convergence, i.e. if x t is obtained via t ∈ N steps of the Newton method Eq. (5.2), then ν λ (x t ) ≤ 2 -(2 t -1) ν λ (x 0 ). Finally, the approximate Newton method starting from x 0 has linear convergence, i.e. if x t is obtained via t ∈ N steps of Eq. (5.3), with ∆ t ∈ LinApprox(H λ (x t ), ∇f λ (x t ), ρ) and ρ ≤ 1/7, then ν λ (x t ) ≤ 2 -t ν λ (x 0 ).

Proof. The three points are obtained in the following lemmas, assuming x ∈ D λ (1/7). • The convergence rate of the approximate Newton method starting in D λ (1/7) is obtained in proposition 5.5.

• For 1 4 ν λ (x) 2 ≤ f λ (x) -f λ (x λ ) ≤ ν λ (x) 2 ,

5.B .4 Sketching the Hessian only once in each Dikin ellispoid

In this section, we provide a lemma which shows in essence that if we are in a small Dikin ellipsoid, then we can keep the Hessian of the starting point and compute approximations of H -1 λ (x 0 )∇f λ (x t ); they will be good approximations to H -1 λ (x t )∇f λ (x t ) as well.

Lemma 5.12. Let c < 1 and x 0 ∈ D λ (c) be fixed.

Let H be an approximation of the Hessian at x 0 , approximation wich we quantify with

t := H -1/2 λ (x 0 ) H λ (x 0 ) -H H -1/2 λ (x 0 ) . Assume 1 + t < 2(1 -c) 2 . Let b ∈ H. If ∆ ∈ LinApprox( H λ , b, ρ), then ∀x ∈ D λ (c), ∆ ∈ LinApprox(H λ (x), b, ρ), ρ = (ρ -1)(1 -c) 2 + (1 + t) 2(1 -c) 2 -(1 + t) . In particular, if c ≤ 1 30 , x 0 ∈ D λ (c), ∀x ∈ D λ (c), ∀b ∈ H, ∆ ∈ LinApprox(H λ (x 0 ), b, 1 20 ) =⇒ ∆ ∈ LinApprox(H λ (x), b, 1 7
).

Proof. First, start with a general theoretical result.

1.

Let A and B be two positive semi-definite hermitian operators. Let λ > 0, b ∈ H and ∆ ∈ LinApprox(B λ , b, ρ). Decompose

A -1 λ b -∆ A λ ≤ A -1 λ b -B -1 λ b A λ + B -1 λ b -∆ A λ ≤ A 1/2 λ (A -1 λ -B -1 λ )A 1/2 λ b A -1 λ + A 1/2 λ B -1/2 λ B -1 λ b -∆ B λ .

Now using the fact that

A -1 λ -B -1 λ = B -1 λ (B -A)A -1 λ , A 1/2 λ (A -1 λ -B -1 λ )A 1/2 λ ≤ A -1/2 λ (B -A)A -1/2 λ A 1/2 λ B -1 λ A 1/2 λ = A -1/2 λ (B -A)A -1/2 λ A 1/2 λ B -1/2 λ 2 .
Moreover,

B -1 λ b -∆ B λ ≤ ρ b B -1 λ ≤ A 1/2 B -1/2 b A -1 λ .
Putting things together, and noting that from Lemma 5.21,

A 1/2 B -1/2 2 ≤ 1 1-A -1/2 λ (B-A)A -1/2 λ as soon as A -1/2 λ (B -A)A -1/2 λ < 1, it holds: ∆ ∈ LinApprox(A λ , b, ρ), ρ = ρ + A -1/2 λ (B -A)A -1/2 λ 1 -A -1/2 λ (B -A)A -1/2 λ .
The aim is now to apply this lemma to A = H(x) and B = H.

CHAPTER 5. SECOND ORDER STRIKES BACK

2.

Let x, x 0 ∈ D λ (c). Using Lemma 5.22, we see that

1 + H -1/2 λ (x)( H -H(x))H -1/2 λ (x) ≤ (1 + t)(1 + H -1/2 λ (x)(H(x 0 ) -H(x))H -1/2 λ (x) ).
Using Eq. ( 5.17), it holds:

(e -t(x-x 0 ) -1)I H -1/2 λ (x)(H(x 0 ) -H(x))H -1/2 λ (x) (e t(x 0 -x) -1)I.
Thus,

H -1/2 λ (x)(H(x 0 ) -H(x))H -1/2 λ (x) ≤ max(1 -e -t(x-x 0
) , e t(x-x 0 ) -1) = e t(x-x 0 ) -1.

Finally, using the fact that

x 0 , x ∈ D λ (c) for c < 1 yields t(x -x 0 ) ≤ 2 log 1 1-c . Hence 1 + H -1/2 λ (x)(H(x 0 ) -H(x))H -1/2 λ (x) ≤ 1 (1 -c) 2 .
Thus,

H -1/2 λ (x)( H -H(x))H -1/2 λ (x) ≤ 1 + t (1 -c) 2 -1.
The result then follows.

5.C . GLOBALIZATION SCHEME

• For any c < 1, x ∈ H, if Rνµ(x) √ µ ≤ c
3 , then the following holds:

R √ µ x H -1 µ (x) c ≤ 1 + 1 1 -c/3 1 3 + 1 1 -c/3 R √ µ x µ H -1 µ (x µ ) c .
Likewise, it can be shown that under the same conditions:

R x c ≤ R x µ c + 1 3 φ(-log(1 -c/3)).
Proof. The first bound is obvious. Moreover, the fact that ν µ (x) ≤ c 3 implies that t(xx µ ) ≤ log 1 1-c/3 . Thus, we get the classical bounds on the Hessian using Eq. ( 5.14):

e -t(x-x µ ) H(x) H(x µ ) e t(x-x µ ) H(x).
1. Bound on µ x H -1 µ (x) . Using Eqs. ( 5.17) and ( 5.18),

µ x H -1 µ (x) = ∇f µ (x) -∇f (x) + ∇f (x µ ) -∇f (x µ ) H -1 µ (x) ≤ ν µ (x) + 1 0 H µ (x) -1/2 H(x t )(x -x µ ) dt + ∇f (x µ ) Hµ(x) , x t = tx + (1 -t)x µ . Now bound H µ (x) -1/2 H(x t )(x-x µ ) ≤ H µ (x) -1/2 H µ (x t ) 1/2 x-x µ H(xt)
and use Eq. ( 5.17) and Eq. ( 5.14) to get:

H µ (x) -1/2 H(x t )(x -x µ ) ≤ e t t(x-x µ ) x -x µ H(x) .
Integrating this yields:

1 0 H µ (x) -1/2 H(x t )(x -x µ ) dt ≤ φ(t(x -x µ )) x -x µ H(x) ≤ e t(x-x µ ) ν µ (x).
Where the last inequality is obtained using the bounds between gradient and hessian distance Eq. (5.18). Finally, using the bound on t(xx µ ),

µ x H -1 µ (x) ≤ 1 + 1 1 -c/3 ν µ (x) + 1 1 -c/3 ∇f (x µ ) H -1 µ (x µ ) . 2. Bound on R x . Start by decomposing R x ≤ R x µ + R x -x µ . Now bound R x -x µ ≤ R √ µ x -x µ Hµ(x) .
Using Eq. ( 5.17), xx µ Hµ(x) ≤ φ(-log(1 -c/3))ν µ (x). Hence:

R x ≤ R x µ + φ(-log(1 -c/3)) Rν µ (x) √ µ .
3. Now assume x ∈ D µ (c/3). Using the bound on µ x H -1 µ (x) , and noting that 1 r µ (x) ≤ e t(x-x µ )/2 1 r µ (x µ ) , it holds:

µ x H -1 µ (x) c r µ (x) ≤ 1 3 1 + 1 1 -c/3 + 1 1 -c/3 µx µ H -1 µ (x µ ) c r µ (x µ )
.

Now assume

Rνµ(x)

√ µ ≤ c 3 .
. We know that in particular, x ∈ D µ (c/3) and hence:

R √ µ x H -1 µ (x) ≤ 1 + 1 1 -c/3 Rν µ (x) √ µ + 1 1 -c/3 Rµ x µ H -1 µ (x µ ) √ µ ≤ 1 + 1 1 -c/3 c 3 + 1 1 -c/3 R √ µ x µ H -1 µ (x µ ) . Hence R √ µ x H -1 µ (x) c ≤ 1 + 1 1 -c/3 1 3 + 1 1 -c/3 R √ µ x µ H -1 µ (x µ ) c . Likewise: R x c ≤ R x µ c + 1 3 φ(-log(1 -c/3)).
We can get the following simpler bounds.

Corollary 5.3 (Application to c = 1 7 ). Applying Lemma 5.14 to c = 1 7 , we get the following bounds. Let µ > 0.

• For any x ∈ H, if x ∈ D µ (c/3), then the following holds:

7µ x H -1 µ (x) r µ (x) ≤ 1 + 8 µx µ H -1 µ (x µ ) r µ (x µ ) . • For any c < 1, x ∈ H, if Rνµ(x)
√ µ ≤ c 3 , then the following hold:

7R √ µ x H -1 µ (x) ≤ 1 + 8R √ µ x µ H -1 µ (x µ ) . 7R x ≤ 7R x µ + 1.

5.C .2 Proof of main theorems

In this section, we bound the number of iterations of our scheme in different cases.

Recall the proposed globalization scheme in the paper, where ANM ρ (f, x, t) is a method performing t successive ρ-relative approximate Newton steps of f starting at x.

Proposed Globalization Scheme

Phase I: Getting in the Dikin ellispoid of

f λ Start with x 0 ∈ H, µ 0 > 0, t, T ∈ N and (q k ) k∈N ∈ (0, 1]. For k ∈ N x k+1 ← ANM ρ (f µ k , x k , t) µ k+1 ← q k+1 µ k Stop when µ k+1 < λ and set x last ← x k . K ← k
Phase II: reach a certain precision starting from inside the Dikin ellipsoid Return x ← ANM ρ (f λ , x last , T ) Throughout this section, we will denote with K the value of k when the scheme stops, i.e. the first value of k such that µ k+1 < λ.

Adaptive methods We start by presenting an adaptive way to select µ k+1 from µ k , with theoretical guarantees. The main result is the following. Proposition 5.6 (Adaptive, simple version). Assume that we perform phase I starting at x 0 such that

Rν µ 0 (x 0 ) √ µ 0 ≤ 1 7 .
Assume that at each step k, we compute x k+1 using t = 2 iterations of the ρ-relative approximate Newton method. Then if at each iteration, we set:

µ k+1 = q k+1 µ k , q k+1 := 1 3 + 7R x k+1 1 + 7R x k+1 .
Then the following hold:

1. ∀k ≤ K + 1, Rνµ k (x k ) √ µ k ≤ 1 7 .
2. The decreasing parameter q k+1 is bounded above before reaching K:

∀k ≤ K, q k+1 ≤ 4 3 + 7R x µ k 2 + 7R x µ k ≤ 4 3 + 7R x λ 2 + 7R x λ . 3. K is finite, K ≤      log µ 0 λ log 2+7R x λ 4 3 +7R x λ      ≤ (3 + 11R x λ ) log µ 0 λ , and
Rν λ (x K+1 ) √ λ ≤ 1 7 .
Proof. Let us prove the three points one by one.

1. This is easily proved by induction, the keys to the induction hypothesis being:

• Using the induction hypothesis, x k ∈ D µ k (c) and hence, using proposition 5.5 shows that after two iterations of the approximate Newton scheme,

νµ k (x k+1 ) νµ k (x k ) ≤ 1 3 which implies Rνµ k (x k+1 ) √ µ k ≤ c 3 .
• Now using Lemma 5.13, we see that that since

7R x k+1 = R x k+1 c ≥ R √ µ k x k+1 H -1 µ k (x k+1 )
c , the hypotheses to guarantee the bound for q k+1 hold, hence

Rν µ k+1 (x k+1 ) √ µ k+1 ≤ c.
2. Using the second bullet point of Cor. 5.3, we see that the previous point implies

∀k ≤ K, 7R x k+1 ≤ 7R x µ k + 1 =⇒ q k+1 ≤ 4/3 + 7R x µ k 2 + 7R x µ k .
Now using the fact that for any k ≤ K, µ k > λ, we can use the simple fact that x λ ≥ x µ k to get the desired bound for q k+1 .

3. Using the previous point clearly shows the following bound:

∀k ≤ K + 1, µ k ≤ 4 3 + 7R x λ 2 + 7R x λ k µ 0 .
As this clearly converges to 0 when k goes to infinity, K is necessarily finite. Applying this for k = K, we see that:

λ ≤ µ K ≤ 4 3 + 7R x λ 2 + 7R x λ K µ 0 .
This shows that K ≤

log µ 0 λ log 2+7R x λ 4 3 +7R x λ .
The final bound is obtained noting that

2 + 7R x λ 4 3 + 7R x λ = 1 + 1 t , t = 2 + 21 2 R x λ ,
and using the classical bound:

1 log(1 + 1 t ) ≤ t + 1.
Finally, the fact that

Rν λ (x K+1 ) √ λ
≤ c is just a consequence of the fact that µ K+1 ≤ λ ≤ µ K and thus that λ = qµ K with q ≥ q K+1 , which is shown to satisfy the condition in Lemma 5.13. Hence, the lemma holds not only for µ K+1 but also for λ.

Remark 11 (µ 0 ). In the previous proposition, we assume start at x 0 , µ 0 such that

Rν µ 0 (x 0 ) √ µ 0 ≤ 1 7 .
A simple way to have such a pair is simply to select:

x 0 = 0, µ 0 = 7R ∇f (0) , since Rνµ 0 (x 0 ) √ µ 0 = R ∇f (0) H -1 µ 0 (0) √ µ 0 ≤ R ∇f (0) µ 0
.

Alternatively, if one can approximately compute x H -1 µ (x) , one can propose the following variant, whose proof is completely analogous. Proposition 5.7 (Adaptive, small variant version). Assume that we perform phase I starting at x 0 such that Rν µ 0 (x) √ µ 0 ≤ 1 7 .

Then if at each iteration, we set:

t k+1 = 7 7 6 R √ µ k √ x k+1 • s k+1 , s k+1 ∈ LinApprox(H µ k (x k+1 ), x k+1 , 1 7 ),
and

µ k+1 = q k+1 µ k , q k+1 := 1 3 + t k+1 1 + t k+1 .
Then the following hold:

1. ∀k ≤ K, Rνµ k (x k ) √ µ k ≤ 1 7 .
2. The decreasing parameter q k+1 is bounded above before reaching K:

∀k ≤ K, q k+1 ≤ sup µ 0 ≥µ≥λ 7 3 + 10R √ µ x µ H -1 µ (x µ ) 3 + 10R √ µ x µ H -1 µ (x µ ) ≤ 7 3 + 10R x λ 3 + 10R x λ .

3.

K is finite,

K ≤ 9 2 + 15 sup λ≤µ≤µ 0 R √ µ x µ H -1 µ (x µ ) log µ 0 λ ,
and

Rν λ (x K+1 ) √ λ ≤ 1 7 .
Proof. The main thing to note is that because of the properties of 1 7 -approximations, if

s k+1 ∈ LinApprox(H µ k (x k+1 ), x k+1 , 1 7 ), (1 - 1 7 ) x k+1 2 H -1 µ k (x k+1 ) ≤ x k+1 • s k+1 ≤ (1 + 1 7 ) x k+1 2 H -1 µ k (x k+1 ) .
Hence,

x k+1 H -1 µ k (x k+1 ) ≤ 7 6 √ x k+1 • s k+1 ≤ 4 3 x k+1 H -1 µ k (x k+1 ) . Hence, t k+1 ≥ 7R √ µ k x k+1 H -1 µ k (x k+1
) , and we can apply Lemma 5.13 to get the first point. To get the second point, we bound t k+1 above:

t k+1 ≤ 7 4 3 R √ µ k x k+1 H -1 µ k (x k+1
) . Now use Cor. 5.3 to find:

t k+1 ≤ 4 3 1 + 8R √ µ k x µ k H -1 µ k (x µ k ) ≤ 2 + 10R √ µ k x µ k H -1 µ k (x µ k
) .

5.C . GLOBALIZATION SCHEME

205

Thus,

q k+1 ≤ 7 3 + 10R √ µ k x µ k H -1 µ k (x µ k ) 3 + 10R √ µ k x µ k H -1 µ k (x µ k )
.

Note that as long as k ≥ K,

q k+1 ≤ sup µ≥λ 7 3 + 10R √ µ x µ H -1 µ (x µ ) 3 + 10R √ µ x µ H -1 µ (x µ ) ≤ 7 3 + 10R x λ 3 + 10R x λ .
This guarantees convergence.

For the last point, the proof is exactly the same as in the previous proposition.

General non-adaptive result. As mentioned in the core of the article, in practice, we do not select q k+1 at each iteration using a safe adaptative value, but rather decrease µ k+1 = qµ k with a constant q, which we see as a parameter to tune. The following result shows that for q large enough, this is justified, and that the lower bound we get for q depends on the radius of the Dikin ellipsoid r µ (x), instead of √ µ R in the previous bounds, which is somewhat finer and shows that if the data is structured such that this radius is very big, then q might actually be very small. Proposition 5.8 (Fixed q). Assume that we perform phase I starting at x 0 such that

x 0 ∈ D µ 0 ( 1 7
).

Assume we perform the method with a fixed q k+1 = q, satisfying q ≥ sup

λ≤µ≤µ 0 4 3 + 8 µ x µ H -1 µ (x µ ) rµ(x µ ) 2 + 8 µ x µ H -1 µ (x µ ) rµ(x µ )
.

Then the following hold:

1. ∀k ≤ K + 1, x k ∈ D µ k ( 1 7 ). 2. K is finite, K ≤ 1 1 -q log µ 0 λ , and x K+1 ∈ D λ ( 1 7 ).
Proof. Let us prove the two points.

1. Let us prove the result by induction. The initialization is trivial. Now assume x k ∈ D µ k ( 1 7 ). Performing two iterations of the approximate Newton method guarantees that

x k+1 ∈ D µ k ( 1 21 ),
as show in proposition 5.5. Now using Lemma 5.13, we see that

x k+1 ∈ D qµ k ( 1 7 ), provided that q ≥ 1 3 + 7µ k x k+1 H -1 µ k (x k+1 ) rµ k (x k+1 ) 1 + 7µ k x k+1 H -1 µ k (x k+1 ) rµ k (x k+1 )
. Now using Cor. 5.3, we get that

7µ k x k+1 H -1 µ k (x k+1 ) r µ k (x k+1 ) ≤ 1 + 8µ k x µ k H -1 µ k (x µ k ) r µ k (x µ k ) ≤ 1 + 8 sup λ≤µ≤µ 0 µ x µ H -1 µ (x µ ) r µ (x µ ) .
Hence the result.

2. This point just follows, using the bound

1 log 1 q ≤ 1 1-q .

5.C .3 Proof of Theorem 5.1

Using Remark 11, the fact that x 0 = 0 and µ 0 = 7R ∇f (0) , as well as the hypotheses of the theorem, we can apply proposition 5.6, and show that the number of steps K performed in the first phase is bounded:

K ≤ (3 + 11R x λ ) log(7R ∇f (0) /λ) .
Moreover, this proposition also shows that Rν λ (x last )/ √ λ ≤ 1 7 . Hence, we can use proposition 5.5: if

t ≥ T = log 2 λε -1 R 2 ≥ log 2 ν λ (x last ) √ ε , then it holds ν λ (x) ≤ √ ε and f λ (x) -f λ (x λ ) ≤ ε.
5.D . NON-PARAMETRIC LEARNING 207

5.D Non-parametric learning with generalized self-concordant functions

In this section, the aim is to provide a fast algorithm in the case of Kernel methods which achieves the optimal statistical guarantees.

5.D .1 General setting and assumptions, statistical result for regularized ERM.

In this section, we consider the supervised learning problem of learning a predictor f : X → Y from training samples (x i , y i ) 1≤i≤n which we assume to be realisations from a certain random variable

Z = (X, Y ) ∈ Z = X × Y whose distribution is ρ.
In what follows, for simplification purposes, we assume Y = R; however, this analysis can easily be adapted (although with heavier notations) to the setting where Y = R p . Our aim is to compute the predictor of minimal generalization error

inf f ∈H L(f ) := E z∼ρ [ z (f (x))], (5.29)
where H is a space of candidate solutions and z : R → R is a loss function comparing the prediction f (x) to the objective y.

Kernel methods. Kernel methods consider a space of functions H K implicitly constructed from a symmetric positive semi-definite Kernel K : X × X → and whose basic functions are the K x : t ∈ X → K(x, t) and the linear combinations of such functions f = m j=1 α j K x j .

It is endowed with a scalar product such that:

∀x 1 , x 2 ∈ X , K x 1 • K x 2 = K(x 1 , x 2 )
, and as a consequence, H K satisfies the self-reprocucing property:

∀x ∈ X , ∀f ∈ H, f (x) = f, K x H .
In order to find a good predictor for Eq. ( 5.29), the following estimator, called the regularized ERM estimator, is often computed:

f λ := arg min f ∈H L λ (f ) := 1 n n i=1 z i (f (x i )) + λ 2 f 2 H .
The properties of this estimator have been studied in [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF] for the square loss and Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] for generalized self-concordant functions. In Sec. 5.H , we recall the full setting of Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], and extend it to include the statistical properties of the projected problem.

Assumptions In this section, we will make the following assumptions, which are reformulations of the assumptions of Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], which we recall in Sec. 5.H , in order to have the statistical properties of the regularized ERM. First, we assume that the (x i , y i ) are i.i.d. samples. Assumption 5.1 (i.i.d. data). The samples (z i ) 1≤i≤n = (x i , y i ) 1≤i≤n ∈ Z n are independently and identically distributed according to ρ.

In the case where Y = R, we make the following assumptions on the loss, which leads to the self concordance of the mappings f → z (f (x)) and that of L, L...

208

CHAPTER 5. SECOND ORDER STRIKES BACK Assumption 5.2 (Technical assumptions). The mapping (z, t) ∈ Z × R → z (t) is measurable. Moreover,

• there exists R < ∞ such that for all z ∈ supp(Z),

∀t ∈ R, | (3) z (t)| ≤ R z (t),
• the random variables

| Z (0)|, | Z (0)|, | Z (0)| are are bounded;
• The kernel is bounded, i.e. ∀x ∈ supp(X), K(x, x) ≤ κ 2 for a certain κ.

Using these assumptions, we see that the following properties are satisfied. Define L z (f ) := z (f (x)). Then the L z satisfy the following properties:

• For any z ∈ Z, (L z , {R K x }) is a generalized self-concordant function in the sense of definition 5.4.

• The mapping (z, f ) ∈ Z × H → L z (f ) is measurable; • the random variables L Z (0) , ∇L Z (0) , Tr(∇ 2 L Z (0)) are bounded by | Z (0)|, κ| Z (0)|, κ 2 | Z (0)|; • G := {R K x : z ∈ supp(Z)} is a bounded subset of H, bounded by R = R κ.
This shows that Assumption 5.7 and Assumption 5.8 are satisfied by the L z and hence, using proposition 5.16 in the next appendix, L is well-defined, generalized self-concordant with G. Moreover, the empirical loss

L = 1 n n i=1 L z i , is also generalized self-concordant with G := {R K x i : 1 ≤ i ≤ n}.
Finally, as in Sec. 5.H , we make an assumption on the regularity of the problem; namely, we assume that a solution to the learning problem exists in H.

Assumption 5.3 (Existence of a minimizer). There exists

f ∈ H such that L(f ) = inf f ∈H L(f ).
We adopt all the notations from Sec. 5.H , doing the distinction between expected an empirical problems by adding a • over the quantities related to the empirical problem. We continue using the standard notations for L: for any f ∈ H and λ > 0,

L λ (f ) = L(f ) + λ 2 f 2 , L λ (f ) = L(f ) + λ 2 f 2 H(f ) = ∇ 2 L(f ), H λ (f ) = ∇ 2 L λ (f ) = H(f ) + λI H(f ) = ∇ 2 L(f ), H λ (f ) = ∇ 2 L λ (f ) = H(f ) + λI
Recall that f λ is defined as the minimizer of L λ .

Define the following bounds on the second order derivatives:

∀f ∈ H, b 2 (f ) = sup z∈supp(Z) z (f (x)).

5.D . NON-PARAMETRIC LEARNING

209

Statistical properties of the estimator The statistical properties of the estimator f λ have been studied in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] in the case of generalized self concordance, an are reported in the main lines in Sec. 5.H . The statistical rates of this estimator and the optimal choice of λ is determined by two parameters, defined in proposition 5.17 and which we adapt to the Kernel problem here.

• the bias b λ = H λ (f ) -1/2 ∇L λ (f ) = λ f H -1 λ (f )
, which characterizes the regularity of the optimum. The faster b λ decreases to zero, the more regular f is.

• the effective dimension

df λ = E H λ (f ) -1/2 ∇L Z (f ) 2 .
(5.30)

This quantity characterizes the size of the space H with respect to the problem; the slower it explodes as λ goes to zero, the smaller the size of H.

For more complete explanations on the meaning of these quantities, we refer to Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF].

Moreover, as mentioned in proposition 5.17, one can define

B 1 := sup z∈supp(Z) ∇L z (f ) , B 2 := sup z∈supp(Z) Tr(∇ 2 L z (f )), Q = B 1 B 2 , b 2 = b 2 (f ). (5.31)
We assume the following regularity condition on the minimizer f , in order to get statistical bounds.

Assumption 5.4 (Source condition). There exists r > 0 and g ∈ H such that f = H r (f )g. This implies the following decrease rate of the bias:

b λ ≤ Lλ 1/2+r , L = g H .
This is a stronger assumption than the existence of the minimizer as r > 0 is crucial for our analysis.

We also quantify the effective dimension df λ : (however, since it always holds for α = 1, this is not, strictly speaking, an additional assumption).

Assumption 5.5 (Effective dimension). The effective dimension decreases as df λ ≤ Qλ -1/α .

If these two assumptions hold, define:

β = α 1 + α(1 + 2r) , γ = (1 + 2r)α 1 + α(1 + 2r)
.

Under these assumptions, one can obtain the following statistical rates (which can be found in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] or in Cor. 5.4).

Proposition 5.9. Let δ ∈ (0, 1/2]. Under Assumptions 5.1 to 5.5, when n ≥ N and λ = (C 0 /n) β , then with probability at least 1 -2δ, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], and satisfying

L( f λ ) -L(f ) ≤ C 1 n -γ log 2 δ , with C 0 = 256(Q/L) 2 , C 1 = 8(256) γ (Q γ L 1-γ ) 2 and N defined in Marteau-Ferey, Ostrovskii,
N = O(poly(B 1 , B 2 , L, Q, R, log(1/δ))).

5.D .2

Reducing the dimension: projecting on a subspace using Nyström sub-sampling.

Computations Using a representer theorem, one of the key properties of Kernel spaces is that, owing to the reproducing property,

f λ ∈ H n := n i=1 α i K x i : (α i ) ∈ R n .
This means that solving the regularized empirical problem can be turned into a finite dimensional problem in α. Indeed f λ = n i=1 α i K x i where α = (α i ) 1≤i≤n is the solution to the following problem:

α = arg min α∈R n 1 n n i=1 z i (α K nn e i ) + λ 2 α K nn α, K nn = (K(x i , x j )) 1≤i,j≤n ∈ R n×n .
The previous problem is usually too costly to solve directly for large values of n, both in time and memory, because of the operations involving K nn . A solution consists in looking for a solution in a smaller dimensional sub-space H M constructed from sub-samples of the data {x 1 , ..., xM } ⊂ {x 1 , ..., x n }:

H M :=    M j=1 αj K xj : α ∈ R M    .
In this case, the minimizer f M,λ = arg min f ∈H M L λ (f ) can be written f M,λ = M j=1 αj K xj , where α is the solution to the following problem:

α = arg min α∈R M 1 n n i=1 z i (α K M n e i ) + λ 2 α K M M α,
where

K nM = (K(x i , xj )) 1≤i≤n 1≤j≤M , K M n = K nM , K M M := (K(x i , xj )) 1≤i,j≤M .
Let T be an upper triangular matrix such that T T = K M M . One can re-parametrize the previous problem in the following way. For any

β ∈ R M , define f β = M j=1 [T † β] j K xj . This implies in particular that f β H = β R M . Then f M,λ = f β M,λ , where β M,λ = arg min β∈R M L M,λ (β) := 1 n n i=1 z i (e i K nM T † β) + λ 2 β 2 .
Using the properties the z , one easily shows that β

→ z i (e i K nM T † β) is R T -K M n e i generalized self-concordant, and R T -K M n e i ≤ R K(x i , x i ).
Thus, L M is also generalized self-concordant, and the associated G M is bounded by R = R κ. It will therefore be possible to apply the second order scheme presented in this paper to approximately compute β M,λ .

Statistics Let ν λ,M (β) denote the Newton decrement of L λ,M at point β and P M denote the orthogonal projection on H M . Then the following statistical result shows that provided β is a good enough approximation of the optimum, and provided H M is large enough, then f β has the same generalization error as the empirical risk minimizer f λ .

Recall the following result proved in proposition 5.19 in Sec. 5.H .3.

Proposition 5.10 (Behavior of an approximation to the projected problem). Suppose that Assumptions 5.1 to 5.

3 are satisfied. Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Whenever n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , C 1 df λ ∨ (Q ) 2 n log 2 δ ≤ λ 1/2 R , C 1 b λ ≤ λ 1/2 R , if H 1/2 (f )(I -P M ) 2 ≤ λ √ 2 480 , 126 ν M,λ (β) ≤ λ 1/2 R ,
the following holds, with probability at least 1 -2δ.

L(f β ) -L(f ) ≤ K 1 b 2 λ + K 2 df λ ∨ (Q ) 2 n log 2 δ + K 3 ν 2 M,λ (β), R f β -f H ≤ 10,
where K 1 ≤ 6.0e4, K 2 ≤ 6.0e6 and K 3 ≤ 810, C 1 is defined in Lemma 5.19, and the other constants are defined in Theorem 5.8.

In particular, if we apply the previous result for a fixed λ, the following theorem holds (for a proof, see Sec. 5.H .4).

Theorem 5.5 (Quantitative result with source r > 0). Suppose that Assumptions 5.1 to 5.5 are satisfied. Let n ≥ N and δ ∈ (0,

1 2 ]. If λ = Q L 2 1 n α α(1+2r)+1
, and if

H 1/2 (f )(I -P M ) 2 ≤ λ √ 2 480 , ν M,λ (β) ≤ Q γ L 1-γ n -γ/2 ,
then with probability at least 1 -2δ,

L(f β ) -L(f ) ≤ K Q γ L 1-γ 2 1 n γ log 2 δ , R f β -f ≤ 10,
where N is defined in Eq. (5.42) and K ≤ 7.0e6.

The proof of the previous result is quite technical and can be found in Sec. 5.H , in Theorem 5.9.

5.D .3 A note on sub-sampling techniques

Let Z be a random variable on a Polish space Z and (v z ) z∈Z be a family of vectors in

H such that ||v|| L ∞ (Z) := sup z∈supp(Z) v z < ∞ is bounded. Assume that z 1 , ..., z n are i.i.d. samples from Z.
Define the following trace class Hermitian operators:

A = E [v Z ⊗ v Z ] , A = 1 n n i=1 v z i ⊗ v z i . 212 CHAPTER 5. SECOND ORDER STRIKES BACK Define N A (λ) := Tr(A -1 λ A), N A ∞ (λ) := sup z∈supp(Z) A -1/2 λ v z 2 .
(5.32)

We typically have:

N A (λ) ≤ N A ∞ (λ) ≤ v 2 L ∞ (Z)
λ .

We define the leverage scores associated to the points z i and A:

∀1 ≤ i ≤ n, ∀t > 0, l A i (t) = A -1/2 t v z i 2 = n (G nn + tnI) -1 G nn ii , (5.33)
where

G nn = (v z i • v z j ) 1≤i
,j≤n denotes the Gram matrix associated to the family v z i .

As in [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF], definition 1, we give the following definition for leverage scores.

Definition 5.5 (q-approximate leverage scores). given t 0 , a family ( lA i (t)) 1≤i≤n is said to be a family of q-approximate leverage scores with respect to A if

∀1 ≤ i ≤ n, ∀t ≥ t 0 , 1 q l A i (t) ≤ lA i (t) ≤ q l A i (t).
We say that a subset of m points {z 1 , ..., zm } ⊂ {z i : 1 ≤ i ≤ n} is:

• Sampled using q-approximate leverage scores for t if the zj = z i j where the i j are m i.i.d. samples from {1, ..., n} using the probability vector

p i = lA i (t) n ĩ=1 lA ĩ (t) .
In that case, we define A m := 1 m m j=1

1 np i j v zj ⊗ v zj .
• Sampled uniformly if the {i j : 1 ≤ j ≤ m} is a uniformly chosen subset of {1, ..., n} of size m. In this case, we define A m := 1 m m j=1 v zj ⊗ v zj . In Sec. 5.I .1, we present technical lemmas which allow us to show that if m is large enough, the following hold:

• A η (I -P m ) 2 ≤ 3η
, where P m is the orthogonal projection on the subspace induced by the v zj ;

• A m,λ is equivalent to A λ .

Remark 12 (cost of computing q-approximate leverage scores). In [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF], one can show that the complexity of computing q-approximate leverage scores can be achieved in: c samp = O(q 2 N A (λ) 2 min(n, 1/λ)) time (where a unit of time is a scalar product evaluation) and O(N A (λ) 2 + n) in memory.

5.D .4 Selecting the M Nyström points

In order for Theorem 5.5 to hold, we must subsample the M points such as to guarantee

H 1/2 (f )(I -P M ) 2 ≤ √ 2λ
480 . Since we must sub-sample the M points a priori, i.e. before performing the method, it is necessary to have sub-sampling schemes which do not depend heavily on the point. Define the covariance operator:

Σ = E [K X ⊗ K X ] . Since H(f ) = E [ Z (f (X)) K X ⊗ K X ], it is easy to see that H(f ) b 2 Σ. Note that for Σ, since Σ = 1 n n i=1 K x i ⊗ K x i
, the leverage scores have the following form:

∀1 ≤ i ≤ n, l Σ i (t) = n (K nn + λnI) -1 K nn ii . Proposition 5.11 (Selecting Nyström points). Let δ > 0. Let η = min( Σ , λ √ 2 1440(b 2 ∨1)
). Assume the samples {x 1 , ..., xM } are obtained with one of the following. 1. n ≥ M ≥ 10 + 160N Σ ∞ (η) log 8κ 2 ηδ using uniform sampling; 2. M ≥ 6 + 486q 2 N Σ (η) log 8κ 2 ηδ using q-approximate leverage scores with respect to

Σ for t = η, t 0 ∨ 19κ 2 n log n 2δ < η, n ≥ 405κ 2 ∨ 67κ 2 log 12κ 2 δ .
Then it holds, with probability at least 1 -δ:

Σ 1/2 η (I -P M ) ≤ 3η =⇒ H 1/2 (f )(I -P M ) 2 ≤ λ √ 2 480 .
Proof. The proof is a direct application of the lemmas in Sec. 5.I .1. Indeed, note that since

Σ = E [K X ⊗ K X ],
then the results can be applied with Z ← X and v z ← K x . Indeed, from Assumption 5.2, it holds: sup

x∈supp(X) K x 2 ≤ κ 2 .
We can now combine proposition 5.11 and proposition 5.10 to obtain the following statistical bounds for the optimizer of the projected Nyström problem β M,λ .

Theorem 5.6. Suppose that Assumptions 5.1 to 5. ) . Assume the samples {x 1 , ..., xM } are obtained with one of the following. 1. n ≥ M ≥ 10 + 160N Σ ∞ (η) log 8κ 2 ηδ using uniform sampling; 2. M ≥ 6 + 486q 2 N Σ (η) log 8κ 2 ηδ using q-approximate leverage scores with respect to

3 are satisfied. Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 ∧ 720 √ 2(b 2 ∨ 1) Σ . Assume n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , C 1 df λ ∨ (Q ) 2 n log 2 δ ≤ λ 1/2 R , C 1 b λ ≤ λ 1/2 R , Let η = λ √ 2 1440(b 2 ∨1
Σ for t = η, t 0 ∨ 19κ 2 n log n 2δ < η, n ≥ 405κ 2 ∨ 67κ 2 log 12κ 2 δ .
The following holds, with probability at least 1 -3δ.

L(f β M,λ ) -L(f ) ≤ K 1 b 2 λ + K 2 df λ ∨ (Q ) 2 n log 2 δ , R β M,λ ≤ R f + 10,
where K 1 ≤ 6.0e4, K 2 ≤ 6.0e6 and K 3 ≤ 810, C 1 is defined in Lemma 5.19, and the other constants are defined in Theorem 5.8.

Proof. This is simply a reformulation of proposition 5.10, noting that ν M,λ (β M,λ ) = 0 and that proposition 5.11 implies the condition on the Hessian at the optimum.

Provided source condition holds with r > 0, the conditions of this theorem are not void.

5.D .5 Performing the globalization scheme to approximate β M,λ

In order to apply proposition 5.10, one needs to control ν M,λ (β).

We will apply our general scheme to L M,λ in order to obtain such a control.

Performing approximate Newton steps

The key element in the globalization scheme is to be able to compute 1 7 -approximate Newton steps.

Note that at a given point β and for a given µ > 0 the Hessian is of the form:

H M,µ (β) = 1 n T -K M n D n (β)K nM T -1 + µI M ,
where

D n (β) = diag((d i (β)) 1≤i≤n
) is a diagonal matrix whose elements are given by d i (β) =

z i (e i K nM T -1 β).
Note that we can always write

H M,µ (β) = 1 n n i=1 u i (β)u i (β) + µI, u i (β) = d i (β)T -K M n e i
The gradient can be put in the following form:

∇ L M,µ (β) = 1 n T -K M n v + µβ, v = ( z i (e i K nM T -1 β)) 1≤i≤n .
Computing the gradient at one point therefore costs O(nM + M 2 ), this being the cost of computing K nM times a vector costs O(nM ) and computing T -1 times a vector takes O(M 2 ) since T is triangular. Moreover, the cost in memory is O(M 2 + n), M 2 being needed for the saving of T and n for the saving of the gradient; K nM times a vector can also be done in O(n) memory, provided we compute it by blocks.

On the other hand, computing the full Hessian matrix would cost nM 2 operations, which is un-tractable. However, computing a Hessian vector product can be done in O(nM + M 2 ) time, as for the gradient, which suggest using an iterative solver with preconditioning.

Computing x ∈ LinApprox(A, b, ρ) through pre-conditioned conjugate gradient descent. Assume we wish to solve the problem Ax = b where A ∈ R M ×M is a positive definite matrix and b is a vector of R M . If one uses the conjugate gradient method starting from zero, then if x k denotes the k-the iterate of the conjugate gradient algorithm, Theorem 6.6 in [START_REF] Saad | Iterative Methods for Sparse Linear Systems[END_REF] shows that

x k ∈ LinApprox(A, b, ρ), ρ = 2 Cond(A) -1 Cond(A) + 1 k .
where Cond(A) is the condition number of the matrix A, namely the ratio λmax(A) λ min (A) . If Cond(A) is large, this convergence can be very slow. The idea of preconditioning is to compute an

approximation matrix A such that 1 2 A A 3 2 A.
(5.34)

We then compute B a triangular matrix such that B B = A using a cholesky decomposition, which can be done in O(M 3 ), and note that B -AB -1 is very well conditioned; indeed, its condition number is bounded by 3.

Perform a conjugate gradient method to solve the pre-conditioned problem B -AB -1 z = B -b, and denote with z τ the τ -th iteration of this method. Then using the bound on the condition number, we find

z τ ∈ LinApprox(B -AB -1 , B -b, ρ), ρ = 2 √ 3 -1 √ 3 + 1 τ ,
which in turn implies that by setting x τ := B -1 z τ ,

x τ ∈ LinApprox(A, b, ρ), ρ = 2 √ 3 -1 √ 3 + 1 τ .
This shows that after at most τ = 3 iterations, provided A satisfies Eq. ( 5.34), x τ ∈ LinApprox(A, b, 1 7 ). The cost of this method is therefore O(M 3 + nM ) in time, and O(n + M 2 ) due to the computing of the preconditioner and computing matrix vector products by block. This does not include the cost of finding a suitable A.

Computing a suitable approximation of H M,µ (β) To compute a good pre-conditioner, we will subsample Q points i 1 , ..., i Q points from {1, ..., n}, and sketch the Hessian using these Q points.

Proposition 5.12 (Computing approximate newton steps). Let δ > 0. Let β ∈ R M and µ ≥ λ, and assume

19b 2 (f β )κ 2 n log n 2δ < λ and n ≥ 405b 2 (f β )κ 2 ∨ 67b 2 (f β )κ 2 log 12b 2 (f β )κ 2 δ
. Let μ = min(µ, H(f β ) ). Assume one of the following properties is satisfied

1. Q ≥ 10 + 160N H(f β ) ∞ (μ) log 8b 2 (f β )κ 2 μδ
with uniform sampling of the {i 1 , ..., i Q }. We set

D Q = diag( z i j (f β (x i j ))) 1≤j≤Q 2. Q ≥ 6 + 486q 2 N H(f β ) (μ) log 8b 2 (f β )κ 2 μδ
using q-approximate leverage scores associated to

H(f β ) for t = μ. We set D Q = diag z i j (f β (x i j )) p i j
, where the p i j are the probabilities computed from the leverage scores.

Assume we use a pre-conditioner B such that

B B = 1 Q T -K M Q D Q K QM T -1 + µI M , K QM = (K(x i j , xk )) 1≤j≤Q 1≤k≤M .
If we perform τ = log(ρ/2)/ log(( √ 3 + 1)/ √ 3 -1) iterations of the conjugate gradient descent on the pre-conditioned Newton system using B as a preconditioner, then with probability at least 1 -δ, this procedure is returns ∆ ∈ LinApprox( H M,λ (β), ∇ L M,λ (β), ρ), and the computational time is of order O(τ (M n + M 2 Q + M 3 + c samp )), and the memory requirements can be reduced to O(M 2 + n). Here c samp stands for the complexity of computing Nystrom leverage scores, and using Remark 12 or [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF], c samp = O(1) if uniform sampling is used, and

c samp = O(N H(f β ) (μ) 2 /λ) if Nystrom sub-sampling is used. Note that for τ = 3, ρ = 1 7 .
Proof. Start by defining the following operators:

• K n : f ∈ H → (f (x i )) 1≤i≤n ∈ R n ; • K M : f ∈ H → (f (x j )) 1≤j≤M ∈ R M ; • V = K * M T -1
, where T is an upper triangular matrix such that

T T = K M M = K M K * M . Note that K n V = K nM T -1 . Now note that ∀f ∈ H, H(f ) = E [v z ⊗ v z ] , H(f ) = 1 n n i=1 v z i ⊗ v z i , v z = z (f (x))K x .
Since for any f ∈ H,

H(f ) = 1 n K * n D n (f )K n , where D n (f ) = diag( z i (f (x i ))), we see that H M,µ (β) = V * H(f β )V + µI M .
Thus, the last lemma of Sec. 5.I .1 can be applied, using the fact that v z 2 ≤ b 2 (f )κ 2 , to get that in both cases of the proposition, under the corresponding assumptions:

1 2 1 Q T -K M Q D Q K QM T -1 + µI M H M,µ (β) 3 2 1 Q T -K M Q D Q K QM T -1 + µI M .
The rest of the proposition follows from the previous discussion.

Applying the globalization scheme to control ν M,λ (β)

In order to apply proposition 5.12 to each point β in our method, we need to have a globalized version of the condition of this proposition.

First, we start by localizing the different values of β we will visit throughout the algorithm.

Definition 5.6 (path of regularized solutions). Let λ > 0, ε > 0. Define the path of regularized solutions

Γ M λ := {β M,µ : µ ≥ λ} . (5.35)
And the ε approximation of this path:

Γ M λ,ε := β ∈ R M : d(β, Γ M λ ) ≤ ε .
(5.36)

Note that we always have Γ M λ ⊂ B R M ( β M,λ ). We now state a lemma proving that all the values visited during the algorithm will lie in an approximation of this path.

Lemma 5.15. Define Let β ∈ R M such that ν M,µ (β) ≤ µ 1/2
7R for some µ ≥ λ. Then the following holds:

β ∈ Γ M λ, 1 6R . Proof. Bound R β -β M,µ ≤ R µ 1/2 β -β M,µ H M,µ (β) ≤ 1 φ(t M (β -β M,µ )) R ν M,µ (β) µ 1/2 .
Just apply Eq. ( 5.18) to obtain R ββ M,µ ≤ 1 6 .

We now introduce the following quantities which will allow to control the number of sub-samples throughout the whole algorithm.

Definition 5.7. Define

• b 2 := sup β∈ Γ M λ,1/6R b 2 (f β ). • N H (λ) = sup β∈ Γ M λ,1/6R N H(f β ) (λ). • N H ∞ (λ) = sup β∈ Γ M λ,1/6R N H(f β ) ∞ (λ). • H = min β∈ Γ M λ,1/6R
H(f β ) .

Proposition 5.13 (Performance of the globalization scheme). Let ε > 0, δ > 0, λ = min(λ, H ).

Assume 19b 2 κ 2 n log n 2δ < λ and n ≥ 405b 2 κ 2 ∨ 67b 2 κ 2 log 12b 2 κ 2 δ .
Assume we perform the globalization scheme with the parameters in Theorem 5.1, where in order to compute any ρ approximation of a regularized Newton step, we use a conjugate gradient descent on the pre-conditioned system, where the pre-conditioner is computed as in proposition 5.12 using

1. Q ≥ 10 + 160N H ∞ ( λ) log 8b 2 κ 2 λδ if using uniform sampling 2. Q ≥ 6 + 486q 2 N H ( λ) log 8b 2 κ 2 λδ if using Nyström leverage scores
Recall that t denotes the number of approximate Newton steps performed at for each µ in Phase I and T denotes the number of approximate Newton steps performed in Phase II, and that using Theorem 5.1, t = 2 and T = log 2 1 ∨ (λε -1 /R 2 ) . Moreover, recall that K denotes the number of steps performed in Phase I. Define

N ns := 2 (3 + 11R β M,λ ) log 2 (7R ∇ L M (0) /λ) + log 2 1 ∨ (λε -1 /R 2 ) .
Then with probability at least (1 -δ) Nns :

• The method presented in proposition 5.12 returns a 1/7approximate Newton step at each time it is called in the algorithm.

• If β denotes the result of the method, ν M,λ (β) ≤ √ ε.

• The number of approximate Newton steps computed during the algorithm is bounded by N ns ; the complexity of the method is therefore of order O(N ns (M 2 max(M, Q) + nM + c samp (λ))) in time and O(M Q + M 2 + n) in memory, where c samp (λ) is a bound on the complexity associated to the computing of leverage scores (see [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF] for details).

The algorithm is detailed in Sec. 5.E , in algorithm 1. Note however that the notations are those of the main paper, which are slightly different from the ones used here.

Proof. If we take the globalization scheme, using the parameters of Theorem 5.1. Assume that all previous approximate Newton steps have been computed in a good way. Then the β at which we are belongs to Γ M λ,1/6R . Thus, the hypotheses of this proposition imply that the hypothesis of proposition 5.12 are satisfied; and hence, up to a (1 -δ) probability factor, we can assume that the next approximate Newton step is performed correctly, continuing the globalization scheme in the right way. Thus, the globalization scheme converges as in Theorem 5.1.

5.D .6 Statistical properties of the algorithm

The following theorem describes the computational and statistical behavior of our algorithm.

Proposition 5.14 (Behavior of an approximation to the projected problem). Suppose that Assumptions 5.1 to 5.3 are satisfied.

Let n ∈ N, ε > 0, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Define λ = min(λ, H ) and assume 19b 2 κ 2 n log n 2δ < λ, n ≥ 405b 2 κ 2 ∨ 67b 2 κ 2 log 12b 2 κ 2 δ , and
n ≥ 1 B 2 λ log 8 2 1 B 2 λδ . Assume C 1 df λ ∨ (Q ) 2 n log 2 δ ≤ λ 1/2 R , C 1 b λ ≤ λ 1/2 R , 126 √ ε ≤ λ 1/2 R .
Assume that the M points x1 , ..., xM are obtained through Nyström sub-sampling using η = Σ ∧ λ √ 2 1440(b 2 ∨1) , with either 1. M ≥ 10 + 160N Σ ∞ (η) log 8κ 2 ηδ if using uniform sampling; 2. M ≥ 6 + 486q 2 N Σ (η) log 8κ 2 ηδ if using q-approximate leverage scores for η, associated to the co-variance operator Σ.

Assume we perform the globalization scheme as in proposition 5.13, i.e. with the parameters in Theorem 5.1, where in order to compute any ρ approximation of a regularized Newton step, we use a conjugate gradient descent on the pre-conditioned system, where the pre-conditioner is computed as in proposition 5.12 using

1. Q ≥ 10 + 160N H ∞ ( λ) log 8b 2 κ 2 λδ if using uniform sampling 2. Q ≥ 6 + 486q 2 N H ( λ) log 8b 2 κ 2
λδ if using Nyström leverage scores Let N ns be defined as in proposition 5.13. Recall N ns is an upper bound for the number of approximate Newton steps performed in the algorithm. One can bound

N ns ≤ 2 (113 + 11R f ) log 2 7R ∇ L M (0) λ + log 2 λ 1/2 Rε .
Moreover, with probability at least 1 -(N ns + 2)δ, the following holds:

L(f β ) -L(f ) ≤ K 1 b 2 λ + K 2 df λ ∨ (Q ) 2 n log 2 δ + K 3 ε.

5.D . NON-PARAMETRIC LEARNING

219

where K 1 ≤ 6.0e4, K 2 ≤ 6.0e6 and K 3 ≤ 810, C 1 is defined in Lemma 5.19, and the other constants are defined in Theorem 5.8.

Proof. This is a simple combination between propositions 5.10, 5.11 and 5.13. To bound the number of Newton steps N ns , one simply uses the fact that under the conditions of the theorem, R β M,λ ≤ 10 + R f .

Remark 13 (Complexity). Let L = b 2 κ 2 . The complexity of the previous method using leverage scores computed for Σ for the Nystrom projections and for H(f β ) for choosing the Q points at the different stages is the following. The total complexity in time will be of order:

O N ns nN H (λ) log(Lλ -1 δ -1 ) + b 3 2 N Σ (λ) 3 log 3 (Lλ -1 δ -1 ) + L/λ b 2 2 N Σ (λ) 2 .
The memory complexity can be bounded by

O(b 2 2 N Σ (λ) 2 log 2 (Lλ -1 δ -1 ) + n).
Here, we use the fact that H ≤ b 2 Σ.

We can now write down the previous proposition by classifying problems using Assumptions 5.4 and 5.5 and in order to get optimal rates.

Theorem 5.7 (Performance of the scheme using pre-conditioning). Let δ > 0. Assume Assumptions 5.1 to 5.5 are satisfied. Let n ≥ Ñ , where Ñ is characterized in the proof,

λ = Q L 2 1 n α α(1+2r)+1 .
Assume that the M points x1 , ..., xM are obtained through Nyström sub-sampling using η = λ √ 2 1440(b 2 ∨1) , with either 1. M ≥ 10 + 160N Σ ∞ (η) log 8κ 2 ηδ if using uniform sampling; 2. M ≥ 6 + 486q 2 N Σ (η) log 8κ 2 ηδ if using q-approximate leverage scores for η, associated to the co-variance operator Σ.

Assume we perform the globalization scheme as in proposition 5.13, i.e. with the parameters in Theorem 5.1, where in order to compute any ρ approximation of a regularized Newton step, we use a conjugate gradient descent on the pre-conditioned system, where the pre-conditioner is computed as in proposition 5.12 using

1. Q ≥ 10 + 160N H ∞ (λ) log 8b 2 κ 2 λδ if using uniform sampling 2. Q ≥ 6 + 486q 2 N H (λ) log 8b 2 κ 2
λδ if using Nyström leverage scores Let N ns be defined as in proposition 5.13. Recall N ns is an upper bound for the number of approximate Newton steps performed in the algorithm. One can bound

N ns ≤ (227 + 22R f ) log 2 7R ∇ L M (0) + log 2 nL 2 Q 2 + log 2 1 RL .
Moreover, with probability at least 1 -(N ns + 2)δ, the following holds:

• all of the approximate Newton methods yield 1 7 -approximate Newton steps • The scheme finishes, and the number of approximate Newton steps is bounded by N ns . The total complexity of the method is therefore

O((nM + M 3 + M 2 Q + c samp )N ns ) in time , O(n + M 2 ) in memory.
• The returned β is statistically optimal:

L(f β ) -L(f ) ≤ K Q γ L 1-γ 2 1 n γ log 2 δ ,
where K is defined in Theorem 5.5.

Proof. The proof consists mainly of combining propositions 5.11 and 5.13 and Theorem 5.5.

Recall that we set λ

= Q 2 L 2 1 n α α(1+2r)+1 .
1. Start by defining Ñ such that:

• Ñ ≥ N where N is defined in Theorem 5.5;

• ∀n ≥ Ñ , λ ≤ H . This is possible as α α(1+2r)+1 is a strictly positive exponent.

• ∀n ≥ Ñ , 19b 2 ∨1 κ 2 n log n 2δ < λ; this is possible as soon as α α(1+2r)+1 < 1, i.e. this is satisfied since r > 0;

• Ñ ≥ 405b 2 ∨ 1 κ 2 ∨ 67b 2 ∨ 1 κ 2 log 12b 2 ∨1 κ 2 δ ; • ∀n ≥ Ñ , λ √ 2 1440(b 2 ∨1) ≤ Σ .
We see that such a Ñ can be defined explicitly.

2.

Combining the assumptions on Ñ with the ones on M , we see that all the assumptions of proposition 5.11 are satisfied and thus that with probability at least 1 -δ, all the hypotheses for Theorem 5.5 are satisfied except the bound on ν M,λ (β).

3. Applying proposition 5.13, taking

√ ε = Q γ L 1-γ n -γ/2 and λ = Q 2 L 2
1 n α α(1+2r)+1 , we see that under these hypotheses,

N ns :=2 (3 + 11R β M,λ ) log 2 7R ∇ L M (0) nL 2 Q 2 α α(1+2r)+1 + log 2 1 RL nL 2 Q 2 rα α(1+2r)+1
. Now we can bound this harshly:

N ns ≤ (7 + 22R β M,λ ) log 2 7R ∇ L M (0) + log 2 nL 2 Q 2 + log 2 1 RL . Now bounding R β M,λ ≤ 10 + R f , we get N ns ≤ (227 + 22R f ) log 2 7R ∇ L M (0) + log 2 nL 2 Q 2 + log 2 1 RL .

4.

Finally, we use a union bound to conclude.

5.E . ALGORITHM 221

5.E Algorithm

Let N, M ∈ N with M ≤ N . In Alg. 1, leverage-scores-sampling((z i ) N i=1 , M, k, λ) returns a subset of (z i ) N i=1 of cardinality M sampled by using (approximate) leverage scores at scale λ > 0 and computed using the kernel k. An explicit example of an algorithm computing leverage-scores-sampling is in [START_REF] Rudi | On fast leverage score sampling and optimal learning[END_REF]. Moreover kernel-matrix((

x i ) N i=1 , (x i ) M i=1 , k) computes the kernel matrix K ∈ R N ×M where K ij = k(x i , x j ), with N, M ∈ N.
Algorithm 1 Algorithm efficient non-parametric learning for generalized self-concordant losses with optimal statistical guarantees discussed in Sec. 5.4 of the main paper.

Input: (xi, yi) n i=1 , n ∈ N, loss function, k kernel function and λ > 0. Return: estimated function g : X → R Parameters: Q, M, T ∈ N, µ0 > 0, (q k ) k∈N . Fixed parameters: t = 2 from Theorem 5.1, τ = 3 from proposition 5.12 in Sec. 5.D .5.

(xj) M j=1 ← leverage-scores-sampling((xi) n i=1 , M, λ, k) K ← kernel-matrix((xj) M j=1 , (xj) M j=1 ) T ← cholesky-upper-triangular(K) define the function v(•) = (k(x1, •), . . . , k(xM , •)) ∈ R M
define compute-preconditioner:

Input: α ∈ R M , λ > 0 ci ← (2) (v(xi) T -1 α, yi) for all i = 1, . . . , n define the function k (•, •) as k (•, •) := c• × c• × k(x•, x•) for •, • ∈ {1, . . . , n} (hs) Q s=1 ← leverage-scores-sampling((i) n i=1 , Q, λ, k ) G ← kernel-matrix((xj) M i=1 , (x hs ) Q s=1 , k) H ← T -× G × diag((c 2 l h ) Q h=1 ) × G × T -1 B ← cholesky-upper-triangular( 1 Q H + λI) return B define preconditioned-conj-grad: Input: α ∈ R M , µ > 0, r ∈ R M , τ ∈ N, B ∈ R M ×M p ← r, s0 ← r 2 , β ← 0 For i = 1, . . . , τ z ← µB -B -1 p + 1 n n i=1 (2) (v(xi) T -1 α, yi) (v(xi) T -1 B -1 p) B -T -v(xi) a ← s0/(p z) β ← β + ap r ← r -az, s1 ← r 2 p ← r + (s1/s0)p s0 ← s1 return β
define appr-linear-solver:

Input: α ∈ R M , µ > 0, g ∈ R M B ← compute-preconditioner(α, µ) u ← preconditioned-conjugate-gradient(α, µ, B -g, τ = 3, B) return B -1 u define approximate-Newton: Input: α0 ∈ R M , µ > 0, t ∈ N For j = 1, . . . , t g ← µαj-1 + 1 n n i=1 (1) (v(xi) T -1 αj-1, yi) T -v(xi) αj ← αj-1 -appr-linear-solver(αj-1, µ, g) return αt α0 ← 0 For k ∈ N α k+1 ← approximate-Newton(α k , µ k , t = 2) µ k+1 ← q k+1 µ k Stop when µ k+1 < λ and set α last ← α k α ← approximate-Newton(α last , λ, T ) return g(•) := v(•) T -1 α

5.F Experiments

We present our algorithm's performance for logistic regression on two large scale data sets: Higgs and Susy. We have implemented our method using pytorch, and performed computations on one node of a Tesla P100-PCIE-16GB GPU. Recall that in the case of logistic regression,

(x,y) (t) = log(1 + e -yt ).
In what follows, denote with n the cardinality of the data set and d the number of features of this data set. The error is measured in terms of classification error for both data sets. In both cases, we pre-process the data by substracting the mean and dividing by the standard deviation for each feature. The data sets are the following.

Susy (n = 5 × 10 6 , d = 18, binary classification). We always use a Gaussian Kernel with σ = 5 for logistic loss (obtained through a grid search; note that in [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF], σ = 4 is used for the square loss), and will always use 10 4 Nystrom points.

Higgs (n = 1.1 × 10 7 , d = 28, binary classification). We then apply a Gaussian Kernel with σ = 5, as in Rudi, Carratino, and Rosasco (2017) (we have also performed a grid search).

For these data sets, we do not have a fixed test set, and thus set apart 20% of the data set at random to be the test set, and use the rest of the 80% to train the classifier.

In practice, we perform our globally convergent scheme with the following parameters.

• We use Q = M uniform random features to compute the pre-conditioner for each approximate Newton step;

• In the first phase, we decrease µ in a very fast way to λ by starting at µ = 1 and dividing µ by 1000 after performing only a single approximate Newton step (using 2 iterations of conjugate gradient descent);

• In the second phase, we perform 10 approximate Newton steps (each ANS is computed using 8 iterations of conjugate gradient descent).

Selection of λ In the introduction, we claim that in many a learning problem, the parameter λ obtained through cross validation is often much smaller than the ones obtained in statistical bounds which are usually of order 1 √ n . This leads to very ill conditioned problems. For both data sets, we select λ (and σ, but we omit the double tables from this paper) by computing the test loss and classification errors for different values of λ, and report the evolution of these losses as a function of the parameter λ in Fig. 5.2 for the Higgs data set, and Fig. 5.3 for the Susy data set. We see that the optimal λ yield strongly ill-conditioned problems.

Comparison with accelerated methods Given the M Nystrom points, our aims to minimize L M,λ . From an optimization point of view, i.e. from a point of view where the aim is to minimize L M,λ , we compare our method with a large mini-batch version of Katyusha accelerated SVRG (see [START_REF] Allen-Zhu | Katyusha: The first direct acceleration of stochastic gradient methods[END_REF]). Indeed, we perform this method using batch sizes of size M ; the theoretical bounds provided in [START_REF] Allen-Zhu | Katyusha: The first direct acceleration of stochastic gradient methods[END_REF] show that the algorithm has linear convergence, with a time complexity of order O(nM + M 3 + M 2 L λ ) log 1 ε to reach precision ε. In the following plots, we compare both methods in terms of passes and time.

By pass, we mean the following.

• In the case of our second-order scheme, we define a pass on the data to be one step of the conjugate gradient descent used to compute approximate newton steps.

• In the case of Katyusha SVRG, we define a pass on the data to be either a full gradient computation or n/M computations of the type K τ M T -1 β where T is an upper triangular matrix, and K τ M is a M × M kernel matrix, associated to one batch gradient.

We use this notion to measure the speed of our method as they both correspond to natural O(nM ) operations, and incorporate the essential of the computing time. However, the second point is often much slower to compute than the first, due to the solving of the triangular system. Thus, the notion of passes is to take with precaution, as a pass for the accelerated SVRG algorithm takes much longer to run that a pass for our method. This is confirmed by the time plots (see Fig. 5.5 for in instance).

Comparison between the two methods -Due to the running time of K-SVRG, we compare both methods for M = 10000 Nyström points for both data sets. We compare the performance of these two algorithm with respect to the distance to the optimum in function values as well as classification error Fig. 5.4 for the Higgs data set, and in Fig. 5.5 for the Susy data set. Note on the need for precise optimization -As noted in the introduction, we see in both Fig. 5.5 and Fig. 5.4 that precise optimization of the objective function is needed in order to get a good classification error. This justifies a posteriori the use of a second order method. In particular, in Fig. 5.5, one notes the difference in behavior between the two methods : the second order method converges linearly in a fast way while the first order method slows down because of the condition number.

Note on ill-conditioning -First note that in order to optimize test error, one gets very poorly conditioned problems. As predicted by the rates, we observe that K-SVRG is more sensible to ill-conditioning than our second order scheme. Indeed, in Fig. 5.6, we have plotted the results for Susy for a smaller condition number with λ = 10 -8 , compared to λ = 10 -10 to get optimal test error in Fig. 5.5. We see that the difference in number of passes needed to reach a certain precision is much lower when λ = 10 -8 in Fig. 5.6, confirming that K-SVRG behaves better when the condition number is smaller. Performance of our method. In Table 5.1, we record the performance of the following methods, taking the λ values we have obtained previously for the different data sets.

For FALKON (see [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF]), we take the parameters suggested in the paper (except for the number of Nyström points needed for Higgs, as our computational capacity is limited).

5.G Solving a projected problem to reduce dimension

5.G .1 Introduction and notations

In this section, we give ourselves a generalized self-concordant function f whose associated subset we denote with G. Once again, we will always omit the subscript f in the notations associated to f . The aim of this section is the following. Given f and λ > 0, computing an approximate solution to

x λ = arg min x∈H f λ (x),
is often too costly. Instead, we look for a solution in a small subset of H which we see as the image of a certain orthogonal projector P and which we denote H P . Usually, this subset will be finite dimensional and admit an easy parametrization. Thus we will compare an approximation of x λ to an approximation of

x * P,λ = arg min x∈H P f λ (x) = arg min x∈H f (Px) + λ 2 x 2 .
Denote with f P the mapping x ∈ H → f (Px). It is easy to see that, as f is a generalized self-concordant function with G, f P is naturally a generalized self-concordant with G P := PG = {Pg : g ∈ G}. Moreover, x * P,λ = x f P ,λ .

We will adopt the following notations for the quantities related to the generalized self-concordant function f P . Essentially, we always replace f P simply by P from our definitions in appendix.

• For the regularized function :

∀x ∈ H, ∀λ > 0, f P,λ (x) = f P (x) + λ 2 x 2 .
• For the Hessians ∀x ∈ H, λ > 0, H P,λ (x) = H f P ,λ (x) = PH(Px)P + λI.

• ∀h ∈ H, t P (h) := t f P (h) = t(Ph).

• For the Newton decrement:

∀x ∈ H, λ > 0, ν P,λ (x) = ν f P ,λ (x) = ∇f P,λ H -1 P,λ (x) = P∇f (Px) + λx H -1 P,λ (x) .
• For the Dikin ellipsoid radius:

∀λ > 0, ∀x ∈ H, r P,λ (x) := r f P ,λ (x) = 1 sup g∈G Pg H -1 λ,P (x)
;

• For the Dikin ellipsoid: ∀λ > 0, ∀c ≥ 0, D P,λ (c) := D f P ,λ (c).

CHAPTER 5. SECOND ORDER STRIKES BACK

Note that for any x ∈ H P , r P,λ (x) ≥ r λ (x).

We will now introduce the key quantities in order to compare an approximation of x * P,λ to an approximation of x λ . Definition 5.8 (key quantities). Define the following quantities

• For any λ > 0, the source term s λ := λ x λ H -1 λ (x λ ) = ∇f (x λ ) H -1 λ (x λ ) ; • Given an orthogonal projector P, λ > 0, and x ∈ H, the capacity of the projector C P (x, λ) := H(x) 1/2 (I-P) 2 λ .

5.G .2 Relating the projected to the original problem

Given x ∈ H P , our aim is to bound ν λ (x) given ν λ,P (x) and s λ .

Proposition 5.15. Let x ∈ H P . If

s λ r λ (x λ ) ≤ 1 4 , C P (x λ , λ) ≤ 1 120 , ν P,λ (x) ≤ r P,λ (x) 2 ,
Then it holds:

ν λ (x) ≤ 3(ν P,λ (x) + s λ ).
Moreover, under these conditions,

• xx λ ≤ 7λ -1/2 (ν P,λ (x) + s λ );

• λ x H -1 P,λ (x) ≤ 7ν P,λ (x) + 9s λ .

Proof. In this proof, introduce the following auxiliary quantity:

γ λ := s λ r λ (x λ ) .
1) Start by bounding t(Px λ -x λ ). It holds:

t(Px -x λ ) = sup g∈G |g • (I -P)x λ | ≤ 1 r λ (x λ ) (I -P)x λ H λ (x λ ) ≤ 1 r λ (x λ ) H λ (x λ ) 1/2 (I -P)H λ (x λ ) 1/2 H -1/2 λ (x λ )x λ = (1 + C P (x λ , λ)) λ H -1/2 λ (x λ )x λ r λ (x λ ) = (1 + C P (x λ , λ)) γ λ .

5.G . PROJECTED PROBLEM 229

2) Then bound t(x * P,λ -Px λ ) First, bound ν P,λ (Px λ ):

ν P,λ (Px λ ) = P∇f λ (Px λ ) H λ,P (Px λ ) -1 ≤ ∇f λ (Px λ ) H λ (Px λ ) -1 .
Using Eq. ( 5.17), we get ∇f λ (Px λ ) H λ (Px λ ) -1 ≤ e t((I-P)x λ )/2 ν λ (Px λ ). Using Eq. ( 5.20), we can bound

ν λ (Px λ ) ≤ φ(t((I -P)x λ )) (I -P)x λ H λ (x λ ) ≤ φ(t((I -P)x λ )) (1 + C P (x λ , λ))s λ .
Putting things together, ν P,λ (Px λ ) ≤ e t((I-P)x λ )/2 φ(t((

I -P)x λ )) (1 + C P (x λ , λ))s λ . Now 1 r P,λ (Px λ ) ≤ 1 r λ (Px λ )
≤ e t((I-P)x λ )/2 1 r λ (x λ ) .

Hence,

ν P,λ (Px λ ) r P,λ (Px λ ) ≤ e tλ φ( tλ ) tλ , tλ = (1 + C P (x λ , λ))γ λ .
Since t → e t φ(t) t is an increasing function whose value in 0 is 0, we find numerically that for t = 3 10 , e t φ(t) t ≤ 1 2 . Hence, if (1 + C P (x λ , λ))γ λ ≤ 3 10 , then ν P,λ (Px λ ) r P,λ (Px λ ) ≤ 1 2 . Using Lemma 5.5, this shows that t P (Px λ -x * P,λ ) = t(Px λ -x * P,λ ) ≤ log 2.

3) Getting a bound for t(xx λ ). To do so, combine the two previous bounds with the fact that if ν P,λ (x) ≤ r P,λ (x) 2

, then using Lemma 5.5 with f P , t P (xx * P,λ ) = t(xx * P,λ ) ≤ log 2. Thus, if

(1 + C P (x λ , λ))γ λ ≤ 3 10 , ν P,λ (x) ≤ r P,λ (x) 2 , then it holds t(x -x λ ) ≤ 3 10 + 2 log 2. 4) A technical result to bound H λ (x) -1/2 H P,λ (x) 1/2
. Using the fact that Px = x, and Lemma 5.23, applied to A = H(x), we get

H λ (x) -1/2 H P,λ (x) 1/2 ≤ 1 + C P (x, λ).
Then, one can easily bound C P (x, λ) ≤ e t(x-x λ ) C P (x λ , λ).

230

CHAPTER 5. SECOND ORDER STRIKES BACK 5) Let us now bound ν λ (x). First, decompose the term

ν λ (x) = ∇f λ (x) H -1 λ (x) ≤ P∇f λ (x) H -1 λ (x) + (I -P)∇f (x) H -1 λ (x) . Since x ∈ H P , P∇f λ (x) H -1 λ (x) = ∇f P,λ (x) H -1 λ (x)
, and using the previous point, we get

P∇f λ (x) H -1 λ (x) ≤ 1 + e t(x-x λ )/2 C P (x λ , λ) ν P,λ (x).
Let us now bound the second term. We divide it into two terms:

(I -P)∇f (x) H -1 λ (x) ≤ (I -P) (∇f (x) -∇f (x λ )) H -1 λ (x) + (I -P)∇f (x λ ) H -1 λ (x)
. The second term can be bounded in the following way:

(I -P)∇f (x λ ) H -1 λ (x) ≤ 1 √ λ (I -P)H 1/2 λ (x λ ) ∇f (x λ ) H -1 λ (x λ ) ≤ 1 + C P (x λ , λ) s λ .
For the first term, we proceed in the following way.

(

I -P) (∇f (x) -∇f (x λ )) H -1 λ (x) = 1 0 H -1/2 λ (x)(I -P)H(x t )(x -x λ ) dt ≤ 1 √ λ 1 0 (I -P)H 1/2 (x t ) H 1/2 (x t )(x -x λ ) dt ≤ C P (x λ , λ) φ(t(x -x λ )) x -x λ H(x λ )
≤ C P (x λ , λ) e t(x-x λ ) ν λ (x).

Hence the final bound:

1 -C P (x λ , λ) e t(x-x λ ) ν λ (x) ≤ 1 + e t(x-x λ )/2 C P (x λ , λ) ν P,λ (x) + 1 + C P (x λ , λ) s λ .

Now if C P (x λ , λ) ≤ 1 120 , we see that C P (x λ , λ) e t(x-x λ ) ≤ 1 2 , and hence, using the bound on t(xx λ ),

ν λ (x) ≤ 3(ν P,λ (x) + s λ ).
6) Showing the last two points . We leverage the fact that ν λ (x) ≤ 3(ν P,λ (x) + s λ ) and t(xx λ ) ≤ 3 10 + 2 log 2. To show the first bound, we plug in the previous results in the following equation:

x -x λ ≤ λ -1/2 x -x λ H λ (x) ≤ 1 φ(t(x -x λ )) λ -1/2 ν λ (x).
The last inequality is obtained using Eq. ( 5.18).

To show the second point, we use the fact that x ∈ H P to show that

λ x H -1 P,λ (x) ≤ λ x H -1 λ (x) ≤ λ x -x λ H λ (x) + λ x λ H -1 λ (x) .

5.G . PROJECTED PROBLEM 231

Then applying Eq. ( 5.17) and Eq. ( 5.18):

λ x H -1 P,λ (x) ≤ 1 φ(t(x -x λ ))
ν λ (x) + e t(x-x λ )/2 s λ .

We then use the previous results to conclude.

5.G .3 Finding a good projector

Lemma 5.16. If for a certain η ≤ λ and for a certain constant C, H

1/2 η (x)(I -P) 2 ≤ Cη, then C P (x, λ) ≤ Cη λ .
Proof. This is completely direct, using the fact that H 1/2 (x) H

1/2 η (x).

5.H Relations between statistical problems and empirical problem.

In this section, we recall and reformulate the framework from Marteau-Ferey, Ostrovskii, Bach, and Rudi (2019).

5.H .1 Statistical problem and ERM estimator

Let Z be a Polish space and Z be a random variable on Z with distribution ρ. Let H be a separable Hilbert space, with norm • , and let (f z ) z∈Z be a family of functions on H. Our goal is to minimize the expected risk with respect to x ∈ H:

inf x∈H f (x) := E [f Z (x)] .
Given (z i ) n i=1 ∈ Z n , we define the empirical risk :

f (x) := 1 n n i=1 f z i (x),
and consider the following estimator based on regularized empirical risk minimization given λ > 0 (note that the minimizer is unique in this case):

x λ = arg min x∈H f λ (x) := f (x) + λ 2 x 2 ,
where we assume the following.

Assumption 5.6 (i.i.d. data). The samples (z i ) 1≤i≤n are independently and identically distributed according to ρ.

We make the following assumption on the family (f z ) (this is a reformulation of Assumption 8 in Marteau-Ferey, Ostrovskii, Bach, and Rudi (2019))

Assumption 5.7 (Generalized self-concordance). For any z ∈ Z, there exists an associated subset G z ⊂ H such that (f z , G z ) is generalized self-concordant in the sense of definition 5.3.

Moreover we require the following technical assumption to guarantee that f and and its derivatives are well defined for any x ∈ H (this is a reformulation of Assumptions 3 and 4 in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], and the necessary conditions to obtain proposition 5.3).

Assumption 5.8 (Technical assumptions). The mapping

(z, x) ∈ Z × H → f z (x) is measurable. Moreover,
• the random variables f Z (0) , ∇f Z (0) , Tr(∇ 2 f Z (0)) are are bounded;

• G := z∈supp(Z) G z is a bounded subset of H.
The assumptions above are usually easy to check in practice. In particular, if the support of ρ is bounded, the mappings z → z (0), ∇ z (0), Tr(∇ 2 z (0)) are continuous, and z → G z is uniformly bounded on bounded sets, then they hold. Proposition 5.16. Under Assumptions 5.7 and 5.8, the function (f, G) (or simply f ) is generalized self-concordant. Moreover, under Assumption 5.6, define

G := n i=1 G z i . Then ( f , G) (or simply f ) is generalized self-concordant. Moreover, note that G ⊂ G.
The main regularity assumption we make on our statistical problems follows (see Assumption 5 in Marteau-Ferey, Ostrovskii, Bach, and Rudi (2019)).

Assumption 5.9 (Existence of a minimizer). There exists x ∈ H such that f (x ) = inf x∈H f (x).

Notations We adopt all the notations from Sec. 5.A for f and f , which are generalized self-concordant functions with associated subsets given in proposition 5.16 with the following conventions:

• For all quantities relating to f , we omit the subscript f as usual;

• For all quantities relating to f , we omit the subscript f and instead put a hat over all these quantities. For instance:

H(x) := H f (x) = 1 n n i=1 ∇ 2 f z i (x), r λ (x) := r f ,λ (x) = 1 sup g∈ G g H -1 λ (x)
, etc... Recall the two main quantities introduced in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] to establish the quality of our estimator x λ (in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], this is a mix between Proposition 2 and Definition 3). Proposition 5.17 (Bias, degrees of freedom). Suppose Assumptions 5.7 to 5.9 are satisfied. The following key quantities are well defined:

• the bias b λ = H λ (x ) -1/2 ∇f λ (x ) ; • the effective dimension df λ = E H λ (x ) -1/2 ∇f Z (x ) 2 .
Moreover, we also introduce the following quantities:

B 1 := sup z∈supp(Z) ∇f z (x ) , B 2 := sup z∈supp(Z) Tr(∇ 2 f z (x )), Q = B 1 B 2 .
We can now recall the main theorem of Marteau-Ferey, Ostrovskii, Bach, and Rudi (2019) (Theorem 4), which quantifies the behavior of the ERM estimator:

Theorem 5.8 (Bound for the ERM estimator).

Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Whenever n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , 2 df λ ∨ (Q ) 2 n log 2 δ ≤ r λ (x ), 2b λ ≤ r λ (x ),
then with probability at least 1 -2δ, it holds

f ( x λ ) -f (x ) ≤ C bias b 2 λ + C var df λ ∨ (Q ) 2 n log 2 δ , (5.37)
where C bias , C var , 1 ≤ 414, 1 , 2 ≤ 5184.

5.H .2 Link between a good approximation of x λ and x

In this paper, we provide an algorithm which can effectively compute a good approximation of x λ (as it is a finite sum problem which can be solved). This algorithm will return a certain x ∈ H, whose precision with respect to the empirical problem will be characterized by ν λ (x). The aim of the following lemma is to see how this approximation x behaves with respect to the statistical problem.

Lemma 5.17. Suppose the conditions for Theorem 5.8 are satisfied, i.e. let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 and suppose

n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , 2 df λ ∨ (Q ) 2 n log 2 δ ≤ r λ (x ), 2b λ ≤ r λ (x ).
Let x be an approximation of x λ characterized by its Newton decrement ν λ (x). If

ν λ (x) ≤ r λ (x) 2 , ν λ (x) ≤ r λ (x ) 2 ,
then with probability at least 1 -2δ, it holds

f (x) -f (x ) ≤ 14(f ( x λ ) -f (x )) + 30 ν λ (x) 2 .
Proof. Using Eq. ( 5.16),

f (x) -f ( x λ ) ≤ ∇f ( x λ ), x -x λ H + ψ(t(x -x λ )) x -x λ 2 H λ ( x λ ) ≤ 1 2 ∇f ( x λ ) 2 H -1 λ ( x λ ) + ψ(t(x -x λ )) + 1 2 x -x λ 2 H λ ( x λ ) . 1. Let us bound ∇f ( x λ ) H -1 λ ( x λ ) ∇f ( x λ ) H -1 λ ( x λ ) ≤ 1 0 H -1/2 λ ( x λ )H(x t )( x λ -x ) dt, x t = (1 -t) x λ + tx ≤ 1 0 H -1/2 λ ( x λ )H 1/2 (x t ) H 1/2 (x t )( x λ -x ) dt.
Now using equation Eq. ( 5.14)

H(x t ) e tt( x λ -x ) H( x λ ), H(x t ) e (1-t)t( x λ -x ) .
Thus:

∇f ( x λ ) H -1 λ ( x λ ) ≤ e t( x λ -x )/2 x λ -x H(x ) .
Finally, using equation Eq. ( 5.16)

∇f ( x λ ) H -1 λ ( x λ ) ≤ e t( x λ -x )/2 ψ(-t( x λ -x )) 1/2 (f ( x λ ) -f (x )) 1/2 .
2. Let us bound the terms involving xx λ H λ ( x λ ) Note that using Eq. ( 5.18) and Eq. ( 5.17) applied to f ,

x -x λ H λ ( x λ ) ≤ H 1/2 λ ( x λ ) H -1/2 λ ( x λ ) e t(x-x λ )/2 φ( t(x -x λ )) ν λ (x).
This also leads to:

t(x -x λ ) ≤ 1 r λ ( x λ ) H 1/2 λ ( x λ ) H -1/2 λ ( x λ ) x -x λ H λ ( x λ ) ≤ 1 r λ ( x λ ) H 1/2 λ ( x λ ) H -1/2 λ ( x λ ) e t(x-x λ )/2 φ( t(x -x λ ))
ν λ (x).

Putting things together

In the end, we get

f (x) -f (x ) ≤ 1 + e t( x λ -x ) ψ(-t( x λ -x )) (f ( x λ ) -f (x )) + ψ(t(x -x λ )) + 1 2 e t( x λ -x λ )/2 H 1/2 λ (x λ ) H -1/2 λ (x λ ) e t(x-x λ )/2 φ( t(x -x λ )) ν λ (x) 2 .
Moreover, we bound

t(x -x λ ) ≤ e (t(x -x λ )+t( x λ -x λ ))/2 H 1/2 λ (x λ ) H -1/2 λ (x λ ) e t(x-x λ )/2 φ( t(x -x λ )) ν λ (x) r λ (x )
.

Plugging in previous results

Under the assumptions of this lemma, which include the assumptions of Theorem 4. in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], we get the following bounds.

• In Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF],the assumptions of Theorem 4 imply that we can use Lemma 9, which uses Lemma 8 in which we show that with probability at least 1 -δ,

H -1/2 λ (x λ )H λ (x λ ) 1/2 2 ≤ 2.
• Still using the assumptions of Theorem 4, we see in the proof of this theorem that the assumptions of Theorem 7 of Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] are satisfied in the case where b λ ≤ r λ (x ) 2 , and thus that

t( x λ -x λ ) ≤ log 2, t(x λ -x ) ≤ log 2.
Plugging in all these bounds, we get

1 + e t( x λ -x ) ψ(-t( x λ -x )) ≤ 14, t(x -x λ ) ≤ 6, ψ(t(x -x λ )) + 1 2 e t( x λ -x λ )/2 H 1/2 λ (x λ ) H -1/2 λ (x λ ) e t(x-x λ )/2 φ( t(x -x λ ))
≤ 30.

5.H .3 Bounds when we solve a projected empirical problem

In this section, we place ourselves in the setting of Sec. 5.G . In this section, we had argued that for computational purposes, it was less costly to compute an approximate solution to a projected problem.

In this section, we assume that we are going to project the regularized empirical problem, that is solve approximately

x ≈ arg min x∈H f P,λ (x) = f (Px) + λ 2 x 2 .
for a given orthogonal projection P. Recall from Sec. 5.G that there is a natural way of seeing f P as a generalized self-concordant function. We import all the notations from this section, keeping a • over all notations to mark the fact that we are projecting f and not f .

To quantify the quality of the approximation x, we will use the Newton decrement for the empirical projected problem ν P,λ (x) := ν f P ,λ (x).

As we see in proposition 5.15, under certain conditions, bounding ν λ (x) amounts to bounding two terms:

• The empirical source s λ := λ x λ H -1 λ ( x λ ) , • The projected empirical Newton decrement ν P,λ (x).

1. Bounding the empirical source term s λ Start by bounding the source empirical source term using quantities we know.

Lemma 5.18 (Empirical source). Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Whenever n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , 2 df λ ∨ (Q ) 2 n log 2 δ ≤ r λ (x ), 2b λ ≤ r λ (x ).
The following holds, with probability at least 1 -2δ.

s λ ≤ 8 b λ + 80 df λ ∨ (Q ) 2 log 2 δ n .
Moreover, we also have the following bound :

x λ -x ≤ 3 λ -1/2 b λ + 8 λ -1/2 df λ ∨ (Q ) 2 log 2 δ n .
Proof. We first decompose the source term into two terms, and then apply different bounds from Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] to effectively bound it. We will use the following quantity:

Var λ := H 1/2 λ (x λ ) H -1/2 λ (x λ ) 2 ∇ f λ (x λ ) H -1 λ (x λ ) .
It is also defined in equation ( 23) in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF].

1. Dividing s λ into two controllable terms . Decompose

s λ = λ x λ H -1 λ ( x λ ) ≤ H -1/2 λ ( x λ )H 1/2 λ ( x λ ) λ x λ H -1 λ ( x λ ) ≤ H -1/2 λ ( x λ )H 1/2 λ ( x λ ) ∇f λ ( x λ ) H -1 λ ( x λ ) + ∇f ( x λ ) H -1 λ ( x λ ) .
On the one hand, from the previous proof, we get

∇f ( x λ ) H -1 λ ( x λ ) ≤ e t( x λ -x )/2 x λ -x H(x ) ≤ e t( x λ -x )/2 e t(x λ -x ) x λ -x λ H λ (x λ ) + x λ -x H λ (x ) ≤ e t( x λ -x )/2 e t(x λ -x ) φ(t( x λ -x λ )) Var λ + 1 φ(t(x λ -x )) b λ .
In the last line, we use the fact that

x λ -x λ H λ (x λ ) ≤ H 1/2 λ (x λ ) H -1/2 λ (x λ ) x λ -x λ H λ (x λ )
and then bound it using Eq. ( 5.18) applied to f to get

x λ -x λ H λ (x λ ) ≤ 1 φ( t(x λ -x λ )) ∇ f λ (x λ ) H -1 λ (x λ ) ≤ 1 φ(t(x λ -x λ )) H 1/2 λ (x λ ) H -1/2 λ (x λ ) ∇ f λ (x λ ) H -1 λ (x λ ) .
On the other hand, apply successively Eq. ( 5.18) to f and f using the fact that t ≤ t to get

∇f λ ( x λ ) H -1 λ ( x λ ) = ∇f λ ( x λ ) -∇f λ (x λ ) H -1 λ ( x λ ) ≤ e t( x λ -x λ )/2 φ(t( x λ -x λ )) x λ -x λ H λ (x λ ) ≤ e t( x λ -x λ )/2 φ(t( x λ -x λ )) H 1/2 λ (x λ ) H -1/2 λ (x λ ) x λ -x λ H λ (x λ ) ≤ e t( x λ -x λ )/2 φ(t( x λ -x λ )) φ(t( x λ -x λ )) H 1/2 λ (x λ ) H -1/2 λ (x λ ) 2 ∇ f λ (x λ ) H λ (x λ ) = e 3t( x λ -x λ )/2 Var λ .
Putting things together:

s λ ≤ H -1/2 λ ( x λ )H 1/2 λ ( x λ ) e 3t(x λ -x λ )/2 1 + 1 φ(t(x λ -x λ )) Var λ + e t(x λ -x λ )/2 φ(t(x λ -x )) b λ .

238

CHAPTER 5. SECOND ORDER STRIKES BACK 2. We now import the results from Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] . The following hypotheses imply those of Thms 4 and 7 in Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]:

Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Whenever n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , n ≥ 2 df λ ∨ (Q ) 2 r λ (x ) 2 log 2 δ , b λ ≤ r λ (x ) 2 .
In particular, they imply that with probability at least 1 -2δ:

• Var λ ≤ 1 2 b λ + 4 1 df λ ∨(Q ) 2 log 2 δ n ; • H 1/2 λ (x λ ) H -1/2 λ (x λ ) ≤ √ 2;
• t(xx λ ) ≤ log 2;

• t( x λ -x λ ) ≤ log 2.
Hence, plugging these bounds in the previous equation, we get

s λ ≤ 8b λ + 80 df λ ∨ (Q ) 2 log 2 δ n .
3. Note that in what has been done previously, we can bound:

x λ -x λ H λ (x λ ) ≤ 1 φ(t(x λ -x λ )) Var λ ≤ b λ + 8 df λ ∨ (Q ) 2 log 2 δ n .
Moreover,

x λ -x H λ (x ) ≤ 1 φ(t(x λ -x )) ∇f λ (x ) H -1 λ (x ) ≤ 2b λ .
Hence:

x λ -x ≤ 3 λ -1/2 b λ + 8 λ -1/2 df λ ∨ (Q ) 2 log 2 δ n .
2. Final bound for the projected ERM approximation In this paragraph, denote with C P (x, λ) the quantity H 1/2 (x)(I-P) 2 λ and C P (x, λ) the quantity

H 1/2 (x)(I-P) 2 λ Lemma 5.19. Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Whenever n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , C 1 df λ ∨ (Q ) 2 n log 2 δ ≤ r λ (x ), C 1 b λ ≤ r λ (x ), if C P (x , λ) ≤ √ 2 480 , ν P,λ (x) ≤ r P,λ (x) 2 ∧ r λ (x ) 126 ,
the following holds, with probability at least 1 -2δ.

ν λ (x) ≤ r λ (x) 2 , ν λ (x) ≤ r λ (x ) 2 .
Here, C 1 = 1008.

Proof. Proceed in the following way.

1.

It is easy to see that the conditions of this lemma imply the conditions of Theorem 5.8. Hence, as in the previous proofs, the following hold:

• H 1/2 λ (x λ ) H -1/2 λ (x λ ) ≤ √ 2;
• t(xx λ ) ≤ log 2;

• t( x λ -x λ ) ≤ log 2.

2.

Let us now apply proposition 5.15 to f . If

s λ r λ ( x λ ) ≤ 1 4 , C P ( x λ , λ) ≤ 1 120 , ν P,λ (x) ≤ r P,λ (x) 2 ,
Then it holds:

ν λ (x) ≤ 3( ν P,λ (x) + s λ ), t(x -x λ ) ≤ 3 10 + 2 log 2. (5.38)
where the second bound is obtained in the proof of this proposition. Now since 1 r λ ( x λ ) ≤ e t( x λ -x λ )/2 1 r λ (x λ ) Eq. (5.17)

≤ e t( x λ -x λ )/2 H 1/2 λ (x λ ) H -1/2 λ (x λ ) sup g∈ G g H -1 λ (x λ ) Def ≤ e t( x λ -x λ )/2 H 1/2 λ (x λ ) H -1/2 λ (x λ ) sup g∈G g H -1 λ (x λ ) G ⊂ G = e t( x λ -x λ )/2 H 1/2 λ (x λ ) H -1/2 λ (x λ ) 1 r λ (x λ ) Def ≤ e (t( x λ -x λ )+t(x λ -x ))/2 H 1/2 λ (x λ ) H -1/2 λ (x λ ) 1 r λ (x )
Eq. (5.17)

≤ 2 √ 2 r λ (x )
. previous bounds

In a similar way, we get C P ( x λ , λ) ≤ 2 √ 2C P (x , λ). Thus, the conditions above are satisfied if the following conditions are satisfied:

s λ r λ (x ) ≤ √ 2 16 , C P (x , λ) ≤ √ 2 480 , ν P,λ (x) ≤ r P,λ (x) 2 .
Finally, note that under these conditions,

1 r λ (x) ≤ e t(x-x λ )/2 r λ (x) ≤ 7 r λ (x )
.

(5.39) using the previous bound and the bound on t(xx λ ).

CHAPTER 5. SECOND ORDER STRIKES BACK

Let us assume

s λ r λ (x ) ≤ √ 2 16 , C P (x , λ) ≤ √ 2 480
, ν P,λ (x) ≤ r P,λ (x) 2 .

According to Eq. ( 5.39), and to Eq. ( 5.38), if

ν P,λ (x) + s λ ≤ r λ (x ) 42 , then it holds ν λ (x) ≤ r λ (x) 2 , ν λ (x) ≤ r λ (x ) 2 .
We simplify this condition as:

ν P,λ (x) ≤ r λ (x ) 126 , s λ ≤ 2r λ (x ) 126 .
4. Now using the fact that under the conditions of this lemma, those of Lemma 5.18 are satisfied:

s λ ≤ 8 b λ + 80 df λ ∨ (Q ) 2 log 2 δ n .
Thus,

s λ ≤ 2r λ (x ) 126 holds, provided b λ ≤ r λ (x ) C 1 , n ≥ C 2 1 df λ ∨ (Q ) 2 log 2 δ r λ (x ) 2 ,
where C 1 = 1008.

Proposition 5.18 (Behavior of an approximation to the projected problem). Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Let x ∈ H P . Whenever

n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , C 1 df λ ∨ (Q ) 2 n log 2 δ ≤ r λ (x ), C 1 b λ ≤ r λ (x ), if C P (x , λ) ≤ √ 2 480 , ν P,λ (x) ≤ r P,λ (x) 2 ∧ r λ (x ) 126 .
The following holds, with probability at least 1 -2δ.

f (x) -f (x ) ≤ K 1 b 2 λ + K 2 df λ ∨ (Q ) 2 n log 2 δ + K 3 ν 2 P,λ (x),
where K 1 ≤ 6.0e4, K 2 ≤ 6.0e6 and K 3 ≤ 810, C 1 are defined in Lemma 5.19, and the other constants are defined in Theorem 5.8.

Remark 14 (Constants). In this result, absolutely huge constants are obtained. They are (of course) totally sub-optimal. Indeed, this analysis has been simplified by dividing the bound into blocks: error of the empirical risk minimization with regularization, error of the projection compared to this empirical risk minimizer. Going back and forth from empirical to statistical, from projected to non projected induces exponential explosion of the constants. There is a way of doing the analysis directly by projecting the statistical problem. However, in order to relate to our previous work Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] and avoid re-doing all of our work we discarded this. If we were to perform this more direct analysis, we could keep the constants to a reasonable level, of order 10 2 .

Proof. We apply Lemma 5.17, using the previous lemma to guarantee the conditions.

1. Under the conditions of this proposition, applying Lemma 5.19, the conditions of Lemma 5.17 are satisfied. Thus,

f (x) -f (x ) ≤ 14(f ( x λ ) -f (x )) + 30 ν λ (x) 2 .
Moreover, from the previous proof,

ν λ (x) ≤ 3( ν P,λ (x) + s λ ),
and seeing as Lemma 5.18 is satisfied,

s λ ≤ 8 b λ + 80 df λ ∨ (Q ) 2 log 2 δ n .
This therefore yields:

ν λ (x) 2 ≤ 27 ν P,λ (x) 2 + 1726b 2 λ + 172600 df λ ∨ (Q ) 2 log 2 δ n .
2. Moreover, from Theorem 5.8, it holds:

f ( x λ ) -f (x ) ≤ 414 b 2 λ + 414 df λ ∨ (Q ) 2 n log 2 δ .

3.

Putting things together:

f (x) -f (x ) ≤ K 1 b 2 λ + K 2 df λ ∨ (Q ) 2 n log 2 δ + K 3 ν 2 P,λ (x).
We bound the constants in the theorem.

Lemma 5.20. Under the conditions of the previous theorem, the following hold:

• 1 r P,λ (x) ≤ 8 r λ (x ) ; • λ 1/2 x -x ≤ 7 ν P,λ (x) + 59b λ + 568 df λ ∨(Q ) 2 log 2 δ n ; • λ x H -1 P,λ (x) ≤ 7 ν P,λ (x) + 72b λ + 720 df λ ∨(Q ) 2 log 2 δ n .
In particular,

λ x H -1 P,λ (x)
r P,λ (x)

≤ 11.

Proof. Let us prove the three statements.

1. Write 1 r P,λ (x) = sup g∈ G Pg H -1 P,λ (x) . Now sup g∈ G Pg H -1 P,λ (x) ≤ sup g∈ G g H -1 λ (x) ≤ e t(x-x λ )/2 sup g∈ G g H -1 λ ( x λ ) . Now bound sup g∈ G g H -1 λ ( x λ ) ≤ e t(x λ -x λ )/2 sup g∈ G g H -1 λ (x λ ) ≤ e t(x λ -x λ )/2 H 1/2 λ (x λ ) H -1/2 λ (x λ ) sup g∈ G g H -1 λ (x λ ) .
Finally bound

sup g∈ G g H -1 λ (x λ ) ≤ e t(x -x λ )/2 1 r λ (x ) .
Now using the fact that under the previous assumptions t

(x -x λ ), t(x λ -x λ ) ≤ log 2, t(x -x λ ) ≤ 3 10 + 2 log 2 and H 1/2 λ (x λ ) H -1/2 λ (x λ ) ≤ √ 2
, we get the first equation.

2.

In order to bound λ 1/2 xx , decompose

λ 1/2 x -x ≤ λ 1/2 x -x λ + λ 1/2 x λ -x .
Now use proposition 5.15 to bound λ 1/2 xx λ ≤ 7( ν P,λ (x) + s λ ). Using Lemma 5.18, under the conditions above,

s λ ≤ 8 b λ + 80 df λ ∨ (Q ) 2 log 2 δ n .
Hence

λ 1/2 x -x λ ≤ 7 ν P,λ (x) + 56b λ + 560 df λ ∨ (Q ) 2 log 2 δ n .
Moreover, using again Lemma 5.18

λ 1/2 x λ -x ≤ 3 b λ + 8 df λ ∨ (Q ) 2 log 2 δ n .
Combining these two inequalities, we get:

λ 1/2 x -x ≤ 7 ν P,λ (x) + 59b λ + 568 df λ ∨ (Q ) 2 log 2 δ n .

3.

In order to bound λ x H -1 P,λ (x) , use proposition 5.15 to get λ x H -1 P,λ (x) ≤ 7 ν P,λ (x) + 9 s λ . Now using Lemma 5.18, the following bound holds:

λ x H -1 P,λ (x) ≤ 7 ν P,λ (x) + 72b λ + 720 df λ ∨ (Q ) 2 log 2 δ n . Proposition 5.19 (Simplification). Let n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 . Let x ∈ H P . Whenever n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , C 1 df λ ∨ (Q ) 2 n log 2 δ ≤ √ λ R , C 1 b λ ≤ √ λ R , if C P (x , λ) ≤ √ 2 480 , ν P,λ (x) ≤ √ λ 126R ,
then the following holds, with probability at least 1 -2δ.

f (x) -f (x ) ≤ K 1 b 2 λ + K 2 df λ ∨ (Q ) 2 n log 2 δ + K 3 ν 2 P,λ (x),
where K 1 ≤ 6.0e4, K 2 ≤ 6.0e6 and K 3 ≤ 810, C 1 are defined in Lemma 5.19, and the other constants are defined in Theorem 5.8.

Moreover, in that case, R xx ≤ 10.

5.H .4 Optimal choice of λ, specific source conditions

In this part, we continue to assume Assumptions 5.6 to 5.9. We present a classification of distributions ρ and show that we can achieve better rates than the classical slow rates, as presented in Appendix F of Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF].

Classification of distributions and statistical bounds for the ERM

We use the following classification for distributions.

Definition 5.9 (class of distributions). Let α ∈ [1, +∞] and r ∈ [0, 1/2].

We denote with P α,r the set of probability distributions ρ such that there exists L, Q ≥ 0,

• b λ ≤ L λ 1+2r 2 ; • df λ ≤ Q 2 λ -1/α ;
where this holds for any 0 < λ ≤ 1. For simplicity, if α = +∞, we assume that Q ≥ Q .

Note that given our assumptions, we always have

ρ ∈ P 1,0 , L = x , Q = B 1 .
(5.40)

We also define

λ 1 = Q Q 2α ∧ 1, (5.41) such that ∀λ ≤ λ 1 , df λ ∨ (Q ) 2 ≤ Q 2 λ 1/α .

Interpretation of the classes

• The bias term b λ characterizes the regularity of the objective x . In a sense, if r is big, then this means x is very regular and will be easier to estimate. The following results reformulates this intuition.

Remark 15 (source condition). Assume there exists 0 ≤ r ≤ 1/2 and v ∈ H such that

P H(x ) x = H(x ) r v.
Then it holds:

∀λ > 0, b λ ≤ L λ 1+2r 2 , L = H(x ) -r x .
• The effective dimension df λ characterizes the size of the space H with respect to the problem. The higher α, the smaller the space. If H is finite dimensional for instance, α = +∞.

In this section, for any given pair (α, r) characterizing the regularity and size of the problem, we associate

β = 1 1 + 2r + 1/α , γ = α(1 + 2r) α(1 + 2r) + 1 .
In Marteau-Ferey, Ostrovskii, Bach, and Rudi (2019) (see corollary 3), explicit bounds are given for the performance of the regularized expected risk minimizer x λ depending on which class ρ belongs to, i.e., as a function of α, r.

Corollary 5.4. Let δ ∈ (0, 1/2]. Under Assumptions 5.6 to 5.9, if ρ ∈ P α,r with r > 0 , when n ≥ N and λ = (C 0 /n) β , then with probability at least 1 -2δ, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF], and satisfying

f ( x λ ) -f (x ) ≤ C 1 n -γ log 2 δ , with C 0 = 256(Q/L) 2 , C 1 = 8(256) γ (Q γ L 1-γ ) 2 and N defined in Marteau-Ferey, Ostrovskii,
N = O(poly(B 1 , B 2 , L, Q, R, log(1/δ))).

Quantitative bounds for the projected problem

In this part, the aim is to show that if we approximately solve the projected problem up to a certain precision, then this approximation has the same statistical rates as the regularized ERM with the good choice of λ. For the sake of simplicity, we will assume that r > 0.

In what follows, we define

N = Q 2 L 2 (B 2 ∧ λ 0 ∧ λ 1 ) -1/β ∨ 2.1e4 1 1 -β A log 1.4e6 1 1 -β A 2 1 δ 1/(1-β) , (5.42)
where

A = B 2 L 2β Q 2β , λ 0 = (C 1 LR log 2 δ ) -1/r ∧ 1 and λ 1 = Q 2α (Q ) 2α .
Theorem 5.9 (Quantitative result with source r > 0). Let ρ ∈ P α,r and assume r > 0. Let δ ∈ (0, 1 2 ]. Let P be an orthogonal projection, x ∈ H. If

n ≥ N, λ = Q L 2 1 n β , C P (x , λ) ≤ √ 2 480 , ν P,λ (x) ≤ Q γ L 1-γ n -γ/2
then with probability at least 1 -2δ,

f (x) -f (x ) ≤ K Q γ L 1-γ 2 1 n γ log 2 δ ,
where N is defined in Eq. (5.42) and K ≤ 7.0e6. Moreover, R xx ≤ 10.

Proof. Using the definition of λ 1 , as soon as λ ≤ λ 1 ,it holds:

df λ ∨ (Q ) 2 ≤ Q 2 λ -1/α .
Let us formulate proposition 5.19 using the fact that ρ ∈ P α,r .

Let

n ∈ N, δ ∈ (0, 1/2], 0 < λ ≤ B 2 , x ∈ H P . Whenever n ≥ 1 B 2 λ log 8 2 1 B 2 λδ , C 1 Q 2 λ 1/α n log 2 δ ≤ λ 1/2 R , C 1 Lλ 1/2+r ≤ λ 1/2 R , if C P (x , λ) ≤ √ 2 480 , ν P,λ (x) ≤ Lλ 1/2+r ,
The following holds, with probability at least 1 -2δ.

f (x) -f (x ) ≤ (K 1 + K 3 )L 2 λ 1+2r + K 2 Q 2 λ 1/α n log 2 δ , R x -x ≤ 10,
where all constants are defined in proposition 5.19.

Assume that r > 0 . Define

λ 0 = (C 1 LR log 2 δ ) -1/r ∧ 1.
Then for any λ ≤ λ 0 :

Lλ 1/2+r ≤ 1 C 1 √ λ R .
1) First, we find a simple condition to guarantee

r λ (x ) 2 λ 1/α ≥ C 2 Q 2 1 n log 2 δ .
We see that if λ ≤ λ 0 , then r λ ≥ C 1 Lλ 1/2+r log 2 δ . Hence, this condition is satisfied if

λ ≤ λ 0 , C 2 1 L 2 λ 1+2r+1/α ≥ C 2 Q 2 1 n .

CHAPTER 5. SECOND ORDER STRIKES BACK

Using the fact that C 2 = C 2 1 , we reformulate:

λ ≤ λ 0 , L 2 λ 1+2r+1/α ≥ Q 2 1 n .
2) Now fix

λ 1+2r+1/α = Q 2 L 2 1 n ⇐⇒ λ = Q 2 L 2 1 n β .
where β = 1/(1 + 2r + 1/λ) ∈ [1/2, 1).

Using our restatement of proposition 5.18, with probability at least 1 -2δ,

L(x) -L(x ) ≤ K 1 + K 3 + K 2 log 2 δ L 2 λ 1+2r ≤ K log 2 δ L 2 λ 1+2r ,
where K = K 1 + K 3 + K 2 ≤ 7.0e6 (see proposition 5.18). This result holds provided

0 < λ ≤ B 2 ∧ λ 0 ∧ λ 1 , n ≥ 1 B 2 λ log 8 2 1 B 2 λδ .
(5.43) Indeed, it is shown in the previous point that the other conditions are satisfied.

3) Let us now work to guarantee the conditions in Eq. ( 5.43). First, to guarantee n ≥ 1

B 2 λ log 8 2 1 B 2 λδ , bound B 2 λ = B 2 L 2β n β Q 2β log β 2 δ ≤ 2 B 2 L 2β Q 2β n β .
Then apply lemma 15 from Marteau-Ferey, [START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF] with a 1 = 2 1 ,

a 2 = 16 2 1 , A = B 2 L 2β Q 2β .
Since β ≥ 1/2, using the bounds in Theorem 5.8, we find a 1 ≤ 10400 and a 2 ≤ 64, hence the following sufficient condition:

n ≥ 2.1e4 1 1 -β A log 1.4e6 1 1 -β A 2 1 δ 1/(1-β)
.

Then, to guarantee the condition

λ ≤ B 2 ∧ λ 0 ∧ λ 1 , we simply need n ≥ Q 2 L 2 (B 2 ∧ λ 0 ∧ λ 1 ) -1/β . Hence, defining N = Q 2 L 2 (B 2 ∧ λ 0 ∧ λ 1 ) -1/β ∨ 2.1e4 1 1 -β A log 1.4e6 1 1 -β A 2 1 δ 1/(1-β)
, we see that as soon as n ≥ N , Eq. ( 5.43) holds.

5.I . BOUNDS FOR HERMITIAN OPERATORS

247

5.I Multiplicative approximations for Hermitian operators

In this section, we put together useful tools for approximating linear operators and solving linear systems with regularization.

In this section, A and B will always denote positive semi-definite Hermitian operators on a Hilbert space H, and P will denote an orthogonal projection operator. Moreover, given a positive semi-definite operator A, and λ > 0, A λ will stand for the regularized operator A + λI.

Lemma 5.21 (Equivalence of Hermitian operators). Let A and B be two semi-definite Hermitian operators. Let λ > 0. Assume you have access to

t := A -1/2 λ (B -A)A -1/2 λ . It holds: A -1/2 λ B 1/2 λ 2 ≤ 1 + t ⇔ B λ (1 + t)A λ . Moreover, if t < 1, B -1/2 λ A 1/2 λ 2 ≤ 1 1 -t ⇔ (1 -t)A λ B λ .
Proof. For the first point, simply note that:

A -1/2 λ B 1/2 λ 2 = A -1/2 λ B λ A -1/2 λ = I + A -1/2 λ (B -A) A -1/2 λ ≤ 1 + t.
For the second point,

B -1/2 λ A 1/2 λ 2 = A -1/2 λ B λ A -1/2 λ -1 = I + A -1/2 λ (B -A) A -1/2 λ -1
.

Moreover, we know that if H < 1 with H a Hermitian operator, then (I + H) -1 ≤ 1 1-H . The result follows.

We will now state a technical lemma which describes how combining approximation behaves.

Lemma 5.22 (Combination of approximations). Let N ≥ 1. Let (A i ) 1≤i≤N +1 be a sequence of positive semi-definite Hermitian operators. Define

t i := A -1/2 i,λ (A i+1 -A i )A -1/2 i,λ .
For any 1 ≤ i, j ≤ N + 1, define

t i:j := A -1/2 i,λ (A j -A i )A -1/2 i,λ .
In particular, t i = t i:i+1 . Then the following holds:

∀1 ≤ i ≤ j ≤ N, 1 + t i:j ≤ j-1 k=i (1 + t k ) Moreover, if t i < 1, then it holds: A -1/2 i+1,λ (A i -A i+1 )A -1/2 i+1,λ ≤ t i 1 -t i Hence, in that case ∀1 ≤ j ≤ i ≤ N, 1 + t j:i ≤ j-1 k=i 1 1 -t k 248 CHAPTER 5. SECOND ORDER STRIKES BACK
Proof. Let us prove everything for a sequence of three operators; the rest follows by induction. Let A 1 , A 2 , A 3 be three positive semi-definite operators.

1.

Bound

t 1:3 = A -1/2 1,λ (A 1 -A 3 ) A -1/2 1,λ ≤ A -1/2 1,λ (A 1 -A 2 ) A -1/2 1,λ + A -1/2 1,λ (A 2 -A 3 ) A -1/2 1,λ ≤ t 1:2 + A -1/2 1,λ A 1/2 2,λ 2 t 2:3 ≤ t 1:2 + (1 + t 1:2 )t 2:3 .
The last line comes from Lemma 5.21. Thus 1 + t 1:3 ≤ 1 + t 1:2 + t 2:3 + t 1:2 t 2:3 = (1 + t 1:2 )(1 + t 2:3 ).

2.

Let us now bound t 2:1 knowing t 1:2 . This will imply the rest of the lemma. Indeed, simply note that

t 2:1 = A -1/2 2,λ (A 2 -A 1 )A -1/2 2,λ ≤ A -1/2 2,λ A 1/2 1,λ 2 t 1:2 . Using Lemma 5.21, if t 1:2 < 1, A -1/2 2,λ A 1/2 1,λ 2 ≤ 1 1-t 1:2 , hence t 2:1 ≤ t 1:2 1 -t 1:2 .
Lemma 5.23 (Projection of Hermitian operators). For any Hermitian operator A and orthogonal projection P, the following holds:

A -1/2 λ (A -PAP)A -1/2 λ ≤ 1 + A 1/2 (I -P) √ λ 2 -1.
In particular,

A -1/2 λ (PAP + λI) 1/2 ≤ 1 + A 1/2 (I -P) √ λ . Moreover, if A 1/2 (I -P) √ λ < √ 2 -1, then it holds A 1/2 λ (PAP + λI) -1/2 2 ≤ 1 2 -1 + A 1/2 (I-P) √ λ 2 .
We also always have:

(PAP + λI) -1/2 PA 1/2 λ 2 ≤ 1.

5.I . BOUNDS FOR HERMITIAN OPERATORS 249

Proof. For any Hermitian operator A, the following computation holds:

A -1/2 λ (A -PAP)A -1/2 λ = A -1/2 λ (A -(I -(I -P))A(I -(I -P))A -1/2 λ ≤ 2 A -1/2 λ (I -P)AA -1/2 λ + A -1/2 λ (I -P)A(I -P)A -1/2 λ ≤ 2 A 1/2 (I -P) √ λ + A 1/2 (I -P) 2 λ = 1 + A 1/2 (I -P) √ λ 2 -1.
Lemma 5.24 (Relationship between approximations). Let A and B be two positive semi-definite hermitian operators. Let λ > 0, b ∈ H and ρ > 0. If

A -1/2 λ (B -A)A -1/2 λ ≤ 1 2 ∧ ρ 4 , ∆ ∈ LinApprox(B λ , b, ρ/4),
then it holds:

∆ ∈ LinApprox(A λ , b, ρ).
Proof. Assume ∆ ∈ LinApprox(B λ , b, ρ/4) for a certain ρ. Decompose

A -1 λ b -∆ A λ ≤ A -1 λ b -B -1 λ b A λ + B -1 λ b -∆ A λ ≤ A 1/2 λ (A -1 λ -B -1 λ )A 1/2 λ b A -1 λ + A 1/2 λ B -1/2 λ B -1 λ b -∆ B λ .

Now using the fact that

A -1 λ -B -1 λ = B -1 λ (B -A)A -1 λ , A 1/2 λ (A -1 λ -B -1 λ )A 1/2 λ ≤ A -1/2 λ (B -A)A -1/2 λ A 1/2 λ B -1 λ A 1/2 λ = A -1/2 λ (B -A)A -1/2 λ A 1/2 λ B -1/2 λ 2 .
Moreover,

B -1 λ b -∆ B λ ≤ ρ b B -1 λ ≤ A 1/2 B -1/2 b A -1 λ .
Putting things together, and noting that from Lemma 5.21,

A 1/2 B -1/2 2 ≤ 1 1-A -1/2 λ (B-A)A -1/2 λ as soon as A -1/2 λ (B -A)A -1/2 λ < 1, it holds: ∆ ∈ LinApprox(A λ , b, ρ), ρ = ρ + A -1/2 λ (B -A)A -1/2 λ 1 -A -1/2 λ (B -A)A -1/2 λ .
Choosing the right values for ρ and

A -1/2 λ (B -A)A
-1/2 λ yields the result.

CHAPTER 5. SECOND ORDER STRIKES BACK

5.I .1 Results for Nystrom sub-sampling

Recall the notations from Sec. 5.D .

We write without proof the following lemmas, which are just restatements of lemmas 9 and 10 of [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF].

Lemma 5.25 (Uniform sampling). Let δ > 0. If {z 1 , ..., zm } are sampled uniformly, then if

0 < λ < A , m ≤ n and m ≥ 10 + 160N A ∞ (λ) log 8 v 2 L ∞ (Z)
λδ .

Then it holds, with probability at least 1 -δ:

A -1/2 λ ( A -A)A -1/2 λ ≤ 1 2 , A -1/2 m,λ ( A -A m ) A -1/2 m,λ ≤ 1 2 .
Lemma 5.26 (Nystrom sampling). Let δ > 0. If {z 1 , ..., zm } are sampled using q-approximate leverage scores for t = λ, then if

t 0 ∨ 19 v 2 L ∞ (Z) n log n 2δ < λ < A , and n ≥ 405 v 2 L ∞ (Z) ∨ 67 v 2 L ∞ (Z) log 12 v 2 L ∞ (Z) δ , if m ≥ 6 + 486q 2 N A (λ) log 8 v 2 L ∞ (Z)
λδ .

Then it holds, with probability at least 1 -δ:

A -1/2 λ ( A -A)A -1/2 λ ≤ 1 2 , A -1/2 m,λ ( A -A m ) A -1/2 m,λ ≤ 1 2 .
Lemma 5.27. Let λ > 0. Assume:

A -1/2 λ ( A -A)A -1/2 λ ≤ 1 2 , A -1/2 m,λ ( A -A m ) A -1/2 m,λ ≤ 1 2 .
Denote with P m the projection on span(v zj ) 1≤j≤m . Then the following holds:

A 1/2 λ (I -P m ) 2 ≤ 3λ,
and for any partial isometry V ,

1 2 V * A m V + λI V * AV + λI 3 2 V * A m V + λI .
Proof. For the first point, use the well known fact that

I -P m ≤ λ A -1 m,λ ,
since the range of P m contains that of A m . Thus,

A 1/2 λ (I -P m ) 2 ≤ λ A 1/2 λ A -1/2 m,λ 2 .
Now using Lemma 5.22,

A -1/2 λ ( A -A)A -1/2 λ ≤ 1 2 =⇒ A -1/2 λ ( A -A) A -1/2 λ ≤ 1.

5.I . BOUNDS FOR HERMITIAN OPERATORS

251

Hence, again using Lemma 5.22,

A -1/2 m,λ ( A m -A) A -1/2 m,λ
≤ 2, and therefore, using Lemma 5.21,

A 1/2 λ A -1/2 m,λ 2 ≤ 3.
For the second point, this is only a consequence of Lemma 5.21.

Now state two results which show that

Lemma 5.28 (Uniform sampling yielding ρ-approximation). Let 0 < ρ ≤ 1 and δ > 0. Let b ∈ H. If {z 1 , ..., zm } are sampled uniformly, 0 < λ < A , m ≤ n and

m ≥ 2 + 48 ρ + 5000 ρ 2 N A ∞ (λ) log 8 v 2 L ∞ (Z)
λδ .

Then it holds, with probability at least 1 -δ:

x ∈ LinApprox( A m,λ , b, ρ/4) =⇒ x ∈ LinApprox(A λ , b, ρ).

In particular, with probability 1 -δ,

A -1 m,λ b ∈ LinApprox(A λ , b, ρ).
Proof. Apply Lemma 9 from Rudi, Carratino, and Rosasco ( 2017) with η = ρ 12 < 1 2 . We find that under the conditions above, with probability at least 1 -δ,

A -1/2 λ ( A -A)A -1/2 λ ≤ η, A -1/2 m,λ ( A -A m ) A -1/2 m,λ ≤ η.

Now use Lemma 5.22 to see that

A -1/2 λ ( A m -A)A -1/2 λ ≤ (1 + η 2 ) -1 ≤ 3η ≤ ρ/4.
Thus, we can apply Lemma 5.24 to get the desired result.

Lemma 5.29 (Leverage scores Nystrom sampling yielding ρ-approximation). Let δ > 0. If {z 1 , ..., zm } are sampled using q-approximate leverage scores for t = λ, then if

t 0 ∨ 19 v 2 L ∞ (Z) n log n 2δ < λ < A , and n ≥ 405 v 2 L ∞ (Z) ∨ 67 v 2 L ∞ (Z) log 12 v 2 L ∞ (Z) δ , if m ≥ 2 + 24 ρ + 13000q 2 ρ 2 N A (λ) log 8 v 2 L ∞ (Z)
λδ .

Then it holds, with probability at least 1 -δ:

x ∈ LinApprox( A m,λ , b, ρ/4) =⇒ x ∈ LinApprox(A λ , b, ρ).

In particular, with probability 1 -δ,

A -1 m,λ b ∈ LinApprox(A λ , b, ρ).
Proof. The proof is exactly the same as that of the previous lemma, using Lemma 10 instead of Lemma 9 in [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF].

Introduction : representing non-negative functions in a flexible way 6 PSD models 7 Sampling from arbitrary functions via PSD models

Introduction : representing non-negative functions in a flexible way

This part is based on the two articles by Marteau-Ferey, Bach, andRudi (2020, 2022a). In this part, we develop a model for non-negative functions based on reproducing kernel Hilbert spaces, and apply this model to sample from probability distributions given their un-normalized density function, breaking the curse of dimensionality with regularity. This introduction will be less detailed than the one in the two other parts of thes thesis for two reasons. The first is that, as we introduce a new model for non-negative functions, there is less backround on this topic.

The second is that these articles are short and easily readable without context. We therefore do not go too deep into the related works and results of this part to avoid repetitions, and instead present a high level overview. In particular, we freely include elements from the introductions of chapters 6 and 7 and Sec. 2.3.2

In chapter 6, we describe the work by Marteau-Ferey, [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. We consider a class of models with non-negative outputs, as well as outputs in an affine convex cone, which exhibit the same properties as linear models and kernel methods. This model is to consider functions parametrized by a PSD operator on a Hilbert space H on which the input space X is mapped through φ : X → H with linear models notations and x ∈ X → k x ∈ H in RKHS notation (see Eq. (6.4) in Sec. 6.3 ):

{f A : A 0} f A (x) = φ(x) Aφ(x) or f A (x) = k x , Ak x H , x ∈ X .
(5.44)

We call these models PSD models. As this model is itself linear, it can directly be used in the same way to solve e.r.m. problems Eqs. (2.47) and (2.50), although different spectral regularizations may be used (see chapter 6). Moreover we derive a representer theorem similar to Theorem 2.2 for our models in the context of empirical risk minimization in Theorem 6.1, and provide a convex finite-dimensional dual formulation of the learning problem, depending only on the training examples in Theorem 6.2.

In terms of statistics, we prove that the proposed model is a universal approximator and is strictly richer than commonly used generalized linear models. Moreover, we show that its Rademacher complexity is comparable with the one of kernel methods (for more details, see Sec. 6.4 ).

In terms of algorithms, we show the effectiveness of the method through the problems of density estimation, regression with Gaussian heteroscedastic errors, and multiple quantile regression. We

TABLE OF CONTENTS

derive the corresponding learning algorithms for convex dual formulation, and compare it with standard techniques used for the specific problems on a few reference simulations. In this case, we use proximal optimization techniques, and in particular FISTA, presented in Sec. 2.1.3.

In chapter 7, we describe the work by Marteau-Ferey, Bach, and Rudi (2022a). This work applies the modeling framework above to the problem of sampling n i.i.d. samples from a distribution whose density is known up to a constant through function evaluations. Contrary to most of the existing methods in the literature such as rejection sampling or Monte-Carlo Markov chain methods [START_REF] Gelman | Bayesian Data Analysis[END_REF][START_REF] Jun | Monte Carlo Strategies in Scientific Computing[END_REF][START_REF] Lelièvre | Free Energy Computations[END_REF][START_REF] Robert | Monte Carlo statistical methods[END_REF], we propose a two-step procedure by first modelling this density using a positive semidefinite model, and then sampling from this PSD model using an adapted algorithm. In particular, we use PSD models with the Gaussian kernel defined in Sec. 2.2.2 (we will call these models Gaussian PSD models), that is we approximate the target density with a function of the form

f A (x) = n i,j=1 A ij k σ (x i , x)k σ (x j , x), A 0, k σ (x, x ) = exp(-x -x 2 /(2σ 2 )).
(5.45)

Modeling probability distributions with Gaussian PSD models has been developped by [START_REF] Rudi | Psd representations for effective probability models[END_REF], showing that a) they are stable under key operations for probabilistic inference, such as marginalization, integration (also called "sum-rule"), and product, which can be done efficiently in practice, and b) they concisely approximate a large class of probability distributions. The contributions of this chapter are the following.

We derive an algorithm that is easy to implement and which can generate an arbitrary number of i.i.d. samples from a given Gaussian PSD model, with any given precision (see Sec. 7.3 ). This answers one of the open questions outlined by [START_REF] Rudi | Psd representations for effective probability models[END_REF] and shows that one can indeed efficiently sample from a Gaussian PSD model.

We then show that we can sample an arbitrary number of i.i.d. samples from a target probability distribution that is regular enough, with any given precision. The algorithm consists in (a) approximating the un-normalized density p via a PSD model, using evaluations of p, and (b) extracting i.i.d. samples from the PSD model. We show that for sufficiently regular densities the resulting PSD model is concise and avoids the curse of dimensionality : to achieve error ε (we measure this error in Wasserstein, total variation and Hellinger distance), the Gaussian PSD model requires a number of parameters and a number of evaluations of p that are in the order ε -2-d/β , where d is the dimension of the space and β is the order of differentiability of the density. For regular probabilities, i.e., when β ≥ d, the rate does not depend exponentially on d and is bounded by O(ε -3 ) (the constant term instead may depend exponentially on d). Note that the main technical difficulty lies in showing that the approximation phase can be done in a good way statistically (see Sec. 7.4 ), although this is very close in terms of tools to the setting of part I and especially to the least squares setting.

Chapter 6

Non-parametric models for non-negative functions

Introduction

The richness and flexibility of linear models, with the aid of possibly infinite-dimensional feature maps, allowed to achieve great effectiveness from a theoretical, algorithmic, and practical viewpoint in many supervised and unsupervised learning problems, becoming one of the workhorses of statistical machine learning in the past decades [START_REF] Hastie | The Elements of Statistical Learning[END_REF][START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF]. Indeed linear models preserve convexity of the optimization problems where they are used. Moreover they can be evaluated, differentiated and also integrated very easily.

260

CHAPTER 6. PSD MODELS Linear models are adapted to represent functions with unconstrained real-valued or vector-valued outputs. However, in some applications, it is crucial to learn functions with constrained outputs, such as functions which are non-negative or whose outputs are in a convex set, possibly with additional constraints like an integral equal to one, such as in density estimation, regression of multiple quantiles (Bondell, Reich, and Wang, 2010), and isotonic regression [START_REF] Barlow | The isotonic regression problem and its dual[END_REF]. Note that the convex pointwise constraints on the outputs of the learned function must hold everywhere and not only on the training points. In this context, other models have been considered, such as generalized linear models [START_REF] Mccullagh | Generalized Linear Models[END_REF], at the expense of losing some important properties that hold for linear ones.

In this paper, we make the following contributions:

-We consider a class of models with non-negative outputs, as well as outputs in a chosen convex cone, which exhibit the same key properties of linear models. They can be used within empirical risk minimization with convex risks, preserving convexity. They are defined in terms of an arbitrary feature map and they can be evaluated, differentiated and integrated exactly.

-We derive a representer theorem for our models and provide a convex finite-dimensional dual formulation of the learning problem, depending only on the training examples. Interestingly, in the proposed formulation, the convex pointwise constraints on the outputs of the learned function are naturally converted to convex constraints on the coefficients of the model.

-We prove that the proposed model is a universal approximator and is strictly richer than commonly used generalized linear models. Moreover, we show that its Rademacher complexity is comparable with the one of linear models based on kernels.

-To show the effectiveness of the method in terms of formulation, algorithmic derivation and practical results, we express naturally the problems of density estimation, regression with Gaussian heteroscedastic errors, and multiple quantile regression. We derive the corresponding learning algorithms for convex dual formulation, and compare it with standard techniques used for the specific problems on a few reference simulations.

Background

In a variety of fields ranging from supervised learning, to Gaussian processes (Williams and Rasmussen, 2006), inverse problems [START_REF] Werner Engl | Regularization of Inverse Problems[END_REF], scattered data approximation techniques [START_REF] Wendland | Scattered Data Approximation[END_REF], and quadrature methods to compute multivariate integrals (Bach, 2017b), prototypical problems can be cast as

f * ∈ arg min f ∈F L(f (x 1 ), . . . , f (x n )) + Ω(f ). (6.1)
Here L : R n → R is a (often convex) functional, F a class of real-valued functions, x 1 , . . . , x n a given set of points in X , and Ω a suitable regularizer [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF].

Linear models for the class of functions F are particularly suitable to solve such problems. They are classically defined in terms of a feature map φ : X → H where X is the input space and H is a separable Hilbert space. Typically, H = R D with D ∈ N, but H can also be infinite-dimensional.

A linear model is determined by a parameter vector w ∈ H as f w (x) = φ(x) w, (6.2) leading to the space F = {f w | w ∈ H}. These models are particularly effective for problems in the form Eq. ( 6.1) because they satisfy the following key properties.

. BACKGROUND

261

P1. They preserve convexity of the loss function. Indeed, given x 1 , . . . , x n ∈ X , if L : R n → R is convex, then L(f w (x 1 ), . . . , f w (x n )) is convex in w.

P2. They are universal approximators. Under mild conditions on φ and H (universality of the associated kernel function [START_REF] Micchelli | Universal kernels[END_REF])) linear models can approximate any continuous function on X . Moreover they can represent many classes of functions of interest, such as the class of polynomials, analytic functions, smooth functions on subsets of R d or on manifolds, or Sobolev spaces [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF].

P3. They admit a finite-dimensional representation. Indeed, there is a so-called representer theorem [START_REF] Cucker | On the mathematical foundations of learning[END_REF]. Let L be a possibly non-convex functional, F = {f w | w ∈ H}, and assume Ω is an increasing function of w w (see the work by [START_REF] Schölkopf | A generalized representer theorem[END_REF] for more generality and details). Then, the optimal solution f * of (6.1) corresponds to f * = f w * , with w * = n i=1 α i φ(x i ), and α 1 , . . . α n ∈ R. Denoting by k the kernel function k(x, x ) := φ(x) φ(x ) for x, x ∈ X (see, e.g., the work by [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF]), f * can be rewritten as

f * (x) = n i=1 α i k(x, x i ).
(6.3) P4. They are differentiable/integrable in closed form. Assume that the kernel k(x, x ) is differentiable in the first variable. Then ∇ x f w * (x) = n i=1 α i ∇ x k(x, x i ). Also the integral of f w * can be computed in closed form if we know how to integrate k. Indeed, for p : X → R integrable, we have f w * (x)p(x)dx = n i=1 α i k(x, x i )p(x)dx.

Vector-valued models. By juxtaposing scalar-valued linear models, we obtain a vector valued linear model, i.e. f w 1 •••wp : X → R p defined as f w 1 •••wp (x) = (f w 1 (x), . . . , f wp (x)) ∈ R p .

.1 Models for non-negative functions or functions with constrained outputs

While linear models provide a powerful formalization for functions from X to R or R p , in some important applications arising in the context of unsupervised learning, non-parametric Bayesian methods, or graphical models, additional conditions on the model are required. In particular, we will focus on pointwise output constraints. That is, given Y R p , we want to obtain functions satisfying f (x) ∈ Y for all x ∈ X . A prototypical example is the problem of density estimation.

Example 6.1 (density estimation problem). The goal is to estimate the density of a probability ρ on X , given some i.i.d. samples x 1 , . . . , x n . It can be formalized in terms of Eq. (6.1) (e.g., through maximum likelihood), with the constraint that f is a density, i.e., f (x) ≥ 0, ∀x ∈ X , and X f (x)dx = 1.

Despite the similarity with Eq. (6.1), linear models cannot be applied because of the constraint f (x) ≥ 0. Existing approaches to deal with the problem above are reported below, but lack some of the crucial properties P1-4 that make linear models so effective for problems of the form Eq. (6.1).

Generalized linear models (GLM). Given a suitable map ψ : R p → Y, these models are of the form f (x) = ψ(w φ(x)). In the case of non-negative functions, common choices are ψ(z) = e z , leading to the exponential family, or the positive part function ψ(z) = max(0, z).

GLM have an expressive power comparable to linear models, being able to represent a wide class of functions, and admit a finite-dimensional representation [START_REF] Cheney | A Course in Approximation Theory[END_REF] (they thus satisfy P2 and P3). However, in general they do not preserve convexity of the functionals where they are used (except for specific cases, such as L = -n i=1 log z i and ψ(z) = e z (McCullagh and Nelder, 1989)). Moreover they cannot be integrated in closed form, except for specific φ, requiring some Monte Carlo approximations [START_REF] Robert | Monte Carlo statistical methods[END_REF] (thus missing P1 andP4). An elegant way to obtain a GLM-like non-negative model is via non-parametric mirror descent (Yang, Wang, Kiyavash, and He, 2019) (see, e.g., their Example 4). A favorable feature of this approach is that the map ψ is built implicitly according to the geometry of Y. However, still the resulting model does not always satisfy P3, does not satisfy P1 and P4 , and is only efficient in small-dimensional input spaces.

Non-negative coefficients models (NCM). Leveraging the finite-dimensional representation of linear models in Eq. ( 6.3), the NCM models represent non-negative functions as f (x) = n i=1 α i k(x, x i ), with α 1 , . . . α n ≥ 0, given a kernel k(x, x ) ≥ 0 for any x, x ∈ X , such as the Gaussian kernel e -x-x 2 or the Abel kernel e -x-x . By construction these models satisfy P1, P3, P4. However, they do not satisfy P2. Indeed the fact that α 1 , . . . , α n ≥ 0 does not allow cancellation effects and thus strongly constrains the set of functions that can be represented, as illustrated below.

Example 6.2. The NCM model cannot approximate arbitrarily well a function with a width strictly smaller that the width of the kernel. Take k(x, x ) = e -x-x 2 and try to approximate the function e -x 2 /2 on [-1, 1]. Independently of the chosen n or the chosen locations of the points (x i ) n i=1 , it will not be possible to achieve an error smaller than a fixed constant (Sec. 6.D for a simulation).

Partially non-negative linear models (PNM).

A partial solution to have a linear model that is pointwise non-negative is to require non-negativity only on the observed points (x i ) n i=1 . That is, the model is of the form w φ(x), with w ∈ {w ∈ H | w φ(x 1 ) ≥ 0, . . . , w φ(x n ) ≥ 0}. While this model is easy to integrate in Eq. (6.1), this does not guarantee the non-negativity outside of a neighborhood of (x i ) n i=1 . It is possible to enrich this construction with a set of points that cover the whole space X (i.e., a fine grid, if X = [-1, 1] d ), but this usually leads to exponential costs in the dimension of X and is not feasible when d ≥ 4.

Proposed Model for Non-negative Functions

In this section we consider a non-parametric model for non-negative functions and we show that it enjoys the same benefits of linear models. In particular, we prove that it satisfies at the same time all the properties P1, . . . , P4. As linear models, the model we consider has a simple formulation in terms of a feature map φ : X → H.

Let S(H) be the set of bounded Hermitian linear operators from H to H (set of symmetric D × D matrices if H = R D with D ∈ N) and denote by A 0 the fact that A is a positive semi-definite operator (a positive semi-definite matrix, when H is finite-dimensional) [START_REF] Reed | Methods of Modern Mathematical Physics: Functional Analysis[END_REF][START_REF] Horn | Matrix analysis[END_REF]. The model is defined for all x ∈ X as

f A (x) = φ(x) Aφ(x),
where A ∈ S(H), A 0. (6.4)

. PROPOSED MODEL FOR NON-NEGATIVE FUNCTIONS

263

The proposed model1 is parametrized in terms of the operator (or matrix when H is finite dimensional) A, as in the work by [START_REF] Blondel | Convex factorization machines[END_REF], but with an additional positivity constraint. Note that, by construction, it is linear in A and at the same time nonnegative for any x ∈ X , due to the positiveness of the operator A, as reported below (the complete proof in Sec. 6.B .1).

Proposition 6.1 (Pointwise positivity and linearity in the parameters). Given A, B ∈ S(H) and α, β ∈ R, then

f αA+βB (x) = αf A (x) + βf B (x). Moreover, A 0 ⇒ f A (x) ≥ 0, ∀x ∈ X .
An important consequence of linearity of f A in the parameter is that, despite the pointwise non-negativity in x, it preserves P1, i.e., the convexity of the functional where it is used. First define the set S(H) + as S(H) + = {A ∈ S(H) | A 0} and note that S(H) + is convex [START_REF] Boyd | Convex optimization[END_REF].

Proposition 6.2 (The model satisfies P1). Let L : R n → R be a jointly convex function and x 1 , . . . , x n ∈ X . Then the function

A → L(f A (x 1 ), . . . , f A (x n )) is convex on S(H) + .
proposition 6.2 is proved in Sec. 6.B .2. The property above provides great freedom in choosing the functionals to be optimized with the proposed model. However, when H has very high dimensionality or it is infinite-dimensional, the resulting optimization problem may be quite expensive. In the next subsection we provide a representer theorem and finite-dimensional representation for our model, that makes the optimization independent from the dimensionality of H.

.1 Finite-dimensional representations, representer theorem, dual formulation

Here we will provide a finite-dimensional representation for the solutions of the following problem, inf A 0 L(f A (x 1 ), . . . , f A (x n )) + Ω(A), (6.5)

given some points x 1 , . . . , x n ∈ H. However, the existence and uniqueness of solutions for the problem above depend crucially on the choice of the regularizer Ω as it happens for linear models when H is finite-dimensional [START_REF] Werner Engl | Regularization of Inverse Problems[END_REF]. To derive a representer theorem for our model, we need to specify the class of regularizers we are considering. In the context of linear models a typical regularizer is Tikhonov regularization, i.e., Ω(w) = λw w, for w ∈ H.

Since the proposed model is expressed in terms of a symmetric operator (matrix, if H is finitedimensional), the equivalent of the Tikhonov regularizer is a functional that penalizes the squared Frobenius norm of A, i.e., Ω(A) = λ Tr(A A), for A ∈ S(H) also written as [START_REF] Werner Engl | Regularization of Inverse Problems[END_REF]. However, since A is an operator, we can also consider different norms on its spectrum. From this viewpoint, an interesting regularizer corresponds to the nuclear norm A , which induces sparsity on the spectrum of A, leading to low-rank solutions [START_REF] Recht | Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization[END_REF][START_REF] Blondel | Convex factorization machines[END_REF]. In this paper, for the sake of simplicity we will present the results for the following regularizer, which is the matrix/operator equivalent of the elastic-net regularizer [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF]:

Ω(A) = λ A 2 F (Engl,
Ω(A) = λ 1 A + λ 2 A 2 F , ∀A ∈ S(H), (6.6)
with λ 1 , λ 2 ≥ 0 and λ 1 + λ 2 > 0. Note that Ω is strongly convex as soon as λ 2 > 0; we will therefore take λ 2 > 0 in practice in order to have easier optimization. Recall the definition of the kernel k(x, x ) := φ(x) φ(x ), x, x ∈ X [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF]. We have the following theorem.

Theorem 6.1 (Representer theorem, P3). Let L : R n → R ∪ {+∞} be lower semi-continuous and bounded below, and Ω as in Eq. (6.6). Then Eq. (6.5) has a solution A * which can be written as

n i,j=1 B ij φ(x i )φ(x j ) ,
for some matrix B ∈ R n×n , B 0. (6.7)

A * is unique if L is convex and λ 2 > 0. By Eq. (6.4), A * corresponds to a function of the form

f * (x) = n i,j=1 B ij k(x, x i )k(x, x j ), for some matrix B ∈ R n×n , B 0.
The proof of the theorem above is in Sec. 6.B .3, where it is derived for the more general class of spectral regularizers (this thus extends a result from [START_REF] Abernethy | A new approach to collaborative filtering: Operator estimation with spectral regularization[END_REF], from linear operators between potentially different spaces to positive self-adjoint operators).

A direct consequence of Theorem 6.1 is the following finite-dimensional representation of the optimization problem in Eq. (6.5). Denote by K ∈ R n×n the matrix K i,j = k(x i , x j ) and assume w.l.o.g. that it is full rank (always true when the n observations are distinct and k is a universal kernel such as the Gaussian kernel [START_REF] Micchelli | Universal kernels[END_REF]). Let V be the Cholesky decomposition of K, i.e., K = V V. Define the finite dimensional model fA (x) = Φ(x) AΦ(x), A ∈ R n×n , A 0, (6.8)

where Φ : X → R n , defined as Φ(x) = V -v(x), with v(x) = (k(x, x i )) n i=1 ∈ R n , is the classical empirical feature map. In particular, fA = f A where A is of the form Eq. (6.7) with B = V -1 AV -. We will say that fA is a solution of Eq. (6.5) if the corresponding A is a solution of Eq. (6.5). Proposition 6.3 (Equivalent finite-dimensional formulation in the primal). Under the assumptions of Theorem 6.1, the following problem has at least one solution, which is unique if λ 2 > 0 and L is convex :

min A 0 L( fA (x 1 ), . . . , fA (x n )) + Ω(A).
(6.9)

Moreover, for any given solution A * ∈ R n×n of Eq. (6.9), the function fA * is a minimizer of Eq. (6.5). Finally, note that problems Eq. (6.5) and Eq. (6.9) have the same condition number if it is exists.

The proposition above (proof in Sec. 6.B .4) characterizes the possibly infinite-dimensional optimization problem of Eq. (6.5) in terms of an optimization on n × n matrices. A crucial property is that the formulation in Eq. (6.9) preserves convexity, i.e., it is convex as soon as L is convex. To conclude, Sec. 6.B .4 provides a construction for V valid for possibly rank-deficient K.

We now provide a finer characterization in terms of a dual formulation on only n variables.

. APPROXIMATION PROPERTIES OF THE MODEL 265

Convex dual formulation. We have seen above that the problem in Eq. (6.5) admits a finitedimensional representation and can be cast in terms of an equivalent problem on n × n matrices.

Here, when L is convex, we refine the analysis and provide a dual optimization problem on only n variables. The dual formulation is particularly suitable when L is a sum of functions as we will see later. In the following theorem [A] -corresponds to the negative part2 of A ∈ S(R n ).

Theorem 6.2 (Convex dual problem). Assume L is convex, lower semi-continuous and bounded below. Assume Ω is of the form Eq. (6.6) with λ 2 > 0. Assume that the problem has at least a strictly feasible point, i.e., there exists A 0 0 such that L is continuous in [START_REF] Boyd | Convex optimization[END_REF], problem Eq. (6.9) has the following dual formulation:

(f A 0 (x 1 ), ..., f A 0 (x n )) ∈ R n (
sup α∈R n -L * (α) -1 2λ 2 [V diag(α)V + λ 1 I] - 2 F , (6.10)
and this supremum is atteined. Moreover, if α * ∈ R n is a solution of (6.10), a solution of (6.5) is obtained via (6.7), with B ∈ R n×n defined as

B = λ -1 2 V -1 [V diag(α * )V + λ 1 I] -V -. (6.11)
The result above (proof in Sec. 6.B .5) is particularly interesting when L can be written in terms of a sum of functions, i.e., L(z 1 , . . . , z n ) = n i=1 i (z i ) for some functions i : R → R. Then the Fenchel dual is L * (α) = n i=1 * i (α i ), where * i is the Fenchel dual of i , and the optimization can be carried by using accelerated proximal splitting methods as FISTA [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]

, since [V diag(α)V + λ 1 I] - 2 F is differentiable in α.
This corresponds to a complexity of O(n 3 ) per iteration for FISTA, due to the computation of Eq. (6.11), and can be made comparable with fast algorithms for linear models based on kernels [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF], by using techniques from randomized linear algebra and Nyström approximation [START_REF] Halko | Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions[END_REF]) (see more details in Sec. 6.B .5).

Approximation Properties of the Model

The goal of this section is to study the approximation properties of our model and to understand its "richness", i.e., which functions it can represent. In particular, we will prove that, under mild assumptions on φ, (a) the proposed model satisfies the property P2, i.e., it is a universal approximator for non-negative functions, and (b) that it is strictly richer than the family of exponential models with the same φ. First, define the set of functions belonging to our model

F • φ = {f A | A ∈ S(H), A 0, A • < ∞}, where • • is a suitable norm for S(H).
In particular, norms that we have seen to be relevant in the context of optimization are the nuclear norm • and the Hilbert-Schmidt (Frobenius) norm

• F . Given norms • a , • b , we denote the fact that • a is stronger (or equivalent) than

• b with • a • b (for example, • • F ).
In the next theorem we prove that when the feature map is universal [START_REF] Micchelli | Universal kernels[END_REF], such as the one associated to the Gaussian kernel k(x, x ) = exp(-xx 2 ) or the Abel kernel k(x, x ) = exp(-xx ), then the proposed model is a universal approximator for non-negative functions over X (in particular, in the sense of cc-universality [START_REF] Micchelli | Universal kernels[END_REF][START_REF] Bharath | Universality, characteristic kernels and rkhs embedding of measures[END_REF], see Sec. 6.B .6 for more details and the proof). Proposition 6.4 (Integrability in closed form, P4). Let A ∈ S(H) with A bounded and φ uniformly bounded. Let p : X → R be an integrable function. There exists a trace class operator W p ∈ S(H) such that X f A (x)p(x)dx = Tr(AW p ) and W p = X φ(x)φ(x) p(x)dx.

The result can be extended to derivatives and more general linear functionals on f A (see Sec. 6.B .9). In particular, note that if we consider the empirical feature map Φ in Eq. (6.8), which characterizes the optimal solution of Eq. (6.5), by Theorem 6.1, we have that W p is defined explicitly as W p = V -M p V -1 with (M p ) i,j = k(x, x i )k(x, x j )p(x)dx, for i, j = 1, . . . , n and it is computable in closed form. Then, assuming an equality and an inequality constraint on the integral w.r.t. two functions p and q and two values c 1 , c 2 ∈ R, the resulting problem takes the following finite-dimensional form min A∈S(R n ) L( fA (x 1 ), . . . , fA (x n )) + Ω(A), (6.12)

s.t. A 0, Tr(AW p ) = c 1 , Tr(AW q ) ≤ c 2 .
Representing function with outputs in convex polyhedral cones. We represent a vectorvalued function with our model as the juxtaposition of p scalar valued models, with p ∈ N, as follows

f A 1 •••Ap (x) = (f A 1 (x), . . . , f Ap (x)) ∈ R p , ∀ x ∈ X .
We recall that a convex polyhedral cone Y is defined by a set of inequalities as follows

Y = {y ∈ R p | c 1 y ≥ 0, . . . , c h y ≥ 0}, (6.13)
for some c 1 , . . . , c h ∈ R p and h ∈ N. Let us now focus on a single constraint c y ≥ 0. Note that, by definition of positive operator (i.e., A 0 implies v Av ≥ 0 for any A), we have that p s=1 c s A s 0 implies φ(x) ( p s=1 c s A s )φ(x) ≥ 0 for any x ∈ X , which, by linearity of the inner product and the definition of

f A 1 •••Ap is equivalent to c f A 1 •••Ap (x) ≥ 0.
From this reasoning we derive the following proposition (see complete proof in Sec. 6.B .10). Proposition 6.5. Let Y be defined as in Eq. (6.13). Let A 1 , . . . , A p ∈ S(H). Then the following holds

p s=1 c t s A s 0 ∀t = 1, . . . , h ⇒ f A 1 •••Ap (x) ∈ Y ∀x ∈ X .
Note that the set of constraints on the l.h.s. of the equation above defines in turn a convex set on A 1 , . . . , A p . This means that we can use it to constrain a convex optimization problem over the space of the proposed vector-valued models as follows min

A 1 ,...,Ap∈S(H) L(f A 1 •••Ap (x 1 ), . . . , f A 1 •••Ap (x n )) + p s=1 Ω(A s ) (6.14) s.t. p s=1 c t s A s 0, ∀ t = 1, . . . , h.

By proposition 6.5, the function f

A * 1 •••A *
p , where (A * 1 , . . . , A * p ) is the minimizer above, will be a function with output in Y. Moreover, the formulation above admits a finite-dimensional representation analogous to the one for non-negative functions, as stated below (see proof in Sec. 6.B .11) Theorem 6.6 (Representer theorem for model with output in convex polyhedral cones). Under the assumptions of Theorem 6.1, the problem in Eq. (6.14) admits a minimizer

(A * 1 , • • • , A * p ) of the form A * s = n i,j=1 [B s ] i,j φ(x i )φ(x j ) =⇒ (f * (x)) s = n i,j=1 [B s ] i,j k(x i , x)k(x j , x), s = 1, ..., p,
where

f * := f (A * 1 ,...,A * p )
is the corresponding function and the B s ∈ S(R n ) are symmetric n × n matrices which satisfy the conic constraints p s=1 c t s B s 0, t = 1, ..., h. Remark 16 (Efficient representations when the ambient space of Y is high-dimensional). When p h, or when Y is a polyhedral cone with Y ⊂ G and G an infinite-dimensional space, it is still possible to have an efficient representation of functions with output in Y by using the representation of Y in terms of conical hull [START_REF] Boyd | Convex optimization[END_REF], i.e., Y = { t i=1 α i y i | α i ≥ 0} for some y 1 , . . . , y t and t ∈ N. In particular, given A 1 , . . . , A t 0, the model

f A 1 ...At (x) = t i=1 f A i (x)y i satisfies f A 1 ...At (x) ∈ Y for any x ∈ X .
Moreover it is possible to derive a representer theorem as Theorem 6.6.

Remark 17. By extending this approach, we believe it is possible to model (a) functions with output in the cone of positive semidefinite matrices, (b) convex functions. We leave this for future work.

Numerical Simulations

In this section, we provide illustrative experiments on the problems of density estimation, regression with Gaussian heteroscedastic errors, and multiple quantile regression. We derive the algorithm according to the finite-dimensional formulation in Eq. (6.12) for non-negative functions with constraints on the integral, and to Eq. (6.14) with the finite-dimensional representation suggested by Theorem 6.6. Optimization is performed applying FISTA [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] on the dual of the resulting formulations. More details on implementation and the specific formulations are given below and in Sec. 6.E . The algorithms are compared with careful implementations of Eq. (6.1) with the models presented in Sec. 6.2 .1, i.e., partially non-negative models (PNM), non-negative coefficients models (NCM) and generalized linear models (GLM). For all methods we used Ω(A) = λ A * + 0.01 2 A 2 F or Ω(w) = λ w 2 . We used the Gaussian kernel k(x, x ) = exp(-xx 2 /(2σ 2 )) with width σ. Full cross-validation has been applied to each model independently, to find the best λ (see Sec. 6.E ).

Density estimation. This problem is illustrated in Example 6.1. Here we considered the log-likelihood loss as a measure of error, i.e., L(z 1 , . . . , z n ) = -1 n n i=1 log(z i ), which is jointly convex and with an efficient proximal operator [START_REF] Chaux | A variational formulation for frame-based inverse problems[END_REF]. We recall that the problems are constrained to output a function whose integral is 1. In Fig. 6.1, we show the experiment on n = 50 i.i.d. points sampled from ρ(x) = 1 2 N (-1, 0.3) + 1 2 N (1, 0.3) and where for all the models we used σ = 1, to illustrate pictorially the main interesting behaviors. Instead in Sec. 6.E , we perform a multivariate experiment in d = 10 and n = 1000, where we cross-validated σ for each algorithm and show the same effects more quantitatively. Note that PNM (left) is non-negative on the training points, but it achieves negative values on the regions not covered by examples. This effect is worsened by the constraint on the integral that borrows areas from negative regions to reduce the log-likelihood on the dataset. NCM (center-left) produces a function whose integral is one and that is non-negative everywhere, but the poor approximation properties of the model do not allow to fit the density of interest (see Example 6.2). GLM (center-right) produces a function that is non-negative and approximates quite well ρ, however, the obtained function does not sum to one, but to 0.987, since the integral constraint can be enforced only approximately via Monte Carlo sampling (GLM does not satisfy P4). Estimating the integral is easy in low dimensions but becomes soon impractical in higher dimensions [START_REF] Robert | Monte Carlo statistical methods[END_REF]. Finally the proposed model (right) leads to a convex problem and produces a non-negative function whose integral is 1 and that fits the density ρ quite well.

Heteroscedastic Gaussian process estimation. The goal is to estimate µ : R → R and v : R → R + determining the conditional density ρ of the form ρ(y|x) = (2πv(x)) -1/2 exp(-(yµ(x)) 2 /(2v(x))) from which the data are sampled. The considered functional corresponds to the negative log-likelihood, i.e., L = n i=1 1 2 log v(x i ) + (y i -µ(x i )) 2 /(2v(x i )) that becomes convex in η, θ via the so called natural parametrization η(x) = µ(x)/v(x) and θ(x) = 1/v(x) [START_REF] Quoc | Heteroscedastic gaussian process regression[END_REF]. We used a linear model to parametrize η and the non-negative models for θ. The experiment on the same model of (Le, Smola, and Canu, 2005; Yuan and [START_REF] Yuan | Doubly penalized likelihood estimator in heteroscedastic regression[END_REF] is reported in Fig. 6.1. Modeling θ via PNM (left) leads to a convex problem and reasonable performance. In particular, the fact that θ = 0 corresponds to v = +∞ prevents the model for θ from crossing zero. NCM (center-left) leads to a convex problem, but very sensitive to the kernel width σ and with poor approximation properties. GLM (center-right) leads to a non-convex problem and we need to restart the method randomly to have a reasonable convergence. Our model (right) leads to a convex problem and produces a non-negative function for θ, that fits well the observed data.

Multiple quantile regression.

The goal here is to estimate multiple quantiles of a given conditional distribution P (Y |x). Given τ ∈ (0, 1), q τ defined by

P (Y > q τ (x)|x) = τ is the τ -quantile of ρ. By construction 0 < τ -h ≤ • • • ≤ τ h < 1 implies q τ -h (x) ≤ • • • ≤ q τ h (x). If
we denote by q : X → R 2h+1 the list of quantiles, we have by construction q

(x) ∈ Y where Y is a convex cone Y = {y ∈ R h | y -h ≤ • • • ≤ y h }.
To regress quantiles, we used the pinball loss L τ (convex, non-smooth) considered by [START_REF] Koenker | Quantile Regression[END_REF]; [START_REF] Steinwart | Estimating conditional quantiles with the help of the pinball loss[END_REF], obtaining L = h j=-h n i=1 L τ j (f (x i ), y i ). In Fig. 6.1, we used τ -1 = 1 4 , τ 0 = 1 2 , τ 1 = 3 4 . Using PNM, (left) the ordering is enforced by explicit constraints on the observed dataset [START_REF] Takeuchi | Nonparametric quantile regression[END_REF][START_REF] Howard | Noncrossing quantile regression curve estimation[END_REF]. The resulting problem is convex. However, in regions with low density of points, PNM quantiles do not respect their natural order. To enforce the order constraint, a fine grid covering the space would be needed [START_REF] Takeuchi | Nonparametric quantile regression[END_REF]. For NCM, GLM and our model, we represented the quantiles as q τ ±j = q τ 0 ± i=1 v ±i where the v's are non-negative functions and q τ 0 , with τ 0 = 1 2 , is the median and is modeled by a linear model. NCM (center-left) leads to a convex problem and quantiles that respect the ordering, but the estimation is very sensitive to the chosen σ and has poor approximation properties. GLM (center-right) leads to a non-convex non-differentiable problem, with many local minima, which is difficult to optimize with standard techniques (see Sec. 6.E ). GLM does not succeed in approximating the quantiles. Our model (right) leads to a convex optimization problem that approximates the quantiles relatively well and preserves their natural order everywhere.

6.A Notation and basic definitions

• H is a separable Hilbert space.

• X is a Polish space (we will require explicitly compactness in some theorems).

• φ : X → H is a continous map. We also assume it to be uniformly bounded i.e. sup x∈X φ(x) ≤ c for some c ∈ (0, ∞), if not differently stated.

• k(x, x ) := φ(x) φ(x ) is the kernel function associated to the feature map φ [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF][START_REF] Berlinet | Reproducing Kernel Hilbert Spaces in Probability and Statistics[END_REF].

6.B Proofs and additional discussions 6.B .1 Proof of proposition 6.1

In this section, let us extend the definition in Eq. ( 6.4) to any operator A ∈ S(H), without the implied positivity restriction (in Eq. ( 6.4), we ask that A 0) :

∀A ∈ S(H), ∀x ∈ X , f A (x) := φ(x) Aφ(x). ( 4bis
)
Proof of proposition 6.1. To prove linearity, let A, B ∈ S(H) and α, β ∈ R. Since S(H) is a vector space, αA + βB ∈ S(H). Let x ∈ X . By definition for the first equality and linearity for the second,

f αA+βB (x) = φ(x) (αA + βB)φ(x) = αφ(x) Aφ(x) + βφ(x) Bφ(x).
Finally, since by definition, f A (x) = φ(x) Aφ(x) and f B (x) = φ(x) Bφ(x), it holds :

f αA+βB (x) = αφ(x) Aφ(x) + βφ(x) Bφ(x) = αf A (x) + βf B (x).
Since this holds for all x ∈ X , this shows

f αA+βB = αf A + βf B .
To prove the non-negativity, assume now that A 0. By definition of of positive semi-definiteness, ∀h ∈ H, h Ah ≥ 0.

In particular, for any x ∈ X , the previous inequality applied to h = φ(x) yields

f A (x) = φ(x) Aφ(x) ≥ 0.
Hence, f A ≥ 0.

6.B .2 Proof of proposition 6.2

Recall the definition of f A for any A ∈ S(H) in Eq. (4bis). We have the lemma: Lemma 6.1 (Linearity of evaluations). Let x 1 , . . . , x n ∈ X . Then the map

A ∈ S(H) → (f A (x i )) 1≤i≤n ∈ R n is linear from S(H) to R n .
Proof. This just follows from the fact that the definition of

f A (x i ), f A (x i ) := φ(x i ) Aφ(x i ), is linear in A.
Proof of proposition 6.2. Let L : R n → R be a jointly convex function and x 1 , . . . , x n ∈ X . The function A ∈ S(H) → L(f A (x 1 ), . . . , f A (x n )) can be written L • R, where

R : A ∈ S(H) → (f A (x i )) 1≤i≤n ∈ R n .
Since L is convex, and R is linear by Lemma 6.1, their composition is convex.

Moreover, since S(H) + is a convex subset of S(H), the restriction of

A ∈ S(H) → L(f A (x 1 ), . . . , f A (x n ))
on S(H) + is also convex.

6.B .3 Proof of Theorem 6.1

In this section, we prove Theorem 6.1 for a more general class of spectral regularizers.

Compact operators and spectral functions

In this section, we briefly introduce compact self-adjoint operators and the spectral theory of compact self-adjoint operators. For more details, see for instance the work by [START_REF] Gohberg | Basic Classes of Linear Operators[END_REF]. We start by defining a compact self-adjoint operator (Gohberg, Goldberg, and Kaashoek, 2004, Section 2.16) and stating its main properties: Definition 6.1 (compact operators). Let H be a separable Hilbert space. A bounded self-adjoint operator A ∈ S(H) is said to be compact if its range is included in a compact set. We denote with S ∞ (H) the set of compact self adjoint operators on H. It is a closed subspace of S(H) for the operator norm and the closure of the set of finite rank operators.

Proposition 6.6 ( Gohberg, Goldberg, and Kaashoek (2004, Spectral theorem)). Let H be a separable Hilbert space and let A be a compact self adjoint operator on H. Then there exists a spectral decomposition of A, i.e., an orthonormal system (u k ) ∈ H of eigenvectors of A and corresponding eigenvalues (σ k ) such that for all h ∈ H, it holds

Ah = k σ k u k h u k =: k σ k u k u k h.
Moreover, if σ k is an infinite sequence, it converges to zero. Furthermore, we say that the orthonormal system (u k ) of eigenvectors of A and the corresponding eigenvalues (σ k ) is a basic system of eigenvectors of A if all the σ k are non zero. In this case, if P 0 denotes the orthogonal projection on Ker(A), then it holds

∀h ∈ H, h = Π 0 h + k u k u k h
In what follows, to simplify notations, we will usually write A = U diag(σ)U in order to denote a basic system of eigenvectors of A. Moreover, if A is positive semi-definite, we will assume that the eigenvalues are sorted in decreasing order, i.e., σ k+1 ≤ σ k .

Definition 6.2 (Spectral function on S ∞ (H) [START_REF] Gohberg | Basic Classes of Linear Operators[END_REF]). Let q : R → R be a lower semi-continuous function such that q(0) = 0. Let H be any separable Hilbert space. For any A ∈ S ∞ (H) and any basic system A = U diag(σ)U , we define the spectral function q q(A)

= U diag(q(σ)))U T = k q(σ k )u k u k .

Classes of regularizers

Let us now state our main assumption on regularizers.

Assumption 6.1 (Assumption on regularizers). Ω is of of the form

∀A ∈ S(H), Ω(A) = Tr(q(A)) = k q(σ k ) if A = U diag(σ)U ∈ S ∞ (H), k q(σ k ) < ∞ +∞ otherwise,
where q : R → R + is:

• non-decreasing on R + with q(0) = 0;

• lower semi-continuous;

• q(σ) -→ |σ|→+∞ +∞.
Note that in this case, Ω is defined on S(H) for any Hilbert space H.

Remark 18. Ω(A) = λ 1 A + λ 2 2 A 2 F satisfies Assumption 6.1, with q(σ) = λ 1 |σ| + λ 2 σ 2 . Lemma 6.2 (Properties of Ω).
Let Ω satisfying Assumption 6.1. Then the following properties hold.

(i) For any separable Hilbert spaces H 1 , H 2 and any linear isometry O : H 1 → H 2 , i.e., such that O * O = I H 1 , it holds ∀A ∈ S(H 1 ), Ω(OAO * ) = Ω(A).

(ii) For any separable Hilbert space H and any orthogonal projection Π ∈ S(H 1 ), i.e., satisfying Π = Π * , Π 2 = Π, it holds ∀A 0, Ω(ΠAΠ) ≤ Ω(A).

(iii) For any finite dimensional Hilbert space H n ,

Ω is lower semi-continuous (l.s.c), Ω(A) -→ A op→+∞

+∞

where we denoted by • op the operator norm.

Proof. (i) Write A = k σ k u k u k where the (u k ) form a basic system of eigen-vectors for A.

The (v k ) = (Ou k ) form a basic system of eigen-vectors for OAO * , as

OAO * = k σ k v k v k , σ k = 0.
Hence, by definition, q(OAO * ) = k q(σ k )v k v k . By definition of the trace, we have

Ω(OAO * ) = k q(σ k ) = Ω(A).
(ii) Let A be a compact self-adjoint semi-definite operator. Let A = U diag(σ)U be a basic system of eigenvectors of A, where the σ k are positive and in decreasing order. Define B = U diag( √ σ)U and note that in this case, A = B 2 = B * B. Using Exercise 23 by [START_REF] Gohberg | Basic Classes of Linear Operators[END_REF], we have that for any orthogonal projection operator Π and any index k, σ k (ΠB * BΠ) ≤ σ k (B * B) and hence σ k (ΠAΠ) ≤ σ k (A). Since q is non decreasing, it holds q(σ k (ΠAΠ)) ≤ q(σ k (A)) and hence

Ω(ΠAΠ) = k q(σ k (ΠAΠ)) ≤ k q(σ k (A)) = Ω(A).
(iii) Let H n be a finite dimensional Hilbert space and let • op be the operator norm on S(H n ).

If q is continuous, then A ∈ H n → q(A) is continuous and hence Ω is continuous (since the trace is continuous in finite dimensions). Now assume q is lower semi-continuous, and define for n ∈ N, q n (t) := inf s∈R q(s) + n|t -s|. We have q n ≥ 0, q n (0) = 0 q n is uniformly continuous and q n is an increasing sequence of functions such that q n → q point-wise. Now it is easy to see that Tr(q(A)) = sup n Tr(q n (A)) and hence Ω is lower semi-continuous as a supremum of continuous functions.

The fact that Ω goes to infinity is a direct consequence of the fact that q goes to infinity, by Assumption 6.1.

Remark 19. The three conditions of the previous lemma are in fact the only conditions needed in the proof. We could loosen Assumption 6.1 to satisfy only these three properties.

Finite-dimensional representation and existence of a solution

Fix n ∈ N, a loss function L : R n → R ∪ {+∞}, a separable Hilbert space H, a regularizer Ω on S(H) a feature map φ : X → H and points (x 1 , ..., x n ) ∈ X n .

Recall the problem in Eq. (6.5):

inf A 0 L(f A (x 1 ), . . . , f A (x n )) + Ω(A). (6.5)
Define H n to be the finite-dimensional subset of H spanned by the φ(x i ), i.e.,

H n := span (φ(x i )) 1≤i≤n = n i=1 α i φ(x i ) : α ∈ R n .
Define Π n is the orthogonal projection on H n , i.e.,

Π n ∈ S(H), Π 2 n = Π n , range(Π n ) = H n .
Define S n (H) + to be the following subspace of S(H) + :

S n (H) + := Π n S(H) + Π n = {Π n AΠ n : A ∈ S(H) + } .
Proposition 6.7. Let L be a lower semi-continuous function which is bounded below, and assume Ω satisfies Assumption 6.1. Then Eq. (6.5) has a solution A * which is in S n (H) + .

CHAPTER 6. PSD MODELS

Remark 20 (Constructing V). In the case where the kernel matrix K is full rank, V ∈ R n×n and is invertible, and O n can be simply written S * n V -1 . In the case where the kernel matrix K is not full-rank, we build V as V = Σ 1/2 U , where Σ ∈ R r×r is diagonal and U ∈ R n×r is unitary and correspond to the economy eigendecomposition of K where r is the rank of K, i.e., K = UΣU .

Consider the following generalization of the finite dimensional model proposed in Eq. (6.8) in the case where K is not necessarily full rank : (6.8) where Φ :

fA (x) = Φ(x) AΦ(x), A ∈ R r×r , A 0,
X → R r is defined as Φ(x) = O * n φ(x) = (VV ) -1 Vv(x), where v(x) = (k(x i , x)) 1≤i≤n ∈ R n .
We are now ready to prove proposition 6.3.

Proof of proposition 6.3. Recall min A 0 L( fA (x 1 ), . . . , fA (x n )) + Ω(A).

(6.9)

The fact that Eq. ( 6.9) has a solution, and that this solution is unique if λ 2 > 0 and L is convex can be seen as a simple consequence of Theorem 6.7 in the case where the model considered is the finite dimensional model defined in Eq. (6.8). Let us now prove the other part of the proposition.

Start by noting that with our definition of O n , for all A ∈ R r×r , A 0,

f OnAO * n = fA . (a)
Moreover,

O n AO * n : A ∈ R r×r , A 0 = S n (H) + . (b)
Finally, since O n is an isometry which implies Ω(O n AO * n ) = Ω(A) and by Eq. (a), for any A ∈ S(R n ) + , it holds :

L(f OnAO * n (x 1 ), ..., f OnAO * n (x n )) + Ω(O n AO * n ) = L( fA (x 1 ), ..., fA (x n )) + Ω(A). (c)
Now combining Eq. (c) and Eq. (b), any solution A * to Eq. (6.9) corresponds to a solution

A * ∈ arg min A∈Sn(H) + L(f A (x 1 ), ..., f A (x n )) + Ω(A), where A * = O n A * O * n
. Now using Eq. ( 6.15) in the proof of proposition 6.7, we see that A * is also a minimizer of Eq. (6.5) hence the result.

Note that the fact that the condition number of the problem, if it exists, is preserved because O n is an isometry.

6.B .5 Proof of Theorem 6.2 and algorithmic consequence.

In this section, we prove Theorem 6.2 and explain how to derive an efficient algorithm to solve it in certain cases.

Let us start by proving the following lemma. Lemma 6.5. Let λ 1 , λ 2 ≥ 0 and assume λ 2 > 0. Let Ω + be defined on S(R r ) as follows :

Ω + (A) = λ 1 A + λ 2 2 A 2 F if A 0; +∞ otherwise .
Then Ω + is a closed convex function, and its Fenchel conjugate is given for any B ∈ S(R r ) by the formula:

Ω * + (B) = 1 2λ 2 [B -λ 1 I] + 2 F .
Moreover, Ω + is differentiable at every point, and is 1/λ 2 smooth. Its gradient is given by:

∇Ω * + (B) = 1 λ 2 [B -λ 1 I] + . Proof. Write Ω + (A) = ι S(R r ) + + λ 1 A + λ 2 2 A 2 F .
Here, ι C stands for the characteristic function of the convex set C, i.e. ι C (x) = 0 if x ∈ C and +∞ otherwise. Since • 2 F and • are both convex, continuous, and real valued, and since ι S(R r ) + is closed since S(R r ) + is a closed non-empty convex subset of S(R r ), this shows that Ω + is indeed convex and closed. Note that it is continuous on its domain S(R r ) + . Moreover, it is strongly convex since λ 2 > 0. Fix B ∈ S(R r ) and consider the problem sup

A∈S(R r ) Tr(AB) -Ω + (A) = sup A 0 Tr(A(B -λ 1 I)) - λ 2 2 A 2 F
Since Ω + is strongly convex, we know there exists a unique solution to this problem.

Note that A * = arg max Tr(AB) -Ω + (A) if and only if

A * = arg min A∈S(R r ) + 1 2 A - 1 λ 2 (B -λ 1 I) 2 .
That is A * is the orthogonal projection of B-λ 1 I λ 2 on S(R r ) + for the Frobenius scalar product.

Hence,

A * = B-λ 1 I λ 2 + .
Here, for any symetric matrix C, we denote with [C] + resp [C] -its positive resp negative part. Given an eigendecomposition C = U ΣU T with Σ diagonal, they are defined by [C] + = U max(0, Σ)U T and [C] -= U max(0, -Σ)U T . Hence, the Fenchel conjugate of Ω + is given by

Ω * + (B) = 1 2λ 2 [B -λ 1 I] + 2 F . Consider ω * + : σ ∈ R → max(0, σ 2 ) ∈ R. ω *
+ is 1-smooth and differentiable, and (ω * + ) (σ) = max(0, σ). Hence, the function

B → Tr(ω * + (B)) = [B] + 2 F
is differentiable and 1-smooth, with differential given by the spectral function

(ω * + ) (B) = [B] + . Hence, Ω + is differentiable and ∇Ω * + (B) = 1 λ 2 [B -λ 1 I] + , and is 1/λ 2 smooth.
Then, by denoting with h(x) = g(x) -φ(x) w √ g,Z, Q , we have

g -f A g,Z, C(Z) = sup x∈Z |g(x) -(φ(x) w √ g,Z, Q ) 2 | (6.17) = sup x∈Z g(x) -φ(x) w √ g,Z, Q g(x) + φ(x) w √ g,Z, Q (6.18) = sup x∈Z |h(x)(2 g(x) -h(x))| (6.19) ≤ h C(Z) (2 √ g C(Z) + h C(Z) ) (6.20) ≤ Q 2 g 1/2 C(Z) + Q ≤ . (6.21)
The last step is due to the fact that /Q ≤ √ , then 2 g

1/2 C(Z) + Q ≤ Q.
6.B .7 Proof and additional discussion of Theorem 6.4

In Theorem 6.10, stated below, we prove that E φ ⊆ F φ under the very general assumption that G φ is a multiplication algebra, i.e.. if G φ is closed under pointwise product of the functions. In Theorem 6.11 we specify this result when G φ is a Sobolev space, proving that E φ F • φ . Theorem 6.4 is a direct consequence of the latter theorem.

General result when G φ is a multiplication algebra. First we endow G φ with a Hilbertian norm. Define • G φ as f w G φ = w H , for any w ∈ H. Definition 6.3. G φ is a multiplication algebra, when there exists a constant C such that the unit function u : X → R that maps x → 1 for any x ∈ X is in G φ and (6.22) where we denote by f • g the pointwise multiplication, i.e., (f • g)(x) = f (x)g(x) for all x ∈ X .

f • g G φ ≤ C f G φ g G φ , ∀ f, g ∈ G φ ,
Remark 21 (Renormalizing the constant). Note that when G φ is a multiplication algebra for a constant C, it is always possible to define an equivalent norm

• G φ as • G φ = C • G φ for which G φ is a multiplication algebra with constant 1.
Theorem 6.10 (General version when G φ is an algebra). Let • • • . Let X be a compact space and φ be a bounded continuous map such that G φ is a multiplication algebra, then E φ ⊆ F • φ .

Proof. Let g ∈ E φ and take f ∈ G φ such that g(x) = e f (x) for all x ∈ X . First we prove that

E φ ⊆ F • φ .
With this goal, first we prove that √ g ∈ G φ and then we construct a rank one positive operator such that f Ag (x) = g(x) for every x ∈ X . We start noting that, given f ∈ G φ and t ∈ N,

f t defined by f • f t-1 for t ∈ N satisfies f t ∈ G φ , with f t G φ ≤ C t f t G φ
, by repeated application of the Eq. (6.22). Moreover note that the function s = t∈N

1 2 t t! f t , satisfies s ∈ G φ , indeed s G φ ≤ t∈N 1 2 t t! f t G φ ≤ t∈N 1 2 t t! C t f t G φ ≤ e C f G φ /2 .
Moreover s satisfies s(x) = g(x) for all x ∈ X , indeed for x ∈ X we have

s(x) = φ(x) s = t∈N 1 2 t t! φ(x) f t = t∈N 1 2 t t! f t (x) = e f (x)/2 = g(x)
.

CHAPTER 6. PSD MODELS

Now let A g = s ⊗ s, we have that A g • ≤ A g by assumption, and

A g = s 2 G φ < ∞, so the function f Ag ∈ F • φ and for any x ∈ X f Ag (x) = φ(x) A g φ(x) = φ(x) (s ⊗ s)φ(x) = (φ(x) s) 2 = g(x).
Since for any g ∈ E φ there exists f Ag ∈ F • φ that is equal to g on their domain of definition, we have that

E φ ⊆ F • φ .
Now we are going to specialize the result above for Sobolev spaces.

Result for Sobolev spaces The result below is based on the general result in Theorem 6.10, however it is possible to do a proof based only on norm inequalities for compositions of functions in Sobolev space (see for example the work by [START_REF] Brezis | Gagliardo-nirenberg, composition and products in fractional sobolev spaces[END_REF]). While more technical, this second approach would allow to derive also a more quantitative analysis on the norms of the functions in G φ and F • φ . We will leave this for a longer version of this work.

Theorem 6.11. Let • • • . Let X ⊆ R d and X compact with locally Lipschitz boundary and let G φ = W m 2 (X ). Let x 0 ∈ X . Then the following holds:

(a) E φ F • φ . (b) The function f x 0 (x) = e -x-x 0 -2 ∈ C ∞ (X ) satisfies f x 0 ∈ F • φ and f x 0 / ∈ E φ .
Proof. First we prove that E φ ⊆ F • φ , via Theorem 6.10, then we. To apply this result we need first to prove that G φ = W m 2 (X ) is a multiplication algebra when W m 2 (X ) is a RKHS as in our case.

Step 1, m > d/2. First note that G φ satisfies m > d/2 since W m 2 (X ) admits a representation in terms of a separable Hilbert space H and a feature map φ : X → H, i.e., it is a reproducing Kernel Hilbert space and for the same reason [START_REF] Wendland | Scattered Data Approximation[END_REF].

• G φ is equivalent to • W m 2 (X )
Step 2. G φ is a multiplication algebra. Applying Theorem 6.10. Since G φ = W m 2 (X ) with m > d/2, then it is a multiplication algebra. This result is standard (e.g. see pag. 106 by [START_REF] Adams | Sobolev Spaces[END_REF] for m ∈ N and X = R d ) and we report it in Lemma 6.8 in Sec. 6.C . Then we apply Theorem 6.10 obtaining E φ ⊆ F • φ .

Step 3. Proving that

f x 0 ∈ F • φ and not in E φ . By construction the function v(x) = e -1/(2 x-x 0 2 ) is in C ∞ (X ) and so in W m 2 (X ) for any m ≥ 0. Since G φ = W m 2 (X ), then v ∈ G φ , i.e., there exists w ∈ H such that w φ(•) = v(•). Define A v = w ⊗ w, then f Av (x) = φ(x) A v φ(x) = (w φ(x)) 2 = v 2 (x) = f x 0 (x), ∀x ∈ X . Then f x 0 = f Av on X , i.e., f x 0 ∈ F • φ .
To conclude note that, f x 0 does not belong to E φ , since x 0 ∈ X and f x 0 (x 0 ) = 0, while for any g ∈ E φ we have inf x∈X g(x) > 0. Indeed, we have that for

any f ∈ G φ , f C(X ) = sup x∈X |f (x)| < ∞, since G φ = W m 2 (X ) ⊂ C(X ).
Moreover, given g ∈ G φ , and denoting by f ∈ G φ the function such that g = e f , we have that inf

x∈X g(x) ≥ e -f C(X ) > 0. Finally, since E φ ⊆ F • φ , but there exists f x 0 ∈ F • φ and not in E φ , then E φ F • φ .
Proof of Theorem 6.4. This result is a direct application of Theorem 6.11, since X = [-R, R] d , with R ∈ (0, ∞) is a compact set with Lipschitz boundary.

6.B .8 Proof of Theorem 6.5

We recall here the Rademacher complexity and prove Theorem 6.5. This latter theorem is obtained from the following Theorem 6.12 that bounds the empirical Rademacher complexity introduced below. First we recall that the function class F • φ,L is defined as

F • φ,L = {f A | A 0, A • ≤ L},
for a given norm • • on operators, a feature map φ : X → H and L > 0. Now we define the empirical Rademacher complexity and the Rademacher complexity [START_REF] Bartlett | Rademacher and gaussian complexities: Risk bounds and structural results[END_REF]. Given x 1 , . . . , x n ∈ X , the empirical Rademacher complexity for a class F of functions mapping X to R, is defined as

R n (F) = 2E sup f ∈F 1 n n i=1 σ i f (x i ) ,
where σ i independent Rademacher random variables, i.e., σ i = -1 with probability 1/2 and +1 with probability 1/2 and the expectation is on σ 1 , . . . , σ n . Let ρ be a probability distribution on X and x 1 , . . . , x n sampled independently according to ρ. The Rademacher complexity R n (F) is defined as

R n (F) = E R n (F),
where the last expectation is on x 1 , . . . , x n . In the following theorem we bound R n .

Theorem 6.12.

Let • • • F . Let x 1 , . . . , x n ∈ X , L ≥ 0. R n (F • φ,L ) ≤ 2L n n i=1 φ(x i ) 4 . Proof. Given f A ∈ F • φ,L , since • • is stronger or equivalent to Hilbert-Schmidt norm, we have that A F ≤ A • ≤ L. Since A is bounded and φ(•) ∈ H, by linearity of the trace we have f A (x) = φ(x) Aφ(x) = Tr(A φ(x) ⊗ φ(x)) for any x ∈ X . Then, by linearity of the trace Rn (F • φ,L ) = 2E sup f ∈F • φ,L 1 n n i=1 σ i f (x i ) = 2E sup A 0, A •≤L 1 n n i=1 σ i φ(x i ) Aφ(x i ) (6.23) = 2E sup A 0, A •≤L 1 n n i=1 σ i Tr(A (φ(x i ) ⊗ φ(x i ))) (6.24) = 2E sup A 0, A •≤L Tr A 1 n n i=1 σ i φ(x i ) ⊗ φ(x i ) (6.25) Now since • • is stronger or equivalent to • F this means that {A ∈ S(H) | A • ≤ L} ⊆ {A ∈ S(H) | A F ≤ L}, then 2E sup A 0, A •≤L Tr A 1 n n i=1 σ i φ(x i ) ⊗ φ(x i ) (6.26) ≤ 2E sup A 0, A F ≤L Tr A 1 n n i=1 σ i φ(x i ) ⊗ φ(x i ) (6.27) ≤ 2E sup A 0, A F ≤L A F 1 n n i=1 σ i φ(x i ) ⊗ φ(x i ) F (6.28) ≤ 2L E 1 n n i=1 σ i φ(x i ) ⊗ φ(x i ) F . (6.29)
To conclude denote by ζ i the random variable

σ i φ(x i ) ⊗ φ(x i ). Then E 1 n n i=1 σ i φ(x i ) ⊗ φ(x i ) 2 F = E 1 n n i=1 ζ i F = E Tr 1 n n i=1 ζ i * 1 n n i=1 ζ i = E Tr 1 n 2 n i,j=1 ζ i ζ j .
By Jensen inequality, the concavity of the square root, and the linearity of the trace

E Tr 1 n 2 n i,j=1 ζ i ζ j ≤ E Tr 1 n 2 n i,j=1 ζ i ζ j = 1 n 2 n i,j=1
Tr(Eζ i ζ j ). Now note that for i ∈ {1, . . . , n}, we have

E σ i ζ i = 0, moreover Eσ 2 i = φ(x i ) 2 φ(x i ) ⊗ φ(x i
). Finally, given x 1 , . . . , x n , we have that ζ i is independent from ζ j , when i = j. Then when i = j we have Tr(Eζ

i ζ j ) = Tr((E σ i ζ i )(E σ j ζ j )) = 0. When i = j we have Tr(Eζ i ζ j ) = Tr(Eζ 2 i ) = φ(x i ) 4 . So 1 n 2 n i,j=1 Tr(Eζ i ζ j ) = 1 n 2 n i=1 φ(x i ) 4 .
From which we obtain the desired result.

Now we are ready to bound R n as follows

Proof Theorem 6.5. The proof is obtained by applying Theorem 6.12 and considering that φ(x) is uniformly bounded by c on X .

6.B .9 Proof of proposition 6.4

See Sec. 6.A for the basic technical assumptions on X , H and φ. In particular X is Polish and φ is continuous and uniformly bounded by a constant c.

Proof of proposition 6.4. In the following we will consider integrability and measurability with respect to a measure dx on X . In particular p : X → R is an integrable function on X with respect to the measure dx. Now define Ψ(x) = p(x)φ(x)φ(x) . We have that Ψ is measurable, since φ and p are measurable. Since p is integrable, p is finite almost everywhere, and hence Ψ(x) = p(x)φ(x)φ(x) is defined and trace class almost everywhere, and satisfies

Ψ(x) = |p(x)| φ(x) 2 H ≤ |p(x)|c 2 almost everywhere.
Since the space of trace class operators is separable, this shows that Ψ is Bochner integrable and thus that the operator W p = x∈X φ(x)φ(x) p(x)dx is well defined and trace class, with trace norm bounded by κ 2 p L 1 (X ) . Moreover, by linearity of the integral, for any A ∈ S(H),

Tr(AW p ) = X Tr(Aφ(x)φ(x) )p(x)dx = X f A (x)p(x)dx,
where the last equality follows from the definition of f A and the fact that

Tr(Aφ(x)φ(x) ) = Tr(φ(x) Aφ(x)) = φ(x) Aφ(x) = f A (x).
Remark 22 (Extension to more general linear functionals.). Note that the linearity of the model in A allows to generalize very easily the construction above to any linear functional that we want to apply to the model. This is especially true when the model has a finite dimensional representation as Eq. (6.7), i.e. f B = n ij=1 B i,j k(x, x i )k(x, x j ) with B 0. In this case, given a linear functional L : C(X ) → R, we have

L(f B ) = n i,j=1 B i,j L(k(x, x i )k(x, x j )) = Tr(BW L ),
where (W L ) i,j = L(k(x, x i )k(x, x j )) for i, j = 1, . . . , n.

6.B .10 Proof of proposition 6.5

In Sec. 6.B .10 and Sec. 6.B .11, we will use the following notations.

Let h, p ∈ N and H, H 1 , H 2 be separable Hilbert spaces.

• A = (A s ) 1≤s≤p ∈ S(H) p will denote a family of self-adjoint operators;

• Given a feature map φ : X → H and A = (A s ) 1≤s≤p ∈ S(H) p we will define the function

f A as follows ∀x ∈ X , f A (x) = (f As (x)) 1≤s≤p = φ(x) A s φ(x) 1≤s≤p ∈ R p , f A : X → R p • Given a matrix C ∈ R p×h which corresponds to a list of column vectors (c t ) 1≤t≤h ∈ (R p ) h , we define K C (H) := A = (A s ) 1≤s≤p ∈ S(H) p : p s=1 c t s A s 0, 1 ≤ t ≤ h • For any A = (A s ) 1≤s≤p ∈ S(
= (A s ) 1≤s≤p ∈ S(H) p , A ∈ K C (H) =⇒ ∀x ∈ X , f A (x) ∈ Y. Assume A ∈ K C (H) and let x ∈ X . By definition, f A (x) = (φ(x) A s φ(x)) 1≤s≤p ∈ R p . Hence, C f A (x) = p s=1 c t s φ(x) A s φ(x) 1≤t≤h = φ(x) p s=1 c t s A s φ(x) 1≤t≤h . Since A ∈ K C (H), for all 1 ≤ t ≤ h, it holds p s=1 c t s A s 0. In particular, this implies φ(x) p s=1 c t s A s φ(x) ≥ 0 for all 1 ≤ t ≤ h. Hence C f A (x) ≥ 0 =⇒ f A (x) ∈ Y.

6.B .11 Proof of Theorem 6.6

Using the notations of the previous section, the goal of this section is to solve a problem of the form inf (6.14) for given p, h ∈ N, C ∈ R p×h , separable Hilbert space H, feature map φ : X → H, regularizer Ω, loss function L : R n → R ∪ +∞ and x 1 , ..., x n ∈ X .

A∈K C (H) L(f A (x 1 ), ..., f A (x n )) + Ω(A),
We start by stating the form of the regularizers we will be using.

Assumption 6.2. Let p ∈ N. For any separable Hilbert space H and any A = (A s ) 1≤s≤p ∈ S(H) p , Ω is of the form

Ω(A) = p s=1 Ω s (A s ), Ω s (A s ) = λ s,1 A s + λ s,2 2 A s 2 F ,
where λ s,1 , λ s,2 ≥ 0 and λ s,1 + λ s,2 > 0.

Lemma 6.6 (Properties of Ω). Let Ω be a regularizer such that Ω satisfies Assumption 6.2.

Then Ω satisfies the following properties.

(i) For any separable Hilbert spaces H 1 , H 2 and any linear isometry O :

H 1 → H 2 , i.e., such that O * O = I H 1 , it holds ∀A ∈ S(H 1 ) p , Ω(OAO * ) = Ω(A).
(ii) For any separable Hilbert space H and any orthogonal projection Π ∈ S(H 1 ), i.e. satisfying Π = Π * , Π 2 = Π, it holds ∀A ∈ S(H) p , Ω(ΠAΠ) ≤ Ω(A).

6.B . PROOFS AND ADDITIONAL DISCUSSIONS

289

(iii) For any finite dimensional Hilbert space H n , taking ||A s || op to be the operator norm on H n ,

Ω is continuous, Ω(A) -→ sup s ||As||op→+∞ +∞ Proof. Note that since Ω(A) = p s=1 Ω s (A s ), Ω s (A s ) = λ s,1 A s + λ s,2 2 A s 2 F ,
where λ s,1 , λ s,2 ≥ 0 and λ s,1 + λ s,2 > 0, it is actually sufficient to prove the following result.

Let λ 1 , λ 2 ≥ 0 and assume

λ 1 + λ 2 > 0. Let for any A ∈ S(H), Ω(A) = λ 1 A + λ 2 2 A 2 F .
Then the following hold:

(i) For any separable Hilbert spaces H 1 , H 2 and any linear isometry O :

H 1 → H 2 , i.e., such that O * O = I H 1 , it holds ∀A ∈ S(H 1 ) p , Ω(OAO * ) = Ω(A).
(ii) For any separable Hilbert space H and any orthogonal projection Π ∈ S(H 1 ), i.e. satisfying Π = Π * , Π 2 = Π, it holds ∀A ∈ S(H) p , Ω(ΠAΠ) ≤ Ω(A).

(iii) For any finite dimensional Hilbert space

H n , Ω is continuous, Ω(A) -→ A op→+∞ +∞,
where we denote by • op the operatorial norm.

1. (i) has already been proven in Lemma 6.2.

Let us prove (ii)

. Let H be a separable Hilbert space, Π an orthogonal projection on H and A ∈ S(H).

Using the fact that B = sup C op≤1 Tr(BC), where C op denotes the operator norm on S(H), we have by property of the trace ΠAΠ = sup

C op≤1 Tr(ΠAΠC) = sup C op≤1 Tr(A(ΠCΠ)). Now since ΠCΠ op ≤ C op ≤ 1, it holds sup C op≤1 Tr(A(ΠCΠ)) ≤ sup C op≤1 Tr(AC) = A . Thus: ΠAΠ ≤ A .
Moreover, since Π I, it holds ΠAΠAΠ ΠA 2 Π. Hence, ΠAΠ 2 F = Tr(ΠAΠΠAΠ) ≤ Tr(ΠA 2 Π) Now using the fact that Tr(ΠA 2 Π) = Tr(AΠA), we can once again use the fact that Π I to show that AΠA A 2 and hence Tr(AΠA) ≤ Tr(A 2 ). Putting things together, we have shown

Tr(ΠAΠΠAΠ) ≤ Tr(A 2 ) =⇒ ΠAΠ 2 F ≤ A 2 F .
Thus, by summing the inequalities, Ω(ΠAΠ) ≤ Ω(A). CHAPTER 6. PSD MODELS Theorem 6.13. Let L be lower semi-continuous and bounded below, and Ω satisfying Assumption 6.2. Then Eq. (6.5) has a solution of the form

f * (x) = n i,j=1 [B s ] i,j k(x, x i )k(x, x j ) 1≤s≤p , for some family B = (B s ) 1≤s≤p ∈ K C (R n ).
Moreover, if L is convex, this solution is unique.

Proof of Theorem 6.13. The proof is completely analoguous to that of Theorem 6.7, combining Lemma 6.7 and proposition 6.9.

6.C Additional proofs

Lemma 6.8. Let X ⊂ R d , d ∈ N, be a compact set with Lipschitz boundary. Let m > d/2. Then W m 2 (X ) is a multiplication algebra (see definition 6.3).

Proof. When m ∈ N and m > d/2, then W m 2 (R d ) is a multiplication algebra [START_REF] Adams | Sobolev Spaces[END_REF]. When m / ∈ N, by Eq. 2.69 pag. 138 by [START_REF] Triebel | Theory of Function Spaces III[END_REF] we have that F m 2,2 (R d ) is a multiplication algebra when m > d/2, where F m 2,2 is the Triebel-Lizorkin space of smoothness m and order 2, 2 and corresponds to [START_REF] Triebel | Theory of Function Spaces III[END_REF].

W m 2 (R d ), i.e., F m 2,2 (R d ) = W m 2 (R d )
So far we have that m > d/2 implies that W m 2 (R d ) is a multiplication algebra, now we extend this result to W m 2 (X ). Note that since X is compact and with Lipschitz boundary, for any

f ∈ W m 2 (X ) there exists an extension f ∈ W m 2 (R d ) such that f | X = f and f W m 2 (R d ) ≤ C 1 f W m 2 (X )
with C 1 depending only on m, d, X (see Thm. 5.24 pag. 154 for m ∈ N and 7.69 when m / ∈ N pag. 256 by [START_REF] Adams | Sobolev Spaces[END_REF]). Then, since for any f : R d → R, by construction we have [START_REF] Adams | Sobolev Spaces[END_REF]. Then, for any f, g ∈ W m 2 (X ), denoting by f , g the extensions of f, g, we have

f | X W m (X ) ≤ f W m (R d )
f • g W m 2 (X ) = f | X • g| X W m 2 (X ) ≤ f • g W m 2 (R d ) (6.30) ≤ C f W m 2 (R d ) g W m 2 (R d ) ≤ CC 2 1 f W m 2 (X ) g W m 2 (X ) . (6.31) To conclude u : X → R that maps x → 1 has bounded norm corresponding to u 2 W m 2 (X ) = X dx. So W m
2 (X ) when m > d/2 and X is compact with Lipschitz boundary is a multiplication algebra.

6.D Additional details on the other models

Recall that the goal is to solve a problem of the form Eq. (6.1), i.e.

min f ∈F L(f (x 1 ), ..., f (x n )) + Ω(f ).
In this section, φ : X → H will always denote a feature map, k : X × X → R a positive semi definite kernel on X (k(x, x ) = φ(x) φ(x ) if k is the positive semi-definite kernel associated to φ). Given a kernel k, K ∈ R n×n will always denote the positive semi-definite kernel matrix with coefficients K i,j = k(x i , x j ), 1 ≤ i, j ≤ n.

Generalized linear models (GLM). Consider generalized linear models of the form, f w (x) = ψ(w φ(x)). Assume the regularizer is of the form Ω(f w ) = λ 2 w 2 . Using the representer theorem 6.D . ADDITIONAL DETAILS ON THE OTHER MODELS 293 by [START_REF] Cheney | A Course in Approximation Theory[END_REF], any solution to Eq. ( 6.1) is of the form w = n i=1 α i φ(x i ) and thus Eq. ( 6.1) becomes the following finite dimensional problem in α:

min α∈R n L(ψ(Kα)) + λ 2 α Kα. (6.32)
In the case where one wishes to learn a density function with respect to a basis measure ν, a common choice of model is functions of the form

p α (x) = exp(g(x)) x∈X exp(g(x))dν(x) , g(x) = n i=1 α i k(x i , x).
where k is a positive semi-definite kernel on X . The prototypical problem one solves to find the best p α is

min α∈R n L(p α (x 1 ), ..., p α (x n )) + λ 2 α Kα. (6.33)
In the specific case where the loss function is the negative log likelihood L(z 1 , ..., z n ) =

1 n n i=1 -log(z i ), it can be shown that Eq. ( 6.33) is convex in α.

In practice, we solve Eq. ( 6.32) by applying standard gradient descent with restarts, as the problem is non convex.

To solve Eq. (6.33), since the problem is convex, the algorithm is guaranteed to converge. However, since we can only estimate the quantity x∈X exp(g(x))dν(x); we do so by taking a measure ν from which we can sample. However, this becomes intractable as the dimension grows, as the experiments on density estimation will put into light.

Non-negative coefficients models (NCM). Recall the definition of an NCM. It represent non-negative functions as f α (x) = n i=1 α i k(x, x i ), with α 1 , . . . α n ≥ 0, given a kernel k(x, x ) ≥ 0 for any x, x ∈ X . In this case, the prototypical problem is of the form :

min α≥0 L(Kα) + λ 2 α Kα. (6.34)
If we are performing density estimation with respect to the measure ν, one wishes to impose X f α (x)dν(x) = 1, which can be seen as an affine constraint over α, since

X f α (x)dν(x) = u α, u = X k(x, x i )dν(x) 1≤i≤n ∈ R n .
In this case, the prototypical problem will be of the form If L is a convex smooth function, both problems Eq. ( 6.34) and Eq. ( 6.35) can be solved using projected gradient descent, since the projections on the set α ≥ 0 and the simplex α ∈ R n : α ≥ 0, u α = 1 can be computed in closed form.

min α≥0 u α=1 L(Kα) + λ 2 α Kα. ( 6
In the main paper, we mention that NCM models do not satisfy P2 i.e. that they cannot approximate any function arbitrarily well. We implement Example 6.2 in the following way. Let g(x) = e -x 2 /2 . Take k(x, x ) = e -x-x 2 , n points (x 1 , ..., x n ) taken uniformly in the interval [-5, 5]. To find the function f α which best approximates g, we perform least squares regression, i.e. solve the prototypical problem Eq. ( 6.34) with the square loss function

L(y) = 1 2n n i=1 |y i -g(x i )| 2 .
We perform cross validation to select the value of λ for each value of n. In Fig. 6.2, we show the obtained function f α for n = 100, 1000, 10000. This clearly illustrates that with this model, we cannot approximate g in a good way, no matter how many points n we have.

Partially non-negative linear models (PNM). Consider partially non negative models of the form f w (x) = w φ(x), with w ∈ {w ∈ H | w φ(x 1 ) ≥ 0, . . . , w φ(x n ) ≥ 0} (that is we impose f w (x i ) ≥ 0). Take Ω to be of the form λ 2 w 2 in Eq. (6.1). Using the representer theorem by [START_REF] Cheney | A Course in Approximation Theory[END_REF], we can show that there is a solution of this problem of the form f α = n i=1 α i k(x, x i ), leading to the following optimization problem in α to recover the optimal solution:

min Kα≥0 L(Kα) + λ 2 α Kα (6.36)
If we want to impose that the resulting f α sums to one for a given measure ν on X , we proceed as in Eq. ( 6.35) and solve

min Kα≥0 u α=1 L(Kα) + λ 2 α Kα. (6.37)
However, there is no guarantee that the resulting f α will be a density, as will be made clear in the next section on density estimation.

In the experiments, we solve Eq. ( 6.36) and Eq. (6.37) in the following way. We first compute a cholesky factor of K : K = V V. Changing variables by setting Vβ = α, the objective functions become strongly convex in β. We then compute the dual of these problems and apply a proximal algorithm like FISTA, since the proximal operator of L is always known in our experiments.

6.E . ADDITIONAL DETAILS ON THE EXPERIMENTS 295

6.E Additional details on the experiments

In this section, we provide additional details on the experiments. The code will be available online. Recall that we consider four different models for functions with non-negative outputs : GLM, PNM, NCM and our model.

Kernels.

All the models we consider depend on certain positive semi definite kernels k. In all the experiments, we have taken the kernels to be Gaussian kernels with width σ:

∀x, x ∈ R d , k(x, x ) = exp - x -x 2 2σ 2 .
Regularizers. For GLM, PNM and NCM, the regularizer for the underlying linear models are always of the form λ 2 w 2 where w is the parameter of the linear model, which translates to λ 2 α Kα where the α are the coefficients of the finite dimensional representation. For our model, we always take the regularizer to be of the form λ A + 0.05 A 2 F .

Parameter selection. In all experiments except for the one on density estimation in the main paper (in which we fix σ = 1 and select λ), we select the parameters σ of the kernels involved as well as the parameters λ for the regularizers using K fold cross validation with K = 7. This means that once the data set has been generated, we randomly divide it into two sets : the training set containing 70% of the data and the test set containing 30% of the data. We then train our model for the given σ, λ and report the performance on the test set. We repeat this operation K = 7 times and consider the mean performance on the test set to be a good indicator of the performance of our model for a given set of parameters. We then select the best parameters by doing a grid search. The code for this cross-validation will be available online.

Formulations and algorithms. The formulations of our three problems : density estimation, regression with Gaussian heteroscedastic errors, and multiple quantile regression, have been expressed in the main paper in a generic way involving functions with unconstrained outputs, and functions with outputs constrained to be non negative and sometimes summing to one. We always model functions with unconstrained outputs with a linear model with gaussian kernel, and model the functions with constrained outputs with the four models for non-negative functions we consider: ours, PNM, GLM and NCM.

In practice, we implement the methods PNM, GLM and NCM as explained in Sec. 6.D . In particular, we use FISTA for PNM, and our model, dualizing the equality constraints for density estimation. This relies on the fact that the proximal operators of the log likelihood, the objective function for heteroscedastic regression as well as the pinball loss can be computed in closed form, and that the regularization is smooth in the right coordinates.

Details on the experiments of the main text. Here, we add a few precisions on the toy distributions we have used to sample data and the number of sampled used when not specified in the main text.

• For heteroscedastic regression, the data was generated as the toy data in section 5 by [START_REF] Quoc | Heteroscedastic gaussian process regression[END_REF], with n = 80 points. • For quantile regression, the data points (x i , y i ) were generated according to the following distribution for (X, Y ) :

X ∼ 1 2 U (0, 1/3) + 1 2 U (2/3, 1) and Y |x ∼ N (0, σ(x))
where

σ(x) =      -x + 1/3 for 0 ≤ x ≤ 1/3 x -2/3 for 2/3 ≤ x ≤ 1 0 otherwise .
.

Here, U stands for the uniform distribution. Moreover, in order to perform the experiments in the main paper, we have used 500 sample points.

Density estimation in dimension 10 with n = 1000. In this paragraph, we consider the following experiment. Let d = 10, X ∈ R d be a random variable distributed as a mixture of Gaussians :

X ∼ 1 2 N (-2e 1 , 1/ √ 2πI d ) + 1 2 N (2e 1 , 1/ √ 2πI d )
where e 1 is the first vector of the canonical basis of R d .

Let n = 1000 and let (x 1 , ..., x n ) be n iid samples of X. We perform the four different methods, cross validating both the regularization parameter λ and the kernel parameter σ at each time.

We learn the density in the form

p(x) = f (x)ν(x),
ν is the density associated with N (0, 5I d ).

We then use our models for densities to compute the best f in its class using the negative log-likelihood as a loss function. It is crucial that we can sample from ν in order to approximate the integral in the case of GLMs.

In order to visualize the results of the different algorithms in Fig. 6.3, we compute the learnt distribution p, and then sample randomly n 0 = 500 points from a uniform distribution on the box centered at 0 and of width 5 in order to explore regions where the density is close to zero, n 0 points sampled from the true distribution of the data, in order to explore points where the density is representative, and n 0 points on the line [-4, 4] × {0} d-1 where the density is at its highest. We then project onto the first coordinate, i.e. given a point x = (x i ) 1≤i≤d and the associated predicted density p(x), we plot the point (x 1 , p(x)). Note that for readability, we have used the same scale for our model and the PNM, and a smaller scale for the two others since the learnt density is much flatter.

Let us now analyse the results in Fig. 6.3. Note that in terms of performance, i.e. log likelihood on the test set, the first two models (PNM and our model) are quite close and are better than the two others.

6.F . RELATION TO SIMILAR WORK

• PNM. As in d = 1 we see that for d = 10 the problems of non-negativity for PNM are exacerbated, making it not suitable to learn a probability distribution. Indeed there are low density regions where the optimization problem pushes the model to be negative. Since by constraint we have f dν = 1, the volume of the negative regions is used to push up the function in the regions with high density. So |f |dν 1, while it should be |f |dν = 1. This is confirmed by the behavior of the cross validation.

• Our model Our model seems to perform reasonably well.

• NCM. This problem is particularly difficult for NCM. Indeed, as the width of the kernel decreases, the model is unable to learn since it overfits in the direction e 1 and it would require way more points than n = 1000. However, as soon as the width of the kernel is good for e 1 , the learnt distribution becomes too heavy tailed in the direction orthogonal to e 1 .

• GLM. It is interesting to note that GLM completely fails, because the measure ν which we take as a reference measure has a support which has only double variance compared to p, but in 10 dimensions it corresponds to a support with way larger volume compared to the one of the target distribution. In particular, the estimation of the integral, which was possible in d = 1 with 10000 i.i.d. points from ν, in 10 dimensions becomes almost impossible (it would require way more sampling points). Note that we sample the points from ν to simulate the real-world situation where p is a measure from which it is difficult to sample from, while ν is an simple measure to sample from which contains the support of p. Further experiments show that if one takes the target distribution to sample, one obtains a good model, which reassures us in the fact that this is not a coding error but a real phenomenon.

6.F Relationship to the work by Bagnell and Farahmand (2015)

As mentioned in the main paper, the model in Eq. ( 6.4) has already been considered by [START_REF] Bagnell | Learning positive functions in a Hilbert space[END_REF] with a similar goal as ours. This paper is a workshop publication that has only be lightly peer-reviewed and contains fundamental flaws. In particular, they provide an incorrect characterization of the solution of Eq. (6.5), that limits the representation power of the model to the one of non-negative coefficients models, that, as we have seen in Sec. 6.2 .1 and in Example 6.2, has poor approximation properties and cannot be universal. This severe limitation affects also the optimization framework (which also only relies on general-purpose toolboxes such as CVX (http://cvxr.com/cvx/), which are not scalable to large n).

Indeed, in their main result, the representer theorem incorrectly characterizes A * the solution of Eq. (6.5) as

A * ∈ R n ∩ S(H) + , R n = n i=1 α i φ(x i ) ⊗ φ(x i ) | α ∈ R n ,
and S(H) + = {A ∈ S(H) | A 0}. Note, however that R n ⊆ S(H n ) ⊂ S(H) by construction, where H n = span{φ(x 1 ), . . . , φ(x n )}. So their characterization corresponds to

A * ∈ A = n i=1 α i φ(x i ) ⊗ φ(x i ) | α ∈ R n , A 0 .
Now, for simplicity, consider the interesting case where φ is universal and x 1 , . . . , x n are distinct points. Then (φ(x i )) n i=1 forms a basis for H n and the only α 1 , . . . , α n ∈ R that guarantee A 0 CHAPTER 6. PSD MODELS are α 1 ≥ 0, . . . , α n ≥ 0, i.e.,

R n = A = n i=1 α i φ(x i ) ⊗ φ(x i ) | α 1 ≥ 0, . . . , α n ≥ 0 .
Note that this class of operators leads only to non-negative coefficients models. Indeed, let A ∈ R n and denote by k(x, x ) the function k(x, x ) = (φ(x) φ(x )) 2 , then

f A (x) = φ(x) Aφ(x) = n i=1 α i (φ(x) φ(x i )) 2 = n i=1 α i k(x, x i ), ∀ x ∈ X .
Since k is a kernel (it is an integer power of φ(x) φ(x ) that is a kernel [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF]) and α 1 ≥ 0, . . . , α n ≥ 0, then f A belongs to the non-negative coefficients models.

Instead, we know by our Theorem 6.1 that A * ∈ S(H n ) + and more explicitly, by Theorem 6.2 that A * , the solution of Eq. (6.5) is characterized by the non-positive part operator of a symmetric matrix [•] + . By Theorem 6.3 we already know that our model is universal while NCM is not and thus the characterization by [START_REF] Bagnell | Learning positive functions in a Hilbert space[END_REF] cannot be universal.

300

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS Moreover, they are hard to adapt to probabilities which are known up to their renormalization constant, which is often the case when dealing with exponential models that are common in applications [START_REF] Robert | Monte Carlo statistical methods[END_REF].

More involved methods have been developed to address these dimensionality and renormalization issues, in the class of so-called Markov chain Monte Carlo (MCMC) methods. However, they are complex to set up: in particular, independence between samples is not directly guaranteed, convergence can be slow and hard to measure non-asymptotically [START_REF] Lelièvre | Free Energy Computations[END_REF][START_REF] Robert | Monte Carlo statistical methods[END_REF].

In this work, we address the problem in a different way, by incorporating a modeling step. Instead of sampling directly from the target density, we first model this density using a positive semi-definite (PSD) model [START_REF] Rudi | Finding global minima via kernel approximations[END_REF][START_REF] Rudi | Psd representations for effective probability models[END_REF], and then sample from this PSD model.

PSD models have been introduced by Marteau-Ferey, [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] and their relevance for modeling probability distributions has been further established by [START_REF] Rudi | Psd representations for effective probability models[END_REF], showing that i) they are stable under key operations for probabilistic inference, such as marginalization, integration (also called "sum-rule"), and product, which can be done efficiently in practice, and ii) they concisely approximate a large class of probability distributions. We present these models in Sec. 7.2 . Building on this work, we show that these models are also relevant in the context of sampling, making the following main contributions.

(1) In Sec. 7.3 , we derive an algorithm that is easy to implement and which can generate an arbitrary number of i.i.d. samples from a given PSD model, with any given precision. This answers one of the open questions outlined by [START_REF] Rudi | Psd representations for effective probability models[END_REF] and shows that one can indeed efficiently sample from a PSD model.

(2) In Sec. 7.4 we show that we can sample an arbitrary number of i.i.d. samples from a target probability distribution that is regular enough, with any given precision. The algorithm consists in (a) approximating the un-normalized density p via a PSD model, using evaluations of p, and (b) extracting i.i.d. samples from the PSD model. We show that for sufficiently regular densities the resulting PSD model is concise and avoids the curse of dimensionality: to achieve error ε, the PSD model requires a number of parameters and a number of evaluations of p that are in the order ε -2-d/β , where d is the dimension of the space and β is the order of differentiability of the density. For regular probabilities, i.e., when β ≥ d, the rate does not depend exponentially on d and is bounded by O(ε -3 ) (the constant term instead may depend exponentially on d).

In Sec. 7.5 , we also present numerical simulations which demonstrate the quality of both our sampling technique and approximation results.

Backround on Positive Semi-Definite (PSD) models

Denote by R d ++ the vectors of R d with positive components and S m + the set of positive semi-definite m by m matrices. Following [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]; [START_REF] Rudi | Psd representations for effective probability models[END_REF], a Gaussian PSD model is parametrized by a triplet (A, X, η) ∈ S m + × R m×d × R d ++ , and is defined for any x ∈ R d as

f (x; A, X, η) = m i,j=1 A ij k η (x, x i )k η (x, x j ), (7.1)
where, with diag(η) being the diagonal matrix with diagonal η, k η (x, x ) = e -(x-x ) diag(η)(x-x ) is the Gaussian kernel of parameter η , X ∈ R n×d is the matrix whose rows corresponds to the 7.2 . BACKROUND ON POSITIVE SEMI-DEFINITE (PSD) MODELS 301 centers x 1 , . . . , x n of the Gaussian PSD model, and A is a matrix of coefficients which is positive semi-definite, to guarantee the non-negativity of f . Note that when A = aa , a ∈ R m , is a rank-1 operator, a Gaussian PSD model is simply the square of a linear model f (x; A, X, η) = g(x; a, X, η) 2 of the form,

g(x; a, X, η) = m i=1 a i k η (x, x i ), (7.2)
for any x ∈ R d . This particular case of PSD model will appear when approximating an arbitrary probability density p in Sec. 7.4 .2.

.1 Main properties of PSD models

As explained in the introduction, PSD models show properties that make them particularly well suited to model non-negative functions and probability distributions. Such properties are analyzed by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] and [START_REF] Rudi | Psd representations for effective probability models[END_REF], here we recall the ones that are important for our purpose.

Non-negativity. Since A is positive semidefinite, then the PSD model f (

x; A, X, η) satisfies f (x; A, X, η) ≥ 0 for all x ∈ R d .
Preservation of convex functionals. Using the PSD model to represent non-negative functions in a problem of the form min f ≥0 L(f ), where L is a convex functional, leads to a convex problem min A∈S + (R m ) L(f (•; A, X, η)). Indeed, the constraint A ∈ S + (R m ) is convex, the PSD model f (•; A, X, η) is linear in the parameter matrix A and a composition of a convex function L with a linear function is convex. This allows, e.g., to perform empirical risk minimization for the square and logarithmic losses.

Conciseness of the representation. under mild conditions, recalled in Assumption 7.1, a PSD model can approximate a probability density that is β-times differentiable with error ε, using a number of centers m = O(ε -d/β ) (which is minimax optimal). [START_REF] Rudi | Psd representations for effective probability models[END_REF] provide also an algorithm to learn the PSD model given i.i.d. samples from the probability. However, we cannot use this result in our context since we do not assume to have samples from our density.

Integration over hyper-rectangles in closed form. As integration of PSD models will play a key role in the algorithm developed for sampling in Sec. 7.3 , both for theoretical an computational reasons, we develop this integration aspect in greater detail. For X ∈ R m×d and η ∈ R d ++ , we denote with K X,η ∈ R m×m the kernel matrix such that [K X,η ] ij = k η (x i , x j ). The integral of a PSD model in Eq. ( 7.1) over a hyper-rectangle can be expressed with simple matrices, leveraging the fact that for any pair (x i , x j ), it holds

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS k η (x, x i )k η (x, x j ) = k η/2 (x i , x j )k 2η (x, (x i + x j )/2). Then we have I(Q; A, X, η) := Q f (x; A, X, η) dx = m i,j=1 A ij k η 2 (x i , x j ) Q k 2η (x, x i +x j 2 ) dx = m i,j=1 A ij [K X,η/2 ] ij [G X,2η,Q ] ij , (7.3) where [G X,η,Q ] ij = Q ij k η (x, 0) dx, and Q ij = Q -(x i + x j )/2
. These integrals can be computed by 2d calls to the erf function, as, for any i, j ∈ {1, ..., m}:

[G X,η,Q ] ij = c η d k=1 [erf( √ η k B ijk ) -erf( √ η k A ijk )], (7.4)
where c η = (π/4) d/2 det diag(η) -1/2 , A, B ∈ R d×m×m , A is the tensor of bottom left corners and B is the tensor of top right corners, defined formally from the means tensor (7.5) This shows that, for any hyper-rectangle Q, we can compute G X,η,Q with exactly 2dm 2 calls to the erf function and dm 2 arithmetic operations (so there is no dependence on the dimension of the hyper-rectangle).

X ijk = 1 2 (X ik + X jk ) as A ijk = a k -X ijk , B ijk = b k -X ijk .

A sampling algorithm for PSD models

In this section, we fix a Gaussian PSD model on R d parametrized by (A, X, η) ∈ S m + ×R m×d ×R d ++ for a given m ∈ N. To simplify notations, we will omit the parameters of the PSD model using f (x) as a shorthand for f (x; A, X, η) and I(Q) as a shorthand of I(Q) = I(Q; A, X, η).

Given a bounded hyper-rectangle Q (see Sec. 7.3 .1), denote by p Q the function (7.6) where 1 Q (x) = 1 when x ∈ Q and 0 otherwise. In Sec. 7.3 .2, we explain that even in the case of an infinite hyper-rectangle (e.g., Q = R d ), we can easily find a finite hyper-rectangle Q on which the whole mass of f is essentially concentrated, and thus approximately sample in this case as well. We end this section with a discussion on the main elements needed to sample, and which could allow to generalize this approach to PSD models with different kernels.

p Q (x) = f (x)1 Q (x)/I(Q),

.1 A sampling algorithm on a finite hyper-rectangle

Given the function f , the algorithm will take three inputs (Q, N, ρ): the hyper-rectangle Q (with sides parallel to the axes) from which we would like to sample, the number of i.i.d. samples N which we would like to obtain, and a parameter ρ which defines the quality of the approximation of p Q from which the algorithm generates samples. The effect of ρ on the precision of the algorithm is formally established in Theorem 7.2.

We start with the case N = 1. Starting from Q, we cut Q in half in its longest direction forming two sub-rectangles

Q 1 , Q 2 . If X Q were a random variable following the law of p Q , then X Q ∈ Q i with probability p i = I(Q i )/I(Q), and X Q |{X Q ∈ Q i } follows the law of p Q i .
Therefore, when looking for a sample from p Q , we randomly choose with probability p i one of the two smaller sub-rectangles Q i in which to look for the sample and then call the algorithm recursively to get a sample from p Q i . Of course, we need a stopping criterion: when the maximal side of Q has length smaller than ρ then we stop and we return a point sampled uniformly at random in Q.

The complete algorithm is presented in algorithm 2 and is explained below.

Details for algorithm 2. In line 1, we define the recursive function SampleRec which will generate samples recursively. The main algorithm Sample in line 15 simply calls the function SampleRec and randomly reshuffles the samples in order to guarantee independence (see RandomPerm line 17). In line 4, the function MaxLen applied to Q returns the maximum of the lengths of the sides of Q; the condition can therefore be translated as "if all sides of Q are smaller than ρ". If it is the case, in line 5, we return N i.i.d. samples from the uniform distribution on Q using SampleUniform. If it is not, in line line 7 we cut the hyper-rectangle Q in half along its largest side with minimal index (i.e., along side k = min arg max (b i -a i )), yielding two sub hyper-rectangles Q 1 , Q 2 . This is the purpose of the function SplitLargestSide.

In line 8, we compute the probability q that a given sample from p Q belongs to Q 1 using the fact that we can integrate the PSD model exactly. Since we have to generate N samples, we will select k of them from Q 1 and Nk from Q 2 where k is a sample from a binomial law of paramter q: this is the purpose of the function SampleBinomial and line 9. We then call the algorithm recursively to generate the k samples from Q 1 using p Q 1 and the Nk samples from Q 2 from p Q 2 (lines 10 and 11).

Algorithm 2 Approximately sampling from p Q 1: function SampleRec(Q, N, ρ)

2:

if N = 0 then 3:

return EmptyList 4:

else if MaxLen(Q) ≤ ρ then 5:
return SampleUniform(Q, N ) 6:

else 7: Q 1 , Q 2 = SplitLargestSide(Q) 8: q = I(Q 1 )/I(Q) 9:
k = SampleBinomial(N, q) 10:

L 1 = SampleRec(Q 1 , k, ρ) 11: L 2 = SampleRec(Q 2 , N -k, ρ) 12: return Concatenate(L 1 , L 2 ) 13:
end if 14: end function

15: function Sample(Q, N, ρ) 16: L = SampleRec(Q, n, ρ) 17:
return RandomPerm(L) 18: end function Guarantees of the algorithm. Given (Q, N, ρ), algorithm 2 does not sample N i.i.d. samples from the exact distribution p Q but rather from an approximation p Q,ρ of p Q , controlled by the parameter ρ. More formally, let D Q,ρ be the set of dyadic sub-rectangles of Q with largest possible size smaller than ρ (see Appendix 7.D for a formal definition). Our algorithm will CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS effectively sample from a piece-wise constant approximation of p on the elements of D Q,ρ : Note that the theorem gives us that the complexity is essentially O(N m 2 d 2 log(1/ρ)). This quadratic dependence in the dimension d is verified in practice and the slicing procedure does not yield any time or computational difficulties. Note however that in our two step procedure detailed in the next section, the number m will a priori depend on the dimension, but this is confined to the learning phase; once the m centers are set, the complexity is quadratic. Moreover, note that we verify the claim that computing integrals is the computational bottleneck in practice in Sec. 7.D .4.

p Q,ρ = 1 I(Q) Qρ∈D Q,ρ I(Qρ) |Qρ| 1 Qρ , ( 7
Approximation error of the algorithm. Since by Theorem 7.1, the algorithm does not generate samples exactly from p Q but rather from the piecewise constant approximation p Q,ρ defined in Eq. (7.7), it is necessary to quantify the distance between p Q and its approximation p Q,ρ . We do so in Theorem 7.2 for three different distances.

The weakest distance will be the Wasserstein-1 distance (also called earth mover's distance) [START_REF] Santambrogio | Optimal Transport for Applied Mathematicians[END_REF]. It quantifies the discrepancies in the allocation of mass between two distributions, and is defined as (7.8) where Lip(f ) is the Lipschitz constant of f for the Euclidean norm. It is structurally the most adapted to the approximation p Q,ρ since on each hyper-rectangle of D Q,ρ , p Q,ρ has the same mass as p Q but distributes it uniformly. Hence, the discrepancy in mass allocation will be confined to small hyper-rectangles whose sides are of size at most ρ.

W 1 (p 1 , p 2 ) = sup Lip(f )≤1 X f (x)(p 1 (x) -p 2 (x))dx ,
We will also use two stronger distances: the total variation (TV) distance

d T V (p 1 , p 2 ) = p 1 - p 2 L 1 (X )
, and the Hellinger distance

H(p 1 , p 2 ) = √ p 1 - √ p 2 L 2 (X )
, which is particularly relevant for exponential models [START_REF] Lo | Asymptotics in Statistics: Some Basic Concepts[END_REF], and, in our paper, when using rank-1 PSD models (see Sec. 7.4 .2). These distances will naturally appear in Sec. 7.4 to quantify the discrepancy between a given probability density and its approximation as a Gaussian PSD model. For more details on these distances, see Appendix 7.A .2. Theorem 7.2 provides bounds on these distances between the target density p Q = f 1 Q /I(Q) and p Q,ρ as a function of ρ, and some Lipschitz constant (where Lip ∞ (g) denotes the Lipschitz constant of g for the norm x ∞ = sup |x i |). A more general theorem is proved in Appendix 7.D .3 as Theorem 7.7.

Theorem 7.2 (Variation bounds). Let Q be a hyper-rectangle, ρ > 0, p Q = f 1 Q /I(Q) and p Q,ρ defined in Eq. (7.7). It holds: Selection of ρ. In Fig. 7.1, we observe the effect of ρ on the quality of sampling, when sampling from a PSD model whose distribution is illustrated by the heat map defined on the top left figure. We highlight the fact that decreasing ρ corresponds to refining the dyadic decomposition of the hyper-rectangle and hence sampling more precisely. In practice, one can therefore choose ρ manually (for instance ρ = 10 -4 , 10 -6 ) and have an upper bound on the distance between p Q,ρ and p Q from Theorem 7.2. If one wishes to select ρ in a more principled way to bound the total variation or Hellinger distance, this can also be done using only accessible quantities. If f is a PSD model with parameters (A, X, η) for η = τ 1 d , and K is a shorthand for K X,η , the Lipschitz constants can be bounded using only τ , K and A (or a s.t. A = aa in the case of a rank one

H(p Q , p Q,ρ ) ≤ |Q| I(Q) Lip ∞ ( f ) ρ (7.9) d T V (p Q , p Q,ρ ) ≤ |Q| I(Q) Lip ∞ (f )ρ (7.10) W 1 (p Q , p Q,ρ ) ≤ √ dρ. ( 7

Sampling from arbitrary distributions using PSD models

The previous section provides an algorithm to approximately sample from a distribution in the form of a PSD model. In this section, we show how to leverage that fact to be able to generate N approximate i.i.d. samples from a very general class of probability distributions on a hyper-rectangle X ⊂ R d . The strategy is simple : a) approximate the target distribution p with a PSD model p, and b) approximately sample from the PSD model p using the algorithm presented in Sec. 7.3 . The main challenge is to quantify the distance between the target distribution p and its approximation p as a PSD model.

Approaching a distribution by a PSD model by accessing the distribution through samples has been done in Sec. 3. by [START_REF] Rudi | Psd representations for effective probability models[END_REF]. Instead, in this work, we access the distribution through function evaluations, as our goal is to be able to generate samples. However, a similar algorithm can be implemented to learn a PSD model from function evaluations. Moreover, it can be analysed under the same conditions (see Assumption 7.1 and Sec. 7.4 .1). This algorithm is based on the solving of a semi-definite program to find the matrix A to form a good approximation f (x; A, X m , η) of the density p. In Sec. 7.4 .2, we instead learn a rank-one PSD model, solving a least-squares problem (and not a semi-definite program) using tools by [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]; [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF]; Meanti, Carratino, Rosasco, and Rudi (2020). This algorithm, faster than the one based on the solving of a semi-definite program, requires a stronger assumption to be analysed, and is naturally adapted to densities of the form p(x) ∝ e -V (x) .

Main hyper-parameters. The two methods presented in this section (see Sec. 7.4 .1 and Sec. 7.4 .2) will have hyper-parameters n, m, τ, λ, ρ.

The parameters n and m are integer; moreover, we will take two sequences of i.i.d. samples uniformly from X : x 1 , ..., x n represented by X ∈ R n×d and x 1 , ..., x m represented by X m ∈ R m×d . We will use an isotropic η = τ 1 d in the Gaussian linear and PSD models for a strictly positive τ .

To simplify notation, take K mm := K Xm,η and K nm := K X, Xm,η . The parameter λ will always be a strictly positive real number.

The parameters m and τ will define the PSD model: m will control the number of points, also called Nyström centers, which we use to represent our PSD model (as n and m increase, the quality of the approximation increases); and τ will control the width of the Gaussian kernel. The parameter n and λ control the learning phase of the algorithm, i.e., the approximation of p by a PSD model. n is the number of points at which we evaluate our probability density to estimate it; λ will control the strength of the regularization. Finally, ρ will control the scale at which we apply algorithm 2.

.1 A general method

In this section, we present a method to approximately sample from the density by a) approximating it by a PSD model solving a semi-definite program (SDP) and b) use algorithm 2 to sample from that PSD model. More precisely, we assume that p is known up to a constant, i.e., that we have a function f p which is proportional to p which we can evaluate.

Step a): approximation of p. To fit a PSD model to p, we use an method similar to the one presented in Section 3 of [START_REF] Rudi | Psd representations for effective probability models[END_REF], and construct a Gaussian PSD model

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS f = f (•; A, X m , η)
, where A ∈ S m + is the solution to the empirical semi-definite problem (7.15) where f (x; A) := f (x; A, X m , η). This problem is a quadratic problem in A and can be solved in polynomial time in m using semi-definite programming. We then define Z = X f (x) dx which can be computed in closed form as the integral over a hyper-cube of a PSD model, and p = f / Z, which is our approximation of p.

A = arg min A∈S m + X f (x; A) 2 dx -2 n i=1 f p (x i )f (x i ; A) + λ K 1/2 mm AK 1/2 mm 2 F ,
Problem Eq. ( 7.15) can be seen as a variation of empirical risk minimization for the square loss, with an additional regularization term λ K

1/2 mm AK 1/2
mm F which is the equivalent of the classical kernel regularization term in the setting of PSD models. Indeed, the function of A being minimized is a proxy of f (

•; A) -f p (•) 2 L 2 (X ) = f (•; A) 2 L 2 (X ) + X f p (x)f (x; a) dx + C.
In Eq. (7.15), X f p (x)f (x; a) dx is approximated by its empirical version, using uniform samples X = (x 1 , ..., x n ) (plus the regularization term). The first term f (•; A) 2 L 2 (X ) is kept as such as it is a quadratic function of A which can be explicitly computed, using the same techniques as those to compute integrals of PSD models, and described by [START_REF] Rudi | Psd representations for effective probability models[END_REF]. Note that here, X, X m , τ, λ are hyper-parameters; n and m will be taken as large as possible with a given computational budget, and λ and τ can be selected by validation on a newly generated test data set (since we assume we can generate samples from X ).

Step b): sampling from the approximation p. We apply algorithm 2 to p with a parameter ρ and on the hyper-rectangle X . We denote with p sample the density p X ,ρ given by Eq. (7.7), from which algorithm 2 effectively samples N i.i.d. samples by Theorem 7.1. This two step strategy is detailed in algorithm 4. SolveSDP simply solves Eq. (7.15).

Algorithm 4 Approximately sampling from p using a SDP Input p, X , N Hyper-parameters (approximation) n, m, τ, λ Hyper-parameters (sampling) ρ Output N approximate samples from p| X 1: function ApproximateSamples(p, X , N, n, m, τ, λ, ρ) 2:

X n = UniformSamples(n, X ) 3: X m =UniformSamples(m, X ) 4: A = SolveSDP(p, X n , X m , τ, λ) 5: p(•) = f (• | A, X m , τ ) 6: X N = Sample(X , N, ρ) from p 7:
return X N 8: end function Theoretical analysis. Recall that p is the target density, proportional to f p and that p is the approximation of p obtained by solving Eq. ( 7.15) and p sample is the distribution from which we effectively sample when applying algorithm 2 to p. In proposition 7.1 and Theorem 7.3, we show that under certain regularity assumptions on p, given ε > 0, we can find hyper-parameters n, m, τ, λ and ρ such that d T V (p, p sample ) ≤ Cε, i.e. that algorithm 4 generates N i.i.d. samples from a distribution Cε close to p.

For simplicity, we will assume X = (-1, 1) d , as is done by [START_REF] Rudi | Psd representations for effective probability models[END_REF]. In principle, we could approximate p on any bounded domain X from which we can sample uniformly, and still obtain analogous results. In that case, we would apply algorithm 2 on a hyper-rectangle containing the domain, and reject a sample outside of it. Our main assumption on p will be that p can be written as a sum of squares of functions belonging to the space W β (X ) = W β 2 (X ) ∩ L ∞ (X ) which is the space of bounded functions whose derivatives of order less or equal to β are square integrable, and which can be equipped with the norm

• W β (X ) = • W β 2 (X ) + • L ∞ (X ) (see Appendix 7.
A .1 for more precise definitions). The key quantities here are the dimension d and the regularity of the density β. This summarized in the following assumption.

Assumption 7.1 (Sum of squares distribution). There exists J ∈ N and functions q 1 , ..., q J belonging to W β (X ) such that p = J j=1 q 2 j . Moreover, we have access to p only through function evaluations of the form f p (x) where f p ≥ 0 is given, is proportional to p, and where the proportionality constant is unknown. We define p sos,X ,β = inf

J j=1 q j 2 W β (X )
where the infimum is taken over all such decompositions of p.

The approximation properties of p w.r.t. p are bounded in total variation distance in the following proposition, proved as proposition 7.10 in Appendix 7.E .

Proposition 7.1 (Performance of p). There exist constants ε 0 > 0 depending only on d, β, and p sos,X ,β and C 1 , C 1 , C 2 , C 3 depending only on d, β such that the following holds. Let δ ∈ (0, 1] and ε ≤ ε 0 , and assume n and m satisfy

m ≥ C 1 ε -d/β log d C 2 ε log C 3 εδ , (7.16) n ≥ ε -2-d/β log d 1 ε log 2 δ .
(7.17)

Let λ = ε 2+2d/β and τ = ε -2/β . With probability at least 1 -2δ, it holds

d T V ( p, p) ≤ C 1 p sos,X ,β ε. (7.18)
The key takeaway from this proposition is that the number of samples n, m needed to perform the first step of the algorithm (approximation) is polynomial in the quantities O(ε -1 ), O(ε -d/β ), thus leveraging the regularity β of p. When this is the case, we can find λ, τ such that the distance d(p, p) is of order ε. We provide a choice for ρ for the second step of the algorithm (sampling), in order to guarantee a bound for the total variation distance between the sampling distribution and the original distribution in the following theorem. It is proved as Theorem 7.8 in Appendix 7.E . In particular, it bounds the total complexity of the algorithm in terms of erf computations, as a function of N and the desired error ε.

Theorem 7.3 (Performance of p sample ). Under the assumptions and notations of proposition 7.1, there exists a constant C 2 depending only on d, β, such that the following holds. If ρ is set either as ε 1+(d+1)/β or adaptively as ρ T V ε , then with probability at least 1 -2δ,

d T V (p, p sample ) ≤ C 2 p sos,X ,β ε. (7.19)
Moreover, the adaptive ρ T V ε is lower bounded by ε 1+(d+1)/β /(C 3 p sos,X ,β ). In both cases, this guarantees that the complexity in terms of erf computations is of order O(N m 2 log(1/ρ)), which in terms of ε yields O N ε -2d/β log 2d+1 1 ε log 2 1 δε , where the O notations is taken with constants depending on d, β, p sos,X ,β .

Algorithm 5 Sampling from p using a rank-1 model Input p, X , N Hyper-parameters (approximation) n, m, τ, λ Hyper-parameters (sampling) ρ Output N approximate samples from p| X 1: function ApproximateSamples(p, X , N, n, m, τ, λ, ρ) 2:

X n = UniformSamples(n, X ) 3: X m =UniformSamples(m, X ) 4: A = SolveHellinger(p, X n , X m , τ, λ) 5: p(•) = f (• | A, X m , τ ) 6: X N = Sample(X , N, ρ) from p 7: return X N 8: end function
Note that this assumption is satisfied if p ∝ e -V (x) for a potential V which is β times continuously differentiable which we can evaluate.

In proposition 7.2 and Theorem 7.4, we show that under certain regularity assumptions on p, given ε > 0, we can find hyper-parameters n, m, τ, λ and ρ such that H(p, p sample ) ≤ Cε, i.e., that algorithm 5 generates N i.i.d. samples from a distribution Cε close to p. Proposition 7.2 (Performance of p). Let ν > min(1, d/(2β)). There exists a constant ε 0 depending only on q W β (X ) , β, d, constants C 1 , C 2 , C 3 , C 4 depending only on β, d and a constant C 1 depending only on β, d, ν such that the following holds.

Let δ ∈ (0, 1] and ε ≤ ε 0 , and assume m and n satisfy

m ≥ C 1 ε -d/β log d C 2 ε log C 3 δε (7.22) n ≥ C 1 ε -2 ν log 8 δ . (7.23)
Let τ = ε -2/β and λ = ε 2+d/β . With probability at least 1 -3δ, it holds

H( p, p) ≤ C 4 q W β (X ) ε. (7.24)
Once again, the key takeaway from this proposition is that the number of samples n, m needed to perform the first step of the algorithm (approximation) is polynomial in the quantities

O(ε -1 ), O(ε -d/β
), thus leveraging the regularity β of q s.t. q 2 = p. When this is the case, we can find λ, τ such that the distance H(p, p) is of order ε. We provide a choice for ρ for the second step of the algorithm (sampling), in order to guarantee a bound for the Hellinger distance between the sampling distribution and the original distribution in the following theorem. It is proved as Theorem 7.10 in Appendix 7.F . In particular, it bounds the total complexity of the algorithm in terms of erf computations, as a function of N and the desired error ε.

Theorem 7.4 (Performance of p sample ). Under the assumptions and notations of proposition 7.2, there exists a constant C 5 depending only on d, β, such that the following holds. If on the one hand ρ is set either as ε 1+(d+2)/(2β) or adaptively as ρ H ε (see Remark 24), then with probability at least 1 -3δ, H(p, p sample ) ≤ C 5 q W β (X ) ε.

(7.25)

Moreover, the adaptive ρ H ε is lower bounded by ε 1+(d+2)/β /(C 5 q W β (X ) ). In both cases, this guarantees that the complexity in terms of erf computations is bounded by O(N m 2 log 1 ρ ), which, in terms of ε, yields O N ε -2d/β log 2d+1 1 ε log 2 1 δε where the O notation incorporates constants depending on d, β, q W β (X ) .

.3 Discussion

The two methods presented in Sec. 7.4 .1 and Sec. 7.4 .2 share many interesting properties, both from a practical and theoretical viewpoint.

On the theoretical side, even though we only have access to the distribution up to a re-normalizing constant, this does not influence the theoretical results, i.e., the bounds we get only depend on the density p through its norm p . Moreover, the number of samples n, m needed (and hence the complexity of the sampling and of the approximation algorithm) is polynomial in the quantities O(ε -1 ), O(ε -d/β ), showing that as soon as β ≥ d, the dimension plays no role in the exponents of these error terms and thus breaking the curse of dimensionality in the rates. However, the constants in the O(•) term can be exponential in d, and without more hypotheses, they are unimprovable [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF]. We therefore keep a form of "curse of dimensionality" in the constants, but not in the rate. Concretely this means that we need a number of points in the order of the constants before having a reasonable error (i.e., ε = 1). However, as soon as this number is reached, one can rapidly gain in precision, if the function is regular. Moreover, in practice, we do not always pay this exponential constant, owing to some additional regularity of the function. Interestingly, this phenomenon is shared with approximation, learning and optimization problems over a wide family of functions (see [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF] for more details).

On the practical side, note that both algorithm 4 and algorithm 5 can be run for any hyperparameter (even though this might not have statistical sense), making it easy to use. More importantly, we can evaluate the learnt model a posteriori using empirical metrics (like the empirical total variation distance or the empirical Hellinger distance for instance) on a new data set generated uniformly from X . We could also evaluate it using certain empirical divergences since we are able to sample from p sample . This can help in both selecting τ and λ by validation, as well as in simply evaluating the performance of the learnt model, with error bars if needed. In Fig. 7.2 for example, we evaluate the performance of learnt PSD models for the empirical Hellinger distance. We perform 5 different tests and plot the associated error bars: this methods seems very robust for evaluation.

m =50 m =100 m =200 m =500 m =1000 10 4 10 5 n 10 -5 10 -4 10 -3 Hellinger distance, d = 2 m =50 m =100 m =200 m =500 m =1000

Experiments

The experiments in this work were executed on a MacBook Pro equipped with a 2.8 GHz Quad-Core Intel Core i7 processor and 16Gb of RAM2 .

Influence of m and n. In Fig. 7.2, we show how m and n interact in order to set the precision of our approximation in the learning phase (step a)). For m = 50, 100, 200, m is so small that increasing n beyond 1000 does not yield better performance (the variations are due to the fact that points are always resampled accross experiments). However, when m = 500, 1000, we see that increasing n yields better performance, before arriving at a plateau. This plateau corresponds to the transition from the phase where n is the limiting statistical factor to the phase where m is.

Qualitative performance of our algorithm. In Fig. 7.3, we show an example of the way our algorithm approximates a certain target density p 1 known up to a renormalization constant: p 1 (x) ∝ 0.08k 0. 7 (x,1)) -0.4k 0. 6 (x,(1,1)) + 0.4k 0. 7 (x,(1,1)). In the top left figure, a heat map of p 1 is plotted. Note that p 1 is not a Gaussian PSD model, as the widths of the Gaussian kernels are not the same. We then use algorithm 5 to approximate p 1 by a rank one PSD model p 1 (whose heat-map is plotted on the top right figure) and then sample N = 1000 samples from this approximation (plotted in the bottom left figure). Note that in order to approximate p 1 by p 1 , n = 10 5 , m = 300 were fixed and τ = 2, λ = 10 -9 were selected on a test set. In Sec. 7.G , we perform and comment another experiment when trying to learn a density which is not smooth (and therefore out of the scope of Theorems 7.3 and 7.4).

Quantitative performance of our algorithm. To further demonstrate the promising nature of our sampling algorithm, we tried learning the density p 2 (x) ∝ (k 1/5 (x, (1, ..., 1))k 1/5 (x, -(1, ..., 1))) 2 on Q = [-1, 1] d , for d = 5. Contrary to p 1 , this is a PSD model, we can sample from it with very high precision (here, we chose ρ = 10 -6 ). Our goal here is to be able to compare methods through the generated samples.

We compared the performance of our model to the naive gridding algorithm which, if allowed n function evaluations, computes a grid G of side n 1/d , which we identify to the set of centers of the tiles of the grid, and evaluates p at each point in the grid. To sample a point, one chooses a point g ∈ G with probability p(g)/ h∈G p(h), and then draws a sample uniformly in that tile. It is the algorithm called 'grid' in the bottom right figure of Fig. 7.3. We compare our algorithm with the gridding algorithm by fixing the number n of function evaluations of p each method is allowed, and computing the distance between each method and the ground truth. The distance we use between distributions is the empirical version of the Maxmium Mean Discrepancy distance (MMD) [START_REF] Bharath | Hilbert space embeddings and metrics on probability measures[END_REF][START_REF] Bharath | Universality, characteristic kernels and rkhs embedding of measures[END_REF], which is defined, for the Gaussian kernel k η of parameter η, as

d η (p, p) = E X∼p [φ η (X)] -E X∼p [φ η (X)]
Hη where φ η is the embedding associated to the Gaussian kernel k η (for more details, see Appendix 7.A ). This distance can be approximated using N samples (x i ) 1≤i≤N from p and N samples (x j ) 1≤j≤N from p as

d η (p, p) = 1 N N i=1 φ η (x i ) -1 N N j=1 φ η (x j )

Hη

. This quantity can be computed explicitly using kernel matrices (Sriperumbudur et al., 2010). However, Tolstikhin, Sriperumbudur, and[START_REF] Ilya O Tolstikhin | Minimax estimation of maximum mean discrepancy with radial kernels[END_REF] show that the minimax rate cannot exceed 1/ √ N , i.e., that d η approximates d η only with precision of order 1/ √ N .

Ground truth Approximation

Samples In our experiments, we take N = 10 4 . We compute the empirical distances d η five times using newly generated samples from each distribution, and compute an empirical mean and standard deviation, reported as error bars on the plot. When approximating p 2 by a PSD model using algorithm 5, we take m = 50, as there is no need to increase m to reach better precision than the target distribution for d η . We take ρ = 10 -3 and select τ, λ by using half of the evaluation points as a test set.

The results reported on the bottom-right plot of Fig. 7.3 show that in dimension 5, the 'grid' method is not competitive anymore, and is close to the uniform distribution in performance for η = 2. Note that the choice of η in a wide range from 0.1 to 10 does not change these results. They also show that when taking only N = 10 4 to approximate the MMD distance, our method is below the noise level.

Extensions, future work

In this paper, we have introduced a method for sampling any distribution from function values by first approximating it with a so-called PSD model and then sampling from this PSD model using the algorithm introduced in Sec. 7.3 . Natural extensions of this work include the fact that while we cast a least squares problem in Sec. 7.4 .1, we can actually minimize more general convex losses adapted to distributions, such as maximum log-likelihood estimation. Moreover, as mentioned in Sec. 7.3 , the proposed algorithm only relies on integral computations, and could therefore be extended to other kernels, provided they can easily be integrated on hyper-rectangles. Future work will start with trying to scale the sampling method up in terms of generation of samples, by both theoretical means (to make computation saving approximations) and computational means (use of GPUs, parallelization).

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS Theorem 7.10.

7.G . Additional experimental details

7.A Definitions and notations

In this section we recall results by [START_REF] Rudi | Psd representations for effective probability models[END_REF] which will be useful in the different statements and proofs.

Basic vector and matrix notations. Let n, d ∈ N. We denote by R d ++ the space vectors in R d with positive entries, R n×d the space of n × d matrices, S n + = S + (R n ) the space of positive semidefinite n × n matrices. Given a vector η ∈ R d , we denote diag(η) ∈ R d×d the diagonal matrix associated to η. We denote by A • B the entry-wise product between two matrices A and B. We denote by A F , A , det(A), vec(A) and A respectively the Frobenius norm, the operator norm (i.e. maximum singular value), the determinant, the (column-wise) vectorization of a matrix and the (conjugate) transpose of A. With some abuse of notation, where clear from context we write element-wise products and division of vectors u, v ∈ R d as uv, u/v. The term 1 n ∈ R n denotes the vector with all entries equal to 1.

Hyper-rectangles

Define a hyper-rectangle Q as a product of the form d k=1 [a k , b k [, where a ≤ b. Given a hyper-rectangle Q we denote its extremities with a(Q) ≤ b(Q) ∈ R d (i.e. Q = d k=1 [a k (Q), b k (Q)[),
and its side-lengths ρ(Q) = b(Q) -a(Q). We sometimes omit Q when it is implied by the context.

We will also use the so-called error function, which is defined as follows :

erf(x) = 2 √ π x 0 e -t 2 dt .
This function is implemented as an elementary function in most libraries.

Multi-index notation Let α ∈ N d , x ∈ R d and f be an infinitely differentiable function on R d , we introduce the following notation

|α| = d j=1 α i , α! = d j=1 α j !, x α = d j=1 x α j j , ∂ α f = ∂ |α| f ∂x α 1 1 • • • ∂x α d d .
We introduce also the notation D α that corresponds to the multivariate distributional derivative of order α and such that

D α f = ∂ α f
for functions that are differentiable at least |α| times [START_REF] Adams | Sobolev Spaces[END_REF].

Fourier Transform Given two functions f, g : R d → R on some set R d , we denote by f • g the function corresponding to pointwise product of f, g, i.e.,

(f • g)(x) = f (x)g(x), ∀x ∈ R d .

7.A . DEFINITIONS AND NOTATIONS

319

Let f, g ∈ L 1 (R d ) we denote the convolution by f g

(f g)(x) = R d f (y)g(x -y)dy.
We now recall some basic properties, that will be used in the rest of the appendix.

Proposition 7.3 (Basic properties of the Fourier transform [START_REF] Wendland | Scattered Data Approximation[END_REF], Chapter 5.2.).

(a) There exists a linear isometry

F : L 2 (R d ) → L 2 (R d ) satisfying F[f ] = R d e -2πi ω x f (x) dx ∀f ∈ L 1 (R d ) ∩ L 2 (R d ),
where i = √ -1. The isometry is uniquely determined by the property in the equation above.

(b) Let f ∈ L 2 (R d ), then F[f ] L 2 (R d ) = f L 2 (R d ) . (c) Let f ∈ L 2 (R d ), r > 0 and define f r (x) = f ( x r ), ∀x ∈ R d , then F[f r ](ω) = r d F[f ](rω). (d) Let f, g ∈ L 1 (R d ), then F[f • g] = F[f ] F[g]. (e) Let α ∈ N d , f, D α f ∈ L 2 (R d ), then F[D α f ](ω) = (2πi) |α| ω α F[f ](ω), ∀ω ∈ R d . (f) Let f ∈ L 1 (R d ) ∩ L 2 (R d ), then F[f ] L ∞ (R d ) ≤ f L 1 (R d ) . (g) Let f ∈ L ∞ (R d ) ∩ L 2 (R d ), then f L ∞ (R d ) ≤ F[f ] L 1 (R d ) .
Reproducing kernel Hilbert spaces for translation invariant kernels. We now list some important facts about reproducing kernel Hilbert spaces in the case of translation invariant kernels on R d . For this paragraph, we refer to the works by [START_REF] Steinwart | Support Vector Machines[END_REF]; [START_REF] Wendland | Scattered Data Approximation[END_REF]. For the general treatment of positive kernels and Reproducing kernel Hilbert spaces, see the works by [START_REF] Aronszajn | Theory of reproducing kernels[END_REF]; [START_REF] Steinwart | Support Vector Machines[END_REF]. Let v : R d → R such that its Fourier transform

F[v] ∈ L 1 (R d
H = {f ∈ L 2 (R d ) | f H < ∞}, f 2 H = R d |F[f ](ω)| 2 F[v](ω) dω, (7.26) (c) H is a separable Hilbert space, whose inner product •, • H is characterized by f, g H = R d F[f ](ω)F[g](ω) F[v](ω) dω.
In the rest of the paper, when clear from the context we will simplify the notation of the inner product, by using f g for f, g ∈ H, instead of the more cumbersome f, g H .

(d) The feature map φ : R d → H is defined as φ

(x) = k(x -•) ∈ H for any x ∈ R d .
(e) The functions in H have the reproducing property, i.e.,

f (x) = f, φ(x) H , ∀f ∈ H, x ∈ R d , in particular k(x , x) = φ(x ), φ(x) H for any x , x ∈ R d .
CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS

We now introduce the main tool of our analysis, the Gaussian RKHS, which will be further explored in Appendix 7.C .

Example 7.1 (Gaussian Reproducing Kernel Hilbert Space). Let η ∈ R d ++ and k η (x, x ) = e -(x-x ) diag(η)(x-x ) , for x, x ∈ R d be the Gaussian kernel with precision η. The function k η is a translation invariant kernel, since

k η (x, x ) = v(x -x ) with v(z) = e -D 1/2 z 2 , D = diag(η) and F[v](ω) = c η e -π 2 D -1/2 ω 2 , c η = π d/2 det(D) -1/2 , for ω ∈ R d is in L 1 (R d ) and satisfies F[v](ω) ≥ 0 for all ω ∈ R d .
The associated reproducing kernel Hilbert space H η is defined according to Eq. (7.26), with norm

f 2 Hη = 1 c η R d |F[f ](ω)| 2 e π 2 D -1/2 ω 2 dω, ∀f ∈ L 2 (R d ). (7.27)
The inner product and the feature map φ η are defined as in the discussion above.

7.A .1 Sobolev spaces

Let β ∈ N, p ∈ [1, ∞] and let Ω ⊆ R d be an open set. The set L p (Ω) denotes the set of p-integrable functions on Ω for p ∈ [1, ∞) and that of the essentially bounded on Ω when p = ∞. The set W β p (Ω) denotes the Sobolev space, i.e., the set of measurable functions with their distributional derivatives up to β-th order belonging to L p (Ω),

W β p (Ω) = {f ∈ L p (Ω) | f W β p (Ω) < ∞}, f p W β p (Ω) = |α|≤β D α f p L p (Ω) , (7.28)
where D α denotes the distributional derivative. In the case of p = ∞,

f W β ∞ (Ω) = max |α|≤β D α f L ∞ (Ω)
We now recall some basic results about Sobolev spaces that are useful for the proofs in this paper.

g| Ω W s p (Ω) ≤ g W s p (Ω )
, and so g| Ω ∈ W s p (Ω) for any g ∈ W s p (Ω ). Now we recall the extension properties of Sobolev spaces, which will allow us to consider the case The formal definition of a set with Lipschitz boundary is provided by [START_REF] Adams | Sobolev Spaces[END_REF]. Note that if X = (-1, 1) d , as will be the case later on for simplicity, then X is bounded and has Lipschitz boundary.

The following result shows that being in an intersection space allows to extend the function to the whole of R d . This will be useful in order to use the properties of translation invariant kernels in order to approximate functions which are a priori defined only on X but which we extend using this result.

Proposition 7.4 (Rudi and Ciliberto (2021, Corollary A.3

)). Let X ⊂ R d be a non-empty open set with Lipschitz boundary. Let β ∈ N, p ∈ [1, ∞]. Then for any function f ∈ W β p (X ) ∩ L ∞ (X ) there exists an extension f on R d , i.e. a function f ∈ W β p (R d ) ∩ L ∞ (R d ) such that f = f | X a.e. on X , f L ∞ (R d ) ≤ C f L ∞ (X ) , f W β p (R d ) ≤ C f W β p (X )
. The constant C depends only on X , d, and the constant C only on X , β, d, p

The following proposition gives an idea of what these intersection spaces contain.

Proposition 7.5 (Rudi and Ciliberto (2021, Proposition A.4)). Let X be an open bounded set with Lipschitz boundary. Let f be a function that is m times differentiable on the closure of X . Then there exists a function

f ∈ W m p (X ) ∩ L ∞ (X ) for any p ∈ [1, ∞], such that f = f on X .
The following proposition provides a useful characterization of the space W β 2 (R d ) in terms of Fourier transform; this will be particularly useful when approximating functions in W β 2 (R d ) by functions in a Gaussian RKHS H η using the characterization of the norm in terms of Fourier transform for those kernels in Eq. (7.26).

Proposition 7.6 (Characterization of the Sobolev space W k 2 (R d ), [START_REF] Wendland | Scattered Data Approximation[END_REF], (Rudi and Ciliberto, 2021, Proposition A.4)). Let k ∈ N. The norm of the Sobolev space

• W k 2 (R d ) is equivalent to the following norm f 2 W k 2 (R d ) = R d |F[f ](ω)| 2 (1 + ω 2 ) k dω, ∀f ∈ L 2 (R d )
and satisfies

1 (2π) 2k f 2 W k 2 (R d ) ≤ f W k 2 (R d ) ≤ 2 2k f 2 W k 2 (R d ) , ∀f ∈ L 2 (R d ) (7.29) Moreover, when k > d/2, then W k 2 (R d
) is a reproducing kernel Hilbert space.

7.A .2 Measuring distances between probability densities

In this work, since our aim is to approximate a probability distribution, we will often compare probability distributions, with different distances.

To simplify definitions, we will only consider distances between probability densities p 1 , p 2 defined on a Borel subset X of R d with respect to the Lebesgue measure. Note that while the total variation distance, the Hellinger distance and the Wasserstein distance do not actually depend on the choice of such a base measure and can be defined intrinsically, the L 2 distance cannot; that is why it is less appropriate from a statistical point of view. We consider it here because it is the natural distance in which we are able to solve Eq. (7.15).

The total variation (TV) or L 1 distance : (7.30) This distance can also be expressed using a dual formulation (Lucien Le Cam, 1990, chapter 3.2).

d T V (p 1 , p 2 ) := p 1 -p 2 L 1 (X ) = X |p 1 (x) -p 2 (x)|dx.
d T V (p 1 , p 2 ) = sup |f |≤1 X f (x)(p 1 (x) -p 2 (x)) dx (7.31)
The Hellinger distance : (this distance is particularly suitable in the case of exponential models; see the works by Lucien Le [START_REF] Lo | Asymptotics in Statistics: Some Basic Concepts[END_REF] and in particular Chapter 3).

H(p 1 , p 2 ) := √ p 1 - √ p 2 L 2 (X ) = X | √ p 1 (x) - √ p 2 (x)| 2 dx 1/2 (7.32) CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS
The Wasserstein distance In the case where X is bounded (for simplicity), the p Wasserstein distance for p ≥ 1 (see chapter 5 by Santambrogio ( 2015)):

W p p (p 1 , p 2 ) = inf γ∈Π(p 1 ,p 2 ) X ×X |x -y| p dγ(x, y), (7.33)
where Π(p 1 , p 2 ) is the set of all probability measures on X × X with marginals p 1 and p 2 . Note that one has the following easier dual formulation when p = 1 (see the chapter on Kantorovich duality by [START_REF] Santambrogio | Optimal Transport for Applied Mathematicians[END_REF]):

W 1 (p 1 , p 2 ) = sup f ∈Lip 1 (X ) X f (x)(p 1 (x) -p 2 (x))dx, (7.34)
where Lip 1 (X ) is the set of 1-Lipschitz functions on X . Wasserstein distances capture the moving of mass; they are quite weak but are well-adapted to capture the behavior of our sampling algorithm which approximates probability densities on each hyper-rectangle.

The L 2 distance :

p 1 -p 2 L 2 (X ) = X (p 1 (x) -p 2 (x)) 2 dx 1/2 (7.35)
Relating these difference distances . The following well known bounds exist between distances.

H 2 (p 1 , p 2 ) ≤ d T V (p 1 , p 2 ) ≤ √ 2H(p 1 , p 2 ). (7.36)
Moreover, if X is bounded, we have for any p ≥ 1, using the Holder inequality:

W p (p 1 , p 2 ) ≤ diam(X ) (p-1)/p W 1 (p 1 , p 2 ) 1/p , (7.37) W 1 (p 1 , p 2 ) ≤ diam(X )d T V (p 1 , p 2 ), (7.38) d T V (p 1 , p 2 ) ≤ |X | 1/2 p 1 -p 2 L 2 (X ) , (7.39)
where diam(X ) denotes the diameter of the set X .

7.A .3 General PSD models

In this section, we recall the definition of a PSD model more generally as introduced by Rudi and Ciliberto (2021).

Following Marteau-Ferey, Bach, and Rudi (2020); [START_REF] Rudi | Psd representations for effective probability models[END_REF], we consider the family of positive semi-definite (PSD) models, namely non-negative functions parametrized by a feature map φ : X → H from an input space X to a suitable feature space H (a separable Hilbert space e.g. R q ) and a linear operator M ∈ S + (H), of the form

f (x; M, φ) = φ(x) M φ(x). (7.40)
PSD models offer a general way to parametrize non-negative functions (since M is positive semidefinite, f (x; M, φ) ≥ 0 for any x ∈ X ) and enjoy several additional appealing properties discussed in the following. In this work. we focus on a special family of models i.e. Gaussian PSD models defined in Sec. 7.2 and Eq. (7.1). These models parametrize probability densities over X ⊂ R d . It is a special case of Eq. (7.40) where i) φ = φ η : R d → H η is a feature map associated to the Gaussian kernel defined in Example 7.1, or by [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF] and, ii) the operator M lives in the span of φ(x 1 ), . . . , φ(x n ) for a given set of points (x i ) n i=1 , namely there exists

A ∈ S + (R n ) such that M = ij A ij φ η (x i )φ η (x j ) .
Thus, given the triplet (A, X, η) characterizing the Gaussian PSD model in Eq. (7.1), we have 1≤i,j≤n

A ij k η (x, x i )k η (x, x j ) = f (x; A, X, η) = f (x; M, φ η ) M = 1≤i,j≤n A ij φ η (x i ) ⊗ φ η (x j ),
where (u ⊗ v)w = uv w = v, w u.

7.B Properties of the Gaussian RKHS

In this section, we introduce notations and results associated to the Gaussian RKHS (see Example 7.1) H η for a given η ∈ R d ++ (η will sometimes be taken in the form τ 1 d ). Recall that the Gaussian embedding is written φ η : R d → H η and that the Gaussian kernel is denoted with k η .

7.B .1 Properties of the Gaussian kernel k η

The following lemma has an immediate proof.

Lemma 7.1 (product of Gaussian kernels). Let K ∈ N, let η 1 , ..., η K ∈ R d ++ and let y 1 , ..., y K ∈ R d . The following equality holds:

∀x ∈ R d , K k=1 k η k (x, y k ) = k η (x, y) K k=1 k η k (y k , y)
where η = K k=1 η k and y = k η k y k /η Let us now state an useful corollary.

Corollary 7.1. Let η ∈ R d ++ , y 1 , y 2 ∈ R d . Then ∀x ∈ R d , k η (x, y 1 )k η (x, y 2 ) = k 2η (x, (y 1 + y 2 )/2)k η/2 (y 1 , y 2 ). (7.41) Lemma 7.2 (Gaussian embedding derivative). Let η ∈ R d ++ , x ∈ R d and α ∈ N d . The derivative ∂ α φ η (x) is well defined in H η , and ∂ α φ η (x) Hη = 2 |α|/2 η α/2 . Moreover, if g ∈ H η , then sup x∈R d |(∂ α g)(x)| ≤ 2 |α|/2 η α/2 g Hη . Proof. Let α ∈ N d and let v η (z) = k η (z, 0) = exp(-z diag(η)z). If the function ∂ α ∂x α k η (x, y) belongs to H η , then ∂ α φ η (x)
is in H η and is equal to that function by the reproducing property.

First, note that ∀x, y ∈ R d , ∂ α ∂x α k η (x, y) = (-1) |α| ∂ α τ x [v η ](y),
where τ x : f → f (• -x), commutes with the differential operator ∂ α , and satisfies the following relation wrt to the Fourier transform : F[τ x g](ξ) = e -2iπxξ F[f ](ξ). Hence, using (e) of proposition 7.3, we get the following fourier transform wrt y:

F y [ ∂ α ∂x α k η (x, y)](ξ) = (-2πi) |α| ξ α e -2iπξx F[v η ](ξ).
Hence, we have using Eq. (7.26):

∂ α ∂x α k η (x, •) 2 Hη = R d (2π) 2|α| ξ 2α F[v η ](ξ) dξ = (-1) |α| R d (2iπ) 2|α| ξ 2α F[v η ](ξ) dξ = (-1) |α| R d F[∂ 2α v η ](ξ) dξ = (-1) |α| ∂ 2α v η (0),
where the last equality comes from the inverse Fourier transform. A simple recursion then shows that (-1) |α| ∂ 2α v η (0) = 2 |α| η α , hence the result. The last point of the lemma is simply a consequence of the fact that ∂ α g(x) = g, ∂ α φ η (x) Hη .

7.B .2 Useful Matrices and Linear Operators on the Gaussian RKHS

Recall that we denote with φ η the embedding associated to the RKHS H η of the Gaussian kernel k η defined in Example 7.1. In this section, we define operators which will be useful throughout the rest of this section and which we will use in Appendixes 7.E and 7.F . In order to make the dependence in η appear (indeed, η will be a parameter to choose in the the next sections), we will keep it as an index for all of these operators. Recall that for any two vectors u, v in a Hilbert space H, we can define their tensor product u ⊗ v which is a linear rank one operator on H defined by (u ⊗ v)w = v, w H u. For the sake of simplicity, we will often write u ⊗ v as uv , so that the formula (u ⊗ v)w = uv w is formally true.

Kernel matrices. We start off by setting the notations for kernel matrices as done by [START_REF] Rudi | Psd representations for effective probability models[END_REF]. Let X ∈ R n×d and X ∈ R n ×d be two matrices corresponding to points x 1 , ..., x n ∈ R d and x 1 , ..., x n ∈ R d . We denote with K X,X ,η the matrix in R n×n such that

∀1 ≤ i ≤ n, ∀1 ≤ j ≤ n , [K X,X ,η ] ij = k η (x i , x j ). (7.42) If X = X , then we just write K X,η and it is positive semi-definite, i.e. K X,η ∈ S + (R n ).
Integration matrices. In this work, we also define, for a given hyper-rectangle

Q = d k=1 [a k , b k ], the following integration matrix G X,X ,η,Q ∈ R n×n : ∀1 ≤ i ≤ n, ∀1 ≤ j ≤ n , [G X,X ,η,Q ] ij = Q k η (x -(x i + x j )/2)dx = d k=1 π 4η k erf( √ η k (b k + (x ik + x jk )/2)) -erf( √ η k (a k + (x ik + x jk )/2)) , (7.43)
where the erf function is defined in the notations section. Similarly, if X = X , we simply write G X,η,Q .

This matrix is defined in order to satisfy the following property, which is a direct application of Eq. (7.41): for any X ∈ R n×d , any A ∈ S n + and η ∈ R d ++ , the following holds.

Q f (x; A, X, η) dx = 1≤i,j≤n [A • K X,η/2 • G X,2η,Q ] ij = vec(A • K X,η/2 • G X,2η,Q ) 1 n 2 (7.44)
Co-variance operator. Let X ⊂ R d be a measurable set of R d with finite Lebesgue measure |X |. Define the associated co-variance operator:

C η ∈ S + (H η ), C η = 1 |X | X φ η (x) ⊗ φ η (x) dx, C η,λ = C η + λI. (7.45)
Note that C η is a trace class operator with and that Tr (C η ) = 1 by linearity of the trace and since Tr (

φ η (x) ⊗ φ η (x)) = φ η (x) 2 = k η (x, x) = 1. Moreover, since C η,λ λI, C η,λ is inverible for any λ > 0.
Note that we do not make the set X appear in the notation of the co-variance operator (which can actually be defined with respect to any probability distribution on R d and not just 1 X dx |X | ). This is because the set X will usually explicit in the next sections, and in particular equal to the unit hyper-cube X = (-1, 1) d . Sampling operators. Let n ∈ N (x 1 , .., x n ) ∈ (R d ) n be points of R d which should be seen as samples from a certain distribution. We define the following sampling operators.

C η ∈ S + (H η ), C η = 1 n n i=1 φ η (x i ) ⊗ φ η (x i ), C η,λ = C η + λI (7.46) S η : H η → R n , S η (g) = 1 √ n (g(x i )) 1≤i≤n (7.47) S * η : R n → H η , S * η (a) = 1 √ n n i=1 a i φ η (x i ) (7.48)
where S * η and S η are adjoint operators. We will usually use the • notation to denote sampling operators, and imply the underlying (x 1 , ..., x n ). These operators will be used in later sections in order to quantify the difference between objects resulting from the sampling of distributions and the "ideal" objects (typically the difference between an empirical risk minimizer and the true expected risk minimizer). For instance, it is clear the C η is an empirical version of C η , if the x i are i.i.d. samples from the uniform distribution on X .

Compression operators. Following the notations used by Rudi and Rosasco (2017); [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]; [START_REF] Rudi | Psd representations for effective probability models[END_REF], a compression operator of size m is an operator Z η,m : H η → R m . We call it a compression operator since we use it to project every element of H η onto the range of the adjoint operator Z * η,m : R m → H η . This range, which we denote with H η,m ⊂ H η , is a subset of dimension at most m. We also denote with P η,m : H η → H η the orthogonal projection onto H η,m , which can also be written

P η,m = Z * η,m ( Z η,m Z * η,m ) † Z η,m
, where † denotes the Moore-Penrose pseudo-inverse.

In this work, we will always use the notation • m to denote a compression operator, and the index m to make the size of the compression explicit.

In this work, we take a specific form of compression operator as in appendix C by [START_REF] Rudi | Psd representations for effective probability models[END_REF]. Indeed, let X m ∈ R m×d be a data point matrix representing vectors x 1 , ..., x m ∈ R d . The compression operator associated to X m is the following :

Z η,m : H η → R m , Z η,m (g) = (g( x j )) 1≤j≤m = (g φ η ( x j )) 1≤j≤m . (7.49) Note that Z η,m Z * η,m = K Xm,η
and hence the projection operator can be written

P η,m = Z * η,m K † Xm,η
Z η,m and that it is simply the projection onto span φ η ( x i ) 1≤i≤m . This compression is also chosen to satisfy the two following properties :

• if h ∈ H η , then P η,m h represents a function of the form g(•; a, X m , η) where a = K † Xm,η
Z η,m h (see Eq. ( 7.2) for the definition of the Gaussian linear model g(x; a, X m , η));

• if M ∈ S + (H η ), then for any x ∈ R d , it holds f (x; P η,m M P η,m , φ η ) = f (x; A, X m , η), A = K † Xm,η Z η,m M Z * η,m K † Xm,η , (7.50)
meaning that compressed linear (resp. PSD) models can be compressed as a sum of m (resp. m 2 ) Gaussian kernel functions. We quantify the effect of this compression in Lemma 7.3 and Theorem 7.5.

7.B .3 Approximation properties of the Gaussian kernel

This section aims in quantifying the approximation power of the Gaussian RKHS. We start in proposition 7.7 by quantifying the approximation power of the Gaussian RKHS by finding an ε approximation of a regular function with controlled norm. We then quantify the "size" of a compression for the Gaussian RKHS in Lemma 7.3, which essentially bounds the possible variations of a function in H η if it is equal to zero on the compression points X m .

Approximation of a Sobolev function. This paragraph remolds results in the proof of Theorem D.4 by [START_REF] Rudi | Psd representations for effective probability models[END_REF] whose goal is to approximate any function

g ∈ W β 2 (R d ) ∩ L ∞ (R d ) by a function in H η . Proposition 7.7 (Approximation of W β 2 (R d ) ∩ L ∞ (R d ) in H η ). Let g be a function in W β 2 (R d ) ∩ L ∞ (R d ) and η ∈ R d ++ .
Denote with |η| the product |η| := d i=1 η i and η 0 = min 1≤i≤d η i . For any ε ∈ (0, 1], there exists θ ∈ H η such that

θ -g L 2 (R d ) ≤ ε g W β 2 (R d ) θ -g L ∞ (R d ) ≤ C 1 ε 1-ν g • , θ Hη ≤ C 2 g W β 2 (R d ) |η| 1/4 1 + ε exp 50 η 0 ε 2/β , (7.51) where g • = g L ∞ (R d ) if β ≤ d/2 and g • = g W β 2 (R d ) if β > d/2, ν = min(1, d/(2β))
and C 1 , C 2 are constants which depend only on d, β.

7.B . PROPERTIES OF THE GAUSSIAN RKHS 327

Proof. Recalling the notations from the proof of Theorem D.4. by [START_REF] Rudi | Psd representations for effective probability models[END_REF], let g t := t -d g 1 (x/t) where g 1 is defined as g in equation (D.2) by [START_REF] Rudi | Psd representations for effective probability models[END_REF]. The following result hold.

• By step 1 of the proof of Theorem D.4, g -

g g t L 2 (R d ) ≤ (2t) β g W β 2 (R d ) .
• By step 2 and the beginning of step 3 of the proof of Theorem D.4,

g g t Hη ≤ 2 β π -d/4 |η| 1/4 (1 + (t/3) β exp( 50 η 0 t 2 )) g W β 2 (R d ) .
• As in step 5 of the proof of Theorem D.4 and in particular the Young convolution inequality combined with the fact that

g 1 L 1 (R d ) is finite, g g t L ∞ (R d ) ≤ g 1 L 1 (R d ) g L ∞ (R d ) which in turn implies g -g g t ≤ (1 + g 1 L 1 (R d ) ) g L ∞ (R d ) .
Replacing t by ε 1/β /2, we get all the bounds except the bound for the L ∞ norm in the case where β > d/2. In that case, we proceed in the following way. Recycling results and notations from the proof of Theorem D.4 by [START_REF] Rudi | Psd representations for effective probability models[END_REF], denoting with F the Fourier transform defined in proposition 7.3, it holds

f -f g t L ∞ (R d ) ≤ F(f -f g t ) L 1 (R d ) proposition 7.3 = F(f )(1 -F(g t )) L 1 (R d ) ≤ (1 + ω 2 ) β/2 F(f ) L 2 (R d ) (1 + ω 2 ) -β/2 F(1 -g t ) L 2 (R d ) ≤ 2 β ω >1/t (1 + ω 2 ) -β dω 1/2 f W β 2 (R d ) Eq. (7.29) = 2 β S d r>1/t r d-1 (1 + r 2 ) -β dr 1/2 f W β 2 (R d ) (spherical coord.) ≤ 5 β/2 S 1/2 d r>1/t r d-1-2β dr 1/2 f W β 2 (R d ) (t < 1/2) = 5 β/2 1 √ 2β-d S 1/2 d t β-d/2 f W β 2 (R d ) = 5 β/2 2 d/2-β S 1/2 d 1 √ 2β-d ε 1-d/(2β) f W β 2 (R d )
, where S d is the surface area of the d -1 dimensional hyper-sphere.

A bound on the performance of compression when using uniform samples from X = (-1, 1) d . In this paragraph, we study the effect of performing compression with a compression operator of the form Z η,m (see Eq. (7.49)) where the associated X m are i.i.d. samples from the uniform measure on the unit hyper-cube X = (-1, 1) d .

Lemma 7.3. Let m ∈ N, δ ∈ (0, 1], τ ≥ 1 and ρ ∈ (0, 1]. Let η = τ 1 d ∈ R d ++ .
Let X m ∈ R m×d be a data matrix corresponding to vectors x 1 , ..., x m which are sampled independently and uniformly from X = (-1, 1) d and let P η,m be the associated projection operator in H η . With probability at

least 1 -δ, if m ≥ C 1 τ d/2 (log C 2 ρ ) d log C 3 δ + log τ + log log C 2 ρ , then it holds : sup x∈X (I -P η,m )φ η (x) ≤ ρ, (7.52)
where C 1 , C 2 , C 3 are constants which depend only on the dimension d and not on τ, m, δ, ρ.

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS Proof. Let h denote the fill distance with respect to X m , i.e.

h = max x∈[-1,1] d min 1≤j≤m
xx j (7.53)

Using Lemma 12 p.19 by [START_REF] Vacher | A Dimension-free Computational Upper-bound for Smooth Optimal Transport Estimation[END_REF], we there exists two constants C 1 , C 2 depending only on

d such that h ≤ C 1 m -1 (log(C 2 m/δ) 1/d .
Applying Theorem C.3 by [START_REF] Rudi | Psd representations for effective probability models[END_REF] in the case where X = (-1, 1) d , η = τ 1 d , there exists constants C 3 , C 4 , C 5 depending only on the dimension d such that when h ≤ τ -1/2 C -1 3 , the following holds :

sup x∈X (I -P η,m )φ η (x) ≤ C 4 e - C 5 τ 1/2 h log C 5 τ 1/2 h (7.54) Now note that taking C 6 = max(C -1 3 , eC 5 ) and C 7 = max(e, C 4 ), as soon as h ≤ C 6 τ -1/2 / log C 7 ρ , it holds a) h ≤ τ -1/2 C -1 3 , b) C 5 τ 1/2
h ≥ e and thus log C 5 τ 1/2 h ≥ 1, and hence c) sup x∈X (I -P η,m )φ η (x) ≤ ρ using Eq. (7.54). Using the bound on h, this is satisfied as soon as

m ≥ C 8 τ d/2 log C 7 ρ d log(C 2 m/δ),
where

C 8 = max(C 1 /C d 6 , e).
Using the fact that C 2 , C 8 ≥ e, and using the reasoning in the proof of Theorem C.5 by [START_REF] Rudi | Psd representations for effective probability models[END_REF], in equation (C.44), a sufficient condition is the following :

m ≥ 2C 8 τ d/2 log C 7 ρ d log(2C 2 C 8 /δ) + d 2 log τ + d log log C 7 ρ . (7.55)
The result in the theorem is obtained by taking

C 1 ← 2C 8 d, C 2 ← C 7 , C 3 ← 2C 2 C 8 .

7.C Properties of Gaussian PSD models

In this section, we detail some of the properties specific to Gaussian PSD models.

7.C .1 Bounds on the support and the derivatives

In this section, we present results which can be used to bound the tail and derivatives of a Gaussian PSD model. These bounds can be used both for theoretical purposes (see Appendixes 7.E and 7.F ) and to perform adaptive bounds in an algorithm (see Sec. 7.3 )

Lemma 7.4 (tail bound). Let δ = (δ k ) ∈ R d , η ∈ R d ++ , X ∈ R n×d and A ∈ S + (R n ). Let f (x; A, X, η) be the associated PSD model. Define x, x : ∀1 ≤ k ≤ d, x k = max 1≤i≤n X ik , x k = min 1≤i≤n X ik . Let Q δ = Q(x -δ, x + δ).
Then the following bound holds:

R d \Q δ |f (x; A, X, η)|dx ≤ 2π d/2 det(diag(2η)) -1/2 d k=1 e -2η k δ 2 k i,j [A • K X,η/2 ] ij (7.56)
Proof. Start by recalling the following simple Chernoff bound:

∀x > 0, +∞ x e -t 2 dt ≤ √ πe -x 2 (7.57) 7.C . PROPERTIES OF GAUSSIAN PSD MODELS 329 Indeed, take λ > 0. Since e -2λx e 2λt ≤ 1 t>x , it holds +∞ x e -t 2 dt ≤ e -2λx e λ 2 +∞ -∞ e -(t-λ) 2 dt ≤ √ πe -x 2 e (λ-x) 2 .
Hence, taking λ = x, we get the bound. Then we perform the following bound.

R d \Q(-δ,δ) k η (x, 0)dx = 1 d k=1 η 1/2 k R d \Q(-δ √ η,δ √ η) k 1 (x, 0)dx ≤ 1 d k=1 η 1/2 k d k=1 π (d-1)/2 2 ∞ δ k √ η k e -t 2 dt ≤ 2π d/2 det(diag(η)) -1/2 d k=1 e -δ 2 k η k ,
where we go from the first to the second line by noting that

R d \ Q(-δ, δ) ⊂ ∪ d k=1 R × .... × R \ [-δ k , δ k ] × ... × R,
and the last inequality comes from a Eq. (7.57).

The result immediately follows from Eq. ( 7.41) as well as the fact that

Q δ contains (x i + x j )/2 + Q(-δ, δ) for all 1 ≤ i, j ≤ n.
Lemma 7.5 (derivative bound for general PSD model).

Let η ∈ R d ++ , M ∈ S + (H η ) X ∈ R n×d and A ∈ S n + .
The following bounds hold :

sup x∈R d |∂ α f (x; M, φ η )| ≤ 2 3|α|/2 η α/2 M (7.58) sup x∈R d |∂ α f (x; A, X, η)| ≤ 2 3|α|/2 η α/2 K 1/2 X,η AK 1/2 X,η (7.59)
Proof. By derivation of a bi-linear form, we get

∂ α f (x; M, φ η ) = β≤α α β ∂ β φ η (x), M ∂ α-β φ η (x) Hη
Hence, using Lemma 7.2, we get, for any

x ∈ R d , |∂ α f (x; M, φ η )| ≤ M β≤α α β 2 |β|/2 η β/2 2 |α-β|/2 η (α-β)/2 = 2 3|α|/2 η α/2 M . (7.60) In particular, since f (x; A, X, η) = f (x; M A , φ η ) with M A = Z * AZ for Z : h ∈ H η → h(x i ) 1≤i≤n ,
and since ZZ * = K X,η , it holds

M A = Z * AZ = A 1/2 ZZ * A 2 = A 1/2 K X,η A 1/2 = K 1/2 X,η AK 1/2 X,η ,
and hence the second equation of the lemma.

330

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS

7.C .2 Compression as a Gaussian PSD model

In this section, we restate Theorem C.4 by [START_REF] Rudi | Psd representations for effective probability models[END_REF] on the compression of a PSD model of the form f (x; M, φ η ) into a Gaussian PSD model.

Let η ∈ R d ++ , M ∈ S + (H η ).
Given a matrix X m ∈ R m×d representing vectors x 1 , ..., x m ∈ R d , and the associated projection operator P η,m (for more details, see Appendix 7.B .2), one can compress the PSD model f (•; M, φ η ) into f (•; P η,m M P η,m ) which is also a Gaussian PSD model of the form f (•; A, X m , η) (A is defined in Eq. (7.50)). The quality of the compression is given by the following theorem.

Theorem 7.5 ((Rudi and Ciliberto, 2021, Theorem C.4)). Using the previous notations, the compressed model associated to P η,m M P η,m of M onto X m has a distance to the original PSD model associated to M bounded, for any x ∈ X , by

|f (x; M, φ η ) -f (x; P η,m M P η,m , φ η )| ≤ f (x; M, φ η ) M 1/2 sup x∈X (I -P η,m )φ η (x) + M sup x∈X (I -P η,m )φ η (x) 2 . (7.61)
We therefore see that the quality of the compression depends mainly on the quantity sup

x∈X (I -P )φ η (x) ,
which can be bounded using Eq. (7.52) in Lemma 7.3.

7.C .3 Approximation properties of Gaussian PSD model

Define, for any measurable Ω ⊂ R d , and any f : Ω → R, the following function (set to +∞ if the set is empty).

f sos,Ω,β = inf    Q i=1 max( f j L ∞ (Ω) , f j W β 2 (Ω) ) 2 | f = Q j=1 f 2 j , Q ∈ [0, +∞]    (7.62)
Here, we recall Theorem D.4 by [START_REF] Rudi | Psd representations for effective probability models[END_REF], refined in a small way to have more control over the dependence in the f j .

Theorem 7.6 ((Rudi and Ciliberto, 2021, Theorem D.4)). Let τ ≥ 1 and ε ∈ (0, 1] and f such that

f sos,R d ,β < ∞. Let η = τ 1 d . There exists M τ, ∈ S + (H η ) such that f τ, := f (•; M τ, , φ η ) is ε close to p in L 2
norm and has controlled trace norm:

f τ, -f L 2 (R d ) ≤ C 1 f sos,R d ,β ε, Tr (M τ, ) ≤ C 2 f sos,R d ,β τ d/2 (1 + ε 2 exp(C 3 ε -2/β /τ )), (7.63)
where the constants C 1 , C 2 , C 3 depend only on β, d.

Proof. Let δ > 0 and take Q δ ∈ [0, +∞] as well as f δ,j such that f = Q δ j=1 f 2 δ,j point-wise and

Q i=1 f δ,j W β 2 (R d ) max( f δ,j L ∞ (R d ) , f δ,j W β 2 (R d ) ) ≤ f sos,β .
Now using exactly the same reasoning than in the proof of Theorem D.4 by [START_REF] Rudi | Psd representations for effective probability models[END_REF] but setting simply t = ε 1/β , it holds the existence of M δ,τ,ε and C 1 , C 2 , C 3 depending only on β, d such that

f δ,τ,ε -f L 2 (R d ) ≤ C 1 ( f sos,R d ,β + δ) ε, Tr (M δ,τ,ε ) ≤ C 2 ( f sos,R d ,β + δ) τ d/2 (1 + ε 2 exp(C 3 ε -2/β /τ )).
Note that in the proof, M δ,τ,ε is well defined since its trace norm is bounded (normal convergence). Now if f sos,R d ,β = 0, then f = 0 and there is nothing to prove. If not, then taking δ = f sos,R d ,β , the theorem holds.

7.D The sampling algorithm

In this section, we formally prove that algorithm 2 converges, as in Theorem 7.1, as well as the different results of Sec. 7.3 . We start by introducing some notations around dyadic decomposition of hyper-rectangles. We then introduce a well founded order relation, which we will then use to both construct the random variables we study, justify the convergence of the algorithm and prove its correctness.

Recall we are given a density (up to a scaling factor) f (x) and that we denote with I(Q) the quantity Q f (x)dx on any hyper-rectangle Q.

7.D .1 Dyadic decompositions and convergence of algorithm 2

Dyadic sub-rectangles Let Q = d k=1 [a k , b k [ be a hyper-rectangle where a ≤ b and let δ = ba. Let q ∈ N d . We define D Q,q to be the set of dyadic sub-rectangles of Q whose k-th size is cut in half q k times, i.e.

D Q,q = d k=1 [a k + δ k s 2 q k , a k + δ k s+1 2 q k [ : s ∈ d k=1 0, 2 q k -1 .
We denote with D Q the set of dyadic sub-rectangles of Q, i.e. the union q∈N d D Q,q . Moreover, if q ρ k = max(0, log 2 δ k ρ ), we also define D Q,ε := D Q,q ρ to be the set of dyadic sub-rectangles whose size is just below ρ.

Well founded order relation on hyper-rectangles For all ρ > 0, we define the following strict order relation. We say that Q ≺ ρ Q if the following conditions hold :

1. Q ∈ D Q ; 2. There exists k ∈ 1, d such that δ k > ρ and δ k < δ k . This relation is obviously transitive. Moreover, if s(Q) := d k=1 δ k (Q), it is easy to show that Q ≺ ρ Q implies s(Q) ≤ s(Q ) -ρ/2. Since s ≥ 0,
this in turn shows that any strictly decreasing sequence for ≺ ρ is finite, and that Q ≺ ρ Q and Q ≺ ρ Q are incompatible.

We are now ready to define the random variable Y ρ,Q,n by structural induction on Q for any n ∈ N. Recall that for Ω ⊂ R d , we denote with U Ω the uniform law on Ω.

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS Definition of the random variable Y ρ,Q,n and relation to the algorithm We now define a random variable from whose distribution we sample when SamplerRec in 2 is applied.

• If δ(Q) ≤ ρ, then for any n ∈ N, Y ρ,Q,n ∼ U ⊗n Q • Else, let n ∈ N and k Q = min arg max 1≤k≤d δ k (Q) be the smallest index amongst the largest sides of Q. Define Q 1 and Q 2 to be the two hyper-rectangles obtained by cutting Q in half along the direction k Q . Since δ k Q > ρ and Q 1 , Q 2 are dyadic sub-rectangles of Q, we have Q 1 , Q 2 ≺ ρ Q.
By structural induction, we give ourselves a probability space on which we take we take the following random variables to be independent :

Y 1,m ∼ Y ρ,Q 1 ,m , Y 2,m ∼ Y ρ,Q 2 ,m for 0 ≤ m ≤ n and M ∼ B(n, I(Q 1 )/I(Q)) and define Y ρ,Q,n = (Y ρ,Q 1 ,M , Y ρ,Q 2 ,n-M ) := n m=0 1 M =m (Y 1,m , Y 2,n-m ).
(7.64)

Lemma 7.6 (Termination of the algorithm and first result). For any inputs ρ > 0, hyperrectangle Q and n ∈ N, SamplerRec in algorithm 2 terminates and returns a sample (y 1 , ...,

y n ) from Y ρ,Q,n .
Proof. This is a simple application of structural induction on the well-founded order ≺ ρ for the termination and then again for the fact that a sample (y 1 , ..., y n ) from Y ρ,Q,n , using the definition of Y above.

7.D .2 Proof of Theorem 7.1

In this section, we prove Theorem 7.1. To do so, we define a random variable X ρ,Q , compute its density with respect to the Lebesgue measure on the hyper-rectangle Q (and show it is our target density), and show that Y = X up to some random shuffling.

Definition of the variable

X ρ,Q Recall the definition of D Q,ρ from Appendix 7.D .1. We define a random variable R ρ,Q on D Q,ρ whose law is defined P (R ρ,Q = r) = I(r)/I(Q).
Recall that for any r ⊂ R d , we denote with U r the uniform law on r. We give ourselves a measure space on which there exists a family of random variables U r ∼ U r for r ∈ D Q,ρ and R ∼ R ρ,Q which are all independent and define

X ρ,Q = U R := r∈D Q,ρ
1 R=r U r (7.65)

Lemma 7.7 (density of X ρ,Q ). The density of X ρ,Q with respect to the Lebesgue measure is given by Eq. (7.7), i.e.

∀x ∈ Q, p X ρ,Q (x) = r∈D Q,ρ I(r) I(Q) 1r(x)
|r| .

(7.66)

7.D . THE SAMPLING ALGORITHM

333

Proof. For any measurable function f , it holds

E[f (X ρ,Q )] = r∈D Q,ρ E[1 R=r f (U r )] = r∈D Q,ρ P (R = r)E[f (U r )] = r∈D Q,ρ I(r) I(Q) R d f (x) 1r(x) |r| dx = R d f (x)   r∈D Q,ρ I(r) I(Q) 1r(x) |r|   dx
Action of a permutation and decomposition Let n ∈ N. For any permutation τ ∈ S n and vector v ∈ R n , denote with τ v the permuted vector (v τ -1 (i) ) 1≤i≤n .

We now define a decomposition of a permutation of n variables as i) a permutation of the first m variables and a permutation of the last nm variables ii) followed by a rearrangement of these variables.

Given I ⊂ 1, n of size m, define τ I as the unique permutation satisfying I = {τ I (1), ..., τ I (m)}, I c = {τ I (m+1), ..., τ I (n)} and τ I (1) < ... < τ I (m) and τ I (m+1) < ... < τ I (n). For any m ∈ 0, n , if P m (n) denotes the set of subsets of {1, ..., n} of size m, the map from

P m (n) × S m × S n-m to S n defined as (I, σ m , σ n-m → i → τ I (σ m (i)) if i ≤ m τ I (m + σ n-m (i -m)) otherwise (7.67) is a bijection. Lemma 7.8. Let ρ > 0. Let n ∈ N, Q be a hyper-rectangle of R d . Let σ be a random permutation independent of Y ρ,Q,n . Then (Y σ(i) ρ,Q,n ) 1≤i≤n ∼ X ⊗n ρ,Q .
Proof. Once again, we prove this by structural induction. Fix ρ > 0. We will prove the following property by structural induction on the set of hyper-rectangles Q equipped with the strict order relation ≺ ρ :

For any n ∈ N, if σ is a random permutation (i.e. distributed uniformly amongst all permutations in S n ), Y Q,n ∼ Y ρ,Q,n and both random variables are independent, then (Y

σ(i) Q,n ) 1≤i≤n ∼ X ⊗n Q,ρ . 1) If δ(Q) ≤ ρ.
On the one hand, by definition of Y ρ,Q,n , it holds that for any

n ∈ N, Y ρ,Q,n ∼ U ⊗n Q and hence Y Q,n ∼ U ⊗n Q .
By invariance of the product measure by permutation, it also holds that (Y

σ(i) Q,n ) 1≤i≤n ∼ U ⊗n
Q . On the other hand, since δ(Q) ≤ ρ, it is easy to see that q ρ = 0 and hence D Q,ρ = {Q}. Hence, by definition of X ρ,Q in Eq. (7.65), R is deterministic and hence

X ρ,Q = U Q ∼ U Q .
Putting things together, this yields (Y

σ(i) Q,n ) 1≤i≤n ∼ X ⊗n ρ,Q . CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS
2) Assume δ(Q) > ρ and take n ∈ N. By definition of Y ρ,Q,n in Eq. (7.64), and since our property only concerns a convergence in law, we can assume that Y Q,n is of the form

Y Q,n = n m=0 1 M =m (Y Q 1 ,m , Y Q 2 ,n-m ),
where

Y Q 1 ,m , Y Q 2 ,m and M are independent and independent of σ, Y Q 1 ,m ∼ Y ρ,Q 1 ,m , Y Q 2 ,m ∼ Y ρ,Q 2 ,m
for 0 ≤ m ≤ n and M ∼ B(n, I(Q 1 )/I(Q)), and Q 1 , Q 2 are defined just before Eq. (7.64). It is easy to see that since

Q 1 Q 2 = Q and Q 1 , Q 2 ≺ ρ Q, it holds D Q,ρ = D Q 1 ,ρ D Q 2 ,ρ
where symbolises a disjoint union.

Fix a measurable function f . Using the independence of M from the other variables and the fact that it is discrete, it holds

E[f (σ Y Q,n )] = n m=0 P (M = m)E[f (σ (Y Q 1 ,m , Y Q 2 ,n-m ))].
Now note that using our bijection Eq. ( 7.67), it holds

E σ,Y Q 1 ,m ,Y Q 2 ,m [f (σ (Y Q 1 ,m , Y Q 2 ,n-m ))] = 1 n! τ ∈Sn E Y Q 1 ,m ,Y Q 2 ,m [f (τ (Y Q 1 ,m , Y Q 2 ,n-m ))] = 1 n! I⊂ 1,n |I|=m σ 1 ∈Sm σ 2 ∈S n-m E Y Q 1 ,m ,Y Q 2 ,m [f (τ I (σ 1 Y Q 1 ,m , σ 2 Y Q 2 ,n-m )))] = 1 n m I⊂ 1,n |I|=m E σ 1 ,σ 2 ,Y Q 1 ,m ,Y Q 2 ,m [f (τ I (σ 1 Y Q 1 ,m , σ 2 Y Q 2 ,n-m )))]
Now note that by induction,

σ 1 Y Q 1 ,m ∼ X ⊗m ρ,Q 1 and σ 2 Y Q 2 ,n-m ∼ X ⊗(n-m) ρ,Q 2 . Let X 1 1 , ..., X n 1 ∼ X ρ,Q 1 and X 1 1 , ..., X n 1 ∼ X ρ,Q 2 be 2n i.i.d. random variables; the previous statement shows that , τ I (σ 1 Y Q 1 ,m , σ 2 Y Q 2 ,n-m ) ∼ (X i 1 1 i∈I + (1 -1 i∈I )X i 2 ) 1≤i≤n (here, I is fixed). Moreover, note that P (M = m) = n m q m (1 -q) n-m where q = I(Q 1 )/I(Q). Hence E[f (σ Y Q,n )] = I⊂ 1,n q |I| (1 -q) n-|I| E X i 1 ,X i 2 (X i 1 1 i∈I + (1 -1 i∈I )X i 2 ) 1≤i≤n
Now let B 1 , ..., B n be n i.i.d. Bernoulli variables of parameter q independent of the X 1 , X 2 . Note that from the previous equation,

E[f (σ Y Q,n )] = E[f ((X i 1 B i + X i 2 (1 -B i )) 1≤i≤n )]
It is easy to see that (X i 1 B i + X i 2 (1 -B i )) 1≤i≤n are i.i.d. and distributed as X ρ,Q , which concludes the proof.

Proof of Theorem 7.1. Theorem 7.1 is now a simple consequence of Lemmas 7.6 to 7.8.The bound on the number of integral computations can be easily obtained by noting that for any sample, at most sum d k=1 q ρ k hyper-rectangles are visited (we do not count the first since this computation is done once and for all in any case). Since q ρ k = log 2 (δ k /ρ) ≤ log 2 (2δ k /ρ), this yields a bound of log 2 (2 d |Q|/ρ d ) = log 2 (|Q|) + d log 2 (2/ρ) per sample, hence the result.

7.D .3

Evaluating the error of the sampling algorithm : proof of Theorem 7.2

Theorem 7.2 is a specific case of the following theorem. For a given function g defined on a hyper-rectangle Q, define its Lipschitz constant with respect to the infinity norm :

∀x ∈ Q, x ∞ = sup 1≤k≤d |x i |, Lip ∞ (g) = sup x,y∈Q x =y |g(x) -g(y)| x -y ∞ . (7.68)
Theorem 7.7 (Variation bounds). Let Q be a hyper-rectangle, ρ > 0, p Q = f /I(Q) and p Q,ρ defined in Eq. (7.7). Recall the definition of Lip ∞ (f ), Lip ∞ ( √ f ) from Eq. (7.68). The following bounds hold.

d T V (p Q , p Q,ρ ) ≤ |Q| I(Q) Lip ∞ (f ) ρ (7.69) H(p Q , p Q,ρ ) ≤ |Q| I(Q) Lip ∞ ( f ) ρ (7.70) W p (p Q , p Q,ρ ) ≤ √ dρ, p ≥ 1. (7.71) Proof. Recall that p Q = f 1 Q /I(Q) and hence ∀x ∈ Q, p Q (x) = 1 I(Q) Qρ∈D Q,ρ f (x)1 Qρ (x)
Combining the previous equation with Eq. (7.7), it holds :

∀x ∈ Q, p Q (x) -p Q,ρ (x) = 1 I(Q) Qρ∈D Q,ρ (f (x) - I(Qρ) |Qρ| )1 Qρ (x) (7.72)
1. Distance between f and its mean on a small cube.

Let Q ρ ∈ D Q,ρ and x ∈ Q ρ , it holds |f (x) - I(Qρ) |Qρ| | ≤ Lip ∞ (f ) ρ. (7.73)
Indeed, expanding the mean, we get f (x) -

I(Qρ) |Qρ| = 1 |Qρ| Qρ (f (x) -f (y)) dy. Moreover, |f (x) - f (y)| ≤ Lip ∞ (f ) x -y ∞ .
Plugging that back in the previous equation and using the fact that xy ∞ ≤ ρ on Q ρ , we get Eq. (7.73) 2. Bounds on the total variation and L 2 distances. Using Eqs. (7.72) and (7.73), we immediately get

Q |p Q (x) -p Q,ρ (x)|dx = 1 I(Q) Qρ∈D Q,ρ Qρ |f (x) - I(Qρ) |Qρ| |dx ≤ |Q| Lip ∞ (f ) ρ I(Q)
.

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS 3. Bound on the Wasserstein norm W p . Consider the following density on

Q × Q: γ(x, y) = 1 I(Q) Qρ∈D Q,ρ f (x)1 Qρ (x) 1 |Qρ| 1 Qρ (y). (7.74)
A simple computation shows that γ ∈ Π(p Q , p Q,ρ ) (see the work by [START_REF] Santambrogio | Optimal Transport for Applied Mathematicians[END_REF] and Eq. (7.33)), i.e. that its marginals are p Q and p Q,ρ . Hence, by definition Eq. (7.33), we have

W p p (p Q , p Q,ρ ) ≤ 1 I(Q) Qρ∈D Q,ρ Qρ×Qρ |x -y| p f (x) |Qρ| dxdy.
Now using the fact that if x, y ∈ Q ρ , we have xy ≤ √ dρ as Q ρ is a hyper-rectangle with all sides of length less than or equal to ρ, we finally get :

W p (p Q , p Q,ρ ) ≤ √ dρ
4. Hellinger distance bound. Note that we could get a looser bound using Eq. ( 7.36) which only relies on the Lipschitz constant of f and not on that of √ f . Here, we concentrate on that case.

Let Q ρ ∈ D Q,ρ . By the intermediate value theorem, there exists z ∈ Q ρ such that f (z) = I(Qρ) |Qρ| and hence for any x ∈ Q ρ , it holds f (x) - I(Qρ) |Qρ| = f (x) -f (z) ≤ Lip ∞ ( f ) x -z ∞ ≤ Lip ∞ ( f ) ρ.
Bounding the distance between p Q,ρ and p Q by decomposing on dyadic hyper-rectangles using the previous expression, it holds

H(p Q , p Q,ρ ) 2 = Qρ∈D Q,ρ Qρ f (x) I(Q) - I(Qρ) |Qρ|I(Q) 2 dx = 1 I(Q) Qρ∈D Q,ρ Qρ f (x) - I(Qρ) |Qρ| 2 dx ≤ (Lip ∞ ( √ f ) ρ) 2 I(Q) Qρ∈D Q,ρ Qρ 1 dx = |Q| I(Q) Lip ∞ ( f ) ρ 2 .

7.D .4 Time complexity

In the Theorem 7.1, we measure the cost of the algorithm in terms of evaluation of integrals of the PSD model and in particular in the number of calls to the erf function (or subtractions) in the computation of such integrals. The fact that this is the true bottleneck of the algorithm can be seen in Sec. 7.D .4, as integrals take 95% of the CPU time.

7.E A general method of approximation and sampling

In this section, we prove proposition 7.1 and Theorem 7.3 using mainly results by [START_REF] Rudi | Psd representations for effective probability models[END_REF]. We introduce those results sequentially, showing the how each one is a building block towards the final result. For this section, fix a probability distribution p on the set X = (-1, 1) d (this is for the sake of simplicity; any hyper-rectangle could do), and assume that Assumption 7.1 holds for a certain β ∈ N, β > 0, i.e. there exists J ∈ N and q 1 , ..., q J ∈ W β 2 (X ) ∩ L ∞ (X ) such that p = j q 2 j . In this section, this probability distribution p is only known through a function f p proportional to its density. Denote with Z p > 0 this proportionality constant, i.e. f p /Z p = p, and with f j the renormalized q j : q j / Z p = f j s.t. f p = j f 2 j . Our goal is to be able to generate i.i.d. samples from a distribution as close as possible to p.

To do so, we first approximate f p by a Gaussian PSD model f τ,m,λ = f (•; A τ,m,λ , X m , η) where η = τ 1 d and τ > 0, X m ∈ R m×d is obtained as ( x 1 , ..., x m ) from m i.i.d. uniform samples from X , and A τ,m,λ is obtained by solving the problem Eq. (7.15) which we rewrite here for a given λ > 0 :

min A∈S + (R m ) X f (x; A, X, η) 2 dx -2 n i=1 f p (x i )f (x i ; A, X, η) + λ K 1/2 AK 1/2 F , (7.15)
where K = K Xm,η and the (x i ) 1≤i≤n represented by X ∈ R n×d are n i.i.d. samples from the uniform distribution on X .

The parameters τ, m, n, λ are selected in order to have an ε approximation of the probability p.

Using the fact that we can easily compute integrals of Gaussian PSD models, we can easily have access to

p τ,m,λ = f τ,m,λ / Z τ,m,λ where Z τ,m,λ = f τ,m,λ L 1 (X ) = X f τ,m,λ (x) dx.
We then apply algorithm 2 to p τ,m,λ , the hyper-rectangle X , the desired number of samples N and a certain ρ controlling the size of the dyadic decomposition of X in order to sample from a distribution whose total variation distance to p is less than a constant times ε.

Existence of a compressed ε-close Gaussian PSD model. We start by invoking Theorem 7.6 in order to obtain an ε-approximation of f p in the form of a general PSD f τ, with associated operator M τ, ∈ S + (H η ). This PSD model can then be compressed using a compression operator as described in Appendix 7.C .2. This is the object of the following proposition.

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS

Proposition 7.8 (Compression of M τ, ). Let ε ∈ (0, 1], τ ≥ ε -2/β and define η = τ 1 d ∈ R d . Let
M τ, be given by Theorem 7.6 applied to f p and satisfying Eq. (7.63) and f τ, the corresponding PSD model.

Let m ∈ N, X m ∈ R m×d be a data matrix corresponding to vectors x 1 , ..., x m which are sampled independently and uniformly from X , and P η,m be the associated orthogonal projection in H η . Let M τ,m, := P η,m M τ, P η,m be the operator associated to the compressed PSD model f τ,m, of f τ, onto X m (see Eq. (7.49) and Eq. (7.50) for the definitions).

Let δ ∈ (0, 1]. If one of the two following are true

m ≥ C 1 τ d/2 log C 2 ε + d 2 log τ d log C 3 δ + d 2 log τ + log log C 2 ε ; (7.75) m ≥ C 1 ε -d/β log C 2 ε d log C 2 ε + log C 3 δ , τ = ε -2/β (7.76)
then with probability at least 1 -δ, it holds

f τ, -f τ,m, L 2 (X ) ≤ 2 d f τ, -f τ,m, L ∞ (X ) ≤ 2C f p sos,R d ,β ε Tr ( M τ,m, ) ≤ Tr (M τ, ) ≤ C f p sos,R d ,β τ d/2 (7.77)
The constants C, C 1 , C 2 , C 3 , C 1 depends only on d, β, and not on τ, ε, m, δ.

Proof. Using Eq. (7.63) in Theorem 7.6 applied to f p , we see that if ε ≤ 1 and τ ≥ ε -2/β , there exists constants C 4 , C 5 depending only on d ,β, and not on τ, ε such that f (

•; M τ, , φ η ) - f p L 2 (X ) ≤ C 4 f p sos,R d ,β ε and Tr (M τ, ) ≤ C 5 f p sos,R d ,β τ d/2 (we set C 5 = C 2 (1 + e C 3 ) where C 2 , C 3 are introduced in Theorem 7.6). Now setting ρ = ε 2 d τ d/2
which is less than 1 since ε ≤ 1 and τ ≥ ε -2/β ≥ 1, we can apply Lemma 7.3 and hence, with probability at least 1 -δ, if

m ≥ C 1 τ d/2 (log C 2 τ d/2 ε ) d log C 3 δ + log τ + log log C 2 τ d/2 ε , (7.78) with C 1 ← C 1 from Lemma 7.3, C 2 ← max(e, C 2 2 d )
where C 2 is given by Lemma 7.3 and

C 3 ← C 3 from Lemma 7.3, it holds sup x∈X (I -P η,m )φ η (x) ≤ ρ (hence C 1 , C 2 , C 3 depend only on d).
1. Let us now show that Eq. (7.78) is implied by Eq. (7.75). Let us bound :

log log C 2 τ d/2 ε = log log C 2 ε 1 + d/2 log τ log C 2 ε = log log C 2 ε + log 1 + d/2 log τ log C 2 ε ≤ log log C 2 ε + d 2 log τ, where the last inequality is obtained since log(1 + t) ≤ t and C 2 /ε ≥ C 2 ≥ e by definition of C 2 and since ε ≤ 1. Setting C 1 = 3C 1 , C 2 = C 2 and C 3 = C 3 ,
it is therefore clear that Eq. (7.75) implies Eq. (7.78).

2. Moreover, Eq. (7.75) is in turn implied by Eq. (7.76). Indeed, in the case where τ = ε -2/β , we have the bound log log

C 2 ε + d 2 log τ ≤ log C 2 ε + d 2 log τ = log C 2 ε + d β log 1 ε ≤ (1 + d/β) log C 2 ε since C 2 ≥ e ≥ 1. Thus, taking C 1 = C 1 (1 + d/β) d+1
, Eq. (7.76) implies Eq. (7.75).

7.E . A GENERAL METHOD OF APPROXIMATION AND SAMPLING 339 3. If Eq. (7.78) holds, then Eq. (7.77) holds with probability at least 1 -δ. Indeed, for the first part, since Eq. (7.78) holds, with probability at least 1-δ, sup

x∈X (I -P η,m )φ η (x) ≤ ρ = ε 2 d τ d/2
Moreover, using Eq. (7.61) combined with the fact that for any x ∈ X ,

|f (x; M τ, , φ η )| = | φ η (x), M τ, φ η (x) | ≤ φ η (x) 2 Hη M τ, = M τ, , since φ η (x) 2 = k η (x, x) = 1, it holds : f (•; M τ, , φ η ) -f (•; M τ,m, , φ η ) L ∞ (X ) ≤ M τ, (ρ 2 + ρ) ≤ 2 M τ, ρ.
We conclude using the fact that for any operator M , and any orthogonal projection P , M ≤ Tr (M ) and Tr (P M P ) ≤ Tr (M ). We then conclude the proof by using the definition of ρ and the fact that X 1 dx = 2 d , and setting C ← C 5 .

Combining Eq. ( 7.63) and Eq. ( 7.77), we see that if m is large enough, one can find a Gaussian PSD model of the form f τ,m, = f ( • ; A τ,m, , X m , τ 1 d ) (where A τ,m, is defined through Eq. (7.50) from M τ,m, ) which is C f p sos,R d ,β ε close to f p and whose trace is controlled. It now remains to compare the performance of f τ,m, with the Gaussian PSD model learned from evaluations of f p , f τ,m,λ , which is the solution of Eq. ( 7.15) which we can compute.

Controlling the L 2 distance between f τ,m,λ and f p . This theorem is a rewriting of Theorem 7 by [START_REF] Rudi | Psd representations for effective probability models[END_REF], but with the point of view of ε instead of n.

Proposition 7.9 (Performance of f τ,m,λ ). Let n ∈ N and let (x 1 , ..., x n ) be n i.i.d. samples from p. Let δ ∈ (0, 1] and ε ≤ 1 e . Assume n satisfies

n ≥ ε -(d+2β)/β log d 1 ε log 2 δ , (7.79)
Let m ∈ N and assume m satisfies Eq. (7.76) and let X m ∈ R m×d be a data matrix corresponding to vectors x 1 , ..., x m which are sampled independently and uniformly from X . Let λ = ε 2(β+d)/β , τ = ε -2/β and f τ,m,λ be the Gaussian PSD model associated to the solution A τ,m,λ of Eq. (7.15) with X m , λ, τ . With probability at least 1 -2δ, the following holds

f τ,m,λ -f p 2 L 2 (X ) + λ M τ,m,λ 2 F 1/2 ≤ C f p sos,X ,β ε, (7.80)
where C is a constant depending only on d, β, and not on ε, δ, λ, m, τ, f p .

Proof. We start by applying the same reasoning as in the proof of Theorem 7 by [START_REF] Rudi | Psd representations for effective probability models[END_REF].

Note that since τ = ε -2/β and Eq. (7.76) is satisfied, with probability at least 1 -δ, it holds

f τ,m, -f τ,m,λ L 2 (X ) ≤ 2C 1 f p sos,R d ,β ε (where C 1 ← C from Eq. (7.77)) and hence f p - f τ,m,λ L 2 (X ) ≤ (C 0 + 2C 1 ) f p sos,R d ,β ε, (where C 0 ← C 1 from Theorem 7.6
). C 0 , C 1 are both constants depending only on d, β. Moreover, since the Frobenius norm is bounded by the trace norm, by definition of τ , we also have

M τ,m,λ F ≤ Tr ( M τ,m,λ ) ≤ C 1 f p sos,R d ,β τ d/2 ≤ C 1 f p sos,R d ,β ε -d/β .
We can modify Theorem E.2 by [START_REF] Rudi | Psd representations for effective probability models[END_REF] by taking v = 1 n n i=1 f p (x i )ψ η (x i ) and v = X f p (x)ψ η (x) dx; all the formulas then remain true and adapt to our problem Eq. (7.15). 340 CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS Applying Theorem E.2 by [START_REF] Rudi | Psd representations for effective probability models[END_REF] to A τ,m, and using Lemma E.3 by [START_REF] Rudi | Psd representations for effective probability models[END_REF] to simplify notation, as well as the bound on the term

Q -1/2 λ ( v -v) combining Lemma E.4 (with ζ = Q -1/2 λ f p (x)ψ η (x)
) using s = d and Lemma E.5 (again, for more details, see part 2 of the proof of Theorem 7 by [START_REF] Rudi | Psd representations for effective probability models[END_REF]) and using the fact that √ a + b ≤ √ a + √ b, a, b ≥ 0, with probability at least 1 -δ, it holds :

f τ,m,λ -f p 2 L 2 (X ) + λ M τ,m,λ 2 F 1/2 ≤ f τ,m, -f p L 2 (X ) + √ λ M τ,m, F + C 2 f p sos,R d ,β log 2 δ nλ 1/4 + C 3 f p sos,R d ,β τ d/4 log 1 λ d/2 log 2 δ 1/2 n 1/2 , (7.81)
where C 2 and C 3 are constants which depend only on d.

Note that in the proof of Lemma E.4 by [START_REF] Rudi | Psd representations for effective probability models[END_REF], ζ is bounded in essential supremum and standard deviation by f p L ∞ (X ) × a quantity independent of f p which is then bounded, hence the previous concentration bound since

f p L ∞ (X ) ≤ f p sos,R d ,β .
Now combining both events in a union bound, and plugging in the fact that λ = ε 2β+2d β and τ = ε -2/β , we see that with probability at least 1 -2δ, the left hand term is bounded by the following quantity:

ε f p sos,R d ,β (C 0 + 3C 1 + T ), (7.82)
T = C 2 ε - 3β+d 2β log 2 δ n + C 3 ε -(d+2β)/2β 2β+2d β log 1 ε d/2 log 2 δ 1/2 n 1/2 .
Now the goal is to bound the term T . Note that as soon as ε ≤ e -1 and δ

≤ 2, if Y = ε -(d+2β)/β log d 1 ε log 2 δ n , then it holds T ≤ C 2 log 2 2 Y + C 3 √ Y . Now note that Y ≤ 1 iif n ≥ ε -(d+2β)/β log d 1 ε log 2 δ .
The theorem therefore holds with C ← 1 + 3C 1 + C 2 / log 2 2 + C 3 . Finally, the fact that all bounds involving f p sos,R d ,β can be replaced, up to constants depending only on β, d, by the norm f p sos,X ,β , is simply a consequence of proposition 7.4.

We now come to the final part of our section detailing the proof of proposition 7.10 and Theorem 7.8, which consists in approximately sampling from the learnt model f τ,m,λ using algorithm 2 with well chosen parameters.

Performance of the re-normalized probability measure p τ,m,λ . We start off with a technical lemma.

Lemma 7.9 (Technical lemma). Let • be a norm on a vector space E, and let x, y ∈ E \ {0}. Then it holds:

a a -b b ≤ 2 a -b a . (7.83) Moreover, if a -b ≤ a /2, it holds a b ≤ 2. (7.84) Proof. Introduce the quantity b a in order to get a a -b b ≤ a a -b a + b a -b b = a -b a + b 1 a -1 b .
One concludes by writing

1 a -1 b = | b -a | a b ≤ b -a a b ,
where the last inequality is simply the triangle inequality. This concludes the proof of Eq. (7.83).

The proof of Eq. (7.84) is simply the result of applying the bound

1 b ≤ 1 a -b-a ≤ 2 a .
Proposition 7.10 (Performance of p τ,m,λ ). Let p be a probability density w.r.t. the Lebesgue measure on X = (-1, 1) d satisfying Assumption 7.1 for a certain β. There exists ε 0 > 0 depending only on d, β, and p sos,X ,β and C 1 , C 2 , C 1 , C 2 , C 3 depending only on d, β such that the following holds.

Let n ∈ N and let (x 1 , ..., x n ) be n i.i.d. samples selected uniformly at random from X . Let δ ∈ (0, 1] and ε ≤ ε 0 , λ = ε 2(β+d)/β and τ = ε -2/β . Assume n satisfies Eq. (7.79), i.e.

n ≥ ε -(d+2β)/β log d 1 ε log 2 δ . (7.79)
Let m ∈ N and assume m satisfies Eq. (7.76), i.e.

m ≥ C 1 ε -d/β log C 2 ε d log C 2 ε + log C 3 δ , (7.76)
and let X m ∈ R m×d be a data matrix corresponding to vectors x 1 , ..., x m which are sampled independently and uniformly from X .

Let f τ,m,λ be the Gaussian PSD model associated to the solution A τ,m,λ of Eq. (7.15) with X m , λ, τ and let p τ,m,λ be the associated probability density on X (i.e. the re-normalization of f τ,m,λ ). Let R τ,m,λ be PSD operator on H η associated to p τ,m,λ . With probability at least 1 -2δ, it holds

d T V ( p τ,m,λ , p) ≤ C 1 p sos,X ,β ε, R τ,m,λ F ≤ C 2 p sos,X ,β ε -d/β . (7.85)
Proof. Since the assumptions of proposition 7.9 are satisfied, we have by Eq. (7.80) the existence of a constant C depending only on d, β, and not on ε, δ, λ, m, τ, f p , such that

f τ,m,λ -f p L 2 (X ) ≤ C f p sos,X ,β ε, M τ,m,λ F ≤ C f p sos,X ,β ε -d/β , (7.86)
where we have used the fact that λ = ε 2+2d/β . Now using the fact that

• L 1 (X ) ≤ 2 d/2 • L 2 (X )
(by Cauchy-Schwarz inequality), Eq. (7.86) shows in particular that f τ,m,λ -f p L 1 (X ) ≤ 2 d/2 C f p sos,X ,β ε. Now applying Eq. (7.83) of Lemma 7.9, using the fact that p τ,m,λ = f τ,m,λ / f τ,m,λ L 1 (X ) and p = f p / f p L 1 (X ) , it holds

d T V ( p τ,m,λ , p) = p τ,m,λ -p L 1 (X ) ≤ 2 f τ,m,λ -f p L 1 (X ) / f p L 1 (X ) (7.87) ≤ 2 d/2+1 C f p sos,X ,β / f p L 1 (X ) ε. CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS Since p = f p / f p L 1 (X ) , we have f p sos,X ,β / f p L 1 (X ) = p sos,X ,β . This shows d T V ( p τ,m,λ , p) ≤ 2 d/2+1 C p sos,X ,β ε. Now set ε 0 = min(e -1 , 2 -d/2-1 C -1 p -1 sos,X ,β ). If ε ≤ ε 0 , we have 2 d/2 C f p sos,X ,β ε ≤ f p L 1 (X ) /2 and hence f τ,m,λ -f p L 1 (X ) ≤ f p L 1 (X ) /2
. By Eq. (7.84) of Lemma 7.9, we therefore have

f p L 1 (X ) / f τ,m,λ L 1 (X ) = Z p / Z τ,m,λ ≤ 2. Now since R τ,m,λ = M τ,m,λ / Z τ,
m,λ , using Eq. ( 7.86), it holds R τ,m,λ ≤ C 2 p sos,X ,β ε -d/β where C 2 = 2C, which depends only on β, d.

Theorem 7.8 (Performance of p sample ). Under the assumptions and notations of the previous theorem (proposition 7.1), there exists a constant C 3 depending only on d, β, such that the following holds.

Let p τ,m,λ be given by the previous proposition. Let p sample be the dyadic approximation of p τ,m,λ on Q = X = (-1, 1) d and of width ρ (see Eq. (7.7)). Recall from Theorem 7.1 that algorithm 2 applied to Q = (-1, 1) d , N, ρ returns N i.i.d. samples from p sample .

If on the one hand ρ is set to ε 1+(d+1)/β , then with probability at least 1 -2δ,

d T V ( p τ,m,λ , p sample ) ≤ C 3 p sos,X ,β ε, d T V (p, p sample ) ≤ (C 1 + C 3 ) p sos,X ,β ε. (7.88)
If on the other ρ is set adaptively to guarantee d T V (p sample , p τ,m,λ ) ≤ ε as in Remark 24 then with probability at least 1 -2δ, ρ ≥ ε 1+(d+1)/β /(C 3 p sos,X ,β ), and hence

d T V ( p τ,m,λ , p sample ) ≤ ε, d T V (p, p sample ) ≤ C 1 p sos,X ,β ε + ε. (7.89)
In any case, this guarantees that the complexity in terms of erf computations is bounded by

O(N m 2 log 1 ρ ) = O N ε -2d/β log 2d+1 1 ε log 1 ε + log 1 δ , (7.90)
where the O notations is taken with constants depending on d, β, p sos,X ,β .

Proof. Let us bound Lip

∞ ( p τ,m,λ ). Note that Lip ∞ ( p τ,m,λ ) ≤ sup x∈X d k=1 ∂ k p τ,m,λ (x). Using Lemma 7.5, we get Lip ∞ ( p τ,m,λ ) ≤ d2 3/2 √ τ R τ,m,λ . Using the fact that τ = ε -2/β
and that by Eq. (7.85

), R τ,m,λ ≤ R τ,m,λ F ≤ C 2 p sos,X ,β ε -d/β , we therefore have Lip ∞ ( p τ,m,λ ) ≤ 2 3/2 dC 2 p sos,X ,β ε -(d+1)/β . Hence, applying Theorem 7.7 to p τ,m,λ , we get d T V (p sample , p τ,m,λ ) ≤ 2 3/2 2 d dC 2 p sos,X ,β ε -(d+1)/β ρ. (7.91)
On the one hand, if we use algorithm 2 with ρ = ε

1+ (d+1)
β , by the previous equation, we get

d T V (p sample , p τ,m,λ ) ≤ 2 3/2 d 2 d C 2 p sos,X ,β ε.
If on the other hand we find ρ adaptively by computing a bound

Lip(A) = 2 3/2 τ 1/2 d K 1/2 AK 1/2 = 2 3/2 τ 1/2 d R τ,m,λ F
(A) ≤ 2 3/2 dC 2 p sos,X ,β ε -(d+1)/β , we will get ρ ≥ ε 1+(d+1)/β 2 d+3/2 d C 2 p sos,X ,β
and hence d T V (p sample , p τ,m,λ ) ≤ ε. The last point is just a consequence of Theorem 7.1 and the bound on m in Eq. (7.76).

7.F Approximation and sampling using a rank one PSD model

In this section, we prove the results in Sec. 7.4 .2, i.e. proposition 7.2 and Theorem 7.4.

For this section, fix a probability which has density p with respect to the Lebesgue measure dx on X = (-1, 1) d , (this is for the sake of simplicity; any hyper-rectangle could do), and assume that Assumption 7.2 holds for a certain β ∈ N, β > 0, i.e. there exists q ∈ W β 2 (X ) ∩ L ∞ (X ) such that p = q 2 . This is the case, for instance, when p ∝ e -V (x) where V is β times differentiable.

One of the main advantages of our method will be to deal with probability measures which are known up to a constant; therefore, in this section, we take f p such that p = f p /Z(f p ) where Z(f p ) = X f p (x)dx. Assuming Assumption 7.2 holds, we take g p ∈ W β 2 (X ) ∩ L ∞ (X ) such that g 2 p = f p as and assume that p is only known through function evaluations of g p , i.e. we can evaluate the function g p (x) for any x ∈ X .

Once again, our goal is to be able to generate N i.i.d. samples from a distribution which is ε-close to p, in a sense which we will define. To do so, we first approximate g p by a Gaussian linear model g τ,m,λ = g(•; a τ,m,λ , X m , η) (see Eq. ( 7.2) for a definition) where η = τ 1 d for some τ > 0, X m ∈ R m×d is obtained as ( x 1 , ..., x m ) from m i.i.d. uniform samples from X , and a τ,m,λ is obtained by solving the problem Eq. (7.20) which we rewrite here for a given λ > 0 and for n i.i.d. samples (x 1 , .., x n ) sampled uniformly from X : (7.20) This yields a Gaussian linear model

a τ,m,λ = arg min a∈R m 1 n n i=1 (g(x i ; a, x m , τ 1 d ) -g p (x i )) 2 + λa K Xm,η a.
g τ,m,λ ∈ H η of g p . Since g 2 τ,m,λ = f τ,m,λ is a PSD model (indeed f τ,m,λ = f (•; A τ,m,λ , X m , τ 1 d ) with A τ,m,λ = a τ,
m,λ a τ,m,λ ), we can see f τ,m,λ as a Gaussian PSD model of f p , and hence its renormalized version p τ,m,λ as a PSD model of p.

The parameters τ, m, λ, n are selected in order to have an ε approximation of the probability p.

Furthermore, note that the first term in the optimized quantity in Eq. (7.20) is an empirical version of the quantity

1 |X | X f τ,m,λ (x) -f p (x) 2 dx ≤ 1 |X | X | g τ,m,λ (x) -g p (x)| 2 dx.
This quantity is related to Hellinger distance H(p, p τ,m,λ ) defined in Eq. (7.32).

This will therefore be the natural measure in which to express the quality of the approximation p τ,m,λ of p in this section.

The bound obtained on the performance of p τ,m,λ can be decomposed into two steps.

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS

• We start by bounding the distance between any g ∈ H η and g τ,m,λ in Theorem 7.9.

• We then select a g τ,ε which is ε-close to g p , and use it as a reference point in order to bound the distance between g p and g τ,m,λ . To do so, we need to apply different concentration inequalities to obtain a final bound in terms of performance for both f τ,m,λ with respect to f p and p τ,m,λ with respect to p in Hellinger distance in proposition 7.2.

Bound on the performance of g τ,m,λ compared to an arbitrary function g. Here, we adapt Theorem 2. by [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF].

Theorem 7.9 (Bounding the error [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]).

Let η ∈ R d ++ and g ∈ H η . C 1/2 η,λ (g -g τ,m,λ ) ≤ θ 2 1 θ 2 g p -S η g R n (7.92) + g Hη (1 + θ 1 θ 2 + θ 2 1 ) sup x∈X (I -P η,m )φ η (x) + λ 1/2 ,
where

θ 1 = C -1/2 η,λ C 1/2 η,λ , θ 2 = C 1/2 η,λ C -1/2 η,λ and g p = (g p (x i )/ √ n) 1≤i≤n ∈ R n .
Proof. Let g ∈ H η . We can apply a modification of Theorem 2 by [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]. Indeed, consider in the notations by [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF] the loss

E(f ) = C 1/2
η (fg) Hη , and note that the assumptions are satisfied with ν = 0 and R = g Hη , since g minimizes E and C -0 g Hη = g Hη . Moreover, note that in the proof of that theorem, one can replace C η by C η,λ without changing the result (indeed, in the proof, one always bounds

C 1/2 η ≤ C 1/2 η C -1/2 η,λ C 1/2 η,λ ≤ C 1/2 η,λ
). Thus, in that setting, without combining the "constant" terms in the bounds and looking into the proof of Theorem 2 by [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF], it holds

C 1/2 η,λ ( g τ,m,λ -g) ≤ θ 2 1 C -1/2 η,λ S * η ( g p -S η g) + R(1 + θ 1 θ 2 ) (I -P η,m )C 1/2 η,λ + Rθ 2 1 λ 1/2 , (7.93)
where

θ 1 = C -1/2 η,λ C 1/2 η,λ and θ 2 = C 1/2 η,λ C -1/2 η,λ . Note that C -1/2 η,λ S * η ( g p -S η g) ≤ C -1/2 η,λ S * η g p -S η g R n ≤ θ 2 g p -S η g R n since C -1/2 η,λ S * η 2 = C -1/2 η,λ C η C -1/2 η,λ ≤ C -1/2 η,λ C η,λ C -1/2 η,λ = θ 2 2 . Moreover, using the definition of C η , it holds (I -P η,m )C 1/2 η,λ 2 = (I -P η,m )C η (I -P η,m ) + λ(I -P η,m ) ≤ 1 |X| X (I -P η,m )φ η (x) ⊗ φ η (x)(I -P η,m ) dx + λ (I -P η,m ) ≤ sup x∈X (I -P η,m )φ η (x) 2 + λ.
Combining these results and using the fact that √ a + b ≤ √ a + √ b for any a, b ≥ 0, we get the bound.

Performance of p τ,m,λ . We can now state the main results of this section, i.e. the bound on the performance of p τ,m,λ .

Proposition 7.11 (Performance of p τ,m,λ ). Let p be a probability density on X = (-1, 1) d , and assume p = q 2 and q ∈ L ∞ (X ) ∩ W β 2 (X ) for some β ≥ 0. Let ν > min(1, d/(2β)). There exists a constant ε 0 depending only on q L ∞ (X ) , q W β 2 (X ) , β, d, constants C 1 , C 2 , C 3 , C 4 , C 5 depending only on β, d and a constant C 1 depending only on β, d, ν such that the following holds.

Let δ ∈ (0, 1] and ε ≤ ε 0 , and assume (x 1 , ..., x n ) and ( x 1 , ..., x m ) are respectively n and m uniform i.i.d. samples on X , satisfying

m ≥ C 1 ε -d/β log d C 2 ε log C 3 δε (7.94) n ≥ C 1 ε -2 ν log 8 δ (7.95) Let τ = ε -2/β , η = τ 1 d and λ = ε 2+d/β
. Let a τ,m,λ ∈ R n be the vector obtained by solving Eq. (7.20) and g τ,m,λ ∈ H η the associated Gaussian linear model (see Eq. (7.2)). Let f τ,m,λ = g 2 τ,m,λ be the associated Gaussian PSD model, Z τ,m,λ = X f τ,m,λ (x) dx be the normalizing constant, and p τ,m,λ = f τ,m,λ / Z τ,m,λ be the renormalized PSD model, which is a probability density. Let R τ,m,λ be PSD operator in S + (H η ) associated to p τ,m,λ .

With probability at least 1 -3δ, it holds

H( p τ,m,λ , p) ≤ C 4 q L ∞ (X )∩W β 2 (X ) ε Tr ( R τ,m,λ ) = g τ,m,λ Z τ,m,λ 2 Hη ≤ C 5 q 2 L ∞ (X )∩W β 2 (X ) ε -d/β , (7.96) where • L ∞ (X )∩W β 2 (X ) = max( • W β 2 (X ) , • L ∞ (X ) ).
Proof. Let τ > 0, and define η = τ 1 d . By proposition 7.4, we can extend g p to the whole of R d and there exists an constant C such that

g p W β 2 (R d ) ≤ g p W β 2 (X ) and g p L ∞ (R d ) ≤ C g p L ∞ (X )
. We still denote with g p such an extension. Let g τ,ε be given by proposition 7.7 when approximating g p .

Setting τ = ε -2/β and λ = ε 2β+d β , since we assume ε ≤ 1, Eq. (7.51) gives us two constants C 1 , C 2 depending only on β, d such that

g τ,ε -g p L 2 (R d ) ≤ ε g p W β 2 (R d ) g τ,ε -g p L ∞ (R d ) ≤ C 1 ε 1-ν g p • g τ,ε Hη ≤ C 2 g p W β 2 (R d ) τ d/4 = C 2 g p W β 2 (R d ) ε - d 2β .
1. Bounding g p -S η g τ,ε R n Apply Theorem 3 by [START_REF] Boucheron | Concentration Inequalities: a non Asymptotic Theory of Independence[END_REF], reformulated in Proposition 10 by [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]. Consider the random variable

ζ = (g τ,ε -g p )(X) 2 -1 |X | g τ,ε -g p 2 L 2 (X )
where X follows the uniform law on X . Then

|ζ| ≤ g τ,ε -g p 2 L ∞ (X ) almost surely, and E[ζ 2 ] ≤ g τ,ε -g p 2 L ∞ (X ) 1 |X | g τ,ε -g p 2 L 2 (X )
. Applying the concentration bound yields that with probability at least 1 -δ, it holds

g p -S η g τ,ε 2 R n -1 |X | g τ,ε -g p 2 L 2 (X ) ≤ 2 gτ,ε-gp 2 L ∞ (X ) log 1 δ 3n + 2 gτ,ε-gp 2 L ∞ (X ) 1 |X | gτ,ε-gp 2 L 2 (X ) log 1 δ n
, and thus

g p -S η g τ,ε 2 R n ≤   1 √ |X | g τ,ε -g p L 2 (X ) + g τ,ε -g p L ∞ (X ) 2 log 1 δ n   2 .
Hence, by Eq. (7.51), and because |X | = 2 d , there exists two constants C 3 and C 4 depending only on d and β such that with probability at least 1 -δ, it holds 2β) Using Lemma 7.3 and proceeding in the same way as in point 2 of the proof of proposition 7.8, we see that there exists constants C 5 , C 6 , C 7 depending only on d and β such that as soon as

g p -S η g τ,ε R n ≤ C 3 ε g p W β 2 (R d ) + C 4 ε g p • log 1 δ ε ν √ n . (7.97) 2. Guaranteeing sup x∈X (I -P η,m )φ η (x) ≤ λ 1/2 = ε 1+d/(
m ≥ C 5 ε -d/β log C 6 ε d log C 7 δε , (7.98)
it holds sup x∈X (I -P η,m )φ η (x) ≤ λ 1/2 with probability at least 1 -δ.

3. Finding a lower bound for C η This will be necessary in the next bound. Let v(z) = k η (0, z) = e -τ z 2 . Then v Hη = 1 and

C 1/2 η v 2 H = 1 |X | X |v(x)| 2 dx = 1 |X | 1 -1 e -2τ t 2 dt d ≥ 1 2 d 1 -1 e -2t 2 dt d τ -d/2 = C 8 τ -d/2 ,
where the last inequality comes from the fact that τ ≥ 1 since ε ≤ 1. Hence,

C η ≥ C 8 τ -d/2
where C 8 is a constant depending only on d. Hence, as soon as λ ≤ C 8 τ -d/2 which rewrites ε ≤ √ C 8 , it holds λ ≤ C η .

4. Bounding θ 1 , θ 2 . Using the same reasoning as that of Proposition 2. by [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF],

if b = C -1/2 η,λ ( C η -C η )C -1/2 η,λ
, then θ 1 ≤ 1/(1 -b) and θ 2 2 ≤ 1 + b. Bounding b can be done using Proposition 8 by [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF]: if λ ≤ C η , and δ ∈ (0, 1] it holds, with probability at least 1 -δ :

C -1/2 η,λ ( C η -C η )C -1/2 η,λ ≤ 2(1 + N ∞ (λ)) log 8 λδ 3n + 2N ∞ (λ) log 8 λδ n , (7.99)
where we have used the fact that Tr ( 2s) for any s > d/2 where C 9 depends only on s, d by a proof completely analogous as that of Step 2 of Lemma E.4 by [START_REF] Rudi | Psd representations for effective probability models[END_REF]. Replacing the values of τ, λ yields :

C η ) ≤ 1. Note that N ∞ (λ) = sup x∈X C -1/2 η,λ φ η (x) 2 ≤ C 9 τ (s-d)d/(2s) λ -d/(
N ∞ (λ) ≤ C 9 ε - 2d(β+s)-d 2 2sβ
.

Note that the function γ

: s ∈]d/2, +∞[ → 2dβ+2ds-d 2 2sβ
is a homography and therefore reaches all the values ν strictly between 2 and d/β. Therefore, for any ν > ν, there exists a constant C 10 depending only on d and ν such that (1 + N ∞ (λ)) log 1 λ ≤ C 10 ε -2 ν . Hence, there exists a constant depending only on d, β, ν such that if n ≥ C 11 ε -2 ν log 8 δ , and if ε ≤ min(1/2, √ C 8 ) then b ≤ 1/3 (here we have bounded log 8 δλ by a constant times log 1 λ log 8 δ provided ε ≤ 1/2 and hence λ ≤ 1/4. Moreover, note that C 11 can be taken large enough, by Eq. (7.97), to guarantee the following, also with probability 1 -δ :

g p -S η g τ,ε R n ≤ C 3 ε g p W β 2 (R d ) + C 4 ε g p • .
(7.100)

5. Applying Theorem 7.9 to g τ,ε . Combining all the previous equations, we get that if

n ≥ C 11 ε -2 ν log 8 δ , ε ≤ min(1/2, √ C 8 ) and m ≥ C 5 ε -d/β log C 6 ε d log C 7
δε , it holds Eq. ( 7.100) and b ≤ 1/3 as well as sup x∈X (I -P η,m )φ η (x) ≤ λ 1/2 and hence, using the bound on g τ,ε , there exists a constant C 12 depending only on d, β such that

C 1/2 η,λ (g τ,ε -g τ,m,λ ) ≤ C 12 max( g p W β 2 (R d ) , g p • )ε.
Thus, using the bound on g τ,ε -g p L 2 (R d ) , and the fact that

gC η g = 1 |X | g 2 L 2 (X ) we get g p -g τ,m,λ L 2 (X ) ≤ C 13 max( g p W β 2 (R d ) , g p • )ε, g τ,m,λ Hη ≤ C 14 max( g p W β 2 (R d ) , g p • ) ε -d/2β . (7.101) 6. Bounding the performance of p τ,m,λ . Note that q = gp gp L 2 (X ) and p τ,m,λ = | g τ,m,λ | g τ,m,λ L 2 (X )
. Thus, using Eq. ( 7.83), it holds

H( p τ,m,λ , p) = gp gp L 2 (X ) - | g τ,m,λ | g τ,m,λ L 2 (X ) L 2 (X ) ≤ 2 g τ,m,λ -g p L 2 (X ) g p L 2 (X ) .
Hence, since q = g p / g p L 2 (X ) , we have by Eq. (7.101) :

H(p, p τ,m,λ ) ≤ 2C 13 max( q W β 2 (R d ) , q • )ε.
Moreover, by Eq. (7.84), if 2C 13 max( q

W β 2 (R d ) , q • )ε ≤ 1, then gτ,ε L 2 (X ) g τ,m,λ L 2 (X )
≤ 2 and hence again by Eq. (7.101), p τ,m,λ Hη ≤ 2C 14 max( q

W β 2 (R d ) , q • ) ε -d/2β
. Setting

ε 0 = min(1/2, C 8 , (2C 13 max( q W β 2 (R d ) , q • )) -1
), , we therefore have all the desired properties.

7. Replacing norms on R d with norm on X . To do so, we just use proposition 7.4, which does not change anything up to multiplicative constants depending only on d, β.

Theorem 7.10 (Performance of p sample ). Under the assumptions and notations of the previous theorem (proposition 7.11), there exists a constant C 6 depending only on d, β, such that the following holds. Let p τ,m,λ be given by the previous proposition. Let p sample be the dyadic approximation of p τ,m,λ on Q = X = (-1, 1) d and of width ρ (see Eq. (7.7)). Recall from Theorem 7.1 that algorithm 2 applied to Q = (-1, 1) d , N, ρ returns N i.i.d. samples from p sample .

If on the one hand ρ is set to ε 1+(d+2)/(2β) , then with probability at least 1 -3δ,

H( p τ,m,λ , p sample ) ≤ C 6 q L ∞ (X )∩W β 2 (X ) ε, H(p, p sample ) ≤ (C 4 + C 6 ) q L ∞ (X )∩W β 2 (X ) ε. (7.102)
If on the other ρ is set adaptively to guarantee H(p sample , p τ,m,λ ) ≤ ε as in Remark 24, then with probability at least 1 -3δ,

ρ ≥ ε 1+(d+2)/β /(C 6 q L ∞ (X )∩W β 2 (X ) ), H( p τ,m,λ , p sample ) ≤ ε, H(p, p sample ) ≤ (C 1 + 1)ε. (7.103)
In any case, this guarantees that the complexity in terms of erf computations is bounded by

O(N m 2 log 1 ρ ) = O N ε -2d/β log 2d+1 1 ε log 1 ε + log 1 δ , (7.104)
where the O notations is taken with constants depending on d, β, q L ∞ (X )∩W β 2 (X ) .

Proof. Let us bound Lip ∞ ( p τ,m,λ ). Note that since for any x, y ∈ X , it holds

p τ,m,λ (x) -p τ,m,λ (y) = || g τ,m,λ (x)| -| g τ,m,λ (y)||/ Z τ,m,λ ≤ | g τ,m,λ (x) -g τ,m,λ (y)|/ Z τ,m,λ , we have Lip ∞ ( p τ,m,λ ) ≤ Lip ∞ ( g τ,m,λ )/ Z τ,m,λ . Now Lip ∞ ( g τ,m,λ ) ≤ sup x∈X d k=1 ∂ k g τ,m,λ (x). Using Lemma 7.2, we get Lip ∞ ( g τ,m,λ ) ≤ d √ 2τ g τ,m,λ Hη . Using the fact that τ = ε -2/β
and that by Eq. (7.96), g τ,m,λ 2β) . Hence, applying Theorem 7.7 to p τ,m,λ , we get

Hη / Z τ,m,λ ≤ √ C 5 q L ∞ (X )∩W β 2 (X ) ε -d/(2β) , we therefore have Lip ∞ ( p τ,m,λ ) ≤ d √ 2C 5 q L ∞ (X )∩W β 2 (X ) ε -(d+2)/(
H(p sample , p τ,m,λ ) ≤ 2 d/2 d 2C 5 q L ∞ (X )∩W β 2 (X ) ε -(d+2)/(2β) ρ. (7.105)
On the one hand, if we use algorithm 2 with ρ = ε

1+ (d+2)
2β , by the previous equation, we get

H(p sample , p τ,m,λ ) ≤ 2 d/2 d √ 2C 5 ε.
If on the other hand we find ρ adaptively by computing an upper bound Lip(a) defined in

s.t. Lip(a) = √ 2τ d K 1/2 a = √ 2τ d g τ,m,λ / Z τ,m,λ ≥ Lip ∞ ( p τ,m,λ ) from p τ,m,λ and finding ρ such that 2 d/2 Lip(a) ρ = ε, we will get ρ ≥ ε 1+(d+2)/(2β) 2 d/2 d √ 2C 5 q L ∞ (X )∩W β 2 (X )
and hence H(p sample , p τ,m,λ ) ≤ ε. The last point is just a consequence of Theorem 7.1 and the bound on m in Eq. (7.94).

7.G . ADDITIONAL EXPERIMENTAL DETAILS

349

7.G Additional experimental details

As mentioned in Sec. 7.5 , we report in Fig. 7.4 an experiment in which we learn the density of the indicator function of [-1, 1] using algorithm 5.

Note that this is out of the setting of Theorem 7.4, as these bounds rely on the regularity of the target density which is not at all the case here.

However, in order to sample approximately from p as a rough approximation, algorithm 5 could be relevant : it shows that we must develop tools which analyse these algorithms beyond notions of regularity, with rougher objectives.

Densities target learnt

Samples from the PSD model problems (already highlighted in Sec. 2.1.4), as well as similar applications, the two methods are different on many aspects and have different behaviors and guarantees.

In the rest of this paragraph, we wil therefore define the general setting of optimization over measures, which is a general framework emcompassing both polynomial optimization and our contributions. We will then present a generic method to solve such problems for polynomials in Sec. 8.1 , developed by [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF]. In Sec. 8.2 , we will follow the same main steps as for the introduction of polynomial methods, to describe the method developed by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. We will end Sec. 8.2 by presenting the work by Marteau-Ferey, Bach, and Rudi (2022b), which gives the theoretical basis to extend our method to different problems. The idea is to highlight the high level ingredients of these two methods, in terms of formulation, guarantees and algorithm, to provide a basis for comparison in Sec. 8.3 . The goal is to make a bridge between the works of the polynomial optimization community and our own, as our method offers the potential to go beyond polynomials, while also facing challenges which appear when the structure is not as rigid as that of polynomials. This comparison will have important repercussion in chapter 11 as it opens up the problems handled by polynomial optimization to other classes of functions and domains.

Optimization over measures

Let us present informally Eqs. (2.25) and ( 8.1) as optimization problems over measures, and their dual counterpart. Note that here, we do not strictly establish duality. Assume that X is equipped with a topology. In this chapter only, denote with M (X ) the set of signed borel measures on X and with M + (X ) the set of non-negative signed measures (i.e. the set of finite measures). The "measure optimization" equivalent of Eq. ( 8.1) is :

inf X f (x)µ(dx) subject to µ ∈ M + (X ), X 1 µ(dx) = 1 (8.2)
This problem seems, of course, much harder than the original problem. However, it is convex and encompasses important information in the measure : if f is continuous, and has a non empty set Γ of minimizers, then the support of µ is concentrated on Γ, and in fact the set of measures minimizing Eq. ( 8.2) is exactly the set of probability distributions with support included in Γ. This fact will be crucial in Sec. 8.1 in order to extract the minimizer. The dual of Eq. ( 8.2) is simply Eq. (2.25) (see 1.2 by Lasserre ( 2010)) :

sup c subject to c ∈ R, g(x) = f (x) -c, g(x) ≥ 0, ∀x ∈ X .
(2.25)

Polynomial optimization

In this section, we will present so-called moment-SOS (for moment sum of squares) hierarchies, to solve the global optimization problem for polynomials. This section follows the works and result by [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF][START_REF] Lasserre | Global optimization with polynomials and the problem of moments[END_REF]; [START_REF] Henrion | The Moment-SOS Hierarchy[END_REF], to name a few. Note that [START_REF] Parrilo | Semidefinite programming relaxations for semialgebraic problems[END_REF] also studied these kinds of hierarchies.

Notations. In this section, we denote with R[x 1 , ..., x d ] = R[x], x = (x 1 , ..., x d ) the set of polynomial functions on R d with real coefficients, and with R r [x] the set of polynomial functions on R d with degree at most r. Given a polynomial function f , we will denote with f α the coefficient in front of the monomial x α , and the sequence (f α ) α∈N d , which has finite support, is identified with the polynomial.

We also define the set N d r wich is the set of α ∈ N d so that |α| = d i=1 α i ≤ r. Note that N d r is of size s(r) = r+d d , and that coefficients of polynomials in R r [x 1 , ..., x d ] are naturally indexed by N d r . We will often identify the set R N d r of families (h α ) |α|≤r with the set of vectors in R s(r) . With a slight abuse of notations, we will therefore use the α ∈ N d r to index such vectors, as well as matrices defined on R s(r) ≈ R N d r . Given x ∈ R d , we will denote with v r (x) the vector (x α ) ∈ R s(r) , so that for any q ∈ R s(r) , it holds q v r (x) = |α|≤r q α x α .

As we will see in Sec. 8.1 .2, it will be interesting to study global optimization of polynomials over subsets K of R d . In particular, we will usually focus on semi-algebraic sets K, i.e., sets of the form K = {x ∈ R d : f j (x) ≥ 0, 1 ≤ j ≤ m} for polynomials f j of degree 2r j or 2r j -1. In the semi-algebraic case, in order to lighten notations, we will not write the dependence of semi-algebraic sets in the f j (even though the quantities and sets we define might depend on the f j ). If K = R d , we consider that the canonical choice of f j is the empty set of polynomial constraints.

In this section, great emphasis will be put on certain sets of non-negative polynomials. We will denote with P + (K) the set of non-negative polynomials on a set K ⊂ R d . We will also use the following notations:

• SOS[x] will denote the set of sum of squares of polynomials of and SOS r [x] will denote the set of sum of squares of polynomials of degree at most r, i.e.,

SOS r [x] := f ∈ R 2r [x] : ∃N ∈ N, ∃h 1 , ..., h N ∈ R r [x], f = N i=1 h 2 i ; (8.3)
• following the notations by [START_REF] Slot | Sum-of-squares hierarchies for polynomial optimization and the christoffel-darboux kernel[END_REF], given K defined by inequalities f j ≥ 0 we denote with Q r (K) the elements of degree less than 2r of the associated "quadratic module", i.e., the set of polynomial functions of the form

Q r (K) = f ∈ R 2r [x] : ∃σ 0 ∈ SOS r [x], ∃σ j ∈ SOS r-r j [x], f = σ 0 + m i=1 f j σ j ; (8.4)
• finally, still following the notations by Slot (2021), we define f J = j∈J f j for any subset J ⊂ {1, ..., m} (note that the inequalities f J ≥ 0 still define K) and with T r (X ) the set of elements of degree less than 2r of the associated "preordering", i.e., the set of polynomial functions of the form

T r (K) =    f ∈ R 2r [x] : ∃σ 0 ∈ SOS r [x], ∃σ J ∈ SOS r J [x], f = σ 0 + J⊂{1,...,m} f J σ J    . (8.5)
Note that we have the following inclusion, for any r ∈ N:

SOS r [x] ⊂ Q r (K) ⊂ T r (K) ⊂ P + (K) (8.6)
Outline. We will start by introducing the moment-SOS hierarchies for lower bounds in Sec. 8.1 .1, which can be seen either as a way of approximating the set of measures through its moments, or the set of non-negative polynomials by sums of squares. We will use this tool to to define surrogate problems for both the unconstrained and constrained polynomial minimizations problems in Sec. 8.1 .2, as well as give the guarantees for the solution of these surrogate problems. In Sec. 8.1 .3, we will briefly present another hierarchy presented by [START_REF] Lasserre | A new look at nonnegativity on closed sets and polynomial optimization[END_REF], whose aim is to provide upper bounds for the minimization problem, and which comes with theoretical guarantees. We conclude by a small summary in Sec. 8.1 .4.

.1 Moment relaxation, SOS strengthening

From measure to moments When dealing with polynomials, a measure µ is only seen through moments, i.e. values of the form R d x α µ(dx). As an example, if we take Eq. ( 8.2), one can write

R d f (x)µ(dx) = α f α y α , where y α = R d x α µ(dx).
The moment point of view on moment-SOS hierarchies focuses precisely on defining and approximating the set of moments sequences associated to measures, and not the measures themselves. An element y

= (y α ) ∈ R N d is the full moment sequence associated to µ if y α = R d x α µ(dx), α ∈ N d .
We denote with M (K) the set of full moment sequences of measures µ with support in K, i.e.,

M (K) = y = (y α ) α∈N n : ∃µ ∈ M (R d ), ∀α ∈ N d , y α = R d x α µ(dx), supp(µ) ⊂ K (8.7)
For a sequence y ∈ R N d , we define the linear form L y on R[x] as L y (f ) = α y α f α , which is linear both in f and y. The global optimization on problem in Eq. ( 8.2) for a polynomial f on a set K can simply be rewritten as

inf L y (f ) subject to y ∈ M (K), y 0 = 1, (8.8)
Which is a finite dimensional linear convex program (indeed, if f is of degree r, we can just optimize on the projection of M (K) on its |α| ≤ r first coefficients), but on a cone which does not seem to have any kind of finite dimensional characterization. This is in a sense exactly the same situation as in the dual problem sup c subject to fc ∈ P + (K), (8.9) where the cone P + (K) of non-negative polynomials over K does not admit any nice finitedimensional respresentation even when restricted ro R r [x].

Note that the cones M (K) and P + (K) are dual cones of each other with respect to the bilinear form L : y, f → L y (f ). The idea of moment relaxation and SOS strengthening (which is the dual operation) is to approximate these cones with cones which are representable as sections of PSD cones, and hence are representable in finite dimension, leading to a solvable semidefinite program.

Unconstrained measures and polynomials

Let us start with the simple case K = R d . If y ∈ M (K), then necessarily, for any non-negative polynomial h, L y (h) ≥ 0. In particular, this is true for all polynomial functions h wich are squares or sums of squares of polynomials, i.e., r) to be the matrix associated to the bilinear form

∀r ∈ N, ∀h ∈ R r [x], L y (h 2 ) ≥ 0 (8.10) For r ∈ N, define the moment matrix M r (y) = (y α+β ) α,β∈N d r ∈ R s(r)×s(
h 1 , h 2 ∈ R r [x] → L y (h 1 h 2 ) in the canonical basis (x α ) α∈N d r . The previous remark shows exactly that if y ∈ M (R d ), then M r (y) 0 for all r ≥ 0. For r ∈ N, define M r (R d ) = {y ∈ R N d : M r (y) 0}
. This set is PSD representable (it can be represented as a linear subset of a PSD cone), and satisfies M r (R d ) ⊃ M (R d ) : it is therefore a PSD relaxation of the cone M (R d ). Moreover, these relaxations for an outer approximation sequence of the moment cone, i.e.,

M 0 (R d ) ⊃ ..... ⊃ M r (R d ) ⊃ ... ⊃ M (R d ). (8.11)
A dual perspective provides the so-called SOS strengthenings. Indeed, the dual of M r (R d ) with respect to L can be shown to be the set of sum of squares of polynomials of degree less than r, defined in Eq. ( 8.3) and which we denote with SOS r [x].Rewriting the definition of this cone as

SOS r [x] = {f ∈ R 2r [x], ∃A ∈ S + (R s(r) ), f (•) = v r (•) Av r (•)}, (8.12)
it is clear that it is a PSD representable cone. Moreover, the cones SOS r [x] ⊂ P + (R d ), as sums of squares are always non-negative, and forms an inner approximation of P + (R d ), i.e.,

SOS 0 [x] ⊂ ..... ⊂ SOS r [x] ⊂ ... ⊂ P + (R d ). (8.13)
As we will see in Sec. 8.1 .2, these unconstrained approximation sequences are usually too weak, in the sense that they do not approximate M (R d ) and

P + (R d ) well enough at infinity. Indeed, if SOS[x] = r≥0 SOS r [x]
is the set of SOS polynomials, SOS[x] is not "dense" in P + (R d ) for certain metrics of interest, such as the uniform convergence, needed in global optimization [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF]. Note that in the one dimensional case, the uncontrained SOS approximation sequence and hence both approximation sequences are tight : every polynomial which is non-negative can be written as a sum of squares of polynomials.

Constrained measures and polynomials

In the case of constrained measures, which is the most common in practice, the necessary condition Eq. (8.10) is not strong enough as it does not incorporate the algebraic inequalities f j which define the set K. Let y ∈ M (K). It is clear that for any 1

≤ j ≤ m, if h is non-negative on {x ∈ R d : f j (x) ≥ 0}, then L y (h) ≥ 0.
In particular, since for any polynomial h, the polynomial f j h 2 is non-negative on {x ∈ R d : f j (x) ≥ 0}, the following sufficient conditions hold .14) where M r (f j y) is called the localizing matrix associated to f j , and is associate to the bilinear form

∀1 ≤ j ≤ m, ∀r ∈ N, M r (f j y) 0. ( 8
h 1 , h 2 ∈ R r [x] → L y (f j h 1 h 2 ). Note that in that case, M r (f j y) α,β = γ f γ y α+β+γ
Following Lasserre (2010), for r ≥ r 0 = max 1≤j≤m (r j ), define

M r (K) := {y ∈ R N d : M r (y) 0, M r-r j (f j y) 0, 1 ≤ j ≤ m}. (8.15)
The set M r (K) is still PSD representable (see the expression of the localization matrices), and are the sets of moments such that the necessary condition Eqs. (8.10) and (8.14) hold with certain degree. They are still relaxations of M (K), and we have the following outer approximation of the moment cone:

M 0 (K) ⊃ ..... ⊃ M r (K) ⊃ ... ⊃ M (K). (8.16)
A dual perspective provides the constrained SOS strengthening and approximation sequence. In this case, the dual of M r (R d ) with respect to L can be shown to be the set Q r (K) defined in Eq. (8.4). These cones are still PSD representable and are strenghtenings of the cone P + (K).

They form an inner approximation sequence of P + (K), i.e.,

Q 0 (K) ⊂ ..... ⊂ Q r (K) ⊂ ... ⊂ P + (K). ( 8

.17)

A note on preorderings

Since SOS r [x] ⊂ Q r (K)
, it is clear that we will better approximate non-negative polynomials on K using the inner approximation sequence given by the Q r (K) than the one given by SOS r [x]. This fact will be made formal in the next section, where we will see that the guarantees obtained using this approximation sequence are much stronger than the one obtained using the simple sequence of SOS polynomials.

Another approximation of non-negative polynomials on K is used and analysed in the literature (see [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF]Slot and Laurent, 2020a;[START_REF] Slot | Sum-of-squares hierarchies for polynomial optimization and the christoffel-darboux kernel[END_REF], based on the notion of preordering.

Instead of considering polynomials of the form σ 0 + m j=1 f j σ j for sum of squares polynomials σ j , polynomials of the form J⊂{1,...,m} f J σ J are considered, where f J = j∈J f j and the σ J are sum of squares. For instance, Slot (2021) define T r (K) as the set of all such polynomials such that deg(f J σ J ) ≤ 2r (see Eq. (8.5) for a formal definition). In a sense, this is just the quadratic module Q r (K) but associated to the augmented family (f J ) J⊂{1,...,m} of inequalities which define the same semi-algebraic set K. Of course, this family has 2 m instead of m + 1 terms in the sum and we can see that T r (K) is a larger set of non-negative functions than Q r (K). Once again, the T r (K) form an inner approximation of the set of non-negative functions :

T 0 (K) ⊂ ..... ⊂ T r (K) ⊂ ... ⊂ P + (K). (8.18)
The associated moments' point of view can of course be derived by considering localizing matrices for all (f J ) J⊂{1,...,m} .

Apart from cases where the number of inequalities is very low, it is too costly to use these inner approximations of the cone P + (K) in practice (as well as the moments' counterpart), as the number of PSD constraints exponentially increases in m. However, this approximation sequence has been more analyzed from a theoretical standpoint, and many guarantees can be obtained by considering this augmented sequence rather than Q r (K), even though it is the one most used in practice.

Finally, note that Q r (K) and T r (K) coincide if K is only defined with one inequality (such as ball of fixed radius).

.2 Moment-SOS hierarchies of lower bounds for optimization, and their guarantees

The case of unconstrained optimization

Let us now consider the approximations by PSD cones described above in order to solve the unconstrained polynomial minimization problem Eqs. (8.8) and (8.9). Let f be a polynomial of degree 2r (note that if the degree of the polynomial is odd, the minimization problem trivially has solution -∞), and let f * be its minimum on R d .

We define the sequence of problems :

ρ r = inf L y (f ) subject to M r (y) 0, y 0 = 1 (8.19)
where we have relaxed the condition that y ∈ M (R d ) with the condition that y ∈ M r (R d ). The dual counterparts of these problems are

ρ * r = sup c subject to f -c ∈ SOS r [X ], (8.20)
where we have strengthened the condition that fc ∈ P + (R d ) with the condition that fc ∈ SOS r (R d ).

The sequences ρ r and ρ * r are a) increasing sequences (see Eq. ( 8.13) and Eq. (8.11)), b) are all lower bounds of f * , and c) satisfy ρ r = ρ * r , that is there is no duality gap [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF]. In the dual problem, note that if the constraint is satisfied, necessarily, fc ∈ SOS r [x]. Thus, there is no need for the full sequence of problems in the unconstrained case, the case r = r: there is actually no hierarchy in the unconstrained case. To solve Eq. (8.19) with r = r, Lasserre (2010) uses interior point methods [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF].

Once an optimal ρ r is obtained (and we know that ρ r ≤ f * ), two questions remain.

(a) Can we say that ρ r = f * ? (b) Can we find a minimizer

x * ∈ R d such that f (x * ) = f * ?
The following result provides a sufficient condition, based on the "stabilization of the rank of the moment matrix", in order to guarantee that ρ r = f * and that a minimizer x * can be extracted. Theorem 8.1 (Lasserre (2010), Theorem 5.5). Let ρ r be a solution to Eq. (8.19) if there is an optimal solution y * , and if rank(M r (y * )) = rank(M r-1 (y * )), then ρ r = f * , ff * is a sum of squares, an a minimizer of f can be extracted using Algorithm 4.2 by [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF].

Unfortunately however, unconstrained minimization for polynomials suffers from the following alternative : CHAPTER 8. A PARALLEL WITH MOMENT-SOS HIERARCHIES (i) if the rank of the moment matrix stagnates, in which case ff * is a sum of squares polynomial, ρ r = f * , and a minimizer can be extracted;

(ii) if the moment stagnation property is not satisfied, and we obtain a lower bound ρ r but without any additional information on f * or a minimizer.

Note that since only sum of squares can hope to satisfy the moment stagnation property, all non-negative polynomials which are not sums of squares (and they exist, see Chapter 2 by Lasserre ( 2010)) are excluded from the first point, and will not be minimized. Since there is no guarantee a priori that a non-negative polynomial is a sum of squares, there is no a priori guarantee that ρ r will be close to f * .

The reason for this is the lack of a proper Positivstellensatz for the unconstrained case, that is we cannot prove that there exists a sequence c n ↑ f * such that fc n , which is striclty positive on R d , can be decomposed as a sum of squares. While such results do not exist for general polynomials (except in the d = 1 case), they exists under certain mild assumptions for the constrained case. Moreover, in practice, we usually look for the minimizer in a large enough ball, and adding a constraint of the form B 2 -d i=1 x 2 i ≥ 0 is enough to make these assumptions hold (it is sometimes called the archimedean assumption).

Finally, note that in the one dimensional case, all non-negative polynomials are sum of squares, and this method works very well.

Constrained optimization and moment-SOS hierarchies

Let us now consider the moment-SOS approximation sequences in Sec. 8.1 .1 to solve the constrained polynomial minimization problem, and show that there are better guarantees in the constrained setting than in the unconstrained one. Let f be a polynomial function, and r = deg(f )/2 such that that f is either of degree 2r or 2r -1. Let K be defined by the polynomial inequalities f j ≥ 0 for 1 ≤ j ≤ m, and r j = deg(f j )/2 such that f j is of degree either 2r j or 2r j -1. Let f K be the minimum of f on K. Let r 0 = min j (r j ). For any r ≥ r 0 , we define the sequence of moment problems

ρ r = inf L y (f ) subject to M r (y) 0, M r -r j (f j y) 0, y 0 = 1 (8.21)
where we have relaxed the condition that y ∈ M (K) with the condition that y ∈ M r (K). The dual counterpart of this sequence of moment problems is the following sequence of problems,

ρ * r = sup c subject to f -c = σ 0 + m j=1 σ j f j , σ 0 ∈ SOS r [x], σ j ∈ SOS r -r j [x], 1 ≤ j ≤ m, (8.22)
where we have strengthened the condition that fc ∈ P + (K) with the condition that fc ∈ Q r (K).

Once again, Eqs. (8.11) and ( 8.13) guarantee that ρ r and ρ * r are both increasing sequences. Moreover, we have

∀r ≥ r 0 , ρ * r ≤ ρ r ≤ f K . (8.23)
These problems are often referred to as the (Lasserre) moment-SOS hierarchies (of lower bounds), as they form a sequence of problems which are both increasingly harder and more precise.

The following algorithm [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF] can be applied to attempt to minimize f , with a parameter R which is the highest order of the hierarchy we are prepared to go to. It is based on the equivalent of Theorem 8.1 in the constrained case, in order to establish if ρ r = f K (Lasserre (2010), Theorem 5.7).

• Solve the problem Eq. ( 8.21) and obtain ρ r .

• If there is no optimal y * , then ρ r ≤ f K ; if r < R, we set r ← r + 1

• If there is an optimal y * , and rank(M r (y)) > rank(M r-r 0 (y)) then ρ r ≤ f K ; if r < R, we set r ← r + 1

• If there is an optimal y * and rank(M r (y)) = rank(M r-r 0 (y)), then f K = ρ r and one can extract the minimizer using algorithm 4.2 by [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF].

Guarantees of the moment-SOS hierarchies

The key difference with the unconstrained case is that the moment-SOS hierarchy converges to the actual minimizer f K . There are essentially two types of results describing this convergence.

• The first results (which were also the first resutls obtained on this subject) describe assumptions under which the hierarchy converges (i.e., ρ r ↑ f K ).

• The second results are more precise and describe the speed of convergence of the hierarchy towards f K as a functions of the order r of the hierarchy. They can have stronger assumptions in than the ones needed for the convergence, in order to prove faster rates. Note that most of these results are obtained using the pre-ordering approximation Eq. (8.5) and is not exactly the hierarchy described in Eq. (8.21). We mention them more precisely in the paragraph below.

Convergence results. Let (A) denote Assumption 2.1 by [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF]. It is satisfied in many cases, and can be enforced as soon as the set K is localized inside a ball of known radius B > 0 (or in the unconstrained case, as soon as we know we are looking for the minima of f in a ball of given radius). In that case, adding the constraint B 2 -d i=1 x 2 i ≥ 0 directly guarantees that (A) is satisfied. The following theorem is the cornerstone to prove the (asymptotic) convergence of the Lasserre moment-SOS hierarchy. Theorem 8.2 (Putinar Positivstellensatz, (Putinar, 1993)). Under assumption (A) on K, any strictly positive polynomial belongs to SOS r (K) for some r ∈ N, i.e., is of the form

σ 0 + m j=1 σ j f j , σ 0 ∈ SOS r [x], σ j ∈ SOS r -r j [x], 1 ≤ j ≤ m. (8.24)
Indeed, once such a result is established, then for any c < f * , fc is strictly positive and hence belongs to a certain SOS r (K) and thus ρ * r ≥ c, showing that ρ * r ↑ f K and hence also ρ r ↑ f K (see [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF], theorem 4.1).

Under the assumption that f has a unique minimizer on K as well as a few additional assumptions, it is also possible to design a sequence x r → x * from the hierarchy solutions (y * ) r , if they exist (see [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF], Theorem 5.6).

CHAPTER 8. A PARALLEL WITH MOMENT-SOS HIERARCHIES

Under stronger assumptions than the ones above, finite convergence of the Lasserre hierarchy has been obtained by [START_REF] Nie | Optimality conditions and finite convergence of Lasserre's hierarchy[END_REF] in Theorem 1.1. These assumptions are based on first and second order sufficiency conditions at the minimizers, and are analoguous to constraint qualification assumptions in non-linear optimization [START_REF] Gilbert | Fragments d'optimisation différentiable -théorie et algorithmes[END_REF], developped for polynomials by [START_REF] Marshall | Representations of non-negative polynomials having finitely many zeros[END_REF]. Under these assumptions, one obtains that ff K ∈ Q r (K) for some large enough r 0 (although no bound is provided on the size of r 0 ), and hence ρ r = f K for all r ≥ r 0 .

Other results on finite convergence of the Lasserre hierarchy have been established for certain classes of polynomials. For instance, [START_REF] Lasserre | Convexity in semi algebraic geometry and polynomial optimization[END_REF] proves finite convergence strictly convex polynomials and gives the order of convergence of the Hierarchy for SOS-convex polynomials (that is polynomials whose Hessians are sum of squares, as defined by Helton and Nie ( 2010)). The strict convexity assumption is has been weakened in further works, as the one by [START_REF] Klerk | On the lasserre hierarchy of semidefinite programming relaxations of convex polynomial optimization problems[END_REF].

Rates of convergence. Under assumption (A), [START_REF] Nie | On the complexity of Putinar's Positivstellensatz[END_REF] proved that the convergence ρ r ↑ f K happens at a rate of order O(1/ log(r ) c ), where c is a constant which depends on K.

More recently, better convergence bounds have been provided in specific cases, such as when K is the unit ball, as stated below.

Theorem 8.3 (Theorem 3 by [START_REF] Slot | Sum-of-squares hierarchies for polynomial optimization and the christoffel-darboux kernel[END_REF]). Assume K is the d-dimensional unit ball and let f be a polynomial of degree r. Then for any r ≥ 2rd, we have

f K -ρ r ≤ C r,d (r ) 2 max K f -min K f (8.25)

Guarantees using the preordering approximation sequence

It is important to note that many convergence rates obtained for the Lasserre hierarchy have been obtained in a setting slightly different than the hierarchy presented in Eq. (8.22). Instead, works such as those by [START_REF] Laurent | An effective version of schmüdgen's positivstellensatz for the hypercube[END_REF]; [START_REF] Slot | Sum-of-squares hierarchies for polynomial optimization and the christoffel-darboux kernel[END_REF] derive rates of convergence for the hierarchy defined by the "preordering" approximation sequence in Eq. ( 8.18) (i.e., replacing the constraint that fc ∈ Q r (K) by fc ∈ T r (K) in Eq. (8.22)).

A summary of those results can be found in Table 1. of the work by [START_REF] Slot | Sum-of-squares hierarchies for polynomial optimization and the christoffel-darboux kernel[END_REF]. In particular, rates of order O(1/(r ) 2 ) such as the one reported in Theorem 8.3 are obtained for both the simplex

K = ∆ d-1 = {x ∈ R d : x ≥ 0, x 1 = 1} and the hypercube K = [-1, 1] d .

.3 A hierarchy of upper bounds

In this section, we briefly present another approximating sequence of the cone of non-negative polynomials on a set K, introduced by [START_REF] Lasserre | A new look at nonnegativity on closed sets and polynomial optimization[END_REF]. This sequence forms an outer approximation and leads to a hierarchy of upper bounds for the minimum. It also comes with theoretical guarantees on the convergence and rate of convergence of the hierarchy.

Outer approximation of the cone of non-negative polynomials

For simplicity, let K be a compact set of R d (for generalizations to non-compact sets, see the original paper by [START_REF] Lasserre | A new look at nonnegativity on closed sets and polynomial optimization[END_REF], where one can deal with the non-compact case with a slight modification). Note that here, K does not have to be a semi-algebraic set. The outer approximation of the cone of non-negative polynomials relies on a fixed reference measure µ, whose support is K. Given this reference measure, one considers the following sets for r ∈ N :

C r r (K) = f ∈ R 2r [x] : ∀q ∈ SOS r [x], K q(x)f (x) dµ(x) (8.26)
Once again, this is an abuse of notation because this set depends not only on K but on µ. Moreover, this set can be defined for any function f and not only for a polynomial. It is clear that we have nested sequence of outer approximations approximations of the set of non-negative polynomials on K of degree at most 2r which we write C r (K). .27) Note that C r r (K) can also be written using a localization matrix (see below Eq. ( 8.14) for the definition), 8.28) This means that if the moments (y α ) are computable for the measure µ, we can have a computable PSD representation of the cone C r r (K).

C r 0 (K) ⊃ ... ⊃ C r r (K) ⊃ ... ⊃ C r (K). ( 8
C r r (K) = f ∈ R r [x] : M r (f y) 0, y α = K x α µ(dx) (

Hierarchy of upper bounds

Given a polynomial function f ∈ R 2r [x], if we relax Eq. (8.9) by replacing fc ∈ P + (K) by fc ∈ C r r (K), we get the following hierarchy of upper bounds indexed by r : .29) Note that this correspond to the dual problem in the previous sections. The dual of Eq. (8.29), which corresponds to the primal problem in the previous sections (i.e., a moment problem), can be written as

λ r = sup c subject to f -c ∈ C r r (K). ( 8
λ * r = inf K f σdµ subject to K σdµ = 1, σ ∈ SOS r [x].
(8.30) Theorem 4.2 by [START_REF] Lasserre | A new look at nonnegativity on closed sets and polynomial optimization[END_REF] shows in particular that if f K > -∞ and K has non-empty interior, there is no duality gap (and hence λ r = λ * r ). As explained by [START_REF] Lasserre | A new look at nonnegativity on closed sets and polynomial optimization[END_REF], these optimization problems correspond to a generalized eigenvalue problem between the localizing matrices M r (y) and M r (f y). Moreover, this hierarchy also comes with a way of approximating the minimizer, when K is convex (see Theorem 4.2 by [START_REF] Lasserre | A new look at nonnegativity on closed sets and polynomial optimization[END_REF]).

CHAPTER 8. A PARALLEL WITH MOMENT-SOS HIERARCHIES

Convergence guarantees

In terms of convergence guarantees, the moment-SOS hierarchy of upper bounds comes with many results. The first, which can be found as Theorem 4.1 by [START_REF] Lasserre | A new look at nonnegativity on closed sets and polynomial optimization[END_REF], establishes the convergence λ r ↓ f K .

Many convergence rates have been established, and are summarized in Table 2 by [START_REF] Slot | Sum-of-squares hierarchies for polynomial optimization and the christoffel-darboux kernel[END_REF]. Since the choice of µ defines the approximation, these results depend on the choice of µ In particular, [START_REF] Slot | Near-optimal analysis of lasserre's univariate measure-based bounds for multivariate polynomial optimization[END_REF] show that under very mild assumptions on K (such as convexity, or being a semi-algebraic set with dense interior), if µ is the Lebesgue measure, then the convergence rate is of order O(log(r ) 2 /(r ) 2 ). For specific cases like the unit ball, the unit simplex or the unit hypercube, O(1/(r ) 2 ) rates have also been shown by Slot and Laurent (2020b).

.4 Summary

In this section, we have presented different moment-SOS hierarchies in order to solve global polynomial optimization on unconstrained and constrained domains.

We started by presenting the first moment-SOS hierarchy which provides lower bounds on the global optimum. It based on a series of moment relaxations or SOS strengthenings (depending on the primal or dual point of view) , which consist in approximating the set moments or nonnegative polynomials on K by a sequence of PSD representable cones, on which the associated moment or SOS problems can be optimized.

Note that using interior point methods, the complexity of optimizing the hierarchy until degree r is roughly or order d r (this complexity can be reduced using sparsity, see [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF], section 4.6.2).

We have seen that the proposed methods return either a) a minimizer and the minimum under a rank condition, or b) a lower bound ρ r for f K otherwise.

In the unconstrained case, if a lower bound is returned, there is no guarantee on its proximity to f * .

In the constrained setting (which is usually naturally the case), under very mild assumption, it can be shown that the hierarchy asymptotically converges, that is ρ r ↑ f K (finite convergence can also be shown in some convex or locally convex cases). Moreover, convergence rates of order 1/ log(r ) c and 1/(r ) 2 in certain specific settings can be proven.

We then presented a second moment-SOS hierarchy of upper bounds, based on the choice of a base measure on K. This moment-SOS hierarchy enjoys very nice convergence properties (close to O(1/(r ) 2 ) in many settings). However, it is less obvious to use in practice, as one must compute the moments of the base measure.

In terms of complexity of minimizing a polynomial function on a subset of R d with error , we would need a degree in the hierarchy of order 1/ √ ε, which leads to a crude estimation of the complexity of order d 1/ √ , and which is therefore exponential in 1/ √ ε.

Global optimization through kernel sums of squares

In this section, we present the work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF], which proposes an algorithm to perform global optimization through sums of squares of kernels. This method is analyzed in the context of approximating a function f with a given regularity.

Here, we take a different approach from the paper, whose verbatim can be found in chapter 9.

We start by presenting the method generally, without guarantees, in Sec. 8.2 .1, before applying it in the setting considered by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] where guarantees can be obtained, in Sec. 8.2 .2.

Notations. In this section, we will consider a RKHS H on a set X associated to the kernel k, whcih we will assume to be separable (i.e., there is a dense at most countable sequence in H). We denote with S(H) the set of bounded symmetric operators on H. In particular, in this section, we will consider the set S 1 (H) of trace class symmetric operators on H [START_REF] Weidmann | Linear Operators in Hilbert Spaces[END_REF] equipped with the trace norm (note that we could take other spectral norms, such as the Hilber-Schmidt norm, as explained in chapter 6). We will denote with S 1,+ (H), or simply S + (H) (the norm considered will always be the trace norm in this section), the set of postivie semidefinite trace-class symmetric operators on H.

.1 A generic methods based on kernel sum of squares

In this section, we present the main ingredients of the method developed by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. We present these ingredients in full generality, without assuming that X = R d for example, but without providing any guarantees at this stage.

The method relies on three main steps.

(i) As in the polynomial case, we start by considering a kernel equivalent of the moment-SOS relaxation.

(ii) We then consider a regularized discretization of this relaxation, in the spirit of empirical risk minimization.

(iii) Finally, using the kernel representer theorem, we effectively solve this regularized discretized version.

Primal and dual relaxations

The first step is to consider the following primal and dual relaxations of the measure problem Eqs. (2.25) and (8.2). It is easier to see things in the dual : we replace the positivity constraint fc ≥ 0 by the constraint that fc be a PSD model introduced in part II. This is done in Eq. ( 9.3) in the work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF].

To align with the notations of the previous section, define .31) Note that as explained in chapter 6, SOS H 0 is simply the set of finite sums of squares of functions in H, while SOS H is its completion for the trace norm, defined as f 1 = min{Tr(A) : f (x) = k x , Ak x H , x ∈ X , A ∈ S + (H)}. Strengthening the constraint fc ≥ 0 by fc ∈ SOS H , we get the following strengthening of the dual : 8.32) This corresponds to the following relaxation of the primal measure problem : (8.33) where the positivity constraint of the measure is relaxed with the PSD constraint X k x ⊗ H k x µ(dx) 0 on signed measures.

SOS H 0 = {f : X → R : f (x) = k x , Ak x H , x ∈ X , A ∈ S + (H), rank(A) < ∞} , SOS H = {f : X → R : f (x) = k x , Ak x H , x ∈ X , A ∈ S + (H)} . ( 8
ρ * H = sup c subject to f (x) -c = g(x), x ∈ X , g ∈ SOS H (
ρ H = inf X f (x)µ(dx) subject to µ ∈ M (X ), X µ(dx) = 1, X k x ⊗ H k x µ(dx) 0,
As these problems are relaxations/strengthenings of the original problem, the following guarantees hold : Note on moments. It is not as obvious to define the moment problem, as in the polynomial case, since we have assumed nothing on f at this point. It is possible to formulate the moment problem if f is assumed to belong in a certain space, as we will see in Sec. 8.3 . For the remaining of this section, we will keep to the measure problem as primal problem, without simplifying it as a moment problem as used to be the case in the polynomial setting.

ρ * H ≤ ρ H ≤ f * = inf x∈X f (x). ( 8
Note on tightness. As we will see when applying this method with the Sobolev kernel in Sec. 8.2 .2, in the infinite dimensional RKHS setting (non-parametric), it is easier to show that Eq. ( 8.32) are actually tight, i.e., ρ H = ρ * H = f * . However, as we will see in the next section, it will be crucial to have more than this type of tightness, and to have an optimal solution in the dual case Eq. (8.32) (i.e., the existence of c * , g * which solve the problem and reach the supremum), in order to get good properties of the method, as we do not solve Eq. (8.32) directly, but approximate it. This corresponds to a well-specified assumption in part I, as we will take the same "regularized e.r.m." approach to solve the problem Eq. (8.32).

Evaluating the constraint on a finite number of points While these primal and dual relaxation are PSD representable, they are not finite dimensional, because of the fact that H is potentially infinite dimensional. In a second step, we therefore adopt the same strategy as when performing empirical risk minimization, and instead of considering the equality in Eq. (8.32) to hold at every point of X , we restrict it to hold only a a finite number of points. To that end, we give ourselves a sequence (x i ) 1≤i≤n ∈ X n of test points, corresponding to a subset X n = {x i : 1 ≤ i ≤ n} ⊂ X of size n, and we solve the following regularized problem corresponding to Eq. ( 8.32) : (8.35) which we can recast as the following problem on semidefinite trace class operators : (8.36) This corresponds to the following rewriting of the primal measure problem, where the measure is restricted to have support in X n :

ρ * n,λ = sup c -λ g 1 subject to f (x i ) -c = g(x i ), 1 ≤ i ≤ n, g ∈ SOS H ,
ρ * n,λ = sup c -λ Tr(A) subject to f (x i ) -c = k x i , Ak x i H , 1 ≤ i ≤ n, A ∈ S + (H).
ρ n,λ = inf Xn f (x)µ(dx) subject to µ ∈ M (X n ), Xn µ(dx) = 1 Xn k x ⊗ H k x µ(dx) + λI H 0. (8.37)
Writing a signed measure with support in X n as µ = n i=1 α i δ x i , we get the following dual of Eq. ( 8.36)

ρ n,λ = inf n i=1 α i f (x i ) subject to (α i ) ∈ R n , n i=1 α i = 1 n i=1 α i k x i ⊗ H k x i + λI H 0. (8.38)
Difference with SOS hierarchies. We want to highlight that it is this modification of the problem Eq. (8.32) which makes this method fundamentally different from the polynomial method. Indeed, by going from Eq. (8.32) to Eq. (8.35) the following phenomemons appear.

(i). Take λ = 0. The restriction to the set X n makes the primal problem harder (we restrict the set of measures to sums of diracs on X n ) and the dual problem easier (as we only enforce the equality constraint on a finite subset of X ). Thus, the operation of restricting to X n is a strengthening of the primal, and a relaxation of the dual : ρ n,0 ≥ ρ H and ρ * n,0 ≥ ρ * H . In the case where we do not know tightness a priori of problems Eqs. (8.32) and (8.33), we cannot say anything about ρ * n,0 , ρ n compared to f * , as we used to have ρ H , ρ * H ≤ f * . Even in the case where Eqs. (8.32) and (8.33) can be shown to be tight a priori, if the class of functions SOS H is too large (and it will usually be the case as soon as H is infinite dimensional, for example in the Sobolev kernel setting of chapter 9), the problem is still complicated because we need regularization. Indeed, in the case where the space SOS H is large, it is possible that it can interpolate any function fc with c ≤ min 1≤i≤n f (x i ). In that case, we have ρ * n,0 = min 1≤i≤n f (x i ), which is a trivial upper bound on the minimum and can be obtained without such an elaborate method.

(ii). As soon as H and therefore SOS H is too large, regularization is necessary to avoid overfitting or interpolation. As the space SOS H is equipped with a natural norm given by the trace norm, it is natural to regularize the dual problem (with the SOS constraint) with that norm. This corresponds to a form of strengthening of the dual and of relaxation of the dual, and we therefore have ρ n,λ ≤ ρ n,0 and ρ * n,λ ≤ ρ * n,0 . Since the bounds go in the other way around from the bounds between the discretized and non discretized problems, there is no way, even when the original SOS problem is tight, to know if ρ * n,λ , ρ n,λ are lower or upper bounds for f * . However, as is the e.r.m. setting, the interaction between n and λ helps derive asymptotic upper bounds between ρ * H and ρ * n,λ under certain conditions, as we will see in Sec. 8.2 .2.

Algorithm

Looking at Eq. (8.35), we see that we can apply proposition 6.3 in chapter 6, i.e., the representer theorem for PSD models. Following the reasoning in chapter 9, define Φ i the ith column of a matrix such that Φ Φ = K, where K is the kernel matrix K = (k(x i , x j )) 1≤i,j≤n . The problem can be written in the dual as (8.39) and in the primal as

ρ * n,λ = sup c -λ Tr(A) subject to f (x i ) -c = Φ i AΦ i , 1 ≤ i ≤ n, A 0,
ρ n,λ = inf n i=1 α i f (x i ) subject to (α i ) ∈ R n , n i=1 α i = 1, n i=1 α i Φ i Φ i + λI R n 0. (8.40)
Both of these problems are linear semidefinite programs in the cone S + (R n ). If, for example, we solve the primal formulation using interior point methods, we can obtain an ε-solution in time O(n 3.5 log 1 ε ). They are detailed in chapter 9, in algorithm 6 and Sec. 9.6 .

Remark on extraction. Contrary to the polynomial case, where a stabilization of the rank was a sufficient condition to find a minimizer, it is not direct to a sufficient condition to extract a minimizer associated to the estimation ρ n,λ or ρ * n,λ of the minimum. One way to go if X is equipped with a metric d X is to look for x * as the solution to the following problem : (8.41) Where δ x is just a dirac in X , and µ n = n i=1 α i δ x i is the signed measure with support in X n which is the solution to Eq. (8.37). In other words, we look for x * such that the dirac measure δ x * is as close as possible in Wasserstein-2 norm [START_REF] Santambrogio | Optimal Transport for Applied Mathematicians[END_REF] to the signed measure µ n . In the case where X = R d , the solution to Eq. ( 8.41) is simply x * = n i=1 α i x i . The way chosen by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] is essentially this one. Note that in order for this strategy to be well motivated, it is important that the minimizer be unique.

x * = arg min x∈X n i=1 α i d X (x, x i ) 2 = X d X (x, x ) 2 µ n (dx ) = X ×X d X (x 1 , x 2 ) 2 µ n (dx 1 )δ x (dx 2 ),

.2 Applying this method to the minimization of regular functions in the Euclidean space

In this section, we explain how [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] apply the method described in Sec. The goal of this work from a high level viewpoint is to leverage regularity in global optimization, even if it is performed using only function values.

Problem setting and worst case bounds

Formally, the high level objective consists in (i) finding an ε-approximation of the minimum, i.e., c such that | cf * | ≤ ε;

(ii) finding an ε-approximation of the minimizer, i.e., x such that |f ( x) -f * | ≤ ε.

Given ε > 0, we wish to find such approximation with the lowest number of function calls (i.e., the smallest possible number of x i , 1 ≤ i ≤ n in Sec. 8.2 .1), and with worst-case guarantees over all functions f in some relevant class of functions F ⊂ C r b (Ω). We want the method to find x to satisfy :

sup f ∈F f (x) -min x∈Ω f (x) ≤ ε.
(8.42) [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF] argues that when having access to n function values, the problem of optimizing f is as hard as the problem of approximating f uniformly (see introduction and section 1.3 by [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF] and Fig. 8.1 for a visual representation). In the setting where f ∈ C r b (Ω), section 1.3.9 by [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF] shows that the problem of optimization is indeed as hard as approximation, and that n =∝ ε -d/r points are needed in order to obtain an ε-approximation (the proportionality constants may depend exponentially in d). The goal in the work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] is to match these bounds, that is exhibit an algorithm which needs only n ∝ ε -d/r evaluation points to achieve ε error.

We say that such an algorithm breaks the curse of dimensionality in the exponent using smoothness in the sense that the number of samples needed in the worst case is not n ∝ ε -d (as is the case in the Lischitz case and basic algorithms), but of order ε -d/r . Note that the curse of dimensionality sometimes denotes the fact that we need much more regularity to optimize in high dimensions : in that case, of course, we do not break that specific curse, but rather add to its statement.

Assumptions

The work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] is done in the following setting. Assume Ω is a bounded open subset of R d , that f ∈ C r b (Ω) for some r ≥ 0, and that the following assumption is satisfied. Assumption 8.1. There exists ε > 0 such that {x ∈ X : f (x) -f * ≤ ε} is compact. Assumption 8.1 states that close enough to the optimum, the level sets are compact. This implies a) the existence of at least one minimizer, and b) that there is no minimizer near the boundary. Finally, [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] take the Sobolev kernel k s with s > d/2 (see Eq. (2.38)) to apply the method presented in Sec. 8.2 .1. Note that this choice of kernel is useful because of the direct link with differentiability, and makes the analysis clear. However, as can be seen in chapter 7, one can actually approximate such a kernel using a Gaussian kernel, with adapted bandwidth, and which is easier to implement in practice.

x f (x) f (x) -ε f (x) + ε f (x)
Unde these assumptions, we can easily prove that Eq. ( 8.32) actually solves the minimization problem, i.e., ρ * H = ρ H = f * . (8.43) This is a main difference with the polynomial setting, where Eq. ( 8.43) only happened if ff * was a sum of squares of polynomials. In this setting, the space SOS H is large enough to contain any close approximation of ff * . However, this does not give us any information on ρ n,λ , which is the result of the method by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF].

Similarly to the well-specified setting presented in part I, in order to obtain guarantees on ρ n,λ , it is useful for the problem to be well specified, i.e., that the solutions to Eqs. (8.32) and (8.33) be reached at certain values µ * for the primal or (c * , g * ) for the dual.

Guarantees on the relaxation

The first step of the work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] is therefore to show that under certain additional assumptions, there is an exact solution to Eq. (8.32). To do so, the following assumption is made on the set of minimizers.

Assumption 8.2. All minimizers of f are strict global minimizers, i.e., the Hessian at all minimizers is positive definite.

The following theorem is a simplification of Theorem 9.3. In particular, note that the condition r ≥ s + 2 will imply that r > d/2 + 2.

Main theorem 12: [START_REF] Rudi | Finding global minima via kernel approximations[END_REF], Theorem 3.

Assume that r -2 ≥ s and recall that H is the reproducing kernel Hilbert space associated to the Sobolev kernel k s . Under the assumptions above (including Assumptions 8.1 and 8.2) on the kernel and the function, there exists a finite rank operator A * ∈ S + (H) with rank at most d|Z| + 1 (where |Z| is the size of the set of minimizers), such that : (8.44) In particular, this shows that Eq. ( 8.32) has an optimal solution (c * , g * ) with c * = f * and

∀x ∈ Ω, f (x) -f * = k x , A * k x H .
g * = k (•) , A * k (•) H .
The proof of this theorem is based on a local decomposition as a sum of squares around each global minimum, before gluing the decompositions together using standard gluing lemmas from differential geometry. Using the fact that the Hessian is positive definite, we perform an exact Taylor-type decomposition around each global minimum to the second order : that is why we lose an exponent 2 in the bounds, i.e., r ≥ s + 2. This approach is generalized in Marteau-Ferey, Bach, and Rudi (2022b) in order to go to functions which have a continuous set of zeros, and which are defined on a manifold (see Sec. 8.2 .3 for more details).

Guarantees after restriction to a finite number of points and near-optimal algorithm

The second step of the method presented by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] is to guarantee the equality constraint in Eq. ( 8.32) only on a finite subset X n = {x 1 , ..., x n } ⊂ Ω and to regularize the problem. To analyse the performance of the resulting ρ * n,λ , the authors use the fill distance h Xn,Ω of X n in Ω, defined as

h Xn,Ω = sup x∈Ω inf x i ∈Xn x -x i R d .
(8.45)

The following simplified theorem is obtained for the performance of ρ * n,λ , which is the solution of Eq. (8.35), whose details can be found in Theorem 9.5 and has originally been proved as Theorem 5. by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF].

Main theorem 13: Rudi, Marteau-Ferey, and Bach (2020), Theorem 5.

Assume that the conditions of Main theorem 12 are satisfied, and that Ω is a ball of radius R for simplicity. Let r < sd/2 There exists constants C r ,d , C r ,d depeding only on r , d and a constant h R,r depending only on R, r such that if g * ∈ S + (H) is the optimal solution satisfying ff * = g * , we have (8.46) as soon as h Xn,Ω ≤ h R,r , λ ≥ C r ,d h r Xn,Ω .

|ρ * n,λ -f * | ≤ C r ,d |f | Ω,r h r Xn,Ω + λ g 1 ,
The proof of this theorem relies on bounds obtained through scattered data analysis [START_REF] Wendland | Scattered Data Approximation[END_REF], and the fact that H s embeds itself in C r (Ω) for any r < sd/2.

Assuming that Ω is a ball of radius R, and that we can generate i.i.d. samples from the uniform measure on the ball, we can choose the points x i randomly with the following guarantee on the fill distance.

Lemma 8.1 (Lemma 9.4). Let X n = {x 1 , . . . , x n } independent points sampled from the uniform distribution on Ω. There exists a constant n d depending only on d such that for any δ > 0, if n ≥ n d log 2 δ , then the following holds with probability at least 1 -δ: (8.47) Note that Lemma 9.4 is a more general result for sets which go beyond the simple ball of radius R.

h Xn,Ω ≤ 11R n -1 d (log 2 d n δ ) 1/d .
In the case of a hypercube, one could simply take a grid X n , and we would have

h Xn ≤ Cn -1/d
where n is the number of points in the grid.

We now state Theorem 9.6, in the case where we subsample uniformly n points from the uniform measure when Ω is a ball of radius R.

Main theorem 14 [START_REF] Rudi | Finding global minima via kernel approximations[END_REF], Theorem 6

Assume that Ω is a ball of radius R, that H is the RKHS associated to the Sobolev kernel k s for s > d/2. Let r ≥ s + 2 and 0 ≤ r < sd/2. There exists constant n 0 and C 0 depending on s, r , d, r such that the following hold. Let f ∈ C r b (Ω) satisfying Assumptions 8.1 and 8.2, ρ * n,λ be the solution of Eq. ( 8.35), and δ

∈ (0, 1]. If n ≥ n 0 log 2 δ and λ satisfies λ ≥ C 0 n -r d (log 2 d n δ ) r d , (8.48)
then, with probability at least 1 -δ, (8.49) In particular, assuming r > d/2 + 2 and taking s = r -2 and r = r -3 -d/2 and taking λ n as in the lower bound of Eq. ( 8.48), we have constants n 1 , C 1 depending only on r, d such that as soon as n ≥ n 1 log 2 δ , it holds (8.50) This result shows, up to logarithmic terms, that with the right choice of regularization, if f ∈ C r b (Ω) and satisfies the additional assumptions Assumptions 8.1 and 8.2, the method presented in Sec. 8.2 .1 achieves an error of order n -r/d+3/d+1/2 up to logarithmic and constant terms (which do not depend on n). This is to compare to the worst case bound obtained by [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF], which was n -r/d . We are short of a term 3/d + 1/2, which is due to the fact that a) our decomposition as a sum of squares reduces the regularity, and b) that we use the very loose fact that C s b (Ω) ⊂ W s 2 (Ω). However, this results still shows that in spirit, as soon as the regularity is of the order of the dimension, one does not pay the dimensionality in the rate (i.e., approximately, | ρ * n,λ -f * | ≤ Cn -1/2 ), although the dimension is still present in the constants.

| ρ * n,λ -f * | ≤ 3 λ g * + |f | Ω,r .
| ρ * n,λ -f * | ≤ C 1 g * + |f | Ω,r n -(r/d-3/d-1/2) (log 2 d n δ ) r/d-2/d-1/2 .
To end this part on the guarantees on the discretized regularized problems, note that the original results, given in chapter 9, also focus on providing explicit bounds for the constants in the above theorems. That is why there is a particular condition on the domain Ω, (although that condition is more general than that of being a ball of radius R, see Assumption 9.1). The same reasoning could be applied on a domain satisfying a uniform cone condition, as in chapter 4 of [START_REF] Adams | Sobolev Spaces[END_REF], and extend these results formally to a wider class of domains, although keeping track of the constants would lead to long and painful computations.

Extraction

Until now, we have only talked about approximating a minimum of f , but not a minimizer. In order to be able to approximate a minimizer, [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] use Eq. (8.41), and set x = n i=1 α i x i , where the α i are the optimal primal variables. This is justified formally in Sec. 9.7 , by making the assumptions that (i) there is a unique minimizer x * of f ;

(ii) there is a parabola of the form ν xx * 2 with small enough ν such that this parabola is a lower bound for f . Under these assumptions, [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] provide an algorithm to retrieve both an approximation of the minimum and of the minimizer, using the same principle as the one described above, and with convergence guarantees of the form ν x n,λ -x * 2 ≤ C n -d/r+3/d+1/2 for a suitable choice of λ (see Theorem 9.8).

.3 Going to functions with continuous sets of zeros on manifolds

The method presented in Sec. 8.2 .1 was quite general, and did not at all need that X be a domain of R d to be applied. However, the analysis we have performed in this section crucially relies on certain properties of RKHS on R d , which are needed to prove that the problem Eq. ( 8.32) is not only tight, but also that there exists solutions to these problems (see Main theorem 12).

The goal of the work by Marteau-Ferey, Bach, and Rudi (2022b) is to extend these results which rely on the fact that certain regular non-negative functions can be decomposed as sums of squares of regular functions. It was motivated by two main lines of applications of this method we would like to explore.

• The first was to be able to decompose functions whose sets of minimizers is not discrete (as implied by Assumption 8.2), but wich can have some smoothness properties, such as being a manifold. This has been motivated by applications to optimal transport, where such sets of minimizers naturally appear (see [START_REF] Vacher | A Dimension-free Computational Upper-bound for Smooth Optimal Transport Estimation[END_REF]).

• The second was to be able to move from R d to any manifold M , in order to apply this algorithm to minimize functions on manifolds.

The high level ideas of the work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] are the following. To generalize from domains of R d to manifolds, one simply notes that the result was obtained in the R d case by gluing decompositions. The same reasoning can be applied for manifolds, and hence the real crux of the problem is to show that certain regular non-negative functions can be locally decomposed as sums of squares of regular function on R d (by composing with a chart). In the work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF], the strategy to show a local decomposition as sums of squares was to use the fact that the Hessian at the optimal point was positive definite (Assumption 8.2) to write and exact second order decomposition, which naturally made a sum of squares decomposition appear (such as the ones obtained through the traditional Morse lemma, see [START_REF] Milnor | Morse theory[END_REF]). In the work by Marteau-Ferey, Bach, and Rudi (2022b), this is adapted to the setting where the set of zeros is a submanifold of R d , and Assumption 8.2 is modified to allow a continuous set of zeros. Instead, it imposes that orthogonally to the manifold of zeros, the Hessian is positive definite. Both of these situations and their difference are illustrated in Fig. 8.2. Denoting with T x N the set of tangeant vectors to a manifold N at x, we prove the following theorem (see chapter 10 for more details and references).

Main theorem 15: Marteau-Ferey, Bach, and Rudi (2022b), Theorem 3.9

Let f be a non-negative function of class C p for p ≥ 2 and let Z denote the set of zeros of f . If Z is a compact sub-manifold of M of class C 1 such that (8.51) then there exists a finite number of functions

∀x 0 ∈ Z, ∀h ∈ T x 0 M \ T x 0 Z, H(x 0 )[h, h] > 0,
f 1 , ..., f N ∈ C p-2 (M ) such that ∀x ∈ M, f (x) = N i=1 f i (x) 2 . ( 8

Similarities and differences between the two approaches

We conclude this chapter by putting together Secs. 8.1 and 8.2 , and showing the structural similarities and differences between the moment-SOS hierarchy of lower bounds for polynomials and the method proposed by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF].

.1 Structure of the methods

First, we compare the structure of the methods, i.e., the different key algorithmic steps to obtain an approximation of the minimum/minimizer. Right: Representation of a non-negative function on the sphere as a color map; it satisfies the assumptions of Marteau-Ferey, Bach, and Rudi (2022b) to be decomposed as a sum of squares, and its null space Z is represented in black.

The same relaxation

The first step of both methods is very similar : we relax the primal problem (or strengthen the dual) into a positive semidefinite problem. If we fix r ∈ N, the set SOS r [x] of SOS polynomials of degree at most r (see Eq. ( 8.3)) corresponds exactly to the set SOS H in Eq. ( 8.31) where we set H = R r [x] equipped with a kernel structure. For example, we can take k(x 1 , x 2 ) = |α|≤r 1 x α x α 2 , where the kernel structure is inherited from the linear representation v r defined in Sec. 8.1 : k(x 1 , x 2 ) = v r (x 1 ) v r (x 2 ) and where the norm of a polynomial f = (f α ) |α|≤r in this RKHS is just the euclidean norm of the coefficients, i.e., f 2 H = |α|≤r f 2 α .

However, while we can identify this first relaxation step, there is a big difference between the two approaches at this point, owing to the necessity of solving opposite problems.

• In the polynomial case, the set H = R r [x] is finite dimensional and hence is not large enough to perform a tight relaxation.

• On the contrary, in the standard kernel sum of squares, the set H can be taken in a way that ρ * H = ρ H = f * (see Eq. (8.43)). However, the relaxed problem is therefore still infinite dimensional, and there is a need to decrease the dimension.

Hierarchichal structure for polynomials and the need for constraints As highlighted above, since for a single r, the moment-SOS strengthening of P + (R d ) by SOS r [x] is not tight enough : we need to make r grow in roder to get closer and closer to P + (R d ), or, from another perspective, for ρ * r to get closer and closer to f * . However, as we have explained in Sec. 8.1 , keeping to R d and using an increasing r is not enough to approximate f * well, i.e., we do not necessarily have ρ * r ↑ f * .

However, it is the case in the constrained setting, where the polynomial contraints add some algebraic structure and allows to for ρ r to go to f * . Indeed, for polynomials, adding constraints loosens the rigidity of the polynomial structure, by allowing more cancellations between coefficients. This setting is the most widespread, as a lot of problems are constrained or can easily be constrained (when the minimum is roughly localized in a given region for instance). On the contrary, in the kernel sum of squares case, there is no need for a hierarchy or for constraints for the proiblem to be tight. However, the initial problem is infinite dimensional and cannot be solved as such.

The technique developed by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] is to reduce the problem of approximating the function ff * by a sum of squares of functions g to a finite number of evaluation points (x 1 , ..., x n ), regularizing in order to avoid undesirable side effects of overfitting. This is a typical machine learning approach, as described in the general introduction.

The drawback of this approach is twofold. The theoretical analysis imposes a stronger assumption than simply tightness (i.e., that ρ * H = f * ), it imposes that an optimal solution exists. This assumption is much stronger than the tightness assumption, and proves to be one of the real challenges for analyzing these methods. Moreover, the a posteriori analysis of the returned solution is not obvious to do.

Solving a semidefinite program

The last structural point which appears in both moment-SOS polynomial hierarchies and in our setting is the need, in the end, to solve semidefinite programs. This can be quite challenging, and we see that the complexity grows in order d r where r is the degree of the hierarchy used in the polynomial setting, while growing in order n 3.5 in the kernel sum of squares setting. This shows that these two methods have really different limiting conditions in terms of computation time. The SOS hierarchy is limited by the complexity of the problem through its dimension and the degree needed in the hierarchy, while the kernel SOS technique is limited by the quality of the approximation of the contraint ff * = g using n points.

.2 Two different type of guarantees

The difference between these two methods is reflected in the guarantees we can obtain. Indeed, as polynomials are more rigid, the a priori convergence guarantees are relatively weak, and speed of convergence of the hierarchy cannot be analyzed. However, the objects involved are much more structured, and one can obtain good a posteriori guarantees. Symmetrically, in the kernel sum of squares setting, we obtain very good rates and non-asymptotic guarantees. However, as the model is implicit and large, it is not obvious to obtain a posteriori guarantees from a returned solution, in particular because the functions f we handle are much less structured.

Guarantees for the moment-SOS hierarchy

In the polynomial case, the a priori guarantees show convergence of the hierarchy of lower and upper bounds, and even finite convergence in some cases. Moreover, for some specific domains K, this convergence can be shown to have error 1/(r ) 2 , where r is the degree of the hierarchy, thus showing that a degree of order 1/ √ ε is needed to reach ε error. However, as soon as the underlying dimension d is large, the number of variables grows exponentially as d r in terms of the order r of the hierarchy, thus making high orders for high precision ε unreachable (even though some sparsification techniques can be applied).

In practice, the a posteriori guarantees can be very strong. When the algorithm finishes, the user can have the following information :

• a lower bound ρ r on f * ;

• if a rank condition holds, the knowledge that ρ r = f * and a procedure to compute a minimizer;

• if not, a technique to approximate the minimizer if certain conditions are satisfied.

As explained above, the situation is quite different in the kernel sum of squares setting.

Guarantees for kernel sum of squares

There are strong a priori guarantees :

• fast rates of convergence with regularity for the minimum (i.e. of order ε -C m/d for a constant C in time and space, to be compared to the space complexity of order d 1/ √ ε in the polynomial case);

• fast rates of convergence for the minimizer under certain additional assumptions.

Howoever, there are no a posteriori guarantees, except for the one given by the approximation of the minimizer (i.e., we will know that f ( x) ≥ f * ).

In term of complexity, this methods has a real advantage, as it can always be run, and the user can hope that it will work well in favorable settings. Moreover, note that the rates of convergence derived in chapter 9 are expressed using certain theoretical quantities which give us insight into what makes a problem difficult or not : the norms of the elements in the sum of squares decomposition for example.

.3 Building connexions

As we have seen in the two previous sections, there are fundamental similarities and differences between these two methods : in one, one seeks to tackle the rigidity of polynomials while in the other, one tackles the "large dimension" of the kernel models.

However, the variety of fields in which the moment-SOS hierarchies have been applied also provides a plethora of interesting task for the kernel sum of squares to tackle, accompanied of course by additional challenges to solve. We will go a bit further into detail about these different directions in chapter 11, and just formulate a few questions which directly appear when looking at the analysis and comparison above.

Are there ways to keep a lower a posteriori bound in the kernel setting ?

One of the main advantages of polynomial optimization from the user point of view is that it provides lower bounds (which can directly be coupled to an upper bound if an approximation of the minimizer is returned). This, of course, cannot be done using the technique developed by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] as the evaluation breaks the guarantee of being a lower bound. Therefore, can we find a way to get both better approximation properties for larger classes of functions with kernels, while keeping an upper bound ?

Constraints

We also have seen that adding constraints helps the optimization guarantees in the polynomial case. Does something similar exist in the kernel case (for example, maybe this reduces the norm of optimal solutions) ?

Moments

In the setting we described for the kernel sum of squares case, we did not once talk about moments, which were central objects in the polynomial case. This is something which would be very interesting to explore in greater detail, and can be defined, although in a way which is a bit more involved than in the polynomial setting.

Let k be a kernel and H the associated RKHS. We consider the RKHS H ⊗2 associated to the kernel k 2 , and note that there is a natural embedding Op :

H ⊗2 → S 2 (H) which extends the map i α i k 2 x i ∈ H ⊗2 → i α i k x i ⊗ H k x i ∈ S 2 (H)
, where S 2 (H) is the set of Hilbert-Schmidt operators on H (in particular, this set contains the set S 1 (H) of trace class operators). In that case, the set of moments would be In particular, we will consider the setting where (a) the function is smooth, that is, f ∈ C m (R d ) with m ∈ N + (f m-times continuously differentiable), and (b) we are able to evaluate it on given points, without the need of computing the gradient. For this class of problems there are known lower-bounds [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF][START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF] that show that it is not possible to achieve a global minimum with error ε with less than O(ε -d/m ) function evaluations. In this paper, we want to achieve this lower bound in terms of function evaluations, while having an optimization algorithm which has a running-time which is polynomial in the underlying dimension and the number of function evaluations.

M H = g ∈ H ⊗2 : ∃µ ∈ M (X ), g = X k 2 x µ(dx), 1, g H ⊗2 = 1,
Several methods are available to solve this class of problems. For example, the function f can be approximated from its values at n sampled points, and the approximation of the function globally minimized instead of f . If the approximation is good enough, then this can be optimal in terms of n, but computationally infeasible. Optimal approximations can be obtained by multivariate polynomials [START_REF] Viktor | On optimum minimization algorithms in classes of differentiable functions[END_REF] or functions in Sobolev spaces [START_REF] Novak | Tractability of Multivariate Problems: Standard Information for Functionals[END_REF], with potentially adaptive ways of selecting points where the function is evaluated (see, e.g., the work by [START_REF] Osborne | Gaussian processes for global optimization[END_REF] and references therein). Alternatively, when the function is itself a polynomial, algorithms based on the "sum-of-squares" paradigm can be used, but their computational complexity grows polynomially on d r/2 , where r is in the most favorable situations the order of the polynomial, but potentially larger when so-called hierarchies are used [START_REF] Lasserre | Global optimization with polynomials and the problem of moments[END_REF][START_REF] Laurent | Sums of squares, moment matrices and optimization over polynomials[END_REF][START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF].

It turns out that the analysis of lower-bounds on the number of function evaluations shows an intimate link between function interpolation and function minimization, i.e., the lower-bounds of one problem are the same for the other problem. However, existing methods consider a two-step approach where (1) the function is approximated optimally, and (2) the approximation is minimized. In this paper, we consider a joint approach where approximation and optimization are done jointly.

We derive an algorithm that casts the possibly non-convex problem in Eq. ( 9.1) in terms of a simple convex problem based on a non-parametric representation of non-negative functions via positive definite operators [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. As shown below, it can be considered as an infinite-dimensional counter-part to polynomial optimization with sums of squares, with two key differences: (1) the relaxation is always tight for the direct formulation, and (2) the computational cost does not depend on the dimension of the model (here infinite anyway), by using a subsampling algorithm and a computational trick common in statistics and machine learning.

The resulting algorithm with n sampled points will be able to achieve an error of ε = O(n -m/d+3/d+1/2 ) as soon as m ≥ 3 + d/2, with n function evaluations to reach the global minimum with precision ε, and a computational complexity of O(n 3.5 log(1/ε)) (with explicit constants). This is still not the optimal complexity in terms of number of function evaluations (which is ε = O(n -m/d )), but this is achieved with a polynomial-time algorithm in n. This is particularly interesting in the contexts where the function to be optimized is very smooth, i.e., m d, possibly C ∞ or a polynomial. For example, if the function is differentiable at least d + 3 times, even if non-convex, the proposed algorithm finds the global minimum with error O(n -1/2 ) and time O(n 3.5 log n).

→ R, Ω, n ∈ N + , λ > 0, s > d/2. 1: X ← {x 1 , . . . , x n } Sampled i.i.d. uniformly on Ω 2: f j ← f (x j ), ∀j ∈ [n] Features computation 3: K ij ← k(x i , x j ) i, j ∈ [n]
k Sobolev kernel of smoothness s, Eq. ( 9.7) 4: R ← cholesky(K) upper triangular Cholesky 5:

Φ j = j-th column of R, ∀j ∈ [n]
Solution of the approximate problem (use any algorithm in Sec. 9.6 )

6: ĉ ← max c∈R,B∈S + (R n ) c -λ Tr(B) such that ∀j ∈ [n], f j -c = Φ j BΦ j 7: return ĉ
Hilbert spaces [START_REF] Berlinet | Reproducing Kernel Hilbert Spaces in Probability and Statistics[END_REF], such as Sobolev spaces [START_REF] Adams | Sobolev Spaces[END_REF].

Subsampling. We approximate the problem above as follows. Given a finite set X = {x 1 , . . . , x n } which is a subset of Ω, we restrict the equality in Eq. ( 9.3) to only x 1 , . . . , x n .

Unlike the case of polynomial optimization where subsampling is exact if n is large enough [START_REF] Lasserre | A bounded degree SOS hierarchy for polynomial optimization[END_REF], in our case subsampling leads to an error that decreases in n and depends on the regularity of f and of the map x → φ(x), Aφ(x) . While f is smooth enough by assumption, we need to control the regularity of the map induced by A, to guarantee that the constraints subsampled on X well approximate the whole set of constraints on Ω. Then we consider a penalization term based on the trace of A and solve the following problem

max c∈R, A∈S + (H) c -λ Tr(A)
such that ∀i ∈ {1, . . . , n}, f (x i ) -c = φ(x i ), Aφ(x i ) , (9.4) for some positive λ (with the implicit assumption that we optimize over operators A with finite trace). We show in this paper that solving Eq. ( 9.4) leads to an approximate optimum of the original problem in Eq. ( 9.2), when n is large enough and λ small enough. Note that the value of c which we obtain after subsampling in not anymore a lower bound on the global minimum, but we can provide both a priori and a posteriori certificates of optimality (see Sec. 9.8 .3).

Finite-dimensional algorithm. The problem in Eq. ( 9.4) is still formulated in an infinitedimensional space. We can leverage the particular choice of penalty by the trace of A and the choice of Hilbert space to obtain a finite-dimensional algorithm. Indeed, for reproducing kernel Hilbert spaces, then, following (Marteau-Ferey, Bach, and Rudi, 2020), we only need to solve the problem in the finite-dimensional Hilbert space spanned by φ(x 1 ), . . . , φ(x n ), that is, we only need to look at A of the form A = n i,j=1 C ij φ(x i ) ⊗ φ(x j ) for some positive semi-definite matrix C ∈ R n×n . We can then write Tr(A) = Tr(CK), with K ∈ R n×n the matrix of dot-products with K ij = φ(x i ), φ(x j ) = k(x i , x j ), and φ(x i ), Aφ(x i ) = (KCK) ii .

Consider the Cholesky decomposition of K as K = R R, with R ∈ R n×n upper-triangular. We can directly solve for B = RCR , noting that KCK = R BR and Tr(CK) = Tr(B). We can thus use a representation in terms of finite-dimensional vectors Φ 1 , . . . , Φ n ∈ R n defined as the columns of R. We thus study the following problem,

max c∈R, B∈S + (R n ) c -λ Tr(B)
such that ∀i ∈ {1, . . . , n}, f (x i ) -c = Φ i BΦ i . (9.5) From an algorithmic viewpoint the problem above can be solved efficiently since this is a semi-definite program. We show in Sec. 9.6 how we can apply Newton method and classical interior-point algorithms, leading to a computational complexity of O(n 3.5 log(1/ε)) in time and O(n 2 ) in space.

Note that in the context of sum-of-squares polynomials, the relationship with reproducing kernel Hilbert spaces had been explored for approximation purposes after a polynomial optimization algorithm is used [START_REF] Marx | Semialgebraic approximation using Christoffel-Darboux kernel[END_REF]. In this paper, we propose to leverage kernel methods within the optimization algorithm.

Why not simply subsampling the inequality? One straightforward algorithm is to subsample the dense set of inequalities in Eq. ( 9.2). Doing this will simply lead to outputting min i∈{1,...,n} f (x i ). This last algorithm, while easy to implement and convergent, is very slow, with a rate of O(n -2/d ) (see the discussion in Sec. 9.11 ). Subsampling the dense set of equalities in Eq. ( 9.3) allows to use smooth interpolation tools. When λ = 0, the optimal value is also min i∈{1,...,n} f (x i ) (if the kernel matrix is invertible, see Sec. 9.6 ), but for λ > 0, we can leverage smoothness as shown below.

Theoretical guarantees. From a theoretical viewpoint, denoting by ĉ the minimizer of Eq. ( 9.5), we provide upper bounds for |f * -ĉ| with explicit constants and that hold under mild geometrical assumptions on f . We prove that the bound depends on how the points in X = {x 1 , . . . , x n } are chosen. In particular we prove that when they are chosen uniformly at random on Ω, the problem in Eq. (9.5) achieves the global minimum with error ε with a precise dependence on n.

The results in this paper hold under the following assumptions. Note that Assumption 9.1(a) can be easily relaxed to Ω having locally Lipschitz-continuous boundaries (Adams and Fournier, 2003, Section 4.9). Assumption 9.1(b) is satisfied if all global minimizers of f are in Ω, and are second-order strict local minimizers. Note that similar assumptions are made to show finite convergence for polynomial optimization hierarchies [START_REF] Nie | Optimality conditions and finite convergence of Lasserre's hierarchy[END_REF]. Relationship to polynomial optimization. When f is a polynomial of degree 2r, then it is natural to consider φ(x) composed of all monomials of degree less than r, leading to a space H of dimension d+r r . All polynomials can be represented as f (x) = c + φ(x) Aφ(x) for some symmetric matrix A. When A 0, by using its eigendecomposition, we can see that the polynomial x → φ(x) Aφ(x) is a sum-of-squares polynomial.

However, in general A may not be positive semi-definite, as non-negative polynomials are not all sum-of-squares. Moreover, even when there exists a matrix A 0, the corresponding c may not be the minimum of f (it only needs to be a lower bound)-see, e.g., the work by [START_REF] Lasserre | Global optimization with polynomials and the problem of moments[END_REF] and references therein.

If f (x) -f * is a sum of squares, then, with λ = 0 and n = d+2r 2r points (to ensure that subsampling is exact), we exactly get the minimum of f , as we are solving exactly the usual optimization problem.

When f (x) -f * is not a sum of squares, then a variety of hierarchies have been designed when optimization is performed on a compact constraint set described with polynomial inequalities (such as taking an 2 -ball for Ω), that augment the problem dimensionality to reach global convergence [START_REF] Lasserre | Global optimization with polynomials and the problem of moments[END_REF][START_REF] Laurent | Sums of squares, moment matrices and optimization over polynomials[END_REF][START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF]. In Sec. 9.9 , we show how our framework fits with one these hierarchies, and also can provide computational gains.

Note that our framework, by looking directly at an infinite-dimensional space circumvents the need for hierarchies, and solves a single optimization problem. The difficulty is that it requires sampling. Moreover by using only kernel evaluations, we circumvent the explicit construction of a basis for H, which is computationally cumbersome when d grows.

Organization of the paper. The paper is organized as follows: in Sec. 9.3 , we present the kernel setting our paper relies on; then, in Sec. 9.4 , we analyze the infinite-dimensional problem and show its equivalence with global minimization. Then, in Sec. 9.5 , we present our theoretical guarantee for the finite-dimensional algorithm, as summarized in Theorem 9.1. In Sec. 9.6 we present the dual algorithm based on self-concordant barriers and the damped Newton algorithm. In Sec. 9.7 , we present our extension to find the global minimizer, while in Sec. 9.8 , we provide certificates of optimality for potentially inexactly solved problems. In Sec. 9.9 , we discuss further relationships with polynomial hierarchies, and provide illustrative experiments in Sec. 9.10 . We conclude in Sec. 9.11 with a discussion opening up to many future problems.

Setting

In this section, we first introduce some definitions and notation about reproducing Kernel Hilbert spaces in Sec. 9.3 .1 (for more details, see the work by [START_REF] Aronszajn | Theory of reproducing kernels[END_REF]; [START_REF] Paulsen | An Introduction to the Theory of Reproducing Kernel Hilbert Spaces[END_REF]), and present our detailed assumptions in Sec. 9.3 .2. In Sec. 9.4 we show how our infinite-dimensional sum-of-squares representation can be built, and in Sec. 9.5 we provide guarantees on subsampling.

.1 Definitions and notation

In this section we denote by u • v, a • v respectively the pointwise multiplication between the functions u and v, and the composition between the functions a and v. We denote by N the set of natural numbers including 0, by N + the set N + = N \ {0} and [n] the set {1, . . . , n} for n ∈ N + . We will always consider R d endowed with the Euclidean norm • if not specified otherwise. We always endow R p with the standard inner product x y = p i=1 x i y i for any x, y ∈ R p . In the case H = R p , with the standard inner product, then A ∈ R p×p is a matrix and the Hilbert-Schmidt norm corresponds to the Frobenius norm. We say that A 0 or A is a positive operator (positive matrix if H is finite dimensional), when A is bounded, self-adjoint, and u, Au ≥ 0, ∀u ∈ H. We denote by S + (H) the space of positive operators on H. Moreover, we denote by A 0, or A strictly positive operator, the case u, Au > 0 for all u ∈ H such that u = 0. Kernels and reproducing kernel Hilbert spaces. For this section we refer to the work by [START_REF] Aronszajn | Theory of reproducing kernels[END_REF]; [START_REF] Steinwart | Support Vector Machines[END_REF]; [START_REF] Paulsen | An Introduction to the Theory of Reproducing Kernel Hilbert Spaces[END_REF], for more details (see also Sec. 9.A .3,page 421). Let Ω be a set. A function k : Ω × Ω → R is called a positive definite kernel if all matrices of pairwise evaluations are positive semi-definite, that is, if it satisfies the following equation n i,j=1

α i α j k(x i , x j ) ≥ 0, ∀n ∈ N, α 1 , . . . , α n ∈ R, x 1 , . . . , x n ∈ Ω.
Given a kernel k, the reproducing kernel Hilbert space (RKHS) H, with the associated inner product •, • , is a space of real functions with domain Ω, with the following properties.

(a) The function k x = k(x, •) satisfies k x ∈ H for any x ∈ Ω.

(b) The inner product satisfies f, k x = f (x) for all f ∈ H, x ∈ Ω. In particular k x , k x = k(x , x) for all x, x ∈ Ω.

In other words, function evaluations are uniformly bounded and continuous linear forms and the k x are the evaluation functionals. The norm associated to H is the one induced by the inner product, i.e., f 2 = f, f . We remark that given a kernel on Ω there exists a unique associated RKHS on Ω [START_REF] Berlinet | Reproducing Kernel Hilbert Spaces in Probability and Statistics[END_REF]. Moreover, the kernel admits a characterization in terms of a feature map φ,

φ : Ω → H, defined as φ(x) = k(x, •) = k x , ∀x ∈ Ω.
Indeed according to the point (b) above, we have k(x, x ) = φ(x), φ(x ) for all x, x ∈ Ω. We will conclude the section with an example of RKHS that will be useful in the rest of the paper.

Example 9.1 (Sobolev kernel [START_REF] Wendland | Scattered Data Approximation[END_REF])). Let s > d/2, with d ∈ N + , and Ω be a bounded open set. Let

k s (x, x ) = c s x -x s-d/2 K s-d/2 ( x -x ), ∀x, x ∈ Ω, (9.7)
where K : R + → R the Bessel function of the second kind (see, e.g., 5.10 in the work by Wendland ( 2004)) and c s = 2 1+d/2-s Γ(s-d/2) . The constant c s is chosen such that k s (x, x) = 1 for any x ∈ Ω.

In the particular case of s = d/2 + 1/2, we have k(x, x ) = exp(-xx ). Note that a scale factor is often added as k(x, x ) = exp(-xx /σ) in this last example. In such case, all bounds that we derive in this paper would then have extra factors proportional to powers of σ. To conclude, when Ω has locally Lipschitz boundary (a sufficient condition is Assumption 9.1(a)) then H = W s 2 (Ω), where W s 2 (Ω) is the Sobolev space of functions whose weak-derivatives up to order s are square-integrable [START_REF] Adams | Sobolev Spaces[END_REF]. Moreover, in this case • H is equivalent to • W s 2 (Ω) . Reproducing kernel Hilbert spaces are classically used in fitting problems, such as appearing in statistics and machine learning, because of function evaluations f → f (x) are bounded operators for any x, and optimization problems involving f only through function evaluations at a finite number of points x 1 , . . . , x n , and penalized with the norm f , can be solved by looking only a f of the form f (x) = n i=1 α i k(x, x i ) [START_REF] Aronszajn | Theory of reproducing kernels[END_REF][START_REF] Paulsen | An Introduction to the Theory of Reproducing Kernel Hilbert Spaces[END_REF]. We will use an extension of this classical "representer theorem" to operators and spectral norms in Sec. 9.5 .

.2 Precise assumptions on reproducing kernel Hilbert space

On top of Assumption 9.1 (made on the function f and the set Ω), we make the following assumptions on the space H and the associated kernel k.

(a) w| Ω ∈ H, ∀w ∈ C ∞ (R d ). Moreover there exists M ≥ 1 such that u • v H ≤ M u H v H , ∀u, v ∈ H. (b) a • v ∈ H, for any a ∈ C ∞ (R p ), v = (v 1 , . . . , v p ), v j ∈ H, j ∈ [p].
(c) Let z ∈ R d , r > 0 s.t. the ball B r (z) is in Ω. For any u ∈ H, there exists g r,z ∈ H s.t.

g r,z (x) = 1 0 (1 -t)u(z + t(x -z))dt, ∀x ∈ B r (z).
(d) H is a RKHS with associated kernel k. For some m ∈ N + and some D m ≥ 1, the kernel k satisfies max

|α|=m sup x,y∈Ω |∂ α x ∂ α y k(x, y)| ≤ D 2 m < ∞.
M = (2π) d/2 2 s+1/2 , D m = (2π) d/4 Γ(m + d/2)Γ(s -d/2 -m) Γ(s -d/2)Γ(d/2
) .

The proof of proposition above is in Sec. 9.D .2, page 429. We make a last assumption regarding the differentiability of f , namely that f and its second-derivatives are in H.

Assumption 9.3 (Analytic properties of f ). The function f satisfies f | Ω ∈ C 2 (Ω) ∩ H and ∂ 2 f ∂x i ∂x j | Ω ∈ H for all i, j ∈ [d].

Equivalence of the infinite-dimensional problem

In Theorem 9.2 and Cor. 9.1, we provide a representation of ff * in terms of an infinitedimensional, but finite-rank, positive operator, under basic geometric conditions on f and algebraic properties of H. In Theorem 9.3 we use this operator to prove that Eq. ( 9.3) achieves the global minimum of f . In this section we analyze the conditions under which the problem in (9.3) has the same solution as the one in Eq. ( 9.2).

The proof follows by explicitly constructing a bounded positive operator A * (which will have finite trace) that satisfy f (x) -f * = φ(x), A * φ(x) for all x ∈ Ω. Note that, by construction ff * is a non-negative function. If w := √ ff * ∈ H then A * = w ⊗ w would suffice. However, denoting by ζ ∈ Ω a global minimizer, note that f (ζ) -f * = 0 and the smoothness of

√ f -f * may degrade around ζ, making √ f -f * / ∈ H even if f -f * ∈ H.
Here we follow a different approach. In Lemma 9.1 we provide a decomposition that represents the function ff * locally around each global optimum using the fact that it is locally strongly convex around the minimizers. In the proof of Theorem 9.2 we provide a decomposition of the function far from the optimal points; we then glue these different decompositions via bump functions.

Lemma 9.1. Let H be a space of functions on Ω that satisfy Assumptions 9.2(a) to 9.2(c).

Let ζ ∈ Ω and r, γ > 0. Let B r (ζ) ⊂ Ω be a ball centered in ζ of radius r and g ∈ C 2 (Ω) satisfy g(ζ) = 0, ∇ 2 g(x) γI for x ∈ B r (ζ) and ∂ 2 ∂x i ∂x j g ∈ H for i, j ∈ [d].
Then, there exists

w j ∈ H, j ∈ [d] such that g(x) = d j=1 w j (x) 2 , ∀x ∈ B r (ζ). ( 9
(x) = (x -ζ) R(x)(x -ζ), R(x) = 1 0 (1 -t)∇ 2 g(ζ + t(x -ζ))dt.
(9.9)

= d j=1 λ j u j u j , √ M = d j=1 λ j u j u j ). Thus, ∀x ∈ B r (ζ), g(x) = (x -ζ) S(x)S(x)(x -ζ) = d i=1 e i S(x)(x -ζ) 2 .
The following steps prove the existence of

w i ∈ H such that w i | Br(ζ) = e i S(•)(• -ζ).
Let (e 1 , ..., e d ) be the canonical basis of R d and S(R d ) be the set of symmetric matrices on R d endowed with Frobenius norm, in the rest of the proof we identify it with the isometric space R d(d+1)/2 (corresponding of taking the upper triangular part of the matrix and reshaping it in form of a vector).

Step 1. There exists a function R :

Ω → S(R d ), such that ∀i, j ∈ [d], e i Re j ∈ H and R| Br(ζ) = R.
This is a direct consequence of the fact that ∂ 2 ∂x i ∂x j g ∈ H for all i ≤ j ∈ [d], of Assumption 9.2(c) and of the definition of R in Eq. (9.9).

Step 2. There exists a function 2018) Thm. 1.1) and U ⊂ S + (R d ) then θ i,j is infinitely differentiable on U , i.e., θ i,j ∈ C ∞ (U ). By proposition 9.10, since K is a compact set in U , there exists θ i,j ∈ C ∞ 0 (S(R d )) such that ∀T ∈ K, θ i,j (T ) = θ i,j (T ). Define S(x) = i,j∈ [d] (θ i,j • R)(x)e i e j for any x ∈ Ω. Applying Assumption 9.2(b),

S : Ω → S(R d ) such that ∀i, j ∈ [d], e i Se j ∈ H and ∀x ∈ B r (ζ), S(x) = √ R(x).
e i Se j = θ i,j • R ∈ H since the R k,l ∈ H, k, l ∈ [d] and θ i,j is in C ∞ 0 (S(R d ))
. Moreover, by construction, for any x ∈ B r (ζ), we have R(x) = R(x) ∈ K and so

S i,j (x) = θ i,j (R(x)) = θ i,j (R(x)) = e i R(x)e j .
Note that here, we have applied proposition 9.10 and Assumption 9.2(b) to S(R d ) and not to R d(d+1)/2 ; this can be made formal by using the linear isomorphism between S(R d ) endowed with the Frobenius norm and R d(d+1)/2 endowed with the Euclidean norm.

Step 3. There exists a function h = (h j ) j∈ [d] :

Ω → R d such that ∀j ∈ [d], h j ∈ H and ∀x ∈ B r (ζ), h(x) = x -ζ.
w 2 i = h S d i=1
e i e i S h = h S 2 h.

Using the previous points,

∀x ∈ B r (ζ), i∈[d] w 2 i (x) = h (x)S 2 (x)h(x) = (x -ζ) R(x)(x -ζ) = g(x).
Now we are going to use the local representations provided by the lemma above to build a global representation in terms of a finite-rank positive operator. Indeed far from the global optima the function ff * is strictly positive and so we can take a smooth extension of the square root to represent it and glue it with the local representations around the global optima via bump functions as follows.

Theorem 9.2. Let Ω be a bounded open set and let H be a space of functions on Ω that satisfy Assumptions 9.2(a) to 9.2(c). Let f satisfy Assumptions 9.1(b) and 9.3. Then there exist w 1 , . . . , w q ∈ H with q ≤ dp + 1 and p ∈ N + the number of minimizers in Ω, such that Step 1. There exists r > 0 and α > 0 such that (i) the B r (ζ l ), l ∈ [p] are included in Ω and (ii) for any x ∈ l∈[p] B r (ζ l ), it holds ∇ 2 g(x) αI.

f (x) -f * = j∈[q] w j (x) 2 , ∀ x ∈ Ω. ( 9
By Assumption 9.1(b), for all ζ ∈ Z, ∇ 2 g(ζ) 0. Since ∇ 2 g is continuous, Z is a finite set, and Ω is an open set, there exists a radius r > 0 and α > 0 such that for all l ∈ [p], B r (ζ l ) ⊂ Ω and ∇ 2 g| Br(ζ l ) αI. For the rest of the proof, fix r, α satisfying this property. For any X ⊂ Ω denote with 1 X the indicator function of a X in Ω. We define χ 0 = 1 Ω\ l∈[p] B r/2 (ζ l ) , and

χ l = 1 Br(ζ l ) , l ∈ [p].
Step 2. There exists w 0 ∈ H s.t. w 2 0 χ 0 = gχ 0 . Ω is bounded and by Assumption 9.1(b), the set of global minimizers of f included in Ω is finite and there is no mimimizer of f on the boundary, i.e., there exists m 1 > 0 and a compact K ⊂ Ω such that ∀x ∈ Ω \ K, g(x) ≥ m 1 .

Moreover, f has no global optima on the compact K \ ζ∈Z B r/2 (ζ) since the set of global optima is Z, hence the existence of

m 2 > 0 such that ∀x ∈ K \ l∈[p] B r/2 (ζ l ), g(x) ≥ m 2 . Taking m = min(m 1 , m 2 ), it holds ∀x ∈ Ω \ l∈[p] B r/2 (ζ l ), g(x) ≥ m > 0. Since f ∈ C 2 (Ω), f is also bounded above on Ω hence the existence of M > 0 such that g ≤ M . Thus ∀x ∈ Ω \ l∈[p] B r/2 (ζ l ), g(x) ∈ I ⊂ (m/2, 2M ), I = [m, M ]. Since √ • ∈ C ∞ ((m/2, 2M )), (m/2, 2M
) is an open subset of R and I is compact, applying proposition 9.10, there exists a smooth extension s I ∈ C ∞ 0 (R) such that s I (t) = √ t for any t ∈ I. Now since g ∈ H and s I ∈ C ∞ 0 (R), by Assumption 9.2(b), w 0 := s I • g ∈ H. Since ∀x ∈ Ω \ l∈[p] B r/2 (ζ l ), g ∈ I, this shows gχ 0 = w 2 0 χ 0 .

Step 3. For all l ∈ [p], there exists (w l,j ) j∈[d] ∈ H d s.t. g(x)χ l = d j=1 w 2 l,j χ l . This is an immediate consequence of Lemma 9.1 noting that ∇g(x) ≥ αI on B r (ζ l ).

Step 4. There exists b l ∈ C ∞ (R d ) s.t. b l = b l χ l for all l ∈ {0, 1, . . . , p} and p l=0 b 2 l = 1. This corresponds to Lemma 9.7,Sec. 9.A .4, page 422 applied to the balls B r (ζ l ), l ∈ [p].

Step 5. Using all the previous steps

g = p l=0 g b 2 l = p l=0 g(χ l b l ) 2 = p l=0 (χ l g) (χ l b 2 l ) = (χ 0 w 2 0 ) (χ 0 b 2 0 ) + p l=1 χ l d j=1 w 2 l,j χ l b 2 l = ([b 0 χ 0 ] w 0 ) 2 + p l=1 d j=1 ([b l χ l ] w l,j ) 2 = (b 0 w 0 ) 2 + p l=1 d j=1 (b l w l,j ) 2 .
Applying Assumption 9.2(a) to each function inside the squares in the previous expressions yields the result.

A direct corollary of the theorem above is the existence of A * ∈ S + (H) when H is a reproducing kernel Hilbert space satisfying the assumptions of Theorem 9.2.

Corollary 9.1. Let k be a kernel whose associated RKHS H satisfies Assumptions 9.2(a) to 9.2(c) and let f satisfy Assumptions 9.1(b) and 9.3, then there exists

A * ∈ S + (H) with rank(A * ) ≤ d|Z| + 1 such that f (x) -f * = φ(x), A * φ(x) for all x ∈ Ω.
Proof. By Theorem 9.2 we know that if f satisfies Assumptions 9.1(b) and 9.3 w.r.t. a space H that satisfies Assumptions 9.2(a) to 9.2(c), there exists w 1 , . . . , w q ∈ H with q ≤ d|Z| + 1 such that f (x) -f * = j∈[q] w 2 j (x) for any x ∈ Ω. Since H is a reproducing kernel Hilbert space, for any h ∈ H, x ∈ Ω we have h(x) = φ(x), h H . Moreover, by the properties of the outer product in Hilbert spaces, for any h, v ∈ H, it holds ( h, v H ) 2 = h, (v ⊗ H v)h . Thus, for any x ∈ Ω, j ∈ [q], it holds w j (x) 2 = φ(x), (w j ⊗ w j )φ(x) and hence

∀x ∈ Ω, f (x) -f * = φ(x), A * φ(x) , A * = j∈[q]
w j ⊗ w j .

To conclude the section we prove the problem in Eq. ( 9.3) admits a maximizer whose non-negative operator is of rank at most d|Z| + 1. Proof. Let p 0 be the maximum of Eq. ( 9.2). Since A 0 implies φ(x), Aφ(x) ≥ 0 for all x ∈ Ω, the problem in Eq. ( 9.2) is a relaxation of Eq. ( 9.3), where the constraint f (x)-c = φ(x), Aφ(x) is substituted by f (x) -c ≥ 0, ∀x ∈ Ω. Then p 0 ≥ p * if a maximum p * exists for Eq. ( 9.3). Moreover if there exists A that satisfies the constraints in Eq. ( 9.3) for the value c * = f * , then p 0 = p * and (c * , A) is a maximizer of Eq. ( 9.3). The proof is concluded by applying Cor. 9.1 that shows that there exists A satisfying the constraints in Eq. ( 9.3) for the value c = f * .

In Cor. 9.1 and Theorem 9.3 we proved the existence of an infinite-dimensional trace-class positive operator A * that satisfies φ(x), A * φ(x) = f (x) -f * for any x ∈ Ω and maximizing Eq. ( 9.3).

The proof is quite general, requiring some geometric properties on f , the fact that f and its second derivatives belong to H and some algebraic properties of the space H, in particular to be closed to multiplication with a C ∞ function, to integration, and to composition with a C ∞ map. The generality of the proof does not allow to derive an easy characterization of the trace of A * .

Properties of the finite-dimensional problem

In the previous section we proved that there exists a finite rank positive operator A * minimizing Eq. (9.3). In this section we study the effect of the discretization of Eq. ( 9.3) on a given a set of distinct points X = {x 1 , . . . , x n }. First, we derive Theorem 9.4 which is fundamental to prove Theorem 9.5, and is our main technical result (we believe it can have a broader impact beyond the use in this paper as discussed in Sec. 9.11 ). Given a smooth function g on Ω, in Theorem 9.4 we prove that if there exists a matrix B ∈ S + (R n ) such that g(x i ) = Φ i BΦ i for i ∈ [n] (the vectors Φ j ∈ R n are defined before Eq. (9.5)), then the inequality g(x) ≥ -ε holds for any x ∈ Ω for an ε depending on the smoothness of the kernel, the smoothness of g and how well the points in X cover Ω. We denote by h X,Ω the fill distance [START_REF] Wendland | Scattered Data Approximation[END_REF],

h X,Ω = sup x∈Ω min i∈[n]
xx i , (9.11) corresponding to the maximum distance between a point in Ω and the set X. In particular, if the kernel and g are m-times differentiable, Theorem 9.4 proves that g(x) ≥ -ε holds with

ε = O(h m X,Ω
) which is an improvement when m 2 with respect to standard discretization results that guarantee exponents of only 1 or 2. Then in Lemma 9.3 we show that there exists a finite-dimensional positive definite matrix B ∈ S + (R n ) such that Tr(B) ≤ Tr(A * ) and Φ i BΦ i = φ(x i ), A * φ(x i ) for all i ∈ [n]. Finally, in Theorem 9.5, we combine Lemma 9.3 with Theorem 9.4, to show that the problem in Eq. ( 9.5) provides a solution that is only O(h m X,Ω ) distant from the solution of the infinite dimensional problem in Eq. ( 9.3).

To start we recall some basic properties of Φ i and φ(x i ), for i ∈ [n], already sketched in Sec. 9.2 . In particular, the next proposition shows that, by construction, Φ i Φ j = φ(x i ) φ(x j ) for any i, j ∈ [n] and more generally that the map V that maps f ∈ H → R -( φ(x 1 ), f , . . . , φ(x n ), f ) ∈ R n is a partial isometry and that Φ i = V φ(x i ). The map V will be crucial to characterize the properties of the finite dimensional version of the operator A * Lemma 9.2 (Characterizing Φ j in terms of φ). Let k be a kernel satisfying Assumption 9.2(a). There exists a linear operator V : H → R n such that

Φ i = V φ(x i ), ∀i ∈ [n].
Moreover V is a partial isometry: V V * is the identity on R n , P = V * V is a rank n projection operator satisfying P φ(

x i ) = φ(x i ), ∀i ∈ [n].
The proof of Lemma 9.2 is given in Sec. 9.C .1 in page 427 and is based on the fact that the kernel matrix K is positive definite and invertible when k is universal [START_REF] Steinwart | Support Vector Machines[END_REF], property that is implied by Assumption 9.2(a), and that R is an invertible matrix that satisfies K = R R.

.1 Uniform inequality from scattered constraints

In this section we derive Theorem 9.4. Here we want to guarantee that a function g satisfies g(x) ≥ -ε on Ω, by imposing some constraints on g(x i ) for i ∈ [n]. If we use the most natural discretization, that consists in the constraints g(x i ) ≥ 0, by Lipschitzianity of g we can guarantee only ε = |g| Ω,1 h X,Ω (recall the definition of | • | Ω,m for m ∈ N from Eq. ( 9.6)). In the case of equality constraints, instead, standard results for functions with scattered zeros [START_REF] Wendland | Scattered Data Approximation[END_REF] (recalled in Sec. 9.B ) guarantee for all

x ∈ Ω |u(x)| ≤ ε, ε = Ch m X,Ω |u| Ω,m
, when u is m-times differentiable and satisfies u(x i ) = 0 for any i ∈ [n] (see the work by [START_REF] Wendland | Scattered Data Approximation[END_REF]; [START_REF] Narcowich | Refined error estimates for radial basis function interpolation[END_REF] or Theorem 9.13 for more details). Thus, in this case the discretization leverages the degree of smoothness of u, requiring much less points to achieve a given ε than in the inequality case.

The goal here is to derive a guarantee for inequality constraints that is as strong as the one for the equality constraints. In particular, given a function g defined on Ω and that satisfies g(x i ) -Φ i BΦ i = 0 on X, with B 0, we first derive a function u defined on the whole Ω and matching g(x i ) -Φ i BΦ i on X. This is possible since we know that Φ i = V φ(x i ), by Lemma 9.2, then u(x) = g(x) -φ(x), V * BV φ(x) satisfies u(x i ) = g(x i ) -Φ i BΦ i for any i ∈ [n]. Finally, we apply the results for functions with scattered zeros on u. The desired result is obtained by noting that, since φ(x), V * BV φ(x) ≥ 0 for any x ∈ Ω, by construction, then for all x ∈ Ω -g(x) ≤ -g(x)

+ φ(x), V * BV φ(x) ≤ |g(x) -φ(x), V * BV φ(x) | = |u(x)| ≤ ε, i.e., g(x) ≥ -ε for all x ∈ Ω with ε = Ch m X,Ω
|u| Ω,m . In the following theorem we provide a slightly more general result, that allows for |g(x i ) -Φ i BΦ i | ≤ τ with τ ≥ 0. Theorem 9.4 (Uniform inequality from scattered constraints). Let Ω satisfy Assumption 9.1(a) for some r > 0. Let k be a kernel satisfying Assumptions 9.2(a) and 9.2(d) (9.12) where the Φ i 's are defined in Sec. 9.2 . The following statement holds: . The constants m, M, D m , defined in Assumptions 9.2(a) and 9.2(d), do not depend on n, X, h X,Ω , B or g. CHAPTER 9. FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS Proof. Let the partial isometry V : H → R n and the projection operator P = V * V be defined as in Lemma 9.2. Given B ∈ S + (R n ) satisfying Eq. ( 9.12), define the operator A ∈ S + (H) as A = V * BV and the functions u, r A : Ω → R as follows r A (x) = φ(x), Aφ(x) , u(x) = g(x) -r A (x), ∀x ∈ Ω.

for some m ∈ N + . Let X = {x 1 , . . . , x n } ⊂ Ω with n ∈ N + such that h X,Ω ≤ r min(1, 1 18(m-1) 2 ). Let g ∈ C m (Ω) and assume there exists B ∈ S + (R n ) and τ ≥ 0 such that |g(x i ) -Φ i BΦ i | ≤ τ, ∀i ∈ [n],
g(x) ≥ -(ε + 2τ ) ∀x ∈ Ω, where ε = Ch m X,Ω , (9
Since Φ i = V φ(x i ) for all i ∈ [n], then for all i ∈ [n]:

r A (x i ) = φ(x i ), V * BV φ(x i ) = (V φ(x i )) B(V φ(x i )) = Φ i BΦ i , and hence u(x i ) = g(x i ) -Φ i BΦ i . Thus, |u(x i )| ≤ τ for any i ∈ [n]
. This allows to apply one of the classical results on functions with scattered zeros [START_REF] Narcowich | Refined error estimates for radial basis function interpolation[END_REF][START_REF] Wendland | Scattered Data Approximation[END_REF] to bound sup x∈Ω |u(x)|, which we derived again in Theorem 9.13 to obtain explicit constants. Since we have assumed h X,Ω ≤ r/ max(1, 18(m -1) 2 ), applying Theorem 9.13, the following holds sup

x∈Ω |u(x)| ≤ 2τ + ε, ε = c R m (u) h m X,Ω
,

where c = 3 max(1, 18(m -1) 2 ) m and R m (v) = |α|=m 1 α! sup x∈Ω |∂ α v(x)
| for any v ∈ C m (Ω) using the multi-index notation (recalled in Sec. 9.3 .1). Since r A (x) = φ(x), Aφ(x) ≥ 0 for any x ∈ Ω as A ∈ S + (H), it holds :

g(x) ≥ g(x) -r A (x) = u(x) ≥ -|u(x)| ≥ -(2τ + ε), ∀x ∈ Ω. ( 9
R m (u) = |α|=m 1 α! sup x∈Ω ∂ α u(x) ≤ |α|=m 1 α! |u| Ω,m = d m m! |u| Ω,m . Since |u| Ω,m ≤ |g| Ω,m + |r A | Ω,m
, combining all the previous bounds, it holds

ε ≤ C 0 (|g| Ω,m + D m M Tr(B)) h m X,Ω , C 0 = 3 d m max(1, 18(m -1) 2 ) m m! .
The proof is concluded by bounding ε in Eq. ( 9.14) with the inequality above.

In the theorem above we used a domain satisfying Assumption 9.1(a) and a version of a bound for functions with scattered zeros (that we derived in Theorem 9.13 following the analysis in the work by [START_REF] Wendland | Scattered Data Approximation[END_REF]), to have explicit and relatively small constants. However, by using different bounds for functions with scattered zeros, we can obtain the same result as Theorem 9.4, but with different assumptions on Ω (and different constants). For example, we can use Corollary 6.4 in the work by [START_REF] Narcowich | Refined error estimates for radial basis function interpolation[END_REF] to obtain a result that holds for Ω = [-1, 1] d or Theorem 11.32 with p = q = ∞, m = 0 in the work by [START_REF] Wendland | Scattered Data Approximation[END_REF] to obtain a result that holds for Ω with locally Lipschitz-continuous boundary. Proof. We divide the proof in few steps.

Step 0. Problem Eq. ( 9.5) admits always a solution.

(a) On the one hand, c cannot be larger than c 0 = min i∈[n] f (x i ), otherwise there would be a point x j for which f (x j ) -c < 0 and so the constraint Φ j BΦ j = f (x j ) -c would be violated, since does not exist any positive semi-definite matrix for which Φ j BΦ j < 0.

(b) On the other, there exists an admissible point. Indeed let (c * , A * ) be the solution of Eq. ( 9.3) such that A * has minimum trace norm. By Theorem 9.3, we know that this solution exists with c * = f * , under Assumptions 9.1 to 9.3. Then, by Lemma 9.3 applied to g(x) = f (x) -c * and A = A * , given X = {x 1 , . . . , x n } we know that there exists B ∈ S + (R n ) satisfying Tr(B) ≤ Tr(A * ) such that the constraints of Eq. ( 9.5) are satisfied for c = c * . Then (c * , B) is admissible for the problem in Eq. ( 9.5).

Thus, since there exists an admissible point for the constraints of Eq. ( 9.5) and its functional cannot be larger than c 0 without violating one constraint, the SDP problem in Eq. ( 9.5) admits a solution (see the work by [START_REF] Boyd | Convex optimization[END_REF]).

Step 1. Consequences of existence of A * . Let (ĉ, B) be one minimizer of Eq. ( 9 (9.17)

Step

2. f | Ω ∈ C m+2 (Ω). Assumption 9.3 guarantees that f | Ω ∈ C 2 (Ω) and that for all i, j ∈ [d], ∂ ∂x i ∂x j f | Ω ∈ H. Since under Assumption 9.2(d), H ⊂ C m (Ω) by Remark 26, we see that ∂ ∂x i ∂x j f | Ω ∈ C m (Ω) for all i, j ∈ [d] and hence f | Ω ∈ C m+2 (Ω).
Step 3. L ∞ bound due to the scattered zeros. Let (ĉ, B) be one minimizer of Eq. ( 9.5) and define ĝ(x) = f (x) -ĉ for all x ∈ Ω. Note that ĝ(

x i ) = Φ i BΦ i for i ∈ [n]. Moreover, ĝ ∈ C m (Ω) because f ∈ C m (Ω) and ĉ is a constant. Considering that h X,Ω ≤ r max(1,18(m-1) 2 )
, by assumption, then all the conditions in Theorem 9.4 are satisfied for g = ĝ, τ = 0 and B = B. Applying Theorem 9.4, we obtain,

∀x ∈ Ω, f (x) -ĉ = ĝ(x) ≥ -η(|ĝ| Ω,m + MD m Tr( B)), η = C 0 h m X,Ω
, where C 0 is defined in Theorem 9.4. Since the inequality above holds for any x ∈ Ω, by evaluating it in one global minimizer ζ ∈ Ω, we have f (ζ) = f * and hence A direct consequence of the theorem above, already stated in Remark 25, is the nearly-optimality of algorithm 6 for the cases of Sobolev functions. Indeed by applying Theorem 9.6 with m equal to the largest integer strictly smaller than sd/2 we have that m ≥ sd/2 -1, and so algorithm 6 achieves the global minimum with a rate that is O(n

-∆ = f * -ĉ = f (ζ) -ĉ = ĝ(ζ) ≥ -η(|ĝ| Ω,m + MD m Tr( B)). Since ĝ = f -ĉ1 Ω ,
-s d + 1 2 + 1 d ).
The lower bounds from information based complexity state that, by observing the functions in n points, it is not possible to find the minimum with error smaller than n -s d + 1 2 for functions in W s 2 (Ω) (see, e.g., the work by Novak (2006), Prop. 1.3.11, page 36). Since in Theorem 9.6 we assume f belongs to W s+2 2 (Ω), the optimal rate would be n -s d + 1 2 -2 d so we are a factor n 3/d slower than the optimal rate. Note that this factor is negligible if the function is very smooth, i.e., s d, or d is very large. An interesting corollary that corresponds to Theorem 9.1, can be derived considering that C s+2 (Ω) ⊆ W s+2 2 (Ω), since Ω is bounded.

Algorithm

We need to solve the following optimization problem:

max B 0,c∈R c -λ Tr(B) such that f (x i ) -c -Φ i BΦ i = 0, ∀i ∈ [n].
This is a semi-definite programming problem with n constraints and a semi-definite constraint of size n. It can thus be solved with precistion ε in time O(n 3.5 log(1/ε)) and memory O(n 2 ) by standard software packages [START_REF] Boyd | Convex optimization[END_REF]. However, to allow applications to n = 1000 or more, and on parallel architectures, we provide a simple Newton algorithm, which relies on penalization by a self-concordant barrier, that is, we aim to solve max

B 0,c∈R c -λ Tr(B) + ε n log det(B) such that f (x i ) -c -Φ i BΦ i = 0, ∀i ∈ [n],
for which we know that at optimum, the deviation with the optimal value is at most ε (Nemirovski, 2004, Sec. 4.4). By standard Lagrangian duality, we get, with Φ ∈ R n×n the matrix with rows Φ 1 , . . . , Φ n , so that ΦΦ = K:

sup B 0,c inf α∈R n c + n i=1 α i f (x i ) -c -Φ i BΦ i -λ Tr(B) + ε n log det(B) = inf α∈R n n i=1 α i f (x i ) - ε n log det Φ Diag(α)Φ + λI + ε n log ε n -ε s. t. α 1 n = 1.
With the barrier term, this thus defines a dual function H(α), and we get the following gradient

H (α) i = f i - ε n Φ i Φ Diag(α)Φ + λI -1 Φ i = f i - ε nα i K(K + λDiag(α) -1 ) -1 ii ,
9.6 . ALGORITHM

and Hessian

H (α) ij = ε n Φ i Φ Diag(α)Φ + λI -1 Φ j 2 ,
which can be rewritten

H (α) ij = ε nα j α i K(K + λDiag(α) -1 ) -1 ij K(K + λDiag(α) -1 ) -1 ji .
We can then compute the step for the Damped Newton algorithm:

α + = α - 1 1+ √ n ε λ(α) ∆, where ∆ = H (α) -1 H (α) -1 n H (α) -1 H (α)
1 n H (α) -1 1n H (α) -1 1 n and λ(α) 2 = ∆ H (α)∆ is the Newton decrement (which can serve as a stopping criterion). Note that the algorithm is always feasible, without a need for any eigenvalue decomposition. The overall complexity is O(n 3 ) per iteration due to matrix inversions and linear systems. Note that the conditioning of these linear systems is at least as bad as the conditioning of the kernel matrix K. Fortunately, for the s-th Sobolev kernels in dimension d, the m-th eigenvalue of the kernel matrix typically decay as m -2s/d (Bach, 2017a, Sec. 2.3).

Retrieving c and B. From an optimal α, we can recover

B = ε n Φ Diag(α)Φ + λI -1 = ε nλ I -Φ (ΦΦ + λDiag(α) -1 ) -1 Φ and c = 1 n H (α) 1 n (since c
is the Lagrange multiplier for the constraint α 1 n = 1). Thus, computing the model for a test point, can be done as ε nλ k(x, x) -q(x) (K + λDiag(α) -1 ) -1 q(x) , where q(x) i = k(x, x i ). Alternatively, when Φ is invertible, we can use q(x) Φ -BΦ -1 q(x).

Retrieving a minimizer. Given the dual solution, based on our localizing arguments presented in Sec. 9.7 , a good candidate solution will be ẑ

= n i=1 α i x i (9.19)
A more principled way to find a minimizer is provided in Sec. 9.7 , of which the equation above corresponds to the limit solution of Eq. ( 9.23) for ν → 0 (see Sec. 9.7 .1).

Number of iterations.

In order to reach a Newton decrement n 1/2 ε -1/2 λ(α) κ, a number of steps equal to a universal constant times n ε [H(α 0 ) -H(α * )] + log log 1 κ is sufficient [START_REF] Nemirovski | Interior point polynomial time methods in convex programming[END_REF].

When initializing with α

0 = 1 n 1 n , we have H(α 0 ) = 1 n n i=1 f i -ε n log det K + nλI + ε n log ε -ε, and H(α * ) c * -λ Tr(A * ) -ε.
This leads to a number of Newton steps less than

n ε f -inf f + log det K + nλI + n ε λ Tr(A * ) + log ε + log log 1 κ .
In our experiments, we do not perform path following (that would lead the classical interior-point method) and instead fixed value ε = 10 -3 , and a few hundred Newton steps.

Behavior for λ = 0. If the kernel matrix K is invertible (which is the case almost surely for Sobolev kernels and points sampled independently from a distribution with a density with respect to the Lebesgue measure), then we show that for λ = 0, the optimal value of of the finite-dimensional problem in Eq. ( 9.5) is equal to min

i∈[n] f (x i ). Since f (x i ) ≥ c + Φ i BΦ i implies f (x i ) ≥ c
, the optimal value has to be less than min i∈[n] f (x i ). We therefore just need to find a feasible B that achieves it. Since K is assumed invertible (and thus its Cholesky factor as well), we can simply take

B = R -Diag[(f (x j ) -min i∈[n] f (x i )) j ]R -1 .

402

CHAPTER 9. FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS

Finding the global minimizer

In this section we provide and study the problem in Eq. ( 9.23), that is a variation of the problem in Eq. (9.5), and allows to find also the minimizer of f as we prove in Theorem 9.8. As in Sec. 9.2 we start from a convex representation of the optimization problem and then we derive our sampled version, passing by an intermediate infinite-dimensional problem that is useful to derive the theoretical properties of the method. While the problem in Eq. ( 9.2) can be seen as finding the largest constant c such that fc is still non-negative, in the problem below we find the parabola of the form p z,γ (x) = ν 2 x 2 -νx z + c = ν 2 xz 2 + cν 2 z 2 with the highest vertex such that fp z,c is still non-negative. Since the height of the vertex of p z,c corresponds to cν 2 z 2 , the resulting optimization problem is the following,

max c∈R,z∈R d c -ν 2 z 2 such that f (x) -ν 2 x 2 + νx z -c ≥ 0 ∀x ∈ Ω. (9.20) It is easy to see that if f ∈ C 2 (R d )
∈ Ω, f (x) -f * ≥ β 2 x -ζ 2 . (9.21)
The remark above is derived in Sec. 9.F .1, page 438. In what follows, whenever f satisfies Assumptions 9.1(b) and 9.4, then β will be assumed to be the supremum among the value satisfying Eq. ( 9.21). Now we are ready to summarize the reasoning above on the fact that Eq. ( 9.20) achieves the minimizer of f .

* = f * + ν 2 ζ 2 .
The lemma above guarantees that the problem in Eq. ( 9.20) achieves the global minimum and the global minimizer of f , when f satisfies the geometric conditions Assumptions 9.1 and 9.4. Now, as we did for Eq. ( 9.2), we consider the following problem of which Eq. ( 9.20) is a tight relaxation. .22) Indeed, since φ(x), Aφ(x) ≥ 0 for any x ∈ Ω and A ∈ S + (H), for any triplet (c, z, A) satisfying the constraints in the problem above, the couple (c, z) satisfies the constraints in Eq. ( 9.20). The contrary may be not true in general. In the next theorem we prove that when H satisfies Assumption 9.2 and Ω, f satisfy Assumptions 9.1, 9.3 and 9.4, then the relaxation is tight The proof of the theorem above is essentially the same of Theorem 9.3 and is reported for completeness in Sec. 9.F .2, page 438. In particular, to prove the existence of A * we applied Cor. 9.1 to the function f (x) -ν 2 xζ 2 that still satisfies Assumptions 9.1 and 9.3 when f does and ν < β. Now we are ready to consider the finite-dimensional version of Eq. (9.22). Given a set of points

max c∈R,z∈R d ,A∈S + (H) c -ν 2 z 2 such that f (x) -ν 2 x 2 + νx z -c = φ(x), Aφ(x) ∀x ∈ Ω. ( 9
X = {x 1 , . . . , x n } with n ∈ N + , max c∈R,z∈R d ,B∈S + (R n ) c -ν 2 z 2 -λ Tr(B) such that ∀i ∈ [n], f (x i ) -ν 2 x i 2 + νx i z -c = Φ i BΦ i . (9.23)
For the problem above we can derive similar convergence guarantees as for Eq. ( 9.5) and also a convergence of the estimated minimizer z to ζ, as reported in the following theorem. The proof of the theorem above is similar to the one of Theorem 9.5 and it is stated for completeness in Sec. 9.F .3, page 440. The same comments to Theorem 9.5 that we reported in the related section and the rates for Sobolev functions, apply also in this case. In the next section we describe the algorithm to solve the problem in Eq. ( 9.23).

.1 Algorithm

We can use the same dual technique as presented in Sec. 9.6 , and obtain a dual problem to Eq. ( 9.23) with the additional penalty ε n log det B. The dual problem can readily be obtained as (up to constants) inf

α∈R n n i=1 α i f (x i ) - ε n log det Φ Diag(α)Φ + λI + ν 2 - n i=1 α i x i 2 2 + n i=1 α i x i 2 2
, such that α 1 n = 1, with the optimal z that can be recovered as z = n i=1 α i x i . We note that when ν tends to zero, we recover the dual problem from Sec. 9.6 , and we keep the candidate above in Ω even when ν = 0.

.2 Warm restart scheme for linear rates

It is worth noting that Theorem 9.8 provides strong guarantees on the distance ẑζ where ẑ is the solution of the problem Eq. ( 9.23) and ζ the global optimum of f . This suggests that we can implement a warm restart scheme that leverage the additional knowledge of the position of ζ. Assume indeed that Ω is a ball of radius R centered in z 0 . For t = 1, . . . , T with T = log 1 ε , we apply Eq. ( 9.23) to a set X t that contains enough points sampled uniformly at random in the ball B r t-1 (z t-1 ) such that Theorem 9.8 guarantees that z t -ζ ≤ r t-1 /e where z t is the solution of Eq. ( 9.23). The cycle is repeated with r t = r t-1 /e and the new center be z t . By plugging the estimate of Lemma 9.4 for h Xt,Br t-1 (z t-1 ) in Theorem 9.8 for each step t, we obtain a total number of points n to achieve z T -ζ ≤ ε with probability 1 -T δ, that is

n = O C d/m d,m F ν d/m

Extensions

In this section we deal with two aspects: (a) the effect of solving approximately the problem in Eq. (9.5), and (b) how can we certify explicitly (no dependence on quantities of theoretical interest as Tr(A * )) how close is a given (approximate) solution to the optimum; (c) we will also cover the case when the function f does not have a positive definite representer A * in S + (H) but in a larger space. This allows to cover the cases of f ∈ C s (R d ) with s ≤ d/2 + 2.

.1 Approximate solutions

In this section we extend Theorem 9.5 to consider the case when we solve Eq. (9.5) in an approximate way. In particular, let λ > 0, n ∈ N + and X = {x 1 , . . . , x n }. Denote by p λ,n the optimal value achieved by Eq. (9.5) for such λ, n. We say that (c, B) is an approximate solution of Eq. (9.5) with parameters θ 1 , θ 2 , τ 1 , τ 2 ≥ 0 if it satisfies the following inequalities The proof of the theorem above is reported for completeness in Sec. 9.G .1, page 441, and is a variation of the one of Theorem 9.5 where we used Theorem 9.4 with τ = τ 1 + τ 2 Tr( B) and we further bound p λ,n via Eq. ( 9.27). From a practical side, the theorem above allows to use a wide 9.8 . EXTENSIONS 405 range of methods and techniques to approximate the solution of Eq. (9.5). In particular, it is possible to use lower dimensional approximations of Φ 1 , . . . , Φ n and algorithms based on early stopping as described in Sec. 9.11 , since τ 1 , τ 2 , θ 1 , θ 2 will take into account the error incurred in the approximations. An interesting application of the theorem above, from a theoretical side is that it allows also to deal with situations where f does not have a representer A * in S + (H) as we are going to discuss in the next section.

p λ,n -c + λ Tr( B) ≤ θ 1 + θ 2 Tr( B), (9.27) |f (x i ) -c -Φ i BΦ i | ≤ τ 1 + τ 2 Tr( B), ∀i ∈ [n]. ( 9

.2 Rates for f with low smoothness

When f ∈ C s+2 (R d ) with s ∈ N, but with a low smoothness, i.e., s ≤ d/2, we can still apply our method to find the global minimum and obtain almost optimal convergence rates, as soon as it satisfies the geometric conditions in Assumption 9.1(b), as we are going to show in Theorem 9.11 and the following discussion.

In this section, for any function u defined on a super-set of Ω and s times differentiable on Ω, we define the following semi norm : .31) We consider the following variation of the problem in Eq. ( 9.5):

u Ω,s = max |α|≤s sup x∈Ω ∂ α u(x) . ( 9
max c∈R, B∈S + (R n ) c -λ Tr(B) such that ∀i ∈ [n], |f (x i ) -c -Φ i BΦ i | ≤ τ. (9.32)
The idea is that f , under the geometric conditions in Assumption 9.1(b), still admits a decomposition in the form f (x) = j∈[p] w j (x) 2 , p ∈ N + for any x ∈ Ω, but now with respect to functions with low smoothness w 1 , . . . , w p ∈ C s (R d ). To prove this we follow the same proof of Sec. 9.4 noting that the assumptions to apply Lemma 9.1 and Theorem 9.2 are that f belongs to a normed vector space space that satisfy the algebraic properties in Assumptions 9.2(a) to 9.2(c) which does not have necessarily to be a RKHS. In particular, note that the space

H = {f | Ω : f ∈ C s (R d )} of restriction to Ω of functions in C s (R d ),
∈ C s (R d ), p ∈ N + , such that ∀x ∈ Ω, f (x) -f * = j∈[p]
w 2 j (x).

By using the decomposition above, when the kernel satisfies Assumption 9.2(a), we build an operator A ε ∈ S + (H) that approximates f with error O( s ) for any > 0. First note that, for any bounded open set Ω ⊂ R d and any s ≤ r, there exists C 1 and C 2 depending only on r, s, Ω such that for any g ∈ C s (R d ) and ε > 0 there exists a smooth approximation g

∈ C ∞ (R d ) such that sup x∈Ω |g(x) -g ε (x)| ≤ C 1 ε s g Ω,s
and such that g ε Ω,r ≤ C 2 ε -(r-s) g Ω,s (see Thm. 5.33 by [START_REF] Adams | Sobolev Spaces[END_REF] for the more general case of Sobolev spaces, or (Cheney and Light, 2009, Chapter 21)

ε = j∈[p] w ε j | Ω ⊗ w ε j | Ω ∈ S + (H).
The reasoning above is formalized in the next theorem (the proof is in Sec. 9.G .3, page 442).

(x) = j∈[p] w 2 j (x), x ∈ Ω with w j ∈ C s (R d ) for j ∈ [p]. Let k r be the Sobolev kernel of smoothness r > max(s, d
2 ) and let H be the associated RKHS. Then, for any ε ∈ (0, 1] there exist A ε ∈ S + (H) such that 9.33) where C = pqw 2 , C = pq w 2 , and w = max j∈[p] w j Ω,s and q, q are constants that depend only on s, r, d, Ω and are defined in the proof.

Tr(A ε ) ≤ Cε -2(r-s) , sup x∈Ω |f (x) -f * -φ(x), A ε φ(x) | ≤ C ε s , (
Denote now by (c, B) one minimizer of Eq. ( 9.32), and consider the problem in Eq. ( 9.5) with respect to

f ε (x) = φ(x), A ε φ(x) + f * , i.e., max c∈R, B∈S + (R n ) c -λ Tr(B) such that ∀i ∈ [n], f ε (x i ) -c = Φ i BΦ i , (9.34)
and denote by p ε λ,n its optimum. Since

f ε (x i ) -c = Φ i BΦ i implies |f (x i ) -c -Φ i BΦ i | ≤ τ when τ ≥ sup x∈Ω |f (x) -f ε (x)|,
then in this case Eq. ( 9.32) is a relaxation of Eq. ( 9.34) and we have that p ε λ,n -cλ Tr( B) ≤ 0. Then, to obtain guarantees on (c, B) (the solution of Eq. ( 9.32)) we can apply Theorem 9.9 to the problem in Eq. ( 9.34) with θ 1 , θ 2 , τ 2 = 0 and τ 1 = τ with the requirement τ ≥ sup x∈Ω |f (x) -f ε (x)|. The reasoning above is formalized in the following theorem and the complete proof is reported in Sec. 9.G .4, page 443. With the same notation and the same conditions on λ of Theorem 9.5, when τ = λ s/(2r-s)

|c -f * | ≤ C 1,f (λ + λ s 2r-s ), Tr( B) ≤ C 2,f (1 + λ -(1-s 2r-s ) ).
with C 1,f , C 2,f defined in the proof and depending only on f and r, s, d, Ω.

The result above allows to derive the following estimate on algorithm 6 applied on the problem in Eq. ( 9.32) in the case of a function f with low smoothness. Consider the application algorithm 6 to the problem in Eq. ( 9.32) to a function f ∈ C s+2 (Ω) satisfying Assumption 9.1(b), with a Sobolev kernel k r , r ≥ s, r > d/2, and with τ = λ s/(2r-s) , λ = O(n -r d +1/2 ) on a set of n points sampled independently and uniformly at random from Ω = B 1 (0), the unit ball of R d . By combining the result of Theorem 9.11 with the condition on λ in Theorem 9.5 and with the upper bound on the fill distance in the case of points sampled uniformly at random in Lemma 9.4, we have that

|c -f * | = O n -s 2d (1-d-s 2r-s ) ,
modulo logarithmic factors, where c is the solution of Eq. ( 9.32). The rate above must be compared with the optimal rates for global minimization of functions in C s+2 (Ω) via function 9.9 . RELATIONSHIP WITH POLYNOMIAL HIERARCHIES 407 evaluations, that is n -s+2 d for any s ∈ N (Prop. 1.3.9, pag. 34 by [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF]). In the low smoothness setting, i.e., s ≤ d/2 when we choose r d/2, then the term 1 -d-s 2r-s → 1 and so the exponent of the rate above differs from the optimal one by a multiplicative factor 1/2 + 1 s , leading essentially to a rate of O(n -s/(2d) ). However, the choice of a large r will impact the hidden constants that are not tracked in the analysis above. Then for a fixed n there is a trade-off in r between the constants and the exponent of the rate. So in practice it would be useful to select r by parameter tuning.

.3 Certificate of optimality

While in Theorem 9.5 we provide a bound on the convergence of Eq. ( 9.5) a priori, i.e., only depending on properties of f, Ω, H, in this section we provide a bound a posteriori, that is a certificate of optimality. Indeed, the next theorem quantifies f (z) -f * for a candidate minimizer z, in terms of only (ĉ, B), an (approximate) solution of Eq. ( 9.5) and |f | Ω,m . A candidate minimizer based on Eq. ( 9.5) is provided in Eq. (9.19). In section Sec. 9.7 we study a different algorithm Eq. ( 9.23) that explicitly provides a minimizer and whose certificate is studied in Sec. 9.G .5. Theorem 9.12 (Certificate of optimality a minimizer from Eq. (9.5)). Let Ω satisfy Assumption 9.1(a) for some r > 0. Let k be a kernel satisfying Assumptions 9.2(a) and 9.2(d) for some m

∈ N + . Let X = {x 1 , . . . , x n } ⊂ Ω with n ∈ N + such that h X,Ω ≤ r 18(m-1) 2 . Let f ∈ C m (Ω) and let ĉ ∈ R, B ∈ S + (R n ) and τ ≥ 0 satisfying |f (x i ) -ĉ -Φ i BΦ i | ≤ τ, i ∈ [n], (9.35)
where the Φ i 's are defined in Sec. 9.2 . Let f * = min x∈Ω f (x). Then the following holds Proof. By applying Theorem 9.4 with g(x) = f (x) -ĉ, we have f (x) -ĉ ≥ -ε -2τ for any x ∈ Ω. In particular this implies that f (ζ) -ĉ ≥ -ε -τ . The proof is concluded by noting that f (z) ≥ f * by definition of f * .

|f (z) -f * | ≤ f (z) -ĉ + ε + 2τ, ∀z ∈ Ω, where ε = Ch m X,Ω , (9

Relationship with polynomial hierarchies

The formulation as an infinite-dimensional sum-of-squares bears some strong similarities with polynomial hierarchies. There are several such hierarchies allowing to solve any polynomial optimization problem [START_REF] Lasserre | Global optimization with polynomials and the problem of moments[END_REF][START_REF] Lasserre | A sum of squares approximation of nonnegative polynomials[END_REF][START_REF] Lasserre | A new look at nonnegativity on closed sets and polynomial optimization[END_REF], but one has a clear relationship to ours. The goal of the following discussion is to shed light on the benefits in terms of condition number and dimensionality of the problem, deriving by using an infinite dimensional feature map in the finite dimensional problem, instead of an explicit finite-dimensional polynomial map as in the case considered by the papers cited above.

Adding small perturbations. We start this discussion from the following result from Lasserre [START_REF] Lasserre | A sum of squares approximation of nonnegative polynomials[END_REF], that is, for any multivariate non-negative polynomial f on R d , and for any η > 0, there exists a degree r(f, η) such that the function

f η (x) = f (x) + η r(f,ε) k=0 1 k! d j=1
x 2k j is a sum of squares, and such that the 1 -norm between the coefficients of f and f η tends to zero (here this 1 -norm is equal to ηd r(f,ε) k=0

1 k! ηde).
This implies that for the kernel k r (x, y) = r k=0 (x y) k k! , with feature map φ r (x) composed of all weighted monomials of degree less than r, the function

f (x) + η φ r (x) 2 2 = f (x) + ηk r (x, x
) is a sum of squares, for any r r(f, η), with η arbitrarily close to zero (this can be obtained by adding the required squares to go from d j=1 x 2k j to x 2k = ( d j=1 x 2 j ) k ). This result implies that minimizing f arbitrarily precisely over any compact set K (such that sup x∈K k r (x, x) is finite), can be done by minimizing f (x) + ηk(x, x), with sum-of-squares polynomials of sufficiently large degree. We already showed that in this paper that if f satisfies the geometric condition in Assumption 9.1(b), our framework is able to find the global minimum by the finite dimensional problem in Eq. ( 9.5), which, in turn, is based on a kernel associated to an infinite dimensional space (as the Sobolev kernel, see Example 9.1). We now show how our framework can provide approximation guarantees and potentially efficient algorithms for the problem above even when Assumption 9.1(b) may not hold and we use a polynomial kernel of degree r (with r that may not be large enough). However, in this case the resulting problem would suffer of a possibly infinite condition number and a larger dimensionality than the one achievable with an infinite dimensional feature map.

Modified optimization problem. Given the representation of x → f (x) -f * + η φ r (x) 2 2 as a sum-of-squares, we can explicitly model the function as f (x) -c + η φ r (x) 2 2 = φ r (x), Aφ r (x) with A positive definite and η ≥ 0. Note that if r is greater than twice the degree of f this problem is always feasible by taking η sufficiently large. Moreover, for feasible (c, η, A), we have for any

x ∈ Ω, f (x) ≥ c -η φ r (x) 2 ≥ c -η sup y∈Ω φ r (y) 2 2 .
Thus, a relaxation of the optimization problem is sup

c∈R,A 0,η 0 c -η sup y∈Ω φ r (y) 2 2 s. t. ∀x ∈ Ω, f (x) = c + φ r (x) Aφ r (x) -η φ(x) 2 2 .
Moreover, if we choose r larger than r(ff * , η), we know that there exists a feasible A which is positive semi-definite, with c = f * -η sup y∈Ω φ r (y) 2 2 , and thus the objective value is greater than f * -η sup y∈Ω φ r (y) 2 2 . Thus, the objective value of the problem above converges to f * , when η go to zero (and thus r(ff * , η) goes to infinity), while always providing a lower bound. Note that if ff * is a sum of squares, then the optimal value η can be taken to be zero, and we recover the initial problem.

Subsampling and regularization. At this point, since r is finite, subsampling d 2r points leads to an equivalent finite-dimensional problem. We can also add some regularization to sub-sample the problem and avoiding such a large number of points. Note here that the kernel matrix will probably be ill-conditioned, and the problem computationally harder to solve and difficult ro regularize.

Infinite-degree polynomials. In the approach outlined above, we need to let r increase to converge to the optimal value. We can directly take r = ∞, since k r (x, y) = r k=0 (x y) k k! tends to the kernel exp(x y), and here use subsampling. Again, it may lead to numerical difficuties. However, we can use Sobolev kernels (with guarantees on performance and controlled conditioning of kernel matrices), on the function f (x) + ηe x 2 2 for which we now there exists a sum of squares representation as soon as f is a polynomial.

Experiments

In this section, we illustrate our results with experiments on synthetic data.

Finding hyperparameters. Given a function to minimize and a chosen kernel, there are three types of hyperparameters: (a) the number n of sample points, (b) the regularization parameter λ, and (c) the kernel parameters. Since n drives the running time complexity of the method, we will always set it manually, while we will estimate the other parameters (regularization and kernel), by "cross-validation" (i.e., selecting the parameters of the algorithm that lead to the minimum value of f at the candidate optimum, among a logarithmic range of parameters). This adds a few function evaluations, but allows to choose good parameters.

Functions to minimize. We consider first a simple functions defined in R 2 with their global minimimizer on [-1, 1] d , which is minus the sum of Gaussian bumps (see Fig. 9.1). To go to higher even dimensions with the possibility of computing the global minimum with high precision by grid search, we consider functions of the form f

(x) = f (x 1 , x 2 ) + f (x 3 , x 4 ) + • • • + f (x d-1 , x d ).
We also consider adding a high-frequency cosine on the coordinate directions representing a more general scenario for a non-convex function. Note that in this second setting the gradient based methods cannot work properly (while ours can) as we are going to see in the simulations.

All results are reported by normalizing function values so that the range of values is 1, that is, max

x∈[-1,1] d f (x) = 1 and min x∈[-1,1] d f (x) = 0.
Baseline algorithms. We compare our algorithm with the exponential kernel and points sampled from a quasi-random sequence in [-1, 1] d , such as the Halton sequence [START_REF] Niederreiter | Random Number Generation and Quasi-Monte Carlo Methods[END_REF], to:

• Random search: select a quasi-random sequence in [-1, 1] d and take the point with minimal function value.

• Random search with gradient descent: starting gradient descent for a certain number of iterations from quasi-random points, with a number of initialization divided by d + 1 and the number of gradient steps, to account for gradient evaluations based on d + 1 function evaluations (by finite-difference). The step-size for gradient descent is taken constant, but its values is optimized for smallest final value while providing a descent algorithm.

Illustration in two dimensions. We show in Fig. 9.1 a function in two dimensions, with sampled point in purple, the trajectory of the candidate optimum along Newton iterations in red, and the final model of the function. We also compare to gradient descent with random starting points. We consider two functions below, one without extra high-frequency component (top), and one with (bottom). We can make the following observations: 9.37)). Right: the trajectory of gradient descent starting from random points. As it is possible to see, even a small local non-convexity prevents the random+GD algorithms to converge properly, while the proposed method is quite robust to it.

• Our algorithm outperforms random search, that is, it improves on the function values of the sampled points.

• For the smoother function, gradient descent performs quite well, but is not robust when high-frequency components are added.

Note that the proposed algorithm provides also a model of the function reconstructed starting from its evaluation on the sampled points. In particular, if (ĉ, B) is a solution of the algorithm, the approximate function ĝ

≈ f -f * corresponds to ĝ(x) = φ(x), V * BV φ(x) = v(x) R -1 BR -v(x), ∀x ∈ Ω (9.37)
with v(x) = (k(x 1 , x), . . . , k(x n , x)) for x ∈ Ω and where V : H → R n is in Sec. 9.5 .

Higher dimensions. We compare the algorithms on a problem in dimension d = 8, as n increases, in order to assess how we approach the global optimum. We perform 4 replications with different random seeds for the sampling of points in [-1, 1] d . The function to be minimized is built as described at the beginning of this section and is shifted and rescaled to have output in [0, 1] and the minimum in 0. We can see that as n gets large, the performance of the proposed algorithm improves, and that with high frequency components, gradient descent with random restarts has worse performance and seem to show a slower rate overall, even in the case of the function without high-frequency components.

.1 Experiments on benchmarks for global optimization

In this section, we perform experiments using the algorithm described in the section above on the more than 200 global optimization problems in multiple dimensions constituting the well-known benchmark "Global Optimization Benchmarks" [START_REF] Ali | A numerical evaluation of several stochastic algorithms on selected continuous global optimization test problems[END_REF][START_REF] Jamil | A literature survey of benchmark functions for global optimisation problems[END_REF][START_REF] Molga | Test functions for optimization needs[END_REF] The functions to optimize come with their minimizer and their minimum to be used as a ground truth and with a region of interest where to look for the minimizers.

In this section, we present only the results for dimensions 4 and higher, as our method seems particularly interesting for these dimensions. The results for dimensions 2 and 3 can be found in Table 9.4, in the appendix, page 452. In Table 9.1, we report the results obtained by our algorithm. The algorithm we implemented is warm-restart scheme described in Sec. 9.7 .2. The implementation details are reported in Sec. 9.H . The algorithm was performed with N = 200 restart iterations, and was repeated 5 times (we select the best estimator out of the 5 restarts, to take into account of the high-probability factors). In Table 9.1, we report the following : (a) the problem name; (b) its dimension; (c) the number of iterations needed to achieve a threshold of 0.01 relative error; we define the relative error as r(x) = f (x)-f (x * ) f (x 1 )-f (x * ) ; (d) the final absolute error f ( x) -f (x * ); (e) the number m of new function evaluations at each step (without counting those in order to select λ).

Note that the dimension of the optimization problem is n = 3m and that the SDP constraint is also of size n × n. Moreover, the choice we make to evaluate the relative error is in order to avoid very high values of the function f ; comparing to f (x 1 ) somewhat shows the importance of the iterative scheme.

Discussion, interpretation The set of functions on which we have tested our algorithm originally is a challenge allowing a maximum of 2000 function evaluations to reach a precision in absolute error of order 10 -6 . We do not try to compete in this specific challenge, which models the fact that the number of function evaluations in certain real-life problems is very costly. In order to tackle this challenge, we would need to reduce the cost in function evaluations of certain steps such as that of the selection of λ (which we believe can be done without much difficulty).

Note that the fact that we achieve a relative error of 0.01 in almost all cases shows that the iterative scheme is indeed effective. Our method is statistical and therefore does not enjoy the same precision that gloptipoly3 achieves in the case when it converges. However, our method is clearly more scalable in the sense that it returns an approximate solution for polynomials of high degree and dimension, for any chosen dimension of the PSD matrix, and very small matrices allow to achieve already interesting precisionsm, as it is possible to see in Table 9.3.

Discussion

In this section, we discuss our results and propose a series of extensions.

Main technical contribution and extensions. We see that from Eq. ( 9.2), the problem of minimization can be easily written in terms of an infinite set of inequality constraints on u(x) = f (x) -c that must hold for every x ∈ Ω. While it is well known how to approximate efficiently an infinite set of equality constraints via a finite subset (e.g. via bounds on functions with scattered zeros [START_REF] Wendland | Scattered Data Approximation[END_REF] from the field of approximation theory), leading to optimal rates for the approximation problem, the situation is more difficult in the case of an infinite set of inequality constraints. The main technical contribution of this paper, on which the whole result of the paper is based, is Theorem 9.4, that allows to deal with an infinite set of inequality constraints as efficiently as in the equality case as discussed in Sec. 9.5 .1. In particular, we rewrite the infinite set of inequalities g(x) ≥ 0, ∀x ∈ Ω in terms of a very sparse set of constraints of the form g(x i ) = Φ i BΦ i , for some points x 1 , . . . , x n ∈ Ω and a matrix B ∈ S + (R n ), with n in the same order of the one required by the equality case. Assume for simplicity that Ω is contained in the unit ball and the points are uniformly distributed in Ω. From Theorem 9.4 we derive that if B exists, g(x) ≥ -C n -m/d (|g| Ω,m + Tr(B)), modulo logarithmic factors, where m is the order of smoothness of g. This result is particularly useful for two reasons. First, it recovers the same dependence on m, the smoothness of g, and n the number of sample points, as in the case of equality constraints. This is particularly convenient when m d, e.g. with m ≥ d the rate becomes O(n -1 ), that is independent from d in the exponent (the dependence of d is still present in the hidden constants and it is exponential in the worst case). Second, if used in an optimization problem, the matrix B can be found via a convex formulation, by requiring u(x i ) = Φ i BΦ i for i ∈ [n] and penalizing Tr(B) in the functional. This technique allows, for example, to deal with more general optimization problems with infinite constraints than the one considered in this paper, as min θ∈Θ F (θ) such that g(θ, x) ≥ 0, ∀x ∈ Ω, by translating it as follows min θ∈Θ,B 0

F (θ) + λ Tr(B) such that g(θ, x i ) = Φ i BΦ i ∀i ∈ [n].
If F and u are convex in θ and Θ a convex set, then the second is a convex problem that has the potential to approximate very efficiently the first, due to Theorem 9.4. From this viewpoint this paper is an application of this principle to Eq. ( 9.2).

Duality. Beyond using duality in Sec. 9.6 for algorithmic purposes, there is also a dual for the infinite-dimensional problem, which can be written as, inf

p:Ω→R Ω p(x)f (x)dx such that Ω p(x)dx = 1 and Ω p(x)φ(x)⊗φ(x)dx 0.
Replacing the constraint Ω p(x)φ(x) ⊗ φ(x)dx 0 by ∀x ∈ Ω, p(x) 0 leads to the usual relaxation of optimization with probability measures. Thus, like for polynomial optimization [START_REF] Henrion | The Moment-SOS Hierarchy[END_REF], our formulation corresponds also to a relaxation in the dual formulation to signed measures.

Comparison with algorithms based on SOS polynomials. According to recent results on SOS polynomials (see the work by Slot and Laurent (2020a) and references therein) which apply to polynomial relaxations as described in Sec. 9.9 , when f is a polynomial, such algorithms can achieve the global mininum with a rate O(1/r 2 ) via an SDP problem based on the representation of SOS polynomials of degree r in terms of positive definite matrices. Since the dimension of the corresponding matrix is n = d+r r corresponding to n = O(r d ), by expressing the rate with respect to the dimensionality of the matrix, such methods achieve the global minimum with an error that is in the order of O(n -2/d ). This can be compared with the approach proposed in this paper as algorithm 6. By sampling n points from the domain of interest, we cast an SDP problem in terms of a n-dimensional positive definite matrix, achieving a rate that is C s,d n -s/d+1/2 (see Theorem 9.6) modulo logaritmic factors, by using a Sobolev kernel k s+3 with s > d/2 (see Example 9.1). Since the polynomials are arbitrarily differentiable, we can choose s arbitrarily large at the cost of a larger constant C s,d completely characterized in Theorem 9.6. For example, by choosing s = 5d/2 we achieve the global minimum with a rate O(n -2 ) that does not suffer of the curse of dimensionality except in the constants, and that is faster than the one obtained by SOS polynomial methods especially when d 1. It must be noted that our result holds under the sufficient assumption Assumption 9.1(b) that can be relaxed according to Remark 27, but that it is not required by SOS polynomial methods. It would be of interest to know if such methods can achieve our rates under the same assumption.

Comparison with simpler algorithms. Similar reasoning can be done with respect to simple algorithms for global optimization. We consider here the algorithm that consists in sampling n points at random in Ω and taking the one with minimum value. A simple analysis, that we report below shows that this method achieves a rate of O(n -2/d ). So our method is strictly better than taking the minimum f (x i ) for i ∈ [n] when f is at least 3-times differentiable (see Sec. 9.8 .2). Note that even in the case when the function f is infinitely differentiable, the the algorithm that consists in sampling n points at random in Ω and taking the one with minimum value cannot go faster that O(n -2/d ). To see this, consider Ω = [0, 1] d and the points x i to be chosen as a grid of step τ . This means that n = O(τ -d ). Now let f (x) = xy 2 for some y ∈ [0, 1] d . This function is infintely differentiable. Nevertheless, in general the best approximation of y on the grid will be ỹ = τ y/τ + 1/2 (componentwise). Since, for any τ , there exists always an y ∈ [0, 1] d such that y/τ + 1/2 -y/τ = 1/2, we have that in the worst case

f (ỹ) -f (y) = ỹ -y 2 = τ 2 y/τ + 1/2 -y/τ 2 = τ 2 /4.
Now if we express τ w.r.t., n, i.e., τ = n -1/d , we see that we obtain an error that is in the order of n -2/d . So this simple algorithm cannot be better than n -2/d even if the function is infinitely differentiable. A similar argument can be obtained when the points are a generic covering of Ω.

Obtaining optimal rates. Our current analysis, even for functions f in Sobolev spaces, does not lead to the optimal rate of convergence (we obtain an extra term of 2/d in the exponents). We conjecture, that this could be removed by a more refined analysis (in particular in the construction of the operator A * ).

Modelling gradients. Our current framework only used function values. If gradients are observed, it could be possible to use them to reduce the number of sampled points, using tools by [START_REF] Zhou | Derivative reproducing properties for kernel methods in learning theory[END_REF].

The choice of Ω. Since we assume that f has at least one global minimum, then there exists always an open set Ω that contains it and that satisfies the required properties. In this work, we don't discuss how to find Ω. While, in general, this could be not an easy problem. In practice, many non-convex optimization problems come already with a region of interest where to look for the global minimum. Such a region is typically obtained by considering some basic properties of the function of interest. For example, if are minimizing a polynomial of the form f

(x) = B(x) + p(x), where B(x) = x 2r 1 + • • • + x 2r
d for some r ∈ N and p(x) is a polynomial of degree q ≤ 2r -1. Note that by construction f admits a global minimum, since it goes to +∞ at infinity and has p(0) < ∞ (while any polynomial without this structure does not have a minimizer). Now it is possible to easily derive a hypercube that contains the global minimum. Indeed by construction f (x * ) ≤ f (0) = p(0). Denote by L the sum of the absolute values of the coefficients of p. Now take the smallest R ≥ 1 such that R m -LR q ≥ p(0) + ε for an ε > 0. For any x / ∈ (-R, R), we have

f (x) = B(x) + p(x) ≥ R m -LR q > p(0) ≥ f (x * ).
Then the region (-R, R) contains all the global minimizers.

Efficient kernel approximations. The current algorithm has a complexity of O(n 3 ) for n sampled points, partly due to the need to compute inverse of kernel matrices. There is a large literature within machine learning aiming at providing low-rank approximations, either from approximations of K from a subset of its columns (see, e.g., the work by [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF]; [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF] and references therein) or using random feature vectors (see, e.g., the work by Rudi and Rosasco (2017); Bach (2017b) and references therein). This requires to relax the equality constraint on the subset X to an mean square deviations, as allowed by Sec. 9.8 .

Constrained optimization. Following the work by [START_REF] Lasserre | Global optimization with polynomials and the problem of moments[END_REF], we can apply the same algorithmic technique to constrained optimization, by formulating the problem of minimizing f (x) such that g(x) 0 as maximizing c such that f (x) = c + p(x) + g(x)q(x), and p, q nonnegative functions. We can then replace the non-negative constraints by p(x) = φ(x), Aφ(x) and q(x) = φ(x), Bφ(x) for positive operators A and B. We can then subsample and penalize the traces of A and B to obtain an algorithm. A detailed study of the approximation properties of this algorithm remains to be done.

9.A Additional notations and definitions

We provide here some basic notation that will be used in the rest of the appendices.

Multi-index notation. Let α ∈ N d , x ∈ R d and f be an infinitely differentiable function on R d , we introduce the following notation

|α| = j∈[d] α i , α! = j∈[d] α j !, x α = j∈[d] x α j j , ∂ α f = ∂ |α| f ∂x α 1 1 • • • ∂x α d d .
Some useful space of functions. Let Ω be an open set. In this paper we will denote by C s (Ω), s ∈ N, the set of s-times differentiable functions on Ω and by C s 0 (Ω) the set of functions that are differentiable at least s times and that are supported on a compact in Ω. Denote by L p (Ω) the Lebesgue space of p-integrable functions with respect to the Lebesgue measure and denote by • L p (Ω) the associated norm [START_REF] Adams | Sobolev Spaces[END_REF].

9.A .1 Fourier Transform.

Given two functions f, g : Ω → R on some set Ω, we denote by f • g the function corresponding to pointwise product of f, g, i.e.,

(f • g)(x) = f (x)g(x), ∀x ∈ Ω. Let f, g ∈ L 1 (R d ) we denote the convolution by f g (f g)(x) = R d f (y)g(x -y)dy. Let f ∈ L 1 (R d ).
The Fourier transform of f is denoted by f and is defined as

f (ω) = (2π) -d 2 R d e -i ω x f (x) dx,
We now recall some basic properties, that will be used in the rest of the appendix. Proposition 9.2 (Basic properties of the Fourier transform [START_REF] Wendland | Scattered Data Approximation[END_REF], Chapter 5.2.).

(a) Let f ∈ L 1 (R d ) and let r > 0. Denote by f its Fourier transform and by f r the function

f r (x) = f (x/r) for all x ∈ R d , then fr (ω) = r d f (rω). (b) Let f, g ∈ L 1 (R d ), then f • g = (2π) d/2 f g. (c) Let α ∈ N d 0 , f : R d → R and f, ∂ α f ∈ L 1 (R d ), then ∂ α f (ω) = i |α| ω α f (ω), ∀ω ∈ R d . (d) Let f ∈ L 1 (R d ), then f L ∞ (R d ) ≤ (2π) -d/2 f L 1 (R d ) . 9.A . ADDITIONAL NOTATIONS AND DEFINITIONS 419 (e) Let f ∈ L 1 (R d ) and assume that f ∈ L 1 (R d ), then f (x) = (2π) -d 2 R d e i ω x f (ω) dx, and f L ∞ (R d ) ≤ (2π) -d/2 f L 1 (R d ) .
(f) There exists a linear isometry

F : L 2 (R d ) → L 2 (R d ) satisfying Ff = f , f ∈ L 2 (R d ) ∩ L 1 (R d ).
The isometry is uniquely determined by the property in the equation above. For any f ∈ L 2 (R d ) we denote by f the function f = Ff .

9.A .2 Sobolev Spaces

For this section we refer to the work by [START_REF] Adams | Sobolev Spaces[END_REF]. For any α ∈ N d 0 we say that

v α ∈ L 1 loc (R d ) is the α-weak derivative of u ∈ L 1 loc (R d ) if, for all compactly supported smooth functions τ ∈ C ∞ 0 (R d ), we have R d v α (x)τ (x)dx = (-1) |α| R d u(x)(∂ α τ )(x)dx, and we denote v α by D α u. Let Ω ⊆ R d be an open set. For s ∈ N, p ∈ [1, ∞] the Sobolev spaces W s p (Ω) are defined as W s p (Ω) = {f ∈ L p (Ω) | f W s p (Ω) < ∞}, f W s p (Ω) = |α|≤s D α f L p (Ω) .
We now recall some basic results about Sobolev spaces that are useful for the proofs in this paper.

g| Ω W s p (Ω) ≤ g W s p (Ω ) ,
and so g| Ω ∈ W s p (Ω) for any g ∈ W s p (Ω ). Now we recall the extension properties of Sobolev spaces.

Proposition 9.3 (Extension operator, 5.24 in the work by [START_REF] Adams | Sobolev Spaces[END_REF]). Let Ω be a bounded open subset of R d with locally Lipschitz boundary [START_REF] Adams | Sobolev Spaces[END_REF] Ω) with C 3 = E op . Proposition 9.4 (Approximation property of Sobolev spaces, Thm 5.33 in the work by [START_REF] Adams | Sobolev Spaces[END_REF]). Let Ω be a bounded open subset of R d with locally Lipschitz boundary [START_REF] Adams | Sobolev Spaces[END_REF], or Ω = R d . Let s, d ∈ N, r ≥ s and p ∈ [1, ∞]. There exists C 1 depending only on s, d, p and C 2 depending only on r, s, d, p such that for any ε ∈ (0, 1] and g ∈ W s p (Ω) there exists

. Let β ∈ N and p ∈ [1, ∞]. There exists a bounded operator E : W β p (Ω) → W β p (R d ) and a constants C 3 depending only on β, p, Ω such that for any h ∈ W β p (Ω) the following holds (a) h = (Eh)| Ω (b) Eh W β p (R d ) ≤ C 3 h W β p (
g ε ∈ C ∞ (Ω) satisfying (i) g is the restriction to Ω of a certain g ∈ C ∞ (R d ) and (ii) g -g ε L p (Ω) ≤ C 1 ε s g W s p (Ω) , g ε W r p (Ω) ≤ C 2 ε -(r-s) g W s p (Ω) .

9.A .3 Reproducing Kernel Hilbert spaces

For this section we refer to the work by [START_REF] Aronszajn | Theory of reproducing kernels[END_REF]; [START_REF] Steinwart | Support Vector Machines[END_REF]; [START_REF] Paulsen | An Introduction to the Theory of Reproducing Kernel Hilbert Spaces[END_REF]. Let S be a set and k : S × S → R be a p.d. kernel. We denote by H k (S) the reproducing kernel Hilbert space (RKHS) associated to the kernel k, and by •, • k the associated inner product. In particular, we will omit the dependence in k from H and •, • when the used kernel is clear from the context. We will omit also the dependence on S when S = Ω, the region we are using in this paper. In particular we will use the following shortcuts

H = H k (Ω) and H(R d ) = H k (R d ).
Concrete constructions and useful characterizations. In the rest of the section we provide other methods to build RKHS and some interesting characterizations of H k (S) and •, • k that will be useful int the rest of the appendix.

Proposition 9.7 (Construction of RKHS given S, φ, Thm. 4.21 by [START_REF] Steinwart | Support Vector Machines[END_REF]). Let φ : S → V be a continuous map, where V is separable Hilbert space with inner product •, • V . Let k(x, x ) = φ(x), φ(x ) V for any x, x ∈ S. Then k is a p.d. kernel and the associated RKHS is characterized as follows

H k (S) = { w, φ(•) V | w ∈ V }, f H k (S) = inf u∈V u V s.t. f = u, φ(•) V .
Proposition 9.8 (Restriction of a RKHS H k 1 (S 1 ) on a subset S 0 ⊂ S 1 [START_REF] Aronszajn | Theory of reproducing kernels[END_REF][START_REF] Paulsen | An Introduction to the Theory of Reproducing Kernel Hilbert Spaces[END_REF]). Let k 0 be the restriction on S 0 of the kernel k 1 defined on S 1 . Then the following holds

(a) k 0 is a p.d. kernel, (b) the RKHS H k 0 (S 0 ) is characterized as H k 0 (S 0 ) = {f | S 0 | f ∈ H k 1 (S 1 )}, (c) the norm • H k 0 (S 0 ) is characterized by f H k 0 (S 0 ) = inf g∈H k 1 (S 1 ) g H k 1 (S 1 ) , s.t. f (x) = g(x) ∀x ∈ S 0 , (d
) there exist a linear bounded extension operator E :

H k 0 (S 0 ) → H k 1 (S 1 ) such that (Ef )(x) = f (x) for any x ∈ S 0 and f ∈ H k 0 (S 0 ) and such that f H k 0 (S 0 ) = Ef H k 1 (S 1 ) , ∀f ∈ H k 0 (S 0 ), (e
) there exist a linear bounded restriction operator R :

H k 1 (S 1 ) → H k 0 (S 0 ) such that (Rf )(x) = f (x) for any x ∈ S 0 and f ∈ H k 1 (S 1 ),
(f) R and E are partial isometries. In particular E = R * and RE is the identity on H k 0 (S 0 ), while ER is a projection operator on H k 1 (S 1 ).

Proposition 9.9 (Translation invariant kernels on R d ). Let v : R d → R such that its Fourier transform ṽ is integrable and satisfies ṽ ≥ 0 on R d . Then

H k (R d ) = {f ∈ L 2 (R d ) | f H k (R d ) < ∞}, f 2 H k (R d ) = (2π) -d 2 R d |(Ff )(ω)| 2 ṽ(ω) dω, CHAPTER 9
. FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS

9.B Fundamental results on scattered data approximation

We recall here some fundamental results about local polynomial approximation. In particular, we report here the proofs to track explicitly the constants. The proof techniques are essentially from the work by [START_REF] Narcowich | Refined error estimates for radial basis function interpolation[END_REF]; [START_REF] Wendland | Scattered Data Approximation[END_REF]. Denote by π k (R d ) the set of multivariate polynomials of degree at most k, with k ∈ N. In this section B r (x) ⊂ R d denotes the open ball of radius r and centered in x.

= 0 or h Y ,B δ ≤ δ 9k 2 , there exist u j : B δ → R with j ∈ [m] such that (a) j∈[m] p(y j )u j (x) = p(x), ∀x ∈ B δ , p ∈ π k (R d ) (b) j∈[m] |u j (x)| ≤ 2, ∀x ∈ B δ . Lemma 9.
8 (Bounds on functions with scattered zeros on a small ball [START_REF] Narcowich | Refined error estimates for radial basis function interpolation[END_REF][START_REF] Wendland | Scattered Data Approximation[END_REF]

). Let k ∈ N, d, m ∈ N + and δ > 0. Let B δ ⊂ R d be a ball of radius δ in R d . Let f ∈ C k+1 (B δ ). Let Y = {y 1 , . . . , y m } ⊂ B δ be a non empty finite subset of B δ . If either k = 0 or h Y ,B δ ≤ δ 9k 2 , it holds: sup x∈B δ |f (x)| ≤ 3Cδ k+1 + 2 max i∈[m] |f (y i )|, C := |α|=k+1 1 α! ∂ α f L ∞ (B δ ) . Proof. Note that since either k = 0 or h Y ,B δ ≤ δ 9k
∈ B δ , |f (x)| ≤ |f (x) -p(x)| + |p(x) -s f, Y (x)| + |s f, Y (x)| ≤ |f (x) -p(x)| + j∈[m] |p(y j ) -f (y j )||u j (x)| + max j∈[m] |f (y j )| j∈[m] |u j (x)| ≤ f -p L ∞ (B δ ) (1 + j∈[m] |u j (x)|) + τ j∈[m] |u j (x)| ≤ 3 f -p L ∞ (B δ ) + 2τ.
In particular, consider the Taylor expansion of f at the center x 0 of B δ up to order k (e.g. the work by [START_REF] Brenner | The Mathematical Theory of Finite Element Methods[END_REF] Eq. 4.2.5 pag 95). For any x ∈ B δ , it holds Theorem 9.13 (Bounds on functions with scattered zeros [START_REF] Narcowich | Refined error estimates for radial basis function interpolation[END_REF][START_REF] Wendland | Scattered Data Approximation[END_REF]). Let k, m ∈ N s.t. k ≤ m and n, d ∈ N + . Let r > 0 and Ω an open set of

f (x) = |α|≤k 1 α! ∂ α f (x 0 )(x -x 0 ) α + |α|=k+1 k + 1 α! (x -x 0 ) α 1 0 (1 -t) k ∂ α f ((1 -t)x 0 + tx)dt. By choosing p(x) = |α|≤k 1 α! ∂ α f (x 0 )(x -x 0 ) α ∈ π k (R d ) it holds: f -p L ∞ (B δ ) ≤ |α|=k+1 δ k+1 α! ∂ α f L ∞ (B δ ) = Cδ k+1 , 9
R d of the form Ω = x∈S B r (x) for some subset S of R d . Let X = {x 1 , . . . , x n } be a non-empty finite subset of Ω. Let f ∈ C m+1 (Ω). If h X,Ω ≤ r max(1, 1 18k 2 ), then sup x∈Ω |f (x)| ≤ CC f h k+1 X,Ω + 2 max i∈[n] |f (x i )|, where C = 3 max(1, 18 k 2 ) k+1 and C f = |α|=k+1 1 α! ∂ α f L ∞ (Ω) .
Proof. First, note that the condition that there exists a set S such that Ω = x∈S B r (x) implies ∀δ ≤ r, Ω =

x 0 ∈S δ B δ (x 0 ), S δ = {x ∈ Ω : ∃x ∈ S, x -x ≤ r -δ}.
We will now prove the theorem for k ≥ 1 and then the easier case k = 0, where we will use essentially only the Lipschitzianity of f .

Proof of the case k ≥ 1. The idea of the proof is to apply Lemma 9.8 to a collection of balls of radius δ for a well chosen δ ≤ r and centered in x 0 ∈ S δ defined above. Given X, to apply Lemma 9.8 on a ball of radius δ we have to restrict the points in X to the subset belonging to that ball, i.e., Y x 0 ,δ = X ∩ B δ (x 0 ), x 0 ∈ S δ and δ > 0. The set Y x 0 ,δ will have a fill distance h x 0 ,δ = h Y x 0 ,δ ,B δ (x 0 ) . First we are going to show that Y x 0 ,δ is not empty, when r > δ > h X,Ω . To obtain this result we need to study also the ball B δ (x 0 ) with δ = δh X,Ω .

Step 1. Showing that Y x 0 ,δ is not empty and for any y ∈ B δ (x 0 ) there exists z ∈ Y x 0 ,δ satisfying yz ≤ h X,Ω . Let x 0 ∈ S δ and δ ≤ r. This implies that B δ (x 0 ) ⊆ Ω by the characterization of Ω in terms of S δ we gave above. Define now δ = δh X,Ω and note that B δ (x 0 ) is non empty, since δ > 0, and that B δ (x 0 ) ⊂ B δ (x 0 ) ⊆ Ω. Now note that by definition of fill distance, for any y ∈ B δ (x 0 ) there exists a z ∈ X such that zy ≤ h X,Ω . Moreover

note that z ∈ B δ (x 0 ), since x 0 -z ≤ x 0 -y + y -z < δ -h X,Ω + h X,Ω = δ. Since z ∈ X
and also in B δ (x 0 ), then z ∈ Y x 0 ,δ by definition of Y x 0 ,δ .

Step 2. Showing that h x 0 ,δ ≤ 2h X,Ω . Let x ∈ B δ (x 0 ). We have seen in the previous step that the ball B δ (x 0 ) is well defined and non empty, with δ = δh X,Ω . Now note that also B h X,Ω (x) ∩ B δ (x 0 ) is not empty, indeed the distance between the centers x, x 0 is strictly smaller than the sum of the two radii, indeed x -

x 0 < δ = δ + h X,Ω , since x ∈ B δ (x 0 ). Take w ∈ B h X,Ω (x) ∩ B δ (x 0 ). Since w ∈ B δ (x 0 ) by Step 1 we know that there exists z ∈ Y x 0 ,δ with w -z ≤ h X,Ω . Since w ∈ B h X,Ω (x), then we know that x -w < h X,Ω . So x -z ≤ x -w + w -z < 2h X,Ω .
Step 3. Applying Lemma 9.8. Since, by assumption h X,Ω ≤ r/(18k 2 ) and k ≥ 1, then the choice δ = 18k 2 h X,Ω implies r ≥ δ > h X,Ω . So we can use the characterization of Ω in terms of S δ and the results in the previous two steps, obtaining that for any x 0 ∈ S δ the set B δ (x 0 ) ⊆ Ω and moreover the set Y x 0 ,δ is not empty and covers B δ (x 0 ) with a fill distance h x 0 ,δ ≤ 2h X,Ω . Since, h x 0 ,δ ≤ 2h X,Ω ≤ δ/(9k 2 ) then we can apply Lemma 9.8 to each ball B δ (x 0 ) obtaining sup

x∈B δ (x 0 ) |f (x)| ≤ 3C δ,x 0 δ k+1 + 2 max z∈ Y x 0 ,δ |f (z)|, C δ,x 0 := |α|=k+1 1 α! ∂ α f L ∞ (B δ (x 0 )) .
The proof is concluded by noting that Ω = x 0 ∈S δ B δ (x 0 ) and that for any x 0 ∈ S δ we have

C δ,x 0 ≤ C f , δ k+1 ≤ (18k 2 ) k+1 h k+1 X,Ω and moreover that max z∈ Y x 0 ,δ |f (z)| ≤ max i∈[n] |f (x i )|, since Y x 0 ,δ ⊆ X by construction.
Proof of the case k = 0 Since h X,Ω ≤ r, by assumption, then δ = h X,Ω implies that Ω admits a characterization as Ω = x 0 ∈S δ B δ (x 0 ). Now let x ∈ Ω and choose x 0 ∈ S δ such that x ∈ B δ (x 0 ). One the one hand, since the segment [x 0 , x] is included in Ω, by Taylor inequality, |f

x) -f (x 0 )| ≤ C f x -x 0 ≤ C f h X,Ω and C f = |α|=1 1 α! ∂ α f L ∞ (Ω) (
. One the other hand, by definition of h X,Ω , there exists z ∈ X ⊂ Ω such that zx 0 ≤ h X,Ω = δ. Since both the open segment [x 0 , z) ⊂ B δ (x 0 ) ⊂ Ω and z ∈ Ω, then the whole segment [x 0 , z] ⊂ Ω and hence we can apply Taylor inequality to show f

(x 0 ) -f (z) ≤ C f z -x 0 ≤ C f h X,Ω . Then we have |f (x)| ≤ |f (x) -f (x 0 )| + |f (x) -f (z)| + |f (z)| ≤ 2C f h X,Ω + max i∈[n] |f (x i )|.
The proof of the step k = 0 is concluded by noting that the previous inequality holds for every x ∈ Ω.

9.C Auxiliary results on RKHS

We recall that the nuclear norm of a compact linear operator A is defined as A = Tr( √ A * A) or equivalently A = j∈N σ j , where (σ j ) j∈N are the singular values of A (Chapter 7 by [START_REF] Weidmann | Linear Operators in Hilbert Spaces[END_REF] or [START_REF] Bhatia | Matrix Analysis[END_REF] for the finite dimensional analogue). Lemma 9.9. Let Ω be a set, k be a kernel and H the associated RKHS. Let A : H → H be a trace class operator. If H satisfies Assumption 9.2(a), then r A H ≤ M A , where r A (x) := φ(x), Aφ(x) , ∀x ∈ Ω, and A is the nuclear norm of A. We recall that if A ∈ S + (H) then A = Tr(A).

Proof. Since A is compact, it admits a singular value decomposition A = i∈N σ i u i ⊗ v i . Here, (σ j ) j∈N is a non-increasing sequence of non-negative eigenvalues converging to zero, and (u j ) j∈N and (v j ) j∈N are two orthonormal families of corresponding eigenvectors, (a family (e j ) is said to be orthonormal if for i, j ∈ N, e i , e j = 1 if i = j and e i , e j = 0 otherwise) [START_REF] Weidmann | Linear Operators in Hilbert Spaces[END_REF]. Note that we can write r A using this decomposition as r A

(x) = i∈N σ i u i (x)v i (x) = i∈N σ i (u i •v i )(x),
for all x ∈ Ω, where we denote by • the pointwise multiplication between two functions (this equality is justified by the following absolute convergence bound). By Assumption 9.2(a), the fact that A is trace-class (i.e., A < ∞) and the fact that u j , v j satisfy u j H = v j H = 1, j ∈ N, the following holds

r A H = j∈N σ j (u j • v j ) H ≤ j∈N σ j u j • v j H ≤ M j∈N σ j u j H v j H ≤ M j∈N σ j = M A .
In the case where A ∈ S + (H), we have A = Tr( √ A * A) = Tr(A). Given the kernel k, the associated RKHS H and the canonical feature map φ : Ω → H and a set of distinct points X = {x 1 , . . . , x n } define the kernel matrix K ∈ R n×n as K i,j = φ(x i ), φ(x j ) = k(x i , x j ) for all i, j ∈ [n]. Note that, since k is a p.d. kernel, then K is positive semidefinite, moreover when k is universal, then φ(x 1 ), . . . , φ(x n ) are linearly independent, so K is full rank and hence invertible. Universality of k is guaranteed since H contains the C ∞ 0 (Ω) functions, by Assumption 9.1(a), and so can approximate continuous functions over compacts in Ω [START_REF] Steinwart | Support Vector Machines[END_REF]. Denote by R the upper triangular matrix corresponding to the Cholesky decomposition of K, i.e., R satisfies K = R R. We are ready to start the proof of Lemma 9.2.

Proof. Denote by S : H → R n the linear operator that acts as follows

Sg = ( φ(x 1 ), g , . . . , φ(x n ), g ) ∈ R n , ∀g ∈ H.
Define S * : R n → H, i.e., the adjoint of S, as S * β = n i=1 β i φ(x i ) for β ∈ R n . Note, in particular, that K = S S * and that S * e j = φ(x i ), where e j is the j-th element of the canonical basis of R n . We define the operator V = R -S and its adjoint V * = S * R -1 . By using the definition of V , the fact that K = R R by construction of R, and the fact that K = S S * , we derive two facts.

On the one hand,

V V * = R -S S * R -1 = R -KR -1 = R -R RR -1 = I.
On the other hand, P is a projection operator, i.e., P 2 = P , P is positive definite and its range is range P = span φ(x i ) | i ∈ [n], implying P φ(x i ) = φ(x i ) for all i ∈ [n]. Indeed, using the equation above, P 2 = V * V V * V = V * (V V * )V = V * V = P , and the positive-semi-definiteness of P is given by construction since it is the product of an operator and its adjoint. Moreover, the range of P is the same as that of V * which in turn is the same as that of S * , since R is invertible : range

P = span φ(x i ) | i ∈ [n].
Finally, note that since k(x, x ) = φ(x), φ(x ) , for any x, x ∈ Ω, then for any j ∈ [n], Φ j is characterized by

Φ j = R -(k(x 1 , x j ), . . . , k(x n , x j )) = R -( φ(x 1 ), φ(x j ) , . . . , φ(x n ), φ(x j ) ) = R -Sφ(x j ) = V φ(x j ).
9.D The constants of translation invariant and Sobolev kernels 9.D .1 Results for translation invariant and Sobolev kernels Lemma 9.10. Let Ω be a set and let k(x, x ) = v(xx ) for all x, x ∈ Ω, be a translation invariant kernel for some function v : R d → R. Denote by ṽ the Fourier transform of v. Let H be the associated RKHS. For any f, g ∈ H we have

f • g H ≤ C f H g H , C = (2π) d/4 ṽ ṽ ṽ 1/2 L ∞ (R d ) .
In particular, if there exists a non-increasing g :

[0, ∞] → (0, ∞] s.t. ṽ(ω) ≤ g( ω ), then C ≤ √ 2(2π) d/2 v(0) 1/2 sup ω∈R d g( 1 2 ω ) ṽ(ω) .
Proof. First note that by as recalled in proposition 9.8, there exists an extension operator, i.e., a partial isometry E : H → H(R d ) such that r = Eu satisfies r(x) = u(x) for all x ∈ Ω and u H = r H , for any u ∈ H. Moreover there exists a restriction operator R : H(R d ) → H, as recalled in proposition 9.8, such that RE : H → H is the identity operator and ER :

H(R d ) → H(R d ) is a projection operator whose range is H. Moreover, note that f • g = R(Ef • Eg) since for any x ∈ Ω, (R(Ef • Eg))(x) = (Ef )(x)(Eg)(x) = f (x)g(x) = (f • g)(x). Since ER is a projection operator, then ER op ≤ 1, hence f • g H = R(Ef • Eg) H = ER(Ef • Eg) H(R d ) ≤ ER op Ef • Eg H(R d ) ≤ Ef • Eg H(R d ) .
Ef • Eg 2 H(R d ) = a • b 2 H(R d ) = (2π) -d/2 R d | a • b (ω)| 2 ṽ(ω) dω. Now we bound a • b. Since a • b = (2π) d/2 ã b (see proposition 9.
2) where corresponds to the convolution, by expanding it and by applying Cauchy-Schwarz we obtain

(2π) -d/2 | a • b(ω)| 2 = |(ã b)(ω)| 2 = R d ã(σ) b(ω -σ)dσ 2 = R d ã(σ) ṽ(σ) b(ω -σ) ṽ(ω -σ) √ ṽ(σ) √ ṽ(ω -σ)dσ 2 ≤ R d ã2 ṽ (σ) b2 ṽ (ω -σ)dσ R d ṽ(σ)ṽ(ω -σ)dσ = ã2 ṽ b2 ṽ (ω) (ṽ ṽ)(ω).
By using the bound above together with Hölder inequality and Young inequality for convolutions, we have

(2π) -d/2 R d | a • b (ω)| 2 ṽ(ω) dω ≤ R d ã2 ṽ b2 ṽ (ω) (ṽ ṽ)(ω) ṽ(ω) dω ≤ ã2 ṽ b2 ṽ L 1 (R d ) ṽ ṽ ṽ L ∞ (R d ) ≤ ã2 ṽ L 1 (R d ) b2 ṽ L 1 (R d ) ṽ ṽ ṽ L ∞ (R d ) = (2π) d/2 ṽ ṽ ṽ L ∞ (R d ) a 2 H(R d ) b 2 H(R d ) = C 2 ,
where in the last step we used the definitions of inner products for translation invariant kernels. The proof is concluded by noting that a H(R d ) = Ef H(R d ) = f H and the same holds for b, i.e., b H(R d ) = g H . A final consideration is that C can be further bounded by applying proposition 9.12 and noting that v(0) = (2π) -d/2 ṽ(ω)dω = (2π) -d/2 ṽ L 1 (R d ) , via the characterization of v in terms of ṽ in proposition 9.2(e), since ṽ(ω) ≥ 0 and integrable.

9.D . THE CONSTANTS OF TRANSLATION INVARIANT AND SOBOLEV KERNELS429

Proposition 9.12. Let u ∈ L 1 (R d ) ∩ C(R d ) be u(x) ≥ 0 for x ∈ R d and such that there exists a non-increasing function g

: [0, ∞) → (0, ∞) satisfying u(x) ≤ g( x ) for all x ∈ R d . Then it holds : ∀x ∈ R d , 0 ≤ (u u)(x) ≤ 2 u L 1 (R d ) g( 1 2 x ). In particular, if u > 0, it holds u u u L ∞ (R d ) ≤ 2 u L 1 (R d ) sup x∈R d g( 1 2 x ) u(x) . Proof. For any x ∈ R d , (u u)(x) = sup x∈R d R d u(y)u(x -y)dy. Let S x = {y | x -y ≤ 1 2 x }. Note that, when y ∈ R d \ S x , then x -y > 1 2 x . Instead, when y ∈ S x , then 1 2 x ≤ x -x -y ≤ y . Since g is non-increasing, for any x ∈ R d we have R d u(y)u(x -y)dy = Sx u(y)u(x -y)dy + R d \Sx u(y)u(x -y)dy ≤ Sx g( y )u(x -y)dy + R d \Sx u(y)g( x -y )dy ≤ Sx g( 1 2 x )u(x -y)dy + R d \Sx u(y)g( 1 2 x )dy ≤ R d g( 1 2 x )u(x -y)dy + R d u(y)g( 1 2 x )dy = R d g( 1 2 x )u(y)dy + R d u(y)g( 1 2 x )dy = 2 g( 1 2 x ) R d u(y)dy,
where: in the first inequality we bounded u(y) with g( y ) and u(xy) with g( xy ), in the first and the second integral, respectively; in the second inequality we bounded g( y ) with g( 1 2 x ), since y ≥ 1 2 x when y ∈ S x and we bounded g( xy ) with g( 1 2 x ), since xy ≥ 1 2 x when y ∈ R d \ S x ; in the third we extended the integration domains to R d .

9.D .2 Proof of proposition 9.1

Proof. We prove here that the Sobolev kernel satisfies Assumption 9.2. Let k = k s from Eq. (9.7).

As we have seen in Example 9.1

H = W s 2 (Ω) and • W s 2 (Ω) is equivalent to • H , when s > d/2
and Ω satisfies Assumption 9.1(a) since this assumption implies that Ω satisfies the cone condition [START_REF] Wendland | Scattered Data Approximation[END_REF]. [START_REF] Wendland | Scattered Data Approximation[END_REF]. In the rest of the proof, C 0 will always refer to this constant.

Recall that k is translation invariant, i.e., k(x, x ) = v(x -x ) for any x, x ∈ R d , with v defined in Example 9.1. The Fourier transform of v is ṽ(ω) = C 0 (1 + ω 2 ) -s with C 0 = 2 d/2 Γ(s) Γ(s-d/2)
We are going to divide the proof in one step per point of Assumption 9.2.

Proof of Assumption 9.2(d) for the Sobolev kernel.

Let α ∈ N d , m = |α|. Assume m < s-d/2, i.e., m ∈ {1, . . . , s-(d+1)/2 }. Since k is translation invariant, then ∂ α x ∂ α y k(x, y) = (-1) m v 2α (x -y) with v 2α (z) = ∂ 2α z v(z) for all z ∈ R d . So sup x,y∈Ω |∂ α x ∂ α y k(x, y)| = sup x,y∈Ω |∂ α x ∂ α y v(x -y)| ≤ sup z∈R d |∂ 2α z v(z)| ≤ (2π) -d/2 ω 2α ṽ(z) L 1 (R d ) ,
where in the last step we used elementary properties of the Fourier transform (in particular the ones recalled in propositions 9.2(c) and 9.2(e)). Let S d-1 = 2 π d/2 Γ(d/2) be the area of the d -1 dimensional sphere. Since m < sd/2 and ṽ ≥ 0,

ω 2α ṽ(z) L 1 (R d ) ≤ R d ω 2m ṽ(ω)dω = C 0 S d-1 ∞ 0 r 2m+d-1 (1 + r 2 ) s dr = C 0 S d-1 ∞ 0 t m+d/2-1 2(1 + t) s dt = C 0 S d-1 Γ(m+d/2)Γ(s-d/2-m) 2Γ (s)
,

where we performed a change of variable r = √ t and dr = dt 2 √ t and applied Eq. 5.12.3 pag. 142 by [START_REF] Frank | NIST Handbook of Mathematical Functions[END_REF] to the resulting integral. Thus, Assumption 9.2(d) holds with

D 2 m = C 0 π d/2 Γ(m + d/2)Γ(s -m -d/2) Γ(d/2)Γ(s) = (2π) d/2 Γ(m + d/2)Γ(s -d/2 -m) Γ(s -d/2)Γ(d/2) .
Proof of Assumption 9.2(a) for the Sobolev kernel. First, note that

C ∞ (R d )| Ω ⊂ W s ∞ (Ω) ⊂ W s 2 (Ω). Indeed, since Ω is bounded, for any f ∈ C ∞ (R d ), ∂ α f | Ω L ∞ (Ω) < ∞ for any α ∈ N d . This shows that f | Ω ∈ W s ∞ (Ω). Moreover W s ∞ (Ω) ⊂ W s 2 (Ω) since • L 2 (Ω) ≤ vol(Ω) 1/2 • L ∞ (Ω)
because Ω is bounded. Second, since ṽ(ω) = g s ( ω ) with g s (t) = C 0 (1+t 2 ) -s , positive and non-increasing, we can apply Lemma 9.10. Therefore, for C

= √ 2(2π) d/2 v(0) 1/2 sup t≥0 gs(t/2) gs(t) 1/2 it holds f •g H ≤ C f H g H .
In particular we have sup t≥0 gs(t/2) gs(t)

1/2 ≤ 2 s and v(0) = 1, since lim t→0 t s-d/2 K s-d/2 (t) = Γ(s -d/2)/2 1+d/2-s = 1/C 0 (
(R d ), h L ∞ (R d ) ≤ C s h W s 2 (R d ) . In particular, this shows that W s 2 (R d ) ⊂ L ∞ (R d ).
Fix such a constant C s in the rest of the proof. Let p ∈ N and g ∈ C ∞ (R p ) with g(0, 0, . . . , 0) = 0. From (i) of Thm. 11 in the work by [START_REF] Sickel | Superposition of functions in sobolev spaces of fractional order. a survey[END_REF], there exists a constant c g depending only on g, p, s such that for any h

1 , . . . , h p ∈ W s 2 (R d ) ∩ L ∞ (R d ), it holds g(h 1 , . . . , h p ) W s 2 (R d ) ≤ c g sup i∈[p] h i W s 2 (R d ) 1 + h i max(0,s-1) L ∞ (R d )
.

Since s > d/2, the bound above shows, in particular, that for any h

1 , . . . , h p ∈ W s 2 (R d ), it holds g(h 1 , . . . , h p ) W s 2 (R d ) ≤ c g sup i∈[p] h i + h i max(1,s) W s 2 (R d ) , c g = c g max 1, C max(0,s-1) s . Since W s 2 (R d ) = H(R d ) and • W s 2 (R d ) and • H(R d )
are equivalent (see the work by [START_REF] Adams | Sobolev Spaces[END_REF]), the previous inequality holds for • H(R d ) with a certain constant c g depending only on g, p, s, d. In particular, this implies that g(h 1 , . . . , h p ) ∈ H(R d ) for any h 1 , . . . , h p ∈ H(R d ). Now we are going to prove the same implication for the restriction on Ω.

9.D . THE CONSTANTS OF TRANSLATION INVARIANT AND SOBOLEV KERNELS431

First note that any function in a ∈ C ∞ (R p ) can be written as a(z) = q 1(z) + g(z), z ∈ R p where q = a(0, 0, • • • , 0) ∈ R, g ∈ C ∞ (R p ) with g(0, 0, • • • , 0) = 0 and 1(z) = 1 for all z ∈ R p . Recall the definition and basic results on the extension operator E : H → H(R d ) from proposition 9.8. For any f 1 , . . . , f p ∈ H, note that g((Ef 1 )(x), . . . , (Ef p )(x)) = g(f 1 (x), . . . , f p (x)) for all x ∈ Ω. We can now apply the results of proposition 9.8 to show that g(f 1 , . . . , f p ) ∈ H :

g(f 1 , . . . , f p ) H = inf u u H(R d ) s.t. u(x) = g(f 1 (x), . . . , f p (x)) ∀x ∈ Ω ≤ g(Ef 1 , . . . , Ef p ) H(R d ) ≤ c g sup j∈[p] Ef j H(R d ) + Ef j max(1,s) H(R d ) = c g sup j∈[p] f j H + f j max(1,s) H < ∞,
where in the last step we used the fact that

• H = E • H(R d )
. The proof of this point is concluded by noting that, a(f 1 , . . . , f p ) ∈ H, since 1 ∈ H, due to the Point (a) above, and

a(f 1 , . . . , f p ) H ≤ q 1 H + g(f 1 , . . . , f p ) H < ∞.
Proof of Assumption 9.2(c) for the Sobolev kernel. This proof is done in Lemma 9.11, right below.

Before stating Lemma 9.11 we are going to recall some properties. First, recall the Young inequality :

∀f ∈ L 2 (R d ), ∀g ∈ L 1 (R d ), f g L 2 (R d ) ≤ f L 2 (R d ) g L 1 (R d ) .
Moreover, by definition of the Sobolev kernel, it is a translation-invariant kernel with v defined in Example 9.1, with Fourier transform ṽ(ω) = C 0 (1 + ω 2 ) -s . Let H(R d ) be the reproducing kernel Hilbert space on R d associated to the Sobolev kernel k s . As recalled in proposition 9.9, the H

(R d )-norm is characterized by ∀f ∈ H(R d ), f H(R d ) = (2π) -d/4 f / √ ṽ L 2 (R d ) , (9.38)
where f = F(f ) is the Fourier transform of f (see the work by [START_REF] Adams | Sobolev Spaces[END_REF]).

Then we recall that ṽ ∈ .39) where 9.40) where

L 1 (R d ), since s > d/2, so for any f ∈ H(R d ) f L 1 (R d ) = √ ṽ f / √ ṽ L 1 (R d ) ≤ √ ṽ L 2 (R d ) f / √ ṽ L 2 (R d ) = C 1 f H(R d ) . ( 9
C 1 = (2π) d/4 √ ṽ L 2 (R d ) . A useful consequence of the inequality above is obtained by considering that f L ∞ (R d ) is bounded by the L 1 norm of f (see proposition 9.2(e)), then f L ∞ ≤ (2π) -d/2 f L 1 (R d ) ≤ C 2 f H(R d ) , (
C 2 = (2π) -d/4 √ ṽ L 2 (R d ) .
Lemma 9.11 (Assumption 9.2(c) for Sobolev Kernels). Let H be the RKHS associated to the translation invariant Sobolev Kernel defined in Example 9.1, with s > d/2. Then Assumption 9.2(c) is satisfied.

C 0 /ṽ(ω) for any ω ∈ R d , then J s χ L 2 (R d ) ≤ √ C 0 χ/ √ ṽ L 2 (R d ) = (2π) d/4 √ C 0 χ H(R d )
, where we used Eq. (9.38)

. Second term, χ L 1 (R d ) < ∞, since χ L 1 (R d ) ≤ C 1 χ H(R d )
, via Eq. ( 9.39) (the constant C 1 is defined in the same equation) and we have seen already that χ H(R d ) is bounded. Third term, by a change of variable τ = ω/t, 9.39) and the fact that w H(R d ) = u H as recalled at the beginning of the proof. Finally, fourth term, for t ∈ R \ {0},

w t L 1 (R d ) = R d | w t (ω)|dω = R d |t| -d | w(ω/t)|dω = R d | w(τ )|dτ = w L 1 (R d ) , moreover w L 1 (R d ) ≤ C 1 w H(R d ) = C 1 u H via Eq. (
J s w t 2 L 2 (R d ) = R d ω 2s | w t (ω)| 2 dω = t -2d R d ω 2s | w(ω/t)| 2 dω = t 2s-d R d τ 2s | w(τ )| 2 dτ ≤ t 2s-d R d (1 + τ 2 ) s | w(τ )| 2 dτ = t 2s-d (2π) d/2 C 0 w 2 H(R d ) .
where we performed a change of variable ω = t τ , t d dτ = dω and used the definition in Eq. ( 9.38) and the fact that τ 2s ≤ (1 + τ 2 ) s for any τ ∈ R d . The proof of the bound of the fourth term is concluded by recalling that w H(R d ) = u H as discussed in the proof of the bound for the previous term.

Conclusion. Putting all our bounds together, we get :

∀t ∈ R \ {0}, h t H(R d ) ≤ (A + B t s-d/2 ) χ H(R d ) u H , where A = c 1 C 2 + cc 1 C 1 (2π) d/4 √ C 0 and B = cc 1 C 1 (2π) d/4 √ C 0
, where c = 2 max(s-1,0) , c 1 = 2 max(s/2-1,0) , while C 1 is defined in Eq. ( 9.39), C 2 in Eq. (9.40). Now define

∀x ∈ R d , g z,r (x) = 1 0 (1 -t)h t (x)dt,
and note that, by construction g z,r (x) = 1 0 (1 -t)u(tz + (1 -t)x)dt for any x ∈ B since u and χw coincide on B. Note that the map t ∈ (0, 1) → (1-t) h t H(R d ) is measurable, using the expression in Eq. (9.38). Moreover, since for all t ∈ (0, 1), it holds h

t H(R d ) ≤ (A+Bt s-d/2 ) χ H(R d ) u H ≤ (A + B) χ H(R d ) u H since s > d/2, the map t → (1 -t)h t is in integrable, and thus g z,r H(R d ) = 1 0 (1 -t)h t dt H(R d ) ≤ 1 0 |1 -t| h t H(R d ) dt ≤ (A + B) χ H(R d ) u H < ∞,
which implies that the function g z,r belongs to H(R d ). Finally, denote by R Ω : H(R d ) → H the restriction operator (see proposition 9.8 for more details). By construction (R Ω g)(x) = g(x) for any g ∈ H(R d ) and x ∈ Ω, defining g z,r = R Ω g z,r the lemma is proven.

9.E Proofs for algorithm 6

We start with two technical lemmas that will be used by the proofs in this section. Proof. Note that the function x → log(βx)

x is strictly decreasing on [exp(1)/β, +∞].

Moreover, 2α log(2βα) ≥ 2 log 4 ≥ exp(1)/2 ≥ exp(1)/β since β ≥ 2 and α ≥ 1. Now assume n ≥ cα with c = 2 log(2βα). It holds:

α log(βn) n ≤ log(βcα) c ≤ log( c 2 ) + log(2αβ) c ≤ 1 2 + 1 2 2 log(2βα) c ≤ 1,
where we used the definition of c and the fact that log(c/2)

≤ c/2 -1 ≤ c/2. Lemma 9.13. Let - → u ∈ S d-1 = {x ∈ R d | x = 1}, α ∈ [0, π/2],
x 0 ∈ R d and t > 0. Define the cone centered at x 0 , directed by -→ u of radius t with aperture α:

C α x 0 , -→ u ,t = x ∈ B t (x 0 ) | x-x 0 x-x 0 • - → u ≤ cos(α), x = x 0 ,
where we denoted by • the scalar product among vectors. Then the volume of this cone is lower bounded as

vol(C α x 0 , -→ u ,t ) ≥ ( √ π sin(α)) d-1 (t cos α) d dΓ((d + 1)/2) . Moreover, let x 0 ∈ R d and r > 0. Let x ∈ B r (x 0 ) and 0 < t ≤ r. The intersection B t (x) ∩ B r (x 0 ) contains the cone C π/3
x, -→ u ,t , where -→ u = x-x 0 x-x 0 if x = x 0 and any unit vector otherwise.

Proof. 1. Bound on the volume of the cone. Without loss of generality, assume x 0 = 0 and -→ u = e 1 since the Lebesgue measure is invariant by translations and rotations. A simple change of variable also shows that vol(C α 0, -→ u ,t ) = t d vol(C α 0, -→ u ,1 ). Now note the following inclusion (the proof is trivial):

C := x = (x 1 , z) ∈ R d = R × R d-1 : z ≤ cos(α), z R d-1 ≤ x 1 sin(α) ⊂ C α 0,e 1 ,1 .
It is possible to compute the volume of the left hand term explicitly :

vol( C) = R 1 x 1 ≤cos(α) R d-1 1 z ≤x 1 sin(α) dz dx 1 = cos(α) 0 V d-1 (sin αx 1 ) d-1 dx = V d-1 sin d-1 (α) cos d (α) d ,
where V d-1 = π (d-1)/2 /Γ((d -1)/2 + 1) denotes the volume of the d -1 dimensional ball.

Proof of the second point

The case where x = x 0 is trivial since t ≤ r. Assume therefore x = x 0 and note that by definition, C π/3

x, -→ u ,t ⊂ B t (x). We will now show that

C π/3 x, -→ u ,t ⊂ B r (x 0 ). Let y ∈ C π/3
x, -→ u ,t and assume y = x (if y = x then y ∈ B r (x 0 )). Expanding the dot product

y -x 0 2 = y -x 2 + 2(y -x) • (x -x 0 ) + x -x 0 2 = y -x 2 -2 y -x x 0 -x y-x y-x • - → u + x -x 0 2 ≤ x -y 2 -x -y x -x 0 + x -x 0 2 ,
where the last inequality comes from the definition of the cone and cos π/3 = 1 2 . Let us distinguish two cases:

• if t > x 0 -x , we havexy x 0 -x ≤ -t 2 and hence yx 0 2 ≤ t 2 ≤ r 2 ;

• otherwise xy ≤ t ≤ x 0 -x and thus yx 0 2 ≤ xx 0 2 ≤ r 2 .

In any case, y ∈ B r (x 0 ), which concludes the proof.

9.E .1 Proof of Lemma 9.4 Proof of Lemma 9.4. Fix Ω as in Lemma 9.4. Let U be the uniform probability over Ω, i.e., U (A) = vol(A∩Ω) vol(Ω) for any Borel-measurable set A. Let P = U ⊗n over Ω n . Throughout this proof, we will use the notation V d to denote the volume of the d-dimensional unit ball (recall that

V d = π d/2 Γ(d/2+1) ).
Step

|Y | t/2 (t/2) d V d = vol ∪ y∈Y t B t (y) ≤ vol(B R+t/2 (x 0 )) = (R + t/2) d V d .
Step 2. Probabilistic analysis. Note that for any (x 1 , ..., x n ) ∈ Ω n , writing X = {x 1 , .., x n }, it holds:

h X,Ω = max x∈Ω min i∈[n] x -x i = max x∈Xt max x∈Bt(x)∩Ω min i∈[n] x -x i ≤ t + max x∈Xt min i∈[n]
xx i .

Define E to be the following event :

E = {(x 1 , . . . , x n ) ∈ Ω n | max j∈[m] min i∈[n]
x j -x i < t}.

The n tuple (x 1 , .., x n ) belongs to E if for each x ∈ X t there exists at least one i ∈ [n] for which xx i < t. E can therefore be rewritten as follows :

E = x∈Xt i∈[n] {(x 1 , . . . , x n ) ∈ Ω n | x -x i < t}.
In particular, note that

E c = Ω n \ E = x∈Xt i∈[n] {(x 1 , . . . , x n ) ∈ Ω n | x -x i ≥ t} = x∈Xt (Ω \ B t (x)) n .
Applying a union bound, we get

P(E c ) = P x∈Xt (Ω \ B t (x)) n ≤ x∈Xt P (Ω \ B t (x)) n = j∈[m] U (Ω \ B t (x))) n ,
where the last step is due to the fact that P is a product measure and so P(

A n ) = U ⊗n (A n ) = U (A) n . Now we need to evaluate U (Ω \ B t (x)) = 1 -U (B t (x)) for x ∈ X t . Since X t ⊂ Ω, it holds ∀x ∈ X t , U (B t (x)) = vol(Bt(x)∩Ω) vol(Ω)
≥ min x∈Ω vol(Bt(x)∩Ω) vol(Ω)

.

Step

(Ω ∩ B t (x)) ≥ vol(B r (z) ∩ B t (x)) ≥ vol(C x,z,t ) ≥ 2V d-1 √ 3d √ 3 4 d t d .
Step 4. Expressing t with respect to n and δ and guaranteeing that t ≤ r. To conclude,

let C = V d-1 2d vol(Ω) √ 3 4 d-1
. Since N t ≤ (1 + 2R/t) d , and (1 -c) x ≤ e -cx for any x ≥ 0 and c ∈ [0, 1], then

P(E) ≥ 1 -N t 1 -Ct d n ≥ 1 -e -Ct d n+d log(1+2R/t) ≥ 1 -δ,
where the last step is obtained by setting

t = (Cn) -1/d log (1 + 2R(Cn) 1/d ) d δ 1/d .
Then h X,Ω ≤ 2t with probability at least 1 -δ, when t ≤ r. The desired result is obtained by further bounding C and t as follows.

Bounding C. It holds

2V d-1 √ 3dV d = 4 3d 2 π 1/2 Γ(d/2+1) Γ(d/2+1/2) . Using Gautschi's inequality and the fact that d ≥ 1, 2 3dπ 1/2 ≤ 2V d-1 √ 3dV d ≤ 2(d+2) 3d 2 π 1/2 ≤ 1. Since 3dπ 2 1/2d 4 √ 3 ≤ 2 √ 2π for all d ≥ 1, and since V d r d ≤ vol(Ω) ≤ V d R d , it holds (2 √ 2πR) -d ≤ C ≤ (4r/ √ 3) -d =⇒ n 1/d 2 √ 2πR ≤ (Cn) 1/d ≤ √ 3n 1/d 4r ≤ n 1/d 2r . Bounding t. Since, (1 + x) d ≤ (2x) d for any x ≥ 1 and 2R(Cn) 1/d ≤ R r n 1/d , and R r n 1/d ≥ 1, it holds t ≤ 2 √ 2πRn -1/d (log n δ + d log 2R r ) 1/d .
Guaranteeing t ≤ r. Applying Lemma 9.12 to α = (2π) 9.E .2 Proof of Theorem 9.6

Proof. Recall that s > d/2 and m < sd 2 is a positive integer. Assume that Ω satisfies Assumption 9.1(a) for a certain r and that the diameter of Ω is bounded by 2R. In particular, if Ω is a ball of radius R, then Ω satisfies Assumption 9.1(a) with r = R. In the first step of the proof we guarantee that n is large enough to apply Lemma 9.4 and that h X,Ω , controlled by Lemma 9.4, satisfies the assumptions of Theorem 9.5. Then we apply Theorem 9.5.

Step 1. Guaranteeing n large enough and h X,Ω ≤ r/(18(m -1) 2 ). Applying Lemma 9.12 to

α = 2R r d max(3, 10(m -1)) 2d and β = (2R) d r d δ , it holds that if n ≥ 2α log(2αβ) = 2R r d max(3, 10(m -1)) 2d 2 log 2 δ + 4d log R r max(6, 20(m -1)) , then α/n log(βn) ≤ 1, which implies n -1/d (log n δ + d log β) 1/d ≤ r 2R max(3, 10(m -1)) 2 .
In particular, n satisfying the condition above is large enough to satisfy the requirement of Lemma 9.4 (since r ≤ R). Therefore, by applying Lemma 9.4 we have that with probability at least

1 -δ, h X,Ω ≤ 11R n -1 d (log (2R) d n r d δ ) 1/d ≤ r max(1, 18(m -1) 2 ) .
Step 2. Applying Theorem 9.5. In the previous step we provided a condition on n such that h X,Ω satisfies h X,Ω ≤ r max(1,18(m-1) 2 ) . By proposition 9.1, Assumption 9.2 holds for the Sobolev kernel with smoothness s, for any m ∈ N since m < sd/2. Then the conditions to apply Theorem 9.5 are satisfied. Applying Theorem 9.5 with λ ≥ 2η max(1, MD m ) and η = 3 max(1,18(m-1) 2 ) m d m (9.42) Thus, taking β = min( β 1 2 , 2(m-f * )

f (x) -f * = f (x) -m + m -f * ≥ m -f * = 2(m -f * ) 2R 2 R 2 ≥ 2(m -f * ) 2R 2 x -ζ 2 .
R 2

) and combining Eqs. (9.41) and ( 9.42), it holds

∀x ∈ Ω, f (x) -f * ≥ β 2 x -ζ 2 .
9.F .2 Proof of Theorem 9.7

Proof. Let us divide the proof into four steps.

Step 1: Extending the parabola outside of Ω Since Ω is an open set containing ζ, there exists t > 0 such that the ball B t (ζ) ⊂ Ω. Define δ = β-ν 2 t 2 . It holds : Step 2. L ∞ bound due to the scattered zeros. Note that the solution (ĉ, ẑ, B) satisfies ĝ(x i ) = Φ i BΦ i for i ∈ [n], where the function ĝ is defined as ĝ(x) = f (x) -ν 2 x 2 + νx ẑĉ for x ∈ Ω, moreover h X,Ω ≤ 9.G Proofs for the extensions 9.G .1 Proof of Theorem 9.9

∀x ∈ R d \ Ω, β 2 x -ζ 2 ≥ ν 2 x -ζ 2 + δ. ( 9
Proof. Let (ĉ, B) be a minimum trace-norm solution of Eq. (9.5). The minimum p λ,n of Eq. (9.5) then corresponds to p λ,n = ĉλ Tr( B). Combining Eq. ( 9.27) with Eq. (9.17) from the proof of Theorem 9.5 and the fact that θ 2 ≤ λ/8, we have that The desired result is obtained by combining Eq. ( 9.50) and Eq. ( 9.49) as we did in Step 3 of Theorem 9.5. Note that this norm is well defined since for any g ∈ H, since there exists f ∈ C s (R d ) such that g = f | Ω , since all the derivatives of f are continuous hence bounded on Ω which is bounded, so are all the derivatives of g. (1 -t)u(z + t(xz))dt. v z is in C s (R d ) by dominated convergence, and v z = v| Ω satisfies the desired property (in this case, there is no need to depend on r and one can simply take g r,z = v z ).

Now note that

Moreover, if f ∈ C s+2 (R d ), then in particular, for any i, j ∈ [d], ∂f ∂x i ∂x j ∈ C s (R d ) and hence its restriction to Ω is in H. Moreover, in that case, it is obvious that since s ≥ 0, f | Ω ∈ H. This shows that f satisfies Assumptions 9.1(b) and 9.3. Therefore, Theorem 9.2 can be applied, and there exist w1 , . . . , wp H, p ∈ N + , such that

∀x ∈ Ω, f (x) -f * = j∈[p] w 2 j (x).
By definition of H, taking w 1 , ..., w p such that w j | Ω = wj , the corollary holds.

9.G .3 Proof of Theorem 9.10

Proof. In this proof we will use the results recalled in Sec. 9.A .2 about Sobolev spaces. By Cor. 9.2 we have that there exists w 1 , . . . , w p ∈ C s (R d ) such that f (x) = j∈[p] w 2 j (x) for any x ∈ Ω. Define w j = w j | Ω . Note that by proposition 9.5, w j ∈ W s ∞ (Ω) for j ∈ [p]. Now let ε ∈ (0, 1], for any j ∈ [p], let w ε j ∈ C ∞ (Ω) be the ε approximation of w j as defined in proposition 9.4, i.e., w j = w j | Ω where w j ∈ C ∞ (R d ) and w j -w ε j L ∞ (Ω) (2 w j L ∞ (Ω) + w j -w ε j L ∞ (Ω) )

w j -w ε j L ∞ (Ω) ≤ C 1 ε s w j W s ∞ (Ω) , w ε j W r ∞ (Ω) ≤ C 2 ε s-r w j W s ∞ (Ω) . ( 9
≤ j∈[p] C 1 ε s w j W s ∞ (Ω) (2 w j W s ∞ (Ω) + C 1 ε s w j W s ∞ (Ω) ) ≤ ε s p C 1 (2 + C 1 ) max j∈[p] w j 2 W s ∞ (Ω) ,
where we use the first equation of Eq. ( 9.51) to go from the second to the third line.

Recall that H is defined to be a RKHS associated to the Sobolev kernel k r defined in Example 9.1 for a given r > max(s, d/2). As mentioned in Example 9.1, we have H = W r 2 (Ω), and • H is equivalent to • W r 2 (Ω) , i.e., there exists C 4 depending on Ω, r, d such that 1

C 4 • W r 2 (Ω) ≤ • H ≤ C 4 • W r
2 (Ω) . Since w ε j ∈ W r 2 (Ω) = H by Eq. ( 9.51) for all j ∈ [p], we can define :

A ε = j∈[p]
w ε j ⊗ H w ε j .

It holds :

Tr(A ε ) ≤ p max where we used Eq. ( 9.51) and the fact that there exists C 5 such that

• W r ∞ (Ω) ≤ C 5 • W r 2 (Ω)
(see proposition 9.5). To conclude, we use proposition 9.6 to bound • W s ∞ (Ω) ≤ C 6 • Ω,s . 9.G .4 Proof of Theorem 9.11 Proof. The proof of the existence of a minimizer corresponds essentially to the first part of the proof of Theorem 9.5 and we skip it. Let ε ∈ (0, 1], by applying Theorem 9.10 to f we know that there exists A ε ∈ S + (H) satisfying Eq. (9.33). Define f ε = φ(x), A ε φ(x) for all x ∈ Ω, by Theorem 9.10 we have -s) , sup Proof. We divide the proof in two steps

Tr(A ε ) ≤ C f -2(r
Step 1. First note that ĝ(x) := f (x) -ν 2 x 2 + νx ẑĉ = f (x) -ν 2 xẑ 2 -f . Step 2. Now to obtain Eq. ( 9.53) we need to do a slightly different construction. Let u j (x) = e j (xẑ) for any x ∈ Ω. Note that since u j is the restriction to Ω of a C ∞ function on R d , by Assumption 9.2(a), u j ∈ H. Moreover, note that ν 2 xẑ 2 = ν 2 d j=1 u j (x) 2 . Take ûj ∈ R n defined as ûj = V * u j and note that Φ i ûj = V φ(x i ), V * u j = V * V φ(x i ), u j = P φ(x i ), u j = u j (x i ).

Then, defining Ĝ = ν 2 . In particular, some basic linear algebra leads to

Tr( Ĝ) = ν 2 R -(X -1 n ẑ ) 2 .

9.H Details on the algorithmic setup used in the benchmark experiments

In this section, we explain exactly the algorithmic setup which we used to perform the experiments in Section Sec. 9.10 .1. In all the following problems, the set Ω on which we will minimize the function will be a hyper-rectangle. Given a hyper-rectangle R, we will identify it with its center c R ∈ R d and its width w R ∈ R d , such that R = d i=1 ((c R ) i -(w R ) i /2, (c R ) i + (w R ) i /2). We start by defining algorithm 7 whose main goal is to find a global minimizer as described in the previous sections given sample points (x 1 , ..., x n ). However, the kernel k, and the hyper-parameters λ also have to be chosen.

Therefore, algorithm 7 will use as inputs :

• The function f to minimize;

• the evaluation points x i , 1 ≤ i ≤ n, summarized in a matrix X ∈ R n×d ;

• the kernel k;

• two parameters λ min and λ max such that we can choose λ in [λ min , λ max ];

• The paramter ε, which controls the log barrier.

For simplicity, we hide the hyperparameters linked to the solving of Eq. ( 9.55) through a Newton method, as explained in Sec. 9.6 . algorithm 7 automatically selects the hyperparameter λ by minimizing the function wich to λ associates the function value of the resulting x on a log scale (ScalarFunction). This function is minimized in the range [λ min , λ max ] through the function MinimizeScalar. Hence, the number of function evaluations inherent to running this algorithm is n + n min where n min is the number of In our experiments, we use ε = 10 -3 , λ min = 10 -12 , λ max = 1 and we use either the Brent method or simply a grid search with a maximum number of 100 points. This minimization does not have to be very precise.

The full algorithm we use is an iterative scheme and is written down in algorithm 8, computing a sequence (x k ) of approximations of a minimizer of f by iteratively reducing the size of the hyper-rectangle from which the points used in algorithm 7 are sampled.

More precisely, we start from points sampled from a hyper-rectangle with center x 0 = c Ω and with width w 0 = w Ω (that is the hyper-rectangle Ω) to form m -1 samples which, together with x 0 , form the points X 0 ∈ R m×d used to compute the first approximation of the minimizer using FindMinimizer : x 1 . Then at each step k, we use the last approximation of the minimizer x k as the new center of the hyper-rectangle, with width w k which is set through the predefined function Contraction as w k = Contraction(k)w 0 . As for the first iteration, we then form the concatenation X k ∈ R m×d of m -1 samples from this hyper-rectangle plus x k . In order to keep track of the previous points (as a kind of momentum), we apply FindMinimizer with

X k = [ X k , X k-1 , X k-2 ]
, that is keeping the two last set of points as well as the ones sampled for the k-th step. • a kernel function x, x , σ → k σ (x, x ) such that σ is a parameter to adapt to the the typical width of the data;

• the initial hyper-rectangle Ω;

• the function f ;

• the contraction function Contraction to set the width of the successive hyper-rectangles;

• the number m of new points sampled and used at each iteration;

• the number N of iterations.

In our implementation, we use the following parameters.

• For σ > 0 and x, y ∈ R d , we will use the following kernel, which is a mix between the Gaussian (very regular functions) and the Abel kernel (Sobolev functions of order s = (d + 1)/2 functions), plus a small term 0.01 which allows to handle the constant component of a function more easily.

k σ (x, y) = 0.01 + exp(-xy 2 /(2σ 2 )) + exp(-xy /σ). (9.57)

• We will use the following contraction function, which depends on the dimension as well as the number of iterations :

Contraction(k) = max 1 + 1 d -k
, 1 1+k 0.6 . (9.58)

• The number N of iterations will be set to N = 200 unless stated otherwise.

• m will be specified in the experiments : indeed, the higher the dimension, the larger m has to be in order to get meaningful results. Note that one actually uses n = 3m points (from the third iteration onwards) to form the optimization problem, hence the dimension of the SDP solved with the Newton method will be 3m.

Remark 30. It is equivalent to minimize a function f and minimize the function f f +c for a positive constant c. This allows to minimize a function in [0, 1] instead of minimizing a real-valued function : however, this also makes higher derivatives behave differently than those of the original function. In practice, instead of minimizing f directly, we minimize f f +c , where c is chosen such that f f +c will be spread evenly over [0, 1], typically by selecting c as a quantile of the (f (x i )) 1≤i≤n (we choose the 0.25 quantile). We performed experiments by comparing this renormalization method with simply minimizing f , and this yields slightly better results. The importance of preserving regularity

9.H .1 Additional experiments for global optimization

In this work, we state sufficient conditions for a non-negative function f to be written as a sum of squares of functions f i . Of course, if no other constraints are added, this is a trivial problem as writing f = ( √ f ) 2 would offer an immediate solution. What we want to understand here are sufficient conditions which allow to inherit a form of regularity of the function f in the sum of squares decomposition. This is not necessarily the case when taking the square root: for example, the map (x, y) → x 2 + y 2 is smooth and a sum of smooth squares, but its square root is not differentiable at (0, 0).

In general, being able to decompose the function with a certain regularity is important. Of course, there is a complex interaction between the structural constraint of being a sum of squares and the original regularity of the function, and the two may not work very well together (see Theorem 10.4). However, for certain theoretical and applied problems, it is crucial to maintain some regularity. For example, in the setting introduced by Rudi, Marteau-Ferey, and Bach (2020), the speed of convergence of the presented algorithm of global optimization depends on the regularity of the sum of squares representation of the function ff * (where f * is the global minimum of f ). In polynomial sum-of-squares (SoS) optimization, the running time depends on the degree in the sum of squares decomposition of P -P * .

Abstractly, we can formulate the following generic question. If f ∈ C 1 where C 1 describes a form of regularity, can we write f as a sum of squares of functions of class C 2 , where C 2 inherits the regularity properties C 1 as much as possible?

Problem setting

In this work, we will concentrate on the class C p of p times differentiable functions with continuous p-th derivatives on R d (or any d-dimensional manifolds M ). For simplicity, in this introduction, we will state the main results for functions on R d . We will show that under a certain condition on the set of zeros Z of f , if f is a C p non-negative function on R d , it can be decomposed as

f = i∈I f 2 i , f i ∈ C p-2 (R d ), (10.1)
where (f i ) is an at most countable family and has locally finite support. Two elements are important in Eq. ( 10.1): the locally finite aspect and the regularity of the functions f i , i.e., p -2. This is a consequence of the fact that we will consider second order sufficient conditions, hence the loss of two derivatives.

.1 Intuition and previous results

Let us give an intuition as to how we obtain decompositions in the form Eq. (10.1). First, we start by proving that this decomposition holds locally in a neighborhood of any x 0 ∈ R d . It is then possible to invoke a result to "glue" the local decompositions together; we develop the tools to do so in Sec. 10.3 .2 (note that this is one of the key differences between results for polynomials and results for functions). For any fixed x 0 ∈ R d , if f (x 0 ) > 0, then f 1 := √ f is well defined and of class C p around x 0 , and so Eq. (10.1) holds locally around x 0 since f = f 2 1 . The crux of the problem is to determine whether f can be decomposed as a sum of squares around a point in the set of zeros Z of f , i.e., the set of points x such that f (x) = 0. Since f is non-negative, all such points are necessarily minimizers of f , hence the following necessary second-order condition: ∀x 0 ∈ Z, ∇f (x 0 ) = 0, ∇ 2 f (x 0 ) 0. (10.2)

Around any x 0 ∈ Z, f can be approximated by a parabola since the eigenvalues of ∇ 2 f (x 0 ) are non-negative: f (x) = x ∇ 2 f (x 0 )x + o( x 2 ) using a Taylor expansion. Since any parabola can be written as the sum of at most d squares of linear functions (just write the eigen-decomposition of ∇ 2 f (x 0 )), we see that up to the o( x 2 ) factor, we can indeed write f as a sum of at most d squares around x 0 . The whole difficulty of the following results is to go beyond this o( x 2 ) approximation and have an exact decomposition, using the Taylor expansion with integral remainder.

It turns out that in the case where ∇ 2 f (x 0 ) 0, that is when the Hessian has strictly positive eigenvalues, this decomposition can be made exact. We will call this condition the strict Hessian condition (SHC) at x 0 . This result exists in recent work: it is a particular case of Theorem 2 by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF], applied to the set H = C p-2 . Precisely, it states Theorem 10.1 (Rudi, Marteau-Ferey, and Bach (2020, Theorem 2)). Let f be a non-negative function of class C p for p ≥ 2, and assume that the zeros Z of f satisfy the strict Hessian condition: ∀x 0 ∈ Z, ∇ 2 f (x 0 ) 0. (10.3)

If f has a finite number m = |Z| of zeros, then f satisfies Eq. (10.1) with dm + 1 functions f i .

This situation is illustrated on the left hand side of Sec. 10.1 .1, where the Hessian is positive definite at all four zeros of f and hence satisfies the SHC: by Theorem 10.1, it can be decomposed as a sum of squares. It is not the case on the right hand side, where there is a continuous subspace of zeros: in that case, f does not satisfy the SHC.

Contribution

While the SHC condition Eq. ( 10.3) already offers a nice result in Theorem 10.1, we see that there is a big difference with the necessary condition Eq. (10.2). Previous results in the literature show that Eq. (10.2) is not sufficient to be decomposed as a sum of squares of C p-2 functions as soon as the dimension d is greater than 3 (see Theorem 10.4 in the background section form more details). On the other hand, Eq. ( 10.3) is very restrictive. In particular, it implies that the set Z of zeros is discrete. However, in some situations [START_REF] Vacher | A Dimension-free Computational Upper-bound for Smooth Optimal Transport Estimation[END_REF], the set of zeros has a natural structure, which can be a sub-manifold of R d (consider for instance the extreme case where f = 0). In this paper, we show that if the set Z of zeros is a sub-manifold of R d such that the Hessian of f along this manifold is positive along all directions which are not tangent to Z, then Eq. (10.1) still holds. This is the case for the function depicted in the right hand side of Sec. 10.1 .1, and illustrates the difference between previous works and our contributions. More formally, we prove the following result.

Theorem 10.2. Let f be a non-negative function of class C p for p ≥ 2 and let Z denote the set of zeros of f . If Z is a sub-manifold of R d of class C 1 such that ∀x 0 ∈ Z, ∀h ∈ R d \ T x 0 Z, h ∇ 2 f (x 0 )h > 0, (10.4) then f satisfies Eq. (10.1), and Z is of class C p-1 . Here, T x 0 Z denotes the tangent space to Z at x 0 , which is a vector sub-space of R d . This theorem is proved as Theorem 10.6 in Sec. 10.2 , and the assumption Eq. ( 10.4) will be referred to as the normal Hessian condition (or NHC). Note that the NHC assumption encompasses that of the SHC assumption of Theorem 10.1; in that case, the results presented in this paper make the result tighter by removing the assumption that Z be finite and by needing only d + 1 squares to represent the function, and not d|Z| + 1 (see the full version of Theorem 10.6).

The proof techniques used to prove this theorem differ from the proof by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] and use tools from differential geometry and Morse theory. In particular, the proof extends naturally to functions defined on d-dimensional manifolds, which is the object of Sec. 10.3 and Theorem 10.8. This opens the way to new problems, wich are more naturally defined on standard manifolds like the d-dimensional sphere S d or the d-dimensional torus

T d ≈ (S 1 ) d .

Background

The problem of decomposing C p functions as sums of squares has appeared in the context of symbolic calculus, in the proof of the Fefferman-Phong inequality, which is an important regularity result for partial differential operators (see the original article by [START_REF] Fefferman | On positivity of pseudo-differential operators[END_REF], and the article by Bony (1998Bony ( -1999) ) for the link with sum of squares decompositions, as well as the monograph by [START_REF] Tataru | On the fefferman-phong inequality and related problems[END_REF]). In this context, the following result is proved (with C k,1 loc denoting the set of k times differentiable functions with locally Lispchitz k-th derivative):

Theorem 10.3 [START_REF] Fefferman | On positivity of pseudo-differential operators[END_REF], Bony, Broglia, Colombini, and Pernazza (2006, Theorem 1.1)). Let Ω be an open set of R d , d ≥ 1 and f ∈ C 3,1 loc (Ω) be a non-negative function. Then f can be written as a finite sum of squares of C 1,1 loc (Ω) functions. In the context of preserving regularity, a natural question which arises is whether increasing the regularity of f can increase the regularity of the functions in a sum of square decomposition. [START_REF] Bony | Nonnegative functions as squares or sums of squares[END_REF]; [START_REF] Bony | Sommes de carrés de fonctions dérivables[END_REF] show that the general answer (under no further assumptions) is negative. More precisely, if f is a function defined on a neighborhood of 0, a local decomposition of f around 0 of class C is a finite family (f i ) i∈I of functions of class C defined on an open neighborhood U of 0 such that i∈I f 2 i = f on U . Theorem 10.4 (Bony, Broglia, Colombini, and Pernazza (2006, Theorem 2.1)). In all the following cases, there exists f ∈ C ∞ defined on an open neighborhood of 0 in R d such that the following holds:

• if d ≥ 4, f has no local decomposition of class C 2 ;

• if d = 3, f has no local decomposition of class C 3 .

The case d = 1 is explored by [START_REF] Bony | Sommes de carrés de fonctions dérivables[END_REF]: it is shown in Theorem 1 that if f is of class C 2m for m finite, then f can be written as the sum of squares of two functions of class C m . Moreover, this is shown to be tight: there exists a function f ∈ C 2m with no local decomposition as a sum of squares of functions of class C m+k , for k ≥ 1. The case d = 2 has been explored less in the literature (some results exist when dealing with flat minima, see for example Theorem 2 by [START_REF] Bony | Sommes de carrés de fonctions dérivables[END_REF]).

To summarize, these results show that without additional assumptions, as soon as the dimension is greater than 3, inheriting the C p regularity properties of the function f in the sum of squares decomposition is not possible in a satisfactory way, and motivates the introduction of additional geometric assumptions.

Polynomials Decomposing non-negative polynomials as sums of squares has been related to important problems in algebraic geometry during the 20th century. In 1927, on his way to the resolution of Hilbert's 17th problem, [START_REF] Artin | Über die Zerlegung definiter Funktionen in Quadrate[END_REF] proved that any non-negative polynomial is a sum of squares of rational functions (that is formal fractions of polynomials P (x)/Q(x)). Moreover, Hilbert had earlier proved that there exist non-negative polynomials which cannot be written as sum of squares of polynomials by Hilbert (1888) (for more than 3 variables and with degree at least 6 for example). In algebraic geometry, the set of SoS polynomials has very interesting properties, and finding sufficient conditions for a non-negative polynomial or even a positive polynomial to be a sum of squares is an important question. More generally, one usually wishes to understand under which sufficient conditions a polynomial P which is non-negative (or positive) on an algebraic set, i.e., defined by polynomial inequalities of the form Q i ≥ 0 for polynomials Q i , can be written in the form P = P 0 + N i=1 P i Q i where the P i are SoS. The theoretical literature regroups these results under the name "Positivstellensatz". The most often seen in the SoS optmization literature are the [START_REF] Stengle | A Nullstellensatz and a Positivstellensatz in Semialgebraic Geometry[END_REF]; [START_REF] Schmügden | The k-moment problem for compact semi-algebraic sets[END_REF]; [START_REF] Putinar | Positive Polynomials on Compact Semi-algebraic Sets[END_REF] Positivstellensätzen.

If these algebraic geometry considerations seem far from applications and from decomposing smooth functions as sums of squares (indeed, polynomials are much more rigid than smooth functions) at first glance, they are actually related in two ways.

First, as smooth functions can be locally approximated by polynomials, results on polynomials give a good intuition of the difficulties one can encounter at the local level when decomposing a function as a sum of squares. Indeed, on the one hand, the general impossibility results proved by [START_REF] Bony | Sommes de carrés de fonctions dérivables[END_REF]; [START_REF] Bony | Nonnegative functions as squares or sums of squares[END_REF] (see Theorem 10.4) are obtained using Hilbert's theorem on the existence of non-negative polynomials which are not sum of squares. On the other hand, the fact that there is hope using our second-order assumptions is also due to the fact that second order non-negative polynomials can always be written as sums of squares.

Second, the certificates given by Positivstellensatz on the decomposability of certain non-negative polynomials can be algorithmically checked in some cases, using semi-definite programming. This has paved the way to so-called SoS hierarchies, and optimization of polynomial objective functions with polynomial constraints. These have been developed by Lasserre [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF] (based on the Postivstellensatz by [START_REF] Putinar | Positive Polynomials on Compact Semi-algebraic Sets[END_REF]) and [START_REF] Parrilo | Semidefinite programming relaxations for semialgebraic problems[END_REF] (based on the Positivstellensatz by [START_REF] Stengle | A Nullstellensatz and a Positivstellensatz in Semialgebraic Geometry[END_REF]; [START_REF] Schmügden | The k-moment problem for compact semi-algebraic sets[END_REF]). Using these theoretical results, they can provide certificates of lower bounds for certain optimization problems (or upper bound in the dual "moment problem", see the work by [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF]). Moreover, to have more interpretable results for these more applied settings, theses works have motivated more practical Positivstellensatz, like that by [START_REF] Marshall | Representations of non-negative polynomials having finitely many zeros[END_REF], which provides a condition for writing a polynomial with a finite set of zeros as as sum of squares (this condition is actually a second order condition which greatly resembles ours in the polynomial setting, although it deals more with the constraints Q i ).

p-times differentiable functions In the same spirit as the polynomial hierarchies, recent works by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]; [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]; [START_REF] Rudi | Psd representations for effective probability models[END_REF] have developed models and methods based on sum of squares of regular functions. The computational properties of these methods are based on the fact that regular functions can be well-approximated by functions of the form i α i k(•, x i ) where k is a so-called positive definite kernel [START_REF] Aronszajn | Theory of reproducing kernels[END_REF] and can be adapted to the regularity. In order to obtain guarantees on these methods, it is crucial to have the equivalent of Positivstellensatz in the case of regular functions. Contrary to the case of algebraic geometry, where such results existed for other purposes, there is a need to build such results for regular functions from scratch. Certain results like Theorem 10.1 have been presented. However, the aim of the present paper is to provide more general results, to be used in most situations.

Organisation of the work

In Sec. 10.2 , we formalize the different notions needed to state Theorem 10.2 in the case of non-negative functions defined on open sets of R d . In particular, we start by presenting a local decomposition in Theorem 10.5, which will be the cornerstone of the work. In Sec. 10.3 , we extend Theorem 10.2 to the manifold setting, and detail the procedure in which we glue local decompositions into a global one, using traditional tools from differential geometry. In Sec. 10.4 , we formally prove Theorem 10.5. We finish by a discussion on the result presented in this paper, as well as possible extensions in Sec. 10.5 .

Decomposition under second order conditions (Euclidean case)

In this section, we present our results on decomposing a C p function f as a sum of squares of C p-2 functions on open sets of R d .We start with a brief presentation of the notion of sub-manifold of R d in Sec. 10.2 .1. It is the key geometric object we use to represent the set of zeros Z of the function f . In Sec. 10.2 .2, we present the cornerstone result of this paper in Theorem 10.5, as well as a sketch of its proof, which is done extensively in Sec. 10.4 . It shows that as soon as a non negative function has positive Hessian in the orthogonal direction to its zeros at a given point, then it can be decomposed as a sum of squares around that point. Finally, in Sec. 10.2 .3, we present Theorem 10.6, which shows that given a function defined on an open subset Ω of the Euclidean space R d , and under conditions on the Hessian of f at its zeros, f can be decomposed as a locally finite sum of squares of functions defined on Ω.

Definitions and notations

In general, given two topological sets M and N as well as x 0 ∈ M and y 0 ∈ N , we will say that φ : (x 0 , M ) → (y 0 , N ) is a local map satisfying a property (P ) if there exists an open neighborhood U of x 0 in M such that φ : U → N is well defined, satisfies φ(x 0 ) = y 0 and property (P ). We will say that φ : U ⊂ R d → R e defined on an open set U is of class C k if it is k times differentiable, 10.2 . DECOMPOSITION UNDER SECOND ORDER CONDITIONS (EUCLIDEAN CASE)459

and its derivatives of order k are continuous. For any function φ : (x, R d ) → R e of class C 1 , we denote with dφ(x) its differential at x. It is an element of Hom(R d , R e ) the set of linear maps from R d to R e . We will write dφ(x)ξ or dφ(x)[ξ] the evaluation of dφ(x) at ξ. The Jacobian of φ at x is the matrix J φ (x) ∈ R e×d which is the matrix of dφ(x) in the canonical bases. Writing the coordinates of φ: φ = (φ 1 , ..., φ e ), we have [J φ ] ij = ∂φ i ∂x j (x).

10.2 .1 Sub-manifolds of R d

One of the main assumptions in order to achieve our results will be that the set of zeros of the non-negative function f is a sub-manifold of R d . In this section, we restrict ourselves to introducing the definitions and results needed to state and prove those in this paper. For a more comprehensive introduction, see chapter 1 by [START_REF] Lafontaine | An introduction to differential manifolds[END_REF], section 2.2 by [START_REF] Paulin | Géométrie différentielle élémentaire[END_REF] (in French) or [START_REF] Spivak | A Comprehensive Introduction to Differential Geometry[END_REF]. The notion of sub-manifold generalizes the notion of a curve in R d (a one dimensional manifold) or a surface in R d (a two dimensional manifold). Intuitively, a sub-manifold N is a subset of R d such that at each point x ∈ N , N "looks like" R d 0 where d 0 is the dimension of the sub-manifold at x (one for a line, two for a surface,etc.). Another way to put this is that N can be locally parametrized by R d 0 . To formalize this, we need the following definitions. We fix a subset N ⊂ R d .

A map φ : U → R d defined on an open neighborhood U of 0 in R d 0 is said to be a local parameterization of N around x 0 of class C k for k ≥ 1 if φ is of class C k , and if there exists an open set V ⊂ R d such that the following conditions are satisfied:

(i) φ(0) = x 0 , φ(U ) = N ∩ V , and φ : U → φ(U ) is a homeomorphism, i.e., it is bijective and has continuous inverse;

(ii) its differential at 0 is injective (one to one), i.e., dφ(t 0 ) ∈ Hom(R d 0 , R d ) is injective.

The second condition guarantees that the local dimension of N is indeed d 0 , that φ is not an over-parameterization. N is said to be a sub-manifold of R d and of class C k if there exists a local parameterization φ of class C k around each point x ∈ N . Given a point x ∈ N , the dimension d x of the local parametrization is independent of the parametrization (two local parametrizations will necessarily be of same dimension); it is called the dimension of N at x. Similarly, the subspace T x N := dφ(x)R dx , which is a subspace of R d of dimension d x is independent of the local parametrization: it is the linear approximation of N at x and is called the tangent space to N at x (see Sec. 10.2 .1 and Sec. 10.2 .2 for more visual representations).

A sub-manifold N of R d is said to be connected if it cannot be written as a union of disjoint open sets. Equivalently, it is connected if any two points in N can be connected by a continuous path γ : [0, 1] → N . On a connected sub-manifold N , the dimension d x is the same at every point x, it is called the dimension of the connected sub-manifold N . This implies that all the tangent spaces T x N have the same dimension. .2: Examples of sub-manifolds of R 2 ; points are denoted with pt. Left: connected sub-manifold of dimension 1 (a circle). Center: a sub-manifold of 4 connected components which are all points, i.e., of dimension 0 (their tangent space is not represented since it is reduced to {0}). Right: a sub-manifold of two connected components, one point pt of dimension 0 and one of dimension 1.

These results, their proof and their broader context can be found in chapter 1.5 by [START_REF] Lafontaine | An introduction to differential manifolds[END_REF]. In particular, Theorem 1.21 presents equivalent definitions of a sub-manifold. Section 2.2 by [START_REF] Paulin | Géométrie différentielle élémentaire[END_REF] is also a good reference (in French).

.2 Local decomposition as a sum of squares

In this section, f will always denote a non-negative function defined on an open set of R d . We will assume that f is of class C p for p ≥ 2. We will also denote with Z the set of zeros of f , i.e., the set of zeros of f . In this section, we will make local assumptions on the Hessian of f at points x ∈ Z such that the function f can be decomposed as a sum of squares locally around x.

We will denote with d 2 f (x) the second differential of f (which we will sometimes call abusively its Hessian), which is a symmetric bilinear form on R d . We denote with d 2 f (x)[ξ, η] its evaluation on vectors ξ, η. We denote with ∇ 2 f (x) ∈ R d×d the Hessian matrix of f at x, which is the matrix of d 2 f (x) in the canonical basis of R d , and we have d 2 f (x)[ξ, η] = η ∇ 2 f (x)ξ. For any vector sub-space space S ⊂ R d , and any bilinear form H on R d , we denote with H| S the restriction of H to S, which is a bilinear form on F . We say that a bilinear form H is positive semi-definite if H[ξ, ξ] ≥ 0 for any ξ ∈ R d , and is positive definite if H[ξ, ξ] > 0 for all ξ ∈ R d \ {0}. We use the same terminology for matrices.

We are now ready to state Theorem 10.5, which is the cornerstone of this work. For the rest of this section (Sec. 10.2 .2), let x 0 ∈ R d and f : (x 0 , R d ) → R be a non-negative C p function, for p ≥ 2, such that f (x 0 ) = 0. We claim that if there is a sub-manifold of class C 1 and of dimension d 0 around x 0 of zeros of f , and if the Hessian of f at x 0 has rank dd 0 (which we will call the normal Hessian condition), then it can be decomposed as a sum of squares as in Eq. (10.1).

Definition 10.1 (normal Hessian condition). Let Z denote the set of zeros of f . We say that f satisfies the normal Hessian condition (NHC) at x 0 if there exists a dimension 0 ≤ d 0 ≤ d and a sub-manifold N of class C k with k ≥ 1 and of dimension d 0 such that x 0 ∈ N ⊂ Z, and on one of the following equivalent conditions is satisfied: (i) the rank of ∇ 2 f (x 0 ) at x 0 is dd 0 ;

(ii) the restriction of d 2 f (x 0 ) to T x 0 N ⊥ is positive definite. Left: connected sub-manifold of dimension 2 (the sphere S 1 ). Right: a sub-manifold of two connected components, one of dimension 2 (homeomorphic to the torus T 2 on which lies x 0 ), and one of dimension 1 on which lies x 1 .

The complete proof of the equivalence of these conditions as well as the proof of Theorem 10.5 can be found in Sec. 10.4 . To illustrate the definition of the normal Hessian condition, we refer to Sec. 10.2 .2 which represents the local behavior of functions f defined locally around a point x 0 ∈ R 2 in the set of zeros and which satisfies the NHC for d 0 = 1. Main steps of the proof. The main steps of this proof are represented geometrically in Sec. 10.2 .2.

Step 1. We show that under the NHC at x 0 , we have T x 0 N = ker(∇ 2 f (x 0 )) and hence that d 2 f (x 0 )| Tx 0 N ⊥ is positive definite.

Step 2. Re-parametrizing f on a basis adapted to T x 0 N ⊥ ⊕ T x 0 N as f (x ⊥ , x ), we apply the Morse lemma (see Lemma 10.10), which decomposes the function f in the form f (x ⊥ , x ) = f (ϕ(x ), x ) + 1 2 d 2 f (x 0 )| Tx 0 N ⊥ [ξ(x ⊥ , x ), ξ(x ⊥ , x )], (10.6) for a certain function ϕ of class C p-1 and ξ of class C p-2 in a certain open set around x 0 (for an easy visualization, see Sec. 10.2 .2).

Step 3. We show that the second term of the right hand side of Eq. (10.6) can actually be seen as a sum of squares of dd 0 functions of class C p-2 .

Step 4. We characterize the manifold of zeros around x 0 . Step 5. We show that the first term of the result of the Morse lemma is equal to zero using the previous characterization, which shows Eq. (10.5).

Example 10.2 (case where d 0 = 0). When d 0 = 0, the NHC at x 0 is simply the SHC Eq. (10.3), that is the condition that x 0 be a strict minimum. In that case, Theorem 10.5 simply states that there exists an open neighborhood U of x 0 such that U ∩ Z = {x 0 } and on which f can be decomposed as the sum of d squares.

Remark 31 (Smoothing effect). Note that Theorem 10.5 induces a smoothing effect: indeed, if we simply assume that there exists a d 0 dimensional manifold of class C 1 of zeros satisfying the NHC, one sees that this manifold is actually of class C p-1 in a neighborhood of x 0 .

.3 Global decomposition as a sum of squares for functions on R d

In this section, we fix f to be a non-negative C p function defined on an open subset Ω of R d . Once again, we assume p ≥ 2. The goal is to find conditions on f to be written as a sum of squares of functions defined on Ω. These conditions will be that the NHC holds at every x 0 ∈ Z. We will start by reformulating this assumption in a more global and geometric way. We introduce the following definition of a manifold to which f is positively normal.

Definition 10.2 (positive normally to a sub-manifold). Let N be a sub-manifold of R d of class C k for k ≥ 1 and included in Ω. We say that f is positive normally to N if: a) N is included in the set of critical points of f (df (x) = 0 for all x ∈ N ); The set A is called an atlas. The manifold M is said to be of class C k for k ≥ 0 if all the transition maps φ i • φ -1 j are of class C k . It is said to be of dimension d if all its charts are in R d . As for sub-manifolds of R n , a manifold can always be decomposed as the union of its connected components, and the dimension is the same on each connected component. Note that with this definition, the restriction of a manifold to any open set is still a manifold (just by restricting the charts).

If M is a manifold of class at least C 1 , we can define at each point its tangent space T x M . Informally, this set T x M is all the possible derivatives γ (0) of curves γ : I → M defined on an open interval I around 0 such that γ(0) = x. Of course γ (0) is not yet formally defined. Formally, T x M can be defined as the classes of C 1 curves defined on an open interval I around 0 such that γ(0) = x, where we identify two curves γ and γ if (φ • γ) (0) = (φ • γ) (0) for a (or equivalently any) chart φ of M around x. We denote with [γ] ∈ T x M the equivalence class of a curve γ. It can be shown that T x M is a vector space (with the natural definition λ[γ] + µ[γ] = [λγ + µγ]) and that it is of dimension d where d is the dimension of M at x. A map g : M → R p defined on a manifold M is said to be of class C q if M is of class at least C q and if for any chart φ, the map g • φ -1 is of class C q . If q ≥ 1, then we can define the differential of g at any point x ∈ M as df (x)[γ] = (f • γ) (0). Hence, df (x) ∈ Hom(T x M, R p ).

Example 10.3. All sub-manifolds of R d are manifolds. The notions of regularity, dimension, and tangent space coincide.

For more precise definitions of topological spaces, atlases, charts, and details on the "Hausdorff second-countable" condition, see for instance the works by [START_REF] Lafontaine | An introduction to differential manifolds[END_REF]; [START_REF] Paulin | Géométrie différentielle élémentaire[END_REF]; [START_REF] Jänich | Topologie[END_REF]; [START_REF] Spivak | A Comprehensive Introduction to Differential Geometry[END_REF]. The main idea behind the introduction of manifolds as opposed to sub-manifolds of R d is to consider the intrinsic geometric object, and not its relation to the euclidean space it is embedded in (as such an embedding is not unique). An example of manifold as well as a representation of the tangent space is provided in the left of Sec. 10.3 .

. GLOBAL DECOMPOSITION AS A SUM OF SQUARES FOR FUNCTIONS ON MANIFOLDS467

.1 Assumptions in the manifold case

In this section, we formulate the local NHC in the case of manifolds, and rewrite Theorem 10.5 in this setting. We also extend the definitions of being positively normal and the global NHC (Lemma 10.1).

Fix p ∈ N, p ≥ 2, and a manifold M of regularity at least C p and of dimension d ∈ N. To start with, let f : Ω → R be a non-negative function defined on an open set of M and of class C p . As before, define Z to be the set of zeros of f . Contrary to the R d case, the second differential of the function f cannot be identified to a symmetric bi-linear form everywhere. However, it is the case at so-called critical points, i.e., points x ∈ Ω such that df (x) = 0. In particular, since all the zeros of a C 1 non-negative function are critical points, this Hessian will be defined at all points in the set of zeros Z of f . Lemma 10.2 (definition of the Hessian). Let x be a critical point of f . Then there exists a unique symmetric bi-linear form H f (x) : T x M × T x M → R such that for any local chart φ : (M, x) → (R d , 0) it holds:

∀ξ, η ∈ T x M × T x M, H f (x)[ξ, η] = d 2 (f • φ -1 )(0)[dφ(x)ξ, dφ(x)η].
In order to prove this lemma, we simply define the bilinear form as such for a given chart φ around x, and then show that this definition does not depend on the chart φ using the fact that x is a critical point. This is completely proved in section 2 of [START_REF] Milnor | Morse theory[END_REF]. In order to formulate the definition of the normal Hessian condition in the setting of manifolds, we further need a definition of what a sub-manifold of M is. A subset N ⊂ M is said to be a sub-manifold of M of class C k if M is of class C k and if, for any x ∈ N and any local chart φ : U → R d defined on a neighborhood of x, φ(U ∩ N ) is a sub-manifold of R d of class C k . In the literature, this is also called a proper sub-manifold (see on the left of Sec. 10.3 for an example).

Definition 10.3 (Normal Hessian condition for a manifold). Let x ∈ Ω be a point in the domain of f . We say that f satisfies the normal Hessian condition (NHC) at x if there exists a dimension 0 ≤ d 0 ≤ d and a sub-manifold N of M of class C k with k ≥ 1 and of dimension d 0 such that x ∈ M ⊂ Z, and one of the following equivalent conditions is satisfied: (i) the rank of H f (x) is dd 0 ;

(ii) H f (x)[ξ, ξ] > 0 for any vector ξ ∈ T x M \ T x N .

Using any local chart around the point x in the domain of f , one can apply Theorem 10.5 to obtain the following theorem as a corollary.

Theorem 10.7. If f satisfies the NHC at x 0 with regularity k and dimension d 0 , there exists an open neighborhood U of x 0 in M on which f is defined and such that U ∩ Z is a sub-manifold of M of dimension d 0 and of class C max(k,p-1) ; and there exist functions f i ∈ C p-2 (U ) where (10.8) Exactly in the same way as for the definition of the NHC for manifolds, we can similarly extend the definition of a function being positively normal to a sub-manifold in definition 10.2 as well as the global NHC in Lemma 10.1. We will therefore say that f : M → R which is non-negative satisfies the global NHC if it satisfies the local NHC at every point in its set of zeros Z, or ∇ 2 x x g(0, 0) = P 1 ∇ 2 f (x 0 )P 1 0. We can therefore apply the Morse lemma Lemma 10.10 to g: there exists two open neighborhoods of zero V ⊂ R d-d 0 , W ⊂ R d 0 as well as ϕ : W → V of class C p-1 such that {(x , y ) ∈ V × W : ∇ x g(x , y ) = 0} = {(x , y ) ∈ V × W : x = ϕ(y )} and z : V × W → R d-d 0 of class C p-2 such that ∀(x , y ) ∈ V × W, g(x , y ) = g(ϕ(y ), y ) + 1 2 z(x , y ) H z(x , y ), (10.13) where H is the positive definite matrix P 1 ∇ 2 f (x 0 )P 1 .

1 ≤ i ≤ d -d 0 such that ∀x ∈ U, f (x) = d-d 0 i=1 f 2 i (x).
Step 3: making the sum of squares appear. Since H ∈ S + (R d-d 0 ) and H 0, we can decompose it using the spectral theorem: H = d-d 0 i=1 λ i u i u i where the λ i > 0. Defining g i (x , y ) = λ i /2u i z(x , y ), Eq. ( 10.13) can be rewritten as ∀(x , y ) ∈ V × W, g(x , y ) = g(ϕ(y ), y ) + d-d 0 i=1 g 2 i (x , y ).

(10.14)

Note that the g i are of class C p-2 since z is of class C p-2 . We see that if we can show that g(ϕ(y ), y ) = 0 in a neighborhood of (0, 0), since we can go back to the original coordinate system through A -1 , we will have shown the theorem.

Step 4: characterizing Z in a neighborhood of x 0 . Denote with G ϕ = {(ϕ(y), y) : y ∈ W } the graph of ϕ, and which is a sub-manifold of class C p-1 of R d-d 0 × R d (see theorem 1.21, point (iv) by [START_REF] Lafontaine | An introduction to differential manifolds[END_REF]). Since A is an isometry, the set A(G ϕ ) is also a sub-manifold of class C p-1 of R d .

Let W = φ -1 (A(V × W )): it is an open neighborhood of 0. Note that φ( W ) ⊂ Z ∩ A(V × W ) by assumption, and since for any x ∈ Z, we have ∇f (x) = 0, it holds in particular that for any x ∈ Z ∩ A(V × W ), we have ∇ x g(A -1 (x)) = P 1 ∇f (x) = 0. Hence, by the result of the Morse lemma, it holds A -1 (φ( W )) ⊂ A -1 (Z) ∩ (V × W ) ⊂ G ϕ .

Define ψ : (x , y ) ∈ V × W → (xϕ(y ), y ) which is a C p-1 diffeomorphism onto its image with inverse (t, u) → (t + ϕ(u), u). Note that ψ maps G ϕ onto {0 R d-d 0 } × W . If π 2 denotes the canonical projection π 2 : R d-d 0 × R d 0 → R d 0 , we see that π 2 • ψ maps G ϕ injectively onto W ⊂ R d 0 .

Take Φ = π 2 • ψ • A -1 • φ : W → R d 0 , which is well defined by definition of W , and C 1 by composition. Note that it is an immersion at 0. Indeed i) φ maps 0 onto x 0 and is an immersion at 0 by assumption, hence dφ 0 is injective; ii) ψ • A -1 is a C p-1 diffeomorphism from A(V × W ) (containing x 0 ) to its image, and hence its differential is invertible at x 0 , and thus by composition, the differential d(ψ • A -1 • φ)(0) is injective; iii) since A -1 (φ( W )) ⊂ G ϕ by a previous statement, and since ψ(G ϕ ) ⊂ {0} × W also by a previous statement, it, it holds that the differential d(ψ • A -1 • φ)(0)R d 0 ⊂ {0} × R d 0 and hence applying π 2 does not change the injectivity of the differential; hence Φ is an immersion at 0. But since dΦ 0 is a linear map from R d 0 to R d 0 , dΦ 0 being injective is equivalent to dΦ 0 being invertible. Hence, by the local inversion theorem Theorem 10.9, there exists an open neighborhood of 0 W ⊂ W and an open neighborhood of 0 W ⊂ W such that Φ is a C 1 diffeomorphism from W to W .

Define U = (π 2 • ψ • A -1 ) -1 (W ) = A(ψ -1 (R d-d 0 × W )) ,which is an open neighborhood of x 0 . Note that since Φ is a diffeomorphism from W to W , we have φ( W ) ⊂ U . Moreover, since ψ is defined on V × W , we have U ⊂ A(V × W ). Finally, let u ∈ U ∩ A(G ϕ ). Since u ∈ U , there exists w ∈ W such that π 2 • ψ • A -1 (φ( w )) = π 2 • ψ • A -1 (u). Moreover, since π 2 • ψ is injective on G ϕ , and since both A -1 (φ( w )) and A -1 (u) belong to G ϕ (the first using the 10.5 . DISCUSSION AND POSSIBLE EXTENSIONS 473 previous point since W ⊂ W and the second by assumption), we have A -1 (φ( w )) = A -1 (u) and hence u = φ( w ) since A is one to one. This shows that U ∩ A(G ϕ ) ⊂ φ( W ).

Moreover, a previous point shows that A -1 (φ( W )) ⊂ A -1 (Z) ∩ (V × W ) ⊂ G ϕ . Now since A is one to one and since W ⊂ W we have φ( W ) ⊂ Z ∩ (A(V × W )) ⊂ A(G ϕ ). Since φ( W ) ⊂ U , we therefore have φ( W ) ⊂ Z ∩ U ⊂ A(G ϕ ) ∩ U . Combining this with the previous result, we finally have

φ( W ) ⊂ Z ∩ U ⊂ A(G ϕ ) ∩ U ⊂ φ( W ) =⇒ φ( W ) = Z ∩ U = A(G ϕ ) ∩ U.
(10.15)

Step 5: conclusion. Eq. ( 10.15) shows that φ( W ) = Z ∩ U = A(G ϕ ) ∩ U .

One the one hand, this shows that U ∩ Z is the intersection between an open set U and a sub-manifold A(G ϕ ) of R d of class C p-1 (since it is the composition of the graph of ϕ which is C p-1 , which is a C p-1 manifold by [START_REF] Lafontaine | An introduction to differential manifolds[END_REF], by an isometry which is in particular a diffeomorphism). Moreover, since φ is a C k immersion which is a homeomorphism on its image, φ( W ) is a sub-manifold of class C k . Thus, U ∩ Z is a sub-manifold of R d of class C max(k,p-1) .

On the other, since A -1 (U ) ⊂ V × W , Eq. (10.14) becomes ∀u ∈ U, g(A -1 (u)) = g(ϕ(y ), y ) +

d-d 0 i=1
g 2 i (A -1 (u)), (x , y ) = A -1 (u). (10.16)

Let u ∈ U and write (x , y ) = A -1 (u). First, note that A(ϕ(y ), y ) ∈ A(G ϕ ). Moreover, since A -1 u ∈ ψ -1 (R d-d 0 × W ) by definition of U , this shows that y ∈ W and hence (ϕ(y ), y ) = ψ -1 (0, y ) ∈ ψ -1 (R d-d 0 × W ). This in turn shows that A(ϕ(y ), y ) ∈ U . Hence, A(ϕ(y ), y ) ∈ A(G ϕ )∩U = Z ∩U and thus g((ϕ(y ), y )) = f (A(ϕ(y ), y )) = 0. Finally, using this in Eq. (10.16), recalling that g = f • A, and defining f i : u ∈ U → g i (A -1 u), we have

∀u ∈ U, f (u) = d-d 0 i=1
f 2 i (u).

(10.17)

We see that f i is of class C p-2 since g i was of class C p-2 and A -1 is an isometry; this concludes the proof of the theorem.

Discussion and possible extensions

In this work, we have provided second order sufficient conditions in order for a non-negative C p function to be written as a sum of squares of C p-2 functions. We hope this will help provide a theoretical basis to algorithms which use functional sum of squares methods [START_REF] Rudi | Finding global minima via kernel approximations[END_REF][START_REF] Vacher | A Dimension-free Computational Upper-bound for Smooth Optimal Transport Estimation[END_REF][START_REF] Rudi | Psd representations for effective probability models[END_REF], which rely on the smoothness of such decompositions. As these conditions are sufficient and not necessary, one main problems is understanding this gap. This seems a highly difficult, and while very interesting, we present three other more reachable subjects for future work.

The first is to have an explicit bound for the number of squares needed in the sum of squares decomposition in the compact case. We believe that using finer tools from differentiable geometry, we should be able to obtain a bound depending on meaningful quantities, and upper bounded by a constant n d depending only on the dimension d of the manifold on which the function is defined.

Proof of Lemma 10.3. The proof of this lemma is immediate. Indeed, by multiplication, we already know that χg is well defined and C q on U . Moreover, for any point x in V = M \ supp(χ), which is an open set, (χg)(x) = 0 (by definition if x ∈ M \ U and since χ(x) = 0 if x ∈ U ) and hence is C q on V . Since V ∪ U = M as supp(χ) ⊂ U , the property holds. Moreover, since χg = 0 on V , supp(χg) ⊂ supp(χ) ⊂ U .

Proof of Lemma 10.4. The proof of this result is a consequence of Lemma 10.8. Indeed, this result shows that there exists a family of function χ i : M → [0, 1] of class C k such that i) for all i ∈ I, supp( χ i ) ⊂ U i , ii) the support of ( χ i ) is locally finite and iii) i χ i = 1.

Define φ = i χ 2 i . Since i χ i = 1, and χ ≥ 0, necessarily, φ > 0. Hence √ φ is of class C k , and hence χ i := χ i / √ φ is of class C k , and satisfies all the desired properties.

10.A .2 Connected components

Connectedness is a key topological notion for manifolds, and allows to decompose manifold into separate blocks. Recall that two points x, x of a topological set X are connected if there exists no two open sets U, V such that X = U ∪ V , x ∈ U and x ∈ V . Since being connected is an equivalence relation, we can partition X in classes with respect to that relation, which are called "connected components". Connected components are both open and closed 4 . On a connected component of a manifold, the dimension d of the charts φ : U → R d is the same, and is called the dimension of that connected component (for more details, see any of the references on manifolds). Note that as a manifold M is assumed to be second-countable, it has at most a countable number of connected components. Recall that a sub-manifold is defined in the main text as follows (such a definition can be found in section 2.4.2 by Paulin ( 2006)). Note that the second-countable Hausdorff condition directly follows from that of M . Moreover, the C k compatibility of the charts is evident. From now on, when considering a sub-manifold N ⊂ M as a manifold, it will be with this structure. The reason for the introduction of all these concepts is to obtain the following lemma, which while it seems natural, we have not found as such in the literature.

Lemma 10.9. Let N be a sub-manifold of a manifold M . Let (N i ) i∈I be the connected components of N . There exists a collection of disjoint open sets (U i ) i∈I of M such that each N i ⊂ U i .

Proof. This proof relies mainly on paracompactness.

Step 1. For all x ∈ N , there exists

.1 Beyond least squares

In part I, we extended results from the least-squares setting to the class of generalized selfconcordant functions, in terms of statistical rates for the regularized empirical risk minimizer (defining the bias, effective dimension, and showing a bias-variance decomposition), as well as algorithms.

The least-squares setting has served as a major model not only in itself, but also as a toy model that can be analyzed in order to understand wider phenomena, which are observed on other loss functions. Many results, statistical and algorithmic, exist in the least squares setting, and their proof critically relies on the simple form of the loss function, and the closed form solutions for the least-squares problem. Extending these results to other losses, starting with the widespread logistic loss for classification, is an obvious perspective and is directly in the line of what we have done in part I. As we will see, there is still a lot to be done, especially in the stochastic setting.

Related works

Before formulating open questions and resarch perspectives, we wish to come back on two works which are related to ours, and which already provide additional steps in generalizing results from least squares to the generalized self concordant setting.

Library including our method. Meanti, Carratino, Rosasco, and Rudi (2020) have developed a library, based on fast kernel computation routines [START_REF] Charlier | Kernel operations on the gpu, with autodiff, without memory overflows[END_REF], and computations on GPUs, in order to make kernel methods scalable to billions of points.

In their library, they include least square regression as well as our method for logistic regression.

Extending the statistical rates to very regular problems. [START_REF] Beugnot | Beyond tikhonov: faster learning with self-concordant losses, via iterative regularization[END_REF] have developed a statistical analysis of the performance of regularized empirical risk minimization for GSC functions in the case where the target function is very regular. Indeed, note that in main corollary 3, we have assumed that the source condition f * = H r-1/2 h holds for some h ∈ H with r ≤ 1.

In the least squares case, when r ≥ 1, the Tikhonov regularization is not optimal anymore : it behaves as if the functions satisfies the source condition with r = 1/2 and does not leverage the additionnal regularity. This is referred to in the literature on least squares as the saturation effect of Tikhonov regularization [START_REF] Gerfo | Spectral algorithms for supervised learning[END_REF], and is linked to the fact that the bias term introduced in definition 3.1 cannot go faster than b λ ≤ R 2 λ 2 due to the specific form of Tikhonov regularization. In the work by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF], it is shown that the optimal rates when r ≥ 1 match the expression in Eq. (3.12), that is there is no saturation effect in the optimal rates, and that these rates can be achieved usign a different spectral regularization than Tikhonov, satisfying a certain qualification property.

The same phenomenon happens in the context of GSC function : our bound on the bias term does not allow a faster convergence of the bias than the one obtained for r = 1. [START_REF] Beugnot | Beyond tikhonov: faster learning with self-concordant losses, via iterative regularization[END_REF] circumvent this problem by introducing an estimator based on iterating Tikhonov regularization steps. They define :

f t n,λ = arg min f ∈H R n (f ) + λ 2 f -f t-1 n,λ 2
H , f 0 n,λ = 0. (11.1)

Analyzing this method as a form of proximal algorithm, they show that as soon as t is large enough, the qualification of the regularization is large enough to achieve optimal rates, without CHAPTER 11. CONCLUSION AND PERSPECTIVES saturation effect.

Perspectives from a statistics point of view

From a statistics point of view, we see two main perspectives. The first one seems relatively easily reachable to us, while the second is more vague as it goes further away from our setting.

Minimax rates. The first is to formally derive minimax upper and lower rates in the setting of GSC losses. For the upper rates, we believe that using Main theorem 8 as a starting point, and applying the same method as in the work by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF], we would obtain minimax upper bounds (of course, in the r ≥ 1/2 setting, we should not use Tikhonov regularization but the one proposed by [START_REF] Beugnot | Beyond tikhonov: faster learning with self-concordant losses, via iterative regularization[END_REF]). We also believe that analoguous lower rates can be obtained, using the fact that GSC losses are essentially equal to a quadratic function in a small region around their minimum. However, one would have to be more careful than in the least squared case, as that quadratic depends on the distribution. The proof by [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] using the scheme by Tsybakov (2008) will have to be studied in depth to see if it can be generalized as such, as we would have to find densities ρ 1 , ..., ρ N which are close enough to be compared using a quadratic approximation (i.e., they must induce functions f ρ i which are in the same Dikin ellipsoids), while far enough from each other to obtain the desired lower bound.

The misspecified setting. Our second perspective is to derive rates and provable algorithms in the misspecified setting, that is when we do not assume that f ρ belongs to the space H. In the least squares case, the degree of misspecification can be quantified by the kernel operator T k which compares the space L 2 (X , ρ X ) with the space H, and rates can still be obtained for certain algorithms (although they are usually upper rates and not optimal rates). We would have to define similar objects for more general losses, where the natural metric is not the L 2 metric but (in the case of GSC losses), the metric induced by the Hessian at the optimum, which is a form of weighed L 2 metric.

Extending fast rates of stochastic gradient optimization

In the least squares setting, a vast literature exists on the problem of minimizing the square loss when having access only to a stochastic oracle of the gradient [START_REF] Dieuleveut | Harder, better, faster, stronger convergence rates for least-squares regression[END_REF]. Recall that in the kernel least squares setting, the goal is to solve (11.4) 11.2 . RESEARCH PERSPECTIVES

min f ∈H R(f ) = 1 2 E f (X) -Y 2 = 1 2 f, Σf H -E [ f, Y k X H ] + cste,

483

where Σ is the covariance operator of the kernel k with respect to the measure ρ X of X. Consider the following regularized stochastic gradient descent scheme :

∀n ≥ 1, f n = f n-1 -γG n (f n-1 ) -γλ(f n-1 -f 0 ), (11.5) where G n (f n-1 ) is an unbiased estimator of ∇R(f n-1 ) (we assume E [G n (f )|F n-1 ] = ∇R(f ), where F n-1 is the σ-algebra generated by f 0 , ..., f n-1 . Write G n (f ) = ∇R(f ) -ξ n , where ξ i is the noise of the gradient. Under the additive noise assumption, that is [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] show the following bounds on the performance of

ξ n = y n k xn -E [Y k X ],
f n = 1 n n i=1 f i : E R(f n ) -R(f ρ ) ≤ λ + 1 γn 2 Σ 1/2 (Σ + λI) -1 (f 0 -f ρ ) 2 + τ 2 Tr(Σ 2 (Σ + λI) -2 ) n (11.6)
This bound shows a real bias variance trade off.

• The first term is a bias term controlled by the regularity of f 0 -f ρ .

• The second term is a variance term; in particular, it is controlled by the quantity Tr(Σ 2 (Σ + λI) -2 ) ≤ d λ where d λ is the effective dimension for least squares introduced in chapter 3.

Note that [START_REF] Dieuleveut | Harder, better, faster, stronger convergence rates for least-squares regression[END_REF] provide bounds for accelerated methods as well under the additive noise assumption. Moreover, they prove similar bounds in the case where there is multiplicative and additive noise (see [START_REF] Dieuleveut | Harder, better, faster, stronger convergence rates for least-squares regression[END_REF], Sec. 3. for details), although not for accelerated methods.

From our point of view, it would be very interesting to extend these rates to the logistic regression and GSC functions setting. Indeed, the fact that both a complex bias and variance term appear, which match those we encounter in the setting described in chapter 3, shows that there is hope to derive similar rates with meaningful quantities in the logistic case. However, the extension is not trivial, and exploring this area of research would help us understand if properties like GSC actually help us to analyze first order methods or not.

.2 A broader understanding of PSD models

In part II, we introduced PSD models, which are linear models of the form g A (x) = φ(x), Aφ(x) H for a given feature map φ representing X in a Hilbert space H, and where A is constrained to belong to the PSD cone of compact positive semidefinite operators. Denote with SOS H this set of functions. While we have presented some key properties of these models, and their applications to sampling, a lot of questions and directions are left to explore to better understand their advantages and limits.

We will divide this section into three parts. First, we will present our main lines of inquiry in order to better understand and use PSD models from a generic point of view (statistics, algorithms). Then, we will focus on sampling, and present research directions motivated by chapter 7. Finally, we will take a step back and raise some questions about modeling constraints in general.

Towards a better understanding of approximation and algorithmic properties of PSD models

Approximation properties. As is usually the case for non-parametric models such as RKHSs, it is not enough to know that our model for non-negative functions can approximate any non-negative functions pointwise or on a compact set to get precise rates non asymptotic rates of convergence.

Indeed, in order to get more precise (fast) rates, as in the least squares and GSC settings described in part I, we usually need the target function to lie in the space SOS H , or in an interpolation space between SOS H and another "natural" space for the problem at hand (such as L 2 (X , ρ X ) in the least squares setting). In any case, we need to better understand the conditions for a non negative function to be in SOS H , that is for the model to be well-specified (see chapters 8 and 10 as well as Sec. 11.2 .3). Finding such conditions is therefore a natural perspective. Note that this has been treated in part for functions defined on R d an on differentiable manifolds in the case where the kernel quantifies the C r regularity of a function, such as Sobolev RKHSs (see chapters 9 and 10). Obtaining such results for different spaces X and kernels on X would be an interesting direction to investigate. Moreover, the results we have derived so far for representability as a sum of squares of functions show that A is finite rank. This leads to the following question : does allowing the number of squares to go to infinity help have better approximation properties ? Finally, a question can be raised in relation to the importance of the spectral norm chosen on SOS H , as it characterizes the regularization. Given a situation or problem, can we have elements to characterize the ideal spectral norm to use (for now, we have heuristically chosen the trace norm as the optimal solutions are assumed to be finite rank) ?

Dimension reduction. A second key element which we must investigate further, and that has both statistical and algorithmic repercussions, is the dimension reduction aspect, for analoguous reasons to the ones described in chapter 3. Indeed, even though there is a reprenter theorem for PSD models Theorem 6.1, it is not always applicable (for example, when we must enforce an affine constraint such as X g A (x)dx = 1 in the case of densities), and leads to a SDP of size n which can quickly become untractable. It is therefore crucial to reduce the dimension of the resulting SDP. The way presented by [START_REF] Rudi | Psd representations for effective probability models[END_REF]; Marteau-Ferey, Bach, and Rudi (2022a) is to sample m points x 1 , ..., x m uniformly from the domain X and to use the following finite dimensional model, parametrized by B ∈ S + (R m ) and defined as ij B ij k(x, xi )k(x, xj ). This model is included in the set SOS H where H is the RKHS associated to the kernel k. This corresponds to uniform Nyström sampling, described in chapter 3. Note that sampling the m points uniformly does not seem to be the best idea a priori, and understanding good ways of selecting the centroids is also a very interesting perspective (we will detail this perspective a bit more in the next section).

Library. Finally, as is done by [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF], it is crucial that we implement a fast, well documented library implementing PSD models in order to explore the properties of this model in different settings, relying on fast SDP solvers. This is important both to test the quality and flexibility of the method on real data sets and loss, as well as to get some insight on what the right setting is for this model.

Perspectives around the use of PSD models to do sampling

In chapter 7, we derived an algorithm to approximately sample from an un-normalized distribution with density p by modelling that density with a PSD model from which we can sample. This algorithm models the density by fitting it to a PSD model p using m centers x 1 , ..., x m , which are sampled uniformly at random from X (we look for p in the form ij B ij k(x, xi )k(x, xj )).

Doing better than approximation on a uniform subset of points. Sampling the m basis points uniformly at random from X seems to be a bit of a waste. If, for example, the distributions p which we have to sample from has support essentially contained in a small dimensional suspace of R d , the m centers will probably never capture any information on the distribution p. An important open problem is therefore to design sampling methods which can identify a region in which the support of p is included.

. RESEARCH PERSPECTIVES

485

An idea to do this is to consider a damped method. Assume that p is of the form p λ (x) = e -λV (x)

where V is a regular potential and λ is an inverse-temperature parameter. A natural idea in order to sample from the normalized version of p λ would be to create successive approximations p t of e -λtV for a decreasing sequence of λ t ↓ λ, starting at λ 0 = 0 (in that case, the measure is uniform). To approximate p t := p λt with p t , we would use m centers computed by sampling from the approximation p t-1 of p t-1 . In practice, at least in low dimensional settings (d ≤ 15), this method seems to be quite efficient. However, we still lack proper theoretical results, and would like to explore it in future work. Note that applying such a scheme could help sample points to perform dimension reduction.

Applications. Following the previous line of thought, one interesting application of being able to sample from a density proportional to e -λV (x) would be to sample from Gibbs measures, which are exactly of this form. This is a very important in statistical physics and chemistry, as they appear as stationnary measures (i.e., limit states, see the book by [START_REF] Lelièvre | Free Energy Computations[END_REF]). This is a field where Monte-Carlo Markov chain (MCMC) methods are widely used. These methods suffer from the fact that they do not automatically generate i.i.d. samples, and hence get stuck in certain regions for a long time (this is related to called the multimodal and entropy barrier problems). We can therefore try to see if our method can help in that setting, by either directly sampling from the Gibbs measure, or by providing good candidates in order to reduce the amount of samples rejected in a MCMC methods which uses rejection sampling.

One other interesting application would be to sample from a uniform measure defined on a subset of X . This would require to abandon the C r regularity condition, and to look instead at the regularity of the subset or a regularization of the indicator function of the subset.

A more general question : how to model constrained outputs

In part II, we have explored the modelling of a non-negativity constraint using PSD models.

There are of course a lot of interesting problems where the constraint on the output is not a non-negativity constraint, but a constraint of another type. We could look for a predictor with outputs in :

• an affine cone;

• a PSD cone;

• a compact set.

Proposing a good model in these different situations proves to be quite a challenge, for which PSD models are not always adapted. Note that in the work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF], we show that it is easy to generalize the non negative setting to the affine cone setting.

Handling PSD cones has partially been done by [START_REF] Muzellec | Learning psd-valued functions using kernel sums-of-squares[END_REF]. Note that in that setting, the SOS guarantees for approximation are weaker than those existing for the non-negative case.

The real difficulty, of course, is to provide a good model for outputs in compact sets. Even in the case where the compact set is of the form [a, b], this is not easy. Indeed, if we use the PSD model naively, we would ask fa = g 1 ∈ SOS H and bf = g 2 ∈ SOS H , and hence, in particular, g 1 + g 2 = a + b. This affine constraint means that we cannot apply the representer theorem.

.3 Going forward with kernel sum of squares

In part III and in particular in chapter 8, we have made a parallel between the method to perform global optimization proposed in chapter 9 and the one used in polynomial optimization. In this section, we present future directions of research which are, for the greater part, inspired by this parallel.

Towards a more complete understanding of the properties of kernel sums of squares in the manifold setting Constrained case. As we have seen in chapter 8, constraints are necessary in the polynomial case to guarantee the convergence of the moment-SOS hierarchy. A first interesting question is whether or not it is possible to adapt Main theorem 15 to the constrained setting for kernel sum of squares. More formally, we wish to find sufficient conditions in order for a function f which is non-negative on a set K = {x ∈ X : f j (x) ≥ 0, 1 ≤ j ≤ m} to be written in the form : (11.7) We claim this is possible under certain second order conditions which resemble those made by [START_REF] Marshall | Representations of non-negative polynomials having finitely many zeros[END_REF] in the polynomial case, and which correspond to the "easy" second order condition for optimality given by Gilbert (2020) called the linear independence qualification of constraints (see p.182, QC-IL). For simplicity, we will not write down the full condition here. However, we believe that the result is true, and that we can assume that the σ j are simply squares of functions for 1 ≤ j ≤ m. The proof will rely on the introduction of a well chosen Lagrangian which incorporates only the activated constraints, and which are assumed to always be linearly independent.

f = σ 0 + m j=1 σ j f j , (σ j ) 0≤j≤m ∈ SOS H .
Bounding the number of functions in the SOS decomposition. Going into the proof of Main theorem 15, we can see that in order to decompose a function f satisfying the assumptions of the theorem into a finite sum of squares, we glue local decompositions as the sum of d squares.

While in Main theorem 15 does not provide any guarantees on the necessary number of functions, we make the following conjecture, with the same assumptions. A very rough sketch of proof of this result is the following. Take one connected component of the set of zeros, N . We use the tubular neighborhood theorem to show that there exists an open set U containing N in M such that ν M N is diffeomorpic to U , where ν M N is called the normal bundle to N in M . We therefore assume that f is defined on ν M N . We perform a handle decomposition [START_REF] Wall | Theory of handle decompositions[END_REF] on N and show that we can build a Riemannian structure on the fibers of ν M N , such that f (p, v) = v, v p by successively gluing such sructures using the handle decomposition ( •, • p denotes the riemannian metric at point p on the fiber above p). We then use the Nash embedding theorem, to show that there exists a neighborhood around each connected component of the minimizers such that f is decomposable as a sum of O(d 3 ) squares.

. RESEARCH PERSPECTIVES

487

We conclude by gluing intelligently all these decompositions, in particular assuming the open sets containing each connected component N are disjoint.

Moments. In Sec. 8.3 .3, we showed that we can define the cone of moments in the kernel setting, in a slightly more elaborate way than for polynomials as we do not have the nice product structure in the kernel case anymore. Moreover, when constraints are added, the problem will become even more complicated, as we will have to consider the products of three functions, and hence the space H ⊗3 , with kernel k 3 . We really wish to explore this moments point of view of kernels, as we hope it will help us better understand where the extraction procedure does not work in the kernel setting. Moreover, we expect it to help us perform relaxations which keep the lower bound guarantees of the polynomial setting.

General objective

After having explained what we believe to be open problems within reach, let us take a step back, and formulate two long term goals.

• We would like to see if it is possible to find interesting classes of functions which are adapted to our problems and which provide better guarantees (i.e., that are really compatible with PSD representations).

• We would like to move out of the setting of R d and C b regularity, and instead focus on different types of regularity, such as the low dimensionality of the support for example.

Can we provide good a posteriori guarantees ?

As we have seen in chapter 8, the fact that we sample the equality constraint in the kernel SOS setting kills the guarantees we used to have at the step before, and for polynomials, which is that we have a lower bound of the original problem.

An interesting research direction in this context is to see if we can adapt the moment formulation or the sum of squares formulation to keep a guarantee, and even have stronger ones (upper bounds and lower bounds). This is done in a work by [START_REF] Woodworth | Non-convex optimization with certificates and fast rates through kernel sums of squares[END_REF] in the case where we have access to the discrete Fourier transform of the function we want to minimize. Indeed, in this work, all errors (optimization, statistical) are explicitely controlled and allow to derive a certificate whose bound is fairly tight.

Applying the method to typical problems tackled by moment-SOS hierarchies in the polynomial case

In the book by [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF], there is an entire section where polynomial optimization is used to tackle certain problems, ranging from optimal control to probability theory. In particular, these problems can usually be formulated as optimization problems on non-negative measures, or optimization problems with non-negativity constraints on a function (i.e., we ask for a measure to be non-negative or a function to be non-negative). [START_REF] Berthier | Infinite-Dimensional Sums-of-Squares for Optimal Control[END_REF] have started applying the kernel sum of squares method to optimal control. This is quite difficult, as the optimal control problems are usually highly irregular. On the bright side, it is almost impossible to solve them in an efficient way, and so it is relevant to try to apply our method even if the needed guarantees do not hold a priori.

  Contents 1.1 Introduction aux méthodes à noyau . . . . . . . . . . . . . . . . . . . 9 1.2 Taux et algorithmes rapides pour les fonctions auto-concordantes généralisées . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.3 Modèles semi-définis positifs : théorie et applications . . . . . . . . 29 1.4 Sommes de carrés de fonctions et optimisation globale . . . . . . . . 31

  et constructions d'espaces à noyau reproduisant (ENR)

1. 2 .

 2 TAUX ET ALGORITHMES RAPIDES POUR LES FONCTIONS AUTO-CONCORDANTES G ÉN ÉR Assumption 1.1 (Gradients and Hessiennes bornés). Il existe des constantes réelles b 1 , b 2 telles que b 1

  3.1.1, nous définissons des classes de mesures de test M pour lesquelles nous obtenons des taux statistiques précis. Formellement, pour R, Q, b 1 , b 2 > 0, r ∈ [1/2, 1] et b ≥ 1, nous définirons la classe de mesures d'essai M(R, r, Q, b, b 1 , b 2 ) comme l'ensemble des mesures ρ telles que les conditions suivantes sont remplies.

  paramétrisation de H m dans Eq. (3.35), le problème Eq. (1.34) devientα n,λ,m = arg min e i K n,m α) + λ 2 α K mm α, K n,m = (k(x i , x j )) 1≤i≤n 1≤j≤m , (1.35) et le minimiseur de Eq. (1.34) peut s'écrire f n,λ,m = k(x) α n,λ,m . Comme dans le cas des moindres carrés, une bien meilleure façon de paramétrer le problème est d'écrire K mm = T T en utilisant une décomposition de Cholesky, où T est une matrice triangulaire supérieure. On remarque alors que si on définit φ(x) = T -k m (x) ∈ R m , φ est une représentation qui définit le noyau k m : k m (x, x ) = φ(x) φ(x ). Donc, H m = {α φ(•) : α ∈ R m }. Si Φ est la matrice dont les colonnes sont les Φ i = φ(x i ), c'est-à-dire Φ = K n,m T -1 , le problème Eq. (1.34) peut être formulé comme suit α n,λ,m = arg min de Eq. (1.34) peut s'écrire f n,λ,m = k(x) (T -1 α n,λ,m ). Bien sûr, cela se fait au prix du calcul et de la sauvegarde d'un facteur de Cholesky T, ce qui représente un temps d'ordre O(m 3 ) et une complexité en mémoire d'ordre O(m 2 ).

Formellement

  trouver une ε-approximation du minimum, c'est-à-dire, c tel que | cf * | ≤ ε ; (ii) trouver une ε-approximation du minimiseur, c.-à-d. , x tel que |f ( x) -f * | ≤ ε.

  CHAPTER 1. R ÉSUM É EN FRANC ¸AIS Main theorem 5[START_REF] Rudi | Finding global minima via kernel approximations[END_REF], Théorème 5.Supposons que les conditions du théorème principal 4 soient satisfaites, et que Ω soit une boule de rayon R pour simplifier. Soit r < sd/2. Il existe des constantes C r ,d , C r ,d dépendant uniquement de r , d et une constante h R,r dépendant uniquement de R, r telle que si g * ∈ S + (H) est la solution optimale satisfaisant f

  Figure 2.1: Different losses for binary classification

Figure 2

 2 Figure 2.2: Evolution of the approximation error, estimation error and total excess risk for a prototypical machine learning problem.

Figure 2 . 3 :

 23 Figure 2.3: Learning from n = 15 noisy observation of a polynomial of degree 6, for different maximum degrees.

  Figure 2.4: Learning from n = 20 noisy observation of a polynomial of degree 6, by solving ridge regression on H = R 20 [x] for different regularization values.

Figure 2 . 5 :

 25 Figure 2.5: Double descent : where the estimation error decreases after interpolation.

  .27) As in the previous section, such a bias-variance decomposition can also be used to derive upper rates of convergence for certain classes of function. Recall from Sec. 3.1.1 the definition of the class M < (R, r, β, b, σ, M ) of distributions ρ, which we shorthand with M < θ when r, β, b are fixed and θ ∈ Θ = {(R, σ, M ) ∈ R 3 ++ : σ ≤ M }. Combining Cor. 3.1 with Theorem 3.4, setting m θ,n = m λ θ,n , it is possible to show an upper rate of convergence for the dimension reduced problem for the class M < θ . Theorem 3.5 (Prototypical upper rates with dimension reduction). Fix r ∈ [1/2, 1], β, b > 0.

Figure 3 . 1 :

 31 Figure 3.1: Training loss and test error as as function of the number of passes on the data for our algorithm vs. K-SVRG. on the (left) Susy and (right) Higgs data sets.

  Assumption 4.4 (Definition in 0). | Z (0)|, ∇ Z (0) and Tr(∇ 2 Z (0)) are almost surely bounded.

  Assumption 4.4 (Definition in 0). | Z (0)|, ∇ Z (0) and Tr(∇ 2 Z (0)) are almost surely bounded.

  Definition 4.3. For θ ∈ H, denote by t(θ) the function t(θ) = sup z∈supp(ρ) sup g∈ϕ(z) |θ • g| , and define

  Theorem 4.6 (General bound, simplified setting). Let n ∈ N and 0

  which means that N is bounded by a rational function of the arguments of poly.Proof. of Cor. 4.2 We simply apply Cor. 4.4 for α = 1 and Q = B 1 .

Lemma 5 . 1 .

 51 Let x, b ∈ H and assume that(2) i ≤ a for a > 0. With probability 1-δ the following methods output an element in LinApprox(H

Figure 5 . 1 :

 51 Figure 5.1: Training loss and test error as as function of the number of passes on the data for our algorithm vs. K-SVRG. on the (left) Susy and (right) Higgs data sets.

  seeLemma 5.6 in Sec. 5.A .1.• The convergence rate of the full Newton method starting in D λ (1/7) is obtained in Lemma 5.10.

Figure 5 Figure 5

 55 Figure 5.2: (Left) Classification error as a function of the regularization parameter and (Right) test loss as a function of the regularization parameter, when performing a logistic regression with M = 2 × 10 4 Nyström features on the entire Higgs data set; we select λ = 10 -9 .

Figure 5 Figure 5 . 5 :

 555 Figure 5.4: (Left) Distance to optimum as a function of time and (Right) distance to optimum and classification error as a function of the number of passes on the data when performing our second order scheme and K-SVRG to minimize the train loss on Higgs, with 1.0 × 10 4 Nyström points and λ = 10 -9 .

Figure 5

 5 Figure 5.6: (Left) Distance to optimum as a function of time and (Right) distance to optimum and classification error as a function of the number of passes on the data when performing our second order scheme and K-SVRG to minimize the train loss on Susy, with 1.0 × 10 4 Nyström points and λ = 10 -8 .

Figure 6

 6 Figure 6.1: Details in Sec. 6.6 . (top) density estimation, (center) regression with Gaussian heteroscedastic errors. (bottom) multiple quantile regression. Shades of blue: estimated curves. Orange: ground truth. Models: (left) PNM, (center-left) NCM, (center-right), GLM, (right) Our model.

Figure 6 . 2 :

 62 Figure 6.2: Best approximation of g using NCM with (left) n = 100 (center) n = 1000 (right) n = 10000 points.

Figure 6 . 3 :

 63 Figure 6.3: Representation of the densities learned by the different models.

A

  hyper-rectangle Q ⊂ R d can be parametrized with its corners a, b ∈ R d , a ≤ b, by writing Q = d k=1 [a k , b k [; a corresponds to the "bottom left" corner and b to the "top right" one.

  Qρ is the indicator function of Q ρ . The guarantees of the algorithm are established in the following theorem, proved formally in Appendix 7.D .2. Theorem 7.1. Given (Q, N, ρ) where Q is a bounded hyper-rectangle of R d , ρ > 0 and N ∈ N, the function Sample in algorithm 2 returns N i.i.d. samples from the distribution p Q,ρ defined in Eq. (7.7). Moreover, the number of integral computations of the form I( Q) performed during the algorithm is bounded by N log 2 (|Q|) + N d log 2 2 ρ + 1, and the number of erf computations is O(N m 2 d (log 2 (2|Q|) + d log 2 (2/ρ))), where m is the dimension of the PSD model.

Figure 7 . 1 :

 71 Figure 7.1: Samples obtained from algorithm 2 using different values for ρ

  Hellinger distance, d = 10

Figure 7 . 2 :

 72 Figure 7.2: Evolution of the empirical Hellinger distance on a test set, between learnt distribution p and target distribution p when increasing the number of evaluation points, for fixed values of m. We learn p as a rank one PSD model through Eq. (7.20). (left) Learning p 2 with d = 10 defined in Sec. 7.5 . (right) Learning p 1 defined in Sec. 7.5 .

  Figure 7.3: (top left) Plot of the distribution p 1 , (top right) heat map of an approximation p 1 of p 1 . (bottom left) samples generated from p 1 , (bottom right) performance of our method in MMD distance.

  ) and satisfies F[v](ω) ≥ 0 for all ω ∈ R d . Then, the following hold. (a) The function k : R d × R d → R defined as k(x, x ) = v(xx ) for any x, x ∈ R d is a positive kernel and is called translation invariant kernel. (b) The reproducing kernel Hilbert space (RKHS) H and its norm • H are characterized by

  First we start by recalling the restriction properties of Sobolev spaces. Let Ω ⊆ Ω ⊆ R d be two open sets. Let β ∈ N and p ∈ [1, ∞]. By definition of the Sobolev norm above we have

  7.F . APPROXIMATION AND SAMPLING USING A RANK ONE PSD MODEL 343 from p τ,m,λ as in Remark 24, and finding ρ such that 2 d Lip(A) ρ = |Q| I(Q) Lip(A) ρ = ε, since the adaptive bound will have computed Lip

Figure 7 . 4 :

 74 Figure 7.4: Trying to learn a non-continuous function using a rank one PSD model. (left) Plot of the target and learnt distributions using algorithm 5. (right) 1000 samples generated from the learnt distribution p sample .

  8.2 .1 in the setting of minimizing a function f belonging to C r b (Ω), where Ω is an open domain of R d , and where C r b (Ω) denotes the set of r continuously differentiable functions with bounded derivatives.

Figure 8 . 1 :

 81 Figure 8.1: Optimization is as hard as approximation.

Figure 8 . 2 :

 82 Figure 8.2: Plots of functions z = f (x, y), where the zeros of f are highlighted in black. left: f satisfies Assumption 8.2, right: f does not satisfy Assumption 8.2 but satisfies the assumptions needed for a sum of squares decomposition in Marteau-Ferey, Bach, and Rudi (2022b).

Figure 8 . 3 :

 83 Figure 8.3: Left: Representation of the manifold M = S 2 as well as a sub-manifold N homeomorphic to a circle. The tangent spaces at a given point x 0 ∈ N ⊂ M are represented as well.Right: Representation of a non-negative function on the sphere as a color map; it satisfies the assumptions of Marteau-Ferey,Bach, and Rudi (2022b) to be decomposed as a sum of squares, and its null space Z is represented in black.

CHAPTER 8 .

 8 A PARALLEL WITH MOMENT-SOS HIERARCHIESReducing dimension in the kernel sum of squares setting

CHAPTER 9 .

 9 FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS Algorithm 6 Global minimum. Given f : R d

Assumption 9 .

 9 1 (Geometric properties on Ω and f ). The following holds:(a) Let Ω = ∪ x∈S B r (x), where S is a bounded subset of R d and B r (x) is the open ball of radius r > 0, centered in x. (b) The function f is in C 2 (R d ). Ω contains at least one global minimizer. The minimizers in Ω are isolated points with strictly positive Hessian and their number is finite. There is no minimizer on the boundary of Ω.

Theorem 9 . 1 (

 91 Main result, informal). Let Ω ⊂ R d be a ball of radius R > 0. Let s > d/2 and let k be Sobolev kernel of smoothness s (see Example 9.1). Let f ∈ C s+3 (R d ) and that satisfies Assumption 9.1(b). Let ĉ be the result of algorithm 6 executed with n ∈ N + points chosen uniformly at random in Ω and λ > 0. Let δ > 0. There exist n s,d,δ , C s,d > 0 such that, when n > n s,d,δ , and λ ≥ C s,d n -s/d+1/2 (log n δ ) s/d-1/2 ,

(

  Moreover we denote by B r (z) the open ball B r (z) = {x ∈ R d | xz < r}. Let Ω ⊆ R d be an open set. Let α ∈ N d . We introduce the following multi-index notation |α| = α 1 + • • • + α d and ∂ Adams and Fournier, 2003). For m ∈ N, and Ω an open set of R d , denote by C m (Ω) the set of m-times differentiable functions on Ω with continuous m-th derivatives. For any function u defined on a superset of Ω and m times differentiable on Ω, define the following semi norm. |u| Ω,m = max Positive definite matrices and operators. Let H be a Hilbert space, endowed with the inner product •, • . Let A : H → H be a linear operator and denote by A * the adjoint operator, by Tr(A) the trace of A and by • F the Hilbert-Schmidt norm A 2 F = Tr(A * A).

Assumption 9 . 2 (

 92 Properties of the space H). Given a bounded open set Ω ⊂ R d , let H be a space of functions on Ω with norm • H , satisfying the following conditions

. 8 )

 8 Proof. Let x ∈ B r (ζ) and consider the functionh(t) = g(ζ + t(xζ)) on [0, 1]. Note that h(0) = g(ζ) and h(1) = g(x). Taking the Taylor expansion of h of order 1, we have h(1) = h(0) + h (0) + 1 0 (1 -t)h (t)dt, with h(0) = g(ζ), h (0) = (xζ) ∇g(ζ) and h (t) = (xζ) ∇ 2 g(ζ + t(xζ))(xζ).Since g(ζ) = 0 by construction and ∇g(ζ) = 0 since ζ is a local minimizer of g, we have h(0) = h (0) = 0 leading to g

9. 4 .

 4 EQUIVALENCE OF THE INFINITE-DIMENSIONAL PROBLEM391Note that for x ∈ B r (ζ) we have ∇ 2 g(x) γI and so R(x) γI. In particular, this implies that for anyx ∈ B r (ζ), S(x) = R(x) is well defined ( √ • : S + (R d ) → S + (R d) is the spectral square root, where for any M ∈ S + (R d ) and any eigen-decomposition M

  Let τ := sup x∈Br(ζ) R(x) op = R(x) op , which is well defined because R is continuous since g ∈ C 2 (Ω). Define the compact set K = {T ∈ S(R d ) | γI T τ I} and the open set U = {T ∈ S(R d ) | γ 2 I ≺ T ≺ 2τ I}. Note that K ⊂ U ⊂ S(R d ). Fix i, j ∈ [d]and consider the function θ i,j : U → R defined by θ i,j (M ) = e i √ M e j . Since the square root √ • : S + (R d ) → S + (R d ) is infinitely differentiable (see e.g. the explicit construction in the work by Del Moral and Niclas (

Fix

  j ∈ [n]. Define B r (ζ) = K ⊂ U = B 2r (ζ) and apply proposition proposition 9.10 to x ∈ U → e j (xζ) to get h j ∈ C ∞ 0 (R d ) which coincides with e j (• -ζ) on K hence on B r (ζ). Applying Assumption 9.2(a), the restriction h j = h j | Ω is in H, and hence h = j∈[d] h j e j satisfies the desired property.

CHAPTER 9 .

 9 FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS Step 4. The w i = e i S h, i ∈ [d] have the desired property.It is clear that the w i are in H as a linear combination of products of functions in H (see Assumption 9.2(a)), since w i = j∈[d] S ij (x)h j (x) for any x ∈ Ω. Moreover, i∈[d]

  .10) Proof. Let Z = {ζ 1 , . . . , ζ p }, p ∈ N + be the non-empty set of global minima of f , according to Assumption 9.1(b). Denote by f * = min x∈Ω f (x) the global minimum of f , and by g: Ω → R the function g = f | Ω -f * 1| Ω where 1 is the function 1(x) = 1 for any x ∈ R d . Assumption 9.3 implies that ∇ 2 g = ∇ 2 f | Ω is continuous, an that ∂ 2 g ∂x i ∂x j ∈ H for any i, j ∈ [d]. Moreover, g ∈ H. Indeed, by construction f * 1 is in C ∞ (R d ),and since H satisfies Assumption 9.2(a), f * 1| Ω ∈ H. Since f | Ω ∈ H by Assumption 9.3, then g ∈ H.

Theorem 9 . 3 .

 93 Let Ω ⊂ R d be an open set, k be a kernel, H the associated RKHS, and f : R d → R. Under Assumptions 9.1 to 9.3, the problem in Eq. (9.3) admits an optimal solution (c * , A * ) with c * = f * , and A * a positive operator on H with rank at most d|Z| + 1.

  .13)and C = C 0 (|g| Ω,m + MD m Tr(B)) with C 0 = 3 max( √ d,3 √ 2d(m-1)) 2m m!

  .14) The last step is bounding R m (u). Recall the definition of | • | Ω,m from Eq. (9.6). First, note that A = V * BV is finite rank (hence trace-class). Applying the cyclicity of the trace and the fact that V V * is the identity on R n , it holds Tr(A) = Tr(V * BV ) = Tr(BV V * ) = Tr(B). Since k satisfies Assumption 9.2(a), by Lemma 9.9, page 426, r A ∈ H and r A H ≤ M Tr(A) = M Tr(B) where M is fixed in Assumption 9.2(a). Moreover, since the kernel k satisfies Assumption 9.2(d) with m and D m , then |v| Ω,m ≤ D m v H , for any v ∈ H as recalled in Remark 26. In particular, this implies |r A | Ω,m ≤ D m r A H ≤ D m M Tr(B). To conclude, note that, by the multinomial theorem,

CHAPTER 9 .,

 9 FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS when h X,Ω ≤ r min(1, 1 18(m-1) 2 ) and λ ≥ 2MD m η. Here C 0 = 3 max( D m , M are defined in Assumption 9.2 and A * is given by Theorem 9.3. Moreover, under the same conditions Tr( B) ≤ 2 Tr(A * ) + 2 η λ |f | Ω,m .(9.16)

  .5). The existence of the admissible point (c * , B) proven in the step above implies that ĉλ Tr( B) ≥ c * -λ Tr(B) ≥ f * -λ Tr(A * ), from which we derive, λ Tr( B) -λ Tr(A * ) ≤ ∆, ∆ := ĉf * .

  and since for any m ∈ N + , |1 Ω | Ω,m = 0, we have |ĝ| Ω,m ≤ |f | Ω,m + |1 Ω | Ω,m = |f | Ω,m . Injecting this in the previous equation yields ∆ ≤ η|f | Ω,m + ηMD m Tr( B). (9.18) uniformly at random in Ω and λ > 0. Let δ ∈ (0, 1]. When m ∈ N + satisfies m < sd/2 and n ≥ max(4, 15(m -1)) m C m,s,d , where C m,s,d = 11 m C 0 max(1, MD m ) with C 0 defined in Theorem 9.5 and MD m defined in proposition 9.1. Note that C m,s,d is explicitely bounded in the proof in page 437 in terms of s, m, d. Then, with probability at least 1 -δ, the following holds |ĉ -f * | ≤ 3 λ (Tr(A * ) + |f | Ω,m ) .

  has a unique minimizer ζ that belongs to Ω and is locally strongly convex around ζ then there exists a ν > 0 such that the problem above achieves an optimum (c * , z * ) with z * = ζ and c * = f * + ν 2 ζ 2 . In particular, to characterize ν explicitly we introduce the stronger assumption below. Assumption 9.4 (Geometric assumption to find global minimizer). The function f : R d → R has a unique global minimizer in Ω. If f satisfies Assumptions 9.1(b) and 9.4, denote with ζ the unique minimizer of f in Ω and with f * = f (ζ) the corresponding minimum. Remark 28. Under Assumptions 9.1(b) and 9.4 f can be lower bounded by a parabola with value f * at ζ, i.e., there exists β > 0 such that ∀x

Lemma 9 . 5 .

 95 Suppose f satisfies Assumptions 9.1 and 9.4. Let ζ be the unique minimizer of f in Ω and f * = f (ζ) be the corresponding minimum. Let β > 0 such that Eq. (9.21) holds. If ν < β then the problem in Eq. (9.20) has a unique solution (c * , z * ) such that z * = ζ and c

  particular, when ν < β, there exists a finite rank operator A * such that the triplet (f * + ν 2 ζ 2 , ζ, A * ) is optimal. Theorem 9.7. Let Ω ⊂ R d be an open set, k be a kernel, H the associated RKHS, and f : R d → R satisfying Assumptions 9.1 to 9.3, and Assumption 9.4. Let β satisfying Eq. (9.21). For any ν < β, the problem in Eq. (9.22) admits an optimal solution (c * , z * , A * ) with c * = f * + ν 2 ζ 2 , z * = ζ, and A * a positive semi-definite operator on H with rank at most d + 1.

Theorem 9 . 8 (

 98 Convergence rates of Eq.(9.23) to the global minimizer). Let Ω be a set satisfying Assumption 9.1(a) for some r > 0. Let X = {x 1 , . . . , x n } ⊂ Ω with fill distance h X,Ω . Let k be a kernel satisfying Assumption 9.2 for some m ≥ 2 and f satisfying Assumptions 9.1, 9.3 and 9.4. The problem in Eq. (9.23) admits a solution. Denote by (ĉ, ẑ, B) any solution of Eq. (9.23), for a given λ > 0. Then for any ν < β, ν 2 ẑζ 2 ≤ 3η(|f | Ω,m + ν) + 2λ Tr(A * ), 2 and λ ≥ 2MD m η. Here C = 3 (3 √ 2d(m-1)) 2m m! and D m , M are defined in Assumption 9.2. A * is from Theorem 9.7. Moreover under the same conditions |ĉ -ν 2 ẑ 2 -f * | ≤ 2η |f | Ω,m + λ Tr(A * ) + 2ην, (9.25) Tr( B) ≤ 2 Tr(A * ) + 2 η λ |f | Ω,m + 2ν η λ .(9.26)

  in n and δ, where C d,m = 3 m CMD m with C defined in Theorem 9.8 and F = |f | Ω,m + ν + Tr(A * ). This means that under the additional assumption of a unique minimizer in Ω, we achieve a convergence rate that is only logarithmic in ε, moreover when m d also the dependence with respect to C d,m (which is exponential in m and d in the case of the Sobolev kernel) and F improves, since d/m tends to 0.

Theorem 9 .

 9 10. Let d, p, s ∈ N. Let Ω satisfy Assumption 9.1(a) and f

Theorem 9 .

 9 11 (Global minimum for functions with low smoothness). Let s ∈ N. Let k r be a Sobolev kernel with smoothness r ≥ s, r > d/2 and H be the associated RKHS. Let Ω ⊂ R d satisfying Assumption 9.1(a) and f ∈ C s+2 (R d ), satisfying Assumption 9.1(b). The problem in Eq. (9.32) admits a minimizer. Denote by (c, B) any of its minimizers for a given λ > 0, τ > 0.

  .36) and C = C 0 (|f | Ω,m + MD m Tr( B)). The constants C 0 , defined in Theorem 9.4, and m, M, D m , defined in Assumptions 9.2(a) and 9.2(d), do not depend on n, X, h X,Ω , ĉ, B or f .

Figure 9

 9 Figure 9.1: Top: 2D function without small-amplitude high-frequency components. Bottom: 2D function with small-amplitude high-frequency components. Left: sampled points and the trajectory of the proposed algorithm. Center: model reconstructed by the algorithm (see Eq. (9.37)). Right: the trajectory of gradient descent starting from random points. As it is possible to see, even a small local non-convexity prevents the random+GD algorithms to converge properly, while the proposed method is quite robust to it.

  First we start by recalling the restriction properties of Sobolev spaces. Let Ω ⊆ Ω ⊆ R d be two open sets. Let β ∈ N and p ∈ [1, ∞]. By definition of the Sobolev norm above we have

( a )

 a The function k : R d × R d → R defined as k(x, x ) = v(xx ) for any x, x ∈ R dis a kernel and is called translation invariant kernel. (b) The RKHS H k (R d ) and the norm • H k (R d ) are characterized by

Proposition 9 .

 9 11 ([START_REF] Wendland | Scattered Data Approximation[END_REF], Corollary 3.11. Local polynomial reproduction on a ball). Let k ∈ N, d, m ∈ N + and δ > 0. Let B δ be an open ball of radius δ > 0 in R d . Let Y = {y 1 , . . . , y m } ⊂ B δ be a non empty finite subset of B δ . If either k

  2 , then we can apply proposition 9.11 obtaining u j with j ∈ [m] with the local polynomial reproduction property. Define the function s f, Y = j∈[m] f (y j )u j and let τ = max i∈[m] |f (y i )|. Now, by using both propositions 9.11(a) and 9.11(b), we have that for any p ∈ π k (R d ) and any x

  .B . FUNDAMENTAL RESULTS ON SCATTERED DATA APPROXIMATION 425 where C = |α|=k+1 1 α! ∂ α f L ∞ (B δ ) is defined in the lemma. Gathering the previous equations, sup x∈B δ |f (x)| ≤ 2τ + 3Cδ k+1 .

9 .

 9 D . THE CONSTANTS OF TRANSLATION INVARIANT AND SOBOLEV KERNELS427 9.C .1 Proof of Lemma 9.2

  Let a = Ef and b = Eg. Denote by ã, b their Fourier transform and by a • b the Fourier transform of a • b (see proposition 9.2 for more details). By expanding the definition of the Hilbert norm of translation invariant kernel

Lemma 9 .

 9 12 (Technical result). Let α ≥ 1, β ≥ 2 and n ∈ N. If n ≥ 2α log(2βα), then it holds α log(βn) n ≤ 1.

.

  |ĉ -f * | ≤ 2η|f | Ω,m + λ Tr(A * ) ≤ 3λ(|f | Ω,m + Tr(A * )),Thus, under this condition, we have with probability at least 1 -δ,|ĉ -f * | ≤ C m,s,d R m n -m/d (s,d = 6 × 11 m × max(1, 18(m -1) 2 ) m d m m! max(1, MD m ).Step 3. Bounding the constant term C m,s,d in terms of m, s, d. Note thatΓ(m + d/2) Γ(d/2) = (d/2)...(d/2 + m -1) ≤ (d/2 + m -1) m-1 and Γ(sd/2 -m) Γ(sd/2) = 1 (sd/2 -m)....(sd/2 -1) D m ≤ (2π) d/4 d/2 + m -1 sd/2 -m (m-1)/2.Moreover, using the bound on M, we getD m M ≤ 2 s+1/2 (2π) 3d/4 d/2 + m -1 sd/2 -m (m-1)/2This yields the following bound for C m,s,d :C m,s,d ≤ 6 max(1, 18(m -1) 2 ) m (11d) m m! max 1, 2 s+1/2 (2π) 3d/4 d/2 + m -1 sd/2 -mSince f satisfies both Assumptions 9.1(b) and 9.4, denote by ζ the unique minimizer of f in Ω. Since ζ is a strict minimum by Assumption 9.1(b), there existsβ 1 > 0 such that ∇ 2 f (ζ) β 1 I. Thus, since f ∈ C 2 (R d ), there exists a small radius t > 0 such that ∇ 2 f (x) β 1 2 I for all x ∈ B t (ζ) and hence ∀x ∈ Ω ∩ B t (ζ), f (x) -f * = f (x) -f (ζ) -∇f (ζ) ≥ β 1 4 xζ 2 . (9.41)Moreover, since f has no minimizer on the boundary of Ω and since ζ is the unique minimizer of f on Ω, f has no minimizer on K = Ω \ B t (x) which is a compact set. Denote by m the minimum of f on K. Since K is compact, this minimum is reached and since f does not reach its global minimum f * on K, we have mf * > 0. Let R be a radius such that Ω ⊂ B R (ζ), which exists since Ω is bounded. Then, since for any x ∈ Ω, xζ < R, it holds for any x ∈ K :

  .43) 9.F . GLOBAL MINIMIZER. PROOFS.

  2 by assumption, since m ≥ 2. Then we can apply Theorem 9.4 with g = ĝ, τ = 0 and B = B obtaining for all x ∈ Ωf (x) -ν 2 x 2 + νx ẑĉ = ĝ(x) ≥ -η(|ĝ| Ω,m + MD m Tr( B)), η = C 0 h m X,Ω ,where C 0 is defined in Theorem 9.4 andC 0 = 3 (18d) m (m-1) 2m m!since m ≥ 2. Since the inequality above holds for any x ∈ Ω, by evaluating it in the global minimizer ζ ∈ Ω, we havef (ζ) = f * and so -∆ -ν 2 ẑζ 2 = ĝ(ζ) ≥ -η(|ĝ| Ω,m + MD m Tr( B)). Now we bound |ĝ| Ω,m . Since ĝ(x) = f (x) -p ẑ,ĉ (x), where p ẑ,ĉ is a second degree polynomials defined as p ẑ,ĉ = ν 2 x 2 -νx ẑ + ĉ, we have|ĝ| Ω,m ≤ |f | Ω,m + |p ẑ,ĉ | Ω,m ≤ |f | Ω,m + ν, (9.47) since for m = 2, we have |p ẑ,ĉ | Ω,2 = sup i,j∈[d],x∈Ω | ∂ 2 p ẑ,ĉ (x) ∂x i ∂x j | = ν and also |p ẑ,ĉ | Ω,m = 0 for m > 2. Then ∆ ≤ ∆ + ν 2 ẑζ 2 ≤ η|f | Ω,m + ηMD m Tr( B) + ην. (9.48) Conclusion. Combining Eq. (9.48) with Eq. (9.46), since ν 2 ẑζ 2 ≥ 0 and since λ ≥ 2MD m η by assumption, we have λ 2 Tr( B) ≤ (λ -MD m η) Tr( B) ≤ η|f | Ω,m + ην + λ Tr(A * ), from which we obtain Eq. (9.26). Moreover, the inequality Eq. (9.25) is derived by bounding ∆ from below as ∆ ≥ -λ Tr(A * ) by Eq. (9.46), since Tr( B) ≥ 0 by construction, and bounding it from above as ∆ ≤ 2η|f | Ω,m + 2ην + λ Tr(A * ), that is obtained by combining Eq. (9.48) with Eq. (9.26) and with the assumption MD m η ≤ λ/2. Finally from Eq. (9.48) we obtain ν 2 ẑζ 2 ≤ |∆| + η|f | Ω,m + ηMD m Tr( B) + ην, from which we derive the bound ν 2 ẑ -ζ 2 in Eq. (9.24), by bounding |∆| and Tr( B) via Eq. (9.25) and Eq. (9.26).

7 8 λ

 8 Tr( B) -λ Tr(A * ) -θ 1 ≤ ∆, ∆ := cf * . (9.49) Analogously to Step 3 of the proof of Theorem 9.5, by applying Theorem 9.4 to Eq. (9.28) with g(x) = f (x) -c, B = B and τ = τ 1 + τ 2 Tr( B), we obtain for any x ∈ Ω f (x) -c ≥ -2τ 1 -2τ 2 Tr( B) -η(|g| Ω,m + MD m Tr( B)), η = C 0 h m X,Ω , with C 0 defined in Theorem 9.4. Now evaluating the inequality above for x = ζ, noting that |g| Ω,m = |f | Ω,m since m ≥ 1, and considering that by assumption τ 2 ≤ λ/8 and MD m η ≤ λ/2 we derive ∆ = -(f (ζ) -c) ≤ 2τ 1 + 3 4 λ Tr( B) + η|f | Ω,m .(9.50)

  Proof. Define H = {g ∈ C s (Ω) : ∃f ∈ C s (R d ), f | Ω = g},endowed with the following norm : ∀g ∈ H, g H = sup |α|≤s sup x∈Ω ∂ α g(x) .

  H satisfies Assumptions 9.2(a) to9.2(c). Indeed, given u, v ∈ H the first assumption is satisfied as a simple consequence of the Leibniz formula, since for anyx ∈ Ω, ∂ α (u • v)(x) = β≤α α β ∂ β u(x)∂ α-β v(x) which in turn implies that for any |α| ≤ s and x ∈ Ω, ∂ α (u • v)(x) ≤ 2 |α| u H v H and hence u • v H ≤ 2 s u H v H . Assumption 9.2(b) is trivially satisfied and Assumption 9.2(c) is a simple consequence of the dominated convergence theorem. Indeed, if u ∈ H and u ∈ C s (R d ) such that u| Ω = u, define ∀x, z ∈ R d , v z (x) = 1 0

  .51) with C 1 , C 2 depending only on r, s, d, Ω. Now, since f (x) -f * = j∈[p] w 2 j (x), for any x ∈ Ω, we haveff * --w ε j )(2w j -(w j -w ε j )) L ∞ (Ω) ≤ j∈[p]

  x∈Ω |f (x) -f ε (x)| ≤ C f ε s .Now consider the problem in Eq.(9.34) and denote by p ε λ,n its optimum. Since f ε (x i )-c = Φ i BΦ i implies |f (x i ) -c -Φ i BΦ i | ≤ τ , since we required τ ≥ sup x∈Ω |f (x) -f ε (x)|. Then in this case Eq. (9.32) is a relaxation of Eq. (9.34) and we have that p ε λ,n -cλ Tr( B) ≤ 0. So, we can apply Theorem 9.9 to f ε with θ 1 , θ 2 , τ 2 = 0 and τ 1 = τ , obtaining for any m ∈ N and m < sd/2 |c -f ε * | ≤ 14τ + 7η|f | Ω,m + 6λ Tr(A ), Tr( B) ≤ 8 Tr(A ) + 8 η λ |f | Ω,m + 16 τ λ .where f ε * is the infimum of f ε (see Remark 27), and satisfies|f * -f ε * | = | min x∈Ω f (x) -inf x∈Ω f ε (x)| = | inf x∈Ω f (x) -inf x∈Ω f ε (x)| ≤ sup x∈Ω |f (x) -f ε (x)| ≤ τ.By the same reasoning in the proof of Theorem 9.4 used to bound |g| Ω,m , we have that|f ε | Ω,m ≤ MD m Tr(A ε ) ≤ C f MD m ε -2r+2s .Combining together the inequalities above, with the fact that λ ≥ 2MD m η, we have|c -f * | ≤ 10λ C f -2(r-s) + 15τ, Tr( B) ≤ 12 C f -2(r-s) + 16 τ λ .Now we set ε as large as possible such that τ≥ sup x∈Ω |f (x) -f ε (x)| holds. In particular we know that requiring τ ≥ C f ε s guarantees τ ≥ sup x∈Ω |f (x) -f ε (x)|. Then by setting ε = 1 when τ ≥ C f , we have |c -f * | ≤ 10 λ C f + 15τ , Tr( B) ≤ 12 C f + 16 τ λ . By setting ε = (τ /C f ) 1/s when τ ≤ C f , we have |c -f * | ≤ 10 λ C f (C f ) 2 r-s s τ -2 r-s s + 15τ , Tr( B) ≤ 12 C f (C f ) 2 r-s s τ -2and combining the inequality for the two cases above, leads to|c -f * | ≤ C f (λ + λ s 2r-s ), Tr( B) ≤ 12 C f + C f λ -(1-s 2r-s ) .whereC f = max 10C f (C f ) 2 r-s s + 15, 10C f , C f = 12 C f (C f ) 2 r-s s + 169.G .5 Certificate of optimality for the global minimizer candidate of Eq. (9.23) Theorem 9.14 (Certificate of optimality for Eq. (9.23)). Let Ω satisfy Assumption 9.1(a) for some r > 0. Let k be a kernel satisfying Assumptions 9.2(a) and 9.2(d) for some m ≥ 2.Let X = {x 1 , . . . , x n } ⊂ Ω with n ∈ N such that h X,Ω ≤ r 18(m-1) 2 . Let f ∈ C m (Ω) and let ĉ ∈ R, ẑ ∈ R d , B ∈ S + (R n ) and τ ≥ 0 satisfying |f (x i ) -ν 2 x i 2 + νx i ẑĉ -Φ i BΦ i | ≤ τ, i ∈ [n] (9.52)where Φ i are defined in Sec. 9.2 . Let f * = min x∈Ω f (x) and f = ĉν 2 ẑ 2 . Then,|f (ẑ) -f * | ≤ f (ẑ) -f + 2τ + C 1 h m X,Ω , (9.53) ν 2 ζẑ 2 ≤ f (ẑ) -f + 2τ + C 2 h m X,Ω . (9.54) andC 1 = C 0 (|f | Ω,m + MD m Tr( B) + MD m Ĉ), C 2 = C 0 (|f | Ω,m + ν + MD m Tr( B)), where Ĉ = ν 2 R -(X -1 n ζ ) 2, with X ∈ R n×d the matrix whose i-th row corresponds to the point x i and 1 n ∈ R n the vector where each element is 1. The constants C 0 , defined in Theorem 9.4, and m, M, D m , defined in Assumptions 9.2(a) and 9.2(d), do not depend on n, X, h X,Ω , ĉ, B or f .

9 .

 9 H . DETAILS ON THE ALGORITHMIC SETUP USED IN THE BENCHMARK EXPERIMENTS445

  By applying Theorem 9.4 with g = ĝ and B = B we have that for anyx ∈ Ω f (x)-ν 2 x-ẑ 2 -f = ĝ(x) ≥ -ε -2τ , where ε = C 0 (|ĝ| Ω,m + MD m Tr( B))h m X,Ωand C 0 is defined in Theorem 9.4. In particular this implies thatf (ζ) -fν 2 xẑ 2 ≥ -ε -2τ,from which Eq. (9.54) is obtained by considering that f (ẑ) ≥ f (ζ) since ζ is a minimizer of f . To conclude the proof of Eq. (9.54) note that |ĝ| Ω,m ≤ |f | Ω,m + ν since m ≥ 2.

  j ∈ S + (R n ) we have ν 2 x i -ẑ 2 = Φ i ĜΦ i , ∀i ∈ [n].Substitutingν 2 x i 2 + νx i ẑ with ν 2 ẑ 2 -Φ i ĜΦ i in the inequality in Eq. (9.52), we obtain|f (x i ) -f -Φ i ( B + Ĝ)Φ i | ≤ τ, ∀i ∈ [n].By applying Theorem 9.4 with g(x) = f (x) -f andB = B + Ĝ we have that f (x) -f ≥ -ε -2τ for all x ∈ Ω, where ε = C h m X,Ω with C = C 0 (|g| Ω,m + MD m Tr( B + Ĝ)). In particular, f (ζ) -f ≥ -ε -2τ , from which Eq. (9.53) is obtained considering that f (ẑ) ≥ f * since ζ is a minimizer of f . Finally, note that |g| Ω,m ≤ |f | Ω,m since m ≥ 1.The proof is concluded by noting that using the definition of V we have ûj = R -vj with vj ∈ R n corresponding to vj = (u j (x 1 ), . . . , u j (x n )) for j ∈ [d] and that Tr( Ĝ) = ν 2 j∈[d] ûj

CHAPTER 9 .

 9 FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS Algorithm 7 Finding a minimizer given pointsX function FindMinimizer(f, X, k(•, •), λ min , λ max , ε) K = (k(x i , x j )) 1≤i,j≤n ∈ R n×n Φ such that Φ Φ = K (cholesky decomposition) f X = (f (x i )) 1≤i≤n ∈ R n function ScalarFunction(t) λ = e t α solution to Eq. (9.55) with λ, ε, Φ, f X x = n i=1 α i x i f = f ( x) return f , x end function f , x = MinimizeScalar(ScalarFunction ,t min = log(λ min ), t max = log(λ max )) return f , x end functionRecall that the algorithm introduced in sections 6 and 7.1 computes a minimizer by solving problem : α = arg min α∈R n Φ Φ = K for K = (k(x i , x j )) 1≤i,j≤n ∈ R n×n , and choosing x as the approximation of the minimizer, defined by

9 .

 9 H . DETAILS ON THE ALGORITHMIC SETUP USED IN THE BENCHMARK EXPERIMENTS447

Algorithm 8

 8 Converging to the minimumfunction FindMinimizerIter(f, Ω, m, N, k (•) (•, •), Contraction) ε = 10 -3 , λ min = 10 -12 , λ max = 1 X -2 , X -1 = [], [] x 0 , w 0 = c Ω , w Ω for k = 0 to N -1 do w k = Contraction(k)×w 0 σ k = w k /2 X k = [x k ,Uniform(x k , w k , m -1) ] X k = [ X k-2 , X k-1 , X k ] f k+1 , x k+1 =FindMinimizer(f, X k , k σ k , λ min , λ max , ε) end for return f N , x N end functionThe function FindMinimizerIter uses the following parameters:

  Figure 10.1: Plots of functions z = f (x, y), where the zeros of f are highlighted in black. left: f satisfies the SHC, right: f satisfies the NHC but not the SHC.

Example 10. 1 .

 1 Figure10.2: Examples of sub-manifolds of R 2 ; points are denoted with pt. Left: connected sub-manifold of dimension 1 (a circle). Center: a sub-manifold of 4 connected components which are all points, i.e., of dimension 0 (their tangent space is not represented since it is reduced to {0}). Right: a sub-manifold of two connected components, one point pt of dimension 0 and one of dimension 1.

  Figure 10.3: Two examples of sub-manifolds of R 3 . The blue affine spaces represent tangent spaces.Left: connected sub-manifold of dimension 2 (the sphere S 1 ). Right: a sub-manifold of two connected components, one of dimension 2 (homeomorphic to the torus T 2 on which lies x 0 ), and one of dimension 1 on which lies x 1 .

Theorem 10. 5 .

 5 If f satisfies the NHC at x 0 (definition 10.1) with regularity k and dimension d 0 , there exists an open neighborhood U of x 0 in R d on which f is defined and such that U ∩ Z is a sub-manifold of R d of dimension d 0 and of class C max(k,p-1) , and there exist functionsf i ∈ C p-2 (U ) where 1 ≤ i ≤ dd 0 such that ∀x ∈ U, f (x) =

⊥Figure 10 . 5 :

 105 Figure 10.4: Local view of the function around a minimum lying on a 1-dimensional manifold.Top left: function around the minimum x 0 . Top right: decomposition of R 2 at x 0 between tangent space and normal tangent space T x 0 N + T x 0 N ⊥ , and positive eigen-vector of the Hessian in red. Bottom left: reparametrization in the right coordinate system, and representation of the map ϕ(x) given by the Morse Lemma. Bottom right: vector field ξ given by the Morse lemma.

  Figure10.6: Left: Representation of the manifold M = S 2 as well as a sub-manifold N homeomorphic to a circle. The tangent spaces at a given point x 0 ∈ N ⊂ M are represented as well. Right: Representation of a non-negative function on the sphere as a color map; it satisfies the NHC, and its null space Z is represented in black.

  Definition 10.4. Let M be a manifold of class C k , k ≤ k . N is a sub-manifold of M of class C k if for any x ∈ N , and any chart φ : U → R d defined around x and of class C k , φ(U ∩ N ) is a sub-manifold (in the sense of R d , see Sec. 10.2 .1) of R d around φ(x). It is equivalent to ask the existence of one such chart per point x.Let N be a sub-manifold of class C k of a manifold M of class C k . Then it is naturally a manifold of class C k in its own right. Indeed, consider that i) N is equipped with the topology of M , i.e. V is open in N iif V = U ∩ N for some open set of M , and ii) the atlas of N is (the completion of) the set of restrictions of charts φ| U ∩N where φ: U → R d is a C k chart on M such that φ(U ∩ N ) ⊂ R d × {0}, where d is the dimension of N at x ∈ U (we identify R d × {0} ≈ R d ).

  U x an open set in M such that U x ∩ N is included in the unique connected component of x in N . Indeed, by definition 10.4, there exists a chart φ : U → N where U is an open neighborhood of x. But since φ(U ∩ N ) is a sub-manifold of R d of class C k around φ(x), by Theorem 2.5 by Paulin (2006), there exists a C k diffeomorphism

Conjecture 11. 1 :

 1 Minimax upper bounds for GSC lossesFix b 1 , b 2 > 0, r > 1/2, Q > 0,R > 0, and b ≥ 1 and let M = M(R, r, Q, b, b 1 , b 2 ), as defined in main corollary 3. Let θ denote the variables θ = (Q, R). Setting λ n,θ = min Q 2 R 2 n 1 2br+1 , 1 , a n,θ = R 2 Q 2 R 2 n 2br 2br+1 ,(11.2)for all ψ(•) = | • | p for p > 0, the following minimax rates of convergence hold with rate a n

  Conjecture 11.2: Number of functions in the decompositionUnder the same assumptions as Main theorem 15, there exists a finite number of functionsf 1 , ..., f N ∈ C p-2 (M ) with N = O(d 3 ) such that ∀x ∈ M, f (x) =

  Il existe donc un isomorphisme naturel entre H φ et H, et H peut donc être naturellement équipé de la structure de Hilbert de H φ . Le fait que k est le noyau reproduisant de H en découle. Notons que l'isomorphisme entre H et H φ montre directement que H = span({k H φ . De plus, le produit interne de H, bien qu'hérité de celui de H, n'est pas exactement le même. En effet, si f i

x : x ∈ X }). En fait, la représentation x ∈ X → k x ∈ H est une façon de représenter X dans l'espace de Hilbert H (c'est une représentation). La raison pour laquelle nous distinguons H et H est que l'espace H est un espace de fonctions sur X alors que H ne l'est pas (il peut être plus grand). En effet, tout espace de Hilbert sur lequel existe un encastrement φ : X → H contient un sous-espace isométrique à H,

  models for non-negative functions. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12816-12826. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper/2020/file/968b15768f3d19770471e9436d97913c-Paper.pdf • Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Sampling from arbitrary functions via psd models. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 2823-2861. PMLR, 28-30 Mar 2022a. URL https://proceedings.mlr.press/v151/marteau-ferey22a.html

  1.4. SOMMES DE CARR ÉS DE FONCTIONS ET OPTIMISATION GLOBALEAssumption 1.3. Tous les minimiseurs de f sont des minimiseurs globaux stricts, i.e., la Hessienne en tous les minimiseurs est définie positive.Le théorème suivant est une simplification de Theorem 9.3. En particulier, notez que la condition r ≥ s + 2 implique que r > d/2 + 2. Supposons que r -2 ≥ s et rappelons que H est l'ENR associé au noyau de Sobolev k s . Sous les hypothèses ci-dessus (y compris Assumptions 1.2 and 1.3) sur le noyau et la fonction, il existe un opérateur de rang fini A * ∈ S + (H) avec un rang au plus égal à d|Z| + 1 (où |Z| est la taille de l'ensemble des minimiseurs), tel que :

	Main theorem 4: Rudi et al. (2020), Theorem 3.

  sont les suivantes. Pour généraliser des domaines de R d aux variétés, on note simplement que le résultat a été obtenu dans le cas de R d en collant des décompositions. Le même raisonnement peut être appliqué pour les variétés, et donc le véritable coeur du problème est de montrer que certaines fonctions régulières positives peuvent être localement décomposées comme des sommes de carrés de fonctions régulières sur R d (en composant avec une carte). Dans les travauxde Rudi, Marteau- Ferey, and Bach (2020), la stratégie pour montrer une décomposition locale en sommes de carrés consistait à utiliser le fait que la Hessienne au point optimal était définie positive (Assumption 1.3) pour écrire une décomposition exacte du second ordre, ce qui faisait naturellement apparaître une décomposition en sommes de carrés (comme celles obtenues par le lemme traditionnel de

Morse, voir Milnor (1963)

). Dans les travaux de

Marteau-Ferey, Bach, and Rudi (2022b)

, ceci est adapté au cas où l'ensemble des zéros est une sous-variété de R d , et Assumption 1.3 est modifiée pour permettre un ensemble continu de zéros. Au lieu de cela, il impose qu'orthogonalement à la variété de zéros, la Hessienne soit définie positive. Ces deux situations et leur différence sont illustrées dans ??. En notant T x N l'ensemble des vecteurs tangeants à une variét'e N em x, nous prouvons le théorème suivant (voir chapter 10 pour plus de détails et de références).

  2.2.

	underfitting	estimation error approximation error	overfitting
		excess risk of ERM
	error
		size of 

  ). Let k be a p.d. kernel on a space X . Then there exists a unique RKHS H with reproducing kernel k.

	Definition as a Hilbert space of functions with the reproducing property. The last
	way of defining a reproducing kernel Hilbert space is through the function space itself. Assume
	you are given a Hilbert space of functions H. It is a reproducing kernel Hilbert space if, and only
	if the evalutations mappings f ∈ H → f (x) are continuous. Indeed, in that case, by the Riesz
	representation theorem, there exists an element k x ∈ H such that f (x) = f, k x H for all f ∈ H.
	Thus H is a reproducing kernel Hilbert space with kernel k(x, x ) := k x , k x H .

  .., x n , has a minimizer in H n = span({k x 1 , ..., k xn }). This is easily proved by decomposing f on H n and its orthogonal. There are two valuable consequences of this theorem.

	The first is that the empirical risk minimization problem Eq. (2.47) can be formulated as a
	n-dimensional problem :
	α ∈ arg min α∈R n	1 n	n i=1	z i (e i Kα) +	λ 2	α Kα,	(2.50)

  Fast rates for least-squares . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.2 Generalized self-concordance . . . . . . . . . . . . . . . . . . . . . . . 107 3.3 Main results and contributions of this part . . . . . . . . . . . . . . . 114

	Table of Contents Chapter 3
	Backround and main results
	3 Backround and main results
	4 Beyond least squares Contents
	5 Second order strikes back 3.1

  .3) Empirical operators. Given the n samples z 1 , ..., z n ∼ Z, we can define the analog finite dimensional operators S n : f ∈ H → (f (x i )) 1≤i≤n ∈ R n , where R n is equipped with the scalar product associated to the mean : (α|β) = 1

	S * n : α → 1 n	n i=1 α i β i . When equipped with that structure, i=1 α i k x i (recall that the adjoint satifies (α| S n f ) = S * n n n α, f H ). The empirical
	covariance is defined as

  ), and computing gradients of the above functional would cost O(n 2 ) per gradient. Moreover, solving the linear system would take O(n 3 ) (indeed, α n,λ = (K + nλI) -1 Ky). Either way, we seem not to be able to escape a cost of O(n 2 ) if not O(n 3 ), which is prohibitive in the settings we wish to consider (i.e. n in the order of millions or billions of points).

	3.1. FAST RATES FOR LEAST-SQUARES	99
	of order O(n 2 The goal of dimension reduction is to find a low dimensional set of functions H m of dimension
	m, equipped with a RKHS structure, such that replacing H by H m in Eq. (2.47) does not cost
	anything statistically. If f n,λ,m is the solution to
	leading to the n-dimensional problem
	α n,λ = arg min α∈R n	1 n	Kα -y 2 + λα Kα.	(2.50)
	where K = (k(x i , x j )) 1≤i,j≤n is called the kernel matrix. The fact that the problem Eq. (2.50) is
	n-dimensional is computationally limiting : computing the full matrix K already has a complexity

  Proposition 4.8. Let λ ≥ 0. If a probability measure µ and satisfy Assumptions 4.3, 4.4 and 4.8, the function L µ,λ (θ) := E µ [ z (θ)] + λ θ 2 and ∇L µ,λ (θ), ∇ 2 L µ,θ (θ) are well-defined for any θ ∈ H, and we can differentiate under the expectation. Moreover,

	∀θ ∈ H,	sup
		z∈supp(ρ)

  this condition is satisfied in simple cases; see examples in Sec. 6.B .5). Denoting with L * the Fenchel conjugate of L (

  Proof of proposition 6.5. Let p, h ∈ N and let C ∈ R p×h be a matrix representing the column vectors c 1 ...c h . Let Y be the polyhedral cone defined by C, i.e. Y = y ∈ R p : C y ≥ 0 . Let H be a separable Hilbert space and φ : X → H be a fixed feature map. With our previous notations, our goal is to prove that for any A

	CHAPTER 6. PSD MODELS

H 1 ) p and any bounded linear operator L : H 1 → H 2 , LAL * will be a slight abuse of notation to denote the family (LA s L * ) 1≤s≤p ∈ S(H 2 ) p .

  .11) Combining the result of Theorems 7.1 and 7.2, we have that, given a PSD model on m centers, an hyper-rectangle of interest Q and an error ρ, algorithm 2 provides N i.i.d.samples whose distribution is distant √ dρ in terms of W 1 from the density represented by the PSD model over the hyper-rectangle. In particular, algorithm 2 computes the N i.i.d. samples with a cost of O(N m 2 d (log 2 (2|Q|) + d log 2 (2/ρ))).

Table 7 .

 7 1: Main computing times (% of the CPU time)

	PART	MAIN OPERATION TIME
	Integration	Eqs. (7.3) to (7.5)
	Computing K X,η/2		71%
	Computing X		6%
	Computing G X,2η,Q Computing A, B	8%
		Calls to erf	6%
		Other	8%
	Other		1%
	Sampling	algorithm 2
	Computing I(Q)	Calls to erf	34%
		Computing A, B Mulitplications √ η	26% 11%
		Other	24%
	Other		5%

  .34) Note on polynomials. As we will see in Sec. 8.3 , the relaxation presented above corresponds exactly to the polynomial unconstrained moment-SOS relaxations, if we consider the RKHS H = R r [x] (the RKHS structure can easily be obtained, see Sec. 8.3 ).

  .52)In Figs.8.2 and 8.3 , we give examples of what functions which satisfy this assumption look like, when defined on R d and on a manifold. While we have not formally done so yet, this result allows to apply exactly the same methodology as for functions with strict minima in open sets of R d , with, we believe, the same guarantees.

  Op(g) ∈ S + (H) ,(8.53) assuming of course that the function 1 belongs to H ⊗2 . We have yet to study this set of moments in the kernel setting, and its link with extraction procedures. Ulysse Marteau-Ferey, and Francis Bach. Finding global minima via kernel approximations, 2020. URL https://arxiv.org/abs/2012.11978. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Outline of contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Equivalence of the infinite-dimensional problem . . . . . . . . . . . . 9.5 Properties of the finite-dimensional problem . . . . . . . . . . . . . . 9.6 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Finding the global minimizer . . . . . . . . . . . . . . . . . . . . . . . 9.8 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9 Relationship with polynomial hierarchies . . . . . . . . . . . . . . . . 9.10 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.11 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.A Additional notations and definitions . . . . . . . . . . . . . . . . . . . 9.B Fundamental results on scattered data approximation . . . . . . . . 9.C Auxiliary results on RKHS . . . . . . . . . . . . . . . . . . . . . . . . 9.D The constants of translation invariant and Sobolev kernels . . . . . 9.E Proofs for algorithm 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.F Global minimizer. Proofs. . . . . . . . . . . . . . . . . . . . . . . . . . 9.G Proofs for the extensions . . . . . . . . . . . . . . . . . . . . . . . . . . 9.H Details on the algorithmic setup used in the benchmark experiments445

	Chapter 9
	Finding global minima via kernel
	approximations
	This chapter is a verbatim of the work :
	Alessandro Rudi, Contents
	9.1

  We consider the general problem of unconstrained optimization. Let f : R d → R be a possibly non-convex function. Our goal is to solve the following problem

	9.1 Introduction
	min x∈R d	f (x).	(9.1)

  Assumption 9.2(d) guarantees that H ⊆ C m (Ω) and |u| Ω,m ≤ D m u H .Note that Assumptions 9.2(a) to 9.2(c) are the only required in Sec. 9.4 to prove the crucial decomposition in Theorem 9.2 and are satisfied by notable spaces (that are not necessarily RKHS) like C s (Ω) or Sobolev spaces W s p (Ω) with s > d/p and p ∈ [1, ∞]. Instead, Assumption 9.2(d) is required for the analysis of the finite-dimensional problem and in particular Theorems 9.4 and 9.5. In the following proposition we show that W s 2 (Ω) with s > d/2 and Ω satisfying Assumption 9.1(a) satisfy the whole of Assumption 9.2. Proposition 9.1 (Sobolev kernels satisfy Assumption 9.2). Let Ω be a bounded open set of R d . The Sobolev kernel with s > d/2 recalled in Example 9.1 satisfies Assumption 9.2 for any m ∈ N + , m < sd 2 and

	Assumptions 9.2(a) to 9.2(c) above require essentially that functions in H (a) can be multiplied
	by other functions in H, or by infinitely smooth functions, and still be in H; (b) that can
	be composed with infinitely smooth functions, or (c) integrated, and still be in H. Moreover
	Assumption 9.2(d) requires that H is a RKHS with a kernel that is m-times differentiable. An
	interesting consequence of Assumption 9.2(d) is the following remark (for more details, see, e.g.,
	(Steinwart and Christmann, 2008, Corollary 4.36)).
	Remark 26.

  .28) Theorem 9.9 (Error of approximate solutions of Eq. (9.5)). Let (c, B) be an approximate solution of Eq. (9.5) for a given n ∈ N + , λ > 0 as defined in Eqs. (9.27) and (9.28) w.r.t. τ 1 , τ 2 , θ 1 , θ 2 ≥ 0. Under the same assumptions and notation of Theorem 9.5 and Remark 27, when τ 2 , θ 2 ≤ λ

			8
	|c -f * | ≤ 7(2τ 1 + η|f | Ω,m ) + 6(θ 1 + λ Tr(A * )),	(9.29)
	Tr( B) ≤ 8 Tr(A * ) + 8 η λ |f | Ω,m + 8 θ 1 +2τ 1 λ	.	(9.30)

  endowed with the norm • Ω,s defined in Eq. (9.31) (and is always finite on H since Ω is bounded) satisfies such assumptions. The reasoning above leads to the following corollary of Theorem 9.2 (the details can be found in Sec. 9.G .2 page 442). Let Ω be a bounded open set and f ∈ C s+2 (R d ), s ∈ N, satisfying Assumption 9.1(b). Then there exist w 1 , . . . , w p

	Corollary 9.2.

  CHAPTER 9. FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS rich enough that the associated RKHS H contains smooth functions (see Assumption 9.2(a)), then we have that w ε j | Ω ∈ H for any j ∈ [p]. Then

	A

for explicit construction in terms of convolutions with smooth functions). Denote by w ε j the smooth approximation of w j on Ω for any j ∈ [p]. Since we consider kernels

  Multivariate case d = 8. Minimization error of our algorithm (gloptikernel) compared with random evaluations or random evaluations + GD. The function considered is built as described at the beginning of this section with domain [-1, 1] d and shifted and rescaled to have minimum in 0 and output in [0, 1]. Left: function without small-amplitude high-frequency components. Right: function with small-amplitude high-frequency components.

		Without high-frequency component	With high-frequency component
		0.5 0.6					0.5 0.6
	normalized function values	0.1 0.2 0.3 0.4	Gloptikernel random random+GD		normalized function values	0.1 0.2 0.3 0.4	Gloptikernel random random+GD
			100	300	1000	3000		100	300	1000	3000
				n					n
	Figure 9.2:

http://infinity77.net/global optimization/index.html.

Table 9 .

 9 3: Results of the experiments when using both gloptipoly3 and our method. the underlying domain; (c) the degree deg of the polynomial function; (d) wether or not gloptipoly3 has converged (cv column); (e) the relaxation order we have tested for before computational issues (relax column); (f) the dimensions of the PSD constraints for the Lasserre hierarchy (PSD moment matrix + the ones due to the constraints); (g) the gap between our solution and the Lasserre lower bound; (h) the dimension of the PSD constraint in our mehod.

	d	deg cv	relax PSD dim glopti	gap	PSD dim (ours)
	Poly1 2	6	True 3	10x10+4x(6x6)	0.0000E+00 63x63
	Poly3 7	6	True 4	330x330+14x(120x120) 5.0819E-02	138x138
	Poly4 7	4	False 4	330x330+14x(120x120) 1.9073E-05	138x138
	Poly2 8	6	True 3	165x165+16x(45x45)	1.7166E-04	153x153
	Poly9 9	6	False 3	455x455+20x(91x91)	5.1392E-02	168x168
	Poly5 10 4	False 3	286x286+20x(66x66)	2.4796E-05	183x183
	Poly8 12 4	False 2	91x91+24x(13x13)	3.6388E-02	213x213
	Poly6 15 4	False 2	136x136+30x(16x16)	2.2190E-02	258x258
	Poly7 17 4	False 2	171x171+34x(18x18)	1.6233E+00 288x288

  1. Covering Ω. Let t > 0. We say that a subset X of Ω is a t (interior) covering of Ω if Ω ⊂ x∈X B t (x). Denote with N t the minimal cardinal |X| of a t interior covering of Ω and fix X t a t interior covering of Ω whose cardinal is minimum, i.e.,|X t | = N t . Since the diameter of Ω is bounded by 2R, it is known that N t ≤ (1 + 2R/t) dTo prove this fact , one defines a maximal t/2-packing of Ω as a maximal set Y t/2 ⊂ Ω such that the balls B t/2 (y) are disjoint. It is then easy to check that if Y t/2 is a maximal t/2-packing, then it is also a t-covering and hence N t ≤ |Y t/2 |. Finally, since Ω is included in a ball of radius B 2R (x 0 ) for some x 0 ∈ R d and since Y t/2 ⊂ Ω, it holds y∈Y t B t (y) ⊂ B R+t/2 (x 0 ). Since the B t (y) are two by two disjoint, the result follows from the following equation:

  3. Bounding vol(B t (x) ∩ Ω) when t ≤ r. Let us now find a lower bound for min x∈Ω vol(B t (x) ∩ Ω). Recall that since Ω satisfiesAssumption 9.1(a), Ω can be written Ω = ∪ z∈S B r (z). .Let t ≤ r, x ∈ Ω. By the previous point, there exists z ∈ S such that x ∈ B r (z) ⊂ Ω and hence B t (x) ∩ B r (z) ⊂ B t (x) ∩ Ω. Let C x,z,t denote the cone centered in x and directed to z with aperture π/3. It is easy to see geometrically that B r (z) ∩ B t (x) contains the cone C x,z,t (this fact is proved in Lemma 9.13). Moreover, using the lower bound for the volume of this cone provided inLemma 9.13, it holds: vol

Controle de l'erreur d'approximation : universalité. La manière standard de décomposer l'excès de risque du mimiseur de risque empirique f n,λ consiste à séparer l'erreur d'approximation et l'erreur d'estimation : R(f n,λ ) -R * = (R H -R * ) + (R( f n,λ ) -R H ).Dans les méthodes à noyaux, comme nous le verrons dans le paragraphe suivant, on ne s'intéresse généralement qu'à l'erreur d'estimation.

√ λ R ).

Note that the model in Eq. (6.4) has already been considered by[START_REF] Bagnell | Learning positive functions in a Hilbert space[END_REF] with a similar goal as ours. However, this workshop publication has only be lightly peer-reviewed, the representer theorem they propose is incorrect, the optimization algorithm is based on an incorrect representation and inefficient at best. See Appendix 6.F for details.

Given the eigendecomposition A = UΛU with U ∈ R n×n unitary and Λ ∈ R n×n diagonal, then [A]-= UΛ-U , with Λdiagonal, defined as (Λ-)i,i = min(0, Λi,i) for i = 1, . . . , n.

f A g,Z, (x) = φ(x) A g,Z, φ(x) = φ(x) w √ g,Z, Q ⊗ w √ g,Z, Q φ(x) = (φ(x) w √ g,Z, Q ) 2 .(6.16)

See Lemma 7.5 in Appendix 7.C .1 for a proof of a the bound on Lip ∞ (f ) when f is a PSD model and Lemma 7.2 in Appendix 7.B .1 for a proof of of a bound on Lip ∞ ( √ f ) in the case where f is a rank one PSD model.

The code is available at https://github.com/umarteau/sampling psd models

(Ω). Let ĉ be the result of algorithm 6 executed with n ∈ N + points chosen

The performance on certain problems is bad, but this seems to be linked to the fact that the functions at hand have very high oscillations (hence high derivatives).

A topological space is said to be second countable if there exists a countable sequence of open sets Un such that any open set U in the topology is a reunion of a part of the Un.

A topological space is Hausdorff if for any two points x = x , there exists two open sets U, V such that x ∈ U and x ∈ V and U ∩ V = ∅

An isometry is simply a map which preserves distances, and can be defined as an orthogonal transformation plus an affine shift.

Remerciements

"Je deumerai longtemps errant dans Césarée" Racine, Bérénice.

Acknowledgments

This work was funded in part by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute). We also acknowledge support of the European Research Council (grant SEQUOIA 724063).

6.A . NOTATION AND BASIC DEFINITIONS

Acknowledgements. This work was funded in part by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001(PRAIRIE 3IA Institute). We also acknowledge support from the European Research Council (grants SEQUOIA 724063 and REAL 947908), and support by grants from Région Ile-de-France.

Acknowledgements. We would like to thank Jean-Bernard Lasserre and Edouard Pauwels for their feedback on an earlier version of the manuscript. This work was funded in part by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute). We also acknowledge support from the European Research Council (grant SEQUOIA 724063).

Chapter 5

Globally convergent newton methods for ill-conditioned generalized self-concordant losses

5.C Proof of bounds for the globalization scheme

In this section, we prove that the scheme of decreasing µ towards λ converges.

5.C .1 Main technical lemmas

Lemma 5.13 (Next µ). Let µ > 0, c < 1.

c rµ(x)

c rµ(x)

.

Proof. For any µ < µ, note that

, and in particular that 1 r µ (x) ≤ µ/ µ 1 rµ(x) .

Using this fact, it holds:

r µ (x) .

Hence, if ν µ (x) ≤ c 3 , a condition to obtain ν µ (x) ≤ c is the following:

r µ (x) .

This yields the second point of the lemma. The analysis is completely analoguous for the first.

Lemma 5.14 (Useful bounds for q). Let µ > 0. Then the following hold:

Moreover, we can bound all of these quantities using x µ :

• For any c < 1, x ∈ H, if x ∈ D µ (c/3), then the following holds:

.

CHAPTER 6. PSD MODELS Theorem 6.3 (Universality, P2). Let H be a separable Hilbert space, φ : X → H a universal map [START_REF] Micchelli | Universal kernels[END_REF], and • • • . Then F • φ is a universal approximator of non-negative functions over X .

The fact that the proposed model can approximate arbitrarily well any non-negative function on X , when φ is universal, makes it a suitable candidate in the context of nonparametric approximation/interpolation or learning [START_REF] Wendland | Scattered Data Approximation[END_REF][START_REF] Alexandre | Introduction to Nonparametric Estimation[END_REF] of non-negative functions. In the following theorem, we give a more precise characterization of the functions contained in F • φ . Denote by G φ the set of linear models induced by φ, i.e., G φ = {w φ(•) | w ∈ H} and by E φ the set of exponential models induced by φ,

Theorem 6.4 (F • φ strictly richer than the exponential model).

for some m > 0, where W m 2 (X ) is the Sobolev space of smoothness m [START_REF] Adams | Sobolev Spaces[END_REF]. Let x 0 ∈ X . The following hold:

Theorem 6.4 shows that if φ is rich enough, then the space of exponential models is strictly contained in the space of functions associated to the proposed model. In particular, the proposed model can represent functions that are exactly zero on some subset of X as showed by the example f x 0 in Theorem 6.4, while the exponential model can represent only strictly positive functions, by construction. Discussion on the condition W m 2 (X ) = G φ , proof of Theorem 6.4 and its generalization to X ⊆ R d are in App. 6.B .7. Here we note only that the condition W m 2 (X ) = G φ is quite mild and satisfied by many kernels such as the Abel kernel k(x, x ) = exp(-xx ) [START_REF] Wendland | Scattered Data Approximation[END_REF][START_REF] Berlinet | Reproducing Kernel Hilbert Spaces in Probability and Statistics[END_REF]. We conclude with a bound on the Rademacher complexity [START_REF] Boucheron | Theory of classification: A survey of some recent advances[END_REF] of F • φ , which is a classical component for deriving generalization bounds [START_REF] Shalev | Understanding Machine Learning: From Theory to Algorithms[END_REF]. Define F • φ,L = {f A | A 0, A • ≤ L}, for L > 0. Theorem 6.5 shows that the Rademacher complexity of F • φ,L depends on L and not on the dimensionality of X , as for regular kernel methods [START_REF] Boucheron | Theory of classification: A survey of some recent advances[END_REF]. Theorem 6.5 (Rademacher complexity of F • φ ). Let • • • F and sup x∈X φ(x) ≤ c < ∞. Let (x i ) n i=1 be i.i.d. samples, L ≥ 0. The Rademacher complexity of F • φ,L on (x i ) n i=1 is upper bounded by 2Lc 2 √ n (proof in Sec. 6.B .8).

Extensions: Integral Constraints and Output in Convex Cones

In this section we cover two extensions. The first one generalizes the optimization problem in Eq. (6.5) to include linear constraints on the integral of the model, in order to deal with problems like density estimation in Example 6.1. The second formalizes models with outputs in convex cones, which is crucial when dealing with problems like multivariate quantile estimation [START_REF] Howard | Noncrossing quantile regression curve estimation[END_REF], detailed in Sec. 6.6 .

Constraints on the integral and other linear constraints. We can extend the definition of the problem in Eq. (6.5) to take into account constraints on the integral of the model. Indeed by linearity of integration and trace, we have the following (proof in Sec. 6.B .9).

Proof. In this proof, denote by J the function defined by ∀A ∈ S(H), J(A) := L(f A (x 1 ), ..., f A (x n )) + Ω(A).

Our goal is to prove that the problem inf A∈S(H) + J(A) has a solution which is in S n (H) + , i.e., of the form Π n AΠ n for some A ∈ S(H) + .

1. Let us start by fixing A ∈ S(H) + . First note that since Π n is the orthogonal projection on span(φ(x i )) 1≤i≤n , in particular Π n φ(x i ) = φ(x i ) for all 1 ≤ i ≤ n. Thus, for any 1

Here, the first and last equalities come from the definition of f A and f ΠnAΠn . Thus, J(A) = L(f ΠnAΠn (x 1 ), ..., f ΠnAΠn (x n )) + Ω(A). Now since Ω satisfies Assumption 6.1, by the second point of Lemma 6.2, it holds Ω(Π n AΠ n ) ≤ Ω(A), hence J(Π n AΠ n ) ≤ J(A).

This last inequality combined with the fact that S n (H) + = Π n S(H) + Π n ⊂ S(H) + show that inf A∈Sn(H) + J(A) = inf A 0 J(A). (6.15)

2. Let us now show that inf A∈Sn(H) + J(A) has a solution. Let us exclude the case where J = +∞, in which case A = 0 can be taken to be a solution.

Let V n be the injection V n : H n → H. Note that V n V * n = Π n and V * n V n = I Hn . These simple facts easily show that

Thus, our goal is to show that inf Ã∈S(Hn) + J(V n AV * n ) has a solution.

By the first point of Lemma 6.2, since V * n V n = I Hn , it holds

Let Ã0 ∈ S(H n ) + be a point such that J 0 := J(V n Ã0 V * n ) < ∞. Let c 0 be a lower bound for L. By the third point of Lemma 6.2, there exists a radius R 0 such that for all à ∈ S(H n ), à F > R 0 =⇒ Ω( Ã) > J 0 -c 0 .

Since c 0 is a lower bound for L, this implies inf Ã∈S(Hn) + J(V n ÃV * n ) = inf Ã∈S(Hn) + , à F ≤R 0 J(V n ÃV * n ). Now since L is lower semi-continuous, Ω is lower semi-continuous by the last point of Lemma 6.2, and à → (f Vn ÃV * n (x i )) 1≤i≤n is linear hence continuous, the mapping A → J(V n ÃV * n ) is lower semi-continuous. Hence, it reaches its minimum on any non empty compact set. Since H n is finite dimensional, the set à ∈ S(H n ) + : à F ≤ R 0 is compact (closed and bounded) and non empty since it contains Ã0 , and hence there exists à * ∈ S(H n ) + such that J(V n à * V * n ) = inf Ã∈S(Hn) + , à F ≤R 0 J(V n ÃV * n ). Going back up the previous equalities, this shows that A * = V n à * V * n ∈ S n (H) + and J(A * ) = inf A 0 J(A).

CHAPTER 6. PSD MODELS Proof of Theorem 6.1

We will prove the following Theorem 6.7 whose statement is that of Theorem 6.1 with more general assumptions.

Theorem 6.7. Let L be lower semi-continuous and bounded below, and Ω satisfying Assumption 6.1. Then Eq. (6.5) has a solution A * which can be written in the form n i,j=1

B ij φ(x i )φ(x j ) , for some matrix B ∈ R n×n , B 0.

Moreover, if L is convex, and Ω is of the form Eq. (6.6) with λ 2 > 0, this solution is unique. By Eq. (6.4), A * corresponds to a function of the form

B ij k(x, x i )k(x, x j ).

Lemma 6.3. The set S n (H) + can be represented in the following way

In particular, for any A ∈ S n (H) + , there exists a matrix B ∈ R n×n , B 0 such that A = 1≤i,j≤n B i,j φ(x i )φ(x j ) =⇒ ∀x ∈ X , f A (x) = 1≤i,j≤n B i,j k(x i , x)k(x j , x).

Proof. Define S n : H → R n to be the operator such that ∀h, S n (h) = h φ(x i ) 1≤i≤n , with adjoint S * n : R n → H such that

Note that for any B ∈ R n×n , S * n BS n = i,j B i,j φ(x i )φ(x j ) .

1. Proving S n (H) + ⊂ 1≤i,j≤n B i,j φ(x i )φ(x j ) , : B ∈ R n×n , B 0 . Let Π n AΠ n ∈ S n (H) + . Using the previous equality, we want to show there exists B ∈ R n×n , B 0 such that Π n AΠ n = S * n BS n . Using Lemma 6.4, we see that Π n can be written in the form S * n T n where T n : H → R n (write Π n = O n O * n and note that O n is of the form S * n Õn ). Hence, defining B to be the matrix associated to the operator T n AT * n : R n → R n , it holds Π n AΠ n = S * n BS n . Moreover, A 0 implies B = T n AT * n 0.

6.B . PROOFS AND ADDITIONAL DISCUSSIONS 277

Proving

1≤i,j≤n B i,j φ(x i )φ(x j ) , : B ∈ R n×n , B 0 ⊂ S n (H) + . Let B ∈ R n×n , B 0. Since B 0, A := S * n BS n 0. Since S * n has its range included in H n , Π n S * n = S * n . Thus, Π n AΠ n = A and hence A ∈ S n (H) + .

The second statement comes from the definition of f A (x). Indeed assume A ∈ S n (H) + . By definition, f A (x) = φ(x) Aφ(x). Moreover, by the previous point, there exists B ∈ R n×n , B 0 such that A = 1≤i,j≤n B i,j φ(x i )φ(x j ) . Combining these two facts yields:

The last equality comes from the definition k(x, x) = φ(x) φ(x).

Proof of Theorem 6.7. Under the assumptions of Theorem 6.7, one satisfies the assumptions of proposition 6.7. Thus, Eq. ( 6.5) has a solution A * which is in S n (H) + . Now applying Lemma 6.3, A * can be written in the form

Uniqueness in the case where Ω is of the form Eq. (6.6) with λ 2 > 0 comes from the fact that the loss function is strongly convex in this case, and thus the minimizer is unique.

6.B .4 Proof of proposition 6.3

Recall the definitions of S n : H → R n and its adjoint S * n : R n → H :

Note that the kernel matrix K = (k(x i , x j )) 1≤i,j≤n can also be written as K = S n S * n . Let r be the rank of K and V ∈ R r×n be a matrix such that

Note that V is of rank r and hence VV is invertible, making the following definition of O n : R r → H valid:

n Õn for a certain Õn and the range of S * n is span(φ(x i )) 1≤i≤n . Finally since the rank of S * n is also the rank of S n S * n which is r, we deduce that the range of span(φ(x i )) 1≤i≤n is of dimension r and hence, since

n is of rank r, putting things together, Πn = Π n .

CHAPTER 6. PSD MODELS Theorem 6.8 (Convex dual problem). Let L : R n → R ∪ {+∞} be convex closed function and L * be the Fenchel conjugate of L (see the book by [START_REF] Boyd | Convex optimization[END_REF] for the definition of closed and of the dual conjugate). Assume Ω is of the form Eq. (6.6). Assume there exists A ∈ R r×r , A 0 such that L is continuous in ( fA (x i )) 1≤i≤n .

Then the problem in Eq. (6.9) has the following dual formulation, (6.10) and this supremum is atteined. Let α * ∈ R n be a solution of (6.10). Then, the solution of (6.5) is obtained via (6.7), with B ∈ R r×r , B 0 as

Proof of Theorem 6.8. We apply theorem 3.3.1 by [START_REF] Borwein | Convex analysis and nonlinear optimization: theory and examples[END_REF] with the following parameters (on le left, the ones in theorem 3.3.1 by [START_REF] Borwein | Convex analysis and nonlinear optimization: theory and examples[END_REF] and on the right the ones by which we replace them).

Indeed, for all 1 ≤ i ≤ n, if Φ is defined in Eq. (6.8), Φ(x i ) = Ve i and thus fA (

The following properties are satisfied :

• L is lower semi-continuous, convex and bounded below hence closed [START_REF] Borwein | Convex analysis and nonlinear optimization: theory and examples[END_REF];

• similarly, Ω + is a non negative closed convex function, with dual Ω * + given in Lemma 6.5 which is differentiable and smooth;

• The dual d can therefore be re-expressed as Eq. (6.10), using the expressions for Ω * + and R * :

• Assume there exists A ∈ R r×r , A 0 such that L is continuous in ( fA (x i )) 1≤i≤n . Then there exists a point of continuity of g such which is also in R dom f , hence the assumption of theorem 3.3.1 by [START_REF] Borwein | Convex analysis and nonlinear optimization: theory and examples[END_REF] is satisfied.

Applying theorem 3.3.1 by [START_REF] Borwein | Convex analysis and nonlinear optimization: theory and examples[END_REF], the following properties hold:

6.B . PROOFS AND ADDITIONAL DISCUSSIONS

281

• d is atteined for a certain α * ∈ R n . Indeed, there exists A ∈ dom Ω + such that R(A) ∈ dom(L). Thus , L(R(A)) + Ω + (A) < +∞ and hence d < +∞. Moreover, since L and Ω + are lower bounded, this shows that d is lower bounded and hence d > -∞. Hence d is finite and thus is atteined by theorem 3.3.1. Now using Exercise 4.2.17 by [START_REF] Borwein | Convex analysis and nonlinear optimization: theory and examples[END_REF] since L and Ω + are closed convex and since Ω * + is differentiable, we see that the optimal solution of the primal problem A * is given by the following formula:

Thus, for any x ∈ X , using the definition of Φ(x), it holds

Thus, setting

, this shows the result. In particular, note that when V is invertible (i.e. when K is full rank) then the equation above is exactly Eq. (6.11), since V (VV ) -1 = V -1 .

Proof of Theorem 6.2. It is a direct consequence of the previous theorem.

Note that the conditions of theorem Theorem 6.2 are satisfied in many interesting cases, such as the ones described in the following proposition.

Proposition 6.8. Assume one of the following conditions is satisfied :

Then there exists A ∈ S(R n ) + such that L is continuous in ( fA (x 1 ), ..., fA (x n )).

Proof. Let us prove these points.

• if dom(L) = R n , since L is convex, L is continuous everywhere. Taking A = 0, the result holds.

• if k(x i , x i ) > 0 for all i > 0, then taking A = I r , we have ( fA

++ and hence, A satisfies the desired property.

CHAPTER 6. PSD MODELS

• Let α 0 be a continuity point of L in R n + . If we assume K is full rank, then in particular, V ∈ R n×n is of rank n and invertible. Thus, there exists A ∈ S(R r ) + such that

Discussion on how to solve Eq. (6.10) Proximal splitting methods can be applied to solve Eq. ( 6.10) such as FISTA [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], provided the proximal operator of L * can be computed (see the work by [START_REF] Parikh | Proximal algorithms[END_REF] for the definition of the proximal operator). Indeed, Eq. (6.10) can be written as

where Ω * + has been defined in Lemma 6.5 and has been shown to be smooth and differentiable. Thus, since α → V diag(α)V is linear, f is smooth and differentiable. Moreover, one can have access to the gradient of f by performing an eigenvalue decomposition of V diag(α)V whose complexity is bounded above by O(r 3 ). Thus, one can apply one of the algorithms in section 4 by [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] in order to compute an optimal solution to Eq. (6.10). Moreover, a bound on the performance of the algorithm is given in theorem 4.4 of this same work. Note that if L is of the form L(α) = n i=1 i (α i ), it suffices to be able to compute the proximal operator of the i to get a proximal operator for L * [START_REF] Parikh | Proximal algorithms[END_REF].

6.B .6 Proof and additional discussion of Theorem 6.3

We recall the notion of universality [START_REF] Micchelli | Universal kernels[END_REF], in particular cc-universality [START_REF] Bharath | Universality, characteristic kernels and rkhs embedding of measures[END_REF], here explicited in the context of non-negative functions. A set F is a universal approximator for non-negative functions on X if, for any compact subset Z of X , we have that the set F| Z of restrictions on Z, defined as F| Z = {f | Z | f ∈ F}, is dense in the set C + (Z) of non-negative continuous functions over Z in the maximum norm. In the following theorem we prove the cc-universality of the proposed model Theorem 6.9. Let X be a locally compact Hausdorff space, H a separable Hilbert space and

φ is a cc-universal approximator for the non-negative functions on X .

Proof. Proving that the proposed model is a cc-universal approximator for non-negative functions, is equivalent to require that given a compact set Z ⊆ X , a non-negative function g : Z → R + and > 0, there exists

. Note that for any x ∈ X , CHAPTER 6. PSD MODELS 3. The proof of (iii) is straightforward. The continuity of Ω comes from the fact that it is a norm on any finite dimensional Hilbert space. Moreover, since λ 1 > 0 or λ 2 > 0 , Ω goes to infinity.

Remark 23. As in the previous sections, the fact that Ω satisfies these three properties is actually sufficient to complete the proof.

Recall that H n is the finite dimensional subset of H spanned by the φ(x i ). Recall that Π n is the orthogonal projection on H n , i.e.

Define K C n (H) to be the following subspace of K C (H) :

It is straightforward to show that K C n (H) ⊂ K C (H) since projecting left and right preserves the linear inequalities. Proposition 6.9. Let L be a lower semi-continuous function which is bounded below, and assume Ω satisfies Assumption 6.2. Then Eq. (6.14) has a solution

Proof. In this proof, denote with J the function defined by

Our goal is to prove that the problem inf A∈K C (H) J(A) has a solution which is in K C n (H), i.e. of the form Π n AΠ n for some A ∈ K C n (H).

Let us start by fixing

First note that since Π n is the orthogonal projection on span(φ(x i )) 1≤i≤n , in particular Π n φ(x i ) = φ(x i ) for all 1 ≤ i ≤ n. Thus, for any 1

Here, the first and last equalities come from the definition of f A and f ΠnAΠn . Thus,

Now since Ω satisfies Assumption 6.2, by the second point of Lemma 6.6, it holds Ω(

This last inequality combined with the fact that Let us now show that inf A∈K C n (H) J(A) has a solution. Let us exclude the case where J = +∞, in which case A = 0 can be taken to be a solution.

Let V n be the injection V n : H n → H. Note that V n V * n = Π n and V * n V n = I Hn . These simple facts easily show that

Thus, our goal is to show that inf Ã∈K C n (Hn) J(V n AV * n ) has a solution.

By the first point of Lemma 6.6, since V * n V n = I Hn , it holds

Let c 0 be a lower bound for L. By the third point of Lemma 6.6, there exists a radius R 0 such that for all à ∈ S(H n ),

Now since L is lower semi-continuous, Ω is continuous by the last point of Lemma 6.6, and à → (f Vn ÃV * n (x i )) 1≤i≤n is linear hence continuous, the mapping A → J(V n ÃV * n ) is lower semi-continuous. Hence, it reaches its minimum on any non empty compact set. Since H n is finite dimensional, the set à ∈ K C (H n ) : à F ≤ R 0 is compact (closed and bounded) and non empty (it contains Ã0 ), and hence there exists

Going back up the previous equalities, this shows that

Lemma 6.7. The set K C n (H) can be represented in the following way

In particular, for any

Proof. The proof is exactly analoguous to the proof of Lemma 6.3.

We will prove the following Theorem 6.13 which statement is that of Theorem 6.6 with more precise assumptions.

Introduction

In many fields such as biochemistry, statistical mechanics and machine learning, effectively sampling arbitrary numbers of independent and identically distributed (i.i.d.) samples from probability distributions is a key task [START_REF] Gelman | Bayesian Data Analysis[END_REF][START_REF] Jun | Monte Carlo Strategies in Scientific Computing[END_REF][START_REF] Lelièvre | Free Energy Computations[END_REF].

Basic sampling methods include rejection sampling and gridding, and rely on simple properties of the density. However, they are suitable only in small dimensions, except for very structured cases. (7.13) where for Eq. (7.13), A = aa is assumed to be a rank-1 operator 1 . These quantities only depend on a, A, K and can be computed explicitly. Combining these bounds with Eqs. (7.9) and (7.10), ρ can be selected in an adaptive way in algorithm 2.

Remark 24 (Adaptive selection of ρ). Let ε > 0. Let f be a PSD model with matrix of coefficients A. Define

where

.2 Discussion

Sampling from the distribution on R d . It is possible to approximately sample from an infinite hyper-rectangle. To do so, one has to find a large enough hyper-rectangle Q such that almost all the mass is contained on Q and then apply the previous algorithm to this hyper-rectangle. One can, for instance, use algorithm 3.

Note that one can also concentrate f a priori using only its parameters (X, A, η), using Eq. (7.56) of Lemma 7.4 in Appendix 7.C .1. One can use this bound to bound the number of steps in algorithm 3.

Generality of the algorithm. algorithm 2 only relies on the fact that one can compute integrals on hyper-cubes of the model f . If we were to replace the Gaussian kernel k η by a kernel k, and therefore have a PSD model of the form ij A ij k(x, x i )k(x, x j ) with another positive definite kernel and A ∈ S m + , then one would be able to run the algorithm as soon as computations of the form Q k(x, x i )k(x, x j )dx were tractable. This would extend this framework to more general PSD models, described by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF].

CHAPTER 7. SAMPLING FROM ARBITRARY FUNCTIONS VIA PSD MODELS

.2 Efficient method with a rank one model

In this section, we present a method to approximately sample from the density p by approximating it by a PSD model solving a linear system (as opposed to a SDP). This simpler and faster method comes at the expense of the stronger Assumption 7.2 needed to provide guarantees. As for algorithm 4, we first approximate the density with a PSD model and then sample from it using algorithm 2. The difference lies in the approximation step. We assume that we can evaluate a function g p such that g 2 p ∝ p (usually, this function will be proportional to the square root of p). We then approximate g p with a Gaussian linear model Eq. (7.2) by solving a regularized empirical least squares problem, which is much faster than the solving of a SDP. Taking the square of that linear model, we obtain a PSD approximation of p from which we can sample using algorithm 2.

Step a): approximation of p. To fit a PSD model to p, we start by approximating g p by a linear model g = g(•; a, X m , η) (see Eq. (7.2)), where a ∈ R m is the solution to the empirical problem min (7.20) where g(x; a) := g(x; a, X m , η) and g n = (g p (x i )) 1≤i≤n . a is the solution to the system :

which can be solved either directly in time O(nm 2 + m 3 ) [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF] or using a preconditioned conjugate gradient method in time O(m 3 + nm) (Rudi et al., 2017;Meanti et al., 2020;[START_REF] Marteau-Ferey | Beyond leastsquares: Fast rates for regularized empirical risk minimization through self-concordance[END_REF]. We then define f = g 2 which is a rank-1 PSD model with coefficients A = a a , Z = X f (x)dx = g 2 L 2 (X ) which is computable in closed form as the integral of a PSD model (see Eq. (7.3)), and our approximation p = f / Z of p.

Solving Eq. (7.20) can be seen as solving a regularized empirical risk minimization problem for the Hellinger distance (see Eq. (7.32) in Sec. 7.3 .1); the regularization term λa K mm a being a regularization in the norm of the RKHS associated to the Gaussian kernel (see Appendix 7.B ). The Hellinger distance is particularly adapted to exponential models of the form exp(-V (x)) for a real-valued potential V , as the square root is simply exp(-V (x)/2).

Step b): sampling from the approximation p. We apply algorithm 2 to p with a parameter ρ and on the hyper-rectangle X . We denote with p sample the density p X ,ρ given by Eq. (7.7), from which algorithm 2 effectively samples N i.i.d. samples by Theorem 7.1. This two step strategy is detailed in algorithm 5. SolveHellinger simply solves Eq. (7.20).

Theoretical Analysis

We use the same notation as introduced in Sec. 7.4 .1. Once again, we assume that X = (-1, 1) d for simplicity. In order to obtain good learning rates for algorithm 5, we make the following assumption, which is stronger than Assumption 7.1: it assumes that p can be written as a single square q 2 , where q belongs to W β (X ).

Assumption 7.2 (Square distribution). There exists a function q belonging to W β (X ) such that p = q 2 . Moreover, we have access to p only through function evaluations of the form g p (x), where g p ∝ q and where the proportionality constant is unknown.

Organization of the Supplementary Material

7.A . Definitions and notations

We set the main notations and tools of the appendix (Fourier transform, vector and matrix notations, notations concerning hyper-rectangles, RKHS and specifically the Gaussian kernel).

7.A .1. Sobolev spaces

In this section, we focus more on notations and basic results concerning Sobolev spaces, as they will be our main tool to measure the regularity of a function.

7.A .2. Measuring distances between probability densities

In this section, we define and compare the basic distances we will be using to compare probability distributions in the paper, since we are always "approximating" a certain distribution with another. In particular, we define the total variation, Hellinger and Wasserstein distances.

7.A .3. General PSD models

We define PSD models in general [START_REF] Rudi | Finding global minima via kernel approximations[END_REF][START_REF] Rudi | Psd representations for effective probability models[END_REF]. They will be our main tool for approximation and sampling, and relates to the more restrictive definition in Sec. 7.2 .

7.B . Properties of the Gaussian RKHS

Throughout the paper the Gaussian kernel k η and the associated Gaussian RKHS will be central objects. We introduce different properties and results.

7.B .1. Properties of the Gaussian kernel k η

We introduce certain properties of the Gaussian kernel involving products, as well as a bound on the derivative of the associated embedding in Lemma 7.2.

7.B .2. Useful Matrices and Linear Operators on the Gaussian RKHS

We introduce the most important theoretical objects of the paper. We introduce kernel matrices, matrices which will appear in the integration of Gaussian PSD models, operators which relate L 2 to the RKHS H η , operators which allow to discretize using samples and "compression" operators which allow concise representations.

7.B .3. Approximation properties of the Gaussian kernel

We prove two important results concerning the approximation properties of the Gaussian RKHS in proposition 7.7 and the concise representation of models in Lemma 7.3.

7.C . Properties of Gaussian PSD models

We present the results specific to Gaussian PSD models. These results are often reformulations of theorems presented by [START_REF] Rudi | Psd representations for effective probability models[END_REF].

7.C .1. Bounds on the support and the derivatives

We present result to understand how the mass of a Gaussian PSD model is concentrated (Lemma 7.4) and how the derivative of a Gaussian PSD model can be bounded using only its parameters (Lemma 7.5).

7.C .2. Compression as a Gaussian PSD model

We restate Theorem C.4 of [START_REF] Rudi | Psd representations for effective probability models[END_REF] as Theorem 7.5 on the effect of a compression operator on a PSD model.

7.C .3. Approximation properties of Gaussian PSD model

We refine Theorem D.4 of [START_REF] Rudi | Psd representations for effective probability models[END_REF] in Theorem 7.6 in order to approximate a sum of squares using a PSD model on the Gaussian RKHS H η .

7.D . The sampling algorithm

We prove that the sampling algorithm indeed returns N i.i.d. samples from the right distribution, and characterize the distance between the sampling distribution and the original PSD distribution.

7.D .1. Dyadic decompositions and convergence of algorithm 2

We formally prove that algorithm 2 finishes and returns N samples from a distribution characterized by a structural induction formula (see Lemma 7.6).

7.D .2. Proof of Theorem 7.1

We prove Theorem 7.1 by structural induction, showing that when the samples are randomly shuffled, we end up with N i.i.d. samples from the distribution defined in Eq. (7.7). This is done by matching the distribution with the one from the previous section using a structural induction.

7.D .3. Evaluating the error of the sampling algorithm : proof of Theorem 7.2

We prove Theorem 7.2 in Theorem 7.7, bounding the distance between the distribution of the PSD model and the actual distribution from which algorithm 2 samples (see Eq. (7.7)). This is done in different distances, all related to the problem in different way (Wasserstein is the most adapted in spirit, but we also need stronger distances such as total variation and Hellinger, which can be bounded using Lipschitz constants of the PSD models).

7.D .4. Time complexity

We illustrate that the time complexity of the algorithm is indeed taken up by the integral computations.

7.E . A general method of approximation and sampling

We prove that we can approximate any probability distribution satisfying Assumption 7.1 using non necessarily normalized function values, by solving Eq. ( 7.15) with the right parameters in proposition 7.10 which is labeled in the main text as proposition 7.1. We then show that applying algorithm 2 with the right value of ρ yields a good sampling algorithm from a good approximation of the distribution. This proves Theorem 7.3 and is proved here as Theorem 7.8.

7.F . Approximation and sampling using a rank one PSD model

We prove that we can approximate any probability distribution satisfying Assumption 7.2 using non necessarily normalized function values, by solving Eq. (7.20) with the right parameters in proposition 7.11 which is labeled in the main text as proposition 7.2. This has an advantage compared to the previous method which is that the approximation phase is much faster (it solves a linear system instead of an SDP). We then show that applying algorithm 2 with the right value of ρ yields a good sampling algorithm from a good approximation of the distribution. This proves Theorem 7.4 and is proved here as In this part, we present the works by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]; Marteau-Ferey, Bach, and Rudi (2022b). As explained in Sec. 2.3.3, the general goal is to minimize a function f defined on a set X , i.e., solve the minimization problem

As explained in Secs. 2.1.4 and 2.3.3, this method is based on the reformulation of the minimization as a convex optimization problem with a potentially infinite number of constraint. This problem simply maximizes a lower bound on the function :

(2.25)

In the work by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF], we assume to have access to n function values of f at points x 1 , ..., x n (these points will be sampled randomly). We will leverage smoothness in order to design an algorithm which approximate and solves Eq. (2.25). This method is related, in spirit, to the method of optimizing polynomials [START_REF] Lasserre | Global optimization with polynomials and the problem of moments[END_REF][START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF], which is based on the same convex reformulation.

In this introduction to the works by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]; Marteau-Ferey, Bach, and Rudi (2022b), we will therefore make a parallel with the works on polynomial optimization. Parallel is not a randomly chosen word : while there are structural similarities between the Note that the (typically exponential) dependence on the dimensionality d is only in the constants and tracked explicitly in the rest of the paper.

Moreover the algorithm is based on simple interior-point methods for semidefinite programming, directly implementable and based only on function evaluations and matrix operations. It can thus leverage multiple GPU architectures to reach large values of n, which are needed when the dimension grows.

Outline of contributions

In this section, we present our framework, our algorithm and summarize the associated guarantees.

Denote by ζ ∈ R d a global minimizer of f and assume to know a bounded open region Ω ⊂ R d that contains ζ. We start with a straightforward and classical convex characterization of the problem in Eq. ( 9.1), with infinitely many constraints:

Note that the solution c * of the problem above corresponds to c * = f (ζ) = f * , the global minimum of f . The problem above is convex, but typically intractable to solve, due to the dense set of inequalities that c must satisfy.

To solve Eq. ( 9.2) our main idea is to represent the dense set of inequalities in terms of a dense set of equalities and then to approximate them by subsampling.

Tight relaxation. We start by introducing a quadratic form φ(x), Aφ(x) with A a selfadjoint positive semidefinite operator from H to H, for a suitable map φ : Ω → H and an infinite-dimensional Hilbert space H, to define the following problem

where S + (H) is the set of bounded self-adjoint positive semi-definite operators on H.

The problem in Eq. ( 9.3) has a smaller optimized objective function than the problem in Eq. ( 9.2) because we constrain A to be positive semi-definite and any feasible point for Eq. ( 9.3) is feasible for Eq. ( 9.2). In fact, when f is a polynomial and φ(x) is composed of monomials of degree less than half the degree of f (and thus H finite-dimensional), then we recover the classical "sum-of-squares" relaxation of polynomial optimization. In that situation, the relaxation is tight only if ff * is itself a sum-of-squares, which is known to not always be the case. Then, to make the relaxation tight in the limit, several hierarchies of polynomial optimization problems have been considered using polynomials of increasing degrees [START_REF] Lasserre | Global optimization with polynomials and the problem of moments[END_REF][START_REF] Laurent | Sums of squares, moment matrices and optimization over polynomials[END_REF][START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF].

In this paper, we consider a well-chosen infinite-dimensional space H, and we prove that if f is smooth enough (i.e., m-times differentiable with m > 3 + d/2), under mild geometrical assumptions on f then there always exists a map φ, and a finite rank A * ∈ S + (H) for which the problem in Eq. ( 9.2) and the one above are equivalent, that is, the relaxation is tight.

Note that, the resulting φ, despite being infinite-dimensional, has an explicit and easy-to-compute (O(d) in memory and time) inner product k(x, x ) = φ(x), φ(x ) that will be the only quantity required to run the algorithm. We will thus use Hilbert spaces H which are reproducing kernel CHAPTER 9. FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS then, with probability at least 1 -δ,

where A * is any solution of Eq. (9.3).

Note that A * exists since f ∈ C s+3 (R d ) and it satisfies the geometrical mild condition in Assumption 9.1(b) (as we prove in Sec. 9.4 ), and that all constants can be made explicit (see Theorem 9.6). From the result above, and with m = s + 3, for s > d/2, we can achieve an error of order n -s/d+1/2 , which translates to ε = O(n -m/d+3/d+1/2 ) as soon as m > d/2 + 3. We pay the additional exponent 3 since we construct the candidate matrix representing the solution by requiring that each component of the Hessian of f , which is m -2 times differentiable belongs to the RKHS. This accounts for the 2 term, the last 1 is paid simply since s can be not integer.

The rate for the class of functions C m (Ω) is sub-optimal by an exponent 1/2 + 3/d. In the following remark we are going to show that our algorithm achieves nearly-optimal convergence rates when the function to optimize is in a Sobolev space. Denote by W s 2 (Ω) the Sobolev space of squared-integrable functions of smoothness s > 0, i.e., the space of functions whose weak derivatives up to order s are square-integrable on Ω, (see the work by [START_REF] Adams | Sobolev Spaces[END_REF]).

Remark 25 (Nearly optimal rates for Sobolev spaces.). If Ω satisfies Assumption 9.1(a), f satisfies Assumption 9.1(b) and f ∈ W s 2 (Ω), with s > d/2 + 3, then algorithm 6 with Sobolev kernel of smoothnes s -3 achieves the convergence rate O n -s/d+1/2+3/d , modulo logarithmic factors, as proven in Theorem 9.6. When d is large, then the error exponent is asymptotically optimal, since the term 3/d becomes negligible, leading to the optimal exponent -s/d + 1/2 (see, e.g., (Novak and Woźniakowski, 2008, Prop. 1.3.11)).

Finding the global minimizer. In Sec. 9.7 we derive an extension of the problem in Eq. (9.5), with the goal of finding the global minimizer. Under the additional assumption that the minimizer is unique we obtain the similar rate as Theorem 9.5 for the localization of the global minimizer.

Warm restart scheme for linear rates. Applying a simple warm restart scheme, we prove, in Sec. 9.7 .2, that when f has a unique global minimum, then it is possible to achieve it with error ε, with a number of observations that is only logarithmic in

for some constant C d,m that can be exponential in d (note that the added assumption of unique minimizer makes this result not contradict the lower bound in ε -d/m ).

Rates for functions with low smoothness m ≤ d/2 or functions that are not in H. In Sec. 9.8 .2 we study a variation of the problem in Eq. (9.5) that allows to have some error τ > 0 on the constraints. When f ∈ C m+2 (Ω), by tuning τ appropriately with respect to λ, we show that algorithm 6 applied on this different formulation achieves an error in the order

where r is now the index of the Sobolev kernel and can be chosen arbitrarily large. The exponent of the rate above matches the optimal one for C m+2 functions (that is -(m + 2)/d) up to a multiplicative factor of 1 2 1 1+2/m . 9.3 . SETTING

.2 Convergence properties of the finite-dimensional problem

Now we use Theorem 9.4 to bound the error of Eq. (9.5). First, to apply Theorem 9.4 we need to prove the existence of at least one finite-dimensional B 0 that satisfies the constraints of Eq. ( 9.5) and such that the trace of B is independent of n and h X,Ω . This is possible since we proved in Theorem 9.3 that there exists at least one finite rank operator A that solves Eq. ( 9.3) and thus satisfies its constraints, of which the ones in Eq. ( 9.5) constitute a subset. In the next lemma we construct

where A * is one solution of Eq. ( 9.3) with minimum trace-norm, since the bound in Theorem 9.4 depends on the trace of the resulting matrix.

Proof. Let V : H → R n be the partial isometry defined in Lemma 9.2 and P = V * V be the associated projection operator. Define B ∈ R n×n as B = V AV * . Since by Lemma 9.2, Φ i = V φ(x i ) and The proof is concluded by noting that, since A 0 and P op ≤ 1 because P is a projection, then Tr(AP ) ≤ P op Tr(|A|) = P op Tr(A) ≤ Tr(A).

We are now ready to prove the convergence rates of Eq. ( 9.5) to the global minimum. We will use the bound for the inequality on scattered data that we derived Theorem 9.4 and the fact that there exists B 0 that satisfies the constraints of Eq. ( 9.5) with a trace bounded by Tr(A * ) as we proved in the lemma above (that is in turn bounded by the the trace of the operator explicitly constructed in Theorem 9.2). The proof is organized as follows. We will first show that Eq. ( 9.5) admits a minimizer, that we denote by (ĉ, B). The existence of B allows to derive a lower-bound on ĉf * . Using Theorem 9.4 on the constraints of Eq. ( 9.5) and evaluating the resulting inequality in one minimizer ζ of f allows to find an upper bound on ĉf * and an upper bound for Tr( B).

Theorem 9.5 (Convergence rates of Eq. ( 9.5) to the global minimum). Let Ω be a set satisfying Assumption 9.1(a) for some r > 0. Let n ∈ N + and X = {x 1 , . . . , x n } ⊂ Ω with fill distance h X,Ω .

Let k be a kernel and H the associated RKHS satisfying Assumption 9.2 for some m ∈ N + . Let f be a function satisfying Assumption 9.1(b) and Assumption 9.3 for H. The problem in Eq. (9.5) admits a solution. Let (ĉ, B) be any solution of Eq. (9.5), for a given λ > 0. The following holds

Conclusion. Combining Eq. ( 9.18) with Eq. ( 9.17), and since λ ≥ 2MD m η by assumption,

Note that Eq. ( 9.16) is obtained from the one above, by dividing by λ 2 . Finally the inequality Eq. ( 9.15) is derived by bounding ∆ from below as ∆ ≥ -λ Tr(A * ) by Eq. ( 9.17), since Tr( B) ≥ 0 by construction, and bounding it from above as

obtained by combining Eq. ( 9.18) with Eq. ( 9.16) and with the assumption MD m η ≤ λ 2 .

The result above holds for any kernel satisfying Assumption 9.2 and any function f, Ω satisfying the geometric conditions in Assumption 9.1 and with f ∈ C 2 (Ω) and

. The latter requirement is quite easy to verify for example when H contains C s (Ω) and f ∈ C s+2 (Ω) for some s > 0 as in the case of H being a Sobolev space with s > d/2. Moreover the proposed result holds for any discretization X (random, or deterministic). We would like to conclude with the following remark on the sufficiency of the assumptions on f . Remark 27 (Sufficiency of Assumptions 9.1(b) and 9.3). Assumptions Assumptions 9.1(b) and 9.3 are sufficient for Theorems 9.3 and 9.5 to hold. However, by inspecting their proof it is clear that they hold by requiring only the existence of a trace-class operator

Note that this is implied by Assumptions 9.1(b) and 9.3 via Cor. 9.1.

In the next subsection we are going to apply the theorem above to the specific setting of algorithm 6.

.3 Result for Sobolev kernels and discussion

In this we are going to apply Theorem 9.5 to algorithm 6 which corresponds to H be the Sobolev space of smoothness s and the points X selected independently and uniformly at random. First, in the next lemma we bound in high probability the fill distance h X,Ω with respect to the number of points n that we sample, i.e., the cardinality of X. Lemma 9.4 (Random sets of points). Let Ω ⊂ R d be a bounded set with diameter 2R, for some R > 0, and satisfying Assumption 9.1(a) for a given r > 0. Let X = {x 1 , . . . , x n } independent points sampled from the uniform distribution on Ω. When n ≥ 2( 6R r ) d log 2 δ + 2d log 4R r , then the following holds with probability at least 1 -δ:

The proof of Lemma 9.4 is in Sec. 9.E .1, page 435 and is a simpler version (with explicit constants) of more general results (Penrose, 2003, Thm. 13.7). In the next theorem we apply the bound in the lemma above with the explicit constants for Sobolev spaces derived in proposition 9.1 to Theorem 9.5. The derivation of the theorem below is in Sec. 9.E .2, page 437.

.2 Comparison with SOS polynomials

In this section, we present a second set of experiments with the same setting as before but optimizing polynomial functions.

One of the reference algorithms in order to optimize polynomials (on semi-algebraic sets) is the Lasserre Hierarchy, implemented in the toolbox gloptipoly 3 [START_REF] Henrion | Gloptipoly 3: Moments, optimization and semidefinite programming[END_REF]. Applying this toolbox on a minimization problem constrained on a hyper-rectangle will yield either a lower bound (if the hierarchy does not converge) or the exact minimum as well as a minimizer if the hierarchy does converge.

The idea of this section is not to compete with the Lasserre hierarchies, which are taylored for polynomials. Rather, we wish to compare both methods, and show that they can complement each other in the case where gloptipoly does not converge, by providing an approximation of the minimizer with a certificate (i.e. with an upper bound on its distance to the optimum). This is particularly interesting in high dimensions, or with polynomials with high degree : in that case, the size of the polynomial problem becomes untractable (for our computer at least), while our algorithm still runs and returns a solution.

In this experiment, we consider polynomials whose expression can be found below, and wish to find a minimizer for these polynomials in the hypercube [-2, 2] d . Note that this domain is chosen such that we can easily sample from it. In particular, Lasserre hierarchies can adapt to much more flexible situations in terms of constraints.

Selection of polynomials Almost all of the polynomials considered in these experiments are of the form

We randomly select a few non zero indices for the Q as well as a random integer. The exact expressions of all the polynomials used can be found below in Table 9.2.

Results and Discussion. We report the results of these experiments in [START_REF] Adams | Sobolev Spaces[END_REF]. The result holds also for W s p (Ω) when Ω has Lipschitz boundaries as discussed in the work by [START_REF] Adams | Sobolev Spaces[END_REF], above Theorem 5.33. The result is obtained considering that when Ω has Lipschitz boundaries, then there exists a bounded extension operator between W s p (Ω) and W s p (R d ) [START_REF] Adams | Sobolev Spaces[END_REF]. Here we provide the proof for the sake of completeness. Let g ∈ W s p (Ω) and let ε ∈ (0, 1]. Then, by proposition 9.3 since Ω has Lipschitz boundary, there exists a bounded extension operator E : W s p (Ω) → W s p (R d ). Denote by g the function g = Eg and note that g ∈ W s p (R d ). Then, by applying Thm. 5.33 in the work by [START_REF] Adams | Sobolev Spaces[END_REF] we have that there exists gε

for some C depending only on s, p and C depending on r, s, p. Since by proposition 9.3 we have g

In the next proposition we recall some aspects of the more general Sobolev embedding theorem [START_REF] Adams | Sobolev Spaces[END_REF].

In particular there exists a constant C 5 such that

Finally, note that for any

Proof. The main statement of the proposition is a subcase of the more general Sobolev embedding theorem [START_REF] Adams | Sobolev Spaces[END_REF].

Finally, we recall that, since f ∈ C r (R d ) and Ω is bounded, then ∂ α f is uniformly bounded on Ω, for any α ∈ N d satisfying |α| ≤ r. This implies that f

Finally, note that the semi-norm • Ω,r defined in Eq. ( 9.31) and the Sobolev norm • W r ∞ are equivalent in the following sense. Proposition 9.6. Let Ω ⊂ Ω be two bounded open sets. Let r ∈ N. For any u ∈ C r (Ω ), recall the definition of u Ω,r from Eq. (9.31). There exists an explicit constant C 6 > 0 such that

. Note that this inequality holds also when the norms are unbounded, by using the convention +∞ ≤ +∞.

Proof. Since by Eq. (9.31)

d-1 (where this is taken to be equal to k + 1 in the case where d = 0), the result holds for

where Ff is the Fourier transform of f (see proposition 9.2 for more details on F).

(c) The inner product •, • k is characterized by Proof. By Thm. 1.4.1, pag. 25 by [START_REF] Hörmander | The Analysis of Linear Partial Differential Operators I: Distribution Theory and Fourier Analysis[END_REF] there exists z K,U ∈ C ∞ 0 (U ), i.e., a smooth function with compact support, such that z K,U (x) ∈ [0, 1] for any x ∈ U and z(x) = 1 for any

Proof. Assume without loss of generality that ζ = 0 and r = 1. Consider the following functions :

It is non-negative, bounded by 1, and infinitely differentiable as a product. Moreover :

To conclude the proof, given r > 0 and ζ ∈ R d we will take u(x) = u 0 ((xζ)/r).

Proof. For all n ∈ [N ], take u n as in Lemma 9.6 with r = r n , ζ = ζ n and define

By point (iii) of Lemma 9.6, u n = 1 on B n for all n ∈ [N ], which shows that u 0 = 0 on N n=1 B n and hence

Step 2. The function

By definition of u 0 , if u 0 (x) = 0, then there exists n ∈ [N ] such that u n (x) = 1. Since all the u n are non-negative, this shows that s := N n=0 u 2 n > 0. Moreover, consider the closed ball B of radius R and centered in 0. Since B is compact, s is continuous and s(x) > 0 for any x ∈ B, then there exists 0

Moreover, since for any x ≥ R, u 0 (x) = 1 and ∀n ∈ [N ], u n (x) = 0, we see that

Then s ∈ [m, M ] for any x ∈ R d , where m = min(m R , 1) and M = max(M R , 1).

Since the interval I = [m, M ] is a compact set included in the open set (0, ∞) and 1/ √ • is infinitely differentiable on (0, ∞) then by proposition 9.10 there exists q I ∈ C ∞ 0 (R) such that q I (x) = 1/ √ x for any x ∈ I. Since s(x) ∈ I for any x ∈ R d we have

Step 3.

Finally, defining v

since u 0 satisfies the same equality and v 0 is the product of u 0 by the strictly positive function 1/ √ s. Analogously v n = v n • 1 Bn , ∀n ∈ {1, . . . , N }, since u n satisfy the same equality and v n is the product of u n by the strictly positive function 1/ √ s.

CHAPTER 9. FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS Proof. For the rest of the proof we fix u : Ω → R with u ∈ H, r > 0 and z ∈ R d such that B r (z) ⊂ Ω. Let E Ω : H → H(R d ) be the extension operator from Ω to R d (its properties are recalled in proposition 9.8).Let χ ∈ C ∞ 0 (R d ) be given by Lemma 9.6 such that χ = 1 on B r (z),

In particular we recall that, since E Ω is a partial isometry (see proposition 9.8) then w H(R d ) = u H .

Step 1. Fourier transform of w t . Denote with w the Fourier transform of w which is well defined since w ∈ H(R d ) ⊂ L 2 (R d ) (see the work by [START_REF] Adams | Sobolev Spaces[END_REF]), with χ the Fourier transform of χ. Since For any t = 0, denote with w t the Fourier transform of w t which is well defined using the results of proposition 9.2, and which satisfies

Step 2. Separating low and high order derivatives of h t , and bounding the low order terms. For t = 0, denote with h t the Fourier transform of h t which is well defined since χ is bounded and w t ∈ L 2 (R d ). We will now bound h t H(R d ) for all t = 0, by using the characterization in Eq. ( 9.38). Since (x + y) s ≤ 2 max(s-1,0) (x s + y s ) for any x, y ≥ 0, s ≥ 0, then (1 + ω 2 ) s/2 ≤ c 1 (1 + ω s ) for any ω ∈ R d , with c 1 = 2 max(s/2-1,0) so using Eq. ( 9.38), we have

The first term on the right hand side can easily be bounded using the fact that the Fourier transform is an isometry of L 2 (R d ) (see proposition 9.2 for more details), indeed

) by definition, so it it bounded and has compact support, implying that

as recalled in Eq. (9.40) (the constant C 2 is defined in the same equation).

Step 3. Decomposing the high order derivatives of h t . Note that since h t = χ • w t , by property of the Fourier transform (see proposition 9.2(b)), χ • w t = (2π) d/2 χ w t . Moreover, since ω s ≤ ( ωη + η ) s ≤ c s ( ωη s + η s ) for any ω, η ∈ R d , with c = 2 max(s-1,0) , then, for all ω ∈ R d we have

where we denoted by J s the function J s (ω) = ω s for any ω ∈ R d . Applying Young's inequality, it holds :

Step 4. Bounding the elements of the decomposition. Now we are ready to bound the four terms of the decomposition of

), and J s (ω) ≤ Now define the following open set :

It is open since f is continuous. Moreover, it contains the closure of Ω which we denote with Ω which is compact since it is closed and bounded in R d . Theorem 1.4.2 in the work by [START_REF] Hörmander | The Analysis of Linear Partial Differential Operators I: Distribution Theory and Fourier Analysis[END_REF] applied to X = Ω and K = Ω shows the existence of χ :

. p ν satisfies the following properties :

The first, second and third properties are direct consequences of the properties of χ and the fact that ν < β. The last property comes from combining Eq. ( 9.43) with the definition of Ω and the fact that χ ∈ [0, 1] :

Step 2: Extending x → f (x) -ν 2 xζ 2 outside of Ω. Define g(x) = f (x) -p ν (x) on R d . Then g satisfies Assumption 9.1(b), g has exactly one minimizer in Ω which is ζ, and its minimum is g(ζ) = f * . Indeed, the fact that g ∈ C 2 (R d ) comes from the fact that f ∈ C 2 (R d ) by Assumption 9.1(b) on f and the fact that p ν ∈ C ∞ (R d ). Moreover, g ≥ f * on R d and g -f * ≥ δ/2 on ∂Ω. Indeed, first note that since ν < β, it holds

where the last inequality comes from Eq. ( 9.21). Second, since p ν = 0 on R d \ Ω and since f * is the minimum of f , for any

Finally, by the last point of the previous step, we see that g(x)

and that this minimum is reached at ζ and is not reached on the boundary of Ω. The fact that ζ is the unique minimum on Ω comes from the fact that since ν < β and by Eq. ( 9.21) we have that for any x ∈ Ω \ {ζ} the following holds

The fact that this minimum is not reached on the boundary of Ω comes from the fact stated above that g(x) ≥ f * + δ/2 for any x ∈ ∂Ω. Finally, the fact that ζ is a strict minimum of g also comes from Eq. ( 9.44) which implies that ∇ 2 g(ζ) (βν)I since g reaches a minimum in ζ, g is C 2 and ν < β.

CHAPTER 9. FINDING GLOBAL MINIMA VIA KERNEL APPROXIMATIONS Note that g also satisfies Assumption 9.3 since f satisfies Assumption 9.3 and

Step 3: Applying Cor. 9.1 to g. The previous point shows that g satisfies Assumptions 9.1(b) and 9.3and that g has a unique minimum in Ω. Moreover, H satisfies Assumption 9.2. Hence, Cor. 9.1 to g and H, the following holds : there exists A * ∈ S + (H) with rank(A * ) ≤ d + 1 such that g(x) -f * = φ(x), A * φ(x) for all x ∈ Ω.

Step 4. Let p 0 be the maximum of Eq. ( 9.20). In Lemma 9.5 we have seen that the solution of Eq. ( 9.20) is p 0 = f * . Since A 0 implies φ(x), Aφ(x) ≥ 0 for all x ∈ Ω, the problem in Eq. ( 9.20) is a relaxation of Eq. (9.22), where the constraint f (x)

Then p 0 ≥ p * if a maximum p * exists for Eq. ( 9.22). Thus, if there exists A that satisfies the constraints in Eq. ( 9

9.F .3 Proof of Theorem 9.8

Proof. The proof is a variation of the the one for Theorem 9.5, the main difference is that we take care of the additional term zζ.

Step 0. The SDP problem in Eq. ( 9.23) admits a solution (a) Under the constraints of Eq. ( 9.23), cν 2 z 2 cannot be larger than min i∈[n] f (x i ). Indeed, for any i ∈ [n], since B 0, the i-th constraint implies

There exists an admissible point. Indeed let (c * , z * , A * ) be the solution of Eq. ( 9.22) such that A * has minimum trace norm (by Theorem 9.7, we know that this solution exists with c * = f * and z * = ζ, under Assumptions 9.1 to 9.4). Then, by Lemma 9.3 applied to g

x n } we know that there exists B ∈ S + (R n ) satisfying Tr(B) ≤ Tr(A * ) s.t. the constraints of Eq. ( 9.23) are satisfied for c = c * and z = z * . Then (c * , z * , B) is admissible for the problem in Eq. (9.23). Since there exists an admissible point for the constraints of Eq. ( 9.23) and its functional cannot be larger than max i∈[n] f (x i ), then the SDP problem in Eq. ( 9.23) admits a solution [START_REF] Boyd | Convex optimization[END_REF].

Step 1. Consequences of existence of A * . Let (ĉ, ẑ, B) one minimizer of Eq. (9.23). The existence of the admissible point (c

Introduction

The relationship between non-negative functions and functions decomposable as sums of squares is a fundamental question in both theoretical and applied mathematics. From a theoretical viewpoint, the decomposability of a non-negative function in terms of sum of squares is the basis of important theoretical objects and properties: quadratic modules [START_REF] Marshall | Representations of non-negative polynomials having finitely many zeros[END_REF] in algebraic geometry, regularizing operators such as Laplacians or sub-Laplacians in (sub-)Riemannian geometry [START_REF] Hörmander | Hypoelliptic second order differential equations[END_REF]Bony, 1998Bony, -1999)), non-negative symbols in pseudo-differential calculus [START_REF] Hörmander | The Analysis of Linear Partial Differential Operators III[END_REF][START_REF] Tataru | On the fefferman-phong inequality and related problems[END_REF]. From an applicative viewpoint, representing a non-negative function in terms of sum of squares allows to simplify the analysis of probability representations and optimization problems [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF][START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. Restricting to the case of non-negative polynomials this has been applied to global optimization and generalized methods of moments [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF][START_REF] Henrion | The Moment-SOS Hierarchy[END_REF]. More generally, the decomposition of non-negative p-times differentiable functions allowed to derive simple and fast optimization algorithms in the context of global optimization (Rudi,CHAPTER 10. SOS DECOMPOSITIONS OF SMOOTH FUNCTIONS b) for any

The intuition of this definition is that if f is positively normal to N , then f grows quadratically normally to N , which is a local minimum valley on which f is constant. Note that there can be more than one connected component in N : this will correspond to multiple local minima valleys (see second and third examples in Sec. 10.2 .3). We now reformulate the fact that the NHC holds at every point in Z as a more geometric global assumption, using definition 10.2.

Lemma 10.1 (Global normal Hessian condition). The following statements are equivalent and define the global NHC condition:

(i) for all x 0 ∈ Z, f satisfies the NHC;

(ii) Z is a sub-manifold of R d (not necessarily connected) of class C 1 such that the Hessian of f is positive normally to Z;

This equivalence is a direct consequence of the local description of Z obtained in Theorem 10.5 under the local NHC. Examples of functions satisfying the global normal Hessian condition can be found in Sec. 10.2 .3. They have manifolds of zeros which are depicted in the same order in Sec. 10.2 .1. Under this geometric condition, we will show in Theorem 10.6 that f can be written as a sum of squares of C p-2 functions with locally finite support, defined below.

Locally finite support. Let X be a topological space (see the book by [START_REF] Jänich | Topologie[END_REF] for full definitions). We say that a family (S i ) of subsets of X is locally finite if for every x ∈ X, there exists an open set U x containing x which intersects a finite number of the S i , i.e., |{i ∈ I : U x ∩ S i = ∅}| < ∞. A family (f i ) of functions on a topological space X has locally finite support if the family of supports (supp(f i )) i∈I is locally finite (recall that supp(f i ) = {x : f (x) = 0}). In particular, if (f i ) has locally finite support, the function i∈I f 2 i is well defined and it is also of class C q if the functions are of class C q . Using this terminology, the global result can be stated as follows.

Theorem 10.6. If f satisfies the global normal Hessian condition in Lemma 10.1, there exists an at most countable family (f i ) i∈I ∈ (C p-2 (Ω)) I with locally finite support such that

(10.7)

Moreover:

• if f satisfies the strict Hessian condition, Z is discrete and we can find such a decomposition such that |I| ≤ d + 1.

• if Z is compact, then |I| can be taken to be finite.

For the formal proof of this result, we refer to the next section, where this result will be proved more generally for functions defined on manifolds (see Theorem 10.8 and Sec. 10.3 ).

Main steps of the proof. For subtleties pertaining to the SHC case, we refer to Sec. 10.3 . The gluing done in that section is slightly more elaborate.

Step 1. Since the local NHC holds at any point in Z, using Theorem 10.5 shows that at any point x, there exists an open neighborhood U x of x , an integer n x and functions (f x,j ) 1≤j≤nx

. GLOBAL DECOMPOSITION AS A SUM OF SQUARES FOR FUNCTIONS ON MANIFOLDS465

of class C p-2 on U x such that f = nx j=1 f 2 x,j on U x . The collection of sets U x is then an open covering of Z. Since R d is Hausdorff and second-countable (see Sec. 10.3 for precise definitions), only an at most countable subsets of them are necessary to cover Z (even a finite number if Z is included in a compact, since it is then itself a compact). Denote with (U i ) i∈I this open covering, and replace x by i to denote the associated f i,j and n i .

Step 2. Since Z is closed, as the set of zeros of a continuous function, the set

and satisfies f 2 1 = f . We can therefore add U >0 to the collection (U i ) and still guarantee the following property: for all i ∈ I, there exists n i ∈ N and f i,j ∈ C p-2 (U i ) such that f = n i j=1 f 2 i,j . Moreover, (U i ) becomes an open covering of Ω; in particular, if U i was a finite covering of Z, it now becomes a finite covering of Ω.

Step 3. Using Lemma 10.4, we can take a partition of unity (χ i ) adapted to the open covering i U i such that i χ 2 i = 1 and which is locally finite. Define fi,j = f i,j χ i which is now defined on the whole of Ω (indeed, it can be extended as zero to Ω \ U i since the support ot χ i is included in U i ). The fi,j satisfy i,j f 2 i,j = f on the whole of Ω, and is a finite family if the covering U i is finite (if Z is assumed to be compact for example).

Global decomposition as a sum of squares for functions on manifolds

In this section, we present results analogous to those of Sec. 10.2 but in the more general context of manifolds. After a brief recap on the terminology and definitions related to manifolds, in Sec. 10.3 .1, we will adapt the definitions of the local and global normal Hessian conditions, as well as state the equivalent result to Theorem 10.5 in the context manifolds. In Sec. 10.3 .2, we will introduce the tools to glue local decompositions as sum of squares together. Finally, in Sec. 10.3 .3, we prove Theorem 10.8, the equivalent of Theorem 10.6 in the broader context of manifolds.

Additional definitions and notations for manifolds

In this section, we introduce the basic definitions we will need concerning manifold. For more formal introductions to manifolds, we refer to the works by [START_REF] Lafontaine | An introduction to differential manifolds[END_REF]; [START_REF] Paulin | Géométrie différentielle élémentaire[END_REF]; [START_REF] Spivak | A Comprehensive Introduction to Differential Geometry[END_REF]. Informally, a manifold of dimension d is a set which "looks like R d " locally. This means that at every point x ∈ M , we can find a chart φ which topologically maps a neighborhood

More generally, we define a chart on a topological space M as a map φ : U → R d for some d ∈ N, defined on an open set U of M , and which is a homeomorphism onto its image. We define a manifold M as a second-countable 1 , Hausdorff 2 topological space equipped with a collection A = (φ i ) i∈I of charts such that (i) all transition maps

CHAPTER 10. SOS DECOMPOSITIONS OF SMOOTH FUNCTIONS equivalently if it is positive normally to Z which is a sub-manifold of M of class C 1 (or C p-1 ). On the right hand side of Sec. 10.3 , we represent a function which satisfies the NHC on the sphere S 2 through a colormap, with a continuous set of zeros. The goal is to prove that such a function can be decomposed as a sum of squares on S 2 .

.2 Gluing local decompositions to form a global one

In this section, we present and develop the tools to glue local decompositions such as Theorem 10.7 into a global one, which will lead to Theorem 10.8.

The first result we need is a simple result to "extend" a function defined on an open set U of M to M by multiplying it by a function defined on M whose support lies in U (Lemma 10.3). The second one is a variant of the fundamental result of existence of partitions of unity on a manifold, adapted to our sum of squares setting (Lemma 10.4). Recall that the support of a function has been defined in Sec. 10.2 .3. The proof of these results can be found in Sec. 10.A .1.

Lemma 10.3 (Extension lemma). Let q ∈ N, M be a manifold of class at least C q . Let U be an open set of M , g : U → R be a C q function defined on U , and χ : M → R be a C q function defined on the whole of M but with support included in U . Then the function χg : U → R extended as 0 on M \ U , is of class C q on the whole of M and has support included in supp(χ) ⊂ U . We still denote with χg its extension to M . Lemma 10.4 (Gluing lemma). Let (U i ) i∈I be an open covering of a manifold M of class C k (i.e. i∈I U i = M ). There exists a family of functions χ i : M → [0, 1] of class C k with locally finite support, such that supp(χ i ) ⊂ U i for all i ∈ I and satisfying:

We can now proceed from local to global in two steps. First, we use the gluing lemma to glue decompositions in a single connected component of the manifold of zeros (Lemma 10.5). We then glue these different decompositions into a single global one (Lemma 10.6).

Lemma 10.5. Assume f satisfies the global NHC. Let N be a connected component of its manifold of zeros Z.

There exists an open neighborhood U of N as well as a locally finite, at most countable family (f j ) j∈J of functions of class C p-2 such that

(10.9)

Moreover, we can find J such that a)

Proof. The case where N = {x 0 } is simply Theorem 10.7 applied to x 0 . In the other cases, note that for all x ∈ N , by Theorem 10.5 since the NHC is satisfied at x, there exists an open neighborhood U x of x as well as functions (f

and b) J is at most countable otherwise, since N is second-countable and Hausdorff. Denote with (U j ) j∈J this open covering, and replace x by j to denote the associated f j,i . Denote with U the open set j U j .

Applying Lemma 10.4 to the manifold U , we can find a family of functions (χ j ) j∈J with locally finite support, such that supp(χ j ) ⊂ U j and j χ 2 j = 1 on U . By the extension Lemma 10.3, we

. GLOBAL DECOMPOSITION AS A SUM OF SQUARES FOR FUNCTIONS ON MANIFOLDS469

can therefore define the functions f j,i := χ j f j,i for i ∈ {1, ..., d} and j ∈ J which are defined on the whole of M . Note that since supp( fj,i ) ⊂ supp(χ j ) and since 1 ≤ i ≤ d, the support of ( fj,i ) is also locally finite. To conclude, we use the property that j χ 2 j = 1 on U as well as the fact that i f 2 j,i = f on supp(χ j ) ⊂ U j to show that i,j f 2 j,i = f on U . The number of functions fj,i is finite if N is compact since J is finite, and is at most countable else since J is at most countable.

Lemma 10.6. Let Z = i∈I N i be the manifold of zeros decomposed along its connected components. Assume that there exists an index set J, such that for all i ∈ I, there exists an open neighborhood U i of N i on which f can be decomposed as a sum of squares indexed by J: (10.10) and such that the families (f i,j ) j∈J are all locally finite. Then there exists a locally finite family (g j ) j∈J∪{ } of C p-2 functions on M (we add an extra element to J), such that

Proof. By Lemma 10.9, there exist disjoint open sets

proper sub-manifold of M by Lemma 10.1 (directly adapted to the manifold case). Hence, we can assume that the U i are disjoint (consider instead U i ∩ V i , the property still holds). Define U = {f > 0}. Note that since the U i 's cover Z, U ∪ i∈I U i covers M since f is non-negative: take χ , (χ i ) i∈I to be a gluing family adapted to that covering given by Lemma 10.4. Note that for any i, i ∈ I, we have χ i χ i = 0 since χ i is supported on U i and the U i are disjoint. Consider the function g j = i∈I χ i f i,j , which is well defined on M and C p-2 by Lemma 10.3. We have g 2 j = i∈I χ 2 i f 2 i,j since χ i χ i = 0 when i = i .

Assertion : the family (g j ) j∈J has locally finite support. Let x ∈ R d and assume g j (x) = 0. Then there exists i ∈ I such that χ i (x) > 0, and hence there exists an open set U x around x such that U x ⊂ U i . But in that case, χ i (x ) = 0 for all other i and for all x ∈ U x since the U i are disjoint and χ i is supported on U i . Moreover, since (f i,j ) j∈J is locally finite, there exists an open set V x around x as well as a finite J 0 ⊂ J such that f i,j = 0 on V x for all j ∈ J \ J 0 . Hence, for any j ∈ J \ J 0 and any x ∈ U x ∩ V x , we have f i,j (x ) = 0 and χ i (x ) = 0 thus g j (x ) = 0. Thus, U x ∩ V x ⊂ M \ supp(g j ) for all j / ∈ J 0 : the family g j is locally finite.

Since the addition of one function changes nothing to the locally finite property of a family of functions, the family (g j ) j∈J ∪g is still locally finite. Using the fact that g 2 j = i∈I χ 2 i f 2 i,j , that j∈I∪{ } χ 2 i = 1 and Eq. (10.10), it holds

CHAPTER 10. SOS DECOMPOSITIONS OF SMOOTH FUNCTIONS

.3 Main results

We are now ready to state our main result on manifolds. On the right hand side of Sec. 10.3 , we represent a case where this theorem applies for a non-negative function defined on S 2 .

Theorem 10.8. Let M be a manifold and f : M → R be a non-negative map of class C p . Assume f satisfies the global normal Hessian condition. Then there exists I which is at most countable and functions f i ∈ C p-2 (M ) for i ∈ I such that the family (f i ) has locally finite support and ∀x ∈ M, f (x) = i∈I f i (x) 2 . (10.12)

Moreover:

• if f satisfies the strict Hessian condition, Z is discrete and we can find such a decomposition such that |I| ≤ d + 1.

• if Z is compact, then |I| can be taken to be finite.

Proof. The proof of this theorem is a simple consequence of Lemma 10.5 and Lemma 10.6. Note that the global NHC Lemma 10.1 shows that Z is a sub-manifold of M . Let N i denote the connected components of Z. By Lemma 10.9, we can find disjoints open sets U i such that

General case. Without any more assumptions, we know from Lemma 10.5 that on any connected component N i , we can have a decomposition of the form f = j∈J f 2 i,j with f i,j ∈ C p-2 a family with locally finite support on an open neighborhood V i of N i . Moreover, we know that J is at most countable. Adding zeros when necessary, and reindexing, we can assume that J = N. Now applying Lemma 10.6, we prove the general case.

Compact case. If we assume that N is compact, since the U i cover N , necessarily the number of connected components is finite (just extract a finite covering of N from the U i ). We know from Lemma 10.5 that on any connected component N i , we can have a decomposition of the form f = n i j=1 f 2 i,j with f i,j ∈ C p-2 and n i ∈ N on an open neighborhood V i of N i , since N i is compact. Hence, up to adding f i,j = 0, we can assume that n i = n = max i (n i ) since there are a finite number of connected components. Now applying Lemma 10.6 with J = {1, ..., n}, the result is proven in the compact case with n + 1 functions.

SHC case. If we assume that the SHC holds, every connected component N i is a singleton {x i }: we know from Lemma 10.5 we can have a decomposition of the form f = d j=1 f 2 i,j with f i,j ∈ C p-2 on an open neighborhood V i of N i , since N i is compact. Now applying Lemma 10.6 with J = {1, ..., d}, the result is proven with d + 1 functions.

Remark 32. Note that the difference between the number of functions in the SHC case is better than the one obtained by [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. This is because of the two step procedure in the gluing: first in a connected component, and then between connected components. The long term goal is to be able to prove that we need only a finite number N (d) of functions per connected component (in the compact case), and hence to have an explicit bound after gluing the connected components together, rather than just relying on a compact extraction argument, which is not as precise. In this section, we formally prove the key result of the paper, Theorem 10.5.

Proof. Note that the existence of the sub-manifold N of dimension d 0 around x 0 implies the existence of a local parametrization around x 0 (see Lafontaine, 2015, Theorem 1.21): there exists an open neighborhood W 0 of 0 in R d 0 , an open neighborhood U x 0 of x 0 in R d and a C k immersion φ : W 0 → U x 0 of class C k such that φ is a homeomorphism from W 0 onto U x 0 ∩ N . Since restricting N to N ∩ U x 0 does not change the assumptions of the theorem, we will will assume that N = Im(φ) for a C k immersion φ : (0, R d 0 ) → (x 0 , R d ). We will denote with T x 0 := dφ 0 (R d 0 ) = T x 0 N the tangent space to N at x 0 .

Before starting the proof, recall that for any x ∈ Z, it holds df (x) = 0 and d 2 f (x) 0 (or equivalently ∇ 2 f (x) 0). Moreover, note that if A ∈ S + (R d ) is a symmetric positive semidefinite matrix, if a vector k ∈ R d satisfies k Ak = 0, then Ak = 0 (this is a trivial consequence of the spectral theorem by decomposing k along an orthonormal basis of eigenvectors).

Step 1: characterizing the null-space of the Hessian. We will prove that under the assumptions of the theorem, a) T x 0 is equal to the null-space ker(∇ 2 f (x 0 )) of the Hessian of f at x 0 and b) that for any supplementary S to T x 0 , the restricted Hessian ∇ 2 f (x 0 )| S is positive definite.

To prove a), assume that there exists an element in k

and let x t = φ(th) which is defined for t in an open neighborhood of 0. Using the Taylor expansion of f around x 0 :

where (x) →

x →0

0. Now applying this for x t , since f (x t ) = f (x 0 ) = 0 and df (x 0 ) = 0, it holds:

Using the fact that φ is differentiable at 0 yields x t -x 0 = tdφ(0

Injecting this in the equation above yields

). Hence, necessarily, k ∇ 2 f (x 0 )k = 0, which is a contradiction. This proves that T x 0 ⊂ ker(∇ 2 f (x 0 )), and in particular, d 0 ≤ dim(ker(∇ 2 f (x 0 ))). Since the rank of ∇ 2 f (x 0 ) is actually dd 0 , the rank theorem shows that dim(ker(∇ 2 f (x 0 ))) = d 0 and hence T x 0 = ker(∇ 2 f (x 0 )).

To prove b), we just need to prove that the restriction to any supplementary to the null-space of ∇ 2 f (x 0 ) is positive definite. Using the small result at the beginning of the proof, any vector

In particular, this means that the restriction of ∇ 2 f (x 0 ) to any supplementary subspace S of T x 0 is positive definite.

Step 2: applying the Morse lemma. Let P = (P 1 , P 2 ) ∈ O d (R) be the matrix of an orthonormal basis adapted to the decomposition R d = T ⊥

x 0 ⊕ T x 0 . Note that P 1 ∈ R d×(d-d 0 ) and P 2 ∈ R d×d 0 are also orthonormal matrices, and that since P 1 spans T ⊥ x 0 , in particular P 1 ∇ 2 f (x 0 )P 1 0. Define g : (x , y ) ∈ R d-d 0 × R d 0 → f (P 1 x + P 2 y + x 0 ) = f (A(x , y )), where A(x , y ) = P (x , y ) + x 0 is an isometry 3 (A -1 x = P (xx 0 )). We have ∇ x g(0, 0) = P 1 ∇f (x 0 ) = 0 and CHAPTER 10. SOS DECOMPOSITIONS OF SMOOTH FUNCTIONS

The second is, as in the polynomial case, to handle functions f which are non-negative on a constrained set defined by inequalities f i ≥ 0. More precisely, we would like to show second order sufficient conditions to write f = g + i g i f i where the g, g i are sum of squares of regular functions when f and the f i are regular. This would open up the field of constrained optimization for methods developed for functions, such as kernel sum of squares [START_REF] Rudi | Finding global minima via kernel approximations[END_REF]. In the polynomial case, such conditions are given by so-called Positivstellensätzen [START_REF] Putinar | Positive Polynomials on Compact Semi-algebraic Sets[END_REF][START_REF] Stengle | A Nullstellensatz and a Positivstellensatz in Semialgebraic Geometry[END_REF][START_REF] Schmügden | The k-moment problem for compact semi-algebraic sets[END_REF], but usually assume the polynomial is positive. Second order conditions have been developed more specifically by [START_REF] Marshall | Representations of non-negative polynomials having finitely many zeros[END_REF] to deal with non-negative polynomials with zeros.

The third is to handle functions with conic outputs which are more general than the non-negativity one. For example, in order to represent functions which have values in a cone defined by linear inequalities [START_REF] Rudi | Finding global minima via kernel approximations[END_REF] or in the PSD cone [START_REF] Muzellec | Learning psd-valued functions using kernel sums-of-squares[END_REF].

10.A . AROUND PARTITIONS OF UNITY AND GLUING FUNCTIONS

10.A Around partitions of unity and gluing functions

In this section, we detail a few topological properties of manifolds, in order to a) decompose a manifold or a sub-manifold in connected components and b) use partitions of unity as a tool to glue functions together. These specific properties are needed for Sec. 10.3 .2. For basics on topological spaces (what is a topology, the notion of continuity, of homeomorphism), we refer to Chapter 1 by [START_REF] Jänich | Topologie[END_REF]. Main references for manifold are the works by [START_REF] Lafontaine | An introduction to differential manifolds[END_REF]; [START_REF] Spivak | A Comprehensive Introduction to Differential Geometry[END_REF]; [START_REF] Paulin | Géométrie différentielle élémentaire[END_REF]. Recall from Sec. 10.3 .3 the definition of a manifold M equipped with its atlas A of class C k , and of a chart on M . A chart φ is said to be of class C k for k ≤ k if it compatible with the atlas up to k smoothness, i.e. if the transitions maps φ • φ -1 i and φ i • φ -1 are all C k . A priori, the atlas of a manifold of class C k is not unique in the sense that more than one atlas generate the same structure. To make it so, and to be able to say the atlas of M of class C k , we consider the maximal atlas on M , i.e. the collection of all charts of class C k on M .

10.A .1 Paracompactness and partitions of unity

The main point of asking a (differential) manifold to be second countable and Hausdorff, (and not just to be locally homeomorphic to R d ), is for the manifold to be paracompact, and and hence to be equipped with partitions of unity. In this section, we introduce the main definitions and results on this topic.

Recall that a family of subsets (U α ) of a space X is said to be a covering of X if α U α = X. It is said to be locally finite if for any x ∈ X, there exists an open neighborhood U of x which intersects only a finite number of the U α . A family (V β ) is said to be a refinement of (U α ) if for all β, there exists an α such that V β ⊂ U α .

A topological space X is said to be paracompact if for any open covering (U α ) of X, there exists an open refinement (V β ) of (U α ) such that (V β ) is locally finite, and is an open covering of X. The following lemma is proved in the first part of proposition 2.3 by [START_REF] Paulin | Géométrie différentielle élémentaire[END_REF] or can be found in Theorem 2.13 by [START_REF] Spivak | A Comprehensive Introduction to Differential Geometry[END_REF].

Lemma 10.7. A manifold is paracompact.

Note that for [START_REF] Spivak | A Comprehensive Introduction to Differential Geometry[END_REF], a manifold is defined to be a metric space locally like R d . In proposition 2.2 by [START_REF] Paulin | Géométrie différentielle élémentaire[END_REF], it is shown that being metric and second countable is equivalent to the countable Hausdorff condition (under the condition of being locally homeomorphic to R d ). The condition by [START_REF] Spivak | A Comprehensive Introduction to Differential Geometry[END_REF] is a bit more general; in fact, it allows a manifold M to be a union of a possible non-countable connected component (as theorem 2 by [START_REF] Spivak | A Comprehensive Introduction to Differential Geometry[END_REF] shows that any connected component of a metric space locally homeomorphic to R d is actually second countable).

Paracompactness is an important property as it yields the existence of partitions of unity. The following lemma is standard (a proof can be found by [START_REF] Paulin | Géométrie différentielle élémentaire[END_REF], proposition 2.3). The result is of course also true for k = 0, but is more technical to prove.

which is connected, we have that U x ∩ N is connected and hence is included in the unique connected component of x.

Step 2. Consider the collection of open sets U x . By paracompactness of U := x∈N U x (it is a manifold), we can find an open cover (U α ) of U which is locally finite, and which still satisfies the condition that for all α, U α ∩ N is included in at most one connected component of N . Let (N i ) i∈I denote the connected components of N . For i

Step 3. For all i ∈ I, define F i = j∈I\{i},β∈A j V j,β and for all α ∈ A i , consider the set

Since the (V j,β ) are locally finite, there exists V x ⊂ V i,α such that V x intersects a finite number of the V j,β and hence of the V j,β . Hence, V x \ F i is still open. Hence W i,α is open. The second condition comes from the fact that V i,α ∩ N ⊂ N i , and that the connected components are disjoint Finally, taking W i = α∈A i W i,α , the W i satisfy all the desired properties (they are disjoint thanks to the previous point and cover N i since the V i,α covered N i ).

10.B Morse lemma

In order for this article to be self contained, we restate the following classical lemmas from differential geometry and topology. Recall that a C k -diffeomorphism is a map φ

The following results are classical.

Theorem 10.9 (Theorem 1.13 by [START_REF] Lafontaine | An introduction to differential manifolds[END_REF]). Let f : (x 0 , R d ) → R d be a function of class C k (k ≥ 1) defined around x 0 and such that df (x 0 ) is invertible. Then there exists a neighborhood U of x 0 such that f (U ) is open and f

Theorem 10.10 (Theorem 1.18 by [START_REF] Lafontaine | An introduction to differential manifolds[END_REF]). Let f :

We restate and reprove Lemma C.6.1 by [START_REF] Hörmander | The Analysis of Linear Partial Differential Operators III[END_REF], which is a generalization of the so-called Morse Lemma (see lemma 2.2 by [START_REF] Milnor | Morse theory[END_REF]), and which is the basis of Morse Theory. We will consider a function of two variables f (x, y) defined on R d 1 × R d 2 . We will denote with ∇ x f (x, y) its gradient with respect to the first variable taken at point (x, y); it is an element of R d 1 . Similarly, we will use the notation ∇ 2 xx f (x, y) ∈ R d 1 ×d 1 to denote the Hessian matrix taken with respect to the first coordinate at point (x, y). It is symmetric. Lemma 10.10 (Hörmander (2007, Lemma C.6

There exists an open convex neighborhood V of 0 in R d 1 and an open convex neighborhood W of

We can now prove our main result.

Proof of Lemma 10.10. Fix V 0 , W 0 satisfying the properties of Lemma 10.11. Let (x, y) ∈ V 0 ×W 0 . For t ∈ [0, 1], define x t = ϕ(y) + t(xϕ(y)). By convexity of V 0 , (x t , y) ∈ V 0 × W 0 ⊂ U 1 ⊂ U 0 for all t ∈ [0, 1]. Thus, the map g : t ∈ [0, 1] → f (x t , y) is well defined, and we can apply the Taylor formula g(1) = g(0)+g (0)+ 1 0 (1 -t)g (t)dt and the fact that g (0) = ∇ x f (ϕ(y), y)•(x-ϕ(y)) = 0 to obtain [START_REF] Hörmander | The Analysis of Linear Partial Differential Operators III[END_REF]. Theorem 10.10 shows there exists an

Summary of the thesis

In this thesis, we have explored three main topics, building on the great versatility of kernel methods. First, we focused on a statistical learning issue, extending fast rates and algorithms which existed in the least-square setting to other losses, and in particular to the logistic regression setting. Second, we developed a kernel based model to handle non-negativity constraints (PSD models), opening the way to the modelling of functions with constrained outputs, while keeping the good properties of kernel methods in terms of algorithms and approximation results. We then used it to model probability densities and to sample from the associated distribution. Third, using the analogy between PSD models and the sum of squares model for polynomials, we considered the problem of global optimization of regular functions using a so-called "kernel sum of squares" approach. We designed an algorithm which approximately minimizes a regular function using n evaluations points, and provided guarantees which shows that this algorithm adapts to the regularity of the function to be minimized, needing less evaluation points to reach a certain precision if the function is more regular.

In all these different settings, we have tried to put modelling at the center of the analysis. We have tried to show how the statistical and algorithmic properties can be characterized using properties of the models we consider, and argue that choosing the right model for the right task remains a (if not the) important choice of a practitionner, but also of the theoretician. While this is only one point of view, we believe it to be a nice way of looking at the different machine learning and applied mathematics problems from a central perspective.

We now summarize the contributions of this thesis in (slightly) greater detail.

In part I, we extended the classical "slow rates" known for logistic regression to match the "fast rates" known for least squares [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF][START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF]. We introduced two key quantities : the bias and the effective dimension (see definition 3.1), which charaterize the complexity of the problem in this setting, and which generalize the corresponding quantities defined in the least-square case. These quantities as well as the derived trade-offs and CHAPTER 11. CONCLUSION AND PERSPECTIVES rates, helped us understand what makes a logistic regression problem either complicated or simple, and the interaction between the complexity of the problem and the necessary regularization. These quantities have allowed us to show that under certain assumptions, regularized empirical risk minimization can leverage the regularity of the solution and of the RKHS, and perform better than the slow rates O(1/ √ n), going as fast as O(1/n) in the most favorable settings. In certain of these favorable settings, we have shown that the regularization needs to be chosen very small, i.e., of order O(1/n), in order to obtain optimal rates. From a more applied perspective, we have also derived results based on the bias-variance decomposition to design a meothod to solve the empirical risk minimization problem which is both efficient and statistically optimal.

In particular, we have adapted the dimension reduction techniques of least-squares [START_REF] Rudi | Less is more: Nyström computational regularization[END_REF] to handle the fact that a priori, the empirical risk minimization problem is represented in a large dimensional space. We then proposed a novel second-order scheme, resembling an interior point method, to optimize the modified problem in a way which is less depednent on the conditioning of the problem than first order methods : this was crucial because as was shown in the statistical part and was verified empirically on real world data, the regularization can be very small. Finally, we derived optimal rates for the algorithm, as in the work by [START_REF] Rudi | Falkon: An optimal large scale kernel method[END_REF] in the least squares setting, showing that we have an optimal algorithm with limited dependence to the condition number, and low complexity.

In part II, we introduced and analyzed a model for non-negative functions based on kernels (PSD models), which is PSD representable, i.e., is parametrized by a (subspace of a) PSD matrix cone. We ansalysed the basic properties of this model in terms of approximation and optimization guarantees. We showed that it comes with the same advantages as (non-parametric) linear models, inheriting the nice properties of kernel methods to model real or vector valued functions.

We have applied this model to sampling from an un-normalized density function, using it to model the density in a way where we can easily sample from it.

Finally in part III, we have used PSD models in a completely different context, to derive an algorithm for global (non-convex) optimization via function evaluations by modelling a non-negativity constrain using a PSD model. We derived guarantees for that algorithm when minimizing a function defined on a bounded domain of R d , assuming that it is regular, of class C r . In particular, we showed that our algorithm almost matches the worst case lower bounds for global optimization of a C r function based on function evaluations given by [START_REF] Novak | Deterministic and Stochastic Error Bounds in Numerical Analysis[END_REF], showing that our global optimization algorithm reaches error ε using roughly n = ε -d/r evaluation points. The method to obtain these guarantees was crucially based on a decomposition of the non-negative function fmin(f ) as a sum of squares of regular functions. In a second work in this part, we therefore explored second-order sufficient conditions in order to decompose C r functions as sums of squares of C r-2 functions, in order to understand what can be achieved by our algorithm in the context of C r regularity, and extend the cornerstone results to handle functions with continuous sets of minimizers and which are defined on a manifold, thus paving the way to extend the guarantees of the method to a wider variety of spaces when working with the C r -type regularity.

Research perspectives

We conclude this manuscript with some perspectives and open questions related to the different areas and problems we have worked with. We will keep the same structure as that of the thesis, developing questions and perspectives raised by each part. Note that this is an informal presentation of these perspectives, and that the conjectures we will formulate are not made to be exact; we will omit certain elements for simplicity, keeping to the main ideas.

. RESEARCH PERSPECTIVES CHAPTER 11. CONCLUSION AND PERSPECTIVES

Without making a list of the different field of applications which can be found in the book by [START_REF] Lasserre | Moments, Positive Polynomials and their Applications[END_REF], we can cite the particular example of finding Lyapunov functions for certain dynamics using kernel sum of squares, which we would like to study in the future.

Making a toolbox

Finally, as we can see with the development of gloptipoly, building a good toolbox with our tools for global optimization will be necessary both for us, in order to see if we are really able to tackle real-life problems with an off the shelf method, and for users if they are to try it in a user friendly way. However, we believe we must wait for the method to be slightly more mature to start implementing and publishing this. In particular, we have to see if there is no way of having a posteriori guarantees, and implement automatic ways to tune the hyperparameter. Finally, we must incorporate the constrained setting from an algorithmic point of view, in order to deal with these problems as well.

MOTS CL ÉS

Apprentissage statistique, estimation non param étrique, espaces de Hilbert à noyau reproduisant, mod èles semid éfinis positifs, sommes de carr és, optimisation globale, m éthode du deuxi ème ordre, m éthode de Newton, conditions du deuxi ème ordre, échantillonnage.

R ÉSUM É

Cette th èse se situe à l'interaction entre les domaines de l'apprentissage statistique, de l'optimisation, et de la mod élisation. Nous étudions diff érents mod èles de fonctions, fond és sur les espaces de Hilbert à noyau reproduisant. Nous commenc ¸ons par étudier les propri ét és statistiques et algorithmiques de l'estimateur de minimisation du risque empirique r égularis é, dans le cadre de la minimisation du risque (en esp érance) sur des espaces à noyau reproduisant. Nous g én éralisons la d écomposition biais-variance tr ès pr écise bien connue dans le cadre de la r égression lin éaire à une plus grande classe de fonctions en controlant pr écis ément l' évolution locale de leurs Hessiennes. Cette classe de fonctions appel ées fonctions autoconcordante g én éralis ées contient la perte logistique, qui est tr ès utilis ée en classification. Nous étendons également les taux de convergence rapides ainsi que les algorithmes rapides établis dans le cadre de la r égression lin éaire à cette classe de fonctions, caract érisant au passage la difficult é du probl ème statistique sous-jacent à l'aide de deux quantit és caract érisant la r égularit é de la solution, ainsi que celle de l'espace à noyau reproduisant utilis é. Dans un deuxi ème temps, nous introduisons un mod èle pour les fonctions positives, qui est lin éairement param étr é par un op érateur symmetrique sur un espace à noyau reproduisant, et o ù la contrainte de positivit é est garantie au moyen d'une contrainte de positivit é semid éfinie sur cet op érateur. Nous étudions les propri ét és de ce mod èle, et montrons qu'il a de bonnes propri ét és d'approximation, qu'il pr éserve la convexit é des fonctions de perte standard en apprentissage statistique, et qu'il est int égrable et diff érentiable en forme close lorsqu'il est d éfini avec des noyaux particuliers. En outre, cette derni ère propri ét é nous permet d'utiliser ce mod èle pour estimer des densit és de probabilit é, et nous d éveloppons un algorithme d' échantillonage d'une densit é de probabilit é quelconque, accessible à partir de ses valeurs en certains points. Enfin, nous appliquons ce mod èle de fonctions positives afin d'approcher le minimum global d'une fonction r éguli ère. Dans le m ême esprit que les hi érarchies de Lasserre pour l'optimisation polynomiale, nous cherchons le minimum global d'une fonction comme sa plus grande borne inf érieure, et affinons les garanties d'approximation du paragraphe pr éc édent afin d'obtenir un algorithme quasi-optimal pour trouver le minimum global d'une fonction à partir de ses valeurs en un nombre donn é de points. Nous établissons également des r ésultats th éoriques permettant de g én éraliser cette approche à des fonctions d éfinies sur des vari ét és.

ABSTRACT

In this thesis at the boundary between statistical learning, optimization and modelling, we study different non-parametric models for functions based on reproducing kernel Hilbert spaces. We first study the expected risk minimization problem through the solving of the regularized empirical risk minimization problem on a reproducing kernel Hilbert space. We extend the precise bias-variance trade offs known for least squares regression to a broader class of functions by controlling the evolution of their Hessians. This class of functions, called generalized self-concordant functions, includes the logistic loss, which is widely used for classification. We also extend the fast rates and fast algorithms known for least squares to this class of functions, characterizing the difficulty of a problem through interpretable quantities.

We then introduce a model for non-negative functions, which is linearly parametrized by a symmetric operator and which enforces non-negativity through a positive semidefinite constraint on this operator. We study the properties of this model and show that it has good approximation properties, preserves the convexity of machine learning loss functions, and is integrable in closed form in certain cases. In particular, this last property allows us to use this model to approximate probability densities, and we develop a sampling algorithm based on these models to sample from an arbitrary probability density known via function evaluations. Finally, we apply this model for non-negative functions in order to approximate the global minimum of a function with certain regularity properties. In the same spirit as moment-SOS hierarchies for polynomial optimization, we look for the global minimum as the maximum lower bound of the function, and refine our approximation guarantees from the previous paragraph in order to obtain a near optimal algorithm to solve global optimization given a fixed number of function evaluations. We also pave the way to generalize this approach to functions defined on manifolds, by introducing a second order sufficient condition for a non-negative regular function to be decomposable as sums of squares of regular functions.

KEYWORDS

Statistical learning, non-parametric estimation, reproducing Kernel Hilbert spaces, PSD models, sum of squares, global optimization, second order methods, Newton methods, second order conditions, sampling.