Mes premiers remerciements vont à Vianney qui a encadré ma thèse et qui, tout en me laissant une grande liberté dans le choix des problèmes que j'ai explorés, m'a partagé ses connaissances, ses idées, ainsi que sa manière d'aborder les problèmes d'apprentissage séquentiel. Je garderai en mémoire les nombreuses séances passées devant le tableau blanc, puis le tableau numérique, à écrire des équations en oubliant volontairement toutes les constantes.

Je tiens également à remercier l'ensemble des membres de mon jury de thèse. Pouvoir vous présenter mes travaux a été une joie et un honneur. Merci en particulier à Gilles qui a animé avec brio et bonne humeur ma soutenance. Antoine et Panayotis, merci tout spécialement d'avoir relu mon manuscrit. Merci pour l'intérêt que vous y avez porté et pour vos nombreuses remarques qui ont permis d'en améliorer la qualité.

Cette thèse n'aurait bien évidemment jamais pu voir le jour sans un goût prononcé pour les mathématiques que j'ai développé au fil des années. Pour cela, je tiens à remercier l'ensemble de mes professeurs de mathématiques qui ont su me transmettre leur passion, et en particulier mes professeurs de prépa à Ginette, Monsieur Nougayrède pour sa pédagogie et Monsieur de Pazzis pour sa rigueur.

Merci également à l'ensemble du personnel du CMLA qui s'est occupé à merveille de toutes les démarches administratives qu'un doctorant souhaite éviter : merci à Véronique, Virginie et Alina. Merci également d'avoir contribué au bon déroulement des séminaires et autres groupes de travail en assurant la partie essentielle : commander les sandwiches.

J'en profite pour remercier tous mes camarades de thèse qui ont animé le célèbre bureau des doctorants de Cachan. On pourra se vanter d'être la dernière promotion de thésards à avoir connu la cave, ses infiltrations de bourdons et ses conserves de civet ! En particulier merci à Valentin pour le puits de connaissances que tu étais et pour la bibliothèque parallèle que tu avais constituée, ainsi qu'à Pierre pour les centaines de questions et d'idées que tu as présentées sur la vitre de la fenêtre qui faisait office de tableau derrière mon bureau. Un grand merci aussi à Tina pour tes questions existentielles et les nombreux gâteaux dont tu nous as gâtés avant la triste arrivée de ton chien qui a bouleversé l'ordre de tes priorités ! Je garderai aussi en mémoire la ponctualité de Jérémy qui nous a permis de profiter quotidiennement à 11h45, avant le flux de lycéens, du restaurant l'Arlequin (à ne pas tester).

Merci également à tous ceux qui ont su me détacher des mathématiques ces trois années. Vous m'avez apporté l'équilibre indispensable pour tenir sur le long terme. Merci notamment à Jean-Nicolas, à Côme et à Gabriel. Merci aux groupes Even et Bâtisseurs qui m'ont accompagné tout au long de cette thèse et en particulier au Père Masquelier. Merci pour tous ces topos, apéros, week-ends et pélés qui m'ont tant apporté.

Mes deux premières années de thèse sont indissociables d'une aventure dans la jungle meudonnaise. Merci aux 32 louveteaux dont j'ai eu la charge au cours de ces deux années comme Akela. Mieux que quiconque vous avez su me changer les idées et me faire oublier la moindre équation. Merci pour vos sourires que je n'oublierai jamais. Merci également à Kaa, Bagheera et Baloo d'avoir formé la meilleure maîtrise que j'aurais pu imaginer. Merci aussi au Père Roberge pour tout ce que vous m'avez apporté aux louveteaux et aujourd'hui encore.

Merci finalement à ma famille. Pendant ces trois années mes frères n'ont pas manqué une occasion de me demander comment avançait la thèse, maintenant ainsi une pression constante sur mes épaules. Merci à mes parents d'avoir accepté mes choix, même s'ils ne comprenaient pas pourquoi je n'avais pas un "vrai" métier. Même si je n'ai jamais vraiment su vous expliquer ma thèse, merci de m'avoir soutenu dans cette voie.

Merci aussi à vous tous qui allez vous aventurer au-delà des remerciements, vous donnez du sens à cette thèse.

Enfin, merci à toi mon Hermine. Ton soutien inconditionnel pendant cette thèse m'a été précieux. Tu as été ma motivation et ma plus grande source de joie pendant ces années. Merci pour ta douceur et ton amour jour après jour.

Abstract

Stochastic optimization algorithms are a central tool in machine learning. They are typically used to minimize a loss function, learn hyperparameters and derive optimal strategies. In this thesis we study several machine learning problems that are all linked with the minimization of a noisy function, which will often be convex. Inspired by real-life applications we choose to focus on sequential learning problems which consist in situations where the data has to be treated "on the fly" i.e., in an online manner. The first part of this thesis is thus devoted to the study of three different sequential learning problems which all face the classical "exploration vs. exploitation" trade-off. In each of these problems a decision maker has to take actions in order to maximize a reward or to evaluate a parameter under uncertainty, meaning that the rewards or the feedback of the possible actions are unknown and noisy. The optimization task has therefore to be conducted while estimating the unknown parameters of the feedback functions, which makes those problems difficult and interesting. As in many sequential learning problems we are interested in minimizing the regret of the algorithms we propose i.e., minimizing the difference between the achieved reward and the best possible reward that can be done with the knowledge of the feedback functions. We demonstrate that all of these problems can be studied under the scope of stochastic convex optimization, and we propose and analyze algorithms to solve them. We derive for these algorithms minimax convergence rates using techniques from both the stochastic convex optimization field and the bandit learning literature. In the second part of this thesis we focus on the analysis of the Stochastic Gradient Descent (SGD) algorithm, which is likely one of the most used stochastic optimization algorithms in machine learning. We provide an exhaustive analysis in the convex setting and in some non-convex situations by studying the associated continuous-time model. The new analysis we propose consists in taking an appropriate energy function to derive convergence results for the continuous-time model using stochastic calculus, and then in transposing this analysis to the discrete case by using a similar discrete energy function. The insights gained by the continuous case help to design the proof in the discrete setting, which is generally more intricate. This analysis provides simpler proofs than existing methods and allows us to obtain new optimal convergence results in the convex setting without averaging as well as new convergence results in the weakly quasi-convex setting. Our method emphasizes the links between the continuous and discrete models by presenting similar statements of the theorems as well as proofs with the same structure.

Résumé

Les algorithmes d'optimisation stochastique sont centraux en apprentissage automatique et sont typiquement utilisés pour minimiser une fonction de perte, apprendre des hyperparamètres ou bien trouver des stratégies optimales. Dans cette thèse nous étudions plusieurs problèmes d'apprentissage automatique qui feront tous intervenir la minimisation d'une fonction bruitée qui sera souvent convexe. Du fait de leurs nombreuses applications nous avons choisi de nous concentrer sur des problèmes d'apprentissage séquentiel, dans lesquels les données doivent être traitées "à la volée", ou en ligne. La première partie de cette thèse est donc consacrée à l'étude de trois différents problèmes d'apprentissage séquentiel qui font tous intervenir le compromis classique entre "exploration et exploitation". En effet, dans chacun de ces problèmes on considère un agent qui doit prendre des décisions pour maximiser une récompense ou bien pour évaluer un paramètre dans un environnement incertain, c'est-à-dire que les récompenses ou les résultats des actions possibles sont inconnus et bruités. Il faut donc mener à bien la tâche d'optimisation tout en estimant les paramètres inconnus des fonctions de récompense, ce qui fait toute la difficulté et l'intérêt de ces problèmes. Comme dans de nombreux problèmes d'apprentissage séquentiel, nous cherchons à minimiser le regret de nos algorithmes, qui est la différence entre la meilleure récompense que l'on pourrait obtenir avec la pleine connaissance des paramètres du problème, et la récompense que l'on a effectivement obtenue. Nous mettons en évidence que tous ces problèmes peuvent être étudiés grâce à des techniques d'optimisation stochastique convexe, et nous proposons et analysons différents algorithmes pour résoudre ces problèmes. Nous prouvons des vitesses de convergence optimales pour nos algorithmes en utilisant à la fois des outils d'optimisation stochastique et des techniques propres aux problèmes de bandits. Dans la seconde partie de cette thèse nous nous concentrons sur l'analyse de l'algorithme de descente de gradient stochastique, qui est vraisemblablement l'un des algorithmes d'optimisation stochastique les plus utilisés en apprentissage automatique. Nous en présentons une analyse complète dans le cas convexe ainsi que dans certaines situations non convexes, en analysant le modèle continu qui lui est associé. L'analyse que nous proposons est nouvelle et consiste à étudier une fonction d'énergie bien choisie pour obtenir des résultats de convergence pour le modèle continu avec des techniques de calcul stochastique, puis à transposer cette analyse au cas discret en utilisant une énergie discrète similaire. Le cas continu apporte donc une intuition très utile pour construire la preuve du cas discret, qui est généralement plus complexe. Notre analyse donne donc lieu à des preuves plus simples que les méthodes précédentes et nous permet d'obtenir de nouvelles vitesses de convergence optimales dans le cas convexe sans moyennage, ainsi que de nouveaux résultats de convergence dans le cas faiblement quasi-convexe. Nos travaux mettent en lumière les liens entre les modèles discret et continu en présentant des théorèmes similaires et des preuves qui partagent la même structure.

Introduction 1 Motivations

Optimization problems are encountered very often in our everyday life: how to optimize our time, how to minimize the duration of a trip, how to maximize the gain of a financial investment under some risk constraints? Constrained and unconstrained optimization problems appear in various mathematical fields, such as control theory, operations research, finance, optimal transport or machine learning. The main focus of this thesis will be to study optimization problems that arise in the machine learning field. Despite its numerous and very different domains of application, such as Natural Language Processing, Image Processing, online advertisement, etc., all machine learning algorithms rely indeed on the concept of optimization, and more precisely on stochastic optimization. One usually analyzes machine learning under the framework of statistical learning, which aims at finding (or learning) on a precise task the best predictive function based on some data, i.e., the most probable function fitting the data. In order to reach this goal optimization techniques are often used, for example to minimize a loss function, to find appropriate hyperparameters or to maximize an expected gain.

In this thesis we will focus on the study of a specific class of statistical learning problems where data is obtained and treated on the fly, which is known as sequential or online learning [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF], as opposed to batch or offline learning where data have been collected beforehand. The major difficulty of sequential learning problems is precisely the fact that the decision maker has to construct a predictor function without knowing all the data. That is why online algorithms usually perform worse than their offline counterpart where the decision maker has access to the whole dataset. However online settings can have advantages as well when the decision maker plays an active role in the data collection process. In this domain of machine learning, usually called active learning [START_REF] Settles | Active learning literature survey[END_REF], the decision maker will be able to choose which data to collect and to label. Being part of the data selection process can improve the performance of the machine learning algorithm, since the decision maker will collect the most informative data. In sequential learning problems the decision maker may be required to take decisions at each time step, for example to select an action to perform, which will impact the rest of the learning process. For example, in bandit problems [START_REF] Bubeck | Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems[END_REF], which are a simple way to model sequential decision making under uncertainty, an agent has to choose between several actions (generally called "arms") in order to maximize a reward. This maximization objective implies therefore choices of the agent, who can choose to select the current best arm, or instead to select another arm in order to explore the different options and to acquire more knowledge about them. This trade-off between exploitation and exploration is one of the major issues in bandit-related problems. In the first three chapters of the present thesis we will study sequential or active learning problems where this kind of trade-off appears. The goal will always be to minimize a quantity, known as "regret" which quantifies the difference between the best policy that would have been chosen by an omniscient decision maker, and the actual policy.

In machine learning, the optimization problems we usually deal with concern objective functions that have the particularity to be either unknown or noisy. For example, in the classical stochastic bandit problem [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF][START_REF] Auer | Finite-time Analysis of the Multiarmed Bandit Problem[END_REF] the decision maker wants to maximize a reward which depends on the unknown probability distributions of the arms. In order to gain information on these distributions, the decision maker receives at each time step a feedback (typically, the reward of the selected arm) that will be used to make future choices. In the bandit setting, we usually speak of "limited feedback" (or "bandit feedback") as opposed to the "full-information setting" where the rewards of all the arms (and not only the selected one) are revealed to the decision maker. The difficulty of such problems does not only lie in the limited feedback setting, but also in the noisiness of the information: the rewards of the arms correspond indeed to noisy values of the arms' expectations. This is also the case of the Stochastic Gradient Descent (SGD) algorithm [START_REF] Robbins | A stochastic approximation method[END_REF] which is used when one wants to minimize a differentiable function with only access to noisy evaluations of its gradient. This is why machine learning needs to use stochastic optimization, which consists in optimizing functions whose values depend on random variables. Since the algorithms we deal with are stochastic, we will usually want to obtain results in expectation or in high probability. The field of stochastic optimization is very broad and we will present different aspects of it in this thesis.

One of the main characteristics of an optimization algorithm, apart from actually minimizing the function, is the speed at which it will reach the minimum, or the precision it can guarantee after a fixed number of iterations, or within a fixed budget. For example, the objective of bandit algorithms is to obtain a sublinear bound (in T , the time horizon of the algorithm) on the regret, and the objective of SGD is to bound E[f (x n )]-min x∈R d f by a quantity depending on the number of iterations n. A machine learning algorithm has indeed to be efficient and precise, meaning that the optimization algorithms it uses need to have fast convergence guarantees. Deriving convergence results for the algorithms we study will be one of the major theoretical issues that we tackle in this thesis. Furthermore, after having established a convergence bound of an optimization algorithm, one has to ask the question whether this bound can be improved, either by a more careful analysis of the algorithm, or by a better algorithm to solve the problem at hand. There exist two ways to answer this question. The first and obvious one is to compare the algorithm performance against known results from the literature. The second one is to prove a "lower bound" on the considered problem, which is a convergence rate that cannot be beaten. If this lower bound matches the convergence rate of the algorithm (known as "upper bound"), the algorithm is said to be "minimax-optimal", meaning that it is the best that can be developed. In this thesis, whenever it is possible, we will compare our results with the literature, or establish lower bounds, in order to obtain an insight of the relevance of our algorithms.

An important tool to derive convergence rates of optimization algorithms is the complexity of the problem at hand. The more complex the problem (or the less specified), the slower the algorithms. For example, trying to minimize an arbitrary function over R d is much more complicated than minimizing a differentiable and strongly convex function. In this thesis, the complexity of a problem will often be characterized by measures of the regularity of the functions we consider: the more regular, the easier the problem. Thus each chapter will begin with a set of assumptions that will be made on the problem, in order to make it tractable and to derive convergence results. We will see how relaxing some of the assumptions will impact the convergence rates. For example, in Chapter 3 and Chapter 4 we will establish convergence rates of stochastic optimization algorithms depending on the exponent of the Łojasiewicz inequality [START_REF] Łojasiewicz | Ensembles semi-analytiques[END_REF][START_REF] Karimi | Linear convergence of gradient and proximal-gradient methods under the Polyak-Łojasiewicz condition[END_REF]. We will see that varying this exponent increases or decreases the complexity of the problem, thus influencing to the convergence rates we obtain. However real-life problems and applications are not always convex or smooth and do not always verify such inequalities. For example, stochastic optimization algorithms such as SGD have often known guarantees [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF] in the convex (or even strongly convex) setting, whereas very few results are available in the non-convex setting, which is nevertheless the most common case, for example in deep learning applications. Tackling those issues will be one of the challenges of this thesis.

The actual performances of an optimization algorithm can be considerably better than the theoretical rates that can be proved. This is typically the case of the aforementioned stochastic optimization algorithms which are extensively used in deep learning without proven convergence guarantees. In order to compare against reality we will illustrate the convergence results we obtain in this thesis with numerical experiments.

In the rest of this opening chapter we will present the different statistical learning and optimization problems that we have studied in this thesis, as well as the main mathematical tools needed. We will conclude with a detailed chapter-by-chapter summary of the contributions of the present thesis and a list of the publications it has led to.

Presentation of the problems 2.1 Stochastic contextual bandits (Chapter 1)

Consider a decision maker who has access to K ∈ N * arms, each corresponding to an unknown probability distribution ν i , for i ∈ {1, . . . , K}. Suppose that at each time step t ∈ {1, . . . , T },1 the decision maker can sample one of those arms i t ∈ {1, . . . , K} and receives a reward Y (it) t distributed from ν it , of expectation µ it . The goal of the decision maker is then to maximize his cumulative total reward T t=1 Y

(it) t

. Since the rewards are stochastic we will rather aim at maximizing the expected total reward E T t=1 µ it , where the expectation is taken on the randomness of the decision maker's actions. Consequently we are usually interested in minimizing the regret (or more precisely the "pseudo-regret")

R(T ) = T max 1≤i≤K µ i -E T t=1 µ it .
(1) This is the classical formulation of the "Stochastic Multi-Armed Bandit problem" [START_REF] Bubeck | Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems[END_REF] which can be solved with the famous Upper-Confidence Bound (UCB) algorithm introduced by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]. This problem can be used to model various situations where an "exploration vs. exploitation" trade-off has to be found. This is for example the case in clinical trials or online advertisement where one wants to evaluate the best ad to display while maximizing the number of clicks. However such a setting seems too limited to propose an appropriate solution to the clinical trials problem or to the online advertisement problem. Indeed, all patients or Internet users do not behave the same way, and an ad can be well-suited for someone and completely inappropriate for someone else. We see here that the aforementioned setting is too restricted, an in particular the hypothesis that each arm i has a fixed expectation µ i is unrealistic. For this reason we need to introduce a context set X = [0, 1] d which corresponds to the different possible profiles of patients or web users of our problem. Each context x ∈ X characterizes a user and we now suppose that the rewards of the K arms depend on the context x. This problem, known as bandits with side information [START_REF] Wang | Bandit problems with side observations[END_REF] or contextual bandits [START_REF] Langford | The epoch-greedy algorithm for multi-armed bandits with side information[END_REF], models more accurately the clinical trials or online advertisement situations. We will now suppose that at each time step t ∈ {1, . . . , T }, the decision maker is given a random context variable X t ∈ X and has to choose an arm i t whose reward Y (it) t will depend on the context variable X t . We denote therefore for each i ∈ {1, . . . , K}, µ i : X → R the conditional expectation of the reward of arm i with respect to the context variable X, which is now a function of the context x:

E[Y (i) |X = x] = µ i (x), for all x ∈ X .
In order to take full advantage of the context variables, we have to make some regularity assumptions on the reward functions. We want indeed to ensure that the rewards of an arm will be similar for two close context values (i.e., two similar individuals). A way to model this natural assumption is for example to suppose that the µ i functions are Lipschitz-continuous. This setting of nonparametric contextual stochastic bandits has been studied by [START_REF] Rigollet | Nonparametric Bandits with Covariates[END_REF] for the case of K = 2 and then by [START_REF] Perchet | The multi-armed bandit problem with covariates[END_REF] for the general case. In this setting the objective of the decision maker is to find a policy π : X → {1, . . . , K}, mapping a context variable to an arm to pull. Of course, as in classical stochastic bandits, the action chosen by the decision maker will depend on the history of the previous pulls. We can now define the optimal policy π and the optimal reward function µ which are π (x) ∈ arg max i∈{1,...,K} µ i (x) and µ (x) = max i∈{1,...,K} µ i (x) .

This gives the following expression of the regret after T samples:

R(T ) = T t=1 E µ (X t ) -µ π(Xt) (X t ) .
(2)

Even if (2) is very close to (1), one of the difficulties in minimizing ( 2) is that one cannot expect to collect several rewards for the same context value since the context space can be uncountable.

In nonparametric statistics [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF] a common idea to estimate an unknown function f over X is to use "regressograms", which are piecewise constant estimators of the function. They work similarly to histograms, by using a partition of X into bins and by estimating f (x) by its mean value on the corresponding bin. Regressograms are an alternative technique to Nadaraya-Watson estimators [START_REF] Nadaraya | On estimating regression[END_REF][START_REF] Watson | Smooth regression analysis[END_REF] which rather use kernels as weighting functions instead of fixed bins.

A possible solution to the problem of stochastic contextual bandits is to draw inspiration from these regressograms and to use a partition of the context space X into bins and to treat the contextual bandit problem as separate independent instances of classical stochastic (without context) bandit problems on each bin. This is done by running a classical bandit algorithm such as UCB or ETC [START_REF] Even-Dar | Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems[END_REF] separately on each of the bins, leading for example to the "UCBogram" policy [START_REF] Rigollet | Nonparametric Bandits with Covariates[END_REF]. Such a strategy is of course possible only because of the smoothness assumption we have previously done, which ensures that considering the reward functions µ i constant on each bin does not lead to a high error.

Instead of assuming that the µ i functions are Lipschitz-continuous, Perchet and Rigollet (2013) make a weaker assumption that is very classical in nonparametric statistics, and assume that the µ i functions are β-Hölder for β ∈ (0, 1], meaning that for all i ∈ {1, . . . , K}, for all (x, y) ∈ X 2 , |µ i (x) -µ i (y)| ≤ L x -y β , and obtain under this assumption the following classical bound on the regret R(T ) (where we only kept the dependency in T , and not in K) R(T ) T 1-β/(2β+d) . Now that we have a solution for the contextual stochastic bandit problem we can wonder whether this setting is still realistic. Indeed, let us take again the example of online advertisement. Suppose that an online advertisement company wishes to use a contextual bandit algorithm to define its policy. The company was using other techniques but does not want to risk to lose too much money by setting up a new policy. This situation is part of a much wider problem which is known as safe reinforcement learning [START_REF] García | A comprehensive survey on safe reinforcement learning[END_REF] which deals with learning policies while respecting some safety constraints. In the more specific domain of bandit algorithms, [START_REF] Wu | Conservative bandits[END_REF] have proposed an algorithm called "Conservative UCB" whose goal is to run a UCB algorithm while maintaining uniformly in time a guarantee that the reward achieved by this UCB strategy is at least larger than 1 -α times the reward that would have been obtained with a previous strategy. In order to do that the authors' idea is to add an additional arm corresponding to the old strategy and to pull it as soon as there is a risk to violate the reward constraint. In Chapter 1 we will adopt another point of view on this problem: instead of imposing a constraint on the reward we will add a regularization term to force the obtained policy to be close to a fixed policy chosen in advance.

In bandit problems the decision maker has to choose actions in order to maximize a reward but he is generally not interested in precisely estimating the mean value of each of the arms. This is a different problem that also has its own interest. However the task of estimating the mean of each of the arms is not compatible with the one of maximizing the reward, since one also has to sample the suboptimal arms. In the next section we will discuss a generalization of this problem which consists in wisely choosing which arm to sample in order to maximize the knowledge about an unknown parameter (which can be the vector of the means of all the arms).

From linear regression to online optimal design of experiments (Chapter 2)

Let us now consider the widely-studied problem of linear regression. In this problem a decision maker has access to a dataset of input/output pairs {(x i , y i )} i=1,...,n of n observations, where (x i , y i ) ∈ R p × R for every i ∈ {1, . . . , n}. These data points are assumed to follow a linear model: ∀i ∈ {1, . . . , n} , y i = x i β + ε i , where β ∈ R p is the parameter vector2 and ε = (ε 1 , . . . , ε n ) is a noise vector which models the error term of the regression. In the following we will assume that this noise is centered and that is has finite variance:

∀i ∈ {1, . . . , n} , E ε 2 i = σ 2 i < ∞ .

We first consider the homoscedastic case, meaning that σ 2 i = σ 2 for all i ∈ {1, . . . , n}. In order to deal with linear regression problems, one usually introduces the "design matrix" X and the observation vector Y defined as follows

X =    • • • x 1 • • • . . . • • • x n • • •    ∈ R n×p and Y =    y 1 . . . y n    ∈ R n , which gives Y = Xβ + ε .
The goal of linear regression is to estimate the parameter β by a β ∈ R p in order to minimize the least squares error L(β) between the true observation values y i and the predicted ones X i β:

L(β) = n i=1 (y i -x i β) 2 = Y -Xβ 2 2 .
We define then β arg min β∈R p L(β) as the optimal estimator of β . Using standard computations we obtain the well-known formula of the Ordinary Least Square (OLS) estimator: β = (X X) -1 X Y , giving the following relation between β and β:

β = β + (X X) -1 X ε .
Consequently, the covariance matrix of the estimation error β -β is

Ω E (β -β)(β -β) = σ 2 (X X) -1 = σ 2 n i=1 x i x i -1
, which characterizes the precision of the estimator β.

As demonstrated above, linear regression is a simple and well-understood problem. However it can be the starting point of several more complex and more interesting problems. Let us for example assume that the vectors x 1 , . . . , x n are not fixed any more, but that they rather could be chosen among a set of candidate covariate vectors of size K > 0 {X 1 , . . . , X K }. The decision maker has now to choose each of the of the x i as one of the X k (with the possibility to choose several times the same X k ). The motivation comes from situations where one can perform different experiments (corresponding to the covariates X 1 , . . . , X K ) to estimate an unknown vector β . The goal of the decision maker is then to choose appropriately the experiments to perform in order to minimize the covariance matrix Ω of the estimation error. Denoting n k the number of times that the covariate vector X k has been chosen, one can rewrite

Ω = σ 2 n k=1 n k X k X k -1
. This problem, as formulated above, is known under the name of "optimal experiment design" [START_REF] Boyd | Convex optimization[END_REF][START_REF] Pukelsheim | Optimal Design of Experiments[END_REF]. Minimizing Ω is an illformulated problem since there is no complete order on the cone of positive semi-definite matrices. Therefore several criteria have been proposed, see [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF], among which the most used are the D-optimal design which aims at minimizing det(Ω), the Eoptimal design which minimizes Ω 2 and the A-optimal design whose goal is to minimize Tr(Ω), all these minimization problems being under the constraint that K k=1 n k = n. All of them are convex problems, which are therefore easily solved, if one relaxes the integer constraint on the n k .

Let us now remove the homoscedasticity assumption and consider the more general heteroscedastic setting where the variances of the points X k are not supposed to be equal. The covariance matrix Ω becomes then

Ω = n k=1 n k σ 2 k X k X k -1 .
Note that the heteroscedastic setting corresponds actually to the homoscedastic one with the X k rescaled by 1/σ k and therefore the previous analysis still applies. However it becomes completely different if the variances σ k are unknown. Indeed minimizing Ω with unknown variances requires to estimate these variances. However using too many samples to estimate the values of σ k can increase the value of Ω. We face therefore again in this setting an "exploration vs. exploitation" dilemma. This setting corresponds now to online optimal experiment design, since the decision maker has to construct sequentially the best experiment plan by taking into account the feedback gathered so far about the previous experiments. It is also close to the "active learning" setting where the agent has to choose which data point to label or not. As explained in [START_REF] Willett | Faster rates in regression via active learning[END_REF] there are two categories of active learning: selective sampling where the decision maker is presented a series of samples and chooses which one to label or not, and adaptive sampling where the decision maker chooses which experiment to perform based on previous results. The setting we described above corresponds to adaptive sampling applied to the problem of linear regression. Using active learning can have many benefits compared to standard offline learning. Indeed some points can have a very large variance and obtaining precise information requires therefore many samples thereof. Using active learning techniques for linear regression should therefore improve the precision of the obtained estimator.

Let us now consider the simpler case where p = K and where the points X k are actually the canonical basis vectors e 1 , . . . , e K of R K . If we note also µ β , we see that X k β = e k µ = µ k and we can identify this setting with a multi-armed bandit problem with K arms of means µ 1 , . . . , µ K . The goal is now to obtain estimates μ1 , . . . , μK of the means µ 1 , . . . , µ K of each of the arms. This is setting has been studied by [START_REF] Antos | Active learning in heteroscedastic noise[END_REF] and [START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF] with the objective to minimize max 1≤k≤K E (µ k -μk ) 2 , which corresponds to estimating equally well the mean of each arm. Another criterion that could be minimized instead of the ∞ -norm of the estimation errors is their 2 -norm:

K k=1 E (µ k -μk ) 2 = E K k=1 (β k -βk ) 2 = E β -β 2 2 .
Note that this problem is very much related to the optimal experiment design problem presented above since E[ β -β 2 2 ] = Tr(Ω). Thus minimizing the 2 -norm of the estimation errors of the means in a Multi-Armed Bandits (MAB) problem corresponds to solving online an A-optimal design problem. The solutions proposed by [START_REF] Antos | Active learning in heteroscedastic noise[END_REF] and [START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF] can be adapted to the 2 -norm setting, and leverage ideas that are common in the bandit literature to deal with the exploration vs. exploitation trade-off. [START_REF] Antos | Active learning in heteroscedastic noise[END_REF] use a greedy algorithm that samples the arm k maximizing the current estimate of E (µ k -μk ) 2 while using forced sampling to maintain each n k greater than α √ n, where α > 0 is a well-chosen parameter. In this algorithm the forced sampling guarantees to explore the options that could have been underestimated. In [START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF] the authors use a similar strategy since they pull the arm that minimizes σ2 k /n k (which estimates E (µ k -μk ) 2 ) corrected by a UCB term to perform exploration. Both strategies obtain similar regret bounds which scale in O(n -3/2 ). However they heavily rely on the fact that the covariates X 1 , . . . , X k form the canonical basis of R K . In order to deal with the general setting one will have to use more sophisticated ideas.

We have seen that actively constructing a design matrix for linear regression requires to use stochastic convex optimization techniques. In the next section we will actually exhibit more fundamental links between active learning and stochastic convex optimization, highlighting the fact that both fields are deeply related to each other.

Active learning and adaptive stochastic optimization (Chapter 3)

Despite their apparent differences the fields of stochastic convex optimization and active learning bear many similarities beyond their sequential aspect. Feedback is indeed central in both fields to decide which action to choose, or which point to explore. The links between active learning and stochastic optimization have been exhibited by [START_REF] Raginsky | Information complexity of black-box convex optimization: A new look via feedback information theory[END_REF] and then further explored by Ramdas and Singh (2013a,b) among others, who present an interesting relation between the complexity measures used in active learning and in stochastic convex optimization. Consider for example a (ρ, µ)-uniformly convex differentiable function f on [0, 1] [START_REF] Zǎlinescu | On uniformly convex functions[END_REF][START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] i.e., a function verifying, for µ > 0 and ρ ≥ 2, 3

∀(x, y) ∈ [0, 1] 2 , f (y) ≥ f (x) + ∇f (x), y -x + µ 2 x -y ρ .
Suppose now that one wants to minimize this function f over [0, 1] i.e., to find its minimum x that we suppose to lie in (0, 1). We have, for all x ∈ [0, 1],

f (x) -f (x ) ≥ µ 2 x -x ρ .
Notice that this condition is very similar to the so-called Tsybakov Noise Condition (TNC) which arises in statistical learning [START_REF] Castro | Minimax bounds for active learning[END_REF]. Consider now the standard classification task on [0, 1]: a decision maker has access to a dataset D = {(X 1 , Y 1 ), . . . , (X n , Y n )} of n independent random copies of (X, Y ) ∈ [0, 1] × {-1, +1}, where Y i is the label of the point X i . His goal is to learn a decision function g : [0, 1] → {-1, +1} minimizing the probability of classification error, often called risk R(g) = P (g(X) = Y ) .

It is well known that the optimal classifier is the Bayes classifier g defined as follows

g (x) = 21 η(x)≥1/2 -1 ,
where η(x) = P (Y = 1 |X = x) is the posterior probability function. We say that η satisfies the TNC with exponent κ > 1 if there exists λ > 0 such that

∀x ∈ [0, 1], |η(x) -1/2| ≥ λ x -x κ .
Now, go back to the minimization problem of the uniformly convex function f on [0, 1]. Suppose we want to use a stochastic first order algorithm i.e., an algorithm that has access to an oracle giving noisy evaluations ĝ(x) of ∇f (x) at each step. Suppose also for simplicity that ĝ(x) = ∇f (x) + z where z is distributed from a standard gaussian random variable independent of x. Moreover, observe that f (x) ≤ 0 for x ≤ x and f (x) ≥ 0 for x ≥ x since f is convex. We can now notice that if all points x ∈ [0, 1] are assigned a label equal to sign(ĝ(x)) then the problem of minimizing f is equivalent to the one of finding the best classifier of the points on [0, 1], since in this case η(x) = P (ĝ(x)

≥ 0 | x) ≥ 1/2 iff x ≥ x .
The analysis conducted by [START_REF] Ramdas | Optimal rates for first-order stochastic convex optimization under tsybakov noise condition[END_REF] shows that for x ≥ x ,

η(x) = P (ĝ(x) ≥ 0 | x) = P f (x) + z ≥ 0 | x = P (z ≥ 0) + P z ∈ -f (x), 0 ≥ 1/2 + λf (x) for λ > 0 ,
and similarly for x ≤ x , η(x) ≥ 1/2 + λ|f (x)| .

Using Cauchy-Schwarz inequality, the convexity of f and finally its uniform convexity we obtain that

|∇f (x)||x -x | ≥ ∇f (x), x -x ≥ f (x) -f (x ) ≥ µ 2 x -x ρ .
This finally shows that

∀x ∈ [0, 1] , |η(x) -1/2| ≥ λµ 2 x -x ρ-1 ,
meaning that η satisfies the TNC with exponent κ = ρ -1 > 1. This simple analysis exhibits clearly the links between actively classifying points in [0, 1] and optimizing a uniformly convex function on [0, 1] using stochastic first-order algorithms. In (Ramdas and Singh, 2013a) the authors leverage this connection to derive a stochastic convex optimization algorithm of a uniformly convex function only using noisy gradient signs, by running an active learning subroutine at each epoch. An important concept in both active learning and stochastic optimization is to quantify the convergence rate of any algorithm. This rate generally depends on regularity measures of the objective function and in the aforementioned setting it will depend either on the exponent κ in the Tsybakov Noice Condition or on the uniform convexity constant ρ. [START_REF] Ramdas | Optimal rates for first-order stochastic convex optimization under tsybakov noise condition[END_REF] show for example that the minimax function error rate of the stochastic first-order minimization problem of a ρ-uniformly convex and Lipschitz continuous function is Ω n -ρ/(2ρ-2) where n is the number of oracle calls. Remark that we recover the Ω(n -1 ) rate of strongly convex functions (ρ = 2) and the Ω(n -1/2 ) rate of convex functions (ρ → ∞). Note moreover that this convergence rate shows that the intrinsic difficulty of a minimization problem is due to the local behavior of the function around the minimum x : the bigger ρ, the flatter the function and consequently the harder the minimization.

One major issue in stochastic optimization is that one might not know the actual regularity of the function to minimize, and more particularly its uniform convexity exponent. Despite this fact many algorithms rely on these values to adujst their own parameters. For example the algorithm EpochGD [START_REF] Ramdas | Optimal rates for first-order stochastic convex optimization under tsybakov noise condition[END_REF]) leverages the -unrealistic in practice -knowledge of ρ to minimize the function. This is why one actually needs "adaptive" algorithms that are agnostic to the constants of the problem at hand but that will adapt to them to achieve the desired convergence rates. Building on the work [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF], [START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] and Ramdas and Singh (2013a) have proposed adaptive algorithms to perform stochastic minimization of uniformly convex functions. They obtained the same convergence rate O(n -ρ/(2ρ-2) ), but this time without using the knowledge of ρ. Both of these algorithms used a succession epochs where an approximate value of x is computed using averaging or active learning techniques.

Despite the fact that stochastic convex optimization is often performed using firstorder methods i.e., with noisy gradient feedback, other settings can be interesting to consider. For example in the case of noisy zeroth-order convex optimization [START_REF] Bach | Highly-Smooth Zero-th Order Online Optimization[END_REF] one has to optimize the function using only noisy values of the current evaluation point f (x t ) + ε. This corresponds actually to using "bandit feedback" i.e., to knowing only a noisy value of the chosen point, to optimize the function f . Generally when speaking of bandit feedback one is more interested in minimizing the regret

R(T ) = T t=1 f (x t ) -f (x ) ,
rather than the function error f (x T ) -f (x ). The former is actually more challenging because the errors made at the beginning of the optimization stage count in the regret. This problem of stochastic convex optimization with bandit feedback has been studied by [START_REF] Agarwal | Stochastic convex optimization with bandit feedback[END_REF] who proposed for the 1D case an algorithm sampling three equally-spaced points x l < x c < x r in the feasible region, and which discards a portion of the feasible region depending on the value of f on these points. This algorithm achieves the optimal rate of O( √ T ) regret. The idea developed by [START_REF] Agarwal | Stochastic convex optimization with bandit feedback[END_REF] have similarities with the binary search, except that they discard a quarter of the feasible region instead of half of it. We also note that some algorithms performing active learning or convex optimization with gradient feedback actually use binary searches. It is for example the case of [START_REF] Burnashev | An interval estimation problem for controlled observations[END_REF] on which the work of [START_REF] Castro | Upper and lower error bounds for active learning[END_REF] is built.

It is interesting to see that stochastic optimization methods using gradient feedback usually aim at minimizing the function error, while it could also be relevant to minimize the regret as in the bandit setting. It is for example the case in the problem of resource allocation that we will define later.

We have discussed so far of many stochastic optimization algorithms using first-order gradient feedback. In the next section we will study the well-known gradient descent algorithm and its stochastic counterpart with an emphasis on the convergence rate of the last point iterate f (x T ) -f (x ).

Gradient Descent and continuous models (Chapter 4)

Consider the minimization problem of a convex and L-smooth4 function f : R d → R:

min x∈R d f (x) .
(3)

There exist plenty of methods to provide solutions to this problem. The most used ones are likely first-order methods i.e., methods using the first derivative, as gradient descent, to minimize the function f . These methods are very popular today because of the constantly increasing sizes of the datasets, which rule out second-order methods (as Newton's method).

The gradient descent algorithm starts from a point x 0 ∈ R d and iteratively constructs a sequence of points approaching x = arg min x∈R d f (x) based on the following recursion:

x k+1 = x k -η∇f (x k ) with η = 1/L . ( 4
)
Even if there exists a classical proof of convergence of this gradient descent algorithm, see [START_REF] Bertsekas | Nonlinear programming[END_REF] for instance, we propose here an alternative proof based on the analysis of the continuous counterpart of (4). Consider a regular function X : R + → R d such that X(kη) = x k for all k ≥ 0. Using a Taylor expansion of order 1 gives

x k+1 -x k = -η∇f (x k ) X((k + 1)η) -X(kη) = -η∇f (X(kη)) η Ẋ(kη) + O(η) = -η∇f (X(kη)) Ẋ(kη) = -∇f (X(kη)) + O(1) ,
suggesting to consider the following Ordinary Differential Equation (ODE)

Ẋ(t) = -∇f (X(t)), t ≥ 0 . ( 5
)
The ODE (5), which is the continuous counterpart of the discrete scheme (4), can be easily analyzed by considering the following energy function, where f = f (x ),

E(t) t(f (X(t)) -f ) + 1 2 X(t) -x 2 .
Differentiating E and using the convexity of f give, for all t ≥ 0,

E (t) = f (X(t)) -f + t ∇f (X(t)), Ẋ(t) + X(t) -x , Ẋ(t) = f (X(t)) -f -t ∇f (X(t)) 2 -∇f (X(t)), X(t) -x ≤ -t ∇f (X(t)) 2 ≤ 0 .
Consequently E is non-increasing and for all t ≥ 0, we have

t(f (X(t)) -f ) ≤ E(t) ≤ E(0) = 1 2 X(0) -x 2
. This gives the following proposition Proposition 1. Let X : R d → R be given by (5). Then for all t > 0

f (X(t)) -f ≤ 1 2t X(0) -x 2 .
We now want to transpose this short and elegant analysis to the discrete setting. We propose therefore to introduce the following discrete energy function

E(k) = kη (f (x k ) -f (x )) + 1 2 x k -x 2 .
First state and prove the following lemma.

Lemma 1. If x k and x k+1 are two iterates of the gradient descent scheme (4), it holds that

f (x k+1 ) ≤ f (x ) + 1 η x k+1 -x k , x -x k - 1 2η x k+1 -x k 2 . ( 6
)
Proof. We have

x k+1 = x k -η∇f (x k ) which gives ∇f (x k ) = x k -x k+1 η .
The descent lemma (Nesterov, 2004, Lemma 1.2.3) and then the convexity of f give

f (x k+1 ) ≤ f (x k ) + ∇f (x k ), x k+1 -x k + L 2 x k+1 -x k 2 ≤ f (x ) + ∇f (x k ), x k -x + x k -x k+1 η , x k+1 -x k + 1 2η x k+1 -x k 2 ≤ f (x ) + 1 η x k+1 -x k , x -x k - 1 2η x k+1 -x k 2 .
This second lemma is immediate and well-known Lemma 2. If x k and x k+1 are two iterates of the gradient descent scheme with have

f (x k+1 ) ≤ f (x k ) - 1 2η x k+1 -x k 2 . ( 7
)
Proof. The descent lemma (Nesterov, 2004, Lemma 1.2.3) gives

f (x k+1 ) ≤ f (x k ) + ∇f (x k ), x k+1 -x k + L 2 x k+1 -x k 2 ≤ f (x k ) - 1 2η x k+1 -x k 2 .
Let us now analyze E(k). Multiplying Equation ( 6) by 1/(k + 1) and Equation ( 7) by k/(k + 1) we obtain

f (x k+1 ) ≤ k k + 1 f (x k ) + 1 k + 1 f (x ) - 1 2η x k+1 -x k 2 + 1 k + 1 1 η x k+1 -x k , x -x k f (x k+1 ) -f (x ) ≤ k k + 1 (f (x k ) -f (x )) - 1 2η x k+1 -x k 2 + 1 k + 1 1 η x k+1 -x k , x -x k (k + 1)η (f (x k+1 ) -f (x )) ≤ kη (f (x k ) -f (x )) - k + 1 2 x k+1 -x k 2 + x k+1 -x k , x -x k . We note A k (k + 1)η (f (x k+1 ) -f (x )) -kη (f (x k ) -f (x )). It gives A k ≤ - k + 1 2 x k+1 -x k 2 + x k+1 -x k , x -x k ≤ k + 1 2 -x k+1 -x 2 -x k -x 2 + 2 x k+1 -x , x k -x + x k+1 -x , x -x k + x k -x 2 ≤ - k + 1 2 x k+1 -x 2 - k -1 2 x k -x 2 + k x k+1 -x , x k -x .
Thus we have

E(k + 1) = (k + 1)η (f (x k+1 ) -f (x )) + 1 2 x k+1 -x 2 ≤ kη (f (x k ) -f (x )) - k 2 x k+1 -x 2 - k 2 x k -x 2 + 1 2 x k -x 2 + k x k+1 -x , x k -x ≤ E(k) - k 2 x k+1 -x 2 + x k -x 2 -2 x k+1 -x , x k -x ≤ E(k) - k 2 x k+1 -x k 2 ≤ E(k) .
This shows that (E(k)) k≥0 is non-increasing and consequently E(k) ≤ E(0) = 1 2 x 0 -x 2 . This allows us state the following proposition, which is the discrete analogous of Proposition 1.

Proposition 2. Let (x k ) k∈N be given by (4) with f : R d → R convex and L-smooth. It holds that for all k ≥ 1,

f (x k ) -f (x ) ≤ L 2k x 0 -x 2 .
With this simple example we have demonstrated the interest of using the continuous counterpart of a discrete problem to gain intuition on a proof scheme for the original discrete problem. Note that the discrete proof is more involved than the continuous one, and that will always be the case in this manuscript. One reason is that we can compute the derivative of the energy function in the continuous case, whereas this is not possible in the discrete setting. In order to circumvent this we can use the descent lemma (Nesterov, 2004, Lemma 1.2.3) which can be seen as a discrete derivative, but at the price of additional terms and computations.

Following these ideas, [START_REF] Su | A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights[END_REF] have recently proposed a continuous model of the famous Nesterov accelerated gradient descent method [START_REF] Nesterov | A method for solving the convex programming problem with convergence rate O(1/k 2 )[END_REF]. Nesterov accelerated method is an improvement over the momentum method [START_REF] Polyak | Some methods of speeding up the convergence of iteration methods[END_REF] which was already an improvement over the standard gradient descent method, which actually goes back to [START_REF] Cauchy | Méthode générale pour la résolution des systemes d'équations simultanées[END_REF]. The idea behind the momentum method is to dampen oscillations by using a fraction of the past gradients into the update term. By doing that, the update uses an exponentially weighted average of all the past gradients and smooth the sequence of points since it will mainly keep the true direction of the gradient and discard the oscillations. However, even if momentum experimentally fastens gradient descent, it does not improve its theoretical convergence rate given by Proposition 2, contrarily to Nesterov's accelerated method, which can be stated as follows

   x k+1 = y k -η∇f (y k ) with η ≤ 1/L y k = x k + k -1 k + 2 (x k -x k-1 ) . ( 8
)
Nesterov's method still uses the idea of momentum but together with a lookahead computation of the gradient, which leads to an improved rate of convergence: Theorem 1. Let f be a convex and L-smooth function. Then Nesterov's accelerated gradient descent method satisfies for all k ≥ 1

f (x k ) -f (x ) ≤ 2L x 0 -x 2 k 2 .
This convergence rate which improves the one of Proposition 2 matches the lower bound of (Nesterov, 2004, Theorem 2.1.7), but the proof is not very intuitive, nor the ideas leading to scheme (8). The continuous scheme introduced by [START_REF] Su | A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights[END_REF] provides more intuition on the acceleration phenomenon by proposing to study the second-order differential equation

Ẍ(t) + 3 t Ẋ(t) + ∇f (X(t)) = 0, t ≥ 0 .
The authors prove the following convergence rate for the continuous model:

for all t > 0, f (X(t)) -f ≤ 2 X(0) -x 2 t 2 ,
again by introducing an appropriate energy function, which they choose to be in this case

E(t) = t 2 (f (X(t)) -f ) + 2 X(t) + t Ẋ(t)/2 -x 2
and which they prove to be non-increasing.

After having investigated the gradient descent algorithm and some of its variants, a natural line of research is to consider the stochastic case. One important use case of gradient descent is indeed machine learning, and more particularly deep learning, where variants of gradient descent are used to minimize the loss functions of neural networks and to learn the weights of these neurons. In deep learning applications, practitioners are usually interested in minimizing a function f of the form

f (x) = 1 N N i=1 f i (x) , ( 9
)
where f i is associated with the i-th observation of the training set (of size N , usually very large). Consequently computing the gradient of f is very costly since it requires to compute the N gradients ∇f i . In order to accelerate training one usually uses stochastic gradient descent by approximating the gradient of f by ∇f i with i chosen uniformly at random between 1 and N . A compromise between this choice and the standard classical gradient descent algorithm is to use "mini-batches" which are small sets of points in {1, . . . , N } to estimate the gradient:

∇f (x) ≈ 1 M M i=1 ∇f σ(i) (x) ,
where σ is a permutation of {1, . . . , N } and M is the size of mini-batch. Both of these choices provide approximations ĝ(x) of the true gradient ∇f (x), and since the points used to compute those approximations are chosen uniformly at random we have E [ĝ(x)] = ∇f (x). Using these stochastic approximations of ∇f (x) instead of the true gradient value in the gradient descent algorithm leads to the "Stochastic Gradient Descent algorithm" (SGD), which has a more general formulation than the one derived above. SGD can indeed be used to deal with the minimization problem (3) with noisy evaluations of ∇f for a wider class of functions than the ones of the form (9).

Obtaining convergence results for SGD is more challenging than for gradient descent, due to the stochastic uncertainties. In the case of SGD, the goal is to bound E [f (x k )]-f because the sequence (x k ) k≥0 is now stochastic. Convergence results in the case where f is strongly convex are well-known [START_REF] Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF][START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF] but convergence results in the convex case are not as common. Most of the convergence results in the convex case are indeed obtained for the Polyak-Ruppert averaging framework [START_REF] Polyak | Acceleration of Stochastic Approximation by Averaging[END_REF][START_REF] Ruppert | Efficient estimations from a slowly convergent Robbins-Monro process[END_REF] where instead of considering the last iterate x N , convergence rates are derived for the average xN defined as follows

xN = 1 N N k=1 x k .
Obtaining convergence rates in the case of averaging, as done by [START_REF] Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF], is easier than obtaining non-asymptotic convergence rates for the last iterate. Indeed if one is able to derive non-asymptotic rates for the last iterate, using Jensen inequality directly gives the convergence results in the averaged setting. Note moreover that all the algorithms presented in Section 2.3 do not consider the final iterate but rather some averaged version of the previous iterates. To the author's knowledge there is no general convergence results in the convex and smooth case for SGD. One of the only results for the last iterate is obtained by [START_REF] Shamir | Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes[END_REF] who assume compactness of the iterates, a strong assumption. Moreover [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF] conjectured that the optimal convergence rate of SGD in the convex case is O(k -1/3 ), which we disprove in Chapter 4.

Outline and contributions

This thesis is be divided into four chapters, each corresponding to one distinct problem.

Each of these chapters led to a publication or a pre-publication. We decided to group the first three chapters in a first part about sequential learning, while the last chapter will be the object of a second part, which is quite different, about stochastic optimization.

Chapter 3 can be seen as a link between both parts. We present in the following a summary of our main contributions and of the results obtained in the next chapters of this thesis. The goal of the following sections is to summarize our results, not to give exhaustive statements of all the hypotheses and theorems. We tried to keep this part easily readable and refer the reader to the corresponding chapters to obtain all the necessary details.

Part I Chapter 1

In this chapter we study the problem of stochastic contextual bandits with regularization, with a nonparametric point of view. More precisely, as introduced in Section 2.1, we consider a set of K ∈ N * arms with reward functions µ k : X → R corresponding to the conditional expectations of the rewards of each arm given the context values drawn uniformly at random from a set X = [0, 1] d . We assume that each of these functions is β-Hölder continuous and, denoting p : X → ∆ K the occupation measure of each arm we aim at minimizing the loss function

L(p) = X µ(x), p(x) + λ(x)ρ(p(x)) dx ,
where ∆ K is the unit simplex of R K , ρ : ∆ K → R is a convex regularization function (typically the entropy) and λ : X → R is a regularization parameter function. Both are supposed to be differentiable and chosen by the decision maker.

We denote by p the optimal proportion function

p = arg inf p∈{f :X →∆ K } L(p) ,
and we design in Chapter 1 an algorithm whose aim is to produce after T iterations a proportion function (or occupation measure) p T minimizing the regret

R(T ) = E [L(p T )] -L(p ) .
Since p T is actually the vector of the empirical frequencies of each arm, R(T ) has to be considered as a cumulative regret.

We analyze the proposed algorithm to obtain upper bounds on this regret under different assumptions. The algorithm we propose uses a binning of the context space and solves separately a convex optimization problem on each bin.

We begin by establishing slow rates for constant λ under mild assumptions. We call "slow rates" convergence results slower than O(T -1/2 ) (and conversely by "fast rates" convergence bounds faster than O(T -1/2 )).

Theorem 2. If λ is constant and ρ is a convex and smooth function we obtain the following slow bound on the regret after T ≥ 1 samples:

R(T ) ≤ O   T log(T ) -β 2β+d   .
If we further assume that ρ is strongly convex and that the minimum of the loss function on each bin is reached far from the boundaries of ∆ K , then we can obtain faster rates.

Theorem 3. If λ is constant and ρ is a strongly convex and smooth function and if L reaches its minimum far5 from ∂∆ K , we obtain the following fast bound on the regret after T ≥ 1 samples:

R(T ) ≤ O   T log(T ) 2 -2β 2β+d   .
However this fast rate hides a multiplicative constant involving 1/λ and 1/η (where η is the distance of the optimum to ∂∆ K ) which can be arbitrarily large. We consider therefore also the case where λ is a function of the context value, meaning that the agent can modulate the weight of the regularization depending on the context. In that case the distance of the optimum to the boundary will also depend on the context value and we define the function η as follows

η(x) := dist(p (x), ∂∆ K ) ,
where p (x) ∈ ∆ K is the point where (p → µ(x), p + λ(x)ρ(p)) reaches its minimum. In order to remove the dependency in λ and η in the bound of the regret, while achieving faster rates than the ones of Theorem 2, we have to consider an additional assumption limiting the possibility for λ and η to take small values (that lead to large constant factors in Theorem 3). This is classical in nonparametric estimation and we make therefore the following assumption known as a "margin condition":

Assumption 1. There exist δ 1 > 0, δ 2 > 0, α > 0 and C m > 0 such that ∀δ ∈ (0, δ 1 ], P X (λ(x) < δ) ≤ C m δ 6α and ∀δ ∈ (0, δ 2 ], P X (η(x) < δ) ≤ C m δ 6α .
This condition involves a margin parameter α that controls the difficulty of the problem and allows us to obtain intermediate convergence rates that interpolate perfectly between the slow and the fast rates, without any dependency in η or λ. Theorem 4. If ρ is a convex function then with a margin condition of parameter α ∈ (0, 1) we obtain the following rates for the regret after T ≥ 1 samples

R(T ) = O   T log 2 (T ) -β 2β+d (1+α)   .
We can wonder whether the convergence results obtained in the three theorems presented above are optimal or not. Note first that the convergence rates we obtain are classical in nonparametric estimation [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. Moreover we derive a lower bound on the considered problem showing that the fast upper bound of Theorem 3 is optimal up to the logarithmic terms.

Theorem 5. For any algorithm with bandit input and output pT , for ρ that is strongly convex and µ β-Hölder, there exists a universal constant C such that

inf p sup ρ,µ E[L(p T )] -L(p ) ≥ C T -2β 2β+d .
We conclude the chapter with numerical experiments on synthetic data to illustrate empirically our convergence results.

Part I Chapter 2

In this chapter we consider the problem of actively estimating a design matrix for linear regression, detailed in Section 2.2. Our goal is to obtain the most precise estimate of the parameter β of the linear regression i.e., to produce with T samples an estimate β which minimizes the

2 -norm E[ β -β 2 ]. If we introduce the matrix Ω(p) = K k=1 ( p k /σ 2 k )X k X k ,
for p ∈ ∆ K , our problem corresponds to minimizing the trace of its inverse (which is the covariance matrix), since

E β -β 2 = 1 T Tr(Ω(p) -1 ) .
This shows that our problem consists actually in performing A-optimal design in an online manner. More precisely we introduce the loss function L(p) = Tr(Ω(p) -1 ) which is strictly convex and which admits therefore a minimum p . Our goal is then to minimize the regret of the algorithm i.e., the gap between the achieved loss and the best loss that can be reached. We define therefore

R(T ) = E β -β 2 -min algo E β(algo) -β 2 = 1 T (E [L(p T )] -L(p )) .
Note that, similarly to Section 3.1, R(T ) is again not a simple regret but a cumulative one.

In Chapter 2 we construct an active learning algorithm building on the work [START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF] to solve the problem of online A-optimal design. We obtain a concentration result on the variances of subgaussian random variables and we use it to analyze our algorithm. Note that in the case where K < d, the matrix op is degenerate and hence the regret is linear, unless we restrict the analysis to the subspace spanned by the covariates. Therefore we consider from now on that K ≥ d.

We consider two cases in our analysis. The first one handles the case where the number K of possible covariates is equal to their dimension d. In this case we know that all the covariates have to be sampled. The control of the number of samples of each arm that can be sampled is crucial and our algorithm uses a well-designed pre-sampling phase to force the loss function to be locally smooth, which helps us to achieve a fast convergence result.

Theorem 6. In the case where K = d we obtain the following fast rate for all T ≥ 1

R(T ) = O log 2 (T ) T 2 .
We need to mention that this fast rate is hard to obtain. In Section 2.3 we propose indeed a naive algorithm for our problem using UCB-like techniques and we prove that it only achieves O(T -3/2 ) regret.

In the second case where K > d the problem is much more difficult. Different situations can arise and the optimal allocation p can be reached either by not sampling some covariate points, or by sampling all of them. Finding out which is the optimal scenario is a hard problem justifying the worse upper bound we obtain in this case Theorem 7. In the case where K > d we obtain the following upper-bound on the regret for all T ≥ 1

R(T ) = O log(T ) T 5/4 .
This upper bound is not tight as we were able to derive the following lower bound in the case K > d: Theorem 8. For any algorithm, there exists a set of parameters such that R(T ) T -3/2 .

The numerical experiments we perform at the end of Chapter 2 illustrate the fact that the case where K > d is more challenging and that the optimal convergence rate certainly lies between T -5/4 and T -3/2 .

Part I Chapter 3

In this chapter we study a problem that lies at the boundary between sequential learning and stochastic convex optimization. We consider the problem of resource allocation which we formulate as follows. A decision maker has access to a set of K different resources, on which he can allocate an amount x k , generating a reward f k (x k ). At each time step the agent can only allocate a fixed budget, meaning that K k=1 x k = 1. Consequently the decision maker receives at each time step t ∈ {1, . . . , T } the reward

F (x (t) ) = K k=1 f k (x (t) k ) with x (t) = (x (t) 1 , . . . , x (t) K ) ∈ ∆ K ,
that has to be maximized. Noting x ∈ ∆ K the optimal allocation maximizing F , the goal of the decision maker can be equivalently restated as minimizing the cumulative regret

R(T ) = F (x ) - 1 T T t=1 K k=1 f k (x (t) k ) = max x∈∆ K F (x) - 1 T T t=1 F (x (t) ) .
Resource allocation has been considered in many fields for centuries and we make therefore a classical assumption that goes back to [START_REF] Smith | An Inquiry into the Nature and Causes of the Wealth of Nations[END_REF] which is known as the "diminishing returns" assumption that postulates that the reward functions are concave. In this chapter we assume that the decision maker has also access at each time step to a noisy value of ∇F (x (t) ) in order to perform minimization, which makes us compete against other firstorder stochastic optimization algorithms.

In order to measure the complexity of the problem at hand we make an additional assumption that is based on the Łojasiewicz inequality (Łojasiewicz, 1965), which is a weaker form of uniform convexity. The precise assumption is explained in details in Section 3.2.3 but we state here a particular case for simplicity

Assumption 2. For all k ∈ {1, • • • , K}, f k is ρ-uniformly concave.
With this assumption we say that we verify "inductively" the Łojasiewicz inequality with parameter β = ρ ρ-1 , see Proposition 3.5. The goal of Chapter 3 is to design an algorithm adaptive to the unknown Łojasiewicz exponent and which minimizes the regret. Going back to the discussion of Section 2.3 we are interested in the more challenging task of regret minimization instead of function error minimization, which actually rules out the algorithms proposed by [START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] or Ramdas and Singh (2013a), which only achieve linear regret.

The algorithm we design uses as its central ingredient the concept of binary search. Let us sketch it in the simpler case of K = 2 resources. In that case

F (x) = f 1 (x 1 ) + f 2 (x 2 ) = f 1 (x 1 ) + f 2 (1 -x 1 ) f 1 (x) + f 2 (1 -x)
can be seen as a function defined over [0,1]. The idea of the algorithm is to sample each query point x a sufficient number of times to obtain with high confidence the sign of ∇F (x), which will tell whether x lies to the right or to the left of x . We run therefore a binary search by discarding half of the search interval at each epoch. Since points that are far from x will be sampled a small number of times, because the sign of their gradient will be quickly found, this algorithm achieves sublinear regret. It is easy to show that our algorithm achieves O(T -1 ) regret in the strongly concave case, reaching therefore the classical rate of stochastic optimization of strongly convex functions. In the more general case we obtain the following rate, using imbricated binary searches.

Theorem 9. Assume that our problem satisfies inductively the Łojasiewicz inequality with β ≥ 1. Then we obtain the following bound on the regret after T ≥ 1 samples

in the case β > 2, E[R(T )] ≤ O K log(T ) log 2 (K) T ; in the case β ≤ 2, E[R(T )] ≤ O   K log(T ) log 2 (K)+1 T β/2   .
Note that in the case of Assumption 2, β = ρ/(ρ -1) ≤ 2 and we obtain a bound on the regret which scales as T -ρ/(2ρ-2) , which is exactly what was obtained by Ramdas and Singh (2013a,b) and [START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF], but this time for the regret and not for the function error. As in the previous chapters we also analyze the optimality of the upper bound obtained in the previous theorem. We prove the following lower bound in the case where β ∈ [1, 2] Theorem 10. For any algorithm there exists a pair of concave non-decreasing functions f 1 and f 2 such that

E [R(T )] ≥ c β T -β 2 ,
where c β > 0 is some constant independent of T .

This result proves that our upper bound is minimax optimal up to the logarithmic terms. We finally illustrate these theoretical findings with numerical experiments performed on synthetic datasets.

Moreover we also demonstrate how our setting can generalize the case of Multi-Armed Bandit by considering linear resources. In Section 3.3.5 we retrieve the classical log(T )/(T ∆) rate of Multi-Armed Bandit algorithms.

Part II Chapter 4

In this chapter we analyze the widely-used algorithm of Stochastic Gradient Descent (SGD) that was discussed in Section 2.4. Let f : R d → R the objective function to minimize. We will assume that f is continuously differentiable and smooth, and that we do not have access to ∇f (x) but rather to unbiased estimates given by H(x, z) where z is a realization of a random variable Z on Z of density µ Z verifying

∀x ∈ R d , Z H(x, z)dµ Z (z) = ∇f (x) .
We then define SGD as follows

X n+1 = X n -γ(n + 1) -α H(X n , Z n+1 ) , ( 10
)
where γ > 0 is the initial stepsize, α ∈ [0, 1] allows the use of decreasing stepsizes and (Z n ) n∈N is a sequence of independent random variables distributed from µ Z . As explained in Section 2.4 we want to study SGD by performing the analysis of its continuous counterpart that we show to be the following time-inhomogeneous Stochastic Differential Equation (SDE)

dX t = -(γ α + t) -α {∇f (X t )dt + γ 1/2 α Σ(X t ) 1/2 dB t } , ( 11
)
where

γ α = γ 1/(1-α) , Σ(x) = µ Z ({H(x, •) -∇f (x)}{H(x, •) -∇f (x)} ) and (B t ) t≥0 is a d-dimensional Brownian motion.
One of the contributions of Chapter 4 is to propose a new method to derive the convergence rates of SGD, by analyzing the corresponding SDE. We argue that this method is simpler than existing ones. We demonstrate first its efficiency on the case of strongly convex functions. The method we propose consists in using an appropriate energy function to obtain convergence results in the continuous case, and then to adapt the proof to the discrete case by using similar techniques. The continuous case gives therefore intuition. For example we are able to prove the following result in the strongly convex case Theorem 11. If f is a strongly convex and smooth function then the SGD scheme (10) with decreasing stepsizes of parameter α ∈ (0, 1] has the following convergence speed for any N ≥ 1,

E X N -x 2 ≤ CN -α .
Even if this theorem is well-known, the proof we propose is simpler than the one of [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF]. In order to prove our results in the continuous case, we use Dynkin's lemma (which consists essentially in taking the expectation in Itô's lemma, see Lemma 4.13) in order to compute the derivative of the energy function. In the discrete case, we replace Dynkin's lemma by the descent lemma (Nesterov, 2004, Lemma 1.2.3) which is an approximate discrete counterpart of Dynkin's lemma, but without a secondorder derivative term, which will lead to some differences in the proofs.

The main contribution of Chapter 4 is a complete analysis of SGD in the convex setting, for the function value error of the last iterate. We consider the case where f is convex and smooth and we do not make any compactness assumption. We prove the following two results thanks to similar proofs. The first one concerns the convergence rate of the SDE (11).

Theorem 12. If f is a smooth and convex function there exists C ≥ 0 such that the sequence (X t ) t≥0 given by the SDE (11) with α ∈ (0, 1) verifies for any T ≥ 1,

E [f (X T )] -f ≤ C(1 + log(T )) 2 /T α∧(1-α) .
We derive a second similar result in the discrete case. The proof is a bit more involved, since the correspondence between Dynkin's lemma and the descent lemma is not perfect. We nevertheless obtain the following result, whose resemblance with Theorem 12 emphasizes the links between discrete and continuous models.

Theorem 13. If f is a convex and smooth function there exists C ≥ 0 such that the sequence of SGD (10) defined for α ∈ (0, 1) verifies for any N ≥ 1,

E [f (X N )] -f ≤ C(1 + log(N + 1)) 2 /(N + 1) α∧(1-α) .
This result disproves the conjecture of [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF] who postulated that the optimal rate for the last point iterate in SGD was N -1/3 .

Finally we study a relaxation of the convexity assumption. We consider a generalization of the "weakly quasi-convex" setting [START_REF] Hardt | Gradient Descent Learns Linear Dynamical Systems[END_REF] by assuming that there exist r 1 ∈ (0, 2), r 2 ≥ 0, τ > 0 such that for all x ∈ R d ,

f (x) -f (x ) ≤ ∇f (x) r 1 x -x r 2 /τ .
This condition embeds also the Łojasiewicz inequality mentioned in Section 3.3 which can be defined as follows, for β ∈ (0, 2) and c > 0,

∀x ∈ R d , f (x) -f (x ) ≤ c ∇f (x) β ,
which has been widely used in optimization.

In this setting we are also able to derive convergence rates for both the SDE (11) and the discrete SGD scheme (10). Our results, which are precisely stated in Section 4.3.4 generalize and outperform the results obtained in the weakly quasi-convex case by [START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF].

List of publications

This thesis has led to the following publications:

• (Fontaine et al., 2019a) In addition, the author also participated in the following publication that is not discussed in the present thesis: In what follows we made the choice to postpone the long proofs to the end of each chapter in order to improve readability.

Introduction en français 1 Motivations

Les problèmes d'optimisation sont très fréquents aujourd'hui et peuvent servir par exemple à utiliser au mieux notre temps, à minimiser la durée d'un trajet, ou à maximiser le gain d'un produit financier avec des contraintes de risque. Les problèmes d'optimisation avec ou sans contraintes sont présents dans de nombreux domaines des mathématiques comme la théorie du contrôle, la recherche opérationnelle, la finance, le transport optimal ou l'apprentissage automatique. Nous nous intéresserons principalement dans cette thèse aux problèmes d'optimisation qui apparaissent en apprentissage automatique. Malgré ses domaines nombreux et variés d'application, comme le traitement automatique du langage, l'analyse d'image, la publicité ciblée, etc., tous les algorithmes d'apprentissage reposent en effet sur le concept d'optimisation, et plus précisément sur l'optimisation stochastique. Les algorithmes d'apprentissage automatique sont généralement analysés à travers le formalisme de l'apprentissage statistique, dont le but est d'estimer (ou d'apprendre) la meilleure fonction de prédiction sur une tâche précise à l'aide de données, c'est-à-dire de trouver la fonction la plus probable qui corresponde aux données. Pour ce faire on utilise souvent des techniques d'optimisation, par exemple pour minimiser une fonction de perte, pour trouver les bons hyperparamètres ou pour maximiser un gain moyen.

Dans cette thèse nous nous concentrons sur l'étude d'une classe spécifique de problèmes d'apprentissage statistique où les données sont obtenues et traitées à la volée, et qui est connue sous le nom d'apprentissage séquentiel, ou apprentissage en ligne [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF], en opposition à l'apprentissage hors-ligne où les données ont été récupérées avant l'apprentissage. La principale difficulté de l'apprentissage séquentiel est précisément le fait que l'agent que l'on considère doit construire une fonction de prédiction sans connaître toutes les données. C'est pour cela que les algorithmes en ligne sont généralement moins performants que les algorithmes hors-ligne pour lesquels l'agent a accès à l'ensemble des données. Les situations d'apprentissage en ligne peuvent cependant avoir aussi des avantages quand l'agent joue un rôle actif dans le processus de collection des données. Dans ce domaine de l'apprentissage automatique, que l'on appelle généralement apprentissage actif [START_REF] Settles | Active learning literature survey[END_REF], l'agent est capable de choisir quelles données collecter et quelles données labelliser. Faire partie intégrante du processus de sélection des données peut améliorer la performance de l'algorithme d'apprentissage puisque l'agent choisira les données qui apporteront le plus d'information. Dans les problèmes d'apprentissage séquentiel on considère donc un agent qui doit prendre des décisions à chaque pas de temps, sachant que les actions qu'il aura choisies pourront avoir une influence sur la suite du processus d'apprentissage. Ainsi dans les problèmes de bandits [START_REF] Bubeck | Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems[END_REF], qui sont une façon simple de modéliser les prises de décision avec incertitude, l'agent doit sélectionner une action (que l'on appelle généralement "bras") parmi plusieurs dans le but de maximiser son gain. Pour atteindre son but l'agent doit donc faire des choix, par exemple sélectionner le meilleur bras actuel, ou bien en choisir un autre afin d'explorer les différentes options à sa disposition et ainsi obtenir des informations sur celles-ci. Ce compromis entre l'exploitation des données et leur exploration est une des principales difficultés des problèmes liés aux bandits. Dans les trois premiers chapitres de cette thèse nous allons étudier des problèmes d'apprentissage séquentiel ou actif où ce genre de compromis est très présent. Notre but sera toujours de minimiser une quantité, que l'on nomme "regret" et qui quantifie la différence entre la meilleure stratégie qu'aurait choisie un agent omniscient et la stratégie effectivement adoptée.

Les problèmes d'optimisation que l'on traite en apprentissage automatique ont généralement la particularité de concerner des fonctions qui sont inconnues ou bruitées. Par exemple, dans le cas classique des bandits stochastiques [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF][START_REF] Auer | Finite-time Analysis of the Multiarmed Bandit Problem[END_REF] on veut maximiser une récompense qui dépend des distributions de probabilité des bras, qui sont inconnues. Pour en savoir davantage sur ces distributions l'agent reçoit à chaque pas de temps un retour d'information, ou "feedback" (typiquement la récompense du bras choisi) qui sera utilisé pour prendre les décisions suivantes. Dans les problèmes de bandits, on parle d'information "limitée" (ou de type bandit), en opposition à l'information "complète" obtenue quand les récompenses de tous les bras (et pas seulement de celui qui a été choisi) sont dévoilées à l'agent. En outre, la difficulté des problèmes de bandits n'est pas uniquement due à l'information limitée mais aussi à son caractère bruité : les récompenses des bras sont effet des valeurs bruitées des moyennes de chaque bras. C'est aussi le cas pour l'algorithme de Descente de Gradient Stochastique (SGD en anglais) [START_REF] Robbins | A stochastic approximation method[END_REF] qui sert à minimiser une fonction différentiable au moyen des valeurs bruitées de son gradient. Du fait de l'aléatoire inhérent aux problèmes d'apprentissage on utilise donc en apprentissage automatique des méthodes d'optimisation stochastique, qui consistent en l'optimisation de fonctions dont les valeurs dépendent de variables aléatoires. Puisque l'on travaille avec des fonctions aléatoires, les résultats que nous obtiendront seront donc généralement énoncés en espérance ou avec grande probabilité.

Une des caractéristiques principales d'un algorithme d'optimisation (en plus de réaliser effectivement sa tâche de minimisation) est la vitesse à laquelle il atteint le minimum, ou bien la précision qu'il peut garantir après un certain nombre d'itérations, ou avec un budget fixé. Par exemple l'objectif des algorithmes de bandits est d'obtenir un regret sous-linéaire en T (l'horizon de temps de l'algorithme), et l'objectif de l'algorithme SGD est de borner E[f (x n )] -min x∈R d f en fonction du nombre d'itérations n. Il faut en effet que les méthodes d'apprentissage automatique soient efficaces et précises, c'est-à-dire que les algorithmes d'optimisation qu'elles utilisent doivent converger rapidement. Obtenir des vitesses de convergence pour les algorithmes que l'on étudie ici sera l'un des objectifs théoriques majeurs de cette thèse. En outre, après avoir obtenu une borne de convergence il faut se demander si cette vitesse peut être améliorée, soit grâce à une analyse plus précise de l'algorithme, soit par un meilleur algorithme pour le problème que l'on considère. Il existe deux réponses à cette question. La première réponse est évidente et consiste à comparer les performances obtenues avec celles d'autres méthodes de la littérature. La deuxième réponse consiste à trouver une "borne inférieure" pour le problème, c'est-à-dire une vitesse de convergence qui ne peut pas être battue. Si cette borne inférieure coïncide avec la vitesse de convergence de l'algorithme (appelée aussi "borne supérieure") on dit que l'algorithme est "min-max optimal", c'est-à-dire que l'on ne peut pas faire mieux. Dans ce manuscrit nous nous efforcerons dès que possible de comparer nos résultats avec l'état de l'art et d'établir des bornes inférieures, afin de pouvoir avoir un avis sur la pertinence de nos algorithmes.

Pour obtenir des vitesses de convergence en optimisation il est important de s'intéresser à la complexité du problème considéré. Plus le problème est complexe (ou moins il est précis), plus les algorithmes seront lents. Par exemple, il est bien plus compliqué de minimiser une fonction arbitraire sur R d que de minimiser une fonction différentiable et fortement convexe. Dans cette thèse on caractérisera la complexité d'un problème par des mesures de complexité des fonctions que l'on considère : plus les fonctions seront régulières, plus le problème sera facile. C'est pourquoi chaque chapitre débutera par un ensemble d'hypothèses qui permettront à la fois de trouver une solution au problème et d'obtenir des vitesses de convergence. Nous verrons comment un changement de ces hypothèses modifiera les vitesses convergence obtenues. Par exemple dans les Chapitres 3 et 4 nous obtiendrons des vitesses de convergence pour des algorithmes d'optimisation stochastique qui dépendront de l'exposant dans une inégalité de Łojasiewicz (Łojasiewicz, 1965;[START_REF] Karimi | Linear convergence of gradient and proximal-gradient methods under the Polyak-Łojasiewicz condition[END_REF]. Nous verrons que des variations de cet exposant font augmenter ou baisser la complexité du problème, et donc les vitesses de convergence associées. Une difficulté que nous rencontrerons dans ce manuscrit est due au fait que les problèmes que l'on étudie ne vérifient pas toujours ce genre d'hypothèses de régularité. Les problèmes réels ou les applications pratiques ne concernent en effet pas toujours des fonctions lisses ou convexes. Ainsi, les algorithmes d'optimisation stochastique tels que SGD ont souvent des garanties de convergence [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF] dans le cas convexe (ou même fortement convexe), alors que très peu de résultats existent dans le cas non convexe, qui demeure néanmoins la situation la plus fréquente, par exemple en apprentissage profond. Un des défis de cette thèse sera donc de traiter ces cas. Notons par ailleurs que la performance réelle d'un algorithme d'optimisation peut être bien meilleure que sa vitesse théorique. C'est typiquement le cas des algorithmes d'optimisation stochastique mentionnés plus haut qui sont beaucoup utilisés dans les réseaux de neurones sans avoir pour autant de garanties théoriques dans ce cas. Afin de pouvoir comparer les performances théoriques et pratiques de nos algorithmes nous présenterons dans cette thèse des simulations numériques de nos méthodes.

Dans la suite de ce chapitre introductif nous présenterons les différents problèmes d'apprentissage séquentiel et d'optimisation que nous avons étudiés au cours de cette thèse, ainsi que les principaux outils mathématiques dont nous aurons besoin. Nous conclurons par des explications détaillées chapitre par chapitre des différentes contributions de cette thèse ainsi que par la liste des publications réalisées.

Présentation des problèmes étudiés 2.1 Bandits stochastiques contextuels (Chapitre 1)

Considérons un agent qui a accès à K ∈ N * bras qui sont chacun associés à une distribution de probabilité ν i , pour tout i ∈ {1, . . . , K}. Supposons qu'à chaque pas de temps t ∈ {1, . . . , T },1 l'agent peut choisir un bras i t ∈ {1, . . . , K} et qu'il reçoit une récompense Y (it) t qui suit la distribution ν it , loi de probabilité sur R + d'espérance µ it . L'agent a comme objectif de maximiser sa récompense totale T t=1 Y

(it) t

. Puisque les récompenses sont aléatoires, nous allons plutôt tâcher de maximiser l'espérance de la récompense totale E T t=1 µ it , où l'espérance porte sur l'aléa des décisions de l'agent.

Ainsi nous sommes généralement intéressés par minimiser le regret (ou plus précisément le "pseudo-regret")

R(T ) = T max 1≤i≤K µ i -E T t=1 µ it .
(1)

C'est la formulation classique du problème de "bandits stochastiques multi-bras" [START_REF] Bubeck | Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems[END_REF] qui peut être résolu en utilisant l'algorithme bien connu UCB (Upper Confidence Bound) qui est dû à [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF].

Ce problème peut servir à modéliser de nombreuses situations où un compromis de type "exploration vs. exploitation" apparaît. C'est par exemple le cas des essais cliniques ou bien de la publicité en ligne où l'on veut trouver la meilleure publicité à afficher tout en maximisant le nombre de clics. Cependant, le modèle décrit ci-dessus semble trop simpliste pour proposer une solution adéquate aux problèmes mentionnés ci-dessus. En effet tous les patients, ou tous les internautes, ne se comportent pas de la même façon, et une publicité peut être appropriée pour quelqu'un et ne pas du tout être adaptée pour quelqu'un d'autre. Nous comprenons donc que le modèle évoqué plus haut est trop restrictif, et nous voyons en particulier que l'hypothèse que l'on a faite que chaque bras i a une espérance fixe µ i n'est pas réaliste. C'est pour cela que nous devons introduire ce que l'on va appeler un ensemble de contextes X = [0, 1] d qui correspond aux différents profils possibles de patients ou d'internautes de notre problème. Chaque contexte x ∈ X donne les caractéristiques d'un utilisateur et nous allons donc maintenant supposer que les récompenses des K bras dépendent du contexte x. Ce problème est connu sous le nom de bandits avec information [START_REF] Wang | Bandit problems with side observations[END_REF] ou de bandits contextuels [START_REF] Langford | The epoch-greedy algorithm for multi-armed bandits with side information[END_REF] et modélise mieux les problèmes d'essais cliniques ou bien de publicité ciblée. Nous supposerons donc maintenant qu'à chaque pas de temps t ∈ {1, . . . , T } l'agent observe un contexte aléatoire X t ∈ X et doit choisir un bras i t dont la récompense Y (it) dépendra du contexte X t . Notons donc pour chaque bras i ∈ {1, . . . , K}, µ i : X → R l'espérance conditionnelle de la récompense du bras i par rapport à la variable de contexte X, qui est maintenant une fonction du contexte x :

E[Y (i) |X = x] = µ i (x), pour tout x ∈ X .
Afin de tirer profit des contextes nous devons faire quelques hypothèses de régularité sur les récompenses. Nous voulons en effet nous assurer qu'un bras donnera des récompenses similaires pour deux variables de contexte proches (c'est-à-dire deux utilisateurs qui ont un profil semblable). Une façon de modéliser cette hypothèse naturelle est par exemple de supposer que les fonctions µ i sont Lipschitz. Ce problème non paramétrique de bandits contextuels stochastiques a été étudié par [START_REF] Rigollet | Nonparametric Bandits with Covariates[END_REF] dans le cas de K = 2 bras et ensuite par [START_REF] Perchet | The multi-armed bandit problem with covariates[END_REF] dans le cas général. Dans ces deux travaux, le but de l'agent est de trouver une stratégie π : X → {1, . . . , K} qui associe à un contexte un bras à tirer. Bien évidemment, comme dans le cas des bandits stochastiques classiques, l'action choisie dépendra de l'historique des précédents tirages, et la dépendance de π en temps est implicite. Nous pouvons maintenant définir la stratégie optimale π ainsi que la fonction de récompense optimale µ :

π (x) ∈ arg max i∈{1,...,K} µ i (x) et µ (x) = max i∈{1,...,K} µ i (x) .
Cela donne donc l'expression suivante pour le regret après T tirages

R(T ) = T t=1 E µ (X t ) -µ π(Xt) (X t ) .
(2)

Même si les expressions (2) et (1) sont similaires, l'une des difficultés que l'on va rencontrer pour minimiser le regret est due au fait que l'on ne peut pas espérer obtenir plusieurs récompenses d'un bras pour une même valeur de contexte, puisque l'espace des contextes est indénombrable. Une idée courante en statistiques non paramétriques [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF] pour estimer une fonction inconnue f sur X est d'utiliser des "régressogrammes", qui sont des estimateurs constants par morceaux de la fonction. Leur construction est similaire à celle d'histogrammes, en partitionnant X en différents sous-ensembles et en approximant f par sa valeur moyenne sur chacun des sous-ensembles de la partition. Les régressogrammes sont une alternative aux estimateurs de Nadaraya-Watson [START_REF] Nadaraya | On estimating regression[END_REF][START_REF] Watson | Smooth regression analysis[END_REF] qui eux utilisent des noyaux en guise de poids au lieu d'utiliser une partition de l'espace.

Une façon de résoudre le problème de bandits stochastiques contextuels consiste à s'inspirer des régressogrammes et à partitionner l'espace de contextes X en différents sous-ensembles et à traiter le problème de bandits contextuels en différentes instances indépendantes d'un problème de bandits sans contexte sur chacun des sous-ensembles de la partition. Cela peut-être réalisé au moyen d'un algorithme classique de bandits comme UCB ou ETC [START_REF] Even-Dar | Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems[END_REF] déployé indépendamment sur chaque sous-ensemble, ce qui donne lieu à une stratégie appelée "UCBogramme" [START_REF] Rigollet | Nonparametric Bandits with Covariates[END_REF]. Une telle stratégie n'est bien sûr possible que grâce à l'hypothèse de régularité que l'on a faite précédemment, et qui garantit que considérer une approximation constante des fonctions µ i ne crée par une erreur trop importante.

Au lieu de supposer que les fonctions µ i sont Lipschitz, [START_REF] Perchet | The multi-armed bandit problem with covariates[END_REF] font une hypothèse plus faible et très classique en estimation non paramétrique qui consiste à supposer que ces fonctions sont β-Hölder pour β ∈ (0, 1], c'est-à-dire que pour tout i ∈ {1, . . . , K} et pour tout (x, y) ∈ X 2 ,

|µ i (x) -µ i (y)| ≤ L x -y β .
Avec cette hypothèse ils obtiennent la borne classique suivante sur le regret R(T ) (où l'on a uniquement fait figurer la dépendance en T et pas celle en K) R(T ) T 1-β/(2β+d) .

Maintenant que l'on a obtenu une solution pour le problème de bandits stochastiques contextuels nous pouvons nous demander s'il est toujours réaliste. Prenons en effet à nouveau l'exemple de la publicité en ligne. Supposons donc qu'une société de publicité ciblée souhaite utiliser des bandits contextuels pour définir sa stratégie. L'entreprise utilisait d'autres techniques précédemment et ne veut pas risquer de perdre trop d'argent en mettant en place sa nouvelle stratégie. Cette situation est une instance d'un problème bien plus vaste que l'on appelle apprentissage par renforcement sécurisé [START_REF] García | A comprehensive survey on safe reinforcement learning[END_REF] et qui étudie les politiques d'apprentissage qui doivent respecter certaines contraintes de sécurité. Dans le cas spécifique des bandits, [START_REF] Wu | Conservative bandits[END_REF] ont proposé un algorithme qu'ils ont appelé "UCB conservatif" qui consiste à faire tourner UCB tout en garantissant uniformément dans le temps que la récompense obtenue est plus grande qu'une portion 1 -α de ce qui aurait été obtenu par la stratégie précédente. Pour obtenir ce résultat les auteurs ajoutent un bras supplémentaire correspondant à l'ancienne stratégie qu'ils tirent dès que la contrainte sur la récompense risque de ne plus être vérifiée. Dans le Chapitre 1 nous adoptons un autre point de vue sur ce problème : au lieu d'imposer une contrainte sur la récompense nous ajoutons un terme de régularisation pour forcer la nouvelle stratégie à être proche d'une stratégie déterminée à l'avance.

Dans les problèmes de bandits l'agent doit choisir des actions pour maximiser une récompense mais il n'est généralement pas intéressé par les valeurs précises de chacun des bras. Estimer ces valeurs est un autre problème qui présente aussi un certain intérêt. En revanche les deux tâches d'estimation des moyennes des bras et de maximisation de la récompense totale ne sont pas compatibles puisque la tâche d'estimation nécessite aussi d'échantillonner les bras sous-optimaux. Dans la section suivante nous nous intéresserons à une généralisation du problème d'estimation qui consiste à choisir intelligemment quel bras tirer pour maximiser sa connaissance sur un paramètre inconnu (et qui peut typiquement être le vecteur des moyennes de chaque bras).

De la régression linéaire à la planification en ligne d'expériences de façon optimale (Chapitre 2)

Considérons maintenant le problème de régression linéaire qui a déjà été très étudié en apprentissage. Dans ce problème un agent a accès à un ensemble de données et de labels {(x i , y i )} i=1,...,n de n observations, où (x i , y i ) ∈ R p × R pour tout i ∈ {1, . . . , n}. On suppose que ces points sont linéairement reliés : ∀i ∈ {1, . . . , n} , y i = x i β + ε i , où β ∈ R p est le vecteur des paramètres2 et ε = (ε 1 , . . . , ε n ) est le vecteur de bruits qui modélise le terme d'erreur de la régression. Dans la suite de cette section nous supposerons que le bruit est centré, c'est-à-dire que E [ε] = 0 et qu'il a une variance finie :

∀i ∈ {1, . . . , n} , E ε 2 i = σ 2 i < +∞ .
Nous considérons tout d'abord le cas homoscédastique, c'est-à-dire que les variances σ 2 i sont toutes supposées égales à σ 2 pour chaque bras i ∈ {1, . . . , n}. En régression linéaire on considère généralement la "matrice de design" X ainsi que le vecteur des observations Y définis ainsi

X =    • • • x 1 • • • . . . • • • x n • • •    ∈ R n×p et Y =    y 1 . . . y n    ∈ R n , ce qui donne Y = Xβ + ε .
L'objectif d'une régression linéaire est d'estimer le paramètre β par un β ∈ R p afin de minimiser l'erreur des moindres carrés L(β) entre les vraies valeurs observées y i et les valeurs prédites par le modèle linéaire X i β :

L(β) = n i=1 (y i -x i β) 2 = Y -Xβ 2 2 .
Nous définissons donc l'estimateur optimal de β comme β arg min β∈R p L(β) et nous obtenons aisément la formule bien connue des moindres carrés β = (X X) -1 X Y , ce qui donne la relation suivante entre β et β :

β = β + (X X) -1 X ε .
Nous pouvons donc définir la matrice de covariance de l'erreur d'estimation β -β

Ω E (β -β)(β -β) = σ 2 (X X) -1 = σ 2 n i=1 x i x i -1
, qui caractérise la précision de l'estimateur β.

Comme nous l'avons montré ci-dessus, la régression linéaire est un problème simple et bien compris aujourd'hui. Il peut toutefois être la brique de base de plusieurs problèmes plus complexes et plus intéressants. Supposons par exemple que les vecteurs x 1 , . . . , x n ne sont plus fixes, mais qu'ils peuvent être choisis parmi un ensemble de points de taille K > 0 {X 1 , . . . , X K }. L'agent doit maintenant choisir chacun des points x i parmi les X k (avec la possibilité de choisir plusieurs fois un même point X k ). Ce problème présente un intérêt quand l'on peut réaliser différentes expériences (qui correspondent aux points X k ) pour estimer un vecteur inconnu β . Le but de l'agent est donc de choisir de façon adéquate les expériences à réaliser pour minimiser la matrice de covariance Ω de l'erreur d'estimation. Si on note n k le nombre de fois que le vecteur X k a été choisi, on peut écrire Ω sous la forme suivante

Ω = σ 2 n k=1 n k X k X k -1 .
Ce problème, tel qu'il a été formulé ci-dessus, a été étudié sous le nom de "conception optimale d'expériences" [START_REF] Boyd | Convex optimization[END_REF][START_REF] Pukelsheim | Optimal Design of Experiments[END_REF]. Nous pouvons toutefois remarquer que le problème de minimisation de Ω est mal posé. En effet il n'existe pas de relation d'ordre total sur le cône des matrices symétriques positives. C'est pourquoi plusieurs critères ont été proposés [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF], dont les plus utilisés sont le critère D qui minimise det(Ω), le critère E qui minimise Ω 2 , ainsi que le critère A dont le but est de minimiser Tr(Ω). Tous ces problèmes de minimisation ont lieu sous la contrainte que K k=1 n k = n. Ce sont tous des problèmes convexes pour lesquels il est donc facile de trouver une solution, pourvu que l'on relâche la contrainte qui force les n k à être entiers.

Omettons maintenant l'hypothèse d'homoscédasticité et considérons la situation hétéroscédastique plus générale suivante, où l'on ne suppose plus que les variances des points X k sont égales. La matrice de covariance Ω se récrit donc

Ω = n k=1 n k σ 2 k X k X k -1 .
Remarquons que la situation hétéroscédastique correspond en fait au cas homoscédastique en faisant subir aux points X k une homothétie de facteur 1/σ k . Ainsi l'analyse effectuée plus haut continue à s'appliquer dans ce cas général. En revanche la situation devient complètement différente si l'on ne connaît pas les valeurs de σ k . En effet il faut commencer par estimer les variances pour pouvoir minimiser 3 Ω dans ce cas-là. Cependant la tâche n'est pas aisée puisque l'on risque d'augmenter la valeur de Ω si l'on utilise trop de points pour estimer certaines variances σ k . Nous faisons donc à nouveau face à un compromis de type "exploration vs. exploitation". Cette situation correspond maintenant à la conception optimale d'expériences en ligne, puisque l'agent doit construire de façon séquentielle le meilleur plan d'expériences en prenant en compte les résultats obtenus lors des expériences précédentes. Ce problème se rapproche donc de l'"apprentissage actif" dans lequel l'agent peut choisir quel point labelliser. Comme l'expliquent [START_REF] Willett | Faster rates in regression via active learning[END_REF] il y a deux types d'apprentissage actif : l'échantillonnage sélectif dans lequel l'agent peut choisir de labelliser ou non les données qui lui sont présentées, et l'échantillonnage adaptatif où l'agent choisit quelles expériences réaliser en fonction du résultat des expériences passées. La situation que nous avons décrite plus haut correspond au cas de l'échantillonnage adaptatif appliqué au problème de la régression linéaire. Utiliser un algorithme d'apprentissage actif peut améliorer les performances des algorithmes par rapport à l'apprentissage hors-ligne. En effet certains points peuvent avoir des variances élevées et il est donc nécessaire de faire un grand nombre d'expériences sur ce point pour obtenir une réponse précise. On doit donc pouvoir améliorer la précision de l'estimateur en utilisant des techniques d'apprentissage actif pour la régression linéaire.

Considérons maintenant le cas plus simple où p = K et où les points X k sont les vecteurs e 1 , . . . , e K de la base canonique de R K . Si nous notons µ = β nous voyons que X k β = e k µ = µ k et nous pouvons identifier ce problème avec celui des bandits avec K bras de moyennes µ 1 , . . . , µ K . L'objectif est désormais d'obtenir des estimateurs μ1 , . . . , μK des moyennes µ 1 , . . . , µ K des bras. Ce problème a été étudié par [START_REF] Antos | Active learning in heteroscedastic noise[END_REF][START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF] avec comme objectif la minimisation de

max 1≤k≤K E (µ k -μk ) 2 ,
qui correspond à estimer avec la même précision les moyennes de chacun des bras. Il peut être intéressant de minimiser un autre critère que cette norme ∞ , et par exemple nous pouvons considérer la norme 2 des erreurs d'estimation

K k=1 E (µ k -μk ) 2 = E K k=1 (β k -βk ) 2 = E β -β 2 2 .
Notons que ce problème est très relié à celui de la planification optimale d'expériences dont nous avons parlé plus haut puisque E[ β -β 2 2 ] = Tr(Ω). Ainsi donc, minimiser la norme 2 des erreurs d'estimation dans un problème de bandits multi-bras correspond à résoudre en ligne un problème de planification optimale d'expériences avec le critère A. Les solutions proposées par [START_REF] Antos | Active learning in heteroscedastic noise[END_REF][START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF] peuvent être adaptées à la norme 2 , et utilisent des techniques classiques de la littérature de bandits pour gérer le compromis exploration vs. exploitation. [START_REF] Antos | Active learning in heteroscedastic noise[END_REF] utilisent un algorithme glouton qui sélectionne le bras k qui maximise l'estimation courante de E (µ k -μk ) 2 tout en forçant à sélectionner les bras qui ont été choisis moins souvent que α √ n fois, où α > 0 est un paramètre bien choisi. Les étapes d'échantillonnage forcé garantissent que les options qui ont pu être sous-estimées soient quand même explorées. La stratégie proposée par [START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF] est similaire puisque les auteurs choisissent de tirer le bras qui minimise la quantité σ2 k /n k (qui est une estimation de E (µ k -μk ) 2 ) corrigée par un terme d'exploration similaire à celui d'UCB. Les deux méthodes obtiennent des regrets au même comportement asymptotique en O(n -3/2 ) mais s'appuient sur le fait que les covariables X 1 , . . . , X K forment la base canonique de R K . Il nous faudra donc des idées plus élaborées pour traiter le cas général.

Nous avons donc vu que construire activement une matrice de design pour faire une régression linéaire nécessite des techniques d'optimisation stochastique convexe. Dans la section suivante nous mettrons en évidence des liens encore plus forts entre l'apprentissage actif et l'optimisation stochastique convexe, montrant à quel point ces deux domaines sont liés.

Apprentissage actif et optimisation stochastique adaptative (Chapitre 3)

Malgré leurs apparentes différences, les domaines de l'optimisation stochastique convexe et de l'apprentissage actif ont de nombreuses similitudes qui sont dues à leur aspect séquentiel. Le feedback est en effet essentiel dans ces deux domaines pour décider quelle nouvelle action choisir, ou quel point explorer. Ces liens ont été mis en évidence par Raginsky et [START_REF] Raginsky | Information complexity of black-box convex optimization: A new look via feedback information theory[END_REF] et ont ensuite été étudiés plus en détail par Ramdas et Singh (2013a,b) entre autres, qui ont présenté un lien entre les mesures de complexité utilisées en apprentissage séquentiel et en optimisation stochastique convexe. Considérons par exemple une fonction f sur [0, 1] différentiable et (ρ, µ)-uniformément convexe [START_REF] Zǎlinescu | On uniformly convex functions[END_REF][START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF], c'est-à-dire une fonction qui vérifie, pour

µ > 0 et ρ ≥ 2, 4 ∀(x, y) ∈ [0, 1] 2 , f (y) ≥ f (x) + ∇f (x), y -x + µ 2 x -y ρ .
Supposons maintenant que l'on souhaite minimiser cette fonction f sur [0, 1], cest-à-dire, trouver son minimum x que l'on supposera appartenir à (0, 1). Nous avons donc, pour tout x ∈ [0, 1],

f (x) -f (x ) ≥ µ 2 x -x ρ .
Notons que cette condition est très proche de ce qu'on appelle la condition de bruit de Tsybakov (TNC en anglais) qui apparaît en apprentissage statistique [START_REF] Castro | Minimax bounds for active learning[END_REF]. Considérons maintenant la tâche classique de classification sur [0, 1] : un agent a accès à un ensemble de données D = {(X 1 , Y 1 ), . . . , (X n , Y n )} qui contient n copies alétaoires indépendantes de (X, Y ) ∈ [0, 1]×{-1, +1}, où Y i est l'étiquette du point X i . Son objectif est d'apprendre une fonction de décision g : [0, 1] → {-1, +1} qui minimise la probabilité de faire une erreur de classification, que l'on appelle souvent le risque

R(g) = P (g(X) = Y ) .
Le meilleur classifieur est le classifieur de Bayes g qui est défini comme suit

g (x) = 21 η(x)≥1/2 -1 , où η(x) = P (Y = 1 |X = x)
est la distribution de probabilité a posteriori. On dit alors que η vérifie la condition TNC avec exposant κ > 1 s'il existe λ > 0 tel que

∀x ∈ [0, 1], |η(x) -1/2| ≥ λ x -x κ .
Revenons maintenant au problème de minimisation d'une fonction uniformément convexe f sur [0, 1]. Supposons que l'on veuille utiliser pour cela un algorithme d'optimisation stochastique du premier ordre, c'est-à-dire un algorithme ayant accès à un oracle qui donne des évaluations bruitées ĝ(x) de ∇f (x) à chaque étape. Pour plus de simplicité, supposons que ĝ(x) = ∇f (x)+z où z suit une loi normale standard. Observons maintenant que f (x) ≤ 0 pour x ≤ x et que f (x) ≥ 0 pour x ≥ x puisque f est convexe. Nous voyons donc que si l'on associe à tous les points x ∈ [0, 1] le label sign(ĝ(x)), alors le problème de minimisation de f est équivalent à la classification de ces points sur [0, 1] puisque dans ce cas η(x) = P (ĝ(x) ≥ 0 | x) ≥ 1/2 si x ≥ x . L'analyse de Ramdas et Singh (2013b)

montre que pour x ≥ x , η(x) = P (ĝ(x) ≥ 0 | x) = P f (x) + z ≥ 0 | x = P (z ≥ 0) + P z ∈ -f (x), 0 ≥ 1/2 + λf (x) for λ > 0 , et que de même pour x ≤ x , η(x) ≥ 1/2 + λ|f (x)| .
Remarquons maintenant qu'en utilisant l'inégalité de Cauchy-Schwarz, la convexité de f et ensuite son uniforme convexité, nous trouvons que

|∇f (x)||x -x | ≥ ∇f (x), x -x ≥ f (x) -f (x ) ≥ µ 2 x -x ρ .
Cela montre finalement que

∀x ∈ [0, 1] , |η(x) -1/2| ≥ λµ 2 x -x ρ-1 ,
ce qui signifie que η vérifie le TNC avec l'exposant κ = ρ -1 > 1. Cette analyse montre assez simplement les liens qui existent entre le problème de classification active sur [0, 1] et la minimisation d'une fonction uniformément convexe sur [0, 1] en utilisant un algorithme d'optimisation stochastique du premier ordre. Dans (Ramdas et Singh, 2013a) les auteurs mettent à profit ce lien pour obtenir un algorithme d'optimisation stochastique convexe d'une fonction uniformément convexe en utilisant seulement une information bruitée sur les signes du gradient de la fonction. L'algorithme qu'ils proposent utilise une succession de blocs qui contiennent tous une routine d'apprentissage actif.

Un concept important à la fois en apprentissage actif et en optimisation stochastique est la quantification de la vitesse de convergence des algorithmes. Cette vitesse dépend généralement de la mesure de régularité de la fonction à optimiser, et ainsi dans les problèmes détaillés plus haut cette vitesse dépendra soit de l'exposant κ de la condition TNC, soit de la constante de convexité uniforme ρ. [START_REF] Ramdas | Optimal rates for first-order stochastic convex optimization under tsybakov noise condition[END_REF] ont par exemple montré que la vitesse de convergence min-max pour le problème de minimisation stochastique du premier ordre d'une fonction ρ-uniformément convexe et Lipschitz était Ω(n -ρ/(2ρ-2) ) où n est le nombre d'appels à l'oracle. Nous remarquons d'ailleurs que l'on retrouve la vitesse de convergence Ω(n -1 ) pour les fonctions fortement convexes (ρ = 2) et la vitesse Ω(n -1/2 ) pour les fonctions convexes (ρ → ∞). Notons en outre que cette vitesse de convergence montre bien que la difficulté intrinsèque d'un problème de minimisation est à chercher dans le comportement local de la fonction autour du minimum x : plus ρ est grand, plus la fonction est plate autour du minimum et plus il est donc compliqué de la minimiser.

Cependant lorsque l'on essaye d'optimiser une fonction on ne connaît pas forcément sa régularité, et plus particulièrement son exposant de convexité uniforme. Cela constitue donc l'une des difficultés de l'optimisation stochastique. Malgré cela plusieurs algorithmes ont besoin de connaître ces valeurs pour ajuster leurs propres paramètres. Par exemple, l'algorithme EpochGD [START_REF] Ramdas | Optimal rates for first-order stochastic convex optimization under tsybakov noise condition[END_REF] utilise la valeur de ρ, ce qui peut être irréaliste en pratique. C'est pour cela que l'on a besoin d'algorithmes "adpatatifs", qui n'ont pas besoin des valeurs des paramètres du problème considéré mais qui peuvent s'y adapter afin d'obtenir les vitesses de convergence souhaitées. En s'inspirant de [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF], [START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] et Ramdas et Singh (2013a) ont proposé des algorithmes adaptatifs pour le problème de minimisation stochastique de fonctions uniformément convexes. Ils ont obtenu la même vitesse de convergence O(n -ρ/(2ρ-2) ) que précédemment, mais cette fois-ci sans utiliser la valeur de ρ. Ces deux algorithmes utilisent une succession de blocs dans lequels une valeur approchée de x est calculée en utilisant soit des techniques de moyennage soit d'apprentissage actif.

Malgré le fait que les méthodes d'optimisation stochastique convexe soient souvent du premier ordre, c'est-à-dire qu'elles utilisent des valeurs bruitées du gradient, il est intéressant d'étudier d'autres modèles. Par exemple les méthodes d'optimisation convexe d'ordre zéro [START_REF] Bach | Highly-Smooth Zero-th Order Online Optimization[END_REF] visent à optimiser une fonction en utilisant uniquement des valeurs bruitées du point courant f (x t ) + ε. Cela correspond en fait à utiliser un feedback de type "bandit", c'est-à-dire à connaître seulement la valeur de la fonction au point choisi pour optimiser f . Généralement, lorsque l'on parle de feedback de type bandit on est plutôt intéressé par minimiser le regret

R(T ) = T t=1 f (x t ) -f (x ) ,
plutôt que l'erreur f (x T )-f (x ). Minimiser le regret est d'ailleurs plus compliqué puisque les erreurs faites au début de la phase d'optimisation comptent dans le regret. Ce problème d'optimisation stochastique avec feedback de type bandit a été étudié par [START_REF] Agarwal | Stochastic convex optimization with bandit feedback[END_REF] qui ont proposé dans le cas unidimensionnel un algorithme qui utilise trois points équidistants x l < x c < x r de l'intervalle à explorer, et qui rejette une partie de cet intervalle en fonction des valeurs de f aux trois points. Cet algorithme réalise le regret optimal O( √ T ). L'idée proposée par [START_REF] Agarwal | Stochastic convex optimization with bandit feedback[END_REF] est assez semblable à la dichotomie, sauf que les auteurs choisissent de rejeter un quart de l'intervalle au lieu de la moitié dans une dichotomie. Notons en outre qu'il existe des algorithmes d'apprentissage actif ou d'optimisation convexe du premier ordre qui utilisent effectivement des dichotomies. C'est par exemple le cas de [START_REF] Burnashev | An interval estimation problem for controlled observations[END_REF] sur lequel s'appuie le travail de Castro et Nowak (2006).

Il est intéressant de voir que les méthodes d'optimisation stochastiques qui utilisent le gradient ont généralement comme objectif la minimisation de l'erreur sur la fonction, alors qu'il pourrait aussi être pertinent de minimiser le regret, comme dans les problèmes de bandits. Ce sera par exemple le cas dans le problème d'allocation de ressources que nous définirons prochainement.

Nous avons évoqué ici de nombreux algorithmes d'optimisation stochastique qui utilisent le gradient. Ainsi, dans la prochaine section nous étudierons le célèbre algorithme de descente de gradient ainsi que sa version stochastique en insistant particulièrement sur l'analyse des vitesses de convergence pour le dernier point f (x T ) -f (x ).

Descente de gradient et modèles continus (Chapitre 4)

Considérons maintenant le problème de minimisation d'une fonction f : R d → R convexe et L-lisse 5min

x∈R d f (x) .
(3) S'il existe de nombreuses méthodes pour résoudre ce problème, les plus courantes sont vraisemblablement les méthodes du premier ordre, c'est-à-dire celles qui utilisent la dé-rivée première pour minimiser f . C'est par exemple le cas de l'algorithme de descente de gradient. Ces méthodes sont très en vogue aujourd'hui puisque la taille des données explose et rend impossible la mise en oeuvre de méthodes du second ordre, telles que la méthode de Newton. L'algorithme de descente de gradient part d'un point x 0 ∈ R d et construit de façon itérative une suite de points approchant x = arg min x∈R d f (x) avec la récurrence suivante

x k+1 = x k -η∇f (x k ) avec η = 1/L . ( 4
)
Même s'il existe une preuve classique de la convergence de cet algorithme, par exemple dans [START_REF] Bertsekas | Nonlinear programming[END_REF], nous voulons proposer ici une analyse différente qui s'appuie sur l'équivalent continu de (4). Considérons donc la fonction régulière X : R + → R d qui est telle que X(kη) = x k pour tout k ≥ 0. En utilisant un développement de Taylor à l'ordre 1 on trouve

x k+1 -x k = -η∇f (x k ) X((k + 1)η) -X(kη) = -η∇f (X(kη)) η Ẋ(kη) + O(η) = -η∇f (X(kη)) Ẋ(kη) = -∇f (X(kη)) + O(1) ,
ce qui nous incite à considérer l'Équation Différentielle Ordinaire (EDO) suivante

Ẋ(t) = -∇f (X(t)), t ≥ 0 . (5)
L'EDO (5), qui est l'équivalent continu du schéma discret (4) peut être facilement étudiée en analysant la fonction d'énergie suivante, où l'on a noté f = f (x ),

E(t) t(f (X(t)) -f ) + 1 2 X(t) -x 2 .
En dérivant E et en utilisant la convexité de f on obtient pour tout t ≥ 0,

E (t) = f (X(t)) -f + t ∇f (X(t)), Ẋ(t) + X(t) -x , Ẋ(t) = f (X(t)) -f -t ∇f (X(t)) 2 -∇f (X(t)), X(t) -x ≤ -t ∇f (X(t)) 2 ≤ 0 . Ainsi E est décroissante, et pour tout t ≥ 0 on a t(f (X(t)) -f ) ≤ E(t) ≤ E(0) = 1 2 X(0) -x 2 . Cela nous conduit à la proposition suivante Proposition 1. Supposons que X : R d → R vérifie (5). Alors, pour tout t > 0 f (X(t)) -f ≤ 1 2t X(0) -x 2 .
Nous voulons maintenant transposer cette preuve rapide et élégante au cas discret. Nous proposons donc d'introduire la fonction d'énergie discrète suivante

E(k) = kη (f (x k ) -f (x )) + 1 2 x k -x 2 .
Commençons par un premier lemme.

Lemma 1. Si x k et x k+1 sont deux points successifs de la descente de gradient (4) alors

f (x k+1 ) ≤ f (x ) + 1 η x k+1 -x k , x -x k - 1 2η x k+1 -x k 2 . ( 6
)
Preuve. On a

x k+1 = x k -η∇f (x k ) ce qui donne ∇f (x k ) = x k -x k+1 η .
Le lemme de descente [START_REF] Nesterov | Introductory lectures on convex optimization[END_REF], Lemme 1.2.3) et ensuite la convexité de f donnent

f (x k+1 ) ≤ f (x k ) + ∇f (x k ), x k+1 -x k + L 2 x k+1 -x k 2 ≤ f (x ) + ∇f (x k ), x k -x + x k -x k+1 η , x k+1 -x k + 1 2η x k+1 -x k 2 ≤ f (x ) + 1 η x k+1 -x k , x -x k - 1 2η x k+1 -x k 2 .
Ce deuxième lemme est immédiat et bien connu Lemma 2. Si x k et x k+1 sont deux points successifs de la descente de gradient (4) alors

f (x k+1 ) ≤ f (x k ) - 1 2η x k+1 -x k 2 . ( 7
)
Preuve. Le lemme de descente [START_REF] Nesterov | Introductory lectures on convex optimization[END_REF], Lemme 1.2.3) donne

f (x k+1 ) ≤ f (x k ) + ∇f (x k ), x k+1 -x k + L 2 x k+1 -x k 2 ≤ f (x k ) - 1 2η x k+1 -x k 2 .
Nous étudions maintenant E(k). En multipliant l'Équation (6) par 1/(k + 1) et l'Équation (7) par k/(k + 1) on obtient

f (x k+1 ) ≤ k k + 1 f (x k ) + 1 k + 1 f (x ) - 1 2η x k+1 -x k 2 + 1 k + 1 1 η x k+1 -x k , x -x k f (x k+1 ) -f (x ) ≤ k k + 1 (f (x k ) -f (x )) - 1 2η x k+1 -x k 2 + 1 k + 1 1 η x k+1 -x k , x -x k (k + 1)η (f (x k+1 ) -f (x )) ≤ kη (f (x k ) -f (x )) - k + 1 2 x k+1 -x k 2 + x k+1 -x k , x -x k . Notons alors A k (k + 1)η (f (x k+1 ) -f (x )) -kη (f (x k ) -f (x )). Il vient A k ≤ - k + 1 2 x k+1 -x k 2 + x k+1 -x k , x -x k ≤ k + 1 2 -x k+1 -x 2 -x k -x 2 + 2 x k+1 -x , x k -x + x k+1 -x , x -x k + x k -x 2 ≤ - k + 1 2 x k+1 -x 2 - k -1 2 x k -x 2 + k x k+1 -x , x k -x .
Et ainsi on a

E(k + 1) = (k + 1)η (f (x k+1 ) -f (x )) + 1 2 x k+1 -x 2 ≤ kη (f (x k ) -f (x )) - k 2 x k+1 -x 2 - k 2 x k -x 2 + 1 2 x k -x 2 + k x k+1 -x , x k -x ≤ E(k) - k 2 x k+1 -x 2 + x k -x 2 -2 x k+1 -x , x k -x ≤ E(k) - k 2 x k+1 -x k 2 ≤ E(k) . Cela montre que (E(k)) k≥0 est décroissante et donc que E(k) ≤ E(0) = 1 2 x 0 -x 2 .
Cela nous permet donc d'établir la proposition suivante qui est l'analogue discret de la Proposition 1.

Proposition 2. Soit (x k ) k∈N vérifiant (4) avec f : R d → R une fonction convexe et L-lisse. Alors, pour tout k ≥ 1, f (x k ) -f (x ) ≤ L 2k x 0 -x 2 .
Avec cet exemple simple nous avons montré l'intérêt d'utiliser l'équivalent continu d'un problème discret pour intuiter un schéma de preuve pour le problème discret initial. Nous pouvons remarquer que la preuve dans le cas discret est plus complexe que dans le cas continu. Ce sera toujours le cas au long de ce manuscrit. Une des raisons est qu'il est possible de calculer la dérivée de la fonction d'énergie dans le cas continu alors que c'est impossible pour une fonction discrète. Un moyen de contourner ce problème est d'utiliser le lemme de descente [START_REF] Nesterov | Introductory lectures on convex optimization[END_REF], Lemme 1.2.3) qui peut être vu comme une façon de calculer une dérivée discrète, mais avec des termes et des calculs supplémentaires.

À la suite de ces idées, [START_REF] Su | A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights[END_REF] ont récemment proposé un modèle continu pour la célèbre méthode d'accélération de Nesterov [START_REF] Nesterov | A method for solving the convex programming problem with convergence rate O(1/k 2 )[END_REF]. La méthode d'accélération de Nesterov est une amélioration de la méthode du moment [START_REF] Polyak | Some methods of speeding up the convergence of iteration methods[END_REF] qui était déjà elle-même une amélioration de la descente de gradient standard, qui date en fait de Cauchy (1847). L'idée sous-jacente derrière la méthode du moment est de faire diminuer les oscillations en utilisant une fraction des anciennes valeurs des gradients pour calculer le nouveau point. Ce faisant, la récursion utilise donc une moyenne pondérée (avec des poids qui décroissent de façon exponentielle) des précédents gradients et lisse donc la suite de points en maintenant principalement la direction de descente et en supprimant les oscillations. Cependant, même si la méthode du moment accélère expérimentalement la descente de gradient, elle n'améliore pas sa vitesse théorique donnée dans la Proposition 2, à la différence de la méthode d'accélération de Nesterov que l'on peut écrire ainsi

   x k+1 = y k -η∇f (y k ) avec η ≤ 1/L y k = x k + k -1 k + 2 (x k -x k-1 ) . ( 8
)
La méthode de Nesterov utilise aussi l'idée d'un moment, combinée avec un calcul tardif du gradient, ce qui conduit à une meilleure vitesse de convergence :

Théorème 1. Soit f une fonction convexe et L-lisse. Alors la méthode d'accélération de Nesterov vérifie pour tout k ≥ 1 f (x k ) -f (x ) ≤ 2L x 0 -x 2 k 2 .
Cette vitesse de convergence qui améliore celle de la Proposition 2 atteint la borne inférieure de (Nesterov, 2004, Théorème 2.1.7), mais la preuve n'est pas du tout intuitive ni les idées aboutissant au schéma (8). Le schéma continu introduit par [START_REF] Su | A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights[END_REF] apporte en revanche davantage de compréhension au phénomène d'accélération en proposant d'étudier l'équation différentielle du deuxième ordre

Ẍ(t) + 3 t Ẋ(t) + ∇f (X(t)) = 0, t ≥ 0 .
Les auteurs prouvent la vitesse de convergence suivante pour le modèle continu

pour tout t > 0, f (X(t)) -f ≤ 2 X(0) -x 2 t 2 ,
à nouveau en introduisant une énergie appropriée, et dans ce cas

E(t) = t 2 (f (X(t)) -f )+ 2 X(t) + t Ẋ(t)/2 -x 2 qui est décroissante.
Après avoir effectué cette analyse de l'algorithme de descente de gradient ainsi que de certaines de ses variantes, il est naturel de s'intéresser au cas stochastique. La descente de gradient est en effet très utilisée en apprentissage automatique, et plus particulièrement en apprentissage profond où des algorithmes proches de la descente de gradient servent à minimiser les fonctions de perte de réseaux de neurones, et à apprendre les poids de ces neurones. En apprentissage profond on est généralement intéressé par minimiser des fonctions f qui ont la forme suivante

f (x) = 1 N N i=1 f i (x) , (9)
où f i est associée à la i-ème observation des données d'entraînement (qui sont en nombre N , généralement très grand). C'est pour cela que calculer le gradient de f est très coûteux puisque que cela nécessite de calculer les N gradients ∇f i . Afin d'accélérer la phase d'entraînement on choisit donc généralement d'approximer le gradient de f par ∇f i où i est choisi uniformément au hasard entre 1 et N . On peut aussi faire un compromis entre ce choix et la descente de gradient classique en utilisant un "mini lot", c'est-à-dire un petit ensemble de points de {1, . . . , N } pour calculer le gradient :

∇f (x) ≈ 1 M M i=1 ∇f σ(i) (x) ,
où σ est une permutation de {1, . . . , N } et M est la taille du mini lot. Ces deux choix permettent de calculer une valeur approchée ĝ(x) du vrai gradient ∇f (x) qui en constitue un estimateur non biaisé (E [ĝ(x)] = ∇f (x)) puisque les points utilisés pour calculer ces approximations sont choisis uniformément au hasard. En utilisant ces approximations stochastiques de ∇f (x) à la place de la valeur exacte du gradient dans l'algorithme de descente de gradient on obtient l'algorithme de "Descente de Gradient Stochastique" (SGD en anglais), qui a une formulation plus générale que celle obtenue ci-dessus. En effet l'algorithme SGD apporte une solution au problème de minimisation (3) en utilisant des valeurs bruitées de ∇f et ne se restreint pas aux fonctions de la forme (9). Obtenir des vitesses de convergence pour SGD est bien plus complexe que pour la descente de gradient, à cause des incertitudes dues à son côté stochastique. Dans le cas de SGD le but est en réalité de borner E [f (x k )] -f parce que la suite (x k ) k≥0 est maintenant aléatoire. Dans le cas où f est fortement convexe les résultats de convergence sont bien connus [START_REF] Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF][START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF] mais dans le cas où f est simplement convexe ils ne sont pas aussi communs. En effet la majorité des résultats de convergence connus dans le cas convexe sont obtenus dans le cadre du moyennage de Polyak-Ruppert [START_REF] Polyak | Acceleration of Stochastic Approximation by Averaging[END_REF][START_REF] Ruppert | Efficient estimations from a slowly convergent Robbins-Monro process[END_REF]

Plan du manuscript et contributions

Cette thèse sera divisée en quatre chapitres, qui correspondent chacun à l'un des problèmes que nous avons étudiés. Chacun de ces chapitres a donné lieu à une publication ou à une pré-publication. Nous avons décidé de regrouper les trois premiers chapitres en une première partie qui porte sur l'apprentissage séquentiel, tandis que le dernier chapitre fera l'objet d'une seconde partie assez différente sur l'optimisation stochastique. Le Chapitre 3 peut être vu comme un lien entre les deux parties.

Nous présentons maintenant un résumé de nos contributions principales ainsi que des résultats obtenus dans les prochains chapitres de cette thèse. L'objectif des sections qui suivent est de résumer ces résultats, et non de donner les énoncés exacts et exhaustifs des différents hypothèses et théorèmes. Nous nous sommes efforcés de rendre cette partie facilement lisible et nous invitons le lecteur à se diriger vers les chapitres correspondants pour obtenir tous les détails souhaités.

Partie I Chapitre 1

Dans ce chapitre nous étudions le problème de bandits stochastiques contextuels avec régularisation en adoptant un point de vue non paramétrique. Plus précisément, comme expliqué dans la Section 2.1 nous considérons un ensemble de K ∈ N * bras à qui l'on associe les fonctions de récompense µ k : X → R qui correspondent aux espérances conditionnelles des récompenses de chaque bras sachant le contexte, qui est tiré uniformément au hasard parmi un ensemble X = [0, 1] d . Chacune de ces fonctions est supposée β-Hölder. En notant p : X → ∆ K la mesure d'occupation de chaque bras notre objectif est alors de minimiser la fonction de perte

L(p) = X µ(x), p(x) + λ(x)ρ(p(x)) dx , où ρ : ∆ K → R
est une fonction de régularisation convexe (typiquement l'entropie) et λ : X → R est une fonction modulant la régularisation. Nous allons supposer que ces deux fonctions sont connues par l'agent et sont différentiables.

Nous notons p la fonction des proportions optimales

p = arg inf p∈{f :X →∆ K } L(p) ,
et nous développons dans le Chapitre 1 un algorithme qui renvoie au bout de T itérations une fonction de proportions p T qui minimise le regret

R(T ) = E [L(p T )] -L(p ) .
Puisque p T est en fait le vecteur des fréquences empiriques de chaque bras, R(T ) doit être vu comme un regret cumulé. Nous analysons ensuite l'algorithme que nous avons proposé afin d'obtenir des bornes supérieures sur le regret avec différentes hypothèses. Notre algorithme utilise une partition de l'espace des contextes et résout de façon indépendante un problème d'optimisation convexe sur chacun des sous-ensembles de la partition.

Nous commençons par établir des vitesses de convergence dans le cas où λ est une fonction constante et avec des hypothèses faibles sur les autres paramètres du problème. Nous appellerons "vitesses lentes" les vitesses plus lentes que O(T -1/2 ) (et réciproquement "vitesses rapides" les bornes de convergence plus rapides que O(T -1/2 )).

Théorème 2. Si λ est constante et que ρ est une fonction convexe et lisse alors nous obtenons la vitesse de convergence lente suivante pour le regret, pour tout

T ≥ 1, R(T ) ≤ O   T log(T ) -β 2β+d   .
Si nous supposons en plus que ρ est fortement convexe et que le minimum de la fonction de perte sur chaque sous-ensemble de la partition est atteint loin des bords de ∆ K alors nous obtenons des vitesses plus rapides Théorème 3. Si λ est constante et que ρ est une fonction fortement convexe et lisse, et si L atteint son minimum loin6 de ∂∆ K , alors nous obtenons la vitesse de convergence rapide suivante pour le regret, pour tout T ≥ 1,

R(T ) ≤ O   T log(T ) 2 -2β 2β+d   .
Cette vitesse rapide cache cependant un facteur multiplicatif qui fait intervenir 1/λ et 1/η (où η est la distance de l'optimum aux bords de ∆ K ) et qui peut être arbitrairement grand. Nous nous intéressons donc maintenant au cas où λ est une fonction du contexte, c'est-à-dire que l'agent peut moduler le poids de la régularisation en fonction du contexte. Dans ce cas la distance de l'optimum au bord ∂∆ K dépendra aussi de la valeur du contexte et nous définissons donc la fonction η comme suit p)) atteint son minimum. Afin de pouvoir supprimer toute dépendence en λ et η dans notre borne du regret, tout en obtenant des vitesses plus rapides que celles du Théorème 2 nous devons faire une hypothèse supplémentaire qui va empêcher λ et η de prendre trop souvent des petites valeurs (qui sont la raison d'un facteur multiplicatif trop important dans le Théorème 3). Ceci est classique en estimation non paramétrique et nous faisons donc l'hypothèse suivante, autrement appelée "condition de marge"

η(x) := dist(p (x), ∂∆ K ) , où p (x) ∈ ∆ K est le point où (p → µ(x), p + λ(x)ρ(
Hypothèse 1. Il existe δ 1 > 0, δ 2 > 0, α > 0 et C m > 0 tels que ∀δ ∈ (0, δ 1 ], P X (λ(x) < δ) ≤ C m δ 6α et ∀δ ∈ (0, δ 2 ], P X (η(x) < δ) ≤ C m δ 6α .
Cette condition dépend d'un paramètre de marge α qui contrôle la difficulté du problème et qui permet d'obtenir des vitesses de convergence intermédiaires qui interpolent parfaitement entre les vitesses lentes et les vitesses rapides, sans avoir de dépendance en η ou en λ.

Théorème 4. Si ρ est une fonction convexe, alors avec une condition de marge de paramètre α ∈ (0, 1) nous obtenons la vitesse de convergence suivante pour le regret, pour tout T ≥ 1,

R(T ) = O   T log 2 (T ) -β 2β+d (1+α)   .
Nous pouvons nous demander si les résultats de convergence obtenus dans les trois théorèmes présentés ci-dessus sont optimaux ou pas. Remarquons déjà que ces vitesses de convergence sont classiques en estimation non paramétrique [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. Qui plus est, nous prouvons aussi à la fin du chapitre une borne inférieure pour notre problème, qui montre que la vitesse de convergence du Théorème 3 est optimale aux termes logarithmiques près.

Théorème 5. Pour tout algorithme de bandits qui renvoie pT , avec ρ fortement convexe et µ β-Hölder, il existe une constante universelle C telle que

inf p sup ρ,µ E[L(p T )] -L(p ) ≥ C T -2β 2β+d .
Nous terminons ce chapitre avec des expériences numériques sur des données synthétiques qui illustrent de façon empirique nos résultats de convergence.

Partie I Chapitre 2

Dans ce chapitre nous cherchons à estimer de façon active la matrice de design pour le problème de régression linéaire détaillé à la Section 2.2. Le but ici est d'obtenir la meilleure estimation possible du paramètre β de la régression linéaire, c'est-à-dire de construire un estimateur β à l'aide de T échantillons qui minimise la norme

2 E[ β -β 2 ]. En introduisant la matrice suivante Ω(p) = K k=1 ( p k /σ 2 k )X k X k ,
nous voyons que notre problème correspond en réalité à minimiser la trace de l'inverse de la matrice Ω(p) (qui est la matrice de covariance du problème), puisque

E β -β 2 = 1 T Tr(Ω(p) -1 ) .
Ainsi notre problème est équivalent à la planification en ligne et optimale d'expériences avec le critère A. Plus précisément, introduisons la fonction de perte L(p) = Tr(Ω(p) -1 ) qui est strictement convexe et qui admet donc un minimum p . Notre objectif se reformule donc en la minimisation du regret de notre algorithme, c'est-à-dire que nous cherchons à minimiser l'écart entre la perte de l'algorithme et la plus petite perte réalisable

R(T ) = E β -β 2 -min algo E β(algo) -β 2 = 1 T (E [L(p T )] -L(p )) .
De la même façon qu'à la Section 3.1 notons que R(T ) n'est pas un regret simple mais bien un regret cumulé. Dans le Chapitre 2 nous construisons un algorithme d'apprentissage actif pour résoudre le problème de planification en ligne et optimale d'expériences en nous appuyant sur le travail de Berthet et Perchet (2017). Remarquons que lorsque K < d la matrice Ω(p) est dégénérée, ce qui conduit à un regret linéaire, à moins de restreindre l'analyse au sous-espace induit par les covariables. C'est ce que nous ferons par la suite, ce qui permet donc de considérer maintenant que K ≥ d.

Après avoir obtenu un résultat de concentration sur les variances de variables aléatoires sous-gaussiennes nous analysons notre algorithme, en distinguant deux cas. Dans le premier cas le nombre de covariables K est égal à la dimension de l'espace d. Nous savons donc que tous ces points doivent être tirés un nombre non nul de fois, mais le contrôle de la quantité de tirages est crucial. Nous utilisons donc au début de l'algorithme une phase de pré-échantillonnage de chaque bras qui force la fonction de perte à être localement lisse et qui nous permet d'obtenir des vitesses de convergence rapides. Théorème 6. Dans le cas où K = d nous obtenons la vitesse de convergence rapide suivante, pour tout T ≥ 1,

R(T ) = O log 2 (T ) T 2 .
Il est important de mentionner que cette vitesse rapide n'est pas facile à obtenir. En effet, à la Section 2.3 nous présentons un algorithme naïf qui s'appuie sur des techniques similaires à celles utilisées par UCB, et qui n'atteint qu'un regret en O(T -3/2 ).

Dans le second cas où K > d le problème est bien plus difficile. En effet de nombreuses situations différentes peuvent avoir lieu et le point d'optimum p peut être atteint soit en n'échantillonnant pas certains points, soit en les tirant tous. Trouver la stratégie optimale est donc un problème difficile, ce qui explique la vitesse de convergence plus faible que nous obtenons dans ce cas

Partie I Chapitre 3

Dans ce chapitre nous étudions un problème à la frontière entre l'apprentissage séquentiel et l'optimisation convexe stochastique, qui est un problème d'allocation de ressources que nous formulons de la façon suivante. Supposons qu'un agent a accès à un ensemble de K différentes ressources auxquelles il peut allouer un montant x k , qui génère une récompense f k (x k ). À chaque pas de temps l'agent ne peut qu'allouer un budget total fini, c'est-à-dire que K k=1 x k = 1. Ainsi l'agent reçoit à chaque pas de temps t ∈ {1, . . . , T } la récompense

F (x (t) ) = K k=1 f k (x (t) k ) avec x (t) = (x (t) 1 , . . . , x (t) K ) ∈ ∆ K ,
qui doit être maximisée. En notant x ∈ ∆ K l'allocation optimale qui maximise F , l'objectif de l'agent est équivalent à la minimisation du regret cumulé

R(T ) = F (x ) - 1 T T t=1 K k=1 f k (x (t) k ) = max x∈∆ K F (x) - 1 T T t=1 F (x (t) ) .
Les problèmes d'allocation de ressources ont été étudiés pendant des siècles dans de nombreux domaines d'application et nous faisons maintenant une hypothèse classique qui remonte à Smith (1776) et qui porte le nom d'hypothèse des "rendements décroissants", et qui peut être modélisée par des fonctions de récompense concaves. Dans ce chapitre nous supposerons que l'agent a aussi accès à chaque pas de temps à une valeur bruitée de ∇F (x (t) ) pour réaliser la minimisation du regret, ce qui place l'agent en compétition avec d'autres algorithmes d'optimisation stochastique du premier ordre. Afin de mesurer la complexité du problème que nous étudions nous faisons une hypothèse supplémentaire qui s'appuie sur l'inégalité de Łojasiewicz (Łojasiewicz, 1965), qui correspond à une forme plus faible de la convexité uniforme. L'hypothèse précise sur laquelle nous travaillons est expliquée en détails à la Section 3.2.3 mais nous en énonçons un cas particulier ici par simplicité.

Hypothèse 2. Pour tout

k ∈ {1, • • • , K}, f k est ρ-uniformément concave.
Avec cette hypothèse nous dirons que nous vérifions "inductivement" l'inégalité de Łojasiewicz avec le paramètre β = ρ ρ-1 , comme le montre la Proposition 3.5. L'exposant dans l'inégalité de Łojasiewicz étant supposé inconnu, l'objectif du Chapitre 3 est de construire un algorithme adaptatif à cet exposant et qui minimise le regret. Si nous revenons à la discussion de la Section 2.3 nous cherchons ici à minimiser le regret, ce qui est plus compliqué que de minimiser l'erreur sur la fonction. Ce faisant, cet objectif nous empêche d'utiliser les algorithmes proposés par [START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] ou Ramdas et Singh (2013a) qui obtiennent un regret linéaire.

Nous construisons un algorithme dont le concept central est la dichotomie. Nous en présentons ici un aperçu dans le cas de K = 2 ressources, où l'on a donc

F (x) = f 1 (x 1 ) + f 2 (x 2 ) = f 1 (x 1 ) + f 2 (1 -x 1 ) f 1 (x) + f 2 (1 -x)
, qui peut être vue comme une fonction définie sur [0, 1]. L'idée de l'algorithme est d'évaluer chaque point d'échantillonnage x un nombre suffisant de fois pour obtenir avec grande probabilité le signe de ∇F (x), qui nous dira si x est à droite ou à gauche du point courant x. Notre algorithme consiste donc en une dichotomie qui supprime la moitié de l'intervalle de recherche à chaque phase. Puisque les points qui sont loin de x seront peu échantillonnés (car le signe du gradient à ces points est facile à déterminer) notre algorithme a un regret sous-linéaire. Il est facile de montrer que son regret est même O(T -1 ) dans le cas fortement convexe, ce qui coïncide avec la vitesse classique d'optimisation stochastique pour des fonctions fortement convexes. Dans le cas général nous obtenons la vitesse de convergence suivante en imbricant plusieurs dichotomies les unes dans les autres.

Théorème 9. Si le problème vérifie inductivement l'inégalité de Łojasiewicz avec β ≥ 1 alors nous obtenons la borne suivante sur le regret, après T ≥ 1 échantillons

dans le cas β > 2, E[R(T )] ≤ O K log(T ) log 2 (K) T ; dans le cas β ≤ 2, E[R(T )] ≤ O   K log(T ) log 2 (K)+1 T β/2   .
Notons que sous l'Hypothèse 2, β = ρ/(ρ -1) ≤ 2 et nous obtenons donc une borne sur le regret en T -ρ/(2ρ-2) , ce qui est exactement la vitesse obtenue par Ramdas et Singh (2013a,b) et [START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF], mais cette fois pour le regret et non pour la minimisation de la fonction. Comme dans les chapitres précédents nous analysons l'optimalité de la borne supérieure du théorème précédent en prouvant la borne inférieure suivante dans le cas où β ∈ [1, 2] Théorème 10. Pour tout algorithme il existe une paire de fonctions concaves et croissantes

f 1 et f 2 telles que E [R(T )] ≥ c β T -β 2 , où c β > 0 est une constante indépendante de T .
Ce résultat montre que notre borne supérieure est optimale aux facteurs logarithmiques près. Nous finissons le chapitre en illustrant nos résultats théoriques par des expériences numériques réalisées sur des données synthétiques.

Par ailleurs nous mettons aussi en évidence dans ce chapitre que notre problème généralise le problème des bandits multi-bras, en s'intéressant au cas des ressources linéaires. Nous retrouvons ainsi à la Section 3.3.5 la borne classique en log(T )/(T ∆) des bandits multi-bras.

Partie II Chapitre 4

Dans ce chapitre nous analysons l'algorithme de Descente de Gradient Stochastique (SGD) que nous avons évoqué à la Section 2.4. Soit f : R d → R la fonction à minimiser, que l'on suppose continûment dérivable et lisse. Nous faisons l'hypothèse que nous n'avons pas accès ∇f (x) mais plutôt à des estimations non biaisées H(x, z) où z est une réalisation d'une variable aléatoire Z sur Z de densité µ Z qui vérifie

∀x ∈ R d , Z H(x, z)dµ Z (z) = ∇f (x) .
Nous définissons alors SGD comme suit

X n+1 = X n -γ(n + 1) -α H(X n , Z n+1 ) , ( 10
)
où γ > 0 est le pas initial, α ∈ [0, 1] est un paramètre permettant d'utiliser des pas décroissants et (Z n ) n∈N est une suite de variables aléatoires indépendantes distribuées selon µ Z . Comme expliqué dans la Section 2.4 nous étudions SGD en analysant son pendant continu qui vérifie l'Équation Différentielle Stochastique (EDS) suivante

dX t = -(γ α + t) -α {∇f (X t )dt + γ 1/2 α Σ(X t ) 1/2 dB t } , ( 11
) où γ α = γ 1/(1-α) , Σ(x) = µ Z ({H(x, •) -∇f (x)}{H(x, •) -∇f (x)} ) et (B t ) t≥0 est un mouvement brownien d-dimensionnel.
Une de nos contributions dans le Chapitre 4 est de proposer une nouvelle méthode pour obtenir les vitesses de convergence de SGD, qui utilise l'analyse de l'EDS correspondante. Cette méthode a l'avantage d'être plus simple que les preuves existantes, et nous mettons cela en évidence avec l'exemple des fonctions fortement convexes. Notre méthode consiste à trouver une fonction d'énergie appropriée qui va donner les vitesses de convergence dans le cas continu, et ensuite à adapter la preuve au cas discret en utilisant des techniques similaires, le cas continu servant donc à obtenir l'intuition de la preuve. Nous prouvons par exemple le résultat suivant dans le cas fortement convexe.

Théorème 11. Si f est une fonction lisse et fortement convexe, le schéma SGD (10) qui utilise des pas décroissants de paramètre α ∈ (0, 1] a la vitesse de convergence suivante pour tout N ≥ 1,

E X N -x 2 ≤ CN -α .
Malgré le fait que ce résultat est déjà connu, la preuve que nous proposons est beaucoup plus simple que celle de Bach et Moulines (2011). Pour faire l'analyse du schéma continu nous utilisons le lemme de Dynkin (qui consiste essentiellement à prendre l'espérance dans le lemme d'Itô, voir Lemme 4.13) afin de calculer la dérivée de l'énergie. Dans le cas discret nous remplaçons le lemme de Dynkin par le lemme de descente [START_REF] Nesterov | Introductory lectures on convex optimization[END_REF], Lemme 1.2.3) qui est un équivalent approché discret du lemme de Dynkin, mais qui ne fait pas intervenir la dérivée seconde, ce qui va conduire à des preuves légèrement différentes.

La contribution principale du Chapitre 4 est une analyse exhaustive de SGD dans le cas convexe pour la convergence de la fonction au dernier itéré. Nous considérons en effet la situation où f est convexe et lisse sans faire d'hypothèse de compacité. Nous prouvons alors les deux résultats suivants grâce à des arguments assez similaires. Le premier théorème donne la vitesse de convergence de l'EDS (11).

Théorème 12. Si f est une fonction lisse et convexe, alors il existe C ≥ 0 tel que la suite (X t ) t≥0 définie par l'EDS (11) avec paramètre α ∈ (0, 1) vérifie pour tout T ≥ 1,

E [f (X T )] -f ≤ C(1 + log(T )) 2 /T α∧(1-α) .
Nous prouvons un deuxième résultat similaire dans le cas discret. La preuve est un peu plus complexe du fait des différences entre le lemme de Dynkin et le lemme de descente, mais nous obtenons néanmoins le résultat suivant dont la ressemblance avec le Théorème 12 illustre bien les liens entre les modèles discret et continu.

Théorème 13. Si f est une fonction lisse et convexe alors il existe C ≥ 0 tel que la suite de SGD définie par (10) pour α ∈ (0, 1) vérifie pour tout N ≥ 1,

E [f (X N )] -f ≤ C(1 + log(N + 1)) 2 /(N + 1) α∧(1-α) .
Ce dernier résultat vient contredire la conjecture de Bach et Moulines (2011) qui supposaient que la vitesse optimale de convergence pour le dernier itéré dans SGD était N -1/3 . Pour finir nous nous intéressons au cas où f n'est plus convexe. Nous considérons pour cela une généralisation du cas "faiblement quasi-convexe" [START_REF] Hardt | Gradient Descent Learns Linear Dynamical Systems[END_REF]

en supposant qu'il existe r 1 ∈ (0, 2), r 2 ≥ 0, τ > 0 tels que pour tout x ∈ R d , f (x) -f (x ) ≤ ∇f (x) r 1 x -x r 2 /τ .
Notons en outre que cette condition englobe le cas où f vérifie l'inégalité de Łojasiewicz mentionnée à la Section 3.3 qui peut être définie de la façon suivante pour β ∈ (0, 2)

et c > 0, ∀x ∈ R d , f (x) -f (x ) ≤ c ∇f (x) β ,
et qui a été abondamment utilisée en optimisation. Sous ces hypothèses nous sommes à nouveau en mesure d'obtenir des vitesses de convergence, à la fois pour l'EDS (11) et pour le schéma discret de SGD (10). Ces résultats qui sont rigoureusement établis à la Section 4.3.4 généralisent et améliorent les bornes précédemment obtenues dans le cas faiblement quasi-convexe par [START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF].

Liste des publications

Cette thèse a donné lieu aux publications suivantes :

• (Fontaine et al., 2019a) Dans la suite de cette thèse nous avons fait le choix de déplacer les preuves les plus longues dans les appendices de fin de chapitre pour des raisons de lisibilité.

57

Part I

Sequential learning 1 Regularized contextual bandits

In this chapter we consider the stochastic contextual bandit problem with additional regularization. The motivation comes from problems where the policy of the agent must be close to some baseline policy known to perform well on the task. To tackle this problem we use a nonparametric model and propose an algorithm splitting the context space into bins, solving simultaneously -and independently -regularized multiarmed bandit instances on each bin. We derive slow and fast rates of convergence, depending on the unknown complexity of the problem. We also consider a new relevant margin condition to get problem-independent convergence rates, yielding intermediate rates interpolating between the aforementioned slow and fast rates1 .

Introduction and related work

In sequential optimization problems, an agent takes successive decisions in order to minimize an unknown loss function. An important class of such problems, nowadays known as bandit problems, has been mathematically formalized by Robbins in his seminal paper [START_REF] Robbins | Some aspects of the sequential design of experiments[END_REF]. In the so-called stochastic multi-armed bandit problem, an agent chooses to sample (or "pull") among K arms returning random rewards. Only the rewards of the selected arms are revealed to the agent who does not get any additional feedback. Bandit problems naturally model the exploration/exploitation trade-offs which arise in sequential decision making under uncertainty. Various general algorithms have been proposed to solve this problem, following the work of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] who obtain a logarithmic regret for their sample-mean based policy. Further bounds have been obtained by [START_REF] Agrawal | Sample mean based index policies by o(log(n)) regret for the multiarmed bandit problem[END_REF] and [START_REF] Auer | Finite-time Analysis of the Multiarmed Bandit Problem[END_REF] who developed different versions of the well-known UCB algorithm.

The setting of classical stochastic multi-armed bandits is unfortunately too restrictive for real-world applications. The choice of the agent can and should be influenced by additional information (referred to as "context" or "covariate") revealed by the environment. It encodes features having an impact on the arms' rewards. For instance, in online advertising, the expected Click-Through-Rate depends on the identity, the profile and the browsing history of the customer. These problems of bandits with covariates have been initially introduced by [START_REF] Woodroofe | A one-armed bandit problem with a concomitant variable[END_REF] and have attracted much attention since [START_REF] Wang | Bandit problems with side observations[END_REF]; [START_REF] Goldenshluger | Woodroofe's one-armed bandit problem revisited[END_REF]. This particular class of bandits problems is now known under the name of contextual bandits following the work of [START_REF] Langford | The epoch-greedy algorithm for multi-armed bandits with side information[END_REF].

Contextual bandits have been extensively studied in the last decades and several improvements upon multi-armed bandits algorithms have been applied to contextual bandits, including Thompson sampling [START_REF] Agrawal | Thompson sampling for contextual bandits with linear payoffs[END_REF], Explore-Then-Commit strategies [START_REF] Perchet | The multi-armed bandit problem with covariates[END_REF], and policy elimination [START_REF] Dudik | Efficient Optimal Learning for Contextual Bandits[END_REF]. They are quite intricate to study as they borrow aspects from both supervised learning and reinforcement learning. Indeed they use features to encode the context variables, as in supervised learning but also require an exploration phase to discover all the possible choices. Applications of contextual bandits are numerous, ranging from online advertising [START_REF] Tang | Automatic ad format selection via contextual bandits[END_REF], to news articles recommendation [START_REF] Li | A contextual-bandit approach to personalized news article recommendation[END_REF] or decision-making in the health and medicine sectors [START_REF] Tewari | From Ads to Interventions: Contextual Bandits in Mobile Health[END_REF][START_REF] Bastani | Online Decision-Making with High-Dimensional Covariates[END_REF].

Among the general class of stochastic multi-armed bandits, different settings can be studied. One natural hypothesis that can be made is to consider that the arms' rewards are regular functions of the context i.e., that two close context values have similar expected rewards. This setting has been studied in [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design[END_REF], [START_REF] Perchet | The multi-armed bandit problem with covariates[END_REF] and [START_REF] Slivkins | Contextual bandits with similarity information[END_REF]. A possible approach to this problem is to take inspiration from the regressograms used in nonparametric estimation [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF] and to divide the context space into several bins. This technique also used in online learning [START_REF] Hazan | Online Learning with Prior Knowledge[END_REF] leads to the concept of UCBograms [START_REF] Rigollet | Nonparametric Bandits with Covariates[END_REF] in bandits.

We introduce regularization to the problem of stochastic multi-armed bandits. It is a widely-used technique in machine learning to avoid overfitting or to solve ill-posed problems. Here the regularization forces the solution of the contextual bandits problem to be close to an existing known policy. As an example of motivation, an online-advertiser or any decision-maker may wish not to diverge too much from a handcrafted policy that is known to perform well. This has already motivated previous work such as Conservative Bandits [START_REF] Wu | Conservative bandits[END_REF], where an additional arm corresponding to the handcrafted policy is added. By adding regularization, the agent can be sure to end up close to the chosen policy. Within this setting the form of the objective function is not a classical bandit loss anymore, but contains a regularization term on the global policy. We fall therefore in the more general setting of online optimization and borrow tools from this field to build and analyze an algorithm on contextual multi-armed bandits. As a substitute of the UCB algorithm we use the recently introduced Upper Confidence-Frank Wolfe algorithm [START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF].

Our main contribution consists in an algorithm with proven slow or fast rates of convergence, depending on the unknown complexity of the problem at hand. These rates are better than the ones obtained for classical nonparametric contextual bandits. Based on nonparametric statistics we obtain parameter-independent intermediate convergence rates when the regularization function depends on the context value.

The remainder of this chapter is organized as follows. We present the setting and problem in Section 1.2. Our algorithm is described in Section 1.3. Sections 1.4 and 1.5 are devoted to deriving the convergence rates. Lower bounds are detailed in Section 1.6 and experiments are presented in Section 1.7. Section 1.8 concludes the chapter. Postponed proofs are put in Appendix 1.A.

Problem setting and definitions 1.2.1 Problem description

We consider a stochastic contextual bandit problem with K ∈ N * arms and time horizon T . It is defined as follows. At each time t ∈ {1, . . . , T }, Nature draws a context variable X t ∈ X = [0, 1] d uniformly at random. This context is revealed to an agent who chooses an arm π t amongst the K arms. Only the loss Y (πt) t ∈ [0, 1] is revealed to the agent. For each arm k ∈ {1, . . . , K} we note µ k (X) E(Y (k) |X) the conditional expectation of the arm's loss given the context. We impose classical regularity assumptions on the functions µ k borrowed from nonparametric estimation [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. Namely we suppose that the functions µ k are (β, L β )-Hölder, with β ∈ (0, 1]. We note H β,L β this class of functions.

A 1.1 (β-Hölder). There exists β ∈ (0, 1] and

L β > 0 such that for all k ∈ [K] 2 , µ k is β-Hölder i.e., ∀x, y ∈ X , |µ k (x) -µ k (y)| ≤ L β x -y β .
Unless explicitly specified we will only consider the classical euclidean norm on R d in this chapter. We denote by p : X → ∆ K the proportion function of each arm (also called occupation measure), where ∆ K is the unit simplex of R K . In classical stochastic contextual bandits the goal of the agent is to minimize the following loss function

L(p) = X µ(x), p(x) dx .
We add a regularization term representing the constraint on the optimal proportion function p . For example we may want to encourage p to be close to a chosen proportion function q, or to be far from ∂∆ K . In order to do that we consider a convex regularization function ρ : ∆ K × X → R and a regularization parameter λ : X → R. Both ρ and λ are known and given to the agent, while the µ k functions are unknown and must be learned. We want to minimize the loss function

L(p) = X µ(x), p(x) + λ(x)ρ(p(x), x) dx . (1.1)
This is the most general form of the loss function. We study first the case where the regularization does not depend on the context (i.e., when λ is a constant and when ρ is only a function of p).

The function λ modulates the weight of the regularization and is chosen to be regular enough. More precisely we make the following assumption on the regularization term.

A 1.2. λ is a C ∞ non-negative function and ρ is a C 1 convex function.
We will see later the convexity of ρ is not enough and that we actually need strong convexity.

Definition 1.1. We say that ρ is a C 1 ζ-strongly convex function with ζ > 0 if ρ is continuously differentiable and if ∀ (p, q) ∈ (∆ K ) 2 , ρ(q) ≥ ρ(p) + ∇ρ(p), q -p + ζ 2 p -q 2 .
In the rest of this chapter all strongly convex functions will be assumed to be of class C 1 . We will also be led to consider S-smooth functions.

Definition 1.2. A continuously differentiable and real-valued function f defined on a set

D ⊂ R K is S-smooth (with S > 0) if its gradient is S-Lipschitz continuous, i.e., ∀(x, y) ∈ D 2 , ∇f (x) -∇f (y) ≤ S x -y .
The optimal proportion function is denoted by p and verifies

p = arg inf p∈{f :X →∆ K } L(p) .
If an algorithm aiming at minimizing the loss L returns a proportion function p T we define the regret as follows.

Definition 1.3. The regret of an algorithm outputting p

T ∈ {p : X → ∆ K } is R(T ) = EL(p T ) -L(p ) .
In the previous definition the expectation is taken on the choices of the algorithm. The goal is to find after T samples a p T ∈ {p : X → ∆ K } the closest possible to p in the sense of minimizing the regret. Note that R(T ) is actually a cumulative regret, since p T is the vector of the empirical frequencies of each arm i.e., the normalized total number of pulls of each arm. Earlier choices affect this variable unalterably so that we face a trade-off between exploration and exploitation.

Examples of regularizations

The most natural regularization function considered throughout this chapter is the (negative) entropy function defined as follows:

ρ(p) = K i=1 p i log(p i ) for p ∈ ∆ K . Since ∇ 2 ii ρ(p) = 1/p i ≥ 1, ρ is 1-strongly convex.
Using this function as a regularization forces p to go to the center of the simplex, which means that each arm will be sampled a linear amount of time.

We can consider instead the Kullback-Leibler divergence between p and a known proportion function q ∈ ∆ K :

ρ(p) = D KL (p||q) = K i=1 p i log p i q i for p ∈ ∆ K .
Instead of pushing p to the center of the simplex, the KL divergence will push p towards q. This is typically motivated by problems where the decision maker should not alter too much an existing policy q, known to perform well on the task. Another way to force p to be close to a chosen policy q is to use the 2 -regularization ρ(p) = p -q 2 2 . These two last examples have an explicit dependency on x since q depends on the context values, which was not the case of the entropy (which only depends on x through p). Both the KL divergence and the 2 -regularization have a special form that allows us to remove this explicit dependency on x. They can indeed be written as Indeed,

ρ(p(x), x) = H(p(x)) + p(x), k(x) + c(x) ,
D KL (p||q) = K i=1 p i (x) log p i (x) q i (x) = K i=1 p i (x) log p i (x) H(p(x)) + p(x), -log q(x) k(x)
.

And

p(x) -q(x) 2 2 = p(x) 2 H(p(x)) + p(x), -2q(x) k(x) + q(x) 2 c(x)
.

With this specific form the loss function writes as

L(p) = X µ(x), p(x) + λ(x)ρ(p(x), x) dx = X µ(x) + λ(x)k(x), p(x) + λ(x)H(p(x)) dx + X λ(x)c(x) dx .
Since we aim at minimizing L with respect to p, the last term X λ(x)c(x) dx is irrelevant for the minimization. Let us now note μ = µ + λk. We are now minimizing

L(p) = X μ(x), p(x) + λ(x)H(p(x)) dx .
This is actually the standard setting of Section 1.2.1 with a regularization function H independent of x. In order to preserve the regularity of μ we need λρ to be β-Hölder which is the case if q is sufficiently regular. Nonetheless, we remark that the relevant regularity is the one of µ since λ and ρ are known. As a consequence, from now on we will only consider regularization functions ρ that only depend on p.

The Upper-Confidence Frank-Wolfe algorithm

We now briefly present the Upper-Confidence Frank-Wolfe algorithm (UC-FW) from [START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF], that will be an important tool of our own algorithm. This algorithm is designed to optimize an unknown convex function L : ∆ K → R. At each time step t ≥ 1 the feedback available is a noisy estimate of ∇L(p t ), where p t is the vector of proportions of each action. The algorithm chooses the arm k minimizing a lower confidence estimate of the gradient value (similarly to the UCB algorithm [START_REF] Auer | Finite-time Analysis of the Multiarmed Bandit Problem[END_REF]) and updates the proportions vector accordingly. Slow and fast rates for this algorithm are derived by the authors.

Description of the algorithm 1.3.1 Idea of the algorithm

As the time horizon T is finite, even if we could use the doubling-trick, and the reward functions µ k are smooth, we choose to split the context space X into B d cubic bins of side size 1/B, where B ∈ N * . Inspired by UCBograms [START_REF] Rigollet | Nonparametric Bandits with Covariates[END_REF] we are going to construct a (bin by bin) piecewise constant solution pT of (1.1).

We denote by B the set of bins introduced. If b ∈ B is a bin we note |b| = B -d its volume and diam(b) = √ d/B its diameter. Since pT is piecewise constant on each bin b ∈ B (with value pT (b)), we rewrite the loss function into

L(p T ) = X µ(x), pT (x) + λ(x)ρ(p T (x)) dx = b∈B b µ(x), pT (b) + λ(x)ρ(p T (b)) dx = 1 B d b∈B μ(b), pT (b) + λ(b)ρ(p T (b)) = 1 B d b∈B L b (p T (b)) , (1.2)
where Receive context X t from the environment

5: b t ← bin of X t 6:
Perform one iteration of the UC-FW algorithm for the L bt function on bin b t 7: return the proportion vector (p T (1), . . . , p T (B d ))

Line 2 of Algorithm 1.1 consists in a pre-sampling stage where all arms are sampled a certain amount of time. It guarantees that p T (k) is bounded away from 0 so that p T is bounded away from the boundary of ∆ K , which will be required when L b is not smooth on ∂∆ K . We will see how this can be used to enforce constraints on the p i and especially to force p to be far from the boundaries of ∆ K . More details on this pre-sampling stage will be given in the following sections.

In the remaining of this chapter, we derive slow and fast rates of convergence for this algorithm, depending on the complexity of the current instance of the problem.

Estimation and approximation errors

In order to obtain a bound on the regret, we decompose it into an estimation error and an approximation error. The approximation error is the minimal achievable error within the class of piecewise constant functions.

Definition 1.4. The approximation error A(p) is the error between the best piecewise constant function p and the optimal solution p .

A(p ) = L(p ) -L(p ) .
The estimation error is due to the errors made by the algorithm.

Definition 1.5. The estimation error E(p T ) is the error between the result of the algorithm p T and the best piecewise constant function p .

E(p T ) = EL(p T ) -L(p ) = 1 B d b∈B EL b (p T (b)) -L b (p b ) ,
where the last equality comes from (1.2).

We naturally have R(T ) = E(p T ) + A(p ). In order to bound R(T ) we want to obtain bounds on both the estimation and the approximation error terms.

Convergence rates for constant λ

In this section we consider the case where λ is constant.

A 1.3. λ is a constant positive function on R d .
We derive slow and fast rates of convergence.

Slow rates

In order to derive the slow rates we begin with a lemma on the concentration of T b , the number of context samples falling in a bin b.

Lemma 1.1. For all b ∈ B, let T b the number of context samples falling in the bin b. We have

P ∃b ∈ B, T b - T B d ≥ 1 2 T B d ≤ 2B d exp - T 12B d . Proof. For a bin b ∈ B and t ∈ {1, . . . , T }, let Z (b) t = 1 {Xt∈B} which is a random Bernoulli variable of parameter 1/B d .
We have

T b = T t=1 Z t and E[T b ] = T /B d .
Using a multiplicative Chernoff's bound (Vershynin, 2018) we obtain:

P |T b -E[T b ]| ≥ 1 2 E[T b ] ≤ 2 exp - 1 3 1 2 2 T B d = 2 exp - T 12B d .
We conclude with an union bound on all the bins.

The analysis of the UC-FW algorithm gives the following bound.

Proposition 1.1. Assume A1.1, A1.2, A1.3 and that ρ is S-smooth on ∆ K . If p T is the result of Algorithm 1.1 and p the best piecewise constant function on the set of bins B, then for all T ≥ 1, the following bound on the estimation error holds3

EL(p T ) -L(p ) = O   √ KB d/2 log(T ) T   .
Proof. We have

E(p T ) = EL(p T ) -L(p ) = 1 B d b∈B EL b (p T (b)) -L b (p b ) .
Let us now consider a single bin b ∈ B. We have run the UC Frank-Wolfe [START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF] algorithm for the function L b on the bin b with

T b iterations. For all p ∈ ∆ K , L b (p) = μ(b), p + λρ(p), then for all p ∈ ∆ K , ∇L b (p) = μ(b) + λ∇ρ(p) and ∇ 2 L b (p) = λ∇ 2 ρ(p). Since ρ is a S-smooth convex function, L b is a λS-smooth convex function.
We consider the event A:

A ∀b ∈ B, T b ∈ T 2B d , 3T 2B d . Lemma 1.1 shows that P(A ) ≤ 2B d exp - T 12B d .
Using [START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF], Theorem 3) we obtain on event A:

EL b (p T (b)) -L b (p b ) ≤ 4 3K log(T b ) T b + S log(eT b ) T b + π 2 6 + K 2 ∇L b ∞ + L b ∞ T b ≤ 4 6K log(T ) T /B d + 2S log(eT ) T /B d + 2 π 2 6 + K 2 ∇L b ∞ + L b ∞ T /B d .
Since ρ is of class C 1 , ρ and ∇ρ are bounded on the compact set ∆ K . It is also the case for L b and consequently L b ∞ and ∇L b ∞ exist and are finite and can be expressed in function

of ρ ∞ , ∇ρ ∞ and λ ∞ . On event A , EL b (p T (b)) -L b (p b ) ≤ 2 L b ∞ ≤ 2 + 2 λρ ∞ .
Summing over all the bins in B we obtain:

EL(p T ) -L(p ) ≤ 4B d/2 6K log(T ) T + B d 2S log(eT ) T + 4KB d 4 + 2 λ∇ρ ∞ + λρ ∞ T + 4B d (1 + λρ ∞ )e -T 12B d . (1.3)
The first term of Equation (1.3) dominates the others and we can therefore write that

EL(p T ) -L(p ) = O √ KB d/2 log(T ) T ,
where the O is valid for T → ∞.

Some regularization functions are not S-smooth on ∆ K , for example the entropy whose Hessian is not bounded on ∆ K . The following proposition shows that the previous result still holds, at least for the entropy.

Proposition 1.2. Assume A1.1, A1.2, A1.3 and that ρ is the entropy function, then for all T ≥ 1, the following bound on the estimation error holds

EL(p T (b)) -L(p ) = O B d/2 log(T ) √ T .
The idea of the proof is to force the result of the algorithm to be "inside" the simplex ∆ K (in the sense of the induced topology) by pre-sampling each arm.

Proof. We consider a bin b ∈ B containing t samples. We now have to distinguish two cases.

Let S p ∈ ∆ K | ∀i ∈ [K], p i ≥ λ √ t .
(a) Case 1: pb = p b , i.e., the minimum of

L b is in S. For all p ∈ ∆ K , L b (p) = μ(b), p + λρ(p), then for all p ∈ ∆ K , ∇L b (p) = μ(b) + λ(1 + log(p)) and ∇ 2 ii L b (p) = λ/p i . Therefore on S we have ∇ 2 ii L b (p) ≤ √ t .
And consequently

L b is √ t-smooth. And since ∇ i L b (p) = 1 + λ log(p i ), ∇L b (p) ∞ log(t)
. We can apply the same steps as in the proof of Proposition 1.1 to find that

EL b (p t (b)) -L b (p b ) ≤ 4 3K log(t) t + √ t log(et) t + π 2 6 + K 2 log(t) + log(K) t = O log(t) √ t .
(b) Case 2: pb = p b . By strong convexity of L b , pb cannot be a local minimum of L b and therefore pb ∈ ∂S.

The Case 1 shows that

EL b (p t (b)) -L b (p b ) = O log(t) √ t . Let π = (π 1 , . . . , π K ) with π i max(λ/ √ t, pb,i ). We have π -pb 2 ≤ √ Kλ/ √ t.
Let us derive an explicit formula for p b knowing the explicit expression of ρ. In order to find the optimal ρ value let us minimize (p → L b (p)) under the constraint that p lies in the simplex ∆ K . The KKT equations give the existence of ξ ∈ R such that for each i ∈ [K], μi (b)+λ log(p i )+λ+ξ = 0 which leads to p b,i = e -μi(b)/λ /Z where Z is a normalization factor.

Since Z = K i=1 e -μi(b)/λ we have Z ≤ K and p b,i ≥ e -1/λ /K. Consequently for all p on the segment between π and p b we have p i ≥ e -1/λ /K and therefore λ(1+log

(p i )) ≥ λ(1-log K)-1 and finally |∇ i L b (p)| ≤ 4 λ ∞ log(K). Therefore L b is 4 √ K log(K)-Lipschitz and L b (p b ) -L b (π) 2 ≤ 4 λ ∞ √ K log(K) π -pb 2 ≤ 4K log(K) λ 2 ∞ / √ t = O(1/ √ t) . Finally, since L b (π) ≥ L b (p b ) (because π ∈ S), we have EL b (p t (b)) -L b (p b ) ≤ EL b (p t (b)) -L b (p b ) + L b (p b ) -L b (p b ) = O log(t) √ t + L(π) -L(p b ) = O log(t) √ t .
We conclude by summing on the bins and using that t ∈ [T /2B d , 3T /2B d ] with high probability, as in the proof of Proposition 1.1.

In order to obtain a bound on the approximation error we notice that

L b (p b ) = inf p∈∆ K L b (p) = inf p∈∆ K {λρ(p) --μ(b), p } = -(λρ) * (-μ(b)) = -λρ * - μ(b) λ ,
where ρ * is the Legendre-Fenchel transform of ρ.

Similarly, b µ(x), p (x) + λρ(p (x)) dx = b inf p∈∆ K --µ(x), p + λρ(p) dx = b -(λρ) * (-µ(x)) dx = b -λρ * - µ(x) λ dx .
We want to bound

A(p ) = b∈B b µ(x), p (x) + λρ(p (x)) -µ(x), p (x) -λρ(p (x)) dx = b∈B b μ(b), p b + λρ(p b ) -µ(x), p (x) -λρ(p (x)) dx = b∈B b L b (p b ) dx - b µ(x), p (x) + λρ(p (x)) dx = λ b∈B b ρ * (-µ(x)/λ) -ρ * (-μ(b)/λ) dx
L(p ) -L(p ) ≤ L β Kd β B -β .
Proof. We have to bound the quantity

L(p ) -L(p ) = λ b∈B b ρ * (-µ(x)/λ) -ρ * (-μ(b)/λ) dx .
Classical results on convex conjugates (Hiriart-Urruty and Lemaréchal, 2013a) give that ∇ρ * (y) = arg min x∈∆ K ρ(x) -x, y for all y ∈ R K . Consequently, ∇ρ * (y) ∈ ∆ K and for all y ∈ R K , ∇ρ * (y) ≤ 1 showing that ρ * is 1-Lipschitz continuous. This leads to

L(p ) -L(p ) ≤ λ b∈B b µ(x) -μ(b) λ dx ≤ b∈B b L β K √ d B β dx ≤ L β Kd β B -β ,
because all the µ k are (L β , β)-Hölder.

Combining Propositions 1.1 and 1.3 we obtain the following theorem Theorem 1.1 (Slow rates). Assume A1.1, A1.2, A1.3 and that ρ is S-smooth. Applying Algorithm 1.1 with choice B = Θ (T / log(T )) 1/(2β+d) gives4 for all T ≥ 1,

R(T ) = O L β ,β,K,d   T log(T ) -β 2β+d   .
Proposition 1.2 directly shows that the result of this theorem also holds when ρ is the entropy function.

The proof of this theorem consists in choosing a value of B balancing between the estimation and the approximation errors. Since β ∈ (0, 1], we see that the exponent of the convergence rate is below 1/2 and that the proposed rate is slower than T -1/2 , hence the denomination of slow rate.

Proof. We will denote by C k with increasing values of k the constants. Since the regret is the sum of the approximation error and the estimation error we obtain

R(T ) ≤ L β d β KB -β + C 1 √ KB d/2 log(T ) T + B d 2S log(eT ) T + C 2 K B d T + 4B d (1 + λρ ∞ ) exp - T 12B d .
With the choice of

B = C 2 β L β d β/2-1 1/(β+d/2) T log(T ) 1/(2β+d)
, we find that the three last terms of the regret are negligible with respect to the first two. This gives

R(T ) = O 3 √ KL d/(4β+2d) β d β(4+d)/(4β+2d) (C 2 β) -β/(2β+d) T log(T ) -β/(2β+d)
.

When λ = 0 we are in the usual contextual bandit setting. The propositions of this section hold and we recover the slow rates from [START_REF] Perchet | The multi-armed bandit problem with covariates[END_REF].

Fast rates

We now consider possible fast rates i.e., convergence rates faster than O(T -1/2 ). The price to pay to obtain these quicker rates compared to the ones from Section 1.4.1 is to have problem-dependent bounds i.e., convergence rates depending on the parameters of the problem, and especially on λ.

As in the previous section we can obtain a bound on the estimation error based on the convergence rates of the Upper-Confidence Frank-Wolfe algorithm.

This result needs additional assumptions, namely strong convexity and the following assumption that we will make in the rest of this section. It consists in assuming that the minimum of the loss function on each bin is reached far from the boundaries of ∆ K .

EL(p T ) -L(p ) = O B d Sλ + K λ 2 ζ 2 η 4 log 2 (T )
T .

Proof. The proof is very similar to the one of Proposition 1.1. We decompose the estimation error on the bins:

EL(p T ) -L(p ) = 1 B d b∈B EL b (p T (b)) -L b (p b ) .
Let us now consider a single bin b ∈ B. We have run the UCB Frank-Wolfe algorithm for the function L b on the bin b with T b samples.

As in the proof of Proposition 1.1 we consider the event A.

(Berthet and Perchet, 2017, Theorem 7) applied to L b which is a λS-smooth λζ-strongly convex function shows that on event A:

EL(p T ) -L(p ) ≤ 2c 1 log 2 (T ) T /B d + 2c 2 log(T ) T /B d + c3 2 T /B d , with c1 = 96K ζλη 2 , c2 = 24 ζλη 3 + λS and c3 = 24 20 ζλη 2 2 K + λζη 2 2 + λS. Consequently EL(p T ) -L(p ) ≤ 2c 1 log 2 (T ) T /B d + 2c 2 log(T ) T /B d + c3 2 T /B d + 4B d (1 + λρ ∞ ) exp - T 12B d .
In order to have a simpler expression we can use the fact that λ and η are constants that can be small while S can be large. Consequently c3 is the largest constant among c1 , c2 and c3 and we obtain

EL(p T ) -L(p ) ≤ O K λ 2 ζ 2 η 4 + Sλ B d log 2 (T ) T ,
because the other terms are negligible.

The previous bound depends on several parameters of the problem: λ, distance η of the optimum to the boundary of the simplex, strong convexity and smoothness constants. Since λ can be arbitrarily small, η can be small as well and S large. Therefore the "constant" factor can explode despite the convergence rate being "fast": these terms describe only the dependency in T .

As in the previous section we want to consider regularization functions ρ that are not smooth on ∂∆ K . To do so we force the vectors p to be inside the simplex by pre-sampling all arms at the beginning of the algorithm. The following lemma shows that this is valid.

Lemma 1.2. On a bin b ∈ B if there exists α ∈ (0, 1/2] and p o ∈ ∆ K such that p b αp o (component-wise) then for all i ∈ [K], the agent can safely sample arm i αp o i T times at the beginning of the algorithm without changing the convergence results.

The intuition behind this lemma is that if all arms have to be sampled a linear amount of times to reach the optimum value, it is safe to pre-sample each of the arms linearly at the beginning of the algorithm. The goal is to ensure that the current proportion vector p t will always be far from the boundary in order to leverage the smoothness of ρ in the interior of the simplex.

Proof. We consider a single bin b ∈ B. Let us consider the function

Lb : p → L b (αp o + (1 -α)p) .
Since for all i, p b,i ≥ αp o i and since ∆ K is convex we know that min p∈∆ K Lb (p) = L b (p b ). If p is the frequency vector obtained by running the UCB-Frank Wolfe algorithm for function Lb with (1 -α)T samples then minimizing Lb is equivalent to minimizing L with a presampling stage.

Consequently the whole analysis on the regret still holds with T replaced by (1 -α)T . Thus fast rates are kept with a constant factor 1/(1 -α) ≤ 2.

Proposition 1.5. If ρ is the entropy function, sampling each arm T e -1/λ /K times during the presampling phase guarantees the same estimation error as in Proposition 1.4 with constant S = Ke 1/λ . Proof. For the entropy regularization, we have

p b,i = exp(-μ(b) i /λ) K j=1 exp(-μ(b) j /λ) ≥ exp(-1/λ) K .
We apply Lemma 1.2 with p o = 1 K , . . . , 1 K and α = exp(-1/λ). Consequently each arm is presampled T exp(-1/λ)/K times and finally we have

∀i ∈ [K], p i ≥ exp(-1/λ) K . Therefore we have ∀i ∈ [K], ∇ ii ρ(p) = 1 p i ≤ K exp(1/λ) , showing that ρ is K exp(1/λ)-smooth.
In order to obtain faster rates for the approximation error we use Equation (1.4) and the fact that ∇ρ * is 1/ζ-Lipschitz since ρ is ζ-strongly convex.

Proposition 1.6. Assume A1.1, A1.2, A1.3 and that ρ is ζ-strongly convex. If p is the piecewise constant function on the set of bins B minimizing the loss function L, the following bound on the approximation error holds

L(p ) -L(p ) ≤ L β Kd β 2ζλ B -2β .
In order to prove Proposition 1.6 we will need the following lemma which is a direct consequence of a result on smooth convex functions.

Lemma 1.3. Let f : R d → R be a convex function of class C 1 and L > 0. Let g : R d x → L 2 x 2 -f (x). Then g is convex if and only if ∇f is L-Lipschitz continuous.
Proof. Since g is continuously differentiable we can write

g convex ⇔ ∀x, y ∈ R d , g(y) ≥ g(x) + ∇g(x), y -x ⇔ ∀x, y ∈ R d , L 2 y 2 -f (y) ≥ L 2 x 2 -f (x) + Lx -∇f (x), y -x ⇔ ∀x, y ∈ R d , f (y) ≤ f (x) + ∇f (x), y -x + L 2 y 2 + x 2 -2 x, y ⇔ ∀x, y ∈ R d , f (y) ≤ f (x) + ∇f (x), y -x + L 2 x -y
We can now prove Proposition 1.6.

Proof. Since ρ is ζ-strongly convex then ∇ρ * is 1/ζ-Lipschitz continuous (see for example (Hiriart-Urruty and Lemaréchal, 2013b, Theorem 4.2.1, page 82)). Since ρ * is also convex, Lemma 1.3 shows that g :

x → 1 2ζ x 2 -ρ * (x) is convex.
Let us now consider the bin b and the function µ = (µ 1 , . . . , µ k ). Jensen's inequality gives:

1

|b| b g(-µ(x)/λ) dx ≥ g 1 |b| b - µ(x) λ dx . This leads to b g(-µ(x)/λ) dx ≥ b g(-μ(b)/λ) dx b 1 2ζ -µ(x) 2 /λ 2 -ρ * (-µ(x)/λ) dx ≥ b 1 2ζ -μ(b) 2 /λ 2 -ρ * (-μ(b)/λ) dx b ρ * (-µ(x)/λ) -ρ * (-μ(b)/λ) dx ≤ 1 2ζλ 2 b µ(x) 2 -μ(b) 2 dx .
We use the fact that b µ(x) -μ(b)

2 dx = b µ(x) 2 + μ(b) 2 -2 µ(x), μ(b) dx = b µ(x) 2 + μ(b) 2 dx -2 μ(b), b µ(x) dx = b µ(x) 2 + μ(b) 2 dx -2 μ(b), |b|μ(b) = b µ(x) 2 -μ(b) 2 dx and we get finally b ρ * (-µ(x)/λ) -ρ * (-μ(b)/λ) dx ≤ 1 2ζλ 2 b µ(x) -μ(b) 2 dx .
Equation (1.4) shows that

L(p ) -L(p ) ≤ 1 2ζλ b∈B b μ(b) -µ(x) 2 dx ≤ b∈B b L β K 2ζλ √ d B 2β dx ≤ L β Kd β 2ζλ 1 B 2β , because each µ k is (L β , β)-Hölder.
Combining Propositions 1.4 and 1.6, we obtain fast rates for our problem.

Theorem 1.2 (Fast rates). Assume A1.1, A1.2, A1.3, A1.4 and that ρ is ζ-strongly convex and S-smooth. Then applying Algorithm 1.1 with the choice B = Θ T / log 2 (T )

1/(2β+d)
gives the following bound on the regret for all T ≥ 1,5

R(T ) = O L β ,β,K,d,λ,η,ζ,S   T log 2 (T ) -2β 2β+d   .
Proof. We denote again by C k the constants. We sum the approximation and the estimation errors (given in Propositions 1.6 and 1.4) to obtain the following bound on the regret:

R(T ) ≤ C 1 L β Kd β ζλ B -2β + C 2 log 2 (T ) T B d 1 ζλη 3 + K ζ 2 λ 2 η 4 + λζη 2 + λS + 4B d (1 + λρ ∞ ) exp - T 12B d .
For the sake of clarity let us note

ξ 1 C 1 L β Kd β ζλ and ξ 2 C 2 1 ζλη 3 + K ζ 2 λ 2 η 4 + λζη 2 + λS . We have R(T ) ≤ ξ 1 B -2β + ξ 2 B d log 2 (T ) T + 4B d (1 + λρ ∞ ) exp - T 12B d . Taking B = 2ξ 1 β ξ 2 1/(2β+d) T log 2 (T ) 1/(d+2β)
, we notice that the third term is negligible and we conclude that

R(T ) = O 2ξ 1 2ξ 1 β ξ 2 -2β/(2β+d) T log 2 (T ) -2β/(2β+d)
.

The rate of Theorem 1.2 matches the rates obtained in nonparametric estimation [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. However, as shown in the proof this fast rate is obtained at the price of a factor involving λ, η and S, which can be arbitrarily large. It is the goal of the next section to see how to remove this dependency in the parameters of the problem.

Proposition 1.5 shows that the previous theorem can also be applied to the entropy regularization.

Convergence rates for non-constant λ

In this section, we study the case where λ is a function of the context value and do not assume any more A1.3. This is quite interesting as agents might want to modulate the weight of the regularization term depending on the context.

Estimation and approximation errors

Equation (1.2) implies that the estimation errors obtained in Propositions 1.1 and 1.4 are still correct if λ is replaced by λ(b). This is unfortunately not the case for the approximation error propositions because Equation (1.4) does not hold anymore. Indeed the approximation error becomes

A(p ) = b∈B b µ(x), p (x) + λ(x)ρ(p (x)) -µ(x), p (x) -λ(x)ρ(p (x)) dx = b∈B b μ(b), p b + λ(x)ρ(p b ) -µ(x), p (x) -λ(x)ρ(p (x)) dx = b∈B b L b (p b ) dx - b µ(x), p (x) + λ(x)ρ(p (x)) dx = b∈B b -( λ(b)ρ) * (-μ(b)) + (λ(x)ρ) * (-µ(x)) dx = b∈B b λ(x)ρ * - µ(x) λ(x) -λ(b)ρ * - μ(b) λ(b) dx . (1.5)
The main difference with Equation (1.4) is that λ is not constant anymore. From this expression we obtain the following slow and fast rates of convergence. These rates are the same as in Section 1.4 in term of the powers of B but have worse dependency in λ.

Proposition 1.7. If ρ is a strongly convex function and λ a C ∞ integrable non-negative function whose inverse is also integrable, we have on a bin b:

b (λ(x)ρ) * (-µ(x)) -( λ(b)ρ) * (-μ(b)) dx = O(L β d β/2 B -β-d ) .
We begin with a lemma on convex conjugates that will help us proving Proposition 1.7.

Lemma 1.4. Let λ, µ > 0 and let y ∈ R n and ρ a non-negative convex function on

∆ K . Then (λρ) * (y) -(µρ) * (y) ≤ |λ -µ| ρ ∞ . Proof. Let λ, µ > 0 and let y ∈ R n . (λρ) * (y) = sup x∈∆ K x, y -λρ(x)
x λ , y -λρ(x λ ), where x λ ∈ ∆ K is the point where the supremum of the concave problem is reached.

And (µρ)

* (y) = sup x∈∆ K x, y -µρ(x) x µ , y -µρ(x µ ) ≥ x λ , y -µρ(x λ )
, where x µ ∈ ∆ K is defined as above.

Then, (λρ

) * (y) -(µρ) * (y) ≤ x λ , y -λρ(x λ ) -( x λ , y -µρ(x λ )) = (µ -λ)ρ(x λ ). Finally (λρ) * (y) -(µρ) * (y) ≤ |λ -µ| ρ ∞ , because ρ is continuous hence bounded on the compact set ∆ K .
Proof of Proposition 1.7. There exists x 0 ∈ b such that λ(b) = λ(x 0 ) and x 1 ∈ b such that μ(b) = µ(x 1 ). We use Lemma 1.4 to derive a bound for the approximation error.

(λ(x)ρ) * (-µ(x)) -( λ(b)ρ) * (-μ(b)) dx = b (λ(x)ρ) * (-µ(x)) -(λ(x)ρ) * (-μ(b)) dx + b (λ(x)ρ) * (-μ(b)) -( λ(b)ρ) * (-μ(b)) dx ≤ b λ(x) ρ * - µ(x) λ(x) -ρ * - μ(b) λ(x) dx + b |λ(x) -λ(b)| ρ ∞ dx ≤ b λ(x) µ(x) λ(x) - μ(b) λ(x) dx + ρ ∞ b |λ(x) -λ(x 0 )| dx ≤ b L β |x -x 1 | β dx + ρ ∞ b λ ∞ |x -x 0 | dx ≤ B -d L β d β/2 B -β + ρ ∞ λ ∞ √ dB -1 = O(B -β-d ) .
The important point is that the bound does not depend on λ min , which is not the case when we want to obtain fast rates for the approximation error.

In order to do that we need a stronger assumption on λ than the one made by A1.2 A 1.5. λ is a C ∞ non-negative integrable function whose inverse is also integrable.

(λ(x)ρ) * (-µ(x)) -( λ(b)ρ) * (-μ(b)) dx = O KdL 2 β ∇λ 2 ∞ B -2β-d ζλ 3 min .
For clarity reasons we postpone the proof to Appendix 1.A.1. The rate in B is improved compared to Proposition 1.7 at the expense of the constant 1/λ 3 min which can unfortunately be arbitrarily high.

Margin condition

We begin by giving a precise definition of the function η, the distance of the optimum to the boundary of ∆ K .

Definition 1.6. Let x ∈ X a context value. We define by p (x) ∈ ∆ K the point where (p → µ(x), p + λ(x)ρ(p)) attains its minimum, and

η(x) := dist(p (x), ∂∆ K ) . Similarly, if p b is the point where L b : p → μ(b), p + λ(b)ρ(p) attains its minimum, we define η(b) := dist(p b , ∂∆ K ) .
The fast rates obtained in Section 1.4.2 provide good theoretical guarantees but may be useless in practice since they depend on a constant that can be arbitrarily large. We would like to discard the dependency on the parameters, and especially λ (that controls η and S).

We begin by proving that η is β-Hölder continuous.

∈ B ∀(x, y) ∈ b 2 , |η(x) -η(y)| ≤ K K -1 λ ∞ + λ ∞ ζλ min (b) 2 x -y β = C L λ min (b) 2 x -y β . Proof. Let x ∈ X . Since η(x) = dist(p b , ∂∆ K ) we obtain η(x) = K K -1 min i p i (x) .
And

p (x) = arg min { µ(x), p(x) + λ(x)ρ(p(x))} = ∇(λ(x)ρ) * (-µ(x)) = ∇ρ * - µ(x) λ(x) . Since ρ is ζ-strongly convex, ∇ρ * is 1/ζ-Lipschitz continuous. Let b ∈ B. We have, for (x, y) ∈ b 2 , |p (x) -p (y)| ≤ 1 ζ µ(x) λ(x) - µ(y) λ(y) ≤ 1 ζ µ(x) -µ(y) λ(x) + 1 ζ |µ(y)| 1 λ(x) - 1 λ(y) ≤ 1 ζλ min (b) x -y β + 1 ζ λ ∞ λ min (b) 2 x -y ,
since all µ k are bounded by 1 (the losses are bounded by 1).

Difficulties arise when λ and η take values that are very small, meaning for instance that we consider nearly no regularization. This is not likely to happen since we do want to study contextual bandits with regularization. To formalize that we make an additional assumption, which is common in nonparametric regression [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF] and is known as a margin condition: A 1.6 (Margin Condition). We assume that there exist δ 1 > 0 and δ 2 > 0, α > 0 and

C m > 0 such that ∀δ ∈ (0, δ 1 ], P X (λ(x) < δ) ≤ C m δ 6α and ∀δ ∈ (0, δ 2 ], P X (η(x) < δ) ≤ C m δ 6α .
The non-negative parameter α controls the importance of the margin condition. The presence of a factor 6 is most likely a proof artifact.

The margin condition limits the number of bins on which λ or η can be small. Therefore we split the bins of B into two categories, the "well-behaved bins" on which λ and η are not too small, and the "ill-behaved bins" where λ and η can be arbitrarily small. The idea is to use the fast rates on the "well-behaved bin" and the slow rates (independent of λ and η) on the "ill-behaved bins". This is the point of Section 1.5.3.

Let C L = K K -1 λ ∞ + ∇λ ∞ ζ , c 1 = 1 + ∇λ ∞ d β/6 and c 2 = 1 + C L d β/2 .
We define the set of "well-behaved bins" WB as

WB = {b ∈ B, ∃ x 1 ∈ b, λ(x 1 ) ≥ c 1 B -β/3 and ∃ x 2 ∈ b, η(x 2 ) ≥ c 2 B -β/3 } ,
and the set of "ill-behaved bin" as its complementary set in B.

With the smoothness and regularity Assumptions 1.1 and 1.2, we derive lower bounds for λ and η on the "well-behaved bins".

Lemma 1.6. Assume A1.1 and A1.2 and that ρ is a

ζ-strongly convex function. If b is a well-behaved bin then ∀x ∈ b, λ(x) ≥ B -β/3 and ∀x ∈ b, η(x) ≥ B -β/3 .
Proof. We consider a well-behaved bin b. There exists

x 1 ∈ b such that λ(x 1 ) ≥ c 1 B -β/3 . Since λ is C ∞ on [0, 1] d , it is in particular Lipschitz-continuous on b. And therefore ∀x ∈ b, λ(x) ≥ c 1 B -β/3 -λ ∞ diam(b) ≥ c 1 B -β/3 -λ ∞ diam(b) β/3 = B -β/3 .
Lemma 1.5 shows that η is β-Hölder continuous (with constant denoted by C L /λ 2 min ) and therefore we have

∀x ∈ b, η(x) ≥ c 2 B -β/3 - C L λ min (b) 2 diam(b) β = B -β/3 .

Intermediate rates

We summarize the different error rates obtained in the previous sections.

B -d/2 log(T ) T log 2 (T ) T Sλ + 1 η 4 λ 2 Approx. B -d B -β B -2β-d λ 3 B T log(T ) 1 2β+d T log 2 (T ) 1 2β+d R(T ) T log(T ) -β 2β+d T log 2 (T ) -2β 2β+d
For the sake of clarity we removed the dependency on the bin, writing λ instead of λ(b), and we only kept the relevant constants, that can be very small (λ and η), or very large (S).

Table 1.1 shows that the slow rates do not depend on the constants, so that we can use them on the "ill-behaved bins".

Theorem 1.3 (Intermediate rates). Assume A1.1, A1.2, A1.6 with parameter α ∈ (0, 1) and that ρ is the entropy function. Applying Algorithm 1.1 with the choice B = Θ T / log 2 (T )

1 2β+d
gives the following bound of the regret for all T ≥ 1,

R(T ) = O K,d,α,β,L β T log 2 (T ) -β 2β+d (1+α)
.

As explained in the proof in Appendix 1.A.2 we use a pre-sampling stage on each bin to force the entropy to be smooth, as in the proofs of Propositions 1.2 and 1.5.

We consider now the extreme values of α. If α → 0, there is no margin condition and the speed obtained is T -β 2β+d which is exactly the slow rate from Theorem 1.1. If α → 1, there is a strong margin condition and the rate of Theorem 1.3 tends to T -2β 2β+d which is the fast rate from Theorem 1.2. Consequently we get that the intermediate rates from Theorem 1.3 do interpolate between the slow and fast rates obtained previously.

Lower bounds

The results in Theorems 1.1 and 1.2 have optimal exponents in the dependency in T . For the slow rate, since the regularization can be equal to 0, or a linear form, the lower bounds on contextual bandits in this setting apply [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF][START_REF] Rigollet | Nonparametric Bandits with Covariates[END_REF], matching this upper bound. For the fast rates, the following lower bound holds, based on a reduction to nonparametric regression [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF][START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF].

Theorem 1.4. For any algorithm with bandit input and output pT , for ρ that is 1-strongly convex, we have

inf p sup µ∈H β ρ∈1-str. conv. E[L(p T )] -L(p ) ≥ C T -2β 2β+d , for a universal constant C.
Proof. We consider the model with K = 2 where µ(x) = (-η(x), η(x)) , where η is a β-Hölder function on X = [0, 1] d . We note that η is uniformly bounded over X as a consequence of smoothness, so one can take λ such that |η(x)| < λ. We denote by e = (1/2, 1/2) the center of the simplex, and we consider the loss

L(p) = X µ(x), p(x) + λ p(x) -e 2 dx .
Denoting by p 0 (x) the vector e + µ(x)/(2λ), we have that p 0 (x) ∈ ∆ 2 for all x ∈ X . Further, we have that

µ(x), p(x) + λ p(x) -e 2 = λ p(x) -p 0 (x) 2 + 1/(4λ) µ(x) 2 ,
since µ(x), e = 0. As a consequence, L is minimized at p 0 and

L(p) -L(p 0 ) = X λ p(x) -p 0 (x) 2 dx = 1/(2λ) X |η(x) -η 0 (x)| 2 dx ,
where η is such that p(x) = 1/2 -η(x)/(2λ), 1/2 + η(x)/(2λ) . As a consequence, for any algorithm with final variable pT , we can construct an estimator ηT such that

E[L(p T )] -L(p 0 ) = 1/(2λ)E X |η T (x) -η 0 (x)| 2 dx ,
where the expectation is taken over the randomness of the observations Y t , with expectation ±η(X t ), with sign depending on the known choice π t = 1 or 2. As a consequence, any upper bound on the regret for a policy implies an upper bound on regression over β-Hölder functions in dimension d, with T observations. This yields that, in the special case where ρ is the 1-strongly convex function equal to the squared 2 -norm inf p sup

µ∈H β ρ = 2 2 E[L(p T )] -L(p 0 ) ≥ inf η sup η∈H β 1/(2λ)E X |η T (x) -η 0 (x)| 2 dx ≥ CT -2β 2β+d .
The final bound is a direct application of (Györfi et al., 2006, Theorem 3.2).

The upper and lower bound match up to logarithmic terms. This bound is obtained for K = 2, and the dependency of the rate in K is not analyzed here.

Empirical results

We present in this section experiments and simulations for the regularized contextual bandits problem. The setting we consider uses K = 3 arms, with an entropy regularization and a fixed parameter λ = 0.1. We run successive experiments for values of T ranging from 1 000 to 100 000, and for different values of the smoothness parameter β. The arms' rewards follow 3 different probability distributions (Poisson, exponential and Bernoulli), with β-Hölder mean functions.

The results presented in Figure 1.1 shows that (T → T • R(T )) growths as expected, and the lower β, the slower the convergence rate, as shown on the graph. 0 25,000 50,000 75,000 100,000 500 1,000

1,500 T R(T ) • T β = 0.3 β = 0.5 β = 0.7 β = 0.9 Figure 1.1 -Regret as a Function of T
In order to verify that the fast rates proven in Section 1.4.2 are indeed reached, we plot on Figure 1.2 the ratio between the regret and the theoretical bound on the regret T / log2 (T )

-2β
2β+d . We observe that this ratio is approximately constant as a function of T , which validates empirically the theoretical convergence rates.

Conclusion

We proposed an algorithm for the problem of contextual bandits with regularization reaching fast rates similar to the ones obtained in nonparametric estimation, and validated by our experiments. We can discard the parameters of the problem in the convergence rates by applying a margin condition that allows us to derive intermediate convergence rates interpolating perfectly between the slow and fast rates.

1.A Proof of the intermediate rates results

In this section we prove Proposition 1.8 and Theorem 1.3.

1.A.1 Proof of Proposition 1.8

Proof of Proposition 1.8. As in the proof of Proposition 1.6 we consider a bin b ∈ B and the goal is to bound

b λ(x)ρ * - µ(x) λ(x) -λ(b)ρ * - μ(b) λ(b) dx .
We use a similar method and we apply Jensen inequality with density

λ(x) |b| λ(b)
to the function

g : x → 1 2ζ x 2 -ρ * (x) which is convex. g b - µ(x) λ(x) λ(x) |b| λ(b) dx ≤ b g - µ(x) λ(x) λ(x) |b| λ(b) dx g - μ(b) λ(b) ≤ b g - µ(x) λ(x) λ(x) |b| λ(b) dx 1 2ζ - μ(b) λ(b) 2 -ρ * - μ(b) λ(b) ≤ 1 |b| λ(b) b 1 2ζ - µ(x) λ(x) 2 -ρ * - µ(x) λ(x) λ(x) dx b λ(x)ρ * - µ(x) λ(x) -λ(b)ρ * - μ(b) λ(b) dx ≤ 1 2ζ b µ(x) 2 λ(x) - μ(b)
Consequently we have proven that

b λ(x)ρ * - µ(x) λ(x) -λ(b)ρ * - μ(b) λ(b) dx ≤ 1 2ζ b µ(x) 2 λ(x) - μ(b) 2 λ(b) dx ≤ 1 2ζ K k=1 b µ k (x) 2 λ(x) - μk (b) 2 λ (b) dx .
Therefore we have to bound, for each

k, I = b µ k (x) 2 λ(x) - μk (b) 2 λ(b) dx.
Let us omit the subscript k and consider a β-Hölder function µ.

We have

I = b µ(x) 2 λ(x) - μ(b) 2 λ(b) dx = b µ(x) 2 λ(x) - µ(x) 2 λ(b) + µ(x) 2 λ(b) - μ(b) 2 λ(b) dx = b µ(x) 2 -μ(b) 2 1 λ(x) - 1 λ(b) dx I1 + b μ(b) 2 1 λ(x) - 1 λ(b) dx I2 + b 1 λ(b) µ(x) 2 -μ(b) 2 dx I3 .
We now have to bound these three integrals.

(a) Bounding I 1 :

I 1 = b µ(x) 2 -μ(b) 2 1 λ(x) - 1 λ(b) dx = b (µ(x) + μ(b)) (µ(x) -μ(b)) 1 λ(x) - 1 λ(b) dx ≤ b 2|µ(x) -μ(b)| 1 λ(x) - 1 λ(b) dx ≤ 2L β √ d B β b 1 λ(x) - 1 λ(b) dx .
Since 1/λ is of class C 1 , Taylor-Lagrange inequality yields, using the fact that there exists

x 0 ∈ b such that λ(b) = λ(x 0 ) 1 λ(x) - 1 λ(b) ≤ 1 λ ∞ |x -x 0 | ≤ λ ∞ λ 2 min √ d B .
We obtain therefore

I 1 ≤ 2L β λ ∞ √ d β+1 1 λ 2 min B -(1+β+d) = O B -(1+β+d) λ 2 min . (b) Bounding I 2 :
We have

I 2 = μ(b) 2 b 1 λ(x) - 1 λ(b) dx ≤ b 1 λ(x) - 1 λ(b) dx , because b 1 λ(x) - 1 λ(b)
dx ≥ 0 from Jensen's inequality.

Without loss of generality we can assume that the bin b is the closed cuboid [0, 1/B] d . We suppose that for all x ∈ b, λ(x) > 0.

Since λ is of class C ∞ , we have the following Taylor series expansion

λ(x) = λ(0) + d i=1 ∂λ(0) ∂x i x i + 1 2 i,j ∂ 2 λ(0) ∂x i ∂x j x i x j + O( x 2 ) .
Integrating over the bin b we obtain

λ(b) = λ(0) + 1 2 1 B d i=1 ∂λ(0) ∂x i + 1 8 1 B 2 i =j ∂ 2 λ(0) ∂x i ∂x j + 1 6 1 B 2 d i=1 ∂ 2 λ(0) ∂x 2 i + O 1 B 2 . Consequently b dx λ(b) = 1 B d λ(b) = 1 B d λ(0) 1 1 + 1 2λ(0) 1 B d i=1 ∂λ(0) ∂x i + 1 λ(0) 1 B 2   1 8 i =j ∂ 2 λ(0) ∂x i ∂x j + 1 6 d i=1 ∂ 2 λ(0) ∂x 2 i   + O 1 B 2 = 1 B d λ(0) 1 - 1 2λ(0) 1 B d i=1 ∂λ(0) ∂x i - 1 λ(0) 1 B 2   1 8 i =j ∂ 2 λ(0) ∂x i ∂x j + 1 6 d i=1 ∂ 2 λ(0) ∂x 2 i   + 1 4λ(0) 2 1 B 2 d i=1 ∂λ(0) ∂x i 2 + O 1 B 2 = 1 B d λ(0) - 1 2λ(0) 2 1 B d+1 d i=1 ∂λ(0) ∂x i - 1 λ(0) 2 1 B d+2   1 8 i =j ∂ 2 λ(0) ∂x i ∂x j + 1 6 d i=1 ∂ 2 λ(0) ∂x 2 i   + 1 4λ(0) 3 1 B d+2 d i=1 ∂λ(0) ∂x i 2 + O 1 B 2 .
Let us now compute the Taylor series development of 1/λ. We have

∂ ∂x i 1 λ(x) = - 1 λ(x) 2 ∂λ(x) ∂x i and ∂ 2 ∂x i ∂x j 1 λ(x) = - 1 λ(x) 2 ∂ 2 λ(x) ∂x i ∂x j + 2 λ(x) 3 ∂λ(x) ∂x i ∂λ(x) ∂x j .
This lets us write

1 λ(x) = 1 λ(0) - 1 λ(0) 2 d i=1 ∂λ(0) ∂x i x i - 1 2 1 λ(0) 2 i,j ∂ 2 λ(0) ∂x i ∂x j x i x j + 1 λ(0) 3 i,j ∂λ(0) ∂x i ∂λ(0) ∂x j x i x j + O( x 2 ) b dx λ(x) = 1 λ(0) 1 B d - 1 2λ(0) 2 1 B d+1 d i=1 ∂λ(0) ∂x i - 1 λ(0) 2 1 B d+2   1 8 i =j ∂ 2 λ(0) ∂x i ∂x j + 1 6 d i=1 ∂ 2 λ(0) ∂x 2 i   + 1 λ(0) 3 1 B d+2   1 4 i =j ∂λ(0) ∂x i ∂λ(0) ∂x j + 1 3 d i=1 ∂λ(0) ∂x i 2   + O 1 B d+2 .

And then

I 2 ≤ 1 12 1 λ(0) 3 1 B d+2 d i=1 ∂λ(0) ∂x i 2 + O 1 B d+2 .
Since the derivatives of λ are bounded we obtain that

I 2 = O B -2-d λ 3 min .
(c) Bounding I 3 :

I 3 = b 1 λ(b) µ(x) 2 -μ(b) 2 dx = 1 λ(b) b (µ(x) -μ(b)) 2 dx ≤ 1 λ min L 2 β d β B -(2β+d) = O B -(2β+d) λ min .
Putting I 1 , I 2 and I 3 together we have

I = O (dL 2 β ∇λ 2 ∞ ) B -(2β+d) λ 3 min
. And finally

L(p ) -L(p ) = O KdL 2 β ∇λ 2 ∞ B -2β ζλ 3 min .

1.A.2 Proof of Theorem 1.3

Before proving the theorem, we need a simple lemma.

Lemma 1.7. If ρ is convex, η is an increasing function of λ.

Proof. As in the proof of Proposition 1.2 we use the KKT conditions to find that on a bin b (without the index k for the arm):

μ(b) + λ(b)∇ρ(p b ) + ξ = 0 .
Therefore

p b = (∇ρ) -1 - ξ + μ(b) λ(b) .
Since ρ is convex, ∇ρ is an increasing function and its inverse as well. Consequently p b is an increasing function of λ(b), and since η(b) = K/(K -1) min i p b,i , η is also an increasing function of λ(b).

Proof of Theorem 1.3. Since B will be chosen as an increasing function of T we only consider T sufficiently large in order to have c 1 B -β/3 < δ 1 and c 2 B -β/3 < δ 2 . To ensure this we can also take smaller δ 1 and δ 2 . Moreover we lower the value of δ 2 or δ 1 to be sure that

δ2 c2 = η( δ1 c1
). These are technicalities needed to simplify the proof. The proof will be divided into several steps. We will first obtain lower bounds on λ and η for the "well-behaved bins". Then we will derive bounds for the approximation error and the estimation error. And finally we will put that together to obtain the intermediate convergence rates.

As in the proofs on previous theorems we will denote the constants C k with increasing values of k. We divide the rest of the proof into 4 steps.

(a) Lower bounds on η and λ: Using a technique from [START_REF] Rigollet | Nonparametric Bandits with Covariates[END_REF] we notice that without loss of generality we can index the B d bins with increasing values of λ(b). Let us note IB = {1, . . . , j 1 } and WB = {j 1 + 1, . . . , B d }. Since η is an increasing function of λ (cf Lemma 1.7), the η(b j ) are also increasingly ordered.

Let j 2 ≥ j 1 be the largest integer such that λ(b j ) ≤ δ 1 c 1

. Consequently we also have that j 2 is the largest integer such that η(b j ) ≤ δ 2 c 2 .

Let j ∈ {j 1 + 1, . . . , j 2 }. The bin b j is a well-behaved bin and Lemma 1.6 shows that λ(b j ) ≥ B -β/3 . Then λ(b j ) + (c 1 -1)B -β/3 ≤ c 1 λ(b j ) ≤ δ 1 and we can apply the margin condition (cf Assumption 1.6) which gives

P X (λ(x) ≤ λ(b j ) + (c 1 -1)B -β/3 ) ≤ C m (c 1 λ(b j )) 6α .
But since the context are uniformly distributed and since the λ(b j ) are increasingly ordered we also have that

P X (λ(x) ≤ λ(b j ) + (c 1 -1)B -β/3 ) ≥ P X (λ(x) ≤ λ(b j )) ≥ j B d .
This gives λ(b j ) ≥ 1

c 1 C 1/6α m j B d 1/6α
. The same computations give η(b j ) ≥ 1

c 2 C 1/6α m j B d 1/6α . We note C γ min((c 1 C 1/6α m ) -1 , (c 2 C 1/6α m ) -1 )) and γ j C γ j B d 1/α . Consequently λ(b j ) ≥ γ j and η(b j ) ≥ γ j .
Let us now compute the number of ill-behaved bins:

#{b ∈ B, b / ∈ WB} = B d P(b / ∈ WB) = B d P(∀x ∈ B, η(x) ≤ c 2 B -β/3 or ∀x ∈ B, λ(x) ≤ c 1 B -β/3 ) ≤ B d P(η(x) ≤ c 2 B -β/3 or λ(x) ≤ c 1 B -β/3 ) ≤ C m (c 6α 1 + c 6α 2 )B d B -2αβ C I B d B -2αβ ,
where x is the mean context value in the bin b.

Consequently if j ≥ j C I B d B -2αβ , then b j ∈ WB. Let ĵ C I B d B -αβ ≥ j . Consequently for all j ≥ j , b j ∈ WB.
We want to obtain an upper-bound on the constant S λ(b j ) + K η(b j ) 4 λ(b j ) 2 that arises in the fast rate for the estimation error. For the sake of clarity we will remove the dependency in b j and denote this constant C = Sλ + K λ 2 η 4 . In the case of the entropy regularization S = 1/ min i p i . Since η = K/(K -1) min i p i , we have that min i p i = (K -1)/Kη ≥ η/2. Consequently S ≤ 2/γ j and, on a well-behaved bin b j , for j ≤ j 2 ,

C ≤ K + 2 λ ∞ γ 6 j C F γ 6 j , (1.6)
where the subscript F stands for "Fast". When j ≥ j 2 , we have λ(b j ) ≥ δ 1 /c 1 and η(b j ) ≥ δ 2 /c 2 and consequently

C ≤ K (δ 1 /c 1 ) 2 (δ 2 /c 2 ) 4 + 2 λ ∞ δ 2 /c 2 C max .
Let us notice than λ being known by the agent, the agent knows the value of λ(b) on each bin b and can therefore order the bins. Consequently the agent can sample, on every well-behaved bin, each arm T γ j /2 times and be sure that min i p i ≥ γ j /2. On the first ĵ bins the agent will sample each arm λ(b) T /B d times as in the proof of Proposition 1.2.

(b) Approximation Error:

We now bound the approximation error. We separate the bins into two sets: {1, . . . , j } and { j , . . . , B d }. On the first set we use the slow rates of Proposition 1.7 and on the second set we use the fast rates of Proposition 1.8.

We obtain that, for α < 1/2,

L(p ) -L(p ) ≤ L β d β/2 j j=1 B -β-d + ρ ∞ ∇λ ∞ √ d j j=1 B -1-d + (KdL 2 β ∇λ 2 ∞ ) B d j= j B -2β-d λ(b j ) 3 ≤ C I L β d β/2 B -β B -2αβ + (KdL 2 β ∇λ 2 ∞ )   j2 j= j B -2β-d γ 3 j + B d j=j2+1 B -2β-d (c 1 /δ 1 ) 3   + O(B -2αβ-β ) ≤ C I L β d β/2 B -2αβ-β + (KdL 2 β ∇λ 2 ∞ )   B -2β-d C 3 γ j2 j= j j B d -1/2α + B -2β δ 1 c 1 3   + O(B -2αβ-β ) ≤ C I L β d β/2 B -2αβ-β + (KdL 2 β ∇λ 2 ∞ ) 1 C 3 γ B -2β 1 C I B -2αβ x -1/2α dx + O(B -2αβ-β ) ≤ C I L β d β/2 + KdL 2 β ∇λ 2 ∞ 2α 1 -2α C (2α-1)/2α I C 3 γ B -β-2αβ + O(B -2αβ-β ) = O B -β-2αβ ,
since α < 1/2. We step from line 3 to 4 thanks to a series-integral comparison. For α = 1/2 we get

L(p ) -L(p ) ≤ C I L β d β/2 + KdL 2 β ∇λ 2 ∞ (δ 3 1 c -3 1 + 2βC -3 γ log(B)) B -2β + O(B -2β ) = O B -2β log(B) .
And for α > 1/2 we have

L(p ) -L(p ) ≤ KdL 2 β ∇λ 2 ∞ 1 C 3 γ 2α 2α -1 + δ 1 c 1 3 B -2β + O(B -2β ) = O B -2β , because β + 2αβ > 2β. Let us note ξ 1 C I L β d β/2 + KdL 2 β ∇λ 2 ∞ 2α 1 -2α C (2α-1)/2α I C 3 γ ; ξ 2 C I L β d β/2 + KdL 2 β ∇λ 2 ∞ (δ 3 1 c -3 1 + 2βC -3 γ log(B)) ; ξ 3 KdL 2 β ∇λ 2 ∞ 1 C 3 γ 2α 2α -1 + δ 1 c 1 3 ; ξ app max(ξ 1 , ξ 2 , ξ 3 ) .
Finally we obtain that the approximation error is bounded by ξ app B -min(β+2αβ,2β) log(B) with α > 0.

(c) Estimation Error:

We proceed in a similar manner as for the approximation error, except that we do not split the bins around j but around ĵ.

In a similar manner to the proofs of Theorems 1.1 and 1.2 we only need to consider the terms of dominating order from Propositions 1.1 and 1.4. As before we consider the same event A (cf the proof of Proposition 1.1) and we note C A 4B d (1 + λρ ∞ ). We obtain, for α < 1, using (1.6):

EL(p T ) -L(p ) = 1 B d b∈B EL b (p T ) -L(p b ) = 1 B d B d j= ĵ EL b (p T ) -L(p b ) + 1 B d ĵ j=1 EL b (p T ) -L(p b ) ≤ 1 B d B d j= ĵ 2C log 2 (T ) T /B d + 1 B d ĵ j=1 4 √ 12K log(T ) T /B d + C A e -T 12B d ≤ 2C F j2 j= ĵ log 2 (T ) T γ -6 j + B d j=j2+1 2C max log 2 (T ) T + 6 √ 3K log(T ) T B d/2 B -αβ + C A e -T 12B d ≤ 2C F C 6 γ log 2 (T ) T j2 j= ĵ j B d -1/α + 2C max log 2 (T ) T B d + 6 √ 3K log(T ) T B d/2-αβ + C A e -T 12B d ≤ 2C F C 6 γ log 2 (T ) T B d 1 C I B -αβ x -1/α dx + 2C max log 2 (T ) T B d + 6 √ 3K log(T ) T B d/2-αβ + C A e -T 12B d ≤ 2C F C 6 γ log 2 (T ) T B d α 1 -α B β(1-α) + 2C max log 2 (T ) T B d + 6 √ 3K log(T ) T B d/2-αβ + C A e -T 12B d ≤ 2C F C 6 γ log 2 (T ) T α 1 -α B d+β-αβ + 6 √ 3K log(T ) T B d/2-αβ + 2C max log 2 (T ) T B d + C A e -T 12B d .
(d) Putting things together:

We note C α 2C F C 6 γ α 1 -α
. This leads to the following bound on the regret:

R(T ) ≤ C α log 2 (T ) T B d+β-αβ + 6 √ 3K log(T ) T B d/2-αβ + 2C max log 2 (T ) T B d + C A e -T 12B d + ξ app B -min(2β,β+2αβ) log(B) . Choosing B = T log 2 (T ) 1/(2β+d) we get R(T ) ≤ (C α + 6 √ 3K) T log 2 (T ) -β(1+α)/(2β+d) + O T log 2 (T ) -β(1+α)/(2β+d)
which is valid for α ∈ (0, 1). Finally we have

R(T ) = O T log 2 (T ) -β(1+α)/(2β+d)
.

2 Online A-optimal design and active linear regression

We consider in this chapter the problem of optimal experiment design where a decision maker can choose which points to sample to obtain an estimate β of the hidden parameter β of an underlying linear model. The key challenge of this work lies in the fact that we allow heteroscedasticity, meaning that each covariate can have a different and unknown variance. The goal of the decision maker is then to figure out on the fly the optimal way to allocate the total budget of T samples between covariates, as sampling several times a specific one will reduce the variance of the estimated model around it (but at the cost of a possible higher variance elsewhere). By trying to minimize the 2 -loss E[ β -β 2 ] the decision maker is actually minimizing the trace of the covariance matrix of the problem, which corresponds then to online A-optimal design. Combining techniques from bandit and convex optimization we propose a new active sampling algorithm and we compare it with existing ones. We provide theoretical guarantees of this algorithm in different settings, including a O(T -2 ) regret bound in the case where the covariates form a basis of the feature space, generalizing and improving existing results. Numerical experiments validate our theoretical findings1 .

Introduction and related work

A classical problem in statistics consists in estimating an unknown quantity, for example the mean of a random variable, parameters of a model, poll results or the efficiency of a medical treatment. In order to do that, statisticians usually build estimators which are random variables based on the data, supposed to approximate the quantity to estimate. A way to construct an estimator is to make experiments and to gather data on the estimand.

In the polling context an experiment consists for example in interviewing people in order to know their voting intentions. However if one wants to obtain a "good" estimator, typically an unbiased estimator with low variance, the choice of which experiment to run has to be done carefully. Interviewing similar people might indeed lead to a poor prediction. In this work we are interested in the problem of optimal design of experiments, which consists in choosing adequately the experiments to run in order to obtain an estimator with small variance. We focus here on the case of heteroscedastic linear models with the goal of actively constructing the design matrix. Linear models, though possibly sometimes too simple, have been indeed widely studied and used in practice due to their interpretability and can be a first good approximation model for a complex problem.

The original motivation of this problem comes from use cases where obtaining the label of a sample is costly, hence choosing carefully which points to sample in a regression task is crucial. Consider for example the problem of controlling the wear of manufacturing machines in a factory [START_REF] Antos | Active learning in heteroscedastic noise[END_REF], which requires a long and manual process. The wear can be modeled as a linear function of some features of the machine (age, number of times it has been used, average temperature, ...) so that two machines with the same parameters will have similar wears. Since the inspection process is manual and complicated, results are noisy and this noise depends on the machine: a new machine, slightly worn, will often be in a good state, while the state of heavily worn machines can vary a lot. Thus evaluating the linear model for the wear requires additional examinations of some machines and less inspection of others. Another motivating example comes from econometrics, typically in income forecasting. It is usually assumed that the annual income is influenced by the individual's education level, age, gender, occupation, etc. through a linear model. Polling is also an issue in this context: what kind of individual to poll to gain as much information as possible about an explanatory variable? Finally the setting we investigate is also relevant to the design of nuclear fusion experiments [START_REF] Stoian | Spatial and temporal laser pulse design for material processing on ultrafast scales[END_REF], which are costly, and require the parametrization of a large quantity of dynamic variables (to control the target quality and the temporal laser pulse shape). Using machine learning techniques to reach a controlled thermonuclear fusion can only be done on small size experiment history. It is therefore crucial to design the experiences in order to improve the predictive model in the best possible way.

The field of optimal experiment design [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF] aims precisely at choosing which experiment to perform in order to minimize an objective function within a budget constraint. In experiment design, the distance of the produced hypothesis to the true one is measured by the covariance matrix of the error [START_REF] Boyd | Convex optimization[END_REF]. There are several criteria that can be used to minimize a covariance matrix, the most popular being A, D and E-optimality. In this chapter we focus on A-optimal design whose goal is to minimize the trace of the covariance matrix. Contrary to several existing works which solve the A-optimal design problem in an offline manner in the homoscedastic setting [START_REF] Sagnol | Optimal design of experiments with application to the inference of traffic matrices in large networks: second order cone programming and submodularity[END_REF][START_REF] Yang | On optimal designs for nonlinear models: a general and efficient algorithm[END_REF][START_REF] Gao | Efficient computational algorithm for optimal allocation in regression models[END_REF] we are interested here in proposing an algorithm which solves this problem sequentially, with the additional challenge that each experiment has an unknown and different variance.

Our problem is therefore close to "active learning" which is more and more popular nowadays because of the exponential growth of datasets and the cost of labeling data. Indeed, the latter may be tedious and require expert knowledge, as in the domain of medical imaging. It is therefore essential to choose wisely which data to collect and to label, based on the information gathered so far. Usually, machine learning agents are assumed to be passive in the sense that the data is seen as a fixed and given input that cannot be modified or optimized. However, in many cases, the agent can be able to appropriately select the data [START_REF] Goos | Optimal design of experiments: a case study approach[END_REF]. Active learning specifically studies the optimal ways to perform data selection [START_REF] Cohn | Active learning with statistical models[END_REF] and this is crucial as one of the current limiting factors of machine learning algorithms are computing costs, that can be reduced since all examples in a dataset do not have equal importance [START_REF] Freund | Selective sampling using the query by committee algorithm[END_REF]. For example [START_REF] Bordes | Fast kernel classifiers with online and active learning[END_REF] proposed a SVM algorithm where example selection yields faster training and higher accuracy compared to classical passive SVM techniques. This approach has many practical applications: in online marketing where one wants to estimate the potential impact of new products on customers, or in online polling where the different options do not have the same variance [START_REF] Atkeson | The Oxford handbook of polling and survey methods[END_REF].

There exist different variants of active learning (perhaps depending on the different understandings of the word "active"). Maybe the most common one is the so-called "pool-based" active learning [START_REF] Mccallumzy | Employing EM and pool-based active learning for text classification[END_REF], where the decision maker has access to a pool of examples and chooses which one to query and to label. Another variant is the "retraining-based" active learning [START_REF] Yang | Active learning using uncertainty information[END_REF] whose principle is to retrain the model on well-chosen examples, for instance the ones that had the higher uncertainty. [START_REF] Castro | Minimax bounds for active learning[END_REF] have proven general minimax bounds for active learning, for a general class of functions, with rates depending on noise conditions and on the regularity of the decision boundary (see also [START_REF] Tosh | Diameter-Based Active Learning[END_REF]; [START_REF] Hanneke | Minimax analysis of active learning[END_REF]).

In this chapter we consider therefore a decision maker who has a limited experimental budget of T ≥ 1 samples and who aims at learning some latent linear model. The goal is to build a predictor β that estimates the unknown parameter of the linear model β , and that minimizes E[ β -β 2 ]. The key point here is that the design matrix is constructed sequentially and actively by the agent: at each time step, the decision maker chooses a "covariate" X k ∈ R d and receives a noisy output X k β + ε. The quality of the predictor is measured through its variance. The agent will repeatedly query the different available covariates in order to obtain more precise estimates of their values. Instinctively a covariate with small variance should not be sampled too often since its value is already quite precise. On the other hand, a noisy covariate will be sampled more often. The major issue lies in the heteroscedastic assumption: the unknown variances must be learned to wisely sample the points. [START_REF] Antos | Active learning in heteroscedastic noise[END_REF] introduced a specific variant of our setting where the environment providing the data is assumed to be stochastic and i.i.d. across rounds. More precisely, they studied this problem using the framework of stochastic multi-armed bandits (MAB) by considering a set of K probability distributions (or arms), associated with K variances. Their objective is to define an allocation strategy over the arms to estimate their expected values uniformly well. Later, the analysis and results have been improved by [START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF]. However, this line of work is actually focusing on the case where the covariates are only vectors of the canonical basis of R d , which gives a simpler closed form linear regression problem.

There have been some recent works on MAB with heteroscedastic noise [START_REF] Cowan | Normal bandits of unknown means and variances[END_REF][START_REF] Kirschner | Information Directed Sampling and Bandits with Heteroscedastic Noise[END_REF] with natural connections to this chapter. Indeed, covariates could somehow be interpreted as contexts in contextual bandits. The most related setting might be the one of [START_REF] Soare | Sequential Resource Allocation in Linear Stochastic Bandits[END_REF]. However, they are mostly concerned about best-arm identification while recovering the latent parameter β of the linear model is a more challenging task (as each decision has an impact on the loss). In that sense we improve the results of [START_REF] Soare | Sequential Resource Allocation in Linear Stochastic Bandits[END_REF] by proving a bound on the regret of our algorithm. Other works as [START_REF] Chen | Active Regression via Linear-Sample Sparsification[END_REF] propose active learning algorithms aiming at finding a constant factor approximation of the classification loss while we are focusing on the statistical problem of recovering β . Yet another similar setting has been introduced in (Riquelme et al., 2017a). In this setting the agent has to estimate several linear models in parallel and for each covariate (that appears randomly), the agent has to decide which model to estimate. Other works studied the problem of active linear regression, and for example [START_REF] Sugiyama | Active learning with model selection in linear regression[END_REF] proposed an algorithm conducting active learning and model selection simultaneously but without any theoretical guarantees. More recently [START_REF] Riquelme | Online active linear regression via thresholding[END_REF] have studied the setting of active linear regression with thresholding techniques in the homoscedastic case. An active line of research has also been conducted in the domain of random design linear regression [START_REF] Hsu | An analysis of random design linear regression[END_REF][START_REF] Sabato | Active Regression by Stratification[END_REF][START_REF] Dereziński | Unbiased estimators for random design regression[END_REF]. In these works the authors aim at controlling the mean-squared regression error E[(X β -Y ) 2 ] with a minimum number of random samples X k . Except from the loss function that they considered, these works differ from ours in several points: they generally do not consider the heteroscedastic case and their goal is to minimize the number of samples to use to reach an ε-estimator while in our setting the total number of covariates K is fixed. Allen-Zhu et al. ( 2020) provide a similar analysis but under the scope of optimal experiment design. Another setting similar to ours is introduced in [START_REF] Hazan | Hard-margin active linear regression[END_REF], where active linear regression with a hard-margin criterion is studied. However, the minimization of the classical 2 -norm of the difference between the true parameter of the linear model and its estimator seems to be a more natural criterion, which justifies our investigations.

In this work we adopt a different point of view from the aforementioned existing works. We consider A-optimal design under the heteroscedasticity assumption and we generalize MAB results to the non-coordinate basis setting with two different algorithms taking inspiration from the convex optimization and bandit literature. We prove optimal O(T -2 ) regret bounds for d covariates and provide a weaker guarantee for more than d covariates. Our work emphasizes the connection between MAB and optimal design, closing open questions in A-optimal design. Finally we corroborate our theoretical findings with numerical experiments.

The remainder of this chapter is organized as follows. We describe the setting of our problem in Section 2.2, then we present a naive algorithm in Section 2.3 and a faster algorithm in Section 2.4. We discuss the case K > d in Section 2.5 and present numerical simulations in Section 2.6. Finally Section 2.7 concludes the chapter. Appendix 2.A contains the postponed proofs.

Setting and description of the problem

Motivations and description of the setting

Let X 1 , . . . , X K ∈ R d be K covariates available to some agent who can successively sample each of them (several times if needed). Observations Y are generated by a standard linear model, i.e.,

Y = X β + ε with β ∈ R d .
Each of these covariates correspond to an experiment that can be run by the decision maker to gain information about the unknown vector β . The goal of optimal experiment design is to choose the experiments to perform from a pool of possible design points {X 1 , . . . , X K } in order to obtain the best estimate β of β within a fixed budget of T ∈ N * samples. In classical experiment design problems the variances of the different experiments are supposed to be equal. Here we consider the more challenging setting where each covariate has a specific and unknown variance σ 2 k , i.e., we suppose that when X k is queried for the i-th time the decision maker observes

Y (i) k = X k β + ε (i) k , where E[ε (i) k ] = 0, Var[ε (i) k ] = σ 2 k > 0 and ε (i)
k is κ2 -subgaussian. We assume also that the ε (i) k are independent from each other. This setting corresponds actually to online optimal experiment design since the decision maker has to design sequentially the sampling policy, in an adaptive manner.

A naive sampling strategy would be to sample the covariates X k with the static homoscedastic proportions. In our heteroscedastic setting, this will not produce the most precise estimate of β because of the different variances σ 2 k . Intuitively a point X k with a low variance will provide very precise information on the value X k β while a point with a high variance will not give much information (up to the converse effect of the norm X k ). This indicates that a point with high variance should be sampled more often than a point with low variance. Since the variances σ 2 k are unknown, we need at the same time to estimate σ 2 k (which might require lots of samples of X k to be precise) and to minimize the estimation error (which might require only a few examples of some covariate X k ). There is then a trade-off between gathering information on the values of σ 2 k and using it to optimize the loss; the fact that this loss is global, and not cumulative, makes this trade-off "exploration vs. exploitation" much more intricate than in standard multi-armed bandits.

Usual algorithms handling global losses are rather slow [START_REF] Agrawal | Bandits with concave rewards and convex knapsacks[END_REF][START_REF] Mannor | Approachability in unknown games: Online learning meets multi-objective optimization[END_REF] or dedicated to specific well-posed problems with closed form losses [START_REF] Antos | Active learning in heteroscedastic noise[END_REF][START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF]. Our setting can be seen as an extension of the two aforementioned works that aim at estimating the means of a set of K distributions. Noting µ = (µ 1 , . . . , µ K ) the vector of the means of those distributions and X i = e i the i th vector of the canonical basis of R K , we see (since X i µ = µ i ) that their objective is actually to estimate the parameter µ of a linear model. This setting is a particular case of ours since the vectors X i form the canonical basis of R K .

Definition of the loss function

As we mentioned it before, the decision maker can be led to sample several times the same design point X k in order to obtain a more precise estimate of its response X k β . We denote therefore by T k ≥ 0 the number of samples of X k , hence T = K k=1 T k . For each k ∈ [K] 2 , the linear model yields the following

T -1 k T k i=1 Y (i) k = X T k β + T -1 k T k i=1 ε (i) k . We define Ỹk = T k i=1 Y (i) k /σ k √ T k , Xk = √ T k X k /σ k and εk = T k i=1 ε (i) k /σ k √ T k so that for all k ∈ [K], Ỹk = XT k β + εk , where E[ε] = 0 and Var[ε k ] = 1. We denote by X = ( X 1 , • • • , X K ) ∈ R
K×d the induced design matrix of the policy. Under the assumption that X has full rank, the above Ordinary Least Squares (OLS) problem has an optimal unbiased estimator β = (X X) -1 X Ỹ . The overarching objective is to upperbound E[ β -β 2 ], which can be easily rewritten as follows:

E β -β 2 = Tr((X X) -1 ) = Tr K k=1 Xk X k -1 = 1 T Tr K k=1 p k X k X k /σ 2 k -1
, where we have denoted for every k ∈ [K], p k = T k /T the proportion of times the covariate X k has been sampled. By definition, p = (p 1 , . . . , p K ) ∈ ∆ K , the simplex of dimension K -1. We emphasize here that minimizing E[ β -β 2 ] is equivalent to minimizing the trace of the inverse of the covariance matrix X X, which corresponds actually to A-optimal design [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF]. Denote now by Ω(p) the following weighted matrix

Ω(p) = K k=1 p k σ 2 k X k X k = X X .
The objective is to minimize over p ∈ ∆ K the loss function

L(p) = Tr Ω(p) -1 with L(p) = +∞ if (p → Ω(p)) is not invertible, such that E β -β 2 = 1 T Tr Ω(p) -1 = 1 T L(p) .
For the problem to be non-trivial, we require that the covariates span R d . If it is not the case then there exists a vector along which one cannot get information about the parameter β . The best algorithm we can compare against can only estimate the projection of β on the subspace spanned by the covariates, and we can work in this subspace. The rest of this work is devoted to design an algorithm minimizing Tr Ω(p) -1 with the difficulty that the variances σ 2 k are unknown. In order to do that we will sequentially and adaptively choose which point to sample to minimize Tr Ω(p) -1 . This corresponds consequently to online A-optimal design. As developed above, the norms of the covariates have a scaling role and those can be renormalized to lie on the sphere at no cost, which is thus an assumption from now on: ∀k ∈ [K], X k 2 = 1. The following proposition shows that the problem we are considering is convex.

Proposition 2.1. L is strictly convex on ∆ d and continuous in its relative interior ∆d .

Proof. Let p, q ∈ ∆d , so that Ω(p) and Ω(q) are invertible, and λ ∈ [0, 1]. We have L(p) = Tr(Ω(p) -1 ) and L(λp + (1 -λ)q) = Tr(Ω(λp + (1 -λ)q) -1 ), where

Ω(λp + (1 -λq)) = d k=1 λp k + (1 -λ)q k σ 2 k X k X k = λΩ(p) + (1 -λ)Ω(q).
It is well-known [START_REF] Whittle | A multivariate generalization of Tchebichev's inequality[END_REF]) that the inversion is strictly convex on the set of positive definite matrices. Consequently,

Ω(λp + (1 -λq)) -1 = (λΩ(p) + (1 -λ)Ω(q)) -1 ≺ λΩ(p) -1 + (1 -λ)Ω(q) -1 . 3
Taking the trace this gives

L(λp + (1 -λ)q) < λL(p) + (1 -λ)L(q).
Hence L is strictly convex.

Proposition 2.1 implies that L has a unique minimum p in ∆d and we note p = arg min

p∈∆ d L(p) .
Finally, we evaluate the performance of a sampling policy in term of "regret" i.e., the difference in loss between the optimal sampling policy and the policy in question.

Definition 2.1. Let p T denote the sampling proportions after T samples of a policy. Its regret is then

R(T ) = 1 T (E [L(p T )] -L(p )) .
3 where ≺ denotes the strict Loewner ordering between symmetric matrices.

We will construct active sampling algorithms to minimize R(T ). A key step is the following computations of the gradient of L.

Since ∇ k Ω(p) = X k X T k /σ 2 k , it follows ∂ p k L(p) = - 1 σ 2 k Tr Ω(p) -2 X k X T k = - 1 σ 2 k Ω(p) -1 X k 2 2 .
As in several works [START_REF] Hsu | An analysis of random design linear regression[END_REF][START_REF] Allen-Zhu | Near-optimal discrete optimization for experimental design: A regret minimization approach[END_REF] we will have to study different cases depending on the values of K and d. The first one corresponds to the case K ≤ d.

As we explained it above, if K < d, the matrix Ω(p) is not invertible and it is impossible to obtain a sublinear regret, which makes us work in the subspace spanned by the covariates X k . This corresponds to K = d. We will treat this case in Sections 2.3 and 2.4. The case K > d is considered in Section 2.5.

Concentration arguments

Before going on with algorithms to solve the problem described in Section 2.2.2, we present results on the concentration of the variance for subgaussian random variables.

Traditional results on the concentration of the variances [START_REF] Maurer | Empirical Bernstein Bounds and Sample-Variance Penalization[END_REF][START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF] are obtained in the bounded setting. We propose results in a more general framework. Let us begin with some definitions.

Definition 2.2 (Sub-gaussian random variable). A random variable X is said to be κ

2 - sub-gaussian if ∀λ ≥ 0, exp(λ(X -EX)) ≤ exp(λ 2 κ 2 /2) .
And we define its ψ 2 -norm as

X ψ 2 = inf t > 0 | E[exp(X 2 /t 2 )] ≤ 2 .
We can bound the ψ 2 -norm of a subgaussian random variable as stated in the following lemma.

Lemma 2.1 (ψ 2 -norm). If X is a centered κ 2 -sub-gaussian random variable then

X ψ 2 ≤ 2 √ 2 √ 3 κ .
Proof. A proposition stated in [START_REF] Wainwright | High-dimensional statistics: A non-asymptotic viewpoint[END_REF] shows that for all λ ∈ [0, 1), a sub-gaussian variable X verifies

E λX 2 2κ 2 ≤ 1 √ 1 -λ .
Taking λ = 3/4 and defining u = 2

√ 2 √ 3 κ gives E(X 2 /u 2 ) ≤ 2 . Consequently X ψ2 ≤ u.
A wider class of random variables is the class of sub-exponential random variables that are defined as follows.

Definition 2.3 (Sub-exponential random variable). A random variable X is said to be sub-exponential if there exists K > 0 such that

∀ 0 ≤ λ ≤ 1/K, E[exp(λ|X|)] ≤ exp(Kλ) .
And we define its ψ 1 -norm as

X ψ 1 = inf {t > 0 | E[exp(|X|/t)] ≤ 2} .
A result from [START_REF] Vershynin | High-dimensional probability: An introduction with applications in data science[END_REF] gives the following lemma, that makes a connection between subgaussian and subexponential random variables.

Lemma 2.2. A random variable X is sub-gaussian if and only if X 2 is sub-exponential, and we have X 2

ψ 1 = X 2 ψ 2 .
We now want to obtain a concentration inequality on the empirical variance of a subgaussian random variable. We give use the following notations to define the empirical variance.

Definition 2.4. We define the following quantities for n i.i.d. repetitions of the random variable X.

µ = E[X] and μn = 1 n n i=1 X i , µ (2) = E[X 2 ] and μ(2) n = 1 n n i=1 X 2 i .
The variance and empirical variance are defined as follows

σ 2 = µ (2) -µ 2 and σ2 n = μ(2) n -μ2 n .
We are now able to state the main result of this section.

Theorem 2.1. Let X be a centered and κ 2 -sub-gaussian random variable sampled n ≥ 2 times. Let δ ∈ (0, 1). Let c = (e -1)(2e(2e -1)) -1 ≈ 0.07. With probability at least 1 -δ, the following concentration bound on its empirical variance holds

σ2 n -σ 2 ≤ 3κ 2 • max   log(4/δ) cn , log(4/δ) cn   .
This theorem provides a concentration result on the empirical variance of a subgaussian random variable, whereas usual concentration bounds are generally obtained for bounded random variables [START_REF] Maurer | Empirical Bernstein Bounds and Sample-Variance Penalization[END_REF][START_REF] Carpentier | Upperconfidence-bound algorithms for active learning in multi-armed bandits[END_REF], for which the concentration bound is easier to obtain.

Proof. We have

σ2 n -σ 2 = μ(2) n -μ2 n -(µ (2) -µ 2 ) ≤ μ(2) n -µ (2) + μ2 n -µ 2 ≤ μ(2) n -µ (2) + |μ n -µ||μ n + µ| ≤ μ(2) n -µ (2) + |μ n | 2 since µ = 0.
We now apply Hoeffding's inequality [START_REF] Vershynin | High-dimensional probability: An introduction with applications in data science[END_REF] to the X t variables that are κ 2subgaussian, to get

P 1 n n i=1 X i -µ > t ≤ exp - n 2 t 2 2nκ 2 = exp - nt 2 2κ 2 .

96

And finally

P |μ n -µ| > κ 2 log(2/δ) n ≤ δ.
Consequently with probability at least 1 -δ,

|μ n | 2 ≤ 2κ 2 log(2/δ) n .
The variables X 2 t are sub-exponential random variables. We can apply Bernstein's inequality as stated in [START_REF] Chafaï | Interactions between compressed sensing random matrices and high dimensional geometry[END_REF] to get for all t > 0:

P 1 n n i=1 X 2 i -µ (2) > t ≤ 2 exp -cn min t 2 s 2 , t m ≤ 2 exp -cn min t 2 m 2 , t m . with c = e-1 2e(2e-1) , s 2 = 1 n n i=1 X 2 i ψ1 ≤ m 2 and m = max 1≤i≤n X 2 i ψ1 .
Inverting the inequality we obtain

P μ(2) n -µ (2) > m • max log(2/δ) cn , log(2/δ) cn ≤ δ.
And finally, with probability at least 1 -δ,

σ2 n -σ 2 ≤ m • max log(4/δ) cn , log(4/δ) cn + 2κ 2 log(4/δ) n .
Using Lemmas 2.2 and 2.1 we obtain that m ≤ 8κ 2 /3. Finally,

σ2 n -σ 2 ≤ 8 3 κ 2 • max log(4/δ) cn , log(4/δ) cn + 2cκ 2 log(4/δ) cn ≤ 3κ 2 • max log(4/δ) cn , log(4/δ) cn ,
since 2c ≤ 1/3. This gives the expected result.

We now state a corollary of this result.

Corollary 2.1. Let T ≥ 2. Let X be a centered and κ 2 -sub-gaussian random variable.

Let c = (e -1)(2e(2e -1)) -1 ≈ 0.07. For n = 72κ 4 cσ 4 log(2T ) , we have with probability

at least 1 -1/T 2 , σ2 n -σ 2 ≤ 1 2 σ 2 . Proof. Let δ ∈ (0, 1). Let n = log(4/δ) c 6κ 2 σ 2 2 . Then log(4/δ) cn ≤ σ 2 6κ 2 2 < 1, since σ 2 ≤ κ 2
, by property of subgaussian random variables.

With probability 1 -δ, Theorem 2.1 gives

|σ 2 n -σ 2 | ≤ 3κ 2 σ 2 6κ 2 ≤ 1 2 σ 2 .
Now, suppose that δ = 1/T 2 . Then, with probability 1 -1/T 2 , for n = 72κ 4 cσ 4 log(2T ) samples,

|σ 2 n -σ 2 | ≤ 1 2 σ 2 .

A naive randomized algorithm

We begin by proposing an obvious baseline for the problem at hand. One naive algorithm would be to estimate the variances of each of the covariates by sampling them a fixed amount of time. Sampling each arm cT times (with c < 1/K) would give an approximation σk of σ k of order 1/ √ T . Then we can use these values to construct Ω(p) an approximation of Ω(p) and then derive the optimal proportions pk to minimize Tr( Ω(p) -1 ). Finally the algorithm would consist in using the remainder of the budget to sample the arms according to those proportions. However, such a trivial algorithm would not provide good regret guarantees. Indeed the constant fraction c of the samples used to estimate the variances has to be chosen carefully; it will lead to a 1/T regret if c is too big (if c > p k for some k). That is why we need to design an algorithm that will first roughly estimate the p k . In order to improve the algorithm it will also be useful to refine at each iteration the estimates pk . Following these ideas we propose Algorithm 2.1 which uses a pre-sampling phase (see Lemma 2.6 for further details) and which constructs at each iteration lower confidence estimates of the variances, providing an optimistic estimate L of the objective function L. Then the algorithm minimizes this estimate (with an offline A-optimal design algorithm, see e.g., [START_REF] Gao | Efficient computational algorithm for optimal allocation in regression models[END_REF]). Finally the covariate X k is sampled with probability pt,k . Then feedback is collected and estimates are updated. Compute pt ∈ arg min L, where L is the same function as L, but with variances replaced by lower confidence estimates of the variances (from Theorem 2.1).

6:

Draw π(t) randomly according to probabilities pt and sample covariate X π(t)

7:

Update p t+1 = p t + 1 t+1 (e π(t+1) -p t ) and σ2

R(T ) = O Γ,σ k √ log T T 3/2 ,
where Γ is the Gram matrix of X 1 , . . . , X K .

Notice that we avoid the problem discussed by [START_REF] Erraqabi | Trading off Rewards and Errors in Multi-Armed Bandits[END_REF] (that is due to infinite gradient on the simplex boundary) thanks to presampling, allowing us to have positive empirical variance estimates with high probability.

Proof. We now conduct the analysis of Algorithm 2.1. Our strategy will be to convert the error L(p T ) -L(p ) into a sum over t ∈ [T ] of small errors. Notice first that for i ∈ [K], the quantity Ω(p) -1 X i can be upper bounded by

1 σ i λ min (Γ) max k∈[K] σ 2 k 0.
5p o , for p = p T , where we have denoted by Γ the Gram matrix of X 1 , . . . , X K and where λ min (Γ) denotes the smallest eigenvalue of Γ.

For p = pt , we can also bound this quantity by 4 σ i λ min (Γ) max k∈[K] σ 2 k 0.5p o , using Lemma 2.6 to express pt with respect to lower estimates of the variances -and thus with respect to real variance thanks to Corollary 2.1. Then using the convexity of L we have

L(p T ) -L(p ) = L(p T ) -L 1/T T t=1 pt + L 1 T T t=1 pt -L(p ) ≤ k -Ω(p T ) -1 X k σ k 2 2 p k,T - 1 T T t=1 pk,t + 1 T T t=1 (L(p t ) -L(p )) .
Using Hoeffding inequality,

p k,T -1 T T t=1 pk,t = 1 T T t=1 (I{k is sampled at t} -pk,t ) is bounded by log(2/δ) T
with probability 1 -δ. It thus remains to bound the second term

1 T T t=1 (L(p t ) -L(p )).
First, notice that L(p) is an increasing function of σ i for any i. If we define L be replacing each σ 2 i by lower confidence estimates of the variances σ2 i (see Theorem 2.1), then

L(p t ) -L(p ) ≤ L(p t ) -L(p ) = L(p t ) -L(p t ) + L(p t ) -L(p * ) ≤ L(p t ) -L(p t ).
Since the gradient of L with respect to σ 2 is 2pi

σ 3 i Ω(p) -1 X i 2 2 i , we can bound L(p t ) -L(p t ) by 1/σ 3 min sup k Ω(p t ) -1 X k 2 2 i 2p i,t |σ 2 i -σ2 i | .
Since pi,t is the probability of having a feedback from covariate i, we can use the probabilistically triggered arm setting of [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF] to prove that 1 T

T t=1 i 2p i |σ 2 i -σ2 i | = O log(T ) T
. Taking δ of order T -1 gives the desired result.

A faster first-order algorithm

We now improve the relatively "slow" dependency in T in the rates of Algorithm 2.1due to its naive reduction to a MAB problem, and because it does not use any estimates of the gradient of L -with a different approach based on convex optimization techniques, that we can leverage to gain an order in the rates of convergence.

Description of the algorithm

The main algorithm is described in Algorithm 2.2 and is built following the work of [START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF]. The idea is to sample the arm which minimizes a proxy of the gradient of L corrected by a negative error term, as in the UCB algorithm [START_REF] Auer | Finite-time Analysis of the Multiarmed Bandit Problem[END_REF].

Algorithm 2.2 Bandit algorithm

Require: K, T Require: N 1 , . . . , N K of sum N 1: Sample N k times each covariate X k 2: p N ←-(N 1 /N, . . . , N K /N ) 3: Compute empirical variances σ2 1 , . . . , σ2 K 4: for N + 1 ≤ t ≤ T do 5:
Compute ∇ L(p t ), where L is the same function as L, but with variances replaced by empirical variances.

6:

for k ∈ [K] do 7: ĝk ←-∇ k L(p t ) -2 3 log(t) T k 8:
π(t) ←-arg min k∈ [d] ĝk and sample covariate X π(t)

9:

Update p t+1 = p t + 1 t+1 (e π(t+1) -p t ) and update σ2

π(t)
N 1 , . . . , N K are the number of times each covariate is sampled at the beginning of the algorithm. This stage is needed to ensure that L is smooth. More details about that will be given with Lemma 2.6.

Concentration of the gradient of the loss

The cornerstone of the algorithm is to guarantee that the estimates of the gradients concentrate around their true value. To simplify notations, we denote by G k = ∂ p k L(p) the true k th derivative of L and by Ĝk its estimate. More precisely, if we note Ω

(p) = K k=1 ( p k /σ k )X k X k , we have G k = -σ -2 k Ω(p) -1 X k 2 2 and Ĝk -σ -2 k Ω(p) -1 X k 2 2 .
Since Ĝk depends on the σ2 k , we need a concentration bound on the empirical variances of sub-gaussian random variables.

Using Theorem 2.1 we claim the following concentration argument, which is the main ingredient of the analysis of Algorithm 2.2. Proposition 2.3. For every k ∈ [K], after having gathered T k ≤ T samples of covariates X k , there exists a constant C > 0 (explicit and given in the proof) such that, with probability at least 1 -δ

|G k -Ĝk | ≤ C σ -1 k max i∈[K] σ 2 i p i 3 • max   log(4T K/δ) T k , log(4T K/δ) T k   .
For clarity reasons we postpone the proof to Appendix 2.A. Proving this proposition was one of the main technical challenges of our analysis. Now that we have it proven we can turn to the analysis of Algorithm 2.2.

Analysis of the convergence of the algorithm

In convex optimization several classical assumptions can be leveraged to derive fast convergence rates. Those assumptions are typically strong convexity, positive distance from the boundary of the constraint set, and smoothness of the objective function, i.e., that it has Lipschitz gradient. We prove in the following that the loss L satisfies them, up to the smoothness because its gradient explodes on the boundary of ∆ d . However, L is smooth on the relative interior of the simplex. Consequently we will circumvent this smoothness issue by using a technique from Chapter 1 consisting in pre-sampling every arm a linear number of times in order to force p to be far from the boundaries of ∆ d .

We denote X 0 (X 1 , • • • , X d ) and Γ X 0 X 0 = Gram(X 1 , . . . , X d ). Noting also Cof(M ) ij the (i, j) cofactor of a matrix M and Com(M ) the comatrix (matrix of cofactors) of M , we prove the following lemmas.

Lemma 2.3. The diagonal coefficients of Ω(p) -1 can be computed as follows:

∀i ∈ [d], Ω(p) -1 ii = d j=1 σ 2 j Cof(X 0 ) 2 ij det(X 0 X 0 ) 1 p j .
Proof. We suppose that ∀i ∈ [d], p i = 0 so that Ω(p) is invertible.

We know that Ω(p) -1 = Com(Ω(p)) det(Ω(p)) . We compute now det(Ω(p)).

det(Ω(p)) = det

d k=1 p k X k X k σ 2 k = det(( √ T -1 X) √ T -1 X) = T -d det(X ) 2 = T -d . . . X1 . . . Xd . . . 2 = . . . √ p 1 σ 1 X 1 . . . √ p d σ d X d . . . 2 = det(X 0 ) 2 p 1 σ 2 1 • • • p d σ 2 d .
We now compute Com(Ω(p)) ii .

Com(Ω(p)) = Com(T -1/2 X T -1/2 X) = Com(T -1/2 X ) Com(T -1/2 X ) . Let us note M T -1/2 X =       • • • √ p 1 σ 1 X 1 • • • . . . • • • √ p K σ K X K • • •       . Therefore Com(Ω(p)) ii = d j=1 Com(M ) 2 ij = d j=1 k =j p k σ 2 k Cof(X 0 ) 2 ij .
Finally,

Ω(p) -1 ii = d j=1 σ 2 j Cof(X 0 ) 2 ij det(X 0 X 0 ) 1 p j .
This allows us to derive the exact expression of the loss function L.

Lemma 2.4. The loss function L verifies for all p ∈ ∆ d ,

L(p) = 1 det(X 0 X 0 ) d k=1 σ 2 k p k Cof(X 0 X 0 ) kk .
101

Proof. Using Lemma 2.3 we obtain

L(p) = Tr(Ω(p) -1 ) = d k=1 Ω(p) -1 kk = 1 det(X X) d k=1 σ 2 k p k d i=1 Cof(X 0 ) 2 ik = 1 det(X 0 X 0 ) d k=1 σ 2 k p k Com(X 0 X 0 ) kk .
With this expression, the optimal proportion p can be easily computed using the KKT theorem, with the following closed form:

p k = σ k Cof(Γ) kk / d i=1 σ i Cof(Γ) ii .
(2.1)

This yields that L is strongly convex on ∆ d , with strong convexity parameter

µ = 2 det(Γ) -1 min i Cof(Γ) ii σ 2 i .
Moreover, this also implies that p is far away from the boundary of ∆ d .

Lemma 2.5. Let η dist(p , ∂∆ d ) be the distance from p to the boundary of the simplex. Then

η = K K -1 min i σ i Cof(Γ) ii d k=1 σ k Cof(Γ) kk .
Proof. This is immediate with (2.1) since η = K K -1 min i p i .

It remains to recover the smoothness of L. This is done using a pre-sampling phase. We have proved that p k is bounded away from 0 and thus a pre-sampling would be possible. However, this requires to have some estimate of each σ 2 k . The upside is that those estimates must be accurate up to some multiplicative factor (and not additive factor) so that a logarithmic number of samples of each arm is enough to get valid lower/upper bounds (see Corollary 2.1). Indeed, the estimate σ 2 k obtained satisfies, for each k ∈

[d], that σ 2 k ∈ [σ 2 k /2, 3σ 2 k /2]. Consequently we know that ∀k ∈ [d], p k ≥ 1 √ 3 σ k Cof(Γ) kk d i=1 σ i Cof(Γ) ii ≥ 1 2 p o , where p o = σ k Cof(Γ) kk d i=1 σ i Cof(Γ) ii . (2.2)
This will let us use Lemma 2.6 and with a presampling stage as prescribed, p is forced to remain far away from the boundaries of the simplex in the sense that p t,i ≥ p o i /2 at each stage t subsequent to the pre-sampling, and for all i ∈ [d]. Consequently, this logarithmic phase of estimation plus the linear phase of pre-sampling ensures that in the rest of the process, L is actually smooth. Proof. We use the fact that for all i ∈ [d], p i ≥ p o i /2. We have that for all i ∈ [d],

∇ 2 ii L(p) = Cof(Γ) ii σ 2 i det(Γ) 2 p 3 i ≤ 2 Cof(Γ) ii σ 2 i det(Γ)(p o i /2) 3 .
We have

p o k = σ k Cof(Γ) kk d i=1 σ i Cof(Γ) ii which gives ∇ 2 ii L(p) ≤ 16 σ 2 max d k=1 σ k Cof(Γ) kk 3 det(Γ)σ 3 min min k Cof(Γ) kk C S .
And consequently L is C S -smooth.

We can obtain an upper bound on C S using Corollary 2.1, which tells that

σ k /2 ≤ σ k ≤ 3σ k /2: C S ≤ 432 σ 2 max d k=1 σ k Cof(Γ) kk 3 det(Γ)σ 3 min min k Cof(Γ) kk .
We can now state our main theorem.

R(T ) = O Γ,σ k log 2 (T ) T 2 .
This theorem provides a fast convergence rate for the regret R(T ) and emphasizes the importance of using the gradient information in Algorithm 2.2 compared to Algorithm 2.1.

Proof. Proposition 2.3 gives that

|G i -Ĝi | ≤ 678K σ max σ 4 min 1 σ i λ min (Γ) max k∈[K] σ 2 k p k 3 • κ 2 max • max   log(4T K/δ) T i , log(4T K/δ) T i   .
Since each arm has been sampled at least a linear number of times we guarantee that log(4T K/δ)/T i ≤ 1 such that

|G i -Ĝi | ≤ 678K σ max σ min 7 1 λ min (Γ) 3 κ 2 max p 3 min log(4T K/δ) T i .
Thanks to the presampling phase of Lemma 2.6, we know that p min ≥ p o /2. For the sake of

clarity we note C 678K σ max σ min 7 8 p o3 λ min (Γ) 3 κ 2 max such that |G i -Ĝi | ≤ C log(4T K/δ) T i .
We have seen that L is µ-strongly convex, C L -smooth and that dist(p , ∂∆ d ) ≥ η. Consequently, since Lemma 2.6 shows that the pre-sampling stage does not affect the convergence result, we can apply [START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF], Theorem 7) (with the choice δ T = 1/T 2 , which gives that

E[L(p T )] -L(p ) ≤ c 1 log 2 (T ) T + c 2 log(T ) T + c 3 1 T , with c 1 = 96C 2 K µη 2 , c 2 = 24C 2 µη 3 +S and c 3 = 3072 2 K µ 2 η 4 L ∞ + µη 2 2 +C S .
With the presampling stage and Lemma 2.4, we can bound L ∞ by

L ∞ ≤ j σ 2 j Cof(Γ) jj σ min Cof(Γ) min   j σ j Cof(Γ) jj   .
We conclude the proof using the fact that R(T ) = 1 T (L(p T ) -L(p )).

Discussion and generalization to K > d

We discuss in this section the case where the number K of covariate vectors is greater than d.

Discussion of the Case K > d

In the case where K > d it may be possible that the optimal p lies on the boundary of the simplex ∆ K , meaning that some arms should not be sampled. This happens for instance as soon as there exist two covariate points that are exactly equal but with different variances. The point with the lowest variance should be sampled while the point with the highest one should not. All the difficulty of an algorithm for the case where K > d is to be able to detect which covariate should be sampled and which one should not. In order to adopt another point of view on this problem it might be interesting to go back to the field of optimal design of experiments. Indeed by choosing v k = X k /σ k , our problem consists exactly in the following constraint minimization problem given v

1 . . . , v K ∈ R d : min Tr   K j=1 p j v j v j   -1 under contraints p ∈ ∆ K . (P)
It is known [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF]) that the dual problem of A-optimal design consists in finding the smallest ellipsoid, in some sense, containing all the points v j : max Tr( √ W ) 2 under contraints W 0 5 and v j W v j ≤ 1 for all 1 ≤ j ≤ K .

(D)

In our case the role of the ellipsoid can be easily seen with the KKT conditions.

Proposition 2.4. The points X k /σ k lie within the ellipsoid defined by the matrix Ω(p ) -2 .

Proof. We want to minimize L on the simplex ∆ K . Let us introduce the Lagrangian function

L : (p 1 , . . . , p K , λ, µ 1 , . . . , µ K ) ∈ R K × R × R K + → L(p) + λ K k=1 p k -1 -µ, p
Applying Karush-Kuhn-Tucker theorem gives that p verifies

∀k ∈ [d], ∂L ∂p k (p ) = 0.
5 W 0 means here that W is symmetric positive definite.

104 Consequently ∀k ∈ [d], Ω(p ) -1 X k σ k 2 2 = λ -µ k ≤ λ.
This shows that the points X k /σ k lie within the ellipsoid defined by the equation x Ω(p ) -2 x ≤ λ.

This geometric interpretation shows that a point X k with high variance is likely to be in the interior of the ellipsoid (because X k /σ k is close to the origin), meaning that µ k > 0 and therefore that p k = 0 i.e., that X k should not be sampled. Nevertheless since the variances are unknown, one is not easily able to find which point has to be sampled. Geometrically the dual problem (D) is equivalent to finding an ellipsoid containing all data points X k /σ k such that the sum of the inverse of the semi-axis is maximized. The points that lie on the boundary of the ellipsoid are the one that have to be sampled. We see here that we have to sample the points that are far from the origin (after being rescaled by their standard deviation) because they cause less uncertainty.

We see that several cases can occur as shown on Figure 2.1. If one covariate is in the interior of the ellipsoid it is not sampled because of the KKT equations (see Proposition 2.4). However if all the points are on the ellipsoids some of them may not be sampled. It is the case on Figure 2.1b where X 1 is not sampled. This is due to the fact that a little perturbation of another point, for example X 3 can change the ellipsoid such that X 1 ends up inside the ellipsoid as shown on Figure 2.1d. This case can consequently be seen as a limit case.

A Theoretical Upper-Bound and a Lower Bound

We derive now a bound for the convergence rate of Algorithm 2.2 in the case where K > d.

Theorem 2.3. Applying Algorithm 2.2 with K > d covariate points gives the following bound on the regret, after T ≥ 1 samples

R(T ) = O log(T ) T 5/4 .
Proof. In order to ensure that L is smooth we pre-sample each covariate n times. We note α = n/T ∈ (0, 1). This forces p i to be greater than α for all i. Therefore L is C S -smooth

with C S ≤ 2 max k Cof(Γ) kk σ 2 max α 3 det(Γ)
C α 3 . We use a similar analysis to the one of [START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF]

T ρ T ≤ nKρ nK + C S log(eT ) + T t=nK ε t L(p T ) -L(p ) ≤ Kα(L(p nK ) -L(p )) + C α 3 log(eT ) T + 1 T T t=nK ε t .
We bound T t=nK ε t /T as in Theorem 3 of [START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF]) by 4

3K log(T )

T + π 2 6 + K 2 ∇L ∞ + L ∞ T = O log(T ) T .
We are now interested in bounding α(L(p nK )-

L(p )).

By convexity of L we have

L(p nK ) -L(p ) ≤ ∇L(p nK ), p nK -p ≤ ∇L(p nK ) 2 p nK -p 2 ≤ 2 ∇L(p nK ) 2 .
We have also

∂L ∂p k (p nK ) = -Ω(p nK ) -1 X k σ k 2 2 . Proposition 2.5 shows that Ω(p) -1 2 ≤ 1 λ min (Γ) σ 2 max min k p k .
In our case, min k p nK = 1/K. Therefore

Ω(p nK ) -1 2 ≤ Kσ 2 max λ min (Γ) .
And finally we have

∇L(p nK ) 2 ≤ K λ min (Γ) σ max σ min . We note C 1 2K 2 λ min (Γ)
σ max σ min . This gives

L(p T ) -L(p ) ≤ αC 1 + C α 3 log(T ) T + O log(T ) T .
The choice of α = T -1/4 finally gives

L(p T ) -L(p ) = O log(T ) T 1/4 .
One can ask whether this result is optimal, and if it is possible to reach the bound of Theorem 2.2. The following theorem provides a lower bound showing that it is impossible in the case where there are d covariates. However the upper and lower bounds of Theorems 2.3 and 2.4 do not match. It is still an open question whether we can obtain better rates than T -5/4 . Theorem 2.4. In the case where K > d, for any algorithm on our problem, there exists a set of parameters such that R(T ) T -3/2 . Proof. For simplicity we consider the case where d = 1 and K = 2. Let us suppose that there are two points X 1 and X 2 that can be sampled, with variances σ 2 1 = 1 and σ 2 2 = 1 + ∆ > 1, where ∆ ≤ 1. We suppose also that X 1 = X 2 = 1 such that both points are identical.

The loss function associated to this setting is

L(p) = p 1 σ 2 1 + p 2 σ 2 2 -1 = 1 + ∆ p 2 + p 1 (1 + ∆) = 1 + ∆ 1 + ∆p 1 .
The optimal p has all the weight on the first covariate (of lower variance): p = (1, 0) and

L(p ) = 1. Therefore L(p) -L(p ) = 1 + ∆ 1 + ∆p 1 -1 = p 2 ∆ 1 + ∆p 1 ≥ ∆ 2 p 2 .
We see that we are now facing a classical 2-arm bandit problem: we have to choose between arm 1 giving expected reward 0 and arm 2 giving expected reward ∆/2. Lower bounds on multi-armed bandits problems show that

EL(p T ) -L(p ) 1 √ T .
Thus we obtain R(T ) 1 T 3/2 .

Numerical simulations

We now present numerical experiments to validate our results and claims. We compare several algorithms for active matrix design: a very naive algorithm that samples equally each covariate, Algorithm 2.1, Algorithm 2.2 and a Thompson Sampling (TS) algorithm [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF]. We run our experiments on synthetic data with horizon time T between 10 4 and 10 6 , averaging the results over 25 rounds. We consider covariate vectors in R K of unit norm for values of K ranging from 3 to 100. All the experiments ran in less than 15 minutes on a standard laptop.

Let us quickly describe the Thompson Sampling algorithm. We choose Normal Inverse Gamma distributions for priors for the mean and variance of each of the arms, as they are the conjugate priors for gaussian likelihood with unknown mean and variance. At each time step t, for each arm k ∈ [K], a value of σk is sampled from the prior distribution. An approximate value of ∇ k L(p) is computed with the σk values. The arm with the lowest gradient value is chosen and sampled. The value of this arm updates the hyperparameters of the prior distribution.

In our first experiment we consider only 3 covariate vectors. We plot the results in log-log scale in order to see the convergence speed which is given by the slope of the plot. Results on Figure 2.2 show that both Algorithms 2.1 and 2.2, as well as Thompson sampling have regret O(1/T 2 ) as expected. We see that Thompson Sampling performs well on low-dimensional data. However it is approximately 200 times slower than Algorithm 2.2 -due to the sampling of complex Normal Inverse Gamma distributions -and therefore inefficient in practice. On the contrary, Algorithm 2.2 is very practical. Indeed its computational complexity is linear in time T and its main computational cost is due to the computation of the gradient ∇ L. This relies on inverting Ω ∈ R d×d , whose complexity is O(d 3 ) (or even O(d 2.807 ) with Strassen algorithm). Thus the overall complexity of Algorithm 2.2 is O(T (d 2.8 + K)) hence polynomial. This computational complexity advocates that Algorithm 2.2 is practical for moderate values of d, as in linear regression problems. Figure 2.2 shows that Algorithm 2.1 performs nearly as well as Algorithm 2.2. However, the minimization step of L is time-consuming when K > d, since there is no close form for p , which leads to approximate results. Therefore Algorithm 2.1 is not adapted to K > d. We also have conducted similar experiments in this case, with K = d + 1. The offline solution of the problem indicates that one covariate should not be sampled, i.e., p ∈ ∂∆ K . Results presented on Figure 2.3 prove the performances of Algorithm 2.2.

One might argue that the positive results of Figure 2.3 are due to the fact that it is "easy" for the algorithm to detect that one covariate should not be sampled, in the sense that this covariate clearly lies in the interior of the ellipsoids mentioned in Section 2.5.1. In the very challenging case where two covariates are equal but with variances separated by only 1/ √ T , we obtain the results described on Figure 2.4. The observed experimental convergence rate is of the order of T -1.36 which is much slower than the rates of Figure 2.3, and between the rates proved in Theorems 2.3 and Theorem 2.4. Finally we run a last experiment with larger values of K = d. We plot the convergence rate of Algorithm 2.2 for values of K ranging from 5 to 100 in loglog scale on Figure 2.5. The slope is again approximately of -2, which is coherent with Theorem 2.2. We note furthermore that larger values of d do not make Algorithm 2.2 impracticable, as inferred by its cubic complexity.

Conclusion

We have proposed an algorithm mixing bandit and convex optimization techniques to solve the problem of online A-optimal design, which is related to active linear regression with repeated queries. This algorithm has proven fast and optimal rates O(T -2 ) in the case of d covariates that can be sampled in R d . One cannot obtain such fast rates in the more general case of K > d covariates. We have therefore provided weaker results in this very challenging setting and conducted more experiments showing that the problem is indeed more difficult.

2.A Proof of gradient concentration

In this section we prove Proposition 2.3.

Proof of Proposition 2.3. Let p ∈ ∆ K and let i ∈ [K]. We compute G i -Ĝi = Ω(p) -1 X i σi 2 2 -Ω(p) -1 X i σ i 2 2 ≤ Ω(p) -1 X i σi -Ω(p) -1 X i σ i 2 Ω(p) -1 X i σi + Ω(p) -1 X i σ i 2 .
Let us now note A Ω(p)σ i and B Ω(p)σ i . We have, supposing that

X k 2 = 1, Ω(p) -1 X k σk -Ω(p) -1 X k σ k 2 = (A -1 -B -1 )X k 2 ≤ A -1 -B -1 2 X k 2 ≤ A -1 2 B -1 2 B -A 2 .
One of the quantity to bound is B -1 2 . We have

B -1 2 = ρ(B -1 ) = 1 min(Sp(B))
,

where Sp(B) is the spectrum (set of eigenvalues) of B. We know that Sp(B) = σ i Sp(Ω(p)). Therefore we need to find the smallest eigenvalue λ of Ω(p). Since the matrix is invertible we know λ > 0.

We will need the following lemma.

Lemma 2.8.

Let X 0 = X 1 , • • • , X k . We have λ min (Ω(p)) ≥ min k∈[K] p k σ 2 k λ min (X 0 X 0 ).
Proof. We have for all p ∈ ∆ K , min

i∈[K] p i σ 2 i K k=1 X k X k K k=1 p k σ 2 k X k X k . Therefore min k∈[K] p k σ 2 k X 0 X 0 Ω(p) .

And finally min

k∈[K] p k σ 2 k λ min (X 0 X 0 ) ≤ λ min (Ω(p)) .
Note now that the smallest eigenvalue of X 0 X 0 is actually the smallest non-zero eigenvalue of X 0 X 0 , which is the Gram matrix of (X 1 , . . . , X d ), that we note now Γ.

This directly gives the following Proposition 2.5. If B is defined as Ω(p)σ i for i ∈ [K], we have the following bound

B -1 2 ≤ 1 σ i λ min (Γ) max k∈[K] σ 2 k p k .
We jump now to the bound of A -1 2 . We could obtain a similar bound to the one of B -1 2 but it would contain σk values. Since we do not want a bound containing estimates of the variances, we prove the Proposition 2.6. If A is defined as Ω(p)σ i and B as Ω(p)σ i for i ∈ [K] we have the following inequality

A -1 2 ≤ 2 B -1 2 . Proof. We have, if we note H = A -B, A -1 2 = (B + A -B) -1 2 ≤ B -1 2 (I n + B -1 H) -1 2 ≤ 2 B -1 2 ,
from a certain rank.

Let us now bound B -A 2 . We have

B -A 2 = σ i K k=1 p k X k X k σ 2 k -σi K k=1 p k X k X k σ2 k 2 110 = K k=1 p k X k X k σ i σ 2 k - σi σ2 k 2 ≤ K k=1 p k σ i σ 2 k - σi σ2 k X k 2 2 ≤ K k=1 p k σ i σ 2 k - σi σ2 k .
The next step is now to use Theorem 2.1 in order to bound the difference

σ i σ 2 k - σi σ2 k .
Proposition 2.7. With the notations introduced above, we have

B -A 2 ≤ 113Kσ max σ 4 min κ 2 max • max   log(4T K/δ) T i , log(4T K/δ) T i   Proof. Corollary 2.1 gives that for all k ∈ [K], 1 2 σ 2 k ≤ σ2 k ≤ 3 2 σ 2 k . A consequence of Theorem 2.1 is that for all k ∈ [K],
if we note T k the (random) number of samples of covariate k, we have, with probability at least 1 -δ,

∀k ∈ [K], σ 2 k -σ2 k ≤ 8 3 κ 2 k • max   log(4T K/δ) cT k , log(4T K/δ) cT k   + 2κ 2 k log(4T K/δ) T k .
We note ∆ k the r.h.s of the last equation. We begin by establishing a simple upper bound of ∆ k . Using the fact that 1/c ≤ 1/c and that 8/(3c) ≤ 38, we have

∆ k ≤ 8 3c κ 2 k • max   log(4T K/δ) T k , log(4T K/δ) T k   + 2κ 2 k log(4T K/δ) T k ≤ 38κ 2 k • max   log(4T K/δ) T k , log(4T K/δ) T k   + 2κ 2 k log(4T K/δ) T k ≤ 40κ 2 k • max   log(4T K/δ) T k , log(4T K/δ) T k   . Let k ∈ [K]. We have σ i σ 2 k - σi σ2 k = σ i σ2 k -σi σ 2 k σ 2 k σ2 k = σ i σ2 k -σ i σ 2 k + σ i σ 2 k -σi σ 2 k σ 2 k σ2 k ≤ σ i (σ 2 k -σ 2 k ) σ 2 k σ2 k + σ i -σi σ2 k ≤ σ i (σ 2 k -σ 2 k ) σ 2 k σ2 k + σ 2 i -σ2 i σ2 k (σ i + σi ) ≤ σ i (σ 2 k -σ 2 k ) σ 2 k σ2 k + σ 2 i -σ2 i σ2 k σ i ≤ σ2 k -σ 2 k σ i σ 2 k σ2 k + σ 2 i -σ2 i 1 σ2 k σ i ≤ ∆ k 2σ max σ 4 min + ∆ i 2 √ 2 σ 3 min .
Finally we have, using the fact that

T ≥ T k for all k ∈ [K] B -A 2 ≤ K k=1 p k σ i σ 2 k - σi σ2 k 111 ≤ 2σ max σ 4 min K k=1 p k ∆ k + √ 2 K k=1 p k ∆ i ≤ 2σ max σ 4 min   K k=1 T k T 40κ 2 k • max   log(4T K/δ) T k , log(4T K/δ) T k   + √ 2∆ i   ≤ 2σ max σ 4 min K k=1 40κ 2 k • max log(4T K/δ) T , T k T log(4T K/δ) T + √ 2∆ i ≤ 2σ max σ 4 min K k=1 40κ 2 k • max log(4T K/δ) T , log(4T K/δ) T + √ 2∆ i ≤ 2σ max σ 4 min   K40κ 2 max • max   log(4T K/δ) T i , log(4T K/δ) T i   + √ 2∆ i   ≤ (K + √ 2) 80σ max σ 4 min κ 2 max • max   log(4T K/δ) T i , log(4T K/δ) T i   .
The last quantity to bound to end the proof is Ω

(p) -1 X k σk + Ω(p) -1 X k σ k 2 .
Proposition 2.8. We have for any k ∈

[K], Ω(p) -1 X k σk + Ω(p) -1 X k σ k 2 ≤ 3 B -1 2 .
Proof. For any k ∈ [K], we have

Ω(p) -1 X k σk + Ω(p) -1 X k σ k 2 = (A -1 + B -1 )X k 2 ≤ A -1 + B -1 2 X k 2 ≤ (A -1 -B -1 ) + 2B -1 2 ≤ A -1 -B -1 2 + 2 B -1 2 .
For T sufficiently large we have Ω

(p) -1 X k σk + Ω(p) -1 X k σ k 2 ≤ 3 B -1 2 .
Combining Propositions 2.5, 2.6, 2.7 and 2.8 we obtain that

G i -Ĝi ≤ 6 B -1 3 2 B -A 2 and G i -Ĝi ≤ 678K σ max σ 4 min 1 σ i λ min (Γ) max k∈[K] σ 2 k p k 3 • κ 2 max • max   log(4T K/δ) T i , log(4T K/δ) T i   ,
which proves Proposition 2.3.

Adaptive stochastic optimization for resource allocation

In this chapter, we consider the classical problem of sequential resource allocation where a decision maker must repeatedly divide a budget between several resources, each with diminishing returns. This can be recast as a specific stochastic optimization problem where the objective is to maximize the cumulative reward, or equivalently to minimize the regret. We construct an algorithm that is adaptive to the complexity of the problem, expressed in term of the regularity of the returns of the resources, measured by the exponent in the Łojasiewicz inequality (or by their universal concavity parameter). Our parameter-independent algorithm recovers the optimal rates for strongly concave functions and the classical fast rates of multi-armed bandit (for linear reward functions). Moreover, the algorithm improves existing results on stochastic optimization in this regret minimization setting for intermediate cases1 .

Introduction and related work

In the classical resource allocation problem, a decision maker has a fixed amount of budget (money, energy, work, etc.) to divide between several resources. Each of these resources is assumed to produce a positive return for any amount of budget allocated to them, and zero return if no budget is allocated to them [START_REF] Samuelson | Macroeconomics. McGraw-Hill international editions[END_REF]. The resource allocation problem is an age-old problem that has been theoretically investigated by [START_REF] Koopman | The optimum distribution of effort[END_REF] and that has attracted much attention afterwards [START_REF] Salehi | Stochastic-based robust dynamic resource allocation for independent tasks in a heterogeneous computing system[END_REF][START_REF] Devanur | Near optimal online algorithms and fast approximation algorithms for resource allocation problems[END_REF] due to its numerous applications (e.g., production planning or portfolio selection) described for example by [START_REF] Gross | Notes on Linear Programming: Class of Discrete-type Minimization Problems[END_REF] and [START_REF] Katoh | Resource allocation problems[END_REF].

Other applications include cases of computer scheduling, where concurrent processes compete for common and shared resources. This is the exact same problem encountered in load distribution or in project management where several tasks have to be done and a fixed amount of money/time/workers has to be distributed between those tasks. Flexible Manufacturing Systems (FMS) are also an example of application domain of our problem [START_REF] Colom | The Resource Allocation Problem in Flexible Manufacturing Systems[END_REF] and motivate our work. Resource allocation problems arise also in the domain of wireless communications systems, for example in the new 5G networks, due to the exponential growth of wireless data [START_REF] Zhang | Energyefficient resource allocation in NOMA heterogeneous networks[END_REF]. Finally, utility maximization in economics is also an important application of the resource allocation problem, which explains that this problem has been particularly studied in economics, where classical assumptions have been made for centuries [START_REF] Smith | An Inquiry into the Nature and Causes of the Wealth of Nations[END_REF]. One of them is the diminishing returns assumption that states that "adding more of one factor of production, while holding all others constant, will at some point yield lower incremental per-unit returns"2 . This natural assumption means that the reward or utility per invested unit decreases, and can be linked to submodular optimization [START_REF] Korula | Online submodular welfare maximization: Greedy beats 1/2 in random order[END_REF].

In this chapter we consider the online resource allocation problem with diminishing returns. A decision maker has to partition, at each stage, $1 between K resources. Each resource has an unknown reward function which is assumed to be concave and increasing. As the problem is repeated in time, the decision maker can gather information about the reward functions and sequentially learn the optimal allocation. We assume that the reward itself is not observed precisely, but rather a noisy version of the gradient is observed. As usually in sequential learning -or bandit -problems [START_REF] Bubeck | Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems[END_REF], the natural objective is to maximize the cumulative reward, or equivalently, to minimize the difference with the obtained allocation, namely the regret.

This problem is a generalization of linear resource allocation problems, widely studied in the last decade [START_REF] Lattimore | Linear Multi-Resource Allocation with Semi-Bandit Feedback[END_REF][START_REF] Dagan | A Better Resource Allocation Algorithm with Semi-Bandit Feedback[END_REF], where the reward functions are assumed to be linear, instead of being concave. Those approaches borrowed ideas from linear bandits [START_REF] Dani | Stochastic Linear Optimization under Bandit Feedback[END_REF][START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF]. Several UCBstyle algorithms with nearly optimal regret analysis have been proposed for the linear case. More general algorithms were also developed to optimize an unknown convex function with bandit feedback [START_REF] Agarwal | Stochastic convex optimization with bandit feedback[END_REF][START_REF] Agrawal | Sample mean based index policies by o(log(n)) regret for the multiarmed bandit problem[END_REF]Devanur, 2014, 2015;[START_REF] Berthet | Fast rates for bandit optimization with upperconfidence Frank-Wolfe[END_REF] to get a generic O( √ T )3 regret bound which is actually unavoidable with bandit feedback [START_REF] Shamir | On the Complexity of Bandit and Derivative-Free Stochastic Convex Optimization[END_REF]. We consider instead that the decision maker has a noisy gradient feedback, so that the regularity of the reward mappings can be leveraged to recover faster rates (than √ T ) of convergence when possible. There are several recent works dealing with (adaptive) algorithms for first order stochastic convex optimization. On the contrary to classical gradient-based methods, these algorithms are agnostic and adaptive to some complexity parameters of the problem, such as the smoothness or strong convexity parameters. For example, [START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] proposed an adaptive algorithm to optimize uniformly convex functions and Ramdas and Singh (2013a) generalized it using active learning techniques, also to minimize uniformly convex functions. Both obtain optimal bounds in O T -ρ/(2ρ-2) for the function-error f (x t ) -f * where f is supposed to be ρ-uniformly convex (see Section 3.2.3 for a reminder on this regularity concept). However those algorithms would only achieve a √ T regret (or even a linear regret) because they rely on a structure of phases of unnecessary lengths. So in that setting, regret minimization appears to be much more challenging than function-error minimization. To be precise, we actually consider an even weaker concept of regularity than uniform convexity: the Łojasiewicz inequality [START_REF] Bierstone | Semianalytic and subanalytic sets[END_REF][START_REF] Bolte | Characterizations of Łojasiewicz inequalities: Subgradient flows, Talweg, Convexity[END_REF]. Our objective is to devise an algorithm that can leverage this assumption, without the prior knowledge of the Łojasiewicz exponent, i.e., to construct an adaptive algorithm unlike precedent approaches [START_REF] Karimi | Linear convergence of gradient and proximal-gradient methods under the Polyak-Łojasiewicz condition[END_REF].

The algorithm we are going to introduce is based on the concept of dichotomy, or bi-nary search, which has already been slightly investigated in stochastic optimization [START_REF] Burnashev | An interval estimation problem for controlled observations[END_REF][START_REF] Castro | Minimax bounds for active learning[END_REF][START_REF] Ramdas | Optimal rates for first-order stochastic convex optimization under tsybakov noise condition[END_REF]. The specific case of K = 2 resources is studied in Section 3.3. The algorithm proposed is quite simple: it queries a point repeatedly, until it learns the sign of the gradient of the reward function, or at least with arbitrarily high probability. Then it proceeds to the next step of a standard binary search. We will then consider, in Section 3.4, the case of K ≥ 3 resources by defining a binary tree of the K resources and handling each decision using the K = 2 algorithm as a black-box. Our main result can be stated as follows: if the base reward mappings of the resources are β-Łojasiewicz functions, then our algorithm has a O(T -β/2 ) regret bound if β ≤ 2 and O(T -1 ) otherwise. We notice that for β ≤ 2 we recover existing bounds (but for the more demanding regret instead of function-error minimization) [START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF]Ramdas and Singh, 2013a) since a ρ-uniformly convex function can be proven to be β-Łojasiewicz with β = ρ/(ρ -1). We complement our results with a lower bound that indicates the tightness of these bounds. Finally we corroborate our theoretical findings with some experimental results.

Our main contributions are the design of an efficient algorithm to solve the resource allocation problem with concave reward functions. We show that our algorithm is adaptive to the unknown complexity parameters of the reward functions. Moreover we propose a unified analysis of this algorithm for a large class of functions. It is interesting to notice that our algorithm can be seen as a first-order convex minimization algorithm for separable loss functions. The setting of separable loss functions is still common in practice, though not completely general. Furthermore we prove that our algorithm outperforms other convex minimization algorithms for a broad class of functions. Finally we exhibit links with bandit optimization and we recover classical bandit bounds within our framework, highlighting the connection between bandits theory and convex optimization.

The remainder of this chapter is organized as follows. First, let us introduce in Section 3.2 the general model and the different regularity assumptions mentioned above. We study the case K = 2 in Section 3.3 and the case K ≥ 3 is Section 3.4. Numerical experiments are presented in Section 3.5 and Section 3.6 concludes the chapter. Postponed proofs are put in Appendices 3.A, 3.B and 3.C.

Model and assumptions

Problem setting

Assume a decision maker has access to K ∈ N * different resources. We assume naturally that the number of resources K is not too large (or infinite). At each time step t ∈ N * , the agent has to split a total budget of weight 1 and to allocate x

(t) k to each resource k ∈ [K] which generates the reward f k (x k (t))
. Overall, at this stage, the reward of the decision maker is then

F (x (t) ) = k∈[K] f k (x (t) k ) with x (t) = (x (t) 1 , . . . , x (t) K ) ∈ ∆ K ,
where the simplex ∆ K = (p 1 , . . . , p K ) ∈ R K + ; k p k = 1 is the set of possible convex weights.

We note x ∈ ∆ K the optimal allocation that maximizes F over ∆ K ; the objective of the decision maker is to maximize the cumulated reward, or equivalently to minimize the regret R(T ), defined as the difference between the optimal reward F (x ) and the average reward over T ∈ N * stages t) ) .

R(T ) = F (x ) - 1 T T t=1 K k=1 f k (x (t) k ) = max x∈∆ K F (x) - 1 T T t=1 F (x (
The following diminishing return assumption on the reward functions f k is natural and ensures that F is concave and continuous, ensuring the existence of x .

A 3.1. The reward functions f k : [0, 1] → R are concave, non-decreasing and verify f k (0) = 0. Moreover we assume that they are differentiable, L-Lipschitz continuous and L -smooth.

This assumption means that the more the decision maker invest in a resource, the greater the revenue. Moreover, investing 0 gives nothing in return. Finally the marginal increase of revenue decreases.

We now describe the feedback model. At each time step the decision maker observes a noisy version of ∇F (x (t) ), which is equivalent here to observing each ∇f k (x

(t) k ) + ζ (t) k , where ζ (t)
k ∈ R is some white bounded noise. The assumption of noisy gradients is classical in stochastic optimization and is similarly relevant for our problem: this assumption is quite natural as the decision maker can evaluate, locally and with some noise, how much a small increase/decrease of an allocation x (t) k affects the reward. Consequently, the decision maker faces the problem of stochastic optimization of a concave and separable function over the simplex (yet with a cumulative regret minimization objective). Classical stochastic gradient methods from stochastic convex optimization would guarantee that the average regret decreases as O K/T 1/2 in general and as O (K/T ) if the f k are known to be strongly concave. However, even without strong concavity, we claim that it is possible to obtain better regret bounds than O K/T and, more importantly, to be adaptive to some complexity parameters.

The overarching objective is then to leverage the specific structure of this natural problem to provide a generic algorithm that is naturally adaptive to some complexity measure of the problem. It will, for instance, interpolate between the non-strongly concave and the strongly concave rates without depending on the strong-concavity parameter, and recover the fast rate of classical multi-armed bandit (corresponding more or less to the case where the f k functions are linear). Existing algorithms for adaptive stochastic convex optimization (Ramdas and Singh, 2013a;[START_REF] Juditsky | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] are not applicable in our case since they work for function-error minimization and not regret minimization (because of the prohibitively large stage lengths they are using).

Reminders on the Łojasiewicz inequality

We present here results on the famous Łojasiewicz inequality [START_REF] Łojasiewicz | Ensembles semi-analytiques[END_REF][START_REF] Bierstone | Semianalytic and subanalytic sets[END_REF][START_REF] Bolte | Characterizations of Łojasiewicz inequalities: Subgradient flows, Talweg, Convexity[END_REF]. In this section we state all the results in their most general and classical form, i.e., for convex functions. Their equivalents for concave functions are easily obtained by symmetry. Let us begin with some definitions.

Definition 3.1. A function

f : R d → R satisfies the Łojasiewicz inequality if ∀x ∈ X , f (x) -min x ∈X f (x ) ≤ µ ∇f (x) β .
Definition 3.2. A function f : R d → R is uniformly-convex with parameters ρ ≥ 2 and µ > 0 if and only if for all x, y ∈ R d and for all α ∈ [0, 1],

f (αx + (1 -α)y) ≤ αf (x) + (1 -α)f (y) - µ 2 α(1 -α) α ρ-1 + (1 -α) ρ-1 x -y ρ .
A first interesting result is the fact that every uniformly convex function verifies the Łojasiewicz inequality.

Proposition 3.1. If f is a differentiable (ρ, µ)-uniformly convex function then it satisfies the Łojasiewicz inequality with parameters β = ρ/(ρ -1) and c = 2 µ 1/(ρ-1) ρ -1 ρ ρ/(ρ-1) .

Proof. A characterization of differentiable uniformly convex function (see for example (Juditsky and Nesterov, 2014)) gives that for all x, y ∈ R d

f (y) ≥ f (x) + ∇f (x), y -x + 1 2 µ x -y ρ . Consequently, noting f (x ) = inf f (x), f (x ) ≥ inf y f (x) + ∇f (x), y -x + 1 2 µ x -y ρ g(y)
.

We now want to minimize the function g which is a strictly convex function. We have

∇g(y) = ∇f (x) + µ 2 ρ x -y ρ-2 (y -x).
g reaches its minimum for ∇g(y) = 0 and ∇f -1) .

(x) = - µ 2 ρ x -y ρ-2 (y -x). This gives f (x ) ≥ f (x) + µ 2 x -y ρ (1 -ρ). Since ∇f (x) = µρ 2 x -y ρ-1 we obtain f (x) -f (x ) ≤ (ρ -1) µ 2 2 µρ ∇f (x) ρ/(ρ-1) ≤ 2 µ 1/(ρ-1) ρ -1 ρ ρ/(ρ-1) ∇f (x) ρ/(ρ
In particular a µ-strongly convex function verifies the Łojasiewicz inequality with β = 2 and c = 1/(2µ).

In the case of a convex function we have the following result.

Proposition 3.2. Let f : R → R be a convex differentiable function. Let x = arg min x∈R f (x). Then for all x ∈ R, f (x) -f (x ) ≤ |f (x)||x -x | ,
meaning that f satisfies the Łojasiewicz inequality for β = 1.

Proof. Let x ∈ R. f (x) -f (x ) = x x f (y) dy .
Let us distinguish two cases depending on x < x or x > x .

(a) x < x : since f is non-decreasing (because f is convex) we have for all y ∈ [x, x ], f (y) ≥ f (x) and therefore, since f (x) ≤ 0

f (x) -f (x ) ≤ - x x f (y) dy ≤ |x -x||f (x)| .
(b) x > x : similarly we have for all y ∈ [x , x], f (y) ≤ f (x) and therefore, since f (x) ≥ 0,

f (x) -f (x ) = x x f (y) dy ≤ |x -x|f (x) = |f (x)||x -x| .
We recall now the definition of the local Tsybakov Noise Condition (TNC) [START_REF] Castro | Minimax bounds for active learning[END_REF], around the minimum x of a function f with vanishing gradient.

> 1 if ∀x ∈ X , f (x) -min x ∈X f (x ) ≥ µ x -x κ ,
where in the above the x on the r.h.s. is the minimizer of f the closer to x (in the case where f has non-unique minimizers).

Uniform convexity, TNC and Łojasiewicz inequality are connected since it is well known that if a function f is uniformly convex, it satisfies both the local TNC and the Łojasiewicz inequality. Those two concepts are actually equivalent for convex mappings. Proposition 3.3. If f is a convex differentiable function locally satisfying the TNC with parameters κ and µ then it satisfies the Łojasiewicz equation with parameters κ/(κ -1) and µ -1/(κ-1) .

Proof. Let x, y ∈ R d . Since f is convex we have, noting x = arg min f ,

f (y) ≥ f (x) + ∇f (x), y -x f (x) -f (x ) ≤ ∇f (x), x -x f (x) -f (x ) ≤ ∇f (x) x -x . The TNC gives f (x)-f (x ) ≥ µ x -x κ , which means that x -x ≤ µ -1/κ (f (x) -f (x )) 1/κ
and consequently, -1) .

f (x) -f (x ) ≤ ∇f (x) µ -1/κ (f (x) -f (x )) 1/κ (f (x) -f (x )) 1-1/κ ≤ µ -1/κ ∇f (x) (f (x) -f (x )) ≤ µ -1/(κ-1) ∇f (x) κ/(κ
This concludes the proof.

We now show that the two classes of uniformly convex functions and Łojasiewicz functions are distinct by giving examples of functions that verify the Łojasiewicz inequality and that are not uniformly convex.

Example 3.1. The function f : (x, y) ∈ R 2 → (x -y) 2 verifies the Łojasiewicz inequality but is not uniformly convex on R 2 .

Definition and properties of the complexity class

As mentioned before, our algorithm will be adaptive to some general complexity parameter of the set of functions F = {f 1 , . . . , f K }, which relies on the Łojasiewicz inequality [START_REF] Bierstone | Semianalytic and subanalytic sets[END_REF][START_REF] Bolte | Characterizations of Łojasiewicz inequalities: Subgradient flows, Talweg, Convexity[END_REF] that we state now, for concave functions (rather than convex).

Definition 3.4. A function

f : R d → R satisfies the Łojasiewicz inequality with respect to β ∈ [1, +∞) on its domain X ⊂ R d if there exists a constant c > 0 such that ∀x ∈ X , max x ∈X f (x ) -f (x) ≤ c ∇f (x) β .
Given two functions f, g : [0, 1] → R, we say that they satisfy pair-wisely the Łojasiewicz inequality with respect to β ∈ [1, +∞) if the function (z → f (z) + g(x -z)) satisfies the Łojasiewicz inequality on [0, x] with respect to β for every x ∈ [0, 1].

It remains to define the finest class of complexity of a set of functions F. It is defined with respect to binary trees, whose nodes and leaves are labeled by functions. The trees we consider are constructed as follows. Starting from a finite binary tree of depth log 2 (|F|) , its leaves are labeled with the different functions in F (and 0 for the remaining leaves if |F| is not a power of 2). The parent node of f left and f right is then labeled by the function

x → max z≤x f left (z) + f right (x -z).
We say now that F satisfies inductively the Łojasiewicz inequality for β ≥ 1 if in any binary tree labeled as above, any two siblings4 satisfy pair-wisely the Łojasiewicz inequality for β.

We provide now some properties of the labeled tree constructed above. We begin with a technical and useful lemma.

Lemma 3.1. Let f and g be two differentiable concave functions on [0, 1]. For x ∈ [0, 1] define φ x : z ∈ [0, x] → f (z) + g(x -z). And z x arg max z∈[0,x] φ x (z). We have the following results:

• φ x is concave; • ∀ 0 ≤ x ≤ y ≤ 1, z x ≤ z y and x -z x ≤ y -z y . In particular the function x → z x is 1-Lipschitz continuous.
Proof. The fact that φ x is concave is immediate since f and g are concave functions. z x > 0 and φ (z x ) ≥ 0. This shows that φ y (z x ) ≥ 0 and consequently, that the maximum z y of the concave function φ y is reached after z x . And

If 0 ≤ x ≤ y ≤ 1, we have g (y -z x ) ≤ g (x -z x ) since y -z x ≥ x -z x and g is non- increasing (because g is concave). Consequently, φ y (z x ) = f (z x ) -g (y -z x ) ≥ φ x (z x ). If z x = 0, z y ≥ z x is immediate. Otherwise,
z y ≥ z x .
The last inequality is obtained in a symmetrical manner by considering the function

ψ x : z ∈ [0, x] → f (x -z) + g(x) whose maximum is reached at z = x -z x . This gives x -z x ≤ y -z y .
We now prove two simple lemmas.

Lemma 3.2. If f and g are two concave L-Lipschitz continuous and differentiable functions, then H

: x → max z∈[0,x] f (z) + g(x -z) is L-Lipschitz continuous.
Proof. With the notations of the previous lemma, we have

H(x) = φ x (z x ) for all x ∈ [0, 1]. Let x, y ∈ [0, 1].
Without loss of generality we can suppose that x ≤ y. We have

|H(x) -H(y)| = |f (z x ) + g(x -z x ) -f (z y ) -g(y -z y )| ≤ L|z x -z y | + L|x -z x -(y -z y | ≤ L(z y -z x ) + L(y -z y -x + z x ) ≤ L|y -x|.
We have used the conclusion of Lemma 3.1 in the third line.

Lemma 3.3. If f and g are two concave L -smooth and differentiable functions, then

H : x → max z∈[0,x] f (z) + g(x -z) is L -smooth. Proof. Let x, y ∈ [0, 1].
Without loss of generality we can suppose that x ≤ y. We treat the case where φ x ∈ (0, x) and φ y ∈ (0, y). The other (extremal) cases can be treated similarly. The envelop theorem gives that ∇H(x) = ∇f (z x ) and ∇H(y

) = ∇f (z y ). Therefore |∇H(x) -∇H(y)| = |∇f (z x ) -∇f (z y )| ≤ L |z x -z y | ≤ L |x -y| with Lemma 3.1.
Proposition 3.5 and Lemmas 3.2 and 3.3 show directly the following proposition:

Proposition 3.4. If the functions f 1 , . . . , f K are concave differentiable L-Lipschitz continuous and L -smooth then all functions created in the tree are also concave differentiable L-Lipschitz continuous and L -smooth.

Examples of class of functions satisfying inductively the Łojasiewicz inequality

Since the definition proposed above is quite intricate we can focus on some easier insightful sub-cases. In particular, a set of functions of cardinality 2 satisfies inductively the Łojasiewicz inequality if and only if these functions satisfy it pair-wisely. Another crucial property of our construction is that if f left and f right are concave, non-decreasing and zero at 0, then these three properties also hold for their parent x → max z≤x f left (z) + f right (xz). As a consequence, if these three properties hold at the leaves, they will hold at all nodes of the tree. See Proposition 3.5 for similar alternative statements.

Proposition 3.5. Assume that F = {f 1 , . . . , f K } is finite then F satisfies inductively the Łojasiewicz inequality with respect to some β F ∈ [1, +∞). Moreover, 1. if f k are all concave, non-decreasing and f k (0) = 0, then all functions created inductively in the tree satisfy the same assumption.

2. If f k are all ρ-uniformly concave, then so are all the functions created and F satisfies inductively the Łojasiewicz inequality for

β F = ρ ρ-1 . 3. If f k are concave, then F satisfies inductively the Łojasiewicz inequality w.r.t. β F = 1.
4. If f k are linear then F satisfies inductively the Łojasiewicz inequality w.r.t. any β F ≥ 1.

More specifically, if F is a finite subset of the following class of functions

C α := x → θ(γ -x) α -θγ α ; θ ∈ R -, γ ≥ 1 , if α > 1
then F satisfies inductively the Łojasiewicz inequality with respect to β = α α-1 .

Proof.

1. We just need to prove that the mapping x → H(x) = max z≤x f 1 (z) + f 2 (x -z) = max z≤x G(z; x) satisfies the same assumption as f 1 and f 2 , the main question being concavity. Given x 1 , x 2 , λ ∈ [0, 1], let us denote by z 1 the point where G(• ; x 1 ) attains its maximum (and similarly z 2 where G(• ; x 2 ) attains its maximum). Then the following holds

H(λx 1 + (1 -λ)x 2 ) ≥ f 1 (λz 1 + (1 -λ)z 2 ) + f 1 (λx 1 + (1 -λ)x 2 -λz 1 -(1 -λ)z 2 ) ≥ λf 1 (z 1 ) + (1 -λ)f 1 (z 2 ) + λf 2 (x 1 -z 1 ) + (1 -λ)f 2 (x 2 -z 2 ) = λH(x 1 ) + (1 -λ)H(x 2 )
so that concavity is ensured. The fact that H(0) = 0 and H(•) is non-decreasing are trivial.

Let us prove that the mapping

(x → H(x) = max 0 ≤ z ≤ xf 1 (z) + f 2 (x -z)) is also ρ-uniformly concave. Let α ∈ (0, 1). Let (x, y) ∈ R 2 . Let us denote by z x the point in (0, x) such that H(x) = f 1 (z x ) + f 2 (x -z x )
and by z y the point in (0, y) such that

H(y) = f 1 (z y ) + f 2 (y -z y ).
We have

αH(x) + (1 -α)H(y) = αf 1 (z x ) + αf 2 (x -z x ) + (1 -α)f 1 (z y ) + (1 -α)f 2 (y -z y ) ≤ f 1 (αz x + (1 -α)z y ) - µ 2 α(1 -α) α ρ-1 + (1 -α) ρ-1 z x -z y ρ + f 2 (α(x -z x ) + (1 -α)(y -z y )) - µ 2 α(1 -α) α ρ-1 + (1 -α) ρ-1 x -z x -y + z y ρ ≤ H(αx + (1 -α)y) - µ 2 α(1 -α)( x -y /2) ρ
where we used the fact that f 1 and f 2 are ρ-uniformly concave, and the definition of

H(αx + (1 -α)y), and that a ρ + b ρ ≥ ((a + b)/2) ρ , for a, b ≥ 0.
This proves that H is (ρ, µ/2 ρ )-uniformly convex. Finally Proposition 3.1 shows that F satisfies inductively the Łojasiewicz inequality for β F = ρ/(ρ -1).

3. This point is actually a direct consequence of the Proposition 3.2.

4. If f 1 and f 2 are linear, then x → max z ≤ xf 1 (z) + f 2 (x -z) is either equal to f 1 or to f 2 (depending on which one is the biggest). Hence it is linear. 5. Assume that f i = θ i (γ i -x) α -θ i γ α i for some parameter γ i > 1 and θ i < 0. Then easy computations show that H is equal to either f 1 or f 2 on a small interval near 0 (depending on the size of ∇f i (0)) and then H(x) = θ 0 (γ 0 -x) α -c 0 for some parameters θ 0 < 0 and γ 0 > 1. As a consequence, H is defined piecewisely by functions in C α , a property that will propagate in the binary tree used in the definition of inductive satisfiability of Łojasiewicz inequality. The fact that those functions satisfies the Łojasiewicz inequality with respect to β = α α-1 has already been proved in Example 3.3.

One could ask why the class of Łojasiewicz functions is interesting. A result of Łojasiewicz (1965) shows that all analytic functions satisfy the Łojasiewicz inequality with a parameter β > 1. This is a strong result motivating our interest for the class of functions satisfying the Łojasiewicz inequality. More precisely we have the following proposition.

Proposition 3.6. If the functions {f 1 , . . . , f K } are real analytic and strictly concave then the class F satisfy inductively the Łojasiewicz inequality with a parameter β F > 1.

The proof relies on the following lemma. Lemma 3.4. If f and g are strictly concave real analytic functions then H : x → max 0≤z≤x f (z) + g(x -z) is also a strictly concave real analytic function.

Proof. The fact that H is strictly concave comes from Proposition 3.5. Since f and g are real analytic functions we can write

f (x) = n≥0 a n x n and g(x) = n≥0 b n x n .
Let us consider the function

φ x : z → f (z) + g(x -z) for z ∈ [0, x]. Now, for all 0 ≤ z ≤ x, we have φ x (z) = f (z) + g(x -z) = n≥0 a n z n + n≥0 b n (x -z) n = n≥0 a n z n + n≥0 b n n k=0 n k x n-k (-1) k z k = k≥0 a k z k + k≥0   n≥k b n (-1) k x n-k   z k = k≥0 c k (x)z k , with c k (x) = a k + n≥k b n (-1) k x n-k .
Since f and g are concave, φ x is also concave. Let z x arg max z∈[0,x] φ x (z). We have

H(x) = φ x (z x ) If z x ∈ (0, x) then ∇φ x (z x ) = 0 because φ x is concave. Consequently k≥0 c k+1 (x)(k + 1)z k x = 0. Let us consider the function Ψ : (x, z) → k≥0 c k+1 (x)(k + 1)z k x = ∇φ x (z). Provided that ∇ z Ψ(x, z x ) is invertible then z x is
unique and is an analytic function of x thanks to the analytic implicit function theorem [START_REF] Berger | Nonlinearity and functional analysis: lectures on nonlinear problems in mathematical analysis[END_REF]. Since f and g are strictly concave the invertibility condition is satisfied since ∇ z Ψ(x, z x ) = f (z) + g (x -z), and the result is proved.

Proof of Proposition 3.6. Let us show that F satisfies inductively the Łojasiewicz inequality. Let f and g be two siblings of the tree defined in Section 3.2.3. Inductively applying Lemma 3.4 shows that (x → max 0≤z≤x f (z) + g(x -z) is a strictly concave real analytic function. Since a real analytic function verifies the Łojasiewicz inequality (Łojasiewicz, 1965), the result is proved. We set β to be the minimum of all Łojasiewicz exponents in the tree.

In the following section, we introduce a generic, parameter free algorithm that is adaptive to the complexity β F ∈ [1, +∞) of the problem. Note that β F is not necessarily known by the agent and therefore the fact that the algorithm is adaptive to the parameter is particularly interesting. The simplest case K = 2 provides many insights and will be used as a subroutine for more resources. Therefore, we will first focus on this case.

Stochastic gradient feedback for K = 2

We first focus on only K = 2 resources. In this case, we rewrite the reward function F as

F (x) = f 1 (x 1 ) + f 2 (x 2 ) = f 1 (x 1 ) + f 2 (1 -x 1 ).
For the sake of clarity we simply note x = x 1 and we define g(x) F (x) -F (x ). Note that g(x ) = 0 and that g is a non-positive concave function. Using these notations, at each time step t the agent chooses xt ∈ [0, 1], suffers |g(x t )| and observes g (x t ) + ε t where ε t ∈ [-1, 1] i.i.d.

Description of the main algorithm

The basic algorithm we follow to optimize g is a binary search. Each query point x (for example x = 1/2) is sampled repeatedly and sufficiently enough (as long as 0 belongs to some confidence interval) to guarantee that the sign of g (x) is known with arbitrarily high probability, at least 1 -δ. Algorithm 3.1 Binary search algorithm Require: T time horizon, δ confidence parameter 1: Search interval I 0 ← [0, 1] ; t ← 1 ; j ← 1 2: while t ≤ T do 3:

x j ← center(I j-1 ); S j ← 0; N j ← 0 4:

while 0 ∈ S j N j ± 2 log( 2T δ ) N j do 5:
Sample x j and get X t , noisy value of ∇g(x j )

6: S ← S j + X t , N j ← N j + 1 7: if S j N j > 2 log( 2T δ ) N j then 8: I j ← [x j , max(I j-1 )] 9:
else 10:

I j ← [min(I j-1 ), x j ] 11: t ← t + N j ; j ← j + 1 12: return x j
Algorithm 3.1 is not conceptually difficult (but its detailed analysis of performances is however): it is just a binary search where each query point is sampled enough time to be sure on which "direction" the search should proceed next. Indeed, because of the concavity and monotone assumptions on f 1 and f 2 , if x < x then

x < x ⇐⇒ ∇g(x) = ∇f 1 (x) -∇f 2 (1 -x) < 0 .
By getting enough noisy samples of ∇g(x), it is possible to decide, based on its sign, whether x lies on the right or the left of x. If x j is the j-th point queried by the binary search (and letting j max be the total number of different queries), we get that the binary search is successful with high probability, i.e., that with probability at least 1 -δT for each j ∈ {1, . . . , j max }, |x j -x | ≤ 2 -j . We also call N j the actual number of samples of x j which is bounded by 8 log(2T /δ)/|g (x j ) 2 | by Lemma 3.5.

Lemma 3.5. Let x ∈ [-1, 1] and δ ∈ (0, 1). For any random variable X ∈ [x -1, x + 1] of expectation x, at most N x = 8 x 2 log (2T /δ) i.i.d. samples X 1 , X 2 , .
. . , X n are needed to figure out the sign of x with probability at least 1 -δ. Indeed, one just need stop sampling as soon as

0 ∈   1 n n t=1 X t ± 2 log(2T /δ) n  
and determine the sign of x is positive if

1 n n t=1 X t ≥ 2 log(2T /δ)
n and negative otherwise.

Proof. This lemma is just a consequence of Hoeffding inequality. Indeed, it implies that, at stage n ∈ N,

P 1 n n t=1 X t -x ≥ 2 log( 2T δ ) n ≤ δ T ,

124

thus with probability at least 1 -δ, x belongs to

1 Nx Nx t=1 X t ± 2 log( 2T δ ) Nx
and the sign of x is never mistakenly determined.

On the other hand, at stage N x , it holds on the same event that 1 Nx Nx t=1 X t is x 2 -close to x, thus 0 no longer belongs to the interval

1 Nx Nx t=1 X t ± 2 log( 2T δ ) Nx .
The regret of the algorithm then rewrites as

R(T ) = 1 T T t=1 |g(x)| = 1 T jmax j=1 N j |g(x j )| + δ ≤ 8 T log(2T /δ) jmax j=1 |g(x j )| g (x j ) 2 + δ . (3.1)
Our analysis of the algorithm performances are based on the control of the last sum in Equation (3.1).

Strongly concave functions

First, we consider the case where the functions f 1 and f 2 are strongly concave.

Theorem 3.1. Assume A3.1 and that g is a L -smooth and α-strongly concave function on [0, 1]. If Algorithm 3.1 is run with δ = 2/T 2 , then there exists a universal positive constant κ such that

ER(T ) ≤ κ α log(T ) T .
This results shows that our algorithm reaches the same rates as the stochastic gradient descent in the smooth and strongly concave case.

Proof. Let j ∈ [j max ]. By concavity of g, we have that -g(x j ) ≤ |g (x j )||x -x j |. Since g is negative, this means that |g(x j )| ≤ |g (x j )||x -x j |.
Since g is of class C 2 and α-strongly concave,

g (x j ) -g (x )|x j -x ≤ -α x j -x 2 -α x j -x 2 ≥ g (x j ) -g (x )|x j -x ≥ -|g (x j )| x j -x |g (x j )| ≥ α x j -x . Then |g(x j )| g (x j ) 2 ≤ |g (x j )||x -x j | g (x j ) 2 = |x -x j | |g (x j )| ≤ 1 α .
Consequently we have

R(T ) ≤ j max T α .
We have for all j ∈ [j max ],

N j = 2 log(2T /δ) 1 g (x j ) 2 . Then T = 8 log(2T /δ) jmax j=1 1 g (x j ) 2 ≥ 8 log(2T /δ) jmax j=1 1 L 2 (x j -x ) 2 ≥ 8 log(2T /δ) 1 L 2 (x jmax -x ) 2 ≥ 8 log(2T /δ) 4 jmax L 2 .

125

where we used the fact that g is L -Lipschitz continuous. Therefore j max ≤ log 4 T L 2 8 log(2T /δ) log(T ). And finally

R(T ) = O 1 α log(T ) T .

Analysis in the non-strongly concave case

We now consider the case where g is only concave, without being necessarily strongly concave.

Theorem 3.2. Assume A3.1 and that g satisfies the local Łojasiewicz inequality w.r.t. β ≥ 1 and c > 0. If Algorithm 3.1 is run with δ = 2/T 2 , then there exists a universal constant κ > 0 such that in the case where

β > 2, E[R(T )] ≤ κ c 2/β L 1-2/β 1 -2 2/β-1 log(T ) T ;
in the case where

β ≤ 2, E[R(T )] ≤ κc log(T ) 2 T β/2
.

The proof of Theorem 3.2 relies on bouding the sum in Equation (3.1), which can be recast as a constrained minimization problem. It is postponed to Appendix 3.A for clarity reasons.

Lower bounds

We now provide a lower bound for our problem that indicates that our rates of convergence are optimal up to poly(log(T )) terms. For β ≥ 2, it is trivial to see that no algorithm can have a regret smaller than Ω(1/T ), hence we shall focus on β ∈ [1, 2]. Theorem 3.3. Given the horizon T fixed, for any algorithm, there exists a pair of functions f 1 and f 2 that are concave, non-decreasing and such that f i (0) = 0, such that

ER(T ) ≥ c β T -β 2 ,
where c β > 0 is some constant independent of T .

The proof and arguments are rather classical now [START_REF] Shamir | On the Complexity of Bandit and Derivative-Free Stochastic Convex Optimization[END_REF][START_REF] Bach | Highly-Smooth Zero-th Order Online Optimization[END_REF]: we exhibit two pairs of functions whose gradients are 1/ √ T -close with respect to the uniform norm. As no algorithm can distinguish between them with arbitrarily high probability, the regret will scale more or less as the difference between those functions which is as expected of the order of T -β/2 . More details can be found in Appendix 3.B.

The specific case of linear (or dominating) resources -the Multi-Armed Bandit case

We focus in this section on the specific case where the resources have linear efficiency, meaning that f i (x) = α i x for some unknown parameter α i ≥ 0. In that case, the optimal allocation of resource consists in putting all the weights to the resource with the highest parameter α i .

More generally, if f 1 (1) ≥ f 2 (0), then one can easily check that the optimal allocation consists in putting again all the weight to the first resource (and, actually, the converse statement is also true).

It happens that in this specific case, the learning is fast as it can be seen as a particular instance of Theorem 3.2 in the case where β > 2. Indeed, let us assume that arg max x∈R g(x) > 1, meaning that max x∈[0,1] g(x) = g(1), so that, by concavity of g it holds that g (x) ≥ g (1) > 0 thus g is increasing on [0, 1]. In particular, this implies that for every β > 2:

∀x ∈ [0, 1], g(1) -g(x) = |g(x)| ≤ g(0) ≤ g(0) g (1) β g (1) β ≤ g(0) g (1) β g (x) β = c|g (x)| β ,
showing that g verifies the Łojasiewicz inequality for every β > 2 and with constant c = g(0)/g (1) β . As a consequence, Theorem 3.2 applies and we obtain fast rates of convergence in O (log(T )/T ). However, we propose in the following an alternative analysis of the algorithm for that specific case. Recall that regret can be bounded as

R(T ) = 8 T log(2T /δ) jmax j=1 |g(x j )| g (x j ) 2 = 8 T log(2T /δ) jmax j=1 |g(1 -1/2 j )| g (1 -1/2 j ) 2 .
We now notice that

g 1 -2 -j = g(1) -g 1 -2 -j = 1 1-1/2 j g (x) dx ≤ 2 -j g 1 -2 -j .
And finally we obtain the following bound on the regret:

R(T ) ≤ 8 T log(2T /δ) jmax j=1 1 2 j 1 g (1) ≤ 8 T log(2T /δ) g (1) ≤ 24 ∆ log(T ) T since g (1 -1/2 j ) > g (1)
and with the choice of δ = 2/T 2 . We have noted ∆ g (1) in order to enlighten the similarity with the multi-armed bandit problems with 2 arms. We have indeed g (1) = f 1 (1) -f 2 (0) > 0 which can be seen as the gap between both arms. It is especially true in the linear case where f i (x) = α i x as ∆ = |α 1 -α 2 | and the gap between arms is by definition of the multi-armed bandit problem |f (1)-f (0

)| = |α 1 -α 2 |.

Stochastic gradient feedback for K ≥ 3 resources

We now consider the case with more than 2 resources. The generic algorithm still relies on binary searches as in the previous section with K = 2 resources, but we have to imbricate them in a tree-like structure to be able to leverage the Łojasiewicz inequality assumption. The goal of this section is to present our algorithm and to prove the following theorem, which is a generalization of Theorem 3.2.

Theorem 3.4. Assume A3.1 and that F = {f 1 , f 2 , . . . , f K } satisfies inductively the Łojasiewicz inequality w.r.t. the parameters β F ≥ 1 and c > 0. Then there exists a universal constant κ > 0 such that our algorithm, run with δ = 2/T 2 , ensures

in the case β F > 2, E[R(T )] ≤ κ c 2/β F L 1-2/β F 1 -2 2/β F -1 K log(T ) log 2 (K) T ; in the case β F ≤ 2, E[R(T )] ≤ κcK log(T ) log 2 (K)+1 T β F /2 .
Let us first mention why the following natural extension of the algorithm for K = 2 does not work. Assume that the algorithm would sample repeatedly a point x ∈ ∆ K until the different confidence intervals around the gradient ∇f k (x k ) do not overlap. When this happens with only 2 resources, then it is known that the optimal x allocates more weight to the resource with the highest gradient and less weight to the resource with the lowest gradient. This property only holds partially for K ≥ 3 resources. Given x ∈ ∆ K , even if we have a (perfect) ranking of gradient ∇f 1 (x 1 ) > . . . > ∇f K (x K ) we can only infer that x 1 ≥ x 1 and x K ≤ x K . For intermediate gradients we cannot (without additional assumptions) infer the relative position of x j and x j .

To circumvent this issue, we are going to build a binary tree, whose leaves are labeled arbitrarily from {f 1 , . . . , f K } and we are going to run inductively the algorithm for K = 2 resources at each node, i.e., between its children f left and f right . The main difficulty is that we no longer have unbiased samples of the gradients of those functions (but only those located at the leaves).

Detailed description of the main algorithm

To be more precise, recall we aim at maximizing the mapping (and controlling the regret)

F (x) = K k=1 f k (x k ) with x = (x 1 , . . . , x K ) ∈ ∆ K .
As we have a working procedure to handle only K = 2 resources, we will adopt a divideand-conquer strategy by diving the mapping F into two sub-mapping F

(1) 1 and F

(1) 2 defined by

F (1) 1 (x) = K/2 k=1 f k (x k )
and F

(1)

2 (x) = K k= K/2 +1 f k (x k ).
Since the original mapping F is separable, we can reduce the optimization of F over the simplex ∆ K to the optimization of a sum of two functions over the simplex of dimension 1 (thus going back to the case of K = 2 resources). Indeed, max

x 1 =1 F (x) = max z∈[0,1] max x 1 =z F (1) 1 (x) + max x 1 =1-z F (1) 2 (x) max z∈[0,1] H (1) 1 (z) + H (1) 2 (1 -z) .
The overall idea is then to separate arms recursively into two bundles, creating a binary tree whose root is F and whose leaves are the f k . We explain in this section the algorithm, introducing the relevant definitions and notations for the proof.

We will denote by F (i) j the function created at the nodes of depth i, with j an increasing index from the left to the right of the tree; in particular

F (0) 1 = F = K k=1 f k (x k )
. This is the function we want to maximize.

Definition 3.5. Starting from

F (0) 1 = F = K k=1 f k (x k ), the functions F (i) j
are constructed inductively as follows. If

F (i) j (x) = k 2 k=k 1 f k (x k ) is not a leaf (i.e., k 1 < k 2 ) we 128 define F (i+1) 2j-1 (x) = (k 1 +k 2 )/2 k=k 1 f k (x k ) and F (i+1) 2j (x) = k 2 k= (k 1 +k 2 )/2 +1 f k (x k ) .
The optimization of F (i) j can be done recursively since, for any z n ∈ [0, 1], max

x 1 =zn F (i) j (x) = max z n+1 ∈[0,zn] max x 1 =z n+1 F (i+1) 2j-1 (x) + max x 1 =zn-z n+1 F (i+1) 2j (x) .
The recursion ends at nodes that are parents of leaves, where the optimization problem is reduced to the case of K = 2 resources studied in the previous section.

For the sake of notations, we introduce the following functions.

Definition 3.6. For every i and j in the constructed binary tree of functions,

H (i) j (z) max x 1 =z F (i) j (x) and
G (i) j (z; y) H (i+1) 2j-1 (z) + H (i+1) 2j (y -z).
With these notations, it holds that for all z n ∈ [0, 1],

H (i) j (z n ) = max z n+1 ∈[0,zn] G (i) j (z n+1 ; z n ) = max z n+1 ∈[0,zn] H (i+1) 2j-1 (z n+1 ) + H (i+1) 2j (z n -z n+1 ).
In order to compute H (i) j (z n ), we aim to apply the machinery of K = 2 resources to the reward mappings H are not basis functions in F and F can be "divided" to compute recursively the gradients of each H (k) j up to H

(1) 1 and H

(1) 2 , up to the noise and some estimation errors that must be controlled.

As a consequence, the gradients of H (i) j can be recursively approximated using estimates of the gradients of their children (in the binary tree). Indeed, assume that one has access to ε-approximations of ∇H . Then Lemma 3.6 directly implies that a ε-approximation of its gradient ∇H (i) j (z) can be computed by a binary search on [0, z]. Moreover, notice that if a binary search is optimizing H (i) j on [0, z] and is currently querying the point ω, then the level of approximation required (and automatically set to) is equal to

|∇H (i+1) 2j-1 (ω) -∇H (i+1) 2j
(z -ω)|. This is the crucial property that allows a control on the regret.

The main algorithm can now be simply summarized as performing a binary search for the maximization of H

(1)

1 (z) + H (1)
2 (1 -z) using recursive estimates of ∇H We detail now how to perform those binary searches. In order to compute H (i) j (z), one has to maximize the function u → G (i) j (u; z) (see Definition 3.6). In order to maximize it, we run a binary search over [0, z], starting at u 1 = z/2: Definition 3.7. We note

D (i) j (v) the binary search run to maximize w → G (i) j (w; v) . We define z (i) j (v) as arg max G (i) j (• ; v) and we also call T (i) j (v) the total number of queries used by D (i) j (v).
Inductively, the binary search D

i) j (u m ; z n ) = ∇H (i+1) 2j-1 (u m )-∇H (i+1) 2j
(z n -u m ), we need to further estimate ∇H (z nu m ). This is done using Lemma 3.6.

Thanks to Lemma 3.6 we are able to compute the gradients ∇G (i) j (v; u) for all nodes in the tree. This is done recursively by imbricating dichotomies.

The goal of the binary searches

D (i+1) 2j-1 (v) and D (i+1) 2j (u -v) is to compute an approx- imate value of ∇G (i) j (v; u). Indeed we have ∇G (i) j (v; u) = ∇H (i+1) 2j-1 (v) -∇H (i+1) 2j (u -v) ,
and to compute H

(i+1) 2j-1 (v) (respectively ∇H (i+1) 2j (u -v))
we need to run the binary search

D (i+1) 2j-1 (v) (respectively D (i+1) 2j (u-v)). Let us denote by ∇G (i) j (v; u) the approximate value of ∇G (i) j (v; u) computed at the end of the binary searches D (i+1) 2j-1 (v) and D (i+1) 2j (u -v), that compute themselves ∇H (i+1) 2j-1 (v), approximation of ∇H (i+1) 2j-1 (v) and ∇H (i+1) 2j (u -v), approximation of ∇H (i+1) 2j (u -v).
The envelop theorem gives that ∇H

(i+1) 2j-1 (v) = ∇H (i+2) 4j-3 (w ) = ∇H (i+2) 4j-2 (v -w ) where w = arg max G (i+1) 2j-1 (w; v). Therefore in order to compute ∇H (i+1) 2j-1 (v) we run the binary search D (i+1) 2j-1 (v) that aims at maximizing the function w → G (i+1) 2j-1 (w; v) . At iteration N of D (i+1) 2j-1 (v), we have |∇G (i+1) 2j-1 (w N ; v)| = |∇H (i+2) 4j-3 (w N ) -∇H (i+2) 4j-2 (v -w N )| . 130
We use the following estimate for ∇H

(i+1) 2j-1 (v): ∇H (i+1) 2j-1 (v) 1 2 ∇H (i+2) 4j-3 (w N ) + ∇H (i+2) 4j-2 (v -w N ) . Since w ∈ (w N , v -w N ) (or (v -w N , w N )), we have that | ∇H (i+1) 2j-1 (v) -∇H (i+1) 2j-1 (v)| ≤ 1 2 |∇G (i+1) 2j-1 (w N ; v)| .
Consequently we can say that with high probability,

∇G (i) j (v; u) ∈ ∇G (i) j (v; u) -α, ∇G (i) j (v; u) + α , where α = 1 2 |∇G (i+1) 2j-1 (w N ; v)| + |∇G (i+1) 2j (v -w N ; v)| .
In order to be sure that the algorithm does not make an error on the sign of ∇G (i) j (v; u) (as in Section 3.3) we have to run the binary searches

D (i+1) 2j-1 (v) and D (i+1) 2j (u -v) until 0 / ∈ ∇G (i) j (v; u) -α, ∇G (i) j (v; u) + α which is the case as soon as α < |∇G (i) j (v; u)|. Therefore we decide to stop the binary D (i+1) 2j-1 (v) when |∇G (i+1) 2j-1 (w N ; v)| < |∇G (i) j (v; u)| and to stop the binary D (i+1) 2j (u -v) when |∇G (i+1) 2j (v -w N ; v)| < |∇G (i) j (v; u)|.
This leads to the following lemma:

-1 (w; v)| ≥ |∇G (i) j (v)| . (i+1) 2j
And during the binary search D (i+1) 2j

(v) we have, for all point w tested by this binary search,

|∇G (i+1) 2j (v -w; v)| ≥ |∇G (i) j (v)| .

Analysis of the algorithm

Before going on with the proof of Theorem 3.4, we begin with a very natural intuition in the case of strongly concave mappings or β > 2, as well as the main ingredients of the general proof.

Recall that in the case where β > 2, the average regret of the algorithm for K = 2 scales as log(T )/T . As a consequence, running a binary search induces a cumulative regret of the order of log(T ). The generic algorithm is defined recursively over a binary tree of depth log 2 (K) and each function in the tree is defined by a binary search over its children. So at the end, to perform a binary search over H

(1)

1 (z) + H (1)
2 (1 -z), the algorithm imbricates log 2 (K) binary searches to compute gradients. The error made by these binary searches cumulate (multiplicatively) ending up in a cumulative regret term of the order of log(T ) log 2 (K) .

For β < 2, the analysis is more intricate, but the main idea is the same one; to compute a gradient, log 2 (K) binary searches must be imbricated and their errors cumulate to give Theorem 3.4.

In order to analyze our algorithm, we associate a regret for each binary search.

Definition 3.8. We define R

(i) j (w; v) .
This notion of subregret is crucial for our induction since the regret of the algorithm after T samples satisfies R(T ) = R (0) 0 (1)/T . Since we have more than 2 resources we have to imbricate the binary searches in a recursive manner in order to get access to the gradients of the functions H (i) j . This will lead to a regret R (i) j (v) for the binary search D (i) j (v) that will recursively depend on the regrets of the binary searches corresponding to the children (in the tree) of D (i) j (v). This will lead to the following proposition, which is one of the main ingredients of the proof of Theorem 3.4.

Proposition 3.7. The regret R (i) j (v) of the binary search D (i) j (v) is bounded by: R (i) j (v) ≤ rmax r=1 8 log(2T /δ) g (i) j (w r ; v) ∇g (i) j (w r ; v) 2 log(T ) log 2 (K)-1-i + R (i+1) 2j-1 (w r ) + R (i+1) 2j (v -w r ),
where {w 1 , . . . , w rmax } are the different samples of

D (i) j (v) and g (i) j (• ; v) G (i) j (• ; v) - max z G (i) j (z; v).
This proposition is a direct consequence of the following two lemmas: Lemma 3.8 gives an expression to compute the subregret R (i) j (v) and Lemma 3.9 gives a bound on the number of samples needed to compute ∇G (i) j (w; v) at a given precision.

Proof. The statement of Proposition 3.7 is a restatement of Lemma 3.8 using the fact that each different point of the binary search

D (i) j (v) is sampled a number of times equal to 8 log(2T /δ) log(T ) p ∇G (i) j (w; v)
2 thanks to Lemma 3.9. The fact that r max ≤ log 2 (T ) comes from the fact that running a binary search to a precision smaller than 1/LT does not give improved bound on the regret since the reward functions are L-Lipschitz continuous. Therefore the binary searches are stopped after more than log 2 (T ) samples.

Lemma 3.8. The subregret

R (i) j (v) verifies R (i) j (v) = T (i) j (v) t=1 G (i) j (zij(t); v) -G (i) j (z (i) j (v); v)) + z∈{zij(t),t=1,...,T (i) j (v)} R (i+1) 2j-1 (z) + R (i+1) 2j (v -z)
where z

(i) j (v)
is the point where G Lemma 3.9. A point w tested by the binary search D (i) j (v) has to be sampled at most a number of times equal to

8 log(2T /δ) log(T ) p ∇G (i) j (w; v) 2 ,
where p is the distance of the node D (i) j (v) to the bottom of the binary tree: p = log 2 (K) -1 -i.

D (i) j (v) is a leaf) D (i+1) 2j-1 (w m ) and D (i+1) 2j (v -w m ).
Let us now prove the result by recurrence on the distance p of D (i) j to the closest leaf of the tree.

• p = 0:

D (i)
j is a leaf. The point w m needs to be sampled 8 log(2T /δ)/ ∇g

2j-1 (w m ). This binary search is at distance p -1 of the closest leaf. Therefore by hypothesis recurrence each point x k will be sampled a number of times equal to

N k = 8 log(2T /δ) log(T ) p-1 ∇G (i+1) 2j-1 (x k ) 2 . Now Lemma 3.7 shows that ∇G (i+1) 2j-1 (x k ) ≥ ∇G (i) j (w m ) . This gives N k ≤ 8 log(2T /δ) log(T ) p-1 ∇G (i) j (w m )
.

The same reasoning applies for the binary search D .

This proves the result for the step p. • Finally the recurrence is complete and the result is shown.

Finally it now remains to control the different ratios g

(i) j (w r ; v) / ∇g (i) j (w r ; v) 2
, using the Łojasiewicz inequality and techniques similar to the case of K = 2. The main difference is the binary tree we construct that imbricates binary searches. The overall idea is that each layer of that tree adds a multiplicative factor of log(T ).

The goal of the remaining of the proof of Theorem 3.4 is to bound R (0)

1 (1). The very natural way to do it is to use the previous proposition with the Łojasiewicz inequality to obtain a simple recurrence relation between the successive values of R (i) j . The end of the proof is then similar to the proofs done in the case K = 2. Finally we can note that the statement of Proposition 3.7 shows clearly that adding more levels to the tree results in an increase of the exponent of the log(T ) factor. The detailed proof is postponed to Appendix 3.C for clarity reasons.

Numerical experiments

In this section, we illustrate the performances of our algorithm on generated data with K = 2 resources. We have considered different possible values for the parameter β ∈ [1, ∞).

In the case where β = 2 we have considered the following functions:

f 1 : x → 5 6 - 5 48 (2 -x) 3 and f 2 : x → 6655 384 - 5 48 11 5 -x 3 , such that g(x) = -(x -0.4) 2 .
g verifies the Łojasiewicz inequality with β = 2 and the functions f 1 and f 2 are concave, non-decreasing and take value 0 at 0. We have computed the cumulated regret of our algorithm in various settings corresponding to different values of β and we have plotted the two references rates: the lower bound T -β/2 (even if the functions considered in our examples are not those used to prove the lower bound), and the upper bound (T / log 2 (T )) -β/2 .

0 5 • 10 5 1 • 10 6 1.5 • 10 6 2 • 10 6 1 • 10 -3 2 • 10 -3 3 • 10 -3 T T / log(T ) 2 -β/2 R(T ) T -β/2 (a)
0 5 • 10 5 1 • 10 6 1.5 • 10 6 2 • 10 6 1 • 10 -3 2 • 10 -3 3 • 10 -3 T T / log(T ) 2 -β/2 R(T ) T -β/2 (a)
Our experimental results on Figures 3.1, 3.2, 3.3 and 3.4 indicate that our algorithm has the correct expected behavior, as its regret is "squeezed" between T -β/2 and (T / log 2 (T )) -β/2 for β ≤ 2 and between T -1 and log(T )/T for β ≥ 2. Moreover, the loglog scale also illustrates that -β/2 is indeed the correct speed of convergence for functions that satisfy the Łojasiewicz inequality with respect to

β ∈ [1, 2]. 0 5 • 10 5 1 • 10 6 1.5 • 10 6 2 • 10 6 1 • 10 -3 2 • 10 -3 3 • 10 -3 T log(T ) 2 /T R(T ) T -1 (

Conclusion

We have considered the problem of multi-resource allocation under the classical assumption of diminishing returns. This appears to be a concave optimization problem and we proposed an algorithm based on imbricated binary searches to solve it. Our algorithm is particularly interesting in the sense that it is fully adaptive to all parameters of the problem (strong convexity, smoothness, Łojasiewicz exponent, etc.). Our analysis provides meaningful upper bound for the regret that matches the lower bounds, up to logarithmic factors. The experiments we conducted validate as expected the theoretical guarantees of our algorithm, as empirically regret seems to decrease polynomially with T with the right exponent.

3.A Analysis of the algorithm with K = 2 resources

In this section we prove Theorem 3.2. We split the proof into 3 subsections, depending on the different values of β.

3.A.1 Proof of Theorem 3.2, when

β > 2 Proof. Let x ∈ [0, 1]. We know that |g(x)| ≤ c|g (x)| β . Then 1 |g (x)| 2 ≤ c 2/β |g(x)| 2/β , and
|g(x)| |g (x)| 2 ≤ c 2/β |g(x)| 1-2/β . Since g is L-Lipschitz on [0, 1], we have |g(x) -g(x )| ≤ L|x -x |. Since g(x ) = 0 then |g(x)| |g (x)| 2 ≤ c 2/β L 1-2/β |x -x | 1-2/β . For j ∈ [j max ], |g(x j )| |g (x j )| 2 ≤ c 2/β L 1-2/β 1 2 1-2/β j , because |x -x j | ≤ 2 -j , as a conse- 136 quence of the binary search. Since 1 -2/β > 0, jmax j=1 1 2 1-2/β j < 1 1 -2 2/β-1 .
Finally we have, using that δ = 2/T2 ,

R(T ) = 8 T log(2T /δ) jmax j=1 |g(x j )| |g (x j )| 2 ≤ 24c 2/β L 1-2/β 1 -2 2/β-1 log(T ) T .

3.A.2 Proof of Theorem 3.2, when β = 2

Proof. As in the previous proof, we want to bound

R = jmax j=1 |g(x j )| g (x j ) 2 ≤ jmax j=1 min c, 1 g j 2 j .
Let us note ĝj 1 c2 j , we have to distinguish two cases:

       if g j > ĝj , then min c, 1 g j 2 j = 1 2 j g j if g j < ĝj , then min c, 1 g j 2 j = c .
We note J 1 {j ∈ [j max ], g j > ĝj } and J 2 {j ∈ [j max ], g j < ĝj }.

We have

R ≤ j∈J1 1 2 j g j R1 + j∈J2 c R2 .
We note as well

T 1 j∈J1 1 g 2 j and T 2 j∈J2 1 g 2 j such that T = T 1 + T 2 .
We now analyze J 1 and J 2 separately.

(a) on J 2 :

T 2 = j∈J2 1 g 2 j > j∈J2 1 ĝ2 j ≥ j∈J2 c 2 4 j ≥ 4 j2,max . Which gives j 2,max ≤ log(T ). Finally, R 2 = j∈J2 c ≤ cj 2,max ≤ c log(T ) . (b) on J 1 : We want to maximize R 1 = j∈J1 1 2 j g j under the constraint T 1 = j∈J1 1 g 2 j .
Karush-Kuhn-Tucker conditions give the existence of λ > 0 such that for all j ∈ J 1 , g j = 2λ • 2 j . As in the previous proof this shows that R 1 = 2λT 1 . We can show as well that, if

j ∈ J 1 , 2λ ≤ 2 √ And since j ∈ J 1 , g j > 1 c2 j and then 2λ 2 j > 1 c2 j which means 2λ > 1 c4 j1,min .
Putting these inequalities together gives

T 1 ≤ 2c √ 3 2 j1,min .
Finally,

R 1 = 2λT 1 ≤ 2 √ 3 2 -j1,min √ T 1 T 1 ≤ 4c 3 .
This shows that

R(T ) c log(2T /δ) log(T ) T c log(T ) 2 T .

3.A.3 Proof of Theorem 3.2, when β < 2

Proof. We know that

R(T ) = 1 T jmax j=1 |g(x j )|N j = 8 log(2T /δ) 1 T jmax j=1 |g(x j )| h 2 j ≤ 8 log(2T /δ) 1 T jmax j=1 |g(x j )| g (x j ) 2 .
where h j ≥ g j is such that

N j = 8 log(2/δ) h 2 j . We note R T R(T ) 8 log(2T /δ) = jmax j=1 |g(x j )| h 2 j . By hypothesis, ∀x ∈ [0, 1], |g(x)| ≤ c|g (x)| β . Moreover Proposition 3.2 gives |g(x j )| ≤ |g (x j )||x j -x | ≤ |g (x j )|2 -j . If we note g j |g (x j )| we obtain R ≤ jmax j=1 min cg β j , g j 2 j 1 h 2 j .
Let us now note T T 8 log(2T /δ) .

We have the constraint

T = jmax j=1 1 h 2 j .
Our goal is to bound R. In order to do that, one way is to consider the functional

F : (g 1 , . . . , g jmax ) ∈ R * + jmax → jmax j=1 min cg β j , g j 2 j /h 2 j 138
and to maximize it under the constraints

T = jmax j=1 1 h 2 j and g j ≤ h j .
Therefore the maximum of the previous problem is smaller than the one of maximizing

F : (h 1 , . . . , h jmax ) ∈ R * + jmax → jmax j=1 min ch β-2 j , 1 h j 2 j
and to maximize it under the constraints

T = jmax j=1 1 h 2 j .
For the sake of simplicity we identify g j with h j . The maximization problem can be done with Karush-Kuhn-Tucker conditions: introducing the Lagrangian

L(g 1 , . . . , g jmax , λ) = F(g 1 , . . . , g jmax ) + λ   T - jmax j=1 1 h 2 j   , we obtain ∂L ∂g j =        c(β -2)g β-3 j + 2λ g 3 j , if g j < ĝj - 1 2 j g j + 2λ g 3 j , if g j > ĝj , where ĝj = 1 2 j c 1/(β-1)
. ĝj is the point where the two quantities in the min are equal. And finally

     g j = 2λ c(2 -β) 1/β , if g j < ĝj g j = 2λ • 2 j , if g j > ĝj .
We note J 1 {j ∈ [j max ], g j > ĝj } and J 2 {j ∈ [j max ], g j < ĝj }. We have

F(g 1 , . . . , g jmax ) = j∈J1 1 2 j g j F1 + j∈J2 cg β-2 j F2 .
We note as well

T 1 j∈J1 1 g 2 j and T 2 j∈J2 1 g 2 j such that T = T 1 + T 2 .
We again analyze J 1 and J 2 separately.

(a) on J 2 :

Since g j < ĝj on J 2 , noting g 2 2λ c(2 -β) 1/β = g j , T 2 = j∈J2 1 g 2 j = |J 2 | 1 g 2 2 > |J 2 | 1 ĝ2 j for all j ∈ J 2 .
In particular,

T ≥ T 2 > |J 2 | 1 c 2 4 j2,max -1/(β-1) ≥ |J 2 | c 2 4 |J2| 1/(β-1) ≥ 4 |J2| 1/(β-1) ,
139 because c can be chosen greater than 1. This gives

|J 2 | ≤ β -1 log(4) log(T ).
And we know that

T 2 = j∈J2 1 g 2 j = |J 2 | 2λ c(2 -β) -2/β . This gives 2λ c(2 -β) = T 2 |J 2 | -β/2
.

We can now compute the cost of J 2 :

F 2 = j∈J2 cg β-2 j = |J 2 |c 2λ c(2 -β) (β-2)/β = |J 2 |c T 2 |J 2 | 1-β/2 = cT 1-β/2 2 |J 2 | β/2 ≤ cT 1-β/2 2 β -1 log(4) log(T ) β/2 cT log(T ) T β/2 . (b) on J 1 :
We know that ∀j ∈ J 1 , g j = 2λ 2 j . This gives

T 1 = j∈J1 1 g 2 j = 1 4λ 2 j∈J1 1 4 j 2λ = j∈J1 4 -j T 1 2λ ≤ 4 • 4 -j1,min 3T 1 .
Since j ∈ J 1 , we know that g j ≥ ĝj and 2λ 2 j ≥ 1 2 j c 1/(β-1)

, and 2λ ≥ c -1/(β-1) (2 j ) -β/(β-1) .

With j = j 1,min we obtain

c -1/(β-1) (2 j1,min ) -β/(β-1) ≤ 4 • 4 -j1,min 3T 1 √ 3 2 2 j1,min -1/(β-1) c -1/(β-1) ≤ 1 √ T 1 c -2 4 -j1,min T 1-β 1 .
And we have

F 1 = j∈J1 1 2 j 2λ 2 j = 1 2λ j∈J1 1 4 j = 2λT 1 T 1 2 -j1,min cT 1-β/2 1 cT 1-β/2 . Finally we have shown that R cT log(T ) T β/2 and consequently T R(T ) 8 log(2T /δ) c T 8 log(2T /δ) log(T ) T β/2 R(T ) c (8 log(2T /δ)) β/2 log(T ) T β/2
.

And using the fact that β < 2 and δ = 2/T 2 , we have

R(T ) c log(T ) 2 T β/2 .

3.B Analysis of the lower bound

In this section we prove Theorem 3.3.

Proof. The proof is very similar to the one of [START_REF] Shamir | On the Complexity of Bandit and Derivative-Free Stochastic Convex Optimization[END_REF] (see also [START_REF] Bach | Highly-Smooth Zero-th Order Online Optimization[END_REF])) so we only provide the main different ingredients.

Given T and β, we are going to construct 2 pairs of functions f 1 , f 2 and f 1 , f 2 such that

f i -f i ∞ ≤ c β √ T and ∇f i -∇ f i ∞ ≤ c β √ T .
As a consequence, using only T samples5 , it is impossible to distinguish between the pair f 1 , f 2 and the pair f 1 , f 2 . And the regret incurred by any algorithm is then lower-bounded (up to some constant) by min

x max{g -g(x) ; g -g(x)}
where we have defined g(x) = f 1 (x) + f 2 (1 -x) and g = max x g(x) and similarly for g.

To define all those functions, we first introduce g and g defined as follows, where γ is a parameter to be fixed later.

g : x →    -x β/(β-1) if x ≤ γ - β β -1 γ 1/(β-1) x + 1 β -1 γ β/(β-1) otherwise and g : x →    -|x -γ| -β/(β-1) if x ≤ 2γ - β β -1 γ 1/(β-1) x + β + 1 β -1 γ β/(β-1) otherwise. .
The functions have the form of Proposition 3.5 near 0 and then are linear with the same slope. Proposition 3.5 ensures that g 1 and g 2 verify the Łojasiewicz inequality for the parameter β.

The functions g 1 and g 2 are concave non-positive functions, reaching their respective maxima at 0 and γ.

We also introduce a third function h defined by

h : x →            (γ -x) β/(β-1) -x β/(β-1) if γ 2 ≤ x ≤ γ 2 β β -1 ( γ 2 ) 1/(β-1) ( γ 2 -x) if x ≤ γ 2 - β β -1 γ 1/(β-1) x + 1 β -1 γ β/(β-1) if x ≥ γ .
The functions f i and f i are then defined as

f 1 (x) = 0 and f 1 (x) = g(x) -g(x) + h(x) -g(0) -g(0) + h(0) f 2 (x) = g(1 -x) -g(1) and f 2 (x) = g(1 -x) -h(1 -x) -g(1) + h(1) It immediately follows that f 1 (x) + f 2 (1 -x) is equal to g(x) and similarly f 1 (x) + f 2 (1 -x)
is equal to g(x) (both up to some additive constant).

We observe that for all x ∈ [0, 1]:

∇g(x) =      - β β -1 x 1/(β-1) if x ≤ γ - β β -1 γ 1/(β-1) otherwise and ∇ g(x) =      - β β -1 sign(x -γ)|x -γ| 1/(β-1) if x ≤ 2γ - β β -1 γ 1/(β-1) otherwise .
Similarly, we can easily compute the gradient of h:

∇h(x) =            -β β-1 (γ -x) 1/(β-1) + x 1/(β-1) if γ 2 ≤ x ≤ γ -2 β β -1 ( γ 2 ) 1/(β-1) if x ≤ γ 2 - β β -1 γ 1/(β-1) if x ≥ γ . We want to bound ∇g -∇ g ∞ as it is clear that ∇h ∞ ≤ β β-1 γ 1/(β-1) . • For x ≤ γ, |∇g(x) -∇ g(x)| = β β -1 -x 1/(β-1) -(γ -x) 1/(β-1) = β β -1 x 1/(β-1) + (γ -x) 1/(β-1) ≤ β β -1 x 1/(β-1) + (γ -x) 1/(β-1) ≤ 2 β β -1 γ 1/(β-1) . • For γ ≤ x ≤ 2γ, |∇g(x) -∇ g(x)| = β β -1 (x -γ) 1/(β-1) -x 1/(β-1) ≤ β β -1 (x -γ) 1/(β-1) + x 1/(β-1) ≤ (1 + 2 1/(β-1) ) β (β -1) γ 1/(β-1) . • For x ≥ 2γ, |∇g(x) -∇ g(x)| = 0.
Finally we also have that ∇g -∇ g ∞ γ 1/(β-1) , where the notation hides a multiplicative constant factor.

Combining the control on ∇g -∇ g ∞ and ∇h ∞ , we finally get that

∇f 1 -∇ f 1 ∞ γ 1/(β-1) and ∇f 2 -∇ f 2 ∞ γ 1/(β-1) .
As a consequence, the specific choice of γ = T (1-β)/2 ensures that γ 1/(β-1) ≤ 1/ √ T and thus the mappings f i are indistinguishable from the f i .

Finally, we get

R(T ) ≥ T min x max(|g(x)|, | g(x)|) ≥ T g(γ/2) γ β/(β-1) T -β/2 .

3.C Analysis of the algorithm with K ≥ 3 resources

We provide here the complete proof of Theorem 3.4. As before, we divide it into 3 subsections, depending on the value of β.

We begin with a very simple arithmetic lemma that will be useful in the following.

Lemma 3.10. Let (u n ) n∈N ∈ N N defined as follows:

u 0 = 1 and u n+1 = 2u n + 1. Then ∀n ∈ N, u n = 2 n+1 -1. Proof. Let consider the sequence v n = u n +1. We have v 0 = 2 and v n+1 = 2v n . Consequently v n = 2 • 2 n = 2 n+1 .

3.C.1 Proof of Theorem 3.4 with β > 2

Proof. Let us first bound a sub-regret R (i) j (v) for i = 0. Proposition 3.7 gives with p the distance from D (i) j (v) to the bottom of the tree,

R (i) j (v) ≤ log 2 (T ) m=1 8 log(2T /δ) g (i) j (w m ; v) ∇g (i) j (w m ; v) 2 log 2 (T ) p + R (i+1) 2j-1 (w m ) + R (i+1) 2j (v -w m ) .
For the sake of simplicity we will note g = g (i) j (• ; v), and we will begin by bounding

R = log(T ) p log 2 (T ) m=1 |g(w m )| |∇g(w m )| 2 .
We use the Łojasiewicz inequality to obtain that |g(w

m )| ≤ c|∇g(w m )| β . This gives R ≤ c 2/β log 2 (T ) p log 2 (T ) m=1 |g(w m )| 1-2/β
We are now in a similar situation as in the proof of Theorem 3.2 in the case where

β > 2. Using the fact that |g(w m )| ≤ L2 -m , we have R ≤ 1 1 -2 2/β-1 c 2/β L 1-2/β log 2 (T ) p . Let us note C 1 1 -2 2/β-1 c 2/β L 1-2/β . We have R ≤ C log 2 (T ) p .
We use now Proposition 3.7 which shows that

R (i) j (v) ≤ 8 log(2T /δ) • C log 2 (T ) p + log 2 (T ) m=1 R (i+1) 2j-1 (w m ) + log 2 (T ) m=1 R (i+1) 2j (v -w m ).

143

Let us now define the sequence A p = 2A p-1 + 1 for p ≥ 1, and A 0 = 1. The bound we have just shown let us show by recurrence that

R (i) j (v) ≤ 8 log(2T /δ) • A p C log(T ) p .
Lemma 3.10 shows that A p = 2 p+1 -1 ≤ 2 p+1 . Moreover for i = 0, we have p = log 2 (K) -1. Consequently for i = 0, A p ≤ K.

With the choice of δ = 2/T 2 we have finally that

R(T ) = R (0) 1 (1) T ≤ 8 • KC log(T ) log 2 (K) T 1 1 -2 2/β-1 c 2/β L 1-2/β K log(T ) log 2 (K)
T .

3.C.2 Proof of Theorem 3.4 with β = 2

Proof. Let us first bound a sub-regret R (i) j (v) for i = 0. Proposition 3.7 gives with p the distance from D (i) j (v) to the bottom of the tree,

R (i) j (v) ≤ log 2 (T ) m=1 8 log(2T /δ) g (i) j (w m ; v) ∇g (i) j (w m ; v) 2 log 2 (T ) p + R (i+1) 2j-1 (w m ) + R (i+1) 2j (v -w m ).
For the sake of simplicity we will note g = g (i) j (• ; v) and and we will begin by bounding

R = log(T ) m=1 |g(w m )| |∇g(w m )| 2 log(T ) p . Łojasiewicz inequality gives |g(w m )| ≤ c|∇g(w m )| 2 , leading to R ≤ log(T ) m=1 c log(T ) p ≤ c log(T ) p+1 .
An immediate recurrence gives that, as in the case where β > 2,

R (i) j (v) ≤ 8A p c log(2T /δ) log(T ) p+1 .
And finally we have, noting g g

(0) 1 (• ; 1) and p = log 2 (K) -1 R (0) 1 (1) ≤ 8A p c log(2T /δ) log(T ) log d (K) .
Giving finally, with the choice δ = 2/T 2 and since

A p ≤ K for p = log 2 (K) -1, R(T ) = 8A p c log(2T /δ) R T ≤ 24cK log(T ) log 2 (K)+1
T .

3.C.3 Proof of Theorem 3.4 with β < 2

Proof. Let us first bound a sub-regret R (i) j (v) for i = 0. Proposition 3.7 gives with p the distance from D (i) j (v) to the bottom of the tree,

R (i) j (v) ≤ log(T ) m=1 8 log(2T /δ) g (i) j (w m ; v) ∇g (i) j (w m ; v) 2 log(T ) p + R (i+1) 2j-1 (w m ) + R (i+1) 2j (v -w m ).
For the sake of simplicity we will note g = g (i) j (• ; v) and we will begin by bounding

R = log 2 (T ) m=1 |g(w m )| |∇g(w m )| 2 log(T ) p . Łojasiewicz inequality gives |g(w m )| ≤ c|∇g(w m )| β , leading to R ≤ log 2 (T ) m=1 c|∇g(w m )| β-2 log 2 (T ) p .
We want to prove by recurrence that, with p = log 2 (K) -1 -i and A p defined in Appendix 3.C.1.

R (i) j (v) ≤ 8 log(2T /δ)cA p rmax r=1 ∇G (i) j (w r ; v) β-2 log 2 (T ) p . (3.2)
The result is true for p = 0 using what has be done previously. Suppose that it holds at level i + 1 in the tree. Then, Proposition 3.7 shows that

R (i) j (v) ≤ rmax r=1 8 log(2T /δ) g (i) j (w r ; v) ∇G (i) j (w r ; v) 2 log(T ) p + R (i+1) 2j-1 (w r ) + R (i+1) 2j (v -w r ) ≤ 8 log(2T /δ) log 2 (T ) p rmax r=1 c ∇G (i) j (w r ) β-2 + rmax r=1 cA p-1 smax s=1 ∇G (i+1) 2j-1 (x s ; w r ) β-2 log 2 (T ) p-1 + rmax r=1 cA p-1 smax s=1 ∇G (i+1) 2j (x s ; v -w r ) β-2 log 2 (T ) p-1 .
We have noted by x s and xs the points tested by the binary searches D

2j-1 (w r ) and D

(vw r ) and s max ≤ log 2 (T ) the number of points tested by those binary searches. We now use the fact that β -2 < 0 and Lemma 3.7 shows that ∇G

(i+1) 2j-1 (x s ; w m ) ≥ ∇G (i) j (w r ) , giving R (i) j (v) ≤ (1 + 2A p-1 )c • 8 log(2T /δ) rmax r=1 ∇G (i) j (w r ; v) β-2 log 2 (T ) p ,
proving Equation (3.2). And finally we have, as in the proof of Theorem 3.2, noting g g

(0) 1 (• ; 1), R (0) 1 (1) ≤ Kc • 8 log(2T /δ) rmax r=1 |∇g(u r )| β-2 log(T ) log 2 (K)-1 .
We note now g r |∇g(u r )| and we have the constraint, with

T = T 8 log(2T /δ) log(T ) log 2 (K)-1 T = rmax r=1 1 g 2 r .
We want to maximize R rmax r=1 g β-2 r under the above constraint. In order to do that we introduce the following Lagrangian function:

L : (g 1 , . . . , g rmax , λ) → rmax r=1 g β-2 r + λ T - rmax r=1 1 g 2 r .
The Karush-Kuhn-Tucker theorem gives 0 = ∂L ∂g r (g 1 , . . . , g rmax , λ)

145 0 = (β -2)g β-3 r + λ 2g -3 r 0 = (β -2)g β r + 2λ g r = 2λ 2 -β 1/β . The expression of T gives T = rmax r=1 g -2 r T = rmax r=1 2λ 2 -β -2/β λ -2/β = T rmax r=1 (1 -β/2) 2/β λ = T -β/2 r β/2 max (1 -β/2) .
We can now bound R:

R ≤ rmax r=1 g β-2 r ≤ rmax r=1 2λ 2 -β 1-2/β ≤ r max (1 -β/2) 2/β-1 λ 1-2/β ≤ r max (1 -β/2) 2/β-1 T -β/2 r β/2 max (1 -β/2) 1-2/β ≤ r β/2 max T 1-β/2 . Now we use the fact that R(T ) = R (0) 1 (1) T and R (0)
1 (1) ≤ Kc•8 log(2T /δ) log(T ) log 2 (K)-1 R. Taking δ = 2/T 2 , we have log(2T /δ) = 3 log(T ). We have, since r max ≤ log(T ),

R(T ) ≤ 1 T Kc • 8 log(2T /δ) log(T ) log 2 (K)-1 R ≤ 24Kc T log(T ) log 2 (K) R ≤ 24Kc T log(T ) log 2 (K) r β/2 max T 1-β/2 ≤ 24Kc T log(T ) log 2 (K) log(T ) β/2 T 24 log(T ) log 2 (K) 1-β/2 ≤ 24 β/2 Kc log(T ) log 2 (K)+1 T β/2
.

Continuous and discrete-time analysis of Stochastic Gradient Descent

This chapter proposes a thorough theoretical analysis of Stochastic Gradient Descent (SGD) with decreasing step sizes. First, we show that the recursion defining SGD can be provably approximated by solutions of a time inhomogeneous Stochastic Differential Equation (SDE). Then, motivated by recent analyses of deterministic and stochastic optimization methods by their continuous counterpart, we study the longtime convergence of the continuous processes at hand and establish non-asymptotic bounds. To that purpose, we develop new comparison techniques which we think are of independent interest. This continuous analysis allows us to develop an intuition on the convergence of SGD and, adapting the technique to the discrete setting, we show that the same results hold to the corresponding sequences. In our analysis, we notably obtain non-asymptotic bounds in the convex setting for SGD under weaker assumptions than the ones considered in previous works. Finally, we also establish finite time convergence results under various conditions, including relaxations of the famous Łojasiewicz inequality, which can be applied to a class of non-convex functions1 .

Introduction and related work

Recently, first-order optimization algorithms [START_REF] Su | A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights[END_REF] have been shown to share similar long-time behavior with solutions of certain Ordinary Differential Equations (ODE).

The starting point of this kind of analysis is that these schemes can also be regarded as discretization methods. In particular, gradient descent (GD) defines the same sequence as the Euler discretization of the gradient flow corresponding to the objective function f , i.e., the ODE dx(t)/dt = -∇f (x(t)). Then, the analysis of the long-time behavior of solutions of this gradient flow equation can give insights on the convergence of GD. This idea has been adapted to the optimal Nesterov acceleration scheme [START_REF] Nesterov | A method for solving the convex programming problem with convergence rate O(1/k 2 )[END_REF] by [START_REF] Su | A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights[END_REF] which derived that this algorithm has a limiting continuous flow associated with a second-order ODE. This result then allows for a much more intuitive analysis of this scheme and the technique has been subsequently applied to prove tighter results [START_REF] Shi | Understanding the Acceleration Phenomenon via High-Resolution Differential Equations[END_REF] or to analyze different settings [START_REF] Krichene | Accelerated Mirror Descent in Continuous and Discrete Time[END_REF][START_REF] Aujol | Optimal convergence rates for Nesterov acceleration[END_REF][START_REF] Apidopoulos | Convergence rates of an inertial gradient descent algorithm under growth and flatness conditions[END_REF]. Following this approach, this work consists in a new analysis of the Stochastic Gradient Descent (SGD) algorithm to optimize a continuously differentiable function f : R d → R given stochastic estimates of its gradient. This problem naturally appears in many applications in statistics and machine learning, see e.g., [START_REF] Berger | Statistical inference[END_REF][START_REF] Gentle | Computational statistics: an introduction[END_REF][START_REF] Bottou | On-line learning for very large data sets[END_REF][START_REF] Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF]. Nowadays, SGD [START_REF] Robbins | A stochastic approximation method[END_REF], and its variants [START_REF] Polyak | Acceleration of Stochastic Approximation by Averaging[END_REF][START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] are very popular due to their efficiency. Using ODEs, and in particular the gradient flow equation, to study SGD has already been applied in numerous papers [START_REF] Ljung | Analysis of recursive stochastic algorithms[END_REF][START_REF] Kushner | Stochastic approximation methods for constrained and unconstrained systems[END_REF]Métivier andPriouret, 1984, 1987;[START_REF] Benveniste | Adaptive algorithms and stochastic approximations[END_REF][START_REF] Benaim | A dynamical system approach to stochastic approximations[END_REF][START_REF] Tadić | Asymptotic bias of stochastic gradient search[END_REF]. However, to take into account more precisely the noisy nature of SGD, it has been recently suggested to use Stochastic Differential Equations (SDE) as continuous-time models for the analysis of SGD. (Li et al., 2017) introduced Stochastic Modified Equations and established weak approximations theorems, gaining more intuition on SGD, in particular to obtain new hyper-parameter adjustment policies. In another line of work, [START_REF] Feng | Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximation[END_REF] derived uniform in time approximation bounds using ergodic properties of SDEs. To our knowledge, these techniques have only been applied to the study of SGD with fixed stepsize.

The first aim and contribution of this chapter is to show that SDEs can also be used as continuous-times processes properly modeling SGD with non-increasing stepsizes. In Section 4.2, we show that SGD with non-increasing stepsizes is a discretization of a certain class of stochastic continuous processes (X t ) t≥0 solution of time inhomogeneous SDEs. More precisely, we derive strong and weak approximation estimates between the two processes. These estimates emphasize the relevance of these continuous dynamics to the analysis of SGD.

However, most of approximation bounds between solutions of SDEs and recursions defined by SGD are derived under a finite time horizon T and the error between the discrete and the continuous-time processes does not go to zero as T goes to infinity, which is a strong limitation to study the long-time behavior of SGD, see (Li et al., 2017[START_REF] Li | Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations[END_REF]. Our goal is not to address this problem here, showing uniform in time bounds between the two processes, but to highlight how the long-time behavior of the continuous process related to SGD can be used to gain more intuition and insight on the convergence of SGD itself. In that sense our work follows the same lines as [START_REF] Su | A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights[END_REF][START_REF] Krichene | Accelerated Mirror Descent in Continuous and Discrete Time[END_REF][START_REF] Aujol | Optimal convergence rates for Nesterov acceleration[END_REF] which use continuous-time approaches to provide intuitive ways of deriving convergence results. More precisely, in Section 4.3 we first study the behavior of (t → E[f (X t )] -min R d f ) which can be quite easily analyzed under different sets of assumptions on f , including a convex and weakly quasi-convex setting. Then, we propose a simple adaptation of the main arguments of this analysis to the discrete setting. This allows us to show, under the same conditions, that (E[f (X n )] -min R d f ) n∈N also converges to 0 with explicit rates, where (X n ) n∈N is the recursion defined by SGD.

Based on this interpretation, we provide much simpler proofs of existing results and, in some settings, obtain sharper convergence rates for SGD than the ones derived in previous works [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF][START_REF] Taylor | Stochastic first-order methods: non-asymptotic and computer-aided analyses via potential functions[END_REF][START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF]. In the convex setting, we prove for the first time that the convergence rates of SGD match the minimax lower-bounds [START_REF] Agarwal | Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization[END_REF] in the case where the variance is bounded and f is convex with Lipschitz gradient. As a consequence, we disprove a conjecture stated in [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF] on the optimal rate of convergence for SGD. Finally, we consider a relaxation of the weakly quasi-convex setting introduced in [START_REF] Hardt | Gradient Descent Learns Linear Dynamical Systems[END_REF]. Indeed, since in many applications, and especially in deep learning, the objective function is not convex, studying SGD in non-convex settings has become necessary. A recent work of [START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF] uses SDEs to analyze SGD and derive convergence rates in some non-convex settings. However the rates they obtained are not optimal and in this chapter we show that our analysis leads to better rates under weaker assumptions.

The remainder of this chapter is organized as follows. We present the discrete and continuous models in Section 4.2 and we give convergence results in Section 4.3. Finally Section 4.4 concludes the chapter. The postponed proofs are deferred to Appendices 4.A, 4.B, 4.C and 4.D.

From a discrete to a continuous process

Problem setting and main assumptions

Throughout the chapter let f : R d → R be an objective function satisfying the following condition.

A 4.1. f is continuously differentiable and L-smooth with L > 0, i.e., for any x, y ∈ R d , ∇f (x) -∇f (y) ≤ L x -y .

We consider the general case where we do not have access to ∇f but only to unbiased estimates.

A4.2. There exists a probability space

(Z, Z, µ Z ), η ≥ 0 and a function H : R d × Z → R d such that for any x ∈ R d Z H(x, z)dµ Z (z) = ∇f (x) , Z H(x, z) -∇f (x) 2 dµ Z (z) ≤ η .
Note that A4.2 is classical [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF][START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF] and weaker than the bounded gradient assumption considered in [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF][START_REF] Shamir | Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes[END_REF][START_REF] Feng | Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximation[END_REF][START_REF] Rakhlin | Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization[END_REF]. Under A4.1 and A4.2, we consider now the sequence (X n ) n∈N starting from X 0 ∈ R d corresponding to SGD with non-increasing stepsizes and defined for any n ∈ N by

X n+1 = X n -γ(n + 1) -α H(X n , Z n+1 ) , ( 4.1)
where γ > 0, α ∈ [0, 1] and (Z n ) n∈N is a sequence of independent random variables on the probability space (Ω, F, P) valued in (Z, Z) such that for any n ∈ N, Z n is distributed from µ Z . We now turn to the continuous counterpart of (4.1). Define for any x ∈ R d , the semi-definite positive matrix Σ

(x) = µ Z ({H(x, •) -∇f (x)}{H(x, •) -∇f (x)} ) and, for α ∈ [0, 1), consider the time inhomogeneous SDE, dX t = -(γ α + t) -α {∇f (X t )dt + γ 1/2 α Σ(X t ) 1/2 dB t } , ( 4.2)
where γ α = γ 1/(1-α) and (B t ) t≥0 is a d-dimensional Brownian motion. For solutions of this SDE to exist, we consider the following assumption on x → Σ(x) 1/2 .

A 4.3. There exists M ≥ 0 such that for any

x, y ∈ R d , Σ(x) 1/2 -Σ(y) 1/2 ≤ M x -y .
Indeed, using (Karatzas and Shreve, 1991, Chapter 5, Theorem 2.5), strong solutions (X t ) t∈R + exist if A4.1 and A4.3 hold. In the sequel, the process (X t ) t∈R + is referred to as the continuous SGD process in contrast to (X n ) n∈N which is referred to as the discrete SGD process.

Approximations results

In this section, we prove that (X t ) t≥0 solution of (4.2) is indeed, under some conditions, a continuous counterpart of (X n ) n∈N given by (4.1). First, let ( Xt ) t≥0 be the linear interpolation of (X n ) n∈N , i.e., such that for any -α) . Using a first-order Taylor expansion and assuming that the noise is roughly Gaussian with zero-mean and covariance matrix Σ( Xnγα ), we have the following approximation,

t ∈ [nγ α , (n + 1)γ α ], n ∈ N, Xt = γ -1 α (t -nγ α )X n+1 + γ -1 α ((n + 1)γ α -t)X n , with γ α = γ 1/(1
X(n+1)γα -Xnγα = X n+1 -X n ≈ -γ(n + 1) -α H( Xnγα , Z n+1 ) ≈ -γ α (nγ α + γ α ) -α {∇f ( Xnγα ) + Σ( Xnγα ) 1/2 G} ≈ - (n+1)γα nγα (s + γ α ) -α ∇f ( Xs )ds -γ 1/2 α (n+1)γα nγα (s + γ α ) -α Σ( Xs ) 1/2 dB s , (4.3)
where G is a d-dimensional standard Gaussian random variable. The next result justifies the ansatz (4.3) and establishes strong approximation bounds for SGD.

Proposition 4.1. Let γ > 0 and α ∈ [0, 1). Assume A4.1, A4.2 and A4.3. Let ((X n ) n∈N , (X t ) t≥0 ) such that (X t ) t≥0 is solution of (4.2) and (X n ) n∈N is defined by (4.1) with X 0 = X 0 (a) Assume that (Z n ) n∈N and (B t ) t≥0 are independent. Then for any T ≥ 0, there exists C ≥ 0 such that for any γ ∈ (0, γ], n ∈ N with γ α = γ 1/(1-α) , nγ α ≤ T we have

E 1/2 X nγα -X n 2 ≤ Cγ δ (1 + log(γ -1 )) , with δ = min(1, (2 -2α) -1 ). (4.4) (b) If (Z, Z) = (R d , B(R d )) and for any x ∈ R d , z ∈ R d and n ∈ N , H(x, z) = ∇f (x) + Σ(x) 1/2 z, Z n = γ -1/2 α nγα
(n-1)γα dB s then (4.4) holds with δ = 1. For clarity reasons we postpone the proof to Appendix 4.A.3. It relies on a coupling argument which is made explicit in in the supplementary Lemma 4.9. To the best of our knowledge, this strong approximation result is new and illustrates the fundamental difference between SGD and discretization of SDEs such as the Euler-Maruyama (EM) discretization. Consider the SDE

dY t = b(Y t )dt + σ(Y t )dB t , (4.5)
where b : R d → R d , and σ : R d → R d×d are Lipschitz functions, so solutions (Y t ) t≥0 of (4.5) exist and are pathwise unique, see (Karatzas and Shreve, 1991, Chapter 5, Theorem 2.5). Let (Y n ) n∈N be the EM discretization of (4.5) defined for any n ∈ N by

Y n+1 = Y n + γb(Y n ) + √ γσ(Y n )G n+1
, where γ > 0 is the stepsize and (G n ) n∈N is a sequence of i.i.d. d-dimensional standard Gaussian random variables. Then for any T ≥ 0, there exists C ≥ 0 such that for any γ > 0,

n ∈ N, nγ ≤ T , E 1/2 [ Y nγ -Y n 2 ]
≤ Cγ δ where δ = 1/2 if σ is non-constant and δ = 1 otherwise; see e.g., [START_REF] Kloeden | Numerical Solution of Stochastic Differential Equations[END_REF][START_REF] Milstein | Numerical integration of stochastic differential equations[END_REF].

Another difference (for strong approximation) between SGD and the EM discretization scheme is the noise which can be used in these algorithms. Indeed, if (G n ) n∈N in EM is no longer a sequence of Gaussian random variables then for b = 0, σ = I d , (but it holds under mild conditions on b and σ), there exists C ≥ 0 such that for any T ≥ 0, γ > 0,

n ∈ N, nγ ≤ T , E 1/2 [ Y nγ -Y n 2 ] ≥ C √ T , i.e.
, no strong approximation holds. The behavior is different for SGD for which we obtain a strong approximation of order 1/2 at least, whatever the noise is in the condition A4.2.

Proof. Assume that there exists n ∈ N such that u n > B, and let n 1 = inf {n ≥ 0 : u n > B}. By definition of B we have n 1 ≥ n 0 + 1. Moreover we have u n1 -u n1-1 ≤ F (n 1 -1, u n1-1 ).

Since n 1 -1 ≥ n 0 we get that u n1 -u n1-1 ≤ A 2 and u n1-1 ≥ u n1 -A 2 ≥ A 1 . Consequently, F (n 1 -1, u n1-1 ) < 0 and u n1 < u n1-1 , which is a contradiction.

Strongly convex case

First, we illustrate the simplicity and effectiveness of our approach by recovering optimal convergence rates under the first following assumption. A4.4. f is µ-strongly convex with µ > 0, i.e., for any x, y ∈ R d , ∇f (x)-∇f (y), x-y ≥ µ x -y 2 .

The results presented below are not new, see [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF] for the discrete case and [START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF] for the continuous one, but they can be obtained very easily within our framework. For clarity reasons, stochastic calculus technicalities such as Dynkin's lemma Lemma 4.13 are presented in Appendix 4.B.

Continuous case

First, we derive convergence rates on the last iterates. Denote under A4.4 by x the unique minimizer of f . Theorem 4.1. Let α, γ ∈ (0, 1) and (X t ) t≥0 be given by (4.2). Assume A4.1, A4.2, A4.3 and A4.4. Then there exists C ≥ 0 (explicit in the proof) such that for any -α) . Using Dynkin's formula, see Lemma 4.13, we have for any t ≥ 0,

T ≥ 1, E[ X T -x 2 ] ≤ CT -α . Proof. Let α, γ ∈ (0, 1] and consider E : R + → R + defined for t ≥ 0 by E(t) = E[(t + γ α ) α X t -x 2 ], with γ α = γ 1/(1
E(t) = E(0) + α t 0 E(s) s + γ α ds + t 0 γ α E [Tr(Σ(X s ))] (s + γ α ) α ds -2 t 0 E [ ∇f (X s ), X s -x ] ds .
We now differentiate this expression with respect to t and using A4.4 and A4.2, we get for any t > 0,

dE(t)/dt = αE(t)(t + γ α ) -1 -2E [ ∇f (X t ), X t -x ] + γ α E [Tr(Σ(X t ))] (t + γ α ) -α ≤ αE(t)/(t + γ α ) -2µE[ X t -x 2 ] + γ α η/(t + γ α ) α ≤ F (t, E(t)) = αE(t)(t + γ α ) -1 -2µE(t)(t + γ α ) -α + γ α η(t + γ α ) -α ,
where we have used in the penultimate line that Tr(Σ(x)) ≤ η for any x ∈ R d by A4.2. Hence, since F satisfy the conditions of Lemma 4.1 with t 0 = (α/µ) 1/(1-α) and A = 2γ α η/µ, applying this result we get, for any t ≥ 0, E(t) ≤ B with B = max(max s∈[0,t0] E(s), A) which concludes the proof.

We state now an immediate corollary on the function error, which converges at the same rate.

Corollary 4.1. Let α, γ ∈ (0, 1) and (X t ) t≥0 be given by (4.2). Assume A4.1, A4.2 A4.3 and A4.4. Then there exists C ≥ 0 such that for any

T > 0, E [f (X T )]-min R d f ≤ CT -α .
We state now an equivalent result of Theorem 4.1 under weaker assumptions, namely the Łojasiewicz inequality with r = 2, that we restate as it is usually given, with c > 0 (see also Section 3.2.2 for additional details on the Łojasiewicz inequality)

∀x ∈ R d , f (x) -f (x ) ≤ c ∇f (x) 2 .
(4.6)

Note that (4.6) is verified for all strongly convex functions (see Proposition 3.1). Under this condition we have the following proposition.

Proposition 4.3. Let α, γ ∈ (0, 1) and (X t ) t≥0 be given by (4.2). Assume A4.1, A4.2, A4.3 and that f verifies (4.6). Then there exists C > 0 such that for any T > 0,

E [f (X T ) -f ] ≤ CT -α .
Proof. Let α, γ ∈ (0, 1) and (X t ) t≥0 be given by (4.2). Without loss of generality we can assume that f = min x∈R d f (x) = 0. We note E(t) = E [f (X t )] and we apply Lemma 4.13 to the stochastic process ((t+γ α ) α f (X t )) t≥0 , and using A4.1, A4.2, A4.3, (4.6) and Lemma 4.12 this gives, for all t > 0,

E(t) -E(0) = t 0 α(s + γ α ) α-1 E [f (X s )] ds - t 0 E ∇f (X s ) 2 ds + (γ α /2) t 0 (s + γ α ) -α E Tr(∇ 2 f (X s )Σ(X s )) ds dE(t)/ dt ≤ αE(t)(t + γ α ) -1 -(1/c)E(t)(t + γ α ) -α + Lη(t + γ α ) -α .
We can now apply Lemma 4.1 to -α) and A = 4cLη, which shows the existence of C > 0 such that for all t > 0, E(t) ≤ C, concluding the proof.

F (t, x) = αx(t + γ α ) -1 -(1/c)x(t + γ α ) -α + Lη(t + γ α ) -α with t 0 = (2cα) 1/(1
Note that in the statement of Theorem 4.1 and Corollary 4.1 we did not precise the dependency of C with respect to the parameters µ, η and the initial condition. In order to obtain that (i) the constant in front of the asymptotic term T -α scales as η/µ (ii) the initial condition is forgotten exponentially fast, we need a more careful analysis, that we propose now.

We first state a specific version of Lemma 4.1 in the case where there exists t 0 > 0 such that for any t ≥ 0 and F (t, x) ≥ -f(x)g(t) with f superlinear.

Lemma 4.3. Let F ∈ C 1 (R + × R, R) and v ∈ C 1 (R + , R + )
such that for all t ≥ 0, dv(t)/dt ≤ F (t, v(t)). Assume that there exists f : R → R, g ∈ C(R + , R + ), t 0 > 0, A ≥ 0 and β > 0 such that the following conditions hold.

(a) For any t ≥ t 0 , r ∈ (0, 1] and x ≥ 0, rF (t, x) ≤ F (t, rx).

(b) For any t ≥ t 0 and x ≥ 0, F (t, x) ≤ -f(x)g(t).

(c) For any x ≥ A, f(x) > βx. and that G is non-decreasing since for any t ≥ 0, g(t) ≥ 0, we have for any t ∈ (0, T ]

Then, for any

t ≥ 0, v(t) ≤ max(A, exp[β(G(t 0 ) -G(t))] max s∈[0,t 0 ] v(s)) and G(t) = t 0 g(s)ds. Proof. Let T ≥ 0 and y T (t) = v(t) exp[β(G(t) -G(T ))]. Using
dy T (t)/dt ≤ exp[β(G(t) -G(T ))]F (t, v(t)) + βg(t)y T (t) ≤ F (t, y T (t)) + βg(t)y T (t) .
Using this result and Lemma 4.3-(b)-(c), we have for any t ≥ t 0 such that y T (t) ≥ A dy T (t)/dt ≤ -f(y T (t))g(t) + βy T (t)g(t) < 0 .

(4.7)

Let B = max(A, max s∈[0,t0] y T (s)). Assume that A = {t ∈ [0, T ] : y T (t) ≥ B} = ∅ and let t 1 = inf A. Note that t 1 ≥ t 0 and y T (t 1 ) ≥ A. Therefore, using (4.7) we have dy T (t 1 )/dt < 0 and therefore, there exists 0 < t 2 < t 1 such that y T (t 2 ) > y T (t 1 ) but then t 2 ∈ A and t 2 < inf A. Hence, A = ∅ and we get that for any t ∈ [0, T ], y T (t) ≤ B. Therefore, we get that for any t ≥ 0,

v(t) = y t (t) ≤ max(A, exp[β(G(t 0 ) -G(t))] max s∈[0,t0] v(s)) ,
which concludes the proof.

Theorem 4.2. Let α, γ ∈ (0, 1) and (X t ) t≥0 be given by (4.2). Assume A4.1, A4.2, A4.3 and A4.4. Then there exists C ≥ 0 (explicit in the proof) such that for any T ≥ 0,

E[ X T -x 2 ] ≤ max 4γ α η/µ, CE[ X 0 -x 2 ] exp[-µ(γ α + T ) 1-α /(2 -2α)] (γ α +T ) -α .
Proof. Let α, γ ∈ (0, 1] and consider E : R + → R + defined for t ≥ 0 by -α) . Using Dynkin's formula, see Lemma 4.13, we have for any t ≥ 0,

E(t) = E[(t + γ α ) α X t -x 2 ], with γ α = γ 1/(1
E(t) = E(0) + α t 0 E(s) s + γ α ds + t 0 γ α E [Tr(Σ(X s ))] (s + γ α ) α ds -2 t 0 E [ ∇f (X s ), X s -x ] ds .
We now differentiate this expression with respect to t and using A4.4 and A4.2, we get for any t > 0,

dE(t)/dt = αE(t)(t + γ α ) -1 -2E [ ∇f (X t ), X t -x ] + γ α E [Tr(Σ(X t ))] (t + γ α ) -α ≤ αE(t)/(t + γ α ) -2µE[ X t -x 2 ] + γ α η/(t + γ α ) α ≤ F (t, E(t)) = αE(t)(t + γ α ) -1 -2µE(t)(t + γ α ) -α + γ α η(t + γ α ) -α ,
where we have used in the penultimate line that Tr(Σ(x)) ≤ η for any x ∈ R d by A4.2. Let t 0 = max((α/µ) 1/(1-α) -γ α , γ α ). We have for any t ≥ t 0 , and x ≥ 0

F (t, x) ≤ -f(x)g(t) , g(t) = (t + γ α ) -α , f(x) = µx -γ α η .
Hence the conditions (a) and (b) of Lemma 4.3 are satisfied. Let β = µ/2 and A = 4γ α η/µ. We obtain that for any t ≥ t 0 and x ≥ A, f(x) > µx/2 and therefore condition (c) of Lemma 4.3 is satisfied. Applying Lemma 4.3, we obtain that for any t ≥ 0

x(t) ≤ max(4γ α η/µ, exp[-µ(γ α + t) 1-α /(2 -2α)]B) , with B = exp[µ(γ α + t 0 ) 1-α /(2 -2α)] max s∈[0,t0] x(s). We have that max s∈[0,t0] x(s) ≤ (t 0 + γ α ) α max s∈[0,t0] E[ X s -x ] 2
. Using Dynkin's formula, see Lemma 4.13, we have for

any t ≥ 0, E [ X t -x ] 2 ≤ E [ X 0 -x ] 2 + ηΨ(α, t 0 ) , with Ψ(α, t 0 ) =      γ 2 /(2α -1) if 2α > 1 , γ α log(γ -1 α (t 0 + γ α ) 1/(1-α) ) if 2α = 1 , γ α (t 0 + γ α ) (1-2α)/(1-α) /(1 -2α) otherwise .
We conclude the proof upon letting

C = (1 + ηΨ(α, t 0 )) exp[µ(γ α + t 0 ) 1-α /(2 -2α)](γ α + t 0 ) α .

Discrete case

We extend now Theorem 4.1 to the discrete setting using Lemma 4.2 and recover the rates obtained in (Bach and Moulines, 2011, Theorem 1) in the case where α ∈ (0, 1]. In particular, if α = 1 then we obtain a convergence rate of order O(T -1 ) which matches the minimax lower-bounds established in [START_REF] Nemirovsky | Problem complexity and method efficiency in optimization[END_REF][START_REF] Agarwal | Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization[END_REF].

We state now a discrete analogous of Theorem 4.1. Note that the proof is considerably simpler than the one of [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF].

Theorem 4.3. Let γ ∈ (0, 1) and α ∈ (0, 1]. Let (X n ) n≥0 be given by (4.1). Assume A4.2 and A4.4. Then there exists C > 0 such that for all N ≥ 1,

E X N -x 2 ≤ CN -α .
In the case where α = 1 we have to assume additionally that γ > 1/(2µ).

Proof. Let γ ∈ (0, 1) and α ∈ (0, 1]. Let (X n ) n≥0 be given by (4.1). Using A4.4 we get for all n ≥ 0,

E X n+1 -x 2 F n = E X n -x -γ(n + 1) -α H(X n , Z n+1 ) 2 F n (4.8) = X n -x 2 + γ 2 (n + 1) -2α E H(X n , Z n+1 ) 2 F n -2γ(n + 1) -α E [ X n -x , H(X n , Z n+1 ) |F n ] ≤ X n -x 2 + γ 2 (n + 1) -2α η + ∇f (X n ) 2 -2γ(n + 1) -α X n -x , ∇f (X n ) E X n+1 -x 2 ≤ E X n -x 2 1 -2γ(n + 1) -α µ + γ 2 (n + 1) -2α L 2 + ηγ 2 (n + 1) -2α .
We note now u n E X n -x 2 and v n n α u n . Using (4.8) and Bernoulli's inequality we have, for all n ≥ 0

v n+1 -v n = (n + 1) α u n+1 -n α u n = (n + 1) α (u n+1 -u n )) + u n ((n + 1) α -n α ) ≤ -2γµ + γ 2 L 2 (n + 1) -α u n + ηγ 2 (n + 1) -α + u n n α [(1 + 1/n) α -1] ≤ -2γµ + γ 2 L 2 (n + 1) -α + αn α-1 u n + ηγ 2 (n + 1) -α .
Therefore, in the case where α < 1, there exists n 0 ≥ 0 such that for all n ≥ n 0 ,

v n+1 -v n ≤ -γµu n + ηγ 2 (n + 1) -α ≤ -γµn -α v n + ηγ 2 (n + 1) -α ≤ (n + 1) -α (-γµv n + ηγ 2 ) .
And in the case where α = 1, if γ > 1/(2µ) we have the existence of n 1 ≥ 0 such that for all n ≥ n 1 ,

v n+1 -v n ≤ (1/2 -γµ) + γ 2 L 2 (n + 1) -α + αn α-1 u n + ηγ 2 (n + 1) -α .
Using Lemma 4.2 this shows that, for α ∈ (0, 1], there exists a constant C > 0 such that for all n ≥ 0, v n ≤ C. This proves the result.

Using A4.1 and the descent lemma (Nesterov, 2004, Lemma 1.2.3) we have the immediate corollary Corollary 4.2. Let α ∈ (0, 1] and γ ∈ (0, 1). Let (X n ) n≥0 be given by (4.1). Assume A4.1, A4.2 and A4.4. Then there exists C > 0 such that for all N ≥ 1,

E [f (X N ) -f ] ≤ CN -α .
If α = 1 we have also assumed that γ > 1/(2µ).

We now state the discrete counterpart of Proposition 4.3, which is an equivalent of Corollary 4.2, under the Łojasiewicz inequality (4.6).

Proposition 4.4. Let α ∈ (0, 1] and γ ∈ (0, 1). Let (X n ) n≥0 be given by (4.1). Assume A4.1, A4.2 and that f verifies (4.6). Then there exists C > 0 such that for all N ≥ 1,

E [f (X N ) -f ] ≤ CN -α .
In the case where α = 1 we have to assume additionally that γ > 2/c.

Proof. Let α ∈ (0, 1] and γ ∈ (0, 1). Let (X n ) n≥0 be given by (4.1). Let n ≥ 0. Applying the descent lemma (using A4.1) gives

E [f (X n+1 )|F n ] = E [f (X n -γ/(n + 1) α H(X n , Z n+1 )|F n ] ≤ f (X n ) -γ/(n + 1) α E [ ∇f (X n ), H(X n , Z n+1 ) |F n ] + γ 2 /(n + 1) 2α (L/2)E H(X n , Z n+1 ) 2 F n ≤ f (X n ) -γ/(n + 1) α ∇f (X n ) 2 + (Lγ 2 /2)(n + 1) -2α η + ∇f (X n ) 2 E [f (X n+1 )] -f ≤ E [f (X n )] -f + γ(n + 1) -α E ∇f (X n ) 2 -1 + (Lγ/2)(n + 1) -α + (Lγ 2 /2)(n + 1) -2α η .
This shows the existence of n 2 ≥ 0 such that using (4.6) we have for all n ≥ n 2 ,

E [f (X n+1 )] -f ≤ E [f (X n )] -f -(γ/2)(n + 1) -α E ∇f (X n ) 2 + (Lγ 2 /2)(n + 1) -2α η ≤ (E [f (X n )] -f ) 1 -(γc -1 /2)(n + 1) -α + (Lγ 2 /2)(n + 1) -2α η .
We note now for all n ≥ 0,

u n = E [f (X n )] -f and v n = n α u n . We have v n+1 -v n = (n + 1) α u n+1 -n α u n = (n + 1) α (u n+1 -u n )) + u n ((n + 1) α -n α ) ≤ -(γc -1 /2)u n + (Lγ 2 η/2)(n + 1) -α + u n n α [(1 + 1/n) α -1] ≤ u n (-(γc -1 /2) + αn α-1 ) + (Lγ 2 η/2)(n + 1) -α .
If α < 1, or if 1 -γc -1 /2 < 0 we have the existence of n 3 ≥ n 2 and C > 0 such that for all n ≥ n 3 ,

v n+1 -v n ≤ -Cu n + (Lγ 2 η/2)(n + 1) -α ≤ -Cv n + (Lγ 2 η/2) (n + 1) -α
This proves the existence of C > 0 such that for all n ≥ 0,

v n ≤ C ,
concluding the proof.

In Figure 4.1a and Figure 4.1b, we experimentally check that the results we obtain are tight in the simple case where f (x) = x 2 and using synthetic data. In our experiments E[f (X n )] is approximated by Monte-Carlo using 10 4 SGD trajectories. We emphasize that the strong convexity assumption can be relaxed if we only assume that f is weakly µ-strongly convex, i.e., for any x ∈ R d , ∇f (x), x -x ≥ µ x -x 2 . In [START_REF] Kleinberg | An Alternative View: When Does SGD Escape Local Minima[END_REF], the authors experimentally show that modern neural networks satisfy a relaxation of this last condition and it was proved in [START_REF] Li | Convergence Analysis of Two-layer Neural Networks with ReLU Activation[END_REF]) that two-layer neural networks with ReLU activation functions are weakly µ-strongly convex if the inputs are Gaussian. Finally, under the additional assumption that f is smooth, we show in Corollary 4.1 and Corollary 4.2 that Theorem 4.1 also implies convergence rates for the process (E [f (X t )] -min R d f ) t≥0 and its discrete counterpart.

Convex case

In this section, we relax the strong-convexity condition.

A 4.5. f is convex, for any x, y ∈ R d , ∇f (x) -∇f (y), x -y ≥ 0 and there exists x ∈ arg min R d f .

We start by studying the continuous process as for the strong convex case under this weaker condition. The discrete analog is given in Theorem 4.6 after.

Theorem 4.4. Let α, γ ∈ (0, 1) and (X t ) t≥0 be given by (4.2). Assume f ∈ C 2 (R d , R), A4.1, A4.2, A4.3 and A4.5. Then, there exists C ≥ 0 (explicit and given in the proof) such that for any T ≥ 1

E [f (X T )] -min R d f ≤ C(1 + log(T )) 2 /T α∧(1-α) .
To the best of our knowledge, these non-asymptotic results are new for the continuous process (X t ) t≥0 defined by (4.2). Note that for α = 1/2 the convergence rate is of order O(T -1/2 log 2 (T )) which matches (up to a logarithmic term) the minimax lowerbound [START_REF] Agarwal | Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization[END_REF] and is in accordance with the tight bounds derived in the discrete case under additional assumptions [START_REF] Shamir | Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes[END_REF]. The general proof is postponed to Appendix 4.C.1 for readability reasons. The main strategy to show Theorem 4.4 is to carefully analyze a continuous version of the suffix averaging [START_REF] Shamir | Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes[END_REF][START_REF] Harvey | Tight analyses for nonsmooth stochastic gradient descent[END_REF], introduced in the discrete case by [START_REF] Zhang | Solving large scale linear prediction problems using stochastic gradient descent algorithms[END_REF].

We can relax the assumption f ∈ C 2 (R d , R) if we assume that the set arg min R d f is bounded.

Theorem 4.5. Let α, γ ∈ (0, 1) and (X t ) t≥0 be given by (4.2). Assume that arg min R d f is bounded, A4.1, A4.2, A4.3 and A4.5. Then, there exists C ≥ 0 (explicit and given in the proof) such that for any T ≥ 1, (1-α) .

E [f (X T )] -min R d f ≤ C(1 + log(T )) 2 /T α∧
The proof relies on the fact that if f is convex then for any ε > 0, f * g ε is also convex, where (g ε ) ε>0 is a family of non-negative mollifiers. We now turn to the discrete counterpart of Theorem 4.4.

Proof. Let α, γ ∈ (0, 1] and T ≥ 0. (f ε ) ε>0 be given by Lemma 4.14. Let δ = min(α, 1 -α). We can apply, Theorem 4.4 to f ε for each ε > 0. Therefore there exists C (c)

ε such that E [f (X T,ε )] -f (x ε ) ≤ C (c) ε log(T ) 2 T -δ + log(T )T -δ + T -δ + (T -1) -2α , (4.9)
where (X t,ε ) t≥0 is given by (4.2) with X 0,ε = X 0 (upon replacing f by f ε ) and

C (c) ε = 4 max(2C (c) 2,α + 2 X 0 -x ε 2 , (γ α η + 2αC (c) 1,α )(1 -α) -1 ) .
Using (4.9) and Lemma 4.14 we have

E [f (X T )] -f ≤ lim inf ε→0 E [f ε (X t,ε )] -lim sup ε→0 f ε (x ε ) ≤ lim inf ε→0 {E [f ε (X t,ε )] -f ε (x ε )} ≤ lim inf ε→0 C (c) ε log(T ) 2 T -δ + log(T )T -δ + T -δ + (T -1) -2α ≤ C (c) 1 log(T ) 2 T -δ + log(T )T -δ + T -δ + (T -1) -2α , with C (c) 1 = 3 max(2C (c) 2,α +4 X 0 2 +4C 2 , (γ α η+2C (c) 1,α )(1-α) -1
), where C = max y∈arg min R d f y .

Theorem 4.6. Let γ, α ∈ (0, 1) and (X n ) n≥0 be given by (4.1). Assume A4.1, A4.2 and A4.5. Then, there exists C ≥ 0 (explicit and given in the proof) such that for any N ≥ 1,

E [f (X N )] -min R d f ≤ C(1 + log(N + 1)) 2 /(N + 1) α∧(1-α) .
The proof is postponed to Appendix 4.C.2 and takes its inspiration from the proof of the continuous counterpart Theorem 4.4. Note that in the case α = 1/2 we recover (up to a logarithmic term) the rate O(N -1/2 log(N + 1)) derived in [START_REF] Shamir | Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes[END_REF], Theorem 2) which matches the minimax lower-bound [START_REF] Agarwal | Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization[END_REF], up to a logarithmic term. We also extend this result to the case α = 1/2. Note however that our setting differs from theirs. (Shamir and Zhang, 2013, Theorem 2) established the convergence rate for a projected version of SGD onto a convex compact set of R d under the assumption that f is convex (possibly non-smooth) and (E[ H(X n , Z n+1 ) 2 ]) n∈N is bounded. In that sense the result provided in Theorem 4.6 is new and optimal with respect to minimax bounds [START_REF] Agarwal | Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization[END_REF]. Our main contributions in the convex setting are summarized in

(L) α ∈ (0, 1/3) α × × α ∈ (1/3, 1/2) α (3α -1)/2 × α ∈ (1/2, 2/3) 1 -α α/2 α/2 α ∈ (2/3, 1) 1 -α 1 -α 1 -α
In addition to these two conditions, one crucial part of the analysis of (Shamir and Zhang, 2013, Theorem 2) uses that (E[ X n -x 2 ]) n∈N is bounded which is possible since (X n ) n∈N in their setting stays in a compact. In Theorem 4.6, we replace the conditions considered in (Shamir and Zhang, 2013, Theorem 2) On the other hand, the setting we consider is the same as [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF]), but we always obtain better convergence rates and in particular we get an optimal choice for α (1/2) different from theirs (2/3), see Table 4.1. Hence, we disprove the conjecture formulated in [START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF] which asserts that the minimax rate for SGD in this setting is 1/3.

In Figure 4.2, we experimentally assess the results of Theorem 4.6. We perform SGD on the family of functions (ϕ p ) p∈N , where for any x ∈ R, p ∈ N

ϕ p (x) = x 2p , if x ∈ [-1, 1] , 2p(|x| -1) + 1 , otherwise .
For any p ∈ N, ϕ p satisfies and A4.1 and A4.5. Denoting α p the non-increasing rate α for which the convergence rate r p is maximum, we experimentally check that lim p→+∞ r p = 1/2 and lim p→+∞ α p = 1/2. Note also that α p decreases as p grows, which is in accordance with the deterministic setting where the optimal rate in this case is given by p/(p -2), see [START_REF] Bolte | From error bounds to the complexity of first-order descent methods for convex functions[END_REF][START_REF] Frankel | Splitting methods with variable metric for Kurdyka-Łojasiewicz functions and general convergence rates[END_REF].

As an immediate consequence of Theorem 4.6, we can show that (E[ ∇f (X n ) 2 ]) n∈N enjoys the same rates of convergence as (E[f (X n )] -min R d f ) n∈N , using that f is smooth.

Corollary 4.3. Let γ, α ∈ (0, 1) and (X n ) n≥0 be given by (4.1). Assume A4.1, A4.2 and A4.5. Then, there exists C ≥ 0 (explicit and given in the proof) such that for any N ≥ 1, E ∇f (X N ) 2 ≤ C(1 + log(N + 1)) 2 /(N + 1) α∧ (1-α) .

In particular, (E[ ∇f (X n ) 2 ]) n∈N is bounded which is often found as an assumption for the study of the convergence of SGD in the convex setting [START_REF] Shalev-Shwartz | Pegasos: primal estimated sub-gradient solver for SVM[END_REF][START_REF] Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF][START_REF] Hazan | Beyond the Regret Minimization Barrier: Optimal Algorithms for Stochastic Strongly-Convex Optimization[END_REF][START_REF] Shamir | Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes[END_REF][START_REF] Recht | Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent[END_REF]. Our result shows that this assumption is unnecessary.

We present now a corollary of the previous theorem under a different setting. Let us assume, as in [START_REF] Shamir | Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes[END_REF], that ∇f is not Lipschitz-continuous but bounded instead.

Corollary 4.4. Let γ, α ∈ (0, 1) and X 0 ∈ R d and (X n ) n≥0 be given by (4.1). Assume A4.5, A4.2 and ∇f bounded. Then there exists C > 0 such that, for all N ≥ 1,

E [f (X N )] -f ≤ C(1 + log(N + 1)) 2 /(N + 1) min(α,1-α) .
The proof follows the same line of proof as the one of Theorem 4.6 and is consequently postponed to Appendix 4.C.2.

Weakly quasi-convex case

In this section, we no longer consider that f is convex but a relaxation of this condition.

We will analyze the convergence of SGD under the following assumption.

A 4.6. There exist r 1 ∈ (0, 2), r 2 ≥ 0, τ > 0 such that for any

x ∈ R d ∇f (x) r 1 x -x r 2 ≥ τ (f (x) -f (x )) , where x ∈ arg min R d f = ∅ .
This setting is a generalization of the weakly quasi-convex assumption considered in [START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF] and introduced in [START_REF] Hardt | Gradient Descent Learns Linear Dynamical Systems[END_REF] as follows.

A 4.6b. The function f is weakly quasi-convex if there τ > 0 such that for any

x ∈ R d ∇f (x), x -x ≥ τ (f (x) -f (x )) , where x ∈ arg min R d f = ∅ .
This last condition itself is a modification of the quasi-convexity assumption [START_REF] Hazan | Beyond Convexity: Stochastic Quasi-Convex Optimization[END_REF]. It was shown in [START_REF] Hardt | Gradient Descent Learns Linear Dynamical Systems[END_REF]) that an idealized risk for linear dynamical system identification is weakly quasi-convex, and in [START_REF] Yuan | Stagewise Training Accelerates Convergence of Testing Error Over SGD[END_REF], the authors experimentally check that a residual network (ResNet20) used on CIFAR-10 (with differentiable activation units) satisfy the weakly quasi-convex assumption.

The assumption A4.6 also embeds the setting where f satisfies some Kurdyka-Łojasiewicz condition [START_REF] Bolte | From error bounds to the complexity of first-order descent methods for convex functions[END_REF], i.e., if there exist r ∈ (0, 2) and τ > 0 such that for any

x ∈ R d , ∇f (x) r ≥ τ (f (x) -f (x )) , (4.10)
then A4.6 is satisfied with r 1 = r, r 2 = 0 and τ = τ . Kurdyka-Łojasiewicz conditions have been often used in the context of non-convex minimization [START_REF] Attouch | Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the Kurdyka-Łojasiewicz inequality[END_REF][START_REF] Noll | Convergence of non-smooth descent methods using the Kurdyka-Łojasiewicz inequality[END_REF]. Even though the case r 1 = 2 and r 2 = 0 is not considered in A4.6, one can still derive convergence of order α for α ∈ (0, 1), see Proposition 4.4, extending the results obtained in the strongly convex setting. We now state the main theorem of this section.

Theorem 4.7. Let α, γ ∈ (0, 1) and (X t ) t≥0 be given by (4.2). Assume f ∈ C 2 (R d , R), A4.1, A4.2, A4.3 and A4.6. In addition, assume that there exist β, ε ≥ 0 and C β,ε ≥ 0 such that for any t ≥ 0,

E[ X t -x r 2 r 3 ] ≤ C β,ε (γ α + t) β (1 + log(1 + γ -1 α t)) ε ,
where γ α = γ 1/(1-α) and r 3 = (1 -r 1 /2) -1 . Then, there exists C ≥ 0 (explicit and given in the proof) such that for any T ≥ 1

E [f (X T )] -min R d f ≤ CT -δ 1 ∧δ 2 [1 + log(1 + γ -1 α T )] ε ,
where

δ 1 = (r 1 /2)(1 -r 1 /2) -1 (1 -α) -β and δ 2 = (r 1 /2)α -β(1 -r 1 /2) . (4.11)
Note that if f satisfies a Kurdyka-Łojasiewicz condition of type (4.10) then A4.6 is satisfied with r 1 = r and r 2 = 0 and the rates in Theorem 4.7 simplify and we obtain that δ = min((r/2)(1 -r/2) -1 (1 -α), (r/2)α). The rate is maximized for α = (2 -r/2) -1 and in this case, δ = r/(4 -r). Therefore, if r → 2, then δ → 1 and we obtain at the limit the same convergence rate that the case where f is strongly convex A4.4.

Proof. Without loss of generality, we assume that f = 0. Let α, γ ∈ (0, 1), x 0 ∈ R d , a t = γ α + t, t = 1 + log(1 + γ -1 α t) for any t ≥ 0 and δ = min(δ 1 , δ 2 ) with δ 1 and δ 2 given in Theorem 4.7. Using Lemma 4.13, we have for any t ≥ 0

E f (X t )a δ t -ε t -f (X 0 )γ δ α = t 0 --ε s a δ-α s E[ ∇f (X s ) 2 ] +(γ α /2) -ε s a δ-2α s E ∇ 2 f (X s ), Σ(X s ) +δ -ε s a δ-1 s E [f (X s )] -ε -ε-1 s a δ s E [f (X s )] ds .
Define for any t ≥ 0,

E(t) = E[f (X t )]a δ t -ε t . (t → E(t)
) is differentiable and using A4.1 and A4.2 we have for any t > 0,

dE(t)/ dt ≤ --ε t a δ-α t E ∇f (X t ) 2 + (γ α /2) -ε t a δ-2α t Lη + δa -1 t E(t) .
Using, A4.6 and Hölder's inequality we have for any

t ≥ 0 τ E [f (X t )] ≤ E [ X t -x r2r3 ] r -1 3 E[ ∇f (X t ) 2 ] r1/2 .
Noting that (r 3 r 1 ) -1 = r -1 1 -1/2, we get for any t ≥ 0

E[ ∇f (X t ) 2 ] ≥ τ 2r -1 1 E [f (X t )] 2r -1 1 E [ X t -x r2r3 ] 1-2r -1 1 ≥ τ 2r -1 1 C 1-2r -1 1 β,ε a β(1-2r -1 1 ) t ε(1-2r -1 1 ) t E [f (X t )] 2r -1 1 ≥ τ 2r -1 1 C 1-2r -1 1 β,ε a β(1-2r -1 1 )-2r -1 1 δ t ε(1-2r -1 1 )-2r -1 1 -ε t E(t) 2r -1 1 .
Therefore, we have for any t ≥ 0 0)), then for any t ≥ 0, E(t) ≤ C, which concludes the proof.

dE(t)/ dt ≤ -τ 2r -1 1 C 1-2r -1 1 β,ε a (1-2r -1 1 )(δ+β)-α t E(t) 2r -1 1 + γ α -ε t a δ-2α t Lη + δa -1 t E(t) . Let D 3 = max(D 1 , D 2 ) with D 1 = (|δ|C 2r -1 1 -1 β,ε τ -2r -1 1 γ (2r -1 1 -1)(δ+β)+α-1 α ) (2r -1 1 -1) -1 , D 2 = ((Lη/2)C 2r -1 1 -1 β,ε τ -2r -1 1 γ (2r -1 1 -1)(δ+β)+δ-α+1 α ) r1/2 . If E(t) ≥ D 3 then dE(t)/ dt ≤ 0. Let C = max(D 3 , E(
In the general case r 2 = 0, the convergence rates obtained in Theorem 4.7 depend on β where (E[ X t -x r 2 r 3 ](γ α + t) -β ) t≥0 has at most logarithmic growth. If β = 0, then the convergence rates deteriorate. In what follows, we shall consider different scenarios under which β can be explicitly controlled. These estimates imply explicit convergence rates for SGD using Theorem 4.7.

Corollary 4.5. Let α, γ ∈ (0, 1) and (X t ) t≥0 given by (4.2). Assume f ∈ C 2 (R d , R), A4.1, A4.2 and A4.3. (a) If A4.6b holds, then there exists C ≥ 0 such that for any T ≥ 1

E [f (X T )] -min R d f ≤ C[T (1-3α)/2 + T -α/2 + T α-1 ] .
(b) If A4.6b holds and there exist c, R > 0 such that for any x ∈ R d with x -x ≥ R, f (x) -f (x ) ≥ c x -x then there exists C ≥ 0 such that for any T ≥ 1

E [f (X T )] -min R d f ≤ C[T -α/2 + T α-1 ] .
(4.12) (c) If A4.6 holds and if there exists R ≥ 0 such that for any x ∈ R d with x ≥ R, ∇f (x), x -x ≥ m x -x 2 , then there exists C ≥ 0 such that for any T ≥ 1, (4.12) holds.

The proof is postponed to Appendix 4.D. The main ingredient of the proof is to control the growth of t → E[ X t -x 2 ] using either the SDE satisfied by ( X t -x 2 ) t≥0 in the case of (a) and (c), or the SDE satisfied by (f (X t ) -min R d f ) t≥0 in the case of (b).

Under A4.6b, we compare the rates we obtain using Corollary 4.5-(a) with the ones derived by [START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF] in Table 4.2 and Figure 4.3b. Note that compared to [START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF], we establish that SGD converges as soon as α > 1/3 and not α > 1/2. In addition, the convergence rates we obtain are always better than the ones of [START_REF] Orvieto | Continuous-time Models for Stochastic Optimization Algorithms[END_REF] in the case α > 1/2. However, note that in both cases, the optimal convergence rate is 1/3 obtained using α = 2/3. Finally, under additional growth conditions on the function f , and using Corollary 4.5-(b)-(c) we show that the convergence of SGD in the weak quasi-convex case occurs as soon as α > 0.

α ∈ (0, 1/3) × α/2 × α ∈ (1/3, 1/2) (3α -1)/2 α/2 × α = 1/2 1/4 + log. 1/4 + log. × α ∈ (1/2, 2/3) α/2 1 -α 2α -1 α ∈ (2/3, 1) 1 -α 1 -α 1 -α
As in the previous sections, we extend our results to the discrete setting.

Theorem 4.8. Let α, γ ∈ (0, 1) and (X n ) n∈N be given by (4.1). Assume A4.1, A4.2 and A4.6. In addition, assume that there exist β, ε, C β,ε ≥ 0 such that for any n ∈ N,

E [ X n -x r 2 r 3 ] ≤ C β,ε (n + 1) β {1 + log(1 + n)} ε
, where r 3 = (1 -r 1 /2) -1 . Then, there exists C ≥ 0 (explicit and given in the proof) such that for any N ≥ 1

E [f (X N )] -min R d f ≤ CN -δ 1 ∧δ 2 (1 + log(1 + N ))) ε ,
where δ 1 , δ 2 are given in (4.11).

Proof. Without loss of generality, we assume that f = 0. Let α, γ ∈ (0, 1), x 0 ∈ R d . Let δ = min(δ 1 , δ 2 ), with δ 1 , δ 2 given in Theorem 4.8 and let (E k ) k∈N such that for any k ∈ N,

E k = (k + 1) δ E [f (X k )] (1 + log(k + 1)) -ε .
There exists c δ ∈ R such that for any x ∈ [0, 1],

(1 + x) δ ≤ 1 + c δ x. Hence, for any n ∈ N we have

(n + 2) δ -(n + 1) δ ≤ (n + 1) δ (1 + (n + 1) -1 ) δ -1 ≤ c δ (n + 1) δ-1 . (4.13)
Using (Nesterov, 2004, Lemma 1.2.3) and A4.2 we have for any n ∈ N such that n ≥ (2Lγ

) 1/α E [f (X n+1 )|F n ] ≤ f (X n ) -γ(n + 1) -α E [ ∇f (X n ), H(X n , Z n+1 ) |F n ] (4.14) + (L/2)γ 2 (n + 1) -2α E H(X n , Z n+1 ) 2 F n 164 E [f (X n+1 )] ≤ E [f (X n )] -γ(n + 1) -α E ∇f (X n ) 2 + Lγ 2 (n + 1) -2α E ∇f (X n ) 2 + Lγ 2 (n + 1) -2α η ≤ E [f (X n )] -γ(n + 1) -α 1 -Lγ(n + 1) -α E ∇f (X n ) 2 + Lγ 2 (n + 1) -2α η ≤ E [f (X n )] -γ(n + 1) -α E ∇f (X n ) 2 /2 + Lγ 2 (n + 1) -2α η .
Combining (4.13) and (4.14) we get for any n ∈ N such that n ≥ (2Lγ

) 1/2 E n+1 -E n = (n + 2) δ E [f (X n+1 )] (1 + log(n + 2)) -ε -(n + 1) δ E [f (X n )] (1 + log(n + 1)) -ε (4.15) ≤ (1 + log(n + 1)) -ε (n + 2) δ -(n + 1) δ (E [f (X n+1 )]) +(n + 1) δ {E [f (X n+1 )] -E [f (X n )]} ≤ (1 + log(n + 1)) -ε (n + 2) δ -(n + 1) δ (E [f (X n )] + Lγ 2 (n + 1) -2α η) +(n + 1) δ -γ(n + 1) -α E ∇f (X n ) 2 /2 + Lγ 2 (n + 1) -2α η ≤ (1 + log(n + 1)) -ε c δ (n + 1) δ-1 (E [f (X n )] + 2γ 2 (n + 1) -2α η) +(n + 1) δ -γ(n + 1) -α E ∇f (X n ) 2 /2 + Lγ 2 (n + 1) -2α η ≤ c δ E n + 2Lγ 2 (1 + c δ )(n + 1) δ-2α (1 + log(n + 1)) -ε η -γ(n + 1) δ-α (1 + log(n + 1)) -ε E ∇f (X n ) 2 /2 .
Using (4.6) and the fact that for any k

∈ N, E [ X k -x r2r3 ] ≤ C β,ε (k + 1) β (1 + log(1 + k)) ε
and Hölder's inequality and that r 1 r 3 = 2(2r -1 1 -1) -1 , we have for any

k ∈ N E ∇f (X k ) 2 ≥ E [f (X k )] 2r -1 1 C -(2r -1 1 -1) -1 β,ε τ 2r -1 1 (k+1) -β(2r -1 1 -1) (1+log(k+1)) -ε(2r -1 1 -1) .
(4.16) Combining (4.15) and (4.16) we get that for any n ∈ N with n ≥ (4γ

) 1/α E n+1 -E n ≤ c δ E n + 2Lγ 2 (1 + c δ )(n + 1) δ-2α (1 + log(n + 1)) -ε η -γ(n + 1) δ-α-β(2r -1 1 -1) E [f (X n )] 2r -1 1 C -(2r -1 1 -1) -1 β,ε τ 2r -1 1 (1 + log(n + 1)) -ε2r -1 1 /2 ≤ c δ E n + 2Lγ 2 (1 + c δ )(n + 1) δ-2α (1 + log(n + 1)) -ε η -γ(n + 1) α-(δ+β)(2r -1 1 -1) E 2r -1 1 n C -(2r -1 1 -1) -1 β,ε τ 2r -1 1 /2 . Let D 3 = max(D 1 , D 2 ) with    D 1 = (2|c δ |C 2r -1 1 -1 β,ε τ -2r -1 1 ) 2r -1 1 -1 , D 2 = (4Lγ 2 (1 + c δ )C 2r -1 1 -1 β,ε τ -2r -1 1 ) r1/2 . If E n ≥ D 3 and n ≥ (4γ) 1/α then E n+1 ≤ E n . Therefore, we obtain by recursion that E n ≤ C with C = max(E 0 , . . . , E (2Lγ) 1/α , D 3 ).
We can conduct the same discussion as the one after Theorem 4.7 and Corollary 4.5 can be extended to the discrete case.

Corollary 4.6. Let α, γ ∈ (0, 1) and x 0 ∈ R d . Assume A4.1, A4.2. Then we have: (a) if A4.6b holds then, there exists C ≥ 0 such that for any

N ∈ N E [f (X N )] -f ≤ C N (1-3α)/2 + N -α/2 + N α-1 , (b) if A4.6 holds and if there exists R ≥ 0 such that for any x ∈ R d with x ≥ R, ∇f (x), x -x ≥ m x -x 2 , then there exists C ≥ 0 such that for any N ∈ N E [f (X N )] -f ≤ C N -α/2 + N α-1 .
Proof. Let α, γ ∈ (0, 1) and x 0 ∈ R d . We have for any n ∈ N,

E X n+1 -x 2 = E X n -x 2 + 2E [ X n -x , X n+1 -X n ] + E X n+1 -X n 2 (4.17) ≤ E X n -x 2 -2γ(n + 1) -α E [ X n -x , ∇f (X n ) ] + 2γ 2 (n + 1) -2α E ∇f (X n ) 2 + 2γ(n + 1) -2α η .
We now divide the proof into two parts.

(a) Using A4.6b and Lemma 4.16 we have for any

x ∈ R d , ∇f (x), x -x ≥ τ (f (x) -f (x )) ≥ τ ∇f (x) 2 /(2L) . (4.18)
Using A4.1, (4.17) and (4.18) we have for any n ≥ (4γL/τ )

1/α E X n+1 -x 2 ≤ E X n -x 2 + 2γ(n + 1) -α (-τ /(2L) + γ(n + 1) -α )E ∇f (X n ) 2 + 2γ(n + 1) -2α η ≤ E X n -x 2 + 2γ(n + 1) -2α η .
Therefore, there exist β, ε ≥ 0 and

C β,ε ≥ 0 such that E[ X n -x 2 ] < C β,ε (n + 1) -β (1 + log(1 + n)) ε with β = 0 and ε = 0 if α > 1/2, β = 1 -2α and ε = 0 if α < 1/2 and β = 0 and ε = 1 if α = 1/2.
Combining this result and Theorem 4.8 concludes the proof.

(b) Finally, assume that there exists R ≥ 0 such that for any x ∈ R d with x ≥ R, ∇f (x), x -x ≥ m x -x 2 . Therefore, since (x → ∇f (x)) is continuous, there exists a ≥ 0 such that for any x ∈ R d , ∇f (x), x -x ≥ m x -x 2a. Combining this result and (4.17) we get that for any n ∈ N such that n ≥ (2/γ) α -1

E X n+1 -x 2 ≤ (1 -γ(n + 1) -α )E X n -x 2 + 2γ(n + 1) -α a + 2γ 2 (n + 1) -2α η . Hence, if n ≥ (2/γ) -α -1 and E[ X n -x 2 ] ≥ max(2a, 2γη) then E[ X n+1 -x 2 ] ≤ E[ X n - x 2 ]
. Therefore, we obtain by recursion that for any n ∈ N, that (E[ X n -x 2 ]) n∈N is bounded which concludes the proof by applying Theorem 4.8.

Conclusion

In this chapter we investigated the connection of SGD with solutions of a particular time inhomogenuous SDE. We first proved approximation bounds between these two processes motivating convergence analysis of continuous SGD. Then, we turned to the convergence behavior of SGD and showed how the continuous process can provide a better understanding of SGD using tools from ODE and stochastic calculus. In particular, we obtained optimal convergence rates in the strongly convex and convex cases. In the non-convex setting, we considered a relaxation of the weakly quasi-convex condition and improved the state-of-the art convergence rates in both the continuous and discrete-time setting.

4.A Proofs of the approximation results

In this section2 , we present the proof of Proposition 4.1 in Appendix 4.A.3 and the one of Proposition 4.2 in Appendix 4.A.4. We begin this section by some useful technical lemmas and results on moment bounds. Throughout this section we will denote all the constants by the letter A followed by some subscript.

4.A.1 Technical Lemmas

The following lemma is well-known but is recalled as well as its proof for completeness

Lemma 4.4. Let f ∈ C 1 (R d , R).
Assume that there exists L ≥ 0 such that for any x, y ∈ R d , ∇f is L-Lipschitz and that f admits a minimum. Then for any

x ∈ R d ∇f (x) 2 ≤ (L/2)(f (x) -min f ) . (4.19)
Proof. Using (Nesterov, 2004, Lemma 1.2.3), we have for any

x, y ∈ R d f (y) -f (x) ≤ ∇f (x), y -x + (L/2) y -x 2 .
We obtain (4.19) by minimizing both side of the previous inequality.

Lemma 4.5. Let (u n ) n∈N , (v n ) n∈N and (w n ) n∈N such that for any n ∈ N, u n , v n , w n ≥ 0, u 0 ≥ 0 and

u n+1 ≤ (1 + v n )u n + w n . Then for any n ∈ N u n ≤ exp n-1 k=0 v k u 0 + n-1 k=0 w k .
Proof. The proof is a straightforward consequence of the discrete Grönwall's lemma.

Lemma 4.6. Let r > 0, γ > 0 and α ∈ [0, 1). Then for any T ≥ 0, there exists A α,r ≥ 0 such that for any N ∈ N with N γ α ≤ T we have

γ r N -1 k=0 (k + 1) -αr ≤ A α,r γ r (1 + log(γ -1 ))(1 + log(T )) , if α ≥ 1/r , A α,r γ r γ αr-1 α T 1-αr otherwise .
Note that if r = 1 then γ r N -1 k=0 (k + 1) -αr ≤ A α,1 T 1-α . Using a slight modification of Lemma 4.6 we also obtain that there exists à such that if r = 1 then γ r N -1 k=0 (k +1) -αr ≤ T 1-α + Ã.

Proof. Let r > 0, γ > 0 and α ∈ [0, 1). If α > 1/r then there exists A α,r ≥ 0 such that

γ r N -1 k=0 (k + 1) -αr ≤ A α,r γ r . If α < 1/r then there exists A α,r ≥ 0 such that γ r N -1 k=0 (k + 1) -αr ≤ A α,r γ r N -αr+1 ≤ A α,r γ r γ αr-1 α T 1-αr . if α = 1/r then there exists A α,r ≥ 0 such that γ r N -1 k=0 (k + 1) -αr ≤ γ r (1 + log(N )) ≤ A α,r γ r (1 + log(T ))(1 + log(γ -1 )) .

4.A.2 Moment bounds

The following result is well-known in the field of SDE but its proof is given for completeness.

Lemma 4.7. Let p ∈ N, γ > 0 and α ∈ [0, 1). Assume A4.1 and A4.2. Then for any T ≥ 0, there exists A T,1 ≥ 0, such that for any s ≥ 0 and t ∈ [s, s + T ], γ ∈ (0, γ] and

X 0 ∈ R d , we have E 1 + X t 2p ≤ A T,1 (1 + X 0 2p ) ,
where (X t ) t≥0 is the solution of (4.2) such that X s = X 0 . If in addition, for any x ∈ R d , µ Z ( H(x, •)-∇f (x) 2p ) ≤ η p , with η p ≥ 0, then for any T ≥ 0, there exists ÃT,1 ≥ 0, such that for any k 0 ≥ 0, γ ∈ (0, γ] and k ∈ {k 0 , . . . , k 0 + N } with N γ α ≤ T , and X 0 ∈ R d , we have

E 1 + X k 2p ≤ ÃT,1 (1 + X 0 2p ) ,
where (X k ) k∈N satisfies the recursion (4.1) with

X k 0 = X 0 Proof. Let p ∈ N, α ∈ [0, 1), s, T ∈ [0, +∞), t ∈ [s, s + T ], X 0 ∈ R d and g p ∈ C 2 (R d , [0, +∞)) such that for any x ∈ R d , g p (x) = 1 + x 2p .
Let γ > 0 and γ ∈ (0, γ]. We divide the proof into two parts (a) Let (X t ) t≥0 be a solution to (4.2) such that X s = X 0 . We have for any

x ∈ R d ∇g p (x) = 2p x 2(p-1) x , ∇ 2 g p (x) = 4p(p -1) x 2(p-2) xx + 2p x 2(p-1) Id . (4.20)
Let n ∈ N, and set τ n = inf{u ≥ 0 : g p (X u ) > n}. Applying Itô's lemma and using (4.2) and (4.20) we get

E [g p (X t∧τn )] -E [g p (X s∧τn )] = E t∧τn s∧τn -(γ α + u) -α ∇f (X u ), ∇g p (X u ) du (4.21) + (γ α /2)E t∧τn s∧τn (γ α + u) -2α Σ(X u ), ∇ 2 g p (X u ) du .
Using A4.1, (4.20) and the Cauchy-Schwarz inequality we get that for any u ∈

[s, s + T ] | ∇f (X u ), ∇g p (X u ) | ≤ 2p X u 2(p-1) {| ∇f (X u ) -∇f (0), X u | + ∇f (0) X u }(4.22) ≤ 2p(L + ∇f (0) )g p (X u ) .
In addition, using A4.1, A4.2, (4.20) and the Cauchy-Schwarz inequality we get that for any

u ∈ [s, s + T ] Σ(X u ), ∇ 2 g p (X u ) = 2p X u 2(p-1) Z ∇f (X u ) -H(X u , z) 2 dµ Z (z) + 4p(p -1) X u 2(p-2) Z X u , H(X u , z) -∇f (X u ) 2 dµ Z (z) ≤ 2p(2p -1) X u 2(p-1) η ≤ 2p(2p -1)ηg p (X u ) . ( 4
∈ N E [g p (X t∧τn )] -g p (X 0 ) ≤ 2p(L + ∇f (0) )E t∧τn s g p (X u )du + γα p(2p -1)E t∧τn s g p (X u )du ≤ {2p(L + ∇f (0) ) + γα p(2p -1)} t s E [g p (X ∧τn )] du . Using Grönwall's lemma we obtain E [g p (X t∧τn )] ≤ g p (X 0 ) exp [T {2p(L + ∇f (0) ) + γα p(2p -1)}] .
We conclude upon using Fatou's lemma and remarking that lim n τ n = +∞, since X t is well-defined for any t ≥ 0.

(b) Let (X k ) k∈N be a sequence which satisfies the recursion (4.1) with

X k0 = X 0 . Let A k = X k -γ(k + 1) -α ∇f (X k ) and B k = γ(k + 1) -α (∇f (X k ) -H(X k , Z k+1 )).
We have, using Cauchy-Schwarz inequality and the binomial formula,

X k+1 2p = A k + B k 2p = A k 2 + 2 A k , B k + B k 2 p ≤ p i=0 i j=0 p i i j A k 2(p-i)+j B k 2i-j × 2 j ≤ A k 2p + 2 p p i=1 i j=0 p i i j A k 2(p-i)+j B k 2i-j . (4.24) Using A4.1, there exists Ã(a) T,1 , Ã(b) T,1 , Ã(c) T,1 ≥ 0 such that for any ∈ {0, . . . , 2p} A k ≤ m=0 m (1 + γ(k + 1) -α L) m X k m γ(k + 1) -α ∇f (0) -m ≤ (1 + γ(k + 1) -α Ã(a) T,1 ) X k + γ(k + 1) -α Ã(b) T,1 (1 + X k 2p ) ≤ (1 + γ(k + 1) -α Ã(c) T,1 )(1 + X k 2p ) .
Combining this result, (4.24), Jensen's inequality and that for any

∈ N, E B k 2 F k ≤ γ 2 (k + 1) -2α η we have E X k+1 2p F k ≤ (1 + γ(k + 1) -α Ã(a) T,1 ) X k + γ(k + 1) -α Ã(b) T,1 (1 + X k 2p ) + 2 p (1 + γ(k + 1) -α Ã(c) T,1 )(1 + X k 2p ) p i=1 i j=0 p i i j η 1/2 2i-j γ 2i-j (k + 1) -α(2i-j) .
Therefore, there exists Ã(d) T,1 ≥ 0 such that

E 1 + X k+1 2p ≤ (1 + Ã(d) T,1 γ(k + 1) -α )E 1 + X k 2p + Ã(d) T,1 γ(k + 1) -α .
We conclude combining this result, Lemma 4.5 and Lemma 4.6.

We use the previous result to prove the following lemma.

Lemma 4.8. Let p ∈ N, γ > 0 and α ∈ [0, 1). Assume A4.1, A4.2 and that for any

x ∈ R d , µ Z ( H(x, •) -∇f (x) 2p ) ≤ η p , with η p ≥ 0.
Then for any T ≥ 0, there exists A T,2 ≥ 0 such that for any γ ∈ (0, γ], k ∈ N with (k + 1)γ α ≤ T , t ∈ [kγ α , (k + 1)γ α ] and

X 0 ∈ R d , we have max E X k+1 -X 0 2p , E X t -X 0 2p ≤ A T,2 (k + 1) -2αp γ 2p (1 + X 0 2p ) ,
where (X k ) k∈N satisfies the recursion (4.1) with X k = X 0 and (X t ) t≥0 is the solution of (4.2) with X kγα = X 0 .

Proof. Let p ∈ N, α ∈ [0, 1), γ > 0, γ ∈ (0, γ], k ∈ N, t ∈ [kγ α , (k + 1)γ α ] and X 0 ∈ R d .
We divide the rest of the proof into two parts.

(a) Let (X t ) t≥0 be a solution to (4.2) such that X kγα = X 0 . Using A4.1, A4.2, Jensen's inequality, Burkholder-Davis-Gundy's inequality (Rogers and Williams, 2000, Theorem 42.1) and Lemma 4.7 there exists B p ≥ 0 such that

E X t -X 0 2p ≤ 2 2p-1 E t kγα (γ α + s) -α ∇f (X s )ds 2p + 2 2p-1 γ p α E t kγα (γ α + s) -α Σ(X s ) 1/2 dB s 2p ≤ 2 2p-1 γ 2p-1 α t kγα (γ α + s) -2αp E ∇f (X s ) 2p ds + B p 2 2p-1 γ p α t kγα (γ α + s) -2α E [Tr(Σ(X s ))] ds p ≤ 2 2p-1 γ 2p-1-2αp α (k + 1) -2αp (B p + 1) t kγα E ∇f (X s ) 2p ds + t kγα E [Tr(Σ(X s ))] p ds ≤ 2 4p-2 (1 + L 2p )γ 2p γ -1 α (k + 1) -2αp (B p + 1) t kγα ∇f (0) 2p ds + t kγα E X s 2p + η p ds ≤ 2 4p-2 (1 + L 2p )γ 2p (k + 1) -2αp (B p + 1) ∇f (0) 2p +η p + sup s∈[kγα,t] E X s 2p ≤ 2 4p-2 (1 + L 2p )γ 2p (k + 1) -2αp (B p + 1) ∇f (0) 2p + η p + A T,1 g p (X 0 ) . (4.25) (b) Let (X n ) n∈N satisfying the recursion (4.1) with X k = X 0 . Using A4.1 and A4.2 we get that E X k+1 -X 0 2p = E -γ(k + 1) -α (∇f (X 0 ) + H(X 0 , Z k+1 ) -∇f (X 0 )) 2p ≤ γ 2p (k + 1) -2αp 3 2p-1 L 2p ∇f (0) 2p + L 2p X 0 2p + Z H(X 0 , z) -∇f (X 0 ) 2p dµ Z (z) ≤ γ 2p (k + 1) -2αp 3 2p-1 (1 + L 2p ) 1 + ∇f (0) 2p + η p (1 + X 0 2p ) . (4.26)
Combining (4.25) and (4.26) and setting

A T,2 = 2 4p-2 (1 + L 2p ) ∇f (0) 2p + η p + max(A T,1 , 1) ,
conclude the proof upon remarking that η p ≤ η p .

4.A.3 Mean-square approximation

Now consider the stochastic process (X t ) t≥0 defined by X 0 = X 0 and solution of the following SDE

dX t = -γ -1 α +∞ k=0 1 [kγα,(k+1)γα) (t)(1 + k) -α γ ∇f (X kγα )dt + γ 1/2 α Σ(X kγα ) 1/2 dB t .
(4.27) Note that for any k ∈ N, we have

X (k+1)γα = X kγα -γ(k + 1) -α ∇f (X kγα ) + Σ(X kγα ) 1/2 G k , with G k = γ -1/2 α (k+1)γα kγα dB s .
Hence, for any k ∈ N, X kγα has the same distribution as X k given by (4.1) with H(x, z) = ∇f (x) + Σ(x) 1/2 z, (Z, Z) = (R d , B(R d )) and µ Z the Gaussian probability distribution with zero mean and covariance matrix identity. Lemma 4.9. Let γ > 0 and α ∈ [0, 1). Assume A4.2. Then for any T ≥ 0, there exists A T,3 ≥ 0 such that for any γ ∈ (0, γ], k ∈ N with (k + 1)γ α ≤ T and X 0 ∈ R d we have

E X (k+1)γα -X k+1 2 ≤ A T,3 γ 2 (k + 1) -2α (1 + X 0 2 ) ,
where (X k ) k∈N satisfies the recursion (4.1) with X k = X 0 and (X t ) t≥0 is the solution of (4.27) with X kγα = X 0 .

Proof. Let α ∈ [0, 1), γ > 0, γ ∈ (0, γ], k ∈ N, t ∈ [kγ α , (k + 1)γ α ] and X 0 ∈ R d . Let (X k ) k∈N satisfy the recursion (4.1) with X k = X 0 and (X t ) t≥0 be the solution of (4.27) with X kγα = X 0 . Using A4.2 we have

E X (k+1)γα -X k+1 2 = γ 2 (k + 1) -2α E ∇f (X 0 ) + Σ 1/2 (X 0 )G k -H(X 0 , Z k ) 2 ≤ 2γ 2 (k + 1) -2α E ∇f (X 0 ) -H(X 0 , Z k ) 2 + E Σ 1/2 (X 0 )G k 2 ≤ 4γ 2 (k + 1) -2α η ,
which concludes the proof.

Lemma 4.10. Let γ > 0 and α ∈ [0, 1). Assume A4.1, A4.2 and A4.3. Then for any T ≥ 0, there exists A T,4 ≥ 0 such that for any γ ∈ (0, γ], k ∈ N with (k + 1)γ α ≤ T and

X 0 ∈ R d we have E X (k+1)γα -X (k+1)γα 2 ≤ A T,4 γ 4 (k + 1) -4α + γ 2 (k + 1) -2(1+α) (1 + X 0 2 ) ,
where (X t ) t≥0 be the solution of (4.2) with X kγα = X 0 and (X t ) t≥0 be the solution of (4.27) with X kγα = X 0 .

Proof

. Let α ∈ [0, 1), γ > 0, γ ∈ (0, γ], k ∈ N, t ∈ [kγ α , (k + 1)γ α ] and X 0 ∈ R d . Let (X t ) t≥0
is the solution of (4.2) with X kγα = X 0 and (X t ) t≥0 is the solution of (4.27) with X kγα = X 0 . Using Jensen's inequality and that γ α γ -1 = γ α α we have

E X (k+1)γα -X (k+1)γα 2 ≤ E - (k+1)γα kγα (γ α + s) -α ∇f (X s )ds -γ 1/2 α (k+1)γα kγα (γ α + s) -α Σ(X s ) 1/2 dB s +γ(k + 1) -α ∇f (X 0 ) + γγ -1/2 α (k + 1) -α Σ(X 0 ) 1/2 (k+1)γα kγα dB s 2   ≤ 2E   -γ -α α (k+1)γα kγα (1 + γ -1 α s) -α ∇f (X s )ds + γ(k + 1) -α ∇f (X 0 ) 2   + 2E -γ 1/2-α α (k+1)γα kγα (1 + γ -1 α s) -α Σ(X s ) 1/2 dB s +γγ -1/2 α (k + 1) -α Σ(X 0 ) 1/2 (k+1)γα kγα dB s 2   ≤ 2γ -2α α E   (k+1)γα kγα (k + 1) -α ∇f (X 0 ) -(1 + γ -1 α s) -α ∇f (X s ) ds 2   + 2γ 1-2α α E   (k+1)γα kγα (k + 1) -α Σ(X 0 ) 1/2 -(1 + γ -1 α s) -α Σ(X s ) 1/2 dB s 2   .
(4.28)

We now treat each term separately. Using Jensen's inequality, Fubini-Tonelli's theorem, the fact that for any u > 0, u -α -(u + 1) -α ≤ αu -(α+1) , A4.1 and Lemma 4.8 we get that

E   (k+1)γα kγα (k + 1) -α ∇f (X 0 ) -(1 + γ -1 α s) -α ∇f (X s ) ds 2   ≤ γ 2 α sup s∈[kγα,(k+1)γα] E (k + 1) -α ∇f (X 0 ) -(1 + γ -1 α s) -α ∇f (X s ) 2 ds ≤ 2γ 2 α sup s∈[kγα,(k+1)γα] ∇f (X 0 ) 2 |(k + 1) -α -(1 + γ -1 α s) -α | 2 +(1 + γ α s -1 ) -2α E ∇f (X s ) -∇f (X 0 ) 2 ≤ 2γ 2 α α 2 ∇f (X 0 ) 2 (k + 1) -2(1+α) + (k + 1) -2α L 2 sup s∈[kγα,(k+1)γα] E X s -X 0 2 ≤ 2γ 2 α α 2 ∇f (X 0 ) 2 (k + 1) -2(1+α) + (k + 1) -4α L 2 A T,2 γ 2 (1 + X 0 2 ) ≤ 2γ 2 α α 2 ( ∇f (0) 2 + L 2 )(k + 1) -2(1+α) + (k + 1) -4α L 2 A T,2 γ 2 (1 + X 0 2 ) . (4.29)
In addition, using Jensen's inequality, Itô isometry, Fubini-Tonelli's theorem, A4.1, A4.3 and Lemma 4.8 we have

E   (k+1)γα kγα (k + 1) -α Σ(X 0 ) 1/2 -(1 + γ -1 α s) -α Σ(X s ) 1/2 dB s 2   ≤ 2 (k + 1) -2α | (k+1)γα kγα E Σ(X 0 ) 1/2 -Σ(X s ) 1/2 2 ds| +η| (k+1)γα kγα {(k + 1) -α -(1 + γ -1 α s)} 2 ds| ≤ 2γ α (k + 1) -2α M 2 sup s∈[kγα,(k+1)γα] E X s -X 0 2 + ηα 2 (k + 1) -2(1+α) ≤ 2γ α (k + 1) -4α M 2 A T,2 γ 2 + ηα 2 (k + 1) -2(1+α) (1 + X 0 2 ) . (4.30)
Combining (4.28), (4.29) and (4.30) concludes the proof upon setting

A T,4 = 4 M 2 A T,2 + ηα 2 + α 2 ( ∇f (0) 2 + L 2 ) + L 2 A T,2 .
Proposition 4.5. Let γ > 0 and α ∈ [0, 1). Assume A4.1, A4.2 and A4.3. Then for any T ≥ 0, there exists A T,5 ≥ 0 such that for any γ ∈ (0, γ], k ∈ N with (k + 1)γ α ≤ T and X 0 ∈ R d we have

E X (k+1)γα -X k+1 2 ≤ A T,5 γ 4 (k + 1) -4α + γ 2 (k + 1) -2α (1 + X 0 2 ) ,
where (X k ) k∈N satisfies the recursion (4.1) with X k = X 0 and (X t ) t≥0 is the solution of (4.2) with X kγα = X 0 .

Proof. The proof is straightforward upon combining Lemma 4.9 and Lemma 4.10.

We obtain now the following proposition which is a restatement of Proposition 4.1.

Proposition 4.6. Let γ > 0 and α ∈ [0, 1). Assume A4.1, A4.2 and A4.3. Then for any T ≥ 0, there exists A 1 ≥ 0 such that for any γ ∈ (0, γ], k ∈ N with kγ α ≤ T we have

E 1/2 X kγα -X k 2 ≤ A 1 γ δ (1 + log(γ -1 )) , with δ = min(1, (1 -α) -1 /2). If in addition, (Z, Z) = (R d , B(R d )) and for any x ∈ R d , z ∈ R d and n ∈ N, H(x, z) = ∇f (x) + Σ(x) 1/2 z , Z n+1 = γ -1 α (n+1)γα nγα dB s , then δ = 1. Proof. Let p ∈ N, α ∈ [0, 1), γ > 0, γ ∈ (0, γ], k ∈ N, and X 0 ∈ R d . Let (E k ) k∈N such that for any k ∈ N, E k = E X kγα -X k 2 . Note that E 0 = 0. Let Y (k+1)γα = X kγα -γ(k + 1) -α H(X kγα , Z k+1 ).
We have

E k+1 = E X (k+1)γα -X k+1 2 = E X (k+1)γα -Y (k+1)γα + Y (k+1)γα -X k+1 2 = E X (k+1)γα -Y (k+1)γα 2 + 2E X (k+1)γα -Y (k+1)γα , Y (k+1)γα -X k+1 + E Y (k+1)γα -X k+1 2 = E X (k+1)γα -Y (k+1)γα 2 + E Y (k+1)γα -X k+1 2 + 2E X (k+1)γα -Y (k+1)γα , X kγα -X k + 2γ(k + 1) -α E X (k+1)γα -Y (k+1)γα , H(X k , Z k+1 ) -H(X kγα , Z k+1 ) . (4.31) Let a k = γ 4 (k + 1) -4α + γ 2 (k + 1) -2α
. We now bound each of the four terms appearing in (4.31) (a) First, we have using Proposition 4.5 and Lemma 4.7

E X (k+1)γα -Y (k+1)γα 2 = E E X (k+1)γα -Y (k+1)γα 2 X kγα ≤ E A T,5 (γ 4 (k + 1) -4α + γ 2 (k + 1) -2α ) 1 + X kγα 2 ≤ A T,1 A T,5 (γ 4 (k + 1) -4α + γ 2 (k + 1) -2α ) 1 + X 0 2 ≤ A (a) T,6 a k , ( 4
≤ 2a 2 + 2b 2 E Y (k+1)γα -X k+1 2 = E X kγα -X k -γ(k + 1) -α (H(X kγα , Z k+1 ) -H(X k , Z k+1 )) 2 = E X kγα -γ(k + 1) -α ∇f (X kγα ) -X k + γ(k + 1) -α ∇f (X k ) 2 + γ 2 (k + 1) -2α E [ H(X kγα , Z k+1 ) -∇f (X kγα ) +H(X k , Z k+1 ) -∇f (X k ) 2 ≤ (1 + γL(k + 1) -α ) 2 X kγα -X k 2 + 4γ 2 (k + 1) -2α ≤ (1 + 2γL(k + 1) -α + γ 2 L 2 (k + 1) -2α )E k + 4γ 2 (k + 1) -2α ≤ (1 + A (b) T,6 a 1/2 k )E k + 4a k , (4.33) with A (b)
T,6 ≥ 0 which does not depend on γ and k.

(c) Let Y (k+1)γα = X kγα -γ(k+1) -α ∇f (X kγα ) + Σ(X kγα ) 1/2 G k , with G k = γ -1/2 α (k+1)γα kγα dB s . Let b k = γ 3 (k + 1) -3α + γ(k + 1) -2(1+α/2) . Using A4.2 we have E Y (k+1)γα σ(X kγα ) = E Y (k+1)γα σ(X kγα ) .
Combining this result, the Cauchy-Schwarz inequality, Lemma 4.10, Lemma 4.7 and that for any a, b ≥ 0, (a+b

) 1/2 ≤ a 1/2 + b 1/2 and 2ab ≤ a 2 + b 2 we obtain E X (k+1)γα -Y (k+1)γα , X kγα -X k = E E X (k+1)γα -Y (k+1)γα σ(X kγα , X k ) , X kγα -X k = E E X (k+1)γα -Y (k+1)γα σ(X kγα , X k ) , X kγα -X k ≤ E E 1/2 X (k+1)γα -Y (k+1)γα 2 σ(X kγα , X k ) X kγα -X k ≤ E 1/2 X (k+1)γα -Y (k+1)γα 2 E 1/2 X kγα -X k 2 ≤ A 1/2 T,1 A 1/2 T,4 γ 4 (k + 1) -4α + γ 2 (k + 1) -2(1+α) 1/2 (1 + X 0 2 )E 1/2 k ≤ A 1/2 T,1 A 1/2 T,4 γ 3/2 (k + 1) -3α/2 + γ 1/2 (k + 1) -(1+α/2) (1 + X 0 2 )γ 1/2 (k + 1) -α/2 E 1/2 k ≤ A (c) T,6 γ 3 (k + 1) -3α + γ(k + 1) -2(1+α/2) /2 + a 1/2 k E k /2 ≤ A (c) T,6 b k /2 + a 1/2 k E k /2 . (4.34) with A (c)
T,6 ≥ 0 which does not depend on γ and k. (d) Finally, using the Cauchy-Schwarz inequality, (4.32), A4.2 and A4.1 and that for any a, b ≥ 0, (a + b) 1/2 ≤ a 1/2 + b 1/2 , we have

γ(k + 1) -α E X (k+1)γα -Y (k+1)γα , H(X k , Z k+1 ) -H(X kγα , Z k+1 ) ≤ γ(k + 1) -α E 1/2 X (k+1)γα -Y (k+1)γα 2 E 1/2 H(X k , Z k+1 ) -H(X kγα , Z k+1 ) 2 ≤ (A (a) T,6 ) 1/2 γ(k + 1) -α a 1/2 k LE 1/2 X kγα -X k 2 + √ 2η ≤ (A (a) T,6 ) 1/2 γ(k + 1) -α a 1/2 k √ 3LE 1/2 X kγα -X k 2 + √ 6 √ η ≤ (A (a) T,6 ) 1/2 γ(k + 1) -α a 1/2 k 2LE 1/2 k + √ 6 √ η(A (a) T,6 ) 1/2 γ(k + 1) -α a 1/2 k ≤ A (a) T,6 γ 2 (k + 1) -2α a 1/2 k L 2 + a 1/2 k E k + √ 6 √ η(A (a) T,6 ) 1/2 γ(k + 1) -α a 1/2 k ≤ A (d) T,6 a k + a 3/2 k + a 1/2 k E k , (4.35) with A (d)
T,6 ≥ 0 which does not depend on γ and k. Finally, we have using (4.32), (4.33), (4.34) and (4.35) in (4.31)

E k+1 ≤ 1 + (2 + A (b) T,6 )a 1/2 k E k + (4 + A (a) T,6 + A (d) T,6 )a k + A (d) T,6 a 3/2 k + A (c) T,6 b k (4.36) ≤ 1 + (2 + A (b) T,6 )a 1/2 k E k + (4 + A (a) T,6 + 2A (d) T,6 + A (c) T,6 )(a k + a 3/2 k + b k ) .
Using Lemma 4.6 and that a

1/2 k ≤ γ(k + 1) -α + γ 2 (k + 1) -2α , there exists A (e)
T,6 ≥ 0 which does not depend on γ and k such that

(2 + A (b) T,6 ) N -1 k=0 a 1/2 k ≤ A (e)
T,6 .

(4.37)

In addition, we have

a k + a 3/2 k + b k ≤ (1 + 2 3/2 ) γ 2 (k + 1) -2α + γ 3 (k + 1) -3α + γ 4 (k + 1) -4α + γ 6 (k + 1) -6α + γ(k + 1) -2(1+α) .
Therefore, using that γγ α α = γ α and Lemma 4.6 there exists A (f ) T,6 ≥ 0 which does not depend on γ and k such that

N -1 k=0 (4 + A (a) T,6 + 2A (d) T,6 + A (c) T,6 )(a k + a 3/2 k + b k ) ≤    A (f ) T,6 γ 2 (1 + log(γ -1 )) if α ≥ 1/2 , A (f ) T,6 γ α if α < 1/2 . (4.38) We denote v k = (2 + A (b) T,6 )a 1/2 k and w k = (4 + A (a) T,6 + 2A (d) T,6 + A (c) T,6 )(a k + a 3/2 k + b k )
. Using (4.36) and Lemma 4.5 we obtain that

E k ≤ N -1 k=0 w k + exp N -1 k=0 v k N -1 k=0 v k w k (4.39) ≤ N -1 k=0 w k + exp N -1 k=0 v k N -1 k=0 v k N -1 k=0 w k .
Combining (4.37), (4.38) and (4.39) concludes the first part of the proof.

For the second part of the proof H(x, z) = ∇f (x) + Σ(x) 1/2 z and for any k ∈ N, we have

Z k+1 = (k+1)γα kγα dB s . We denote c k = γ 4 (k + 1) -4α + γ 2 (k + 1) -2(1+α
(c) T,6 a 1/2 k c k + a 1/2 k E k .
The rest of the proof is similar to the general case.

4.A.4 Weak approximation

We recall that G p is the set of twice continuously differentiable functions from R d to R such that for any g ∈ G p , there exists K ≥ 0 such that for any

x ∈ R d max ∇g(x) , ∇ 2 g(x) ≤ K(1 + x p ) , (4.40)
with p ∈ N.

The following lemma will be useful.

Lemma 4.11. Let p ∈ N, g ∈ G p and let K ≥ 0 as in (4.40). Then, for any

x, y ∈ R d |g(y) -g(x) -∇g(x), y -x | ≤ K(1 + x p + y p ) x -y 2 .
Proof. Using that for any x → x p is convex, and Cauchy-Schwarz inequality we get for any

x, y ∈ R d |g(x) -g(y) -∇g(x), y -x | ≤ 1 0 |∇ 2 g(x + t(y -x))(y -x) ⊗2 |dt ≤ x -y 2 1 0 |∇ 2 g(x + t(y -x))(y -x) ⊗2 |dt ≤ K(1 + x p + y p ) x -y 2 .
Before giving the proof of Proposition 4.2, we highlight that the result is straightforward for α ∈ [1/2, 1). Proposition 4.7. Let γ > 0 and α ∈ [1/2, 1) and p ∈ N. Assume A4.1, A4.2 and A4.3.

In addition, assume that for any

x ∈ R d , µ Z ( H(x, •) -∇f (x) 2p ) ≤ η p , with η p ≥ 0.
Then for any T ≥ 0 and g ∈ G p , there exists A T,7 ≥ 0 such that for any γ ∈ (0

, γ], k ∈ N with kγ α ≤ T and X 0 ∈ R d we have E [|g(X kγα ) -g(X k )|] ≤ A T,7 γ(1 + log(γ -1 )) ,
where (X k ) k∈N satisfies the recursion (4.1) and (X t ) t≥0 is the solution of (4.2) with X

0 = X 0 Proof. Let p ∈ N, g ∈ G p , α ∈ [1/2, 1), γ > 0, γ ∈ (0, γ], k ∈ N, and X 0 ∈ R d . Using that for any x → x p is convex, for any x, y ∈ R d we get |g(x) -g(y)| ≤ 1 0 | ∇g(x + t(y -x)), y -x |dt ≤ y -x 1 0 ∇g(x + t(y -x)) dt ≤ y -x K(1 + x p + y p ) . Proof. For any k ∈ N with kγ α ≤ T , let g k (x) = E [g(X kγα )] with X 0 = x. Since f ∈ G p,4 , Σ 1/2 ∈ G p,3
and g ∈ G p,2 one can show, see [START_REF] Blagovescenskii | Some properties of diffusion processes depending on a parameter[END_REF] or (Kunita, 1981, Proposition 2.1), that there exists m ∈ N and K ≥ 0 such that for any k ∈

N g k ∈ C m (R d , R) and max { g k (x) , . . . , ∇ m g k (x) } ≤ K(1 + x p ) .
Therefore, g k ∈ G p,m with constants uniform in k ∈ N. In addition, for any k ∈ N with kγ α ≤ T , let h

(1)

k (x) = E [g k (X k+1 )] with X k = x and h (2) k (x) = E g k (X (k+1)γα ) with X kγα = x. Using Proposition 4.8 we have for any k ∈ N, kγ α ≤ T |h (1) k (x) -h (2) k (x)| ≤ A T,8 γ 2 (k + 1) -2α + γ(k + 1) -(1+α) (1 + x m+2 ) .
Therefore, using Lemma 4.7 we have for any k ∈ N with kγ α ≤ T and j ≤ k,

|E h

(1)

k-j-1 (X j ) -h (2) k-j-1 (X j ) | ≤ ÃT,1 A T,8 γ 2 (k + 1) -2α + γ(k + 1) -(1+α) (1+ X 0 m+2 ) .
(4.41) Now, let k ∈ N with kγ α ≤ T and consider the family {(X j ) ∈N : j = 0, . . . , N }, defined by the following recursion: for any j ∈ {0, . . . , N } X j 0 = X 0 and for any ∈ N:

(a) if ≥ j, X j +1 = X j -γ(k + 1) -α H(X j , Z +1 ) , (b) if < j, X j +1 = X j ( +1)γα
, where X j γα = X j and for any t ∈ [ γ α , ( + 1)γ α ] we have

X j t = X j - t γα (γ α + s) -α ∇f (X j s )ds -γ 1/2 α t γα (γ α + s) -α Σ 1/2 (X j s )dB s .
We have

|E [g(X kγα ) -g(X k )]| = |E g(X k k ) -g(X 0 k ) | = k-1 j=0 |E g(X j+1 k ) -g(X j k ) | .
Using (4.41) we get

|E g(X j+1 k ) -g(X j k ) | = |E E g(X j k ) -g(X j+1 k ) X j k | = |E h (1) k-j-1 (X j ) -h (2) k-j-1 (X j ) | ≤ ÃT,1 A T,8 γ 2 (k + 1) -2α + γ(k + 1) -(1+α) (1 + X 0 m+2 ) ≤ A (a) T,9 γ 2 (k + 1) -2α + γ(k + 1) -(1+α) , with A (a)
T,9 ≥ 0 which does not depend on k or γ In addition, using Lemma 4.6 there exists A

(b) T,9 ≥ 0 such that N -1 k=0 γ 2 (k + 1) -2α + γ(k + 1) -(1+α) ≤ A (b) T,9 γ .
Combining these last two results concludes the proof.

4.A.5 Tightness of the mean-square approximation bound

In this section, we show that the upper-bound derived in Proposition 4.1 is sharp (up to a logarithmic term). Proposition 4.10. Let γ > 0, α ∈ [0, 1), (Z, Z) = (R d , B(R d )), (Z k ) k∈N a sequence of independent d-dimensional Gaussian random variables independent from ( (k+1)γα kγα dB s ) k∈N , H(x, z) = z and f = 0. Then there exists Ã1 ≥ 0 such that for any γ ∈ (0, γ] we have

E 1/2 X kγα -X k 2 ≥ Ã1 γ δ ,
with N = T /γ α and δ = min(1, (1 -α) -1 /2).

Proof. First, remark that for any x ∈ R d , Σ(x) = Id. We have using Itô's isometry

E X N γα -X N 2 = E   γ 1/2 α N γα 0 (s + γ α ) -α dB s -γ N -1 k=0 (k + 1) -α Z k+1 2   = N -1 k=0 γ α E (k+1)γα kγα (s + γ α ) -2α ds + γ 2 (k + 1) -2α ≥ γ 2 N -1 k=0 (k + 1) -2α ≥ γ 2 N +1/2 1/2 (s + 1) -2α .
We now distinguish three cases.

(a

) If α = 1/2 then E X N γα -X N 2 ≥ γ 2 (log(N + 1/2) -log(1/2)) ≥ Ã(a) 1 γ 2 , with Ã(a) 1 which does not depend on N or γ. (b) If α > 1/2, E X N γα -X N 2 ≥ γ 2 (3/2) -2α+1 (2α -1) -1 ≥ Ã(b) 1 γ 2 , with Ã(b) 1 which does not depend on N or γ. (c) If α < 1/2, E X N γα -X N 2 ≥ γ 2 (N + 3/2) -2α+1 (1 -2α) -1 ≥ γ 2 γ 2α-1 α (T + 3γ α /2) -2α+1 (1 -2α) -1 ≥ Ã(c) 1 γ α ,
with Ã(c) 1 which does not depend on N or γ.

4.B Technical results

We present in this section technical results that are useful for all the proofs of the different convergence rates. Most of them are known but are recalled here for clarity and completeness.

Lemma 4.12. Let f ∈ C 2 (R d , R). Assume A4.1 and A4.2. Then for any

x ∈ R d we have | ∇ 2 f (x), Σ(x) | ≤ Lη , | ∇f (x)∇f (x) , Σ(x) | ≤ η ∇f (x) 2 . Proof. Let x ∈ R d . Using Cauchy-Schwarz's inequality, we have | ∇ 2 f (x), Σ(x) | ≤ ∇ 2 f (x) Σ(x) * ,
where • is the operator norm and • * is the nuclear norm. Using A4.1 we have ∇ 2 f (x) ≤ L for all x ∈ R d . In addition, denoting (λ i ) i∈{1,...,d} the eigenvalues of Σ(x), using that Σ is positive semidefinite and A4.2 we have

Σ(x) * = d i=1 |λ i | = d i=1 λ i = Tr(Σ(x)) ≤ η .
This concludes the first part of the proof.

For the second part we have

| ∇f (x)∇f (x) , Σ(x) | ≤ sup i∈{1,...,d} λ i ∇f (x) 2 ≤ η ∇f (x) 2 .
The following lemma consists into taking the expectation in Itô's formula, and is known as Dynkins's lemma. Lemma 4.13. Let α ∈ [0, 1) and γ > 0. Assume f, g ∈ C 2 (R d , R), A4.1, A4.2 and A4.3 and let (X t ) t≥0 solution of (4.2). Then for any ϕ

∈ C 1 ([0, +∞) , R), Y ∈ F 0 and E Y 2 + |g(Y )| < +∞, we have the following results: (a) For any t ≥ 0, E X t -Y 2 ϕ(t) = E X 0 -Y 2 ϕ(0) -2 t 0 (γ α + s) -α ϕ(s)E [ ∇f (X s ), X s -Y ] ds + γ α t 0 (γ α + s) -2α ϕ(s)E [Tr(Σ(X s ))] ds + t 0 ϕ (s)E X s -Y 2 ds , (4.42) (b) For any t ≥ 0, E [(f (X t ) -g(Y ))ϕ(t)] = E [(f (X 0 ) -g(Y ))ϕ(0)] - t 0 (γ α + s) -α ϕ(s)E ∇f (X s ) 2 ds + (γ α /2) t 0 (γ α + s) -2α ϕ(s)E ∇ 2 f (X s ), Σ(X s ) ds + t 0 ϕ (s)E [f (X s ) -g(Y )] ds . (c) If E[ Y 2p ] < +∞, then for any t ≥ 0 E X t -Y 2p ϕ(t) = E X 0 -Y 2p ϕ(0) -2p t 0 (γ α + s) -α ϕ(s)E ∇f (X s ), X s -Y X t -Y 2(p-1) ds + γ α p t 0 (γ α + s) -2α ϕ(s)E Tr(Σ(X s )) X s -Y 2(p-1) ds + γ α 2p(p -1) t 0 (γ α + s) -2α ϕ(s)E Σ(X s ), (X t -Y )(X t -Y ) X s -Y 2(p-2) ds + t 0 ϕ (s)E (f (X s ) -g(Y )) 2p ds . (d) If E[|g(Y )| p ] < +∞, then for any t ≥ 0 E [(f (X t ) -g(Y )) p ϕ(t)] = E [(f (X 0 ) -g(Y )) p ϕ(0)] -p t 0 (γ α + s) -α ϕ(s)E ∇f (X s ) 2 (f (X s ) -g(Y )) p-1 ds + γ α (p/2) t 0 (γ α + s) -2α ϕ(s)E ∇ 2 f (X s ), Σ(X s ) (f (X s ) -g(Y )) p-2 ds + γ α p(p -1)/2 t 0 (γ α + s) -2α ϕ(s)E ∇f (X s )∇f (X s ) , Σ(X s ) (f (X s ) -g(Y )) p-2 + t 0 ϕ (s)E [(f (X s ) -g(Y )) p ] ds .
Proof. Let α ∈ [0, 1), γ > 0 and (X t ) t≥0 the solution of (4.2). Note that for any t ≥ 0, we have

X t = γ α t 0 (γ α + s) -2α Tr(Σ(X s ))ds .
We divide the rest of the proof into our parts.

(a) First, let y ∈ R d and

F y : [0, +∞) × R d such that for any t ∈ [0, +∞), x ∈ R d , F y (t, x) = ϕ(t)
x -y 2 . Since (X t ) t≥0 is a strong solution of (4.2) we have that (X t ) t≥0 is a continuous semi-martingale. Using this result, the fact that F ∈ C 1,2 ([0, +∞) , R d ) and Itô's lemma (Karatzas and Shreve, 1991, Chapter 3, Theorem 3.6) we obtain that for any t ≥ 0 almost surely

F y (t, X t ) = F y (0, X 0 ) + t 0 ∂ 1 F y (s, X s )ds + t 0 ∂ 2 F y (s, X s ), dX s (4.43) + (1/2) t 0 ∂ 2,2 F y (s, X s ), d X s = F y (0, X 0 ) + t 0 ϕ (s) X s -y 2 ds + t 0 ∂ 2 F y (s, X s ), dX s + (1/2) t 0 ∂ 2,2 F y (s, X s ), d X s = F y (0, X 0 ) + t 0 ϕ (s) X s -y 2 ds -2 t 0 (γ α + s) -α ϕ(s) ∇f (X s ), X s -y ds + 2γ 1/2 α t 0 (γ α + s) -α ϕ(s) X s -y, Σ(X s ) 1/2 dB s + γ α t 0 (γ α + s) -2α ϕ(s) Tr(Σ(X s ))ds . Using A4.1 have for any x ∈ R d , | ∇f (x), x -y | ≤ ∇f (0) x -y + L x x -y .
Therefore, using this result Lemma 4.7, Cauchy-Schwarz's inequality and that E[ Y 2 ] < +∞, we obtain that for any t ≥ 0 there exists Ā ≥ 0 such that sup

s∈[0,t] E X s -Y 2 ≤ Ā , sup s∈[0,t] E [| ∇f (X s ), X s -Y |] ≤ Ā . (4.44)
In addition, we have using A4.2 that for any

t ≥ 0, E[| Tr(Σ(X s ))|] = E[Tr(Σ(X s ))] ≤ η.
Combining this result, (4.44), (4.43), that (

t 0 (γ α + t) -α ϕ(t) X t -Y, Σ(X t ) 1/2 dB t ) t≥0
is a martingale and Fubini-Lebesgue's theorem we obtain for any t ≥ 0

E ϕ(t) X t -Y 2 = E [E [F Y (t, X t )|F 0 ]] = E ϕ(0) X 0 -Y 2 + t 0 ϕ (s)E X s -Y 2 ds -2 t 0 (γ α + s) -α ϕ(s)E [ ∇f (X s ), X s -Y ] ds 181 + γ α t 0 (γ α + s) -2α ϕ(s)E [Tr(Σ(X s ))] ds ,
which concludes the proof of (4.42).

(b) Second, let y ∈ R d and F :

[0, +∞) × R d such that for any t ∈ [0, +∞), x ∈ R d , F y (t, x) = ϕ(t)(f (x) -g(y)).
Using that (X t ) t≥0 is a continuous semi-martingale, the fact that F ∈ C 1,2 ([0, +∞) , R d ) and Itô's lemma (Karatzas and Shreve, 1991, Chapter 3, Theorem 3.6) we obtain that for any t ≥ 0 almost surely

F y (t, X t ) = F y (0, X 0 ) + t 0 ∂ 1 F y (s, X s )ds + t 0 ∂ 2 F y (s, X s ), dX s + (1/2) t 0 ∂ 2,2 F y (s, X s ), d X s = F y (0, X 0 ) + t 0 ϕ (s)(f (X s ) -g(y))ds + t 0 ∂ 2 F y (s, X s ), dX s + (1/2) t 0 ∂ 2,2 F y (s, X s ), d X s = F y (0, X 0 ) + t 0 ϕ (s)(f (X s ) -g(y))ds - t 0 (γ α + s) -α ϕ(s) ∇f (X s ) 2 ds + γ 1/2 α t 0 (γ α + s) -α ϕ(s) ∇f (X s ), Σ(X s ) 1/2 dB s + (γ α /2) t 0 (γ α + s) -2α ϕ(s) ∇ 2 f (X s ), Σ(X s ) ds . Using A4.1 and that for any a, b ≥ 0, (a + b) 2 ≤ 2(a 2 + b 2 ) we have for any x, y ∈ R d , |f (x)-g(y)| ≤ |f (0)|+ ∇f (0) x +(L/2) x 2 +|g(y)| , ∇f (x) 2 ≤ 2 ∇f (0) 2 +2L 2 x 2 .
Therefore, using this result Lemma 4.7, Cauchy-Schwarz's inequality and that E[g(Y ) 2 ] < +∞, we obtain that for any t ≥ 0 there exists Ā ≥ 0 such that sup

s∈[0,t] E [|f (X s ) -g(Y )|] ≤ Ā , sup s∈[0,t] E ∇f (X s ) 2 ≤ Ā .
Combining this result, Lemma 4.12, the fact that ( t 0 ϕ(s) ∇f (X s ), Σ(X s ) 1/2 dB s ) t≥0 is a martingale and Fubini-Lebesgue's theorem we obtain that for any t ≥ 0

E [F y (t, X t )] = E [E [F Y (t, X t )|F 0 ]] = E [ϕ(0)(f (X 0 ) -g(Y ))] + t 0 ϕ (s)E [(f (X s ) -g(Y ))] ds - t 0 (γ α + s) -α ϕ(s)E ∇f (X s ) 2 ds + (γ α /2) t 0 (γ α + s) -2α ϕ(s)E ∇ 2 f (X s ), Σ(X s ) ds . (c) Let y ∈ R d and F y : [0, +∞) × R d such that for any t ∈ [0, +∞), x, y ∈ R d , F y (t, x) = ϕ(t) x -y 2p .
Using that (X t ) t≥0 is a continuous semi-martingale, the fact that F y ∈ C 1,2 ([0, +∞) , R d ) and Itô's lemma (Karatzas and Shreve, 1991, Chapter 3, Theorem 3.6) we obtain that for any t ≥ 0 almost surely

F y (t, X t ) = F y (0, X 0 ) + t 0 ∂ 1 F y (s, X s )ds + t 0 ∂ 2 F y (s, X s ), dX s + (1/2) t 0 ∂ 2,2 F y (s, X s ), d X s = F y (0, X 0 ) + t 0 ϕ (s) X s -y 2p ds + t 0 ∂ 2 F y (s, X s ), dX s + (1/2) t 0 ∂ 2,2 F y (s, X s ), d X s = F y (0, X 0 ) + t 0 ϕ (s) X s -y 2p ds -2p t 0 (γ α + s) -α ϕ(s) ∇f (X s ), X s -y X s ) -y 2(p-1) ds + 2pγ 1/2 α t 0 (γ α + s) -α ϕ(s) X s -y, Σ(X s ) 1/2 X s -y 2(p-1) dB s + pγ α t 0 (γ α + s) -2α ϕ(s) Tr(Σ(X s )) X s -y 2(p-1) ds + 2p(p -1) t 0 (γ α + s) -2α ϕ(s) (X s -y)∇(X s -y) , Σ(X s ) X s -y 2(p-2) ds .
Using A4.1 and that for any a, b ≥ 0, (a+b) 2 ≤ 2(a 2 +b 2 ) we have for any x, y ∈ R d , Therefore, using this result Lemma 4.7, Cauchy-Schwarz's inequality and that E[ Y 2 ] < +∞, we obtain that for any t ≥ 0 there exists Ā ≥ 0 such that sup

s∈[0,t] E X s -Y 2p ≤ Ā , sup s∈[0,t] E | ∇f (X s ), X s -Y X s -Y 2(p-1) | ≤ Ā .
Combining this result, Lemma 4.12, the fact that (

t 0 ϕ(s) ∇f (X s ), Σ(X s ) 1/2 (f (X s )-g(Y )
) p-1 dB s ) t≥0 is a martingale and Fubini-Lebesgue's theorem we obtain that for any t ≥ 0

E [F y (t, X t )] = E [E [F Y (t, X t )|F 0 ]] = E ϕ(0) X 0 -Y 2p + t 0 ϕ (s)E X s -Y 2p ds -2p t 0 (γ α + s) -α ϕ(s)E ∇f (X s ), X s -y X s ) -y 2(p-1) ds + γ α p t 0 (γ α + s) -2α ϕ(s)E Tr(Σ(X s )) X s -y 2(p-1) ds + 2γ α p(p -1) t 0 (γ α + s) -2α ϕ(s)E (X s -y)∇(X s -y) , Σ(X s ) X s -y 2(p-2) ds . (d) Let y ∈ R d and F : [0, +∞) × R d such that for any t ∈ [0, +∞), x, y ∈ R d , F y (t, x) = ϕ(t)(f (x) -g(y)) 2p .
Using that (X t ) t≥0 is a continuous semi-martingale, the fact that F ∈ C 1,2 ([0, +∞) , R d ) and Itô's lemma (Karatzas and Shreve, 1991, Chapter 3, Theorem 3.6) we obtain that for any t ≥ 0 almost surely

F y (t, X t ) = F y (0, X 0 ) + t 0 ∂ 1 F y (s, X s )ds + t 0 ∂ 2 F y (s, X s ), dX s + (1/2) t 0 ∂ 2,2 F y (s, X s ), d X s = F y (0, X 0 ) + t 0 ϕ (s)(f (X s ) -g(y)) 2p ds + t 0 ∂ 2 F y (s, X s ), dX s + (1/2) t 0 ∂ 2,2 F y (s, X s ), d X s = F y (0, X 0 ) + t 0 ϕ (s)(f (X s ) -g(y)) 2p ds -2p t 0 (γ α + s) -α ϕ(s) ∇f (X s ) 2 (f (X s ) -g(y)) 2(p-1) ds + 2pγ 1/2 α t 0 (γ α + s) -α ϕ(s) ∇f (X s ), Σ(X s ) 1/2 (f (X s ) -g(y)) 2(p-1) dB s + pγ α t 0 (γ α + s) -2α ϕ(s) ∇ 2 f (X s ), Σ(X s ) (f (X s ) -g(y)) 2(p-1) ds + 2p(p -1) t 0 (γ α + s) -α ϕ(s) ∇f (X s )∇f (X s ) , Σ(X s ) (f (X s ) -g(y)) 2(p-2) ds
Using A4.1 and that for any a, b ≥ 0, (a + b) 2 ≤ 2(a 2 + b 2 ) we have for any

x, y ∈ R d , |f (x) -g(y)| 2p ≤ 4 2p-1 |f (0)| 2p + 4 2p-1 ∇f (0) 2p x 2p + (4 2p-1 L/2) x 4p + 4 2p-1 |g(y)| 2p , ∇f (x) 2 ≤ 2 ∇f (0) 2 + 2L 2 x 2 .
Therefore, using this result Lemma 4.7, Lemma 4.12, Hölder's inequality and that E[g(Y ) 2 ] < +∞, we obtain that for any t ≥ 0 there exists Ā ≥ 0 such that sup

s∈[0,t] E |f (X s ) -g(Y )| 2p ≤ Ā , sup s∈[0,t] E ∇f (X s ) 2 |f (X s ) -g(Y )| 2(p-1) ≤ Ā , sup s∈[0,t] E | ∇f (X s )∇f (X s ) , Σ(X s ) (f (X s ) -g(Y )) 2(p-2) | ≤ Ā .
Combining this result, Lemma 4.12, the fact that (

t 0 ϕ(s) ∇f (X s ), Σ(X s ) 1/2 (f (X s )-g(Y )
) p-1 dB s ) t≥0 is a martingale and Fubini-Lebesgue's theorem we obtain that for any t ≥ 0

E [F y (t, X t )] = E [E [F Y (t, X t )|F 0 ]] = E ϕ(0)(f (X 0 ) -g(Y )) 2p + t 0 ϕ (s)E (f (X s ) -g(Y )) 2p ds -2p t 0 (γ α + s) -α ϕ(s)E ∇f (X s ) 2 (f (X s ) -g(y)) 2(p-1) ds + γ α p t 0 (γ α + s) -2α ϕ(s)E ∇ 2 f (X s ), Σ(X s ) (f (X s ) -g(Y )) 2(p-1) ds + 2γ α p(p -1) t 0 (γ α + s) -2α ϕ(s)E ∇f (X s )∇f (X s ) , Σ(X s ) (f (X s ) -g(Y )) 2(p-2) ds .
The following lemma is a useful tool that converts results on C 2 functions to smooth functions.

Lemma 4.14. Assume A4.1, A4.5, A4.3 and that arg min x∈R d f is bounded. Then there exists (f ε ) ε>0 such that for any ε > 0, f ε is convex, C 2 with L-Lipschitz continuous gradient. In addition, there exists C ≥ 0 such that the following properties are satisfied.

(a) For all ε > 0, f ε admits a minimize x ε and lim sup

ε→0 f ε (x ε ) ≤ f (x ). (b) lim inf ε→0 x ε ≤ C. (c) for any T ≥ 0, lim ε→0 E [|f ε (X T,ε ) -f (X T )|] = 0 , where (X t,ε ) t≥0 is the solution of (4.2) replacing f by f ε . Proof. Let ϕ ∈ C ∞ c (R d , R + ) be an even compactly-supported function such that R d ϕ(z)dz = 1. For any ε > 0 and x ∈ R d , let ϕ ε (x) = ε -d ϕ(x/ε) and f ε = ϕ ε * f . Since ϕ ∈ C ∞ c (R d , R + ) and is compactly-supported, we have f ε ∈ C ∞ (R d , R).
In addition, we have for any ε > 0, (∇f ) ε = ∇f ε .

First, we show that for any ε, f is convex and ∇f ε is L-Lipschitz continuous. Let ε > 0, x, y ∈ R d and t ∈ [0, 1]. Using A4.5 we have

f ε (tx + (1 -t)y) = R d f (tx + (1 -t)y -z)ϕ ε (z) dz ≤ R d {tf (x -z) + (1 -t)f (y -z)} ϕ ε (z) dz ≤ tf ε (x) + (1 -t)f ε (y) .
Hence, f ε is convex. In addition, using A4.1 and that

R d ϕ ε (z)dz = 1 we have ∇f ε (x) -∇f ε (y) ≤ R d ∇f (x -z) -∇f (y -z) ϕ ε (z) dz ≤ L x -y , which proves that ∇f ε is L-Lipschitz continuous.
Second we show that f ε and ∇f ε converge uniformly towards f and ∇f . Let ε > 0, x ∈ R d . Using the convexity of f and that ϕ ε is even, we get

f ε (x) -f (x) = R d (f (x -z) -f (x))ϕ ε (z) dz (4.45) ≥ - R d ∇f (x), z ϕ ε (z) dz ≥ -∇f (x), R d zϕ ε (z) dz ≥ 0 ,
Conversely, using the descent lemma (Nesterov, 2004, Lemma 1.2.3) and that ϕ ε is even, we have

f ε (x) -f (x) = R d (f (x -z) -f (x))ϕ ε (z) dz (4.46) ≤ R d -∇f (x), z + (L/2) z 2 ϕ ε (z) dz ≤ (L/2) R d ε 2 z/ε 2 ε -d ϕ(z/ε) dz ≤ (L/2)ε 2 R d u 2 ϕ(u) du .
Combining (4.45) and (4.46) we get that lim ε→0 f -f ε ∞ = 0. Using A4.1 we have for any

x ∈ R d ∇f ε (x) -∇f (x) ≤ (∇f ) ε (x) -∇f (x) ≤ R d ∇f (x-z)-∇f (x) ϕ ε (z)dz ≤ Lε R d z ϕ(z)dz ,
Hence, we obtain that lim ε→0 ∇f ε -∇f ∞ = 0. Finally, since f is coercive (Bertsekas, 1997, Proposition B.9) and (f ε ) ε>0 converges uniformly towards f we have that for any ε > 0, f ε is coercive. We divide the rest of the proof into three parts.

(a) Let ε > 0. Since f ε is coercive and continuous it admits a minimizer x ε . In addition, we have

f ε (x ε ) ≤ f ε (x ) ≤ f (x ) + f ε -f ∞ . (4.47) Therefore, lim sup ε→0 f ε (x ε ) ≤ f (x ). (b) Let ε ∈ (0, 1]. Using (4.47), we obtain that |f ε (x )| ≤ |f (x )| + sup ε∈(0,1] f ε -f ∞ .
Since f is coercive, we obtain that (x ε ) ε∈(0,1] is bounded and therefore there exists C ≥ 0 such that lim inf ε→0 x ε ≤ C.

(c) Let ε > 0, T ≥ 0 and (X t,ε ) t≥0 be the solution of (4.2) replacing f by f ε . Using (4.2), the fact that lim ε→0 ∇f -∇f ε ∞ = 0, A4.1 and Grönwall's inequality (Pachpatte, 1998, Theorem 1.2.2) we have

E X T,ε -X T 2 ≤ E   T 0 (γ α + s) -α {-∇f ε (X t,ε ) + ∇f (X t )} dt 2   (4.48) ≤ 2γ -2α α T T 0 E ∇f (X t,ε ) -∇f (X t ) 2 dt + 2γ -2α α T 2 ∇f -∇f ε 2 ∞ ≤ 2Lγ -2α α T T 0 E X t,ε -X t 2 dt + 2γ -2α α T 2 ∇f -∇f ε 2 ∞ ≤ 2γ -2α α T 2 ∇f -∇f ε 2 ∞ exp 2Lγ -2α α T 2 .
185 Therefore lim ε→0 E X T,ε -X T 2 = 0. In addition, using the Cauchy-Schwarz inequality, A4.1 and Lemma 4.7 we have

E [|f (X T,ε ) -f (X T )|] ≤ E 1 0 ∇f (X T + t(X T,ε -X T )) X T,ε -X T dt (4.49) ≤ E [( X T,ε + X T + x ) X T,ε -X T ] ≤ 3 1/2 x 2 + E X T 2 + E X T,ε 2 1/2 E X T,ε -X T 2 1/2 ≤ 3 1/2 ( x + 2A T,1 ) 1/2 (1 + x 0 2 ) 1/2 E X T,ε -X T 2 1/2 .
Therefore, using (4.48), (4.49) and the fact that lim ε→0 f -f ε ∞ = 0 we obtain that

lim ε→0 E [|f ε (X T,ε ) -f (X T )|] ≤ lim ε→0 E [|f (X T,ε ) -f (X T )|] + lim ε→0 f -f ε ∞ = 0 , which concludes the proof. Lemma 4.15. Let x, y ≥ 1. Let α ∈ (0, 1/2]. If y < x then x α -y α ≤ x 1-α -y 1-α . Proof. Let λ ∈ (0, 1) such that y = λx. Then x α -y α = x α (1 -λ α ) ≤ x 1-α (1 -λ 1-α ) = x 1-α -y 1-α because x > 1, λ < 1 and α ≤ 1 -α.
The following property is a well-known property of functions with Lipschitz gradient but is recalled here for completeness.

∈ R d , ∇f (x) 2 ≤ 2L(f (x) -f ). Proof. Using A4.1 and that f = min R d f , we have for any x ∈ R d f -f (x) ≤ f (x -∇f (x)/L) -f (x) ≤ -∇f (x) 2 /L + ∇f (x) 2 /(2L) ≤ -∇f (x) 2 /(2L) ,
which concludes the proof.

4.C Analysis of SGD in the convex case

4.C.1 Proof of Theorem 4.4

In this section we prove Theorem 4.4. We begin with a lemma to bound E X t -x 2 .

Lemma 4.17. Assume A4.1, A4.2, A4.3 and A4.5. Let (X t ) t≥0 be given by (4.2). Then, for any α, γ ∈ (0, 1), there exists

C (c) 1,α ≥ 0 and C (c) 2,α ≥ 0 and a function Φ (c) α : R + → R + such that, for any t ≥ 0, E X t -x 2 ≤ C (c) 1,α Φ (c) α (t + γ α ) + C (c) 2,α .
And we have

Φ (c) α (t) =        t 1-2α if α < 1/2 , log(t) if α = 1/2 , 0 if α > 1/2 .
The values of the constants are given by

C (c) 1,α =        γ α η(1 -2α) -1 if α < 1/2 , γ α η if α = 1/2 , 0 if α > 1/2 . C (c) 2,α =        X 0 -x 2 if α < 1/2 , X 0 -x 2 -γ α η log(γ α ) if α = 1/2 , X 0 -x 2 + (2α -1) -1 γ 2-2α α η if α > 1/2 , Proof.
Let α, γ ∈ (0, 1) and t ≥ 0. Let (X t ) t≥0 be given by (4.2). We consider the function

F : R × R d → R + defined as follows ∀(t, x) ∈ R × R d , F (t, x) = x -x 2 .
Applying Lemma 4.13 to the stochastic process (F (t, X t )) t≥0 and using A4.5 and A4.2 gives that for all t ≥ 0,

E X t -x 2 -E X 0 -x 2 = -2 T 0 (t + γ α ) -α E [ X t -x , ∇f (X t ) ] dt + T 0 γ α (t + γ α ) -2α E [Tr(Σ(X t ))] dt ≤ γ α η T 0 (t + γ α ) -2α dt .
We now distinguish three cases:

(a) Case where α < 1/2: In that case we have:

E X t -x 2 ≤ X 0 -x 2 + γ α η(1 -2α) -1 ((T + γ α ) 1-2α -γ 1-2α α ) ≤ X 0 -x 2 + γ α η(1 -2α) -1 (T + γ α ) 1-2α .
(b) Case where α = 1/2: In that case we obtain:

E X t -x 2 ≤ X 0 -x 2 + γ α η(log(T + γ α ) -log(γ α ))
≤ γ α η log(T + γ α ) + X 0 -x 2 -γ α η log(γ α ) .

(c) Case where α > 1/2: In that case we have:

E X t -x 2 ≤ X 0 -x 2 + γ α η(1 -2α) -1 ((T + γ α ) 1-2α -γ 1-2α α ) ≤ X 0 -x 2 + (2α -1) -1 γ 2-2α α η .
We now turn to the proof of Theorem 4.4.

Proof of Theorem 4.4. Let f ∈ C 2 (R d , R). Let γ ∈ (0, 1) and α ∈ (0, 1] and T ≥ 1. Let (X t ) t≥0 be given by (4.2). Let S : [0, T ] → [0, +∞) defined by

     S(t) = t -1 T T -t {E [f (X s )] -f } ds , if t > 0 , S(0) = E [f (X T )] .
With this notation we have E [f (X T )] -f = S(0) -S(1) + S(1) -S(T ) + S(T ) -f .

We preface the rest of the proof with the following computation. For any y 0 ∈ R d we define the function F y0 : R + × R d → R by F y0 (t, x) = (t + γ α ) α x -y 0 2 .

187

In the following we will choose either y 0 = x or y 0 = X s for s ∈ [0, T ]. Using Lemma 4.17, that Φ (c) α is non-decreasing and that for any a, b ≥ 0, (a + b) 2 ≤ 2(a 2 + b 2 ), we have

E X t -y 0 2 = E (X t -x ) + (x -y 0 ) 2 ≤ 2E X t -x 2 + 2E y 0 -x 2 ≤ 2C (c)
1,α Φ (c) α (t + γ α ) + 4C 1,α = 0 if α > 1/2. Notice that the additional log(T + γ α ) term is only needed in the case where α = 1/2. For any (t, x) ∈ R + × R d , we have

∂ t F y0 (t, x) = α(t+γ α ) α-1 x -y 0 2 , ∂ x F y0 (t, x) = 2(t+γ α ) α (x-y 0 ) , ∂ xx F y0 (t, x) = 2(t+γ α ) α .
Using Lemma 4.13 on the stochastic process (F y0 (t, X t )) t≥0 , we have that for any u ∈ [0, T ] And in the case where α > 1/2, for all u ∈ [0, T ]:

E [F y0 (T, X T )] -E [F y0 (T -u, X T -u )] = T T -u α(t + γ α ) α-1 E X t -
(t + γ α ) α-1 dt -2 T T -u E [f (X t ) -f (y 0 )] dt ≤ C (c) 3,α ((T + γ α ) α -(T -u + γ α ) α ) -2 T T -u E [f (X t ) -f (y 0 )] dt + (γ α η + 2αC (c) 1,α )(1 -α) -1 (T + γ α ) 1-α -(T -u + γ α ) 1-α log(T + γ α ) + 2C
(T + γ α ) 1-α -(T -u + γ α ) 1-α ≤ ((T + γ α ) α -(T -u + γ α ) α ) ,
and we also have, for all u ∈ [0, T ]:

(T + γ α ) α -(T + γ α -u) α = ((T + γ α ) α -(T + γ α -u) α )((T + γ α ) 1-α + (T + γ α -u) 1-α ) (T + γ α ) 1-α + (T + γ α -u) 1-α ≤ (T + γ α ) -(T + γ α -u) + (T + γ α ) α (T + γ α -u) 1-α -(T + γ α ) 1-α (T + γ α -u) α (T + γ α ) 1-α ≤ 2u/(T + γ α ) 1-α .
Now, plugging y 0 = X T -u in (4.52) we obtain, for all u ∈ [0, T ]:

E T T -u f (X t ) -f (X T -u ) dt ≤ 2C 1 log(T + γ α )(T + γ α ) -min(α,1-α) u . (4.54)
Since S is a differentiable function and using (4.54), we have for all u ∈ (0, T ),

S (u) = -u -2 T T -u E [f (X t )] dt + u -1 E [f (X T -u )] = -u -1 (S(u) -E [f (X T -u )]
) . (4.55)

This last result implies -S (u) ≤ 2C 1 log(T + γ α )/(T + γ α ) -min(α,1-α) u -1 and integrating we get S(1) -S(T ) ≤ 2C 1 log(T + γ α ) log(T )(T + γ α ) -min(α,1-α) .

(b) Bounding S(T ) -f : Using (4.51), with u = T and y 0 = x , and X 0 -x ≤ C 1 we obtain

T 0 E [f (X s )] ds -T f ≤ (C 1 /2) (T + γ α ) α -γ α α + (T + γ α ) 1-α -γ log(T + γ α ) + (1/2)γ α E X 0 -x 2 .
(4.56)

Using this result we have

S(T ) -f ≤ T -1 C 1 (T + γ α ) max(1-α,α) log(T + γ α ) + C 1 γ α T -1 /2 ≤ 2C 1 T -min(α,1-α) log(T + γ α ) .

4.C.2 Proof of Theorem 4.6

In this section we prove Theorem 4.6. The proof is clearly more involved than the one of Theorem 4.4. We will follow a similar way as in the proof of Theorem 4.4, with more technicalities. One of the main argument of the proof is the suffix averaging technique that was introduced in [START_REF] Shamir | Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes[END_REF].

We begin with the discrete counterpart of Lemma 4.17.

Lemma 4.18. Assume A4.1, A4.2 and A4.5. Let α, γ ∈ (0, 1) and let (X n ) n≥0 be given by (4.1). Then there exists C

Φ (d) α (t) =        t 1-2α if α < 1/2 , log(t) if α = 1/2 , 0 if α > 1/2 .
The values of the constants are given by

C (d) 1,α =        2γ 2 η(1 -2α) -1 if α < 1/2 , γ 2 η if α = 1/2 , 0 if α > 1/2 . C (d) 2,α =          2 max k≤(γL/2) 1/α E X k -x 2 if α < 1/2 , 2 max k≤(γL/2) 1/α E X k -x 2 + 2γ 2 η if α = 1/2 , 2 max k≤(γL/2) 1/α E X k -x 2 + γ 2 η(2α -1) -1 if α > 1/2 ,
Proof. Let f : R d → R verifying assumptions A4.1 and A4.5. We consider (X n ) n≥0 satisfying (4.1). Let x ∈ R d be given by A4.5. We have, using (4.1) and A4.2 that for all n ≥ (γL/2) 1/α ,

E X n+1 -x 2 F n = E X n -x -γ(n + 1) -α H(X n , Z n+1 ) 2 F n (4.59) = X n -x 2 -2γ/(n + 1) α X n -x , E [H(X n , Z n+1 )|F n ] + γ 2 (n + 1) -2α E H(X n , Z n+1 ) 2 F n = X n -x 2 -2γ/(n + 1) α X n -x , ∇f (X n ) 190 + γ 2 (n + 1) -2α E H(X n , Z n+1 ) -∇f (X n ) + ∇f (X n ) 2 F n = X n -x 2 -2γ/(n + 1) α X n -x , ∇f (X n ) + γ 2 (n + 1) -2α E H(X n , Z n+1 ) -∇f (X n ) 2 F n + γ 2 (n + 1) -2α E ∇f (X n ) 2 F n + 2E [ H(X n , Z n+1 ) -∇f (X n ), ∇f (X n ) |F n ] = X n -x 2 -2γ/(n + 1) α X n -x , ∇f (X n ) + γ 2 η(n + 1) -2α + γ 2 (n + 1) -2α ∇f (X n ) 2 ≤ X n -x 2 -2γ/L(n + 1) -α ∇f (X n ) 2 + γ 2 η(n + 1) -2α + γ 2 (n + 1) -2α ∇f (X n ) 2 ≤ X n -x 2 + γ/(n + 1) α ∇f (X n ) 2 [γ/(n + 1) α -2/L] + γ 2 η(n + 1) -2α ≤ X n -x 2 + γ 2 η(n + 1) -2α E X n+1 -x 2 ≤ E X n -x 2 + γ 2 η(n + 1) -2α ,
where we used the co-coercivity of f . Summing the previous inequality leads to

E X n -x 2 -E X 0 -x 2 ≤ γ 2 η n k=1 k -2α .
As in the previous proof we now distinguish three cases:

(a) Case where α < 1/2: In that case we have:

E X n -x 2 ≤ X 0 -x 2 + γ 2 η(1 -2α) -1 (n + 1) 1-2α ≤ X 0 -x 2 + 2γ 2 η(1 -2α) -1 n 1-2α .
(b) Case where α = 1/2: In that case we obtain:

E X n -x 2 ≤ X 0 -x 2 + γ 2 η(log(n) + 2) .
(c) Case where α > 1/2: In that case we have:

E X n -x 2 ≤ X 0 -x 2 + γ 2 η(2α -1) -1 .
We now turn to the proof of Theorem 4.6 by stating an intermediate result where we assume a condition bounding E ∇f (X n ) 2 . This Proposition provides non-optimal convergence rates for SGD but will be used as a central tool to improve them and obtain optimal convergence rates. Proposition 4.11. Let γ, α ∈ (0, 1) and x 0 ∈ R d and (X n ) n≥0 be given by (4.1). Assume A4.1, A4.2 and A4.5. Suppose additionally that there exists α ∈ [0, 1/2], β > 0 and C 0 ≥ 0 such that for all n ∈ {0,

• • • , N } E ∇f (X n ) 2 ≤ C 0 (n + 1) β log(n + 1) if α ≤ α , C 0 if α > α .
(4.60)

Then there exists Cα ≥ 0 such that, for all N ≥ 1, E [f (X N )] -f ≤ Cα (1 + log(N + 1)) 2 /(N + 1) min(α,1-α) Ψ α (N + 1) + 1/(N + 1) , with

Ψ α (n) = n β (1 + log(n)) if α ≤ α , 1 if α > α .
Proof. Let α, γ ∈ (0, 1) and N ≥ 1. Let (X n ) n≥0 be given by (4.1). Let (S k ) k∈{0,••• ,N } defined by

∀k ∈ {0, • • • , N } , S k = (k + 1) -1 N t=N -k E [f (X t )] .
With this notation we have E [f (X N )] -f = (S 0 -S N ) + (S N -f ). As in the proof of Theorem 4.4, we preface the proof with the following computation. Let ∈ {0, • • • , N }, let k ≥ , let y 0 ∈ F . Using A4.5 we have + (2γ) -1 (N -u + 1) α E X N -u -y 0 2 .

E X k+1 -y 0 2 F k = E X k -y 0 -γ(k + 1) -α H(X k , Z k+1 ) 2 F k (4.61) = X k -y 0 2 + γ 2 (k + 1) -2α E H(X k , Z k+1 ) 2 F k -2γ(k + 1) -α X k -y 0 , ∇f (X k ) E [f (X k ) -f (y 0 )] ≤ (2γ) -1 (k + 1) α E X k -y 0 2 -E X k+1 -y 0 2 + (γ/2)(k + 1) -α E E H(X k , Z k+1 ) 2 F k E [f (X k ) -f (y 0 )] ≤ (2γ) -1 (k + 1) α E X k -y 0 2 -E X k+1 -y 0 2 + (γ/2)(k + 1) -α η + E ∇f (X k )
In the following we will take for y 0 either x or X m for m ∈ [0, N ]. We now have to run separate analyses depending on the value of α.

(a) Case α ≤ α : In that case (4.60) gives that E ∇f (X k ) 2 ≤ C 0 (N + 1) β log(N + 1), and Lemma 4.18 gives that for all k ∈ {0, . . . , N },

E X k -y 0 2 ≤ 2E X k -x 2 + 2E y 0 -x 2 ≤ 2C (d)
1,α (k + 1) 1-2α log(k + 1) + 2C (b) Case α ∈ (α , 1/2]: In that case Lemma 4.18 gives that for all k ∈ {0, . . . , N },

E X k -y 0 2 ≤ 2E X k -x 2 + 2E y 0 -x 2 ≤ 2C (d)
1,α (k + 1) 1-2α log(k + 1) + 2C ≤ 2(u + 1)/(N + 1) 1-α .

Finally, putting the three cases above together we obtain d) (u + 1)/(N + 1) min(α,1-α) (1 + log(N + 1))Ψ α (N + 1) (4.63)

E N k=N -u f (X k ) -f (X N -u ) ≤ 2C (
+ (2γ) -1 (N -u + 1) α E X N -u -y 0 2 , with Ψ α (n) = n β (1 + log(n)) if α ≤ α , 1 if α > α .
Note that the additional log(N + 1) factor can be removed if α = 1/2. We bound now the quantities (S 0 -S N ) and (S N -f ). We have now, using (4.64),

uS u-1 = (u + 1)S u -E [f (X N -u )] (4.65) = uS u + S u -E [f (X N -u )]
≤ uS u + 2C (d) (N + 1) -min(α,1-α) (1 + log(N + 1))Ψ α (N + 1) S u-1 -S u ≤ 2C (d) u -1 (N + 1) -min(α,1-α) log(N + 1) S 0 -S N ≤ 2C (d) (N + 1) -min(α,1-α) (1 + log(N + 1))Ψ α (N + 1) N u=1

(1/u) S 0 -S N ≤ 2C (d) (N + 1) -min(α,1-α) (1 + log(N + 1)) 2 Ψ α (N + 1) . f (X k ) -f (x ) ≤ 2C (d) (1 + log(N + 1))(N + 1) -min(α,1-α) Ψ α (N + 1) (4.66)

194 + (2γ) -1 (N + 1) -1 X 0 -x 2 S N -f ≤ 2C (d) (1 + log(N + 1)) 2 (N + 1) -min(α,1-α) Ψ α (N + 1) + (2γ) -1 (N + 1) -1 X 0 -x 2 .

And finally, choosing Cα 2 max((2γ) -1 X 0 -x 2 , 2C (d) ) and putting Equations (4.65) and (4.66) together gives E [f (X N )] -f ≤ Cα (1 + log(N + 1)) 2 /(N + 1) min(α,1-α) Ψ α (N + 1) + 1/(N + 1) .

We can finally conclude the proof of Theorem 4.6.

Proof of Theorem 4.6. We begin by proving by induction over m ∈ N * the following statement H m :

For any α > 1/(m+1), there exists C + α > 0 such that for all n ∈ {0, . . . , N } , E ∇f (X n ) 2 ≤ C + α , and for any α ≤ 1/(m+1), there exists C - α > 0 such that for all n ∈ {0, . . . , N } , E ∇f (X n ) 2 ≤ C - α n 1-(m+1)α (1 + log(n)) 3 . For m = 1, H 1 is an immediate consequence of A4.1 and Lemma 4.18, with

C + α = L 2 C (d)
2,α and

C - α = L 2 max(C (d) 1,α , C (d)
2,α ). Now, let m ∈ N * and suppose that H m holds. Let α ∈ (0, 1). Setting α = 1/(m + 1) we see that (4.60) is verified with β = 1 -(m + 1)α.

Consequently, using A4.1, A4.5, A4.2 we can apply Proposition 4.11 which shows that, for α ≤ 1/(m + 1): E [f (X N )] -f ≤ Cα (1 + log(N + 1)) 2 /(N + 1) min(α,1-α) Ψ α (N + 1) + 1/(N + 1) (4.67) ≤ Cα (1 + log(N + 1)) 3 (N + 1) -α (N + 1) 1-(m+1)α + 1/(N + 1) ≤ Cα (1 + log(N + 1)) 3 (N + 1) 1-(m+2)α + 1/(N + 1) .

In particular, if α > 1/(m+2) we have the existence of Cα > 0 such that for all n ∈ {0, • • • , N }, E [f (X n )] -f ≤ Cα . And using A4.1 and Lemma 4.16 we get that, for all n ∈ {0,

• • • , N } E ∇f (X n ) 2 ≤ 2LE [f (X n ) -f ] ≤ 2L Cα ,
which proves H m+1 for α > 1/(m + 2), with C + α = 2L Cα . And (4.67) proves H m+1 for α ≤ 1/(m + 2) with C - α = 2 Cα . Finally this proves that H m is true for any n ≥ 1. Now, let α ∈ (0, 1). Since R is archimedean, there exists m ∈ N * such that α > 1/(m + 1) and therefore H m shows the existence of C 0 > 0 such that E ∇f (X n )

2 ≤ C 0 for all n ∈ N * .

Applying Proposition 4.11 gives the existence of C (d) > 0 such that for all N ≥ 1 d) (1 + log(N + 1)) 2 /(N + 1) min(α,1-α) , with C (d) = 2 Cα . Choosing C = C (d) concludes the proof.

E [f (X N )] -f ≤ C (
We finally prove Corollary 4.4.

Proof of Corollary 4.4. The proof follows the same lines as the ones of Lemma 4.18 and Proposition 4.11. We show that both conclusions hold under the assumption that ∇f is bounded instead of being Lipschitz-continuous.

In order to prove that Lemma 4.18 still holds, let us do the following computation. We consider (X n ) n≥0 satisfying (4.1). We have, using (4.1), A4.5 and A4.2 that for all n ≥ 0,

E X n+1 -x 2 F n = E X n -x -γ(n + 1) -α H(X n , Z n+1 ) 2 F n = X n -x 2 -2γ/(n + 1) α X n -x , E [H(X n , Z n+1 )|F n ] + γ 2 (n + 1) -2α E H(X n , Z n+1 ) 2 F n = X n -x 2 -2γ/(n + 1) α X n -x , ∇f (X n ) + γ 2 η(n + 1) -2α + γ 2 (n + 1) -2α ∇f (X n ) 2 E X n+1 -x 2 ≤ E X n -x 2 + γ 2 (η + ∇f ∞ )(n + 1) -2α .
And we obtain the same equation as in (4.59), with a different constant before (n + 1) -2α . Hence the conclusions of Lemma 4.18 still hold, because A4.1 is never used in the rest of the proof.

We can now apply safely Proposition 4.11 (since A4.1 is only used to use Lemma 4.18) with α = 0. This concludes the proof.

4.D Analysis of SGD in the weakly quasi-convex case

In this section we give the proof of Corollary 4.5.

4.D.1 Technical lemmas

We begin with a series of technical lemmas. Proof. Since f is continuous there exists a ≥ 0 such that for any x ∈ R d , f (x) -f (x ) ≥ c x -xa. Therefore, using Jensen's inequality and that D 4 ≥ 1 we have Lemma 4.20. Assume A4.6 with r 1 = r 2 = 1. Then for any p ∈ N with p ≥ 2 and d-dimensional random variable X we have E ∇f (X) 2 (f (X) -f (x )) p-1 ≥ E [(f (X) -f (x )) p ] 1+1/p E X -x 2p -1/p , Proof. Let p ∈ N with p ≥ 2 and let = 2p/(p + 1). Using A4.6 we have for any x ∈ R d

E X -x 2p ≤ c -2p 2p k=0 k 2p E (f (X) -f (x )) k a 2p-k ≤ c -2p 2p k=0 k 2p E (f (X) -f (x )) 2p k/(2p) a 2p-k ≤ c -2p 2p
x -x ∇f (x) (f (x) -f (x )) (p-1)/2 ≥ (f (x) -f (x )) (p+1)/2 ≥ (f (x) -f (x )) p .

Let ς = 2 -1 = 1 + p -1 and κ such that ς -1 + κ -1 = 1. Using Hölder's inequality the fact that κ = 2p we have

E X -x ∇f (X) (f (X) -f (x )) (p-1)/2 ≤ E ∇f (X) 2 (f (X) -f (x )) p-1 1/ς E X -x 2p 1/κ .
Since, κ -1 = (1 + p) -1 we have

E ∇f (X) 2 (f (X) -f (x )) p-1 ≥ E [(f (X) -f (x )) p ] 1+1/p E X -x 2p -1/p
, which concludes the proof.

Lemma 4.21. Let α, γ ∈ (0, 1). Assume A4.1, A4.2, A4.3 and A4.6b holds. Then for any p ∈ N, there exists D p,4 ≥ 0 such that for any t ≥ 0 E X t -x 2p 1/p ≤ D p,4 1 + (γ α + t) 1-2α .

Proof. Let α, γ ∈ (0, 1) and p ∈ N. Let E t,p = E X t -x 2p . Using Lemma 4.12 and Lemma 4.13 we have for any t > 0 dE t,p / dt = -2p(γ α + t) -α E ∇f (X t ), X t -x X t -x 2(p-1) (4.68)

+ pγ α (γ α + t) -2α E Tr(Σ(X t )) X t -x 2(p-1)
+2(p -1)E (X t -x ) (X t -x ), Σ(X t ) X t -x 2(p-2))

≤ 2pγ α η(2p -1)(γ α + t) -2α E X t -x 2(p-1)

≤ pγ α η(2p -1)(γ α + t) -2α E t,(p-1) .

If p = 1, the proposition holds and by recursion and using (4.68) we obtain the result for p ∈ N.

4.D.2 Control of the norm in the convex case

Proposition 4.12. Let α, γ ∈ (0, 1). Let m ∈ [0, 2] and ϕ > 0 such that for any p ∈ N there exists D p,2 ≥ 0 such that for any t ≥ 0, E[ X t -x 2p ] 1/p ≤ D p,1 {1 + (γ α + t) m-ϕα }. Assume A4.1, A4.2, A4.3 and A4.6b and that there exist R ≥ 0 and c > 0 such that for any x ∈ R d , with x ≥ R, f (x) -f (x ) ≥ c x -x . Then, for any p ∈ N, there exists D p,2 ≥ 0 such that for any t ≥ 0, E X t -x 2p 1/p ≤ D p,2 {1 + (γ α + t) m-(1+ϕ)α } .

Proof. If α ≥ m/ϕ the proof is immediate since sup t≥0 {E[ X t -x 2p ] 1/p } < +∞. Now assume that α < m/ϕ. Let p ∈ N, δ p = p(1 + ϕ)α -pm and (t → E t,p ) such that for any t ≥ 0, E t,p = (f (X t ) -f (x )) 2p (γ α + t) δp . Using Lemma 4.13 we have for any t > 0 dE t,p / dt = -2p(γ α + t) -α+δp E ∇f (X t )

2 (f (X t ) -f (x )) 2p-1 (4.69)

+ pγ α (γ α + t) -2α+δp E ∇ 2 f (X t ), Σ(X t ) (f (X t ) -f (x )) p . Hence, for any t ≥ 0, E (f (X t ) -f (x )) 2p ≤ D (a) p (1 + (γ α + t) pm-p(1+ϕ)α ) . Using Lemma 4.19, there exists D 5 ≥ 0 such that E X t -x 2p ≤ D 5 (1 + (γ α + t) pm-p(1+ϕ)α ) , which concludes the proof upon using that for any a, b ≥ 0, (a + b) 1/2 ≤ a 1/2 + b 1/2 .

The following corollary is of independent interest.

Corollary 4.7. Let α, γ ∈ (0, 1). Assume A4.1, A4.2, A4.3 and A4.5 and that arg min R d f is bounded. Then, for any p ≥ 0 and t ≥ 0, E [ X t -x p ] < +∞ .

Proof. Without loss of generality we assume that x = 0 and f (x ) = 0. First, since arg min R d f is bounded, there exists R ≥ 0 such that for any x ∈ R d with x ≥ R, f (x) > 0.

Let S = {x ∈ R d , x = 1} and consider m : S → (0, +∞) such that for any θ ∈ S, m(θ) = f ( Rθ). m is continuous since f is convex and therefore it attains its minimum and there exists m > 0 such that for any θ ∈ S, m(θ) ≥ m . Let x ∈ R d with x ≥ 2 R. Since

f x : [0, +∞) → R such that f x (t) = f (tx) is convex we have (f (x) -f ( Rx/ x ))( x -R) -1 ≥ (f ( Rx/ x )) R-1 ≥ m R-1 .
Therefore, there exists c > 0 and R ≥ 0 such that for any x ∈ R d with x ≥ R, f (x) ≥ c x . Let p ∈ N. Noticing that A4.5 implies that A4.6b holds we can apply Lemma 4.21 and Proposition 4.12 with m = 1 and ϕ = 2. Applying repeatedly Proposition 4.12 we obtain that there exists D p ≥ 0 such that

E X t -x

4.D.3 Proof of Corollary 4.5

Proof of Corollary 4.5. Let α, γ ∈ (0, 1) and X 0 ∈ R d . Using Lemma 4.13, we have for any t ≥ 0 We divide the proof into three parts.

E X t -x 2 = X 0 -x 2 -(γ α + s) -α f (X s ),
(a) First, assume that A4.6b holds. Combining this result and (4.71), we get that for any t ≥ 0, E t ≤ γ α Lη 2 d(γ α + t) -2α . Therefore, there exist β, ε ≥ 0 and C β,ε ≥ 0 such that E[ X t -x 2 ] < C β,ε (γ α + t) -β (1 + log(1 + γ -1 α t)) ε with β = 0 and ε = 0 if α > 1/2, β = 1 -2α and ε = 0 if α < 1/2 and β = 0 and ε = 1 if α = 1/2. Combining this result and Theorem 4.7 concludes the proof.

(b) We can apply Lemma 4.21 and Proposition 4.12 with m = 1 and ϕ = 2. Applying repeatedly Proposition 4.12 we obtain that there exists D p ≥ 0 such that

E X t -x 2p 1/p ≤ D p {1+(γ α +t) m-α -1 α } ≤ D p {1+(γ α +t) m-m/α α } ≤ D p {1+γ m-m/α α α } ,
which concludes the proof.

(c) Finally, assume that there exists R ≥ 0 such that for any x ∈ R d with x ≥ R, ∇f (x), x -x ≥ m x -x 2 . Therefore, since (x → ∇f (x)) is continuous, there exists a ≥ 0 such that for any x ∈ R d , ∇f (x), x -x ≥ m x -x 2a. Combining this result and (4.71), we get that for any t ≥ 0, E t ≤ -m(γ α + t) -α E t + (γ α + t) -α a + γ α Lη(γ α + t) -2α

Hence, if E t ≥ max(a/m, Lη) we have that E t ≤ 0 and for any t ≥ 0, E t ≤ max(a/m, Lη, E 0 ) and is bounded. Therefore, there exist β, ε ≥ 0 and C β,ε ≥ 0 such that E[ X t -x 2 ] < C β,ε (γ α + t) -β (1 + log(1 + γ -1 α t)) ε with β = ε = 0, which concludes the proof.

199

Conclusion

As we have seen in this thesis, stochastic optimization techniques are central in statistical learning and machine learning. Moreover they are very diverse because of the large amount of problems and settings that can be encountered in machine learning. In the first part of this thesis we have focused our study on sequential learning and we have exhibited links between sequential learning and stochastic optimization, and particularly for convex functions. The situations that we analyzed in the previous chapters of this manuscript, though relatively different, were all instances of the classical trade-off between exploration and exploitation that arises often in sequential or active learning. In these chapters we demonstrated how stochastic convex optimization algorithms could help to solve these problems. Thus in Chapter 1 we explored the problem of stochastic contextual bandits in the case where regularization has been added to the loss function. This study was motivated by situations where the decision maker did not want to deviate too much from an existing policy by using bandits techniques. We adopted a strategy which consisted in partitioning the context space into bins on which separate convex optimization problems had to be solved. We constructed then a piecewise constant solution using an algorithm mixing convex optimization and UCB techniques. Using regularity assumptions on the reward functions we were able to obtain fast convergence rates for this problem, that coincide with classical nonparametric regression rates. We further discarded the dependency in the problem parameters by adding a margin condition on the regularization term and obtained convergence rates interpolating between slow and fast rates for this problem.

The results we obtained show that it is possible to implement a contextual bandit strategy without diverging too much from an existing policy. This is an incentive for decision makers to try and adopt bandit algorithms by continuously decreasing the weight of the regularization term. A possible extension of this line of research would be to consider the adversarial setting and not only the stochastic one.

In Chapter 2 we considered an active learning problem where the goal was to perform linear regression in an online manner, or equivalently to solve online A-optimal design. It consisted in choosing actively which experiment to perform while the variance of the experiments were unknown. The goal was therefore to be able to estimate those variances while minimizing the estimation error on the parameter to measure. Once again the exploration/exploitation trade-off appeared and we used a similar idea as in the previous chapter to deal with it. With a stochastic convex optimization algorithm using confidence estimates on the variances of the different covariates we obtained optimal convergence rates in the particular setting where the number of points is equal to the dimension and probably suboptimal convergence rates in the general case. Despite the fact that the results we obtained are probably not optimal in the general case our algorithm has still good experimental results and can be used as a first brick toward finding a minimax optimal algorithm for this problem. We do not know yet if the lower bound we provided can be reached by an algorithm i.e., whether it is the lower bound or the upper bound that has to be improved.

In Chapter 3 we continued on the path linking active learning and convex optimization. In this chapter we exhibited connections between those two fields by studying the problem of resource allocation under the diminishing returns assumption. In this problem our goal was to repeatedly allocate a fixed budget between several resources. We proposed an algorithm using imbricated binary searches to find the optimal allocation. Since we were working under a noisy gradient feedback assumption, the idea was to sample several times each query point in order to obtain the sign of the gradient of the function to maximize with high probability, which helped us to find out which region of the feasible domain to discard. Thus we have seen that in order to solve what was originally a sequential learning problem we ended up designing a stochastic convex optimization algorithm. In this problem we were interested in the more challenging objective of regret minimization instead of function error minimization. In order to quantify the difficulty of the problem at hand we assumed that the reward functions followed what we called an inductive Łojasiewicz assumption that we leveraged to obtain convergence rates depending on the exponent in the Łojasiewicz inequality. We obtained this result, which we showed to be minimax optimal up to the logarithmic terms by deriving a lower bound, in an adaptive manner, meaning that the algorithm does not need to know the values of the parameters of the objective function to be run, but will adapt to them. Obtaining adaptive algorithms is currently an active domain of convex optimization because in most of the real-world situations the decision makers do not know the convexity constants and the other regularity measures parameters. Future work on this subject of adaptive regret minimization could consist in dealing with functions that are not restricted to the resource allocation problem, and hence of a more general form.

In the second part of this thesis we did not work on particular machine learning problems but we rather focused on one of the most used stochastic optimization algorithms, which is Stochastic Gradient Descent (SGD). After having explored several sequential learning problems and having worked on their links with stochastic optimization we wanted to analyze the central brick of stochastic optimization, which is used in many domains of application, such as neural network training. The goal of Chapter 4 is to provide an extensive analysis of SGD in the convex case and in some non-convex situations, such as the weakly quasi-convex setting. In order to do that we adopted a new viewpoint consisting in analyzing the continuous-time model associated with the discrete scheme of SGD, that can be obtained by considering the limit of SGD when the learning rate goes to 0. This continuous-time model consists in a Stochastic Differential Equation (SDE), which is non-homogeneous in time since we consider the case of decreasing stepsizes in SGD. Using appropriate Lyapunov energy functions we can derive convergence results for the associated SDE and this provided us with good insights to design similar discrete energy functions for the analysis of SGD. We were thus able to obtain more intuitive and shorter proofs in the strongly convex case, as well as new optimal results in the convex case, removing the compactness assumption that was made in a previous work. In this chapter we obtained convergence bound for the last point iterate, while most of the works analyzed the case of averaging, which is easier. Our analysis in the convex setting finally disproved a conjecture on the rate of SGD. We concluded the chapter with new rates in the weakly quasi-convex setting which outperform existing results. These results, together with the new analysis framework we developed in this chapter, can be interestingly used to push forward the analysis of SGD in other non-convex landscapes.

Finally we have studied in this thesis several learning and optimization problems which 202 are all linked together, with a particular emphasis on settings which can be applied to real-world situations. This was the goal of adding regularization in contextual bandits, this was also the idea of using active learning techniques in optimal design, because online learning is much more appropriate than passive learning in many real situations. This was also the aim of designing adaptive algorithms in Chapter 3. In the second part of this thesis we wanted to study a well-known stochastic optimization algorithm to understand why and how it worked. We have continued in this direction with the work [START_REF] De Bortoli | Quantitative Propagation of Chaos for SGD in Wide Neural Networks[END_REF] which we did not discussed in the present thesis, where we investigated the behavior of SGD in wide neural networks in the over-parameterized setting, the goal of this work being to gain a better understanding of neural networks. All of these works, though different in appearance, are therefore similar in the sense that they aim at finding solutions to real-life learning and optimization problems. We can consequently say that using mathematics to solve and understand real-world learning problems was one of the goals of this thesis, which we wish to continue exploring in the future. Résumé: Dans cette thèse nous étudions plusieurs problèmes d'apprentissage automatique qui sont tous liés à la minimisation d'une fonction bruitée, qui sera souvent convexe. Du fait de leurs nombreuses applications nous nous concentrons sur des problèmes d'apprentissage séquentiel, qui consistent à traiter des données "à la volée", ou en ligne. La première partie de cette thèse est ainsi consacrée à l'étude de trois différents problèmes d'apprentissage séquentiel dans lesquels nous rencontrons le compromis classique "exploration vs. exploitation". Dans chacun de ces problèmes un agent doit prendre des décisions pour maximiser une récompense ou pour évaluer un paramètre dans un environnement incertain, dans le sens où les ré-compenses ou les résultats des différentes actions sont inconnus et bruités. Nous étudions tous ces problèmes à l'aide de techniques d'optimisation stochastique convexe, et nous proposons et analysons des algorithmes pour les résoudre. Dans la deuxième partie de cette thèse nous nous concentrons sur l'analyse de l'algorithme de descente de gradient stochastique qui est vraisemblablement l'un des algorithmes d'optimisation stochastique les plus utilisés en apprentissage automatique. Nous en présentons une analyse complète dans le cas convexe ainsi que dans certaines situations non convexes en étudiant le modèle continu qui lui est associé, et obtenons de nouveaux résultats de convergence optimaux.

Title: Sequential learning and stochastic optimization of convex functions Keywords: Stochastic optimization, sequential learning Abstract: In this thesis we study several machine learning problems that are all linked with the minimization of a noisy function, which will often be convex. Inspired by real-life applications we focus on sequential learning problems which consist in treating the data "on the fly", or in an online manner. The first part of this thesis is thus devoted to the study of three different sequential learning problems which all face the classical "exploration vs. exploitation" trade-off. Each of these problems consists in a situation where a decision maker has to take actions in order to maximize a reward or to evaluate a parameter under uncertainty, meaning that the re-wards or the feedback of the possible actions are unknown and noisy. We demonstrate that all of these problems can be studied under the scope of stochastic convex optimization, and we propose and analyze algorithms to solve them. In the second part of this thesis we focus on the analysis of the Stochastic Gradient Descent algorithm, which is likely one of the most used stochastic optimization algorithms in machine learning. We provide an exhaustive analysis in the convex setting and in some non-convex situations by studying the associated continuoustime model, and obtain new optimal convergence results.

Université Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France

Théorème 7 .

 7 Dans le cas où K > d nous obtenons la vitesse de convergence suivante pour le regret, pour tout T ≥ 1 R(T ) = O log(T ) T 5/4 . Cette borne supérieure n'est pas optimale et nous prouvons d'ailleurs la borne inférieure suivante dans le cas où K > d Théorème 8. Pour tout algorithme sur notre problème il existe un ensemble de paramètres tels que R(T ) T -3/2 . Les expériences numériques que nous réalisons à la fin du Chapitre 2 illustrent bien le fait que le cas K > d est bien plus complexe que le cas K = d et que la vitesse de convergence optimale se trouve certainement entre T -5/4 et T -3/2 .

  Regularized Contextual Bandits, Xavier Fontaine, Quentin Berthet and Vianney Perchet, International Conference on Artifical Intelligence and Statistics (AISTATS), 2019 • (Fontaine et al., 2019b) Online A-Optimal Design and Active Linear Regression, Xavier Fontaine, Pierre Perrault, Michal Valko and Vianney Perchet, soumis • (Fontaine et al., 2020b) An adaptive stochastic optimization algorithm for resource allocation, Xavier Fontaine, Shie Mannor and Vianney Perchet, International Conference on Algorithmic Learning Theory (ALT), 2020 • (Fontaine et al., 2020a) Convergence rates and approximation results for SGD and its continuous-time counterpart, Xavier Fontaine, Valentin De Bortoli and Alain Durmus, soumis. L'auteur a aussi participé à la publication suivante, qui n'est pas traitée dans cette thèse : • (De Bortoli et al., 2020) Quantitative Propagation of Chaos for SGD in Wide Neural Networks, Valentin de Bortoli, Alain Durmus, Xavier Fontaine and Umut Şimşekli, Advances in Neural Information Processing Systems, 2020.

  with H a ζ-strongly convex function of p, k a β-Hölder function of x and c any function of x.

  L b (p) = μ(b), p + λ(b)ρ(p) and μ(b) = 1 |b| b µ(x) dx and λ(b) = 1 |b| b λ(x) dx are the mean values of µ and λ on the bin b. Consequently we just need to minimize the unknown convex loss functions L b for each bin b ∈ B. We fall precisely in the setting of Section 1.2.3 and we propose consequently the following algorithm: for each time step t ≥ 1, given the context value X t , we run one iteration of the UC-FW algorithm for the loss function L b corresponding to the bin b X t . We note p T (b) the results of the algorithm on each bin b. Algorithm 1.1 Regularized Contextual Bandits Require: K number of arms, T time horizon Require: B = {1, . . . , B d } set of bins Require: t → α k (T /B d ) times arm k for all k ∈ [K] 3: for t ≥ 1 do 4:

  We note for all bins b ∈ B, p b = arg inf p∈∆ K L b (p) the minimum of L b on the bin b. We note p the piecewise constant function taking the values p b on the bin b.

  In order to force all the successive estimations of p b to be in S we sample each arm λ √ t times. Thus we have ∀i ∈ [K], p i ≥ λ/ √ t. Then we apply the UCB-Frank Wolfe algorithm on the bin b. Let pb min p∈S L b (p) and p b min p∈∆ K L b (p) .

A1. 4 .

 4 There exists η > 0 such that for all b ∈ B, dist(p b , ∂∆ K ) ≥ η, where p b is the point where L b : p → μ(b), p + λ(b)ρ(p) attains its minimum. Proposition 1.4. Assume A1.1, A1.2, A1.3, A1.4 and that ρ is ζ-strongly convex and S-smooth. Then running Algorithm 1.1 gives the following estimation error for all T ≥ 1,

  b

Proposition 1. 8 .

 8 Assume A1.1, A1.5 and that ρ is a ζ-strongly convex function. Then we have on each bin b ∈ B:b

Figure 1

 1 Figure 1.2 -Normalized Regret as a Function of T

Algorithm 2 . 1

 21 Naive randomized algorithmRequire: d, T , δ confidence parameter Require: N 1 , . . . , N d of sum N 1: Sample N k times each covariate X k 2: p N ←-(N 1 /N,. . . , N d /N ) 3: Compute empirical variances σ2 1 , . . . , σ2 d 4: for N + 1 ≤ t ≤ T do 5:

Lemma 2. 7 .

 7 With the pre-sampling of Lemma 2.6, L is smooth with constant C S where C

  Figure 2.1 -Different minimal ellipsoids

Figure 2 . 3 -

 23 Figure 2.2 -Regret as a function of T in log-log scale in the case of K = 3 covariates in R 3 .

  Figure 2.4 -Regret as a function of T in log-log scale in the case of K = 4 covariates in R 3 in a challenging setting.

  Figure 2.5 -Regret as a function of T for different values of K in log-log scale.

  Definition 3.3. A function f : R d → R satisfies locally the Tsybakov Noise Condition with parameters µ ≥ 0 and κ

  searches on the left or on the right of u m , depending on the sign of ∇G

  m ; z n ). As it holds that, by definition, ∇G

(

Lemma 3. 7 .

 7 During the binary search D (i+1) 2j-1 (v) we have, for all point w tested by this binary search, |∇G

  the regret induced by the binary search D (i) j (v) as the regret suffered when optimizing the function w → G

  v) reaches its maximum and where the successive points tested by the binary search D

  are the (not necessarily distinct) zij(t).

Proof.

  The regret of the binary search D

  is the sum for all steps t ∈ [T (i) j (v)] of the sum of two terms: the difference of the function values of G (i) j (• ; v) between the optimal value z

  (i) j (v) and zij(t) and the sub-regrets R (i+1) 2j-1 (zij(t)) and R (i+1) 2j

  (v -zij(t)) of the binary searches that are the children of D(i) j (v).

Proof.

  The binary search D (i) j (v) aims at minimizing the function (w → G (i) j (w; v)). Let us note w 1 , . . . , w m , . . . the values that are tested by this binary search. During the binary search the signs of the values of ∇G

  m ; v) are needed. In order to compute them the algorithm runs sub-binary searches (unless

  in Section 3.3). • p ∈ N * : the point w m has to be sampled a number of times equal to the number of iterations of D (i+1) 2j-1 (w m ) and D (i+1) 2j (v -w m ). Let us therefore compute the number of samples used by D

  w m ). Since there are at most log 2 (T ) different points x k that are tested during the binary search D (i+1) 2j-1 (w m ), we have a final number of iterations for w m which is 8 log(2T /δ) log(T ) p ∇G (i) j (w m )

  Figure 3.1 -Regret, Upper-bound and Lower bound for β = 1.5

  Figure 3.2 -Regret, Upper-bound and Lower bound for β = 1.75

12

  Figure 3.3 -Regret, Upper-bound and Lower bound for β = 2

  Figure 4.1 -In (a) we show (log(E[f (X n )] -min R d f )) n∈N and in (b) we observe that empirical rates match theoretical rates for different values of α.

Figure 4 . 2 -

 42 Figure 4.2 -Convergence rates for the functions ϕ p match the theoretical results of Theorem 4.6 asymptotically, i.e., when p is large.

Figure 4 . 3 -

 43 Figure 4.3 -Comparison of the convergence rates in the convex and weakly quasi-convex settings.

  ≥ 0 which does not depend on γ and k. (b) Second, we have using A4.1, A4.5 and that for any a, b ≥ 0, (a + b) 2

Lemma 4 .

 4 16. Assume A4.1. Then for any x

  . This gives in particular, for every t ∈ [0, T ],(t + γ α ) α-1 E X t -(T + γ α ) 1-2α log(T + γ α ) (t + γ α ) α-1 (4log(T + γ α )(t + γ α ) -α ,

E

  [ X t -y 0 , ∇f (X t ) ] dt + T T -u γ α (t + γ α ) -α E [Tr(Σ(X t ))] dt .Combining this result, A4.5, A4.2, (4.50) and (4.51) we obtain for any u ∈ [0, T ]-(T -u + γ α ) α E X T -uγ α ) -α dt + (T + γ α ) 1-2α T T -u

  log(T + γ α ) {(T + γ α ) α -(T -u + γ α ) α } (T + γ α ) 1-2α . Therefore, we get for any u ∈ [0, T ] T T -u E [f (X t ) -f (y 0 )] dt ≤ (C 1 /2) ((T + γ α ) α -(T -u + γ α ) α ) (4.52) + (1/2)(T -u + γ α ) α E X T -u -y 0 2 + (C 1 /2) (T + γ α ) 1-α -(T -u + γ α ) 1-α log(T + γ α ) , with C 1 = max(C (c) 3,α , (γ α η + 4αC (c) 1,α )(1 -α) -1). We divide the rest of the proof into three parts, to bound the quantities S(1) -S(T ), S(T ) -f and S(0) -S(1).

  (a) Bounding S(1) -S(T ): In the case where α ≤ 1/2, Lemma 4.15 gives that for all u ∈ [0, T ]:((T + γ α ) α -(T -u + γ α ) α ) ≤ (T + γ α ) 1-α -(T -u + γ α ) 1-α ,and we also have, for all u ∈ [0, T ]:(T + γ α ) 1-α -(T + γ α -u) 1-α (4.53) = ((T + γ α ) 1-α -(T + γ α -u) 1-α )((T + γ α ) α + (T + γ α -u) α ) ((T + γ α ) α + (T + γ α -u) α ) ≤ (T + γ α ) -(T + γ α -u) + (T + γ α ) 1-α (T + γ α -u) α -(T + γ α ) α (T + γ α -u) 1-α (T + γ α ) α ≤ 2u/(T + γ α ) α .

  f (X T )] -E [f (X s )]) ds . (4.57)Using Lemma 4.13 on the stochastic process f (X t ) t≥0 and A4.1, we have for all s ∈[T -1, T ] E [f (X T )] -E [f (X s )] = -T s (γ α + t) -α E ∇f (X t ) 2 dt + (L/2)γ α T s (t + γ α ) -2α E [Tr(Σ(X t ))] dt ≤ (ηL/2)γ α T s (t + γ α ) -2α dt ≤ (C 1 L/2)(s + γ α ) -2α (T -s) .Plugging this result into (4.57) yieldsS(0) -S(1) ≤ (C 1 L/2) s)(s + γ α ) -2α ds ≤ C 1 L(T -1 + γ α ) -2α ≤ C 1 L(T -1) -2α .(4.58)Combining (4.55), (4.56) and (4.58) gives the desired resultE [f (X T )]-f ≤ C (c) log(T ) 2 T -min(α,1-α) + log(T )T -min(α,1-α) + T -min(α,1-α) + (T -1) -2α ,with C (c) = 4C 1 (1 + L). We note C = C (c) .

  α : R + → R + such that, for any n ≥ 0, E X n -x 2 ≤ C

2 .

 2 Let u ∈ {0, • • • , N }. Summing now (4.61) between k = N -u and k = N gives E N k=N -u f (X k ) -f (y 0 ) ≤ (γη/2)

+

  . Equation (4.62) leads therefore to, withC (b) = ((γη/2) + (γ/2)C 0 )(1α) -1 . E N k=N -u f (X k ) -f (y 0 ) ≤ (γη/2)(1 -α) -1 (N + 1) 1-α -(N -u) 1-α (N + 1) 1-2α log(N + 1) ((N + 1) α -(N -u + 1) α ) + (γ/2)C 0 (N + 1) β log(N + 1)(1 -α) -1 (N + 1) 1-α -(N -u) 1-α ≤ C (b) (N + 1) β (1 + log(N + 1)) 2 (N + 1) 1-α -(N -u) 1-α + (2γ) -1 (N -u + 1) α E X N -u -((N + 1) α -(N -u) α ) + (2γ) -1 4C (d) 1,α (N + 1) 1-α -(N -u) 1-α ≤ C (d) (N + 1) β (1 + log(N + 1)) 2 (N + 1) 1-α -(N -u) 1-α + (2γ) -1 (N -u + 1) α E X N -u -y 0 2 ,where we used Lemma 4.15 and where we noted C (d) C (b) + (2γ) -1 (C now that, similarly to Equation (4.53) we have(N + 1) 1-α -(N -u) 1-α = (N + 1) 1-α -(N -u) 1-α ((N + 1) α + (N -u) α ) ((N + 1) α + (N -u) α ) -1≤ 2(u + 1)/(N + 1) α .

  k ) -f (y 0 ) ≤ (γη/2)(1 -α) -1 (N + 1) 1-α -(N -u) 1-α + (2γ) -1 (N -u + 1) α E X N -ulog(N + 1)(N + 1) 1-2α ((N + 1) α -(N -u + 1) α ) + (γ/2)C 0 (1 -α) -1 (N + 1) 1-α -(N -u) 1-α ≤ C (b) (N + 1) 1-α -(N -u) 1-α + (2γ) -1 (N -u + 1) α E X N -u -(1 + log(N + 1)) ((N + 1) α -(N -u) α ) ≤ C (d) (1 + log(N + 1)) (N + 1) 1-α -(N -u) 1-α + (2γ) -1 (N -u + 1) α E X N -u -y 0 2 .(c) Case α > 1/2: In that case, α > α and Lemma 4.18 gives∀k ∈ {0, . . . , N } , E X k -y 0 2 ≤ 2E X k -x 2 + 2E y 0 -x f (X k ) -f (y 0 ) ≤ ((γη/2) + γC 0 /2)(1 -α) -1 (N + 1) 1-α -(N -u) 1-α + (2γ) -1 (N -u + 1) α E X N -u -((N + 1) α -(N -u + 1) α ) ≤ C (b) (N + 1) 1-α -(N -u) 1-α + (2γ) -1 (N -u + 1) α E X N -u -((N + 1) α -(N -u) α ) ≤ C (d) ((N + 1) α -(N -u) α ) + (2γ) -1 (N -u + 1) α E X N -u -y 0 2 .Notice now that, similarly to Equation (4.53) we have(N + 1) α -(N -u) α= ((N + 1) α -(N -u) α ) (N + 1) 1-α + (N -u) 1-α (N + 1) 1-α + (N -u) 1-α -1

( a )≤

 a Bounding (S 0 -S N ): Let u ∈ {0, . . . , N }. Equation (4.63) with the choice y 0 = X N -u gives E N k=N -u f (X k ) -f (X N -u ) ≤ 2C (d) (u + 1)/(N + 1) min(α,1-α) (1 + log(N + 1))Ψ α (N + 1) . 2C (d) (N + 1) -min(α,1-α) (1 + log(N + 1))Ψ α (N + 1) + E [f (X N -u )] .

( b )

 b Bounding (S N -f ): Equation (4.63) with the choice y 0 = x and u = N gives (N + 1) -1 E N k=0

Lemma 4. 19 .

 19 Assume that f is continuous, that x ∈ arg min x∈R d f (x) and that there exist c, R ≥ 0 such that for anyx ∈ R d with x-x ≥ R we have f (x)-f (x ) ≥ c x-x . Let p ∈ N, X a d-dimensional random variable and D 4 ≥ 1 such that E[(f (X)-f (x )) 2p ] ≤ D 4 . Then there exists D 5 ≥ 0 such that E X -x 2p ≤ D 5 D 4 .

≤

  D p {1+(γ α +t) m-α -1 α } ≤ D p {1+(γ α +t) m-m/α α } ≤ D p {1+γ m-m/α α α } ,which concludes the proof.

203Titre:

  Apprentissage séquentiel et optimisation stochastique de fonctions convexes Mots clés: Optimisation stochastique, apprentissage séquentiel

Propagation of Chaos for SGD in Wide Neural

  •[START_REF] De Bortoli | Quantitative Propagation of Chaos for SGD in Wide Neural Networks[END_REF]) Quantitative

Networks, Valentin de Bortoli, Alain Durmus, Xavier Fontaine and Umut Şimşekli, Advances in Neural Information Processing Systems, 2020.

  où au lieu de considérer le dernier point x N on considère la valeur moyenne xN Il est plus facile d'obtenir des vitesses de convergence dans le cas du moyennage (Nemirovski et al., 2009) que d'obtenir des vitesses non asymptotiques pour le dernier point. En effet ces dernières impliquent directement avec l'inégalité de Jensen des vitesses de convergence dans le cas du moyennage. Il est d'ailleurs intéressant de remarquer à ce propos que les algorithmes présentés dans la Section 2.3 portent sur la version moyennée des itérés et non sur le dernier point. À notre connaissance il n'existe pas de résultat de convergence général dans le cas convexe et lisse pour SGD.

	xN =	1 N	N k=1	x k .

L'un des seuls résultas dont on dispose pour le dernier itéré est en effet dû à Shamir et

[START_REF] Shamir | Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes[END_REF]

qui font l'hypothèse que les points restent dans un compact, ce qui est évidemment une hypothèse forte. Finalement

[START_REF] Bach | Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning[END_REF]

conjecturent que la vitesse de convergence optimale pour SGD dans le cas convexe est O(k -1/3 ), ce que nous contredisons dans le Chapitre 4.

Proposition 1.3. Assume A1.1, A1.2, A1.3. If

	.	(1.4)
	With Equation (1.4) and convex analysis tools we prove the

p is the piecewise constant function on the set of bins B minimizing the loss function L, we have the following bound

Table 1 .

 1 1 -Slow and Fast Rates for Estimation and Approximation Errors on a Bin

	Error	Slow	Fast
	Estim.

  . Let us note ρ t L(p t ) -L(p ) and ε t+1 (e π(t+1) -e t+1 ) ∇L(p t ) with e t+1 = arg max p∈∆ K p ∇L(p t ). (Berthet and Perchet, 2017, Lemma 12) gives, for t ≥ nK,

	(t + 1)ρ t+1 ≤ tρ t + ε t+1 +	C S t + 1	.

Summing for t ≥ nK gives

  Table 4.1 and Figure 4.3a.

Table 4 .

 4 1 -Convergence rates for convex SGD under different settings (B: Bounded Gradients, L: Lipschitz Gradient), up to the logarithmic terms

	Reference	Theorem 4.6 (L) (BM'11) (B, L) (BM'11)

  by A4.1. Actually our proof can be very easily adapted to the simpler setting where (E[ H(X n , Z n+1 2 ]) n∈N is supposed to be bounded instead of A4.1. We present this result in Corollary 4.4.

		0.7	x 4 x 6
	rate of convergence	0.2 0.3 0.4 0.5 0.6	x 8 x 10 Theoretical rate
		0.1
		0.1	0.2	0.3	0.4	α	0.5	0.6	0.7	0.8

Table 4 .

 4 2 -Rates for continuous SGD with non-convex assumptions

	Reference	Corollary 4.5-(a) Corollary 4.5-(b) (OL'19)

  2p-1 197 + (2p -1)E ∇f (X t )∇f (X t ) , Σ(X t ) (f (X t ) -f (x ) 2p-2 ) + δ p (γ α + t) -1 E t,p .Combining (4.69), Lemma 4.12, Lemma 4.16, Lemma 4.20 and the fact that for any t ≥ 0,E[ X t -x 4p ] 1/(2p) ≤ D p,1 {1 + (γ α + t) m-ϕα } we get dE t,p / dt ≤ -2p(γ α + t) -α+δp E (f (X t ) -f (x )) 2p 1+1/(2p) E X t -x + pγ α (γ α + t) -2α+δp LηE (f (X t ) -f (x )) 2p-1 + L(2p -1)ηE (f (X t ) -f (x )) 2p-1 + δ p (γ α + t) -1 E t,p ≤ -2p(γ α + t) -α-δp/(2p) E + pγ α (d + 2p -1)Lη(1 + η)(γ α + t) -2α+δp/(2p) E + δ p (γ α + t) -1 E t,p ≤ -2p(γ α + t) -α-δp/(2p) E {1 + (γ α + t) m-ϕα } -1 + pγ α (d + 2p -1)Lη(1 + η)(γ α + t) -2α+δp/p E + δ p (γ α + t) -1 E t,p ≤ -2p(γ α + t) (ϕ-1)α-δp/(2p)-m E {1 + (γ α + t) -m+ϕα } -1 + 2pγ α (d + 2p -1)Lη(1 + η)(γ α + t) -2α+δp/(2p) E + δ p (γ α + t) -1 E t,p ≤ -pD -1 p,1 {1 + γ -m+ϕα α } -1 (γ α + t) (ϕ-1)α-δp/(2p)-m E + 2pγ α (d + 2p -1)Lη(1 + η)(γ α + t) -2α+δp/(2p) E

					4p	-1/(2p)
	1+1/(2p) t,p	E X t -x	2p	-1/(2p)
					1-1/(2p)
					t,p
	1+1/(2p) t,p	D -1 p,1 1-1/p
				t,p
	1+1/(2p) t,p	D -1 p,1 1-1/(2p)
					t,p
					1+1/(2p)
					t,p
					1-1/(2p)
					t,p

+ δ p (γ α + t) -1 E t,p . Since m ∈ [0, 2], we have that 1 -m + (ϕ -1)α ≥ (1 + ϕ)α/2 -m/2. Hence, (1 -ϕ)α -δ p /(2p) -m ≤ 2α + δ p /(2p) , (1 -ϕ)α -δ p /(2p) -m ≤ 1 .

Therefore, using Lemma 4.1, there exists D (a)

p ≥ 1 such that for any t ≥ 0, E t,p ≤ D (a)

  X s -x ds (4.70)+ (γ α /2) (γ α + s) -2α Σ(X s ), ∇ 2 f (X s ) ds . Let E t = E[ X t -x 2 ].Using, (4.70) we have for any t ≥ 0,E t ≤ -(γ α + t) -α E [ ∇f (X s ), X s -x ] + (γ α Lη/2)(γ α + t) -2α .(4.71)

The time horizon T ∈ N * is supposed here to be known, even if the so-called "doubling trick"[START_REF] Auer | Gambling in a rigged casino: The adversarial multi-armed bandit problem[END_REF] could circumvent this issue.

One can add an intercept term and assume that yi = β 0 + x i β + εi, with β ∈ R p+1 , which does not alter much the discussion of this section.

More details on uniformly convex functions will be given in Section 3.2.2.

A L-smooth function is a function whose gradient is L-Lipschitz-continuous.

See Section 1.4.2 for a more precise statement.

On suppose ici que l'horizon de temps T ∈ N * est connu, même si le "doubling-trick"[START_REF] Auer | Gambling in a rigged casino: The adversarial multi-armed bandit problem[END_REF] permettrait de s'affranchir de cette contrainte.

On peut aussi ajouter un terme d'ordonnée à l'origine et supposer plutôt que yi = β 0 + x i β + εi, avec β ∈ R p+1 , mais cela ne changera pas grand chose à la discussion qui va suivre.

Nous sous-entendons ici que l'on cherche à minimiser un critère A, D ou E pour Ω.

Plus de détails sur les fonctions uniformément convexes sont donnés dans la Section 3.2.2.

Nous appellerons ici une fonction L-lisse une fonction dérivable dont le gradient est L-Lipschitz.

Voir Section 1.4.2 pour un énoncé plus précis.

This chapter is joint work with Quentin Berthet and Vianney Perchet. It has led to the following publication:(Fontaine et al.,

2019a) Regularized Contextual Bandits, Xavier Fontaine, Quentin Berthet and Vianney Perchet, International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.

[K] = {1, . . . , K} .

The Landau notation O(•) has to be understood with respect to T . The precise bound is given in the proof.

The notation O L β ,β,K,d means that there is a hidden constant depending on L β , β, K and d. The constant can be found in the proof.

⇔ ∇f is L-Lipschitz continuous, where the last equivalence comes from(Nesterov, 2004, Theorem 2.1.5).

The precise dependency in the constants is again given in the proof below.

λ(b)dx .

This chapter is joint work with Pierre Perrault, Michal Valko and Vianney Perchet. It has led to the following publication:(Fontaine et al.,

2019b) Online A-Optimal Design and Active Linear Regression,Xavier Fontaine, Pierre Perrault, Michal Valko and Vianney Perchet, submitted.

[K] = {1, . . . , K}.

2

The notation O Γ,σ k means that there is a hidden constant depending on Γ and on the σ k . The explicit dependency on these parameters is given in the proof.

This chapter is joint work with Shie Mannor and Vianney Perchet. It has led to the following publication:(Fontaine et al.,

2020b) An adaptive stochastic optimization algorithm for resource allocation, Xavier Fontaine, Shie Mannor and Vianney Perchet, International Conference on Algorithmic Learning Theory (ALT), 2020.

See https://en.wikipedia.org/wiki/Diminishing_returns

The O(•) notation is used to hide poly-logarithmic factors.

To be precise, we could only require that this property holds for any siblings that are not children of the root. For those two, we only need that the mapping f left (z) + f right (1 -z) satisfies the local Łojasiewicz inequality.

-j1,min √ T 1 .

Formally, we just need to control the ∞ distance between the gradients, as we assume that the feedbacks of the decision maker are noisy gradients. But we could have assumed that he also observes noisy evaluations of f1(x1) and f2(x2). This is why we also want to control the ∞ distance between the functions fi and fi.

This chapter is joint work with Valentin De Bortoli and Alain Durmus. It has led to the following publication:(Fontaine et al.,

2020a) Convergence rates and approximation results for SGD and its continuous-time counterpart , XavierFontaine, Valentin De Bortoli, Alain Durmus, submitted.

Proof. The proof is a direct consequence of A4.1,(Nesterov, 2004, Lemma 1.2.3) and Theorem 4.1.

This section is mainly the work of Valentin De Bortoli, but is put here for completeness.

Remerciements

Proof. ∇f (x, y) = 2(x -y, y -x) and ∇f (x, y) 2 = 8(x -y) 2 = 8f (x, y). Consequently, since f is minimal at 0, f verifies the Łojasiewicz inequality for β = 2 and c = 1/8.

Let a = (0, 0) and b = (1, 1). If f is uniformly convex on R 2 with parameters ρ and µ then, for α = 1/2,

This is a contradiction since µ > 0 and ρ ≥ 2.

Example 3.2. The function g : (x, y, z) ∈ ∆ 3 → (x -1) 2 + 2(1 -y) + 2(1 -z) is not uniformly convex on the simplex ∆ 3 but verifies the Łojasiewicz inequality.

Proof. g is constant on the set {x = 0} (since y + z = 1). And therefore g is not uniformly convex (take two distinct points in {x = 0}).

We have ∇g(x, y, z) = (2x -2, -2, -2) and ∇g(x, y, z) 2 = 4((x -1) 2 + 2) ≥ 8. Since

2 and g verifies the Łojasiewicz inequality on ∆ 3 .

We conclude this section by giving additional examples of functions verifying the Łojasiewicz inequality.

Then h verifies the Łojasiewicz inequality with respect to the parameters β = α/(α -1) and c = √ K.

The last example is stated in the concave case because it is an important case of application of our initial problem.

Otherwise, the inequality is satisfied on ∆ K for any β ≥ 1 (with a different constant for each β).

Proof. Indeed, let x ∈ ∆ K . If there exists at least one positive a k , then F is quadratic, so if we denote by x its maximum and H its Hessian (it is the diagonal matrix with -a k on coordinate k), we have

Hence F satisfies the Łojasiewicz conditions with β = 2 and c = 1/(4 min k a k ). If all f k are linear, then F (x ) -F (x) ≤ max j b j -min j b j and ∇F (x) = b . Given any β ≥ 1, it holds that

The complexity class

We present now the complexity class of our problem. In all the following, we will handle concave functions. All results from Section 3.2.2 remain valid, with considering their concave counterpart.

. The major issue is that we do not have directly access to the gradients ∇H of those functions because they are defined via an optimization problem, unless they correspond to leaves in the aforementioned tree. In that case their gradient is accessible and using the envelope theorem [START_REF] Afriat | Theory of maxima and the method of Lagrange[END_REF] we can recursively compute all gradients. We indeed have the following lemma (whose proof is immediate and omitted).

Recall that gradients of H

(1) 1 (z) and H

(1)

2 (1 -z) were needed to apply the K = 2 machinery to the optimization of F once this problem is rewritten as max z H

(1)

2 (1 -z). Lemma 3.6 provides them, as the gradient of yet other functions H

(2) 1 and/or H

(2) 2 . Notice that if K = 4, then those two functions are actually the two basis functions f 1 and f 2 , so the agent has direct access to their gradient (up to some noise). It only remains to find the point ω * z which is done with the binary search introduced in the previous section.

If K > 4, the gradient of H

(2) 1 (and, of course, of H

(2)

2 ) is not directly accessible, but we can again divide H

(2) 1 into two other functions H

(3) 1 and H

(3) 2 . Then the gradient of H

(2) 1 will be expressed, via Lemma 3.6, as gradients of H

(3) 1 and/or H

(3) 2 at some specific point (again, found by binary searches as in K = 2). We can repeat this process as

Part II

Stochastic optimization

We also derive weak approximation estimates of order 1 between continuous and discrete SGD. Note that in the case where α ≥ 1/2, these weak results are a direct consequence of Proposition 4.1. Denote by G p,k the set of k-times continuously differentiable functions g such that there exists K ≥ 0 such that for any

In addition, assume that for any m ∈ N and x ∈ R d , µ Z ( H(x, •) -∇f (x) 2m ) ≤ η m with η m ≥ 0. Then for any T ≥ 0, there exists C ≥ 0 such that for any γ ∈ (0, γ], n ∈ N with γ α = γ 1/(1-α) , nγ α ≤ T we have

These results extend (Li et al., 2017, Theorem 1.1 (a)) to the decreasing stepsize case. Once again, the result obtained in Proposition 4.2 must be compared to similar weak error controls for SDEs. For example, under appropriate conditions, [START_REF] Talay | Expansion of the global error for numerical schemes solving stochastic differential equations[END_REF] shows that the EM discretization

approximation of order 1 of (4.5). For clarity reasons the proof of Proposition 4.2 is postponed to Appendix 4.A.4.

Convergence of the continuous and discrete SGD processes 4.3.1 Two basic comparison lemmas

We now turn to the convergence of SGD. In the continuous-time setting, in order to derive sharp convergence rates for (4.2), we will consider appropriate energy functions V : R + × R d → R + which will depend on the conditions imposed on the function f . Then, we show that (t → v(t) = E[V(t, X t )]) satisfies an ODE and prove that it is bounded using the following simple lemma.

Lemma 4.1.

If there exists t 0 > 0 and A > 0 such that for all t ≥ t 0 and for all u ≥ A, F (t, u) < 0, then there exists B > 0 such that for all t ≥ 0, v(t) ≤ B, with

Proof. Assume that there exists t ≥ 0 such that v(t) > B, and let t 1 = inf {t ≥ 0 : v(t) > B}.

By definition of B, t 1 ≥ t 0 , and by continuity of v, v(t 1 ) = B. By assumption, F (t 1 , v(t 1 )) < 0. Then dv(t 1 )/dt < 0 and there exists t 2 < t 1 such that v(t 2 ) > v(t 1 ) = B, hence the contradiction.

Considering discrete analogues of the energy functions and ODEs found in the study of the continuous SGD process solution of (4.2), we also derive explicit convergence bounds for the discrete SGD process. To that purpose, we establish a discrete analog of Lemma 4.1. Note that we have to add an additional assumption to F in order to have a correct statement.

Lemma 4.2. Let

Assume that there exist n 0 ∈ N and A 1 > 0 such that for all n ≥ n 0 and for all x ≥ A 1 , F (n, x) < 0. In addition, assume that there exists A 2 > 0 such that for all n ≥ n 0 and for all x ≥ 0, F (n, x) ≤ A 2 . Then, there exists B > 0 such that for all n ∈ N u n ≤ B with B = max(max n≤n 0 +1 u n , A 1 ) + A 2 .

Combining this result, Proposition 4.6, Lemma 4.7 and the Cauchy-Schwarz inequality we get that

which concludes the proof.

Proposition 4.8. Let p ∈ N and g ∈ G p . Let γ > 0 and α ∈ [0, 1). Assume A4.1, A4.2, A4.3 and that for any x ∈ R d , µ Z ( H(x, •) -∇f (x) 2p ) ≤ η p , with η p ≥ 0. Then for any T ≥ 0, there exists A T,8 ≥ 0 such that for any γ ∈ (0, γ], k ∈ N with (k + 1)γ α ≤ T and

where (X k ) k∈N satisfies the recursion (4.1) with X k = X 0 and (X t ) t≥0 is the solution of (4.2) with X kγα = X 0 .

Proof

Using Lemma 4.7, Lemma 4.8, Lemma 4.10, Lemma 4.11 and the Cauchy-Schwarz inequality we have

(1 + ÃT,1 )

T,4 γ 2 (k + 1) -2α + γ(k + 1) -(1+α) (1 + X 0 )

T,2 γ 2 (k + 1) -2α (1 + X 0 2 )(1 + ÃT,1 ) 1/2 (1 + X 0 p ) , which concludes the proof.

Proposition 4.9. Let γ > 0 and α ∈ [0, 1). Assume that f ∈ G p,4 , Σ 1/2 ∈ G p,3 A4.1, A4.2 and A4.3. Let p ∈ N and g ∈ G p,2 . In addition, assume that for any m ∈ N and x ∈ R d , µ Z ( H(x, •) -∇f (x) 2m ) ≤ η m with η m ≥ 0. Then for any T ≥ 0, there exists A T,9 ≥ 0 such that for any γ ∈ (0, γ], k ∈ N with kγ α ≤ T and X 0 ∈ R d we have

where (X k ) k∈N satisfies the recursion (4.1) and (X t ) t≥0 is the solution of (4.2) with X 0 = X 0 .