Vincent Edouard Delaitre

Grave

Damien Garreau

Rémy Degenne

Rémi Leblond

Horia Mania

John Weed Mais

Jean-Baptiste Alayrac

Piotr Bojanowski

Nicolas Boumal

Guilhem Chéron

Théophile Dalens

Christophe Dupuy

Fajwel Fogel

Vadim Kantorov

Sesh Kumar

Rémi Lajugie

Loic Landrieu

Maxime Oquab

Julia Peyre

Anastasia Po- Dosinnikova

Antoine Recanati

Vincent Roulet

Damien Scieur

Guillaume Seguin

Nino Shervashidze

Matthew Trager

Gül Varol

J'ai Beaucoup Voyagé

Afonso Bandeira

Victor-Emmanuel Brunel

Cheng Mao

Quentin Berthet

Alain Durmus

Pierre Gaillard

Vianney Perchet

J'ai Aussi Pu

Ulysse G Et

Valentin M Merci

la gare de Lyon, Aymeric Dieuleveut et

Keywords: Convex optimization, acceleration, averaging, stochastic gradient, leastsquares regression, stochastic approximation, dual averaging, mirror descent, discriminative clustering, convex relaxation, sparsity, statistical seriation, permutation learning, minimax estimation, shape constraints. viii

Many problems in machine learning are naturally cast as the minimization of a smooth function defined on a Euclidean space. For supervised learning, this includes least-squares regression and logistic regression. While small-scale problems with few input features may be solved efficiently by many optimization algorithms (e.g., Newton's method), large-scale problems with many high-dimensional features are typically solved with first-order techniques based on gradient descent, leading to algorithms with many cheap iterations.

In this manuscript, we consider the particular case of the quadratic loss. In the first part, we are interested in its minimization, considering that its gradients are only accessible through a stochastic oracle that returns the gradient at any given point plus a zero-mean finite variance random error. We propose different algorithms to efficiently solve these minimization problems in many cases. In the second part, we consider two applications of the quadratic loss in machine learning: unsupervised learning, specifically clustering and statistical estimation, specifically estimation with shape constraints.

In the first main contribution of the thesis, we provide a unified framework for optimizing non-strongly convex quadratic functions, which encompasses accelerated gradient descent, averaged gradient descent and the heavyball method. They are studied through second-order difference equations for which stability is equivalent to an O(1/n 2 ) convergence rate. This new framework suggests an alternative algorithm that exhibits the positive behavior of both averaging and acceleration.

The second main contribution aims at obtaining the optimal prediction error rates for least-squares regression, both in terms of dependence on the noise of the problem and of forgetting the initial conditions. Our new algorithm rests upon averaged accelerated gradient descent and is analyzed under finer assumptions on the covariance matrix of the input data and the initial conditions of the algorithm which leads to tighter convergence rates expressed with dimension-free quantities.

The third main contribution of the thesis deals with the minimization of composite objective functions composed of the expectation of quadratic functions and a convex function. We show that stochastic dual averaging with a constant step-size has a convergence rate O(1/n) without strong convexity assumption, extending earlier results on least-squares regression to any regularizer and any geometry represented by a Bregman divergence.

As a fourth contribution, we consider the problem of clustering high-dimensional data. We present a novel sparse extension of the discriminative clustering framework and propose a natural extension for the multi-label scenario. We also provide the first theoretical analysis of this formulation with a simple probabilistic model and vii an efficient iterative algorithm with better running-time complexity than existing methods.

The fifth main contribution of the thesis deals with the seriation problem, which consists in permuting the rows of a given matrix in such way that all its columns have the same shape. We propose a statistical approach to this problem where the matrix of interest is observed with noise and study the corresponding minimax rate of estimation of the matrices. We also suggest a computationally efficient estimator whose performance is studied both theoretically and experimentally.

Résumé

De nombreux problèmes en apprentissage automatique sont formellement équivalents à la minimisation d'une fonction lisse définie sur un espace euclidien. Plus précisément, dans le cas de l'apprentissage automatique supervisé, cela inclut la régression par moindres carrés et la régression logistique. Alors que les problèmes de petite taille, avec peu de variables, peuvent être résolus efficacement à l'aide de nombreux algorithmes d'optimisation (la méthode de Newton par exemple), les problèmes de grande échelle, avec de nombreuses données en grande dimension, sont, quant à eux, généralement traités à l'aide de méthodes du premier ordre, dérivées de la descente de gradient, conduisant à des algorithmes avec de nombreuses itérations peu coûteuses.

Dans ce manuscrit, nous considérons le cas particulier de la perte quadratique. Dans une première partie, nous nous intéressons à la minimisation de celle-ci dans l'hypothèse où nous accédons à ses gradients par l'intermédiaire d'un oracle stochastique. Celui-ci retourne le gradient évalué au point demandé plus un bruit d'espérance nulle et de variance finie. Nous proposons différents algorithmes pour résoudre efficacement ce problème dans de multiples cas. Dans une seconde partie, nous considérons deux applications différentes de la perte quadratique à l'apprentissage automatique : la première en apprentissage non-supervisé, plus spécifiquement en partitionnement des données, et la seconde en estimation statistique, plus précisément en estimation sous contrainte de forme.

La première contribution de cette thèse est un cadre unifié pour l'optimisation de fonctions quadratiques non-fortement convexes. Celui-ci comprend la descente de gradient accélérée, la descente de gradient moyennée et la méthode de la balle lourde. Ces méthodes sont étudiées grâce à des équations aux différences finies du second ordre dont la stabilité est équivalente à une vitesse de convergence O(1/n 2 ) de la méthode étudiée. Ce nouveau cadre nous permet de proposer un algorithme alternatif qui combine les aspects positifs du moyennage et ceux de l'accélération.

La deuxième contribution est d'obtenir le taux optimal d'erreur de prédiction pour la régression par moindres carrés en fonction de la dépendance, à la fois au bruit du problème et à l'oubli des conditions initiales. Notre nouvel algorithme tire son origine de la descente de gradient accélérée et moyennée et nous l'analysons sous des hypothèses plus fines sur la matrice de covariance des données et sur les conditions initiales de l'algorithme. Cette nouvelle analyse aboutit à des taux de convergence plus tendus qui ne font pas intervenir la dimension du problème.

La troisième contribution de cette thèse traite du problème de la minimisation ix de fonctions composites qui sont la somme de l'espérance de fonctions quadratiques et d'une régularisation convexe. Nous montrons qu'utilisée avec un pas constant, la méthode duale moyennée converge vers la solution du problème à la vitesse O(1/n) sans hypothèse de forte convexité. Cela étend les résultats existants sur la régression par moindres carrés aux cas régularisés et aux différentes géométries induites par une divergence de Bregman. Dans une quatrième contribution, nous considérons le problème de partitionnement de données de grande dimension. Nous présentons ainsi une nouvelle extension parcimonieuse du partitionnement discriminatif et son extension naturelle au cas de données avec de multiples labels. Nous analysons aussi cette formulation théoriquement, et ce pour la première fois, à l'aide d'un modèle probabiliste simple et nous proposons un nouvel algorithme itératif ayant une meilleure complexité que les méthodes existantes.

La dernière contribution de cette thèse aborde le problème de la sériation. Celuici consiste à permuter les lignes d'une matrice afin que ses colonnes aient toutes une forme identique. Nous adoptons une approche statistique et, dans le cas où la matrice est observée avec du bruit, nous étudions les taux d'estimation minimax correspondants. Nous proposons aussi un estimateur computationellement efficace et nous étudions ses performances d'un point de vue théorique et pratique.

Mots-clés : Optimisation convexe, accélération, moyennage, gradient stochastique, régression par moindres carrés, approximation stochastique, algorithme dual moyenné, descente miroir, partitionnement discriminatif, relaxation convexe, parcimonie, sériation statistique, apprentissage de permutation, estimation minimax, contraintes de forme.

x Chapter 1 Introduction

Machine Learning

Machine learning is a recent scientific domain at the interface of applied mathematics, statistics and computer science. It aims at giving a machine the ability to produce a predictive analysis based on existing data and it applies to a tremendous variety of domains ranging from computer vision, natural language processing, advertising, bio-informatics, robotics, speech processing and economics. Its utmost challenge is to design general methods that can be applied across all of these domains. Machine learning has been particularly important in the current context of big data, that is a context of increasing data volumes, improved access to data and facilitated data collection.

Learning can be very loosely defined as the ability to answer a question after observing data. The related learning problems may be described through the interaction between the learner and the environment. An algorithm able to detect if a cat appears in an image, first needs to be trained on a dataset of examples. This dataset would consist of many pictures with a label saying if a cat is present or not. On the basis of this training it would design a rule to decide if there is a cat in a new picture. This is what is called supervised learning. In contrast for image segmentation, the learner receives all the images without label and he has to automatically detect the contours of objects in it. This is what is called unsupervised learning.

This thesis is centered around optimization methods for supervised learning. Nonetheless we will still consider extensions to unsupervised learning in Chapter 5. We note there are also other extensions (e.g., reinforcement learning) but they are outside the scope of this manuscript.

Supervised Machine Learning

Supervised machine learning aims at understanding the relationship between elements of an arbitrary input set X and elements of an arbitrary output set Y. Typically X = R d for a large d and Y is a finite set or a subset of R. For example:

-X is a set of images that contains a hand-written number and Y is the set of associated numbers. A picture is coded through the grey level of its pixels and X = [0, 1) d for d pixels and Y = {0, . . . , 9}.

-X summarizes the air pollution, the crime rate, the high-school quality, the percent of green spaces around certain neighborhoods and Y = R depicts the housing prices. Unfortunately an output y 2 Y cannot be always expressed exactly as a function of an input x 2 X since there may be some random noise or unknown factors. So, instead the couple (X, Y ) is modeled as random variables. Therefore we aim to predict the output Y associated to the input X where it is given that (X, Y ) is sampled from an unknown distribution D. We do this prediction through a predictor which is defined as a measurable function h : X ! Y. The set of all predictors is denoted by F(X , Y). A predictor is also called a hypothesis or an estimator. In order to measure the performance of a predictor we define a loss function `: Y ⇥ Y ! R where `(y, y 0

) is the loss incurred when the true output is y whereas y 0 is predicted. For instance: Classification: Y is a finite set and `(y, y 0 ) = y6 =y 0 is the 0 1 loss. In binary classification this would be Y = {0, 1}.

Least-squares regression: X = R and `(y, y 0 ) = |y y 0 | 2 . Least-squares regression is the main topic of this thesis and one of the most classical problems of statistical learning. It will be used to illustrate numerous examples throughout this introduction. In particular we shall be considering the parametric leastsquares framework: we shall assume a linear parameterization of the predictor, h(x) = h✓, (x)i where the features (x) 2 R d are designed by experts using their knowledge of the phenomenon, or are independently learned, e.g., with neural networks [START_REF] Bengio | Representation learning: A review and new perspectives[END_REF]. These features have a key importance in practice because linear predictors with relevant features may be more efficient than non-linear predictors. Furthermore we note that a linear parametrization in ✓ does not imply a linear parametrization in x since may be non-linear. The quality of a prediction is measured by the generalization error (also called risk ) of a predictor defined by

L(h) = E (X,Y )⇠D `(Y, h(X)
). This is the averaged loss incurred when the learner predicts Y by h(X) and the data (X, Y ) are sampled following D. Thus the learner wants to solve the minimization problem min h2F (X ,Y)

L(h).

The solution of this problem (when it exists) is called the Bayes predictor. For least-squares regression the generalization error of a predictor h can be decomposed as

L(h) = 1 2 E[Y E(Y |X)] 2 + 1 2 E[E[Y |X] h(X)] 2 .
In other words E[Y |X] is the Euclidean projection of Y onto the set L 2 (X) of the square integrable functions of X. Consequently L(h) is always larger than 2 and the Bayes predictor is (in this particular case)

1 2 E[Y E[Y |X]]
h ⇤ (x) = E[Y |X = x].
Regrettably since the distribution D is unknown, this function is not computable. Therefore we assume that the learner is observing a finite training set of points S n = ((X 1 , Y 1 ), . . . , (X n , Y n )) which are sampled independently from the unknown law D and wants to use them to predict the output Y associated to the input X, where (X, Y ) is sampled from D independently from S n . This task is formalized through a learning algorithm: a function A : [ n 1 (X ⇥ Y) n ! F(X ⇥ Y) which relates a training set S n to a predictor A(S n ). Since the training set S n is random, the generalization error of the predictor A(S n ) is a random variable and the quality of a learning algorithm A is measured by E Sn L(A(S n )), where the expectation is measured with respect to the distribution of S n . Alternatively L(A(S n )) may be controlled in probability.

Let us stress that we adopt the distribution-free approach of Vapnik and Chervonenkis [1971,1974]; see, e.g., [START_REF] Devroye | A Probabilistic Theory of Pattern Recognition[END_REF]. There is no assumptions on the distribution D and the method has to be efficient independently of it. In contrast classical statistics first assumes a particular statistical model for the distribution D and then estimate the parameters of this model. This approach will be temporarily taken in Chapter 6. It is also different from the probably approximately correct (PAC) learning framework introduced by [START_REF] Valiant | A theory of the learnable[END_REF] but the comparison is outside the scope of this introduction. Interested readers can see the monograph by [START_REF] Shalev-Shwartz | Understanding Machine Learning: From Theory to Algorithms[END_REF] for more precisions on these frameworks.

As the distribution D is unknown the Bayes predictor cannot be directly computed and the generalization error cannot even be minimized. Instead the training set S n is used to approximate these objects. There are two principal ways to address this problem:

Local averaging methods: They estimate the Bayes predictor E[Y |X = x] by averaging the Y i 's corresponding to the X i 's close to x. This includes, for example, the Nadaraya-Watson estimator [START_REF] Nadaraya | On estimating regression[END_REF][START_REF] Watson | Smooth regression analysis[END_REF] or the k-nearest neighbors algorithm [START_REF] Cover | Nearest neighbor pattern classification[END_REF]. These methods are well studied and efficient for dimensions of small to middle scale compared to n [see, e.g., [START_REF] Hastie | The Elements of Statistical Learning[END_REF]. Empirical Risk Minimization (ERM): It approximates the generalization error by the training error (also called empirical risk ) L defined by the average error over the sample S n

L(g) = 1 n n X i=1 `(Y i , h(Y i )),
and considers its minimizer [see, e.g., [START_REF] Vapnik | Statistical Learning Theory. Adaptive and Learning Systems for Signal Processing, Communications, and Control[END_REF]]. This method raises two principal issues: Statistical problem: How well does the minimizer of the empirical risk perform? This question will be studied in Section 1. 4. Optimization problem: How to minimize the empirical risk L? This question will be examined in Section 1.5. We will succinctly address these two questions through the lens of the least-squares regression. First of all we will introduce the mathematical framework adopted in this thesis.

Mathematical Framework

We shall now consider the Euclidean space R d of dimension d 2 N ⇤ endowed with the natural inner product h•, •i and the Euclidean norm k • k 2 . The central concept in this thesis is convexity:

Definition 1. A set C 2 R d is said to be convex if for all ✓ 1 , ✓ 2 2
C and all t 2 (0, 1),

(1 t)✓ 1 + t✓ 2 2 C. While classical functional analysis does not attach the same importance to convex functions as to convex sets [START_REF] Clarke | Functional Analysis, Calculus of Variations and Optimal Control[END_REF], the former is a key concept for optimization theory:

Definition 2. A extended-valued function f : R d ! R [ {+1} is said to be convex if for all ✓ 1 , ✓ 2 2 R d and all t 2 (0, 1), f (1 t)✓ 1 + t✓ 2 )  (1 t)f (✓ 1 ) + tf (✓ 2 )
. When f is twice differentiable this condition is equivalent to a symmetric positivedefinite Hessian. It is worth noting that these concepts of convexity are independent of the notions of norm or even distance. Indeed convex sets and convex functions only need the basic operations of a vector space to be defined. The class of convex functions is very general and this thesis is restricted, for convenience, to the convex functions which are closed (their epigraphs epi(f ) = {(✓, ↵) 2 R d ⇥ R : f (✓)  ↵} are closed sets) and proper (not identically +1). We refer to the monographes by [START_REF] Rockafellar | Convex Analysis[END_REF], Hiriart-Urruty and [START_REF] Hiriart-Urruty | Fundamentals of Convex Analysis[END_REF] for more details on convex analysis.

Conceptually, convexity enables to turn a local information about the function into a global one. For example a local minimum is automatically a global minimum or even more importantly, the gradient rf (✓) at a point ✓ 2 R d provides a global linear lower-bound on the function f . Functions met in machine learning applications often have additional properties. A continuously differentiable function f is said to be L-smooth for L 2 R + when the gradient of f is L-Lipschitz. When f is twice differentiable, this is equivalent to a global upper-bound on the Hessian of f . Thus this assumption provides a quadratic upper bound on the function value. For example, the empirical risk

L(h) = 1 n P n i=1 `(Y i , h(X i )
) is usually smooth when the loss function `is smooth and the data X i are almost surely (abbreviated to from now on as a.s.) bounded.

On the other side a function f is µ-strongly convex for µ 2 R

+ if f µ 2 k • k 2 2 is convex.
When f is twice differentiable this is equivalent to a global lower-bound on the Hessian of f and then this assumption provides a quadratic lower-bound on the function value. In this sense, µ measures the curvature of the function f . strongly convex and smooth functions have several interesting properties we will not review here [see, e.g., [START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF]. Moreover these two assumptions are connected: if f is µ-strongly convex then its Fenchel conjugate f ⇤ is 1/µ-smooth [Hiriart- Urruty and Lemaréchal, 2001, Theorem 4.2.1]. On the other hand, contrary to convexity, both smoothness and strong convexity depend on the norm considered and [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF] recently relaxed this dependency.

Analysis of Empirical Risk Minimization

Since the data distribution D is unknown, the generalization error is approximated by the training error L(h) = 1 n P n i=1 `(Y i , h(Y i )) which may be optimized to obtain a predictor h. Firstly we notice that the predictor h(x) = y i x i (x) has a training error L(h) = 0 whereas its generalization error may be arbitrarily large. This is the overfitting phenomenon which appears when the prediction fits the data too closely and does not try to generalize enough. To prevent this effect, one may either adopt: Constraint formulation: The hypothesis class is restricted to a smaller subset G ⇢ F. The empirical risk minimization for this class is the learning algorithm defined by ĥ = arg min h2G L(h).

A bias may be created by restricting too much the class of predictor and the sub-class has to be fixed in advance (without observing the data). Penalized formulation: A penalization (h) is added to the empirical risk L(h) and one solves the problem min h2F L(h) + (h) for 2 R + .

The penalization (h) controls the complexity of the predictor h by implicitly inducing a tradeoff between predictors with a small training error but a large value of (h) and predictors with larger training error but smaller penalization (h). The penalization also induces a bias since the predictor would be the minimizer of the penalized problem which is different from the initial non-penalized problem. Even if the penalized and constraint formulations are equivalent by convex duality [Borwein and Lewis, 2000, Sec. 4.3], the penalized formulation is easier to use in practice from an algorithmic point of view since (a) unconstrained optimization is easier, (b) there are efficient ways to set the value . On the other hand the constraint formulation is more appropriate for the theoretical analysis [Bach et al., 2012, p.7].

The generalization error of the ERM predictor ĥ can be decomposed as follows:

L( ĥ) L(h ⇤ ) = L( ĥ) min Approximation error: It measures how much is lost from restricting the set of predictors. It does not depend on the sample size. Thus there is a so-called bias-variance tradeoff : choosing a large class results in a small approximation error and a large estimation error since it leads to overfitting. We note that choosing a small class with dedicated structure may help in terms of computational complexity. It is also worth noting that only the estimation error depends on the predictor ĥ. We denote by h = arg min h2G L(h). The estimation error is uniformly bounded by L( ĥ) L( h) = L( ĥ) L( ĥ) + L( ĥ) L( h)

| {z } 0 + L( h) L( h)  2 sup g2G |L(h) L(h)|.
Under Lipschitz assumptions on the loss, this supremum is typically of order O(1/ p n) [START_REF] Boucheron | Concentration Inequalities[END_REF]. When the loss function is, in addition, µ-strongly convex, [START_REF] Sridharan | Fast rates for regularized objectives[END_REF], [START_REF] Boucheron | A high-dimensional Wilks phenomenon[END_REF] show that the estimation error is of order O(1/µn) by directly bounding L( ĥ) L( h) without using a uniform bound as above.

We only consider here linear predictors, i.e., linearly parameterized by ✓ 2 R d as h(x) = h✓, (x)i for features (x) 2 R d . The error of a parameter ✓ is directly defined as L(✓) = L(h✓, (•)i) by a slight abuse of notation. The generalization error is assumed to be minimized among all linear predictors and we denote by ✓ ⇤ one of its minimizer1 : ✓ ⇤ 2 arg min

✓2R d L(✓).
It is important to emphasize that the Bayes predictor h ⇤ is never assumed to be linear as in the classical regression analysis (which assumes a well-specified linear model Y = h✓ ⇤ , Xi + " for some independent zero-mean noise ").

Parametric least-squares regression. For linear least-squares regression this bias-variance decomposition takes the form

L(✓) L(h ⇤ ) = 1 2 E[E[Y |X] h✓ ⇤ , (x)i] 2 + 1 2 h✓ ⇤ ✓, E[ (x) ⌦ (x)](✓ ⇤ ✓)i = 1 2 E[E[Y |X] h✓ ⇤ , (x)i] 2 + 1 2 k✓ ⇤ ✓k 2 ⌃ ,
where the covariance matrix of the features is denoted by ⌃ = E[ (x) ⌦ (x)]. By setting the gradient of the generalization error to 0, the optimum ✓ ⇤ satisfies the normal equation

⌃✓ ⇤ = E[ (X)Y ]
. This implies, when ⌃ is full-rank, that ✓ ⇤ = ⌃ 1 E[ (X)Y ]. We will consider the ordinary least-squares estimator ✓ which is the minimizer of the training error over linear predictors. Some authors also consider the ridge estimator [START_REF] Hoerl | Application of ridge analysis to regression problems[END_REF] which is the solution of the `2-regularized training error minimization min ✓2R d L(✓) + k✓k 2 2 . We will now study the estimation error in the linear least-squares framework. It is a random variable which depends on the training-set S n and its expectation can be decomposed as:

1 2 Ek ✓ ✓ ⇤ k 2 ⌃ = kE ✓ ✓ ⇤ k 2 ⌃ | {z } Estimation bias + EkE ✓ ✓k 2 ⌃ | {z } Estimation variance
.

Note that we only compare to the class of linear predictors. In particular, this predictor has a greater error than the Bayes predictor. This analysis becomes very simple when the input data X n are assumed to be deterministic: this is the fixed design framework.

Fixed design analysis. We consider here that the input observations (X 1 , . . . , X n ) are deterministic. To emphasize this aspect, we denote them by (x 1 , . . . , x n ). The only randomness is thus in the sampling of the Y i . We use the notation ⌃ =

1 n P n i=1 (x i )⌦ (x i ) = > and Ỹ = [Y 1 , . . . , Y n ] > 2 R n .
For the sake of clarity, the design matrix is assumed to be of rank d. This assumption may be relaxed using Moore-Penrose pseudo-inverse. Then

✓ ⇤ = ( > ) 1 > E[ Ỹ ] and ✓ = ( > ) 1 > Ỹ .
Therefore the estimation error is

L( ✓) L(✓ ⇤ ) = ⇧(E Ỹ Ỹ ) 2 ,
where the orthogonal projection matrix is denoted by ⇧ = ( > ) 1 > . And the expected estimation error is

EL( ✓) L(✓ ⇤ ) = tr(⇧ var Ỹ ). If Ỹ satisfies var Ỹ 4 2 I. Then L( ✓) L(✓ ⇤ )  2 d/n.
Equality is obtained when var(Y i ) = 2 and the full-rank assumption on the design matrix may be relaxed to provide a bound of the form 2 rank(X)/n. We also note that one obtains high-probability bounds using extra-assumptions on the distribution of the noise [START_REF] Van Der Vaart | Weak Convergence and Empirical Processes[END_REF].

This result provides an upper-bound on the estimation error of the least-squares estimator. However a different proof could improve upon this bound and a different estimator could also provide better performance. The minimax theory [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF] was developed as a way to handle at once all possible learning algorithms from a given class. This field which takes its roots in information theory [START_REF] Cover | Elements of Information Theory[END_REF] provides a systematic way to show uniform bounds on their performance, even though this may at first appear daunting. An estimator ✓ is minimax optimal and a positive sequence ( n ) n 0 is the minimax rate of estimation over the class ⇥ if there exist constants c, C > 0 such that

sup ✓2⇥ Ek ✓ ✓k 2 2  c n ,
and

lim inf n!1 inf ✓ sup ✓2⇥ 1 n Ek ✓ ✓k 2 2 C 0 ,
where the infimum is over all estimators (measurable functions of the data). In the fixed design setting, it is possible to show that 2 d/n is optimal over R d [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF].

Random design analysis. The input observations are not assumed deterministic anymore. We start with the classical asymptotic study which assumes d fixed and n going to 1. We denote by " = h✓ ⇤ , (X)i Y and ⇠ = " (X). By the law of large numbers > P ! ⌃ and

Y P ! E[ (X)Y ].
We also assume here ⌃ < µI for µ > 0.

Therefore ✓ P ! ✓ ⇤ by the continuous mapping theorem. The Delta method [van der Vaart, 1998, Chapter 3] may be applied to obtain asymptotic normality:

p n( ✓ ✓ ⇤ ) D ! N (0, ⌃ 1 E[⇠ ⌦ ⇠]⌃ 1
).

This matches the Cramer-Rao bound [START_REF] Casella | Statistical Inference. Statistics/Probability Series[END_REF] and therefore this estimator is asymptotically optimal. One proceeds in the same way to asymptotically bound the estimation error

n[L( ✓) L(✓ ⇤ )] D ! 1 2 tr W(⌃ 1/2 E[⇠ ⌦ ⇠]⌃ 1/2 , 1),
where W(⌃, k) is the Wishart distribution with variance matrix ⌃ and k degrees of freedom. Under the assumption E⇠ ⌦⇠ 4 2 ⌃, the asymptotic variance is bounded by

2 d
n as in the fixed design analysis. Using Taylor expansions, this asymptotic analysis can be extended to any loss functions.

The non-asymptotic study is more involved. It has been pursued by [START_REF] Györfi | A Distribution-Free Theory of Nonparametric Regression[END_REF], [START_REF] Audibert | Robust linear least squares regression[END_REF], [START_REF] Hsu | Random design analysis of ridge regression[END_REF], [START_REF] Lecué | Performance of empirical risk minimization in linear aggregation[END_REF], [START_REF] Oliveira | The lower tail of random quadratic forms with applications to ordinary least squares[END_REF]. [START_REF] Hsu | Random design analysis of ridge regression[END_REF] obtain results from fixed design analysis using that ⌃ 1/2 ⌃ 1 ⌃ 1/2 is concentrated around the identity. They consider ✓ = E[ ✓|X 1 , . . . , X n ], the conditional expectation of the least-squares estimator which is equal to

✓ = ⌃ 1 Ê[Xh ⇤ (Y )],
where the empirical expectation is denoted by Ê. We sketch below the main lines of their proof which holds on the decomposition

⌃ 1/2 ( ✓ ✓ ⇤ ) = ⌃ 1/2 ( ✓ ✓) + ⌃ 1/2 ( ✓ ✓ ⇤ ) = ⌃ 1/2 ⌃ 1/2 Ê[ ⌃ 1/2 X(Y h ⇤ (Y ))] +⌃ 1/2 ⌃ 1 ⌃ 1/2 Ê[⌃ 1/2 X(h ⇤ (X) X > ✓ ⇤ )].
They first observe that Ê[ ⌃ 1/2 X(Y h ⇤ (Y ))] is related to the fixed design excess loss and goes to E[Y h ⇤ (X)] = 0. The second term Ê[⌃ 1/2 X(h ⇤ (X) X > ✓ ⇤ )] corresponds to the approximation error and will be close to E[X(h ⇤ (X) X > ✓ ⇤ )] = 0 by optimality of ✓ ⇤ . Under sub-Gaussian assumptions on the noise ", [START_REF] Hsu | Random design analysis of ridge regression[END_REF] show that L( ✓) L(✓ ⇤ ) = O(x 2 d/n) probability with exponentially close to 1 (at least 1 exp( cdx)). Similar results were obtained by [START_REF] Lecué | Learning subgaussian classes: Upper and minimax bounds[END_REF].

Under much weaker assumptions on the noise, [START_REF] Lecué | Performance of empirical risk minimization in linear aggregation[END_REF] provide a weak polynomial probability estimate:

Theorem 1. There exist absolute constants c 0 , c 1 , c 2 for which the following holds. Assume there exists  for which Eh✓, (X)i 4  (Eh✓, (X)i 2

) 2 8✓ 2 R d , and = (E(" 4

)) 1/4 < 1. Let n (c 0 ) 2 d. Then, for every > 0 with probability

1 exp( n/c 1  2 ) , L( ✓) L(✓ ⇤ )  c 2  6 2 d n .
Oliveira [2016] obtains stronger polynomial probability under stronger assumptions on the noise. Even so [START_REF] Lecué | Performance of empirical risk minimization in linear aggregation[END_REF] explain it is impossible to get an exponential probability estimate without stronger moment assumptions. [START_REF] Tsybakov | Optimal rates of aggregation[END_REF], [START_REF] Shamir | The sample complexity of learning linear predictors with the squared loss[END_REF] show this rate is also optimal in the random design setting. Besides, it is worth noting a lower bound in the fixed design framework does not directly imply a lower bound in the random design setting [Tsybakov, 2009, Sec 2.7.2].

To conclude, for the parametric least-squares regression, the prediction error is O( 2 d/n) under very light assumptions. Yet these sharp results are very challenging and their non asymptotic analysis is very recent. Even though (as will become clear later) this thesis is centered on techniques which avoid minimizing the empirical risk, we will now present some optimization methods which achieve that goal.

Complexity Results in Convex Optimization

This thesis is focused on convex optimization and mainly on the study of optimization algorithms for quadratic functions. Nowadays, optimization is indeed divided between convex problems (which can be efficiently solved) and non-convex problems (for which there are no efficient and generic methods).

Interestingly, this distinction was initially drawn between linear and non-linear problems. Linear programming is now essentially solved: (a) in practice since 1947 by Dantzig's simplex algorithm (with an exponential worst-case complexity); (b) in theory by [START_REF] Khachiyan | A polynomial algorithm in linear programming[END_REF] who achieved polynomial complexity using the ellipsoid method (with very slow convergence in practice); and (c) in both theory and practice since [START_REF] Karmakar | An algorithm for finding a circuit of even length in a directed graph[END_REF] proposed the first efficient polynomial-time algorithm using interior point methods.

This winding path from practical to theoretical progress has fed and inspired convex optimization since the seminal works of von [START_REF] Neumann | Discussion of a maximum problem. volume VI of Collected Works[END_REF] and those of [START_REF] Kuhn | Nonlinear programming[END_REF]. [START_REF] Nemirovski | Effective methods for the solution of convex programming problems of large dimensions[END_REF] first showed that all convex minimization problems with a first-order oracle can theoretically be solved in polynomial time using the ellipsoid method, but with slow convergence in practice. Then [START_REF] Nesterov | Interior-Point Polynomial Algorithms in Convex Programming[END_REF] extended the interior point method to efficiently solve a wider class of convex problems.

However when the complexity of problems becomes higher, this generic solver becomes inefficient and older first-order methods are preferred at the cost of precision to the advantage of many cheap iterations. Unfortunately there is no unified analysis for these large-scale problems and the algorithmic choices are constrained by the problem structure. This has prompted both theoreticians and practitioners to move away from black-box optimization and leverage the specifics of the problem being considered to design better adapted algorithms.

Therefore large-scale optimization contributes to blur the frontier between convexity and non-convexity by defining new common territories, i.e., algorithms which can efficiently solve different problems indifferently of their convex aspects [START_REF] Ge | Matrix completion has no spurious local minimum[END_REF][START_REF] Jin | Provable efficient online matrix completion via non-convex stochastic gradient descent[END_REF].

Black Box Optimization and Lyapunov Analysis

We consider the general optimization problem:

min ✓2C f (✓),
where C is a convex set and f is a convex function. In order to conceptualize the optimization task, we adopt the black box framework defined by [START_REF] Nemirovski | Effective methods for the solution of convex programming problems of large dimensions[END_REF]. The convex set C is known to the algorithm but not the objective function f . The only assumption made on f is that it belongs to some class of functions F. The information about f is obtained through interacting with an oracle. When queried at a given point ✓, the first-order oracle we consider here, answers by giving the function value f (✓) and the gradient rf (✓). Therefore a first-order black box procedure is a sequence of mappings ( n ) n 0 where n :

C n ⇥ (R d ) n ⇥ R n ! C.
The algorithm starts from ✓ 0 = 0 and then iterates: ✓ n = n 1 ✓ 0 , . . . , ✓ n 1 , rf (✓ 0 ), . . . , rf (✓ n 1 ), f(✓ 0 ), . . . , f(✓ n 1 ) . Our aim is to determine the oracle complexity of the problem. This is the number of oracle queries necessary to solve the problem at a precision of ", i.e., to find an admissible point ✓ 2 C such that f (✓) min ✓2C f ( ✓)  ". To that purpose we should:

(a) find an algorithm whose convergence rate matches the desired rate, this would imply an upper bound on the complexity; (b) provide a lower-bound on the complexity by showing that no admissible method can solve the problem faster. This is usually done by finding a particular function for which it is possible to bound from below the performance of any method. We note that oracle complexity does not directly provide information about the computational complexity of the method since the oracle query and the mapping n may be computationally demanding. Therefore we will also pay attention to the computational complexity of each method. This explains why we often restrict ourselves to algorithms with finite-order iteration of the form:

✓ n = n 1 (✓ n k , . . . , ✓ n 1 , rf (✓ n k ), . . . , rf (✓ n 1 ), f(✓ n k ) . . . , f(✓ n 1 )).
They can be reformulated (after a change of variable ⇥ n = (✓ n 1 , . . . , ✓ n )) as iterative processes of the form

⇥ n = F n 1 (⇥ n 1 ).
To prove the convergence of these dynamical systems, it is common to rely on Lyapunov theory. It goes back to [START_REF] Lyapunov | The general problem of the stability of motion[END_REF] who showed that all the trajectories of the ordinary differential equation (ODE)

ẋ(t) = Ax(t)
goes to zero (the ODE is stable) if and only if there exists a symmetric positive-definite matrix P such that A > P + P A 4 0. This is the origin of Lyapunov's first method which is based on the equivalence for the linear iterative process

✓ n = A✓ n 1 between lim n!1
✓ n = 0 [START_REF] Oldenburger | Infinite powers of matrices and characteristic roots[END_REF] () ⇢(A) < 1 [START_REF] Stein | Some general theorems on iterants[END_REF] () 9P < 0; A > P A P 4 0.

The matrix P solution of the Lyapunov inequality A > P A P 4 0 can be found analytically using convex optimization [START_REF] Boyd | Linear Matrix Inequalities in System and Control Theory[END_REF] and ensures the convergence of the iterative process. Lyapunov theory has been frequently applied to control theory with major works from [START_REF] Lur | On the theory of stability of control systems[END_REF], [START_REF] Popov | Absolute stability of nonlinear systems of automatic control[END_REF], [START_REF] Kalman | Lyapunov functions for the problem of Lur'e in automatic control[END_REF], [START_REF] Yakubovich | The S-procedure in nonlinear control theory[END_REF]. This method can be extended to non-linear processes, by linearizing the iterative procedure [START_REF] Perron | Über stabilität und asymptotisches verhalten der integrale von differentialgleichungssystemen[END_REF][START_REF] Ostrowski | Solution of Equations and Systems of Equations[END_REF]. It can also be applied directly to optimization [START_REF] Polyak | Introduction to Optimization[END_REF] or by using more complex control theory tools [START_REF] Lessard | Analysis and design of optimization algorithms via integral quadratic constraints[END_REF].

Most stability results for difference equations [START_REF] Ortega | Iterative Solution of Nonlinear Equations in Several Variables[END_REF]] have been obtained as discrete analogues of corresponding results for differential equations [see, e.g., LaSalle and Lefschetz, 1961]. Lyapunov's second method [Hahn, 1958, Kalman and[START_REF] Kalman | Control system analysis and design via the "second method" of Lyapunov. II. Discrete-time systems[END_REF] is the most common and general method to prove con-

✓ n ✓ n+1 n rf (✓ n ) Figure 1-1 -Gradient descent.
vergence of iterative processes. Its idea is to introduce a nonnegative function V (the so-called Lyapunov function) which will decrease along the sequence of iterates, i.e., V (✓ n+1 )  V (✓ n ) and therefore ensure the convergence of the method [see Polyak, 1987, Sec 2.2, for more details]. Therefore finding such a function V may prove the convergence of the iterative system and converse results ensure existence of a Lyapunov function for a stable iterative system [START_REF] Halanay | Quelques questions de la théorie de la stabilité pour les systèmes aux différences finies[END_REF][START_REF] Driver | Note on a paper of Halanay on stability for finite difference equations[END_REF]. Unfortunately there is no systematic way to find Lyapunov functions in practice. There are some classical solutions such as f (✓) f (✓ ⇤ ) or k✓ ✓ ⇤ k 2 , but these do not always work. In general, theoreticians have to use their experience to design them.

Smooth Optimization

In this section we present a panel of methods for optimizing a smooth function. We do not aim at being comprehensive and we refer to [START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF], [START_REF] Bubeck | Convex optimization: Algorithms and complexity[END_REF] for surveys. We first present the gradient method dating back to [START_REF] Cauchy | Méthode générale pour la résolution des systemes d'équations simultanées[END_REF] which will be the starting point for more refined first-order techniques.

Gradient descent. With a first-order oracle, the simplest minimization algorithm is to consider the Euler discretization of the continuous gradient flow ⌘ = rf (⌘). It starts from a vector ✓ 0 , and iterates

✓ n = ✓ n 1 n rf (✓ n 1 ),
for a step-size sequence ( n ). The gradient step is illustrated in Figure 1-1. An L-smooth function f is quadratically upper-bounded as

f (✓)  f (✓ n 1 ) + hrf (✓ n 1 ), ✓ ✓ n 1 i + L 2 k✓ ✓ n 1 k 2 ,
and therefore the gradient update is the minimizer of this first-order approximation of f

✓ n = arg min ✓2R d n hrf (✓ n 1 ), ✓i + 1 2 n k✓ ✓ n 1 k 2 o . (1.1)
This step is also the steepest descent direction since arg min kdk1 hrf (✓ n 1 ), di = rf (✓ n 1 ) krf (✓ n 1 )k .

We now present the first convergence result.

Proposition 1. Let f be a L-smooth convex function. Then gradient descent with

n = 1/L satisfies f (✓ n ) f (✓ ⇤ )  Lk✓ 0 ✓ ⇤ k 2 n .
Moreover, if f is µ-strongly convex,

f (✓ n ) f (✓ ⇤ )  (1 µ/L) n [f (✓ 0 ) f (✓ ⇤ )].
Convergence rates for gradient descent were first studied for quadratic functions by [START_REF] Kantorovitch | On an effective method of solving extremal problems for quadratic functionals[END_REF] and for convex functions by [START_REF] Vainberg | On the convergence of the method of steepest descents for non-linear equations[END_REF], [START_REF] Goldstein | Cauchy's method of minimization[END_REF], [START_REF] Polyak | Gradient methods for minimizing functionals[END_REF], [START_REF] Levitin | Minimization methods in the presence of constraints[END_REF]. Proofs of convergence date back to seminal works of [START_REF] Temple | The general theory of relaxation methods applied to linear systems[END_REF], [START_REF] Curry | The method of steepest descent for non-linear minimization problems[END_REF], [START_REF] Crockett | Gradient methods of maximization[END_REF]. We sketch the proof for quadratic convex functions since it will be used as a stepping stone when proving more general results later in the manuscript. We denote f (✓) = 1 2 h✓, ⌃✓i hb, ✓i, which attains its global optimum at ✓ ⇤ = ⌃ 1 b. The gradient descent iteration is explicitly

✓ n ✓ ⇤ = ⇣ I 1 L ⌃ ⌘ (✓ n 1 ✓ ⇤ ) = ⇣ I 1 L , ⌃ ⌘ n (✓ 0 ✓ ⇤ )
and therefore

f (✓ n ) f (✓ ⇤ ) = 1 2 ⌃ 1/2 ⇣ I 1 L ⌃ ⌘ n (✓ 0 ✓ ⇤ ) 2  (1 µ/L) 2n [f (✓ 0 ) f (✓ ⇤ )],
which concludes the proof in the strongly convex case. Otherwise when µ = 0, we use the inequality ⌃ I 1 L ⌃ 2n 4 L/nI. The gradient descent is adaptive to the problem difficulty. Indeed the step-size does not depend on the value of µ. On the other hand the result is proven for constant step-size, but line-search techniques are also possible [Boyd and Vandenberghe, 2004, Sec 9.3].

Lower-bounds on the convergence rates. After having shown that gradient descent algorithms yield convergence rate O( 1 n ) for L-smooth problems and O((1 µ/L) n

) when the function is also µ-strongly convex, it is natural to wonder whether these rates are optimal or if other algorithms exist with better convergence rates. This question was first answered positively by [START_REF] Nemirovski | Effective methods for the solution of convex programming problems of large dimensions[END_REF] and later presented, in a simplified form, by [START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF]. Interested readers can also see the recent work of [START_REF] Arjevani | On the iteration complexity of oblivious first-order optimization algorithms[END_REF] on this topic.

Proposition 2. Let n  (d 1)/2, ✓ 0 2 R d and L > 0. There exists a L-smooth convex function f with a global minimizer ✓ ⇤ such that for any first-order method.

f (✓ n ) f (✓ ⇤ ) 3 32 Lk✓ ⇤ ✓ 0 k 2 (n + 1) 2 .
The number of iterations can not be too large compared to the dimension of the problem. Note this restriction is necessary since the ellipsoid method [Ben-Tal and Nemirovski, 2001, Sec. 5.2 ] converges at exponential rate for n larger than d. Moreover this lower-bound is still informative about the first steps, when n is small compared to d. Surprisingly, the worst-case function designed to prove this result is quadratic. In this sense, quadratic functions are not easier to optimize than smooth functions. Regarding the strongly convex case, [START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF]

⌘

for the oracle complexity. Clearly, there is a gap between the lower bounds and the performance of gradient descent in Proposition 1. But the lower bounds are not to blame as it turns out they are matched by a slightly more involved algorithm, which we describe now. Accelerated gradient descent. Taking inspiration from conjugate gradient, Nemirovsky and Yudin [1983] proposed the first method with optimal convergence rates. However this method needed some kind of line-search and was not practical. [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF] presented an optimal algorithm which only needs a first-order oracle. Nesterov's accelerated gradient can be described as follows: start from ✓ 0 = ⌘ 0 2 R d and then iterate

✓ n = ⌘ n 1 1 L rf (⌘ n 1 ) ⌘ n = ✓ n + n (✓ n ✓ n 1 ),
where the momentum coefficient n 2 R is chosen to accelerate the convergence rate and has its roots in the heavy-ball algorithm from [START_REF] Polyak | Some methods of speeding up the convergence of iteration methods[END_REF]. Figure 1-2 illustrates the difference between gradient and accelerated gradient steps. For this algorithm one obtains the following result.

Proposition 3. Let f be a L-smooth convex function. Then for n = n 1 n 2 , f (✓ n ) f (✓ ⇤ )  2Lk✓ 0 ✓ ⇤ k 2 (n + 1) 2 . Moreover let f be also µ strongly convex. Then for n = p L p µ p L+ p µ , f (✓ n ) f (✓ ⇤ )  Lk✓ 0 ✓ ⇤ k 2 ⇣ 1 r µ L ⌘ n .
The algorithm was initially proposed by [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF][START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF] with a more complex momentum term. The simplified version we present here is due to [START_REF] Tseng | On accelerated proximal gradient methods for convex-concave optimization[END_REF]. [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], [START_REF] Schmidt | Convergence rates of inexact proximal-gradient methods for convex optimization[END_REF] both proposed very concise proofs of these optimal convergence rates. It is worth noting that the accelerated gradient method is not adaptive to strong convexity. Indeed the value of µ is needed to obtain the linear convergence rate. [START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF], O'Donoghue and Candès [2013] proposed the use of restart heuristics to resolve this issue. Indeed [START_REF] Arjevani | On the iteration complexity of oblivious first-order optimization algorithms[END_REF] show that no methods with fixed sequences of step-sizes can achieve the accelerated rate, unless the parameter µ is known. This method is not a descent algorithm and often exhibits an oscillatory behavior. This will be further detailed in Chapter 2. Until recently, accelerated gradient descend lacked an intuitive interpretation, contrary to the standard gradient descent. This has been addressed by [START_REF] Allen-Zhu | Linear coupling: An ultimate unification of gradient and mirror descent[END_REF], [START_REF] Bubeck | A geometric alternative to nesterov's accelerated gradient descent[END_REF], and in a series of works on the connection with differential equations by [START_REF] Su | A differential equation for modeling Nesterov's accelerated gradient method: theory and insights[END_REF], [START_REF] Krichene | Accelerated mirror descent in continuous and discrete time[END_REF], [START_REF] Wibisono | A variational perspective on accelerated methods in optimization[END_REF], [START_REF] Wilson | A Lyapunov analysis of momentum methods in optimization[END_REF], Attouch et al. [2016a], Attouch and Peypouquet [2016].

Specificity of Quadratic Functions

Minimizing a quadratic function f is equivalent to solving a linear system of equations, i.e., rf (✓) = 0. For instance, in least-squares regression, the ERM predictor is directly given by the normal equation

> ✓ = Ỹ , (1.2)
which is a set of d equations for d unknowns. Hence, the methods we presented so far may appear inefficient or needlessly complicated for solving such problems. Nevertheless the worst case function used to prove the lower bound in Proposition 2 is quadratic and accordingly quadratic problems are just as hard as any smooth problem. We introduce presently two methods dedicated to minimizing quadratic functions.

Numerical algebra. First one may prefer to directly solve the system in Eq. (1.2) by using numerical linear algebra algorithms. The complexities of these methods can usually be decomposed as

O(d 3 + nd 2 ) to form > , O(nd 2
) to form Ỹ and O(d 3 ) to solve the linear system. Gaussian elimination could be used to solve the linear system in Eq. (1.2) as any linear system but this approach is oblivious to the specific form of the empirical covariance > . One may thus prefer methods based on singular value decomposition or QR decomposition when is full rank [see, e.g., [START_REF] Golub | Matrix Computations[END_REF]. These methods rely on well-known linear algebra, but can only be used for small to medium scale problems (typically d  10 4 ). For large scale problems they are not suitable since they do not leverage the special structure of the problem and the conjugate gradient algorithm may be favored instead.

Conjugate gradient algorithm. [START_REF] Hestenes | Methods of conjugate gradients for solving linear systems[END_REF] introduced the conjugate gradient method which corresponds to the momentum algorithm

✓ n = ✓ n 1 n rf (✓ n 1 ) + n (✓ n 1 ✓ n 2 )
with ( n , n ) solutions of the two dimensional optimization problem:

min , f ✓ n 1 rf (✓ n 1 ) + (✓ n 1 ✓ n 2 ) .
This line search procedure can be explicitly solved when f is a quadratic function with the same computational complexity as a gradient computation. The explicit method is detailed by [START_REF] Golub | Matrix Computations[END_REF]. Surprisingly this method converges to the solution ✓ ⇤ of the problem in at most d steps and the behavior of the method is globally controlled by [START_REF] Polyak | Introduction to Optimization[END_REF] f

(✓ n ) f (✓ ⇤ )  min n Lk✓ 0 ✓ ⇤ k 2 2 8n 2 , 4 exp ( 2 p µ/Ln)[f (✓ 0 ) f (✓ ⇤ )]
o .

We note that the method converges at the optimal linear rate without knowing the strong convexity constant µ, in contrast to the accelerated gradient method. Nonlinear extensions of the conjugate gradient are presented by [START_REF] Nocedal | Numerical Optimization[END_REF].

Extension to Composite Optimization

When the function f is no longer smooth, gradient descent with constant step size fails be to consistent [START_REF] Bertsekas | Nonlinear Programming[END_REF]. [START_REF] Shor | An application of the method of gradient descent to the solution of the network transportation problem. Notes, Scientific seminar on theory and application of cybernetics and operations research[END_REF], [START_REF] Ermoliev | Methods of solution of nonlinear extremal problems[END_REF], [START_REF] Polyak | A general method for solving extremal problems[END_REF] solve this issue by showing that the projected sub-gradient method with decreasing step size O(1/ p n) converges at rate O(1/ p n) when the average of the iterates is considered as output of the method [START_REF] Shor | Minimization Methods for Non-Differentiable Functions[END_REF]. When the function is also µstrongly convex, the projected sub-gradient method with step-size O(1/µn) converges at rate O(log(n)/µn) and at rate O(1/µn) when non-uniform average is used. Moreover [START_REF] Nemirovsky | Problem Complexity and Method Efficiency in Optimization[END_REF] show these rates are optimal among first-order methods.

C

✓ n ✓ n+1 ⇧ C n rf (✓ n ) Figure 1-3 -Projected subgradient. We denote by ⇧ C the Euclidean projection on the convex set C.
Yet there is a crucial difference between the class of smooth problems (with bounded smoothness constant) and the class of all convex problems since convergence rates are downgraded from O(1/n 2

) to O(1/ p n). However the structure of the function as well as the reason for non-smoothness are often known. For instance the objective function is sometimes the sum of a smooth and a non-smooth function, as in Lasso or constrained problems. These are composite problems of the form:

min ✓2R d f (✓) + g(✓),
where f is an L-smooth function accessible through a first-order oracle and g is a simple convex function in a sense that will be explained below.

As with the gradient descent for smooth functions, at each iteration the smooth function f is linearized around the current iterate ✓ n . Then proximal gradient methods, also called forward-backward splitting methods [see, e.g., [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Wright | Sparse reconstruction by separable approximation[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF] consider the iterate

✓ n+1 = arg min ✓2R d ⇢ f (✓ n ) + hrf (✓ n ), ✓ ✓ n i + L 2 k✓ ✓ n k 2 2 | {z }
Linearization of the smooth component

+ g(✓) |{z}

Non-smooth term .

For instance when g = C is the indicator function of a convex set C, proximal gradient is tantamount to projected gradient as depicted in Figure 1-3. This update may be efficiently written in the terms of proximal operator defined by [START_REF] Moreau | Fonctions convexes duales et points proximaux dans un espace Hilbertien[END_REF] as Prox g (⌘) = arg min ✓2R d { 1 2 k✓ ⌘k 2 2 + g(✓)}:

✓ n+1 = arg min ✓2R d ⇢ 1 L g(✓) + 1 2 ✓ ✓ n 1 L rf (✓ n ) 2 2 = Prox 1 L g ⇣ ✓ n 1 L rf (✓ n ) ⌘ .
Therefore this method is only used in practice with simple functions g whose proximal operator is computable effectively. This is the case if there exists a closed form expression and several examples are provided by [START_REF] Bach | Optimization with sparsityinducing penalties[END_REF]. Surprisingly [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], [START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF] show convergence results similar to smooth optimization with the proofs following the same general guidelines.

Extension to Non-Euclidean Geometry

Until now, we have derived dimension-free convergence rates under Lipschitz assumptions on the function or on its gradient with respect to the Euclidean geometry. When these assumptions are with respect to different norms, the above convergence rates still apply but the dimension d of the space appears in the bound. For example, if the gradient of a function

f is bounded in the `1-norm by a constant B > 0, it is bounded in the `2-norm as krf (✓)k 2  B p d.
Thus algorithms which work with different geometries may yield convergence rates with better dependency on the dimension.

Let h : R d ! R be a differentiable convex function such that its gradient rh is a bijection of R d . The Bregman divergence associated with the function h is defined as

D h (✓ 1 , ✓ 2 ) = h(✓ 1 ) h(✓ 2 ) hrh(✓ 2 ), ✓ 1 ✓ 2 i, ✓ 1 , ✓ 2 2 R d .
It may be interpreted as a generalized squared distance which provides a new geometry standing in for the Euclidean one. Incidentally, the latter is recovered by considering

h(•) = 1 2 k • k 2 2 .
With the aim of solving the constrained problem min ✓2C f (✓) under the Bregman geometry, [START_REF] Nemirovski | Effective methods for the solution of convex programming problems of large dimensions[END_REF] introduce the mirror descent algorithm which may be defined by the update

✓ n = arg min ✓2C n hrf (✓ n 1 ), ✓i + D h (✓, ✓ n 1 ) , started from ✓ 0 2 C.
The Bregman divergence has replaced the Euclidean distance in the definition of the gradient descent iteration in Eq. (1.1). This contemporary proximal definition is due to [START_REF] Beck | Mirror descent and nonlinear projected subgradient methods for convex optimization[END_REF]. Mirror descent can be equivalently written as a greedy gradient update followed by a projection step

rh( n ) = rh(✓ n 1 ) n rf (✓ n 1 ) (greedy update) ✓ n = arg min ✓2C D h (✓, n ). (projection step)
Let us take a step backward and forget about the Euclidean structure. The gradient rf (✓ n 1 ) belongs to the dual space whereas the iterate ✓ n 1 lives in the primal space.

Thus the gradient iteration no longer makes sense. Instead [START_REF] Nemirovski | Effective methods for the solution of convex programming problems of large dimensions[END_REF] propose first to map the current iterate ✓ n 1 to the dual space with rh(✓ n 1 ), and perform there the gradient update rh(✓ n 1 ) n rf (✓ n 1 ). The resulting point is then mapped back in the primal space and projected with regards to the Bregman divergence in the convex set C. This is depicted in Figure 1234. Mirror descent has to be associated with its cousin dual averaging initially introduced by [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF] which considers a lazy gradient update started from

dual space dual space primal space primal space C C ✓ n ✓ n rh rh(✓ n ) n rf (✓ n ) rh( n+1 ) rh 1 ⇧ h C n+1 ✓ n+1 rh( n ) rh( n 1 ) rh( n+1 ) rh( n+1 ) n rf (✓ n ) rh 1 rh 1 ⇧ h C n+1 ⇧ h C n
✓ 0 , 0 2 C rh( n ) = rh( n 1 ) n rf (✓ n 1 ) (lazy update) ✓ n = arg min ✓2C D h (✓, n ).
(projection step)

Taking 0 such that rh( 0 ) = 0, dual averaging can also be written under the following simpler expression (still started from ✓ 0 2 C)

✓ n = arg min ✓2C n ⌦ n 1 X i=1 i+1 rf (✓ i ), ✓ ↵ + h(✓) o .
The greedy and lazy aspects of these methods as well as what differentiates them will be explained in detail in Chapter 4. As initially desired, mirror descent and dual averaging algorithms adapt to the geometry provided by the Bregman divergence. When the function f is assumed to be B-Lipschitz with respect to an arbitrary norm k • k and the mirror map h is µ-strongly convex with respect to the same norm, then these methods yield an optimal convergence O(BD/ p µn) where [START_REF] Nesterov | Smoothing technique and its applications in semidefinite optimization[END_REF], [START_REF] Krichene | Accelerated mirror descent in continuous and discrete time[END_REF] have accelerated these algorithms when the function is smooth with respect to some non-Euclidean geometry. [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF] have recently redefined the concept of smoothness for non-Euclidean geometry. Furthermore these methods have also been extended to the composite setting by [START_REF] Duchi | Composite objective mirror descent[END_REF], [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF].

D 2 = sup ✓ 1 ,✓ 2 2C {h(✓ 1 ) h(✓ 2 )}.

Convex

µ-Strongly convex B-Lipschitz Projected sub-gradient BD/ p n B 2 D/µn L-Smooth Accelerated gradient Lk✓ 0 ✓ ⇤ k 2 /n 2 (1 p L/µ) n k✓ 0 ✓ ⇤ k 2 Quadratic Conjugate gradient min{d, Lk✓ 0 ✓ ⇤ k 2 /n 2 } min{d, (1 p L/µ) n k✓ 0 ✓ ⇤ k 2 }
Table 1.1 -Convergence rates for deterministic optimization.

Conclusion

We summarize the different rates of convergence provided so far in Table 1.1. Our goal is to minimize the generalization error of a machine learning problem. But we have seen that the estimation error was O(1/ p n) in the Lipschitz case and O(1/n) in the strongly convex case. This is the key insight of [START_REF] Bottou | The tradeoffs of large scale learning[END_REF], [START_REF] Shalev-Shwartz | Svm optimization: Inverse dependence on training set size[END_REF] who explain there is no need to optimize below estimation error for machine learning applications. We present now a different approach which directly minimizes the generalization error.

Stochastic Approximation

Classical Stochastic Approximation

Stochastic approximation historically aims at finding a zero of a function h : R d ! R d which cannot be directly computed but is accessible through samples. For instance in the dosage of chemical products, experiments make it possible to sample the function h at certain points: the result of the experiment is the true value perturbed by some random noise h(✓) + ".

Traditional deterministic techniques may be used to solve the non-linear system h(✓) = 0. At each iteration, an estimate of the value of h(✓ n ) is built, for example, by approximating it by its means over the samples. This is the sample average approximation approach by [START_REF] Kleywegt | The sample average approximation method for stochastic discrete optimization[END_REF]; see, e.g., [START_REF] Shapiro | Lectures on Stochastic Programming: Modeling and Theory[END_REF] for a review. This technique is unstable and time-consuming because at each iteration enough samples need to be computed to properly approximate the function value. Instead [START_REF] Robbins | A stochastic approxiation method[END_REF] propose to use each sample once by considering the noisy fixed point iteration:

✓ n = ✓ n 1 n [h(✓ n ) + " n ],
where " n is typically assumed to be an i.i.d. sequence of zero-mean finite variance noise and ( n ) is a sequence of step-size. Convergence results are usually asymptotic and assume the existence of a well behaving Lyapunov function [START_REF] Polyak | Convergence and convergence rate of iterative stochastic algorithms[END_REF][START_REF] Duflo | Random Iterative Models[END_REF]. [START_REF] Ermoliev | The method of generalized stochastic gradients and stochastic quasi-Fejér sequences[END_REF], [START_REF] Robbins | A convergence theorem for non negative almost supermartingales and some applications[END_REF] show the almost-surely convergence of the iterates and [START_REF] Chung | On a stochastic approximation method[END_REF], [START_REF] Sacks | Asymptotic distribution of stochastic approximation procedures[END_REF], [START_REF] Fabian | On asymptotic normality in stochastic approximation[END_REF] show their asymptotic normality, both if the step-size satisfies the conditions

1 X i=1 i = 1 and 1 X i=1 2 i < 1.
These results directly extend to minimization of a function f by finding a zero of its gradient rf (we note that no convexity is assumed).

Therefore the goal of stochastic approximation can be equivalently defined as minimizing a function f given only unbiased estimates rf n+1 (✓ n ) of its gradient rf (✓ n ) at certain points ✓ n . Indeed in a lot of applications, the exact gradient of a function is not tractable because of errors in the measurements or in the Monte Carlo evaluation of expected values. Yet noisy unbiased estimates of the gradient are accessible and cheap. As noticed by [START_REF] Bottou | On-line learning for very large data sets[END_REF] this directly includes the usual machine learning situation of minimization of the generalization error

L(✓) = Ef n (✓) for f n (✓) = l(y n , h✓, x n i), observing estimate rf n (✓) = l 0 (y n , h✓, x n i)x n of the true gradient rL(✓) = Erf n (✓)
. But its applicability is much broader and goes far beyond convex optimization [Benveniste et al., 1990, Kushner and[START_REF] Kushner | Stochastic Approximation and Recursive Algorithms and Applications[END_REF].

Convex Stochastic Optimization

When we want to minimize a function which is only available through unbiased estimates of the function values or its gradients, stochastic gradient descent is the key algorithm. It takes the form

✓ n = ✓ n 1 n rf n (✓ n 1 ),
where n is a step-size sequence chosen depending on the situation. Here we consider the classical stochastic approximation framework [START_REF] Kushner | Stochastic Approximation and Recursive Algorithms and Applications[END_REF]Yin, 2003, Borkar, 2008]. That is, we let (F n ) n 0 be an increasing family of -fields such that for each ✓ 2 R d and for all n 1 the random variable rf n (✓) is square-integrable and [START_REF] Polyak | A new method of stochastic approximation type[END_REF], [START_REF] Ruppert | Efficient estimations from a slowly convergent Robbins-Monro process[END_REF] consider instead the average of the iterates ✓n = 1 n P n 1 i=1 ✓ i as the output of the algorithm. This is the averaged stochastic gradient descent.

F n - measurable with E[rf n (✓)|F n 1 ] = rf (✓).
In machine learning applications, when the objective function is the expectation of a loss function, the running-time complexity of the stochastic gradient is O(nd) after n iterations. This has to be compared with the O(n 2 d) complexity of gradient descent on the training error computed with n data (computing a gradient which is an average of n terms costs O(nd)).

The global minimax rates of convergence are known since [START_REF] Nemirovsky | Problem Complexity and Method Efficiency in Optimization[END_REF] for convex non-smooth problems and [START_REF] Tsypkin | Attainable accuracy of adaptation algorithms[END_REF], [START_REF] Nazin | Information inequalities in a problem of gradient stochastic optimization and optimal realizable algorithms[END_REF] for strongly convex non-smooth problems. For convex problems the minimax rate is O(1/ p n) and it is attained by averaged stochastic gradient descent with step-size n = C/ p n [START_REF] Zhang | Solving large scale linear prediction problems using stochastic gradient descent algorithms[END_REF]. When in addition, the function is assumed µ-strongly convex, the minimax rate becomes O(1/(µn)) and is still attained by averaged stochastic gradient but with a smaller step-size n = C/µn and a non-uniform average [START_REF] Rakhlin | Making gradient descent optimal for strongly convex stochastic optimization[END_REF][START_REF] Lacoste-Julien | A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method[END_REF]]. Yet the minimax rate is the same for nonsmooth functions as it is in deterministic optimization. Moreover the proofs in the two settings are very similar. The proof techniques are different for the lower-bounds and use tools from statistics and information theory. They are clearly presented and extended by [START_REF] Agarwal | Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization[END_REF], [START_REF] Raginsky | Information-based complexity, feedback and dynamics in convex programming[END_REF].

The stochastic gradient algorithm has been extended to the proximal case by [START_REF] Duchi | Efficient online and batch learning using forward backward splitting[END_REF], [START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF], [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] and to the non-Euclidean setting by [START_REF] Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF], [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF].

Smooth and Strongly Convex Stochastic Optimization

We have seen that smoothness plays a central role in the context of deterministic optimization but for stochastic optimization, smoothness only leads to improvements on constants, not on the rate itself. The minimax rate remains O(1/ p n) for nonstrongly convex problems. Moreover accelerated gradient descent is notorious for not being robust to random or deterministic noise in the gradient [START_REF]Smooth optimization with approximate gradient[END_REF][START_REF] Schmidt | Convergence rates of inexact proximal-gradient methods for convex optimization[END_REF][START_REF] Devolder | First-order methods of smooth convex optimization with inexact oracle[END_REF] and [START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF], [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF]

advocate small step-size C/n 3/2 to obtain convergence rate O(1/ p n).
The story is different for strongly convex problems. When the function is nonsmooth, stochastic gradient descent is not adaptive to strong convexity since the step-size has to depend on µ in order to obtain fast rates. Moreover the choice of the constant in the step-size C/µn is very sensitive as noted by [START_REF] Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF].

Yet for smooth and strongly convex functions, [START_REF] Polyak | Acceleration of stochastic approximation by averaging[END_REF] show that averaged stochastic gradient with larger step size n = Cn ↵ for ↵ 2 (1/2, 1) has a convergence rate of order O(1/n) independent of the strong convexity constant. This idea is developed by [START_REF] Bach | Non-asymptotic analysis of stochastic approximation algorithms for machine learning[END_REF] who provide a non-asymptotic analysis of averaging for smooth functions. They show that averaged stochastic gradient with step-size n = C/n ↵ has a rate of convergence O(1/n + 1/(µn) 2

) for µstrongly convex problems and O(1/n 1 ↵

) for convex problems. Therefore this method with step-size n = C/ p n is adaptive to strong convexity with convergence rate O(min(1/µn), 1/ p n) for all smooth problems. [START_REF] Bach | Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression[END_REF] extends these results to logistic regression which is not globally strongly convex. We also note than the benefits of averaging may be understood through the lens of two-time scale stochastic approximation [START_REF] Borkar | Stochastic approximation with two time scales[END_REF].

Typical high-dimensional machine learning problems have very correlated variables so that the strong convexity constant µ may be arbitrarily small. This makes convergence rates O(1/(µn)) useless and makes non-strongly convex scenario more suitable. Thus we aim at obtaining algorithms with convergence rate O(1/n) that are robust to arbitrarily small strong convexity constants.

✓ 0 ✓ ⇤ ✓n ✓ n Figure 1-5 -Markov chain interpretation. 1.6.4 Least-Mean-Squares Algorithm When applied to least-squares f (✓) = 1 2 E[(y n h (x n ), ✓i) 2
], stochastic gradient descent is known as the least-mean-square algorithm [START_REF] Macchi | Adaptive Processing: the Least-Mean-Squares Approach with Applications in Transmission[END_REF]. In signal processing theory, it is usually studied (a) without averaging, (b) with decreasing step-size and (c) with strong convexity assumption ⌃ = E[ (X) ⌦ (X)] < µI. [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF], following [START_REF] Györfi | On the averaged stochastic approximation for linear regression[END_REF], propose a non-asymptotic analysis of the averaged least-mean-square algorithm with constant-step size

✓ n = ✓ n 1 (h (x n ), ✓ n 1 i y n ) (x n ). (1.3)
The sequence (✓ n ) is a homogeneous Markov chain [START_REF] Meyn | Markov Chains and Stochastic Stability[END_REF] which, under appropriate conditions, converges in distribution to its unique stationary distribution ⇡ . The average of the stationary distribution is denoted by ✓ = R ✓⇡ (d✓). Taking the expectation in Eq. (1.3) and remembering that

✓ ⇤ = ⌃ 1 E[ (x)y] one obtains E✓ n = E✓ n 1 ⌃(E✓ n 1 ✓ ⇤ ). Recall that lim n!1 E✓ n = ✓ and ⌃ is invertible. It follows that ✓ = ✓ ⇤ .
The ergodic theorem then shows that ✓n converges almost surely to ✓ . The central limit theorem for Markov chains also implies that nEk ✓n ✓ ⇤ k 2 ⌃ has a finite limit. Therefore, the Markov chain interpretation explains the averaged iterates converge to ✓ ⇤ pointwise, and that the rate of convergence of

E[f (✓ n ) f (✓ ⇤ )] is of order O(1/n), whereas ✓ n
does not converge to ✓ ⇤ but oscillates around it. This is summarized in Figure 12345.

In addition [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] show that under similar assumptions to that of the analysis of least-squares with random design and without assuming strong convexity,

Ef ( ✓n ) f (✓ ⇤ )  4R 2 k✓ 0 ✓ ⇤ k 2 n + 4 2 d n .
The performance of this algorithm is then decomposed as the sum of: (a) a bias term which depends on the deviation between the initial point of the algorithm and the solution of the problem and characterizes how fast initial conditions are forgotten and (b) a variance term, which depends on the covariance of the noise in the gradients and describes the effect of this noise.

We have already seen that the variance term O( 2 d/n) is optimal over all estimators. However the bias term O(1/n) is suboptimal since the optimal optimization algorithm forgets the initial condition as O(1/n 2

). Therefore there is a mismatch between the optimization and the statistics theories which will be investigated to a great extent in Chapter 3. This is relevant in practice since the bias term can be significantly larger than the variance term. It may consequently be useful to accelerate its convergence.

Finite Sum Problems

We end this section with an intermediate problem, related both to deterministic optimization and to stochastic approximation, which has gained a lot of attention these past years. We consider here a finite sum problem of the form:

f (✓) = 1 k k X i=1 f i (✓),
where f 1 , . . . , f k are L-smooth-functions, possibly µ-strongly convex. An important example is the empirical risk studied in Section 1.4.

The function f is deterministic, thus contrary to stochastic approximation we have a full access to its gradient. The function f can therefore be optimized with gradient descent:

✓ n = ✓ n 1 n 1 k k X i=1 rf i (✓ n 1 )
with a linear convergence rate O(exp( nµ/L)). However the complexity of computing rf (✓ n 1 ) is O(dk) at each iteration. The running time complexity is then O(dkL/µ log(1/")) and can be decreased to O(dk p L/µ log(1/")) using accelerated gradient descent.

Stochastic gradient descent can also be applied to f , sampling with replacement, at each iteration, i(n) 2 {1, . . . , n} (see recent work by [START_REF] Gürbüzbalaban | Why random reshuffling beats stochastic gradient descent[END_REF], [START_REF] Shamir | Without-replacement sampling for stochastic gradient methods[END_REF] for results on sampling without replacement) and iterating

✓ n = ✓ n 1 n 1 rf i(n) (✓ n 1 ).
The convergence rate O(L/(µn)) is slower but the running time complexity O(Ld/(µ")) is independent of k. So stochastic gradient descent achieves rapidly low accuracy whereas deterministic algorithm should be preferred when high accuracy is desired. [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF], Shalev-Shwartz and Zhang [2013] propose two new methods, respectively stochastic averaged gradient (SAG) and stochastic dual coordinate ascent (SDCA) which both combine linear convergence and O(d) complexity by iteration. These methods have a running time complexity O(d(k + L/µ) log(1/")). Initially, SDCA applied only to a restricted set of problems (minimization `2-regularized train-

Convex µ-Strongly convex B-Lipschitz Averaged gradient descent BD/ p n B 2 D/µn L-Smooth Averaged gradient descent O(1/ p n) O(1/µn) Least-squares Averaged gradient descent O( 2 d/n + R 2 k✓ ⇤ k 2 2 ) [Jain et al., 2016] Finite sum Accelerated SDCA, SVRG O ⇣ exp n k + p Lk n 2 ⌘ O ⇣ exp n k+ p Lk/µ ⌘ Table 1.2 -Convergence rates for stochastic approximation.
ing error with linear predictions) and required duality but these two assumptions have been recently lifted by Shalev-Shwartz [2016a]. [START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF] obtain the same running time complexity through a different method that can be understood in terms of variance reduction.. These methods do not contradict lower bounds seen in the previous section since they hold due to the special structure of the objective function.

Shalev-Shwartz and Zhang [2014], [START_REF] Zhang | Stochastic primal-dual coordinate method for regularized empirical risk minimization[END_REF], [START_REF] Nitanda | Stochastic proximal gradient descent with acceleration techniques[END_REF], [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] obtain accelerated versions of these algorithms with running time complexities O(d(k + p kL/µ) log(1/")) and O(k log 1/") + p kL/") when the function is not strongly convex. They are shown to be optimal by [START_REF] Agarwal | A lower bound for the optimization of finite sums[END_REF], [START_REF] Lan | An optimal randomized incremental gradient method[END_REF], [START_REF] Arjevani | On the iteration complexity of oblivious first-order optimization algorithms[END_REF], [START_REF] Woodworth | Tight complexity bounds for optimizing composite objectives[END_REF].

Conclusion

The different rates of convergence for stochastic convex optimization are summarized in Table 1.2. Stochastic approximation techniques enable to directly optimize the generalization error and provide predictors with optimal estimation error. Moreover their estimation rates are much simpler to prove than analyzing (a) the statistical performance of the minimizer of the empirical risk as in Section 1.4 and (b) the convergence rate of the optimization method (studied in Section 1.5). Shalev-Shwartz [2016b] points out that the minimizer of the empirical risk may have better statistical properties than the predictor obtained with stochastic gradient descent even if they have an estimation error of the same order. In particular this latter seems to be more efficient to predict rare events. Therefore this fact emphasizes the importance to efficiently optimize finite sum objective functions with dedicated methods (SAG, SDCA, SVRG) with the intended target of rapidly minimizing the empirical risk [START_REF] Frostig | Un-regularizing: Approximate proximal point and faster stochastic algorithms for empirical risk minimization[END_REF].

Stochastic approximation methods have the significant advantage of being online and new data are instantly taken into account. Yet the i.i.d. assumption on the streamed data is quite strong. We will explain now how online learning lifts the assumption that the sequence being observed is independently sampled from an unknown distribution.

Online Convex Optimization

Online learning studies sequential decision processes. It may be viewed as an extension of statistical learning to a sequential setting without independence of the data. The setting is even harder: an adversary maliciously and adaptively picks the data in order to confuse the learner. A comprehensive survey on this very active field, closely related to game theory, was written by [START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF]. We present here the online convex optimization framework defined by [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF] following [START_REF] Gordon | Regret bounds for prediction problems[END_REF]: let C be a non empty convex set. At each iteration n the learner -predicts a vector ✓ n -observes a convex loss function

f n : C ! R
-suffers the loss f n (✓ n ). He wants to minimize his regret:

R n = n X i=1 f i (✓ i ) inf ✓2C n X i=1 f i (✓).
Following [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF], [START_REF] Hazan | The convex optimization approach to regret minimization[END_REF], only linear functions f n (✓) = hx n , ✓i for x n 2 R d are considered here. Indeed results on linear functions are straightforwardly lifted to general convex functions, using the inequality

f n (✓ n ) f n (✓)  hrf n (✓ n ), ✓ n ✓i for all ✓ 2 C and considering the linear proxy fn (✓) = hrf n (✓ n ), ✓i.
Hannan [1957] initially proposes to predict ✓ n = arg min ✓2C P n 1 i=1 f i (✓), the best vector so far. However [START_REF] Kalai | Efficient algorithms for online decision problems[END_REF] show that the regret R n of this simple strategy (they name follow the leader ) can sometimes be O(n). Following the time-varying potential method presented by Cesa-Bianchi and Lugosi [2006, Sec. 11.6], [START_REF] Shalev-Shwartz | A primal-dual perspective of online learning algorithms[END_REF] and [START_REF] Abernethy | Competing in the dark: An efficient algorithm for bandit linear optimization[END_REF] propose to add a regularization h : C ! R to the previous approach. The follow the regularized leader (FTRL) algorithm considers at each iteration n the update

✓ n = arg min ✓2C n 1 X i=1 f i (✓) + h(✓).
It may be directly interpreted as the online extension of the dual averaging algorithm presented in Section 1.5.5. Nevertheless the primal-dual approach of Shalev-Shwartz and Singer [2007] is more general and may lead to different algorithms. We also note FTRL recovers for special choices of constraint set C and regularization h the online gradient descent by [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF] and the exponentiated gradient by [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF].

We can also take a different iterative approach and consider the online extensions of mirror descent and dual averaging algorithms introduced Section 1.5.5. In online learning, they are respectively denoted by greedy and lazy online mirror descent. As their deterministic counterparts, these two methods consider two sequences ( n ) and (✓ n ) and respectively update:

rh( n ) = rh(✓ n 1 ) ⌘f n 1 (greedy update) rh( n ) = rh( n 1 ) ⌘f n 1 , (lazy update)
Then ✓ n is obtained by projecting according to the Bregman divergence

✓ n = arg min ✓2C D h (✓, n ).
In the linear case, the lazy online mirror descent is equivalent to the FTRL approach as remarked by [START_REF] Hazan | Extracting certainty from uncertainty: Regret bounded by variation in costs[END_REF].

Under strong convexity assumptions on the regularization h, and B-Lipschitz assumptions on the functions f n , Shalev-Shwartz [2007], [START_REF] Abernethy | Competing in the dark: An efficient algorithm for bandit linear optimization[END_REF] show that the regret of FTRL is

R n = O(B p n).
Moreover when the loss functions f n are also µ-strongly convex, [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF], [START_REF] Shalev-Shwartz | Mind the duality gap: Logarithmic regret algorithms for online optimization[END_REF] show the regret becomes logarithmic R n = O(B 2 log n/µ). These rates are known to be optimal [START_REF] Takimoto | The minimax strategy for gaussian density estimation[END_REF]Warmuth, 2000, Shalev-Shwartz, 2007].

Furthermore [START_REF] Littlestone | From on-line to batch learning[END_REF] explains online learning results can be converted into statistical learning results by considering the average ✓n = 1 n P n 1 i=0 ✓ i as output of the learning algorithm. This has been strengthened by [START_REF] Cesa-Bianchi | On the generalization ability of on-line learning algorithms[END_REF], [START_REF] Zhang | Data dependent concentration bounds for sequential prediction algorithms[END_REF] to results on high probability.

Application to online linear regression. We illustrate this setting on online linear regression where we observe sequentially deterministc data (x 1 , y 1 , . . . , x n , y n ) 2 (X ⇥ Y) n and want to design a predictor which achieves small regret

R n = n X i=1 (hx i , ✓ i i y i ) 2 inf ✓2C n X i=1 (hx i , ✓i y i ) 2 .
This is the online learning setting applied to the least-squares functions

f n (✓) = (hx n , ✓i y n ) 2 .
The FTRL approach may be used with an `2-regularization h(✓) = 1 2 k✓k 2 2 to obtain a predictor

✓ n = arg min ✓2R d n 1 X i=1 (h✓, x i i y i ) 2 + k✓k 2
similar to the one of the ridge regression [START_REF] Hoerl | Ridge regression: Biased estimation for nonorthogonal problems[END_REF]. Its regret is not generally bounded, though [Cesa-Bianchi and Lugosi, 2006, Theorem 11.7]. [START_REF] Vovk | Competitive on-line statistics[END_REF], [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF] solve this issue making the predictor non linear. The Vovk-Azoury-Warmuth predictor is the non linear predictor h n (x) = h✓ x n , xi where ✓ x n is given by

✓ x n = arg min ✓2R d n 1 X i=1 (h✓, x i i y i ) 2 + k✓k 2 + h✓, xi 2 .
The additional term (compared with the ridge regression predictor) h✓, xi 2 may be interpreted as the loss being incurred at time n when y n is estimated by 0. ✓ xn n has the following close form:

✓ xn n = ⇣ I + n X i=1 x i ⌦ x i ⌘ 1 ⇣ n 1 X i=1 y i x i ⌘ ,
which can be sequentially computed with complexity O(d 2 ) thanks to the Sherman-Morrison formula. For this predictor, the following convergence result holds [Cesa-Bianchi and Lugosi, 2006, Theorem 11.8].

Proposition 4. The Vovk-Azoury-Warmuth predictor on (x

1 , y 1 , . . . , x n , y n ) 2 (R d ⇥ R) n satisfies 1 n n X i=1 (h✓ x i i , x i i y i ) 2 inf ✓2R d n 1 n n X i=1 (h✓, x i i y i ) 2 + 1 n k✓k 2 2 o  dB 2 log(1 + nR 2 /d) n , (1.4)
where B 2 = max i2{1,...,n} y 2 i and R 2 = max i2{1,...,n} kx i k 2 2 . Therefore in the setting of online learning, with no assumption on the generation of the data, we obtain an O(d log(n)/n) bound on the regret for least-squares problems. Furthermore [START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF] show the optimality of this bound. We also note [START_REF] Rakhlin | Online non-parametric regression[END_REF] extend this method to online non-parametric regression.

Digest of Least-Squares Regression

We summarize here the different results seen so far for the particular example of the least-squares regression:

min ✓2R d L(✓) = 1 2 E(h (X), ✓i Y ) 2 ,
where (X, Y ) are random variables of unknown distribution D. We remind that the covariance matrix is denoted by ⌃ = E[ (X) ⌦ (X)] and is only assumed invertible. Thus its eigenvalues may be arbitrarily small, in contrast with a recent work by [START_REF] Jain | Parallelizing stochastic approximation through mini-batching and tail-averaging[END_REF]. The global minimum of L(✓) is denoted by ✓ ⇤ . We consider not being limited in terms of sample complexity and are instead interested in the running time complexity of the different methods.

Empirical risk minimization. In Section 1.

2 d n ) d O ( 2 d 3 " ) Regularized ERM Accelerated SDCA O( 2 d n ) Õ( 2 d+k✓⇤k 2 R d 1/2 "
) Õ(

2 d 2 +k✓⇤k 2 R d 3/2 " ) S.A. SGD O( 2 d+R 2 k✓⇤k 2 n ) O( 2 d+R 2 k✓⇤k 2 " ) O( 2 d 2 +dR 2 k✓⇤k 2 " ) Online Learning hn predictor O( 2 d log(n)+R 2 k✓⇤k 2 n ) Õ( 2 d+R 2 k✓⇤k 2 "
) ?

Table 1.3 -Complexities of least-squares regression. Õ ignores logarithmic factors. S.A. denotes stochastic approximation.

achieves under reasonable assumptions on (X, Y ):

L( ✓) L(✓ ⇤ ) = O ⇣ 2 d n ⌘ .
This quadratic function is efficiently solved with the conjugate gradient algorithm presented in Section 1.5.3. The solution ✓ is obtained in at most d iterations and the complexity of each iteration is O(nd) (the cost of computing a gradient of L).

Therefore the total running time complexity is O(

2 d 3 " ).
Regularized risk minimization. The `2-regularized training error L (✓) = k✓k2 2 + 1 2n

P n i=1 (h (x i ), ✓i y i ) 2 may also be considered. [START_REF] Hsu | Random design analysis of ridge regression[END_REF] show its minimizer ✓ achieves, under suitable assumptions, a similar estimation rate

L( ✓ ) L(✓ ⇤ ) = O ⇣ 2 d n + k✓ ⇤ k 2 2 ⌘ . Thus = O( 2 d k✓⇤k 2 2 n ) is taken to obtain L( ✓ ) L(✓ ⇤ ) = O( 2 d/n).
Furthermore the regularization may be leveraged by using fast optimization methods dedicated to finite sum of strongly convex functions. Accelerated-SDCA has a running time complexity

O(d(n + R p n/ log(1/"))).
Replacing with the value of , the global running time complexity becomes Õ(

2 d 2 +k✓⇤k 2 R d 3/2 "
).

Stochastic approximations. The generalization error may also be directly optimized using stochastic gradient descent with constant step size and averaging, as in Section 1.

(x) = 1 n n X i=1 h✓ x i , xi
can be taken into account. Classic online to batch conversion techniques ensure an estimation error O(

2 d/n + R 2 k✓ ⇤ k 2 2 /n).
The main downside of this method is that it is unclear how to efficiently compute this average. It may be relaxed in hn (x) = h ✓n , xi where ✓n is defined by ✓n = 1 n P n i=1 ✓ x i i but the independence between ✓n and x n is then lost and the statistical properties of this estimator are not well understood.

Hence the best running time complexity of these methods is O(

2 d 2 +dR 2 k✓⇤k 2 "
) with a minimax optimal variance term 2 d 2 /" and a suboptimal bias term O( dR 2 k✓⇤k 2

"

). Actually a bias term with a decrease in 1/ p " is expected since it is the optimal rate of convergence for first-order optimization of smooth non-strongly convex quadratic functions. This issue will be the quest of the two following chapters of this manuscript.

Part I Stochastic Approximation and

Least-Squares Regression 33

Introduction

Many problems in machine learning are naturally cast as convex optimization problems over a Euclidean space; for supervised learning this includes least-squares regression, logistic regression, and the support vector machine. Faced with large amounts of data, practitioners often favor first-order techniques based on gradient descent, leading to algorithms with many cheap iterations. For smooth problems, two extensions of gradient descent have had important theoretical and practical impacts: acceleration and averaging.

Acceleration techniques date back to [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF] and have their roots in momentum techniques and conjugate gradient [START_REF] Polyak | Introduction to Optimization[END_REF]. For convex problems, with an appropriately weighted momentum term which requires to store two iterates, Nesterov [1983] showed that the traditional convergence rate of O(1/n) for the function values after n iterations of gradient descent goes down to O(1/n 2) for accelerated gradient descent, such a rate being optimal among first-order techniques that can access only sequences of gradients [START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF]. Like conjugate gradient methods for solving linear systems, these methods are however more sensitive to noise in the gradients; that is, to preserve their improved convergence rates, significantly less noise may be tolerated [START_REF]Smooth optimization with approximate gradient[END_REF][START_REF] Schmidt | Convergence rates of inexact proximal-gradient methods for convex optimization[END_REF][START_REF] Devolder | First-order methods of smooth convex optimization with inexact oracle[END_REF].

Averaging techniques which consist in replacing the iterates by the average of all iterates have also been thoroughly considered, either because they sometimes lead to simpler proofs, or because they lead to improved behavior. In the noiseless case where gradients are exactly available, they do not improve the convergence rate in the convex case; worse, for strongly convex problems, they are not linearly convergent while regular gradient descent is. Their main advantage comes with random unbiased gradients, where it has been shown that they lead to better convergence rates than the unaveraged counterparts, in particular because they allow larger step-sizes [Polyak andJuditsky, 1992, Bach and[START_REF] Bach | Non-asymptotic analysis of stochastic approximation algorithms for machine learning[END_REF]. For example, for least-squares regression with stochastic gradients, they lead to convergence rates of O(1/n), even in the non-strongly convex case [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF].

In this chapter, we show that for quadratic problems, both averaging and acceleration are two instances of the same second-order finite difference equation, with different step-sizes. They may thus be analyzed jointly, together with a non-strongly convex version of the heavy-ball method [Polyak, 1987, Section 3.2]. In presence of random zero-mean noise on the gradients, this joint analysis allows to design a novel intermediate algorithm that exhibits the good aspects of both acceleration (quick forgetting of initial conditions) and averaging (robustness to noise).

In this chapter, we make the following contributions: -We show in Section 2.2 that accelerated gradient descent, averaged gradient descent and the heavy-ball method for quadratic non-strongly convex problems may be reformulated as constant parameter second-order difference equation algorithms, where stability of the system is equivalent to convergence at rate O(1/n 2 ). -In Section 2.3, we provide a detailed analysis of the eigenvalues of the corresponding linear dynamical system, showing various oscillatory and non-oscillatory behaviors, together with a sharp stability result with explicit constants. -In Section 2.4, we consider the situation where noisy gradients are available, where we extend our general convergence result, which suggests an alternative algorithm (i.e., with different step sizes) that exhibits the good aspects of both averaging and acceleration. -In Section 2.5, we illustrate our results with simulations on synthetic examples.

Second-Order Iterative Algorithms for Quadratic Functions

In this chapter, we consider minimizing a convex quadratic function f : R d ! R defined as:

f (✓) = 1 2 h✓, H✓i hq, ✓i, (2.1)
with H 2 R d⇥d a symmetric positive semi-definite matrix and q 2 R d . Without loss of generality, H is assumed invertible (by projecting onto the orthogonal of its null space), though its eigenvalues could be arbitrarily small. The solution is known to be ✓ ⇤ = H 1 q, but the inverse of the Hessian is often too expensive to compute when d is large. The excess cost function may be simply expressed as

f (✓ n ) f (✓ ⇤ ) = 1 2 h✓ n ✓ ⇤ , H(✓ n ✓ ⇤ )i.

Second-order Algorithms

In this chapter we study second-order iterative algorithms of the form:

✓ n+1 = A n ✓ n + B n ✓ n 1 + c n , (2.2)
started with

✓ 1 = ✓ 0 in R d , with A n 2 R d⇥d , B n 2 R d⇥d and c n 2 R d for all n 2 N ⇤ .
We impose the natural restriction that the optimum ✓ ⇤ is a stationary point of this recursion, that is, for all n 2 N ⇤ :

✓ ⇤ = A n ✓ ⇤ + B n ✓ ⇤ + c n . (✓ ⇤ -stationarity) By letting n = ✓ n ✓ ⇤ we then have n+1 = A n n + B n n 1 , started from 0 = 1 = ✓ 0 ✓ ⇤ .
Thus, we restrict our problem to the study of the convergence of an iterative system to 0.

In connection with accelerated methods, we are interested in algorithms for which

f (✓ n ) f (✓ ⇤ ) = 1 2 h n , H n i (2.3)
converges to 0 at a speed of O(1/n 2 ). Within this context we impose that A n and B n have the form:

A n = n n+1 A and B n = n 1 n+1 B 8n 2 N with A, B 2 R d⇥d .
(n-scalability)

By letting ⌘ n = n n = n(✓ n ✓ ⇤ ), we can now study the simple iterative system with constant terms

⌘ n+1 = A⌘ n + B⌘ n 1 , started at ⌘ 0 = 0 and ⌘ 1 = ✓ 0 ✓ ⇤ . Showing that the function values f (⌘ n ) remain bounded, we directly have from Eq. (2.3), the convergence of f (✓ n ) to f (✓ ⇤ ) at the speed O (1/n 2
). Thus the n-scalability property allows to switch from a convergence problem to a stability problem.

For feasibility concerns the method can only access H through matrix-vector products. Therefore A and B should be polynomials in H and c a polynomial in H times q, if possible of low degree. The following theorem clarifies the general form of iterative systems which share these three properties (see proof in Appendix 2.B).

Theorem 2. Let (P n , Q n , R n ) 2 (R[X]
) 3 for all n 2 N, be a sequence of polynomials. If the iterative algorithm defined by Eq. (2.2) with A n = P n (H), B n = Q n (H) and c n = R(H)q satisfies the ✓ ⇤ -stationarity and n-scalability properties, there are polynomials

( Ā, B) 2 (R[X]) 2 such that: A n = 2 n n+1 I 1 2 Ā(H) + B(H) H , 37
B n = n 1 n+1 I B(H)H and c n = 1 n+1 n Ā(H) + B(H) q.
Note that our result prevents A n and B n from being zero, thus requiring the algorithm to strictly be of second order. This illustrates the fact that first-order algorithms as gradient descent do not have the convergence rate in O(1/n 2

). We now restrict our class of algorithms to lowest possible order polynomials, that is, Ā = ↵I and B = I with (↵, ) 2 R 2 , which correspond to the fewest matrixvector products per iteration, leading to the constant-coefficient recursion for ⌘ n = n n = n(✓ n ✓ ⇤ ):

⌘ n+1 = (I ↵H) ⌘ n + (I H) (⌘ n ⌘ n 1 ) . (2.4)
Expression with gradients of f . The recursion in Eq. ( 2.4) may be written with gradients of f in multiple ways. In order to preserve the parallel with accelerated techniques, we rewrite it as:

✓ n+1 = 2n n+1 ✓ n n 1 n+1 ✓ n 1 n↵+ n+1 f 0 n(↵+ ) n↵+ ✓ n (n 1) n↵+ ✓ n 1 . (2.5)
It may be interpreted as a modified gradient recursion with two potentially different affine (i.e., with coefficients that sum to one) combinations of the two past iterates. This reformulation will also be crucial when using noisy gradients. The allowed values for (↵, ) 2 R 2 will be determined in the following sections.

Examples

Averaged gradient descent. We consider averaged gradient descent (referred to from now on as "Av-GD") [START_REF] Polyak | Acceleration of stochastic approximation by averaging[END_REF] with step-size 2 R defined by

n+1 = n f 0 ( n ) and ✓ n+1 = 1 n+1 P n+1 i=1 i .
When computing the average online as ✓ n+1 = ✓ n + 1 n+1 ( n+1 ✓ n ) and seeing the average as the main iterate, the algorithm becomes (see proof in Appendix 2.B.2):

✓ n+1 = 2n n+1 ✓ n n 1 n+1 ✓ n 1 n+1 f 0 n✓ n (n 1)✓ n 1 .
This corresponds to Eq. (2.5) with ↵ = 0 and = .

Accelerated gradient descent. We consider the accelerated gradient descent (referred to from now on as "Acc-GD") [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF] with step-sizes ( , n ) 2 R 2 :

✓ n+1 = ! n f 0 (! n ), ! n = ✓ n + n (✓ n ✓ n 1 ).
For smooth optimization, the accelerated literature [Nesterov, 2004, Beck and[START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] uses the step-size n = 1 3 n+1 and their results are not valid for bigger stepsize n . However n = 1 2 n+1 is compatible with the framework of [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF] and is more convenient for our set-up. This corresponds to Eq. (2.5) with ↵ = and = . Note that accelerated techniques are more generally applicable, e.g., to composite optimization with smooth functions [Nesterov, 2013, Beck and[START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF].

Heavy ball. We consider the heavy-ball algorithm (referred to from now on as "HB") [START_REF] Polyak | Some methods of speeding up the convergence of iteration methods[END_REF] with step-sizes ( , n ) 2 R 2 :

✓ n+1 = ✓ n f 0 (✓ n ) + n (✓ n ✓ n 1 ),
when n = 1 2 n+1 (we note that typically n is constant for strongly convex problems). This corresponds to Eq. (2.5) with ↵ = and = 0.

Convergence with Noiseless Gradients

We study the convergence of the iterates defined by Eq. (2.4). This is a secondorder iterative system with constant coefficients that it is standard to cast in a linear framework [see, e.g., [START_REF] Ortega | Iterative Solution of Nonlinear Equations in Several Variables[END_REF]. We may rewrite it as: [START_REF] O'donoghue | Adaptive restart for accelerated gradient schemes[END_REF], if we consider an eigenvalue decomposition of H, i.e., H = P Diag(h)P > with P an orthogonal matrix and (h i ) the eigenvalues of H, sorted in decreasing order: 2.4) may be rewritten as:

⇥ n = F ⇥ n 1 , with ⇥ n = ✓ ⌘ n ⌘ n 1 ◆ and F = ✓ 2I (↵ + ) H H I I 0 ◆ 2 R 2d⇥2d . Thus ⇥ n = F n ⇥ 0 . Following O'Donoghue
h d = L h d 1 • • • h 2 h 1 = µ > 0, then Eq. (
P > ⌘ n+1 = (I ↵Diag (h)) P > ⌘ n + (I Diag (h)) P > ⌘ n P > ⌘ n 1 . (2.6)
Thus there is no interaction between the different eigenspaces and we may consider, for the analysis only, d different recursions with ⌘ i n = p > i ⌘ n , i 2 {1, ..., d}, where p i 2 R d is the i-th column of P :

⌘ i n+1 = (1 ↵h i ) ⌘ i n + (1 h i ) ⌘ i n ⌘ i n 1 .
(2.7)

Characteristic Polynomial and Eigenvalues

In this section, we consider a fixed i 2 {1, . . . , d} and study the stability in the corresponding eigenspace. This linear dynamical system may be analyzed by studying the eigenvalues of the 2 ⇥ 2-matrix

F i = ✓ 2 (↵ + )h i h i 1 1 0 ◆ . These eigenvalues
are the roots of its characteristic polynomial which is:

det(xI F i ) = (x (x 2 + (↵ + )h i ) + 1 h i ) = x 2 2x 1 ↵+ 2 h i + 1 h i .
To compute the roots of the second-order polynomial, we compute its discriminant:

i = 1 ↵+ 2 h i 2 1 + h i = h i ↵+ 2 2 h i ↵ .
Depending on the sign of the discriminant i , there will be two real distinct eigenvalues ( i > 0), two complex conjugate eigenvalues ( i < 0) or a single real eigen-

0 1 2 3 4 0 1 2 AvGD AccGD HB Real Complex h i ↵h i
Figure 2-1 -Area of stability of the algorithm, with the three traditional algorithms represented. In the interior of the triangle, the convergence is linear.

value ( i = 0).

We will now study the sign of i . In each different case, we will determine under what conditions on ↵ and the modulus of the eigenvalues is less than one, which means that the iterates (⌘ i n ) n remain bounded and the iterates (✓ n ) n converge to ✓ ⇤ . We may then compute function values from Eq. (2.3)

as f (✓ n ) f (✓ ⇤ ) = 1 2n 2 P d i=1 (⌘ i n ) 2 h i = 1 2 P d i=1 ( i n ) 2 h i .
The various regimes are summarized in Figure 2-1: there is a triangle of values of (↵h i , h i ) for which the algorithm remains stable (i.e., the iterates (⌘ n ) n do not diverge), with either complex or real eigenvalues. In the following lemmas (see proof in Appendix 2.C), we provide a detailed analysis that leads to Figure 2-1.

Lemma 1 (Real eigenvalues). The discriminant i is strictly positive and the algorithm is stable if and only if

↵ 0, ↵+ 2  4/h i , ↵+ > 2 p ↵/h i .
We then have two real roots r ± i = r i ± p i , with r i = 1 ( ↵+ 2 )h i . Moreover, we have:

( i n ) 2 h i = ( i 1 ) 2 h i 4n 2 ⇥ (r i + p i ) n (r i p i ) n ⇤ 2 i . (2.8)
Therefore, for real eigenvalues, (( i n ) 2 h i ) n will converge to 0 at a speed of O(1/n 2 ) however the constant i may be arbitrarily small (and thus the scaling factor arbitrarily large). Furthermore we have linear convergence if the inequalities in the lemmas are strict.

Lemma 2 (Complex eigenvalues). The discriminant i is stricly negative and the algorithm is stable if and only if ↵ 0, 0, ↵+ < p ↵/h i .

We then have two complex conjugate eigenvalues: r ± i = r i ± p 1 p i . Moreover, we have:

( i n ) 2 h i = ( i 1 ) 2 n 2 sin 2 (! i n) ↵ ( ↵+ 2 ) 2 h i ⇢ 2n .
(2.9)

with ⇢ i = p 1 h i , and ! i defined through sin(! i ) = p i /⇢ i and cos(! i ) = r i /⇢ i .

Therefore, for complex eigenvalues, there is a linear convergence if the inequalities in the lemma are strict. Moreover, (( i n ) 2 h i ) n oscillates to 0 at a speed of O(1/n 2 ) even if h i is arbitrarily small.

Coalescing eigenvalues. When the discriminants go to zero in the explicit formulas of the real and complex cases, both the denominator and numerator of (( i n ) 2 h i ) n will go to zero. In the limit case, when the discriminant is equal to zero, we will have a double real eigenvalue. This happens for = 2 p ↵/h i ↵. Then the eigenvalue is r i = 1 p ↵h i , and the algorithm is stable for 0 < ↵ < 4/h i , we then have

( i n ) 2 h i = h i ( i 1 ) 2 (1 p ↵h i ) 2(n 1)
. This can be obtained by letting i goes to 0 in the real and complex cases (see also Appendix 2.C.3). Summary. To conclude, the iterate

(⌘ i n ) n = (n(✓ i n ✓ i ⇤ )) n will be stable for ↵ 2 [0, 4/h i ] and 2 [0, 2/h i ↵/2].
According to the values of ↵ and this iterate will have a different behavior. In the complex case, the roots are complex conjugate with magnitude p 1 h i . Thus, when > 0, (⌘ i n ) n will converge to 0, oscillating, at rate p 1 h i . In the real case, the two roots are real and distinct. However the product of the two roots is equal to p 1 h i , thus one will have a higher magnitude and

(⌘ i n ) n will converges to 0 at rate higher than in the complex case (as long as ↵ and belong to the interior of the stability region).

Finally, for a given quadratic function f , all the d iterates (⌘ i n ) n should be bounded, therefore we must have ↵ 2 [0, 4/L] and 2 [0, 2/L ↵/2]. Then, depending on the value of h i , some eigenvalues may be complex or real.

Classical Examples

For particular choices of ↵ and , displayed in Figure 2-1, the eigenvalues are either all real or all complex, as shown in the table below.

Av-GD

Acc-GD Heavy ball

↵ 0 0 i ( h i ) 2 h i (1 h i ) h i (1 h i 4 ) r ± i 1, 1 h i p 1 h i e ±i! i e ±i! i cos(! i ) p 1 h i 1 2 h i ⇢ i p 1 h i 1
Averaged gradient descent loses linear convergence for strongly convex problems, because r + i = 1 for all eigensubspaces. Similarly, the heavy ball method is not adaptive to strong convexity because ⇢ i = 1. However, accelerated gradient descent, although designed for non-strongly convex problems, is adaptive because ⇢ i = p 1 h i depends on h i while ↵ and do not. These last two algorithms have an oscillatory behavior which can be observed in practice and has been already studied [START_REF] Su | A differential equation for modeling Nesterov's accelerated gradient method: theory and insights[END_REF].

Note that all the classical methods choose step-sizes ↵ and either having all the eigenvalues real or complex; whereas we will see in Section 2.4 that it is significant to combine both behaviors in the presence of noise.

General Bound

Even if the exact formulas in Lemmas 1 and 2 are computable, they are not easily interpretable. In particular when the two roots become close, the denominator will go to zero, which prevents us from bounding them easily. When we further restrict the domain of (↵, ), we can always bound the iterate by the general bound (see proof in Appendix 2.D):

Theorem 3. For ↵  1/h i and 0   2/h i ↵, we have

(⌘ i n ) 2  min ⇢ 2(⌘ i 1 ) 2 ↵h i , 8(⌘ i 1 ) 2 n (↵ + )h i , 16(⌘ i 1 ) 2 (↵ + ) 2 h 2 i .
(2.10)

These bounds are shown by dividing the set of (↵, ) in three regions where we obtain specific bounds. They do not depend on the regime of the eigenvalues (complex or real); this enables us to get the following general bound on the function values, our main result for the deterministic case.

Corollary 1. For 0  ↵  1/L and 0   2/L ↵: .11) We can make the following observations:

f (✓ n ) f (✓ ⇤ )  min ⇢ k✓ 0 ✓ ⇤ k 2 ↵n 2 , 4k✓ 0 ✓ ⇤ k 2 (↵ + )n . ( 2
-The first bound k✓ 0 ✓⇤k 2 ↵n 2 corresponds to the traditional acceleration result, and is only relevant for ↵ > 0 (that is, for Nesterov acceleration and the heavy-ball method, but not for averaging). We recover the traditional convergence rate of second-order methods for quadratic functions in the singular case, such as conjugate gradient [Polyak, 1987, Section 6.1].

-While the result above focuses on function values, like most results in the nonstrongly convex case, the distance to optimum k✓ n ✓ ⇤ k 2 typically does not go to zero (although it remains bounded in our situation).

-When ↵ = 0 (averaged gradient descent), then the second bound 4k✓ 0 ✓⇤k 2 (↵+ )n provides a convergence rate of O(1/n) if no assumption is made regarding the starting point

(✓ 0 ✓⇤)k 2 (↵+ ) 2 n 2
, that is a rate of O(1/n 2), only for some starting points. -As shown in Appendix 2.E by exhibiting explicit sequences of quadratic functions, the inverse dependence in ↵n 2 and (↵ + )n in Eq. (2.11) is not improvable.

Quadratic Optimization with Additive Noise

In many practical situations, the gradient of f is not available for the recursion in Eq. (2.5), but only a noisy version. In this chapter, we only consider additive uncorrelated noise with finite variance.

Stochastic Difference Equation

We now assume that the true gradient is not available and we rather have access to a noisy oracle for the gradient of f in Eq. (2.5). We assume that the oracle outputs a noisy gradient f 0 n(↵+ ) n↵+ ✓ n (n 1) n↵+ ✓ n 1 " n+1 . The noise (" n ) is assumed to be uncorrelated zero-mean with bounded covariance, i.e., E[" n ⌦ " m ] = 0 for all n 6 = m and E[" n ⌦ " n ] 4 C, where A 4 B means that B A is positive semi-definite.

For quadratic functions, for the reduced variable ⌘ n = n n = n(✓ n ✓ ⇤ ), we get:

⌘ n+1 = (I ↵H)⌘ n + (I H)(⌘ n ⌘ n 1 ) + [n↵ + ]" n+1 .
(2.12)

Note that algorithms with ↵ 6 = 0 will have an important level of noise because of the term n↵" n+1 . We denote by

⇠ n+1 = ✓ [n↵ + ]" n+1 0
◆ and we now have the recursion:

⇥ n+1 = F ⇥ n + ⇠ n+1 , (2.13)
which is a standard noisy linear dynamical system [see, e.g., [START_REF] Arnold | Random Dynamical Systems[END_REF]] with un-correlated noise process (⇠ n ). We may thus express ⇥ n directly as ⇥ n = F n 1 ⇥ 1 + P n k=2 F n k ⇠ k , and its expected second-order moment as

E ⇥ n ⇥ > n = F n 1 ⇥ 1 ⇥ > 1 (F n 1 ) > + n X k=2 F n k E ⇠ k ⇠ > k (F n k ) > .
In order to obtain the expected excess cost function, we simply need to compute

tr ✓ 0 0 0 H ◆ E ⇥ n ⇥ > n ,
which thus decomposes as a term that only depends on initial conditions (which is exactly the one computed and studied in Section 2.3.3), and a new term that depends on the noise.

Convergence Result

For a quadratic function f with arbitrarily small eigenvalues and uncorrelated noise with finite covariance, we obtain the following convergence result (see proof in Appendix 2.F); since we will allow the parameters ↵ and to depend on the time we stop the algorithm, we introduce the horizon N :

Theorem 4 (Convergence rates with noisy gradients).

With E[" n ⌦ " n ] = C for all n 2 N, for ↵  1 L and 0   2 L ↵.
Then for any N 2 N, we have:

Ef (✓ N ) f (✓ ⇤ )  min ⇢ k✓ 0 ✓ ⇤ k 2 ↵N 2 + (↵N + ) 2 ↵N tr(C), 4k✓ 0 ✓ ⇤ k 2 (↵ + )N + 4(↵N + ) 2 ↵ + tr(C) .
We can make the following observations:

-Although we only provide an upper-bound, the proof technique relies on direct moment computations in each eigensubspace with few inequalities, and we conjecture that the scalings with respect to N are tight.

-For ↵ = 0 and = 1/L (which corresponds to averaged gradient descent), the second bound leads to 4Lk✓ 0 ✓⇤k 2 N + 4 tr(C) L , which is bounded but not converging to zero. We recover a result from Bach and Moulines [2011, Theorem 1].

-For ↵ = = 1/L (which corresponds to Nesterov's acceleration), the first bound leads to Lk✓ 0 ✓⇤k 2

N 2 + (N +1) tr(C) L
, and our bound suggests that the algorithm diverges, which we have observed in our experiments in Appendix 2.A.

-For ↵ = 0 and = 1/L p N , the second bound leads to 4Lk✓ 0 ✓⇤k 2 p N

+ 4 tr(C) L p
N , and we recover the traditional rate of 1/ p N for stochastic gradient in the non-strongly convex case.

-When the values of the bias and the variance are known we can choose ↵ and such that the trade-off between the bias and the variance is optimal in our bound, as the following corrollary shows. Note that in the bound below, taking a non-zero enables the bias term to be adaptive to hidden strong convexity.

Ef (✓ N ) f (✓ ⇤ )  2Lk✓ 0 ✓ ⇤ k 2 N 2 + 4 p tr Ck✓ 0 ✓ ⇤ k p N .

Structured Noise and Least-Squares Regression

When only the noise total variance tr(C) is considered, as shown in Section 2.4.4, Corollary 2 recovers existing (more general) results. Our framework however leads to improved result for structured noise processes frequent in machine learning, in particular in least-squares regression which we now consider but also in others problems [see, e.g. [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF].

Assume we observe independent and identically distributed pairs (x n , y n ) 2 R d ⇥R and we want to minimize the expected loss

f (✓) = 1 2 E[(y n h✓, x n i) 2
]. We denote by H = E(x n ⌦ x n ) the covariance matrix which is assumed invertible. The global minimum of f is attained at ✓ ⇤ 2 R d defined as before and we denote by r n = y n h✓ ⇤ , x n i the statistical noise, which we assume bounded by . We have E[r n x n ] = 0. In an online setting, we observe the gradient (x n ⌦ x n )(✓ ✓ ⇤ ) r n x n , whose expectation is the gradient f 0 (✓). This corresponds to a noise in the gradient of

" n = (H x n ⌦ x n )(✓ ✓ ⇤ ) + r n x n .
Given ✓, if the data (x n , y n ) are almost surely bounded, the covariance matrix of this noise is bounded by a constant times H. This suggests to characterize the noise convergence by tr(CH 1

), which is bounded even though H has arbitrarily small eigenvalues.

However, our result will not apply to stochastic gradient descent (SGD) for leastsquares, because of the term (H x n ⌦ x n )(✓ ✓ ⇤ ) which depends on ✓, but to a "semi-stochastic" recursion where the noisy gradient is H(✓ ✓ ⇤ ) r n x n , with a noise process " n = r n x n , which is such that E[" n ⌦ " n ] 4 2 H, and has been used by [START_REF] Bach | Non-asymptotic analysis of stochastic approximation algorithms for machine learning[END_REF] and [START_REF] Dieuleveut | Non-parametric stochastic approximation with large step sizes[END_REF] to prove results on regular stochastic gradient descent. We conjecture that our algorithm (and results) also applies in the regular SGD case, and we provide encouraging experiments in Section 2.5.

For this particular structured noise we can take advantage of a large :

Theorem 5 (Convergence rates with structured noisy gradients).

Let ↵  1 L and 0   3 2L ↵ 2 . For any N 2 N, Ef (✓ N ) f (✓ ⇤ ) is upper-bounded by: min ⇢ k✓ 0 ✓ ⇤ k 2 N 2 ↵ + (↵N + ) 2 ↵ N 2 tr(CH 1 ), 4Lk✓ 0 ✓ ⇤ k 2 (↵ + )N + 8(↵N + ) 2 tr(CH 1 ) (↵ + ) 2 N .
We can make the following observations:

-For ↵ = 0 and = 1/L (which corresponds to averaged gradient descent), the second bound leads to 4Lk✓ 0 ✓⇤k 2 N + 8 tr(CH 1 ) N

. We recover a result from Bach and Moulines [2013, Theorem 1]. Note that when C 4 2 H, tr(CH 1

)  2 d. -For ↵ = = 1/L (which corresponds to Nesterov's acceleration), the first bound leads to Lk✓ 0 ✓⇤k 2 N 2

+ tr(CH 1

), which is bounded but not converging to zero (as opposed to the the unstructured noise where the algorithm may diverge).

-For ↵ = 1/(LN a

) with 0  a  1 and = 1/L, the first bound leads to

Lk✓ 0 ✓⇤k 2 N 2 a + tr(CH 1 ) N a
. We thus obtain an explicit bias-variance trade-off by changing the value of a.

-When the values of the bias and the variance are known we can choose ↵ and with an optimized trade-off, as the following corrollary shows:

Corollary 3. For ↵ = min n k✓ 0 ✓⇤k p L tr(CH 1 )N
, 1/L o and = min {N↵, 1/L} we have:

Ef (✓ N ) f (✓ ⇤ )  max ⇢ 5 tr(CH 1 ) N , 5
p tr(CH 1 )Lk✓ 0 ✓ ⇤ k N , 2k✓ 0 ✓ ⇤ k 2 L N 2 .

Related Work

Acceleration and noisy gradients. Several authors [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF][START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF][START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] have shown that by using a step-size proportional to 1/N 3/2 accelerated methods with noisy gradients lead to the same convergence rate of

O Lk✓ 0 ✓⇤k 2 N 2 + k✓ 0 ✓⇤k p tr(C) p N
as in Corollary 2, for smooth functions. Thus, for unstructured noise, our analysis provides insights in the behavior of second-order algorithms, without improving bounds. We get significant improvements for structured noises.

Least-squares regression. When the noise is structured as in least-square regression and more generally in linear supervised learning, [START_REF] Bach | Non-asymptotic analysis of stochastic approximation algorithms for machine learning[END_REF] have shown that using averaged stochastic gradient descent with constant step-size leads to the convergence rate of

O Lk✓ 0 ✓ 0 k 2 N + 2 d
N . It has been highlighted by [START_REF] Défossez | Averaged least-mean-squares: bias-variance trade-offs and optimal sampling distributions[END_REF] that the bias term Lk✓ 0 ✓⇤k 2 N may often be the dominant one in practice. Our result in Corollary 3 leads to an improved bias term in O(1/N 2

) with the price of a potentially slightly worse constant in the variance term. However, with optimal constants in Corollary 3, the new algorithm is always an improvement over averaged stochastic gradient descent in all situations. If constants are unknown, we may use ↵ = 1/(LN a

) with 0  a  1 and = 1/L and we choose a depending on the emphasis we want to put on bias or variance.

Minimax convergence rates. For noisy quadratic problems, the convergence rate nicely decomposes into two terms, a bias term which corresponds to the noiseless problem and the variance term which corresponds to a problem started at ✓ ⇤ . For each of these two terms, lower bounds are known. For the bias term, if N  d, then the lower bound is, up to constants, Lk✓ [Nesterov, 2004, Theorem 2.1.7]. For the variance term, for the general noisy gradient situation, we show in Appendix 2.H that for N  d, it is (tr C)/(L p N ), while for least-squares regression, it is 2 d/N [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. Thus, for the two situations, we attain the two lower bounds simultaneously for situations where respectively Lk✓

0 ✓ ⇤ k 2 /N 2
Lk✓ 0 ✓ ⇤ k 2  d 2 .
It remains an open problem to achieve the two minimax terms in all situations.

Other algorithms as special cases. We also note as shown in Appendix 2.G that in the special case of quadratic functions, the algorithms of [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF], [START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF], [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] could be unified into our framework (although they have significantly different formulations and justifications in the smooth case).

Experiments

In this section, we illustrate our theoretical results on synthetic examples. We consider a matrix H that has random eigenvectors and eigenvalues 1/k m , for k = 1, . . . , d and m 2 N. We take a random optimum ✓ ⇤ and a random starting point ✓ 0 such that r = k✓ 0 ✓ ⇤ k = 1 (unless otherwise specified). In Appendix 2.A, we illustrate the noiseless results of Section 2.3, in particular the oscillatory behaviors and the influence of all eigenvalues, as well as unstructured noisy gradients. In this section, we focus on noisy gradients with structured noise (as described in Section 2.4.3), where our new algorithms (referred to as "OA") show significant improvements.

We compare our algorithm to other stochastic accelerated algorithms, that is, AC-SA [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF], SAGE [START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF] and Acc-RDA [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] which are presented in Appendix 2.G. For all these algorithms (and ours) we take the optimal step-sizes defined in these papers. We show results averaged over 10 replications.

Homoscedastic noise. We first consider an i.i.d. zero mean noise whose covariance matrix is proportional to H. We also consider a variant "OA-at" of our algorithm with an any-time step-size function of n rather than N (for which we currently have no proof of convergence). In Figure 2-3, we take into account two different set-ups. In the left plot, the variance dominates the bias (with r = k✓ 0 ✓ ⇤ k = ). We see that (a) Acc-GD does not converge to the optimum but does not diverge either, (b) Av-GD and our algorithms achieve the optimal rate of convergence of O( 2 d/n), whereas (c) other accelerated algorithms only converge at rate O(1/ p n). In the right plot, the bias dominates the variance (r = 10 and = 0.1). In this situation our algorithm outperforms all others.

Application to least-squares regression. We now see how these algorithms behave for least-squares regressions and the regular (non-homoscedastic) stochastic gradients described in Section 2.4.3. We consider normally distributed inputs. The covariance matrix H is the same as before. The outputs are generated from a linear function with homoscedatic noise with a signal-to-noise ratio of . We consider d = 20. We show results averaged over 10 replications. In Figure 2-4, we consider again a situation where the variance dominates the bias (left) and vice versa (right). We see that our algorithm has the same good behavior as in the homoscedastic noise case and we conjecture that our bounds also hold in this situation.

Conclusion

We have provided a joint analysis of averaging and acceleration for non-strongly convex quadratic functions in a single framework, both with noiseless and noisy gradients. This allows us to define a class of algorithms that can benefit simultaneously from the known improvements of averaging and acceleration: faster forgetting of initial conditions (for acceleration), and better robustness to noise when the noise covariance is proportional to the Hessian (for averaging).

Our current analysis of our class of algorithms in Eq. (2.5), that considers two different affine combinations of previous iterates (instead of one for traditional acceleration), is limited to quadratic functions; an extension of its analysis to all smooth or self-concordant-like functions would widen its applicability. Similarly, an extension to least-squares regression with natural heteroscedastic stochastic gradient, as suggested by our simulations, would be an interesting development. At this point, it is tempting to consider algorithms which use the last three iterates rather than the last two. In particular, we investigate the effect of blending acceleration and averaging in Chapter 3.

Appendix

2.A Additional Experimental Results

In this appendix, we provide additional experimental results to illustrate our theoretical results.

2.A.1 Deterministic Convergence

Comparaison for d = 1. In Figure 2.A.1, we minimize a one-dimensional quadratic function f (✓) = 1 2 ✓ 2 for a fixed step-size ↵ = 1/10 and different step-sizes . In the left plot, we compare Acc-GD, HB and Av-GD. We see that HB and Acc-GD both oscillate and that Acc-GD leverages strong convexity to converge faster. In the right plot, we compare the behavior of the algorithm for different values of . We see that the optimal rate is achieved for = ⇤ defined to be the one for which there is a double coalescent eigenvalue, where the convergence is linear at speed O(1 p ↵L) n . When > ⇤ , we are in the real case and when < ⇤ the algorithm oscillates to the solution.

Comparison between the different eigenspaces. (i.e., not appearing as strongly convex since nµ remains small). The convergence is at the beginning the same for HB and Acc-GD, with oscillation at speed O(1/n 2), since the small eigenvalue prevents Acc-GD from having a linear convergence. Then for large n, the convergence becomes linear for Acc-GD, since µn becomes large. In the right plot, we optimize a quadratic function in dimension d = 5 with eigenvalues from 1 to 0.1. We show the function values of the projections of the iterates ⌘ n on the different eigenspaces. We see that high eigenvalues first dominate, but converge quickly to zero, whereas small ones keep oscillating, and converge more slowly.

Comparison for d = 20. In Figure 2.A.3, we optimize two 20-dimensional quadratic functions with different eigenvalues with Av-GD, HB and Acc-GD for a fixed step-size = 1/10. In the left plot, the eigenvalues are 1/k 2 and in the right one, they are 1/k 8 , for k = 1, . . . , d. We see that in both cases, Av-GD converges at a rate of O(1/n) and HB at a rate of O(1/n 2

). For Acc-GD the convergence is linear when µ is large (left plot) and becomes sublinear at a rate of O(1/n 2

) when µ becomes small (right plot).

2.A.2 Noisy Convergence with Unstructured Additive Noise

We optimize the same quadratic function, but now with noisy gradients. We compare our algorithm to other stochastic accelerated algorithms, that is, AC-SA [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF], SAGE [START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF] and Acc-RDA [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF], which are presented in Appendix 2.G. For all these algorithms (and ours) we take the optimal step-sizes defined in these papers. We plot the results averaged over 10 replications.

We consider in Figure 2.A.4 an i.i.d. zero mean noise of variance C = I. We see that all the accelerated algorithms achieve the same precision whereas Av-GD with constant step-size does not converge and Acc-Gd diverges. However SAGE and AC-SA are anytime algorithms and are faster at the beginning since their step-sizes are decreasing and not a constant (with respect to n) function of the horizon N .

2.B Proofs of Section 2.2

2.B.1 Proof of Theorem 2 Let (P n , Q n , R n ) 2 (R[X]
) 3 for all n 2 N be a sequence of polynomials. We consider the iterates defined for all n 2 N ⇤ by

✓ n+1 = P n (H)✓ n + Q n (H)✓ n 1 + R(H)q, started from ✓ 0 = ✓ 1 2 R d . The ✓ ⇤ -stationarity property gives for n 2 N ⇤ : ✓ ⇤ = P n (H)✓ ⇤ + Q n (H)✓ ⇤ + R n (H)q.
Since ✓ ⇤ = H 1 q we get for all q 2 R d

H 1 q = P n (H)H 1 q + Q n (H)H 1 q + R n (H)q.
For all q 2 R d we apply this relation to vectors q = H q:

q = P n (H)q + Q n (H)q + R n (H)H q 8q 2 R d ,
and we get

I = P n (H) + Q n (H) + R n (H)H 8n 2 N ⇤ .
Therefore there are polynomials ( Pn , Qn ) 2 (R[X]) 2 and q n 2 R for all n 2 N ⇤ such that we have for all n 2 N:

P n (X) = (1 q n )I + X Pn (X) Q n (X) = q n I + X Qn (X) R n (X) = ( Pn (X) + Qn (X)).
(2.14)

The n-scalability property means that there are polynomials (P, Q) 2 (R[X]) 2 independent of n such that:

P n (X) = n n + 1 P (X), Q n (X) = n 1 n + 1 Q(X).
And in connection with Eq. ( 2.14) we can rewrite P and Q as:

P (X) = p + X P (X), Q(X) = q + X Q(X),
with (p, q) 2 R 2 and ( P , Q) 2 (R[X]) 2 . Thus for all n 2 N:

q n = n 1 n + 1 q (2.15) Qn (X) = n 1 n Q(X) n n + 1 p = (1 q n ) (2.16) Pn (X) = n n + 1 P (X).
Eq. ( 2.15) and Eq. (2.16) give:

n n + 1 p = ✓ 1 n 1 n + 1 q◆ .
Thus for n = 1, we have p = 2. Then n 1 n+1 q = 2n n+1 1 = n 1 n+1 and q = 1. Therefore

P n (H) = 2n n + 1 I + n n + 1 P (H)H Q n (H) = n 1 n I + Q(H)H R n (H) = ✓ n P (H) + (n 1) Q(H) n + 1 ◆ .
We let Ā = ( P + Q) and B = Q so that we have:

P n (H) = 2n n + 1 ✓ I ✓ Ā(H) + B(H) 2 ◆ H ◆ Q n (H) = n 1 n I B(H)H R n (H) = ✓ n Ā(H) + B(H) n + 1 ◆ ,
and with n = ✓ n ✓ ⇤ for all n 2 N, the algorithm can be written under the form:

n+1 =  I ✓ n n+1 Ā(H)+ 1 n+1 B(H) ◆ H n + ✓ 1 2 n+1 ◆ ⇥ I B(H)H ⇤ ( n n 1 ).

2.B.2 Av-GD as Two-Steps Algorithm

We show now that when the averaged iterate of Av-GD is seen as the main iterate we have that Av-GD with step-size 2 R is equivalent to:

✓ n+1 = 2n n + 1 ✓ n n 1 n + 1 ✓ n 1 n + 1 f 0 n✓ n (n 1)✓ n 1 .
We remind

n+1 = n f 0 ( n ), ✓ n+1 = ✓ n + 1 n + 1 ( n+1 ✓ n ).
Thus, we have: 2.C Proof of Section 2.3

✓ n+1 = ✓ n + 1 n + 1 ( n+1 ✓ n ) = ✓ n + 1 n + 1 ( n f 0 ( n ) ✓ n ) = ✓ n + 1 n + 1 (✓ n + (n 1)(✓ n ✓ n 1 ) f 0 (✓ n + (n 1)(✓ n ✓ n 1 )) ✓ n ) = 2n n + 1 ✓ n n 1 n + 1 ✓ n 1 n + 1 f 0 (n✓ n (n 1)✓ n 1 ). 0 1 2 3 4 0 1 2 h i ↵h i r + i  1 r i 1 i > 0 r + i = 1 r i = 1 i = 0

2.C.1 Proof of Lemma 1

The discriminant i is strictly positive when ↵+ 2 2 h i ↵ > 0. This is always true for ↵ strictly negative. For ↵ positive and for h i 6 = 0, this is true for

| ↵+ 2 | > p ↵/h i . Thus the discriminant i is strictly positive for ↵ < 0 or ↵ 0 and n < ↵ 2 p ↵/h i or > ↵ + 2 p ↵/h i o .
Then we determine when the modulus of the eigenvalues is less than one (which corresponds to 1  r i  r + i  1).

r + i  1 , v u u t h i ✓ ↵ + 2 ◆ 2 h i ↵ !  ✓ + ↵ 2 ◆ h i , h i ✓ + ↵ 2 ◆ 2 h i ↵ !  ✓ + ↵ 2 ◆ h i 2 and ↵ + 2 0 , h i ↵ 0 and ↵ + 2 0 , ↵ 0 and ↵ + 0.
Moreover, we have :

r i 1 , v u u t h i ✓ + ↵ 2 ◆ 2 h i ↵ !  2 ✓ + ↵ 2 ◆ h i , h i ✓ + ↵ 2 ◆ 2 h i ↵ !   2 ✓ + ↵ 2 ◆ h i 2 and 2 ✓ + ↵ 2 ◆ h i 0 , h i ✓ + ↵ 2 ◆ 2 h i ↵ !  4 4 ✓ + ↵ 2 ◆ h i + ✓ + ↵ 2 ◆ h i 2 and ✓ + ↵ 2 ◆  2/h i , h i ↵  4 4 ✓ + ↵ 2 ◆ h i and 2  2/h i ↵ 2 ,  2/h i ↵/2 and  4/h i ↵. Figure 2.
C.1 (where we plot all the constraints we have so far) enables to conclude that the discriminant i is strictly positive and the algorithm is stable when the following three conditions are satisfied:

↵ 0 ↵ + 2  4/h i ↵ + 2 p ↵/h 1 .
For any of those ↵ et we will have:

⌘ i n = c 1 (r i ) n + c 2 (r + i ) n . Since ⌘ i 0 = 0, c 1 + c 2 = 0 and for n = 1, c 1 = ⌘ i 1 /(r i r + i )
; we thus have:

⌘ i n = ⌘ i 1 2 (r + i ) n (r i ) n p i .
Thus, we get the final expression:

( i n ) 2 h i = ( i 1 ) 2 4n 2 ⇥ r i + p i ⇤ n ⇥ r i p i ⇤ n 2 i /h i .

2.C.2 Proof of Lemma 2

The discriminant i is strictly negative if and only if

↵+ 2 2 h i ↵ < 0. This implies | ↵+ 2 | < p ↵/h i . The modulus of the eigenvalues is |r ± i | 2 = 1 h i .
Thus the discriminant i is strictly negative and the algorithm is stable for For any of those ↵ et we have:

↵, 0 ↵ + < p ↵/h i , as shown in Figure 2.C.2. 0 1 2 3 4 0 1 2 h i ↵h i 1 < 0 |r ± i |  1 |r ± i | = 1 i = 0
⌘ i n = [c 1 cos(! i n) + c 2 sin(! i n)]⇢ n i , with ⇢ i = p 1 h i , sin(! i ) = p i /⇢ i and cos(! i ) = r i /⇢ i . Since ⌘ i 0 = 0, c 1 = 0 and we have for n = 1, c 2 = ⌘ i 1 /(sin(! i )⇢ i ). Therefore ⌘ i n = ⌘ i 1 sin(! i n) p i (1 h i ) n/2 ,
and

( i n ) 2 h i = ( i 1 ) 2 n 2 sin 2 (! i n) sin 2 (! i )/h i (1 h i ) n 1 .

2.C.3 Coalescing Eigenvalues

When = 2 p ↵/h i ↵, the discriminant i is equal to zero and we have a double real eigenvalue:

r i = 1 p ↵h i .
Thus the algorithm is stable for ↵ < 4 h i . For any of those ↵ et we have:

⌘ i n = (c 1 + nc 2 )r n . This gives with ⌘ i 0 = 0, c 1 = 0 and c 2 = ⌘ i 1 /r. Therefore ⌘ i n = n⌘ i (1 p ↵h i ) n 1 ,
and:

( i n ) 2 h i = h i ( i 1 ) 2 (1 p ↵h i ) 2(n 1) .
In the presence of coalescing eigenvalues the convergence is linear if 0 < ↵ < 4/h i and h i > 0, however one might worry about the behavior of (( i n ) 2 h i ) n when h i becomes small. Using the bound x 2 exp( x)  1 for x  1, we have for ↵ < 4/h i :

h i (1 p ↵h i ) 2n = h i exp(2n log(|1 p ↵h i |))  hi exp( 2n min{ p ↵h i , 2 p ↵h i })  h i min{ p ↵h i , 2 p ↵h i } 2  max ⇢ 1 ↵ , h i (2 p ↵h i ) 2 .
Therefore we always have the following bound for ↵ < 4/h i :

( i n ) 2 h i  ( i 1 ) 2 4n 2 max ⇢ 1 ↵ , h i (2 p ↵h i ) 2 .
Thus for ↵h i  1 we get:

( i n ) 2 h i  ( i 1 ) 2 4n 2 ↵ .

2.D Proof of Theorem 3

2.D.1 Sketch of the Proof

We divide the domain of validity of Theorem 3 in three subdomains as explained in Figure 2.D.4. On the domain described in Figure 2.D.1 we have a first bound on the iterate ⌘ i n :

Lemma 3. For 0  ↵  1/h i and 1 p

1 ↵h i < h i < 1 + p
1 ↵h i , we have:

(⌘ i n ) 2  (⌘ i 1 ) 2 ↵h i .
And on the domain described Figure 2.D.2 we also have:

Lemma 4. For 0  ↵  1/h i and  ↵ we have:

(⌘ i n ) 2  2(⌘ i 1 ) 2 ↵h i .
These two lemmas enable us to prove the first bound of Theorem 3 since the domain of this theorem is included in the intersection of the two domains of these lemmas as shown in Figure 2.D.4.

Then we have the following bound on domain described in Figure 2.D.3:

Lemma 5. For 0  ↵  2/h i and 0   2/h i ↵, we have:

|⌘ i n |  min ( 2 p 2n p (↵ + )h i , 4 (↵ + )h i ) .

2.D.2 Outline of the Proofs of the Lemmas

-We find a Lyapunov function G from R 2 to R such that the sequence (G(⌘ i n , ⌘ i n 1 )) decrease along the iterates. -We also prove that G(⌘ i n , ⌘ i n 1 ) dominates ck⌘ i n k 2 when we want to have a bound on k⌘ i n k 2 of the form 1 c G(⌘ i 1 , ⌘ i 0 ) = 1 c G(✓ i 0 ✓ i ⇤ , 0).
For readability, we remove the index i and take h i = 1 without loss of generality.

2.D.3 Proof of Lemma 3

We first consider a quadratic Lyapunov function

✓ ⌘ n ⌘ n 1 ◆ > G 1 ✓ ⌘ n ⌘ n 1 ◆ with G 1 = ✓ 1 ↵ 1 ↵ 1 1 ↵ ◆ . We note that G 1 is symmetric positive semi-definite for ↵  1. We recall F i = ✓ 2 (↵ + ) 1 1 0 ◆ .
For the result to be true we need for 0  ↵  1 and 1

p 1 ↵ < < 1 + p 1 ↵ two properties: F > i G 1 F i 4 G 1 , (2.17) and ↵ ✓ 1 0 0 0 ◆ 4 G 1 . (2.18)
Proof of Eq. (2.18). We have:

G 1 ↵ ✓ 1 0 0 0 ◆ = (1 ↵) ✓ 1 1 1 1 ◆ < 0 for ↵  1.
Proof of Eq. ( 2.17

). Since 7 ! F i ( ) > G 1 F i ( ) G 1 is convex in (G
1 is symmetric positive semi-definite), we only have to show Eq. ( 2.17) for the boundaries of the interval in . For

x 2 R ⇤ + : ✓ x 2 x x 1 0 ◆ > ✓ 1 x 2 x 2 x 2 ◆ ✓ x 2 x x 1 0 ◆ ✓ 1 x 2 x 2 x 2 ◆ = (1 x 2 ) 2 ✓ 1 0 0 0 ◆ 4 0.
This especially shows Eq. ( 2.17) for the boundaries of the interval with x = ± p 1 ↵.

Bound. Thus, because ⌘ 0 = 0, we have

↵⌘ 2 n+1  ⇥ > n G 1 ⇥ n  ⇥ > n 1 G 1 ⇥ n 1  ⇥ > 0 G 1 ⇥ 0  ⌘ 2 1
. This shows that for 0  ↵  1/h i and 1 p

1 ↵h i < h i < 1 + p 1 ↵h i : (⌘ i n ) 2  (⌘ i 1 ) 2 ↵h i .

2.D.4 Proof of Lemma 4

We consider now a second Lyapunov function G

2 (⌘ n , ⌘ n 1 ) = (⌘ n r⌘ n 1 ) 2 (⌘ n 1 ) 2 . We have: G 2 (⌘ n , ⌘ n 1 ) = (⌘ n r⌘ n 1 ) 2 ⌘ 2 n 1 = (r⌘ n 1 (1 )⌘ n 2 ) 2 ⌘ 2 n 1 = (r 2 )⌘ 2 n 1 + (1 ) 2 ⌘ 2 n 2 2(1 )r⌘ n 1 ⌘ n 2 = ((1 )⌘ 2 n 1 + (1 )(r 2 )⌘ 2 n 2 2(1 )r⌘ n 1 ⌘ n 2 = (1 )[(⌘ n 1 r⌘ n 2 ) 2 (⌘ n 2 ) 2 ]. = (1 )G 2 (⌘ n 1 , ⌘ n 2 ).
Where we have used twice r 2

= (1 ) and

⌘ n = 2r⌘ n 1 (1 )⌘ n 2 . Moreover G 2 (⌘ n , ⌘ n 1
) can be rewritten as:

G 2 (⌘ n , ⌘ n 1 ) = (1 ↵ + 2 )(⌘ n ⌘ n 1 ) 2 + ↵ 2 (⌘ n 1 ) 2 + ↵ + 2 (⌘ n ) 2 .
Thus for ↵ +  2 and  ↵ we have:

↵ 2 (⌘ n ) 2  G 2 (⌘ n , ⌘ n 1 ) = (1 ) n 1 G 2 (⌘ 1 , ⌘ 0 ) = (1 ) n 1 (⌘ 1 ) 2 .
Therefore for ↵ +  2/h i and  ↵, we have:

(⌘ i n ) 2  2(⌘ i 1 ) 2 ↵h i .

2.D.5 Proof of Lemma 5

We may write ⌘ n as

⌘ n = r⌘ n 1 + (r + ) n + (r ) n . Moreover, we have: |(r + ) n + (r ) n |  2, therefore for ↵ +  2, |⌘ n |  r|⌘ n 1 | + 2  2 1 r n 1 r  2 1 (1 ( ↵+ 2 )) n ( ↵+ 2 )
.

Thus |⌘ n |  2 ( ↵+ 2 )h
.

Moreover for all u 2 [0, 1] and n 1 we have

1 (1 u) n  p nu, since 1 (1 u) n  1 and 1 (1 u) n = u P (1 u) k  nu. Thus |⌘ n |  2 p n q ( ↵+ 2 )
.

Therefore for 0  ↵  2/h i and ↵ +  2/h i we have:

|⌘ i n |  min ( 2 p 2n p (↵ + )h i , 4 (↵ + )h i ) .

2.E Lower Bounds

We have the following lower-bound for the bound shown in Corollary 1, which shows that depending on which of the two terms dominates, we may always find a sequence of functions that makes it tight.

Proposition 5. Let L 0. For all sequences 0  ↵ n  1/L and 0  n  2/L ↵ n , such that ↵ n + n = o(n↵ n ) there exists a sequence of one-dimensional quadratic functions (f n ) n with second-derivative less than L such that:

lim ↵ n n 2 (f n (✓ n ) f n (✓ ⇤ )) = k✓ 0 ✓ ⇤ k 2 2 . For all sequences 0  ↵ n  1/L and 0  n  2/L ↵ n , such that n↵ n = o(↵ n + n ),
there exists a sequence of one-dimensional quadratic functions (g n ) n with secondderivative less than L such that:

lim n(↵ n + n )(g n (✓ n ) g n (✓ ⇤ )) = (1 exp( 2)) 2 k✓ 0 ✓ ⇤ k 2 4 .
Proof of the first lower-bound. For the first lower bound we consider 0

 ↵ n  1/L and 0  n  2/L ↵, such that ↵ n + n = o(n↵ n ). We define f n = ⇡ 2 /(4↵ n n 2
) and we consider the sequence of quadratic functions f n (✓) = fn✓ 22 . We consider the iterate (⌘ n ) n defined by our algorithm. We will show that

lim ↵ n f n (⌘ n ) = ⌘ 2 1 2 .
We have, from Lemma 2,

f n (⌘ n ) = ⌘ 2 n f n 2 = ⌘ 2 1 sin 2 (! n n)⇢ 2n n 2↵ n (1 ⇡ 2 (↵n+ n) 2 (4↵nn) 2 )
.

Moreover,

⇢ 2n n = ✓ 1 n ⇡ 2 4↵ n n 2 ◆ n = exp ✓ n log ✓ 1 n ⇡ 2 4↵ n n 2 ◆◆ = 1 + o(1), since n ↵nn = o(1). Also, 1 ⇡ 2 (↵n+ n) 2 (4↵nn) 2 = 1 + o(1), since ↵ n + n = o(n↵ n ). Moreover sin(! n ) = p n ⇢ n = p f n q ↵ n (↵n+ n) 2 4 f n p 1 n f n = ⇡/(2n) + o(1/n), thus ! n = ⇡/(2n) + o(1/n) and sin(n! n ) = 1 + o(1).
Proof of the second lower-bound. We consider now the situation where the second bound is active. Thus we take sequences

(↵ n ) and ( n ), such that n↵ n = o(↵ n + n ). We define g n = that: lim n(↵ n + n )(g n (✓ n ) g n (✓ ⇤ )) = (1 exp( 2)) 2 k✓ 0 ✓ ⇤ k 2 4 .
We will use Lemma 1. We first have

n = ✓ ↵ n + n 2 ◆ 2 g 2 n ↵ n g n = g n ✓ ↵ n + n 2 ◆ 1 n . Thus (n n )/g n = ↵n+ n 2 and p n = s ✓ 1 n ◆ 2 + 2↵ n n(↵ n + n ) = 1 n r 1 + 2↵ n n ↵ n + n = 1 n + ↵ n ↵ n + n + o ✓ ↵ n ↵ n + n ◆ . Moreover r n = 1 ↵ n + n 2 g n = 1 1 n 2↵ n ↵ n + n . Thus r + = 1 ↵ n ↵ n+ n + o ✓ ↵ n ↵ n + n ◆ , and
r n + = exp(n log(r + )) = exp ✓ n↵ n ↵ n + n ◆ + o ✓ n↵ n ↵ n + n ◆ = 1 + o(1). Furthermore r = 1 2 n 3↵ n ↵ n+ n + o ✓ ↵ n ↵ n + n ◆ ,
and

r n = exp(n log(r + )) = exp ✓ 2 3↵ n n ↵ n + n ◆ + o ✓ n↵ n ↵ n + n ◆ = exp( 2) + o(1). Thus (r n + r n ) 2 = (1 exp( 2)) 2 + o(1)
. Finally, we have:

(↵ n + n )n[g n (✓ n ) g n (✓ ⇤ )] = ↵ n + n 2n k✓ 0 ✓ ⇤ k 2 [r n + r n ] 2 4 n /g n = k✓ 0 ✓ ⇤ k 2 4 [r n + r n ] 2 = k✓ 0 ✓ ⇤ k 2 4 (1 exp( 2)) 2 + o(1).
2.F Proofs of Section 2.4

2.F.1 Proofs of Theorem 4 and Theorem 5

We decompose again vectors in an eigenvector basis of H with

⌘ i n = p > i ⌘ n and " i n = p > i " n : ⌘ i n+1 = (1 ↵h i )⌘ i n + (1 h i )(⌘ i n ⌘ i n 1 ) + (n↵ + )" i n+1 .
We denote by

⇠ i n+1 = ✓ [n↵ + ]" i n+1 0 ◆
and we have the reduced equation:

⇥ i n+1 = F i ⇥ i n + ⇠ i n+1 .
Unfortunately F i is not Hermitian and this formulation will not be convenient for calculus. Without loss of generality, we assume

r i 6 = r + i even if it means having r i r + i goes to 0 in the final bound. Let Q i = ✓ r i r + i 1 1 ◆ be the transfer matrix of F i , i.e., F i = Q i D i Q 1 i with D i = ✓ r i 0 0 r + i ◆ and Q 1 i = 1 r i r + i ✓ 1 r + i 1 r i ◆ .
We can reparametrize the problem in the following way:

Q 1 i ⇥ i n+1 = Q 1 i F i ⇥ i n + Q 1 i ⇠ i n+1 = Q 1 i F i Q i Q 1 i ⇥ i n + Q 1 i ⇠ i n+1 = D i (Q 1 i ⇥ i n ) + Q 1 i ⇠ i n+1 . With ⇥i n = Q 1 i ⇥ i n and ⇠i n = Q 1 i ⇠ i n we now have: ⇥i n+1 = D i ⇥i n + ⇠i n+1 , (2.19)
with now D i Hermitian (even diagonal). Thus it is easier to tackle using standard techniques for stochastic approximation [see, e.g., Polyak andJuditsky, 1992, Bach and[START_REF] Bach | Non-asymptotic analysis of stochastic approximation algorithms for machine learning[END_REF]:

⇥i n = D n 1 i ⇥i 1 + n X k=2 D n k i ⇠i k . Let M i = ✓ h 1/2 i h 1/2 i 0 0
◆ , we then get using standard martingale square moment inequalities, since for n 6 = m, " i n and " i m are uncorrelated (i.e., E[" i n " i m ] = 0):

EkM i ⇥i n k 2 = kM i D n 1 i ⇥i 1 k 2 + E n X k=2 kM i D n k i ⇠i k k 2 .
This is a bias-variance decomposition; the left term only depends on the initial con-dition and the right term only depends on the noise process.

We have with

M i = ✓ h 1/2 i h 1/2 i 0 0 ◆ , M i Q 1 i = ✓ 0 h 1/2 i 0 0 ◆ , and M i ⇥i n = ✓p h i ⌘ i n 1 0 ◆ .
Thus, we have access to the function values through:

kM i ⇥i n k 2 = h i (⌘ i n 1 ) 2 .
Moreover we have

⇥ i 1 = ✓ i 1 /(r i r + i ) i 1 /(r i r + i ) ◆ .
Thus

kM i D n 1 i ⇥i 1 k 2 = ( i 1 ) 2 h i (r + i ) n 1 (r i ) n 1 2 (r + i r i ) 2 .
This is the bias term we have studied in Section 2.3.3 which we bound with Theorem 3. The variance term is controlled by the next proposition.

Proposition 6. With E[(" i n ) 2 ] = c i for all n 2 N, for ↵  1/h i and 0   2/h i ↵, we have 1 (n 1) 2 E n X k=2 kM i D n k i ⇠i k k 2  min ( 2(↵(n 1) + ) 2 ↵ (4 (↵ + 2 )h i )(n 1) 2 c i h i , 16((n 1)↵ + ) 2 (n 1)(↵ + ) 2 c i h i , 2 (↵(n 1) + ) 2 (n 1)↵ c i , 8(↵(n 1) + ) 2 ↵ + c i
) .

The last two bounds prove Theorem 4.

We note that if we restrict to  3/(2h i ) ↵/2, then 4 (↵ + 2 )h i 1 and the first bound of Proposition 6 is simplified to 2(↵(n 1)+ ) 2 ↵ (n 1) 2 c i h i . This allows to conclude to prove Theorem 5.

2.F.2 Proof of Corollary 3

We let ⌫ = k✓ 0 ✓⇤k p L tr(CH 1 )

and consider three different regimes depending on ⌫ and

L. If ⌫ < 1/L, we have ⌫/N < 1/L and thus ↵ = ⌫/N and = ⌫. Therefore k✓ 0 ✓ ⇤ k 2 N 2 ↵ + (↵N + ) 2 ↵ N 2 tr(CH 1 ) = k✓ 0 ✓ ⇤ k 2 ⌫N + 4 tr(CH 1 ) N  p L tr(CH 1 )k✓ 0 ✓ ⇤ k N + 4 tr(CH 1 ) N  5 tr(CH 1 ) N ,
where we have used

p Lk✓ 0 ✓ ⇤ k < p tr(CH 1 ) since ⌫ < 1/L. If ⌫ > 1/L and ⌫ < N/L, we have ↵ = ⌫/N and = 1/L. Therefore k✓ 0 ✓ ⇤ k 2 N 2 ↵ + (↵N + ) 2 ↵ N 2 tr(CH 1 )  k✓ 0 ✓ ⇤ k 2 ⌫N + 4 tr(CH 1 ) L⌫N  p L tr(CH 1 )k✓ 0 ✓ ⇤ k N + 4 tr(CH 1 ) N  5 p L tr(CH 1 )k✓ 0 ✓ ⇤ k N ,
where we have used

p Lk✓ 0 ✓ ⇤ k > p tr(CH 1 ) since ⌫ > 1/L. If ⌫ > N/L, we have ↵ = 1/L and = 1/L. Therefore k✓ 0 ✓ ⇤ k 2 N 2 ↵ + (↵(N 1) + ) 2 ↵ N 2 tr(CH 1 ) = Lk✓ 0 ✓ ⇤ k 2 N 2 + tr(CH 1 )  Lk✓ 0 ✓ ⇤ k 2 N 2 + Lk✓ 0 ✓ ⇤ k 2 N 2  2Lk✓ 0 ✓ ⇤ k 2 N 2 ,
where we have used that the real bound in Proposition 6 is in fact in (N 1)↵ + , (see Lemma 6) and that tr(CH 1 )

< Lk✓ 0 ✓⇤k 2 N 2 since ⌫ > N/L.

2.F.3 Proof of Proposition 6

Proof Outline

To prove Proposition 6 we will use Lemmas 6, 7 and 8, that are stated and proved in Section 2.F.3.

We want to bound E[

P n k=2 kM i D n k i ⇠i k k 2
] and according to Lemma 6, we have an explicit expansion using the roots of the characteristic polynomial:

EkM i D n k i ⇠i k k 2 = h i ((k 1)↵ + ) 2 E[(" i ) 2 ] [(r i ) n k (r + i ) n k ] 2 (r i r + i ) 2 .
Thus, by bouding (k 1)↵ + by (n 1)↵ + , we get

E n X k=2 kM i D n k i ⇠i k k 2  h i ((n 1)↵ + ) 2 E[" i 2 ] n X k=2 [(r i ) n k (r + i ) n k ] 2 (r i r + i ) 2
.

(2.20)

Then, we have from Lemma 7 the inequality:

n 2 X k=0 [(r i ) k (r + i ) k ] 2 [(r i ) (r + i )] 2  2 h i 4↵ h 2 i (1 ( 1 4 ↵ + 1 2 )h i ) . Therefore E n X k=2 kM 1/2 i D n k i ⇠i k k 2  E[" i 2 ] h i ((n 1)↵ + ) 2 4↵ 2 h i (1 ( 1 4 ↵ + 1 2 )h i )
.

This allows to prove the first part of the bound. The other parts are much simpler and are done in Lemma 8. Thus, adding these bounds gives for ↵  1/h i and

0   2/h i ↵: 1 (n 1) 2 E n X k=2 kM i D n k i ⇠i k k 2  min ( 2(↵(n 1) + ) 2 ↵ (n 1) 2 (4 (↵ + 2 )h i ) c h i , 16((n 1)↵ + ) 2 (n 1)(↵ + ) 2 c h i , 2 (↵(n 1) + ) 2 (n 1)↵ c i , 8((n 1)↵ + ) 2 ↵ + c i
) .

Some Technical Lemmas

We first compute an explicit expansion of the noise term as a function of the eigenvalues of the dynamical system. Lemma 6. For all ↵  1/h i and 0   2/h i ↵ we have

EkM i D n k i ⇠i k k 2 = h i ((k 1)↵ + ) 2 E[(" i ) 2 ] [(r i ) n k (r + i ) n k ] 2 (r i r + i ) 2 .
Proof. We first turn the Euclidean norm into a trace, using that tr[AB] = tr[BA] for two matrices A and B and that tr[x] = x for a real x. 2.21) This enables us to separate the noise term from the rest of the formula. Then we compute the latter from the definition of ⇠i k in Eq. (2.19) :

EkM i D n k i ⇠i k k 2 = TrD n k i M i > M i D n k i E[ ⇠i k ( ⇠i k ) > ], (
E[ ⇠i k ( ⇠i k ) > ] = ((k 1)↵ + ) 2 (r i r + i ) 2 E[(" i ) 2 ] ✓ 1 1 1 1 ◆ .
And the first part of Eq. ( 2.21) is equal to:

D n k i M i > M i D n k i = h i (r i ) 2(n k) (r i ) (n k) (r + i ) (n k) (r i ) (n k) (r + i ) (n k) (r + i ) 2(n k) ! , because D i = ✓ r i 0 0 r + i ◆ and M i = ✓ h 1/2 i h 1/2 i 0 0 ◆ . Therefore: EkM i D n k i ⇠i k k 2 = h i ((k 1)↵ + ) 2 (r i r + i ) 2 E[" i 2 ][(r i ) n k (r + i ) n k ] 2 .
In the following lemma, we bound a certain sum of powers of the roots.

Lemma 7. For all ↵  1/h i and 0   2/h i ↵ we have

n 2 X k=0 ⇥ (r i ) k (r + i ) k ] 2 [(r i ) (r + i ) ⇤ 2  2 h i 4↵ h 2 i (1 ( 1 4 ↵ + 1 2 )h i )
.

We first note that when the two roots become close, the denominator and the numerator will go to zero, which prevents from bounding the numerator easily. We also note that this bound is very tight since the difference between the two terms goes to zero when n goes to infinity.

Proof. We first expand the square of the difference of the powers of the roots and compute their sums.

n 2 X k=0 ⇥ (r i ) k (r + i ) k ⇤ 2 = n 2 X k=0 ⇥ r + i 2k + r i 2k 2(r + i r i ) k ⇤ = 1 r + i 2(n 1) 1 r + i 2 + 1 r i 2(n 1) 1 r i 2 2 1 (r + i r i ) n 1 1 (r + i r i ) = 1 1 r + i 2 + 1 1 r i 2 2 1 (r + i r i )  r + i 2(n 1) 1 r + i 2 + r i 2(n 1) 1 r i 2 2 (r + i r i ) (n 1) 1 (r + i r i ) = 1 1 r + i 2 + 1 1 r i 2 2 1 (r + i r i ) I n 1 , with I n =  r + i 2n 1 r + i 2 + r i 2n 1 r i 2 2 (r + i r i ) n 1 (r + i r i )
. This sum is therefore equal to the sum of one term we will compute explicitly and one other term which will go to zero. We have for the first term:

1 1 r + i 2 + 1 1 r i 2 2 1 (r + i r i ) = (1 r i 2 )(1 (r + i r i )) (1 r i 2 )(1 r + i 2 ) (1 r + i 2 )(1 r i 2 )(1 (r + i r i )) + (1 r + i 2 )(1 (r + i r i )) (1 r i 2 )(1 r + i 2 ) (1 r + i 2 )(1 r i 2 )(1 (r + i r i )) , with (1 r i 2 )(1 (r + i r i )) (1 r i 2 )(1 r + i 2 ) = (1 r i 2 )[(1 (r + i r i )) (1 r + i 2 )] = r + i (1 r i 2 )(r + i r i ),
and

(1 r + i 2 )(1 (r + i r i )) (1 r i 2 )(1 r + i 2 ) = r i (1 r i 2 )(r + i r i ),
and with

⇤ = r + i (1 r i 2 )(r + i r i ) r i (1 r i 2 )(r + i r i ), ⇤ = (r + i r i )[r + i (1 r i 2 ) r i (1 r i 2 )] = (r + i r i )[r + i r i + r + i r i (r + i r i )] = (r + i r i ) 2 [1 + r + i r i ].
Therefore the first term is equal to:

1 1 r + i 2 + 1 1 r i 2 2 1 (r + i r i ) = (r + i r i ) 2 [1 + r + i r i ] (1 r + i 2 )(1 r i 2 )(1 (r + i r i ))
, and the sum can be expanded as:

n 2 X k=0 [(r i ) k (r + i ) k ] 2 [r i r + i ] 2 = [1 + r + i r i ] (1 r + i 2 )(1 r i 2 )(1 (r + i r i )) J n 1 , with J n = In [(r i ) (r + i )] 2 .
Then we simplify the first term of this sum using the explicit values of the roots.

We recall

r ± i = r i ± p i = 1 ↵+ 2 h i ± q ↵+ 2 2 h 2 i ↵h i , therefore r + i r i = r 2 i 2 i = ✓ 1 ✓ ↵ + 2 ◆ h i ◆ 2 ✓ ↵ + 2 ◆ h i 2 + ↵h i = 1 h i , and
(1 r + i 2 )(1 r i 2 ) = [(1 r i )(1 r + i )][(1 + r + i )(1 + r + i )] = [(1 r i + p i )((1 r i p i )][(1+r i + p i )((1 + r i p i )] = [(1 r i ) 2 i ][(1 + r i ) 2 i ][(1 r i ) 2 i ] = 4↵h i ✓ 1 ✓ 1 4 ↵ + 1 2 ◆ h i ◆ . Thus n 2 X k=0 [(r i ) k (r + i ) k ] 2 [(r i ) (r + i )] 2 = 2 h i 4↵ h 2 i (1 ( 1 4 ↵ + 1 2 )h i ) J n 1 .
Even if J n will be asymptotically small, we want a non-asymptotic bound, thus we will show that J n is always positive.

In the real case

[(r i ) (r + i )] 2 0 and using a 2 + b 2 2ab, for all (a, b) 2 R 2 , we have r + i 2n 1 r + i 2 + r i 2n 1 r i 2 2 (r + i r i ) n q (1 r + i 2 )(1 r i 2 )
, and using r

+ i 2 + r i 2 2r i r i we have q (1 r + i 2 )(1 r i 2 )  1 (r + i r i ), since (1 r + i 2 )(1 r i 2 ) [1 (r + i r i )] 2 = 1 r + i 2 r i 2 + (r + i r i ) 2 1 + 2r + i r i (r + i r i ) 2 = 2r + i r i r + i 2 r i 2  0. Thus r + i 2n 1 r + i 2 + r i 2n 1 r i 2 2 (r + i r i ) n 1 (r + i r i ) 0.
and J n 0 in the real case.

In the complex case, [(r i ) (r + i )] 2  0, and using

z 2 + z2  2z z for all z 2 C, we have r + i 2n 1 r + i 2 + r i 2n 1 r i 2  2 (r + i r i ) n q (1 r + i 2 )(1 r i 2 )
, and using r

+ i 2 + r i 2  2r i r i we have q (1 r + i 2 )(1 r i 2 ) 1 (r + i r i ). Thus r + i 2n 1 r + i 2 + r i 2n 1 r i 2 2 (r + i r i ) n 1 (r + i r i )  0.
and J n 0 in the complexe case. Therefore we always have:

J n 0,
and

n 2 X k=0 [(r i ) k (r + i ) k ] 2 [(r i ) (r + i )] 2  2 h i 4↵ h 2 i (1 ( 1 4 ↵ + 1 2 )h i )
.

However we can also bound roughly Eq. (2.20) using Theorem 3 since we recall we have

⌘ i n = [(r i ) n (r + i ) n ] 2 (r i r + i ) 2
. This gives us the following lemma which enables to prove the second part of Proposition 6.

Lemma 8. For all ↵  1/h i and 0   2/h i ↵ we have

E n X k=2 kM i D n k i ⇠i k k 2  E[(" i ) 2 ](n 1)((n 1)↵ + ) 2 min ⇢ 2 ↵ , 8(n 1) ↵ + , 16 h i (↵ + ) 2 .
Proof. From Lemma 6, we get

E n X k=2 kM i D n k i ⇠i k k 2 = h i E[(" i ) 2 ] n X k=2 ((k 1)↵ + ) 2 [(r i ) n k (r + i ) n k ] 2 (r i r + i ) 2  E[(" i ) 2 ](n 1)((n 1)↵ + ) 2 min ⇢ 2 ↵ , 8(n 1) ↵+ , 16 h i (↵+ ) 2 .

2.G Comparison with Additional Other Algorithms

2.G.1 Summary

When the objective function f is quadratic and for correct choices of step-sizes, the AC-SA algorithm of [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF], the SAGE algorithm of [START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF] and the Accelerated RDA algorithm of [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] are all equivalent to:

✓ n+1 = [I n+1 H n+1 ]✓ n + n 2 n + 1 [I n+1 H n+1 ](✓ n ✓ n 1 ) + n+1 " n+1 ,
where we use H n ✓ + " n as an unbiased estimate of the gradient and n as step-size which values will be specified later. [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF] and [START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF] only consider bounded cases by projecting their iterates on a bounded space. [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] deals with the unbounded case and prove the following convergence result: Theorem 6. [Xiao, 2010, Theorem 6].

With E[" n ⌦ " n ] = C, for step-size n  n 1 n with  1/L, we have Ef (✓ n ) f (✓ ⇤ )  4k✓ 0 ✓ ⇤ k 2 n 2 + n 2 tr C 3 .
This result is significantly more general than ours since it is valid for composite optimization and general noise on the gradients.

We now present the different algorithms and show they all share the same form.

2.G.2 AC-SA

Lemma 9. AC-SA algorithm with step size n and n and gradient estimate H n+1 ✓ n + " n+1 is equivalent to:

✓ n+1 = (I n n H n+1 )✓ n + n 1 1 n (I n n H n+1 )(✓ n ✓ n 1 ) + n n " n+1 .
Proof. We recall the general AC-SA algorithm:

-Let the initial points x ag 1 = x 1 , and the step-sizes { n } n1 and { n } n1 be given. Set n = 1

-Step 1. Set x md n = 1 n x n + (1 1 n )x ag n , -Step 2. Call the Oracle for computing G(x md n , ⇠ n ) where E[G(x md n , ⇠ n )] = f 0 (x md n ). Set x n+1 = x n n G(x md n , ⇠ n ), x ag n+1 = 1 n x n+1 + (1 1 n )x ag n , -Step 3. Set n ! n + 1 and go to step 1. When f is quadratic we will have G(x md n , ⇠ n ) = H n+1 x md n " n+1 , thus x n+1 = x n n H n+1 x md n + n " n+1
, and:

x ag n+1 = 1 n x n+1 + (1 1 n )x ag n = 1 n (x n n H n+1 x md n + n " n+1 ) + (1 1 n )x ag n = 1 n ( n x md n + (1 n )x ag n n H n+1 x md n + n " n+1 ) + (1 1 n )x ag n = x md n n n H n+1 x md n + n n " n+1 , and
x md n = 1 n x n + (1 1 n )x ag n = 1 n n 1 x ag n + 1 n (1 n 1 )x ag n 1 + (1 1 n )x ag n = x ag n + n 1 1 n [x ag n x ag n 1 ].
These give the result for ✓ n = x ag n .

2.G.3 SAGE

Lemma 10. The algorithm SAGE with step-sizes L n and ↵ n is equivalent to:

✓ n+1 = (I 1/L n+1 H n+1 )✓ n + (1 ↵ n ) ↵ n+1 ↵ n [I 1/L n+1 H n+1 ](✓ n ✓ n 1 ) + 1/L n+1 " n+1 .
Proof. We recall the general SAGE algorithm:

-Let the initial points x 0 = z 0 = 0, and the step-sizes { n } n1 and {L n } n1 be given.

Set n = 1

-Step 1. Set x n = (1 ↵ n )y n 1 + ↵ n z n 1 , -Step 2. Call the Oracle for computing G(x n , ⇠ n ) where E[G(x n , ⇠ n )] = f 0 (x n ). Set y n = x n 1/L n G(x n , ⇠ n ), z n = z n 1 ↵ 1 n (x n y n ) -Step 3. Set n ! n + 1 and go to step 1.
We have

y n = (I 1/L n H n )x n + n " n ,
and

z n = z n 1 ↵ 1 n (x n y n ) = z n 1 ↵ 1 n [(1 ↵ n )y n 1 + ↵ n z n 1 y n ] = ↵ 1 n y n ↵ 1 n (1 ↵ n )y n 1 .
Thus

x n = (1 ↵ n )y n 1 + ↵ n z n 1 = (1 ↵ n )y n 1 + ↵ n [↵ 1 n 1 y n 1 ↵ 1 n 1 (1 ↵ n 1 )y n 2 ] = y n 1 + (1 ↵ n 1 ) ↵ n ↵ n 1 [y n 1 y n 2 ].
These give the result for ✓ n = y n .

2.G.4 Accelerated RDA Method

Lemma 11. The algorithm AccRDA with step-sizes and ↵ n is equivalent to:

✓ n+1 = (I n+1 H n+1 )✓ n + (1 ↵ n ) ↵ n+1 ↵ n [I n+1 H n+1 ](✓ n ✓ n 1 ) + n+1 " n+1 , with n = ↵n✓n L+ .
Proof. We recall the general Accelerated RDA method:

-Let the initial points w 0 = v 0 , A 0 = 0, g0 = 0 and the step-sizes {↵ n } n1 and { n } n1 be given.

Set n = 1

-Step 1. Set A n = A n 1 + ↵ n and ✓ n = ↵n An . -Step 2. Compute the query point u n = (1 ✓ n )w n 1 + ✓ n v n 1 -Step 3. Call the Oracle for computing g n = G(u n , ⇠ n ) where E[G(u n , ⇠ n )] = f 0 (u n ),
and update the weighted average

gn gn = (1 ✓ n )g n 1 + ✓ n g n . -Step 4. Set v n = v 0 An L+ n gn . -Step 5. Set w n = (1 ✓ n )w n 1 + ✓ n v n .
-Step 6. Set n ! n + 1 and go to step 1.

First we have

v n = v 0 A n L + n gn = v 0 A n L + n [(1 ✓ n )g n 1 + ✓ n g n ] = v 0 A n L + n [(1 ✓ n )g n 1 + ✓ n (H n+1 u n + " n+1 )] = v 0 + (1 ✓ n ) A n (L + n 1 ) (L + n )A n 1 v n 1 A n L + n ✓ n (H n+1 u n + " n+1 )] = v 0 + (1 ✓ n ) A n (L + n 1 ) (L + n )A n 1 v n 1 ↵ n L + n (H n+1 u n + " n+1 )]. With n = we have v n = v n 1 ↵n L+ (H n+1 u n + " n+1 )]
and

w n = (I ↵ n ✓ n L + H n+1 )u n + ↵ n ✓ n L + " n+1 . Since v n 1 = ✓ 1 n 1 w n 1 ✓ 1 n 1 (1 ✓ n 1 )w n 2 , then u n = (1 ✓ n )w n 1 + ✓ n (✓ 1 n 1 w n 1 ✓ 1 n 1 (1 ✓ n 1 )w n 2 ),
and

u n = w n 1 + ↵ n A n 2 ↵ n 1 A n [w n 1 w n 2 ].

2.H Lower Bound for Stochastic Optimization for Least-Squares

In this section, we show a lower bound for optimization of quadratic functions with noisy access to gradients. We follow very closely the framework of [START_REF] Agarwal | Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization[END_REF] and use their notations. The only difference with their Theorem 1 in the different choice of two functions f + i and f i , which we choose to be:

f ± i (x) = c i (x i ± r 2 ) 2 ,
with a non-increasing sequence (c i ) to be chosen later. The function g ↵ that is optimized is thus:

g ↵ (x) = 1 d d X i=1 ( 1 2 + ↵ i )f + i (x) + ( 1 2 ↵ i )f i (x) .
This function is quadratic and its Hessian has eigenvalues equal to 2c i /d. Thus, its largest eigenvalue is 2c 1 /d, which we choose equal to L. Noisy gradients are obtained by sampling d independent Bernoulli random variables b i , i = 1, . . . , d, with parameters ( 1 2 + ↵ i ) and using the gradient of the random function

1 d P d i=1 b i f + i (x) + (1 b i )f i (x)
. The variance of the random gradient is equal to

V = d X i=1 1 d 2 var ⇣ b i ⇥ c i (x i + r/2) c i (x i r/2) ⇤ ⌘ = 1 d 2 d X i=1 c 2 i r 2 (1/4 2 ).
The function g ↵ is minimized for x = ↵ r, and the discrepancy measure between two functions g ↵ and g is greater than

1 d d X i=1 ⇢ inf x f + i (x) + f i (x) inf x f + i (x) inf x f i (x) 1 ↵ i 6 = i 1 d d X i=1 3c i r 2 2 4 1 ↵ i 6 = i 1 d 3c d r 2 2 4 (↵, ).
Since the vectors ↵, 2 { 1, 1} d are so that their Hamming distance (↵, ) d/4

for ↵ 6 = , we have a discrepancy measure greater than 3c d r 2 2 16 . Thus, for a an approximate optimality of " = c d r 2 2 38 , we have, following the proof of Theorem 1 (equation ( 29)) from [START_REF] Agarwal | Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization[END_REF], for N iterations of any method that accesses a random gradient, we have:

1/3 1 2 16Nd 2 + log 2 d log(2/ p e) .
Thus, for d large, we get, up to constants, 2 1/N and thus " r 2 c d N . For c 1 = 2Ld and c i = L p d for the remaining ones, we get (up to constants):

" V L p d N .
This leads to the desired result for N  d.

Chapter 3

Optimal Convergence Rates for Least-Squares Regression through Stochastic Approximation

Abstract

We consider the optimization of a quadratic objective function whose gradients are only accessible through a stochastic oracle that returns the gradient at any given point plus a zero-mean finite variance random error. We present the first algorithm that achieves jointly the optimal prediction error rates for least-squares regression, both in terms of forgetting the initial conditions in O(1/n 2), and in terms of dependence on the noise and dimension d of the problem, as O(d/n). Our new algorithm is based on averaged accelerated regularized gradient descent, and may also be analyzed through finer assumptions on initial conditions and the Hessian matrix, leading to dimensionfree quantities that may still be small in some distances while the "optimal" terms above are large.

This chapter is extracted from the paper Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression, in collaboration with A. Dieuleveut and F. Bach, accepted in the Journal of Machine Learning Research.

We do not present here the extension to kernel regression and content ourself with a brief excerpt of Section 3.5 on tighter bounds. Indeed these two contributions will be presented in the thesis of Aymeric Dieuleveut.

Introduction

Many supervised machine learning problems are naturally cast as the minimization of a smooth function defined on a Euclidean space. This includes least-squares regression, logistic regression [see, e.g., [START_REF] Hastie | The Elements of Statistical Learning[END_REF] or generalized linear models [START_REF] Mccullagh | Generalized Linear Models[END_REF]. While small problems with few or low-dimensional input features may be solved precisely by many potential optimization algorithms (e.g., Newton method), large-scale problems with many high-dimensional features are typically solved with simple gradient-based iterative techniques whose per-iteration cost is small.

In this chapter, we consider a quadratic objective function f whose gradients are only accessible through a stochastic oracle that returns the gradient at any given point plus a zero-mean finite variance random error. In this stochastic approximation framework [START_REF] Robbins | A stochastic approxiation method[END_REF], it is known that two quantities dictate the behavior of various algorithms, namely the covariance matrix V of the noise in the gradients, and the deviation ✓ 0 ✓ ⇤ between the initial point of the algorithm ✓ 0 and any of the global minimizer ✓ ⇤ of f . This leads to a "bias/variance" decomposition [Bach andMoulines, 2013, Hsu et al., 2014] of the performance of most algorithms as the sum of two terms: (a) the bias term characterizes how fast initial conditions are forgotten and thus is increasing in a well-chosen norm of ✓ 0 ✓ ⇤ ; while (b) the variance term characterizes the effect of the noise in the gradients, independently of the starting point, and with a term that is increasing in the covariance of the noise.

For quadratic functions with (a) a noise covariance matrix V which is proportional (with constant 2 ) to the Hessian of f (a situation which corresponds to least-squares regression) and (b) an initial point characterized by the norm k✓ 0 ✓ ⇤ k 2 , the optimal bias and variance terms are known separately from the optimization and statistical theories. On the one hand, the optimal bias dependency after n iterations is propor-

tional to Lk✓ 0 ✓⇤k 2 n 2
, where L is the largest eigenvalue of the Hessian of f . This rate is achieved by accelerated gradient descent [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF][START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF], and is known to be optimal if the number of iterations n is less than the dimension d of the underlying predictors, but the algorithm is not robust to random or deterministic noise in the gradients [START_REF]Smooth optimization with approximate gradient[END_REF][START_REF] Schmidt | Convergence rates of inexact proximal-gradient methods for convex optimization[END_REF][START_REF] Devolder | First-order methods of smooth convex optimization with inexact oracle[END_REF]. On the other hand, the optimal variance term is proportional to 2 d n [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]; it is known to be achieved by averaged gradient descent [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF], for which the bias term only achieves Lk✓ 0 ✓⇤k 2 n instead of Lk✓ 0 ✓⇤k 2 n 2 . Our first contribution in this chapter is to present a novel algorithm which attains optimal rates for both the variance and the bias terms. This algorithm analyzed in Section 3.4 is averaged accelerated gradient descent; beyond obtaining jointly optimal rates, our result shows that averaging is beneficial for accelerated techniques and provides a provable robustness to noise.

While optimal when measuring performance in terms of the dimension d and the initial distance to optimum k✓ 0 ✓ ⇤ k 2 , these rates are not adapted in many situations where either d is larger than the number of iterations n (i.e., the number of observations for regular stochastic gradient descent) or Lk✓ 0 ✓ ⇤ k 2 is much larger than n 2 . Our second contribution is to provide in Section 3.5 an analysis of a new algorithm (based on some additional regularization) that can adapt our bounds to finer assumptions on ✓ 0 ✓ ⇤ and the Hessian of the problem, leading in particular to dimension-free quantities that can thus be extended to the Hilbert space setting (in particular for non-parametric estimation).

This chapter is organized as follows: in Section 3.2, we present the main problem we tackle, namely least-squares regression, then introduce the two algorithms that we consider in Section 3. Theorem 7 ⌃ \ Table 3.1 -Organization of Chapter 3. ⌃: We extend results from [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] to the setting in which extra regularization is added; \: it is still an open problem to get results in the accelerated setting for a multiplicative noise oracle.

Section 3.2.3. In Section 3.3, we present new results for averaged stochastic gradient descent that set the stage for Section 3.4, where we present our main novel result leading to an accelerated algorithm which is robust to noise. Our tighter analysis of convergence rates based on finer dimension-free quantities is presented in Section 3.5. Organization of the main results is summarized in the Table 3.1 bellow.

Least-Squares Regression

In this section, we present our least-squares regression framework, which is risk minimization with the square loss, together with the main assumptions regarding our model and our algorithms. These algorithms will rely on stochastic gradient oracles, which will come in two kinds, an additive noise which does not depend on the current iterate, which will correspond in practice to the full knowledge of the covariance matrix, and a "multiplicative/additive" noise, which corresponds to the regular stochastic gradient obtained from a single pair of observations. This second oracle is much harder to analyze.

Statistical Assumptions

We consider the following general setting:

-H is a d-dimensional Euclidean space with d 1. -The observations (x n , y n ) 2 H ⇥ R, n
1, are independent and identically distributed (i.i.d.), and such that Ekx n k 2 and Ey 2 n are finite. -We consider the least-squares regression problem, namely the minimization of the expected loss f (✓) = 1 2 E(hx n , ✓i y n ) 2 which is a quadratic function. We first introduce an assumption on the distribution of x n .

Covariance matrix. We denote by ⌃ = E(x n ⌦ x n ) 2 R d⇥d the population covariance matrix, which is the Hessian of f at all points. Without loss of generality, we can assume ⌃ is invertible by reducing H to the minimal subspace where all x n , n 1, lie almost surely. This implies that all eigenvalues of ⌃ are strictly positive (but they may be arbitrarily small). Following [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF], we assume there exists R > 0 such that

Ekx n k 2 x n ⌦ x n 4 R 2 ⌃, (A 1 ) where A 4 B means that B A is positive semi-definite. This assumption implies in particular that (a) Ekx n k 4 is finite and (b) tr ⌃ = Ekx n k 2  R 2 since
k 2  R 2 almost surely implies Ekx n k 2 x n ⌦ x n 4 E ⇥ R 2 x n ⌦ x n ⇤ = R 2 ⌃.
This assumption is also true for data with infinite support and a bounded kurtosis for the projection of the covariates x n on any direction z 2 H, e.g, for which there exists  > 0, such that:

8z 2 H, Ehz, x n i 4  hz, ⌃zi 2 .
(3.1) Indeed, by Cauchy-Schwarz inequality, Equation (3.1) implies for all (z, t) 2 H 2 , the following bound Ehz, x n i 2 ht, x n i 2  hz, ⌃ziht, ⌃ti, which in turn implies that for all positive semi-definite symmetric matrices M, N, we have

Ehx n , Mx n ihx n , Nx n i   tr(M ⌃) tr(N ⌃). Equation (3.1), which is true for Gaussian vectors with  = 3, thus implies (A 1 ) for R 2 =  tr ⌃ = Ekx n k 2 .
In the next two paragraphs, we introduce some quantities that will be important in the analysis, in order to get tighter bounds.

Eigenvalue decay. Most convergence bounds depend on the dimension d of H. However it is possible to derive dimension-free and often tighter convergence rates by considering bounds depending on the value tr ⌃ b for b 2 [0, 1]. Given b, if we consider the eigenvalues of ⌃ ordered in decreasing order, which we denote by s i , then tr ⌃ b = P i s b i , and the eigenvalues decay 1 at least as

(tr ⌃ b ) 1/b i 1/b . Moreover, it is known that (tr ⌃ b
) 1/b is decreasing in b and thus, the smaller the b, the stronger the assumption. For b going to 0 then tr ⌃ b tends to d and we are back in the classical lowdimensional case. When b = 1, we simply get tr ⌃ = Ekx n k 2 , which will correspond to the weakest assumption in our context.

Optimal predictor. In finite dimension the regression function

f (✓) = 1 2 E(hx n , ✓i y n ) 2 always admits a global minimum ✓ ⇤ = ⌃ † E(y n x n ).
When initializing algorithms at ✓ 0 = 0 or regularizing by the squared norm, rates of convergence generally depend on k✓ ⇤ k, a quantity which could be arbitrarily large.

However there exists a systematic upper-bound 2 k⌃

1 2 ✓ ⇤ k  2 p Ey 2
n . This leads naturally to the consideration of convergence bounds depending on k⌃ r/2 ✓ ⇤ k for r  1.

Indeed for any

i 1, we have is b i  P i t=1 s b t  tr(⌃ b ). 2. Indeed for all ✓ 2 R d
and in particular ✓ = 0, by Minkowski's inequality, k⌃

1 2 ✓ ⇤ k p Ey 2 n = p Eh✓ ⇤ , x n i 2 p Ey 2 n  p E(h✓ ⇤ , x n i y n ) 2  p E(h✓, x n i y n ) 2  p E(y n ) 2 .
In infinite dimension this will correspond to assuming k⌃ r/2 ✓ ⇤ k < 1. This new assumption relates the optimal predictor with sources of ill-conditioning (since ⌃ is the Hessian of the objective function f ), the smaller r, the stronger our assumption, with r = 1 corresponding to no assumption at all, r = 0 to ✓ ⇤ in H and r = 1 to a convergence of the bias of least-squares regression with averaged stochastic gradient descent in O k⌃ 1/2 ✓⇤k 2 n 2 [Dieuleveut and[START_REF] Défossez | Averaged least-mean-squares: bias-variance trade-offs and optimal sampling distributions[END_REF][START_REF] Bach | Duality between subgradient and conditional gradient methods[END_REF]. In this chapter, we will use arbitrary initial points ✓ 0 and thus our bounds will depend on k⌃ r/2 (✓ 0 ✓ ⇤ )k. Finally, we make an assumption on the joint distribution of (x n , y n ).

Noise. We denote by " n = y n h✓ ⇤ , x n i the residual for which we have E[" n x n ] = 0. Although we do not have E[" n |x n ] = 0 in general unless the model is well-specified, we assume the noise to be a structured process such that there exists > 0 with

E[" 2 n x n ⌦ x n ] 4 2 ⌃. (A 2 ) Assumption (A
2 ) is satisfied for example for data almost surely bounded or when the model is well-specified, (e.g., y n = h✓ ⇤ , x n i + " n , with (" n ) n2N i.i.d. of variance 2 and independent of x n ).

Averaged Gradient Methods and Acceleration

We focus in this chapter on stochastic gradient methods with and without acceleration for the least-squares function regularized by 2 k✓ ✓ 0 k 2 for 2 R + . The regularization will be useful when deriving tighter convergence rates in Section 3.5, and it has the additional benefit of making the problem -strongly convex. Stochastic gradient descent (referred to from now on as "SGD"), applied to the regularized problem, can be described for n 1 as

✓ n = ✓ n 1 f 0 n (✓ n 1 ) (✓ n 1 ✓ 0 ), (3.2)
starting from ✓ 0 2 H, where > 0 is either called the step-size in optimization or the learning rate in machine learning, and f 0 n (✓ n 1 ) is an unbiased estimate of the gradient of f at ✓ n 1 , that is, its conditional expectation given all other sources of randomness is equal to f 0 (✓ n 1 ). Accelerated stochastic gradient descent is defined, for the regularized problem, by an iterative system with two parameters (✓ n , ⌫ n ) satisfying for n 1

✓ n = ⌫ n 1 f 0 n (⌫ n 1 ) (⌫ n 1 ✓ 0 ) ⌫ n = ✓ n + ✓ n ✓ n 1 , (3.3) starting from ✓ 0 = ⌫ 0 2 H, with , 2 R 2 and f 0 n (✓ n 1
) described as before. It may be reformulated as the following second-order recursion

✓ n = (1 ) ✓ n 1 + (✓ n 1 ✓ n 2 ) f 0 n ✓ n 1 + (✓ n 1 ✓ n 2 ) + ✓ 0 .
The momentum coefficient 2 R is chosen to accelerate the convergence rate [Nesterov, 1983, Beck and[START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] and has its roots in the heavy-ball algorithm from [START_REF] Polyak | Some methods of speeding up the convergence of iteration methods[END_REF]. We especially concentrate here, following [START_REF] Polyak | Acceleration of stochastic approximation by averaging[END_REF], on the average of the sequence

✓n = 1 n + 1 n X i=0 ✓ n , (3.4)
and we note that it can be computed online as ✓n = n n+1 ✓n 1 + 1 n+1 ✓ n . The key ingredient in the algorithms presented above is the unbiased estimate on the gradient f 0 n (✓), which can take two forms that we now describe in our setting.

Additive versus Multiplicative Stochastic Oracles on the Gradient

We consider the standard stochastic approximation framework [START_REF] Kushner | Stochastic Approximation and Recursive Algorithms and Applications[END_REF]. That is, we let (F n ) n 0 be the increasing family of -fields that are generated by all variables (x i , y i ) for i  n, and such that for each ✓ 2 H the random variable

f 0 n (✓) is square-integrable and F n -measurable with E[f 0 n (✓)|F n 1 ] = f 0 (✓), for all n 0. Consequently it is of the form f 0 n (✓) = f 0 (✓) ⇠ n , (A 3 ) where the noise process ⇠ n is F n -measurable with E[⇠ n |F n 1 ] = 0 and E[k⇠ n k 2
] is finite. We will consider two different gradient oracles.

Additive noise. The first oracle is the sum of the true gradient f 0 (✓) and an independent zero-mean noise that does not depend on ✓. This oracle is equal to

f 0 n (✓) = ⌃✓ y n x n . (3.5)
Since f 0 (✓) = ⌃✓ Ey n x n , the oracle above has a noise vector ⇠ n = y n x n Ey n x n independent of ✓ and therefore satisfies Assumption (A 3 ). Furthermore we also assume that there exists ⌧ 2 R such that 4 ) that is, the noise has a particular structure adapted to least-squares regression. For optimal results for unstructured noise, with convergence rate for the noise part in O(1/ p n), see [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF]. Our oracle above with an additive noise which is independent of the current iterate corresponds to the first setting studied in stochastic approximation [START_REF] Robbins | A stochastic approxiation method[END_REF][START_REF] Duflo | Random Iterative Models[END_REF][START_REF] Polyak | Acceleration of stochastic approximation by averaging[END_REF]. While used by [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] as an artifact of proof, for least-squares regression, such an additive noise corresponds to the situation where the distribution of x is known so that the population covariance matrix is computable, but the distribution of the outputs (y n ) n2N remains unknown. Thus it may be seen as an intermediate set-up between regression estimation with fixed and random design [see, e.g., Györfi et al., 2006, Section 1.9].

E[⇠ n ⌦ ⇠ n ] 4 ⌧ 2 ⌃, ( A
Assumption (A 4 ) will be satisfied, for example if the outputs are almost surely bounded because

E[⇠ n ⌦ ⇠ n ] 4 E[y 2 n x n ⌦ x n ] 4 ⌧ 2 ⌃ if y 2 n  ⌧ 2
almost surely. But it will also be for data satisfying Equation (3.1) since we will have

E[⇠ n ⌦ ⇠ n ] 4 E[y 2 n x n ⌦ x n ] = E[(h✓ ⇤ , x n i + " n ) 2 x n ⌦ x n ] 4 2E[h✓ ⇤ , x n i 2 x n ⌦ x n ] + 2 2 ⌃ 4 2(k⌃ 1/2 ✓ ⇤ k 2 + 2 )⌃ 4 2(4E[y 2 n ] + 2 )⌃,
and thus Assumption (A 4) is satisfied with

⌧ 2 = 2(4E[y 2 n ] + 2 ).
Stochastic noise ("multiplicative/additive"). This corresponds to:

f 0 n (✓) = (hx n , ✓i y n )x n = (⌃ + ⇣ n )(✓ ✓ ⇤ ) ⌅ n , (3.6)
with

⇣ n = x n ⌦ x n ⌃ and ⌅ n = (y n hx n , ✓ ⇤ i)x n = " n x n .
This oracle corresponds to regular SGD, which is often referred to as the least-mean-square (LMS) algorithm for least-squares regression, where the noise comes from sampling a single pair of observations. While still satisfying Assumption (A 3 ), it combines an additive noise ⌅ n independent of ✓ as in Eq. (3.5) and a multiplicative noise ⇣ n . This multiplicative noise makes this stochastic oracle harder to analyze which explains why it is often approximated by an additive noise oracle. However it is the most widely used and most practical one. Note that for the oracle in Eq. (3.6), from Equation ( A2 ), we have

E[⌅ n ⌦ ⌅ n ] 4 2
⌃. It has a similar form to Assumption (A 4 ) which is valid for the additive noise oracle in Eq. (3.5): we use different constants 2 and ⌧ 2 to highlight the difference between these two oracles.

Averaged Stochastic Gradient Descent

In this section, we provide convergence bounds for regularized averaged stochastic gradient descent. The main novelty compared to the work of [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] is (a) the presence of regularization, which will be useful when deriving tighter convergence rates in Section 3.5 and (b) a much simpler proof. We first consider the additive noise in Section 3.3.1 before considering the multiplicative/additive noise in Section 3.3.2.

Additive Noise

We study here the convergence of the averaged SGD recursion defined by Eq. (3.2) under the simple oracle defined in Eq. (3.5). For least-squares regression, it takes the form:

✓ n = ⇥ I ⌃ I ⇤ ✓ n 1 + y n x n + ✓ 0 . (3.7)
This is an easy adaptation of the work of Bach and Moulines [2013, Lemma 2] for the regularized case.

Lemma 12. Assume (A 4 ). Consider the recursion in Eq. (3.7) with any regularization parameter 2 R + and any constant step-size such that (⌃ + I) 4 I. Then

Ef ( ✓n ) f (✓ ⇤ )  ⇣ + 1 n ⌘ 2 k⌃ 1/2 (⌃+ I) 1 (✓ 0 ✓ ⇤ )k 2 + ⌧ 2 tr ⇥ ⌃ 2 (⌃ + I) 2 ⇤ n . (3.8)
We can make the following observations:

-The proof (see Appendix 3.A) relies on the fact that ✓ n ✓ ⇤ is obtainable in closed form since the cost function is quadratic and thus the recursions are linear, and follows from [START_REF] Polyak | Acceleration of stochastic approximation by averaging[END_REF].

-The constraint on the step-size is equivalent to (L + ) 6 1 where L is the largest eigenvalue of ⌃ and we thus recover the usual step-size from deterministic gradient descent [START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF].

-When n tends to infinity, the algorithm converges to the minimum of f (✓) +

2 k✓ ✓ 0 k 2 and our performance guarantee becomes 2 k⌃ 1/2 (⌃+ I) 1 (✓ 0 ✓ ⇤ )k 2 . This is the standard "bias term" from regularized ridge regression [START_REF] Hsu | Random design analysis of ridge regression[END_REF] which we naturally recover here. The term ⌧ 2 n tr

⇥ ⌃ 2 (⌃ + I) 2 ⇤
is usually referred to as the "variance term" [START_REF] Hsu | Random design analysis of ridge regression[END_REF], and is equal to

⌧ 2 n times the quantity tr ⇥ ⌃ 2 (⌃ + I) 2 ⇤
, which is often called the degrees of freedom of the ridge regression problem [START_REF] Gu | Smoothing Spline ANOVA Models[END_REF].

-For finite n, the first term in Eq. (3.8) is the usual bias term which depends on the distance from the initial point ✓ 0 to the objective point ✓ ⇤ with an appropriate norm. It includes a regularization-based component which is proportional to 2 and optimization-based component which depends on ( n) 2 . The regularization-based bias appears because the algorithm tends to minimize the regularized function instead of the true function f .

-Given Eq. (3.8), it is natural to set = 1 n , and the two components of the bias term are exactly of the same order leading to 4

2 n 2 k⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2 .
It corresponds up to a constant factor to the bias term of regularized leastsquares [START_REF] Hsu | Random design analysis of ridge regression[END_REF], but it is achieved by an algorithm accessing only n stochastic gradients. Note that when or depend on n, this term is not necessarily of order O(n 2), as the numerator might be arbitrarily large. Note also that here as in the rest of the chapter, we only prove results in the finite horizon setting, meaning that the number of samples is known in advance and the parameters , may be chosen as functions of n, but remain constant along the iterations (when or depend on n, our bounds only hold for the last iterate).

-Note that the bias term can also be bounded by et al., 2009].

1 n k⌃ 1/2 (⌃ + I) 1/2 (✓ 0 ✓ ⇤ )k 2 only when k✓ 0 ✓ ⇤ k is finite (note the
-Overall we get the same performance as the empirical risk minimizer with fixed design, but with an algorithm that performs a single pass over the data.

-When = 0 we recover Lemma 2 of [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF]. In this case the variance term ⌧ 2 d n is optimal over all estimators in H [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF] even without computational limits, in the sense that no estimator that uses the same information can improve upon this rate.

Multiplicative/Additive Noise

When the general stochastic oracle in Eq. (3.6) is considered, the regularized LMS algorithm defined by Eq. (3.2) takes the form:

✓ n = ⇥ I x n ⌦ x n I ⇤ ✓ n 1 + y n x n + ✓ 0 .
(3.9)

We have a very similar result with an additional corrective term (second line below) compared to Lemma 12.

Theorem 7. Assume (A 1,2 ). Consider the recursion in Eq. (3.9). For any regularization parameter 2 R + and for any constant step-size such that 2 (R 2 + 2 )  1 we have:

Ef ( ✓n ) f (✓ ⇤ ) 6 3 ⇣ 2 + 1 n ⌘ 2 k⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2 + 6 2 n + 1 tr ⇥ ⌃ 2 (⌃+ I) 2 ⇤ +3 (⌃ + I) 1/2 (✓ 0 ✓ ⇤ ) 2 tr(⌃(⌃ + I) 1 ) 2 (n + 1) 2 .
We can make the following remarks:

-The proof (see Appendix 3.B) relies on a bias-variance decomposition, each term being treated separately. We adapt a proof technique from [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] which considers the difference between the recursions in Eq. (3.9) and in Eq. (3.7).

-As in Lemma 12, the bias term can also be bounded by 4 and 3.B.5). This is useful in particular when considering unstructured noise.

1 n k⌃ 1/2 (⌃+ I) 1/2 (✓ 0 ✓ ⇤ )k 2 and the variance term by tr[⌃(⌃ + I) 1 ⇠ n ⌦ ⇠ n ] (see proof in Appen- dices 3.B.
-The variance term is the same as in the previous case. However there is a residual term that now appears when we go to the fully stochastic oracle (second line). This term will go to zero when tends to zero and can be compared to the corrective term which also appears when [START_REF] Hsu | Random design analysis of ridge regression[END_REF] go from fixed to random design. Nevertheless our bounds are more concise than theirs, making significantly fewer assumptions and relying on an efficient single-pass algorithm.

-In this setting, the step-size may not exceed 1/(2(R 2 + 2 )), whereas with an additive noise in Lemma 12 the condition is  1/(L+ ), a quantity which can be much bigger than 1/(2(R 2 + 2 )), as L is the spectral radius of ⌃ whereas R 2 is of the order of tr(⌃). Note that in practice, computing L is as hard as computing ✓ ⇤ so that the step-size / 1/R 2 is a good practical choice. See [START_REF] Défossez | Averaged least-mean-squares: bias-variance trade-offs and optimal sampling distributions[END_REF] for larger allowed step-sizes that require more information.

-For = 0 the error is bounded by 3(1+d) (

n) 2 k⌃ 1/2 (✓ 0 ✓ ⇤ )k 2 + 6 2 d n+1 .
We recover results from [START_REF] Défossez | Averaged least-mean-squares: bias-variance trade-offs and optimal sampling distributions[END_REF] with a non-asymptotic bound but we lose the advantage of having an asymptotic equivalent (i.e., a limit rather than an upper-bound). We note that the assumption (A 1,2 ) are close to the minimal assumptions required to obtain the optimal rate of convergence of 2 d/n [Lecué andMendelson, 2016, Oliveira, 2016]

Accelerated Stochastic Averaged Gradient Descent

We study the convergence under the stochastic oracle from Eq. (3.5) of averaged accelerated stochastic gradient descent defined by Eq. (3.3) which can be rewritten for the least-squares function f as a second-order iterative system with constant coefficients:

✓ n = ⇥ I ⌃ I ⇤⇥ ✓ n 1 + (✓ n 1 ✓ n 2 ) i + y n x n + ✓ 0 .
(3.10)

When using averaging, we refer to this algorithm as "averaged-accelerated-SGD".

Theorem 8. Assume (A 4 ). For any regularization parameter 2 R + and for any constant step-size (⌃ + I) 4 I, we have for any 2 ⇥

1 p 1+ p , 1 ⇤
, for the recursion in Eq. (3.10):

Ef ( ✓n ) f (✓ ⇤ )  2 ⇣ + 36 (n + 1) 2 ⌘ k⌃ 1/2 (⌃+ I) 1/2 (✓ 0 ✓ ⇤ )k 2 +8⌧ 2 tr ⇥ ⌃ 2 (⌃ + I) 2 ⇤ n + 1 .
The numerical constants are partially artifacts of the proof (see Appendices 3.C and 3.D). Thanks to a wise use of tight inequalities, the bound is independent of and valid for all 2 R + . This results in the simple following corollary for = 0, which corresponds to the particularly simple recursion (with averaging to obtain ✓n ):

✓ n = ⇥ I ⌃ ⇤ (2✓ n 1 ✓ n 2 ) + y n x n . (3.11)
Corollary 4. Assume (A 4 ). For any constant step-size ⌃ 4 I, we have for = 1,

Ef ( ✓n ) f (✓ ⇤ )  36 k✓ 0 ✓ ⇤ k 2 (n + 1) 2 + 8 ⌧ 2 d n + 1 . (3.12)
We can make the following observations:

-The proof technique relies on direct moment computations in each eigensubspace obtained by O'Donoghue and Candès [2013] in the deterministic case. Indeed as ⌃ is a symmetric matrix, the space can be decomposed on an orthonormal eigenbasis of ⌃, and the iterations are decoupled in such an eigenbasis. Although we only provide an upper-bound, this is in fact an equality plus other exponentially small terms as shown in the proof which relies on linear algebra, with difficulties arising from the fact that this second-order system can be expressed as a linear stochastic dynamical system with non-symmetric matrices.

We only provide a result for additive noise.

-The first bound 1 n 2 k✓ 0 ✓ ⇤ k 2 in Eq. (3.12) corresponds to the usual accelerated rate. It has been shown by [START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF] to be the optimal rate of convergence for optimizing a quadratic function with a first-order method that can access only to sequences of gradients when n  d. We recover by averaging an algorithm dedicated to strongly convex function the traditional convergence rate for non-strongly convex functions. Even if it seems surprising, the algorithm works also for = 0 and = 1 (see also simulations in Section 3.6).

-The second bound in Eq. (3.12) also matches the optimal statistical performance ⌧ 2 d n described in the observations following Lemma 12. Accordingly this algorithm achieves joint bias/variance optimality (when measured in terms of ⌧ 2 and k✓

0 ✓ ⇤ k 2 ).
-We have the same rate of convergence for the bias when compared to the regular Nesterov acceleration without averaging studied in Chapter 2, which corresponds to choosing n = 1 2/n for all n. However if the problem is µ-strongly convex, this latter was shown to also converge at the linear rate O (1 µ) n and thus is adaptive to hidden strong convexity (since the algorithm does not need to know µ to run). This explains that it ends up converging faster for quadratic function since for large n the convergence at rate 1/n 2 becomes slower than the one at rate (1 µ) n even for very small µ. This is confirmed in our experiments in Section 3.6. Thanks to this adaptivity, we can also show using the same tools and considering its weighted average

✓n = 2 n(n+1) P n k=0 k✓ k that the bias term of Ef ( ✓n ) f (✓ ⇤ ) has a convergence rate of order 2 + 1 2 (n+1) 4 k⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2 without
any change in the variance term. This has to be compared to the bias of averaged SGD

+ 1 (n+1) 2 k⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2 in Section 3.
3 and may lead to faster convergence for the bias in presence of hidden strong convexity.

-Overall, the bias term is improved whereas the variance term is not degraded and acceleration is thus robust to noise in the gradients. Thereby, while secondorder finite difference methods for optimizing quadratic functions in the singular case, such as conjugate gradient [Polyak, 1987, Section 6.1] are notoriously highly sensitive to noise, we are able to propose a version which is robust to stochastic noise. -Note that when there is no assumption on the covariance of the noise we still have the variance bounded by n 2 tr

⇥ ⌃(⌃ + I) 1 V ⇤ ; setting = 1/n 3/2 and
= 0 leads to the bound k✓ 0 ✓⇤k 2 p n + tr V p n . We recover the usual rate for accelerated stochastic gradient in the non-strongly convex case [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF]. When the values of the bias and the variance are known, we can achieve the optimal trade-off of Lan

[2012] R 2 k✓ 0 ✓⇤k 2 n 2 + k✓ 0 ✓⇤k p tr V p n for = min n 1/R 2 , k✓ 0 ✓⇤k p tr V n 3/2
o .

Tighter Dimension-Independent Convergence Rates

We have seen in Corollary 4 that the averaged accelerated gradient algorithm achieves the lower bounds ⌧ 2 d/n and L n 2 k✓ 0 ✓ ⇤ k 2 for the prediction error. However the algorithm behaves often better than in the worst-case scenarios corresponding to the lower bounds. Indeed the algorithm still predicts well when the dimension d is much larger than n or when the norm of the optimal predictor k✓ ⇤ k 2 is huge. Actually gradients algorithms are adaptive to the difficulty of the problem as presented in the following corollary for the averaged accelerated algorithm. , for the recursion in Eq. (3.10):

Ef ( ✓n ) f (✓ ⇤ )  min r2[0,1], b2[0,1]  74 k⌃ r/2 (✓ 0 ✓ ⇤ )k 2 1 r (n + 1) 2(1 r) + 8 ⌧ 2 b tr(⌃ b ) (n + 1) 1 2b .
We can make the following observations: -This corollary is a direct consequence of Theorem 9 by [START_REF] Dieuleveut | Better, Faster, Stronger Convergence Rates for Least-Squares Regression[END_REF] considered for = ( n 2 ) 1 . It extends previously known bounds in the kernel least-mean-squares setting [START_REF] Dieuleveut | Non-parametric stochastic approximation with large step sizes[END_REF] with the use of an extra regularization.

-The algorithm is independent of r and b, thus all the bounds for different values of (r, b) are valid. This is a strong property of the algorithm, which is indeed adaptative to the regularity and the effective dimension of the problem (once is chosen). In situations in which either d is larger than n or Lk✓ 0 ✓ ⇤ k 2 is larger than n 2 , the algorithm can still enjoy good convergence properties, by adapting to the best values of b and r.

-For b = 0 we recover the variance term of Corollary 4, but for b > 0 and fast decays of eigenvalues of ⌃, the bound may be much smaller; note that we lose in the dependency in n, but typically, for large d, this can be advantageous. -For r = 0 we recover the bias term of Corollary 4 and for r = 1 (no assumption at all) the bias is bounded by k⌃ 1/2 ✓ ⇤ k 2  4R 2 , which is not going to zero. The smaller r is, the stronger the decrease of the bias with respect to n is (which is coherent with the fact that we have a stronger assumption). In the companion paper of [START_REF] Dieuleveut | Better, Faster, Stronger Convergence Rates for Least-Squares Regression[END_REF], we show that in the setting of non parametric learning in kernel spaces, these bounds lead to the optimal statistical rate of convergence.

Experiments

We illustrate now our theoretical results on synthetic examples. For d = 25 we consider normally distributed inputs x n with random covariance matrix ⌃ which has eigenvalues 1/i 3 , for i = 1, . . . , d, and random optimum ✓ ⇤ and starting point ✓ 0 such that k✓ 0 ✓ ⇤ k = 1. The outputs y n are generated from a linear function with homoscedastic noise with unit signal to noise-ratio ( 2= 1), we take R 2 = tr ⌃ the average radius of the data and a step-size = 1/R 2 and = 0. The additive noise oracle is used. We show results averaged over 10 replications.

We compare the performance of averaged SGD (AvSGD), AccSGD (usual Nesterov acceleration for convex functions) and our novel averaged accelerated SGD from Section 3.4 AvAccSGD (which is not the averaging of AccSGD because the momentum term is proportional to 1 3/n for AccSGD instead of being equal to 1 for AvAcc-SGD), on two different problems: one deterministic (k✓

0 ✓ ⇤ k = 1, 2
= 0) which will illustrate how the bias term behaves, and one purely stochastic (k✓ ). However, as mentioned in the observations following Corollary 4, AccSGD takes advantage of the hidden strong convexity of the leastsquares function and starts converging linearly at the end. For the variance (right plot of Figure 3-1), AccSGD is not converging to the optimum and keeps oscillating whereas AvSGD and AvAccSGD both converge to the optimum at a speed O(1/n). However AvSGD remains slightly faster in the beginning.

0 ✓ ⇤ k = 0,
Note that for small n, or when the bias Lk✓ 0 ✓ ⇤ k 2 /n 2 is much bigger than the variance 2 d/n, the bias may have a stronger effect, although asymptotically, the variance always dominates. It is thus essential to have an algorithm which is optimal in both regimes; this is achieved by AvAccSGD.

Conclusion

In this chapter, we showed that stochastic averaged accelerated gradient descent was robust to structured noise in the gradients present in least-squares regression. Beyond being the first algorithm which is jointly optimal in terms of both bias and finite-dimensional variance, it is also adapted to finer assumptions such as fast decays of the covariance matrices or optimal predictors with large norms.

Our current analysis is performed for least-squares regression. While it could be directly extended to smooth losses through efficient online Newton methods [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF], an extension to all smooth or self-concordant-like functions [START_REF] Bach | Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression[END_REF] would widen its applicability. Moreover, our accelerated gradient analysis is performed for additive noise (i.e., for least-squares regression, with knowledge of the population covariance matrix) and it would be interesting to study the robustness of our results in the contexts of least-mean squares and online learning. Our analysis relies on single observations per iteration and could be made finer by using minibatches [START_REF] Cotter | Better mini-batch algorithms via accelerated gradient methods[END_REF][START_REF] Dekel | Optimal distributed online prediction using mini-batches[END_REF], which should not change the variance term but could impact the bias term. Finally, it would be appealing to consider proximal extensions of this algorithm. In Chapter 4 we take a first step towards this goal by studying a proximal extension of averaged gradient descent. Appendix 3.A Proofs of Section 3.3

3.A.1 Proof of Lemma 12

We proof here Lemma 12 which is the extension of Lemma 2 of [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] for the regularized case. The proof technique relies on the fact that recursions in Eq. (3.7) are linear since the cost function is quadratic which allows us to obtain

✓ n ✓ ⇤ in closed form.
For any regularization parameter 2 R + and any constant step-size (⌃+ I) 4 I we may rewrite the regularized stochastic gradient recursion in Eq. (3.7) as:

✓ n ✓ ⇤ = ⇥ I ⌃ I ⇤ (✓ n 1 ✓ ⇤ ) + ⇠ n + (✓ 0 ✓ ⇤ ).
We thus get for n 1 the expansion

✓ n ✓ ⇤ = (I ⌃ I) n (✓ 0 ✓ ⇤ ) + n X k=1 (I ⌃ I) n k ⇠ k + n X k=1 (I ⌃ I) n k (✓ 0 ✓ ⇤ ) = (I ⌃ I) n (✓ 0 ✓ ⇤ ) + n X k=1 (I ⌃ I) n k ⇠ k + ⇥ I (I ⌃ I) n ⇤ (⌃ + I) 1 (✓ 0 ✓ ⇤ ) = (I ⌃ I) n [I (⌃ + I) 1 ](✓ 0 ✓ ⇤ ) + n X k=1 (I ⌃ I) n k ⇠ k + (⌃ + I) 1 (✓ 0 ✓ ⇤ ).
We then have using the definition of the average

n( ✓n 1 ✓ ⇤ ) = n 1 X j=0 (✓ j ✓ ⇤ ) = n 1 X j=0 (I ⌃ I) j [I (⌃ + I) 1 ](✓ 0 ✓ ⇤ ) + n 1 X j=0 j X k=1 (I ⌃ I) n k ⇠ k + n (⌃ + I) 1 (✓ 0 ✓ ⇤ ).
For which we will compute the two sums separately

n 1 X j=0 (I ⌃ I) j [I (⌃ + I) 1 ](✓ 0 ✓ ⇤ ) = 1 ⇥ I (I ⌃ I) n ⇤ (⌃ + I) 1 [I (⌃ + I) 1 ](✓ 0 ✓ ⇤ ),
and

n 1 X j=0 j X k=1 (I ⌃ I) j k ⇠ k = n 1 X k=1 ⇣ n 1 X j=k (I ⌃ I) j k ⌘ ⇠ k = n 1 X k=1 ⇣ n 1 k X j=0 (I ⌃ I) j ⌘ ⇠ k = n 1 X k=1 ⇥ I (I ⌃ I) n k ⇤ (⌃ + I) 1 ⇠ k .
Gathering the three terms together, we thus have

n( ✓n 1 ✓ ⇤ ) = 1 ⇥ I (I ⌃ I) n ⇤ (⌃ + I) 1 [I (⌃ + I) 1 ](✓ 0 ✓ ⇤ ) + n 1 X k=1 ⇥ I (I ⌃ I) n k ⇤ (⌃+ I) 1 ⇠ k +n (⌃+ I) 1 (✓ 0 ✓ ⇤ ) = h 1 ⇥ I (I ⌃ I) n ⇤ [I (⌃+ I) 1 ]+n I i (⌃+ I) 1 (✓ 0 ✓ ⇤ ) + n 1 X k=1 ⇥ I (I ⌃ I) n k ⇤ (⌃ + I) 1 ⇠ k .
Using standard martingale square moment inequalities which amount to consider ⇠ i , i = 1, . . . , n independent, the variance of the sum is the sum of variances and we have for

V = E⇠ n ⌦ ⇠ n n 2 Ek⌃ 1/2 ( ✓n 1 ✓ ⇤ )k 2 = P n 1 k=1 tr ⇥ I (I ⌃ I) n k ⇤ 2 ⌃(⌃ + I) 2 V + h 1 ⇥ I (I ⌃ I) n ⇤ [I (⌃+ I) 1 ]+n I i ⌃ 1/2 (⌃+ I) 1 (✓ 0 ✓ ⇤ ) 2 . (3.13)
Since all the matrices in this equality are symmetric positive-definite we are allowed to bound

h 1 ⇥ I (I ⌃ I) n ⇤ [I (⌃ + I) 1 ] + n I i 4 ⇣ 1 + n ⌘ I (3.14) ⇥ I (I ⌃ I) n k ⇤ 2 4 I.
This concludes proof of the Lemma 12

Ek⌃ 1/2 ( ✓n 1 ✓ ⇤ )k 2  ⇣ 1 n + ⌘ 2 k⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2 + 1 n tr ⌃(⌃ + I) 2 V. (3.15) 3.A.2 Proof When Only k✓ 0 ✓ ⇤ k Is Finite
Unfortunately k⌃ 1 (✓ 0 ✓ ⇤ )k may not be finite. However we can use that for all u 2 [0, 1] we have

1 (1 u) n nu  1 since 1 (1 u) n u = P n k=0 (1 u) k  n and it yields h 1 ⇥ I (I ⌃ I) n ⇤ [I (⌃ + I) 1 ] + n I i [⌃ + I] 1 4 h 1 ⇥ I (I ⌃ I) n ⇤ + n I i [⌃ + I] 1 4 h 1 ⇥ I (I ⌃ I) n ⇤ [⌃ + I] 1 + n [⌃ + I] 1 i 4 I + nI.
Combining with Eq. (3.14) we have

h 1 ⇥ I (I ⌃ I) n ⇤ [I (⌃ + I) 1 ] + n I i ⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ ) 2  (n + 1) ⇣ 1 + n ⌘ k⌃ 1/2 (⌃ + I) 1/2 (✓ 0 ✓ ⇤ )k 2 ,
which implies that

Ek⌃ 1/2 ( ✓n 1 ✓ ⇤ )k 2  2 ⇣ 1 n + ⌘ k⌃ 1/2 (⌃ + I) 1/2 (✓ 0 ✓ ⇤ )k 2 + 1 n tr ⌃(⌃ + I) 2 V, (3.16)
which is interesting when only k✓ 0 ✓ ⇤ k is finite.

3.A.3 Proof When The noise Is Not Structured

The bound in Eq. (3.15) becomes less interesting when the noise is not structured. However using the same technique we have that

⇥ I (I ⌃ I) n k ⇤ 2 (⌃+ I) 1 4 (n k) I
(I ⌃ I) n k ⇤ 2 ⌃(⌃ + I) 2 V  n X k=1 (n k) tr ⌃(⌃ + I) 1 V  n(n + 1) 2 tr ⌃(⌃ + I) 1 V.
Therefore we get

Ek⌃ 1/2 ( ✓n 1 ✓ ⇤ )k 2  ⇣ 1 n + ⌘ 2 k⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2 + tr ⌃(⌃ + I) 1 V, (3.17)
which is meaningful when the noise is not structured.

3.B Proof of Theorem 7

In this section, we will prove Theorem 7. The proof relies on a decomposition of the error as the sum of three main terms which will be studied separately. We state decomposition in Section 3.B.1 then prove upper bounds for the different terms in Sections 3.B.2 and 3.B.3.

3.B.1 Expansion of the Recursion

We may rewrite the regularized stochastic gradient recursion as:

✓ n = ⇥ I x n ⌦ x n I ⇤ ✓ n 1 + " n x n + hx n , ✓ ⇤ ix n + ✓ 0 ✓ n ✓ ⇤ = ⇥ I x n ⌦ x n I ⇤ (✓ n 1 ✓ ⇤ ) + " n x n + (✓ 0 ✓ ⇤ ). For i > k, let M (i, k) = ⇥ I x i ⌦ x i I ⇤ • • • ⇥ I x k ⌦ x k I ⇤
be an operator from H to H. We have the expansion

✓ n ✓ ⇤ = M (n, 1)(✓ 0 ✓ ⇤ ) + n X k=1 M (n, k + 1)" k x k + n X k=1 M (n, k + 1) (✓ 0 ✓ ⇤ ).
Our goal is to study these three terms separately and bound k⌃ 1/2 ( ✓n ✓ ⇤ )k for each of them.

3.B.2 Regularization-Based Bias Term

This is the term:

✓ n ✓ ⇤ = P n k=1 M (n, k + 1) (✓ 0 ✓ ⇤ ), which corresponds to the recursion ✓ n ✓ ⇤ = I x n ⌦ x n I (✓ n 1 ✓ ⇤ ) + (✓ 0 ✓ ⇤ ), (3.18)
initialized with ✓ 0 = ✓ ⇤ , and no noise. Following the proof technique of [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF], we are going to consider a related recursion by replacing in Equation (3.18) the operator x n ⌦ x n by its expectation ⌃. Thus, we consider ⌘ n defined as

⌘ n ✓ ⇤ = n X k=1 (I ⌃ I) n k (✓ 0 ✓ ⇤ ),
which satisfies the recursion (with initialization ⌘ 0 = ✓ ⇤ ) and

⌘ n ✓ ⇤ = ⇥ I ⌃ I ⇤ (⌘ n 1 ✓ ⇤ ) + (✓ 0 ✓ ⇤ ).
In order to bound k⌃ 1/2 (✓ n ✓ ⇤ )k, we will independently bound

k⌃ 1/2 (⌘ n ✓ ⇤ )k and k⌃ 1/2 (✓ n ⌘ n )k using Minkowski's inequality. Bounding k⌃ 1/2 (✓ n ⌘ n )k. We have ✓ 0 ⌘ 0 = 0, and ✓ n ⌘ n = ⇥ I x n ⌦ x n I ⇤ (✓ n 1 ⌘ n 1 ) + ⇥ ⌃ x n ⌦ x n ⇤ (⌘ n 1 ✓ ⇤ ).
We can now bound the recursion for ✓ n ⌘ n as follows, using standard online learning proofs [START_REF] Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF]:

k✓ n ⌘ n k 2 6 k✓ n 1 ⌘ n 1 k 2 2 ⌦ ✓ n 1 ⌘ n 1 , (x n ⌦ x n + I)(✓ n 1 ⌘ n 1 ) ↵ +2 ⌦ ✓ n 1 ⌘ n 1 , ⇥ ⌃ x n ⌦ x n ⇤ (⌘ n 1 ✓ ⇤ ) ↵ + 2 ⇥ x n ⌦ x n + I ⇤ (✓ n 1 ⌘ n 1 ) ⇥ ⌃ x n ⌦ x n ⇤ (⌘ n 1 ✓ ⇤ ) 2 .
By taking conditional expectations given F n 1 , we get, using first the fact that E(⌃

x n ⌦ x n |F n 1 ) = 0 and the inequality (a + b) 2  2(a 2 + b 2
), then developing and using

E[(x n ⌦ x n ) 2 ]  R 2 ⌃, which is assumption A 1 . E k✓ n ⌘ n k 2 |F n 1 6 k✓ n 1 ⌘ n 1 k 2 2 ⌦ ✓ n 1 ⌘ n 1 , (⌃ + I)(✓ n 1 ⌘ n 1 ) ↵ +2 2 E ⇥ x n ⌦ x n + I ⇤ (✓ n 1 ⌘ n 1 ) 2 |F n 1 +2 2 E ⇥ ⌃ x n ⌦ x n ⇤ (⌘ n 1 ✓ ⇤ ) 2 |F n 1 6 k✓ n 1 ⌘ n 1 k 2 2 ⌦ ✓ n 1 ⌘ n 1 , (⌃ + I)(✓ n 1 ⌘ n 1 ) ↵ +2 2 ⌦ ✓ n 1 ⌘ n 1 , (R 2 ⌃ + 2 I + 2 ⌃)(✓ n 1 ⌘ n 1 ) ↵ +2 2 R 2 h⌘ n 1 ✓ ⇤ , ⌃i 6 k✓ n 1 ⌘ n 1 k 2 + 2 2 R 2 h⌘ n 1 ✓ ⇤ , ⌃(⌘ n 1 ✓ ⇤ )ii 2 ⇥ 1 (R 2 + 2 ) ⇤⌦ ✓ n 1 ⌘ n 1 , ⌃(✓ n 1 ⌘ n 1 )
This leads by taking full expectations and moving terms to

E ⌦ ✓ n 1 ⌘ n 1 , ⌃(✓ n 1 ⌘ n 1 ) ↵ 6 1 2 ⇥ 1 (R 2 +2 ) ⇤ ⇥ Ek✓ n 1 ⌘ n 1 k 2 Ek✓ n ⌘ n k 2 ⇤ + R 2 1 (R 2 + 2 ) h⌘ n 1 ✓ ⇤ , ⌃(⌘ n 1 ✓ ⇤ )i. Thus, if (R 2 + 2 ) 6 1 2 E ⌦ ✓ n 1 ⌘ n 1 , ⌃(✓ n 1 ⌘ n 1 ) ↵ 6 1 ⇥ Ek✓ n 1 ⌘ n 1 k 2 Ek✓ n ⌘ n k 2 ⇤ +2 R 2 Eh⌘ n 1 ✓ ⇤ , ⌃(⌘ n 1 ✓ ⇤ )i.
This leads to, summing and using initial conditions ✓ 0 ⌘ 0 = 0, then using convexity to upper bound ⌦ ✓n ⌘n , ⌃( ✓n ⌘n )

↵  1 n+1 P n k=0 ⌦ ✓ k ⌘ k , ⌃(✓ k ⌘ k ) ↵ , E ⌦ ✓n ⌘n , ⌃( ✓n ⌘n ) ↵ 6 2 R 2 n + 1 n X k=0 h⌘ k ✓ ⇤ , ⌃(⌘ k ✓ ⇤ )i.
Bounding k⌃ 1/2 (⌘ n ✓ ⇤ )k. Moreover we have:

⌘ n ✓ ⇤ = (⌃ + I) 1 (✓ 0 ✓ ⇤ ) (I ⌃ I) n ⇥ (⌃ + I) 1 (✓ 0 ✓ ⇤ ) ⇤ ⌘n ✓ ⇤ = (⌃ + I) 1 (✓ 0 ✓ ⇤ ) 1 n + 1 n X k=0 (I ⌃ I) k ⇥ (⌃ + I) 1 (✓ 0 ✓ ⇤ ) ⇤ = (⌃ + I) 1 (✓ 0 ✓ ⇤ ) 1 n + 1 1 (⌃ + I) 1 ⇥ I (I ⌃ I) n+1 ⇤⇥ (⌃ + I) 1 (✓ 0 ✓ ⇤ ) ⇤ .
This leads using Minkowski inequality to

Ek⌃ 1/2 (⌘ n ✓ ⇤ )k 2 1/2 6 k ⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k Ek⌃ 1/2 (⌘ n ✓ ⇤ )k 2 1/2 6 k ⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k.
Thus this part is such that

q Ek⌃ 1/2 ( ✓n ✓ ⇤ )k 2 6 k ⌃ 1/2 (⌃+ I) 1 (✓ 0 ✓ ⇤ )k+ q 2 R 2 k ⌃ 1/2 (⌃+ I) 1 (✓ 0 ✓ ⇤ )k 2 6 k ⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 1 + p 2 R 2 ,
that gives the first bound on the regularization-based bias

Ek⌃ 1/2 ( ✓n ✓ ⇤ )k 2 6 k ⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2 1 + p 2 R 2 2 . (3.19)

3.B.3 Expansion without the Regularization Term

We will follow here the outline of the proof of [START_REF] Györfi | On the averaged stochastic approximation for linear regression[END_REF] which considers a full expansion of the function value k⌃ 1/2 ( ✓n ✓ ⇤ )k 2 . This corresponds to

✓ n ✓ ⇤ = M (n, 1)(✓ 0 ✓ ⇤ ) + n X k=1 M (n, k + 1)" k x k .
We have

E n X i=0 n X j=0 h✓ i ✓ ⇤ , ⌃(✓ j ✓ ⇤ )i = E n X i=0 k⌃ 1/2 (✓ i ✓ ⇤ )k 2 +2E n 1 X i=0 n X j=i+1 h✓ i ✓ ⇤ , ⌃(✓ j ✓ ⇤ )i.
Moreover,

E n 1 X i=0 n X j=i+1 h✓ i ✓ ⇤ , ⌃(✓ j ✓ ⇤ )i = E n 1 X i=0 n X j=i+1 ⌧ ✓ i ✓ ⇤ , ⌃  M (j, i + 1)(✓ i ✓ ⇤ ) + j X k=i+1 M (j, k + 1) " k x k = E n 1 X i=0 n X j=i+1 h✓ i ✓ ⇤ , ⌃M (j, i + 1)(✓ i ✓ ⇤ )i because " k x k and ✓ i are independent, = E n 1 X i=0 n X j=i+1 h✓ i ✓ ⇤ , ⌃(I ⌃ I) j i (✓ i ✓ ⇤ )i because M (j, i + 1) and ✓ i are independent, = E n 1 X i=0 D ✓ i ✓ ⇤ , 1 ⌃(⌃+ I) 1 ⇥ (I ⌃ I) (I ⌃ I) n i+1 ⇤ (✓ i ✓ ⇤ ) E 6 E n X i=0 D ✓ i ✓ ⇤ , 1 ⌃(⌃ + I) 1 (I ⌃ I)(✓ i ✓ ⇤ ) E using (⌃ + I) 4 I, = 1 E n X i=0 h✓ i ✓ ⇤ , ⌃(⌃ + I) 1 (✓ i ✓ ⇤ )i E n X i=0 h✓ i ✓ ⇤ , ⌃(✓ i ✓ ⇤ )i.
We thus simply need to bound

1 E P n i=0 h✓ i ✓ ⇤ , ⌃(⌃ + I) 1 (✓ i ✓ ⇤ )i, to get a bound on n 2 Ek⌃ 1/2 ( ✓n ✓ ⇤ )k 2 .
Recursion on operators. We have:

E ⇥ M (i, k)⌃(⌃+ I) 1 M (i, k) ⇤ ⇤ = E h M (i, k + 1) ⇥ I x k ⌦ x k I ⇤ ⌃(⌃ + I) 1 ⇥ I x k ⌦ x k I ⇤ M (i, k + 1) ⇤ i = E h M (i, k + 1) ⇣ ⌃(⌃ + I) 1 2 ⌃ + 2 ⇥ x k ⌦ x k + I ⇤ ⌃(⌃ + I) 1 ⇥ x k ⌦ x k + I ⇤ ⌘ M (i, k + 1) ⇤ i 4 E h M (i, k + 1) ⇥ ⌃(⌃ + I) 1 2 ⌃ + 2 (R 2 + 2 )⌃ ⇤ M (i, k + 1) ⇤ i = E h M (i, k + 1)⌃(⌃ + I) 1 M (i, k + 1) ⇤ i (2 (R 2 + 2 ))E h M (i, k + 1)⌃M (i, k + 1) ⇤ i , which leads to E h M (i, k + 1)⌃M (i, k + 1) ⇤ i 4 1 (2 (R 2 + 2 )) ⇣ E h M (i, k + 1)⌃(⌃ + I) 1 M (i, k + 1) ⇤ i E h M (i, k)⌃(⌃ + I) 1 M (i, k) ⇤ i⌘ .
(3.20)

Using the operator T on matrices defined below, this corresponds to showing

(I T ) ⇥ ⌃(⌃ + I) ⇤ 4 ⌃(⌃ + I) ⌃.
Noise term. For ✓ 0 ✓ ⇤ = 0, we have:

Eh✓ i ✓ ⇤ , ⌃(⌃ + I) 1 (✓ i ✓ ⇤ )i = 2 E i X k=1 i X j=1 " j x ⇤ j M (i, j + 1) ⇤ ⌃(⌃ + I) 1 M (i, k + 1)" k x k by expanding all terms, = 2 E i X k=1 " k x ⇤ k M (i, k + 1) ⇤ ⌃(⌃ + I) 1 M (i, k + 1)" k x k using independence, = 2 tr ✓ i X k=1 E" 2 k x k x ⇤ k EM (i, k + 1) ⇤ ⌃(⌃ + I) 1 M (i, k + 1) ◆ 6 2 2 tr ✓ i X k=1 EM (i, k + 1)⌃M (i, k + 1) ⇤ ⌃(⌃ + I) 1
◆ using our assumption regarding the noise.

Using the recurrence between operators

Eh✓ i ✓ ⇤ , ⌃(⌃ + I) 1 (✓ i ✓ ⇤ )i 6 2 2 (R 2 + 2 ) tr i X k=1 ✓ E h M (i, k + 1)⌃(⌃ + I) 1 M (i, k + 1) ⇤ ⌃(⌃ + I) 1 i E h M (i, k)⌃(⌃ + I) 1 M (i, k) ⇤ ⌃(⌃ + I) 1 i ◆ 6 2 2 (R 2 + 2 ) tr ✓ E h M (i, i + 1)⌃(⌃ + I) 1 M (i, i + 1) ⇤ ⌃(⌃ + I) 1 i E h M (i, 1)⌃(⌃ + I) 1 M (i, 1) ⇤ ⌃(⌃ + I) 1 i ◆ by summing,
This implies that for the noise process

Ek⌃ 1/2 ( ✓n ✓ ⇤ )k 2 6 ✓ 2 n + 1 tr ⇥ ⌃ 2 (⌃ + I) 2 ⇤ ◆ 1 1 (R 2 /2 + )
.

Note that when tends to zero, we recover the optimal variance term.

Noiseless term. Without noise, we then need to bound:

1 E n X i=0 h✓ i ✓ ⇤ , ⌃(⌃ + I) 1 (✓ i ✓ ⇤ )i, with ✓ i ✓ ⇤ = M (i, 1)(✓ 0 ✓ ⇤ ), that is 1 E n X i=0 tr h M (i, 1) ⇤ ⌃(⌃ + I) 1 M (i, 1)(✓ 0 ✓ ⇤ )(✓ 0 ✓ ⇤ ) ⇤ i .
We follow here the proof of [START_REF] Défossez | Averaged least-mean-squares: bias-variance trade-offs and optimal sampling distributions[END_REF] and consider the operator T from symmetric matrices to symmetric matrices defined as

T A = (⌃ + I)A + A(⌃ + I) E ⇥ (x n ⌦ x n + I)A(x n ⌦ x n + I) ⇤ .
of the form T A = (⌃ + I)A + (⌃ + I)A SA. The operator S is self-adjoint and positive. Moreover:

hA, SAi = E tr ⇥ A(x n ⌦ x n + I)A(x n ⌦ x n + I) ⇤ = tr ⇥ 2A 2 ⌃ + 2 A 2 ⇤ + E tr ⇥ hx n , Ax n i 2 ⇤ 6 tr ⇥ 2A 2 ⌃ + 2 A 2 ⇤ + E tr ⇥ kx n k 2 x n ⌦ x n , A 2 ⇤ (Cauchy-Schwarz inequality) 6 tr ⇥ 2A 2 ⌃ + 2 A 2 ⇤ + R 2 tr ⌃A 2 6 (R 2 + 2 ) tr ⇥ ⌃ + I]A 2 .
We have for any symmetric matrix A:

EM (i, 1) ⇤ AM (i, 1) = (I T ) i A.
Thus,

E n X i=0 tr h M (i, 1) ⇤ ⌃(⌃ + I) 1 M (i, 1)(✓ 0 ✓ ⇤ )(✓ 0 ✓ ⇤ ) ⇤ i = E n X i=0 hh(I T ) i A, E 0 ii with E 0 = (✓ 0 ✓ ⇤ )(✓ 0 ✓ ⇤ ) ⇤ and A = ⌃(⌃ + I) 1 . This leads to 1 Ehh 1 T 1 (I (I T ) n+1
)A, E 0 ii, where hh•, •ii denote the dot-product between self-adjoint operators.

The sum is less than its limit for n ! 1, and thus, we can get rid of the term (I T ) n+1 , and we need to bound

2 hhM, E 0 ii = 2 hhT 1 (⌃(⌃ + I) 1 ), E 0 ii, with M := T 1 ⇥ ⌃(⌃ + I) 1 ⇤ , i.e., such that ⌃(⌃ + I) 1 = (⌃ + I)M + M (⌃ + I) E(x n ⌦ x n + I)M (x n ⌦ x n + I) = (⌃ + I)M + M (⌃ + I) SM. (3.21)
So that :

M = ⇥ (⌃+ I)⌦I +I ⌦ (⌃+ I) ⇤ 1 ⇥ ⌃(⌃+ I) 1 ⇤ + ⇥ (⌃+ I)⌦I +I ⌦(⌃+ I) ⇤ 1 SM = 1 2 ⌃(⌃ + I) 2 + ⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 SM.
The operator (⌃ + I) ⌦ I + I ⌦ (⌃ + I) is self adjoint, and so is its inverse, thus:

hhM, E 0 ii = hh 1 2 ⌃(⌃ + I) 2 + 3 ⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 SM, E 0 ii = 1 2 hh⌃(⌃ + I) 2 , E 0 ii + hhSM, ⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 E 0 ii = 1 2 tr(⌃(⌃ + I) 2 E 0 ) + hhSM, ⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 E 0 ii.
Moreover,

E 0 = (✓ 0 ✓ ⇤ )(✓ 0 ✓ ⇤ ) ⇤ = (⌃ + I) 1/2 (⌃ + I) 1/2 (✓ 0 ✓ ⇤ )(✓ 0 ✓ ⇤ ) ⇤ (⌃ + I) 1/2 (⌃ + I) +1/2 4 [(✓ 0 ✓ ⇤ ) ⇤ (⌃ + I) 1 (✓ 0 ✓ ⇤ )] (⌃ + I), as (⌃ + I) 1/2 (✓ 0 ✓ ⇤ )(✓ 0 ✓ ⇤ ) ⇤ (⌃ + I) 1/2 4 (✓ 0 ✓ ⇤ ) ⇤ (⌃ + I) 1 (✓ 0 ✓ ⇤ )I. Thus, as [(⌃ + I) ⌦ I + I ⌦ (⌃ + I)] 1 is an non-decreasing operator on (S n (R), 4) (see technical Lemma 16 in Appendix 3.D): ⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 E 0 4 ⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 [(✓ 0 ✓ ⇤ ) ⇤ (⌃ + I) 1 (✓ 0 ✓ ⇤ )](⌃ + I) = (✓ 0 ✓ ⇤ ) ⇤ (⌃ + I) 1 (✓ 0 ✓ ⇤ ) 2 I.
Thus as SM is positive :

2 hhM, E 0 ii  1 2 2 tr(⌃(⌃ + I) 2 E 0 ) + (✓ 0 ✓ ⇤ ) ⇤ (⌃ + I) 1 (✓ 0 ✓ ⇤ ) 2 tr(SM ).
Moreover we can upper bound tr(SM ) : using Equation (3.21) we have

tr(⌃(⌃ + I) 1 ) = 2 tr(⌃ + I)M tr E(x n ⌦ x n + I)M (x n ⌦ x n + I)
then, using Assumption (A 1 ) :

tr E(x n ⌦x n + I)M (x n ⌦x n + I) 6 R 2 trM ⌃+2 trM ⌃ + 2 tr M 6 (R 2 +2 ) trM (⌃+ I).
This implies

tr ⇥ ⌃(⌃+ I) 1 ⇤ > 2 R 2 + 2 tr E(x n ⌦ x n + I)M (x n ⌦ x n + I), > 1 R 2 +2 tr E(x n ⌦ x n + I)M (x n ⌦ x n + I) since (R 2 +2 ) 6 1, > 1 R 2 + 2 tr SM.
Thus finally:

2 hhM, E 0 ii  1 2 2 tr E 0 ⌃(⌃ + I) 2 + (✓ 0 ✓ ⇤ ) ⇤ (⌃ + I) 1 (✓ 0 ✓ ⇤ ) 2 (R 2 + 2 ) tr(⌃(⌃ + I) 1 ),
which leads to the desired error term.

3.B.4 Proof When Only k✓ 0 ✓ ⇤ k Is Finite

When = 0, without noise, we then need to bound:

1 E n X i=0 h✓ i ✓ ⇤ , ✓ i ✓ ⇤ i, with ✓ i ✓ ⇤ = M (i, 1)(✓ 0 ✓ ⇤ ), that is 1 E n X i=0 tr h M (i, 1) ⇤ M (i, 1)(✓ 0 ✓ ⇤ )(✓ 0 ✓ ⇤ ) ⇤ i .
By definition of M (i, 1) we have that EM (i, 1) ⇤ M (i, 1) 4 I leading to

1 E n X i=0 h✓ i ✓ ⇤ , ✓ i ✓ ⇤ i  (n + 1)k✓ 0 ✓ ⇤ k 2 .
For the regularization-based bias we also have

k ⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2  k⌃ 1/2 (⌃ + I) 1/2 (✓ 0 3.B.5 Proof When the Noise Is Not Structured For k✓ 0 ✓ ⇤ k = 0 we have ✓ n ✓ ⇤ = P n k=1 M (n, k + 1)" k x k which leads to Ek⌃ 1/2 (✓ n ✓ ⇤ )k 2 = 2 n X k=1 tr EM (n, k + 1) ⇤ ⌃M (n, k + 1)V, where V = E" 2 k x k x ⇤ k .
And using the recursion on operators in Eq. (3.20) by changing order of elements we have

E h M (n, k + 1) ⇤ ⌃M (n, k + 1) i 4 1 (2 (R 2 + 2 )) ⇣ E h M (n, k + 1) ⇤ ⌃(⌃ + I) 1 M (n, k + 1) i E h M (n, k) ⇤ ⌃(⌃ + I) 1 M (n, k) i⌘ .
And by adding the terms

Ek⌃ 1/2 (✓ n ✓ ⇤ )k 2 4 2 (2 (R 2 + 2 )) tr ⌃(⌃ + I) 1 V,
We conclude by convexity

Ek⌃ 1/2 ( ✓n ✓ ⇤ )k 2 4 2 (2 (R 2 + 2 )) tr ⌃(⌃ + I) 1 V.

3.C Convergence of Accelerated Averaged Stochastic Gradient Descent

We now prove Theorem 8. We thus consider iterates satisfying Eq. (3.10), under Assumptions (A 3 ), (A 4 ). We consider a fixed step size such that (⌃+ I) 4 I. Seeing Eq. (3.10) as a linear second order for ✓ n , we will derive from exact calculations a decomposition of the errors a sum of three terms that will be studied independently. The proof is organized as follows: in Section 3.C.1, we state the formulation as a second order linear system and derive the three main terms that have to be studied (see Lemma 13). Section 3.C.2 studies asymptotic behaviors of the three terms, ignoring some exponentially decreasing terms, in order to give insight of how they behave. This section is not necessary for the proof, indeed a direct and exact calculation in the eigenbasis of ⌃, following O'Donoghue and Candès [2013], is provided in Section 3.C.3. Results are summed up in Section 3.C.4.

3.C.1 General Expansion

We study the regularized stochastic accelerated gradient descent recursion defined for n 1 by

✓ n = ⌫ n 1 f 0 (⌫ n 1 ) (⌫ n ✓ 0 ) + ⇠ n ⌫ n = ✓ n + (✓ n ✓ n 1 ),
starting from ✓ 0 = ⌫ 0 2 H. We may rewrite it for a quadratic function f :

✓ 7 ! 1 2 h✓ ✓ ⇤ , ⌃(✓ ✓ ⇤ )i for n 2 as ✓ n = ⇥ I ⌃ I ⇤⇥ ✓ n 1 + (✓ n 1 ✓ n 2 ) i + ⇠ n + ✓ 0 + ⌃✓ ⇤ , with ✓ 0 2 H and ✓ 1 = ⇥ I ⌃ I ⇤ ✓ 0 + ⇠ 1 + ✓ 0 + ⌃✓ ⇤ .
And by centering around the optimum, we get:

✓ n ✓ ⇤ = ⇥ I ⌃ I ⇤⇥ ✓ n 1 ✓ ⇤ + (✓ n 1 ✓ ⇤ ✓ n 2 + ✓ ⇤ ) i + ⇠ n + (✓ 0 ✓ ⇤ ).
Thus this is a second order iterative system which is standard to cast in a linear form

⇥ n = F ⇥ n 1 + ⌅ n + ⇥ , (3.22) with T = I ⌃ I, F = ✓ (1 + )T T I 0 ◆ , ⇥ n = ✓ ✓ n ✓ ⇤ ✓ n 1 ✓ ⇤ ◆ , ⇥ 0 = ✓ ✓ 0 ✓ ⇤ ✓ 0 ✓ ⇤ ◆ , ⌅ n = ✓ ⇠ n 0 ◆ and ⇥ = ✓ ✓ 0 ✓ ⇤ 0 ◆ .
We are interested in the behavior of the average ⇥n = 1 n+1 P n k=0 ⇥ k for which we have the following general convergence result:

Lemma 13. For all 2 R + and such that (⌃ + I) 4 I and any matrix C the average of the iterates ⇥ n defined by Eq. (3.22) satisfy for P k

(def ) = C 1/2 (I F k )(I F ) 1 , with ⇥0 = ⇥ 0 (I F ) 1 ⇥ , Eh ⇥n , C ⇥n i  2 ( ) 2 kC 1/2 (I F ) 1 ⇥ k 2 + 2 (n + 1) 2 kP n+1 ⇥0 k 2 + 2 (n + 1) 2 n X j=1 tr P j V P > j .
The error thus decomposes as the sum of three main terms:

-the two first ones are bias terms, one arising from the regularization (the first one), and one arising computation (the second one), -a variance term. which is the last one. We remark that as we have assumed that ⌃ is invertible, the matrix I F can be shown to be invertible for all the considered .

The regularization-based term will be studied directly whereas the two others will be studied in two stages. First a heuristic will lead to an asymptotic bound then an exact computation will give a non-asymptotic bound. Proof. The sequence ⇥ n satisfies a linear recursion, from which we get, for all n 1:

⇥ n = F n ⇥ 0 + n X k=1 F n k ⌅ k + n X k=1 F n k ⇥ = F n ⇥ 0 + n X k=1 F n k ⌅ k + (I F n )(I F ) 1 ⇥ .
We study the averaged sequence: ⇥n = 1 n+1 P n k=0 ⇥ k . We get, using the identity

P n 1 k=0 F k = (I F n )(I F ) 1 , ⇥n = 1 n + 1 n X k=0 F k ⇥ 0 + n + 1 n X k=1 k X j=1 F k j ⌅ j + n + 1 n X k=1 (I F k )(I F ) 1 ⇥ . With ⇥0 = ⇥ 0 (I F ) 1 ⇥ , and P n k=1 (I F k ) = P n k=0 (I F k ) = [n + 1 (I F n+1 )(I F ) 1 ].
Using summation formulas for geometric series, we derive:

⇥n = 1 n + 1 (I F n+1 )(I F ) 1 ⇥0 + n + 1 n X k=1 k X j=1 F k j ⌅ j + (I F ) 1 ⇥ = 1 n + 1 (I F n+1 )(I F ) 1 ⇥0 + n + 1 n X j=1 n X k=j F k j ⌅ j + (I F ) 1 ⇥ = 1 n + 1 (I F n+1 )(I F ) 1 ⇥0 + n + 1 n X j=1 n j X k=0 F k ⌅ j + (I F ) 1 ⇥ = 1 n+1 (I F n+1 )(I F ) 1 ⇥0 + n+1 n X j=1 (I F n+1 j )(I F ) 1 ⌅ j + (I F ) 1 ⇥ = 1 n+1 (I F n+1 )(I F ) 1 ⇥0 + n+1 n X j=1 (I F j )(I F ) 1 ⌅ n+1 j + (I F ) 1 ⇥ .
Using martingale square moment inequalities which amount to consider ⌅ i , i = 1, ..., n independent, so that the variance of the sum is the sum of variances, and denoting by

V = E[⌅ n ⌦ ⌅ n ] we have for any positive semi-definite C, Eh ⇥n , C ⇥n i = C 1/2 ✓ 1 n + 1 (I F n+1 )(I F ) 1 ⇥0 + (I F ) 1 ⇥ ◆ 2 + 2 (n + 1) 2 n X j=1 tr(I F j )(I F ) 1 V (I F > ) 1 (I F j ) > C,
where C 1/2 denotes a symmetric square root of C. Define P k

(def ) = C 1/2 (I F k )(I F ) 1 , we have, Using Minkowski's inequality and inequality (a + b) 2  2(a 2 + b 2 ) for any a, b 2 R, Eh ⇥n , C ⇥n i = 1 n + 1 P n+1 ⇥0 + C 1/2 (I F ) 1 ⇥ 2 + 2 (n + 1) 2 n X j=1 tr P j V P > j  2 ( ) 2 kC 1/2 (I F ) 1 ⇥ k 2 + 2kP n+1 ⇥0 k 2 (n + 1) 2 + 2 (n+1) 2 n X j=1 tr P j V P > j .
This concludes proof of Lemma 13.

3.C.2 Asymptotic Expansion

To give the main terms that we expect, we first provide an asymptotic analysis, which shall only be understood as an insight and is not necessary for the proof. Operator F will have only eigenvalues smaller than 1, thus |||F j ||| will decrease exponentially to 0 as j ! 1 (even if |||F ||| 3 might be bigger than 1). The asymptotic analysis relies on ignoring all terms in which F j appears. We thus approximately have:

Eh ⇥n , C ⇥n i  2 ( ) 2 kC 1/2 (I F ) 1 ⇥ k 2 + 2 C 1/2 1 n + 1 (I F n+1 )(I F ) 1 ⇥0 2 + 2 (n + 1) 2 n X j=1 tr(I F j )(I F ) 1 V (I F > ) 1 (I F j ) > C ⇡ 2 ( ) 2 kC 1/2 (I F ) 1 ⇥ k 2 + 2 C 1/2 1 n + 1 (I F ) 1 ⇥0 2 + 2 (n + 1) 2 n X j=1 tr(I F ) 1 V (I F > ) 1 C,
where, as it has been explained ⇡ stands for an equality up to terms that will decay exponentially. However, these terms have to be studied very carefully, what will be done in the Section 3.C.3.

Using the matrix inversion lemma we have for

C = ✓ c 0 0 0 ◆ , I F = ✓ (1 + )( ⌃ + I) I (I ( ⌃ + I)) I I ◆ (I F ) 1 = ✓ ( ⌃ + I) 1 I ( ⌃ + I) 1 ( ⌃ + I) 1 (1 + )I ( ⌃ + I) 1 ◆ (3.23) C 1/2 (I F ) 1 = ✓ c 1/2 ( ⌃ + I) 1 c 1/2 I ( ⌃ + I) 1 0 0 ◆ .
3. |||F ||| denotes the operator norm of F , i.e., sup kxk1 kF xk.

Regularization based term. This gives for the regularization based term

C 1/2 (I F ) 1 ⇥ 2 = ✓ c 1/2 ( ⌃+ I) 1 c 1/2 I ( ⌃+ I) 1 0 0 ◆ ✓ ✓ 0 ✓ ⇤ 0 ◆ 2 = ✓ 1 ◆ 2 k(c 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ ))k 2 . (3.24)
The computation of this term is exact (not asymptotic).

Bias term. For the bias term we have

⇥0 = ⇥ 0 (I F ) 1 ⇥ = ✓ ✓ 0 ✓ ⇤ ✓ 0 ✓ ⇤ ◆ ✓ ( ⌃ + I) 1 I ( ⌃ + I) 1 ( ⌃ + I) 1 (1 + )I ( ⌃ + I) 1 ◆ ✓ ✓ 0 ✓ ⇤ 0 ◆ = ✓ ✓ 0 ✓ ⇤ ✓ 0 ✓ ⇤ ◆ ✓ ( ⌃ + I) 1 (✓ 0 ✓ ⇤ ) ( ⌃ + I) 1 (✓ 0 ✓ ⇤ ) ◆ = ✓ [I (⌃ + I) 1 ](✓ 0 ✓ ⇤ ) [I (⌃ + I) 1 ](✓ 0 ✓ ⇤ ) ◆ .
Thus this gives for the dominant term

C 1/2 (I F ) 1 ⇥0 2 = ✓ c 1/2 ( ⌃ + I) 1 c 1/2 I ( ⌃ + I) 1 0 0 ◆ ⇥0 2 = k(c 1/2 [(1 )( ⌃+ I) 1 + I][I (⌃+ I) 1 ](✓ 0 ✓ ⇤ )k 2 . And if c commutes with ⌃ we have the bound for 2 [ 1 p 1+ p , 1] C 1/2 (I F ) 1 ⇥0 2  ( (1 ) + )k(c 1/2 [I (⌃ + I) 1 ](✓ 0 ✓ ⇤ )k 2  ( 2 p + 1)k(c 1/2 [I (⌃ + I) 1 ](✓ 0 ✓ ⇤ )k 2 .
Variance term. And for the variance term with

V = ✓ v 0 0 0 ◆ , we have C 1/2 (I F ) 1 V 1/2 = ✓ c 1/2 ( ⌃ + I) 1 v 1/2 0 0 0 ◆ , and
tr C 1/2 (I F ) 1 V (I F > ) 1 C 1/2 = tr c( ⌃ + I) 1 v( ⌃ + I) 1 .
This gives the three dominant terms. However in order to control the remainders we have to compute the eigenvalues more carefully, as done in the next section.

3.C.3 Direct Computation without the Regularization Based Term

We derive now direct computation both the bias and variance terms. This is not required for the regularization based term whose previous expression in Eq. (3.24) is already non-asymptotic. Following O'Donoghue and Candès [2013] we consider an eigen-decomposition of the matrix F , in order to study independently the recursion on eigenspaces. We assume ⌃ has eigenvalues (s i ) and we decompose vectors in an eigenvector basis of ⌃ we denote by (p i ), with ✓ i n = p > i ✓ n and ⇠ i n = p > i ⇠ n and we have the reduced equation:

⇥ i n+1 = F i ⇥ i n + ⌅ i n+1 .
with

⇥ i 0 = ⇥i 0 , F i = ✓ (1 + )T i T i 1 0 ◆ , with T i = 1 s i .
Computing initial point ⇥i 0 .

⇥i 0 = ⇥ i 0 (I F i ) 1 ⇥ i , with ⇥ i 0 = ✓ ✓ i 0 ✓ i ⇤ ✓ i 0 ✓ i ⇤ ◆ , ⇥ i = ✓ ✓ i 0 ✓ i ⇤ 0 ◆ and (I F i ) 1 given in Eq. (3.23). Thus ⇥i 0 = ✓ ✓ i 0 ✓ i ⇤ ✓ i 0 ✓ i ⇤ ◆ ( s i + ) ✓ 1 (( s i + ) 1) 1 (1 + )( s i + ) ◆ ✓ ✓ i 0 ✓ i ⇤ 0 ◆ = ✓ (1 +s i )(✓ i 0 ✓ i ⇤ ) (1 +s i )(✓ i 0 ✓ i ⇤ ) ◆ . (3.25)
Study of spectrum of F i . Depending on , F i may have two distinct complex eigenvalues of same modulus, only one (double) eigenvalue, or two real eigenvalues.

We only consider the two former cases, which we detail below. Indeed, the characteristic polynomial

F i (X) def = det(XI F i ) = X 2 (1 + )(1 (s i + ))X + (1 (s i + )) has discriminant i = (1 (s i + ))((1+ ) 2 (1 (s i + )) 4 ) which is non positive as far as 2 [ ; + ], with = 1 p (s i + ) 1+ p (s i + ) , + = 1+ p (s i + ) 1 p (s i + )
.

Two Distinct Eigenvalues

We first assume that F i has two distinct complex eigenvalues

r ± = (1 + )(1 (s i + )) ± p 1 p i 2 which are conjugate. Thus the roots are of the form ⇢ i e ±i! i with ⇢ i = p (1 (s i + )), cos(! i ) = (1+ )(1 (s i + )) 2⇢ i , ! i 2 [ ⇡/2; ⇡/2] and sin(! i ) = p i 2⇢ i . Let Q i = ✓ r i r + i 1 1
◆ be the transfer matrix into an eigenbasis of F i , i.e., F i =

Q i D i Q 1 i with D i = ✓ r i 0 0 r + i ◆ and Q 1 i = 1 r i r + i ✓ 1 r + i 1 r i ◆ .
Computing P i,k . We first compute the matrix P i,k : With

C 1/2 i = ✓ p c i 0 0 0 ◆ , C 1/2 i Q i = ✓ r i p c i r + i p c i 0 0 ◆ we have C 1/2 i Q i (I D k i )(I D i ) 1 = p c i 1 (r i ) k 1 r i r i 1 (r + i ) k 1 r + i r + i 0 0 ! ,
and, when developing and regrouping terms which depend on k, we get :

P i,k = C 1/2 i Q i (I D k i )(I D i ) 1 Q 1 i = p c i r i r + i 1 (r i ) k 1 r i r i 1 (r + i ) k 1 r + i r + i 1 (r + i ) k 1 r + i r i r + i 1 (r i ) k 1 r i r + i r i 0 0 ! = p c i 1 (1 r i )(1 r + i ) r + i r i (1 r i )(1 r + i ) 0 0 ! p c i r i r + i (r i ) k+1 1 r i (r + i ) k+1 1 r + i (r + i ) k+1 1 r + i r i (r i ) k+1 1 r i r + i 0 0 ! .
We also have

P i,k = C 1/2 i Q i (I D k i )(I D i ) 1 Q 1 i = P k 1 j=0 R i,j with R i,j = C 1/2 i Q i D j i Q 1 i = p c i ✓ (r i ) j+1 (r + i ) j+1 0 0 ◆ Q 1 i = p s i r i r + i ✓ (r i ) j+1 (r + i ) j+1 r + i (r i ) j+1 + r i (r + i ) j+1 0 0
◆ , but computing error terms based in R i,j before summing these errors gives a looser error bound than a tight calculation using P i,k . More precisely, if we use

P i,k ⇥ i 0 = P k 1 j=0 R i,j ⇥ i 0 to upper bound kP i,k ⇥ i 0 k  P k 1 j=0 kR i,j ⇥ i 0 k
, we end up with a worse bound.

Bias term. Thus, for the bias term:

P i,k ⇥ i 0 = p c i ✓ i 0 1 r + i r i (1 r i )(1 r + i ) p c i ✓ i 0 r i r + i h (r i ) k+1 1 r + i 1 r i (r + i ) k+1 1 r i 1 r + i i 0 ! = p c i ✓ i 0 p (1 r i )(1 r + i ) 0 @ ⇥ (1 r + i r i ) ⇢ k i A 1 ⇤ p (1 r i )(1 r + i ) 0 1 A ,
where

⇢ k i A 1 = (r i ) k+1 (1 r + i ) 2 (r + i ) k+1 (1 r i ) 2 r i r + i .
This can be bound with the following lemma Lemma 14. For all ⇢ 2 (0, 1) and ! 2 [ ⇡/2; ⇡/2] and r ± = ⇢(cos(!) ± p 1 sin(!)) we have:

1 r + r ⇢ k |A 1 | |1 r + |  3 + 3⇢ k  6 (3.26)
We note that the exact constant seems empirically to be 2. This lemma is proved as Lemma 17 in Appendix 3.D. This gives for the bias term

kP i,k ⇥ i 0 k = p c i (✓ i 0 ) p (1 r i )(1 r + i ) ⇥ 1 p (1 r i )(1 r + i ) (1 r + i r i ) ⇢ k i A 1 ⇤  6 p c i (✓ i 0 ) p (s i + )
, since:

(1 r i )(1 r + i ) = 1 2 Re (r + i ) + |r + i | 2 = 1 (1 + )(1 (s i + )) + (1 (s i + )) = (s i + ).
We also have a looser bound using P i,k ⇥ i 0 =

P k 1 j=0 R i,j ⇥ i 0 . R i,j ⇥ i 0 = p c i ✓ i 0 r i r + i (1 r + i )(r i ) j+1 (1 r i )(r + i ) j+1 = p c i ✓ i 0 ✓ (r i ) j+1 (r + i ) j+1 r i r + i r + i (r i ) j+1 r i (r + i ) j+1 r i r + i ◆ (De Moivre's formula) = p c i ✓ i 0 ✓ ⇢ j+1 i sin(! i (j + 1)) ⇢ i sin(! i ) ⇢ i e i! i ⇢ j+1 i e i! i (j+1) ⇢ i e i! i ⇢ j+1 i e +i! i (j+1) ⇢ i e i! i ⇢ i e i! i ◆ = p c i ✓ i 0 ✓ ⇢ j+1 i sin(! i (j + 1)) ⇢ i sin(! i ) ⇢ j+1 i e i! i j e +i! i j e i! i e i! i ◆ = p c i ✓ i 0 ✓ ⇢ j i sin(! i (j + 1)) sin(! i ) ⇢ j+1 i sin(! i j) sin(! i ) ◆  (1 + e 1 ) p c i ✓ i 0 using Lemma 18 (see proof in Appendix 3.D),
which also gives for the bias term

kP i,k ⇥ i 0 k  (1 + e 1 ) p c i ✓ i 0 k.
Thus we have the final bound:

kP i,k ⇥ i 0 k 2  min ( 36 c i (✓ i 0 ) 2 (s i + ) , 6n(1 + e 1 ) c i (✓ i 0 ) 2 p (s i + ) , n 2 (1 + e 1 ) 2 c i (✓ i 0 ) 2 ) .(3.27)
Variance term. As for the variance term, with

V i = ✓ v i 0 0 0 ◆ , we have tr P i,k V i P i,k = P i,k ✓ p v i 0 ◆ 2 . P i,k ✓ p v i 0 ◆ = p v i c i (1 r i )(1 r + i ) " 1 + (r i ) k+1 (1 r + i ) (r + i ) k+1 (1 r i ) r + i r i # = p v i c i (s i + ) " 1 ⇢ k i B i,k # ,
where

⇢ k i B i,k = (r i ) k+1 (1 r + i ) (r + i ) k+1 (1 r i ) r + i r i ,
which we can bound using the following Lemma:

Lemma 15. For all ⇢ 2 (0, 1) and ! 2 [ ⇡/2; ⇡/2] and r ± = ⇢(cos(!) ± p 1 sin(!)) we have:

⇢ k B k  1.75.
Where we note that the exact majoration seems to be 1.3. This Lemma is proved as Lemma 19 in Appendix 3.D.

We can also have a looser bound using P i,k

✓ v 1/2 i 0 ◆ = P k 1 j=0 R i,j ✓ v 1/2 i 0 ◆ and R i,j ✓ v 1/2 i 0 ◆ = p c i v i r i r + i (r i ) j+1 (r + i ) j+1 = p c i v i ⇢ j+1 i sin(! i (j + 1)) ⇢ i sin(! i )  (j + 1) p c i v i , using the inequality | sin(k! i )|  k| sin(! i )| and P i,k ✓ v 1/2 i 0 ◆  p c i v i (k+1)k 2 .
This gives for the Variance term

n X k=1 tr P i,k V i P i,k  v i c i n X k=1 min ( ⇥ 1 ⇢ k i B 1,k ⇤ 2 2 (s i + ) 2 , ⇥ 1 ⇢ k i B 1,k ⇤ k(k + 1) 2 (s i + ) , k 2 (k + 1) 2 4 )  v i c i min ( 8n 2 (s i + ) 2 ,
(n + 1) 3 2 (s i + )

, (n + 1) 5 20

) .

(3.28)

One Coalescent Eigenvalue

We now turn to the case where F has two coalescent eigenvalues, which happens when the discriminant = 0. We assume that F i has one coalescent eigenvalue

r i = (1+ )(1 (s i + )) 2 . Then, with = 1 p (s i + ) 1+ p (s i + ) , r i = (1+ )(1 (s i + )) 2 = 1 p (s i + ).
Then F i can be trigonalized as

F i = Q i D i Q 1 i with Q i = ✓ r i 1 1 0 ◆ , D i = ✓ r i 1 0 r i ◆ and Q 1 i = ✓ 0 1 1 r i ◆ .
We note that for all k 0, then

D k i = r k 1 i ✓ r i k 0 r i ◆ .
Computing P i,k . We first compute P i,k :

(I 2 D i ) 1 = ✓ 1 1 r i 1 (1 r i ) 2 0 1 1 r i ◆ and (I 2 D k i )(I 2 D i ) 1 = 1 r k i 1 r i 1 r k i (1 r i ) 2 kr k 1 i 1 r i 0 1 r k i 1 r i ! . Thus with C 1/2 i Q i = ✓ p c i r i p c i 0 0 ◆ we have C 1/2 i Q i (I 2 D k i )(I 2 D i ) 1 = p c i 1 r k i 1 r i r i 1 r k i (1 r i ) 2 kr k i 1 r i 0 0 ! .
And, computing as previously the matrices products, we derive:

P i,k = C 1/2 i Q i (I 2 D k i )(I 2 D i ) 1 Q 1 i = p c i 1 r k i (1 r i ) 2 kr k i 1 r i 1 r k i 1 r i r i ( 1 r k i (1 r i ) 2 kr k i 1 r i )r i 0 0 ! = p c i 1 r k i (1 r i ) 2 kr k i 1 r i 1 r k i (1 r i ) 2 (r i ) 2 + kr k+1 i 1 r i 0 0 ! = p c i 1 r i 1 r k i 1 r i kr k i 1 r k i 1 r i (r i ) 2 + kr k+1 i 0 0 ! .
Bias term. We thus have:

P i,k ⇥ i 0 = p c i 1 r i 1 r k i 1 r i kr k i 1 r k i 1 r i (r i ) 2 + kr k+1 i 0 0 ! ✓ ✓ i 0 ✓ i 0 ◆ 109 = ✓ i 0 p c i ✓ (1 r k i ) 1+r i 1 r i kr k i 0 ◆ ,
and this gives for the bias term:

kP i,k ⇥ i 0 k 2 = (✓ i 0 ) 2 c i h (1 r k i ) 1 + r i 1 r i kr k i i 2 = (✓ i 0 ) 2 c i h 1 + r i 1 r i ⇣ k + 1 + r i 1 r i ⌘ r k i i 2 ,
developing the product, then using formulas for r i ,

kP i,k ⇥ i 0 k 2 = (✓ i 0 ) 2 c i h 2 p (s i + ) p (s i + ) ⇣ k+ 2 p (s i + ) p (c i + ) ⌘ ( 1
p (s i + )) k i 2 = (✓ i 0 ) 2 c i (s i + ) h 2 p (s i + ) k p (s i + ) +2 p (s i + ) (1 p (s i + )) k i 2 = (✓ i 0 ) 2 c i (s i + ) h 2 p (s i + ) 2 + (k 1) p (s i + ) (1 p (s i + )) k i 2  4 (✓ i 0 ) 2 c i (s i + )
, using Lemma 20 in Appendix 3.D.

(3.29)

Variance term. With V = ✓ v i 0 0 0 ◆ , tr P i,k V P i,k = s i (1 r i ) 2 1 r k i 1 r i kr k i 1 r k i 1 r i (r i ) 2 +kr k+1 i 0 0 ! ✓ v i 0 0 0 ◆ 1 r k i 1 r i kr k i 1 r k i 1 r i (r i ) 2 +kr k+1 i 0 0 ! > = s i v i (1 r i ) 2 h 1 r k i 1 r i kr k i i 2 = v i h i (s i + ) h 1 r k i 1 r i kr k i i 2 = v i h i (s i + )(1 r i ) 2 h 1 r k i (1 r i )kr k i i 2 = v i h i 2 (s i + ) 2 h 1 (1 + k p (s i + ))(1 p (s i + )) k i 2 ,
and

n X k=1 tr P i,k V P i,k = v i s i 2 (s i + ) 2 n X k=1 h 1 (1 + k p (s i + ))(1 p (s i + )) k i 2  n v i s i 2
(s i + ) 2 using Lemma 20 in Appendix 3.D.

(3.30)

Alternative bounds for the bias and the variance term, as in Equations(3.24), (3.27) may be derived as well. Combining all these results, we are now able to state Theorem 8.

3.C.4 Conclusion

Combining results from Lemma 13, and Equations (3.24), (3.27), (3.28), with c = ⌃, and using the following simple facts:

-For the least squares regression function, with c = ⌃, Eh ⇥n , C ⇥n i = Ef ( ✓n ) f (✓ ⇤ ).

-Under assumption A

3 , A 4 , we have V 4 ⌧ 2 ⌃.
-The squared norm of a vector is the sum of its squared components on the orthonormal eigenbasis. For example

kP n+1 ⇥ 0 k 2 = P d i=1 kP i,n+1 ⇥ i 0 k 2 . -
Ef ( ✓n ) f (✓ ⇤ )  2 k 1/2 ⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2 + d X i=1 2 (n + 1) 2 min ⇢ 36 c i ( ✓i 0 ) 2 (s i + ) , 6n(1 + e 1 ) c i ( ✓i 0 ) 2 p (s i + ) , n 2 (1 + e 1 ) 2 c i ( ✓i 0 ) 2 + d X i=1 2 (n + 1) 2 v i c i min ( 8n 2 (s i + ) 2 , (n + 1) 3 2 (s i + ) , (n + 1) 5 20
) .

This implies, using the Equation (3.25) for the initial point, using c i = i and regrouping sums as traces or norms:

Ef ( ✓n ) f (✓ ⇤ )  2 k 1/2 ⌃ 1/2 (⌃ + I) 1 (✓ 0 ✓ ⇤ )k 2 + 2 min ⇢ 36k⌃ 1/2 (⌃+ I) 1/2 (✓ 0 ✓ ⇤ )k 2 (n + 1) 2 , (1+e 1 ) 2 k⌃ 1/2 (✓ 0 ✓ ⇤ )k 2 + min ( 8 tr(V ⌃(⌃ + I) 2 ) n + 1 , n tr(V ⌃(⌃ + I) 1 )
) ,

which gives exactly Theorem 8 using V 4 ⌧ 2 ⌃ in the Variance term, and 1/2 (⌃ + I) 1/2 4 I in the first term.

3.D Technical Lemmas

The following sequence of Lemmas appear in the proof. They are mostly independent and rely on simple calculations.

Lemma 16. The operator ⇥ (⌃+ I)⌦I +I ⌦(⌃+ I)

⇤ 1 is a non-decreasing operator on (S n , 4).

Proof. Lemma means that for two matrices M, N 2 S n (R) such that M 4 N , then

⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 M 4 ⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 N.
It is equivalent to show that for any symmetric positive matrix A 2 S + n ,

⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 A 2 S + n (R).
We consider a matrix A 2 S + n (R). A can be decomposed as a sum of (at most) n rank one matrices

A = P n i=1 ! i ! > i , with ! i 2 R n .
We thus just have to prove that for some

! 2 R n , ⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 !! > 2 S + n (R). Let ⌃ = P i>0 µ i e i ⌦ e i is the eigenvalue decomposition of ⌃, then ⇥ (⌃ + I) ⌦ I + I ⌦ (⌃ + I) ⇤ 1 !! > = X i,j>0 h!, e i ih!, e j i µ i + µ j + 2 e i ⌦ e j .
Thus, in the orthonormal basis of eigenvectors, this is thus Hadamard product between X i,j>0

h!, e i ih!, e j ie i ⌦ e j = !! > and the matrix C = ⇣

1 µ i +µ j +2 i,j>0

⌘

. Matrix C is a Cauchy matrix and is thus positive. Moreover the Hadamard product of two positive matrices is positive, which concludes the proof.

Remark: surprisingly, the inverse operator (⌃ + I) ⌦ I + I ⌦ (⌃ + I) is not non-decreasing. Indeed, 4 is not a total order on S n so we may have that an operator is non-decreasing and its inverse is not.

Lemma 17. For all ⇢ 2 (0, 1) and ! 2 [ ⇡/2; ⇡/2] and r ± = ⇢(cos(!) ± p 1 sin(!)) we have:

1 r + r ⇢ k |A 1 | |1 r + |  min{1 + ⇢ + e 1 + 4⇢ k , 2 + ⇢ + p 5⇢ k+1 }  6.
(3.31)

Proof. We note that ⇢ k i A 1 is a real number as is is a quotient of pure complex numbers, which come from the difference between a complex and its conjugate. We first write A 1 as a combination of sine and cosine functions:

⇢ k i A 1 = (r i ) k+1 (1 r + i ) 2 (r + i ) k+1 (1 r i ) 2 r i r + i = (r i ) k+1 (r + i ) k+1 2r i r + i ((r i ) k (r + i ) k ) + (r i r + i )(r i ) k 1 (r + i ) k 1 ) ⇢ i sin ! i = ⇢ k+1 i sin((k + 1)! i ) 2⇢ k+2 i sin(k! i ) + ⇢ k+3 i sin((k 1)! i ) ⇢ i sin ! i .
This quantity can be simplified when ⇢ ! 1 or ! ! 0. We thus modify the expression of A 1 to make these dependencies clearer:

A 1 = sin((k + 1)! i ) 2⇢ i sin(k! i ) + ⇢ 2 i sin((k 1)! i ) sin ! i = (cos(!) ⇢)(sin(k!) ⇢ sin((k 1)!))+cos(k!) sin(!) ⇢ cos((k 1)!) sin(!) sin ! i developing sin(a + b) = sin
= (cos(!) ⇢) 2 sin((k 1)!) + (cos(!) ⇢) sin(!) cos((k 1)!) sin ! i + cos(k!) sin(!) ⇢ cos((k 1)!) sin(!) sin ! i = (cos(!) ⇢) 2 sin((k 1)!) sin ! i +(cos(!) ⇢) cos((k 1)!)+cos(k!) ⇢ cos((k 1)!) simplifying expression, then developing the cosine, = (cos(!) ⇢) 2 sin((k 1)!) sin ! i +2(cos(!) ⇢) cos((k 1)!)+sin(!) sin((k 1)!). (3.32)
So that in that final expression all the terms behave relatively simply when ⇢ ! 1 or ! ! 0. We want to upper bound:

1 r + r ⇢ k |A 1 | |1 r + | .
We thus consider separately the first and second term.

1 r + i r i |1 r + i | = 1 ⇢ 2 |1 r + i |  1 + ⇢ (exact if ! = 0).
Then, using Equation (3.32):

⇢ k i |A 1 | |1 r + i | = ⇢ k (cos(!) ⇢) 2 sin((k 1)!) sin ! i +2(cos(!) ⇢) cos((k 1)!) + sin(!) sin((k 1)!) p (1 ⇢ cos !) 2 + ⇢ 2 sin 2 (!)
.

Considering separately the three terms in the numerator, using numerous times that for any a, b 2 [0; 1], |a b|  1 ab:

⇢ k (cos(!) ⇢) 2 sin((k 1)!) sin ! i p (1 ⇢ cos !) 2 + ⇢ 2 sin 2 (!)  ⇢ k (cos(!) ⇢) sin((k 1)!) sin ! i as |(cos(!) ⇢)|  1 ⇢ cos(!),  ⇢ k (cos(!) 1) sin((k 1)!) sin ! i +⇢ k (1 ⇢) sin((k 1)!) sin ! i writing cos(!) ⇢ = cos(!) 1 + 1 ⇢  ⇢ k (1 ⇢)(k 1) + ⇢ k (cos(!) 1) sin((k 1)!) sin ! i as | sin((k 1)!)|  |(k 1) sin(!)|,  ⇢ k (1 ⇢)k (1 ⇢)⇢ k +⇢ k (cos(!) 1) sin((k 1)!) sin ! i writing cos(!) 1 = 2 sin 2 (!/2),  ⇢ k (1 + (1 ⇢)) k ⇢ k (1 ⇢)⇢ k + ⇢ k 2 sin 2 (!/2) sin ! i using 1 + (1 ⇢)k  (1 + (1 ⇢)) k ,  ⇢ k (1 + (1 ⇢)) k ⇢ k (1 ⇢)⇢ k + ⇢ k tan(!/2) and as tan(!/2)  1 for |!|  ⇡/2,  1 (1 ⇢)⇢ k using ⇢ k (1 + (1 ⇢)) k = (1 (1 ⇢) 2 ) k  1,
And for the second and third term:

2 ⇢ k (cos(!) ⇢) cos((k 1)!) p (1 ⇢ cos !) 2 + ⇢ 2 sin 2 (!)  2⇢ k , ⇢ k + sin(!) sin((k 1)!) p (1 ⇢ cos !) 2 + ⇢ 2 sin 2 (!)  ⇢ k .
Thus:

1 r + i r i ⇢ k i |A 1 | |1 r + i |  1 + ⇢ + 1 + 3⇢ k .
We also have

| ⇢ k (cos(!) ⇢) 2 sin((k 1)!) sin ! i p (1 ⇢ cos !) 2 +⇢ 2 sin 2 (!) |  ⇢ k (cos(!) ⇢) sin((k 1)!) sin ! i  ⇢ k (1 ⇢)(k 1) + ⇢ k (cos(!) 1) sin((k 1)!) sin ! i  (1 1 k + 1 ) k+1 (1 ⇢)⇢ k +⇢ k (cos(!) 1) sin((k 1)!) sin ! i  e 1 (1 ⇢)⇢ k + ⇢ k sin 2 (!/2) sin ! i . Using that k sup x2[0;1] x k (1 x) = k 1 k + 1 (1 1 k + 1 ) k = (1 1 k + 1 ) k+1
= exp((k + 1) ln((1

1 k + 1
))  e 1 , (3.33) we get

1 r + i r i ⇢ k i |A 1 | |1 r + i |  1 + ⇢ + e 1 + 4⇢ k We can also change 3⇢ k into p 5⇢ k We have used that |(⇢ cos(!))|  (1 ⇢ cos(!)).
Lemma 18. For any ⇢ i 2 (0; 1), for any

! i 2 [ ⇡/2; ⇡/2] ⇢ j i sin(! i (j + 1)) sin(! i ) ⇢ j+1 i sin(! i j) sin(! i )  1 + e 1 .
Proof.

⇢ j i sin(! i (j + 1)) sin(! i ) ⇢ j+1 i sin(! i j) sin(! i ) = ⇢ j i ✓ sin(! i (j + 1)) ⇢ i sin(! i j) sin(! i ) ◆ = ⇢ j i ✓ (cos(! i ) ⇢ i ) sin(! i j) sin(! i ) + cos(j! i ) ◆  ⇢ j i ((1 ⇢ i )j + 1)  1 + e 1 using (3.33).
Lemma 19. For all ⇢ 2 (0, 1) and ! 2 [ ⇡/2; ⇡/2] and r ± = ⇢(cos(!) ± p 1 sin(!)) we have:

⇢ k i B 1,k  1.75 (3.34)
Proof. Once again, as the considered quantity is real, we first express it as a combination of sine and cosine functions. We then use some simple trigonometric trics to upper bound the quantity.

⇢ k i B 1,k = (r i ) k+1 (1 r + i ) (r + i ) k+1 (1 r i ) r + i r i = 2Im ⇥ (r i ) k+1 (1 r + i )
⇤ p i as it is the difference between a complex 115 and its conjugate,

= Im ⇥ ⇢ k i e (k+1)i! i (1 ⇢ i cos(! i ) i⇢ i sin(! i )) ⇤ sin ! i ⇢ i developing the product, = ⇢ k i cos((k + 1)! i ) sin(! i )⇢ i + sin((k + 1)! i )(1 ⇢ i cos(! i )) sin ! i ⇢ i = ⇢ k i h ⇢ i cos((k + 1)! i ) + (1 ⇢ i cos(! i )) sin((k + 1)! i ) sin ! i i and simplifying.
Let's turn our interest to the second part of the quantity. We denote by

⇤ = ⇢ k i (1 ⇢ i cos(! i )) sin((k + 1)! i ) sin ! i , and
⇤ = ⇢ k i (1 ⇢ i + ⇢ i (1 cos(! i ))) sin((k + 1)! i ) sin ! i introducing an artificial + ⇢ i ⇢ i ,  ⇢ k i (1 ⇢ i ) sin((k + 1)! i ) sin ! i + ⇢ k i ⇢ i (1 cos(! i )) sin((k + 1)! i ) sin ! i by triangular inequality,  ⇢ k i (1 ⇢ i )(k + 1) + ⇢ k i ⇢ i sin 2 ( ! 2 ) 1 2 cos( ! 2 ) sin( ! 2 ) using 1 cos(! i ) = 2 sin 2 ( ! 2 )  ⇢ k i (1 ⇢ i )k + ⇢ k i (1 ⇢) + ⇢ k i ⇢ i sin 2 ( ! 2 ) 1 2 cos( ! 2 ) sin( ! 2 )  (1 (1 ⇢ i )) k (1 + (1 ⇢ i )) k ⇢ k i + 1 2(k + 1) + ⇢ k i ⇢ i 2 tan( ! 2 )  (1 (1 ⇢ i ) 2 ) k + 1 4 + 1 2  1 + 1 4 + 1 2 ⇢ k i .
Thus

⇢ k i B 1,k = ⇢ k i + 1 + 1 4 + 1 2
⇢ k i  1 + 1 4 + 1 2 = 1.75.
Lemma 20. For any s i , , 2 R 3 + such that (s i + )  1, for any k 2 N, we have the two following highly related identities:

0  2 p (s i + ) 2 + (k 1) p (s i + ) (1 p (s i + )) k  2 0  1 (1 + k p (s i + ))(1 p (s i + )) k  1.
Proof. Proof relies on the trick, for any ↵ 2 R, n 2 N: 1 + n↵  (1 + ↵) n . For the first one:

p (s i + ) + 2 + (k 1) p (s i + ) (1 p (s i + )) k = = p (s i + ) + (1 p (s i + )) k + 1 + (k 1) p (s i + ) (1 p (s i + )) k  p (s i + ) + (1 p (s i + )) + 1 + (k 1) p (s i + ) (1 p (s i + )) k 1  1 + (1 (s i + )) k 1  2.
For the second one:

0  (1 + k p (s i + ))(1 p (s i + )) k  (1 (s i + )) k  1. 117

Introduction

Many learning problems may be cast as the optimization of an objective function defined as an expectation of random functions, and which can be accessed only through samples. In this chapter, we consider composite problems of the form min

✓2R d E z `(z, ✓) + g(✓), (4.1)
where for any z, `(z, •) is a convex quadratic function (plus some linear terms) and g is any extended-value convex function.

In a machine learning context, `(z, ✓) is the loss occurred for the observation z and the predictor parameterized by ✓, f (✓) = E z `(z, ✓) is its generalization error, while the function g represents some additional regularization or constraints on the predictor. Thus in this chapter we consider composite least-squares regression problems, noting that solving such problems effectively leads to efficient algorithms for all smooth losses by using an online Newton algorithm [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF], with the same running-time complexity of O(d) per iteration for linear predictions.

When g = 0, averaged stochastic gradient descent with a constant step-size achieves the optimal convergence rate of O(1/n) after n observations, even in illconditioned settings without strong convexity [START_REF] Jain | Parallelizing stochastic approximation through mini-batching and tail-averaging[END_REF], with precise nonasymptotic results that depend on the statistical noise variance 2 of the least-squares problem, as 2 d/n, and on the squared Euclidean distance between the initial predictor ✓ 0 and the optimal predictor ✓ ⇤ , as k✓ 0 ✓ ⇤ k 2 2 /n. In this chapter, we extend this O(1/n) convergence result in two different ways: -Composite problems: we provide a new algorithm that deals with composite problems where g is (essentially) any extended-value convex function, such as the indicator function of a convex set for constrained optimization, or a norm or squared norm for additional regularization. This situation is common in many applications in machine learning and signal processing [see, e.g., Rish and Grabarnik, 2014, and references therein]. Because we consider large steps-sizes (that allow robustness to ill-conditioning), the new algorithm is not simply a proximal extension; for example, in the constrained case, averaged projected stochastic gradient descent with a constant step-size is not convergent, even for quadratic functions. -Beyond Euclidean geometry: Following mirror descent [START_REF] Nemirovski | Effective methods for the solution of convex programming problems of large dimensions[END_REF]] and recent work of [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF], our new algorithm can take into account a geometry obtained with a Bregman divergence D h associated with a convex function h, which can typically be the squared Euclidean norm (leading to regular stochastic gradient descent in the non-composite case), the entropy function, or the squared `p-norm. This will allow convergence rates proportional to D h (✓ ⇤ , ✓ 0 )/n, which may be significantly smaller than k✓ 0 ✓ ⇤ k 2 /n in many situations. In order to obtain these two extensions, we consider the stochastic dual averaging algorithm of [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF] and [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] which we present in Section 4.2, and study under the particular set-up of constant step-size with averaging, showing in Section 4.3 that it also achieves a convergence rate of O(1/n) even without strong convexity. This is achieved by a new proof technique that relates stochastic and deterministic recursions.

Given that known lower-bounds for this class of problems are proportional to 1/ p n for function values, we established our O(1/n) results with a different criterion, namely the Mahalanobis distance associated with the Hessian of the least-squares problem. In our simulations in Section 4.5, the two criteria behave similarly. Finally, in Section 4.4, we shed additional insights of the relationships between mirror descent and dual averaging, in particular in terms of continuous-time interpretations.

Dual Averaging Algorithm

In this section, we introduce dual averaging as well as related frameworks, together with new results in the deterministic case.

with h n = h + n g and 2 (0, 1) (commonly referred to as the step-size in optimization or the learning rate in machine learning). We note that equivalently

✓ n 2 arg max ✓2R d {h⌘ n , ✓i h n (✓)}. When h = 1 2 k • k 2 2
and g = 0, we recover gradient descent.

Two iterates (⌘ n , ✓ n ) are updated in DA. The dual iterate ⌘ n is simply proportional to the sum of the gradients evaluated in the primal iterates (✓ n ). The update of the primal iterate ✓ n is more complex and raises two different issues: its existence and its tractability. We discuss the first point in Appendix 4.A and assume, as of now, that the method is generally well defined in practice. The tractability of ✓ n is essential and the algorithm is only used in practice if the functions h and g are simple in the sense that the gradient rh ⇤ n may be computed effectively. This is the case if there exists a closed form expression. Usual examples are given in Appendix 4.I.

Euclidean case and proximal operators. In the Euclidean case, Eq. ( 4.3) may be written in term of the proximal operator defined by [START_REF] Moreau | Fonctions convexes duales et points proximaux dans un espace Hilbertien[END_REF] as Prox g (⌘) = arg min ✓2X {1 2 k✓ ⌘k 2 2 + g(✓)}:

✓ n = arg min ✓2X n h ⌘ n , ✓i+n g(✓)+ 1 2 k✓k 2 2 o = arg min ✓2X n 1 2 k✓ ⌘ n k 2 2 +n g(✓) o = Prox ng (⌘ n ).
DA is in this sense related to proximal gradient methods, also called forward-backward splitting methods [see, e.g., [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Wright | Sparse reconstruction by separable approximation[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF]. These methods are tailored to composite optimization problems: at each iteration f is linearized around the current iterate ✓ n and they consider the following update

✓ n+1 = arg min ✓2X n h rf (✓ n ), ✓i + g(✓) + 1 2 k✓ ✓ n k 2 2 o = Prox g (✓ n rf (✓ n )).
Note the difference with DA which considers a dual iterate and a proximal operator for the function n g instead of g (see additional insights in Section 4.4).

From non-smooth to smooth optimization. DA was initially introduced by [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF] to optimize a non-smooth function f with possibly convex constraints (g = 0 or g = C ). It was extended to the general stochastic composite case by [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] who defined the iteration as

✓ n = arg min ✓2X n 1 n n 1 X i=0 hz i , ✓i + g(✓) + n n h(✓) o ,
where z i is an unbiased estimate 1 of a subgradient in @f (✓ i ) and ( n ) n 1 a nonnegative and nondecreasing sequence of real numbers. This formulation is equivalent to Eq. ( 4.3) for constant sequences n = 1/ . [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] proved convergence rates of order O(1/ p n) for convex problems with decreasing step-size C/ p n and O(1/(µn))

for problems with µ-strongly convex regularization with constant step-size 1/µ. DA was also studied with decreasing step-sizes in the distributed case by Duchi et al.

[2012], [START_REF] Dekel | Optimal distributed online prediction using mini-batches[END_REF], [START_REF] Colin | Gossip dual averaging for decentralized optimization of pairwise functions[END_REF] and combined with the alternating direction method of multipliers (ADMM) by [START_REF] Suzuki | Dual averaging and proximal gradient descent for online alternating direction multiplier method[END_REF]. It was further shown to be very efficient in manifold identification by [START_REF] Lee | Manifold identification in dual averaging for regularized stochastic online learning[END_REF] and [START_REF] Duchi | Local asymptotics for some stochastic optimization problems: optimality, constraint identification, and dual averaging[END_REF].

Relationship with mirror descent. The DA method should be associated with its cousin mirror descent algorithm (referred to from now on as "MD"), introduced by Nemirovski and Yudin [1979] for the constrained case and written under its modern proximal form by [START_REF] Beck | Mirror descent and nonlinear projected subgradient methods for convex optimization[END_REF] ✓ n = arg min

✓2X hrf (✓ n 1 ), ✓i + D h (✓, ✓ n 1 ) ,
where we denote by D h (↵, ) = h(↵) h( ) hrh( ), ↵ i the Bregman divergence associated with h. Moreover it was later extended to the general composite case by [START_REF] Duchi | Composite objective mirror descent[END_REF] ✓

n = arg min ✓2X hrf (✓ n 1 ), ✓i + g(✓) + D h (✓, ✓ n 1 ) . (4.4)
DA was initially motivated by [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF] to avoid new gradients to be taken into account with less weight than previous ones. However, as an extension of the Euclidean case, DA essentially differs from MD on the way the regularization component is dealt with. See more comparisons in Section 4.4.

Relationship with online learning. DA was traditionally studied under the online learning setting [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF] of regret minimization and is related to the "follow the leader" approach [see, e.g., [START_REF] Kalai | Efficient algorithms for online decision problems[END_REF] as noted by [START_REF] Mcmahan | Follow-the-regularized-leader and mirror descent: Equivalence theorems and l1 regularization[END_REF]. More generally, the DA method may be cast in the primal-dual algorithmic framework of Shalev-Shwartz and Singer [2006] and Shalev-Shwartz and Kakade [2009].

Deterministic Convergence Result for Dual Averaging

In this section we present the convergence properties of the DA method for optimizing deterministic composite problems of the form in Eq. ( 4.2), for any smooth function f (see proof in Appendix 4.B).

Proposition 8. Assume (A1-4). For any step-size such that h f is convex on

X we have for all ✓ 2 X (✓ n ) (✓)  D h (✓, ✓ 0 ) (n + 1)
.

Moreover assume g = 0, and there exists µ 2 R such that f µh is also convex on X

123 then we have for all ✓ 2 X f (✓ n ) f (✓)  (1 µ) n D h (✓, ✓ 0 ) .
We can make the following remarks:

-We adapt the proofs of [START_REF] Chen | Convergence analysis of a proximal-like minimization algorithm using Bregman functions[END_REF], [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF] to the composite case and the DA method by including the regularization component g in the Bregman divergence. If g was differentiable we would simply use D hn = D h+n g and prove the following recursion:

D hn (✓ ⇤ , ✓ n ) D h n 1 (✓ ⇤ , ✓ n 1 ) = D h n 1 (✓ n , ✓ n 1 ) + hrf (✓ n 1 ), ✓ n 1 ✓ n i (g(✓ n ) g(✓)) hrf (✓ n 1 ), ✓ n 1 ✓ ⇤ i.
Since g is not differentiable, we extend instead the notion of Bregman divergence to the non-smooth case in Appendix 4.B.2 and show the proof works in the same way.

-Related work: A result on MD with analogue assumptions was first presented by [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF]. DA was analyzed for smooth functions in the noncomposite case where g = 0, by [START_REF] Dekel | Optimal distributed online prediction using mini-batches[END_REF] in the stochastic setting and by [START_REF] Lu | Relatively-smooth convex optimization by firstorder methods, and applications[END_REF] in the deterministic setting. The technique to extend the Bregman divergence to analyze the regularization component has its roots in the time-varying potential method in online learning [Cesa-Bianchi and Lugosi, 2006, Chapter 11.6] and the "follow the regularized leader" approach [START_REF] Abernethy | Competing in the dark: An efficient algorithm for bandit linear optimization[END_REF].

-This convergence rate is suboptimal for the class of addressed problems. Indeed accelerated gradient methods achieve the convergence rate of O(L/n 2 ) in the composite setting [START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF], such a rate being optimal for optimizing smooth functions among first-order techniques that can access only sequences of gradients [START_REF] Nesterov | Introductory Lectures on Convex Optimization[END_REF].

-Classical results on the convergence of optimization algorithms in non-Euclidean geometries assume on one hand that the function h is strongly convex and on the other hand the function f is Lipschitz or smooth. Following Bauschke et al.

[2016], we consider a different assumption which combines the smoothness of f and the strong convexity of h on the single condition h f convex. For the Euclidean geometry where h(✓) = 1 2 k✓k 2 2 , this condition is obviously equivalent to the smoothness of the function f with regards to the `2-norm. Moreover, under arbitrary norm k • k, this is also equivalent to assuming h µ-strongly convex and f L-smooth (with respect to this norm). However it is much more general and may hold even when f is non-smooth, which precisely justifies the introduction of this condition [see examples described by [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF].

-The bound adapts to the geometry of the function h through the Bregman divergence between the starting point ✓ 0 and the solution ✓ ⇤ and the step-size which is controlled by h. Therefore the choice of h influences the constant in the bound. Examples are provided in Appendix 4.I.

Stochastic Convergence Results for Quadratic Functions

In this section, we consider a symmetric positive semi-definite matrix ⌃ 2 R d⇥d and a convex quadratic function f defined as (A5) f (✓) = 1 2 h✓, ⌃✓i hq, ✓i, with q 2 R d in the column space of ⌃, so that f has a global minimizer ✓ ⌃ 2 R d . Without loss of generality 2 , ⌃ is assumed invertible, though its eigenvalues could be arbitrarily small. The global solution is known to be ✓ ⌃ = ⌃ 1 q, but the inverse of the Hessian is often too expensive to compute when d is large. The function may be simply expressed as

f (✓ n ) = 1 2 h✓ n ✓ ⌃ , ⌃(✓ n ✓ ⌃ )i + f (✓ ⌃ )
and the excess of the cost function = f + g as

(✓ n ) (✓ ⇤ ) = h✓ ⇤ ✓ ⌃ , ⌃(✓ n ✓ ⇤ )i + g(✓ n ) g(✓ ⇤ ) (linear part) + 1 2 h✓ n ✓ ⇤ , ⌃(✓ n ✓ ⇤ )i (quadratic part) .
The first-order condition of the optimization problem in Eq. ( 4.2) is 0 2 rf (✓ ⇤ ) + @g(✓ ⇤ ) + @ X (✓ ⇤ ). By convexity of g, we have g(✓ n ) g(✓ ⇤ ) hz, ✓ n ✓ ⇤ i for any z 2 @g(✓ ⇤ ). Moreover X (✓ n )

X (✓ ⇤ ) = 0 hz, ✓ n ✓ ⇤ i for any z 2 @ X (✓ ⇤ ) since ✓ n , ✓ ⇤ 2 X by definition. Therefore this implies that the linear part g(✓ n ) g(✓ ⇤ ) + hrf (✓ ⇤ ), ✓ n ✓ ⇤ )i is non-negative and we have the bound

1 2 k✓ n ✓ ⇤ k 2 ⌃  (✓ n ) (✓ ⇤ ). (4.5)
We derive, in this section, convergence results in terms of the distance k✓ n ✓ ⇤ k ⌃ which takes into account the ill-conditioning of the matrix ⌃ and is a lower bound in the excess of function values. Furthermore it directly implies classical results for strongly convex problems. In many practical situations, the gradient of f is not available for the recursion in Eq. ( 4.3), and we have only access to an unbiased estimate rf n+1 (✓ n ) of the gradient of f at ✓ n . We consider in this case the stochastic dual averaging method (referred to from now on as "SDA") defined the same way as DA as

⌘ n = ⌘ n 1 rf n (✓ n 1 ) ✓ n = rh ⇤ n (⌘ n ), (4.6)
for ✓ 0 2 dom h and ⌘ 0 = rh(✓ 0 ). Here we consider the stochastic approximation 2. By decomposing ✓ in ✓ = ✓ k + ✓ ? with ✓ ? 2 Null(⌃) and h✓ ? , ✓ k i = 0 and considering

(✓) = f (✓ k ) + g(✓ k ) where g(✓ k ) = inf ✓ ? 2Null(⌃) g(✓ ? + ✓ k ).
framework [START_REF] Kushner | Stochastic Approximation and Recursive Algorithms and Applications[END_REF]. That is, we let (F n ) n 0 be an increasing family of -fields such that for each ✓ 2 R d and for all n 1 the random variable rf n (✓) is square-integrable and F n -measurable with E[rf n (✓)|F n 1 ] = rf (✓). This includes (but also extends) the usual machine learning situation where rf n is the gradient of the loss associated with the n-th independent observation. We will consider in the following two different gradient oracles.

Additive Noise

We study here the convergence of the SDA recursion defined in Eq. ( 4.6) under an additive noise model:

(A6) For all n 1, rf n (✓) = rf (✓) ⇠ n , where the noise (⇠ n ) n 1 is a square- integrable martingale difference sequence (i.e., E[⇠ n |F n 1 ] = 0) with bounded covariance E[⇠ n ⌦ ⇠ n ] 4 C.
With this oracle and for the quadratic function f , SDA takes the form

⌘ n = ⌘ n 1 (⌃✓ n 1 q) + ⇠ n ✓ n = rh ⇤ n (⌘ n ). (4.7)
We obtain the following convergence result on the average ✓n = 1 n P n 1 k=0 ✓ k which is an extension of results from [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] to non-Euclidean geometries and to composite settings (see proof in Appendix 4.C).

Proposition 9. Assume (A2-6). Consider the recursion in Eq. ( 4.7) for any constant step-size such that h f is convex. Then

1 2 Ek ✓n ✓ ⇤ k 2 ⌃  2 min ( D h (✓ ⇤ , ✓ 0 ) n ; krh(✓ 0 ) rh(✓ ⇤ )k 2 ⌃ 1 ( n) 2 ) + 4 n tr ⌃ 1 C.
We can make the following observations:

-The proof in the Euclidean case [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] highly uses the equality

✓ n ✓ ⌃ = (I ⌃)(✓ n 1 ✓ ⌃ )
which is no longer available in the non-Euclidean or proximal cases. Instead we adapt the classic proof of convergence of averaged SGD of [START_REF] Polyak | Acceleration of stochastic approximation by averaging[END_REF] which rests upon the expansion

P n k=0 rf k+1 (✓ k ) = P n k=0 (⌘ k ⌘ k+1 )/ = (⌘ 0 ⌘ n+1 )/ .
The crux of the proof is then to consider the difference between the iterations with and without noise, ⌘ sto n ⌘ det n , which happens to satisfy a similar recursion as Eq. ( 4.7) but started from the solution ✓ ⇤ . The quadratic nature of f is used twice: (a) to bound

k⌘ sto n ⌘ det n k ⌃ 1 ⇠ p n, and (b) to expand rf ( ✓n ) = rf (✓ n ) ⇠ ⌘ sto n ⌘ 0 n + 1/ p n.
-As for Proposition 8, the constraint on the step-size depends on the function h. Moreover the step-size is constant, contrary to previous works on SDA [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] which prove results for decreasing step-size n = C/ p n for the convex case (and with a convergence rate of only O(1/ p n)). -The first term is the "bias" term. It only depends on the "distance" from the initial point ✓ 0 to the solution ✓ ⇤ as the minimum of two terms. The first one recovers the deterministic bound of Proposition 8. The second one, specific to quadratic objectives, leads to an accelerated rate of O(1/n 2 ) for some good starting points such that krh(✓ 0 ) rh(✓ ⇤ )k 2 ⌃ 1 < 1, thus extending the result from Chapter 2.

-The second term is the "variance" term which depends on the noise in the gradients. When the noise is structured (such as for least-squares regression), i.e, there exists > 0 such that C 4 2 ⌃, the variance term becomes

2 d
n which is optimal over all estimators in R d without regularization [START_REF] Tsybakov | Optimal rates of aggregation[END_REF]. However the regularization g does not bring statistical improvement as possible, for instance, with `1-regularization. We believe this is due to our proof technique. Indeed, in the case of linear constraints, [START_REF] Duchi | Local asymptotics for some stochastic optimization problems: optimality, constraint identification, and dual averaging[END_REF] recently showed that the primal iterates (✓ n ) follow a central limit theorem (CLT), namely p n ✓n is asymptotically normal with a covariance precisely restricted to the active constraints. This supports that SDA may leverage the regularization (the active constraints in their case) to get better statistical performance. We leave such non-asymptotic results to future work. Assumption (A6) on the gradient noise is quite general, since the noise (⇠ n ) is allowed to be a martingale difference sequence (correct conditional expectation given the past, but not necessarily independence from the past). However it is not verified by the oracle corresponding to regular SDA for least-squares regression, where the noise combines both an additive and a multiplicative part, and its covariance is then no longer bounded in general (it will be for g the indicator function of a bounded set).

Least-Squares Regression

We consider now the least-squares regression framework, i.e, risk minimization with the square loss. Following [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF], we assume that:

(A7) The observations (x n , y n ) 2 R d ⇥ R,
(✓) = 1 2 E(hx n , ✓i y n ) 2 . (A9)
We denote by ⌃ = E[x n ⌦ x n ] the population covariance matrix, which is the Hessian of f at all points. Without loss of generality, we reduce R d to the minimal subspace where all x n , n 1, lie almost surely. Therefore ⌃ is invertible and all the eigenvalues of ⌃ are strictly positive, even if they may be arbitrarily small.

(A10) We denote the residual by ⇠

n = (y n h✓ ⇤ , x n i)x n . We have E[⇠ n ] = 0 but E[⇠ n |x n ] 6 = 0 in general (unless the model is well-specified). There exists > 0 such that E[⇠ n ⌦ ⇠ n ] 4 2 ⌃. (A11) There exists  > 0 such that for all z 2 R d , Ehz, x n i 4  hz, ⌃zi 2 .
(A12) The function g is lower bounded by some constant which is assumed by sake of simplicity to be 0.

(A13) There exists L > 0 such that Lh 1 2 k • k 2 ⌃ is convex. Assumptions (A7-9) are standard for least-squares regression, while Assumption (A10) defines a bounded statistical noise. Assumption (A11) is commonly used in the analysis of least-mean-square algorithms [START_REF] Macchi | Adaptive Processing: the Least-Mean-Squares Approach with Applications in Transmission[END_REF] and says the projection of the covariates x n on any direction z 2 R d have a bounded kurtosis. It is true for Gaussian vectors with  = 3. Assumption (A13) links up the geometry of the function h and the objective function f ; for example for `p-geometries, L is proportional to Ekxk 2 q where 1/p + 1/q = 1 (see Corollary 7 in Appendix 4.I). For the least-squares regression problem, the SDA algorithm defined in Eq. (4.6) takes the form:

⌘ n = ⌘ n 1 hx n , ✓ n 1 i y n x n ✓ n = rh ⇤ n (⌘ n ). (4.8)
This corresponds to a stochastic oracle of the form

rf n (✓) = (⌃ + ⇣ n )(✓ ✓ ⌃ ) ⇠ n for ✓ 2 R d , with ⇣ n = x n ⌦x n ⌃.
This oracle combines an additive noise ⇠ n satisfying the previous Assumption (A6) and a multiplicative noise ⇣ n which is harder to analyze.

We obtain a similar result compared to Proposition 9 at the cost of additional corrective terms.

Proposition 10. Assume (A2-4) and (A7-13). Consider the recursion in Eq. ( 4.8) for any constant step-size such that  1 4Ld . Then

1 2 Ek ✓n ✓ ⇤ k 2 ⌃  2 D h (✓ ⇤ , ✓ 0 ) n + 32d n 2 + k✓ ⇤ ✓ ⌃ k 2 ⌃ + 16d n 2 ✓ 5D h (✓ ⇤ , ✓ 0 ) + g(✓ 0 ) ◆ .
We can make the following remarks:

-The proof technique is similar to the one of Proposition 9. Nevertheless its complexity comes from the extra multiplicative noise ⇣ n in the gradient estimate (see Appendix 4.D). -The result is only proven for  1/(4Ld) which seems to be a proof artifact. Indeed we empirically observed (see Section 4.5) that the iterates still converge to the solution for all 6 1/(2Ekx n k 2 2 ). -The global bound leads to a rate of O(1/n) without strong convexity, which is optimal for stochastic approximation, even with strong convexity [START_REF] Nemirovsky | Problem Complexity and Method Efficiency in Optimization[END_REF]. We recover the terms of Proposition 9 pertubed by: (a) one corrective term of order O(d/n) which depends on the distance between the solution ✓ ⇤ and the global minimizer ✓ ⌃ of the quadratic function f , which corresponds to the covariance of the multiplicative noise at the optimum, and (b) two residual terms of order O(d/n 2

). It would be interesting to study whether these two terms can be removed.

-As in Proposition 9, the bias is also O 1 tr ⌃). Moreover the result holds then for any step-size 6 1/L, which is bigger than allowed for g = 0 [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF].

( n) 2 krh(✓ 0 ) rh(✓ ⇤ )k

Convergence Results on the Objective Function

In this section we present the convergence properties of the SDA method on the objective function = f + g rather than on the norm k • k ⌃ . We first start with a disclaimer: it is not possible to obtain general non-asymptotic results on the convergence of the SDA iterates in term of function values without additional assumptions on the regularization g. We indeed show in Appendix 4.E that, even in the simple case of a linear function f (✓) = ha, ✓i, for a 2 R d , we can always find, for any finite time horizon N , a quadratic non-strongly convex regularization function g N such that for any unstructured noise of variance 2 , the function value

N (✓) = f (✓) + g N (✓) evaluated in the SDA iterates at time N is lowerbounded by N ( ✓N ) N (✓ ⇤ ) 2 12
.

This lower bound is specific to the SDA algorithm and we underline that the regularization g N depends on the horizon N . However this result still prevents the possibility of a universal non-asymptotic convergence result on the function value for the SDA iterates for general quadratic and linear functions. We note that this does not apply to the setting of Proposition 9 and Proposition 10 since ⌃ = 0 for a linear function and the vector q defining the linear term hq, ✓i cannot be in the column space of ⌃, thus violating Assumption (A5). We conjecture that in the setting of Assumption (A5), the lower bound is O(1/ p n) as well. We now provide some specific examples for which we can prove convergence in function values.

Quadratic objectives with smooth regularization. When there exists a constant L g 0 such that L g f g is convex on X then results from Propositions 9 and 10 directly imply convergence of the composite objective to the optimum through

( ✓n ) (✓ ⇤ )  (L g + 1) 2 k ✓n ✓ ⇤ k 2 ⌃ = O(1/n),
with precise constants from Propositions 9 and 10. Indeed we have in that case (L g + 1)f convex and this would be directly implied by Proposition 11 in Appendix 4.B. An easy but still interesting application is the non-regularized case (g = 0) when the optimum

✓ ⇤ is the global optimum ✓ ⌃ of f , because then (✓) (✓ ⇤ ) = 1 2 k✓ ✓ ⇤ k 2 ⌃ .
Thus this extends previous results on function values [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] to non-Euclidean geometries.

f ( ✓n ) f (✓ ⇤ ) = hrf (✓ ⇤ ), ✓n ✓ ⇤ i + 1 2 k ✓n ✓ ⇤ k 2 ⌃  k✓ ⇤ ✓ ⌃ k 2 k ✓n ✓ ⇤ k ⌃ + 1 2 k ✓n ✓ ⇤ k 2 ⌃ = O ⇣ k✓ ⇤ ✓ ⌃ k 2 p n ⌘ ,
with precise constants from Propositions 9 and 10. Hence we obtain a global rate of order O(1/ p n) for the convergence of the function value in the constrained case. These rates may be accelerated to O(1/n) for certain specific convex constraints or when the global optimum ✓ ⌃ 2 C; [START_REF] Duchi | Local asymptotics for some stochastic optimization problems: optimality, constraint identification, and dual averaging[END_REF] recently obtained asymptotic convergence results for the iterates in the cases of linear and `2-ball constraints for linear objective functions. Their results can be directly extended to asymptotic convergence of function values and very probably to all strongly convex sets [see, e.g., [START_REF] Vial | Strong and weak convexity of sets and functions[END_REF]]. However, even for the simple `2-ball constrained problem, we were not able to derive non-asymptotic convergence rates for function values.

However the global rate of order O(1/ p n) is statically non-improvable in general. In Appendix 4.F, we relate the stochastic convex optimization problem [START_REF] Agarwal | Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization[END_REF] to the statistical problem of convex aggregation of estimators [START_REF] Tsybakov | Optimal rates of aggregation[END_REF][START_REF] Lecué | Optimal oracle inequality for aggregation of classifiers under low noise condition[END_REF]]. These authors showed lower bounds on the performance of such estimators which provide us lower bounds on the performance of any stochastic algorithm to solve constrained problems. In Proposition 14 and Proposition 16 of Appendix 4.F, we derive more precisely lower bound results for linear and quadratic functions for certain ranges of n and d confirming the optimality of the convergence rate O(1/ p n). This being said, in our experiments in Section 4.5, we observed that the convergence of function values follows closely the convergence in the Mahalanobis distance.

Parallel Between Dual Averaging and Mirror Descent

In this section we compare the behaviors of DA and MD algorithms, by highlighting their similarities and differences, in particular in terms of continuous-time interpretation.

Lazy versus Greedy Projection Methods

DA and MD are often described in the online-learning literature as "lazy" and "greedy" projection methods [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF]. Indeed, the difference between these two methods is more apparent in the Euclidean projection case (when g = C and h = 1 2 k • k 2 2 ). MD is then projected gradient descent and may be written under its

C ✓ 0 ⌘ md 1 = ⌘ da 1 ⌘ da 2 ⌘ md 2 ⌘ md 3 ⌘ da 3 ✓ ⇤ ✓ md,da 1 ✓ md 2 ✓ da 2 ✓ da 3 ✓ md 3 Figure 4
-1 -Lazy versus greedy projection. In red: greedy projection. In black: lazy projection.

primal-dual form as:

⌘ md n = ✓ md n 1 g md n with g md n 2 @f (✓ md n 1 ) and ✓ md n = arg min ✓2C k⌘ md n ✓k 2 .
Whereas DA takes the form

⌘ da n = ⌘ da n 1 g da n with g da n 2 @f (✓ da n 1 ) and ✓ da n = arg min ✓2C k⌘ da n ✓k 2 .
Therefore, imagining the subgradients g n are provided by an adversary without the need to compute the primal sequence (✓ n ), no projections are needed to update the dual sequence (⌘ da n ), and this one moves far away in the asymptotic direction of the gradient at the optimum rf (✓ ⇤ ). Furthermore the primal iterate ✓ da n is simply obtained, when required, by projecting back the dual iterate in the constraint set. Conversely, the MD dual iterate ⌘ md n update calls for ✓ md n 1 , and therefore a projection step is unavoidable. Thereby MD iterates (⌘ md n , ✓ md n ) are going, at each iteration, back-and-forth between the boundary and the outside of the convex set C. This is illustrated in Figure 4-1

Strongly Convex Cases

MD converges linearly for smooth and strongly convex functions f , in the absence of a regularization component [START_REF] Lu | Relatively-smooth convex optimization by firstorder methods, and applications[END_REF] or for Euclidean geometries [START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF]. However we were not able to derive faster convergence rates for DA when the function f or the regularization g are strongly convex. Moreover the only results we found in the literature are about (a) an alteration of the dual gradient method [Devolder et al., 2013, Section 4] which is itself a modification of DA with an additional projection step proposed by [START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF] for smooth optimization, (b) the strongly convex regularization g which enables [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] to obtain a O(1/µn) convergence rate in the stochastic case.

At the simplest level, for h = 1 2 k • k 2 2 and f = 0, MD is equivalent to the proximal point algorithm [START_REF] Martinet | Breve communication. Régularisation d'inéquations variationnelles par approximations successives[END_REF]

✓ md n = arg min ✓2R d g(✓) + 1 k✓ ✓ md n 1 k 2 2 , whereas DA, which is not anymore iterative, is such that ✓ da n = arg min ✓2R d g(✓) + 1 n k✓k 2 2 . For the squared `2-regularization g(✓) = ⌫ 2 k✓ ✓ ⇤ k 2 2 , we compute exactly (see Ap- pendix 4.G) g(✓ md n ) g(✓ ⇤ ) = ⇣ 1 ⌫ ⌘ n [g(✓ md 0 ) g(✓ ⇤ )] and g(✓ da n ) g(✓ ⇤ ) = g(✓ da 0 ) g(✓ ⇤ ) (1 + ⌫ n) 2 .
Therefore the convergence of DA can be dramatically slower than MD. However when noise is present, its special structure may be leveraged to get interesting results.

Continuous Time Interpretation of DA et MD

Following [START_REF] Nemirovsky | Problem Complexity and Method Efficiency in Optimization[END_REF], [START_REF] Bolte | Smooth optimization with approximate gradient[END_REF], [START_REF] Krichene | Accelerated mirror descent in continuous and discrete time[END_REF], [START_REF] Wibisono | A variational perspective on accelerated methods in optimization[END_REF] we propose a continuous interpretation of these methods for g twice differentiable. Precise computations are derived in Appendix 4.H.

The MD iteration in Eq. ( 4.4) may be viewed as a forward-backward Euler discretization of the MD ODE [START_REF] Bolte | Smooth optimization with approximate gradient[END_REF]:

✓ = r 2 h(✓) 1 [rf (✓) + rg(✓)]. (4.9)
On the other hand, the ODE associated to DA takes the form

✓ = r 2 (h(✓) + tg(✓)) 1 (rf (✓) + rg(✓)). (4.10)
It is worth noting that these ODEs are very similar, with an additional term tg(✓) in the inverse mapping r 2 (h(✓) + tg(✓)) 1 which may slow down the DA dynamics. In analogy with the discrete case, the Bregman divergences D h and D h+tg are respectively Lyapunov functions for the MD and the DA ODEs [see, e.g., [START_REF] Krichene | Accelerated mirror descent in continuous and discrete time[END_REF] and we notice in Appendix 4.H the continuous time argument really mimics the proof of Proposition 8 without the technicalities associated with discrete time. Moreover we recover the variational interpretation of [START_REF] Krichene | Accelerated mirror descent in continuous and discrete time[END_REF], [START_REF] Wibisono | A variational perspective on accelerated methods in optimization[END_REF], [START_REF] Wilson | A Lyapunov analysis of momentum methods in optimization[END_REF]: the Lyapunov function generates the dynamic in the sense that a function L is first chosen and secondly a dynamics, for which L is a Lyapunov function, is then designed. In this way MD and DA are the two different dynamics associated to the two different Lyapunov functions D h and D h+tg . We also provide in Appendix 4.H a slight extension to the noisy-gradient case.

Experiments

In this section, we illustrate our theoretical results on synthetic examples. We provide additional experiments on a machine learning benchmark in Appendix 4.K. Simplex-constrained least-squares regression with synthetic data. We consider normally distributed inputs x n 2 R d with a covariance matrix ⌃ that has random eigenvectors and eigenvalues 1/k, for k = 1, . . . , d and a random global optimum ✓ ⌃ 2 [0, +1) d . The outputs y n are generated from a linear function with homoscedatic noise with unit signal to noise-ratio ( 2= 1). We denote by R 2 = tr ⌃ the average radius of the data and we show results averaged over 10 replications.

0 2 4 6 log 10 (n) -4 -3 -2 -1 0 log 10 [f( θ)-f( θ * )] SGD C/R 2 SGD C/R 2 n 1/2 SDA C/R 2 SDA C/R 2 n 1/2 0 2 4 6 log 10 (n) -4 -3 -2 -1 0 log 10 || θ-θ * || Σ 2 SGD C/R 2 SGD C/R 2 n 1/2 SDA C/R 2 SDA C/R 2 n 1/2
We consider the problem of least-squares regression constrained on the simplex d of radius r = k✓ ⌃ k 1 /2 , i.e., min ✓2r d E(hx n , ✓i y n ) 2 , for d = 100. We compare the performance of SDA and SGD algorithms with different settings of the stepsize n , constant or proportional to 1/ p n. In the left plot of Figure 4-1 we show the performance on the objective function and on the right plot, we show the performance on the squared Mahalanobis norm k • k 2 ⌃ . All costs are shown in log-scale, normalized so that the first iteration leads to f (✓ 0 ) f (✓ ⇤ ) = 1. We can make the following observations (we only show results on Euclidean geometry since results under the negative entropy geometry were very similar):

-With constant step-size, SDA converges to the solution at rate O(1/n) whereas the SGD algorithm does not converge to the optimal solution.

-With decaying step-size n = 1/(2R 2 p n), SDA and SGD converge first at rate O(1/ p n), then at rate O(1/n), taking finally advantage of the strong convexity of the problem.

-We note (a) there is no empirical difference between the performance on the objective function and the squared distance k • k 2 ⌃ , (b) with decreasing step-size, SGD and SDA behave very similarly.

Conclusion

In this chapter, we proposed and analyzed the first algorithm to achieve a convergence rate of O(1/n) for stochastic composite objectives, without the need for strong convexity. This was achieved by considering a constant step-size and averaging of the primal iterates in the dual averaging method.

Our results only apply to expectations of quadratic functions (but to any additional potentially non-smooth terms). In fact, constant step-size stochastic dual averaging is not convergent for general smooth objectives; however, as done in the noncomposite case by [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF], one could iteratively solved quadratic approximations of the smooth problems with the algorithm we proposed in this chapter to achieve the same rate of O(1/n), still with robustness to ill-conditioning and efficient iterations. Finally, it would be worth considering accelerated extensions to achieve a forgetting of initial conditions in O(1/n 2) as was done for averaged gradient descent in Chapter 3. After having studied the stochastic optimization of quadratic functions, we now shift our focus towards two applications of the quadratic loss in machine learning.

4.B Proof of Convergence of Deterministic DA

We first describe a new notion of smoothness defined by [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF]. Then we present our extension of the Bregman divergence to the non-smooth function g to finally prove Proposition 8.

4.B.1 A Lipschitz-Like/Convexity Condition

Classical results on the convergence of optimization algorithms in non-Euclidean geometry assume on one hand that the function h is strongly convex and on the other hand the function f is Lipschitz or smooth. Following [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF], we consider a different assumption which combines the smoothness of f and the strong convexity of h on a single condition called Lipschitz-like/Convexity Condition by [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF] and denoted by (LC): (LC) There exists a constant L 2 (0, +1) such that Lh f is convex on X . For Euclidean geometry, this condition is obviously equivalent to the smoothness of the function f with regards to the `2-norm. Moreover, under an arbitrary norm k • k, assuming h µ-strongly convex and f L-smooth clearly implies, by simple convex computation, (LC) with constant L/µ. However (LC) is much more general and may hold even when f is non-smooth what precisely justifies the introduction of this condition. Many examples are described by [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF]. Furthermore this notion has the elegance of pairing well with Bregman divergences and leading to more refined proofs as shown in the following proposition which summarizes equivalent properties of (LC).

Proposition 11 [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF]). Assume (A1-4). For L > 0 the following conditions are equivalent: -Lh f is convex on X , i.e., (LC) holds, -D f (↵, )  LD h (↵, ) for all (↵, ) 2 X ⇥ X . Furthermore, when f and h are assumed twice differentiable, then the above is equivalent to r 2 f (✓) 4 Lr 2 h(✓) for all ✓ 2 X .

4.B.2 Generalized Bregman Divergence

The Bregman divergence was defined by [START_REF] Bregman | A relaxation method of finding a common point of convex sets and its application to the solution of problems in convex programming[END_REF] for a differentiable convex function h as

D h (↵, ) = h(↵) h( ) hrh( ), ↵ ), for (↵, ) 2 dom h ⇥ dom h. (4.11)
It behaves as a squared distance depending on the function h and extends the computational properties of the squared `2-norm to non-Euclidean spaces. Indeed most proofs in Euclidean space rest upon the expansion

k✓ n ✓ ⇤ rf (✓ n )k 2 2 = k✓ n ✓ ⇤ k 2 2 + 2 krf (✓ n )k 2 2 2 hrf (✓ n ), ✓ n ✓ ⇤ i
which is not available in non-Euclidean geometry. Therefore the Bregman divergence comes to rescue and is used to compute a deviation between the current iterate of the algorithm and the solution of the problem and, seemingly, used as an non-Euclidean Lyapunov function [START_REF] Chen | Convergence analysis of a proximal-like minimization algorithm using Bregman functions[END_REF]. It has been widely used in optimization [see, e.g., Bauschke and Borwein, 1997, for a review].

We follow this path and include the regularization component g of the objective function = f + g in the Bregman divergence for the sake of the analysis. If g was differentiable we would simply use D h+n g . Since g is not differentiable, D hn is not well defined. However for (↵, ⌘) 2 dom h ⇥ dom h ⇤ n , we denote by extension for

✓ = rh ⇤ n (⌘): Dn (↵, ⌘) = h n (↵) h n (✓) h⌘, ↵ ✓i. (4.12)
This extension is different from the one defined by [START_REF] Kiwiel | Proximal minimization methods with generalized Bregman functions[END_REF]. It is worth noting that if there exists µ such that ↵ = rh ⇤ n (µ), we recover the classical formula Dn (↵, ⌘) = D h ⇤ n (⌘, µ) which is well defined since h ⇤ n is differentiable. Yet Dn is defined more generally since such a µ does not always exist. The next lemma relates Dn to D h and is obvious if g is differentiable since

D hn = D h + nD g . Lemma 22. Let n 0, ↵ 2 dom h and ⌘ 2 dom h ⇤ n , then with ✓ = rh ⇤ n (⌘), Dn (↵, ⌘) D h (↵, ✓). (4.13) Proof. ✓ = rh ⇤ n (⌘), thus ⌘ 2 @h n (✓)
and by elementary calculus rule @h n (✓) = rh(✓) + n @g(✓). Consequently ⌘ rh(✓) 2 n @g(✓) and by convexity of g

Dn (↵, ⌘) D h (↵, ✓) = n  g(↵) g(✓) ⌧ ⌘ rh(✓) n , ↵ ✓ 0.

4.B.3 Proof of Proposition 8

We assume their exists a constant L > 0 such that Lh f is convex on X and we assume the step-size  1/L. We first show that the Bregman divergence decreases along the iterates [see, e.g., [START_REF] Chen | Convergence analysis of a proximal-like minimization algorithm using Bregman functions[END_REF][START_REF] Beck | Mirror descent and nonlinear projected subgradient methods for convex optimization[END_REF][START_REF] Bach | Duality between subgradient and conditional gradient methods[END_REF]. For all ✓ 2 X ,

Dn (✓, ⌘ n ) Dn 1 (✓, ⌘ n 1 ) = h n 1 (✓ n 1 ) h n (✓ n ) + h n (✓) h n 1 (✓) h⌘ n , ✓ ✓ n i + h⌘ n 1 , ✓ ✓ n 1 i = h n 1 (✓ n 1 ) h n 1 (✓ n ) (g(✓ n ) g(✓)) +h⌘ n 1 , ✓ n ✓ n 1 i + h⌘ n ⌘ n 1 , ✓ n ✓i = Dn 1 (✓ n , ⌘ n 1 ) (g(✓ n ) g(✓)) hrf (✓ n 1 ), ✓ n ✓i.
Therefore for all ✓ 2 X ,

Dn (✓, ⌘ n ) Dn 1 (✓, ⌘ n 1 ) = Dn 1 (✓ n , ⌘ n 1 ) + hrf (✓ n 1 ), ✓ n 1 ✓ n i (g(✓ n ) g(✓)) hrf (✓ n 1 ), ✓ n 1 ✓i. (4.14)
It follows from Proposition 11 and Lemma 22

f (✓ n ) f (✓ n 1 ) + hrf (✓ n 1 ), ✓ n ✓ n 1 i  LD h (✓ n , ✓ n 1 )  LD n 1 (✓ n , ✓ n 1 ),
and from the convexity of f ,

hrf (✓ n 1 ), ✓ n 1 ✓i  f (✓) f (✓ n 1 ).
And Eq. ( 4.14) is bounded by

Dn (✓, ⌘ n ) Dn 1 (✓, ⌘ n 1 )  ( (✓) (✓ n )) + ( L 1)D h (✓ n , ✓ n 1 ). Thus for  1/L, Dn (✓, ⌘ n ) Dn 1 (✓, ⌘ n 1 )  ( (✓) (✓ n )).
Taking ✓ = ✓ n 1 we note that the sequence { (✓ n )} n 0 is decreasing and we obtain for  1/L,

(✓ n ) (✓)  1 n + 1 n X k=0 [ (✓ i ) (✓)]  D h (✓, ✓ 0 ) Dn (✓, ⌘ n ) (n + 1) . ( 4

.15)

We assume now that the non-smooth part g = 0 and there exists µ 0 such that f µh is convex. So Proposition 11 implies

hrf (✓ n 1 ), ✓ n 1 ✓i  f (✓) f (✓ n 1 ) µD h (✓, ✓ n 1 ),
which gives with Eq. (4.14) the better bound

D h (✓, ✓ n ) D h (✓, ✓ n 1 )  (f (✓) f (✓ n )) µD h (✓, ✓ n 1 ) + ( L 1)D h (✓ n , ✓ n 1 ).
And for  1/L, this can be simplified as

D h (✓, ✓ n )  (1 µ)D h (✓, ✓ n 1 ) + (f (✓) f (✓ n )).
The sequence {f (✓ n )} n 0 is still decreasing and we obtain by expanding the recursion

D h (✓, ✓ n )  (1 µ) n D h (✓, ✓ 0 ) + n X k=1 (1 µ) n k (f (✓) f (✓ k ))  (1 µ) n D h (✓, ✓ 0 ) + n X k=1 (1 µ) n k (f (✓) f (✓ k ))  (1 µ) n D h (✓, ✓ 0 ) + n X k=1 (1 µ) n k (f (✓) f (✓ n ))  (1 µ) n D h (✓, ✓ 0 ) + 1 (1 µ) n µ (f (✓) f (✓ n )).
Thus for all ✓ 2 X ,

1 (1 µ) n µ (f (✓ n ) f (✓)) + D h (✓, ✓ n )  (1 µ) n D h (✓, ✓ 0 ), and
f (✓ n ) f (✓)  µ(1 µ) n 1 (1 µ) n D h (✓, ✓ 0 )  (1 µ) n D h (✓, ✓ 0 ) , since (1 µ) 2  1 µ implies µ/(1 (1 µ) n )  1.

4.C Proof of Proposition 9

In this section, we will prove Proposition 9. The proof relies on considering the difference between the iteration with noise we denote by (⌘ n , ✓ n ) and without noise we denote by (! n , n ), which happens to verify a similar recursion as the SDA recursion.

-

4.C.1 Two Technical Lemmas

We first present and prove two technical lemmas.

Bound on the Difference of Two Dual Iterates

In the following lemma we show that the difference between two dual iterates that follow the same recursion is of order n. This will be used with the iteration with noise

(⌘ n , ✓ n ) and without noise (! n , n ). Lemma 23. Let us consider two sequences of iterates (µ k , ↵ k ) and (⌫ k , k ) which satisfy the recursion µ n ⌫ n = µ n 1 ⌫ n 1 ⌃(↵ n 1 n 1 ) + ⇠ n , ↵ n = rh ⇤ n (µ n ) and n = rh ⇤ n (⌫ n ) and assume that is such that 2h f is convex then for all n 0 Ekµ n ⌫ n k 2 ⌃ 1  kµ 0 ⌫ 0 k 2 ⌃ 1 + n 2 tr ⌃ 1 C.
Proof. We first expand the square.

kµ n+1 ⌫ n+1 k 2 ⌃ 1 = kµ n ⌫ n k 2 ⌃ 1 + 2 k[⌃(↵ n n ) ⇠ n+1 ]k 2 ⌃ 1 2 h⌃(↵ n n ) ⇠ n+1 , ⌃ 1 (µ n ⌫ n )i = kµ n ⌫ n k 2 ⌃ 1 + 2 k↵ n n k 2 ⌃ + 2 k⇠ n+1 k 2 ⌃ 1 2 2 h↵ n n , ⇠ n+1 i 2 h↵ n n ⌃ 1 ⇠ n+1 , µ n ⌫ n i.
And taking the expectation

E[kµ n+1 ⌫ n+1 k 2 ⌃ 1 |F n ] = kµ n ⌫ n k 2 ⌃ 1 + 2 E[k⇠ n+1 k 2 ⌃ 1 |F n ] + 2 k↵ n n k 2 ⌃ 2 2 E[h↵ n n , ⇠ n+1 i|F n ] 2 E[h↵ n n ) ⌃ 1 ⇠ n+1 , µ n ⌫ n i|F n ] = kµ n ⌫ n k 2 ⌃ 1 + 2 tr ⌃ 1 E[⇠ n+1 ⌦ ⇠ n+1 |F n ] + 2 k↵ n n k 2 ⌃ 2 2 h↵ n n , E[⇠ n+1 |F n ]i 2 h↵ n n ) ⌃ 1 E[⇠ n+1 |F n ], µ n ⌫ n i = kµ n ⌫ n k 2 ⌃ 1 + 2 tr ⌃ 1 C + 2 k↵ n n k 2 ⌃ 2 h↵ n n , µ n ⌫ n i.
Moreover, using the definition of ↵ n and n , with

⇤ = k↵ n n k 2 ⌃ 2h↵ n n , µ n ⌫ n i, ⇤ = h ⌃(↵ n n ) 2(µ n ⌫ n ), ↵ n n i = h rf (↵ n ) rf ( n ) 2(rh(↵ n ) rh( n )), ↵ n n i 2h(µ n rh(↵ n )) (⌫ n rh( n )), ↵ n n i = hr( f 2h)(↵ n ) r( f 2h)( n ), ↵ n n i 2h(µ n rh(↵ n )) (⌫ n rh( n )), ↵ n n i.
Using the h-smoothness of f and assuming that is such 2h f is convex,

hr( f 2h)(↵ n ) r( f 2h)( n ), ↵ n n i  0,
and as explained in the proof of Lemma 22, µ n rh(↵ n ) 2 @n g(↵ n ) and ⌫ n rh( n ) 2 @n g( n ) and consequently

h(µ n rh(↵ n )) (⌫ n rh( n )), ↵ n n i 0,
by convexity of g. This explains that

k↵ n n k 2 ⌃ 2h↵ n n , µ n ⌫ n i  0.
Then, taking the global expectation, we have shown that

Ekµ n+1 ⌫ n+1 k 2 ⌃ 1  Ekµ n ⌫ n k 2 ⌃ 1 + 2 tr ⌃ 1 C,
which concludes the proof.

Bound on the Difference of the Average of Two Primal Iterates

In the following lemma we adapt the classic proof of averaged SGD by [START_REF] Polyak | Acceleration of stochastic approximation by averaging[END_REF] to show that the difference between two averaged primal iterates, which follow the same recursion, is of order O(1/n).

Lemma 24. Let us consider two sequences of iterates (µ k , ↵ k ) and (⌫ k , k ) which satisfy the recursion

µ n ⌫ n = µ n 1 ⌫ n 1 ⌃(↵ n 1 n 1 ) + ⇠ n , ↵ n = rh ⇤ n (µ n ) and n = rh ⇤ n (⌫ n ) and assume that is such that 2h f is convex then for all n 0 Ek↵ n ¯ n k 2 ⌃  4 kµ 0 ⌫ 0 k 2 ⌃ 1 ( n) 2 + 4 n tr ⌃ 1 C.
Proof. Let us consider two sequences of iterates (µ k , ↵ k ) and (⌫ k , k ) which satisfy the recursion

µ n ⌫ n = µ n 1 ⌫ n 1 ⌃(↵ n 1 n 1 ) + ⇠ n , ↵ n = rh ⇤ n (µ n ) and n = rh ⇤ n (⌫ n ).
This can be written as

⌃(↵ n n ) = µ n ⌫ n µ n+1 + ⌫ n+1 + ⇠ n+1 .
Thus we obtain

⌃ 1/2 n 1 X i=0 (↵ i i ) = ⌃ 1/2 (µ 0 ⌫ 0 µ n + ⌫ n ) + n 1 X i=0 ⌃ 1/2 ⇠ i+1 .
Finally, using that by convexity

(a + b) 2  2(a 2 + b 2
), this leads to

n 2 Ek↵ n ¯ n k 2 ⌃  2E ⌃ 1/2 (µ 0 ⌫ 0 ) + n 1 X i=0 ⌃ 1/2 ⇠ i+1 2 2 + 2E ⌃ 1/2 (µ n ⌫ n ) 2 2 .
Using martingale second moment expansions, we obtain

Ek↵ n ¯ n k 2 ⌃  2E kµ 0 ⌫ 0 k 2 ⌃ 1 ( n) 2 + 2 Ekµ n ⌫ n k 2 ⌃ 1 ( n) 2 + 2 n 2 n 1 X i=0 tr ⌃ 1 E(⇠ i+1 ⌦ ⇠ i+1 ).
We compute

P n 1 i=0 tr ⌃ 1 E(⇠ i+1 ⌦ ⇠ i+1 ) = P n 1 i=0 tr ⌃ 1 C = n tr ⌃ 1 C and, using Lemma 23, we bound Ekµ n ⌫ n k 2 ⌃ 1 as Ekµ n ⌫ n k 2 ⌃ 1 ( n) 2  kµ 0 ⌫ 0 k 2 ⌃ 1 ( n) 2 + 1 n tr ⌃ 1 C.

This implies the final bound

Ek↵ n ¯ n k 2 ⌃  4 kµ 0 ⌫ 0 k 2 ⌃ 1 ( n) 2 + 4 n tr ⌃ 1 C.

4.C.2 Application of Lemma 24 to Prove Proposition 9

First of all we define the sequence

⌘ ⇤ n = rh(✓ ⇤ ) n rf (✓ ⇤ ). (4.16) By definition of ✓ ⇤ , rf (✓ ⇤ ) 2 @g(✓ ⇤ ) then ⌘ ⇤ n 2 @(h + n g)✓ ⇤ and ✓ ⇤ = rh ⇤ n (⌘ ⇤ n ). Therefore the sequence ⌘ ⇤
n is obtained by iterating DA started from the solution of the problem ✓ ⇤ .

We note then than Lemma 24 applied to

(µ n = ⌘ n , ↵ n = ✓ n ) and (⌫ n = ⌘ ⇤ n , n = ✓ ⇤ )
gives the first bound of Proposition 9.

On the other hand, when considering the noiseless iterates (! n , n ) defined by

! n = ! n 1 ⌃( n 1 ✓ ⌃ ) and n = rh ⇤ n (! n )
, started from the same point 0 = ✓ 0 , we obtain, following Proposition 8, for such that h f is convex, the bound

1 2 k ¯ n ✓ ⇤ k 2 ⌃  ( ¯ n ) (✓ ⇤ )  D h (✓ ⇤ , ✓ 0 ) n .
Therefore, considering the difference between the semi-stochastic and the noiseless iterate (⌘ n ! n ) which verifies the same equation

⌘ n ! n = ⌘ n 1 ! n 1 ⌃(✓ n 1 n 1 ) + ⇠ n with ✓ 0 0 =
0 as initial value, we may apply Lemma 24 to show

Ek ✓n ¯ n k 2 ⌃  4 n tr ⌃ 1 C.
And by the Cauchy-Schwarz inequality

Ek ✓n ✓ ⇤ k 2 ⌃  2Ek ✓n ¯ n k 2 ⌃ + 2Ek ¯ n ✓ ⇤ k 2 ⌃  8 n tr ⌃ 1 C + 4 D h (✓ ⇤ , ✓ 0 ) n ,
which proves the second bound of Proposition 9.

It is worth noting that the condition on the step-size of Lemma 23 is less restrictive than in Proposition 9. Indeed for all such that 2h f is convex, the difference between the dual iterates of the stochastic and deterministic recursions stay close but the deterministic iterates only converge to the solution for such that h f is convex.

4.D Proof of Proposition 10

In this section, we prove Proposition 10. The proof technique is similar to Proposition 9 but with the additional difficulty of the multiplicative noise.

We first note that Assumption (A11) is equivalent by the Cauchy-Schwarz inequality to

Ehx n , Mx n ihx n , Nx n i   tr(M ⌃) tr(N ⌃), (4.17) for all positive semi-definite symmetric matrices M and N , see, e.,g., proof in Chapter 3. We will often use, in the following demonstrations, Eq. ( 4.17) and its direct corollary (4.18) without always referring to it.

hx n , Mx n ix n ⌦ x n 4  tr(M ⌃)⌃,

4.D.1 A Simple Proof for the Bounded Constrained Case

We first prove Proposition 10 for the constrained case. It is then a simple corollary of Proposition 9.

Let us denote by C a bounded convex set and consider the constrained problem (g = 1 C ). We remind that the general stochastic oracle for SDA in least-squares regression is

rf n (✓) = (⌃ + ⇣ n )(✓ ✓ ⌃ ) ⇠ n , for ✓ 2 R d , with ⇣ n = x n ⌦ x n ⌃.
We denote by r = max ✓2C k✓ ✓ ⌃ k 2 and we show that the noise covariance is directly bounded, despite the multiplicative noise:

E h rf n (✓) rf (✓) ⌦ rf n (✓) rf (✓) i 4 2E ⇥ ⇣ n (✓ ✓ ⌃ ) ⌦ (✓ ✓ ⌃ )⇣ n ⇤ + 2E⇠ n ⌦ ⇠ n ,
and using Assumption (A11)

E ⇥ ⇣ n (✓ ✓ ⌃ ) ⌦ (✓ ✓ ⌃ )⇣ n ⇤ 4 r 2 E⇣ n ⇣ n 4 r 2 (tr ⌃)⌃. Therefore E h rf n (✓) rf (✓) ⌦ rf n (✓) rf (✓) i 4 2 2 + r 2 (tr ⌃) ⌃.
Hence Proposition 9 already implies for all step-size such that h f is convex

1 2 Ek ✓n ✓ ⇤ k 2 ⌃  2 D h (✓ ⇤ , ✓ 0 ) n + 8d n ( 2 + r 2 tr ⌃).

4.D.2 A General Result

We prove in this section a more general result than Proposition 10 under the additional assumption

(A14) There exists b 2 [0, 1] and µ b > 0 such that h µ b 2 k • k 2 ⌃ b is convex.
Proposition 12. Assume (A2-4) and (A7-14). Consider the recursion in Eq. ( 4.8).

For any constant step-size such that

 min{ µ b 4 tr ⌃ 1 b , 1 Ld }. Then 1 2 Ek ✓n ✓ ⇤ k 2 ⌃  2 D h (✓ ⇤ , ✓ 0 ) n + 24 n tr ⌃ 1 C + 16d nµ b tr C⌃ b + 8d n ✓ 4 tr ⌃ 1 b µ b +3 ◆ k✓ ⇤ ✓ ⌃ k 2 ⌃ +80 d n 2 D h (✓ ⇤ , ✓ 0 )+ 16d n 2 g(✓ 0 ).
We note Assumption (14) is always satisfied for b = 1, for which it is Assumption (13). Therefore Proposition 12 directly implies Proposition 10 as a corollary. We prove now two auxiliary lemmas which will be used in the proof of the Proposition 12.

4.D.3 Two Auxiliary Results for Least-Squares Objectives

For b 2 [0, 1], we denote by T b the operator

T b = E[hx, ⌃ b xix ⌦ x].
We first prove that, for least-square objectives, the sum of the function evaluated along the primal iterates remains bounded.

Lemma 25. Let us consider the recursion ⌘ n = ⌘ n 1

x n ⌦ x n (✓ n 1 ✓ ⇤ ) + ⇠ n and assume g is positive and there exist

µ b such that h µ b 2 k • k 2 ⌃ b is convex and  such that T b 4  tr(⌃ 1 b )⌃, then for  µ b /(4 tr ⌃ 1 b ) and ✓ 2 X we have E n X i=0 [ (✓ i ) (✓)] + ⇣ 1 4  tr(⌃ 1 b )/µ b ⌘ n X i=0 1 2 Ek✓ i ✓k 2 ⌃  D h (✓, ✓ 0 ) ED h (✓, ✓ n+1 ) +(n+1) /µ b tr ⌃ b C+4(n+1) tr(⌃ 1 b )/µ b f (✓)+g(✓ 0 ).
We note that we can also obtain a bound depending on 2 (✓) rather than 4f (✓) with a similar proof.

Proof. Let denote by f n (✓) = x n ⌦ x n (✓ ✓ ⌃ ) + ⇠ n .
Then following the proof of Proposition 8 (see Eq. ( 4.14)) we have the expansion

Dn (✓, ⌘ n ) Dn 1 (✓, ⌘ n 1 )  (g(✓ n ) g(✓)) hrf n (✓ n 1 ), ✓ n 1 ✓i D h (✓ n , ✓ n 1 ) + hrf n (✓ n 1 ), ✓ n 1 ✓ n i. (4.19) Since h µ b 2 k • k 2 ⌃ b is convex, using Proposition 11, we get that D h (✓ n , ✓ n 1 ) µ b 2 k✓ n ✓ n 1 k 2 ⌃ b . Let denote by A = D h (✓ n , ✓ n 1 ), + hrf n (✓ n 1 ), ✓ n 1 ✓ n i, A  µ b 2 k✓ n ✓ n 1 k 2 ⌃ b + hx n ⌦ x n (✓ n 1 ✓ ⌃ ) + ⇠ n , ✓ n 1 ✓ n i  µ b 2 k✓ n ✓ n 1 k 2 ⌃ b + D ⌃ b/2 p µ b [x n ⌦ x n (✓ n 1 ✓ ⌃ ) + ⇠ n ], ⌃ b/2 p µ b ✓ n 1 ✓ n E  2 µ b 2 kx n ⌦ x n (✓ n 1 ✓ ⌃ ) + ⇠ n k 2 ⌃ b 1 2 k ⌃ b/2 p µ b (✓ n ✓ n 1 ) ⌃ b/2 p µ b [x n ⌦ x n (✓ n 1 ✓ ⌃ ) + ⇠ n ]k 2 2  2 2µ b kx n ⌦ x n (✓ n 1 ✓ ⌃ ) + ⇠ n k 2 ⌃ 1  2 µ b k✓ n 1 ✓ ⌃ k 2 T b + 2 µ b k⇠ n k 2 ⌃ b .
Thus, taking the conditional expectation and assuming that  is such that

T b 4  tr(⌃ 1 b )⌃ we obtain E[D h (✓ n , ✓ n 1 )F n 1 ] + Ehrf n (✓ n 1 ), ✓ n 1 ✓ n iF n 1 ]  2  tr(⌃ 1 b ) µ b k✓ n 1 ✓ ⌃ k 2 ⌃ + 2 µ b tr ⌃ b C.
Taking again the conditional expectation in Eq. ( 4.19), we have for ✓ 2 X

E[ Dn (✓, ⌘ n )|F n 1 ] Dn 1 (✓, ⌘ n 1 )  2  µ b tr(⌃ 1 b )k✓ n 1 ✓ ⌃ k 2 ⌃ + 2 µ b tr ⌃ b C E[hx n ⌦ x n (✓ n 1 ✓ ⌃ ) + ⇠ n , ✓ n 1 ✓i|F n 1 ] (E[g(✓ n )|F n 1 ] g(✓))  2  µ b tr(⌃ 1 b )k✓ n 1 ✓ ⌃ k 2 ⌃ h✓ n 1 ✓ ⌃ , ⌃(✓ n 1 ✓)i + 2 µ b tr ⌃ b C (E[g(✓ n )|F n 1 ] g(✓)).
And we note that

h✓ n 1 ✓ ⌃ , ⌃(✓ n 1 ✓ ⇤ )i = [f (✓ n 1 ) f (✓ ⇤ )] 2 k✓ n 1 ✓k 2 ⌃ .
Therefore

E[ Dn (✓ ⇤ , ⌘ n )|F n 1 ] Dn 1 (✓ ⇤ , ⌘ n 1 )  [f (✓ n 1 ) f (✓ ⇤ ) + E[g(✓ n )|F n 1 ] g(✓ ⇤ )] 2 ⇣ 1 4  µ b tr(⌃ 1 b ) ⌘ k✓ n 1 ✓ ⌃ k 2 ⌃ +2 2  µ b tr(⌃ 1 b )k✓ ✓ ⌃ k 2 ⌃ + 2 µ b tr ⌃ b C.
Taking the total expectation we obtain

Ef (✓ n 1 ) f (✓ ⇤ ) + Eg(✓ n ) g(✓ ⇤ )] + 1 2 ⇣ 1 4  µ b tr(⌃ 1 b ) ⌘ k✓ n 1 ✓ ⌃ k 2 ⌃  E Dn 1 (✓ ⇤ , ⌘ n 1 ) E Dn (✓ ⇤ , ⌘ n ) + 2  µ b tr(⌃ 1 b )k✓ ✓ ⌃ k 2 ⌃ + µ b tr ⌃ b C, which, summing from i = 0 to i = n, leads to n X i=0 [Ef (✓ i ) f (✓ ⇤ ) + Eg(✓ i ) g(✓ ⇤ )]] + ⇣ 1 4  µ b tr(⌃ 1 b ) ⌘ n X i=0 1 2 k✓ i ✓ ⌃ k 2 ⌃  D h (✓ ⇤ , ✓ 0 ) E Dn+1 (✓ ⇤ , ⌘ n+1 ) + 4  µ b tr(⌃ 1 b )(n + 1)k✓ ✓ ⌃ k 2 ⌃ + (n + 1) µ b tr ⌃ b C Eg(✓ n+1 ) + g(✓ 0 ).
The result follows if g is non negative.

We now present an extension of Lemma 23 to least-squares objectives.

Lemma 26. Let us consider two sequences of iterates (µ k , ↵ k ) and (⌫ k , k ) which satisfy the recursion

µ n ⌫ n = µ n 1 ⌫ n 1 x n ⌦x n (↵ n 1 n 1 )+ ⇠ n , ↵ n = rh ⇤ n (µ n ) and n = rh ⇤ n (⌫ n ) and denote by C = E[x n ⌦ x n ] for n 0. Assume that is such that h T is convex. Then Ekµ n ⌫ n k 2 ⌃ 1  Ekµ 0 ⌫ 0 k 2 ⌃ 1 + 2 2 n tr ⌃ 1 C.
We note that the condition h T is rather restrictive since bounds on T are often of the form d times a matrix. For instance Eq. ( 4.17) directly implies T 4 d⌃. Even for independent normal data x n with diagonal covariance matrix ⌃ we are able to derive the equality T = (d + 2)⌃.

Proof. We expand

kµ n ⌫ n k 2 ⌃ 1 = kµ n 1 ⌫ n 1 k 2 ⌃ 1 + 2 kx n ⌦ x n (↵ n 1 n 1 ) + ⇠ n k 2 ⌃ 1 2 hx n ⌦ x n (↵ n 1 n 1 ) + ⇠ n , ⌃ 1 (µ n 1 ⌫ n 1 )i.
Taking conditional expectations, we get

E[kµ n ⌫ n k 2 ⌃ 1 |F n 1 ] = kµ n 1 ⌫ n 1 k 2 ⌃ 1 + 2 E[kx n ⌦ x n (↵ n 1 n 1 )+⇠ n k 2 ⌃ 1 |F n 1 ] 2 E[hx n ⌦ x n (↵ n 1 n 1 ) + ⇠ n , ⌃ 1 (µ n 1 ⌫ n 1 )i|F n 1 ] = kµ n 1 ⌫ n 1 k 2 ⌃ 1 + 2 E[kx n ⌦ x n (↵ n 1 n 1 )+⇠ n k 2 ⌃ 1 |F n 1 ] 2 h↵ n 1 n 1 , µ n 1 ⌫ n 1 i.
Using (a+b) 2  2a 2 +2b 2 and denoting by

B = E[kx n ⌦x n (↵ n 1 n 1 )+⇠ n k 2 ⌃ 1 |F n 1 ], this leads to B  2E[kx n ⌦ x n (↵ n 1 n 1 )k 2 ⌃ 1 |F n 1 ] + 2E[k⇠ n k 2 ⌃ 1 |F n 1 ]  2k↵ n 1 n 1 k 2 E[xn⌦xn⌃ 1 xn⌦xn|F n 1 ] + 2 tr ⌃ 1 E[" n ⌦ " n |F n 1 ]  2k↵ n 1 n 1 k 2 T + 2 tr ⌃ 1 C, with T = E[x ⌦ x⌃ 1 x ⌦ x]. Thus we obtain E[kµ n ⌫ n k 2 ⌃ 1 |F n 1 ]  kµ n 1 ⌫ n 1 k 2 ⌃ 1 + 2 2 tr ⌃ 1 C 2 hµ n 1 ⌫ n 1 T (↵ n 1 n 1 ), ↵ n 1 n 1 i  kµ n 1 ⌫ n 1 k 2 ⌃ 1 + 2 2 tr ⌃ 1 C, assuming that is such h 1 2 k • k 2
T is convex (as in the proof of Lemma 23). Taking global expectations, we have shown that

Ekµ n ⌫ n k 2 ⌃ 1  Ekµ n 1 ⌫ n 1 k 2 ⌃ 1 + 2 2 tr ⌃ 1 C.

4.D.4 Bound on the Difference between Two Averages of Primal Variables

We present now the following lemma with is an analogue of Lemma 24 for the leastsquares problem. It shows that the difference between the average of two sequences of primal iterates which follow the same recursion is O(1/n).

Lemma 27. Let us consider two sequences of iterates (µ k , ↵ k ) and (⌫ k , k ) which satisfy the recursion

µ n ⌫ n = µ n 1 ⌫ n 1 x n ⌦x n (↵ n 1 n 1 )+ ⇠ n , ↵ n = rh ⇤ n (µ n ) and n = rh ⇤ n (⌫ n ). For n 0 denote by C = E[x n ⌦ x n ].
Assume that is such that h T is convex and there exists  such that T 4 d⌃. Then

Ek↵ n ¯ n k 2 ⌃  4 d 1 n 2 n 1 X i=0 Ek↵ i i k 2 ⌃ + 4 k⌘ 0 µ 0 k 2 ⌃ 1 ( n) 2 + 8 n tr ⌃ 1 C. Proof. Using the expansion µ n ⌫ n = µ n 1 ⌫ n 1 x n ⌦ x n (↵ n 1 n 1 ) + ⇠ n , we derive ⌃(↵ n n ) = (⌃ x n+1 ⌦ x n+1 )(↵ n n ) + x n+1 ⌦ x n+1 (↵ n n ) = (⌃ x n+1 ⌦ x n+1 )(↵ n n ) + µ n ⌫ n µ n+1 + ⌫ n+1 + ⇠ n+1 .
We obtain by summing n times

⌃ 1/2 n 1 X i=0 (↵ i i ) = n 1 X i=0 ⌃ 1/2 X i+1 + ⌃ 1/2 (µ 0 ⌫ 0 µ n + ⌫ n ) + n 1 X i=0 ⌃ 1/2 ⇠ i+1 ,
where we denote by

X i = (⌃ x i ⌦ x i )(↵ i 1 i 1 ) which is a square-integrable martingale difference sequence. We use (a + b) 2  2(a 2 + b 2 ) to obtain k(↵ n ¯ n )k 2 ⌃  2 n 2 n 1 X i=0 ⌃ 1/2 X i+1 + ⌃ 1/2 (µ 0 ⌫ 0 ) + n 1 X i=0 ⌃ 1/2 ⇠ i+1 2 2 +2 kµ n ⌫ n k 2 ⌃ 1 ( n) 2
.

Therefore using martingale square moment inequalities which here amount to considering the variance of the sum as the sum of the variance, we have

Ek(↵ n ¯ n )k 2 ⌃  4 n 2 n 1 X i=0 EkX i+1 k 2 ⌃ 1 + 2 kµ 0 ⌫ 0 k 2 ⌃ 1 ( n) 2 + 2 Ekµ n ⌫ n k 2 ⌃ 1 ( n) 2 + 4 n 2 n 1 X i=0 tr ⌃ 1 E(⇠ i+1 ⌦ ⇠ i+1 ). (4.20)

147

-The variance term may be bounded as

n 1 X i=0 tr ⌃ 1 E(⇠ i+1 ⌦ ⇠ i+1 )  n 1 X i=0 tr ⌃ 1 C  n tr ⌃ 1 C.
-Following Lemma 26 we bound the dual iterates

Ekµ n ⌫ n k 2 ⌃ 1 as Ekµ n ⌫ n k 2 ⌃ 1 ( n) 2  kµ 0 ⌫ 0 k 2 ⌃ 1 ( n) 2 + 2 n tr ⌃ 1 C.
-The martingale difference sequence (X i ) satisfies

Ek⌃ 1/2 X i+1 k 2 2  Eh(⌃ x i+1 ⌦ x i+1 )(↵ i i ), ⌃ 1 (⌃ x i+1 ⌦ x i+1 )(↵ i i )i  h↵ i i , E[(⌃ x i+1 ⌦ x i+1 ) > ⌃ 1 (⌃ x i+1 ⌦ x i+1 )](↵ i i )i  h↵ i i , [E(x i+1 ⌦ x i+1 ) > ⌃ 1 x i+1 ⌦ x i+1 ⌃](↵ i i )i  h↵ i i , [T ⌃](↵ i i )i  (d 1)k↵ i i k 2 ⌃ .
Consequently we obtain in Eq. ( 4.20)

Ek⌃ 1/2 (↵ n ¯ n )k 2 2  4 d 1 n 2 n 1 X i=0 Ek↵ i i k 2 ⌃ + 4 kµ 0 ⌫ 0 k 2 ⌃ 1 ( n) 2 + 8 n tr ⌃ 1 C.

4.D.5 Application of Lemma 27 to the Proof of Proposition 12

We are now able to prove Proposition 12 using Lemma 27.

Firstly we can directly apply Lemma 27 to

(µ n = ⌘ n , ↵ n = ✓ n ) and (⌫ n = ⌘ ⇤ n , n = ✓ ⇤ ) where (⌘ ⇤ n , ✓ ⇤ )
are defined in Eq. (4.16). This implies

Ek⌃ 1/2 ( ✓n ✓ ⇤ )k 2 2  4 d 1 n 2 n 1 X i=0 Ek✓ i ✓ ⇤ k 2 ⌃ + 4 krh(✓ 0 ) rh(✓ ⇤ )k 2 ⌃ 1 ( n) 2 + 8 n tr ⌃ 1 C.
Following Lemma 25, the primal variables

(✓ i ) satisfy n 1 X i=0 Ek✓ i ✓ ⇤ k 2 ⌃  2 D h (✓ ⇤ , ✓ 0 ) + 2 n µ b tr ⌃ b C + 8n  tr ⌃ 1 b µ b f (✓ ⇤ ) + 2g(✓ 0 ).
This leads to the final bound

Ek⌃ 1/2 ( ✓n ✓ ⇤ )k 2 2  8 d 1 n 2 D h (✓ ⇤ , ✓ 0 ) + 4 krh(✓ 0 ) rh(✓ ⇤ )k 2 ⌃ 1 ( n) 2 + 8 1 n tr ⌃ 1 C + 8 d 1 n µ b [tr ⌃ b C + 4 tr ⌃ 1 b f (✓ ⇤ )] + 8 d 1 n 2 g(✓ 0 ). (4.21)
This bound depends on k • k ⌃ 1 which may be infinite. For this reason we compare again the noisy iterate ✓ n to the noiseless iterate we still denote by ( n ). We remind these iterates verify the recursion

! n = ! n 1 ⌃( n 1 ✓ ⌃ )
. Therefore the difference (⌘ n ! n ) satisfies the same form of recursion as (⌘ n ):

⌘ n ! n = r⌘ n 1 ! n 1 x n ⌦ x n (✓ n 1 n 1 ) + " n , with a different noise " n = ⇠ n [x n ⌦ x n ⌃]( n 1 ✓ ⌃ )
and 0 for initial value. Although the noise " n is different from ⇠ n , its covariance is still bounded by

1 3 E[" n ⌦ " n ] 4 E[⇠ n ⌦ ⇠ n ] + E[[x n ⌦ x n ⌃]( n 1 ✓ ⇤ ) ⌦ ( n 1 ✓ ⇤ )[x n ⌦ x n ⌃]] +E[[x n ⌦ x n ⌃](✓ ⇤ ✓ ⌃ ) ⌦ (✓ ⇤ ✓ ⌃ )[x n ⌦ x n ⌃]] 4 E[⇠ n ⌦ ⇠ n ] E[⌃( n 1 ✓ ⇤ ) ⌦2 ⌃] E[⌃(✓ ⇤ ✓ ⌃ ) ⌦2 ⌃] +E[x n ⌦ x n ( n 1 ✓ ⇤ ) ⌦2 x n ⌦ x n ] + E[x n ⌦ x n (✓ ⇤ ✓ ⌃ ) ⌦2 x n ⌦ x n ] 4 E[⇠ n ⌦ ⇠ n ] + ( 1) k n 1 ✓ ⇤ k 2 ⌃ + k✓ ⇤ ✓ ⌃ k 2 ⌃ ⌃
, where we have use that for z 2 R d , Ehz, x n i 4  hz, ⌃zi. We may apply Proposition 8 and obtain

E[" n ⌦ " n ] 4 3C + 6( 1) n D h (✓ ⇤ , ✓ 0 )⌃ + 6( 1)f (✓ ⇤ ).
Thereby Lemma 27 can be applied with ✓ 0 = ↵ 0 and we get

Ek ✓n ¯ n k 2 ⌃  4 d 1 n 2 n 1 X i=0 Ek✓ i i k 2 ⌃ + 8 n tr ⌃ 1 E[" n ⌦ " n ].
As before we apply Lemma 25 to have

n 1 X i=0 Ek✓ i i k 2 ⌃  " 2 n 1 X i=0 Ek✓ i ✓ ⇤ k 2 ⌃ + 2 n 1 X i=0 k i ✓ ⇤ k 2 ⌃ #  " 8D h (✓ ⇤ , ✓ 0 ) + n µ b 4 tr ⌃ b C + 16n  tr ⌃ 1 b µ b f (✓ ⇤ ) + 4g(✓ 0 ) # . Therefore Ek ✓n ¯ n k 2 ⌃  4 d 1 n 2 " 8 D h (✓ ⇤ , ✓ 0 ) + n µ b 4 tr ⌃ b C + 16n  tr ⌃ 1 b µ b f (✓ ⇤ ) + 4g(✓ 0 ) # + 8 n " 3 tr ⌃ 1 C + 6( 1) n D h (✓ ⇤ , ✓ 0 )d + 6( 1)f (✓ ⇤ )d # .
And rearranging terms we obtain

Ek ✓n ¯ n k 2 ⌃  80 d n 2 D h (✓ ⇤ , ✓ 0 )+ 64 2 d nµ b tr ⌃ 1 b f (✓ ⇤ )+ 16d nµ b tr C⌃ b + 16d n 2 g(✓ 0 )+ 24 n tr ⌃ 1 C + 48d n f (✓ ⇤ ).
And by the Cauchy-Schwarz inequality (Ek ✓n

✓ ⇤ k 2 ⌃  2Ek ✓n ¯ n k 2 ⌃ + 2Ek ¯ n ✓ ⇤ k 2 ⌃ ) 1 2 Ek ✓n ✓ ⇤ k 2 ⌃  2 D h (✓ ⇤ , ✓ 0 ) n + 24 n tr ⌃ 1 C + 16d nµ b tr C⌃ b + 16d n ⇣ 4 tr ⌃ 1 b µ b + 3 ⌘ f (✓ ⇤ ) + 80 d n 2 D h (✓ ⇤ , ✓ 0 ) + 16d n 2 g(✓ 0 ),
which proves the second bound of Proposition 12.

4.D.6 A Corollary of Proposition 12 for h with an Euclidean Behavior

When h rather behaves as an Euclidean norm, we may replace Assumptions (A12-13) by the following:

(A12') There exists µ h > 0 such that h µ h 2 k • k 2 2 is convex. (A13 ') There exists R 2 such that E[kx n k 2 2 x n ⌦ x n ] 4 R 2 ⌃.
And Proposition 12 implies the following corollary. Corollary 6. Assume For any constant step-size such that  min{ µ h 4R 2 , R 2 4d }. Then

1 2 Ek ✓n ✓ ⇤ k 2 ⌃  2 D h (✓ ⇤ , ✓ 0 ) n + 8 n ⇣ 3 + 4 R 2 µ h ⌘⇣ 2 d + dk✓ ⇤ ✓ ⌃ k 2 ⌃ ⌘ + 16d n 2 ⇣ 5D h (✓ ⇤ , ✓ 0 ) + g(✓ 0 ) ⌘ .
This corollary would pave the way for a general result for larger step-size without the condition  R 2 4d . Unfortunately the latter seems not improvable, as noted after Lemma 26.

4.E Lower Bound for Non-Strongly Convex Quadratic Regularization

estimates of the gradient (4.22) where (⇠ n ) is an uncorrelated zero-mean noise sequence with bounded covariance.

rf n (✓) = a + ⇠ n for n 1,
Proposition 13. For any d 2, L > 0; > 0 and finite time horizon N 1, there exists a quadratic function g L-smooth such that for any uncorrelated zero-mean noise sequence (⇠ n ) with bounded covariance E[⇠ n ⌦⇠ n ] = 2 LI d , SDA with constant step-size applied with the oracle Eq. ( 4.22) satisfies

( ✓N ) (✓ ⇤ ) 2 12 min{(L ) 2 , 1}.
Proof. For sake of clarity, we consider d = 2 and a = 0. Thus f (✓) = Eh⇠ n , ✓i = 0.

Let g(✓) = 1 2 h✓, A✓i be a quadratic form with A =

✓ L 0 0 µ ◆ for L µ > 0 with
µ possibly arbitrary small. The noise (⇠ n ) is assumed to be uncorrelated zero-mean with bounded covariance

E[⇠ n ⌦ ⇠ n ] = 2 LI 2 .
The stochastic dual algorithm with step-size takes the form:

✓ n = rh ⇤ n ( n ⇠n ) = arg min ✓2R d n h ⇠n , ✓i + 1 2 h✓, A✓i + 1 2n k✓k 2 2 o = n(I + nA) 1 ⇠n . And ✓n = n n 1 X k=1 k X j=1 k(I + kA) 1 1 k ⇠ j = n n 1 X j=1 ⇣ n 1 X k=j (I + kA) 1 ⌘ ⇠ j .
Therefore using standard martingale square moment inequalities

Eh ✓n , A ✓n i = 2 n 2 n X j=1 E D ⇠ j ⇣ n X k=j (I + kA) 1 ⌘ , A ⇣ n X k=j (I + kA) 1 ⌘ ⇠ j E = 2 2 L n 2 tr n X j=1 ⇣ n X k=j (I + kA) 1 ⌘ A ⇣ n X k=j (I + kA) 1 ⌘ I 2 = 2 2 L n 2 n X j=1 h L ⇣ n X k=j 1 1 + Lk ⌘ 2 + µ ⇣ n X k=j 1 1 + µk ⌘ 2 i . And Eh ✓n , A ✓n i 2 2 L n 2 h L (1 + Ln) 2 + µ (1 + µn) 2 i n X j=1 (n j) 2 n 2 2 L 3 h L (1 + Ln) 2 + µ (1 + µn) 2 i n 2 2 3 µ (1 + µn) 2 2 L 12 min ⇣ nµ 2 , 1 µn ⌘ .
Conclude by taking µ = L/N . The proof is the same for d 2 by considering A = diag(L, . . . , L, Lµ) with d 1 L.

4.F Lower Bound for Stochastic Approximation Problems

In this section we relate the problem of aggregation of estimators to the stochastic convex optimization problem, i.e., minimizing a convex function, given only unbiased estimates of its gradients. We will consider the regression and the classification with hinge loss problems which will individually provide lower bounds for quadratic and linear functions. We follow here [START_REF] Tsybakov | Optimal rates of aggregation[END_REF], [START_REF] Lecué | Optimal oracle inequality for aggregation of classifiers under low noise condition[END_REF], [START_REF] Agarwal | Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization[END_REF].

4.F.1 Oracle Complexity of Stochastic Convex Optimization

Beforehand we describe the stochastic oracle model formalism as done by [START_REF] Nemirovsky | Problem Complexity and Method Efficiency in Optimization[END_REF], [START_REF] Agarwal | Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization[END_REF], [START_REF] Raginsky | Information-based complexity, feedback and dynamics in convex programming[END_REF]. For a given class of problems we aim to determine lower bounds on the number of queries to a stochastic first-order oracle needed to optimize to a certain precision any function in this class. To this end we have the following definition.

Definition 3 [START_REF] Agarwal | Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization[END_REF]). For a given constraint convex set C, and a function class S, a first-order stochastic oracle is a random mapping ⇡ :

C ⇥ S ! R ⇥ R d of the form (✓, f ) = ( f (✓), g(✓)), such that E f (✓) = f (✓); Eg(✓) = rf (✓),
and there exists a constant C < 1 such that for every

✓ 2 R d E[kg(✓) rf (✓)k 2 ]  C(1 + k✓k 2 ).
The class of first-order stochastic oracle is denoted by . A stochastic approximation algorithm M is a method which approximately minimizes a function f by querying, at each iteration i, the oracle at the point ✓ i . The oracle answers with the information (✓ i , f) and the method uses all the information { (✓ 0 , f), . . . , (✓ i , f)} to build a new point ✓ i+1 . For n 2 N we denote by M n the class of all such methods that are allowed to make n queries. As done by [START_REF] Agarwal | Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization[END_REF], we denote the error of the method M on the function f after n steps as

" n (M, f, C, ) = f (✓ n ) min ✓2C f (✓).
Given a class of functions S, an oracle and a convex constraint set C, [START_REF] Agarwal | Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization[END_REF] also defines the minimax error as

" ⇤ n (S, C, ) = inf M 2Mn sup f 2S E " n (M, f, C, ).
We will lower bound this minimax error by relating convex stochastic approximation with convex aggregation of estimators [Juditsky andNemirovski, 2000, Tsybakov, 2003].

4.F.2 Convex Aggregation of Estimators

Let (X , A) be a measurable space and Y ⇢ R . We consider random variables (X, Y ) on X ⇥ Y with probability distribution denoted by ⇡. We observe n i.i.d. pairs D n = {(X 1 , Y 1 ), . . . , (X n , Y n )} which follow the law ⇡ and we want to predict the output Y for any feature X 2 X by a prediction f (X) for a measurable function f from X to R. For this purpose we want to minimize the risk defined by

A(f ) = E[`(f (X), Y )],
for any measurable function f from X to R and `: Y ⇥ Y ! R a loss function.

We consider we have access to d different arbitrary estimators F = {f 1 , . . . , f d } with values in Y. We denote their convex hull by C = conv(f 1 , . . . , f d ). The aim of convex aggregation is to build a new estimator which is a convex combination of the different f i and behaves as the best among the estimators f i . The aggregation problem is equivalent to a minimization problem over the simplex d since for f 2 C there is

✓ 2 d such that f = P d i=1 ✓(i)f i . Therefore, defining B(✓) = A ⇣ P d i=1 ✓(i)f i ⌘ , we have min f 2C A(f ) = min ✓2 d B(✓).
We denote by F :

X ! R d , x 7 ! (f 1 (x), . . . , f d (x)
) the function whose the ith coordinate is the function f i , and we have

B(✓) = E[`(hF (X), ✓i, Y )].
Therefore the convex aggregation problem of minimizing A(f ) over the convex hull of F is formally equivalent to the stochastic approximation problem of minimizing, over the simplex d , the function B(✓) = E[`(hF (X), ✓i, Y )], given only unbiased estimates of its gradient rB n (✓) = r`(hF (x n ), ✓i, y n ). Hence lower bounds on convex aggregation problems provide lower bounds on stochastic approximation problems studied in this chapter.

4.F.3 Aggregation in Regression and Application to Oracle Complexity of Stochastic Quadratic Optimization

We first consider the regression problem for which Y = R. We rely substantially on [START_REF] Tsybakov | Optimal rates of aggregation[END_REF]. The regression model is

Y i = f ⇤ (X i ) + ⇠ i , for i = 1, . . . , n,
where X 1 , . . . , X n are i.i.d. random vectors of X of law P X and ⇠ i are i.i.d. Gaussian N (0, 2) random variables such that (⇠ 1 , . . . , ⇠ n ) is independent of (X 1 , . . . , X n ) and f ⇤ : X ! R is the regression function. Regression problem aims to estimate the unknown regression function f ⇤ based on the data D n by minimizing the risk

A reg (f, f ⇤ ) = E(f (X) f ⇤ (X)) 2 .
The problem of the optimal rate of convex aggregation has been studied by [START_REF] Tsybakov | Optimal rates of aggregation[END_REF]. We reintroduce his notations and assumptions for sake of completeness.

Let denote by F 0 = {f : kf k 1  L} for L > 0 and assume that (B1) There exists a cube S ⇢ X such that P X admits a bounded density µ on S w.r.t. the Lebesgue measure and µ(x) µ 0 > 0 for all x 2 S. (B2) There exists a constant c 0 such that d  c 0 exp(n). We have the following result Theorem 9 (Theorem 2, [START_REF] Tsybakov | Optimal rates of aggregation[END_REF]). Under assumptions (B1-2) we have

sup f 1 ,...,f d 2F 0 inf Tn sup f⇤2F 0 ⇥ E Dn A reg (T n , f ⇤ ) min f 2C A reg (f, f ⇤ ) ⇤ c⇣ n (d),
for some constant c > 0 and any integer n, where inf Tn denotes the infimum over all estimators, E Dn denotes the expectation with regard to the probability distribution of the data D n and

⇣ n (d) = ( d/n if d  p n q 1 n log d p n + 1 if d > p n.
We relate now the problem of convex aggregation of regression functions to the problem of stochastic quadratic functions optimization. Consider F = {f 1 , . . . , f d } the set of estimators given by Proposition 9 and denote by F :

X ! R d , x 7 ! (f 1 (x), . . . , f d (x)). For f 2 C, there is ✓ 2 d such that f = P d i=1 ✓(i)f i and we obtain A reg (f, f ⇤ ) = E[(h✓, F (X)i f ⇤ (X)) 2 ] = B(✓), where B(✓) = h✓, E[F (X) ⌦ F (X)], ✓i 2h✓E[f ⇤ (X)F (X)]i + E[f ⇤ (X) 2
] is a quadratic function. This set enables us to construct a difficult subclass of quadratic functions:

G quad = n B(✓) = 1 2 E[(h✓, F (X)i f ⇤ (X)) 2 ]; f ⇤ 2 F 0 o .
We also define the first-order stochastic oracle quad on G quad as follows

quad (✓, f ) = ✓ 1 2 (h✓, F (x)i f ⇤ (x)) 2 , (h✓, F (x)i f ⇤ (x))F (x) ◆ , for x ⇠ P X .
We can optimize B with a stochastic approximation algorithm M 2 M n to obtain ✓ n 2 d and therefore build a estimator

T n = P d i=1 ✓ n (i)f i which belongs to C. Moreover we have A reg (T n , f ⇤ ) = B(✓ n ) and min f 2C A reg (f, f ⇤ ) = min ✓2 d B(✓).
Consequently, for the oracle quad and the class G quad Proposition 9 implies that -For

" ⇤ n (G quad , d , quad ) c⇣ n (d). ( 4
p n  d  c 0 exp(n) sup 2 " ⇤ n (S quad , d , ) c s 1 n log ⇣ d p n + 1 ⌘ .
We note that without assumption on d the lower-bound for the class of quadratic functions is of order O(1/n) but in high-dimensional settings it becomes of order (1/ p n). Nevertheless we will see in the next section this lower-bound is always of order (1/ p n) for the class of linear functions.

4.F.4 Aggregation in Classification and Application to Oracle Complexity of Stochastic Linear Optimization

We consider now the classification problem with the hinge loss for which Y = { 1, 1}. We follow very closely the framework of [START_REF] Lecué | Optimal oracle inequality for aggregation of classifiers under low noise condition[END_REF][START_REF] Lecué | Optimal rates of aggregation in classification under low noise assumption[END_REF] and use their notations. We still consider random variables (X, Y ) on X ⇥ Y with probability distribution denoted by ⇡. We observe n i.i.d. pairs

D n = {(X 1 , Y 1 ), . . . , (X n , Y n )}
which follow the law ⇡ and we want to predict the label Y for any feature X 2 X by minimizing the hinge risk defined by

A cla (f ) = E max(1 Y f(X), 0),
for any measurable function f from X to R. We consider we have access to

d different estimators F = {f 1 , . . . , f d } with values in [ 1, 1].
We denote their convex hull by C = conv(f 1 , . . . , f d ). Lecué [2006, Theorem 1] and Lecué [2007, Theorem 2] provide a lower bound on this aggregation problem for classification we adapt to our specific case.

Proposition 15 ( Adaptation of Theorem 2 of [START_REF] Lecué | Optimal rates of aggregation in classification under low noise assumption[END_REF] for  = 1). Let d, n be two integers such that 2 log 2 d  n. We assume that the input space X is infinite. There exists an absolute constant c > 0, and a set of prediction rules F = {f 1 , . . . , f n } such that for any real-valued procedure T n , there exists a probability measure ⇡, for which

E Dn [A cla (T n )] min f 2C (A cla (f )) c r log d n .
Proof. Theorem 2 of [START_REF] Lecué | Optimal rates of aggregation in classification under low noise assumption[END_REF] is stated under an additional Margin assumption MAH() (see definition and notation below Eq. ( 9) in [START_REF] Lecué | Optimal rates of aggregation in classification under low noise assumption[END_REF]) on the probability distribution ⇡, i.e., there exists a constant c

0 such that E[|f (X) f ⇤ (X)|]  c 0 (A(f ) A ⇤ ) 1/ ,
for any function f on X with values in [ 1,1]. Therefore taking  ! 1, we can always consider c 0 = 2. And the constant c() in Theorem 2 of [START_REF] Lecué | Optimal rates of aggregation in classification under low noise assumption[END_REF] is

c() = c  0 (4e) 1 2 2( 1)/(2 1) (log 2) /(2 1) ,
which goes when  ! 1 to c 1 = p 2/(4e p log 2). Hence taking  ! 1 in Theorem 2 of [START_REF] Lecué | Optimal rates of aggregation in classification under low noise assumption[END_REF] implies Proposition 15. We could also have plugged arguments of the proof of Theorem 14.5 of [START_REF] Devroye | A Probabilistic Theory of Pattern Recognition[END_REF] to directly prove this result.

We relate now the problem of convex aggregation of classifiers to the problem of optimizing a linear function on the simplex. Consider the set of prediction rules F = {f 1 , . . . , f n } given by Proposition 15 and denote by F :

X ! R d , x 7 ! (f 1 (x), . . . , f d (x)). For f 2 C, there is ✓ 2 d such that f = P d i=1 ✓(i)f i and we obtain A cla (f ) = E max(1 Y hF (X), ✓i, 0).
On the other hand, when the f i are valued in [ 1, 1], the classification problem becomes equivalent to maximize the expectation EY f(X) since the hinge loss is linear on [ 1, 1]:

Y 2 { 1, 1}, f(X) 2 [ 1, 1]=)Y f(X) 2 [ 1, 1]=)E max(1 Y f(X), 0) = 1 EY f(X).
Combining both, we obtain that

A cla (f ) = 1 hE[Y F (X)], ✓i = 1 + C(✓),
where

C(✓) = hE[Y F (X)],
✓i is a linear function. This set enables us to construct a difficult subclass of linear functions

G lin = {C(✓) = hE[Y F (X)], ✓i; (X, Y ) ⇠ ⇡}.
We also define the first-order stochastic oracle lin on G lin as follows

lin (✓, f ) = ⇣ hyF (x), ✓i, yF (x) ⌘ , for (x, y) ⇠ ⇡.
As before we may optimize C with a stochastic approximation algorithm M 2 M n to obtain ✓ n 2 d and therefore build a estimator

T n = P d i=1 ✓ n (i)f i which belongs to C. Moreover we have A cla (T n ) = C(✓ n ) and min f 2C A cla (f ) = min ✓2 d C(✓).
Consequently, for the oracle lin and the class G lin Proposition 9 implies that

" ⇤ n (G lin , d , lin ) c r log d n . ( 4

4.G Lower Bounds on the Rates of Convergence of DA and MD Algorithms

Let us consider in this section that f = 0,

g(✓) = 1 2⌫ k✓ ✓ ⇤ k 2 2 and h = 1 2 k✓k 2 2 . In this case, for n 1, MD iterates (✓ md n ) verify ✓ md n = arg min ✓2R d n 1 2⌫ k✓ ✓ ⇤ k 2 2 + 1 2 k✓ ✓ md n 1 k 2 2 o . Therefore ✓ md n = ✓ ⇤ + 1 ⌫ (✓ md n 1 ✓ ⇤ ), ✓ md n ✓ ⇤ = 1 ( ⌫) n (✓ md 0 ✓ ⇤ ) and g(✓ md n ) g(✓ ⇤ ) = g(✓ md 0 ) g(✓ ⇤ ) ( ⌫) 2n
.

Whereas DA iterates (✓ da n ) satisfy

✓ da n = arg min ✓2R d n 1 2⌫ k✓ ✓ ⇤ k 2 2 + 1 2 n k✓k 2 2 o .
We compute ✓ da n = ⌫n ⌫n+1 ✓ ⇤ and

g(✓ da n ) g(✓ ⇤ ) = g(✓ da 0 ) g(✓ ⇤ ) (1 + ⌫ n) 2 .

4.H Continuous Time Interpretation of DA et MD

Following [START_REF] Nemirovsky | Problem Complexity and Method Efficiency in Optimization[END_REF], [START_REF] Bolte | Smooth optimization with approximate gradient[END_REF], [START_REF] Krichene | Accelerated mirror descent in continuous and discrete time[END_REF] we propose a continuous interpretation of these methods for g twice differentiable. We note this could be extended for g non-smooth with differential inclusions.

Derivation of the ordinary differential equation (ODE).

The first-order optimality condition of the MD iteration in Eq. (4.4)

rf (✓ n ) + rg(✓ n+1 ) + rh(✓ n+1 ) rh(✓ n ) can be rearranged as rh(✓ n+1 ) rh(✓ n ) = rf (✓ n ) rg(✓ n+1 ).
Noting @ t rh(✓) = r 2 h(✓) ✓, this is exactly a forward-backward Euler discretization of the MD ODE [START_REF] Bolte | Smooth optimization with approximate gradient[END_REF]:

✓ = r 2 h(✓) 1 [rf (✓) + rg(✓)]. (4.25)
On the other hand, considering the DA iteration in Eq. ( 4.3) we obtain

⌘ n ⌘ n 1 = rf (✓ n 1 ) and ⌘ n = n rg(✓ n ) + rh(✓ n ). (4.26)
Combining both parts in Eq. ( 4.26) leads to the single equation

n rg(✓ n ) rg(✓ n 1 ) + rg(✓ n 1 ) + rh(✓ n ) rh(✓ n 1 ) = rf (✓ n 1 ),
which is the explicit Euler discretization of the ODE @ t (trg(✓) + rh(✓)) = rf (✓).

Therefore the ODE associated to DA takes the form

✓ = r 2 (h(✓) + tg(✓)) 1 (rf (✓) + rg(✓)). (4.27)
It is worth noting that this ODE is very similar to the MD ODE in Eq. ( 4.25), with an additional term tg(✓) in the inverse mapping r 2 (h(✓) + tg(✓)) 1 which may thus slow down the DA dynamic.

Lyapunov analyzes. Lyapunov functions are used to prove convergence of the solutions of ODEs. In analogy with the discrete case, the Bregman divergence is a Lyapunov function for these ODEs [see, e.g., Bolte andTeboulle, 2003, Krichene et al., 2015] since

@ t D h (✓ ⇤ , ✓(t)) = @ t [h(✓ ⇤ ) h(✓(t)) hrh(✓(t)), ✓ ⇤ ✓(t)i] = hrh(✓(t)), ✓(t)i + hr 2 h(✓(t)) ✓(t), ✓(t) ✓ ⇤ i + hrh(✓(t)), ✓(t)i = hr 2 h(✓(t)) ✓(t), ✓(t) ✓ ⇤ i.
For the MD ODE in Eq. ( 4.25) we obtain

@ t D h (✓ ⇤ , ✓(t)) = hrf (✓(t)) + rg(✓(t)), ✓(t) ✓ ⇤ i  (✓ ⇤ ) (✓(t))
(by convexity of ).

Integrating, this yields with Jensen inequality

( ✓(t)) (✓ ⇤ )  1 t Z t 0 (✓(s)) (✓ ⇤ ) ds  D h (✓ ⇤ , ✓(0)) D h (✓ ⇤ , ✓(t)) t ,
for ✓(t) = 1 t R t 0 ✓(s)ds. This is the same convergence result as in the discrete time. For the DA ODE in Eq. ( 4.27) we obtain

@ t D h+tg (✓ ⇤ , ✓(t)) = @ t [(h + tg)(✓ ⇤ ) (h + tg)(✓(t)) hr(h + tg)(✓(t)), ✓ ⇤ ✓(t)i] = g(✓ ⇤ ) h(rh(✓(t)) + trg(✓(t))), ✓(t)i + g(✓(t)) +h@ t (rh + trg)(✓(t)), ✓(t) ✓ ⇤ i + h(r + trg)h(✓(t)), ✓(t)i = g(✓ ⇤ ) g(✓ t ) hrf (✓(t)), ✓(t) ✓ ⇤ i.
Therefore by convexity of f , @ t D h+tg (✓ ⇤ , ✓(t))  (✓ ⇤ ) (✓(t)) and we obtain

( ✓(t)) (✓ ⇤ )  D h (✓ ⇤ , ✓(0)) D h+tg (✓ ⇤ , ✓(t)) t .
The continuous time argument really mimics the proof of Proposition 8 without the technicalities associated with the discrete time. We remind that we recover the variational interpretation of [START_REF] Krichene | Accelerated mirror descent in continuous and discrete time[END_REF], [START_REF] Wibisono | A variational perspective on accelerated methods in optimization[END_REF], [START_REF] Wilson | A Lyapunov analysis of momentum methods in optimization[END_REF]: the Lyapunov function generates the dynamic in the sense that a function L is first chosen and secondly a dynamics, for which L is a Lyapunov function, is then designed. In this way MD and DA are the two different dynamics associated to the two different Lyapunov functions D h and D h+tg .

Extension to the noisy-gradient case. We consider now we only have access to noisy estimates of the gradient as in Section 4.3 and propose a continuous-time interpretation of these stochastic methods. Stochastic MD and SDA may be viewed, in their primal-dual forms, as discretizations of the following stochastic differential When ⌃ = E(x⌦x) is a covariance matrix as in Section 4.3.2, h↵, ⌃↵i = Ehx, ↵i 2  Ekxk 2 q k↵k 2 p by Hölder inequality, and Proposition 17 admits the following corollary.

Corollary 7. Assume that f (✓) = 1 2 E(hx, ✓i y) 2 , h(✓) = 1 2 k✓k 2 p and q such that 1/p + 1/q = 1. Then h f is convex for any constant step-size such that

 1/Ekxk 2 q .
Therefore we may use the algorithm with bigger step-size than in the Euclidean case. Moreover when the algorithm is started from ✓ 0 = 0, the Bregman divergence is

D h (✓ ⇤ , ✓ 0 ) = 1 2(p 1) k✓ ⇤ k 2
p and the bias in Proposition 8 would be bounded by

Ekxk 2 q k✓⇤k 2 p 2(p 1) . For high-dimension problems, taking q = 1 + log(d) (with p ⇠ 1 and q ⇠ +1) yields to bounds depending on the `1-norm of the optimal predictor and the `1-norm of the features which is advisable for sparse problems.

4.J Proof of Proposition 17

We consider here h

(✓) = 1 2(p 1) k✓k 2 p . For ✓ 2 R d , h is twice differentiable. Its gradient is r i h(✓) = sign(✓(i))|✓(i)| p 1 (p 1)k✓k p 2 p ,
and its Hessian may be written for ↵ = 2 p (p 1) k✓k

2(p 1) p , u(i) = k✓k 2 p p ✓(i) p 2 and v(i) = ✓(i) p 1 for i = 1, . . . , d, as r 2 h(✓) = Diag(u) + ↵vv > , The function h f is convex if and only if r 2 h(✓) 4 ⌃ for all ✓ 2 R d . This condition is equivalent to min ✓ min ↵ h↵, r 2 h(✓)↵i h↵, ⌃↵i .
A sufficient condition is that Diag u < ⌃. After a change of variables, u may be written as

u(i) = ⌘(i) p 2 where ⌘(i) = |✓(i)|/k✓k p satisfies P d i=1 ⌘(i) p = 1 and ⌘(i) 0. Hence for all ✓, ↵ 2 R d h↵, r 2 h(✓)↵i d X i=1 ↵(i) 2 u(i) = d X i=1 ↵(i) 2 ⌘(i) p 2 , which implies min ✓2R d h↵, r 2 h(✓)↵i min ⌘2R d d X i=1 ↵(i) 2 ⌘(i) p 2 such that d X i=1 ⌘(i) p = 1 and ⌘(i) 0.
This optimization problem is equivalent with v(i) = ⌘(i) p to the one the simplex d min

v2R d d X i=1 ↵(i) 2 v(i) 1 2/p such that d X i=1 v(i) = 1 and ⌫(i) 0,
for which we define the Lagrangian L(v, , µ

) = P d i=1 ↵(i) 2 v(i) 1 2/p h , vi + ⌫(1 P d i=1 v(i)) for 2 R d + and µ 2 R. Its gradient is r v(i) L(v, , µ) = (1 2/p)↵(i) 2 /v(i) 2/p (i) ⌫.
Writing the KKT condition for this problem [see, e.g., [START_REF] Boyd | Convex Optimization[END_REF], we have that (v, , ⌫) is optimal if and only if

(1 2/p)↵(i) 2 /v(i) 2/p (i) ⌫ = 0, P d i=1 v(i) = 1 and for all i; (i) 0, v(i) 0 and (i)v(i) = 0. These conditions are satisfied by v(i) = ↵(i) p P d i=1 a(j) p , ↵(i) = 0 and ⌫ = (1 2p)( P d i=1 a(j) p ) 2/p . Hence the minimum value is d X i=1 ↵(i) 2 v(i) 1 2/p = d X i=1 ↵(i) 2 ↵(i) p 2 ( P d i=1 a(j) p ) 1 2/p = P d i=1 a(j) ( P d i=1 a(j) p ) 1 2/p = k↵k 2 p .
Consequently h↵, r 2 h(✓)↵i k↵k 2 p ,

and h f is convex for  min ↵2R d k↵k 2 p h↵,⌃↵i .

4.K Standard Benchmarks

We have considered the sido dataset which is often used for comparing largescale optimization algorithms. This is a finite binary classification dataset with finite number of observations with outputs in { 1, 1}. We have followed the following experimental protocol: (1) remove all outliers, i.e., sample points x n whose norms is greater than 5 times the average norm. ( 2) divide the dataset in two equal parts, one for training, one for testing, (3) start the algorithms from ✓ 0 = 0, (4) sample within the training dataset with replacement, for 100 times the number of observations in the training set; a dashed line marks the first effective pass in all plots, (5) compute averaged cost on training and testing data based on 10 replications. All cost are shown in log-scale, normalized to that the first iteration leads to (✓ 0 ) (✓ ⇤ ) = 1. We solved a `1-regularized least-squares regression for three different values of `1regularization: (1) one with the ⇤ which corresponds to the best generalization error after 500 effective passes through the train set, (2) one with ⇤ /8 and (3) one with 256 ⇤ .

We compare five algorithms: averaged SGD with constant step-size, average SGD with decreasing step-size C/(R 2 p n), SDA with constant step-size, SDA with decreasing step-size C/(R 2 p n) and SAGA with constant step-size [START_REF] Defazio | SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives[END_REF], which showed state-of-the-art performance in the set-up of finite data sets. We consider the theoretical value of step-size which ensures convergence. We note the behaviors are comparable to the situation where step-sizes with the best testing error after one effective pass through the data (testing powers of 4 times the theoretical step-size) are used.

We can make the following observations:

-We show results for = ⇤ in Figure 4.K.1. SAGA, constant-step-size SDA and constant-step-size SGD exhibit the best behavior for both settings of stepsize. However the training error of SGD does not converge to 0. On the other hand, SGD and SDA with step-size decaying as C/R 2 p n are slower. SAGA and constant-step-size SDA exhibit some overfitting after more than 10 passes on the regularized objective .

-We show results for = ⇤ /8 in Figure 4.K.2. The problem is then very little regularized and the behavior of constant-step-size SGD gets closer to constantstep-size SDA. There is here still overfitting for the regularized objective . -We show results for = 256 ⇤ in Figure 4.K.1. The problem is then much more regularized. In this case the regularization has an important weight and the stochasticity of the quadratic objective plays a minor role. Therefore SAGA exhibits the best behavior, despite strong early oscillations, with a linear convergence but reaches a saturation point after few passes over the data. On the other hand, constant-step-size SDA exhibits a sublinear convergence which is faster at the beginning and catches up with SAGA at the end. Constant-stepsize SGD is not converging to the solution. To conclude, constant-step-size SDA behaves similarly to SAGA which is specially dedicated to the set-up of finite data sets. For larger datasets, where only a single pass is possible, SAGA could not be run. Moreover SAGA does not come with generalization guarantees while SDA does (if a single pass is made). log 10 (n sido square C=1 test Chapter 5

0 1 2 3 4 5 log 10 (n) -1.8 -1.6 -1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 log 10 [psi( θ)-psi( θ * )] sido square C=1 test SGD C/R 2 SGD C/R 2 n 1/2 SDA C/R 2 SDA C/R 2 n 1/2 SAGA 0 1 2 3 4 5 log 10 (n) -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 log 10 [psi( θ)-psi( θ * )] sido square C=1 train SGD C/R 2 SGD C/R 2 n 1/2 SDA C/R 2 SDA C/R 2 n 1/2 SAGA
) -1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 log 10 [psi( θ)-psi( θ * )] sido square C=1 test SGD C/R 2 SGD C/R 2 n 1/2 SDA C/R 2 SDA C/R 2 n 1/2 SAGA 0 1 2 3 4 5 log 10 (n) -3 -2.5 -2 -1.5 -1 -0.5 0 log 10 [psi( θ)-psi( θ * )] sido square C=1 train SGD C/R 2 SGD C/R 2 n 1/2 SDA C/R 2 SDA C/R 2 n 1/2 SAGA
SGD C/R 2 SGD C/R 2 n 1/2 SDA C/R 2 SDA C/R 2 n 1/2 SAGA 0 1 2 3 4 5 log 10 (n) -5 -4.5 -4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 log 10 [psi( θ)-psi( θ * )] sido square C=1 train SGD C/R 2 SGD C/R 2 n 1/2 SDA C/R 2 SDA C/R 2 n 1/2 SAGA

Application to Discriminative Clustering

Abstract Clustering high-dimensional data often requires some form of dimensionality reduction, where clustered variables are separated from "noise-looking" variables. We cast this problem as finding a low-dimensional projection of the data which is wellclustered. This yields a one-dimensional projection in the simplest situation with two clusters, and extends naturally to a multi-label scenario for more than two clusters. In this chapter, (a) we first show that this joint clustering and dimension reduction formulation is equivalent to previously proposed discriminative clustering frameworks, thus leading to convex relaxations of the problem; (b) we propose a novel sparse extension, which is still cast as a convex relaxation and allows estimation in higher dimensions; (c) we propose a natural extension for the multi-label scenario; (d) we provide a new theoretical analysis of the performance of these formulations with a simple probabilistic model, leading to scalings over the form d = O( p n) for the affine invariant case and d = O(n) for the sparse case, where n is the number of examples and d the ambient dimension; and finally, (e) we propose an efficient iterative algorithm with running-time complexity proportional to O(nd 2), improving on earlier algorithms for discriminative clustering with the square loss, which had quadratic complexity in the number of examples.

This chapter is extracted from the paper Robust Discriminative Clustering with Sparse Regularizers, in collaboration with B. Palaniappan and F. Bach, accepted in the Journal of Machine Learning Research.

Introduction

Clustering is an important and commonly used pre-processing tool in many machine learning applications, with classical algorithms such as K-means [START_REF] Macqueen | Some methods for classification and analysis of multivariate observations[END_REF], linkage algorithms [START_REF] Gower | Minimum spanning trees and single Linkage cluster analysis[END_REF] or spectral clustering [START_REF] Ng | On spectral clustering: Analysis and an algorithm[END_REF]. In high dimensions, these unsupervised learning algorithms typically have problems identifying the underlying optimal discrete nature of the data; for example, they are quickly perturbed by adding a few noisy dimensions. Clustering highdimensional data thus requires some form of dimensionality reduction, where clustered variables are separated from non-informative "noise-looking" (e.g., Gaussian) variables.

Several frameworks aim at linearly separating noise from signal, that is finding projections of the data that extracts the signal and removes the noise. They differ in the ways signals and noise are defined. A line of work that dates back to projection pursuit [START_REF] Friedman | Projection pursuit regression[END_REF] and independent component analysis [START_REF] Hyvärinen | Independent Component Analysis[END_REF] defines the noise as Gaussian while the signal is non-Gaussian [START_REF] Blanchard | In search of non-Gaussian components of a high-dimensional distribution[END_REF][START_REF] Roux | Local component analysis[END_REF][START_REF] Diederichs | Sparse non-Gaussian component analysis by semidefinite programming[END_REF]. In this work, we follow De la Torre and Kanade [2006], [START_REF] Ding | Adaptive dimension reduction using discriminant analysis and Kmeans clustering[END_REF], along the alternative route where one defines the signal as being clustered while the noise is any nonclustered variable. In the simplest situation with two clusters, we may project the data into a one-dimensional subspace. Given a data matrix X 2 R n⇥d composed of n d-dimensional points, the goal is to find a direction w 2 R d such that Xw 2 R n is well-clustered, e.g., by K-means. This is equivalent to identifying both a direction to project, represented as w 2 R d and the labeling y 2 { 1, 1} n that represents the partition into two clusters.

Most existing formulations are non-convex and typically perform a form of alternating optimization [De la [START_REF] De La Torre | Discriminative cluster analysis[END_REF]Kanade, 2006, Ding and[START_REF] Ding | Adaptive dimension reduction using discriminant analysis and Kmeans clustering[END_REF], where given y 2 { 1, 1} n , the projection w is found by linear discriminant analysis (or any binary classification method), and given the projection w, the clustering is obtained by thresholding Xw or running K-means on Xw. As shown in Section 5.2, this alternating minimization procedure happens to be equivalent to maximizing the (centered) correlation between y 2 { 1, 1} n and the projection Xw 2 R d , that is

max w2R d ,y2{ 1,1} n (y > ⇧ n Xw) 2 k⇧ n yk 2 2 k⇧ n Xwk 2 2 ,
where

⇧ n = I n 1 n 1 n 1 >
n is the usual centering projection matrix (with 1 n 2 R n being the vector of all ones, and I n the n ⇥ n identity matrix). This correlation is equal to one when the projection is perfectly clustered (independently of the number of elements per cluster). Existing methods are alternating minimization algorithms with no theoretical guarantees.

In this chapter, we relate this formulation to discriminative clustering formulations [Xu et al., 2004, Bach and[START_REF] Bach | DIFFRAC : a discriminative and flexible framework for clustering[END_REF], which consider the problem

min v2R d , b2R, y2{ 1,1} n 1 n ky Xv b1 n k 2 2 , (5.1)
with the intuition of finding labels y which are easy to predict by an affine function of the data. In particular, we show that given the relationship between the number of positive labels and negative labels (i.e., the squared difference between the respective number of elements), these two problems are equivalent, and hence discriminative clustering explicitly performs joint dimension reduction and clustering.

While the discriminative framework is based on convex relaxations and has led to interesting developments and applications [START_REF] Zhang | Maximum margin clustering made practical[END_REF][START_REF] Li | Tighter and convex maximum margin clustering[END_REF], Joulin et al., 2010a[START_REF] Wang | Linear time maximum margin clustering[END_REF][START_REF] Niu | Maximum volume clustering: a new discriminative clustering approach[END_REF][START_REF] Huang | Maximin separation probability clustering[END_REF], it has several shortcomings when used with the square loss: (a) the running-time complexity of the semi-definite formulations is at least quadratic in n, and typically much more, (b) no theoretical analysis has ever been performed, (c) no convex sparse extension has been proposed to handle data with many irrelevant dimensions, (d) balancing of the clusters remains an issue, as it typically adds an extra hyperparameter which may be hard to set. In this chapter, we focus on addressing these concerns.

When there are more than two clusters, one considers either the multi-label or the multi-class settings. The multi-class problem assumes that the data are clustered into distinct classes, i.e., a single class per observation, whereas the multi-label problem assumes the data share different labels, i.e., multiple labels per observation. We show in this work that discriminative clustering framework extends more naturally to multi-label scenarios and that this extension has the same convex relaxation.

A summary of the contributions of this chapter follows:

-In Section 5.2, we relate discriminative clustering with the square loss to a joint clustering and dimension reduction formulation. The proposed formulation takes care of the balancing hyperparameter implicitly.

-We propose in Section 5.3 a novel sparse extension to discriminative clustering and show that it can still be cast through a convex relaxation.

-When there are more than two clusters, we extend naturally the sparse formulation to a multi-label scenario in Section 5.4.

-We then proceed to provide a theoretical analysis of the proposed formulations with a simple probabilistic model in Section 5. ), the first to be linear in the number of observations n for discriminative clustering with the square loss.

Throughout this chapter we assume that X 2 R n⇥d is centered, a common preprocessing step in unsupervised (and supervised) learning. This implies that X > 1 n = 0 and ⇧ n X = X.

Joint Dimension Reduction and Clustering

In this section, we focus on the single binary label case, where we first study the usual non-convex formulation, before deriving convex relaxations based on semidefinite programming. Some of the following results are already known in the literature however we state them here for completeness.

Non-Convex Formulation

Following De la Torre and Kanade [2006], [START_REF] Ding | Adaptive dimension reduction using discriminant analysis and Kmeans clustering[END_REF], [START_REF] Ye | Discriminative k-means for clustering[END_REF], we consider a cost function which depends on y 2 { 1, 1} n and w 2 R d , which is such that alternating optimization is exactly (a) running K-means with two clusters on Xw to obtain y given w (when we say "running K-means", we mean solving the vector quantization problem exactly), and (b) performing linear discriminant analysis to obtain w given y.

Proposition 18 (Joint clustering and dimension reduction for two clusters). Given X 2 R n⇥d such that X > 1 n = 0 and X has rank d, consider the optimization problem

max w2R d ,y2{ 1,1} n (y > Xw) 2 k⇧ n yk 2 2 kXwk 2 2 .
(5.2)

Given y, the optimal w is obtained as w = (X > X) 1 X > y, while given w, the optimal y is obtained by running K-means on Xw.

This equivalence might be straightforward however it has not been precisely stated in the literature to the best of our knowledge.

Proof. Given y, we need to optimize the Rayleigh quotient w > X > yy > Xw w > X > Xw with a rankone matrix in the numerator, which leads to w = (X > X) 1 X > y. Given w, we show in Appendix 5.A, that the averaged distortion measure of K-means once the means have been optimized is exactly equal to (y > Xw) 2 /k⇧ n yk 2 2 .

Algorithm. The proposition above leads to an alternating optimization algorithm. Note that K-means in one dimension may be run exactly in O(n log n) [START_REF] Bellman | A note on cluster analysis and dynamic programming[END_REF].

After having optimized with respect to w in Eq. ( 5.2), we then need to maximize with respect to y the function y > X(X > X) 1 X > y k⇧nyk 2

2

, which happens to be exactly performing Kmeans on the whitened data (which is now in high dimension and not in 1 dimension). At first, it seems that dimension reduction is simply equivalent to whitening the data and performing K-means; while this is a formally correct statement, the resulting K-means problem is not easy to solve as the clustered dimension is hidden in noise; for example, algorithms such as K-means++ [START_REF] Arthur | K-means++: The advantages of careful seeding[END_REF], which have a multiplicative theoretical guarantee on the final distortion measure, are not provably effective here because the minimal final distortion is not small (since the clusters are corrupted by some noisy dimensions), and the multiplicative guarantee is then meaningless.

Convex Relaxation and Discriminative Clustering

The discriminative clustering formulation in Eq. ( 5.1) may be optimized for any

y 2 { 1, 1} n in closed form with respect to b as b = 1 > n (y Xv) n = 1 > n y
n since X is centered. Substituting b in Eq. ( 5.1) leads us to

min v2R d 1 n k⇧ n y Xvk 2 2 = 1 n k⇧ n yk 2 2 max w2R d (y > Xw) 2 kXwk 2 2 , (5.3)
where v is obtained from any solution w as v = w y > Xw kXwk 2

2

. Thus, given

(y > 1 n ) 2 n 2 = 1 n 2 #{i, y i = 1} #{i, y i = 1} 2 = ↵ 2 [0, 1], (5.4)
which characterizes the asymmetry between clusters and with k⇧ n yk 2 = n(1 ↵), we obtain from Eq. ( 5.3), an equivalent formulation to Eq. ( 5.2) (with the added constraint) as

min y2{ 1,1} n , v2R d 1 n k⇧ n y Xvk 2 2 such that (y > 1 n ) 2 n 2 = ↵. (5.5)
This is exactly equivalent to a discriminative clustering formulation with the square loss [START_REF] Bach | DIFFRAC : a discriminative and flexible framework for clustering[END_REF] with an explicit cluster balance constraint. Consequently we have formally established that the discriminative clustering formulation in Eq. ( 5.5) is related to the joint clustering and dimension reduction formulation in Eq. ( 5.2). Following [START_REF] Bach | DIFFRAC : a discriminative and flexible framework for clustering[END_REF], we may optimize Eq. ( 5.5) in closed form with respect to v as v = (X > X) 1 X > y. Substituting v in Eq. ( 5.5) leads us to

min y2{ 1,1} n 1 n y > ⇧ n X(X > X) 1 X > y such that (y > 1 n ) 2 n 2 = ↵. (5.6)
This combinatorial optimization problem is NP-hard in general [START_REF] Karp | Reducibility among combinatorial problems[END_REF][START_REF] Garey | Some simplified NP-complete graph problems[END_REF]. Hence in practice, it is classical to consider the following convex relaxation of Eq. (5.6) [START_REF] Luo | Semidefinite relaxation of quadratic optimization problems[END_REF]. For any admissible y 2 { 1, +1} n , the matrix Y = yy > 2 R n⇥n is a rank-one symmetric positive semi-definite matrix with unit diagonal entries and conversely any such Y may be written in the form Y = yy > such that y is admissible for Eq. (5.6). Moreover by rewriting Eq. (5.6) as

min y2{ 1,1} n 1 n tr yy > ⇧ n X(X > X) 1 X > such that 1 > n (yy > )1 n n 2 = ↵,
we see that the objective and constraints are linear in the matrix Y = yy > and Eq. ( 5.6) is equivalent to

min Y <0, rank(Y )=1 diag(Y )=1 1 n tr Y ⇧ n X(X > X) 1 X > such that 1 > n Y 1 n n 2 = ↵.
Then dropping the non-convex rank constraint leads us to the following classical convex relaxation:

min Y <0, diag(Y )=1 1 n tr Y ⇧ n X(X > X) 1 X > such that 1 > n Y 1 n n 2 = ↵. (5.7)
This is the standard (unregularized) formulation, which is cast as a semi-definite program. The complexity of interior-point methods is O(n 7

), but efficient algorithms in O(n 2) for such problems have been developed due to the relationship with the maxcut problem [START_REF] Journée | Low-rank optimization on the cone of positive semidefinite matrices[END_REF][START_REF] Wen | Block coordinate descent methods for semidefinite programming[END_REF]. We note that convex relaxation techniques are also used for semi-supervised methods [START_REF] De Bie | Convex methods for transduction[END_REF].

Given the solution Y , one may traditionally obtain a candidate y 2 { 1, 1} n by running K-means on the largest eigenvector of Y or by sampling [START_REF] Goemans | Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming[END_REF]. In this chapter, we show in Section 5.5 that it may be advantageous to consider the first two eigenvectors.

Unsuccessful Full Convex Relaxation

The formulation in Eq. ( 5.7) imposes an extra parameter ↵ that characterises the cluster imbalance. It is tempting to find a direct relaxation of Eq. (5.2). It turns out to lead to a trivial relaxation, which we outline below.

When optimizing Eq. ( 5.2) with respect to w, we obtain the following optimization problem

max y2{ 1,1} n y > X(X > X) 1 X > y y > ⇧ n y , leading to a quasi-convex relaxation as max Y <0, diag(Y )=1 tr Y X(X > X) 1 X > tr ⇧ n Y ,
whose solution is found by solving a sequence of convex problems [Boyd and Vandenberghe, 2004, Section 4.2.5]. As shown in Appendix 5.B, this may be exactly reformulated as a single convex problem:

max M <0, diag(M )=1+ 1 > M 1 n 2 tr MX(X > X) 1 X > .
Unfortunately, this relaxation always leads to trivial solutions, and we thus need to consider the relaxation in Eq. (5.7) for several values of ↵ = 1 > n Y 1 n /n 2 (and then the non-convex algorithm can be run from the rounded solution of the convex problem, using Eq. ( 5.2) as a final objective). Alternatively, we may solve the following penalized problem for several values of ⌫ > 0:

min Y <0, diag(Y )=1 1 n tr Y ⇧ n X(X > X) 1 X > + ⌫ n 2 1 > n Y 1 n . (5.8)
For ⌫ = 0, Y = 1 n 1 > n is always a trivial solution. As outlined in our theoretical section and as observed in our experiments, it is sufficient to consider ⌫ 2 [0, 1].

By convex duality [Borwein and Lewis, 2000, Sec. 4.3], both constrained relaxation in Eq. ( 5.7) and penalized relaxation in Eq. ( 5.8) are formally equivalent for specific choice of constraint parameter ↵ and penalization parameter ⌫. We will see in Section 5.6 that the formulation in Eq. ( 5.8) is more suitable for algorithmic design [START_REF] Bach | Optimization with sparsityinducing penalties[END_REF].

Equivalent Relaxations

Optimizing Eq. ( 5.5) with respect to v in closed form as in Section 5.2.2 is feasible with no regularizer or with a quadratic regularizer. However, if one needs to add more complex regularizers, we need a different relaxation. Therefore we now propose a new formulation of the discriminative clustering framework. We start from the penalized version of Eq. (5.5),

min y2{ 1,1} n , v2R d 1 n k⇧ n y Xvk 2 2 + ⌫ (y > 1 n ) 2 n 2 , (5.9)
which we expand as:

min y2{ 1,1} n , v2R d 1 n tr ⇧ n yy > 2 n tr Xvy > + 1 n tr X > Xvv > + ⌫ (y > 1 n ) 2 n 2 , (5.10)
and relax as, using Y = yy > , P = yv > and V = vv > , min

V,P,Y 1 n tr ⇧ n Y 2 n tr P > X + 1 n tr X > XV + ⌫ 1 > n Y 1 n n 2 s.t. ✓ Y P P > V ◆ < 0, diag(Y ) = 1.
(5.11)

When optimizing Eq. ( 5.11) with respect to V and P , we get exactly Eq. (5.8). Indeed, the optimum is attained for V =(X > X) 1 X > Y X(X > X) 1 and P=Y X(X > X) 1 as shown in Appendix 5.C.1. Therefore, the convex relaxation in Eq. (5.11) is equivalent to Eq. (5.8).

However, we get an interesting behavior when optimizing Eq. ( 5.11) with respect to P and Y also in closed form. For ⌫ = 1, we obtain, as shown in Appendix 5.C.2, the following closed form expressions:

Y = Diag(diag(XV X > )) 1/2 XV X > Diag(diag(XV X > )) 1/2 P = Diag(diag(XV X >
)) 1/2 XV, leading to the problem:

min V <0 1 2 n n X i=1 p (XV X > ) ii + 1 n tr(V X > X).
(5.12)

The formulation above in Eq. ( 5.12) is interesting for several reasons: (a) it is formulated as an optimization problem in V 2 R d⇥d , which will lead to algorithms whose running time will depend on n linearly (see Section 5.6), (b) it allows for easy adding of regularizers (see Section 5.3), which may be formulated as convex functions of V = vv > . At first sight this seems to be valid only for ⌫ = 1. However we now propose a reformulation which can handle all possible ⌫ 2 [0, 1) through a simple data augmentation.

Reformulation for any ⌫. When ⌫ 2 [0, 1), we may reformulate the objective function in Eq. ( 5.9) as follows:

k⇧ n y Xvk 2 2 + ⌫ (y > 1 n ) 2 n = k⇧ n y Xv + ⌫ y > 1 n n 1 n k 2 2 ⌫y > 1 n 2 n + ⌫ y > 1 n 2 n = ky Xv (1 ⌫) y > 1 n n 1 n k 2 2 + ⌫ (1 ⌫)y > 1 n 2 (1 ⌫)n = min b2R ky Xv b1 n k 2 2 + ⌫n 1 ⌫ b 2 , (5.13) since ky Xv b1 n k 2 2 + ⌫n 1 ⌫ b 2 can be optimized in closed form with respect to b as b = (1 ⌫) y > 1n n . Note that the weighted imbalance ratio (1 ⌫) y > 1n n
is made as an optimization variable in Eq. (5.13). Thus we have the following reformulation

min v2R d , y2{ 1,1} n 1 n k⇧ n y Xvk 2 2 + ⌫ (y > 1 n ) 2 n 2 = min v2R d ,b2R, y2{ 1,1} n 1 n ky Xv b1 n k 2 2 + ⌫ 1 ⌫ b 2 , (5.14)
which is a non-centered penalized formulation on a higher-dimensional problem in the variable v b 2 R d+1 . In the rest of the chapter, we will focus on the case ⌫ = 1 for ease of exposition. This enables the use of the formulation in Eq. ( 5.12), which is easier to optimize. It is worth noting that this is not an algorithmic restriction. Of course any problem with ⌫ 2 [0, 1) can be treated with equal ease by adding a constant term and a quadratic regularizer.

Regularization

There are several natural possibilities. We consider norms ⌦ such that ⌦(w) 2 = (ww >

) for a certain convex function ; all norms have that form [Bach et al., 2012, Proposition 5.1]. When ⌫ = 1, Eq. ( 5.12) then becomes

max V <0 2 n n X i=1 p (XV X > ) ii 1 n tr(V X > X) (V ).
(5.15)

The quadratic regularizers (V ) = tr ⇤V have already been tackled by [START_REF] Bach | DIFFRAC : a discriminative and flexible framework for clustering[END_REF]. They consider the regularized version of problem in Eq. ( 5.3) (5.16) optimize in closed form with respect to v as v = (X > X + n⇤) 1 X > y. Substituting v in Eq. ( 5.16) leads them to

min v2R d 1 n k⇧ n y Xvk 2 2 + v > ⇤v,
min Y <0, diag(Y )=1 1 n tr Y ⇧ n X(X > X + n⇤) 1 X .
In this chapter, we propose a novel sparse extension to discriminative clustering framework with the square loss. Specifically we formulate a non-trivial sparse regularizer which is a combination of weighted squared `1-norm and `2-norm. It leads to

(V ) = tr[Diag(a)V Diag(a)] + k Diag(c)V Diag(c)k 1 , (5.17)
such that (vv > ) =

P d i=1 a 2 i v 2 i + P d i=1 c i |v i | 2 .
This allows to treat all situations simultaneously, with ⌫ = 1 or with ⌫ 2 [0, 1). To be more precise, when ⌫ 2 [0, 1), we can consider in Eq. ( 5.14), a problem of size d + 1 with a design matrix [X, 1 n ] 2 R n⇥(d+1) , a direction of projection v b 2 R d+1 and different weights for the last variable with a d+1 = ⌫ 1 ⌫ and c d+1 = 0.

Note that the sparse regularizers on V introduced in this chapter are significantly different when compared to the sparse regularizers on variable v in Eq. ( 5.3), for example, considered by [START_REF] Wang | Multi-view clustering and feature learning via structured sparsity[END_REF]. A straightforward sparse regularizer on v in Eq. ( 5.3), despite leading to a sparse projection, does not yield natural generalizations of the discriminative clustering framework in terms of theory or algorithms.

In our analysis and experiments for the balanced clusters (when ⌫ = 1), the sparse regularization = k • k 1 , for 2 R will often be considered. This is equivalent to setting a = 0 d and c = p 1 d in Eq. ( 5.17). The problem in Eq. ( 5.15) then becomes

max V <0 2 n n X i=1 p (XV X > ) ii 1 n tr(V X > X) kV k 1 .
(5.18)

The sparse regularizers considered in this chapter have a significant algorithmic appeal for certain applications in computer vision [START_REF] Bojanowski | Finding actors and actions in movies[END_REF][START_REF] Alayrac | Unsupervised learning from narrated instruction videos[END_REF], audio processing [START_REF] Lajugie | A weakly-supervised discriminative model for audio-to-score alignment[END_REF] and natural language processing [START_REF] Grave | A convex relaxation for weakly supervised relation extraction[END_REF]. They also lead to robust cluster recovery under minor assumptions as will be illustrated on a simple example in Section 5.5. The practical benefits of the sparse regularizers will be further demonstrated using empirical evaluation on synthetic and real data sets in Section 5.7.

Extension to Multiple Labels

The discussion so far has focussed on two clusters. Yet it is key in practice to tackle more clusters. It is worth noting that the discrete formulations in Eq. ( 5.2) and Eq. ( 5.5) extend directly to more than two clusters. However two different extensions of the initial problems Eq. ( 5.2) or Eq. (5.5) are conceivable. They lead to problems with different constraints on different optimization domains and, consequently, to different relaxations. We discuss these possibilities next.

One extension is the multi-class case. The multi-class problem which is dealt with by [START_REF] Bach | DIFFRAC : a discriminative and flexible framework for clustering[END_REF] assumes that the data are clustered into K classes and the various partitions of the data points into clusters are represented by the Kclass indicator matrices y 2 {0, 1} n⇥K such that y1 K = 1 n . The constraint y1 K = 1 n ensures that one data point belongs to only one cluster. However as discussed by [START_REF] Bach | DIFFRAC : a discriminative and flexible framework for clustering[END_REF], by letting Y = yy > , it is possible to lift these K-class indicator matrices into the outer convex approximations [START_REF] Frieze | Improved approximation algorithms for MAX k-CUT and MAX BISECTION[END_REF], which is different for all values of K. Note that letting K = 2 corresponds to the previous sections.

C K = {Y 2 R n⇥n : Y = Y > , diag(Y ) = 1 n , Y < 0, Y 4 1 K 1 n 1 > n } [
In this chapter, we consider a different novel extension for discriminative clustering to the multi-label case. The multi-label problem assumes that the data share k labels and the data-label membership is represented by matrices y 2 { 1, +1} n⇥k . In other words, the multi-class problem embeds the data in the extreme points of a simplex, while the multi-label problem does so in the extreme points of the hypercube.

The discriminative clustering formulation of the multi-label problem is

min v2R d⇥k , y2{ 1,1} n⇥k 1 n k⇧ n y Xvk 2 F , (5.19)
where the Frobenius norm is defined for any vector or rectangular matrix as kAk 2 F = tr AA > = tr A > A. Letting k = 1 here corresponds to the previous sections. The discrete ensemble of matrices y 2 { 1, +1} n⇥k can be naturally lifted into

D k = {Y 2 R n⇥n : Y = Y > , diag(Y ) = k1 n , Y < 0}, since diag(Y ) = diag(yy > ) = P k i=1 y 2 i,i = k.
As the optimization problems in Eq. ( 5.7) and Eq. ( 5.8) have linear objective functions, we can change the variable from Y to Ỹ = Y/k to change the constraint diag(Y ) = k1 n to diag( Ỹ ) = 1 n without changing the optimizer of the problem. Thus the problems can be solved over the relaxed domain

D = {Y 2 R n⇥n : Y = Y > , diag(Y ) = 1 n , Y < 0} which is independent of k.
Note that the domain D is similar to that considered in the problems in Eq. (5.8) and Eq. ( 5.11) and these convex relaxations are the same regardless of the value of k. Hence the multi-label problem is a more natural extension of the discriminative framework, with a slight change in how the labels y are recovered from the solution Y (we discuss this in Section 5.5.3).

However the problem in Eq. ( 5.8) should have a unique solution in order to always recover the correct assignment y. Unfortunately the semidefinite constraint Y < 0 of the relaxation makes the second-order information arduous to study. Due to this reason, we consider the other equivalent relaxation in Eq. ( 5.12) for which V ⇤ = vv > is also solution with v / (X > X) 1 X > y (see Lemma 36 in Appendix 5.D.3). Fortunately the semidefinite constraint V < 0 of the problem in Eq. ( 5.12) may be ignored since the second-order information in V of the objective function already provides unicity for the unconstrained problem. Hence we are able to ensure the uniqueness of the solution with high probability.

Proposition 19. Let us assume d 3, > 1 and m 2 3 2(d+ 4) : (a) If n d 2 R 4 1+(d+ )m 2 m 2
( 1) , V ⇤ is the unique solution of the problem in Eq. ( 5.12) with high probability.

(b) If n d 2 R 4 min{m 2
( 1),2m 2 ,2m} , v is the principal eigenvector of any solution of the problem in Eq. ( 5.12) with high probability.

Let us make the following observations: -Proof technique: The proof relies on a computation of the Hessian of f (V ) =

2 n P n i=1 p (XV X > ) ii
1 n tr X > XV which is the objective function in Eq. ( 5.12). We first derive the expectation of r 2 f (V ) with respect to the distribution of X. By the law of large numbers, it amounts to have n going to infinity in r 2 f (V ). Then we expand the spectrum of this operator Er 2 f (V ) to lower-bound its smallest eigenvalue. Finally we use concentration theory on matrices, following [START_REF] Tropp | User-friendly tail bounds for sums of random matrices[END_REF], to bound the Hessian r 2 f (V ) for finite n.

-Effect of kurtosis: We remind that > 1, with equality if and only if z follows a Rademacher law (P(z = +1) = P(z = 1) = 1/2). Thus, if the noisy dimensions are clustered, then unsurprisingly, our guarantee is meaningless. Note that the constant behaves like a distance of the distribution z to the Rademacher distribution. Moreover, = 3 if z follows a standard normal distribution.

-Scaling between d and n: If the noisy variables are not evenly clustered between the same clusters {±1} (i.e., > 1), we recover a rank-one solution as long as n = O(d 3 ); while, as long as n = O(d 2 ), the solution is not unique but its principal eigenvector recovers the correct clustering. Moreover, as explained in the proof, its spectrum would be very spiky.

-The assumption m 2 3 2(d+ 4) is generally satisfied for large dimensions. Note that m 2 d is the total variance of the irrelevant dimensions, and when it is small, i.e., when m 2 

[2006] and Gaussian mixtures by [START_REF] Kalai | Efficiently learning mixtures of two Gaussians[END_REF], [START_REF] Moitra | Settling the polynomial learnability of mixtures of Gaussians[END_REF], where separation conditions between the two clusters are derived, under which the clustering problem is efficiently solved. It would be of great interest to relate these separation conditions to our condition on n and d but this is outside the scope of this work.

Noise Robustness for the 1-Dimensional Balanced Problem

We assume now that the data are one-dimensional and are perturbed by some noise " 2 R n such that X = y + " with y 2 { 1, 1} n . The solution of the relaxation in Eq. ( 5.8) recovers the correct y in this setting only when each component of y and y + " have the same sign (this is shown in Appendix 5.D.5). This result comes out naturally from the information on whether the signs of y and y + " are the same or not. Further if we assume that y and " are independent, this condition is equivalent to k"k 1 < 1 almost surely.

Unbalanced Problem

When the clusters are imbalanced (y > 1 n 6 = 0), the natural rank-one candidates Y ⇤ = yy > and V ⇤ = vv > are no longer solutions of the relaxations in Eq. ( 5.8) (for ⌫ = 1) and Eq. ( 5.12), as proved in Appendix 5.D.6. Nevertheless we are able to characterize some solutions of the penalized relaxation in Eq. ( 5.8) for ⌫ = 0.

Lemma 28. For ⌫ = 0 and for any non-negative a, b 2 R such that a + b = 1,

Y = ayy > + b1 n 1 > n
is solution of the penalized relaxation in Eq. (5.8).

Hence any eigenvector of this solution Y would be supported by the directions y and 1 n . Moreover when the value ↵ ⇤ = ( 1 > n y n ) 2 is known, it turns out that we can characterize some solutions of the constrained relaxation in Eq. (5.7), as stated in the following lemma.

Lemma 29. For ↵ ↵ ⇤ , Y = 1 ↵ 1 ↵ ⇤ yy > + ⇣ 1 1 ↵ 1 ↵ ⇤ ⌘ 1 n 1 > n
is a rank-2 solution of the constrained relaxation in Eq. ( 5.7) with constraint parameter ↵.

The eigenvectors of Y enable to recover y for ↵ ⇤  ↵ < 1. We conjecture (and checked empirically) that this rank-2 solution is unique under similar regimes to those considered for the balanced case. The proof would be more involved since, when ⌫ 6 = 1, we are not able to derive an equivalent problem in V for the penalized relaxation in Eq. (5.8) similar to Eq. ( 5.12) for the balanced case. We also note that Lemmas 28 and 29 will be direct consequences of Lemma 32 in Section 5.5.3. Thus Y being rank-2, one should really be careful and consider the first two eigenvectors when recovering y from a solution Y . This can be done by rounding the principal eigenvector of ⇧ n Y ⇧ n = 1 ↵ 1 ↵⇤ ⇧ n y(⇧ n y) > as discussed in the following lemma.

Lemma 30. Let y ev be the principal eigenvector of ⇧ n Y ⇧ n where Y is defined in Lemma 29, then sign(y ev ) = y.

Proof. By definition of Y , y ev = q 1 ↵ 1 ↵⇤ ⇧ n y thus sign(y ev ) = sign(⇧ n y) and since ↵  1 then sign(⇧ n y) = sign(y p ↵1 n ) = y.
In practice, contrary to the standard procedure, we should, for any ⌫, solve the penalized relaxation in Eq. ( 5.8) and then do K-means on the principal eigenvector of the centered solution ⇧ n Y ⇧ n instead of the solution Y to recover the correct y. This procedure is followed in our experiments on real-world data in Section 5.7.2.

Analysis for 2 Clusters: 1-Sparse Problems

We assume here that the direction of projection v (such that Xv = y) is l-sparse (by l-sparse we mean kvk 0 = l). The `1-norm regularized problem in Eq. ( 5.18) is no longer invariant by affine transformation and we cannot consider that X = [y, Z] without loss of generality. Yet the relaxation Eq. ( 5.18) seems experimentally to only have rank-one solutions for the simple l = 1 situation. Hence we are able to derive some theoretical analysis only for this case. It is worth noting the l = 1 case is simple since it can be solved in O(d) by using K-means separately on all dimensions and ranking them. Nonetheless the proposed scaling also holds in practice for l > 1 (see Figure 5-1b).

Thereby we consider data X = [y, Z] with y 2 { 1, 1} n and Z 2 R n⇥(d 1) which are clustered in the direction v = [1, 0, . . . , 0] > 2 R d . When adding a `1-penalty, the initial problem in Eq. (5.5) for

↵ = 0 is min y2{ 1,1} n , v2R d 1 n ky Xvk 2 2 + kvk 2 1 . (5.20)
When optimizing in v this problem is close to the Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] and a solution is known to be v

⇤ i = (y > y + n ) 1 y > y = 1 1+ , 8i 2 J and v ⇤ i = 0, 8i 2 {1, 2, . . . , d} \ J, where J is the support of v ⇤ . The candidate V ⇤ = v ⇤ v ⇤> is
still a solution of the relaxation in Eq. ( 5.18) (see Lemma 39 in Appendix 5.E.1) and we will investigate under which conditions on X this solution is unique. Let us assume as before (z i ) i=1,...,d are i.i.d. with distribution z symmetric with Ez = Ez 3 = 0, and denote by Ez 2 = m and Ez 4 /(Ez 2 ) 2 = . We also assume that kzk 1 is almost surely bounded by 0  R  1. We are able to ensure the uniqueness of the solution with high-probability. ( 1) , V ⇤ is the unique solution of the problem Eq. ( 5.12) with high probability.

(b) If n dR 2 m 2
( 1) , v ⇤ is the principal eigenvector of any solution of the problem Eq. ( 5.12) with high probability.

The proof technique is very similar to the one of Proposition 19. With the function g

(V ) = 2 n P n i=1 p (XV X > ) ii kV k 1 1
n tr X > XV , we can certify that g will decrease around the solution V ⇤ by analyzing the eigenvalues of its Hessian.

The rank-one solution V ⇤ is recovered by the principal eigenvector of the solution of the relaxation Eq. ( 5.18) as long as n = O(d). Thus we have a much better scaling when compared to the non-sparse setting where n = O(d 2). We also conjecture a scaling of order n = O(ld) for a projection in a l-sparse direction (see Figure 5-1b for empirical results).

The proposition does not state any particular value for the regularizer parameter . This makes sense since the proposition only holds for the simple situation when l = 1. We propose to use = 1/ p n by analogy with the Lasso.

Analysis for the Multi-Label Extension

In this section, the signals share k labels which are corrupted by some extra noisy dimensions. We assume the centered design matrix to be X = [⇧ n y, Z] where y 2 { 1, +1} n⇥k and Z 2 R n⇥ (d k) . We also assume that y is full-rank1 . We denote by y = [y 1 , . . . , y k ] and (5.22) As explained in Section 5.4, these two discrete problems admit the same relaxations in Eq. ( 5.7) and Eq. ( 5.8) we have studied for one label. We now investigate when the solution of the problems in Eq. ( 5.21) and in Eq. (5.22) generate solutions of the relaxations in Eq. ( 5.7) and Eq. (5.8).

↵ i = ⇣ y > i 1n n ⌘ 2 for i = 1, . . . ,
By analogy with Lemma 28, we want to characterize the solutions of these relaxations which are supported by the constant vector 1 n and the labels (y 1 , . . . , y k ). Their general form is Y = ỹAỹ > where A 2 R k⇥k is symmetric semi-definite positive and ỹ = [1 n , y]. However the initial y is easily recovered from the solution Y only when A is diagonal. To that end the following lemma derives some condition under which the only matrix A such that the corresponding Y satisfies the constraint of the relaxations in Eq. ( 5.7) and Eq. (5.8) is diagonal.

Lemma 31. The solutions of the matrix equation diag(ỹAỹ > ) = 1 n with unknown variable A are diagonal if and only if the family {1 n , (y i )

1ik , (y i y j ) 1i<jk } is linearly independent where we denoted by the Hadamard (i.e., pointwise) product between matrices.

In this way we are able to characterize the solution of relaxations in Eq. ( 5.7) and Eq. ( 5.8) with the following result:

Lemma 32. Let us assume that the family {1 n , (y i )

1ik , (y i y j ) 1i<jk } is linearly independent. If ↵ ↵ min = min 1ik {↵ i } with (↵ i ) 1ik defined above Eq. ( 5.21), the solutions of the constrained relaxation in Eq. ( 5.7) supported by the vectors (1 n , y 1 , . . . , y k ) are of the form:

Y = a 2 0 1 n 1 > n + k X i=1 a 2 i y i y > i ,
where (a i ) 0ik satisfies

P k i=0 a 2 i = 1 and a 2 0 + P k i=1 a 2 i ↵ i = ↵.
Moreover the solutions of the penalized relaxation in Eq. ( 5.8) for ⌫ = 0 which are supported by the vectors (1 n , y 1 , . . . , y k ) are of the form:

Y = a 2 0 1 n 1 > n + k X i=1 a 2 i y i y > i ,
where (a i ) 0ik satisfies

P k i=0 a 2 i = 1.
In the multi-label case, some combinations of the constant matrix 1 n 1 > n and the rank-one matrices y i y > i are solutions of constrained or penalized relaxations. Furthermore, under some assumptions on the labels (y i )

1ik , these combinations are the only solutions which are supported by the vectors (1 n , y 1 , . . . , y k ). And we conjecture (and checked empirically) that under assumptions similar to those made for the balanced one-label case, all the solutions of the relaxation are supported by the family (1 n , y 1 , . . . , y k ) and consequently share the same form as in Lemma 32. Thus the eigenvector of the solution Y would be in the span of the directions (1 n , y 1 , . . . , y k ). Let us consider an eigenvalue decomposition of Y = F F > = P k i=0 i e i e > i and denote by M = [a 0 1 n , a 1 y 1 , . . . , a k y k ] where (a i ) 0ik are defined in Lemma 32. Since MM > = F F > , there is an orthogonal transformation R such that F R = M . We also denote the product F R by F R = [⇠ 0 , . . . , ⇠ K ]. We propose now an alternating minimization procedure to recover the labels (y 1 , . . . , y k ) from M . Lemma 33. Consider the optimization problem

min M 2M, R2R k⇥k : R > R=I k kF R M k 2 F , where M = {[a 0 1 n , a 1 y 1 , . . . , a k y k ], a 2 R k+1 : kak 2 = 1, y i 2 {±1} n }.
Given M , the problem is equivalent to the orthogonal Procrustes problem [START_REF] Schönemann | A generalized solution of the orthogonal Procrustes problem[END_REF]. Denote by U V > a singular value decomposition of F > M . The optimal R is obtained as R = UV > . While given R, the optimal M is obtained as

M = 1 p k⇠ 1 k 2 1 + k⇠ 2 k 2 1 + . . . + k⇠ k k 2 1 [k⇠ 0 k 1 sign(⇠ 0 ), . . . , k⇠ k k 1 sign(⇠ k )].
Proof. We give only the argument for the optimization problem with respect to M . Given R, the optimization problem in M is equivalent to max

a2R k+1 tr(F R) > M s.t. kak 2 = 1, y 2 { 1, 1} n⇥k and tr(F R) > M = a 0 ⇠ > 0 1 n + P k i=1 a i ⇠ > i y i .
Thus by property of the dual norms the solution is given by y i = sign(⇠ i ) and a i =

k⇠ i k 1 p k⇠ 1 k 2 1 +k⇠ 2 k 2 1 +...+k⇠ k k 2 1 .
The minimization problem in Lemma 33 is non-convex; however we observe that performing few alternating optimizations is sufficient to recover the correct (y 1 , . . . , y k ) from M .

Discussion

In this section we studied the tightness of convex relaxations under simple scenarios where the relaxed problem admits low-rank solutions generated by the solution of the original non-convex problem. Unfortunately the solutions lose the characterized rank when the initial problem is slightly perturbed since the rank of a matrix is not a continuous function. Nevertheless, the spectrum of the new solution is really spiked, and thus these results are quite conservative. We empirically observe that the principal eigenvectors keep recovering the correct information outside these scenarios. However this simple proof mechanism is not easily adaptable to handle perturbed problems in a straightforward way since it is difficult to characterize the properties of eigenvectors of the solution of a semi-definite program. Hence we are able to derive a proper theoretical study only for these simple models.

Algorithms

In this section, we present an optimization algorithm which is adapted to large n settings, and avoids the n-dimensional semidefinite constraint.

Reformulation

We aim to solve the general regularized problem which correponds to Eq. ( 5.15) ).

max V <0 2 n n X i=1 p (XV X > ) ii 1 n tr V (X > X + n Diag(a) 2 ) k Diag(c)V Diag(c)k 1 . ( 5
Unbalanced case. We generate an unbalanced problem for d = 10, n = 80 and ↵ ⇤ = 0.25 and we average the results over 10 replications. We compare the clustering error for the constrained and the penalized relaxations in Eq. ( 5.7) and Eq. ( 5.8) when we consider the sign of the first or second eigenvector and when we use projection technique defined as

(⇧ n Y (2) ⇧ n ) (1)
where Y (k) is the best rank-k approximation of Y , to extract the information of y. We see in Figure 5-2 that (a) for the constrained case, the range of ↵ such that the sign of y is recovered is cut in two parts where one eigenvector is correct, whereas the projection method performs well on the whole set. (b) For the penalized case, the correct sign is recovered for ⌫ close to 0 by the first eigenvector and the projection method whereas the second one performs always badly. (c) When there is zero noise the rank of the solution is one for ↵ 2 {↵ ⇤ , 1}, two for ↵ 2 (↵ ⇤ , 1) and greater otherwise. These findings confirm our analysis. However, when y is corrupted by some noise this result is no longer true.

Runtime experiments. We generated data with a k-sparse direction of projection v by adding d k noise variables to a randomly generated and rotated k-dimension data. The scalability of the FISTA based optimization algorithm illustrated in Sec- in our framework, hinge loss is used in max-margin clustering. We have also included the behavior of K-means and alternating optimization methods in Figure 5-5 for completeness. From this plot, it is clear that the max-margin clustering is sensitive to noisy dimensions present in the data. Sparse discriminative clustering with square loss is able to maintain zero cluster error for a large number of noisy dimensions, while the performance of max-margin clustering starts deteriorating after adding a few noisy dimensions. However, we note from Figure 5-5 that for large dimensions, the hinge loss used in max-margin clustering is observed to provide a better solution than the square loss used in our framework.

Experiments on Real-World Data

Experiments on two-class data. Experiments were conducted on real two-class classification datasets2 to compare the performance of sparse discriminative clustering against non-sparse discriminative clustering, alternating optimization, K-means and max-margin clustering algorithms. For sparse and non-sparse discriminative clustering, we consider the problem in Eq. ( 5.24) and the algorithm detailed in Section 5.6.3 (with the regularization c = 0 for the non-sparse case). The alternating optimization method is described in Proposition 18. For the two-class datasets, the clustering performance for a cluster ȳ 2 {+1, 1} n obtained from an algorithm under comparison, was computed as 1 (ȳ > y/n) 2 , where y is the original labeling. Here we explicitly compare the output of clustering with the original labels of the data points.

The dataset details and clustering performance results are summarized in Table 5.1. The experiments for discriminative clustering were conducted for different values of a, c 2 {10 3 , 10 2 , 10 1 }1 d associated with the `2-regularizer and `1regularizer respectively. The range of cluster imbalance parameter was chosen to be ⌫ 2 {0.01, 0.25, 0.5, 0.75, 1}. Note that for ⌫ 6 = 1, the reformulation given in Eq. (5.14) was used, as explained in Section 5.3 after Eq. ( 5.17). The results given in Table 5.1 pertain to the best choices of these parameters. Similarly, the values of regularization parameter for max-margin clustering [START_REF] Li | Tighter and convex maximum margin clustering[END_REF] were chosen from the set {10 5 , 10 4 , 10 3 , 10 2 , 0.1, 1, 10} and the cluster balance parameter was chosen from {0.1, 0.2, . . . , 0.9}. The results for alternating optimization and K-means show the average cluster error (and standard deviation) over 10 different runs. These results show that the cluster error is quite high for many datasets. This is primarily due to the absence of an ambient low-dimensional clustering of the two-class data, which can be identified by the simple linear model presented in this chapter. Since K-means does not provide explicit dimensionality reduction, it might not be able to take advantage of the existence of an ambient low-dimensional clustering of the twoclass data and its performance is poor. The results show that max-margin clustering achieves best clustering performance on most datasets. This improved performance of max-margin clustering may be due to the use of hinge loss, as opposed to square loss used in discriminative clustering in this chapter. However for the heart dataset, we note that the sparse version with the square loss performs significantly better than the non-sparse version with the hinge loss (see additional experiments in . The results also show that adding sparse regularizers to discriminative clustering helps in a better cluster identification when compared to the non-sparse case. Experiments on real multi-label data. Experiments were also conducted on the Microsoft COCO dataset 3 to demonstrate the effectiveness of the proposed method in discovering multiple labels. We considered n = 2000 images from the dataset, each of which was labeled with a subset of K = 80 labels. The labels identified the objects in the images like person, car, chair, table, etc. and the corresponding features for each image were extracted from the last layer of a conventional convolutional neural network (CNN). The CNN was originally trained over the imagenet data [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF].

For each image in the dataset, we obtained d = 1000 features. We then performed discriminative clustering on the 2000 ⇥ 1000 data matrix X and obtained the label matrix Y which was then subjected to the alternating optimization procedure (see Section 5.5.3).

It is clearly unlikely to recover perfect labels; therefore we now describe a way of measuring the amount of information which is recovered. In order to extract meaningful cluster information from the result so-obtained, we computed the correlation matrix Y k ⇧ n Y true where Y true is the n⇥K label matrix containing actual labels and ⇧ n is the n ⇥ n centering matrix I n

1 n 1 n 1 > n .
The k predicted labels are present in the Y k matrix. In order to choose an appropriate value of k, we plotted Tr( Ytrue Y k ) (shown in Figure 5-6 along with a K-means baseline), where

Y k = Y k (Y k > Y k ) 1 Y > k .
From these plots, we chose k = 30 to be a suitable value for our interpretation purposes.

After choosing an arbitrary value of k = 30, we plotted the correlations between the actual and predicted labels. The heatmap of the normalized absolute correlations is given in Figure 567, where the columns and rows corresponding to the 80 true labels and 30 predicted labels respectively, are ordered according to the sum of squared correlations (the top-scoring labels appear to the left-bottom). From this plot, we extract following highly correlated labels: person, dining table, car, chair, cup, tennis racket, bowl, truck, fork, pizza, showing that these labels were partially recovered by our unsupervised technique (note that the CNN features are learned with supervision on the different dataset Imagenet, hence there is still some partial supervision).

Conclusion

In this chapter, we provided a sparse extension of the discriminative clustering framework, and gave a first analysis of its theoretical performance in the totally unsupervised situation, highlighting provable scalings between ambient dimension d, number of observations and "clusterability" of irrelevant variables. We also proposed an efficient algorithm which is the first of its kind to be linear in the number of observations for discriminative clustering with the square loss. Our work could be extended in a number of ways, e.g., extending the sparse analysis to l-sparse case with higher l,extending the framework to nonlinear clustering using kernels, considering related weakly supervised learning extensions [START_REF] Joulin | A convex relaxation for weakly supervised classifiers[END_REF], going beyond uniqueness of rank-one solutions, and improving the complexity of our algorithm to O(nd), for example using stochastic gradient techniques.

with a rank-one matrix in the numerator, which leads to w = (X > X) 1 X > y. Given w, we will show that the averaged distortion measure of K-means once the means have been optimized is exactly equal to (y > ⇧ n Xw) 2 /k⇧ n yk 2 2 . Given the data matrix X 2 R n⇥d , K-means to cluster the data into two components will tend to approximate the data points in X by the centroids c

+ 2 R d and c 2 R d such that X ⇡ (y + 1 n ) 2 c > + (y 1 n ) 2 c > (since y 2 { 1, 1} n ) = y 2 (c > + c > ) + 1 2 1 n (c > + + c > ).
The objective of K-means can now be written as problem KM:

min y,c + ,c X y 2 (c > + c > ) 1 2 1 n (c > + + c > ) 2 F = min y,c + ,c X (y + 1 n ) 2 c > + (1 n y) 2 c > 2 F = min y,c + ,c kXk 2 F + kc > + k 2 F (y + 1 n ) 2 2 + kc > k 2 F (1 y n ) 2 2 +2c > c + (y + 1 n ) 2 > (1 n y) 2 2 tr X > ✓ (y + 1 n ) 2 c > + + (1 n y) 2 c > ◆ = min y,c + ,c kXk 2 F + kc > + k 2 F 1 2 (n + 1 > n y) + kc > k 2 F 1 2 (n 1 > n y) 2c > + X > ✓ y + 1 n 2 ◆ 2c > X > ✓ 1 n y 2 ◆ .
Fixing y and minimizing with respect to c + and c , we get closed-form expressions for c

+ and c as

c + = X > (y + 1 n ) (n + 1 > n y) and c = X > (1 n y) (n 1 > n y)
.

Substituting these expressions in KM, we have the following optimization problem in y:

min y kXk 2 F 1 2 kX > (y + 1 n )k 2 F (n + 1 > n y) 1 2 kX > (1 n y)k 2 F (n 1 > n y) = min y kXk 2 F 1 2 tr XX > (y + 1 n )(y + 1 n ) > (n + 1 > n y) 1 2 tr XX > (1 n y)(1 n y) > (n 1 > n y) = min y kXk 2 F 2 (n + 1 > n y) tr XX > ✓ y + 1 n 2 ◆✓ y + 1 n 2 ◆ > 2 (n 1 > n y) tr XX > ✓ 1 n y 2 ◆✓ 1 n y 2 ◆ > = min y tr XX > 2 (n + 1 > n y) tr XX > ✓ y + 1 n 2 ◆✓ y + 1 n 2 ◆ > 2 (n 1 > n y) tr XX > ✓ 1 n y 2 ◆✓ 1 n y 2 ◆ > = min y tr XX > ✓ I 1 2(n + 1 > n y) (yy > + 1 n 1 > n + y1 > n + 1 n y > ) 1 2(n 1 > n y) (1 n 1 > n + yy > 1 n y > y1 > n ) ◆ .
By the centering of X, we have 1 > n X = 0 and hence tr

XX > 1 n 1 > n = tr XX > 1 n y > = tr XX > y1 > n = 0. Therefore, we obtain min y tr XX > ✓ I 1 2(n + 1 > n y) (yy > ) 1 2(n 1 > n y) (yy > ) ◆ = min y tr XX > ✓ I (yy > ) ✓ 1 2(n + 1 > n y) + 1 2(n 1 > n y) ◆◆ = min y tr XX > ✓ I (yy > ) ✓ n n 2 (1 > n y) 2 ) ◆◆ = min y tr XX > ✓ I nyy > n 2
(1 > n y) 2

◆ .

Thus we have the equivalent K-means problem as

min y2{ 1,1} n 1 n tr Xww > X > ⇣ I n n 2 (y > 1) 2 yy > ⌘ = 1 max y2{ 1,1} n (w > X > y) 2 n 2 (y > 1) 2 .
Thus the averaged distortion measure of K-means with the optimized means is

(y > ⇧nXw) 2 k⇧nyk 2 2 .

5.B Full (Unsuccessful) Relaxation

It is tempting to find a direct relaxation of Eq. (5.2). It turns out to lead to a trivial relaxation, which we outline in this section. When optimizing Eq. ( 5.2) with respect to w, we obtain max

y2{ 1,1} n y > X(X > X) 1 X > y y > ⇧ny
, leading to a quasi-convex relaxation as max

Y <0, diag(Y )=1 tr Y X(X > X) 1 X > tr ⇧nY
. Unfortunately, this relaxation always leads to trivial solutions as described below.

Consider the quasi-convex relaxation .27) By definition of ⇧ n this relaxation is equal to:

max Y <0,diag(Y )=1 tr Y X(X > X) 1 X > tr ⇧ n Y . ( 5
max Y <0,diag(Y )=1 1 n tr Y X(X > X) 1 X > 1 1 > n Y 1n n 2
.

Let A = {Y < 0, diag(Y ) = 1} the feasible set of this problem and define

B = {M < 0, diag(M ) = 1 + 1 > n M 1n n 2 }. Let Y 2 A, then M defined by M = Y 1 1 > n Y 1n n 2
belongs to

B since 1 + 1 > n M 1n n 2 = 1 + 1 > n Y 1n n 2 1 > n Y 1n = 1 1 1 > n Y 1n n 2 = diag(M ). Reciprocally for M 2 B, we can define Y = M 1+ 1 > n M 1n n 2
, such that diag(Y ) = 1 and Y 2 A and then verify that

M = Y 1 1 > n Y 1n n 2
. Thus the problem Eq. ( 5.27) is equivalent to the relaxation

max M <0,diag(M )=1+ 1 > n M 1n n 2 1 n tr MX(X > X) 1 X > .
(5.28)

Using L(A) and the psd constraint ✓ Y P P > V ◆ < 0, we write the dual problem as

min A tr AY 0 s.t. ✓ 1 n ⇧ n + ⌫ n 2 1 n 1 > n + A 1 n X 1 n X > 1 n X > X ◆ < 0.
From the Schur's complement condition of

✓ 1 n ⇧ n + ⌫ n 2 1 n 1 > n + A 1 n X 1 n X > 1 n X > X ◆ < 0, we obtain 1 n ⇧ n + ⌫ n 2 1 n 1 > n + A < 1 n X(X > X) 1 X > .
Substituting the bound for A we get the optimal objective function value

D ⇤ = 1 n tr X(X > X) 1 X > Y 0 1 n tr ⇧ n Y 0 ⌫ n 2 1 > n Y 0 1 n .
Note that the optimal dual objective value D ⇤ corresponds to a fixed Y 0 . Hence by maximizing with respect to Y we obtain exactly Eq. ( 5.8) and therefore, the convex relaxation in Eq. (5.11) is equivalent to Eq. (5.8). Moreover the Karush-Kuhn-Tucker (KKT) conditions gives

P > X + V X > X = 0 and Y X + P X > X = 0
Thus the optimum is attained for P=Y X(X > X) 1 and V = (X > X) 1 X > Y X(X > X) 1 .

5.C.2 Second Equivalent Relaxation

For ⌫ = 1, we solve the problem in Eq. ( 5.31) by fixing the matrix V = V 0 . Then the Lagrangian function of this problem can be written as

L(µ, B) = 1 n tr ⇧ n Y 2 n tr P > X + 1 n tr X > XV + ⌫ 1 > n Y 1 n n 2 +µ > (diag(Y ) 1 n ) + tr B(V V 0 ) = ✓ Y P P > V ◆ ✓ 1 n I n + diag(µ) 1 n X 1 n X > 1 n X > X + B ◆ µ > 1 n tr BV 0 .
Using L(µ, B) and the psd constraint

✓ Y P P > V ◆ < 0, the dual problem is given by min µ,B µ > 1 n + tr BV 0 s.t. ✓ 1 n I n + diag(µ) 1 n X 1 n X > 1 n X > X + B ◆ < 0.
From the Schur's complement condition of

✓ 1 n I n + diag(µ) 1 n X 1 n X > 1 n X > X + B ◆ < 0, we obtain B < 1 n 2 X > diag(µ + 1 n /n) 1 X 1 n X > X.
Substituting the bound for B we get the dual problem as .

min µ µ > 1 n + 1 n 2 tr V 0 X > diag(µ + 1 n /n) 1 X 1 n tr V 0 X > X with µ 1 m 2 (
: f (V ⇤ ) f (V ⇤ + ) > (1 )µ 2 k k 2 F + o(k k 2 ) 0, where µ 2 = min{2m 2 , m 2 (
This lemma directly implies Proposition 19.

Proof. For 2 S(d) and 2 R we compute for

f (V ) = 1 n P n i=1 p x > i V x i , d 2 d 2 f (V + ) = 1 4n n X i=1 (x > i x i ) 2 p x > i (V + )x i 3 .
Thus the second directional derivative in V = V ⇤ along is

r 2 f (V ⇤ ) = lim !0 d 2 d 2 f (V + ) = 1 4n n X i=1 (x > i x i ) 2 .
Let T x be the semidefinite positive quadratic form of S(d) defined for 2 S(d), by

T x : 7 ! (x > x) 2 .
(5.33)

Then it exists a positive linear operator T x from S(d) to S(d) such that T x ( ) = h , T x i.

Therefore the function f will be stricly concave if for all directions 2 S(d)

1 n n X i=1 T x i ( ) > 0.
(5.34)

We will bound the empirical expectation in Eq. ( 5.34) by first showing that its expectation remains away from 0. Then we will use a concentration inequality for matrices to control the distance between the sum in Eq. ( 5.34) and its expectation.

We first derive conditions so that the result is true in expectation, i.e. for the operator T defined by T = ET x for x following the same law as (y, z >

) > . We denote by m = Ez 2 and by = Ez 4 /m 2 its kurtosis.

We let = ✓ a b > b C ◆ and then have x > x = a + 2yb > z + z > Cz. Thus T x ( ) = a 2 + 4ayb > z + 2az > Cz + 4b > (zz > )b + (z > Cz) 2 + 4yb > z(z > Cz).
Therefore we can express the value of the operator T only in function of the elements 200 of :

T ( ) = (a + m tr C) 2 + 4mkbk 2 2 + 2m 2 kC Diag(diag(C))k 2 F + m 2 ( 1)k diag(C)k 2 ,
where we have used

E(z > Cz) 2 = E X i,j,k,l z i z j z k z l c i,j c k,l = E X i (z i ) 4 c 2 i,i + E X i,k6 =i z 2 i z 2 k c i,i c k,k + 2E X i,j6 =i z 2 i z 2 j c 2 i,j = m 2 X i c 2 i,i + m 2 X i,k6 =i c i,i c k,k + 2m 2 X i,j6 =i c 2 i,j = m 2 ( 3) X i c 2 i,i + m 2 X i,k c i,i c k,k + 2m 2 X i,j c 2 i,j = m 2 ( 3)k diag(C)k 2 + m 2 2kCk 2 F + tr(C) 2 = m 2 ( 3)k diag(C)k 2 + m 2 2kC Diag(diag(C))k 2 F + tr(C) 2 . Since 1, we get T ( ) (a + m tr C) 2 + 4mkbk 2 2 + 2m 2 (kCk 2 F k diag(C)k 2 ).
Thus T ( ) = 0 if and only if = 1 with b = 0 d 1 and C = diag(c) with c > 1 d = a m 2 . With the condition = 1 meaning that var(z 2 ) = 0 and thus z 2 is constant a.s., i.e. z follows a Rademacher law.

However we would like to bound T ( ) away from zero by some constant and for that we are looking for the smallest eigenvalue of the operator ET x . Unfortunately we are not able to solve the optimization problem min

2S(d),k k 2 F =1 T ( ),
and we have to compute all the spectrum of this operator to be able to find the smallest using ET x = 1/2rT ( ) .

We have

1/2rT ( ) = 0 @ a + m tr(C) 2 mb > 2mb (a + m tr(C))m 2 I d 1 + 2m 2 C +m 2 ( 3) Diag(diag(C)) 1 A .
-For all b 2 R d 1 we have for =

✓ 0 b > b 0 ◆ , 1/2rT ( ) = 2m . Thus 2m is an eigenvalue of multiplicity d 1.
-For all C 2 R (d 1)⇥(d 1) with diag(C) = 0 d 1 we have for

= ✓ 0 0 0 C ◆ , 1/2rT ( ) = 2m 2 . Thus 2m 2 is an eigenvalue of multiplicity (d 1)(d 2)
-For all c 2 R d 1 with c > 1 d 1 = 0 we have for =

✓ 0 0 0 diag(C) ◆ , 1/2rT ( ) = m 2 ( 1) . Thus m 2 ( 1) is an eigenvalue of multiplicity d 2. -For all a, c 2 R 2 we have for = ✓ a 0 0 cI d 1 ◆ , 1/2rT ( ) = ✓ a + m(d 1)c 0 0 [ma + m 2 (d + 2)c]I d 1 ◆ = Diag h ✓ 1 m1 > d 1 m1 d 1 (d + 2)m 2 I d 1 ◆ ✓ a c1 d 1 ◆ i .
Thus an eigenvalue of

✓ 1 (d 1)m m (d + 2)m 2 ◆
with an eigenvector [a, c] > would be an eigenvalue of the operator ET x with a corresponding eigenvector

✓ a 0 0 cI d 1 ◆ .
This matrix has two simple eigenvalues

µ ± = 1 + (d + 2)m 2 ± p (1 + (d + 2)m 2 ) 2 4m 2 ( 1) 2 .
(5.35)

Moreover when we add all the multiplicity of the found eigenvalues we get d 1 +

(d 1)(d 2) 2 + d 2 + 2 = d(d+1) 2
which is the dimension of S(d), therefore we have found all the eigenvalues of the linear operator ET x .

We will prove now than the smallest eigenvalue is µ when the dimension d is large enough with regards to m 2 and 2m 2 otherwise. .

Unfortunately µ can become small when the dimension increases as explained by the tight bound µ m 2

( 1) 1+(d+ 2)m 2 . However the corresponding eigenvector have a particular structure we will be able to exploit.

Proof. First we note that µ  m 2

( 1) and compute .

µ 2m 2 , 1 + (d + 2)m 2 p (1 + (d + 2)m 2 ) 2 4m 2 ( 1) 4m 2 0 , 1 + (d + 2)m 2 4m 2 p (1 + (d + 2)m 2 ) 2 4m 2 ( 1) , (1 + (d + 2)m 2 4m 2 ) 2 (1 + (d + 2)m 2 ) 2 4m 2 (
Therefore |c min |  m |(d + 2)m 2 1|
.

We will control now the behavior of the empirical expection by its expectation thanks to concentration theory. By definition T x is a symmetric positive linear operator as its projection T ?

x onto the orthogonal space of min . We can thus apply the Matrix Chernoff inequality from Tropp [2012, Theorem 5.1.1] to these two operators using kT

x k op  kxx > k 2  tr(xx > ) 2  kxk 4 2  R 4 d 2 Then: P min ⇣ X k=1 T x k ⌘  n µ 1 !  d h e (1 ) i nµ 1 /(2R 4 d 2 )  de (1 ) 2 nµ 1 /(4R 4 d 2 )
, before, we get x > i V ⇤ x i = (⇡y i ) 2 and with I ± the set of indices such that ⇧ n y i = ⇡y ± , the KKT conditions for V = V ⇤ can be written as

1 n h X i2I + ⇣ 1 ⇡y + 1 ⌘ x i x > i + X i2I ⇣ 1 ⇡y 1 ⌘ x i x > i i = A n 4 0 and A n V ⇤ = 0.
We check that with n ± = #{I ± }:

w > A n w = 0 = X i2I + ⇣ 1 ⇡y + 1 ⌘ (⇡y + ) 2 + X i2I ⇣ 1 ⇡y 1 ⌘ (⇡y ) 2 = n + ⇣ 1 ⇡y + 1 ⌘ (⇡y + ) 2 + n ⇣ 1 ⇡y 1 ⌘ (⇡y ) 2 = n + ⇡y + n ⇡y n + (⇡y + ) 2 + n (⇡y ) 2 = y > ⇧ n y (⇧ n y) > ⇧ n y = y > ⇧ n y y > ⇧ n y = 0.
And

A n = 1 2n ⇥ P i2I + ↵ + x i x > i + P i2I ↵ x i x > i ⇤ with ↵ + = 1 ⇡y + 1 and ↵ = 1 ⇡y
1 . Unfortunately ↵ + ↵  0, and A n is not necessary negative. Even worse we will show that EA is not semi-definite negative which will conclude the proof since by the law of large number lim n!1

1 n A n = EA.
Assume that the proportions of the two clusters stay constant with n ± = ⇢ ± n, then

EA = ⇢ + ↵ + ✓ (⇡y + ) 2 0 0 I ◆ + ⇢ ↵ ✓ (⇡y ) 2 0 0 I ◆ .
And ⇢ + ↵ + (⇡y + ) 2 + ⇢ ↵ (⇡y ) 2 = 0 since w > A n w = 0. Then

⇢ + ↵ + + ⇢ ↵ = ⇢ + ⇡y ⇢ ⇡y + ⇡y + ⇡y ⇡y + ⇡y = (⇢ + + ⇢ ) 1 > n y n (⇢ + ⇢ ) + (1 (1 > n y) 2 ) (1 ( 1 > n y n ) 2 ) = 1 > n y n (⇢ + ⇢ ) + ( 1 > n y n ) 2 ) (1 ( 1 > n y n ) 2 ) = 2( 1 > n y n ) 2 (1 ( 1 > n y n ) 2 ) 0. Thus A = 2(1 > n y) 2 (n 2 (1 > n y) 2 )
✓ 0 0 0 I ◆ is not semi-definite negative and V ⇤ is not solution of the relaxation Eq. (5.12).

5.E Auxiliary Results for Sparse Extension

5.E.1

There Is a Rank-One Solution of the Relaxation Eq. (5.18) Lemma 39. The rank-one solution V ⇤ = v ⇤ v ⇤> is solution of the relaxation Eq. ( 5.18) if the design matrix X is such that 1 n X > X has all its diagonal entries less than one.

Proof. The KKT conditions are

1 n n X i=1 x i x > i p x > i W x i U 1 n X > X = A 4 0 and AW = 0, with U such that U ij = sign(W ij ) if W ij 6 = 0 and U ij 2 [ 1, 1] otherwise. For V ⇤ = v ⇤ v ⇤> this gives A = (1 + ) n X > X U 1 n X > X = h X > X n U i ,
with U 1,1 = 1, and U i,j 2 [ 11] otherwise. We check that AV ⇤ = 0. If the design matrix X satsifies assumption (A1), we can choose a sub-gradient U such that the dual variable A = 0 and thus V ⇤ is solution. Otherwise by property of semi-definite matrices, there is a diagonal entry of 1 n X > X which is bigger than 1 which prevents A to be semi-definite negative since the corresponding diagonal entry of X > X n U will be positive. This shows that V ⇤ does not solve the problem.

5.E.2 Proof of Proposition 20

Lemma 40. For 2 [0, 1), with probability 1 5d 2 exp

2 n( 1) 2dR 4

(1/m 2 + +d) , for any direction such that V ⇤ + < 0, we have:

g(V ⇤ ) g(V ⇤ + ) > (1 ) h k Diag( )k 1 + 1 + d + 1/m 2 (1 + ) 3 4 k Diag( )k 2 2 i +o(k k 2 ) 0.
Moreover we also have with probability at least 1 5d 2 exp 2 nm 2

( 1) 2dR 4

, for any symmetric matrix such that V ⇤ + < 0 and Diag( ) 2 (e min ) ? :

g(V ⇤ ) g(V ⇤ + ) > (1 ) h k Diag( )k 1 +m 2 ( 1) (1+ ) 3 4 k Diag( )k 2 2 i +o(k k 2 ) 0.
where e min = [1, c min 1 d 1 ] is defined in the proof and satisfies

|c min |  m |(d + 2)m 2 1|
.

Proof Outline

We will investigate under which conditions on X the solution is unique, first for a deterministic design matrix. We make the following deterministic assumptions on X for , ⇣ 0 and S ⇢ R d :

(A1) k X > X n k 1  1 (A3) k Z > Z n Diag(diag( 1 n Z > Z))k 1  (A2) k Z > y n k 1  (A4) S min X 2 (X 2 ) > n ⇣ > 0.
Where we denoted by the Hadamard (i.e., pointwise) product between matrices and S min the minimum eigenvalue of a linear operator restricted to a subspace S.

Then with g(V ) = 2 n P n i=1 p x > i V x i kV k 1 1
n tr X > XV , we can certify that g will decrease around the solution V ⇤ .

Lemma 41. Let us assume that the noise matrix verifies assumption (A1,A2,A3,A4), then for all direction such that V ⇤ + < 0 and diag( ) 2 S we have:

g(V ⇤ ) g(V ⇤ + ) (1 )k Diag(diag( ))k 1 +⇣ (1 + ) 3 4 k Diag( )k 2 2 +o(k k 2 ) > 0.
Let us assume now that (z i ) i=1,.,d are i.i.d of law z symmetric with Ez = Ez 3 = 0,

Ez 2 = m = 1, Ez 4 /(Ez 2 ) 2
= and such that kzk 1 is a.s. bounded by 0  R  1. Then the matrix X satisfies a.s. assumption (A1). Using multiple Hoeffding's inequalities we have Lemma 42. If z does not follow a Rademacher law, the design matrix X satsifies assumptions (A1,A2,A3,A4) with probability greater than 1 8d 2 exp .

This lemma concludes the proof of proposition 20. We will now prove these two lemmas.

Proof of Lemma 41

Proof. Since the dual variable A for the PSD constraint is 0 (see the proof of lemma 39), this constraint W < 0 is not active and we will show that the function decreases in a set of directions which include the one for which V ⇤ + < 0.

Therefore we consider a direction

= ✓ a b > b C ◆ , with C < 0, which is slightly more general than V ⇤ + < 0. We denote by f (W ) = 2 n P n i=1 p x > i W x i 1
n tr X > XW the smooth part of g. By Taylor-Young, we have for all W :

f (W ) f (W + ) = hf 0 (W ), i 1 2 h , f 00 (W ) i + o(k k 2 ).
Thus:

g(W ) g(W + ) = hf 0 (W ), i 1 2 h , f 00 (W ) i+ (kW + k 1 kW k 1 )+o(k k 2 ). 207 In W = V ⇤ this gives with X > X = ✓ n y > Z Z > y Z > Z ◆ , g(W ) g(W + ) = h X > X n , i 1 2 h , f 00 (V ⇤ ) i + (a + 2kbk 1 + kCk 1 ) + o(k k 2 ) = ⇥ 2(kbk 1 1 n b > Z > y) + kCk 1 1 n tr(Z > ZC) ⇤ 1 2 h , f 00 (V ⇤ ) i + o(k k 2 ).
And with Hölder's inequality and assumption (A2)

kbk 1 1 n b > Z > y kbk 1 (1 k 1 n Z > yk 1 ) (1 )kbk 1 .
Nevertheless we will show in lemma 43 that kCk

1 1 n tr(Z > ZC) (1 )kC diag(C)k 1 , thus g(W ) g(W + ) (1 )(2kbk 1 + kC diag(C)k 1 ) + o(k k 2
).

(5.36) However in Eq. ( 5.36), g(W ) g(W + ) = 0 for b = 0 and C diagonal, therefore we have to investigate second order conditions, i.e. to show for = diag(e) with e 2 R d that h , f 00 (V ⇤ ) i > 0.

And with assumption (A4)

4 (1 + ) 3 hdiag(e), f 00 (V ⇤ ) diag(e)i = 1 n n X i=1 (x > i diag(e)x i ) 2 = 1 n n X i=1 ( d X j=1 e j (x j i ) 2 ) 2 = 1 n n X i=1 e > [x 2 i (x 2 i ) > ]e min X 2 (X 2 ) > n kek 2 ⇣kek 2 2 .
Thus we can conclude:

g(W ) g(W + ) (1 )(2kbk 1 + kC diag(C)k 1 ) + ⇣ (1 + ) 3 4 kek 2 2 + o(k k 2 ).
We proceed as in the proof of proposition 19 to show that X satisfies assumption (A4). We first derive a condition to have the result in expectation, then we use an inequality concentration on matrix to bound the empirical expectation. This will be very similar, but we will get a better scaling since is diagonal.

Using the same arguments as in the proof of proposition 19 we have for the diagonal matrix = diag(e) with e = (a, c) 2 R d :

e > E(x 2 (x 2 ) > )e = E(x > x) 2 = (a + mc > 1 n 1 ) 2 + m 2 ( 1)kck 2 2 > 0 if > 1.
We can show that m 2 ( 1) is an eigenvalue of multiplicity d 2 and µ ± are eigenvalues of multiplicity one of the operator 7 ! E(x > x) 2 with eigenvectors e ± . Thus we have

min (Ex 2 (x 2 ) > ) = 1 + (d + 2)m 2 p (1 + (d + 2)m 2 ) 2 4m 2 ( 1) 2 m 2 ( 1) 1 + (d + 2)m 2 ,
(5.37) and e ?

min (Ex 2

(x 2 ) > ) = m 2 ( 2). Moreover max ⇣ x 2 (x 2 ) > ⌘ = (x 2 ) > x 2 = d X j=1 (x i ) 4  dR 4 .
Thus we can apply the Matrix Chernoff inequality from [START_REF] Tropp | User-friendly tail bounds for sums of random matrices[END_REF] for µ S = S min (Ex 2 (x 2 ) > ):

P S min ⇣ X 2 (X 2 ) > n ⌘  (1 )µ S !  de 2 nµ S /(2dR 4
) .

Thus with probability 1 5d 2 exp( 2 nµ /(2dR 4

)) the design matrix X satisfies assumption (A1,A2,A3,A4) with ⇣ = ( 1)µ and S = R d . And with probability 1 5d 2 exp( 2 n min{ 1, 2}/(2dR 4 )) the design matrix X satisfies assumption (A1,A2,A3,A4) with ⇣ = ( 1) min{ 1, 2} and S = e ? .

5.F Proof of Multi-Label Results

We first prove the lemma 31:

Proof. Let A 2 R k⇥k symmetric semi-definite positive such that diag(ỹAỹ > ) = 1 n , then diag(ỹAỹ > ) = k X i=0 a i,i 1 n + 2 k X i=1 a 0,i y i + 2 X 1i<jk a i,j y i y j
Algorithm 1 FISTA Algorithm to solve Eq. ( 5.38)

1: Input X.

2: Compute Lipschitz constant L.

3: Let (u 0 , C 0 ) be an arbitrary starting point. 4: Let (ū 0 , C0 ) = (u 0 , C 0 ), t 0 = 1. 5: Set the maximum iterations to be K. 6: for k = 1, 2, . . . , K do . The loop can also be terminated based on duality gap.

7:

(ū k 1 2 , Ck 1 2 ) = ⇣ ūk 1 L r ūk , Ck 1 L r Ck ⌘ . 8: Obtain u k = arg min u2R n + n L 2 ku ūk 1 2 k 2 + 1 2n P n i=1 1 u i } by Algorithm 2.
9:

Obtain C k = arg min C n I C:|C ij |6c i c j + L 2 kC Ck 1 2 k 2 F o by thresholding. 10: t k = 1+ p 1+4t 2 k 1 2 . 11: (ū k , Ck ) = (u k , C k ) + (t k 1 1) t k ⇣ (u k , C k ) (u k 1 , C k 1 )
⌘ .

12: end for 13: Output (u K , C K ).

Algorithm 2 Newton method to solve u sub-problem

1: Input u k 1 2 , n, L. 2: u 0 i = max(u k 1 2 i , 1 (2nL)
1 3

), i = 1, 2, . . . , n.

3: Set M to be the max number of Newton steps. for i = 1, 2, . . . , n do 6:

u t i = 2nL(u t 1 i ) 3 u k 1 2 i +3u t i 2(nL(u t 1 i ) 3 +1)
.

7:

end for 8: end for 9: Output max(u M , 0).

Introduction

The consecutive 1's problem (C1P) [START_REF] Fulkerson | Incidence matrices with the consecutive 1's property[END_REF] is defined as follows. Given a binary matrix A the goal is to permute its rows in such a way that the resulting matrix enjoys the consecutive 1's property: each of its columns is a vector v = (v 1 , . . . , v n ) > where v j = 1 if and only if a  j  b for two integers a, b between 1 and n.

This problem has its roots in archeology and especially sequence dating where the goal is to recover the chronological order of sepultures based on artifacts found in these sepultures where the entry A i,j of matrix A indicates the presence of artifact j in sepulture i. In his seminal work, egyptologist Flinders Petrie [1899] formulated the hypothesis that two sepultures should be close in the time domain if they present similar sets of artifacts. Already in the noiseless case, this problem presents an in-teresting algorithmic challenge and is reducible to the famous Travelling Salesman Problem [START_REF] Gertzen | Flinders Petrie, the travelling salesman problem, and the beginning of mathematical modeling in archaeology[END_REF] as observed by statistician David [START_REF] Kendall | A statistical approach to Flinders Petrie's sequence-dating[END_REF][START_REF] Kendall | Incidence matrices, interval graphs and seriation in archeology[END_REF][START_REF] Kendall | A mathematical approach to seriation[END_REF][START_REF] Kendall | Abundance matrices and seriation in archaeology. Z. Wahrscheinlichkeitstheorie und Verw[END_REF]] who employed early tools from multidimensional scaling as a heuristic to solve it. C1P belongs to a more general class of so-called seriation problems that consist in optimizing various criteria over the discrete set of permutations. While such problems are hard in general, it can be shown that a subset of the these problems, including C1P, can be solved efficiently using spectral method [START_REF] Atkins | A spectral algorithm for seriation and the consecutive ones problem[END_REF] or convex optimization [Fogel et al., 2013, Lim and[START_REF] Lim | Beyond the birkhoff polytope: Convex relaxations for vector permutation problems[END_REF]. However, little is known about the robustness to noise of such methods.

In order to set the benchmark for the noisy case, we propose a statistical seriation model and study optimal rates of estimation in this model. Assume that we observe an n ⇥ m matrix Y = ⇧A + Z, where ⇧ is an unknown n ⇥ n permutation matrix, Z is an n ⇥ m noise matrix and A 2 IR n⇥m is assumed to belong to a class of matrices that satisfy a certain shape constraint. Our goal is to give estimators ⇧ and  so that ⇧  is close to ⇧A. The shape constraint can be the consecutive 1's property, but more generally, we consider the class of matrices that have unimodal columns, which also include monotonic columns as a special case. These terms will be formally defined at the end of this section.

The rest of the chapter is organized as follows. In Section 6.2 we formulate the model and discuss related work. Section 6.3 collects our main results, including uniform and adaptive upper bounds for the least-squares estimator together with corresponding minimax lower bounds in the general unimodal case. In Section 6.4, for the special case of monotone columns, we propose a computationally efficient alternative to the least-squares estimator and study its rates of convergence both theoretically and numerically. Appendix 6.A is devoted to the proofs of the upper bounds, which use the metric entropy bounds proved in Appendix 6.B. The proofs of the informationtheoretic lower bounds are presented in Appendix 6.C. In Appendix 6.D, we study the rate of estimation of the efficient estimator for the monotonic case. Appendix 6.E contains a delayed proof of a trivial upper bound. Appendix 6.F presents new bounds for unimodal regression implied by our analysis, which are minimax optimal up to logarithmic factors.

Notation. For a positive integer n, define [n] = {1, . . . , n}. For a matrix A 2 IR n⇥m , let kAk F denote its Frobenius norm, and let A i,• be its i-th row and A •,j be its j-th column. Let B n (a, t) denote the Euclidean ball of radius t centered at a in IR n . We use C and c to denote positive constants that may change from line to line. For any two sequences (u n ) n and (v n ) n , we write u n . v n if there exists an absolute constant C > 0 such that u n  Cv n for all n. We define u n & v n analogously. Given two real numbers a, b, define a ^b = min(a, b) and a _ b = max(a, b).

Denote the closed convex cone of increasing 1 sequences in IR n by

S n = {a 2 IR n : a 1  • • •  a n }.
We define S m to be the Cartesian product of m copies of S n and we identify S m to the set of n ⇥ m matrices with increasing columns.

For any l 2

[n], define the closed convex cone C l = {a 2 IR n : a 1  • • •  a l }\{a 2 IR n : a l • • • a n }
, which consists of vectors in IR n that increase up to the l-th entry and then decrease. Define the set U of unimodal sequences in IR n by U = S n l=1 C l . We define U m to be the Cartesian product of m copies of U and we identify U m to the set of n⇥m matrices with unimodal columns. It is also convenient to write U m as a union of closed convex cones as follows. For l

= (l 1 , . . . , l m ) 2 [n] m , let C m l = C l 1 ⇥ • • • ⇥ C lm . Then U m is the union of the n m closed convex cones C m l , l 2

Problem Setup and Related Work

In this section, we formally state the problem of interest and discuss several lines of related work.

The Seriation Model

Suppose that we observe a matrix Y 2 IR n⇥m , n 2 such that

Y = ⇧ ⇤ A ⇤ + Z , (6.1)
where A ⇤ 2 U m , ⇧ 2 S n and Z is a centered sub-Gaussian noise matrix with variance proxy 2 > 0. More specifically, Z is a matrix such that E[Z] = 0 and, for any

M 2 IR n⇥m , E ⇥ exp Tr(Z > M ) ⇤  exp ⇣ 2 kM k 2 F 2 ⌘ ,
where Tr(•) is the trace operator. We write Z ⇠ subG n,m ( 2 ) or simply Z ⇠ subG( 2 ) when dimensions are clear from the context.

Given the observation Y , our goal is to estimate the unknown pair (⇧ ⇤ , A ⇤ ). The performance of an estimator ( ⇧, Â) 2 S n ⇥ U m , is measured by the quadratic loss:

1 nm k ⇧ Â ⇧ ⇤ A ⇤ k 2 F .
In particular, its expectation is the mean squared error. Since we are interested in estimating ⇧ ⇤ A ⇤ 2 M, we can also view M as the parameter space.

In the general unimodal case, upper bounds on the above quadratic loss do not imply individual upper bounds on estimation of the matrix ⇧ ⇤ or the matrix A ⇤ due to lack of identifiability. Nevertheless, if we further assume that the columns of A ⇤ are monotone increasing, that is A ⇤ 2 S m , then the following lemma holds.

Lemma 44. If A ⇤ , à 2 S m , then for any ⇧ ⇤ , ⇧ 2 S n , we have that k à A ⇤ k 2 F  k ⇧ à ⇧ ⇤ A ⇤ k 2 F ,
and that

k ⇧A ⇤ ⇧ ⇤ A ⇤ k 2 F  4k ⇧ Ã ⇧ ⇤ A ⇤ k 2 F . Proof. Let a, b 2 S n and b ⇡ = (b ⇡(1) , . . . , b ⇡(n) ) where ⇡ : [n] ! [n] is a permutation. It is easy to check that P n i=1 a i b i P n i=1 a i b ⇡(i) , so ka bk 2 2  ka b ⇡ k 2 2 .
Applying this inequality to columns of matrices, we see that

k à A ⇤ k 2 F  k à ⇧ 1 ⇧ ⇤ A ⇤ k 2 F = k ⇧ à ⇧ ⇤ A ⇤ k 2 F , since A ⇤ , à 2 S m . Moreover, k ⇧A ⇤ ⇧ Ãk F = kA ⇤ Ãk F , so k ⇧A ⇤ ⇧ ⇤ A ⇤ k F  kA ⇤ Ãk F + k ⇧ à ⇧ ⇤ A ⇤ k F  2k ⇧ à ⇧ ⇤ A ⇤ k F ,
by the triangle inequality and the previous display.

Lemma 44 guarantees that k ⇧A ⇤ ⇧ ⇤ A ⇤ k F is a pertinent measure of the performance of ⇧. Note further that k ⇧A ⇤ ⇧ ⇤ A ⇤ k F is large if ⇧ misplaces rows of A ⇤ that have large differences, and is small if ⇧ only misplaces rows of A ⇤ that are close to each other. We argue that, in the seriation context, this measure of distance between permutations is more natural than ad hoc choices such as the trivial 0/1 distance or popular choices such as Kendall's ⌧ or Spearman's ⇢.

Apart from Section 6.4 (and Appendix 6.D), the rest of this chapter focuses on the least-squares (LS) estimator defined by

( ⇧, Â) 2 argmin (⇧,A)2Sn⇥U m kY ⇧Ak 2 F . (6.2)
Taking M = ⇧ Â, we see that it is equivalent to define the LS estimator by

M 2 argmin M 2M kY M k 2 F . (6.3)
Note that in our case, the set of parameters M is not convex, but is a union of n!n m closed convex cones and it is not clear how to compute the LS estimator efficiently. We discuss this aspect in further details in the context of monotone columns in Section 6.4. Nevertheless, the main focus of this chapter is the least-squares estimator which, as we shall see, is near-optimal in a minimax sense and therefore serves as a benchmark for the statistical seriation model.

Related Work

Our work falls broadly in the scope of statistical inference under shape constraints but presents a major twist: the unknown latent permutation ⇧ ⇤ .

Shape Constrained Regression

To set our goals, we first consider the case where the permutation is known and assume without loss of generality that ⇧ ⇤ = I n . In this case, we can estimate individ- .5) This result was extended by [START_REF] Bellec | Sharp oracle inequalities for least squares estimators in shape restricted regression[END_REF] to a sharp oracle inequality where C = 1. This bound was also shown to be optimal in a minimax sense by Chatterjee et al.

en k(✓) ⌘ . ( 6
[2015], [START_REF] Bellec | Sharp oracle bounds for monotone and convex regression through aggregation[END_REF]. Unlike its monotone counterpart, unimodal regression where ✓ ⇤ 2 U has received sporadic attention [START_REF] Shoung | Least squares estimators of the mode of a unimodal regression function[END_REF][START_REF] Köllmann | Unimodal regression using Bernstein-Schoenberg splines and penalties[END_REF][START_REF] Chatterjee | Adaptive risk bounds in unimodal regression[END_REF]. This state of affairs is all the more surprising given that unimodal density estimation has been the subject of much more research [START_REF] Bickel | Some problems on the estimation of unimodal densities[END_REF][START_REF] Birge | Estimation of unimodal densities without smoothness assumptions[END_REF][START_REF] Eggermont | Maximum likelihood estimation of smooth monotone and unimodal densities[END_REF][START_REF] Daskalakis | Learning k-modal distributions via testing[END_REF], 2013[START_REF] Turnbull | Unimodal density estimation using bernstein polynomials[END_REF]. It was recently shown by [START_REF] Chatterjee | Adaptive risk bounds in unimodal regression[END_REF] that the LS estimator also adapts to V (✓ ⇤

) and k(✓ ⇤ ) for unimodal regression:

1 n k ✓ ✓ ⇤ k 2 2 . min ⇣ 4/3 ⇣ V (✓ ⇤ ) + n ⌘ 2/3 , 2 n k(✓ ⇤ ) 3/2 (log n) 3/2 ⌘ (6.6)
with probability at least 1 n ↵ for some ↵ > 0. The exponent 3/2 in the second term was improved to 1 in the new version by [START_REF] Chatterjee | Adaptive risk bounds in unimodal regression[END_REF] after the first version of our current paper was posted. Note that the exponents in Eq. (6.6) are different from the isotonic case. Our results will imply that they are not optimal and in fact the LS estimator achieves the same rate as in isotonic regression. See Corollary 10 for more details. The algorithmic aspect of unimodal regression has received more attention [START_REF] Frisen | Unimodal regression[END_REF][START_REF] Geng | Algorithm as 257: Isotonic regression for umbrella orderings[END_REF][START_REF] Bro | Least squares algorithms under unimodality and nonnegativity constraints[END_REF][START_REF] Boyarshinov | Linear time isotonic and unimodal regression in the L 1 and L 1 norms[END_REF] and [START_REF] Stout | Unimodal regression via prefix isotonic regression[END_REF] showed that the LS estimator can be computed with time complexity O(n) using a modified version of PAVA. Hence there is little difference between isotonic and unimodal regressions from both computational and statistical points of views.

Latent Permutation Learning

When the permutation ⇧ ⇤ is unknown the estimation problem is more involved. Noisy permutation learning was explicitly addressed by [START_REF] Collier | Minimax rates in permutation estimation for feature matching[END_REF] where the problem of matching two sets of noisy vectors was studied from a statistical point of view. Given n ⇥ m matrices Y = A ⇤ + Z and Ỹ = ⇧ ⇤ A ⇤ + Z, where A ⇤ 2 IR n⇥m is an unknown matrix and ⇧ ⇤ 2 IR n⇥n is an unknown permutation matrix, the goal is to recover ⇧ ⇤ . It was shown by [START_REF] Collier | Minimax rates in permutation estimation for feature matching[END_REF]

that if min i6 =j kA i,• A j,• k 2 c (log n) 1/2 _ (m log n) 1/4
, then the LS estimator defined by ⇧ = argmin ⇧2Sn k⇧Y Ỹ k 2 F recovers the true permutation with high probability. However they did not directly study the behavior of k ⇧A ⇤ ⇧ ⇤ A ⇤ k 2 F . In his celebrated paper on matrix estimation, Sourav [START_REF] Chatterjee | Matrix estimation by universal singular value thresholding[END_REF] describes several noisy matrix models involving unknown latent permutations. One is the nonparametric Bradley-Terry-Luce (NP-BTL) model where we observe a matrix Y 2 IR n⇥n with independent entries Y i,j ⇠ Ber(P i,j ) for some unknown parameters P = {P i,j } 1i,jn where P i,j 2 [0, 1] is equal to the probability that item i is preferred over item j and P j,i = 1 P i,j . Crucially, the NP-BTL model assumes the socalled strong stochastic transitivity (SST) [START_REF] Davidson | Experimental tests of a stochastic decision theory[END_REF]Marschak, 1959, Fishburn, 1973] assumption: there exists an unknown permutation matrix ⇧ 2 IR n⇥n such that the ordered matrix

A = ⇧ > P ⇧ satisfies A 1,k  • • •  A n,k for all k 2 [n]
. Note that the NP-BTL model is a special case of our model in Eq. ( 6.1) where m = n and Z ⇠ subG(1/4) is taken to be Bernoulli. [START_REF] Chatterjee | Matrix estimation by universal singular value thresholding[END_REF] proposed an estimator P that leverages the fact that any matrix P in the NP-BTL model can be approximated by a low rank matrix and proved [Chatterjee, 2015, Theorem 2.11] that n 2 k P P k 2 F . n 1/4 , which was improved to n 1/2 by [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF] for a variation of the estimator. This method does not yield individual estimators of ⇧ or A, and Chatterjee and Mukherjee [2016] proposed estimators ⇧ and  so that ⇧  ⇧> estimates P with the same rate n 1/2 up to a logarithmic factor. The non-optimality of this rate has been observed by [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF] who showed that the correct rate should be of order n 1 up to a possible log n factor. However, it is not known whether a computationally efficient estimator could achieve the fast rate. A recent work by [START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF] explored a new notion of adaptivity for which the authors proved a computational lower bound, and also proposed an efficient estimator whose rate of estimation matches that lower bound. Also mentioned by [START_REF] Chatterjee | Matrix estimation by universal singular value thresholding[END_REF] is the so-called stochastic block model that has since received such extensive attention in various communities that it is futile to attempt to establish a comprehensive list of references. Instead, we refer the reader to the article by [START_REF] Gao | Rate-optimal graphon estimation[END_REF] and references therein. This paper establishes the minimax rates for this problem and its continuous limit, the graphon estimation problem and, as such, constitutes the state-of-the-art in the statistical literature. In the stochastic block model with k 2 blocks, we assume that we observe a matrix Y = P + Z where P = ⇧A⇧ > , ⇧ 2 IR ⇥n is an unknown permutation matrix and A has a block structure, namely, there exist positive integers n 1 < . . . < n k < n k+1 := n, and k 2 real numbers a s,t , (s, t)

2 [k] 2 such that A has entries A i,j = X (s,t)2[k] 2 a s,t 1I{n s  i  n s+1 , n t  j  n t+1 } , i, j 2 [n] .
While traditionally, the stochastic block model is a network model and therefore pertains only to Bernoulli observations, the more general case of sub-Gaussian additive error is also explicitly handled by [START_REF] Gao | Rate-optimal graphon estimation[END_REF]. For this problem, Gao, Liu and Zhou have established that the least-squares estimator P satisfies n 2 k P P k 2 F . k 2 /n 2 + (log k)/n together with a matching lower bound. Using piecewise constant approximation to bivariate Hölder functions, they also establish that this estimator with a correct choice of k leads to minimax optimal estimation of smooth graphons. Both results exploit extensively the fact that the matrix P is equal to or can be well approximated by a piecewise constant matrix and our results below take a similar route by observing that monotone and unimodal vectors are also well approximated by piecewise constant ones. Moreover, we allow for rectangular matrices.

In fact, our result can be also formulated as a network estimation problem but on a bipartite graph, thus falling at the intersection of the above two examples. Assume that n left nodes represent items and that m right nodes represent users. Assume further that we observe the n ⇥ m adjacency matrix Y of a random graph where the presence of edge (i, j) indicates that user j has purchased or liked item i. Define P = E[Y ] and assume SST across items in the sense that there exists an unknown n⇥n permutation matrix ⇧ ⇤ such that P

= ⇧ ⇤ A ⇤ and A ⇤ is such that A ⇤ 1,j  • • •  A ⇤ n,j
for all users j 2 [m]. This model falls into the scope of the statistical seriation model in Eq. ( 6.1).

Main Results

Adaptive Oracle Inequalities

For a matrix A 2 U m , let k(A •,j ) = Card({A 1,j , . . . , A n,j }) be the number of values taken by the j-th column of A and define K

(A) = P m j=1 k(A •,j ). Observe that K(A)
m. The first theorem shows that the LS estimator adapts to the complexity K.

Theorem 10. For A ⇤ 2 IR n⇥m and Y = ⇧ ⇤ A ⇤ + Z, let ( ⇧, Â) be the LS estimator defined in Eq. ( 6.2). Then the following oracle inequality holds

1 nm k ⇧ Â ⇧ ⇤ A ⇤ k 2 F . min A2U m ⇣ 1 nm kA A ⇤ k 2 F + 2 K(A) nm log enm K(A) ⌘ + 2 log n m (6.7)
with probability at least 1 e c(n+m) , c > 0. Moreover,

1 nm Ek ⇧ Â ⇧ ⇤ A ⇤ k 2 F . min A2U m ⇣ 1 nm kA A ⇤ k 2 F + 2 K(A) nm log enm K(A) ⌘ + 2 log n m . (6.8)
Note that while we assume that A ⇤ 2 U m in Eq. ( 6.1), the above oracle inequalities hold in fact for any A ⇤ 2 IR n⇥m even if its columns are not assumed to be unimodal.

The above oracle inequalities indicate that the LS estimator automatically trades off the approximation error kA A ⇤ k 2 F for the stochastic error 2 K(A) log(enm/K(A)). If A ⇤ is assumed to have unimodal columns, then we can take A = A ⇤ in Eq. ( 6.7) and Eq. ( 6.8) to get the following corollary.

Corollary 8. For A ⇤ 2 U m and Y = ⇧ ⇤ A ⇤ + Z, the LS estimator ( ⇧, Â) satisfies 1 nm k ⇧ Â ⇧ ⇤ A ⇤ k 2 F . 2 ⇣ K(A ⇤ ) nm log enm K(A ⇤ ) + log n m
⌘ with probability at least 1 e c(n+m) , c > 0. Moreover, the corresponding bound with the same rate holds in expectation.

The two terms in the adaptive bound can be understood as follows. The first term corresponds to the estimation of the matrix A ⇤ with unimodal columuns if the permutation ⇧ ⇤ is known. It can be viewed as a matrix version of the adaptive bound in Eq. (6.5) in the vector case. The LS estimator adapts to the cardinality of entries of A ⇤ as it achieves a provably better rate if K(A ⇤

) is smaller while not requiring knowledge of K(A ⇤

). The second term corresponds to the error due to the unknown permutation ⇧ ⇤ . As m grows to infinity this second term vanishes, because we have more samples to estimate ⇧ ⇤ better. If m n, it is easy to check that the permutation term is dominated by the first term, so the rate of estimation is the same as if the permutation is known.

Global Oracle Inequalities

The bounds in Theorem 10 adapt to the cardinality of the oracle. In this subsection, we state another type of upper bounds for the LS estimator ( ⇧, Â). They are called global bounds because they hold uniformly over the class of matrices whose columns are unimodal and that have bounded variation. Recall that we call variation of a vector a 2 IR n the scalar V (a) 0 defined by

V (a) = max 1in a i min 1in a i . For m  K 0  nm and V 0 > 0, define U m K 0 = A 2 U m : K(A)  K 0 and U m (V 0 ) = A 2 U m : V (A)  V 0 .
We present below two lower bounds, one for the adaptive rate uniformly over U m K 0 and one for the global rate uniformly over U m (V 0 ). This splitting into two cases is solely justified by better readability but it is worth noting that a stronger lower bound that holds on the intersection U m K 0 \ U m (V 0 ) can also be proved and is presented as Proposition 21.

Theorem 12. There exists a constant c 2 (0, 1) such that for any K 0 m, and any estimator ( ⇧, Â), it holds that

sup (⇧,A)2Sn⇥U m K 0 IP ⇧A h 1 nm k ⇧ Â ⇧Ak 2 F & 2 ⇣ K 0 nm + log l m ⌘i c,
where l = min(K 0 m, m) + 1 and IP ⇧A is the probability distribution of Y = ⇧A + Z. It follows that the lower bound with the same rate holds in expectation.

In fact, the lower bound holds for any estimator of the matrix ⇧ ⇤ A ⇤ , not only those of the form ⇧  with  2 U m . The above lower bound matches the upper bound in Corollary 8 up to logarithmic factors.

Note the presence of a log l factor in the second term. If l = 1 then K 0 = m which means that each column of A is simply a constant block, so ⇧A = A for any ⇧ 2 S n . In this case, the second term vanishes because the permutation does not play a role. More generally, the number l 1 can be understood as the maximal number of columns of A on which the permutation does have an effect. The larger l, the harder the estimation. It is easy to check that if l n the second term in the lower bound will be dominated by the first term in the upper bound.

A lower bound corresponding to Corollary 9 also holds:

Theorem 13. There exists a constant c 2 (0, 1) such that for any V 0 0, and any estimator ( ⇧, Â), it holds that

sup (⇧,A)2Sn⇥U m (V 0 ) IP ⇧A h 1 nm k ⇧ Â ⇧Ak 2 F & 2 V 0 n 2/3 + 2 n + 2 m ^m2 V 2 0 i c ,
where IP ⇧A is the probability distribution of Y = ⇧A + Z. The lower bound with the same rate also holds in expectation.

There is a slight mismatch between the upper bound of Corollary 9 and the lower bound of Theorem 13 above. Indeed the lower bound features a term

2 m ^m2 V 2 0 instead of just 2 m . In the regime m 2 V 2 0 < 2 m
, where A has very small variation, the LS estimator may not be optimal. Proposition 22 indicates that a matrix with constant columns obtained by averaging achieves optimality in this extreme regime.

Further Results in the Monotone Case

A particularly interesting subset of unimodal matrices is S m , the set of n ⇥ m matrices with monotonically increasing columns. While it does not amount to the seriation problem in its full generality, this special case is of prime importance in the context of shape constrained estimation as illustrated by the discussion and references in Section 6.2.2. In fact, it covers the example of bipartite ranking discussed at the end of Section 6.2.2. In the rest of this section, we devote further investigation to this important case. To that end, consider the model in Eq. ( 6.1) where we further assume that A ⇤ 2 S m . We refer to this model as the monotone seriation model. In this context, define the LS estimator by

( ⇧, Â) 2 argmin (⇧,A)2Sn⇥S m kY ⇧Ak 2 F .
Since S m is a convex subset of U m , it is easily seen that the upper bounds in Theorem 10 and 11 remain valid in this case. The lower bounds of Theorem 12 (with log l replaced by 1) and Theorem 13 also extend to this case; see Appendix 6.C. Although for unimodal matrices the established error bounds do not imply any bounds on estimation of A ⇤ or ⇧ ⇤ in general, for the monotonic case, however, Lemma 44 yields that

k  A ⇤ k 2 F _ 1 4 k( ⇧ ⇧ ⇤ )A ⇤ k 2 F  k ⇧  ⇧ ⇤ A ⇤ k 2 F .
so that the LS estimator ( ⇧, Â) also leads to good individual estimators of ⇧ ⇤ and A ⇤ respectively.

Because it requires optimizing over a union of n! cones ⇧S m , no efficient way of computing the LS estimator is known since. As an alternative, we describe a simple and efficient algorithm to estimate (⇧ ⇤ , A ⇤

) and study its rate of estimation. Let K(A) and V (A) be defined as before. Moreover, for a matrix A 2 S m , let J denote the set of pairs of indices (i, j) 2 [n] 2 such that A i,• and A j,• are not identical. Define the quantity R(A) by 1 with equality achieved when all entries of u are the same.

R(A) = 1 n max I⇢[n] 2 |I|=n X (i,j)2I\J ⇣ kA i,• A j,• k 2 2 kA i,• A j,• k 2 1 ^mkA i,• A j,• k 2 2 kA i,• A j,• k 2 1 ⌘ . ( 6
For matrices with small R(•) values, it is possible to aggregate the information across each row to learn the unknown permutation ⇧ ⇤ in a simple fashion. Recovering the permutation ⇧ ⇤ , is equivalent to ordering (or ranking reversely) the rows of ⇧ ⇤ A ⇤ from their noisy version Y .

One simple method to achieve this goal, which we call RankSum, is to permute the rows of Y so that they have increasing row sums. However, it is easy to observe worth noting that R(A ⇤ ) is not needed to construct ( ⇧, Ã), so the estimator adapts to R(A ⇤ ) automatically. Remark 4. In the same way that Theorem 11 follows from Theorem 10, we can deduce from Theorem 14 a global bound for the estimator ( ⇧, Ã) which has rate

⇣ 2 V (A ⇤ ) log n n ⌘ 2/3 + 2 ⇣ log n n + R(A ⇤ ) log(nm) m ⌘ .
We conclude this section with a numerical comparison between the RankSum and RankScore procedures. Consider the model in Eq. ( 6.1) with A ⇤ 2 S m and assume without loss of generality that ⇧ ⇤ = I n . For various n⇥m matrices A ⇤ , we generate observations Y = A ⇤ +Z where entries of Z are i.i.d. standard Gaussian variables. The performance of the estimators given by RankScore and RankSum defined above is compared to the performance of the oracle Âoracle defined by the projection of Y onto the cone S m . For the RankScore estimator we take ⌧ = 6. The curves are generated based on 30 equally spaced points on the base-10 logarithmic scale, and all results are averaged over 10 replications. The vertical axis represents the estimation error of an estimator ⇧ Â, measured by the sample mean of log

10 1 nm k ⇧ Â A ⇤ k 2
F unless otherwise specified. We begin with two simple examples for which we set n = m. In the left plot of Figure 6-1, A ⇤ is defined as in Eq. (6.11). As expected, RankSum fails to estimate the true permutation and performs very poorly. On the other hand, RankScore succeeds in recovering the correct permutation and has roughly the same performance as the oracle. Because the difference of any two rows of A ⇤ is 1-sparse, R(A ⇤ ) = 1 according to Eq. ( 6.10) and the discussion thereafter. Hence, Theorem 14 predicts the fast rate, which is verified by the experiment. The right plot illustrates another extreme case; more precisely, we set A ⇤ to be the matrix with all m columns equal to 1 n (1, • • • , n) > . The difference of any two rows of A ⇤ is constant across all entries, so again we have R(A ⇤ ) = 1 by Eq. (6.10). Thus RankScore achieves the fast rate as expected. Note that RankSum also performs well in this case. of ⇧ by Lemma 61 and 44, and the plot suggests that the rates of estimation achieved by ⇧A ⇤ and ⇧ à are about the same order. Finally à seems to have a slightly faster rate of estimation than ⇧ Ã, so in practice à could be used to estimate A. However we refrain from making an explicit conjecture about the rate.

Discussion

While computational aspects of the seriation problem have received significant attention, the robustness of this problem to noise was still unknown to date. To overcome this limitation, we have introduced in this chapter the statistical seriation model and studied optimal rates of estimation by showing, in particular, that the least-squares estimator enjoys several desirable statistical properties such as adaptivity and minimax optimality (up to logarithmic terms).

While this work paints a fairly complete statistical picture of the statistical seriation model, it also leaves many unanswered questions. There are several logarithmic gaps in the bounds. In the case of adaptive bounds, some logarithmic terms are unavoidable as illustrated by Theorem 12 (for the permutation term) and also by statistical dimension consideration explained by [START_REF] Bellec | Sharp oracle inequalities for least squares estimators in shape restricted regression[END_REF] (for the estimation term). However, a more refined argument for the uniform bound, namely one that uses covering in `2-norm rather than `1-norm, would allow us to remove the log n factor from the estimation term in the upper bound of Corollary 9. Such an argument is provided by [START_REF] Birman | Piecewise polynomial approximations of functions of classes W p[END_REF], [START_REF] Anuchina | Teoreticheskie osnovy i konstruirovanie chislennykh algoritmov zadach matematicheskoȋ fiziki[END_REF], van de Geer [1991] for the larger class of vectors with bounded total variation [see, e.g., Mammen and van de Geer, 1997] but we do not pursue sharp logarithmic terms in this work. For the permutation term, log n in the upper bound of Corollary 8 and log l in the lower bound of Theorem 12 do not match if l < n. We do not seek answers to these questions in this chapter but note that their answers may be different for the unimodal and the monotone case.

Perhaps the most pressing question is that of computationally efficient estimators. Indeed, while statistically optimal, the least-squares estimator requires searching through n! permutations, which is not realistic even for problems of moderate size, let alone genomics applications. We gave a partial answer to this question in the specific context of monotone columns by proposing and studying the performance of a simple and efficient estimator called RankScore. This study reveals the existence of a potentially intrinsic gap between the statistical performance achievable by efficient estimators and that achievable by estimators with access to unbounded computation. A similar gap is also observed in the SST model for pairwise comparisons by [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF]. We conjecture that achieving optimal rates of estimation in the seriation model is computationally hard in general but argue that the planted clique assumption that has been successfully used to establish statistical vs. computational gaps by [START_REF] Berthet | Complexity theoretic lower bounds for sparse principal component detection[END_REF], [START_REF] Ma | Computational barriers in minimax submatrix detection[END_REF], [START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF] for example, is not the correct primitive. Instead, one has to seek for a primitive where hardness comes from searching through permutations rather than subsets.

6.A.1 A Variational Formula for the Error of the LS Estimator

We begin the proof by stating an extension of Chatterjee's variational formula. While we only need this lemma to hold for a union of closed convex sets we present a version that holds for all closed sets. The latter extension was suggested to us by Pierre C. [START_REF] Bellec | [END_REF]

⇣ ka a ⇤ k 2 2 2ha a ⇤ , zi + kzk 2 2 ⌘ = argmax a2C ⇣ ha a ⇤ , zi 1 2 ka a ⇤ k 2 2 ⌘ .
Together with the definition of f a ⇤ , this implies that

f a ⇤ (kâ a ⇤ k 2 ) hâ a ⇤ , zi 1 2 kâ a ⇤ k 2 2 sup a2C\B d (a ⇤ ,t) ⇣ ha a ⇤ , zi 1 2 ka a ⇤ k 2 2 ⌘ sup a2C\B d (a ⇤ ,t) ha a ⇤ , zi t 2 2 = f a ⇤ (t) .
Therefore Eq. ( 6.13) follows. Furthermore, suppose that there is t

⇤ > 0 such that f a ⇤ (t) < 0 for all t t ⇤ . Since f a ⇤ (kâ a ⇤ k 2 ) f a ⇤ (0) = 0, we have kâ a ⇤ k 2  t ⇤ .
Note that this structural result holds for any error vector z 2 IR d and any closed set C which is not necessarily convex. In particular, this extends the results by [START_REF] Chatterjee | A new perspective on least squares under convex constraint[END_REF] and [START_REF] Chatterjee | Adaptive risk bounds in unimodal regression[END_REF] which hold for convex sets and finite unions of convex sets respectively.

6.A.2 Proof of Theorem 10

For our purpose, we need a standard chaining bound on the supremum of a sub-Gaussian process that holds in high probability. The interested readers can see the proof, for example, by van Handel [2014, Theorem 5.29], and refer to the monograph of [START_REF] Ledoux | Probability in Banach Spaces[END_REF] for a more detailed account of the technique.

Lemma 46 (Chaining tail inequality). Let ⇥ ⇢ IR d and z ⇠ subG( 2) in IR d . For any ✓ 0 2 ⇥, it holds that

sup ✓2⇥ h✓ ✓ 0 , zi  C Z diam(⇥) 0 p log N (⇥, k • k 2 , ") d" + s with probability at least 1 C exp( cs 2 2 diam(⇥) 2 )
where C and c are positive constants. Let à 2 U m . To ligthen the notation, we define two rates of estimation:

R 1 = R 1 ( Ã, n) = ⇣ r K( Ã) log enm K( Ã) + p n log n ⌘ (6.14) and R 2 = R 2 ( Ã, n) = 2 ⇣ K( Ã) log enm K( Ã) + n log n ⌘ . (6.15) Note that R 2  R 2 1  2R 2 . Lemma 47. Suppose Y = A ⇤ +Z where A ⇤ 2 IR n⇥m and Z ⇠ subG( 2 ). For à 2 U m and all t > 0, define f Ã(t) = sup A2M\B nm ( Ã,t) hA Ã, Y Ãi t 2 2
.

Then for any s > 0, it holds simultaneously for all t > 0 that 6.16)

f Ã(t)  CR 1 t + tkA ⇤ Ãk F t 2 2 + st (
with probability at least 1 C exp( cs 2
2 ), where C and c are positive constants. Proof. Define ⇥ = ⇥ M ( Ã, 1) = S 0 {B Ã : B 2 M \ B nm ( Ã, 1)} (see also Definition in Eq. ( 6.22)). In particular, ⇥ ⇢ B nm (0, 1) and 0 2 ⇥. Since M is a finite union of convex cones and thus is star-shaped, by scaling invariance,

sup A2M\B nm ( Ã,t) hA Ã, Zi = t sup B2M\B nm (t 1 Ã,1) hB t 1 Ã, Zi  t sup M 2⇥ hM, Zi.
By Lemma 46, with probability at least

1 C exp( cs 2 2 ), sup M 2⇥ hM, Zi  C Z 2 0 p log N (⇥, k • k F , ") d" + s .
Moreover, it follows from Lemma 54 that log N (⇥, k • k F , ")  C" 1 K( Ã) log enm K( Ã) + n log n .

and thus

k ⇧ Â A ⇤ k 2 F . kA ⇤ Ãk 2 F + 2 K( Ã) log enm K( Ã) + 2 n log n .
Finally, Eq. ( 6.7) follows by taking the infimum over à 2 U m on the right-hand side and dividing both sides by nm.

Next, to prove the bound in expectation, observe that Eq. ( 6.17) yields

IP h k ⇧ Â A ⇤ k 2 F C(R 2 + kA ⇤ Ãk 2 F ) s i  C exp( cs 2 )
, where R 2 is defined in Eq. (6.15). Integrating the tail probability, we get that

Ek ⇧ Â A ⇤ k 2 F C(R 2 + kA ⇤ Ãk 2 F ) . Z 1 0 exp( cs
2 ) ds = 2 c
and therefore

Ek ⇧ Â A ⇤ k 2 F . R 2 + kA ⇤ Ãk 2
F . Dividing both sides by nm and minimizing over à 2 U m yields Eq. ( 6.8).

6.A.3 Proof of Theorem 11

In the setting of isotonic regression, [START_REF] Bellec | Sharp oracle bounds for monotone and convex regression through aggregation[END_REF] derived global bounds from adaptive bounds by a block approximation method, which also applies to our setting. [START_REF] Bellec | Sharp oracle bounds for monotone and convex regression through aggregation[END_REF] and their proof also extends to the unimodal case with minor modifications. We present the result with proof for completeness. consider the intervals

I j = h a + j 1 k V (a), a + j k V (a)
i , and I k = h a + k 1 k V (a), āi . Also for j 2 [k], let J j = {i 2 [n] : a i 2 I j }. We define the vector ã 2 IR n by ãi = a + j 1/2 k V (a) for i 2 [n], where j is uniquely determined by i 2 I j . Since a is increasing on {1, . . . , i 0 } and decreasing {i 0 , . . . , n}, so is ã. Thus ã 2 U k . Moreover, |ã i a i |  V (a) 2k for i 2 [n], which implies Eq. ( 6.18). Next we prove the latter two assertions. Since k ⇤ = d V (a) Proof. Applying Lemma 48 to columns of A, we see that there exists à 2 U m k ⇤ such

6.B.1 Cartesian Product of Cones

Lemma 51 below bounds covering numbers of product spaces and is useful in later proofs. We start with a well-known result on the covering number of a Euclidean ball with respect to the `1-norm [see, e.g., Massart, 2007, Lemma 7.14, for an analogous result].

Lemma 50. For any " 2 (0, 1],

N ⇣ B m (0, 1), k • k 1 , " p m ⌘  (C/") m ,
for some constant C > 0.

Proof. We aim at bounding the covering number of a Euclidean ball by cubes. Let {x 1 , . . . , x M } be a maximal " p m -packing of B m (0, 1) with respect to the `1-norm, where a -packing of a set G with respect to a norm k • k is a set {w 1 , • • • , w N } ⇢ G such that kw i w j k for all distinct i, j 2 [N ]. Then this set is necessarily an " p m -net of B m (0, 1) by maximality, so N (B m (0, 1), k • k 1 , " p m )  M . Consider the cubes with side length " p m centered at x i for 1  i  M . These cubes are disjoint and contained in the set B m (0, 1) + Q m ( " p m ), where Q m ( " p m ) is the cube with side length "

p m centered at the origin in IR m . Since Q m ( " p m ) ⇢ B m (0, "),

M Vol ⇣ Q m " p m ⌘  Vol ⇣ B m (0, 1) + Q m " p m ⌘  Vol(B m (0, 1 + "))  Vol(B m (0, 2)).
This proves the following bound on the covering number in terms of a volume ratio:

N ⇣ B m (0, 1), k • k 1 , " p m ⌘  Vol(B m (0, 2)) Vol(Q m ( " p m ))  C m m m/2
" m m m/2 = (C/") m .

Now we study the metric entropy of a Cartesian product of convex cones. Let {I i } m i=1 be a partition of [n] with |I i | = n i and P m i=1 n i = n. For a 2 IR n , the restriction of a to the coordinates in I i is denoted by a I i 2 IR n i . Let C i be a convex cone in IR n i and C = C 1 ⇥ • • • ⇥ C m . Lemma 51. With the notation above, suppose that a I i 2 C i \ ( C i ). Then for any t > 0 and " 2 (0, t],

log N C \ B n (a, t), k • k 2 , "  m log Ct " + m X i=1 log N ⇣ C i \ B n i (a I i , t), k • k 2 , " 3
⌘
for some constant C > 0.

Proof. Since a product of balls B n 1 (0, " p m ) ⇥ • • • ⇥ B nm (0, " p m ) is contained in B n (0, "), one could try to cover C \ B n (a, t) by such products of balls. It turns out that this yields an upper bound of order m 3/2 , which is too loose for our purpose. Fortunately, the following argument corrects this dependency.

Without loss of generality, we assume that t = 1. We construct a 3"-net of C \ B n (a, 1) as follows. First, let N B be an " Note that µ i + 1 p m > 0 for µ 2 N D , and let N µ i be a

(µ i + 1 p m )"-net of C i \B n i (a I i , µ i + 1 p m ). Define N µ = N µ 1 ⇥ • • • ⇥ N µm , i
|N µ i | = N ⇣ C i \ B n i 0, µ i + 1 p m , k • k 2 , µ i + 1 p m " ⌘ = N (C i \ B n i (0, 1), k • k 2 , ") = N (C i \ B n i (a I i , 1), k • k 2 , ") . As |N µ |  Q m i=1 |N µ i |, therefore [ µ2N D N µ  ⇣ C " ⌘ m m Y i=1 N (C i \ B n i (a I i , 1), k • k 2 , ") .
Taking the logarithm completes the proof.

6.B.2 Unimodal Vectors and Matrices

Recall that S n denotes the closed convex cone of increasing vectors in IR n . First, we prove a result on the metric entropy of S n intersecting with a ball using Lemma 51.

Lemma 52. Let b 2 IR n be such that b 1 = • • • = b n . Then for any t > 0 and " > 0, log N (S n \ B n (b, t), k • k 2 , ")  C" 1 t log(en). Proof. The majority of the proof is due to Lemma 5.1 in an old version of the article of [START_REF] Chatterjee | Adaptive risk bounds in unimodal regression[END_REF], but we improve their result by a factor p log n and provide the whole proof for completeness.

The bound holds trivially if " > t, since the left-hand side is zero. It also clearly holds when n = 1. Hence we can assume without loss of generality that "  t and n = 2n 0 2. Moreover, assume that t = 1 for simplicity and the proof will work for any t > 0. Let I = {1, . . . , n 0 } and observe that For each a 2 S n \B n (0, 1), it holds that |a i |  1 p i for i 2 I (since either |a l | |a i | for all l  i or |a l | |a i | for all i  l  n; see e.g. [START_REF] Dai | Aggregation of affine estimators[END_REF]), so max i2A j |a i |  2 j/2 . Also m j  2 j , so we get that Proof. Assume that "  t since otherwise the left-hand side is zero and the bound holds trivially. For j 2 [m], define I j,1 = [l j ] and I j,2 = [n]\[l j ]. Define k j,1 = k(A I j,1 ,j ) and k j,2 = k(A I j,2 ,j ). Let { = P m j=1 (k j,1 + k j,2 ) and observe that K(A)  {  2K(A). Moreover, let {I j,1 1 , . . . , I j,1 k j,1 } be the partition of I j,1 such that A I j,1 i ,j is a constant vector for i 2 [k j,1 ]. Note that elements of I j,1 i need not to be consecutive. Define the partition for I j,2 analogously.

log N ⇣ S m j \ B m j (b A j , 1), k • k 2 , " 3
For j 2 [m] and i 2 [k j,1 ] (resp. [k j,2 ]), let S I j,1 i ,j (resp. S I j,2 i ,j ) denote the set of increasing (resp. decreasing) vectors in the component space IR |I j,1 i | (resp. IR |I j,2 i | ). Lemma 52 implies that log N (S I j,r i ,j \ B |I j,r i | (A I j,r i ,j , t), k • k F , ")  C" 1 t log(e|I j,r i |).

As a matrix in IR n⇥m can be viewed as a concatenation of { = P m j=1 (k j,1 + k j,2 ) vectors of length |I j,r i |, r 2 [2], j 2 [m], we define the cone S ⇤ in IR n⇥m by

S ⇤ = Q m j=1 Q 2 r=1 Q k j,r
i=1 S I j,r i ,j , which is clearly a superset of C m l . It also follows that A 2 S ⇤ \ ( S ⇤

), and thus by Lemma 51 and the previous display,

log N (S ⇤ \ B nm (A, t), k • k F , ")  { log Ct " + m X j=1 2 X r=1 k j,r X i=1 C" 1 t log(e|I j,r i |)  C" 1 t { + C" 1 t { log e P j,r,i |I j,r i | {  C" 1 t K(A) log enm K(A) ,
where we used the concavity of the logarithm and Jensen's inequality in the second step, and that K(A)  {  2K(A) in the last step. Since A 2 S ⇤ \( S ⇤ ) (the cone S ⇤ is pointed at A) we have that S ⇤ \B nm ( A, t) [Tsybakov, 2009, Theorem 2.5] with ↵ = 1 10 gives the conclusion.

6.C.1 Proof of Theorem 12

We define

U m K 0 (V 0 ) = U m K 0 \ U m (V 0 ) and M K 0 (V 0 ) = S ⇧2Sn ⇧ U m K 0 (V 0 )
. Define the subset of M K 0 (V 0 ) containing permutations of monotonic matrices by for some c, c 0 > 0, where IP M is the probability with respect to Y = M + Z. This bound remains valid for the parameter subset M S K 0 (V 0 ) if l = 1 or 2.

M S K 0 (V 0 ) = {⇧A 2 M K 0 (V 0 ) : ⇧ 2 S n , A 2
Note that the bound clearly holds for the larger parameter space M K 0 = S ⇧2Sn ⇧U m K 0 . By taking l = min(K 0 m, m) + 1 and V 0 large enough, we see that the assumption in Proposition 21 is satisfied and the second term becomes simply 2 m log l, so Theorem 12 follows. In the monotonic case, by the last statement of the proposition, if K 0 m + 1 then taking l = 2 and V 0 large enough yields a lower bound of rate 2 ( K 0 nm + 1 m ) for the set of matrices A with increasing columns and K(A)  K 0 . The proof of Proposition 21 has two parts which correspond to the two terms respectively. First, the term 2 K 0 nm is derived from the proof of lower bounds for isotonic regression by [START_REF] Bellec | Sharp oracle bounds for monotone and convex regression through aggregation[END_REF]. Then we derive the other term 1. We can permute the rows of M 0 so that each column has consecutive ones (or all zeros), so M 2 M. Moreover, Since log l 1 2 log(l + 1) for l 2, applying Lemma 56 completes the proof.

Combining Lemma 57, 58 and 60, and dividing the bound by nm, we get Eq. ( 6.23) because the max of two terms is lower bounded by a half of their sum. The last statement in Proposition 21 holds since Lemma 57 and 58 are proved for matrices with increasing columns.

6.C.2 Proof of Theorem 13

The proof will only use Lemma 57 and 58, so the lower bound of rate ( ) is achieved by Lemma 58, so we focus on the tradeoff between the first two terms. Suppose that ( 16n

2 ) 1/3 V ). Then the proof of Theorem 10 implies that with probability at least 1 e c(n+m) ,

k ⇧ Ã A ⇤ k 2 F . 2 K(A) log enm K(A) + k ⇧A A ⇤ k 2 F . 2 K(A) log enm K(A) + kA A ⇤ k 2 F + k ⇧A ⇤ A ⇤ k 2 F .
Minimizing over A 2 S m yields the desired result.

The idea of splitting the error into two terms as in Lemma 61 has appeared in previous works by [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF], [START_REF] Chatterjee | On estimation in tournaments and graphs under monotonicity constraints[END_REF].

6.D.2 Permutation

By virtue of Lemma 61, it remains to control the permutation error k ⇧A ⇤ ⇧ ⇤ A ⇤ k 2

F
where ⇧ is given by the RankScore procedure defined in Section 6.4. Recall that for i, i 0 2 [n],

A ⇤ (i, i 0 ) = max j2[m] (A ⇤ i 0 ,j A ⇤ i,j ) _ 1 p m m X j=1 (A ⇤ i 0 ,j A ⇤ i,j )
and Y (i, i 0 ) is defined analogously. Since columns of A ⇤ are increasing, (6.25) Recall that the RankScore procedure is defined as follows. First, for i 2 [n], we associate with the i-th row of Y a score s i defined by

A ⇤ (i, i 0 ) = kA ⇤ i 0 ,• A ⇤ i,• k 1 _ 1 p m kA ⇤ i 0 ,• A ⇤ i,• k 1 .
s i = n X l=1
1I( Y (l, i) 2⌧ ) (6.26) for the threshold ⌧ := 3 p log(nm 1 ) where is the probability of failure. Then we order the rows of Y so that the scores are increasing with ties broken arbitrarily. This is equivalent to requiring that the corresponding permutation ⇡ : [n] ! [n] satisfies that if s i < s i 0 then ⇡ 1 (i) < ⇡ 1 (i 0 ). Define ⇧ to be the n ⇥ n permutation matrix corresponding to ⇡ so that ⇧⇡(i),i = 1 for i 2 [n] and all other entries of ⇧ are zero. Moreover, let ⇡ ⇤ : [n] ! [n] be the permutation corresponding to ⇧ ⇤ . To control the permutation error, we first state a lemma which asserts that if the gap between two rows of A ⇤ is sufficiently large, then the permutation defined above will recover their relative order with high probability.

Lemma 62. There is an event E of probability at least 1 on which the following holds. For any i, i

0 2 [n], if A ⇤ (i, i 0 ) 4⌧ , then ⇡ 1 ⇡ ⇤ (i) < ⇡ 1 ⇡ ⇤ (i 0 ).
Proof. Since Z ⇠ subG( 2), Z i,j and 1 p m P m j=1 Z i,j are sub-Gaussian random variables with variance proxy 2 . A standard union bound yields that max

⇣ max i2[n],j2[m] |Z i,j |, max i2[n] 1 p m m X j=1 Z i,j ⌘  ⌧ = 3 p log(nm 1 )
on an event E of probability at least 1 2(nm + n) exp( ⌧ 2 2 2 ) 1 . In the sequel, we make statements that are valid on the event E. Since Y ⇡ ⇤ (i),j = A ⇤ i,j + Z i,j , by the triangle inequality, (6.27) Suppose that A ⇤ (i, i 0 ) 4⌧ . We claim that s ⇡ ⇤ (i) < s ⇡ ⇤ (i 0

| Y (⇡ ⇤ (i), ⇡ ⇤ (i 0 )) A ⇤ (i, i 0 )|  2⌧.
) . If Y (⇡ ⇤ (l), ⇡ ⇤ (i)) 2⌧ , for l 2 [n], then A ⇤ (l, i) 0 by Eq. (6.27). Since A ⇤ has increasing columns, A ⇤ (l, i 0 ) 4⌧ . Again by Eq. ( 6.27), Y (⇡ ⇤ (l), ⇡ ⇤ (i 0 )) 2⌧ . By definition in Eq. ( 6.26), we see that s ⇡ ⇤ (i)  s ⇡ ⇤ (i 0

) . Moreover, A ⇤ (i, i 0 ) 4⌧ so Y (⇡ ⇤ (i), ⇡ ⇤ (i 0 )) 2⌧ . Therefore s ⇡ ⇤ (i) < s ⇡ ⇤ (i 0

) . According to the construction of ⇡, ⇡ 1 ⇡ ⇤ (i) < ⇡ 1 ⇡ ⇤ (i 0 ).

Next, recall that for a matrix A 2 S m , J denotes the set of pairs of indices (i, j) 2 [n] 2 such that A i,• and A j,• are not identical. The quantity R(A) is defined by

R(A) = 1 n max I⇢[n] 2 |I|=n X (i,j)2I\J ⇣ kA i,• A j,• k 2 2 kA i,• A j,• k 2 1 ^mkA i,• A j,• k 2 2 kA i,• A j,• k 2 1 ⌘ .
For any nonzero vector u 2 IR m , kuk 2 2 /kuk 2 1 1 with equality achieved when kuk 0 = 1, and kuk 2 2 /kuk 2 1 m 1 with equality achieved when all entries of u are the same. Hence R(A)

1. Moreover, kuk 2 2  kuk 1 kuk 1 by Hölder's inequality, so Intuitively, the quantity R(A) is small if the difference of any two rows of A is either very sparse or very dense.

Lemma 63. There is an event E of probability at least 1 on which

k ⇧A ⇤ ⇧ ⇤ A ⇤ k 2 F . 2 R(A ⇤
) n log(nm 1 ) .

Proof. Throughout the proof, we restrict ourselves to the event E defined in Lemma 62.

To simplify the notation, we define (6.28) where I is the set of indices i for which ↵ i is nonzero. For each i 2 I, 6.29) by Eq. ( 6.25).

↵ i = A ⇤ ⇡ 1 ⇡ ⇤ (i),• A ⇤ i,• . Then k ⇧A ⇤ ⇧ ⇤ A ⇤ k 2 F = n X i=1 kA ⇤ ⇡(i),• A ⇤ ⇡ ⇤ (i),• k 2 2 = X i2I k↵ i k 2 2 ,
k↵ i k 2 2 = min ⇣ k↵ i k 2 2 k↵ i k 2 1 , mk↵ i k 2 2 k↵ i k 2 1 ⌘ • max ⇣ k↵ i k 2 1 , k↵ i k 2 1 m ⌘ = min ⇣ k↵ i k 2 2 k↵ i k 2 1 , mk↵ i k 2 2 k↵ i k 2 1 ⌘ • A ⇤ i, ⇡ 1 ⇡ ⇤ (i) 2 (
Next, we proceed to showing that | A ⇤ (i, ⌫(i))|  4⌧ for any i 2 [n], where ⌫ = ⇡ 1 ⇡ ⇤ . To that end, note that if A ⇤ (i, ⌫(i)) > 4⌧ , in which case A ⇤ (i, i 0 ) > 4⌧ for all i 0 2 I 0 := {i 0 2 [n] : i 0 ⌫(i)}, then it follows from Lemma 62 that on E, ⌫(i) < ⌫(i 0

), 8 i 2 I 0 . Note that |⌫(I 0 )| = |I 0 | = n ⌫(i)+1. Hence ⌫(i) < ⌫(i 0 ), 8 i 2 I 0 implies that ⌫(i)  n |⌫(I 0 )| = ⌫(i) 1, which is a contradiction. Therefore, there does not exist such i 2 [n] on E. The case where A ⇤ (i, ⌫(i)) < 4⌧ is treated in a symmetric manner.

Combining this bound with Eq. (6.28) and Eq. ( 6.29), we conclude that

k ⇧A ⇤ ⇧ ⇤ A ⇤ k 2 F . X i2I min ⇣ k↵ i k 2 2 k↵ i k 2 1 , mk↵ i k 2 2 k↵ i k 2 1 ⌘ • ⌧ 2 . 2 R(A ⇤
) n log(nm 1 ) .

by the definitions of R(A ⇤ ) and ⌧ .

6.D.3 Proof of Theorem 14

The bound is an immediate consequence of Lemma 61 and Lemma 63 with = (nm) C for C > 0.

6.E Upper bounds in a Trivial Case

In Theorem 13, we have observed the term 2 m ^m2 V (A) 2 , whereas the LS estimator only has 1 exp( m). On this event,

k ⇧ Â ⇧ ⇤ A ⇤ k 2 F . nm 3 V (A) 2 + m 2 .
Dividing the previous two displays by nm completes the proof.

6.F Unimodal Regression

If the permutation in the main model in Eq. ( 6.1) is known, then the estimation problem simply becomes a concatenation of m unimodal regressions. In fact, our proofs imply new oracle inequalities for unimodal regression. Recall that U denotes the cone of unimodal vectors in IR n . Suppose that we observe

y = ✓ ⇤ + z ,
where ✓ ⇤ 2 IR n and z is a sub-Gaussian vector with variance proxy The corresponding bounds in expectation also hold.

Proof. The proof closely follows that of Theorem 10 and Theorem 11. First note that the term n log n in the bound of Lemma 54 comes from a union bound applied to the set of permutations, so it is not present if we consider only the set of unimodal matrices U m instead of M. Hence taking m = 1 in the lemma yields that

log N ⇥ U ( ✓, t), k • k 2 , "  C" 1 t k( ✓) log en k( ✓) .
For ✓ 2 U, define

f ✓(t) = sup ✓2U \B n ( ✓,t)
h✓ ✓, y ✓i t 2

2

.

Following the proof of Lemma 47 and using the above metric entropy bound, we see ). Then the proof of Theorem 10 gives that with probability at least 1 C exp( cs 2

2 ),

k ✓ ✓ ⇤ k 2  C ⇣ r k( ✓) log en k( ✓) + k ✓ ✓ ⇤ k 2 ⌘ + 2s .
Taking s = C p ↵ log n for ↵ 1 and C sufficiently large, we get that with probability at least

1 n ↵ , k ✓ ✓ ⇤ k 2 2 . 2 k( ✓) log en k( ✓) + k ✓ ✓ ⇤ k 2 2 + ↵ 2 log n .
Minimizing over ✓ 2 U yields Eq. ( 6.30). The corresponding bound in expectation follows from integrating the tail probability as in the proof of Theorem 10. Finally, we can apply the proof of Theorem 11 with m = 1 to achieve the global bound.

Note that the bounds in Corollary 10 match the minimax lower bounds for isotonic regression of [START_REF] Bellec | Sharp oracle bounds for monotone and convex regression through aggregation[END_REF] up to logarithmic factors. Since every monotonic vector is unimodal, lower bounds for isotonic regression automatically hold for unimodal regression. Therefore, we have proved that the LS estimator is minimax optimal up to logarithmic factors for unimodal regression.

A result similar to Eq. ( 6.30) was obtained by [START_REF] Bellec | Sharp oracle inequalities for least squares estimators in shape restricted regression[END_REF] in its revision that was prepared independently and contemporaneously to this chapter. [START_REF] Chatterjee | Adaptive risk bounds in unimodal regression[END_REF] also improved their bounds to having optimal exponents after the first version of our current paper was posted. Interestingly [START_REF] Bellec | Sharp oracle inequalities for least squares estimators in shape restricted regression[END_REF] employs bounds on the statistical dimension by leveraging results from [START_REF] Amelunxen | Living on the edge: phase transitions in convex programs with random data[END_REF], and [START_REF] Chatterjee | Adaptive risk bounds in unimodal regression[END_REF] use both the variational formula and the statistical dimension. Moreover, their results are presented in the well-specified case where ✓ ⇤ 2 U and ✓ = ✓ ⇤ .

Chapter 7

Conclusion and Future Work

Summary of the Thesis

In this thesis, we focused on some problems specific to optimization of quadratic functions and their applications to machine learning.

In our first contribution we provided a unified framework for minimizing nonstrongly convex quadratic functions, both with noiseless and noisy gradients. It encompassed averaged gradient descent, accelerated gradient descent and the heavy ball method. They were jointly analyzed as constant parameter second-order difference equation algorithms, where stability of the system was equivalent to convergence at rate O(1/n 2). This suggested a new class of algorithms that could profit at the same time from the known enhancements of averaging and acceleration: faster forgetting the initial conditions (for acceleration), and improved robustness to noise when the noise covariance was proportional to the Hessian (for averaging).

In our second main contribution, we continued this pursuit and showed that stochastic averaged accelerated gradient descent was robust to structured noise in the gradients present in least-squares regression. Consequently we proposed the first algorithm that achieved simultaneously the optimal prediction error rates for leastsquares regression, both in terms of forgetting the initial conditions in O(1/n 2 ), and in terms of dependence on the noise and dimension d of the problem, as O( 2 d/n). Furthermore we also provided an analysis adapted to finer assumptions such as fast decays of the covariance matrices or optimal predictors with large norms leading to dimension-free quantities that may still be small in some distances while the "optimal" terms above were large.

In our third main contribution we extended the problem studied beforehand to composite settings where the objective function is composed of the expectation of quadratic functions and an arbitrary convex function. We showed that stochastic dual averaging algorithm with a constant step-size achieved a convergence rate of O(1/n) for stochastic composite objectives, without strong convexity assumptions. Accordingly we widened the previous results on least-squares regression of [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] to all convex regularizers and all geometries which may be represented by a Bregman divergence.

In our fourth main contribution we considered the problem of clustering highdimensional data by finding a low-dimensional projection of the data which was wellclustered. We related this formulation to the discriminative clustering framework with the square loss for which we proposed a novel sparse extension and provided the first theoretical analysis. We also proposed a new efficient algorithm with an improved linear complexity in the number of observations and a natural extension to the multi-label scenario.

In our final contribution we considered the seriation problem which consists in permuting the rows of an observed matrix in such way that all its columns have the same shape. When the matrix was observed with additional random noise, we analyzed the minimax rates of estimation and we also designed a computationally efficient estimator in case the columns of the initial matrix were monotone increasing. A theoretical and experimental studies of this estimator were also provided.

Perspectives

Our work has triggered a few questions, which are still open. 1. Our current analysis of algorithms presented in Chapter 2 and Chapter 3 is only provided for additive noise, e.g., for least-squares regression, with knowledge of the population covariance matrix. This drawback raises two different issues: (a) common applications use the stochastic oracle with a multiplicative noise, (b) the stochastic oracle with additive noise has a computational complexity O(d 2 ) (for multiplying the Hessian and the iterate). Thus the total running-time complexity is O( 2 d 3 /n) which is comparable to the one obtained by minimizing the empirical risk with the conjugate gradient. Nevertheless [START_REF] Jain | Accelerating stochastic gradient descent[END_REF] have recently provided a lower-bound which prevents from directly extending our results to least-squares regression with multiplicative noise without assuming extra assumptions. This will lead to future work. 2. Algorithms studied in Chapter 2, Chapter 3 and Chapter 4 only work with quadratic functions. In fact, these constant step-size stochastic algorithms are not converging for general smooth objectives. However each of these methods may be extended to non-quadratic functions through online Newton algorithm [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] which iteratively solves quadratic approximations of the smooth problems with the algorithm to achieve the same rate O(1/n).

Research on connections between discrete and continuous dynamics has recently

gained interest in the optimization community. For the moment it enables to have a deeper intuition on accelerated gradient descent by drawing links between the proofs of convergence of the discrete and continuous dynamics. One of the strength of continuous dynamics is that their corresponding Lyapunov functions are significantly easier to design. This advantage can be leveraged to provide Lyapunov functions for discrete dynamics by discretizing the one corresponding to the continuous dynamic. Therefore further investigating these connections may yield to systematic proofs of convergence of optimization algorithms. This may be used to prove the convergence of online Newton algorithm [START_REF] Bach | Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)[END_REF] by considering its connection with continuous dynamics studied by [START_REF] Alvarez | A second-order gradient-like dissipative dynamical system with Hessian-driven damping. Application to optimization and mechanics[END_REF], Attouch et al. [2016b]. On the other hand an extension to stochastic approximation and corresponding stochastic differential equations would provide great insights by studying and discretizing the resulting diffusions. 4. While stochastic gradient descent is broadly used by practitioners and intensively studied by theoreticians, the choice of its step-size remains heuristic.

Yet its performance depends highly on how the step-size is tuned and decreased over time. For quadratic functions, we studied the behavior of constant step-size which is easy to calibrate. However for general convex functions the stochastic gradient descent with constant step-size is not converging and the choice of its decrease remains challenging. Thus parameter-free methods are very popular [see, e.g., [START_REF] Kingma | A method for stochastic optimization[END_REF] and deserve further investigations. 5. Designing better stochastic algorithms for non-convex optimization is also very challenging. It could take several forms to avoid saddle-points and insure convergence to local minima: (a) by exploiting second order information (b) by adding Gaussian noise to the gradient estimate in stochastic gradient descent as in Langevin algorithms [START_REF] Dalalyan | Theoretical guarantees for approximate sampling from a smooth and log-concave density[END_REF][START_REF] Durmus | Non-asymptotic convergence analysis for the unadjusted langevin algorithm[END_REF][START_REF] Durmus | Stochastic gradient Richardson-Romberg markov chain monte carlo[END_REF]]. 6. We have only provided a computationally efficient estimator for statistical seriation in the simple case of matrix with monotonic columns. We conjectured that achieving optimal rates of estimation in the seriation model is computationally hard in general. However we could also investigate approximate message passing [Donoho et al., 2009, Mézard andMontanari, 2009], and auto-encoder [START_REF] Hazan | A non-generative framework and convex relaxations for unsupervised learning[END_REF] techniques which might efficiently solved this problem.

257

Résumé

De multiples problèmes en apprentissage automatique consistent à minimiser une fonction lisse sur un espace euclidien. Pour l'apprentissage supervisé, cela inclut les régressions par moindres carrés et logistique. Si les problèmes de petite taille sont résolus e cacement avec de nombreux algorithmes d'optimisation, les problèmes de grande échelle nécessitent en revanche des méthodes du premier ordre issues de la descente de gradient.

Dans ce manuscrit, nous considérons le cas particulier de la perte quadratique. Dans une première partie, nous nous proposons de la minimiser grâce à un oracle stochastique. Dans une seconde partie, nous considérons deux de ses applications à l'apprentissage automatique : au partitionnement de données et à l'estimation sous contrainte de forme.

La première contribution est un cadre unifié pour l'optimisation de fonctions quadratiques non-fortement convexes. Celui-ci comprend la descente de gradient accélérée et la descente de gradient moyennée. Ce nouveau cadre suggère un algorithme alternatif qui combine les aspects positifs du moyennage et de l'accélération.

La deuxième contribution est d'obtenir le taux optimal d'erreur de prédiction pour la régression par moindres carrés en fonction de la dépendance au bruit du problème et à l'oubli des conditions initiales. Notre nouvel algorithme est issu de la descente de gradient accélérée et moyennée.

La troisième contribution traite de la minimisation de fonctions composites, somme de l'espérance de fonctions quadratiques et d'une régularisation convexe. Nous étendons les résultats existants pour les moindres carrés à toute régularisation et aux di érentes géométries induites par une divergence de Bregman.

Dans une quatrième contribution, nous considérons le problème du partitionnement discriminatif. Nous proposons sa première analyse théorique, une extension parcimonieuse, son extension au cas multi-labels et un nouvel algorithme ayant une meilleure complexité que les méthodes existantes.

La dernière contribution de cette thèse considère le problème de la sériation. Nous adoptons une approche statistique où la matrice est observée avec du bruit et nous étudions les taux d'estimation minimax. Nous proposons aussi un estimateur computationellement e cace.

Mots Clés

Optimisation convexe, accélération, moyennage, gradient stochastique, régression par moindres carrés, approximation stochastique, algorithme dual moyenné, descente miroire, partitionnement discriminatif, relaxation convexe, parcimonie, sériation statistique, apprentissage de permutation, estimation minimax, contraintes de forme.

Abstract

Many problems in machine learning are naturally cast as the minimization of a smooth function defined on a Euclidean space. For supervised learning, this includes least-squares regression and logistic regression. While small problems are e ciently solved by classical optimization algorithms, large-scale problems are typically solved with first-order techniques based on gradient descent.

In this manuscript, we consider the particular case of the quadratic loss. In the first part, we are interested in its minimization when its gradients are only accessible through a stochastic oracle. In the second part, we consider two applications of the quadratic loss in machine learning: clustering and estimation with shape constraints.

In the first main contribution, we provided a unified framework for optimizing non-strongly convex quadratic functions, which encompasses accelerated gradient descent and averaged gradient descent. This new framework suggests an alternative algorithm that exhibits the positive behavior of both averaging and acceleration.

The second main contribution aims at obtaining the optimal prediction error rates for least-squares regression, both in terms of dependence on the noise of the problem and of forgetting the initial conditions. Our new algorithm rests upon averaged accelerated gradient descent.

The third main contribution deals with minimization of composite objective functions composed of the expectation of quadratic functions and a convex function. We extend earlier results on least-squares regression to any regularizer and any geometry represented by a Bregman divergence.

As a fourth contribution, we consider the the discriminative clustering framework. We propose its first theoretical analysis, a novel sparse extension, a natural extension for the multi-label scenario and an e cient iterative algorithm with better running-time complexity than existing methods.

The fifth main contribution deals with the seriation problem. We propose a statistical approach to this problem where the matrix is observed with noise and study the corresponding minimax rate of estimation. We also suggest a computationally e cient estimator whose performance is studied both theoretically and experimentally.

Keywords

Convex optimization, acceleration, averaging, stochastic gradient, least-squares regression, stochastic approximation, dual averaging, mirror descent, discriminative clustering, convex relaxation, sparsity, statistical seriation, permutation learning, minimax estimation, shape constraints.

  It appears because the training error is an estimate of the generalization error. It depends on the size of the training set and on the complexity of the class G.

Figure 1 - 2 -

 12 Figure 1-2 -Accelerated gradient descent.

Figure 1 - 4 -

 14 Figure 1-4 -Mirror descent versus dual averaging. Left: mirror descent. Right: dual averaging. We denote by ⇧ h C the Bregman projection on the convex set C defined in the projection step.

Figure 2 - 2 -

 22 Figure 2-2 -Trade-off between averaged and accelerated methods for noisy gradients.

Corollary 2 .

 2 For ↵ = min n min{N↵, 1/L}], we have:

0✓Figure 2 - 3 -

 23 Figure 2-3 -Quadratic optimization with regression noise. Left = 1, r = 1. Right = 0.1, r = 10.

Figure 2 - 4 -

 24 Figure 2-4 -Least-Square Regression. Left = 1, r = 1. Right = 0.1, r = 10.

Figure 2 .Figure 2 .

 22 Figure 2.A.1 -Deterministic case for d = 1 and ↵ = 1/10. Left: classical algorithms, right: different oscillatory behaviors.

Figure 2 .

 2 Figure 2.A.2 -Left: Deterministic quadratic optimization for d = 2. Right: Function value of the projection of the iterate on the different eigenspaces (d = 5).

Figure 2 .

 2 Figure 2.A.3 -Deterministic case for d = 20 and = 1/10. Left: m = 2. Right: m = 8.

Figure 2 .

 2 Figure 2.A.4 -Quadratic optimization with additive noise.

Figure 2 .

 2 Figure 2.C.1 -Stability in the real case, with all contraints plotted.

Figure 2 .C. 2 -

 22 Figure 2.C.2 -Stability in the complex case, with all constraints plotted.

Figure 2 .D. 4 -

 24 Figure 2.D.1 -Validity of Lemma 3

Corollary 5 .

 5 Assume (A 4 ), for any constant step-size (⌃ + I) 4 I, we have for =

Figure 3

 3 Figure 3-1 -Synthetic problem (d = 25) and = 1/R 2 . Left: Bias. Right: Variance.

2 = 1 )

 21 which will illustrate how the variance term behaves. For the bias (left plot of Figure3-1), AvSGD converges at speed O(1/n), while AvAccSGD and AccSGD converge both at speed O(1/n 2

  and we get the following upper-bound on the variance n X k=1 tr ⇥ I

  the iterate. The end of the section is devoted to the proof of this lemma.

  (a) cos(b) + cos(a) sin(b) and regrouping terms,

  Constrained problems. When g is the indicator function of a convex set C then by definition the primal iterate ✓ n 2 C and by convexity ✓n 2 C. Therefore ( ✓n ) = f ( ✓n ) + C ( ✓n ) = f ( ✓n ) and we obtain with the Cauchy-Schwarz inequality:

Figure 4 -

 4 Figure 4-1 -Simplex-constrained least-squares regression with synthetic data. Left: Performance on the objective function. Right: Performance on the Mahalanobis norm k • k 2 ⌃ .

  .23) And we have proven the following minimax oracle complexity.Proposition 14. Let d be the simplex. Then there exists universal constants c 0 > 0 and c > 0 such that the minimax oracle complexity over the class S quad of quadratic functions satisfies the following lower bounds:

2 "

 2 .24) And we have proven the following minimax oracle complexity. Proposition 16. Let d be the simplex. Then there exists universal constant c > 0 such that the minimax oracle complexity over the class S lin of linear functions satisfies the following lower bound for 2 log 2 d  n sup ⇤ n (S lin , d , ) c r log(d) n .

Figure 4 .

 4 Figure 4.K.1 -Test and train performances for `1-regularized least-squares regression on the sido dataset with = opt . Left: test performance. Right: train performance.

Figure 4 .K. 2 -

 42 Figure 4.K.2 -Test and train performances for `1-regularized least-squares regression on the sido dataset with = opt 8 . Left: test performance. Right: train performance.

Figure 4 .

 4 Figure 4.K.3 -Test and train performances for `1-regularized least-squares regression on the sido dataset with = 256 opt . Left: test performance. Right: train performance.

  5, which effectively leads to scalings over the form d = O( p n) for the affine invariant case and d = O(n) for the 1-sparse case. -Finally, we propose in Section 5.6 efficient iterative algorithms with runningtime complexity for each step equal to O(nd 2

Proposition 20 .

 20 Let us assume d 3. (a) If n dR 2 1+(d+ )m 2 m 2

  Phase transition for rank-one solution. Left: affine invariant case. Right: 1-sparse case. Phase transition for clustering error. Left: affine invariant case . Right: 4-sparse case.

Figure 5 -

 5 Figure 5-1 -Phase transition plots. error. This illustrates, in practice, this solver aims to find a rank-one solution under the improved scaling n = O(d 2).

  cvx, max-cut comparison with = 0. Top: n varied with d = 50, k = 6. cvx crashed for n ⇡ 1000. Bottom: d varied with n = 100, k = 2. cvx comparison with = 0.001. Top: n varied with d = 50, k = 6. cvx crashed for n ⇡ 1000. Bottom: d varied with n = 100, k = 2.

Figure 5 -

 5 Figure 5-3 -Scalability experiments.

Figure 5 - 4 -

 54 Figure 5-4 -Comparison with k-means and alternating optimization, n = 100.

Figure 5 - 5 -

 55 Figure 5-5 -Comparison with k-means, alternating optimization and max-margin clustering (mmc), n = 100. The plots for FISTA, cvx and mmc correspond to the best choice of regularization parameters.

Figure 5 - 6 -

 56 Figure 5-6 -Plot of Tr( Ytrue Y k ).

Figure 5 -

 5 Figure 5-7 -Heatmap of correlations, Y k ⇧ n Y true with k = 30, with columns and rows ordered according to the sum of squared correlations.

Lemma 38 .

 38 Let µ 1 and µ 2 be the two smallest eigenvalues of the operator ET x . Let us assume that d 3 (the case d = 2 will also be done in the proof )to µ for which we have set without loss of generality the first component a = 1.

  c min 1 d 1 ] ? where c min is defined in the proof and satisfies |e

  [n] m . Finally, let S n be the set of n⇥n permutation matrices and define M = S ⇧2Sn ⇧U m where ⇧U m = {⇧A : A 2 U m }, so that M is the union of the n!n m closed convex cones ⇧C m l , ⇧ 2 S n , l 2 [n] m .

Figure 6 - 1 -

 61 Figure 6-1 -Estimation errors of three estimators for two deterministic A ⇤ of size n ⇥ n. Left: rows of A ⇤ are 1-sparse; Right: columns of A ⇤ are identical.

Figure 6

 6 Figure 6-3 -Various estimation errors of the oracle and RankScore for the triangular matrix.

For k 2

 2 [n], let U k = a 2 U : Card({a 1 , . . . , a n })  k . below is very similar to Lemma 2 of Bellec

Lemma 48 .

 48 For a 2 U and k 2 [n], there exists ã 2 U k such that Let a = min(a 1 , a n ), ā = max i2[n] a i and i 0 2 argmax i2[n] a i . For j 2 [k 1],

2 log

 2 On the other hand, if k ⇤ > 1, then It is straightforward to generalize the lemma to matrices. For k 2 [n] m , we write k = (k 1 , . . . , k m ) and letU m k = {A 2 U m : Card({A 1,j , . . . , A n,j }) = k j for 1  j  m}. Then K(A) = P m j=1 k j for A 2 U m k . Define k ⇤ by k ⇤ j = l⇣ V (A •,j ) 2 n For A 2 U m , there exists à 2 U m k

  log N (S n \ B n (b, 1), k • k 2 , ")  2 log N (S n 0 \ B n 0 (b I , 1), k • k 2 , "/ p 2 ) .Let k be the smallest integer for which 2 k > n 0 . We partitionI into k blocks A j = I \ [2 j , 2 j+1 ) for j 2 [k] and let m j = |A j |. Since S n 0 ⇢ S m 1 ⇥ • • • ⇥ S m k , Lemma 51 yields that log N S n 0 \ B n 0 (b I , 1), k • k 2 ,Chatterjee [2014, Lemma 4.20] that for any c  d and n 1,log N ⇣ S n \ [c, d] n \ B n (b, 1), k • k 2 ,

j 2 2 )

 22 [k]. Substituting this bound into Eq. (6.21) and noting that k  log 2 n, we reach the conclusionlog N (S n 0 \ B n 0 (b I , 1), k • k 2 , "/ p  C" 1 log(en) .Next, we study the metric entropy of the set of matrices with unimodal columns.Recall thatC l = {a 2 IR n : a 1  • • •  a l } \ {a 2 IR n : a l • • • a n } for l 2 [n]. For l = (l 1 , . . . , l m ) 2 [n] m , define C m l = C l 1 ⇥ • • • ⇥ C lm . Moreover, for A 2 IR n⇥m , t > 0 and C ⇢ IR n⇥m , define ⇥ C (A, t) = [ Note that in particular ⇥ C (A, t) ⇢ B nm (0, t). Lemma 53. Given A 2 IR n⇥m and l = (l 1 , . . . , l m ) 2 [n] m , define K(A) = P m j=1 k(A •,j ) and k(A •,j ) = Card({A 1,j , . . . , A n,j }).Then for any t > 0 and " > 0, log N ⇥ C m l (A, t), k • k F , "  C" 1 t K(A) log enm K(A) .

  S m }. Since each estimator pair ( ⇧, Â) gives an estimator M = ⇧ Â of M = ⇧A, it suffices to prove a lower bound on k M M k 2 F . In fact, we prove a stronger lower bound than the one in Theorem 12.Proposition 21. Suppose that K 0  m( 16n 2 )

2 m

 2 log l for any 1  l  min(K 0 m, m)+1, which is due to the unknown permutation.we have that |P| l 7n/8( 1 8e ) n/8 . Hence for l 2

2 m

 2 even if the matrices are required to have increasing columns. The last term min( , m 2 V 2 0

  product of the two terms is no larger than m. The equality is achieved by u = (1, . . . , 1, 0, . . . , 0) where the first p m entries are equal to one.

  4, the generalization error L is first approximated by the training error L(✓) = 1

		Estimation	Oracle	Running-time
		error	complexity	complexity
	ERM
	Conjugate gradient	O(

2n

P n i=1 (h (x i ), ✓i y i ) 2 . Its minimizer ✓

  Online learning. The Vovk-Azoury-Warmuth predictor presented in Section 1.7 guarantees an O(d log(n)/n) bound on the regret for any sequence (x 1 , y 1 , . . . , x n , y n ). When the data are actually i.i.d., the average predictor hn defined by hn

6.4

. One directly obtains a running time complexity O(

  2.2, as well as the two types of oracles on the gradient in

		Averaged	Averaged
		Algorithm Accelerated
			Algorithm
	Dimension dependent
	rates Additive Noise	Section 3.3 Lemma 12 ⌃	Section 3.4 Theorem 8
	Multiplicative Noise

  taking the trace of the previous inequality we get Ekx n k 4  R 2 Ekx n k 2 and using Cauchy-Schwarz inequality we get Ekx n k 2  p Ekx n k 4  R p Ekx n k 2 . Assumption (A 1 ) is satisfied, for example, for least-square regression with almost surely bounded data, since kx n

  difference in the powers of n and (⌃ + I) 1 ). See the proof in Appendix 3.A.2 for details.-The second term in Eq. (3.8) is the variance term. It depends on the noise in the gradient. When this one is not structured the variance turns to be also bounded by tr ⌃(⌃+ I) 1 E[⇠ n ⌦⇠ n ] (see Appendix 3.A.3) and we recover for = O(1/ p n), the usual rate of 1 p n for SGD in the smooth case [Shalev-Shwartz

  For any regularization parameter 2 R + and for any constant step-size (⌃ + I) 4 I, for any 2 ⇥

	1 1+ eigenvalues or two coalescent eigenvalues. p ⇤ , matrix F will have only two distinct complex p , 1
	Proposition 7. Under (A	1 1+	p p , 1	⇤	, for the recursion in
	Eq. (3.10):

4,5

), for any regularization parameter 2 R + and for any constant step-size (⌃ + I) 4 I we have for any 2 ⇥

  We consider the least-squares regression problem which is the minimization of the quadratic function f

		n	1, are i.i.d. distributed with finite
	(A8)	variances Ekx n k 2 2 < 1 and Ey 2 n < 1.

  2 ⌃ 1 for specific starting points (see proof in Appendix 4.D for details). -It is worth noting that in the constrained case (g = Ek ✓n ✓ ⇤ k 2 ⌃  2 D h (✓⇤,✓ 0 )

	2 (see Appendix 4.D.1 for details). Therefore 2 n ⌃ k Proposition 9 already implies 1 + 8d n ( 2 + r 2

C for a bounded convex set C), the covariance of the noisy oracle is simply bounded by ( tr ⌃r 2 + 2 )⌃ where we denote by r = max ✓2C k✓ ✓

  We first show in Lemma 23 that the distance Ek⌘ n ! n k 2 ⌃ 1 is of order n.-Then in Lemma 24 we show that Ek ✓n ¯ n k 2 ⌃ is of order O(1/n), by: (a) noticing that Ek ✓n ¯ n k 2 ⌃ is of order

	with the result of Lemma 23.	Ek⌘n !nk 2 ⌃ 1 n 2	+ variance n	, (b) combining this

Table 5 .

 5 1 -Experiments on two-class datasets (S.D.C. means sparse discriminative clustering, and Non-S. D.C. non sparse discriminative clustering.)

  1) 1+(d+ 2)m 2 . Moreover we also have with probability at least 1 d exp 2 nµ 2 4R 4 d 2 , for any symmetric matrix 2 ? min

   1 for  4 and is always satisfied otherwise.-If d 3, the first equation implies that 3 for which the second equation is always satisfied. It also implies that m 2 

														1)
			and 1 + (d +	6)m 2		0
			8m 2 and 1 + (d + (1 + (d + , 16m 4 6)m 2		2)m 2 0	)	4m 2	(	1)
			, 2(d +			4)m 2 	3 and 1 + (d +	6)m 2	0.
	-If d = 2,
		-If  3 we have necessary that  2 and the first equation gives m 2 3 2(2 ) and the second m 2  1/(4 ). Thus we should have (4 )(3 )  2(2 ) which is not possible since the polynomial 2 5 + 8 0.
		-If 1/(4		3, the first equation gives m 2  3 )  2( 2) 3 2( 2)  1 and the second m 2  3 2(d+ 4)  1. ✓ ◆
	We denote by	min =	1 0 c	0 min I d 1	the eigenvector for which we have set without
	loss of generality a = 1 and
	c	min =		1	h	p ((d +	2)m 2	1) 2	+ 4(d 1)m 2	(d +	2)m 2	+ 1	i	.
			2(d 1)m
	Consequently c	min  0 and by convexity of the square root we have
	p ((d +	2)m 2	1) 2	+ 4(d 1)m 2  ((d +	2)m 2	1) +	2(d 1)m 2 |(d + 2)m 2	1|

  Intuitively, the quantity R(A) is small if the difference u of any two rows of A is either very sparse (kuk

	2 /kuk 1 1 is small). Indeed, for any nonzero vector u 2 IR m , 1 with equality achieved when kuk is small) or very dense (mkuk 2 /kuk kuk 2 2 /kuk 2 1 0 = 1, and mkuk 2 2 /kuk 2 1

.10) It can be shown (see Appendix 6.D) that 1  R(A)  p m.

  in a private communication. Lemma 45. Let C be a closed subset of IR d . Suppose that y = a ⇤ + z where a ⇤ 2 C and z 2 IR d . Let â 2 argmin a2C ky ak 2 2 be a projection of y onto C. Define the function f a ⇤ Moreover, if there exists t ⇤ > 0 such that f a ⇤ (t) < 0 for all t t ⇤ , then kâ a ⇤ k 2  t ⇤ .

	: IR	+ ! IR by
		f a ⇤	(t) =	sup a2C\B d (a ⇤ ,t)	ha a ⇤ , zi	2 t 2	.
	Then we have
			kâ a ⇤ k	2 2 argmax t 0	f a ⇤	(t).	(6.13)
	Proof. By definition,
		â 2 argmin a2C

  .e.,N µ = {w 2 IR n : w = (w I 1 , • • • , w Im ), w I i 2 N µ i }.We claim thatS µ2N D N µ is an 3"-net of C \ B n (a, 1). Fix v 2 C \ B n(a, 1). Let v I i 2 IR n i be the restriction of v to the component spaceIR n i . Then v I i 2 C i . Let 2 IR m be defined by i = kv I i a I i k 2 , so k k 2 = kv ak 2  1. Hence we can find µ 2 N B such that kµ k 1  " , so µ 2 N D . Moreover, kv I i a I i k 2 = i < µ i + 1 p m and v I i 2 C i , so by definition of N µ i , there exists w I i 2 N µ i such that kw I i v I i k 2  (µ i + 1 p m )". Let w = (w I 1 , . . . , w Im ) 2 N µ . Since It remains to bound the cardinality of this net. By Lemma 50, |N D |  |N B |  (C/") m . Moreover, recall that N µ i is a (µ i + 1 p m )"-net of C i \ B n i (a I i , µ i + 1 p m ). Since a I i 2 C i \ ( C i ), for any t > 0, C i \ B n i (a I i , t) = {x + a I i : x 2 C i \ B n i (0, t)}.Hence we can choose the net so that

	µ i	i	2	" p m	1 p m m 2 X i=1 µ 2 i 			2 i=1	p m . In particular, for all i 2 [m], 2 2 i + " 2 2 2  5 ,
	we conclude that	kw vk 2 2 	m X i=1	⇣	µ i +	1 p m	⌘	2	" 2  7" 2 .
	Therefore	S	µ2N

m X i=1 ( i + | i µ i |) 2  m X D N µ is a 3"-net of C \ B n (a, 1).

  Lemma 56. Consider the model y = ✓ + z where ✓ 2 ⇥ ⇢ IR d and z ⇠ N (0, 2 I d ). Suppose that |⇥| 3 and for distinct ✓, ✓ 0 2 ⇥, 4  k✓ ✓ 0 k 2 2  2 8 log |⇥| where > 0. Then there exists c > 0 such that Proof. Let IP ✓ denote the probability with respect to ✓+z. Then the Kullback-Leibler divergence between IP ✓ and IP ✓ 0 satisfies that KL(IP ✓ , IP ✓ 0 ) = k✓ ✓ 0 k 2

		inf ✓ sup ✓2⇥	IP ✓	⇥	k ✓ ✓k 2 F	⇤	c.
			2 2	F		log |⇥| 16		10 log(|⇥| 1)	,
	since |⇥|	3. Applying

  Proof. Assume without loss of generality that ⇧ ⇤ = I n . Let A 2 S m and define = (n, . . . , n), by Lemma 53,log N ⇥ ⇧S m (A, t), k • k F , "  C" 1 t K(A) log

			f	⇧A (t) =	sup M 2 ⇧S m \B nm	( ⇧A,t)	hM ⇧A, Y	⇧Ai	2 t 2	.
	Since S m	= C m l with l enm K(A)	.
	Using the proof of Lemma 47, we see that
		f	⇧A (t)  C t	r	K(A) log	enm K(A)	+ tk ⇧A A ⇤ k F	2 t 2	+ st
	with probability at least 1 C exp( cs 2 2
									2/3 0	3, in which case the first

  2 . Define the LS estimator ✓ by Moreover let k(✓) = Card({✓ 1 , . . . , ✓ n }) and V (✓) = max i2[n] ✓ i min i2[n] ✓ i .Corollary 10. There exists a constant c > 0 such that with probability at least 1 n ↵ ,

									✓ 2 argmin ✓2U	k✓ yk 2 2 .
	↵ 1,
	1 n	k ✓ ✓ ⇤ k 2 2 . min ✓2U	⇣	1 n	k✓ ✓ ⇤ k 2 2 + 2 k(✓) n	log	en k(✓)	⌘	+ ↵ 2 log n n	(6.30)
	and	1 n	k ✓ ✓ ⇤ k 2 2 . min ✓2U	h	1 n	k✓ ✓ ⇤ k 2 2 +	⇣	2 V (✓) log n n	⌘	2/3 i	+ ↵ 2 log n n	.

The generalization error does not always attain its global minimum, e.g., in logistic regression when the model is not well-specified[START_REF] Bach | Self-concordant analysis for logistic regression[END_REF].

d 2 +dR 2 k✓⇤k 2 ").

n(↵n+ n) + 4↵n (↵n+ n) 2 and consider the sequence of quadratic functions g n (✓) = gn✓2 2 . We will show for the iterate (⌘ n ) defined by our algorithm

2 (R 2 + 2 ) tr ⌃ 2 (⌃ + I) 2 .

Their results remain true in the more general setting of online learning.

We derive, in this section, a lower bound on the performance of SDA when f is the linear form f (✓) = ha, ✓i with a 2 R d and g is a non-strongly convex quadratic function. We assume that the vector a is not available and we only have access to

2(d+ 4) , the problem is particularly simple, and we can also show that V ⇤ is the unique solution of the problem in Eq. (5.12) with high probability if n d 2 R 4 m 2 . Finally, note that for sub-Gaussian distributions (where  3), the extra constraint is vacuous, while for super-Gaussian distributions (where 3), this extra constraint only appears for small m. -This result provides the first guarantee for discriminative clustering. However similar theoretical results have been derived for K-means by Ostrovsky et al.

This assumption is fairly reasonable since the probability of a matrix whose entries are i.i.d. Rademacher random variables to be singular is conjectured to be 1/2 + o(1)[START_REF] Bourgain | On the singularity probability of discrete random matrices[END_REF].

The data sets were obtained from https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/ datasets/

k(✓) n log

m log n in the upper bounds. The next proposition shows that in the case

Remerciements

Assumptions

We consider the Euclidean space R d of dimension d endowed with the natural inner product h•, •i and an arbitrary norm k • k (which may not be the Euclidean norm). We denote by k • k ⇤ its dual norm and for any symmetric positive-definite matrix A, by k • k A = p h•, A•i the Mahalanobis norm. For a vector ✓ 2 R d , we denote by ✓(i) its i-th coordinate and by k✓k p = ( P d i=1 |✓(i)| p ) 1/p its `p-norm. We also denote the convex conjugate of a function f by f ⇤ (⌘) = sup ✓2R d h⌘, ✓i f (✓). We remind that a function f is L-smooth with respect to a norm k

k and is µ-strongly convex if for all (↵, ) 2 R d ⇥ R d and g 2 @f ( ), f (↵) f ( ) + hg, ↵ i + µ 2 k↵ k 2 [see, e.g., [START_REF] Shalev-Shwartz | Online learning meets optimization in the dual[END_REF].

We consider problems of the form:

where X ⇢ R d is a closed convex set with non empty interior. Throughout this chapter, we make the following general assumptions: (A1) f : R d ! R [ {+1} is a proper lower semicontinuous convex function and is differentiable on X (the interior of X ). (A2) g : R d ! R [ {+1} is a proper lower semicontinuous convex function. (A3) h : R d ! R [ {+1} with dom h \ dom g = X , dom h \ dom g 6 = ;. Moreover h is a Legendre function [Rockafellar, 1970, chap. 26]:

-h is a proper lower semicontinuous strictly convex function, differentiable on dom h. -The gradient of h is diverging on the boundary of dom h (i.e., for any sequence (✓ n ) converging to a boundary point of dom h, lim n!+1 krh(✓ n )k = 1). Note that rh is then a bijection from dom h to dom h ⇤ whose inverse is the gradient of the conjugate rh ⇤ . (A4) The function = f +g attains its minimum over X at a certain ✓ ⇤ 2 R d (which may not be unique). Note that we adopt the same framework as [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF] with the difference that a convex constraint C can be handled with more flexibility: either by considering a Legendre function h whose domain is C or by considering the hard constraint g(✓) = C (✓) (equal to 0 if ✓ 2 C and +1 otherwise).

Dual Averaging Algorithm

In this section we present the dual averaging algorithm (referred to from now on as "DA") for solving composite problems of the form of Eq. (4.2). It starts from ✓ 0 2 dom h and ⌘ 0 = rh(✓ 0 ) and iterates for n 1 the recursion

Appendix

4.A Unambiguity of the Primal Iterate

We describe here conditions under which the primal iterate ✓ n in Eq. ( 4.3) is correctly defined. Since h is strictly convex, h ⇤ n is continuously differentiable on dom h ⇤ n [see Hiriart-Urruty and Lemaréchal, 2001, Theorem 4.1.1]. Therefore the primal iterate ✓ n is well defined if the dual iterate ⌘ n 2 dom h ⇤ n . It is, for example, the case under two natural assumptions as shown by the next lemma which is an adaption of Lemma 2 by [START_REF] Bauschke | A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications[END_REF].

Lemma 21. We make the following assumptions: (B1) h or g is supercoercive.

(B2) arg min ✓2X (✓) is compact and h bounded below. Under (B1) or (B2) the primal iterates (✓ n ) defined in Eq. ( 4.3) are well defined.

Proof. Since h is strictly convex, h ⇤ n is continuously differentiable on dom h ⇤ n [see Hiriart-Urruty and Lemaréchal, 2001, Theorem 4.1.1]. Therefore the primal iterate ✓ n is well defined if the dual iterate ⌘ n 2 dom h ⇤ n . -If h or g is supercoercive then h n is supercoercive [see Bauschke and Combettes, 2011, Proposition 11.13] and it follows from Hiriart-Urruty and Lemaréchal [2001, Chapter E, Proposition 1.3.8] that dom

by convexity of f . Therefore ⌘ n 2 dom h ⇤ n since + X is coercive and h bounded below [see Bauschke and Combettes, 2011, Proposition 11.15]. equations (SDE). For stochastic MD d⌘(t) = [rf (✓(t)) + rg(✓(t))]dt + dW (t)dt and ⌘(t) = rh(✓(t)), and for SDA d⌘(t) = rf (✓(t))dt + dW (t)dt and ⌘(t) = r(h + tg)(✓(t)),

where W t is a Wiener process and > 0. We note that the regularization g does not take part in the SDA SDE which explains this dynamic is efficient in presence of noise.

In contrast, the stochastic MD SDE is corrupted by the presence of the gradient rg which may not behaves well for non-smooth g. This continuous-time interpretation of stochastic algorithms could lead to further insights but is outside the scope of this chapter.

4.I Examples of Different Geometries

We describe now different examples of concrete geometries and how SDA is then implemented for well known regularizations g.

Euclidean distance. The simplest geometry is obtained by taking the function

. Therefore (LC) is equivalent to the smoothness of the function f and we return to classic results on proximal gradient descent.

-Projection: Let g = C be the indicator of a convex set C. The SDA method yields to the projected method

.

-`2-regularization:

which is a standard gradient descent on f + g with a structured decreasing step-size n = ( 1 I + nQ) 1 .

-`1-regularization:

. Therefore the SDA method is equivalent, for i = 1, . . . , d, to the iteration:

◆ .

Yet since convergence results hold on the average of the iterates ✓n , SDA provides less sparse solutions than other methods which rather consider final iterates as outputs.

Kullback-Leibler divergence. The negative entropy h(✓) = P n i=1 ✓(i) log(✓(i)) is a Legendre function on dom h = (0, 1) n whose associated Bregman divergence is the Kullback-Leibler divergence

and its conjugate gradient mapping is r i h ⇤ (⌘) = exp(⌘ i ) 1 for i = 1, . . . , d. When constrained on the simplex d , h is 1-strongly convex with respect to the `1-norm [see, e.g., Beck and Teboulle, 2003, Proposition 5.1], and (LC) holds, for example, if f is smooth with regards to the `1-norm. This illustrates one of the non-Euclidean benefit since Lipschitz constants under the `1-norm are smaller than under the `2-norm.

This geometry is particularly appropriated to constrained minimization on the simplex d . With g(✓) = d , SDA update is the dual averaging analogue of the exponentiated gradient algorithm [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF]: , 2] is believed to adapt to the geometry of learning problem and is often used with p = 1 + 1/ log(d) in association with `1-regularization [see, e.g., [START_REF] Duchi | Composite objective mirror descent[END_REF]. Its Fenchel conjugate is the squared conjugate norm h ⇤ = 1 2(q 1) k • k 2 q for 1/p + 1/q = 1 and its conjugate gradient mapping is

(q 1)k⌘k q 2 q [see, e.g., [START_REF] Gentile | The robustness of the p-norm algorithms[END_REF]. For `1-regularization, this yields to:

The function h is 1-strongly convex with respect to the `p-norm [see, e.g., [START_REF] Hanner | On the uniform convexity of L p and l p[END_REF]. Therefore (LC) holds if f is smooth with respect to the the `p-norm. However when the function f considered is quadratic as in Section 4.3, we can directly show that (LC) holds under tighter conditions on the Hessian matrix ⌃ (see proof in Appendix 4.J).

Part II

Applications of the Quadratic Loss in Machine Learning

Theoretical Analysis

In this section, we provide the first theoretical analysis for the discriminative clustering framework with the square loss. We start with the 2-clusters situation: the non-sparse case is considered first and analysis is provided for both balanced and imbalanced clusters. Our study for the sparse case currently only provides results for the simple 1-sparse solution. However, the analysis also yields valuable insights on the scaling between n and d. We then derive results for multi-label situation.

For ease of analysis, we consider the constrained problem in Eq. ( 5.7), the penalized problem in Eq. (5.8) or their equivalent relaxations in Eq. ( 5.12) or Eq. ( 5.18) under various scenarios, for which we use the same proof technique. We first try to characterize the low-rank solutions of these relaxations and then show in certain simple situations the uniqueness of such solutions, which are then non-ambiguously found by convex optimization. Perturbation arguments could extend these results by weakening our assumptions but are not within the scope of this chapter, and hence we do not investigate them further in this section.

Analysis for 2 Clusters: Non-Sparse Problems

In this section, we consider several noise models for the problem, either adding irrelevant dimensions or perturbing the label vector with noise. We consider these separately for simplicity, but they could also be combined (with little extra insight).

Irrelevant Dimensions

We consider an "ideal" design matrix X 2 R n⇥d such that there exists a direction v along which the projection Xv is perfectly clustered into two distinct real values c 1 and c 2 . Since Eq. (5.2) is invariant by affine transformation, we can rotate the design matrix X to have X = [y, Z] with y 2 { 1, 1} n , which is clustered into +1 or 1 along the direction v = 1 0 d 1 . Then after being centered, the design matrix is written as

. The columns of Z represent the noisy irrelevant dimensions added on top of the signal y.

Balanced Problem

When the problem is well balanced (y > 1 n = 0), y is already centered and ⇧ n y = y. Thus the design matrix is represented as X = [y, Z]. We consider here the penalized formulation in Eq. (5.8) with ⌫ = 1 which is the only scenario where we are able to provide a theoretical analysis.

Let us assume that the columns (z i ) i=1,...,d 1 of Z are i.i.d. with symmetric distribution z, with Ez = Ez 3 = 0 and such that kzk 1 is almost surely bounded by R 0. We denote by Ez 2 = m its second moment and by Ez 4 /(Ez 2 ) 2 = its (unnormalized) kurtosis.

Surprisingly the clustered vector y happens to generate a solution yy > of the relaxation Eq. (5.8) for all possible values of Z (see Lemma 35 in Appendix 5.D.2 ).

We consider a slightly different optimization problem:

When c is equal to zero, then Eq. ( 5.24) is exactly equivalent to Eq. (5.23); when c is small (as will typically be the case in our experiments), the solutions are very similar-in fact, one can show by Lagrangian duality that by a sequence of problems in Eq. ( 5.24), one may obtain the solution to Eq. (5.23).

Smoothing

By letting A = X > X n +Diag(a) 2 , we consider a strongly convex approximation of Eq. ( 5.24) as:

) log(A

where tr M log(M ) is a spectral convex function called the von Neumann entropy [START_REF] Neumann | Thermodynamik quantummechanischer Gesamheiten[END_REF]. The difference in the two problems is known to be " log(d) [START_REF] Nesterov | Smoothing technique and its applications in semidefinite optimization[END_REF]. As shown in Appendix 5.G.1, the dual problem is (5.26) where " (M ) is an "-smooth approximation to the maximal eigenvalue of the matrix M .

Optimization Algorithm

In order to solve Eq. ( 5.26), we split the objective function into a smooth part

We may then apply FISTA [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] updates to the smooth function "

(A 1 2

), along with a proximal operator for the non-smooth terms I |C ij |6c i c j and )). Since we stop when the duality gap is " log(d) (as we use smoothing, it is not useful to go lower), the number of iterations is of order 1/(" p log(d)). The proposed algorithm is a clear improvement over the existing approach by [START_REF] Bach | DIFFRAC : a discriminative and flexible framework for clustering[END_REF] which is quadratic in n.

Experiments

We implemented the proposed algorithm in Matlab. The code is available in https://drive.google.com/uc?export=download&id=0B5Bx9jrp7celMk5pOFI4UGt0ZEk.

Two sets of experiments were performed: one on synthetically generated data sets and the other on real-world data sets. The details about experiments follow.

Experiments on Synthetic Data

In this section, we illustrate our theoretical results and algorithms on synthetic examples. The synthetic data were generated by assuming a fixed clustering with ↵ ⇤ 2 [0, 1], along a single direction and the remaining variables were whitened. We consider clustering error defined for a predictor ȳ as 1 (ȳ > y/n) 2 , with values in [0, 1] and equal to zero if and only if y = ȳ.

Phase transition. We first illustrate our theoretical results for the balanced case in Figure 5-1. We solve the relaxation in Eq. ( 5.12) and Eq. ( 5.18) for a large range of d and n using the cvx solver [Grant andBoyd, 2008, 2014]. We show the results averaged over 4 replications and take = 1/ p n for the sparse problems. In Figure 5-1a we investigate whether cvx finds a rank-one solution for a problem of size (n, d) (the value is 1 if the solution is rank-one and 0 otherwise). We compare the performance of the algorithms without `1-regularization in the affine invariant case and with `1regularization in the 1-sparse case. We observe a phase transition with a scaling over the form n = O(d 2) for the affine invariant case and n = O(d) for the 1-sparse case. This is better than what is expected by the theory and corresponds rather to the performance of the principal eigenvector of the solution. It is worth noting that it may be uncertain to really distinguish between a rank-one solution and a spiked solution.

We also solve the relaxation for 4-sparse problems of different sizes d and n and plot the clustering error. We compare, in Figure 5-1b, the performance of the formulation in Eq. ( 5.12) (without `1-regularization) which corresponds to the affine invariant case, against the `1-regularized formulation in Eq. (5.18). We notice a phase transition of the clustering error with a scaling over the form n = O(d 2) for the affine invariant case and n = O(d) for the 4-sparse case. It supports our conjecture on the scaling of order n = O(ld) for l-sparse problems. Comparing left plots of Figure 5-1a and Figure 5-1b, we observe that the two phase-transitions occur at the same scaling between n and d. Thus there are few values of (n, d) for which the cvx solver finds a solution whose rank is stricly larger than one and whose principal eigenvector has a low clustering tion 5.6.3 to solve Eq. ( 5.24) (with c = p 1 d , a = 0 d ) was compared against a benchmark cvx solver (which solves Eq. ( 5.18)). Experiments were performed for = 0 and = 0.001, the coefficient associated with the sparse kV k 1 term. For a fixed d, cvx breaks down for large n values (typically n > 1000). Similarly, the runtime required by cvx is generally high for = 0 and is comparable to our method for = 0.001. This behavior is illustrated in Figure 5-3. When = 0, the problem reduces exactly to the original Diffrac problem [START_REF] Bach | DIFFRAC : a discriminative and flexible framework for clustering[END_REF]. In the plots in Figure 5-3a our implementation using FISTA is compared to the baseline Diffrac which is solved with max-cut SDP [START_REF] Boumal | Manopt, a Matlab toolbox for optimization on manifolds[END_REF]. We observed that our method is comparable in terms of runtime and clustering performance of low-rank methods for max-cut. However, for > 0, the equivalence with max-cut disappears.

The plots in these figures show the behavior of FISTA for two different stopping criteria: " = 10 2 / log(d) and " = 10 3 / log(d). It is observed that the choice 10 3 / log(d) gives a better accurate solution at the cost of more number of iterations (and hence higher runtime). For sparse problems in Figure 5-3b, we see that cvx gets a better clustering performance (while crashing for large n); the difference would be reduced with a smaller duality gap for FISTA.

Clustering performance. Experiments comparing the proposed method (Eq. ( 5.24) with c = p 1 d and a = 0 d solved using FISTA based optimization algorithm, and Eq. ( 5.18) solved using benchmark cvx solver) with K-means and alternating optimization are given in Figure 5-4. K-means is run on the whitened variables in R d . Alternating optimization is another popular method proposed by [START_REF] Ye | Discriminative k-means for clustering[END_REF] for dimensionality reduction with clustering (where alternating optimization of w and y is performed to solve the non-convex formulation (5.2)). The plots show that both Kmeans and alternating optimization fail when only a few dimensions of noise variables are present. The plots also show that with the introduction of a sparse regularizer (corresponding to the non-zero ) the proposed method becomes more robust to noisy dimensions. As observed earlier, the performance of FISTA is also sensitive to the choice of ".

Finally we give a comparison of sparse discriminative clustering (cvx and FISTA) with max-margin clustering [START_REF] Li | Tighter and convex maximum margin clustering[END_REF] in Figure 5-5. While square loss is used Appendix

5.A Joint Clustering and Dimension Reduction

Given y, we need to optimize the Rayleigh quotient

The Lagrangian function of this problem can be written as:

Using L(µ) and the PSD constraint M < 0, the dual problem is given by

Since X(X > X) 1 X > < 0, this implies for the dual variable µ:

However for ⌫ 2 R n , the harmonic mean ⇥

1 is always smaller than the arithmetic mean 1 n P n i=1 ⌫ i with equality if and only if ⌫ = c1 n for c 2 R. Thus the dual variable µ is constant and the diagonal constraint simplifies itself as a trace constraint. Therefore the problem is equivalent to the trivial relaxation whose each eigenvector of X(X

5.C Equivalent Relaxation

5.C.1 First Equivalent Relaxation

We start from the penalized version of Eq. ( 5.5),

which we expand as: (5.30) and relax as, using Y = yy > , P = yv > and V = vv > , min

When optimizing Eq. ( 5.31) with respect to V and P , we get exactly Eq. ( 5.8). Indeed we solve this problem by fixing the matrix Y such that Y = Y 0 and diag(Y 0 ) = 1 n . Then the Lagrangian function of the problem in Eq. ( 5.31) can be written as

Solving for µ i , we get

Substituting µ ⇤ i into the dual obkective function, we get the optimal objective function value

Furthermore the KKT conditions gives

Thus we obtain the following closed form expressions:

The optimal dual objective value D corresponds to a fixed V 0 . Therefore, maximizing with respect to V leads to the problem:

(5.32)

5.D Auxiliary Results for Section 5.5.1

5.D.1 Auxiliary Lemma

The matrix X(X > X) 1 X > has the following properties [see, e.g., [START_REF] Freedman | Statistical Models: Theory and Practice[END_REF].

Lemma 34. The matrix H = X(X > X) 1 X > is the orthogonal projection onto the column space of the design matrix X since:

- Proof. We give an elementary proof of this result without using convex optimization tools. Using lemma 34 we have Hy = y, thus tr HY ⇤ = tr Hyy > = tr yy > = n.

Moreover all M < 0 can always be decomposed as P n i=1 i u i u > i with i 0 and (u i ) i=1,...,n an orthonormal family. Since H is an orthogonal projection

Then for all matrix M feasible we have tr HM  n since diag(M ) = 1 n and tr HY ⇤ = n which conclude the lemma.

5.D.3

Rank-One Solution of the Relaxation Eq. ( 5.12)

Lemma 36. The rank-one solution V ⇤ = vv > is always solution of the relaxation Eq. ( 5.12).

Proof. The Karush-Kuhn-Tucker (KKT) optimality conditions for the problem are for the dual variable A 4 0:

and the dual variable A = 0 satisfy the KKT conditions and then V ⇤ is solution of this problem.

5.D.4 Proof of Proposition 19

In the following lemma, we use a Taylor expansion to lower-bound f around its minimum.

Lemma 37. For d 3 and 2 [0, 1).

If 3 and m 2  3 2(d+ 4) , then with probability at least 1 d exp

2R 4 d 2 , for any symmetric matrix :

Otherwise with probability at least 1 d exp

For m = 1 and d 3 we have µ

5.D.5 Noise Robustness for the 1-Dimensional Balanced Problem

We want a condition on " such that the solution of the relaxation recovers the right y. We recall the dual problem of the relaxation Eq. ( 5.8)

The KKT conditions are:

-Dual feasibility: Diag(µ) < X(X > X) 1 X > .

-Primal feasibility: Diag(Y ) = 1 n and Y < 0.

-Complimentary slackness : Y [Diag(µ) X(X > X) 1 X > ] = 0 For Y = yy > a rank one matrix, the last condition implies Diag(µ)y = Hy and

For X = y+", we denote by ỹ = y+", then X(X > X) 1 X > = ỹ ỹ> kỹk 2 and X(X > X) 1 X > y = ỹ> y kỹk 2 ỹ. Thus

Assume that all ỹi y i have the same sign, without loss of generality we assume ỹi y i > 0.

By definition of µ, µ 0. To show the dual feasibility we have to show that Diag(µ) < H which is equivalent to Diag( ỹi y i ) < ỹ ỹ> ỹ> y , to I n Diag( q y i ỹi ) ỹ ỹ> ỹ> y Diag( q y i ỹi ) < 0 and to P y i ỹi  ỹ> y which is obviously true. Reciprocally if µ is dual feasible then Diag(µ) < 0 and all the ỹi y i have the same sign. Therefore we have shown that y is solution of the relaxation Eq. ( 5.8) if and only if all the ỹi y i have the same sign. If " and y are independent this is equivalent to k"k 1  1 a.s.

5.D.6 The Rank-One Candidates Are Not Solutions of the Relaxation

We assume now that 1 > n y 6 = 0 thus y 6 = ⇧ n y, which means we do not have the same proportion in the two clusters. Let us assume that ⇧ n y takes two values {⇡y , ⇡y + } that is by definition of ⇧ n ⇡y

n and ⇡y = 1 1 > n y n . For V ⇤ defined as

Auxiliary Lemma

Lemma 43. For all matrix C symmetric semi-definite positive we have under assumptions (A1) and (A3):

Proof. We denote by ⌃ n = Z > Z n . We always have kCk

If there is a diagonal element of C which is 0, then all the corresponding line and column in C will also be 0 and we can look at the same problem as before by erasing of C and ⌃ n the corresponding column and line.

Proof of Lemma 42

Proof. We will first show that the noise matrix Z satisfies assumptions (A2,A3). By Hoeffding's inequality we have with probability 1 2 exp( 2 n/(2R 2

))

Then, since the law of z is symmetric y i z i will have the same law as z i and with probability 1 2 exp( 2 n/(2R 2

)), the design matrix Z satisfies assumption (A2):

Likewise we have with probability

)) that for j 6

Thus we also have with probability

)) that Z satisfies assumption (A3):

Thus with probability 1 4d 2 exp( 2 n/(2R 4

)), the noise matrix Z satisfies assumptions (A1, A2, A3). thus

And this system admits as unique solution 0 n if and only if the family {1 n , (y i ) 1ik , (y i y j ) 1i<jk } is linearly independent. Then we prove the lemma 32:

We have already seen that such Y satisfies the constraint. The KKT conditions are:

)

and tr(HY

= B and B > = B, thus B is a symmetric projection and consequently symmetric semi-definit positive.

Hence the primal variable Y and the dual variables µ = 1 n and ⌫ = 1/n satisfy the KKT conditions, thus Y is solution of this problem.

5.G Efficient Optimization Problem

5.G.1 Dual Computation

We consider the following strongly convex approximation of Eq. ( 5.24), augmented with the von-Neumann entropy:

Introducing dual variables, we have

By fixing u and C, and letting

, we can write the max problem as

( 1 2n

This problem is of the form

2 and i (Q) denotes the i-th largest eigen value of the matrix Q. If we consider the matrix D to be of the form D = U Diag(✓)U > with ✓ denoting the vector of ordered eigen values of D, then it turns out that at optimality Q has the form Q = U Diag( )U > , with denoting the ordered vector of eigen values of Q.

Therefore the above optimization problem can be cast in terms of as:

The solution of this problem is i = e ✓ i /"

P n j=1 e ✓ j /" , which leads to

⌘ .

In terms of the original matrix variables, we have min " (D) = " log tr e D " . Using the appropriate expansion of D, we have the overall optimization problem as

( 1 2n

).

(5.38)

At optimality, we have

The error of approximation is at most " log d and the Lipschitz constant associated with the function " ( ) is 1 " .

5.G.2 Algorithm Details

We write the optimization problem Eq. ( 5.38) as:

where

( 1 2n

)

is the smooth part and

is the non-smooth part.

The gradient r u of H(u, C) with respect to u is

where

).

The Lipschitz constant L associated with the gradient rH(u, C) is

where max (M ) denotes the maximum eigen value of matrix M . Computing L takes O(max(n, d) 3

) time and L needs to be computed once at the beginning of the algorithm.

The resultant FISTA procedure is described in Algorithm 1. Note that the FISTA procedure first computes intermediate iterates

Step 7, Algorithm 1) by taking descent steps along the respective gradient directions. Then two distinct problems in u and C (respectively Steps 8 and 9 in Algorithm 1) are solved. The sub-problem in u (Step 8) can be efficiently solved using a Newton procedure followed by a thresholding step, as illustrated in Algorithm 2. The sub-problem in C (Step 9) can also be solved using a simple thresholding step. ually each column A ⇤

•,j by an estimator •,j and then get an estimator  for the whole matrix by concatenating the columns •,j . Thus the task is reduced to estimation of a vector ✓ ⇤ which satisfies a certain shape constraint from an observation y = ✓ ⇤ + z where z ⇠ subG n,1 ( 2). When ✓ ⇤ is assumed to be increasing we speak of isotonic regression [START_REF] Barlow | Statistical Inference under Order Restrictions. The Theory and Application of Isotonic Regression[END_REF]. The LS estimator defined by ✓ = argmin ✓2Sn k✓ yk 2 2 can be computed in closed form in O(n) using the Pool-Adjacent-Violators algorithm (PAVA) [START_REF] Ayer | An empirical distribution function for sampling with incomplete information[END_REF][START_REF] Barlow | Statistical Inference under Order Restrictions. The Theory and Application of Isotonic Regression[END_REF], Robertson et al., 1988] and its statistical performance has been studied by [START_REF] Zhang | Risk bounds in isotonic regression[END_REF] (see also works by [START_REF] Nemirovski | The rate of convergence of nonparametric estimates of maximum likelihood type[END_REF], [START_REF] Donoho | Gelfand n-widths and the method of least squares[END_REF], van de [START_REF] Van De Geer | Estimating a regression function[END_REF], [START_REF] Mammen | Estimating a smooth monotone regression function[END_REF], van de Geer [1993] for similar bounds using empirical process theory) who showed in the Gaussian case z ⇠ N (0, 2 I n ) that the mean squared error behaves like .4) where

Note that 2/3 = 2 /(2 + 1) for = 1 so that this is the minimax rate of estimation of Lipschitz functions [see, e.g., [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF].

The rate in Eq. ( 6.4) is said to be global since it holds uniformly over the set of monotone vectors with variation V (✓ ⇤

). Recently, Chatterjee et al. [2015] have initiated the study of adaptive bounds that may be better if ✓ ⇤ has a simpler structure in some sense. To define this structure, let k Chatterjee et al. [2015] showed that the LS estimator satisfies the adaptive bound

We extend this notion to a matrix A 2 IR n⇥m by defining

.

While this 2/3-norm may seem odd at first sight, it turns out to be the correct extrapolation from vectors to matrices, at least in the context under consideration here. Indeed, the following upper bound, in which this quantity naturally appears, is matched by the lower bound of Theorem 13 up to logarithmic terms.

Theorem 11. For A ⇤ 2 IR n⇥m and Y = ⇧ ⇤ A ⇤ + Z, let ( ⇧, Â) be the LS estimator defined in Eq. ( 6.2). Then it holds that

with probability at least 1 e c(n+m) , c > 0. Moreover, the corresponding bound with the same rate holds in expectation.

If A ⇤ 2 U m , then taking A = A ⇤ in Theorem 11 leads to the following corollary that indicates that the LS estimator is adaptive to the quantity V (A ⇤

). Corollary 9. For A ⇤ 2 U m and Y = ⇧ ⇤ A ⇤ + Z, the LS estimator ( ⇧, Â) satisfies

with probability at least 1 e c(n+m) , c > 0. Moreover, the corresponding bound with the same rate holds in expectation.

Akin to the adaptive bound, the above inequality can be viewed as a sum of a matrix version of Eq. ( 6.4) and an error due to estimation of the unknown permutation.

Having stated the main upper bounds, we digress a little to remark that the proofs of Theorem 10 and Theorem 11 also yield a minimax optimal rate of estimation (up to logarithmic factors) for unimodal regression, which improves the bound in Eq. (6.6). We discuss the details in Appendix 6.F.

Minimax Lower Bounds

Given the model Y = ⇧ ⇤ A ⇤ + Z where entries of Z are i.i.d. N (0, 2 ) random variables, let ( ⇧, Â) denote any estimator of (⇧ ⇤ , A ⇤ ), i.e., any pair in S n ⇥IR n⇥m that is measurable with respect to the observation Y . We will prove lower bounds that match the rates of estimation in Corollary 8 and Corollary 9 up to logarithmic factors. The combination of upper and lower bounds, implies simultaneous near optimality of the least-squares estimator over a large scale of matrix classes. (6.11) where A ⇤ i,1 = i p m and entries of Z are i.i.d. standard Gaussian variables, because the sum of noise in a row has order p m which is no less than the gaps between row sums of A ⇤ . In fact, R(A ⇤ ) = 1 and it should be easy to distinguish the two types of rows of A ⇤ , for example, by looking at the first entry of a row. This motivates us to consider the following method called RankScore.

For i, i 0 2 [n], define

and define Y (i, i 0 ) analogously. The RankScore procedure is defined as follows: 1. For each i 2 [n], define the score s i of the i-th row of Y by

where ⌧ := C p log(nm) for some tuning constant C (see Appendix 6.D for more details). 2. Then order the rows of Y so that their scores are increasing, with ties broken arbitrarily. The RankScore procedure recovers an order of the rows of Y , which leads to an estimator ⇧ of the permutation. Then we define à 2 S m so that ⇧ à is the projection of Y onto the convex cone ⇧S m . The estimator ( ⇧, Ã) enjoys the following rate of estimation. Theorem 14. For A ⇤ 2 S m and Y = ⇧ ⇤ A ⇤ + Z, let ( ⇧, Ã) be the estimator defined above using the RankScore procedure with threshold ⌧ = 3 p (C + 1) log(nm), C > 0. Then it holds that

with probability at least 1 e c(n+m) (nm) C for some constant c > 0.

The quantity R(A ⇤

) only depends on the matrix

) is bounded logarithmically, the estimator ( ⇧, Ã) achieves the minimax rate up to logarithmic factors. In any case, R(A ⇤ )  p m, so the estimator is still consistent with the permutation error (the last term) decaying at a rate no slower than Õ( 1 p m ). Furthermore, it is

Figure 6-2 -Estimation errors of the oracle (dashed lines) and RankScore (solid lines) for different regimes of (n, m) and randomly generated A ⇤ of size n⇥m. Left:

In Figure 6-2, we compare the performance of RankScore to that of the oracle in three regimes of (n, m). The matrices A ⇤ are randomly generated for different values of n and m as follows. For the right plot, A ⇤ is generated so that V (A ⇤

)  1, by sorting the columns of a matrix with i.i.d. U (0, 1) entries. For the left plot, we further require that K(A ⇤ ) = 5m by uniformly partitioning each column of A ⇤ into five blocks and assigning each block the corresponding value from a sorted sample of five i.i.d. U (0, 1) variables.

Since the oracle knows the true permutation, its behavior is independent of m, and its rates of estimation are bounded by log n n for K(A ⇤ ) = 5m and ( log n n )

) = 1 respectively by Theorem 10 and 11. (The difference is minor in the plots as n is not sufficiently large). For RankScore, the permutation term dominates the estimation term when m = n 1/2 by Theorem 14. From the plots, the rates of estimation are better than Õ(n 1/4

) predicted by the worst-case analysis in both examples. For m = n, we also observe rates of estimation faster than the worst-case rate Õ(n 1/2

) and close to the oracle rates. We could explain this phenomenon by R(A ⇤ ) < p m, but such an interpretation may not be optimal since our analysis is based on worst-case deterministic A ⇤ . Potential study of random designs of A ⇤ is left open. Finally, for m = n 3/2 , the permutation term is of order Õ(n 3/4

) theoretically, in between of the oracle rates for the two cases. Indeed RankScore has almost the same performance as the oracle experimentally. Overall Figure 6-2 illustrates the good behavior of RankScore in this random scenario.

To conclude our numerical experiments, we consider the n ⇥ n lower triangular matrix A ⇤ defined by A ⇤ i,j = 1I(i j). For this matrix, it is easy to check that K(A ⇤ ) = 2n 1 and R(A ⇤ ) ⇡ p n. We plot in Figure 6-3 the estimation errors of ⇧ Ã, ⇧A ⇤ and à given by RankScore, in addition to the oracle. By Theorem 14, the rate of estimation achieved by ⇧ à is of order Õ(n 1/2 ), while that achieved by the oracle is of order Õ(n 1

) since there is no permutation term. The plot confirms this discrepancy. Moreover,

F is an appropriate measure of the performance Appendix

6.A Proof of the Upper Bounds

Before proving the main theorems, we discuss two methods adopted in recent works to bound the error of the LS estimator in shape constrained regression, in a general setting. Consider the least-squares estimator ✓ of the model y = ✓ ⇤ + z, where ✓ ⇤ lies in a parameter space ⇥ and z is Gaussian noise. One way to study Ek ✓ ✓ ⇤ k 2 2 is to use the statistical dimension [START_REF] Amelunxen | Living on the edge: phase transitions in convex programs with random data[END_REF]

This has been successfully applied to isotonic and more general shape constrained regression by Chatterjee et al. [2015], [START_REF] Bellec | Sharp oracle inequalities for least squares estimators in shape restricted regression[END_REF].

Another prominent approach is to express the error of the LS estimator via what is known as Chatterjee's variational formula, proved by [START_REF] Chatterjee | A new perspective on least squares under convex constraint[END_REF] and given by .12) Note that the first term is related to the Gaussian width [see, e.g., [START_REF] Chandrasekaran | The convex geometry of linear inverse problems[END_REF] of ⇥ defined by E[sup ✓2⇥ h✓, zi], whose connection to the statistical dimension was studied by [START_REF] Amelunxen | Living on the edge: phase transitions in convex programs with random data[END_REF]. The variational formula was first proposed for convex regression by [START_REF] Chatterjee | A new perspective on least squares under convex constraint[END_REF], and later exploited in several different settings, including matrix estimation with shape constraints by [START_REF] Chatterjee | On matrix estimation under monotonicity constraints[END_REF] and unimodal regression by [START_REF] Chatterjee | Adaptive risk bounds in unimodal regression[END_REF]. Similar ideas have appeared in other works, for example, analysis of empirical risk minimization [START_REF] Mendelson | Learning without concentration[END_REF], ranking from pairwise comparison [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF] and isotonic regression [START_REF] Bellec | Sharp oracle inequalities for least squares estimators in shape restricted regression[END_REF]. [START_REF] Bellec | Sharp oracle inequalities for least squares estimators in shape restricted regression[END_REF] has used the statistical dimension approach to prove spectacularly sharp oracle inequalities that seem to be currently out of reach for methods based on Chatterjee's variational formula in Eq. ( 6.12). On the other hand, Chatterjee's variational formula seems more flexible as computations of the statistical dimension based on tools developed by [START_REF] Amelunxen | Living on the edge: phase transitions in convex programs with random data[END_REF] are currently limited to convex sets ⇥ with a polyhedral structure. In this chapter, we use exclusively Chatterjee's variational formula.

Combining the previous three displays, we see that

2 with probability at least 1 C exp( cs 2

2 ) simultaneously for all t > 0.

We are now in a position to prove the adaptive oracle inequalities in Theorem 10. Recall that ( ⇧, Â) denotes the LS estimator defined in Eq. (6.2). Without loss of generality, assume that ⇧ ⇤ = I n and Y = A ⇤ + Z. Fix à 2 U m and define f à as in Lemma 47. We can apply Lemma 45 with a ⇤ = Ã, z = Y Ã, y = Y and â = ⇧  to achieve an error bound on k ⇧  Ãk F , since ⇧  2 argmin M 2M kY M k 2 F . To be more precise, for any s > 0 we define

where C 1 is the constant in Eq. (6.16). Then it follows from Lemma 47 that with probability at least 1 C exp( cs 2 2 ), it holds for all t t ⇤ that 6.17) with probability at least 1 C exp( cs 2 2 ). In particular, if s = R 1 , then s p n + m as K( Ã) m. We see that with probability at least 1 C exp( cs 2

2 ) 1 e c(n+m) ,

Summing over 1  j  m, we get that

and similarly

For A 2 U m , choose à 2 U m k ⇤ according to Lemma 49. Then

log n (6.19) by noting that log(en)  2.5 log n for n 2, and similarly (6.20) Plugging Eq. ( 6.19) and Eq. ( 6.20) into the right-hand side of Eq. ( 6.7) and Eq. ( 6.8), and then minimizing over A 2 U m , we complete the proof.

6.B Metric Entropy

In this section, we study various covering numbers or metric entropy related to the parameter space of the model in Eq. ( 6.1). First recall some standard definitions that date back at least to [START_REF] Kolmogorov | -entropy and "-capacity of sets in functional space[END_REF]. An "-net of a subset G ⇢ IR n with respect to a norm k • k is a set {w 1 , • • • , w N } ⇢ G such that for any w 2 G, there exists i 2 [N ] for which kw w i k  ". The covering number N (G, k•k, ") is the cardinality of the smallest "-net with respect to the norm k • k. Metric entropy is defined as the logarithm of a covering number. In the following, we will consider the Euclidean norm unless otherwise specified.

A = S ⇤ \ B nm (0, t) for any 0. In view of Definition in Eq. ( 6.22), it holds

In particular, taking = 1, we get

Thus the metric entropy of ⇥ C m l (A, t) is subject to the above bound as well.

Finally, we consider the metric entropy of ⇥ M (A, t) for A 2 IR n⇥m , t > 0 and M = S ⇧2Sn ⇧U m . The above analysis culminates in the following lemma which we use to prove the main upper bounds.

Lemma 54. Let A 2 IR n⇥m and K(A) be defined as in the previous lemma. Then for any " > 0 and t > 0,

+ n log n.

Proof. Assume that "  t since otherwise the left-hand side is zero and the bound holds trivially. Note that U m = S l2[n] m C m l , and that M = S ⇧2Sn ⇧U m . Thus M is the union of n m n! cones of the form ⇧C m l . By Definition in Eq. ( 6.22), ⇥ M (A, t) is also the union of n m n! sets ⇥ ⇧C m l (A, t), each having metric entropy subject to the bound in Lemma 53. Therefore, a union bound implies that

where the last step follows from that K log(enm/K) m log n for m  K  nm and that "  t.

6.C Proof of the Lower Bounds

For minimax lower bounds, we consider the model Y = ⇧ ⇤ A ⇤ + Z where entries of Z are i.i.d. N (0, 2

). The Varshamov-Gilbert lemma [Massart, 2007, Lemma 4.7] is a standard tool for proving lower bounds. We also need the following useful lemma.

Lemma 57. Suppose that K

where IP M is the probability with respect to Y = M + Z.

Proof. We adapt the proof of Bellec and Tsybakov [2015, Theorem 4] to the case of matrices. Let

On the other hand,

Applying Lemma 56 completes the proof.

For the second term in Eq. ( 6.23), we first note that the bound is trivial for l = 1 since log l = 0. The next lemma deals with the case l = 2.

Lemma 58. There exist constants c, c 0 > 0 such that for any K

where IP M is the probability with respect to Y = M + Z.

Proof. By Lemma 55, there exists ⌦ ⇢ {0, 1} n such that log |⌦| n/8 and (!, ! 0 ) n/4 for distinct !, ! 0 2 ⌦. For each ! 2 ⌦, define M ! 2 IR n⇥m by setting the first column of M ! to be ↵! and all other entries to be zero, where

M )  V 0 and we can permutate the rows of M ! so that its first column is increasing;

Applying Lemma 56 completes the proof.

For the previous two lemmas, we have only used matrices with increasing columns. However, to achieve the second term in Eq. ( 6.23) for l 3, we need matrices with unimodal columns. The following packing lemma is the key. For " > 0, define k = b " 2 n 2 c. Then there exists an "

Proof. There are l choices of entries to put the one in each row of M , so

Moreover, let P be a maximal " p n-packing of M. Then P is also an "

We conclude that |P| l n k ( k en ) k . For notational simplicity, we now consider 2  l  min(K

6.D Matrices with Increasing Columns

For the model Y = ⇧ ⇤ A ⇤ + Z where A ⇤ 2 S m and Z ⇠ subG( 2), a computationally efficient estimator ( ⇧, Ã) has been constructed in Section 6.4 using the RankScore procedure. We will bound its rate of estimation in this section. Recall that the definition of ( ⇧, Ã) consists of two steps. First, we recover an order (or a ranking) of the rows of Y , which leads to an estimator ⇧ of the permutation. Then define à 2 S m so that ⇧ à is the projection of Y onto the convex cone ⇧S m . For the analysis of the algorithm, we deal with the projection step first, and then turn to learning the permutation.

6.D.1 Projection

In fact, for any estimator ⇧, if à is defined as above by the projection corresponding to ⇧, then the error k ⇧ à ⇧ ⇤ A ⇤ k 2 F can be split into two parts: the permutation error k( ⇧ ⇧ ⇤ )A ⇤ k 2 F and the estimation error of order Õ( 2 K(A ⇤ )). The proof of the following oracle inequality is very similar to that of Theorem 10, so we will sketch the proof without providing all the details. Lemma 61. Consider the model Y = ⇧ ⇤ A ⇤ + Z where A ⇤ 2 S m and Z ⇠ subG( 2). For any ⇧ 2 S n , define à 2 S m so that ⇧ à is the projection of Y onto ⇧S m . Then with probability at least 1 e c(n+m) ,

, we can simply use an averaging estimator that achieves the term m 2 V (A) 2 .

Proposition 22. For Y = ⇧ ⇤ A ⇤ + Z where Z ⇠ subG( 2), let ⇧ = I n and  be defined by Âi,j = 1 n P n k=1 Y k,j for all (i, j) 2 [n] ⇥ [m]. Then,

with probability at least 1 exp( m) and

Proof. Recall that V (A) = ( 1 m P m j=1 V j (A) 2/3 ) 3/2 . Since the `2-norm of a vector is no larger than the `2 3 -norm,

On the other hand,

Z k,j , so we have that

where g j = 1

p n P n k=1 Z k,j for j 2 [m] so that g 1 , . . . , g m are centered sub-Gaussian variables with variance proxy 2 . It is well-known that Eg 2 j . 2 , so

Moreover, since (g 1 , . . . , g m ) is a sub-Gaussian vector with variance proxy 2 , it follows from Hsu et al. [2012, Theorem 2.1] that P m j=1 g 2 j . 2 m with probability at least

List of Figures

List of Tables