Yang Hua

Erwan Le Roux

Henrique Morimitsu

Jérôme Revaud

Pavel Tokmakov

Alberto Bietti

Nicolas Chesneau

Guosheng Hu

Dan Oneata

Mattis Paulin

Federico Pierucci

Philippe Weinzaepfel

Karteek Alahari

Dexiong Chen

Matthijs Douze

Ghislain Durif

Nikita Dvornik

Thomas Lucas

Vicky Kalogeiton

Xavier Martin

Xiaojiang Peng

Danila Potapov

Gregory Rogez

Shreyas Saxena

Konstantin Shmelkov

Vladyslav Sydorov

Jakob Verbeek

Daan Wynen

First of all, I would like to thank my two supervisors Zaid Harchaoui and Julien Mairal for their continuous support and patience during all these years, for offering me a very fascinating research topic, for their enlightening remarks and fruitful advices.

Résumé

Les problèmes d'optimisation apparaissent naturellement pendant l'entrainement de modèles d'apprentissage supervisés. Un exemple typique est le problème de minimisation du risque empirique (ERM), qui vise à trouver un estimateur en minimisant le risque sur un ensemble de données. Le principal défi consiste à concevoir des algorithmes d'optimisation efficaces permettant de traiter un grand nombre de données dans des espaces de grande dimension. Dans ce cadre, les méthodes classiques d'optimisation, telles que l'algorithme de descente de gradient et sa variante accélérée, sont coûteux en termes de calcul car elles nécessitent de passer à travers toutes les données à chaque évaluation du gradient. Ce défaut motive le développement de la classe des algorithmes incrémentaux qui effectuent des mises à jour avec des gradients incrémentaux. Ces algorithmes réduisent le coût de calcul par itération, entraînant une amélioration significative du temps de calcul par rapport aux méthodes classiques. Une question naturelle se pose : serait-il possible d'accélérer davantage ces méthodes incrémentales ? Nous donnons une réponse positive en introduisant plusieurs schémas d'accélération génériques, qui s'appliquent aux méthodes d'optimisation de premier ordre.

Dans le chapitre 2, nous développons une variante proximale de l'algorithme Finito/MISO, qui est une méthode incrémentale initialement conçue pour des problèmes lisses et fortement convexes. Nous introduisons une étape proximale dans la mise à jour de l'algorithme pour prendre en compte la pénalité de régularisation qui est potentiellement non lisse. Lorsque le problème est fortement convexe, l'algorithme admet un taux de convergence linéaire similaire à l'algorithme Finito/MISO original. L'avantage de cette méthode est qu'elle donne systématiquement un certificat d'optimalité à chaque itération, basé sur des bornes inférieures quadratiques. Contrairement aux algorithmes duals comme SDCA, la construction de notre algorithme est purement primale. Ni la preuve de convergence ni l'algorithme ne nécessitent la dualité de Fenchel, ce qui le rend facile à évaluer en pratique.

Dans le chapitre 3, nous introduisons un schéma d'accélération générique pour les problèmes convexes, appelé Catalyst, lequel accélère les méthodes d'optimisation de premier ordre dans le sens de Nesterov. Il s'applique à une grande classe d'algorithmes, y compris la descente de gradient, la descente de gradient par coordonnées, des algorithmes incrémentaux tels que SAG, SAGA, SDCA, SVRG, Finito/MISO et leurs variantes proximales. Pour toutes ces méthodes, nous montrons qu'une accélération est obtenue en appliquant Catalyst sur les problèmes convexes. Il est important de remarquer que notre approche est capable de traiter des objectifs non fortement convexes même si certaines méthodes incrémentales ne sont définies que sur des problèmes fortement convexes. En outre, Catalyst peut être vu comme une variante approchée de l'algorithme de point proximal accéléré. À chaque itération, il utilise la méthode que l'on souhaite accélérer pour résoudre un sous problème, ii qui revient à approcher un opérateur proximal. La clé de l'accélération est de choisir attentivement un critère d'arrêt et de contrôler l'effort de calcul requis. Nous montrons ensuite que l'accélération est bien effectuée en pratique.

Dans le chapitre 4, nous présentons une seconde approche générique qui applique les principes Quasi-Newton pour accélérer les méthodes de premier ordre, appelée QuickeNing. Le schéma s'applique à la même classe de méthodes que Catalyst. À notre connaissance, QuickeNing est le premier algorithme de type Quasi-Newton compatible avec les objectifs composites et la structure de somme finie. L'algorithme admet une simple interprétation comme une combinaison de l'algorithme L-BFGS et de la régularisation Moreau-Yosida. Nous proposons une nouvelle stratégie pour déterminer le pas à chaque itération afin d'éviter des recherches linéaires coûteuses, ce qui rend l'algorithme pratique à l'utilisation. Nous montrons expérimentalement que QuickeNing apporte une amélioration significative par rapport aux méthodes concurrentes dans les problèmes d'apprentissage à grande échelle.

Nous concluons cette thèse en proposant une extension de l'algorithme Catalyst au cas non convexe. Il s'agit d'un travail en collaboration avec Dr. Courtney Paquette et Pr. Dmitriy Drusvyatskiy, de l'Université de Washington, et mes encadrants de thèse. Le point fort de cette approche réside dans sa capacité à s'adapter automatiquement à la convexité. En effet, aucune information sur la convexité de la fonction n'est nécessaire avant de lancer l'algorithme. Ceci est réalisé par une étape spécifique, appelée auto-adaptation, qui exploite l'information de convexité locale. Lorsque l'objectif est convexe, l'approche proposée présente les mêmes taux de convergence que l'algorithme Catalyst convexe, entraînant une accélération. Lorsque l'objectif est non-convexe, l'algorithme converge vers les points stationnaires avec le meilleur taux de convergence pour les méthodes de premier ordre. Des résultats expérimentaux prometteurs sont observés en appliquant notre méthode à des problèmes de factorisation de matrice parcimonieuse et à l'entrainement de models de réseaux de neurones.

Abstract

Optimization problems arise naturally in machine learning for supervised problems. A typical example is the empirical risk minimization (ERM) formulation, which aims to find the best a posteriori estimator minimizing the regularized risk on a given dataset. The current challenge is to design efficient optimization algorithms that are able to handle large amounts of data in high-dimensional feature spaces. Classical optimization methods such as the gradient descent algorithm and its accelerated variants are computationally expensive under this setting, because they require to pass through the entire dataset at each evaluation of the gradient. This was the motivation for the recent development of incremental algorithms. By loading a single data point (or a minibatch) for each update, incremental algorithms reduce the computational cost per-iteration, yielding a significant improvement compared to classical methods, both in theory and in practice. A natural question arises: is it possible to further accelerate these incremental methods? We provide a positive answer by introducing several generic acceleration schemes for first-order optimization methods, which is the main contribution of this manuscript.

In chapter 2, we develop a proximal variant of the Finito/MISO algorithm, which is an incremental method originally designed for smooth strongly convex problems. In order to deal with the non-smooth regularization penalty, we modify the update by introducing an additional proximal step. The resulting algorithm enjoys a similar linear convergence rate as the original algorithm, when the problem is strongly convex. More interestingly, a practical optimality certificate based on quadratic lower bounds is systematically given. In contrast to other dual-type algorithms like proximal SDCA, the construction of our algorithm is purely primal. Neither the proof of convergence nor the algorithm requires the Fenchel conjugate, which makes it more practical.

In chapter 3, we introduce a generic acceleration scheme, called Catalyst, for accelerating gradient-based optimization methods in the sense of Nesterov. Our approach applies to a large class of algorithms, including gradient descent, block coordinate descent, incremental algorithms such as SAG, SAGA, SDCA, SVRG, Finito/MISO, and their proximal variants. For all of these methods, we provide acceleration and explicit support for non-strongly convex objectives. The Catalyst algorithm can be viewed as an inexact accelerated proximal point algorithm, applying a given optimization method to approximately compute the proximal operator at each iteration. The key for achieving acceleration is to appropriately choose an inexactness criteria and control the required computational effort. We provide a global complexity analysis and show that acceleration is useful in practice.

In chapter 4, we present another generic approach called QuickeNing, which applies Quasi-Newton principles to accelerate gradient-based optimization methods. The scheme is applicable to the same class of functions as Catalyst, including most iv of the well-known first-order algorithms. To the best of our knowledge, QuickeNing is the first Quasi-Newton type algorithm compatible with both composite objectives and the finite sum setting. The algorithm can be interpreted as a combination of inexact L-BFGS algorithm and the Moreau-Yosida regularization. A particular warm start strategy is proposed in order to avoid any line search in the determination of the stepsize. This makes the algorithm practical and we provide extensive experiments showing that QuickeNing gives significant improvement over competing methods in large-scale machine learning problems.

We conclude the thesis by extending the Catalyst algorithm into the nonconvex setting. This is a joint work with Courtney Paquette and Dmitriy Drusvyatskiy, from University of Washington, and my PhD advisors. The strength of the approach lies in the ability of the automatic adaptation to convexity, meaning that no information about the convexity of the objective function is required before running the algorithm. A particular step called Auto-adapt is designed in the algorithm to exploit the local convexity information. When the objective is convex, the proposed approach enjoys the same convergence result as the convex Catalyst algorithm, leading to acceleration. When the objective is nonconvex, it achieves the best known convergence rate to stationary points for first-order methods. Promising experimental results have been observed when applying to sparse matrix factorization problems and neural network models. v Chapter 1

Introduction

Mathematical optimization lies at the heart of machine learning. The first learning algorithm, Rosenblatt's Perceptron [START_REF] Duda | Pattern Classification (2Nd Edition)[END_REF], was performing incremental optimization on the empirical risk objective with a linear hinge loss for supervised binary classification [START_REF] Bottou | Stochastic gradient descent tricks[END_REF]. The Adaline algorithm similarly performed incremental minimization on the empirical risk with a least-squares loss function. Incremental and stochastic optimization algorithms were the prevailing approaches to train learning methods in the early days of the computing era [START_REF] Tsypkin | Foundations of the theory of learning systems[END_REF]Nikolic, 1971, Tsypkin, 1973] where datasets were small by today's standards but the computational resources were very limited.

The need to train machine learning models on huge datasets commonly arises in modern applications of machine learning such as visual object recognition and natural language processing, and has recently led to revisit these incremental optimization algorithms with a new point of view. Indeed recent fundamental advances in mathematical optimization allow to provably automatically accelerate first-order or gradient-based optimization methods under broad smoothness assumptions of the objective. The fast gradient method [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF] and its popular variant FISTA [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] are prominent examples of accelerated firstorder methods. These methods were successfully applied to an array of machine learning, signal and image processing, statistics problems; see e.g. [START_REF] Mallat | A wavelet tour of signal processing: the sparse way[END_REF][START_REF] Bach | Optimization with sparsity-inducing penalties[END_REF][START_REF] Mairal | Sparse modeling for image and vision processing[END_REF][START_REF] Hastie | Statistical Learning With Sparsity: The Lasso And Generalizations[END_REF].

We present in this thesis several acceleration schemes allowing to improve the convergence of first-order optimization algorithms applied to large-scale machine learning problems. We first consider the minimization of convex composite objectives, decomposable into a smooth part and a non-smooth part, using first-order methods. In Chapter 2, we present an incremental optimization algorithm, called Prox-MISO, based on the principle of surrogate minimization and establish its convergence guarantee. In Chapter 3, we describe an acceleration scheme called Catalyst consisting of an outer-loop performing an extrapolation step on iterates produced by first-order methods applied to auxiliary objectives. We quantify the acceleration in terms of worst-case non-asymptotic convergence guarantees when applying Catalyst to various firstorder methods. In Chapter 4, we describe a different acceleration scheme called QuickeNing also 1 CHAPTER 1. INTRODUCTION based on an inner-outer loop construction but performing quasi-Newton-type updates on the iterates produced by first-order methods applied to auxiliary objectives. We demonstrate the effectiveness of the acceleration through extensive numerical benchmarks on several machine learning datasets. Finally, in Chapter 5, we consider the minimization of possibly non-convex composite objectives. We present an extension of the Catalyst scheme that is automatically adaptive to the hidden convexity of the problem.

In this introductory chapter, we shall review the main concepts at work in the acceleration schemes we propose. We first formulate the optimization problem and review two important classes of smooth convex optimization methods: first-order methods and Quasi-Newton methods. Then we review the notion of proximal operator in Section 1.4, which is an important tool for extending smooth optimization methods to non-smooth settings. In Section 1.5, we present a survey of incremental optimization algorithms, which are particularly attractive when dealing with datasets with large number of data-points. A natural question to ask is can we do better? To address this question, we introduce the Moreau-Yosida regularization in Section 1.6, which gives a natural interpretation of the generic acceleration scheme that we develop in this thesis. Finally, we outline the technical contributions in Section 1.7.

Formulation of the optimization problem

The goal of machine learning is to have a machine learn from data by itself without human interventions. Often a specific task is given, such as visual object recognition or speech recognition, and we would like to ask machines to solve the corresponding task based on the data we provided. For example, in the context of visual object recognition, we would like to learn a function that maps images to an object label, indicating what object category appears in the image, such as a cat, a dog; or a plane, etc. However, the set of functions from images to labels are so rich that given any finite many images, there will always be a perfect match by simply memorizing the correct label of each image. This perfect matching is contrary to our intention of learning high-level representations of the image, because it may behaves arbitrarily bad for any unseen image. Thus it is necessary and important to impose some hypothesis on the nature of the function that we want to learn in order to get a good generalization ability [START_REF] Vapnik | The nature of statistical learning theory[END_REF]. Different models encode different assumption about the function space: in linear model, the function class is simply all linear functions; in more complex model like neural networks, the function space is a collection of connected artificial neurons mimicking human brain's structure [START_REF] Friedman | The elements of statistical learning[END_REF]. Thus the learning model presumes the underlying structure of the problem and provides a much smaller searching space of functions. Once we determine which model to be applied, we can formulate the problem as an optimization problem among the providing searching space.

More formally speaking, we are given two spaces of objects A and B and we assume that there is a joint probability distribution P (a, b) over A × B. The goal is to learn a function h : A → B predicting the label in B given an object a ∈ A under a certain predefined searching

FIRST-ORDER METHODS FOR SMOOTH OPTIMIZATION

space h ∈ H. The quality of the prediction function h is measured by the risk,

R(h) = E P (a,b) [ℓ(h(a), b)],
where ℓ is a distance function measuring the difference between the predicted label h(a) and the true label b. Ideally, we would like to find the best prediction function minimizing the risk which gives the optimization problem min h∈H R(h).

(1.1) However, the latent distribution P is unknown and therefore the risk function R is directly uncomputable. Instead, we assume that n i.i.d samples (a i , b i ) i∈ [1,n] from P are provided, and we define the empirical risk with respect to this training set as

R emp (h) = 1 n n ∑ i=1 ℓ(h(a i ), b i ).
By the law of large numbers, the empirical risk converges to the expected risk when the number of samples n goes to infinity. Thus, we minimize the empirical risk among the class of functions defined by the model, which forms the empirical risk minimization problem (ERM) [START_REF] Shalev-Shwartz | Understanding machine learning: From theory to algorithms[END_REF]. When the model is parametric, this naturally transforms to an optimization problem

min x∈R p { 1 n n ∑ i=1 ℓ(h(a i , x), b i ) } , (1.2)
where x represents the model parameters. In linear regression, x corresponds to the coefficient of the linear representation a T i x; in neural networks, x contains all the layers' weights. Typical examples of loss functions are the square loss for least squares regression, i.e., ℓ(h, b) = (bh) 2 with b ∈ R and the logistic loss ℓ(h, b) = log(1 + exp(-bh)) for logistic regression with b ∈ {-1, +1}. As a consequence, the learning problem in the setting of supervised learning turns out to be an optimization problem, which motivates our work towards fast and efficient algorithms to solve the ERM problems.

First-order methods for smooth optimization

For the sake of clarity, we write the empirical risk minimization problem in the general form

min x∈R p f (x) .
(1.3) Convexity and smoothness. We shall first focus on the case where f is smooth and convex, which includes classical models such as linear regression and logistic regression. We assume the smoothness assumption at this first stage only for the sake of clarity, it allow us to define the gradient which is a natural descent direction. The extension to composite function settings will be discussed later in Section 1.4. If,in addition,inequality (1.4) can be improved with an additional quadratic term, namely there exists µ > 0 such that for any x, y in R p ,

f (y) ≥ f (x) + ⟨∇f (x), y -x⟩ + µ 2 ∥y -x∥ 2 , (1.5)
then we say that the function f is µ-strongly convex, where µ is the strong convexity parameter.

Convexity guarantees that all stationary points of f are in fact global minima. The strong convexity further guarantees that a global minimum exists and is unique. When f is two times continuously differentiable, the strong convexity assumption is equivalent to saying that the eigenvalues of the Hessian ∇ 2 f (x) are uniformly lower bounded by µ > 0 for all x in R p . Intuitively, a strongly convex function enjoys better local curvature and steeper slope, which allows optimization algorithms to converge faster.

As we can see, Equation (1.4) and (1.5) provide simple lower bounds on f . In the same spirit, we also assume that an upper bound is available, leading to the definition of L-smoothness. We say that f is L-smooth if f is differentiable and its gradient is L-Lipschitz, i.e. for any x, y in R p ∥∇f (y) -∇f (x)∥ ≤ L∥y -x∥.

(1.6)

Elementary calculation shows that this condition implies that for any x, y in R p , f (y) ≤ f (x) + ⟨∇f (x), y -x⟩ + L 2 ∥y -x∥ 2 , (1.7) which provides a quadratic upper bound; see details in Theorem 2.1.5 of [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]. Moreover, when f is two times continuously differentiable, L-smoothness is equivalent to having the eigenvalues of the Hessian ∇ 2 f (x) uniformly upper bounded by L, for any x in R p . To summarize, L and µ represent upper and lower bounds on the eigenvalues of the Hessian respectively, which provide simple quadratic upper and lower bounds on f . This brings us to the condition number of the function f , defined as

Q f = L µ , (1.8)
which plays an important role in the characterization of the convergence rate of first-order algorithms applied to f .

Gradient descent algorithm

We come back to our optimization problem (1.3). The goal is to design algorithms which move towards the minimum iteratively. Given that we are at point x, a natural idea is to follow the direction giving the steepest decrease in terms of the function value. Since the objective function f is differentiable, given any unit direction u, the local variation of function f around x following the direction u is given by df x (u) = ∇f (x) ⊤ u.

Thus, the direction in which the function decreases the most is the direction of anti gradient -∇f (x). This yields the classical gradient descent algorithm:

Gradient Descent algorithm

Initialization:

x 0 ∈ R d .
For k ≥ 0,

x k+1 = x k -η k ∇f (x k ),
where η k > 0 is the stepsize.

CHAPTER 1. INTRODUCTION

There are several strategies to set the sequence of stepsizes (η k ) k≥0 . A simple strategy is to choose a constant stepsize η k = η for all k ≥ 0, which enjoys the following convergence guarantee.

Theorem 1 (Convergence of gradient descent method). If f is convex and L-smooth, then by letting η k = 1 L , the sequence (x k ) k≥0 generated by the gradient descent algorithm satisfies

f (x k ) -f * ≤ L∥x 0 -x * ∥ 2 2k , ∀k ≥ 1, (1.9)
where f * is the minimum of f and x * is an optimal point with f (x * ) = f * . Moreover, if f is µ-strongly convex, then

f (x k ) -f * ≤ ( 1 - µ L ) k (f (x 0 ) -f * ) .
(1.10)

The proof is recalled in Appendix A.1. It is worth mentioning that the convergence rate shown in (1.9) is not tight, an exact worst case convergence rate for gradient methods has been recently derived by Taylor et al. [2017], De [START_REF] De Klerk | On the worst-case complexity of the gradient method with exact line search for smooth strongly convex functions[END_REF]. More precisely, they show that

∥x k -x * ∥ 2 ≤ ( 1 - µ L ) 2k ∥x 0 -x * ∥ 2 , (1.11) f (x k ) -f * ≤ ( 1 - µ L ) 2k (f (x 0 ) -f * ) , (1.12) and ∥∇f (x k )∥ 2 ≤ ( 1 - µ L ) 2k
∥∇f (x 0 )∥ 2 , (1.13) when applying gradient descent method with stepsize 1 L on a µ-strongly convex function. The proof is elementary and we refer readers who are interested to the original papers.

The convergence rate in Theorem 1 is stated respect to the function value gap, one can directly derive from it that the norm of the gradient ∥∇f (x k )∥ vanishes by using Theorem 2.1.5 of [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]:

1 2L ∥∇f (x k )∥ 2 ≤ f (x k ) -f * .
The quantity ∥∇f (x k )∥ may be naturally checked in practice and it provides information about convergence to a stationary point. So far, we have been considering convex objective functions, for which all stationary points are also global minima. Therefore, for convex objective functions, we are able to state stronger convergence guarantees in terms of function evaluations f (x k )f * . However, when the function becomes non-convex, we cannot hope to converge to a global minimum, and as a substitute, we study the convergence to a stationary point which is characterized by the gradient norm.

As shown by the theorem, the asymptotic convergence rate of the gradient descent algorithm is linear when the problem is strongly convex, which is much faster compared to the rate for non-strongly convex problems:

1.2. FIRST-ORDER METHODS FOR SMOOTH OPTIMIZATION Definition 2. We say that an algorithm M is linearly convergent on a problem f if there exists τ M in (0, 1], C M > 0 such that for any initial point x 0 in R p , M generates a sequence of iterates

(x k ) k≥0 satisfying f (x k ) -f * ≤ C M (1 -τ M ) k ∆(x 0 ), (1.14)
where ∆(x 0 ) is a quantity depending on x 0 measuring how good the initial point x 0 is. We will call τ M the linear convergence parameter of the algorithm M.

Note that the asymptotic behavior of the algorithm does not change as long as ∆(x 0 ) is finite, which is why we leave it unspecified on purpose. Typical examples are f (x 0 )f * , ∥x 0x * ∥ 2 or some kind of duality gap, depending on the considered algorithm. Unlike ∆(x 0 ), τ M and C M are quantities that do not depend on the initial point but on the algorithm M and the problem f itself. As we can see, for gradient descent, C M = 1 and τ M = µ/L, which is the inverse of the condition number of f . Consequently, the asymptotic behavior of gradient descent depends only on the condition number. The larger it is, the slower the algorithm converges, in which case the problem is called ill-conditioned.

To give an example of this phenomenon, let us consider a quadratic objective function f (x) = 1 2 x T Dx, where D is a diagonal matrix with coefficients

σ 1 ≥ σ 2 ≥ • • • ≥ σ d > 0.
The Hessian of f is then constant: ∇ 2 f (x) = D for any x ∈ R p . Consequently, we have L = σ 1 , µ = σ d , and the condition number Q = σ 1 /σ d . Now let us see how the gradient descent algorithm solves this problem. Given an initial point x 0 , one step of gradient descent gives

x 1 = x 0 - 1 σ 1 ∇f (x 0 ) = ( I - 1 σ 1 D ) x 0 .
Consequently, by induction, we have

x k = ( I - 1 σ 1 D ) k x 0 -→ x * = 0.
The rate of convergence of ∥x kx * ∥ is asymptotically controlled by the last coordinate of the vector, which is

x (d) k = ( 1 - σ d σ 1 ) k x (d) 0 = ( 1 - µ L ) k x (d) 0 .
This matches the convergence rate in terms of distance to the optimum shown in (1.11) and one can easily derive similar result in terms of function value gap. In the best case when L = µ, the quadratic function is isotropic and one step of gradient descent at any arbitrary point attains the minimum. Intuitively, the smaller the condition number is, the more isotropic the objective function is, which makes it more favorable to the gradient descent algorithm.

For later comparison of the computational complexity, we translate our convergence result in terms of numbers of gradient evaluations, which equals to the number of iterations in the gradient descent algorithm: CHAPTER 1. INTRODUCTION Corollary 1. Let ε > 0, the number of iterations N required to achieve an ε-solution f (x)-f * ≤ ε by the gradient descent method is

   O ( L ε ) if f is convex, O(Q f log ( 1 ε ) ) if f is µ-strongly convex.
(1.15)

There are other, sometimes more effective, strategies to set the sequence of stepsizes (η k ) k≥0 [START_REF]Updating quasi-Newton matrices with limited storage[END_REF]Wright, 2006, Bonnans et al., 2006]. A standard technique is to perform a line search. The exact line search strategy is described as follows: at each iteration k, we choose η k which gives the steepest decrease, namely

η k = arg min η>0 f (x k -η∇f (x k )).
Such a strategy is aggressive since we always take the best step-size possible given the direction of gradient. However, the exact stepsize is often not computable. A more practical way is to apply a backtracking line search strategy, that looks for a stepsize η k such that the following descent condition is satisfied,

f (x k -η k ∇f (x k )) ≤ f (x k ) - η k 2 ∥∇f (x k )∥ 2 .
When the inequality is violated, we decrease the stepsize η k by a constant factor and recheck it again. Note that the descent condition is satisfied for any stepsize η k < 1 L , thus the line search procedure is guaranteed to terminate. In terms of convergence result, the backtracking line search enjoys a similar asymptotic convergence rate as the constant step-size, see section 9.2 of [START_REF] Boyd | Convex optimization[END_REF]. Thus for later discussion, by gradient descent algorithm, we refer to the one with constant step-size [START_REF] Bertsekas | Convex Optimization Algorithms[END_REF].

Accelerated gradient method

A natural question is whether or not the gradient descent method is optimal. The answer is provided in part by the concept of minimax lower bound analysis, which indicates the best possible rate of convergence expected for a given class of algorithms. We constrain ourselves on firstorder methods because problems from machine learning potentially contain high-dimensional features, where the evaluation of higher-order differentials is infeasible in practice due to memory issues and prohibitive computational effort. Let us now introduce the notion of the iterative first-order method [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]: Assumption 1. We call an algorithm M an iterative first-order method if it generates a sequence of iterates (x k ) k≥0 such that

x k ∈ x 0 + Span {∇f (x 0 ), • • • , ∇f (x k-1 )} , for k ≥ 1.

FIRST-ORDER METHODS FOR SMOOTH OPTIMIZATION

This assumption may seem a bit restrictive because we constrain the trajectory to be a linear combination of past gradients, but we remark that most interesting first-order algorithms satisfy this definition including the gradient descent algorithm, its accelerated variants, conjugate gradient methods and quasi-Newton algorithms such as BFGS and L-BFGS algorithms. 1 We shall therefore recall the minimax lower bounds for first-order methods as defined by Assumption 1.

Theorem 2 (Lower bounds for convex functions [START_REF] Nemirovskii | Problem complexity and method efficiency in optimization[END_REF][START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]). Given the dimension d, for any k with 1 ≤ k ≤ 1 2 (d -1), and any x 0 in R p , there exists a convex L-smooth function f such that for any first-order method M satisfying Assumption 1,

f (x k ) -f * ≥ 3L∥x 0 -x * ∥ 2 32(k + 1) 2 , ∥x k -x * ∥ 2 ≥ 1 8 ∥x 0 -x * ∥ 2 .
Note that the above result is valid only when the number of iterations is bounded by half of the dimension, which does not provide any information about the asymptotic convergence. A dimension free lower bound in a similar rate was recently developed by [START_REF] Arjevani | Dimension-free iteration complexity of finite sum optimization problems[END_REF] relying on a stronger assumption, which excludes quasi-Newton methods.

Theorem 3 (Lower bounds for strongly convex functions [START_REF] Nemirovskii | Problem complexity and method efficiency in optimization[END_REF][START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]). Consider the space of square-summable sequences ℓ 2 (R). For any x 0 in ℓ 2 (R), any constant µ > 0, Q > 1, there exists a µ-strongly convex function f with condition number Q such that for any first-order method M satisfying Assumption 1, we have,

∥x k -x * ∥ 2 ≥ ( 1 - 2 √ Q + 1 ) 2k ∥x 0 -x * ∥ 2 , and f (x k ) -f * ≥ µ 2 ( 1 - 2 √ Q + 1 ) 2k ∥x 0 -x * ∥ 2 , for any k ≥ 1.
As we can see, there is a gap between the lower bounds and the rate of convergence obtained from the gradient descent algorithm, which gives hope for algorithms with faster rate. In 1983, Nesterov proposed an accelerated version of gradient method, which is optimal with respect to the minimax lower bound. Accelerated Gradient method input x 0 ∈ R d and α 0 ∈ (0, 1).

1: Set y 0 = x 0 and q = µ L .

2: At k-th iteration, (k ≥ 0) (a) Compute x k+1 = y k - 1 L ∇f (y k ). (b) Compute α k+1 ∈ (0, 1) from equation α 2 k+1 = (1 -α k+1 )α 2 k + qα k+1 .
(c) Compute y k with

y k+1 = x k+1 + β k (x k+1 -x k ) with β k = α k (1 -α k ) α 2 k + α k+1 .
The key ingredient for acceleration is to take the gradient at the extrapolated point y k instead of x k , where y k is a linear combination (but not necessarily an average) of the previous iterates. Nesterov's acceleration falls into the general class of schemes called extrapolation schemes in numerical analysis. Intuitively, y k moves one step further from x k following the "descent direction" x kx k-1 , yielding faster convergence. A lot of work has been devoted to provide an intuitive explanation of the magic behind Nesterov's acceleration scheme. For instance, y k can be viewed as a linear combination of a gradient descent step and a mirror descent step [START_REF] Allen-Zhu | Linear coupling: An ultimate unification of gradient and mirror descent[END_REF]; or geometrically, the algorithm can be interpreted as sequentially performing the intersection of balls [START_REF] Bubeck | A geometric alternative to nesterov's accelerated gradient descent[END_REF] or sequentially averaging some quadratic functions [Drusvyatskiy et al., 2016]; or from a continuous point of view, it can be seen as applying a multi-step integration schemes on the gradient flow equation [START_REF] Scieur | Integration methods and accelerated optimization algorithms[END_REF]. The important message is that by simply taking the gradient step at a different point, we can achieve the optimal rate of convergence: Theorem 4 (Convergence of accelerated gradient method [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]). If f is convex, the accelerated gradient algorithm generates a sequence (x k ) k≥0 such that

f (x k ) -f * ≤ 4L (k + 2) 2 ∥x 0 -x * ∥ 2 .
Moreover, when f is µ-strongly convex, we have

f (x k ) -f * ≤ L ( 1 - √ µ L ) k ∥x 0 -x * ∥ 2 .

QUASI-NEWTON METHODS

Corollary 2. Let ε > 0, the number of iterations N required to achieve an ε-solution f (x)-f * ≤ ε by the accelerated gradient descent is

   O (√ L ε ) if f is convex, O( √ Q log ( 1 ε ) ) if f is µ-strongly convex.
(1.16)

Due to the extrapolation step, the accelerated gradient method is not guaranteed to be a descent algorithm, meaning that it is possible to have f (x k+1 ) > f (x k ). One way to ensure the descent property is to introduce a restart scheme, which restarts the algorithm once some descent conditions are violated [START_REF] Candes | Adaptive restart for accelerated gradient schemes[END_REF]Candes, 2015, Fercoq and[START_REF] Fercoq | Restarting accelerated gradient methods with a rough strong convexity estimate[END_REF]. This forces the algorithm to make progress at each iteration and achieves significant improvement in practice. To summarize, the accelerated gradient descent algorithm improves the convergence rate compared to standard gradient methods and achieves the optimal minimax rate. However, as mentioned before, such a minimax rate only holds when the number of iteration is small, there may be hope to achieve even faster algorithms as the number of iterations increases. As we will see later, under slightly stronger assumption, quasi-Newton algorithms can asymptotically achieve superlinear convergence.2

Quasi-Newton methods

We have focused so far on first-order optimization algorithms. If the objective function is smooth enough, using higher order information may result in faster algorithms. Newton's method is one of the most successful optimization method combining first and second derivatives. Instead of performing the first-order approximation of the function, Newton's method goes one step further. It uses the second order Taylor expansion to approximate the objective function,

f (x k + h) ≈ f (x k ) + ∇f (x k ) T h + 1 2 h T ∇ 2 f (x k )h.
(1.17)

Minimizing this quadratic function leads to the update

x k+1 = x k - ( ∇ 2 f (x k ) ) -1 ∇f (x k ). (1.18)
When the objective is quadratic, one step of Newton's method gives the exact minimum. In general, when (x k ) k≥0 converges and the minimum has a nondegenerate Hessian, the sequence converges with a quadratic rate. However, for a general nonlinear function, the iterate (1.18) is not guaranteed to converge for any arbitrary initial point x 0 . This is because the step

(∇ 2 f (x k )) -1 ∇f (x k
) may be large and in which case the Taylor expansion (1.17) may no longer CHAPTER 1. INTRODUCTION be a good approximation of f . A quick solution to guarantee the convergence is to introduce a stepsize parameter η k > 0 in the update,

x k+1 = x k -η k ( ∇ 2 f (x k ) ) -1 ∇f (x k ), (1.19)
and ensure that the function value is decreasing, e.g. f (x k+1 ) < f (x k ). Such stepsize exists as long as the Hessian at x k is positive definite. While Newton's method enjoys a fast convergence rate, it is computationally expensive. At each iteration, the Hessian matrix needs to be evaluated and to be inverted, requiring at least an O(d 3 ) computational effort. Thus, the per-iteration cost of Newton's method is d 2 times higher than the gradient methods, which is often too expensive for solving large-scale machine learning problems. This is why Hessian-free and inverse-free modifications of Newton's method have been developed, giving birth to the class of Quasi-Newton methods, defined as the generalization of (1.18):

x k+1 = x k -B -1 k ∇f (x k ), (1.20)
The matrix B k represents an approximation of the Hessian matrix, varying from iteration to iteration, giving the alternative name: variable metric methods. The update in (1.20) can be viewed as performing a gradient descent step according to the metric induced by B k . The metrics B k are generated iteratively without evaluating the Hessian, and its inverse H k = B -1 k is given in a closed-form update, without any operation of matrix inversion. This makes Quasi-Newton algorithm computationally efficient. For historical reasons, we use B to denote approximations of the Hessian matrix and H to denote the inverse of the Hessian matrix.

The challenging point of Quasi-Newton algorithms lies in the construction of the Hessian approximations. In a few words, Quasi-Newton methods use the finite difference of the gradient to approximate the Hessian matrix (see Appendix C), which gives a fundamental equation called the secant equation: at (k + 1)-th iteration, the Hessian approximation B k+1 is required to satisfy the relationship

B k+1 s k = y k , with s k = x k+1 -x k , y k = ∇f (x k+1 ) -∇f (x k ).
(1.21)

Intuitively, the function f is locally approximated by the quadratic function

Q k+1 (x) = f (x k+1 ) + ∇f (x k+1 ) T (x -x k+1 ) + 1 2 (x -x k+1 ) T B k+1 (x -x k+1 ).
The secant equation (1.21) is requiring the gradient ∇f to match the gradient of the quadratic approximation ∇Q k+1 at the past iterate x k . Moreover, since the Hessian matrix is always symmetric, and positive definite if the function is strongly convex, it is natural to ask B k+1 to have the same properties. Thus, the goal is to construct a symmetric and positive definite matrix B k+1 satisfying the secant equation. However, this is an underdetermined problem based on the local estimation. In order to capture the past information, we would expect B k+1 to be "close" to B k , yielding the two most popular Quasi-Newton algorithms called Davidon-Fletcher-Powell

QUASI-NEWTON METHODS

(DFP) update [Davidon, 1991, Fletcher and[START_REF] Fletcher | A rapidly convergent descent method for minimization[END_REF] and Broyden-Fletcher-Goldfarb-Shanno (BFGS) update [START_REF] Broyden | The convergence of a class of double-rank minimization algorithms 1. general considerations[END_REF][START_REF] Fletcher | A new approach to variable metric algorithms[END_REF][START_REF] Goldfarb | A family of variable-metric methods derived by variational means[END_REF][START_REF] Shanno | Conditioning of quasi-newton methods for function minimization[END_REF].

In the DFP update, the matrix B k+1 is defined by

B DFP k+1 = ( I - y k s T k ⟨y k , s k ⟩ ) B k ( I - s k y T k ⟨y k , s k ⟩ ) + y k y T k ⟨y k , s k ⟩ , (1.22)
and the corresponding inverse matrix is obtained by applying the Sherman-Morrison formula [START_REF] Sherman | Adjustment of an inverse matrix corresponding to a change in one element of a given matrix[END_REF], giving

H DFP k+1 = H k + s k s T k ⟨s k , y k ⟩ - H k y k y T k H k y T k H k y k . (1.23)
In the BFGS update, the inverse Hessian approximation H k+1 is directly updated by

H BFGS k+1 = ( I - s k y T k ⟨y k , s k ⟩ ) H k ( I - y k s T k ⟨y k , s k ⟩ ) + s k s T k ⟨y k , s k ⟩ . (1.24)
A perfect symmetry can be observed from (1.22) and (1.24) by interchanging s k ↔ y k and B k ↔ H k . In fact, the underlying mechanism of the two algorithms are identical, where DFP approximates the Hessian matrix and BFGS approximates the inverse Hessian matrix. This is why BFGS is sometimes named the complementary DFP algorithm.

Quasi-Newton Method

: DFP/ BFGS input x 0 ∈ R p and H 0 ∈ R d×d . 1: At (k + 1)-th iteration, (k ≥ 0) (a) Compute the iterate x k+1 = x k -η k H k ∇f (x k ),
where the stepsize η k is determined by a line-search strategy or constant.

(b) Update H k+1 with s k = x k+1 -x k and y k = ∇f (x k+1 ) -∇f (x k ): H DFP k+1 = H k + s k s T k ⟨s k , y k ⟩ - H k y k y T k H k y T k H k y k ; H BFGS k+1 = ( I - s k y T k ⟨y k , s k ⟩ ) H k ( I - y k s T k ⟨y k , s k ⟩ ) + s k s T k ⟨y k , s k ⟩ . CHAPTER 1. INTRODUCTION
Both DFP and BFGS algorithms inherit a nice property of Newton's method, which is the affine invariance. More precisely, when considering a new problem g(z) = f (Az), where A in R d×d is a non singular matrix. Applying DFP and BFGS algorithm to the problem min z g(z) provide identical updates as the problem min x f (x), if the initialization metric satisfies the affine invariance relation H f 0 = AH g 0 A T . This suggests that the convergence of Quasi-Newton methods are independent of the condition number if we initialize them with an exact Hessian matrix. However, in practice, H 0 is usually set to be proportional to the identity matrix, which breaks the affine invariance property.

In order to guarantee convergence, the step size of Quasi-Newton updates need to be carefully chosen. A typical example is to perform a line-search:

(a) Exact line-search: Given a direction d, the step-size η is chosen to minimize the function in this direction:

η exact = arg min η f (x + ηd).
(b) Wolfe's conditions: Given a direction d, two scalars 0 < α < β < 1, the setpsize η is chosen to satisfy

f (x + ηd) ≤ f (x) + αη∇f (x) T d, (Armijo condition) ∇f (x + ηd) T d ≥ β∇f (x) T d. (curvature condition)
Note that the exact line-search is computationally expensive in practice, but it provides interesting theoretical insight. The convergence of the DFP or BFGS algorithms depends on the selected line-search strategy.

Theorem 5. Assume that the objective function f is twice differentiable, L-smooth and µstrongly convex.

1. If f is quadratic and the exact line search is used, both DFP and BFGS find the exact minimum in at most d iterations [START_REF] Broyden | Quasi-newton methods and their application to function minimisation[END_REF].

2. For a general function f , if the exact line search is used, then both DFP and BFGS algorithms converge linearly [START_REF] Powell | On the convergence of the variable metric algorithm[END_REF]. Furthermore, if the Hessian is Lipschitz, then both algorithms converge superlinearly, meaning that

∥x k+1 -x * ∥ ∥x k -x * ∥ → 0.
3. If the stepsize is always equal to 1, and the initialization is close enough to the solution, then both DFP and BFGS algorithms converge linearly. Furthermore, if the Hessian is Lipschitz, then both algorithms converge superlinearly. [START_REF] Dennis | A characterization of superlinear convergence and its application to quasi-newton methods[END_REF] 1.3. QUASI-NEWTON METHODS

4.

If the stepsize is determined by Wolfe's conditions, then the BFGS algorithm converges linearly. Furthermore, if the Hessian is Lipschitz, then the algorithm converges superlinearly. [START_REF] Byrd | Global convergence of a case of quasi-Newton methods on convex problems[END_REF]Nocedal, 1989] We remark that when equipped with the exact line-search, both DFP and BFGS algorithms are closely related to the conjugate gradient method, which exhibits a finite termination when the objective function is quadratic. Moreover, a global convergence result is proved for both methods using the exact line-search. However the scenario becomes quite different when a practical strategy is used, like a constant step-size or Wolfe's conditions. The DFP algorithm only enjoys local linear convergence while the BFGS algorithm achieves global convergence. An example is given in [START_REF] Powell | How bad are the BFGS and DFP methods when the objective function is quadratic?[END_REF] illustrating such a different behavior while using the constant step-size η k = 1. The objective function is a quadratic function, but the initialization B 0 is far from the true Hessian with an extremely large eigenvalue λ. The BFGS algorithm manages to correct it in a linear rate, namely λ k+1 ≈ λ k /2; but the DFP algorithm reduces it at a much slower rate, giving an update λ k+1 ≈ λ k -1, see details in [START_REF] Powell | How bad are the BFGS and DFP methods when the objective function is quadratic?[END_REF]. Therefore, it may take a large amount of iterations for the DFP algorithm to get into the region with linear convergence. This might explain why the DFP algorithm does not perform as well as the BFGS algorithm in practice. Thus, we will focus on the BFGS algorithm in the following discussion. In terms of convergence rate, the theoretical rate of BFGS algorithm with Wolfe's conditions is not better than the gradient descent algorithm, requiring O

( L µ log( 1 ε )
) iterations to achieve an ε-solution. On contrary, in practice, it is often observed that the BFGS algorithm significantly outperforms the gradient descent algorithm and its accelerated variant. Moreover, all the Quasi-Newton methods we mention are first-order algorithms satisfying Assumption 1 if the initialization B 0 or H 0 is proportional to the identity matrix. To verify this, it suffices to decompose the matrix-vector product H k ∇f (x k ) into a weighted sum of previous gradients; see Proposition 18 in the Appendix for more details.

When comparing to classical gradient methods, the sole additional ingredient used by Quasi-Newton methods is to maintain a Hessian approximation at each iteration. However, in the large-scale setting, the required d 2 space may still be too large to fit into memory. This point motivated the study of limited memory Quasi-Newtons methods, giving birth to the limited memory BFGS algorithm (L-BFGS) [START_REF] Liu | On the limited memory BFGS method for large scale optimization[END_REF]. The main idea is to use the decomposition of matrix-vector product mentioned before, where the update x k+1x k can be expressed as a weighted sum of the previous gradients. Thus, instead of keeping in memory the approximate Hessian matrix, one can replace it by a list of past gradients. The trick to limit the memory is to fix the maximum length ℓ for such a list and only keep the most recent gradients. As a consequence, the required memory is reduced from d 2 to ℓd where ℓ is selected by the user. In order to compute the matrix-vector product H k ∇f (x k ), a two-loop recursion strategy has been developed, yielding a complexity O(ld) instead of O(d 2 ) [START_REF] Nocedal | Numerical optimization[END_REF]. Thus the L-BFGS algorithm improves the BFGS algorithm both in terms of memory space and computational effort per iteration, which makes it practical. However, the superlinear convergence result is no longer valid since the limited memory may be too small to well approximate the Hessian matrix. It is shown that L-BFGS enjoys a linear convergence rate when the function is strongly convex; but the rate is again not better than the gradient methods.

As a summary, Quasi-Newton algorithms maintain a Hessian approximation to exploit the curvature of the objective function, a detailed overview with additional algorithms apart from DFP or BFGS is provided in Section C. In practice, the step-size needs to be selected carefully. The line-search using Wolfe's conditions usually provides promising performance [START_REF] Nocedal | Numerical optimization[END_REF]. However, in contrast to the significant speed up, the theoretical analysis does not give better convergence results compared to the gradient method, leaving a huge gap between theory and practice.

Proximal algorithms

Overfitting is a common issue when applying the empirical risk minimization framework. This phenomenon usually occurs when the number of samples is too small or the dimension of the features is too large, reminding that the training set is just a finite sample of the latent distribution. If the training set is too small, it may not reflect the underlying structure of the true distribution and, fitting a model on this dataset will most likely capture some irrelevant artifacts. If the dimension of the features is too large, the model may successfully predict any training data including noise and outliers, which will not generalize well to unseen data.

One solution for preventing overfitting is to add a regularization penalty in the ERM problem (1.2), e.g.

min x∈R p { 1 n n ∑ i=1 ℓ(h(a i , x), b i ) + ψ(x) } .
(1.25)

The purpose of the regularizer ψ is to impose some constraints and prior knowledge about the model parameters. For example, by adding the squared ℓ 2 -norm, known as Ridge regularization [START_REF] Engl | Regularization of inverse problems[END_REF], we penalize large entries in the parameter x. If we want to limit the number of selected features, we can apply sparsity-inducing regularizers. A natural candidate is the ℓ 0 -penalty defined by

∥x∥ 0 = #{i = 1, • • • , d | x i ̸ = 0}.
However, the ℓ 0 -penalty is non convex which leads to an NP-hard problem [START_REF] Natarajan | Sparse approximate solutions to linear systems[END_REF]. Therefore, to make the problem computationally tractable, we substitute the ℓ 0 -penalty by its convex relaxation -the ℓ 1 -norm. Regularizing the linear regression with the ℓ 1 -norm yields the well known Lasso problem [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]. By forcing the sum of the absolute values of the model parameter to be small, Lasso is able to encourage certain coefficients to be zero and induces sparsity. A number of Lasso variants have been developed later to improve the performance under different scenarios including Elastic-net regularization [Zou and Hastie, 2005], Group-Lasso with or without overlap [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF][START_REF] Jacob | Group lasso with overlap and graph lasso[END_REF][START_REF] Jenatton | Structured variable selection with sparsity-inducing norms[END_REF] and other sparsity-inducing regularizers [START_REF] Bach | Optimization with sparsity-inducing penalties[END_REF].

PROXIMAL ALGORITHMS

Another benefit of considering composite minimization is that it covers convex constrained minimization problems. This is achieved by defining ψ as the extended-valued indicator function 1 C of a convex domain C, which takes the value 0 inside the convex set C and +∞ outside (see [Hiriart-Urruty and Lemaréchal, 1996]). Thus, we consider the following composite optimization problem

min x∈R p { f (x) ≜ 1 n n ∑ i=1 f i (x) + ψ(x) } (1.26)
where each f i is convex, L-smooth and ψ is convex, but not necessarily differentiable. In the following, we will denote f 0 to be the smooth part of the objective, namely

f 0 = 1 n ∑ n i=1 f i .
A side effect of the regularization is that the objective function may be no longer differentiable, which is problematic to apply smooth optimization algorithms. In order to handle this issue, we introduce the notion of proximal operator: Definition 3. Given a convex function ψ : R p → R, the proximal operator of ψ is the function prox ψ : R p → R defined by

prox ψ (x) = arg min z∈R p { ψ(z) + 1 2 ∥z -x∥ 2 } (1.27)
When the regularization ψ is relatively simple, the associated proximal operator has a closed form solution, which is the case for ℓ 1 regularizer, the Elastic-net regularizer or Group-Lasso without overlap [START_REF] Combettes | Proximal splitting methods in signal processing[END_REF]. Moreover, when ψ is the indicator function 1 C , the proximal operator is simply the orthogonal projection onto the convex domain C. Thus, by using the proximal operator, gradient descent and accelerated gradient descent algorithms can be extended to solve composite problems such as

min x∈R p {f (x) = f 0 (x) + ψ(x)} ,
where f 0 is differentiable and L-smooth, representing the smooth part of the function. Proximal gradient methods perform alternatively the gradient descent step and the proximal step, yielding the class of iterative shrinkage-thresholding algorithms (ISTA) [START_REF] Daubechies | An iterative thresholding algorithm for linear inverse problems with a sparsity constraint[END_REF], also called forward-backward splitting method [START_REF] Gabay | Chapter ix applications of the method of multipliers to variational inequalities[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF][START_REF] Raguet | A generalized forward-backward splitting[END_REF][START_REF] Stella | Forward-backward quasi-newton methods for nonsmooth optimization problems[END_REF]. The accelerated variant is later proposed in [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Tseng | On accelerated proximal gradient methods for convex-concave optimization[END_REF][START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF], yielding an acceleration in the sense of Nesterov.

ISTA with constant stepsize

Initialization: x 0 ∈ R d and η = 1 L . For k ≥ 0, x k+1 = prox ηψ (x k -η∇f 0 (x k )) .
Compared to their smooth counterpart, proximal algorithms perform an additional proximal step after each gradient descent step. This can be interpreted as a maximization-minimization scheme, where the smooth part f 0 is approximated by a quadratic upper bound, namely

x k+1 = arg min x∈R p { f 0 (z) + ⟨∇f 0 (z), x -z⟩ + L 2 ∥x -z∥ 2 + ψ(x) } , (1.28)
where z is the center of the quadratic approximation. When ψ is the indicator of a convex set, the proximal operator projects the gradient step back to the constrained set, yielding a generalization of the projected gradient method. In terms of convergence, these proximal variants enjoy the exact same rate as the corresponding smooth variants. In practice, a more aggressive strategy to determine the stepsize is proposed by [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]. It consists in applying the backtracking line search and check whether a sufficient decrease is achieved. This strategy removes the dependency of the Lipschitz parameter L, which makes the method more practical.

FISTA with constant stepsize input x 0 ∈ R d and α 0 ∈ (0, 1).

1: Set y 0 = x 0 , η = 1 L and q = µ L . 2: At k-th iteration, (k ≥ 0) (a) Compute x k+1 = prox ηψ (y k -η∇f 0 (y k )). (b) Compute α k+1 ∈ (0, 1) from equation α 2 k+1 = (1 -α k+1 )α 2 k + qα k+1 .
(c) Compute y k with

y k+1 = x k+1 + β k (x k+1 -x k ) with β k = α k (1 -α k ) α 2 k + α k+1 .
While proximal gradient methods have gained a lot of success, efficiently extending Quasi-Newton methods to the composite setting is still not completely well understood. There are several attempts to develop a proximal Quasi-Newton method [START_REF] Byrd | An inexact successive quadratic approximation method for L-1 regularized optimization[END_REF][START_REF] Lee | Proximal Newton-type methods for convex optimization[END_REF][START_REF] Scheinberg | Practical inexact proximal quasi-Newton method with global complexity analysis[END_REF][START_REF] Yu | A quasi-Newton approach to non-smooth convex optimization[END_REF], providing the recurrence

x k+1 = arg min x∈R p { f 0 (x k ) + ⟨∇f 0 (x k ), x -x k ⟩ + 1 2 (x -x k ) T B k (x -x k ) + ψ(x) } .
(1.29)

A SURVEY OF INCREMENTAL ALGORITHMS

This is a direct extension of (1.28) by incorporating the Hessian approximation B k . However, the subproblem (1.29) does not have a closed form solution in general since B k is varying from iteration to iteration and it is usually non isotropic. Thus, an additional effort is required to solve the subproblem by applying an optimization algorithm, giving an inexact solution for the proximal operator. If the errors of the subproblem are small enough [START_REF] Byrd | An inexact successive quadratic approximation method for L-1 regularized optimization[END_REF], then the inexact proximal Newton method enjoys the same rate as in the smooth case in terms of the number of iterations. But the per-iteration cost is much higher. Moreover, the computational effort to solve the subproblem is hard to characterize since the condition number of B k may be poor. In practice, the heuristic of applying several passes of the randomized coordinate descent is commonly used. Another drawback is that the storage of the entire matrix B k seems inevitable, ruling out the limited memory variant.

Besides the proximal operator, another way to handle non differentiability is to introduce the notion of subgradients [Hiriart-Urruty and Lemaréchal, 1996]:

Definition 4. Given a convex function f : R p → R, for any point x ∈ R p , we define ∂f (x) = {g ∈ R p | ∀y ∈ R p , f (y) ≥ f (x) + ⟨g, y -x⟩}.
(1.30)

When f is differentiable at x, the set of the subgradients ∂f (x) reduces to the singleton {∇f (x)}. Then, a natural generalization of the gradient method is to replace the gradient ∇f (x k ) by an element g k in the subdifferential set ∂f (x k ). However, the stepsize of the subgradient method needs to be decreasing in order to ensure convergence, giving a much slower convergence rate compared to the smooth case. The number of iterations required to reach an ε-solution is O( 1 ε 2 ) when the problem is convex and is O( 1 ε ) when the problem is strongly convex [START_REF] Bertsekas | Nonlinear programming[END_REF][START_REF] Shor | Minimization methods for non-differentiable functions[END_REF]. Moreover, when we are looking for a sparse solution, the subgradient method usually will not lead to coordinates equal to zero. In contrast, the proximal gradient methods do provide sparse solutions by taking into account the composite structure.

A survey of incremental algorithms

For modern machine learning problems, both the amount of data n and the dimension of features d can be very large, which excludes higher-order optimization methods. Under this setting, the gradient methods that we have presented so far are also computationally expensive, because evaluating the full gradient requires to pass through all the n training points. This motivates the development of incremental gradient methods which have a cost per-iteration independent of n.

Let us start with the case where the objective function is smooth, i.e., min

x∈R p { f (x) ≜ 1 n n ∑ i=1 f i (x) } , (1.31)
and n is large. Due to the decomposable structure of the objective, incremental gradient methods can operate on a single component f i at each iteration instead of the entire sum in the objective. This reduces the computational complexity per iteration by a factor n. If each iteration of the incremental method yields reasonable progress in the objective, the resulting incremental method will outperform its non-incremental counterpart. This point explains the success of incremental methods in a wide variety of applications. One of the best known incremental method is the stochastic gradient descent method (SGD), which generates a sequence

x k+1 = x k -η k ∇f i k (x k ), (1.32)
where η k is the stepsize and i k is an index uniformly generated among 1 to n. Intuitively, we are performing an inexact gradient method at each iteration, where the full gradient ∇f (x k ) is approximated by the incremental gradient

∇f i k (x k ). The difference ∇f i k (x k ) -∇f (x k )
represents the noise of the approximation and its magnitude is controlled by the variance

E i k [∥∇f i k (x k ) -∇f (x k )∥ 2 ] = 1 n n ∑ i=1 ∥∇f i (x k ) -∇f (x k )∥ 2 .
In general, this variance is non zero even at the minimum x * , where the full gradient ∇f (x * ) vanishes but ∇f i (x * ) may not. Thus assuming that at some iteration we attain the exact minimum x k = x * , we can move away from it at the next iteration, meaning that x * is not a fixed point with respect to such dynamics. As a consequence, in order to ensure the convergence to the minimum, we need to decrease the stepsize η k , which reduces the magnitude of error. It is shown that with the choice η k = O(1/k), the algorithm achieves a sublinear convergence rate.

A proximal version of it is given in [START_REF] Bertsekas | Incremental proximal methods for large scale convex optimization[END_REF] enjoying a similar rate. However, unlike full gradient methods for which linear convergence can be obtained, the stochastic gradient descent does not converge faster than O(1/k) even for strongly convex functions. In practice, we also observe that it converges quickly in the first iterations but then slows down and attain some saturation due to the decreasing stepsize [START_REF] Bottou | Large-scale machine learning with stochastic gradient descent[END_REF].

A lot of recent work has been devoted to the study of faster incremental methods, including SAG [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF], SAGA [Defazio et al., 2014a], Finito/MISO [Defazio et al., 2014b[START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF], SDCA [START_REF] Shalev-Shwartz | Proximal stochastic dual coordinate ascent[END_REF] and SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF]. All of them converge linearly for strongly convex objectives, which significantly improves the performance of the stochastic gradient descent method. The main idea is to reduce the variance of the stochastic update such that the magnitude of the noise goes to zero while the iterates (x k ) k≥0 converge to x * . This will allow constant stepsizes, leading to a faster convergence.

Primal variance-reduction methods.

The key to reduce the variance is to incorporate information regarding past iterates. For primal methods such as SAG, SAGA and SVRG, the update is given by:

x k+1 = x k -η [ ∇f i k (x k ) -g i k σ + 1 n n ∑ i=1 g i ]
.

(1.33) 20

A SURVEY OF INCREMENTAL ALGORITHMS

where η is the stepsize, σ is a scaling parameter and g i represents a past incremental gradient of the component f i , for i ∈ [1, n]. The natural choice of σ is 1, in which case,

E i k [ ∇f i k (x k ) -g i k + 1 n n ∑ i=1 g i ] = ∇f (x k ),
leading to an unbiased gradient update. This is the case for SVRG and SAGA. Although they enjoy a similar update expression, a fundamental difference arises in the way of updating g i .

1. In SVRG, all the g i 's are updated simultaneously every m iterations, where m is a predefined parameter. More precisely, at initialization, all g i 's are initialized by ∇f i (x 0 ), for all i ∈ [1, n]; then they are kept unchanged during m iterations; at the m-th iteration, they are simultaneously updated by the gradients at x m namely ∇f i (x m ). Then again unchanged in the next m iterations, and the next update will be performed at 2m-th iteration, and so on.

2. In SAGA, the strategy is different. SAGA updates one g i at each iteration. At k-th iteration, one index i k ∈ [1, n] is selected and x k+1 is obtained following (1.33). After that, g i k is updated by the latest gradient, namely

g i k = ∇f i k (x k ).
Then we move to the next iteration.

Intuitively, SVRG updates the past gradients with delay while SAGA updates them in an online fashion, which makes SAGA more aggressive but less stable. SAG updates the past gradient in an identical way as SAGA does, the only difference is that SAG chooses σ = n instead of σ = 1. The biased gradient update provides further reduction of the variance, see [Defazio et al., 2014a] for more details.

The difference in terms of the update strategy leads to different requirements in terms of memory. For SVRG, the required memory is two vectors of dimension d, one for the iterate x km and the other one for the full gradient ∇f (x km );3 for SAGA, the required memory is n vectors of dimension d, one for each g i . This makes a huge difference, since when n × d is large, the amount of memory required by SAGA becomes prohibitive. For the incremental methods we presented here, including the upcoming dual methods SDCA, Finito or MISO, SVRG is the only one enjoying such "memory-free" property. Fortunately, for problems involving linear models, it is possible to reduce the memory requirement of these methods from n vectors to n scalars.

Dual variance reduction methods. Now we consider the dual-based incremental algorithms, such as SDCA and Finito/MISO. In contrast to the primal algorithm which controls the primal gap f (x k )f * , dual-based algorithms construct some lower bounds d k of the objective function and ensure the convergence of the duality gap f * -d k (x k ). In SDCA, the stochastic coordinate ascent algorithm is applied on the dual problem, which is obtained thanks to the Fenchel conjugate. Amazingly, while SDCA uses the concept of duality in behind, the update of the algorithm can be performed without evaluating the dual function, named as dual-free SDCA [START_REF] Shalev-Shwartz | SDCA without duality, regularization, and individual convexity[END_REF]. The advantage of such an approach comes from the fact that the gradient of the Fenchel conjugate is not always easy to evaluate, a purely primal update makes the algorithm more practical. Closely related to SDCA, MISO substitutes the Fenchel conjugate by the quadratic lower bounds given by the strong convexity. Since the quadratic functions are more computationally friendly compared to the convex conjugate function, MISO provides systematically a quantity d k (x k ) lower bounding f * .

Despite the difference in the choice of the lower bound function, SDCA and MISO enjoy a very similar update rule. Both of them keep in memory n auxiliary vectors z 1 , . . . , z n , one for each component f i . At each iteration, an index i k ∈ [1, n] is randomly selected and the corresponding auxiliary vector is updated by:

(SDCA & MISO) z i k = (1 -δ)z i k + δ ( x k - 1 µ ∇f i k (x k ) ) .
Then, the next iterate is obtained by averaging over all of the auxiliary vectors:

x k+1 = 1 n n ∑ i=1 z i .
Each z i carries the information of the i-th component and contributes equally to the iterate x k , which reflects the averaging structure of f . The only difference between SDCA and MISO comes from the choice of the parameter δ: in SDCA δ = µn L+µn and in MISO, δ = min{1, µn 2(L-µ) }. It is interesting to see that different approaches converging to a similar algorithm even though the proof cannot be adapted from one to another. The benefit of dual-based methods, is that an upper bound on the function value gap f (x k )f * is provided by the duality gap f (x k )d k (x k ), which is easily computable and thus can be used to stop the algorithm. In general, the function value gap f (x k )-f * is not available for primal algorithms simply because f * is unknown. When the function is smooth, the norm of the gradient can be used as a substitution, but it will be more challenging for non smooth functions.

Another related direction is to develop incremental algorithms for the dual-averaging method [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF]. The main idea is to move along the accumulated gradient, namely

x t+1 = x t - η t t ∑ k=1 ∇f (x k ). (1.34)
An incremental version of this has been provided by [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] by replacing the full gradient ∇f (x k ) by the incremental gradient ∇f i k (x k ) in (1.34). The resulting algorithm enjoys a similar convergence rate as SGD, which is sublinear even for strongly convex objectives. Variancereduced variants have been later studied by [START_REF] Murata | Stochastic dual averaging methods using variance reduction techniques for regularized empirical risk minimization problems[END_REF], mimicking the approach of SVRG.

A SURVEY OF INCREMENTAL ALGORITHMS

So far, we have only considered smooth problems. In order to adapt to the composite setting, proximal variants are typically developed based on the original smooth incremental method [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF][START_REF] Shalev-Shwartz | Proximal stochastic dual coordinate ascent[END_REF][START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF], Defazio et al., 2014a].

The first contribution of this manuscript is to introduce a proximal variant of Finito/MISO algorithm and provide the convergence analysis, which is the main focus of Chapter 2.

The comments about convergence and memory for the smooth versions still hold for the proximal versions: proximal SVRG is the only algorithm that requires an amount of O(d) space in memory instead of O(nd). In summary, the mentioned variance reduction incremental algorithms all enjoy linear convergence rates when the objective function is strongly convex. This outperforms classical SGD algorithms in terms of optimization error. The downside is that a non negligible amount of information needs to be stored in memory, except for SVRG. Nevertheless, the performance of SVRG is often slower in practice than the other memory-consuming methods. Thus, we have to make a choice regarding the trade-off between memory and speed.

Compared to full gradient methods, the per-iteration cost of incremental methods is n times smaller. In order to make a fair comparison, we count the computational complexity in the numbers of evaluations for a single gradient ∇f i . Thus evaluating the full gradient requires to sum up the gradient of each component, yielding a complexity O(n). The computational complexity for reaching an ε-solution is provided in the following table :  In the strongly convex setting, the complexity of full gradient methods (FG) depends multiplicatively on the number of components n and the condition number L f /µ. In comparison, incremental methods like SAG/SAGA, Finito/MISO, SDCA and SVRG all enjoy a better complexity, where the dependence on the number of components and the condition number is additive. This improvement becomes significant when the condition number is large. In typical machine learning problems like ridge regression, the regularization parameter is in the order of 1/n giving a condition number in the order of n, in which case the full gradient methods lose a large factor n compared to incremental methods.

In contrast, when the problem is not strongly convex, there is no theoretical improvement in terms of complexity; SAG/SAGA enjoys the same complexity as full gradient methods. Moreover, several algorithms like Finito/MISO, SDCA and SVRG are not defined in the non strongly convex setting. Thus two natural questions arise: are these algorithms optimal and can we adapt them in the convex but non-strongly convex settings?

The first accelerated incremental method was developed by Shalev-Shwartz and [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF], where an accelerated version of SDCA has been proposed. The complexity of the resulting algorithm is given in Table 1.1, where the dependence on the condition number is improved by its squared root, yielding an acceleration in the sense of Nesterov. Compared to SDCA, the acceleration occurs when the condition number L/µ is larger than n, e.g., for poorly

CHAPTER 1. INTRODUCTION Strongly convex: µ > 0 Convex: µ = 0 FG O ( n L f µ log ( 1 ε ) ) O ( n L f ε ) SAG O (( n + L µ ) log ( 1 ε ) ) 4 O ( n L ε ) SAGA Finito/MISO not avail. SDCA SVRG Acc-FG O ( n √ L f µ log ( 1 ε ) ) O ( n √ L f ε ) Acc-SDCA Õ (( n + √ nL µ ) log ( 1 ε ) ) not avail. Lower bounds O (( n + √ nL µ ) log ( 1 ε ) ) O (√ nL ε
)

Table 1.1 -Comparison of rates of convergence, respectively in the strongly-convex and non strongly-convex cases. The notation Õ for acc-SDCA hides logarithmic terms in the condition number and n. We remark that there is a difference between the Lipschitz constant L f used for full gradient method and the one L used in incremental methods. In full gradient method, the Lipschitz constant is with respect to the entire function f , whereas each component f i needs to be L-smooth for incremental methods. Thus, we always have L f ≤ L, but L f may be much smaller than L. Incremental methods are thus outperforming full gradient methods for the finite-sum problem we consider here unless there is a big mismatch between L and L f . conditioned problems. However, the provided analysis is restricted to strongly convex problems like the original SDCA. A more general scheme has been developed later by [START_REF] Frostig | Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization[END_REF] for smooth strongly convex problems.

Whether or not incremental algorithms other than SDCA can be accelerated in the composite setting remained an open question at the beginning of this thesis. Moreover, it was also not clear whether an acceleration may be obtained for convex but not strongly convex problems. This brings us to one of the main contribution of this thesis. We present in Chapter 3 the first generic acceleration scheme called Catalyst [START_REF] Lin | A universal catalyst for first-order optimization[END_REF], which provides a positive answer for both open questions. It can be applied to a large class of gradient-based algorithms, yielding the acceleration in the sense of Nesterov for both strongly convex and non-strongly convex objectives.

The lower bound for the class of incremental methods have been investigated later by [START_REF] Agarwal | A lower bound for the optimization of finite sums[END_REF], [START_REF] Woodworth | Tight complexity bounds for optimizing composite objectives[END_REF], [START_REF] Arjevani | Dimension-free iteration complexity of finite sum optimization problems[END_REF]. It is shown that accelerated SDCA [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF] and Catalyst [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] are optimal

A SURVEY OF INCREMENTAL ALGORITHMS

up to a logarithmic factor. A lot of other works have recently been developed towards obtaining optimal incremental methods, such as APCG [START_REF] Lin | An accelerated proximal coordinate gradient method[END_REF], SDPC [Zhang and Xiao, 2015a], RPDG [START_REF] Lan | An optimal randomized incremental gradient method[END_REF], Point-SAGA [START_REF] Defazio | A simple practical accelerated method for finite sums[END_REF] and Katyusha [Allen-Zhu, 2016]. We remark that their techniques are algorithm-specific and cannot be directly generalized into a unified scheme. These methods enjoy the tight optimal convergence rate while our method loses a logarithmic factor. This may be the price to pay for providing a generic framework. While first-order incremental algorithms have caught a lot of attention, incorporating curvature information in the large finite sum setting remains a challenging problem. The direct application of substituting the full gradient ∇f by an incremental gradient ∇f i in the Quasi-Newton update is unfortunately not convergent. The reason is that such a substitution does not provide unbiased estimation of the Quasi-Newton update B -1 k ∇f since there is a matrix inversion. Thus, it seems hopeless to incrementally update both the gradient and the Hessian estimate at the same time. An alternative for getting rid of this issue is to use a sub-sampled Hessian for estimating B k . This gives the so called stochastic Quasi-Newton method (SQN) [START_REF] Mokhtari | Global convergence of online limited memory bfgs[END_REF][START_REF] Byrd | A stochastic quasi-Newton method for large-scale optimization[END_REF][START_REF] Gower | Stochastic block BFGS: Squeezing more curvature out of data[END_REF][START_REF] Moritz | A linearly-convergent stochastic L-BFGS algorithm[END_REF]. However, the construction of SQN highly depends on the smoothness of the objective function, which makes it difficult to generalize to the composite setting. Moreover, as mentioned in section 1.4, proximal Quasi-Newton methods suffer the drawback of requiring a non isotropic proximal operator at each iteration, where an exact solution is infeasible in practice.

Thus, a theoretically grounded variable metric algorithm in the composite setting is still missing in the literature. In Chapter 4, we introduce the first generic variable metric framework, QuickeNing, applicable to the composite finite sum structure. The algorithm can be interpreted as applying an inexact L-BFGS algorithm to the classical smoothing technique called infimum convolution or the Moreau-Yosida regularization, that we will present in Section 1.6. Global convergence analysis is provided followed by promising experimental results.

We should mention that the idea of combining second-order or quasi-Newton methods with Moreau-Yosida regularization is in fact relatively old. It may be traced back to variable metric bundle methods [START_REF] Chen | Proximal quasi-Newton methods for nondifferentiable convex optimization[END_REF][START_REF] Fukushima | A globally and superlinearly convergent algorithm for nonsmooth convex minimization[END_REF][START_REF] Mifflin | A quasi-second-order proximal bundle algorithm[END_REF], which use BFGS updates on the Moreau-Yosida smoothing of the objective and bundle methods to approximately solve the corresponding sub-problems. However, no global complexity was known, especially no complexity guarantee was given regarding the complexity of solving the subproblems. Therefore, the first contribution of QuickeNing lies in the global convergence analysis. Moreover, QuickeNing can be applied to incremental algorithms. The resulting algorithm successfully incorporate the curvature information with the incremental algorithms as expected.

Up to now, we have only focused on the minimization of convex problems. However, many interesting machine learning applications involves non-convex problems in extremely large-scale CHAPTER 1. INTRODUCTION settings, such as neural networks. Thus, efficiently extending incremental algorithms to nonconvex problems has high interest.

A variant of SVRG for non-convex problems has been recently introduced and analyzed [Reddi et al., 2016b]. One challenge we tackle in Chapter 5 is to extend all the previous incremental methods designed for convex objectives to work in a non-convex context. As we will see, we propose a variant of Catalyst that does not require any knowledge about the convexity of the function to operate.

The Moreau-Yosida regularization

In this section, we introduce a fundamental tool from convex analysis called the Moreau-Yosida envelope [START_REF] Moreau | Fonctions convexes duales et points proximaux dans un espace hilbertien[END_REF][START_REF] Yosida | Functional analysis[END_REF], which plays a key role for understanding generic algorithms such as Catalyst and QuickeNing. This tool can be seen as a smoothing technique, that can turn any convex lower semicontinuous function f into a smooth function with Lipschitz gradients. More precisely, it consists of the infimal convolution of f with a quadratic penalty:

F (x) ≜ min z∈R p { f (z) + κ 2 ∥z -x∥ 2 } , (1.35)
where κ is a positive regularization parameter. Recalling that the proximal operator is then defined as the unique minimizer of the problem-that is,

p(x) ≜ prox f /κ (x) = arg min z∈R p { f (z) + κ 2 ∥z -x∥ 2 } .
(1.36)

Unlike the proximal operator of the simple regularizer ψ defined in (1.27), p(x) does not admit a closed-form solution in general. Therefore, computing it requires to solve a sub-problem with high accuracy thanks to some iterative algorithm.

∇F (x) = κ(x -p(x)).
(1.37)

Moreover the gradient ∇F is Lipschitz continuous with L F = κ.

3. If f is µ-strongly convex, then F is µ F -strongly convex with µ F = µκ µ+κ .

Interestingly, F inherits from the convexity of f and more importantly it is always L-smooth. Moreover, when f is µ-strongly convex, the condition number of F is given by

Q F = L F µ F = µ + κ µ ,
which can be arbitrarily well conditioned by choosing a small value of κ. Naturally, since both functions admit the same solutions, a naive approach for minimizing a non-smooth function f is to apply a smooth optimization algorithm on F . Different variants of proximal point algorithm are obtained by respectively applying the gradient descent method, the Nesterov's accelerated gradient method and Quasi-Newton methods on the Moreau-Yosida envelope F .

The proximal point algorithm. Let us consider gradient descent steps on F :

x k+1 = x k - 1 L F ∇F (x k ).
Noticing from (1.37), we have ∇F (x k ) = κ(x kp(x k )) and L F = κ. Thus,

x k+1 = p(x k ) = arg min z∈R p { f (z) + κ 2 ∥z -x k ∥ 2 } ,
which is exactly the proximal point algorithm [START_REF] Martinet | Régularisation d'inéquations variationnelles par approximations successives[END_REF][START_REF] Rockafellar | Monotone operators and the proximal point algorithm[END_REF].

Accelerated proximal point algorithm. We now consider applying Nesterov's accelerated gradient method on F :

x k+1 = y k - 1 L F ∇F (y k ) and y k+1 = x k+1 + β k+1 (x k+1 -x k ),
where β k+1 is Nesterov's extrapolation parameter [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]. Again from (1.37), the update can be rewritten as

x k+1 = p(y k ) and y k+1 = x k+1 + β k+1 (x k+1 -x k ),
which is known as the accelerated proximal point algorithm of [START_REF] Güler | New proximal point algorithms for convex minimization[END_REF].

CHAPTER 1. INTRODUCTION

Variable metric proximal point algorithm. Let us now apply Quasi-Newton methods on F :

x k+1 = x k -H k ∇F (x k ) = x k -κH k (x k -p(x k )),
where the inverse Hessian matrix H k is constructed using previous gradients of F . This is known as the variable metric proximal point algorithm [START_REF] Burke | A variable metric proximal point algorithm for monotone operators[END_REF]. As a consequence, the generated iterates (x k ) k≥0 converge to the minimum of F , which is also the minimum of f . Furthermore, the convergence rate of the above algorithms can be directly derived from the convergence of the underlying smooth optimization methods. Since the condition number of F can be arbitrarily good, the convergence of them can be arbitrarily fast with a small enough regularization parameter κ.

While these algorithms are conceptually elegant, they suffer from a major drawback in practice, which is the exact evaluation of the proximal operator p(x) at each iteration. Unless a closed form is available, which is almost never the case, an iterative algorithm is required to approximate the proximal operator. This raises two essential issues: first, the approximate proximal operator encounters inexactness in the gradient evaluation, the magnitude of the inexactness should be carefully addressed in order to ensure the convergence of the algorithm; second, an additional computational effort is required by applying an iterative algorithm to approximate the proximal operator, the resulting computational cost needs to be controlled and included in the convergence analysis.

The inexactness criteria have been intensively investigated. Different criteria have been developed in the context of the proximal point algorithm, starting from the pioneer work of [START_REF] Rockafellar | Monotone operators and the proximal point algorithm[END_REF], followed by [START_REF] Auslender | Numerical methods for nondifferentiable convex optimization[END_REF], [START_REF] Correa | Convergence of some algorithms for convex minimization[END_REF], [START_REF] Solodov | A unified framework for some inexact proximal point algorithms[END_REF], [START_REF] Fuentes | Descentwise inexact proximal algorithms for smooth optimization[END_REF]. While the proximal point algorithm has caught a lot of attention, very few works have focused on its accelerated variant. The first accelerated proximal point algorithm with inexact gradients was proposed by [START_REF] Güler | New proximal point algorithms for convex minimization[END_REF]. Then, [START_REF] Salzo | Inexact and accelerated proximal point algorithms[END_REF] developed a more rigorous convergence analysis, generalized later by [START_REF] Devolder | First-order methods of smooth convex optimization with inexact oracle[END_REF]. However, the considered inexactness criteria for the accelerated variants are unpractical, leaving some space for improvement. As a direct extension, the strategy of combining Quasi-Newton methods and the Moreau-Yosida regularization has been applied to develop variable metric bundle methods [START_REF] Fukushima | A globally and superlinearly convergent algorithm for nonsmooth convex minimization[END_REF][START_REF] Mifflin | A quasi-second-order proximal bundle algorithm[END_REF], Lemaréchal and Sagastizábal, 1997b[START_REF] Chen | Proximal quasi-Newton methods for nondifferentiable convex optimization[END_REF].

In the meantime, the computational complexity for approximating the proximal operator is not investigated. This is the reason why the global complexity of proximal point algorithms is not well understood. To the best of our knowledge, the first attempt in this direction is provided in the accelerated SDCA algorithm [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF], which is an instance of inexact accelerated proximal point algorithm, even though this was not explicitly stated in the original paper. Their analysis only holds in the strongly convex setting and it was originally designed for the stochastic dual coordinate ascent method. Inspired by this, we extend their idea into generic acceleration schemes, providing global convergence analysis for different variants of the proximal point algorithms.

CONTRIBUTIONS OF THIS THESIS

Contributions of this thesis

We outline here the main contributions of this thesis following the organization of the manuscript.

• In Chapter 2, we present Prox-MISO, the proximal version of the incremental algorithm Finito/MISO [Defazio et al., 2014b[START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF]. The resulting algorithm extends Finito/MISO to the composite problems (1.26). Moreover, it removes the big data condition n ≥ 2 L µ originally required in Finito/MISO. We show that the algorithm is linearly convergent when the problem is strongly convex and that the number of iterations required to achieve an ε-solution is given by

O (( n + L µ ) log ( 1
ε ) )
.

(1.38)

Since Prox-MISO builds a simple lower bound of the objective function, a practical optimality certificate is systematically obtained, which is a benefit compared to primal algorithms.

• In Chapter 3, we introduce a generic acceleration scheme that applies to a large class of algorithms. We call our approach a "catalyst" by analogy with substances that increase chemical reaction rates. Catalyst takes an optimization method M as input and applies it to solve a series of subproblems in order to achieve acceleration. A method M may be accelerated if it enjoys linear convergence rate for strongly convex problems. This is the case for full gradient methods [Beck andTeboulle, 2009, Nesterov, 2013] and block coordinate descent methods [Nesterov, 2012b, Richtárik and[START_REF] Richtárik | Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function[END_REF], which already have wellknown accelerated variants. More importantly, it also applies to incremental algorithms such as SAG [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF], SAGA [Defazio et al., 2014a], Finito/MISO [Defazio et al., 2014b[START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF], SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF] and SDCA [START_REF] Shalev-Shwartz | Proximal stochastic dual coordinate ascent[END_REF]. Whether or not these methods could be accelerated was an important open question. It was only known to be the case for dual coordinate ascent approaches such as SDCA for strongly convex objectives [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF].

Our work provides a universal positive answer regardless of the strong convexity of the objective. Let us illustrate the phenomena of acceleration with the example of the Prox-MISO algorithm.

1. When the problem is strongly convex, Catalyst applying to Prox-MISO yields the complexity

Õ (( n + √ n L µ ) log ( 1 ε ) ) , (1.39)
where Õ hides logarithmic constant in terms of n and the condition number. The obtained complexity outperforms the one of (1.38) when the problem is ill-conditioned.

CHAPTER 1. INTRODUCTION 2.
When the problem is convex, Catalyst applying to Prox-MISO yields the complexity

Õ   √ nL ε   , (1.40)
where Õ hides logarithmic constant in terms of n, the condition number and 1 ε . We should mention that the Prox-MISO algorithm is not defined for non-strongly convex objectives, but it is theoretically grounded to apply it inside Catalyst. Thus, our work automatically provides support for non-strongly convex objectives even though the input method is not defined under such setting. This is a side achievement besides the acceleration.

The resulting complexity in (1.39) and (1.40) match the minimax lower bounds later developed by [START_REF] Woodworth | Tight complexity bounds for optimizing composite objectives[END_REF], [START_REF] Arjevani | Dimension-free iteration complexity of finite sum optimization problems[END_REF] up to a logarithmic factor. More generally, for all the incremental algorithms mentioned before, applying Catalyst provides both acceleration and support for non-strongly convex objectives with a similar convergence rate. Experiments are conducted to show that the acceleration is useful in practice, especially for ill-conditioned problems where we observe significant improvement.

• Chapter 4 is dedicated to another generic acceleration approach called QuickeNing, which uses the Quasi-Newton principles to accelerate gradient-based optimization methods. To the best of our knowledge, QuickeNing is the first Quasi-Newton type algorithm compatible with both the composite objectives and the finite sum setting.

The algorithm admits a simple interpretation: it may be seen as applying the L-BFGS algorithm with inexact gradients to the Moreau-Yosida regularization of the objective. Given an optimization method M, we apply M to solve subproblems which correspond to approximating gradients of the Moreau-Yosida regularization. As a result, our approach is generic, and can be applied to the same class of functions as Catalyst. A particular warm start strategy is proposed in order to avoid the line search step in the determination of the stepsize, which makes the algorithm practical.

The global complexity of the algorithm is given, showing that a linear convergence rate can be obtained when the problem is strongly convex. This is the first global complexity result of the variable metric proximal point algorithm involving the complexity for solving the subproblems. More importantly, the convergence analysis provides us an explicit way to set up parameters, which are keys to obtain fast convergence in practice.

We remark that QuickeNing algorithm does not provide any theoretical acceleration in the worst case complexity analysis. This comes from the fact that the theoretical guarantee of L-BFGS algorithm is not better than the gradient descent method. However, we provide extensive experiments showing that QuickeNing gives significant improvement over competing methods in large-scale machine learning problems.

CONTRIBUTIONS OF THIS THESIS

• In Chapter 5, we extend Catalyst algorithm to the nonconvex setting. This is a joint work with Courtney Paquette and Dmitriy Drusvyatskiy from University of Washington. The strength of the approach lies in the ability of automatically adapting the convexity, meaning that we can run the algorithm without specifying that the objective is convex or not. A particular step called Auto-adapt is designed in the algorithm to exploit the local convexity.

When the objective is convex, the proposed approach enjoys the same convergence rate as the original Catalyst algorithm, leading to an acceleration. When the objective is nonconvex, it achieves the best known convergence rate to stationary points for first-order methods. As a consequence, it automatically transforms a convex optimization method to adapt to the nonconvex setting. We present promising experimental results when applied it to problems of sparse matrix factorization and neural networks.

Chapter 2

Proximal minimization by incremental surrogate optimization (MISO)

Chapter abstract:

The approach MISO, proposed by [START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF], is an incremental technique for minimizing unconstrained finite sum problems. The original algorithm is limited to smooth strongly convex problems and requires a big data condition such that the number of components n is larger than the condition number L/µ. In this chapter, we allow the MISO approach to deal with composite problems. Moreover, we remove the big data assumption with a new proof technique. The resulting algorithm provides systematically a lower bound of the optimum which is simple to compute, leading to a practical optimality certificate.

The material of this chapter is part of the following publication:

H. Lin, J. Mairal, and Z. Harchaoui. A universal catalyst for first-order optimization. In Advances in Neural Information Processing Systems (NIPS), 2015

Introduction

Incremental algorithms have recently caught a lot of attention due to their ability of solving large-scale machine learning problems. Just like gradient descent methods, incremental algorithms are first studied in the smooth convex setting and extended later to composite problems. For instance, Proximal-SDCA [START_REF] Shalev-Shwartz | Proximal stochastic dual coordinate ascent[END_REF] and Proximal-SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF] are direct proximal extensions of SDCA [START_REF] Shalev-Shwartz | Stochastic dual coordinate ascent methods for regularized loss minimization[END_REF] and SVRG [START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF]; SAGA [Defazio et al., 2014a] is an implicit proximal variant of SAG [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF]. The smooth variants usually enjoy simpler expressions

CHAPTER 2. PROXIMAL MINIMIZATION BY INCREMENTAL SURROGATE OPTIMIZATION (MISO)

and easier convergence analysis compared to the proximal variants. For this reason, the smooth setting is often considered in the first place. Finito/MISO is another incremental approach, proposed simultaneously by Defazio et al. [2014b] and [START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF] for solving smooth unconstrained µ-strongly convex problems. We will call it MISO in the following discussion, which refers to Minimization by Incremental Surrogate Optimization. The concept of surrogate is about to replace the original objective f by a simpler approximation h and minimize the approximation h instead of f . A typical example is the Lipschitz gradient surrogate. Given a point y, we define a majorant surrogate of f near the point y by

h y : x → f (y) + ⟨∇f (y), x -y⟩ + L 2 ∥y -x∥ 2 .
Minimizing h y gives the classical gradient descent update y -1 L ∇f (y). Thus, the gradient descent method can be viewed as sequentially minimizing the Lipchitz gradient surrogate near the previous iterate. In the same way, proximal gradient methods can be obtained from the proximal gradient surrogate which is the sum of the non-smooth regularization and the Lipschitz gradient surrogate. Other well known optimization methods including accelerated gradient methods, block coordinate descent method, Franck-Wolfe method can also be interpreted as the minimization of surrogate functions [START_REF]Optimization with first-order surrogate functions[END_REF].

The original MISO algorithm

Consider the following smooth unconstrained convex minimization problem

min x∈R p { f (x) ≜ 1 n n ∑ i=1 f i (x) } , (2.1)
where each f i is differentiable with L-Lipschitz continuous derivatives and are µ-strongly convex.

The idea of MISO 1 is to construct a list of quadratic surrogate functions (d i ) i∈ [1,n] , where d i represents the "dual" of the component f i . At each iteration, one of the surrogates is updated, using information of the latest iterate. More precisely, at k-th iteration, an index i k is randomly picked up among [1, n] and we perform the following update given the last iterate x k-1 :

d k i (x) = { f i (x k-1 ) + ⟨∇f i (x k-1 ), x -x k-1 ⟩ + µ 2 ∥x -x k-1 ∥ 2 if i = i k d k-1 i (x) otherwise ,
where d k i denotes the i-th surrogate at the k-th iteration. Then, the iterate is updated as the minimum of the entire surrogate,

x k = arg min x∈R p { dk (x) = 1 n n ∑ i=1 d k i (x) } .
(2.2)

The subproblem (2.2) is nothing but a minimization of a quadratic function which can be easily solved. Let us denote z k i as the minimum point of d k i , namely the expression

d k i (x) = c k i + µ 2 ∥x -z k i ∥ 2 ,
holds for some constant c k i . Then the update rule of z k i is given by

z k i = { x k-1 -1 µ ∇f i (x k-1 ) if i = i k z k-1 i otherwise ,
Consequently, the iterate x k is simply the average of the centers z k i :

x k = 1 n n ∑ i=1 z k i = x k-1 + 1 n ( z k i k -z k-1 i k ) .
Interestingly, since dk is a lower-bound of f we have dk (x k ) ≤ f * . Thus, the quantity f (x k )dk (x k ) can be used as an optimality certificate that upper-bounds f (x k )f * . Furthermore, this certificate was shown to converge to zero with a rate similar to SAG/SDCA/SVRG/SAGA under the condition n ≥ 2L/µ [START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF]:

E[f (x k ) -dk (x k )] ≤ C(1 -τ MISO ) k (f (x 0 ) -d0 (x 0 )).
with τ MISO = 1/(3n) (also refined in 1/(2n) in [Defazio et al., 2014b]). However, we need to keep in memory the list of n centers z k i in order to iteratively update the x k . This may be expensive if the dimension of the feature is large. It is worth remarking that in many machine learning problems, each function f i (x) has the specific form f i (x) = l i (⟨x, w i ⟩) + µ 2 ∥x∥ 2 . In such cases, the vectors z k i can be obtained by storing only O(n) scalars, this remark is still valid for other incremental methods like SAG/SAGA/SDCA.

In the next section, we extend MISO to the composite optimization problem and remove the big data condition n ≥ 2L/µ.

The proximal MISO algorithm

We now consider the composite optimization problem below,

min x∈R p { f (x) = 1 n n ∑ i=1 f i (x) + ψ(x) } ,
where the functions f i are differentiable with L-Lipschitz derivatives and µ-strongly convex. As in typical composite optimization problems, ψ is convex but not necessarily differentiable. We assume that the proximal operator of ψ can be computed easily.

CHAPTER 2. PROXIMAL MINIMIZATION BY INCREMENTAL SURROGATE OPTIMIZATION (MISO)

Taking the similar approach as the original MISO algorithm, we construct lower bounds of f , by adding the regularization penalty ψ,

x k = arg min x∈R p { D k (x) = 1 n n ∑ i=1 d k i (x) + ψ(x) } .
(2.3)

In order to remove the big data condition n ≥ 2L/µ, we introduce a "shrinking" parameter δ in (0, 1). Let us denote l k i as the latest lower bound with respect to the component f i ,

l k i (x) = f i (x k-1 )+⟨∇f i (x k-1 ), x -x k-1 ⟩+ µ 2 ∥x -x k-1 ∥ 2 .
Then we perform the update of the surrogates d k i by following

d k i (x) = { (1 -δ)d k-1 i (x)+ δl k i (x) if i = i k , d k-1 i (x)
otherwise.

(2.4)

This immediately implies the following algorithm.

Algorithm 1 Prox-MISO: MISO algorithm with proximal support.

input (z 0 i ) i=1,...,n such that (A1) holds; N (number of iterations);

1: initialize z0 = 1 n ∑ n i=1 z 0 i and x 0 = prox ψ/µ [z 0 ]; 2: define δ = min(1, µn 2(L-µ) ); 3: for k = 1, . . . , N do 4: randomly pick up an index i k in [1, n] and update z k i = { (1 -δ)z k-1 i + δ ( x k-1 -1 µ ∇f i (x k-1 ) ) if i = i k z k-1 i otherwise zk = zk-1 + 1 n ( z k i k -z k-1 i k ) = 1 n n ∑ i=1 z k i x k = prox ψ/µ [z k ].
(2.5)

5: end for output x N (final estimate).

To illustrate the difference from the algorithm presented in the last section, we consider a pathological case when the sample size n = 1. If we apply the original MISO algorithm, it gives the gradient descent method with a large stepsize 1/µ, which has no guarantee of convergence. If we apply the new update, we take a less aggressive step with the stepsize shrunk by δ, which reduces the variance of the update. This is the key for removing the big data condition. Then, we remark that under the large sample size condition n ≥ 2L/µ and ψ = 0, we have δ = 1 and the update of the quantities z k i in (2.5) is identical as the original MISO algorithm. More generally, the choice of the parameter δ is driven from the convergence analysis that we will discuss later.

Relationship with the variance reduction approch. When the problem is smooth, i.e. ψ = 0, the Prox-MISO algorithm boils down to the following update:

x k = x k-1 + δ n ( x k-1 - 1 µ ∇f i k (x k-1 ) -z k-1 i k ) ,
where i k is the index randomly selected at k-th iteration. Remarking that

E i k [ x k-1 - 1 µ ∇f i k (x k-1 ) -z k-1 i k ] = - 1 µ ∇f (x k-1 ),
the update of MISO is thus an unbiased gradient update. It enjoys a similar variance reduction interpretation as the primal incremental methods discussed in Section 1.5. ). It is however not clear that Prox-MISO actually performs dual ascent steps in the sense of SDCA since the proof of convergence of SDCA cannot be directly modified to use the stepsize of proximal MISO and furthermore, the convergence proof of Prox-MISO does not use the concept of duality. Another difference lies in the optimality certificate of the algorithms. Whereas Proximal-SDCA provides a certificate in terms of linear convergence of a duality gap based on Fenchel conjugate, Proximal-MISO ensures linear convergence of a gap that relies on strong convexity but not on the Fenchel conjugate (at least explicitly).

Relationship with

Relationship with Dual Averaging methods. The Prox-MISO algorithm is closely related to the dual averaging methods when the function f i enjoys a particular form f i (x) = fi (x) + µ 2 ∥x∥ 2 . In this case, the update of z k i is given by,

z k i = (1 -δ)z k-1 i - δ µ ∇ fi (x k-1 ).
Thus, the quantities z k i are simply non uniform average of the past gradients which plays a similar rule as the dual averaging term in the update of RDA algorithm [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF].

Initialization of surrogates.

We now take a particular focus on the initialization of the surrogates that we have omitted to discuss so far. The algorithm needs to be initialized with some quadratic lower bounds for the functions f i :

f i (x) ⩾ d 0 i (x) ≜ µ 2 ∥x -z 0 i ∥ 2 + c 0 i , (A1)
which are guaranteed to exist due to the µ-strong convexity of f i . For typical machine learning applications, such initialization is easy. For example, logistic regression with ℓ 2 -regularization

CHAPTER 2. PROXIMAL MINIMIZATION BY INCREMENTAL SURROGATE OPTIMIZATION (MISO)

satisfies (A1) with z 0 i = 0 and c 0 i = 0. Then, the Prox-MISO scheme is given in Algorithm 1. Note that if no simple initialization is available, we may consider any initial estimate z0 in R p and define z 0 i = z0 -(1/µ)∇f i (z 0 ), which requires performing one pass over the data.

Optimality certificate and stopping criterion. Similar to the original MISO algorithm, Proximal MISO maintains a list (d k i ) of lower bounds of the functions f i . As a consequence, the following function is a lower bound of the objective f :

D k (x) = 1 n n ∑ i=1 d k i (x) + ψ(x), (2.6)
and the update (2.5) can be shown to exactly minimize D k . As a lower bound of f , we have that

D k (x k ) ≤ f * and thus f (x k ) -f * ≤ f (x k ) -D k (x k ).
The quantity f (x k ) -D k (x k ) can then be interpreted as an optimality gap, and the analysis below will show that it converges linearly to zero.

Convergence analysis. We show that the certificate f (x k ) -D k (x k ) enjoys a linear convergence rate:

Theorem 6 (Convergence of Prox-MISO). Let (x k ) k≥0 be obtained by Prox-MISO, then

E[f (x k ) -D k (x k )] ⩽ 1 τ (1 -τ ) k (f * -D 0 (x 0 )) with τ ⩾ min { µ 4L , 1 2n } .
Furthermore, the convergence of f (x k )f * can be directly derived from it,

E[f (x k )] -f * ⩽ 1 τ (1 -τ ) k+1 (f (x 0 ) -D 0 (x 0 )) .
(2.7)

Before we prove this theorem, we note that this rate is slightly better than the one proven in MISO [START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF], which converges as (1 -1 3n ) k . The main idea of the proof is to show that the lower bound D k (x k ) converges to f * in a linear rate, acting as a dual method. In order to build connections between

D k (x k ) and D k-1 (x k-1 ), we introduce an intermediate term D k (x k-1 ) and control respectively the difference D k (x k-1 ) -D k (x k ) and D k (x k-1 ) -D k-1 (x k-1 ).
We start by recalling a classical lemma that provides useful inequalities. Its proof may be found in [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF].

Lemma 1 (Classical Quadratic Upper and Lower Bounds). For any function h : R p → R which is µ-strongly convex and differentiable with L-Lipschitz derivatives, we have for all

x, y in R p , µ 2 ∥x -y∥ 2 ≤ h(x) -h(y) + ⟨∇h(y), x -y⟩ ≤ L 2 ∥x -y∥ 2 .

THE PROXIMAL MISO ALGORITHM

Then, we proceed with a sequence of technical lemmas

Lemma 2 (Lower Bound on D k -D k-1 ). For all k ≥ 1 and x in R p , D k (x) ⩾ D k-1 (x) - δ(L -µ) 2n ∥x -x k-1 ∥ 2 , ∀x ∈ R p .
(2.8)

Proof. For all i ∈ [1, n], f i satisfies the assumptions of Lemma 1. Therefore, for all k ≥ 0 and i = i k , the following inequality holds for all x in R p ,

d k i (x) = (1 -δ)d k-1 i (x) + δ[f i (x k-1 ) + ⟨∇f i (x k-1 ), x -x k-1 ⟩ + µ 2 ∥x -x k-1 ∥ 2 ] ⩾ (1 -δ)d k-1 i (x) + δf i (x) - δ(L -µ) 2 ∥x -x k-1 ∥ 2 ⩾ d k-1 i (x) - δ(L -µ) 2 ∥x -x k-1 ∥ 2 ,
where the first inequality uses Lemma 1, and the last one uses the inequality

f i ≥ d k-1 i
. From this inequality, we can obtain (2.8) by simply using

D k (x) = D k-1 (x)+ 1 n ( d k i k (x) -d k-1 i k (x)
)

.

Lemma 3 (Relation between D k and D k-1 ). For all k ≥ 0, for all x and y in R p ,

D k (x) -D k (y) = D k-1 (x) -D k-1 (y) -µ⟨z k -zk-1 , x -y⟩.
(2.9)

Proof. This is straightforward from the fact that D k is the sum of ψ and a quadratic function centered at zk , namely

D k (x) = C k + µ 2 ∥x -zk ∥ 2 + ψ(x).
with some constant C k ∈ R.

Lemma 4 (Lower Bound on D k (x k-1 )). For all k ≥ 1, we have

D k (x k-1 ) ≥ D k-1 (x k-1 ) + nµ 2 2δ(L -µ) ∥z k -zk-1 ∥ 2 .
(2.10)

Proof. Take x = x k-1 and y = x k-1 + nµ δ(L-µ) (z k -zk-1 ) in (2.9) gives, D k (x k-1 ) -D k-1 (x k-1 ) = D k (y) -D k-1 (y) + µ⟨z k -zk-1 , y -x k-1 ⟩ by (2.8) ⩾ - δ(L -µ) 2n ∥y -x k-1 ∥ 2 + µ⟨z k -zk-1 , y -x k-1 ⟩ = nµ 2 2δ(L -µ) ∥z k -zk-1 ∥ 2 . Lemma 5 (Upper Bound on D k (x k-1 )). For any k ⩾ 1, D k (x k-1 ) ≤ D k (x k ) + µ 2 ∥z k -zk-1 ∥ 2 . (2.11) Proof. Apply Lemma 3 with x = x k-1 and y = x k yields D k (x k-1 ) -D k (x k ) = D k-1 (x k-1 ) -D k-1 (x k ) -µ⟨z k -zk-1 , x k-1 -x k ⟩. Since x k-1 is the minimum of D k-1 which is µ-strongly convex. Thus, D k-1 (x k-1 ) + µ 2 ∥x k -x k-1 ∥ 2 ⩽ D k-1 (x k ).
As a consequence,

D k (x k-1 ) -D k (x k ) ⩽ - µ 2 ∥x k -x k-1 ∥ 2 -µ⟨z k -zk-1 , x k-1 -x k ⟩ ⩽ µ 2 ∥z k -zk-1 ∥ 2 .
We are now in shape to prove the main convergence result.

Proof of Theorem 6. From Lemma 4 and Lemma 5, we have

D k (x k-1 ) -D k-1 (x k-1 ) ≥ nµ 2 2δ(L -µ) ∥z k -zk-1 ∥ 2 ≥ nµ δ(L -µ) [D k (x k-1 ) -D k (x k )] .
(2.12)

Recall that by construction,

D k (x k-1 ) = D k-1 (x k-1 ) + δ n (f i k (x k-1 ) -d k-1 i k (x k-1 )).
Take expectation on i k gives,

E[D k (x k-1 )] = ( 1 - δ n ) E[D k-1 (x k-1 )] + δ n E[f (x k-1 )].
(2.13) Apply (2.13) to (2.12) and rearrange the expression gives

τ E[f (x k-1 )] -E[D k (x k )] ≤ -(1 -τ )E[D k-1 (x k-1 )].
where

τ = ( 1 - δ(L -µ) nµ ) δ n , (2.14)

IMPLEMENTATION DETAILS AND CONCLUSIONS

Interpolating with f * yields

τ (E[f (x k-1 )] -f * ) + (f * -E[D k (x k )]) ⩽ (1 -τ ) (f * -E[D k-1 (x k-1 )]) . (2.15) On one hand, since f (x k-1 ) ⩾ f * , we have f * -E[D k (x k )] ≤ (1 -τ ) (f * -E[D k-1 (x k-1 )]) ≤ • • • ≤ (1 -τ ) k (f * -D 0 (x 0 )) . (2.16)
On the other hand, since

f * ⩾ D k (x k ), then τ (E[f (x k-1 )] -f * ) ⩽ (1 -τ ) (f * -E[D k-1 (x k-1 )]) ⩽ (1 -τ ) k (f * -D 0 (x 0 )) .
(2.17)

This gives (2.7). Moreover,

E[f (x k ) -D k (x k )] = E[f (x k ) -f * ] + E[f * -D k (x k )] ≤ 1 τ (1 -τ ) k+1 (f * -D 0 (x 0 )) + (1 -τ ) k (f * -D 0 (x 0 )) (From 2.17 and 2.16) = 1 τ (1 -τ ) k (f * -D 0 (x 0 )).
This gives (6). We conclude the proof by appropriately choosing the parameter δ which determines τ . Given the expression of the linear convergence parameter τ in (2.14), we choose δ to maximize τ , yielding

δ = min { 1, nµ 2(L -µ) } .
By distinguishing the two cases, we have

1. when nµ 2(L-µ) ⩽ 1, we have δ = nµ 2(L-µ) and τ = µ 4(L-µ) . 2. when 1 ⩽ nµ 2(L-µ) , we have δ = 1 and τ = 1 n -L-µ n 2 µ ⩾ 1 2n .
Therefore, τ ≥ min

( 1 2n , µ 4(L-µ)
)

, which concludes the theorem.

Implementation details and conclusions

In practice, the Prox-MISO algorithm provides a convenient stopping criterion, using the duality

gap f (x k ) -D k (x k ).
In order to keep track of the value D k (x k ) efficiently, we introduce an additional list of scalars (C i ) i∈ [1,n] to be kept in memory. Intuitively, each C i represents the contribution of d k i to the value D k (x k ) and only one of them is updated at each iteration. More precisely, we define C i by rewriting the quadratic surrogate d k i as

d k i (x) = c k i + µ 2 ∥x -z k i ∥ 2 = C k i -µ⟨x, z k i ⟩ + µ 2 ∥x∥ 2 , (2.18)
where

C k i = c k i + µ 2 ∥z k i ∥ 2 . When the incremental surrogate d i is updated, the corresponding C k i
is also updated following the rule

C k i = (1 -δ)C k-1 i + δ ( f i (x k-1 ) -⟨∇f i (x k-1 ), x k-1 ⟩ + µ 2 ∥x k-1 ∥ 2 ) .
This yields the practical implementation Algorithm 2.

Algorithm 2 Prox-MISO with stopping criterion.

input (z 0 i , c 0 i ) i=1,...,n such that (A1) holds; ε (target accuracy); 1: initialize z0 = 1 n ∑ n i=1 z 0 i and x 0 = prox ψ/µ [z 0 ], set C 0 i = c 0 i + µ 2 ∥z 0 ∥ 2 for all i in [1, n] ; 2: Define δ = min ( 1, µn 2(L-µ)
) and k = 0;

3: while 1 n ∑ n i=1 (f i (x k ) -C k i ) + µ⟨x k , zk ⟩ -µ 2 ∥x k ∥ 2 > ε do 4: for l = 1, . . . , n do 5: k ← k + 1; 6:
randomly pick up an index i k in [1, n] and perform the update (2.5);

7:

in the same time, update

C k i = { (1 -δ)C k-1 i + δ ( f i (x k-1 ) -⟨∇f i (x k-1 ), x k-1 ⟩ + µ 2 ∥x k-1 ∥ 2 ) if i = i k C k-1 i otherwise .
(2.19)

8:
end for 9: end while output x N (final estimate such that f (x N )f * ≤ ε).

To explain the stopping criterion in Algorithm 2, we remark that the entire surrogate D k is the sum of d k i and ψ, leading to the expression

D k (x) = ( 1 n n ∑ i=1 C k i ) -µ⟨x, zk ⟩ + µ 2 ∥x∥ 2 + ψ(x), Thus, f (x k ) -D k (x k ) = ( 1 n n ∑ i=1 f i (x k ) -C k i ) + µ⟨x k , zk ⟩ - µ 2 ∥x k ∥ 2 ,
when the problem is strongly convex and the resulting linear rate is similar to other variance reduction based incremental algorithms. Extensions to non-uniform sampling strategy and to stochastic setting are developed later by [START_REF] Bietti | Stochastic optimization with variance reduction for infinite datasets with finite-sum structure[END_REF]. However, we remark that our algorithm is restricted to strongly convex problems and the obtained convergence rate is not optimal. The generic acceleration scheme Catalyst that we are presenting in the next chapter will solve both of these issues: a) Catalyst provides a way to apply Prox-MISO or other incremental algorithms to non strongly convex problems; b) The obtained convergence rate of applying Catalyst is optimal up to a logarithmic constant, in both strongly convex and non-strongly convex settings. The practical performance of Prox-MISO algorithm is similar to other incremental methods. Thus, we leave the experimental evaluation to the next chapter, at the same time as the evaluation of the acceleration scheme.

Chapter 3

Catalyst acceleration

Chapter abstract:

We introduce a generic scheme for accelerating gradient-based optimization methods in the sense of Nesterov. Our approach, called Catalyst, builds upon the inexact accelerated proximal point algorithm for minimizing a convex objective function, and consists of approximately solving a sequence of well-chosen auxiliary problems, leading to faster convergence. One of the key to achieve acceleration in theory and in practice is to solve these sub-problems with appropriate accuracy by using the right stopping criterion and warm start strategy. In this chapter, we discuss these practical issues and also provide a global complexity analysis. We show that our approach applies to a large class of algorithms, including gradient descent, block coordinate descent, incremental algorithms such as SAG, SAGA, SDCA, SVRG, Finito/MISO, and their proximal variants. For all of these methods, we provide acceleration and explicit support for non-strongly convex objectives. We conclude with extensive experiments showing that acceleration is useful in practice, especially for ill-conditioned problems.

The material of this chapter is based on the following paper in preparation:

H. Lin, J. Mairal, and Z. Harchaoui. Catalyst Acceleration for Gradient-Based Optimization: from Theory to Practice. 2017.

This paper extends the original conference publication:

H. Lin, J. Mairal, and Z. Harchaoui. A universal catalyst for first-order optimization. In

Advances in Neural Information Processing Systems (NIPS), 2015

The code for reproducing the figures in this chapter is publicly available at: https://github.com/hongzhoulin89/Catalyst-QNing

Introduction

A large number of machine learning and signal processing problems are formulated as the minimization of a convex composite objective function f : R p → R:

min x∈R p { f (x) ≜ f 0 (x) + ψ(x) } , (3.1)
where f 0 is convex and L-smooth1 , and ψ is convex but may not be differentiable. In statistics or machine learning, the variable x may represent model parameters, and the role of f 0 is to ensure that the estimated parameters fit some observed data. Specifically, f 0 is often a large sum of functions and (3.1) is a regularized empirical risk

min x∈R p { f (x) ≜ 1 n n ∑ i=1 f i (x) + ψ(x) } . (3.2)
Each term f i (x) measures the fit between x and a data point indexed by i, whereas the function ψ acts as a regularizer; it is typically chosen to be the squared ℓ 2 -norm, which is smooth, or to be a non-differentiable penalty such as the ℓ 1 -norm or another sparsity-inducing norm [START_REF] Bach | Optimization with sparsity-inducing penalties[END_REF]. Composite minimization also encompasses constrained minimization when considering extended-valued indicator functions ψ that may take the value +∞ outside of a convex set and 0 inside [see Hiriart-Urruty and Lemaréchal, 1996].

In this chapter, we provide a generic framework that is able to accelerate gradient-based or first-order methods, with a particular focus on large sums of functions. By "accelerating", we mean generalizing a mechanism invented by [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF] that improves the convergence rate of the gradient descent algorithm. More precisely, when ψ = 0, gradient descent steps produce iterates

(x k ) k≥0 such that f (x k ) -f * ≤ ε in O(1/ε) iterations,
where f * denotes the minimum value of f . Furthermore, when the objective f is µ-strongly convex, the previous iteration-complexity becomes O((L/µ) log(1/ε)), which is proportional to the condition number L/µ. However, these rates were shown to be suboptimal for the class of first-order methods, and a simple strategy of taking the gradient step at a well-chosen point different from x k yields the optimal complexity-O(1/ √ ε) for the convex case and O( √ L/µ log(1/ε)) for the µ-strongly convex one [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF]. Later, this acceleration technique was extended to deal with non-differentiable penalties ψ for which the proximal operator defined below is easy to compute [Beck andTeboulle, 2009, Nesterov, 2013].

prox ψ (x) ≜ arg min z∈R p { ψ(z) + 1 2 ∥x -z∥ 2 } , (3.3)
where ∥.∥ denotes the Euclidean norm.

For machine learning problems involving a large sum of n functions, a recent effort has been devoted to developing fast incremental algorithms such as SAG [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF], SAGA 3.1. INTRODUCTION [Defazio et al., 2014a], SDCA [START_REF] Shalev-Shwartz | Proximal stochastic dual coordinate ascent[END_REF], SVRG [START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF]Zhang, 2013, Xiao and[START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF], or Finito/MISO [Defazio et al., 2014b[START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF], which can exploit the particular structure (3.2). Unlike full gradient approaches, which require computing and averaging n gradients (1/n) ∑ n i=1 ∇f i (x) at every iteration, incremental techniques have a cost per-iteration that is independent of n. The price to pay is the need to store a moderate amount of information regarding past iterates, but the benefits may be significant in terms of computational complexity. More precisely, in order to achieve an ε-accurate solution for a µ-strongly convex objective, the number of gradient evaluations required by these methods is given by O

(( n + L µ ) log( 1 ε )
)

, where L is either the maximum Lipschitz constant across the gradients ∇f i , or the average value, depending on the algorithm variant considered. Unless there is a big mismatch between L and L (global Lipschitz constant for the sum of gradients), incremental approaches significantly outperform the full gradient method whose complexity in terms of gradient evaluations is given by O

( n L µ log( 1 ε )
)

. Yet, these incremental approaches do not use Nesterov's extrapolation steps and whether or not they could be accelerated was an important open question when these methods were introduced. It was indeed only known to be the case for SDCA [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF] for strongly convex objectives.

In this chapter, we propose a generic acceleration scheme that provides a universal positive answer to the previous open question. By analogy with substances that increase chemical reaction rates, we call our approach "Catalyst". Given an optimization method M as input, Catalyst outputs an accelerated version of it, possibly the same algorithm if the method M is already optimal. The sole requirement on the method in order to achieve acceleration is that it should have linear convergence rate for strongly convex problems. This is the case for full gradient methods [Beck andTeboulle, 2009, Nesterov, 2013] and block coordinate descent methods [Nesterov, 2012b, Richtárik and[START_REF] Richtárik | Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function[END_REF], which already have well-known accelerated variants. More importantly, it also applies to the previously-mentioned incremental methods, whose complexity become

Õ (( n + √ n L/µ ) log( 1 ε ) ) after Catalyst acceleration,
where Õ hides some logarithmic dependencies on the condition number L/µ. This improves upon the non-accelerated variants, when n is smaller than the condition number. Besides, acceleration occurs regardless of the strong convexity of the objective-that is, with µ = 0-which brings us to our second achievement. Some approaches such as Finito/MISO, SDCA, or SVRG are only defined for strongly convex objectives. A classical trick to apply them to general convex functions is to add a small regularization ε∥x∥ 2 to the objective [START_REF] Shalev-Shwartz | Proximal stochastic dual coordinate ascent[END_REF]. The drawback of this strategy is that it requires choosing in advance the parameter ε, which is related to the target accuracy. A consequence of our work is to automatically provide a direct support for non-strongly convex objectives, thus removing the need of selecting ε beforehand. Moreover, an accelerated rate is directly obtained from the resulting algorithm.

A short version of this chapter has been published at the NIPS conference in 2015 [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]; in addition to simpler convergence proofs and more extensive numerical evaluation, we extend the conference paper with two important new contributions:

Moreau-Yosida smoothing interpretation. We provide a new interpretation of Catalyst as an accelerated gradient descent method with inexact gradients, applied to the Moreau-Yosida envelope of the objective function. While the link between the proximal point algorithm and the Moreau-Yosida smoothing is well known [see [START_REF] Themelis | Forward-backward envelope for the sum of two nonconvex functions: Further properties and nonmonotone line-search algorithms[END_REF], the fact that smoothing may be useful to accelerate existing algorithms, as in Catalyst, is less intuitive and opens new perspectives. In particular, it was a key observation to our subsequent work to Catalyst, called QuickeNing [START_REF] Lin | A generic quasi-newton algorithm for faster gradient-based optimization[END_REF], which relies on limited memory Quasi-Newton principles instead of Nesterov's acceleration. Besides, the point of view of Moreau-Yosida smoothing also led to new practical strategies, which we now present.

More effective stopping criteria and warm start strategies. Catalyst is a two-loop algorithm, which requires solving sub-problems in an inner-loop with enough accuracy. In this chapter, we introduce new parameter-free stopping criteria for solving these sub-problems that are more practical than those we originally presented in [START_REF] Lin | A universal catalyst for first-order optimization[END_REF], and new warm start strategies with significantly better empirical performance.

The paper is structured as follows: We complete this introductory section with some related work in Section 3.1.1, and give a short description of the two-loop Catalyst algorithm in Section 3.1.2. Then, Section 3.2 introduces the Moreau-Yosida regularization and its inexact variant. In Section 3.3, we introduce formally the main algorithm, and its convergence analysis is presented in Section 3.4. Section 3.6 is devoted to numerical experiments and Section 3.7 concludes the chapter.

Related Work

Catalyst can be interpreted as a variant of the proximal point algorithm [START_REF] Rockafellar | Monotone operators and the proximal point algorithm[END_REF][START_REF] Güler | On the convergence of the proximal point algorithm for convex minimization[END_REF], which is a central concept in convex optimization, underlying augmented Lagrangian approaches, and composite minimization schemes [Bertsekas, 2015, Parikh and[START_REF] Parikh | Proximal algorithms[END_REF]. The proximal point algorithm consists of solving (3.1) by minimizing a sequence of auxiliary problems involving a quadratic regularization term. In general, these auxiliary problems cannot be solved with perfect accuracy, and several notations of inexactness were proposed by [START_REF] Güler | New proximal point algorithms for convex minimization[END_REF], [START_REF] He | An accelerated inexact proximal point algorithm for convex minimization[END_REF] and [START_REF] Salzo | Inexact and accelerated proximal point algorithms[END_REF]. The Catalyst approach hinges upon (i) an acceleration technique for the proximal point algorithm originally introduced in the pioneer work of [START_REF] Güler | New proximal point algorithms for convex minimization[END_REF]; (ii) a more practical inexactness criterion than those proposed in the past.2 As a result, we are able to control the rate of convergence for approximately solving the auxiliary problems with an optimization method M. In turn, we are also able to obtain the computational complexity of the global procedure for solving (3.1), which was not possible with

INTRODUCTION

previous analysis [START_REF] Güler | New proximal point algorithms for convex minimization[END_REF][START_REF] He | An accelerated inexact proximal point algorithm for convex minimization[END_REF][START_REF] Salzo | Inexact and accelerated proximal point algorithms[END_REF]. When instantiated in different first-order optimization settings, our analysis yields systematic acceleration.

Beyond [START_REF] Güler | New proximal point algorithms for convex minimization[END_REF], several papers have inspired our work. In particular, accelerated SDCA [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF] is an instance of an inexact accelerated proximal point algorithm, even though this was not explicitly stated in the original paper. Catalyst can be seen as a generalization of their algorithm, originally designed for stochastic dual coordinate ascent approaches. Their proof of convergence relies on different tools than ours. Specifically, we introduce an approximate sufficient descent condition, which, when satisfied, grants acceleration to any optimization method, whereas the direct proof of Shalev-Shwartz and [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF], in the context of SDCA, does not extend to non-strongly convex objectives. Another useful methodological contribution was the convergence analysis of inexact proximal gradient methods of Schmidt et al. [2011b] and [START_REF] Devolder | First-order methods of smooth convex optimization with inexact oracle[END_REF]. Finally, similar ideas appear in the independent work [START_REF] Frostig | Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization[END_REF]. Their results overlap in part with ours, but both papers adopt different directions. Our analysis is for instance more general and provides support for non-strongly convex objectives.

Then, beyond accelerated SDCA [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF], a lot of other works have been recently developed towards obtaining optimal incremental methods, such as APCG [START_REF] Lin | An accelerated proximal coordinate gradient method[END_REF], SDPC [Zhang and Xiao, 2015b], RPDG [START_REF] Lan | An optimal randomized incremental gradient method[END_REF], Point-SAGA [START_REF] Defazio | A simple practical accelerated method for finite sums[END_REF] and Katyusha [Allen-Zhu, 2016]. We remark that their techniques are algorithmspecific and cannot be directly generalized into a unified scheme. However, we should mention that the complexity obtained by applying Catalyst acceleration to incremental methods only matches the optimal bound up to a logarithmic factor, which may be the price to pay for providing a generic framework.

Finally, another recent line of work has also combined smoothing techniques with outerloop algorithms such as Quasi-Newton methods [Themelis et al., 2016, Giselsson and[START_REF] Giselsson | Nonsmooth minimization using smooth envelope functions[END_REF], even though their purpose is not to accelerate existing techniques, but rather to derive new algorithms for nonsmooth optimization.

Overview of Catalyst

Before introducing Catalyst precisely in Section 3.3, we provide here a first overview of the algorithm and its main ideas. Catalyst is a generic approach that wraps an algorithm M into an accelerated one A, which can achieve the same accuracy as M with reduced computational complexity. The resulting method A is an inner-outer loop construct, presented in Algorithm 3, where in the inner loop the method M is called to solve an auxiliary stronglyconvex optimization problem, and where in the outer loop the sequence of iterates produced by M are extrapolated for faster convergence. There are therefore three main ingredients in Catalyst: a) a smoothing technique that produces strongly-convex sub-problems; b) an extrapolation technique to accelerate the convergence; c) a balancing principle to optimally tune the inner and outer computations.

Algorithm 3 Catalyst -Overview input initial estimate x 0 in R p , smoothing parameter κ, optimization method M.

1: Initialize y 0 = x 0 . 2: while the desired accuracy is not achieved do 3:

Find x k using M x k ≈ arg min x∈R p { h k (x) ≜ f (x) + κ 2 ∥x -y k-1 ∥ 2 } . (3.4) 4:
Compute y k using an extrapolation step, with β k in (0, 1)

y k = x k + β k (x k -x k-1
).

5: end while output x k (final estimate).

Smoothing by infimal convolution

Catalyst can be used on any algorithm M that enjoys a linear-convergence guarantee when minimizing strongly-convex objectives. However the objective at hand may be poorly conditioned or even might not be strongly convex.

In Catalyst, we use M to approximately minimize an auxiliary objective h k at iteration k, defined in (3.4), which is strongly convex and has better conditioning than f . The classical notion of Moreau envelope allows to build, using smoothing by infimal convolution with a quadratic, a well-conditioned convex function F from any poorly-conditioned convex function f (see Section 3.3 for a refresher on Moreau envelopes). We shall show in Section 3.3 that a notion of approximate Moreau envelope allows to define precisely the first-order information collected when approximately minimizing the auxiliary objective.

Extrapolation by Nesterov acceleration Catalyst uses an extrapolation scheme "à la Nesterov" to build a sequence (y k ) k≥0 updated as

y k = x k + β k (x k -x k-1 ) ,
where (β k ) k≥0 is a positive decreasing sequence, which we define in Section 3.3.

We shall show in Section 3.4 that we can get faster rates of convergence thanks to this extrapolation step when the smoothing parameter κ, the inner-loop stopping criterion, and the sequence (β k ) k≥0 are carefully built.

Balancing inner and outer complexities

The optimal balance between inner loop and outer loop complexity derives from the complexity bounds established in Section 3.4. Given a target accuracy and a condition number estimate of f , our bounds dictate a choice of κ that gives the optimal setting for the inner-loop stopping criterion and all technical quantities

Without Catalyst

With Catalyst

µ > 0 µ = 0 µ > 0 µ = 0 FG O ( n L f µ log ( 1 ε )) O ( n L f ε ) Õ( n √ L f µ log ( 1 ε ) ) Õ( n √ L f ε ) SAG/SAGA O (( n + L µ ) log ( 1 ε )) O ( n L ε ) Õ(( n + √ n L µ ) log ( 1 ε ) ) Õ( √ n L ε ) MISO not avail. SDCA SVRG Acc-FG O ( n √ L f µ log ( 1 ε ) ) O ( n L f √ ε ) no acceleration Acc-SDCA Õ(( n + √ n L µ ) log ( 1 ε
) )

not avail.

Table 3.1 -Comparison of rates of convergence, before and after the Catalyst acceleration, in the strongly-convex and non strongly-convex cases, respectively. The notation Õ hides logarithmic factors. The constant L f is the global Lipschitz constant of the gradient with respect to the entire objective function, while L is the average Lipschitz constants of the gradients ∇f i , or the maximum value, depending on the algorithm's variants considered.

involved in the algorithm. We shall demonstrate in particular the power of an appropriate warm start strategy to achieve near-optimal complexity.

Overview of the complexity results

Finally, we provide in Table 3.1 a brief overview of the complexity results obtained from the Catalyst acceleration, when applied to various optimization methods M for minimizing a large finite sum of n functions. Note that the complexity results obtained with Catalyst are optimal, up to some logarithmic factors [see [START_REF] Agarwal | A lower bound for the optimization of finite sums[END_REF][START_REF] Arjevani | Dimension-free iteration complexity of finite sum optimization problems[END_REF][START_REF] Woodworth | Tight complexity bounds for optimizing composite objectives[END_REF].

The Moreau-Yosida envelope and its inexact variant

We now briefly recall the definition of the Moreau-Yosida envelope [START_REF] Moreau | Fonctions convexes duales et points proximaux dans un espace hilbertien[END_REF][START_REF] Yosida | Functional analysis[END_REF]], which we have already introduced in Section 1.6 and which plays a key role for understanding the Catalyst acceleration. More precisely, the Moreau-Yosida envelope results from the infimal convolution of f with a quadratic penalty:

F (x) ≜ min z∈R p { f (z) + κ 2 ∥z -x∥ 2 } , (3.5)
where κ is the positive regularization parameter. Note that the proximal operator is then defined as the unique minimizer of the problem-that is,

p(x) ≜ prox f /κ (x) = arg min z∈R p { f (z) + κ 2 ∥z -x∥ 2 } .
Unlike the proximal operator of ψ defined in (3.3), p(x) does not admit a closed form in general. Therefore, computing it requires to solve a sub-problem to high accuracy with some iterative algorithm.

In Section 1.6, we discussed basic properties of the Moreau-Yosida envelope. In particular, minimizing F and f is equivalent in the sense that both optimal function values and solution sets coincide. F is always L F = κ-smooth and its gradient is given by

∇F (x) = κ(x -p(x)). (3.6)
Finally, if f is µ-strongly convex, then F is µ F -strongly convex with µ F = µκ µ+κ . Note that F can then be arbitrarily well conditioned when f is µ-strongly convex, by choosing a small value of κ.

A Fresh Look at Catalyst

We have also seen that first-order methods applied to F provide us several known algorithms; gradient descent on F yields the proximal point algorithm [START_REF] Martinet | Régularisation d'inéquations variationnelles par approximations successives[END_REF][START_REF] Rockafellar | Monotone operators and the proximal point algorithm[END_REF]] and Nesterov's accelerated method yields the the accelerated proximal point algorithm of [START_REF] Güler | New proximal point algorithms for convex minimization[END_REF]. While these algorithms are conceptually elegant, each update requires to evaluate the gradient of F , and thus to compute proximal operator p(x) exactly. Unless a closed form is available, which is almost never the case, an iterative algorithm is required, which yields the two-loop scheme (see Algorithm 3). Catalyst can then be interpreted as an accelerated gradient descent method that computes inexact sub-problems using an optimization method M:

x k+1 ≈ p(y k ) and y k+1 = x k+1 + β k+1 (x k+1 -x k ),
where β k+1 is Nesterov's extrapolation parameter [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]. When the minimization is exact,

x k+1 = p(y k ) = y k -1 L F ∇F (y k )
, and we recover a gradient step computed at the extrapolated point y k . The main challenge that will be addressed in Section 3.3 is how to control the complexity of the inner-loop minimization.

The Inexact Moreau-Yosida Envelope

Since Catalyst uses the Moreau-Yoside envelope with inexact gradient, it is natural to introduce inexactness criteria. We start with the most natural one.

Inexactness through absolute accuracy. The following definition introduces a set of approximate proximal operators, which we will use in Catalyst. Given a proximal center x, a smoothing parameter κ, and an accuracy ε > 0, we denote the set of ε-approximations of the proximal operator p(x) by

p ε (x) ≜ {z ∈ R p s.t. h(z) -h * ≤ ε} where h(z) = f (z) + κ 2 ∥x -z∥ 2 , (C1)
and h * is the minimum function value of h. Checking the condition z in p ε (x) may typically be done by computing a duality gap, or by using another certificate of the type

h(z) -h * ≤ h(z)-d(z)
, where the lower-bound d(z) ≤ h * is obtained from duality such as in SDCA [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF]. In other methods such as MISO [START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF], d(z) may be obtained from a quadratic approximation of the objective function and may have a simple closed form.

Note that a criterion based on the subdifferential may also be used as in the extension of Catalyst recently developed for non-convex optimization by [START_REF] Paquette | Catalyst acceleration for gradient-based non-convex optimization[END_REF]. Here, we focus on the convex case, where other simple criterions may be appropriate, as shown in the next lemma.

Lemma 6 (Checking the absolute accuracy criterion). Consider a proximal center x, a smoothing parameter κ and an accuracy ε > 0. Consider an objective with composite form (3.1), and the function h defined in (C1). Then, define

[z] η = prox ηψ (z -η∇h 0 (z)) , (3.7)
where h 0 is the smooth part of h, namely ∇h 0 (z) = ∇f 0 (z) + κ(zx) and η = 1 κ+L . Then,

∥z -[z] η ∥ ≤ η √ 2κε implies that [z] η ∈ p ε (x).
The proof is provided in Appendix B.2[see also the link with criterions based on subdifferential in [START_REF] Paquette | Catalyst acceleration for gradient-based non-convex optimization[END_REF]. Then, with an approximate proximal operator in hand, we may now obtain an approximate gradient of the Moreau-Yosida envelope, defined as

g(z) ≜ κ(x -z), (3.8)
for any z in p ε (x), according to the exact gradient formula ∇F (x) = κ(xp(x)). As a consequence, we may immediately draw a link

z ∈ p ε (x) =⇒ ∥z -p(x)∥ ≤ √ 2ε κ ⇐⇒ ∥g(z) -∇F (x)∥ ≤ √ 2κε, (3.9)
where the upper bound on ∥zp(x)∥ is a consequence of the strong convexity of h and the fact that p(x) is its exact minimizer.

Relative error criterion.

We have seen that z in p ε (x) yields an upper-bound of the form ∥g(z)-∇F (x)∥ ≤ ε ′ . Another criterion that arise naturally in the convergence analysis provides instead an upper bound of the form ∥g(z) -∇F (x)∥ ≤ δ ′ ∥∇F (x)∥ for some δ ′ > 0-that is, providing a relative error in terms of gradient approximation. Given a proximal center x, a smoothing parameter κ and a relative accuracy δ in [0, 1), we denote the set of δ-relative approximations by

g δ (x) ≜ { z ∈ R p s.t. h(z) -h * ≤ δκ 2 ∥x -z∥ 2 } , (C2)
Criterion (C2) may be interpreted as (C1) with an adaptive ε that depends on the point z.

Therefore, given a fixed z,

checking if z is in g δ (x) is equivalent to checking if z is in p ε (x) with ε = δκ 2 ∥x -z∥ 2 .
Then, by following similar steps as in (3.9), notice that z in g δ (x) implies

∥z -p(x)∥ ≤ √ δ∥x -z∥ ≤ √ δ(∥x -p(x)∥ + ∥p(x) -z∥),
and by rearranging, and using closed forms of g(z) and ∇F (x),

z ∈ g δ (x) =⇒ ∥g(z) -∇F (x)∥ ≤ δ ′ ∥∇F (x)∥ with δ ′ = √ δ 1 - √ δ ,
which provides a relative gradient approximation. Finally, note that Lemma 6 may be used as well to obtain a point [z] η in g δ (x) after taking an extra gradient and prox steps and setting the value ε = δκ 2 ∥x -z∥.

A few remarks on related works. Inexactness criteria with respect to subgradient norms have been investigated in the past, starting from the pioneer work of [START_REF] Rockafellar | Monotone operators and the proximal point algorithm[END_REF] in the context of the inexact proximal point algorithm. Later, different works have been dedicated to more practical inexactness criteria3 [START_REF] Auslender | Numerical methods for nondifferentiable convex optimization[END_REF][START_REF] Correa | Convergence of some algorithms for convex minimization[END_REF][START_REF] Solodov | A unified framework for some inexact proximal point algorithms[END_REF][START_REF] Fuentes | Descentwise inexact proximal algorithms for smooth optimization[END_REF]. Here, we provide a more intuitive point of view using the Moreau-Yosida envelope.

While the proximal point algorithm has caught a lot of attention, very few works have focused on its accelerated variant. The first accelerated proximal point algorithm with inexact gradients was proposed by [START_REF] Güler | New proximal point algorithms for convex minimization[END_REF]. Then, [START_REF] Salzo | Inexact and accelerated proximal point algorithms[END_REF] proposed a more rigorous convergence analysis, and more inexactness criteria, which are typically stronger than ours. In the same way, a more general inexact oracle framework has been proposed later by [START_REF] Devolder | First-order methods of smooth convex optimization with inexact oracle[END_REF]. To achieve the Catalyst acceleration, our main effort was to propose and analyze criteria that allow us to control the complexity for finding approximate solutions of the subproblems.

Catalyst acceleration

In this section, we introduce the Catalyst algorithm, and discuss its main features.

Main Algorithm

Catalyst is presented in Algorithm 4. As discussed in Section 3.2, this scheme can be interpreted as an inexact accelerated proximal point algorithm, or equivalently as an accelerated gradient descent method applied to the Moreau-Yosida envelope of the objective with inexact gradients. Since an overview has already been presented in Section 3.1.2, we now present important details to obtain acceleration in theory and in practice.

Algorithm 4 Catalyst input Initial estimate x 0 in R p , smoothing parameter κ, strong convexity parameter µ, optimization method M and a stopping criterion based on a sequence of accuracies

(ε k ) k≥0 , or (δ k ) k≥0 , or a fixed budget T . 1: Initialize y 0 = x 0 , q = µ µ+κ . If µ > 0, set α 0 = √ q, otherwise α 0 = 1.
2: while the desired accuracy is not achieved do 3:

Compute an approximate solution of the following problem with M

x k ≈ arg min x∈R p { h k (x) ≜ f (x) + κ 2 ∥x -y k-1 ∥ 2 } ,
using the warm start strategy of Section 3.3.1 and one of the following stopping criteria:

(a) absolute accuracy: find x k in p ε k (y k-1 ) by using criterion (C1);

(b) relative accuracy: find x k in g δ k (y k-1 ) by using criterion (C2);

(c) f ixed budget: run M for T iterations and output x k .

4:

Update α k in (0, 1) by solving the equation

α 2 k = (1 -α k )α 2 k-1 + qα k (3.10) 5:
Compute y k with Nesterov's extrapolation step .11) 6: end while output x k (final estimate).

y k = x k + β k (x k -x k-1 ) with β k = α k-1 (1 -α k-1 ) α 2 k-1 + α k . ( 3
Requirement: linear convergence of the method M. One of the main characteristic of Catalyst is to apply the method M to strongly-convex sub-problems, without requiring strong convexity of the objective f . As a consequence, Catalyst provides direct support for convex but non-strongly objectives to M, which may be useful to extend the scope of application of techniques that need strong convexity to operate. Yet, Catalyst requires to solve these subproblems efficiently enough in order to control the complexity of the inner-loop computations. More precisely, when applying M to minimize a strongly-convex function h, we assume that M is able to produce a sequence of iterates (z t ) t≥0 such that

h(z t ) -h * ≤ C M (1 -τ M ) t (h(z 0 ) -h * ), (3.12)
where z 0 is the initial point given to M, and τ M in (0, 1), C M > 0 are two constants. In such a case, we say that M admits a linear convergence rate. The quantity τ M controls the speed of convergence for solving the sub-problem: the larger is τ M , the faster is convergence. For a given algorithm M, the quantity τ M depends usually on the condition number of h. For instance, for the proximal gradient method and many first-order algorithms, we simply have

τ M = O((µ + κ)/(L + κ))
, as h is (µ + κ)-strongly convex and (L + κ)-smooth. Catalyst can also be applied to randomized methods M that satisfy (3.12) in expectation:

E[h(z t ) -h * ] ≤ C M (1 -τ M ) t (h(z 0 ) -h * ), (3.13)
Then, the complexity results of Section 3.4 also hold in expectation. This allows us to apply Catalyst to randomized block coordinate descent algorithms [see [START_REF] Richtárik | Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function[END_REF], and references therein], and some incremental algorithms such as SAG, SAGA, or SVRG. For other methods that admit a linear convergence rates in terms of duality gap, such as SDCA, MISO/Finito, Catalyst can also be applied as explained in Section 3.5.

Stopping criteria.

Catalyst may be used with three types of stopping criteria for solving the inner-loop problems. We now detail them below.

(a) absolute accuracy: we use a sequence (ε k ) k≥0 of accuracies, and stop the method M by using criterion (C1). Typically, (C1) requires to compute a duality gap to guarantee that h k (z t )h * k ≤ ε k . Our analysis suggests the following choice for (ε k ) k≥0 :

-if f is µ-strongly convex, use α 0 = √ q and ε k = 2 9 (1 -ρ) k (f (x 0 ) -f * ) with ρ < √ q.
if f is convex but not strongly convex, use α 0 = 1.

ε k = 2(f (x 0 ) -f * ) 9(k + 2) 4+γ with γ > 0.
Typically, γ = 0.1 and ρ = 0.9 √ q are reasonable choices, both in theory and in practice.

Of course, the quantity f (x 0 )f * is unknown, but it may be replaced safely by an upper bound such as f (x 0 ) if f is non-negative or by a duality gap.

(b) relative accuracy: To use criterion (C2), our analysis suggests the following choice for the sequence (δ k ) k≥0 :

-if f is µ-strongly convex, δ k = √ q 2 - √ q .
if f is convex but not strongly convex,

δ k = 1 (k + 1) 2 ,
and in both cases, the value of α 0 is the same as for criterion (C1).

With no a priori knowledge required about the objective function f , these relative accuracy criteria (b) are sometimes simpler to use than criteria (a).

(c) fixed budget: Finally, the simplest way of using Catalyst is to fix in advance the number T of iterations of the method M for solving the sub-problems without checking any optimality criterion. Whereas our analysis provides theoretical budgets that are compatible with this strategy, we found them to be pessimistic and impractical. Instead, we propose an aggressive strategy for incremental methods that simply consists of setting T = n. This setting was called the "one-pass" strategy in the original Catalyst paper [START_REF] Lin | A universal catalyst for first-order optimization[END_REF].

Warm-starts in inner loops. Besides linear convergence rate, an adequate warm start strategy needs to be used to guarantee that the sub-problems will be solved in reasonable computational time. Specifically, the following choices arise from the convergence analysis that will be detailed in Section 3.4. Consider a sub-problem consisting of minimizing the function

h k (z) ≜ f (z) + κ 2 ∥z -y k-1 ∥ 2 .
Then, M will produce a sequence of iterates (z t ) t≥0 with initial point z 0 , which enjoys linear convergence (3.12) or (3.13). To achieve the Catalyst acceleration, the choice of z 0 is critical.

Here, we propose the following rules, which are both compatible with the theory, and more effective in practice than the one introduced in the original Catalyst paper [START_REF] Lin | A universal catalyst for first-order optimization[END_REF].

(a) when using criterion (C1) to find

x k in p ε k (y k-1 ), choose -if f is smooth (ψ = 0), then choose z 0 = x k-1 + κ κ+µ (y k-1 -y k-2 ). -if f is composite as in (3.1), then define w 0 = x k-1 + κ κ+µ (y k-1 -y k-2 ) and z 0 = prox ηψ (w 0 -ηg) with g = ∇f 0 (w 0 ) + κ(w 0 -y k-1 ) and η = 1 L + κ . (3.14)
At the cost of an extra function evaluation, we also noticed that the warm start strategy can be improved by choosing the starting point among x k-1 and (3.14) that yields the smallest function value for h.

(b) when using criteria (C2) to find x k in g δ k (y k-1 ),

-if f is smooth (ψ = 0), then choose z 0 = y k-1 .
if f is composite as in (3.1), then choose

z 0 = prox ηψ (y k-1 -η∇f 0 (y k-1 )) with η = 1 L + κ .
(c) when using a fixed budget T , choose the same warm start strategy as in (a).

Note that the original paper about Catalyst [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] only considers criterion (C1) with the warm start rule z 0 = x k-1 , which does not perform as well as the ones proposed here.

Optimal balance: choice of parameter κ. Finally, the last ingredient is to find an optimal balance between the inner-loop (for solving each sub-problem) and outer-loop computations. To do so, we minimize our global complexity bounds with respect to the value of κ. As studied in the experimental section, this strategy turns out to be reasonable in practice. Then, as shown in the theoretical section, the resulting rule of thumb is

We select κ by maximizing the ratio τ M / √ µ + κ.

We recall that τ M characterizes how fast M solves the sub-problems, according to (3.12); typically, τ M depends on the condition number L+κ µ+κ and is a function of κ.4 In Table 3.2, we illustrate the choice of κ for different methods. Note that the resulting rule for incremental methods is very simple for the pracitioner: select κ such that the condition number L+κ µ+κ is of the order of n; then, the inner-complexity becomes O(n log(1/ε)).

Method M Inner-complexity τ M Choice for κ FG O ( n L+κ µ+κ log ( 1 ε )) ∝ µ+κ L+κ L -2µ SAG/SAGA/SVRG O (( n + L+κ µ+κ ) log ( 1 ε )) ∝ µ+κ n(µ+κ)+ L+κ L-µ n+1 -µ Table 3.2 -Example
of choices of the parameter κ for the full gradient (FG) and incremental methods SAG/SAGA/SVRG. See Table 3.1 for details about the complexity of the methods.

Convergence and complexity analysis

We now present the complexity analysis of Catalyst. In Section 3.4.1, we analyze the convergence rate of the outer loop, regardless of the complexity for solving the sub-problems. Then, we analyze the complexity of the inner-loop computations for our various stopping criteria and warm start strategies in Section 3.4.2. Section 3.4.3 combines the outer-and inner-loop analysis to provide the global complexity of Catalyst applied to a given optimization method M.

Complexity Analysis for the Outer-Loop

The complexity analysis of the first variant of Catalyst we presented in [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] used a tool called "estimate sequence", which was introduced by [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]. Here, we provide a simpler proof. We start with criterion (C1), before extending the result to (C2).

Analysis for Criterion (C1)

The next theorem describes how the errors (ε k ) k≥0 accumulate in Catalyst.

Theorem 7 (Convergence of outer-loop for criterion (C1)). Consider the sequences (x k ) k≥0 and (y k ) k≥0 produced by Algorithm 4, assuming that

x k is in p ε k (y k-1 ) for all k ≥ 1, Then, f (x k ) -f * ≤ A k-1   √ (1 -α 0 )(f (x 0 ) -f * ) + γ 0 2 ∥x * -x 0 ∥ 2 + 3 k ∑ j=1 √ ε j A j-1   2 ,
where

γ 0 = (κ + µ)α 0 (α 0 -q) and A k = k ∏ j=1 (1 -α j ). (3.15)
Before we prove this theorem, we note that by setting ε k = 0 for all k, the speed of convergence of f (x k )f * is driven by the sequence (A k ) k≥0 . Lemma 2.2.4 of [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF], which we recall below, provides us this speed.

Lemma 7 (Lemma 2.2.4 of Nesterov 2004). Consider the quantities γ 0 , A k defined in (3.15) and the α k 's defined in Algorithm 4. Then, if γ 0 ≥ µ,

A k ≤ min      (1 - √ q) k , 4 ( 2 + k √ γ 0 κ ) 2     
.

For non-strongly convex objectives, A k follows the classical accelerated O(1/k 2 ) rate of convergence, whereas it achieves a linear convergence rate for the strongly convex case. Note that the quantity q = µ µ+κ is simply the inverse square root of the condition number of the Moreau-Yosida regularization F of the objective f . We now provide the proof of the theorem below.

Proof. We start by defining an approximate sufficient descent condition inspired by a remark of [START_REF] Chambolle | A remark on accelerated block coordinate descent for computing the proximity operators of a sum of convex functions[END_REF] regarding accelerated gradient descent methods. A related condition was also used by [START_REF] Paquette | Catalyst acceleration for gradient-based non-convex optimization[END_REF] in the context of non-convex optimization.

Approximate sufficient descent condition. Let us define the function h

k (x) = f (x) + κ 2 ∥x -y k ∥ 2 . Since p(y k )
is the unique minimizer of h k , we may start with a strong convexity inequality: for any k ≥ 1, for all x in R d and θ k ≥ 0,

h k (x) ≥ h * k + κ + µ 2 ∥x -p(y k-1 )∥ 2 ≥ h * k + κ + µ 2 (1 -θ k ) ∥x -x k ∥ 2 - κ + µ 2 ( 1 θ k -1 ) ∥x k -p(y k-1 )∥ 2 ≥ h k (x k ) -ε k + κ + µ 2 (1 -θ k ) ∥x -x k ∥ 2 - κ + µ 2 ( 1 θ k -1 ) ∥x k -p(y k-1 )∥ 2 ,
where the (µ + κ)-strong convexity of h k is used in the first inequality; Lemma 16 is used in the second inequality, and the last one uses the relation

h k (x k ) -h * k ≤ ε k . Moreover, when θ k ≥ 1, the last term is non-negative and h k (x) ≥ h k (x k ) -ε k + κ + µ 2 (1 -θ k ) ∥x -x k ∥ 2 . If instead θ k ≤ 1, the coefficient 1 θ k -1 is non-negative and we have - κ + µ 2 ( 1 θ k -1 ) ∥x k -p(y k-1 )∥ 2 ≥ - ( 1 θ k -1 ) (h k (x k ) -h * k ) ≥ - ( 1 θ k -1 ) ε k .
In this case, we have

h k (x) ≥ h k (x k ) - ε k θ k + κ + µ 2 (1 -θ k ) ∥x -x k ∥ 2 .
As a result, we have for all value of θ k ≥ 0,

h k (x) ≥ h k (x k ) + κ + µ 2 (1 -θ k ) ∥x -x k ∥ 2 - ε k min{1, θ k } .
After using the definition of h k , we then obtain the approximate descent condition

f (x k ) + κ 2 ∥x k -y k-1 ∥ 2 + κ + µ 2 (1 -θ k ) ∥x -x k ∥ 2 ≤ f (x) + κ 2 ∥x -y k-1 ∥ 2 + ε k min{1, θ k } . (3.16)
Definition of the Lyapounov function. we introduce a sequence (S k ) k≥0 that will act as a Lyapounov function, with

S k = (1 -α k )(f (x k ) -f * ) + α k κη k 2 ∥x * -v k ∥ 2 . (3.17)
where x * is a minimizer of f , (v k ) k≥0 is a sequence defined by v 0 = x 0 and

v k = x k + 1 -α k-1 α k-1 (x k -x k-1 ) for k ≥ 1,
and (η k ) k≥0 is an auxiliary quantity defined by

η k = α k -q 1 -q .
The way we introduce these variables allow us to write the following relationship,

y k = η k v k + (1 -η k )x k , for all k ≥ 0,
which is obtained after simple calculation. Then, we have the following relations for all k ≥ 1 and

z k = α k-1 x * + (1 -α k-1 )x k-1 . f (z k ) ≤ α k-1 f * + (1 -α k-1 )f (x k-1 ) - µα k-1 (1 -α k-1 ) 2 ∥x * -x k-1 ∥ 2 ; z k -x k = α k-1 (x * -v k ),
and also the following one

∥z k -y k-1 ∥ 2 = ∥(α k-1 -η k-1 )(x * -x k-1 ) + η k-1 (x * -v k-1 )∥ 2 = α 2 k-1 ( 1 - η k-1 α k-1 ) (x * -x k-1 ) + η k-1 α k-1 (x * -v k-1 ) 2 ≤ α 2 k-1 ( 1 - η k-1 α k-1 ) ∥x * -x k-1 ∥ 2 + α 2 k-1 η k-1 α k-1 ∥x * -v k-1 ∥ 2 = α k-1 (α k-1 -η k-1 )∥x * -x k-1 ∥ 2 + α k-1 η k-1 ∥x * -v k-1 ∥ 2 ,
where we used the convexity of the norm and the fact that η k ≤ α k . Using the previous relations in (3.16) with

x = z k = α k-1 x * + (1 -α k-1 )x k-1 , gives for all k ≥ 1, f (x k ) + κ 2 ∥x k -y k-1 ∥ 2 + κ + µ 2 (1 -θ k ) α 2 k-1 ∥x * -v k ∥ 2 ≤ α k-1 f * + (1 -α k-1 )f (x k-1 ) - µ 2 α k-1 (1 -α k-1 )∥x * -x k-1 ∥ 2 + κα k-1 (α k-1 -η k-1 ) 2 ∥x * -x k-1 ∥ 2 + κα k-1 η k-1 2 ∥x * -v k-1 ∥ 2 + ε k min{1, θ k } .
Remark that for all k ≥ 1,

α k-1 -η k-1 = α k-1 - α k-1 -q 1 -q = q(1 -α k-1 ) 1 -q = µ κ (1 -α k-1 ),
and the quadratic terms involving x *x k-1 cancel each other. Then, after noticing that for all k ≥ 1,

η k α k = α 2 k -qα k 1 -q = (κ + µ)(1 -α k )α 2 k-1 κ ,
which allows us to write

f (x k ) -f * + κ + µ 2 α 2 k-1 ∥x * -v k ∥ 2 = S k 1 -α k . (3.18)
We are left, for all k ≥ 1, with

1 1 -α k S k ≤ S k-1 + ε k min{1, θ k } - κ 2 ∥x k -y k-1 ∥ 2 + (κ + µ)α 2 k-1 θ k 2 ∥x * -v k ∥ 2 . (3.19)

Control of the approximation errors for criterion (C1).

Using the fact that

1 min{1, θ k } ≤ 1 + 1 θ k , we immediately derive from equation (3.19) that 1 1 -α k S k ≤ S k-1 + ε k + ε k θ k - κ 2 ∥x k -y k-1 ∥ 2 + (κ + µ)α 2 k-1 θ k 2 ∥x * -v k ∥ 2 . (3.20)
By minimizing the right-hand side of (3.20) with respect to θ k , we obtain the following inequality

1 1 -α k S k ≤ S k-1 + ε k + √ 2ε k (µ + κ)α k-1 ∥x * -v k ∥, 62

CONVERGENCE AND COMPLEXITY ANALYSIS

and after unrolling the recursion,

S k A k ≤ S 0 + k ∑ j=1 ε j A j-1 + k ∑ j=1 √ 2ε j (µ + κ)α j ∥x * -v j ∥ A j-1 .
From Equation (3.18), the lefthand side is larger than

(µ+κ)α 2 k-1 ∥x * -v k ∥ 2 2A k-1
. We may now define

u j = √ (µ+κ)α j-1 ∥x * -v j ∥ √ 2A j and a j = 2 √ ε j √ A j
, and we have

u 2 k ≤ S 0 + k ∑ j=1 ε j A j-1 + k ∑ j=1 a j u j for all k ≥ 1.
This allows us to use Lemma 17, which yields

S k A k ≤    S 0 + k ∑ j=1 ε j A j-1 + 2 k ∑ j=1 √ ε j A j-1    2 , ≤   √ S 0 + 3 k ∑ j=1 √ ε j A j-1   2 which provides us the desired result given that f (x k ) -f * ≤ S k 1-α k and that v 0 = x 0 .
We are now in shape to state the convergence rate of the Catalyst algorithm with criterion (C1), without taking into account yet the cost of solving the sub-problems. The next two propositions specialize Theorem 7 to the strongly convex case and convex, but not strongly convex cases, respectively. Their proofs are provided in Appendix B.2.

Proposition 2 (µ-strongly convex case, criterion (C1)).

In Algorithm 4, choose α 0 = √ q and

ε k = 2 9 (f (x 0 ) -f * )(1 -ρ) k with ρ < √ q.
Then, the sequence of iterates

(x k ) k≥0 satisfies f (x k ) -f * ⩽ 8 ( √ q -ρ) 2 (1 -ρ) k+1 (f (x 0 ) -f * ).

Proposition 3 (Convex case, criterion (C1)).

When µ = 0, choose α 0 = 1 and

ε k = 2(f (x 0 ) -f * ) 9(k + 1) 4+γ with γ > 0.
Then, Algorithm 4 generates iterates (x k ) k≥0 such that

f (x k ) -f * ⩽ 8 (k + 1) 2 ( κ 2 ∥x 0 -x * ∥ 2 + 4 γ 2 (f (x 0 ) -f * )
)

.

Analysis for Criterion (C2)

Then, we may now analyze the convergence of Catalyst under criterion (C2), which offers similar guarantees as (C1), as far as the outer loop is concerned.

Theorem 8 (Convergence of outer-loop for criterion (C2)). Consider the sequences (x k ) k≥0 and (y k ) k≥0 produced by Algorithm 4, assuming that x k is in g δ k (y k-1 ) for all k ≥ 1 and δ k in (0, 1). Then,

f (x k ) -f * ≤ A k-1 ∏ k j=1 (1 -δ j ) ( (1 -α 0 )(f (x 0 ) -f * ) + γ 0 2 ∥x 0 -x * ∥ 2 ) ,
where γ 0 and (A k ) k≥0 are defined in (3.15) in Theorem 7.

Proof.

x k in g δ k (y k-1 ) is equivalent to x k in p ε k (y k-1 ) with an adaptive ε k = δ k κ 2 ∥x k -y k-1 ∥ 2 .
All steps of the proof of Theorem 7 hold for such values of ε k and from (3.19), we may deduce

S k 1 -α k - (κ + µ)α 2 k-1 θ k 2 ∥x * -v k ∥ 2 ≤ S k-1 + ( δ k κ 2 min{1, θ k } - κ 2 ) ∥x k -y k-1 ∥ 2 .
Then, by choosing θ k = δ k < 1, the quadratic term on the right disappears and the left-hand side is greater than

1-δ k 1-α k S k . Thus, S k ≤ 1 -α k 1 -δ k S k-1 ≤ A k ∏ k j=1 (1 -δ j ) S 0 , which is sufficient to conclude since (1 -α k )(f (x k ) -f * ) ≤ S k .
The next propositions specialize Theorem 8 for specific choices of sequence (δ k ) k≥0 in the strongly and non strongly convex cases.

Proposition 4 (µ-strongly convex case, criterion (C2)). In Algorithm 4, choose α 0 = √ q and

δ k = √ q 2 - √ q .
Then, the sequence of iterates

(x k ) k≥0 satisfies f (x k ) -f * ⩽ 2 ( 1 - √ q 2 ) k (f (x 0 ) -f * ) .
Proof. This is a direct application of Theorem 8 by remarking that γ 0 = (1 -√ q)µ and

S 0 = (1 - √ q) ( f (x 0 ) -f * + µ 2 ∥x * -x 0 ∥ 2 ) ≤ 2(1 - √ q)(f (x 0 ) -f * ). 1 -α k 1 -δ k = 1 - √ q 2
Proposition 5 (Convex case, criterion (C2)).

When µ = 0, choose α 0 = 1 and

δ k = 1 (k + 1) 2 .
Then, Algorithm 4 generates iterates (x k ) k≥0 such that

f (x k ) -f * ⩽ 4κ∥x 0 -x * ∥ 2 (k + 1) 2 . (3.21)
Proof. This is a direct application of Theorem 8 by remarking that

γ 0 = κ, A k ≤ 4 (k+2) 2 (Lemma 18) and k ∏ i=1 ( 1 - 1 (i + 1) 2 ) = k ∏ i=1 i(i + 2) (i + 1) 2 = k + 2 2(k + 1) ≥ 1 2 .
Remark 1. In fact, the choice of δ k can be improved by taking δ k = 1 (k+1) 1+γ for any γ > 0, which comes at the price of a larger constant in (3.21).

Analysis of Warm Start Strategies for the Inner Loop

In this section, we study the complexity of solving the subproblems with the proposed warm start strategies. The only assumption we make on the optimization method M is that it enjoys linear convergence when solving a strongly convex problem-meaning, it satisfies either (3.12) or its randomized variant (3.13). Then, the following lemma gives us a relation between the accuracy required to solve the sub-problems and the corresponding complexity.

Lemma 8 (Accuracy vs. complexity). Let us consider a strongly convex objective h and a linearly convergent method M generating a sequence of iterates (z t ) t≥0 for minimizing h. Consider the complexity T (ε) = inf{t ≥ 0, h(z t )h * ≤ ε}, where ε > 0 is the target accuracy and h * is the minimum value of h. Then, 1. If M is deterministic and satisfies (3.12), we have

T (ε) ≤ 1 τ M log ( C M (h(z 0 ) -h * ) ε ) .
2. If M is randomized and satisfies (3.13), we have

E[T (ε)] ≤ 1 τ M log ( 2C M (h(z 0 ) -h * ) τ M ε ) + 1
The proof of the deterministic case is straightforward and the proof of the randomized case is provided in Appendix B.2.4. From the previous result, a good initialization is essential for fast convergence. More precisely, it suffices to control the initialization h(z 0 )-h * ε in order to bound the number of iterations T (ε). For that purpose, we analyze the quality of various warm start strategies.

Warm Start Strategies for Criterion (C1)

The next proposition characterizes the quality of initialization for (C1).

Proposition 6 (Warm start for criterion (C1)). Assume that M is linearly convergent for strongly convex problems with parameter τ M according to (3.12), or according to (3.13) in the randomized case. At iteration k + 1 of Algorithm 4, given the previous iterate x k in p ε k (y k-1 ), we consider the following function

h k+1 (z) = f (z) + κ 2 ∥z -y k ∥ 2 ,
which we minimize with M, producing a sequence (z t ) t≥0 . Then,

• when f is smooth, choose z 0 = x k + κ κ+µ (y k -y k-1 ); • when f = f 0 + ψ is composite, choose z 0 = prox ηψ (w 0 -η∇h 0 (w 0 )) with w 0 = x k + κ κ+µ (y k -y k-1 ), η = 1 L+κ and h 0 = f 0 + κ 2 ∥ • -y k ∥ 2 .
We also assume that we choose α 0 and (ε k ) k≥0 according to Proposition 2 for µ > 0, or Proposition 3 for µ = 0. Then,

1. if f is µ-strongly convex, h k+1 (z 0 ) -h * k+1 ≤ Cε k+1 where, C = L + κ κ + µ ( 2 1 -ρ + 2592(κ + µ) (1 -ρ) 2 ( √ q -ρ) 2 µ ) if f is smooth, (3.22) or C = L + κ κ + µ ( 2 1 -ρ + 23328(L + κ) (1 -ρ) 2 ( √ q -ρ) 2 µ ) if f is composite. (3.23)
2. if f is convex with bounded level sets, there exists a constant B > 0 that only depends on f, x 0 and κ such that

h k+1 (z 0 ) -h * k+1 ≤ B. (3.24)
Proof. We treat the smooth and composite cases separately.

Smooth and strongly-convex case. When f is smooth, by the gradient Lipschitz assumption,

h k+1 (z 0 ) -h * k+1 ≤ (L + κ) 2 ∥z 0 -p(y k )∥ 2 .
Moreover,

∥z 0 -p(y k )∥ 2 = ∥x k + κ κ + µ (y k -y k-1 ) -p(y k )∥ 2 = ∥x k -p(y k-1 ) + κ κ + µ (y k -y k-1 ) -(p(y k ) -p(y k-1 ))∥ 2 ≤ 2∥x k -p(y k-1 )∥ 2 + 2 κ κ + µ (y k -y k-1 ) -(p(y k ) -p(y k-1 )) 2 . Since x k is in p ε k (y k-1
), we may control the first quadratic term on the right by noting that

∥x k -p(y k-1 )∥ 2 ≤ 2 κ + µ (h k (x k ) -h * k ) ≤ 2ε k κ + µ .
Moreover, by the firmly non-expansiveness property of the proximal operator, it is possible to show that

κ κ + µ (y k -y k-1 ) -(p(y k ) -p(y k-1 )) 2 ≤ ∥y k -y k-1 ∥ 2 .
As a consequence,

h k+1 (z 0 ) -h * k+1 ≤ (L + κ) 2 ∥z 0 -p(y k )∥ 2 ≤ (L + κ)(∥x k -p(y k-1 )∥ 2 + ∥y k -y k-1 -(p(y k ) -p(y k-1 ))∥ 2 ), ≤ 2 L + κ µ + κ ε k + (L + κ)∥y k -y k-1 ∥ 2 , (3.25)
Then, we need to control the term ∥y ky k-1 ∥ 2 . Inspired by the proof of accelerated SDCA of Shalev-Shwartz and [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF],

∥y k -y k-1 ∥ = ∥x k + β k (x k -x k-1 ) -x k-1 -β k-1 (x k-1 -x k-2 )∥ ⩽ (1 + β k )∥x k -x k-1 ∥ + β k-1 ∥x k-1 -x k-2 ∥ ⩽ 3 max {∥x k -x k-1 ∥, ∥x k-1 -x k-2 ∥} ,
The last inequality was due to the fact that β k ≤ 1. In fact,

β 2 k = ( α k-1 -α 2 k-1 ) 2 ( α 2 k-1 + α k ) 2 = α 2 k-1 + α 4 k-1 -2α 3 k-1 α 2 k + 2α k α 2 k-1 + α 4 k-1 = α 2 k-1 + α 4 k-1 -2α 3 k-1 α 2 k-1 + α 4 k-1 + qα k + α k α 2 k-1 ≤ 1,
where the last equality uses the relation

α 2 k + α k α 2 k-1 = α 2 k-1 + qα k from (3.10). Then, ∥x k -x k-1 ∥ ⩽ ∥x k -x * ∥ + ∥x k-1 -x * ∥,
and by strong convexity of f

µ 2 ∥x k -x * ∥ 2 ⩽ f (x k ) -f * ⩽ 36 ( √ q -ρ) 2 ε k+1 ,
where the last inequality is obtained from Proposition 2. As a result,

∥y k -y k-1 ∥ 2 ⩽ 9 max { ∥x k -x k-1 ∥ 2 , ∥x k-1 -x k-2 ∥ 2 } ⩽ 36 max { ∥x k -x * ∥ 2 , ∥x k-1 -x * ∥ 2 , ∥x k-2 -x * ∥ 2 } ⩽ 2592 ε k-1 ( √ q -ρ) 2 µ .
Since

ε k+1 = (1 -ρ) 2 ε k-1
, we may now obtain (3.22) from (3.25) and the previous bound.

Smooth and convex case. When µ = 0, Eq. (3.25) is still valid but we need to control ∥y ky k-1 ∥ 2 in a different way. From Proposition 3, the sequence (f (x k )) k≥0 is bounded by a constant that only depends on f and x 0 ; therefore, by the bounded level set assumption, there exists R > 0 such that

∥x k -x * ∥ ≤ R, for all k ≥ 0.
Thus, following the same argument as the strongly convex case, we have

∥y k -y k-1 ∥ ≤ 36R 2 for all k ≥ 1,
and we obtain (3.24) by combining the previous inequality with (3.25).

Composite case. By using the notation of gradient mapping introduced in (3.7), we have z 0 = [w 0 ] η . By following similar steps as in the proof of Lemma 6, the gradient mapping satisfies the following relation

h k+1 (z 0 ) -h * k+1 ≤ 1 2(κ + µ) 1 η (w 0 -z 0 ) 2 ,
and we need to bound ∥w 0 -

z 0 ∥ = ∥w 0 -[w 0 ] η ∥.
For that, we introduce

[x k ] η = prox ηψ (x k -η(∇f 0 (x k ) + κ(x k -y k-1 ))).
Then,

∥w 0 -[w 0 ] η ∥ ≤ ∥w 0 -x k ∥ + ∥x k -[x k ] η ∥ + ∥[x k ] η -[w 0 ] η ∥, (3.26)

CONVERGENCE AND COMPLEXITY ANALYSIS

and we will bound each term on the right. By construction

∥w 0 -x k ∥ = κ κ + µ ∥y k -y k-1 ∥ ≤ ∥y k -y k-1 ∥.
Next, it is possible to show that the gradient mapping satisfies the following relation [see [START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF],

1 2η ∥x k -[x k ] η ∥ 2 ≤ h k (x k ) -h * k ≤ ε k .
And then since

[x k ] η = prox ηψ (x k -η(∇f 0 (x k ) + κ(x k -y k-1 ))) and [w 0 ] η = prox ηψ (w 0 - η(∇f 0 (w 0 ) + κ(w 0 -y k ))).
From the non expansiveness of the proximal operator, we have

∥[x k ] η -[w 0 ] η ∥ ≤ ∥x k -η(∇f 0 (x k ) + κ(x k -y k-1 )) -(w 0 -η(∇f 0 (w 0 ) + κ(w 0 -y k ))) ∥ ≤ ∥x k -η(∇f 0 (x k ) + κ(x k -y k-1 )) -(w 0 -η(∇f 0 (w 0 ) + κ(w 0 -y k-1 ))) ∥ + ηκ∥y k -y k-1 ∥ ≤ ∥x k -w 0 ∥ + ηκ∥y k -y k-1 ∥ ≤ 2∥y k -y k-1 ∥.
We have used the fact that ∥x -η∇h(x) -(y -η∇h(y))∥ ≤ ∥x -y∥. By combining the previous inequalities with (3.26), we finally have

∥w 0 -[w 0 ] η ∥ ≤ √ 2ηε k + 3∥y k -y k-1 ∥.
Thus, by using the fact that (a + b) 2 ≤ 2a 2 + 2b 2 for all a, b,

h k+1 (z 0 ) -h * k+1 ≤ L + κ κ + µ ( 2ε k + 9(L + κ)∥y k -y k-1 ∥ 2 ) ,
and we can obtain (3.23) and (3.24) by upper-bounding ∥y ky k-1 ∥ 2 in a similar way as in the smooth case, both when µ > 0 and µ = 0.

Finally, the complexity of the inner loop can be obtained directly by combining the previous proposition with Lemma 8.

Corollary 3 (Inner-loop Complexity for Criterion (C1)). Consider the setting of Proposition 6; then, the sequence (z t ) t≥0 minimizing h k+1 is such that the complexity

T k+1 = inf{t ≥ 0, h k+1 (z t ) -h * k+1 ≤ ε k+1 } satisfies T k+1 ≤ 1 τ M log (C M C) if µ > 0 =⇒ T k+1 = Õ ( 1 τ M ) ,
where C is the constant defined in (3.22) or in (3.23) for the composite case; and

T k+1 ≤ 1 τ M log ( 9C M (k + 2) 4+η B 2(f (x 0 ) -f * ) ) if µ = 0 =⇒ T k+1 = Õ ( log(k + 2) τ M ) ,
where B is the uniform upper bound in (3.24). Furthermore, when M is randomized, the expected complexity E[T k+1 ] is similar, up to a factor 2/τ M in the logarithm-see Lemma 8, and we have

E[T k+1 ] = Õ(1/τ M ) when µ > 0 and E[T k+1 ] = Õ(log(k + 2)/τ M ).
Here, Õ(.) hides logarithmic dependencies in parameters µ, L, κ, C M , τ M and f (x 0 )f * .

Warm Start Strategies for Criterion (C2)

We may now analyze the inner-loop complexity for criterion (C2) leading to upper bounds with smaller constants and simpler proofs. Note also that in the convex case, the bounded level set condition will not be needed, unlike for criterion (C1). To proceed, we start with a simple lemma that gives us a sufficient condition for (C2) to be satisfied.

Lemma 9 (Sufficient condition for criterion (C2)). If a point z satisfies

h k+1 (z) -h * ≤ δ k+1 κ 8 ∥p(y k ) -y k ∥ 2 , then z is in g δ k+1 (y k ).
Proof.

h k+1 (z) -h * k+1 ≤ δ k+1 κ 8 ∥p(y k ) -y k ∥ 2 ≤ δ k+1 κ 4 ( ∥p(y k ) -z∥ 2 + ∥z -y k ∥ 2 ) ≤ δ k+1 κ 4 ( 2 µ + κ (h k+1 (z) -h * k+1 ) + ∥z -y k ∥ 2 ) ≤ 1 2 ( h k+1 (z) -h * k+1 ) + δ k+1 κ 4 ∥z -y k ∥ 2 .
Rearranging the terms gives the desired result.

With the previous result, we can control the complexity of the inner-loop minimization with Lemma 8 by choosing ε = δ k+1 κ 8 ∥p(y k )y k ∥ 2 . However, to obtain a meaningful upper bound, we need to control the ratio

h k+1 (z 0 ) -h * k+1 ε = 8(h k+1 (z 0 ) -h * k+1 ) δ k+1 κ∥p(y k ) -y k ∥ 2 .
Proposition 7 (Warm start for criterion (C2)). Assume that M is linearly convergent for strongly convex problems with parameter τ M according to (3.12), or according to (3.13) in the randomized case. At iteration k + 1 of Algorithm 4, given the previous iterate x k in g δ k (y k-1 ), we consider the following function

h k+1 (z) = f (z) + κ 2 ∥z -y k ∥ 2 ,
which we minimize with M, producing a sequence (z t ) t≥0 . Then,

• when f is smooth, choose z 0 = y k ;

• when f = f 0 + ψ is composite, choose z 0 = prox ηψ (y k -η∇f 0 (y k )) with η = 1 L+κ .
Then,

h k+1 (z 0 ) -h * k+1 ≤ L + κ 2 ∥p(y k ) -y k ∥ 2 . (3.27) Proof. When f is smooth, the optimality conditions of p(y k ) yield ∇h k+1 (p(y k )) = ∇f (p(y k )) + κ(p(y k ) -y k ) = 0. As a result, h k+1 (z 0 ) -h * k+1 = f (y k ) - ( f (p(y k )) + κ 2 ∥p(y k ) -y k ∥ 2 ) ≤ f (p(y k )) + ⟨∇f (p(y k )), y k -p(y k )⟩ + L 2 ∥y k -p(y k )∥ 2 - ( f (p(y k )) + κ 2 ∥p(y k ) -y k ∥ 2 ) = L + κ 2 ∥p(y k ) -y k ∥ 2 .
When f is composite, we use the inequality in Lemma 2.3 of [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]: for any z,

h k+1 (z) -h k+1 (z 0 ) ≥ L + κ 2 ∥z 0 -y k ∥ 2 + (L + κ)⟨z 0 -y k , y k -z⟩,
Then, we apply this inequality with z = p(y k ), and thus,

h k+1 (z 0 ) -h * k+1 ≤ - L + κ 2 ∥z 0 -y k ∥ 2 -(L + κ)⟨z 0 -y k , y k -p(y k )⟩ ≤ L + κ 2 ∥p(y k ) -y k ∥ 2 .
We are now in shape to derive a complexity bound for criterion (C2), which is obtained by combining directly Lemma 8 with the value ε = δ k+1 κ 8 ∥p(y k )y k ∥ 2 , Lemma 9, and the previous proposition.

Corollary 4 (Inner-loop Complexity for Criterion (C2)). Consider the setting of Proposition 7 when M is deterministic; assume further that α 0 and (δ k ) k≥0 are chosen according to Proposition 4 for µ > 0, or Proposition 5 for µ = 0.

Then, the sequence (z t ) t≥0 is such that the complexity

T k+1 = inf{t ≥ 0, z t ∈ g δ k+1 (y k )} satisfies T k+1 ≤ 1 τ M log ( 4C M (L + κ) κ 2 - √ q √ q ) when µ > 0,
and

T k+1 ≤ 1 τ M log ( 4C M (L + κ) κ (k + 2) 2 ) when µ = 0.
When M is randomized, the expected complexity is similar, up to a factor 2/τ M in the logarithm-see Lemma 8, and we have

E[T k+1 ] = Õ(1/τ M ) when µ > 0 and E[T k+1 ] = Õ(log(k + 2)/τ M ).
The inner-loop complexity is asymptotically similar with criterion (C2) as with criterion (C1), but the constants are significantly better.

Global Complexity Analysis

In this section, we combine the previous outer-loop and inner-loop convergence results to derive a global complexity bound. We treat here the strongly convex (µ > 0) and convex (µ = 0) cases separately.

Strongly Convex Case

When the problem is strongly convex, we remark that the subproblems are solved in a constant number of iterations

T k = T = Õ ( 1 τ M )
for both criteria (C1) and (C2). This means that the iterate x k in Algorithm 4 is obtained after s = kT iterations of the method M. Thus, the true convergence rate of Catalyst applied to M is of the form

f s -f * = f ( x s T ) -f * ≤ C ′ (1 -ρ) s T (f (x 0 ) -f * ) ≤ C ′ ( 1 - ρ T ) s (f (x 0 ) -f * ), (3.28)
where f s = f (x k ) is the function value after s iterations of M and ρ = √ q/2 for criterion (C2)

and ρ may be chosen to be 0.9 √ q for criterion (C1). Then, choosing κ consists of maximizing the rate of convergence (3.28). In other words, we want to maximize √ q/T = Õ( √ qτ M ). Since q = µ µ+κ , this naturally lead to the maximization of τ M / √ µ + κ. As we will discuss later, this choice recovers classical accelerated rates of convergence. We now state more formally the global convergence result in terms of complexity.

Proposition 8 (Global Complexity for strongly convex objectives). When f is µstrongly convex and all parameters are chosen according to Propositions 2 and 6 when using criterion (C1), or Propositions 4 and 7 for (C2), then Algorithm 4 finds a solution x such that f (x)f * ≤ ε in at most N M iterations of a deterministic method M with 1. when criterion (C1) is used,

N M ≤ 1 τ M ρ log (C M C) • log ( 8(f (x 0 ) -f * ) ( √ q -ρ) 2 ε ) = Õ ( 1 τ M √ q log ( 1 ε ) )
,

where ρ = 0.9 √ q and C is the constant defined in (3.22) or (3.23) for the composite case;

2. when criterion (C2) is used,

N M ≤ 2 τ M √ q log ( 4C M L + κ κ 2 - √ q √ q ) • log ( 2(f (x 0 ) -f * ) ε ) = Õ ( 1 τ M √ q log ( 1 ε ) )
.

Note that similar results hold in terms of expected number of iterations when the method M is randomized (see the end of Proposition 6).

Proof. Let K be the number of iterations of the outer-loop algorithm required to obtain an ε-accurate solution. From Proposition 2, using (C1) criterion yields

K ≤ 1 ρ log ( 8(f (x 0 ) -f * ) ( √ q -ρ) 2 ε ) .
From Proposition 4, using (C2) criterion yields

K ≤ 2 √ q log ( 2(f (x 0 ) -f * ) ε ) .
Then since the number of runs of M is constant for any inner loop, the total number N M is given by KT where T is respectively given by Corollaries 3 and 4.

Convex, but not Strongly Convex Case

When µ = 0, the number of iterations for solving each subproblems grows logarithmically, which means that the iterate x k in Algorithm 4 is obtained after s ≤ kT log(k + 2) iterations of the method M, where T is a constant. By using the global iteration counter s = kT log(k + 2), we finally have

f s -f * ≤ C ′ log 2 (s) s 2 ( f (x 0 ) -f * + κ 2 ∥x 0 -x * ∥ 2 ) . (3.29)
This rate is near-optimal, up to a logarithmic factor, when compared to the optimal rate O(1/s 2 ). This may be the price to pay for using a generic acceleration scheme. As before, we formally detail the global complexity bound for convex in the next proposition.

Proposition 9 (Global complexity for convex objectives). When f is convex and all parameters are chosen according to Propositions 3 and 6 when using criterion (C1), or Propositions 5 and 7 for criterion (C2), then Algorithm 4 finds a solution x such that f (x)f * ≤ ε in at most N M iterations of a deterministic method M with 1. when criterion (C1) is applied

N M ≤ 1 τ M K log ( 9C M BK 4+γ 2(f (x 0 ) -f * ) ) = Õ ( 1 τ M √ κ ε log ( 1 ε ))
, where,

K ε = 8 ( κ 2 ∥x 0 -x * ∥ 2 + 4 γ 2 (f (x 0 ) -f * ) ) ε ;
2. when criterion (C2) is applied,

N M ≤ 1 τ M √ 4κ∥x 0 -x * ∥ 2 ε log ( 16C M (L + κ)∥x 0 -x * ∥ 2 ε ) = Õ ( 1 τ M √ κ ε log ( 1 ε ))
.

Note that similar results hold in terms of expected number of iterations when the method M is randomized (see the end of Proposition 7).

Proof. Let K denote the number of outer-loop iterations required to achieve an ε-accurate solution. From Proposition 3, when (C1) is applied, we have

K ≤ 8 ( κ 2 ∥x 0 -x * ∥ 2 + 4 γ 2 (f (x 0 ) -f * ) ) ε .
From Proposition 5, when (C2) is applied, we have

K ≤ √ 4κ∥x 0 -x * ∥ 2 ε .
Since the number of runs in the inner loop is increasing, we have

N M = K ∑ i=1 T i ≤ KT K .
Respectively apply T K obtained from Corollary 3 and Corollary 4 gives the result.

CONVERGENCE AND COMPLEXITY ANALYSIS

Theoretical foundations of the choice of κ. The parameter κ plays an important rule in the global complexity result. The linear convergence parameter τ M depends typically on κ since it controls the strong convexity parameter of the subproblems. The natural way to choose κ is to minimize the global complexity given by Proposition 8 and Proposition 9, which leads to the following rule

Choose κ to maximize τ M √ µ + κ ,
where µ = 0 when the problem is convex but not strongly convex. We now illustrate two examples when applying Catalyst to the classical gradient descent method and to the incremental approach SVRG.

Gradient descent. When M is the gradient descent method, we have

τ M = µ + κ L + κ .
Maximizing the ratio

τ M √ µ + κ gives κ = L -2µ, when L > 2µ.
Consequently, the complexity in terms of gradient evaluations for minimizing the finite sum 3.2, where each iteration of M cost n gradients, is given by

N M =      Õ ( n √ L µ log ( 1 ε
))

when µ > 0;

Õ ( n √ L ε log ( 1 ε
))

when µ = 0.

More precisely, when (C2) is applied

N M ≤        √ 2(L-µ) µ log ( 2(f (x 0 )-f * ) ε ) log ( 16 √ L-µ µ ) when µ > 0; 4 √ L∥x 0 -x * ∥ 2 ε log ( 32∥x 0 -x * ∥ 2 ε ) when µ = 0.
These rates are near-optimal up to logarithmic constants according to the first-order lower bound [START_REF] Nemirovskii | Problem complexity and method efficiency in optimization[END_REF][START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF].

SVRG.

For SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF] applied to the same finite-sum objective,

τ M = 1 600 ( n + L+κ µ+κ ) . 5
Thus, maximizing the corresponding ratio gives

κ = L -µ n + 1 -µ, when L > (n + 2)µ.
Consequently, the resulting global complexity, here in terms of expected number of gradient evaluations, is given by

E[N M ] =      Õ (√ n L µ log ( 1 ε
))

when µ > 0 and L > (n + 2)µ;

Õ (√ nL ε log ( 1 ε
))

when µ = 0.

Note that we treat here only the ill-conditioned case L > (n + 2)µ to simplify. More precisely, when (C2) is applied

E[N M ] ≤      4800 √ n L µ log ( 2(f (x 0 )-f * ) ε ) log ( 76800n √ n L µ )
when µ > 0 and L > (n + 2)µ; 9600

√ nL ε log ( 153600 nL∥x 0 -x * ∥ 2 ε )
when µ = 0.

We end up with large constant coefficients since we use a pessimistic convergence rate of SVRG algorithm, which is purely theoretical. This rate is near-optimal up to logarithmic factors according to the first-order lower bound [START_REF] Woodworth | Tight complexity bounds for optimizing composite objectives[END_REF]Srebro, 2016, Arjevani and[START_REF] Arjevani | Dimension-free iteration complexity of finite sum optimization problems[END_REF]. We remark that applying Catalyst to other incremental algorithms such as SAG/SAGA [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF], Defazio et al., 2014a] or dual-type algorithm Finito/MISO [Defazio et al., 2014b[START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF] or SDCA [START_REF] Shalev-Shwartz | Proximal stochastic dual coordinate ascent[END_REF]] yields similar convergence rates, which is different only in the constant. Thus, we successfully accelerate a large class of first-order algorithms, see Table 3.1 for a summary.

Practical Aspects of the Theoretical Analysis

So far, we have not discussed the fixed budget criterion mentioned in Section 3.3.1. The idea is quite natural and simple to implement: we predefine the number of iterations to run for solving each subproblems and stop worrying about the stopping condition. For example, when µ > 0 and M is deterministic, we can simply run T M iterations of M for each subproblem where T M is greater than the value given by Corollaries 3 or 4, then the criterions (C1) and (C2) are guaranteed to be satisfied. Unfortunately, the theoretical bound of T M is relatively poor and does not lead to a practical strategy. On the other hand, using a more aggressive strategy such as T M = n for incremental algorithms, meaning one pass over the data, seems to provide outstanding results, as shown in the experimental part of this chapter. Finally, one could argue that choosing κ according to a worst-case convergence analysis is not necessarily a good choice. In particular, the convergence rate of the method M, driven by the parameter τ M is probably often under estimated in the first place. This suggests that using a smaller value for κ than the one we have advocated earlier is a good thing. In practice, we have 3.5. CATALYST FOR MISO/FINITO/SDCA observed that indeed Catalyst is often robust to smaller values of κ than the theoretical one, but we have also observed that the theoretical value performs reasonably well (see experimental section).

Catalyst for MISO/Finito/SDCA

In this section, we present the application of Catalyst to MISO/Finito [START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF], Defazio et al., 2014b], which may be seen as a variant of SDCA [START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF]. The reason why these algorithms require a specific treatment is due to the fact that their linear convergence rates are given in a different form than (3.12); specifically, Theorem 4.1 of [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] tells us that MISO produces a sequence of iterates (z t ) t≥0 for minimizing the auxiliary objective h

(z) = f (z) + κ 2 ∥z -y∥ 2 such that E[h(z t )] -h * ≤ C M (1 -τ M ) t+1 (h * -d 0 (z 0 )),
where d 0 is a lower-bound of h defined as the sum of a simple quadratic function and the composite regularization ψ. More precisely, these algorithms produce a sequence (d t ) t≥0 of such lower-bounds, and the iterate z t is obtained by minimizing d t in closed form. In particular, z t is obtained from taking a proximal step at a well chosen point w t , providing the following expression,

z t = prox ψ/(κ+µ) (w t ).
Then, linear convergence is achieved for the duality gap

E[h(z t ) -h * ] ≤ E[h(z t ) -d t (z t )] ≤ C M (1 -τ M ) t (h * -d 0 (z 0 )).
Indeed, the quantity h(z t )d t (z t ) is a natural upper-bound on h(z t )h * , which is simple to compute, and which can be naturally used for checking the criterions (C1) and (C2). Consequently, the expected complexity of solving a given problem is slightly different compared to Lemma 8.

Lemma 10 (Accuracy vs. complexity). Let us consider a strongly convex objective h. Finito/MISO/SDCA generate a sequence of iterates (z t ) t≥0 . Consider the complexity T (ε) = inf{t ≥ 0, h(z t )d 0 (z 0 ) ≤ ε}, where ε > 0 is the target accuracy and h * is the minimum value of h. Then,

E[T (ε)] ≤ 1 τ M log ( 2C M (h * -d 0 (z 0 )) τ M ε ) + 1,
where d 0 is a lower bound of f constructed by the algorithm.

For the convergence analysis, the outer-loop complexity does not change as long as the algorithm finds approximate proximal points satisfying criterions (C1) and (C2). It is then sufficient to control the inner loop complexity. As we can see, we now need to bound the dual gap h *d 0 (z 0 ) instead of the primal gap h(z 0 )h * , leading to slightly different warm start strategies. Here, we show how to warm start MISO/Finito. Proposition 10 (Warm start for criterion (C1)). Consider applying Catalyst with the same parameter choices as in Proposition 6 to MISO/Finito. At iteration k + 1 of Algorithm 4, assume that we are given the previous iterate x k in p ε k (y k-1 ), the corresponding dual function d(x) and its prox-center w k satisfying x k = prox ψ/(κ+µ) (w k ). Then, initialize the sequence (z t ) t≥0 for minimizing

h k+1 = f + κ 2 ∥ • -y k ∥ 2 with, z 0 = prox ψ/(κ+µ) ( w k + κ κ + µ (y k -y k-1 ) ) ,
and initialize the dual function as

d 0 (x) = d(x) + κ 2 ∥x -y k ∥ 2 - κ 2 ∥x -y k-1 ∥ 2 .
Then, 1. when f is µ-strongly convex, we have h * k+1d 0 (z 0 ) ≤ Cε k+1 with the same constant as in (3.22) and (3.23), where d 0 is the dual function corresponding to z 0 ; 2. when f is convex with bounded level sets, there exists a constant B > 0 identical to the one of (3.24) such that

h * k+1 -d 0 (z 0 ) ≤ B.
Proof. The proof is given in Lemma D.5 of [START_REF] Lin | A universal catalyst for first-order optimization[END_REF], which gives

h * k+1 -d 0 (z 0 ) ≤ ε k + κ 2 2(κ + µ) ∥y k -y k-1 ∥ 2 .
This term is smaller than the quantity derived from (3.25), leading to the same upper bound.

Proposition 11 (Warm start for criterion (C2)). Consider applying Catalyst with the same parameter choices as in Proposition 7 to MISO/Finito. At iteration k + 1 of Algorithm 4, we assume that we are given the previous iterate x k in g δ k (y k-1 ) and the corresponding dual function d(x). Then, initialize the sequence (z t ) t≥0 for minimizing

h k+1 = f + κ 2 ∥ • -y k ∥ 2 by z 0 = prox ψ/(κ+µ) ( y k - 1 κ + µ ∇f 0 (y k ) ) ,
where f = f 0 + ψ and f 0 is the smooth part of f , and set the dual function d 0 by

d 0 (x) = f 0 (y k ) + ⟨∇f 0 (y k ), x -y k ⟩ + κ + µ 2 ∥x -y k ∥ 2 + ψ(x).
Then,

h * k+1 -d 0 (z 0 ) ≤ (L + κ) 2 2(µ + κ) ∥p(y k ) -y k ∥ 2 .
(3.30)

EXPERIMENTAL STUDY

Proof. Since p(y k ) is the minimum of h k+1 , the optimality condition provides

-∇f 0 (p(y k )) -κ(p(y k ) -y k ) ∈ ∂ψ(p(y k )).
Thus, by convexity,

ψ(p(y k )) + ⟨-∇f 0 (p(y k )) -κ(p(y k ) -y k ), z 0 -p(y k )⟩ ≤ ψ(z 0 ), f 0 (p(y k )) + κ 2 ∥p(y k ) -y k ∥ 2 + ⟨∇f 0 (p(y k )) + κ(p(y k ) -y k ), y k -p(y k )⟩ ≤ f 0 (y k ).
Summing up gives

h * k+1 ≤ f 0 (y k ) + ψ(z 0 ) + ⟨∇f 0 (p(y k )) + κ(p(y k ) -y k ), z 0 -y k ⟩. As a result, h * k+1 -d 0 (z 0 ) ≤ f 0 (y k ) + ψ(z 0 ) + ⟨∇f 0 (p(y k )) + κ(p(y k ) -y k ), z 0 -y k ⟩ -d 0 (z 0 ) = ⟨∇f 0 (p(y k )) + κ(p(y k ) -y k ) -∇f 0 (y k ), z 0 -y k ⟩ - κ + µ 2 ∥z 0 -y k ∥ 2 ≤ 1 2(κ + µ) ∥∇ f 0 (p(y k )) -∇f 0 (y k ) ∥•∥≤L∥p(y k )-y k ∥ +κ(p(y k ) -y k )∥ 2 ≤ (L + κ) 2 2(µ + κ) ∥p(y k ) -y k ∥ 2 .
The bound obtained from (3.30) is similar to the one form Proposition 7, and differs only in the constant factor. Thus, the inner loop complexity in Section 3.4.2 still holds for MISO/Finito up to a constant factor. As a consequence, the global complexity of MISO/Finito applied to Catalyst is similar to one obtained by SVRG, yielding an acceleration for ill-conditioned problems.

Experimental study

In this section, we conduct various experiments to study the effect of the Catalyst acceleration and its different variants, showing in particular how to accelerate SVRG, SAGA and MISO.

Datasets, formulations, and metric

Datasets. We consider four standard machine learning datasets with different characteristics in terms of size and dimension, which are described below: Formulations. We consider three common optimization problems in machine learning and signal processing, which admit a particular structure (large finite sum, composite, strong convexity). For each formulation, we also consider a training set (b i , a i ) n i=1 of n data points, where the b i 's are scalars in {-1, +1} and the a i are feature vectors in R d . Then, the goal is to fit a linear model x in R p such that the scalar b i can be well predicted by the inner-product ≈ a ⊤ i x, or by its sign. Specifically, the three formulations we consider are listed below.

• ℓ 2 2 -regularized Logistic Regression:

min x∈R d 1 n n ∑ i=1 log ( 1 + exp(-b i a T i x) ) + µ 2 ∥x∥ 2 ,
which leads to a µ-strongly convex smooth optimization problem.

• ℓ 1 -regularized Linear Regression (LASSO):

min x∈R d 1 2n n ∑ i=1 (b i -a T i x) 2 + λ∥x∥ 1 ,
which is non smooth and convex but not strongly convex.

• ℓ 1ℓ 2 2 -regularized Linear Regression (Elastic-Net):

min x∈R d 1 2n n ∑ i=1 (b i -a T i x) 2 + λ∥x∥ 1 + µ 2 ∥x∥ 2 ,
which is based on the Elastic-Net regularization [Zou and Hastie, 2005] leading to stronglyconvex optimization problem.

Each feature vector a i is normalized, and a natural upper-bound on the Lipschitz constant L of the un-regularized objective can be easily obtained with L logistic = 1/4 and L lasso = 1. The regularization parameter µ and λ are choosing in the following way:

• For Logistic Regression, we set µ = 0.01/n which corresponds to relatively illconditioned problems.

• For Elastic-Net, we set µ = 0.01/n as in logistic regression and add a small l 1regularization penalty with λ = 1/n that produces sparse solutions.

• For the Lasso problem, we consider a logarithmic grid 10 i /n, with i = -3, -2, . . . , 3, and we select the parameter λ that provides a sparse optimal solution closest to 10% non-zero coefficients, which leads to λ = 10/n or 100/n.

Note that for the strongly convex problems, the regularization parameter µ yields a lower bound on the strong convexity parameter of the problem.

Metric used. In this chapter, and following previous work about incremental methods [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF], we plot objective values as a function of the number of gradients evaluated during optimization, which appears to be the computational bottleneck of all previously mentioned algorithms. Since no metric is perfect for comparing algorithms speed, we shall make the two following remarks, such that the reader can interpret our results and the limitations of our study with no difficulty.

• Ideally, CPU-time would be a gold standard but CPU time is implementation-dependent and hardware-dependent. Thus, we have chosen the number of gradient computations as a proxy for CPU time that does not suffer from unwanted externalities.

• We have chosen to count only gradients computed with random data access. Thus, computing n times a gradient f i by picking each time one function at random counts as "n gradients", whereas we ignore the cost of computing a full gradient ∇(1/n)

∑ n i=1 f i , where the f i 's can be accessed in sequential order. Similarly, we ignore the cost of computing the function value

f (x) = (1/n) ∑ n i=1 f i (x)
, which is typically performed every pass on the data when computing a duality gap. While this assumption may be inappropriate in some contexts, the cost of sequential gradient computations was indeed insignificant in our experiments, where (i) datasets fit into memory; (ii) computing full gradients was done in C++ by calling BLAS2 functions exploiting multiple cores.

Choice of hyper-parameters and variants

Before presenting the numerical results, we discuss the choice of default parameters used in the experiments as well as different variants.

Choice of method M. We consider the acceleration of incremental algorithms which are able to adapt to the problem structure we consider: large sum of functions and composite regularization.

• The proximal SVRG algorithm [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF] with stepsize η = 1/L.

• SAGA algorithm [Defazio et al., 2014a] with stepsize η = 1/3L.

• The proximal MISO algorithm that we developed in Section 2.3.

Choice of regularization parameter κ.

As suggested by the theoretical analysis, we take κ to minimize the global complexity. Leading to the choice

For SVRG/SAGA/MISO, κ = L -µ n + 1 -µ.
Stopping criteria for the inner loop. The choice of the accuracies are driven from the theoretical analysis. Here, we specify it for the problems we consider:

• Stopping criterion (C1). Stop when h k (z t ) -h * k ≤ ε k , where ε k =      2 9 (1 -ρ) k f (x 0 ) 6 with ρ = 0.9 √ µ µ+κ when µ > 0; 2f (x 0 ) 9(k+1) 4.1 when µ = 0.
The duality gap h(w t )h * can be estimated either by evaluating the Fenchel conjugate or by computing the squared norm of the gradient.

• Stopping criterion (C2). Stop when h

k (z t ) -h * k ≤ δ k • κ 2 ∥z t -y k-1 ∥ 2 ,
where

δ k =      √ q 2-
√ q with q = µ µ+κ when µ > 0;

1 (k+1) 2
when µ = 0.

• Stopping criterion (C3) . Perform exactly one pass over the data in the inner loop without checking any stopping criteria.7

Warm start for the inner loop. This is an important point to achieve acceleration which was not highlighted in the conference paper [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]. At iteration k, we consider the minimization of

h k (z) = f 0 (z) + κ 2 ∥z -y k-1 ∥ 2 + ψ(z).
Let x k-1 be the approximate minimizer of h k-1 , obtained from the last iteration.

• Initialization for (C1). Let us define η = 1 L+κ , then initialize at

z C1 0 =    w 0 ≜ x k-1 + κ κ+µ (y k-1 -y k-2 ) if ψ = 0;
prox ηψ (w 0ηg) with g = ∇f 0 (w 0 ) + κ(w 0y k-1 ) otherwise.

• Initialization for (C2). Intialize at

z C2 0 =    y k-1 if ψ = 0;
prox ηψ (y k-1 -η∇f 0 (y k-1 )) otherwise.

EXPERIMENTAL STUDY

• Initialization for (C3). Take the best initial point among x k-1 and z C1

0 z C3 0 such that h k (z C3 0 ) = min{h k (x k-1 ), h k (z C1 0 )}.
• Initialization for (C1 * ). Use the strategy (C1) with z C3 0 . The warm start at z C3 0 requires to choose the best point between the last iterate x k-1 and the point z C1 0 . The motivation is that since the one-pass strategy is an aggressive heuristic, the solution of the subproblems may not be as accurate as the ones obtained with other criterions. Allowing to use the iterate x k-1 turned out to be significantly more stable in practice. Then, it is also natural to use a similar strategy for criterion (C1), which we call (C1 * ) . Using a similar strategy for (C2) turned out not to provide any benefit in practice and is thus omitted from the list here.

Comparison of Stopping Criteria and Warm Start Strategies

First, we evaluate the performance of the previous strategies when applying Catalyst to SVRG, SAGA and MISO. The results are presented in Figures 3.1, 3.2, and 3.3, respectively.

Observations for Catalyst-SVRG. We remark that in most of the cases, the curve of (C3) and (C1 * ) are superimposed, meaning that one pass through the data is enough for solving the subproblem up to the required accuracy. Moreover, they give the best performance among all criterions. Regarding the logistic regression problem, the acceleration is significant (even huge for the covtype dataset) except for alpha, where only (C3) and (C1 * ) do not degrade significantly the performance. For sparse problems, the effect of acceleration is more mitigated, with 4 cases out of 8 exhibiting important acceleration and 4 cases no acceleration. As before, (C3) and (C1 * ) are the only strategies that never degrade performance.

One reason explaining why acceleration is not systematic may be the ability of incremental methods to adapt to the unknown strong convexity parameter µ ′ ≥ µ hidden in the objective's loss, or local strong convexity near the solution. When µ ′ /L ≥ 1/n, we indeed obtain a well-conditioned regime where acceleration should not occur theoretically since the complexity O(n log(1/ε)) of the unaccelerated method is already optimal. For sparse problems, conditioning of the problem with respect to the linear subspace where the solution lies might also play a role, even though our analysis does not study this aspect. Therefore, this experiment suggests that adaptivity to unknown strong convexity is of high interest for incremental optimization.

Observations for Catalyst-SAGA. Our conclusions with SAGA are almost the same as with SVRG. However, in a few cases, we also notice that criterion C1 lacks stability, or at least exhibits some oscillations, which may suggest that SAGA has a larger variance compared to SVRG. The difference in the performance of (C1) and (C1 * ) can be huge, while they differ from each other only by the warm start strategy. Thus, choosing a good initial point for solving the sub-problems is a key for obtaining acceleration in practice. Observations for Catalyst-MISO. The warm start strategy of MISO is different from primal algorithms because parameters for the dual function need to be specified. The most natural way for warm starting the dual functions is to set

D k+1 0 (x) = D k (x) + κ 2 ∥x -y k ∥ 2 - κ 2 ∥x -y k-1 ∥ 2 ,
where D k is the last dual function of the previous subproblem h k . This gives the warm start

z 0 = prox ( x k + κ κ + µ (y k -y k-1 )
)

.

For other choices of z 0 , the dual function needs to be recomputed from scratch, which is computationally expensive and unstable for ill-conditioned problems. Thus, we only present the experimental results with respect to criterion (C1) and the one-pass heuristic (C3) . As we observe, a huge acceleration is obtained in logistic regression and Elastic-net formulations. For Lasso problem, the original Prox-MISO is not defined since the problem is not strongly convex. Thus, in order to make a comparison, we compare with Catalyst-SVRG which shows that the acceleration achieves a similar performance. This confirms the theoretical result that Catalyst applied to incremental algorithms yields a similar convergence rate. Notice also that the original MISO algorithm suffers from numerical instability in this ill-conditioned regime chosen for our experiments. Catalyst not only accelerates MISO, but it also stabilizes it.

Acceleration of Existing Methods

Then, we put the previous curves into perspective and make a comparison of the performance before and after applying Catalyst across methods. We show the best performance among the three developed stopping criteria, which corresponds to be (C3) .

Observations. We observe that by applying Catalyst, we accelerate the original algorithms up to the limitations discussed above (comparing the dashed line and the solid line of the same color). In three datasets (covtype, real-sim and rcv1), significant improvements are achieved as expected by the theory for the ill-conditioned problems in logistic regression and Elastic-net. For dataset alpha, we remark that an relative accuracy in the order 10 -10 is attained in less than 10 iterations. This suggests that the problems is in fact well-conditioned and there is The dashed lines correspond to the original algorithms and the solid lines correspond to accelerated algorithms by applying Catalyst. We plot the relative function value gap (f (x k )f * )/f * in the number of gradient evaluations, on a logarithmic scale. some hidden strong convexity for this dataset. Thus, the incremental algorithms like SVRG or SAGA are already optimal under this situation and no improvement is obtained by applying Catalyst.

Study of the Parameter κ

Finally, we provide evaluations of the performance for different κ. Observations for different choices of κ. We consider a logarithmic grid κ = 10 i κ 0 with i = -2, -1, • • • , 2 and κ 0 is the optimal κ given by the theory. We observe that for ill-conditioned problems, using optimal choice κ 0 provides much better performance than other choices, which confirms the theoretical result. For the dataset of alpha or Lasso problems, we observe that the best choice is given by the smallest κ = 0.01κ 0 . This suggests that, as discussed before, strong convexity may be hidden in the loss for this dataset.

Discussions and concluding remarks

In this chapter, we have introduced a generic algorithm, Catalyst, that brings Nesterov's acceleration to a large class of first-order methods. In particular, it can be applied to previously unaccelerated incremental algorithms such as SAG/SAGA, SVRG, MISO/Finito. The resulting algorithms after applying Catalyst achieve the optimal complexity up to a logarithmic factor. We studied experimentally different variants of it and show the acceleration does occur in practice. We compare different choices of parameters and show that the theoretical choice provides promising performance, especially for ill-conditioned problems. The current version of the algorithm requires the knowledge of both the Lipschitz constant parameter and the strong convexity parameter. As we have seen in the experiments, the acceleration degrades when the parameter are not correctly estimating, such as the existence of hidden strong convexity. Designing a parameter free algorithm is one of our important future directions.

Chapter 4

QuickeNing: acceleration with Quasi-Newton principles

Chapter abstract:

We propose a generic approach to accelerate gradient-based optimization algorithms with quasi-Newton principles. The proposed scheme, called QuickeNing, can be applied the same class of methods as Catalyst, including classical gradient methods and incremental first-order methods. It is a variable metric approach compatible with composite objectives, meaning that it has the ability to deal with sparsity-inducing regularizations and provide exactly sparse solutions. QuickeNing relies on applying an inexact limited-memory BFGS rules to the Moreau-Yosida regularization, which is also known as variable metric proximal point algorithm. We provide a global convergence analysis, showing that QuickeNing enjoys a worst-case linear convergence rate for strongly convex problems. We present experimental results where QuickeNing gives significant improvements over competing methods for solving large-scale high-dimensional machine learning problems.

The material of this chapter is based on the following paper, which is currently under review: H. Lin, J. Mairal, and Z. Harchaoui. A generic quasi-newton algorithm for faster gradientbased optimization. arXiv:1610.00960v2, 2017

The code for reproducing the figures in this chapter is publicly available at: https://github.com/hongzhoulin89/Catalyst-QNing

Introduction

First-order methods are often the default choice in machine learning for solving the empirical risk minimization problem,

min x∈R p { f (x) = 1 n n ∑ i=1 f i (x) + ψ(x) } . (4.1)
However, it is also known that standard Quasi-Newton approaches can sometimes be surprisingly effective in the smooth case-that is when ψ = 0, see, e.g., [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF] for extensive benchmarks. Since the dimension of the problem d is typically very large (d ≥ 10 000), "limited memory" variants of these algorithms, such as L-BFGS, are necessary to achieve the desired scalability [START_REF] Liu | On the limited memory BFGS method for large scale optimization[END_REF]Nocedal, 1989, Nocedal, 1980]. The theoretical guarantees offered by L-BFGS are somewhat weak, meaning that it does not outperform accelerated first-order methods in terms of worst-case convergence rate, and also it is not guaranteed to correctly approximate the Hessian of the objective. Yet, it remains one of the greatest practical success of smooth optimization. Adapting L-BFGS to composite and structured problems, such as the finite sum of functions (4.1), is of utmost importance nowadays. For instance, there have been several attempts to develop a proximal Quasi-Newton method [START_REF] Byrd | An inexact successive quadratic approximation method for L-1 regularized optimization[END_REF][START_REF] Lee | Proximal Newton-type methods for convex optimization[END_REF][START_REF] Scheinberg | Practical inexact proximal quasi-Newton method with global complexity analysis[END_REF][START_REF] Yu | A quasi-Newton approach to non-smooth convex optimization[END_REF]. These algorithms typically require computing many times the proximal operator of ψ with respect to a variable metric, which may be as computationally demanding as solving the original problem. Quasi-Newton steps were also incorporated as local search steps into accelerated first-order methods to further enhance their numerical performance [START_REF] Ghadimi | Generalized Uniformly Optimal Methods for Nonlinear Programming[END_REF]. More related to our work, L-BFGS is combined with SVRG for minimizing smooth finite sums in [START_REF] Gower | Stochastic block BFGS: Squeezing more curvature out of data[END_REF]. The scope of our approach is broader beyond the case of SVRG. We present a generic Quasi-Newton scheme, applicable to a large-class of first-order methods for composite optimization, including other incremental algorithms [Defazio et al., 2014a,b, Mairal, 2015[START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF][START_REF] Shalev-Shwartz | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF] and block coordinate descent methods [Razaviyayn et al., 2013, Richtárik and[START_REF] Richtárik | Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function[END_REF] More precisely, our main contribution is a generic meta-algorithm, called QuickeNing (the letters "Q" and "N" stand for Quasi-Newton), which uses a given optimization method to solve a sequence of auxiliary problems up to some appropriate accuracy, resulting in faster global convergence in practice. The resulting scheme admits a simple interpretation: it may be seen as applying the L-BFGS algorithm with inexact (but accurate enough) gradients to the Moreau-Yosida regularization of the objective. As a result, our approach is (i) generic, as stated previously; (ii) despite the smoothing of the objective, the sub-problems that we solve are composite ones, which may lead to exactly sparse iterates when a sparsity-inducing regularization is involved, e.g., the ℓ 1 -norm; (iii) it admits a worst-case linear convergence rate for strongly convex problems similar to that of gradient descent, which is typically the best guarantees obtained for L-BFGS schemes in the literature; (iv) it is easy to use and does

RELATED WORK AND PRELIMINARIES

not require using a line search algorithm, which is sometimes computationally expensive and difficult to calibrate in classical Quasi-Newton methods.

The idea of combining second-order or quasi-Newton methods with Moreau-Yosida regularization is in fact relatively old. It may be traced back to variable metric proximal bundle methods [START_REF] Chen | Proximal quasi-Newton methods for nondifferentiable convex optimization[END_REF][START_REF] Fukushima | A globally and superlinearly convergent algorithm for nonsmooth convex minimization[END_REF][START_REF] Mifflin | A quasi-second-order proximal bundle algorithm[END_REF], which use BFGS updates on the Moreau-Yosida smoothing of the objective and bundle methods to approximately solve the corresponding sub-problems. Our approach revisits this principle with a limited-memory variant (to deal with large dimension d), with an alternative strategy to line search schemes-which is useful when f is a large sum of n functions as in (4.1)-and with warm start strategies for the sub-problems with a global complexity analysis that is more relevant than convergence rates that do not take into account the cost per iteration.

To demonstrate the effectiveness of our scheme in practice, we evaluate QuickeNing on regularized logistic regression and regularized least-squares, with smooth and non smooth regularization penalities such as the Elastic-Net [Zou and Hastie, 2005]. We use large-scale machine learning datasets and show that QuickeNing performs at least as well as the recently proposed Catalyst [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] and as the classical L-BFGS scheme in all experiments, and significantly outperforms them in many cases.

This chapter is organized as follows: Section 4.2 presents related work on Quasi-Newton methods such as L-BFGS; we introduce QuickeNing in Section 4.3, and we provide a convergence analysis in Section 4.4; Section 4.5 is devoted to experiments and Section 4.6 concludes the chapter.

Related work and preliminaries

The history of quasi-Newton methods can be traced back to the 1950's [START_REF] Bonnans | Numerical Optimization: Theoretical and Practical Aspects[END_REF], Hiriart-Urruty and Lemaréchal, 1996[START_REF] Nocedal | Numerical optimization[END_REF]. Quasi-Newton methods often lead to significantly faster convergence in practice compared to simpler gradient-based methods for solving smooth optimization problems [Schmidt et al., 2011a]. Yet, a theoretical analysis of quasi-Newton methods that explains their impressive empirical behavior on a wide range of problems is still an open topic. Here, we recall the well-known BFGS algorithm in Section 4.2.1, its limited memory variant [START_REF]Updating quasi-Newton matrices with limited storage[END_REF], and a few recent extensions. Then, we present earlier works that combine proximal point and Quasi-Newton algorithms in Section 4.2.2, which is related to the strategy we use in this chapter.

Quasi-Newton methods for smooth optimization

In Section 1.3, we introduce Quasi-Newton methods for smooth optimization. Here, we recall the updates of BFGS, named after its inventors (Broyden-Fletcher-Goldfarb-Shanno), and its limited variant L-BFGS [START_REF] Nocedal | Numerical optimization[END_REF]. These approaches will be the workhorses of the QuickeNing meta-algorithm. Consider a smooth convex objective f to be minimized, the BFGS method constructs at iteration k a couple (x k , B k ) with the following update:

x k+1 = x k -α k B -1 k ∇f (x k ) and B k+1 = B k - B k s k s ⊤ k B k s ⊤ k B k s k + y k y ⊤ k y ⊤ k s k , (4.2)
where α k is a suitable stepsize and

s k = x k+1 -x k , y k = ∇f (x k+1 ) -∇f (x k ).
The condition y ⊤ k s k > 0 and the positive definiteness of B k are guaranteed as soon as f is strongly convex. To determine the stepsize α k , Wolfe's line-search is a simple choice which provides linear convergence rate in the worst case. In addition, if the objective is twice differentiable and the Hessian is Lipschitz continuous, the convergence is asymptotically superlinear [START_REF] Nocedal | Numerical optimization[END_REF].

The limited memory variant L-BFGS [START_REF]Updating quasi-Newton matrices with limited storage[END_REF] overcomes the issue of storing B k for large d, by replacing it by another positive definite matrix-say Bk -which can be built from a "generating list" of at most l pairs of vectors {(s k i , y k i )} i=0...j along with an initial diagonal matrix B0 . Formally, Bk can be computed by applying at most l times a recursion similar to (4.2) involving all pairs of the generating list. Between iteration k and k + 1, the generating list is incrementally updated, by removing the oldest pair in the list (when j = l) and adding a new one. What makes the approach appealing is the ability of computing H k z = B -1 k z for any vector z with only O(ld) floating point operations instead of O(d 3 ) for a naive implementation with matrix inversion. The price to pay is that superlinear convergence becomes out of reach in contrast to BFGS, (see Section 1.3).

L-BFGS is thus appropriate for high-dimensional problems (when d is large), but it still requires computing the full gradient at each iteration, which may be cumbersome in the large sum setting (4.1). This motivated stochastic counterparts of the Quasi-Newton method (SQN) [START_REF] Byrd | A stochastic quasi-Newton method for large-scale optimization[END_REF]. The direct application substituting the full gradient ∇f (x k ) by a stochastic estimate is unfortunately not convergent. Instead, the SQN method [START_REF] Byrd | A stochastic quasi-Newton method for large-scale optimization[END_REF] uses a product of a sub-sampled Hessian and s k to approximate y k . SQN can be complemented by a variance reduction scheme such as SVRG [START_REF] Gower | Stochastic block BFGS: Squeezing more curvature out of data[END_REF][START_REF] Moritz | A linearly-convergent stochastic L-BFGS algorithm[END_REF].

Combining the proximal point algorithm and Quasi-Newton

There are several ways to extend Quasi-Newton methods for nonsmooth optimization. A first approach consists in minimizing successive quadratic approximations, providing the update

x k+1 = arg min x∈R p { f 0 (x k ) + ⟨∇f 0 (x k ), x -x k ⟩ + 1 2 (x -x k ) T B k (x -x k ) + ψ(x) } , (4.3)
which is known as proximal Quasi-Newton method [START_REF] Byrd | An inexact successive quadratic approximation method for L-1 regularized optimization[END_REF][START_REF] Lee | Proximal Newton-type methods for convex optimization[END_REF][START_REF] Scheinberg | Practical inexact proximal quasi-Newton method with global complexity analysis[END_REF][START_REF] Yu | A quasi-Newton approach to non-smooth convex optimization[END_REF]. A major drawback is then the computation of the proximal operator. Since B k is dense and changes over the iterations, an exact solution to the proximal operator is usually not available. An inexact variant has been analyzed in [START_REF] Byrd | An inexact successive quadratic approximation method for L-1 regularized optimization[END_REF], but it is unclear whether or not the quadratic approximation could be solved inexactly in an efficient manner.

A second approach consists in applying Quasi-Newton methods after Moreau-Yosida smoothing or equivalently a proximal point algorithm on the objective function [START_REF] Bonnans | A family of variable metric proximal methods[END_REF][START_REF] Fukushima | A globally and superlinearly convergent algorithm for nonsmooth convex minimization[END_REF][START_REF] Chen | Proximal quasi-Newton methods for nondifferentiable convex optimization[END_REF][START_REF] Burke | On the superlinear convergence of the variable metric proximal point algorithm using Broyden and BFGS matrix secant updating[END_REF][START_REF] Fuentes | Descentwise inexact proximal algorithms for smooth optimization[END_REF]. The underlying idea is to first smooth and improve the conditioning of the objective function, then apply a Quasi-Newton method on the smoothed function. We recall here the definition of the Moreau-Yosida regularization, which we have presented in Section 1.6

F (x) ≜ min z∈R p { f (z) + κ 2 ∥z -x∥ 2 } , (4.4)
where κ is a positive scalar. Applying Quasi-Newton methods on F gives a sequence of subproblems in the shape of (4.4). The main difference between the proximal Newton approach and this smoothing approach is that the subproblems generated by (4.3) change the metric at every iteration while the subproblems in (4.4) have a fixed conditioning. Moreover, the conditioning of the subproblems is explicitly controlled by the parameter κ, which is free to choose. Thus, it suffices to take an appropriate value of κ to make the subproblem well conditioned and apply an optimization method to approximate the solution. For instance, in [START_REF] Chen | Proximal quasi-Newton methods for nondifferentiable convex optimization[END_REF], the sub-problems are solved inexactly by proximal bundle methods, and in [START_REF] Fuentes | Descentwise inexact proximal algorithms for smooth optimization[END_REF] by using a sophisticated sufficient descent criterion. While the proposed algorithms are elegant, the required complexity of solving these subproblems has not been investigated, which explains the lack of global complexity result. The proposed QuickeNing algorithm, which will be presented in the next section, follows this line of research. We provide a global convergence analysis taking into account the complexity for solving the subproblems. More importantly, the complexity result allows us to provide an explicit choice for the parameter κ, which hardly discussed in the previous work. Moreover, we propose a simple strategy to set the parameters, removing the need of a line-search scheme in the determination of stepsizes. Several theoretical grounded warm start strategies are proposed for efficiently solving the sub-problems, leading to a significant acceleration in practice.

QuickeNing: a Quasi-Newton meta-algorithm

We now present the QuickeNing method in Algorithm 5, which consists of applying L-BFGS rules on the smoothed objective F with inexact gradients. Each gradient approximation is the result of a minimization problem tackled with the algorithm M, used as a sub-routine. The outer loop of the algorithm performs L-BFGS quasi-Newton updates. The method M can be any algorithm of the user's choice, as long as it enjoys linear convergence rate for strongly convex problems. More details about the choice of the parameter κ and about the inexactness criterion to use will be given next.

Algorithm 5 QuickeNing input Initial point x 0 in R p ; number of iterations K; parameter κ > 0; minimization algorithm M. 1: Initialization: (g 0 , F 0 , z 0 ) = ApproxGradient (x 0 , M); L-BFGS matrix H 0 = (1/κ)I. 2: for k = 0, . . . , K -1 do 3:

Perform the Quasi-Newton step

x test = x k -H k g k . 4:
Estimate the new gradient and the Moreau-Yosida function value

(g test , F test , z test ) = ApproxGradient (x test , M) . 5:
if sufficient approximate decrease is obtained

F test ≤ F k - 1 2κ ∥g k ∥ 2 , (4.5) then 6:
Accept the new iterate: (x k+1 , g k+1 , F k+1 , z k+1 ) = (x test , g test , F test , z test ). Update the current iterate with the proximal mapping:

x k+1 = z k . (g k+1 , F k+1 , z k+1 ) = ApproxGradient (x k+1 , M) . 9:
end if 10:

update H k+1 = L-BFGS(H k , x k+1x k , g k+1g k ). 11: end for output last proximal mapping z K (solution).

Algorithm 6 Generic procedure ApproxGradient input Current point x in R p ; smoothing parameter κ > 0.

1: Compute the approximate proximal mapping using an optimization method M:

z ≈ arg min w∈R p { h(w) ≜ f (w) + κ 2 ∥w -x∥ 2 } , (4.6)
using the warm start strategies and stopping criterions described later in this section. 2: Estimate the gradient ∇F (x) of the Moreau-Yosida objective function

g = κ(x -z).
output approximate gradient estimate g, objective value F a ≜ h(z), proximal mapping z.

The main algorithm

We now discuss the main algorithm components and its main features.

Outer-loop: inexact L-BFGS. The L-BFGS rule is the standard one and consists in updating incrementally a generating list of vectors {(s i , y i )} i=1...j , which implicitly defines the L-BFGS matrix. We use here the two-loop recursion detailed in Algorithm 7.4 of [START_REF] Nocedal | Numerical optimization[END_REF] and use skipping steps when the condition s ⊤ i y i > 0 is not satisfied, in order to ensure the positive-definiteness of the L-BFGS matrix H k (see [START_REF] Friedlander | Hybrid deterministic-stochastic methods for data fitting[END_REF]).

Inner-loop: approximate Moreau-Yosida envelope. The minimization algorithm M is used to compute an approximate Moreau-Yosida envelope of the objective. The procedure ApproxGradient () calls the minimization algorithm M to minimize the sub-problem (4.6). When the problem is solved exactly, the function returns the exact values g = ∇F (x), F a = F (x) and z = p(x). Otherwise, a stopping criterion should be used. In this chapter, we consider the three following cases:

(a) defining a pre-defined decreasing sequence (ε k ) k≥0 and stopping criterion based on function values-that is, stop the optimization method M when the approximate solution of (4.6) satisfies h(z)h * ≤ ε k , where h * = min z∈R d h(z), which may be checked by computing a duality gap. This is the less practical of the three strategies presented here.

(b) defining an adaptive sequence (ε k ) k≥0 , which depends on quantities that are available at iteration k.

(c) using a pre-defined budget T M in terms of number of iterations of the method M at iteration k.

Section 4.4 will provide theoretical and practical insight about these parameter choices. Before that, we discuss the requirements on f and M and warm start strategies to solve the subproblems.

Requirements on f . When f is µ-strongly-convex, we show that the method achieves a linear convergence rate (which takes into account the complexity of solving the sub-problems with M); to obtain this guarantee, the accuracy for solving the sub-problems needs to decrease at the same linear rate. In principle, all strategies (a)-(b)-(c) above may be used to stop M, as discussed in Section 4.4. When f is convex but not strongly convex (meaning µ = 0), it is possible to achieve the classical O(1/k) worst-case convergence rate on function values, by using a particular strategy (b) and by replacing the approximate descent condition (4.5) by a slightly stronger one f (x test ) ≤ f (z k ), see Proposition 15 for more details.

Requirements on M and warm start. We have mentioned that QuickeNing applies to methods M with linear convergence rates for strongly-convex problems. More precisely, we consider methods M that are always able to produce a sequence of iterates (w t ) t≥0 for solving each sub-problem (4.6) such that

h(w t ) -h * ≤ C M (1 -τ M ) t (h(w 0 ) -h * ) for some constants C M , τ M > 0, (4.7)
where w 0 is the initial point given to M. When (4.7) is satisfied, we call M a primal method.

QuickeNing can also afford non-deterministic methods M that satisfy (4.7) in expectation; in such a case, our complexity results also hold in expectation. The condition typically holds for many primal gradient-based optimization techniques, such as block-coordinate descent algorithms [Nesterov, 2012b, Richtárik and[START_REF] Richtárik | Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function[END_REF], or some incremental algorithms [Defazio et al., 2014a[START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF]. The warm start strategy derived from our convergence analysis in Section 4.4 is simply w 0 = x test . The class of dual methods MISO/Finito [Defazio et al., 2014b[START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF] can also be applied with a different warm start in the same way as Catalyst [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]. Finally, before giving more precise theoretical statements, we discuss briefly two important features of the algorithm.

Handling composite objective functions. In machine learning or signal processing, convex composite objectives (4.1) with a non-smooth penalty ψ are typically formulated to encourage solutions with specific characteristics; in particular, the ℓ 1 -norm is known to provide sparsity. Smoothing techniques [START_REF] Nesterov | Smooth minimization of non-smooth functions[END_REF] may allow us to solve the optimization problem up to some chosen accuracy, but they provide solutions that do not inherit the properties induced by the non-smoothness of the objective. To illustrate what we mean by this sentence, we may consider smoothing the ℓ 1 -norm, leading to a solution vector with small coefficients, but not with exact zeroes. When the goal is to perform model selection-that is, understanding which variables are important to explain a phenomenon, exact sparsity is seen as an asset, and optimization techniques dedicated to composite problems such as FISTA [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] are often preferred.

Then, one might be concerned that our scheme operates on the smoothed objective F , leading to iterates (x k ) k≥0 that may suffer from the above "non-sparse" issue, assuming that ψ is the ℓ 1 -norm. Yet, our approach also provides iterates (z k ) k≥0 that are computed using the original optimization method M we wish to accelerate. When M handles composite problems without smoothing, typically when M is a proximal block-coordinate, or incremental method, the iterates (z k ) k≥0 may be sparse. For this reason, our theoretical analysis presented in Section 4.4 studies the convergence of the sequence (f (z k )) k≥0 to the solution f * .

On the absence of line-search scheme. Another key property of the QuickeNing algorithm is that it does not require using a line-search scheme to select a step-size, which is typically necessary to ensure the monotonic descent (and convergence) of classical BFGS and L-BFGS algorithms [START_REF]Updating quasi-Newton matrices with limited storage[END_REF]. In the context of the Moreau-Yosida regularization, any line-search would be prohibitive, since it would require to evaluate the function F multiple times, hence solving the sub-problems (4.6) as many times at every iteration.1 Here, we choose a simple strategy that selects x k+1 = z k when the sufficient descent condition (4.5) is not satisfied. z k is indeed a good candidate since it is obtained by performing one step of the inexact proximal point algorithm, which we have described above, and whose convergence properties are well understood. In practice, we have observed that the sufficient decrease condition is almost all the time satisfied, given the choice of parameters provided in the experimental section.

Convergence and complexity analysis

In this section, we study the convergence of the QuickeNing algorithm-that is, the rate of convergence to zero of the quantities (f (z k )f * ) k≥0 and (F (x k ) -F * ) k≥0 , and also its computational complexity, which takes into account the cost of solving the sub-problems (4.6). We consider several cases, leading to different variants of the parameter choices, warm start and approximation strategies. We start by stating the main properties of the gradient approximation in Section 4.4.1. Then, we analyze the convergence of the outer loop algorithm in Section 4.4.2, and then Section 4.4.3 is devoted to the global complexity analysis.

Properties of the gradient approximation

The next lemma is classical and provides approximation guarantees about the quantities returned by the ApproxGradient procedure (Algorithm 6); see [Bertsekas, 2015, Fukushima and[START_REF] Fukushima | A globally and superlinearly convergent algorithm for nonsmooth convex minimization[END_REF]. We recall here the proof for completeness.

Lemma 11 (Approximation quality of the gradient approximation). Consider a vector x in R p , a positive scalar ε and compute

z ≈ arg min v∈R p { h(v) ≜ f (v) + κ 2 ∥v -x∥ 2 } , such that h(z) -h * ≤ ε, where h * = min v∈R p h(v).
As in Algorithm 6, define g = κ(xz) and F a = h(z). Then, the following inequalities hold

F (x) ≤ F a ≤ F (x) + ε, (4.8) ∥z -p(x)∥ ≤ √ 2ε κ , (4.9) ∥g -∇F (x)∥ ≤ √ 2κε.
(4.10)

Proof. (4.8) is straightforward by definition of h(z). Since f is convex, the function h is κstrongly convex, and (4.9) follows from

κ 2 ∥z -p(x)∥ 2 ≤ h(z) -h(p(x)) = h(z) -h * ≤ ε,
where we recall that p(x) minimizes h. Finally, we obtain (4.10) from

g -∇F (x) = κ(x -z) -κ(x -p(x)) = κ(p(x) -z),
by using the definitions of g and the property (1.37).

This lemma allows us to quantify the quality of the gradient and function value approximations in terms of ε, which is a key to control the error accumulation of our algorithm. Before moving to the first convergence result, a few additional technical inequalities are needed.

Lemma 12 (Simple inequalities regarding the gradient approximation). Consider the same quantities introduced in Lemma 11. Then,

f (z) = F a - 1 2κ ∥g∥ 2 , (4.11) 1 2 ∥∇F (x)∥ 2 -2κε ≤ ∥g∥ 2 ≤ 2(∥∇F (x)∥ 2 + 2κε).
(4.12)

Proof. Eq. ( 4.11) is obtained by using the definitions of g and noticing that F a = h(z). Eq. ( 4.12) follows from

∥∇F (x)∥ 2 ≤ 2(∥∇F (x) -g∥ 2 + ∥g∥ 2 ) ≤ 2(2κε + ∥g∥ 2 ) (from (4.10)).
Interchanging ∇F (x) and g gives the right-hand side inequality.

Convergence analysis of the outer loop

We are now in shape to establish the convergence of the QuickeNing meta-algorithm, without considering yet the cost of solving the sub-problems (4.6). We first remark that the following relation always holds

(g k , F k , z k ) = ApproxGradient(x k , M) , (4.13)
which allows us to apply Lemma 12 for all k ≥ 0. In the following analysis, we will always call ε k the precision (in the sense of Lemma 11) of solving the sub-problem (4.6) related to the call (4.13).

Lemma 13 (Approximate descent property). Consider the sequences (x k ) k≥0 and (z k ) k≥0 generated by Algorithm 5 where z k is obtained by solving a sub-problem (4.6) with accuracy ε k . Then,

F (x k+1 ) ≤ f (z k ) ≤ F (x k ) - 1 4κ ∥∇F (x k )∥ 2 + 2ε k . ( 4
f (z k ) = F k - 1 2κ ∥g k ∥ 2 .
To show the first inequality, we consider two cases:

• When the condition (4.5) is violated, then we set x k+1 = z k . As a result,

F (x k+1 ) = min z∈R k { f (z) + κ 2 ∥z -x k+1 ∥ 2 } ≤ f (x k+1 ) = f (z k ).
• Otherwise, we have

F (x k+1 ) ≤ F k+1 ≤ F k -1 2κ ∥g k ∥ 2 = f (z k ).
Thus, we have F (x k+1 ) ≤ f (z k ) in both cases and we may now prove the second inequality:

f (z k ) = F k - 1 2κ ∥g k ∥ 2 ≤ F (x k ) + ε k - ( 1 4κ ∥∇F (x k )∥ 2 -ε k )
(from (4.8) and (4.12))

= F (x k ) - 1 4κ ∥∇F (x k )∥ 2 + 2ε k .
This concludes the proof.

This lemma gives us a first intuition about the natural choice of the accuracy ε k to use, which should be of the same order as ∥∇F (x k )∥ 2 . Unfortunately, the quantity ∥∇F (x k )∥ 2 is not known since computing its value requires solving exactly the subproblem (4.6). We propose in the next section several practical strategies to set the sequences (ε k ) k≥0 . Before that, we now use the approximate descent property of the previous lemma to control the accumulation of errors across iterations.

Strongly-convex objectives.

We start by analyzing the convergence of our algorithm for strongly-convex objectives.

Proposition 12 (Convergence of Algorithm 5 with theoretical sequence (ε k ) k≥0 .). Assume that f is µ-strongly convex and define ρ = µ 4(µ+κ) . Consider the sequences (x k ) k≥0 and (z k ) k≥0 generated by Algorithm 5 where z k is obtained by solving a sub-problem (4.6) with accuracy ε k where

ε k ≤ 1 16κ ∥∇F (x k )∥ 2 . (4.15)
Then,

F (x k+1 ) -F * ≤ f (z k ) -f * ≤ (1 -ρ) k+1 (f (x 0 ) -f * ).
Proof. We may apply Lemma 13, and from (4.14), we obtain

F (x k+1 ) -F * ≤ f (z k ) -F * ≤ F (x k ) -F * - 1 8κ ∥∇F (x k )∥ 2 ≤ F (x k ) -F * - µ F 4κ (F (x k ) -F * ) = (1 -ρ) (F (x k ) -F * ) ≤ (1 -ρ) k+1 (f (x 0 ) -f * ),
where the second inequality uses a classical relation Theorem 2.1.10 in [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF] for strongly-convex functions, and the third one simply uses the definition of ρ given in Proposition 14.

The previous proposition is interesting from a theoretical point of view, but, as discussed previously, the quantity ∥∇F (x k )∥ 2 is unknown in advance. Therefore, the above criterion does not seem practical at first sight. The next proposition presents one way to use them concretely.

Proposition 13 (On using Proposition 12 in practice). The following condition implies the inequality (4.15) in Proposition 12:

ε k ≤ 1 36κ ∥g k ∥ 2 .
Proof. We may use (4.12) from Lemma 12, which gives

∥g k ∥ 2 ≤ 2 ( ∥∇F (x k )∥ 2 + 1 18 ∥g k ∥ 2 ) , which implies 4 9 ∥g k ∥ 2 ≤ ∥∇F (x k )∥ 2 and thus ε k ≤ 1 36κ ∥g k ∥ 2 ≤ 1 16κ ∥∇F (x k )∥ 2 .
While we do not have access to ∥∇F (x k )∥ 2 , we do have access to the estimate gradient g k = κ(x kz k ). This immediately suggests the following stopping criterion when applying the method M to (4.6)

h(w t ) -h * ≤ κ 36 ∥w t -x∥ 2 (4.16)
where (w t ) t≥0 is the sequence produced by M to solve the sub-problem (4.6). Again, such a condition can often be checked by computing duality gap. This choice ensures the linear convergence of the sequences (F (x k )) k≥0 and (f (z k )) k≥0 to f * . We shall also see in Section 4.4.3 that the condition may also be enforced by simply using a constant number of iterations of the method M when the right warm start strategy is used. Another possibility is to use a predefined sequence (ε k ) k≥0 , similar to Catalyst [START_REF] Lin | A universal catalyst for first-order optimization[END_REF].

Proposition 14 (Convergence of Algorithm 5 with pre-defined sequence). Assume that f is µ-strongly convex and define ρ = µ 4(µ+κ) . Consider the sequences (x k ) k≥0 and (z k ) k≥0 generated by Algorithm 5 where z k is obtained by solving a sub-problem (4.6) with accuracy ε k for all k. Then,

F (x k+1 ) -F * ≤ f (z k ) -f * ≤ (1 -2ρ) k+1 (f (x 0 ) -f * ) + 2 k ∑ i=0 (1 -2ρ) k-i ε i .
Proof. First, we recall from Proposition 1 that F is µ F -strongly convex with parameter µ F = µκ µ+κ and that the optimal value of the objectives f and F coincide-in other words, and F * = f * . We may then apply Lemma 13 and

F (x k+1 ) -F * ≤ f (z k ) -f * ≤ F (x k ) -F * - 1 4κ ∥∇F (x k )∥ 2 + 2ε k , ≤ F (x k ) -F * - µ F 2κ (F (x k ) -F * ) + 2ε k , = (1 -2ρ) (F (x k ) -F * ) + 2ε k , ≤ (1 -2ρ) k+1 (F (x 0 ) -F * ) + 2 k ∑ i=0 (1 -2ρ) k-i ε i , ≤ (1 -2ρ) k+1 (f (x 0 ) -f * ) + 2 k ∑ i=0 (1 -2ρ) k-i ε i ,
where the last inequality simply comes from the upper-bound F ≤ f .

Corollary 5 (Recommended pre-defined sequence (ε k ) k≥0 ). If the sequence (ε k ) k≥0 in Proposition 14 satisfies

ε k = 1 2 C(1 -ρ) k+1 with C ≥ f (x 0 ) -f * . then F (x k+1 ) -F * ≤ f (z k ) -f * ≤ C ρ (1 -ρ) k+2 .
(4.17)

Proof. From Proposition 14, we have

F (x k+1 ) -F * ≤ f (z k ) -f * ≤ (1 -2ρ) k+1 (f (x 0 ) -f * ) + 2 k ∑ i=0 (1 -2ρ) k-i ε i , ≤ (1 -2ρ) k+1 C + k ∑ i=0 (1 -2ρ) k-i (1 -ρ) i+1 C, = C (1 -ρ) k+2 -(1 -2ρ) k+2 (1 -ρ) -(1 -2ρ) , ≤ C ρ (1 -ρ) k+2 .
We remark that, of course, the quantity f (x 0 )f * is unknown in practice, but the proposition only requires an upper-bound, e.g., a duality gap, or simply f (x 0 ) if the function is nonnegative. Next, we present a similar complexity analysis for convex, but not strongly-convex objectives.

Convex but not strongly-convex objectives.

The goal of this section is to show that a minor modification of the algorithm yields the classical O(1/k) convergence rate of the sequences (F (x k )) k≥0 and (f (z k )) k≥0 , which can be shown to be slightly stronger than the previous one, when the accuracy ε k is small enough. Specifically, we replace the descent condition (4.5) by f (x test ) ≤ f (z k ). We may now use the new descent condition to derive a convergence rate for non-strongly convex function.

Proposition 15 (Convergence of Algorithm 5 for convex, but not strongly-convex objectives). Let f be a convex function, which attains its minimum at x * . Assume that the level sets of f are bounded. More precisely, assume that there exists R > 0 such that

∀y ∈ R p s.t. f (y) ≤ f (x 0 ) then ∥y -x * ∥ ≤ R.
Assume that the sub-problems (4.6) are solved up to accuracy ε k ≤ 1 2κ ∥g k ∥ 2 for any k, and replace the descent condition (4.5) in Algorithm 5 by f (x test ) ≤ f (z k ); then, the sequence

(x k ) k≥1 satisfies f (x k ) -f * ≤ 2κR 2 k + 3 .
Proof. First, we remark that with the new condition, we always have

f (x k+1 ) ≤ f (z k ), since x k+1 = z k when the descent condition f (x test ) ≤ f (z k ) is not satisfied. Moreover, according to Lemma 11, f (z k ) + κ 2 ∥z k -x k ∥ 2 = F k ≤ F (x k ) + ε k ,
and then, by combining the two previous inequalities

f (x k+1 ) ≤ f (z k ) ≤ F (x k ) - κ 2 ∥z k -x k ∥ 2 + ε k = F (x k ) - 1 2κ ∥g k ∥ 2 + ε k ≤ F (x k ).

CONVERGENCE AND COMPLEXITY ANALYSIS

This relation allows us to use a similar proof technique introduced in [START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF] in the context of the proximal gradient descent method

f (x k+1 ) ≤ F (x k ) = min z∈R p { f (z) + κ 2 ∥z -x k ∥ 2 } ≤ min α∈[0,1] { f (αx * + (1 -α)x k ) + κα 2 2 ∥x * -x k ∥ 2 } ≤ min α∈[0,1] { αf (x * ) + (1 -α)f (x k ) + κα 2 2 ∥x * -x k ∥ 2 } .
Since f (x k ) is necessarily decreasing (take α = 0 in the previous relation), we may use the bounded level set assumption, which leads to

f (x k+1 ) -f * ≤ min α∈[0,1] { (1 -α)(f (x k ) -f * ) + κα 2 R 2 2 } . 1. If f (x k ) -f * ≥ κR 2 , then α * = 1 and f (x k+1 ) -f * ≤ κR 2 2 . 2. Otherwise, α * = f (x k )-f * κR 2
, which gives

r k+1 ≤ r k (1 - r k 2κR 2 ) ≤ κR 2 2 ,
where

r k ≜ f (x k ) -f * . Thus 1 r k+1 ≥ 1 r k (1 -r k 2κR 2 ) ≥ 1 r k ( 1 + r k 2κR 2 ) = 1 r k + 1 2κR 2 .
In both cases, r 1 ≤ κR 2 2 and there

1 r k+1 ≥ 1 r 1 + k 2κR 2 ≥ k + 4 2κR 2 ,
which is sufficient to conclude.

Complexity analysis of the inner loop

In this section, we compute the complexity of solving the sub-problems (4.6) using the method M, which allows us to obtain the global complexity of QuickeNing, when using the various approximation strategies described in the previous section. We analyze here primal approaches M, which achieve the convergence guarantees (4.7). Dual methods like MISO/Finito whose convergence rate does not satisfy (4.7) require a different warm start as presented in the appendix of [START_REF] Lin | A generic quasi-newton algorithm for faster gradient-based optimization[END_REF]. In both cases, our main result is that all sub-problems can be solved with an appropriate accuracy in a constant number T M of iterations. One of the key is to choose the right warm start point, as detailed next. The first lemma characterizes the quality of the initialization w 0 = x for solving the sub-problem (4.6) when the objective function is smooth.

Lemma 14 (Warm start for primal methods -smooth case). If f is differentiable with L-Lipschitz continuous gradients, we can initialize the method M with w 0 = x for solving (4.6). Then, we have the guarantee that

h(w 0 ) -h * ≤ L + κ 2κ 2 ∥∇F (x)∥ 2 . (4.18)
Proof. Denote by w * the minimizer of h. Then, we have the optimality condition ∇f (w * ) + κ(w *x) = 0. As a result,

h(w 0 ) -h * = f (x) - ( f (w * ) + κ 2 ∥w * -x∥ 2 ) ≤ f (w * ) + ⟨∇f (w * ), x -w * ⟩ + L 2 ∥x -w * ∥ 2 - ( f (w * ) + κ 2 ∥w * -x∥ 2 ) = L + κ 2 ∥w * -x∥ 2 = L + κ 2κ 2 ∥∇F (x)∥ 2 .
The inequality in the proof of Lemma 14 relies on the smoothness of f , which does not hold for composite problems. The next lemma addresses this issue.

Lemma 15 (Warm start for primal methods -composite case). Consider the composite optimization problem (4.1). We may initialize the method M for solving (4.6) with

w 0 = arg min w∈R p { f 0 (x) + ⟨∇f 0 (x), w -x⟩ + L + κ 2 ∥w -x∥ 2 + ψ(w) } . (4.19) Then, h(w 0 ) -h * ≤ L 2κ 2 ∥∇F (x)∥ 2 . Proof.
We use the inequality corresponding to Lemma 2.3 in [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]: for any w,

h(w) -h(w 0 ) ≥ L ′ 2 ∥w 0 -x∥ 2 + L ′ ⟨w 0 -x, x -w⟩, (4.20)
with L ′ = L + κ. Then, we apply this inequality to w = w * , and

h(w 0 ) -h * ≤ - L ′ 2 ∥w 0 -x∥ 2 -L ′ ⟨w 0 -x, x -w * ⟩ ≤ L ′ 2 ∥x -w * ∥ 2 = L + κ 2κ 2 ∥∇F (x)∥ 2 .
The lemmas presented above are important, since they allow us to derive the global complexity of QuickeNing in the scenarios described in Propositions 12, 13, and 15. Specifically, it shows us that the conditions in both cases will be satisfied in a constant number of iterations of the method M.

Proposition 16 (Inner-loop complexity for Algorithm 5). Assume that f is differentiable with L-Lipschitz continuous gradients. Consider Algorithm 5 with the warm start strategy described in Lemma 14 or in Lemma 15. Assume that the primal method M applied in the inner loop produces a sequence (w t ) t≥0 for each sub-problem (4.6) such that

h(w t ) -h * ≤ C M (1 -τ M ) t (h(w 0 ) -h * ) for some constants C M , τ M > 0.
(4.21)

Then, the sub-problems are solved with enough accuracy to apply Propositions 12, 13, and 15 in at most T M iterations with

T M = 1 τ M log ( 38C M L + κ κ ) .
Proof. The proposition is a direct consequence of the warm start strategy in previous lemmas. Consider at iteration k, we want to approximate the proximal mapping according to x k . With the given T M (which we abbreviate by T ), we have

h(w T ) -h * ≤ C M (1 -τ M ) T (h(w 0 ) -h * ) ≤ C M e -τ M T (h(w 0 ) -h * ) ≤ C M e -τ M T L + κ 2κ 2 ∥∇F (x k )∥ 2
(By Lemma 14 and Lemma 15)

= 1 76κ ∥∇F (x k )∥ 2 .
It remains to show that this accuracy is in fact small enough to apply Propositions 12, 13, and 15. From Lemma 12, we know that ∥g k ∥ and ∥∇F (x k )∥ are related through:

1 2 ∥∇F (x k )∥ 2 ≤ ∥g k ∥ 2 + 2κε k ≤ ∥g k ∥ 2 + 1 38 ∥∇F (x k )∥ 2 . Thus h(w T ) -h * ≤ 1 76κ ∥∇F (x k )∥ 2 ≤ 1 36κ ∥g k ∥ 2 .
This is exactly the accuracy required in Proposition 13. From which we easily derived the accuracy in Propositions 12 and 15.

Global complexity of QuickeNing

Finally, we can use the previous result to upper-bound the complexity of the QuickeNing algorithm in terms of iterations of the method M for minimizing f up to ε.

Proposition 17 (Worst-case global complexity for Algorithm 5). Under the setting of Proposition 16, the number of iterations of the method M (or expected number if M is non-deterministic) to guarantee the optimality condition f (z k )f * ≤ ε is

• for µ-strongly-convex problems:

2T M × µ + κ µ log ( f (x 0 ) -f * ε ) = 2 µ + κ τ M µ log ( f (x 0 ) -f * ε ) log ( 38C M L + κ κ ) .
• for convex but not strongly convex problems:

2T M × 2κR 2 ε = 4κR 2 τ M ε log ( 38C M L + κ κ ) .
Proof. Since in the worst case, there are two proximal problems to be solved at each outer-loop iteration, the total number of calls of method M is at most 2T M times the number of outer-loop iterations. The result follows immediately from Propositions 12 and 15.

We give below the worst-case global complexity of QuickeNing when applied to two optimization methods M of interest. As we shall see, in terms of worst-case complexity bounds, QuickeNing does not lead to an improved convergence rate. It is worthwhile to underline, though, that this result is not surprising since it is often the case for L-BFGS-type methods, for which an important gap remains between theory and practice. When comparing the vanilla L-BFGS algorithm with the gradient descent method, one outperforms the other in many practical cases, but never in theory.

Proposition 17 and its application to the two examples below show that, in terms of worsecase complexity, the QuickeNing scheme leaves the convergence rate unchanged. Linearlyconverging methods remain linear-converging when used within the QuickeNing scheme.

Example 1. Consider gradient descent with fixed constant step-size 1/L as the optimization method M. Gradient descent (GD) minimizes f to ε accuracy in L/µ log(1/ε) iterations. Quickening-GD minimizes f to ε accuracy in 2(L + κ)/µ log(1/ε) iterations (when ignoring logarithmic terms).

Example 2. Consider the stochastic variance-reduced gradient (SVRG) as the optimization method M. SVRG minimizes f to ε accuracy in

O ( max { n, L µ } log ( 1 ε ) )
Choice of method M. We consider the acceleration of the proximal SVRG algorithm [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF] since it is able to adapt to the problem structure we consider: large sum of functions and composite regularization. We also consider accelerating the ISTA algorithm [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], which allow us to perform a comparison with the natural baselines FISTA and L-BFGS for smooth problems.

Stopping criteria for the inner loop. The default stopping criteria is to solve the subproblem until accuracy ε k such that ε k ≤ 1 36κ ∥g k ∥ 2 , which reduces to the simple criterion (4.16). Although we have shown that such accuracy is attainable in constant T M iteration, the theoretical value of T M could be very large depending on the conditioning of the problem. A natural heuristic is to always perform exactly one pass over the data in the inner loop without checking any stopping criteria as proposed in [START_REF] Lin | A universal catalyst for first-order optimization[END_REF].

When applying QuickeNing to SVRG and ISTA, we call the one-pass variant QuickeNing-SVRG1 and QuickeNing-ISTA1, respectively, and the one using stopping criterion (4.16) QuickeNing-SVRG2 and QuickeNing-ISTA2.

stopping criterion for min w h(w)

QuickeNing1

one-pass through the data.

QuickeNing2 stop when h(w t ) -h * ≤ κ 36 ∥w t -x k ∥ 2 .
We upper bound the duality gap h(w t )h * either by evaluating the Fenchel conjugate or by computing the squared norm of the gradient. Finally, another variant that could be considered is to give a pre-defined sequence of (ε k ) k≥0 and stop when the current accuracy is smaller than ε k , which is checked by using Fenchel conjugate; typically, this variant is less practical and is thus omitted from the experiments.

Choice of regularization parameter κ.

As the global complexity is a function of κ, the most intuitive choice is to choose κ to minimize the complexity. However, such analysis leads to κ = 0, which is not quite helpful since this suggests directly performing M on the original problem. The reason is that the convergence of L-BFGS is hard to characterize and theoretical rates of convergence can be pessimistic. Noting that for smooth functions, L-BFGS often outperforms Nesterov's accelerated gradient method, it is reasonable to expect QuickeNing achieves a similar complexity bound as Catalyst.

Choose κ to maximize τ M √ µ + κ , Therefore, we choose κ identical as in Catalyst algorithm, which leads to κ = L for gradient descent and κ = L/2n for SVRG. Later in this section, a comparison of different scaling of κ is performed in order to demonstrate the relevance of this strategy.

Choice of limited memory parameter l. The default setting is l = 100. We show a comparison with smaller values to study the influence of this parameter.

Sufficient decrease condition.

We use the sufficient decrease condition suggested by the theory:

• for strongly convex objectives, we use

F test ≤ F k -1 2κ ∥g k ∥ 2 ;
• for convex, but non-strongly convex objectives, we use f (x k+1 ) ≤ f (z k ). Nevertheless, we will also evaluate experimentally the strategy for strongly-convex objectives in this context.

Comparison metric.

For all experiments, we use the number of gradient evaluations as a natural metric, assuming this is the computational bottleneck of all methods considered. This is indeed the case here since the L-BFGS step cost O(dl) floating-point operations [START_REF] Nocedal | Numerical optimization[END_REF], whereas evaluating the gradient of the full objective costs O(nd), with d ≤ n and l ≪ d. A similar choice was made in other works [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF].

QuickeNing-SVRG for minimizing large sums of functions

We now apply QuickeNing to SVRG and compare different variants.

• SVRG: the Prox-SVRG algorithm of [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF] with default parameters m = 1 and η = 1/L, where L is the upper-bound on Lipschitz constant of the gradient, as described in the Section 4.5.2.

• Catalyst-SVRG: The catalyst meta-algorithm of [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] applied to Prox-SVRG;

• L-BFGS (for smooth objectives): Since implementing effectively L-BFGS with a linesearch algorithm is a bit involved, we use the implementation of Mark Schmidt2 , which has been widely used in other comparisons [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF]. The limited memory parameter l is also set to 100. We use it for the logistic regression experiment since L-BFGS is defined only for smooth objectives. .1 -Experimental study of the performance of QuickeNing-SVRG for minimizing large sums of functions. We plot the value F (x k )/F * -1 as a function of the number of gradient evaluations, on a logarithmic scale; the optimal value F * is estimated with a duality gap.

• QuickeNing-SVRG1 as detailed above. For the Lasso problem, we also evaluate this heuristic with the sufficient decrease condition used for strongly convex problems. We call this heuristic variant QuickeNing-SVRG1*.

• QuickeNing-SVRG2, as detailed above.

The result of the comparison is presented in Figure 4.1 and leads to the conclusions below, showing that QuickeNing-SVRG1 is a safe heuristic, which never decreases the speed of the method SVRG:

• L-BFGS is less competitive than other approaches that exploit the sum structure of the objective, except on the dataset real-sim; the difference in performance with the SVRGbased approaches can be very important (see dataset alpha).

• QuickeNing-SVRG1 is significantly faster than or on par with SVRG and QuickeNing-SVRG2.

• QuickeNing-SVRG2 is significantly faster than, or on par with, or only slightly slower than SVRG.

• QuickeNing-SVRG1 is significantly faster, or on par with Catalyst-SVRG except in two cases where Catalyst-SVRG performs slightly better.

• there is no clear conclusion regarding the performance of QuickeNing-SVRG1* vs QuickeNing-SVRG1 for the Lasso problem.

QuickeNing-ISTA and comparison with L-BFGS

The previous experiments have included a comparison between L-BFGS and approaches that are able to exploit the sum structure of the objective. It is then interesting to study the behavior of QuickeNing when applied to a basic proximal gradient descent algorithm such as ISTA. Specifically, we now consider

• ISTA: the classical proximal gradient descent algorithm ISTA [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] with back-tracking line-search to automatically adjust the Lipschitz constant of the gradient objective;

• FISTA: the accelerated variant of ISTA from [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF].

• L-BFGS, QuickeNing-ISTA1, QuickeNing-ISTA2, and QuickeNing-ISTA1*, as in the previous section when replacing SVRG by ISTA.

The results are reported in Figure 4.2 and lead to the following conclusions

• L-BFGS is slightly better on average than QuickeNing-ISTA1 for smooth problems, which is not surprising since we use a state-of-the-art implementation with a wellcalibrated line search.

• QuickeNing-ISTA1 is always significantly faster than, or on par with QuickeNing-ISTA2.

• The QuickeNing-ISTA approaches are significantly faster than FISTA in 15 cases out of 16.

• for Lasso problems, QuickeNing-ISTA1* is sometimes faster than QuickeNing-ISTA1. We plot the value F (x k )/F * -1 as a function of the number of gradient evaluations, on a logarithmic scale; the optimal value F * is estimated with a duality gap.

Experimental study of hyper-parameters l and κ

In this section, we study the influence of the limited memory parameter l and of the regularization parameter κ in QuickeNing. More precisely, we start with the parameter l and try the method QuickeNing-SVRG1 with the values l = 1, 2, 5, 10, 20, 100. Note that all previous experiments were conducted with l = 100, which is the most expensive in terms of memory and computational cost for the L-BFGS step. The results are presented in Figure 4.3. Interestingly, the experiment suggests that having a large value for l is not necessarily the best choice, especially for composite problems where the solution is sparse, where l = 5 seems to be a reasonable choice in practice.

The next experiment consists of studying the robustness of QuickeNing to the smoothing parameter κ. We present in . κ 0 denotes the default choice used in the previous experiments. We plot the value F (x k )/F * -1 as a function of the number of gradient evaluations, on a logarithmic scale; the optimal value F * is estimated with a duality gap. i = -3, -2, . . . , 2, 3, where κ 0 is the default parameter that we used in the previous experiments. The conclusion is clear: QuickeNing clearly slows down when using a larger smoothing parameter than κ 0 , but it is very robust to small values of κ (and in fact it even performs better for smaller values than κ 0 in one experiment).

Discussions and concluding remarks

A few questions naturally arise regarding the QuickeNing scheme: one may wonder whether or not our convergence rates may be improved, or if the Moreau-Yosida regularization could be replaced by another smoothing technique. In this section, we discuss these two points and 116 4.6. DISCUSSIONS AND CONCLUDING REMARKS present concluding remarks.

Discussion of convergence rates

In this chapter, we have established the linear convergence of QuickeNing for strongly convex objectives when sub-problems are solved with enough accuracy. Since QuickeNing uses Quasi-Newton steps, one might have expected a superlinear convergence rate as several Quasi-Newton algorithms often enjoy [START_REF] Byrd | Global convergence of a case of quasi-Newton methods on convex problems[END_REF]. The situation is as follows. Consider the BFGS Quasi-Newton algorithm (without limited memory). As shown in [START_REF] Chen | Proximal quasi-Newton methods for nondifferentiable convex optimization[END_REF], if the sequence (ε k ) k≥0 decreases super-linearly, then, it is possible to design a scheme similar to QuickeNing that indeed enjoys a super-linear convergence rate. There are two major downsides though. The scheme of [START_REF] Chen | Proximal quasi-Newton methods for nondifferentiable convex optimization[END_REF] with such a fast rate requires performing a line-search on F and a super-linearly decreasing sequence (ε k ) k≥0 implies an exponentially growing number of iterations in the inner-loops. These two issues make this approach impractical.

Another potential strategy for obtaining a faster convergence rate consists in interleaving a Nesterov-type extrapolation step in the QuickeNing algorithm. Indeed, the convergence rate of QuickeNing scales linearly in the condition number µ F /L F , which suggests that a faster convergence rate could be obtained using a Nesterov-type acceleration scheme. Empirically, we did not observe any benefit of such a strategy, probably because of the pessimistic nature of the convergence rates that are typically obtained for Quasi-Newton approaches based on L-BFGS. Obtaining a linear convergence rate for an L-BFGS algorithm is still an important sanity check, but to the best of our knowledge, the gap in performance between these worst-case rates and practice has always been huge for this class of algorithms.

Other types of smoothing

Algorithm 6 (ApproxGradient) corresponds to applying the Moreau-Yosida regularization first before computing an estimate g of the gradient, which is a particular instance of infimal convolution smoothing [START_REF] Beck | Smoothing and first order methods: A unified framework[END_REF], whose family also encompasses the so-called Nesterov smoothing [START_REF] Beck | Smoothing and first order methods: A unified framework[END_REF]. Other ways to smooth a function include randomization techniques [START_REF] Duchi | Randomized smoothing for stochastic optimization[END_REF] or specific strategies tailored for the objective at hand.

One of the main purposes of the Moreau-Yosida regularization is to provide a better conditioning. As seen in Proposition 1, the gradient of the Moreau-Yosida-smoothed function F is Lipschitz continuous regardless of whether the original function is continuously differentiable or not. Furthermore, the conditioning of F is improved with respect to the original function, with a condition number depending on the amount of smoothing. As highlighted in [START_REF] Beck | Smoothing and first order methods: A unified framework[END_REF], this property is also shared by other types of infimal convolutions. Therefore, QuickeNing could potentially be extended to such types of smoothing in place of the Moreau-Yosida regularization. A major advantage of our approach, though, is its outstanding simplicity.

Concluding remarks

To conclude, we have proposed a generic mechanism, QuickeNing, to accelerate existing firstorder optimization algorithms with quasi-Newton-type rules to update a variable metric along the iterations. QuickeNing's main features are the compatibility with composite optimization and its practical performance when combined with incremental approaches. The absence of linesearch scheme makes it also easy to implement and use, making it a promising tool for solving large-scale machine learning problems. A few questions remain however open regarding the use of the method in a pure stochastic optimization setting, and the gap in performance between worst-case convergence analysis and practice is significant. We are planning to address the first question about stochastic optimization in future work; the second question is unfortunately difficult and is probably one of the main open question in the literature about L-BFGS methods.

Introduction

We consider optimization problems of the form

min x∈R p { f (x) := f 0 (x) + ψ(x) } , where f 0 (x) := 1 n n ∑ i=1 f i (x) .
(5.1)

Here each function f i : R p → R is smooth, the regularization ψ : R p → R may be nonsmooth, and R := R ∪ {∞}. By considering extended-real-valued functions, this composite setting also encompasses constrained minimization by letting ψ be the indicator function of the constraints on x. Minimization of a regularized empirical risk objective of form (5.1) is central in machine learning. Whereas a significant amount of work has been devoted to this setting for convex problems, leading in particular to fast incremental algorithms [see, e.g., Defazio et al., 2014a[START_REF] Lan | An optimal randomized incremental gradient method[END_REF][START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF][START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF][START_REF] Woodworth | Tight complexity bounds for optimizing composite objectives[END_REF][START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF], the question of minimizing efficiently (5.1) when the functions f i and ψ may be nonconvex is still largely open today.

Yet, nonconvex problems in machine learning are of high interest. For instance, the variable x may represent the parameters of a neural network, where each term f i (x) measures the fit between x and a data point indexed by i, or (5.1) may correspond to a nonconvex matrix factorization problem (see Section 5.6). Besides, even when the data-fitting functions f i are convex, it is also typical to consider nonconvex regularization functions ψ, for example for feature selection in signal processing [START_REF] Hastie | Statistical Learning With Sparsity: The Lasso And Generalizations[END_REF]. In this work, we address two questions from nonconvex optimization:

1. How to apply a method for convex optimization to a nonconvex problem?

2. How to design an algorithm which does not need to know whether the objective function is convex while obtaining the optimal convergence guarantee if the function is convex? Several pioneering works attempted to transfer ideas from the convex world to the nonconvex one, see, e.g., [Ghadimi andLan, 2016, Ghadimi et al., 2015]. Our work has a similar goal and studies the extension of Nesterov's acceleration for convex problems [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF] to nonconvex composite ones. For nonconvex and L-smooth problems, gradient descent is optimal among first-order methods in terms of information-based complexity to find an ε-stationary point [Carmon et al., 2017][Thm. 2 Sec. 5]. Without additional assumptions on the function f , worst case complexity for first-order methods can not achieve better than O(ε -2 ) oracle queries [START_REF] Cartis | On the complexity of steepest descent, newton's and regularized newton's methods for nonconvex unconstrained optimization problems[END_REF][START_REF] Cartis | On the complexity of finding first-order critical points in constrained nonlinear optimization[END_REF]. Under a stronger assumption that the objective function is C 2smooth, state-of-the-art methods [e.g., [START_REF] Carmon | Accelerated methods for non-convex optimization[END_REF]Carmon et al., , 2017] ] only achieve marginal gain with complexity O(ε -7/4 log(1/ε)), and do not appear to generalize to composite or stochastic settings. For this reason, our work fits within a broader stream of recent research on methods that do not perform worse than gradient descent in the nonconvex case (in terms of worst-case complexity), while automatically accelerating for minimizing convex functions. The hope is to see acceleration in practice for non-convex problems, by exploiting "hidden" convexity in the objective (e.g., local convexity near the optimum, or convexity along the trajectory of iterates).

INTRODUCTION

Our main contribution is a generic meta-algorithm, dubbed 4WD-Catalyst, which is able to use an optimization method M, originally designed for convex problems, and turn it into an accelerated scheme that also applies to nonconvex objective functions. The proposed 4WD-Catalyst can be seen as a 4-Wheel-Drive extension of Catalyst [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] to all optimization "terrains". Specifically, without knowing whether the objective function is convex or not, our algorithm may take a method M designed for convex optimization problems with the same structure as (5.1), e.g., SAGA [Defazio et al., 2014a], SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF], and apply M to a sequence of sub-problems such that it provides a stationary point of the nonconvex objective. Overall, the number of iterations of M to obtain a gradient norm smaller than ε is Õ(ε -2 ) in the worst case, while automatically reducing to Õ(ε -2/3 ) if the function is convex.1 Related work. Inspired by Nesterov's fast gradient method for convex optimization [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF], the first accelerated method performing universally well for nonconvex and convex problems was introduced in [START_REF] Ghadimi | Accelerated gradient methods for nonconvex nonlinear and stochastic programming[END_REF]. Specifically, [START_REF] Ghadimi | Accelerated gradient methods for nonconvex nonlinear and stochastic programming[END_REF] addresses composite problems such as (5.1) with n = 1, and, provided the iterates are bounded, performs no worse than gradient descent on nonconvex instances with complexity O(ε -2 ) on the gradient norm. When the problem is convex, it accelerates with complexity O(ε -2/3 ). Extensions to Gauss-Newton methods were also recently developed in [START_REF] Drusvyatskiy | Efficiency of minimizing compositions of convex functions and smooth maps[END_REF]. To the best of our knowledge, convergence guarantees for accelerated gradient methods directly applied to nonconvex problem are unknown; however their performance escaping saddle points faster than gradient descent has been observed [START_REF] O'neill | Behavior of Accelerated Gradient Methods Near Critical Points of Nonconvex Problems[END_REF]Wright, 2017, Jin et al., 2017].

In [START_REF] Li | Accelerated proximal gradient methods for nonconvex programming[END_REF], a similar strategy is proposed, focusing instead on convergence guarantees under the so-called Kurdyka-Łojasiewicz inequality-a property corresponding to polynomial-like growth of the function, as shown by [START_REF] Bolte | From error bounds to the complexity of first-order descent methods for convex functions[END_REF]. Our scheme is in the same spirit as these previous papers, since it monotonically interlaces proximal-point steps (instead of proximal-gradient as in [START_REF] Ghadimi | Generalized Uniformly Optimal Methods for Nonlinear Programming[END_REF]) and extrapolation/acceleration steps. A fundamental difference is that our method is generic and accommodates inexact computations, since we allow the subproblems to be approximately solved by any method we wish to accelerate.

By considering C 2 -smooth nonconvex objective functions f with Lipschitz continuous gradient ∇f and Hessian ∇ 2 f , the authors of [START_REF] Carmon | Accelerated methods for non-convex optimization[END_REF] propose an algorithm with complexity O(ε -7/4 log(1/ε)), based on iteratively solving convex subproblems closely related to the original problem. It is not clear if the complexity of their algorithm improves in the convex setting. Note also that the algorithm proposed in [START_REF] Carmon | Accelerated methods for non-convex optimization[END_REF] is inherently for C 2smooth minimization. This implies that the scheme does not allow incorporating nonsmooth regularizers and cannot exploit finite sum structure.

Finally, a method related to SVRG [START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF] for minimizing large sums, while automatically adapting to the weak convexity constant of the objective function, is pro-posed in [START_REF] Allen-Zhu | Natasha: Faster stochastic non-convex optimization via strongly non-convex parameter[END_REF]. When the weak convexity constant is small (i.e., the function is nearly convex), the proposed method enjoys an improved efficiency estimate. This algorithm, however, does not automatically accelerate for convex problems, in the sense that the rate is slower than O(ε -3/2 ) in terms of target accuracy ε on the gradient norm.

Organization of the chapter. Section 5.2 presents mathematical tools for non-convex and non-smooth analysis. Section 5.3 introduces our algorithm, while Section 5.5 presents global convergence guarantees of the scheme and convergence guarantees when the algorithm wraps specific algorithms such as SAGA and SVRG. Section 5.6 is devoted to experiments on neural networks and matrix factorization.

Tools for nonconvex and nonsmooth optimization

Convergence results for nonsmooth optimization typically rely on the concept of subdifferential, which does not admit a unique definition in a nonconvex context [START_REF] Borwein | Convex analysis and nonlinear optimization: theory and examples[END_REF]. In this chapter, we circumvent this issue by focusing on a broad class of nonconvex functions known as weakly convex or lower C 2 functions, for which all these constructions coincide. Weakly convex functions cover most of the interesting cases of interest in machine learning and resemble convex functions in many aspects.

Definition 5 (Weak convexity). A function f : R p → R is ρ-weakly convex if for any points x, y ∈ R p and λ ∈ [0, 1], the approximate secant inequality holds:

f (λx + (1 -λ)y) ≤ λf (x) + (1 -λ)f (y) + ρλ(1 -λ) 2 ∥x -y∥ 2 .
Notice that ρ-weak convexity with ρ = 0 is exactly the definition of a convex function and it is easy to show that f is ρ-weakly convex if and only if the function x → f (x) + ρ 2 ∥x∥ 2 is convex. In particular, a C 1 -smooth function f is ρ-weakly convex if the gradient ∇f is ρ-Lipschitz, while a C 2 -smooth function f is ρ-weakly convex if and only if ∇ 2 f (x) ⪰ -ρI for all x. This resembles an equivalent condition for C 2 -smooth and µ-strongly convex functions, namely ∇ 2 f (x) ⪰ µI.

Useful characterizations of ρ-weakly convex functions rely on differential properties. Since the functions we consider here are nonsmooth, we use a generalized derivative construction. We mostly follow the standard monograph on the subject by Rockafellar and Wets [START_REF] Rockafellar | Variational analysis, volume 317 of Grundlehren der Mathematischen Wissenschaften[END_REF]].

Definition 6 (Subdifferential). Consider a function f : R p → R and a point x with f (x) finite. The subdifferential of f at x is the set

∂f (x) := {v ∈ R p : f (y) ≥ f (x) + v T (y -x) + o(∥y -x∥) ∀y ∈ R p }.

THE 4WD-CATALYST ALGORITHM

Thus, a vector v lies in ∂f (x) whenever the linear function y → f (x) + v T (yx) is a lower-model of f , up to first-order around x. In particular, the subdifferential ∂f (x) of a differentiable function f is the singleton {∇f (x)}, while for a convex function f it coincides with the subdifferential in the sense of convex analysis [see Rockafellar and Wets, 1998, Exercise 8.8]. It is useful to keep in mind that the sum rule, ∂(f + g)(x) = ∂f (x) + ∇g(x), holds for any differentiable function g.

In non-convex optimization, standard complexity bounds are derived to guarantee dist ( 0, ∂f (x)

) ≤ ε .
When ε = 0, we are at a stationary point and first-order optimality conditions are satisfied; that is, the directional derivative at the point is nonnegative in every direction. For functions that are nonconvex, first-order methods search for points with small subgradients, which does not necessarily imply small function values, in contrast to convex functions where the two criteria are much closer related.

The 4WD-Catalyst algorithm

We present here our main algorithm called 4WD-Catalyst. The proposed approach extends the Catalyst method [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] to potentially nonconvex problems, while enjoying the two following properties: 1. When the problem is non convex, the algorithm automatically adapts to the unknown weak convexity constant ρ.

2. When the problem is convex, the algorithm automatically accelerates in the sense of Nesterov, providing near-optimal convergence rates for first-order methods.

Main goal. As in the regular Catalyst algorithm of [START_REF] Lin | A universal catalyst for first-order optimization[END_REF], the proposed scheme wraps in an outer loop a minimization algorithm M used in an inner loop. The goal is to leverage a method M that is able to exploit the problem structure (finite-sum, composite) in the convex case, and benefit from this feature when dealing with a new problem with unknown convexity; remarkably, M does not need to have any convergence guarantee for nonconvex problems to be used in 4WD-Catalyst.

Two-step subproblems. In each iteration, 4WD-Catalyst forms subproblems of the form

min x f κ (x; y) := f (x) + κ 2 ∥x -y∥ 2 . (P)
We call y the prox-center of the subproblem and any minimizer a proximal point. The perturbed function f κ (x; y) satisfies the important property: f κ (•; y) is (κρ)-strongly convex for any κ > ρ. The addition of the quadratic to f makes the subproblem more "convex". That is,

123

CHAPTER 5. CATALYST FOR NON-CONVEX OPTIMIZATION when f is non convex, a large enough κ yields convex subproblems; even when f is convex, the quadratic perturbation nonetheless improves conditioning. We now describe the k'th iteration of Algorithm 8. To this end, suppose we have available iterates x k-1 and v k-1 . At the center of our Algorithm 8 are two main sequences of iterates (x k ) k and (x k ) k , obtained from approximately solving two subproblems of the form P.

1. Proximal point step. We first perform an inexact proximal point step with prox-center x k-1 :

xk ≈ arg min x f κ (x; x k-1 ) [Proximal-point step]
(5.2)

2. Accelerated proximal point step. Then we build the next prox-center y k as the combination

y k = α k v k-1 + (1 -α k )x k-1 .
(5.3)

Next we use y k as a prox-center and update the next extrapolation term:

xk ≈ arg min x f κ (x; y k ) [Accelerated proximal-point step]
(5.4)

v k = x k-1 + 1 α k (x k -x k-1 ) [Extrapolation] (5.5)
where α k+1 ∈ (0, 1) is a sequence of coefficients satisfying (1α k+1 )/α 2 k+1 = 1/α 2 k . Essentially, the sequences (α k ) k , (y k ) k , (v k ) k are built upon the extrapolation principles of [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF].

Picking the best. At the end of iteration k, we have two iterates, resp. xk and xk . Following [START_REF] Ghadimi | Accelerated gradient methods for nonconvex nonlinear and stochastic programming[END_REF], we simply choose the best of the two in terms of their objective values, that is we choose x k such that

f (x k ) ≤ min {f (x k ), f (x k )} .
The proposed scheme blends the two steps in a synergistic way, allowing us to recover the near-optimal rates of convergence in both worlds: convex and non-convex. Intuitively, when xk is chosen, it means that Nesterov's extrapolation step "fails" to accelerate convergence.

We present now our strategy to set the parameters of 4WD-Catalyst so that the resulting algorithm a) automatically adapts to the unknown weak convexity constant ρ; b) enjoys a near-optimal rate of convergence in both convex and non-convex settings.

Optimal parameters and adaptation

When κ is large enough, the subproblems become strongly convex; thus globally solvable. Henceforth, we will assume that M satisfies the following natural linear convergence assumption.

OPTIMAL PARAMETERS AND ADAPTATION

Algorithm 7 Auto-adapt (x, κ, T ) input x ∈ R p , method M, κ > 0, number of iterations T .

Repeat Run T iterations of M initializing from z 0 = x, or z 0 = prox ηψ (x -η∇f 0 (x; y)) with η = 1/(L + κ) for composite objectives f = f 0 + ψ with L-smooth function f 0 , to obtain z T ≈ arg min

z∈R p f κ (z; x). If f κ (z T ; x) ≤ f κ (x;
x) and dist(0, ∂f κ (z T ; x)) ≤ κ ∥z T -x∥, then go to output. else repeat with κ → 2κ. output (z T , κ).

Algorithm 8 4WD-Catalyst input Fix a point x 0 ∈ dom f , real numbers κ 0 , κ cvx > 0 and T, S > 0, and an opt. method M. initialization:

α 1 = 1, v 0 = x 0 . repeat for k = 1, 2, . . . 1. Compute (x k , κ k ) = Auto-adapt (x k-1 , κ k-1 , T ).

Compute y

k = α k v k-1 + (1 -α k )x k-
v k = x k-1 + 1 α k (x k -x k-1 ) and α k+1 = √ α 4 k + 4α 2 k -α 2 k 2 .
4. Choose x k to be any point satisfying f

(x k ) = min{f (x k ), f (x k )}.
until the stopping criterion dist

( 0, ∂f (x k ) ) < ε
Linear convergence of M for strongly-convex problems. We assume that for any κ > ρ, there exist A κ ≥ 0 and τ κ ∈ (0, 1) so that the following hold: 1. For any prox-center y ∈ R p , define f * κ (y) = min z f κ (z, y). For any initial point z 0 ∈ R p , the iterates {z t } t≥1 generated by M on the problem min z f κ (z; y) satisfy

dist 2 (0, ∂f κ (z t ; y)) ≤ A κ (1 -τ κ ) t ( f κ (z 0 ; y) -f * κ (y)
) .

(5.7)

125

CHAPTER 5. CATALYST FOR NON-CONVEX OPTIMIZATION 2. The rates τ κ and constants A κ are increasing in κ. When the function is strongly convex, the first condition is equivalent to the error f κ (z k ; y)f * κ (y) decreasing geometrically to zero (convergence in sub-gradient norm and function values are equivalent in this case). If the method is randomized, we allow (5.7) to hold in expectation; see Sec. 4. in Appendix. All algorithms of interest, (e.g., gradient descent, SVRG, SAGA) satisfy these properties.

Adaptation to weak convexity and predefining T . Recall that we add a quadratic to f to make each subproblem convex. Thus, we should set κ > ρ, if ρ were known. On the other other hand, we do not want κ too large, as that may slow down the overall algorithm. In any case, it is difficult to have an accurate estimate of ρ for machine learning problems such as neural networks. Thus, we propose a procedure described in Algorithm 7 to automatically adapt to ρ.

The idea is to fix in advance a number of iterations T , let M run on the subproblem for T iterations, output the point z T , and check if a sufficient decrease occurs. We show that if we set T = Õ(τ -1 L ), where Õ hides logarithmic dependencies in L and A L , where L is the Lipschitz constant of the smooth part of f ; then, if the subproblem were convex, the following conditions would be guaranteed:

1. Descent condition:

f κ (z T ; x) ≤ f κ (x; x); 2. Adaptive stationary condition: dist ( 0, ∂f κ (z T ; x) ) ≤ κ ∥z T -x∥ .
Thus, if either condition is not satisfied, then the subproblem is deemed not convex and we double κ and repeat. The procedure yields an estimate of ρ in a logarithmic number of increases; see lem. D.3.

The descent condition is a sanity check, which ensures the iterates generated by the algorithm always decrease the function value. Without it, the stationarity condition alone is insufficient because of the existence of local maxima in nonconvex problems.

The adaptive stationarity property controls the inexactness of the subproblem in terms of subgradient norm. In a non convex setting, the subgradient norm is convenient, since we cannot access f κ (z T , x)f * κ (x). Furthermore, unlike the stationary condition dist ( 0, ∂f κ (z T ; x)

) < ε, where an accuracy ε is predefined, the adaptive stationarity condition depends on the iterate z T . This turns out to be essential in deriving the global complexity. Sec. 3 in the Appendix contains more details.

Relative stationarity and predefining S. One of the main differences of our approach with the Catalyst algorithm of [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] is to use a pre-defined number of iterations, T and S, for solving the subproblems. We introduce κ cvx , a M dependent smoothing parameter and set it in the same way as the smoothing parameter in [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]. The automatic acceleration of our algorithm when the problem is convex is due to extrapolation steps in Step 2-3 of Algo. 8. We show that if we set S = Õ

( τ -1 κcvx )
, where Õ hides logarithmic dependencies

GLOBAL CONVERGENCE AND APPLICATIONS TO EXISTING ALGORITHMS

in L, κ κ , and A κcvx , then we can be sure that, for convex, (5.8) This relative stationarity of xk , including the choice of κ cvx , shall be crucial to guarantee that the scheme accelerates in the convex setting. An additional k + 1 factor appears compared to the previous adaptive stationary condition because we need higher accuracy for solving the subproblem to achieve the accelerated rate in 1/ √ ε. Therefore, an extra log(k + 1) factor of iterations is needed; see Sec. 3 and Sec. 4 in the Appendix.

dist ( 0, ∂f κcvx (x k ; y k ) ) < κ cvx k + 1 ∥x k -y k ∥ .
We shall see, in Sec. 5.6, that our strategy of predefining T and S works quite well. The theoretical bounds we derive are, in general, too conservative; we observe in our experiments that one may choose T and S significantly smaller than the theory suggests and still retain the stopping criteria.

Global convergence and applications to existing algorithms

With the previous section in mind, we can now present the complexity of our algorithm, which takes into account the cost of approximately solving the subproblems (5.2) and (5.4).

Theorem 9 (Global complexity bounds for 4WD-Catalyst). Choose T = Õ(τ -1 L ) and S = Õ(τ -1 κcvx ) (see Theorem 4.5 in Appendix). Then the following are true. 1. Algorithm 8 generates a point x satisfying dist ( 0, ∂f (x)

) ≤ ε after at most Õ ( ( τ -1 L + τ -1 κcvx ) • L(f (x 0 ) -f * ) ε 2
) iterations of the method M.

2. If f is convex, Algorithm 8 generates a point x satisfying dist ( 0, ∂f (x)

)

≤ ε after at most

Õ   ( τ -1 L + τ -1 κcvx ) • L 1/3 (κ cvx ∥x * -x 0 ∥ 2 ) 1/3 ε 2/3   iterations of the method M. 3. If f is convex, Algorithm 8 generates a point x satisfying f (x) -f * ≤ ε after at most Õ   ( τ -1 L + τ -1 κcvx ) • √ κ cvx ∥x * -x 0 ∥ 2 √ ε   iterations of the method M.
Here Õ hides universal constants and logarithmic dependencies in A κcvx , A L , κ 0 , κ cvx , ε, and ∥x *x 0 ∥ 2 .

If M is a first order method, the convergence guarantee in the convex setting is near-optimal, up to logarithmic factors, when compared to O(1/ √ ε) [Lin et al., 2015, Woodworth and[START_REF] Woodworth | Tight complexity bounds for optimizing composite objectives[END_REF]. In the non-convex setting, our approach matches, up to logarithmic factors, the best known rate for this class of functions, namely O(1/ε2 ) [START_REF] Cartis | On the complexity of steepest descent, newton's and regularized newton's methods for nonconvex unconstrained optimization problems[END_REF][START_REF] Cartis | On the complexity of finding first-order critical points in constrained nonlinear optimization[END_REF]. Moreover, our rates dependence on the dimension and Lipschitz constant equals, up to log factors, the best known dependencies in both the convex and nonconvex setting. These logarithmic factors may be the price we pay for having a generic algorithm.

Choice of κ cvx . The parameter κ cvx drives the convergence rate of 4WD-Catalyst in the convex setting. To determine κ cvx , we compute the global complexity of our scheme as if ρ = 0, hence using the same reasoning as [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]. The rule consists in maximizing the ratio τ κ / √ L + κ. Then, the choice of κ 0 is independent of M; it is an initial lower estimate for the weak convexity constant, ρ. We provide a detailed derivation of all the variables for each of the considered algorithms in the appendix (Sec.5 in Appendix)

Applications

We now compare the guarantees obtained before and after applying 4WD-Catalyst to some specific optimization methods M (full gradient, SAGA, and SVRG). In the convex setting, the accuracy is stated in terms of optimization error, f (x)f * ≤ ε and in the nonconvex setting, in terms of stationarity condition dist(0, ∂f (x)) < ε.

Full gradient method. First, we consider the simplest case of applying our method to the full gradient method (FG). Here, the optimal choice for κ cvx is L. In the convex setting, we get the accelerated rate O(n √ L/ε log(1/ε)) which is consistent with Nesterov's accelerated variant (AFG) up to logarithmic factors. In the nonconvex case, our approach achieves no worse rate than O(nL/ε 2 log(1/ε)), which is consistent with the standard gradient descent up to logarithmic factors. We note that under stronger assumptions, namely C 2 -smoothness of the objective, the accelerated algorithm in [Carmon et al., 2017] achieves the same rate as (AFG) for the convex setting and O(ε -7/4 log(1/ε)) for the nonconvex setting. Their approach, however, does not extend to composite setting nor to stochastic methods. Our marginal loss is the price we pay for considering a much larger class of functions.

Randomized incremental gradient. We now consider randomized incremental gradient methods such as SAGA [Defazio et al., 2014a] and (prox) SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF]. Here, the optimal choice for κ cvx is O(L/n). Under the convex setting, we achieve an accelerated rate of O( √ n √ L/ε log(1/ε)). Direct applications of SVRG and SAGA have no convergence guarantees in the non-convex setting. With our approach, the resulting algorithm matches the guarantees for FG up to log factors.

Table 5.1 -Comparison of rates of convergence, before and after the 4WD-Catalyst , resp. in the non-convex and convex cases. For the comparison, in the convex case, we only present the number of iterations to obtain a point x satisfying f (x)f * < ε. In the non-convex case, we show the number of iterations to obtain a point x satisfying dist(0, ∂f (x)) < ε.

Stepsize Nonconvex Convex FG O ( 1
L ) O ( n L ε 2 ) O ( n L ε ) 4WD-Catalyst-FG O ( 1
L ) Õ ( n L ε 2 ) Õ   n √ L ε   Stepsize Nonconvex Convex SVRG/SAGA 2 O ( 1
L ) not avail. O ( n L ε ) ncvx-SVRG/SAGA 3 O ( 1 √ nL ) O ( n 2/3 L ε 2 ) O ( √ n L ε ) 4WD-Catalyst-SVRG/SAGA O ( 1
L ) Õ ( n L ε 2 ) Õ   √ n √ L ε  

Experiments

We investigate the performance of 4WD-Catalyst in two standard non-convex problems in machine learning. We report experimental results of 4WD-Catalyst when applied to two different algorithms: SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF] and SAGA [Defazio et al., 2014a]. We compare the following algorithms:

• Nonconvex Prox-SVRG/SAGA [Reddi et al., 2016b]: stepsize η = 1/Ln • Convex Prox-SVRG/SAGA [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF]Zhang, 2014, Defazio et al., 2014a] use the logistic loss to fit the estimators to the true labels and report experimental results on the two datasets alpha and covtype. The weights of the network are randomly initialized and we fix the number of hidden neurons to d = 100.

Computational cost. For the Convex-SVRG and Nonconvex-SVRG, one iteration corresponds to one pass over the data in the plots. On the one hand, since 4WD-Catalyst-SVRG solves two sub-problems per iteration, the cost per iteration is twice that of the Convex-SVRG and Nonconvex-SVRG. On the other hand, in the experiments, we observe that, everytime acceleration occurs, then xk is almost always preferred to xk in step 4 of 4WD-Catalyst, half of the computations are in fact not performed when running 4WD-Catalyst-SVRG.

Experimental conclusions. In matrix factorization experiments, we observe that 4WD-Catalyst-SVRG always outperforms the competing algorithms. Nonconvex-SVRG has slower convergence in objective values and Convex-SVRG is not always converging; see in particular right panel in Fig. 5.1. Therefore 4WD-Catalyst-SVRG offers a more stable option than Convex-SVRG for minimizing nonconvex objectives. Furthermore, in these experiments 4WD-Catalyst-SVRG enjoys a faster convergence in objective values. This confirms the remarkable ability of 4WD-Catalyst-SVRG to adapt to nonconvex terrains. Similar conclusions hold when applying 4WD-Catalyst to SAGA; see Sec.6. in Appendix, which demonstrate the genericity of 4WD-Catalyst.

In neural network experiments, we observe that 4WD-Catalyst-SVRG converges much faster in terms of objective values than the competing algorithms. Nonconvex-SVRG with the theoretically recommended sequence of step-sizes [Reddi et al., 2016b] compares unfavorably here, which implies that the recommended step-sizes are too pessimistic hence too small. We also observe an interesting phenomenon: the subgradient norm may increase at some point then decrease, while the function value keeps decreasing, as the algorithm proceeds. This suggests that the extrapolation step, or the Auto-adapt procedure, is helpful to escape bad stationary points, e.g., saddle-points. We leave study of this particular phenomenon as a potential direction for future work.

Chapter 6

Discussions and concluding remarks

In this manuscript, we have introduced several generic acceleration schemes for accelerating gradient-based optimization methods. Our approaches are adapted to large-scale and highdimensional convex problems a with composite finite sum structure. Moreover, one of our methods is extended to non convex formulations, which allows us to apply it to a large range of machine learning problems. In this section, we summarize our contributions and discuss possible directions for future work.

First, we have presented a proximal variant of the Finito/MISO algorithm. The algorithm we developed is able to deal with non smooth regularizations and it removes a constraint on the number of data points required in the original Finito/MISO algorithm. We have shown that our algorithm converges linearly with a similar rate as other variance reduction based incremental algorithms, when the objective function is strongly convex.

Second, we introduce a generic acceleration scheme called Catalyst. It provides acceleration in the sense of Nesterov to a large class of first-order optimization methods. We prove that for both convex and strongly convex problems, a near-optimal convergence rate, up to a logarithmic factor, is achieved by applying Catalyst. We experimentally demonstrate that the acceleration is useful, especially for ill-conditioned problems.

We then present another generic mechanism, QuickeNing, to accelerate existing first-order optimization algorithms with quasi-Newton principles. It is able to exploit the composite structure of the objective function and take into account the curvature information. To the best of our knowledge, QuickeNing is the first generic algorithm enjoying both of these features. We show that it provides significant improvement in practice over competing methods including the Catalyst acceleration. However, in contrast to Catalyst where a theoretical acceleration is guaranteed, QuickeNing does not improve the convergence rate in the worst case analysis. Thus a huge gap remains between the practical performance and the theoretical analysis, which is a common observation for quasi-Newton methods.

Finally, we extend Catalyst acceleration to non convex objective functions. The proposed algorithm does not require to know the function is convex or not and it automatically adapts to the convexity. When the objective is non convex, it converges to a stationary point with a CHAPTER 6. DISCUSSIONS AND CONCLUDING REMARKS complexity not worse than the gradient descent method; when the objective is convex, acceleration will be obtained as the original Catalyst algorithm. We provide experiments on problems of matrix factorization and neural networks to show that our approach converges faster than existing algorithms.

Catalyst is our first attempt to the field of non convex optimization, it is possible to extend QuickeNing in a similar way which we leave to future work. The main difference between convex and non convex optimization is that when the function is convex, all stationary points are global minima; but when the function is non convex, a stationary point may not even be a local minimum. Thus, an important subject in non convex optimization is to study whether an optimization method can escape bad stationary points, e.g., saddle-points. On one hand, positive results have been proven by incorporating Hessian information. [START_REF] Nesterov | Cubic regularization of newton method and its global performance[END_REF] shows that a cubic regularization of Newton method converges to second-order stationary points. Later, algorithms based on Hessian-vector products has been proposed to relax the requirement of the evaluation of Hessian matrix [START_REF] Agarwal | Finding approximate local minima for nonconvex optimization in linear time[END_REF][START_REF] Carmon | Accelerated methods for non-convex optimization[END_REF][START_REF] Allen-Zhu | Natasha: Faster stochastic non-convex optimization via strongly non-convex parameter[END_REF]. On the other hand, it seems difficult to ask the same property for first-order methods since they are blind to second-order information. Amazingly, it is shown that classical gradient descent method with a random initialization converges almost surely to a local minimum [START_REF] Lee | Gradient descent only converges to minimizers[END_REF]. Moreover, with additional perturbed noise, gradient descent method or stochastic gradient descent (SGD) are guaranteed to escape saddle points [START_REF] Ge | Escaping from saddle points-online stochastic gradient for tensor decomposition[END_REF], Jin et al., 2017].

Thus, there may be a hope to prove similar properties for our algorithm since Catalyst can be interpreted as an inexact accelerated proximal point algorithm. In fact, the experimental results in Section 5.6 do suggest our algorithm may be helpful to escape bad stationary points. Indeed, in the experiments for neural networks, we observe that when applying Catalyst, the subgradient norm may increase at some point then decrease, while the function value keeps decreasing. A more systematic study is required to confirm such a observation, which we leave as a potential direction of future work.

A.2. QUASI-NEWTON METHODS

meaning that it suffices to show that H k ∇f (x k ) ∈ G k+1 . By the definition of the BFGS's update,

H k = ( I - s k-1 y T k-1 ⟨y k-1 , s k-1 ⟩ ) L k-1 H k-1 ( I - y k-1 s T k-1 ⟨y k-1 , s k-1 ⟩ ) R k-1 + s k-1 s T k-1 ⟨y k-1 , s k-1 ⟩ , with s k-1 = x k -x k-1 and y k-1 = ∇f (x k ) -∇f (x k-1
). The L k-1 and R k-1 represent the left and right matrix multiplied to H k-1 respectively. Unrolling the recursion gives

H k = L k-1:0 H 0 R 0:k-1 + k-1 ∑ i=0 L k-1:i+1 s i s T i y T i s i R i+1:k-1 , (A.3)
where

L j:i = { L j L j-1 • • • L i if j > i I otherwise , R i:j = { R i R i+1 • • • R j if j > i I otherwise . (A.4)
We remark that for any vector z in R d ,

L j:i z ∈ z + Span{s j , s j-1 , • • • , s i }; and R i:j z ∈ z + Span{y i , y i+1 , • • • , y j };
This immediately shows that

H k ∇f (x k ) ∈ ∇f (x k ) + Span{s 0 , • • • , s k-1 , y 0 , • • • , y k-1 } ⊂ G k+1 ,
where the last inclusion is true because all s i and y i are in G k+1 .

A natural implication of relation (A.3) is the two loop recursion for BFGS update which can be extended to the limited memory variant:

Proof. This lemma is identical to the Lemma A.10 in the original Catalyst paper [START_REF] Lin | A universal catalyst for first-order optimization[END_REF], inspired by a lemma of Schmidt et al. [2011b] for controlling errors of inexact proximal gradient methods.

First, We give here an elementary proof for completeness based on induction. The relation (B.2) is obviously true for k = 0. Then, we assume it is true for k -1 and prove the relation for k. First, we remark that from (B.1),

( u k - a k 2 ) 2 ≤ S k + k-1 ∑ i=1 a i u i + a 2 k 4 ,
and then

u k ≤ S k + k-1 ∑ i=1 a i u i + a 2 k 4 + a k 2 .
We may now prove the relation (B.2) by induction,

S k + k ∑ i=1 a i u i ≤ S k + k-1 ∑ i=1 a i u i + a k   a k 2 + S k + k-1 ∑ i=1 a i u i + a 2 k 4   ≤ S k + k-1 ∑ i=1 a i u i + a k   a k + S k + k-1 ∑ i=1 a i u i   ≤   S k + k-1 ∑ i=1 a i u i + a k   2 =   (S k -S k-1 ) + (S k-1 + k-1 ∑ i=1 a i u i ) + a k   2 ≤     (S k -S k-1 ) + ( √ S k-1 + k-1 ∑ i=1 a i ) 2 + a k     2 (by induction) ≤ ( √ S k + k ∑ i=1 a i ) 2
.

The last inequality is obtained by developing the square

( √ S k-1 + ∑ k-1 i=1 a i ) 2
and use the increasing assumption S k-1 ≤ S k .

Lemma 18 (Growth of the sequence (A k ) k≥0 ). Let (A k ) k≥0 be the sequence defined in (3.15) where (α k ) k≥0 is produced by (3.10) with α 0 = 1 and µ = 0. Then, we have the following bounds for all k ≥ 0, 2 (k + 2) 2 ≤ A k ≤

B.2. PROOFS OF AUXILIARY RESULTS

Proof. The righthand side is directly obtained from Lemma 7 by noticing that γ 0 = κ with the choice of α 0 . Using the recurrence of α k , we have for all k ≥ 1,

α 2 k = (1 -α k )α 2 k-1 = k ∏ i=1 (1 -α i )α 2 0 = A k ≤ 4 (k + 2) 2 .
Thus, α k ≤ 2 k+2 for all k ≥ 1 (it is also true for k = 0). We now have all we need to conclude the lemma:

A k = k ∏ i=1 (1 -α i ) ≥ k ∏ i=1 ( 1 - 2 i + 2 ) = 2 (k + 2)(k + 1) ≥ 2 (k + 2) 2 .

B.2 Proofs of auxiliary results

B.2.1 Proof of lemma 6

Proof. Let us introduce the quantity h

′ (z) ≜ 1 η (z -[z] η ), which is often called "gradient map- ping". Since [z] η is a proximal operator, we have 1 η ((z -η∇h 0 (z)) -[z] η ) ∈ ∂ψ([z] η ).
Therefore, since h = h 0 + ψ, we may define

u ≜ 1 η (z -[z] η ) -(∇h 0 (z) -∇h 0 ([z] η )) ∈ ∂h([z] η ),
and, by strong convexity,

h * ≥ h([z] η ) + u ⊤ (p(x) -[z] η ) + κ 2 ∥p(x) -[z] η ∥ 2 ≥ h([z] η ) - 1 2κ ∥u∥ 2 .
Moreover,

∥u∥ 2 = 1 η (z -[z] η ) 2 - 2 η ⟨z -[z] η , ∇h 0 (z) -∇h 0 ([z] η )⟩ + ∥∇h 0 (z) -∇h 0 ([z] η )∥ 2 ≤ ∥h ′ (z)∥ 2 -∥∇h 0 (z) -∇h 0 ([z] η )∥ 2 ≤ ∥h ′ (z)∥ 2 ≤ (κε) 2 ,
where the first inequality comes from the relation [Nesterov, 2004, Theorem 2.1.5]

η∥∇h 0 (z) -∇h 0 ([z] η )∥ 2 ≤ ⟨∇h 0 (z) -∇h 0 ([z] η ), z -[z] η ⟩, since h 0 is (1/η)-smooth. Thus, h([z] η ) -h * ≤ 1 2κ ∥u∥ 2 ≤ 1 2κ ∥h ′ (z)∥ 2 = 1 2κη 2 ∥z -[z] η ∥ 2 ≤ ε. (B.3)

B.2.2 Proof of Proposition 2

Proof. We simply use Theorem 7 and specialize it to the choice of parameters. The initialization α 0 = √ q leads to a particularly simple form of the algorithm, where α k = √ q for all k ≥ 0.

Therefore, the sequence (A k ) k≥0 from Theorem 7 is also simple since we indeed have A k = (1 -√ q) k . Then, we remark that γ 0 = µ(1 -√ q) and thus, by strong convexity of f ,

S 0 = (1 - √ q) ( f (x 0 ) -f * + µ 2 ∥x 0 -x * ∥ 2 ) ⩽ 2(1 - √ q)(f (x 0 ) -f * ).
Therefore,

√ S 0 + 3 k ∑ j=1 √ ε j A j-1 ⩽ √ 2(1 - √ q)(f (x 0 ) -f * ) + 3 k ∑ j=1 √ ε j A j-1 = √ 2(1 - √ q)(f (x 0 ) -f * )        1 + k ∑ j=1   1 -ρ 1 - √ q   η j        = √ 2(1 - √ q)(f (x 0 ) -f * ) η k+1 -1 η -1 ⩽ √ 2(1 - √ q)(f (x 0 ) -f * ) η k+1 η -1 .
Therefore, Theorem 7 combined with the previous inequality gives us

f (x k ) -f * ⩽ 2A k-1 (1 - √ q)(f (x 0 ) -f * ) ( η k+1 η -1 ) 2 = 2 ( η η -1 ) 2 (1 -ρ) k (f (x 0 ) -f * ) = 2   √ 1 -ρ √ 1 -ρ - √ 1 - √ q   2 (1 -ρ) k (f (x 0 ) -f * ) = 2   1 √ 1 -ρ - √ 1 - √ q   2 (1 -ρ) k+1 (f (x 0 ) -f * ). Since √ 1 -x + x 2 is decreasing in [0, 1], we have √ 1 -ρ + ρ 2 ⩾ √ 1 - √ q + √ q 2 . Consequently, f (x k ) -f * ⩽ 8 ( √ q -ρ) 2 (1 -ρ) k+1 (f (x 0 ) -f * ).

B.2.3 Proof of Proposition 3

Proof. The initialization α 0 = 1 leads to γ 0 = κ and S

0 = κ 2 ∥x * -x 0 ∥ 2 . Then, √ γ 0 2 ∥x 0 -x * ∥ 2 + 3 k ∑ j=1 √ ε j A j-1 ≤ √ κ 2 ∥x 0 -x * ∥ 2 + 3 k ∑ j=1 √ (j + 1) 2 ε j 2 (from Lemma 18) ⩽ √ κ 2 ∥x 0 -x * ∥ 2 + √ f (x 0 ) -f *   k ∑ j=1 1 (j + 1) 1+γ/2   ,
where the last inequality uses Lemma 18 to upper-bound the ratio ε j /A j . Moreover,

k ∑ j=1 1 (j + 1) 1+γ/2 ⩽ ∞ ∑ j=2 1 j 1+γ/2 ⩽ ∫ ∞ 1 1 x 1+γ/2 dx = 2 γ .
Then applying Theorem 7 yields

f (x k ) -f * ⩽ A k-1 ( √ κ 2 ∥x 0 -x * ∥ 2 + 2 γ √ f (x 0 ) -f * ) 2 ⩽ 8 (k + 1) 2 ( κ 2 ∥x 0 -x * ∥ 2 + 4 γ 2 (f (x 0 ) -f * )
)

.

The last inequality uses (a + b) 2 ⩽ 2(a 2 + b 2 ).

B.2.4 Proof of Lemma 8

Proof. We abbreviate τ M by τ and C = C M (h(z 0 )h * ) to simplify the notation. Set

T 0 = 1 τ log ( 1 1 -e -τ C ε ) . For any t ≥ 0, we have E[h(z t ) -h * ] ⩽ C(1 -τ ) t ⩽ C e -tτ .
By Markov's inequality,

P[h(z t ) -h * > ε] = P[T (ε) > t] ⩽ E[h(z t ) -h * ] ε ⩽ C e -tτ ε . (B.4)
Together with the fact P ⩽ 1 and t ≥ 0. We have

P[T (ε) ⩾ t + 1] ⩽ min { C ε e -tτ , 1 } . Therefore, E[T (ε)] = ∞ ∑ t=1 P[T (ε) ⩾ t] = T 0 ∑ t=1 P[T (ε) ⩾ t] + ∞ ∑ t=T 0 +1 P[T (ε) ⩾ t] ⩽ T 0 + ∞ ∑ t=T 0 C ε e -tτ = T 0 + C ε e -T 0 τ ∞ ∑ t=0 e -tτ = T 0 + C ε e -τ T 0 1 -e -τ = T 0 + 1.
As simple calculation shows that for any τ ∈ (0, 1), τ 2 ⩽ 1e -τ and then

E[T (ε)] ⩽ T 0 + 1 = 1 τ log ( 1 1 -e -τ C ε ) + 1 ⩽ 1 τ log ( 2C τ ε ) + 1.
why an additional assumption is usually required for analyzing Quasi-Newton methods such as Lipschitz continuous Hessians.

Let us now introduce the secant equation, which is plays a crucial rule in the construction of Quasi-Newton mtehods. At (k + 1)-th iteration, given the past iterate x k and its Hessian approximation B k , we evaluate the new iterate x k+1 through the Quasi-Newton update

x k+1 = x k -B -1 k ∇f (x k ).
Then, the new Hessian approximation B k+1 at x k+1 is required to satisfy the secant equation,

B k+1 s k = y k , (C.2)
where

s k = x k+1 -x k and y k = ∇f (x k+1 ) -∇f (x k ).
The methods that we are presenting in this section all satisfy the secant equation (C.2), making it the core object to be understood. Many explanations have been given to justify the requirement of the secant equation. One way is to introduce the local quadratic approximation

Q k+1 (x) = f (x k+1 ) + ⟨∇f (x k+1 ), x -x k+1 ⟩ + 1 2 (x -x k+1 ) T B k+1 (x -x k+1 ).
The secant equation is just asking the gradient of Q k+1 to match the gradient of f , at the previous iterate x k . While this interpretation is easy to follow, it does not provide further information about how we can construct B k+1 . Since the secant equation is an underdetermined system which may admit an infinity amount of solutions, itself alone does not uniquely determine B k+1 .

A different way to interpret the secant equation is through the point of view of discretization. More precisely, by the definition of the Hessian matrix,

lim h→0 ∇f (x k+1 + h) -∇f (x k+1 ) -∇ 2 f (x k+1 )h ∥h∥ = 0.
The limit is valid along any direction h. The secant equation (C.2) is nothing else than a discretization of the above limit along the direction h = x kx k+1 = -s k . The reason for choosing this direction is two-fold: first, x k is the closest point to x k+1 among the past iterates, which makes the discretization error small; second, by letting h = -s k , the gradient ∇f (x k+1 + h) becomes ∇f (x k ), which is already evaluated and does not require an additional gradient evaluation. Since the secant equation is the only information we have at iteration k + 1, there is no justification to update B k+1 in other directions which are orthogonal to s k . This leads to the condition

B k+1 z = B k z, ∀⟨z, s k ⟩ = 0. (C.3)
Together with the secant equation (C.2), the matrix B k+1 is uniquely determined, giving the update of (Good) Broyden's method,

B k+1 = B k + (y k -B k s k )s T k ∥s k ∥ 2 . (C.4)
From the variational point of view, the constructed B k+1 is the closest matrix to B k satisfying the secant equation [START_REF] Dennis | Quasi-newton methods, motivation and theory[END_REF]:

Proposition 19. Given B k in L(R d ), y k in R d and a nonzero vector s k ∈ R d , the matrix B k+1 defined in (C.4) is the unique solution of the problem min B∈L(R d ) {∥B -B k ∥ F | Bs k = y k }, (C.5)
where ∥ • ∥ F denotes the Frobenius norm.

The interpretation based on the discretization provides us a more exhaustive point of view about the construction of Quasi-Newton methods. The key idea is to iteratively correct the Hessian estimation by adding new information regarding the current iterate. Broyden's update is the simplest one which does not guarantee the symmetry nor the positive definiteness of the matrix. This may be the reason why the corresponding theoretical result for Broyden's update is weaker than other well known Quasi-Newton methods. When the objective function is quadratic, Broyden's method is globally and superlinearly convergent to the exact minimum under mild modifications, see [START_REF] Moré | On the global convergence of broyden's method[END_REF]].2 Discarding the convergence analysis, let us show that we benefit a reduction in the computational complexity. Especially, we improve the O(d 3 ) operations required for the matrix inversion to O(d 2 ) operations, by applying the Sherman-Morrison formula [START_REF] Sherman | Adjustment of an inverse matrix corresponding to a change in one element of a given matrix[END_REF].

Lemma 19. Let u, v ∈ R d and A ∈ L(R d ) is non singular. Then A + uv T is non singular if and only if σ = 1 + ⟨v, A -1 u⟩ ̸ = 0. If σ ̸ = 0, then (A + uv T ) -1 = A -1 - 1 σ A -1 uv T A -1 . (C.6)
As a consequence, one can implement Broyden's method by considering the update on the inverse of Hessian: set H k = B -1 k , then apply Lemma 19 to (C.4) yields

H k+1 = H k + (s k -H k y k )s T k H k s T k H k y k , (C.7) under the condition s T k H k y k ̸ = 0.
Then, the next iterate can be obtained by performing

x k+2 = x k+1 -H k+1 ∇f (x k+1 ).
The matrix inversion is no longer required which reduces the complexity to O(d 2 ) due to the multiplication of matrix vector. Broyden's method is originally designed for solving nonlinear equations, in which case the underlying structure can be neither symmetric nor positive definite. However, in the unconstrained convex optimization setting, these properties are naturally Intuitively, C 2j+1 is the projection from C 2j to the affine space defined by the solutions of the secant equation, along the "direction" z, and C 2j+2 is the projection of C 2j+1 on the space of symmetric matrices. This is nothing but an alternating projection procedure. It turns out that the sequence C j converge to a limit C ∞ with a rank two update

C ∞ = B + (y -Bs)z T + z(y -
∥A∥ F,W = ∥W 1/2 AW 1/2 ∥ F .
The update given by (C.12) is always symmetric and unintentionally it is a rank two update. Now it suffices to choose an appropriate direction z. By taking z k = s k , the underlying single rank update is Broyden's method, this gives the Powell symmetric Broyden update, called PSB update [START_REF] Powell | A new algorithm for unconstrained optimization[END_REF]:

B PSB k+1 = B k + r k s T k + s k r T k ∥s k ∥ 2 - ⟨r k , s k ⟩s k s T k ∥s k ∥ 4 , (C.14)
where r k = y k -B k s k is the residual vector. Unfortunately, the PSB update does not necessarily maintain the positive definiteness (P). This can be verified by checking the determinant of the matrix. Amazingly, a positive determinant is in fact a sufficient condition to guarantee the positive definiteness (P) of the update (C.12). The main reason is that the update can be rewritten as

B k+1 = B k + aa T -bb T ,
for some vectors a, b ∈ R d . Given that B k is positive definite, B k+1 has at most one negative eigenvalue, meaning that the positive determinant is sufficient to ensure the positive definiteness.

To make this reasoning rigorous, a result from perturbation theory is required [START_REF] Wilkinson | The algebraic eigenvalue problem[END_REF]:

Lemma 20. Let A ∈ L(R d ) be symmetric with eigenvalues λ 1 ≤ λ 2 ≤ • • • ≤ λ d . Let u ∈ R d be a non zero vector and B = A + σuu T . Then if σ ≥ 0, then the eigenvalues of B satisfies λ 1 ≤ λ B 1 ≤ λ 2 ≤ • • • ≤ λ d ≤ λ B d ; otherwise if σ < 0, then the eigenvalues of B satisfies λ B 1 ≤ λ 1 ≤ λ B 2 ≤ • • • ≤ λ B d ≤ λ d .
From the expression (C.12), one can derive that

det(B k+1 ) = det(B k ) ⟨z k , s k ⟩ 2 [ ⟨z k , y k ⟩ 2 H k -∥z k ∥ 2 H k ∥y k ∥ 2 H k + ∥z k ∥ 2 H k ⟨y k , s k ⟩ ] , (C.15)
where

H k = B -1 k and ⟨u, v⟩ H k = u T H k v. Thus the determinant is positive is equivalent to ∥z k ∥ 2 H k ⟨y k , s k ⟩ > ∥z k ∥ 2 H k ∥y k ∥ 2 H k -⟨z k , y k ⟩ 2 H k . (C.16)
For the PSB update, we have z k = s k and the inequality does not always hold because it depends essentially on the spectrum of the metric H k = B -1 k . In order to make the update inherit the positive definiteness, a simple choice is to set z k = y k , in which case the righthand side vanishes. This gives the Davidon-Fletcher-Powell (DFP) update [Davidon, 1991, Fletcher and[START_REF] Fletcher | A rapidly convergent descent method for minimization[END_REF],

B DFP k+1 = B k + (y k -B k s k )y T k + y k (y k -B k s k ) T ⟨y k , s k ⟩ - ⟨y k -B k s k , s k ⟩y k y T k ⟨y k , s k ⟩ 2 = ( I - y k s T k ⟨y k , s k ⟩ ) B k ( I - s k y T k ⟨y k , s k ⟩ ) + y k y T k ⟨y k , s k ⟩ . (C.17) Thus, B DF P k+1 is positive definite if B k is positive definite and ⟨y k , s k ⟩ > 0. Remind that y k = ∇f (x k+1 ) -∇f (x k ) and s k = x k+1 -x k , then ⟨y k , s k ⟩ > 0 holds
whenever the function is strongly convex. Finally, to get rid of the matrix inversion, we apply the Sherman-Morrison formula twice, leading to the expression

H DFP k+1 = H k + s k s T k ⟨s k , y k ⟩ - H k y k y T k H k y T k H k y k . (C.18)
So far, we have been focusing on approximating the Hessian matrix and apply the Sherman-Morrison formula to develop the inverse Hessian update. Indeed, it is also natural to directly approximate the inverse of the Hessian matrix H. In this case, the secant equation B k+1 s k = y k transforms to the inverse secant equation, which is

H k+1 y k = s k . (C.19)
We can construct H k+1 in the same way as we construct B k+1 . For instance, one can derive the (Bad) Broyden's method by following the reasoning of (Good) Broyden's method; the SR1 method remains the same. More importantly, by mimicking the DFP update, one can derive the Broyden-Fletcher-Goldfarb-Shanno update (BFGS) [START_REF] Broyden | The convergence of a class of double-rank minimization algorithms 1. general considerations[END_REF][START_REF] Fletcher | A new approach to variable metric algorithms[END_REF][START_REF] Goldfarb | A family of variable-metric methods derived by variational means[END_REF][START_REF] Shanno | Conditioning of quasi-newton methods for function minimization[END_REF],

H BFGS k+1 = ( I - s k y T k ⟨y k , s k ⟩ ) H k ( I - y k s T k ⟨y k , s k ⟩ ) + s k s T k ⟨y k , s k ⟩ . (C.20)
The corresponding Hessian update is given by

B BFGS k+1 = B k + y k y T k ⟨y k , s k ⟩ - B k s k s T k B k s T k B k s k . (C.21)
Even though the underlying mechanism of BFGS is the same as the DFP update, the matrix H BFGS k+1 is in fact different from the inverse of the DFP update H DFP k+1 . One explanation is that the symmetrization operation (A + A T )/2 does not commute with the inverse operation of matrices.

In this part, we have provided a review of several well-known Quasi-Newton methods. The construction starts from approximating the Hessian or inverse Hessian matrix, then desired properties such as symmetry and positive definiteness are carefully incorporated. The procedure that we reviewed provides a straightforward way for developing new variants of Quasi-Newton methods. In particular, we can replace the secant equation by

B k+1 h = ∇f (x k+1 + h) -∇f (x k+1 ),
where h is direction to be determined. The drawback of always choosing h = -s k in the secant equation is that s k are usually quite "similar" for recent iterations, meaning that we may keep correcting our approximation in similar directions and ignoring the others. Ideally, as in the discrete Newton method, an Hessian estimation need to perform updates in all directions. Privileging one of them may not be a good choice since the other directions may be seriously wrong from the beginning. This effect does happen in practice that sometimes Quasi-Newton methods are stuck in a region for a long time. This may be an interesting direction for the future work. Finally, we list all the mentioned Quasi-Newton algorithm in the following table: We omit the inverse Hessian update for PSB method since it is relatively complicate, an explicit formula can be found in equation ( 23) of [START_REF] Powell | A new algorithm for unconstrained optimization[END_REF].

D.1. INTRODUCTION

a gradient norm smaller than ε is Õ(ε -2 ) in the worst case, while automatically reducing to Õ(ε -2/3 ) if the function is convex.1 Related work. Inspired by Nesterov's acceleration method for convex optimization [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF], the first accelerated method performing universally well for nonconvex and convex problems was introduced in [START_REF] Ghadimi | Accelerated gradient methods for nonconvex nonlinear and stochastic programming[END_REF]. Specifically, this work addresses composite problems such as (D.1) with n = 1, and, provided the iterates are bounded, it performs no worse than gradient descent on nonconvex instances with complexity O(ε -2 ) on the gradient norm.

When the problem is convex, it accelerates with complexity O(ε -2/3 ). Extensions to accelerated Gauss-Newton type methods were also recently developed in [START_REF] Drusvyatskiy | Efficiency of minimizing compositions of convex functions and smooth maps[END_REF]. In a follow-up work, the authors of [START_REF] Ghadimi | Generalized Uniformly Optimal Methods for Nonlinear Programming[END_REF] proposed a new scheme that monotonically interlaces proximal gradient descent steps and Nesterov's extrapolation; thereby achieving similar guarantees as [START_REF] Ghadimi | Accelerated gradient methods for nonconvex nonlinear and stochastic programming[END_REF] but without the need to assume the iterates to be bounded. Extensions when the gradient of ψ is only Hölder continuous can also be devised.

In [START_REF] Li | Accelerated proximal gradient methods for nonconvex programming[END_REF], a similar strategy is proposed, focusing instead on convergence guarantees under the so-called Kurdyka-Łojasiewicz inequality-a property corresponding to polynomial-like growth of the function, as shown by [START_REF] Bolte | From error bounds to the complexity of first-order descent methods for convex functions[END_REF]. Our scheme is in the same spirit as these previous papers, since it monotonically interlaces proximal-point steps (instead of proximal-gradient as in [START_REF] Ghadimi | Generalized Uniformly Optimal Methods for Nonlinear Programming[END_REF]) and extrapolation/acceleration steps. A fundamental difference is that our method is generic and accommodates inexact computations, since we allow the subproblems to be approximately solved by any method we wish to accelerate.

By considering C 2 -smooth nonconvex objective functions f with Lipschitz continuous gradient ∇f and Hessian ∇ 2 f , the authors of [START_REF] Carmon | Accelerated methods for non-convex optimization[END_REF] propose an algorithm with complexity O(ε -7/4 log(1/ε)), based on iteratively solving convex subproblems closely related to the original problem. It is not clear if the complexity of their algorithm improves in the convex setting. Note also that the algorithm proposed in [START_REF] Carmon | Accelerated methods for non-convex optimization[END_REF] is inherently for C 2 -smooth minimization and requires exact gradient evaluations. This implies that the scheme does not allow incorporating nonsmooth regularizers and can not exploit finite sum structure.

Finally, a stochastic method related to SVRG [START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF] for minimizing large sums while automatically adapting to the weak convexity constant of the objective function is proposed in [START_REF] Allen-Zhu | Natasha: Faster stochastic non-convex optimization via strongly non-convex parameter[END_REF]. When the weak convexity constant is small (i.e., the function is nearly convex), the proposed method enjoys an improved efficiency estimate. This algorithm, however, does not automatically accelerate for convex problems, in the sense that the overall rate is slower than O(ε -3/2 ) in terms of target accuracy ε on the gradient norm.

Organization of the chapter. Section D.2 presents mathematical tools for non-convex and non-smooth analysis, which are used throughout the chapter. In Sections D.3 and D.4, we introduce the main algorithm and important extensions, respectively. Finally, we present experimental results on matrix factorization and training of neural networks in Section D.6.

D.2 Tools for nonconvex and nonsmooth optimization

Convergence results for nonsmooth optimization typically rely on the concept of subdifferential, which does not admit a unique definition in a nonconvex context [START_REF] Borwein | Convex analysis and nonlinear optimization: theory and examples[END_REF]. In this chapter, we circumvent this issue by focusing on a broad class of nonconvex functions known as weakly convex or lower C 2 functions, for which all these constructions coincide. Weakly convex functions cover most of the interesting cases of interest in machine learning and resemble convex functions in many aspects. In this section, we formally introduce them and discuss their subdifferential properties.

Definition 7 (Weak convexity). A function f : R p → R is ρ-weakly convex if for any points x, y ∈ R p and λ ∈ [0, 1], the approximate secant inequality holds:

f (λx + (1 -λ)y) ≤ λf (x) + (1 -λ)f (y) + ρλ(1 -λ) ∥x -y∥ 2 .
Notice that ρ-weak convexity with ρ = 0 is exactly the definition of a convex function. An elementary algebraic manipulation shows that f is ρ-weakly convex if and only if the function x → f (x) + ρ 2 ∥x∥ 2 is convex. In particular, a C 1 -smooth function f is ρ-weakly convex if the gradient ∇f is ρ-Lipschitz, while a C 2 -smooth function f is ρ-weakly convex if and only if ∇ 2 f (x) ⪰ -ρI for all x. This closely resembles an equivalent condition for C 2 -smooth and µ-strongly convex functions, namely ∇ 2 f (x) ⪰ µI with µ > 0.

Useful characterizations of ρ-weakly convex functions rely on differential properties. Since the functions we consider in the paper are nonsmooth, we use a generalized derivative construction. We mostly follow the standard monograph on the subject by Rockafellar and Wets [START_REF] Rockafellar | Variational analysis, volume 317 of Grundlehren der Mathematischen Wissenschaften[END_REF].

Definition 8 (Subdifferential). Consider a function f : R p → R and a point x with f (x) finite. The subdifferential of f at x is the set

∂f (x) := {v ∈ R p : f (y) ≥ f (x) + v T (y -x) + o(∥y -x∥) ∀y ∈ R p }.
Thus, a vector v lies in ∂f (x) whenever the linear function y → f (x) + v T (yx) is a lower-model of f , up to first-order around x. In particular, the subdifferential ∂f (x) of a differentiable function f is the singleton {∇f (x)}, while for a convex function f it coincides with the subdifferential in the sense of convex analysis [see Rockafellar and Wets, 1998, Exercise 8.8]. It is useful to keep in mind that the sum rule, ∂(f + g)(x) = ∂f (x) + ∇g(x), holds for any differentiable function g.

D.3. THE BASIC 4WD-CATALYST ALGORITHM FOR NON-CONVEX OPTIMIZATION

We are interested in deriving complexity bounds on the number of iterations required by a method M to guarantee dist ( 0, ∂f (x)

) ≤ ε .

Recall when ε = 0, we are at a stationary point and satisfy first-order optimality conditions. In our convergence analysis, we will also use the following differential characterization of ρ-weakly convex functions, which generalize classical properties of convex functions. A proof follows directly from Theorem 12.17 of [START_REF] Rockafellar | Variational analysis, volume 317 of Grundlehren der Mathematischen Wissenschaften[END_REF]] by taking into account that f is ρ-weakly convex if and only if f

+ ρ 2 ∥ • ∥ 2 is convex.
Theorem 11 (Differential characterization of ρ-weakly convex functions).

For any lower-semicontinuous function f : R p → R, the following properties are equivalent:

1. f is ρ-weakly convex.

(subgradient inequality) The inequality

f (y) ≥ f (x) + v T (y -x) - ρ 2 ∥y -x∥ 2
holds for all x, y ∈ R p and v ∈ ∂f (x).

(hypo-monotonicity)

The inequality

(v -w) T (x -y) ≥ -ρ∥x -y∥ 2
holds for all x, y ∈ R p and v ∈ ∂f (x), w ∈ ∂f (y).

Weakly convex functions have appeared in a wide variety of contexts, and under different names. Some notable examples are globally lower-C 2 [START_REF] Rockafellar | Favorable classes of Lipschitz-continuous functions in subgradient optimization[END_REF], prox-regular [START_REF] Poliquin | Prox-regular functions in variational analysis[END_REF], proximally smooth functions [START_REF] Clarke | Proximal smoothness and the lower-C 2 property[END_REF], and those functions whose epigraph has positive reach [START_REF] Federer | Curvature measures[END_REF].

D.3 The Basic 4WD-Catalyst algorithm for non-convex optimization

We now present a generic scheme (Algorithm 9) for applying a convex optimization method to minimize

min x∈R p f (x), (D.2)
where f is only ρ-weakly convex. Our goal is to develop a unified framework that automatically accelerates in convex settings. Consequently, the scheme must be agnostic to the constant ρ.

D.3.1 Basic 4WD-Catalyst : a meta algorithm

At the center of our meta algorithm (Algorithm 9) are two sequences of subproblems, with better convexity properties, obtained by adding simple quadratics to f . The proposed approach extends the Catalyst acceleration of [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] and comes with a simplified convergence analysis. We next describe in detail each step of the scheme.

Two-step subproblems. The proposed acceleration scheme builds two main sequences of iterates (x k ) k and (x k ) k , obtaining from approximately solving two subproblems. These subproblems are simple quadratic perturbation of the original problem f having the form:

min x { f κ (x; y) := f (x) + κ 2 ∥x -y∥ 2 } .
Here κ is a regularization parameter and y is called the prox-center. By adding the quadratic, we make the problem more "convex": when f is non convex, with a large enough κ, the subproblem will be convex; when f is convex, we improve the conditioning of the problem. At the k-th iteration, given a previous iterate x k-1 and the extrapolation term v k-1 , we construct the two following subproblems.

1. Proximal point step. We first perform an inexact proximal point step with prox-center

x k-1 :

xk ≈ arg min

v k = x k-1 + 1 α k (x k -x k-1 ) [Extrapolation] (D.4)
where α k+1 ∈ (0, 1) is a sequence of coefficients satisfying (1α k+1 )/α 2 k+1 = 1/α k 2 . Essentially, the sequences (α k ) k , (y k ) k , (v k ) k are built upon the extrapolation principles of [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF].

Picking the best. At the end of iteration k, we have at hand two iterates, resp. xk and xk . Following [START_REF] Ghadimi | Accelerated gradient methods for nonconvex nonlinear and stochastic programming[END_REF], we simply choose the best of the two in terms of their objective values, that is we choose x k such that f (x k ) ≤ min {f (x k ), f (x k )} . )

< κ ∥x kx k-1 ∥ and f κ (x k ; x k-1 ) ≤ f κ (x k-1 ; x k-1 ). )

Set

y k = α k v k-1 + (1 -α k )x k-

< ε

The proposed scheme blends the two steps in a synergistic way, allowing us to recover the near-optimal rates of convergence in both worlds: convex and non-convex. Intuitively, when xk is chosen, it means that Nesterov's extrapolation step "fails" to accelerate convergence.

Stopping criterion for the subproblems. In order to derive complexity bounds, it is important to properly define the stopping criterion for the proximal subproblems. When the subproblem is convex, a functional gap like f κ (z; x)inf z f κ (z; x) may be used as a control of the inexactness, as in [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]. Without convexity, this criterion can not be used since

It is important to notice that this convergence result is valid for any κ and does not require it to be larger than the weak convexity parameter. As long as the stopping criteria for the proximal subproblems are satisfied, the quantities dist(0, ∂f ( xj )) tend to zero. The proof is inspired by that of inexact proximal algorithms [START_REF] Bertsekas | Convex Optimization Algorithms[END_REF][START_REF] Güler | On the convergence of the proximal point algorithm for convex minimization[END_REF], Lin et al., 2015] and appears in the appendix (see Appendix D.8).

If the function f turns out to be convex, the scheme achieves a faster convergence rate both in function values and in stationarity: This theorem establishes a rate of O(N -2 ) for suboptimality in function value and convergence in O(N -3/2 ) for the minimal norm of subgradients. The first rate is optimal in terms of information-based complexity for the minimization of a convex composite function [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF][START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF]. The second can be improved to O(N -2 log(N )) through a regularization technique, if one knew in advance that the function is convex and had an estimate on the distance of the initial point to an optimal solution [Nesterov, 2012a]. The proof appears in the appendix; see Appendix D.9.1.

Towards an automatically adaptive algorithm. So far, our analysis has not taken into account the cost of obtaining the iterates xj and xj by the algorithm M. We emphasize again that the two results above do not require any assumption on κ, which leaves us a degree of freedom. In order to develop the global complexity, we need to evaluate the total number of iterations performed by M throughout the process. Clearly, this complexity heavily depends on the choice of κ, since it controls the magnitude of regularization we add to improve the convexity of the subproblem. This is the point where a careful analysis is needed, because our algorithm must adapt to ρ without knowing it in advance. The next section is entirely dedicated to this issue. In particular, we will explain how to automatically adapt the parameter κ (Algorithm 10).

D.4 The 4WD-Catalyst algorithm

In this section, we work towards understanding the global efficiency of Algorithm 9, which automatically adapts to the weak convexity parameter. For this, we must take into account the cost of approximately solving the proximal subproblems to the desired stopping criteria. First, we require some natural conditions on the optimization method M.

Linear convergence of M: When κ is sufficiently large, the subproblems become strongly convex; thus easy to solve. As such, we assume that M has a linear convergence rate, namely for any κ > ρ, there exist A κ ≥ 0 and τ κ ∈ (0, 1) so that the following hold:

1. For any prox-center y ∈ R p and initial z 0 ∈ R p the iterates {z t } t≥1 generated by M on the problem min z f κ (z; y) satisfy dist 2 (0, ∂f κ (z t ; y)) ≤ A κ (1τ κ ) t (f κ (z 0 ; y)f * κ (y)) (D.14)

where f κ (y) * := inf z f κ (z; y). If the method M is randomized, we allow this inequality to take an expectation

E [ dist 2 ( 0, ∂f κ (z t ; y) )]
≤ A κ (1τ κ ) t (f κ (z 0 ; y)f * κ (y)) .

2. The rates τ κ and the constants A κ are increasing in κ.

Remark 2. The linear convergence we assume here for M differs from the one considered by [START_REF] Lin | A universal catalyst for first-order optimization[END_REF], which was given in terms of function values. However, if the problem is a composite one, both points of view are near-equivalent, as discussed in Section D.7 and the precise relationship is given in Appendix D.9.

We chose the norm of the subgradient as our measurement because the complexity analysis is easier. This allow us to bound the computational complexity to achieve an ε approximate stationary point.

Lemma 21. Given a strongly convex problem f κ (•; y) and a linear convergent method M generating {z t } t≥0 . Given a target accuracy ε, denote T (ε) = inf{t ≥ 1, dist ( 0, ∂f κ (z t ; y)

) ≤ ε}. 1. If M is deterministic, then T (ε) ≤ 1 τκ log ( Aκ(fκ(z 0 ;y)-f * κ (y)) ε 2 ) . 2. If M is randomized, then E [T (ε)] ≤ 1 τκ log ( Aκ(fκ(z 0 ;y)-f * κ (y)) τκε 2
) . [Lemma C.1 of [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]]

As we can see, we lose a factor in the log term by passing from deterministic to randomized algorithms. For the sake of simplicity, we perform our analysis only for deterministic algorithms and we use τ κ to denote the linear rate for f κ .

Algorithm 10 4WD-Catalyst input Fix a point x 0 ∈ dom f , real numbers κ 0 , κ cvx > 0 and T, S > 0, and an opt. method M. initialization: α 1 = 1, v 0 = x 0 . repeat for k = 1, 2, . . . 1. Compute (x k , κ k ) = Auto-adapt (x k-1 , κ k-1 , T ).

Compute y

k = α k v k-1 + (1 -α k )x k-
v k = x k-1 + 1 α k (x k -x k-1 ) and α k+1 = √ α 4 k + 4α 2 k -α 2 k 2 .
4. Choose x k to be any point satisfying f (x k ) = min{f (x k ), f (x k )}.

until the stopping criterion dist The first idea consists of using a pre-defined number of iterations for solving (D.7), corresponding to the maximal number of iterations of M that we are willing to tolerate for evaluating xk . The main motivation is three-fold: (i) the complexity of such a strategy is known in advance; (ii) in the convex case, we can choose a small κ, denoted by κ cvx , that ensures we obtain the optimal convergence guarantee; (iii) for nonconvex problems, we remark from the proof of Theorem 12 that the convergence guarantees are obtained regardless of the quality of xk for solving (D.7). In other words, the conclusions of Theorem 12 are valid even when the iterates xk do not satisfy ∥∂f κ (x k )∥ < κ k+1 ∥x ky k ∥ for non-convex functions.This allows us to choose the same strategy for computing xk in the convex and non-convex case-that is, we use the pre-defined number of iterations of M and a "convex" smoothing parameter κ cvx .

The second idea lies in an adaptive procedure, Auto-adapt; see Algorithm 11. This procedure uses a pre-defined number of iterations of M to compute the iterates xk while automatically choosing the smoothing parameter κ. We show that as soon as the number of iterations allocated to M for computing xk is large enough, acceleration is preserved in the convex case, while GRADIENT-BASED NON-CONVEX OPTIMIZATION Theorem 17 (Global complexity bounds for 4WD-Catalyst). Choose T and S as in Õ(τ -1 L ) and Õ(τ -1 κcvx ) (see Theorem 16 in appendix). We let Õ hide universal constants and logarithmic dependencies in A L , A κcvx , L, ε, κ 0 , κ cvx , and ∥x *x 0 ∥ 2 . Then the following are true.

1. Algorithm 10 will generate a point x satisfying dist ( 0, ∂f (x)

)
≤ ε after at most

Õ ( ( τ -1 L + τ -1 κcvx ) • L(f (x 0 ) -f * ) ε 2
) iterations of the method M.

2. If f is convex, then Algorithm 10 will generate a point x satisfying dist 3. If f is convex, then Algorithm 10 will generate a point x satisfying f (x)f * ≤ ε after at most

Õ   ( τ -1 L + τ -1 κcvx ) • √ κ cvx ∥x * -x 0 ∥ 2 √ ε  
iterations of the method M.

D.5 Applications to Existing Algorithms

We show how to accelerate existing algorithms M and compare the convergence guaranties before and after 4WD-Catalyst. In particular, the algorithms considered are gradient descent and incremental methods, SAGA and SVRG. For all the algorithms considered, we state the convergence guaranties in terms of the total number of iterations (in expectation, if appropriate) to reach an accuracy of ε; in the convex setting, the accuracy is stated in terms of functional Randomized incremental gradient. We now consider randomized incremental gradient methods such as SAGA [Defazio et al., 2014a] and (prox) SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF]. Here, the optimal choice for κ cvx is O(L/n). Under the convex setting, we achieve an accelerated rate of O( √ n √ L/ε log(1/ε)). A direct application of SVRG and SAGA have no convergence guarantees in the non-convex setting. With our approach, the resulting algorithm matches the guarantees for FG up to log factors. The global complexity for gradient descent is 1. Algorithm 10 will generate a point x satisfying dist ( 0, ∂f (x)

Derivation of the variables for the existing algorithms

)

≤ ε after at most

O [ n log ( L κ 0 ) log ( L 2 (f (x 0 ) -f * ) 2 ε 4 ) • ( L(f (x 0 ) -f * ) ε 2 ) ]
gradient computations.

2. If f is convex, then Algorithm 10 will generate a point x satisfying dist ) ] gradient computations.

D.5. APPLICATIONS TO EXISTING ALGORITHMS

3. If f is convex, then Algorithm 10 will generate a point x satisfying f (x)f * ≤ ε after at most

O [ n log ( L κ 0 ) • log ( L ∥x 0 -x * ∥ 2 ε ) • √ L ∥x * -x 0 ∥ √ ε ] gradient computations.
For SVRG, we have the following ) }

.

The global complexity for SVRG when n is sufficiently large is 1. Algorithm 10 will generate a point x satisfying dist ( 0, ∂f (x)

)

≤ ε after at most

O [ n log ( L κ 0 ) log ( nL 2 (f (x 0 ) -f * ) 2 ε 4 ) • ( L(f (x 0 ) -f * ) ε 2 ) ]
gradient computations.

2. If f is convex, then Algorithm 10 will generate a point x satisfying dist • log

( nL ∥x 0 -x * ∥ 2 ε ) • √ L ∥x * -x 0 ∥ √ ε ] gradient computations.
For SAGA, we have the following

A 4L Method M constant 4L
We observe that the variables for SAGA are only constant multiples as those for SVRG. Therefore, the global complexities results for SAGA are, up to constants, the same as SVRG.

We present a summary of common choices for the parameters when minimizing the composite form, 1 n ∑ n i=1 f i (x) + ψ(x) where L is the Lipschitz constant.

Variable Description Initialization

x 0 ∈ R p Starting point; x 0 ∈ dom(f ) Random

D.6 Experiments

We investigate the performance of 4WD-Catalyst in two standard non-convex problems in machine learning. We report experimental results of 4WD-Catalyst when applied to two different algorithms: SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF] and SAGA [Defazio et al., 2014a]. We compare the following algorithms:

• Nonconvex SVRG/SAGA [Reddi et al., 2016b]: stepsize η = 1/Ln 2/3 • Convex SVRG/SAGA [Defazio et al., 2014a, Xiao and[START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF]: stepsize η = 1/2L • 4WD-Catalyst SVRG/SAGA: stepsize η = 1/2L

The original version of SVRG (resp. SAGA), convex SVRG (resp. SAGA), was designed for minimizing convex objectives. We report their results, while there is no theoretical guarantee on their behavior when venturing into nonconvex terrains. We also report the results of recently proposed variants, Nonconvex SVRG/SAGA, designed for minimizing nonconvex objectives. The proposed algorithms 4WD-Catalyst SVRG and 4WD-Catalyst SAGA enjoy the strong theoretical guarantees stated in Sec. 3.

Parameter settings

We start from an initial estimate of the Lipschitz constant L and use the theoretically recommended κ 0 = κ cvx = 2L/n. The number of inner iterations is to T = S = n in all experiments, which boils down to making one pass at most over the data for solving each sub-problem. We simply drop the log(k) dependency while solving the subproblem in (D.17). These choices turn out to be justified a posteriori, as both SVRG and SAGA have a much better convergence rate in practice than the theoretical rate derived from a worst-case analysis. Indeed, in all experiments, one pass over the data to solve each sub-problem is enough to guarantee sufficient descent.

Sparse matrix factorization a.k.a. dictionary learning. Dictionary learning consists of representing a dataset X = [x 1 , • • • , x n ] ∈ R m×n as a product X ≈ DA, where D in R m×p is called a dictionary, and A in R p×n is a sparse matrix. The classical non-convex formulation [see [START_REF] Mairal | Sparse modeling for image and vision processing[END_REF] represent class labels, and a i in R p are feature vectors. The estimator of a label class is now given by a two-layer neural network b = sign(W ⊤ 2 σ(W ⊤ 1 a)), where W 1 in R p×d represents the weights of a hidden layer with d neurons, W 2 in R d carries the weight of the network's second layer, and σ(u) = log(1 + e u ) is a non-linear function, applied pointwise to its arguments. We fix the number of hidden neurons to d = 100 and use the logistic loss to fit the estimators to the true labels. Since the memory required by SAGA becomes n times larger than SVRG for nonlinear models, which is problematic for large n, we can only perform experiments with SVRG. The experimental results are reported on two datasets alpha and covtype in Convex-SVRG for minimizing nonconvex objectives. Furthermore, in these experiments 4WD-Catalyst-SVRG enjoys a faster convergence in objective values. This confirms the remarkable ability of 4WD-Catalyst-SVRG to adapt to nonconvex terrains. Similar conclusions hold when applying 4WD-Catalyst to SAGA; see Sec.6. in Appendix, which demonstrates how general 4WD-Catalyst is.

In neural network experiments, we observe that 4WD-Catalyst-SVRG converges much faster in terms of objective values than the competing algorithms. Nonconvex-SVRG with the theoretically recommended sequence of step-sizes [Reddi et al., 2016b] compares unfavorably here, which implies that the recommended step-sizes are too pessimistic hence too small. We also observe an interesting phenomenon: the subgradient norm may increase at some point then decrease, while the function value keeps decreasing, as the algorithm proceeds. This suggests that the extrapolation step, or the Auto-adapt procedure, is helpful to escape bad stationary points, e.g., saddle-points. A more systematic study is required to confirm such observation, we leave it as a potential direction of future work.

D.7 Note on convergence rates in strongly-convex composite minimization

We now briefly discuss convergence rates, which are typically given in different forms in the convex and non-convex cases. If the weak-convex constant is known, we can form a strongly convex approximation similar to [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]. For that purpose, we consider a stronglyconvex composite minimization problem min x∈R p h(x) := f 0 (x) + ψ(x),

D.8. THEORETICAL ANALYSIS OF THE BASIC ALGORITHM

Proof. This result is noted without proof in a remark of [START_REF] Tseng | On accelerated proximal gradient methods for convex-concave optimization[END_REF]. For completeness, we give below a simple proof using induction. Clearly, the statement holds for k = 1. Assume the inequality on the right-hand side holds for k. By using the induction hypothesis, we get

α k+1 = √ α 4 k + 4α 2 k -α 2 k 2 = 2 √ 1 + 4/α 2 k + 1 ≤ 2 √ 1 + (k + 1) 2 + 1 ≤ 2 k + 2 ,
as claimed and the expression for α k+1 is given by explicitly solving (D.9). To show the lower bound, we note that for all k ≥ 1, we have

α 2 k+1 = (1 -α k+1 )α 2 k = k+1 ∏ i=2 (1 -α i )α 2 1 = k+1 ∏ i=2 (1 -α i ).
Using the established upper bound α k ≤ 2 k+1 yields

α 2 k+1 ≥ k+1 ∏ i=2 ( 1 - 2 i + 1 ) = 2 (k + 2)(k + 1) ≥ 2 (k + 2) 2 .
The result follows.

Lemma 23 (Prox-gradient and near-stationarity). Suppose y + satisfies dist(0, ∂f κ (y + ; y)) < ε. Then, the inequality holds: Proof. We can find ξ ∈ ∂f κ (y + ; y) with ∥ξ∥ ≤ ε. Taking into account ∂f κ (y + ; y) = ∂f (y + ) + κ(y +y) the result follows.

Next we establish convergence guarantees of Theorem 12 and Theorem 13 for Basic 4WD-Catalyst .

Proof of Theorem 13 and Theorem 13. The proof of Theorem 12 follows the analysis of inexact proximal point method [START_REF] Lin | A universal catalyst for first-order optimization[END_REF][START_REF] Güler | On the convergence of the proximal point algorithm for convex minimization[END_REF][START_REF] Bertsekas | Convex Optimization Algorithms[END_REF]. The descent condition in (D.11) implies {f (x k )} k≥0 are monotonically decreasing. From this, we deduce

f (x k-1 ) = f κ (x k-1 ; x k-1 ) ≥ f κ (x k ; x k-1 ) ≥ f (x k ) + κ 2 ∥x k -x k-1 ∥ 2 . (D.23)
Using the adaptive stationarity condition (D.11), we apply Lemma 23 with y = x k-1 , y + = xk and ε = κ ∥x kx k-1 ∥; hence we obtain dist(0, ∂f (x k )) ≤ 2 ∥κ(x kx k-1 )∥ .

D.9. ANALYSIS OF 4WD-CATALYST AND AUTO-ADAPT

Denote A k := 1θ 2 k . Subtracting f * from both sides and using the inequality 1-α k

α 2 k = 1 α 2 k-1
and α 1 ≡ 1, we derive the following recursion argument:

f (x k ) -f * α 2 k + A k κ 2 ∥x * -v k ∥ 2 ≤ 1 -α k α 2 k ( f (x k-1 ) -f * ) + κ 2 ∥x * -v k-1 ∥ 2 ≤ 1 A k-1 ( f (x k-1 ) -f * α 2 k-1 + A k-1 κ 2 ∥x * -v k-1 ∥ 2
) .

The last inequality follows because 0 < A k-1 ≤ 1. Iterating N times,we deduce

f (x N ) -f * α 2 N ≤ N ∏ j=2 1 A j-1 ( κ 2 ∥x * -v 0 ∥ 2
) .

(D.26)

We note

N ∏ j=2 1 A j-1 = 1 ∏ N j=2 ( 1 -1 (j+1) 2
) ≤ 2; This yields the descent condition which completes the proof for T . The proof for S κ is similar to T κ , so we omit many of the details. In this case, we only need to show the adaptive stationary condition. For convenience, we denote S = S κ . Following the same argument as in Equation (D.39) but with S log(k + 1) number of iterations, we deduce dist 2 (0, ∂f κ (z S ; y)) ≤ (κρ) 2 64(k + 1) 2 ∥y * -y∥ 2 .

By applying Lemma 26, we obtain dist(0, ∂f κ (z S ; y)) ≤ (κρ)

4 ∥z T -y∥ ≤ κ k + 1 ∥z S -y∥ ,
which proves the desired result for z S .

Assuming Proposition 21 and Proposition 22 hold as well as Lemma 27, we begin by providing the proof of Theorem 16.

Proof of Theorem 16. We consider two cases: (i) the function f is non-convex and (ii) the function f is convex. First, we consider the non-convex setting. To produce xk , the method M is called T log

( 4L κ 0 )
/ log(2) (D.40) number of times. This follows from Proposition 21 and Lemma 27. The reasoning is that once κ > ρ + L, which only takes at most log(4L/κ 0 ) number of increases of κ to reach, then the iterate xk satisfies the stopping criteria (D.18). Each time we increase κ we run M for T iterations. Therefore, the total number of iterations of M is given by multiplying T with log(4L/κ 0 ). To produce xk , the method M is called S log(k + 1) number of times. (Note: the proof of Theorem 15 does not need xk to satisfy (D.19) in the non-convex case).

Next, suppose the function f is convex. As before, to produce xk the method M is called (D.40) times. To produce xk , the method M is called S log(k + 1) number of times. By Proposition 22, the iterate xk satisfies (D.19); a key ingredient in the proof of Theorem 15.

1. 2 .

 2 Figure 1.1 -Graphical illustration of L-smoothness and µ-strong convexity.

  Proximal SDCA [Shalev-Shwartz and Zhang, 2012]. The algorithm Prox-MISO is almost identical to variant 5 of proximal SDCA [Shalev-Shwartz and Zhang, 2012], which performs the same updates with δ = µn/(L + µn) instead of δ = min(1, µn 2(L-µ)

Figure 3

 3 Figure3.1 -Experimental study of different stopping criterions for Catalyst-SVRG. We plot the value f (x k )/f * -1 as a function of the number of gradient evaluations, on a logarithmic scale; the optimal value f * is estimated with a duality gap.

Figure 3

 3 Figure 3.2 -Experimental study of different stopping criterions for Catalyst-SAGA, with a similar setting as in Figure 3.1.

Figure 3

 3 Figure 3.3 -Experimental study of different stopping criterions for Catalyst-MISO, with a similar setting as in Figure 3.1

Figure 3

 3 Figure3.4 -Experimental study of the performance of Catalyst applying to SVRG, SAGA and MISO. The dashed lines correspond to the original algorithms and the solid lines correspond to accelerated algorithms by applying Catalyst. We plot the relative function value gap (f (x k )f * )/f * in the number of gradient evaluations, on a logarithmic scale.

  Figure3.5 -Evaluations of Catalyst-SVRG for different κ using stopping criterion C1, where κ 0 is the theoretical choice given by the complexity analysis.

  Figure 4.1 -Experimental study of the performance of QuickeNing-SVRG for minimizing large sums of functions. We plot the value F (x k )/F * -1 as a function of the number of gradient evaluations, on a logarithmic scale; the optimal value F * is estimated with a duality gap.

Figure 4

 4 Figure4.2 -Experimental study of the performance of QuickeNing-ISTA. We plot the value F (x k )/F * -1 as a function of the number of gradient evaluations, on a logarithmic scale; the optimal value F * is estimated with a duality gap.

  Figure 4.4 an experiments by trying the values κ = 10 i κ 0 , for

Figure 4 . 3 -

 43 Figure4.3 -Experimental study of influence of the limited-memory parameter l for QuickeNing-SVRG1. We plot the value F (x k )/F * -1 as a function of the number of gradient evaluations, on a logarithmic scale; the optimal value F * is estimated with a duality gap.

Figure 4 . 4 -

 44 Figure4.4 -Experimental study of influence of the smoothing parameter κ for QuickeNing-SVRG1. κ 0 denotes the default choice used in the previous experiments. We plot the value F (x k )/F * -1 as a function of the number of gradient evaluations, on a logarithmic scale; the optimal value F * is estimated with a duality gap.

Figure 5 Figure 5

 55 Figure 5.1 -Dictionary learning experiments. We plot the function value (top) and the subgradient norm (bottom). From left to right, we vary the size of the dataset from n = 1 000 to n = 100 000

x

  f κ (x; x k-1 ) [Proximal-point step]2. Accelerated proximal point step. Then we build the next prox-center y k as the convex combinationy k = α k v k-1 + (1α k )x k-1 . (D.3)Next we use y k as a prox-center and update the next extrapolation term:xk ≈ arg min x f κ (x; y k ) [Accelerated proximal-point step]

  158D.3. THE BASIC 4WD-CATALYST ALGORITHM FOR NON-CONVEX OPTIMIZATIONAlgorithm 9 Basic 4WD-Catalyst input Fix a point x 0 ∈ dom f , real numbers κ > 0, and an optimization method M.initialization:α 1 ≡ 1, v 0 ≡ x 0 . repeat for k = 1, 2, . . . 1. Choose xk using M such that xk ≈ arg min x f κ (x; x k-1 ) κ (x k ; x k-1 )

  xk using M such that xk ≈ arg min x f κ (x; y k ) where dist ( 0, ∂f κ (x k ; y k ) x k to be any point satisfying f (x k ) ≤ min {f (x k ), f (x k )}. (D.10) until the stopping criterion dist ( 0, ∂f (x k )

Theorem 13 (

 13 Outer-loop complexity, convex case). If the function f is convex, then for any κ > 0 and N ≥ 1, the iterates generated by Algorithm 9 satisfyf (x N )f (x * ) ≤ 16κ (N + 1) 2 ∥x *x 0 ∥ 2 + 1) 2 ∥x *x 0 ∥ 2where x * is any minimizer of the function f .

  Adaptive versions of Basic 4WD-Catalyst. To solve the previous theoretical and practical challenges, we introduce an adaptive variant, called 4WD-Catalyst, of Basic 4WD-Catalyst. The adaptive algorithm is presented in Algorithm 10. There are two main modifications to Algorithm 9.

  output zS = xk satisfies ∥∂f κcvx (z S )∥ < κcvx k+1 ∥z S ky k ∥ for all k ≥ 1.We can now derive global complexity bounds by combining Theorem 15 and Theorem 16, and a good choice for the constant κ cvx .

  3 (κ cvx ∥x *x 0 ∥ 2 )

  Figure D.3 -Neural network experiments on subsets of dataset alpha. From left to right, we vary the size of the dataset's subset from n = 1 000 to n = 100 000.

FigureFigure

  Figure D.5 -We run 50 iterations of 4WD-Catalyst SVRG with different choice of S on twolayer neural network. The data is a subset of dataset covtype. The x-axis is the number of gradient evaluations on the left, which is T + S k per iteration with T = 1; and the number of iterations on the right.

  dist

( 0 ,

 0 ∂f (y + ) ) ≤ ε + κ(y +y) .

  (D.27) thereby concluding the result. Summing up (D.24) from j = N + 1 to 2N , we obtainmin j=1,...,2N { dist 2 (0, ∂f (x j )) x N )f * )Combining this inequality with (D.26), the result is shown.D.9 Analysis of 4WD-Catalyst and Auto-adaptLinear convergence interlude. Our assumption on the linear rate of convergence of M (see (D.14)) may look strange at first sight. Nevertheless, most linearly convergent first-order methods M for composite minimization either already satisfy this assumption or can be made to satisfy it by introducing an extra prox-gradient step. To see this, recall the convex composite minimization problem from Section D.7min z∈R p h(z) := f 0 (z) + ψ(z),where D.10. INNER-LOOP COMPLEXITY: PROOF OF THEOREM 16 which gives the adaptive stationary condition. Next, we show we also have the descent condition. By the (κρ)-strong convexity of f κ (•; y), for any v ∈ ∂f κ (z T ; y) we deduce f κ (y; y) ≥ f κ (z T ; y) + ⟨v, yz T ⟩ + κρ 2 ∥z T -y∥ 2 ≥ f κ (z T ; y) -∥v∥ ∥yz T ∥ + κρ 2 ∥z T -y∥ 2≥ f κ (z T ; y).

Basic Properties of the Moreau-Yosida Envelope. The smoothing effect of the Moreau- Yosida regularization can be characterized by the next proposition [see Lemaréchal and Sagas- tizábal, 1997a, for elementary proofs]. Proposition 1 (Regularization properties of the Moreau-Yosida Envelope). Given

	1.6. THE MOREAU-YOSIDA REGULARIZATION
	2. F is continuously differentiable even when f is not and
	a
	convex continuous function f and a regularization parameter κ > 0, we consider the Moreau-
	Yosida envelope F defined in (1.35). Then,
	1. F is convex and minimizing f and F are equivalent in the sense that
	min x∈R p F (x) = min x∈R p f (x),
	and the solution set of the two above problems coincide with each other.

  1 and apply S log(k + 1) iterations of M to find

	xk ≈ arg min x∈R p	f κcvx (x, y k ).	(5.6)
	3. Update v k and α k+1 by

  Given B ∈ L(R d ) be symmetric, and let y, s and z be the vectors in R d such that ⟨z, s⟩ > 0. Let W ∈ L(R d ) be any nonsingular, symmetric matrix such that W z = s. Then the matrix C ∞ defined in (C.12) is the unique solution of the problem

	⟨z, s⟩	Bs)	-	⟨y -Bs, s⟩ ⟨z, s⟩	zz T .	(C.12)
	In the variational point of view, this procedure converges to the solution of the following
	minimization problem:
	Proposition 20. min B∈L(R d )

{∥ B -B∥ F,W | B symmetric, Bs = y}, (C.13)

where ∥ • ∥ F,W denotes the weighted Frobenius norm defined by

Bs)(y -Bs) T ⟨y -Bs, s⟩ H + (s -Hy)(s -Hy) T ⟨s -Hy, y⟩ PSB B + rs T + sr T ∥s∥ 2 - ⟨r, s⟩ss T

				Hessian estimation: B k+1	Inverse Hessian estimation: H k+1	(S) (P)
	Broyden			B +	(y -Bs)s T ∥s∥ 2				H +	(s -Hy)s T H s T Hy
	SR1			B +	(y -∥s∥ 4				( * )
	DFP	(	I -	ys T ⟨y, s⟩	)	B	( I -	sy T ⟨y, s⟩	)	+	yy T ⟨y, s⟩	H +	ss T ⟨s, y⟩	-	Hyy T H y T Hy
	BFGS			B +	yy T ⟨y, s⟩	-	Bss T B s T Bs		( I -	sy T ⟨y, s⟩	)	H	( I -	ys T ⟨y, s⟩	)	+	ss T ⟨y, s⟩

Table

C

.1 -For simplicity, we abbreviate B k , H k , s k , y k by B, H, s, y, respectively. The vector r is the residual defined by r = y -Bs. The bold formulas are obtained directly form the construction and the others are obtained from applying the Sherman-Morrison formula (C.6).

  1 and apply S log(k + 1) iterations of M to find xk ≈ arg min x∈R p f κcvx (x, y k ). (D.17) 3. Update v k and α k+1 by

  For gradient descent, we have the following table of values

	Variable Description	Value
	1/τ L	linear convergence with κ = L	2
	κ cvx	smooth parameter for convex set-ting	L
	1/τ κcvx linear convergence with κ cvx = L	2
	A 4L	Method M constant	4L
	Using these values for gradient descent, the number of iterations in the inner loop are
		T ≥ 2 log(640)
		S log(k + 1) ≥ 2 log

( 64(k + 1) 2 )

  table of values κcvx linear convergence with κ cvx = L/n 2n + 1 Using these values for SVRG, the number of iterations in the inner loop are

	Variable Description					Value
	1/τ L	linear convergence with κ = L	n + 2
	κ cvx	smooth parameter for convex set-ting	L/n
	1/τ A 4L	Method M constant			4L
	T ≥ (n + 2) log(640) S log(k + 1) ≥ max { (2n + 1) log	(	32 •	n L	( L n	+ L

)

• (k + 1) 2

  table of values

	Variable Description		Value
	1/τ L	linear convergence with κ = L		n +	√	2n
	κ cvx	smooth parameter for convex set-ting	L/(2n + 1)
	1/τ κcvx	linear convergence with κ cvx = L/(2n + 1)	n +	√ 2n 2 + n + 1

See Proposition 18 in the Appendix.

Meaning ∥x k+1 -x * ∥ ∥x k -x * ∥ → 0.

As a recompense, we need to additionally evaluate g i k at each iteration.

The L we are using here is in fact the maximum of L i , where L i is the Lipschitz constant with respect to the i-th component f i . An improvement of this dependency to the average Lipschitz constant can be achieved by applying a non uniform sampling strategy; we omit the discussion here for simplicity.

Note that even though we call this algorithm MISO (or Finito), it was called MISOµ in[Mairal,

2015], whereas "MISO" was originally referring to an incremental majorization-minimization procedure that uses upper bounds of the functions f i instead of lower bounds.

We call a function L-smooth when it is differentiable and its gradient is L-Lipschitz continuous.

Note that our inexact criterion was also studied, among others, by[START_REF] Salzo | Inexact and accelerated proximal point algorithms[END_REF], but their analysis led to the conjecture that this criterion was too weak to warrant acceleration. Our analysis refutes this conjecture.

Such as, duality gap, ε-subdifferential or decrease in terms of function value.

Note that the rule for the non strongly convex case, denoted here by µ = 0, slightly differs from[START_REF] Lin | A universal catalyst for first-order optimization[END_REF] and results from a tighter complexity analysis.

The coefficient

corresponds to the choice θ = 0.1 and m = 100 L µ in[START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF].

This stopping criterion is heuristic since one pass may not be enough to achieve the required accuracy. What we have shown is that with a large enough T M , then the convergence will be guaranteed. Here we take heuristically T M as one pass.

Note that in the context of BFGS applied to F , a heuristic line-search that requires evaluating f instead of F is also proposed in[START_REF] Chen | Proximal quasi-Newton methods for nondifferentiable convex optimization[END_REF], but this heuristic does not guarantee the algorithm convergence and is still computationally demanding when f is a large sum of functions as in (4.1).

available here http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html

In this section, the notation Õ only displays the polynomial dependency with respect to ε for simplicity.

See[START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] Zhang, 2014, Defazio et al., 2014a]

See[Reddi et al., 2016a,b]. Here we present the convergence result without any mini-batching. In the mini-batch case, the same convergence can be obtained with a batch size b = n 2/3 and the large stepsize O(1/L) is allowed.

(k + 2) 2 .

Remind that for historical reasons, we use B for denoting Hessian approximations and use H for denoting the approximations of the inverse Hessian matrix.

Convergence results are beyond the focus of this part, we invite readers who are interested in that to go through the discussion in Section 1.3.

In this section, the notation Õ only displays the polynomial dependency with respect to ε for the clarity of exposition.

If f is convex, then Algorithm 10 will generate a point x satisfying f (x)f * ≤ ε after at most

Acknowledgements

Acknowledgements v

iterations in expectation. QuickeNing-SVRG with minimizes f to ε accuracy in

iterations in expectation.

Choice of κ. The worst-case complexity theory also does not seem to offer any guidance regarding the choice of κ for the QuickeNing acceleration to be most effective. We shall experiment with a heuristic, consisting in choosing the κ that would lead to the most acceleration for the related Catalyst acceleration scheme [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]. We present empirical evidence in support of this heuristic in Section 4.5.

Experiments and practical details

In this section, we present experimental results obtained by applying QuickeNing on SVRG (Section 4.5.3) and on the proximal gradient descent algorithm ISTA (Section 4.5.4), and we compare the resulting methods to other acceleration techniques and to the L-BFGS algorithm.

We also study the behavior of QuickeNing when changing the smoothing parameter κ and the limited-memory parameter l (Section 4.5.5). Before that, we first present in Section 4.5.1 the formulations and datasets that we use, and also discuss in Section 4.5.2 various experimental and practical choices.

Formulations and datasets

We consider the same problems as in Section 3.6.1 including regularized logistic regression, Elastic-net and Lasso problem. The datasets are also identical to the that in the experiments of Catalyst with different characteristics in terms of size and dimension:

Choice of hyper-parameters and variants

Before presenting the numerical results, we discuss the choice of default parameters used in the experiments as well as different variants.

Chapter 5

Catalyst for non-convex optimization

Chapter abstract:

We introduce a generic scheme to solve nonconvex optimization problems using gradientbased algorithms originally designed for minimizing convex functions. When the objective is convex, the proposed approach enjoys the same properties as the Catalyst approach of [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]. When the objective is nonconvex, it achieves the best known convergence rate to stationary points for first-order methods. Specifically, the proposed algorithm does not require knowledge about the convexity of the objective; yet, it obtains an overall worst-case efficiency of Õ(ε -2 ) and, if the function is convex, the complexity reduces to the near-optimal rate Õ(ε -2/3 ). We conclude the chapter by showing promising experimental results obtained by applying the proposed approach to SVRG and SAGA for sparse matrix factorization and for learning neural networks.

The material of this chapter is a short version of the following technical report, which is a joint work with Courtney Paquette and Dmitriy Drusvyatskiy from University of Washington and my PhD advisors:

C. Paquette, H. Lin, D. Drusvyatskiy, J. Mairal, and Z. Harchaoui. Catalyst acceleration for gradient-based non-convex optimization. arXiv preprint arXiv:1703.10993v2, 2017

The paper is in preparation for submission to AISTATS 2018. For completeness, the technical report is presented in appendix D, which includes the proofs of all technical results of this chapter.

Following the same experimental setting as [Reddi et al., 2016b], we evaluate algorithms in the absent of mini-batch. The original version of SVRG (resp. SAGA), convex SVRG (resp. SAGA), was designed for minimizing convex objectives. We report their results, while there is no theoretical guarantee on their behavior when venturing into nonconvex terrains. We also report the results of recently proposed variants, Nonconvex SVRG/SAGA, designed for minimizing nonconvex objectives. The proposed algorithms 4WD-Catalyst SVRG and 4WD-Catalyst SAGA enjoy the strong theoretical guarantees stated in Sec. 3.

Parameter settings

We start from an initial estimate of the Lipschitz constant L and use the theoretically recommended κ 0 = κ cvx = 2L/n. The number of inner iterations is to T = S = n in all experiments, which boils down to making one pass at most over the data for solving each sub-problem. We simply drop the log(k) dependency while solving the subproblem in (5.6). These choices turn out to be justified a posteriori, as both SVRG and SAGA have a much better convergence rate in practice than the theoretical rate derived from a worst-case analysis. Indeed, in all experiments, one pass over the data to solve each sub-problem is enough to guarantee sufficient descent. We focus in the main text on the results for SVRG. We relegate results for SAGA and details about experiments to Sec.6 in Appendix.

Sparse matrix factorization a.k.a. dictionary learning. Dictionary learning consists of representing a dataset

is called a dictionary, and A in R p×n is a sparse matrix. The classical non-convex formulation [see [START_REF] Mairal | Sparse modeling for image and vision processing[END_REF] can be reformulated as the equivalent finite-sum problem min D∈C

(5.9)

ψ is a sparsity-inducing regularization and C is chosen as the set of matrices whose columns are in the ℓ 2 -ball; see Sec.6 in Appendix. We consider elastic-net regularization ψ(α) = µ 2 ∥α∥ 2 + λ∥α∥ 1 of [Zou and Hastie, 2005], which has a sparsity-inducing effect, and report the corresponding results in Figure 5.1, learning a dictionary in R m×p with p = 256 elements, on a set of whitened normalized image patches of size m = 8 × 8. Parameters are standard ones in this literature [START_REF] Mairal | Sparse modeling for image and vision processing[END_REF]-that is, a small value µ = 1e -5, and λ = 0.25, leading to sparse matrices A (on average ≈ 4 non-zero coefficients per column of A).

Neural networks. We consider simple binary classification problems for learning neural networks. Assume that we are given a training set {a i , b i } n i=1 , where the variables b i in {-1, +1} represent class labels, and a i in R p are feature vectors. The estimator of a label class is now given by a two-layer neural network b = sign(w ⊤ 2 σ(W ⊤ 1 a)), where W 1 in R p×d represents the weights of a hidden layer with d neurons, w 2 in R d carries the weight of the network's second layer, and σ(u) = log(1 + e u ) is a non-linear function, applied point-wise to its arguments. We

Appendix A

Proofs of simple results

A.1 Convergence of the gradient descent method

Theorem 10 (Convergence of gradient descent). If f is convex and L-smooth, then by letting η k = 1 L , the sequence {x k } k∈N generated by the gradient descent algorithm satisfies

where f * is the minimum of f and x * is an optimal point with f

Proof. By the L-smoothness of the function f and the iterate update

This shows that the sequence {f (x k )} k∈N is decreasing. Moreover by convexity, we have

As a result, since f (x i ) is decreasing, we have

When f is strongly convex,

for any x, y ∈ R d . Taking both side the minimum according to y, we have

Rearranging the terms gives

A.2 Quasi-Newton methods

Proposition 18. The BFGS algorithm satisfies the first-order oracle defined in Assumption 1 if H 0 is initialized proportional to the identity matrix.

Proof. We prove by recurrence that

When k = 0, this is true if H 0 is proportional to the identity matrix. Assume that this is true for all i ≤ k. Then for k + 1, we have

Two loop recursion strategy

Input: ∇f (x k ) and two lists {s 0 ,

Initialize z = ∇f (x k ).

For i = k -1 to 0 do

Update z = H 0 z.

Output the vector z which verifies z = H k ∇g k .

Thus, instead of storing the dense matrix B k or H k , an alternative way is to store two list of vectors of s i and y i and use the two loop recursion strategy to build the matrix. This relax the requirement of the matrix storage and provide the possibility to limited memory variant.

Appendix B Proof and technical results of Chapter 3 B.1 Useful lemmas

Lemma 16 (Simple lemma on quadratic functions). Consider three vectors x, y and z in R d . Then, for all θ > 0,

Proof.

Lemma 17 (Simple lemma on non-negative sequences). Consider a increasing sequence (S k ) k≥0 and two non-negative sequences (a k ) k≥0 and (u k ) k≥0 such that for all k,

Then,

Appendix C

Short review regarding Quasi-Newton approximations

We have been presenting Quasi-Newton methods as a "natural" extension of Newton's method. But readers may wonder how the update of BFGS algorithm has been discovered. The expression of the update (4.2) is indeed not trivial at all. In order to provide a better understanding about Quasi-Newton methods, we perform in this chapter a literature review of them, aiming to give intuitions behind their constructions.

The whole story starts from the estimation of the Hessian matrix. For the purpose of reducing the computational complexity, we would like to approximate the Hessian matrix instead of evaluating it. More formally speaking, given a twice differentiable function f and a point x, we are looking for a reasonable approximation 1 B of the Hessian matrix ∇ 2 f (x), based on gradient informations. A naive way is to apply the finite difference method, approximating each column by B(:

where h > 0 and (e 1 , . . . , e d ) is the canonical basis. This yields the discrete Newton's method which replaces the Hessian matrix in Newton's by the above approximation. However, in terms of computational complexity, discrete Newton's method requires d evaluations of gradients at each iteration, which is as expensive as evaluation of the Hessian matrix. Thus, in order to reduce the per-iteration cost, only a small amount of gradients are allowed to be evaluated, say one or two gradients, at each iteration. The main difficulty is that the Hessian matrix lives in the vector space of d 2 dimension but the informations we have is in the order of d.

where z k is any vector non-orthogonal to s k . More precisely, the solutions of the secant equation forms an affine subspace V = {B | y k = Bs k } and the update (C.8) is the projection of B k on V along the rank one subspace {uz T k | u ∈ R d } generated by z k . The vector z k represents the "direction" in which we project the metric. It is immediate to see that

as long as B k is symmetric and ⟨y k -B k s k , s k ⟩ ̸ = 0. This update is known as the symmetric single rank formula, called the SR1 method [START_REF] Broyden | Quasi-newton methods and their application to function minimisation[END_REF]. One may use the to develop an update of the inverse Hessain matrix, see Table C.1. As its name indicates, the SR1 method ensures the metric B k being symmetric, but unfortunately it does not guarantee the positive definiteness, (a counter example can be found in Example 11.2 in [START_REF] Nocedal | Numerical optimization[END_REF]). Moreover, the vectors s k and y k -B k s k can be orthogonal, in which case the algorithm is not defined. Since the SR1 update is the unique rank one update conserving the symmetry, there is no hope to ensure both the symmetry and the positive definiteness by a rank one update. This is the reason why rank two update are investigated.

A generic routine for constructing a rank two update has been proposed by [START_REF] Powell | A new algorithm for unconstrained optimization[END_REF] for deriving extensions of Broyden's method. Given the past metric B, vectors y and s, fix a direction of projection z, the procedure iteratively constructs the following sequence (C j ) j∈N with C 0 = B and

Appendix D Technical report: Catalyst Acceleration for Gradient-Based Non-Convex Optimization

Courtney Paquette University of Washington

Hongzhou Lin Inria

Dmitriy Drusvyatskiy University of Washington

Julien Mairal Inria

Zaid Harchaoui University of Washington

D.1 Introduction

We consider optimization problems of the form

Here each function f i : R p → R is smooth, the regularization ψ : R p → R may be nonsmooth, and R := R ∪ {∞}. By considering extended-real-valued functions, this composite setting also encompasses constrained minimization by letting ψ be the indicator function of the constraints on x. Minimization of regularized empirical risk objectives of form (D.1) is central in machine learning. Whereas a significant amount of work has been devoted to this composite setting for convex problems, leading in particular to fast incremental algorithms [see, e.g., Defazio et al., 2014a[START_REF] Lan | An optimal randomized incremental gradient method[END_REF][START_REF]Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF][START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF][START_REF] Woodworth | Tight complexity bounds for optimizing composite objectives[END_REF], GRADIENT-BASED NON-CONVEX OPTIMIZATION Xiao and Zhang, 2014], the question of minimizing efficiently (D.1) when the functions f i and ψ may be nonconvex is still largely open today.

Yet, nonconvex problems in machine learning are of high interest. For instance, the variable x may represent the parameters of a neural network, where each term f i (x) measures the fit between x and a data point indexed by i, or (D.1) may correspond to a nonconvex matrix factorization problem (see Section D.6). Besides, even when the data-fitting functions f i are convex, it is also typical to consider nonconvex regularization functions ψ, for example for feature selection in signal processing [START_REF] Hastie | Statistical Learning With Sparsity: The Lasso And Generalizations[END_REF]. In this work, we address two questions from nonconvex optimization:

1. How to apply a method for convex optimization to a nonconvex problem? 2. How to design an algorithm which does not need to know whether the objective function is convex while obtaining the optimal convergence guarantee if the function is convex?

Several pioneering works attempted to transfer ideas from the convex world to the nonconvex one, see, e.g., [Ghadimi andLan, 2016, Ghadimi et al., 2015]. Our paper has a similar goal and studies the extension of Nesterov's acceleration for convex problems [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF] to nonconvex composite ones. Unfortunately, the concept of acceleration for nonconvex problems is unclear from a worst-case complexity point of view: gradient descent requires O(ε -2 ) iterations to guarantee a gradient norm smaller than ε [START_REF] Cartis | On the complexity of steepest descent, newton's and regularized newton's methods for nonconvex unconstrained optimization problems[END_REF][START_REF] Cartis | On the complexity of finding first-order critical points in constrained nonlinear optimization[END_REF]. Under a stronger assumption that the objective function is C 2 -smooth, state-of-the-art methods [e.g., [START_REF] Carmon | Accelerated methods for non-convex optimization[END_REF] achieve a marginal gain with complexity O(ε -7/4 log(1/ε)), and do not appear to generalize to composite or finite-sum settings. For this reason, our work fits within a broader stream of recent research on methods that do not perform worse than gradient descent in the nonconvex case (in terms of worst-case complexity), while automatically accelerating for minimizing convex functions. The hope when applying such methods to nonconvex problems is to see acceleration in practice, by heuristically exploiting convexity that is "hidden" in the objective (for instance, local convexity near the optimum, or convexity along the trajectory of iterates).

The main contribution of this part is a generic meta-algorithm, dubbed 4WD-Catalyst, which is able to use a gradient-based optimization method M, originally designed for convex problems, and turn it into an accelerated scheme that also applies to nonconvex objective functions. The proposed 4WD-Catalyst can be seen as a 4-Wheel-Drive extension of Catalyst [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]] to all optimization "terrains" (convex and nonconvex), while Catalyst was originally proposed for convex optimization. Specifically, without knowing whether the objective function is convex or not, our algorithm may take a method M designed for convex optimization problems with the same structure as (D.1), e.g., SAGA [Defazio et al., 2014a], SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF], and apply M to a sequence of sub-problems such that it asymptotically provides a stationary point of the nonconvex objective. Overall, the number of iterations of M to obtain GRADIENT-BASED NON-CONVEX OPTIMIZATION such quantities can not be easily bounded. In particular, first order methods seek points whose subgradient is small. Since small subgradients do not necessarily imply small function values in a non-convex setting, first order methods only test is for small subgradients. In contrast, in the convex setting, small subgradients imply small function values; thus a first order method in the convex setting can "test" for small function values. Hence, we can not direct application of catalyst [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] which uses the functional gap as a stopping criteria. Because we are working in the nonconvex setting, we include a stationarity stopping criteria.

We propose the following two types of stopping criteria:

Without the descent condition, the stationarity condition is insufficient for defining a good stopping criterion because of the existence of local maxima in nonconvex problems. In the nonconvex setting, local maxima and local minima satisfy the stationarity condition. The descent condition ensures the iterates generated by the algorithm always decrease the value of objective function f ; thus ensuring we move away from local maxima. The second criterion, adaptive stationary condition, provides a flexible relative tolerance on termination of algorithm used for solving the subproblems; a detailed analysis is forthcoming.

In Basic 4WD-Catalyst , we use both the stationary condition and the descent condition as a stopping criteria to produce the point x:

(D.11)

For the point x, our "acceleration" point, we use a modified stationary condition:

12)

The k + 1 factor guarantees Basic 4WD-Catalyst accelerates for the convex setting. To be precise, Equation D.27 in the proofs of Theorem 13 and Theorem 13 uses the k + 1 to ensure a produce converges. Note, we do not need the descent condition for x, as the functional decrease in x is enough to ensure the sequence {f (x k )} k≥1 is monotonically decreasing.

D.3.2 Convergence analysis.

We present here the theoretical properties of Algorithm 9. In this first stage, we do not take into account the complexity of solving the subproblems (D.5) and (D.7). For the next two theorems, we assume that the stopping criteria for the proximal subproblems are satisfied in each iteration of Algorithm 9.

Theorem 12 (Outer-loop complexity for Basic 4WD-Catalyst; non-convex case). For any κ > 0 and N ≥ 1, the iterates generated by Algorithm 9 satisfy

D.4. THE 4WD-CATALYST ALGORITHM

Bounding the required iterations when κ > ρ. When κ > ρ and τ κ is explicitly given as a function of κ, the number of inner calls to M is easy to compute. First, we define the notation

and note that f κ (x; y) = f 0 (x; y) + ψ(x). The number of inner calls is given by the following formula:

Theorem 14. Suppose κ > ρ. Define the initialization point z 0 by

If M is initialized at the point z 0 , then 1. the output z T satisfies f κ (z T ; y) ≤ f κ (z 0 ; y) (descent condition) and dist(0, ∂f κ (z T ; y)) ≤ κ ∥z T -y∥ (adaptive stationary condition) in at most T κ iterations where

2. the output z S satisfies dist(0, ∂f κ (z S ; y)) ≤ κ k+1 ∥z S -y∥ (modified adaptive stationary condition) in at most S κ log(k + 1) iterations where

) .

The proof is technical so we will leave it to the end of the supplementary materials (see Section D.7). When ρ is known, such that κ can be chosen large enough to guarantee linear convergene of M, the global complexity for Algorithm 9 is immediate. Herein lies a problemthe smoothing parameter κ drives the outer complexity (see Theorem 12 and Theorem 13). By choosing κ too large, we may fail to obtain the desired optimal complexity guarantees; choosing κ too small the method M converges at suboptimal rates.

keeping the conclusions of Theorem 12 valid. Under our assumptions on M (see Section D.4), we show that the procedure Auto-adapt(x, κ, ε, T ) indeed terminates regardless of the number of iterations T chosen with the assumption that τ κ → 1 as κ → ∞ (see Lemma 24). If the later is not the case, we can still choose T sufficiently large so that Auto-adapt(x, κ, ε, T ) terminates (See Proposition 21). When a formula for τ κ is explicitly given as a function of κ, the number of calls made by Auto-adapt(x, κ, ε, T ) to M is easy to compute.

Algorithm 11 Auto-adapt (x, κ, T )

by running T iterations of M initializing from

then go to repeat with κ → 2κ. else go to output. output (z T , κ).

New stopping criteria for the subproblems Recall, equations (D.11) and (D.12) rely on a fixed parameter κ > ρ. To derive the global complexity results for Basic 4WD-Catalyst that match optimal convergence guarantees, we make a distinction between the κ in the convex setting versus the nonconvex setting. As the weak convexity constant is unknown and κ depends on k, we introduce a parameter κ 0 > 0 as an initial guess for ρ. Algorithm 11 produces a sequence of κ k . In replacement of (D.11), our new stopping criteria to produce the point xk is

To obtain the optimal convergence guarantee in the convex setting, we introduce the parameter κ cvx , the choice of which is same as the smoothing parameter in [START_REF] Lin | A universal catalyst for first-order optimization[END_REF] and depends on the method M. In replacement of (D.12), the new stopping criteria for xk is dist

If in addition the function f is convex and S k is chosen so that xk satisfies (D.19), then

where x * is any minimizer of the function f .

Inner-loop Complexity

In light of Theorem 15, we must now understand how to choose T and S as small as possible, while guaranteeing that xk and xk satisfy (D.18) and (D.19) hold for each k. The quantities T and S must be set beforehand without knowing the true value of the weak convexity constant ρ. Using Theorem 14, we assert the following choices for T and S.

Theorem 16 (Inner complexity for 4WD-Catalyst : determining the values T and S). Suppose the stopping criteria are (D.18) and (D.19) as in in Theorem 15 and we choose T and S in Algorithm 10 to be the smallest numbers satisfying

for all k. In particular,

Then κ max ≤ 4L and the following hold for any index k ≥ 1:

1. Generating xk in Algorithm 10 starting at x 0 k-1 requires at most

iterations of M;

D.4. THE 4WD-CATALYST ALGORITHM

2. Generating xk in Algorithm 10 starting at y 0 k requires at most

where Õ hides universal constants and logarithmic dependencies on k, L, κ cvx , A L , and A κcvx .

Appendix D.10 is devoted to proving Theorem 16, but we outline below the general procedure and state the two main propositions (see Proposition 21 and Proposition 22).

We summarize the proof of Theorem 16 as followed:

1. When κ > ρ + L, we compute the number of iterations of M to produce a point satisfying (D.18). Such a point will become xk .

2. When the function f is convex, we compute the number of iterations of M to produce a point which satisfies the (D.19) condition. Such a point will become the point xk .

3. We compute the smallest number of times we must double κ 0 until it becomes larger than ρ + L. Thus eventually the condition 4L ≥ κ > ρ + L will occur.

4. We always set the number of iterations of M to produce xk and xk as in Step 1 and

Step 2, respectively, regardless of whether

The next proposition shows that Auto-adapt terminates with a suitable choice for xk after T number of iterations.

Proposition 21 (Inner complexity for xk ). Suppose ρ + L < κ ≤ 4L. If the method M is initialized at a point x 0 ∈ R p , where x 0 is defined Theorem 14, and is applied to the problem

for at least the number of iterations

Under the additional assumption that the function f is convex, we produce a point with (D.19) when the number of iterations, S, is chosen sufficiently large.

D.5. APPLICATIONS TO EXISTING ALGORITHMS

error, f (x)inf f < ε and in the nonconvex setting, the appropriate measure is stationarity, namely dist(0, ∂f (x)) < ε. All the algorithms considered have formulations for the composite setting with analogous convergence rates. Table D.1 summarizes our conclusions for the algorithms considered.

Derivation of smoothing parameters. The smoothing parameter, κ cvx drives the convergence rate of 4WD-Catalyst in the convex setting. To determine κ cvx , we pretend ρ = 0 and compute the global complexity of our scheme. As such, we end up with the same complexity result as in [START_REF] Lin | A universal catalyst for first-order optimization[END_REF]. Following their work, the rule of thumb is to maximize the ratio τ κ / √ L + κ. On the other hand, the choice of κ 0 is independent of M; it is an initial lower estimate for the weak convexity constant, ρ. We provide a detailed derivation of all the variables for each of the considered algorithms in the appendix D.4.

Convergence results for existing algorithm. Table D.1 presents convergence rates for SAGA [Defazio et al., 2014a], (prox) SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF], and gradient descent (FG). In the appendix, we provide a complete derivation of the rate for each of the algorithms.

)

SVRG [START_REF] Xiao | A proximal stochastic gradient method with progressive variance reduction[END_REF]] not avail.

)

SAGA [Defazio et al., 2014a] not avail.

1 -Comparison of rates of convergence, before and after the 4WD-Catalyst , resp. in the non-convex and convex cases. For the comparision, in the convex case, we only present the number of iterations to obtain a point x satisfying f (x)f * < ε. In the non-convex case, we show the number of iterations to obtain a point x satisfying dist(0, ∂f (x)) < ε.

Full gradient method.

A first illustration is the algorithm obtained when accelerating the regular "full" gradient (FG). Here, the optimal choice for κ cvx is L. In the convex setting, we get an accelerated rate of O(n √ L/ε log(1/ε)) which agrees with Nesterov's accelerated variant (AFG) up to logarithmic factors. On the other hand, in the nonconvex setting, our approach achieves no worse rate than O(nL/ε 2 log(1/ε)), which agrees with the standard gradient descent up to logarithmic factors. We note that under stronger assumptions, namely C 2 -smoothness of the objective, the accelerated algorithm in [Carmon et al., 2017] achieves the same rate as (AFG) for the convex setting and O(ε -7/4 log(1/ε)) for the nonconvex setting. Their approach, however, does not extend to composite setting nor to stochastic methods. Our marginal loss is the price we pay for considering a much larger class of functions. GRADIENT-BASED NON-CONVEX OPTIMIZATION and the subgradient norm (bottom). From left to right, we vary the size of dataset from n = 1 000 to n = 100 000. learning a dictionary in R m×p with p = 256 elements, on a set of whitened normalized image patches of size m = 8 × 8. Parameters are standard ones in this literature [START_REF] Mairal | Sparse modeling for image and vision processing[END_REF]that is, a small value µ = 1e -5, and λ = 0.25, leading to sparse matrices A (on average ≈ 4 non-zero coefficients per column of A).

Neural networks. We consider now simple binary classification problems for learning neural networks. Assume that we are given a training set {a i , b i } n i=1 , where the variables b i in {-1, +1} GRADIENT-BASED NON-CONVEX OPTIMIZATION Initial estimates of L. The proposed algorithm 4WD-Catalyst requires an initial estimate of the Lipschitz constant L. In the problems we are considering, there is no simple closed form formula available to compute an estimate of L. We use following heuristics to estimate L:

1. For matrix factorization, it can be shown that the function f i defined in (D.22) is differentiable according to Danskin's theorem [see Bertsekas [START_REF] Bertsekas | Nonlinear programming[END_REF], Proposition B.25] and its gradient is given by

If α i (D) is in fact D-independant, the gradient is linear in D and thus admit ∥α i ∥ 2 as Lipschitz constant. Thus when initializing our algorithm at D 0 , we use a Lasso/Elasticnet solver spams to find α i (D 0 ) for any i ∈ [1, n] and use max i∈ [1,n] ∥α i (D 0 )∥ 2 as an estimate of L.

2. For neural network, the formulation we are considering is actually differentiable. We randomly generates two pairs of weight vectors (W 1 , W 2 ) and (W ′ 1 , W ′ 2 ) and use the quantity

} as an estimate of the Lipschitz constant, where f i denotes the loss function respect to i-th training sample (a i , b i ). We separate weights in each layer to estimate the Lipschitz constant per layer. Indeed the scales of the weights can be quite different across layers.

Computational cost. For the Convex-SVRG and Nonconvex-SVRG, one iteration corresponds to one pass over the data in the plots. On the one hand, since 4WD-Catalyst-SVRG solves two sub-problems per iteration, the cost per iteration is twice that of the Convex-SVRG and Nonconvex-SVRG. On the other hand, in the experiments, we observe that, every time acceleration occurs, then xk is almost always preferred to xk in step 4 of 4WD-Catalyst, hence half of the computations are in fact not performed when running 4WD-Catalyst-SVRG. We report in Figure D.5 an experimental study where we vary S on the neural network example. In terms of number of iterations, of course, the larger S k the better the performance. This is not surprising as we solve each subproblem more accurately. Nevertheless, in terms of number of gradient evaluations, the relative performance is reversed. There is clearly no benefit to take larger S k . This justifies in hindsight our choice of setting S = 1.

Experimental conclusions.

In matrix factorization experiments, we observe that 4WD-Catalyst-SVRG always outperforms the competing algorithms. Nonconvex-SVRG has slower convergence in objective values and Convex-SVRG is not always converging; see in particular right panel in Fig. D.1. Therefore 4WD-Catalyst-SVRG offers a more stable option than GRADIENT-BASED NON-CONVEX OPTIMIZATION where f 0 : R p → R is µ-strongly convex and smooth with L-Lipschitz continuous gradient ∇f 0 , and ψ : R p → R is a closed convex function with a computable proximal map prox βψ (y) := arg min

Let x * be the minimizer of h and h * be the minimal value of h. In general, there are three types of measures of optimality that one can monitor: ∥xx * ∥ 2 , h(x)h * , and dist(0, ∂h(x)).

Since h is strongly convex, the three of them are equivalent in terms of convergence rates if one can take an extra prox-gradient step:

To see this, define the displacement vector, also known as the gradient mapping, g L (x) := L(x -[x] L ), and notice the inclusion g L (x) ∈ ∂h([x] L ). In particular g L (x) = 0 if and only if x is the minimizer of h. These next inequalities follow directly from Theorem 2.2.7 in [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]:

Thus, an estimate of any one of the four quantities ∥x-x * ∥, h(x)-h * , ∥g L (x)∥, or dist(0, ∂h(x)) directly implies an estimate of the other three evaluated either at x or at [x] L .

D.8 Theoretical analysis of the basic algorithm

We present here proofs of the theoretical results of the paper. Althroughout the proofs, we shall work under the Assumptions on f stated in Section D.3 and the Assumptions on M stated in Section D.4.

D.8.1 Convergence guarantee of Basic 4WD-Catalyst

In Theorem 12 and Theorem 13 under an appropriate tolerance policy on the proximal subproblems (D.5) and (D.7), Basic 4WD-Catalyst performs no worse than an exact proximal point method in general, while automatically accelerating when f is convex. For this, we need the following observations. Lemma 22 (Growth of (α k )). Suppose the sequence {α k } k≥1 is produced by Algorithm 9. Then, the following bounds hold for all k ≥ 1:

.

GRADIENT-BASED NON-CONVEX OPTIMIZATION

We combine the above inequality with (D.23) to deduce

Next, suppose the function f is convex. Our analysis is similar to that of [Tseng, 2008, Beck and[START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]. Using the stopping criteria (D.12), fix an

For any x ∈ R n , Equation (D.10), and the strong convexity of the function

We substitute x = α k x * + (1α k )x k-1 where x * is any minimizer of f . Using the convexity of f , the norm of ξ k , and Equations (D.6) and (D.8), we deduce

Completing the square on Equation (D.25), we obtain

Hence, we deduce See [START_REF] Parikh | Proximal algorithms[END_REF] for a survey of proximal maps. Typical linear convergence guarantees of an optimization algorithm assert existence of constants A ∈ R and τ ∈ (0, 1) satisfying

for each t = 0, 1, 2, . . . , ∞. To bring such convergence guarantees into the desired form (D.14), define the prox-gradient step

and the displacement vector

and notice the inclusion g L (z) ∈ ∂h([z] L ). The following inequality follows from [START_REF] Nesterov | Gradient methods for minimizing composite functions[END_REF]:

Thus, the linear rate of convergence (D.28) implies

which is exactly in the desired form (D.14).

D.9.1 Convergence analysis of the adaptive algorithm: 4WD-Catalyst

First, under some reasonable assumptions on the method M (see Section D.4), the sub-method Auto-adapt terminates.

Lemma 24 (Auto-adapt terminates). Assume that τ κ → 1 when κ → +∞. The procedure Auto-adapt(x, κ, ε, T ) terminates after finitely many iterations.

Proof. Due to our assumptions on M and the expressions f κ (

) .

(D.29)

D.9. ANALYSIS OF 4WD-CATALYST AND AUTO-ADAPT

Since τ κ tends to one, for all sufficiency large κ, we can be sure that the right-hand-side is smaller than ε 2 . On the other hand, for κ > ρ, the function f κ (•; x) is (κρ)-strongly convex and therefore we have dist 2 (0, ∂f κ (z

Combining this with (D.29), we deduce

)

.

Letting κ → ∞, we deduce f κ (z T ; x) ≤ f (x), as required. Thus the loop indeed terminates.

We prove the main result, Theorem 15, for 4WD-Catalyst.

Proof of Theorem 15. The proof closely resembles the proofs of Theorem 13 and Theorem 13, so we omit some of the details. The main difference in the proof is that we keep track of the effects the parameters κ cvx and κ 0 have on the inequalities as well as the sequence of κ k . Since {f (x k )} k≥0 are monotonically decreasing, we deduce

Using the adaptive stationary condition (D.18), we apply Lemma 23 with ε

We combine the above inequality with (D.30) to deduce

Suppose the function f is convex. Using in the stopping criteria (D.19) in replacement of (D.11), we deduce a similar expression as (D.25):

Hence, we deduce

Following the standard recursion argument as in the proofs of Theorem 13 and Theorem 13, we conclude

The last inequality follows because 0 < A k-1 ≤ 1. Iterating N times, we deduce

thus the result is shown. Summing up (D.31) from j = N + 1 to 2N , we obtain

Combining this inequality with (D.32), the result is shown.

D.10. INNER-LOOP COMPLEXITY: PROOF OF THEOREM 16

D.10 Inner-loop complexity: proof of Theorem 16

Recall, the following notation Note that if we are not in the composite setting and κ > ρ, then f κ (•, y) is (κ + L)-strongly convex. Using standard bounds for strongly convex functions, Equation (D.34) follows (see [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]). We next show an important lemma for deducing the inner complexities. The last inequality follows because of the assumption ε ≤ κ-ρ 2 . Rearranging the terms above, we get the desired result.

These two lemmas together give us Theorem 14.

Proof of Theorem 14. First, we prove that z T satisfies both adaptive stationary condition and the descent condition. Recall, the point y 0 is defined to be the prox or y depending on if f κ (•; y) is a composite form or smooth, respectively (see statement of Theorem 14). By Lemma 25 (or the remark following it), the starting y 0 satisfies Proof of Proposition 21. The idea is to apply Theorem 14. Since the parameter A κ increases with κ, then we upper bound it by A κ k ≤ A 4L . Moreover, we have κ-ρ ≥ ρ+L-ρ = L. Lastly, since τ κ is increasing in κ, we know 1 τκ ≤ 1 τ L . Plugging these bound into Theorem 14, we see that for any smoothing parameter κ satisfying ρ + L < κ < 4L, we get the desired result.

Next, we compute the maximum number of times we must double κ until κ > ρ + 2L.

Lemma 27 (Doubling κ). If we set T and S according to Theorem 16, then the doubling of κ 0 will terminate as soon as κ > ρ + L. Thus the number of times κ 0 must be doubled in Algorithm 11 is at most log .

Since κ is doubled (Algorithm 11) and T is chosen as in Proposition 21 , the maximum the value κ, κ max , takes is 2(ρ + L) ≤ 4L.

D.10.2 Inner complexity for xk : proof of Proposition 22

In this section, we prove Proposition 22, an inner complexity result for the iterates xk . Recall that the inner-complexity analysis for xk is important only when f is convex (see Section D.4). Therefore, we assume throughout this section that the function f is convex. We are now ready to prove Proposition 22.

Proof of Proposition 22. The proof immediately follows from Theorem 14 by setting κ = κ cvx and ρ = 0 as the function f is convex.