Piotr Vincent Guillaume

Damien Aymeric

Théophile, Rémi, Gauthier Julia Nicolas

Guilhem, Christophe, Nastia, Maxime, Dmitri, Tatiana, Gül, Vadim Alexandre ; Matthew

Thomas E Dmitri

Antoine Yana

Maxime Nicolas Jérémy

Aymeric Nathan Lou

Keywords: Convex optimization, Error bound, Sparsity, Sharpness, Structured models with partitions of the data. v Convex optimization, Error bound, Sparsity, Sharpness, Structured models with partitions of the data

In numerous fields such as machine learning, operational research or circuit design, a task is modeled by a set of parameters to be optimized in order to take the best possible decision. Formally, the problem amounts to minimize a function describing the desired objective with iterative algorithms. The development of these latter depends then on the characterization of the geometry of the function or the structure of the problem.

In a first part, this thesis studies how sharpness of a function around its minimizers can be exploited by restarting classical algorithms. Optimal schemes are presented for general convex problems. They require however a complete description of the function that is rarely available. Adaptive strategies are therefore developed and shown to achieve nearly optimal rates. A specific analysis is then carried out for sparse problems that seek for compressed representation of the variables of the problem. Their underlying conic geometry, that describes sharpness of the objective, is shown to control both the statistical performance of the problem and the efficiency of dedicated optimization methods by a single quantity.

A second part is dedicated to machine learning problems. These perform predictive analysis of data from large set of examples. A generic framework is presented to both solve the prediction problem and simplify it by grouping either features, samples or tasks. Systematic algorithmic approaches are developed by analyzing the geometry induced by partitions of the data. A theoretical analysis is then carried out for grouping features by analogy to sparse methods.

Résumé

Dans de nombreux domaines tels que l'apprentissage statistique, la recherche opérationnelle ou encore la conception de circuits, une tâche est modélisée par un jeu de paramètres que l'on cherche à optimiser pour prendre la meilleure décision possible. Mathématiquement, le problème revient à minimiser une fonction de l'objectif recherché par des algorithmes itératifs. Le développement de ces derniers dépend alors de la géométrie de la fonction ou de la structure du problème.

Dans une première partie, cette thèse étudie comment l'acuité d'une fonction autour de ses minima peut être exploitée par le redémarrage d'algorithmes classiques. Les schémas optimaux sont présentés pour des problèmes convexes généraux. Ils nécessitent cependant une description complète de la fonction, ce qui est rarement disponible. Des stratégies adaptatives sont donc développées et prouvées être quasioptimales. Une analyse spécifique est ensuite conduite pour les problèmes parcimonieux qui cherchent des représentations compressées des variables du problème. Leur géométrie conique sous-jacente, qui décrit l'acuité de la fonction de l'objectif, se révèle contrôler à la fois la performance statistique du problème et l'efficacité des procédures d'optimisation par une seule quantité.

Une seconde partie est dédiée aux problèmes d'apprentissage statistique. Ceux-ci effectuent une analyse prédictive de données à l'aide d'un large nombre d'exemples. Une approche générique est présentée pour à la fois résoudre le problème de prédiction et le simplifier en groupant soit les variables, les exemples ou les tâches. Des méthodes algorithmiques systématiques sont développées en analysant la géométrie induite par une partition des données. Une analyse théorique est finalement conduite lorsque les variables sont groupées par analogie avec les méthodes parcimonieuses.

Mots

Bibliography xii

Extended abstract

This thesis studies how optimization procedures can take advantage of the geometry of a problem around its solutions in a first part and how they can handle combinatorial structures in a second part. Its outline is the following.

Part 1

The first part analyzes sharpness of functions around their minimizers by so-called error bounds, first discovered by Łojasiewicz.

Chapter 1: This chapter sets up the framework of optimization studied in the first part. Starting from a brief introduction to convex optimization, it further presents Łojasiewicz inequality that guarantees convergence of restart schemes developed in this part. We also briefly present an interpretation of accelerated algorithms that we restart.

Chapter 2: This chapter presents restart schemes of classical algorithms for convex optimization. They provide optimal rates of convergence under generic error bounds on the minimum of the function. Precisely, we study convex optimization problems of the form minimize 𝑓 (𝑥) in variable 𝑥 ∈ R 𝑑 , where 𝑓 is a convex function.

Problem setting. To tackle both smooth and non-smooth problems, we assume that there exist 1 ≤ 𝑠 ≤ 2 and 𝐿 > 0 such that

‖∇𝑓 (𝑥) -∇𝑓 (𝑦)‖ ≤ 𝐿‖𝑥 -𝑦‖ 𝑠-1 , for all 𝑥, 𝑦 ∈ R 𝑑 , (1)
where ∇𝑓 (𝑥) is any sub-gradient of 𝑓 at 𝑥. This encompasses smooth convex optimization for 𝑠 = 2 and classical assumption for non-smooth convex optimization for 𝑠 = 1. The optimal rate of convergence for such functions is bounded as 𝑂(1/𝑁 𝜌 ), where 𝑁 is the total number of iterations and

𝜌 = 3𝑠/2 -1,
which gives 𝜌 = 2 for smooth functions and 𝜌 = 1/2 for non-smooth functions.

1 Furthermore, we assume that 𝑓 satisfies a Hölderian error bound on a compact 𝐾 that contains the set of minimizers 𝑋 * of the problem, that is, there exist 𝑟 ≥ 1, and 𝜇 > 0 such that 𝜇𝑑(𝑥, 𝑋 * ) 𝑟 ≤ 𝑓 (𝑥) -𝑓 * for all 𝑥 ∈ 𝐾 (2)

where 𝑓 * is the minimum of 𝑓 and 𝑑(𝑥, 𝑋 * ) is the distance from 𝑥 to 𝑋 * , the set of minimizers of the problem. This encompasses sharp functions for 𝑟 = 1, strongly convex functions for 𝑟 = 2 and it is known to be satisfied for a broad class of functions as the Łojasiewicz inequality.

The smoothness assumption (1) defines an upper bound on the function. Combined with the previous lower bound, we observe that necessary 𝑠 ≤ 𝑟. The convergence rate of our restart schemes will then depend on the following condition number, based on exponents 𝑠 ad 𝑟, 𝜏 = 1 -𝑠/𝑟 ∈ [0, 1[.

Optimal restart schemes.

Under these assumptions, we restart classical algorithms for convex optimization, namely Nesterov's accelerated algorithm if the function is smooth (𝑠 = 2) or a fast universal gradient algorithm for general convex functions satisfying equation (1). Precisely, we run these algorithms until some number of iterations scheduled in advance, stop them and restart them from the last iterate. If all parameters of the function are known, we retrieve optimal rates of convergence for the class of functions satisfying our hypotheses, i.e., the precision reached at the last point x is upper bounded as 𝑓 (x) -𝑓 * = 𝑂 ( exp(-𝜅 -𝑠 2𝜌 𝑁 ) ) , when 𝜏 = 0, while,

𝑓 (x) -𝑓 * = 𝑂 ( 𝜅 𝑠 2𝜏 𝑁 -𝜌 𝜏 ) , when 𝜏 > 0,
where 𝑁 is the total number of iterations and 𝜅 is a generalized condition number that matches the classical one for smooth and strongly convex functions. The error bound assumption, incorporated in the parameter 𝜏 , therefore results in faster rates than the ones obtained with only the smoothness hypothesis. Our proof is simple and the use of universal gradient algorithms enables to cover the non-smooth case.

Adaptive restart schemes.

We then develop an adaptive restart scheme if the function is smooth (𝑠 = 2) and if the parameters 𝜇, 𝑟 of the error bound (2) are unknown. Namely, for a fixed budget of iterations, a log-scale grid-search on parameters 𝜇, 𝑟 is proven to be nearly optimal up to an additional cost of log(𝑁 ). We also develop restart schemes using a stopping criterion based on the gap 𝑓 (𝑥) -𝑓 * , when the optimal value 𝑓 * is known in advance. These methods do not need any parameters of the function and are proven to be optimal.

Extensions.

Finally, we extend our results for composite problems in non-Euclidean settings. This enables to treat constrained or ℓ 1 regularized problems for which our restarts are shown empirically to outperform plain implementations of accelerated algorithms.

Chapter 3: This chapter presents sparse problems that originally attempt to decode a vector with few non-zero coordinates given some linear observations of it. Dedicated optimization procedures and recovery problems of group sparse vectors or low rank matrices are discussed.

Chapter 4: This chapter studies how optimization complexity and recovery performance of sparse problems are related. Namely, we study recovery problems of a sparse vector 𝑥 * ∈ R 𝑛 given 𝑝 linear observations 𝑏 𝑖 ≈ 𝑎 𝑇 𝑖 𝑥 * for 𝑖 = 1, . . . , 𝑛. If observations are exact, 𝑥 * is decoded by the exact recovery problem minimize ‖𝑥‖ 1 subject to 𝐴𝑥 = 𝑏.

(3) in 𝑥 ∈ R 𝑛 . Otherwise, the robust recovery problems reads minimize

‖𝑥‖ 1 subject to ‖𝐴𝑥 -𝑏‖ 2 ≤ 𝜖, (4)
in 𝑥 ∈ R 𝑛 , where 𝜖 is a tolerance on the noise in the observations. The performance of these decoding procedures to recover 𝑥 * has been extensively studied in the compressed sensing literature, either to know how many observations are needed to retrieve 𝑥 * in the exact case or how sensitive is the robust problem to noise. Here, we relate these statistical measures of performance to the computational complexities of these problems.

Exact recovery.

For problem (3), we show that recovery of 𝑥 * is equivalent to the sharpness of the problem around its minimizers, which reads 𝛾‖𝑥 -𝑥 * ‖ 1 < ‖𝑥‖ 1 -‖𝑥 * ‖ 1 , for all 𝑥 s.t. 𝐴𝑥 = 𝑏 where 0 ≤ 𝛾 < 1. We develop then linearly convergent restart schemes of the smoothing technique of Nesterov, whose rate is controlled by the sharpness constant 𝛾. A statistical analysis reveals that 𝛾 is then controlled by the recovery threshold of the problem, i.e., the maximal sparsity level of 𝑥 * such that it can be recovered. Overall, this shows that the more performing the decoding procedure is, the easier the decoding problem is.

Robust recovery. The sensitivity to noise of the robust recovery problem is wellknown to be measured by conically restricted singular values of the coding matrix 𝐴. We show that these latter control the sharpness of the exact recovery problem, hence linear rate of our restart scheme, but also the complexity of oracle based techniques such as the ellipsoid method. Overall, we identify a single quantity that controls both the recovery performance and the computational complexity of decoding procedures. This is then further illustrated by numerical experiments. depends on a restricted isometric property of the problem along the subspaces defined by partitions. While for sparsity problem this condition is satisfied for a number of samples that is only a fraction of the number of features, the problem of grouping features requires as many samples as features to ensure convergence. The underlying combinatorial problem appears too complex to be solved with few samples by a projected gradient scheme.

Extension to sparse and grouped vectors. Finally a projected gradient scheme to both select and group variables is presented which enjoys the theoretical guarantees of sparse problems while highly reducing the dimensionality of the problem by grouping features.

Chapter 7: This chapter studies the problem of grouping samples while solving a prediction task. It can either be seen as a supervised clustering task or a prediction problem allowed to output diverse predictions. As for features, the problem is formulated as a supervised learning problem constrained by partitions on the data. Same strategies are used to solve it: a convex relaxation using Frank-Wolfe and a projected gradient scheme, both requiring the solution of a k-means problem to perform an iterative resolution of the problem.

Chapter 8: This chapter studies the problem of grouping tasks for classification. The underlying partitioning problem is isolated, which enables the development of a projected gradient scheme for general losses. In the special case of a squared loss, we show that the problem reduces to a k-means problem.

The publications related to this manuscript are listed below: a) Interpretation of Accelerated algorithm in Chapter 1, Section 1.4 is based on the article: Integration Methods and Accelerated Optimization Algorithms, D. Scieur, V. Roulet, F. Bach and A. d'Aspremont to

Introduction

Optimization problems consist in finding extremal values of functions, such as the minimal cost of a task, and take the general form minimize 𝑓 (𝑥)

(1.1) in variable 𝑥 ∈ R 𝑑 , where 𝑓 , the objective function, has at least one minimizer.

Analytical solutions of such problems are generally not available. Iterative algorithmic procedures are therefore developed to get an approximate solution x of (1.1). The precision achieved by a method is measured by the gap 𝑓 (x)-𝑓 * between the estimate solution 𝑓 (x) and the true solution 𝑓 * = min 𝑥 𝑓 (𝑥). The complexity of an algorithm to solve (1.1) is then defined as the number of iterations needed to achieve an accuracy 𝜀.

Development of algorithms to solve the optimization problem depend on the information available at each iteration. Throughout this thesis we are interested in first order algorithms that have access to gradient (or sub-gradients defined below) ∇𝑓 (𝑥) of the function at any querying point 𝑥 ∈ dom 𝑓 . When dimension 𝑑 is large, their cheap iteration cost makes them more appropriate than second order methods that require to compute the Hessian of the function.

Assumptions on the function can then be exploited to build appropriate methods. In this introduction we recall how convexity and smoothness were used to develop efficient algorithms. We then present Łojasiewicz inequality that merely describes the behavior of the objective function around its minimizers and show how it can be exploited by restart schemes. Finally, we briefly present how accelerated gradient algorithms for convex functions can be interpreted as discretization methods of the gradient flow.

Convex optimization

Convexity has quickly raised the attention of researchers as a key property that enables efficient resolution of optimization problems. Several books [Boyd and [START_REF] Boyd | Convex optimization[END_REF][START_REF] Bertsekas | Nonlinear programming[END_REF][START_REF] Borwein | Convex analysis and nonlinear optimization: theory and examples[END_REF][START_REF] Nocedal | Numerical Optimization[END_REF]Nesterov, 2013b] provide exhaustive presentation of the subject. Here we briefly present the key assumptions and classical algorithms to highlight the interest of characterizing the geometry of the problem around its minimizers.

Convexity

We first recall elementary definitions. A convex set, as defined below, is a set that contains any segment of its points.

Definition 1.1.1. Convex sets A set 𝑄 in R 𝑑 is convex if for any 𝑥, 𝑦 ∈ 𝑄 and 𝜃 ∈ [0, 1],

𝜃𝑥 + (1 -𝜃)𝑦 ∈ 𝑄.

Convexity of a function 𝑓 can then be defined by the convexity of its epigraph ℰ(𝑓 ) = {(𝑥, 𝑡) ∈ R 𝑑 × R : 𝑓 (𝑥) ≤ 𝑡} [START_REF] Rockafellar | Convex analysis[END_REF] or equivalently as follows.

Definition 1.1.2. Convex functions 0 th order A function 𝑓 is convex if its domain is convex and if for any 𝑥, 𝑦 ∈ dom 𝑓 and 𝜃 ∈ [0, 1],

𝑓 (𝜃𝑥 + (1 -𝜃)𝑦) ≤ 𝜃𝑓 (𝑥) + (1 -𝜃)𝑓 (𝑦).
If the function is in addition differentiable, then it is convex if it is lower bounded at any point in its domain by its linear approximation. It is detailed in following equivalent definition.

Definition 1.1.3. Convex functions 1 st order A differentiable function 𝑓 is convex if its domain is convex and if for any 𝑥 ∈ dom 𝑓 𝑓 (𝑦) ≥ 𝑓 (𝑥) + ⟨∇𝑓 (𝑥), 𝑦 -𝑥⟩, for every 𝑦 ∈ dom 𝑓 .

This property is generalized for non-differentiable convex functions through the notion of sub-gradients defined below. Definition 1.1.4. Sub-gradient Let 𝑓 be a convex function. A vector 𝑔 is called the sub-gradient of 𝑓 at point 𝑥 ∈ dom 𝑓 if 𝑓 (𝑦) ≥ 𝑓 (𝑥) + ⟨𝑔, 𝑦 -𝑥⟩, for every 𝑦 ∈ dom 𝑓 .

Closed 1 convex functions have a sub-gradient at any point in the interior of their domain [Nesterov, 2013b]. In the following, we denote ∇𝑓 (𝑥) any sub-gradient of a closed convex function 𝑓 at a given point 𝑥. Convexity offers a simple certificate of optimality : if 𝑥 ∈ R 𝑑 possesses a null (sub)gradient ∇𝑓 (𝑥) = 0 then it is a global minimizer. While convexity offers affine lower bounds at each point, quadratic lower bounds can be obtained through the notion of strong convexity as detailed in next section.

Strong convexity

We first recall general definition of strong convexity.

Definition 1.1.5. Strong convexity 0 th order A function 𝑓 is strongly convex if its domain is convex and if there exists 𝜇 ≥ 0 such that for any points 𝑥, 𝑦 ∈ dom 𝑓 and 𝜃 ∈ [0, 1],

𝑓 (𝜃𝑥 + (1 -𝜃)𝑦) ≤ 𝜃𝑓 (𝑥) + (1 -𝜃)𝑓 (𝑦) -𝜃(1 -𝜃) 𝜇 2 ‖𝑥 -𝑦‖ 2 2
It follows that strong convexity implies convexity. Besides when the function is differentiable, following equivalent definition details lower bound induced by strong convexity.

Definition 1.1.6. Strong convexity 1 st order A differentiable function 𝑓 is strongly convex if its domain is convex and if there exists 𝜇 ≥ 0 such that for any points 𝑥 ∈ dom 𝑓 , 𝑓 (𝑦) ≥ 𝑓 (𝑥) + ⟨∇𝑓 (𝑥), 𝑦 -𝑥⟩ + 𝜇 2 ‖𝑥 -𝑦‖ 2 2 , for every 𝑦 ∈ dom 𝑓 .

In general closed convex functions can also be lower bounded by quadratics in the interior of their domain, several lower bounds may exist at one point defined by the different sub-gradients of the function. Strong convexity can be refined to analyze monomial lower bounds on the function through the notion of uniform convexity. We simply give its definition for differentiable functions to highlight the resulting lower bound and refer to e.g. [START_REF] Juditski | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF]; [START_REF] Bauschke | Convex analysis and monotone operator theory in Hilbert spaces[END_REF] for further details.

Definition 1.1.7. Uniform convexity A differentiable function 𝑓 is uniformly convex if its domain is convex and if there exists 𝑟 ≥ 2, 𝜇 ≥ 0 such that for any points 𝑥 ∈ dom 𝑓 , 𝑓 (𝑦) ≥ 𝑓 (𝑥) + ⟨∇𝑓 (𝑥), 𝑦 -𝑥⟩ + 𝜇 2 ‖𝑥 -𝑦‖ 𝑟 2 , for every 𝑦 ∈ dom 𝑓 .

As it will be shown in Section 1.1.4, strong convexity or uniform convexity enables much more efficient resolution of optimization problems than plain convexity. However it is a strong assumption as it requires quadratic lower bounds at any point and any direction. We present in Section 1.2, how it can be relaxed by much weaker assumption on the function while still getting fast rates of convergence.

Smoothness

A standard way to design algorithms for (1.1) is to minimize an upper bound on the objective function 𝑓 at each iteration. These can be derived from its Taylor expansion at a point 𝑥 ∈ dom 𝑓 that reads for a differentiable function 𝑓 , 𝑓 (𝑦) = 𝑓 (𝑥) + ∫ 1 0 ⟨∇𝑓 (𝑥 + 𝜏 (𝑦 -𝑥)), 𝑦 -𝑥⟩𝑑𝜏 = 𝑓 (𝑥) + ⟨∇𝑓 (𝑥), 𝑦 -𝑥⟩ + ∫ 1 0 ⟨∇𝑓 (𝑥 + 𝜏 (𝑦 -𝑥)) -∇𝑓 (𝑥), 𝑦 -𝑥⟩𝑑𝜏 Bounds on the gradient lead then to upper bounds on the function. In this section we use Euclidean norm to define smoothness, refined assumptions are presented in Section 1.1.6.

Smooth functions

The most common assumption for differentiable function is that their gradient is Lipschitz continuous as defined below.

Definition 1.1.8. Smooth functions A differentiable function 𝑓 is smooth if there exists 𝐿 > 0 such that ‖∇𝑓 (𝑥) -∇𝑓 (𝑦)‖ 2 ≤ 𝐿‖𝑥 -𝑦‖ 2 , for every 𝑥, 𝑦 ∈ dom 𝑓 .

At a given point 𝑥 ∈ dom 𝑓 , using the Taylor expansion of 𝑓 , smoothness implies 𝑓 (𝑦) ≤ 𝑓 (𝑥) + ⟨∇𝑓 (𝑥), 𝑦 -𝑥⟩ + 𝐿 2 ‖𝑥 -𝑦‖ 2 2 , for every 𝑦 ∈ dom 𝑓 .

Denoting 𝑥 + = 𝑥 -1 𝐿 ∇𝑓 (𝑥) the minimizer of this upper bound, we get

𝑓 (𝑥 + ) -𝑓 (𝑥) ≤ - 1 2𝐿 ‖∇𝑓 (𝑥)‖ 2 2 ,
that ensures decreasing of the objective if the current point is not a stationary point (∇𝑓 (𝑥) = 0). This is then used to derive convergence rates of the gradient descent for convex functions [Nesterov, 2013b] or gradient dominated functions [START_REF] Karimi | Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition[END_REF].

Non-smooth functions

If the function is not differentiable, Taylor expansions cannot be used. However, if the function is closed convex, bound on its sub-gradients still offer local upper bounds on the function. To this end we make the following assumption.

Assumption 1.1.9. Non-smooth functions For a non-differentiable closed convex function 𝑓 , there exists 𝐿 > 0 such that ‖∇𝑓 (𝑥)‖ 2 ≤ 𝐿 2 , for every 𝑥 ∈ dom 𝑓 ,

where ∇𝑓 (𝑥) is any sub-gradient of 𝑓 at 𝑥.

At a given point 𝑥 ∈ dom 𝑓 , with a given sub-gradient ∇𝑓 (𝑥), this ensures 𝑓 (𝑦) ≤ 𝑓 (𝑥) + ⟨∇𝑓 (𝑥), 𝑦 -𝑥⟩ + 𝐿‖𝑥 -𝑦‖ 2 , for every 𝑦 ∈ dom 𝑓 .

Minimization of this upper bound may not be tractable but approximate minimizations can exist [START_REF] Devolder | First-order methods of smooth convex optimization with inexact oracle[END_REF][START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF]. Notice that non-smooth convex optimization methods such as sub-gradient [Nesterov, 2013b], dual averaging [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF], double dual averaging [START_REF] Nesterov | Quasi-monotone subgradient methods for nonsmooth convex minimization[END_REF], generally rely cutting planes arguments. Here we focus on upper bounds to derive generic bounds that encompass both smooth and non-smooth cases.

Hölder-smooth functions

Smoothness of a differentiable function 𝑓 can be refined by considering Hölder Lipschitzity of its gradient. While difficult to observe, it allows a common treatment of the smooth and non-smooth cases by the following definition.

Definition 1.1.10. Hölder smooth functions A closed convex function 𝑓 is Hölder smooth if there exist 1 ≤ 𝑠 ≤ 2, 𝐿 > 0 such that ‖∇𝑓 (𝑥) -∇𝑓 (𝑦)‖ 2 ≤ 𝐿‖𝑥 -𝑦‖ 𝑠-1 2 , for every 𝑥, 𝑦 ∈ dom 𝑓 ,

where ∇𝑓 (𝑥), ∇𝑓 (𝑦) are any sub-gradient of 𝑓 at respectively 𝑥 and 𝑦.

Notice that if 𝑠 > 1, the function is necessarily differentiable, since the set of sub-gradients at any point is then reduced to a singleton. If 𝑠 = 2, we retrieve the definition of smooth functions, if 𝑠 = 1 the assumption on non-smooth convex functions and for general 1 ≤ 𝑠 ≤ 2 we get a refined upper bound on the function at a given point 𝑥 ∈ dom 𝑓 that reads 𝑓 (𝑦) ≤ 𝑓 (𝑥) + ⟨∇𝑓 (𝑥), 𝑦 -𝑥⟩ + 𝐿 𝑠 ‖𝑥 -𝑦‖ 𝑠 2 , for every 𝑦 ∈ 𝑄.

(1.2)

Even though exact minimization of such upper bounds might not be tractable, approximate minimizations [START_REF] Devolder | First-order methods of smooth convex optimization with inexact oracle[END_REF][START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF] are possible and lead to universal gradient methods presented in next section.

Classical algorithms for convex optimization

In this section we present classical algorithms to solve unconstrained convex optimization problems that read minimize 𝑓 (𝑥) (1.3) in variable 𝑥 ∈ R 𝑑 , where 𝑓 is a closed convex function. We detail their complexity depending on additional assumptions on the function such as smoothness or strong convexity. For each class of problems, [START_REF] Nemirovskii | Problem complexity and method efficiency in optimization[END_REF] computed the least number of iterations an algorithm must build to achieve a given precision for any function in the class. These define lower complexity bounds of convex optimization problems. Algorithms that achieve these lower complexity bounds are then called optimal.

In the following we denote 𝑥 0 the starting point of the algorithms and 𝑑(𝑥 0 , 𝑋 * ) = min 𝑦∈𝑋 * ‖𝑥 0 -𝑦‖ 2 the Euclidean distance between the starting point 𝑥 0 and the set of solutions 𝑋 * = argmin 𝑦∈𝑄 𝑓 (𝑦). Detailed implementations for general convex problems are presented in Appendix A.

Gradient descent

Gradient descent dates back from [START_REF] Cauchy | Méthode générale pour la résolution des systemes d'équations simultanées[END_REF]. It simply moves along the opposite direction of the gradient at each iteration to progressively decrease the objective values as

𝑥 𝑡+1 = 𝑥 𝑡 -ℎ 𝑡 ∇𝑓 (𝑥 𝑡 ),
where 𝑥 𝑡 is the current iterate, 𝑥 𝑡+1 is the next iterate and ℎ 𝑡 is the step size. Several choices exist for the step-size depending on the assumptions made on the function.

• Smooth convex functions If the function 𝑓 is 𝐿-smooth, a constant step-size ℎ 𝑡 = 1 𝐿 ensures convergence. Gradient descent outputs then after 𝑡 iterations a point x that satisfies 𝑓 (x) -𝑓 * ≤ 𝐿 𝑡 𝑑(𝑥 0 , 𝑋 * ) 2

If 𝐿 is not known in advance, a backtracking line-search as presented by Nesterov [2013a] estimates it on the fly to get the same rate with an additional log factor cost.

• Smooth strongly convex functions If the function 𝑓 is 𝐿-smooth and 𝜇strongly convex, constant step size still ensures convergence, however a bigger step-size can be used, namely ℎ 𝑡 = 2 𝜇+𝐿 . Gradient descent shows then linear convergence, it outputs after 𝑡 iterations a point x that satisfies

𝑓 (x) -𝑓 * ≤ ( 1 -𝜇/𝐿 1 + 𝜇/𝐿 ) 𝑡 (𝑓 (𝑥 0 ) -𝑓 * ).
Notice that if 𝜇 is not known in advance, constant step size ℎ 𝑡 = 1 𝐿 still ensures linear convergence as

𝑓 (x) -𝑓 * ≤ ( 1 - 𝜇 𝐿 ) 𝑡 (𝑓 (𝑥 0 ) -𝑓 * ).
Although worse than previous one, this complexity bound illustrates that gradient descent adapts to the assumptions on the problem. If the function is strongly convex it automatically shows linear rate. In fact the function only needs to satisfy gradient dominated property presented later to get linear rate. However since gradient descent is not optimal, more elaborate schemes can get better linear rates.

• Non-smooth settings We briefly discuss classical algorithms for non-smooth optimization. They can be tackled by universal gradient algorithms or restart schemes presented later. If the function is non-smooth, sub-gradients can be used instead of gradients leading e.g. to sub-gradient methods [Nesterov, 2013b], dual averaging method [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF] or double dual averaging methods [START_REF] Nesterov | Quasi-monotone subgradient methods for nonsmooth convex minimization[END_REF], unified recently by [START_REF] Ito | A family of subgradient-based methods for convex optimization problems in a unifying framework[END_REF]. They all output after 𝑡 iterations a point 𝑡 that satisfies the optimal rate for this class of function, i.e.

𝑓 (x) -𝑓 * = 𝑂(1/𝑡 1/2 ).
However they all need to know in advance an additional parameter such as a bound 𝑅 ≥ 𝑑(𝑥 0 , 𝑋 * ) on the distance to the set of minimizers, which can be simply the size of the constrained set for constrained optimization presented later.

If the function is non-smooth but strongly convex, sub-gradient methods as presented e.g. by [START_REF] Lacoste-Julien | A simpler approach to obtaining an o (1/t) convergence rate for the projected stochastic subgradient method[END_REF] can achieve after 𝑡 iterations the optimal rate in this case, i.e.

𝑓 (x) -𝑓 * = 𝑂(1/𝑡).

Accelerated gradient descent

• Smooth convex functions In his seminal work [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate 𝑂(1/𝑘 2 )[END_REF] developed an accelerated scheme for the optimization of 𝐿-smooth convex functions 𝑓 . It was originally motivated by the construction of a lower bound on the function using its convexity. It has been revisited several times by the author himself [Nesterov, 2013a[START_REF] Nesterov | Smooth minimization of non-smooth functions[END_REF][START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF] or by others [START_REF] Tseng | On accelerated proximal gradient methods for convex-concave optimization[END_REF] to cite a few. Overall, even if implementation may vary among the authors, accelerated gradient algorithm outputs after 𝑡 iterations a point x that satisfies 𝑓 (x) -𝑓 * ≤ 4𝐿 𝑡 2 𝑑(𝑥 0 , 𝑋 * ) 2 It achieves then optimal rate of convergence for the class of smooth convex functions Here again the smoothness constant 𝐿 can be estimated on the fly [Nesterov, 2013a], such that the algorithm does not need to know any parameter of the function to be run.

• Smooth strongly convex functions Nesterov [2013b] presented also acceleration algorithms for 𝐿-smooth 𝜇-strongly convex functions 𝑓 , which outputs after 𝑡 iterations a point x that satisfies the optimal rate of convergence, precisely

𝑓 (x) -𝑓 * ≤ ( 1 - √ 𝜇 𝐿 ) 𝑡 (𝑓 (𝑥 0 ) -𝑓 * ).
As ratio 𝜇/𝐿 is often small, getting a √ 𝜇/𝐿 rate has important impact on the convergence. However while gradient descent automatically adapts to strong convexity, implementation of the accelerated algorithm varies considerably for strongly convex functions. Strong convexity parameter 𝜇 can be estimated by doing outer loops of the algorithm as mentioned by Nesterov [2013a] and further developed by [START_REF] Fercoq | Adaptive restart of accelerated gradient methods under local quadratic growth condition[END_REF].

Notice that for quadratic functions defined by positive definite matrices, accelerated algorithm of Nesterov [2013b] is not exactly optimal. Conjugate gradients [START_REF] Hestenes | Methods of conjugate gradients for solving linear systems[END_REF] or Heavy ball method [START_REF] Polyak | Some methods of speeding up the convergence of iteration methods[END_REF] achieve a slightly better rate. However these are not proven to converge for general smooth and strongly convex functions [START_REF] Lessard | Analysis and design of optimization algorithms via integral quadratic constraints[END_REF].

Universal gradient methods

Hölder smoothness was first tackled by [START_REF] Nemirovskii | Optimal methods of smooth convex minimization[END_REF] when parameters 𝑠 and 𝐿 of the function were known. Recently [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF] presented universal methods able to get optimal rates on this class of function without knowing parameters 𝑠 and 𝐿 in advance but only the desired target precision. A so-called universal gradient method was developed that simply approximates the minimization of the upper bound (1.2). An accelerated version incorporates then the convexity assumption similarly as in the classical smooth case. For a Hölder smooth function with parameters (𝑠, 𝐿) and required accuracy 𝜀, it starts from a feasible point 𝑥 0 and outputs after 𝑡 iterations a point x that satisfies

𝑓 (𝑥) -𝑓 * ≤ 𝜀 2 + 𝑐 𝑠 𝐿 2 𝑠 𝑑(𝑥 0 , 𝑋 * ) 𝜀 2 𝑠 𝑡 3𝑠-2 𝑠 𝜀 2 ,
where 𝑐 𝑠 = 2 4-2/𝑠 . If the smoothness parameters 𝑠, 𝐿 are known, target accuracy 𝜀 can be optimized which leads to optimal rates for the class of Hölder smooth convex functions [START_REF] Nemirovskii | Optimal methods of smooth convex minimization[END_REF], i.e. after 𝑡 iterations, 𝑓 (𝑥) -𝑓 * ≤ 𝑂(1/𝑡 3𝑠/2-1 ).

In particular if the function is smooth (𝑠 = 2) optimal target accuracy is 𝜀 = 0 and we retrieve optimal rate for smooth convex functions 𝑂(1/𝑡 2 ). For non-smooth functions with bounded sub-gradients, we also get the optimal rate 𝑂(1/𝑡 1/2 ).

Other algorithms

Although not studied here, quasi-Newton methods such as BFGS Broyden [1970]; [START_REF] Fletcher | A new approach to variable metric algorithms[END_REF]; [START_REF] Goldfarb | A family of variable-metric methods derived by variational means[END_REF]; [START_REF] Shanno | Conditioning of quasi-newton methods for function minimization[END_REF] that approximate the Hessian along the iterations show fast numerical convergence. Their theoretical analysis is however more complex. Recently [START_REF] Scieur | Regularized nonlinear acceleration[END_REF] proposed also regularized polynomial extrapolation methods that shows good numerical performance. Finally non-smooth uniform convex functions were tackled by restart schemes by [START_REF] Juditski | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] that lead to optimal rate.

Constrained and composite problems

Previous problems were presented for convex unconstrained problems of the form (1.3) for sake of clarity. We detail here how constrained or regularized problems are treated.

Constrained convex optimization problems read minimize 𝑓 (𝑥) subject to 𝑥 ∈ 𝑄,

(1.4) in variable 𝑥 ∈ R 𝑑 , where 𝑓 is closed convex and 𝑄 ⊂ R 𝑑2 , that encodes constraints on variables, is a closed convex set. For such problems we assume that we can compute feasible points for problem (1.1) from any 𝑥 ∈ R 𝑑 . This generally amounts to have access to the Euclidean projection on 𝑄 but refined assumptions exist such as linear minimization [START_REF] Frank | An algorithm for quadratic programming[END_REF][START_REF] Jaggi | Revisiting frank-wolfe: Projection-free sparse convex optimization[END_REF] or proximal operation on the set for mirror gradient descent algorithms [START_REF] Beck | Mirror descent and nonlinear projected subgradient methods for convex optimization[END_REF]. Additional assumptions on the function such as smoothness or strong convexity presented in Sections 1.1.3 and 1.1.2 need only to be valid on the set of constraints. Parameters 𝐿 and 𝜇 of these assumptions can then be much better when restricted to the set 𝑄.

Algorithms can then take advantage of the specific geometry of the function on the set as presented by d [START_REF] Aspremont | An optimal affine invariant smooth minimization algorithm[END_REF].

Constrained convex optimization problems belong to the more general class of composite problems that read minimize 𝑓 (𝑥) + 𝑔(𝑥)

(1.5) in variable 𝑥 ∈ R 𝑑 , where 𝑓 is closed convex and 𝑔 is a "simple" convex function in the sense that its proximal operator

prox 𝛾 𝑔 (𝑥) = argmin 𝑦∈R 𝑑 𝑔(𝑦) + 𝛾 2 ‖𝑥 -𝑦‖ 2 2
can be solved at a cheap computational cost for any 𝑥 ∈ R 𝑑 and 𝛾 ≥ 0. Constrained problems can be cast as composite problems by taking 𝑔 the indicator function of the set 𝑄 whose proximal operator is the Euclidean projection on 𝑄. But this encompasses also least squares problems regularized by sparsity inducing norms [START_REF] Bach | Optimization with sparsity-inducing penalties[END_REF], such as the ℓ 1 norm. Algorithms presented in Section 1.1.4 can be adapted to incorporate the "simple" function 𝑔 [Nesterov, 2013a;[START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Tseng | On accelerated proximal gradient methods for convex-concave optimization[END_REF]. They achieve same rates of convergence for the classes of functions that 𝑓 belongs to. However, as already mentioned, parameters of these assumptions can considerably change. In other words constraining or regularizing the problem may make the problem easier.

Beyond Euclidean geometry

We defined in Sections 1.1.3 and 1.1.2 with respect to the Euclidean norm. However, in some cases, different norms are more appropriate to describe the function on a constrained set 𝑄. To handle them, Bregman divergences of strongly convex functions are used as surrogates of the norm of interest (see e.g. [START_REF] Beck | Mirror descent and nonlinear projected subgradient methods for convex optimization[END_REF] for a detailed presentation). This leads to the family of mirror gradient [START_REF] Nemirovskii | Problem complexity and method efficiency in optimization[END_REF] or accelerated mirror gradient descents [START_REF] Krichene | Accelerated mirror descent in continuous and discrete time[END_REF] and can been incorporated in numerous algorithms such dual averaging [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF] or universal gradient algorithms [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF].

Recently [START_REF] Lu | Relatively-smooth convex optimization by first-order methods, and applications[END_REF]; [START_REF] Bauschke | A descent lemma beyond lipschitz gradient continuity: first-order methods revisited and applications[END_REF] introduced smoothness and strong convexity with respect to another convex function that refines even more the description of convex functions.

The Łojasiewicz inequality

Definitions

In his pioneering work, Łojasiewicz [1958, 1961, 1965] proved two inequalities on semi-algebraic functions. The first one lower bounds the function values by the distance to the set of minimizers. We give a simple definition that will be used in Chapter 2. Definition 1.2.1. Łojasiewicz error bound inequality A lower bounded continuous function 𝑓 on R 𝑑 with non-empty set of minimizers 𝑋 * satisfies the Łojasiewicz error bound inequality on a set 𝐾 ⊃ 𝑋 * if there exists 𝜇 > 0, 𝑟 ≥ 1 such that

𝜇𝑑(𝑥, 𝑋 * ) 𝑟 ≤ 𝑓 (𝑥) -𝑓 * , for every 𝑥 ∈ 𝐾, (1.6)
where 𝑓 * = min 𝑥∈R 𝑑 𝑓 (𝑥) and 𝑑(𝑥, 𝑋 * ) = min 𝑦∈𝑋 * ‖𝑥 -𝑦‖ 2 is the Euclidean distance between 𝑥 and the set of minimizers of 𝑓 .

The first result on error bounds dates back from [START_REF] Hoffman | On approximate solutions of systems of linear inequalities[END_REF] who analyzed systems of linear inequalities. It was then studied for convex functions by [START_REF] Robinson | An application of error bounds for convex programming in a linear space[END_REF]; [START_REF] Mangasarian | A condition number for differentiable convex inequalities[END_REF]; [START_REF] Auslender | Global regularity theorems[END_REF]. Łojasiewicz [1958, 1961, 1965] extended these results. As it merely describes function in the neighborhood of its minimizers, Łojasiewicz error bound is generally satisfied. Its proof for general subanalytic functions can be found for example in Bierstone and Milman [1988, Theorem 6.4] using topological arguments. Error bounds have been more generally expressed using residual function of the set of minimizers, see e.g. [START_REF] Pang | Error bounds in mathematical programming[END_REF] for a review. Quantitative results were developed by Luo and Luo [1994]; [START_REF] Luo | Error bounds for analytic systems and their applications[END_REF]; [START_REF] Luo | Error bounds for quadratic systems[END_REF]; [START_REF] Dedieu | Penalty functions in subanalytic optimization[END_REF] and recently further analyzed by [START_REF] Li | Global error bounds for piecewise convex polynomials[END_REF]; Li, Mordukhovich and Pham [2015]; [START_REF] Li | Error bounds for parametric polynomial systems with applications to higher-order stability analysis and convergence rates[END_REF]; [START_REF] Beck | Linearly convergent away-step conditional gradient for non-strongly convex functions[END_REF]; [START_REF] Vui | Global holderian error bound for nondegenerate polynomials[END_REF].

Although Łojasiewicz inequality is widely applicable, pathological behavior appears when the function is extremely flat or present wild oscillations around their minimizers such as 𝑓 (𝑥) = exp(-1/𝑥 2 ) or 𝑓 (𝑥) = sin(1/𝑥) exp(-1/𝑥 2 ) extended by continuity on 0. In the presented definition, the larger is 𝑟, the flatter is the function around its minimizers and so the worse it is conditioned as presented in Chapter 2. Notice finally that Łojasiewicz error bound inequality is a local assumption that may not generalize to the whole domain of 𝑓 . Yet, if this property is valid on an open set 𝐾 ⊃ 𝑋 * , it will also be valid on any compact set 𝐾 ′ ⊃ 𝐾 with the same exponent 𝑟 but with a potentially lower constant 𝜇.

The second Łojasiewicz inequality upper bounds gradient magnitude by function values. Again, we give a simple definition that can be generalized. First discovered by Łojasiewicz for analytic functions, above gradient inequality was generalized by [START_REF] Kurdyka | On gradients of functions definable in o-minimal structures[END_REF] for semi-algebraic functions by introducing desingularizing functions. It has recently been generalized to the non-smooth case by [START_REF] Bolte | The łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems[END_REF].

Variants of strong convexity

As strong convexity appears too restrictive for many problems, it was relaxed in several manners using the definitions of essential strong convexity, weak strong convexity or restricted secant inequality [START_REF] Karimi | Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition[END_REF][START_REF] Necoara | Linear convergence of first order methods for non-strongly convex optimization[END_REF][START_REF] Liu | Asynchronous stochastic coordinate descent: Parallelism and convergence properties[END_REF][START_REF] Zhang | The restricted strong convexity revisited: analysis of equivalence to error bound and quadratic growth[END_REF]. Among these assumptions, the quadratic error bound ((1.6) with 𝑟 = 2) is proven to be the weakest [START_REF] Karimi | Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition[END_REF].

More generally uniform convexity with parameter 𝑟 can easily be seen to satisfy Łojasiewicz error bound with some exponent by looking at its first order definition. By allowing exponent 𝑟 to vary, Łojasiewicz error bound can also describes sharp functions (𝑟 = 1) studied for example by [START_REF] Gilpin | First-order algorithm with 𝒪(log 1/𝜖) convergence for 𝜖-equilibrium in two-person zero-sum games[END_REF]. Combined with Hölder smoothness, these offer a generic description of convex problems.

Applications

Łojasiewicz gradient inequality (1.7) for 𝜃 = 1/2 is better known in the optimization community as the gradient dominated property introduced by [START_REF] Polyak | Gradient methods for minimizing functionals[END_REF]. It suffices to prove linear convergence of convex functions as recalled by [START_REF] Karimi | Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition[END_REF]. Generic error bounds were also considered by [START_REF] Nemirovskii | Optimal methods of smooth convex minimization[END_REF] to derive generic optimal algorithms for convex functions. However these require to know in advance parameters 𝜇 and 𝑟 which are generally hard to estimate outside the strongly convex case.

Recently the growing interest about Łojasiewicz inequalities led to new convergence analysis of many important algorithms in non-smooth and even non-convex settings [START_REF] Attouch | Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized gauss-seidel methods[END_REF][START_REF] Bolte | Proximal alternating linearized minimization for nonconvex and nonsmooth problems[END_REF][START_REF] Frankel | Splitting methods with variable metric for kurdyka-łojasiewicz functions and general convergence rates[END_REF]. Generic methods to derive convergence rates for convex optimization were notably studied by [START_REF] Bolte | From error bounds to the complexity of first-order descent methods for convex functions[END_REF]. In Chapter 2, we present simple restart schemes for convex optimization that take advantage of Łojasiewicz error bound and show adaptivity to parameters 𝜇 and 𝑟 for smooth functions.

Łojasiewicz inequalities have also an impact beyond optimization. They notably lead to the characterization of smooth or non-smooth dynamical systems [START_REF] Simon | Asymptotics for a class of non-linear evolution equations, with applications to geometric problems[END_REF][START_REF] Bolte | The łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems[END_REF] or can provide a simple framework for the development of concentration inequalities [START_REF] Blanchet | A family of functional inequalities: lojasiewicz inequalities and displacement convex functions[END_REF].

Restart schemes

A simple way to achieve acceleration for strongly convex problems is simply to restart the accelerated method for smooth convex functions at regular time intervals [Nesterov, 2013a]. Consider the unconstrained minimization of a smooth strongly convex functions. At the unique minimizer 𝑥 * , strong convexity reads

𝑓 (𝑥) -𝑓 * ≥ 𝜇 2 ‖𝑥 -𝑥 * ‖ 2 2
After 𝑡 = ⌈𝑒 √ 8𝐿/𝜇⌉ iterations of the accelerated method for smooth convex functions, one obtains a point x satisfying

𝑓 (x) -𝑓 * ≤ 4𝐿 𝑡 2 ‖𝑥 0 -𝑥 * ‖ 2 2 ≤ 8𝐿 𝜇𝑡 2 (𝑓 (𝑥 0 ) -𝑓 * ) ≤ 𝑒 -2 (𝑓 (𝑥 0 ) -𝑓 * )
Therefore repeating this operation and counting the total number of iterations leads to the optimal complexity for strongly convex functions. Observe that only the Łojasiewicz error bound implied by strong convexity is used to achieve such rate. This remark was further explored by [START_REF] Nemirovskii | Optimal methods of smooth convex minimization[END_REF] for general convex problems.

The scheduling of the restart schemes is their drawback. In the strongly convex case for example, strong convexity parameter 𝜇 must be known in advance. Numerous heuristics strategies have thus been developed to restart the accelerated method for smooth convex functions. O'Donoghue and Candes [2015] proposed to enforce monotonicity of the objective values, that is, to restart the algorithm when the objective values increase. [START_REF] Giselsson | Monotonicity and restart in fast gradient methods[END_REF] proved that such heuristics converge at least as fast as the accelerated scheme for smooth convex functions. However in practice restart show much better performance than a plain implementation of the accelerated algorithm, in particular for composite problems such as LASSO [START_REF] Tibshirani | Regression shrinkage and selection via the LASSO[END_REF]. Continuous time interpretations of the accelerated scheme were also used to justify restart heuristics. They rely on an inertia argument of the continuous time dynamic [START_REF] Su | A differential equation for modeling nesterov's accelerated gradient method: Theory and insights[END_REF][START_REF] Wibisono | A variational perspective on accelerated methods in optimization[END_REF]. However no theoretical guarantees of this heuristic were proven.

Developing algorithms adaptive to unknown strongly convex parameter was first studied by Nesterov [2013a]; [START_REF] Lin | An adaptive accelerated proximal gradient method and its homotopy continuation for sparse optimization[END_REF] by running several outer loops of the accelerated algorithm for strongly convex functions. Recently Fercoq and Qu [2016] proposed restart schemes for generic quadratic error bounds ((1.6) with 𝑟 = 2) robust to misspecification of the parameters. [START_REF] Fercoq | Adaptive restart of accelerated gradient methods under local quadratic growth condition[END_REF] refined their algorithm to provide adaptive restarts with theoretical guarantees under the same assumptions, which offer guarantees of accelerated linear rates for the LASSO.

Restart schemes were also studied for non-smooth convex optimization. [START_REF] Juditski | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] restart the dual averaging method [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF] to optimize non-smooth uniformly convex functions. They present an adaptive algorithm in this case for a fixed number of iterations and with an additional estimate of the distance to the set of minimizers. [START_REF] Gilpin | First-order algorithm with 𝒪(log 1/𝜖) convergence for 𝜖-equilibrium in two-person zero-sum games[END_REF] restart smoothing technique of [START_REF] Nesterov | Smooth minimization of non-smooth functions[END_REF] to solve zero-sum games. Interestingly, as the optimum value of their problem is known, their technique do not need additional parameters. Finally smooth convex optimization problems satisfying (1.7) with 𝜃 > 1/2 were studied by [START_REF] Yang | Adaptive accelerated gradient converging methods under holderian error bound condition[END_REF], where they use magnitude of the gradient to provide criteria for restarts.

Interpretation of accelerated algorithm

Nesterov's accelerated gradient algorithm [Nesterov, 2013b] was designed with optimal complexity in mind, but the proof relies on purely algebraic arguments and the key mechanism behind acceleration remains elusive. Various alternative to the accelerated algorithm were developed with different proofs [Nesterov, 2013a;[START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Auslender | Interior gradient and proximal methods for convex and conic optimization[END_REF][START_REF] Chambolle | Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage[END_REF]] sometimes simpler [START_REF] Tseng | On accelerated proximal gradient methods for convex-concave optimization[END_REF] but still difficult to interpret. Recent stream of papers use differential equations to model the acceleration behavior and offer another interpretation of Nesterov's algorithm [START_REF] Su | A differential equation for modeling nesterov's accelerated gradient method: Theory and insights[END_REF][START_REF] Krichene | Accelerated mirror descent in continuous and discrete time[END_REF][START_REF] Wibisono | A variational perspective on accelerated methods in optimization[END_REF][START_REF] Wilson | A lyapunov analysis of momentum methods in optimization[END_REF]. However, the differential equation is often quite complex, being reverse-engineered from Nesterov's method itself, thus losing the intuition. Moreover, integration methods for these differential equations are often ignored or are not derived from standard numerical integration schemes, since convergence proofs do not require the continuous-time interpretation.

In [START_REF] Scieur | Integration methods and accelerated optimization algorithms[END_REF], we analyzed how optimization methods, in particular Nesterov's algorithm, can be interpreted as discretization methods of the simple gradient flow equation that reads

ẋ(𝑡) = -∇𝑓 (𝑥(𝑡)), 𝑥(0) = 𝑥 0 .
(1.8)

For example gradient descent can be seen as an Euler's explicit scheme of integration for this differential equation. Proximal point algorithm [START_REF] Rockafellar | Monotone operators and the proximal point algorithm[END_REF] translates as an Euler's implicit scheme. Finally extra-gradient algorithm, studied for example by [START_REF] Nemirovski | Prox-method with rate of convergence o (1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convexconcave saddle point problems[END_REF], that solves saddle point problems, is a predictor-corrector method [Press, 1992, §16.7] on the associated gradient flow equation.

In [START_REF] Scieur | Integration methods and accelerated optimization algorithms[END_REF], we study specifically multi-step discretization methods for gradient flow equation derived from smooth and strongly convex functions. These schemes use several past points to build next iterate of the approximate solution of (1.8). Numerical analysis literature [START_REF] Gautschi | Numerical analysis[END_REF] provides conditions on the parameters to ensure consistency, i.e. convergence of the discrete approximation to the continuous solution of the differential equation on a finite time interval for infinitesimal step-sizes, which is essential for continuous time interpretation. Stability for infinite time horizon can be derived for linear gradient flows (quadratic optimization) by some other conditions. Nesterov's algorithm and Polyak's heavy ball method can then be identified as consistent and stable multi-step methods of order 2 that allow bigger step-size of integration compared to simple Euler's scheme, which explains the acceleration phenomenon. Similar considerations applied for the non-strongly convex case by identifying accelerated method as a consistent integration scheme and identifying a bigger step-size. A full analysis of multi-step methods of order 2 is carried out for linear gradient flows (quadratic optimization). Future work is to certify stability of multi-step methods for smooth and strongly convex functions and explain why Heavy ball's method fails to solve some smooth strongly convex functions [START_REF] Lessard | Analysis and design of optimization algorithms via integral quadratic constraints[END_REF].

Introduction

We study convex optimization problems of the form minimize 𝑓 (𝑥) (P)

where 𝑓 is a convex function defined on R 𝑛 . The complexity of these problems using first order methods is generically controlled by smoothness assumptions on 𝑓 such as Lipschitz continuity of its gradient. Additional assumptions such as strong convexity or uniform convexity provide respectively linear [Nesterov, 2013b] and faster polynomial [START_REF] Juditski | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF] rates of convergence. However, these assumptions are often too restrictive to be applied. Here, we make a much weaker and generic assumption that describes the sharpness of the function around its minimizers by constants 𝜇 ≥ 0 and 𝑟 ≥ 1 such that

𝜇 𝑟 𝑑(𝑥, 𝑋 * ) 𝑟 ≤ 𝑓 (𝑥) -𝑓 * , for every 𝑥 ∈ 𝐾, (Sharp)
where 𝑓 * is the minimum of 𝑓 , 𝐾 ⊂ R 𝑛 is a compact set, 𝑑(𝑥, 𝑋 * ) = min 𝑦∈𝑋 * ‖𝑥 -𝑦‖ is the distance from 𝑥 to the set 𝑋 * ⊂ 𝐾 of minimizers of 𝑓 1 for the Euclidean norm ‖ • ‖. This defines a lower bound on the function around its minimizers: for 𝑟 = 1, 𝑓 shows a kink around its minimizers and the larger is 𝑟 the flatter is the function around its minimizers. We tackle this property by restart schemes of classical convex optimization algorithms.

Sharpness assumption (Sharp) is better known as a Hölderian error bound on the distance to the set of minimizers. Hoffman [START_REF] Hoffman | On approximate solutions of systems of linear inequalities[END_REF] first introduced error bounds to study system of linear inequalities. Natural extensions were then developed for convex optimization [START_REF] Robinson | An application of error bounds for convex programming in a linear space[END_REF][START_REF] Mangasarian | A condition number for differentiable convex inequalities[END_REF][START_REF] Auslender | Global regularity theorems[END_REF], notably through the concept of sharp minima [START_REF] Polyak | Sharp minima institute of control sciences lecture notes, moscow, ussr[END_REF][START_REF] Burke | Weak sharp minima in mathematical programming[END_REF][START_REF] Burke | Weak sharp minima revisited part i: basic theory[END_REF]. But the most striking discovery was made by Łojasiewicz [Łojasiewicz, 1963, 1993] who proved inequality (Sharp) for real analytic and subanalytic functions. It has then been extended to non-smooth subanalytic convex functions by [START_REF] Bolte | The łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems[END_REF]. Overall, since (Sharp) essentially measures the sharpness of minimizers, it holds somewhat generically. On the other hand, this inequality is purely descriptive as we have no hope of ever observing either 𝑟 or 𝜇, and deriving adaptive schemes is crucial to ensure practical relevance.

Łojasiewicz inequalities either in the form of (Sharp) or as gradient dominated properties [START_REF] Polyak | Sharp minima institute of control sciences lecture notes, moscow, ussr[END_REF] led to new simple convergence results [START_REF] Karimi | Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition[END_REF], in particular for alternating and splitting methods [START_REF] Attouch | Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the kurdyka-łojasiewicz inequality[END_REF][START_REF] Frankel | Splitting methods with variable metric for kurdyka-łojasiewicz functions and general convergence rates[END_REF], even in the non-convex case [START_REF] Bolte | Proximal alternating linearized minimization for nonconvex and nonsmooth problems[END_REF]]. Here we focus on Hölderian error bounds as they offer simple explanation of accelerated rates of restart schemes.

Restart schemes were already studied for strongly or uniformly convex functions [START_REF] Nemirovskii | Optimal methods of smooth convex minimization[END_REF]Nesterov, 2013a;[START_REF] Juditski | Primal-dual subgradient methods for minimizing uniformly convex functions[END_REF][START_REF] Lin | An adaptive accelerated proximal gradient method and its homotopy continuation for sparse optimization[END_REF]. In particular, [START_REF] Nemirovskii | Optimal methods of smooth convex minimization[END_REF] link a "strict minimum" condition akin to (Sharp) with faster convergence rates using restart schemes which form the basis of our results, but do not study the cost of adaptation and do not tackle the non-smooth case. In a similar spirit, weaker versions of this strict minimum condition were used more recently to study the performance of restart schemes in [START_REF] Renegar | Efficient first-order methods for linear programming and semidefinite programming[END_REF][START_REF] Freund | New computational guarantees for solving convex optimization problems with first order methods, via a function growth condition measure[END_REF][START_REF] Roulet | Renegar's condition number, shaprness and compressed sensing performance[END_REF]. The fundamental question of a restart scheme is naturally to know when must an algorithm be stopped and relaunched. Several heuristics [O'Donoghue and Candes, 2015;[START_REF] Su | A differential equation for modeling nesterov's accelerated gradient method: Theory and insights[END_REF][START_REF] Giselsson | Monotonicity and restart in fast gradient methods[END_REF] studied adaptive restart schemes to speed up convergence of optimal methods. The robustness of restart schemes was then theoretically studied by [START_REF] Fercoq | Restarting accelerated gradient methods with a rough strong convexity estimate[END_REF] for quadratic error bounds, i.e. (Sharp) with 𝑟 = 2, that LASSO problem satisfies for example. [START_REF] Fercoq | Adaptive restart of accelerated gradient methods under local quadratic growth condition[END_REF] extended recently their work to produce adaptive restarts with theoretical guarantees of optimal performance, still for quadratic error bounds. Previous references focus on smooth problems, but error bounds appear also for non-smooth ones, [START_REF] Gilpin | First-order algorithm with 𝒪(log 1/𝜖) convergence for 𝜖-equilibrium in two-person zero-sum games[END_REF] prove for example linear converge of restart schemes in bilinear matrix games where the minimum is sharp, i.e. (Sharp) with 𝑟 = 1.

Our contribution here is to derive optimal scheduled restart schemes for general convex optimization problems for smooth, non-smooth or Hölder smooth functions satisfying the sharpness assumption. We then show that for smooth functions these schemes can be made adaptive with nearly optimal complexity (up to a squared log term) for a wide array of sharpness assumptions. We also analyze restart criterion based on a sufficient decrease of the gap to the minimum value of the problem, when this latter is known in advance. In that case, restart schemes are shown to be optimal without requiring any additional information on the function.

The paper is organized as follows. In Section 2.1, we present the assumptions we make on the problem, i.e. smoothness and sharpness of the function, and discuss their link. In Section 2.2, scheduled restart strategies are presented for smooth unconstrained convex minimization problems satisfying the Łojasiewicz inequality, together with adaptive variants. In Section 2.3, these results are generalized to functions with Hölder continuous gradients. Adaptive restart schemes when the optimal value is known are introduced in Section 2.4. In Section 2.5, our results are extended to composite problems using Bregman divergences, and in particular prox-friendly constrained problems. Finally, numerical experiments are provided in Section 2.6.

Notations

For a real 𝑎, ⌈𝑎⌉ and ⌊𝑎⌋ denote respectively the smallest integer larger than or equal to 𝑎 and the largest integer smaller than or equal to 𝑎.

Problem assumptions

In the following, we present the geometry of the problem with respect to the Euclidean norm, such that 𝑑(𝑥, 𝑋 * ) = min 𝑦∈𝑋 * ‖𝑥 -𝑦‖ 2 is the Euclidean distance from a point 𝑥 ∈ R 𝑛 to the set of minimizers. In Section 2.5, we detail how our approach generalizes to other geometries handled by Bregman divergences.

Smoothness

Convex optimization problems (P) are generally divided in two classes : smooth problems, for which 𝑓 has Lipschitz continuous gradients, and non-smooth problems for which 𝑓 is not differentiable. [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF] proposed to unify point of views by assuming generally that there exist constant 1 ≤ 𝑠 ≤ 2 and 𝐿 > 0 such that

‖∇𝑓 (𝑥) -∇𝑓 (𝑦)‖ 2 ≤ 𝐿‖𝑥 -𝑦‖ 𝑠-1 2 , for all 𝑥, 𝑦 ∈ R 𝑛 (Smooth)
where ∇𝑓 (𝑥) is any sub-gradient of 𝑓 at 𝑥 if 𝑠 = 1 (otherwise this implies differentiability of 𝑓 ). For 𝑠 = 2, we retrieve the classical definition of smoothness [Nesterov, 2013b]. For 𝑠 = 1 we get a classical assumption made in non-smooth convex optimization, i.e. that sub-gradients of the function are bounded. For 1 < 𝑠 < 2, this assumes gradient of 𝑓 to be Hölder Lipschitz. In a first step, we will analyze restart schemes for smooth convex optimization problems, then generalize to general smoothness assumption (Smooth) using appropriate accelerated algorithms developed by [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF].

Error bounds

In general, an error bound is an inequality of the form

𝑑(𝑥, 𝑋 * ) ≤ 𝜔(𝑓 (𝑥) -𝑓 * ),
where 𝜔 is an increasing function at 0, called the residual function, and 𝑥 may evolve either in the whole space or in a bounded set, see [START_REF] Bolte | From error bounds to the complexity of first-order descent methods for convex functions[END_REF] for more details. We focus on Hölderian Error Bounds (Sharp) as they are the most common in practice, they are notably satisfied by a analytic and subanalytic functions but the proof (see e.g. [START_REF] Bierstone | Semianalytic and subanalytic sets[END_REF]) is shown using topological arguments that are far from constructive, hence outside of some particular cases (e.g. strong convexity), we cannot assume that the constants in (Sharp) are known, even approximately.

Error bounds can generically be linked to Łojasiewicz inequality that upper bounds magnitude of the gradient by values of the function [START_REF] Bolte | From error bounds to the complexity of first-order descent methods for convex functions[END_REF]. Such property paved the way to many recent results in optimization [START_REF] Attouch | Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the kurdyka-łojasiewicz inequality[END_REF][START_REF] Frankel | Splitting methods with variable metric for kurdyka-łojasiewicz functions and general convergence rates[END_REF][START_REF] Bolte | Proximal alternating linearized minimization for nonconvex and nonsmooth problems[END_REF]. Here we will see that (Sharp) is sufficient to acceleration of convex optimization algorithms by their restart. Note finally that in most cases, error bounds are local properties hence the convergence results that follow will generally be local.

Sharpness and smoothness

Let 𝑓 be a convex function on R 𝑛 satisfying (Smooth) with parameters (𝑠, 𝐿). This property ensures that, 𝑓 (𝑥) ≤ 𝑓

Scheduled restarts for smooth convex problems

In this section 𝑓 is assumed to be smooth, i.e. satisfies (Smooth) with 𝑠 = 2 and 𝐿 > 0. Without further assumptions on 𝑓 , an optimal algorithm to solve the smooth convex optimization problem (P) is Nesterov's accelerated gradient method [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate 𝑂(1/𝑘 2 )[END_REF]. Given an initial point 𝑥 0 , this algorithm outputs, after 𝑡 iterations, a point 𝑥 = 𝒜(𝑥 0 , 𝑡) such that

𝑓 (𝑥) -𝑓 * ≤ 𝑐𝐿 𝑡 2 𝑑(𝑥 0 , 𝑋 * ) 2 , (2.3)
where 𝑐 > 0 denotes a universal constant (whose value will be allowed to vary in what follows, with 𝑐 = 4 here). We assume without loss of generality that 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 ). [Bierstone and Milman, 1988, Theorem 6.4]. The scheduled restart schemes presented assume (Sharp) on the whole sublevel set defined by the initial point and are not adaptive to the best local constant 𝜇. On the other hand, restarts on criterion introduced in Section 2.4, assuming that 𝑓 * is known, adapt to the value of 𝜇. We now describe a restart scheme exploiting this extra regularity assumption to improve the computational complexity of solving problem (P) using accelerated methods.

Scheduled restarts

Here we schedule the number of iterations 𝑡 𝑘 made by Nesterov's algorithm between restarts, with 𝑡 𝑘 the number of (inner) iterations at the 𝑘 th algorithm run (outer iteration). Our scheme is described below.

Algorithm 1 Scheduled restarts for smooth convex minimization (RESTART) Inputs : 𝑥 0 ∈ R 𝑛 and a sequence 𝑡 𝑘 for 𝑘 = 1, . . . , 𝑅. for 𝑘 = 1, . . . , 𝑅 do

𝑥 𝑘 := 𝒜(𝑥 𝑘-1 , 𝑡 𝑘 ) (RESTART)
end for Output : x := 𝑥 𝑅 The analysis of this scheme and the following ones relies on two steps : first choose schedules that ensure linear convergence in the iterates 𝑥 𝑘 at a given rate, then adjust this linear rate to minimize the complexity in terms of the total number of iterations.

We begin by a technical lemma which assumes linear convergence holds, and connects the growth of 𝑡 𝑘 , the precision reached and the total number of inner iterations 𝑁 .

Lemma 2.2.1. Let 𝑥 𝑘 be a sequence whose 𝑘 th iterate is generated from the previous one by an algorithm that runs 𝑡 𝑘 iterations and write 𝑁 = ∑ 𝑅 𝑘=1 𝑡 𝑘 the total number of iterations to output a point 𝑥 𝑅 . Suppose setting 𝑡 𝑘 = 𝐶𝑒 𝛼𝑘 , 𝑘 = 1, . . . , 𝑅 for some 𝐶 > 0 and 𝛼 ≥ 0 ensures that outer iterations satisfy

𝑓 (𝑥 𝑘 ) -𝑓 * ≤ 𝜈𝑒 -𝛾𝑘 , (2.4)
for all 𝑘 ≥ 0 with 𝜈 ≥ 0 and 𝛾 ≥ 0. Then precision at the output is given by, 𝑓 (𝑥 𝑅 ) -𝑓 * ≤ 𝜈 exp(-𝛾𝑁/𝐶), when 𝛼 = 0, and

𝑓 (𝑥 𝑅 ) -𝑓 * ≤ 𝜈 (𝛼𝑒 -𝛼 𝐶 -1 𝑁 + 1) 𝛾 𝛼
, when 𝛼 > 0.

Proof. When 𝛼 = 0, 𝑁 = 𝑅𝐶, and inserting this in (2.4) at the last point 𝑥 𝑅 yields the desired result. On the other hand, if

𝛼 > 0, then 𝑁 = ∑ 𝑅 𝑘=1 𝑡 𝑘 = 𝐶𝑒 𝛼 𝑒 𝛼𝑅 -1 𝑒 𝛼 -1 , which gives 𝑅 = log ( 𝑒 𝛼 -1 𝑒 𝛼 𝐶 𝑁 + 1
) /𝛼. Inserting this in (2.4) at the last point, this leads

𝑓 (𝑥 𝑅 ) -𝑓 * ≤ 𝜈 exp ( - 𝛾 𝛼 log ( 𝑒 𝛼 -1 𝑒 𝛼 𝐶 𝑁 + 1 )) ≤ 𝜈 (𝛼𝑒 -𝛼 𝐶 -1 𝑁 + 1) 𝛾 𝛼
, using that 𝑒 𝑥 -1 ≥ 𝑥. This yields the second part of the result.

The last approximation in the case 𝛼 > 0 simplifies the analysis that follows without significantly affecting the bounds. We also show in Appendix 2.A that using t𝑘 = ⌈𝑡 𝑘 ⌉ does not significantly affect the bounds above. Remark that convergence bounds are generally linear or polynomial such that one can extract a subsequence that converges linearly. Therefore our approach does not restrict the analysis of our scheme. It simplifies it and can be used for other algorithms like the gradient descent as detailed in Section 2.2.3.

We now analyze restart schedules 𝑡 𝑘 that ensure linear convergence. Our choice of 𝑡 𝑘 will heavily depend on the ratio between 𝑟 and 𝑠 (with 𝑠 = 2 for smooth functions here), incorporated in the parameter 𝜏 = 1 -𝑠/𝑟 defined in (2.2). The following Proposition shows that if 𝜏 = 0, a constant schedule is sufficient to ensure linear convergence. When 𝜏 > 0, this requires a geometrically increasing number of iterations for each cycle.

Proposition 2.2.2. Let 𝑓 be a smooth convex function satisfying (Smooth) with parameters (2, 𝐿) and (Sharp) with parameters (𝑟, 𝜇) on a set 𝐾. Assume that we are given 𝑥 0 ∈ R 𝑛 such that {𝑥| 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )} ⊂ 𝐾. Run (RESTART) from 𝑥 0 with iteration schedule 𝑡 𝑘 = 𝐶 * 𝜅,𝜏 𝑒 𝜏 𝑘 , for 𝑘 = 1, . . . , 𝑅, where

𝐶 * 𝜅,𝜏 𝑒 1-𝜏 (𝑐𝜅) 1 2 (𝑓 (𝑥 0 ) -𝑓 * ) -𝜏 2 , (2.5)
with 𝜅 and 𝜏 defined in (2.2) and 𝑐 = 4𝑒 2/𝑒 here. The precision reached at the last point x is given by,

𝑓 (x) -𝑓 * ≤ exp ( -2𝑒 -1 (𝑐𝜅) -1 2 𝑁 ) (𝑓 (𝑥 0 ) -𝑓 * ) = 𝑂 ( exp(-𝜅 -1 2 𝑁 ) ) , when 𝜏 = 0, (2.6) while, 𝑓 (x) -𝑓 * ≤ 𝑓 (𝑥 0 ) -𝑓 * ( 𝜏 𝑒 -1 (𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 2 (𝑐𝜅) -1 2 𝑁 + 1 ) 2 𝜏 = 𝑂 ( 𝑁 -2 𝜏 ) , when 𝜏 > 0, (2.7)
where 𝑁 = ∑ 𝑅 𝑘=1 𝑡 𝑘 is the total number of iterations.

Proof. Our strategy is to choose 𝑡 𝑘 such that the objective is linearly decreasing, i.e.

𝑓 (𝑥 𝑘 ) -𝑓 * ≤ 𝑒 -𝛾𝑘 (𝑓 (𝑥 0 ) -𝑓 * ), (2.8)
for some 𝛾 ≥ 0 depending on the choice of 𝑡 𝑘 . This directly holds for 𝑘 = 0 and any 𝛾 ≥ 0. Combining (Sharp) with the complexity bound in (2.3) gives

𝑓 (𝑥 𝑘 ) -𝑓 * ≤ 𝑐𝜅 𝑡 2 𝑘 (𝑓 (𝑥 𝑘-1 ) -𝑓 * ) 2 𝑟 ,
where 𝑐 = 4𝑒 2/𝑒 using that 𝑟 2/𝑟 ≤ 𝑒 2/𝑒 . Assuming recursively that (2.8) is satisfied at iteration 𝑘 -1 for a given 𝛾,

𝑓 (𝑥 𝑘 ) -𝑓 * ≤ 𝑐𝜅𝑒 -𝛾 2 𝑟 (𝑘-1) 𝑡 2 𝑘 (𝑓 (𝑥 0 ) -𝑓 * ) 2 𝑟 ,
and to ensure (2.8) at iteration 𝑘, this imposes

𝑐𝜅𝑒 -𝛾 2 𝑟 (𝑘-1) 𝑡 2 𝑘 (𝑓 (𝑥 0 ) -𝑓 * ) 2 𝑟 ≤ 𝑒 -𝛾𝑘 (𝑓 (𝑥 0 ) -𝑓 * ).
Rearranging terms in this last inequality, using 𝜏 defined in (2.2),

𝑡 𝑘 ≥ 𝑒 𝛾(1-𝜏 ) 2 (𝑐𝜅) 1 2 (𝑓 (𝑥 0 ) -𝑓 * ) -𝜏 2 𝑒 𝜏 𝛾 2 𝑘 .
(2.9)

For a given 𝛾 ≥ 0, we can set 𝑡 𝑘 = 𝐶𝑒 𝛼𝑘 where

𝐶 = 𝑒 𝛾(1-𝜏 ) 2 (𝑐𝜅) 1 2 (𝑓 (𝑥 0 ) -𝑓 * ) -𝜏 2 and 𝛼 = 𝜏 𝛾/2, (2.10)
and Lemma 2.2.1 then yields,

𝑓 (x) -𝑓 * ≤ exp ( -𝛾𝑒 -𝛾 2 (𝑐𝜅) -1 2 𝑁 ) (𝑓 (𝑥 0 ) -𝑓 * ),
when 𝜏 = 0, while

𝑓 (x) -𝑓 * ≤ (𝑓 (𝑥 0 ) -𝑓 * ) ( 𝜏 2 𝛾𝑒 -𝛾 2 (𝑐𝜅) -1 2 (𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 2 𝑁 + 1 ) 2 𝜏 ,
when 𝜏 > 0. These bounds are minimal for 𝛾 = 2, which yields the desired result.

When 𝜏 = 0, bound (2.6) matches the classical complexity bound for smooth strongly convex functions [Nesterov, 2013b]. When 𝜏 > 0 on the other hand, bound (2.7) highlights a much faster convergence rate than accelerated gradient methods. The sharper the function (i.e. the smaller 𝑟), the faster the convergence. This matches the lower bounds for optimizing smooth and sharp functions functions [Nemirovskii and Nesterov, 1985, Page 6] up to constant factors. Also, setting 𝑡 𝑘 = 𝐶 * 𝜅,𝜏 𝑒 𝜏 𝑘 yields continuous bounds on precision, i.e. when 𝜏 → 0, bound (2.7) converges to bound (2.6), which also shows that for 𝜏 near zero, constant restart schemes are almost optimal.

Adaptive scheduled restart

The previous restart schedules depend on the parameters (𝑟, 𝜇) in (Sharp). In general of course, these values are neither observed nor known a priori. Making our restart scheme adaptive is thus crucial to its practical performance. Fortunately, we show below that a simple logarithmic grid search strategy on these parameters is enough to guarantee nearly optimal performance.

In that purpose we need first the following Corollary of Proposition 2.2.2.

Corollary 2.2.3. Let 𝑓 be a smooth convex function satisfying (Smooth) with parameters (2, 𝐿) and (Sharp) with parameters (𝑟, 𝜇) on a set 𝐾. Assume that we are given 𝑥 0 ∈ R 𝑛 such that {𝑥 : 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )} ⊂ 𝐾. Run (RESTART) from 𝑥 0 with general schedules of the form

{ 𝑡 𝑘 = 𝐶 if 𝜏 = 0, 𝑡 𝑘 = 𝐶𝑒 𝛼𝑘 if 𝜏 > 0,
we have the following complexity bounds, if 𝜏 = 0 and 𝐶 ≥ 𝐶 * 𝜅,0 ,

𝑓 (x) -𝑓 * ≤ ( 𝑐𝜅 𝐶 2 ) 𝑁 𝐶 (𝑓 (𝑥 0 ) -𝑓 * ), (2.11) while, if 𝜏 > 0 and 𝐶 ≥ 𝐶(𝛼), 𝑓 (x) -𝑓 * ≤ 𝑓 (𝑥 0 ) -𝑓 * (𝛼𝑒 -𝛼 𝐶 -1 𝑁 + 1) 2 𝜏 , (2.12)
where (2.13) and 𝑁 = ∑ 𝑅 𝑘=1 𝑡 𝑘 is the total number of iterations.

𝐶(𝛼) 𝑒 𝛼(1-𝜏 ) 𝜏 (𝑐𝜅) 1 2 (𝑓 (𝑥 0 ) -𝑓 * ) -𝜏 2 ,
Proof. Given general schedules of the form

{ 𝑡 𝑘 = 𝐶 if 𝜏 = 0, 𝑡 𝑘 = 𝐶𝑒 𝛼𝑘 if 𝜏 > 0,
the best value of 𝛾 satisfying condition (2.9) for any 𝑘 ≥ 0 in Proposition 2.2.2 are given by

{ 𝛾 = log ( 𝐶 2 𝑐𝜅 ) if 𝜏 = 0 and 𝐶 ≥ 𝐶 * 𝜅,0 , 𝛾 = 2𝛼 𝜏 if 𝜏 > 0 and 𝐶 ≥ 𝐶(𝛼).
As in Proposition 2.2.2, plugging these values into the bounds of Lemma 2.2.1 yields the desired result.

This Corollary shows that scheduled restarts are theoretically efficient only if the algorithm itself makes a sufficient number of iterations. With this Corollary, an analysis of a grid search of the schedules can be made.

We run several schemes with a fixed number of inner iterations 𝑁 to perform a log-scale grid search on 𝜏 and 𝜅. These schemes are defined as follows.

{

𝒮 𝑖,0 : (RESTART) scheme with 𝑡 𝑘 = 𝐶 𝑖 , 𝒮 𝑖,𝑗 : (RESTART) scheme with 𝑡 𝑘 = 𝐶 𝑖 𝑒 𝜏 𝑗 𝑘 , (2.14)

where 𝐶 𝑖 = 2 𝑖 and 𝜏 𝑗 = 2 -𝑗 . These schemes are stopped when the total number of inner algorithm iterations has exceed 𝑁 , i.e. at the smallest 𝑅 such that ∑ 𝑅 𝑘=1 𝑡 𝑘 ≥ 𝑁 . The size of the grid search in 𝐶 𝑖 is naturally bounded as the algorithm cannot be restarted after more than 𝑁 total inner iterations, so 𝑖 ∈ [1, . . . , ⌊log 2 𝑁 ⌋]. We will also show that when 𝜏 is smaller than 1/𝑁 , a constant schedule performs as well as the optimal geometrically increasing schedule, which crucially means that we can also choose 𝑗 ∈ [1, . . . , ⌈log 2 𝑁 ⌉] and limits the cost of grid search.

(x) -𝑓 * ≤ exp ( -𝑒 -1 (𝑐𝜅) -1 2 𝑁 ) (𝑓 (𝑥 0 ) -𝑓 * ).
If 𝜏 > 0, there exist 𝑖 ∈ [1, . . . , ⌊log 2 𝑁 ⌋] and 𝑗 ∈ [1, . . . , ⌈log 2 𝑁 ⌉] such that scheme 𝒮 𝑖,𝑗 achieves a precision given by

𝑓 (x) -𝑓 * ≤ 𝑓 (𝑥 0 ) -𝑓 * ( 𝜏 𝑒 -1 (𝑐𝜅) -1 2 (𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 2 (𝑁 -1)/4 + 1 ) 2 𝜏 .
Overall, running the logarithmic grid search has a complexity (log 2 𝑁 ) 2 times higher than running 𝑁 iterations using the optimal (oracle) scheme.

Proof. Denote 𝑁 ′ = ∑ 𝑅 𝑘=1 𝑡 𝑘 ≥ 𝑁 the number of iterations of a scheme 𝒮 𝑖,𝑗 . We necessarily have 𝑁 ′ ≤ 2𝑁 for our choice of 𝐶 𝑖 and 𝜏 𝑗 . Hence the cost of running all methods is of the order (log 2 𝑁 ) 2 .

If 𝜏 = 0 and 𝑁 ≥ 2𝐶 * 𝜅,0 , then 𝑖 = ⌈log 2 𝐶 * 𝜅,0 ⌉ ≤ ⌊log 2 𝑁 ⌋. Therefore 𝒮 𝑖,0 has been run and bound (2.11) to shows then that the last iterate x satisfies

𝑓 (x) -𝑓 * ≤ ( 𝑐𝜅 𝐶 2 𝑖 ) 𝑁 𝐶 𝑖 (𝑓 (𝑥 0 ) -𝑓 * ). Using that 𝐶 * 𝜅,0 ≤ 𝐶 𝑖 ≤ 2𝐶 * 𝜅,0 , 𝑓 (x) -𝑓 * ≤ ( 𝑐𝜅 (𝐶 * 𝜅,0 ) 2 ) 𝑁 2𝐶 * 𝜅,0 (𝑓 (𝑥 0 ) -𝑓 * ) ≤ exp ( -𝑒 -1 (𝑐𝜅) -1 2 𝑁 ) (𝑓 (𝑥 0 ) -𝑓 * ).
If 𝜏 ≥ 1 𝑁 and 𝑁 ≥ 2𝐶 * 𝜅,𝜏 , then 𝑗 = ⌈log 2 𝜏 ⌉ ≤ ⌈log 2 𝑁 ⌉ and 𝑖 = ⌈log 2 𝐶 * 𝜅,𝜏 ⌉ ≤ ⌊log 2 𝑁 ⌋. Therefore scheme 𝒮 𝑖,𝑗 has been run. As 𝐶 𝑖 ≥ 𝐶 * 𝜅,𝜏 ≥ 𝐶(𝜏 𝑗 ), where 𝐶(𝜏 𝑗 ) is defined in (2.13), bound (2.12) shows then that the last iterate x of scheme 𝒮 𝑖,𝑗 satisfies

𝑓 (x) -𝑓 * ≤ 𝑓 (𝑥 0 ) -𝑓 * ( 𝜏 𝑗 𝑒 -𝜏 𝑗 𝐶 -1 𝑖 𝑁 + 1 ) 2 𝜏 .
Finally, by definition of 𝑖 and 𝑗, 2𝜏 𝑗 ≥ 𝜏 and 𝐶 𝑖 ≤ 2𝐶 * 𝜅,𝜏 , so

𝑓 (x) -𝑓 * ≤ 𝑓 (𝑥 0 ) -𝑓 * ( 𝜏 𝑒 -𝜏 𝑗 (𝐶 * 𝜅,𝜏 ) -1 𝑁/4 + 1 ) 2 𝜏 = 𝑓 (𝑥 0 ) -𝑓 * ( 𝜏 𝑒 -1 (𝑐𝜅) -1 2 (𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 2 𝑁/4 + 1 ) 2 𝜏 ,
where we concluded by expanding 𝐶 * 𝜅,𝜏 = 𝑒 1-𝜏 (𝑐𝜅)

1 2 (𝑓 (𝑥 0 ) -𝑓 * ) -𝜏 2 and using that 𝜏 ≥ 𝜏 𝑗 .

If 1 𝑁 > 𝜏 > 0 and 𝑁 > 2𝐶 * 𝜅,𝜏 , then 𝑖 = ⌈log 2 𝐶 * 𝜅,𝜏 ⌉ ≤ ⌊log 2 𝑁 ⌋, so scheme 𝒮 𝑖,0 has been run. Its iterates 𝑥 𝑘 satisfy, with 1 -𝜏 = 2/𝑟,

𝑓 (𝑥 𝑘 ) -𝑓 * ≤ 𝑐𝜅 𝐶 2 𝑖 (𝑓 (𝑥 𝑘-1 ) -𝑓 * ) 2 𝑟 ≤ ( 𝑐𝜅 𝐶 2 𝑖 ) (1-(1-𝜏) 𝑘 )/𝜏 (𝑓 (𝑥 0 ) -𝑓 * ) (1-𝜏 ) 𝑘 ≤ ( 𝑐𝜅(𝑓 (𝑥 0 ) -𝑓 * ) -𝜏 𝐶 2 𝑖 ) (1-(1-𝜏) 𝑘 )/𝜏 (𝑓 (𝑥 0 ) -𝑓 * ). Now 𝐶 𝑖 ≥ 𝐶 * 𝜅,𝜏 = 𝑒 1-𝜏 (𝑐𝜅) 1 2 (𝑓 (𝑥 0 ) -𝑓 * ) -𝜏 2 and 𝐶 𝑖 𝑅 ≥ 𝑁 , therefore last iterate x satisfies 𝑓 (x) -𝑓 * ≤ exp ( -2(1 -𝜏 ) 1 -(1 -𝜏 ) 𝑁/𝐶 𝑖 𝜏 ) (𝑓 (𝑥 0 ) -𝑓 * ).
As 𝑁 ≥ 𝐶 𝑖 , since

ℎ(𝜏 ) = (1 -𝜏 ) ( 1 -(1 -𝜏 ) 𝑁 𝐶 𝑖 ) 1 -(1 -𝜏 )
is decreasing with 𝜏 and 1 𝑁 > 𝜏 > 0, we have

𝑓 (x) -𝑓 * ≤ exp ( -2(𝑁 -1) ( 1 - ( 1 - 1 𝑁 ) 𝑁/𝐶 𝑖 )) (𝑓 (𝑥 0 ) -𝑓 * ) ≤ exp ( -2(𝑁 -1) ( 1 -exp ( - 1 𝐶 𝑖 ))) (𝑓 (𝑥 0 ) -𝑓 * ) ≤ exp ( -2 𝑁 -1 𝐶 𝑖 ( 1 - 1 2𝐶 𝑖 )) (𝑓 (𝑥 0 ) -𝑓 * ).
having used the facts that (1 + 𝑎𝑥)

𝑏 𝑥 ≤ exp(𝑎𝑏) if 𝑎𝑥 ≥ -1, 𝑏 𝑥 ≥ 0 and 1 -𝑥 + 𝑥 2 2 ≥ exp(-𝑥) when 𝑥 ≥ 0. By assumption 𝐶 * 𝜅,𝜏 ≥ 1, so 𝐶 𝑖 ≥ 1 and finally 𝑓 (x) -𝑓 * ≤ exp ( - 𝑁 -1 𝐶 𝑖 ) (𝑓 (𝑥 0 ) -𝑓 * ) ≤ exp ( - 𝑁 -1 2𝐶 * 𝜅,𝜏 ) (𝑓 (𝑥 0 ) -𝑓 * ) ≤ 𝑓 (𝑥 0 ) -𝑓 * ( 𝜏 (𝐶 * 𝜅,𝜏 ) -1 (𝑁 -1)/4 + 1 ) 2 𝜏 ≤ 𝑓 (𝑥 0 ) -𝑓 * ( 𝜏 (𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 2 𝑒 -1 (𝑐𝜅) -1 2 (𝑁 -1)/4 + 1 ) 2 𝜏 .
using the fact that 𝑒 𝜏 ≥ 1.

As shown in Corollary 2.2.3, theoretical efficiency of scheduled restarts is ensured only for a sufficient number of iterations of the algorithm itself. Therefore 𝑁 needs to be large enough to ensure the efficiency of the adaptive method. If 𝜏 = 0, then naturally 𝐶 * 𝜅,0 ≥ 1, therefore if 1 𝑁 > 𝜏 > 0 and 𝑁 is large, assuming 𝐶 * 𝜅,𝜏 ≈ 𝐶 * 𝜅,0 , it results 𝐶 * 𝜅,𝜏 ≥ 1. This adaptive bound is similar to the one of Nesterov [2013a] to optimize smooth strongly convex functions in the sense that we lose approximately a log factor of the condition number of the function. However our assumptions are weaker and we are able to tackle all regimes of the sharpness property, i.e. any exponent 𝑟 ∈ [2, +∞], not just the strongly convex case. Finally the step size chosen for the grid search was set to 2. Proof can adapted for a generic step size ℎ, the size of the grid may be reduced but corresponding bounds will suffer an ℎ 2 approximation loss compared to the best schedule.

We end this section by analyzing the behavior of gradient descent in light of the sharpness assumption.

Comparison to gradient descent

Given only the smoothness hypothesis, the gradient descent algorithm, recalled in Appendix A.3, starts from a point 𝑥 0 and outputs iterates 𝑥 𝑡 = 𝒢(𝑥 0 , 𝑡) such that

𝑓 (𝑥 𝑡 ) -𝑓 * ≤ 𝐿 𝑡 𝑑(𝑥 0 , 𝑋 * ) 2 ,
While accelerated methods use the last two iterates to compute the next one, simple gradient descent algorithms use only the last iterate, so the algorithm can be seen as (implicitly) restarting at each iteration. Its convergence can therefore be written for 𝑘 ≥ 1,

𝑓 (𝑥 𝑘+𝑡 ) -𝑓 * ≤ 𝐿 𝑡 𝑑(𝑥 𝑘 , 𝑋 * ) 2 .
(2.15)

and we analyze it in light of the restart interpretation using the error bound in the following proposition.

Proposition 2.2.5. Let 𝑓 be a smooth convex function satisfying (Smooth) with parameters (2, 𝐿) and (Sharp) with parameters (𝑟, 𝜇) on a set 𝐾. Assume that we are given 𝑥 0 ∈ R 𝑛 such that {𝑥| 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )} ⊂ 𝐾. Denote 𝑥 𝑡 = 𝒢(𝑥 0 , 𝑡) the iterate sequence generated by the gradient descent algorithm started at 𝑥 0 to solve (P). Define

𝑡 𝑘 = 𝑒 1-𝜏 𝑐𝜅(𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 𝑒 𝜏 𝑘 ,
with 𝜅 and 𝜏 defined in (2.2) and 𝑐 = 𝑒 2/𝑒 here. The precision reached after 𝑁 = ∑ 𝑛 𝑘=1 𝑡 𝑘 iterations is given by,

𝑓 (𝑥 𝑁 ) -𝑓 * ≤ exp ( -𝑒 -1 (𝑐𝜅) -1 𝑁 ) (𝑓 (𝑥 0 ) -𝑓 * ) = 𝑂 ( exp(-𝜅 -1 𝑁 ) ) , when 𝜏 = 0, while, 𝑓 (𝑥 𝑁 ) -𝑓 * ≤ 𝑓 (𝑥 0 ) -𝑓 * (𝜏 𝑒 -1 (𝑐𝜅) -1 (𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 𝑁 + 1) 1 𝜏 = 𝑂 ( 𝑁 -1 𝜏 ) , when 𝜏 > 0.
Proof. For a given 𝛾 ≥ 0, we construct a subsequence 𝑥 𝜑(𝑘) of 𝑥 𝑡 such that 𝑓 (𝑥 𝜑(𝑘) ) -𝑓 * ≤ 𝑒 -𝛾𝑘 (𝑓 (𝑥 0 ) -𝑓 * ).

(2.16)

Define 𝑥 𝜑(0) = 𝑥 0 . Assume that (2.16) is true at iteration 𝑘 -1, then combining complexity bound (2.15) and (Sharp), for any 𝑡 ≥ 1,

𝑓 (𝑥 𝜑(𝑘-1)+𝑡 ) -𝑓 * ≤ 𝑐𝜅 𝑡 (𝑓 (𝑥 𝜑(𝑘-1) ) -𝑓 * ) 2 𝑟 ≤ 𝑐𝜅 𝑡 𝑒 -𝛾 2 𝑟 (𝑘-1) (𝑓 (𝑥 0 ) -𝑓 * ) 2 𝑟 .
where 𝑐 = 𝑒 2/𝑒 , using that 𝑟 2/𝑟 ≤ 𝑒 2/𝑒 . Taking 𝑡 𝑘 = 𝑒 𝛾(1-𝜏 ) 𝑐𝜅(𝑓 (𝑥 0 ) -𝑓 * ) -𝜏 𝑒 𝛾𝜏 𝑘 and 𝜑(𝑘) = 𝜑(𝑘 -1) + 𝑡 𝑘 , (2.16) holds at iteration 𝑘. Using Lemma 2.2.1, we obtain at iteration

𝑁 = 𝜑(𝑛) = ∑ 𝑛 𝑘=1 𝑡 𝑘 , 𝑓 (𝑥 𝑁 ) -𝑓 * ≤ exp ( -𝛾𝑒 -𝛾 (𝑐𝜅) -1 𝑁 ) (𝑓 (𝑥 0 ) -𝑓 * ), if 𝜏 = 0, and
𝑓 (𝑥 𝑁 ) -𝑓 * ≤ 𝑓 (𝑥 0 ) -𝑓 * (𝜏 𝛾𝑒 -𝛾 (𝑐𝜅) -1 (𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 𝑁 + 1) 1 𝜏 , if 𝜏 > 0.
These bounds are minimal for 𝛾 = 1 and the results follow.

We observe that restarting accelerated gradient methods reduces complexity from 𝑂(1/𝜀 𝜏 ) to 𝑂(1/𝜀 𝜏 /2 ) compared to simple gradient descent. More general results on the convergence of (sub)gradient descent algorithms under a Łojasiewicz inequality assumption were developed by [START_REF] Bolte | From error bounds to the complexity of first-order descent methods for convex functions[END_REF]. We extend now this restart scheme to solve general convex optimization problem under an Hölderian error bound assumption.

Universal scheduled restarts for convex functions

In this section we use the framework introduced by [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF] to describe smoothness of a function 𝑓 on a set 𝐽 ⊂ R 𝑛 . We recall that it assumes that there exist 𝑠 ∈ [1, 2] and 𝐿 > 0 such that ‖∇𝑓 (𝑥) -∇𝑓 (𝑦)‖ ≤ 𝐿‖𝑥 -𝑦‖ 𝑠-1 , for every 𝑥, 𝑦 ∈ 𝐽, If the function is non smooth, it satisfies (Smooth) with 𝑠 = 1 and 𝐿 taken as the maximum norm of subgradients on 𝐽. Without further assumptions on 𝑓 , an optimal algorithm to solve the convex optimization problem (P) is the universal fast gradi-ent method [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF]. Given a target accuracy 𝜀, the universal fast gradient method starts at a point 𝑥 0 and outputs after 𝑡 iterations a point 𝑥 𝒰(𝑥 0 , 𝜀, 𝑡), such that

𝑓 (𝑥) -𝑓 * ≤ 𝜀 2 + 𝑐𝐿 2 𝑠 𝑑(𝑥 0 , 𝑋 * ) 2 𝜀 2 𝑠 𝑡 2𝜌 𝑠 𝜀 2 , (2.17)
where 𝑐 is a constant (𝑐 = 8) and

𝜌 3𝑠 2 -1 (2.18)
is the optimal rate of convergence for 𝑠-smooth functions. More details about the universal fast gradient method are given in Appendix A.1. We will again assume that 𝑓 satisfies (Sharp) with parameters (𝑟, 𝜇) on a set 𝐾 ⊇ 𝑋 * , i.e.

𝜇 𝑟 𝑑(𝑥, 𝑋 * ) 𝑟 ≤ 𝑓 (𝑥) -𝑓 * , for every 𝑥 ∈ 𝐾.

As mentioned in Section 2.1, if 𝑟 > 𝑠, smoothness or sharpness are local properties, i.e. either 𝐽 or 𝐾 or both are bounded, our analysis is therefore local. In the following we assume for simplicity, given an initial point 𝑥 0 , that smoothness and sharpness are satisfied simultaneously on the sublevel set {𝑥| 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )}. The key difference with the smooth case described in the previous section is that here we schedule both the target accuracy 𝜀 𝑘 used by the algorithm and the number of iterations 𝑡 𝑘 made at the 𝑘 th run of the algorithm. Our scheme is described in Algorithm Universal RESTART.

Algorithm 2 General scheduled restarts for convex minimization (Universal RESTART) where 𝜌 is defined in (2.18), 𝜅 and 𝜏 are defined in (2.2) and 𝑐 = 8𝑒 2/𝑒 here. The precision reached at the last point x is given by,

Inputs : 𝑥 0 ∈ R 𝑛 , 𝜀 0 ≥ 𝑓 (𝑥 0 ) -𝑓 * ,
𝑓 (x) -𝑓 * ≤ exp ( -𝜌𝑒 -1 (𝑐𝜅) -𝑠 2𝜌 𝑁 ) 𝜀 0 = 𝑂 ( exp(-𝜅 -𝑠 2𝜌 𝑁 ) ) , when 𝜏 = 0, while, 𝑓 (x) -𝑓 * ≤ 𝜀 0 ( 𝜏 𝑒 -1 (𝑐𝜅) -𝑠 2𝜌 𝜀 𝜏 𝜌 0 𝑁 + 1 ) -𝜌 𝜏 = 𝑂 ( 𝜅 𝑠 2𝜏 𝑁 -𝜌 𝜏 ) , when 𝜏 > 0,
where 𝑁 = ∑ 𝑅 𝑘=1 𝑡 𝑘 is total number of iterations.

Proof. Our goal is to ensure that the target accuracy is reached at each restart, i.e.

𝑓 (𝑥 𝑘 ) -𝑓 * ≤ 𝜀 𝑘 .

(2.19)

By assumption, (2.19) holds for 𝑘 = 0. Assume that (2.19) is true at iteration 𝑘 -1, combining (Sharp) with the complexity bound in (2.17), then

𝑓 (𝑥 𝑘 ) -𝑓 * ≤ 𝜀 𝑘 2 + 𝑐𝜅(𝑓 (𝑥 𝑘-1 ) -𝑓 * ) 2 𝑟 𝜀 2 𝑠 𝑘 𝑡 2𝜌 𝑠 𝑘 𝜀 𝑘 2 ≤ 𝜀 𝑘 2 + 𝑐𝜅 𝑡 2𝜌 𝑠 𝑘 𝜀 2 𝑟 𝑘-1 𝜀 2 𝑠 𝑘 𝜀 𝑘 2 ,
where 𝑐 = 8𝑒 2/𝑒 using that 𝑟 2/𝑟 ≤ 𝑒 2/𝑒 . By definition 𝜀 𝑘 = 𝑒 -𝛾𝑘 𝜀 0 , so to ensure (2.19) at iteration 𝑘 this imposes

𝑐𝜅𝑒 𝛾 2 𝑟 𝑒 -𝛾( 2 𝑟 -2 𝑠 )𝑘 𝑡 2𝜌 𝑠 𝑘 𝜀 2 𝑟 -2 𝑠 0 ≤ 1.
Rearranging terms in last inequality, using 𝜏 defined in (2.2),

𝑡 𝑘 ≥ 𝑒 𝛾 1-𝜏 𝜌 (𝑐𝜅) 𝑠 2𝜌 𝜀 -𝜏 𝜌 0 𝑒 𝛾𝜏 𝜌 𝑘 .
Choosing 𝑡 𝑘 = 𝐶𝑒 𝛼𝑘 , where

𝐶 = 𝑒 𝛾 1-𝜏 𝜌 (𝑐𝜅) 𝑠 2𝜌 𝜀 -𝜏 𝜌 0 and 𝛼 = 𝛾𝜏 𝜌 ,
and using Lemma 2.2.1 then yields, (2.20) when 𝜏 = 0, while,

𝑓 (x) -𝑓 * ≤ exp(-𝛾𝑒 -𝛾 𝜌 (𝑐𝜅) -𝑠 2𝜌 𝑁 )𝜀 0 ,
𝑓 (x) -𝑓 * ≤ 𝜀 0 ( 𝛾𝜏 𝜌 𝑒 -𝛾 𝜌 (𝑐𝜅) -𝑠 2𝜌 𝜀 𝜏 𝜌 0 𝑁 + 1 ) 𝜌 𝜏 .
(2.21)

when 𝜏 > 0. These bounds are minimal for 𝛾 = 𝜌 and the results follow.

This bound matches the lower bounds for optimizing smooth and sharp functions [Nemirovskii and Nesterov, 1985, Page 6] up to constant factors. However, the rate of convergence of this method is controlled by the ratio between 𝜏 and 𝜌. If these are unknown, a log-scale grid search will not be able to reach the optimal rate even if 𝜌 is known since we will miss the optimal rate, controlled by the ratio 𝜌/𝜏 , by a constant factor. If both are known, in the case of non-smooth strongly convex functions for example, a grid-search on 𝐶 recovers nearly the optimal bound. Now we will see that if 𝑓 * is known, restart produces adaptive optimal rates.

Restart with termination criterion

Here, we assume that we know the optimum 𝑓 * of (P), or have an exact termination criterion. This is the case for example in zero-sum matrix games problems. We assume again that 𝑓 satisfies (Smooth) with parameters (𝑠, 𝐿) on a set 𝐽 and (Sharp) with parameters (𝑟, 𝜇) on a set 𝐾. Given an initial point 𝑥 0 we assume that smoothness and error bound are satisfied simultaneously on the sublevel set {𝑥| 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )}. We use again the universal gradient method 𝒰. Here however, we can stop the algorithm when it reaches the target accuracy as we know the optimum 𝑓 * , i.e. we stop after 𝑡 𝜀 inner iterations such that 𝑥 = 𝒰(𝑥 0 , 𝜀, 𝑡 𝜀 ) satisfies 𝑓 (𝑥) -𝑓 * ≤ 𝜀, and write the output of this method 𝑥 𝒞(𝑥 0 , 𝜀).

Here we simply restart this method and decrease the target accuracy by a constant factor after each restart. Our scheme is described in Algorithm 𝜀-RESTART.

Algorithm 3 Restart on criterion (𝜀-RESTART)

Inputs : 𝑥 0 ∈ R 𝑛 , 𝑓 * , 𝛾 ≥ 0, 𝜀 0 = 𝑓 (𝑥 0 ) -𝑓 * for 𝑘 = 1, . . . , 𝑅 do 𝜀 𝑘 := 𝑒 -𝛾 𝜀 𝑘-1 , 𝑥 𝑘 := 𝒞(𝑥 𝑘-1 , 𝜀 𝑘 ) (𝜀-RESTART) end for Output : x := 𝑥 𝑅
The following result describes the convergence of this method. It relies on the idea that it cannot do more iterations than the best scheduled restart to achieve the target accuracy at each iteration.

Proposition 2.4.1. Let 𝑓 be a convex function satisfying (Smooth) with parameter (𝑠, 𝐿) on a set 𝑄 and (Sharp) with parameters (𝑟, 𝜇) on a set 𝐾. Given 𝑥 0 ∈ R 𝑛 assume that {𝑥, 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )} ⊂ 𝑄 ∩ 𝐾. Run (𝜀-RESTART) scheme from 𝑥 0 with parameter 𝛾 = 𝜌. The precision reached at the point 𝑥 𝑅 is given by,

𝑓 (x) -𝑓 * ≤ exp ( -𝜌𝑒 -1 (𝑐𝜅) -𝑠 2𝜌 𝑁 ) (𝑓 (𝑥 0 ) -𝑓 * ) = 𝑂 ( exp(-𝜅 -𝑠 2𝜌 𝑁 ) ) , when 𝜏 = 0, while, 𝑓 (x) -𝑓 * ≤ 𝑓 (𝑥 0 ) -𝑓 * ( 𝜏 𝑒 -1 (𝑐𝜅) -𝑠 2𝜌 (𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 𝜌 𝑁 + 1 ) 𝜌 𝜏 = 𝑂 ( 𝜅 𝑠 2𝜏 𝑁 -𝜌 𝜏 ) , when 𝜏 > 0,
where 𝑁 is the total number of iterations.

Proof. Given 𝛾 ≥ 0, linear convergence of our scheme is ensured by our choice of target accuracies 𝜀 𝑘 . It remains to compute the number of iterations 𝑡 𝜀 𝑘 needed by the algorithm before the 𝑘 th restart. Following proof of Proposition 2.3.1, for 𝑘 ≥ 1 we know that target accuracy is necessarily reached after

t𝑘 = 𝑒 𝛾 1-𝜏 𝜌 (𝑐𝜅) 𝑠 2𝜌 𝜀 -𝜏 𝜌 0 𝑒 𝛾𝜏 𝜌 𝑘
iterations, such that 𝑡 𝜀 𝑘 ≤ t𝑘 . So (𝜀-RESTART) scheme achieves linear convergence while needing less inner iterates than the scheduled restart presented in Proposition 2.3.1, its convergence is therefore at least as good. For a given 𝛾 bounds (2.20) and (2.21) follow with 𝜀 0 = 𝑓 (𝑥 0 ) -𝑓 * and taking 𝛾 = 𝜌 is optimal.

Therefore if 𝑓 * is known, this method is adaptive to 𝜇 and 𝑟, contrary to the general case in Proposition 2.3.1. It can even adapt to the local values of 𝐿 or 𝜇 as we use a criterion instead of a preset schedule. Here, stopping using 𝑓 (𝑥 𝑘 ) -𝑓 * implicitly yields optimal choices of 𝐶 and 𝜏 . A closer look at the proof shows that the dependency in 𝛾 of this restart scheme is a factor ℎ(𝛾) = 𝛾𝑒 -𝛾/𝜌 of the number of iterations. Taking 𝛾 = 1, leads then to a suboptimal constant factor of at most ℎ(𝜌)/ℎ(1) ≤ 𝑒/2 ≈ 1.3 for 𝜌 ∈ [1/2, 2], so running this scheme with 𝛾 = 1 makes it parameter-free while getting nearly optimal bounds.

Composite problems & Bregman divergences

The restart schemes detailed so far focused on unconstrained problems in an Euclidean setting. Here, we extend them to more general convex optimization problems of the form minimize 𝑓 (𝑥) 𝜑(𝑥) + 𝑔(𝑥),

where 𝜑 is a convex function whose smoothness is described by parameters (𝐿, 𝑠), such that

‖∇𝜑(𝑥) -∇𝜑(𝑦)‖ * ≤ 𝐿‖𝑥 -𝑦‖ 𝑠-1 for every 𝑥, 𝑦 ∈ 𝐽, (Generic Smooth)
for a given norm ‖ • ‖ where ‖ • ‖ * is its dual norm, and 𝑔 is a simple convex function (the meaning of simple will be clarified later).

To exploit the smoothness of 𝜑 with respect to a generic norm, we assume that we have access to a prox function ℎ with dom(𝑓 ) ⊂ dom(ℎ), strongly convex with respect to the norm ‖ • ‖ with convexity parameter equal to one, which means

ℎ(𝑦) ≥ ℎ(𝑥) + ∇ℎ(𝑥) 𝑇 (𝑦 -𝑥) + 1 2 ‖𝑥 -𝑦‖ 2 , for any 𝑥, 𝑦 ∈ dom(ℎ).
We define the Bregman divergence associated to ℎ as

𝐷 ℎ (𝑦, 𝑥) = ℎ(𝑦) -ℎ(𝑥) -∇ℎ(𝑥) 𝑇 (𝑦 -𝑥), for 𝑥, 𝑦 ∈ dom(ℎ), so that 𝐷 ℎ (𝑦, 𝑥) ≥ 1 2 ‖𝑥 -𝑦‖ 2 . For ℎ(𝑥) = 1 2 ‖𝑥‖ 2 2 , we get 𝐷 ℎ (𝑦, 𝑥) = 1 2 ‖𝑥 -𝑦‖ 2
2 and recover the Euclidean setting. Given the problem geometry, appropriate choices of prox functions and associated Bregman divergences can lead to significant performance gains in high dimensional settings.

We now formally state the assumption that 𝑔 is simple. Given 𝑥, 𝑦 ∈ dom(𝑓 ) and 𝜆 ≥ 0 we assume that min 𝑧 { 𝑦 𝑇 𝑧 + 𝑔(𝑧) + 𝜆𝐷 ℎ (𝑧, 𝑥) } can be solved either in a closed form or by some fast computational procedure. Examples of such settings include sparse optimization problems, such as the LASSO, where 𝜑(𝑥) = ‖𝐴𝑥 -𝑏‖ 2 2 , with 𝐴 ∈ R 𝑚×𝑛 , 𝑏 ∈ R 𝑚 , 𝑔(𝑥) = 𝜆‖𝑥‖ 1 , with 𝜆 ≥ 0 and ℎ(𝑥) = 1 2 ‖𝑥‖ 2 . This setting also includes constrained optimization problems, where 𝑔 is the indicator function of a closed convex set. We see in numerical experiments that restart schemes applied in these two settings lead to significant performance improvements.

To apply our analysis of restart schemes we need two things: an accelerated algorithm that tackles such setting and an appropriate notion of sharpness. We first introduce the notion of relative error bound.

Definition 2.5.1. A convex function 𝑓 is called relatively sharp with respect to a strictly convex function ℎ on a set 𝐾 ⊂ dom(𝑓 ) iff there exist 𝑟 ≥ 1, 𝜇 > 0 such that

𝜇 𝑟 𝐷 ℎ (𝑥, 𝑋 * ) 𝑟 2 ≤ 𝑓 (𝑥) -𝑓 * for any 𝑥 ∈ 𝐾 (Relative Sharp)
where 𝐷 ℎ (𝑥, 𝑋 * ) = min 𝑥 * ∈𝑋 * 𝐷 ℎ (𝑥, 𝑥 * ) and 𝐷 ℎ is the Bregman divergence associated to ℎ.

If ℎ = 1 2 ‖𝑥‖ 2 2
we recover the definition of sharpness in the Euclidean setting (with slightly modified constants). This assumption is as generic as our first one in (Sharp) as it is satisfied if 𝑓 and ℎ are subanalytic [Bierstone and Milman, 1988, Th. 6.4].

The universal gradient is then the candidate in this setting. Given a target accuracy 𝜀 and an initial point 𝑥 0 , it outputs, after 𝑡 iterations, a point 𝑥 = 𝒰(𝑥 0 , 𝜀, 𝑡) such that

𝑓 (𝑥) -𝑓 * ≤ 𝜀 2 + 𝑐𝐿 2 𝑠 𝐷 ℎ (𝑥 0 , 𝑋 * ) 𝜀 2 𝑠 𝑡 2𝜌 𝑠 𝜀 2 ,
where 𝑐 = 16 here. All our previous results can then directly be transposed to the setting here, as their proofs rely only on this convergence bound. We restate them in this setting below. First if 𝜑 is known to be smooth (𝑠 = 2) the universal fast gradient algorithm simplifies as the accelerated gradient algorithm (see Appendix A.2) and the next Corollary generalizes Proposition 2.2.2.

Corollary 2.5.2. Let 𝜑, 𝑔, ℎ defining the composite problem (Composite) described above with 𝜑 satisfying (Generic Smooth) with parameters (2, 𝐿) and 𝑓 = 𝜑 + 𝑔 satisfies the (Relative Sharp) condition with respect to ℎ with parameters (𝑟, 𝜇) on a set 𝐾. Assume that we are given

𝑥 0 ∈ R 𝑛 such that {𝑥| 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )} ⊂ 𝐾. Run (RESTART) from 𝑥 0 with iteration schedule 𝑡 𝑘 = 𝐶 * 𝜅,𝜏 𝑒 𝜏 𝑘 , for 𝑘 = 1, . . . , 𝑅, where 𝐶 * 𝜅,𝜏 𝑒 1-𝜏 (𝑐𝜅) 1 2 (𝑓 (𝑥 0 ) -𝑓 * ) -𝜏 2 ,
with 𝜅 and 𝜏 defined in (2.2) and 𝑐 = 16𝑒 2/𝑒 . The precision reached at the last point x is given by,

𝑓 (x) -𝑓 * ≤ exp ( -2𝑒 -1 (𝑐𝜅) -1 2 𝑁 ) (𝑓 (𝑥 0 ) -𝑓 * ) = 𝑂 ( exp(-𝜅 -1 2 𝑁 ) ) , when 𝜏 = 0, while, 𝑓 (x) -𝑓 * ≤ 𝑓 (𝑥 0 ) -𝑓 * ( 𝜏 𝑒 -1 (𝑓 (𝑥 0 ) -𝑓 * ) 𝜏 2 (𝑐𝜅) -1 2 𝑁 + 1 ) 2 𝜏 = 𝑂 ( 𝜅 1 𝜏 𝑁 -2 𝜏 ) , when 𝜏 > 0,
where 𝑁 = ∑ 𝑅 𝑘=1 𝑡 𝑘 is the total number of iterations.

For general convex functions, the following Corollary generalizes Proposition 2.3.1.

Corollary 2.5.3. Let 𝜑, 𝑔, ℎ defining the composite problem (Composite) described above with 𝜑 satisfying (Generic Smooth) with parameters (𝑠, 𝐿) on a set 𝐽 and 𝑓 = 𝜑 + 𝑔 satisfying (Relative Sharp) with respect to ℎ with parameters (𝑟, 𝜇) on a set 𝐾.

Given 𝑥 0 ∈ R 𝑛 assume that {𝑥| 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )} ⊂ 𝑄 ∩ 𝐾. Run (Universal RESTART) scheme from 𝑥 0 for given 𝜀 0 ≥ 𝑓 (𝑥 0 ) -𝑓 * , 𝛾 = 𝜌, 𝑡 𝑘 = 𝐶 * 𝜅,𝜏,𝜌 𝑒 𝜏 𝑘 , where 𝐶 * 𝜅,𝜏,𝜌 𝑒 1-𝜏 (𝑐𝜅) 𝑠 2𝜌 𝜀 -𝜏 𝜌 0
where 𝜌 is defined in (2.18), 𝜅 and 𝜏 are defined in (2.2) and 𝑐 = 16𝑒 2/𝑒 . The precision reached at the last point x is given by,

𝑓 (x) -𝑓 * ≤ exp ( -𝜌𝑒 -1 (𝑐𝜅) -𝑠 2𝜌 𝑁 ) 𝜀 0 = 𝑂 ( exp(-𝜅 -𝑠 2𝜌 𝑁 ) ) , when 𝜏 = 0, while, 𝑓 (x) -𝑓 * ≤ 𝜀 0 ( 𝜏 𝑒 -1 (𝑐𝜅) -𝑠 2𝜌 𝜀 𝜏 𝜌 0 𝑁 + 1 ) 𝜌 𝜏 = 𝑂 ( 𝜅 𝑠 2𝜏 𝑁 -𝜌 𝜏 ) , when 𝜏 > 0,
where 𝑁 = ∑ 𝑅 𝑘=1 𝑡 𝑘 is total number of iterations.

The results regarding adaptive schemes and those with termination criterion generalize similarly under the relative sharpness assumption.

Numerical results

We illustrate our results by testing our adaptive restart methods, denoted Adap and Crit (𝜀-Restart), introduced respectively in Sections 2.2.2 and 2.4 on several problems and compare them against simple gradient descent (Grad ), accelerated gradient methods (Acc), and the restart heuristic enforcing monotonicity (Mono in [O'Donoghue and Candes, 2015]). For Adap we plot the convergence of the best method found by grid search to compare with the restart heuristic. This implicitly assumes that the grid search is run in parallel with enough servers. For Crit we use the optimal 𝑓 * found by another solver. This gives an overview of its performance in order to potentially approximate it along the iterations as with Polyak steps [START_REF] Poljak | Introduction to optimization[END_REF]. All restart schemes are using the accelerated gradient with backtracking line search detailed in the Supplementary Material, with large dots representing restart iterations.

In Figures 2-1, we solve classification problems with various losses on the UCI Sonar data set [START_REF] Newman | Uci machine learning repository[END_REF] with (𝑛, 𝑑) = (208, 60). For least square loss on sonar data set, we observe much faster convergence of the restart schemes compared to the accelerated method. These results were already observed by O'Donoghue and Candes [2015]. For logistic loss, we observe that restart does not provide much improvement. For hinge loss, we regularized by a squared norm and optimize the dual, which means solving a quadratic problem with box constraints. We observe here that the scheduled restart scheme converges much faster, while restart heuristics may be activated too late. We observe similar results for the LASSO problem. In general Crit ensures acceleration rates consistent with the theory but Adap exhibits more consistent behavior. This highlights the benefits of a Hölderian error bound assumption for these last two problems. Precisely quantifying sharpness of the function from data/problem structure is a key open problem.

To account for the grid search effort, in Figure 2-2, we multiplied the number of iterations made by the Adap method by the size of the grid. This is for the LASSO problem on Sonar data set with a grid step size of 4. This shows that the benefits of the restart schemes make the grid search effort acceptable both on paper and in practice. More clever grid search strategies for scheduled restarts run in parallel would even reduce the grid search effort.

Conclusion

We have shown that Hölderian error bounds that generically describe convex functions can be optimally treated by restart schemes of fast convex optimization algorithms for smooth or non-smooth functions. These restarts are shown to be robust to misspecification of the parameters of the error bound as a log-scale grid search on these parameters offer near-optimal rates for smooth convex functions. Schedules are however still a drawback of our approach as they do not adapt to local geometries of the function. Error bounds can indeed be formulated at the distance to a sublevel set of the function instead of the set of minimizers. Our restart with criterion brings a partial resolution of the question when 𝑓 * is known. Recently Fercoq and Qu [2017] developed generic adaptive schemes for quadratic local error bounds that give theoretical guarantees of acceleration for ℓ 1 regularized or constrained problems by restart Lemma 2.A.1. Let 𝑥 𝑘 be a sequence whose 𝑘 th iterate is generated from previous one by an algorithm that needs 𝑡 𝑘 iterations and denote 𝑁 = ∑ 𝑅 𝑘=1 𝑡 𝑘 the total number of iterations to output a point x = 𝑥 𝑅 . Suppose setting

𝑡 𝑘 = ⌈𝐶𝑒 𝛼𝑘 ⌉, 𝑘 = 1, . . . , 𝑅
for some 𝐶 > 0 and 𝛼 ≥ 0 ensures that objective values 𝑓 (𝑥 𝑘 ) converge linearly, i.e.

𝑓 (𝑥

𝑘 ) -𝑓 * ≤ 𝜈𝑒 -𝛾𝑘 , (2.22)
for all 𝑘 ≥ 0 with 𝜈 ≥ 0 and 𝛾 ≥ 0. Then precision at the output is given by, 𝑓 (x) -𝑓 * ≤ 𝜈 exp(-𝛾𝑁/(𝐶 + 1)), when 𝛼 = 0,

and 𝑓 (x) -𝑓 * ≤ 𝜈 (𝛼𝑒 -𝛼 𝐶 -1 𝑁 ′ + 1) 𝛾 𝛼
, when 𝛼 > 0,

where

𝑁 ′ = 𝑁 - log((𝑒 𝛼 -1)𝑒 -𝛼 𝐶 -1 𝑁 +1) 𝛼 . Proof. At the 𝑅 th point generated, 𝑁 = ∑ 𝑅 𝑘=1 𝑡 𝑘 . If 𝑡 𝑘 = ⌈𝐶⌉, define 𝜀 = ⌈𝐶⌉-𝐶 such that 0 ≤ 𝜀 < 1. Then 𝑁 = 𝑅(𝐶 + 𝜀), injecting it in (2.22) at the 𝑅 th point, 𝑓 (x) -𝑓 * ≤ 𝜈𝑒 -𝛾 𝑁 𝐶+𝜀 ≤ 𝜈𝑒 -𝛾 𝑁 𝐶+1 . Now, if 𝑡 𝑘 = ⌈𝐶𝑒 𝛼𝑘 ⌉, define 𝜀 𝑘 = ⌈𝐶𝑒 𝛼𝑘 ⌉ -𝐶𝑒 𝛼𝑘 , such that 0 ≤ 𝜀 𝑘 < 1. On one hand 𝑁 ≥ 𝑅 ∑ 𝑘=1 𝐶𝑒 𝛼𝑘 , such that 𝑅 ≤ log ((𝑒 𝛼 -1)𝑒 -𝛼 𝐶 -1 𝑁 + 1) 𝛼 .
On the other hand,

𝑁 = 𝑅 ∑ 𝑘=1 𝑡 𝑘 = 𝐶𝑒 𝛼 𝑒 𝛼 -1 (𝑒 𝛼𝑅 -1) + 𝑅 ∑ 𝑘=1 𝜀 𝑘 ≤ 𝐶𝑒 𝛼 𝑒 𝛼 -1 (𝑒 𝛼𝑅 -1) + 𝑅 ≤ 𝐶𝑒 𝛼 𝑒 𝛼 -1 (𝑒 𝛼𝑅 -1) + log ((𝑒 𝛼 -1)𝑒 -𝛼 𝐶 -1 𝑁 + 1) 𝛼 , such that 𝑅 ≥ log (𝛼𝑒 -𝛼 𝐶 -1 𝑁 ′ + 1) 𝛼 .
Injecting it in (2.22) at the 𝑅 th point the result follows.

Chapter 3

A brief introduction to sparse problems

Sparsity has been studied early on by statisticians to select relevant features in data fitting problem, see [START_REF] Guyon | An introduction to variable and feature selection[END_REF] for a review. A whole field, namely compressed sensing, has then been dedicated to its analysis as it provides new codingdecoding procedures that break Shannon compression rate barrier, see [START_REF] Candès | An introduction to compressive sampling[END_REF] for an introduction. Here we briefly recall models that tackle sparsity assumptions and their structure for which geometrical analysis provides insights on their optimization process.

Original sparse problems

The term sparsity has been extensively used to qualify diverse structures of variables that compress their information. Originally a vector is said to be sparse if it has few non-zero coefficients. Formally, a vector 𝑥 ∈ R 𝑑 is 𝑠-sparse if its support Supp(𝑥) = {𝑖 ∈ 1, 𝑑 : 𝑥 𝑖 ̸ = 0} has cardinality at most 𝑠. Sparsity assumption can be used for example in linear regression problems. Assume that we are given 𝑛 noisy observations 𝑏 1 , . . . , 𝑏 𝑛 of a 𝑠-sparse vector 𝑥 * ∈ R 𝑑 by data points 𝑎 1 , . . . , 𝑎 𝑛 ∈ R 𝑑 such that 𝑏 𝑖 = 𝑎 𝑇 𝑖 𝑥 𝑖 + 𝜂 𝑖 , for every 𝑖 ∈ 1, 𝑑 , where 𝜂 𝑖 are bounded i.i.d. noise. Simple least-square estimation can retrieve 𝑥 * if the number of samples 𝑛 is larger than the number of features 𝑑. However as it does not exploit the sparsity assumption, estimated solution will be dense and some coefficients only capture noise signal. Moreover if the number of samples is strictly less than the number of features, least-square estimation simply fails.

Models

Several models have thus been developed to exploit sparsity assumption and recover the original signal even if 𝑛 < 𝑑. When observations are not perturbed by noise a straight strategy to recover original signal is to minimize the number of non-zero coefficients of vectors satisfying the observations. However this problem is NP-hard [START_REF] Natarajan | Sparse approximate solutions to linear systems[END_REF] so one rather seeks to solve its convex relaxation called Basis Pursuit [START_REF] Chen | Atomic decomposition by basis pursuit[END_REF]

that reads minimize ‖𝑥‖ 1 subject to 𝐴𝑥 = 𝑏 (3.1)
in variable 𝑥 ∈ R 𝑑 , where 𝐴 = (𝑎 1 , . . . , 𝑎 𝑛 ) 𝑇 ∈ R 𝑛×𝑑 is the matrix of data points and 𝑏 = (𝑏 1 , . . . , 𝑏 𝑛 ) ∈ R 𝑛 is the vector of observations. To account for noise, a robust version of (3.1) reads

minimize ‖𝑥‖ 1 subject to ‖𝐴𝑥 -𝑏‖ 2 ≤ 𝛿‖𝐴‖ 2 , (3.2)
in variable 𝑥 ∈ R 𝑑 , where 𝛿 is an estimation of the level of noise and ‖𝐴‖ 2 is the spectral norm of the observation matrix 𝐴.

Another approach to retrieve the original signal is to minimize a data fitting term under sparsity constraints. As these latter are not convex, one consider the relaxed versions that constrains the ℓ 1 norm of the variable. Its penalized version is well known as the LASSO [START_REF] Tibshirani | Regression shrinkage and selection via the LASSO[END_REF] problem that reads

minimize 1 2 ‖𝐴𝑥 -𝑏‖ 2 2 + 𝜆‖𝑥‖ 1 (3.3) in variable 𝑥 ∈ R 𝑑 ,
where 𝜆 ≥ 0 is a regularization parameter. Finally by looking at the optimal conditions of the LASSO problem, another estimator called Dantzig selector [START_REF] Candes | The dantzig selector: Statistical estimation when p is much larger than n[END_REF] was developed.

Recovery performance

The main question that drew interest of the compressed sensing community was to certify that, by solving these problems, one can retrieve the original vector at least approximatively, i.e. decode it from the observations. Precisely, they were interested in the number of observations needed to recover the signal and the robustness to noise of these procedures. Naturally the answer depends on properties of the observation matrix 𝐴, i.e. how the original vector was encoded by linear observations.

In their seminal works [START_REF] Candès | Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information[END_REF]; [START_REF] Candes | Near-optimal signal recovery from random projections: Universal encoding strategies?[END_REF]; [START_REF] Donoho | For most large underdetermined systems of linear equations the minimal l1-norm solution is also the sparsest solution[END_REF] show that for an original 𝑠-sparse vector, 𝑂(𝑠 log 𝑑) random observations of it are sufficient to decode it by solving one of these problems. To this end, they introduce the restricted isometric property defined below. Definition 3.1.1. A matrix 𝐴 ∈ R 𝑛×𝑑 satisfies the Restricted Isometric Property (RIP) at level 𝑠, if there exists 𝛿 𝑠 such that for any submatrix 𝐴 𝑠 ∈ R 𝑛×𝑠 and any 𝑠-sparse vector 𝑥,

(1 [START_REF] Candès | Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information[END_REF]; [START_REF] Candes | Near-optimal signal recovery from random projections: Universal encoding strategies?[END_REF] then proved that the RIP at level 𝑠 with sufficiently small parameter 𝛿 𝑠 ensures recovery of 𝑠-sparse signals and that if the matrix of observations 𝐴 is drawn from a Gaussian random distribution then the RIP is satisfied.

-𝛿 𝑠 )‖𝑥‖ 2 2 ≤ ‖𝐴 𝑠 𝑥‖ 2 2 ≤ (1 + 𝛿 𝑠 )‖𝑥‖ 2 2 .
Further assumptions on the observation matrix 𝐴 were then defined to ensure recovery. [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Feuer | On sparse representation in pairs of bases[END_REF]; [START_REF] Cohen | Compressed sensing and best k-term approximation[END_REF] notably introduced the null space property defined below.

Definition 3.1.2. A matrix 𝐴 satisfies the Null Space Property (NSP) at order 𝑠 with constant 𝛼 ≥ 1 if for any support 𝑆 ⊂ {1, . . . 𝑝} of cardinality at most 𝑠 and any

𝑧 ∈ Null(𝐴) ∖ {0}, 𝛼‖𝑧 𝑆 ‖ 1 < ‖𝑧 𝑆 𝑐 ‖ 1 ,
where Null(𝐴) is the null space of 𝐴, and 𝑧 𝑆 ∈ R 𝑝 denotes the vector obtained by zeroing all coefficients of 𝑧 that are not in 𝑆.

NSP provides sufficient and necessary conditions for exact recovery of signals in the absence of noise when solving problem(3.1). For LASSO problem (3.3) or Dantzig selector, numerous properties were defined, [START_REF] Van De Geer | On the conditions used to prove oracle results for the lasso[END_REF] summarized and linked them. Minimal conically restricted singular values defined by [START_REF] Bickel | Simultaneous analysis of lasso and dantzig selector[END_REF] that read for a matrix 𝐴,

𝜅(𝑠, 𝛼) = inf 𝑆∈ 1,𝑝 Card(𝑆)≤𝑠 inf ‖𝑥 𝑆 𝑐 ‖ 1 ≤𝛼‖𝑥 𝑆 ‖ 1 ‖𝑥 𝑆 ‖ 2 =1
‖𝐴𝑥‖ 2 appeared to be the essential quantity that control exact and robust recovery. These assumptions were linked early to Kolmogorov width [START_REF] Kashin | A remark on compressed sensing[END_REF] or Gaussian widths [START_REF] Chandrasekaran | The convex geometry of linear inverse problems[END_REF][START_REF] Raskutti | Restricted eigenvalue properties for correlated gaussian designs[END_REF] of sections of unit balls which allow a simpler and more geometrical approach to these problems.

Overall, these works provide a fine description of the geometry of the problem that can be used for optimization purposes.

Optimization procedures

On an optimization point of view problems (3.1) and (3.2) on one side and problem (3.3) have different structures and are therefore tackled by different procedures that we present now.

Basis pursuits

As the ℓ 1 norm is non-smooth, problems (3.1) and (3.2) belong to the class of non-smooth convex problems and shall have complexity 𝑂(1/ √ 𝜀). However analysis does not take into account the structure of the problem, namely that the ℓ 1 norm can be written as a maximum of linear functions. In his seminal paper, [START_REF] Nesterov | Smooth minimization of non-smooth functions[END_REF] presented how this additional information can be exploited to get an approximate solution in 𝑂(1/𝜀) iterations by smoothing the objective function and applying his accelerated algorithm for smooth convex optimization. This remark was exploited by [START_REF] Becker | Nesta: A fast and accurate first-order method for sparse recovery[END_REF] on problems (3.1) and (3.2). They also provide restarting heuristics of their algorithm, called continuation steps, that show remarkable performance but they do not explain this phenomenon.

Łojasiewicz inequality can then help. While the ℓ 1 norm is clearly not strongly convex, it is sharp, i.e. it satisfies

𝛾𝑑(𝑥, 𝑋 * ) ≤ 𝑓 (𝑥) -𝑓 * , for every 𝑥 ∈ R 𝑑
where 𝑓 (𝑥) = ‖𝑥‖ 1 , 𝑓 * = 0, 𝑋 * = 0, 𝑑(𝑥, 𝑋 * ) is the ℓ 1 distance of 𝑥 to the minimizer and 𝛾 = 1. This property generalizes to problem (3.1) and can be exploited to produce linearly convergent restart schemes as shown in Chapter 4. The sharpness constant depends then on the data 𝐴 that defines the set of constraints. Interestingly, sharpness can be estimated by the same conic properties of the problem that control recovery performance. A finer analysis in Chapter 4 show that a single quantity controls then both statistical and computational performance.

LASSO

The LASSO problem (3.3) drew more attention of the optimization community for its applicability in prediction problems. Accelerated methods for smooth convex composite problems apply in this setting, they were first developed by [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] with FISTA algorithm. As the ℓ 1 norm is decomposable, coordinate accelerated methods were also proposed [START_REF] Fercoq | Accelerated, parallel, and proximal coordinate descent[END_REF][START_REF] Lee | Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems[END_REF][START_REF] Nesterov | Efficiency of coordinate descent methods on huge-scale optimization problems[END_REF]. Some specialized schemes discard features that are certified to be zero along the optimization process see Fercoq et al. [2015] and references herein. Finally homotopy methods such LARS [START_REF] Efron | Least angle regression[END_REF] build the regularization path of the method, i.e. the curve of solutions for varying regularization parameters.

In the undetermined case, where 𝑛 < 𝑑, problem (3.3) is convex but not strongly convex. However here again restart schemes were shown to outperform simple accelerated methods [O'donoghue and Candes, 2015]. Further properties of the problem were explored to prove linear convergence of a simple gradient descent [START_REF] Yen | Constant nullspace strong convexity and fast convergence of proximal methods under highdimensional settings[END_REF][START_REF] Agarwal | Fast global convergence rates of gradient methods for high-dimensional statistical recovery[END_REF]. A closer inspection of the structure highlighted that LASSO problem satisfies the Łojasiewicz inequality with exponent 𝑟 = 2 [START_REF] Bolte | From error bounds to the complexity of first-order descent methods for convex functions[END_REF][START_REF] Zhang | The restricted strong convexity revisited: analysis of equivalence to error bound and quadratic growth[END_REF], which explained once more why restart schemes accelerate standard methods and allow theoretical guarantees as done by Fercoq andQu [2016, 2017].

Generalized sparse structure

Original sparsity is one assumption among others that present how some variables can be compressed. For example if the variable is a matrix rather than a vector, compression can mean low rank of the matrix. Several point of views were taken to ensure such compressed structures, loosely called sparse structures. A first approach is to consider these structures as the description of variables by a finite set of "atoms" [START_REF] Chandrasekaran | The convex geometry of linear inverse problems[END_REF]. The norm induced by the convex hull of these atoms can then be used to build specific estimators. Their geometrical definition has then been used to analyze recovery performance of problems of the form (3.1), (3.2) or (3.3) [START_REF] Raskutti | Restricted eigenvalue properties for correlated gaussian designs[END_REF].

Combinatorial penalties can also be used to induce sparse structures on the variables. Original sparsity for example is the cardinality of the support of the variable. It was generalized by using submodular functions on sets defined by a vector, like its support [START_REF] Bach | Learning with submodular functions: A convex optimization perspective[END_REF]. Their continuous extensions can then be used as regularizers for machine learning problems [START_REF] Bach | Optimization with sparsity-inducing penalties[END_REF]. Furthermore by identifying the underlying combinatorial problem, this approach provides practical operators for the resulting regularizer. Notice that generally the support function of the symmetric base polytope of the support function is the dual norm of an atomic norm, such that both point of views merge.

Finally several frameworks attempt to unify sparse structures under generic assumptions to identify their statistical properties [START_REF] Negahban | A unified framework for high-dimensional analysis of 𝑚-estimators with decomposable regularizers[END_REF]. Recovery performances of corresponding problems require then generalization of the previous properties as done e.g. by [START_REF] Juditsky | On a unified view of nullspacetype conditions for recoveries associated with general sparsity structures[END_REF]; [START_REF] Oymak | New null space results and recovery thresholds for matrix rank minimization[END_REF].

51

Chapter 4

On computational and statistical performances of sparse recovery problems

Chapter Abstract

In the past years, sparse recovery problems have received a lot of attention from various perspectives. On one side, an extensive literature explores attainable statistical performance, that is, the possibility to retrieve the underlying original signal, depending on the number of observations or the noise level. On the other side, a long list of algorithms now solve these problems at a low computational cost. Here, we provide evidence that these two aspects are intimately related. Loosely stated, the more accurate the recovery is, the faster it is to compute. As a first step, we develop a linearly convergent restart scheme for exact recovery whose rate is controlled by the recovery threshold of the problem. Then, we analyze the underlying conic geometry of sparse recovery problems. On the statistical side, minimal conically-restricted singular values of the observation matrix control robust recovery performance. On the computational side, Renegar's condition number, generalizing classical condition numbers in linear algebra to conic systems, is shown to control computational complexity of optimality certificates for exact recovery and the restart scheme we present. Numerical experiments illustrate its impact for several other classical algorithms. By observing that the worst-case value of this algorithmic complexity measure taken over all signals matches the minimal conically-restricted singular value, we argue that, in these problems, a single parameter directly controls computational complexity and statistical recovery performance

Introduction

Sparse recovery problems have received a lot of attention from various perspectives. On one side, an extensive literature explores the limits of recovery performance. On the other side, a long list of algorithms now solve these problems very efficiently. Early on, it was noticed empirically by e.g. [START_REF] Donoho | Fast solution of ℓ 1 -norm minimization problems when the solution may be sparse-norm minimization problems when the solution may be sparse[END_REF], that recovery problems which are easier to solve from a statistical point of view (i.e., where more samples are available), are also easier to solve numerically. Here, we show that these two aspects are indeed intimately related.

Recovery problems consist in retrieving a signal 𝑥 * , lying in some Euclidean space 𝐸, given linear observations. If the signal is "sparse", namely if it can be efficiently compressed, a common approach is to minimize the corresponding sparsity inducing norm ‖ • ‖ (e.g. the ℓ 1 norm in classical sparse recovery). in the variable 𝑥 ∈ 𝐸 where 𝜆 > 0 is a penalization parameter. This last problem is known as the LASSO [START_REF] Tibshirani | Regression shrinkage and selection via the LASSO[END_REF] in the ℓ 1 case.

When 𝑥 * has no more than 𝑠 non zero values, [START_REF] Donoho | Sparse nonnegative solutions of underdetermined linear equations by linear programming[END_REF] and [START_REF] Candes | Near-optimal signal recovery from random projections: Universal encoding strategies?[END_REF] have shown that, for certain linear operators 𝐴, 𝑂(𝑠 log 𝑝) observations suffice for stable recovery of 𝑥 * by solving the exact formulation (4.1) using the ℓ 1 norm (a linear program). These results have then been generalized to many other recovery problems with various assumptions on signal structure (e.g., where 𝑥 is a block-sparse vector, a low-rank matrix, etc.) and corresponding convex relaxations were developed in those cases (see e.g. [START_REF] Chandrasekaran | The convex geometry of linear inverse problems[END_REF] and references therein). Recovery performance is often measured in terms of the number of samples required to guarantee exact or robust recovery given a level of noise.

On the computational side, many algorithms were developed to solve these problems at scale. Besides specialized methods such as LARS [START_REF] Efron | Least angle regression[END_REF], FISTA [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] and NESTA [START_REF] Becker | Nesta: A fast and accurate first-order method for sparse recovery[END_REF], solvers use accelerated gradient methods to solve robust recovery problems, with efficient and flexible implementations covering a wide range of compressed sensing instances developed by e.g. [START_REF] Becker | Templates for convex cone problems with applications to sparse signal recovery[END_REF]. Recently, linear convergence results have been obtained for the LASSO [START_REF] Agarwal | Fast global convergence rates of gradient methods for high-dimensional statistical recovery[END_REF][START_REF] Yen | Constant nullspace strong convexity and fast convergence of proximal methods under highdimensional settings[END_REF][START_REF] Zhou | l1, p-norm regularization: Error bounds and convergence rate analysis of first-order methods[END_REF] using variants of the classical strong convexity assumption. Some restart schemes have also been developed in e.g. [O'Donoghue and Candes, 2015;[START_REF] Su | A differential equation for modeling nesterov's accelerated gradient method: Theory and insights[END_REF][START_REF] Giselsson | Monotonicity and restart in fast gradient methods[END_REF] while [START_REF] Fercoq | Restarting accelerated gradient methods with a rough strong convexity estimate[END_REF] showed that generic restart schemes can offer linear convergence given a rough estimate of the behavior of the function around its minimizers.

As mentioned above, [START_REF] Donoho | Fast solution of ℓ 1 -norm minimization problems when the solution may be sparse-norm minimization problems when the solution may be sparse[END_REF] was one of the first reference to connect statistical and computational performance in this case, showing empirically that recovery problems which are easier to solve from a statistical point of view (i.e., where more samples are available), are also easier to solve numerically (using homotopy methods). More recently, [START_REF] Chandrasekaran | Computational and statistical tradeoffs via convex relaxation[END_REF]; Amelunxen et al.

[2014] studied computational and statistical trade offs for increasingly tight convex relaxations of shrinkage estimators. They show that recovery performance is directly linked to the Gaussian squared-complexity of the tangent cone with respect to the constraint set and study the complexity of several convex relaxations. In [START_REF] Chandrasekaran | Computational and statistical tradeoffs via convex relaxation[END_REF]Amelunxen et al., 2014] however, the structure of the convex relaxation is varying and affecting both complexity and recovery performance, while in [START_REF] Donoho | Fast solution of ℓ 1 -norm minimization problems when the solution may be sparse-norm minimization problems when the solution may be sparse[END_REF] and in what follows, the structure of the relaxation is fixed, but the data (i.e. the observation matrix 𝐴) varies.

Here, as a first step, we study the exact recovery case and show that the null space property introduced by [START_REF] Cohen | Compressed sensing and best k-term approximation[END_REF] can be seen as a measure of sharpness on the optimum of the sparse recovery problem. On one hand this allows us to develop linearly convergent restart schemes whose rate depends on this sharpness. On the other hand we recall how the null space property is linked to the recovery threshold of the sensing operator 𝐴 for random designs, thus producing a clear link between statistical and computational performance.

We then analyze the underlying conic geometry of recovery problems. Robust recovery performance is controlled by a minimal conically restricted singular value. We recall Renegar's condition number and show how it affects the computational complexity of optimality certificates for exact recovery and the linear convergence rate of restart schemes. By observing that the minimal conically restricted singular value matches the worst case value of Renegar's condition number on sparse signals, we provide further evidence that a single quantity controls both computational and statistical aspects of recovery problems. Numerical experiments illustrate its impact on various classical algorithms for sparse recovery.

The first two sections focus on the ℓ 1 case for simplicity. We generalize our results to non-overlapping group norms and the nuclear norm in a third section.

Notations

For a given integer 𝑝 ≥ 1, 1, 𝑝 denotes the set of integers between 1 and 𝑝. For a given subset 𝑆 ⊂ 1, 𝑝 , we denote 𝑆 𝑐 = 1, 𝑝 ∖ 𝑆 its complementary and Card(𝑆) its cardinality. For a given vector 𝑥 ∈ R 𝑝 , we denote Supp(𝑥) = {𝑖 ∈ 1, 𝑝 : 𝑥 𝑖 ̸ = 0} the support of 𝑥, ‖𝑥‖ 0 = Card(Supp(𝑥)) its sparsity and ‖𝑥‖ 𝑝 its 𝑝-norm. For a given vector 𝑥 and integer subset 𝑆 ⊂ 1, 𝑝 , 𝑥 𝑆 ∈ R 𝑝 denotes the vector obtained by zeroing all coefficients of 𝑥 that are not in 𝑆. For a given linear operator or matrix 𝐴, we denote Null(𝐴) its null space, ℑ(𝐴) its range, and ‖𝑋‖ 2 its operator norm with respect to the Euclidean norm (for matrices this is the spectral norm). The identity operator is denoted I. In a linear topological space 𝐸 we denote int(𝐹 ) the interior of 𝐹 ⊂ 𝐸. Finally for a given real 𝑎, we denote ⌈𝑎⌉ the smallest integer larger than or equal to 𝑎 and ⌊𝑎⌋ the largest integer smaller than or equal to 𝑎.

Recovery performance and linear convergent restart scheme for exact recovery

In this section and the following one, we discuss sparse recovery problems using the ℓ 1 norm. Given a matrix 𝐴 ∈ R 𝑛×𝑝 and observations 𝑏 = 𝐴𝑥 * on a signal 𝑥 * ∈ R 𝑝 , recovery is performed by solving the ℓ 1 minimization program

minimize ‖𝑥‖ 1 subject to 𝐴𝑥 = 𝑏 (ℓ 1 recovery) in the variable 𝑥 ∈ R 𝑝 .
In what follows, we show that the Null Space Property condition (recalled below) can be seen as measure of sharpness for ℓ 1 -recovery of a sparse signals 𝑥 * , with

‖𝑥‖ 1 -‖𝑥 * ‖ 1 > 𝛾‖𝑥 -𝑥 * ‖ 1 (Sharp)
for any 𝑥 ̸ = 𝑥 * such that 𝐴𝑥 = 𝑏, and some 0 ≤ 𝛾 < 1. This first ensures that 𝑥 * is the unique minimizer of problem (ℓ 1 recovery) but also has important computational implications. It allows us to produce linear convergent restart schemes whose rates depend on sharpness. By connecting null space property to recovery threshold for random observation matrices, we thus get a direct link between computational and statistical aspects of sparse recovery problems.

Null space property & sharpness for exact recovery

Although the definition of null space property appeared in earlier work [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF][START_REF] Feuer | On sparse representation in pairs of bases[END_REF]] the terminology of restricted null space is due to [START_REF] Cohen | Compressed sensing and best k-term approximation[END_REF]. The following definition differs slightly from the original one in order to relate it to intrinsic geometric properties of the problem in Section 4.2.

𝛼‖𝑧 𝑆 ‖ 1 < ‖𝑧 𝑆 𝑐 ‖ 1 .
(NSP)

The matrix 𝐴 satisfies the Null Space Property at order 𝑠 with constant 𝛼 ≥ 1 if it satisfies it on every support 𝑆 of cardinality at most 𝑠.

The Null Space Property is a necessary and sufficient condition for the convex program (ℓ 1 recovery) to recover all signals up to some sparsity threshold. Necessity will follow from results recalled in Section 4.2.2. We detail sufficiency of (NSP) using sharpness in the following proposition.

Proposition 4.1.2. Given a coding matrix 𝐴 ∈ R 𝑛×𝑝 satisfying (NSP) at order 𝑠 with constant 𝛼 ≥ 1, if the original signal 𝑥 * is 𝑠-sparse, then for any 𝑥 ∈ R 𝑝 satisfying 𝐴𝑥 = 𝑏, 𝑥 ̸ = 𝑥 * , we have

‖𝑥‖ 1 -‖𝑥 * ‖ 1 > 𝛼 -1 𝛼 + 1 ‖𝑥 -𝑥 * ‖ 1 .
This implies signal recovery, i.e. optimality of 𝑥 * for (ℓ 1 recovery), and the sharpness bound (Sharp) with 𝛾 = 𝛼-1 𝛼+1 .

Proof. The proof follows the one in Cohen et al. [2009, Theorem 4.4]. Let 𝑆 = supp(𝑥 * ), with Card(𝑆) ≤ 𝑠, and let 𝑥 ̸ = 𝑥 * such that 𝐴𝑥 = 𝑏, so 𝑧 = 𝑥 -𝑥 * ̸ = 0 satisfies 𝐴𝑧 = 0. Then

‖𝑥‖ 1 = ‖𝑥 * 𝑆 + 𝑧 𝑆 ‖ 1 + ‖𝑧 𝑆 𝑐 ‖ 1 ≥ ‖𝑥 * 𝑆 ‖ 1 -‖𝑧 𝑆 ‖ 1 + ‖𝑧 𝑆 𝑐 ‖ 1 = ‖𝑥 * ‖ 1 + ‖𝑧‖ 1 -2‖𝑧 𝑆 ‖ 1 . Now as 𝐴 satisfies (NSP) on support 𝑆, ‖𝑧‖ 1 = ‖𝑧 𝑆 ‖ 1 + ‖𝑧 𝑆 𝑐 ‖ 1 > (1 + 𝛼)‖𝑧 𝑆 ‖ 1 hence ‖𝑥‖ 1 -‖𝑥 * ‖ 1 > 𝛼 -1 𝛼 + 1 ‖𝑧‖ 1 = 𝛼 -1 𝛼 + 1 ‖𝑥 -𝑥 * ‖ 1 .
As 𝛼 ≥ 1, this implies that 𝑥 * is the solution of program (ℓ 1 recovery) and the corresponding sharpness bound.

Sharpness is a crucial property for optimization problems that can be exploited to accelerate the performance of classical optimization algorithms [START_REF] Nemirovskii | Optimal methods of smooth convex minimization[END_REF][START_REF] Roulet | Sharpness, restart and acceleration[END_REF]. Before that we remark that it is in fact equivalent to (NSP) at order 𝑠.

Proposition 4.1.3. Given a matrix 𝐴 ∈ R 𝑛×𝑝 such that problem (ℓ 1 recovery) is sharp on every 𝑠-sparse signal 𝑥 * , i.e. there exists 0 ≤ 𝛾 < 1 such that

‖𝑥‖ 1 -‖𝑥 * ‖ 1 > 𝛾‖𝑥 -𝑥 * ‖ 1 ,
for any 𝑥 ̸ = 𝑥 * such that 𝐴𝑥 = 𝐴𝑥 * . Then, 𝐴 satisfies (NSP) at order 𝑠 with constant 𝛼 = 1+𝛾 1-𝛾 ≥ 1.

Proof. Let 𝑆 ⊂ 1, 𝑝 with Card(𝑆) ≤ 𝑠 and 𝑧 ∈ Null(𝐴), 𝑧 ̸ = 0, such that 𝐴𝑧 𝑆 = -𝐴𝑧 𝑆 𝑐 and 𝑧 𝑆 ̸ = -𝑧 𝑆 𝑐 . Using sharpness of problem (ℓ 1 recovery) with 𝑥 * = 𝑧 𝑆 , and 𝑥 = -𝑧 𝑆 𝑐 , we get

‖𝑧 𝑆 𝑐 ‖ 1 -‖𝑧 𝑆 ‖ 1 > 𝛾‖𝑧‖ 1 = 𝛾‖𝑧 𝑆 ‖ 1 + 𝛾‖𝑧 𝑆 𝑐 ‖ 1 .
Rearranging terms and using 𝛾 < 1, this reads

‖𝑧 𝑆 𝑐 ‖ 1 > 1 + 𝛾 1 -𝛾 ‖𝑧 𝑆 ‖ 1 ,
which is (NSP) on support 𝑆 with the announced constant. As 𝑆 was taken arbitrarily, this means (NSP) holds at order 𝑠.

Restarting first-order methods

In this section, we seek to solve the recovery problem (ℓ 1 recovery) and exploit the sharpness bound (Sharp). The NESTA algorithm [START_REF] Becker | Nesta: A fast and accurate first-order method for sparse recovery[END_REF] uses the smoothing argument of Nesterov [2013a] to solve (ℓ 1 recovery). In practice, this means using the optimal algorithm of [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate 𝑂(1/𝑘 2 )[END_REF] to minimize

𝑓 𝜀 (𝑥) sup ‖𝑢‖∞≤1 { 𝑢 𝑇 𝑥 - 𝜀 2𝑝 ‖𝑢‖ 2 2 }
for some 𝜀 > 0, which approximates the ℓ 1 norm uniformly up to 𝜀/2. This is the classical Huber function, which has a Lipschitz continuous gradient with constant equal to 𝑝/𝜀. Overall given an accuracy 𝜀 and a starting point 𝑥 0 this method outputs after 𝑡 iterations a point 𝑥 = 𝒜(𝑥 0 , 𝜀, 𝑡) such that

‖𝑥‖ 1 -‖x‖ 1 ≤ 2𝑝‖𝑥 0 -x‖ 2 2 𝜀𝑡 2 + 𝜀 2 , (4.2)
for any x solution of problem (ℓ 1 recovery). Now if the sharpness bound is satisfied, restarting this method, as described in the (Restart) scheme presented below, accelerates its convergence.

Algorithm 4 Restart Scheme (Restart)

Input: Initial point 𝑦 0 ∈ R 𝑝 , initial gap 𝜀 0 ≥ ‖𝑦 0 ‖ 1 -‖x‖ 1 , decreasing factor 𝜌, restart clock 𝑡 For 𝑘 = 1 . . . , 𝐾 compute 𝜀 𝑘 = 𝜌𝜀 𝑘-1 , 𝑦 𝑘 = 𝒜(𝑦 𝑘-1 , 𝜀 𝑘 , 𝑡) (Restart)
Output: A point ŷ = 𝑦 𝐾 approximately solving (ℓ 1 recovery).

Optimal restart scheme

We begin by analyzing an optimal restart scheme assuming the sharpness constant is known. We use a non-integer clock to highlight its dependency to the sharpness. Naturally clock and number of restarts must be integer but this does not affect much bounds as detailed in Appendix 4.A. The next proposition shows that algorithm 𝒜 needs a constant number of iterations to decrease the gap by a constant factor, which means restart leads to linear convergence. Proposition 4.1.4. Given a coding matrix 𝐴 ∈ R 𝑛×𝑝 and a signal 𝑥 * ∈ R 𝑝 such that the sharpness bound (Sharp) is satisfied with 𝛾 > 0, i.e.

‖𝑥‖ 1 -‖𝑥 * ‖ 1 > 𝛾‖𝑥 -𝑥 * ‖ 1 , for any 𝑥 ̸ = 𝑥 * such that 𝐴𝑥 = 𝐴𝑥 * , running the (Restart) scheme with 𝑡 ≥ 2 √ 𝑝 𝛾𝜌 ensures ‖𝑦 𝑘 ‖ 1 -‖𝑥 * ‖ 1 ≤ 𝜀 𝑘 , (4.3)
at each iteration, with 𝑥 * the unique solution of problem (ℓ 1 recovery). Using optimal parameters

𝜌 * = 𝑒 -1 𝑎𝑛𝑑 𝑡 * = 2𝑒 √ 𝑝 𝛾 , (4.4)
we get a point ŷ such that

‖ŷ‖ 1 -‖𝑥 * ‖ 1 ≤ exp ( - 𝛾 2 √ 𝑝 𝑒𝑁 ) 𝜀 0 . (4.5)
after running a total of 𝑁 inner iterations of Algorithm Restart with 𝑡 = 𝑡 * (hence 𝑁/𝑡 restarts).

Proof. By the choice of 𝜀 0 , (4.3) is satisfied for 𝑘 = 0. Assuming it holds at iteration 𝑘, combining sharpness bound (Sharp) and complexity bound (4.2) leads to, for 𝑥 = 𝒜(𝑦 𝑘-1 , 𝜀 𝑘 , 𝑡),

‖𝑥‖ 1 -‖𝑥 * ‖ 1 ≤ 2𝑝(‖𝑦 𝑘-1 ‖ 1 -‖𝑥 * ‖ 1 ) 2 𝛾 2 𝜀 𝑘 𝑡 2 + 𝜀 𝑘 2 ≤ 4𝑝 𝜌 2 𝛾 2 𝑡 2 𝜀 𝑘 2 + 𝜀 𝑘 2 .
Therefore after 𝑡 ≥ 2 √ 𝑝 𝛾𝜌 iterations, the method has achieved the decreased accuracy 𝜀 𝑘 which proves (4.3). The overall complexity after a total of 𝑁 inner iterations, hence 𝑁/𝑡 restarts, is then

‖ŷ‖ 1 -‖𝑥 * ‖ 1 ≤ 𝜌 𝑁/𝑡 𝜀 0 .
If 𝛾 is known, using exactly

2 √ 𝑝
𝛾𝜌 inner iterations at each restart leads to

‖ŷ‖ 1 -‖𝑥 * ‖ 1 ≤ exp ( 𝛾 2 √ 𝑝 𝑁 𝜌 log 𝜌 ) 𝜀 0 .
Optimizing in 𝜌 yields 𝜌 * = 𝑒 -1 , and with 𝑡 * inner iterations the complexity bound (4.5) follows.

To run NESTA, 𝐴 𝑇 𝐴 is assumed to be an orthogonal projector (often the case in compressed sensing applications) such that the projection on the feasible set is easy. [START_REF] Becker | Nesta: A fast and accurate first-order method for sparse recovery[END_REF] already studied restart schemes that they called "acceleration with continuation". However their restart criterion depends on the relative variation of objective values, not on the number of iterates, and no linear convergence was proven. We further note that linear convergence of restart schemes only requires an assumption of the form

𝑓 (𝑥) -𝑓 * ≥ 𝛾𝑑(𝑥, 𝑋 * ), (4.6)
where 𝑑(𝑥, 𝑋 * ) is the distance (in any norm) from 𝑥 to the set of minimizers of the objective function 𝑓 (here 𝑓 (𝑥) = ‖𝑥‖ 1 ). This type of bound is known as Łojasiewicz's inequality, studied for example in [START_REF] Bolte | The łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems[END_REF] for non-smooth convex functions.

Here (NSP) ensures that the set of minimizers is reduced to a singleton, the original signal.

Practical restart scheme

Several parameters are needed to run the optimal scheme above. The optimal decreasing factor is independent of the data. The initial gap 𝜀 0 can be taken as ‖𝑦 0 ‖ 1 for 𝐴𝑦 0 = 𝑏. The sharpness constant 𝛾 is for its part mostly unknown such that we cannot choose the number 𝑡 * of inner iterations a priori. However, given a budget of iterations 𝑁 , a log scale grid search can be performed on the optimal restart clock to get nearly optimal rates as detailed in the following corollary (contrary to the general results in [START_REF] Roulet | Sharpness, restart and acceleration[END_REF], the sharpness exponent is known here, simplifying the parameter search).

Corollary 4.1.5. Given a coding matrix 𝐴 ∈ R 𝑛×𝑝 , a signal 𝑥 * ∈ R 𝑝 such that the sharpness bound (Sharp) is satisfied with 𝛾 > 0, a budget of 𝑁 iterations, run the following schemes from an initial point 𝑦 0

(Restart) with 𝑡 = ℎ 𝑗 , 𝑗 = 1, . . . , ⌊log ℎ 𝑁 ⌋
with ℎ the grid search precision. Stop restart iteration when the total number of iterations has exceeded the budget 𝑁 . Then, provided that 𝑁 ≥ ℎ𝑡 * , where 𝑡 * is defined in (4.4), at least one of these restart schemes achieves a precision given by

‖ŷ‖ 1 -‖𝑥 * ‖ 1 ≤ exp ( - 𝛾 2ℎ √ 𝑝 𝑒𝑁 ) 𝜀 0 . (4.7)
Overall running the logarithmic grid search has a complexity log ℎ 𝑁 times higher than running 𝑁 iterations in the optimal scheme.

Proof. All schemes stop after at most 𝑁 + ℎ 𝑗 ≤ 2𝑁 iterations. As we assumed 𝑁 ≥ ℎ𝑡 * , 𝑗 = ⌈log ℎ 𝑡 * ⌉ ≤ log ℎ 𝑁 and (Restart) has been run with 𝑡 = ℎ 𝑗 . Proposi-tion 4.1.4 ensures, since 𝑡 ≥ 𝑡 * , that the output of this scheme achieves after 𝑁 ′ ≥ 𝑁 total iterations a precision ‖ŷ‖ 1 -‖x‖ 1 ≤ 𝑒 -𝑁 ′ /𝑡 𝜀 0 ≤ 𝑒 -𝑁/𝑡 𝜀 0 and as 𝑡 ≤ ℎ𝑡 * ‖ŷ‖ 1 -‖x‖ 1 ≤ 𝑒 -𝑁/(ℎ𝑡 * ) 𝜀 0 which gives the result. Finally the logarithmic grid search costs log ℎ 𝑁 to get this approximative optimal bound.

Sharpness therefore controls linear convergence of simple restart schemes to solve (ℓ 1 recovery). We now turn back to (NSP) estimates and connect them to recovery thresholds of the sampling matrix. This will give us a direct link between computational complexity and recovery performance for exact recovery problems.

Recovery threshold

If (NSP) is satisfied at a given order 𝑠 it holds also for any 𝑠 ′ ≤ 𝑠. However, the constant, and therefore the speed of convergence, may change. Here we show that this constant actually depends on the ratio between the maximal order at which 𝐴 satisfies (NSP) and the sparsity of the signal that we seek to recover.

To this end, we give a more concrete geometric meaning to the constant 𝛼 in (NSP), connecting it with the diameter of a section of the ℓ 1 ball by the null space of the matrix 𝐴 (see e.g. [START_REF] Kashin | A remark on compressed sensing[END_REF] for more details).

Lemma 4.1.6. Given a matrix 𝐴 ∈ R 𝑛×𝑝 , denote

1 2 diam(𝐵 𝑝 1 ∩ Null(𝐴)) = sup 𝐴𝑧=0 ‖𝑧‖ 1 ≤1 ‖𝑧‖ 2 ,
the radius of the section of the ℓ 1 ball 𝐵 𝑝 1 by the null space of the matrix 𝐴 and

𝑠 𝐴 1/ diam(𝐵 𝑝 1 ∩ Null(𝐴)) 2 , (4.8)
a recovery threshold. Then 𝐴 satisfies (NSP) at any order 𝑠 < 𝑠 𝐴 with constant

𝛼 = 2 √ 𝑠 𝐴 /𝑠 -1 > 1. (4.9)
Proof. For any 𝑧 ∈ Null(𝐴) and support set 𝑆 with Card(𝑆) ≤ 𝑠, using equivalence of norms and definition of the radius,

‖𝑧 𝑆 ‖ 1 ≤ √ 𝑠 ‖𝑧‖ 2 ≤ 1 2 √ 𝑠 𝑠 𝐴 ‖𝑧‖ 1 = 1 2 √ 𝑠 𝑠 𝐴 (‖𝑧 𝑆 ‖ 1 + ‖𝑧 𝑆 𝑐 ‖ 1 ),
which means, as 𝑠 < 𝑠 𝐴 ,

‖𝑧 𝑆 𝑐 ‖ 1 ≥ (2 √ 𝑠 𝐴 /𝑠 -1)‖𝑧 𝑆 ‖ 1 ,
hence the desired result.

With 𝑠 𝐴 defined in (4.8), for any signal 𝑥 * of sparsity 𝑠 < 𝑠 𝐴 , the sharpness bound (Sharp) then reads

‖𝑥‖ 1 -‖𝑥 * ‖ 1 ≥ ( 1 - √ 𝑠/𝑠 𝐴 ) ‖𝑥 -𝑥 * ‖ 1 ,
and the optimal restart scheme defined in Proposition 4.1.4 has complexity

‖ŷ‖ 1 -‖𝑥 * ‖ 1 ≤ exp ( - ( 1 - √ 𝑠/𝑠 𝐴 ) 𝑒 2 √ 𝑝 𝑁 ) 𝜀 0 ,
which means that, given a sensing matrix 𝐴 with recovery threshold 𝑠 𝐴 , the sparser the signal, the faster the algorithm. Precise estimates of the diameter of random sections of norm balls can be computed using classical results in geometric functional analysis. The low 𝑀 * estimates of Pajor and Tomczak-Jaegermann [1986] (see [Vershynin, 2011, Theorem 3.1] for a concise presentation) show that when 𝐸 ⊂ R 𝑝 is a random subspace of codimension 𝑛 (e.g. the null space of a random matrix 𝐴 ∈ R 𝑛×𝑝 ), then

diam(𝐵 𝑝 1 ∩ 𝐸) ≤ 𝑐 √ log 𝑝 𝑛 ,
with high probability, where 𝑐 > 0 is an absolute constant. This means that the recovery threshold 𝑠 𝐴 satisfies

𝑠 𝐴 ≥ 𝑛/(𝑐 2 log 𝑝),
with high probability and leads to the following corollary.

Corollary 4.1.7. Given a random sampling matrix 𝐴 ∈ R 𝑛×𝑝 and a signal 𝑥 * with sparsity 𝑠 < 𝑛/(𝑐 2 log 𝑝), (Restart) scheme with optimal parameters defined in (4.4) outputs a point ŷ such that

‖ŷ‖ 1 -‖𝑥 * ‖ 1 ≤ exp ( - ( 1 -𝑐 √ 𝑠 log 𝑝 𝑛 ) 𝑒 2 √ 𝑝 𝑁 ) 𝜀 0 ,
with high probability, where 𝑐 is a universal constant and 𝑁 is the total number of iterations.

A conic view for sparse recovery problems

We first gave concrete evidence of the link between optimization complexity and recovery performance for the exact recovery problem by highlighting sharpness properties of the objective around the true signal, given by the null space condition. We now take a step back and consider results on the underlying conic geometry of recovery problems that also control both computational and statistical aspects.

On the statistical side, minimal conically restricted singular values are known to control recovery performance in robust recovery problems. On the computational side, Renegar's condition number, a well known computational complexity measure for conic convex programs, controls the cost of obtaining optimality certificates for exact recovery and the sharpness of exact recovery problems (hence computational complexity of the (Restart) scheme presented in the previous section). Numerical experiments will then illustrate its relevance to control numerous other classical algorithms. By observing that minimal conically restricted singular values match the worst case of Renegar's condition number on sparse signals, our analysis shows once more that one single geometrical quantity controls both statistical robustness and computational complexity of recovery problems.

Conic linear systems

Conic linear systems arise naturally from optimality conditions of the exact recovery problem. To see this, define the tangent cone at point 𝑥 with respect to the ℓ 1 norm, that is, the set of descent directions for ‖ • ‖ 1 at 𝑥, as

𝒯 (𝑥) = cone{𝑧 : ‖𝑥 + 𝑧‖ 1 ≤ ‖𝑥‖ 1 }.
(4.10)

As shown for example by Chandrasekaran et al. [2012, Prop 2.1] a point 𝑥 is then the unique optimum of the exact recovery problem (ℓ 1 recovery) if and only if Null(𝐴) ∩ 𝒯 (𝑥) = {0}, that is, there is no point satisfying the linear constraints that has lower ℓ 1 norm than 𝑥. Correct recovery of an original signal 𝑥 * is therefore certified by the infeasibility of a conic linear system of the form find 𝑧 s.t. 𝐴𝑧 = 0 𝑧 ∈ 𝐶, 𝑧 ̸ = 0,

(P A,C )
where 𝐶 is a closed convex cone and 𝐴 a given matrix. For both computational and statistical aspects we will be interested in the distance to feasibility. On the computational side this will give a distance to ill-posedness that plays the role of a condition number. On the statistical side it will measure the amount of perturbation that the recovery can handle.

Definition 4.2.1 (Distance to feasibility). Writing ℳ 𝐶 = {𝐴 ∈ R 𝑛×𝑝 : (P A,C ) is infeasible}, distance to feasibility is defined as

𝜎 𝐶 (𝐴) inf Δ𝐴 {‖∆𝐴‖ 2 : 𝐴 + ∆𝐴 / ∈ ℳ 𝐶 }. (4.11)
A geometric analysis of the problem explicits the distance to feasibility in terms of minimal conically restricted singular value, as recalled in the following lemma.

Lemma 4.2.2. Given a matrix 𝐴 ∈ R 𝑝×𝑛 and a closed convex cone 𝐶, the distance to feasibility of (P A,C ) is given by

𝜎 𝐶 (𝐴) = min 𝑥∈𝐶 ‖𝑥‖ 2 =1 ‖𝐴𝑥‖ 2 .
(4.12)

Proof. We recall the short proof of Amelunxen and Lotz [2014, Lemma 3.2]. Similar results have been derived by Freund and Vera [1999b, Theorem 2] and Belloni and Freund [2009, Lemma 3.2]. Let 𝑧 ∈ 𝐶, with ‖𝑧‖ 2 = 1, achieve the minimum above. Then ∆𝐴 = -𝐴𝑧𝑧 𝑇 satisfies (𝐴 + ∆𝐴)𝑧 = 0, so 𝐴 + ∆𝐴 / ∈ ℳ 𝐶 and

𝜎 𝐶 (𝐴) ≤ ‖∆𝐴‖ 2 = ‖𝐴𝑧‖ 2 ‖𝑧‖ 2 = min 𝑥∈𝐶 ‖𝑥‖ 2 =1 ‖𝐴𝑥‖ 2 .
On the other hand denote ∆𝐴 a perturbation such that 𝐴 + ∆𝐴 / ∈ ℳ 𝐶 . Then there exists 𝑧 ∈ 𝐶 ∖ {0} such that (𝐴 + ∆𝐴)𝑧 = 0. Thus we have

‖∆𝐴‖ 2 ≥ ‖∆𝐴𝑧‖ 2 ‖𝑧‖ 2 = ‖𝐴𝑧‖ 2 ‖𝑧‖ 2 ≥ min 𝑥∈𝐶 ‖𝑥‖ 2 =1 ‖𝐴𝑥‖ 2 .
Taking the infimum on the left-hand side over all ∆𝐴 such that 𝐴 + ∆𝐴 / ∈ ℳ 𝐶 concludes the proof. Expression (4.12) writes distance to infeasibility as a cone restricted eigenvalue. Minimal cone restricted eigenvalues also directly characterize recovery performance as we recall now.

Recovery performance of robust recovery

The previous section showed that recovery of a signal 𝑥 * is ensured by infeasibility of the conic linear system (P A,𝒯 (𝑥 * ) ), i.e. positiveness of the minimal conically restricted singular value 𝜎 𝒯 (𝑥 * ) (𝐴). We now show how this quantity also controls recovery performance in the presence of noise. In that case, the robust recovery problem attempts to retrieve an original signal 𝑥 * by solving Proof. We recall the short proof of Chandrasekaran et al. [2012, Prop. 2.2]. Both x and 𝑥 * are feasible for (Robust ℓ 1 recovery) and x is optimal, so that ‖x‖ 1 ≤ ‖𝑥 * ‖ 1 . Thus, the error vector x -𝑥 * is in the tangent cone 𝒯 (𝑥 * ). By the triangle inequality,

‖𝐴(x -𝑥 * )‖ 2 ≤ ‖𝐴x -𝑏‖ 2 + ‖𝐴𝑥 * -𝑏‖ 2 ≤ 2𝛿‖𝐴‖ 2 .
Furthermore, by definition of 𝜎 𝒯 (𝑥 * ) (𝐴),

‖𝐴(x -𝑥 * )‖ 2 ≥ 𝜎 𝒯 (𝑥 * ) (𝐴) ‖x -𝑥 * ‖ 2 .
Combining the two concludes the proof.

Therefore the robustness of the coding matrix 𝐴 on all 𝑠-sparse signals is measured by 𝜇 𝑠 (𝐴) inf

𝑥 : ‖𝑥‖ 0 ≤𝑠 min 𝑧∈𝒯 (𝑥) ‖𝑧‖ 2 =1 ‖𝐴𝑧‖ 2 . (4.14)
Expression of this minimal conically singular value can the be simplified by identifying the tangent cones on 𝑠-sparse signals, as done in the following lemma.

Lemma 4.2.4. For any subset 𝑆 ⊂ 1, 𝑝 , let

ℰ 𝑆 = {𝑧 : ‖𝑧 𝑆 𝑐 ‖ 1 ≤ ‖𝑧 𝑆 ‖} and ℱ 𝑆 = ⋃ 𝑥 : 𝑥=𝑥 𝑆 𝒯 (𝑥), then ℰ 𝑆 = ℱ 𝑆 .
Proof. Let 𝑧 ∈ ℰ 𝑆 , take 𝑥 = -𝑧 𝑆 , then

‖𝑥 + 𝑧‖ 1 = ‖𝑧 𝑆 𝑐 ‖ 1 ≤ ‖𝑧 𝑆 ‖ 1 = ‖𝑥‖ 1 .
Therefore 𝑧 ∈ 𝒯 (𝑥) ⊂ ℱ 𝑆 as 𝑥 = 𝑥 𝑆 .

Conversely let 𝑧 ∈ ℱ 𝑆 , and 𝑥 ∈ R 𝑝 , with 𝑥 = 𝑥 𝑆 , such that 𝑧 ∈ 𝒯 (𝑥). Then

‖𝑥 + 𝑧‖ 1 = ‖𝑥 + 𝑧 𝑆 ‖ 1 + ‖𝑧 𝑆 𝑐 ‖ 1 ≥ ‖𝑥‖ 1 -‖𝑧 𝑆 ‖ 1 + ‖𝑧 𝑆 𝑐 ‖ 1 .
As 𝑧 ∈ 𝒯 (𝑥), this implies ‖𝑧 𝑆 𝑐 ‖ 1 ≤ ‖𝑧 𝑆 ‖ 1 , so 𝑧 ∈ ℰ 𝑆 and we conclude that ℰ 𝑆 = ℱ 𝑆 .

Therefore, the previous expression for the minimal conically restricted singular value (4.14) can be equivalently stated as

𝜇 𝑠 (𝐴) = min 𝑆⊂ 1,𝑝 Card(𝑆)≤𝑠 min ‖𝑧 𝑆 𝑐 ‖ 1 ≤‖𝑧 𝑆 ‖ 1 ‖𝑧‖ 2 =1 ‖𝐴𝑧‖ 2 . (4.15)
This quantity was introduced in [START_REF] Bickel | Simultaneous analysis of lasso and dantzig selector[END_REF] and further explored in e.g. [START_REF] Van De Geer | On the conditions used to prove oracle results for the lasso[END_REF]. [START_REF] Bickel | Simultaneous analysis of lasso and dantzig selector[END_REF] notably showed that it controls estimation performance of LASSO and Dantzig selector. Observe that positiveness of 𝜇 𝑠 (𝐴) is equivalent to (NSP) at order 𝑠 with constant 1 which shows necessity of (NSP) for sparse recovery.

Since both null space property and conically restricted singular values are necessary and sufficient conditions for exact recovery they may have been linked previously in the literature. Here we derive estimates for the constant in (NSP) from the minimal cone restricted singular value using tools form conic linear systems. We search for 𝛼 such that (NSP) is satisfied at order 𝑠. Equivalently we search for 𝛼 such that for any support 𝑆 of cardinality at most 𝑠, the conic linear system

find 𝑧 s.t. 𝐴𝑧 = 0 ‖𝑧 𝑆 𝑐 ‖ 1 ≤ 𝛼‖𝑧 𝑆 ‖ 1 , 𝑧 ̸ = 0 (4.16)
is infeasible. Notice that system (4.16) for 𝛼 > 1 is a perturbed version of the case 𝛼 = 1, so the problem reduces to studying the sensitivity to perturbations of conic linear systems as shown in the following lemma.

Lemma 4.2.5. Given a matrix 𝐴 ∈ R 𝑛×𝑝 and an integer 𝑠 ∈ 1, 𝑝 , if the minimal conically restricted singular value 𝜇 𝑠 (𝐴) in (4.14) and (4.15) is positive, then 𝐴 satisfies (NSP) at order 𝑠 for any constant

𝛼 ≤ ( 1 - 𝜇 𝑠 (𝐴) ‖𝐴‖ 2 ) -1 .
Proof. For a support 𝑆 of cardinality at most 𝑠, write 𝑃 the orthogonal projector on this support (that is, 𝑃 𝑥 = 𝑥 𝑆 ), P = I -𝑃 its orthogonal projector and define the closed convex cone 𝐶 𝑆 = {𝑧 :

‖𝑧 𝑆 𝑐 ‖ 1 ≤ ‖𝑧 𝑆 ‖ 1 }. Given 𝛼 ≥ 1, denote 𝐻 = 𝛼 -1 𝑃 + P = I -(1 -𝛼 -1 )𝑃 . Observe that {𝑧 : ‖𝑧 𝑆 𝑐 ‖ 1 ≤ 𝛼‖𝑧 𝑆 ‖ 1 } = 𝐻𝐶 𝑆 .
Therefore, the conic linear system (4.16) reads

find 𝑧 s.t. 𝐴𝑧 = 0 𝑧 ∈ 𝐻𝐶 𝑆 , 𝑧 ̸ = 0.
As 𝐻 is invertible, this is equivalent to find 𝑧 s.t. 𝐴𝐻𝑧 = 0 𝑧 ∈ 𝐶 𝑆 , 𝑧 ̸ = 0.

(4.17)

Therefore, if the conic linear system

find 𝑧 s.t. 𝐴𝑧 = 0 𝑧 ∈ 𝐶 𝑆 , 𝑧 ̸ = 0
is infeasible, that is 𝜎 𝐶 𝑆 (𝐴) > 0, by Lemma 4.2.2, which is true for 𝜇 𝑠 > 0, then by definition of the distance to feasibility, (4.17) is also infeasible provided ‖𝐴𝐻 -𝐴‖ 2 ≤ 𝜎 𝐶 𝑆 (𝐴), which holds for any 𝛼 ≥ 1 such that

(1 -𝛼 -1 )‖𝐴𝑃 ‖ 2 ≤ 𝜎 𝐶 𝑆 (𝐴).
Using that ‖𝐴𝑃 ‖ 2 ≤ ‖𝐴‖ 2 , infeasibility is ensured in particular for any 𝛼 such that

1 - 𝜎 𝐶 𝑆 (𝐴) ‖𝐴‖ 2 ≤ 𝛼 -1 .
To ensure infeasibility of the conic linear systems (4.16) for any support 𝑆, it suffices to take 𝛼 such that 1 -

𝜇 𝑠 (𝐴) ‖𝐴‖ 2 ≤ 𝛼 -1 .
This means that (NSP) at order 𝑠 is satisfied for any

𝛼 ≤ ( 1 - 𝜇 𝑠 (𝐴) ‖𝐴‖ 2 ) -1
where we used that, by definition of the minimal conically restricted singular value, 𝜇 𝑠 (𝐴) ≤ ‖𝐴‖ 2 (in case of equality (NSP), will be satisfied for any 𝛼 ≥ 1).

We now relate the minimal cone restricted singular value to computational complexity measures.

Computational complexity of recovery problems

Computational complexity for convex optimization problems is often described in terms of polynomial functions of problem size. This produces a clear link between problem structure and computational complexity but fails to account for the nature of the data. If we use linear systems as a basic example, unstructured linear systems of dimension 𝑛 can be solved with complexity 𝑂(𝑛 3 ) regardless of the matrix values, but iterative solvers will converge much faster on systems that are better conditioned. The seminal work of Renegar [1995b[START_REF] Renegar | A mathematical view of interior-point methods in convex optimization[END_REF] extends this notion of conditioning to optimization problems, producing data-driven bounds on the complexity of solving conic programs, and showing that the number of outer iterations of interior point algorithms increases as the distance to ill-posedness decreases.

Renegar's condition number

Renegar's condition number [Renegar, 1995b,a;[START_REF] Peña | Understanding the geometry of infeasible perturbations of a conic linear system[END_REF] provides a data-driven measure of the complexity of certifying infeasibility of a conic linear system of the form presented in (P A,C ) (the larger the condition number, the harder the problem). It is rooted in the sensible idea that certifying infeasibility is easier if the problem is far from being feasible. It is defined as the scale invariant reciprocal of the distance to feasibility 𝜎 𝐶 (𝐴), defined in (4.11), of problem (P A,C ), i.e.

ℛ 𝐶 (𝐴) ‖𝐴‖ 2 𝜎 𝐶 (𝐴) = ‖𝐴‖ 2 / min 𝑥∈𝐶 ‖𝑥‖ 2 =1 ‖𝐴𝑥‖ 2 .
(4.18)

Notice that, if 𝐶 were the whole space R 𝑝 , and if 𝐴 𝑇 𝐴 were full-rank (never the case if 𝑛 < 𝑝), then 𝜎 𝐶 (𝐴) would be the smallest singular value of 𝐴. As a result, ℛ 𝐶 (𝐴) would reduce to the classical condition number of 𝐴 (and to ∞ when 𝐴 𝑇 𝐴 is rankdeficient). Renegar's condition number is necessarily smaller (better) than the latter, as it further incorporates the notion that 𝐴 need only be well conditioned along those directions that matter with respect to 𝐶.

Complexity of certifying optimality

In a first step, we study the complexity of the oracle certifying optimality of a candidate solution 𝑥 to (ℓ 1 recovery) as a proxy for the problem of computing an optimal solution to this problem. As mentioned in Section 4.2.1, optimality of a point 𝑥 is equivalent to infeasibility of find 𝑧 s.t. 𝐴𝑧 = 0 𝑧 ∈ 𝒯 (𝑥), 𝑧 ̸ = 0,

(P A,𝒯 (𝑥) )
where the tangent cone 𝒯 (𝑥) is defined in (4.10). By a theorem of alternative, infeasibility of (P A,𝒯 (𝑥) ) is equivalent to feasibility of the dual problem

find 𝑦 s.t. 𝐴 𝑇 𝑦 ∈ int(𝒯 (𝑥) ∘ ), (D A,𝒯 (𝑥) )
where 𝒯 (𝑥) ∘ is the polar cone of 𝒯 (𝑥). Therefore, to certify infeasibility of (P A,𝒯 (𝑥) ) it is sufficient to exhibit a solution for the dual problem (D A,𝒯 (𝑥) ).

Several references have connected Renegar's condition number and the complexity of solving such conic linear systems using various algorithms [Renegar, 1995b;Freund and Vera, 1999a;[START_REF] Epelman | Condition number complexity of an elementary algorithm for computing a reliable solution of a conic linear system[END_REF][START_REF] Renegar | A mathematical view of interior-point methods in convex optimization[END_REF][START_REF] Vera | A primal-dual symmetric relaxation for homogeneous conic systems[END_REF]Belloni et al., 2009]. In particular, [START_REF] Vera | A primal-dual symmetric relaxation for homogeneous conic systems[END_REF] linked it to the complexity of solving the primal dual pair (P A,𝒯 (𝑥) )-(D A,𝒯 (𝑥) ) using a barrier method. They show that the number of outer barrier method iterations grows as

𝑂 ( √ 𝜌 log ( 𝜌 ℛ 𝒯 (𝑥) (𝐴) )) ,
where 𝜌 is the barrier parameter, while the conditioning (hence the complexity) of the linear systems arising at each interior point iteration is controlled by ℛ 𝒯 (𝑥) (𝐴) 2 . This link was also tested empirically on linear programs using the NETLIB library of problems by [START_REF] Ordóñez | Computational experience and the explanatory value of condition measures for linear optimization[END_REF], where computing times and number of iterations were regressed against estimates of the condition number computed using the approximations for Renegar's condition number detailed by [START_REF] Freund | On the complexity of computing estimates of condition measures of a conic linear system[END_REF].

Studying the complexity of computing an optimality certificate gives insights on the performance of oracle based optimization techniques such as the ellipsoid method. We now show how Renegar's condition also controls the number steps in the (Restart) scheme presented in Section 4.1.2.

Complexity of restart scheme with Renegar's condition number

Convergence of the (Restart) scheme presented in Section 4.1.2 is controlled by the sharpness of the problem deduced from (NSP). We now observe that sharpness is controlled by the worst case Renegar condition number for the optimality certificates (P A,𝒯 (𝑥) ) on all 𝑠-sparse signals, defined as Proof. From Lemma 4.2.3, if ℛ 𝑠 (𝐴) < +∞, then (NSP) is satisfied for 𝛼 = (1 -ℛ 𝑠 (𝐴) -1 ) -1 > 1, so from Lemma 4.2.5 (4.1) is sharp with constant 𝛾 = 𝛼-1 𝛼+1 = (2ℛ 𝑠 (𝐴) -1) -1 and Proposition 4.1.4 leads to the linear convergence rate above. This shows that Renegar's condition number explicitly controls the convergence of an algorithmic scheme devoted to the exact recovery problem (ℓ 1 recovery), through its link with sharpness.

ℛ 𝑠 (𝐴) sup 𝑥 : ‖𝑥‖ 0 ≤𝑠 ℛ 𝒯 (𝑥) (𝐴) = ‖𝐴‖ 2 /
On the statistical side, we observed that the minimal conically restricted singular value controls recovery performance of robust procedures and that its positivity ensures exact recovery. On the computational side, we presented the role of Renegar's condition number as a computational complexity measure for sparse recovery problems. A key observation is that the worst case of Renegar's condition number ℛ 𝑠 (𝐴), defined in (4.19), matches the minimal conically restricted singular value defined in (4.14). Once again, a single quantity controls both aspects. This at least partially explains the common empirical observation (see, e.g., [START_REF] Donoho | Fast solution of ℓ 1 -norm minimization problems when the solution may be sparse-norm minimization problems when the solution may be sparse[END_REF]) that problem instances where statistical estimation succeeds are computationally easier to solve.

Computational complexity for inexact recovery

When the primal problem (P A,𝒯 (𝑥) ) is feasible, so that 𝜎 𝒯 (𝑥) (𝐴) = 0, Renegar's condition number as defined here is infinite. While this correctly captures the fact that, in that regime, statistical recovery does not hold, it does not properly capture the fact that, when (P A,𝒯 (𝑥) ) is "comfortably" feasible, certifying so is easy, and algorithms terminate quickly (although they return a useless estimator). From both a statistical and a computational point of view, the truly delicate cases correspond to problem instances for which both (P A,𝒯 (𝑥) ) and (D A,𝒯 (𝑥) ) are only barely feasible or infeasible. This is illustrated in simple numerical example by Boyd and Vandenberghe [2004, §11.4.3] and in our numerical experiments, corresponding to the peaks in the CPU time plots of the right column in Figure 4-4: problems where sparse recovery barely holds/fails are relatively harder. For simplicity, we only focused here on distance to feasibility for problem (P A,𝒯 (𝑥) ). However, it is possible to symmetrize the condition numbers used here as described by Amelunxen and Lotz [2014, §1.3], where a symmetric version of the condition number is defined as

R𝒯 (𝑥) (𝐴) = min { ‖𝐴‖ 𝜎 𝑃 𝒯 (𝑥) (𝐴) , ‖𝐴‖ 𝜎 𝐷 𝒯 (𝑥) (𝐴) } ,
where 𝜎 𝑃 𝒯 (𝑥) (𝐴) and 𝜎 𝐷 𝒯 (𝑥) (𝐴) denote the distance to feasibility of respectively (P A,𝒯 (𝑥) ) and (D A,𝒯 (𝑥) ). This quantity peaks for programs that are nearly feasible/infeasible.

As we noticed in Section 4.1.2, a Łojasiewicz inequality (4.6) for the (ℓ 1 recovery) problem is sufficient to ensure linear convergence of the restart scheme. Connecting the symmetrized Renegar condition number to the Łojasiewicz inequality constant 𝛾 may then produce complexity bounds for the restart scheme beyond the recovery case. Łojasiewicz inequalities for convex programs have indeed proven their relevance. They were used by [START_REF] Fercoq | Restarting accelerated gradient methods with a rough strong convexity estimate[END_REF]; [START_REF] Roulet | Sharpness, restart and acceleration[END_REF] to accelerate classical methods, in particular on the LASSO problem. Lower computational bounds for convex optimization problems satisfying it were also studied by Nemirovskii and Nesterov [1985, Page 6]. Although the Łojasiewicz inequality is proven to be satisfied by a broad class of functions [START_REF] Bolte | The łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems[END_REF], quantifying its parameters is still a challenging problem that would enable better parameter choices for appropriate algorithms.

Other algorithms

The restart scheme presented in Section 4.1.2 is of course not the only one to solve problem (ℓ 1 recovery) in practice and it has not been analyzed in the noisy case. However, we will observe in the numerical experiments of Section 4.4 that the condition number is correlated with the empirical performance of efficient recovery algorithms such as LARS [START_REF] Efron | Least angle regression[END_REF] and Homotopy [START_REF] Donoho | Fast solution of ℓ 1 -norm minimization problems when the solution may be sparse-norm minimization problems when the solution may be sparse[END_REF][START_REF] Asif | Sparse recovery of streaming signals using l1homotopy[END_REF]. On paper, the computational complexities of (ℓ 1 recovery) and (Robust ℓ 1 recovery) are very similar (in fact, infeasible start primal-dual algorithms designed for solving (ℓ 1 recovery) actually solve problem (Robust ℓ 1 recovery) with 𝛿 small). However in our experiments, we did observe sometimes significant differences in behavior between the noisy and noiseless case.

Generalization to common sparsity inducing norms

In this section we generalize previous results to sparse recovery problems in (nonoverlapping) group norms or nuclear norm. Group norms arise in contexts such as genomics to enforce the selection of groups of genes (e.g., [START_REF] Obozinski | Group lasso with overlaps: the latent group lasso approach[END_REF] and references therein.) The nuclear norm is used for low-rank estimation (e.g., [START_REF] Recht | Necessary and sufficient conditions for success of the nuclear norm heuristic for rank minimization[END_REF] and references therein.) We use the framework of decomposable norms introduced by [START_REF] Negahban | A unified framework for high-dimensional analysis of 𝑚-estimators with decomposable regularizers[END_REF] which applies to these norms. This allows us to generalize the null space property and to derive corresponding sharpness bounds for the exact recovery problem in a broader framework. We then again relate recovery performance and computational complexity of these recovery problems.

Decomposable norms

Sparsity inducing norms have been explored from various perspectives. Here, we use the framework of decomposable norms by [START_REF] Negahban | A unified framework for high-dimensional analysis of 𝑚-estimators with decomposable regularizers[END_REF] to generalize our results from ℓ 1 norms to non-overlapping group norms and nuclear norms in a concise form. We then discuss the key geometrical properties of these norms and potential characterization of their conic nature.

We first recall the definition of decomposable norms by [START_REF] Negahban | A unified framework for high-dimensional analysis of 𝑚-estimators with decomposable regularizers[END_REF] in terms of projectors.

Definition 4.3.1. Decomposable norms Given a Euclidean space 𝐸, a norm ‖.‖ on 𝐸 is said to be decomposable if there exists a family of orthogonal projectors 𝒫 such that (i) to each 𝑃 ∈ 𝒫 is associated a non-negative weight 𝜂(𝑃 ) and an orthogonal projector P such that 𝑃 P = P 𝑃 = 0, and (ii) for any 𝑥 ∈ 𝐸 and 𝑃 ∈ 𝒫, ‖𝑃 𝑥 + P 𝑥‖ = ‖𝑃 𝑥‖ + ‖ P 𝑥‖ .

A signal 𝑥 is then said to be 𝑠-sparse if there exists 𝑃 ∈ 𝒫, such that 𝜂(𝑃 ) ≤ 𝑠 and 𝑃 𝑥 = 𝑥.

We now detail the family of projectors for some decomposable norms of interest.

ℓ 1 norm.

In the the ℓ 1 norm case, 𝐸 = R 𝑝 and 𝒫 is the set of projectors on coordinate subspaces of R 𝑝 , that is, 𝒫 contains all projectors which zero out all coordinates of a vector except for a subset of them, which are left unaffected. The maps P are the complementary projectors: P = I-𝑃 . Property (ii) is the classical decomposability of the ℓ 1 norm. Naturally, the complexity level corresponds to the number of coordinates preserved by 𝑃 , i.e., 𝜈(𝑃 ) = Rank(𝑃 ). These definitions recover the usual notion of sparsity.

Group norms.

Given a partition 𝐺 of 1, 𝑝 in (non-overlapping) groups 𝑔 ⊂ 1, 𝑝 , the group norm is defined for

𝑥 ∈ R 𝑝 as ‖𝑥‖ = ∑ 𝑔∈𝐺 ‖𝑥 𝑔 ‖ 𝑟 ,
where ‖𝑥 𝑔 ‖ 𝑟 is the ℓ 𝑟 -norm of the projection of 𝑥 onto the coordinates defined by 𝑔.

The cases 𝑟 = 2, ∞ correspond respectively to ℓ 1 /ℓ 2 and ℓ 1 /ℓ ∞ block norms. Here, 𝐸 = R 𝑝 and the family 𝒫 is composed of orthogonal projectors onto coordinates defined by (disjoint) unions of groups 𝑔, and P = I -𝑃 . Formally, to each 𝑃 we associate 𝐹 ⊂ 𝐺 such that for any 𝑥 ∈ 𝐸, (𝑃 𝑥) 𝑔 = 𝑥 𝑔 if 𝑔 ∈ 𝐹 and (𝑃 𝑥) 𝑔 = 0 otherwise. Decomposability (ii) then clearly holds. To each group 𝑔 we associate a weight 𝜂 𝑔 and for a projector 𝑃 ∈ 𝒫 with associated 𝐹 ⊂ 𝐺, 𝜂(𝑃 ) = ∑ 𝑔∈𝐹 𝜂 𝑔 . A classical choice of weights is 𝜂 𝑔 = 1 for all 𝑔 ∈ 𝐺.

Nuclear norm.

The nuclear norm is defined for matrices 𝑋 ∈ R 𝑝×𝑞 with singular values 𝜎 𝑖 (𝑋) as

‖𝑋‖ = min(𝑝,𝑞) ∑ 𝑘=1 𝜎 𝑘 (𝑋).
Here 𝐸 = R 𝑝×𝑞 and its associated family of projectors contains 𝑃 such that

𝑃 : 𝑋 ↦ → 𝑃 left 𝑋𝑃 right , and
P : 𝑋 ↦ → (I -𝑃 left )𝑋(I -𝑃 right ),
where 𝑃 left ∈ R 𝑝×𝑝 and 𝑃 right ∈ R 𝑞×𝑞 are orthogonal projectors. Their weights are defined as 𝜂(𝑃 ) = max (Rank(𝑃 left ), Rank(𝑃 right )) defining therefore 𝑠-sparse matrices as matrices of rank at most 𝑠. As 𝑃 and P project on orthogonal row and column spaces, condition (ii) holds. Decomposable norms offer a unified nomenclature for the study of sparsity inducing norms. However, they appear to be essentially restricted to the three examples presented above. Moreover, it is not clear if their definition is sufficient to character-ize the conic nature of these norms, in particular in the nuclear norm case that will require additional linear algebra results. In comparison, the framework proposed by [START_REF] Juditsky | On a unified view of nullspacetype conditions for recoveries associated with general sparsity structures[END_REF] can encompass non-latent overlapping groups. For future use, we simplify the third property of their definition [Juditsky et al., 2014, Section 2.1] in Appendix 4.B. It is not clear how this view can be used for latent overlapping group norms presented by [START_REF] Obozinski | Group lasso with overlaps: the latent group lasso approach[END_REF] applied in biology. Moreover the sufficient conditions that [START_REF] Juditsky | On a unified view of nullspacetype conditions for recoveries associated with general sparsity structures[END_REF] present are sufficient but not necessary in the nuclear norm case. Better characterizing the key geometrical properties of these norms is therefore a challenging research direction.

Sharpness and generalized null space property

From now on, we assume that we are given an ambient Euclidean space 𝐸 with one of the three decomposable norms ‖.‖ presented in previous section, i.e. ℓ 1 , group or nuclear norm, and the associated family of orthogonal projectors 𝒫 as introduced in Definition 4.3.1. We study the sparse recovery problem minimize ‖𝑥‖ subject to 𝐴(𝑥) = 𝑏 (Sparse recovery)

in the variable 𝑥 ∈ 𝐸, where 𝐴 is a linear operator onto R 𝑛 and the observations 𝑏 ∈ R 𝑛 are taken from an original point 𝑥 * such that 𝑏 = 𝐴(𝑥 * ). We begin by generalizing the null space property in this setting. The linear operator 𝐴 satisfies the Generalized Null Space Property at order 𝑠 with constant 𝛼 ≥ 1 if it satisfies it for any 𝑃 such that 𝜂(𝑃 ) ≤ 𝑠.

Notice that if P = I -𝑃 , the condition 𝑧 = 𝑃 𝑧 + P 𝑧 is not restrictive. However it will be useful to prove necessity of (GNSP) for the nuclear norm. In that case, observe that it is equivalent to the condition introduced by Oymak and Hassibi [2010], i.e.

∀𝑧 ∈ Null(𝐴) ∖ {0}, 𝛼 𝑠 ∑ 𝑖=1 𝜎 𝑖 (𝑧) < min(𝑝,𝑞) ∑ 𝑖=𝑠+1 𝜎 𝑖 (𝑧),
where 𝜎 𝑖 (𝑧) are the singular values of 𝑧 in decreasing order. Notice also that we recover the classical Definition NSP in the ℓ 1 case. The sharpness bound then easily follows if P = I -𝑃 . In the case of the nuclear norm it requires additional linear algebra results.

Proposition 4.3.3. Given a linear operator 𝐴 that satisfies (GNSP) at order 𝑠 with constant 𝛼, if the original point 𝑥 * is 𝑠-sparse, then for any 𝑥 ∈ 𝐸 satisfying 𝐴(𝑥) = 𝑏, 𝑥 ̸ = 𝑥 * , we have

‖𝑥‖ -‖𝑥 * ‖ > 𝛼 -1 𝛼 + 1 ‖𝑥 -𝑥 * ‖.
This implies recovery, i.e., optimality of 𝑥 * for (Sparse recovery).

Proof. Denote 𝑃 such that 𝜂(𝑃 ) ≤ 𝑠 and 𝑃 𝑥 * = 𝑥 * , which defines its sparsity. Let 𝑥 ̸ = 𝑥 * such that 𝐴(𝑥) = 𝑏, so 𝑧 = 𝑥 -𝑥 * ∈ Null(𝐴) and 𝑧 ̸ = 0. If P = I -𝑃 , 𝑥 = 𝑃 𝑥 + P 𝑥 and using the decomposability (ii), we have

‖𝑥‖ = ‖𝑃 𝑥 * + 𝑃 𝑧‖ + ‖ P 𝑧‖ ≥ ‖𝑥 * ‖ -‖𝑃 𝑧‖ + ‖ P 𝑧‖ = ‖𝑥 * ‖ + ‖𝑧‖ -2‖𝑃 𝑧‖.
By using (GNSP), ‖𝑧‖ = ‖𝑃 𝑧‖+‖ P 𝑧‖ > (1+𝛼)‖𝑃 𝑧‖. The result follows by arranging the terms. If ‖.‖ is the nuclear norm and P ̸ = I -𝑃 , as in [START_REF] Oymak | New null space results and recovery thresholds for matrix rank minimization[END_REF], Lemma 6], we use that (see Horn and Johnson [1990, Theorem 7.4.9.1])

‖𝑥 * + 𝑧‖ ≥ min(𝑝,𝑞) ∑ 𝑖=1 |𝜎 𝑖 (𝑥 * ) -𝜎 𝑖 (𝑧)|,
where 𝜎 𝑖 (𝑥 * ), 𝜎 𝑖 (𝑧) denote the singular values in decreasing order of respectively 𝑥 * and 𝑧. Then, using that 𝑥 * has rank at most 𝑠,

‖𝑥‖ ≥ 𝑠 ∑ 𝑖=1 |𝜎 𝑖 (𝑥 * ) -𝜎 𝑖 (𝑧)| + min(𝑝,𝑞) ∑ 𝑖=𝑠+1 𝜎 𝑖 (𝑧) ≥ 𝑠 ∑ 𝑖=1 𝜎 𝑖 (𝑥 * ) - 𝑠 ∑ 𝑖=1 𝜎 𝑖 (𝑧) + min(𝑝,𝑞) ∑ 𝑖=𝑠+1 𝜎 𝑖 (𝑧) = ‖𝑥 * ‖ -‖𝑄𝑧‖ + ‖ Q𝑧‖,
where 𝑄 is the projector on the 𝑠 largest singular directions of 𝑧 and therefore Q the projector on the 𝑛 -𝑠 others. These can be defined using the singular value decomposition of 𝑧 such that 𝑧 = 𝑄𝑧 + Q𝑧. Then, using (GNSP) and the decomposability (ii) concludes the proof as above.

This shows that the sharpness bound of the form (Sharp) generalizes to nonoverlapping group norms and the nuclear norm. Proposition 4.1.3 can also be generalized directly to this case with our definition of (GNSP). The smoothing argument and restart schemes developed in Section 4.1.2 can then be applied with similar linear convergence rates that essentially depend on the sharpness constant. By looking at the diameter of the section of the unit ball of the norm by the null space of 𝐴, one may also show that the oversampling ratio controls the sharpness bound as in Section 4.1.3.

As in Section 4.2, we now study the conic quantities which control statistical and optimization aspects.

Robust recovery performance and computational complexity

In this section, for a Euclidean space 𝐸 and 𝑥 ∈ 𝐸 we denote ‖𝑥‖ 2 the ℓ 2 norm of its coefficients, if 𝐸 is a matrix space ‖𝑥‖ 2 is then the Frobenius norm of 𝑥.

Generalized cone restricted singular value

We begin by addressing the recovery performance of robust sparse recovery problems that reads minimize ‖𝑥‖ subject to ‖𝐴(𝑥) -𝑏‖ 2 ≤ 𝛿‖𝐴‖ 2 , (Robust sparse recovery) in the variable 𝑥 ∈ 𝐸, with the same linear operator 𝐴, where the observations 𝑏 ∈ R 𝑛 are affected by noise of level 𝛿 > 0. For a linear operator 𝐴 from 𝐸 to R 𝑛 , we denote its operator norm with respect to ‖ • ‖, ‖𝐴‖ 2 = sup 𝑥∈𝐸:‖𝑥‖ 2 ≤1 ‖𝐴(𝑥)‖ 2 . The results of Section 4.2.2 transpose directly to the general case by replacing ‖ • ‖ 1 by ‖ • ‖. Precisely, assuming that 𝑏 = 𝐴𝑥 * + 𝑤 where ‖𝑤‖ 2 ≤ 𝛿‖𝐴‖ 2 , an optimal solution x of problem (Robust sparse recovery) satisfies the error bound

‖x -𝑥 * ‖ 2 ≤ 2 𝛿‖𝐴‖ 2 𝜎 𝒯 (𝑥 * ) (𝐴) ,
where the tangent cone is defined as

𝒯 (𝑥) = cone{𝑧 : ‖𝑥 + 𝑧‖ ≤ ‖𝑥‖},
and robust recovery of 𝑠-sparse signals is therefore controlled by

𝜇 𝑠 (𝐴) = inf 𝑃 ∈𝒫 : 𝜂(𝑃 )≤𝑠 inf 𝑥∈𝐸 : 𝑃 𝑥=𝑥 min 𝑧∈𝒯 (𝑥) ‖𝑧‖ 2 =1 ‖𝐴𝑧‖ 2 . (4.20)
The key point is then to characterize the tangent cones of 𝑠-sparse signals. First, this will allow statistical estimations of 𝜇 𝑠 (𝐴). Second, it will enable us to estimate the constant (GNSP), hence sharpness of the exact recovery problem and computational complexity of associated restart schemes. This is the aim of the following lemma. Since 𝑧 ∈ 𝒯 (𝑥), we have ‖𝑥 + 𝑧‖ ≤ ‖𝑥‖; combined with the previous statement, this implies that 𝑧 ∈ {𝑧 ∈ 𝐸 : 𝑧 = 𝑃 𝑧 + P 𝑧, ‖ P 𝑧‖ ≤ ‖𝑃 𝑧‖} ⊂ ℰ. Now, if ‖.‖ is the nuclear norm, as in the proof of Proposition (4.3.3), we have

‖𝑥 + 𝑧‖ ≥ ‖𝑥‖ -‖𝑄𝑧‖ + ‖ Q𝑧‖,
where 𝑄 is the projector on the 𝑠 largest singular directions of 𝑧 given by the singular value decomposition of 𝑧, so that 𝑧 = 𝑄𝑧 + Q𝑧. Therefore, 𝑧 ∈ 𝒯 (𝑥) implies 𝑧 ∈ {𝑧 ∈ 𝐸 : 𝑧 = 𝑄𝑧 + Q𝑧, ‖ Q𝑧‖ ≤ ‖𝑄𝑧‖} ⊂ ℰ. In all cases we have therefore proven ℰ = ℱ.

Using the previous lemma, the minimal cone restricted singular value reads:

𝜇 𝑠 (𝐴) = inf 𝑃 ∈𝒫, 𝜂(𝑃 )≤𝑠 min 𝑧∈𝐸, ‖𝑧‖ 2 =1 𝑧=𝑃 𝑧+ P 𝑧, ‖ P 𝑧‖≤‖𝑃 𝑧‖ ‖𝐴𝑧‖ 2 .
(4.21)

This quantity can then be linked to the (GNSP) constant, as shown in the following lemma.

Lemma 4.3.5. Given a linear operator 𝐴 on 𝐸, If the minimal cone restricted singular value 𝜇 𝑠 (𝐴), defined in (4.20) and reformulated in (4.21), is positive, then 𝐴 satisfies (GNSP) at order 𝑠 for any constant

𝛼 ≤ ( 1 - 𝜇 𝑠 (𝐴) ‖𝐴‖ 2 ) -1 .
Proof. For a given 𝑃 ∈ 𝒫, denote 𝐶 𝑃 = {𝑧 ∈ ℑ(𝑃 ) + ℑ( P ) : ‖ P 𝑧‖ ≤ ‖𝑃 𝑧‖} and define for 𝛼 ≥ 1 the conic linear system find 𝑧 ∈ ℑ(𝑃 ) + ℑ( P ) s.t. 𝐴(𝑧) = 0 ‖ P 𝑧‖ ≤ 𝛼‖𝑃 𝑧‖, 𝑧 ̸ = 0.

(4.22) Infeasibility of this system for all 𝑃 ∈ 𝒫 such that 𝜂(𝑃 ) ≤ 𝑠 is then equivalent to (GNSP) at order 𝑠 with constant 𝛼. Denote 𝐻 = I -(1 -𝛼 -1 )𝑃 such that {𝑧 ∈ ℑ(𝑃 ) + ℑ( P ) : ‖ P 𝑧‖ ≤ 𝛼‖𝑃 𝑧‖} = 𝐻𝐶 𝑃 .

Since 𝐻 is invertible, we observe as in Lemma 4.2.5 that the conic linear system (4.22) is equivalent to find 𝑧 ∈ ℑ(𝑃 ) + ℑ( P )

s.t. 𝐴 -(1 -𝛼 -1 )𝐴𝑃 𝑧 = 0 𝑧 ∈ 𝐶 𝑃 , 𝑧 ̸ = 0. (4.23)
If this problem is infeasible for 𝛼 = 1, i.e., its distance to feasibility 𝜇 𝐶 𝑃 (𝐴) defined in (4.11) is positive, then (4.23) is infeasible for any 𝛼 ≥ 1 such that

(1 -𝛼 -1 )‖𝐴𝑃 ‖ ≤ 𝜇 𝐶 𝑃 (𝐴).
Now, if 𝜇 𝑠 (𝐴) > 0 the conic linear system (4.22) will still be infeasible for any

𝛼 ≤ ( 1 - 𝜇 𝑠 (𝐴) ‖𝐴‖ 2 ) -1 .
Thus, 𝐴 satisfies (GNSP) at order 𝑠 with 𝛼 as above.

Renegar's condition number

On the computational side, denote ℛ 𝒯 (𝑥) (𝐴) the Renegar condition number of the conic linear system find 𝑧 s.t. 𝐴(𝑧) = 0 𝑧 ∈ 𝒯 (𝑥), 𝑧 ̸ = 0, and the worst-case Renegar condition number on 𝑠-sparse signals

ℛ 𝑠 (𝐴) sup 𝑃 ∈𝒫 : 𝜂(𝑃 )≤𝑠 sup 𝑥∈𝐸 : 𝑃 𝑥=𝑥 ℛ 𝒯 (𝑥) (𝐴) = ‖𝐴‖ 2 /𝜇 𝑠 (𝐴).
First, Renegar's condition number plays the same role as before in computing optimality certificates for the exact recovery problems. Then, combining Lemma 4.3.5 and Proposition 4.3.3 shows that the sharpness bound for exact recovery reads

‖𝑥‖ -‖𝑥 * ‖ > 1 2ℛ 𝑠 (𝐴) -1 ‖𝑥 -𝑥 * ‖.
This sharpness will then control linearly convergent restart schemes for the exact recovery problem. Overall then, as established earlier in this paper, a single geometric quantitynamely, the minimal cone restricted singular value-appears to control both computational and statistical aspects. We now illustrate this statement on numerical experiments.

Numerical results

In this section, we first test the empirical performance of restart schemes and its link with recovery performance. We then perform similar experiments on Renegar's condition number.

Sharpness & restart for exact recovery

We test the (Restart) scheme on ℓ 1 -recovery problems with random design matrices. Throughout the experiments, we use the NESTA code described in [START_REF] Becker | Nesta: A fast and accurate first-order method for sparse recovery[END_REF] as the subroutine in the restart strategy. We generate a random design matrix 𝐴 ∈ R 𝑛×𝑝 with i.i.d. Gaussian coefficients. We then normalize 𝐴 so that 𝐴𝐴 𝑇 = I (to fit NESTA's format) and generate observations 𝑏 = 𝐴𝑥 * where 𝑥 * ∈ R 𝑝 is an 𝑠-sparse vector whose nonzero coefficients are all ones. We denote x the solution given by a common solver run at machine precision and plot convergence 𝑓 (𝑥 𝑡 ) -𝑓 * = ‖𝑥 𝑡 ‖ 1 -‖x‖ 1 (scaled such that 𝑓 (𝑥 0 ) -𝑓 (x) = 1).

Restart scheme performance

First we compare in Figure 4-1 the practical scheme presented in Section 4.1.2 with a plain implementation of NESTA without restart or continuation steps. Dimensions of the problem are 𝑝 = 500, 𝑛 = 200 and 𝑠 = 30. Starting from 𝑥 0 = 𝐴 𝑇 𝑏, we use 𝜀 0 = ‖𝑥 0 ‖ 1 as a first initial guess on the gap and perform a grid search of step size ℎ = 4 for a budget of 𝑁 = 500 iterations. The first and last schemes of the grid search were not run as they are unlikely to produce a nearly optimal restart scheme. The grid search can be parallelized and the best scheme found is plotted with a solid red line. The dashed red line represents the convergence rate accounting for the cost of the grid search. For the plain implementation of NESTA, we used different target precisions. These control indeed the smoothness of the surrogate function 𝑓 𝜀 which itself controls the step size of Nesterov's algorithm. Therefore a high precision slows down the algorithm. However for low precision NESTA can be faster but will not approximate well the original signal. Also, the theoretical bound (4.2) might be very pessimistic, as the surrogate function 𝑓 𝜀 may approximate the ℓ 1 norm for the points of interest at a much better accuracy than 𝜀.

Overall, we observe a clear linear convergence of the restart scheme that outperforms the plain implementation. This was already observed by [START_REF] Becker | Nesta: A fast and accurate first-order method for sparse recovery[END_REF] who developed their continuation steps against which we compare in Figure 4-2. We used default options for NESTA, namely 5 continuation steps with a stopping criterion based on the relative objective change in the surrogate function (specifically, the algorithm stops when these changes are lower than the target accuracy, set to 10 -6 ). We compare continuations steps and best restart found by grid search for different dimensions of the problem, we fix 𝑝 = 500, 𝑠 = 30 and vary the

Inner iterations

Figure 4-1 -Best restarted NESTA (solid red line) and overall cost of the practical restart schemes (dashed red line) versus plain NESTA implementation with low accuracy 𝜀 = 10 -1 (dotted black line) and higher accuracy 𝜀 = 10 -3 (dash-dotted black line) for a budget of 500 iterations. number of samples 𝑛 = {200, 300}. Continuation steps converge faster with better conditioned problems, i.e., more samples. Otherwise they may get stuck due to the termination criterion. Notice that a lot of parameters are involved for both algorithms, in particular the target precision may play an important role, so that more extensive experiments may be needed to refine these statements.

Our goal here is to provide a simple but strong baseline with theoretical guarantees for recovery. Improving on it, as [START_REF] Fercoq | Restarting accelerated gradient methods with a rough strong convexity estimate[END_REF] did for LASSO, is an appealing research direction. Sharpness may be used for example to refine the heuristic strategy of the continuations steps. Table 4.1 -Time to achieve 𝜀 = 10 -2 by the best restart scheme for increasing number of samples 𝑛

Convergence rate and oversampling ratio

We now illustrate the theoretical results of Section 4.1.3 by running the practical scheme presented in Section 4.1.2 for increasing values of the oversampling ratio 𝜏 = 𝑛/𝑠. In Figure 4-3, we plot the best scheme found by the grid search, that approximates the optimal scheme, for a budget of 𝑁 = 500 iterations. We use a fine grid of step size ℎ = 2. Other algorithmic parameters remain unchanged: 𝑥 0 = 𝐴 𝑇 𝑏 and 𝜀 0 = ‖𝑥 0 ‖ 1 . We fix the dimension 𝑝 = 1000 and either make 𝑛 vary for a fixed sparsity 𝑠 = 20 or make 𝑠 vary for a fixed number of samples 𝑛 = 200. In both cases we do observe an improved convergence for increasing oversampling ratio 𝜏 . Notice that linear convergence is observed even for 𝜏 < log 𝑝, that is, outside the recovery region. This suggests that sharpness may also hold in this case in the form of a Łojasiewicz inequality as mentioned in Sections 4.1.2 and 4.2.3. In table 4.1, we report the time needed to achieve an 𝜀 = 10 -2 precision for a fixed sparsity 𝑠 = 20 and increasing number of samples 𝑛 = {100, 200, 400}. While the cost of core operations (namely, matrix-vector multiplications) increases with 𝑛, the overall time required to reach convergence decreases. In other words, getting more samples increases the recovery performance at little or no computational cost.

Renegar's condition number and compressed sensing performance

Our theoretical results showed that Renegar's condition number measures the complexity for the exact recovery problem (4.1). However it does not a priori control convergence of the robust recovery problems defined in the introduction. This numerical section aims therefore at analyzing the relevance of this condition number for general recovery problems in the ℓ 1 case, assuming that their complexity corresponds roughly to that of checking optimality of a given point at each iteration, as mentioned in Section 4.2.3. We first describe how we approximate the value of ℛ 𝒯 (𝑥 * ) (𝐴) as defined in (4.18) for a given original signal 𝑥 * and matrix 𝐴 ∈ R 𝑛×𝑝 . We then detail numerical experiments on synthetic data sets.

Computing ℛ 𝒯 (𝑥 * ) (𝐴)
The condition number ℛ 𝒯 (𝑥 * ) (𝐴) appears here in upper bounds on computational complexities and statistical performances. In order to test numerically whether this quantity truly explains those features (as opposed to merely appearing in a wildly pessimistic bound), we explicitly compute it in numerical experiments.

To compute ℛ 𝒯 (𝑥 * ) (𝐴), we propose a heuristic which computes 𝜎 𝒯 (𝑥 * ) (𝐴) in (4.11) and (4.12), the value of a non convex minimization problem over the cone of descent directions 𝒯 (𝑥 * ). The closure of the latter is the polar of the cone generated by the subdifferential to the ℓ 1 -norm ball at 𝑥 * [Chandrasekaran et al., 2012, §2.3]. Let 𝑆 ⊂ 1, 𝑝 denote the support of 𝑥 * and 𝑠 = Card(𝑆). Then, with 𝑢 = sign(𝑥 * ),

𝒯 (𝑥 * ) = cone { 𝑧 ∈ R 𝑝 : 𝑧 𝑆 = 𝑢 𝑆 , 𝑧 𝑆 𝑐 ∈ [-1, 1] 𝑝-𝑠 ) } ∘ = { 𝑧 ∈ R 𝑝 : ‖𝑧 𝑆 𝑐 ‖ 1 ≤ -𝑢 𝑇 𝑆 𝑧 𝑆 = -𝑢 𝑇 𝑧 } .
Thus, 𝜎 𝒯 (𝑥 * ) (𝐴) is the square root of

min 𝑧∈R 𝑝 𝑧 𝑇 𝐴 𝑇 𝐴𝑧 s.t. ‖𝑧‖ 2 = 1 and ‖𝑧 𝑆 𝑐 ‖ 1 ≤ -𝑢 𝑇 𝑧. (4.24)
Let 𝜆 denote the largest eigenvalue of 𝐴 𝑇 𝐴. If it were not for the cone constraint, solutions of this problem would be the dominant eigenvectors of 𝜆I -𝐴 𝑇 𝐴, which suggests a projected power method [START_REF] Deshpande | Cone-constrained principal component analysis[END_REF] as follows. Given an initial guess

𝑧 0 ∈ R 𝑝 , ‖𝑧 0 ‖ 2 = 1, iterate ẑ𝑘+1 = Proj 𝒯 (𝑥 * ) ( (𝜆I -𝐴 𝑇 𝐴)𝑧 𝑘 ) , 𝑧 𝑘+1 = ẑ𝑘+1 /‖ẑ 𝑘+1 ‖ 2 , (4.25)
where we used the orthogonal projector to 𝒯 (𝑥 * ),

Proj 𝒯 (𝑥 * ) (z) = arg min 𝑧∈R 𝑝 ‖𝑧 -z‖ 2
iteration [START_REF] Luss | Conditional gradient algorithmsfor rank-one matrix approximations with a sparsity constraint[END_REF][START_REF] Journée | Generalized power method for sparse principal component analysis[END_REF]]

𝑧 𝑘+1 ∈ arg max 𝑧∈R 𝑝 𝑧 𝑇 (𝜆I -𝐴 𝑇 𝐴)𝑧 𝑘 s.t. ‖𝑧‖ 2 ≤ 1 and ‖𝑧 𝑆 𝑐 ‖ 1 ≤ -𝑢 𝑇 𝑧.
From the latter, it follows that ‖𝐴𝑧 𝑘 ‖ 2 decreases monotonically with 𝑘. Indeed, owing to convexity of 𝑓 (𝑧) = 1 2 𝑧 𝑇 (𝜆I-𝐴 𝑇 𝐴)𝑧, we have 𝑓 (𝑧)-𝑓 (𝑧 𝑘 ) ≥ (𝑧 -𝑧 𝑘 ) 𝑇 (𝜆I-𝐴 𝑇 𝐴)𝑧 𝑘 . The next iterate 𝑧 = 𝑧 𝑘+1 maximizes this lower bound on the improvement. Since 𝑧 = 𝑧 𝑘 is admissible, the improvement is nonnegative and 𝑓 (𝑧 𝑘 ) increases monotonically.

Thus, the sequence ‖𝐴𝑧 𝑘 ‖ 2 converges, but it may do so slowly, and the value it converges to may depend on the initial iterate 𝑧 0 . On both accounts, it helps greatly to choose 𝑧 0 well. To obtain one, we modify (4.24) by smoothly penalizing the inequality constraint in the cost function, which results in a smooth optimization problem on the ℓ 2 sphere. Specifically, for small 𝜀 1 , 𝜀 2 > 0, we use smooth proxies

ℎ(𝑥) = √ 𝑥 2 + 𝜀 2 1 -𝜀 1 ≈ |𝑥| and 𝑞(𝑥) = 𝜀 2 log(1 + exp(𝑥/𝜀 2 )) ≈ max(0, 𝑥).
Then, with 𝛾 > 0 as Lagrange multiplier, we consider

min ‖𝑧‖ 2 =1 ‖𝐴𝑧‖ 2 2 + 𝛾 • 𝑞 ( 𝑢 𝑇 𝑧 + ∑ 𝑖∈𝑆 𝑐 ℎ(𝑧 𝑖 )
) .

We solve the latter locally with Manopt [START_REF] Boumal | Manopt, a Matlab toolbox for optimization on manifolds[END_REF], itself with a uniformly random initial guess on the sphere, to obtain 𝑧 0 . Then, we iterate the projected power method. The value ‖𝐴𝑧‖ 2 is an upper bound on 𝜎 𝒯 (𝑥 * ) (𝐴), so that we obtain a lower bound on ℛ 𝒯 (𝑥 * ) (𝐴). Empirically, this procedure, which is random only through the initial guess on the sphere, consistently returns the same value, up to five digits of accuracy, which suggests the proposed heuristic computes a good approximation of the condition number. Similarly positive results have been reported on other cones by [START_REF] Deshpande | Cone-constrained principal component analysis[END_REF], where the special structure of the cone even made it possible to certify that this procedure indeed attains a global optimum in proposed experiments. Similarly, a generalized power method was recently shown to converge to global optimizers for the phase synchronization problem (in a certain noise regime) [START_REF] Boumal | Nonconvex phase synchronization[END_REF][START_REF] Zhong | Near-optimal bounds for phase synchronization[END_REF]. This gives us confidence in the estimates produced here.

Sparse recovery performance

We conduct numerical experiments in the ℓ 1 case to illustrate the connection between the condition number ℛ 𝒯 (𝑥 * ) (𝐴), the computational complexity of solving (ℓ 1 recovery), and the statistical efficiency of the estimator (Robust ℓ 1 recovery). Importantly, throughout the experiments, the classical condition number of 𝐴 will remain essentially constant, so that the main variations cannot be attributed to the latter.

We follow a standard setup, similar to some of the experiments by [START_REF] Donoho | Fast solution of ℓ 1 -norm minimization problems when the solution may be sparse-norm minimization problems when the solution may be sparse[END_REF]. Fixing the ambient dimension 𝑝 = 300 and sparsity 𝑠 = ‖𝑥 * ‖ 0 = 15, we let the number of linear measurements 𝑛 vary from 1 to 150. For each value of 𝑛, we generate a random signal 𝑥 * ∈ R 𝑝 (uniformly random support, i.i.d. Gaussian entries, unit ℓ 2 -norm) and a random sensing matrix 𝐴 ∈ R 𝑛×𝑝 with i.i.d. standard Gaussian entries. Furthermore, for a fixed value 𝛿 = 10 -2 , we generate a random noise vector 𝑤 ∈ R 𝑛 with i.i.d. standard Gaussian entries, normalized such that ‖𝑤‖ 2 = 𝛿‖𝐴‖ 2 , and we let 𝑏 = 𝐴𝑥 * + 𝑤. This is repeated 100 times for each value of 𝑛.

For each triplet (𝐴, 𝑥 * , 𝑏), we first solve the noisy problem (Robust ℓ 1 recovery) with the L1-Homotopy algorithm (𝜏 = 10 -7 ) [START_REF] Asif | Sparse recovery of streaming signals using l1homotopy[END_REF], and report the estimation error ‖x -𝑥 * ‖ 2 . Then, we solve the noiseless problem (4.1) with L1-Homotopy and the TFOCS routine for basis pursuit (𝜇 = 1) [START_REF] Becker | Templates for convex cone problems with applications to sparse signal recovery[END_REF]. Exact recovery is declared when the error is less than 10 -5 , and we report the empirical probability of exact recovery, together with the number of iterations required by each of the solvers. The number of iterations of LARS [START_REF] Efron | Least angle regression[END_REF] is also reported, for comparison. For L1-Homotopy, we report the computation time, normalized by the computation time required for one least-squares solve in 𝐴, as in [Donoho and Tsaig, 2008, Fig. 3], which accounts for the growth in 𝑛. Finally, we compute the classical condition number of 𝐴, 𝜅(𝐴), as well as (a lower bound on) the cone-restricted condition number ℛ 𝒯 (𝑥 * ) (𝐴), as per the previous section. As it is the computational bottleneck of the experiment, it is only computed for 20 of the 100 repetitions.

The results of Figure 4-4 show that the cone-restricted condition number explains both the computational complexity of (ℓ 1 recovery) and the statistical complexity of (Robust ℓ 1 recovery): fewer samples mean bad conditioning which in turn implies high computational complexity. We caution that our estimate of ℛ 𝒯 (𝑥 * ) (𝐴) is only a lower bound. Indeed, for small 𝑛, the third plot on the left shows that, even in the absence of noise, recovery of 𝑥 * is not achieved by (Robust ℓ 1 recovery). Lemma 4.2.3 then requires ℛ 𝒯 (𝑥 * ) (𝐴) to be infinite. But the computational complexity of solving (ℓ 1 recovery) is visibly favorable for small 𝑛, where far from the phase transition, problem (P A,𝒯 (𝑥) ) is far from infeasibility, which is just as easy to verify as it is to certify that (P A,𝒯 (𝑥) ) is infeasible when 𝑛 is comfortably larger than needed. This phenomenon is best explained using a symmetric version of the condition number [Amelunxen and Lotz, 2014] (omitted here to simplify computations).

We also solved problem (ℓ 1 recovery) with interior point methods (IPM) via CVX. The number of iterations appeared mostly constant throughout the experiments, suggesting that the practical implementation of such solvers renders their complexity mostly data agnostic in the present setting. Likewise, the computation time required by L1-Homotopy on the noisy problem (Robust ℓ 1 recovery), normalized by the time of a least-squares solve, is mostly constant (at about 150). This hints that the link between computational complexity of (ℓ 1 recovery) and (Robust ℓ 1 recovery) remains to be fully explained.

Conclusion

We studied the geometry of sparse recovery problems around their solutions that control both computational efficiency of restart schemes or oracle-based algorithms for exact recovery, and statistical performance of the decoding procedures, either in the recovery threshold or in the sensitivity to noise. We generally show that minimal conically restricted singular value of the coding matrix is the key quantity that describes the problem and can be seen as the worst case of Renegar's condition number for conic feasibility problems that certificate recovery. This analysis extends then to other sparse structures such as group sparsity or low rank matrices. Several questions remain. First, sharpness constant is only given when exact recovery is ensured. As the ℓ 1 norm is always sharp, our analysis may extend to problems where exact recovery is not possible. In other words, if the decoding procedure is far from being efficient statistically, its complexity may also be small. Furthermore, for robust recovery, sharpness may also be stated. A finer analysis may help to link it with minimal conically restricted singular value on the tangent cone of the original vector which would give instance dependent complexity bounds. Finally a proper definition of sparse structures would ease the development of new compressed structures such as grouped vectors that we present in Chapter 6.

where ‖ • ‖ * is the dual norm of ‖ • ‖ and 𝑃 * is the conjugate mapping of the linear map 𝑃 .

The last condition in Definition 4.B.1 is arguably the least intuitive and following Lemma connects it with the more intuitive notion of decomposable norm. Since 𝑓 and 𝑔 are non-negative, continuous convex functions, 𝑓 2 /2 and 𝑔 2 /2 are also convex continuous and following equivalences hold

𝑓 ≤ 𝑔 ⇔ 𝑓 2 2 ≤ 𝑔 2 2 ⇔ ( 𝑓 2 2 ) * ≥ ( 𝑔 2 2 ) * ,
using that for a convex continuous function ℎ, ℎ ** = ℎ. Now combining the conjugacy result for squared norm [Boyd and Vandenberghe, 2004, Example 3.27] showing that the conjugate of a squared norm ‖𝑥‖ 2 /2 is the squared conjugate norm ‖𝑥‖ 2 * /2, with the result in [Rockafellar, 2015, Th. 16.3], we get Chapter 5

( 𝑓 2 2 ) * (𝑠, 𝑡) = inf

Introduction

Machine learning

Machine learning is a recent field of research that attempts to give to a computer the ability to perform predictive analysis of data. It stems from statistics which model the data and optimization which makes the computer find the best parameters of the model. It has been applied to numerous fields such as computer vision, natural language processing, bio-informatics, robotics, speech processing and economics. Machine learning can be applied for example to predict antigens concentration in blood after a medical treatment. Another classical application is image classification, where the computer must distinguish pictures of humans, animals or objects.

Formally, data consist of objects 𝑥 ∈ 𝒳 and their attributes 𝑦 ∈ 𝒴 that the computer must predict. To perform predictive analysis, the computer has generally access to the representation of the object by 𝑑 features, such that throughout this part 𝒳 = R 𝑑 . Attributes can be quantitative values, i.e. 𝒴 = R, as in prediction of antigens concentration, or qualitative values, 𝒴 = {1, . . . 𝐾} that encode 𝐾 classes of objects, as in image classification.

Several settings exist to analyze data depending on the information that a computer can have access. In this thesis, we study supervised learning problems where a batch of training samples is available. It consists in 𝑛 pairs (𝑥 1 , , 𝑦 1 ), . . . , (𝑥 𝑛 , 𝑦 𝑛 ) of objects and their attributes that the computer uses to fit a model on the data.

Big data challenges

Machine learning appeals a growing interest of researchers because of the large amount of data available in applications that the prediction task can benefit from. By large, one refers first to the number 𝑛 of training samples like the number of documents in a topic classification task. The larger is 𝑛, the more performing is naturally the computed model.

Big data can also concern the number 𝑑 of features that describe the data. For example in genomics, features are frequency of mutations of genes and the number of discriminative genes in ADN is around 3 millions. Similarly, the more features one can have access to describe the data, the more information the computer can use.

Finally, machine learning problems may treat large number 𝐾 of tasks simultaneously. For example, in computer vision, a common task is to identify elements of a picture such as a cat or a car. Each element requires a specific classification task and data sets like ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF] of 1000 classes exist to describe the variety of possible elements in a picture. Here again a large number 𝐾 of tasks may help because similar tasks can share information to improve overall performance.

While this big data setting helps prediction performance, it also increases complexity of the task. First, it rises new computational challenges for the optimization procedure but the resulting model can also be more difficult to interpret. Learning procedures able to perform prediction and simplify the problem are therefore of interest for the user. They not only help interpretation but also increase generalization performance.

Outline of this part

In this part, we study therefore learning procedures that simultaneously solve a prediction problem and reduce its complexity by grouping either features in Chapter 6, samples in Chapter 7 or tasks in Chapter 8. They mix classical supervised learning procedures and partitioning problems. The algebraic tools and optimization procedures are essentially the same for all three settings. We present them in Sections 5.2 and 5.3. A general perspective of our approach is given in Section 5.1. Following chapters detail then each application and specific computations.

Learning in the data cube

For a general supervised machine learning problem involving 𝑛 samples with 𝑑 features to solve 𝐾 tasks, supervised learning methods have at their disposition 𝑛 input data points 𝑥 1 , . . . , 𝑥 𝑛 ∈ R 𝑑 with their corresponding outputs for 𝐾 tasks 𝑦 1 , . . . , 𝑦 𝑛 ∈ R 𝐾 . Linear methods aim then at finding 𝐾 vectors of prediction 𝑤 1 , . . . 𝑤 𝐾 ∈ R 𝑑 that maps inputs to outputs as Notice that to perform prediction we assume that all samples share the same predictor. However, to account for diversity in the samples, this assumption can be relaxed by searching for more than one predictor as detailed in Chapter 7. Diverse outputs are then delivered by the machine depending on the prediction variable chosen. This means that our prediction variable can generally be seen as a tensor 𝒲 = (𝑤 𝑗𝑘𝑖 ) 𝑗∈{1,...𝑑}, 𝑘∈{1,...,𝐾}, 𝑖∈{1,...,𝑛} ∈ R 𝑑×𝐾×𝑛 whose coefficients along the third direction are constrained to be equal for classical prediction which leads to the matrix of prediction 𝑊 presented before. The three dimensions of this tensor dress the data cube as introduced by [START_REF] Harchaoui | Large-scale learning for image classification[END_REF].

𝑦 𝑖𝑘 ≈ 𝑤 𝑇 𝑘 𝑥 𝑖 ,
In this part we study grouping of either features, samples or tasks by constraining coordinates of the tensor 𝒲 along the different directions it defines in the data cube. Precisely, we constraint the coefficients to be equal or closed to each other in some groups defined by partitions.

Partitioning problems

Representation of partitions

We present classical algebraic tools to partition 𝑝 items, where, in our following settings, 𝑝 = 𝑑 for features, 𝑝 = 𝑛 for samples or 𝑝 = 𝐾 for tasks. First, recall the definition of partitions.

Definition 5.2.1. Partitions A collection 𝐺 of subsets of {1, . . . , 𝑝} is a partition of {1, . . . , 𝑝} if for any 𝑔, 𝑔 ′ ∈ 𝐺×𝐺, 𝑔 ̸ = 𝑔 ′ implies 𝑔∩𝑔 ′ = ∅ and if ⋃ 𝑔∈𝐺 𝑔 = {1, . . . , 𝑝}. Partitions can be partially ordered as detailed in Section 6.4.2. They possess a lattice structure [START_REF] Fujishige | Submodular functions and optimization[END_REF] that can be used for combinatorial optimization [START_REF] Topkis | Minimizing a submodular function on a lattice[END_REF][START_REF] Amini | Submodular partition functions[END_REF].

Here we use simple algebraic tools to represent them. First, for a partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) of a set {1, . . . , 𝑝} of 𝑝 elements in 𝑄 groups, one defines assignment matrices 𝑍 ∈ {0, 1} 𝑝×𝑄 that assigns items into the groups 𝑔 1 , . . . , 𝑔 𝑄 by

𝑍 𝑖𝑞 = { 1 if 𝑖 ∈ 𝑔 𝑞 , 0 otherwise.
Observe that a partition 𝐺 into 𝑄 groups 𝑔 1 , . . . 𝑔 𝑄 is independent of the ordering of the groups, namely, 𝑔 𝜋(1) , . . . 𝑔 𝜋(𝑄) , where 𝜋 is a permutation of {1, . . . , 𝑄}, describes as well 𝐺. Consequently a partition can be encoded by several assignment matrices, these are identical up to a permutation of their columns defining the groups.

A binary matrix 𝑍 ∈ {0, 1} 𝑝×𝑄 describes a partition 𝐺 of {1, . . . , 𝑝} into 𝑄 groups, if and only if it satisfies 𝑍1 = 1 as it encodes the fact that each element belongs to exactly one group. Since groups of a partition are disjoints, columns of assignment matrices are orthogonal. Their squared Euclidean norm and ℓ 1 norm are equal to the size of the groups they represent, i.e. ‖𝑍 𝑞 ‖ 2 2 = ‖𝑍 𝑞 ‖ 1 = 𝑍 𝑇 1 = Card(𝑔 𝑞 ), where 𝑍 𝑞 is the 𝑞 th column of an assignment matrix 𝑍 of a partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ). Combining two previous comments, we conclude that size of the groups are the squared singular values of the assignment matrix., i.e. 𝑍 𝑇 𝑍 = diag(𝑠) where 𝑠 = (Card(𝑔 1 , . . . , Card(𝑔 𝑄 )) encodes the size of the groups 𝑔 1 , . . . , 𝑔 𝑄 that 𝑍 represents.

Another way to represent a partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) of {1, . . . , 𝑝} into 𝑄 groups is to use its equivalence matrix 𝑁 ∈ {0, 1} 𝑝×𝑝 that encodes if pair of items belong to the same group as

𝑁 𝑖𝑗 = { 1 if (𝑖, 𝑗) ∈ (𝑔 𝑞 × 𝑔 𝑞 ) 0 otherwise.
They can be derived from assignment matrices 𝑍 of the partition 𝐺 as 𝑁 = 𝑍𝑍 𝑇 . Contrary to assignment matrices, there exists only one equivalence matrix per partition. In other words, partitions are uniquely defined by pairwise relationships of the items. Equivalence matrices are notably used for finding minimal cuts of graph, which partitions its vertices. Resulting problem is then convex but can lead to unsatisfactory solutions as it may only separate one edge to the others [Von [START_REF] Luxburg | A tutorial on spectral clustering[END_REF].

To circumvent this problem, balanced cuts penalize the resulting partitions by the size of the groups. This leads to the definition of normalized equivalence matrices 𝑀 ∈ R 𝑝×𝑝 , that encode a partition 𝐺 = {𝑔 1 , . . . , 𝑔 𝑄 } of {1, . . . , 𝑝} as

𝑀 𝑖𝑗 = { 1/ Card(𝑔 𝑞 ) if (𝑖, 𝑗) ∈ (𝑔 𝑞 × 𝑔 𝑞 ) 0 otherwise.
In terms of assignment matrices, these read 𝑀 = 𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 , where 𝐴 † denotes the pseudo-inverse of a matrix 𝐴, here (𝑍 𝑇 𝑍) † = diag(𝑠 † ), where 𝑠 † 𝑞 = 1/ Card(𝑔 𝑞 ) if 𝑔 𝑞 is non-empty and 𝑠 † 𝑞 = 0 otherwise. Normalized equivalence matrices are orthonormal projectors, i.e. 𝑀 2 = 𝑀 and 𝑀 𝑇 = 𝑀 . While normalized equivalence matrices may better formulate a task, they lead to non-convex problems such as k-means.

K-means problem

A well-known and appealing partitioning problem is the clustering of points into groups of points closed to each other. Its common example is the k-means problem [START_REF] Steinhaus | Sur la division des corp materiels en parties[END_REF][START_REF] Macqueen | Some methods for classification and analysis of multivariate observations[END_REF] that seeks to partition 𝑝 points 𝑥 1 , . . . , 𝑥 𝑝 ∈ R 𝑑 into 𝑄 clusters that minimize distance between points and centers 𝑐 1 , . . . , 𝑐 𝑄 of the clusters. Several distances can be used [START_REF] Dhillon | Kernel k-means: spectral clustering and normalized cuts[END_REF], the original one is the Euclidean distance which leads to following problem minimize

𝑄 ∑ 𝑞=1 ∑ 𝑖∈𝑔𝑞 ‖𝑥 𝑖 -𝑐 𝑞 ‖ 2 2 ,
in the partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) of {1, . . . , 𝑝} and the centroids 𝑐 1 , . . . , 𝑐 𝑄 . By using assignment matrices 𝑍 ∈ {0, 1} 𝑝×𝑄 to represent partitions, the k-means problem reads minimize‖𝑋 -𝑍𝐶‖ 2

𝐹

(5.1) in variables 𝐶 = (𝑐 1 , . . . , 𝑐 𝑄 ) 𝑇 ∈ R 𝑄×𝑑 and 𝑍, where 𝑋 = (𝑥 1 , . . . , 𝑥 𝑝 ) 𝑇 ∈ R 𝑝×𝑑 is the matrix of data points. In following chapters, we develop formulations of our predic-tion problems that group features, samples or tasks into 𝑄 groups using assignment matrices 𝑍 such that constraint set takes the form

{𝑊 = 𝑉 𝑍, 𝑍 ∈ {0, 1} 𝑝×𝑄 }
where 𝑝 is the number of items to be grouped, 𝑊 represents predictors with potentially redundant coefficients and 𝑉 represents the predictors without redundancy. Dimensions of 𝑊 and 𝑉 depend on the setting. Assignment matrices make the constraint set non-convex but projection on it reads (5.1). Therefore projected gradient presented in next section can be approximately performed. K-means problem (5.1) can be simplified by minimizing in the matrix of centroids which leads to minimize ‖𝑈 -𝑍(𝑍 𝑇 𝑍)

-1 𝑍 𝑇 𝑈 ‖ 2 𝐹 subject to 𝑍 ∈ {0, 1} 𝑑×𝑄 , 𝑍1 = 1, (5.2)
in variable 𝑍, where we recognize normalized equivalence matrices of partitions 𝑀 = 𝑍(𝑍 𝑇 𝑍) -1 𝑍 𝑇 . In our prediction problems, when using a squared loss, empirical loss minimization can be performed analytically. It remains a partitioning problem in terms of normalized equivalence matrices that can be solved using conditional gradient presented in next section. Core inner step of this algorithm, namely linear minimization oracle, amounts indeed to the k-means problem (5.2), as detailed in following chapters.

Optimization on non-convex sets

By imposing partitioning structure on a classical learning task we introduce nonconvex constraints in the optimization procedure. Formally, we face a problem of the form minimize 𝑓 (𝑥) subject to 𝑥 ∈ 𝐶, (5.3) in 𝑥 ∈ R 𝑑 , where 𝑓 is a 𝐿-smooth convex function and 𝐶 is a closed set. In this section we review optimization strategies for such problems.

Projected gradient descent

Projected gradient scheme initialized on 𝑥 0 ∈ 𝐶 for (5.3) reads

𝑥 𝑡+1 = 𝑃 𝐶 (𝑥 𝑡 -𝛾 𝑡 ∇𝑓 (𝑥 𝑡 )),
where 𝛾 𝑡 is the step-size and 𝑃 𝐶 (𝑥) ∈ argmin 𝑦∈𝐶 ‖𝑥 -𝑦‖ 2 is the projection on set 𝐶.

If 𝐶 is convex, projection is uniquely defined and a constant step-size 𝛾 𝑡 = 1 𝐿 ensures convergence to a solution of ( 5.3) at rate 𝑂(1/𝑡) [Nesterov, 2013b]. When 𝐶 is non-convex convergence is no more guaranteed without further assumptions on the problem. Yet, projected gradient schemes offer fast and good algorithmic solutions for nu-merous models. In some cases their convergence can even be stated. A well-known example is the Iterative Hard Thresholding algorithm [START_REF] Blumensath | Iterative hard thresholding for compressed sensing[END_REF] used in compressed sensing to decode a given sparse signal from linear observations of it. Provided that linear observations satisfy the restricted isometric property [START_REF] Candès | Decoding by linear programming[END_REF], Iterative Hard Thresholding is shown to approximately recover the sparse signal. Such analysis was extended to low rank estimation of matrices [START_REF] Tanner | Normalized iterative hard thresholding for matrix completion[END_REF] or tensors [START_REF] Rauhut | Low rank tensor recovery via iterative hard thresholding[END_REF]. Key ingredients are the structure of the constraint set, namely a union of subspaces and the restricted strong convexity property that the function 𝑓 satisfies [START_REF] Jain | On iterative hard thresholding methods for high-dimensional m-estimation[END_REF].

In our settings, projected gradient descent offer fast and scalable resolutions of our problems. However their analysis is delicate, since, even if the feasible set is a union of subspaces, the projection step is generally only approximated, as k-means is a NP-hard problem [START_REF] Mahajan | The planar k-means problem is np-hard[END_REF]]. Yet, in the case of regression grouping features, the k-means steps are performed on scalar values and can therefore be solved exactly by dynamic program. This allows us in Chapter 6 to adapt the analysis of classical Iterative Hard Thresholding algorithm in our setting

Convex relaxation

A more classical approach to solve non-convex problems of the form (5.3) is to approximate the non-convex set by a convex one. This ensures then efficient optimization of the resulting problem. The tightest relaxation consists in considering the convex hull of the original set which leads to minimize 𝑓 (𝑥) subject to 𝑥 ∈ hull(𝐶), in 𝑥 ∈ R 𝑑 . Projected gradient descent may then be delicate as it would require to build the convex hull and the projection itself may be costly. In such cases, the conditional gradient method, a.k.a. Frank-Wolfe [START_REF] Frank | An algorithm for quadratic programming[END_REF][START_REF] Jaggi | Revisiting frank-wolfe: Projection-free sparse convex optimization[END_REF], presented in Algorithm 5, can circumvent the problem. It uses a linear minimization oracle (5.4) to produce a sequence of feasible iterates that converge to a solution 𝑥 * of the problem at a rate 1/𝑡, where 𝑡 is the number of iterations. The linear minimization oracle also provides an estimated gap (5.5) for free, since, by definition of the oracle and convexity of 𝑓 ,

⟨𝑥 𝑡 -𝑠 𝑡 , ∇𝑓 (𝑥 𝑡 )⟩ ≥ ⟨𝑥 𝑡 -𝑥 * , ∇𝑓 (𝑥 𝑡 )⟩ ≥ 𝑓 (𝑥 𝑡 ) -𝑓 (𝑥 * ).

Introduction

In a prediction problem, getting a compressed representation of the information needed for the task has been extensively studied to improve prediction performance. Numerous models have been developed to select few features for the task (see e.g. [START_REF] Tang | Feature selection for classification: A review[END_REF]). In particular an extensive literature has been presented to tackle the problem by enforcing sparsity on the prediction vector (see [START_REF] Bach | Optimization with sparsity-inducing penalties[END_REF]). Here we rather focus on the problem of grouping features, which has various applications. For example, in text classification this amounts to group words that have the same meaning for the task (see e.g. [START_REF] Gupta | A survey of text mining techniques and applications[END_REF] and references therein). In biology, this can be used to retrieve groups of genomes that have the same impact on a disease (see e.g. [START_REF] Segal | Regression approaches for microarray data analysis[END_REF][START_REF] Balding | A tutorial on statistical methods for population association studies[END_REF]). More generally this approach can be seen as a supervised quantization of the feature space (see e.g. [START_REF] Nova | A review of learning vector quantization classifiers[END_REF] and references therein).

The idea of grouping features to reduce dimensionality of the problem is of course not new. [START_REF] Hastie | Supervised harvesting of expression trees[END_REF] used for example supervised learning methods to select group of predictive variables formed by hierarchical clustering. Several models also developed mutual information-based algorithms to remove redundant features, e.g. [START_REF] Yu | Feature selection for high-dimensional data: A fast correlation-based filter solution[END_REF][START_REF] Song | A fast clustering-based feature subset selection algorithm for high-dimensional data[END_REF][START_REF] Peng | Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy[END_REF]. More recently, regularizers were developed to enforce grouped vectors [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with oscar[END_REF][START_REF] She | Sparse regression with exact clustering[END_REF][START_REF] Petry | Pairwise fused lasso[END_REF]. In particular, [START_REF] Bach | Optimization with sparsity-inducing penalties[END_REF] analyzed geometrical properties induced by convex relaxations of submodular functions that lead to group structures. This geometrical perspective was also investigated by [START_REF] Bühlmann | Correlated variables in regression: clustering and sparse estimation[END_REF], who showed recovery performance of group norms induced by hierarchical clustering methods based on canonical correlations. Finally [START_REF] Shen | Grouping pursuit through a regularization solution surface[END_REF] developed an homotopy method to extract homogeneous subgroups of predictors.

In this chapter, we study a simple approach to the problem : while sparsity enforces a small number of non-zero coefficient of the prediction vector, we enforce a small number of values. This naturally induces groups of features that share the same weight for the prediction. We formulate our approach for regression and classification tasks in Section 6.1, by using assignment matrices to link features to the representative coefficient of their group. This leads to a non-convex problem that we tackle with several algorithmic schemes.

First, in Section 6.2, we analyze a convex relaxation of our model when a squared loss is used to measure the prediction error. In that case, the underlying combinatorial problem can be isolated such that we can use a conditional gradient algorithm, a.k.a. Frank-Wolfe algorithm [START_REF] Frank | An algorithm for quadratic programming[END_REF][START_REF] Jaggi | Revisiting frank-wolfe: Projection-free sparse convex optimization[END_REF], whose core inner step amounts to solving a clustering problem. Then, in Section 6.3 we present a simple projected gradient scheme similar to the Iterative Hard Thresholding (IHT) [START_REF] Blumensath | Iterative hard thresholding for compressed sensing[END_REF] algorithm used in compressed sensing. While constraints are non-convex, projection on the feasible set also reduces to a clustering subproblem. For both schemes, the clustering problem can be solved exactly with dynamic programming [START_REF] Bellman | A note on cluster analysis and dynamic programming[END_REF][START_REF] Wang | Ckmeans. 1d. dp: optimal k-means clustering in one dimension by dynamic programming[END_REF] for regression where it amounts to a k-means problem in one dimension, or approximated efficiently with k-means++ [START_REF] Arthur | k-means++: The advantages of careful seeding[END_REF] for classification.

We analyze the performance of the projected gradient scheme to recover a vector that generates the observations in Section 6.4. To this end, we detail the geometry induced by the partitioning constraints and demonstrate that, for regression with a least-square penalty and a sufficient number of observations, the projected gradient scheme can be seen as a fixed point algorithm. Although our structure is similar to sparsity, we show that imposing a grouped structure, while helping interpretability, does not allow us to significantly reduce the number of observations to retrieve the original vector, as in the sparse case for example.

We also extend the application of the projected gradient scheme to both select and group features in Section 6.5 by developing a new dynamic program that gives the exact projection on the set of sparse and clustered vectors.

Finally, numerical experiments illustrate the performance of the algorithmic schemes on both synthetic and real datasets involving large corpora of text from movie reviews. The use of k-means steps makes our approach fast and scalable while comparing favorably with standard benchmarks and providing meaningful insights on the data structure.

Problem Formulation

We first present our framework for linear regression tasks and then extend its application to classification.

Regression with grouped features

Given 𝑛 observations 𝑦 1 , . . . , 𝑦 𝑛 ∈ R from data points 𝑥 1 , . . . , 𝑥 𝑛 ∈ R 𝑑 , linear regression aims at finding a regression vector 𝑤 ∈ R 𝑑 that fits the data such that 𝑦 𝑖 ≈ 𝑤 𝑇 𝑥 𝑖 , for all 𝑖 = 1, . . . , 𝑛.

Parameter 𝑤 then serves to predict future observations 𝑦 of new data points 𝑥. To assess the quality of a prediction vector 𝑤, one defines a loss function ℓ that measures its accuracy error ℓ(𝑤 𝑇 𝑥, 𝑦) on a sample (𝑥, 𝑦). A common choice of loss, that we investigate here, is the squared loss ℓ square (𝑤 𝑇 𝑥, 𝑦) = 1 2 (𝑤 𝑇 𝑥 -𝑦) 2 , but several others exist in the literature, see [START_REF] Hastie | The elements of statistical learning: data mining, inference and prediction[END_REF]. A classical approach to compute a linear regression vector is then to minimize the empirical loss function

𝐿(𝑤) = 1 𝑛 𝑛 ∑ 𝑖=1 ℓ(𝑤 𝑇 𝑥 𝑖 , 𝑦 𝑖 ).
In order to prevent the computed prediction parameters from over-fitting the given set of samples, one often adds a regularizer 𝑅(𝑤) of the regression vector to the minimization problem. This notably reduces the effect of noise or outliers in the data. For example, one can use the squared Euclidean norm of the regression vector, i.e. 𝑅 square (𝑤) = 1 2 ‖𝑤‖ 2 . Candidate regression parameters are then given by the minimization problem minimize 𝐿(𝑤) + 𝜆𝑅(𝑤) (6.1) in variable 𝑤 ∈ R 𝑑 , where 𝜆 ≥ 0 is a regularization parameter.

Structural information on the task can then be added. For example, one can enforce the regression vectors 𝑤 to be sparse, i.e. to have few non-zeros coefficients.

Here we rather enforce the regression vectors 𝑤 to have at most 𝑄 values 𝑣 1 , . . . , 𝑣 𝑄 . Each coefficient 𝑣 𝑞 is assigned to a group of features 𝑔 𝑞 such that regression vectors 𝑤 define partitions 𝐺 = {𝑔 1 , . . . , 𝑔 𝑄 } of the features. We encode it by an assignment matrix 𝑍 ∈ {0, 1} 𝑑×𝑄 , whose rows index the features and columns index the groups, such that

𝑍 𝑖𝑞 = { 1 if 𝑖 ∈ 𝑔 𝑞 0 otherwise.
These are presented in Section 5.2. A regression vector 𝑤 satisfying the constraints can then be described by an assignment matrix 𝑍 and the prediction weights 𝑣 1 , . . . , 𝑣 𝑄 such that 𝑤 𝑖 = ∑ 𝑄 𝑞=1 𝑍 𝑖𝑞 𝑣 𝑞 = (𝑍𝑣) 𝑖 . Therefore linear regression enforcing 𝑄 groups of features reads minimize 𝐿(𝑤) + 𝜆𝑅(𝑤) subject to 𝑤 = 𝑍𝑣, 𝑍 ∈ {0, 1} 𝑑×𝑄 , 𝑍1 = 1 (6.2) in variables 𝑤 ∈ R 𝑑 , 𝑣 ∈ R 𝑄 and 𝑍, where 𝜆 ≥ 0 is a regularization parameter.

We detail the geometry of the non-convex set defined by the partitions of the features in Section 6.4.2 and extend now this formulation to classification tasks. Before notice that affine regression problems that seek for a regression vector 𝑤 ∈ R 𝑑 and an intercept 𝑏 ∈ R such that 𝑦 ≈ 𝑤 𝑇 𝑥 + 𝑏 can be treated similarly. It suffices to add a constant feature equals to one to data points 𝑥 and to consider the resulting problem in dimension 𝑑 + 1. In this case regularization function 𝑅 and partitioning constraints apply only on the first 𝑑 dimensions of the resulting problem.

Classification with grouped features

Numerous models have been proposed for classification, we refer the interesting reader to [START_REF] Hastie | The elements of statistical learning: data mining, inference and prediction[END_REF] for a detailed presentation. Here we briefly present one of them, namely one-vs-all linear classification, in order to focus on the optimization problem that will be constrained to group features. In classification, data points 𝑥 1 , . . . , 𝑥 𝑛 ∈ R 𝑑 belong to one of 𝐾 classes, which can be encoded by binary vectors 𝑦 𝑖 ∈ {-1, 1} 𝐾 such that 𝑦 𝑖𝑘 = 1 if 𝑖 th point belongs to class 𝑘 and -1 otherwise. One-vs-all linear classification aims then at computing hyperplanes defining regions of space where points are more likely to belong to a given class. Such hyperplanes are defined by their normals 𝑤 1 , . . . , 𝑤 𝐾 , forming a matrix of linear classifiers 𝑊 ∈ R 𝑑×𝐾 whose classification error on a sample (𝑥, 𝑦) is measured by a loss ℓ(𝑊 𝑇 𝑥, 𝑦) such as the squared loss ℓ square (𝑊 𝑇 𝑥, 𝑦) = 1 2 ‖𝑊 𝑇 𝑥 -𝑦‖ 2 2 . One searches then to minimize the empirical loss function

𝐿(𝑊 ) = 1 𝑛 𝑛 ∑ 𝑖=1 ℓ(𝑊 𝑇 𝑥 𝑖 , 𝑦 𝑖 ).
As for regression, a regularizer 𝑅(𝑊 ) can be added on the linear classifiers such as their squared euclidean norm

𝑅 square (𝑊 ) = 1 2 ∑ 𝐾 𝑘=1 ‖𝑤 𝑘 ‖ 2 2 = 1 2 ‖𝑊 ‖ 2 𝐹 .
Candidate classification parameters are then given by solving minimize 𝐿(𝑊 ) + 𝜆𝑅(𝑊 ) (6.3) in variable 𝑊 ∈ R 𝑑×𝐾 , where 𝜆 ≥ 0 is a regularization parameter.

To group features, we will enforce the classifiers to share the same partition of their coefficients. Namely, if this partition is encoded by an assignment matrix 𝑍 and 𝑣 𝑘 = (𝑣 1𝑘 , . . . , 𝑣 𝑄𝑘 ) represent the 𝑄 different coefficients of the 𝑘 th linear classifier 𝑤 𝑘 , then 𝑤 𝑘 = 𝑍𝑣 𝑘 . Linear classification enforcing 𝑄 groups of constraints then reads minimize 𝐿(𝑊 ) + 𝜆𝑅(𝑊 ) subject to 𝑊 = 𝑍𝑉, 𝑍 ∈ {0, 1} 𝑑×𝑄 , 𝑍1 = 1 (6.4) in variables 𝑊 ∈ R 𝑑×𝐾 , 𝑉 ∈ R 𝑄×𝐾 and 𝑍, where 𝜆 ≥ 0 is a regularization parameter. Observe that constraints in (6.4) are essentially the same as the ones in (6.2), except that these are formulated on matrices. However this simple difference will have important algorithmic implications. Notice that for binary classification, a vector of labels of dimension one is sufficient to encode the class information, such that binary classification reduces to a problem of the form (6.2). As for regression, this setting can be applied to compute affine hyperplanes by extending the problem in 𝑑 + 1 dimension and by applying regularization and constraints only on the first 𝑑 dimensions.

Convex relaxation

We now present a first optimization strategy for solving prediction problems (6.2) and (6.4) that group features, whose general formulation is

minimize 𝐿(𝑊 ) + 𝜆𝑅(𝑊 ) subject to 𝑊 = 𝑍𝑉, 𝑍 ∈ {0, 1} 𝑑×𝑄 , 𝑍1 = 1 (P)
in variables 𝑊 ∈ R 𝑑×𝐾 , 𝑉 ∈ R 𝑄×𝐾 and 𝑍, where 𝐿 and 𝑅 are respectively the loss and the regularizer of the problem and 𝜆 ≥ 0 is a regularization parameter. Regression and binary classification cases corresponds to 𝐾 = 1 and multiclassification to 𝐾 > 1. The difficulty of problem (P) lies in its underlying combinatorial nature that we isolate in the case of a squared loss and squared regularizer. Then, we propose a convex relaxation of the resulting problem which amounts to optimize on the convex hull of the set of constraints by using a conditional gradient algorithm.

Simplified formulation for squared loss

The squared loss has the advantage to provide analytic solutions for prediction problems in variables 𝑊, 𝑉 . By replacing 𝑊 = 𝑍𝑉 , the objective of problem (P) in the remaining variables 𝑍, 𝑉 reads

𝐿 square (𝑍𝑉 ) + 𝜆𝑅 square (𝑍𝑉 ) = 1 2𝑛 𝑛 ∑ 𝑖=1 ‖𝑦 𝑖 -(𝑍𝑉 ) 𝑇 𝑥 𝑖 ‖ 2 2 + 𝜆 2 ‖𝑍𝑉 ‖ 2 𝐹 = 1 2𝑛 Tr(𝑉 𝑇 𝑍 𝑇 𝑋 𝑇 𝑋𝑍𝑉 ) + 𝜆 2 Tr(𝑉 𝑇 𝑍 𝑇 𝑍𝑉 ) - 1 𝑛 Tr(𝑌 𝑇 𝑋𝑍𝑉 ) + 1 2𝑛 Tr(𝑌 𝑇 𝑌 ),
where 𝜆 ≥ 0 is a regularization parameter, 𝑋 = (𝑥 1 , . . . , 𝑥 𝑛 ) 𝑇 ∈ R 𝑛×𝑑 is the matrix of data points and 𝑌 = (𝑦 1 , . . . , 𝑦 𝑛 ) 𝑇 ∈ R 𝑛×𝐾 is the matrix of labels or the vector of observations in the regression case. Assume first 𝑍 to be full rank, then minimization in 𝑉 leads to

min 𝑉 𝐿 square (𝑍𝑉 ) + 𝜆𝑅 square (𝑍𝑉 ) = 1 2𝑛 Tr ( 𝑌 𝑇 ( I -𝑋𝑍(𝑍 𝑇 𝑋 𝑇 𝑋𝑍 + 𝜆𝑛𝑍 𝑇 𝑍) -1 𝑍 𝑇 𝑋 𝑇 ) 𝑌 ) = 1 2𝑛 Tr ( 𝑌 𝑇 ( I + 1 𝑛𝜆 𝑋𝑍(𝑍 𝑇 𝑍) -1 𝑍 𝑇 𝑋 𝑇 ) -1 𝑌 ) ,
where we simplified first expression by using the Sherman-Woodbury-Morrison formula. If 𝑍 is not full rank, some of its columns are null (some of the groups it represents are empty). Previous computations can then be performed by replacing 𝑍 by Z defined with the non-zero columns of 𝑍, such that Z is full rank and Z( Z𝑇 Z) -1 Z𝑇 = 𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 , where 𝐴 † is the pseudo-inverse of 𝐴. Overall, we therefore get

min 𝑉 𝐿 square (𝑍𝑉 ) + 𝜆𝑅 square (𝑍𝑉 ) = 1 2𝑛 Tr ( 𝑌 𝑇 ( I + 1 𝑛𝜆 𝑋𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 𝑋 𝑇 ) -1 𝑌 )
Resulting objective is convex in 𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 where we recognize the normalized equivalence matrices of partitions as presented in Section 5.2. The optimization problem then reads minimize Tr ( 𝑌 𝑇 (

I + 1 𝑛𝜆 𝑋𝑀 𝑋 𝑇 ) -1 𝑌 )
subject to 𝑀 ∈ ℳ.

(6.5) in variable 𝑀 where ℳ = {𝑀 = 𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 , 𝑍 ∈ {0, 1} 𝑑×𝑄 , 𝑍1 = 1} is the set of normalized equivalence matrices for partitions of {1, . . . , 𝑑} into 𝑄 groups.

The resulting problem is still non-convex due to the combinatorial nature of the set of normalized equivalence matrices. However one can then relax the problem by optimizing on its convex hull as presented in Section 5.3.2.

Conditional gradient algorithm

We detail the linear minimization oracle used by Frank-Wolfe Algorithm 5 in this setting. Denote the objective function of problem (6.5) by

𝑓 (𝑀 ) Tr ( 𝑌 𝑇 ( I + 1 𝑛𝜆 𝑋𝑀 𝑋 𝑇 ) -1 𝑌 ) . (6.6)
Its gradient at a given 𝑀 ∈ ℳ is

∇𝑓 (𝑀 ) = - 1 2𝑛 2 𝜆 𝑋 𝑇 ( I + 1 𝑛𝜆 𝑋𝑀 𝑋 𝑇 ) -1 𝑌 𝑌 𝑇 ( I + 1 𝑛𝜆 𝑋𝑀 𝑋 𝑇 ) -1 𝑋. (6.7)
Observe that -∇𝑓 (𝑀 ) is a semi-definite positive matrix of squared root

𝑈 = 1 𝑛 √ 2𝜆 𝑋 𝑇 ( I + 1 𝑛𝜆 𝑋𝑀 𝑋 𝑇 ) -1 𝑌 ∈ R 𝑑×𝐾 .
The linear minimization oracle to minimize 𝑓 (𝑀 ) over the convex hull of the set of normalized equivalence matrices can then be computed as follows

argmin 𝑆∈hull(ℳ) ⟨𝑆, ∇𝑓 (𝑀 )⟩ 𝜗 1 = argmin 𝑆∈ℳ Tr(𝑆 𝑇 ∇𝑓 (𝑀 )) = argmin 𝑆∈ℳ -Tr(𝑆𝑈 𝑈 𝑇 ) = argmin 𝑆∈ℳ Tr((I -𝑆)𝑈 𝑈 𝑇 )) 𝜗 2 = argmin 𝑆∈ℳ ‖𝑈 -𝑆𝑈 ‖ 2 𝐹 . (6.8)
In 𝜗 1 , we used that ℳ is a set of atoms, so its convex hull is a polytope and linear minimization on it is equivalent to linear minimization on its vertices, i.e., ℳ. In 𝜗 2 , we used that normalized equivalence matrices are orthogonal projectors, so 𝐼 -𝑆 is also an orthogonal projector. Now we observe that (6.8) is a k-means problem as presented in Section 5.2.2.

Therefore solving a k-means problem on the rows of the squared root matrix 𝑈 of -∇𝑓 (𝑀 ) offers a solution to the linear minimization oracle. Crucially here if 𝐾 = 1 (i.e. for regression or binary classification), 𝑈 is a vector such that this reduces to a k-means problem in one dimension that can be solved exactly in polynomial time by dynamic programming [START_REF] Bellman | A note on cluster analysis and dynamic programming[END_REF][START_REF] Wang | Ckmeans. 1d. dp: optimal k-means clustering in one dimension by dynamic programming[END_REF]. Otherwise careful initialization as made in k-means++ [START_REF] Arthur | k-means++: The advantages of careful seeding[END_REF] offers logarithmic approximations to the problem.

A conditional gradient method can then be applied in our setting, so that we can solve a relaxed version of (6.5). Once done, it remains to provide an approximate feasible solution for the original constraints. Two projections of the relaxed version are possible: either finding the closest normalized equivalence matrix in Frobenius norm or computing the point that minimizes the gradient of the relaxed solution, i.e. computing its linear minimization oracle. In practice we chose second solution as it provided better results. Notice that the k-means operation used for the linear minimization provides not only a normalized equivalence matrix but also a corresponding assignment matrix 𝑍 that leads to the optimal coefficients that minimize 𝐿 square (𝑍𝑉 ) + 𝜆𝑅 square (𝑍𝑉 ) as

𝑉 (𝑍) = (𝑍 𝑇 𝑋 𝑇 𝑋𝑍 + 𝜆𝑛𝑍 𝑇 𝑍) † 𝑍 𝑇 𝑋 𝑇 𝑌.
(6.9)

To summarize, our convex relaxation for regression with grouped features is presented in Algorithm 6. We denote by 𝑍 = k-means(𝑈, 𝑄) an assignment matrix solution of the k-means problem that cluster rows of 𝑈 in 𝑄 groups.

Inputs: Data (𝑋, 𝑌 ), desired number of groups 𝑄, target precision 𝜀, regularization parameter 𝜆 ≥ 0 Initialize 𝑀 0 ∈ ℳ for 𝑡 = 0, . . . do Compute -∇𝑓 (𝑀 𝑡 ) in (6.7) and its squared root 𝑈 Get linear minimization oracle by computing

𝑍 𝑡 = k-means(𝑈, 𝑄) 𝑆 𝑡 = 𝑍 𝑡 (𝑍 𝑇 𝑡 𝑍 𝑡 ) † 𝑍 𝑇 𝑡 if Tr ( (𝑀 𝑡 -𝑆 𝑡 ) 𝑇 ∇𝑓 (𝑀 𝑡 ) ) ≤ 𝜀 then Stop end if Set 𝑀 𝑡+1 = 𝑀 𝑡 + 1 𝑡+2 (𝑆 𝑡 -𝑀 𝑡 ) end for Ẑ = 𝑍 𝑡 V = 𝑉 ( Ẑ) in (6.9) Output: Ŵ = Ẑ V

Computational complexity

We briefly examine the complexity of Algorithm 6. For regression the k-means operation has a complexity of 𝑂(𝑑 2 𝑄) operations to get an exact solution by dynamic programming. For classification, k-means++ initialization costs 𝑂(𝑄 2 𝑑) operations and standard alternating minimization approximates the k-means operation at a cost of 𝑂(𝑇 𝑄𝑑), where 𝑇 is the number of alternating steps, that is generally small.

The squared root computation is directly given by the computation of ∇𝑓 (𝑀 ). However, this gradient itself requires the inversion of a matrix of size 𝑑 × 𝑑. This burdens its implementation for big data sets. Yet, a few iterations of this algorithm can give a good initialization for non-convex approaches of the problem.

Iterative Hard Clustering

Previous part gave an overview of the tightest convex relation in the simple case of a squared loss. Convexity ensures convergence of the Algorithm 6, at least for regression or binary classification. However, as noticed, this scheme may not scale for big data. Moreover it does not transpose to losses different than the squared one. We therefore propose to tackle directly the non-convex problem minimize 𝐿(𝑊 ) + 𝜆𝑅(𝑊 ) subject to 𝑊 = 𝑍𝑉, 𝑍 ∈ {0, 1} 𝑑×𝑄 , 𝑍1 = 1 (P) in variables 𝑊 ∈ R 𝑑×𝐾 , 𝑉 ∈ R 𝑄×𝐾 and 𝑍, where 𝜆 ≥ 0 is a regularization parameter. We use a projected gradient scheme that amounts to iteratively cluster features at each gradient step. This transposes the Iterative Hard Thresholding [START_REF] Blumensath | Iterative hard thresholding for compressed sensing[END_REF] algorithm, studied in compressed sensing, to the problem of grouping features. It offers a scalable solution for various losses as shown in the numerical experiments. Here we detail its implementation and analyze its performance from a compressed sensing point of view in the next section.

Projected gradient descent

The algorithm relies on the fact that projecting a point 𝑊 on the feasible set, that reads minimize (6.10) in variables 𝑣 1 , . . . , 𝑣 𝑄 ∈ R 𝐾 that are the centroids of the clusters and 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) a partition of {1, . . . , 𝑑}. As noticed in previous section, this k-means problem can be solved approximately with k-means++ if 𝐾 > 1 or exactly in polynomial time if 𝐾 = 1. Given a matrix 𝑊 , whose rows we want to cluster in 𝑄 groups, we denote by [𝑍, 𝑉 ] = k-means(𝑊, 𝑄) respectively the assignment matrix and the matrix of centroids output by a clustering algorithm. A projected gradient scheme for problem (P) is described in Algorithm 7 and its implementations details are provided in next section.

‖𝑊 -𝑍𝑉 ‖ 2 𝐹 subject to 𝑍 ∈ {0, 1} 𝑑×𝑄 , 𝑍1 = 1 in variable 𝑉 ∈ R 𝑄×𝐾 and 𝑍 is a clustering problem minimize 𝑄 ∑ 𝑞=1 ∑ 𝑖∈𝑔𝑞 ‖𝑤 𝑖 -𝑣 𝑞 ‖ 2 2 ,

Algorithm 7 Iterative Hard Clustering

Inputs: Data (𝑋, 𝑌 ), desired number of groups 𝑄, regularization parameter 𝜆 ≥ 0,

step size 𝛾 𝑡 Initialize 𝑊 0 ∈ R 𝑑×𝐾 for t = 1,. . . ,T do 𝑊 𝑡+1/2 = 𝑊 𝑡 -𝛾 𝑡 (∇𝐿(𝑊 𝑡 ) + 𝜆∇𝑅(𝑊 𝑡 )) [𝑍 𝑡+1 , 𝑉 𝑡+1 ] = k-means(𝑊 𝑡+1/2 , 𝑄) 𝑊 𝑡+1 = 𝑍 𝑡+1 𝑉 𝑡+1 end for Output: Ŵ = 𝑊 𝑇

Detailed implementation

In practice, we stop the algorithm when change in objective values of (P) are below some prescribed threshold 𝜀. We use a backtracking line search on the stepsize 𝛾 𝑡 that guarantees decreasing of the objective. At each iteration if

W𝑡+1 = k-means (𝑊 𝑡 -𝛾 𝑡 (∇𝐿(𝑊 𝑡 ) + 𝜆∇𝑅(𝑊 𝑡 )), 𝑄)
decreases the objective value we keep it and we increase the stepsize by a constant factor 𝛾 𝑡+1 = 𝛼𝛾 𝑡 with 𝛼 > 1. If W𝑡+1 increases the objective value we decrease the stepsize by a constant factor 𝛾 𝑡 ← 𝛽𝛾 𝑡 , with 𝛽 < 1, compute new W𝑡+1 and iterate this operation until W𝑡+1 decreases the objective value or the stepsize reaches the stopping value 𝜀 used as a stopping criterion on the objective values. We observed better results with this line search than with constant stepsize, in particular when the number of samples is small. Using this strategy, we observed convergence of the projected gradient algorithm in less than 100 iterations which makes it highly scalable. The complexity of its core operations amounts indeed to k-means operations whose complexities were given in Section 6.2.3.

Recovery performance of Iterative Hard Clustering

We now analyze convergence of the Iterative Hard Clustering scheme to retrieve the true regressor 𝑤 * in the regression problem. To this end we first detail the problem in terms of partitions.

Combinatorial penalty for grouping features

Several works developed tools to encode structural information in optimization problems such as sparsity inducing norms. [START_REF] Bach | Learning with submodular functions: A convex optimization perspective[END_REF] show that these regularizers can generally be seen as convex extensions of combinatorial functions. A given vector 𝑤 defines indeed by its support Supp(𝑤) = {𝑖 ∈ {1, . . . , 𝑑}, 𝑤 𝑖 ̸ = 0} a set that can be constrained to satisfy some combinatorial properties by using submodular functions. For example, classical sparsity enforces the cardinality of this support to be small in order to select a few parameters. On a regression problem (6.1), this reads minimize 𝐿(𝑤) + 𝜆𝑅(𝑤) subject to Card(Supp(𝑤)) ≤ 𝑠, in variable 𝑤 ∈ R 𝑑 , where 𝑠 is the desired sparsity. In our context, we do not use the set defined by the support of a vector, but the partition given by its level sets, that read Part(𝑤) = {𝑔 ⊂ {1, . . . , 𝑑} : (𝑖, 𝑗) ∈ 𝑔 × 𝑔, iff 𝑤 𝑖 = 𝑤 𝑗 }.

Denoting Card(𝐺) the number of the (non-empty) groups of a partition 𝐺, linear regression enforcing 𝑄 group of features (6.2) then reads minimize 𝐿(𝑤) + 𝜆𝑅(𝑤) subject to Card(Part(𝑤)) ≤ 𝑄, (6.11) in variable 𝑤 ∈ R 𝑑 . Assignment matrices introduced in Section 6.1 encode then the possible partitions into at most 𝑄 groups. We detail the geometrical interpretations of assignment and normalized equivalence matrices in Appendix 6.A. Identifying the underlying combinatorial constraint helps then to describe the geometry of the feasible set. It can also be used to derive a norm as discussed in Appendix 6.B.

Geometry induced by partitions

Denote 𝒫 the set of partitions of {1, . . . , 𝑑} whose definition is recalled below.

Definition 6.4.1. Partitions A collection 𝐺 of subsets of {1, . . . , 𝑑} is a partition of {1, . . . , 𝑑} if for any 𝑔, 𝑔 ′ ∈ 𝐺×𝐺, 𝑔 ̸ = 𝑔 ′ implies 𝑔∩𝑔 ′ = ∅ and if ⋃ 𝑔∈𝐺 𝑔 = {1, . . . , 𝑑}.

Denote from now on 𝒫 the set of partitions of {1, . . . , 𝑑}. Pair of partitions can then be compared as follows.

Definition 6.4.2. Sup-and sub-partitions Let 𝐺, 𝐺 ′ ∈ 𝒫 be two partitions. 𝐺 is a sup-partition of 𝐺 ′ (or 𝐺 ′ is a sub-partition of 𝐺), denoted

𝐺 ⪰ 𝐺 ′ , if for any 𝑔 ′ ∈ 𝐺 ′ there exists 𝑔 ∈ 𝐺 such that 𝑔 ′ ⊂ 𝑔 , or equivalently if any 𝑔 ∈ 𝐺 is a union of groups 𝑔 ′ of 𝐺 ′ .
Relation ⪰ is transitive, reflexive and anti-symmetric, such that it is a partial order on the set of partitions. Notice that the number of (non-empty) groups of partitions, Card(•), decreases with the partial order ⪰.

Following proposition highlights the geometry induced by a single partition of the features.

Proposition 6.4.3. Any partition 𝐺 ∈ 𝒫 defines a linear subspace

𝐸 𝐺 = {𝑤 ∈ R 𝑑 : Part(𝑤) ⪰ 𝐺} (6.12)
of vectors 𝑤 whose level sets can be partitioned by the groups of 𝐺. For any partitions

𝐺, 𝐺 ′ ∈ 𝒫, if 𝐺 ⪰ 𝐺 ′ then 𝐸 𝐺 ⊂ 𝐸 𝐺 ′ .
Proof. Given a partition 𝐺 ∈ 𝒫 and a vector 𝑤 ∈ R 𝑑 , 𝐺 is a sub-partition of Part(𝑤), i.e. Part(𝑤) ⪰ 𝐺, if and only if the groups of 𝐺 are subsets of equal coefficients of 𝑤, or equivalently if level sets of 𝑤 can be partitioned by groups of 𝐺. Now, if, for some 𝑤 1 , 𝑤 2 ∈ R 𝑑 , groups of 𝐺 are subsets of equal coefficients of both 𝑤 1 and 𝑤 2 , they will also be subset of equal coefficients of any linear combination of 𝑤 1 , 𝑤 2 . Therefore 𝐸 𝐺 is a linear subspace. Second statement follows from the transitivity of ⪰.

Since 𝑤 ∈ 𝐸 Part(𝑤) , the feasible set for the regression problem (6.11) enforcing 𝑄 groups of features is then a union of subspaces:

{𝑤 ∈ R 𝑑 : Card(Part(𝑤)) ≤ 𝑄} = ⋃ 𝐺∈𝒫 : Card(𝐺)≤𝑄 𝐸 𝐺 = ⋃ 𝐺∈𝒫 : Card(𝐺)=𝑄 𝐸 𝐺 .
Second equality comes from the fact that if a partition 𝐺 ∈ 𝒫 has strictly less than 𝑄 groups, i.e., Card(𝐺) < 𝑄, some of its groups can always be split to form a new partition 𝐺 ′ such that 𝐺 ⪰ 𝐺 ′ , Card(𝐺 ′ ) = 𝑄 and 𝐸 𝐺 ⊂ 𝐸 𝐺 ′ . Therefore it is sufficient to consider subspaces generated by partitions into exactly 𝑄 groups, whose set is denoted 𝒫 𝑄 = {𝐺 ∈ 𝒫 : Card(𝐺) = 𝑄}.

Convergence analysis of Iterative Hard Clustering

We now analyze the convergence of the projected gradient algorithm applied to a regression problem enforcing 𝑄 groups of features. We use a squared loss and no regularization. Therefore our problem reads minimize

1 2𝑛 ‖𝑋𝑤 -𝑦‖ 2 2 subject to Card(Part(𝑤)) ≤ 𝑄 (6.13)
in 𝑤 ∈ R 𝑑 , where 𝑋 = (𝑥 1 , . . . , 𝑥 𝑛 ) 𝑇 ∈ R 𝑛×𝑑 is the matrix of data points and 𝑦 = (𝑦 1 , . . . , 𝑦 𝑛 ) ∈ R 𝑛 is the vector of observations. For the analysis, we use a constant step size 𝛾 𝑡 = 1 and initialize the algorithm with 𝑤 0 = 0. We assume that the observations 𝑦 are generated by a linear model whose coefficients 𝑤 * satisfy the constraints above, up to additive noise, that is

𝑦 = 𝑋𝑤 * + 𝜂,
where 𝜂 ∼ 𝒩 (0, 𝜎 2 ) and Card(Part(𝑤 * )) ≤ 𝑄. Hence we analyze the performance of the algorithm to recover 𝑤 * and the partition Part(𝑤 * ) of the features. If it were not the constraints, a gradient descent applied to (6.13) would act as a fixed point algorithm whose contraction factor depends on the singular values of the Hessian 𝑋 𝑇 𝑋 of the problem. Here we will show that the projected gradient scheme exhibits the same behavior, except that the contraction factor will depend on restricted singular values on small subspaces defined by partitions. These subspaces belong to the following collections

ℰ 1 = {𝐸 𝐺 : 𝐺 ∈ 𝒫 𝑄 } ℰ 2 = {𝐸 𝐺 1 + 𝐸 𝐺 2 : 𝐺 1 , 𝐺 2 ∈ 𝒫 𝑄 } ℰ 3 = {𝐸 𝐺 1 + 𝐸 𝐺 2 + 𝐸 𝐺 3 : 𝐺 1 , 𝐺 2 , 𝐺 3 ∈ 𝒫 𝑄 }. (6.14)
Throughout the rest of the section, for a given subspace 𝐸 of R 𝑑 , we denote 𝑃 𝐸 the orthogonal projector on 𝐸, 𝑈 𝐸 any orthonormal basis of 𝐸 and for a given matrix 𝑋 ∈ R 𝑛×𝑑 we denote 𝜎 min (𝑋𝑈 𝐸 / √ 𝑛) and 𝜎 max (𝑋𝑈 𝐸 / √ 𝑛) respectively the smallest and largest singular values of 𝑋𝑈 𝐸 / √ 𝑛, i.e. the smallest and largest restricted singular values of 𝑋/ √ 𝑛 on 𝐸.

The next proposition adapts the proof of Iterative Hard Thresholding in our context using that feasible set is a union of subspaces, Proposition 6.4.4. Iterative Hard Clustering Algorithm 7 with constant step size 𝛾 𝑡 = 1 and initialization 𝑤 0 = 0, applied to (6.13) outputs iterates 𝑤 𝑡 that converge to the original 𝑤 * as

‖𝑤 * -𝑤 𝑡 ‖ 2 ≤ 𝜌 𝑡 ‖𝑤 * ‖ 2 + 1 -𝜌 𝑡 1 -𝜌 𝜈‖𝜂‖ 2 ,
where

𝜌 = 6 max 𝐸∈ℰ 2 max(𝛿 𝐸 , 𝛿 3 𝐸 ), 𝜈 = 2/ √ 𝑛 max 𝐸∈ℰ 3 𝜎 max (𝑋𝑈 𝐸 / √ 𝑛)
and for any subspace 𝐸 of R 𝑑 , 𝛿 𝐸 is the smallest non-negative constant that satisfies

1 -𝛿 𝐸 ≤ 𝜎 min ( 𝑋𝑈 𝐸 / √ 𝑛 ) ≤ 𝜎 max ( 𝑋𝑈 𝐸 / √ 𝑛 ) ≤ 1 + 𝛿 𝐸 .
Proof. To describe Algorithm 7, we define for 𝑡 ≥ 0,

𝑤 𝑡+1/2 = 𝑤 𝑡 -𝛾 𝑡 ∇𝐿(𝑤 𝑡 ) = 𝑤 𝑡 - 1 𝑛 𝑋 𝑇 𝑋(𝑤 𝑡 -𝑤 * ) + 1 𝑛 𝑋 𝑇 𝜂 𝑤 𝑡+1 = argmin 𝑤∈R 𝑑 : Card(Part(𝑤))≤𝑄 ‖𝑤 -𝑤 𝑡+1/2 ‖ 2 2 ,
where 𝑤 𝑡+1 is given exactly by the solution of a k-means problem in one dimension.

The analysis of convergence relies on the characterization of the subspaces that contain 𝑤 * , 𝑤 𝑡 and 𝑤 𝑡+1 . We define therefore

𝐸 𝑡,* = 𝐸 Part(𝑤𝑡) + 𝐸 Part(𝑤*) 𝐸 𝑡+1,* = 𝐸 Part(𝑤 𝑡+1 ) + 𝐸 Part(𝑤*) 𝐸 𝑡,𝑡+1,* = 𝐸 Part(𝑤𝑡) + 𝐸 Part(𝑤 𝑡+1 ) + 𝐸 Part(𝑤*) ,
and the orthogonal projections on these set respectively 𝑃 𝑡,* , 𝑃 𝑡+1,* , 𝑃 𝑡,𝑡+1,* . Bound on the error can then be computed as follows:

‖𝑤 * -𝑤 𝑡+1 ‖ 2 = ‖𝑃 𝑡+1,* (𝑤 * -𝑤 𝑡+1 )‖ 2 ≤ ‖𝑃 𝑡+1,* (𝑤 * -𝑤 𝑡+1/2 )‖ 2 + ‖𝑃 𝑡+1,* (𝑤 𝑡+1/2 -𝑤 𝑡+1 )‖ 2 . (6.15)
In the second term, as Card(Part(𝑤 * )) ≤ 𝑄 and 𝑤 𝑡+1 = argmin

𝑤∈R 𝑑 : Card(Part(𝑤))≤𝑄 ‖𝑤 - 𝑤 𝑡+1/2 ‖ 2 2 , we have ‖𝑤 𝑡+1 -𝑤 𝑡+1/2 ‖ 2 2 ≤ ‖𝑤 * -𝑤 𝑡+1/2 ‖ 2 2 which is equivalent to ‖𝑃 𝑡+1,* (𝑤 𝑡+1 -𝑤 𝑡+1/2 )‖ 2 2 +‖(𝐼-𝑃 𝑡+1,* )𝑤 𝑡+1/2 ‖ 2 2 ≤ ‖𝑃 𝑡+1,* (𝑤 * -𝑤 𝑡+1/2 )‖ 2 2 +‖(𝐼-𝑃 𝑡+1,* )𝑤 𝑡+1/2 ‖ 2 2
and this last statement implies

‖𝑃 𝑡+1,* (𝑤 𝑡+1 -𝑤 𝑡+1/2 )‖ 2 ≤ ‖𝑃 𝑡+1,* (𝑤 * -𝑤 𝑡+1/2 )‖ 2 .
This means that we get from (6.15)

‖𝑤 * -𝑤 𝑡+1 ‖ 2 ≤ 2‖𝑃 𝑡+1,* (𝑤 * -𝑤 𝑡+1/2 )‖ 2 = 2‖𝑃 𝑡+1,* (𝑤 * -𝑤 𝑡 - 1 𝑛 𝑋 𝑇 𝑋(𝑤 * -𝑤 𝑡 ) - 1 𝑛 𝑋 𝑇 𝜂)‖ 2 ≤ 2‖𝑃 𝑡+1,* (𝐼 - 1 𝑛 𝑋 𝑇 𝑋)(𝑤 * -𝑤 𝑡 )‖ 2 + 2 𝑛 ‖𝑃 𝑡+1,* (𝑋 𝑇 𝜂)‖ 2 = 2‖𝑃 𝑡+1,* (𝐼 - 1 𝑛 𝑋 𝑇 𝑋)𝑃 𝑡,* (𝑤 * -𝑤 𝑡 )‖ 2 + 2 𝑛 ‖𝑃 𝑡+1,* (𝑋 𝑇 𝜂)‖ 2 ≤ 2‖𝑃 𝑡+1,* (𝐼 - 1 𝑛 𝑋 𝑇 𝑋)𝑃 𝑡,* ‖ 2 ‖𝑤 * -𝑤 𝑡 ‖ 2 + 2 𝑛 ‖𝑃 𝑡+1,* 𝑋 𝑇 ‖ 2 ‖𝜂‖ 2 . Now, assuming 2‖𝑃 𝑡+1,* (𝐼 - 1 𝑛 𝑋 𝑇 𝑋)𝑃 𝑡,* ‖ 2 ≤ 𝜌 (6.16) 2 𝑛 ‖𝑃 𝑡+1,* 𝑋 𝑇 ‖ 2 ≤ 𝜈 (6.17)
and developing the latter inequality over 𝑡, using that 𝑤 0 = 0, we get

‖𝑤 * -𝑤 𝑡 ‖ 2 ≤ 𝜌 𝑡 ‖𝑤 * ‖ 2 + 1 -𝜌 𝑡 1 -𝜌 𝜈‖𝜂‖ 2 .
Bounds 𝜌 and 𝜈 can then be given by restricted singular values of 𝑋. For 𝜈 in (6.17), we have

‖𝑃 𝑡+1,* 𝑋 𝑇 ‖ 2 = ‖𝑋𝑃 𝑡+1,* ‖ 2 𝜗 ≤ max 𝐸∈ℰ 2 ‖𝑋𝑃 𝐸 ‖ 2 = max 𝐸∈ℰ 2 𝜎 max (𝑋𝑈 𝐸 ).
For 𝜗, as noticed in previous section, if for example Card(Part(𝑤 * )) < 𝑄, there always exists 𝐺 ∈ 𝒫 such that Part(𝑤 * ) ⪰ 𝐺, Card(𝐺) = 𝑄 and so 𝐸 Part(𝑤*) ⊂ 𝐸 𝐺 . Therefore there exists 𝐹 𝑡+1,* that contain 𝐸 𝑡+1,* and belong to ℰ 2 , such that we can restrict our attention to restricted singular values on subspaces in ℰ 2 (defined from partitions in exactly 𝑄 groups).

For 𝜌 in (6.16), we have

‖𝑃 𝑡+1,* (𝐼 -𝑋 𝑇 𝑋)𝑃 𝑡,* ‖ 2 𝜗 1 ≤ ‖𝑃 𝑡,𝑡+1,* (𝐼 - 1 𝑛 𝑋 𝑇 𝑋)𝑃 𝑡,𝑡+1,* ‖ 2 𝜗 2 ≤ max 𝐸∈ℰ 3 ‖𝑃 𝐸 (𝐼 - 1 𝑛 𝑋 𝑇 𝑋)𝑃 𝐸 ‖ 2 = max 𝐸∈ℰ 3 ‖𝑈 𝐸 (𝐼 - 1 𝑛 𝑈 𝑇 𝐸 𝑋 𝑇 𝑋𝑈 𝐸 )𝑈 𝑇 𝐸 ‖ 2 = max 𝐸∈ℰ 3 ‖𝐼 - 1 𝑛 𝑈 𝑇 𝐸 𝑋 𝑇 𝑋𝑈 𝐸 ‖ 2 ,
where for a subspace 𝐸, 𝑈 𝐸 denotes any orthonormal basis of it. In 𝜗 1 we used that 𝐸 𝑡,𝑡+1,* contain 𝐸 𝑡,* and 𝐸 𝑡+1,* . In 𝜗 2 we use the same argument as for 𝜈 to restrict our attention to subspaces defined by partitions into exactly 𝑄 groups. Finally, for a subspace 𝐸 if 𝛿 𝐸 ≥ 0 satisfies

1 -𝛿 𝐸 ≤ 𝜎 min ( 𝑋𝑈 𝐸 / √ 𝑛 ) ≤ 𝜎 max ( 𝑋𝑈 𝐸 / √ 𝑛 ) ≤ 1 + 𝛿 𝐸 ,
then [Vershynin, 2010, Lemma 5.38] shows

‖𝐼 - 1 𝑛 𝑈 𝑇 𝐸 𝑋 𝑇 𝑋𝑈 𝐸 ‖ 2 ≤ 3 max{𝛿 𝐸 , 𝛿 2 𝐸 },
which concludes the proof by taking the maximum of 𝛿 𝐸 over ℰ 3 .

If the contraction factor is sufficient, convergence of the projected gradient scheme to the original vector is ensured up to a constant error of the order of the noise as the classical IHT algorithm does for sparse signals [START_REF] Blumensath | Iterative hard thresholding for compressed sensing[END_REF].

Recovery performance on random instances

We observe now that for isotropic independent sub-Gaussian data 𝑥 𝑖 the restricted singular values introduced in Proposition 6.4.4 depend on the number of subspaces that define partitions and their dimension. This proposition reformulates results of Vershynin [2010, Theorems 5.39, 5.65] in our context. Proposition 6.4.5. Let ℰ be a collection of subspaces of R 𝑑 of dimension at most 𝐷 and denote 𝑁 their number. If the samples are 𝑛 isotropic independent sub-gaussian random variables forming a design matrix 𝑋 = (𝑥 1 , . . . , 𝑥 𝑛 ) 𝑇 ∈ R 𝑛×𝑑 , then for any 𝐸 ∈ ℰ,

1 -𝛿 -𝜀 ≤ 𝜎 min ( 𝑋𝑈 𝐸 √ 𝑛 ) ≤ 𝜎 max ( 𝑋𝑈 𝐸 √ 𝑛 ) ≤ 1 + 𝛿 + 𝜀,
with probability larger than 1exp(-𝑐𝜀 2 𝑛), where 𝛿 = 𝐶 0 √ 𝐷 𝑛 + √ log(𝑁 ) 𝑐𝑛 and 𝐶 0 , 𝑐 depend only on the sub-gaussian norm of the 𝑥 𝑖 .

Proof. Let 𝐸 ∈ ℰ, denote 𝑈 𝐸 one of its orthonormal basis and 𝐷 𝐸 = dim(𝐸) ≤ 𝐷 its dimension. The rows of 𝑋𝑈 𝐸 are orthogonal projections of the rows of 𝑋 onto 𝐸, so they are still independent sub-gaussian isotropic random vectors. We can therefore apply [Vershynin, 2010, Theorem 5.39] on 𝑋𝑈 𝐸 ∈ R 𝑛×𝐷 𝐸 . Hence, for any 𝑠 ≥ 0, with probability at least 1 -2 exp(-𝑐𝑠 2 ), the smallest and largest singular values of 𝑋𝑈 𝐸 / √ 𝑛 are bounded as , (6.18) where 𝑐 and 𝐶 0 depend only on the sub-gaussian norm of the 𝑥 𝑖 . Now, by taking the union bound, (6.18) holds for any 𝐺 ∈ 𝒫 𝑄 with probability 1 -2𝑁 exp(-𝑐𝑠 2 ).

1 -𝐶 0 √ 𝑄 𝑛 - 𝑠 √ 𝑛 ≤ 𝜎 min ( 𝑋𝑈 𝐸 √ 𝑛 ) ≤ 𝜎 max ( 𝑋𝑈 𝐸 √ 𝑛 ) ≤ 1 + 𝐶 0 √ 𝑄 𝑛 + 𝑠 √ 𝑛
Taking 𝑠 = √ log(𝑁 ) 𝑐 + 𝜀 √ 𝑛, we get for all 𝐺 ∈ 𝒫 𝑄 ,

1 -𝛿 -𝜀 ≤ 𝜎 min ( 𝑋𝑈 𝐸 √ 𝑛 ) ≤ 𝜎 max ( 𝑋𝑈 𝐸 √ 𝑛 ) ≤ 1 + 𝛿 + 𝜀,
with probability at least 1 -2 exp(-𝑐𝜀 2 𝑛), where 𝛿 = 𝐶 0 √

𝑄 𝑛 + √ log(𝑁 ) 𝑐𝑛 .
To ensure approximate recovery of the projected gradient scheme in Proposition 6.4.4, one needs to control restricted singular values of 𝑋 on subspaces in ℰ 3 in order to ensure that the contraction factor 𝜌 is strictly less than one. Precisely, we need to ensure for that for any 𝐸 ∈ ℰ 3 , there exists 0 ≤ 𝛿 < 1/6 such that

1 -𝛿 ≤ 𝜎 min ( 𝑋𝑈 𝐸 √ 𝑛 ) ≤ 𝜎 max ( 𝑋𝑈 𝐸 √ 𝑛 ) ≤ 1 + 𝛿.
Denoting 𝐷 3 and 𝑁 3 respectively the largest dimension of the subspaces in ℰ 3 and the number of these subspaces, last proposition shows that when observations 𝑥 𝑖 are isotropic independent sub-gaussian, their number 𝑛 must therefore satisfy

𝐶 0 √ 𝐷 3 𝑛 < 1 6 and √ log(𝑁 3 ) 𝑐𝑛 < 1 6
which is roughly 𝑛 = Ω(𝐷 3 ) and 𝑛 = Ω(log(𝑁 3 )) (6.19)

The first condition in (6.19) means that subspaces must be low-dimensional, in our case 𝐷 3 = 3𝑄, and we naturally want the number of groups to be small. The second condition in (6.19) means that the structure (partitioning here) is restrictive enough, i.e., that the number of possible configurations, 𝑁 3 , is small enough.

To compute 𝑁 3 , denote 𝑁 𝑄 the number of partitions in exactly 𝑄 groups such that 𝑁 3 = ( 𝑁 𝑄

3

) . The number of partitions into 𝑄 groups is then given by the the Stirling number of second kind 𝑁 𝑄 = { 𝑑 𝑄 } , that can be bounded as

𝑄 𝑑-𝑄 ≤ { 𝑑 𝑄 } ≤ 1 2 (𝑒𝑑/𝑄) 𝑄 𝑄 𝑑-𝑄 . (6.20)
Using standard bounds on the binomial coefficients this means

𝑁 3 ≥ ( 𝑁 𝑄 3 ) 3 ≥ 𝑄 3𝑑-3𝑄 27 .
Therefore although the intrinsic dimension of our variables is of order 3𝑄, the number of subspaces 𝑁 3 is such that we need roughly 𝑛 ≥ 3𝑑 log(𝑄) observations, i.e., approximately as many samples as features, so the grouping structure is not specific enough to reduce the number of samples required by a projected gradient scheme to converge. On the other hand, given this many samples, the algorithm provably converges to the original 𝑤 * , which helps interpretation. As a comparison, classical sparse recovery problems have the same structure [START_REF] Rao | Signal recovery in unions of subspaces with applications to compressive imaging[END_REF], as 𝑠-sparse vectors for instance can be described as {𝑤 = 𝑍𝑣, 𝑍 ∈ {0, 1} 𝑑×𝑠 , 𝑍 𝑇 1 = 1} and so are part of a "union of subspaces". However in the case of sparse vectors the number of subspaces grows as 𝑑 𝑠 which means recovery requires much less samples than features.

Sparse and grouped linear models

Projected gradient schemes are simple but scalable algorithms to tackle constrained structures of linear models. It has been developed for sparsity through the Iterative Hard Thresholding algorithm [START_REF] Blumensath | Iterative hard thresholding for compressed sensing[END_REF], we presented its version to group features in Section 6.3, we now extend it to both select 𝑠 features and group them in 𝑄 groups. Using the notations introduced in Section 6.4.1, a regression problem that enforces predictors to have at most 𝑠 non-zeros coefficients clustered in at most 𝑄 groups reads minimize 𝐿(𝑤) + 𝜆𝑅(𝑤) subject to Card(Supp(𝑤)) ≤ 𝑠, Card(Part(𝑤)) ≤ 𝑄 + 1 (6.21) in variable 𝑤 ∈ R 𝑑 , where 𝐿 and 𝑅 are respectively the loss and the regularizer of the prediction problem as introduced in Section 6.1 and 𝜆 ≥ 0 is a regularization parameter. Naturally we take 𝑄 ≤ 𝑠 as one cannot cluster 𝑠 features in more than 𝑠 groups.

A projected gradient for this problem requires essentially an efficient algorithm for the projection step. To this end, we develop a new dynamic program to get the projection on 𝑠-sparse vectors whose non-zero coefficients form 𝑄 groups of equal coefficients. Analysis of the recovery performance of this scheme will then follow from the previous study for grouped vectors.

Projection on 𝑠-sparse 𝑄-grouped vectors

Given two projectors on different subspaces, the projection on their intersection is not simply given by the composition of the projectors. Similarly here one easily shows that projection on sparse and clustered vectors is not given by a thresholding step followed by a clustering one, finer analysis is necessary as detailed below.

Formulation of the problem

A feasible point 𝑤 ∈ R 𝑑 for problem (6.21) is described by the partition of its coordinates 𝐺 = {𝑔 0 , . . . , 𝑔 𝑄 𝐺 } in groups of equal coefficients, where 𝑔 0 is the group of zero coefficients, and 𝑣 1 , . . . , 𝑣 𝑄 𝐺 the possible values of the non-zero coefficients. 𝑄 𝐺 = Card(𝐺)-1 ≥ 0 denotes here the number of (non-empty) groups of a partition 𝐺 ∈ 𝒫. This problem amounts to select a number 𝑠 ′ ≤ 𝑠 of features and cluster them in a number 𝑄 ′ ≤ 𝑄 groups whose barycenters have maximal magnitude for the objective in (6.23). The objective can then be split into positive and negative barycenters to treat each resulting problem independently and then find the best balance between both parts.

Dynamic programing

To solve problem (6.23), observe first that the objective is clearly increasing with the number of groups, as it allows more degrees of freedom to approximate 𝑥. Furthermore if the number 𝑠 ′ of selected features is fixed, the number of groups cannot exceed it, i.e. 𝑄 ′ ≤ 𝑠 ′ , and it can therefore be set at min(𝑠 ′ , 𝑄).

A solution of (6.23) that selects 𝑠 ′ ≤ 𝑠 features is then composed of a partition of 𝑗 points into 𝑞 groups that define positive barycenters, and a partition of the 𝑠 ′ -𝑗 remaining points into min(𝑠 ′ , 𝑄) -𝑞 groups that define negative centers. We therefore tackle (6.23) by searching for the best parameters 𝑠 ′ , 𝑗, 𝑞 that balance optimally the objective into positive and negative barycenters.

To this end, we define 𝑓 + (𝑗, 𝑞) the optimal value of (6.23) when picking 𝑗 points clustered in 𝑞 groups of positive barycenters, i.e. the solution of the problem maximize

∑ 𝑞 𝑝=1 𝑠 𝑝 𝜇 2 𝑝 subject to 𝜇 𝑝 = 1 𝑠𝑝 ∑ 𝑖∈𝑔𝑝 𝑥 𝑖 > 0 Card ( ⋃ 𝑞 𝑝=1 𝑔 𝑝 ) = 𝑗, (𝑃 + (𝑗, 𝑞))
in disjoint groups 𝑔 1 , . . . , 𝑔 𝑞 ⊂ {1, . . . , 𝑑}. This problem is not always feasible, as it may not be possible to find 𝑞 clusters of positive barycenters with 𝑗 points. In that case we denote its solution 𝑓 + (𝑗, 𝑞) = +∞. We define similarly 𝑓 -(𝑗, 𝑞) the optimal value of (6.23) when picking 𝑗 points clustered in 𝑞 groups forming only negative barycenters. The best balance between the two, which solves (6.23), is then given by solving: maximize 𝑓 + (𝑗, 𝑞) (6.24) in variables 𝑠 ′ , 𝑗 and 𝑞.

+ 𝑓 -(𝑠 ′ -𝑗, 𝑄 ′ -𝑞) subject to 0 ≤ 𝑗 ≤ 𝑠 ′ , 0 ≤ 𝑞 ≤ 𝑄 ′ , 0 ≤ 𝑠 ′ ≤ 𝑠, 𝑄 ′ = min(𝑠 ′ , 𝑄),
It remains to compute 𝑓 + and 𝑓 -efficiently. We present our approach for 𝑓 + that transposes to 𝑓 -. Let 𝑆 + ⊂ {1, . . . , 𝑑} be the optimal subset of indexes taken for (𝑃 + (𝑗, 𝑞)) and 𝑖 ∈ 𝑆 + . If there exists 𝑗 ∈ {1, . . . , 𝑑} ∖ 𝑆 + such that 𝑥 𝑗 ≥ 𝑥 𝑖 , then swapping 𝑗 and 𝑖 would increase the magnitude of the barycenter of the group that 𝑖 belongs to and so the objective. Therefore (𝑃 + (𝑗, 𝑞)) amounts to a partitioning problem on the 𝑗 largest values of 𝑥. From now on, assume coefficients of 𝑥 to be in decreasing order 𝑥 1 ≥ . . . ≥ 𝑥 𝑑 . For (𝑃 + (𝑗, 𝑞)) a feasible problem, denote 𝑔 1 , . . . , 𝑔 𝑞 the optimal partition of {1, . . . , 𝑗} whose corresponding barycenters are in decreasing order. Let 𝑖 be the index of the largest coefficient of 𝑥 in 𝑔 𝑞 , then necessarily 𝑔 1 , . . . , 𝑔 𝑞-1 is optimal to solve (𝑃 + (𝑖-1, 𝑞 -1)). 𝑓 + can then be computed recursively as 𝑓 + (𝑗, 𝑞) = max 𝑓 + is initialized as a grid of 𝑘 + 1 and 𝑄 + 1 columns such that 𝑓 + (0, 𝑞) = 0 for any 𝑞, 𝑓 + (𝑗, 0) = 0 and 𝑓 + (𝑗, 1) = 𝑗𝜇(𝑥 1 , . . . , 𝑥 𝑗 ) 2 for any 𝑗 ≥ 1. 𝐼 + and 𝑣 + are initialized by 𝐼 + (𝑗, 1) = 1 and 𝜇 + (𝑗, 1) = 𝜇(𝑥 1 , . . . , 𝑥 𝑗 ).

Each dynamic program needs only to build the best partitions for the 𝑠 smallest or largest partitions so they cost 𝑂(𝑠 2 𝑄) elementary operations. The grid search and the backtrack cost respectively 𝑂(𝑠 2 𝑄) and 𝑂(𝑄) elementary operations. Overall, the complexity of the projection does not exceed 𝑂(𝑠 2 𝑄).

Recovery performance

Analysis of recovery performance of the projected gradient for sparse clustered vectors follows the one provided in Section 6.4. Our problem is to recover an original vector 𝑤 * such that Card(Supp(𝑤 * )) ≤ 𝑠 and Card(Part(𝑤 * )) ≤ 𝑄+1 that generates 𝑛 noisy observations 𝑦 𝑖 from data points 𝑥 𝑖 as

𝑦 = 𝑋𝑤 * + 𝜂,
where 𝜂 ∼ 𝒩 (0, 𝜎 2 ), where 𝑋 = (𝑥 1 , . . . , 𝑥 𝑛 ) 𝑇 ∈ R 𝑛×𝑑 is the matrix of data points and 𝑦 = (𝑦 1 , . . . , 𝑦 𝑛 ) ∈ R 𝑛 is the vector of observations. To this end we attempt to solve a regression problem enforcing 𝑄 groups of 𝑠 features with a squared loss and no regularization, which reads (6.26) As in Section 6.4, we use a projected gradient scheme with constant step size 𝛾 𝑡 = 1 and initialized at 𝑤 0 = 0, the only difference is the projection step that is given here by the dynamic program presented in last section.

First we detail the geometry of the feasible set of (6.21). A given subset 𝑆 ⊂ {1, . . . , 𝑑} defines a linear subspace

𝐸 𝑆 = {𝑤 ∈ R 𝑑 : Supp(𝑤) ⊂ 𝑆}.
By combining a subset 𝑆 ∈ {1, . . . , 𝑑} with a partition 𝐺 ∈ 𝒫 we get a linear subspace

𝐸 𝑆,𝐺 = {𝑤 ∈ R 𝑑 : Supp(𝑤) ⊂ 𝑆, Part(𝑤) ⪰ 𝐺} = 𝐸 𝑆 ∩ 𝐸 𝐺 .
Vectors in 𝐸 𝑆,𝐺 have at most Card(𝐺) -1 different non-zero coefficients such that dim(𝐸 𝑆,𝐺 ) = Card(𝐺) -1. The feasible set of (6.21) is then a union of subspaces, 𝐸 𝑆,𝐺 .

Analysis of convergence made in Proposition 6.4.4 for the clustered case relies only on the fact that the feasible set is a union of subspaces and that the projection on it can be computed exactly, so it applies also here. However the contraction factor will now depend on restricted singular values of the data on a smaller collection of subspaces. Precisely, define Ẽ = {𝐸 𝑆,𝐺 : 𝑆 ⊂ {1, . . . , 𝑑}, 𝐺 ∈ 𝒫, Card(𝑆) = 𝑠, Card(𝐺) = 𝑄 + 1} Ẽ3 = {𝐸 1 + 𝐸 2 + 𝐸 3 : 𝐸 1 , 𝐸 2 , 𝐸 3 ∈ Ẽ}.

The contraction factor depends then on the restricted singular values of the matrix 𝑋 on subspaces belonging to Ẽ3 . Since dim(𝐸 𝑆,𝐺 ) = Card(𝐺) -1, subspaces in Ẽ3 have a dimension at most 3𝑄. Denoting 𝑁 and 𝑁 3 the cardinality of respectively Ẽ and Ẽ3 , we have 𝑁

) 𝑠 1 2 ( 𝑒 𝑄 ) 𝑄 𝑠 𝑄 𝑄 𝑠-𝑄 ≤ 𝑑 𝑠 𝑠 𝑄 𝑄 𝑠-𝑄
and so

𝑁 3 ≤ ( 𝑒𝑁 3 ) 3 ≤ 𝑁 3 ≤ (𝑑 𝑠 𝑠 𝑄 𝑄 𝑠-𝑄 ) 3
Propositions 6.4.4 and 6.4.5 adapted in this case thus predict that the number of observations must satisfy 𝑛 = 𝑂(𝑠 log 𝑑 + 𝑄 log(𝑠) + (𝑠 -𝑄) log(𝑄))

for a projected gradient scheme to recover approximately 𝑤 * . It produces 𝑄 + 1 cluster of features, one being a cluster of zero features, reducing dimensionality, while needing roughly as many samples as non-zero features.

Numerical experiments

We now test our methods, first on artificial datasets to check their robustness to noisy data, then on real data extracted from movie reviews.

Synthetic dataset

We test the robustness of our algorithms for an increasing number of training samples or level of noise in the labels. We generate a linear model in dimension 𝑑 = 100 with a vector 𝑤 * ∈ R 𝑑 that has only 𝑄 = 5 different values uniformly distributed around 0. We sample 𝑛 Gaussian random points 𝑥 𝑖 with noisy observations 𝑦 𝑖 = 𝑤 𝑇 𝑥 𝑖 + 𝜂, where 𝜂 ∼ 𝒩 (0, 𝜎 2 ). We vary the number of samples 𝑛 or the level of noise 𝜎 and measure ‖𝑤 * -ŵ‖ 2 , the 𝑙 2 norm of the difference between the true vector of weights 𝑤 * and the estimated ones ŵ.

In Table 6.1 and 6.2, we study the performance of our model with a squared loss and regularized by the Euclidean norm. We solve it with the conditional gradient algorithm (CG),the Iterative Hard Clustering algorithm (IHC) (initialized with the solution of Least Square followed by k-means) and the conditional gradient followed by IHC (CGIHC). We compare the proposed algorithms to Least Squares regularized by the squared norm (LS), Least Squares regularized by the squared norm followed by K-means on the weights (using associated centroids as predictors) (LSK) and OSCAR [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with oscar[END_REF]. For OSCAR we used a submodular approach [START_REF] Bach | Optimization with sparsity-inducing penalties[END_REF] to compute the corresponding proximal algorithm, which makes it scalable. "Oracle" refers to the Least Square solution given the true assignments of features and can be seen as the best achievable error rate. When varying the number of samples, noise on labels is set to 𝜎 = 0.5 and when varying level of noise 𝜎 number of samples is set to 𝑛 = 150. Regularization parameters of the models were all crossvalidated using a logarithmic grid. Results were averaged over 50 experiments and figures after the ± sign correspond to one standard deviation. , the 𝑙 2 norm of the difference between the true vector of weights 𝑤 * and the estimated ones ŵ along level of noise 𝜎.

We observe that both IHC and CGIHC give significantly better results than other methods and even reach the performance of the Oracle for 𝑛 > 𝑑 and for small 𝜎, while for 𝑛 ≤ 𝑑 results are in the same range.

6.6.2 Predicting ratings from reviews using groups of words.

We perform "sentiment" analysis of newspaper movie reviews. We use the publicly available dataset introduced by Pang and [START_REF] Pang | Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales[END_REF] which contains movie reviews paired with star ratings. We treat it as a regression problem, taking responses for 𝑦 in (0, 1) and word frequencies as covariates. The corpus contains 𝑛 = 5006 documents and we reduced the initial vocabulary to 𝑑 = 5623 words by eliminating stop words, rare words and words with small TF-IDF mean on whole corpus. We evaluate our algorithms for regression with clustered features against standard regression approaches: Least-Squares (LS), Least-Squares followed by k-means on predictors (LSK), Oscar regularization (OS), LASSO and Iterative Hard Thresholding (IHT). We also tested our projected gradient with sparsity constraint, initialized by the solution of LSK (PGS) or by the solution of CG (CGPGS). Number of clusters, sparsity constraints and regularization parameters were 5-fold cross-validated using respectively grids going from 5 to 15, 𝑑/2 to 𝑑/5 and logarithmic grids. Cross validation and training were made on 80% on the dataset and tested on the remaining 20% it gave 𝑄 = 15 number of clusters and 𝑑/2 sparsity constraint for our algorithms. Results are reported in Table 6.3, figures after the ± sign correspond to one standard deviation when varying the training and test sets on 20 experiments.

All methods perform similarly except IHT and Lasso whose hypotheses do not seem appropriate for the problem. Our approaches have the benefit to reduce dimensionality from 5623 to 15 and provide meaningful cluster of words. The clusters with highest absolute weights are also the ones with smallest number of words, which confirms the intuition that only a few words are very discriminative. We illustrate this in Table 6.4, picking randomly words of the four clusters within which associated predictor weights 𝑣 𝑞 have largest magnitude.

LS

LSK IHC CG CGIHC OS 1.51±0.06 1.53±0.06 1.52±0.06 1.58±0.07 1.49±0.08 1.47±0.07 IHCS CGIHCS IHT LASSO 1.53±0.06 1.49±0.07 2.19±0.12 3.77±0.17 Table 6.3 -100 × mean square errors for predicting movie ratings associated with reviews.

Conclusion

We analyzed how prediction problems can be simplified by constraining groups of features, within each a common weight is assigned. Our formulation enables the development of a projected gradient scheme or a tight convex relaxation for squared loss. This approach can then be generalized for grouping samples or tasks in the following chapters. Our theoretical analysis of the projected gradient highlights that, even if dimension is reduced by grouping features, the treatment of this structure is much more complex than simple sparsity as it requires as many samples as features to ensure efficiency of a projected gradient scheme. Yet, by developing a dynamic program for sparse and grouped vectors we finally obtain a fast and scalable algorithm that drastically reduces the dimensionality of the problem while enjoying properties of sparse problems.

Notice that the combinatorial penalty we use, i.e. the number of non-empty groups of a partition, is not a submodular function on the lattice that forms partitions. Using submodular functions of partitions may lead to efficient algorithmic solutions to group features as it has been done for classical sparsity [START_REF] Bach | Learning with submodular functions: A convex optimization perspective[END_REF].

i.e. the function Ω(𝑤) = Card(Part(𝑤)).

Following [START_REF] Obozinski | Convex relaxation for combinatorial penalties[END_REF] ,we investigate how this combinatorial function can be incorporated in standard Euclidean regularization by finding the tightest convex homogeneous envelope of

Ω 2 = 1 2 ‖𝑤‖ 2 2 + 1 2 Card(Part(𝑤)).
Following proposition details its formulation Proposition 6.B.1. The tightest convex homogeneous envelope of

Ω 2 (𝑤) = 1 2 ‖𝑤‖ 2 2 + 1 2 Card(Part(𝑤)) is ‖𝑤‖ Ω 2 = inf (𝑥 𝑀 ) 𝑀 ∈ℳ 𝑥= ∑ 𝑀 ∈ℳ 𝑀 𝑥 𝑀 ∑ 𝑀 ∈ℳ Tr(𝑀 ) 1/2 ‖𝑀 𝑥 𝑀 ‖ 2 ,
where M defined in (6.27) is the set of normalized equivalence matrices of partitions of {1, . . . , 𝑑}.

‖𝑤‖ Ω 2 is a norm, whose dual norm is

‖𝑤‖ * Ω 2 = max 𝑀 ∈ℳ ‖𝑀 𝑥‖ 2 Tr(𝑀 ) 1/2 .
Proof. First we give an algebraic formulation of the combinatorial function Ω. Given a vector 𝑤 ∈ R 𝑑 , Part(𝑤) is the largest partition (in terms of ⪰ presented in Definition 6.4.2) in groups of equal coefficients of 𝑤. It defines therefore the smallest subspace (see Proposition 6.4.3) on which 𝑤 lies. Card(Part(𝑤)) is then the dimension of the smallest subspace defined from partitions, on which 𝑤 lies. Using normalized equivalence matrices that are orthogonal projections on these subspaces, as mentioned in Appendix 6.A, the combinatorial penalty Ω reads

Ω(𝑤) = Card(Part(𝑤)) = min 𝑀 ∈M 𝑀 𝑤=𝑤
Tr(𝑀 ). Now, following [START_REF] Obozinski | Convex relaxation for combinatorial penalties[END_REF], we begin by computing the homogenized version of Ω 2 defined as ℎ(𝑤) = inf 𝜆>0 Ω 2 (𝜆𝑤) 𝜆

, then we compute the Fenchel bi-conjugate of ℎ. We have

ℎ(𝑤) = inf 𝜆>0 1 2 ‖𝑤‖ 2 2 𝜆 + 1 2 Ω(𝑤)𝜆 -1 . = ‖𝑤‖ 2 Ω(𝑤) 1/2
Fenchel dual of ℎ reads then

ℎ * (𝑥) = sup 𝑤∈R 𝑑 𝑥 𝑇 𝑤 -‖𝑤‖ 2 Ω(𝑤) 1/2 = sup 𝑤∈R 𝑑 max 𝑀 ∈M 𝑀 𝑤=𝑤 𝑥 𝑇 𝑤 -‖𝑤‖ 2 Tr(𝑀 ) 1 2 = max 𝑀 ∈M sup 𝑤∈R 𝑑 𝑀 𝑤=𝑤 𝑥 𝑇 𝑤 -‖𝑤‖ 2 Tr(𝑀 ) 1 2 = max 𝑀 ∈M { 0 if ‖𝑀 𝑥‖ 2 ≤ Tr(𝑀 ) 1/2 +∞ otherwise = { 0 if max 𝑀 ∈ℳ ‖𝑀 𝑥‖ 2 Tr(𝑀 ) -1/2 ≤ 1 +∞ otherwise. Define ‖𝑤‖ * Ω 2 = max 𝑀 ∈M ‖𝑀 𝑥‖ 2 Tr(𝑀 ) -1/2 .
‖𝑤‖ * Ω 2 is convex as a finite maximum of convex functions, it is clearly homogeneous and as

I ∈ ℳ we have ‖𝑤‖ * Ω 2 =⇒ 𝑤 = 0. Hence ‖𝑤‖ * Ω 2 is a norm. ℎ * is then the indicator function of the unit norm ball of ‖𝑤‖ * Ω 2 . Fenchel bi-dual of ℎ is then ℎ ** (𝑤) = sup 𝑥∈R 𝑑 𝑤 𝑇 𝑥 -ℎ * (𝑥) = sup 𝑥∈R 𝑑 𝑤 𝑇 𝑥 - ∑ 𝑀 ∈M sup 𝜆 𝑀 ≥0 𝜆 𝑀 (‖𝑀 𝑥‖ 2 -Tr(𝑀 ) 1/2 ) = inf (𝜆 𝑀 ) 𝑀 ∈M , 𝜆 𝑀 ≥0 ∑ 𝑀 ∈M Tr(𝑀 ) 1/2 𝜆 𝑀 + sup 𝑥∈R 𝑑 𝑤 𝑇 𝑥 - ∑ 𝑀 ∈M 𝜆 𝑀 ‖𝑀 𝑥‖ 2 = inf (𝜆 𝑀 ) 𝑀 ∈M , 𝜆 𝑀 ≥0 ∑ 𝑀 ∈M Tr(𝑀 ) 1/2 𝜆 𝑀 + sup 𝑥∈R 𝑑 𝑤 𝑇 𝑥 - ∑ 𝑀 ∈M 𝜆 𝑀 sup ‖𝑎 𝑀 ‖ 2 ≤1 𝑥 𝑇 𝑀 𝑎 𝑀 = inf (𝜆 𝑀 ) 𝑀 ∈M , 𝜆 𝑀 ≥0 (𝑎 𝑀 ) 𝑀 ∈M , ‖𝑎 𝑀 ‖ 2 ≤1 𝑥= ∑ 𝑀 ∈M 𝜆 𝑀 𝑀 𝑎 𝑀 ∑ 𝑀 ∈M Tr(𝑀 ) 1/2 𝜆 𝑀 = inf (𝑥 𝑀 ) 𝑀 ∈M 𝑥= ∑ 𝑀 ∈M 𝑀 𝑥 𝑀 ∑ 𝑀 ∈M Tr(𝑀 ) 1/2 ‖𝑀 𝑥 𝑀 ‖ 2 = ‖𝑤‖ Ω 2 . Since ℎ * is the indicator function of the unit ball of ‖𝑤‖ * Ω 2 , ‖𝑤‖ Ω 2 is the dual norm of ‖𝑤‖ * Ω 2 .
Computed norm ‖𝑤‖ Ω 2 appears similar to the grouped norms defined for example by [START_REF] Jacob | Group lasso with overlap and graph lasso[END_REF]. However, to our knowledge, no algorithm can compute the norm or its proximal operator such that its utility in practice is unclear.

125

Chapter 7

Grouping samples for diverse predictions

Chapter Abstract

In prediction problems involving a large amount of training samples, reducing complexity of the task by clustering data points can improve performance and help interpretation. Here, rather than separating clustering and prediction steps, we study how both can be done simultaneously. We provide a formulation of our problem for regression or classification and present several algorithms to solve it. First, we develop a projected gradient scheme whose core iteration amounts to a k-means step. Then, we present a convex relaxation of the problem using conditional gradient, a.k.a. Frank-Wolfe algorithm, whose core iteration amounts again to a k-means operation. Numerical experiments illustrate the performance of our methods on synthetic data sets.

On the other side, more and more clustering techniques exist (classical k-means [START_REF] Gan | Data clustering: theory, algorithms, and applications[END_REF], k-means with kernels [START_REF] Schölkopf | Nonlinear component analysis as a kernel eigenvalue problem[END_REF], EM algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the em algorithm[END_REF], EM with Bregman divergence [START_REF] Banerjee | Clustering with bregman divergences[END_REF], spectral clustering [START_REF] Luxburg | A tutorial on spectral clustering[END_REF], DIFFRAC [START_REF] Bach | Diffrac: a discriminative and flexible framework for clustering[END_REF], Clusterpath [START_REF] Hocking | Clusterpath an algorithm for clustering using convex fusion penalties[END_REF] to cite a few) but no common measure of performance allow their comparison. Yet, clustering can be used as a first phase of prediction problem that it simplifies and helps. Performance of the prediction procedure can then be used to compare different clustering of the data.

In this chapter, we investigate how both clustering and prediction can be performed simultaneously. It either can be seen as a prediction problem relaxed to output more than one prediction per sample or as a clustering problem driven by the performance of a prediction task. It applies when data contain hidden information and diverse answers are required as in privacy learning (see [START_REF] Wainwright | Privacy aware learning[END_REF] and references herein), where personal information influences the predicted attributes but are not revealed.

Diversity learning was studied by [START_REF] Guzman-Rivera | Efficiently enforcing diversity in multi-output structured prediction[END_REF] who developed losses to enforce multiple outputs. Here we rather constraint classical learning problems by a clustering one. Mixing clustering and prediction problems is also performed in the mixture of experts framework [START_REF] Jordan | Hierarchical mixtures of experts and the em algorithm[END_REF]. However our setting differs form the latter as we assume that partition of the samples can only be revealed by the observations that have to be predicted and not by the features of the data. Our framework corresponds to the one studied by [START_REF] Zhang | Regression clustering[END_REF] or [START_REF] Bagirov | Nonsmooth nonconvex optimization approach to clusterwise linear regression problems[END_REF] to perform regression clustering with various models of regression.

We present our framework for regression and classification tasks, it incorporates the partitioning problem of the training samples in the minimization of an empirical loss. First, we propose algorithms to tackle the non-convex resulting problem : an alternate minimization between the partitioning problem and the empirical loss minimization and a projected gradient scheme whose projection step amounts to a clustering problem that can be solved approximatively by k-means++. Then, we present a convex relaxation in the case of a squared loss for which the combinatorial problem can be isolated. We use conditional gradient, a.k.a. Frank-Wolfe algorithm [START_REF] Frank | An algorithm for quadratic programming[END_REF][START_REF] Jaggi | Revisiting frank-wolfe: Projection-free sparse convex optimization[END_REF], whose core iteration amounts also to a k-means problem. Finally numerical experiments show the robustness of our method on synthetic data.

Problem Formulation

We first present our framework for linear regression tasks and then extend its application to classification.

Clustered Regression

We briefly present regression problems, more details are provided in Section 6.1.1. Given 𝑛 observations 𝑦 1 , . . . , 𝑦 𝑛 ∈ R from data points 𝑥 1 , . . . , 𝑥 𝑛 ∈ R 𝑑 , we are seek-ing for a vector 𝑤 that predicts linearly outputs from inputs. A loss function ℓ measures its accuracy error ℓ(𝑤 𝑇 𝑥, 𝑦) on a sample (𝑥, 𝑦) such as the squared loss ℓ square (𝑤 𝑇 𝑥, 𝑦) = 1 2 (𝑤 𝑇 𝑥 -𝑦) 2 . A candidate vector 𝑤 can be found by minimizing the empirical loss (7.3) in variables 𝑊 ∈ R 𝑑×𝑛 , 𝑉 ∈ R 𝑑×𝑄 and 𝑍, where 𝜆 ≥ 0 is a regularization parameter. This algebraic formulation eases the presentation of our algorithmic solutions that we present in next section, before we extend our framework to classification problems.

Clustered classification

We briefly present classification problems, more details are provided in Section 6.1.2. We are given 𝑛 data points 𝑥 1 , . . . , 𝑥 𝑛 ∈ R 𝑑 that belong to one of 𝐾 classes, which is encoded by binary vectors 𝑦 𝑖 ∈ {-1, 1} 𝐾 such that 𝑦 𝑖𝑘 = 1 if 𝑖 th point belongs to class 𝐾 and -1 otherwise. We search for 𝐾 linear classifiers 𝑤 1 , . . . , 𝑤 𝐾 , forming a matrix of linear classifiers 𝑊 ∈ R 𝑑×𝐾 whose classification error on a sample (𝑥, 𝑦) is measured by a loss ℓ(𝑊 𝑇 𝑥, 𝑦) such as the squared loss ℓ square (𝑊 𝑇 𝑥, 𝑦) = 1 2 ‖𝑊 𝑇 𝑥 -𝑦‖ 2 2 . Candidate classifiers are given by the minimization of the empirical loss (7.5) in variables 𝑊 1 , . . . , 𝑊 𝑛 ∈ R 𝑑×𝐾 , 𝑉 1 , . . . , 𝑉 𝑄 ∈ R 𝑑×𝐾 and 𝑍, where 𝜆 ≥ 0 is a regularization parameter. As for regression this can also applied to compute affine hyperplanes by adding a constant dimension to the data and solve the problem in dimension 𝑑 + 1. Overall (7.5) possesses same structure as (7.3) by replacing regression vectors with matrices of classifiers. Both problems have convex objectives but non-convex constraints that encode the underlying combinatorial problem. Specifically the feasible set is a union of subspaces generated by assignment matrices of partitions. In the following, we focus on the regression case, classification can be treated with same algorithms by considering matrices instead of vectors.

Non-convex schemes

We present strategies that tackle directly the non-convex clustered prediction problems.

Greedy algorithm

A first strategy to solve problem (7.1) is to alternate minimization on the predictors of each group and assignment of each point to the group where its loss is smallest. This method is fast but unstable and highly dependent on initialization. However, it can be used to refine the solution of others algorithms proposed below. In practice, we stop the algorithm when change in objective values are below some prescribed threshold 𝜀. We use a backtracking line search on the stepsize 𝛾 𝑡 that guarantees decreasing of the objective. Precisely, at each iteration if

Projected gradient descent

W𝑡+1 = k-means (𝑊 𝑡 -𝛾 𝑡 (∇𝐿(𝑊 𝑡 ) + ∇𝑅(𝑊 𝑡 )), 𝑄)
decreases the objective value we keep them and we increase the stepsize by a constant factor 𝛾 𝑡+1 = 𝛼𝛾 𝑡 with 𝛼 > 1. If W𝑡+1 increases the objective value we decrease the stepsize by a constant factor 𝛾 𝑡 ← 𝛽𝛾 𝑡 , with 𝛽 < 1, compute new W𝑡+1 and iterate this operation until W𝑡+1 decreases the objective value or the stepsize reaches the stopping value 𝜀 used as a stopping criterion on the objective values. We observed better results with this line search than with constant stepsize.

Using this strategy we observed convergence of the projected gradient algorithm in less than 100 iterations even for large 𝑛 which makes it scalable. The complexity of its core operations amounts indeed to k-means operations. k-means++ initialization costs 𝑂(𝑄 2 𝑛𝑑) operations and standard alternating minimization approximates the k-means operation at a cost of 𝑂(𝑇 𝑄𝑛𝑑), where 𝑇 , the number of alternating steps, is generally small.

Convex relaxation

Difficulty of clustered regression problems (7.1) lies in its underlying combinatorial nature that can be isolated in the case of a squared loss and squared regularizer. In this case problem (7.1) reads minimize

1 2𝑛 𝑄 ∑ 𝑞=1 ∑ 𝑖∈𝑔𝑞 ( 𝑣 𝑇 𝑞 𝑥 𝑖 -𝑦 𝑖 ) 2 + 𝜆 2 𝑄 ∑ 𝑞=1 𝑠 𝑞 ‖𝑣 𝑞 ‖ 2 2 (7.7)
in the partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) of {1, . . . , 𝑛} and prediction vectors 𝑣 1 , . . . , 𝑣 𝑄 ∈ R 𝑑 , where 𝑠 𝑞 = Card(𝑔 𝑞 ) is the size of group 𝑔 𝑞 and 𝜆 ≥ 0 is a regularization parameter.

In the following, we denote 𝑋 = (𝑥 1 , . . . , 𝑥 𝑛 ) 𝑇 ∈ R 𝑛×𝑑 the matrix of data points and 𝑦 = (𝑦 1 , . . . , 𝑦 𝑛 ) 𝑇 ∈ R 𝑛 the vector of observations.

Simplified formulation for squared loss

The squared loss has the advantage to provide analytic solutions for prediction problems in the continuous variables. Resulting problem on partitions can be expressed in terms of normalized equivalence matrices presented in Section 5.2. Following proposition details the simplified formulations of the clustered prediction problem for squared loss and squared regularizer.

Proposition 7.3.1. Problem (7.7) is equivalent to minimize 1 2𝑛 𝑦 𝑇 (I + 1 𝜆𝑛 𝑋𝑋 𝑇 ∘ 𝑀 ) -1 𝑦 subject to 𝑀 ∈ ℳ (7.8)
where ℳ = {𝑀 : 𝑀 = 𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 , 𝑍 ∈ {0, 1} 𝑛×𝑄 , 𝑍1 = 1} is the set of equivalence matrices.

Proof. We fix a partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) and define for each group 𝑔 𝑞 = {𝑘 1 , . . . , 𝑘 𝑠𝑞 } ⊂ {1, . . . , 𝑛}, the matrix 𝐸 ∈ {0, 1} 𝑠𝑞×𝑛 that picks the 𝑠 𝑞 points of 𝑔 𝑞 , i.e. (𝐸 𝑞 ) 𝑖𝑗 = 1 if 𝑗 = 𝑘 𝑖 and 0 otherwise. Therefore 𝑦 𝑞 = 𝐸 𝑞 𝑦 ∈ R 𝑠𝑞 and 𝑋 𝑞 = 𝐸 𝑞 𝑋 ∈ R 𝑠𝑞×𝑑 are respectively the vector of labels and the matrix of sample vectors of the group 𝑔 𝑞 . We naturally have 𝐸 𝑞 𝐸 𝑇 𝑞 = I as rows of 𝐸 𝑞 are orthonormal and 𝐸 𝑇 𝑞 𝐸 𝑞 is a diagonal matrix where 𝑍 𝑞 = diag(𝐸 𝑇 𝑞 𝐸 𝑞 ) ∈ {0, 1} 𝑛 is the assignment vector in group 𝑔 𝑞 , i.e. (𝑍 𝑞 ) 𝑗 = 1 if 𝑗 ∈ 𝑔 𝑞 and 0 otherwise. 𝑍 = (𝑍 1 , . . . , 𝑍 𝑄 ) is therefore an assignment matrix for the partition 𝐺.

Minimizing in 𝑣 and using the Sherman-Woodbury-Morrison formula, we obtain a function of the partition

𝜓(𝐺) = min 𝑣 1 ,...,𝑣 𝑄 1 2𝑛 𝑄 ∑ 𝑞=1 ‖𝑦 𝑞 -𝑋 𝑞 𝑣 𝑞 ‖ 2 2 + 𝜆 2 𝑄 ∑ 𝑞=1 𝑠 𝑞 ‖𝑣 𝑞 ‖ 2 2 = 1 2𝑛 𝑄 ∑ 𝑞=1 ( ‖𝑦 𝑞 ‖ 2 2 -𝑦 𝑇 𝑞 𝑋 𝑞 (𝑠 𝑞 𝜆𝑛I + 𝑋 𝑇 𝑞 𝑋 𝑞 ) -1 𝑋 𝑇 𝑞 𝑦 𝑞 ) = 1 2𝑛 𝑄 ∑ 𝑞=1 𝑦 𝑇 𝑞 (I + 1 𝑠 𝑞 𝜆𝑛 𝑋 𝑞 𝑋 𝑇 𝑞 ) -1 𝑦 𝑞 .
Formulating terms of the sum as solutions of an optimization problem, we get 𝑛×𝑛 is an orthonormal matrix, we make the change of variable 𝛽 = 𝐸 𝑇 𝛼 (and so 𝛼 = 𝐸𝛽) such that for 𝛼 = (𝛼 1 ; . . . ; 𝛼 𝑄 ), 𝛼 𝑞 ∈ R 𝑠𝑞 , 𝛼 𝑞 = 𝐸 𝑞 𝛽. Decomposing 𝑋 𝑞 and 𝑦 𝑞 and using 𝐸 𝑇 𝑞 𝐸 𝑞 = diag(𝑍 𝑞 ), we get

𝜓(𝐺) = 1 2𝑛 𝑄 ∑ 𝑞=1 max 𝛼𝑞∈R 𝑠𝑞 -𝛼 𝑇 𝑞 (I + 1 𝑠 𝑞 𝜆𝑛 𝑋 𝑞 𝑋 𝑇 𝑞 )𝛼 𝑞 + 2𝑦 𝑇 𝑞 𝛼 𝑞 = 1 2𝑛 max
𝜓(𝐺) = 1 2𝑛 max 𝛽∈R 𝑛 𝑄 ∑ 𝑞=1 -𝛽 𝑇 𝐸 𝑇 𝑞 (I + 1 𝑠 𝑞 𝜆𝑛 𝑋 𝑞 𝑋 𝑇 𝑞 )𝐸 𝑞 𝛽 + 2𝑦 𝑇 𝑞 𝐸 𝑞 𝛽 = 1 2𝑛 max 𝛽∈R 𝑛 𝑄 ∑ 𝑞=1 -𝛽 𝑇 𝐸 𝑇 𝑞 (I + 1 𝑠 𝑞 𝜆𝑛 𝐸 𝑞 𝑋𝑋 𝑇 𝐸 𝑇 𝑞 )𝐸 𝑞 𝛽 + 2𝑦 𝑇 𝐸 𝑇 𝑞 𝐸 𝑞 𝛽 = 1 2𝑛 max 𝛽∈R 𝑛 𝑄 ∑ 𝑞=1 -𝛽 𝑇 diag(𝑍 𝑞 )𝛽 - 1 𝑠 𝑞 𝜆𝑛 𝛽 𝑇 diag(𝑍 𝑞 )𝑋𝑋 𝑇 diag(𝑍 𝑞 )𝛽 + 2𝑦 𝑇 diag(𝑍 𝑞 )𝛽.
For 𝑞 fixed, (

1 𝑠𝑞 diag(𝑍 𝑞 )𝑋𝑋 𝑇 diag(𝑍 𝑞 ) ) 𝑖𝑗 = 1 𝑠𝑞 𝑥 𝑇 𝑖 𝑥 𝑗 if (𝑖, 𝑗) ∈ 𝑔 𝑞 and 0 otherwise. So 𝑄 ∑ 𝑞=1 1 𝑠 𝑞 diag(𝑍 𝑞 )𝑋𝑋 𝑇 diag(𝑍 𝑞 ) = 𝑋𝑋 𝑇 ∘ 𝑀,
where 𝑀 = 𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 is the normalized equivalence matrix of the partition 𝐺 and ∘ denotes the Hadamard product. Using ∑ 𝑄 𝑞=1 diag(𝑍 𝑞 ) = I, we finally get a function of the equivalence matrix

𝑓 (𝑀 ) = 1 2𝑛 max 𝛽∈R 𝑛 -𝛽 𝑇 (I + 1 𝜆𝑛 𝑋𝑋 𝑇 ∘ 𝑀 )𝛽 + 2𝑦 𝑇 𝛽 = 1 2𝑛 𝑦 𝑇 (I + 1 𝜆𝑛 𝑋𝑋 𝑇 ∘ 𝑀 ) -1 𝑦.
The resulting problem is still non-convex due to the combinatorial nature of the set of equivalence matrices. However one can then relax the problem by optimizing on its convex hull as presented in Section 5.3.2.

Conditional gradient algorithm application

We detail the linear minimization oracle used by Frank-Wolfe Algorithm 5 in this setting. Denote the objective function of problem (7.8) by

𝑓 (𝑀 ) 1 2𝑛 𝑦 𝑇 (I + 1 𝜆𝑛 𝑋𝑋 𝑇 ∘ 𝑀 ) -1 𝑦. (7.9)
Its gradient at a given 𝑀 ∈ ℳ is

∇𝑓 (𝑀 ) = - 1 2𝜆𝑛 2 𝑋𝑋 𝑇 ∘ ( (I + 1 𝜆𝑛 𝑋𝑋 𝑇 ∘ 𝑀 ) -1 𝑦𝑦 𝑇 (I + 1 𝜆𝑛 𝑋𝑋 𝑇 ∘ 𝑀 ) -1
) . (7.10)

Observe that -∇𝑓 (𝑀 ) is a semi-definite positive matrix with squared root

𝑈 = 1 √ 2𝜆𝑛 diag ( (I + 1 𝜆𝑛 𝑋𝑋 𝑇 ∘ 𝑀 ) -1 𝑦 ) 𝑋 ∈ R 𝑛×𝑑 .
The linear minimization oracle to minimize 𝑓 (𝑀 ) over the convex hull of the set of normalized equivalence matrices can then be computed as follows In 𝜗 1 , we used that ℳ is a set of atoms, so its convex hull is a polytope and linear minimization on it is equivalent to linear minimization on its vertices, i.e., ℳ. In 𝜗 2 , we used that normalized equivalence matrices are orthogonal projectors, so 𝐼 -𝑆 is also an orthogonal projector. Now we observe that (7.11) is a k-means problem as presented in Section 5.2.2. Therefore solving a k-means problem on the rows of the squared root matrix 𝑈 of -∇𝑓 (𝑀 ) offers a solution to the linear minimization oracle. Careful initialization as made in k-means++ [START_REF] Arthur | k-means++: The advantages of careful seeding[END_REF] offers logarithmic approximations to the problem.

A conditional gradient method with approximate oracle can then be applied to solve a relaxed version of (7.8). Once done, it remains to provide an approximate feasible solution for the original constraints. Two projections of the relaxed version are possible: either finding the closest normalized equivalence matrix in Frobenius norm or computing the point that minimizes the gradient of the relaxed solution, i.e. computing its linear minimization oracle. In practice we chose second solution as it provided better results. Notice that the k-means operation used for the linear minimization provides not only a normalized equivalence matrix but also a corresponding partition that leads to the optimal predictors 𝑣 1 , . . . , 𝑣 𝑄 as (7.12) where 𝑋 𝑞 , 𝑦 𝑞 , 𝑍 𝑞 and 𝐸 𝑞 are defined in the proof of Proposition 7.3.1.

𝑣 𝑞 = (𝑛𝜆𝑠 𝑞 I + 𝑋 𝑇 𝑞 𝑋 𝑞 ) -1 𝑋 𝑇 𝑞 𝑦 𝑞 = (𝑛𝜆𝑠 𝑞 I + 𝑋 𝑇 𝐸 𝑇 𝑞 𝐸 𝑞 𝑋) -1 𝑋 𝑇 𝐸 𝑇 𝑞 𝐸 𝑞 𝑦 = (𝑛𝜆𝑠 𝑞 I + 𝑋 𝑇 diag(𝑍 𝑞 )𝑋) -1 𝑋 𝑇 diag(𝑍 𝑞 )𝑦,
To summarize, our convex relaxation for regression with grouped features is presented in Algorithm 9. We denote by 𝑍 = k-means(𝑈, 𝑄) an assignment matrix solution of the k-means problem that cluster rows of 𝑈 in 𝑄 groups.

Inputs: Data (𝑋, 𝑌 ), desired number of groups 𝑄, target precision 𝜀 Initialize 𝑀 0 ∈ ℳ for 𝑡 = 0, . . . do Compute -∇𝑓 (𝑀 𝑡 ) in (7.10) and its squared root 𝑈 Get linear minimization oracle by computing

𝑍 𝑡 = k-means(𝑈, 𝑄) 𝑆 𝑡 = 𝑍 𝑡 (𝑍 𝑇 𝑡 𝑍 𝑡 ) † 𝑍 𝑇 𝑡 if Tr ( (𝑀 𝑡 -𝑆 𝑡 ) 𝑇 ∇𝑓 (𝑀 𝑡 ) ) ≤ 𝜀 then Stop end if Set 𝑀 𝑡+1 = 𝑀 𝑡 + 1
𝑡+2 (𝑆 𝑡 -𝑀 𝑡 ) end for Output: Optimal predictors 𝑣 1 , . . . , 𝑣 𝑄 in (7.12) from partition given by 𝑍 𝑡 .

We briefly examine the complexity of Algorithm 6. To get linear minimization oracle we use a k-means++ initialization that costs 𝑂(𝑄 2 𝑛𝑑) operations and standard alternating minimization approximates then the k-means operation at a cost of 𝑂(𝑇 𝑄𝑛𝑑), where 𝑇 is the number of alternating steps, that is generally small. The squared root computation is directly given by the computation of ∇𝑓 (𝑀 ). However this gradient itself requires the inversion of a matrix of size 𝑛 × 𝑛. This burdens its implementation for big data sets. However a few iterations of this algorithm can give a good initialization for non-convex approaches of the problem.

Numerical experiments

We test the robustness of our method when data can be clustered using a few features. We generate 𝑛 data points (𝑥 𝑖 , 𝑦 𝑖 ) for 𝑖 = 1, . . . , 𝑛, with 𝑥 𝑖 ∈ R 𝑑 , 𝑑 = 10, and 𝑦 𝑖 ∈ R, divided in 𝑄 = 3 clusters corresponding to regression tasks with weight vectors 𝑣 𝑞 . Regression labels for points 𝑥 𝑖 in group 𝑔 𝑞 are given by 𝑦 𝑖 = 𝑣 𝑇 𝑞 𝑥 𝑖 + 𝜂 𝑦 , where 𝜂 ∼ 𝒩 (0, 𝜎 2 ). We test the robustness of the algorithms to the addition of noisy dimensions by completing 𝑥 𝑖 with 𝛿 dimensions of noise 𝜃 ∼ 𝒩 (0, 𝜏 2 ). For testing the models we take the difference between the true label and the best prediction such that the mean square error is given by

1 2𝑛 𝑛 ∑ 𝑖=1 min 𝑞=1...,𝑄 (𝑦 𝑖 -𝑣 𝑇 𝑞 𝑥 𝑖 ) 2 . (7.13)
The results are reported in Table 7.1 where the intrinsic dimension is 10 and the proportion of dimensions of noise 𝛿/(𝑑+𝛿) increases. On the algorithmic side, "Oracle" refers to the least-squares fit given the true assignments, which can be seen as the best achievable error rate, AM refers to alternate minimization, PG refers to projected gradient with squared loss, CG refers to conditional gradient and RC to regression clustering as proposed by [START_REF] Zhang | Regression clustering[END_REF], implemented using the Harmonic K-means formulation. PG, CG and RC were followed by AM refinement. 1000 points were used for training, 100 for testing. The regularization parameters were 5-fold crossvalidated using a logarithmic grid. Noise on labels is 𝜎 = 10 -1 and noise on added dimensions is 𝜏 = 1. Results were averaged over 50 experiments with figures after the ± sign corresponding to one standard deviation. p = 0 p = 0.25 p = 0.5 p = 0.75 p = 0.9 Oracle 0. Table 7.1 -Test mean square error given by (7.13) along proportion of added dimensions of noise 𝑝 = 𝛿/(𝑑 + 𝛿).

All algorithms perform similarly, RC and AM get better results without added noise. None of the present algorithms get a significantly better behavior with a majority of noisy dimensions.

Conclusion

We developed a framework for diversity learning that finds groups of samples which share same predictor. Projected gradient scheme offers a scalable algorithmic approach for any loss, while convex relaxation for squared loss gives a first good estimate. However core inner steps of these approaches are only approximated by k-means++, a finer analysis would require to both tackle non-convexity and inexact oracles. Yet, this framework may find its application in privacy learning to leverage hidden information.

137

Introduction

A supervised learning problem aims at fitting a model on a set of training data that can generalize to future samples. Generalization performance naturally increases with the number of training samples but can also benefit from the number of tasks performed simultaneously on the data [START_REF] Maurer | The benefit of multitask representation learning[END_REF]. Similar tasks can indeed share information to improve overall performance [START_REF] Argyriou | Convex multi-task feature learning[END_REF][START_REF] Evgeniou | Regularized multi-task learning[END_REF][START_REF] Pan | A survey on transfer learning[END_REF]. For example, in image classification, classifiers associated with different species of cats should be quite similar, but well separated from classifiers associated with cars. This observation motivated [START_REF] Jacob | Group lasso with overlap and graph lasso[END_REF] to regularize multi-task learning problems by a clustering penalty that groups similar tasks. It led to further developments in the transfer learning literature [START_REF] Zhang | A convex formulation for learning task relationships in multi-task learning[END_REF][START_REF] Kumar | Learning task grouping and overlap in multi-task learning[END_REF][START_REF] Ciliberto | Convex learning of multiple tasks and their structure[END_REF][START_REF] Zhou | Clustered multi-task learning via alternating structure optimization[END_REF].

In this work, we investigate new optimization procedures of the clustered multitask of [START_REF] Jacob | Group lasso with overlap and graph lasso[END_REF]. We use same clustering penalty that can be decomposed in three terms : overall regularization of the tasks, regularization of the variance between clusters and regularization of the variance within clusters. However, here, we isolate the clustering problem and do not relax it. First, we develop a projected gradient descent, that clusters predictors at each prediction step. Then, for squared loss we isolate the underlying partitioning problem on the tasks and show that it amounts to a k-means problem. We illustrate the performance on our approach on topics classification of a corpus of texts.

Problem Formulation

For simplicity, we illustrate the case of multi-category classification, which can be extended to the general multi-task setting. We briefly present classification problems, more details are provided in Section 6.1.2. We are given 𝑛 data points 𝑥 1 , . . . , 𝑥 𝑛 ∈ R 𝑑 that belong to one of 𝐾 classes, which is encoded by binary vectors 𝑦 𝑖 ∈ {-1, 1} 𝐾 such that 𝑦 𝑖𝑘 = 1 if 𝑖 th point belongs to class 𝐾 and -1 otherwise. We search for 𝐾 linear classifiers 𝑤 1 , . . . , 𝑤 𝐾 , forming a matrix of linear classifiers 𝑊 ∈ R 𝑑×𝐾 whose classification error on a sample (𝑥, 𝑦) is measured by a loss ℓ(𝑊 𝑇 𝑥, 𝑦) such as the squared loss ℓ square (𝑊 𝑇 𝑥, 𝑦) = 1 2 ‖𝑊 𝑇 𝑥 -𝑦‖ 2 2 . Candidate classifiers are given by the minimization of the empirical loss

𝐿(𝑊 ) = 1 𝑛 𝑛 ∑ 𝑖=1 ℓ(𝑊 𝑇 𝑥 𝑖 , 𝑦 𝑖 ).
Various strategies are used to leverage the information coming from related tasks, such as low rank [START_REF] Argyriou | Convex multi-task feature learning[END_REF] or structured norm penalties [START_REF] Ciliberto | Convex learning of multiple tasks and their structure[END_REF] on the matrix of classifiers 𝑊 . Here we follow the clustered multitask setting introduced in Jacob et al. [2009]. Namely we add a penalty 𝜔 on the classifiers (𝑤 1 , . . . , 𝑤 𝐾 ) which enforce them to be clustered in 𝑄 groups 𝑔 1 , . . . , 𝑔 𝑄 , centered around 𝑄 points 𝑣 1 , . . . , 𝑣 𝑄 ∈ R 𝑑 . By denoting 𝐺 = (𝑔 1 , . . . 𝑔 𝑄 ) the partition of the 𝐾 tasks, 𝑠 𝑞 = Card(𝑔 𝑞 ) the size of the group 𝑔 𝑞 and 𝑉 = (𝑣 1 . . . , 𝑣 𝑄 ) ∈ R 𝑑×𝑄 the matrix of centroids, this penalty can be decomposed in The partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) of the 𝐾 tasks can then be represented by an assignment matrix 𝑍 ∈ {0, 1} 𝐾×𝑄 , whose columns index tasks and rows groups such that

𝑍 𝑘𝑞 = { 1 if 𝑘 ∈ 𝑔 𝑞 0 otherwise.
To represent the barycenter of the predictors, we use the orthogonal projection Π 𝐾 = 1 𝐾 1 𝑇 𝐾 /𝐾 ∈ R 𝐾×𝐾 on the vector 1 𝐾 of all ones in dimension 𝐾 . Given 𝑄 centroids forming a matrix 𝑉 = (𝑣 1 , . . . , 𝑣 𝑄 ) ∈ R 𝑑×𝑄 and a partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) represented by an assignment matrix 𝑍 ∈ {0, 1} 𝐾×𝑄 such that 𝑍1 = 1, previous penalties read [START_REF] Arthur | k-means++: The advantages of careful seeding[END_REF] offers logarithmic approximations to the problem such that approximate projection is available. Given a matrix 𝑊 , whose columns we want to cluster in 𝑄 groups, we denote by [𝑍, 𝑉 ] = k-means(𝑊, 𝑄) respectively the assignment matrix and the matrix of centroids output by a clustering algorithm. A projected gradient scheme for problem (8.2) is described in Algorithm 10. decrease the objective value we keep them and we increase the stepsize by a constant factor 𝛾 𝑡+1 = 𝛼𝛾 𝑡 with 𝛼 > 1. If W𝑡+1 , Ū𝑡+1 increase the objective value we decrease the stepsize by a constant factor 𝛾 𝑡 ← 𝛽𝛾 𝑡 , with 𝛽 < 1, compute new W𝑡+1 , Ū𝑡+1 and iterate this operation until W𝑡+1 , Ū𝑡+1 decrease the objective value or the stepsize reaches the stopping value 𝜀 used as a stopping criterion on the objective values. We observed better results with this line search than with constant stepsize.

Ω
Using this strategy we observed convergence of the projected gradient algorithm in less than 100 iterations even for large dimension (big 𝑛, 𝑑 or 𝐾), which makes it scalable. The complexity of its core operations amounts indeed to k-means operations. k-means++ initialization costs 𝑂(𝑄 2 𝑛) operations and standard alternating minimization approximates the k-means operation at a cost of 𝑂(𝑇 𝑄𝑛), where 𝑇 , the number of alternating steps, is generally small.

Clustered multitask with squared loss

Difficulty of clustered multitask problems (8.2) lies in its underlying combinatorial nature that can be isolated in the case of a squared loss. We first detail the simplified formulation and then show how it reduces to a k-means problem.

Simplification for squared loss

When simplifying the clustered multitask problem, partitions are expressed in terms of normalized equivalence matrices presented in Section 5.2. First, simplification can be made on the clustering penalty (8.1) by minimizing on the variable 𝑉 for 𝑊 and 𝑍 fixed as detailed in the following lemma. Denote 𝑠 = (𝑠 1 , . . . , 𝑠 𝑄 ) ∈ R 𝑄 the vector of the size of the groups 𝑔 1 , . . . , 𝑔 𝑄 that can be computed from the assignment matrix 𝑍 as 𝑠 = 𝑍 𝑇 1 𝐾 , 𝑠 The remaining variable of the problem is the normalized equivalence matrix 𝑀 . By ignoring the constant terms we get the claimed formulation.

Clustered multitask problem reduces therefore to a linear problem in normalized equivalence matrices 𝑀 of partitions of the 𝐾 classes. Though the feasible set is still non-convex, minimization of this linear function on the set of normalized equivalence matrices reduces to a k-means problem that can be approximated efficiently.

Resolution by k-means

As 0 ≤ 𝜌 𝑤𝑏 ≤ 1, we get that 𝐴 defined in (8.5) is semi-definite negative and denote 𝑈 ∈ R 𝐾×𝐾 a squared root of -𝐴 such that -𝐴 = 𝑈 𝑈 𝑇 . Clustered multitask problem (8.4) reads then In 𝜗, we used that normalized equivalence matrices are orthogonal projectors, so 𝐼 -𝑆 is also an orthogonal projector. Now we observe that (8.6) is a k-means problem as presented in Section 5.2.2. Therefore solving a k-means problem on the rows of the squared root matrix 𝑈 of -∇𝑓 (𝑀 ) offers a solution to the clustered multi-task problem with squared loss. Careful initialization as made in k-means++ [START_REF] Arthur | k-means++: The advantages of careful seeding[END_REF] offers logarithmic approximations to the problem. Resolution of the clustered-multitask problem with squared loss by k-means is summarized in Algorithm 11. We denote 𝑍 = k-means(𝑈, 𝑄) the assignment matrix found by a dedicated algorithm that clusters the rows of 𝑈 into 𝑄 groups. Once an equivalence matrix found for the problem, classifiers stacked in 𝑊 = (𝑤 1 , . . . , 𝑤 𝑘 ) are given in (8.7) using computations made in Proposition 8. Overall this solution requires a k-means++ initialization that costs 𝑂(𝑄 2 𝐾 2 ) operations and the resolution of the k-means problem that standard alternating minimization approximates at a cost of 𝑂(𝑇 𝑄𝐾 2 ), where 𝑇 is the number of alternating steps, which is generally small. The main drawback of this solution is the computation of 𝐴 in (8.5) that requires the inversion of matrices of size 𝑑 × 𝑑 and the computation of its squared root. This burdens its implementation for big data sets. However it can give a good initialization for more elaborated approaches that use different losses than the squared loss.

Numerical experiments

We perform classification of documents in topics. We used the publicly available 20NewsGroup dataset which contains 2800 documents that are classified in 20 topics. Some of the topics are very closely related to each other (e.g. pc.hardware and mac.hardware), while others are highly unrelated (e.g. misc.forsale and soc.religion.christian). Using this prior, we try to benefit from the classification of some topics with the use of similar topics. We use a dictionary of 5000 words selected by TF-IDF coefficients and take the word frequencies as covariates for each document.

In Table 8.1, we compare our approach to other classical regularizations such as the Frobenius norm and the trace norm, as implemented by [START_REF] Ciliberto | Convex learning of multiple tasks and their structure[END_REF], using either a ridge or a logistic loss (Log). The algorithm proposed by [START_REF] Jacob | Group lasso with overlap and graph lasso[END_REF] was too slow on this large dataset to compare with. We initialize it by the solution given by the logistic loss. All algorithms were 5-fold cross-validated on 80% of the data then tested on the remaining 20%. The number of clusters was set to 5, as suggested by the names of newsgroups. Figures after the ± sign correspond to one standard deviation when varying the training and test sets. At first glance, our solutions appear to offer better generalization performance, however the groups of tasks found by our method do not have meaning and best regularization parameters are the ones that do not tend to clustered tasks.

Frobenius Log Trace Ridge CG Ridge PG Log 5.8±2.0 4.3±0.8 4.9±0.8 2.6±1.5

Table 8.1 -100 × mean absolute errors for predicting topics on 20NewsGroup dataset, comparing classical regularizers (Frobenius and Trace) with our algorithms. PG refers to projected gradient, CG refers to conditional gradient.

Conclusion

We applied our framework of learning problems with partitions to cluster tasks. Once more a projected gradient scheme can be applied to offer scalable resolution for any loss. However framework of [START_REF] Jacob | Group lasso with overlap and graph lasso[END_REF] requires to tune at least 3 parameters which burdens its application. Moreover regularization of the variance between clusters is generally taken positive to develop convex relaxations. Further experiments that both enforce low variance intra-clusters and high variance inter-clusters may better benefit from the clustering approach. Notice that our computations for squared loss revealed that this framework can be reduced to a k-means problem for which several approaches exist that may refine the framework. 149 Résumé Dans de nombreux domaines tels que l'apprentissage statistique, la recherche opérationnelle ou encore la conception de circuits, une tâche est modélisée par un jeu de paramètres que l'on cherche à optimiser pour prendre la meilleure décision possible. Mathématiquement, le problème revient à minimiser une fonction de l'objectif recherché par des algorithmes itératifs. Le développement de ces derniers dépend alors de la géométrie de la fonction ou de la structure du problème.

Dans une première partie, cette thèse étudie comment l'acuité d'une fonction autour de ses minima peut être exploitée par le redémarrage d'algorithmes classiques. Les schémas optimaux sont présentés pour des problèmes convexes généraux. Ils nécessitent cependant une description complète de la fonction, ce qui est rarement disponible. Des stratégies adaptatives sont donc développées et prouvées être quasi-optimales. Une analyse spécifique est ensuite conduite pour les problèmes parcimonieux qui cherchent des représentations compressées des variables du problème. Leur géométrie conique sous-jacente, qui décrit l'acuité de la fonction de l'objectif, se révèle contrôler à la fois la performance statistique du problème et l'efficacité des procédures d'optimisation par une seule quantité.

Une seconde partie est dédiée aux problèmes d'apprentissage statistique. Ceux-ci effectuent une analyse prédictive de données à l'aide d'un large nombre d'exemples. Une approche générique est présentée pour à la fois résoudre le problème de prédiction et le simplifier en groupant soit les variables, les exemples ou les tâches. Des méthodes algorithmiques systématiques sont développées en analysant la géométrie induite par une partition des données. Une analyse théorique est finalement conduite lorsque les variables sont groupées par analogie avec les méthodes parcimonieuses.

Mots Clés

Optimisation convexe, Borne d'erreur, Parcimonie, Acuité, Modèles structurés par partitions des données.

Abstract

In numerous fields such as machine learning, operational research or circuit design, a task is modeled by a set of parameters to be optimized in order to take the best possible decision. Formally, the problem amounts to minimize a function describing the desired objective with iterative algorithms. The development of these latter depends then on the characterization of the geometry of the function or the structure of the problem.

In a first part, this thesis studies how sharpness of a function around its minimizers can be exploited by restarting classical algorithms. Optimal schemes are presented for general convex problems. They require however a complete description of the function that is rarely available. Adaptive strategies are therefore developed and shown to achieve nearly optimal rates. A specific analysis is then carried out for sparse problems that seek for compressed representation of the variables of the problem. Their underlying conic geometry, that describes sharpness of the objective, is shown to control both the statistical performance of the problem and the efficiency of dedicated optimization methods by a single quantity.

A second part is dedicated to machine learning problems. These perform predictive analysis of data from large set of examples. A generic framework is presented to both solve the prediction problem and simplify it by grouping either features, samples or tasks. Systematic algorithmic approaches are developed by analyzing the geometry induced by partitions of the data. A theoretical analysis is then carried out for grouping features by analogy to sparse methods.

  -clés : Optimisation convexe, Borne d'erreur, Parcimonie, Acuité, Modèles structurés par partitions des données. vii viii A.3 Gradient descent method . . . . . . . . . . . . . . . . . . . . . . . . .

  Figure2-1 -Sonar data set. From top to bottom and left to right: least square loss, logistic loss, dual SVM problem and LASSO. We use adaptive restarts (Adap), gradient descent (Grad), accelerated gradient (Acc) and restart heuristic enforcing monotonicity (Mono). Large dots represent the restart iterations. Regularization parameters for dual SVM and LASSO were set to one.

Figure 2

 2 Figure 2-2 -Comparison of the methods for the LASSO problem on Sonar dataset where number of iterations of the Adaptive method is multiplied by the size of the grid. Grid search step size is set to 4.

  Definition 4.1.1. (Null Space Property) The matrix 𝐴 satisfies the Null Space Property (NSP) on support 𝑆 ⊂ 1, 𝑝 with constant 𝛼 ≥ 1 if for any 𝑧 ∈ Null(𝐴)∖{0},

  Definition 4.3.2. (Generalized Null space Property) A linear operator 𝐴 on 𝐸 satisfies the Generalized Null Space Property (GNSP) for orthogonal projector 𝑃 ∈ 𝒫 with constant 𝛼 ≥ 1 if and only if for any 𝑧 ∈ Null(𝐴) ∖ {0} such that 𝑧 = 𝑃 𝑧 + P 𝑧, 𝛼‖𝑃 𝑧‖ < ‖ P 𝑧‖.(GNSP)

  Lemma 4.3.4. For a given sparsity 𝑠, write ℰ = ⋃ 𝑃 ∈𝒫 : 𝜂(𝑃 )≤𝑠 {𝑧 ∈ 𝐸 : 𝑧 = 𝑃 𝑧 + P 𝑧, ‖ P 𝑧‖ ≤ ‖𝑃 𝑧‖} and ℱ = ⋃ 𝑃 ∈𝒫 : 𝜂(𝑃 )≤𝑠 ⋃ 𝑥∈𝐸 : 𝑥=𝑃 𝑥 𝒯 (𝑥). Then ℰ = ℱ. Proof. Let 𝑧 ∈ ℰ and 𝑃 ∈ 𝒫 such that 𝑧 = 𝑃 𝑧 + P 𝑧. Taking 𝑥 = -𝑃 𝑧 we get ‖𝑥 + 𝑧‖ = ‖ P 𝑧‖ ≤ ‖𝑃 𝑧‖ = ‖𝑥‖. Therefore 𝑧 ∈ 𝒯 (𝑥) ⊂ ℱ. Conversely, if 𝑧 ∈ ℱ, denote 𝑥 ∈ 𝐸 and 𝑃 ∈ 𝒫 such that 𝑥 = 𝑃 𝑥, 𝑧 ∈ 𝒯 (𝑥) and 𝜂(𝑃 ) ≤ 𝑠. If P = I -𝑃 , by decomposability (ii), ‖𝑥 + 𝑧‖ = ‖𝑃 𝑥 + 𝑃 𝑧‖ + ‖ P 𝑧‖ ≥ ‖𝑥‖ -‖𝑃 𝑧‖ + ‖ P 𝑧‖.

  Figure 4-2 -Best restarted NESTA (solid red line) and overall cost of the practical restart schemes (dashed red line) versus NESTA with 5 continuation steps (dotted blue line) for a budget of 500 iterations. Crosses represent the restart occurrences. Left: 𝑛 = 200. Right : 𝑛 = 300.

Figure 4

 4 Figure 4-3 -Best restart scheme found by grid search for increasing values of the oversampling ratio 𝜏 = 𝑛/𝑠. Left : sparsity 𝑠 = 20 fixed. Right : number of samples 𝑛 = 200 fixed.

Figure 4

 4 Figure4-4 -We plot the cone-restricted condition number of 𝐴 (upper left), explaining both the computational complexity of problem (ℓ 1 recovery) (right column) and the statistical complexity of problem (Robust ℓ 1 recovery) (second on the left). Central curves represent the mean (geometric mean in log-scale plots), red curves correspond to 10th and 90th percentile. We observe that high computing times (peaks in the right column) are directly aligned with instances where sparse recovery barely holds/fails (left), i.e. near the phase transition around 𝑛 = 70, where the distance to feasibility for problem (P A,𝒯 (𝑥) ) also follows a phase transition.

  Lemma 4.B.2. Condition (3) above, which reads ‖𝑃 * 𝑢 + P * 𝑣‖ * ≤ max(‖𝑢‖ * , ‖𝑣‖ * ), for any 𝑢, 𝑣 ∈ 𝐸, is equivalent to ‖𝑤‖ ≥ ‖𝑃 𝑤‖ + ‖ P 𝑤‖, for any 𝑤 ∈ 𝐸. Proof. Denote 𝑓 : (𝑢, 𝑣) → ‖𝑃 * 𝑢 + P * 𝑣‖ * and 𝑔 : (𝑢, 𝑣) → max(‖𝑢‖ * , ‖𝑣‖ * ).

𝑤 {‖𝑤‖ 2

 2 /2 : 𝑃 𝑤 = 𝑠, P 𝑤 = 𝑡}, where the infimum is +∞ if the constraints are infeasible. Then the dual of the norm 𝑔 is (𝑠, 𝑡) → ‖𝑠‖ + ‖𝑡‖ therefore condition (3) is equivalent to inf 𝑤 {‖𝑤‖ : 𝑃 𝑤 = 𝑠, P 𝑤 = 𝑡} ≥ ‖𝑠‖ + ‖𝑡‖, for any 𝑠, 𝑡 ∈ 𝐸, which reads ‖𝑤‖ ≥ ‖𝑃 𝑤‖ + ‖ P 𝑤‖, for any 𝑤 ∈ 𝐸.

  Card(Supp(𝑤)) ≤ 𝑠, Card(Part(𝑤)) ≤ 𝑄 + 1.

  {𝑤 ∈ R 𝑑 : Card(Supp(𝑤)) ≤ 𝑠, Card(Part(𝑤)) ≤ 𝑄 + 1} = ⋃ 𝑆∈{1,...,𝑑} : Card(𝑆)≤𝑠 𝐺∈𝒫 : Card(𝐺)≤𝑄+1

  Ẽ are defined by selecting 𝑠 features among 𝑑 and partitioning these 𝑠 features into 𝑄 groups so that their number is 𝑁 = on the binomial coefficient and (6.20), we can roughly bound 𝑁 for 𝑠 ≥ 3, 𝑄 ≥ 3 by 𝑁 ≤ ( 𝑒𝑑 𝑠

-Figure 8

 8 Figure 8-1 -Decomposed clustering penalty.

Algorithm 10

 10 Projected gradient descent for clustered multi-taskInputs: Data (𝑋, 𝑌 ), desired number of groups 𝑄, step size 𝛾 𝑡 Initialize 𝑊 0 ∈ R 𝑑×𝐾 for t = 1,. . . ,T do𝑊 𝑡+1 = 𝑊 𝑡 -𝛾 𝑡 (∇𝐿(𝑊 𝑡 ) + ∇Ω(𝑊 𝑡 , 𝑈 𝑡 )) 𝑈 𝑡+1/2 = 𝑈 𝑡 -𝛾 𝑡 (∇Ω(𝑊 𝑡 , 𝑈 𝑡 )) [𝑍 𝑡+1 , 𝑉 𝑡+1 ] = k-means(𝑈 𝑡+1/2 , 𝑄) 𝑈 𝑡+1 = 𝑉 𝑡+1 𝑍 𝑇 𝑡+1 end for Output: Ŵ = 𝑊 𝑇In practice we stop the algorithm when change in objective values are below some prescribed threshold 𝜀. We use a backtracking line search on the stepsize 𝛾 𝑡 that guarantees decreasing of the objective. Precisely, at each iteration if W𝑡+1 = 𝑊 𝑡 -𝛾 𝑡 (∇𝐿(𝑊 𝑡 ) + ∇Ω(𝑊 𝑡 , 𝑈 𝑡 )) Ū𝑡+1 = k-means (𝑈 𝑡 -𝛾 𝑡 ∇Ω(𝑊 𝑡 , 𝑈 𝑡 ), 𝑄)

  Lemma 8.3.1. For given matrix of classifier 𝑊 ∈ R 𝑑×𝐾 and assignment matrix 𝑍 ∈ {0, 1} 𝐾×𝑄 , clustering penalty (8.1) with regularization parameters 𝜆 𝑚 , 𝜆 𝑏 and 𝜆 𝑤 simplifies by minimizing in the matrix of centroids 𝑉 ∈ R 𝑑×𝑄 asmin 𝑉 Ω(𝑊, 𝑉 𝑍 𝑇 ) = 𝜆 𝑤 2 Tr ( 𝑊 (I 𝐾 -𝜌 𝑤𝑏 (𝑀 -Π 𝐾 ) -𝜌 𝑤𝑚 Π 𝐾 ) 𝑊 𝑇 )where 𝜌 𝑤𝑏 = 𝜆𝑤 𝜆𝑤+𝜆 𝑏 , 𝜌 𝑤𝑚 = 𝜆𝑤 𝜆𝑤+𝜆𝑚 and 𝑀 = 𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 is the normalized equivalence matrix of the partition represented by 𝑍. Proof. min 𝑉 Ω(𝑊, 𝑉 𝑍 𝑇 ) = min

  3.2 and Kronecker's formula. Finally 𝜌 𝑤𝑏 , 𝜌 𝑤𝑚 are defined in Lemma 8.3.1. Algorithm 11 Clustered multitask resolution by k-means Inputs: Data (𝑋, 𝑌 ), desired number of groups 𝑄, regularization parameters 𝜆 𝑤 , 𝜆 𝑏 , 𝜆 𝑚 . Compute squared root 𝑈 of𝐴 = 𝑌 𝑇 𝑋 ( (𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 I 𝑑 ) -1 -(𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑏 )I 𝑑 ) -1 ) 𝑋 𝑇 𝑌 Get 𝑍 = k-means(𝑈, 𝑄), 𝑀 = 𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 Output: 𝑊 = (𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 I 𝑑 ) -1 𝑋 𝑇 𝑌 (I 𝐾 -𝑀 ) +(𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑏 )I 𝑑 ) -1 𝑋 𝑇 𝑌 (𝑀 -Π 𝐾 ) +(𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑚 )I 𝑑 ) -1 𝑋 𝑇 𝑌 Π 𝐾 (8.7)

  appear in Advances in Neural Information Processing Systems 30 (NIPS 2017). b) Chapter 2 is based on the article: Sharpness, Restart and Acceleration, V. Roulet and A. d'Aspremont, to appear in Advances in Neural Information Processing Systems 30 (NIPS 2017). c) Chapter 4 is based on the article: Computational Complexity versus Statistical Performance on Sparse Recovery Problems, V. Roulet, N. Boumal and A. d'Aspremont, under submission to Information and Inference: A Journal of the IMA. d) Chapters 6, 7, 8 are based on the article: Supervised Learning in the Data Chapter 6 is also based on the article: Iterative Hard Clustering of Features, V. Roulet, F. Fogel, F. Bach and A. d'Aspremont, under submission to the 21st International Conference on Artificial Intelligence and Statistics (AISTATS 2018)

	Part I
	Convex optimization with error
	bounds

Cube, V.

Roulet, F. Fogel, F. Bach and A. d'Aspremont, presented at workshop Transfer and Multi-Task Learning: Trends and New Perspectives (NIPS 2015)

and the preprint: Learning with Clustering Penalties, V. Roulet, F. Fogel, F.

Bach and A. d'Aspremont, ArXiv 1506

.04908 e) 7

Chapter 1

  Definition 1.2.2. Łojasiewicz gradient inequality A differentiable function 𝑓 satisfies Łojasiewicz gradient inequality in the neighborhood 𝐾 of a critical point 𝑥 * , if there exists 𝑐 > 0 and 𝜃 ∈ [0, 1[ such that 𝑐‖∇𝑓 (𝑥)‖ 𝜃 2 ≥ 𝑓 (𝑥) -𝑓 (𝑥 * ), for every 𝑥 ∈ 𝐾.

	(1.7)

Setting 𝑦 to be the projection of 𝑥 onto 𝑋 * , this yields the following upper bound on suboptimality

  𝐿 𝑠 ‖𝑥 -𝑦‖ 𝑠 2 , for given 𝑥 ∈ R 𝑛 and 𝑦 ∈ 𝑋 * .

	𝑓 (𝑥) -𝑓 * ≤	𝐿 𝑠	𝑑(𝑥, 𝑋 * ) 𝑠 .		(2.1)
	Now assume that 𝑓 satisfies the error bound (Sharp) on a set 𝐾 with parameters
	(𝑟, 𝜇). Combining (2.1) and (Sharp) this leads for every 𝑥 ∈ 𝐾,
		𝑠𝜇 𝑟𝐿	≤ 𝑑(𝑥, 𝑋 * ) 𝑠-𝑟 .
	This means that necessarily 𝑠 ≤ 𝑟 by taking 𝑥 → 𝑋 * . Moreover if 𝑠 < 𝑟, this last inequality can only be valid on a bounded set, i.e. either smoothness or error bound
	or both are valid only on a bounded set. In the following, we write
	𝜅 𝐿	2 𝑠 /𝜇	2 𝑟	and	𝜏 1 -	𝑠 𝑟	(2.2)
	respectively a generalized condition number for the function 𝑓 and a condition number
	based on the ratio of powers in inequalities (Smooth) and (Sharp). If 𝑟 = 𝑠 = 2, 𝜅
	matches the classical condition number of the function.

* +

  As mentioned before if 𝑟 > 𝑠 = 2, this property is necessarily local, i.e. 𝐾 is bounded. We assume then that given a starting point 𝑥 0 ∈ R 𝑛 , (Sharp) is satisfied on the sublevel set {𝑥| 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )}. Remark that if this property is valid on an open set 𝐾 ⊃ 𝑋 * , it will also be valid on any compact set 𝐾 ′ ⊃ 𝐾 with the same exponent 𝑟 but a potentially lower constant 𝜇

	More details about Nesterov's algorithm are given in Appendix A.2.
	In what follows, we will also assume that 𝑓 satisfies (Sharp) with parameters (𝑟, 𝜇)
	on a set 𝐾 ⊇ 𝑋 * , which means 𝜇 𝑟 𝑑(𝑥, 𝑋

* ) 𝑟 ≤ 𝑓 (𝑥) -𝑓 * , for every 𝑥 ∈ 𝐾.

  Proposition 2.2.4. Let 𝑓 be a smooth convex function satisfying (Smooth) with parameters (2, 𝐿) and (Sharp) with parameters (𝑟, 𝜇) on a set 𝐾. Assume that we are given 𝑥 0 ∈ R 𝑛 such that {𝑥| 𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )} ⊂ 𝐾 and denote 𝑁 a given number of iterations. Run schemes 𝒮 𝑖,𝑗 defined in (2.14) to solve (P) for 𝑖 ∈ [1, . . . , ⌊log 2 𝑁 ⌋]

	and 𝑗 ∈ [0, . . . , ⌈log 2 𝑁 ⌉], stopping each time after 𝑁 total inner algorithm iterations i.e. for 𝑅 such that ∑ 𝑅 𝑘=1 𝑡 𝑘 ≥ 𝑁 . Assume 𝑁 is large enough, so 𝑁 ≥ 2𝐶 * 𝜅,𝜏 , and if 1 𝜅,𝜏 > 1. 𝑁 > 𝜏 > 0, 𝐶 * If 𝜏 = 0, there exists 𝑖 ∈ [1, . . . , ⌊log 2 𝑁 ⌋] such that scheme 𝒮 𝑖,0 achieves a precision given by
	𝑓

  𝛾 ≥ 0 and a sequence 𝑡 𝑘 for 𝑘 = 1, . . . , 𝑅. 𝐿) on a set 𝐽 and (Sharp) with parameters (𝑟, 𝜇) on a set 𝐾. Given 𝑥 0 ∈ R 𝑛 assume that {𝑥|𝑓 (𝑥) ≤ 𝑓 (𝑥 0 )} ⊂ 𝐽 ∩ 𝐾. Run (Universal RESTART) scheme from 𝑥 0 for a given 𝜀 0 ≥ 𝑓 (𝑥 0 ) -𝑓 * with 𝛾 = 𝜌, 𝑡 𝑘 = 𝐶 * 𝜅,𝜏,𝜌 𝑒 𝜏 𝑘 , where 𝐶 *

	for 𝑘 = 1, . . . , 𝑅 do 𝜀 𝑘 := 𝑒 -𝛾 𝜀 𝑘-1 , end for Output : x := 𝑥 𝑅 Our strategy is to choose a sequence 𝑡 𝑘 that ensures 𝑥 𝑘 := 𝒰(𝑥 𝑘-1 , 𝜀 𝑘 , 𝑡 𝑘 ) 𝑓 (𝑥 𝑘 ) -𝑓 * ≤ 𝜀 𝑘 , for the geometrically decreasing sequence 𝜀 𝑘 . The overall complexity of our method (Universal RESTART) will then depend on the growth of 𝑡 𝑘 as described in Lemma 2.2.1. Proposition 2.3.1. Let 𝑓 be a convex function satisfying (Smooth) with parameter (𝑠, 𝜅,𝜏,𝜌 𝑒 1-𝜏 (𝑐𝜅) 𝑠 -𝜏 𝜌 2𝜌 𝜀 0

  Lemma 4.2.3. Given a coding matrix 𝐴 ∈ R 𝑛×𝑝 and an original signal 𝑥 * , suppose we observe 𝑏 = 𝐴𝑥 * + 𝑤 where ‖𝑤‖ 2 ≤ 𝛿‖𝐴‖ 2 and denote an optimal solution of (Robust ℓ 1 recovery) by x. If the minimal singular value 𝜎 𝒯 (𝑥 * ) (𝐴) in (4.12) restricted to the tangent cone 𝒯 (𝑥 * ) in (4.10) is positive, the following error bound holds:‖x -𝑥 * ‖ 2 ≤ 2 𝛿‖𝐴‖ 2 𝜎 𝒯 (𝑥 * ) (𝐴).

	(4.13)

minimize ‖𝑥‖ 1 subject to ‖𝐴𝑥 -𝑏‖ 2 ≤ 𝛿‖𝐴‖ 2 , (Robust ℓ 1 recovery) in the variable 𝑥 ∈ R 𝑝 , with the same design matrix 𝐴 ∈ R 𝑛×𝑝 , where 𝑏 ∈ R 𝑛 are given observations perturbed by noise of level 𝛿 > 0. The following classical result then bounds reconstruction error in terms of 𝜎 𝒯 (𝑥 * ) (𝐴).

  𝜇 𝑠 (𝐴).

	(4.19)
	Connecting Lemmas 4.2.3, 4.2.5 and Proposition 4.1.4 we get the following corollary.

Corollary 4.2.6. Given a coding matrix 𝐴 ∈ R 𝑛×𝑝 and a sparsity level 𝑠 ≥ 1, if ℛ 𝑠 (𝐴) < +∞ in (4.19) then optimal (Restart) scheme achieves an 𝜀 precision in at most 𝑂((2ℛ 𝑠 (𝐴) -1) log 𝜀 -1 ) iterations.

  for 𝑘 ∈ {1, . . . , 𝐾} and 𝑖 ∈ {1, . . . , 𝑛} which reads compactly 𝑌 ≈ 𝑋𝑊,where 𝑌 = (𝑦 1 , . . . , 𝑦 𝑛 ) 𝑇 ∈ R 𝑛×𝐾 , 𝑋 = (𝑥 1 , . . . , 𝑥 𝑛 ) 𝑇 ∈ R 𝑛×𝑑 and 𝑊 = (𝑤 1 , . . . , 𝑤 𝑘 ) ∈ R 𝑑×𝐾 . A loss function is then used to measure the accuracy error of a given set of predictors 𝑊 , as detailed for regression and classification in the following chapters.Its minimization on the training data gives a candidate predictor for future samples.

  Let us fix a point 𝑥 ∈ R 𝑑 , its distance to a feasible point 𝑤 reads {𝑔 0 , . . . 𝑔 𝑄 𝐺 } ∈ 𝒫 and 𝑣 ∈ R 𝑄 𝐺 . For a fixed partition 𝐺 ∈ 𝒫, hence a fixed subspace, minimization in 𝑣 gives the barycenters of the groups 𝑔 1 , . . . , 𝑔 𝑄 𝐺 of non-zero coefficients denoted𝑥 𝑖 for 𝑞 = 1, . . . 𝑄 𝐺 ,where 𝑠 𝑞 = Card(𝑔 𝑞 ) is the size of the 𝑞 th group. Inserting them in (6.22), the distance to a subspace of sparse grouped coefficients defined by a partition 𝐺 ∈ 𝒫 can be developed as Projection on the feasible set of (6.21), that minimizes the above distance for all possible partitions in 𝑄 groups of 𝑠 non-zeros coefficients, amounts then to solve {𝑔 0 , . . . , 𝑔 𝑄 𝐺 } ∈ 𝒫, where 𝜇 𝑞 = 1

										𝑄 𝐺
		‖𝑥 -𝑤‖ 2 2 =	∑ 𝑖∈𝑔 0	𝑥 2 𝑖 +	∑ 𝑞=1	∑ 𝑖∈𝑔𝑞 (𝑥 𝑖 -𝑣 𝑞 ) 2 ,	(6.22)
	for given 𝐺 = 𝜇 𝑞 =	1 𝑠 𝑞	∑ 𝑖∈𝑔𝑞
			𝑄 𝐺							𝑑	𝑄 𝐺
	∑ 𝑖∈𝑔 0	𝑥 2 𝑖 +	∑ 𝑞=1	∑ 𝑖∈𝑔𝑞	(𝑥 2 𝑖 + 𝜇 2 𝑞 -2𝑣 𝑞 𝑥 𝑖 ) =	∑ 𝑖=1	𝑥 2 𝑖 -	𝑞=1 ∑	𝑠 𝑞 𝜇 2 𝑞 .
	maximize subject to Card ∑ 𝑄 𝐺 𝑞=1 𝑠 𝑞 𝜇 2 𝑞 ( ⋃ 𝑄 𝐺 𝑞=1 𝑔 𝑞	)	≤ 𝑠, 0 ≤ 𝑄 𝐺 ≤ 𝑄,	(6.23)
	in the partition 𝐺 = 𝑠𝑞 1.	∑ 𝑖∈𝑔𝑞 𝑥 𝑖 and 𝑄 𝐺 = Card(𝐺)-

  𝑞≤𝑖≤𝑗 𝜇(𝑥 𝑖 ,...,𝑥 𝑗 )>0𝑓 + (𝑖 -1, 𝑞 -1) + (𝑗 -𝑖 + 1)𝜇(𝑥 𝑖 , . . . , 𝑥 𝑗 ) 2 , 𝑥 𝑙 can be computed in constant time using that𝜇(𝑥 𝑖 , . . . , 𝑥 𝑗 ) = 𝑥 𝑖 + (𝑗 -𝑖)𝜇(𝑥 𝑖+1 , . . . , 𝑥 𝑗 ) 𝑗 -𝑖 + 1 . By convention 𝑓 + (𝑗, 𝑞) = -∞ ifis not possible to find 𝑞 clusters of positive barycenters with 𝑗 points such that (𝑃 + (𝑗, 𝑞)) is not feasible. Values of 𝑓 + are stored to compute (6.24). Two auxiliary variables 𝐼 + and 𝑣 + store respectively the indexes of the largest value of 𝑥 in group 𝑔 𝑞 and the barycenter of the group 𝑔 𝑞 . The same dynamic program can be used to compute 𝑓 -, 𝐼 -and 𝑣 -, defined similarly as 𝐼 + and 𝑣 + , by reversing the order of the values of 𝑥. A grid search on 𝑓 (𝑗, 𝑞, 𝑠 ′ ) = 𝑓 + (𝑗, 𝑞) + 𝑓 -(𝑠 ′ -𝑗, 𝑄 ′ -𝑞), with 𝑄 ′ = min(𝑠 ′ , 𝑄), gives the optimal balance between positive and negative barycenters. A backtrack on 𝐼 -and 𝐼 + finally gives the best partition and the projection with the associated barycenters given in 𝑣 -and 𝑣 + .

		(6.25)
	where 𝜇(𝑥 𝑖 , . . . , 𝑥 𝑗 ) = 1 𝑗-𝑖+1	∑ 𝑗 𝑙=𝑖

Table 6 .

 6 1 -Measure of ‖𝑤 * -ŵ‖ 2 , the 𝑙 2 norm of the difference between the true vector of weights 𝑤 * and the estimated ones ŵ along number of samples 𝑛.

		𝑛 = 50	𝑛 = 75	𝑛 = 100	𝑛 = 125	𝑛 = 150
	Oracle	0.16±0.06	0.14±0.04	0.10±0.04	0.10±0.04 0.09±0.03
	LS	61.94±17.63 51.94±16.01 21.41±9.40 1.02±0.18 0.70±0.09
	LSK	62.93±18.05 57.78±17.03 10.18±14.96 0.31±0.19 0.19±0.12
	IHC	63.31±18.24 52.72±16.51 5.52±14.33 0.14±0.09 0.09±0.04
	CG	61.81±17.78 52.59±16.58 17.24±13.87 1.20±1.38 1.05±1.37
	CGIHC 62.29±18.15 50.15±17.43 0.64±2.03 0.15±0.19 0.17±0.53
	OS	61.54±17.59 52.87±15.90 11.32±7.03 1.25±0.28 0.71±0.10
		𝜎 = 0.05	𝜎 = 0.1	𝜎 = 0.5		𝜎 = 1
	Oracle	0.86±0.27	1.72±0.54	8.62±2.70	17.19±5.43
	LS	7.04±0.92	14.05±1.82	70.39±9.20	140.41±18.20
	LSK	1.44±0.46	2.88±0.91	19.10±12.13	48.09±27.46
	IHC	0.87±0.27	1.74±0.52	9.11±4.00	26.23±18.00
	CG	23.91±36.51 122.31±145.77 105.45±136.79 155.98±177.69
	CGIHC 1.52±3.13 140.83±710.32 17.34±53.31	24.80±16.32
	OS	14.43±2.45	18.89±3.46	71.00±10.12	140.33±18.83
	Table 6.2 -Measure of ‖𝑤

* -ŵ‖ 2

Table 6 .

 6 4 -Clustering of words on movie reviews. We show clusters of words within which associated predictor weights have largest magnitude. First row presents ones associated to a negative coefficient and therefore bad feelings about movies, second row ones to a positive coefficient and good feelings about movies.

	2 most	bad, awful,
	negative	worst, boring, ridiculous,
	clusters	watchable, suppose, disgusting,
	2 most	perfect,hilarious,fascinating,great
	positive	wonderfully,perfectly,goodspirited,
	clusters world, intelligent,wonderfully,unexpected,gem,recommendation,
		excellent,rare,unique,marvelous,good-spirited,
		mature,send,delightful,funniest

1

  𝑛∑ 𝑛 𝑖=1 ℓ(𝑤 𝑇 𝑥 𝑖 , 𝑦 𝑖 ). In order to prevent over-fitting, a regularizer 𝑅(𝑤) of the predictor 𝑤 is added such as its squared Euclidean norm, i.e. 𝑇 𝑥 𝑖 , 𝑦 𝑖 ) + 𝜆𝑅(𝑤) in variable 𝑤 ∈ R 𝑑 , where 𝜆 ≥ 0 is a regularization parameter. Affine regression problems can be treated similarly as detailed in Section 6.1.1.Here we search for 𝑄 linear prediction vectors 𝑣 1 , . . . , 𝑣 𝑄 ∈ R 𝑑 , each can be used to predict observations of a given sample. Samples can then be partitioned in 𝑄 groups 𝑔 1 , . . . , 𝑔 𝑄 by assigning for each of them the best predictor. When it comes to compute best predictors 𝑣 1 , . . . , 𝑣 𝑄 on the training data, we face therefore a mix of a partitioning problem and empirical loss minimization. We weight the regularization of each predictor by the number of samples in the group it defines in order to ease the development of our algorithms. Our clustered regression problem reads therefore 𝑑 , where 𝑠 𝑞 = Card(𝑔 𝑞 ) is the size of group 𝑔 𝑞 and 𝜆 ≥ 0 is a regularization parameter.By denoting 𝑤 𝑖 ∈ R 𝑑 the prediction vector used for sample 𝑖 such that if 𝑖 ∈ 𝑔 𝑞 , 𝑤 𝑖 = 𝑣 𝑞 , our problem can be rewritten 𝑤 𝑖 ) subject to 𝑤 𝑖 = 𝑣 𝑞 if 𝑖 ∈ 𝑔 𝑞 , for all 𝑖 ∈ {1, . . . , 𝑛}(7.2)in the partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) of {1, . . . , 𝑛} and prediction vectors 𝑤 1 , . . . , 𝑤 𝑛 ∈ R 𝑑 and 𝑣 1 , . . . , 𝑣 𝑄 ∈ R 𝑑 , where 𝜆 ≥ 0 is a regularization parameter. A partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) of {1, . . . , 𝑛} can then be represented by an assignment matrix 𝑍 ∈ {0, 1} 𝑛×𝑄 such that 𝑖𝑞 𝑣 𝑞 for all 1 ≤ 𝑖 ≤ 𝑛, where 𝑍 is the assignment matrix of a partition. Denoting 𝑊 = (𝑤 1 , . . . , 𝑤 𝑛 ) ∈ R 𝑑×𝑛 the matrix of individual classifiers, 𝐿(𝑊 ) = 1 𝜆𝑅(𝑊 ) subject to 𝑊 = 𝑉 𝑍 𝑇 , 𝑍 ∈ {0, 1} 𝑛×𝑄 , 𝑍1 = 1,

	𝑅 square (𝑤) = 1 2 ‖𝑤‖ 2 . The regression problem is then
	minimize ℓ(𝑤 minimize 𝑛 1 𝑛 ∑ 𝑖=1 𝑄 1 𝑞=1 𝑖∈𝑔𝑞 𝑛 ∑ ∑ ℓ ( 𝑣 𝑇 𝑞 𝑥 𝑖 , 𝑦 𝑖	)	+ 𝜆
	minimize 1 𝑛	∑ 𝑛 𝑖=1 ℓ 𝑖=1 𝑅(𝑍 𝑖𝑞 = ( 𝑤 𝑇 𝑖 𝑥 𝑖 , 𝑦 𝑖 ) + 𝜆 ∑ 𝑛 { 1 if 𝑖 ∈ 𝑔 𝑞 0 otherwise.
	Partitioning constraints of (7.2) read then 𝑤 𝑖 = 𝑞=1 𝑍 𝑛 ∑ 𝑄 ∑ 𝑛 𝑖=1 ℓ ( 𝑤 𝑇 𝑖 𝑥 𝑖 , 𝑦 𝑖	)	and 𝑅(𝑊 ) =	∑ 𝑛 𝑖=1 𝑅(𝑤 𝑖 ),
	clustered regression problem finally reads
	minimize 𝐿(𝑊 ) +

𝑄 ∑ 𝑞=1 𝑠 𝑞 𝑅(𝑣 𝑞 ) (7.1) in the partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) of {1, . . . , 𝑛} and prediction vectors 𝑣 1 , . . . , 𝑣 𝑄 ∈ R

1

  𝑛∑ 𝑛 𝑖=1 ℓ(𝑊 𝑇 𝑥 𝑖 , 𝑦 𝑖 ). As for regression, a regularizer 𝑅(𝑊 ) can be added on the linear classifiers such as their squared euclidean norm 𝑅 square (𝑊 ) = 1 𝑑×𝐾 , where 𝜆 ≥ 0 is a regularization parameter.As for regression, we search not one but several matrices of classifiers 𝑉 1 , . . . , 𝑉 𝑄 ∈ R 𝑑×𝐾 that, for a given sample, outputs diverse estimated classes. Number of possible classifiers 𝑄 does naturally not exceed the number of classes 𝐾. Samples are then partitioned in 𝑄 groups by assigning the best matrix of classifier for each sample. 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) of {1, . . . , 𝑛} and matrices of classifiers 𝑉 1 , . . . , 𝑉 𝑄 ∈ R 𝑑×𝐾 , where 𝑠 𝑞 = Card(𝑔 𝑞 ) is the size of group 𝑔 𝑞 and 𝜆 ≥ 0 is a regularization parameter.By denoting 𝑊 𝑖 the matrix of classifiers used for sample 𝑖 such that if 𝑖 ∈ 𝑔 𝑞 , 𝑊 𝑖 = 𝑉 𝑞 and by using assignment matrices to represent partitions, clustered classification

	problems read then							2	∑ 𝐾 𝑘=1 ‖𝑤 𝑘 ‖ 2 2 = 1 2 ‖𝑊 ‖ 2 𝐹 . Classification
		minimize	1 𝑛
														Us-
	ing same weighted regularization as for regression, our clustered classification problem
	reads	minimize	1 𝑛	𝑄 𝑞=1 ∑	𝑖∈𝑔𝑞 ∑	ℓ	(	𝑉 𝑇 𝑞 𝑥 𝑖 , 𝑦 𝑖	)	+ 𝜆	𝑄 𝑞=1 ∑	𝑠 𝑞 𝑅(𝑉 𝑞 )	(7.4)
	in the partition problem reads
		minimize 1 𝑛 subject to 𝑊 𝑖 = ∑ 𝑛 𝑖=1 ℓ ∑ 𝑄 ( 𝑞=1 𝑍 𝑖𝑞 𝑉 𝑞 , for all 𝑖 ∈ 1, 𝑛 𝑊 𝑇 𝑖 𝑥 𝑖 , 𝑦 𝑖 ) + 𝜆 ∑ 𝑛 𝑖=1 𝑅(𝑊 𝑖 ) 𝑍 ∈ {0, 1} 𝑛×𝑄 , 𝑍1 = 1,

𝑛 ∑ 𝑖=1 ℓ(𝑊 𝑇 𝑥 𝑖 , 𝑦 𝑖 ) + 𝜆𝑅(𝑊 ) in variable 𝑊 ∈ R

  Formulation (7.3) enables to develop projected gradient descent on the individual prediction vectors represented by the variable 𝑊 . Projection of a given 𝑊 = (𝑤 1 , . . . , 𝑤 𝑛 ) ∈ R 𝑑×𝑛 on the feasible set of (7.3) reads 1} 𝑛×𝑄 , 𝑍1 = 1 in variable 𝑉 = (𝑣 1 , . . . , 𝑣 𝑄 ) ∈ R 𝑑×𝑄 and 𝑍. We recognize a k-means problem on the columns of 𝑊 , i.e. , . . . , 𝑣 𝑄 ∈ R 𝑑 that are the centroids of the clusters and 𝐺 = {𝑔 1 , . . . , 𝑔 𝑄 } a partition of {1, . . . , 𝑛}. Careful initialization as made in k-means++[START_REF] Arthur | k-means++: The advantages of careful seeding[END_REF] offers logarithmic approximations to the problem such that approx-Projected gradient descent for clustered regression Inputs: Data (𝑋, 𝑌 ), desired number of groups 𝑄, step size 𝛾 𝑡 Initialize 𝑊 0 ∈ R 𝑑×𝐾 for t = 1,. . . ,T do 𝑊 𝑡+1/2 = 𝑊 𝑡 -𝛾 𝑡 (∇𝐿(𝑊 𝑡 ) + ∇𝑅(𝑊 𝑡 )) [𝑍 𝑡+1 , 𝑉 𝑡+1 ] = k-means(𝑊 𝑡+1/2 , 𝑄) 𝑊 𝑡+1 = 𝑉 𝑡+1 𝑍 𝑇

	minimize ‖𝑊 -𝑉 𝑍 𝑇 ‖ 2 𝐹 = subject to 𝑍 ∈ {0, minimize 𝑄 ∑ 𝑞=1 ∑ ∑ 𝑛 𝑖=1 ‖𝑤 𝑖 -𝑖∈𝑔𝑞 ‖𝑤 𝑖 -𝑣 𝑞 ‖ 2 2 , ∑ 𝑄 𝑞=1 𝑍 𝑖𝑞 𝑣 𝑞 ‖ 2 2 in variables 𝑣 1 𝑡+1 end for Output: Ŵ = 𝑊 𝑇	(7.6)

imate projection is available. Given a matrix 𝑊 , whose columns we want to cluster in 𝑄 groups, we denote by [𝑍, 𝑉 ] = k-means(𝑊, 𝑄) respectively the assignment matrix and the matrix of centroids output by a clustering algorithm. A projected gradient scheme for problem (7.3) is described in Algorithm 8.

Algorithm 8

  Using that 𝐸 = (𝐸 1 ; . . . ; 𝐸 𝑄 ) = (𝐸 𝑇 1 , . . . , 𝐸 𝑇 𝑄 ) 𝑇 ∈ {0, 1}

	𝛼𝑞∈R 𝑠𝑞	𝑄 ∑ 𝑞=1	-𝛼 𝑇 𝑞 (I +	1 𝑠 𝑞 𝜆𝑛	𝑋

𝛼=(𝛼 1 ;...;𝛼 𝑄 ) 𝑞 𝑋 𝑇 𝑞 )𝛼 𝑞 + 2𝑦 𝑇 𝑞 𝛼 𝑞 , where (𝛼 1 ; . . . ; 𝛼 𝑄 ) = (𝛼 𝑇 1 , . . . , 𝛼 𝑇 𝑄 ) 𝑇 stacks vectors 𝛼 𝑞 in one vector of size ∑ 𝑄 𝑞=1 𝑠 𝑞 = 𝑛.

  𝑤𝑖𝑡ℎ𝑖𝑛 (𝑊, 𝑉, 𝑍) = 1 2 Tr(𝑉 𝑍 𝑇 (I 𝐾 -Π 𝐾 )𝑍𝑉 𝑇 ), Tr(𝑉 𝑍 𝑇 Π𝑍𝑉 𝑇 ), Using 𝑈 = 𝑉 𝑍 𝑇 , the total penalty can then be written 𝐾 -Π 𝐾 )𝑈 𝑇 ) + 𝜆 𝑚 2 Tr(𝑈 Π 𝐾 𝑈 𝑇 ). (8.1) where 𝜆 𝑤 , 𝜆 𝑏 , 𝜆 𝑚 are non-negative regularization parameters. Overall, clustered multitask learning problem reads then minimize 𝐿(𝑊 ) + Ω(𝑊, 𝑈 ) s.t. 𝑈 = 𝑉 𝑍 𝑇 , 𝑍 ∈ {0, 1} 𝐾×𝑄 , 𝑍1 = 1, (8.2) in variables 𝑊 ∈ R 𝑑×𝐾 , 𝑈 ∈ R 𝑑×𝐾 , 𝑉 ∈ R 𝑑×𝑄 and 𝑍. This formulation exhibits a convex objective under non-convex constraints that encode the clustering problem. Several optimization procedures can then be used to treat these constraints. 8.2 Projected gradient descent Formulation (8.2) enables to develop projected gradient descent on the auxiliary variable 𝑈 . Projection of a given 𝑈 = (𝑢 1 , . . . , 𝑢 𝐾 ) ∈ R 𝑑×𝐾 on the feasible set of (8.2) reads minimize ‖𝑈 -𝑉 𝑍 𝑇 ‖ 2 𝐹 = ∑ 𝐾 𝑖=1 ‖𝑢 𝑖 -∑ 𝑄 𝑞=1 𝑍 𝑖𝑞 𝑣 𝑞 ‖ 2 2 subject to 𝑍 ∈ {0, 1} 𝐾×𝑄 , 𝑍1 = 1 in variable 𝑉 = (𝑣 1 , . . . , 𝑣 𝑄 ) ∈ R 𝑑×𝑄 and 𝑍. We recognize a k-means problem on the columns of 𝑈 , i.e. 𝑣 1 , . . . , 𝑣 𝑄 ∈ R 𝑑 that are the centroids of the clusters and 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) a partition of {1, . . . , 𝐾}. Careful initialization as made in k-means++

	𝑄
	𝑄 ∑ 𝑞=1 𝑄 ∑ 𝑞=1 1 2 ||v|| 2 ∑ 𝑖∈𝑔𝑞 𝑠 𝑞 ||𝑣 𝑞 -v|| 2 ||𝑤 𝑖 -𝑣 𝑞 || 2 2 = 2 = 𝜆 𝑏 2 2 = 𝜆 𝑚 2 ||𝑊 -𝑈 || 2 Ω 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 (𝑉, 𝑍) = 1 2 Ω 𝑚𝑒𝑎𝑛 (𝑉, 𝑍) = 𝐾 Ω(𝑊, 𝑈 ) = 𝜆 𝑤 2 𝐹 + 𝜆 𝑏 2 ∑ 𝑞=1 ∑ 𝑖∈𝑔𝑞 ‖𝑢 𝑖 -𝑣 𝑞 ‖ 2 𝜆 𝑤 2 ||𝑊 -𝑉 𝑍 𝑇 || 2 𝐹 . 2 , Tr(𝑈 (I minimize in variables	(8.3)

  Tr(𝑉 𝑍 𝑇 (I 𝐾 -Π 𝐾 )𝑍𝑉 𝑇 ) + 𝜆 𝑚 2 Tr(𝑉 𝑍 𝑇 Π 𝐾 𝑍𝑉 𝑇 ) 𝜆 𝑏 )𝑍 𝑇 𝑍 + (𝜆 𝑚 -𝜆 𝑏 )𝑍 𝑇 Π 𝐾 𝑍 ) 𝑉 𝑇 ) -𝜆 𝑤 Tr(𝑉 𝑇 𝑊 𝑍) (𝜆 𝑤 + 𝜆 𝑏 )𝑍 𝑇 𝑍 + (𝜆 𝑚 -𝜆 𝑏 )𝑍 𝑇 Π 𝐾 𝑍 ) -1 𝑍 𝑇 𝑊 𝑇

	𝑉 = min 𝑉 (𝜆 𝑤 + + 𝜆 𝑤 2 ‖𝑊 -𝑉 𝑍 𝑇 ‖ 2 𝐹 + 1 2 Tr ( 𝑉 ( 𝜆 𝑤 2 ‖𝑊 ‖ 2 𝐹	𝜆 𝑏 2
	= -	𝜆 2 𝑤 2	Tr	(	𝑊 𝑍	(	)	+	𝜆 𝑤 2	‖𝑊 ‖ 2 𝐹 .

  𝑠 𝑞 and 𝑠 -1 2 the vector whose coordinates are 1 √ 𝑠 𝑞 if 𝑠 𝑞 ̸ = 0 and 0 otherwise. Inversion in the precedent formula reads𝐽 -1 = ( (𝜆 𝑤 + 𝜆 𝑏 )𝑍 𝑇 𝑍 + (𝜆 𝑚 -𝜆 𝑏 )𝑍 𝑇 Π 𝐾 𝑍 ) -1 = ( (𝜆 𝑤 + 𝜆 𝑏 ) diag(𝑠) + (𝜆 𝑚 -𝜆 𝑏 ) 𝜆 𝑏 (𝑍 𝑇 𝑍) † -𝜆 𝑚 -𝜆 𝑏 (𝜆 𝑤 + 𝜆 𝑚 )(𝜆 𝑤 + 𝜆 𝑏 )where we used in 𝜗 1 that 𝑍 𝑇 𝑍 = diag(𝑠) and in 𝜗 2 that 𝑃 = 𝑠 𝛼𝑃 ) -1 = 𝐼 -𝛼 𝛼+1 𝑃 . Now introducing the equivalence matrix 𝑀 = 𝑍(𝑍 𝑇 𝑍) † 𝑍 𝑇 , and using that𝑍 𝐾 -𝜌 𝑤𝑏 (𝑀 -Π 𝐾 ) -𝜌 𝑤𝑚 Π 𝐾 ) 𝑊 𝑇 ) ,where 𝜌 𝑤𝑏 = 𝜆𝑤 𝜆𝑤+𝜆 𝑏 and 𝜌 𝑤𝑚 = 𝜆𝑤 𝜆𝑤+𝜆𝑚 .The squared loss has now the advantage to provide analytic solutions for multitask problems in the variable 𝑊 which simplifies clustered multitask problem (8.2) as shown in following proposition. Training data is there represented by the design matrix 𝑋 = (𝑥 1 , . . . , 𝑥 𝑛 ) 𝑇 ∈ R 𝑛×𝑑 and the matrix of labels (𝑦 1 , . . . , 𝑦 𝑛 ) 𝑇 ∈ R 𝑛×𝐾 . Proposition 8.3.2. Clustered multitask problem (8.2) with regularization parameters 𝜆 𝑚 , 𝜆 𝑏 and 𝜆 𝑤 is equivalent to minimize Tr(𝐴 𝑇 𝑀 ) subject to 𝑀 ∈ ℳ (8.4)where𝐴 = 𝑌 𝑇 𝑋 ( (𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 I 𝑑 ) -1 -(𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑏 )I 𝑑 ) -1 ) 𝑋 𝑇 𝑌, (8.5) 𝜌 𝑤𝑏 = 𝜆𝑤 𝜆𝑤+𝜆 𝑏 and ℳ = {𝑀 : 𝑀 = 𝑍(𝑍 𝑇 𝑍) -1 𝑍 𝑇 , 𝑍 ∈ {0, 1} 𝑛×𝑄 , 𝑍1 = 1} is the set of equivalence matrices.Proof. Using a squared loss and Kronecker's product formula, we get for a given matrix of classifier 𝑊= (𝑤 1 , . . . , 𝑤 𝐾 ) ∈ R 𝑑×𝐾 , 𝑇 (I 𝐾 ⊗ 𝑋 𝑇 𝑋) Vect(𝑊 ) -1 𝑛 Vect(𝑊 ) 𝑇 Vect(𝑋 𝑇 𝑌 ) + 𝑇 𝑌 ) 𝑇 ( I 𝐾 ⊗ 𝑋 𝑇 𝑋 + 𝜆 𝑤 𝑛𝑃 ⊗ I 𝑑 ) -1 Vect(𝑋 𝑇 𝑌 ) 𝐹 Denote 𝑣 1 , ..., 𝑣 𝑑 ∈ R 𝑑 ,𝜆 1 , ..., 𝜆 𝑑 ∈ R and 𝑢 1 , ..., 𝑢 𝐾 ∈ R 𝐾 , 𝜇 1 , ..., 𝜇 𝐾 ∈ R the eigenvectors and corresponding eigenvalues respectively of matrices 𝑋 𝑇 𝑋 and 𝑃 = (I 𝐾 -𝜌 𝑤𝑏 (𝑀 -Π 𝐾 ) + 𝜌 𝑤𝑚 Π 𝐾 ). The eigenvectors and corresponding eigenvalues of I 𝐾 ⊗ 𝑋 𝑇 𝑋 +𝜆 𝑤 𝑛𝑃 ⊗I 𝑑 are (𝑢 𝑖 ⊗𝑣 𝑗 ) 𝑖∈ 1,𝑛 , 𝑗∈ 1,𝑑 and (𝜆 𝑤 𝑛𝜇 𝑖 +𝜆 𝑗 ) 𝑖∈ 1,𝑛 , 𝑗∈ 1,𝑑 . The inversion in the expression of 𝐺 is then given by𝐽 -1 = ( I 𝐾 ⊗ 𝑋 𝑇 𝑋 + 𝜆 𝑤 𝑛𝑃 ⊗ I 𝑑 ) -1 = 𝜆 𝑤 𝑛𝜇 𝑖 + 𝜆 𝑗 𝑢 𝑖 𝑢 𝑇 𝑖 ⊗ 𝑣 𝑗 𝑣 𝑇 𝑗 .We then note that the set of eigenvectors of 𝑃 can be decomposed into three sets. Matrices I 𝐾 -𝑀, 𝑀 -Π 𝐾 and Π 𝐾 are indeed orthogonal projectors on orthogonal subspaces spanning the entire space. Denote by ℐ 𝑤 , ℐ 𝑏 , ℐ 𝑚 the sets of eigenvectors corresponding respectively to I 𝐾 -𝑀 , 𝑀 -Π 𝐾 and Π 𝐾 , their corresponding eigenvalues in 𝑃 can easily be computed and we obtain𝑃 = I 𝐾 -𝑀 + (1 -𝜌 𝑤𝑏 )(𝑀 -Π 𝐾 ) + (1 -𝜌 𝑤𝑚 )Π 𝐾 = ∑ 𝑖∈ℐ𝑤 𝑢 𝑖 𝑢 𝑇 𝑖 + (1 -𝜌 𝑤𝑏 ) ∑ 𝑖∈ℐ 𝑏 𝑢 𝑖 𝑢 𝑇 𝑖 + (1 -𝜌 𝑤𝑚 ) 𝜆 𝑤 𝑛 + 𝜆 𝑗 𝑢 𝑖 𝑢 𝑇 𝑖 ⊗ 𝑣 𝑗 𝑣 𝑇 𝜆 𝑤 𝑛(1 -𝜌 𝑤𝑏 ) + 𝜆 𝑗 𝑢 𝑖 𝑢 𝑇 𝑖 ⊗ 𝑣 𝑗 𝑣 𝑇 𝜆 𝑤 𝑛(1 -𝜌 𝑤𝑚 ) + 𝜆 𝑗 𝑢 𝑖 𝑢 𝑇 𝑖 ⊗ 𝑣 𝑗 𝑣 𝑇 𝑗 = (I 𝐾 -𝑀 ) ⊗ (𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 I 𝑑 ) -1 + (𝑀 -Π 𝐾 ) ⊗ (𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑏 )I 𝑑 ) -1 + Π 𝐾 ⊗ (𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑚 )I 𝑑 ) -1 . 𝑇 𝑌 ) 𝑇 ( (I 𝐾 -𝑀 ) ⊗ (𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 I 𝑑 ) -1 ) Vect(𝑋 𝑇 𝑌 ) 𝑇 𝑌 ) 𝑇 ( (𝑀 -Π 𝐾 ) ⊗ (𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑏 )I 𝑑 ) -1 )Vect(𝑋 𝑇 𝑌 ) 𝑇 𝑌 ) 𝑇 ( Π 𝐾 ⊗ (𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑚 )I 𝑑 ) -1 ) 𝑇 𝑋(𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 I 𝑑 ) -1 𝑋 𝑇 𝑌 (I 𝐾 -𝑀 )) 𝑇 𝑋(𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑏 )I 𝑑 ) -1 𝑋 𝑇 𝑌 (𝑀 -Π 𝐾 )) 𝑇 𝑋(𝑋 𝑇 𝑋 + 𝑛𝜆 𝑤 (1 -𝜌 𝑤𝑚 )I 𝑑 ) -1 𝑋 𝑇 𝑌 Π 𝐾 ) +

	coordinates are This decomposition can be used for the inversion √ 𝑠𝑠 𝑇	) -1	∑ 𝑖∈ℐ𝑚	𝑢 𝑖 𝑢 𝑇 𝑖 .
		𝐾
	𝜗 1 = ∑ 𝑑 𝜗 2 = = = projector and therefore (𝐼 + 1 𝑄 1 𝑇 1 𝜆 𝑤 + 𝜆 𝑏 diag(𝑠 -1 2 ) ( I 𝑄 + 𝜆 𝑚 -𝜆 𝑏 𝜆 𝑤 + 𝜆 𝑏 𝑠 1 2 𝑠 1 2 𝑇 𝐾 ) -1 diag(𝑠 -1 ∑ 1 𝐽 -1 = 𝑗 2 ) 1 𝜆 𝑤 + 𝜆 𝑏 diag(𝑠 -1 2 ) ( I 𝑄 -𝜆 𝑚 -𝜆 𝑏 𝜆 𝑤 + 𝜆 𝑚 𝑠 1 2 𝑠 1 2 𝑇 𝑚 ) diag(𝑠 -1 2 ) 1 𝜆 𝑤 + 1 𝑄 1 𝑇 𝑄 𝐾 1 𝜆 𝑤 + 𝜆 𝑏 ( (𝑍 𝑇 𝑍) † -1 𝑄 1 𝑇 𝑄 𝐾 ) + 1 𝜆 𝑤 + 𝜆 𝑚 1 𝑄 1 𝑇 𝑄 𝐾 , 1 2 𝑠 1 2 𝑇 𝐾 = 𝑠 1 2 𝑠 1 2 𝑇 ‖𝑠 1 2 ‖ 2 2 is a 𝑄 𝑚 𝑍 𝑇 = Π 𝐾 , we finally obtain min 𝑉 Ω(𝑊, 𝑉 𝑍 𝑇 ) = 𝜆 𝑤 2 Tr(𝑊 𝑊 𝑇 ) -𝜆 2 𝑤 2 Tr ( 𝑊 ( 1 𝜆 𝑤 + 𝜆 𝑏 (𝑀 -Π 𝐾 ) + 1 𝜆 𝑤 + 𝜆 𝑚 Π 𝐾 ) 𝑊 𝑇 ) = 𝜆 𝑤 2 Tr ( 𝑖∈ℐ 𝑊 𝑗=1 1 2𝑛 ‖𝑋𝑊 -𝑌 ‖ 2 𝐹 = 1 2𝑛 Tr(𝑊 𝑇 𝑋 𝑇 𝑋𝑊 ) -1 𝑛 Tr(𝑊 𝑇 𝑋 𝑇 𝑌 ) + 1 2𝑛 ‖𝑌 ‖ 2 𝐹 = 1 2𝑛 Vect(𝑊 ) 1 2𝑛 ‖𝑌 ‖ 2 𝐹 . Simplified clustering penalty computed in Lemma 8.3.1 can also be rewritten using Kronecker's product formula as min + ∑ 𝑖∈ℐ 𝐵 𝑑 ∑ 𝑗=1 1 𝑗 + ∑ 𝑖∈ℐ 𝑊 𝑑 ∑ 𝑗=1 1 Finally clustered multitask problem can be simplified using properties of the Kro-necker product 𝐴 = min 𝑊,𝑉 𝐿(𝑊 ) + Ω(𝑊, 𝑉 𝑍 𝑇 ) 𝑊 (I 𝐿(𝑊 ) = -1 𝑛 Vect(𝑊 ) 𝑇 Vect(𝑋 𝑇 𝑌 ) + 1 2𝑛 ‖𝑌 ‖ 2 𝐹 = -1 2𝑛 Vect(𝑋 + 1 2𝑛 ‖𝑌 ‖ 2 = -1 2𝑛 Vect(𝑋 -1 2𝑛 Vect(𝑋 -1 2𝑛 Vect(𝑋 Vect(𝑋 𝑇 𝑌 ) + 1 2𝑛 𝐹 ‖𝑌 ‖ 2 = -1 2𝑛 Tr(𝑌 -1 2𝑛 Tr(𝑌 -1 2𝑛 Tr(𝑌 1 2𝑛 ‖𝑌 ‖ 2 𝐹 .
	𝑛 ∑	𝑑 ∑	1
	𝑖=1	𝑗=1
					1 2 the vector whose
	143

𝑉

Ω(𝑊, 𝑉 𝑍

𝑇 ) = 𝜆 𝑤 2 Vect(𝑊 ) 𝑇 (𝑃 ⊗ I 𝑑 ) Vect(𝑊 ),

where

𝑃 = I 𝐾 -𝜌 𝑤𝑏 (𝑀 -Π 𝐾 ) -𝜌 𝑤𝑚 Π 𝐾 .

Problem (8.2) can then be simplified as min 𝑊,𝑉 𝐿(𝑊 ) + Ω(𝑊, 𝑉 𝑍 𝑇 ) = min 𝑊 1 2𝑛 Vect(𝑊 ) 𝑇 (I 𝐾 ⊗ 𝑋 𝑇 𝑋 + 𝑃 ⊗ I 𝑑 ) Vect(𝑊 )

Without loss of generality we assume 𝑄 ⊂ dom 𝑓

This means that the complexity of the optimization problem (ℓ 1 recovery) is controlled by the oversampling ratio 𝑛/𝑠. In other words, while increasing the number of samples increases the time complexity of elementary operations of the algorithm, it also increases its rate of convergence.

s.t. ‖𝑧 𝑆 𝑐 ‖ 1 ≤ -𝑢 𝑇 𝑧. (4.26)This convex, linearly constrained quadratic program is easily solved with CVX[START_REF] Grant | CVX: Matlab software for disciplined convex programming[END_REF]. As can be seen from KKT conditions, this iteration is a generalized power

Machine learning aims at analyzing data either to predict observations of future data or to reveal hidden information such as clusters. It has been applied in numerous fields such as computer vision, bio-informatics or economy (see[START_REF] Hastie | The elements of statistical learning: data mining, inference and prediction[END_REF] for an introduction of the field). While prediction and clustering problems are often analyzed independently, they can benefit from each other.On one side, if data are clustered beforehand a prediction problem can use this information[START_REF] Pfeffermann | Regression analysis of data from a cluster sample[END_REF][START_REF] Graubard | Regression analysis with clustered data[END_REF]. If data could be clustered but clusters are unknown then the prediction problem shall take this information into account. For example, in a regression problem where one half of the data 𝑥 with attributes 𝑦 satisfies 𝑦 ≈ 𝑥 and the other one satisfies 𝑦 ≈ -𝑥, a classical algorithm would predict 𝑦 ≈ 0 in average, while it may preferable to output two possible answers 𝑦 = 𝑥 or 𝑦 = -𝑥 which accounts for the diversity of the data. Note that, in this case, the problem can be seen as a special instance of subspace clustering problem[START_REF] Vidal | Subspace clustering[END_REF].

Remerciements

Appendix

2.A Rounding issues

We presented convergence bounds for real sequences of iterate counts (𝑡 𝑘 ) ∞ 𝑘=1 but in practice these are integer sequences. The following Lemma details the convergence of our schemes for an approximate choice t𝑘 = ⌈𝑡 𝑘 ⌉.

Appendix

4.A Practical optimal restart scheme

In Section 4.1.2 we quickly give optimal restart schemes in terms of a potentially non-integer clock. Following corollary details optimal scheme for an integer optimal clock. Corollary 4.A.1. Given a coding matrix 𝐴 ∈ R 𝑛×𝑝 and an original signal 𝑥 * ∈ R 𝑝 such that sharpness bound (Sharp) is satisfied with 𝛾 > 0, running Algorithm Restart with 𝜌 * and 𝑡 = ⌈𝑡 * ⌉ where 𝜌 * and 𝑡 * are defined in (4.4) ensures that after 𝐾 ≥ 1 restarts, i.e. 𝑁 = 𝐾⌈𝑡 * ⌉ total number of iterations,

Proof. Denote 𝛿 = ⌈𝑡 * ⌉ -𝑡 * ∈ [0, 1[. As ⌈𝑡 * ⌉ ≥ 𝑡 * (4.3) is ensured for 𝜌 * . At the 𝐾 th restart, 𝑁 = 𝐾(𝑡 * + 𝛿), and ‖ŷ‖ 1 -‖𝑥 * ‖ 1 ≤ 𝑒 -𝐾 𝜀 0 = exp(-𝑁/(𝑡 * + 𝛿)) ≤ exp(-𝑁/(𝑡 * + 1)).

Replacing 𝑡 * by its value gives the result.

4.B Remark on sparsity inducing norms

We quickly discuss the framework of [START_REF] Juditsky | On a unified view of nullspacetype conditions for recoveries associated with general sparsity structures[END_REF] for sparsity inducing norms and show that it can be simplified. We first recall the definition. Definition 4.B.1. (Sparsity structure [START_REF] Juditsky | On a unified view of nullspacetype conditions for recoveries associated with general sparsity structures[END_REF]) A sparsity structure on a Euclidean space 𝐸 is defined as a norm ‖ • ‖ on 𝐸, together with a family 𝒫 of linear maps of 𝐸 into itself, satisfying three assumptions:

1. Every 𝑃 ∈ 𝒫 is a projector, 𝑃 2 = 𝑃 , 2. Every 𝑃 ∈ 𝒫 is assigned a weight 𝜈(𝑃 ) ≥ 0 and a linear map P on 𝐸 such that 𝑃 P = 0, ⟨∇𝑓 (𝑥 𝑡 ), 𝑠⟩ (5.4)

For any

Get estimated gap

The key observation is that, in our problems, we do have access to a linear minimization oracle that amounts to a k-means problem.

97

Chapter 6

Grouping features for prediction with partitioning constraints

Chapter Abstract

In a prediction problem, grouping features can improve performance, robustness and interpretation of the results. Here we propose to find the best partition of the features for a task by constraining the prediction vector to have a small number of values. We formulate our model for classification and regression and present algorithmic schemes to tackle it. First, we develop a convex relaxation for squared losses using conditional gradient descent to handle the underlying combinatorial problem. Then, we propose a projected gradient scheme that amounts to iteratively cluster the features at each gradient step. We provide a theoretical analysis of this method to recover the information needed for prediction, based on a union of subspaces interpretation of the partitioning structure. We extend these results to combine sparsity and grouping constraints, and develop a new projection algorithm on the set of grouped sparse vectors. Numerical experiments illustrate the performance of our algorithms on synthetic and real data.

Appendix

6.A Geometric interpretation of algebraic tools

In Proposition 6.4.3 we defined subspaces from partitions of{1, . . . , 𝑑}. Here we relate assignment and normalized equivalence matrices to these subspaces. First for a partition 𝐺 = (𝑔 1 , . . . , 𝑔 𝑄 ) into 𝑄 groups, 𝑤 ∈ 𝐸 𝐺 has at most 𝑄 different coefficients and can be encoded using assignment matrices as presented in Section 6.1. In other words, for an assignment matrix 𝑍 of 𝐺, one has

Columns of 𝑍 are orthogonal since since groups are disjoints and not null if 𝐺 has no empty groups. In this case, 𝑍 is therefore an orthogonal basis of 𝐸 𝐺 . The normalized equivalence matrix of 𝐺, that reads 𝑀 = 𝑍(𝑍 𝑇 𝑍) -1 𝑍 𝑇 is then the orthogonal projector on 𝐸 𝐺 .

As mentioned in Section 5.2, several assignment matrices can encode a partition, i.e. several binary matrices form a basis of a subspace 𝐸 𝐺 . However 𝐸 𝐺 and its orthogonal projector, i.e. the normalized equivalence matrix, are for their part uniquely defined by 𝐺.

To represent more generally partitions of {1, . . . , 𝑑} in any number of groups one can use binary matrices 𝑍 ∈ {0, 1} 𝑑×𝑑 that satisfy 𝑍1 = 1. Number of nonzero columns of such matrices are then the number of groups of the partition they represent. Once again partitions can be represented by several assignment matrices but are in bijection with the set of normalized equivalence matrices

Number of groups of a partition 𝐺 is then equal to the rank of its normalized equivalence matrix (the dimension of 𝐸 𝐺 ), i.e. Card(𝐺) = Rank(𝑀 ) = Tr(𝑀 ), since 𝑀 is a projector.

6.B Norm for grouping features

In this section, we seek to develop a norm that induce groups of features by regularization rather than enforcing it by constraints as in (6.2) or (6.4). In Section 6.4, we highlighted that our framework constraints number of level sets of the variables, Chapter 8

Clustered multi-task

Chapter Abstract

Multi-task learning aim at solving several prediction tasks simultaneously with the hope that they can share information to improve overall performance. Based on previous clustered multi-task learning framework of [START_REF] Jacob | Group lasso with overlap and graph lasso[END_REF], we investigate how similar tasks can be clustered to exploit common information. To this end, a clustering penalty on the prediction vectors is added to the empirical loss minimization problem. Here, we isolate the clustering problem and treat it with various optimization strategies. First, we present a projected gradient descent whose core iteration amounts to a k-means problem on the predictors. Then, we show that in case of a squared loss, clustered multi-task reduces to a k-means problem. Numerical experiments detail the performance of our approach on text classification.

Appendix A

Classical algorithms implementation

We present here the classical algorithms for convex optimization that we restart. We present their general form to solve composite optimization problems of the form minimize

where 𝜑, 𝑔 are convex functions and 𝑔 is assumed simple in a sense that will be clarified later.

Algorithms used to solve this problem crucially depend on the smoothness assumption on the (sub)gradients of 𝜑. We assume here that they are Hölder continuous with respect to a given norm ‖ • ‖ on a set 𝐽, i.e. that there exists 𝑠 ∈ [0, 1] and 𝐿 > 0 such that ‖∇𝜑(𝑥) -∇𝜑(𝑦)‖ * ≤ ‖𝑥 -𝑦‖ 𝑠-1 , for every 𝑥, 𝑦 ∈ 𝐽, (Generic Smoothness)

where ‖ • ‖ is the dual norm of ‖ • ‖ and ∇𝜑(𝑥) denotes any subgradient of 𝜑 at 𝑥. To exploit the smoothness of 𝜑 with respect to a generic norm, we assume that we have access to a prox function ℎ with dom(𝑓 ) ⊂ dom(ℎ), strongly convex with respect to the norm ‖ • ‖ with convexity parameter equal to one, i.e.

We define the Bregman divergence associated to ℎ as, for given 𝑥, 𝑦 ∈ dom(ℎ),

such that we naturally have 𝐷 ℎ (𝑦; 𝑥) ≥ 1 2 ‖𝑥 -𝑦‖ 2 . For ℎ(𝑥) = 1 2 ‖𝑥‖ 2 2 , we get 𝐷 ℎ (𝑦; 𝑥) = 1 2 ‖𝑥 -𝑦‖ 2 2 , so we retrieve the Euclidean setting. The assumption that 𝑔 is "simple" can now be stated formally. Given 𝑥, 𝑦 ∈ dom(𝑓 ) and 𝜆 ≥ 0 we assume that we can solve min

either in a closed form or by some cheap computational procedure.

A.1 Universal fast gradient method

An optimal algorithm to solve the (Composite) problem is then the universal fast gradient method [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF]. It is detailed in Algorithm 12. Given a target accuracy 𝜀, it starts at a point 𝑥 0 and outputs after 𝑡 iterations a point 𝑥 𝒰(𝑥 0 , 𝜀, 𝑡), such that

where 𝐷 ℎ (𝑥; 𝑋 * ) = min 𝑥 * ∈𝑋 * 𝐷 ℎ (𝑥; 𝑥 * ) is the Bregman distance from 𝑥 to the set of minimizers and 𝑐 𝑠 = 2

5𝑠-2 𝑠 that we bounded as

is the optimal rate of convergence for 𝑠-smooth functions. In the Euclidean setting, ℎ = 1 2 ‖𝑥‖ 2 2 , 𝐷 ℎ (𝑦; 𝑥) = 1 2 ‖𝑥 -𝑦‖ 2 , such that we get the bound given in (2.17). The method does not need to know the smoothness parameters (𝑠, 𝐿), but the target accuracy 𝜀 is used to parametrize the algorithm. The universal fast gradient method requires an estimate 𝐿 0 of the smoothness parameter 𝐿 to start a line search on 𝐿. This line search is proven to increase the complexity of the algorithm by at most a constant factor plus a logarithmic term and ensures that the overall complexity does not depend on 𝐿 0 but on 𝐿. In our restart schemes we use a first estimate 𝐿 0 when running the algorithm for the first time and we use the last estimate found by the algorithm when restarting it.

Finally if 𝑋 * ̸ = ∅, the universal fast gradient method produces a convergent sequence of iterates. Therefore if the Łojasiewicz inequality is satisfied on a compact set 𝐾, it will be valid for all our iterates after perhaps reducing 𝜇.

A.2 Accelerated gradient method

The accelerated gradient method is a special instance of the universal fast gradient method when the function 𝜑 is known to be smooth (i.e. satisfies (Generic Smoothness) with 𝑠 = 2). In that case the optimal 𝜀 to run the Universal Fast Gradient method is 0 (otherwise it depends on the parameters of the function). Given an initial point 𝑥 0 , accelerated gradient method outputs, after 𝑡 iterations, a point 𝑥 𝒜(𝑥 0 , 𝑡) = 𝒰(𝑥 0 , 0, 𝑡) such that 𝑓 (𝑦) -𝑓 * ≤ 𝑐𝐿 𝑡 2 𝐷 ℎ (𝑥 0 , 𝑋 * ), where 𝐷 ℎ (𝑥; 𝑋 * ) = min 𝑥 * ∈𝑋 * 𝐷 ℎ (𝑥; 𝑥 * ) is the Bregman distance from 𝑥 to the set of minimizers and 𝑐 = 8. In the Euclidean setting, 𝐷 ℎ (𝑦; 𝑥) = 1 2 ‖𝑥 -𝑦‖ 2 , such that we get the bound given in (2.3). Here again smoothness parameter 𝐿 is found by a backtracking line search such that one only needs a first estimate of its value.

Algorithm 12 Universal fast gradient method

Inputs : 𝑥 0 , 𝐿 0 , 𝜀 Initialize : 𝑦 0 := 𝑥 0 , 𝐴 0 := 0, L := 𝐿 0 for 𝑡 = 0, . . . , 𝑇 do

A.3 Gradient descent method

We recall in Algorithm 13 the simple gradient descent method when the function 𝜑 is smooth with constant 𝐿. I can also be found in [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF]. It starts at a point 𝑥 0 and outputs iterates 𝑥 𝑡 = 𝒢(𝑥 0 , 𝑡) such that

where 𝑐 = 2 and 𝐷 ℎ (𝑥; 𝑋 * ) = min 𝑥 * ∈𝑋 * 𝐷 ℎ (𝑥; 𝑥 * ) is the Bregman distance from 𝑥 to the set of minimizers. In the Euclidean setting, 𝐷 ℎ (𝑦; 𝑥) = 1 2 ‖𝑥 -𝑦‖ 2 , such that we get the bound in (2.15). Once again it performs a line search on the smoothness parameter 𝐿 such that 𝐿 0 can be chosen arbitrarily.

List of Tables