Alexandre Gramfort

Jalal Fadili

Carola-Bibiane Schönlieb

Antonia Jérome

Maëliss, Hugo Merci À Hicham

Omar Hubert

Zaccharie Alexis Gaston

Louis Matthieu D Bénédicte

Tackling modern neurological diseases such as Alzheimer or Parkinson appears as a major challenge for which neuroscientists extensively rely on advanced recording and processing of brain activity. Due to non-invasiveness and excellent time resolution, magneto-and electroencephalography (M/EEG) have emerged as tools of choice to monitor brain activity. Reconstructing brain signals from M/EEG measurements can be cast as a high dimensional ill-posed inverse problem. Usual estimators of brain signals involve challenging composite optimization problems. Because of their notoriously hard to tune regularization hyperparameters, sparsity-based estimators are currently not massively used in neuroscience. The goal of this thesis is to provide a simple, fast, and automatic way to calibrate sparse linear models.

We first study some properties of coordinate descent, which is a state-of-art algorithm to solve composite "smooth + nonsmooth separable" optimization problem. Relying on the partial smoothness framework we show that coordinate descent achieves model identification and local linear convergence. These properties are latter used in this thesis for hyperparameter optimization. We also propose an Anderson extrapolation scheme, andersoncd, to accelerate coordinate descent in practice.

We then explore a statistical approach to set the regularization parameter of Lassotype problems. A closed-form formula can be derived for the optimal regularization hyperparameter of sparse penalized linear regressions. Unfortunately, it relies on the true noise level, unknown in practice. To remove this dependency, one can resort to estimators for which the regularization hyperparameter does not depend on the noise level. However, they require to solve challenging "nonsmooth + nonsmooth" optimization problems. We show that partial smoothing preserves their statistical properties, while making the optimization problem amenable to efficient coordinate descent. Extensive experiments on real M/EEG data show the interest of these estimators on visual and auditory tasks.

Finally we investigate hyperparameter selection through the lens of bilevel optimization. It encompasses most popular hyperparameter selection techniques in machine learning and inverse problems: hold-out, cross-validation and proxies of the Stein unbiased risk estimator. This approach relies on bilevel optimization problems with nonsmooth inner problems, that are usually solved with zeros-order methods, such as grid-search or random-search. In this thesis we present an efficient algorithm to solve these bilevel optimization problems using first-order methods. This enabled to efficiently calibrate sparse models with large number of hyperparameters.

In order to promote scientific dissemination and reproducibility all the algorithms developed in this thesis are available online with examples and extensive documentation. In addition, andersoncd has been implemented in the largest python brain signal pro-10 ABSTRACT cessing package, MNE, and is now the default solver for sparse signal estimators. Algorithms for bilevel optimization with nonsmooth inner optimization problems should soon be implemented in the library jaxopt, which extends the automatic differentiation library jax, and allows to differentiate solutions of optimization problems. Tout d'abord je tiens à remercier mes deux directeurs de thèse, pour leur encadrement, leur bienveillance et la confiance qu'ils m'ont portée pendant cette thèse. Merci Alexandre de m'avoir enseigné la méthode de la trotinette, merci pour ta motivation, ta bonne humeur, ainsi que pour nos séances de debugging en visio ou en voiture, ce fut précieux. Merci Joseph pour ta bienveillance, pour ces moments à Montpellier, et pour toutes les petites attentions que tu portes au futur de tes doctorants. Comme mes prédécesseurs, j'ai pu avoir le luxe de me plaindre d'être trop encadré (coucou Eugène et Mathurin).

Thanks again to the reviewers of this thesis, Jalal Fadili and Massimiliano Pontil, I am sincerely honored, your kind words in the reports mean a lot to me. Thanks to Karim Lounici, Peter Ochs, and Carola-Bibiane Schönlieb for accepting to be part of the jury.

Un grand merci à tous mes collaborateurs, en particulier merci à Mathurin, mon partner in crime de l'évitemment du neuroimage, merci de m'avoir épaulé scientifiquement et moralement depuis le début de ma thèse. Ton soutien et nos petits moments au quotidient furent (et sont toujours) très précieux. J'espère nos collaborations encore nombreuses. Merci à Quentin pour nos moments à Dijon et à Saclay, pour son intrépidité intellectuelle. Merci pour ces échanges quotidiens qui auront rendus cette période de covid presque sympathique. Merci à Samuel, la force tranquille, pour nos moments à Dijon et au

Résumé

Comprendre les maladies neurologiques modernes telles que la maladie d'Alzheimer ou de Parkinson apparaît comme un défi majeur pour lequel les neuroscientifiques s'appuient largement sur l'enregistrement et le traitement avancés de l'activité cérébrale. En raison de leur caractère non invasif et de leur excellente résolution temporelle, la magnéto-et l'électroencéphalographie (M/EEG) sont devenues des outils incontournables pour observer l'activité cérébrale. La reconstruction des signaux cérébraux à partir des mesures M/EEG peut être considérée comme un problème inverse en grande dimension mal posé. Les estimateurs classiques des signaux cérébraux sont basés sur la résolution de problèmes d'optimisation composites. Ces estimateurs basés sur la parcimonie ne sont actuellement pas massivement utilisés en neurosciences, principalement en raison de leurs hyperparamètres de régularisation notoirement difficiles à régler. Un des objectif de cette thèse est de fournir une méthode simple, rapide et automatique pour calibrer les modèles linéaires parcimonieux.

Nous étudions d'abord certaines propriétés de la descente par coordonnée, qui est un algorithme de pointe pour résoudre les problèmes d'optimisation composites «lisse + non lisse séparable». En s'appuyant sur la théorie du lissage partiel, nous montrons que la descente par coordonnée permet l'identification du modèle et la convergence linéaire locale. Ces propriétés sont ensuite utilisées dans cette thèse pour l'optimisation des hyperparamètres. Nous proposons également un schéma d'extrapolation d'Anderson, andersoncd, pour accélérer efficacement la descente par coordonnée en pratique.

Nous explorons ensuite une approche statistique pour définir l'hyperparamètre de régularisation des problèmes de type Lasso. Dans ce cas il existe une formule exacte pour l'hyperparamètre de régularisation optimal pour la régression linéaire parcimonieuse. Malheureusement, elle dépend du niveau de bruit réel, inconnu en pratique. Pour éliminer cette dépendance, on peut recourir à des estimateurs pour lesquels le paramètre de régularisation ne dépend pas du niveau de bruit. Cependant, ces derniers nécessitent de résoudre de difficiles problèmes d'optimisation «non lisse + non lisse». Nous montrons que le lissage partiel préserve leurs propriétés statistiques, tout en faisant que le problème d'optimisation puisse être résolu avec des techniques efficaces de descente par coordonnée. Des expériences approfondies sur des données réelles de M/EEG montrent l'intérêt de ces estimateurs sur des tâches visuelles et auditives.

Enfin, nous étudions la sélection d'hyperparamètres sous l'angle de l'optimisation à deux niveaux. Elle englobe les techniques de sélection d'hyperparamètres les plus populaires dans l'apprentissage automatique et les problèmes inverses comme le critère «holdout», la validation croisée, ainsi que certains proxies de l'estimateur du risque sans biais de Stein. Cette approche repose sur des problèmes d'optimisation à deux niveaux avec des problèmes internes non lisses, qui sont généralement résolus avec des méthodes d'ordre zéro, telles que la recherche sur grille ou la recherche aléatoire. Dans cette thèse, nous présentons un algorithme efficace pour résoudre ces problèmes d'optimisation à deux niveaux en utilisant des méthodes du premier ordre. Cela permet de calibrer efficacement des modèles parcimonieux avec un grand nombre d'hyperparamètres.

Afin de promouvoir la diffusion scientifique et la reproductibilité, tous les algorithmes développés dans cette thèse sont disponibles en ligne, avec des exemples et une documentation exhaustive. De plus, andersoncd a été implémenté dans la bibliothèque python la plus populaire de traitement du signal cérébral, MNE, et est maintenant l'algorithme par défaut pour le calcul des estimateurs de signaux parcimonieux. Les algorithmes d'optimisation à deux niveaux devraient bientôt être intégrés au module d'optimisation jaxopt de la bibliothèque de différentiation automatique jax.

Notation

REMERCIEMENTS

General

Equal by definition [p]

Set of integers from 1 to p included

Id n Identity matrix in R n×n • * Nuclear norm

• 2,1 Row-wise 2,1 -mixed norm on matrices A 2,1 = p j=1 A j: • 1,2

Column-wise 1,2 -mixed norm on matrices

A 1,2 = n i=1 A :i • 2,∞
Row-wise 2,∞ -mixed norm on matrices A 2,∞ = max j∈[p] A j:

f 1 f 2 (x)
Infimal convolution of f 1 and f 2 at x inf{f 1 (x-y)+f 2 (y) :

y ∈ R d } L Lipschitz contant of ∇f L j Lipschitz contant of the function ∇ j f ∇ j f (x + he j ) -∇ j f (x) ≤ L j |h j |, ∀x ∈ R p , h ∈ R
For S 1 and S 2 ∈ S n + , S 1 S 2 if S 1 -S 2 ∈ S n + . When we write S 1 S 2 we implicitly assume that both matrices belong to S n + . A function is said to be smooth if it has Lipschitz gradients. Let f be a L-smooth function. Lipschitz constants of the functions ∇ j f are denoted by L j ; hence for all x ∈ R p , h ∈ R:

Model selection for the M/EEG inverse problem

Over the last century, neuroscience has led to significant advances in brain understanding. Breakthroughs include functional localization [START_REF] Penfield | The cerebral cortex of man; a clinical study of localization of function[END_REF], better comprehension of diseases such as epilepsy [START_REF] Penfield | Epilepsy and the functional anatomy of the human brain[END_REF], or precise description of the visual cortex [START_REF] Hubel | Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[END_REF]. Much remains to be understood and tackling modern brain diseases appears as a major challenge. To this aim, neuroscientific studies extensively rely on advanced recording and processing of brain activity.

Neuroimaging

What is brain activity? The nervous system is composed of a complex network of billions of neurons. Each neuron is composed of a cell body (the soma), dendrites, and an axon. The variation of the ionic concentration at the soma's membrane produces an electrical potential. When this potential rapidly rises and falls, an action potential can be triggered, locally propagating potential variations along the axons to ten to thousands neighboring neurons. This leads to excitatory postsynaptic potentials in the dendrites [START_REF] Baillet | Electromagnetic brain mapping[END_REF]. The goal of neuroimaging is to record brain activity, for example through the measure of these simultaneous and localized neural activations.

How to measure the brain activity? Neuroimaging, or brain imaging, is defined as the direct or indirect imaging of the nervous system structure. Main neuroimaging techniques include electrocorticography, electro-and magnetoencephalography and functional magnetic resonance imaging: [START_REF] Blausen | Medical gallery of blausen medical 2014[END_REF]), a grid of sensors is placed onto the brain surface. For electroencephalography (middle 1 ) and magnetoencephalography (right, image from the National Institute of Mental Health), sensors are placed at or close to the head surface and no surgery is needed.

• Electrocorticography (ECoG, [START_REF] Jasper | Electrocorticograms in man: effect of voluntary movement upon the electrical activity of the precentral gyrus[END_REF] measures the electrical field produced by the active neurons, with electrodes directly placed in the head, on cortical surfaces (Figure 1.1a). ECoG leads to signals with very good spatial and temporal resolutions, approximately 1 mm and 1 ms, but is extremely invasive: a part of the skull must be removed in order to place electrodes on cortical surfaces.

• Electro-and magnetoencephalography (EEG [START_REF] Berger | Über das elektroenkephalogramm des menschen[END_REF]MEG Cohen 1968) measure the electric and magnetic fields at the head surface (Figures 1.1b and 1.1c). After an excitatory postsynaptic potential is triggered, all synchronized neurons produce electric and magnetic fields, with sufficiently large amplitudes to be measured by sensors at the head surface. Measurements are very noisy (Figure 1.2a) but have an excellent time resolution, around 1 ms. Spatial resolution of M/EEG is unclear and seems to be more related to the reconstruction problems difficulty than the techniques themselves. As opposed to ECoG, EEG and MEG are non invasive techniques: no brain surgery is required.

• Functional magnetic resonance imaging (fMRI) measures changes in brain blood flow: the latter is coupled to neuronal activity [START_REF] Logothetis | Neurophysiological investigation of the basis of the fmri signal[END_REF] and increases in active brain areas. This blood flow, called haemodynamic response, leads to higher concentrations in oxygenated hemoglobin, which can be detected using blood oxygen-level-dependent (BOLD) signals [START_REF] Ogawa | Brain magnetic resonance imaging with contrast dependent on blood oxygenation[END_REF]. As opposed to EEG and MEG measurements, fMRI is an indirect technique that measures blood flow as a proxy for brain activity. fMRI BOLD signals typically have a high spatial resolution, from 1 to 3 mm, but a poor temporal resolution, from 1 to 3 s.

With their outstanding temporal resolution, MEG and EEG are tools of choice to observe and understand brain activity. In the next paragraphs we detail M/EEG data specificities and present a mathematical modelling for neuroimaging with M/EEG.

Sensors. M/EEG recordings typically involve around 300 sensors, each of which records the amplitude of the electric or magnetic field at a different location at the head surface or periphery. There are three types of sensors (see Figure 1.2a):

1.1. THESIS MOTIVATION Figure 1.2 -M/EEG data and brain signals we aim at reconstructing. Amplitude of the electric field, magnetic field, and gradient of the magnetic field (left) at the head surface or periphery. Amplitude of the current in two areas (middle), and their location (right). Data comes from the MNE software [START_REF] Gramfort | MNE software for processing MEG and EEG data[END_REF].

• Electrodes, recording the amplitude of the electric field (in µV).

• Magnetometers, recording the amplitude of the magnetic field (in fT), which is eight orders of magnitude smaller than the Earth's magnetic field.

• Gradiometers, recording the amplitude of the gradient of the magnetic field (in fT/cm).

Figure 1.2a shows M/EEG measurements after auditory stimulation at time 0 (dashed red line). Each sensor provides a time series of 400 to 500 ms with a sampling rate of 600 Hz (leading to around 50 time points after resampling): M/EEG data recordings consist in a matrix Y of size number of sensors n≈300 × number of time points

T ≈50

.

Sources. Simultaneous postsynaptic potentials can be modelled as an equivalent dipole at the macroscopic scale [START_REF] Williamson | Biomagnetism: an interdisciplinary approach[END_REF]. Source candidates are chosen on a discrete mesh of the cortical surface [START_REF] Nunez | On the relationship of synaptic activity to macroscopic measurements: does co-registration of eeg with fmri make sense?[END_REF]. The goal is then to reconstruct the electric current intensity amplitude (Figure 1.2b) of the equivalent dipole in each point on the mesh (Figure 1.2c). The equivalent dipole direction is assumed to be given, and normal to the cortical surface (Lin et al., 2006a). To summarize, our goal is to recover a matrix B * of size number of source candidates p≈10 4

× number of time points

T ≈50
, where each row is a time series with T time points, corresponding to the amplitude of the electric current on the cortical surfaces along the cognitive experiment [START_REF] Baillet | Electromagnetic brain mapping[END_REF].

Forward model. Physics of the problem leads to a direct relation between the neural activity we aim at reconstructing (Figures 1.2b and 1.2c) and the recorded M/EEG signals (Figure 1.2a). The biomagnetic forward model is well-posed [START_REF] Plonsey | Considerations of quasi-stationarity in electrophysiological systems[END_REF][START_REF] Hämäläinen | Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain[END_REF]: if one knows the number of active sources, their amplitudes and localizations in the brain, one can determine the amplitudes of the observed M/EEG signals. We use the quasi-static approximation [START_REF] Bibliography | Physical concepts and mathematical models[END_REF][START_REF] Heller | Brain stimulation using electromagnetic sources: theoretical aspects[END_REF][START_REF] Hämäläinen | Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain[END_REF]: propagation of the electric and magnetic fields is supposed to be instantaneous (no delayed potentials). This leads to a linear relation (l) . With only one trial (left), one cannot see a brain response, after the auditory stimulation at time 0 (dashed red line). When the number of trials increases (middle and right), one can observe a brain response after the stimulation. [START_REF] Hämäläinen | Interpreting magnetic fields of the brain: minimum norm estimates[END_REF] between the observed signals amplitudes Y ∈ R n×p and the electrical current amplitudes in the brain B * ∈ R p×T , through a design matrix X ∈ R n×p (also called forward operator ). In addition, signals are corrupted with additive noise E ∈ R n×T :

Y = XB * + E . (1.1)
This linear model is frequent in machine learning and inverse problems. In the next paragraph, we present some aspects of it specific to M/EEG data.

Signal characteristics. First, M/EEG recordings are very noisy: noise and signal amplitudes are of the same order of magnitude (Figure 1.3,left). In order to decrease the noise level, a common procedure is to repeat the same cognitive experiment multiple times. The brain response B * is assumed to be the same for all trials. M/EEG data is thus composed of r ∈ N measurements Y (l) ∈ R n×T :

Y (l) = XB * + E (l) , (1.2) with E (l) ∈ R n×p , l ∈ [r].
Data is then averaged across the repetitions Y (l) , called trials in M/EEG, to obtain a better signal-to-noise ratio (Figure 1.3). Another specificity of M/EEG data is that the additive noise is usually not Gaussian i.i.d., but correlated.

In Chapter 5 we will take advantage of the repetitions structure to better handle the correlated Gaussian noise.

Equipped with this mathematical modelling of M/EEG data and brain sources, we now recall some usual techniques to reconstruct neural activity from M/EEG recordings.

Linear inverse problem

With M/EEG data we have access to the amplitudes of the electric and magnetic fields at the surface of the head, yet we seek to recover the intensity of the current inside the brain. There exists three main types of approaches to identify sources responsible for observed M/EEG signals: parametric methods, scanning methods, and imaging methods (Table 1.1).

Parametric methods (dipole fitting, [START_REF] Scherg | Two bilateral sources of the late aep as identified by a spatio-temporal dipole model[END_REF][START_REF] Scherg | Fundamentals of dipole source potential analysis. Auditory evoked magnetic fields and electric potentials[END_REF] assume that observed signals are produced by a small and fixed number of active sources.

Then source locations and orientations are determined minimizing a criterion. This approach leads to challenging non-convex optimization problems, which can be solved using clustering methods, simulated annealing, or genetic algorithms [START_REF] Uutela | Global optimization in the localization of neuromagnetic sources[END_REF]. In addition to complex algorithms, no clear stopping criteria, and sensitive initialization, the number of active sources can also be hard to estimate in advance.

Scanning methods (beamforming Mosher et al. 1999 andMUSIC Mosher and[START_REF] Mosher | Recursive music: a framework for eeg and meg source localization[END_REF] evaluate the contribution of each source on a discrete grid, scanning the source space to determine optimal active source locations [START_REF] Hillebrand | A new approach to neuroimaging with magnetoencephalography[END_REF][START_REF] Sekihara | Performance of an meg adaptive-beamformer technique in the presence of correlated neural activities: effects on signal intensity and time-course estimates[END_REF]. No interference is assumed between the sources, and their contributions are computed independently using correlations between each brain location and the observed signals. More refined methods combine beamforming with signal subspace estimation, leading to MUSIC approaches (Mosher and Leahy, 1999).

Imaging methods rely on a fixed predefined grid of sources. As opposed to scanning methods, they jointly estimate amplitudes of the currents in all potential brain source locations. This leads to a severely ill-posed inverse problem: the number of sensors (the number of samples from a statistical point of view, around 10 2 ) is orders of magnitude lower than the number of potential source locations (around 10 4 ). This leads to nonunique solutions, highly sensitive to noise corruption: to circumvent these problems, one can incorporate some prior knowledge on the desired sources to recover. This prior can be enforced through constrained optimization, regularized optimization, or using Bayesian statistics [START_REF] Gelman | Bayesian data analysis[END_REF]. Most popular priors in M/EEG include 2 regularization (called MNE in the neuroimaging community [START_REF] Hämäläinen | Interpreting magnetic fields of the brain: minimum norm estimates[END_REF], weighted 2 regularization (Lin et al., 2006b), 1 regularization [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF][START_REF] Ou | A distributed spatio-temporal EEG/MEG inverse solver[END_REF][START_REF] Gramfort | Mixed-norm estimates for the m/eeg inverse problem using accelerated gradient methods[END_REF], and iterative reweighting techniques [START_REF] Candès | Enhancing sparsity by reweighted l 1 minimization[END_REF][START_REF] Strohmeier | Improved meg/eeg source localization with reweighted mixed-norms[END_REF].

In the rest of this thesis we focus on imaging methods with sparsity inducing priors, e.g., 1 or 2,1 penalties, and more generally with nonsmooth regularization. A wealth of such imaging techniques has been proposed in the literature, but unfortunately only very few are actually used in practice: they often lack of efficient and automatic ways to tune their hyperparameters. A notable example in M/EEG is the multitask Lasso [START_REF] Argyriou | Convex multi-task feature learning[END_REF][START_REF] Obozinski | Support union recovery in highdimensional multivariate regression[END_REF]:

B(λ) ∈ arg min B∈R p×T 1 2 Y -XB 2 + λ p j=1 B j: B 2,1
.

(1.3)

Modern block coordinate descent algorithms (Tseng and Yun, 2009a;[START_REF] Wright | Coordinate descent algorithms[END_REF]Massias et al., 2020b) efficiently solve Problem (1.3) and selection of the regularization hyperparameter now appears as the main obstacle for these estimators to be massively used by neuroscientists. Figure 1.4 represents estimated active brain locations found when solving Problem (1.3) for multiple values of the regularization parameter λ. Brain source locations are taken as the non zeros rows of the estimator B(λ) defined in Problem (1.3), where λ is chosen as a fraction of λ max X Y 2,∞ (the largest value of the parameter producing non-trivial solutions). Small variations in λ can lead to different numbers of active sources and brain locations. Ignoring the temporal aspect, in Chapter 6 we focus on the setting where the regularization parameter λ trades data fidelity against prior: β(λ) ∈ arg min β∈R p Φ(β, λ) f (β) + λg(β) ,

(1.4)

where f is a convex smooth function (the data fit), and g a convex, proper, lower semicontinuous and usually nonsmooth function (the regularizer).

Figure 1.4 highlights the importance of model selection for sparse linear inverse problems. Hence, in the next section we review some usual procedures to select the regularization hyperparameter λ for estimators based on Problem (1.4).

Hyperparameter selection for sparse models

The hyperparameter selection problem can be tackled in various frameworks. Most of them can be fitted in three categories: the statistical route, Bayesian statistics, and hyperparameter optimization. In this section we assume to have access to a target vector y ∈ R n , and a design matrix X ∈ R n×p .

The statistical route. The Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]:

β(λ) ∈ arg min β∈R p 1 2 y -Xβ 2 + λ β 1 , (1.5)
and its variations [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF][START_REF] Koh | An interior-point method for large-scale l1-regularized logistic regression[END_REF][START_REF] Obozinski | Support union recovery in highdimensional multivariate regression[END_REF][START_REF] Simon | A sparse-group lasso[END_REF], led to a broad literature for hyperparameter tuning [START_REF] Lounici | Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators[END_REF][START_REF] Lounici | Taking Advantage of Sparsity in Multi-Task Learning[END_REF][START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF][START_REF] Belloni | Square-root Lasso: pivotal recovery of sparse signals via conic programming[END_REF]. Under the exact sparse linear model assumption (Lounici, 2008, Ass. 1) and under the mutual incoherence hypothesis on the design matrix X (Lounici, 2008, Ass. 2)

y = Xβ * + ε , with ε ∼ N (0, σ * 2 Id n ), β * ∈ R p , β * 0 ≤ s,
X :j = 1 , and max j =j

|X :j X :j | ≤ 1 7αs , ∀j ∈ [p], for some constant α > 1 , (1.7)
one can show that if the regularization parameter λ is chosen as

λ = A log p n σ * , with A ≥ 2 √ 2 , (1.8)
then the regression coefficients are well estimated, for some C > 0, with high probability

β -β * ∞ ≤ C log p n σ * .
(1.9)

Under the additional hypothesis that the amplitude of the true regression coefficients β * is "large enough", it is possible to show that their sign is recovered with high probability (Lounici, 2008, Thm. 2). The presented properties can be generalized to other estimators, with multiple data fitting terms and penalties (van de Geer, 2016, Chap. 6 and 7). However, the statistical route has multiple weaknesses:

(i) It requires the knowledge of the true noise level σ * . This can be fixed using pivotal estimators [START_REF] Belloni | Square-root Lasso: pivotal recovery of sparse signals via conic programming[END_REF] as we will detail in Chapter 4. Note also than one can estimate the noise level σ * in M/EEG recordings [START_REF] Ledoit | A well-conditioned estimator for large-dimensional covariance matrices[END_REF][START_REF] Engemann | Automated model selection in covariance estimation and spatial whitening of MEG and EEG signals[END_REF][START_REF] Cai | Robust estimation of noise for electromagnetic brain imaging with the champagne algorithm[END_REF].

(ii) Although the mutual incoherence hypothesis in Equation (1.7) is not a necessary condition for support recovery, it is very strong and generally does not hold on real data (Figure 1.5).

CHAPTER 1. MOTIVATION AND CONTRIBUTIONS

To summarize, the statistical route relies on restrictive hypotheses and quantities which are typically unknown in practice: Lasso users still have to resort to other techniques to select the hyperparameter λ.

Other techniques involving Bayesian statistics [START_REF] Kaipio | Statistical and computational inverse problems[END_REF][START_REF] Bolstad | Introduction to Bayesian statistics[END_REF] do not rely on the mutual incoherence hypothesis and provide multiple ways to set the regularization parameter λ in the hierarchical Bayesian framework [START_REF] Molina | Bayesian and regularization methods for hyperparameter estimation in image restoration[END_REF][START_REF] Sato | Hierarchical bayesian estimation for meg inverse problem[END_REF].

Hierarchical Bayesian modeling. In this approach the regularization parameter is incorporated to the model: a distribution p(λ) has to be chosen on the hyperparameter λ. The joint probability distribution writes:

p(β, λ|y) ∼ p(y|β, λ)p(β|λ)p(λ) (1.10) ∼ exp -f (β) -λg(β) + log p(λ) /C(λ) .
(1.11)

One can resort to multiple inference techniques to estimate the regression coefficients β and the regularization parameter λ (Table 1.2):

• One can jointly compute β and λ using a full maximum-a-posteriori approach (full-MAP, [START_REF] Calvetti | Hypermodels in the bayesian imaging framework[END_REF][START_REF] Calvetti | Conditionally gaussian hypermodels for cerebral source localization[END_REF][START_REF] Lucka | Fast markov chain monte carlo sampling for sparse bayesian inference in highdimensional inverse problems using l1-type priors[END_REF]:

( β, λ) ∈ arg max β,λ
p(β, λ|y) .

(1.12)

Note that the constant C(λ) = exp(-λg(β))dβ (which depends on λ) of the probability distribution in Equation (1.11) is unknown and can be hard to estimate. For some specific models, a closed-form formula can be derived [START_REF] Pereyra | Maximum-a-posteriori estimation with unknown regularisation parameters[END_REF], otherwise it is ignored [START_REF] Bekhti | A hierarchical bayesian perspective on majorization-minimization for non-convex sparse regression: application to m/eeg source imaging[END_REF], leading to alternate sampling schemes, where one successively samples from β and λ.

• One can marginalize with respect to β [START_REF] Tipping | Sparse bayesian learning and the relevance vector machine[END_REF][START_REF] Wipf | Sparse bayesian learning for basis selection[END_REF][START_REF] Wipf | A unified Bayesian framework for MEG/EEG source imaging[END_REF][START_REF] Zhang | Sparse signal recovery with temporally correlated source vectors using sparse bayesian learning[END_REF]: (1.14) and then infer the regression coefficients β(λ) with usual optimization tools. This inference is also referred to as λ maximum-a-posteriori (λ-MAP), type-II likelihood or empirical Bayes because the prior p(β|λ) is learned from the data. Integrals in Equation (1.14) can be intractable: one can resort to expectation-minimization [START_REF] Dempster | Maximum likelihood from incomplete data via the emalgorithm[END_REF], or variational inference [START_REF] Jordan | An introduction to variational methods for graphical models[END_REF][START_REF] Mackay | Information theory, inference and learning algorithms[END_REF][START_REF] Friston | Multiple sparse priors for the m/eeg inverse problem[END_REF] to compute them.

λ ∈ arg max λ p(λ|y) (1.13) λ ∈ arg max λ p(β, λ|y)dβ ,
• One can marginalize with respect to λ [START_REF] Figueiredo | Adaptive sparseness using jeffreys prior[END_REF][START_REF] Cotter | Sparse solutions to linear inverse problems with multiple measurement vectors[END_REF][START_REF] Seeger | Variational bayesian inference techniques[END_REF][START_REF] Pereyra | Maximum-a-posteriori estimation with unknown regularisation parameters[END_REF]:

β ∈ arg max β p(β|y) (1.15) β ∈ arg max β p(β, λ|y)dλ . (1.16)
This approach is sometimes referred to as β maximum-a-posteriori (β-MAP, Wipf and Nagarajan 2009, Sec. 4). To conclude, Bayesian approaches are interpretable, flexible, and allow to incorporate prior knowledge in the regularization parameter λ with hierarchical approaches, or to directly use data to select the regularization parameter. The main disadvantages of these techniques are practical:

(i) Usual hyperpriors on λ are parametric and introduce new hyperparameters that have to be selected.

(ii) They lead to non-convex problems for which convergence toward a global minimum might depend on the initialization.

(iii) In addition to the lack of clear stopping criteria, convergence speed of samplers can be dramatically affected by other hyperparameters, such as stepsizes, or initializations (see for instance Vidal et al. 2020, Fig. 3.b).

Hyperparameter optimization. In machine learning the most popular approach for hyperparameter selection is hyperparameter optimization [START_REF] Kohavi | Automatic parameter selection by minimizing estimated error[END_REF][START_REF] Hutter | Beyond manual tuning of hyperparameters[END_REF]Feurer and Hutter, 2019): one selects the hyperparameter λ such that the regression coefficients β(λ) minimize a given criterion C : R p → R. Here C should ensure good generalization, or avoid overcomplex models. Common examples include the hold-out loss [START_REF] Devroye | Distribution-free performance bounds for potential function rules[END_REF], the cross-validation loss (CV, [START_REF] Stone | Estimating WAIS IQ from Shipley Scale scores: Another cross-validation[END_REF], the AIC [START_REF] Akaike | A new look at the statistical model identification[END_REF], BIC (Schwarz, 1978) or SURE [START_REF] Stein | Estimation of the mean of a multivariate normal distribution[END_REF] criteria. Formally, the hyperparameter optimization problem is a bilevel optimization problem [START_REF] Bracken | Mathematical programs with optimization problems in the constraints[END_REF][START_REF] Candler | Multi-level programming and development policy[END_REF][START_REF] Colson | An overview of bilevel optimization[END_REF]:

λ ∈ arg min λ∈R r L(λ) C β(λ) s.t. β(λ) ∈ arg min β∈R p Φ(β, λ) .
(1.19)

CHAPTER 1. MOTIVATION AND CONTRIBUTIONS

The most famous criterion may be the hold-out loss [START_REF] Devroye | Distribution-free performance bounds for potential function rules[END_REF]. Data (X, y) is split in two sets: the training set (X train , y train ), on which the model in trained, and the validation set (X val , y val ), on which the prediction quality of β(λ) is evaluated.

If (X i: , y i ) are i.i.d., (X val , y val ) plays the role of "unseen" new data. For the Lasso it writes

λ ∈ arg min λ∈R 1 2 y val -X val β(λ) 2 s.t. β(λ) ∈ arg min β∈R p 1 2 y train -X train β 2 + λ β 1 .
(1.20)

Hold-out relies on one split of the data, which leads to one estimate of the risk. Crossvalidation principle is to use multiple splits to create multiple estimators of the risk, and then average it. Multiple cross-validation procedures exist (Arlot and Celisse, 2010, Sec. 4.3), most popular include leave-one-out [START_REF] Stone | Cross-validatory choice and assessment of statistical predictions[END_REF][START_REF] Allen | The relationship between variable selection and data agumentation and a method for prediction[END_REF], leave-p-out [START_REF] Shao | Linear model selection by cross-validation[END_REF][START_REF] Zhang | Model selection via multifold cross validation[END_REF] or K-fold cross-validation [START_REF] Bibliography | A predictive approach to the random effect model[END_REF] which is the most common in machine learning. Data (X, y)

is partitioned into K ∈ N * hold-out datasets (X train k , y train k ), k ∈ [K].
The regularization parameter λ is then chosen to minimize the averaged squared norm of the errors

λ ∈ arg min λ∈R 1 K K k=1 y val k -X val k β(λ,k) 2 2 s.t. β(λ,k) ∈ arg min β∈R p 1 2 y train k -X train k β 2 2 + λ β 1 , ∀k ∈ [K] .
(1.21)

Cross-validation is well-suited when the "samples" (X i: , y i ) are assumed to be i.i.d., which is often not the case for inverse problems, and definitely not the case for the M/EEG inverse problem. In addition, if the goal is model selection, i.e., recovering the exact support of the true coefficients β * , then some cross-validation procedures were shown to be inconsistent: they select all active variables in β * , but also select some additional irrelevant variables [START_REF] Shao | Linear model selection by cross-validation[END_REF][START_REF] Shao | An asymptotic theory for linear model selection[END_REF].

This is why other criteria, such as the Stein unbiased risk estimator (SURE, Stein 1981), are more popular in the inverse problem literature [START_REF] Katsaggelos | Methods for choosing the regularization parameter and estimating the noise variance in image restoration and their[END_REF][START_REF] Donoho | Adapting to unknown smoothness via wavelet shrinkage[END_REF][START_REF] Zhang | Adaptive denoising based on sure risk[END_REF][START_REF] Blu | The sure-let approach to image denoising[END_REF][START_REF] Pesquet | A sure approach for digital signal/image deconvolution problems[END_REF][START_REF] Vaiter | Local behavior of sparse analysis regularization: Applications to risk estimation[END_REF]. It is an unbiased estimator of the mean squared error, which tends to penalize overcomplex models through the degree of freedom (dof) [START_REF] Efron | How biased is the apparent error rate of a prediction rule?[END_REF]Deledalle et al., 2014, Sec. 2.1). The SURE criterion writes

SURE(β) = y -Xβ 2 -σ * 2 Tr(X X) + 2dof(β) . (1.22)
Unfortunately the degree of freedom is not always available in closed-form or differentiable in β. To circumvent these problems, multiple SURE proxies have been developed: finite-difference SURE [START_REF] Ye | On measuring and correcting the effects of data mining and model selection[END_REF][START_REF] Shen | Adaptive model selection[END_REF], Monte-Carlo SURE [START_REF] Ramani | Monte-carlo sure: A black-box optimization of regularization parameters for general denoising algorithms[END_REF], iterative differentiation Monte-Carlo SURE [START_REF] Vonesch | Recursive risk estimation for non-linear image deconvolution with a wavelet-domain sparsity constraint[END_REF], or finite-differentiation Monte-Carlo SURE [START_REF] Deledalle | Stein Unbiased GrAdient estimator of the Risk (SUGAR) for multiple parameter selection[END_REF]. For the Lasso, the latter approximation yields the following problem:

λ ∈ arg min λ∈R 1 2 y -X β(λ,1) 2 + σ * 2 β(λ,2) -β(λ,1) , X δ s.t. β(λ,1) ∈ arg min β∈R p 1 2 y -Xβ 2 + λ β 1 β(λ,2) ∈ arg min β∈R p 1 2 y + δ -Xβ 2 + λ β 1 , (1.23)
with > 0 and δ ∼ N (0, Id n ). As the statistical route, the SURE requires the knowledge of the true noise level σ * . Finite-differentiation Monte-Carlo SURE also introduces a new hyperparameter , for which theoretical guidelines exist (Deledalle et al., 2014, Sec. 5.1).

Problems (1.20), (1.21) and (1.23) are bilevel optimization problems with nonsmooth inner optimization problems. These problems can be challenging to solve, especially as the dimension of the hyperparameter becomes larger. So far we have considered models with one hyperparameter λ, but more refined models rely on multiple hyperparameters, for instance the sparse-group Lasso [START_REF] Simon | A sparse-group lasso[END_REF], elastic-net [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF] or the weighted Lasso (Lasso with one regularization parameter per feature, [START_REF] Zou | The adaptive lasso and its oracle properties[END_REF]. In Chapter 6 we propose efficient first-order techniques to solve Problem (1.19) with a potential large number of hyperparameters.

One of the main goal of this thesis is to provide a fast way to select the regularization parameter λ. Most hyperparameter selection techniques rely on efficient resolutions of Problem (1.4), solved with coordinate descent: we also use it intensively (Chapters 2, 3, 5 and 6). The next section proposes an introduction to coordinate descent.

Coordinate descent

Coordinate descent is a variant of gradient descent, which updates the iterates one coordinate at a time (Tseng and Yun, 2009b). Coordinate descent is mostly used in practice in its proximal variation, on composite "smooth + separable nonsmooth" optimization problems:

min β∈R p f (β) + p j=1 g j (β j ) , (1.24)
where f is a smooth convex function, and the functions g j are convex, proper, lower semicontinuous and usually nonsmooth. Proximal coordinate descent has been applied to numerous machine learning problems (Shalev-Shwartz and Zhang, 2013a), in particular the Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF], the elastic net [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF] or the sparse logistic regression [START_REF] Ng | Feature selection, l1 vs. l2 regularization, and rotational invariance[END_REF]. It is currently the state-of-art default algorithm in preeminent machine learning packages such as scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF], glmnet [START_REF] Friedman | glmnet: Lasso and elastic-net regularized generalized linear models[END_REF], libsvm (Fan et al., 2008) or lightning [START_REF] Blondel | Lightning: large-scale linear classification, regression and ranking in python[END_REF]. Coordinate descent can also be applied on group-type penalties (Tseng and Yun, 2009a;[START_REF] Obozinski | Support union recovery in highdimensional multivariate regression[END_REF][START_REF] Simon | A sparse-group lasso[END_REF], and leads to good empirical performance with nonsmooth nonconvex penalties g j [START_REF] Breheny | Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection[END_REF][START_REF] Mazumder | Sparsenet: Coordinate descent with nonconvex penalties[END_REF][START_REF] Ge | Picasso: A sparse learning library for high dimensional data analysis in r and python[END_REF].

CHAPTER 1. MOTIVATION AND CONTRIBUTIONS

For exposition purposes we present the algorithms on an ordinary least squares problem (OLS), with a design matrix X ∈ R n×p and a target vector y ∈ R n :

min β∈R p f (β) 1 2 y -Xβ 2 . (1.25)
We suppose that X X 0, we define µ as the smallest eigenvalue of X X, and L its largest, L X 2 2 . In Section 1.2.1 we give some intuitions on why coordinate descent achieves good performance, and explain implementation details. To give insights, we compare coordinate descent against gradient descent, and instantiate all algorithms on the simpler Problem (1.25). In Section 1.2.2 we recall some theoretical results on coordinate descent on quadratics. In particular we provide a proof of the linear convergence of cyclic coordinate descent for Problem (1.25), which is a simpler case of the proof of Theorem 2.16 from Chapter 2. We also remind under which conditions coordinate descent outperforms gradient descent. Section 1.2.3 presents the generalization of coordinate descent for composite optimization Problem (1.24). Readers can refer to [START_REF] Wright | Coordinate descent algorithms[END_REF]; [START_REF] Shi | A primer on coordinate descent algorithms[END_REF] for comprehensive introductions on coordinate descent.

Introduction and intuitions

In this section we provide numerical intuitions on the success of coordinate descent. We first remind some properties of gradient descent.

Gradient descent. For Problem (1.25), gradient descent iterations with step size 1/L read, for β ∈ R p :

β ← β - 1 L X (Xβ -y) .
(1.26)

Proposition 1.1. Gradient descent with step size 1/L converges linearly with rate 1µ/L < 1.

Proof One update of gradient descent writes:

β ← (Id p - 1 L X X) T GD β - 1 L X y . (1.27)
The spectral radius of T GD has a closed-form: ρ(T GD ) = 1µ/L < 1, thus gradient descent linearly converges to the unique fixed point of β → β -1 L X (Xβy), which is also the minimizer of Problem (1.25) (Polyak, 1987, Thm. 1, Sec. 2.1.2).

In particular, the condition number L/µ of the Hessian H X X of Problem (1.25) controls the convergence of the gradient descent algorithm. We will see that coordinate descent convergence rate is governed by other quantities (Proposition 1.10), the Lipschitz constants L j of ∇ j f |β j , defined as follows:

for all β ∈ R p , h ∈ R, |∇ j f (β + he j ) -∇ j f (β)| ≤ L j |h| . -3 0 4 8 β 1 -8 -4 0 3 β 2 (a) Gradient descent. -3 0 4 8 β 1 -8 -4 0 3 β 2 (b) Coordinate descent.
f (β 1 , β 2 ) = 7β 2 1 + 6β 1 β 2 + 8β 2 2 .
Contour lines of the function f are in blue. Each algorithm starts at point (8, -6) and converges toward the minimizer (0, 0). The darker the point, the larger the iteration number. From a computational point of view, two updates of coordinate descent are equivalent to one update of gradient descent (see Algorithm 1.4).

Coordinate descent. The core idea of coordinate descent is to solve an optimization problem through the resolution of smaller dimension subproblems. These subproblems are usually easier and cheaper to solve. More formally, for a function f : R p → R, the idea is to minimize successively one dimensional functions f |β j : R → R, updating only one coordinate at a time, while the others remain unchanged. There exist many variants of coordinate descent algorithms, the main branchings being:

• The index selection. Many procedures exist to select the coordinate to update, indices can be selected:

-Cyclically in the set [p] {1, . . . , p} (Tseng and Yun, 2009a).

-At random following a given distribution [START_REF] Nesterov | Efficiency of coordinate descent methods on huge-scale optimization problems[END_REF].

-Greedily, to optimize a given criterion [START_REF] Nutini | Coordinate descent converges faster with the Gauss-Southwell rule than random selection[END_REF]: largest decrease of the objective function, or largest gradient norm (Gauss-Southwell rule, [START_REF] Southwell | Relaxation methods in engineering science-a treatise on approximate computation[END_REF]).

• The update rule. The chosen coordinate can be updated in multiple manners:

-Exact minimization of the function f |β j [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF].

-Coordinate gradient descent update (Tseng and Yun, 2009a).

Tables 1.3 and 1.4 summarize index selection and update rule variants. In this introduction we focus on cyclic coordinate descent with coordinate gradient descent update rule: for Problem (1.25), for the j th coordinate, with step size 1/L j , it reads

β j ← β j - 1 L j X :j (Xβ -y) .
(1.28)

Index selection Description

Cyclic j = (k mod p) + 1 Random j is chosen randomly (sampling with in [p] following replacement) P(j = j ) = 1/L j Gauss-Southwell j = arg max j ∇ j f (β) Table 1.3 -Index selection.

Update rule Description

Exact

β j ← arg min β j f |β j Coordinate gradient β j ← β j -1 L j ∇ j f (β) descent Table 1.4 -Update rule.
Five iterations of gradient descent and cyclic coordinate descent on a quadratic function are displayed in Figure 1.6. While gradient descent iterates move towards the minimum orthogonally to the level sets, coordinate descent iterates move successively along the axes β 1 and β 2 . On this example the values of the coordinate specific Lipschitz constants of ∇ j f |β j are L 1 = 7, L 2 = 8, and the the global Lipschitz constant is larger: L ≈ 10.5: coordinate descent uses larger coordinate specific stepsizes than gradient descent. Figure 1.8 will underline stepsizes importance. First we show that for Problem (1.25) L j has a closed-form formula.

Proposition 1.2. With f (β) = 1 2 y -Xβ 2 , the best Lipschitz constant L j of ∇f |β j is: L j = X :j 2 . (1.29)
Proof The gradient of the j th coordinate of f writes

∇ j f (β) = X :j (Xβ -y) .
Then the gradient of ∇ j f (β) restricted to β j writes

∇ 2 j,j f (β) = X :j X :j = X :j 2 .
Since ∇ j f |β j is differentiable with continuous derivatives, X :j 2 is a valid Lipschitz constant for ∇ j f |β j .

Examples, algorithmic and implementation details. Algorithms 1.1 to 1.3 are the instantiations of gradient descent, Kaczmarz algorithm [START_REF] Kaczmarz | Angenaherte auflosung von systemen linearer gleichungen: Bulletin international de l'académie polonaise des sciences et des lettres[END_REF][START_REF] Strohmer | A randomized kaczmarz algorithm with exponential convergence[END_REF], which can be seen as stochastic gradient descent SGD, Robbins and Monro 1951 with constant step size), and coordinate gradient descent on the least squares Problem (1.25).

From an algorithmic point of view, one naive update of coordinate descent (Algorithm 1.3) has a O(np) computation cost per iteration, as it requires computing Xβ. In the case of Problem (1.25) it is possible to make this update O(n) per iteration (Algorithm 1.4). This cheap update trick of coordinate descent generalizes to other sparse linear models, see [START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF] for details.

Whereas coordinate descent iterates on the columns X :j of X, stochastic gradient descent (or Kaczmarz) iterates over lines X i: of the design matrix X (Figure 1.7). From an implementation point of view, accessing efficiently the columns X :j of X requires the Figure 1.7 -Matrix storage. For coordinate descent (right), one needs to access efficiently successively each column X :j of X ∈ R n×p . On the opposite, when applying Kaczmarz or SGD (middle), one needs to access efficient each line X i: of X.

design matrix X to be stored in Fortran order3 . On the opposite accessing efficiently the lines X i: of X requires the design matrix X to be stored in C order. When dealing with sparse matrices, this means that the design matrix X should be stored is compressed sparse column (CSC)4 format for coordinate descent, and compressed sparse row (CSR) for Kaczmarz.

Algorithm 1.1 Gradient descent init : β ∈ R p for k = 0, 1, . . . , do β ← β - 1 X 2 2 X (Xβ -y) // O(np) return β Algorithm 1.2 Kaczmarz init : β ∈ R p for k = 0, 1, . . . , do Select a sample i ∈ [n] β ← β- 1 X i: 2 X i: (X i: β-y i ) // O(p) return β Algorithm 1.3 Naive CD init : β ∈ R p for k = 0, 1, . . . , do Select a coordinate j ∈ [p] β j ← β j -1 X :j 2 X :j (Xβ -y) // O(np) return β Algorithm 1.4 Efficient CD init : β ∈ R p , r = y -Xβ for k = 0, 1, . . . , do Select a coordinate j ∈ [p] β old j ← β j // O(1) β j ← β j + 1 X :j 2 X :j r // O(n) // update the residuals r r -= X :j (β j -β old j ) // O(n) return β
A quick review of coordinate descent. The index selection (Table 1.3) and the update rule (Table 1.4) branchings led to a plethora of results for coordinate descent (Fercoq and Richtárik, 2015, Tab. 1). Some landmark results for cyclic coordinate descent include [START_REF] Luo | On the convergence of the coordinate descent method for convex differentiable minimization[END_REF]; [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF]; Tseng and Yun (2009a); [START_REF] Razaviyayn | A unified convergence analysis of block successive minimization methods for nonsmooth optimization[END_REF] who have shown convergence results for cyclic coordinate descent for nonsmooth optimization problems. Then, [START_REF] Beck | On the convergence of block coordinate type methods[END_REF] showed 1/k convergence rates for convex Lipschitz smooth functions and linear convergence rates in the smooth strongly convex case. Among the results on random coordinate descent 36 CHAPTER 1. MOTIVATION AND CONTRIBUTIONS one can refer to [START_REF] Nesterov | Efficiency of coordinate descent methods on huge-scale optimization problems[END_REF] for the minimization of a smooth function f . It proved global non-asymptotic 1/k in expectation convergence rate in the case of a smooth and convex f . This work was later extended to composite optimization f + j g j for nonsmooth separable functions [START_REF] Richtárik | Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function[END_REF][START_REF] Fercoq | Accelerated, parallel and proximal coordinate descent[END_REF]. Refined convergence rates were also shown by [START_REF] Shalev-Shwartz | Stochastic methods for l1-regularized loss minimization[END_REF]Shalev-Shwartz and Zhang (2013b).

Some branchings simplify for quadratic problems: exact minimization and coordinate gradient descent actually coincide.

Proposition 1.3. On Problem (1.25) exact coordinate descent and coordinate gradient descent lead to the same update rule.

Proof

arg min

β j 1 2 y -Xβ 2 = arg min β j 1 2 y - p j =1 β j X :j 2 = arg min β j 1 2 y - j =j β j X :j -β j X :j 2 = arg min β j 1 2 β j X :j 2 -y - j =j β j X :j , β j X :j = arg min β j 1 2 β 2 j X :j 2 -X :j (y - j =j β j X :j ), β j = arg min β j 1 2 β j - 1 X :j 2 X :j (y - j =j β j X :j ) 2 = 1 X :j 2 X :j (y -Xβ + β j X :j ) = β j - 1 X :j 2 X :j (Xβ -y) .
(1.30)

As shown in Algorithms 1.1 and 1.4, one update of gradient descent costs O(np), whereas on update of coordinate descent costs O(n): in order to do fair comparisons, one must compare one update of gradient descent, and p updates of coordinate descent.

Definition 1.4 (Epoch). An epoch is p updates of coordinate descent. For the cyclic index selection it corresponds to exactly one update of each coordinate.

Intuition on the efficiency of coordinate descent (Figure 1.8). Since X :j = Xe j ≤ sup u∈R p | u ≤1 Xu = X 2 , the coordinate specific step size γ j , j ∈ [p], is larger than the step size γ of gradient descent: γ j = 1/ X :j 2 ≥ 1/ X 2 2 = γ. To confirm the influence of the step size in coordinate descent success let us do a simple experiment: Figure 1.8 compares the performance of cyclic coordinate descent with multiple values of step size. Gradient descent is compared against cyclic coordinate descent with step sizes γ j = δ/L j + (1δ)/L, for multiple values of δ. When δ = 1 one recovers the usual coordinate descent with step size γ j = 1/L j , when δ = 0, one obtains coordinate descent with step size γ j = 1/L. One can see that the performance of coordinate descent with step size 1/L is similar to the performance of gradient descent.

The larger the step size, up to 1/L j , the faster the convergence.

Figure 1.8 suggests that cyclic coordinate descent converges linearly on the least squares Problem (1. 25), and surpasses gradient descent on real-world datasets. Natural questions are: is it possible to show linear convergence for coordinate descent? If yes, is it possible to show the convergence rate of coordinate descent is better than the convergence rate of gradient descent?

Theoretical results on least squares

First, we recall cyclic coordinate descent (Algorithm 1.4) converges.

Proposition 1.5 (Convergence, Bertsekas 2015, Prop. 6.5.1). Cyclic coordinate descent converges to the minimizer of Problem (1.25).

In addition, one can show linear convergence for cyclic coordinate descent.

Proposition 1.6 (Linear convergence of cyclic CD). Cyclic coordinate descent converges linearly on Problem (1.25).

Proof The proof will be in two parts:

• First we will show that one epoch of cyclic coordinate descent leads to a linear iteration (as for one iteration of gradient descent, Proposition 1.1).

• Then we show that the iteration matrix of cyclic coordinate descent has its spectral radius strictly smaller than 1, implying that cyclic coordinate descent converges linearly.

Lemma 1.7 (Linear iteration). On Problem (1.25), one epoch of cyclic coordinate descent can be seen as a linear iteration: (1.32) with γ j = 1/ X :j 2 , and b CD ∈ R p .

β ← T CD β + b CD , (1.31) CHAPTER 1. MOTIVATION AND CONTRIBUTIONS T CD = (Id p -γ p e p e p X X) . . . (Id p -γ 1 e 1 e 1 X X) ∈ R p×p ,
Proof The update of the coordinate j writes:

β j ← β j -γ j X :j (Xβ -y) β ← β -γ j X :j (Xβ -y)e j
β ← βγ j e j X :j Xβγ j X :j ye j β ← βγ j e j e j X Xβγ j X :j ye j β ← (Id p -γ j e j e j X X)βγ j X :j ye j .

(1.33)

Updating the coordinate from j = 1 to j = p, this leads to:

β ← (Id p -γ p e p e p X X) . . . (Id p -γ 1 e 1 e 1 X X)

T CD β + b CD , (1.34)
for some b CD ∈ R p .

We have shown that one epoch of cyclic coordinate descent could be seen as a linear iteration with iteration matrix T CD . In the next lemma we that the modulus of its larger eigenvalue is strictly smaller than 1.

Lemma 1.8. The spectral radius of the iteration matrix of cyclic coordinate descent T CD is strictly smaller than 1: j) is the orthogonal projection onto Span(H 1/2 e j ) ⊥ , we thus have T (j) 2 ≤ 1, and

ρ(T CD ) < 1 . (1.35) Proof With H = X X 0, ρ(T CD ) = ρ((Id p -γ p e p e p H) . . . (Id p -γ 1 e 1 e 1 H)) = ρ(H -1/2 (Id p -γ p H 1/2 e p e p H 1/2 ) . . . (Id p -γ 1 H 1/2 e 1 e 1 H 1/2 )H 1/2 ) = ρ((Id p -γ p H 1/2 e p e p H 1/2 ) . . . (Id p -γ 1 H 1/2 e 1 e 1 H 1/2 )) . For j ∈ [p], let T (j) (Id p -γ j H 1/2 e j e j H 1/2 ), T (
ρ(T (p) × • • • × T (1) ) = T (p) × • • • × T (1) 2 ≤ T (p) 2 × • • • × T (1) 2 ≤ 1 . (1.36)
Suppose there exists a (potentially complex) eigenvalue δ ∈ C and an eigenvector

β ∈ C p = 0 of T (p) × • • • × T (1) such that |δ| = 1, we have T (p) × • • • × T (1) β 2 = β 2 .
Since T (1) ≤ 1, we have T (1) β = 1, leading to β ∈ Span(H 1/2 e 1 ) ⊥ and T (1) β = β.

Recursively we have that β ∈ Span(H 1/2 e 1 , . . . , H 1/2 e p ) ⊥ = {0}, which is a contradiction.

To conclude we have ρ( 1) has no eigenvalue of modulus 1, thus,

T (p) × • • • × T (1) ) ≤ 1 and T (p) × • • • × T (
ρ(T CD ) = ρ(T (p) × • • • × T (1) ) < 1 .
(1.37) We just showed the linear convergence of cyclic coordinate descent on the simple Problem (1.25). The reasoning above is extended in Chapter 2 to show the local linear convergence of cyclic coordinate descent on Problem (1.24).

Does coordinate descent outperforms gradient descent? In other words, for the same cost per update, how do the worst-case convergence rates of gradient and coordinate descent compare? Unfortunately, the answer will depend on the index selection procedure. While being used as a default mode for nonsmooth optimization problems in all preeminent machine learning packages, cyclic coordinate has a poor worst-case convergence rate [START_REF] Beck | On the convergence of block coordinate type methods[END_REF]. Indeed, it is possible to construct adversarial examples to deteriorate the performance of coordinate descent.

Example 1.9 [START_REF] Sun | Worst-case complexity of cyclic coordinate descent: O(n 2 ) gap with randomized version[END_REF]. For c ∈ 0, 1 , we define the quadratic problem:

min β∈R p 1 2 β        1 c . . . c c . . . . . . . . . . . . . . . . . . c c . . . c 1        β .
(1.38)

On this objective, as c approaches 1, cyclic coordinate descent is O(p 2 ) times slower than randomized coordinate descent. Figure 1.9, left, shows the suboptimality as a function of the number of epochs on Example 1.9, with c = 0.8 and p = 100. Regression coefficients β are initialized with random Gaussian i.i.d. entries β ∼ N (0, 1). On this specific example, coordinate descent with cyclic index selection performs worse than gradient descent. However, coordinate descent with random index selection performs much better than gradient descent. Theoretically, it is possible to show that coordinate descent with random index selection perfoms better than gradient descent in expectation.

CHAPTER 1. MOTIVATION AND CONTRIBUTIONS Proposition 1.10 (Convergence rates on quadratics, Sun and Ye 2019). For L-smooth µ-strongly convex quadratic functions f , gradient descent (GD), random (RCD) and cyclic (CCD) coordinate descent converge with the following per epochs complexity rates:

f β (k+1) GD -f β ≤ exp - µ L f β (k) GD -f β , (1.39) Ef β (k+1) RCD -f β ≤ exp   - pµ p j=1 L j   Ef β (k) RCD -f β , (1.40) f (β (k+1) CCD ) -f ( β) ≤ exp   - µ p p j=1 L j   f (β (k) CCD ) -f ( β) , (1.41)
and these rates are tight.

Remark 1.11 (Proposition 1.10). Since p j=1 L j /p ≤ L ≤ p j=1 L j , we have:

exp   - pµ p j=1 L j   RCD ≤ exp - µ L GD ≤ exp   - µ p p j=1 L j   CCD .
(1.42)

The worst-case convergence rate of random coordinate descent is better than the one of gradient descent, which is itself better than the one of cyclic coordinate descent. Note that scikit-learn and glmnet implement the cyclic index selection procedure: on reallife datasets cyclic coordinate descent seems to perform better (Figure 1.9, middle and right). This phenomenon is understood in some very specific cases, when the Hessian is very structured [START_REF] Gurbuzbalaban | When cyclic coordinate descent outperforms randomized coordinate descent[END_REF], but remains largely unexplained in general.

Beyond quadratics

Coordinate descent for ordinary least squares (Algorithm 1.4) can be generalized for nonquadratic smooth optimization problems [START_REF] Nesterov | Efficiency of coordinate descent methods on huge-scale optimization problems[END_REF][START_REF] Beck | On the convergence of block coordinate type methods[END_REF] and for composite Problem (1.24) using its proximal variation (PCD, Richtárik and Takáč 2014, Algorithm 1.5). Note that for sparse generalized linear models (GLMs), cheap coordinate descent updates are possible. If for all β ∈ R p , f (β) = F (Xβ), for some convex smooth function F , then an update of coordinate descent can be made

O(n) (Algorithm 1.6). Algorithm 1.5 Naive PCD init : β ∈ R p , L j > 0 for k = 0, 1, . . . , do Select a coordinate j ∈ [p] β j ←prox g j L j (β j -1 L j ∇ j f (β)) // O(np) return β Algorithm 1.6 Efficient PCD for GLMs init : β ∈ R p , r = Xβ, L j > 0 for k = 0, 1, . . . , do Select a coordinate j ∈ [p] β old j ← β j // O(1) β j ←prox g j L j (β j -1 L j X :j ∇F (Xβ))// O(n) // update efficiently r = Xβ r += X :j (β j -β old j ) // O(n) return β 1.3. CONTRIBUTIONS

Contributions

This thesis is organized as follows: in Part I we investigate some theoretical and empirical properties of coordinate descent to solve "smooth + nonsmooth separable" optimization problems. In Part II we explore the hyperparameter selection from a statistical point of view: we show that for some estimators, the regularization parameter is independent from the noise level. We also provide applications to the brain source localization problem on real M/EEG data. In Part III we present multiple ways of applying first-order methods to tackle bilevel optimization problems with nonsmooth inner optimization problems: these methods are applied to hyperparameter optimization.

Note that each chapter is self-sufficient and can be read independently. In this thesis we strongly emphasize on numerical contributions as well as reproducibility: each part comes with extensively documented and tested open source code.

More precisely in Part I:

• Chapter 2 introduces two properties of coordinate descent: under some mild assumptions on the composite nonsmooth optimization Problem (1.24), it is shown finite time support identification and local linear convergence. For example, zeros coefficients of Lasso solutions β are exactly attained after some iteration K: with S = {j ∈ [p] : βj = 0}, β (k) the iterates of coordinate descent, there exists K ∈ N, such that for all k ≥ K, for all j ∈ S c , β

= βj = 0. After support identification, there exists a regime where the iterates β (k) converge linearly toward β. This properties are extensively used for the rest of the thesis, in particular in Chapter 3 for acceleration, and in Chapter 6 for hyperparameter optimization.

• In Chapter 3, we investigate Anderson acceleration for coordinate descent. First we illustrate that à la Nesterov acceleration for coordinate descent can lead to poor convergence in practice: this contrasts with its optimal theoretically accelerated convergence rate. Then we show numerically that usual proofs of Anderson accelerated rates cannot be applied for coordinate descent, and we provide a way to theoretically accelerate coordinate descent on quadratic optimization problems. Finally, we illustrate extensively the performance of Anderson accelerated coordinate descent on a large number of optimization problems, datasets, and experimental settings. The approach proposed in this chapter can be combined with working sets techniques, and generalized to numerous data fitting terms and penalties implemented in a modular open source package andersoncd: https://github.com/mathurinm/andersoncd. The proposed algorithm is also now implemented5 as the default solver in the preeminent brain signal processing package MNE.

In Part II:

• Chapter 4: in this work we study estimators for which the regularization parameter is independent of the noise level. These estimators rely on challenging "nonsmooth + nonsmooth" optimization problems. One practical way of computing such estimators is to smooth the data fitting term: the larger the smoothing term, the easier the optimization, but the more modified the initial estimator. Provided that the smoothing parameter is not too large, we show that partial smoothing preserves the support recovery and pivotal properties. In addition we precisely quantify how large the smoothing parameter can be, leading to practical guidelines to calibrate this hyperparameter.

• In Chapter 5 we propose an application of pivotal estimators to the M/EEG source localization problem. We introduce an estimator based on an optimization problem with a smoothed trace norm as a data fitting term and an 2,1 norm as a regularizer. In addition to be independent to the noise level, the proposed estimator can take advantage of the M/EEG data multiple repetitions, and is designed to take into account Gaussian correlated noise. Last but not least, thanks to partial smoothing, one can efficiently apply block coordinate descent algorithms to solve the underlying optimization problem. Extensive experiments on real M/EEG data show the interest of the this estimator on visual and auditory tasks. Code and documentation can be found online: https://github.com/QB3/CLaR. The proposed algorithm should soon be implemented in the jaxopt library [START_REF] Blondel | Efficient and modular implicit differentiation[END_REF], which extends the automatic differentiation library jax [START_REF] Bibliography | JAX: composable transformations of Python+NumPy programs[END_REF], and allows to differentiate through solutions of optimization problems.

In Part III:

• In Chapter 6 we study the most popular hyperparameter selection approach in machine learning, encompassing hold-out and cross-validation: hyperparameter optimization. Our approach models this step as a challenging bilevel optimization with nonsmooth inner problems. Usual solvers usually rely on zeros order algorithms, whose complexity scales exponentially with the number of hyperparameters. To circumvent this problem, we propose an efficient implicit differentiation algorithm to compute the hypergradient. The bilevel optimization problem can then be solved using first-order methods. The proposed approach can be applied to set the regularization parameter for a wide range of estimators such as the Lasso, the elastic net, or the SVM: efficient implementation and extensive documentation can be found here: https://github.com/QB3/sparse-ho.

Publications

The work presented in this thesis led to the following publications: For composite nonsmooth optimization problems, which are regular enough, proximal gradient descent achieves model identification after a finite number of iterations. For instance, for the Lasso, this implies that the iterates of proximal gradient descent identify the non-zeros coefficients after a finite number of steps. Identification properties often rely on the framework of partial smoothness. In this work we show simple sufficient conditions to be a partial smooth function when the nonsmooth penalty is separable. In this simplified framework, we show cyclic coordinate descent achieves model identification in finite time, which yields explicit local linear convergence rates. Theses two properties are paramount for other works in this thesis, in particular to design accelerated algorithms for coordinate descent (Chapter 3) and to perform gradient-based hyperparameter optimization with nonsmooth inner problems (Chapter 6). This chapter is based on the following work

• Q. Bertrand, M.
• Q. Klopfenstein, Q. Bertrand, A.

Introduction

Over the last two decades, coordinate descent (CD) algorithms have become a powerful tool to solve large scale optimization problems [START_REF] Friedman | Pathwise coordinate optimization[END_REF][START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF]. Many applications coming from machine learning or compressed sensing have lead to optimization problems that can be solved efficiently via CD algorithms: the Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF][START_REF] Chen | Atomic decomposition by basis pursuit[END_REF], the elastic net [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF] or support-vector machine [START_REF] Boser | A training algorithm for optimal margin classifiers[END_REF]. All the previously cited estimators are based on an op-CHAPTER 2. MODEL IDENTIFICATION AND LOCAL LINEAR CONVERGENCE timization problem which can be written:

β ∈ arg min β∈R p {Φ(β) f (β) + p j=1 g j (β j ) g(β)
} , (2.1) with f a convex smooth (i.e., with a Lipschitz gradient) function and g j proper closed and convex functions. In the past twenty years, the popularity of CD algorithms has greatly increased due to the well suited structure of the new optimization problems mentioned above (i.e., separability of the nonsmooth term), as well as the possible parallelization of the algorithms [START_REF] Fercoq | Accelerated, parallel and proximal coordinate descent[END_REF][START_REF] Richtárik | Parallel coordinate descent methods for big data optimization[END_REF][START_REF] Salzo | Parallel random block-coordinate forward-backward algorithm: a unified convergence analysis[END_REF].

The key idea behind CD (Algorithm 2.1) is to solve small and simple subproblems iteratively until convergence. More formally, for a function Φ : R p → R, the idea is to minimize successively one dimensional functions Φ |β j : R → R, updating only one coordinate at a time, while the others remain unchanged. There exists many variants of CD algorithms, the main branching being:

• The index selection. There are different ways to choose the index of the updated coordinate at each iteration. The main variants can be divided in three categories, cyclic CD (Tseng and Yun, 2009a) when the indices are chosen in the set [p] {1, . . . , p} cyclically. Random CD [START_REF] Nesterov | Efficiency of coordinate descent methods on huge-scale optimization problems[END_REF], where the indices are chosen following a given random distribution. Finally, greedy CD [START_REF] Nutini | Coordinate descent converges faster with the Gauss-Southwell rule than random selection[END_REF] picks an index, optimizing a given criterion: largest decrease of the objective function, or largest gradient norm (Gauss-Southwell rule), for instance.

• The update rule. There also exists several possible schemes for the coordinate update: exact minimization, coordinate gradient descent or prox-linear update (see Shi et al. 2016, Sec. 2.2 for details).

In this work, we will focus on the most popular combination: cyclic CD with proxlinear update rule (Algorithm 2.1), implemented in popular packages such as glmnet [START_REF] Friedman | Pathwise coordinate optimization[END_REF] or sklearn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF].

Among the methods of coordinate selection, random CD has been extensively studied, especially by [START_REF] Nesterov | Efficiency of coordinate descent methods on huge-scale optimization problems[END_REF] for the minimization of a smooth function f . It was the first paper proving global non-asymptotic 1/k convergence rate in the case of a smooth and convex f . This work was later extended to composite optimization f + j g j for nonsmooth separable functions [START_REF] Richtárik | Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function[END_REF][START_REF] Fercoq | Accelerated, parallel and proximal coordinate descent[END_REF]. Refined convergence rates were also shown by Shalev-Shwartz and Tewari (2011); Shalev-Shwartz and Zhang (2013b). These convergence results have then been extended to coordinate descent with equality constraints [START_REF] Necoara | A random coordinate descent algorithm for optimization problems with composite objective function and linear coupled constraints[END_REF] that induce non-separability as found in the SVM dual problem in the presence of the bias term. Different distributions have been considered for the index selection such as uniform distribution [START_REF] Fercoq | Accelerated, parallel and proximal coordinate descent[END_REF][START_REF] Nesterov | Efficiency of coordinate descent methods on huge-scale optimization problems[END_REF][START_REF] Shalev-Shwartz | Stochastic methods for l1-regularized loss minimization[END_REF]Shalev-Shwartz and Zhang, 2013b), importance sampling (Leventhal and [START_REF] Lewis | Randomized methods for linear constraints: convergence rates and conditioning[END_REF][START_REF] Zhang | Solving large scale linear prediction problems using stochastic gradient descent algorithms[END_REF] and arbitrary sampling [START_REF] Necoara | A random coordinate descent algorithm for optimization problems with composite objective function and linear coupled constraints[END_REF]Qu and Richtárik, 2016a,b).

On the opposite, theory on cyclic coordinate descent is more fuzzy, the analysis in the cyclic case being more difficult. First, [START_REF] Luo | On the convergence of the coordinate descent method for convex differentiable minimization[END_REF]; [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF]; Tseng and Yun (2009a); [START_REF] Razaviyayn | A unified convergence analysis of block successive minimization methods for nonsmooth optimization[END_REF] have shown convergence results for (block) CD algorithms for nonsmooth optimization problems (without rates1 ). Then, [START_REF] Beck | On the convergence of block coordinate type methods[END_REF] showed 1/k convergence rates for Lipschitz convex functions and linear convergence rates in the strongly convex case. [START_REF] Saha | On the nonasymptotic convergence of cyclic coordinate descent methods[END_REF] proved 1/k convergence rates for composite optimization f + . 1 under "isotonicity" condition. [START_REF] Sun | Improved iteration complexity bounds of cyclic block coordinate descent for convex problems[END_REF]; [START_REF] Hong | Iteration complexity analysis of block coordinate descent methods[END_REF] have extended the latter results and showed 1/k convergence rates with improved constants for composite optimization f + j g j . [START_REF] Li | On faster convergence of cyclic block coordinate descent-type methods for strongly convex minimization[END_REF] have extended the work of [START_REF] Beck | On the convergence of block coordinate type methods[END_REF] to the nonsmooth case and refined their convergence rates in the smooth case. Finally, as far as we know, the work by [START_REF] Xu | A globally convergent algorithm for nonconvex optimization based on block coordinate update[END_REF] is the first one tackling the problem of local linear convergence. They have proved local linear convergence under the very general Kurdyka-Lojasiewicz hypothesis, relaxing convexity assumptions. Following the line of work by [START_REF] Liang | Local linear convergence of forwardbackward under partial smoothness[END_REF], we use a more restrictive framework, that allows to achieve finer results: model identification as well as improved local convergence results.

Model identification

Nonsmooth optimization problems coming from machine learning such as the Lasso or the support-vector machine (SVM) generally generate solutions lying onto a lowcomplexity model (see Definition 2.7 for details). For the Lasso, for example, a solution β has typically only a few non-zeros coefficients: it lies on the model set T β = {u ∈ R p : supp(u) ⊆ supp( β)}, where supp(β) is the support of β, i.e., the set of indices corresponding to the non-zero coefficients. A question of interest in the literature is: does the algorithm achieve model identification after a finite number of iterations? Formally, does it exist K > 0 such that for all k > K, β (k) ∈ T β ? For the Lasso the question boils down to "does it exist K > 0 such that for all k > K, supp(β (k) ) ⊆ supp( β)"? This finite time identification property is paramount for features selection [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF], but also for potential acceleration methods (Massias et al., 2018b) of the CD algorithm, as well as model calibration (see Chapter 6).

Finite model identification was first proved in [START_REF] Bertsekas | On the Goldstein-Levitin-Polyak gradient projection method[END_REF] for the projected gradient method with non-negative constraints. In this case, after a finite number of steps the sparsity pattern of the iterates is the same as the sparsity pattern of the solution. It means that for k large enough, β

= 0 for all j such that βj = 0. Then, many other results of finite model identification have been shown in different settings and for various algorithms. For the projected gradient descent algorithm, identification was proved for polyhedral constraints [START_REF] Burke | On the identification of active constraints[END_REF], for general convex constraints [START_REF] Wright | Identifiable surfaces in constrained optimization[END_REF], and even non-convex constraints [START_REF] Hare | Identifying active constraints via partial smoothness and prox-regularity[END_REF]. More recently, identification was proved for proximal gradient algorithm [START_REF] Lions | Splitting algorithms for the sum of two nonlinear operators[END_REF][START_REF] Combettes | Signal recovery by proximal forward-backward splitting[END_REF], for the 1 regularized problem [START_REF] Hare | Identifying active manifolds in regularization problems[END_REF]. [START_REF] Liang | Local linear convergence of forwardbackward under partial smoothness[END_REF][START_REF] Liang | Activity Identification and Local Linear Convergence of Forward-Backward-type Methods[END_REF]; [START_REF] Vaiter | Model consistency of partly smooth regularizers[END_REF] have shown model identification and local linear convergence for proximal gradient descent. These results have then been extended to other popular machine learning algorithms such as SAGA, SVRG [START_REF] Poon | Local convergence properties of SAGA/Prox-SVRG and acceleration[END_REF] and ADMM [START_REF] Poon | Trajectory of alternating direction method of multipliers and adaptive acceleration[END_REF], see [START_REF] Iutzeler | Nonsmoothness in machine learning: specific structure, proximal identification, and applications[END_REF] for a review. To our knowledge, apart from the line of work of [START_REF] Nutini | Let's make block coordinate descent go fast: Faster greedy rules, message-passing, active-set complexity, and superlinear convergence[END_REF]; [START_REF] Nutini | Greed is good: greedy optimization methods for large-scale structured problems[END_REF]; [START_REF] Nutini | active-set complexity" of proximal gradient: How long does it take to find the sparsity pattern?[END_REF], which we discuss thoroughly in Remark 2.14, CD has not been extensively studied with a similar generality. Some identification results have been shown for CD, but only on specific models [START_REF] She | Linear convergence and support vector identification of sequential minimal optimization[END_REF]Massias et al., 2020b) or variants of CD [START_REF] Wright | Accelerated block-coordinate relaxation for regularized optimization[END_REF], in general, under restrictive hypothesis.

Coordinate descent. We denote 0 < γ j ≤ 1/L j the local step size and γ = (γ 1 , . . . , γ p ) . To prove model identification we need to "keep track" of the iterates: following the notation from [START_REF] Beck | On the convergence of block coordinate type methods[END_REF] coordinate descent can be written:

Algorithm 2.1 Proximal coordinate descent input : γ 1 , . . . , γ p ∈ R + , n iter ∈ N, β (0) ∈ R p for k = 0, . . . , n iter do β (0,k) ← β (k) for j = 1, . . . , p do // index selection β (j,k) ← β (j-1,k) β (j,k) j ← prox γ j g j β (j-1,k) j -γ j ∇ j f (β (j-1,k) ) // update rule β (k+1) ← β (p,k) return β n iter +1
We consider the optimization problem defined in Equation (2.1) with the following assumptions:

Assumption 2.1 (Smoothness). f is a convex and differentiable function, with a Lipschitz gradient.

Assumption 2.2 (Proper, closed, convex). For any j ∈ [p], g j is proper, closed and convex.

Assumption 2.3 (Existence). The problem admits at least one solution:

arg min β∈R p Φ(β) = ∅ .
(2.2) Assumption 2.4 (Non degeneracy). The problem is non-degenerate: for any β ∈ arg min x∈R p Φ(x) -∇f ( β) ∈ ri ∂g( β) .

(2.3) Remark 2.5. Assumption 2.4 can be seen as a generalization of qualification constraints (Hare and Lewis, 2007, Sec. 1), and is usual in the machine learning literature [START_REF] Bibliography | On model selection consistency of lasso[END_REF][START_REF] Bach | Consistency of the group Lasso and multiple kernel learning[END_REF]. Note that this assumption can be considered too conservative, and one can try to relax it using the mirror-stratifiable framework [START_REF] Fadili | Sensitivity analysis for mirror-stratifiable convex functions[END_REF][START_REF] Fadili | Model consistency for learning with mirror-stratifiable regularizers[END_REF].

The contributions of this chapter are the following, first,

• When the nonsmooth penalty g is separable, we provide simple sufficient conditions for the optimized function to be partly smooth.

Then, under mild assumptions on the g j functions, for the cyclic proximal coordinate descent algorithm:

• We prove finite time model identification (Theorem 2.13).

• We provide local linear convergence rates (Theorem 2.16).

• We illustrate our results on multiple real datasets and estimators (Section 2.4) showing that our theoretical rates match the empirical ones.

Partial smoothness for separable functions

The class of partly smooth functions was first defined in [START_REF] Lewis | Active sets, nonsmoothness, and sensitivity[END_REF]. It encompasses a large number of known nonsmooth machine learning optimization penalties, such as the 1 -norm or box constraints to only name a few, see Vaiter et al. (2018, Section 2.1) for details. Interestingly, this framework enables powerful theoretical tools on model identification such as Hare and Lewis (2004, Thm. 5.3). Loosely speaking, a partly smooth function behaves smoothly as it lies on the related model and sharply if we move normal to that model. Formally, we recall the definition of partly smooth functions restricted to the case of proper, lower semicontinuous and convex functions.

Definition 2.6 (Partial smoothness). Let g : R p → R be a proper closed convex function. g is said to be partly smooth at β relative to a set M ⊆ R n if there exists a neighborhood U of β such that

• (Smoothness) M ∩ U is a C 2 -manifold and g restricted to M ∩ U is C 2 ,
• (Sharpness) The tangent space of M at β is the model tangent space T β where T β = Lin(∂g(β)) ⊥ ,

• (Continuity) The set valued mapping ∂g is continuous at β relative to M.

One of the key assumptions to solve Equation (2.1) using CD is that the nonsmooth function g is separable. In this setting, the set M appearing in Definition 2.6 is related to the notion of support that we now define.

Definition 2.7 (Generalized support, Nutini et al. 2019, Def. 1). For a vector x ∈ R p , its generalized support

S β ⊆ [p] is the set of indices j ∈ [p] such that g j is differentiable at β j : S β {j ∈ [p] : ∂g j (β j ) is a singleton} .
An iterative algorithm is said to achieve finite support identification if its iterates β (k) converge to β ∈ arg min x∈R p Φ(x), and there exists K ≥ 0 such that for all j / ∈ S β , for all k ≥ K, β

(k) j = βj .
This notion can be unified with the definition of model subspace from Vaiter et al. (2015, Sec. 3.1):

Definition 2.8 (Model subspace, [START_REF] Vaiter | Model selection with low complexity priors[END_REF]. We denote the model subspace at x:

T β = {u ∈ R p : ∀j ∈ S c β , u j = 0} , (2.4)
where S β is the generalized support of β (see Definition 2.7). CONVERGENCE

Example 2.9 (The 1 norm). The function g(β) = p j=1 |β j | is certainly the most popular nonsmooth convex regularizer promoting sparsity. Indeed, the 1 norm generates structured solution with model subspace [START_REF] Vaiter | Model consistency of partly smooth regularizers[END_REF]. We have that S β = {j ∈ [p] : β j = 0} since | • | is differentiable everywhere but not at 0, and the model subspace reads:

T β = {u ∈ R p : supp(u) ⊆ supp(β)} .
(2.5)

The 1 -norm is partly smooth at β relative to the set M β = T β .

Example 2.10 (The box constraints indicator function ι [0,C] ). This indicator function appears for instance in box constrained optimization problems such as the dual problem of the SVM. Let

I 0 β = j ∈ [p] : β j = 0 and I C β = j ∈ [p] : β j = C , then T β = {u ∈ R p : I 0 β ⊆ I 0 u and I C β ⊆ I 0 u }.
For the SVM, model identification boils down to finding the active set of the box constrained quadratic optimization problem after a finite number of iterations. The box indicator function of the interval [0, C] is partly smooth at β relative to the set

M β = β + T β .
For separable functions, the next lemma gives an explicit link between the generalized support (Definition 2.7) and the framework of partly smooth functions [START_REF] Hare | Identifying active constraints via partial smoothness and prox-regularity[END_REF].

Lemma 2.11. Let β ∈ dom(g). If for every j ∈ S β , g j is locally C 2 around βj , then g is partly smooth at β relative to β + T β .

Proof We need to prove the three properties of the partial smoothness (Definition 2.6):

• Smoothness. Let us write M β = β + T β the affine space directed by the model subspace and pointed by β. In particular, it is a C 2 -manifold.

For every j ∈ S β , g j is locally C 2 around βj , hence there exists a neighborhood U j of βj such that the restriction of g j to U is twice continuously differentiable.

For j ∈ S c β , let's write U j = R. Take U = j∈[p] U j . This a neighborhood of β (it is open, and contains β). Consider the restriction g |M β of g to M β . It is C 2 at each point of U since each coordinates (for j ∈ S β ) are C 2 around U j .
• Sharpness. Since g is completely separable, we have that

∂g( β) = ∂g 1 ( β1 ) × . . . × ∂g p ( βp ) . (2.6)
Note that ∂g j ( βj ) is a set valued mapping which is equal to the singleton {∇ j g( βj )} if g j is differentiable at βj or it is equal to an interval. The model tangent space T β of g at β is given by e , (2.8) called the model vector.

T β = span(∂g( β)) ⊥ where span(∂g( β)) = aff(∂g( β)) -e β , ( 2
In the particular case of separable functions, we have that

aff ∂g( β) = aff ∂g 1 ( β1 ) × . . . × ∂g p ( βp ) = aff ∂g 1 ( β1 ) × . . . × aff ∂g p ( βp ) .
In this case,

aff ∂g j ( βj ) =    {∇ j g( βj )} if j ∈ S β R otherwise (2.9)
and

e βj =    ∇ j g( βj ) if j ∈ S β 0 otherwise . (2.10) Thus we have that span ∂g( β) = aff ∂g( β) -e β = {x ∈ R p : ∀j ∈ S β , β j = 0} .
Then

T β = span ∂g( β) ⊥ = {x ∈ R p : ∀j ∈ S c β , β j = 0} .
(2.11)

• Continuity. We are going to prove that ∂g is inner semicontinuous at β relative to M β , i.e., that for any sequence (β (k) ) of elements of M β converging to β and any η ∈ ∂g( β), there exists a sequence of subgradients η (k) ∈ ∂g(β (k) ) converging to η.

Let β (k) be a sequence of elements of M β converging to β, or equivalently, let t (k) be a sequence of elements of T β converging to 0, and let η ∈ ∂g( β).

For j ∈ S β , we choose η

(k) j g j ( βj + t (k) j )
, using the smoothness property we have η

(k) j ηj . For all j ∈ S c β β (k) j = βj we choose η (k) j ηj , since β (k) ∈ M β , we have η (k) j ∈ ∂g(β (k) ).
We have that η (k) ∈ ∂g(β k ) and η (k) converges towards η since g j is C 1 around βj for j ∈ S β , hence, g j ( βj + t (k) j ) converges to g j ( βj ) = ηj . Thus, it proves that g is partly smooth at β relative to β + T β .

CHAPTER 2. MODEL IDENTIFICATION AND LOCAL LINEAR CONVERGENCE

Identification

In this section we show the generalized support identification result of cyclic coordinate descent. To ensure model identification, we need the following (mild) assumption:

Assumption 2.12 (Locally C 2 ). For all j ∈ S β , g j is locally C 2 around βj , and f is locally C 2 around β.

It is satisfied for the Lasso and the dual SVM problem mentioned above, but also for sparse logistic regression and elastic net. The following theorem shows that the CD (Algorithm 2.1) has the model identification property with local constant step size 0 < γ j ≤ 1/L j :

Theorem 2.13 (Model identification of CD). Consider a solution β ∈ arg min β∈R p Φ(β) and S = S β . Suppose 1. Assumptions 2.1 to 2.4 and 2.12 hold, 2. The sequence (β (k) ) k≥0 generated by Algorithm 2.1 converges to β.

Then, Algorithm 2.1 identifies the model after a finite number of iterations, which means that there exists K > 0 such that for all k ≥ K, β

S c = βS c . (k)
This result implies that for k large enough, β (k) shares the support of β (potentially smaller).

Proof Thanks to Lemma 2.11, we have that g is partly smooth [START_REF] Lewis | Active sets, nonsmoothness, and sensitivity[END_REF] at β relative to the affine space β + T β .

We now prove that for the CD Algorithm 2.1: dist ∂Φ(β (k) ), 0 → 0, when k → ∞.

As written in Algorithm 2.1, one update of coordinate descent reads:

1 γ j β (j-1,k) j -∇ j f β (j-1,k) - 1 γ j β (j,k) j ∈ ∂g j β (j,k) j 1 γ j β (k) j -∇ j f β (j-1,k) - 1 γ j β (k+1) j ∈ ∂g j β (k+1) j .
Since g is separable with non-empty subdifferential, the coordinatewise subdifferential of g is equal to the subdifferential of g, we then have

1 γ β (k) -∇ j f β (j-1,k) j∈[p] - 1 γ β (k+1) ∈ ∂g(β (k+1) ) , (2.12)
which leads to

1 γ β (k) -∇ j f β (j-1,k) j∈[p] - 1 γ β (k+1) + ∇f (β (k+1) ) ∈ ∂Φ(β (k+1) ) . (2.13) 2.3. LOCAL LINEAR CONVERGENCE 55
To prove support identification using Hare and Lewis (2004, Thm. 5.3), we need to bound the distance between ∂Φ(β (k+1) ) and 0, using Equation (2.13):

dist ∂Φ(β (k+1) ), 0

2 ≤ p j=1 β (k) j γ j -∇ j f (β (j-1,k) ) - β (k+1) j γ j + ∇ j f (β (k+1) ) 2 ≤ β (k) -β (k+1) 2 γ -1 + p j=1 ∇ j f β (j-1,k) -∇ j f β (k+1) 2 ≤ β (k) -β (k+1) 2 γ -1 + L 2 p j=1 β (j-1,k) -β (k+1) 2 ≤ β (k) -β (k+1) 2 γ -1 + L 2 p j=1 p j ≥j β (k) j -β (k+1) j 2 →0 when k→∞ .
We thus have: Poliquin and Rockafellar 1996a) and subdifferentially continuous.

• dist ∂Φ(β (k+1) ), 0 → 0 • Φ(β (k) ) → Φ( β) because Φ is prox-regular (since it is convex, see
Then the conditions to apply Hare and Lewis (2004, Th. 5.3) are met and hence we have model identification after a finite number of iterations.

Remark 2.14 (Link with existing literature). Regarding generic model identification results for CD Algorithm 2.1, Nutini et al. (2017, Lemma 3) shows model identification of coordinate descent, but relies on (Nutini et al., 2019, Lemma 1), assuming the strong convexity of the smooth function f . This is a strong hypothesis in the sparse setting, usually not realistic. Nutini (2018, Sec. 6.2.2, Thm. 9) shows model identification of coordinate descent with step size 1/L, without the strong convexity of f . We proposed a similar results for CD with step size 1/L j , relying a different proof technique: the proof of Nutini (2018, Sec. 6.2.2, Thm. 9) is based on an explicit manipulation of the expression of the minimum distance to the boundary of the subdifferential, whereas our proof is based on a geometrical statement showing that any separable function regular enough enjoys a natural control on this distance, thanks to partial smoothness theory.

Local linear convergence

In this section, we prove the local linear convergence of the CD Algorithm 2.1. After model identification, there exists a regime where the convergence towards a solution of Equation (2.1) is linear. Local linear convergence was already proved in various settings such as for ISTA and FISTA algorithms (i.e., with an 1 penalty, [START_REF] Tao | Local linear convergence of ISTA and FISTA on the LASSO problem[END_REF]) and then for the general Forward-Backward algorithm [START_REF] Liang | Local linear convergence of forwardbackward under partial smoothness[END_REF].

Local linear convergence requires an additional assumption: restricted injectivity. It is classical for this type of analysis as it can be found in [START_REF] Liang | Activity Identification and Local Linear Convergence of Forward-Backward-type Methods[END_REF] and [START_REF] Poon | Trajectory of alternating direction method of multipliers and adaptive acceleration[END_REF].

Assumption 2.15. (Restricted injectivity) For a solution β ∈ arg min β∈R p Φ(β), the restricted Hessian to its generalized support S = S β is definite positive, i.e., ∇ 2 S,S f ( β) 0 .

(2.14)

In order to study local linear convergence, we consider the fixed-point iteration of a complete epoch (an epoch is a complete pass over all the coordinates). Thanks to model identification (Theorem 2.13), we are able to prove that once the generalized support is correctly identified, there exists a regime where CD algorithm converges linearly towards the solution.

Theorem 2.16 (Local linear convergence). Consider a solution β ∈ arg min β∈R p Φ(β) and S = S β . Suppose 1. Assumptions 2.1 to 2.4, 2.12 and 2.15 hold.

2. The sequence (β (k) ) k≥0 generated by Algorithm 2.1 converges to β.

3. The model has been identified i.e., there exists

K ≥ 0 such as for all k ≥ K β (k) S c = βS c .
Then (β (k) ) k≥K converges locally linearly towards β. More precisely, once the model has been identified, there exists ψ : R |S| → R |S| (defined in Equation (2.18)) such that:

β (k+1) S = ψ(β (k) S ) ,
and for any ν ∈ [ρ(J ψ( βS )), 1[, there exists K > 0 and C > 0 such that for all k ≥ K,

β (k) S -βS ≤ Cν (k-K) β (K) S -βS .
Proof To simplify the notations we write S S β , and its elements as follows: S = {j 1 , . . . , j |S| }. We also define π : R |S| → R p for all β S ∈ R |S| and all j ∈ S by

π(β S ) j =    β j if j ∈ S βj if j ∈ S c , (2.15) and for all s ∈ [|S|], P (s) : R |S| → R |S| is defined for all u ∈ R |S| and all s ∈ [|S|] by P (s) (u) s =    u s if s = s prox γ js g js u s -γ js ∇ js f (π(u)) if s = s . (2.16)
Once the model is identified (Theorem 2.13), we have that there exists

K ≥ 0 such that for all k ≥ K β (k)
S c = βS c and (2.17)

β (k+1) S = ψ(β (k) S ) P (|S|) • . . . • P (1) (β (k) S ) . (2.18)
The proof of Theorem 2.16 follows the same steps as the ones of Proposition 1.6 on least squares in Chapter 1:

2.3. LOCAL LINEAR CONVERGENCE

57

• First we show that the fixed-point operator ψ is differentiable at βS (Lemma 2.17).

• Then we show that the Jacobian spectral radius of ψ is strictly bounded by one (Lemma 2.18 (e)). Proof of Lemma 2.18 (e) relies on Lemmas 2.18 (a) to 2.18 (d).

• Finally we conclude to local linear convergence by applying Polyak (1987, Theorem 1, Section 2.1.2).

The following lemma shows that ψ is differentiable at the optimum.

Lemma 2.17 (Differentiability fixed-point operator). ψ is differentiable at βS with Jacobian:

J ψ( βS ) = M -1/2 (Id -B (|S|) ) . . . (Id -B (1) ) A M 1/2 , (2.19) M ∇ 2 S,S f ( β) + diag (u) ∈ R |S|×|S| , (2.20) u ∈ R |S| is defined for all s ∈ [|S|] by u s =      1 γ js ∂x prox γ js g js (ẑ js ) -1 γ js if prox γ js g js (ẑ js ) = 0 0 otherwise, (2.21)
B (s) = M 1/2 :s γj s ∂ x prox γ js g js ẑjs M 1/2 :s ∈ R |S|×|S| , (2.22
)

and ẑ = β -γ ∇f ( β) ∈ R p .
Proof (Lemma 2.17). From Assumption 2.12, we know there exists a neighborhood of βj denoted U such that, for j ∈ S, the restriction of g j to U is C 2 on U. In particular, it means that βj is a differentiable point of g j and given a pair

(u, v) ∈ U × R p such that u = prox γ j g j (v) ∈ U , (2.23) we have 1 γ j (v -u) ∈ ∂g j (u) becomes 1 γ j (v -u) = g j (u) ⇔ v = u + γ j g j (u) ⇔ v = (Id + g j )(u) . (2.24) Let H(u) = (Id + g j )(u)
, since g j is twice differentiable at u, we have that

H (u) = 1 + γ j g j (u) . (2.25)
Thus, H : U → R is continuous and then H :

U → R is continuously differentiable. Hence F (v, u) v -H(u) is C 1 and F (v, u) = 0.
By convexity of g, we have g j (u) ≥ 0 and

∂F ∂u (v, u) = -H (u) = -1 -γ j g (u) = 0 . (2.26)
Using the implicit functions theorem, we have that there exists an open interval

V ⊆ R with v ∈ V and a function h : V → R which is C 1 such as u = h(v).
Using (2.23) we thus have with the choice u = βj , v = βjγ j ∇ j f ( β) that the map h coincides with prox γ j g j on V and is differentiable at s) is differentiable at βS . In addition, P (s) ( βS ) = βS , thus

v = βj -γ j ∇ j f ( β) ∈ V. It follows that P (s) is differentiable at βS . For all s ∈ [|S|], P (
ψ P (|S|) • . . . • P (1) , (2.27)
is also differentiable at βS . To compute, the Jacobian of P (s) at βS , let us first notice that

J P (s) ( βS ) = e 1 . . . e s-1 v s e s+1 . . . e |S| , (2.28)
where v s = ∂ x prox γ js g js ẑjs e jsγ js ∇ 2 js,: f ( β) and ẑj = βjγ j ∇ j f ( β). This matrix can be rewritten as

J P (s) ( βS ) = Id |S| -e s e s + ∂ x prox γ js g js ẑjs e s e s -γ js e s e s ∇ 2 f ( β) = Id |S| -e s e s γ js ∂ x prox γ js g js ẑjs diag(u) + ∇ 2 f ( β) = Id |S| -e s e s γ js ∂ x prox γ js g js ẑjs M = M -1/2 Id |S| -M 1/2 e s e s γ js ∂ x prox γ js g js ẑjs M 1/2 M 1/2 = M -1/2 Id |S| -B (s) M 1/2 , (2.29)
where

M ∇ 2 S,S f ( β) + diag (u) ∈ R |S|×|S| , (2.30) and u ∈ R |S| is defined for all s ∈ [|S|] by u s =      1 γ js ∂x prox γ js g js (ẑ js ) -1 γ js if prox γ js g js (ẑ js ) = 0 0 otherwise, (2.31) and B (s) M 1/2 :s γ js ∂ x prox γ js g js ẑjs M 1/2 :s ∈ R |S|×|S| .
(2.32)

The chain rule leads to

J ψ( βS ) = J P (|S|) ( βS )J P (|S|-1) ( βS ) . . . J P (1) ( βS ) = M -1/2 (Id -B (|S|) )(Id -B (|S|-1) ) . . . (Id -B (1) ) A M 1/2 .
The next series of lemmas (Lemmas 2.18 (a) to 2.18 (d)) will be useful to bound the spectral radius of the Jacobian of the fixed-point operator ψ (Lemma 2.18 (e)).

Lemma 2.18. a) The matrix M defined in (2.20) is symmetric definite positive.

b) For all s ∈ [|S|], the spectral radius of the matrix

B (s) defined in (2.22) is bounded by 1, i.e., B (s) 2 ≤ 1. c) For all s ∈ [|S|], B (s) / B (s) is an orthogonal projector onto Span(M 1/2 :s ). d) For all s ∈ [|S|] and for all u ∈ R S , if (Id -B (s) )u = u then u ∈ Span(M 1/2 :s ) ⊥ .
e) The spectral radius of the Jacobian J ψ( βS ) of the fixed-point operator ψ is bounded by 1

ρ(J ψ( βS )) < 1 . (2.33)
Proof (Lemma 2.18 (a)). Using the non-expansivity of the proximal operator, and the property ∂ x prox γ j g j (ẑ j ) > 0 for j ∈ S, diag(u) is a symmetric semidefinite matrix, so M is a sum of a symmetric definite positive matrix and a symmetric semidefinite matrix, hence M is symmetric definite positive.

Proof (Lemma 2.18 (b)). B (s) is a rank one matrix which is the product of

γ js ∂ x prox γ js g j (ẑ j )M 1/2 :s and M 1/2 :s
, its non-zeros eigenvalue is thus given by

B (s) 2 = M 1/2 :s γ js ∂ x prox γ js g js (ẑ j ) s M 1/2 :s = γ js ∂ x prox γ js g js (ẑ js )M s,s = γ js ∂ x prox γ js g js (ẑ js )         ∇ 2 j,j f ( β) 0≤ +   1 γ js ∂ x prox γ js g js (ẑ js ) - 1 γ js   0≤         . (2.34)
By positivity of the two terms, s) .

B (s) 2 = γ js ∂ x prox γ js g js (ẑ js ) ∇ 2 j,j f ( β) ≤L j ≤ 1 γ js + 1 -∂ x prox γ js g js (ẑ js ) ≤ ∂ x prox γ js g js (ẑ js ) + 1 -∂ x prox γ js g js (ẑ js ) ≤ 1 . (2.35) Proof (Lemma 2.18 (c)). It is clear that B (s) / B (s) is symmetric. We now prove that it is idempotent, i.e., (B (s) / B (s) ) 2 = B (s) / B (
(B (s) / B (s) ) 2 = (γ j ∂ x prox γ j g j (ẑ j )) 2 M 1/2 :s M 1/2 :s M 1/2 :s M 1/2 :s / B (s) 2 = (γ j ∂ x prox γ j g j (ẑ j )) B (s) M 1/2 :s M 1/2 :s / B (s) 2 = B (s) / B (s) .
Hence, B (s) / B (s) is an orthogonal projector.

Proof (Lemma 2.18 (d)).

Id -B (s) = Id -B (s) B (s) B (s) = (1 -B (s) ) Id + B (s) 2 Id -B (s) 2 B (s) B (s) 2 = (1 -B (s) ) Id + B (s) Id - B (s) B (s) 2 projection onto M 1/2⊥ :s . (2.36) Let u / ∈ Span(M 1/2 :s ) ⊥ , then there exists α = 0, u M 1/2⊥ :s ∈ Span(M 1/2 :s ) ⊥ such that u = αM :s + u M 1/2⊥ :s . (2.37)
Combining Equations (2.36) and (2.37) leads to:

(Id -B (s) )u = (1 -B (s) 2 )u + B (s) 2 u M 1/2⊥ :s (Id -B (s) )u ≤ |1 -B (s) 2 | =(1-B (s) 2 ) u + B (s) 2 u M 1/2⊥ :s < u < u .
Proof (Lemma 2.18 (e)). Let u ∈ R s such that

(Id |S| -B (|S|) ) . . . (Id |S| -B (1) )u = u . (2.38) Since (Id |S| -B (|S|) . . . (Id |S| -B (1) ) 2 ≤ (Id |S| -B (|S|) ) 2 ≤1 × • • • × (Id |S| -B (1) ) 2 ≤1 ,
we thus have for all j ∈ S, (Id |S| -B (s) )u = u . One can thus successively apply Lemma 2.18 (d) which leads to:

u ∈ s∈[|S|] Span M 1/2 :s ⊥ ⇔ u ∈ Span M 1/2 :1 , . . . , M 1/2 :|S| ⊥ .
Moreover M 1/2 has full rank (see Lemma 2.18 (a)), thus u = 0 and

(Id |S| -B (|S|) ) . . . (Id |S| -B (1) ) 2 < 1 .
From Lemma 2.18 (e), A 2 < 1. Moreover A and J ψ( βS ) are similar matrices (Equation (2.19)), then ρ(J ψ( βS )) = ρ(A) < 1.

Then all conditions (Lemmas 2.18 (e) and 2.17) are met to apply Polyak (1987, Theorem 1, Section 2.1.2) which proves the local linear convergence.

2.4. EXPERIMENTS 61

Differentiability outside the generalized support

For the sake of completeness, we also show that prox γ j g j is differentiable on the complement of the generalized support at βj -∇ j f ( β). This lemma will be useful in Chapter 6 to derive an implicit differentiation formula.

Lemma 2.19. Consider a solution β ∈ arg min β∈R p Φ(β) and S = S β . Suppose Assumptions 2.1 to 2.4, 2.12 and 2.15 hold.

Then, for all j ∈ S c , prox γ j g j is constant around βj -∇ j f ( β). Moreover, the map

β → prox γ j g j (β j -∇ j f (β)) is differentiable at β with gradient 0.
Proof Let ∂g j ( βj ) = [a; b] and let ẑj = βj -∇ j f ( β), then combining the fixed point equation and Assumption 2.4 leads to:

1 γ j (ẑ j -βj ) ∈ ri ∂g j ( βj ) =]a; b[ . (2.39) Thus, ẑj ∈]γ j a + βj ; γ j b + βj [ . (2.40) For all v ∈]γ j a+ βj ; γ j b+ βj [, we have 1 γ j (v-βj ) ∈]a; b[= ri ∂g j ( βj ) , i.e., prox γ j g j (v) = βj . As f is C 2 in β, we have that β → prox γ j g j (β j -∇ j f (β)
) is differentiable at β with gradient being 0.

Experiments

We now illustrate Theorems 2.13 and 2.16 on multiple datasets and estimators: the Lasso, the logistic regression and the SVM. In this section, we consider a design matrix X ∈ R n×p and a target y ∈ R n for regression (Lasso) and y ∈ {-1, 1} n for classification (logistic regression and support-vector machine). We used classical datasets from libsvm [START_REF] Chang | Libsvm: a library for support vector machines[END_REF] summarized in Table 2.1.

In Figures 2.1 to 2.3 the distance of the iterates to the optimum, β (k) -β as a function of the number of iterations k is plotted as a solid blue line. The vertical red dashed line represents the iteration k where the model has been identified by CD (Algorithm 2.1) illustrating Theorem 2.13. The yellow dashed line represents the theoretical linear rate from Theorem 2.16. Theorem 2.16 gives the slope of the dashed yellow line, the (arbitrary) origin point of the theoretical rate line is chosen such that blue and yellow lines coincide at identification time, i.e., all lines intersect at this point. More precisely, if k denotes the iteration where model identification happens, the equation of the dashed yellow line is:

h(k) = β ( k) -β × ρ(J ψ( βS )) (k-k) .
(2.41)

Once a solution β has been computed, one can calculate J ψ( βS ) and its spectral radius for each estimator.

For the experiments we used three different estimators that we detail here.

practical rate theoretical rate model identification Lasso. [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] The most famous estimator based on a nonsmooth optimization problem may be the Lasso. For a design matrix X ∈ R n×p and a target y ∈ R n it writes:

arg min β∈R p 1 2n Xβ -y 2 + λ β 1 . (2.42)
The CD update for the Lasso is given by

β j ← ST γ j λ β j -γ j X :j (y -Xβ) , (2.43)
where ST λ (β) = sign(β) • max(|β|λ, 0). The solution of Equation (2.42) is obtained using Algorithm 2.1 with constant stepsizes 1/γ j =

X :j 2 n .
practical rate theoretical rate model identification Sparse logistic regression. The sparse logistic regression is an estimator for classification tasks. It is the solution of the following optimization problem, for a design matrix X ∈ R n×p and a target variable y ∈ {-1, 1} n , with σ(z)

1 1+e -z : arg min β∈R p - 1 n n i=1 log σ(y i β X i: ) + λ β 1 .
(2.44)

The CD update for the sparse logistic regression is

β j ← ST γ j λ β j -γ j X :j (y (σ(y Xβ) -1)) .
(2.45)

The constant stepsizes for the CD algorithm to solve Equation (2.44) are given by 1/γ j = X :j 2 4n . Support-vector machine. [START_REF] Boser | A training algorithm for optimal margin classifiers[END_REF] The support-vector machine (SVM) primal optimization problem is, for a design matrix X ∈ R n×p and a target variable

y ∈ {-1, 1} n : arg min β∈R p 1 2 β 2 + C n i=1 max 1 -y i X i: β, 0 . (2.46)
The SVM can be solved using the following dual optimization problem: The CD update for the SVM reads:

arg min w∈R n 1 2 w (y X)(y X) w - n i=1 w i + n i=1 ι 0≤w i ≤C . ( 2
w i ← P [0,C] w i -γ i ((y X) i: (y Xw) -1 ) , (2.48)
where P [0,C] (x) = min(max(0, x), C). The stepsizes of the CD algorithm to solve Equation (2.47) are given by 1/γ i = (y X) i:

2 . The values of the regularization parameter C for each dataset from Figure 2.3 are given in Table 2.2.

Comments on Figures 2.1 to 2.3. Finite time model identification and local linear convergence are illustrated on the Lasso, the sparse logistic regression and the SVM in Figures 2.1 to 2.3. As predicted by Theorem 2.13, the relative model is identified after a finite number of iterations. For the Lasso (Figure 2.1) and the sparse logistic regression (Figure 2.2), we observe that as the regularization parameter gets smaller, the number of iterations needed by the CD algorithm to identify the model increases. To our knowledge, this is a classical empirical observation, that is not backed up by theoretical results. After identification, the convergence towards a solution is linear as predicted by Theorem 2.16. The theoretical local speed of convergence provided by Theorem 2.16 seems like a sharp estimation of the true speed of convergence as illustrated by the three figures.

Note that on Figures 2.1 to 2.3 high values of λ (or small values of C) were required for the restricted injectivity Assumption 2.15 to hold. Indeed, despite its lack of theoretical foundation, it is empirically observed that, in general, the larger the value of λ, the smaller the cardinal of the generalized support: |S|. It makes the restricted injectivity Assumption 2.15: ∇ 2 S,S f ( β) 0 easier to be satisfied. For instance, for λ = λ max /20, the restricted injectivity Assumption 2.15 was not verified for a lot of datasets for the Lasso and the sparse logistic regression (Figures 2.1 and 2.2). In the same vein, values of C for the SVM had to be chosen small enough, in order to make |S| not too large (Figure 2.3).

Note that finite time model identification is crucial to ensure local linear convergence, see for instance 20news dataset on Figure 2.3. However there exists very few quantitative theoretical results for the convergence speed of the model identification. [START_REF] Nutini | active-set complexity" of proximal gradient: How long does it take to find the sparsity pattern?[END_REF]; Sun et al. (2019) tried to obtain some rates on the identification, quantifying "how much the problem is qualified", i.e., how much Assumption 2.4 is satisfied. But these theoretical results do not seem to explain fully the experimental results of the CD: in particular the identification speed of the model compared to other algorithms.

Limits. We would like to point out the limit of our analysis illustrated for the case of λ = λ max /15 for the sparse logistic regression and the rcv1 dataset in Figure 2.2. In this case, the solution may no longer be unique. The support gets larger and Assumption 2.15 is no longer met. In this case, the largest eigenvalue of J ψ( βS ) is exactly one, which leads to the constant rate observed in Figure 2.2. Despite the largest eigenvalue being exactly 1, a regime of locally linear convergence toward a (potentially non unique) minimizer is still observed. Linear convergence of non-strongly convex functions starts to be more and more understood [START_REF] Necoara | Linear convergence of first order methods for non-strongly convex optimization[END_REF]. Figure 2.2 with λ = λ max /15 for rcv1 suggests extensions of Necoara et al. ( 2019) could be possible in the nonsmooth case.

Conclusion

In conclusion, we show finite time model identification for coordinate descent Algorithm 2.1 (Theorem 2.13). Thanks to this identification property we were able to show local linear rates of convergence (Theorem 2.16). These two theoretical results were illustrated on popular estimators (Lasso, sparse logistic regression and SVM dual) and popular machine learning datasets (Section 2.4).

A first natural extension of this chapter would be to investigate block coordinate minimization: Theorem 2.13 could be extended for blocks under general partial smoothness assumption [START_REF] Hare | Identifying active constraints via partial smoothness and prox-regularity[END_REF]). However, it seems that Theorem 2.16 would require a more careful analysis. A second extension could be to show linear convergence without the restricted injectivity (Assumption 2.15), paving the way for a generalization of [START_REF] Necoara | Linear convergence of first order methods for non-strongly convex optimization[END_REF] Acceleration of first order methods is mainly obtained via inertia à la Nesterov, or via nonlinear extrapolation. The latter has known a recent surge of interest, with successful applications to gradient and proximal gradient techniques. On multiple machine learning problems, coordinate descent achieves performance significantly superior to full-gradient methods. Speeding up coordinate descent in practice is notoriously hard: inertially accelerated versions of coordinate descent are theoretically accelerated, but might not always lead to practical speed-ups. Capitalizing on the identification (Theorem 2.13) and local linear convergence (Theorem 2.16) from Chapter 2, we propose an accelerated version of coordinate descent using Anderson extrapolation. This chapter is based on the following work, accepted at AISTATS 2021:

• Q. Bertrand and M. Massias. Anderson acceleration of coordinate descent. In AISTATS, 2021

Introduction

Gradient descent is the workhorse of modern convex optimization [START_REF] Nesterov | Introductory lectures on convex optimization[END_REF][START_REF] Beck | First-Order Methods in Optimization[END_REF]. For composite problems, proximal gradient descent retains the nice properties enjoyed by the latter. In both techniques, inertial acceleration achieves accelerated convergence rates [START_REF] Nesterov | A method for solving a convex programming problem with rate of convergence O(1/k 2 )[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF].

Coordinate descent is a variant of gradient descent, which updates the iterates one coordinate at a time (Tseng and Yun, 2009b;[START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF]. Proximal coordinate descent has been applied to numerous machine learning problems (Shalev-Shwartz and Zhang, 2013a;[START_REF] Wright | Coordinate descent algorithms[END_REF][START_REF] Shi | A primer on coordinate descent algorithms[END_REF], in particular the Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF], elastic net [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF] or sparse logistic regression [START_REF] Ng | Feature selection, l1 vs. l2 regularization, and rotational invariance[END_REF]. It is used in preeminent packages such as scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF], glmnet [START_REF] Friedman | glmnet: Lasso and elastic-net regularized generalized linear models[END_REF], libsvm (Fan et al., 2008) or lightning [START_REF] Blondel | Lightning: large-scale linear classification, regression and ranking in python[END_REF]. On the theoretical side, inertial accelerated versions of coordinate descent [START_REF] Nesterov | Efficiency of coordinate descent methods on huge-scale optimization problems[END_REF][START_REF] Lin | An accelerated proximal coordinate gradient method[END_REF][START_REF] Fercoq | Accelerated, parallel and proximal coordinate descent[END_REF] achieve accelerated rates. Note that usual lower bounds (Nesterov, 2004, Sec. 2.1.2) are derived for algorithms with iterates lying in the span of previous gradients, which is not the case for coordinate descent. However there also exists similar lower bounds for cyclic coordinate descent [START_REF] Sun | Worst-case complexity of cyclic coordinate descent: O(n 2 ) gap with randomized version[END_REF]. To obtain accelerated rates, Anderson extrapolation [START_REF] Anderson | Iterative procedures for nonlinear integral equations[END_REF]) is an alternative to inertia: it provides acceleration by exploiting the iterates' structure. This procedure has been known for a long time, under various names and variants [START_REF] Wynn | Acceleration techniques for iterated vector and matrix problems[END_REF][START_REF] Eddy | Extrapolating to the limit of a vector sequence[END_REF][START_REF] Smith | Extrapolation methods for vector sequences[END_REF], see [START_REF] Sidi | Vector extrapolation methods with applications[END_REF]; [START_REF] Brezinski | Shanks sequence transformations and anderson acceleration[END_REF] for reviews. Anderson acceleration enjoys accelerated rates on quadratics [START_REF] Golub | Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order richardson iterative methods[END_REF], but theoretical guarantees in the nonquadratic case are weaker [START_REF] Scieur | Regularized nonlinear acceleration[END_REF]. Interestingly, numerical performance still shows significant improvements on nonquadratic objectives.

GD GD -Anderson GD -inertial CD CD -Anderson CD -inertial RCD conjugate grad.
Anderson acceleration has been adapted to various algorithms such as Douglas-Rachford [START_REF] Fu | Anderson accelerated Douglas-Rachford splitting[END_REF], ADMM [START_REF] Poon | Trajectory of alternating direction method of multipliers and adaptive acceleration[END_REF] or proximal gradient descent [START_REF] Zhang | Globally convergent type-I Anderson acceleration for non-smooth fixed-point iterations[END_REF][START_REF] Mai | Anderson acceleration of proximal gradient methods[END_REF][START_REF] Poon | Geometry of first-order methods and adaptive acceleration[END_REF]. Among main benefits, the practical version of Anderson acceleration is memory efficient, easy to implement, line search free, has a low cost per iteration and does not require knowledge of the strong convexity constant. Finally, it introduces a single additional parameter, which often does not require tuning (see Section 3.3.1).

In this work:

• We propose an Anderson acceleration scheme for cyclic coordinate descent, which, as visible on Figure 3.1, outperforms inertial and extrapolated gradient descent, as well as inertial and randomized coordinate descent.

• The acceleration is obtained even though the iteration matrix is not symmetric, a notable problem in the analysis of Anderson extrapolation.

Algorithm 3.1 Offline Anderson extrapolation init:

β (0) ∈ R p for k = 1, . . . do // regular linear iteration β (k) = T β (k-1) + b U = [β (1) -β (0) , . . . , β (k) -β (k-1) ]
// solve a linear system of size k

c = (U U ) -1 1 k ∈ R k c /= 1 k c // does not affect β (k) β (k) e-off = k i=1 c i β (i) return β (k) e-off Algorithm 3.2 Online Anderson extrapol- ation init: β (0) ∈ R p , K ∈ N for k = 1, . . . do // regular iteration β (k) = T β (k-1) + b if k = 0 mod K then U = [β (k-K+1) -β (k-K) , . . . , β (k) -β (k-1) ] // solve a linear system of size K c = (U U ) -1 1 K ∈ R K c /= 1 K c β (k) e-on = K i=1 c i β (k-K+i)
// base sequence changes • We empirically highlight that the proposed acceleration technique can generalize in the non-quadratic case (Algorithm 3.3) and significantly improve proximal coordinate descent algorithms (Section 3.3), which are state-of-the-art first order methods on the considered problems.

β (k) = β
Notation. The vector of size K with all one entries is written 1 K .

Anderson extrapolation

Background

Anderson extrapolation is designed to accelerate the convergence of sequences based on fixed point linear iterations, that is:

β (k+1) = T β (k) + b , (3.1)
where the iteration matrix T ∈ R p×p has spectral radius ρ(T ) < 1. There exist two variants: offline and online, which we recall briefly.

CHAPTER 3. ANDERSON ACCELERATION

Offline extrapolation (Algorithm 3.1 and Figure 3.2a), at iteration k, looks for a fixed point as an affine combination of the k first iterates: 1) , and solves for the coefficients c (k) ∈ R k as follows: 1) . The motivation for introducing the coefficients c (k) is discussed in more depth after Prop. 6 in Massias et al. (2020b), and details about the closed-form solution can be found in Scieur et al. (2016, Lem. 2.4). In offline acceleration, more and more base iterates are used to produce the extrapolated point, but the extrapolation sequence does not affect the base sequence. This may not scale well since it requires solving larger and larger linear systems.

β (k) e-off = k 1 c (k) i β (i-
c (k) = arg min k 1 c i =1 k 1 c i β (i-1) -T k 1 c i β (i-1) -b 2 = arg min k 1 c i =1 k 1 c i β (i) -β (i-1) 2 = (U U ) -1 1 k /1 k (U U ) -1 1 k , (3.2) where U = [β (1) -β (0) , . . . , β (k) -β (k-1) ] ∈ R p×k (and hence the objective rewrites U c 2 ). In practice, since β (k) is available when c (k) is computed, one uses β (k) e = k 1 c (k) i β (i) instead of k 1 c (k) i β (i-
A more practical variant is the online version (Algorithm 3.2 and Figure 3.2b) considered in this chapter. The number of points to be extrapolated is fixed to K; β (1) , . . . , β (K) are computed normally with the fixed point iterations, but β (K) e is computed by extrapolating the iterates from β (1) to β (K) , and β (k) is taken equal to β (K) e . K normal iterates are then computed from β (K+1) to β (2K) then extrapolation is performed on these last K iterates, etc.

As we recall below, results on Anderson acceleration mainly concern fixed-point iterations with symmetric iteration matrices T , and results concerning non-symmetric iteration matrices are weaker [START_REF] Bollapragada | Nonlinear acceleration of momentum and primal-dual algorithms[END_REF]. Poon and Liang (2020, Thm 6.4) do not assume that T is symmetric, but only diagonalizable, which is still a strong requirement.

Proposition 3.1 (Symmetric T , Scieur 2019). Let the iteration matrix T be symmetric semi-definite positive, with spectral radius ρ = ρ(T ) < 1. Let β be the limit of the sequence

(β (k) ). Let ζ = (1- √ 1 -ρ)/(1+ √ 1 -ρ).
Then the iterates of offline Anderson acceleration satisfy, with

B = (Id -T ) 2 , β (k) e-off -β B ≤ 2ζ k-1 1+ζ 2(k-1) β (0) -β B , (3.3)
and those of online extrapolation satisfy:

β (k) e-on -β B ≤ 2ζ K-1 1+ζ 2(K-1) k/K β (0) -β B .
(3.4) [START_REF] Scieur | Regularized nonlinear acceleration[END_REF] showed that the offline version in Proposition 3.1 matches the accelerated rate of the conjugate gradient [START_REF] Hestenes | Methods of conjugate gradients for solving linear systems[END_REF]. As it states, gradient descent can be accelerated by Anderson extrapolation on quadratics. (3.5)

A typical instance is overdetermined least squares with full-column rank design matrix X ∈ R n×p , and observations y ∈ R n , such that H = X X and b = -X y. On Problem (3.5) gradient descent with step size 1/L reads:

β (k+1) = Id p -1 L H T GD ∈S p + β (k) + (-b/L b GD
) .

(3.6)

Because they have this linear structure, iterates of gradient descent can benefit from Anderson acceleration, observing that the fixed point of β → T GD β + b GD solves (3.5), with T GD ∈ S p + . Anderson acceleration of gradient descent has therefore been wellstudied beyond the scope of machine learning [START_REF] Pulay | Convergence acceleration of iterative sequences. the case of scf iteration[END_REF][START_REF] Eyert | A comparative study on methods for convergence acceleration of iterative vector sequences[END_REF]. However, on many machine learning problems, coordinate descent achieves far superior performance, and it is interesting to determine whether or not it can also benefit from Anderson extrapolation.

Linear iterations of coordinate descent

To apply Anderson acceleration to coordinate descent, we need to show that its iterates satisfy linear iterations as in (3.6). An epoch of cyclic coordinate descent for Problem (3.5) consists in updating the vector x one coordinate at a time, sequentially, i.e. for j = 1, . . . , p:

β j ← β j - 1 H jj (H j: β + b j ) , (3.7)
which can be rewritten, for j = 1, . . . , p:

β ←   Id p - e j e j H jj H   β - b j H jj e j .
(3.8)

Thus, for primal iterates, one full pass (updating coordinates from 1 to p) leads to a linear iteration (as in Lemma 1.7 from Chapter 1), for some b CD ∈ R p : Note that in the case of coordinate descent we write β (k) for the iterates after one pass of coordinate descent on all features, and not after each update (3.7). The iterates of cyclic coordinate descent therefore also have a fixed-point structure, but contrary to gradient descent, their iteration matrix T CD is not symmetric, which we address in Section 3.2.3. Note that random coordinate descent does not exhibit such a fixed-point structure.

β (k+1) = T CD β (k) + b CD , ( 3

Anderson extrapolation for nonsymmetric iteration matrices

Even on quadratics, Anderson acceleration with non-symmetric iteration matrices is less developed, and the only results concerning its theoretical acceleration are recent and weaker than in the symmetric case.

Proposition 3.2 (Bollapragada et al. 2018, Thm 2.2). When T is not symmetric, and ρ(T ) < 1,

β (k) e-off -T β (k) e-off -b ≤ Id -ρ(T -Id) 2 P * (T )(β (1) -β (0) ) ,
where the unavailable polynomial P * minimizes P (T )(β (1)β (0) ) amongst all polynomials P of degree exactly k -1 whose coefficients sum to 1.

The quality of the bound (in particular, its eventual convergence to 0) crucially depends on P (T ) . Using the Crouzeix conjecture [START_REF] Crouzeix | Bounds for analytical functions of matrices[END_REF], [START_REF] Bollapragada | Nonlinear acceleration of momentum and primal-dual algorithms[END_REF] managed to bound P (T ) , with P a polynomial: [START_REF] Crouzeix | Numerical range and functional calculus in hilbert space[END_REF][START_REF] Crouzeix | The numerical range is a (1+2)-spectral set[END_REF], and W (T ) the numerical range:

P (T ) ≤ c max z∈W (T ) |P (z)| , (3.11) with c ≥ 2 (
W (T ) {β * T β : β 2 = 1, β ∈ C p } . (3.12)
Since there is no general formula for this bound, [START_REF] Bollapragada | Nonlinear acceleration of momentum and primal-dual algorithms[END_REF] used numerical bounds on W (T q ) to ensure convergence. Figure 3.3 displays the numerical range W (T q ) in the complex plane for q ∈ {1, 128, 256, 512}. In order to be able to apply the theoretical result from [START_REF] Bollapragada | Nonlinear acceleration of momentum and primal-dual algorithms[END_REF], one must chose q such that the point (1, 0) is not contained in W (T q ), and extrapolate β (0) , β (q) , β (2q) , . . . One can see on Figure 3.3 that large values of q are needed, unusable in practice: q = 512 is greater than the number of iterations needed to converge on some problems. Moreover, Anderson acceleration seems to provide speed up on coordinate descent even with q = 1 as we perform, which highlights the need for refined bounds for Anderson acceleration on nonsymmetric matrices.

We propose two means to fix this lack of theoretical results: to modify the algorithm in order to have a more amenable iteration matrix (Section 3.2.4), or to perform a simple cost function decrease check (Section 3.2.5).

Pseudo-symmetrization of T

A first idea to make coordinate descent theoretically amenable to extrapolation is to perform updates of coefficients from indices 1 to p, followed by a reversed pass from p to 1. This leads to an iteration matrix which is not symmetric either but friendlier: it writes

T CD-sym Id p - e 1 e 1 H 11 H × • • • × Id p - epe p Hpp H Id p - e p e p H pp H × • • • × Id p - e 1 e 1 H 11 H = H -1/2 SH 1/2 , (3.13) 3.2. ANDERSON EXTRAPOLATION 73 -2 -1 0 1 2 -2 -1 0 1 2 gina agnostic -2 -1 0 1 2 -1 0 1 rcv1 -2 -1 0 1 2 -1 0 1 real-sim -2-1 0 1 2 -2 -1 0 1 2 news20 W (T 1 ) W (T 128 ) W (T 256 ) W (T 512 )
Figure 3.3 -Numerical range of T q as q varies; T is the iteration matrix of Ridge regression problems with conditioning κ = 10 3 , on 4 datasets. The black cross marks the (1, 0) point, which should lie outside the range for the theoretical bound to be useful.

with

S = Id p -H 1/2 e 1 e 1 H 11 H 1 2 × • • • × Id p -H 1 2 epe p Hpp H 1 2 × Id p -H 1 2 epe p Hpp H 1 2 × • • • × Id p -H 1 2 e 1 e 1 H 11 H 1 2 . (3.14)
S is symmetric, thus, S and T (which has the same eigenvalues as S), are diagonalisable with real eigenvalues. We call these iterations pseudo-symmetric, and show that this structure allows to preserve the guarantees of Anderson extrapolation.

Proposition 3.3 (Pseudosym. T = H -1/2 SH 1/2
). Let T be the iteration matrix of pseudo-symmetric coordinate descent: T = H -1/2 SH 1/2 , with S the symmetric positive semidefinite matrix of (3.13). Let β be the limit of the sequence

(β (k) ). Let ζ = (1 - √ 1 -ρ)/(1 + √ 1 -ρ). Then ρ = ρ(T ) = ρ(S)
β (k) e-off -β B ≤ κ(H) 2ζ k-1 1+ζ 2(k-1) β (0) -β B , (3.15)
and thus those of online extrapolation satisfy:

β (k) e-on -β B ≤ κ(H) 2ζ K-1 1+ζ 2(K-1) k/K β (0) -β B . (3.16)
Proof (Proposition 3.3). First we link the quantity computed in Equation (3.2) to the extrapolated quantity k i=1 c i β (i-1) (Lemma 3.4 (a)), then we bound β

(k) e-on -β B (Lemma 3.4 (b)). Lemma 3.4. a) For all c ∈ R k such that k i=1 c i = 1: k i=1 c i (β (i) -β (i-1) ) = (T -Id)   k i=1 c i β (i-1) - β  . (3.17) b) For all c ∈ R k such that k i=1 c i = 1, (T -Id)(β (k) e-off -β) ≤ κ(H) k-1 i=0 c i S i (T -Id)(β (0) -β) . (3.18) CHAPTER 3. ANDERSON ACCELERATION
Here we link the quantity computed in Equation (3.2) to the extrapolated quantity

k i=1 c i β (i-1) . Proof (Lemma 3.4 (a)). Since β (i) = T β (i-1) + ( β -T β), c i (β (i) -β (i-1) ) = c i (T β (i-1) + β -T β -β (i-1) ) = (T -Id)c i (β (i-1) -β) . (3.19) Hence, since k 1 c i = 1, k i=1 c i (β (i) -β (i-1) ) = (T -Id)   k i=1 c i β (i-1) - β  . (3.20)
Here we bound β

(k)
e-on -β B . Proof (Lemma 3.4 (b)). In this proof, we denote by c * the solution of (3.2). We use the fact that for all

c ∈ R k such that k i=1 c i = 1, k i=1 c * i (β (i) -β (i-1) ) = min c∈R k i c i =1 k i=1 c i (β (i) -β (i-1) ) ≤ k i=1 c i (β (i) -β (i-1) ) .
c i ∈ R such that k i=1 c i = 1 : (T -Id)(β e -β) = k i=1 c * i (β (i) -β (i-1) ) ≤ k i=1 c i (β (i) -β (i-1) ) = (T -Id) k i=1 c i (β (i-1) -β) = (T -Id) k i=1 c i T i-1 (β (0) -β) = k i=1 c i T i-1 (T -Id)(β (0) -β) = k i=1 c i T i-1 × (T -Id)(β (0) -β) ≤ H -1/2 k i=1 c i S i-1 H 1/2 × (T -Id)(β (0) -β) ≤ κ(H) k i=1 c i S i-1 × (T -Id)(β (0) -β) . ( 3
= 1- √ 1-ρ(T ) 1+ √ 1-ρ(T ) : k i=1 c Cb i S i-1 ≤ 2ζ k-1 1+ζ 2(k-1) . (3.23)
Combined with Lemma 3.4 (b) this concludes the proof:

(T -Id)(β e -β) ≤ κ(H) k i=1 c i S i-1 (T -Id)(β (0) -β) (3.24) ≤ κ(H) 2ζ k-1 1+ζ 2(k-1) (T -Id)(β (0) -β) . (3.25)
Proposition 3.3 shows accelerated convergence rates for the offline Anderson acceleration, but a κ(H) appears in the rate of the online Anderson acceleration, meaning that K must be large enough that ζ K mitigates this effect. This factor however seems like a theoretical artefact of the proof, since we observed significant speed up of the online Anderson acceleration, even with bad conditioning of H (see Figure 3.4).

Figure 3.4 illustrates the convergence speed of cyclic and pseudo-symmetric coordinate descent on the rcv1 dataset. Anderson acceleration provides speed up for both versions. Interestingly, on this quadratic problem, the non extrapolated pseudo-symmetric iterations perform poorly, worse than cyclic coordinate descent. However, the performance is reversed for their extrapolated counterparts: the pseudo-symmetrized version is better than the cyclic one (which has a nonsymmetric iteration matrix). Finally, Anderson extrapolation on the pseudo-symmetrized version even reaches the conjugate gradient performance.

CHAPTER 3. ANDERSON ACCELERATION Algorithm 3.3 Online Anderson proximal gradient descent (proposed)

init: β (0) ∈ R p for k = 1, . . . do β = β (k-1) for j = 1, . . . p do β old j = β j β j = prox λ L j g j (β j -X :j ∇F (Xβ)/L j ) Xβ += (β j -β old j )X :j β (k) = β // regular iter. O(np) if k = 0 mod K then // extrapol., O(K 3 + pK 2 ) U = [β (k-K+1) -β (k-K) , . . . , β (k) -β (k-1) ] c = (U U ) -1 1 K /1 K (U U ) -1 1 K ∈ R K β e = K i=1 c i β (k-K+i) if F (Xβ e ) + λg(β e ) ≤ F (β (k) ) + λg(β (k) ) then β (k) = β e return β (k)

Generalization to nonquadratic and proposed algorithm

After devising and illustrating an Anderson extrapolated coordinate descent procedure for a simple quadratic objective, our goal is to apply Anderson acceleration on problems where coordinate descent achieve state-of-the-art results, i.e., of the form:

min β∈R p Φ(β) = f (β) + λg(β) F (Xβ) + λ p j=1 g j (β j ) ,
where F : R n → R is convex, smooth and g j 's are proper, closed and convex functions As examples, we allow g = 0, g = β 1 , g = 1 2 β 2 2 , g = β 1 + ρ 2λ x 2 . In the nonquadratic case, updates of proximal coordinate descent do not lead to a linear iteration. In this case, T is not a matrix, but a nonlinear operator. However, as stated in Proposition 3.5, asymptotically, the fixed-point operator T is linear.

Proposition 3.5. Consider a solution β ∈ arg min β∈R p Φ(β). Suppose 1. Assumptions 2.1 to 2.4, 2.12 and 2.15 hold.

2. The sequence (β (k) ) k≥0 generated by Algorithm 2.1 converges to β.

Then there exists K ∈ N, ψ : R |S| → R |S| (defined in Equation (2.18)) such that, for all k ∈ N, k ≥ K:

β (k) j = βj , for all j ∈ S c , (3.26) β (k+1) S -βS = J ψ( βS )(β (k) S -βS ) + O( β (k) S -βS 2 ) , (3.27)
and ρ J ψ( βS ) < 1 .

(3.28)

Proof Proposition 3.5 is a direct consequence of Theorems 2.13 and 2.16.

Figure 3.5 shows the performance of Anderson extrapolation on a 2 -regularized logistic regression problem:

arg min β∈R p n i=1 log(1 + e -y i X i: β ) + λ 2 β 2 2 . (3.29)
One can see that despite the better theoretical properties of the pseudo-symmetrized coordinate descent, Anderson acceleration on coordinate descent seems to work better on the cyclic coordinate descent. We thus choose to apply Anderson extrapolation on the cyclic coordinate descent (Algorithm 3.3), while adding a step checking the decrease of the objective function in order to ensure convergence.

Finally, we can also use Algorithm 3.3 in the non smooth case where g = • 1 , since coordinate descent achieves support identification when the solution is unique, after which the objective becomes differentiable. There is therefore a linear structure after a sufficient number of iterations.

Experiments

An implementation relying on numpy, numba and cython [START_REF] Mckinney | Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython[END_REF][START_REF] Lam | Numba: A LLVM-based Python JIT Compiler[END_REF][START_REF] Behnel | Cython: The best of both worlds[END_REF], with scripts to reproduce the figures, is available at: https: //mathurinm.github.io/andersoncd.

We first show how we set the hyperparameters of Anderson extrapolation (Section 3.3.1).

Then we show that Anderson extrapolation applied to proximal coordinate descent outperforms other first order algorithms on standard machine learning problems (Section 3.3.2).

Parameter setting

Anderson extrapolation relies on 2 hyperparameters: the number of extrapolated points K, and the amount of regularization eventually used when solving the linear system to obtain the coefficients c ∈ R K . Based on the conclusions of this section, we fix these parameters for all the subsequent experiments in Section 3.3.2: no regularization and K = 5.

Influence of the regularization. [START_REF] Scieur | Regularized nonlinear acceleration[END_REF] provided accelerated complexity rates for regularized Anderson extrapolation: a term λ reg c 2 is added to the objective of Equation (3.2). The closed-form formula for the coefficients is then

(U U + λ reg Id K ) -1 1 K /1 K (U U + λ reg Id K ) -1 1 K .
However, similarly to [START_REF] Mai | Anderson acceleration of proximal gradient methods[END_REF] and [START_REF] Poon | Geometry of first-order methods and adaptive acceleration[END_REF] we observed that regularizing the linear system does not seem necessary, and can even hurt the convergence speed. Figure 3.7 shows the influence of the regularization parameter on the convergence on the rcv1 dataset for a sparse logistic regression problem, with K = 5 and λ = λ max /30. The more the optimization problem is regularized, the more the convergence speed is deteriorated. Thus we choose not to regularize when solving the linear system for the extrapolation coefficients. We simply check if the ex- Influence of K. Figure 3.6 shows the impact of K on the convergence speed. Although the performance depends on K, it seems that the dependency is loose, as for K ∈ {10, 20} the acceleration is roughly the same. Therefore, we do not treat K as a parameter and fix it to K = 5.

Computational overhead of Anderson extrapolation. With nnz the number of nonzero coefficients, K epochs (i.e., K updates of all coordinates) of CD without Anderson acceleration cost:

Knnz(X) .

Every K epochs, Algorithm 3.3 requires to solve a K × K linear system. Thus, K epochs of CD with Anderson acceleration cost:

Knnz(X) K passes of CD + K 2 nnz(w) form U U + K 3 solve system
.

With our choice, K = 5, the overhead of Anderson acceleration is marginal compared to a gradient call: K 2 + Knnz(w) nnz(X). This can be observed in Figures 3.8 to 3.10: even before acceleration actually occurs, Anderson PCD is not slower than regular PCD.

Numerical comparison on machine learning problems

We compare multiple algorithms to solve popular machine learning problems: the Lasso, the elastic net, the sparse logistic regression and the group Lasso. The compared algorithms are the following:

• Proximal gradient descent (PGD, [START_REF] Lions | Splitting algorithms for the sum of two nonlinear operators[END_REF][START_REF] Combettes | Signal recovery by proximal forward-backward splitting[END_REF].

• Nesterov-like inertial PGD (FISTA, Beck and Teboulle 2009).

• Anderson accelerated PGD [START_REF] Mai | Anderson acceleration of proximal gradient methods[END_REF][START_REF] Poon | Geometry of first-order methods and adaptive acceleration[END_REF].

• Proximal coordinate descent (PCD, Tseng and Yun 2009b).

• Proximal coordinate descent with random index selection (PRCD, Richtárik and Takáč 2014).

• Inertial PCD [START_REF] Lin | An accelerated proximal coordinate gradient method[END_REF][START_REF] Fercoq | Accelerated, parallel and proximal coordinate descent[END_REF].

• Anderson accelerated PCD (ours, Algorithm 3.3).

We use datasets from libsvm (Fan et al., 2008) and openml (Feurer et al., 2019) (Table 3.1), varying as much as possible to demonstrate the versatility of our approach. Lasso. Figure 3.8 shows the suboptimality Φ(β (k) ) -Φ( β) of the algorithms on the Lasso problem:

arg min β∈R p 1 2 y -Xβ 2 + λ β 1 , (3.30)
as a function of time for multiple datasets and values of λ. We parametrize λ as a fraction of λ max = X y ∞ , smallest regularization strength for which β = 0. Figure 3.8 highlights the superiority of proximal coordinate descent over proximal gradient descent for Lasso problems on real-world datasets, and the benefits of extrapolation for coordinate descent. It shows that Anderson extrapolation can lead to a significant gain of performance. In particular Figure 3.8 shows that without restart, inertial coordinate descent [START_REF] Lin | An accelerated proximal coordinate gradient method[END_REF][START_REF] Fercoq | Accelerated, parallel and proximal coordinate descent[END_REF] can slow down the convergence, despite its accelerated rate. Note that the smaller the value of λ, the harder the optimization: when λ decreases, more time is needed to reach a fixed suboptimality. The smaller λ is (i.e., the harder the problem), the more efficient Anderson extrapolation is.

Elastic net. Anderson extrapolation is easy to extend to other estimators than the Lasso. Figure 3.9 show the superiority of the Anderson extrapolation approach over proximal gradient descent and its accelerated version for the elastic net problem [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF]: In particular, we observe that the more difficult the problem, the more useful the Anderson extrapolation: it is visible on Figure 3.9 that going from ρ = λ/10 to ρ = λ/100 lead to an increase in time to achieve similar suboptimality for the classical proximal coordinate descent, whereas the impact is more limited on the coordinate descent with Anderson extrapolation.

arg min β∈R p 1 2 y -Xβ 2 + λ β 1 + ρ 2 β 2 2 . ( 3
Finally, for a nonquadratic data-fit, here sparse logistic regression, we still demonstrate the applicability of extrapolated coordinate descent. Sparse logistic regression. Figure 3.10 represents the suboptimality as a function of time on a sparse logistic regression problem:

arg min β∈R p n i=1 log(1 + e -y i X i: β ) + λ β 1 , (3.32)
for multiple datasets and values of λ. We parametrize λ as a fraction of λ max = X y ∞ /2. As for the Lasso and the elastic net, the smaller the value of λ, the harder the problem and Anderson CD outperforms its competitors.

Group Lasso. In this section we consider the group Lasso, with a design matrix X ∈ R n×p , a target y ∈ R n , and a partition G of [p] (elements of the partition being the disjoints groups):

arg min β∈R p 1 2 y -Xβ 2 + λ g∈G β g , (3.33)
where for g ∈ G, β g ∈ R |g| is the subvector of β composed of coordinates in g. the group Lasso can be solved via proximal gradient descent and by block coordinate descent (BCD), the latter being amenable to Anderson acceleration. As Figure 3.11 shows, the superiority of Anderson accelerated block coordinate descent is on par with the one observed on the problems studied above.

Conclusion

In this work, we have proposed to accelerate coordinate descent using Anderson extrapolation. We have exploited the fixed point iterations followed by coordinate descent iterates on multiple machine learning problems to improve their convergence speed.

We have circumvented the non-symmetricity of the iteration matrices by proposing a pseudo-symmetric version for which accelerated convergence rates have been derived.

In practice, we have performed an extensive validation to demonstrate large benefits on multiple datasets and problems of interests. For future works, the excellent performance of Anderson extrapolation for cyclic coordinate descent calls for a more refined analysis of the known bounds, through a better analysis of the spectrum and numerical range of the iteration matrices.

On the statistical aspects of partial smoothing In Chapter 3 we saw how to efficiently solve "smooth + nonsmooth separable" optimization problems. In this chapter we are interested in selecting the regularization parameter of Lasso-type problems, trading the data-fidelity term against the sparsity enforcing term. In high dimensional sparse regression, there exist estimators for which the optimal regularization parameter has a closed-form formula and is independent from the true noise level: pivotal estimators. The canonical pivotal estimator is the square-root Lasso, formulated along with its derivatives as a "nonsmooth + nonsmooth" optimization problem. Modern techniques to solve these include smoothing the data fitting term, to benefit from fast efficient proximal algorithms. In this chapter we show minimax sup-norm convergence rates for non smoothed and smoothed, single task and multitask square-root Lasso-type estimators. Thanks to our theoretical analysis, we provide some guidelines on how to set the smoothing hyperparameter, and illustrate on synthetic data the interest of such guidelines. This chapter is based on the following work, accepted at AISTATS 2020:

• M. Massias, Q. Bertrand, A. Gramfort, and J. Salmon. Support recovery and sup-norm convergence rates for sparse pivotal estimation. In AISTATS, 2020a

Introduction

Since the mid 1990's and the development on the Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF], a vast literature has been devoted to sparse regularization for high dimensional regression. Statistical analysis of the Lasso showed that it achieves optimal rates (up to log factor, [START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF]; see also [START_REF] Bühlmann | Statistics for high-dimensional data[END_REF] for an extensive review. Yet, this estimator requires a specific calibration to achieve such an appealing rate: the regularization parameter must be proportional to the noise level. This quantity is generally unknown to the practitioner, hence the development of methods which are adaptive with respect to the noise level. An interesting candidate with such a property is the square-root Lasso ( √ Lasso, [START_REF] Belloni | Square-root Lasso: pivotal recovery of sparse signals via conic programming[END_REF]) defined for an observation vector y ∈ R n , a design matrix X ∈ R n×p and a regularization parameter λ by arg min

β∈R p 1 √ n y -Xβ 2 + λ β 1 . (4.1)
It has been shown to be pivotal with respect to the noise level by [START_REF] Belloni | Square-root Lasso: pivotal recovery of sparse signals via conic programming[END_REF]: the optimal regularization parameter of their analysis does not depend on the true noise level. This feature is also encountered in practice as illustrated by Figure 4.1 (see details on the framework in Section 4.4.1).

Figure 4.1 -Lasso (left) and square-root Lasso (right) optimal regularization parameters λ determined by cross validation on prediction error (blue), as a function of the noise level on simulated values of y. As indicated by theory, the Lasso's optimal λ grows linearly with the noise level, while it remains constant for the square-root Lasso.

Despite this theoretical benefit, solving the square-root Lasso requires tackling a "nonsmooth + nonsmooth" optimization problem. To do so, one can resort to conic programming [START_REF] Belloni | Square-root Lasso: pivotal recovery of sparse signals via conic programming[END_REF] or primal-dual algorithms [START_REF] Chambolle | A first-order primal-dual algorithm for convex problems with applications to imaging[END_REF] for which practical convergence may rely on hard-to-tune hyperparameters. Another approach is to use variational formulations of norms, e.g., expressing the absolute value as |x| = min σ>0

x 2 2σ + σ 2 [START_REF] Bach | Convex optimization with sparsityinducing norms[END_REF], Sec. 5.1, Micchelli et al. 2010). This leads to concomitant estimation [START_REF] Huber | Numerical solution of robust regression problems[END_REF], that is, optimization problems over the regression parameters and an additional variable. In sparse regression, the seminal concomitant approach is the concomitant Lasso [START_REF] Owen | A robust hybrid of lasso and ridge regression[END_REF]:

arg min β∈R p ,σ>0 1 2nσ y -Xβ 2 2 + σ 2 + λ β 1 , (4.2)
which yields the same estimate β as Problem (4.1) whenever y -X β = 0. Problem (4.2) is more amenable: it is jointly convex, and the data fitting term is differentiable. Nevertheless, the data fitting term is still not smooth, as σ can approach 0 arbitrarily:

4.1. INTRODUCTION 87
proximal solvers cannot be applied safely. A solution is to introduce a constraint σ ≥ σ [START_REF] Ndiaye | Efficient smoothed concomitant lasso estimation for high dimensional regression[END_REF], which amounts to smoothing [START_REF] Nesterov | Smooth minimization of non-smooth functions[END_REF][START_REF] Beck | Smoothing and first order methods: A unified framework[END_REF] the square-root Lasso, i.e., replacing its nonsmooth data fit by a smooth approximation (see details in Section 4.1.4).

There exist a straightforward way to generalize the square-root Lasso to the multitask setting (observations Y ∈ R n×T ): the multitask square-root Lasso, arg min

B∈R p×T 1 √ nT Y -XB F + λ B 2,1 , (4.3)
where B 2,1 is the 1 norm of the 2 norms of the rows. Another extension of the square-root Lasso to the multitask case is the multivariate square-root Lasso1 (van de Geer, 2016, Sec. 3.8):

arg min

B∈R p×T 1 nT (n ∧ T ) Y -XB * + λ B 2,1 . (4.4)
It is also shown by van de Geer ( 2016) that when Y -X B is full rank, Problem (4.4) also admits a concomitant formulation, this time with an additional matrix variable:

arg min

B∈R p×T S 0 1 2nT Y -XB 2 S -1 + 1 2n Tr(S) + λ B 2,1 . (4.5)
In the analysis of the square-root Lasso (4.1), the non-differentiability at 0 can be avoided by excluding the corner case where the residuals y -X β vanish. However, analysis of the multivariate square-root Lasso through its concomitant formulation (4.5) has a clear weakness: it requires excluding rank deficient residuals cases, which is far from being a corner case. As illustrated in Figure 4.2, the full rank assumption made by van de Geer and Stucky (2016, Lemma 1) or Molstad (2019, Rem. 1) is not realistic, even for T ≥ n and high values of λ (see Section 4.4 for the setting's details). Motivated by numerical applications, Massias et al. (2018a) introduced a lower bound on the smallest eigenvalue of S (S σ Id n ) in Problem (4.5) to circumvent this issue. As we will see in Chapter 5, this amounts to smoothing the nuclear norm.

Our goal is to prove sup-norm convergence rates and support recovery guarantees for the estimators introduced above, and their smoothed counterparts.

Related works. The statistical properties of the Lasso have been studied under various frameworks and assumptions. [START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF] showed that with high probability, X( ββ * ) 2 vanishes at the minimax rate (prediction convergence), whereas [START_REF] Lounici | Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators[END_REF] proved the sup-norm convergence and the support recovery of the Lasso (estimation convergence), i.e., controlled the quantity ββ * ∞ . The latter result was extended to the multitask case by [START_REF] Lounici | Oracle inequalities and optimal inference under group sparsity[END_REF].

Since then, other Lasso-type estimators have been proposed and studied, such as the square-root Lasso [START_REF] Belloni | Square-root Lasso: pivotal recovery of sparse signals via conic programming[END_REF] or the scaled Lasso [START_REF] Sun | Scaled sparse linear regression[END_REF]. In the multitask case, [START_REF] Liu | Calibrated multivariate regression with application to neural semantic basis discovery[END_REF] introduced the Calibrated Multivariate Regression, and van de Geer and Stucky (2016); Molstad (2019) studied the multivariate square-root Lasso. These estimators have been proved to converge in prediction. However, apart from [START_REF] Bunea | The group square-root Lasso: Theoretical properties and fast algorithms[END_REF] for a particular group square-root Lasso, we are not aware of other works showing sup-norm convergence2 of these estimators.

Within the framework introduced by Lounici ( 2008), our contributions are the following:

• We prove sup-norm convergence and support recovery of the multitask square-root Lasso and its smoothed version.

• We prove sup-norm convergence and support recovery of the multivariate squareroot Lasso (van de Geer and Stucky, 2016, Sec. 2.2), and a smoothed version of it.

• Theoretical analysis leads to guidelines for the setting of the smoothing parameter σ. In particular, as soon as σ ≤ σ * / √ 2, the "optimal" λ and the sup-norm bounds obtained do not depend on σ.

• We show on synthetic data the support recovery performance is little sensitive to the smoothing parameter σ as long as σ ≤ σ * / √ 2.

Our contributions with respect to the existing literature are summarized in Table 4.1.

89

where Y ∈ R n×T , X ∈ R n×p is the deterministic design matrix, B * ∈ R p×T are the true regression coefficients and E ∈ R n×T models a centered noise.

For an estimator B of B * , we aim at controlling B -B * 2,∞ with high probability, and showing support recovery guarantees provided the non-zero coefficients are large enough. To prove such results, the following assumptions are classical: Gaussianity and independence of the noise, and mutual incoherence.

Assumption 4.1. The entries of E are i.i.d. N (0, σ * 2 ) random variables.

Assumption 4.2 (Mutual incoherence). The Gram matrix Ψ 1 n X X satisfies Ψ jj = 1 , and max

j =j |Ψ jj | ≤ 1 7αs , ∀j ∈ [p] , (4.7)
for some integer s ≥ 1 and some constant α > 1.

Mutual

min S⊂[p] |S|≤s min ∆ =0 ∆ S c 2,1 ≤3 ∆ S 2,1 1 √ n X∆ F ∆ S F ≥ 1 -1 α > 0 . (4.8)
In particular, with the choice

∆ B -B * , if ∆ S c * 2,1
≤ 3 ∆ S * 2,1 , the following bound holds:

1 n X∆ 2 F ≥ 1 - 1 α ∆ S * 2 F .
(4.9)

Motivation and general proof structure

Structure of all proofs. We prove results of the following form for several estimators B (summarized in Table 4.1): for some parameter λ independent of the noise level σ * , with high probability,

1 T B -B * 2,∞ ≤ C 1 √ nT log p T σ * . (4.10)
Then, assuming a signal strong enough such that

min j∈S * 1 T B * j: 2 > 2C 1 √ nT log p T σ * , (4.11)
on the same event, for some η > 0,

Ŝ {j ∈ [p] : 1 T Bj: 2 > C(3 + η)λσ * } (4.12)
matches the true sparsity pattern: Ŝ = S * .

We explain here the general sketch proofs for all the estimators. We assume that Assumption 4.2 holds and then place ourselves on an event A such that X Z 2,∞ ≤ λ/2 (for a Z ∈ ∂f (E), where f is the data fitting term) in order to use Lemma 4.5 (b), which links the control of Ψ( B -B * ) 2,∞ to the control of B -B * 2,∞ . To obtain sup-norm convergence it remains for each estimator to: • control the probability of the event A with classical concentration inequalities.

• control the quantity Ψ( B -B * ) 2,∞ , with:

first order optimality conditions, which provide a bound on X Z 2,∞ :

X Ẑ 2,∞ ≤ λ for a Ẑ ∈ ∂f ( Ê),
the definition of the event A, for some estimators, an additional assumption (Assumption 4.8).

Next, we detail the lemmas used in this strategy.

Preliminary lemma

First we show that if Assumption 4.2 holds, then a bound "in prediction", on Ψ∆ 2,∞ , leads to a bound "in estimation", on ∆ 2,∞ .

Lemma 4.4. Let Ψ, α and s satisfy Assumption 4.2, let B be an estimator satisfying:

∆ S c * 2,1 ≤ 3 ∆ S * 2,1 , then: a) ∆ S * F ≤ α α-1 4 √ s Ψ∆ 2,∞ , b) ∆ 2,1 ≤ α α-1 16s Ψ∆ 2,∞ , c) ∆ 2,∞ ≤ 1 + 16 7(α-1)
Ψ∆ 2,∞ .

Proof For Lemma 4.4 (a), the idea is to upper and lower bound 1 n X∆ 2 F . First we bound ∆ 2,1 :

∆ 2,1 = ∆ S c * 2,1 + ∆ S * 2,1 ≤ 4 ∆ S * 2,1 ≤ 4 √ s ∆ S * F . (4.13)
Now we can upper bound 1 n X∆ 2 F with Hölder inequality and Equation (4.13):

1 n X∆ 2 F = ∆, Ψ∆ ≤ ∆ 2,1 Ψ∆ 2,∞ ≤ 4 √ s ∆ S * F Ψ∆ 2,∞ . (4.14) 4.1. INTRODUCTION 91
By Equation (4.9) and Equation (4.14):

(

1 -1 α ) ∆ S * 2 F ≤ 1 n X∆ 2 F ≤ 4 √ s ∆ S * F Ψ∆ 2,∞ ∆ S * F ≤ α α -1 4 √ s Ψ∆ 2,∞ . (4.15)
Lemma 4.4 (b) is a direct consequence of Equation (4.13) and lemma 4.4 (a):

∆ 2,1 ≤ 4 √ s ∆ S * F ≤ α α -1 16s Ψ∆ 2,∞ . (4.16)
Finally, for Lemma 4.4 (c), for any j ∈ [p],

Ψ∆ j: = ∆ j: + j =j Ψ j j ∆ j :

|| Ψ∆ j: -∆ j: || 2 ≤ j =j |Ψ jj | × ||∆ j : || 2 || Ψ∆ j: -∆ j: || 2 ≤ 1 7αs j =j ∆ j : ∆ 2,∞ ≤ Ψ∆ 2,∞ + 1 7αs ∆ 2,1 ≤ 1 + 16 7(α-1)
Ψ∆ 2,∞ .

(4.17) using Assumption 4.2 and lemma 4.4 (b).

We now provide conditions leading to ∆ S c * 2,1 ≤ 3 ∆ S * 2,1 , to be able to apply Lemma 4.3. In this section we consider estimators of the form B arg min

B∈R p×T f (Y -XB) + λ B 2,1 , (4.18)
for a proper, lower semi-continuous and convex function f : R n×T → R (see the summary in Table 4.1). Fermat's rule for Problem (4.18) reads:

0 ∈ X ∂f ( Ê) + λ∂ • 2,1 ( B) , (4.19)
Hence, we can find Ẑ ∈ ∂f ( Ê) such that

X Ẑ 2,∞ ≤ λ . (4.20)
Lemma 4.5. Consider an estimator based on Problem (4.18), and assume that there exists Z ∈ ∂f (E) such that X Z 2,∞ ≤ λ/2. Then:

a) ∆ S c * 2,1 ≤ 3 ∆ S * 2,1 , b) if Ψ and α satisfy Assumption 4.2, ∆ 2,∞ ≤ 1 + 16 7(α -1) Ψ∆ 2,∞ .
Proof For Lemma 4.5 (a), we use the minimality of B:

f ( Ê) -f (E) ≤ λ B * 2,1 -λ B 2,1 . (4.21)
We upper bound the right hand side of Equation (4.21), using B 2,1 = BS * 2,1 + BS c * 2,1 , B * S c * = 0 and with the triangle inequality:

B * 2,1 -B 2,1 = B * S * 2,1 -BS * 2,1 -BS c * 2,1 = B * S * 2,1 -BS * 2,1 -∆ S c * 2,1 ≤ (B * -B) S * 2,1 -∆ S c * 2,1 ≤ ∆ S * 2,1 -∆ S c * 2,1 . (4.22)
We now aim at finding a lower bound of the left hand side of Equation (4.21). By convexity of f , ∂f

(E) = ∅. Picking Z ∈ ∂f (E) such that X Z 2,∞ ≤ λ 2 yields: f (Y -X B) -f (Y -XB * ) ≥ -Z, X( B -B * ) ≥ -X Z, ∆ ≥ -X Z 2,∞ ∆ 2,1 ≥ - 1 2 λ ∆ 2,1 .
Combining Equations (4.21) to (4.23) leads to:

- 1 2 ∆ 2,1 ≤ ∆ S * 2,1 -∆ S c * 2,1 ∆ S c * 2,1 ≤ 3 ∆ S * 2,1 . (4.23)
Proof of Lemma 4.5 (b) is a direct application of Lemmas 4.5 (a) and 4.4 (c).

Equipped with these Assumptions and Lemmas, we will show that the considered estimators reach the minimimax lower bounds, which we recall in the following.

Minimax lower bounds

As said in Section 4.1.1, our goal is to provide convergence rates on the quantity B -B * 2,∞ . To show that our bounds are "optimal" we recall that the considered estimators achieve minimax rate (up to a logarithmic factor). Indeed, under some additional assumptions controlling the conditioning of the design matrix, one can show [START_REF] Lounici | Oracle inequalities and optimal inference under group sparsity[END_REF]) minimax lower bounds.

Assumption 4.6. For all ∆ ∈ R p×T \{0} such that |S(∆)| ≤ 2|S * |: κ ≤ X∆ 2 F n ∆ 2 F ≤ κ . (4.24)
Provided Assumptions 4.1 and 4.6 hold true, Lounici et al. (2011, Thm. 6.1) proved the following minimax lower bound (with an absolute constant R):

inf B sup B * s.t. |S(B * )|≤s E 1 T B -B * 2,∞ ≥ Rσ * κ√ n 1 + log(ep/s) T .

Smoothing

Some of the pivotal estimators studied here are obtained via a technique called smoothing. For L > 0, a convex function φ is L-smooth (i.e., its gradient is L-Lipschitz) if and only if its Fenchel conjugate φ * is 1 L -strongly convex (Hiriart-Urruty and Lemaréchal, 1993, Thm 4.2.1). Therefore, given a smooth function ω, a principled way to smooth a function f is to add the strongly convex ω * to f * , thus creating a strongly convex function, whose Fenchel transform is a smooth approximation of f . Formally, given a smooth convex function ω, the ω-smoothing of f is (f * + ω * ) * . By properties of the Fenchel transform, the latter is also equal to f ω whenever f is convex (Bauschke and Combettes, 2011, Prop. 13.21).

Proposition 4.7. Let ω σ = 1 2σ • 2 F + σ 2 .
The ω σ -smoothing of the Frobenius norm is equal to:

ω σ • F (Z) =    Z F , if Z F ≤ σ , 1 2σ Z 2 F + σ 2 , if Z F ≥ σ . = min σ≥σ 1 2σ Z 2 F + σ 2 . (4.25)

Multitask square-root Lasso

It is clear that the multitask square-root Lasso (Problem (4.3)) suffers from the same numerical weaknesses as the square-root Lasso. A more amenable version is introduced in Chapter 5. The smoothed multitask square-root Lasso is obtained by replacing the nonsmooth function • F with a smooth approximation, depending on a parameter σ > 0:

arg min

B∈R p×T • F 1 2σ • 2 + σ 2 Y -XB √ nT + λ B 2,1 . (4.26)
Plugging the expression of the smoothed Frobenius norm (4.25), the problem formulation becomes:

( B, σ) ∈ arg min B∈R p×T σ≥σ 1 2nT σ Y -XB 2 F + σ 2 + λ B 2,1 , (4.27)
where the data fitting term is (nT σ) -1 -smooth with respect to B. We show that estimators (4.3) and (4.26) reach the minimax lower bound, with a regularization parameter independent of σ * . For that, another assumption is needed.

√ 2 √ nT 1 + A (log p)/T , if σ ≤ σ * √ 2
then with probability at least 1p 1-A 2 /2 -(1 + e 2 )e -nT /24 ,

1 T B -B * 2,∞ ≤ C(3 + η)λσ * . (4.29) SMOOTHING Moreover provided that min j∈S * 1 T B * j: 2 > 2C(3 + η)λσ * , (4.30)
then, with the same probability, the estimated support

Ŝ {j ∈ [p] : 1 T Bj: 2 > C(3 + η)λσ * } (4.31)
recovers the true sparsity pattern: Ŝ = S * .

Proof We first bound Ψ∆ 2,∞ . Let A 1 be the event

A 1 X E 2,∞ √ nT E F ≤ λ 2 ∩ σ * √ 2 < E F √ nT < 2σ * . (4.32)
By Lemma 4.16 (h), P(A 1 ) ≥ 1p 1-A 2 /2 -(1 + e 2 )e -nT /24 . For both estimators, on A 1 we have:

n Ψ∆ 2,∞ = X ( Ê -E) 2,∞ ≤ X Ê 2,∞ + X E 2,∞ ≤ X Ê 2,∞ + λnT σ * , (4.33)
hence we need to bound X Ê 2,∞ . We do so using optimality conditions, that yield for Problem (4.3), with Ê = 0,

X Ê Ê F 2,∞ ≤ λ √ nT 1 nT X Ê 2,∞ ≤ λ Ê F √ nT , (4.34)
and the last equation is still valid if Ê = 0. For Problem (4.26), the optimality conditions yield:

   1 nT X Ê 2,∞ ≤ λ Ê F √ nT , if Ê F √ nT ≥ σ , 1 nT X Ê 2,∞ ≤ λσ , otherwise . (4.35) Therefore, 1 nT X Ê 2,∞ ≤ λ max Ê F √ nT , σ . (4.36)
It now remains to bound Ê F for both estimators, which is done with Assumption 4.8: for Problem (4.3), by minimality of the estimator,

1 √ nT Ê F + λ B 2,1 ≤ 1 √ nT E F + λ B * 2,1 1 √ nT Ê F ≤ 1 √ nT E F + λ B * 2,1 ≤ 2σ * + (1 + η)σ * ≤ (3 + η)σ * , (4.37)
and we can obtain the same bound in the case of Problem (4.26): by the minimality of the estimator we have on A 1 :

• F 1 2σ + σ 2 • 2 F ( Ê √ nT ) + λ B 2,1 ≤ • F 1 2σ • 2 F + σ 2 ) ( E √ nT ) + λ B * 2,1 Ê F + λ B 2,1 ≤ • F 1 2σ • 2 F + σ 2 )( E √ nT ) + λ B * 2,1 since • F ≤ • F 1 2σ • 2 F + σ 2 1 √ nT Ê F ≤ 1 √ nT E F + λ B * 2,1 since 1 √ nT E F ≥ σ * √ 2 ≥ σ ≤ 2σ * + λ B * 2,1 since 1 √ nT E F ≤ 2σ * ≤ 2σ * + (1 + η)σ * since λ B * 2,1 ≤ (1 + η)σ * 1 √ nT Ê F ≤ (3 + η)σ * .
Combining Equations (4.33), (4.34), (4.36) and (4.37) we have in both cases:

1 T Ψ∆ 2,∞ ≤ (3 + η)λσ * . (4.38)
Finally we exhibit an element of ∂f (E) to apply Lemma 4.5 (b). Recall that f =

1 √ nT • F for Problem (4.3), and f = • F 1 2σ • 2 F + σ 2 ( • √ nT ) for Problem (4.26). On A 1 , ∂f (E) is a singleton for both estimators, whose element is E/( E F √ nT ). Additionally, on A 1 the inequality 1 √ nT X E 2,∞ E F ≤ λ
2 holds, meaning we can apply Lemma 4.5 (b) with Z = E/( E F √ nT ). This proves the bound on ∆ 2,∞ . Then, the support recovery property easily follows from Lounici et al. (2009, Cor. 4.1).

Single task case. For the purpose of generality, we proved convergence results for the multitask versions of the square-root/concomitant Lasso and its smoothed version, but the results are also interesting in the single task setting: in this case (T = 1) it is possible to achieve tighter convergence rates. Then with probability at least 1 -

p 1-A 2 /8 -(1 + e 2 )e -n/24 , 1 T β -β * 2,∞ ≤ C(2 + η)λσ . (4.40) Moreover if min j∈S * |β * j | > 2C(2 + η)λσ , (4.41)
then with the same probability:

Ŝ = {j ∈ [p] : | βj | > C(2 + η)λσ} (4.42)
estimate correctly the true sparsity pattern:

Ŝ = S * . (4.43)
Proof All the inequalities leading to Equation (4.38) still hold. The control of the event A 1 can be tighter in the single-task case. Since A

1 = 1 n X ε ∞ ≤ λ 2 σ * √ 2 ∩ σ * √ 2 < ε F √ nT < 2σ *
, with λ = A 2 log p/n, concentration in equalities from Lemmas 4.16 (b) and 4.16 (e) in Section 4.A leads to:

P(A 1 ) ≥ 1 -p 1-A 2 /8 -(1 + e 2 )e -n/24 .
(4.44)

Multivariate square-root Lasso

Here we show that the multivariate square-root Lasso3 and its smoothed version also reach the minimax rate. Recall that the multivariate square-root Lasso is Problem (4.4).

For the numerical reasons mentioned above, as well as to get rid of the invertibility assumption of Ê Ê, we consider the smoothed estimator of Massias et al. (2018a):

arg min B∈R p×T σ Idn S σ Idn 1 2nT Y -XB 2 S -1 + Tr S 2n + λ B 2,1 . (4.45)
The variable introduced by concomitant formulation is now a matrix S, corresponding to the square root of the noise covariance estimate. The multivariate square-root Lasso (4.4) and its concomitant formulation (4.5) have the same solution in B provided Ê Ê is invertible. In this case, the solution of Problem (4.5

) in S is Ŝ = ( 1 T ÊÊ ) 1 2 .
Problem (4.45) is actually a small modification of Massias et al. (2018a), where we have added the second constraint S σ Id n . σ can for example be set as ( 1 T Y Y ) 1/2 2 , as Figure 4.2 illustrates that this is the order of magnitude of Ŝ 2 . Because of these constraints, the solution in S is different from that of Problem (4.5). We write a singular value decomposition of 1

√ T Ê: U DV , with D = diag(γ i ) ∈ R n×n , U ∈ R n×n and V ∈ R T ×n such that U U = V V = Id n . Then the solution in S to Problem (4.45) is Ŝ = U diag [γ i ] σ
σ U (this result can be derived from Massias et al. 2018a, Prop. 2). Ŝ can be used to bound X Ê 2,∞ : Lemma 4.11. For the concomitant multivariate square-root Lasso (4.5) and the smoothed concomitant multivariate square-root (4.45) we have:

X Ê 2,∞ ≤ Ŝ 2 X Ŝ-1 Ê 2,∞ .
(4.46)

Proof

Concomitant multivariate square-root ( Ŝ = ( ÊÊ ) 1/2 ). We recall that U DV is a singular value decomposition of 1

√ T Ê, with D = diag(γ i ) ∈ R r×r , U ∈ R n×r and V ∈ R T ×r such that U U = V V = Id r . 4.3. MULTIVARIATE SQUARE-ROOT LASSO 97 We have, observing that Ŝ-1 Ê = (U D 2 U ) -1/2 √ T U DV = √ T U V : X Ê = √ T X U DV (4.47) = √ T X U V V DV (4.48) = X Ŝ-1 ÊV DV . (4.49) Therefore, X Ê 2,∞ ≤ V DV 2 X Ŝ-1 Ê 2,∞ (4.50) ≤ Ŝ 2 X Ŝ-1 Ê 2,∞ . (4.51)
Equation (4.46) also holds for Problem (4.45).

Smoothed concomitant multivariate square-root (

Ŝ = U diag([γ i ] σ σ )U ). We recall that U DV is a singular value decomposition of 1 √ T Ê, with D = diag(γ i ) ∈ R n×n , U ∈ R n×n and V ∈ R T ×n such that U U = V V = Id n . Observing that Ŝ-1 Ê = U diag([γ i ] σ σ ) -1 U √ T U DV = √ T U diag([γ i ] σ σ ) -1 diag(γ i )V , (4.52)
we have

X Ê = √ T X U diag(γ i )V = √ T X U diag([γ i ] σ σ ) -1 diag(γ i ) diag([γ i ] σ σ )V = √ T X U diag([γ i ] σ σ ) -1 diag(γ i )V T V diag([γ i ] σ σ )V = √ T X Ŝ-1 ÊV diag([γ i ] σ σ )V .
Therefore,

X Ê 2,∞ ≤ X Ŝ-1 Ê 2,∞ V diag([γ i ] σ σ )V 2 (4.53) ≤ X Ŝ-1 Ê 2,∞ Ŝ 2 . (4.54)
We can prove the minimax sup-norm convergence of these two estimators, using the following assumptions.

Assumption 4.12. For the multivariate square-root Lasso, Ê Ê is invertible, and there exists η such that ( 1

T Ê Ê) 1 2 2 ≤ (2 + η)σ * .
We get rid of this very strong hypothesis for the smoothed version, as the estimated noise covariance is invertible because of the constraint S σ Id n , and we can control its operator norm via the constraint S σ Id n . We still need an assumption on σ and σ.

Assumption 4.13. σ, σ and η verify: σ ≤ σ * √ 2 and σ = (2 + η)σ * with η ≥ 1.

Proposition 4.14. For the multivariate square-root Lasso (4.4) (resp. its smoothed version (4.45)), let Assumption 4.1 be satisfied, let α satisfy Assumption 4.2 and let η satisfy Assumption 4.12 (resp. let σ, σ, η satisfy Assumption 4.13). Let C = 1 + 7(α-1) , A ≥ √ 2, and λ = 2 √ 2 √ nT (1 + A (log p)/T ). Then there exists c ≥ 1/64 such that with probability at least 1p

1-A 2 /2 -2ne -cT /n , 1 T B -B * 2,∞ ≤ C(3 + η)λσ * . (4.55) Moreover if min j∈S * 1 T B * j: 2 > 2C(3 + η)λσ * , (4.56)
then with the same probability:

Ŝ {j ∈ [p] : 1 T Bj: 2 > C(3 + η)λσ * } (4.57)
correctly estimates the true sparsity pattern: Ŝ = S * .

Proof Let A 2 be the event:

X E 2,∞ nT ≤ λσ * 2 √ 2 ∩ {2σ * Id T ( E E n ) 1 2 σ * √ 2 Id T } . (4.58) By Lemma 4.16 (i), P(A 2 ) ≥ 1 -p 1-A 2 /2 -2ne -cT /n (c ≤ 1/64).
When the multivariate square-root Lasso residuals are full rank, the optimality conditions for Problems (4.4) and (4.45) read the same, but with differents Ŝ (introduced above):

X Ŝ-1 Ê 2,∞ ≤ λT n . (4.59)
With Lemma 4.11 and eq. (4.59) and Assumption 4.12 for the multivariate square-root Lasso (or Assumption 4.13 for its smoothed version):

n Ψ∆ 2,∞ = X (E -Ê) 2,∞ ≤ X Ê 2,∞ + X E 2,∞ ≤ λT n Ŝ 2 + X E 2,∞ ≤ λ(2 + η)T nσ * + X E 2,∞ . (4.60)
Then on the event A 2 : • 2 S -1 + Tr S 2n for Problem (4.45). We also recall that for a full rank matrix A ∈ R n×T (Koltchinskii et al., 2011, Sec. 2):

1 T Ψ∆ 2,∞ ≤ λ(2 + η)σ * + 1 nT X E 2,∞ ≤ 3 + η λσ * . ( 4
∂ A * = {(AA ) -1/2 A} .
(4.62)

On A 2 , ∂f (E) is a singleton for both estimators, whose element is (EE ) -1/2 E/(n √ T ). Additionally on A 2 :

1 n √ T X (EE ) -1/2 E 2,∞ ≤ 1 nT X E 2,∞ ( EE T ) -1/2 2 ≤ λσ * 2 √ 2 × √ 2 σ * ≤ λ 2 , (4.63)
meaning we can apply Lemma 4.5 (b) with Z = E(E E) -1/2 /n √ T . This proves the bound on ∆ 2,∞ . Then, the support recovery property easily follows from Lounici et al. (2009, Cor. 4.1).

Experiments

We first describe the setting of Figures 4.1 and 4.2. Then we show that empirically that results given by Propositions 4.9 and 4.14 hold in practice. The signal-to-noise ratio (SNR) is defined as XB * F Y -XB F .

Pivotality of the square-root Lasso

In this experiment the matrix X consists of the 10 000 first columns of the climate dataset (n = 864). We generate β * with 20 non-zero entries. Random Gaussian noise is added to Xβ * to create y, with a noise variance σ * controlling the SNR.

For each SNR value, both for the Lasso and the square-root Lasso, we compute the optimal λ on a grid between λ max (the estimator specific smallest regularization level yielding a 0 solution), using cross validation on prediction error on left out data. For each SNR, results are averaged over 10 realizations of y.

Figure 4.1 shows that, in accordance with theory, the optimal λ for the Lasso depends linearly on the noise level, while the square-root Lasso achieves pivotality.

Rank deficiency experiment

For (n, T, p) = (10, 20, 30), we simulate data: entries of X are i.i.d. N (0, 1), B * has 5 non zeros rows, and Gaussian noise is to XB * added to result in a SNR of 1. We reformulate Problem (4.4) as a Conic Program, and solve it with the SCS solver of cvxpy (O' Donoghue et al., 2016;[START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF] for various values of λ (λ max is the smallest regularization value yielding a null solution). We then plot the singulars values of the residuals at optimum, shown on Figure 4.2.

Since the problem is reformulated as a Conic Program and solved approximately (precision = 10 -6 ), the residuals are not exact; however the sudden drop of singular values of Y -X B must be interpreted as the singular value being exactly 0. One can see that even for very high values of λ, the residuals are rank deficient while the matrix Y is not. This is most likely due to the trace penalty on S in the equivalent formulation of Problem (4.5), encouraging singular values to be 0. Therefore, even on simple toy data, the hypothesis used by van de Geer and Stucky (2016); Molstad (2019) does not hold, justifying the need for smoothing approaches, both from practical and theoretical point of views.

(Multitask) smoothed concomitant Lasso

Here we illustrate, as indicated by theory, that when the smoothing parameter σ is sufficiently small, the multitask SCL is able to recover the true support (Proposition 4.9). More precisely, when σ ≤ σ * / √ 2, there exist a λ, independent of σ and σ * , such that the multitask SCL recovers the true support with high probability. We use (n, T, p) = (50,50,1000) with parameter ρ X = 0.5 (correlation between X :i and X :j is ρ |i-j| X ), and its columns have unit Euclidean norm. The true coefficient B * has 5 non-zeros rows whose entries are i.i.d. N (0, 1).

Comments on Figures 4.3 and 4.4

The multitask SCL relies on two hyperparameters: the penalization coefficient λ and the smoothing parameter σ, whose influence we study here. The goal is to show empirically that when σ ≤ σ * / √ 2 the optimal λ does not depend on the smoothing parameter σ. We vary λ and σ on a grid: for each pair (λ, σ) we solve the multitask SCL. For each solution B(λ,σ) we then compute a metric, the hard recovery (Figure 4.3) or the size of the support (Figure 4.4). The metrics are averaged over 100 realizations of the noise. Figure 4.3 shows the latter graph for different values of SNR. We can see that when σ ≤ σ * , support recovery is achieved for λ independent of σ. As soon as σ > σ * the optimal λ depends on σ. When σ reaches a large enough value (i.e., σ * ) then the recovery profile is modified: the optimal λ decreases as σ grows. This is logical, since as soon as the constraint is saturated, the (multitask) SCL boils down to a multitask Lasso with regularization parameter λσ:

B arg min

B∈R p×T 1 2nT Y -XB 2 F + λσ B 2,1 . (4.64)
Figure 4.4 shows that with a fixed λ higher values of σ may lead to smaller support size, see e.g., λ/λ 0 = 0.32.

Smoothed generalized concomitant Lasso (SGCL)

The experimental setting is the same as before, except here we used (n, T, p) = (150, 100, 500). Figure 4.5 illustrates Proposition 4.14. When σ ≤ σ * , there exist a λ that does not depend on σ and such that SGCL finds the true support S * . However, as before, when σ ≥ √ 2σ * , λ depends on σ.

Conclusion

We have proved sup norm convergence rates and support recovery for a family of sparse estimators derived from the square-root Lasso. We showed that they are pivotal too: the optimal regularization parameter does not depend on the noise level. We showed that their smoothed versions retain these properties while being simpler to solve, and requiring more realistic assumptions to be analyzed. These findings were corroborated numerically, in particular for the influence of the smoothing parameter.

1 n X ε ∞ ≤ λ 2 . Take λ = Aσ * (log p)/n and A > 2 √
2, then:

P(C 1 ) ≥ 1 -2p 1-A 2 8 . (4.66) b) Let C 1 1 n X ε ∞ ≤ λ 2 σ * √
2 . Take λ = A (2 log p)/n and A > 2 √ 2, then:

P(C 1 ) ≥ 1 -2p 1-A 2 8 . (4.67) c) Let C 2 1 nT X E 2,∞ ≤ λ 2 . Take λ = 2σ * √ nT 1 + A log p T and A > √ 2, then: P(C 2 ) ≥ 1 -p 1-A 2 /2 . (4.68)
Another possible control is Lounici et al. (2009, Proof of Lemma 3.1, p. 6). Let A > 8 and λ = 2σ * √ nT 1 + A log p √ T , then:

P(C 2 ) ≥ 1 -p min(8 log p,A √ T /8) . (4.69) d) Let C 3 1 nT X E 2,∞ ≤ λσ * 2 √ 2 . Take λ = 2 √ 2 √ nT 1 + A log p T and A > √ 2,
then:

P(C 3 ) ≥ 1 -p 1-A 2 /2 . (4.70) e) Let C 4 σ * √ 2 < ε F √ nT < 2σ * . Then: P(C 4 ) ≥ 1 -(1 + e 2 )e -nT /24 . (4.71) f ) Let C 5      EE T 1 2 σ * √ 2     
. Then with c ≥ 1 32 :

P(C 5 ) ≥ 1 -ne -cT /(2n
) Let C 6      2σ * EE T 1 2     
. Then with c ≥ 1 32 :

P(C 6 ) ≥ 1 -ne -cT /n . (4.73) h) Let us recall that A 1 = X E 2,∞ √ nT E F ≤ λ 2 ∩ σ * √ 2 < E F √ nT < 2σ * , we have P(A 1 ) ≥ 1 -p 1-A 2 /2 -(1 + e 2 )e -nT /24 . (4.74) i) Let us recall that A 2 = X E 2,∞ nT ≤ λσ * 2 √ 2 ∩ 2σ * Id T ( EE n ) 1 2 σ * √ 2 Id T
Proof Lemma 4.16 (a): Lemma 4.16 (c): since E is isotropic, the law of u E is the same for all vectors u ∈ R n of same norm. In particular, X :1 E and √ ne 1 E = √ nE 1: have the same law.

P C c 1 ≤ p P X :1 ε ≥ nλ/2 ≤ p P ε 1 ≥ √ nλ/2 ( X :1 = √ n) ≤ p P ε 1 /σ ≥ √ nλ/(2σ) ≤ 2p exp - nλ 2 8σ 2 (Theorem 4.15) ≤ 2p 1-A 2 8 (λ = Aσ (log p)/n) . ( 4
The variable 1 σ E 1: 2 is a chi variable with T degrees of freedom, and (4.76) where the bound can be proved by recursion. We have:

1 σ E[ E 1: 2 ] = √ 2Γ( T +1 2 ) Γ( T 2 ) ∈ T √ T + 1 , √ T ,
P C c 2 ≤ p P X :1 E 2 ≥ T nλ/2 ≤ p P E 1: 2 ≥ T √ nλ/2
(by isotropy of E)

≤ p P E 1: 2 ≥ σ √ T + Aσ log p (λ = 2σ T √ n ( √ T + A log p)) ≤ p P E 1: 2 ≥ E( E 1: 2 ) + Aσ log p (σ √ T ≥ E( E 1: 2 ) ≤ p 1-A 2 2 (Theorem 4.15) . (4.77)
The proof of the other control of A 2 can be found in Lounici et al. (2009, Proof of Lemma 3.1, p. 6).

Lemma 4.16 (d) is a direct consequence of Lemma 4.16 (c).

Proof of Lemma 4.16 (e) can be found in Giraud (2014, Proof of Lemma 5.4 p. 112), who control the finer event

{ σ √ 2 ≤ ε √ n ≤ (2 -1 √
2 )σ}. Proof of Lemmas 4.16 (f) and 4.16 (g) are particular cases of Gittens and Tropp (2011, Cor. 7.2, p. 15).

Proof of Lemma 4.16 (h) is done using Lemmas 4.16 (d) and 4.16 (e). Indeed we

have A 1 ⊃ 1 nT X E 2,∞ ≤ λσ * 2 √ 2 ∩ σ * √ 2 < E F √ nT < 2σ * = C 3 ∩ C 4 . Hence P(C 1 ) ≥ 1 -P(C c 3 ) -P(C c 4 ) ≥ 1 -p 1-A 2 /2 -(1 + e
2 )e -nT /24 . Proof of Lemma 4.16 (i) is done using Lemmas 4.16 (d), 4.16 (f) and 4.16 (g). Indeed 2n) .

A 2 = C 3 ∩C 5 ∩C 6 . Hence P(A 2 ) ≥ 1-P(C c 3 )-P(C c 5 )-P(C c 6 ) ≥ 1-p 1-A 2 /2 -2ne -cT /(

Application to neuroimaging

Introduction

In many statistical applications, the number of parameters p is much larger than the number of observations n. A popular approach to tackle linear regression problems in such scenarios is to consider convex 1 -type penalties, as popularized by [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]. The use of these penalties relies on a regularization parameter λ trading data fidelity versus sparsity. Unfortunately, [START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF] showed that, in the case of white Gaussian noise, the optimal λ depends linearly on the standard deviation of the noise -referred to as noise level. Because the latter is rarely known in practice, one can jointly estimate the noise level and the regression coefficients, following pioneering work on concomitant estimation [START_REF] Huber | Numerical solution of robust regression problems[END_REF][START_REF] Huber | Robust Statistics[END_REF]. Adaptations to sparse regression [START_REF] Owen | A robust hybrid of lasso and ridge regression[END_REF] have been analyzed under the names of square-root Lasso [START_REF] Belloni | Square-root Lasso: pivotal recovery of sparse signals via conic programming[END_REF] or scaled Lasso [START_REF] Sun | Scaled sparse linear regression[END_REF]. Generalizations have been proposed in the multitask setting, the canonical estimator being Multi-Task Lasso [START_REF] Obozinski | Joint covariate selection and joint subspace selection for multiple classification problems[END_REF].

The latter estimators take their roots in a white Gaussian noise model. However some real-world data (such as magneto-electroencephalographic data) are contaminated with strongly non-white Gaussian noise [START_REF] Engemann | Automated model selection in covariance estimation and spatial whitening of MEG and EEG signals[END_REF]. From a statistical point of view, the non-uniform noise level case has been widely explored: [START_REF] Daye | High-dimensional heteroscedastic regression with an application to eQTL data analysis[END_REF]; [START_REF] Wagener | Bridge estimators and the adaptive Lasso under heteroscedasticity[END_REF]; [START_REF] Kolar | Variance function estimation in high-dimensions[END_REF]; [START_REF] Dalalyan | Learning heteroscedastic models by convex programming under group sparsity[END_REF].

In a more general case, with a correlated Gaussian noise model, estimators based on non-convex optimization problems were proposed [START_REF] Lee | Simultaneous multiple response regression and inverse covariance matrix estimation via penalized Gaussian maximum likelihood[END_REF] and analyzed for sub-Gaussian covariance matrices (Chen and Banerjee, 2017) through the lens of penalized maximum likelihood estimation (MLE). Other estimators [START_REF] Rothman | Sparse multivariate regression with covariance estimation[END_REF][START_REF] Rai | Simultaneously leveraging output and task structures for multiple-output regression[END_REF] assume that the inverse of the covariance (the precision matrix ) is sparse, but the underlying optimization problems remain non-convex. A convex approach to regression with correlated noise, the Smooth Generalized Concomitant Lasso (SGCL) was proposed by Massias et al. (2018a). Relying on smoothing techniques [START_REF] Moreau | Proximité et dualité dans un espace hilbertien[END_REF][START_REF] Nesterov | Smooth minimization of non-smooth functions[END_REF][START_REF] Beck | Smoothing and first order methods: A unified framework[END_REF], the SGCL jointly estimates the regression coefficients and the noise co-standard deviation matrix (the square root of the noise covariance matrix). However, in applications such as M/EEG, the number of parameters in the co-standard deviation matrix (≈ 10 4 ) is typically equal to the number of observations, making it statistically hard to estimate accurately.

In this chapter we consider applications to M/EEG data in the context of neuroscience. M/EEG data consists in recordings of the electric and magnetic fields at the surface or close to the head. Here we tackle the source localization problem, which aims at estimating which regions of the brain are responsible for the observed electro-magnetic signals: this problem can be cast as a multitask high dimensional linear regression [START_REF] Ndiaye | Gap safe screening rules for sparse multi-task and multi-class models[END_REF]. MEG and EEG data are obtained from heterogeneous types of sensors: magnetometers, gradiometers and electrodes, leading to samples contaminated with different noise distributions, and thus non-white Gaussian noise. Moreover the additive noise in M/EEG data is correlated between sensors and rather strong: the noise variance is commonly even stronger that the signal power. It is thus customary to make several repetitions of the same cognitive experiment, e.g., showing 50 times the same image to a subject in order to record 50 times the electric activity of the visual cortex. The multiple measurements are then classically averaged across the experiment's repetitions in order to increase the signal-to-noise ratio. In other words, popular estimators for M/EEG usually discard the individual observations, and rely on Gaussian i.i.d. noise models [START_REF] Ou | A distributed spatio-temporal EEG/MEG inverse solver[END_REF][START_REF] Gramfort | Timefrequency mixed-norm estimates: Sparse M/EEG imaging with non-stationary source activations[END_REF].

In this work we propose Concomitant Lasso with Repetitions (CLaR), an estimator that is

• Designed to exploit all available measurements collected during repetitions of experiments. • Defined as the solution of a convex minimization problem, handled efficiently by proximal block coordinate descent techniques. • Built thanks to an explicit connection with nuclear norm smoothing. This can also be viewed as a partial smoothing of the multivariate square-root Lasso (van de Geer and Stucky, 2016). • Shown (through extensive benchmarks with respect to existing estimators) to leverage experimental repetitions to improve support identification,

• Available as open source code to reproduce all the experiments.

In Section 5.2, we recall the framework of concomitant estimation, and introduce CLaR. In Section 5.2.2, we detail the properties of CLaR, and derive an algorithm to solve it. Finally, Section 5.3 is dedicated to experimental results. (l) . Let X ∈ R n×p be the design (or gain) matrix, with p features stored column-wise: X = [X :1 | . . . |X :p ], where for a matrix A ∈ R m×n its j th column (resp. row) is denoted A :j ∈ R m×1 (resp. A j: ∈ R 1×n . The matrix B * ∈ R p×T contains the coefficients of the linear regression model.

Concomitant estimation with correlated noise

Each measurement (i.e., repetition of the experiment) follows the model:

∀l ∈ [r], Y (l) = XB * + S * E (l) , (5.1)
where the entries of E (l) are i.i.d. samples from standard normal distributions, the E (l) 's are independent, and S * ∈ S n ++ is the co-standard deviation matrix, and S n ++ (resp. S n + ) stands for the set of positive (resp. semi-definite positive) matrices. Note that even if the observations Y (1) , . . . , Y (r) differ because of the noise E (1) , . . . , E (r) , B * and the noise structure S * are shared across repetitions. To leverage the multiple repetitions while taking into account the noise structure, we introduce the Concomitant Lasso with Repetitions (CLaR):

Definition 5.1. CLaR estimates the parameters of Model (5.1) by solving:

( BCLaR , ŜCLaR ) ∈ arg min

B∈R p×T S σ Idn f (B, S)+λ B 2,1 , with f (B, S) r l=1 Y (l) -XB 2 S -1 2nT r + Tr(S) 2n ,
(5.2) where λ > 0 controls the sparsity of BCLaR and σ > 0 controls the smallest eigenvalue of ŜCLaR .

In low SNR settings, a standard way to deal with strong noise is to use the averaged observation Ȳ ∈ R n×T instead of the raw observations. The associated model reads: (5.4)

Ȳ = XB * + S * Ẽ , ( 5
Remark 5.3. Note that ŜCLaR estimates S * , while ŜSGCL estimates S * = S * / √ r. Since we impose the constraint ŜCLaR σ Id n , we rescale the constraint so that ŜSGCL σ/ √ r Id n in (5.4) for future comparisons. Also note that CLaR and SGCL are the same when r = 1 and Y (1) = Ȳ .

The justification for CLaR is the following: if the quadratic loss Y -XB 2 were used, the parameters of Model (5.1) could be estimated by using either Ȳ -XB 2 or 1 r Y (l) -XB 2 as a data fitting term. Yet, both alternatives yield the same solutions as the two terms are equal up to constants. Hence, the quadratic loss does not leverage the multiple repetitions and ignores the noise structure. On the contrary, the more refined data fitting term of CLaR allows to take into account the individual repetitions, leading to improved performance in applications.

Properties of the proposed data fitting term

Let us analyze the data fitting term of CLaR, by connecting it to the Schatten 1-norm. Let us define the following smoothing function: (5.5) and the inf-convolution of functions f 1 and f 2 , f 1 f 2 (y) inf x f 1 (x) + f 2 (yx). The name "smoothing" used in this chapter comes from the following fact: if f 1 is a closed proper convex function, then f * 1 + 1 2 • 2 is strongly convex, and thus its Fenchel transform

ω σ (•) 1 2 • 2 + n σ ,
(f * 1 + 1 2 • 2 ) * = (f * 1 + ( 1 2 • 2 ) * ) * = (f 1 1 2 • 2 ) * * = f 1 1 2 • 2 is smooth.
The next propositions are key to our framework and show the connection between the SGCL, CLaR and the Schatten 1-norm: Proposition 5.4 (Smoothing of the trace norm). For the choice ω(•) = 1 2 • 2 + n 2 , and with n ≤ T , the ω σ -smoothing of the Schatten-1 norm, i.e., the function • S ,1 ω σ : R n×T → R, has the following closed-form formula, for all Z ∈ R n×T :

( • S ,1 ω σ )(Z) = min S σ Idn 1 2 Z 2 S -1 + 1
2 Tr(S) .

(5.6)

Proof The proof relies on a direct calculus. We will evaluate each member of Equation (5.6) on a matrix Z ∈ R n×T , through a singular value decomposition of Z (Lemma 5.6).

Proposition 5.5 (Usual properties of inf-convolution). Let g : R d → R, h : R d → R be closed proper convex functions. Then, the following holds (see Parikh et al. 2013, p. 136):

h * * = h ,
(5.7)

(h g) * = h * + g * , (5.8) σg • σ * = σg * ,
(5.9)

• * p = ι B p * , where
1 p + 1 p * = 1 , (5.10) (h + δ) * = h * -δ, ∀δ ∈ R , (5.11) 1 2 • 2 * = 1 2 • 2 .
(5.12)

We now compute each member of Equation (5.6) on a matrix Z ∈ R n×T , Lemma 5.6. Let Z ∈ R n×T and V diag(γ 1 , . . . , γ n , 0, . . . , 0)W its singular value decomposition, then for the choice ω(•) = 1 2 • 2 + n 2 , and with n ≤ T :

a) • S ,1 ω σ (Z) = 1 2σ γ i ≤σ γ 2 i + γ i >σ γ i + 1 2 γ i ≤σ σ . b) min S σ Idn 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) = 1 2σ γ i ≤σ γ 2 i + γ i >σ γ i + 1 2 γ i ≤σ σ .
We first compute the value of the right-hand member in Equation (5.6) CHAPTER 5. APPLICATION TO NEUROIMAGING Proof (Lemma 5.6 (a)).

• S ,1 ω σ (Z) = • S ,1 ω σ * * (Z) (using eq. (5.7))

= • * S ,1 + ω * σ *
(Z) (using eq. (5.8))

= ι B S ,∞ + σ 2 • 2 - n 2 σ *
(Z) (using eq. ( 5.10))

= σ 2 • 2 + ι B S ,∞ * (Z) + n 2
σ (using eq. ( 5.11))

= sup

U ∈R n×T U, Z - σ 2 U 2 -ι B S ,∞ (U ) + n 2 σ = sup U ∈B S ,∞ U, Z - σ 2 U 2 + n 2 σ = -inf U ∈B S ,∞ σ 2 U 2 -U, Z + n 2 σ = -σ • inf U ∈B S ,∞ 1 2 U 2 -U, Z σ + n 2 σ = -σ • inf U ∈B S ,∞ 1 2 U -Z σ 2 -1 2σ 2 Z 2 + n 2 σ = 1 2σ Z 2 -σ 2 • inf U ∈B S ,∞ U -Z σ 2 + n 2 σ = 1 2σ Z 2 -σ 2 Π B S ,∞ Z σ -Z σ 2 + n 2 σ (with Π B S ,∞ the projection on B S ,∞ ) = 1 2σ Z 2 + n 2 σ -σ 2 V diag γ 1 σ ∧ 1 -γ 1 σ , . . . , γn σ ∧ 1 -γn σ W 2 (using Π B S ,∞ Z σ = V diag γ 1 σ ∧ 1, . . . , γn σ ∧ 1 W , see Beck 2017, Example 7.31) = 1 2σ Z 2 + n 2 σ -σ 2   n i=1 γ i σ ∧ 1 -γ i σ 2   = 1 2σ Z 2 + n 2 σ -σ 2   1 σ 2 n i=1 γ i ∧ σ -γ i 2   = 1 2σ Z 2 + n 2 σ -σ 2    1 σ 2 γ i >σ γ i ∧ σ -γ i 2    = 1 2σ Z 2 + n 2 σ -σ 2    1 σ 2 γ i >σ σ -γ i 2    = 1 2σ Z 2 + n 2 σ -σ 2    1 σ 2 γ i >σ σ 2 + γ 2 i -2σγ i    = 1 2σ Z 2 + n 2 σ -σ 2    γ i >σ 1 + 1 σ 2 γ i >σ γ 2 i -2 1 σ γ i >σ γ i    = 1 2σ Z 2 + n 2 σ - σ 2 γ i >σ 1 - 1 2σ γ i >σ γ 2 i + γ i >σ γ i = 1 2σ γ i ≤σ γ 2 i + γ i >σ γ i + 1 2 γ i ≤σ σ .

CONCOMITANT ESTIMATION WITH CORRELATED NOISE 113

We now compute the value of the left-hand member in Equation (5.6) Massias et al. 2018a, Prop. 2). Thus we have:

Proof (Lemma 5.6 (b)). The minimum of min S σ Idn 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) is at- tained at Ŝ = V diag(γ 1 ∨ σ, . . . , γ n ∨ σ)V (see
min S σ Idn 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) = 1 2 Tr[Z Ŝ-1 Z] + 1 2 Tr( Ŝ) = 1 2 Tr[ Ŝ-1 ZZ ] + 1 2 Tr( Ŝ) = 1 2 Tr[V diag(γ 2 1 /(γ 1 ∨ σ), . . . , γ 2 n /(γ n ∨ σ))V ] + 1 2 Tr[V diag(γ 1 ∨ σ, . . . , γ n ∨ σ)V ] = 1 2 n i=1 γ 2 i γ i ∨ σ 1 2 γ i >σ γ i + 1 2σ γ i ≤σ γ 2 i + 1 2 n i=1 γ i ∨ σ 1 2 γ i >σ γ i + 1 2 γ i ≤σ σ = 1 2σ γ i ≤σ γ 2 i + γ i >σ γ i + 1 2 γ i ≤σ σ .
Proof (Proposition 5.4). Proposition 5.4 is a direct consequence of Lemmas 5.6 (a) and 5.6 (b).

Properties similar to Proposition 5.4 can be traced back to van de Geer and Stucky (2016, Sec 2.2), who introduced the multivariate square-root Lasso: (5.13) and showed that if ( Ȳ -X B)( Ȳ -X B) 0, the latter optimization problem admits a variational2 formulation:

B ∈ arg min B∈R p×T 1 n √ T Ȳ -XB S ,1 + λ B 2,1 ,
( B, Ŝ) ∈ arg min B∈R p×T , S 0 1 2nT Ȳ -XB 2 S -1 + Tr(S) 2n + λ B 2,1 .
(5.14)

In other words Proposition 5.4 generalizes van de Geer (2016, Lemma 3.4) for all matrices Ȳ -X B, getting rid of the condition ( Ȳ -X B)( Ȳ -X B) 0. In the present contribution, the problem formulation in Proposition 5.4 is motivated by computational aspects, as it helps to address the combined non-smoothness of the data fitting term • S ,1 and the penalty term • 2,1 . Note that another smoothing of the nuclear norm was proposed in [START_REF] Argyriou | Convex multi-task feature learning[END_REF]; Bach et al. (2012, Sec. 5.2):

Z → min S 0 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) + σ 2 2 Tr(S -1 ) , (5.15)
which is a σ-smooth nσ-approximation of • S ,1 , therefore less precise than ours (Lemma 5.8).

First let us recall the definition of a smoothable function and a µ-smooth approximation of Beck and Teboulle (2012, Def. 2.1):

Definition 5.7 (Smoothable function, µ-smooth approximation). Let g : E → -∞, +∞ be a closed and proper convex function, and let E ⊆ dom(g) be a closed convex set. The function g is called (α, δ, K)-smoothable on E if there exists δ 1 , δ 2 satisfying δ 1 + δ 2 = δ > 0 such that for every µ there exists a continuously differentiable convex function g µ : E → -∞, +∞ such that the following holds:

a) g(x) -δ 1 µ ≤ g µ (x) ≤ g(x) + δ 2 µ for every x ∈ E .
b) The function ∇g µ has a Lipschitz constant which is less than or equal to K + α µ :

∇g µ (x) -∇g µ (y) ≤ K + α µ
xy for every x, y ∈ E .

(5.16)

The function g is called a µ-smooth approximation of g with parameters (α, δ, K).

The nuclear norm • S ,1 is nonsmooth (and not even differentiable at 0), but one can construct a smooth approximation of the nuclear norm based on the following variational formula, if ZZ 0:

Z S ,1 = min S 0 1 2 Tr[Z S -1 Z] + 1 2
Tr(S) , (5.17) see van de Geer (2016, Lemma 3.4). When ZZ 0, one can approximate • S ,1 with (5.18) as shown in Lemma 5.8. It can be shown that this approximation stays close to the nuclear norm.

min S σ Id 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) = • S ,1 ω σ ,
Lemma 5.8. Let Z ∈ R n×T , (γ 1 , . . . , γ n ) be its singular values, and n ≤ T

a) • S ,1 ω σ is a σ-smooth approximation of • S ,1 with parameters (1, n 2 , 0). More precisely: • S ,1 ω σ has a σ-Lipschitz gradient and 0 ≤ • S ,1 ω σ -• S ,1 = σ 2 γ i <σ 1 - γ i σ 2 ≤ σ 2 n . (5.19) b) Z → min S 0 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) + σ 2 2 Tr(S -1 ) is a σ-smooth approximation of • S ,1 with parameters (1, n, 0). More precicely: Z → min S 0 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) + σ 2
2 Tr(S -1 ) has a gradient σ-Lipschitz and

0 ≤ min S 0 1 2 Tr[Z S -1 Z]+ 1 2 Tr(S)+ σ 2 2 Tr(S -1 )-Z S ,1 = σ i 1 1 + γ 2 i σ 2 + γ i σ ≤ σn .
(5.20)

CONCOMITANT ESTIMATION WITH CORRELATED NOISE

c) It can be shown that with a fixed Lipschitz constant, the proposed smoothing is (at least) uniformly a twice better approximation. This can be quantified even more precisely: 5.22) which means that for a fixed smoothing constant σ, our smoothing is at least twice uniformly better. Moreover the proposed smoothing can be much better, in particular when a lot a singular values are around σ.

0 ≤ • S ,1 ω σ (Z) -Z S ,1 Err 1 (Z) ≤ 1 2       min S 0 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) + σ 2 2 Tr(S -1 ) -Z S ,1 Err 2 (Z)       (5.21) More precisely 1 2 Err 2 (Z) -Err 1 (Z) = σ 2 γ i ≥σ    1 + γ 2 i σ 2 - γ i σ    ≥0 + σ 2 γ i <σ     1 1 + γ 2 i σ 2 + γ i σ -(1 + γ i σ ) 2     ≥0 , (
Proof (Lemma 5.8 (a)). Since ω is 1-smooth, Beck and Teboulle (2012, Thm. 4.1) shows that • S ,1 ω σ is σ-smooth.

Let Z ∈ R n×T and let γ 1 , . . . , γ n be its singular value decomposition:

• S ,1 ω σ (Z) -Z S ,1 = 1 2 n i=1 γ 2 i γ i ∨ σ + 1 2 n i=1 γ i ∨ σ + 1 2 n n+1 σ - n i=1 γ i = 1 2 n i=1 γ 2 i γ i ∨ σ + γ i ∨ σ -2γ i + 1 2 n n+1 σ = 1 2 γ i ≤σ γ 2 i γ i ∨ σ + γ i ∨ σ -2γ i + 1 2 n n+1 σ = 1 2 γ i ≤σ γ 2 i σ + σ -2γ i + 1 2 n n+1 σ = 1 2 γ i ≤σ (γ i -σ) 2 σ .
(5.23)

Hence,

0 ≤ • S ,1 ω σ (Z) -Z S ,1 = 1 2 γ i ≤σ (γ i -σ) 2 σ ≤ σ 2 n .
(5.24)

Moreover this bound is attained when Z = 0.

Proof (Lemma 5.8 (b)). Another regularization was proposed in [START_REF] Argyriou | Convex multi-task feature learning[END_REF]; Bach et al. (2012, p. 62):

min S 0 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) + σ 2 2 Tr(S -1 ) h(S -1 )
.

(5.25)

By putting the gradient of the objective function in Equation (5.25) to zero it follows that: (5.26) leading to :

0 = ∇h( Ŝ-1 ) = ZZ -Ŝ2 + σ 2 Id ,
Ŝ = (ZZ + σ 2 Id) 1 2 .
(5.27)

Let γ 1 , . . . , γ n be the singular values of Z: Beck and Teboulle (2012, Example 4.6).

n i=1 γ 2 i + σ 2 is a σ-smooth approximation of n i=1 γ 2 i = Z S ,1 , see
min S 0 1 2 Tr[Z S -1 Z] + 1 2 Tr(S) + σ 2 2 Tr(S -1 ) -Z S ,1 = 1 2 n i=1     γ 2 i γ 2 i + σ 2 + γ 2 i + σ 2 + σ 2 γ 2 i + σ 2     -Z S ,1 = 1 2 n i=1     γ 2 i + γ 2 i + σ 2 + σ 2 γ 2 i + σ 2     -Z S ,1 = n i=1 γ 2 i + σ 2 -Z S ,1 = n i=1 γ 2 i + σ 2 -γ i = n i=1 σ 2 γ 2 i + σ 2 + γ i = σ n i=1 1 1 + γ 2 i σ 2 + γ i σ
(5.28) ≤ σn .

(5.29)

Moreover this bound is attained when Z = 0.

Proof (Lemma 5.8 (c)). Using the formulas of Err 1 (Equation (5.23)) and Err 2 (Equation (5.28)), Equation (5.22) is direct. In Equation (5.22) the positivity of the first sum is trivial, the positivity of the second can be obtained with an easy function study.

Other alternatives to exploit the multiple repetitions without simply averaging them, would consist in investigating other Schatten p-norms: r) -XB] S ,p + λn B 2,1 .

arg min B∈R p×T 1 √ rT [Y (1) -XB| . . . |Y (
(5.30)

Without smoothing, problems of the form given in Equation (5.30) present the drawback of having two nonsmooth terms, and calling for primal-dual algorithms [START_REF] Chambolle | A first-order primal-dual algorithm for convex problems with applications to imaging[END_REF] with costly proximal operators. Even if the nonsmooth Schatten 1norm is replaced by the formula in Equation (5.6), numerical challenges remain: S can approach 0 arbitrarily, hence, the gradient with respect to S of the data fitting term is not Lipschitz over the optimization domain. Recently, Molstad (2019) proposed two algorithms to directly solve Equation (5.30): a prox-linear ADMM, and accelerated proximal gradient descent, the latter lacking convergence guarantees since the composite objective has two nonsmooth terms. Before that, van de Geer and Stucky (2016) devised a fixed point method, lacking descent guarantees. A similar problem was raised for the concomitant Lasso by [START_REF] Ndiaye | Efficient smoothed concomitant lasso estimation for high dimensional regression[END_REF] who used smoothing techniques to address it.

Finally one can explicitly link the data fitting term of the proposed estimator to the smoothed trace norm.

Proposition 5.9 (Schatten 1-norm (nuclear/trace norm) with repetitions). Let Z (1) , . . . , Z (r) be matrices in R n×T , we define r) ]. Then for the choice ω(•) = 1 2 • 2 + n∧T r 2 , then the following holds true:

Z ∈ R n×T r by Z = [Z (1) | . . . |Z (
• S ,1 ω σ (•) (Z) = min S σ Idn 1 2 r l=1
Tr Z (l) S -1 Z (l) + 1 2 Tr(S) .

(5.31)

Proof The result is a direct application of Proposition 5.4, with Z = [Z (1) | . . . |Z (r) ].

It suffices to notice that Tr Z S -1 Z = r l=1 Tr Z (l) S -1 Z (l) .

Properties of the proposed estimator and algorithmic details

We detail the principal results needed to solve Problem (5.2) numerically, leading to the implementation proposed in Algorithm 5.1. We first recall useful results for alternate minimization of convex composite problems.

Proposition 5.10 (Joint convexity). The function f defined in Problem (5.2) is jointly convex in (B, S). Moreover, f is convex and smooth on the feasible set, and • 2,1 is convex and separable in B j: 's, thus minimizing the objective alternatively in S and in B j: 's (Algorithm 5.1) converges to a global minimum.

Proof

f (B, S) = 1 2nT r r 1 Y (l) -XB 2 S -1 + 1 2n Tr(S) = Tr(Z T S -1 Z) + 1 2n Tr(S) , with Z = 1 √ 2nT r [Y (1) -XB| . . . |Y (r) -XB].
Proposition 5.13. For a fixed S ∈ S n ++ , each step of the block minimization of f (•, S)+ λ • 2,1 in the j th line of B admits a closed-form solution:

B j: = BST B j: + X :j S -1 ( Ȳ -XB) X :j 2 S -1 , λnT X :j 2 S -1
.

(5.35)

Proof The function to minimize is the sum of a smooth term f (•, S) and a nonsmooth but separable term, • 2,1 , whose proximal operator3 can be computed:

• f is X :j 2 S -1 /nT -smooth with respect to B j: , with partial gradient

∇ j f (•, S) = - 1 nT X :j S -1 ( Ȳ -XB) .
(5.36)

• B 2,1 = p j=1 B j: is row-wise separable over B, with

prox λnT / X :j 2 S -1 , • (•) = BST •, λnT X :j 2 S -1
.

(5.37)

Hence, proximal block-coordinate descent converges (Tseng and Yun, 2009a), and the update are given by Equation (5.35). The closed-form formula arises since the smooth part of the objective is quadratic and isotropic with respect to B j: .

As for other Lasso-type estimators, there exists λ max ≥ 0 such that whenever λ ≥ λ max , the estimated coefficients vanish. This λ max helps calibrating roughly λ in practice by choosing it as a fraction of λ max .

Proposition 5.14 (Critical regularization parameter). For the CLaR estimator we have: with

S max ClSqrt 1 T r r l=1 Y (l) Y (l) , σ , ∀λ ≥ λ max 1 nT X S -1 max Ȳ 2,∞ , BCLaR = 0 . (5.38) Proof Fermat's rules states B = 0 ⇔ 0 ∈ ∂ f (•, S max ) + λ • 2,1 (0) ⇔ -∇f (•, S max ) ∈ λB • 2,∞ ⇔ 1 nT X S -1 max Ȳ 2,∞ λ max ≤ λ .
(5.39)

Convex formulation benefits. Thanks to the convex formulation, convergence of Algorithm 5.1 can be ensured using the duality gap as a stopping criterion (as it guarantees a targeted sub-optimality level). In addition, convexity allows to leverage acceleration methods such as working sets strategies [START_REF] Fan | Sure independence screening for ultrahigh dimensional feature space[END_REF][START_REF] Tibshirani | Strong rules for discarding predictors in lasso-type problems[END_REF] CHAPTER 5. APPLICATION TO NEUROIMAGING [START_REF] Johnson | Blitz: A principled meta-algorithm for scaling sparse optimization[END_REF]Massias et al., 2018b) or safe screening rules [START_REF] Ghaoui | Safe feature elimination in sparse supervised learning[END_REF]Fercoq et al., 2015) while retaining theoretical convergence guarantees. Such techniques are trickier to adapt in the non-convex case, as they could change the local minima reached.

Choice of σ. Although σ has a smoothing interpretation, from a practical point of view it remains an hyperparameter to set. Following guidelines from Chapter 4, σ is always chosen as follows: σ = Y /(1000 × nT ). In practice, the experimental results were little affected by the choice of σ.

Remark 5.15. l) is pre-computed, the cost of updating S does not depend on r, i.e., is the same as working with averaged data. Indeed, with R = [Y (1) -XB| . . . |Y (r) -XB], the following computation can be done in O(qn 2 ).

Once cov Y 1 r r 1 Y (l) Y (
RR = RRT(cov Y , Y, X, B) rcov Y + r × (XB)(XB) -r Ȳ (XB) -r(XB) Ȳ .
(5.40)

Proof RR = r l=1 R (l) R (l) = r l=1 (Y (l) -XB)(Y (l) -XB) = r l=1 Y (l) Y (l) - r 1 Y (l) (XB) - r 1 XBY (l) + rXB(XB) = rcov Y -r Ȳ XB -r(XB) Ȳ + rXB(XB) .
(5.41)

Experiments

Our Python code (with Numba compilation, [START_REF] Lam | Numba: A LLVM-based Python JIT Compiler[END_REF] is released as an open source package: https://github.com/QB3/CLaR. The following estimators are compared:

• CLaR defined with Problem (5.2) (ours).

• SGCL defined with Problem (5.4) (Massias et al., 2018a).

• 2,1 -MLE, an 2,1 version of MLE [START_REF] Lee | Simultaneous multiple response regression and inverse covariance matrix estimation via penalized Gaussian maximum likelihood[END_REF][START_REF] Chen | Alternating estimation for structured high-dimensional multiresponse models[END_REF]. When minimizing 2,1 -Maximum Likelihood the natural parameters of the problem are the regression coefficients B and the precision matrix Σ -1 . Since real M/EEG covariance matrices are not full rank, one has to be algorithmically careful when Σ becomes singular. To avoid such numerical errors and to be consistent with the smoothed estimator proposed in the chapter (CLaR), let us define the (smoothed)

2,1 -MLE as following:

( B 2,1 -MLE , Σ 2,1 -MLE ) ∈ arg min B∈R p×T Σ σ 2 /r 2 Ȳ -XB 2 Σ -1 -log det(Σ -1 )+λ B 2,1 , (5.42)
• 2,1 -MLER, a version of the 2,1 -MLE with multiple repetitions:

( B 2,1 MLER , Σ 2,1 MLER ) ∈ arg min B∈R p×T Σ σ 2 r 1 Y (l) -XB 2 Σ -1 -log det(Σ -1 ) + λ B 2,1 .
(5.43) Problems (5.42) and (5.43) are not convex because the objective functions are not convex in (B, Σ -1 ), however they are biconvex, i.e., convex in B and convex in Σ -1 . Alternate minimization can be used to solve Problems (5.42) and (5.43), but without guarantees to converge toward a global minimum.

• MRCER, an 2,1 penalized version of MRCE [START_REF] Rothman | Sparse multivariate regression with covariance estimation[END_REF] with repetitions. MRCE jointly estimates the regression coefficients (assumed to be sparse) and the precision matrix (i.e., the inverse of the covariance matrix), which is supposed to be sparse as well. 2,1 -MRCE is defined as the solution of the following optimization problem:

( BMRCE , ΣMRCE ) ∈ arg min B∈R p×T Σ -1 0 Ȳ -XB 2 Σ -1 -log det(Σ -1 ) + λ B 2,1 + µ Σ -1 1 .
(5.44) Problem (5.44) is not convex, but can be solved heuristically (see [START_REF] Rothman | Sparse multivariate regression with covariance estimation[END_REF] for details) by coordinate descent for the updates in B j: 's and solving a Graphical Lasso [START_REF] Friedman | Sparse inverse covariance estimation with the graphical lasso[END_REF] for the update in Σ -1 .

• MTL, the Multi-Task Lasso [START_REF] Obozinski | Joint covariate selection and joint subspace selection for multiple classification problems[END_REF]. It is the usual estimator used when the additive noise is supposed to be homoscedastic (with no correlation). MTL is obtained by solving BMTL ∈ arg min

B∈R p×T 1 2nT Ȳ -XB 2 + λ B 2,1 .
(5.45)

Each estimator, proposed or compared to is based on an optimization problem to solve. Each optimization problem is solve with block coordinate descent, whether there is theoretical guarantees for it to converge toward a global minimum (for convex formulations, CLaR, SGCL and MTL), or not (for non-convex formulations, 2,1 -MLE, 2,1 -MLER, MRCER). The cost for the updates for each algorithm can be found in Table 5.1. The formula for the updates in B j: 's and S/Σ for each algorithm can be found in Table 5.2. Let f dual be the number of updates of B for one update of S or Σ.

Table 5.1 -Algorithms cost in time summary.

CD epoch cost convex dual gap cost

CLaR one. However, when the SNR decreases (middle), the performance of SGCL and MTL starts to drop, while that of CLaR, 2,1 -MLER and MRCER remains stable (CLaR performing better), highlighting their capacity to leverage multiple repetitions of measurements to handle the noise structure. Finally, when the SNR is too low (right), all algorithms perform poorly, but CLaR, 2,1 -MLER and MRCER still performs better.

O( n 3 +qn 2 f dual + pn 2 + pnq) yes O(rnT + p) SGCL O( n 3 +qn 2 f dual + pn 2 + pnq) yes O(nT + p) 2,1 -MLER O( n 3 +qn 2 f dual + pn 2 + pnq) no not convex 2,1 -MLE O( n 3 +qn 2 f dual + pn 2 + pnq) no not convex MRCER O( O(glasso) f dual + pn 2 + pnq) no not convex MTL O(npT ) yes O(nT + p) CLaR SGCL 2,1-MLER 2,1-MLE 2,
Influence of the number of repetitions. Figure 5.3 shows ROC curves of all compared approaches for different r, starting from r = 1 (left) to 100 (right). Even with r = 20 (middle) CLaR outperforms the other estimators, and when r = 100 CLaR can better leverage the large number of repetitions.

Realistic data

We now evaluate the estimators on realistic magneto-and electroencephalography (M/EEG) data. The M/EEG recordings measure the electrical potential and magnetic fields induced by the active neurons. Data are time series of length T with n sensors and p sources mapping to locations in the brain. Because the propagation of the electromagnetic fields is driven by the linear Maxwell equations, one can assume that the relation between the measurements Y (1) , . . . , Y (r) and the amplitudes of sources in the brain B * is linear.

The M/EEG inverse problem consists in identifying B * . Because of the limited number of sensors (a few hundreds in practice), as well as the physics of the problem, the M/EEG inverse problem is severely ill-posed and needs to be regularized. Moreover, the experiments being usually short (less than 1 s.) and focused on specific cognitive functions, the number of active sources is expected to be small, i.e., B * is assumed to be row-sparse. This plausible biological assumption motivates the framework of Section 5.2 ( [START_REF] Ou | A distributed spatio-temporal EEG/MEG inverse solver[END_REF].

CHAPTER 5. APPLICATION TO NEUROIMAGING CLaR SGCL 2,1-MLER 2,1-MLE 2,1-MRCER MTL 0.0 0.1 0.2
Dataset. We use the sample dataset 4 from the MNE software [START_REF] Gramfort | MNE software for processing MEG and EEG data[END_REF].

The experimental conditions here are auditory stimulations in the right or left ear, leading to two main foci of activations in bilateral auditory cortices (i.e., 2 non-zeros rows for B * ). For this experiment, we keep only the gradiometer magnetic channels.

After removing one channel corrupted by artifacts, this leads to n = 203 signals. The length of the temporal series is T = 100, and the data contains r = 50 repetitions. We choose a source space of size p = 1281 which corresponds to about 1 cm distance between neighboring sources. The orientation is fixed, and normal to the cortical mantle.

Realistic MEG data simulations. We use here true empirical values for X and S by solving Maxwell equations and taking an empirical co-standard deviation matrix. To generate realistic MEG data we simulate neural responses B * with 2 non-zeros rows corresponding to areas known to be related to auditory processing (Brodmann area 22). Each non-zero row of B * is chosen as a sinusoidal signal with realistic frequency (5 Hz) and amplitude (amp ∼ 1 -10 nAm). We finally simulate r MEG signals l) , E (l) being matrices with i.i.d. normal entries.

Y (l) = XB * + S * E (
Preprocessing steps for realistic and real data. When using multi-modal data without whitening, one has to rescale properly data, indeed data needs to have the same order of magnitude, otherwise some mode (for example EEG data) could be (almost) completely ignored by the optimization algorithm. The preprocessing pipeline used to rescale realistic data (Figures 5.4 and 5.5) and real data (Figures 5.6,5.8,5.10 and 5.12) is described in Algorithm 5.2. // rescale each line of X

for i = 1, . . . , n do for l = 1, . . . , r do Y (l) i: ← Y (l) i: / X i: X i: ← X i: / X i:
// rescale each column of X for j = 1, . . . , p do X :j ← X :j / X :j return X, Y (1) , . . . , Y (r) The signals being contaminated with correlated noise, if one wants to use homoscedastic solvers it is necessary to whiten the data first (and thus to have an estimation of the covariance matrix, the later often being unknown). In this experiment we demonstrate that without this whitening process, the homoscedastic solver MTL fails, as well as solvers which does not take in account the repetitions: SGCL and 2,1 -MLE. In this scenario CLaR, 2,1 -MLER and MRCER do succeed in recovering the sources, CLaR leading to the best results. As for the synthetic data, Figures 5.4 and 5.5 are obtained by varying the estimator-specific regularization parameter λ from λ max to λ min on a geometric grid. Amplitude influence. Figure 5.5 shows ROC curves for different values of the amplitude of the signal. When the amplitude is high (right), all the algorithms perform well, however when the amplitude decreases (middle) only CLaR leads to good results, almost hitting the (0, 1) corner. When the amplitude gets lower (left) all algorithms perform worse, CLaR still yielding the best results.

Influence of the number of repetitions. Figure 5.4 shows ROC curves for different number of repetitions r. When the number of repetitions is high (right, r = 50), the algorithms taking into account all the repetitions (CLaR, 2,1 -MLER, MRCER) perform best, almost hitting the (0, 1) corner, whereas the algorithms which do not take into account all the repetitions ( 2,1 -MLE, MTL, SGCL) perform poorly. As soon as the number of repetitions decreases (middle and left) the performance of all the algorithms except CLaR starts dropping severely. CLaR is once again the algorithm taking the most advantage of the number of repetitions.

Real M/EEG data

As before, we use the sample dataset, keeping only the magnetometer magnetic channels (n = 102 signals). We choose a source space of size p = 7498 (about 5 mm between neighboring sources). The orientation is fixed, and normal to the cortical mantle. As for realistic data, X is the empirical design matrix, but this time we use the empirical measurements Y (1) , . . . , Y (r) . As two sources are expected (one in each hemisphere, in bilateral auditory cortices), we vary λ by dichotomy between λ max (returning 0 sources) and a λ min (returning more than 2 sources), until finding a λ giving exactly 2 sources. Results are provided in Figures 5.6 and 5.7. Running times of each algorithm are of the same order of magnitude and can be found in Figure 5.14.

Audiroy stimulations. Comments on Figure 5.6, left auditory stimulations. Sources found by the algorithms are represented by red spheres. SGCL, 2,1 -MLE and MRCER completely fail, finding sources that are not in the auditory cortices at all (SGCL sources are deep, thus not in the auditory cortices, and cannot be seen). MTL and 2,1 -MLER do find sources in auditory cortices, but only in one hemisphere (left for MTL and right for 2,1 -MLER). CLaR is the only one that finds one source in each hemisphere in the auditory cortices as expected.

Comments on Figure 5.7, right auditory stimulations. In this experiment we only keep r = 33 repetitions (out of 65 available) and it can be seen that only CLaR finds correct sources, MTL finds sources only in one hemisphere and all the other algorithms do find sources that are not in the auditory cortices. This highlights the robustness of CLaR, even with a limited number of repetitions, confirming previous experiments (see Figure 5.3).

Figures 5.8 and 5.9 show the solution given by each algorithm on real data after right auditory stimulations. As two sources are expected (one in each hemisphere, in bilateral auditory cortices), we vary λ by dichotomy between λ max (returning 0 sources) and a λ min (returning more than 2 sources), until finding a lambda giving exactly 2 sources. Figure 5.8 (resp. Figure 5.9) shows the solution given by the algorithms taking in account all the repetitions (resp. only half of the repetitions). When the number of repetitions is high (Figure 5.8) only CLaR and 2,1 -MLER find one source in each auditory cortex, MTL does find sources only in one hemisphere, all the other algorithms fail by finding sources not in the auditory cortices at all. Moreover when the number of repetitions is decreasing (Figure 5.9) 2,1 -MLER fails and only CLaR does find 2 sources, one in each hemisphere. Once again CLaR is more robust and performs better, even when the number of repetitions is low.

Visual stimulations. Figures 5.10 and 5.11 show the results for each algorithm after left visual stimulations. As one source is expected (in the right hemisphere), we vary λ by dichotomy between λ max (returning 0 sources) and a λ min (returning more than 1 sources), until finding a lambda giving exactly 1 source. When the number of repetitions is high (Figure 5.10) only CLaR and 2,1 -MLER do find a source in the visual cortex.

When the number of repetitions decreases, CLaR and 2,1 -MLER still find one source in the visual cortex, other algorithms fail. This highlights this importance of taking into account the repetitions. As one source is expected (in the left hemisphere), we vary λ by dichotomy between λ max (returning 0 sources) and a λ min (returning more than 1 sources), until finding a lambda giving exactly 1 source. When the number of repetitions is high (Figure 5.12) only CLaR, 2,1 -MLER and MTL do find a source in the visual cortex. When the number of repetitions decreases (Figure 5.13), only CLaR finds one source in the visual cortex, other algorithms fail. This highlights once again the robustness of CLaR, even with a limited number of repetitions.

Time comparison. The goal of this experiment is to show that our algorithm (CLaR) is as costly as a Multi-Task Lasso or other competitors (in the M/EEG context, i.e., n not too large). The time taken by each algorithm to produce Figure 5.6 (real data, left auditory stimulations) is given in Figure 5.14. In this experiment the tolerance is set to tol=10 -3 , the safe stopping criterion is duality gap < tol (only avail- able for convex optimization problems). The heuristic stopping criterion is "if the objective do not decrease enough anymore then stop" i.e., if objective(B (t) , Σ (t) )objective(B (t+1) , Σ (t+1) ) < tol/10 then stop. The safe stopping criterion is only available for CLaR, SGCL and MTL (it takes too much time i.e., more than 10min for SGCL to have a duality gap under the fixed tol, so we remove it).

CLaR SGCL 2, 1 -MLER 2, 1 -MLE 2, 1 -MRCER MTL Algorithms
Comment on Figure 5.14 Figure 5.14 shows that if we use the heuristic stopping criterion, CLaR is as fast the other algorithm. In addition CLaR has a safe stopping criterion which only take 2 to 3 more time than the heuristic one (less than 10sec).

Conclusion. This work introduces CLaR, a sparse estimator for multitask regression. It is designed to handle correlated Gaussian noise in the context of repeated observations, a standard framework in applied sciences such as neuroimaging. The resulting optimization problem can be solved efficiently with state-of-the-art convex solvers, and the algorithmic cost is the same as for single repetition data. The theory of smoothing connects CLaR to the Schatten 1-Lasso in a principled manner, which opens the way to the use of more sophisticated data fitting terms. The benefits of CLaR for support recovery in the presence of non-white Gaussian noise were extensively evaluated against a large number of competitors, both on simulations and on empirical MEG data. In Chapters 4 and 5 we investigated pivotal estimators, for which the regularization parameter admits a closed-form formula. However, as illustrated in Chapter 1, this route relies on strong hypotheses usually unrealistic on real data (Figure 1.5), and users still have to resort to model calibration (see Section 5.3). In this chapter we investigate another route: hyperparameter optimization. Indeed, finding the optimal hyperparameters of a model can be cast as a bilevel optimization problem, typically solved using zero-order techniques. In this work we study first-order methods when the inner optimization problem is convex but nonsmooth. Capitalizing on model identification (Theorem 2.13) and local linear convergence (Theorem 2.16) results from Chapter 2, we show that the forward-mode differentiation of proximal gradient descent and proximal coordinate descent yield sequences of Jacobians converging toward the exact Jacobian. Using implicit differentiation, we show it is possible to leverage the non-smoothness of the inner problem to speed up the computation. Finally, we provide a bound on the error made on the hypergradient when the inner optimization problem is solved approximately. Results on regression and classification problems reveal computational benefits for hyperparameter optimization, especially when multiple hyperparameters are required. This chapter is based on the following works, one accepted to ICML 2020, the other currently under review for the Journal of Machine Learning Research: Klopfenstein, M. Blondel, S. Vaiter, A. Gramfort, and J. Salmon. Implicit differentiation of Lasso-type models for hyperparameter optimization. ICML, 2020

• Q. Bertrand, Q.
• Q. Bertrand, Q. Klopfenstein, M. Massias, M. Blondel, S. Vaiter, A. Gramfort, and J. Salmon. Implicit differentiation for fast hyperparameter selection in nonsmooth convex learning. arXiv preprint arXiv:2105.01637, 2021

Introduction

Almost all models in machine learning require at least one hyperparameter, the tuning of which drastically affects accuracy. This is the case for many popular estimators, where the regularization hyperparameter controls the trade-off between a data fidelity term and a regularization term. Such estimators, including Ridge regression [START_REF] Hoerl | Ridge regression: Biased estimation for nonorthogonal problems[END_REF], Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF][START_REF] Chen | Atomic decomposition by basis pursuit[END_REF], elastic net [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF], sparse logistic regression [START_REF] Koh | An interior-point method for large-scale l1-regularized logistic regression[END_REF], support-vector machine/SVM [START_REF] Boser | A training algorithm for optimal margin classifiers[END_REF][START_REF] Platt | Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods[END_REF] are often cast as an optimization problem (Table 6.1) Table 6.1 -Examples of nonsmooth inner problems as in (6.1).

Inner problem, Φ f (β)

g j (β j , λ) e λmax Lasso 1 2n y -Xβ 2 e λ |β j | 1 n X y ∞ elastic net 1 2n y -Xβ 2 e λ 1 |β j | + 1 2 e λ 2 β 2 j 1 n X y ∞ sparse log. reg. 1 n n i=1 ln(1 + e -y i X i: β ) e λ |β j | 1 2n X y ∞ dual SVM 1 2 (y X) β 2 -p j=1 β j ι [0,e λ ] (β j ) - β(λ) ∈ arg min β∈R p Φ(β, λ) f (β) + p j=1 g j (β j , λ) g(β,λ) , (6.1)
with smooth f : R p → R (i.e., with Lipschitz gradient), proper closed convex (possibly nonsmooth) functions g j (•, λ), and a regularization hyperparameter λ ∈ R r . In the examples of Table 6.1, the computation of f involves a design matrix X ∈ R n×p ; and the cost of computing ∇f (β) is O(np). In the SVM example, since we consider the dual problem, we chose to reverse the roles of n and p to enforce β ∈ R p . We often drop the λ dependency and write β instead of β(λ) when it is clear from context.

For a fixed λ, the issue of solving efficiently Problem (6.1) has been largely explored.

If the functions g j are smooth, one can use solvers such as L-BFGS [START_REF] Liu | On the limited memory BFGS method for large scale optimization[END_REF], SVRG [START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF][START_REF] Zhang | Linear convergence with condition number independent access of full gradients[END_REF], or SAGA [START_REF] Defazio | Saga: A fast incremental gradient method with support for non-strongly convex composite objectives[END_REF]. When the functions g j are nonsmooth, Problem (6.1) can be tackled efficiently with stochastic algorithms [START_REF] Pedregosa | Breaking the nonsmooth barrier: A scalable parallel method for composite optimization[END_REF] or using working set methods [START_REF] Fan | Sure independence screening for ultrahigh dimensional feature space[END_REF][START_REF] Tibshirani | Strong rules for discarding predictors in lasso-type problems[END_REF] combined with coordinate descent (Tseng and Yun, 2009a), see overview by Massias et al. (2020b). The question of model selection, i.e., how to select the hyperparameter λ ∈ R r (potentially multidimensional), is more open, especially when the dimension r of the regularization hyperparameter λ is large. For the Lasso, a broad literature has been devoted to parameter tuning. Under strong hypothesis on the design matrix X, it is possible to derive guidelines for the setting of the regularization parameter λ [START_REF] Lounici | Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators[END_REF][START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF][START_REF] Belloni | Square-root Lasso: pivotal recovery of sparse signals via conic programming[END_REF].

Unfortunately, these guidelines rely on quantities which are typically unknown in practice, and Lasso users still have to resort to other techniques to select the hyperparameter λ.

A popular approach for hyperparameter selection is hyperparameter optimization [START_REF] Kohavi | Automatic parameter selection by minimizing estimated error[END_REF][START_REF] Hutter | Beyond manual tuning of hyperparameters[END_REF]Feurer and Hutter, 2019): one selects the hyperparameter λ such that the regression coefficients β(λ) minimize a given criterion C : R p → R. Here C should ensure good generalization, or avoid overcomplex models. Common examples (see Table 6.2) include the hold-out loss [START_REF] Devroye | Distribution-free performance bounds for potential function rules[END_REF], the cross-validation loss (CV, Stone and Ramer 1965, see Arlot and Celisse 2010 for a survey), the AIC [START_REF] Akaike | A new look at the statistical model identification[END_REF], BIC (Schwarz, 1978) or SURE [START_REF] Stein | Estimation of the mean of a multivariate normal distribution[END_REF] criteria. Formally, the hyperparameter optimization problem is a bilevel optimization problem [START_REF] Colson | An overview of bilevel optimization[END_REF]: (6.2)

arg
Popular approaches to solve (the generally non-convex) Problem (6.2) include zeroorder optimization (gradient-free) techniques such as grid-search, random-search [START_REF] Rastrigin | The convergence of the random search method in the extremal control of a many parameter system[END_REF][START_REF] Bergstra | Random search for hyper-parameter optimization[END_REF][START_REF] Bergstra | Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms[END_REF] or Sequential Model-Based Global Optimization (SMBO), often referred to as Bayesian optimization [START_REF] Mockus | The bayesian approach to local optimization[END_REF][START_REF] Jones | Efficient global optimization of expensive black-box functions[END_REF][START_REF] Forrester | Engineering design via surrogate modelling: a practical guide[END_REF][START_REF] Brochu | A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning[END_REF][START_REF] Snoek | Practical bayesian optimization of machine learning algorithms[END_REF]. Gridsearch is a naive discretization of Problem (6.2). It consists in evaluating the outer function L on a grid of hyperparameters, solving one inner optimization Problem (6.1) for each λ in the grid (see Figure 6.1). For each inner problem solution β(λ) , the criterion C( β(λ) ) is evaluated, and the model achieving the lowest value is selected.

Random-search has a similar flavor, but one randomly selects where the criterion must be evaluated. Finally, SMBO models the objective function L via a function amenable to uncertainty estimates on its predictions such as a Gaussian process. Hyperparameter values are chosen iteratively to maximize a function such as the expected improvement as described, e.g., by [START_REF] Bergstra | Algorithms for hyper-parameter optimization[END_REF]. However, these zero-order methods share a common drawback: they scale exponentially with the dimension of the search space (Nesterov, 2004, Sec. 1.1.2). When the hyperparameter space is continuous and the regularization path λ → β(λ) is well-defined and weakly differentiable, first-order optimization methods are well suited to solve the bilevel optimization Problem (6.2). Using the chain rule, the gradient of L with respect to λ, also referred to as the hypergradient, evaluates to

∇ λ L(λ) = Ĵ (λ) ∇C( β(λ) ) , (6.3)
with Ĵ (λ) ∈ R p×r the Jacobian of the function λ → β(λ) ,

Ĵ (λ)       ∂ β(λ) 1 ∂λ 1 . . . ∂ β(λ) 1 ∂λr . . . . . . . . . ∂ β(λ) p ∂λ 1 . . . ∂ β(λ) p ∂λr       . (6.4)
An important challenge of applying first-order methods to solve Problem (6.2) is evaluating the hypergradient in Equation (6.3). There are three main algorithms to compute the hypergradient ∇ λ L(λ): implicit differentiation [START_REF] Larsen | Design and regularization of neural networks: the optimal use of a validation set[END_REF][START_REF] Bengio | Gradient-based optimization of hyperparameters[END_REF] and automatic differentiation using the reverse-mode [START_REF] Linnainmaa | The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors[END_REF][START_REF] Lecun | Efficient backprop[END_REF] or the forward-mode [START_REF] Wengert | A simple automatic derivative evaluation program[END_REF][START_REF] Deledalle | Stein Unbiased GrAdient estimator of the Risk (SUGAR) for multiple parameter selection[END_REF][START_REF] Franceschi | Forward and reverse gradientbased hyperparameter optimization[END_REF]. As illustrated in Figure 6.1, once the hypergradient in Equation (6.3) has been computed, one can solve Problem (6.2) with first-order schemes, e.g., gradient descent.

Contributions. We are interested in tackling the bilevel optimization Problem (6.2), with a nonsmooth inner optimization Problem (6.1). More precisely,

• We show that classical algorithms used to compute hypergradients for smooth inner problem have theoretically grounded nonsmooth counterparts. We provide in Theorem 6.9 an implicit differentiation formula for nonsmooth optimization problems. We obtain in Theorem 6.13, for the first time in the nonsmooth case, error bounds with respect to the hypergradient when the inner problem and the linear system involved are only solved approximately. We obtain in Theorem 6.12 convergence rates on the hypergradient for iterative differentiation of nonsmooth optimization problems.

• Based on the former contributions we propose an algorithm to tackle Problem (6.2).

We develop an efficient implicit differentiation algorithm to compute the hypergradient in Equation ( 6.3), leveraging the sparsity of the Jacobian and enabling the use of state-of-the-art solvers (Algorithm 6.5). We combine in Algorithm 6.6 this fast hypergradient computation with a gradient descent scheme to solve Problem (6.2).

• We provide extensive experiments on diverse datasets and estimators (Section 6.4). We first show that implicit differentiation significantly outperforms other hypergradient methods (Section 6.4.1). Then, leveraging sparsity, we illustrate computational benefits of first-order optimization with respect to zero-order techniques for solving Problem (6.2) on Lasso, elastic net and multiclass logistic regression (Section 6.4.2).

• We release our implementation as a high-quality, documented and tested Python package: https://github.com/qb3/sparse-ho.

Notation. The regularization parameter, possibly multivariate, is denoted by λ = (λ 1 , . . . , λ r ) ∈ R r . We denote Ĵ (λ) (∇ λ β(λ) 1 , . . . , ∇ λ β(λ) p ) ∈ R p×r the weak Jacobian [START_REF] Evans | Measure theory and fine properties of functions[END_REF] of β(λ) with respect to λ. For a function f , its gradient restricted to the indices in a set S is denoted ∇ S f . For a function ψ : R p × R r → R p , we denote ∂ z ψ the weak Jacobian with respect to the first variable and ∂ λ ψ the weak Jacobian with respect to the second variable. The proximal operator of g(•, λ) can be seen as such a function ψ of β and λ (see Table 6.1 for examples):

R p × R r → R p (z, λ) → prox g(•,λ) (z) = ψ(z, λ) .
In this case we denote ∂ z prox g(•,λ) ∂ z ψ and ∂ λ prox g(•,λ) ∂ λ ψ. Since we consider only separable penalties g(•, λ), ∂ z prox g(•,λ) is a diagonal matrix, so to make notation lighter, we write ∂ z prox g(•,λ) for its diagonal. We thus have

∂ z prox g(•,λ) = (∂ z prox g j (•,λ) ) j∈[p] ∈ R p (by separability of g) ∂ λ prox g(•,λ) ∈ R p×r .
Explicit partial derivatives formulas for usual proximal operators can be found in Table 6.3.

Bilevel optimization with smooth inner problems

The main challenge to evaluate the hypergradient ∇ λ L(λ) is the computation of the Jacobian J (λ) . We first focus on the case where Φ(•, λ) is convex and smooth for any λ.

Table 6.3 -Partial derivatives of proximal operators used. -e λ 1 sign(ST(z j ,e λ 1 )) 1+e λ 2 , -ST(z j ,e λ 1 )e λ 2 (1+e λ 2 ) 2 ι [0,e λ ] (β j ) max(0, min(z j , e λ )) 1 ]0,e λ [ (z j ) e λ 1 z j >e λ Implicit differentiation. We recall how the implicit differentiation3 formula of the gradient ∇ λ L(λ) is obtained for smooth inner optimization problems. We will provide a generalization to nonsmooth optimization problems in Section 6.3.2.

g j (β j , λ) prox g j (•,λ) (z j ) ∂ z prox g j (•,λ) (z j ) ∂ λ prox g j (•,λ) (z j ) e λ β 2 j /2 z j /(1 + e λ ) 1/(1 + e λ ) -z j e λ /(1 + e λ ) 2
Theorem 6.1 [START_REF] Bengio | Gradient-based optimization of hyperparameters[END_REF]. Let β(λ) ∈ arg min β∈R p Φ(β, λ) be a solution of Problem (6.1). Assume that for all λ > 0, Φ(•, λ) is a convex smooth function, ∇ 2 β Φ( β(λ) , λ) 0, and that for all β ∈ R p , Φ(β, •) is differentiable over ]0, +∞[. Then the hypergradient ∇ λ L(λ) reads:

∇ λ L(λ) ∈R r = -∇ 2 β,λ Φ( β(λ) , λ) ∈R r×p ∇ 2 β Φ( β(λ) , λ) ∈R p×p -1 ∇C( β(λ) ) ∈R p . (6.5)
Proof For a smooth convex function β → Φ(β, λ) the first-order condition writes: (6.6) for any β(λ) solution of the inner problem. Moreover, if λ → ∇ β Φ( β(λ) , λ) is differentiable, differentiating Equation (6.6) with respect to λ leads to:

∇ β Φ( β(λ) , λ) = 0 ,
∇ 2 β,λ Φ( β(λ) , λ) + Ĵ (λ) ∇ 2 β Φ( β(λ) , λ) = 0 . (6.7)
The Jacobian Ĵ (λ) is computed by solving the following linear system:

Ĵ (λ) = -∇ 2 β,λ Φ( β(λ) , λ) ∈R r×p ∇ 2 β Φ( β(λ) , λ) ∈R p×p -1 . (6.8)
Plugging Equation (6.8) into Equation ( 6.3) yields the desired result.

The computation of the gradient via implicit differentiation (Equation (6.5)) involves the resolution of a p×p linear system (Bengio, 2000, Sec. 4). This potentially large linear system can be solved using different algorithms such as conjugate gradient [START_REF] Hestenes | Methods of conjugate gradients for solving linear systems[END_REF]Stiefel 1952, as in Pedregosa 2016) or fixed point methods [START_REF] Lions | Splitting algorithms for the sum of two nonlinear operators[END_REF][START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF]Yun 2009a, as in Grazzi et al. 2020). Implicit differentiation has been used for model selection of multiple estimators with smooth regularization term: kernelbased models [START_REF] Chapelle | Choosing multiple parameters for support vector machines[END_REF][START_REF] Seeger | Cross-validation optimization for large scale structured classification kernel methods[END_REF], weighted Ridge estimator [START_REF] Foo | Efficient multiple hyperparameter learning for log-linear models[END_REF], neural networks [START_REF] Lorraine | Optimizing millions of hyperparameters by implicit differentiation[END_REF] or meta-learning [START_REF] Franceschi | Bilevel programming for hyperparameter optimization and meta-learning[END_REF][START_REF] Rajeswaran | Meta-learning with implicit gradients[END_REF]. In addition to hyperparameter selection, it has been applied successfully in natural language processing [START_REF] Bai | Deep equilibrium models[END_REF] and computer vision [START_REF] Bai | Multiscale deep equilibrium models[END_REF].

Problem (6.1) is typically solved using iterative solvers. In practice, the number of iterations is limited to reduce computation time, and also since very precise solutions are generally not necessary for machine learning tasks. Thus, Equation (6.6) is not exactly satisfied at machine precision, and consequently the linear system to solve Equation (6.5) does not lead to the exact gradient ∇ λ L(λ), see [START_REF] Ablin | Super-efficiency of automatic differentiation for functions defined as a minimum[END_REF] for quantitative convergence results. However, [START_REF] Pedregosa | Hyperparameter optimization with approximate gradient[END_REF] showed that one can resort to approximate gradients when the inner problem is smooth, justifying that implicit differentiation can be applied using an approximation of β. Interestingly, this approximation scheme was shown to yield significant practical speedups when solving Problem (6.2), while preserving theoretical properties of convergence toward the optimum.

Iterative differentiation. Iterative differentiation computes the gradient ∇ λ L(λ) by differentiating through the iterates of the algorithm used to solve Problem (6.1).

Iterative differentiation can be applied using the forward-mode [START_REF] Wengert | A simple automatic derivative evaluation program[END_REF][START_REF] Deledalle | Stein Unbiased GrAdient estimator of the Risk (SUGAR) for multiple parameter selection[END_REF][START_REF] Franceschi | Forward and reverse gradientbased hyperparameter optimization[END_REF] or the reverse-mode [START_REF] Linnainmaa | The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors[END_REF][START_REF] Lecun | Efficient backprop[END_REF][START_REF] Domke | Generic methods for optimization-based modeling[END_REF]. Both rely on the chain rule, the gradient being decomposed as a large product of matrices, computed either in a forward or backward way. Note that forward and reverse modes are algorithm-dependent: in this section we illustrate iterative differentiation for proximal gradient descent (PGD, [START_REF] Lions | Splitting algorithms for the sum of two nonlinear operators[END_REF][START_REF] Combettes | Signal recovery by proximal forward-backward splitting[END_REF], using the forward-mode (Algorithm 6.1), and the reversemode (Algorithm 6.2).

The most popular method in automatic differentiation is the reverse-mode, a cornerstone of deep learning (Goodfellow et al., 2016, Chap. 8). Iterative differentiation for hyperparameter optimization can be traced back to [START_REF] Domke | Generic methods for optimization-based modeling[END_REF], who derived (for smooth loss functions) a reverse-mode with gradient descent, heavy ball and L-BFGS algorithms. It first computes the solution of the optimization Problem (6.1) using an iterative solver, but requires storing the iterates along the computation for a backward evaluation of the hypergradient (Algorithm 6.2). Alternatively, the forward-mode computes jointly the solution along with the gradient ∇ λ L(λ). It is memory efficient (no iterates storage) but more computationally expensive when the number of hyperparameters (r) is large; see [START_REF] Baydin | Automatic differentiation in machine learning: a survey[END_REF] for a survey.

Resolution of the bilevel Problem (6.2). From a theoretical point of view, solving Problem (6.2) using gradient-based methods is also challenging, and results in the literature are quite scarce. [START_REF] Kunisch | A bilevel optimization approach for parameter learning in variational models[END_REF] studied the convergence of a semi-Newton algorithm where both the outer and inner problems are smooth. [START_REF] Franceschi | Bilevel programming for hyperparameter optimization and meta-learning[END_REF] gave similar results with weaker assumptions to unify hyperparameter optimization and meta-learning with a bilevel point of view. They required the inner problem to have a unique solution for all λ > 0 but do not have second-order assumptions on Φ. Recent results [START_REF] Ghadimi | Approximation methods for bilevel programming[END_REF][START_REF] Ji | Provably faster algorithms for bilevel optimization and applications to meta-learning[END_REF][START_REF] Mehmood | Differentiating the value function by using convex duality[END_REF] have provided quantitative convergence toward a global solution of Problem (6.2), but under global joint convexity assumption and exact knowledge of the gradient Lipschitz constant.

Algorithm 6.1 Forward-mode PGD

input : λ ∈ R r , γ > 0, n iter ∈ N, β (0) ∈ R p , J (
z (k) = β (k-1) -γ∇f (β (k-1) ) // GD step dz (k) = J (k-1) -γ∇ 2 f (β (k-1) ) J (k-1) β (k) = prox γg(•,λ) (z (k) )
// prox. step // update the Jacobian

J (k) = ∂ z prox γg(•,λ) (z (k) ) dz (k) J (k) += ∂ λ prox γg(•,λ) (z (k) ) // O(pr) v = ∇C(β n iter ) return β n iter , J n iter v Algorithm 6.2 Reverse-mode PGD input : λ ∈ R r , γ > 0, n iter ∈ N, β (0) ∈ R p // computation of β for k = 1, . . . , n iter do z (k) =β (k-1) -γ∇f (β (k-1) ) // GD step β (k) = prox γg(•,λ) z (k) // prox. step // backward computation of the gradient g v = ∇C(β (n iter ) ), h = 0 R r for k = n iter , n iter -1, . . . , 1 do h += v ∂ λ prox γg(•,λ) (z (k) ) // O(pr) v ← ∂ z prox γg(•,λ) (z (k) ) v // O(p) v ← (Id -γ∇ 2 f (β (k) ))v // O(np)
return β n iter , h

Bilevel optimization with nonsmooth inner problems

We recalled above how to compute hypergradients when the inner optimization problem is smooth. In this section we tackle the bilevel optimization Problem (6.2) with nonsmooth inner optimization Problem (6.1). Handling nonsmooth inner problems requires specific tools detailed in Section 6.3.1. We then show how to compute gradients with nonsmooth inner problems using implicit differentiation (Section 6.3.2) or iterative differentiation (Section 6.3.3). In Section 6.3.4 we tackle the problem of approximate gradient for a nonsmooth inner optimization problem. Finally, we propose in Section 6.3.6 an algorithm to solve the bilevel optimization Problem (6.2).

Theoretical framework

Differentiability of the regularization path. Before applying first-order methods to tackle Problem (6.2), one must ensure that the regularization path λ → β(λ) is almost everywhere differentiable (as in Figure 6.1). This is the case for the Lasso [START_REF] Mairal | Complexity analysis of the lasso regularization path[END_REF] and the SVM [START_REF] Hastie | The entire regularization path for the support vector machine[END_REF][START_REF] Rosset | Piecewise linear regularized solution paths[END_REF] since solution paths are piecewise differentiable (see Figure 6.1). Results for nonquadratic data fitting terms are scarcer: [START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF] address the practical resolution of sparse logistic regression, but stay evasive regarding the differentiability of the regularization path.

In the general case for problems of the form Problem (6.1), we believe it is an open question and leave it for future work.

Differentiability of proximal operators. The key point to obtain an implicit differentiation formula for nonsmooth inner problems is to differentiate the fixed point equation of proximal gradient descent. From a theoretical point of view, ensuring this differentiability at the optimum is non-trivial: Poliquin and Rockafellar (1996b, Thm. 3.8) showed that under a twice epi-differentiability condition the proximal operator is differentiable at optimum. For the convergence of forward and reverse modes in the nonsmooth case, one has to ensure that, after enough iterations, the updates of the algorithms become differentiable. [START_REF] Deledalle | Stein Unbiased GrAdient estimator of the Risk (SUGAR) for multiple parameter selection[END_REF] justified (weak) differentiability of proximal operators as they are non-expansive. However this may not be a sufficient condition, see Bolte and Pauwels (2020a,b). In our case, we show differentiability after

Sparse logistic regression

Figure 6.1 -Regularization paths (coefficient values as a function of λ), on the diabetes and breast cancer datasets for the Lasso, the elastic net and sparse logistic regression. This illustrates the weak differentiability of the paths. We used diabetes for the Lasso and the elastic net, and the 10 first features of breast cancer for the sparse logistic regression.

support identification of the algorithms: active constraints are identified after a finite number of iterations by proximal gradient descent [START_REF] Liang | Local linear convergence of forwardbackward under partial smoothness[END_REF][START_REF] Vaiter | Model consistency of partly smooth regularizers[END_REF] and proximal coordinate descent, see Nutini (2018, Sec. 6.2) or [START_REF] Klopfenstein | Model identification and local linear convergence of coordinate descent[END_REF]. Once these constraints have been identified convergence is linear towards the Jacobian (see Theorem 6.12 and Figures 6.2 to 6.4).

For the rest of this chapter, we consider the bilevel optimization Problem (6.2) with the following assumptions on the inner Problem (6.1).

Assumption 6.2 (Smoothness). The function f : R p → R is a convex, differentiable function, with a L-Lipschitz gradient.

Assumption 6.3 (Proper,closed,convex). For all λ ∈ R r , for any j ∈ [p], the function g j (•, λ) : R → R is proper, closed and convex.

Assumption 6.4 (Non-degeneracy). The problem admits at least one solution:

arg min β∈R p Φ(β, λ) = ∅ ,
and, for any β solution of Problem (6.1), we have

-∇f ( β) ∈ ri ∂ β g( β, λ) .
To be able to extend iterative and implicit differentiation to the nonsmooth case, we need to introduce the notion of generalized support.

Definition 6.5 (Generalized support, Nutini et al. 2019, Def. 1). For a solution β ∈ arg min β∈R p Φ(β, λ), its generalized support Ŝ ⊆ [p] is the set of indices j ∈ [p] such that g j is differentiable at βj :

Ŝ {j ∈ [p] : ∂ β g j ( βj , λ) is a singleton} .
An iterative algorithm is said to achieve finite support identification if its iterates β (k) converge to β, and there exists K ≥ 0 such that for all j / ∈ Ŝ, for all k ≥ K, β

(k) j = βj .
Examples. For the 1 norm (promoting sparsity), g j ( βj , λ) = e λ | βj |, the generalized support is Ŝ {j ∈ [p] : βj = 0}. This set corresponds to the indices of the non-zero coefficients, which is the usual support definition. For the SVM estimator, g j ( βj , λ) = ι [0,e λ ] ( βj ). This function is non-differentiable at 0 and at e λ . The generalized support for the SVM estimator then corresponds to the set of indices such that βj ∈]0, e λ [. Finally, to prove local linear convergence of the Jacobian we assume regularity and strong convexity on the generalized support. Assumption 6.6 (Locally C 2 and C 3 ). The map β → f (β) is locally C 3 around β. For all λ ∈ R r , for all j ∈ Ŝ the map g j (•, λ) is locally C 2 around βj . Assumption 6.7 (Restricted injectivity). Let β be a solution of Problem (6.1) and Ŝ its generalized support. The solution β satisfies the following restricted injectivity condition:

∇ 2 Ŝ, Ŝ f ( β) 0 .
Assumptions 6.2 and 6.3 are classical to ensure inner problems can be solved using proximal algorithms. Assumption 6.4 can be seen as a generalization of constraint qualifications (Hare and Lewis, 2007, Sec. 1) and is crucial to ensure support identification. Assumptions 6.6 and 6.7 are classical for the analysis [START_REF] Liang | Activity Identification and Local Linear Convergence of Forward-Backward-type Methods[END_REF] and sufficient to derive rates of convergence for the Jacobian of the inner problem once the generalized support has been identified.

The next lemma guarantees uniqueness of Problem (6.1) under Assumptions 6.4 and 6.7. Lemma 6.8 (Liang et al. 2017, Prop. 4.1). Assume that there exists a neighborhood Λ of λ such that Assumptions 6.4 and 6.7 are satisfied for every λ ∈ Λ. Then for every λ ∈ Λ, Problem (6.1) has a unique solution, and the map λ → β(λ) is well-defined on Λ.

We first show how implicit and iterative differentiation can be used with a nonsmooth inner problem. [START_REF] Peyré | Learning analysis sparsity priors[END_REF] proposed to smooth the inner optimization problem, [START_REF] Ochs | Bilevel optimization with nonsmooth lower level problems[END_REF]; [START_REF] Frecon | Bilevel learning of the group lasso structure[END_REF] relied on the forward-mode combined with Bregman iterations to get differentiable steps. For nonsmooth optimization problems, implicit differentiation has been considered for (constrained) convex optimization problems (Gould et al., 2016;[START_REF] Amos | Optnet: Differentiable optimization as a layer in neural networks[END_REF][START_REF] Agrawal | Differentiable convex optimization layers[END_REF], Lassotype problems (Mairal et al., 2012;[START_REF] Bertrand | Implicit differentiation of Lasso-type models for hyperparameter optimization[END_REF], total variation penalties [START_REF] Cherkaoui | Learning to solve TV regularised problems with unrolled algorithms[END_REF], dictionary learning [START_REF] Malézieux | Dictionary and prior learning with unrolled algorithms for unsupervised inverse problems[END_REF], and generalized to strongly monotone operators [START_REF] Winston | Neural monotone operator equilibrium networks[END_REF].

Hypergradient computation: implicit differentiation

The exact proof of Theorem 6.1 cannot be applied when β → Φ(β, λ) is nonsmooth, as Equations (6.6) and (6.7) no longer hold. Nevertheless, instead of the optimality condition of smooth optimization, Equation (6.6), one can leverage the fixed point iteration of proximal gradient descent, which we will see in Equation (6.11). The main theoretical challenge is to show the differentiability of the function β → prox γg (β -γ∇f (β)). Besides, taking advantage of the generalized sparsity of the regression coefficients β(λ) , one can show that the Jacobian Ĵ is row-sparse, leading to substantial computational benefits when computing the hypergradient ∇ λ L(λ)) for Problem (6.1), Theorem 6.9 (nonsmooth implicit formula). Suppose Assumptions 6.2, 6.3 and 6.6 hold. Let 0 < γ ≤ 1/L, where L is the Lipschitz constant of ∇f . Let λ ∈ R r , Λ be a neighborhood of λ, and Γ Λ β(λ) -γ∇f ( β(λ) ) : λ ∈ Λ . In addition, (H1) Suppose Assumptions 6.4 and 6.7 hold on Λ.

(H2) Suppose λ → β(λ) is continuously differentiable on Λ.

(H3) Suppose for all z ∈ Γ Λ , λ → prox γg(•,λ) (z) is continuously differentiable on Λ.

(H4) Suppose ∂ z prox γg(•,λ) and ∂ λ prox γg(•,λ) are Lipschitz continuous on Γ Λ × Λ.

Let β β(λ) be the solution of Problem (6.1), Ŝ its generalized support of cardinality ŝ. Then the Jacobian Ĵ of the inner Problem (6.1) is given by the following formula,

ẑ = β -γ∇f ( β), and A Id ŝ -∂ z prox γg(•,λ) (ẑ) Ŝ Id ŝ -γ∇ 2 Ŝ, Ŝ f ( β) : Ĵ Ŝc : = ∂ λ prox γg(•,λ) ẑ Ŝc , (6.9) Ĵ Ŝ: = A -1 ∂ λ prox γg(•,λ) (ẑ) Ŝ -γ∂ z prox γg(•,λ) (ẑ) Ŝ ∇ 2 Ŝ, Ŝc f ( β) Ĵ Ŝc . (6.10)
Proof According to Lemma 6.8, Assumptions 6.4 and 6.7 ensure Problem (6.1) has a unique minimizer and λ → β(λ) is well-defined on Λ. We consider the proximal gradient descent fixed point equation:

β(λ) = prox γg (•,λ) β(λ) -γ∇f ( β(λ) ) . (6.11)
Together with the conclusion of Lemma 6.8, Assumptions 6.2 and 6.6, and given (H2), (H3) and (H4), we have that λ → ψ β (λ) -γ∇f ( β(λ) ), λ

prox γg(•,λ) β(λ) -γ∇f ( β(λ)
) is differentiable at λ. One can thus differentiate Equation (6.11) with respect to λ, which leads to:

Ĵ = ∂ z prox γg(•,λ) (ẑ) Id -γ∇ 2 f ( β) Ĵ + ∂ λ prox γg(•,λ) ẑ , (6.12)
with ẑ = β -γ∇f ( β). In addition to 0 < γ < 1/L ≤ 1/L j , the separability of g and Assumptions 6.2 to 6.4 and 6.6 ensure (see Lemma 2.19) that for any j ∈ Ŝc ,

∂ z prox γg j (•,λ) βj -γ∇ j f ( β) = 0 . (6.13)
Plugging Equation (6.13) into Equation (6.12) ensures Equation (6.9) for all j ∈ Ŝc :

Ĵ j: = ∂ λ prox γg j (•,λ) βj -γ∇ j f ( β) . (6.14)
Plugging Equations (6.13) and (6.14) into Equation (6.12) shows that the Jacobian restricted on the generalized support Ŝ satisfies the following linear system:

Id ŝ -∂ z prox γg(•,λ) ẑ Ŝ (Id ŝ -γ∇ 2 Ŝ, Ŝ f ( β)) Ĵ Ŝ: = -γ∂ z prox γg(•,λ) (ẑ) Ŝ ∇ 2 Ŝ, Ŝc f ( β) Ĵ Ŝc : + ∂ λ prox g (ẑ) Ŝ: . Since 0 < γ ≤ 1/L, ∂ z prox γg(•,λ) (ẑ) Ŝ (Id ŝ -γ∇ 2 Ŝ, Ŝ f ( β)) 2 ≤ ∂ z prox γg(•,λ) (ẑ) Ŝ • Id ŝ -γ∇ 2 Ŝ, Ŝ f ( β) 2 < 1 . (6.15) Since Equation (6.15) holds, A Id ŝ -∂ z prox γg(•,λ) (ẑ) Ŝ (Id ŝ -γ∇ 2 Ŝ, Ŝ f ( β)
) is invertible, which leads to Equation (6.10).

Remark 6.10. In the smooth case a p × p linear system is needed to compute the Jacobian in Equation (6.8). For nonsmooth problems this is reduced to an ŝ × ŝ linear system (ŝ ≤ p being the size of the generalized support, e.g., the number of non-zero coefficients for the Lasso). This leads to significant speedups in practice, especially for very sparse vector β(λ) . Remark 6.11. To obtain Theorem 6.9 we differentiated the fixed point equation of proximal gradient descent, though one could differentiate other fixed point equations (such as the one from proximal coordinate descent). The value of the Jacobian Ĵ obtained with different fixed point equations would be the same, yet the associated systems could have different numerical stability properties. We leave this analysis to future work.

Hypergradient computation: iterative differentiation

Instead of implicit differentiation, it is also possible to use iterative differentiation on proximal solvers. In section Section 6.2 we presented forward and reverse modes differentiation of proximal gradient descent (Algorithms 6.1 and 6.2). In this section we study the iterative differentiation of proximal coordinate descent (Algorithms 6.3 and 6.4). To instantiate algorithms easily on problems such as the Lasso, partial derivatives of usual proximal operators can be found in Table 6.3.

For coordinate descent, the computation of the iterative Jacobian in a forward way involves differentiating the following update:

z j ← β j -γ j ∇ j f (β) β j ← prox γ j g j β j -γ j ∇ j f (β) J j: ← ∂ z prox γ j g j (•,λ) (z j ) ∈R J j: -γ j ∇ 2 j: f (β) J ∈R p + ∂ λ prox γ j g j (•,λ) (z j ) ∈R p .
We address now the convergence of the iterative Jacobian scheme, a question which remained open in Deledalle et al. (2014, Section 4.1). We show next that the forwardmode converges to the Jacobian in the nonsmooth separable setting of this chapter. Moreover, we prove that the iterative Jacobian convergence is locally linear after support identification.

Theorem 6.12 (Local linear convergence of the Jacobian). Let 0 < γ ≤ 1/L. Suppose Assumptions 6.2, 6.3 and 6.6 hold. Let λ ∈ R r , Λ be a neighborhood of λ, and Γ Λ β(λ) -γ∇f ( β(λ) ) : λ ∈ Λ . In addition, suppose hypotheses (H1) to (H4) from Theorem 6.9 are satisfied and the sequence (β (k) ) k∈N generated by Algorithm 6.1 (respectively by Algorithm 6.3) converges toward β.

Algorithm 6.3 Forward-mode PCD input : X ∈ R n×p , y ∈ R n , λ ∈ R r , n iter ∈ N, β ∈ R p J ∈ R p×r ,
z j ← β j -γ j ∇ j f (β) // CD step dz j ← J j: -γ j ∇ 2 j: f (β) J β j ← prox γ j g j (•,λ) (z j ) // prox. step // update the Jacobian // diff. with respect to λ J j: ← ∂ z prox γ j g j (•,λ) (z j )dz j J j: += ∂ λ prox γ j g j (•,λ) (z j ) β (k) = β J (k) = J v = ∇C(β) return β n iter , J v Algorithm 6.4 Reverse-mode PCD input : X ∈ R n×p , y ∈ R n , λ ∈ R r , n iter ∈ N, β ∈ R p , γ 1 , . . . , γ p > 0 // compute coef.
for k = 1, . . . , n iter do for j = 1, . . . , p do // update the regression coefficients Then, the sequence of Jacobians (J (k) ) k≥0 generated by the forward-mode differentiation of proximal gradient descent (Algorithm 6.1) (respectively by forward-mode differentiation of proximal coordinate descent, Algorithm 6.3) converges locally linearly towards Ĵ .

z j ← β j -γ j ∇ j f (β) // CD step β j ← prox γ j g j (•,λ) (z j ) // prox. step β (k,j) = β; z (k) j = z j // store iterates // compute gradient g in a backward way v = ∇C(β n iter ), h = 0 R r for k = n iter , n iter -1, . . . , 1 do for j = p, . . . , 1 do h -= γ j v j ∂ λ prox γ j g j (•,λ) (z (k) j ) v j * = ∂ z prox γ j g j (•,λ) (z (k) j ) v -= γ j v j ∇ 2 j: f (β (k,j) ) // O(np) return β n iter , h
Proof of Theorem 6.12 can be found in Section 6.A.1. the exact solution up to machine precision are Celer (Massias et al., 2018b(Massias et al., , 2020b) ) for the Lasso and Blitz [START_REF] Johnson | Blitz: A principled meta-algorithm for scaling sparse optimization[END_REF] for the sparse logistic regression. Table 6.1 summarizes the values of the hyperparameters λ used in Figures 6.2 to 6.4.

Comments on Figure 6.2. We illustrate the results of Theorem 6.12 on SVM (for the Lasso and sparse logistic regression, see Figures 6.3 and 6.4) for multiple datasets (leukemia, rcv1, news20 and real-sim4 ). The values of the hyperparameters λ are summarized in Table 6.1. Regression coefficients β(λ) were computed to machine precision (up to duality gap smaller than 10 -16 ) using a state-of-the-art coordinate descent solver implemented in Lightning [START_REF] Blondel | Lightning: large-scale linear classification, regression and ranking in python[END_REF]. The exact Jacobian was computed via implicit differentiation (Equation (6.10)). Once these quantities were obtained, we used the forward-mode differentiation of proximal coordinate descent (Algorithm 6.3) and monitored the distance between the iterates of the regression coefficients β (k) and the exact solution β. We also monitored the distance between the iterates of the Jacobian J (k) and the exact Jacobian Ĵ . The red vertical dashed line represents the iteration number where support identification happens. Once the support is identified, Figures 6.2 to 6.4 illustrate the linear convergence of the Jacobian. However, the behavior of the iterative Jacobian before support identification is more erratic and not even monotone.

Hypergradient computation with approximate gradients

As mentioned in Section 6.2, relying on iterative algorithms to solve Problem (6.1), one only has access to an approximation of β(λ) : this may lead to numerical errors when computing the gradient in Theorem 6.9. Extending the result of Pedregosa (2016, Thm. 1), which states that hypergradients can be computed approximately, we give a stability result for the computation of approximate hypergradients in the case of nonsmooth inner problems. For this purpose we need to add several assumptions to the previous framework.

Theorem 6.13 (Bound on the error of approximate hypergradient). For λ ∈ R r , let β(λ) ∈ R p be the exact solution of the inner Problem (6.1), and Ŝ its generalized support. Suppose Assumptions 6.2, 6.3 and 6.6 hold. Let Λ be a neighborhood of λ, and Γ Λ β(λ) -γ∇f ( β(λ) ) : λ ∈ Λ . Suppose hypotheses (H1) to (H4) from Theorem 6.9 are satisfied. In addition suppose (H5) The application β → ∇ 2 f (β) is Lipschitz continuous.

(H6) The criterion β → ∇C(β) is Lipschitz continuous.

(H7) Both optimization problems in Algorithm 6.5 are solved up to precision with support identification:

β (λ) -β(λ) ≤ , A is invertible, and A -1 ∇ Ŝ C(β (λ) ) -v ≤ .
Then the error on the approximate hypergradient h returned by Algorithm 6.5 is of the order of magnitude of the error on β (λ) and v:

∇L(λ) -h = O( ) .
Proof of Theorem 6.13 can be found in Section 6.B. Following the analysis of [START_REF] Pedregosa | Hyperparameter optimization with approximate gradient[END_REF], two sources of approximation errors arise when computing the hypergradient: one from the inexact computation of β, and another from the approximate resolution of the linear system. Theorem 6.13 states that if the inner optimization problem and the linear system are solved up to precision , i.e., β(λ)β (λ) ≤ and A -1 ∇ S C(β (λ) )v ≤ , then the approximation on the hypergradient is also of the order of .

Remark 6.14. The Lipschitz continuity of the proximity operator with respect to λ (H4) is satisfied for usual proximal operators, in particular all the operators in Table 6.3. The Lipschitz continuity of the Hessian and the criterion, hypotheses (H5) and (H6), are satisfied for usual machine learning loss functions and criteria, such as the least squares and the logistic loss.

Remark 6.15. To simplify the analysis, we used the same tolerance for the resolution of the inner Problem (6.1) and the resolution of the linear system. Theorem 6.13 gives intuition on the fact that the inner problem does not need to be solved at high precision to lead to good hypergradients estimation. Note that in practice one does not easily control the distance between the approximate solution and the exact one β (k) -β : most softwares provide a solution up to a given duality gap (sometimes even other criteria), not β (k) -β .

Proposed method for hypergradient computation

We now describe our proposed method to compute the hypergradient of Problem (6.2). In order to take advantage of the sparsity induced by the generalized support, we propose an implicit differentiation algorithm for nonsmooth inner problem that can be found in Algorithm 6.5. First, we compute a solution of the inner Problem (6.1) using a solver identifying the generalized support [START_REF] Liang | Local linear convergence of forwardbackward under partial smoothness[END_REF][START_REF] Klopfenstein | Model identification and local linear convergence of coordinate descent[END_REF]. Then, the hypergradient is computed by solving the linear system in Equation (6.10). This linear system, as mentioned in Section 6.2, can be solved using multiple algorithms, including conjugate gradient or fixed point methods. Table 6.2 summarizes the computational complexity in space and time of the described algorithms. Table 6.2 -Cost in time and space for each method: p is the number of features, n the number of samples, r the number of hyperparameters, and ŝ is the size of the generalized support (Definition 6.5, ŝ ≤ p and usually ŝ p). The number of iterations of the inner solver is noted n iter , the number of iterations of the solver of the linear system is noted n sys . From a practical point of view, once the hypergradient has been computed, first-order methods require the definition of a step size to solve the non-convex Problem (6.2). As the Lipschitz constant is not available for the outer problem, first-order methods need to rely on other strategies, such as:

Differentiation

• Gradient descent with manually adjusted fixed step sizes [START_REF] Frecon | Bilevel learning of the group lasso structure[END_REF][START_REF] Ji | Provably faster algorithms for bilevel optimization and applications to meta-learning[END_REF]. The main disadvantage of this technique is that it requires a careful tuning of the step size for each experiment. In addition to being potentially tedious, it does not lead to an automatic procedure.

• L-BFGS (as in [START_REF] Deledalle | Stein Unbiased GrAdient estimator of the Risk (SUGAR) for multiple parameter selection[END_REF]. L-BFGS is a quasi-Newton algorithm that exploits past iterates to approximate the Hessian and propose a better descent direction, which is combined with some line search [START_REF] Nocedal | Numerical optimization[END_REF]). Yet, due to the approximate gradient computation, we observed that L-BFGS did not always converge.

• ADAM [START_REF] Kingma | A method for stochastic optimization[END_REF]. It turned out to be inappropriate to the present setting. ADAM was very sensitive to the initial step size and required a careful tuning for each experiment.

• Iteration specific step sizes obtained by line search [START_REF] Pedregosa | Hyperparameter optimization with approximate gradient[END_REF]. While the approach from Pedregosa (2016) requires no tuning, we observed that it could diverge when close to the optimum. The adaptive step size strategy proposed in Algorithm 6.6, used in all the experiments, turned out to be robust and efficient across problems and datasets.

Remark 6.16 (Uniqueness). The solution of Problem (6.1) may be non-unique, leading to a multi-valued regularization path λ → β(λ) [START_REF] Liu | A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton[END_REF] and requiring tools such as optimistic gradient (Dempe et al., 2015, Chap. 3.8). Though it is not possible to ensure uniqueness in practice, we did not face experimental issues due to potential nonuniqueness. For the Lasso, this experimental observation can be theoretically justified [START_REF] Tibshirani | The lasso problem and uniqueness[END_REF]: when the design matrix is sampled from a continuous distribution, the solution of the Lasso is almost surely unique.

Remark 6.17 (Initialization). One advantage of the nonsmooth case with the 1 norm is that one can find a good initialization point: there exists a value λ max (see Table 6.1) such that the solution of Problem (6.1) vanishes for λ ≥ λ max . Hence, a convenient and robust initialization value can be chosen as e λ = e λmax /100. This is in contrast with the smooth case, where finding a good initialization heuristic is hard: starting in flat zones can lead to poor performance for gradient-based methods [START_REF] Pedregosa | Hyperparameter optimization with approximate gradient[END_REF].

Experiments

In this section, we illustrate the benefits of our proposed Algorithm 6.5 to compute hypergradients and Algorithm 6.6 to solve Problem (6.2). Our package, sparse-ho, is implemented in Python. It relies on Numpy [START_REF] Harris | Array programming with NumPy[END_REF], Numba [START_REF] Lam | Numba: A LLVM-based Python JIT Compiler[END_REF] and SciPy [START_REF] Virtanen | Scipy 1.0: fundamental algorithms for scientific computing in python[END_REF]. Figures were plotted using matplotlib [START_REF] Hunter | Matplotlib: A 2d graphics environment[END_REF]. The package is available under BSD3 license at https://github.com/qb3/ sparse-ho, with documentation and examples available at https://qb3.github.io/ sparse-ho/. Online code includes scripts to reproduce all figures and experiments of the chapter.

Hypergradient computation

Comparison with alternative approaches (Figure 6.1). First, we compare different methods to compute the hypergradient: duality gap of 10 -6 for the forward-mode and the reverse-mode. cvxpylayers relies on cvxpy, solving Problem (6.1) using a splitting conic solver [START_REF] O'donoghue | SCS: Splitting conic solver[END_REF].

Since the termination criterion of the splitting conic solver is not exactly the duality gap (O'Donoghue et al., 2016, Sec. 3.5), we used the default tolerance of 10 -4 . The hypergradient ∇L(λ) was computed for hold-out mean squared error (see Table 6.2).

The forward-mode differentiation of proximal coordinate descent is one order of magnitude faster than cvxpylayers and two orders of magnitude faster than the reversemode differentiation of proximal coordinate descent. The larger the value of λ, the sparser the coefficients β are, leading to significant speedups in this regime. This performance is in accordance with the lower time cost of the forward mode in Table 6.2.

Combining implicit differentiation with state-of-the art solvers (Figures 6.2 and 6.3). We now compare the different approaches described in Section 6.3:

• Forward-mode differentiation of proximal coordinate descent (Algorithm 6.3).

• Implicit differentiation (Algorithm 6.5) with proximal coordinate descent to solve the inner problem. For efficiency, this solver was coded in Numba [START_REF] Lam | Numba: A LLVM-based Python JIT Compiler[END_REF].

• Implicit differentiation (Algorithm 6.5) with state-of-the-art algorithm to solve the inner problem: we used Celer (Massias et al., 2020b) for the Lasso, and Lightning [START_REF] Blondel | Lightning: large-scale linear classification, regression and ranking in python[END_REF] for the SVM.

Figure 6.2 shows for three datasets and two values of regularization parameters the absolute difference between the exact hypergradient and the approximate hypergradient obtained via multiple algorithms as a function of time. Figure 6.3 reports similar results for the SVM, on the same datasets, except news20, which is not well suited for SVM, due to limited number of samples.

First, it demonstrates that implicit differentiation methods are faster than the forwardmode of proximal coordinate descent (pink). This illustrates the benefits of restricting the gradient computation to the support of the Jacobian, as described in Section 6.3.5. Second, thanks to the flexibility of our approach, we obtain additional speed-ups by combining implicit differentiation with a state-of-the-art solver, Celer. The resulting method (orange) significantly improves over implicit differentiation using a vanilla proximal coordinate descent (green). .3 -SVM with hold-out criterion: absolute difference between the exact hypergradient (using β) and the iterate hypergradient (using β (k) ) of the SVM as a function of time. "Implicit diff. + Lightning" uses Lightning [START_REF] Blondel | Lightning: large-scale linear classification, regression and ranking in python[END_REF], instead of our proximal coordinate descent implementation.

Resolution of the bilevel optimization problem

In this section we compare multiple methods to find the optimal hyperparameters for the Lasso, elastic net and multiclass sparse logistic regression. The following methods are compared:

• Grid-search: for the Lasso and the elastic net, the number of hyperparameters is small, and grid-search is tractable. For the Lasso we chose a grid of 100 hyperparameters λ, uniformly spaced between λ maxln(10 4 ) and λ max . For the elastic net we chose for each of the two hyperparameters a grid of 10 values uniformly spaced between λ max and λ maxln(10 4 ). The product grid thus has 10 2 points.

• Random-search: we chose 30 values of λ sampled uniformly between λ max and λ maxln(10 4 ) for each hyperparameter. For the elastic net we chose 30 points sampled uniformly in [λ maxln(10 4 ), λ max ] × [λ maxln(10 4 ), λ max ]. • SMBO: this algorithm is SMBO using as criterion expected improvement (EI) and the Tree-structured Parzen Estimator (TPE) as model. First it evaluates L using 5 values of λ, chosen uniformly at random between λ max and λ maxln(10 4 ).

Then a TPE model is fitted on the data points (λ (1) , L(λ (1) )), . . . , (λ (5) , L(λ (5) )).

Iteratively, the EI is used to choose the next point to evaluate L at, and this value is used to update the model. We used the hyperopt implementation [START_REF] Bergstra | Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms[END_REF]. • 1st order: first-order method with exact gradient (Algorithm 6.6 with constant tolerances i = 10 -6 ), with λ maxln(10 2 ) as a starting point. • 1st order approx: a first-order method using approximate gradient (Algorithm 6.6 with tolerances i , geometrically decreasing from 10 -2 to 10 -6 ), with λ maxln(10 2 ) as a starting point.

Outer criterion. In the Lasso and elastic net experiments, we pick a K-fold CV loss as outer criterion 5 . Hence, the dataset (X, y) is partitioned into K hold-out datasets (X train k , y train k ), (X val k , y val k ). The bilevel optimization problems then write: (6.16) while Lasso CV is obtained taking λ 2 → -∞ in the former. By considering an extended variable β ∈ R K×p , cross-validation can be cast as an instance of Problem (6.2). Figure 6.4 represents the cross-validation loss in Lasso CV as a function of the regularization parameter λ (black curve, three top rows) and as a function of time (bottom). Each point corresponds to the evaluation of the cross-validation criterion for one λ value. The top rows show cross-validation loss as a function of λ, for the grid-search, the SMBO optimizer and the first-order method. The lightest crosses correspond to the first iterations of the algorithm and the darkest, to the last ones. For instance, Lasso grid-search starts to evaluate the cross-validation function with λ = λ max and then decreases to λ = λ maxln(10 4 ). On all the datasets, first-order methods are faster to find the optimal regularization parameter, requiring only 5 iterations. Figure 6.5 represents the level sets of the cross-validation loss for the elastic net (three top rows) and the cross-validation loss as a function of time (bottom). One can see that after 5 iterations the SMBO algorithm (blue crosses) suddenly slows down (bottom) as the hyperparameter suggested by the algorithm leads to a costly optimization problem to solve, while first-order methods converge quickly as for Lasso CV. In the present context, inner problems are slower to solve for low values of the regularization parameters.

arg min λ=(λ 1 ,λ 2 )∈R 2 L(λ) = 1 K K k=1 y val k -X val k β(λ,k
Multiclass sparse logistic regression (# classes hyperparameters, Figure 6.6). We consider a multiclass classification problem with q classes. The design matrix is noted X ∈ R n×p , and the target variable y ∈ {1, . . . , q} n . We chose to use a one-versus- all model with q regularization parameters. We use a binary cross-entropy for the inner loss:

ψ k (β, λ k ; X, y) - 1 n n i=1
1 y i =k ln(σ(X i: β)) + (1 -1 y i =k ) ln(1σ(X i: β)) + e λ k β 1 , and a multiclass cross-entropy for the outer criterion:

C β(λ 1 ) , . . . , β(λq) ; X, y -

n i=1 q k=1 ln   e X i: β(λ k )
q l=1 e X i: β(λ l )   1 y i =k .

(6.17 (6.18) Figure 6.6 represents the multiclass cross-entropy (top), the accuracy on the validation set (middle) and the accuracy on the test set (unseen data, bottom). When the number of hyperparameter is moderate (q = 10, on mnist and usps), the multiclass cross-entropy

The updates of the Jacobian then become:

J (k+1) Ŝ: = ∂ z prox γg(•,λ) (z (k) ) Ŝ Id -γ∇ 2 Ŝ, Ŝ f (β (k) ) J (k) Ŝ:
+∂ λ prox γg(•,λ) (z (k) ) Ŝ: .

From Assumption 6.6, we have that f is locally C 3 at β, g(•, λ) is locally C 2 at β hence (6.25) The inequality on the derivative of the proximal operator comes from the non-expansiveness of proximal operators. The second inequality comes from Assumption 6.7 and 0 < γ ≤ 1/L. Assumptions 6.2 to 6.4, 6.6 and 6.7 and the convergence of (β (k) ) toward β ensure (β (k) ) k∈N converges locally linearly (Liang et al., 2014, Thm. 3.1). The asymptotic autoregressive sequence in Equation (6.24), ρ(A) < 1, and the local linear convergence of ( (k) ) k∈N , yield our result using Lemma 6.18.

We now prove Theorem 6.12 for proximal coordinate descent.

Proximal coordinate descent. Compared to proximal gradient descent, the analysis of coordinate descent requires studying functions defined as a the composition of p applications, each of them only modifying one coordinate.

Coordinate descent updates read as follows:

β (k,j) j
= prox γ j g j (•,λ) β (k,j-1) j γ j ∇ j f (β (k,j-1) ) z (k,j-1) j . (6.26)

We consider the following sequence: J (k,j) j:

= ∂ z prox γ j g j (•,λ) (z (k,j-1) j

) J

(k,j-1) j:

-γ j ∇ 2 j: f (β (k,j-1) ) J (k,j-1)

+ ∂ λ prox γ j g j (•,λ) (z (k,j-1) j

) . (6.27)

Note that if prox γg(•,λ) is not differentiable with respect to the first variable at z (k) (respectively with respect to the second variable λ), any weak Jacobian can be used. When (H3) holds, differentiating Equation (6.26) with respect to λ yields exactly Equation (6.27).

CHAPTER 6. HYPERPARAMETER OPTIMIZATION Assumptions 6.2 to 6.4 and 6.6 and the convergence of (β (k) ) k∈N toward β ensure proximal coordinate descent has finite identification property (Klopfenstein et al., 2020, Thm. 1): we note K the iteration when identification is achieved. Once the generalized support Ŝ (of cardinality ŝ) has been identified, we have that for all k ≥ K, β = ∂ λ prox γ j g j (•,λ) (z (k,j-1) j

). Then, we have that for any j ∈ Ŝ and for all k ≥ K: J (k,j) j:

= ∂ z prox γ j g j (•,λ) (z (k,j-1) j ) J (k,j-1) j:

-γ j ∇ 2 j, Ŝ f (β (k,j-1) ) J (k,j-1) Ŝ:

+ ∂ λ prox γ j g j (•,λ) (z (k,j-1) j

)γ j ∂ z prox γ j g j (•,λ) (z (k,j-1) j

)∇ 2 j, Ŝc f (β (k,j-1) ) J

(k,j-1) Ŝc :

. Let e 1 , . . . , e ŝ be the vectors of the canonical basis of R ŝ. We can consider the applications

R p → R ŝ β → ∂ z prox γ j g j (•,λ) β j -γ j ∇ j f (β) e j -γ j ∇ 2 j, Ŝ f (β) , and
R p → R ŝ×r
β → ∂ λ prox γ j g j (•,λ) β jγ j ∇ j f (β)γ j ∂ z prox γ j g j (•,λ) β jγ j ∇ j f (β) ∇ 2 j, Ŝc f (β) Ĵ Ŝc : , which are both differentiable at β using Assumption 6.6 and (H4). The Taylor expansion of the previous functions yields: J (k,j) j:

= ∂ z prox γ j g j (•,λ) ẑj e jγ j ∇ 2 j, Ŝ f ( β) J

(k,j-1) Ŝ:

+ ∂ λ prox γ j g j (•,λ) ẑjγ j ∂ z prox γ j g j (•,λ) ẑj ∇ 2 j, Ŝc f ( β) J (k,j-1) Ŝc :

+ o( β (k,j-1) -β ) .

Let j 1 , . . . , j ŝ be the indices of the generalized support of β. When considering a full epoch of coordinate descent, the Jacobian is obtained as the product of matrices of the form The spectral radius of A is strictly bounded by 1 (Klopfenstein et al., 2020, Lemma 8): ρ(A) < 1. Assumptions 6.2 to 6.4 and 6.6 and the convergence of (β (k) ) k∈N toward β ensure local linear convergence of (β (k) ) k∈N (Klopfenstein et al., 2020, Thm. 2). Hence, we can write the update for the Jacobian after an update of the coordinates from 1 to p: J (k+1) Ŝ:

= A J (k) Ŝ:

+b + (k) , (6. 28) with ( (k) ) k∈N converging linearly to 0.

Recalling ρ(A) < 1, Lemma 6.18 and the last display yield our result using.

6.B Proof of the approximate gradient theorem Theorem 6.13 (Bound on the error of approximate hypergradient). For λ ∈ R r , let β(λ) ∈ R p be the exact solution of the inner Problem (6.1), and Ŝ its generalized support. Suppose Assumptions 6.2, 6.3 and 6.6 hold. Let Λ be a neighborhood of λ, and Γ Λ β(λ) -γ∇f ( β(λ) ) : λ ∈ Λ . Suppose hypotheses (H1) to (H4) from Theorem 6.9 are satisfied. In addition suppose (H5) The application β → ∇ 2 f (β) is Lipschitz continuous.

(H6) The criterion β → ∇C(β) is Lipschitz continuous.

(H7) Both optimization problems in Algorithm 6.5 are solved up to precision with support identification: β (λ) -β(λ) ≤ , A is invertible, and A -1 ∇ Ŝ C(β (λ) )v ≤ .

Then the error on the approximate hypergradient h returned by Algorithm 6.5 is of the order of magnitude of the error on β (λ) and v:

∇L(λ)h = O( ) .

Proof

Overview of the proof. Our goal is to bound the error between the approximate hypergradient h returned by Algorithm 6.5 and the true hypergradient ∇L(λ). Following the analysis of [START_REF] Pedregosa | Hyperparameter optimization with approximate gradient[END_REF], two sources of approximation errors arise when computing the hypergradient:

• Approximation errors from the inexact computation of β. Dropping the dependency with respect to λ, we denote β the approximate solution and suppose the problem is solved to precision with support identification (H7):

   β Ŝc = β Ŝc β Ŝ -β Ŝ ≤ .
• Approximation errors from the approximate resolution of the linear system, using (H7) yields: • Using the last two points, the goal is to bound the difference between the exact hypergradient and the approximate hypergradient, ∇L(λ)h . Following Algorithm 6.5, the exact hypergradient reads Let ṽ be the exact solution of the approximate system A ṽ ∇ Ŝ C(β). The following conditions are met:

A -1 ∇ Ŝ C(β) -v ≤ .
• v is the exact solution of the exact linear system and ṽ is the exact solution of the approximate linear system

 v ∇ Ŝ C( β)
A ṽ ∇ Ŝ C(β) .

• One can control the difference between the exact matrix in the linear system  and the approximate matrix A.

A -Â 2 ≤ δ β -β , for a certain δ > 0 (Equation (6.30)).

• One can control the difference between the two right-hand side of the linear systems

∇ Ŝ C(β) -∇ Ŝ C( β) ≤ L C β -β ,
since β → ∇C(β) is L C -Lipschitz continuous (H6).

• One can control the product of the perturbations

δ • β -β • Â-1 2 ≤ ρ < 1 .
Conditions are met to apply the result by Higham (2002, Thm 7.2), which leads to

ṽ -v ≤ 1 -Â-1 δ L C Â-1 + v • Â-1 δ ≤ 1 -ρ L C Â-1 + v • Â-1 δ = O( ) . (6.31)
The bound on ṽv finally yields a bound on the first quantity in Equation ( 6.3), vv : (6.34)

v -v = v -ṽ + ṽ -v ≤ v -ṽ + ṽ -v ≤ A -1 A(v -ṽ) + ṽ -v ≤ A -1 2 × A(v -ṽ) ≤ (
Plugging Equations (6.32) and (6.34) into Equation ( 6.3) yields the desired result: ∇L(λ)h = O( ).

Conclusion and perspectives

«La chose la plus importante à toute la vie est le choix du métier : le hasard en dispose.»

In this thesis we first investigated some theoretical properties of coordinate descent.

We showed finite time model identification and local linear convergence. Relying on these two properties, we proposed an Anderson accelerated version of cyclic proximal coordinate descent: andersoncd. It has been implemented in the largest python brain signal processing package, and is now the default solver for sparse signal estimation.

Then we explored a statistical approach to set the regularization parameter of Lassotype problems. We showed that partial smoothing preserves the statistical properties of pivotal estimators, while making the optimization problem amenable to efficient coordinate descent algorithms. We have extensively illustrated the interest of these estimators on real M/EEG data: on visual and auditory tasks. However this approach relies on unrealistic hypotheses and quantities unknown in practice.

Finally we investigated hyperparameter selection through the lens of bilevel optimization. We extended usual first-order methods for bilevel optimization problems with smooth inner problems, to nonsmooth inner problems. Leveraging sparsity, we were able to speed-up hypergradient computations. This enabled efficient sparse linear models calibration with a large number of hyperparameters.

Over the last decades, convex optimization has lead to a wealth of algorithms to solve single-level optimization problems, leaving the question of model selection of optimization-based estimators to statisticians. We hope we convinced the reader of the interest of solving directly bilevel optimization problems to moderate precision, instead of single-level optimization problems to machine precision.

Even in the case of smooth inner problems, practical packages, such as scikit-learn or glmnet, rely on zero-order methods for hyperparameter optimization, through gridsearch, random-search, or Bayesian techniques. First-order method packages exist (hoag, Pedregosa 2016), but they still rely on other hyperparameters requiring manual calibration: in practice, they do not lead to automated algorithms. In particular, they are currently based on nested for-loops (Algorithm 6.6), which can lack of clear stopping criterion to avoid solving the inner optimisation problems to unnecessary precision. Some "online" algorithms have been proposed, but currently mostly rely on heuristics [START_REF] Baydin | Online learning rate adaptation with hypergradient descent[END_REF][START_REF] Mackay | Self-tuning networks: Bilevel optimization of hyperparameters using structured bestresponse functions[END_REF][START_REF] Vicol | Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies[END_REF].

Note also that bilevel optimization is studied in game theory for almost a century under the name of Stackelberg equilibrium (von Stackelberg, 1934), for which a wealth of specific algorithms have been developed [START_REF] Korpelevich | The extragradient method for finding saddle points and other problems[END_REF][START_REF] Rakhlin | Optimization, learning, and games with predictable sequences[END_REF][START_REF] Mescheder | The numerics of GANs[END_REF]. Unifying bilevel optimization and Stackelberg game theory [START_REF] Sinha | A review on bilevel optimization: from classical to evolutionary approaches and applications[END_REF], with the design of principled, efficient, and practical "online" algorithms appears a major area of research for the coming years. This would allow to select the regularization parameter, without even solving exactly one inner problem. Abstract : Due to non-invasiveness and excellent time resolution, magneto-and electroencephalography (M/EEG) have emerged as tools of choice to monitor brain activity. Reconstructing brain signals from M/EEG measurements can be cast as a high dimensional ill-posed inverse problem. Typical estimators of brain signals involve challenging optimization problems, composed of the sum of a data-fidelity term, and a sparsity promoting term. Because of their notoriously hard to tune regularization hyperparameters, sparsity-based estimators are currently not massively used by practitioners. The goal of this thesis is to provide a simple, fast, and automatic way to calibrate sparse linear models. We first study some properties of coordinate descent: model identification, local linear convergence, and acceleration. Relying on Anderson extrapolation schemes, we propose an effective way to speed up coordinate descent in theory and practice. We then explore a statistical approach to set the regularization parameter of Lasso-type problems. A closed-form formula can be derived for the optimal regularization parameter of L1 penalized linear regressions. Unfortunately, it relies on the true noise level, unknown in practice. To remove this dependency, one can resort to estimators for which the regularization parameter does not depend on the noise level. However, they require to solve challenging "nonsmooth + nonsmooth" optimization problems. We show that partial smoothing preserves their statistical properties and we propose an application to M/EEG source localization problems. Finally we investigate hyperparameter optimization, encompassing held-out or cross-validation hyperparameter selection. It requires tackling bilevel optimization with nonsmooth inner problems. Such problems are canonically solved using zeros order techniques, such as grid-search or random-search. We present an efficient technique to solve these challenging bilevel optimization problems using first-order methods.

Universit é Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France

+

  Set of p by p symmetric positive semidefinite matrices S p ++ Set of p by p symmetric positive definite matrices • Euclidean norm on vectors and matrices Norm induced by the matrix A ∈ S n ++ β A = β Aβ • S ,p Schatten p-norm on matrices for p ∈ [1, +∞]

  Support of β ∈ R p j ∈ [p] : β j = 0 S c Complement of the set S B(x, ) Ball of center x and radius J ψ(x) Jacobian of the function ψ at x Convex analysis prox g (x) Proximity operator of g at x arg min y∈R p 1 2 xy 2 + g(y) aff(C) Affine hull of the convex set C ri(C) Relative interior of the convex set C ι C (x) Indicator function of the set C at x ι C Domain of the function f{x ∈ R p : f (x) < +∞} ∂f (x) Subdifferential of the function f at x ∂f (x) = {s ∈ R p : f (y) ≥ f (x)+ s, y-x , ∀y ∈ dom(f )} f * (u)Fenchel conjugate of the function f at u sup x u, xf (x)

Figure 1 . 1 -

 11 Figure 1.1 -Multiple neuroimaging modalities. In electrocorticography (left, Blausen 2014), a grid of sensors is placed onto the brain surface. For electroencephalography (middle 1 ) and magnetoencephalography (right, image from the National Institute of Mental Health), sensors are placed at or close to the head surface and no surgery is needed.

  Brain sources B * . (c) Brain sources location.

Figure 1 . 3 -

 13 Figure1.3 -Influence of the number of trials. Amplitude of the electric field measured at the surface of the head, as a function of time, for multiple numbers of averaged trials Y(l) . With only one trial (left), one cannot see a brain response, after the auditory stimulation at time 0 (dashed red line). When the number of trials increases (middle and right), one can observe a brain response after the stimulation.

CHAPTER 1 .

 1 Figure 1.4 -Hyperparameter selection for real M/EEG data. Brain source locations (in red) found in the left (top) and in the right (bottom) hemispheres, using multitask Lasso Problem (1.3), for multiple values of λ. Each column corresponds to one value of the hyperparameter λ. Which λ to pick? How to automatically select λ?

  for some s ∈ N , (1.6)

Figure 1 . 5 -

 15 Figure 1.5 -Maximal sparsity that can theoretically be recovered assuming mutual incoherence in Equation (1.7) is displayed for multiple values of number of samples and number of features (left), and for multiple real-life datasets (right). For the synthetic data, entries of the design matrix X are drawn i.i.d. following a Gaussian distribution of mean 0 and variance 1. Real datasets are taken from libsvm (Chang and Lin, 2011) 2 .

Figure 1

 1 Figure 1.6 -Five iterations of gradient descent (left in blue) and cyclic coordinate descent (right in red) to minimize the quadratic functionf (β 1 , β 2 ) = 7β 2 1 + 6β 1 β 2 + 8β 2 2 .Contour lines of the function f are in blue. Each algorithm starts at point (8, -6) and converges toward the minimizer (0, 0). The darker the point, the larger the iteration number. From a computational point of view, two updates of coordinate descent are equivalent to one update of gradient descent (see Algorithm 1.4).

Figure 1 . 8 -

 18 Figure1.8 -Influence of stepsizes for cyclic coordinate descent, OLS. Suboptimality f (β (k) )f ( β) as a function of the number of epochs to solve a least squares problem on multiple datasets from libsvm: leukemia, real-sim and rcv1. Gradient descent is compared against coordinate descent with step sizes γ j = δ/L j + (1δ)/L, for multiple values of δ.

Figure 2 . 1 -

 21 Figure 2.1 -Lasso, linear convergence. Distance to optimum, β (k) -β , as a function of the number of iterations k, on 4 different datasets: leukemia, gisette, rcv1, and real-sim.

Figure 2 . 2 -

 22 Figure 2.2 -Sparse logistic regression, linear convergence. Distance to optimum, β (k) -β , as a function of the number of iterations k, on 4 different datasets: leukemia, gisette, rcv1, and real-sim.

Figure 2 . 3 -

 23 Figure 2.3 -Support vector machine, linear convergence. Distance to optimum, β (k) -β , as a function of the number of iterations k, on 4 different datasets: leukemia, gisette, rcv1 and 20news.

Figure 3

 3 Figure 3.1 -Suboptimality along time for a quadratic problem on the 5000 first features of the rcv1 dataset. GD: gradient descent, CD: cyclic coordinate descent, RCD: randomized coordinate descent.

Figure 3

 3 Figure 3.2 -Illustrations of offline (left) and online (right) Anderson extrapolation.

  < 1 and the iterates of offline Anderson acceleration satisfy, with B = (T -Id) (T -Id) 0:

(3. 21 )

 21 Then we use twice Lemma 3.4 (a) for the left-hand and right-hand side of Equation (3.21). Using Lemma 3.4 (a) with the c * i minimizing Equation (3.2) we have for all

Figure 3 Figure 3

 33 Figure 3.4 -OLS, rcv1. Suboptimality as a function of time on the 5000 first columns of the dataset rcv1.

Figure 3 . 6 -Figure 3 . 7 -

 3637 Figure 3.6 -Influence of K, quadratic, rcv1. Influence of the number of iterates K used to perform Anderson extrapolation with coordinate descent (CD) on a quadratic with the rcv1 dataset (2000 first columns).

Figure 3 . 8 -Figure 3 . 9 -

 3839 Figure 3.8 -Lasso, suboptimality. Suboptimality as a function of time for the Lasso on multiple datasets and values of λ.

Figure 3

 3 Figure 3.10 -1 -regularised logistic regression, suboptimality. Suboptimality as a function of time for 1 -regularized logistic regression on multiple datasets and values of λ.

Figure 3

 3 Figure3.11 -Group Lasso, suboptimality. Suboptimality as a function of time for the group Lasso on the Leukemia dataset, λ = λ max /100. Groups are artificially taken as consecutive blocks of 5 features.

Contents 4 . 1

 41 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.1.1 Motivation and general proof structure . . . . . . . . . . . . . . . 89 4.1.2 Preliminary lemma . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.1.3 Minimax lower bounds . . . . . . . . . . . . . . . . . . . . . . . . 92 4.1.4 Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.2 Multitask square-root Lasso . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.3 Multivariate square-root Lasso . . . . . . . . . . . . . . . . . . . . . . . 96 4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.4.1 Pivotality of the square-root Lasso . . . . . . . . . . . . . . . . . 99 4.4.2 Rank deficiency experiment . . . . . . . . . . . . . . . . . . . . . 99 4.4.3 (Multitask) smoothed concomitant Lasso . . . . . . . . . . . . . 99 4.4.4 Smoothed generalized concomitant Lasso (SGCL) . . . . . . . . . 101 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.A Concentration inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Figure 4

 4 Figure 4.2 -Singular values of the residuals Y -X B of the multivariate square-root Lasso (n = 10, T = 20, p = 30), as a function of λ. The observation matrix Y is full rank, but the residuals are rank deficient even for high values of the regularization parameter, invalidating the classical assumptions needed for statistical analysis.

Notation.

  For any B ∈ R p×T we define S(B) {j ∈ [p] : ||B j: || 2 = 0} the rowwise support of B. We write S * for the row-wise support of the true coefficient matrix B * ∈ R p×T . For any B ∈ R p×T and any subset S of [p] we denote B S the matrix in R p×T which has the same values as B on the rows with indices in S and vanishes on the complement S c . The estimated regression coefficients are written B, their difference with the true parameter B * is noted ∆ B -B * . The residuals at the optimum are noted Ê Y -X B. For a < b, [x] b a max(a, min(x, b)) is the clipping of x at levels a and b. Model. Consider the multitask linear regression model: Y = XB * + E , (4.6) 4.1. INTRODUCTION

  Proposition 4.10. Let Assumption 4.1 be satisfied, let α satisfy Assumption 4.2 and let η satisfy Assumption 4.8. Let C = 2

  .61) Finally we exhibit an element of ∂f (E) to apply Lemma 4.5 (b). Recall that f =

Figure 4

 4 Figure 4.3 -(Synthetic data, n = 50, p = 1000, T = 20) Hard recovery loss for different values of SNR for the multitask SCL.

Figure 4

 4 Figure 4.4 -(Synthetic data, n = 50, p = 1000, T = 20) Hard recovery loss (top) and percent of non-zeros coefficients (bottom) for different values of SNR: SNR = 1 (left), SNR = 2 (right) for the multitask SCL.

Figure 4

 4 Figure 4.5 -(Synthetic data, n = 150, p = 500, T = 100) Hard recovery loss for different values of SNR for the SGCL.

. 75 )

 75 Lemma 4.16 (b) is a direct consequence of Lemma 4.16 (a).

5. 2 . 1

 21 Model and proposed estimator Probabilistic model. Let r be the number of repetitions of the experiment. The r observation matrices are denoted Y (1) , . . . , Y (r) ∈ R n×T with n the number of sensors/samples and T the number of tasks/time samples. The mean over the repetitions of the observation matrices is written Ȳ = 1 r r l=1 Y

Notation.

  The unit p ball is written B p , p ∈ [1, ∞). For a, b ∈ R, we denote (a) + = max(a, 0), a ∨ b = max(a, b) and a ∧ b = min(a, b). The block soft-thresholding operator at level τ > 0, is denoted BST(•, τ ), and reads for any vector x, BST(x, τ ) = 1τ / x + x. Let d ∈ N, and let C be a closed and convex subset of R d .

Figure 5

 5 Figure 5.1 -Influence of noise structure.ROC curves of support recovery (ρ X = 0.6, SNR = 0.03, r = 20) for different ρ S values.

FPR

  Figure 5.4 -Influence of the number of repetitions. ROC curves with empirical X and S and simulated B * (amp = 2 nA.m), for different number of repetitions.

Figure 5 . 6 -

 56 Figure 5.6 -Real data, left auditory stimulations (n = 102, p = 7498, T = 76, r = 63) Sources found in the left hemisphere (top) and the right hemisphere (bottom) after left auditory stimulations.

Figure 5 . 7 -

 57 Figure 5.7 -Real data, right auditory stimulations (n = 102, p = 7498, T = 76, r = 33) Sources found in the left hemisphere (top) and the right hemisphere (bottom) after right auditory stimulations.

Figure 5 . 8 -

 58 Figure 5.8 -Real data (n = 102, p = 7498, T = 76, r = 65) Sources found in the left hemisphere (top) and the right hemisphere (bottom) after right auditory stimulations.

Figure 5 . 9 -

 59 Figure 5.9 -Real data (n = 102, p = 7498, T = 76, r = 33) Sources found in the left hemisphere (top) and the right hemisphere (bottom) after right auditory stimulations.

  Figure 5.10 -Real data (n = 102, p = 7498, T = 48, r = 71) Sources found in the left hemisphere (top) and the right hemisphere (bottom) after left visual stimulations.

  Figure 5.11 -Real data (n = 102, p = 7498, T = 48, r = 36) Sources found in the left hemisphere (top) and the right hemisphere (bottom) after left visual stimulations.

  Figure 5.12 -Real data (n = 102, p = 7498, T = 48, r = 61) Sources found in the left hemisphere (top) and the right hemisphere (bottom) after right visual stimulations.

  Figure 5.13 -Real data (n = 102, p = 7498, T = 48, r = 31) Sources found in the left hemisphere (top) and the right hemisphere (bottom) after right visual stimulations.

  Figure 5.14 -Time comparison, real data, n = 102, p = 7498, T = 54, r = 56 Time for each algorithm to produce Figure 5.6.

Figure 6

 6 Figure 6.1 -5-fold cross-validation error C(β (λ) ): (top) Lasso CV error with respect to λ for multiple hyperparameter optimization methods on the real-sim dataset, and (bottom) elastic net CV error with respect to λ 1 and λ 2 on the rcv1 dataset. Crosses represent the 10 (top) or 25 (bottom) first error evaluations for each method.

e

  λ |β j | ST(z j , e λ )| sign(ST(z j , e λ ))| -e λ sign(ST(z j , e λ ))e λ1 |β j | + 1 2 e λ2 β 2 j ST(z j ,e λ 1 ) 1+e λ 2| sign(ST(z j ,e λ 1 ))| 1+e λ 2

Figure 6 . 2 -

 62 Figure 6.2 -Local linear convergence of the Jacobian for the SVM. Distance to optimum for the coefficients β (top) and the Jacobian J (bottom) of the forward-mode differentiation of proximal coordinate descent (Algorithm 6.3) on multiple datasets. One epoch corresponds to one pass over the data, i.e., one iteration with proximal gradient descent.

Figures 6 .Figure 6 . 3 -Figure 6 . 4 -

 66364 Figures 6.3 and 6.4 are the counterparts of Figure6.2 for the Lasso and sparse logistic regression. It shows the local linear convergence of the Jacobian for the Lasso, obtained by the forward-mode differentiation of coordinate descent. The solvers used to determine

  Lassoe λ = 0.01 e λmax e λ = 0.075 e λmax e λ = 0.3 e λmax e λ = 0.1 e λmax Logistic regression e λ = 0.1 e λmax e λ = 0.25 e λmax e λ = 0.8 e λmax e λ = 0.15 e λmax SVM e λ = 10 -5 e λ = 3 × 10 -2 e λ = 10 -3 e λ = 5 × 10 -2

ForwardFigure 6 . 1 -

 61 Figure 6.1 -Lasso with hold-out criterion: time comparison on the gina dataset to compute a single hypergradient as a function of the number of features, for two values of λ, e λ = e λmax /10 (left) and e λ = e λmax /100 (right).

Figure 6 . 2 -

 62 Figure6.2 -Lasso with hold-out criterion: absolute difference between the exact hypergradient (using β) and the iterate hypergradient (using β (k) ) of the Lasso as a function of time. Results are for three datasets and two different regularization parameters. "Implicit diff. + Celer)" uses Celer(Massias et al., 2020b) instead of our proximal coordinate descent implementation.

Figure 6 . 4 -

 64 Figure 6.4 -Lasso with cross-validation criterion: cross-validation loss as a function of λ (black line, top) and as a function of time (bottom). Lighter markers correspond to earlier iterations of the algorithm.

)

  With a single train/test split, the bilevel problem to solve writes:arg min λ (λ 1 ,...,λq)∈R q C β(λ 1 ) , . . . , β(λq) ; X test , y test s.t. β(λ k ) ∈ arg min β∈R p ψ k (β, λ k ; X train , y train ) ∀k ∈ [q] .

  prox g(•,λ) is locally C 2 . The function β → ∂ z prox γg(•,λ) (β -γ∇f (β)) Ŝ (Id -γ∇ 2 Ŝ, Ŝ f (β)) is differentiable at β. Using (H4) we have that β → ∂ λ prox γg(•,λ) (β -γ∇f (β)) Ŝ:is also differentiable at β. Using the Taylor expansion of the previous functions yields:0 < γ ≤ 1/L, ρ(A) ≤ A 2 ≤ ∂ z prox γg(•,λ) (ẑ) Ŝ ≤1 (non-expansiveness) • Id -γ∇ 2 Ŝ, Ŝ f ( β) 2<1 (Assumption 6.7 and 0 < γ ≤ 1/L) < 1 .

  j ∈ Ŝc , ∂ z prox γ j g j (•,λ)

A

  s = e 1 . . . e s-1 v js e s+1 . . . e ŝ ∈ R ŝ×ŝ , where v js = ∂ z prox γ js g js ẑjs e sγ js ∇ 2 js, Ŝ f ( β) ∈ R ŝ. A full epoch can then be written J for a certain b ∈ R ŝ.

  CHAPTER 6. HYPERPARAMETER OPTIMIZATIONThe exact solution of the exact linear system v satisfies:v = Â-1 ∇ Ŝ C( β) , with A Id | Ŝ| -∂ z prox γg(•,λ) β -γ∇f (β) | Ŝ| -∂ z prox γg(•,λ) β -γ∇f ( β)

  ∇L(λ) = Bv + Ĵ Ŝc : ∇ Ŝc C( β) , and similarly for the approximate versions:h = Bv + J Ŝc : ∇ Ŝc C(β) , with B ∂ λ prox γg(•,λ) β -γ∇f (β) Ŝ: -γ∂ z prox γg(•,λ) β -γ∇f (β) Ŝ ∇ 2 Ŝ, Ŝc f (β) Ĵ Ŝc : B ∂ λ prox γg(•,λ) β -γ∇f ( β) Ŝ: -γ∂ z prox γg(•,λ) β -γ∇f ( β) Ŝ ∇ 2 Ŝ, Ŝc f ( β) Ĵ Ŝc : .We can exploit these decompositions to bound the difference between the exact hypergradient and the approximate hypergradient:∇L(λ)h = Bv -Bv + Ĵ Ŝc : ∇ Ŝc C( β) -Ĵ Ŝc : ∇ Ŝc C(β) ≤ Bv -Bv + Ĵ Ŝc : ∇ Ŝc C( β) -Ĵ Ŝc : ∇ Ŝc C(β) ≤ Bv -Bv + Bv -Bv + Ĵ Ŝc : (∇ Ŝc C( β) -∇ Ŝc C(β)) ≤ v • B -B + B • vv + L C Ĵ Ŝc : • β -β . (6.29)Bounding vv and B -B in Equation (6.29) yields the desired result which is bounding the difference between the exact hypergradient and the approximate hypergradient ∇L(λ)h .Bound on vv . We first prove that A -Â = O( ). Let L H be the Lipschitz constant of the application β → ∇ 2 f (β), then we have:A -Â 2 = CD -Ĉ D 2 ≤ CD -C D 2 + C D -Ĉ D 2 ≤ C 2 ≤1 (non-expansiveness) D -D 2 ≤L H β-β using (H5) + D 2 ≤1 C -Ĉ 2 O( β-β ) using (H4) ≤ L H β -β + O( β -β ) = O( β -β ) . (6.30)

  H7) + ṽv O( ) (Equation (6.31))= O( ) . (6.32) CHAPTER 6. HYPERPARAMETER OPTIMIZATION Bound on B -B 2 . We now bound the second quantity in Equation (6.3) B -B 2 :B -B 2 ≤ ∂ λ prox γg(•,λ) (β -γ∇f (β)) Ŝ: -∂ λ prox γg(•,λ) ( β -γ∇f ( β)) Ŝ: 2 + γ ∂ z prox γg(•,λ) ( β -γ∇f ( β)) Ŝ ∇ 2 Ŝ, Ŝc f ( β) Ĵ Ŝc : (6.33) -∂ z prox γg(•,λ) (β -γ∇f (β)) Ŝ ∇ 2 Ŝ, Ŝc f (β) Ĵ Ŝc : 2 ≤ L 1 β -γ∇f (β) Ŝ: -β + γ∇f ( β) using (H4)+ L 2 ββ • Ĵ Ŝc : using (H4) and Assumption 6.6= O( ββ ) .

Titre:

  S élection d'hyperparam ètres pour l'apprentissage parcimonieux en grande dimension : application à la neuroimagerie Mots cl és : S élection et optimisation d'hyperparam ètre, optimisation bi-niveaux, parcimonie, descente par coordonn ée, probl èmes inverses, neuroimagerie R ésum é : Gr âce à leur caract ère non invasif et leur excellente r ésolution temporelle, la magn étoet l' électroenc éphalographie (M/EEG) sont devenues des outils incontournables pour observer l'activit é c ér ébrale. La reconstruction des signaux c ér ébraux à partir des enregistrements M/EEG peut être vue comme un probl ème inverse de grande dimension mal pos é. Les estimateurs typiques des signaux c ér ébraux se basent sur des probl èmes d'optimisation difficiles à r ésoudre, compos és de la somme d'un terme d'attache aux donn ées et d'un terme favorisant la parcimonie. À cause du param ètre de r égularisation notoirement difficile à calibrer, les estimateurs bas és sur la parcimonie ne sont actuellement pas massivement utilis és par les praticiens. L'objectif de cette th èse est de fournir un moyen simple, rapide et automatis é de calibrer des mod èles lin éaires parcimonieux. Nous étudions d'abord quelques propri ét és de la descente par coordonn ées : identification du mod èle, convergence lin éaire locale, et acc él ération. En nous appuyant sur les sch émas d'extrapolation d'Anderson, nous proposons un moyen efficace d'acc él érer la descente par coordonn ées en th éorie et en pratique. Nous explorons ensuite une approche statistique pour calibrer le param ètre de r égularisation des probl èmes de type Lasso. Il est possible de construire des estimateurs pour lesquels le param ètre de r égularisation optimal ne d épend pas du niveau de bruit. Cependant, ces estimateurs n écessitent de r ésoudre des probl èmes d'optimisation "non lisses + non lisses". Nous montrons que le lissage partiel pr éserve leurs propri ét és statistiques et nous proposons une application aux probl èmes de localisation de sources M/EEG. Enfin, nous étudions l'optimisation d'hyperparam ètres, qui comprend notamment la validation crois ée. Cela n écessite de r ésoudre des probl èmes d'optimisation à deux niveaux avec des probl èmes internes non lisses. De tels probl èmes sont r ésolus de mani ère usuelle via des techniques d'ordre z éro, telles que la recherche sur grille ou la recherche al éatoire. Nous pr ésentons une technique efficace pour r ésoudre ces probl èmes d'optimisation à deux niveaux en utilisant des m éthodes du premier ordre. Title : Hyperparameter selection for high dimensional sparse learning: application to neuroimaging Keywords : Hyperparameter selection and optimization, convex optimization, bilevel optimization, sparsity, coordinate descent, inverse problem, neuroimaging

1

  Motivation and contributionsContents 1.1 Model selection for the M/EEG inverse problem . . . . . . . . . . . . . . 21 1.1.1 Neuroimaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.1.2 Linear inverse problem . . . . . . . . . . . . . . . . . . . . . . . . 24 1.1.3 Hyperparameter selection for sparse models . . . . . . . . . . . . 26 1.2 Coordinate descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.2.1 Introduction and intuitions . . . . . . . . . . . . . . . . . . . . . 32 1.2.2 Theoretical results on least squares . . . . . . . . . . . . . . . . . 37 1.2.3 Beyond quadratics . . . . . . . . . . . . . . . . . . . . . . . . . . 40 1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 1.4 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

  Local linear convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 2.3.1 Differentiability outside the generalized support . . . . . . . . . . 61 2.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

	1.4. PUBLICATIONS
	• Q. Klopfenstein, Q. Bertrand, A. Gramfort, J. Salmon, and S. Vaiter. Model identification and local linear convergence of coordinate descent. arXiv preprint arXiv:2010.11825, 2020 2
	• Q. Bertrand, Q. Klopfenstein, M. Massias, M. Blondel, S. Vaiter, A. Gramfort, and J. Salmon. Implicit differentiation for fast hyperparameter selection in non-Model identification and local linear
	smooth convex learning. arXiv preprint arXiv:2105.01637, 2021 convergence
	Contents
	2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
	2.2 Model identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
	2.2.1 Partial smoothness for separable functions . . . . . . . . . . . . . 51
	2.2.2 Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
	2.3	Part I
	Some properties of coordinate
		descent
	Massias, A. Gramfort, and J. Salmon. Handling correlated
	and repeated measurements with the smoothed multivariate square-root Lasso.
	In NeurIPS, 2019
	• M. Massias, Q. Bertrand, A. Gramfort, and J. Salmon. Support recovery and sup-norm convergence rates for sparse pivotal estimation. In AISTATS, 2020a
	• Q. Bertrand, Q. Klopfenstein, M. Blondel, S. Vaiter, A. Gramfort, and J. Salmon. Implicit differentiation of Lasso-type models for hyperparameter optimization.
	ICML, 2020
	• Q. Bertrand and M. Massias. Anderson acceleration of coordinate descent. In AISTATS, 2021

Table 2

 2

	Datasets #samples n #features p	density
	leukemia	38	7129	1
	gisette	6000	4955	1
	rcv1 real-sim 20news	20 242 72 309 5184	19 959 20 958 155 148	3.6 × 10 -3 2.4 × 10 -3 1.9 × 10 -3

.1 -Characteristics of the datasets.

Table 2 .

 2 2 -C values for SVM.

	dataset leukemia	gisette	rcv1	20news
	C value	10	1.5 10 -2 1.5 10 -2 5 10 -1

  as suggested by Figure 6.4. 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3.2 Anderson extrapolation . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 3.2.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 3.2.2 Linear iterations of coordinate descent . . . . . . . . . . . . . . . 71 3.2.3 Anderson extrapolation for nonsymmetric iteration matrices . . . 72 3.2.4 Pseudo-symmetrization of T . . . . . . . . . . . . . . . . . . . . . 72 3.2.5 Generalization to nonquadratic and proposed algorithm . . . . . 76 3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

3.3.1 Parameter setting . . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.3.2 Numerical comparison on machine learning problems . . . . . . . 78 3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

  Application to quadratics The canonical application of Anderson extrapolation is gradient descent on quadratics. Consider a quadratic problem, with b ∈ R p , H ∈ S p

				++
	such that H 0, L denotes the largest eigenvalue of H, L	H 2 :
	β = arg min β∈R p	1 2	β Hβ + b, β .

Table 3

 3

	.1 -Datasets characteristics
	name	n	p	density
	gina agnostic	3468	970	1
	hiva agnostic	4229	1617	1
	leukemia	72	7129	1
	rcv1_train	20 242 19 960 3.7 10 -3
	real-sim	72 309 20 958 2.4 10 -3
	news20	19 996 632 983 6.1 10 -4

  incoherence of the design matrix (Assumption 4.2) implies the Restricted Eigenvalue Property introduced by Bickel et al. (2009).

	Lemma 4.3 (Restricted Eigenvalue Property, Lounici 2008, Lemma 2). If Assump-
	tion 4.2 is satisfied, then:

  . The design X is random with Toeplitz-correlated features 100 CHAPTER 4. ON THE STATISTICAL ASPECTS OF PARTIAL SMOOTHING

	* σ/σ	2.0 1.15 0.66 0.38 0.21 0.12 0.07 0.04 0.02 0.01		SNR=0.75				SNR=1.00
	* σ/σ	2.0 1.15 0.66 0.38 0.21 0.12 0.07 0.04 0.02 0.01		SNR=1.50				SNR=2.00
		1.0	0.56	0.32	0.18	0.1	1.0	0.56	0.32	0.18	0.1
				λ/λ 0					λ/λ 0

  Contents 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.2 Concomitant estimation with correlated noise . . . . . . . . . . . . . . . 109 5.2.1 Model and proposed estimator . . . . . . . . . . . . . . . . . . . 109 5.2.2 Properties of the proposed data fitting term . . . . . . . . . . . . 110 5.2.3 Properties of the proposed estimator and algorithmic details . . . 117 5.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.3.1 Synthetic data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.3.2 Realistic data . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . 123 5.3.3 Real M/EEG data . . . . . . . . . . . . . . . . . . . . . . . . . . 126

This chapters provide an application of the pivotal estimators presented in Chapter 4: the multivariate square-root Lasso. In addition to have a regularization parameter independent from the noise level, the (smoothed version of the) multivariate square-root Lasso can cope with complex noise structure by using non-averaged measurements. The resulting optimization problem consists in a smoothed trace norm as a data fit, and an usual 2,1 -norm as a penalty. It is jointly convex and amenable to efficient block coordinate descent. Practical benefits are extensively demonstrated on real M/EEG neuroimaging data, including visual and auditory cognitive experiments. This chapter is based on the following work, accepted at NeurIPS 2019:

• Q. Bertrand, M. Massias, A. Gramfort, and J. Salmon. Handling correlated and repeated measurements with the smoothed multivariate square-root Lasso.

In NeurIPS, 2019

  6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 6.2 Bilevel optimization with smooth inner problems . . . . . . . . . . . . . 137 6.3 Bilevel optimization with nonsmooth inner problems . . . . . . . . . . . 140 6.3.1 Theoretical framework . . . . . . . . . . . . . . . . . . . . . . . . 140 6.3.2 Hypergradient computation: implicit differentiation . . . . . . . . 142 6.3.3 Hypergradient computation: iterative differentiation . . . . . . . 144 6.3.4 Hypergradient computation with approximate gradients . . . . . 147 6.3.5 Proposed method for hypergradient computation . . . . . . . . . 148 6.3.6 Resolution of the bilevel optimization Problem (6.2) . . . . . . . 148 6.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 6.4.1 Hypergradient computation . . . . . . . . . . . . . . . . . . . . . 149 6.4.2 Resolution of the bilevel optimization problem . . . . . . . . . . 152 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 6.A Proof of the local linear convergence . . . . . . . . . . . . . . . . . . . . 157 6.A.1 Local linear convergence . . . . . . . . . . . . . . . . . . . . . . . 157 6.B Proof of the approximate gradient theorem . . . . . . . . . . . . . . . . 161

Table 6 .

 6 2 -Examples of outer criteria used for hyperparameter selection.

	6.1. INTRODUCTION

  γ 1 , . . . , γ p

	// jointly compute coef. & Jacobian
	for k = 1, . . . , n iter do
	for j = 1, . . . , p do
	// update the regression
	coefficients

Table 6

 6

	.1 -Dataset characteristics and regularization parameters used in Figures 6.2
	to 6.4.
	Datasets	leukemia	rcv1	news20	real-sim
	# samples	n = 38	n = 20 242	n = 19 996	n = 72 309
	# features	p = 7129	p = 19 959	p = 632 982	p = 20 958

https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/.

see https://scipy-lectures.org/advanced/advanced_numpy/ for details.

see https://scipy-lectures.org/advanced/scipy_sparse/index.html for details.

https://github.com/mne-tools/mne-python/commit/080e7a879d325ad8f0c11fe28a8ff9f5983dfd2f

Note that some local rates are shown inTseng and Yun (2009a) but under some strong hypothesis.

modified here with a row-sparse penalty instead of 1

of particular interest: combined with a large coefficients assumption, it implies support identification

√nT

we keep the name of van de Geer (2016), although a better name in our opinion would be the (multitask) trace norm Lasso, but the name is used by[START_REF] Grave | Trace lasso: a trace norm regularization for correlated designs[END_REF] when the nuclear norm is used as a regularizer

See the definition we consider in eq. (5.46).

also called concomitant formulation since minimization is performed over an additional variable[START_REF] Owen | A robust hybrid of lasso and ridge regression[END_REF][START_REF] Ndiaye | Efficient smoothed concomitant lasso estimation for high dimensional regression[END_REF].

As a reminder, for a scalar t > 0, the proximal operator of a function h : R d → R can be defined for any x0 ∈ R d by prox t,h (x0) = arg min x∈R d 1 2t x -x0 2 + h(x) .

For a linear model y = Xβ + ε, with ε ∼ N (0, σ

), the degree of freedom(dof, Efron 1986) is defined as dof(β) = n i=1 cov(yi, (Xβ)i)/σ 2 . 2 The smoothed Hinge loss is given by (x) = 1 2x if x ≤ 0, 1 2 (1x) 2 if 0 ≤x ≤ 1 and 0 else.

Note that implicit refers to the implicit function theorem, but leads to an explicit formula for the gradient.

Data available on the libsvm website: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/ datasets/

In our experiments the default choice is K = 5.

Remerciements Part III

Hyperparameter selection, the bilevel way

Part II

Hyperparameter selection, between statistics and optimization Appendix

4.A Concentration inequalities

The following theorem is a powerful tool to show a lot of concentration inequalities:

Theorem 4.15 (Giraud 2014, Thm B.6 p. 221). Assume that F : R d → R is 1-Lipschitz and z has N (0, σ 2 Id d ) as a distribution, then there exists a variable ξ, exponentially distributed with parameter 1, such that: S ← ClSqrt 1 T r RR , σ // Eq. (5.33) for j = 1, . . . , p do L j = X :j S -1 X :j for j = 1, . . . , p do // coef. update R ← R + X :j B j: // cheap residuals update B j: ← BST

, λnT L j R ← R -X :j B j: // cheap residuals update return B, S First note that the (joint) function (Z, Σ) → Tr Z Σ -1 Z is jointly convex over R n×T × S n ++ , see Boyd and Vandenberghe (2004, Example 3.4). This means that f is jointly convex in (Z, S), moreover B → 1

Moreover the constraint set is convex and thus solving CLaR is a convex problem.

The function f is convex and smooth on the feasible set and • 2,1 is convex in B and separable in B j: 's, thus (see [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF]Tseng and Yun 2009a) f + λ • 2,1 can be minimized through coordinate descent in S and the B j: 's (on the feasible set).

Hence, for our alternate minimization implementation, we only need to consider solving problems with B or S fixed, which we detail in the next propositions. Let us first defined the clipped square root, which is used in the minimization in S (Proposition 5.12). Definition 5.11 (Clipped Square Root). For Σ ∈ S n + with eigenvalue decomposition Σ = U diag(γ 2 1 , . . . , γ 2 n )U (U is orthogonal), let us define the Clipped Square Root operator:

(5.32) Proposition 5.12 (Minimization in S). Let B ∈ R n×T be fixed. The minimization of f (B, S) with respect to S with the constraint S σ Id n admits the closed-form solution:

(5.33)

Proof Minimizing f (B, •) amounts to solving arg min l) ] .

(5.34)

The solution is ClSqrt ZZ , σ (see Massias et al. 2018a, Appendix A2), with ZZ = 1 r r l=1 Z (l) Z (l) .

CHAPTER 5. APPLICATION TO NEUROIMAGING

, a summary of the updates in S/Σ and B j: 's for each algorithm is given in Table 5.2.

Comments on Table 5.2 The updates in S/Σ and B j: 's are given in Table 5.2. Although the updates may look similar, all the algorithms can lead to very different results, see Figures 5.6, 5.8, 5.10 and 5.12.

no update in S/Σ

Synthetic data

Here we demonstrate the ability of our estimator to recover the support i.e., the ability to identify the predictive features. There are n = 150 observations, p = 500 features, T = 100 tasks. The design X is random with Toeplitz-correlated features with parameter ρ X = 0.6 (correlation between X :i and X :j is ρ |i-j| X ), and its columns have unit Euclidean norm. The true coefficient B * has 30 non-zeros rows whose entries are independent and normally centered distributed. S * is a Toeplitz matrix with parameter ρ S . The SNR is fixed and constant across all repetitions SNR XB * / √ r XB * -Ȳ .

(5.46)

For Figures 5.1 to 5.3, the figure of merit is the ROC curve, i.e., the true positive rate (TPR) against the false positive rate (FPR). For each estimator, the ROC curve is obtained by varying the value of the regularization parameter λ on a geometric grid of 160 points, from λ max (specific to each algorithm) to λ min , the latter also being estimator specific and chosen to obtain a FPR larger than 0.4.

Influence of noise structure. Figure 5.1 represents the ROC curves for different values of ρ S . As ρ S increases, the noise becomes more and more correlated. From left to right, the performance of CLaR, SGCL, MRCER, 2,1 -MRCE, and 2,1 -MLER increases as they are designed to exploit correlations in the noise, while the performance of MTL decreases, as its i.i.d. Gaussian noise model becomes less and less valid.

Influence of SNR. On Figure 5.2 we can see that when the SNR is high (left), all estimators (except 2,1 -MLE) reach the (0, 1) point. This means that for each algorithm (except 2,1 -MLE), there exists a λ such that the estimated support is exactly the true CHAPTER 6. HYPERPARAMETER OPTIMIZATION

// compute the value and the gradient L(λ), ∇L(λ) ← Algorithm 6.5(X, y, λ, i ) • Forward-mode differentiation of proximal coordinate descent (Algorithm 6.3).

• Reverse-mode differentiation of proximal coordinate descent (Algorithm 6.4).

• cvxpylayers [START_REF] Agrawal | Differentiable convex optimization layers[END_REF], a software based on cvxpy [START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF], solving disciplined parametrized programming and providing derivatives with respect to the parameters of the program. It is thus possible to use cvxpylayers to compute gradients with respect to the regularization parameters.

Figure 6.1 compares the time taken by multiple methods to compute a single hypergradient ∇L(λ) for the Lasso (see Table 6.1), for multiple values of λ. It shows the time taken to compute the regression coefficients and the hypergradient, as a function of the number of columns, sampled from the design matrix from the gina dataset. The columns were selected at random and 10 repetitions were performed for each point of the curves. In order to aim for good numerical precision, problems were solved up to a reached by SMBO and random techniques is as good as first-order techniques. This is expected and follows the same conclusion as [START_REF] Bergstra | Random search for hyper-parameter optimization[END_REF]; [START_REF] Frazier | A tutorial on Bayesian optimization[END_REF]: when the number of hyperparameters is moderate, SMBO and random techniques can be used efficiently. However, when the number of hyperparameters increases (rcv1, q = 53 and aloi, q = 1000), the hyperparameter space is too large: zero-order solvers simply fail. On the contrary, first-order techniques manage to find hyperparameters leading to significantly better accuracy. (q = 10 classes), rcv1 (q = 53 classes), aloi (q = 1000 classes).

Conclusion

In this work we considered the problem of hyperparameter optimization to select the regularization parameter of linear models with nonsmooth objective. Casting this problem as a bilevel optimization problem, we proposed to use first-order methods. We showed that the usual automatic differentiation techniques, implicit differentiation, forward and reverse modes, can be used to compute the hypergradient, despite the non-smoothness of the inner problem. Experimentally, we showed the interest of first-order techniques to solve bilevel optimization on a wide range of estimators ( 1 penalized methods, SVM, etc.) and datasets. The presented techniques could also be extended to other criteria [START_REF] Lounici | Muddling labels for regularization, a novel approach to generalization[END_REF] or to more general bilevel optimization problems, in particular implicit differentiation could be well suited for meta-learning problems, with a potentially large number of hyperparameters [START_REF] Franceschi | Bilevel programming for hyperparameter optimization and meta-learning[END_REF]. Appendix 6.A Proof of the local linear convergence

6.A.1 Local linear convergence

We now detail the following result: an asymptotic vector autoregressive sequence, with an error term vanishing linearly to 0, converges linearly to its limit. In a more formal way:

) k∈N be a sequence of R p such that: (6.19) with ( (k) ) k∈N a sequence which converges linearly to 0, then (J (k) ) k∈N converges linearly to its limit Ĵ (Id -A) -1 b.

Proof Assume ( (k) ) k∈N converges linearly. Then, there exists c 1 > 0, 0 < ν < 1 such that:

Applying a standard result on spectral norms (see [START_REF] Polyak | Introduction to optimization. optimization software[END_REF], Chapter 2, Lemma 1) yields a bound on A k 2 . More precisely, for every δ > 0 there is a constant c 2 (δ) = c 2 such that

Without loss of generality, we consider from now on a choice of δ such that ρ(A)+δ < 1.

Since Ĵ = (Id -A) -1 b the limit Ĵ of the sequence satisfies: (6.20) Taking the difference between Equations (6.19) and (6.20) yields:

Unrolling Equation (6.21) yields -k ) . Taking the norm on both sides and using the triangle inequality leads to

HYPERPARAMETER OPTIMIZATION

We can now split the last summand in two parts and obtain the following bound, reminding that ρ(A) + δ < 1:

Thus, (J (k) ) k∈N converges linearly towards its limit Ĵ .

Theorem 6.12 (Local linear convergence of the Jacobian). Let 0 < γ ≤ 1/L. Suppose Assumptions 6.2, 6.3 and 6.6 hold. Let λ ∈ R r , Λ be a neighborhood of λ, and Γ Λ β(λ) -γ∇f ( β(λ) ) : λ ∈ Λ . In addition, suppose hypotheses (H1) to (H4) from Theorem 6.9 are satisfied and the sequence (β (k) ) k∈N generated by Algorithm 6.1 (respectively by Algorithm 6.3) converges toward β.

Then, the sequence of Jacobians (J (k) ) k≥0 generated by the forward-mode differentiation of proximal gradient descent (Algorithm 6.1) (respectively by forward-mode differentiation of proximal coordinate descent, Algorithm 6.3) converges locally linearly towards Ĵ .

Proof We first prove Theorem 6.12 for proximal gradient descent.

Proximal gradient descent case. Solving Problem (6.1) with proximal gradient descent leads to the following updates:

) . (6.22)

Consider the following sequence (J (k) ) k∈N defined by: k) ) . (6.23)

Note that if prox γg(•,λ) is not differentiable with respect to the first variable at z (k) (respectively with respect to the second variable λ), any weak Jacobian can be used. When (H3) holds, differentiating Equation (6.22) with respect to λ yields exactly Equation (6.23).

Assumptions 6.2 to 6.4 and 6.6 and the convergence of (β (k) ) toward β ensure proximal gradient descent algorithm has finite identification property (Liang et al., 2014, Thm. 3.1): we note K the iteration when identification is achieved. As before, the separability of g, Assumptions 6.2 to 6.4 and 6.6 ensure (see Theorems 2.13 and 2.19) ∂ z prox γg(•,λ) (z k ) Ŝc = 0, for all k ≥ K. Thus, for all k ≥ K, J (k) Ŝc :

= Ĵ Ŝc : = ∂ λ prox γg(•,λ) (z (k) ) Ŝc : .

6.A. PROOF OF THE LOCAL LINEAR CONVERGENCE