Albert Cohen

Charles Dossal

Yves Meyer Je

Nicolas Merci À Hugo

Fang, Sira Giacomo Mohammad

FX Jean-Marie Laurent

Keywords: inverse problem, variational regularization, low complexity prior, sparsity, robustness, sensitivity, risk estimation, degrees of freedom, parameter selection, partly smooth function 2 Robust Sparse Analysis Regularization x Contents 10.3 Robust Antisparse Regularization 10.4 Efficient GSURE Computation

This thesis is concerned with recovery guarantees and sensitivity analysis of variational regularization for noisy linear inverse problems. This is cast as a convex optimization problem by combining a data fidelity and a regularizing functional promoting solutions conforming to some notion of low complexity related to their non-smoothness points. Our approach, based on partial smoothness, handles a variety of regularizers including analysis/structured sparsity, antisparsity and low-rank structure. We first give an analysis of the noise robustness guarantees, both in terms of the distance of the recovered solutions to the original object, as well as the stability of the promoted model space. We then turn to sensivity analysis of these optimization problems to observation perturbations. With random observations, we build unbiased estimator of the risk which provides a parameter selection scheme.

Résumé

Cette thèse se consacre aux garanties de reconstruction et de l'analyse de sensibilité de régularisation variationnelle pour des problèmes inverses linéaires bruités. Il s'agit d'un problème d'optimisation convexe combinant un terme d'attache aux données et un terme de régularisation promouvant des solutions vivant dans un espace dit de faible complexité. Notre approche, basée sur la notion de fonctions partiellement lisses, permet l'étude d'une grande variété de régularisations comme par exemple la parcimonie de type analyse ou structurée, l'antiparcimonie et la structure de faible rang. Nous analysons tout d'abord la robustesse au bruit, à la fois en termes de distance entre les solutions et l'objet original, ainsi que la stabilité de l'espace modèle promu. Ensuite, nous étudions la stabilité de ces problèmes d'optimisation à des perturbations des observations. À partir d'observations aléatoires, nous construisons un estimateur non biaisé du risque afin d'obtenir un schéma de sélection de paramètre.

Mots-clés : problème inverse, régularisation variationnelle, a priori de faible complexité, parcimonie, robustesse, sensibilité, estimation du risque, degrés de liberté, sélection de paramètre, fonction partiellement lisse.

Inverse Problems and Regularization

Consider the following challenges:

• You are given an image where half of the sensors in your CCD camera are defective ! Could one recover the original image up to a given accuracy ?

• You work for a major entertainment company which is willing to build a recommender system to provide recommendations on movies based on the user's preferences. However, the data is quite incomplete since users typically rate only a few movies in the database. Could one infer the preference of any user for any movie, including the unrated ones ?

• You were recording the best performance with your rock band. Unfortunately, someone near the microphone was talking during the recording.

Can you remove the voice of this uncivil ?

• You want to build a search engine for large-scale images, whose goal is to retrieve images based on a semantic query. Can one build efficient compact descriptors/features on which efficient retrieval can be based ?

Several strategies have been proposed in the past decades to solve these problems (image inpainting, matrix completion, source separation, large-scale nearest neighbor search). All these problems can be cast in the same framework, where one has access to recover an object of interest (signal, image, video, matrix, etc.) while only partial, indirect and possibly imperfect information of it is available. To handle this class of problems within the same setting, we hinge on the following triad:

(i) Forward model : One has to model the degradation process underlying the incomplete and corrupted observations. Throughout this thesis, we consider the case of linear forward models where both the original object and the observations live in finite-dimensional vector spaces.

(ii) A priori : While recovering a vector from an underdetermined system of linear equations seems hopeless by basic arguments, the situation radically changes if some information is available in the form of a prior.

Here, we consider a variational formulation of this prior encoded into a convex function. More precisely, we focus on functions promoting low complexity objects, for instance piecewise constant, sparse or low rank.

(iii) Computational algorithm : In practice, it is necessary to be able to compute quickly a solution of a convex optimization problem casted as a trade-off between data fidelity (item (i)) and prior (item (ii)), hopefully unique, up to a good accuracy. It is thus important to propose an efficient algorithm, which is the case of the majority of the regularization considered here, using the structure of the problem.

Forward Model

This thesis is concerned with linear inverse problems in finite dimension. This framework is used in many applications in the fields of signal processing, imaging sciences, statistics and machine learning. Although one may object that this does not always conform to real world applications, where the corresponding objects may be infinite-dimensional or even continuous, our setting is sufficiently large to covers a wide spectrum of problems and practical applications in imaging or statistics. It also lends to a unified, generic and rigorous mathematical analysis.

We model physically the observed data with functions defined on a subspace Ω ⊆ R d , where d = 1 for an audio signal, d = 2 for an image, etc. Let us take the example of images. Intrinsically, a physical image is the projection of an object on an optical system. Thus, the image is a function f 0 defined by the quantity of energy f 0 (v) received by the focal plane at the point v, defining a function Ω → R, where Ω corresponds to a sub-domain of the focal plane.

From a mathematical point of view, one assumes that f 0 belongs to some functional space H. Typically, we consider f 0 as a finite energy function, i.e. H = L 2 (Ω).

In order to take into account properties of these signals or images (smooth, piecewise smooth, oscillating) other richer functional spaces are used. For instance, one can consider the space of functions with bounded variation, a Besov or Sobolev space. Sometimes, it is more meaningful to consider f 0 as a distribution. One may think for instances of point sources in an astronomical image, e.g. stars, which can be seen as a sum of Dirac masses. Another alternative, which is not considered in this thesis, is to place a random model on the signals, which corresponds to the Bayesian approach.

In many modern digital systems, the physical quantity (light) available at the focal plane, is directly sampled on a discrete cartesian grid (by construction on CCD or CMOS camera), hence giving directly equi-spaced samples y ∈ R q of the acquired scene. A general forward model relating the original image f 0 to the observations reads

y = Ψ(f 0 ) ⊙ b, (1.1)
where Ψ is degradation operator from the signal space H to the observation space R q , b is a noise term and ⊙ is some composition operator between the degraded data Ψ(f 0 ) and the noise. Typically, this composition is additive or multiplicative depending on the nature of the acquisition device. The operator Ψ model the acquisition device (digital camera, scanner, etc) and typically entails some sort of degradation and loss of resolution (blurring, missing pixels, etc). The noise term b may originate from several causes. It models the fluctuations (deterministic or random) that contaminate the observations (such as thermal noise).

In the overwhelming majority of applications in image and signal processing, the forward operator Ψ is considered as linear, either exactly or to a good approximation, see e.g. [START_REF] Mallat | A wavelet tour of signal processing[END_REF]. Thus, we leave aside the case of nonlinear observations, such as the magnitude of complex measurements, for instance Fourier in interferometric or diffraction imaging [START_REF] Hofmann | Iterative image reconstruction from the bispectrum[END_REF]. Moreover, the noise is considered additive in many cases, so that the forward model (1.1) reduces to the following:

y = Ψf 0 + b.
(1.2)

In practice, the goal of recovering a continuous function f 0 is in many cases hopeless a numerical point of view. Our goal is thus to find a discrete approximation of this function. To achieve that, we set some basis B(L) of a subspace L of H with dimension n, for instance taking finite elements, e.g. piecewise constants on a square grid or piecewise affine on a triangulation. Thus, we obtain an approximation Bx 0 of our original signal f 0 , where x 0 ∈ R n are the coefficients of f 0 in the basis B(L) and B is the matrix whose columns are the atoms of the basis. It leads us to consider the following forward model (cf. Figure 1.1):

y = Φx 0 + w (1.3)
where Φ = ΨB : R n → R q and w = b + Ψ(f 0 -Bx 0 ).

(1. 4) In general, the observation domain R q and the computational one R n are different (q = n). Indeed, q is dictated by the acquisition device, whereas n is a choice made by the numerical user, resulting from a trade-off between computational cost, precision and theoretical limit. This is the forward model that we will consider throughout this manuscript. Typically, Φ is not invertible, or badly conditioned. Beyond signal processing, the linear model is also used in statistics and machine learning under the name of regression. One can find its history in the paper of [START_REF] Seal | Studies in the History of Probability and Statistics. XV The historical development of the Gauss linear model[END_REF]. From now on, we focus on the problem of recovering Bx 0 . This thus corresponds to a finite dimensional problem: finding a good approximation of x 0 from the observation y alone. The behavior when the grid size tends to zero raises many important and difficult issues, which will be not treated here.

For some problem, it is important to take in account the random nature of the noise, and thus to consider the stochastic forward model

Y = Φx 0 + W, (1.5)
where the noise W is a random vector with realizations taking values in R q . Supposing that the noise follows a centered Gaussian density, W ∼ N(0, σ 2 Id), we obviously have Y ∼ N(Φx 0 , σ 2 Id). This classical model is studied in details in [START_REF] Trevor | The elements of statistical learning[END_REF]. Others noise models are considered in image processing, such as Poisson noise for short noise (e.g. CCD cameras, computerized tomography), and multiplicative noise in SAR imaging. We refer to [START_REF] Refregier | Statistical Image Processing techniques for Noisy Images -An application Oriented Approach[END_REF]) and [START_REF] Boncelet | Handbook of Image and Video Processing[END_REF] for a more comprehensive account on noise models in imaging systems.

When no noise corrupts the data, which is hardly the case for real life applications, the forward model becomes y = Φx 0 .

(1.6)

We now list some classical examples of the forward operator Φ used in image processing.

Denoising. The denoising problem is among the most intensively studied in the image processing literature. This step may prove crucial prior to more high-level image analysis and processing tasks, e.g. object segmentation or detection. The model (1.3) thus reduces to y = x 0 + w.

In other words, the operator Φ is nothing more than the identity Φ = Id.

Deconvolution. In the case of photography, we observe a blur when the camera is not adequately stabilized (motion blur), but also a blur due to the point spread function (PSF) of the acquisition system. A reasonable approximation allows to model this degradation as a convolution operator, i.e. Φx = K Φ ⋆ x, where K Φ is the blurring kernel. In particular, the high frequency content of x 0 may be seriously damaged. An important property of the convolution is the fact that it is shift invariant. Estimating both x 0 and K Φ , a.k.a blinddeconvolution, is a difficult problem, but we are solely here concerned with the case where K Φ is known. The deconvolution procedure is popular in many fields in science and engineering [START_REF] Biemond | Iterative methods for image deblurring[END_REF], for instance in astronomy [START_REF] Starck | Deconvolution in astronomy: A review[END_REF], in geophysics [START_REF] Santosa | Linear inversion of band-limited reflection seismograms[END_REF] or microscopy [START_REF] Agard | Optical sectioning microscopy: cellular architecture in three dimensions[END_REF].

Inpainting. In presence of occlusion or damages pixels, the inpainting procedure aims at recovering such parts. In this case, Φ is a binary diagonal operator such that Φ ii is 1 if the data are preserved, 0 otherwise. This operator can be deterministic, or the realization of a random mask. Inpainting is commonly used in many applications, such as medical fluroscopy [START_REF] Chan | Image sequence filtering in quantum-limited noise with applications to low-dose fluoroscopy[END_REF], in colorization [START_REF] Sapiro | Inpainting the colors[END_REF] or in data compression [START_REF] Liu | Image compression with edge-based inpainting[END_REF]).

Compressed Sensing. The conventional wisdom in signal processing is that for a continuous band-limited signal to be reconstructed perfectly from its equi-spaced samples, it has to be acquired at a frequency at least twice its bandwidth; this is the celebrated [START_REF] Shannon | A Mathematical Theory of Communication[END_REF][START_REF] Nyquist | Certain topics in telegraph transmission theory[END_REF] theorem. This theory however precludes many signals of interests that are not bandlimited, but whose intrinsic dimension is small, think for instance of a sparse signal, or of a smooth signal away from a few singularities. The compressed sensing theory (Candès et al. 2006a;[START_REF] Donoho | Nonlinear solution of linear inverse problems by waveletvaguelette decomposition[END_REF] asserts that for such signals, exact and stable reconstruction is possible, hence allowing to break the Shannon-Nyquist limit. The reconstruction is moreover achieved by solving a computationally tractable convex optimization problem. The sampling operator can be modeled with a matrix Φ which is the realization of an appropriate random ensemble, such i.i.d. Gaussian or Bernoulli entries, or partial random Fourier or Hadamard matrices. The corresponding inverse problem can be shown to be efficiently regularized by some popular low complexity priors discussed in the next section. This theory has sparkled a whole research field, and hardware proofs of concept have been also developed. The first one is the single pixel camera [START_REF] Wakin | An architecture for compressive imaging[END_REF] at Rice University, which measures directly random projections on a single CCD element with a binary reflector composed of micro-mirrors. Figure 1.2 illustrates this process. Compressed sensing has been also used for Dirac train recovery in ultrasonic imagery [START_REF] Tur | Innovation rate sampling of pulse streams with application to ultrasound imaging[END_REF]. Introducing a partial randomization of the measurements [START_REF] Lustig | Sparse MRI: The application of compressed sensing for rapid MR imaging[END_REF]) is also promising in medical imaging applications such as fMRI. Compressed sensing is also used in astronomy [START_REF] Bobin | Compressed sensing in astronomy[END_REF], in particular on the telescope Herschel.

Tomography. Tomography is commonly used in medical imaging [START_REF] Newton | Radiology of the Skull and Brain: Technical aspects of computed tomography[END_REF]. Popular CT scanners are X-ray and PET modalities. In this case, the operator Φ is a discrete Radon transform [START_REF] Herman | Fundamentals of computerized tomography: image reconstruction from projections[END_REF], possibly with a sub-sampling to model incomplete measurements. In practice, only a few measurements can be collected, leading to an increase of the ill-posedness of the (continuous) forward operator Ψ.

Variational Regularizations

Solving an inverse problem from the observations (1.3) corresponds to computing an approximation of x 0 from the knowledge of y alone. This problem is said to be well posed (in the sense of [START_REF] Hadamard | Sur les problèmes aux dérivées partielles et leur signification physique[END_REF]) in a space S if Φx = y has x 0 as unique solution on S, and if this solution depends continuously on y. This means that one recovers exactly x = x 0 when there is no noise, and a good approximation if w is small. In general, the matrix Φ is rank deficient or ill-conditionned, so that the problem is not well posed on the whole space S = R n . In order to recover well-posedness it is thus necessary to restrict the inversion process to a well-chosen space S that includes x 0 . A closely related procedure, that we describe next, is to set-up a variational inversion process which is penalized by a well-chosen prior.

A first line of works has considered imposing a random model on the signal x 0 . This corresponds to the Bayesian formalism, see for instance the monograph [START_REF] Hunt | Improving upon standard estimators in discrete exponential families with applications to Poisson and negative binomial cases[END_REF] for an introduction to these methods. We do not explore these strategies in this thesis. We rather directly impose some prior on the (deterministic) x 0 through some penalty function J. This corresponds to the usual notion of variational regularization, which was initially introduced in [START_REF] Tikhonov | Solutions of ill-posed problems[END_REF] as a way to recover well-posedness of the inverse problem under investigation.

Within this framework, the computation of an approximation x ⋆ to x 0 is obtained by solving the following optimization problem

x ⋆ ∈ Argmin x∈R n
F(x, y) + λJ(x).

(P F y,λ )

Here, F : R n × R q → R + is a data fidelity term. Typically, it is a smooth nonnegative convex function. Thus, we expect that F is small when the prediction Φx is close enough from y. The factorization F(x, y) = F 0 (Φx, y) is commonly used, where F 0 : R q × R q → R is smooth, non-negative and strongly convex. This data fidelity term can be the quadratic loss, the Poisson antilog likelyhood or the logistic loss. Statistically, one may interpret (P F y,λ ) as a Maximum A Posteriori (MAP). This interpretation can however be misleading, as exemplified in [START_REF] Gribonval | Should penalized least squares regression be interpreted as Maximum A Posteriori estimation?[END_REF], where failures of the MAP approach are analyzed for sparse distributions.

The function J : R n → R + is the regularization term imposing some prior on the signal class. We assume in this thesis that J is a convex function. Convexity is important to ensure the ability to compute global optima of (P F y,λ ) with fast algorithms, and also enables a fine theoretical analysis of the properties of x ⋆ . It is however important to realize that non-convex penalties, as well as nonvariational methods (e.g. greedy algorithms) are routinely used and often outperform their convex counterparts. This is however beyond the scope of this thesis, and we focus here on convex regularizers. Section 1.1.4 details the basic properties of these regularizers and sketch some important examples.

The scalar λ is the regularization parameter (or hyper-parameter) allowing a trade-off between fidelity and regularization. The choice of λ is an important question, which is treated in the second part of this thesis, and discussed in Section 1.3.

Since Φ is generally not injective, note that the objective function of the problem (P F y,λ ) is not stricly convex. Thus, it may admits several solutions. It is also possible to use the constrained version, in opposition to (P F y,λ ) qualified as penalized or Lagrangian form, coined as the Ivanov form in the inverse problem litterature [START_REF] Ivanov | Theory of Ill-Posed Linear Problems and Its Applications[END_REF]. It reads

x ⋆ ∈ Argmin x∈R n
J(x) subject to F(x, y) ε.

( PF y,ε )

We mainly focus on the Lagrangian version in this dissertation. However, problems ( PF y,ε ) and (P F y,λ ) are equivalent in some sense [START_REF] Ivanov | Theory of Ill-Posed Linear Problems and Its Applications[END_REF][START_REF] Poljak | Introduction to optimization[END_REF]), but one may take care that the mapping between ε and λ is generally not explicit. Some recent work [START_REF] Ciak | Homogeneous penalizers and constraints in convex image restoration[END_REF] in this direction exploit the Fenchel-Rockafellar duality to overcome this difficulty in some particular cases.

When there is no noise, i.e. when the observations follow (1.6), we consider the constrained version of (P y,0 ) which reads

x ⋆ ∈ Argmin x∈R n J(x) subject to Φx = y.

(P y,0 )

As it will be proved formally in Chapter 5 (more precisely Proposition 5.2), problems (P y,λ ) and ( PF y,ε ) converge (in an appropriate sense) to (P y,0 ).

Data Fidelity

The data fidelity is linked to the forward model (1.3). We find in the statistical literature several data fidelity term for the problem (P F y,λ ). Note that many of them does not assume that the forward model is of the form (1.3). This will discussed in details in Chapter 8. One naturally thinks to generalized linear models (GLMs) introduced by [START_REF] Nelder | Generalized linear models[END_REF] which assume that conditionally on Φ, Y i are independent with distribution that belongs to a given (one-parameter) standard exponential family. Well-known examples are Gaussian distribution (linear model), the reciprocal link (Gamma and exponential distributions), and the logit link (Bernoulli distribution, logistic regression).

When the noise is the realization of a white Gaussian noise, it is common to use the quadratic loss as a data fidelity term.

F 0 (µ, y) = 1 2 ||µ -y|| 2 2 .
The functional F 0 can also be chosen for instance as the logistic loss

F 0 (µ, y) = q i=1
log (1 + exp(µ i ))y, µ , or Huber loss (smoothed) or a ℓ p loss. Note that ℓ p loss is not smooth for p < 1. From a deterministic point of view, F 0 can be chosen from the prior on the noise in the continuous case (for instance a noise in a Banach space) or in the discrete setting considered, as a prior of ℓ p -boundness.

In the case where F 0 is the quadratic loss, the problem (P F y,λ ) reads

x ⋆ ∈ Argmin (P y,λ )

This variational formulation is at the core of the first part of this thesis.

Low Complexity Priors

1.1. 4

.1 Combinatorial Model Selections

Penalizing in accordance to some notion of complexity is a key idea, whose roots can be traced back for instance to the statistics literature [START_REF] Mallows | Some Comments on C p[END_REF][START_REF] Akaike | Information theory and an extension of the maximum likelihood principle[END_REF]). This complexity is measured using a functional pen(T ) where T is some linear subspace containing x, and chosen among a fixed collection of spaces T. This approach typically makes use of hierarchy of models of increasing complexity, which should be designed in accordance to some prior knowledge about the data x 0 to recover. A union of linear models is a collection T of subspaces of R n which is usually finite but very large, in the case of finite dimensional problems. These subspaces typically account for some kind of smoothness or simplicity of the signal. A key example is sparsity, which, in its simplest form, corresponds to a problem of selecting few of active variables in the data. In this setting, a subspace T has the form T = {x | supp(x) = I} for some set of indexes I indicating the active variables. With such a set of model at hand, one can use the following prior

J(x) = inf
x∈T pen(T ).

(1.7)

The problem (P y,λ ) can be recast as a model selection problem inf T ∈T,x∈T ||y -Φx|| 2 + λpen(T ).

The model selection literature [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF][START_REF] Barron | Risk bounds for model selection via penalization[END_REF][START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF] proposes many significant results to quantify the performance of these approaches. A major bottleneck of this class of approaches is that the corresponding J function defined in (1.7) is non-convex, thus typically leading to intractable, often NP-hard problems. For instance, the sparsity of coefficients x ∈ R n is measured using the ℓ 0 pseudo-norm

J 0 (x) = ||x|| 0 = | supp(x)|.
Minimizing (P y,λ ) or (P y,0 ) with J = J 0 is known to be NP-hard, see for instance [START_REF] Natarajan | Sparse approximate solutions to linear systems[END_REF]. There is a wide variety of approaches to tackles directly non-convex optimization problems. A line of research considers greedy algorithms. The most popular ones are Matching Pursuit [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF] and Orthogonal Matching Pursuit [START_REF] Pati | Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition[END_REF][START_REF] Davis | Adaptive Time-Frequency Approximations with Matching Pursuits[END_REF], see also the comprehensive reviews [START_REF] Needell | Greedy signal recovery review[END_REF] and references therein. Another line of research, which is the one under study in this thesis, consists in considering convexified versions of (1.7).

Convex Encoding of Models

For any subspace T of a real vector space E, we denote P T the orthogonal vector on T , x T = P T (x) and Φ T = Φ P T . We now introduce the model tangent subspace at a point x for some finite-valued convex functional J.

D efi ni ti on 1 . 1 -M odel Tangent Subspace For any vector x ∈ R N , we denote e x its model vector,

e x = argmin e∈aff ∂J(x)
||e||,

where aff ∂J(x) is the affine hull of the subdifferential (see Definition 2.12) of J at x, and T x = span(∂J(x)) ⊥ .

T

x is coined the model tangent subspace of x associated to J.

This terminology will be clear after we define partly smooth function in Section 1.2.2. When J is Gâteaux-differentiable at x, i.e. ∂J(x) = {∇J(x)}, e x = ∇J(x) and T x = R N . On the contrary, when J is not smooth at x, the dimension of T x is of smaller dimension, and the regularizing functional J essentially promotes elements living on or close to the affine space x + T x . Table 1.1 exemplifies Definition 1.1 on several regularizers that are popular in the literature. The details of the exact derivations is provided in Chapter 3.

Comment

Sparsity

A dictionary D = (d i ) p i=1 is a (possibly redundant, i.e. when p > n) collection of p atoms d i ∈ R n . It can also be viewed as a linear mapping from R p to R n Chapter 1 Introduction which is used to synthesize a signal x ∈ Im(D) ⊆ R n as

x = Dα = p i=1 α i d i ,
where α is the coefficient vector that synthesizes x from the dictionary D. Note that if D is redundant, there is an infinite number of coefficients α such that x = Dα. An issue beyond our work is to build a good dictionary. We may cite the wavelet transform [START_REF] Mallat | A theory for multiresolution signal decomposition: the wavelet representation[END_REF]) and the curvelet transform [START_REF] Candès | Curvelets: A surprisingly effective nonadaptive representation for objects with edges[END_REF] for images that are piecewise smooth away from smooth edge curves, local Fourier basis for sounds [START_REF] Allen | Short-term spectral analysis, and modification by discrete Fourier Transform[END_REF], or union of dictionaries for image and signal decomposition, see for instance cartoon+texture decomposion in [START_REF] Elad | Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA)[END_REF].

Synthesis sparsity. When considering sparsity in the canonical basis, i.e. D = Id, the model subspace and model vector read

T x = x ′ | supp(x ′ ) = supp(x)
and e x = sign(x).

Looking for the sparsest representation of x in the dictionary D amounts to solving min α∈R p ||α|| 0 subject to x = Dα.

Replacing the ℓ 0 norm by the ℓ 1 norm leads to a convex problem. The sparsest set of coefficients, according to the ℓ 1 norm, defines a signal prior which is the image of ||.|| 1 under D, J S (x) = min α∈R p ||α|| 1 subject to x = Dα.

Therefore any solution x of (P y,λ ) using J = J S can be written as x = Dα where α is a solution of min [START_REF] Vaiter | Robust Polyhedral Regularization[END_REF] was introduced in the statistical community in [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] where it was coined Lasso. Note that it was originally introduced as an ℓ 1 -ball constrained optimization and in the over-determined case. It is also known in the signal processing community as Basis Pursuit DeNoising [START_REF] Chen | Atomic decomposition by basis pursuit[END_REF]. Such a problem corresponds to the so-called sparse synthesis regularization, as sparsity is assumed on the coefficients α that synthesize the signal x = Dα. In the noiseless case, the constrained problem (P y,0 ) becomes

α∈R p 1 2 ||y -ΦDα|| 2 Note that || • || 1 is the convexification of || • || 0 restricted to the ℓ ∞ -ball. Prob- lem (1.
min α∈R p ||α|| 1 subject to y = ΦDα, (1.9)
which goes by the name of Basis Pursuit after [START_REF] Chen | Atomic decomposition by basis pursuit[END_REF].

Sparse regularization is a popular class of priors to model natural signals and images, see for instance [START_REF] Mallat | A wavelet tour of signal processing[END_REF]. The idea of ℓ 1 regularization finds its root in the seismic imaging literature [START_REF] Santosa | Linear inversion of band-limited reflection seismograms[END_REF] for deconvolution. It is also used in many applications, see [START_REF] Starck | Sparse image and signal processing: wavelets, curvelets, morphological diversity[END_REF]) for a comprehensive account.

A key problem of active research is to learn and optimize the dictionary in order to represent optimally a set of given exemplar. We refer to the book of (Elad 2010, Chapter 12) for a recent overview of the relevant literature.

Analysis sparsity. Analysis regularization corresponds to using J = J A in (P y,λ ) where

J A (x) = ||D * x|| 1 = p i=1 | d i , x |,
It imposes the sparsity of the correlations ( d j , x ) j=1,...,p between x and the atoms in a dictionary D. In this case,

T x = x ′ | supp(D * x ′ ) = supp(D * x)
and e x = sign(D * x).

Note that synthesis and analysis regularizations are different as soon as D is not an invertible square matrix. Hence, (P y,λ ) reads

min x∈R n 1 2 ||y -Φx|| 2 + λ||D * x|| 1 .
(1.10)

In the noiseless case, the ℓ 1 -analysis equality-constrained problem is [START_REF] Deledalle | Proximal Splitting Derivatives for Risk Estimation[END_REF] In [START_REF] Nam | The cosparse analysis model and algorithms[END_REF], the term cosparse is used, motivated by the role played by the complement of the support (i.e. cosupport) of the vector D * x in the theoretical analysis of (1.11).

min x∈R n ||D * x|| 1 subject to Φx = y. (1.
The adjoint of any synthesis dictionary (see above) can be used to define analysis sparsity prior. Analysis sparsity allows for more intricate operators D * because D * is not required to be a stable frame of the signal space. One of the most popular is the finite difference operator used in the total variation seminorm, first introduced for denoising (in a continuous setting) by [START_REF] Rudin | Nonlinear total variation based noise removal algorithms[END_REF]. Typically, for 1-D discrete signals, D can be taken as a dictionary of forward finite differences D DIF where

D DIF =          +1 0 -1 +1 -1 . . . . . . +1 0 -1         
.

(1.12)

The corresponding prior J A favors piecewise constant signals and images. A comprehensive review of total variation regularization can be found in [START_REF] Chambolle | An Introduction to Total Variation for Image Analysis[END_REF]. One can also use a wavelet dictionary D which is shiftinvariant, such that the corresponding regularization J A can be see as a multiscale total variation in the case of the Haar wavelet [START_REF] Steidl | On the equivalence of soft wavelet shrinkage, total variation diffusion, total variation regularization, and SIDEs[END_REF]) for 1D signals. When using wavelets with m vanishing moment, the corresponding priors favors discrete piecewise polynomial signals of degree m. A numerical exploration of the relative performances of analysis and synthesis regularization is performed in [START_REF] Elad | Analysis versus synthesis in signal priors[END_REF]. [START_REF] Selesnick | Signal restoration with overcomplete wavelet transforms: comparison of analysis and synthesis priors[END_REF] report an extensive numerical exploration where they use shift invariant wavelet dictionaries to compare analysis and synthesis sparsity priors for several inverse problems (e.g. deconvolution and inpainting). As a last example of sparse analysis regularization, we would like to mention the Fused Lasso [START_REF] Tibshirani | Sparsity and smoothness via the fused Lasso[END_REF], where D is the concatenation of D DIF and a weighted identity. The corresponding prior J A promotes both sparsity of the signal and its derivative, hence favoring the grouping of non-zero coefficients in blocks over which the signal is constant.

Structured sparsity. To further improve the performance of sparse regularization, it is useful to group the coefficients, imposing the sparsity in a block-wise manner. It has been first proposed by [START_REF] Hall | Numerical performance of block thresholded wavelet estimators[END_REF][START_REF] Hall | Block threshold rules for curve estimation using kernel and wavelet methods[END_REF]; [START_REF] Cai | Adaptive wavelet estimation: a block thresholding and oracle inequality approach[END_REF] for wavelet block shrinkage. For over-determined regression of the form (1.3), it has been introduced by Bakin (1999); [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF]. Block regularization is popular in image processing because wavelet coefficients of a natural image have a group structure [START_REF] Mallat | A wavelet tour of signal processing[END_REF]. Indeed, edges and textures induce strong local dependencies between coefficients. In multi-task learning [START_REF] Obozinski | Joint covariate selection and joint subspace selection for multiple classification problems[END_REF], it is used to control the sparsity pattern of the covariates. In audio processing, it is also useful to deal with multi-channel data as studied by [START_REF] Gribonval | Atoms of all channels, unite! Average case analysis of multi-channel sparse recovery using greedy algorithms[END_REF] which is also known as the multiple measurements vector (MMV) model, see for instance [START_REF] Cotter | Sparse solutions to linear inverse problems with multiple measurement vectors[END_REF][START_REF] Chen | Theoretical results on sparse representations of multiple-measurement vectors[END_REF].

Suppose what we split the signal space R n into groups without overlapping. We formalize this splitting by a disjoint partition B of {1, . . . , n}, i.e. Then, we define the ℓ 1ℓ 2 norm as J = J B where

J B (x) = b∈B ||x b ||, (1.13)
where x b is a vector of size |b| containing the entries indexed by b. Thus, the model space and model vector reads

T x = x ′ | ∀b ∈ B, x b = 0 ⇒ x ′ b = 0 and e x = x b ||x b || b∈B ,
where we take the convention that if

x b = 0 then x b ||x b || = 0.
It is possible to replace the ℓ 2 norm with more general functionals, such as ℓ p norms for p > 1 [START_REF] Turlach | Simultaneous variable selection[END_REF][START_REF] Negahban | Simultaneous support recovery in high dimensions: Benefits and perils of block-regularization[END_REF][START_REF] Vogt | A Complete Analysis of the l 1,p Group-Lasso[END_REF] or to use analysis block sparsity

J B (x) = b∈B ||D * b x b ||,
where D * b are linear operators from R |b| → R p . For instance, one can express the 2D isotropic total variation by defining D * b x ∈ R 2 to be an approximation by finite differences of the gradient of the image x at the pixel indexed by b. This block analysis sparsity allows us also to take into account overlapping groups [START_REF] Jenatton | Structured variable selection with sparsity-inducing norms[END_REF][START_REF] Cai | Incorporating information on neighbouring coefficients into wavelet estimation[END_REF], or groups structured in a tree [START_REF] Peyré | Adaptive Structured Block Sparsity Via Dyadic Partitioning[END_REF][START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]).

Beyond Sparsity

While sparsity has become mainstream in imaging sciences and machine learning, there is now a flurry of activity to develop novel priors to take into account various types of low-dimensional structures to model the data.

Low rank prior. The natural extension of sparsity to matrices x ∈ R n 1 ×n 2 , where n = n 1 n 2 , is to impose a low rank constraint. This should be understood as imposing the sparsity of the singular values. Denoting x = V x diag(Λ x )U *

x a Singular Value Decomposition of x, where Λ x ∈ R m and m = min(n 1 , n 2 ). Hence the rank reads rank(x) = ||Λ x || 0 . Here, the natural models are not linear subspaces T x but manifolds of matrices with a fixed rank, see Chapter 4. The nuclear norm (or trace, 1-Schatten norm) imposes such a sparsity [START_REF] Fazel | Matrix Rank Minimization with Applications[END_REF] and is defined as

||x|| * = ||Λ x || 1 .
The nuclear norm is the convexification of the rank function with respect to the spectral norm ball, see [START_REF] Fazel | Matrix Rank Minimization with Applications[END_REF][START_REF] Hiriart-Urruty | Convexifying the set of matrices of bounded rank: applications to the quasiconvexification and convexification of the rank function[END_REF]. It has been used for instance to recover low rank matrices by [START_REF] Srebro | Learning with matrix factorizations[END_REF] (Netflix prize) or for model reduction in [START_REF] Fazel | A rank minimization heuristic with application to minimum order system approximation[END_REF].

Spread representation.

In some cases, one expects to recover flat vectors, i.e such that for most i, x i = ||x|| ∞ . A convex function promoting such behavior is the ℓ ∞ norm defined as

||x|| ∞ = max i∈{1,...,n} |x i |.
Such a prior is encoded in a linear model T which is defined w.r.t to the number of saturating coordinates. More precisely,

T x = x ′ | x ′ I = ρx I for some ρ ∈ R , where I = {i | x i = ||x|| ∞ }.
For applications in computer vision such as image retrieval in a database [START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF], it is useful to have a compact signature of signals, typically with only two values ±1. An approach proposed in [START_REF] Jégou | Anti-sparse coding for approximate nearest neighbor search[END_REF] for obtaining this binary quantification is to compute these vectors as spread approximations in a random dictionary. A study of this regularization is done in [START_REF] Fuchs | Spread representations[END_REF], where an homotopy-like algorithm is provided. Moreover, the use of ℓ ∞ regularization is connected to Kashin's representation [START_REF] Lyubarskii | Uncertainty principles and vector quantization[END_REF], which is known to be useful in stabilizing the quantization error for instance. Others applications such as wireless network optimization [START_REF] Studer | Signal Representations with Minimum ℓ ∞ -Norm[END_REF]) also rely on ℓ ∞ prior.

1.1.4.5 From Continuous to Discrete, and Vice Versa.

Even if we focus on finite dimensional problems, an important issue is to understand the link between these models with their continuous counterparts. The underlying mathematical problems, for instance the convergence of discrete models to the continuous ones, have practical implications in order to understand the fine structure of signals computed with these methods, which are generally easier to describe in a continuous setting (for instance edges in images). Two typical questions naturally arise:

(i) How to analyze the convergence of the function Bx ⋆ to f 0 when n tends to +∞, where x ⋆ is some solution of (P y,λ ) and B is the basis defined in (1.4) ? It is often treated by a control on both the estimation error ||x ⋆ -x 0 || and the approximation error ||f 0 -Bx 0 ||.

(ii) How to define a variational problem directly in the continuous setting ? It corresponds to replacing the function J par a function f → J(f) such that J(Bx ⋆ ) is "close enough" to J(x ⋆ ). In this case, we deal with the following optimization problem

f ⋆ ∈ argmin f∈H 1 2 ||y -Ψf|| 2 + λJ(f).
(cP y,λ )

Note that the choice of H should be chosen in accordance to the functional J.

Wavelet sparsity and Besov spaces. Let W(f) ∈ R N be the wavelet transform of f ∈ H in an orthogonal basis introduced by [START_REF] Mallat | A theory for multiresolution signal decomposition: the wavelet representation[END_REF]; [START_REF] Daubechies | Ten Lectures on Wavelets[END_REF]; [START_REF] Meyer | Wavelets and operators[END_REF]. Besov Banach spaces are defined using appropriate sparsity inducing norms of W(f). Besov spaces form an important class of Banach spaces since they are a powerful tool to model piecewise regular signals and image with pointwise singularities, see for instance [START_REF] Meyer | Wavelets and operators[END_REF][START_REF] Chambolle | Nonlinear wavelet image processing: Variational problems, compression, and noise removal through wavelet shrinkage[END_REF]. Wavelets are known to provide optimally sparse representations for functions in Besov spaces [START_REF] Mallat | A wavelet tour of signal processing[END_REF]).

These spaces have been widely used in the statistical community to establish minimaxity of wavelet-based estimators for several problems (e.g. regression, inverse problems, etc.), for instance to quantify the optimality of the soft thresholding for denoising [START_REF] Donoho | Ideal spatial adaptation by wavelet shrinkage[END_REF] or for the waveletvaguelettes [START_REF] Donoho | Nonlinear solution of linear inverse problems by waveletvaguelette decomposition[END_REF] method which corresponds to applying the soft thresholding to Ψ + (y). These results have been extended to group sparsity [START_REF] Hall | Block threshold rules for curve estimation using kernel and wavelet methods[END_REF], for instance in the denoising setting (Chesneau et al. 2010a) or deblurring (Chesneau et al. 2010b).

Sparsity and Radon measures. When we wish to recover highly localized signals, a convenient model is to use sum of Dirac distributions. The finite dimensional problem (P y,λ ), with J = || • || 1 and Φ = Id, is equivalent to considering that these distributions are on a fixed grid whereas (cP y,λ ) can be seen as its continuous (grid-free) counterpart. In this case, H is the space of finite Radon measures and J(f) is the total variation of the measure f which should not be confused with the total variation of a function.

Solving inverse problems on this space of measures has been recently considered by [START_REF] Bredies | Inverse problems in spaces of measures[END_REF] and a theoretical study of the performance is proposed by [START_REF] Candès | Towards a Mathematical Theory of Super-resolution[END_REF] for the case of deconvolution (super-resolution). The convergence of the solutions of (P y,λ ) to those of (cP y,λ ) is studied in [START_REF] Duval | Exact Support Recovery for Sparse Spikes Deconvolution[END_REF][START_REF] Tang | Compressive sensing off the grid[END_REF].

Sparse gradient and bounded variation functions. Analysis sparsity of the gradient (1.12) can be seen as a discretization by finite difference of the total variation of a function. More precisely, for f ∈ L 1 loc (Ω), we denote the total variation J as

J(f) = sup - Ω f div ψ | ψ ∈ C ∞ c (Ω, R n ), ∀x ∈ Ω, |ψ(x)| 1 . (1.14)
f has bounded variation if J(f) < +∞ and we denote BV(Ω) the Banach spaces of functions of bounded variations endowed with the norm

|| • || L 1 (Ω) + J(•). Remark that W 1,1 (Ω) is strictly included in BV(Ω). In fact, if f is C 1 then J(f) = ||∇f|| L 1 (Ω) .
A useful property of this space with respect to any Sobolev space is the fact that the problem (cP y,λ ) can admit non-continuous solutions when using H = BV(Ω). The denoising problem has been studied in [START_REF] Caselles | The Discontinuity Set of Solutions of the TV Denoising Problem and Some Extensions[END_REF] where the discontinuity set of its solution is characterized. We refer to [START_REF] Ambrosio | Functions of bounded variation and free discontinuity problems[END_REF] for a detailed study of this space. Note that higher order priors have been introduced recently, e.g the total generalized variation [START_REF] Bredies | Total generalized variation[END_REF].

Solving the Optimization Problem

Several algorithms exist in order to solve the problem (P y,λ ) or (P y,0 ), depending on the nature of J.

Solving (P y,λ ) corresponds to the minimization of a convex function f. When J is smooth, one can make use of traditional gradient or Newton descent schemes. However, the class of low-complexity regularizations J considered in this thesis are highly non-smooth. It is possible to adapt the gradient descent scheme when f is convex, lower semi-continuous and proper by replacing the descent direction grad f by any element of the subdifferential ∂f(x). This scheme is however quite inefficient for the penalties J considered in this thesis, which are highly structured. Making use of this structure is crucial to obtain fast algorithms.

For a large class of J regularizers, such as those introduced in this section (ℓ 1 , nuclear norm, total variation, etc), the optimization (P y,λ ) can be shown to be equivalent to a conic program. This cone constraint can be enforced using a self-concordant barrier function, and the optimization problem can hence be solved using interior point methods, as pioneered by [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF], see also the monograph [START_REF] Boyd | Convex optimization[END_REF]. This class of methods enjoys fast convergence rate. Each iteration however is typically quite costly. This class of solvers is a wise choice for problem of medium size, and when high accuracy is needed.

Homotopy methods have been introduced in the case of the sparsity J = || • || 1 by [START_REF] Osborne | A new approach to variable selection in least squares problems[END_REF], then adapted to the analysis sparsity J = ||D * • || 1 in [START_REF] Tibshirani | The solution path of the generalized lasso[END_REF]) and spread representations || • || ∞ in [START_REF] Fuchs | Spread representations[END_REF].

The LARS algorithm [START_REF] Efron | Least angle regression[END_REF]) is closely related and compute an approximation of the homotopy path with a faster algorithm. These methods rely on the behavior of λ → x ⋆ (λ), where x ⋆ (λ) is a solution of (P y,λ ). In the case of a polyhedral regularization, such as ℓ 1 or ℓ ∞ , this path turns out to be piecewise polygonal, see Chapter 8.

The cost per iteration of both interior point and homotopy methods scales badly with the dimension, thus preventing them to be used in large scale problems such as those encountered in imaging science. Proximal schemes are attractive alternatives, since they correspond to first order schemes whose iterations are in practice quite cheap. We refer to [START_REF] Beck | Gradient-based algorithms with applications to signal recovery[END_REF][START_REF] Bauschke | Convex analysis and monotone operator theory in Hilbert spaces[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF][START_REF] Parikh | Proximal algorithms[END_REF] for comprehensive reviews.

Their slow convergence rate is thus generally not a big issue in imaging or machine learning, where one typically does not seek for a high precision solution to the optimization problem (P y,λ ).

Robustness: Handling the Impact of Noise

Observations are in general contaminated by noise. It is thus important to study the robustness of (P y,λ ) to analyze its performance. More precisely, we aim to derive criteria quantifiying how x ⋆ is close to to x 0 . This notion of closeness will be analyzed mathematically through two quality criteria: error distance in the sense of the ℓ 2 norm and model selection.

Linear Convergence Rate

Here, we are seeking sufficient conditions under which any solution of (P y,λ ) satisfies

||x ⋆ -x 0 || = O(||w||).
It depends typically on x 0 , while λ should be chosen proportionally to the noise level for the linear convergence to hold. The terminology "linear" in the convergence rate, which stems from the inverse problem community, pertains to the fact that the error is within a factor of the noise level. This rate is made possible by the fact that J is not smooth. For instance, this is not the case for [START_REF] Scherzer | Variational methods in imaging[END_REF].

|| • || 2 2 , see

Dual Certificate and Non-Degeneracy

We introduce the notion of dual certificate which characterizes the solutions of the noiseless problem (P 0 (Φx 0 )). This notion is a key ingredient of our analysis in the sequel.

Defi ni ti on 1 .2 -D ual C erti fi cates. A (dual) certificate for x ∈ R n is a vector p ∈ R q such that the source condition is verified:

Φ * p ∈ ∂J(x). (SC x )
If p is a certificate, and moreover

Φ * p ∈ ri ∂J(x), (SC x )
we say that p is a non-degenerate certificate, where ri denotes the relative interior.

A subspace T ⊆ R n satisfies the restricted injectivity condition

(INJ T ) if Φ is injective on T .
In practice, it might be difficult to find such a non-degenerate certificate. A popular strategy in the literature is to single out a particular certificate (that we coined minimal norm) which in some cases can be actually computed in closed form. The minimal norm certificate p 0 for x ∈ R n is defined by

p 0 = argmin p∈R q ||p|| subject to Φ * p ∈ ∂J(x).
We define also the linearized precertificate p F as

p F = argmin p∈R q ||p|| subject to (Φ * p) T x = e x .
Now, suppose that (INJ T x ) is satisfied. In this case, p F = Φ +, * T x e x , see Lemma 5.5. Then Φ * p F ∈ ri ∂J(x) or Φ * p 0 ∈ ri ∂J(x) implies that p F = p 0 . Thus the linearized precertificate is the minimal norm certificate if it is indeed a nondegenerate certificate, see Chapter 5 for a precise statement. This is important since p F is simple enough to be computed and analyzed mathematically, leading to an easy way to check if p 0 is a non-degenerate certificate. Another crucial point is that p 0 is the certificate that drives the robustness of the model, as detailed in Section 1.2.2.

Main Contribution

We prove the following Theorem which establishes a linear convergence rate for any closed convex function, without particular assumption on it, except the fact that it is finite-valued, hence continuous.

Theorem 1 Let T 0 = T x 0 . Suppose that (SC x 0 ) is verified for Φ * p ∈ ri ∂J(x 0 ) and (INJ T 0 ) holds. If λ = cε, c > 0, then for every minimizer x ⋆ of (P y,λ )

||x ⋆ -x 0 || 2 Cε , where C = C 1 (2 + c||p|| 2 ) + C 2 (1 + c||p|| 2 /2) 2 cC p , C 1 > 0 and C 2 > 0 are two constants independent of p and 0 < C p < 1.
This theorem is proved in Chapter 6. This result holds for any finite-valued convex function and holds for any minimizer of (P y,λ ) (not necessarily unique). However, remark that (INJ T 0 ) makes sense only if J promotes subspace of low dimension. Note that finding a certificate p is not trivial, and that the constant involved in Theorem 1 depends on it. This leaves a degree of freedom to optimize the constant for the certificate. The closer to 1 the constant C p is, the better is the robustness. It measures how far from the relative boundary is p. Finally, the constants C 1 and C 2 are not absolute and may depend on the dimension. Hence, this theorem does not extend straightforwardly to the infinite-dimensional problem (cP y,λ ).

Relations to Previous Works

Convergence rates. The monograph [START_REF] Scherzer | Variational methods in imaging[END_REF]) is dedicated to regularization properties of inverse problems in infinite-dimensional Hilbert and Banach spaces with application to imaging. In particular, Chapter 3 of this book treats the case where J is a coercive gauge for the problem (P y,λ ). In [START_REF] Burger | Convergence rates of convex variational regularization[END_REF], the authors consider the case where J is a proper, convex and l.s.c functional for both the constrained and Lagrangian regularization (cP y,λ ). Under the source condition and a restricted injectivity assumption, they bound the error in Bregman divergence with a linear rate O(||w||). For the classical Thikonov regularization, i.e.

J = || • || L 2 (Ω) , the estimation is in O( ||w||),
which is not a linear convergence. Extensions of these results have been proved in [START_REF] Resmerita | Regularization of ill-posed problems in Banach spaces: convergence rates[END_REF] and [START_REF] Hofmann | A convergence rates result for Tikhonov regularization in Banach spaces with non-smooth operators[END_REF] for the Bregman rate. [START_REF] Lorenz | Convergence rates and source conditions for Tikhonov regularization with sparsity constraints[END_REF] treats the case where J is a ℓ p norm with 1 p 2 and provides a prediction error Φx 0 -Φx ⋆ in O(||w||) and an estimation error x ⋆x 0 in O( ||w||). [START_REF] Grasmair | Necessary and sufficient conditions for linear convergence of l1-regularization[END_REF] is concerned with the special case of ℓ 1 regularization, and draws some connection with the restricted isometry property (RIP), see below. The result which is the closed to our appears in [START_REF] Grasmair | Linear convergence rates for Tikhonov regularization with positively homogeneous functionals[END_REF]. Here, J is a proper, convex, l.s.c and positively homogeneous functional on some Banach space H. Under a source condition and restricted injectivity on a an appropriate cone, a linear convergence rate is proved with respect to J, i.e.

J(x

⋆ -x 0 ) = O(||w||).
This result implies ours, but only if J is injective which precludes many important regularizers, e.g. TV.

Compressed sensing. In a compressed sensing setting, for instance when Φ is drawn from a i.i.d. normal distribution, it was proved [START_REF] Rudelson | On sparse reconstruction from Fourier and Gaussian measurements[END_REF]) that if the number of measurements q is such that q log(n/k) where k = ||x 0 || 0 then there exists with high probability on Φ a non-degenerate certificate when J = || • || 1 , i.e. (SC x ) holds and one can apply the result of Theorem 1.

The performance of compressed sensing recovery has initially been analyzed using the so-called restricted isometry property (RIP) introduced in (Candès et al. 2006a(Candès et al. , 2006b;;Candès and Tao 2006)

for ℓ 1 . It is defined for a couple (Φ, k)
where k is a targeted sparsity, as the smallest constant δ k such that

(1 -δ k )||x|| 2 ||Φx|| 2 (1 + δ k )||x|| 2 , (1.15)
for any vector x such that ||x|| 0 k. It is shown (Candès et al. 2006a) that if δ 2k + δ 3k < 1, then for every vector x 0 of sparsity k, there exists a nondegenerate certificate (Candès et al. 2005, Lemma 2.2) as remarked also by [START_REF] Grasmair | Necessary and sufficient conditions for linear convergence of l1-regularization[END_REF]. This result thus implies linear convergence rate, and is applied in (Candès et al. 2006b) to show the robustness to noise of compressed sensing. This was generalized to analysis sparsity (i.e. J = ||D * • || 1 with D tight frame) in [START_REF] Candès | Compressed sensing with coherent and redundant dictionaries[END_REF], structured sparsity in [START_REF] Candès | Compressed sensing with coherent and redundant dictionaries[END_REF]) and matrix completion [START_REF] Recht | Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization[END_REF]; Candès and Plan 2011b) using J = || • || * . A major shortcoming of this approach is that available designs of matrices satisfying (1.15) for reasonnably large value of k are essentially random. Indeed, in this case, the constant δ k can be shown to be small enough with high probability on Φ for a nearly optimal scaling of (n, q, k). For instance, when Φ is drawn for the Gaussian ensemble, it is the case when q k log(n/k). as proved by Candès and Tao (2006) Note that in general, computing the RIP constants for a given matrix is an NP-hard problem [START_REF] Bandeira | Certifying the Restricted Isometry Property is Hard[END_REF].

The golfing scheme introduced by [START_REF] Gross | Recovering Low-Rank Matrices From Few Coefficients in Any Basis[END_REF] for the nuclear norm allows to consider non-Gaussian distributions, e.g. partial Fourier measurements. It is based on an iterative scheme starting from the linearized precertificate p F in order to construct an (approximate) certificate with high probability on the matrix for a given vector. It was further studied by Candès and Plan (2011a) for ℓ 1 regularization and [START_REF] Koltchinskii | Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion[END_REF].

Model Selection

So far, we were concerned with ℓ 2 -stability/robustness. What can be said about the recovery of the model T 0 = T x 0 underlying the original vector itself x 0 ? To be able to state such a result, the regularization has to enjoy some additional structure. This is the goal of partial smoothness that we introduce formally hereafter.

The notion of partial smoothness [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF] unifies many notions of structured non-smooth functions known in the literature The notion of partial smoothness (as well as identifiable surfaces [START_REF] Wright | Identifiable Surfaces in Constrained Optimization[END_REF])) captures essential features of the geometry of non-smoothness which are along the so-called "active/identifiable manifold". Loosely speaking, a partly smooth function behaves smoothly as we move on the identifiable manifold, and sharply if we move normal to the manifold. In fact, the behavior of the function and of its minimizers (or critical points) depend essentially on its restriction to this manifold, hence offering a powerful framework for sensitivity analysis theory. In particular, critical points of partly smooth functions move stably on the manifold as the function undergoes small perturbations [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF][START_REF] Lewis | Partial Smoothness, Tilt Stability, and Generalized Hessians[END_REF].

Defi ni ti on 1 .3 -Partly Smooth Functi on. A finite-valued convex function J ∈ Γ + c (R n ) is said to be partly smooth (PSF) at x relative to a set M ⊆ R n if there exists a neighborhood U of x such that (i) M ∩ U is a C 2 -manifold and J restricted to M is C 2 around x, (ii) T x is the tangent plane of M at x, i.e. T M (x) = T x ,
(iii) the set-valued mapping ∂J is continuous at x relative to M.

The manifold M is coined the model manifold of x ∈ R n . J is said to be partly smooth relative to a set M if M is a manifold and J is partly smooth at each point x ∈ M relative to M. J is said to be locally partly smooth at x relative to a set M if M is a manifold and there exists a neighbourhood U of x such that J is partly smooth at each point x ′ ∈ M ∩ U relative to M.

Note that in the previous definition, M needs only to be defined locally around x, and it can be shown to be locally unique. Hence the notation M = M x is unambiguous (locally).

ℓ 1 , ℓ 1 -ℓ 2 and nuclear norms are partly smooth, where the first two ones are such that M = T x . This special class of partly smooth functions is dubbed partly smooth with linear manifold functions. Moreover, if J = J 0 • D * and J 0 is partly smooth at z = D * x for the manifold M 0 z , then it is shown in (Lewis 2002, Theorem 4.2) that J is partly smooth at x for

M = u ∈ R N | D * u ∈ M 0 z .
A similar result is also proved for the sum of two partly smooth functions. We detail these results in Section 4.1.

Main Contribution

We prove the following theorem.

Theorem 2 Let J a locally partly smooth function at x 0 relative to M. Assume that (INJ T ) holds and Φ * p F ∈ ri ∂J(x 0 ). Then there exist positive constants C, C ′ such that if w and λ obey

||w|| C and λ = C ′ ||w||, (1.16)
the solution x ⋆ of (P y,λ ) with noisy measurements y is unique, and satisfies

x ⋆ ∈ M and ||x 0 -x ⋆ || = O(||w||).
This theorem is proved in Chapter 7. Obviously, the assumptions of Theorem 2 imply the conclusion of Theorem 1. Contrary to this same result, this theorem is based on an explicit formulation of the precertificate p F , which makes it directly effective. Note that there exist vectors which can be stably recovered in the ℓ 2 sense of Theorem 1, but whose underlying manifold model cannot be stably identified in the sense of Theorem 2, see our numerical experiments in Chapter 10. When J is partly smooth with linear manifold (M = T x ), i.e. the manifold is in fact the model subspace, a more precise statement of Theorem 2 is given with the explicit derivation of the constants C, C ′ and the one involved in the O(.) term, see Chapter 7.

Relation to Previous Works

Special cases. Theorem 2 is a generalization of many previous works that have appeared in the literature. For the ℓ 1 norm, J = || • || 1 , to the best of our knowledge, this result was initially stated by [START_REF] Fuchs | On sparse representations in arbitrary redundant bases[END_REF]. In this setting, the result x ⋆ ∈ M corresponds to the correct identification of the support, i.e. supp(x ⋆ ) = supp(x 0 ). Moving to a setting where both Φ and w are random, the condition p F ∈ ri ∂J(x 0 ) implies model consistency (also known as sparsistency for ℓ 1 ), i.e. the probability that the support is correctly identified tends to 0 when the dimensions of the problem increases. Bach proves respectively in (Bach 2008a) and (Bach 2008b) Theorem 2 (in fact a variant since he considers randomized Φ and w) for ℓ 1ℓ 2 and nuclear norm gauges, in the special case where Φ has full rank (i.e. is injective). Our results thus shows that the same condition ensure rank consistency with the additional constraint that Ker(Φ) ∩ T = {0}. Theorem 2 for a ℓ 1 analysis prior was proved by Vaiter, Peyré, et al. (2013). A similar result was shown in [START_REF] Duval | Exact Support Recovery for Sparse Spikes Deconvolution[END_REF]) for an infinite dimensional sparse recovery problem over the space of Dirac measures, with J the total variation of a measure.

Compressed sensing. Condition Φ * p F ∈ ri ∂J(x 0 ) is often used when Φ is drawn from the Gaussian matrix ensemble to asses the performance of compressed sensing recovery with ℓ 1 norm. It has been proved [START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming (lasso)[END_REF][START_REF] Dossal | Sharp support recovery from noisy random measurements by ℓ 1 -minimization[END_REF] for J = || • || 1 that if Φ is a random matrix drawn from the Gaussian ensemble, then for q > 2s log n, Φ * p F ∈ ri ∂J(x) with high probability on Φ for k = ||x 0 || 0 . One may have observed that the bound on q bears similarities with that of Section 1.2.1 except in the scaling in the log term. It was also used to ensure ℓ 2 robustness of matrix completion in a noisy setting by [START_REF] Candès | Matrix completion with noise[END_REF], and our findings show that it also ensures rank consistency for matrix completion at high signal to low noise levels. It generalizes the result proved for a family of decomposable norms (including in particular ℓ 1 -ℓ 2 norm and the nuclear norm) by [START_REF] Candès | Simple bounds for recovering low-complexity models[END_REF] when w = 0.

Stronger criteria for ℓ1 . Many sufficient conditions can be formulated to ensure that p F is a non-degenerate certificate, and hence to guarantee the model stability. The strongest criterion to ensure a noise robustness for ℓ 1 regularization is the mutual coherence, introduced by [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]. Finer criteria based on Babel functions have been proposed in [START_REF] Gribonval | Beyond sparsity : recovering structured representations by ℓ 1 -minimization and greedy algorithms[END_REF][START_REF] Borup | Beyond coherence : recovering structured time-frequency representations[END_REF]. The Exact Recovery Condition introduced by [START_REF] Tropp | Just relax: Convex programming methods for identifying sparse signals in noise[END_REF] is weaker than the coherence which in turns is greater that the weak-ERC (Dossal 2012).

Sensitivity Analysis and Parameter Selection

Beside studying stability, the second goal of this thesis is to investigate the sensitivity of any solution x ⋆ (y) to the parameterized problem (P y,λ ) to (small) perturbations of y. This sensivity analysis is central to construct an unbiased estimator of the quadratic risk, as described in Section 1.3.2. We suppose here that J is a partly smooth gauge with linear manifold, i.e. such that M x = T x and J is 1-homogeneous. We conjecture that this statement remains true for any finite-valued convex partly smooth function, though this has not been formally proved yet. The technical obstacles faced by this generalization will be discussed in Chapter 9.

Local Differentiability of the Optimal Solutions

The objective here is find a formula of the derivative of x ⋆ (y) with respect to the observations, valid on the biggest set possible. Moreover, since x ⋆ (y) is not uniquely defined, it has to be interpreted as a multivalued mapping. Sensitivity analysis 1 is a major branch of optimization and optimal control theory. Comprehensive monographs on the subject are [START_REF] Bonnans | Perturbation analysis of optimization problems[END_REF][START_REF] Mordukhovich | Sensitivity analysis in nonsmooth optimization[END_REF]). The focus of sensitivity analysis is the dependence and the regularity properties of the optimal solution set and the optimal values when the auxiliary parameters (e.g. y here) undergo a perturbation. In its simplest form, sensitivity analysis of first-order optimality conditions, in the parametric form of the Fermat rule, relies on the celebrated implicit function theorem.

Main Contribution

Because of non-smoothness of the regularizer J, it is a well-known fact in sensitivity analysis that one cannot hope for a global claim, i.e. an everywhere smooth mapping2 y → x ⋆ (y). Rather, the sensitivity behavior will be local. This is the reason why we need to introduce the following transition space H, which will be shown to contain points of non-smoothness of y → x ⋆ (y).

We introduce the transition space H defined as

H = T ∈T H T , where H T = bd(Π n+p,n (A T )),
where Π n+p,n is the canonical projection onto the first n components, bd C is the boundary of C, and

A T = (y, x T ) ∈ R n × T | 1 λ Φ * T (Φx T -y) ∈ rbd ∂J(x T ) .
Here, rbd ∂J(x T ) is the relative boundary of ∂J(x T ) relatively to its affine hull and T = {x ∈ R n | T x = T }. This set corresponds exactly to the observations y such that the model space associated to a solution of (P y,λ ) is not stable with respect to small perturbations. In particular, when J = || • || 1 , we show that this set is in fact a union of hyperplanes and when J = || • || 1,2 it is a semi-algebraic set.

Our main sensitivity analysis is the following.

Theorem 3 Let y ∈ H and x ⋆ a solution of (P y,λ ) such that

Ker Φ T ∩ Ker D 2 J T (x ⋆ ) = {0} (I x ⋆ )
where T = T x ⋆ . Then, there exists an open neighborhood V ⊂ R n of y, and a mapping x : V → T such that (i) For every ȳ ∈ V, x( ȳ) is a solution of (P λ( ȳ)), and x(y) = x ⋆ .

(ii) The mapping x is C 1 (V) and

∀ ȳ ∈ V, D 1 x( ȳ) = -(Φ * T Φ T + λD 2 J T (x ⋆ )) -1 Φ T .
The mapping y → µ(y) = Φx ⋆ is single-valued and C 1 (R n \ H). For every y ∈ H, there exists a solution x ⋆ of P λ (y) such that (I x ⋆ ) is satisfied. Moreover, for any y ∈ H, div( µ)(y) = tr(∆(y))

where

∆(y) = -Φ T (Φ T * Φ T + λD 2 J T (x ⋆ )) -1 • Φ T * .
This theorem is proved in Chapter 8.

Relation to Previous Works

Sensitivity analysis is a major branch of optimization and optimal control theory. Comprehensive monographs on the subject are [START_REF] Bonnans | Perturbation analysis of optimization problems[END_REF][START_REF] Mordukhovich | Sensitivity analysis in nonsmooth optimization[END_REF]). The focus of sensitivity analysis is the dependence and the regularity properties of the optimal solution set and the optimal values when the auxiliary parameters (e.g. y here) undergo a perturbation. In its simplest form, sensitivity analysis of first-order optimality conditions, in the parametric form of the Fermat rule, relies on the celebrated implicit function theorem.

For the Lasso problem, the above divergence formula implies that

div( µ)(y) = | supp(x ⋆ )|,
where x ⋆ is a solution of (P y,λ ) such that (I x ⋆ ) holds, i.e. Φ supp(x ⋆ ) has full rank. This result was proved in [START_REF] Dossal | The degrees of freedom of the Lasso for general design matrix[END_REF], see also [START_REF] Tibshirani | The solution path of the generalized lasso[END_REF] where a similar result is proved without the condition (I x ⋆ ).

The case of analysis sparsity was investigated in [START_REF] Vaiter | Local behavior of sparse analysis regularization: Applications to risk estimation[END_REF] and [START_REF] Tibshirani | The solution path of the generalized lasso[END_REF]. In this case, one has

div( µ)(y) = dim Ker D * Λ , Λ = supp(D * x ⋆ ) c ,
where x ⋆ is such that (I x ⋆ ) holds.

The originality of our contribution in this direction is the following:

(i) We formulate the set H of non-smoothness points, which is crucial for the application to risk estimation exposed bellow.

(ii) We give an explicit formula of the divergence of the prediction.

(iii) Our sensitivity result deals with a set-valued mapping (even if its image by Φ is single-valued).

Unbiased Risk Estimation

The degrees of freedom (DOF) of a statistical procedure quantifies its complexity [START_REF] Efron | The estimation of prediction error: Covariance penalties and cross-validation[END_REF]). Among possible applications is the computation of efficient risk estimators. These estimator allows an objectively guided choice of the hyperparameters associated to the statistical procedure.

Let µ 0 = Φx 0 . Suppose that the observations Y ∼ N(µ 0 , σ 2 Id n ). Following [START_REF] Efron | The estimation of prediction error: Covariance penalties and cross-validation[END_REF], the DOF is defined as

df = n i=1 cov(Y i , µ i (Y)) σ 2
.

The well-known Stein's lemma [START_REF] Stein | Estimation of the Mean of a Multivariate Normal Distribution[END_REF]

E ∂ µ i ∂y i (Y) < ∞, ∀i ,
then its divergence is an unbiased estimator of its DOF, i.e.

df = div( µ)(Y) = tr(D µ(Y)) and E( df) = df ,
where D µ is the Jacobian of y → µ(y). In turn, this allows to get an unbiased estimator of the prediction risk E(|| µ(Y)µ 0 || 2 ) through the SURE (Stein Unbiased Risk Estimate Stein 1981).

Main Contribution

To apply Stein's lemma we need to provide a closed-form of the Jacobian of y → µ(y) which holds true almost everywhere. Roughly speaking, to be able to control the size of R q \ H, the functions J cannot be too oscillating in order to prevent pathological behaviors. In order to do this, we use arguments of o-minimal geometry. More precisely, we ask that the function J is definable in such a structure and that T = (T x ) x∈R n is finite. These assumptions exclude the nuclear norm. Under such assumptions, we prove the following theorem.

Theorem 4 Let Y = Φx 0 + W with W ∼ N(0, σ 2 Id n ). Then, (i) H is of Lebesgue measure zero;
(ii) µ is Lipschitz continuous, hence weakly differentiable, with an essentially bounded gradient.

(iii) df = tr(∆(Y)) is an unbiased estimate of df = E(div( µ(Y)).

(iv) The SURE

SURE( µ)(Y) =||Y -µ(Y)|| 2 + 2σ 2 df -nσ 2 (1.17) is an unbiased estimator of the risk E || µ(Y) -µ 0 || 2 .
This theorem is proved in Chapter 9. This result holds true for the SURE within an exponential family, see Chapter 9.

Relation to Previous Works

In the case of standard Lasso (i.e. ℓ 1 penalty) with Y ∼ N(Φx 0 , σ 2 Id n ) and Φ of full column rank, [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF] showed that the number of nonzero coefficients is an unbiased estimate for the DOF. Their work was generalized in [START_REF] Dossal | The degrees of freedom of the Lasso for general design matrix[END_REF] to any arbitrary design matrix. Under the same Gaussian linear regression model, unbiased estimators of the DOF for the Lasso with ℓ 1 -analysis penalty, were given independently in [START_REF] Tibshirani | The solution path of the generalized lasso[END_REF][START_REF] Vaiter | Local behavior of sparse analysis regularization: Applications to risk estimation[END_REF].

A formula of an estimate of the DOF for the group Lasso when the design is orthogonal within each group was conjectured in [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF]. Kato 2009 studied the DOF of a general shrinkage estimator where the regression coefficients are constrained to a closed convex set C. His work extends that of [START_REF] Meyer | On the degrees of freedom in shaperestricted regression[END_REF] which treats the case where C is a convex polyhedral cone. When Φ is full column rank, [START_REF] Kato | On the degrees of freedom in shrinkage estimation[END_REF] derived a divergence formula under a smoothness condition on the boundary of C, from which an unbiased estimator of the degrees of freedom was obtained. When specializing to the constrained version of the group Lasso, the author provided an unbiased estimate of the corresponding DOF under the same group-wise orthogonality assumption on Φ as [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF]). An estimate of the DOF for the group Lasso was also given by [START_REF] Solo | Threshold selection for group sparsity[END_REF] using heuristic derivations that are valid only when Φ is full column rank, though its unbiasedness is not proved.

Reading Guide

This thesis is organized in 11 chapters. Figure 1.3 provides a description of the dependencies between them. A summary in French of the thesis is provided in appendix.

Chapter 2: Mathematical Background. This chapter provides the necessary common material used in this thesis. In particular, we recall basic definitions of convex analysis (in particular gauges), o-minimality and smooth manifolds.

This chapter contains mainly well known definitions and properties, but we would like to emphasize that some new results are established. The reader is invited to take in account the List of Notations in appendix.

1. ) on an open neighborhood of some solution x ⋆ . Theorem 8.2 shows that the prediction map is well-defined outside the transition space and gives its local behavior.

Chapter 9: Unbiased Risk Estimation. In this chapter, we prove Proposition 9.1 stating that that the transition space has zero measure w.r.t Lebesgue measure. Proposition 9.2 proves that the prediction is Lipschitz continuous. Theorems 9.1 and 9.2 prove that the (G)SURE is an unbiased estimator of the risk for non-linear Gaussian regression and generalized linear model.

Numerical Considerations and Conclusion

Chapter 10: Numerical Considerations. This chapter recaps our results from a numerical point of view. We prove in Theorem 10.1 that under the same hypothesis of non-degeneracy and partial smoothness of Theorem 7.2, the forward-backward algorithm identifies the correct manifold after a finite number of steps. We discuss in Sections 10.2 and 10.3 how the linearized precertificate behaves in different concrete scenarios. We investigate further the noiseless behavior of total variation denoising in Theorem 10.2 and the compressed sensing with ℓ ∞ regularization in Theorem 10.3. We also show how in practice one can use our sensitivity result (Theorem 9.1) to select the best hyperparameter λ for ℓ 1 -analysis regularization in Section 10.4.

Chapter 11: Conclusion. This last chapter summarizes our contributions. We also discuss several open problems. T hi s chapter lays the mathematical foundation of our work. In particular, we recap important results of convex analysis, o-minimal geometry and some properties of smooth manifolds. An emphasis is done on the notion of gauge which will be important in order to capture the structure of the subdifferential of a convex function.

In all the following, R n will be the signal space, R q the observation space and R p the analysis space. The space R n will be endowed with its canonical Euclidian structure and its associated inner product is denoted

•, • , i.e. ∀x, x ′ ∈ R n , x, x ′ = n i=1 x i x ′ i ,
and the associated norm, the ℓ 2 norm1 is denoted

||x|| = n i=1 x 2 i .
For any subspace T of a real vector space E, we denote P T the orthogonal projection on T , and x T = P T (x) and Φ T = Φ P T .

For a subset I of {1, . . . , n}, we denote by I c its complement with respect to {1, . . . , n}, |I| its cardinality, x (I) is the subvector whose entries are those of x restricted to the indices in I, and Φ (I) the submatrix whose columns are those of Φ indexed by I. For any matrix A, A * denotes its adjoint matrix and A + its Moore-Penrose pseudo-inverse. We denote the right-completion of the real line by R = R ∪ {+∞}.

Section 2.1 recalls basics of convex analysis, and Section 2.2 is concerned with differential properties. Then, Section 2.3 details properties of gauges.

Finally, in Section 2.4 we sketch some properties of o-minimal geometry, a generalization of semi-algebraic geometry.

Convex Analysis

In this section, we recall useful concepts from convex analysis in finite dimension. The definitive reference book on this subject is [START_REF] Rockafellar | Convex analysis[END_REF]. One may also refers to [START_REF] Zalinescu | Convex analysis in general vector spaces[END_REF][START_REF] Hiriart-Urruty | Convex Analysis And Minimization Algorithms[END_REF] for more details, or [START_REF] Ekeland | Analyse convexe et problemes variationelles[END_REF] for the infinite dimensional case.

Functions

We recall basic definitions of real analysis. D efi ni ti on 2 . 1 -E pi graph and D omai n. The epigraph of a function f : R n → R is the set

epi f = {(x, α) ∈ R n × R | α f(x)} ⊆ R n+1 .
The (effective) domain of f is the set the projection of epi f under the mapping

(x, α) → x, i.e. dom f = x ∈ R N | f(x) < +∞ . The function f is proper if dom f = ∅.
The epigraph is the set of points lying above its graph. Coercivity and lower semicontinuity will play an important role.

D efi ni ti on 2 . 2 -C oerci vi ty. A function f : R n → R is coercive if lim ||x||→+∞ f(x) = +∞.
An example of coercive function is any norm over R n . However, this is not the case of ||D * • || as soon as D * has a non trivial kernel.

Defi ni ti on 2 .3 -L ower Semi conti nui ty.

A function f : R n → R is lower semi-continuous (l.s.c.) at x ∈ R n if lim inf z→x f(z) f(x).
The fact that a function f is l.s.c. is equivalent to epi f closed in R n × R, see (Rockafellar et al. 1998, Theorem 1.6). For this reason, we also say that f is closed. We recall that lim inf

z→x f(z) = sup δ>0 inf {z| ||z-x||}<δ f(z).
Defi ni ti on 2 .4 -K ernel . The kernel of a function is defined as

Ker f = {x ∈ R n | f(x) = 0} .
Note that the kernel of a function is not necessarily a linear subspace. However, if f is convex, Ker f is a convex cone.

Defi ni ti on 2 .5 -S ublevel Set. The sublevel set slev x J of J passing through x is defined as

slev x J = {z ∈ R n | J(z) J(x)} .

Convexity

All functionals considered in this manuscrit are convex. We recall the definition of convexity and give several examples.

D efi ni ti on 2 . 6 -C onvexi ty. A set C ⊆ R n is said to be convex if ∀x, x ′ ∈ C, ∀µ ∈ [0, 1], µx + (1 -µ)x ′ ∈ C. A function f : R n → R is said to be convex if its epigraph is convex, i.e. ∀x, x ′ ∈ R n , ∀µ ∈ [0, 1], f(µx + (1 -µ)x ′ ) µf(x) + (1 -µ)f(x ′ ). It is strictly convex if ∀x, x ′ ∈ R n , ∀µ ∈ [0, 1], x = x ′ ⇒ f(µx + (1 -µ)x ′ ) < µf(x) + (1 -µ)f(x ′ ).
It is strongly convex of modulus τ if for every x, x ′ ∈ C and every µ ∈ [0, 1],

f(µx + (1 -µ)x ′ ) µf(x) + (1 -µ)f(x ′ ) - τ 2 µ(1 -µ)||x ′ -x|| 2 .
The set of all convex, proper and closed functions is denoted Γ 0 (R n ). The set of all finite-valued, bounded from below, convex, proper (hence continuous) functions is denoted

Γ + c (R n ). D efi ni ti on 2 . 7 -I ndi cator Functi on. Let C a nonempty closed convex subset of R n . The indicator function ι C ∈ Γ 0 (R n ) of C is ι C (x) =    0, if x ∈ C , +∞, otherwise. D efi ni ti on 2 . 8 -C onj ugate. The Legendre-Fenchel conjugate f * ∈ Γ 0 (R n ) of a proper, closed and convex function f ∈ Γ 0 (R n ) is f * (u) = sup x∈dom f u, x -f(x) .
Here, f * is proper, closed and convex, thus f * * = f. For instance, the conjugate of the indicator function ι C is the support function of C defined as:

Defi ni ti on 2 .9 -S upport functi on The support function of a nonempty closed convex subset

C of R n is σ C (u) = sup x∈C u, x . σ C is sublinear, is non-negative if 0 ∈ C,
(i) C 1 ⊆ C 2 ⇔ σ C 1 σ C 2 , (ii) σ C 1 +C 2 = σ C 1 + σ C 2 , (iii) For any ρ ∈ R, σ ρC 1 = ρσ C 1 .
Defi ni ti on 2 .10 -I nfi mal convoluti on. Let f and g be two proper closed convex functions from R n to R. Their infimal convolution is the function

(f + ∨ g)(x) = inf x 1 +x 2 =x f(x 1 ) + g(x 2 ) = inf z∈R N f(z) + g(x -z) .
An important property of optimization with convex function is recapped below.

P roposi ti on 2 .2 Let F 0 be a strictly convex function and J a closed convex function. Then every solutions of the problem min

x∈R n F 0 (Φx) + J(x) (2.1)
share the same image by Φ and the same value J. Moreover, given two solutions of (2.1)

x ⋆ 0 , x ⋆ 1 , there exists δ such that x ⋆ 0 = x ⋆ 1 + δ. p ro o f Let x ⋆ 0 , x ⋆ 1 be two solutions of P(y) such that Φx ⋆ 0 = Φx ⋆ 1 . Take any convex combination x ⋆ t = (1 -t)x ⋆ 0 + tx ⋆ 1 , t ∈]0, 1[. Strict convexity of µ → F 0 (µ)
implies that the Jensen inequality is strict, i.e.

F 0 (Φx ⋆ t ) < (1 -t)F 0 (Φx ⋆ 0 ) + tF 0 (Φx ⋆ 1 ).
The convexity of the regularization implies

J(x ⋆ t ) (1 -t)J(x ⋆ 0 ) + tJ(x ⋆ 1 ) .
Summing these two inequalities we arrive at

F 0 (Φx ⋆ t ) + J(x ⋆ t ) < F 0 (Φx ⋆ 0 ) + J(x ⋆ 0 )
a contradiction since x ⋆ 0 is a minimizer of (2.1).

Special Convex Sets

D efi ni ti on 2 . 11 -C onvex Hull and i ts Closure . The convex hull of a non-empty set C ⊂ R n is the intersection of all convex sets containing C. We denote co (C) the closure of its convex hull.

Defi ni ti on 2 .12 -A ffi ne Hull. Its affine hull aff C is the smallest affine manifold containing it, i.e.

aff C = k i=1 ρ i x i | k > 0, ρ i ∈ R, x i ∈ C, k i=1 ρ i = 1 .
It is included in the linear hull span C which is the smallest subspace containing C.

Defi ni ti on 2 .13 -I nteri or and Relati ve Interi or . The interior of C is denoted int C. The relative interior ri C of a convex set C is the interior of C for the topology relative to its affine full.

Defi ni ti on 2 .14 -

C losed Coni cal H ull . The closed conical hull of a nonempty set C ⊂ R n is cone(C) = cl k i=1 ρ i x i | k > 0, ρ i 0, x i ∈ C .
Note that the closure operation is necessary. In general, the argument of the closure is neither compact nor closed, even if C is a convex compact set.

In the following, we give a handy expression of the tangent cone to a closed convex set, see

C ⊂ R n at x is T C (x) = cone(C -x) = cl t 0 t(C -x) .
The normal cone to C at x is the polar of T C (x), i.e.

N C (x) = {z ∈ R n | ∀c ∈ C, z, x -c 0} .

Multivalued Mappings

We refer to [START_REF] Aubin | Set-Valued Analysis[END_REF] for more details about multivalued mappings.

We need the definition of continuity and Lipschitz-property in this work.

D efi ni ti on 2 . 16 -M ulti valued M appi ng . A multivalued map- ping S : X ⇒ Y from X to Y is a mapping from X to the subsets of Y.
D efi ni ti on 2 . 17 -C onti nui ty. Let S : X ⇒ Y a multivalued mapping. We say that S is

• outer semicontinuous at x if lim sup z→x S(z) ⊆ S(x) where lim sup z→x S(z) = {u | ∃x ν → x, ∃u ν → u with u ν ∈ S(x ν )} .
• inner semicontinuous at x if lim inf z→x S(z) ⊇ S(x) where

lim inf z→x S(z) = {u | ∀x ν → x, with u ν ∈ S(x ν )} . • continuous if both conditions holds. D efi ni ti on 2 . 18 -L i pschi tz Map. Let S : X ⊆ R n ⇒ Y a multival- ued mapping. We say that S is β-Lipschitz around x ∈ X if there exists a neighborhood U of x such that ∀x 1 , x 2 ∈ U, S(x 1 ) ⊆ S(x 2 ) + β||x 1 -x 2 ||B X ,
where B X is the unit ball of X.

The following lemma is important in the study of partly smooth functions in Chapter 4.

Lemma 2 . 1 Let C : X ⇒ Y be a β-Lipschitz multivalued mapping, such that C(x) is a compact convex set for every x ∈ X. Then, for every x 1 , x 2 ∈ X and y ∈ Y, σ C(x 1 ) (y) -σ C(x 2 ) (y) β||x 1 -x 2 ||||y||. p ro o f Since C(x 1 ) ⊆ C(x 2 ) + β||x 1 -x 2 ||B X , we have by 2.1 (i), σ C(x 1 ) σ ⊆C(x 2 )+β||x 1 -x 2 ||B X .
By Proposition 2.1 (ii) and (iii), we obtain

σ C(x 1 ) (y) σ C(x 2 ) (y) + β||x 1 -x 2 ||σ B X (y).
Since σ B X (y) = ||y||, we obtain our claim.

A proof of this statement can also be found in (Hiriart-Urruty et al. 2001, Theorem V.3.3.8).

Asymptotic Cone and Function

Defi ni ti on 2 .19 Let C be a non-empty closed convex set in R n . Its asymptotic cone, or recession cone, C ∞ is the set

C ∞ = {d ∈ R n | x + td ∈ C, ∀t > 0} = t>0 C -x t , ∀x ∈ C .
The closure assumption on the convex set C is crucial and cannot be removed.

The importance of the asymptotic cone is revealed by the following key properties, in particular property (iii). (Auslender et al. 2003, Proposition 2.1.5).

P roposi ti on 2 .3 Let C be a non-empty closed convex set in R n . (i) C ∞ is independent of x. (ii) C ∞ is a closed convex cone. (iii) C is compact if and only if C ∞ = {0}. (iv) If C is non-empty closed convex cone, then C ∞ = C. p ro o f (i)
(ii) (Auslender et al. 2003, Proposition 2.1.5).

(iii) (Auslender et al. 2003, Proposition 2.1.2).

(iv) (Auslender et al. 2003, Proposition 2.1.1(c) and Proposition 2.1.5).

D efi ni ti on 2 . 20 For any function f ∈ Γ 0 (R n ), there exists a unique function f ∞ : R n → R associated with f, called the asymptotic function, or the recession function, such that epi f ∞ = (epi f) ∞ .

In [START_REF] Hiriart-Urruty | Convex Analysis And Minimization Algorithms[END_REF], the notation f ′ ∞ is used which is justified by the properties hereafter.

The epigraph of f ∞ is a closed convex cone, see Proposition 2.3. Moreover, f ∞ enjoys many important properties some of which we summarize as follows.

P roposi ti on 2 .4 Let f ∈ Γ 0 (R n ). (i) f ∞ ∈ Γ 0 (R n ) and positively homogeneous, f ∞ (d) = sup x∈dom(f) f(x + d) -f(x) ,
and

f ∞ (d) = lim t→+∞ f(x + td) -f(x) t = sup t>0 f(x + td) -f(x) t , ∀x ∈ dom(f) . (ii) In particular, if 0 ∈ dom(f), then ∀d ∈ R n f ∞ (d) = lim t→+∞ f(td) t . (iii) (ι C ) ∞ = ι C ∞ , for C a non-empty closed convex set. (iv) If f is a gauge of C containing the origin, then Ker(f) = C ∞ . (v) Let f i ∈ Γ 0 (R n ), i = 1, • • • , m, f := p i=1 f i and m i=1 dom(f i ) = ∅. Then, f ∈ Γ 0 (R n ) and f ∞ = p i=1 (f i ) ∞ . (vi) Let A : R n → R p be a linear map such that Im(A) ∩ dom(f) = ∅. Then (f • A) ∞ (d) = f ∞ (Ad) . p ro o f (i)
The first statement is a consequence of convexity and (Auslender et al. 2003, Proposition 2.5.1(a)), which is in turn uses Proposition 2.3(ii). The equivalent expressions of f ∞ follow from (Auslender et al. 2003, Proposition 2.5.2).

(ii) Since 0 ∈ dom(f), f(0) < ∞ and the formula follows from (i).

(iii) (Auslender et al. 2003, Corollary 2.5.1).

(iv) (Auslender et al. 2003, Proposition 2.6.1).

(v) (Auslender et al. 2003, Proposition 2.6.3).

Differential Properties

The set of continuously differentiable functions from a set X ⊆ R n to R q is denoted C 1 (X, Y) and the Jacobian of a function f :

X → R q at a point x ∈ X is denoted Df(x).
First of all, we should recall the classical implicit function theorem.

Theorem 2 .1 -I mpli ci t Functi on. Let f : R n × R q → R n be a C 1 function in a neighborhood of (x, ȳ) such that f(x, ȳ) = 0.
Assume that the Jacobian matrix D 1 f(x, ȳ) with respect to the first variable is non-singular at (x, ȳ). Then, there exists an open neighborhood U of ȳ and a mapping x :

U → R n such that x is C 1 on U, ∀y ∈ U, f(x(y), y) = 0 and x( ȳ) = x.
Moreover, its Jacobian reads

∀y ∈ U, Dx(y) = -(D 1 f(x(y), y)) -1 D 2 f(x(y), y).
We draw the attention of the reader to the fact that this theorem admits several generalizations such as for instance in the context of multivalued mappings, see the monograph of [START_REF] Dontchev | Implicit functions and solution mappings: A view from variational analysis[END_REF].

Subdifferential

Defi ni ti on 2 .21 -S ubdi fferenti al . The subdifferential ∂f(x) of a convex function f at x is the set

∂f(x) = u ∈ R n | f(x ′ ) f(x) + u, x ′ -x , ∀x ′ ∈ dom f . An element of ∂f(x) is a subgradient. If the convex function f is Gâteaux- differentiable at x, then its only subgradient is its gradient, i.e. ∂f(x) = {∇f(x)}. P roposi ti on 2 .5 Let f ∈ Γ 0 (R n ).
Then ∂f is outer semicontinuous.

p ro o f See (Hiriart-Urruty et al. 2001, Theorem 6.2.4).

This result can be stated as

∀ε > 0, ∃δ > 0, ||x ′ -x|| δ ⇒ ∂f(x ′ ) ⊆ ∂f(x) + B(0, ε).
Note that without additional constraints, ∂f is not inner semicontinuous. This fact will motivate us to introduce the notion of partial smoothness in Chapter 4.

Defi ni ti on 2 .22 -D i recti onal D eri vati ve. The directional derivative f ′ (x, δ) of a finite-valued closed function f at the point x ∈ dom f in the direction δ ∈ R n is f ′ (x, δ) = lim t↓0 f(x + tδ) -f(x) t . When f is convex, then the function δ → f ′ (x, •) exists and is sublinear. The subdifferential ∂f(x) is a non-empty compact convex set of R n whose support function is f ′ (x, •), i.e. f ′ (x, δ) = σ ∂f(x) (δ) = sup η∈∂f(x)
η, δ .

We also recall the fundamental first-order minimality condition of a convex function.

P roposi ti on 2 .6 A vector x ⋆ is the global minimizer of a convex function f if, and only if, 0 ∈ ∂f(x).

We define the Bregman divergence, a classical tool in convex analysis.

D efi ni ti on 2 . 23 The Bregman divergence D J η (x, x 0 ) associated to a convex function J ∈ Γ 0 (R n ) and a vector η ∈ ∂J(x 0 ) between two points x and x 0 ∈ R n is defined as

D J η (x, x 0 ) = J(x) -J(x 0 ) -η, x -x 0 .
It is obvious that by convexity, the Bregman divergence is non-negative. When J is differentiable at x 0 , the unique Bregman divergence is then associated to η = ∇J(x 0 ) and we recover the standard smooth case where

D J (x, x 0 ) = J(x) -J(x 0 ) -∇J(x 0 ), x -x 0 .
Note that the Bregman divergence is not a distance, since it does not satisfy the triangle inequality nor the symmetry axioms. However, it is common in the litterature to find the term Bregman distance. We will drop the exponent J if the context allows it.

Minimizers Gradients for Composite Problems

The following lemma shows that for a minimization problem min

f + g such that f ∈ Γ 0 (R n ) is C 2 and g ∈ Γ 0 (R n ), the solutions share the same gradient. Lemma 2 . 2 Let x ⋆ 0 and x ⋆ 1 be two solutions of min x∈R p f(x) + g(x) (2.2)
where f is proper, convex and C 2 (R p ) function, and g is proper, convex and lower semicontinuous with a non-necessarily full-domain. Then

∇f(x ⋆ 0 ) = ∇f(x ⋆ 1 ).
p ro o f Let x ⋆ 0 and x ⋆ 1 be two distinct solutions of (2.2), otherwise, there is nothing to prove. We denote

x ⋆ t = x ⋆ 0 + th where h = x ⋆ 1 -x ⋆ 0 , t ∈ [0, 1]. By convexity, x ⋆ t is also a minimizer of (2.2). We have -∇f(x ⋆ t ) ∈ ∂g(x ⋆ t ). Convexity of g then yields ∇f(x ⋆ t ) -∇f(x ⋆ 0 ), th 0.
Similarly, convexity of f entails

∇f(x ⋆ t ) -∇f(x ⋆ 0 ), th 0.
Combining these inequalities yields, for any t

∈ [0, 1] ∇f(x ⋆ t ) -∇f(x ⋆ 0 ), h = 0. (2.3) Since f is C 2 (R p ), Taylor expansion gives ∇f(x ⋆ 1 ) -∇f(x ⋆ 0 ) = 1 0 D 2 f(x ⋆ t )hdt , (2.4)
which, after taking the inner product of both sides with h and using (2.3), yields

∇f(x ⋆ 1 ) -∇f(x ⋆ 0 ), h = 1 0 D 2 f(x ⋆ t )h, h dt = 0. (2.5)
By convexity, the Hessian D 2 f(x ⋆ t ) is semidefinite positive, and

(2.5) implies that ∀t ∈ [0, 1], D 2 f(x ⋆ t )h, h = 0,
or equivalently

||D 2 f(x ⋆ t ) 1/2 h|| = 0 ⇔ h ∈ Ker D 2 f(x ⋆ t ) .
Inserting this again in (2.4) yields the desired claim.

Smooth Manifolds

In this thesis, we will not use advanced results of differential geometry. However, we need the structure of smooth manifold to define the central notion of partial smoothness. This section aims to to recall basic notion on smooth manifolds. The reader may refer to [START_REF] Lee | Smooth manifolds[END_REF].

D efi ni ti on 2 . 24 -S mooth mani fold. Let k 1. A C k -manifold M around x ∈ R n of codimension m is a subset of R n such that there exists an open set U of R n and a C k -function g : U → R m satisfying M ∩ U = {x ∈ U | g(x) = 0} ,
and g has surjective derivative throughout U. We say that

M is a C k -manifold if M is a C k -manifold around every x ∈ M of codimension m.
Note that every linear subspace H of R n is a manifold around each point x ∈ H, and this is in particular true for H = R n . Another example, which will be used in this thesis, is the set of matrices of fixed rank [START_REF] Lee | Smooth manifolds[END_REF].

D efi ni ti on 2 . 25 -Tangent space . Let M be a C k -manifold around x ∈ M of codimension m associated to a C k function g. The tangent space of M at x is defined as T x (M) = Ker Dg(x).
We now introduce the Grassmann manifold, which will be used in Chapter 7. [START_REF] Lee | Smooth manifolds[END_REF]).

P roposi ti on 2 .7 -G rassmann M ani fold. Let G k,n be the set of all linear subspaces of R n of dimension k. Then, G k,n is a smooth manifold of dimension k(n -k), coined the Grassman manifold of k-planes. Moreover, (G k,n , d) endowed with d(V, V ′ ) = ||P V -P V ′ || is a compact metric space, where || • || is an operator norm. p ro o f This property is a consequence of the isomorphism between G k,n and O n /(O k × O n-k ), see
An important property is the fact that the projection onto a manifold is locally well-defined as a single-valued mapping.

Lemma 2 . 3 Let M be a C k -manifold with k 2 around a point x ∈ M. Then, there exists a neigborhood U of x such that for every x ∈ U, x has a unique projection P M (x) onto M. Moreover, the function

P M : U → M is C k-1 , with derivative DP M (x) = P T x(M) .
p ro o f See [START_REF] Lewis | Alternating Projections on Manifolds[END_REF], Lemma 2.1).

Gauges

This section gives some general results on gauges. Again, we refer to (Rockafellar 1996) for more insight on this notion. Gauges are equivalently defined as nonnegative, convex and positively homegeneous functions or are parameterized by convex set containing 0. They are the natural extension of norms or seminorms, which are indeed gauges. The classical duality is then replaced by the polarity of convex set parameterizing the gauges. We start by defining formally a gauge, and prove the associated Lemma 2.4 stating the equivalence between gauges and convex sets containing zero.

We begin with the definition of a gauge.

D efi ni ti on 2 . 26 -G auge Let C ⊆ R n be a non-empty closed convex set containing the origin. The gauge of C is the function γ C ∈ Γ 0 (R n ) defined on R n by γ C (x) = inf {λ > 0 | x ∈ λC} .
As usual, γ C (x) = +∞ if the infimum is not attained.

We say that γ C is bounded (or finite-valued) if, for every x ∈ R n , γ C (x) < +∞. This is typically not the case if the gauge is of the form γ

C (x) = f(x) + ι C (x)
where ι C is the indicator function of a convex set C. Some important properties are stated below. In particular, Lemma 2.4((ii)) is a fundamental result of convex analysis that states that there is a one-to-one correspondence between gauge functions and closed convex sets containing the origin. This allows to identify sets from their gauges, and vice versa.

L emma 2 .4 Let C ⊆ R n and γ C the associated gauge.

(i) γ C is a non-negative, lsc and sublinear function.

(ii) Suppose C is a closed convex set containing the origin. Then, f is the gauge associated to C if, and only if, f is positively homegeneous and

C = {x ∈ R n | f(x) 1} . (iii) γ C is bounded if, and only if, 0 ∈ int C, in which case γ C is continuous. (iv) Ker γ C = {0},
= aff C = span C since 0 ∈ C.

Polar Set and Gauges.

Let us now turn to the polar of a convex set and a gauge.

Defi ni ti on 2 .27 -Polar set Let C be a non-empty convex set. The set C • given by

C • = {v ∈ R n | ∀x ∈ C, v, x 1} is called the polar of C.
C • is a closed convex set containing the origin. When the set C is also closed and contains the origin, then it coincides with its bipolar, i.e.

C •• = C.
We are now in position to define the polar gauge.

D efi ni ti on 2 . 28 -Polar G auge The polar of a gauge γ C is the function γ

• C defined by γ • C (u) = inf {µ 0 | ∀x ∈ R n , x, u µγ C (x)} .
Observe that gauges polar to each other have the property

∀ (x, u) ∈ dom γ C × dom γ • C , x, u γ C (x)γ • C (u) ,
just as dual norms satisfy a duality inequality. In fact, polar pairs of gauges correspond to the best inequalities of this type. The following Lemma 2.5

shows the relation between polar sets and polar gauges.

L emma 2 .5 Let C ⊆ R n be a closed convex set containing 0. Then, (i) γ • C is a gauge function and γ •• C = γ C . (ii) γ • C = γ C • , or equivalently C • = {x ∈ R n | γ • C (x) 1} = {x ∈ R n | γ C • (x) 1} .
(iii) The gauge of C and the support function of C are mutually polar, i.e. (Rockafellar 1996, Theorem 15.1). (ii) (Rockafellar 1996, Corollary 15.1.1)

γ C = σ C • and γ C • = σ C . P ro o f (i) follows from

Subdifferential of a Gauge

The subdifferential of a gauge γ C at a point x is completely characterized by the face of its polar set C • exposed by x. Put formally, we have, P roposi ti on 2 .8 Let C be a convex set containing 0. Then,

∂γ C (x) = F C • (x) = η ∈ R N | η ∈ C • and η, x = γ C (x) ,
where The special case of x = 0 has a much simpler structure; it is the polar set

F C • (x)
C • from Lemma 2.5(ii)-(iii), i.e. ∂γ C (0) = η ∈ R N | γ C • (η) 1 = C • .

Polar Calculus

We here derive the expression of the gauge function of the Minkowski sum of two sets, as well as that of the image of a set by a linear operator. These results play an important role in Chapter 7.

First of all, we prove that if a multivalued mapping is Lipschitz, then the polar mapping is also Lipschitz continuous.

Lemma 2 . 6 Let C : X ⇒ R n be a β C -Lipschitz multivalued mapping, such that C(x) is a compact convex set containing 0 for every x ∈ X. Then C • defined by as x → C(x) • is β C -Lipschitz and the mapping x → γ C(x) is β C -Lipschitz. p ro o f Using the Lipschitz continuity of C, there exists β C(x) such that C(x ′ ) ⊆ C(x) + β C ||x ′ -x||B X ,
Using the symmetry of B X , we get that

C(x ′ ) + β C ||x ′ -x||B X ⊆ C(x).
Since the polarity reverse the order for the inclusion, we have

(C(x ′ ) + β C ||x ′ -x||B X ) • ⊇ C(x) • .
Hence,

σ (C(x ′ )+β C ||x ′ -x||B X ) • σ C(x) • , or equivalently γ C(x ′ )+β C ||x ′ -x||B X γ C(x) . (2.6)
According to Lemma 2.7, one has

γ C(x ′ )+β C ||x ′ -x||B X (u) = inf z∈R n max(γ C(x ′ ) (u), γ β C ||x ′ -x||B X (u -z)).
Hence,

γ C(x ′ )+β C ||x ′ -x||B X (u) γ C ( x ′ ) (u) + γ β C ||x ′ -x||B X (u) = γ C(x ′ ) (u) + β C ||x ′ -x||||u||.
Hence, combining with (2.6), we get

γ C(x ′ ) (u) + β C ||x ′ -x||||u|| γ C(x) ,
or equivalently,

γ C(x) • γ C(x ′ ) • (u) + β C ||x ′ -x||||u||,
which concludes the proof.

Minkowski Sum

Recall that the Minkowski sum of two sets A and B, subsets of R n is defined as

A + B = {a + b ∈ R n | a ∈ A, b ∈ B} .
In particular, the (Minkowski) sum of two convex sets containing zero is a convex set containing zero. The following Lemma 2.7 makes a connection between the gauge of C 1 + C 2 with gauges of C 1 and C 2 through the infconvolution operator.

Lemma 2 . 7 Let C 1 and C 2 be nonempty closed convex sets containing the origin. Then

γ C 1 +C 2 (x) = sup ρ∈[0,1] ργ C 1 + ∨ (1 -ρ)γ C 2 (x) . If x is such that γ C 1 (x 1 ) + γ C 2 (x 2 ) is continuous and bounded on {(x 1 , x 2 ) | x 1 + x 2 = x}, then γ C 1 +C 2 (x) = inf z∈R n max(γ C 1 (z), γ C 2 (x -z)) .
p ro o f We have from Lemma 2.5 and calculus rules on support functions,

γ (C 1 +C 2 ) • = σ C 1 +C 2 = σ C 1 + σ C 2 .
Thus,

(C 1 + C 2 ) • = {u | σ C 1 (u) + σ C 2 (u) 1} . (2.7)
Using the fact that the gauge of a set C is the support function of its polar, we have

γ C 1 +C 2 (x) = σ (C 1 +C 2 ) • (x). 64 2.3 Gauges Using (2.7), γ C 1 +C 2 (x) = σ σ C 1 (u)+σ C 2 (u) 1 (x).
By definition of the support function,

γ C 1 +C 2 (x) = sup σ C 1 (u)+σ C 2 (u) 1 u, x . Introduction ρ = σ C 1 (u) + σ C 2 (u), we rewrite it as γ C 1 +C 2 (x) = sup ρ∈[0,1] sup σ C 1 (u) ρ,σ C 2 (u) 1-ρ u, x .
This yields

γ C 1 +C 2 (x) = sup ρ∈[0,1] σ σ C 1 (u) ρ + ∨ σ σ C 2 (u) 1-ρ (x) = sup ρ∈[0,1] ρσ σ C 1 (u) 1 + ∨ (1 -ρ)σ σ C 2 (u) 1 (x).
By definition of the polarity,

γ C 1 +C 2 (x) = sup ρ∈[0,1] ρσ C • 1 + ∨ (1 -ρ)σ C • 2 (x) = sup ρ∈[0,1] σ ρC • 1 + ∨ σ (1-ρ)C • 2 (x) = sup ρ∈[0,1] ργ C 1 + ∨ (1 -ρ)γ C 2 (x) ,
which is the first assertion.

The last identity can be rewritten

γ C 1 +C 2 (x) = sup ρ∈[0,1] inf x 1 +x 2 =x ργ C 1 (x 1 ) + (1 -ρ)γ C 2 (x 2 ) .
Under the boundedness and continuity assumption of the lemma, the objective 2 in the sup inf is a continuous bounded concave-convex function on the set

[0, 1] × {(x 1 , x 2 ) | x 1 + x 2 = x}.
Since the latter sets are non-empty, closed and convex, and [0, 1] is obviously bounded, we have from using (Rockafellar 1996, Corollary 37.3.2)

γ C 1 +C 2 (x) = inf z∈R n sup ρ∈[0,1] ργ C 1 (z) + (1 -ρ)γ C 2 (x -z) = inf z∈R n max(γ C 1 (z), γ C 2 (x -z)) ,
which concludes the proof.

Image of a Set by a Linear Operator

Considering a linear operator

D : R p → R n , one constructs the image D(C) of a convex set C ⊆ R p by D(C) = {Dx ∈ R n | x ∈ C} .
γ (D(C)) • = σ D(C) = ι D(C) * = σ C • D * .
Now, as by assumption 0 ∈ ri C, we have 0 ∈ ri(C • ), and therefore

Im(D * ) ∩ ri(C • ) = ∅.
γ D(C) (x) = σ (D(C)) • (x) = σ σ C •D * (u) 1 (x) = ι σ C (w) 1 • D * * (x) = inf v σ σ C (w) 1 (v) s.t. Dv = x = inf z∈Ker(D) σ σ C (w) 1 (D + x + z) = inf z∈Ker(D) σ σ C (w) 1 (D + x + z) = inf z∈Ker(D) γ C (D + x + z) ,
which concludes the proof.

In particular, if Ker

D = 0, then γ D(C) (x) = γ C (D + x). Using Lemma 2.4(v), one can observe that the infimum is bounded if (D + x + Ker(D)) ∩ span C = ∅.

Lift to Matrix Spaces

We recall the singular value decomposition theorem.

P roposi ti on 2 .9 For any matrix A ∈ R n 1 ×n 2 , there exists three matrices

U ∈ R n 1 ×n 1 , Σ ∈ R n 1 ×n 2 , V ∈ R n 2
×n 2 such that U and V are orthogonal matrices, Σ is empty outside its main diagonal and A = UΣV * . The matrix Σ is unique, up to permutation.

p ro o f See (Horn et al. 2012, Theorem 7.3.3).

Denoting n = min(n 1 , n 2 ), we call the diagonal elements of Σ the singular values of A, denoted (σ i (A)) 1 i n . Thus we define a function σ : R n 1 ×n 2 → R n such that σ 1 (A) σ 2 (A) . . . σ n (A).
We start by the following definition Defi ni ti on 2 .29 A function f : R n → R is an absolutely symmetric gauge if f is a gauge and is absolutely symmetric, i.e.

∀x ∈ R n , ∀P ∈ P n , f(Px) = f(x),
where P n is the set of all signed permutation matrices of {1, . . . , n},

The following proposition makes a connection between absolutely symmetric gauges and unitarily invariant norms, i.e. norms F such that F(UΛ x V * ) = F(Λ x ).

P roposi ti on 2 .10 There is a one-to-one correspondance between absolutely symmetric gauges and unitarily invariant norms. More precisely, (i) If f is an absolutely symmetric gauge, then F = f • σ is a unitarily invariant norm.

(ii) If F is a unitarily invariant norm, then f = F • diag is an absolutely symmetric gauge.

p ro o f The proof might be found in [START_REF] Von Neumann | Collected works[END_REF] or (Horn et al. 2012, Theorem 7.4.7.2, p 464).

For instance, the nuclear norm is nothing more than a unitarily invariant norm induced by the ℓ 1 -norm || • || 1 and the spectral norm is induced by the ℓ ∞ -norm.

Operator Bounds

Since we use more general regularizers than norms, we have to generalize the concept of operator norm. Recall that if (V, || • ||) is a normed vector space, we embedded the set of continuous linear operators from V to W using the operator norm, i.e.

|||A||| = sup ||x|| 1 ||Ax|| = sup ||x||=1 ||Ax|| = sup x∈V ||Ax|| x .
This motivates the following Definition 2.30.

D efi ni ti on 2 . 30 Let J 1 and J 2 be two gauges defined on two vector spaces V 1 and V 2 , and

A : V 1 → V 2 a linear map. The operator bound | | |A| | | J 1 →J 2 of A between J 1 and J 2 is given by | | |A| | | J 1 →J 2 = sup J 1 (x) 1 J 2 (Ax). Note that | | |A| | | J 1 →J 2 < +∞ if, and only if A Ker(J 1 ) ⊆ Ker(J 2 ). In particular, if J 1 is coercive (i.e. Ker J 1 = {0} from Lemma 2.4(iv)), then | | |A| | | J 1 →J 2 is finite. As a convention, | | |A| | | J 1 →||•|| p is denoted as | | |A| | | J 1 →ℓ p .
An easy consequence of this definition is the fact that for every

x ∈ V 1 , J 2 (Ax) | | |A| | | J 1 →J 2 J 1 (x).

O-minimality

The goal of o-minimal geometry is to prevent pathological behavior with respect to the common operations on sets, such as addition and projection.

To expose our motivation, we take the example of [START_REF] Coste | An Introduction to O-minimal Geometry[END_REF]. Consider the function f :

x → sin 1 x defined on R * + and G its graph on R 2 . Denote Ḡ the closure of G in R 2 . Then, dim( Ḡ \ G) = dim G in the Hausdorff
sense. This is typically this kind of behavior that we wish to avoid.

Definition

We briefly recall here the definition and the main properties of o-minimal structures, that are used for our proof. We refer to [START_REF] Dries | Tame topology and o-minimal structures[END_REF][START_REF] Coste | An Introduction to O-minimal Geometry[END_REF] for more details about o-minimal structures.

O-minimal geometry can been seen as a generalization of the notion of semialgebraicity.

Defi ni ti on 2 .31 -S emi-algebrai c Subsets . The semi-algebraic subsets of R n are the smallest set SA n of subsets of R n such that:

(i) For every real polynomial P ∈ R[X 1 , . . . , X n ], {x ∈ R n | P(x) = 0} ∈ SA n and {x ∈ R n | P(x) > 0} ∈ SA n . (ii) If A, B ∈ SA n , then A ∪ B, A ∩ B, R n \ A ∈ SA n .
The following result is central in the study of semi-algebraic sets. Theorem 2 . 2 -Tarski -S ei denberg. The set SA n of semi-algebraic sets is closed under projection.

We now define o-minimal structures.

Defi ni ti on 2 .32 -S tructure . An o-minimal structure O expanding R is a sequence of sets (O n ) n∈N which satisfies the following axioms:

(i) Each O n is a Boolean algebra of subsets of R n , with R n ∈ O n . (ii) Every semi-algebraic subset of R n is in O n , i.e. SA n ⊆ O n . (iii) If A ∈ O n and B ∈ O n ′ , then A × B ∈ O n+n ′ . (iv) If A ∈ O n+1 , then Π(A) ∈ O n ,
, i.e. Ω ⊂ R n is definable if Ω ∈ O n . A map f : Ω → R p is said to be definable if its graph G(f) = {(x, u) ∈ Ω × R p | u = f(x)} ⊆ R n × R p is a definable subset of R n × R p .
Note that in this case, the application p times of axiom (iv) implies that Ω is definable. The fundamental example of o-minimal structure is the set of semi-algebraic sets, which is in some sense the smallest o-minimal structure.

For example, note that in the special case where q is a rational number, the functionals || • || q are actually semi-algebraic. When q ∈ R is not rational, then || • || q is not semi-algebraic, but it can be shown to be definable in a o-minimal structure.

Properties

In the following results, we collect some important stability properties of ominimal structures. To be self-contained, we also provide proofs. To the best of our knowledge, these proofs, although simple, are not reported in the literature or some of them are left as exercices in the authoritative references [START_REF] Dries | Tame topology and o-minimal structures[END_REF][START_REF] Coste | An Introduction to O-minimal Geometry[END_REF]. Moreover, in most proofs, to show that a subset is definable, we could just write the appropriate first-order formula, see (Coste 1999, Page 12) and (Dries 1998, Section Ch1.1.2), and conclude using (Coste 1999, Theorem 1.13). Here, for the sake of clarity and avoid cryptic statements for the non-specialist, we translate the first order formula into operations on the involved subsets, in particular projections, and invoke the above stability axioms of o-minimal structures.

Lemma 2 . 9 -A ddi ti on and Multi pli cati on. Let Ω a subset of R n . Let f : Ω → R p and g : Ω ⊂ R n ⊂ R p be definable functions. Then their pointwise addition and multplication is also definable.

p ro o f Let h = f + g, and

B = (Ω × R × Ω × R × Ω × R) ∩ (Ω × R × G(f) × G(h)) ∩ S
where S = {(x, u, y, v, z, w) | x = y = z, u = v + w} is obviously an algebraic (in fact linear) subset, hence definable by axiom 2. Property 1 implies that B is also definable. Let Π 3n+3p,n+p : R 3n+3p → R n+p be the projection on the first n + p coordinates. We then have

G(h) = Π 3n+3p,n+p (B)
whence we deduce that h is definable by applying 3n + 3p times axiom 4. Definability of the pointwise mutplication follows the same proof taking

u = v • w in S. Lemma 2 . 10 -I nequali ti es i n Defi nable S ets. Let f : Ω ⊂ R n → R be a definable function. Then {x ∈ Ω | f(x) > 0}
, is definable. The same holds when replacing > with <.

Clearly, inequalities involving definable functions are accepted when defining definable sets.

There are many possible proofs of this statement.

p ro o f ( 1 ) Let B = {(x, y) ∈ R × R | f(x) = y} ∩ (Ω × (0, +∞)
, which is definable thanks to axioms 1 and 3, and that the level sets of a definable function are also definable. Thus

{x ∈ Ω | f(x) > 0} = {x ∈ Ω | ∃y, f(x) = y, y > 0} = Π n+1,n (B) ,
and we conclude using again axiom 4.

Yet another (simpler) proof.

p ro o f ( 2 ) It is sufficient to remark that {x ∈ Ω | f(x) > 0} is the projection of the set (x, t) ∈ Ω × R | t 2 f(x) -1 = 0
, where the latter is definable owing to Lemma 2.9.

L emma 2 .11 -D eri vati ve . Let f : I → R be a definable differentiable function on an open interval I of R. Then its devivative f ′ : I → R is also definable.

p ro o f Let g : (x, t) ∈ I × R → g(x, t) = f(x + t) -f(x)
. Note that g is definable function on I × R by Lemma 2.9. We now write the graph of f ′ as

G(f ′ ) = {(x, y) ∈ I × R | ∀ε > 0, ∃δ > 0, ∀t ∈ R, |t| < δ, |g(x, t) -yt| < ε|t|} . Let C = (x, y, v, t, ε, δ) ∈ I × R 5 | ((x, t), v) ∈ G(g)
, which is definable since g is definable and using axiom 3. Let

B = (x, y, v, t, ε, δ) | t 2 < δ 2 , (v -ty) 2 < ε 2 t 2 ∩ C .
The first part in B is semi-algebraic, hence definable thanks to axiom 2. Thus B is also definable using axiom 1. We can now write

G(f ′ ) = R 3 \ Π 5,3 R 5 \ Π 6,5 (B) ∩ (I × R) ,
where the projectors and completions translate the actions of the existential and universal quantifiers. Using again axioms 4 and 1, we conclude.

With such a result at hand, this proposition follows immediately.

P roposi ti on 2 .11 -D i fferenti al and J acobi an.

Let f = (f 1 , • • • , f p ) : Ω → R p be a differentiable function on an open subset Ω of R n .
If f is definable, then so its differential mapping and its Jacobian. In particular, for each

i = 1, • • • , n and j = 1, • • • , p, the partial derivative ∂f i /∂x j : Ω → R is definable.
Lemma 2 . 12 -M argi nal Functi on. Let g : R n × R m → R be a definable function, and Ω a definable subset of R m . The function

f(x) = sup y∈Ω g(x, y)
is definable. The same conclusion holds true with inf instead of sup.

p ro o f Let the subset

B = {(x, u, y) ∈ R n × R × R m | g(x, y) > u} ∩ (R n × R × Ω) .
B is definable thanks to Lemma 2.10 and axiom 1. Projecting on the components (x, u), we get the set

Π n+1+m,n+1 (B) = {(x, u) ∈ R n × R | ∃y ∈ Ω, g(x, y) > u} whose complement is R n+1 \ Π n+1+m,n+1 (B) = {(x, u) ∈ R n × R | ∀y ∈ Ω, g(x, y) u} = epi(f) ,
and therefore, epi(f) is definable using axioms 4 and 1. Similarly, replacing > with < in B, we get that the hypograph hypo(f) is definable. Thus G(f) = hypo(f) ∩ epi(f) is definable by axiom 1. The proof for the inf is similar.

As applications of this result, it follows that the Legendre-Fenchel conjugate of a definable function is definable, that the support function of a definable set is definable, and that the infimal convolution of definable functions is also definable.

P roposi ti on 2 .12 -Polars and Gauges . Let Ω be a closed convex subset of R n containing the origin. Then the following assertions are equivalent:

(i) Ω is definable.

(ii) The polar set Ω • is definable.

(iii) The gauge f of Ω is definable.

(iv) The polar gauge f • is definable.

p ro o f (i) ⇐⇒ (ii):
We have

Ω • = {u ∈ R n | ∀x ∈ Ω, x, u 1} = R n \ Π 2n,n ({(u, x) ∈ R n × Ω | x, u > 1}) .
We conclude that Ω • is definable using axioms 1-4. The converse statement follows by exchanging the roles of Ω and Ω • .

(i) ⇐⇒ (iii): f is the support function of Ω • f(x) = σ Ω • (x) = sup u∈Ω • x, u ,
We get that f is definable using (i)⇒(ii), and applying Lemma 2.12 with g(x, u) = x, u , which is obviously definable by axiom 2. The converse statement follows from Lemma 2.10 since

Ω • = {x ∈ R n | f(x) 1} and (ii)⇒(i).
(ii) ⇐⇒ (iv): The proof follows exactly the lines of the previous item replacing f and Ω with their polars.

P roposi ti on 2 .13 -S ubdi fferenti al of a Gauge . Let f be the gauge of a closed convex subset Ω of R n containing the origin as an interior point. Suppose that f is definable. Then for any x ∈ R n , the subdifferential ∂f(x) is definable.

p ro o f Let Ω • be the polar set of Ω. We have ∀x ∈ R n ∂f(x) = Argmax u∈Ω • x, u = {u ∈ Ω • | x, u = f(x)} ,
i.e. the exposed face of Ω • associated with x, which is a non-empty convex compact set for all x. Thus, since f is definable, so is Ω • by Proposition 2.12, and ∂f(x) is also definable by axioms 1-2 (the intersection of Ω • and a linear set).

Lemma 2 . 13 -G raph of the Relati ve I nteri or . Let f be the gauge of a closed convex subset Ω of R n containing the origin in its interior. Suppose that f is definable. Then, the set

{(x, η) | η ∈ ri ∂f(x)} is definable. p ro o f Denote C = {(x, η) | η ∈ ri ∂f(x)}.
Combining the characterization of the relative interior of a convex set (Rockafellar 1996, Theorem 6.4) and the structure of the subdifferential of a bounded gauge, which is non-empty convex, see Proposition 2.13), we rewrite C in the more convenient form

C =    (β, η) | ∀u ∈ Ω • and u, β = f(β), ∃t > 1 s.t    (1 -t)u + tη ∈ Ω o η, β = f(β)    . Let B u = (R n \ Ω • ) ∪ (R n \ {u | u, β = f(β)}) and B =(R n × R n × B u ×]1, +∞[×Ω • ) ∩ {(β, η, u, t, ξ) | η, β = f(β), ξ = (1 -t)u + tη} .
B u is definable by virtue of Proposition 2.12, axiom 1, and the fact it involves algebraic relations and the level sets of a definable function. In turn, B is definable owing to Proposition 2.12 and axioms 1-3. It then results that

C = R n \ Π 4n+1,2n (B) ,
which is then definable after axioms 4 and 1. T he purpose of this chapter is to introduce one of the main concepts used throughout this thesis, the model tangent subspace associated to a convex function. The main result, Theorem 3.1 of this chapter, proves that the subdifferential of any convex function exhibits some kind of decomposability property.

It is known that the subdifferential, see for instance [START_REF] Fuchs | On sparse representations in arbitrary redundant bases[END_REF]), of the ℓ 1norm exhibits a particular structure. More precisely, for any x ∈ R n ,

∂|| • || 1 (x) = {η ∈ R n | η I = sign(x) I and ||η I c || ∞ 1} ,
where I = supp(x), the support of x. Such a structure is very convenient when dealing with (P y,λ ) or (P y,0 ), since it allows to split the analysis between "active" components of x and non-active. One may ask if such a splitting is always possible for a convex function. More precisely, we aim to split the subdifferential onto an orthogonal decomposition of R n in a structured way. We answer positively in the following sections, replacing the support I by what we coined the tangent model subspace, the sign pattern sign(x) by the model vector and the ℓ ∞ -norm by the subdifferential gauge. However, this decomposition will be pointwise, an issue that is solved in Chapter 4.

Model Tangent Subpace Associated to a Convex Function

This section introduces the model tangent subspace associated to a convex function, and its associated model vector. We can compute the model tangent space of a sum of convex functions at some point, and also by precomposition by a linear operator. This section is illustrated by two examples, the ℓ1 ℓ 2 norm and the ℓ ∞ norm. Section 3.3 provides several other examples to connection these definition to practical applications.

The terminology of model tangent subspace is partly explained in this chapter. Indeed, the tangent aspect is a consequence of a further property, partial smoothness, that is studied in Chapter 4. Nevertheless, we stick with this name right now. We denote S x = Sx -e x = span(∂J(x)) and

Model Tangent Subspace

Let J ∈ Γ + c (R n ) a regularizer 1 , i.
T x = S ⊥ x .
T x is coined the model tangent subspace of x associated to J.

When J is Gâteaux-differentiable at x, i.e. ∂J(x) = {∇J(x)}, e x = ∇J(x) and

T x = R N .
On the contrary, when J is not smooth at x, the dimension of T x is of smaller dimension, and the regularizing function J essentially promotes elements living on or close to this model subspace.

We start by summarizing some key properties of e x and T x .

P roposi ti on 3 .1 Let J ∈ Γ + c (R n ). For any x ∈ R N , one has (i) e x ∈ T x ∩ Sx , (ii) Sx = η ∈ R N | η T x = e x .
Moreover, if J is a gauge, then

(iii) For every u ∈ Sx , J(x) = u, x , (iv) x ∈ T x .
p ro o f (i) This is due to the fact that e x is the orthogonal projection of 0 on the affine space Sx . It is therefore an element of Sx ∩ ( Sx -e x ) ⊥ , i.e. e x ∈ Sx ∩ T x .

(ii) This is straightforward from the fact that

S x = η ∈ R N | η T x = 0 , Sx = S x + e
x and e x ∈ T x from (i).

(iii) Each element of Sx can be written as u = k i=1 ρ i η i , for k > 0, where η i ∈ ∂J(x) and k i=1 ρ i = 1. By Fenchel identity applied to the gauge J, and using Lemma 2.5(iii), we have

x, η i = J(x) + ι C • (η i ), ∀i . Since η i ∈ ∂J(x) ⊆ C • , we get x, η i = J(x), ∀i .
Multiplying by ρ i and summing this identity over i and using the fact that k i=1 ρ i = 1 we obtain the desired result.

(iv) For any v ∈ S x , we have v + e x ∈ Sx since e x ∈ Sx . Thus applying (i), we get

x, e x + v = J(x) and x, e x = J(x). Combining both identities implies that x, v = 0, ∀v ∈ S x , or equivalently that x ∈ S ⊥ x = T x .

In general e x ∈ ∂J(x), which is the situation displayed on Figure 3.1. In this figure , x is an element of T x which is not the case for a convex function. This is however always the case if J is a gauge, as stated in Proposition 3.1(iv).

Algebraic Stability

The following proposition determines the model tangent subspace of the sum of two functions.

H = J + G
in terms of those associated to J and G.

P roposi ti on 3 .2 Let J, G ∈ Γ + c (R n ).
Denote T J and e J (resp. T G and e G ) the model tangent subspace and vector at a point x corresponding to J (resp. G). Then the model tangent subspace at

x of H = J + G reads (i) T H = T J ∩ T G , or equivalently S H = (T H ) ⊥ = span S J ∪ S G . (ii) e H = P T H (e J + e G ). p ro o f (i) First, we have ∂H(x) = ∂J(x) + ∂G(x), Let S J = span(∂J(x) -η J ) and S G = span(∂G(x) -η G ), for any pair η J ∈ ∂J(x) and η G ∈ ∂G(x). Choosing η H = η J + η G ∈ ∂H(x) we have S H = span(∂H(x) -η H ) = span (∂J(x) -η J )+(∂G(x) -η G ) = span span(∂J(x) -η J )+ span(∂G(x) -η G ) = span(S J ∪ S G ).
As a consequence we have

T H = (S H ) ⊥ = T J ∩ T G . (ii) Moreover, since T H ⊥ S J ∪ S G we have from Proposition 3.1(iii) that e H = P T H (∂H(x)) = P T H (∂J(x)+∂G(x)) = P T H (e J + P S J ∂J(x) + e G + P S G ∂G(x)) = P T H (e J + e G ).
Functions of the form J 0 • D * , where

J 0 ∈ Γ + c (R n
) is a bounded regularizing convex function, correspond to the so-called analysis-type regularizers. In the following, we denote T = T x = S ⊥ and e = e x the subspace and vector in the decomposition of the subdifferential of J at a given x ∈ R N . Analogously, It is common in the litterature [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF]) to find regularization of the form

T 0 = S ⊥
J ε (x) = J(x) + ε 2 ||x|| 2
2 in order to stabilize the numerical optimization. More generally, we consider a function G which is Gâteaux-differentiable.

C orollary 3 .1 Let J ∈ Γ + c (R n ), x ∈ R n and G ∈ Γ + c (R n ) a function which is Gâteaux-differentiable at x. Then, T J+G x = T J and e J+G x = e J
x + P T J x ∇G(x).

p ro o f Indeed, since ∂G(x) = {∇G(x)}, we obtain that T G x = R n and e G x = ∇G(x).

Applying Proposition 3.2, we get the result.

Hence, the model tangent space does not vary with the pertubation G (unlike e x ). Remark that the function

G : x → ε 2 ||x|| 2 2 is C ∞ everywhere. If J is a gauge, hence x ∈ T x ,
we get that the model vector reads e J+G x = e J x + εx.

Examples

We illustrate the definition of model tangent subspace with two norms, the ℓ 1 -ℓ 2 norm used in structured sparsity and the ℓ ∞ -norm used for spread representations as discussed in the introduction.

The ℓ 1 -ℓ 2 norm. We consider a uniform disjoint partition

B of {1, • • • , n}, {1, . . . , N} = b∈B b, b ∩ b ′ = ∅, ∀b = b ′ . The ℓ 1 -ℓ 2 norm of x is J(x) = ||x|| B = b∈B ||x b ||. P roposi ti on 3 .4 Let J = || • || B . The tangent model space of J at x = 0 reads T x = η ∈ R N | ∀b ∈ I, η b = 0 , where I = {b ∈ B | x b = 0}
, and its orthogonal S x reads

S x = Sx -e x = {η ∈ R n | ∀b ∈ I, η b = 0} .
It model vector reads

e x = (N(x b )) b∈B ,
where N(a) = a/||a|| if a = 0, and N(0) = 0.

p ro o f The subdifferential of J at x ∈ R n is ∂J(x) = η ∈ R n | ∀b ∈ I, η b = x b ||x b || and ∀g ∈ I, ||η g || 1 .
Thus, the affine hull of ∂J(x) reads

Sx = η ∈ R n | ∀b ∈ I, η b = x b ||x b || .
Hence the projection of 0 onto Sx is

e x = (N(x b )) b∈B and S x = Sx -e x = {η ∈ R n | ∀b ∈ I, η b = 0} , which completes the proof. The ℓ ∞ norm. The ℓ ∞ norm is J(x) = ||x|| ∞ = max 1 i n |x i |. P roposi ti on 3 .5 Let J = || • || ∞ . The tangent model space of J at x = 0 reads T x = η | η (I) = ρ(sign(x)) (I) for ρ ∈ R ,
where For x = 0, we have

I = {i ∈ {1, . . . , n} | |x i | = ||x|| ∞ } and
∂J(x) = {η | ∀ i ∈ I c , η i = 0, η, s = 1, η i s i > 0 ∀ i ∈ I} .
It is clear that Sx is the affine hull of an |I|-dimensional face of the unit ℓ 1 ball exposed by the sign subvector s (I) . Thus e x is the barycenter of that face, i.e.

e x = s/|I| and S x = η | η (I c ) = 0 and η (I) , s (I) = 0 .
In turn, we have the expression of T x .

The Decomposability Property

In the previous section, we defined the model tangent subspace and the model vector. They are going to play a key role in structuring the subdifferential of J.

The following proposition gives an equivalent convenient description of the subdifferential of a gauge γ C at x in terms of a particular supporting hyperplane to C • : the affine hull Sx .

P roposi ti on 3 .6 Let γ C be a finite-valued gauge. Then for

x ∈ R N , one has ∂γ C (x) = Sx ∩ C • . p ro o f Let x ∈ R N . We have ∂γ C (x) = F C • (x) = H ∩ C • ,
where

H = η ∈ R N | η, x = J(x)
is the supporting hyperplane of C • at x. By Proposition 3.1(iii), we have

Sx = aff ∂γ C (x) ⊆ H, which implies that Sx ∩ C • ⊆ H ∩ C • .
The converse inclusion is true since ∂γ

C (x) = H ∩ C • ⊆ Sx .
Note that this property holds only for gauges. In the following, we propose an alternative for any kind of convex function.

The Subdifferential Gauge

Definition

Before providing an equivalent description of the subdifferential of J at x in terms of the geometrical objects e x , T x and S x , we introduce a gauge that plays a prominent role in this description.

D efi ni ti on 3 . 2 -S ubdi fferenti al Gauge . Let J ∈ Γ + c (R n ). Let x ∈ R N \ {0} and f x ∈ ri ∂J(x). The subdifferential gauge associated to f x is the gauge J x,• f x = γ ∂J(x)-f x .
Since ∂J(x)f x is a closed (in fact compact) convex set containing the origin as a relative interior point, it is uniquely characterized by the subdifferential gauge

J x,• f x (see Lemma 2.4(i)).
The following proposition states the main properties of the gauge J x,• f x .

P roposi ti on 3 .7 The subdifferential gauge J x,• f x is such that dom J x,• f x = S x , and is coercive on S x . Moreover, if J is a gauge, then

J x,• f x (η) = inf τ 0 max(J • (τf x + η), τ) + ι S x (η) .
p ro o f The first assertion follows from Lemma 2.4(v) since 0 ∈ ri(∂J(x)f x ).

Let's now turn to the second part. Since

f x ∈ ri ∂J(x) ⊂ Sx , Proposition 3.1 implies that f x = P S x (f x ) + P T x (f x ) = P S x (f x ) + e x .
Hence, using Proposition 3.6, we get

∂J(x) -f x = (C • -f x ) ∩ ( Sx -f x ) = (C • -f x ) ∩ (S x -{P S x (f x )}) = (C • -f x ) ∩ S x . J x,• f x (η) = γ (C • -f x )∩S x (η) = max(γ C • -f x (η), γ S x (η)) = max(γ C • -f x (η), ι S x (η)) = γ C • -f x (η) + ι S x (η) .
At this stage, Lemma 2.7 does not apply straightforwardly since 0 ∈ C • but f x = 0 in general. However, proceeding as in the proof of that lemma, we arrive at

γ C • +{-f x } (η) = sup ρ∈[0,1] ρJ • + ∨ (1 -ρ)σ {-f x } • (η)
where, from Definition 2.27,

{-f x } • = {η | η, f x -1}
, which indeed contains the origin as an interior point. Continuing from the last equality, we get

γ C • +{-f x } (η) = sup ρ∈[0,1] ρJ • + ∨ (1 -ρ)γ {-f x } •• (η) = sup ρ∈[0,1] ρJ • + ∨ (1 -ρ)γ co({-f x }∪{0}) (η) = sup ρ∈[0,1] ρJ • + ∨ (1 -ρ)γ {-µf x | µ∈[0,1]} (η) .
It is easy to see that

γ {-µf x | µ∈[0,1]} (-η) =    τ if η ∈ τf x , τ ∈ R + , +∞ otherwise . Thus γ C • +{-f x } (η) = sup ρ∈[0,1] inf τ 0 ρJ • (τf x + η) + (1 -ρ)τ .
Recalling that J • is a finite-valued gauge, hence continuous, the objective in the sup inf fulfills the assumption of the second assertion of Lemma 2.7, whence we get

γ C • +{-f x } (η) = inf τ 0 max(J • (τf x + η), τ) ,
which completes the proof.

The second claim gives a formula which links J x,• f x to the polar gauge J • . But they are not equal in general unless some additional assumptions are imposed on J, as we will see shortly.

The Polar of the Subdifferential Gauge

We now turn to the gauge polar to the subdifferential gauge defined by the relation

(J x,• f x ) • = J x f x . J x f
x comes into play in several results in the rest of the manuscript. The following proposition summarizes its most important properties.

P roposi ti on 3 .8 The gauge J x f x is such that:

(i) Its has a full domain.

(ii)

J x f x (d) = J x f x (d S ) = sup J x,• fx (η Sx ) 1 η S x , d , where S = S x . (iii) Ker J x f x = T x and J x f x is coercive on S x .
Moreover, if J is a gauge, (iv)

J x f x (d) = J(d S x ) -f S x , d S x
p ro o f The gauge J x f x is the support function of the set

K x def. = ∂J(x) -f x = η ∈ R N | J x,• f x (η) 1 ⊂ S x ,
where the inclusion follows from Proposition 3.7.

(i) Since K x is a bounded set, its support function is also finite-valued (Hiriart-

Urruty et al. 2001, Proposition V.2.1.3). It follows that dom J x f x = R n .
(ii) We have

J x f x (d) = sup η∈K x η, d = sup J • fx (η) 1 η, d = sup J x,• fx (η Sx ) 1 η S x , d = sup η∈K x η, d T x + η, d S x = sup η∈K x η, d S x = J x f x (d S x ) ,
where we used the fact that η, d T x = 0 on K x .

(iii) As a consequence of (ii), J x f x (d T x ) = 0. Clearly, T x ⊂ Ker(J x f x ) and J x f x is constant along all affine subspaces parallel to T x . But, since 0 ∈ ri K x , excluding the origin, the supremum in J f x is always attained at some nonzero η ∈ K x ⊂ S x . Thus J x f x (d) > 0 for all d such that d / ∈ T x . This shows that actually Ker(J x f x ) = T x . In particular, this yields that on S x , the gauge J x f x is coercive.

(iv) Using some calculus rules with support functions and assertion (ii), we have

J x f x (d) = J x f x (d S x ) = σ (C • +{-f x })∩S x (d S x ) = co inf(σ C • +{-f x } (d S x ), σ S (d S x )) = co inf(σ C • +{-f x } (d S x ), ι T (d S x )) = σ C • +{-f x } (d S x ) = σ C • (d S x ) -P S x (f x ), d S x = J(d S x ) -P S x (f x ), d S x ,
which completes the proof.

Main Result

Piecing together the above ingredients yields a fundamental pointwise decomposition of the subdifferential of the regularizer J.

Theorem 3 .1 -D ecomposabi li ty. Let J ∈ Γ + c (R n ). Let x ∈ R N \ {0} and f x ∈ ri ∂J(x). Then the subdifferential of J at x reads ∂J(x) = η ∈ R N | η T x = e x and J x,• f x (P S x (η -f x )) 1 .
p ro o f Invoking Proposition 3.1, we get that for every η ∈ ∂J(x), η T x = e x , and

P T x (f x ) = e x .
It remains now to uniquely characterize the part of the subdifferential lying in S x , i.e. ∂J(x)e x . Since f x ∈ ri ∂J(x), we have from the one-to-one correspondence of Lemma 2.4(i) and the definition of the subdifferential gauge,

η ∈ η ∈ R N | J x,• f x (η S x -P S x (f x )) 1 ⇐⇒ η S x -P S x (f x ) ∈ ∂J(x) -f x ⇐⇒ η S x ∈ ∂J(x) -e x ⇐⇒ η ∈ ∂J(x) ,
which completes the proof.

Capitalizing on Theorem 3.1, we are now able to deduce a convenient necessary and sufficient first-order (global) minimality condition of (P y,λ ) and (P y,0 ). P roposi ti on 3 .9 Let x ∈ R n , and denote for short T = T x and S = S x . The two following propositions hold.

(i) The vector x is a global minimizer of (P y,λ ) if, and only if,

Φ * T (y -Φx) = λe x and J x,• f x (λ -1 Φ * S (y -Φx) -P S (f x )) 1.
(ii) The vector x is a global minimizer of (P y,0 ) if, and only if, there exists a dual vector p ∈ R q such that

Φ * T p = e x and J x,• f x (Φ * S p -P S (f x )) 1.
p ro o f This is a convenient rewriting of the fact that x is a global minimizer if, and only if, 0 is a subgradient of the objective function at x.

(i) For problem (P y,λ ), this is equivalent to

1 λ Φ * (y -Φx) ∈ ∂J(x).
Projecting this relation on T and S yields the desired result.

(ii) Let's turn to problem (P y,0 ). We have at any global minimizer

x 0 ∈ ∂J(x) + Φ * N {p| p=y} (Φx)
where N {p| p=y} (x) is the normal cone of the constraint set {p | p = y} at x, which is obviously the whole space R q . Thus, this monotone inclusion is equivalent to the existence of p ∈ R q such that

Φ * p ∈ ∂J(x) .
Projecting again this on T and S proves the assertion.

These results can be extended easily when 1 2 ||y -Φx|| 2 is replaced by an other data fidelity term.

Decomposability of the Sum and Precomposition by a Linear Operator

Following the same path as for the model space, we establish the subdifferential gauge in the case of the sum and the precomposition by a linear operator. We recall that (Proposition 3.2) denoting T J and e J (resp. T G and e G ) the model tangent subspace and vector at a point x corresponding to J (resp. G), we proved that T H = T J ∩ T G and e H = P T H (e J + e G ). The following proposition describes the subdifferential gauge of H = J + G.

P roposi ti on 3 .10 Let J, G ∈ Γ + c (R n ). Let J x,• f J x and G x,• f G x denote the subd- ifferential gauges for the pairs (J, f J x ∈ ri ∂J(x)) and (G, f G x ∈ ri ∂G(x))
, correspondingly. Then, for the particular choice of

f H x = f J x + f G x we have f H x ∈ ri ∂H(x)
, and for a given η ∈ S H , the subdifferential gauge of H reads

H x,• f H x (η) = inf η 1 +η 2 =η max(J x,• f J x (η 1 ), G x,• f G x (η 2 )) . p ro o f As f J x ∈ ri ∂J(x) and f G x ∈ ri ∂G(x)
, it follows from (Rockafellar 1996, Corollary 6.6.2) that

f H x = f J x + f G x ∈ ri ∂J(x) + ri ∂G(x) = ri (∂J(x) + ∂G(x)) = ri ∂H(x) .
The subdifferential gauge associated to H is then

H x,• f H x = γ ∂H(x)-f H x = γ (∂J(x)-f J x) +(∂G(x)-f G x ) ,
which is coercive and finite-valued on S H according to Proposition 3.7. Invoking Lemma 2.7, we get the desired result since for any ρ 0,

u → ρJ x,• f J x (u) + (1 -ρ)G x,• f G x (η -u) = ργ ∂J(x)-f J x (u) + (1 -ρ)γ ∂G(x)-f G x (η -u)
is finite-valued and continuous on S J ∩ (S G + η), for η ∈ S H = span(S J + S G ).

Similarly, we derive the expression of the subdifferential gauge for an analysistype prior. In this case, according to Proposition 3.3, the tangent model space reads T = Ker(D * S 0 ) = D * T 0 and its model vector e = P T De 0 .

P roposi ti on 3 .11 Let J 0 ∈ Γ + c (R p ). Let J D * x,• 0,f D * x denote the subdifferential gauge for the pair (J 0 , f 0,D * x ∈ ri ∂J 0 (x)). Then, for the particular choice of

f x = Df 0,D * x we have f x ∈ ri ∂J(x), dom J x,• f x = S and for every η ∈ S J x,• f x (η) = inf z∈Ker(D S 0 ) J D * x,• 0,f D * x (D + S 0 η + z) .
The infimum can be equivalently taken over Ker(D) ∩ S 0 .

p ro o f With such a choice of f x , we have

f 0,D * x ∈ ri ∂J 0 (D * x) ⇒ Df 0,D * x ∈ D ri ∂J 0 (D * x) ⇐⇒ f x ∈ ri D∂J 0 (D * x) ⇐⇒ f x ∈ ri ∂J(x) .
We follow the same lines as in the proof of Lemma 2.8, where we additionally invoke Proposition 3.8(ii) to get

J x f x (d) = σ ∂J(x)-f x (d) = σ D(∂J 0 (D * x)-f 0,D * x ) (d) = σ ∂J 0 (D * x)-f 0,D * x (D * d).
Identifying σ ∂J 0 (D * x)-f 0,D * x with the gauge γ ∂J 0 (D * x)-f • 0,D * x , we get

J x f x (d) = J D * x 0,f 0,D * x (D * d) = J D * x 0,f 0,D * x (D * S 0 d) .
Note that J x f x is indeed constant along affine subspaces parallel to Ker(D * S 0 ) = S ⊥ = T . We now get that for every η ∈ S = Ker(D

+ S 0 ) ⊥ J x,• f x (η) = σ J fx (d) 1 (η) = σ J D * x 0,f 0,D * x (D * S 0 d) 1 (η) = ι J D * x 0,f 0,D * x (w) 1 • D * S 0 * (η) = inf v σ J D * x 0,f 0,D * x (w) 1 (v) s.t. D S 0 v = η = inf z∈Ker(D S 0 ) J D * x,• 0,f D * x (D + S 0 η + z) .
The infimum is bounded and is attained necessarily at some z ∈ Ker(D S 0

) ∩ S 0 = ∅ since dom J D * x,• 0,f D * x = S 0 and Im(D + S 0 ) = Im(D * S 0 ) ⊂ S 0 . Moreover, Ker(D S 0 ) ∩ S 0 = Ker(D) ∩ S 0 .
We get the following corollary for smooth perturbation G, see Section 3.1. We recall that in this case, the model space T J+G x = T J

x and e J+G x = e J

x + P T J x ∇G(x).

C orollary 3 .2 Let J ∈ Γ + c (R n ), x ∈ R n and G ∈ Γ + c (R n
) a function which is Gâteaux-differentiable at x.. Then, for the particular choice of

f J+G x = f J x + ∇G(x),
we have f J+G x ∈ ri(J + G)(x) and for a given η ∈ S J x , the subdifferential gauge of J + G reads

(J + G) x,• f J+G x ,x = J x,• f J x ,x .
p ro o f It is sufficient to remark that the smooth perturbation G translates the subdifferential ∂J(x) by ∇G(x). Hence, using our choice of f J+G x , we find the same subdifferential gauge.

Special Cases

Strong Gauge

In this section, we study a particular subclass of convex functions that we dub strong gauges. We start with some definitions. D efi ni ti on 3 . 3 A bounded regularizing gauge J is separable with respect to

T = S ⊥ if ∀ (x, x ′ ) ∈ T × S, J(x + x ′ ) = J(x) + J(x ′ ).
Separability of J is equivalent to the following property on the polar J • . L emma 3 .1 Let J be a bounded gauge. Then, J is separable w.r.t. to T = S ⊥ if, and only if its polar J • satisfies

J • (x + x ′ ) = max J • (x), J • (x ′ ) , ∀ (x, x ′ ) ∈ T × S . p ro o f Let J = γ C , x ∈ T and x ′ ∈ S.
⇒: By virtue of Lemma 2.5, we have

J • (x + x ′ ) = sup u∈C x + x ′ , u = sup J(u) 1 x + x ′ , u = sup J(u T +u S ) 1 x, u T + x ′ , u S = sup J(u T )+J(u S ) 1
x, u T + x ′ , u S using the separability.

Rewriting the condition J(u

T + u S ) 1 as J(u T ) ρ, J(u S ) 1 -ρ for ρ ∈ [0, 1],
we arrive to

J • (x + x ′ ) = sup ρ∈[0,1] sup J(u T ) ρ,J(u S ) 1-ρ x, u T + x ′ , u S = sup ρ∈[0,1] ρ sup J(u T ) 1 x, u T + (1 -ρ) sup J(u S ) 1 x ′ , u S = sup ρ∈[0,1] ρ sup v∈C∩T x, v + (1 -ρ) sup wC∩T x ′ , w = sup ρ∈[0,1] ρσ C∩T (x) + (1 -ρ)σ C∩S (x ′ ) = max(σ C∩T (x), σ C∩S (x ′ )) . Since σ C∩T (x) = co (inf(σ C (x), ι S (x))) = σ C (x) = J • (x) and σ C∩S (x ′ ) = co inf(σ C (x ′ ), ι T (x ′ )) = σ C (x ′ ) = J • (x ′ ) ,
the implication follows.

⇐: Using again Lemma 2.5, we get

J(x + x ′ ) = sup u∈C • x + x ′ , u = sup J • (u T +u S ) 1 x, u T + x ′ , u S .
Using the separability of the polar,

J(x + x ′ ) = sup max(J • (u T ),J • (u S )) 1 x, u T + x ′ , u S = sup J • (u T ) 1,J • (u S ) 1 x, u T + x ′ , u S = sup v∈C • ∩T x, v + sup w∈C • ∩S x ′ , w = σ C • ∩T (x) + σ C • ∩S (x ′ ) = co (inf(σ C • (x), ι S (x))) + co inf(σ C • (x ′ ), ι T (x ′ )) = σ C • (x) + σ C • (x ′ ) = J(x) + J(x ′ ) .
This concludes the proof.

The decomposability of ∂J(x) as described in Theorem 3.1 depends on the particular choice of the map x → f x ∈ ri ∂J(x). An interesting situation is encountered when e x ∈ ri ∂J(x), in which case, one can just choose f x = e x , hence implying that f S x = 0. Strong gauges are precisely a class of gauges for which this situation occurs.

In the sequel, for a given model subspace T , we denote T the set of vectors sharing the same T ,

T = x ∈ R N | T x = T .
Using positive homogeneity, it is easy to show that T ρx = T x and e ρx = e x ∀ρ > 0. Thus T is a non-empty cone which is contained in T by Proposition 3.1(iv).

D efi ni ti on 3 . 4 -S trong Gauge A strong gauge on T is a bounded gauge J such that (i) For every x ∈ T , e x ∈ ri ∂J(x).

(ii) J is separable with respect to T and S = T ⊥ .

The following result shows that the decomposability property of Theorem 3.1 has a simpler form when J is a strong gauge.

P roposi ti on 3 .12 Let J be a strong gauge on T x . Then, the subdifferential of J at x reads

∂J(x) = η ∈ R N | η T x = e x and J • (η S x ) 1 .
p ro o f Let J = γ C . We only need to show that J x,• e x (η S x ) = J • (η S x ). This follows from Proposition 3.7, Lemma 3.1 and Lemma 2.5(ii). Indeed,

J x,• e x (η S x ) = inf τ 0 max(J • (τe x + η S x ), τ) from Proposition 3.7, = inf τ 0 max(τJ • (e x ), J • (η S x ), τ) from Lemma 3.1, = inf τ 0 max(J • (η S x ), τ) from e x ∈ ∂J(x) ⊂ C • , = J • (η S x ) ,
which concludes the proof.

When J is in addition a norm, this coincides with the decomposability definition of [START_REF] Candès | Simple bounds for recovering low-complexity models[END_REF]. Note however that the last part of assertion (ii) in Proposition 3.8 is an intrinsic property of gauges, while it is stated as an assumption in their definition. A notion of decomposability closely related to that of [START_REF] Candès | Simple bounds for recovering low-complexity models[END_REF], but different, was proposed in [START_REF] Negahban | A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers[END_REF]. Typical examples of (strongly) decomposable norms are the ℓ 1 , ℓ 1 -ℓ 2 and nuclear norms. However, strong decomposability excludes many important cases. One can think of analysis-type semi-norms since strong decomposability is not preserved under pre-composition by a linear operator, or the ℓ ∞ norm among many others.

Examples

ℓ 1 Norm

The norm J(x) = ||x|| 1 is a symmetric (bounded) strong gauge. More precisely, we have the following result.

P roposi ti on 3 .13 J = || • || 1 is a symmetric strong gauge with

T x = η ∈ R N | ∀j ∈ I, η j = 0 , S x = η ∈ R N | ∀i ∈ I, η i = 0 , e x = sign(x), f x = e x , J x,• f x = || • || ∞ + ι S x ,
where

I = I(x) = {i | x i = 0}. p ro o f The subdifferential of || • || 1 reads ∂|| • || 1 (x) = η ∈ R N | η (I) = sign(x (I) ) and ||η (I c ) || ∞ 1 .
The expressions of S x , T x , e x and f x follow immediately. Since e x ∈ ri ∂|| • || 1 (x) and || • || 1 is separable, it follows from Definition 3.4 that the ℓ 1 -norm is a strong gauge. Therefore J x,• f x = || • || ∞ , and Proposition 3.12 specializes to the stated subdifferential.

Figure 3.2 shows the underlying geometry of the ℓ 1 regularization in two dimensions. Note that ∂J(x) is included in the dual closed ball.

S x Sx T x • 1 ≤ 1 • ∞ ≤ 1 ∂J(x)
x Figure 3.2: ℓ 1 geometry.

Analysis-ℓ 1 Seminorm

The semi-norm J(x) = ||D * x|| 1 is a symmetric gauge.

P roposi ti on 3 .14 J = ||D * • || 1 is a symmetric (bounded) gauge with

T x = Ker(D * (I c )) ) = η ∈ R N | ∀j ∈ I, d j , η j = 0 , S x = Im(D I c ), e x = P Ker(D * I c ) D sign(D * x), f x = D sign(D * x), J x,• f x (η) = inf z∈Ker(D (I c ) ) ||D + (I c ) η + z|| ∞ , for η ∈ S x ,
where

I = I(x) = {i | d i , x i = 0}.
p ro o f This is a direct consequence of Proposition 3.11 and Proposition 3.13.

3.3.2.3 ℓ ∞ Norm
The norm J(x) = ||x|| ∞ is a symmetric gauge, but unlike the ℓ 1 -norm, it is not strongly so (except for n = 2). In the following proposition, we rule out the trivial case x = 0.

P roposi ti on 3 .15 J = || • || ∞ is a symmetric (bounded) gauge for x = 0 with S x = η | η (I c ) = 0 and η (I) , s (I) = 0 , T x = α | α (I) = ρs (I) for ρ ∈ R , e x = s |I| , f x = e x , J x,• f x (η) = max i∈I (-|I|s i η i ) + for η ∈ S x ,
where s = sign(x) and

I = I(x) = {i | |x i | = ||x|| ∞ }. p ro o f Recall that for J = || • || ∞ , f x = e x = s/|I|, with s = sign(x). Let K x = ∂J(x) -e x .
It can be straightforwardly shown that in this case,

K x = v | ∀ (i, j) ∈ I × I c , v j = 0, v (I) , s (I) = 0, -|I|v i s i 1
This is rewritten as

K x = S x ∩ {v | ∀ i ∈ I, -|I|v i s i 1} =K ′ x .
Thus the subdifferential gauge reads

J x,• f x (η) = γ K x (η) = max(γ S x (η), γ K ′ x (η)).
We have

γ S x (η) = ι S x (η) and γ K ′ x (η) = max i∈I (-|I|s i η i ) +
, where (•) + is the positive part, hence we obtain

J x,• f x (η) =    max i∈I (-|I|s i η i ) + if η ∈ S x +∞ otherwise.
Therefore the subdifferential of || • || ∞ at x takes the form

∂J(x) = η ∈ R N | η T x = e x = s |I| and max i∈I (-|I|s i η i ) + 1 ,
which concludes the proof.

Figure 3.3 shows the underlying geometry of the ℓ ∞ regularization in three dimensions.

3.3.2.4 ℓ 1 -ℓ 2 Norm
The ℓ 1ℓ 2 norm is a symmetric strong gauge.

P roposi ti on 3 .16 The ℓ 1ℓ 2 norm associated to the partition B, as defined in (1.13), is a symmetric (bounded) strong gauge with

T x = η | ∀j / ∈ I, η j = 0 , S x = {η | ∀i ∈ I, η i = 0} , e x = (N(x b )) b∈B , f x = e x , J x,• f x = || • || ∞,2 + ι S x ,
where

I = I(x) = {b | x b = 0},

Nuclear Norm

We show that the nuclear norm is a symmetric strong gauge. Since || • || 1 is an absolutely symmetric gauge, it is immediate to see that || • || * is an unitarily invariant norm according to Proposition 2.10.

P roposi ti on 3 .17 The nuclear norm is a symmetric strong gauge with

S x = U * ⊥ CV ⊥ | C ∈ R (N 1 -r)×(N 2 -r) , T x = UA * + BV * | A ∈ R N 2 ×r , B ∈ R N 1 ×r = Z ∈ R (N 1 -r)×(N 2 -r) | U * ⊥ ZV ⊥ = 0 , e x = UV * , f x = e x , J x,• f x (x) = max i σ i + ι S x ,
where U ⊥ , V ⊥ span the orthogonal of the ranges of U, V.

It can be observed that dim(T x ) = r(N 1 + N 2 -r) and dim(S x ) = N 1 N 2 - dim(T x ) = N 1 N 2 -r(N 1 + N 2 ) + r 2 .
p ro o f The subdifferential of the nuclear norm is a classical result in convex analysis of spectral functions, see e.g. [START_REF] Watson | Characterization of the subdifferential of some matrix norms[END_REF][START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF]. More precisely, let x ∈ R N 1 ×N 2 be a matrix and x = UΣV * its singular value decomposition. Then, the subdifferential ∂J(x) reads

∂J(x) = {UV * + M | ||M|| 1, U * M = 0 and MV = 0} .
The expressions of the subspaces T x , S x and e x follow immediately. Since the nuclear norm is a strong gauge, we get from Proposition 3.12 that the subdifferential gauge is the spectral norm.

Polyhedral Gauges

The ℓ 1 and ℓ ∞ norms are special cases of polyhedral priors. There are two alternative ways to define a polyhedral gauge. The H-representation encodes the gauge through the hyperplanes that support the polygonal facets of its unit level set. The V-representation encodes the gauge through the vertices that are the extreme points of this unit level set. We focus here on the H- representation.

A polyhedral gauge in the H-representation is defined as

J(x) = max 1 i p ( x, d i ) + = J 0 (D * x) where J 0 (u) = max 1 i p (u i ) + ,
and we have defined D = (d i ) p i=1 ∈ R n×p . For instance, J = || • || 1 can be recovered using the matrix D ∈ R n×2 n enumerating all sign patterns and

J = || • || ∞ corresponds to taking D = [-Id n Id n ].
Observe that the polar of a polyhedral gauge is again a polyhedral gauge. Such a polyhedral gauge can also be thought as an analysis gauge. One can then characterize decomposability of J 0 and then invoke Proposition 3.11 to derive those of J. This is what we are about to do. In the following, we denote (a i ) 1 i p the standard basis of R p . Figure 3.5 shows the geometry of this regularization when u is on the positive ray R + (1, 1) in two dimensions. Note that the level-set {J 0 (•) 1} to 1 is unbounded. P roposi ti on 3 .18 J 0 (u) = max 1 i p (u i ) + is a (bounded) gauge and,

J 0 (•) ≤ 1 J 0 , S • (•) ≤ 1 0 u 2 u 1 T u S u
• If u i 0, ∀i ∈ {1, • • • , p}, then S u = span a i i∈I 0 , T u = span a i i/ ∈I 0 , e u = 0, f u = µ i∈I 0 a i , for any 0 < µ < 1, J u,• f u (η) = inf τ max i∈I 0 (-η i ) + /µ max τµ|I 0 | + i∈I 0 η i , τ for η ∈ S u ,
where

I 0 = {i ∈ {1, • • • , p} | u i = J 0 (u) = 0} . • If ∃i ∈ {1, • • • , p} such that u i > 0, then S u = η | η (I c + ) = 0 and η (I + ) , s (I + ) = 0 , T u = α | α (I + ) = µs (I + ) for µ ∈ R , e u = s |I + | , f u = e u , J u,• f u (η) = max i∈I + (-|I + |η i ) + for η ∈ S u ,
where

s = i∈I + a i and I + = {i ∈ {1, • • • , p} | u i = J 0 (u) and u i > 0} .
p ro o f In general, the subdifferential of J 0 reads

∂J 0 (u) =        i∈I ρ i s i a i | ρ ∈ Σ I , s i ∈        {1} if u i > 0 [0, 1] if u i = 0 {0} if u i < 0       
, where Σ I is the canonical simplex in R |I| , and

I = {i ∈ {1, • • • , p} | (x i ) + = J 0 (x)}. • If u i 0, ∀i ∈ {1, • • • , p}, the above expression becomes ∂J 0 (u) =    i∈I 0 ρ i s i a i | ρ ∈ Σ I 0 , s i ∈ [0, 1]    ,
where

I 0 = {i ∈ {1, • • • , p} | u i = J 0 (u) = 0}.
Equivalently, ∂J 0 (u) is the intersection of the unit ℓ 1 ball and the positive orthant on R |I 0 | . The expressions of S u , T u and e u then follow immediately. ∂J 0 (u) then contains e u = 0, but not in its relative interior. Choosing any f u as advocated, we have f u ∈ ri ∂J 0 (u).

To get the subdifferential gauge, we some calculus rules on gauges and apply Lemma 3.7 to get

J u,• f u (η (I 0 ) ) = inf τ 0, τ(f u ) i -η i ∀i∈I 0 max(||τf u + η|| 1 , τ) ,
where the extra-constraints on τ come from the fact that ∂J 0 (u) is in the positive orthant, and the ℓ 1 norm is the gauge of the unit ℓ 1 -ball. We then have

J u,• f u (η (I 0 ) ) = inf τ 0, µτ max i∈I 0 -η i max(τ i∈I 0 µa i + η i , τ) = inf τ max i∈I 0 (-η i ) + /µ max(τµ|I 0 | + i∈I 0 η i , τ) .
• Assume now that u i > 0 for at least one i ∈ {1, • • • , p}. In such a case, J 0 (u) = ||u|| ∞ , and the subdifferential becomes

∂J 0 (u) = Σ I + ,
where

I + {i ∈ {1, • • • , p} | u i = J 0 (u) and u i > 0}.
The forms of S u , T u , e u , f u and the subdifferential gauge can then be retrieved from those of the ℓ ∞ -norm with s (I + ) = 1 and s (I c + ) = 0.

4

Partial Smoothness

Main contributions of this chapter

• Specialization and application of the theory of partial smoothness (Definition 4.1) to popular gauges in imaging and statistics.

• Derivation of explicit partial smoothness Lipschitz-constants for a particular sub-class of partly smooth functions (Definition 4.2). T heorem 3.1 provides a pointwise decomposition of the subdifferential of a convex function. It actually says nothing about the stability of such a formula at points x close enough from x. In order to obtain this stability, one needs to restrict the set of finite-valued convex functions used as regularizers. We propose two different classes of such regularizers, coined partly smooth functions and partly smooth functions relative to a linear manifold . The first one comes directly from the optimization litterature [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF], whereas the second is introduced in order to be able to provide explicit constants in our robustness results.

Contents

Partly Smooth Functions

The notion of "partly smooth" functions [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF] unifies many nonsmooth functions known in the literature. The notion of partial smoothness (as well as identifiable surfaces [START_REF] Wright | Identifiable Surfaces in Constrained Optimization[END_REF])) captures essential features of the geometry of non-smoothness which are along the so-called "active/identifiable manifold". Loosely speaking, a partly smooth function behaves smoothly as we move on the partial smoothness manifold, and sharply if we move normal to the manifold. In fact, the behaviour of the function and of its minimizers (or critical points) depend essentially on its restriction to this manifold, hence offering a powerful framework for sensitivity analysis theory. In particular, critical points of partly smooth functions move stably on the manifold as the function undergoes small perturbations [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF][START_REF] Lewis | Partial Smoothness, Tilt Stability, and Generalized Hessians[END_REF]).

Definition

Specialized to convex functions, the definition of partly smooth functions reads as follows1 .

D efi ni ti on 4 . 1 A function J ∈ Γ + c (R n ) is said to be partly smooth (PSF) at x relative to a set M ⊆ R n if there exists a neighborhood U of x such that (i) Smoothness. M ∩ U is a C 2 -manifold and J restricted to M ∩ U is C 2 , J U∩M ∈ C 2 (M ∩ U).
(ii) Sharpness. The tangent space of M at x is the model tangent space T x , T M (x) = T x .

(iii) Continuity. The set-valued mapping ∂J is continuous at x relative to

M.

The manifold M is coined the model manifold of x ∈ R n . J is said to be partly smooth relative to a set M if M is a manifold and J is partly smooth at each point x ∈ M relative to M. J is said to be locally partly smooth at x relative to a set M if M is a manifold and there exists a neighbourhood U of x such that J is partly smooth at each point x ′ ∈ M ∩ U relative to M.

We denote the set of all partly smooth functions at x relative to a manifold M as S x (M) and the set of all partly smooth functions relative to a manifold M as S(M). The definition of continuity of ∂J is to be understood according to Definition 2.17. Since J is proper convex continuous, the subdifferential of ∂J(x) is everywhere non-empty and compact and every subgradient is regular. Therefore, the Clarke regularity property (Lewis 2002, Definition 2.7(ii)) is automatically verified. In view of (Lewis 2002, Proposition 2.4(i)-(iii)), our sharpness property is equivalent to that of (Lewis 2002, Definition 2.7(iii)).

Obviously, any smooth function J : R n → R is partly smooth relative to the manifold R n . Moreover, any indicator function ι M of a manifold M is partly smooth relative to M.

Remark that in the previous definition, M needs only to be defined locally around x, and it can be shown to be locally unique. Hence the notation M is unambiguous.

Lemma 4 . 1 Let J ∈ S x (M) be a partly smooth function at x ∈ R n relative to both M and M. Then, there exists a neighborhood U of x such that

U ∩ M = U ∩ M.
p ro o f This is proved in Corollary 4.2 of [START_REF] Hare | Identifying Active Manifolds[END_REF]).

Partial Smoothness Calculus

Partial smoothness is preserved under addition, pre-composition by a linear operator and matrix lift. These results are proved in [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF][START_REF] Daniilidis | Orthogonal Invariance and Identifiability[END_REF].

P roposi ti on 4 .1 Let J 0 ∈ S z (M 0 z ) be a partly smooth function at z = D * x relative to the manifold M 0 z . Then J = J 0 • D * ∈ S x (M) for the manifold M = u ∈ R N | D * u ∈ M 0 z .
p ro o f This is proved in (Lewis 2002, Theorem 4.2). Contrary to the general case, the transversality hypothesis is automatically satisfied since J 0 is convex and continuous.

P roposi ti on 4 .2 Let J and G two partly smooth functions at x ∈ R n relative to the manifolds M J and M G . Then J + G ∈ S x (M) for the manifold

M = M J ∩ M G .
p ro o f This is proved in (Lewis 2002, Corollary 4.6). Contrary to the general case, the transversality hypothesis is automatically satisfied since J and G are convex and continuous.

P roposi ti on 4 .3 Let j be an absolutely symmetric gauge and x ∈ R n 1 ×n 2 a symmetric matrix. Then the two following statements are equivalent:

(i) j is a partly smooth function at Λ x relative to the manifold m Λ x .

(ii

) J = j • σ ∈ S x (M) for the manifold M = σ -1 (m Λ x ).
p ro o f See (Daniilidis et al. 2013, Theorem 5.3).

Partly Smooth Functions With Linear Manifolds

In practice, many of the partly smooth functions we consider are associated to linear manifolds (i.e. the tangent model subspace is the model manifold M = T x ). These functions, coined partly smooth functions with linear manifolds, encompass most of the knowns regularizations in the image processing and statistics, such as the ℓ 1 , ℓ 1ℓ 2 , ℓ ∞ norm and their sums and compositions by a linear operator, with the noticeable exception of the nuclear norm.

Definition

We restrict our interest in this section to convex funtions J which are partly smooth at x ∈ R n with respect to a linear subspace. In this case, this subspace is T x . The following theorem proves that such functions, which enjoy the fact that ∂J is Lipschitz on T x are characterized by a set of parameters.

Theorem 4 . 1 Let Γ be any coercive gauge bounded on T x for x ∈ R n . Let J ∈ S x (M x ) for the manifold M x = T x and we assume that ∂J : T x ⇒ R n is Lipschitz around x. Then for any Lipschitz-mapping.

f :

   T x → R n x → f x ∈ ri ∂J(x),
there exist four non-negative reals ν x , µ x , τ x , ξ x such that

∀x ′ ∈ T , Γ (x -x ′ ) ν x ⇒ T x = T x ′ (4.1)
and for every x ′ ∈ T with Γ (xx ′ ) < ν x , one has

Γ (e x -e x ′ ) µ x Γ (x -x ′ ), (4.2)
J x,• f x (P S (f x -f x ′ )) τ x Γ (x -x ′ ), (4.3)
sup u∈S u =0 J x ′ ,• f x ′ (u) -J x,• f x (u) J x,• f x (u) ξ x Γ (x -x ′ ).
(4.4)

Moreover, there exists such a mapping f. p ro o f We prove this result for Γ = || • ||. This is not restrictive, since for every

x ∈ R n , Γ (x) | | |Id| | | Γ →ℓ 2 ||x||.
We start from the hypotheses J ∈ S x (M x ) for the manifold M x = T x and ∂J : T x ⇒ R n is Lipschitz around x.

• Existence of f x . Such a mapping exists according to (Aubin et al. 2009, Theorem 9.4.3).

• ν-stability. Using (Lewis 2002, Proposition 2.10) the sharpness property (ii) is locally stable. Hence, for x ′ ∈ T x in a neighborhood of x, T x ′ = T x . The radius of this neighborhood can be taken as ν x .

• µ-stability. Using (Hiriart-Urruty et al. 2001, Corollary VI.2.1.3), we write for any h ∈ T

x J(x + th) = J(x) + t s, h + o(t),
where s ∈ F ∂J(x) (h). Since J restricted to T x ∩ U is C 2 according to the smoothness property, repeating this argument at order 2 let us concludes that the mapping z ∈ T x ∩ U → e z is C 1 . Hence, this map is Lipschitz.

• τ-stability. One has

J x,• f x (P S (f x -f x ′ )) | | |P S x | | | J x,• fx →ℓ 2 ||f x -f x ′ || τ x ||x -x ′ ||,
where

τ x = | | |P S x | | | J x,
• fx →ℓ 2 β and β is the Lipschitz constant associated to f x , proving (4.3).

• ξ-stability. ∂J is Lipschitz around x and x → f x is Lipschitz. Hence, the application x → (∂J(x) -f x ) is Lipschitz on T x . Using Lemma 2.6, we get that

J x ′ ,• f x ′ (u) -J x,• f x (u) β||x ′ -x||||u||. Since ||u|| | | |Id| | | ℓ 2 →J x,• fx J x,•
f x (u), we get bound (4.4) where

ξ x = β| | |Id| | | ℓ 2 →J x,• fx .
This result motivates the following definition.

D efi ni ti on 4 . 2 -P S F R elati ve to a L i near M ani fold. A finite-valued convex function J is said to be partly smooth relative to a linear manifold at x ∈ R n , if J is partly smooth at x for the manifold M = T x . The set of all partly smooth functions with linear manifolds at x, such that ∂J is Lipschitz around x relative to T x , with parameters (Γ ,

f x , ν x , µ x , τ x , ξ x ) is denoted SL x (Γ , f x , ν x , µ x , τ x , ξ x ).

Stability under the Sum and Precomposition by a Linear Operator

Partial smoothness with linear manifold property is preserved under addition and pre-composition by a linear operator, and one can give explicit bound on the corresponding Lipschitz constants.

Addition

P roposi ti on 4 .4 Let x ∈ R n , J and G two partly smooth functions with linear manifolds such that

J ∈ SL x (Γ J , f J x , ν J x , µ J x , τ J x , ξ J x ) G ∈ SL x (Γ G , f G x , ν G x , µ G x , τ G x , ξ G x ).
Then, H = J + G is also partly smooth with linear manifold at x, for the choice

f H x = f J x + f G
x and Γ H = max(Γ J , Γ G ), with ∂H Lipschitz and the parameters

ν H x = min(ν J x , ν G x ) µ H x = µ J x | | |P T H | | | Γ J →Γ H + µ G x | | |P T H | | | Γ G →Γ H τ H x = τ J x + τ G x + µ J x | | |P S H ∩T J | | | Γ J →H x,• f H x + µ G x | | |P S H ∩T G | | | Γ G →H x,• f H x ξ H x = max(ξ J x , ξ G x ).
p ro o f In the following, all operator bounds that appear are finite owing to the coercivity assumption on the involved gauges in Definition 4.2 of a PSFL.

It is straightforward to see that the function Γ H = max(Γ J , Γ G ) is indeed a gauge, which is bounded and coercive on T H = T J ∩ T G . Moreover, given that both J and G are PSFL at x with corresponding parameters ν J x and ν G x , we have with the advocated choice of Γ H and ν H

x ,

Γ J (x -x ′ ) ν J x and Γ G (x -x ′ ) ν G x ,
for every ∀x ′ ∈ T H x such that Γ H (xx ′ ) ν H x . It follows that:

• Since J and G are both PSFL, then we have

T J x = T J x ′ and T G x = T G
x ′ , and thus by Proposition 3.10(i)

T H x = T J x ∩ T G x = T J x ′ ∩ T G x ′ = T H x ′ = T H .
• µ H x -stability: we have from Proposition 3.10(ii)

Γ H (e H x -e H x ′ ) = Γ H P T H (e J x + e G x -e J x ′ -e G x ′ ) Γ H P T H (e J x -e J x ′ ) + Γ H P T H (e G x -e G x ′ ) | | |P T H | | | Γ J →Γ H Γ J e J x -e J x ′ + | | |P T H | | | Γ G →Γ H Γ G e G x -e G x ′ µ J x | | |P T H | | | Γ J →Γ H + µ G x | | |P T H | | | Γ G →Γ H Γ H (x -x ′ ) ,
where we used µ J x -and µ G x -stability of J and G in the last inequality.

• τ H x -stability: the fact that S J ⊆ S H and S G ⊆ S H and subadditivity of gauges lead to

H x,• f H x P S H (f H x -f H x ′ ) = H x,• f H x P S J (f J x -f J x ′ ) + P S G (f G x -f G x ′ ) + P S H (e J
xe J

x ′ ) + P S H (e G

xe G x ′ )

H x,• f H x P S J (f J x -f J x ′ ) + H x,• f H x P S G (f G x -f G x ′ ) + H x,• f H x P S H (e J x -e J x ′ ) + H x,• f H x P S H (e G
xe G x ′ ) . (4.5)

According to Proposition 3.10(iii), we have

H x,• f H x P S J (f J x -f J x ′ ) = inf η 1 +η 2 =P S J (f J x -f J x ′ ) max(J x,• f J x (η 1 ), G x,• f G x (η 2 )) . Since dom J x,• f J x = S J , (η 1 , η 2 ) = (P S J (f J x -f J x ′ ), 0
) is a feasible point of the last problem, and we get

H x,• f H x P S J (f J x -f J x ′ ) J x,• f J x P S J (f J x -f J x ′ ) .
Moreover, as e J x , e J x ′ ∈ T J (see Proposition 3.1(ii)) and S J ⊆ S H , we have min

η 1 ∈T J ,η 2 S J ,η 1 +η 2 ∈S H ||η 1 + η 2 -(e J x -e J x ′ )|| 2 = min η 1 ∈T J ,η 2 S J ,η 1 +η 2 ∈S H ||η 1 -(e J x -e J x ′ )|| 2 + ||η 2 || 2 = min η 1 ∈T J ,η 2 S J ,η 1 ∈S H ||η 1 -(e J x -e J x ′ )|| 2 + ||η 2 || 2 = min η 1 ∈S H ∩T J ||η 1 -(e J x -e J x ′ )|| 2 .
That is

P S H (e J
xe J

x ′ ) = P S H ∩T J (e J

xe J x ′ ) . Thus

H x,• f H x P S H (e J x -e J x ′ ) | | |P S H ∩T J | | | Γ J →H x,• f H x Γ J e J x -e J x ′
.

Similar reasoning leads to the following bounds

H x,• f H x P S G (f G x -f G x ′ ) G x,• f G x P S G (f G x -f G x ′ ) , H x,• f H x P S H (e G x -e G x ′ ) | | |P S H ∩T G | | | Γ J →H x,• f H x Γ G e G x -e G x ′ .
Having this, we can continue to bound (4.5) as

H x,• f H x P S H (f H x -f H x ′ ) J x,• f J x P S J (f J x -f J x ′ ) + G x,• f G x P S G (f G x -f G x ′ ) + | | |P S H ∩T J | | | Γ J →H x,• f H x Γ J e J x -e J x ′ + | | |P S H ∩T G | | | Γ J →H x,• f H x Γ G e G x -e G x ′ τ J x Γ J (x -x ′ ) + τ G x Γ G (x -x ′ ) + µ J x | | |P S H ∩T J | | | Γ J →H x,• f H x Γ J x -x ′ + µ G x | | |P S H ∩T G | | | Γ G →H x,• f H x Γ G x -x ′ τ J x + τ G x + µ J x | | |P S H ∩T J | | | Γ J →H x,• f H x + µ G x | | |P S H ∩T G | | | Γ G →H x,• f H x Γ H (x -x ′ ) ,
where the last two inequalities J and G follow from µ J x -, τ J x -, µ G x -and τ G xstability of J and G.

• ξ H

x -stability: Proposition 3.10(iii) again yields that for any η ∈ S H

H x ′ ,• f H x ′ (η) = inf η 1 +η 2 =η max(J x ′ ,• f J x ′ (η 1 ), G x ′ ,• f G x ′ (η 2 )) max(J x ′ ,• f J x ′ ( η1 ), G x ′ ,• f G x ′ ( η2 ))
for any feasible ( η1 , η2 ) ∈ S J × S G ∩ {(η 1 , η 2 | η 1 + η 2 = η}. Now both J and G are PRF, hence respectively ξ J x -and ξ G x -stable. Therefore, with the form of Γ H we have

J x ′ ,• f J x ′ ( η1 ) (1 + ξ J x Γ J (x -x ′ ))J x,• f J x ( η1 ) βJ x,• f J x ( η1 ) G x ′ ,• f G x ′ ( η2 ) (1 + ξ G x Γ G (x -x ′ ))G x,• f G x ( η2 ) βG x,• f G x ( η2 ) , where β = 1 + max ξ J x , ξ G x Γ H (x -x ′ ). Whence we get max(J x ′ ,• f J x ′ (η 1 ), G x ′ ,• f G x ′ (η 2 )) β max(J x,• f J x ( η1 ), G x,• f G x ( η2 )) .
Taking in particular

( η1 , η2 ) ∈ Argmin η 1 +η 2 =η max(J x,• f J x (η 1 ), G x,• f G x (η 2 ))
we arrive at

H x ′ ,• f H x ′ (η) β inf η 1 +η 2 =η max(J x,• f J x (η 1 ), G x,• f G x (η 2 )) = βH x,• f H x (η) .
This completes the proof.

Precomposition by a Linear Operator

P roposi ti on 4 .5 Let J 0 be a partly smooth function with linear manifold at u = D * x with parameter

J 0 ∈ SL u (Γ 0 , f 0,u , ν 0,u , µ 0,u , τ 0,u , ξ 0,u ).
Then J = J 0 • D * is partly smooth with linear manifold at x, with the choice f x = Df 0,u and Γ any bounded coercive gauge on T , with ∂J Lipschitz and the parameters

ν x = 1 | | |D * | | | Γ →Γ 0 ν 0,u µ x = µ 0,u | | |P T D| | | Γ →Γ 0 | | |D * | | | Γ →Γ 0 τ x = τ 0,u D + S 0 P S D J u,• 0,f 0,u →J u,• 0,f 0,u + µ 0,u D + S 0 P S D Γ 0 →J u,• 0,f 0,u | | |D * | | | Γ →Γ 0 ξ x = ξ 0,u | | |D * | | | Γ →Γ 0 .
p ro o f In the following, all operator bounds that appear are finite owing to the coercivity assumption on the involved gauges in Definition 4.2 of a PSFL.

• Let x ′ such that Γ (x -x ′ ) 1 | | |D * | | | Γ →Γ 0 ν 0,D * x . Hence, Γ 0 (D * x -D * x ′ ) | | |D * | | | Γ →Γ 0 Γ (x -x ′ ) ν 0,D * x
As J 0 is a PSFL at D * x, we have T 0,D * x = T 0,D * x ′ = T 0 and consequently, using Proposition 3.11(i),

T x = Ker(D * S 0,D * x ) = Ker(D * S 0,D * x ′ ) = T x ′ = T = S ⊥ .
• µ x -stability: we now have

Γ (e x -e ′ x ) = Γ (P T D(e 0,D * x -e 0,D * x ′ )) Proposition 3.11(ii) | | |P T D| | | Γ 0 →Γ Γ 0 (e 0,D * x -e 0,D * x ′ ) µ 0,D * x | | |P T D| | | Γ 0 →Γ Γ 0 (D * x -D * x ′ ) using µ 0,D * x -stability of J 0 µ 0,D * x | | |P T D| | | Γ 0 →Γ | | |D * | | | Γ →Γ 0 Γ (x -x ′ ). • τ x -stability: since f 0,D * x ∈ ∂J 0 (D * x) and f 0,D * x ′ ∈ ∂J 0 (D * x ′ ), one has f 0,D * x -f 0,D * x ′ = P S 0 (f 0,D * x -f 0,D * x ′ ) + e 0,D * x -e 0,D * x ′ .
Thus, subadditivity yields

J x,• f x (P S (f x -f x ′ )) = J x,• f x (P S D(f 0,D * x -f 0,D * x ′ )) J x,• f x (P S D P S 0 (f 0,D * x -f 0,D * x ′ )) + J x,• f x (P S D(e 0,D * x -e 0,D * x ′ )).
Using Proposition 3.11(iii), we get the following bound on the first term

J x,• f x (P S D P S 0 (f 0,D * x -f 0,D * x ′ )) = inf z∈Ker(D)∩S 0 J D * x,• 0,f D * x (D + S 0 P S D P S 0 (f 0,D * x -f 0,D * x ′ ) + z) J D * x,• 0,f D * x (D + S 0 P S D P S 0 (f 0,D * x -f 0,D * x ′ )) D + S 0 P S D J D * x,• 0,f D * x →J D * x,• 0,f D * x J D * x,• 0,f D * x (P S 0 (f 0,D * x -f 0,D * x ′ ))
Using τ 0,D * x -stability of J 0 , we get

J x,• f x (P S D P S 0 (f 0,D * x -f 0,D * x ′ )) τ 0,D * x D + S 0 P S D J D * x,• 0,f D * x →J D * x,• 0,f D * x Γ 0 (D * x -D * x ′ ) τ 0,D * x D + S 0 P S D J D * x,• 0,f D * x →J D * x,• 0,f D * x | | |D * | | | Γ →Γ 0 Γ (x -x ′ ).
Now, combining Proposition 3.11(iii) and µ 0,D * x -stability of J 0 , we obtain the following bound on the second term J x,• f x (P S D(e 0,D * xe 0,D * x ′ )) J D * x,• 0,f D * x (D + S 0 P S D(e 0,D * xe 0,D * x ′ ))

D + S 0 P S D Γ 0 →J D * x,• 0,f D * x Γ 0 (e 0,D * x -e 0,D * x ′ ) µ 0,D * x D + S 0 P S D Γ 0 →J D * x,• 0,f D * x | | |D * | | | Γ →Γ 0 Γ (x -x ′ ).
Combining these inequalities, we arrive at

J x,• f x (P S (f x -f x ′ )) τ 0,D * x D + S 0 P S D J D * x,• 0,f D * x →J D * x,• 0,f D * x + µ 0,D * x D + S 0 P S D Γ 0 →J D * x,• 0,f D * x | | |D * | | | Γ →Γ 0 Γ (x -x ′ ),
whence we get τ x -stability.

123

• ξ x -stability: from Proposition 3.11(iii), we can write for any η ∈ S

J x ′ ,• f x ′ (η) = inf z∈Ker(D)∩S 0 J • f 0,D * x ′ (D + S 0 η + z) J D * x ′ ,• 0,f D * x ′ (D + S 0 η + z)
for any z ∈ Ker(D) ∩ S 0 .

Owing to ξ 0,D * x -stability of J 0 , and since D + S 0 η ∈ S 0 , we have for any feasible z ∈ Ker(D) ∩ S 0

J D * x ′ ,• 0,f D * x ′ (D + S 0 η + z) 1 + ξ 0,D * x Γ 0 (D * x -D * x ′ ) J D * x,• 0,f D * x (D + S 0 η + z) .
Taking in particular z ∈ Argmin z∈Ker(D)∩S 0

J D * x,• 0,f D * x (D + S 0 η + z)
we get the bound

J x ′ ,• f x ′ (η) 1 + ξ 0,D * x Γ 0 (D * x -D * x ′ ) inf z∈Ker(D)∩S 0 J D * x,• 0,f D * x (D + S 0 η + z) = 1 + ξ 0,D * x Γ 0 (D * x -D * x ′ ) J x ′ ,• f x ′ (η) = 1 + ξ 0,D * x | | |D * | | | Γ →Γ 0 Γ (x -x ′ ) J x ′ ,• f x ′ (η) ,
where we used again Proposition 3.11(iii) in the first equality.

Examples

Synthesis Sparsity

The norm J(x) = ||x|| 1 is a strong partly smooth function.

P roposi ti on 4 .6 J = || • || 1 is a strong partly smooth function with

Γ = || • || ∞ , ν x = min i∈I |x i | and µ x = τ x = ξ x = 0, where I = I(x) = {i | x i = 0}.
p ro o f Let x ′ ∈ T , i.e. I(x ′ ) ⊆ I(x), and assume that

||x -x ′ || ∞ < ν x = min i∈I |x i | . This implies that ∀i ∈ I(x), |x ′ i | > ν x -||x -x ′ || ∞ 0
, which in turn yields I(x ′ ) = I(x), and thus T x ′ = T x . Since the sign is also locally constant on the restriction to T of the ℓ ∞ -ball centered at x of radius ν x , one can choose µ x = 0. Finally

τ x = ξ x = 0 because f x = e x .

Analysis Sparsity

Antisparsity

P roposi ti on 4 .8 J = || • || ∞ is a partly smooth function with linear manifold with

Γ = || • || 1 , ν x = ||x|| ∞ -max j/ ∈I |x j | and µ x = τ x = ξ x = 0.
p ro o f Let x ′ ∈ T , and assume that

||x -x ′ || 1 < ν x = ||x|| ∞ -max j/ ∈I |x j | .
This means that x ′ lies in the relative interior of the ℓ 1 -ball (relatively to T ) centered at x of radius ||x|| ∞ -max j/ ∈I |x j |. Within this ball, the support and the sign pattern restricted to the support are locally constant, i.e. I(x) = I(x ′ ) and sign(x (I(x)) ) = sign(x ′ (I(x ′ )) ). Thus T x ′ = T x = T and e x ′ = e x , and from the latter we deduce that µ x = 0. As f x = e x we also conclude that τ x = ξ x = 0, which completes the proof.

Group Sparsity

The ℓ 1ℓ 2 norm is a strong partly smooth function. We start by the following lemma Lemma 4 . 2 Given any pair of non-zero vectors u and v where, ||u -v|| ρ||u||, for 0 < ρ < 1, we have

u ||u|| - v ||v|| C ρ ||u -v|| ||u|| , where C ρ = √ 2 ρ 1 -1 -ρ 2 ∈]1, √ 2[. p ro o f Let d = v -u and β = u, d ||u||||d|| ∈ [-1, 1].
We then have the following identities

u ||u|| - v ||v|| 2 = 2 -2 u, v ||u||||v|| = 2 -2 ||u|| 2 + ||u||||d||β ||u|| ||u|| 2 + ||d|| 2 + 2||u||||d||β , ( 4.6)
for non-zero vectors u and d, the unique maximizer of (4.6) is β ⋆ = -||d||/||u||. Note that the assumption ||d||/||u|| ρ < 1 assures β ⋆ to comply with the admissible range of β and further, the argument of the square root will be always positive. Now, inserting β ⋆ in (4.6), using concavity of √ • on R + , and that ||d||/||u|| ρ, we can deduce the following bound

u ||u|| - v ||v|| 2 2 -2 1 - ||d|| 2 ||u|| 2 = 2 -2 1 - ||d|| 2 ρ 2 ||u|| 2 + ||d|| 2 ρ 2 ||u|| 2 (1 -ρ 2 ) 2 -2 1 - ||d|| 2 ρ 2 ||u|| 2 + ||d|| 2 ρ 2 ||u|| 2 1 -ρ 2 = 2 -2 1 - 1 -1 -ρ 2 ρ 2 ||d|| 2 ||u|| 2 = 2 1 -1 -ρ 2 ρ 2 ||d|| 2 ||u|| 2 .
P roposi ti on 4 .9 The ℓ 1ℓ 2 norm associated to the partition B is a strong partly smooth function with

Γ = || • || ∞,2 , ν x = min b∈I ||x b ||, µ x = √ 2 ν x and τ x = ξ x = 0.
∈ I ||x ′ b || ||x b || -||x b -x ′ b || > ν x -||x -x ′ || ∞,2 0,
and thus I(x ′ ) = I(x), i.e. T x ′ = T x . Moreover, since the gauge is strong, one has τ x = ξ x = 0. To establish the µ x -stability we use Lemma 4.2.

By definition of ν x , we have ||x b || > ν x , ∀b ∈ I, and thus

||x b -x ′ b || < ν x < ||x b ||.
Lemma 4.2 then applies, and it follows that, ∀b ∈ I

||N(x b ) -N(x ′ b )|| C ρ ||x ′ b -x b || ||x b || C ρ ||x ′ b -x b || ν x
, and therefore we get

||N(x) -N(x ′ )|| ∞,2 C ρ ν x ||x ′ -x|| ∞,2 ,
which implies µ x -stability for µ x = C ρ /ν x .

Polyhedral Regularizations

P roposi ti on 4 .10 J 0 (u) = max 1 i N H (u i ) + is a partly smooth function with linear manifold with parameters (assuming I + = ∅)

ν u = max i∈I + u i -max j/ ∈I + ,u j >0 u j , δ ∈]0, 1] and µ u = τ u = ξ u = 0.
p ro o f The parameters are derived following the same lines as for the ℓ ∞ -norm. Let u ′ ∈ T , and assume that

||u -u ′ || 1 < ν u = max i∈I + u i -max j ∈I + ,u j >0 u j .
This means that x ′ lies in the relative interior of the ℓ 1 -ball (relatively to T ) centered at x of radius

max i∈I + u i -max j ∈I + ,u j >0 u j = ||u|| ∞ -max j ∈I + ,u j >0 |u j |
Within this set, one can observe that the set I + associated to u is constant. Moreover, the sign pattern is also constant leading to the fact that T u ′ = T u = T . Hence, we deduce as in the ℓ ∞ -case that µ u = τ u = ξ u = 0.

Nuclear Norm

P roposi ti on 4 .11 Let x ∈ R n×n The nuclear norm is partly smooth at x for the manifold

M = u ∈ R n×n | rank(u) = rank(x) . p ro o f This is a direct consequence of Proposition 4.3 using j(Λ x ) = ||Λ x || 1 .
However, one should note that the nuclear norm is not a partly smooth function with linear manifold. Indeed, according to Proposition 4.11, the model subspace reads

T x = UA * + BV * | A ∈ R n×r , B ∈ R n×r ,
which in particular contains matrices of ranks larger than r. T hi s chapter introduces in particular the notion of dual certificates which is connected to the solution of the dual problem of (P y,0 ). Moreover, we provide a sufficient condition for uniqueness of problem (P y,λ ) and (P y,0 ).

J M T x e x J x,• fx on S x ν x µ x Comment || • || 1 η | ∀j ∈ I, η j = 0 M x sign(x) || • || ∞ min i∈I |x i | 0 I = supp(x) ||D * • || 1 Ker(D * I c ) M x P Ker(D * I c ) sign(D * x) inf z∈Ker(D I c ) ||D + I c η + z|| ∞ min i∈I | d i , x i | 0 I = supp(D * x) || • || 1,2 η | ∀j / ∈ I, η j = 0 M x (N(x b )) b∈B || • || ∞,2 min b∈I ||x b || √ 2 νx I = {g ∈ B | x g = 0} || • || ∞ {α | α I = ρs I for ρ ∈ R} M x sign(x)/|I| max i∈I (-|I|s i η i ) + ||x|| ∞ -max j/ ∈I |x j | 0 I = {i | |x i | = ||x|| ∞ } || • || * {Z | rank(Z) = rank(x)} Z | U * ⊥ ZV ⊥ = 0 UV * || • || sp x = UΛV *

Primal Problem

We consider J ∈ Γ + c (R n ). Let us split y = y 0 + w where y 0 = Φx 0 . We supose that Ker(Φ) ∩ Ker(J ∞ ) = {0}, (5.1)

We rewrite problems (P y,λ ) and (P y,0 ) as a common regularization problem

min x∈R n f(x, θ) where f(x, θ) =    J(x) + ι H y (x) if θ = (y, 0) J(x) + 1 2λ ||Φx -y|| 2 otherwise, (P θ )
where

H y = {x ∈ R n | Φx = y} and θ = (y, λ) ∈ R q × R + . 132
5.1 Primal Problem

Existence of Solutions

We recall that in general (P θ ) might have multiple solutions. Here, based on classical compactness arguments, we show that the set of minimizers of (P θ ) is non-empty, compact and convex.

P roposi ti on 5 .1 Let f ∈ Γ 0 (R n ). Then, (i) If inf f > -∞, then f ∞ (d) 0, ∀d.
(ii) The set of minimizers of f is non-empty and compact ⇐⇒ f is coercive ⇐⇒ the sublevel sets of f are bounded ⇐⇒ f ∞ (d) > 0, ∀d = 0.

p ro o f (i)
The statement follows from the equivalent analytic representation of f ∞ in Proposition 2.4(i).

(ii) (Auslender et al. 2003, Proposition 3.1.2 and Proposition 3.1.3).

Let us now turn to the minimization problem min

x∈R n F(Φx) + J(x) (5.2)
where F ∈ Γ 0 (R p ) and strictly convex, J ∈ Γ 0 (R n ) and continuous on R n , inf J > -∞, and Φ : R n → R p . L emma 5 .1 The set of minimizers of (5.2) is non-empty and compact if and only if Ker(J ∞ ) ∩ Ker(Φ) = {0} , where J ∞ is given by either expressions of Proposition 2.4(i) or (ii).

p ro o f By strict convexity of F, all minimizers of (5.2) share the same image under Φ. Let x ⋆ any minimizer. Thus, (5.2) can be equivalently rewritten

min δ∈Ker(Φ) J(x ⋆ + δ) . Let f = J(x ⋆ + •) + ι Ker(Φ) . Thus, owing to Proposition 5.1(ii) the set of minimizers is compact ⇐⇒ f ∞ (d) > 0 ∀d = 0 Proposition 2.4(v) ⇐⇒ J ∞ (d) + (ι Ker(Φ) ) ∞ (d) > 0 ∀d = 0 Proposition 2.4(iii) ⇐⇒ J ∞ (d) + ι Ker(Φ) ∞ (d) > 0 ∀d = 0 Proposition 2.3(iv) ⇐⇒ J ∞ (d) + ι Ker(Φ) (d) > 0 ∀d = 0 ⇐⇒ J ∞ (d) > 0 ∀d ∈ Ker(Φ) \ {0} Proposition 5.1(i) ⇐⇒ Ker(J ∞ ) ∩ Ker(Φ) = {0} .
This condition (5.1) is from now on assumed in all propositions.

Convergence of the Primal Problem

We first show the convergence of the solutions of the primal problem toward x 0 when (P y 0 ,0 ) has a unique solution x 0 .

P roposi ti on 5 .2 Assume that x 0 is the unique solution of (P y 0 ,0 ). Let θ k = (λ k , y k ) a sequence such that 0 < λ k → 0 and ||y ky 0 || 2 /λ k → 0. Then, for any sequence (x θ k ) k of minimizers to (P θ k ),

x θ k -→ x 0 .
In order to ease the exposition, we will write in the following this convergence statement with the following slight abuse of notation.

x θ → x 0 when    λ → 0 ||w|| 2 λ → 0.
p ro o f This is a classical result, whose proof can be found for instance in (Hofmann et al. 2007, Theorem 3.5). We recall it by sake of clarity.

By optimality of x θ one has f(x θ , θ) f(x 0 , θ) and hence

||Φ(x θ -x 0 ) -w|| 2 ||w|| 2 + 2λJ(x 0 ), (5.3)
J(x θ ) ||w|| 2 2λ + J(x 0 ).

(5.4)

Thanks to (5.1), these bounds show that the sequence {x θ } θ is bounded if ||w|| 2 /λ and λ are bounded. We let x ⋆ be any accumulation point.

For the considered asymptotics, (5.3) implies that Φx ⋆ = Φx 0 , while (5.4) implies that J(x ⋆ ) J(x 0 ). This shows that x ⋆ is a solution of (P y 0 ,0 ) and hence x ⋆ = x 0 .

Certificates and Restricted Injectivity

This section introduces the two main objects of the noise stability study. The first one is the dual certificate, which characterizes the set of solutions of (P y,0 ). The second one is the restricted injectivity condition, to be able to ensure the uniqueness.

Fenchel-Rockafellar Duality

We characterize the dual problem in the following Lemma.

L emma 5 .2 Let θ = (λ, y) with λ 0. The dual problem of (P θ ) reads min p∈R q g(p, θ)

(D θ )
where

g(p, θ) =    J * (Φ * p) -y, p if λ = 0 J * (Φ * p) -y, p + λ
2 ||p|| 2 otherwise. Moreover, there is no duality gap, i.e. min x∈R n f(x, θ) = -min p∈R q g(p, θ).

Observe that domain qualification conditions (on their relative interiors) to ensure closedness of the dual objective (i.e. the min is attained) are verified for the penalized problem since 1 2 || • || 2 2 has full domain.

p ro o f The proof of this result is a simple application of the calculus rules on Fenchel-Rockafellar duality. The case λ = 0 is the Fenchel-Rockafellar duality for linear constraints, see for instance (Borwein et al. 2010, Corollary 3.3.11). The case λ > 0 is due to the fact that

(1/2|| • || 2 ) * = 1/2|| • || 2 .
We now relate the solutions of the primal problem (P θ ) to those of the dual (D θ ). P roposi ti on 5 .3 Let θ = (λ, y) with λ 0 and x θ any solution of (P θ ). Then, (i) if λ > 0, then (D θ ) has a unique solution S θ = {p θ } and

p θ = y -Φx θ λ and α θ = Φ * p θ ∈ ∂J(x θ ).
(ii) if λ = 0, then the set of solutions of (D θ ) is

S θ = {p ∈ R q | Φ * p ∈ ∂J(x θ )} .
p ro o f For the first statement, since J is finite-valued, strong duality holds, hence the result using Fenchel-Rockafellar duality. Similarly, strong duality holds between (P 0,y 0 ) and (D 0,y 0 ), and the primal-dual relationships states that (x 0,y 0 , p 0,y 0 ) form a solution to these problems if and only if Φ * p 0,y 0 ∈ ∂J(x 0,y 0 ).

Dual Certificates

These observations lead us to consider the notion of dual certificate, a terminology introduced in (Candès et al. 2006a) and revitalized in [START_REF] Candès | Simple bounds for recovering low-complexity models[END_REF], which corresponds to Lagrange multipliers, which are solution of the dual problem.

D efi ni ti on 5 . 1 A (dual) certificate for x ∈ R n is a vector p ∈ R q such that the source condition is verified:

Φ * p ∈ ∂J(x). (SC x )
If p is a certificate, and moreover

Φ * p ∈ ri ∂J(x), (SC x )
we say that p is a non-degenerate certificate.

Hence, according to Proposition 5.3, being a dual certificate is equivalent to be a solution of the dual problem (D 0 ) where x 0 = x, y = y 0 . One important certificate is the minimal norm certificate defined as follow D efi ni ti on 5 . 2 The minimal norm certificate for x 0 ∈ R n is defined by

p 0 = argmin p∈S 0,y 0 ||p||.
Since S 0 is a convex set, and p 0 is the projection of 0 onto it, p 0 is well-defined as a single-valued mapping. Moreover, we prove the following proposition, related to the convergence of the dual vectors associated to a solution of (P y,λ ) to the minimal norm certificate.

P roposi ti on 5 .4 One has

||p θ -p 0 || ||y -y 0 || λ + ε(λ),
where ε(λ) → 0 when λ → 0.

p ro o f This result is already proved by [START_REF] Duval | Exact Support Recovery for Sparse Spikes Deconvolution[END_REF] in the special case where J is the TV norm of a Radon measure (an infinite dimensional Banach space). By extension of the Definition 5.2, we denote p 0,y = argmin p∈S 0,y ||p||.

Formulation (D θ ) shows that p θ is the output of proximal operator of the function J * (Φ * •)/λ applied at the point y/λ, This shows that y/λ → p θ is 1-Lipschitz, see Proposition 10.1, and hence

||p θ -p 0 || ||p θ -p λ,y 0 || + ||p λ,y 0 -p 0 || ||w|| λ + ||p λ,y 0 -p 0 ||.
We now prove that

p θ λ→0 -→ p 0,y ,
which gives the desired result when setting y = y 0 in the previous formula.

Since p 0,y is a solution of (D 0,y ), one has -p 0,y , y -p θ , y . ( 5.6) This shows that {p θ } θ is bounded. Let p ⋆ be any cluster point. Operating as in the proof of Proposition 5.2, we have ∀x ∈ {x | y = Φx} ||y -Φx θ || 2 2λJ(x) and J(x θ ) J(x).

Letting λ → 0, we get by continuity that

x 0,y ∈ {x | y = Φx} and J(x θ ) J(x) , or equivalently, that x 0,y is a minimizer of (P y,0 ). Morever, from the primal-dual extremality relationships, we have Φ * p θ ∈ ∂J(x θ ). Since J is a proper closed convex function, the graph of ∂J is sequentially closed, which yields Φ * p ⋆ ∈ ∂J(x 0,y ), i.e. p ⋆ ∈ S 0,y . Now (5.6) implies that ||p ⋆ || ||p 0,y || and hence p ⋆ = p 0,y , which shows that p θ is converging to p 0,y .

The following lemma gives a useful characterization of non-degenerate dual vectors.

L emma 5 .3 Let J ∈ Γ + c (R n ) and x ∈ R n . Then, η ∈ ri ∂J(x) ⇐⇒ ∀u ∈ S \ {0}, ∃η ′ ∈ ∂J(x) such that u, η ′ -η > 0 .
Note that if J is a gauge, u can be normalized in the lemma, e.g. by restricting it to the unit sphere.

p ro o f First, recall that the directional derivative

J ′ (x, u) of J at x in the direction u is J ′ (x, u) = lim t↓0 J(x + tu) -J(x) t
.

From the characterization of the relative interior of a non-empty closed convex set (Hiriart-Urruty et al. 2001, Theorem V.2.2.3) or (Rockafellar 1996, Theorem 13.1), and sublinearity we deduce that η ∈ ri ∂J(x) ⇐⇒ J ′ (x, u) > u, η ∀u such that J ′ (x, u) + J ′ (-x, u) > 0 .

Using Theorem 3.1 shows that

J ′ (x, u) = e x , u + max η∈P S (∂J(x))
η, u .

Sublinearity implies that (Hiriart-Urruty et al. 2001, Corollary V.1.1.5)

J ′ (x, u) + J ′ (x, -u) 0 . Thus J ′ (x, u) + J ′ (x, -u) = max η∈P S (∂J(x)) η, u - min η∈P S (∂J(x))
η, u , whence we obtain

J ′ (x, u) + J ′ (x, -u) > 0 ⇐⇒ u / ∈ T .
Piecing everything together, we get

η ∈ ri ∂J(x) ⇐⇒ ∀u / ∈ T , J ′ (x, u) > u, η ⇐⇒ ∀u / ∈ T , ∃η ′ ∈ ∂J(x) such that u, η ′ > u, η ⇐⇒ ∀u / ∈ T , ∃η ′ ∈ ∂J(x) such that u, η ′ -η > 0 ⇐⇒ ∀u / ∈ T , ∃η ′ ∈ ∂J(x) such that u, η ′ S -η S > 0 ⇐⇒ ∀u ∈ S \ {0}, ∃η ′ ∈ ∂J(x) such that u, η ′ -η > 0 ,
which is the statement announced.

Restricted Injectivity

Let us consider (P y,0 ) when J = || • || 1 . Thus, we want to recover some vector x 0 ∈ R n from the observations y = Φx 0 . Assume that we know the support I 0 of x 0 . Remark that x 0 ∈ span(u i ) i∈I 0 ∩ {x | y = Φx 0 } where (u i ) i∈{1,...,n} is the canonical basis. Hence, to be uniquely recovered, one needs that Φ (I 0 ) has full rank. Conversely, if Φ (I 0 ) has not full rank, then any vector of the form x 0 + h with h ∈ Ker Φ (I 0 ) will be solution of (P y,0 ).

In general, this idea leads us to consider the following condition.

Defi ni ti on 5 .3 A subspace T ⊆ R n satisfies the restricted injectivity condition

(INJ T ) if Φ is injective on T .
For instance, (INJ T ) is equivalent to Φ (I) being full rank in the case of the ℓ 1 -norm, or equivalent to Ker Φ ∩ Ker D * (J) for the analysis ℓ 1 -norm, where J is some cosupport.

Uniqueness

In this section, we provide several results on the uniqueness of the solutions of (P y,λ ) and (P y,0 ).

Sublevel Set and its Cones

In the following, we draw a connection between the sublevel sets of a convex function and the uniqueness of the problem (P y,λ ).

The following proposition summarizes some key properties of the above cones when generated from the sublevel set of a continuous convex function. It will play a pivotal role in our proof of uniqueness (see Theorem 5.1 and Theorem 5.2). P roposi ti on 5 .5 Let J be a continuous convex function on R n . Then, Theorem 5 .1 Let J be a continuous convex function on R n . If Φ is injective on T slev x ⋆ J (x ⋆ ) then x ⋆ is the unique minimizer of (P θ ). In particular, If Φ is injective on T slev x 0 J (x 0 ) x 0 is the unique minimizer of (P y,0 ).

T slev x J (x) ⊂ δ | J ′ (x, δ) 0 . ( 5
p ro o f We provide the proof for (P θ ) when λ > 0, the proof is similar when λ = 0.

According to Proposition 2.2, any other minimizer different from x ⋆ can be written 2 . In both cases, we have the equality (5.7) since

slev x • 1 T slevx • 1 (x) N slevx • 1 (x) x (a) slev x 1 2 • 2 2 T slev x 1 2 • 2 2 (x) N slev x 1 2 • 2 2 (x) x (b)
(1, 0) / ∈ (∂J) -1 (0) = {(0, 0)}.
as x ⋆ + δ, where δ ∈ Ker(Φ) \ {0}, and J(x ⋆ + δ) = J(x ⋆ ). Therefore, we have

δ / ∈ T slev x ⋆ J (x ⋆ ), ∀δ ∈ Ker(Φ) \ {0} , ⇒J(x ⋆ + δ) > J(x ⋆ ), ∀δ ∈ Ker(Φ) \ {0}
⇒J has a unique minimizer x ⋆ , which concludes our proof.

The last statement coincides with that of [START_REF] Chandrasekaran | The Convex Geometry of Linear Inverse Problems[END_REF], Proposition 2.1) for atomic norms.

The Strong Nullspace Property

We are going to restate the previous Theorem 5.1 in a more meaningful way. We now compute the directional derivative of a bounded convex function J.

L emma 5 .4 Let J ∈ Γ + c (R n ). The directional derivative J ′ (x, δ) at point x ∈ R N in the direction δ reads J ′ (x, δ) = e x , δ T x + P S x (f x ), δ S x + J x f x (δ S x ).
p ro o f This comes directly from the structure of J x f x . Indeed, one has

J x f x (δ S x ) = J x f x (δ) Using Proposition 3.8(ii) = sup η∈∂J(x)-{f x } η, δ = -δ, f x + sup η∈∂J(x) η, d = -δ, f x + J ′ (x, δ) = -e x , δ T x -P S x (f x ), δ S x + J ′ (x, δ) ,
which concludes our proof.

The following condition is a generalization of the Null Space Property wellknown for ℓ 1 regularization [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF].

Theorem 5 .2 Let J ∈ Γ + c (R n ).
For a minimizer x ⋆ of (P y,λ ) (resp. a feasible point of (P y,0 )), let T = S ⊥ , e and f the subspace and vectors associated to it. If the Strong Null Space Property holds ∀δ ∈ Ker(Φ) \ {0}, e, δ T + P S (f), δ S < J x f (-δ S ), (NSP S ) then x ⋆ is the unique minimizer of (P y,λ ) (resp. (P y,0 )).

p ro o f From Lemma 5.4, the directional derivative J ′ (x, δ) at x ∈ R N in the direction δ reads

J ′ (x, δ) = e, δ T + P S f, δ S + J x f x (δ S ) .
Combining (5.7) in Proposition 5.5(i) and (68), applied at x ⋆ , together with the fact that Ker(Φ) is a subspace yield

∀δ ∈ Ker(Φ) \ {0}, e, δ T + P S f, δ S < J x f x (-δ S ) ⇐⇒ ∀δ ∈ Ker(Φ) \ {0}, J ′ (x, δ) > 0 =⇒ T slev x ⋆ J (x ⋆ ) ∩ Ker(Φ) = {0} .
We then conclude using Theorem 5.1.

Topological Conditions

A direct consequence of Theorem 5.2 above is the following corollary.

Theorem 5 . 3 Let J ∈ Γ + c (R n ).
For a minimizer x ⋆ of (P y,λ ) (resp. a feasible point of (P y,0 )), let T = S ⊥ , e and f the subspace and vector associated to it. Assume that (SC x ⋆ ) is verified with η = Φ * p ∈ ri ∂J(x ⋆ ), and that (INJ T ) holds. Then, x ⋆ is the unique minimizer of (P y,λ ) (resp. (P y,0 )).

p ro o f The source condition (SC x ⋆ ) implies that ∀ δ ∈ Ker(Φ) \ {0} δ, η = δ, Φ * p = Φδ, p = 0 . Moreover δ, η = δ T , e + δ S , η S = δ T , e + δ S , P S f + δ S , η S -P S f . Thus, applying the duality inequality of gauges we get

δ T , e + δ S , P S f J x f x (-δ S )J • f (η S -P S f) < J x f x (-δ S ) ,
where the last inequality is strict since δ S does not vanish owing to (INJ T ), and

α ∈ ri ∂J(x ⋆ ) is equivalent to η T = e and J x,• f x (η S -P S f) < 1.
The topological condition on the dual certificate required in Theorem 5.3 can be weakened to hold only on a subspace V ⊂ S and the conclusions of the corollary remain valid, and assuming a stronger restricted injectivity assumption. We have the following corollary of of Theorem 5.2.

C orollary 5 .1 With the same notations as in Theorem 5.2, suppose that J is such that

J x f x is separable on S = V ⊕ W. Assume that (SC x ⋆ ) is verified with J x,• f x (η V -P V (f)) < 1
, and (INJ V ) holds. Then, x ⋆ is the unique minimizer of (P y,λ ) (resp. (P y,0 )).

p ro o f We follow the same lines as the proof of Corollary 5.3 and get

δ T , η = δ T , e + δ S , P S f + δ V , η V -P V f + δ W , η W -P W f .
Moreover, by separability of J f on S, we have

J x,• f x (η S ) = sup J x fx (d) 1 d, η S = sup J x fx (d S ) 1 d S , η S = sup J x fx (d V )+J x fx (d W ) 1 d V , η V + d W , η W = sup ρ∈[0,1] sup J x fx (d V ) ρ,J x fx (d W ) 1-ρ d V , η V + d W , η W = sup ρ∈[0,1] ρ sup J x fx (d V ) 1 d V , η V + (1 -ρ) sup J x fx (d W ) 1 d W , η W = sup ρ∈[0,1] ρJ x,• f x (η V ) + (1 -ρ)J x,• f x (η W ) = max(J x,• f x (η V ), J x,• f x (η W )) .
This implies in particular that

J x,• f x (η W -P W f) max(J x,• f x (η V ), J • f (η W )) = J x,• f x (η S -P S ) 1 .
We therefore obtain δ T , e + δ S , P S f

J x f x (-δ V )J x,• f x (η V -P V f) + J x f x (-δ W )J x,• f x (η W -P W f) < J x f x (-δ V ) + J x f x (-δ W ) = J x f x (-δ S ) ,
where we used that δ / ∈ T , J x,• f x (η V -P V f) < 1 and separability of J x f x on S.

Construction of Non-Degenerate Certificates

Linearized Precertificate

Let us first introduce the definition of the linearized precertificate.

Defi ni ti on 5 .4 The linearized precertificate p F for x ∈ R n is defined by

p F = argmin (Φ * p) Tx =e x ||p||.
The intuition behind this definition is well-understood if one realizes that the existence of a dual certificate p is equivalent to η = Φ * p for some p such that η T = e x and J •

f x (η S -f S )
1. Dropping the last constraint, we recover the definition of p F . A nice property of this vector, is that under the restricted injectivity condition, it has a closed form expression. Lemma 5 . 5 Let x ∈ R n and suppose that (INJ T x ) is verified. Then p F is well-defined and

p F = Φ +, * T x e x .
p ro o f The vector p F is in fact the projection of 0 to the set p | (Φ * p) T x = e x . In particular,

Φ * T x p F = e x
Using hypothesis (INJ T x ), we multiply both sides by Φ +, * T x to get the result.

In fact, one can show that p F or p 0 being non-degenerate certificates are equivalent in some sense.

P roposi ti on 5 .6 Under the hypothesis (INJ T ), one has (5.8) (5.9) These conditions implies that x 0 is the unique solution of (P 0 ).

Φ * p F ∈ ri(∂J(x 0 )) =⇒ p 0 = p F ,
Φ * p 0 ∈ ri(∂J(x 0 )) =⇒ p 0 = p F ,
p ro o f Owing to Corollary 5.3,this shows that the left hand side conditions of both (5.8) and (5.9) implies that x 0 is a solution of P 0 .

Proof of (5.8) Under the condition Ker(Φ) ∩ T = {0}, one has, from the definition of Φ * ,+ T , that

p F = argmin p {||p|| | Φ * T p = e} (5.10)
Using Proposition 5.3 for w = 0 with x 0 being solution of (P 0 ), one sees that the constraint of problem (5.10) includes the constraint of the Definition 5.2. Indeed, one has ∀ η ∈ ∂J(x), P T x (η) = e x .

If η F ∈ ri(∂J(x 0 )), then it is a feasible point in the definition of p 0,w when w = 0. Hence, necessarily p 0 = p F .

Proof of (5.9) Since x 0 is a solution of (P 0 ), according to Proposition 5.3, one has that

p 0 = argmin p ||p|| 2 | Φ * T p = e, Φ * S p ∈ U
where we have denoted S = T ⊥ and U = P S (∂J(x 0 )). The first order condition of this problem state the existence of q ∈ R n and u ∈ R q such that p 0 + Φ T q + u = 0 where

   Φ * T p 0 = e, u ∈ N U (p 0 ).
The condition Φ * p 0 ∈ ri(∂J(x 0 )) implies that Φ * S p 0 ∈ ri(U) and thus N U (Φ * S p 0 ) = T . This implies Φ S u = 0 and hence one has the equation

Φ * T p 0 + Φ * T Φ T q = e + Φ * T Φ T q = 0
which leads to p 0 = (Φ T ) +, * e = p F .

Beside condition (INJ T x ) stated above, the following Identifiability Criterion will play a pivotal role.

Defi ni ti on 5 .5 For x ∈ R N such that (INJ T x ) holds, we define the Identifiability Criterion at x as

IC(x) = J x,• f x (Φ * S x Φ +, * T x e x -P S x f x ).
The fact that IC(x) < 1 is totaly equivalent to Φ * p F ∈ ri ∂J(x) but stated in analytical form. Note that if J is a strong gauge on T , then it becomes

IC(x) = J x,• f x (Φ * S x Φ +, *
T x e x ). The Identifiability Criterion clearly brings into play the promoted subspace T x 0 and the interaction between the restriction of Φ to T x 0 and S x 0 . It is a generalization of the irrepresentable condition that has been studied in the literature for some popular regularizers, including the ℓ 1 -norm [START_REF] Fuchs | On sparse representations in arbitrary redundant bases[END_REF]), analysis-ℓ 1 (Vaiter, Peyré, et al. 2013), ℓ 1 -ℓ 2 (Bach 2008a) and nuclear (Bach 2008b).

It turns out that in such a setting, IC(x 0 ) < 1 is a sufficient condition for identifiability without any any other particular assumption on the finite-valued function J, such as partial smoothness. By identifiability, we mean the fact that x 0 is the unique solution of (P y,0 ). P roposi ti on 5 .7 Let x 0 ∈ R N and T = T x 0 . We assume that (INJ T x 0 ) holds and IC(x 0 ) < 1. Then x 0 is the unique solution of (P y,0 ). p ro o f This is a straightforward consequence of the first order condition using p 0 as a dual certificate. Denote e = e x 0 , f = f x 0 and S = T ⊥ . Taking the dual vector p = Φ +, * T e, we have on the one hand

Φ * T Φ +, * T e = e
since e ∈ Im(Φ * T ). On the other hand,

J x,• f x (Φ * S Φ +, * T e -P S f) = IC(x 0 ) < 1.
We conclude thanks to Theorem 5.3.

Analysis Precertificate

In the special case of analysis ℓ 1 regularization, [START_REF] Nam | The cosparse analysis model and algorithms[END_REF] introduced a different precertificate. We extend their idea to any function of the form We can give an explicit form of this certificate using a basis of Ker Φ.

J = J 0 • D * D efi ni
P roposi ti on 5 .8 Let N * be a basis of Ker Φ. Then,

p A = -D(ND S ) + NDe D * x .
p ro o f Since N * be a basis of Ker Φ, one has Im Φ * = Ker N. Thus,

Dω ∈ Im Φ * ⇔ NDω = 0.
Hence, Dω ∈ Im Φ * and ω T = e D * x ⇔ ND(e D * x + P S ω) = 0.

The least-square solution to this linear equation, which coincides with its minimal ℓ 2 norm solution ω, yields

p A = -D(ND S ) + NDe D * x ,
which concludes the proof.

We explore numerically the difference between this precertificate and the linearized in Section 10.2. We draw the attention of the reader to the fact that in [START_REF] Nam | The cosparse analysis model and algorithms[END_REF] T hi s chapter is dedicated to seek sufficient conditions under which for any solution of (P y,λ ), one has

||x ⋆ -x 0 || = O(||w||).
This condition depends on x 0 , while λ must be chosen proportionaly to the noise level ||w||. The terminology "linear" in the convergence rate, which stems from the inverse problems community, pertains to the fact that the error is within a factor of the noise level.

In Section 6.1, we establish the rate of convergence of any solution x ⋆ to x 0 with respect to the Bregman divergence, introduced in Definition 2.23. Section 6.2 states our main result, namely the fact that any if both the source condition and the restricted injectivity hold, then (P y,λ ) enjoys a linear convergence rate with respect to the estimation error. Finally, in Section 6.3, we draw connections with previous works on this subject.

Bregman Rate

The following Lemma 6.1 gives the prediction error and Bregman distance rates for (P y,λ ). Such results can be found in [START_REF] Scherzer | Variational methods in imaging[END_REF].

Lemma 6 . 1 Suppose that (SC x 0 ) is satisfied with η = Φ * p ∈ ∂J(x 0 ). Then, for any minimizer x ⋆ of (P y,λ ), and with λ = cε for some c > 0 and ε = ||w||, we have

D η (x ⋆ , x 0 ) ε (1 + c||p|| 2 /2) 2 c
Bregman divergence rate,

||Φx ⋆ -Φx 0 || 2 ε(2 + c||p|| 2 )
Prediction error.

p ro o f The proof follows the same lines as [START_REF] Scherzer | Variational methods in imaging[END_REF]. For the sake of completness, we provide here a proof.

By definition of x ⋆ , one has

||Φx ⋆ -y|| 2 + λJ(x ⋆ ) ||Φx 0 -y|| 2 + λJ(x 0 ).
Since Φx 0y = -w, one has

||Φx ⋆ -y|| 2 + λJ(x ⋆ ) ε 2 + λJ(x ⋆ ). (6.1)
Now, by definition of D η (x ⋆ , x 0 ), one has

D η (x ⋆ , x 0 ) = J(x ⋆ ) -J(x 0 ) -Φ * p, x ⋆ -x 0 = J(x ⋆ ) -J(x 0 ) -p, Φ(x ⋆ -x 0 ) . (6.2)
Using the Cauchy-Schwarz inequality, we have that

p, Φ(x ⋆ -x 0 ) ||p|| ||Φ(x ⋆ -x 0 )||.
By the fact that Φ(x ⋆x 0 ) = Φx ⋆y + w and the triangle inequality, one has

p, Φ(x ⋆ -x 0 ) ||p|| (||Φx ⋆ -y|| + ||w||) .
Injecting this in (6.2), we get

D η (x ⋆ , x 0 ) J(x ⋆ ) -J(x 0 ) + ||p||||Φx ⋆ -y|| + ||p||ε.
Starting from (6.1), we have

ε 2 ||Φx ⋆ -y|| 2 + λ (D η (x ⋆ , x 0 ) -||p||||Φx ⋆ -y|| -||p||ε) . (6.3) Using that a 2 + b 2 2ab, λ||p||||Φx ⋆ -y|| ||Φx ⋆ -y|| 2 + λ 2 4 ||p|| 2 .
Thus, we get

ε 2 λD η (x ⋆ , x 0 ) -λε||p|| - 1 4 λ 2 ||p|| 2 .
Finally, we have

λD η (x ⋆ , x 0 ) ε 2 + ε||p|| 2 λ 2 + 1 4 λ 2 ||p|| 2 = ε + λ||p|| 2 2 ,
which proves the first inequality (Bregman divergence rate). Now, for the second one, we start from

||Φx ⋆ -Φx 0 || ||Φx ⋆ -y|| + ||y -Φx 0 || = ||Φx ⋆ -y|| + ε.
Using bound (6.3) and the fact that D η (x ⋆ , x 0 ) is nonnegative, we have that

||Φx ⋆ -y|| ε + λ||p||.
Hence,

||Φx ⋆ -Φx 0 || 2ε + λ||p||,
which concludes our proof.

6.2 Linear Convergence Rate

Main Result

We are now ready to state our main convergence results. We denote x 0 ∈ R n and denote T 0 = T x 0 , f 0 = f x 0 . Theorem 6 .1 Assume that (SC x 0 ) holds at x 0 with η = Φ * p ∈ ri ∂J(x 0 ), and that (INJ T 0 ) holds. Choosing λ = cε, c > 0, where ε = ||w||, the following holds for any minimizer x ⋆ of (P y,λ )

||x ⋆ -x 0 || 2 Cε ,
where

C = C 1 (2 + c||p|| 2 ) + C 2 (1 + c||p|| 2 /2) 2 c 1 -J x 0 ,• f 0 (η S 0 -P S 0 f 0 ) (6.4)
and C 1 > 0 and C 2 > 0 are constants independent of p and η.

This result holds for any finite-valued convex function and holds for any minimizer of (P y,λ ) (not necessarily unique). However, remark that (INJ T 0 ) makes sense only if J promotes subspaces of low dimension. Note that finding a certificate p is not trivial, and that the constant involved in Theorem 1 depends on it. This leaves a degree of freedom to optimize the constant for the certificate. The closer to 1 the constant C p = 1 -J x 0 ,• f 0 (η S 0 -P S 0 f 0 ) is, the better is the robustness. It measures how far from the relative boundary of the subdifferential of J at x 0 is p. Finally, the constants C 1 and C 2 are not absolute and may depend on the dimension. Hence, this theorem does not extend straightforwardly to the infinite-dimensional problem (cP y,λ ).

The constants read as follows.

C 1 = C -1 Φ and C 2 = ||Φ|| 2,2 + C Φ C J C Φ ,
where C Φ is the coercivity constant associated to the hypothesis (INJ T 0 ), i.e.

∃ C Φ > 0 s.t. ||Φx|| 2 C Φ ||x|| 2 , ∀x ∈ T 0 ,
and C J is defined by the coercivity of J x 0 f 0 on S 0 :

∃ C J > 0 s.t. ∀x ∈ R n , J x 0 f 0 (η) C J ||η|| 2 .
When the decomposable norm is also separable (see Corollary 5.1), the stability result of Theorem 6.1 remains true assuming that J • f (η V -P V f) < 1 for V ⊂ S 0 . This however comes at the price of the stronger restricted injectivity assumption (INJ V ⊥ ). To show this, the only thing to modify is the statement and the proof of Lemma 6.2 which can be done easily using similar arguments to those in the proof of Corollary 5.1.

Proof of Theorem 6.1

Let T 0 and e 0 be the subspace and generalized sign vector associated to x 0 , and denote S 0 = T 0 ⊥ . We choose some f 0 ∈ ri ∂J(x 0 ). Now as J x 0 f 0 is coercive and bounded on S 0 (see Lemma 3.8), we get

∃ C J > 0 s.t. ∀x ∈ R n , J x 0 f 0 (η) C J ||η|| 2 .
We obtain the following bound on the projected distance between x ⋆ and x 0 . Lemma 6 . 2 Suppose that (SC x 0 ) holds at x 0 with η ∈ ri ∂J(x 0 ). Then,

|| P S 0 (x ⋆ -x 0 )|| 2 D η (x ⋆ , x 0 ) C J 1 -J x 0 ,• f 0 (η S 0 -P S 0 f 0 )
. p ro o f From the properties of J f 0 (see Lemma 3.8), there exists v ∈ S 0 such that

J x 0 ,• f 0 (v) 1 and J x 0 f 0 (x ⋆ -x 0 ) = J x 0 f 0 (P S 0 (x ⋆ -x 0 )) = P S 0 (x ⋆ -x 0 ), v . 158

Relation to Previous Works

Moreover, v + P S 0 f 0 + e 0 ∈ ∂J(x 0 ). Thus

D η (x ⋆ , x 0 ) D η (x ⋆ , x 0 ) -D v+P S 0 f 0 +e 0 (x ⋆ , x 0 ) = v + P S 0 f 0 + e 0 -η, x ⋆ -x 0 = v -(η S 0 -P S 0 f 0 ), x ⋆ -x 0 = J x 0 f 0 (P S 0 (x ⋆ -x 0 )) -η S 0 -P S 0 f 0 , P S 0 (x ⋆ -x 0 ) J x 0 f 0 (P S 0 (x ⋆ -x 0 )) 1 -J x 0 ,• f 0 (η S 0 -P S 0 f 0 ) C J || P S 0 (x ⋆ -x 0 )|| 2 1 -J x 0 ,• f 0 (η S 0 -P S 0 f 0 ) ,
where in the last two inequalities, we used the duality inequality on dom J x 0 ,•

f 0 × dom J x 0 f 0 with dom J x 0 f 0 = R N and dom J x 0 ,• f 0 = S 0 .
We now give the proof of Theorem 6.1.

p ro o f ||x ⋆ -x 0 || 2 || P T 0 (x ⋆ -x 0 )|| 2 + || P S 0 (x ⋆ -x 0 )|| 2 C Φ -1 ||Φ P T 0 (x ⋆ -x 0 )|| 2 + || P S 0 (x ⋆ -x 0 )|| 2 C Φ -1 ||Φ(x ⋆ -x 0 )|| 2 + (1 + C Φ -1 ||Φ|| 2,2 )|| P S 0 (x ⋆ -x 0 )|| 2 ,
where we used assumption (INJ T 0 ), i.e.,

∃ C Φ > 0 s.t. ||Φx|| 2 C Φ ||x|| 2 , ∀x ∈ T 0 .
We finally apply Lemma 6.2 to get

||x ⋆ -x 0 || 2 C Φ -1 ||Φ(x ⋆ -x 0 )|| 2 + ||Φ|| 2,2 +C Φ C J C Φ 1-J x 0 ,• f 0 (η S 0 -P S 0 f 0 ) D η (x ⋆ , x 0 ) .
Using Lemma 6.1 yields the assertion.

Relation to Previous Works

Convergence rates. The monograph [START_REF] Scherzer | Variational methods in imaging[END_REF]) is dedicated to regularization properties of inverse problems in infinite-dimensional Hilbert and Ba-nach spaces with application to imaging. In particular, Chapter 3 of this book treats the case where J is a coercive gauge for the problem (P y,λ ). In [START_REF] Burger | Convergence rates of convex variational regularization[END_REF], the authors consider the case where J is a proper, convex and l.s.c functional for both the constrained and Lagrangian regularization (cP y,λ ). Under the source condition and a restricted injectivity assumption, they bound the error in Bregman divergence with a linear rate O(||w||). For the classical Thikonov regularization, i.e. J = || • || L 2 (Ω) , the estimation is in O( ||w||), which is not a linear convergence. Extensions of these results have been proved in [START_REF] Resmerita | Regularization of ill-posed problems in Banach spaces: convergence rates[END_REF] and [START_REF] Hofmann | A convergence rates result for Tikhonov regularization in Banach spaces with non-smooth operators[END_REF] for the Bregman rate. [START_REF] Lorenz | Convergence rates and source conditions for Tikhonov regularization with sparsity constraints[END_REF] treats the case where J is a ℓ p norm with 1 p 2 and provides a prediction error Φx 0 -Φx ⋆ in O(||w||) and an estimation error x ⋆x 0 in O( ||w||). Grasmair et al. ( 2011) is concerned with the special case of ℓ 1 regularization, and draws some connection with the restricted isometry property (RIP), see below. The results that are the closest to our are contained in [START_REF] Grasmair | Linear convergence rates for Tikhonov regularization with positively homogeneous functionals[END_REF]. Here, J is a proper, convex, l.s.c and positively homogeneous functional on some Banach space H. Under a source condition and restricted injectivity on a an appropriate cone, a linear convergence rate is proved with respect to J, i.e.

J(x ⋆ -x 0 ) = O(||w||).
This result implies ours, but only if J is injective which precludes many important regularizers, e.g. TV.

Compressed sensing. In a compressed sensing setting, for instance when Φ is drawn from a i.i.d. normal distribution, it was proved [START_REF] Rudelson | On sparse reconstruction from Fourier and Gaussian measurements[END_REF]) that if the number of measurements q is such that q k log(n/k) where k = ||x 0 || 0 then there exists with high probability on Φ a non-degenerate certificate when J = || • || 1 , i.e. (SC x ) holds and one can apply the result of Theorem 6.1.

The performance of compressed sensing recovery has initially been analyzed using the so-called restricted isometry property (RIP) introduced in (Candès et al. 2006a(Candès et al. , 2006b;;Candès and Tao 2006) for ℓ 1 . It is defined for a couple

(Φ, k)
where k is a targeted sparsity, as the smallest constant δ k such that

(1 -δ k )||x|| 2 ||Φx|| 2 (1 + δ k )||x|| 2 , ( 6.5)
for any vector x such that ||x|| 0 k. It is shown (Candès et al. 2006a) that if δ 2k + δ 3k < 1, then for every vector x 0 of sparsity k, there exists a nondegenerate certificate [START_REF] Candès | Near-optimal signal recovery from random projections: Universal encoding strategies?[END_REF], Lemma 2.2) as remarked also by [START_REF] Grasmair | Necessary and sufficient conditions for linear convergence of l1-regularization[END_REF]. This result thus implies linear convergence rate, and is applied in (Candès et al. 2006b) to show the robustness to noise of compressed sensing. This was generalized to analysis sparsity (i.e. J = ||D * • || 1 with D tight frame) in [START_REF] Candès | Compressed sensing with coherent and redundant dictionaries[END_REF], structured sparsity in [START_REF] Candès | Compressed sensing with coherent and redundant dictionaries[END_REF]) and matrix completion [START_REF] Recht | Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization[END_REF]; Candès and Plan 2011b) using J = || • || * . A major shortcoming of this approach is that available designs of matrices satisfying (6.5) for reasonnably large value of k are essentially random. Indeed, in this case, the constant δ k can be shown to be small enough with high probability on Φ for nearly optimal scaling of (n, q, k). For instance, when Φ is drawn for the Gaussian ensemble, it is the case when q k log(n/k). as proved by Candès and Tao (2006) Note that in general, computing the RIP constants for a given matrix is an NP-hard problem [START_REF] Bandeira | Certifying the Restricted Isometry Property is Hard[END_REF].

The golfing scheme introduced by Gross (2011) for the nuclear norm allows to consider structured non-Gaussian measurements, e.g. partial Fourier measurements. It is based on an iterative scheme starting from the linearized precertificate p F in order to construct an (approximate) certificate with high probability on the matrix for a given vector. It was further studied by Candès and Plan (2011a) for ℓ 1 regularization.

S

o far, we were concerned with ℓ 2 -stability/robustness. What can be said about the recovery of the model T x 0 underlying the original vector itself x 0 ? To be able to state such a result, the regularization has to enjoy some additional regularity assumption. This is the goal of partial smoothness that we introduced in Definition 4.1.

Section 7.1 states our main result. It ensures that for a partly smooth function J, if the restricted injectivity holds and that the linearized precertificate p F is a non-degenerate certificate, then in a small noise regime, (P y,λ ) has a unique solution and it belongs to the same model manifold M as x 0 . In Section 7.2, we specialize this result to partly smooth functions with linear manifold. This specialization does not cover for instance the nuclear norm regularization, but provides more explicit constants involved in the robustness. Finally, we draw connections with previous works in Section 7.3.

Selection Against Small Noise: General Case

Sensitivity of the Lagrangian Problem

Before diving into our main result, we first show of the theory of partly smooth functions introduced in (Lewis 2002) can be directly applied to study the sensitivity of (P θ ) for λ > 0.

Theorem 7 .1 Let x θ be a solution of (P θ ), with λ > 0, and suppose that J is locally partly smooth at x θ relative to M. If Ker(Φ) ∩ T x θ = {0} and η θ ∈ ri(∂J(x θ )) (7.1) where we have denoted

η θ = Φ * p θ = 1 λ Φ * (y -Φx θ ),
then for θ ′ close enough from θ, the solution x θ ′ of (P θ ) is unique and satisfies

x θ ′ ∈ M.
p ro o f It suffies to apply Theorem 5.7 of [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF]. Indeed, the function

f f(x, θ) = J(x) + 1 2λ ||Φx -y|| 2
is partly smooth at (x θ , θ) relative to the manifold M × Θ, where Θ = R q × R + and condition (7.1) is exactly equivalent to x θ being a strong minimizer of f(•, θ), see (Lewis 2002, Definition 5.6).

Condition (7.1) is not very useful because it depends on the solution x θ and not on the data to recover x 0 . The rationale behind Theorem 7.2 is to make θ tends to 0, and under the conditions λ → 0 and ||w|| λ → 0, Propositions 5.2 and 5.4 ensure that

x θ → x 0 and p θ → p 0 .

Main Result

We now state our main result, which performs a sensivity analysis at λ = 0.

Theorem 7 . 2 Let J ∈ S x 0 (M) a locally partly smooth function at x 0 relative to M such that (INJ T ) with T = T x 0 holds and p F is a non-degenerate certificate, i.e.

Ker(Φ) ∩ T = {0}, and Φ * p F ∈ ri(∂J(x 0 )).

Then there exists positive constants (C, C ′ ) such that if ||w|| C and λ = C ′ ||w||, then the solution x θ of (P θ ) is unique and satisfies

x θ ∈ M and ||x θ -x 0 || = O(||w||). (7.
3)

The heuristic underlying the hypotheses of Theorem 7.2 is that the conditions in (7.1) converge toward those of (7.2). Indeed, according to Proposition 5.6, (7.2) implies p 0 = p F . This is precisely what we need to show in order to prove Theorem 7.2.

Obviously, the assumptions of Theorem 7.2 imply the conclusion of Theorem 6.1. Contrary to the latter, the former is based on an explicit formulation of the precertificate p F . Note that there exist vectors which can be stably recovered in the ℓ 2 sense of Theorem 6.1, but whose underlying manifold model cannot be stably identified in the sense of Theorem 7.2, see our numerical experiments in Chapter 10.

The following proposition shows that Theorem 7.2 is in some sense sharp, since the hypothesis Φ * p F ∈ ri(∂J(x 0 )) (almost) characterizes the stability of M.

P roposi ti on 7 .1 We suppose that x 0 is the unique solution of (P y,0 ) and that Ker(Φ) ∩ T = {0}, and

Φ * p F ∈ ri(∂J(x 0 ))
Then there exists C > 0 such that for ||w|| Cλ and any λ > 0 small enough, then any solution x θ of (P y,λ ) satisfies x θ / ∈ M.

In the particular case where w = 0 (no noise), this result shows that the manifold M is not correctly identified when solving (P y,λ ) for any λ > 0 small enough.

The only case not covered by neither Theorem 7.2 nor Proposition 7.1 is when Φ * p F ∈ rbound(∂J(x 0 )) (the relative boundary). In this case, one cannot conclude, since depending on the noise w, one can have either stability or nonstability of M. We refer to Chapter 10 where an example illustrates this situation for the 1-D total variation J = ||∇ • || 1 (here ∇ is a discretization of the 1-D derivative operator).

We aim at showing that for (||w||/λ, λ) small enough, xθ is the unique solution of (P θ ).

The proof of Proposition 5.2 carries over verbatim to this constrained problem, which shows that xθ → x 0 when

   λ -→ 0, ||w|| 2 /λ -→ 0. (7.5)
In the following, to lighten the notations, we denote T xθ = T .

Convergence of the tangent model subspaces. By definition of the constrained problem (7.4), xθ ∈ M. Moreover, since f(•, θ) is partly smooth at x 0 relative to M, the sharpness property Definition 4.1(ii) holds at all nearby points in the manifold M, see (Lewis 2002, Proposition 2.10). Thus, as soon as (||w|| 2 /λ, λ) is small enough, we have that M is a C 2 -manifold around xθ and T = T M (x θ ). Using the fact that M is of class C 2 , we get the following convergence

T = T M (x θ ) -→ T M (x 0 ) = T when    λ -→ 0, ||w|| 2 /λ -→ 0, (7.6)
where the convergence should be understood over the Grasmanian of linear spaces with the same dimension (or equivalently, as the convergence of the projection operators P T → P T ), see Section 2.2. Since Ker(Φ) ∩ T = {0}, (7.6) implies that for (||w|| 2 /λ, λ) small enough,

Ker(Φ) ∩ T = {0}, (7.7)
which we also assume now.

First order conditions. By partial smoothness, the restriction of J to M is smooth at xθ for θ small enough. Hence, since x → 1 2λ ||y -Φx|| 2 is smooth everywhere, the smooth perturbation rule (Lewis 2002, Corollary 4.7) implies that f(•, θ) is also partly smooth at xθ for M, and thus its restriction to M is smooth at xθ . Therefore, Lewis (2002, Proposition 2.4(b)) applies, and it follows that xθ is a critical point of (7.5) if, and only if,

0 ∈ aff(∂f(x, θ)) = 1 λ Φ * (Φx θ -y) + aff(∂J(x θ )) = 1 λ Φ * (Φx θ -y) + e xθ + T ⊥ .
The first equality comes from the fact that f(•, θ) is a closed convex function and the second one from the decomposability of the subdifferential. Projecting this relation onto T , we get

Φ * T (Φx θ -y) + λe xθ = 0, (7.8)
Convergence of primal variables. Since xθ and x 0 belongs to the same active manifold, and M is a manifold of class C 2 around them, using Lemma 2.3, each point in their neighbourhoods has unique projection on M. In particular, xθ = P M (x θ ) and x 0 = P M (x 0 ). Moreover, P M is of class C 1 near xθ . Thus, continuous differentiability shows

xθ -x 0 = P M (x θ ) -P M (x 0 ) = DP M (x θ )(x θ -x 0 ) + o (||x θ -x 0 ||) .
where DP M (x θ ) is the derivative of P M at xθ . Combining (Lewis et al. 2008, Lemma 2.3) and (Lewis 2002, Proposition 2.4(i)), the derivative DP M (x θ ) is given by DP M (x θ ) = P T .

Inserting this in (7.8), we get

Φ * T Φ(x θ -x 0 ) = Φ * T Φ T (x θ -x 0 ) + o (||x θ -x 0 ||) = Φ * T w -λe xθ .
Using (7.7), Φ T has full rank, and thus xθ -

x 0 = Φ + T w -λ(Φ * T Φ T ) -1 e xθ + o (||x θ -x 0 ||) . (7.9)
Altogether, we obtain the bound

||x θ -x 0 || = O(||w||, λ).
Convergence of dual variables. We define ηθ = Φ * pθ where pθ = y -Φx θ λ , Arguing as above, and using (7.9) we have

λp θ = Φ(x 0 -xθ ) + w = Φ T (x 0 -xθ ) + w + o (||x θ -x 0 ||) = P Im(Φ T ) ⊥ w + λΦ +, * T e xθ + o (||x θ -x 0 ||) .
We thus arrive at

||p θ -p F || = O ||w|| λ , ||Φ +, * T e xθ -Φ +, * T e|| .
Since M is a C 2 manifold, and by partial smoothness x → e x is C 1 on M (recall that J is C 2 on M), one has

||e xθ -e|| = O(||x θ -x 0 ||).
Since A → A +, * is smooth at A = Φ T along the manifold of matrices of constant rank, one has

||Φ +, * T -Φ +, * T || = O(||Φ T -Φ T ||) = O(||P T -P T ||||Φ||) = O(||x θ -x 0 ||).
This implies

||Φ +, * T e xθ -Φ +, * T e|| ||Φ +, * T -Φ +, * T ||||e xθ || + ||e xθ -e||||Φ +, * T || = O(||x θ -x 0 ||).
Altogether, we get the bound

|| ηθ -η F || = O(||w||/λ, λ). (7.10)
Convergence inside the relative interior. Using the hypothesis that p F ∈ ri(∂J(x 0 )), we will show that for (||w||/λ, λ) small enough, pθ ∈ ri(∂J(x θ )).

We follow the line of proof of [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF].

Let us suppose this does not hold. Then there exists a sequence (θ n = (λ n , w n )) n , with (w n /λ n , λ n ) tending to 0, such that pn ∈ rbound(∂J(x n )) (7.12) where we used the shorthand notations xn = xθ n and pn = pθ n .

According to (7.10) and (7.5),

(x n , pn ) → (x 0 , p F ). (7.13)
Condition (7.12) is equivalently stated as, for each n

∃z n ∈ T ⊥ xn , ∀ p ∈ ∂J(x n ), z n , p -pn 0, (7.14)
where one can impose the normalization ||z n || = 1. Up to a sub-sequence (that for simplicity we still denote z n with a slight abuse of notation), since z n is in a compact set, we can suppose that z n → z ⋆ .

Since

T ⊥ xn → T ⊥ because M is a C 2 manifold, one has that z ⋆ ∈ T ⊥ . We now show that ∀ v ∈ ∂J(x 0 ), z ⋆ , p -p F 0.
(7.15) Indeed, let us consider any v ∈ ∂J(x 0 ). By condition 3 of Definition 4.1, ∂J is continuous along M, so that since xn → x 0 there exists v n ∈ ∂J(x n ) with v n → v. Applying (7.14) with p = v n gives z n , v n -pn 0.

Taking the limit n → +∞ in this inequality leads to (7.15), which contradicts the fact that p F ∈ ri(∂J(x 0 )).

Conditions (7.11) and (7.7) implies, using Theorem 5.3, that xθ = x θ is the unique solution of (P θ ).

Proof of Proposition 7.1

Let x θ be a solution of (P y,λ ). Suppose that x θ ∈ M. In particular, x θ is a solution of the non-convex minimization (7.4). Arguing as in the proof of Theorem 7.2, we get the bound (7.10), i.e. .16) Since x 0 is the unique solution of (P y,0 ), p 0 is well defined, hence η 0 = Φ * p 0 ∈ ∂J(x). Thus, there exists K > 0 (for instance

||η θ -η F || = O(||w||/λ, λ) where η θ = Φ * y -Φx θ λ . ( 7
K = d(η F , ∂J(x))) such that ||η F - η 0 || > K. Moreover, ||η F -η 0 || ||η F -η θ || + ||η θ -η 0 ||.
According to (7.16) and (5.4), one has

||η F -η θ || → 0 and ||η θ -η 0 || → 0.
This leads to a contradiction since by assumption η F ∈ ∂J(x 0 ), hence x θ ∈ M.

Selection of Linear Manifold

When J is partly smooth with linear manifold (M = T x ), see Definition 4.2, i.e. the manifold is in fact the model subspace, we derive a more precise result with explicit constants.

Main Result

Theorem 7 . 3 Let x 0 ∈ R n and T = T x 0 . We suppose that J is a partly smooth function with linear manifold at x 0 with the corresponding parameters (Γ , ν x 0 , µ x 0 , τ x 0 , ξ x 0 ) where the constants are defined in (4.2), (4.3) and (4.4). Assume that (INJ T ) holds and IC(x 0 ) < 1. Then there exist positive constants (A T , B T ) that solely depend on T and a constant C(x 0 ) such that if w and λ obey

A T 1 -IC(x 0 ) ||w|| λ ν x 0 min B T , C(x 0 ) (7.17)
the solution x ⋆ of (P y,λ ) with noisy measurements y is unique, and satisfies T x ⋆ = T . Furthermore, one has

||x 0 -x ⋆ || = O max(||w||, λ) .
Clearly this result asserts that exact recovery of T x 0 from noisy partial measurements is possible with the proviso that the regularization parameter λ lies in the interval (7.17). The value λ should be large enough to reject noise, but small enough to recover the entire subspace T x 0 . In order for the constraint (7.17) to be non-empty, the noise-to-signal level ||w||/ν x 0 should be small enough, i.e.

||w|| ν x 0 1 -IC(x 0 ) A T min (B T , C(x 0 )) .
The constant C(x 0 ) involved in this bound depends on x 0 and has the form

C(x 0 ) = 1 -IC(x 0 ) ξ x 0 ν x 0 H D T µ x 0 + τ x 0 ξ x 0 where H(β) = β + 1/2 E T β ϕ 2β (β + 1) 2 and ϕ(u) = √ 1 + u -1 .
The constants (D T , E T ) only depend on T . C(x 0 ) captures the influence of the parameters π x 0 = (µ x 0 , τ x 0 , ξ x 0 ), where the latter reflect the geometry of the regularizing function J at x 0 . More precisely, the larger C(x 0 ), the more tolerant the recovery is to noise. Thus favorable regularizers are those where C(x 0 ) is large, or equivalently where π x 0 has small entries, since H is a strictly decreasing function.

Proof of Theorem 7.3

The proof is similar to Theorem 7.2. To lighten the notations, we let ε = ||w||,

ν = ν x 0 , µ = µ x 0 , τ = τ x 0 , ξ = ξ x 0 , f = f x 0 and T = T x 0 .
The strategy is to construct a vector which is the unique solution to

min x∈T 1 2 ||y -Φx|| 2 + λJ(x) , (P T θ )
and then to show that it is actually the unique solution to (P θ ) under the assumptions of Theorem 7.3.

The following lemma gives a convenient implicit equation satisfied by the unique solution to (P T θ ).

Lemma 7 . 1 Assume that (INJ T ) holds. Then (P T θ ) has exactly one minimizer x, and the latter satisfies

x = x 0 + Φ + T w -λ(Φ * T Φ T ) -1 ẽ
where ẽ ∈ P T (∂J(x)). (7.18) p ro o f Assumption (INJ T ) implies that the objective in (P T θ ) is strongly convex on the feasible set T , whence uniqueness follows immediately. By a change of variable, (P T θ ) be also rewritten in the unconstrained form

x = argmin x∈R N 1 2 ||y -Φ T x|| 2 + λJ(P T x) .
Thus, using Proposition 3.9(i), x has to satisfy

Φ * T (y -Φ T x) + λ ẽ = 0,
for any ẽ ∈ P T (∂J(x)). Owing to the invertibility of Φ on T , i.e. (INJ T ), we obtain (7.18).

We are now in position to prove Theorem 7.3. This is be achieved in three steps:

Step 1: We show that in fact T x = T .

Step 2: Then, we prove that x is the unique solution of (P y,λ ) using Theorem 5.3.

Step 3: We finally exhibit an appropriate regime on λ and ε for the above two statements to hold.

Step 1: Subspace equality. By construction of x in (P T θ ), it is clear that x ∈ T . The key argument now is to use that J is PRG at x 0 , and to show that

Γ (x 0 -x) ν, (7.19)
which in turn will imply subspace equality, i.e. T x = T (see Definition 4.2).

We have from (7.18) and subadditivity that .20) where α 0 = Γ ( ẽ). Consequently, to show that (7.19) is verified, it is sufficient to prove that

Γ (x 0 -x) Γ (-Φ + T w) + λΓ ((Φ * T Φ T ) -1 ẽ) (Φ * T Φ T ) -1 Γ →Γ {Γ (-Φ * T w) + λΓ ( ẽ)} (Φ * T Φ T ) -1 Γ →Γ {| | |Φ * T | | | ℓ 2 →Γ ε + α 0 λ} . ( 7
Aε + Bλ ν, (C 1 )
where we set the positive constants

A = (Φ * T Φ T ) -1 Γ →Γ | | |Φ * T | | | ℓ 2 →Γ , B = α 0 (Φ * T Φ T ) -1 Γ →Γ .
Suppose for now that (C 1 ) holds and consequently, T x = T . Then decomposability of J on T (Theorem 3.1) implies that

ê = P T x (∂J(x)) = P T (∂J(x)) = ẽ,
where we have denoted ê = e x. Thus (7.18) yields the following implicit equation

x = x 0 + Φ + T w -λ(Φ * T Φ T ) -1 ê. (7.21)
Step 2: x is the unique solution of (P y,λ ). Recall that under condition (C 1 ), J is decomposable at x and x 0 with the same model subspace T . To deduce that x is the unique solution of (P y,λ ), it remains to show that

J • f (λ -1 Φ * S (y -Φx) -fS ) < 1. (7.22)
where we use the shorthand notations f = f x and fS = P S f.

Under condition (C 1 ), the ξ-stability property (4.4) of J at x 0 yields

J • f (λ -1 Φ * S (y -Φx) -fS ) 1 + ξΓ (x 0 -x) J • f (λ -1 Φ * S (y -Φx) -fS ). (7.23)
Furthermore, from (7.21), we can derive

λ -1 Φ * S (y -Φx) -fS = Φ * S Φ +, * T ê + λ -1 Φ * S Q T w -fS , (7.24)
where

Q T = Id -Φ T Φ + T = P Ker(Φ * T )
. Inserting (7.24) in ( 7.23), we obtain

J • f (λ -1 Φ * S (y -Φx) -fS ) 1 + ξΓ (x 0 -x) J • f (Φ * S Φ +, * T ê + λ -1 Φ * S Q T w -fS ).
Moreover, subadditivity yields

J • f (Φ * S Φ +, * T ê + λ -1 Φ * S Q T w -fS ) J • f (Φ * S Φ +, * T e -f S ) + J • f (Φ * S Φ +, * T ( ê -e)) + J • f (P S (f -f)) + J • f (λ -1 Φ * S Q T w). (7.25)
We now bound each term of (7.25). In the first term, one recognizes

J • f (Φ * S Φ +, * T e -f S ) IC(x 0 ). (7.26)
Appealing to the µ-stability property, we get

J • f (Φ * S Φ +, * T ( ê -e)) -Φ * S Φ +, * T Γ →J • f Γ (e -ê) µ -Φ * S Φ +, * T Γ →J • f Γ (x 0 -x). (7.27)
From τ-stability, we have

J • f (f S -fS ) τΓ (x 0 -x). (7.28)
Finally, we use a simple operator bound to get

J • f (λ -1 Φ * S Q T w) 1 λ | | |Φ * S Q T | | | ℓ 2 →J • f ε. (7.29)
Following the same steps as for the bound (7.20), except using ẽ = ê here, gives

Γ x 0 -x) (Φ * T Φ T ) -1 Γ →Γ {| | |Φ * T | | | ℓ 2 →Γ ε + λΓ ( ê)} . (7.30)
Plugging inequalities (7.26)-( 7.30) into (7.23) we get the upper-bound

J • f (Φ * S Φ +, * T ê + λ -1 Φ * S Q T w -fS ) (1 + ξΓ (x 0 -x)) IC(x 0 ) + Γ (x 0 -x) µ -Φ * S Φ +, * T Γ →J • f + τ + 1 λ | | |Φ * S Q T | | | ℓ 2 →J • f ε (1 + ξ(c 1 ε + λc 2 )) IC(x 0 ) + (c 1 ε + λc 2 ) μ + 1 λ c 4 ε < 1,
where we have introduced

μ = µc 3 + τ and α 1 = Γ ( ê) = Γ ( ẽ) = α 0 and c 1 = A, c 2 = α 1 (Φ * T Φ T ) -1 Γ →Γ , c 3 = -Φ * S Φ +, * T Γ →J • f , c 4 = Φ * S Q T ℓ 2 →J • f .
If is then sufficient that

(1 + ξ(c 1 ε + λc 2 )) IC(x 0 ) + (c 1 ε + λc 2 ) μ + 1 λ c 4 ε < 1. (7.31)
In particular, if

Cε λ

holds for some constant C > 0 to be fixed later, then inequality (7.31) is true if

P(λ) = aλ 2 + bλ + c > 0 (7.32)
where

       a = -ξ μ (c 1 /C + c 2 ) 2 b = -(c 1 /C + c 2 ) (ξIC(x 0 ) + ξc 4 /C + μ) c = 1 -IC(x 0 ) -c 4 /C
Let us set the value of C to

C = 2c 4 1 -IC(x 0 ) , which, for 0 IC(x 0 ) < 1, it ensures that c = 1-IC(x 0 )
2 is bounded and positive, and thus, the polynomial P has a negative and a positive root λ max equal to

λ max = b 2a ϕ -4 ac b 2
where

       a = -ξ μ((1 -IC(x 0 ))c 1 /(2c 4 ) + c 2 ) 2 b = -((1 -IC(x 0 ))c 1 /(2c 4 ) + c 2 ) ( μ + (1 + IC(x 0 ))ξ/2) c = (1 -IC(x 0 ))/2.
Hence,

λ max = μ + (1 + IC(x 0 ))ξ/2 ξ μ((1 -IC(x 0 ))c 1 /c 4 + 2c 2 ) ϕ 2ξ(1 -IC(x 0 )) μ ( μ + (1 + IC(x 0 ))ξ/2) 2 1 -IC(x 0 ) ξ H( μ/ξ), where ϕ(β) = 1 + β -1, and H(β) = β + 1/2 β(c 1 /c 4 + 2c 2 ) ϕ 2β (β + 1) 2 .
Consequently, we can conclude that the bounds

2c 4 1 -IC(x 0 ) ε λ 1 -IC(x 0 ) ξ H( μ/ξ) (C 2 )
imply condition (7.31), which in turn yields (7.22).

Step 3: (C 1 ) and (C 2 ) are in agreement. It remains now that show the compatibility of (C 1 ) and (C 2 ), i.e. to provide appropriate regimes of λ and ε such that both conditions hold simultaneously. We first observe that (C 1 ) and the left-hand-side of (C 2 ) both hold for λ fulfilling

λ C 0 ν where C 0 = A 2c 4 + B -1 1 -IC(x 0 ) 2c 4 A + B -1
. This updates (C 2 ) to the following ultimate range on λ

2c 4 1 -IC(x 0 ) ε λ min C 0 ν, 1 -IC(x 0 ) ξ H( μ/ξ) .
Now in order to have an admissible non-empty range for λ, the noise level ε must be upper-bounded as

ε 1 -IC(x 0 ) 2c 4 min C 0 ν, 1 -IC(x 0 ) ξ H( μ/ξ) .
Finally, the constants provided in the statement of the theorem (and subsequent discussion) are as follows

A T = 2c 4 , B T = C 0 , D T = c 3
, and

E T = c 1 /c 4 + 2c 2 ,
which completes the proof.

Relation to Previous Works

Special cases. Theorems 7.2 and 7.3 are generalizations of many previous works that have appeared in the literature. For the ℓ 1 norm, J = || • || 1 , to the best of our knowledge, this result was initially stated by [START_REF] Fuchs | On sparse representations in arbitrary redundant bases[END_REF]. In this setting, the result T ⋆ x = T x 0 corresponds to the correct identification of the support, i.e. supp(x ⋆ ) = supp(x 0 ). Moving to a setting where both Φ and w are random, the condition p F ∈ ri ∂J(x 0 ) implies model consistency (also known as sparsistency for ℓ 1 ), i.e. the probability that the support is correctly identified tends to 1 as the number of measurements grows large. Bach proves respectively in (Bach 2008a) and (Bach 2008b) Theorem 7.2 (in fact a variant since he considers randomized Φ and w) for ℓ 1ℓ 2 and nuclear norm gauges, in the special case where Φ has full rank (i.e. is injective). Our result thus shows that the same condition ensures rank consistency with the additional constraint that Ker(Φ) ∩ T = {0}. Theorem 7.3 for a ℓ 1 analysis prior was proved by Vaiter, Peyré, et al. (2013). Theorem 7.2 is extended in [START_REF] Duval | Exact Support Recovery for Sparse Spikes Deconvolution[END_REF] to the TV norm that endows the infinite dimensional Banach space of Radon measures, and where Φ has a finite-dimensional range. In this setting, they show that p F must be replaced by a different pre-certificate.

Compressed sensing. Condition Φ * p F ∈ ri ∂J(x 0 ) is often used when Φ is drawn from the Gaussian matrix ensemble to asses the performance of compressed sensing recovery with ℓ 1 norm, see [START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming (lasso)[END_REF][START_REF] Dossal | Sharp support recovery from noisy random measurements by ℓ 1 -minimization[END_REF]. It has been proved [START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming (lasso)[END_REF][START_REF] Dossal | Sharp support recovery from noisy random measurements by ℓ 1 -minimization[END_REF] for J = || • || 1 that if Φ is a random matrix drawn from the Gaussian ensemble, then for q > 2k log n, Φ * p F ∈ ri ∂J(x) with high probability on Φ for k = ||x 0 || 0 . One may have observed that the bound on q bears similarities with that of Chapter 6.3 except in the scaling in the log term, but induces stronger conclusion. It is also used to ensure ℓ 2 robustness of matrix completion in a noisy setting by [START_REF] Candès | Matrix completion with noise[END_REF], and our findings show that it also ensures rank consistency for matrix completion at high signal to noise levels. It generalizes a result proved for a family of decomposable norms (including in particular ℓ 1 -ℓ 2 norm and the nuclear norm) by [START_REF] Candès | Simple bounds for recovering low-complexity models[END_REF] when w = 0.

Stronger criteria for ℓ 1 . Many sufficient conditions can be formulated to ensure that p F is a non-degenerate certificate, and hence to guarantee the model stability. The strongest criterion to ensure a noise robustness for ℓ 1 regularization is the coherence, introduced by [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]. Finer criteria based on Babel functions have been proposed in [START_REF] Gribonval | Beyond sparsity : recovering structured representations by ℓ 1 -minimization and greedy algorithms[END_REF][START_REF] Borup | Beyond coherence : recovering structured time-frequency representations[END_REF]. The Exact Recovery Condition introduced by [START_REF] Tropp | Just relax: Convex programming methods for identifying sparse signals in noise[END_REF] is weaker than the coherence which in turns is greater that the weak-ERC [START_REF] Dossal | A necessary and sufficient condition for exact sparse recovery by minimization[END_REF]. More precisely, the coherence of a matrix with unit-norm is defined as

µ = max i =j | Φ i , Φ j |,
(x 0 ) = max j ∈I i∈I | Φ i , Φ J | 1 -max j∈I i =j∈I | Φ i , Φ J | .
These quantity obey the following inequality:

IC(x 0 ) ERC(x 0 ) wERC(x 0 ) coh(x 0 ).
In particular, if any of these quantity is less than 1, then p F is a non-degenerate certificate. T hi s chapter is concerned with differentiability of an optimal map of solution to (P F y,λ ). Moreover, we prove that the prediction map is well-defined outside the transition space and gives its derivative. The core of our proof strategy relies on the identification of a certain linear subspace T = T x ⋆ (y) associated to a particular minimizer x ⋆ (y) of (P F y ). We exhibit explicitly a certain set of observations, denoted H (see Definition 8.2), outside which the initial non-smooth optimization (P F y ) boils down locally to a smooth optimization constrained by T . This part of the proof strategy is in close agreement with the one developed in [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF] for the sensitivity analysis of partly smooth functions. The robustness analysis of Chapter 7 relies on the manifold stability when λ = 0. In contrast, we provide in this chapter a sensitivity analysis when λ > 0. Even if we state our result only w.r.t to small variations of y, our result can be extended to analyze the sensibility with respect to other variable parameterizing F, which could be useful for homotopy-like results. This sensivity analysis is central to construct an unbiased estimator of the quadratic risk. We suppose here that J is a partly smooth gauge with linear manifold, i.e. such that M x = T x and J is 1-homogeneous. We conjecture that this statement remains true for any convex partly smooth function, though this has not been formally proved yet. The technical obstacles faced by this generalization will be discussed in Chapter 9.

Part

Main Assumptions

This section details our assumptions on both the data fidelity term F and the regularizer J. We also introduce the restriction and second order derivative of the regularizer J.

Assumptions on the Regularizer

We assume in this chapter that J is a partly smooth gauge with linear manifold, see Definitions 4.1 and 4.2. More precisely, we need the following assumption.

∀ T ∈ T, J ∈ C 2 ( T ) . (C sm ) ∀ x ∈ R n , ∃ν > 0, ∀ x ′ ∈ T x , ||x ′ -x|| < τ ⇒ T x = T x ′ . (C reg )
The set T is finite. (C T )

J is positively homogeneous. (C hom )

We recall that the set T is defined as

T = {T x | x ∈ R n } .
Some remarks are in order. Assumption (C reg ) amounts to saying that there exists a neighbourhood of x on T x on which this subspace model is constant. This condition is a part of the assumptions defining the class of partly smooth function with linear manifold introduced in Definition 4.1. Assumption (C T ) holds in many important cases, including the Lasso (ℓ 1 -norm) and group Lasso (ℓ 1 -ℓ 2 ) penalties, the ℓ ∞ -norm, as well as their analysis-type counterparts.

Assumption on the Data Fidelity

In all the following, we consider a variational regularized problem of the form of (P F y,λ ), i.e. of the form1

x ⋆ (y) ∈ Argmin x∈R n F(x, y) + J(x).

(P F y )

The fidelity term F is of the following form

F(x, y) = F 0 (Φx, y) (8.1)
where F 0 (•, y) is a general loss function assumed to be a proper, convex and sufficiently smooth function of its first argument ∀y. We assume that the fidelity term enjoys the following properties.

∀ (y, x) ∈ R q × R n , F(•, y) ∈ C 2 (R n ) and ∇ 1 F(x, •) ∈ C 1 (R q ). (C F )
Generalized linear models in the exponential family falls into the class of losses we consider. Indeed, taking the negative log-likelihood corresponding to (9.2) gives2

F 0 (µ, y) = 1 λ q i=1 ϕ i (µ i ) -y, µ . (8.2)
It is well-known that if the exponential family is regular, then ϕ i is proper, infinitely differentiable, its hessian is definite positive, and thus it is strictly convex [START_REF] Brown | Fundamentals of Statistical Exponential Families with Applications in Statistical Decision Theory[END_REF]. Therefore, F 0 (•, y) shares exactly the same properties. We recover the squared loss F 0 (µ, y) = 1 2λ ||y -µ|| 2 for the standard linear models (Gaussian case), and the logistic loss F 0 (µ, y) = q i=1 log (1 + exp(µ i ))y, µ for logistic regression (Bernoulli case). GLM estimators with losses (8.2) and ℓ 1 or ℓ 1ℓ 2 (group) penalties have been previously considered and some of their properties studied including in [START_REF] Bunea | Honest variable selection in linear and logistic regression models via ℓ 1 and ℓ 1 + ℓ 2 penalization[END_REF][START_REF] Van De Geer | High-dimensional generalized linear models and the lasso[END_REF][START_REF] Meier | The group lasso for logistic regression[END_REF][START_REF] Bach | Consistency of the group Lasso and multiple kernel learning[END_REF][START_REF] Kakade | Learning exponential families in high-dimensions: Strong convexity and sparsity[END_REF]; see also (B ühlmann et al. 2011, Chapter 3, 4 and 6).

where, for I = supp B (x),

δ x : v ∈ R |I| → (v b /||x b ||) b⊂I ∈ R |I| and P x ⊥ : v ∈ R |I| → (P x ⊥ b v b ) b⊂I ∈ R |I| , where P x ⊥ b v b = v b - x b , v b ||x b || 2 x b
is the orthogonal projector on x ⊥ b .

Local

Behavior of a Solution Mapping

Restricted Injectivity

In this section, we aim at computing the derivative of the map y → x ⋆ (y) whenever this is possible. The following condition plays a pivotal role in this analysis.

Defi ni ti on 8 .1 -R estri cted Inj ecti vi ty A vector x ∈ R n with T = T x is said to satisfy the restricted injectivity condition if, and only if,

T ∩ Ker(D 2 1 F T (x, y)) ∩ Ker(D 2 J T (x)) = {0}. (C x,y )
Lasso For the Lasso problem, i.e. J = || • || 1 and F 0 is the squared loss, condition (C x,y ) reads Ker(Φ I ) = {0}, where I is the support of the vector x. This condition is already known in the literature, see for instance [START_REF] Dossal | The degrees of freedom of the Lasso for general design matrix[END_REF] in the context of DOF estimation.

Group Lasso For the group Lasso, i.e. J = || • || 1,2 and F 0 is the squared loss, condition (C x,y ) amounts to assuming that the collection of vectors (Φ b x b ) b⊂I is linearly independent, where I = supp B (x). This condition appears in [START_REF] Liu | Estimation Consistency of the Group Lasso and its Applications[END_REF]) to establish ℓ 2 -consistency of the group Lasso. It goes without saying that condition (C x,y ) is much weaker than imposing that Φ I is full column rank, which is standard when analyzing the Lasso.

Under this condition, the derivative of the objective function is invertible on T .

L emma 8 .1 Let x ∈ R n , and T = T x . Assume that (C x,y ) holds. Then the linear operator D 2 1 F T (x, y) + D 2 J T (x) : T → T is invertible on T .

p ro o f Since F(•, y) and J are convex and C 2 (T ) by assumptions (C F ) and (C sm ), the (restricted) hessians D 2 1 F T (x, y) and D 2 J T (x) are symmetric semidefinite positive on T . To ensure invertibility of their sum on T , it is necessary and sufficient that their kernels have a trivial intersection, which is exactly what assumption C x,y states.

Transition Space

Let us now turn to the sensitivity of a minimizer x ⋆ (y) of (P F y ) to perturbations of y. Because of non-smoothness of the regularizer J, it is a well-known fact in sensitivity analysis that one cannot hope for a global claim, i.e. an everywhere smooth mapping 3 y → x ⋆ (y). Rather, the sensitivity behaviour is local. This is why the reason we need to introduce the following transition space H, which will be shown to contain points of non-smoothness of x ⋆ (y).

D efi ni ti on 8 . 2 The transition space H is defined as

H = T ∈T H T , where H T = bd(Π q+n,q (A T )),
where we have denoted Π q+n,q : R q × T -→ R q (y, x T ) -→ y 3. To be understood here as a set-valued mapping.

the canonical projection on the first q coordinates, bd C is the boundary of the set C, and

A T = (y, x T ) ∈ R q × T | -∇ 1 F(x T , y) ∈ rbd ∂J(x T ) .
Here, rbd ∂J(x T ) is the relative boundary of ∂J(x T ) defined as its boundary in the topology of its affine hull.

In the particular case where F is the square loss, J = || • || 1 (synthesis sparsity) and J(x) = ||D * x|| 1 (analysis sparsity), the same transition set is introduced in [START_REF] Dossal | The degrees of freedom of the Lasso for general design matrix[END_REF] and [START_REF] Vaiter | Local behavior of sparse analysis regularization: Applications to risk estimation[END_REF]. In these specific cases, since J is a polyhedral gauge, H is a union of affine hyperplane. The geometry of this set can be significantly more complex for other gauges. For instance, for J = || • || 1,2 , it can be shown to be a semi-algebraic set (union of algebraic hyper-surfaces).

Main Result

We are now equipped to state our main sensitivity analysis result. Theorem 8 . 1 Let y ∈ H, and x ⋆ a solution of P λ (y) such that (C x ⋆ ,y ) holds. Then, there exists an open neighborhood V ⊂ R q of y, and a mapping x : V → T such that (i) For all ȳ ∈ V, x( ȳ) is a solution of (P λ( ȳ)), and x(y) = x ⋆ .

(ii) the mapping x is C 1 (V) and for every ȳ ∈ V,

∂ 1 x( ȳ) = -(D 2 1 F T (x ⋆ , ȳ) + D 2 J T (x ⋆ )) -1 • P T •D 2 12 F(x ⋆ , ȳ), (8.3)
where T = T x ⋆ .

One now may wonder whether condition (C x ⋆ ,y ) is restrictive, and in particular, whether there exists always a solution x ⋆ such that it holds. In Section 8.3,

we give an affirmative answer with the proviso that the loss F 0 is strictly convex.

The above result can be extended to the case where the data fidelity is of the form F(x, θ) for some parameter θ, with no particular role of y here. One may think for instance to consider θ = (y, λ). The variations with respect to λ are important for developing homotopy-like algorithm. [START_REF] Vaiter | Local behavior of sparse analysis regularization: Applications to risk estimation[END_REF] proved that if J is polyhedral, then the path is locally affine.

p ro o f Let y ∈ H and x ⋆ be a solution of (P F y ) such that (C x ⋆ ,y ) holds. We denote T x ⋆ = T = S ⊥ .

We define the following mapping

Γ : (x T , y) ∈ T × R q → ∇ 1 F(x T , y) T + e x T .
Observe that owing to Proposition 3.1(iv), the first equation of the first-order condition is equivalent to Γ (x ⋆ T , y) = 0.

Note that any x T ∈ T such that Γ (x T , y) = 0 is a solution of the constrained problem min α∈T F(α, y) + J(α) .

(P(y) T )

It comes from the fact that Γ (x T , y) = 0 is the first-order minimality condition over the subspace T .

We split the proof in three steps. We first show that there exists a mapping ȳ → x( ȳ) ∈ T and an open neighborhood V of y such that every element ȳ of V satisfies Γ (x( ȳ) T , ȳ) = 0 and x( ȳ) S = 0. Then, we prove that x( ȳ) is a solution of (P F ȳ) for ȳ ∈ V. Finally, we obtain (8.3) from the implicit function theorem.

Step 1: construction of x( ȳ). The Jacobian of Γ with respect to the first variable reads

D 1 Γ (x ⋆ T , ȳ) = D 2 1 F T (x ⋆ T , ȳ) T + D 1 e x ⋆ T ,
where D 1 denotes the derivative with respect to the first variable. Moreover, since

x ⋆ ∈ T , Assumption (C sm ) yields D 1 e x ⋆ T = D 2 J T (x ⋆ T )
. Thus, we get

D 1 Γ (x ⋆ T , ȳ) = D 2 1 F T (x ⋆ T , ȳ) + D 2 J T (x ⋆ T ).
The linear operator mapping D 1 Γ (x ⋆ T , y) is invertible on T according to Lemma 8.1. Hence, using the implicit function theorem (Theorem 2.1) restricted to T , there exists a neighborhood V of y such that we can define a mapping xT : V → T which is C 1 ( V), and satisfies for ȳ ∈ V Γ (x T ( ȳ), ȳ) = 0 and xT (y) = x ⋆ T .

We then extend x( ȳ) on S as xS ( ȳ) = 0 which defines a continuous mapping

x : V → T ⊂ R n .
Step 2: checking the first-order minimality condition on S. We now have to check the first order conditions on S, i.e. to check that -∇ 1 F(x( ȳ), ȳ) ∈ ∂J(x( ȳ)).

We distinguish two cases.

(i) Assume that -∇ 1 F(x ⋆ , y) ∈ ri ∂J(x ⋆ ): we show that for a sufficiently small neighbourhood of y, we also have -∇ 1 F(x( ȳ), ȳ) ∈ ri ∂J(x( ȳ)). First, since x is continuous on T , for any ε > 0, there exists a neighborhood

V ⊂ V of y such that ||x( ȳ) -x ⋆ || ε ∀ ȳ ∈ V .
By virtue of Assumption (C reg ), one can then choose ε sufficiently small to conclude that S x( ȳ) = S for any ȳ ∈ V.

Suppose that there is a sequence (y ℓ ) ℓ approaching y such that

-∇ 1 F(x(y ℓ ), y ℓ ) / ∈ ri ∂J(x(y ℓ ))
for all ℓ. This can be equivalently written, owing to Lemma 5.3, as

∃u ℓ ∈ S x(y ℓ ) , ∀v ∈ ∂J(x(y ℓ )) u ℓ , v + ∇ 1 F(x ⋆ , y) 0, ∀ℓ , or ∃u ℓ ∈ S x(y ℓ ) , sup u ℓ , ∂J(x(y ℓ )) + ∇ 1 F(x(y ℓ ), y ℓ ) 0, ∀ℓ . (8.4)
Recall that the sequence u ℓ can be taken on the unit sphere, and therefore has a non-zero cluster point, say u, which belongs to S as S x(y ℓ ) converges to S. We now claim that sup u, ∂J(x ⋆ ) + ∇ 1 F(x ⋆ , y) 0 .

Consider any η ∈ ∂J(x ⋆ ). Since x(y ℓ ) converges to x ⋆ in T , we have from the argument above that T x(y ℓ ) = T for ℓ sufficiently large. This together with Assumption (C sm ), which means that ∂J(β) is continuous on T , allow to deduce that ∂J(x(y ℓ )) converges to ∂J(x ⋆ ). Thus, there exists a sequence η ℓ ∈ ∂J(x(y ℓ )) converging to η. Now, continuity of the mapping

y ℓ ∈ V → ∇ 1 F(x(y ℓ ), y ℓ ) ∈ R n
(since x and ∇ 1 F are both continuous on T and R n × R q ) yields also that ∇ 1 F(x(y ℓ ), y ℓ ) converges to ∇ 1 F(x ⋆ , y). Since

u ℓ , η ℓ + ∇ 1 F(x(y ℓ ), y ℓ ) sup u ℓ , ∂J(x(y ℓ )) + ∇ 1 F(x(y ℓ ), y ℓ ) 0 , ∀ℓ we get that u, η + ∇ 1 F(x ⋆ , y) 0 .
The latter inequality holds for any η ∈ ∂J(x ⋆ ), which, in view of Lemma 5.3, means that -∇ 1 F(x ⋆ , y) / ∈ ri ∂J(x ⋆ ). But this contradicts our initial assumption.

(ii) We now turn to the case where -∇ 1 F(x ⋆ , y) ∈ rbound ∂J(x ⋆ ). Observe that (y, x ⋆ ) ∈ A T . In particular y ∈ Π q+n,q (A T ). Since by assumption y ∈ H, one has y ∈ bd(Π q+n,q (A T )). Hence, there exists an open ball B(y, ε) for some ε > 0 such that B(y, ε) ⊂ Π q+n,q (A T ). Thus for every ȳ ∈ B(y, ε), there exists x ∈ T such that

-∇ 1 F(x, ȳ) ∈ rbound ∂J(x).
Applying Lemma 2.2 with f = F(•, y) and g = J + ι T , where ι T is the indicator function of T , we deduce that all solutions of (P( ȳ) T ) share the same gradient. Thus, we also have that ∇ 1 F(x, ȳ) = ∇ 1 F(x( ȳ), ȳ). This implies in particular that e(x) = e x( ȳ) . Since T ⊂ T is an open set for the topology relative to T and x(y) = x ⋆ ∈ T , for ȳ sufficiently close to y, Assumption (C reg ) allows to deduce that x( ȳ) ∈ T =⇒ T x( ȳ) = T .

Thus, we have T x( ȳ) = T x, hence S x( ȳ) = S x. Combining this with Proposition 3.1 and the claim that both vectors have the same image under e • , yields that they also share the same affine hull, i.e. S x( ȳ) = S x. In turn, this implies the equality of the subdifferential by virtue of Proposition 3.6, i.e. ∂J(x( ȳ)) = ∂J(x). We conclude that

∀ ȳ ∈ B(y, ε), -∇ 1 F(x( ȳ), ȳ) ∈ rbound ∂J(x( ȳ)).
Moreover, by definition of the mapping xT , one has for all ȳ ∈ V ∩ V

∇ 1 F(x T ( ȳ), ȳ) T + e xT ( ȳ) = 0 .
According to Lemma 3.9, the vector x( ȳ) is a solution of P F ȳ.

Step 3: computing the differential. By virtue of step 1., we are in position to use the implicit function theorem, and we get the Jacobian of xT as

Dx T ( ȳ) = -D 1 Γ (x T ( ȳ), ȳ) -1 D 2 Γ (x T ( ȳ), ȳ)
where

D 2 Γ (x T , ȳ) = P T •D 2 12 F(x T , ȳ),
which leads us to (8.3).

Local Behavior of the Prediction Mapping

In this section, we aim to provide a closed-form expression of the local variations of µ(y) with respect to the observation y. Our result is Theorem 8.2. We assume in this section that F takes the form (8.1) and that

∀ y ∈ R q , F 0 (•, y) is strictly convex. (C strict )
Lemma 8 . 3 There exists a solution x ⋆ of (P F y ) such that (C x ⋆ ,y ) holds.

p ro o f Let x ⋆ a solution of (P F y ) such that (C x ⋆ ,y ) does not hold. Consider the associated subspace T = T x ⋆ . Thus, for any h ∈ Ker(Φ) ∩ T ∩ Ker(D 2 J T (x ⋆ )) \ {0}, we have Φ T h = 0 and D 2 J T (x ⋆ ) h = 0. Let v t = x ⋆ + th, ∀ t > 0. By Proposition 3.1, v t ∈ T . Moreover, Φ T v t = Φ T x ⋆ , and thus F(Φ T v t , y) = F(Φ T x ⋆ , y).

Using convexity of J and h ∈ T , we have ∀ η ∈ ∂J(v t )

J(v t ) J(x ⋆ ) + t η, h = J(x ⋆ ) + t η T , h .
Since J obeys Assumption (C reg ) and v t ∈ T , for t sufficiently small, we have T v t = T , whence we get J(v t ) J(x ⋆ ) + t e(v t ), h .

where we used Proposition 3.1. From Assumption (C sm ), Taylor expansion gives

e(v t ) = e(x ⋆ ) + tD 2 J T (x ⋆ )h + tε(th)||h|| = e(x ⋆ ) + tε(th)||h|| ,
with lim t→0 ε(th) = 0. Altogether, we arrive at J(v t ) J(x ⋆ ) + t e(x ⋆ ), h + t||ε(th)||||h|| 2 .

Suppose now that there exists no x ⋆ such that (C x ⋆ ,y ) holds. Then, we can always find a solution x ⋆ such that4 e(x ⋆ ) / ∈ Ker(Φ) ∩ T ∩ Ker(D 2 J T (x ⋆ )) ⊥ , and therefore there is some h ∈ Ker(Φ) ∩ T ∩ Ker(D 2 J T (x ⋆ )) \ {0} such that e(x ⋆ ), h < 0 and thus

F(Φ T v t , y) + J(v t ) < F(Φ T x ⋆ , y) + J(x ⋆ ) ,
for t sufficiently small, leading to a contradiction.

when the auxiliary parameters (e.g. y here) undergo a perturbation. In its simplest form, sensitivity analysis of first-order optimality conditions, in the parametric form of the Fermat rule, relies on the celebrated implicit function theorem.

For the Lasso problem, the above differential formula (8.5) implies that

div( µ)(y) = | supp(x ⋆ )|,
where x ⋆ is any solution of (P y,λ ) such that (C x ⋆ ,y ) holds, i.e. Φ supp(x ⋆ ) has full rank. This result is proved in [START_REF] Dossal | The degrees of freedom of the Lasso for general design matrix[END_REF], see also [START_REF] Tibshirani | The solution path of the generalized lasso[END_REF]. The analysis sparsity case was investigated in [START_REF] Vaiter | Local behavior of sparse analysis regularization: Applications to risk estimation[END_REF] and [START_REF] Tibshirani | The solution path of the generalized lasso[END_REF]. In this case, one has

J = ||D * • || 1 and div( µ)(y) = dim Ker D * Λ , Λ = supp(D * x ⋆ ) c ,
where x ⋆ is such that (C x ⋆ ,y ) holds.

The originality of our contribution in this direction is the following (i) We formulate the set H of non-smoothness points, which is crucial for the application to risk estimation exposed in the next chapter.

(ii) We give an explicit formula of the differential of the prediction.

(iii) Our sensitivity result deals with a set-valued mapping (even if its image by Φ is single-valued).

200

T hi s chapter is concerned with unbiased risk estimation for the variational problem (P F y ). More precisely, given an estimator family x ⋆ λ (y) of x 0 defined as solution of this problem, we aim to find the best parameter λ. Using tools from o-minimal geometry, we prove that the divergence formula (8.5) is valid Lebesgue-a.e. In turn, this allows us to define an unbiased estimate of the DOF and of the prediction risk (Theorem 9.1 and Theorem 9.2) for model (9.1) under two scenarios: (i) Lipschitz continuous non-linearity h and an additive i.i.d. Gaussian noise; (ii) Generalized Linear Models (GLMs) with a continuous exponential family. Our results encompass some previous ones in the literature as special cases. We consider a more general model than a simple linear regression (1. 3)

E(Y|Φ) = h(Φx 0 ), (9.1)
where Y ∈ R q is the response vector, x 0 ∈ R n is the unknown vector, Φ ∈ R q×n is the fixed design matrix whose columns are the n covariate vectors, and the expectation is taken with respect to some σ-finite measure. h is a known smooth function R q → R q . The goal is to design an estimator of x 0 and to study its properties. In the sequel, we do not make any specific assumption on the number of observations q with respect to the number of predictors n. Recall that when q < n, (9.1) is underdetermined, whereas when q n and all the columns of Φ are linearly independent, it is overdetermined. Many examples fall within the scope of model ( 9.1). We here review two of them.

Generalized Linear Models One naturally thinks of generalized linear models (GLMs) introduced by [START_REF] Nelder | Generalized linear models[END_REF] which assume that conditionally on Φ, Y i are independent with distribution that belongs to a given (oneparameter) standard exponential family. Recall that the random variable Z ∈ R has a distribution in this family if its distribution admits a density with respect to some reference σ-finite measure on R of the form

p(z; θ) = B(z) exp(zθ -ϕ(θ)), θ ∈ Θ ⊆ R ,
where Θ is the natural parameter space and θ is the canonical parameter. For model (9.1), the distribution of Y belongs to the n-parameter exponential Suppose that h in (9.1) is the identity and that the observations Y ∼ N(µ 0 , σ 2 Id n ).

Following [START_REF] Efron | The estimation of prediction error: Covariance penalties and cross-validation[END_REF], the DOF is defined as

df = q i=1 cov(Y i , µ i (Y)) σ 2
.

The well-known Stein's lemma [START_REF] Stein | Estimation of the Mean of a Multivariate Normal Distribution[END_REF]

E ∂ µ i ∂y i (Y) < ∞, ∀i ,
then its divergence is an unbiased estimator of its DOF, i.e.

df(Y) = div( µ)(Y) = tr(D µ(Y)) and E( df) = df ,
where D µ is the Jacobian of y → µ(y). In turn, this allows to get an unbiased estimator of the prediction risk E(|| µ(Y)µ 0 || 2 ) through the SURE (Stein Unbiased Risk Estimate, Stein 1981).

Extensions of the SURE to independent variables from an exponential family are considered in [START_REF] Hudson | A natural identity for exponential families with applications in multiparameter estimation[END_REF] for the continuous case, and (Hwang 1982) in the discrete case. [START_REF] Eldar | Generalized SURE for Exponential Families: Applications to Regularization[END_REF] generalizes the SURE principle to continuous multivariate exponential families.

GSURE for Gaussian Observations

Definition

The Stein's lemma is the foundation of risk estimation using the SURE.

Lemma 9 . 1 -S tei n ' s L emma. Let Y = Φx 0 + W with W ∼ N(0, σ 2 Id q ). Assume that g : y → g(y) is weakly differentiable (and a fortiori a singlevalued mapping), then

E W W, g(Y) = σ 2 E W tr ∂g(Y) ∂Y .
p ro o f This result is proved in [START_REF] Stein | Estimation of the Mean of a Multivariate Normal Distribution[END_REF].

We here develop an extended version of GSURE, defined by [START_REF] Eldar | Generalized SURE for Exponential Families: Applications to Regularization[END_REF], that unbiasedly estimates the risk of reconstructing Aµ 0 where A ∈ R m×q is an arbitrary matrix. This allows us to cover in a unified framework unbiased estimation of several classical risks including the prediction risk (with A = Id), the projection risk when Φ is rank deficient (with A = Φ * (ΦΦ * ) + ), and the estimation risk when Φ has full rank (with A = Φ + = (Φ * Φ) -1 Φ * ). A quantity that will enter into play in the risk of estimating Aµ 0 is the degrees of freedom defined as

df A = q i=1 cov w ((Ay) i , (A µ(y)) i ) σ 2
.

Defi ni ti on 9 .1 Let A ∈ R m×q . We define the Generalized Stein Unbiased Risk Estimate (GSURE) associated to A as

Unbiasedness

The next result shows that GSURE A is an unbiased estimator of an appropriate ℓ 2 risk, and df A (y) is an unbiased estimator of df A Theorem 9 .1 Let A ∈ R m×q . Suppose that y → µ(y) is weakly differentiable, so that its divergence is well-defined in the weak sense. If Y = Φx 0 + W with W ∼ N(0, σ 2 Id q ), then

E W GSURE A (Y) = E W ||Aµ 0 -A µ(Y)|| 2 2 and E W df A (Y) = df A .
p ro o f Since y → µ(y) = Φx ⋆ (y) is weakly differentiable, so is A * A µ(y) and we have

∂A * A µ(y) ∂y = A * A ∂ µ(y) ∂y .
Then, using Lemma 9.1, we get

E W w, A * A µ(Y) = σ 2 E W tr A * A ∂ µ(Y) ∂y = σ 2 E W df A (Y) .
Using the decomposition AY = AΦx 0 + AW, we obtain

E W ||AY -A µ(Y)|| 2 2 = E W ||AΦx 0 + AW|| 2 2 -2E W AΦx 0 + AW, A µ(Y) + E W ||A µ(Y)|| 2 2 = E W ||AΦx 0 || 2 2 + σ 2 tr(A * A) -2E W AΦx 0 , A µ(Y) -2E W W, A * A µ(Y) + E W ||A µ(Y)|| 2 2 = E W ||AΦx 0 -A µ(Y)|| 2 2 + σ 2 tr(A * A) -2σ 2 E W df A (Y) . Moreover, i cov W ((AY) i , (A µ(Y)) i ) = E W AW, A µ(Y) , which shows that df A (Y)
is indeed an unbiased estimator of df A .

Theorem 9.1 can be straightforwardly adapted to deal with any white Gaussian noise with a non-singular covariance matrix Σ. It is sufficient to consider the change of variable y → Σ -1/2 y and Φ → Σ -1/2 Φ. This is similar to the work of [START_REF] Eldar | Generalized SURE for Exponential Families: Applications to Regularization[END_REF].

It is also worth noting that if x ⋆ (y) never lies in Ker(Φ), then Risk Π (x 0 ) coincides with the estimation risk up to the additive constant ||(Id -Π)x 0 || 2 2 .

• A = (Φ * Φ) -1 Φ * : in this case Φ has full rank, and the mapping y → x ⋆ (y) is single-valued and weakly differentiable. The maximum likelihood estimator is now x ML (y) = (Φ * Φ) -1 Φ * y , and GSURE (Φ * Φ) -1 Φ * takes the form

GSURE Id (y) = ||x ML (y) -x ⋆ (y)|| 2 2 -σ 2 tr (Φ * Φ) -1 +2σ 2 tr Φ(Φ * Φ) -1 ∂x ⋆ (y) ∂y .
This is an unbiased estimator of the estimation risk given by Risk

Id (x 0 ) = E W ||x ⋆ (Y) -x 0 || 2 2 .

Unbiased Risk Estimation

Throughout this section, we use the same symbols to denote weak derivatives (whenever they exist) as for derivatives. Rigorously speaking, the identities have to be understood to hold Lebesgue-a.e. [START_REF] Evans | Measure theory and fine properties of functions[END_REF].

So far, we have shown that ouside the transition space H, the mapping µ(y) enjoys (locally) nice smoothness properties, which in turn gives closed-form formula of its divergence. To establish that such a formula holds Lebesgue a.e., a key argument that we need to show is that H is of negligible Lebesgue measure. This is where o-minimal geometry enters the picture. In turn, for Y drawn from some appropriate probability measure with density with respect to the Lebesgue measure, this allows us to establish unbiasedness of quadratic risk estimators.

Our o-minimality assumptions requires the existence of an o-minimal structure O, see Definition 2.32, such that the functionals F and J are definable in O.

(C O )
the first-order formula in terms of set operations, and using axioms 1-4 of definability in an o-minimal structure.

• Let D : R n ⇒ R n the set-valued mapping whose graph is

G(D) = {(β, η) | η ∈ ri ∂J(β)} .
From Lemma 2.13, G(D) is definable. Since the graph ∂J is closed (Hiriart-Urruty et al. 2001), and definable (Lemma 2.13), the set

{(β, η) | η ∈ rbound ∂J(β)} = G(∂J) \ G(D) ,
is also definable by axiom 1. This entails that A T is also a definable subset of R q × T since

A T = (R q × T × R n ) ∩ {(y, x, η) | ∃η, η = -∇ 1 F(x T , y)} ∩ {(β, η) | η ∈ rbound ∂J(β)} .
• By axiom 4, the canonical projection Π q+n,q (A T ) is definable, and its boundary H T = bd(Π q+n,q (A T )) is also definable by [START_REF] Coste | An Introduction to O-minimal Geometry[END_REF], Proposition 1.12) with a strictly smaller dimension than Π q+n,q (A T ) (Coste 1999, Theorem 3.22).

• We recall now from (Coste 1999, Theorem 2.10) that any definable subset A ⊂ R q in O can be decomposed (stratified) in a disjoint finite union of q subsets C i , definable in O, called cells. The dimension of A is (Coste 1999, Proposition 3.17( 4)) d = max i∈{1,...,q}

d i q , where d i = dim(C i ). Altogether we get that dim H T = dim bd(Π q+n,q (A T )) < dim Π q+n,q (A T ) = d q
whence we deduce that H is of zero measure with respect to the Lebesgue measure on R q since the union is taken over the finite set T by (C T ).

The Prediction is Lipschitz Continuous

P roposi ti on 9 .2 Suppose that conditions (C τ ) and (C L ) hold. Then, µ is Lipschitz continuous.

p ro o f F 0 (•, y) is strongly convex with modulus τ if, and only if,

F 0 (µ, y) = G(µ, y) + τ 2 ||µ|| 2
where G(•, y) is convex and satisfies (C F ), and in particular its domain in µ is full-dimensional. Thus, (P F y ) amounts to solving

min x∈R n τ 2 ||Φx|| 2 + G(Φx, y) + λJ(x).
It can be recasted as a constrained optimization problem

min µ∈R q ,x∈R n τ 2 ||µ|| 2 + G(µ, y) + λJ(x) s.t. µ = Φx.
Introducing the image (ΦJ) of J under the linear mapping Φ, it is equivalent to

min µ∈R q τ 2 ||µ|| 2 + G(µ, y) + λ(ΦJ)(µ) , (9.3)
where (ΦJ)(µ) = min

{x∈R n | µ=Φx} λJ(x)
. This is a proper closed convex function, which is finite on Im(Φ). The minimization problem amounts to computing the proximal point at 0 of G(•, y) + λ(ΦJ), which is a proper closed and convex function. Thus this point exists and is unique.

Furthermore, by assumption on F 0 , the difference function

F 0 (•, y 1 ) -F 0 (•, y 2 ) = G(•, y 1 ) -G(•, y 2 ) is Lipschitz continuous on R q with Lipschitz constant L||y 1 -y 2 ||.
It then follows from (Bonnans et al. 2000, Proposition 4.32) that µ(•) is Lipschitz continuous with constant 2L/τ.

A Closed Form Expression of the DOF

We now arrive at our main contribution. The following theorem prove that the quantity ∆(y) defined in (8.5)

∂( µ) i ∂y i (Y) < +∞ .
This formula is valid everywhere except on the set H which is of Lebesgue measure zero as shown in Proposition 9.1. We conclude by invoking (i) and Stein's lemma [START_REF] Stein | Estimation of the Mean of a Multivariate Normal Distribution[END_REF] to establish unbiasedness of the estimator df of the DOF. p ro o f By the chain rule (Evans et al. 1992, Remark, Section 4.2.2), the weak derivative of A • µ(•) at y is precisely

A (Y) =||AY -A µ(Y)|| 2 + 2σ 2 df A (Y) -σ 2 tr(AA * ) ( 9
D(A • µ)(y)) = A ( µ(y)) ∆(y) a.e.
This formula is valid everywhere except on the set H which is of Lebesgue measure zero as shown in Proposition 9.1. We conclude by invoking Proposition 9.2 to establish unbiasedness of the estimator df A (Y) and using Theorem 9.1.

GLM with the continuous exponential family. Assume that the observation model ( 9.1) corresponds to the GLM with a distribution which belongs to a continuous standard exponential family as parameterized in (9. Though SURE(Y) depends on µ 0 , which is obviously unknown, it is only through an additive constant.

A Simple Example: DOF of Block Thresholding

Consider that Φ = Id, J = || • || 1,2 and F 0 = 1 λ || • -y|| 2 is the square loss. In this setting, it is known that (P F y ) has a unique solution given by the block thresholding operator, i.e. for every b ∈ B,

µ(y) b = x ⋆ b =      0 if ||y b || λ 1 - λ y b y b otherwise.
The estimator of the degrees of freedom reads then In the case of standard Lasso (i.e. ℓ 1 penalty) with Y ∼ N(Φx 0 , σ 2 Id n ) and Φ of full column rank, [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF] showed that the number of nonzero coefficients is an unbiased estimate for the DOF. Their work was generalized in [START_REF] Dossal | The degrees of freedom of the Lasso for general design matrix[END_REF] to any arbitrary design matrix Φ. Under the same Gaussian linear regression model, unbiased estimators of the DOF for the Lasso with ℓ 1 -analysis penalty, were given independently in [START_REF] Tibshirani | The solution path of the generalized lasso[END_REF][START_REF] Vaiter | Local behavior of sparse analysis regularization: Applications to risk estimation[END_REF].

A formula of an estimate of the DOF for the group Lasso when the design is orthogonal within each group was conjectured in [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF]. Kato 2009 studied the DOF of a general shrinkage estimator where the regression coefficients are constrained to a closed convex set C. His work extends that of [START_REF] Meyer | On the degrees of freedom in shaperestricted regression[END_REF] which treats the case where C is a convex polyhedral cone. When Φ is full column rank, Kato (2009) derived a divergence formula under a smoothness condition on the boundary of C, from which an unbiased estimator of the degrees of freedom was obtained. When specializing to the constrained version of the group Lasso, the author provided an unbiased estimate of the corresponding DOF under the same group-wise orthogonality assumption on Φ as [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF]). An estimate of the DOF for the group Lasso was also given by Solo et al. ( 2010) using heuristic derivations that are valid only when Φ is full column rank, though its unbiasedness is not proved.

Generalized Stein Unbiased Risk Estimator

In [START_REF] Eldar | Generalized SURE for Exponential Families: Applications to Regularization[END_REF], the author derived expressions equivalent to GSURE Π and GSURE Id up to a constant which does not depend on the estimator. However, her expressions were developed separately, whereas we have shown that these GSURE estimates originate from a general result stated in Theorem 9.1. Another distinction between our work and [START_REF] Eldar | Generalized SURE for Exponential Families: Applications to Regularization[END_REF]) lies in the assumptions imposed. [START_REF] Eldar | Generalized SURE for Exponential Families: Applications to Regularization[END_REF] supposes x ⋆ (y) to be a weakly differentiable function of Φ * y/σ2 . In contrast, we just require that the prediction y → µ(y) (a singlevalued map) is weakly differentiable, as classically assumed in the SURE theory.

Indeed, let u = Φ * y/σ 2 , and define x ⋆ (y) = z ⋆ θ (u). Assume that u → z ⋆ θ (u) is weakly differentiable (and a fortiori a single-valued mapping).

When Φ is rank deficient, [START_REF] Eldar | Generalized SURE for Exponential Families: Applications to Regularization[END_REF] proves unbiasedness of the following estimator of the projection risk GSURE (Eldar)

Π (z ⋆ θ (u)) =||Πx 0 || 2 2 + ||Πz ⋆ θ (u)|| 2 2 -2 z ⋆ θ (u), x ML (y) + 2 tr Π ∂z ⋆ θ (u) ∂u .
Since by assumption

∂Φz ⋆ θ (u) ∂u = Φ ∂z ⋆ θ (u)
∂u , and using the chain rule, the following holds

σ 2 tr (ΦΦ * ) + ∂ µ(y) ∂y = σ 2 tr (ΦΦ * ) + ∂Φz ⋆ θ (u) ∂u ∂u ∂y = tr Π ∂z ⋆ θ (u) ∂u whence it follows that GSURE Π (x ⋆ (y)) -GSURE (Eldar) Π (x ⋆ (y)) =||x ML (y)|| 2 2 -||Πx 0 || 2 2 -σ 2 tr (ΦΦ * ) + .
A similar reasoning when Φ has full rank leads to GSURE Id (x ⋆ (y)) -GSURE (Eldar) Id

(x ⋆ (y)) =||x ML (y)|| 2 2 -||x 0 || 2
Both our estimator GSURE Id and those of [START_REF] Eldar | Generalized SURE for Exponential Families: Applications to Regularization[END_REF] are unbiased, but they do not have necessarily the same variance. Given that they only differ by terms that do not depend on x ⋆ (y), and in particular on a parameter (here λ), selecting the latter by minimizing our GSURE Id expressions or those of [START_REF] Eldar | Generalized SURE for Exponential Families: Applications to Regularization[END_REF] leads to the same results.

Let us finally mention that in the context of deconvolution, GSURE Π boils down to the unbiased estimator of the projection risk obtained by [START_REF] Pesquet | A SURE Approach for Digital Signal/Image Deconvolution Problems[END_REF].

where f is a differentiable function with a uniformly Lipschitz gradient. Then, the most common algorithm is the gradient descent, see for instance [START_REF] Boyd | Convex optimization[END_REF], which reads

x k+1 = x k -µ k ∇f(x k ),
where the step size µ k should be small enough to ensure convergence. A major issue is that the objective function of (P y,λ ) is in general not C 1 (R n ) because non-smoothness of J is crucial to induce low complexity models, see Section 1.1.4. Several solutions exist in the litterature. A powerful class of methods to cope with such non-smooth, large scales, optimization problems are so-called proximal splitting schemes. One can refers to [START_REF] Combettes | Proximal splitting methods in signal processing[END_REF] for a detailed review.

Proximity Operator

The proximity operator was introduced by [START_REF] Moreau | Proximité et dualité dans un espace hilbertien[END_REF]. Its definition reads as follows.

D efi ni ti on 10 .1 -P roxi mi ty O perator Let f ∈ Γ 0 (R n ). The mapping

prox f : x → argmin z∈R n f(z) + 1 2 ||x -z|| 2 ,
is a well-defined single-valued mapping over R n , and coined the proximity operator of f.

The following proposition recaps the important properties of the proximity operator. A proof can be found in [START_REF] Bauschke | Convex analysis and monotone operator theory in Hilbert spaces[END_REF].

P roposi ti on 10 .1 -M ai n properti es of prox f Let f ∈ Γ 0 (R n ). (i) prox f is firmly non-expansive, i.e. for every x, z ∈ R n , || prox f (x) -prox f (z)|| 2 + ||(x -prox f (x)) -(z -prox f (z))|| 2 ||x -z|| 2 .
General norms. Let f(x) = ||x|| be a norm on R n . Its conjugate function is the indicator function ι B of the dual norm ball B defined as

B = {x ∈ R n | ||x|| * 1} where ||x|| * = max ||z|| 1
x, z .

Using (10.2), its proximity operator reads prox f = Id -P B .

ℓ 1 norm. The proximity operator of ℓ 1 is the so-called soft-thresholding operator

(prox γ||•|| 1 (x)) i =        x i -γ if x i γ 0 if |x i | γ x i + γ if x i γ.
Nuclear norm. The proximity operator of the nuclear norm is the soft-thresholding operator applies to the singular values. More precisely, if

A = U diag(σ 1 , . . . , σ n )V * is the SVD of A, then prox γ||•|| * (A) = U diag(prox γ||•|| 1 (σ 1 , . . . , σ n ))V * .
See [START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF] for a proof.

ℓ ∞ -norm. Using Moreau identity, we deduce that

prox γ||•|| ∞ (x) = x -P B ℓ 1 x γ
, where P B ℓ 1 is the orthogonal projection onto the ℓ 1 unit-ball. This projector can be computed using soft-thresholding and sorting, see [START_REF] Fadili | Total variation projection with first order schemes[END_REF]) for more details. Point (ii) of Proposition 10.1, namely that the set of fixed-points of prox f coincides with the minimizer of ( 10.1), suggests to define an algorithm, coined proximal fixed point algorithm, where the iteration are of the form

x k+1 = prox γf (x k ),
for γ > 0. Even if such a scheme converges, a major issues with these iterations is that for the functionals f considered, computing prox f cannot be done in closed form, which makes this algorithm intractable. This however suggests the introduction of more advanced iterations obtained by splitting the functional f in sum of simpler functions.

For many application in machine learning and imaging sciences, one may re-write the problem (10.1) as follows

x ⋆ ∈ Argmin x∈R n f(x) + g(x), (10.3)
where f and g enjoy some noticeable properties. For instance f or g might be smooth, or one might be able to compute their proximity operator in closed form.

Forward-Backward

Suppose that f is C 1 (R n ) with a Lipschitz-continuous derivative and g ∈ Γ 0 (R n ). In this case, one can use forward-backward iterations in order to solve (10.3). We denote by β the Lipschitz modulus of ∇f.

Every sequence x (k) generated by Algorithm 1 converges to a solution to the problem (10.3), see for instance [START_REF] Combettes | Signal recovery by proximal forwardbackward splitting[END_REF]. Several versions of the forward-backward algorithm exists, with different relaxation parameters. Again, we refer to [START_REF] Combettes | Signal recovery by proximal forwardbackward splitting[END_REF] for a discussion on this subject. Note that the projection gradient algorithm and the iterative soft-thresholding

Primal Dual Splitting

In the case of analysis models, one has to solve a problem of the form

x ⋆ ∈ Argmin x∈R n f(x) + g(D * x), (10.4)
where D is a linear operator from R p to R n . Since, in general, there is no easy way to compute the proximity operator of such mappings g • D * , it is not possible to apply directly the forward-backward or the Douglas-Rachford algorithms.

Algorithm 3 Relaxed Arrow-Hurwicz primal-dual splitting.

Choose 0 θ 1, στ||D * || 2 < 1 and u

(0) , x (0) , z (0) ∈ R n for k 0 do u (k+1) = prox σg * (u (k) + σD * z (k) ) ⊲ dual step x (k+1) = prox τf (x (k) -τDu (k) ) ⊲ primal step z (k+1) = x (k+1) + θ(x (k+1) -x (k) )
end for

Every sequence x (k) generated by Algorithm 3 converges to a solution to the problem (10.4), see for instance [START_REF] Chambolle | A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging[END_REF].

Identifying Activity with Forward-Backward Splitting

While we showed in Chapter 7 that under some mild conditions, the manifold M is stable, this result only holds for the exact minimizer x ⋆ of (P y,λ ). In practice, x ⋆ is only computed approximately by some iterates x (k) of an optimization scheme. It is thus of practical importance to be able to understand whether the results of Chapter 7 allow to shed some light on the structure of manifolds activated by the sequence of iterates, and their relation to the manifold of the original object to recover. In this section, we answer this question in the case of the forward-backward algorithm when applied to solve (P θ ) with J partly smooth.

The following theorem shows that M is indeed correctly identified by the forward-backward after a finite number of iterations.

Theorem 10 .1 Suppose that the assumptions of Theorem 7.2 hold. Then, for k sufficiently large, x (k) ∈ M, where x (k) is the sequence generated by Algorithm 1.

p ro o f The proof of this result follows the same line as Theorem 7.2 and use (Hare et al. 2007, Theorem 5.3) to concludes. A close inspection of the proof of Theorem 7.2 reveals that η θ = ri(∂J(x θ )) for the assumed regime of (||w||, λ). This in turn implies that the assumptions of (Hare et al. 2007, Theorem 5.3), are fulfilled.

We then conclude arguing in a similar way as in [START_REF] Hare | Identifying Active Manifolds[END_REF], Theorem 2).

Robust Sparse Analysis Regularization

We illustrate in this section our theoretical findings on several examples to study the robustness of the 1-D total variation, shift-invariant Haar and Fused Lasso regularizations, which are special cases of analysis ℓ 1 regularization.

Total Variation Denoising

The discrete 1-D total variation (TV) corresponds to taking D = D DIF as defined in (1.12). We now define a subclass of signals in order to study the stability of there jumps.

D efi ni ti on 10 .

∀ j ∈ J, (∆η) j = 0 and    ∀ i ∈ I, η i = s i , η 0 = η N = 0. reads x ⋆ λ = -1 + λ M 1 l 1 -ε + 2 λ M (1 l 2 -1 l 3 ) + 1 - λ M 1 l 4 , if 0 λ λ1 = -ε M 2 , and x ⋆ λ = -1 + λ M 1 l 1 + 1 - λ M 1 l 4 , if λ1
λ λ2 = M, and 0 if λ > λ2 . Figure 10.2 displays plots of the the coordinates' paths for both cases. It is worth pointing out that when ε > 0, the

x ⋆ λ [i] λ 0 1 λ2 -ε λ1 x ⋆ λ [i] λ 0 1 λ 1 ε λ 2 y[i] i 0 2ǫ y[i] i 0

D-support of x ⋆

λ is always different from that of x 0 whatever the choice of λ, whereas in the case ε < 0, for any λ1 λ λ2 , the D-support of x ⋆ λ and sign of D * x ⋆ λ are exactly those of x 0 .

Total Variation Compressed Sensing

We compare numerically the difference between the linearized precertificate p F and the analysis precertificate, see Definition 5.6. In this case, the analysis certificate reads

p A = -D(ND S x ) + NDe x ,
where N * is a basis of Ker Φ. Note that this precertificate cannot be used in Theorem 7.2, but can be used in Theorem 6.1 of ensure ℓ 2 noise robustness. Figure 10.3 shows an example of p F and p A for a single realization of Φ. We consider the realization Φ drawn from the Gaussian ensemble with redundancy q/n = 1 3 and a signal x with 5 piecewise constant components. In this compressed sensing scenario, one can see that p A behaves much better than p F . Indeed, Phi * p A is strictly within ri(∂J(x 0 )), which is not the case for Phi * p F .

Shift-Invariant Haar Deconvolution

Sparse analysis regularization using a 1-D shift invariant Haar dictionary is efficient to recover piecewise constant signals. This dictionary is built using a set of scaled and dilated Haar filters

ψ (j) i = 1 2 τ(j+1)        +1 if 0 i < 2 j -1 if -2 j i < 0 0 otherwise,
where τ 0 is a normalization exponent. For τ = 1, the dictionary is said to be unit-normed. For τ = 1/2, it corresponds to a Parseval tight-frame. The action on a signal x of the analysis operator corresponding to the translation invariant Haar dictionary D H is

D * H x = ψ (j) ⋆ x 0 j J max ,
where ⋆ stands for the discrete convolution (with appropriate boundary conditions) and J max < log 2 (n), where n is the size of the signal. The analysis regularization ||D * H x|| 1 can also be written as the sum over scales of the TV semi-norms of filtered versions of the signal. As such, it can be understood as a sort of multiscale total variation regularization. Apart from a multiplicative factor, one recovers Total Variation when J max = 0.

We consider a convolution setting (for n = 256) where Φ is a circular convolution operator with a Gaussian kernel of standard deviation σ. We first study the impact of σ on the identifiability criterion IC. The original signal x η is a centered boxcar signal with a support of size 2ηn

x η = 1 {⌊n/2-ηn⌋,...,⌊n/2+ηn⌋} , η ∈ (0, 1/2] .

Figure 10.4 displays the evolution of IC(x 0 ) as a function of σ for three dic-tionaries: the total variation dictionary and the Haar wavelet dictionary with two normalization exponents τ = 1 and τ = 0.5. In this experiment, we chose η = 0.2. One can observe that the three curves pass through 1 for the same value of σ (near 1 here). This shows that for σ small enough, deconvolv- ing a box signal is stable in the sense that the discontinuities are correctly estimated in the presence of a small additive noise in the observations. In addition, in the identifiability regime, IC(x 0 ) appears smaller in the case of the unit-normed normalization (i.e. τ = 1). However, one should avoid to infer stronger conclusions since a detailed computation of the constants involved in Theorem 7.2 would be necessary to completely and fairly compare the stability performance achieved with each of these three dictionaries.

. If x = k i=1 γ i 1 [a i ,b i ]
, where γ i ∈ R and a i b i < a i+1 , then the model space T x reads

T x = k i=1 ρ i 1 [a i ,b i ] | ρ i ∈ R .
This significates that the Fused Lasso favors sparse sums of boxcar signals.

We consider a compressed sensing setting (with the signal size n = 256) and examine the behavior of IC with respect to the undersampling ratio q/n and the true signal properties. Φ is drawn from the standard Gaussian ensemble, i.e. Φ i,j ∼ i.i.d. N(0, 1). The sampled signal x η,ρ is the superposition of two boxcars distant from each other by 2ρN and each of support size ηN x η,ρ = 1 {⌊( 1 2 -η-ρ)n⌋,...,⌊( 1 2 -ρ)n⌋} + 1 {⌊( 1 2 +ρ)n⌋,...,⌊( 1 2 +η+ρ)n⌋} .

In our simulations, we fixed ρ = 0.1.

Figure 10.5 depicts the evolution of the empirical probability with respect to the sampling of Φ of the event IC < 1 as a function of the sampling ratio Q/N ∈ [0.5, 1] and the boxcar support size η ∈ [0.025, 0.15]. This probability is computed from 1000 Monte-Carlo replications of the sampling of Φ. With no surprise, one can clearly see that the probability increases as more measurements are collected. This probability profile also seems to be increasing as η decreases, but this is likely to be a consequence of the choice of the Fused Lasso parameter ε, and the conclusion may be different for other choices. This is indeed confirmed in our last experiment whose results are displayed in Figure 10.6. It shows the evolution of the empirical probability of the event IC < 1 as a function of the Fused Lasso parameter ε ∈ [1/n, 200/n] and the support size η ∈ [0.025, 0.15]. This probability is again computed from 1000 Monte-Carlo replications. Depending on the choice of ε, the probability profile does not necessarily exhibit a monotonic behavior as a function of η. For large values (more weight on Id in the Fused Lasso dictionary), the probability decreases monotonically as η increases which can be explained by the fact that higher η corresponds to less sparse signals. As ε is lowered, higher weight is put on the TV regularization, and the behavior is not anymore monotonic. Now, the probability reaches a peak at intermediate values of η and then vanishes quickly. The peak probability also decreases with decreasing ε.

Robust Antisparse Regularization

In some cases, one aims at recovering flat vectors, i.e such that for most i, x i = ||x|| ∞ . This is for instance the case in computer vision applications when performing quantization of random projections, see [START_REF] Jégou | Anti-sparse coding for approximate nearest neighbor search[END_REF]. One can use as regularizer the ℓ ∞ norm defined as

||x|| ∞ = max i∈{1,...,n} |x i |.
In this case, one has The following result studies the behavior of the linearized precertificate in a compressed sensing scenario.

Theorem 10 .3 Let x be an arbitrary vector with its saturation support I, its model subspace T x = S ⊥ x and generalized sign vector e x as defined above. Let β > 1. For Φ drawn from the standard Gaussian ensemble with m n -|I| + 2β|I| log(|I|/2) , IC(x) < 1 with probability at least 1 -2(|I|/2) -f (β,|I|) where

f(β, |I|) = β 2|I| + β -1 - β 2|I| 2 .
p ro o f To lighten the notation, we drop the dependence on x of T , S and e.

Without of loss of generality, by symmetry of the norm, we will assume that the entries of x are positive.

We follow the same program as in the CS literature. The key ingredient of the proof is the fact that owing to the isotropy of the Gaussian ensemble, p .

Equating the arguments of the exponentials and solving

t 2 4q + t 2|I| - q 2|I| -log |I| 2 = 0
for t to get equal probabilities, we get

t = q |I|    1 + 2|I|   1 -2 2|I| log |I| 2 q   -1    ,
where q = mn + |I| 1 by the injectivity assumption. Setting

β = q 2|I| log |I| 2 ,
we get under the bound on m that β > 1, and

t = 2β log |I| 2 1 + 2|I| β-1 β -1 .
Inserting t in one of the probability terms, and after basic algebraic rearrangements, we get the probability of success with the expression of the function f(β, |I|).

The above bound and probability bears some similarities to what we get with ℓ 1 minimization, except that now the probability of success scales in a power of |I| and not n directly. The reason underlying such a similarity is the proof technique usual in CS-type bounds and the use of the minimal ℓ 2 -norm dual certificate. In particular, a union bound is behind the log factor. If some improvements is sought after, it is on this step that it can be gained.

The map f(β, |I|) is an increasing function of |I|, so that lim |I|→∞ f(β, |I|) = β -1 and the probability of success increases with increasing size of the saturation support. But this comes at the price of a stronger requirement on the number of measurements.

For the noiseless problem (P y,0 ), it can be shown using arguments based on the statistical dimension [START_REF] Amelunxen | Living on the edge: A geometric theory of phase transitions in convex optimization[END_REF] of the descent cone of the ℓ ∞norm that there is a phase transition exactly at n -|I|/2, see (Chandrasekaran et al. 2012, Proposition 3.12). The reason is that each face of the descent cone of the hypercube at a point living on its k-dimensional face is the direct sum of a subspace (the linear hull of the face), and of an orthant of dimension nk (up to an isometry). The statistical dimension is then (nk)/2 + k = (n + k)/2 = n -|I|/2, observing that k = n -|I|.

D efi ni ti on 10 .3 Let Λ be a D-cosupport. Suppose that (INJ T ) holds. We define the matrix Γ [Λ] as (10.6) where U is a matrix whose columns form a basis of Ker D * Λ .

Γ [Λ] = U (U * Φ * ΦU) -1 U * .
Observe that the action of Γ [Λ] can be rewritten as a quadratic optimization under linear constraint

Γ [Λ] u = argmin D * Λ x=0
1 2

||Φx|| 2 -x, u .
The remaining obstacle faced when implementing the GSURE formula of Theorem 9.1 is to compute the divergence term. However, for large scale-data as in image and signal processing, the computational storage required for the matrix in the argument of the trace would be prohibitive. Additionally, computing Γ [Λ] can only be reasonably afforded for small data size. Fortunately, the structure of df A (y) and the definition of Γ [Λ] allows to derive an efficient and principled way to compute the trace term. This is formalized in the next result.

P roposi ti on 10 . Hence denoting ν(z) = Γ [Λ] Φ * z, and using the fact that for any matrix U, tr U = E Z Z, UZ , we arrive at (10.7).

We then use the fact that Γ [Λ] Φ * , the inverse of Φ on Ker D * Λ , is the mapping that solves the following linearly constrained least-squares problem

Γ [Λ] Φ * z = argmin h∈Ker D * Λ ||Φh -z|| 2 2 .
Writing the KKT conditions of this problem leads to (10.8), where ν are the Lagrange multipliers.

In practice, the empirical mean estimator is replaced for the expectation in (10.7), hence giving (10.9) for k realizations z i of Z, where WLNN stands for the Weak Law of Large Numbers. Consequently, the computational bulk of computing an estimate of df A (y) is invested in solving for each ν(z i ) the symmetric linear system (10.8) using e.g. a conjugate gradient solver.

1 k k i=1 ν(z i ), Φ * A * Az i WLLN -→ df A (y) ,

Parameter Selection using the GSURE

Super-Resolution with Total Variation Regularization In this example, Φ is a vertical sub-sampling operator of factor two (hence q/n = 0.5). The noise level has been set such that the observed image y has a peak signal-to-noise ratio (PSNR) of 27.78 dB. We used an anisotropic total variation regularization; i.e. the sum of the ℓ 1 -norms of the partial derivatives in the first and second direction (not to be confused with the isotropic total variation). Fig. 10.7.d depicts the projection risk and its GSURE Π estimate obtained from (10.9) with k = 1 as a function of λ. The curves appear unimodal and coincide even with k = 1 and a single noise realization. Consequently, GSURE Π provides a highquality selection of λ minimizing the projection risk. Close-up views of the central parts of the degraded, restored (using the optimal λ), and true images are shown in Fig. 10.7(a)-(c) for visual inspection of the restoration quality.

Compressed Sensing with Wavelet Analysis Regularization We consider in this example a compressed sensing scenario where Φ is a random partial DCT measurement matrix with an under-sampling ratio q/n = 0.5. The noise is such that input image y has a PSNR set to 27.50 dB. We took D as the shiftinvariant Haar wavelet dictionary with 3 scales. Again, we estimate GSURE Π with k = 1 in (10.9). The results observed on the super-resolution example are confirmed in this compressed sensing experiment both visually and qualitatively, see Fig. 10.8.

Relation to Previous Work

In least-squares regression regularized by a sufficiently smooth penalty term, the DOF can be estimated in closed-form [START_REF] Solo | A SURE-fired way to choose smoothing parameters in illconditioned inverse problems[END_REF]. However even in such simple cases, the computational load and/or storage can be prohibitive for large-scale data.

To overcome the analytical difficult for general non-linear estimators, when no closed-form expression is available, first attempts developed bootstrapbased (asymptotically) unbiased estimators of the DOF [START_REF] Efron | The estimation of prediction error: Covariance penalties and cross-validation[END_REF]. [START_REF] Ye | On measuring and correcting the effects of data mining and model selection[END_REF] and Shen et al. (2002) proposed a data perturbation technique to approximate the DOF (and the SURE) when its closed-form expression is not available or numerically expensive to compute. For denoising, a similar Monte-Carlo approach has been used in [START_REF] Ramani | Monte-Carlo SURE: a black-box optimization of regularization parameters for general denoising algorithms[END_REF] where it was applied to total-variation denoising, wavelet soft-thresholding, and Wiener filtering/smoothing splines.

Alternatively, an estimate can be obtained by recursively differentiating the sequence of iterates that converges to a solution of the original minimization problem. Initially, it has been proposed by [START_REF] Vonesch | Recursive risk estimation for non-linear image deconvolution with a wavelet-domain sparsity constraint[END_REF] to compute the GSURE of sparse synthesis regularization by differentiating the sequence of iterates of the forward-backward splitting algorithm. We have recently proposed a generalization of this methodology to any proximal splitting algorithm, and exemplified it on ℓ 1 -analysis regularization including the isotropic total-variation regularization, and ℓ 1ℓ 2 synthesis regularization which promotes block sparsity [START_REF] Deledalle | Stein Unbiased GrAdient estimator of the Risk (SUGAR) for multiple parameter selection[END_REF].

In our case, we have shown that the computation of an accurate estimator of the DOF for analysis ℓ 1 regularization, and therefore of GSURE A for various risks, can be obtained by solving a linear system. This is more efficient than the previous general-purpose iterative methods that are computationally expensive. A solution x ⋆ (y) of (P y,λ ) at the optimal λ (the one minimizing GSURE Π ). (c) The underlying true image x 0 . (d) Projection risk Risk Π and its GSURE Π estimate obtained from (10.9) using k = 1 random realization.

11

Conclusion

T hi s thesis revolves around the theme of sensitivity analysis of optimization problems. Each part is a variation on a popular theme of sensitivity analysis, such as Lipschitz continuity of the set of the minimizers of (P y,λ ) when λ 0 or manifold stability when λ = 0 and λ > 0. This leads us to different applications: ℓ 2 -robustness (Chapter 6), model identifiability (Chapter 7), local differentiability (Chapter 8), unbiased estimation (Chapter 9) and algorithmic identifiability (Chapter 10).

The theoretical analysis provided by this work draws a connection between these popular applications in imaging, signal processing and machine learning. Partial smoothness allows us to recover results already known in the litterature, within a coherent and unifying framework. It also allows us to significantly extend these results to a larger class of regularizers and to gain a better understanding of the effects of these regularizers.

The research program does not stop here. Many extensions of our work are of interest.

Non convex regularizers and data loss. Non-convex functions are often used in image processing or statistics. There are two differents kinds of nonconvexity which arise, i.e. on the data fidelity term and the regularizer. where 0 < p < 1 and ||u|| p = ( |u i | p ) 1/p . In practice, using such a non-convex functional seems to produce better result in imaging and computer graphics, most probably because it better fits the high level of sparsity of natural image gradients. We believe that our results can be extended to non-convex partly smooth functions.

Unbounded functions. Our analysis does not cover the case of variational formulation with constraints. A typical example is when one imposes nonnegativity constraints. For instance, the problem min

Résumé des travaux

Contexte. Cette thèse concerne la résolution de problèmes inverses linéaires en dimension finie. Elle contribue ainsi à l'étude théorique de thématiques centrales en traitement du signal ou d'image, en statistique ainsi qu'en apprentissage. Un tel problème peut être écrit sous la forme y = Φx 0 + w, o ù y ∈ R q est le vecteur d'observations, x 0 ∈ R n les données inconnues à retrouver, Φ un opérateur linéaire de R q dans R n et w un terme de bruit additif. Ce modèle inclut de nombreux cas typiques en imagerie tels que le débruitage, la déconvolution, l'interpolation, l'échantilonnage compressé ainsi que la tomographie. L'opérateur linéaire Φ est généralement mal conditionné. C'est la raison pour laquelle il s'avère nécessaire de mettre en place une stratégie de reconstruction. Un cadre classique est celui des méthodes variationnelles, pouvant s'écrire sous la forme x ⋆ ∈ Argmin Robustesse. Le premier axe de cette thèse est dédié à l'étude de la robustesse du problème (P y,λ ). Nous cherchons à estimer combien un vecteur solution x ⋆ s'approche du vecteur d'origine x 0 à la fois au sens d'une erreur ℓ 2 , mais également en terme de sélection de modèle. Nous montrons le théorème suivant de convergence linéaire.

Th éor ème 1 Soit T 0 l'espace modèle tangent de x 0 . Supposons que :

• il existe α = Φ * η ∈ ri ∂J(x 0 ), dite condition source, o ù ri ∂J(x) est l'intérieur relatif de ∂J(x) pour la topologie induite par aff ∂J(x),

• Ker Φ ∩ T 0 = {0} (injectivité restreinte).

Si λ = cε, c > 0, alors pour tout minimiseur x ⋆ de (P y,λ ), Sensibilité. Le second axe de cette thèse porte sur l'analyse de sensibilité de (P y,λ ). Cette analyse permet, dans le cadre d'observations aléatoires, la construction d'un estimateur du risque quadratique non biaisé. Nous introduisons l'espace de transition H, correspondant aux observations y telles que l'espace T associé à une solution de (P y,λ ) ne soit pas stable vis-à-vis de petites perturbations de y. Publications Cette thèse reprend le contenu d'articles de journaux internationaux ou de prépublications suivants.

(1) M. J. Fadili, G. Peyré et S. Vaiter. Linear Convergence Rates for Gauge Regularization. En préparation.

Figure 1 . 1 :

 11 Figure 1.1: Forward and Inverse Problem

Figure 1 . 2 :

 12 Figure 1.2: One Pixel Camera project. Source: Rice University

||

  • || 1 η | ∀j ∈ I, η j = 0 sign(x) I = supp(x) ||D * • || 1 Ker(D * I c ) P Ker(D * I c ) sign(D * x) I = supp(D * x)|| • || 1,2 η | ∀j / ∈ I, η j = 0 (N(x b )) b∈B I = {g ∈ B | x g = 0} || • || * Z | U * ⊥ ZV ⊥ = 0 UV * x = UΛV * || • || ∞ {α | α I = ρs I for ρ ∈ R} sign(x)/|I| I = {i | |x i | = ||x|| ∞ }Table 1.1: Examples of Model Tangent Subspace. The notations are precised in the following sections.

  b∈B b = {1, . . . , n} and ∀b, b ′ ∈ B, b ∩ b ′ = ∅.

  is the face of C • exposed by x. The latter is the intersection of C • and the supporting hyperplane η ∈ R N | η, x = γ C (x) . p ro o f See (Hiriart-Urruty et al. 2001, Proposition 3.1.4).

Lemma 2 . 8 Lemma 2 . 8

 2828 connects the gauge associated to C to the one associated to D(C) by an optimization problem over the kernel of D. Let C be a closed convex set such that 0 ∈ ri C, and D a linear operator. Then, for every x ∈ Im(D) γ D(C) (x) = inf z∈Ker(D) γ C (D + x + z) . P ro o f It is immediate to see that D(C) is a closed convex set containing the origin. Moreover, we have Im(D * ) ∩ dom(σ C ) = ∅, since the origin is in both of them. Thus, using (Hiriart-Urruty et al. 2001, Theorem X.2.1.1) and Lemma 2.5, we have

  e. a continuous, bounded from below, proper, and convex function. We now introduce the model tangent subspace at a point x. D efi ni ti on 3 . 1 -M odel Tangent Subspace . For any vector x ∈ R n , we denote by Sx the affine hull of the subdifferential of J at x Sx = aff ∂J(x), and e x , its model vector, the orthogonal projection of 0 onto Sx e x = argmin e∈ Sx ||e||.

Figure 3 . 1 :

 31 Figure 3.1: Illustration of the geometrical elements (S x , T x , e x ) for a gauge.

0

  and e 0 are those of the function J 0 at D * x. The following proposition details the decomposability structure of analysis-type functions. P roposi ti on 3 .3 With the above notation, the model tangent subspace of J = J 0 • D * reads (i) T = Ker(D * S 0 ) = D * T 0 , or equivalently S = Im(D S 0 ) = DS 0 . (ii) e = P T De 0 . p ro o f (i) One has ∂J = D • ∂J 0 • D * , hence S = DS 0 = Im(D S 0 ) and T = S ⊥ = Ker(D * S 0 ). (ii) As S = D S0 = De 0 + S, we get from Proposition 3.1 e ∈ argmin z∈ S ||z|| = argmin z-De 0 ∈S ||z|| = De 0 + argmin h∈S ||h + De 0 ||. The second term is the projection of -De 0 onto the linear subspace S. Thus, e = De 0 + P S (-De 0 ) = (Id -P S )De 0 = P T De 0 , which is the result stated.

  e x is defined by s (I) = (sign(x)) (I) and s (I c ) = 0. p ro o f For x = 0, ∂J(x) is the unit ℓ 1 ball, hence Sx = S x = R N , T x = {0} and e x = 0.

  and N(a) = a/||a|| if a = 0, and N(0) = 0.

Figure 3 . 4 2 1 + x 2 2 +

 3422 Figure 3.4 shows the underlying geometry of the ℓ 1ℓ 2 regularization in three dimensions. We take J(x) = x 2 1 + x 2 2 + |x 3 |.

Figure 3 . 4 :

 34 Figure 3.4: ℓ 1ℓ 2 geometry. In red, the ℓ 1ℓ 2 ball. In blue, the dual ball.

Figure 3 . 5 :

 35 Figure 3.5: Polyhedral geometry.

P

  roposi ti on 4 .7 J = ||D * • || 1 is a strong partly smooth function with parameters ν x = min i∈I | d i , x i | and µ x = τ x = ξ x = 0. p ro o f This is a consequence of Proposition 4.5 with J 0 = || • || 1 .

  whereI = I(x) = {b | x b = 0}.p ro o f Let x ′ ∈ T , i.e. I(x ′ ) ⊆ I(x), and ν x = min b∈I ||x b ||. First, observe that the condition ||xx ′ || ∞,2 = max b∈B ||x bx ′ b || < ν x ensures that for all b

(5. 5 )

 5 By optimality of p θ , one has g(p θ , θ) g(p 0,y , θ), and thus -2 p θ , y + λ||p θ || 2 -2 p 0,y , y + λ||p 0,y || 2 -2 p θ , y + λ||p 0,y || 2 or equivalently ||p θ || ||p 0,y ||.

. 7 )

 7 p ro o f See (Hiriart-Urruty et al. 2001, Proposition III.5.3.1).

Figure 5 .

 5 Figure 5.1 illustrates the tangent cone for the ℓ 1 -norm and a quadratic regularization 1 2 ||x|| 2 2 .

Figure 5 . 1 :

 51 Figure 5.1: Tangent cone generated at x = (1, 0) and its polar for two functions J: a) ||x|| 1 and b) 1 2 ||x|| 2

  ti on 5 . 6 -A nalysi s P recerti fi cate Let x ∈ R n and T = T D * x . The analysis precertificate p A reads p A = D argmin ω∈R p ||ω|| subject to Dω ∈ Im Φ * and ω T = e D * x .

7. 1 . 3

 13 Proof of Theorem 7.2 Constrained problem. We consider the following non-convex constrained minimization problem xθ ∈ Argmin x∈M f(x, θ).

  0 ) I || ∞ = ||Φ * p F || ∞ , where I = supp(x 0 ). The Exact Recovery Condition reads ERC(x 0 ) = ||Φ * I c Φ +, * I || ∞,∞ .The weak-ERC reads wERC

202 9 . 1

 91 Generalized Linear Models and Degrees of Freedom 9.1 Generalized Linear Models and Degrees of Freedom 9.1.1 Generalized Linear Models

  GSURE A (y) =||A(yµ(y))|| 2 2σ 2 tr(A * A) + 2σ 2 df A (

. 4 )

 4 is an unbiased estimator of the risk E ||A µ(Y) -Aµ 0 || 2 , and df A (Y) = tr(A∆(Y)) a.e.

9 . 2

 92 Suppose that conditions (C O ), (C τ ) and (C L ) hold. Then, the SURE SURE(Y) =||∇ log B(Y)µ(Y)|| 2 + 2 df(Y) -(||∇ log B(Y)|| 2 -||µ 0 || 2 ) (9.5) is an unbiased estimator of the risk E || µ(Y)µ 0 || 2 , and df(Y) = tr ∆(Y) a.e. p ro o f The proof is similar but uses the result (Eldar 2009, Theorem 1) to conclude.

  df(y) = |Λ|λ b⊆Λ |b| -1 ||y b || where Λ = {b ∈ B | ||y b || > λ} .

Figure 9 .

 9 Figure 9.1 illustrates this theorem by showing df(y) as a function of y ∈ R 2 for a single block g = {0, 1} of size 2. Note that the DOF is not constant equal to 2 outside R λ = {y | ||y|| > λ} (which would be the case for a 1-D soft thresholding). It tends to 2 when y → +∞ and is equal to 1 on the boundary of R λ .

Figure 9 . 1 :

 91 Figure 9.1: Display of df in 2-D for λ = 0.3.

  2 A signal is said to contain a staircase if there exists i ∈ {1 . . . |I| -1} such that sign(D * I x) i = sign(D * I x) i+1 = ±1.

Figure 10 .Figure 10 . 1 :

 10101 Figure 10.1 shows examples of signals with (left) and without (right) staircase subsignals.

Figure 10 . 2 :

 102 Figure 10.2: Top row: Signals y for ε < 0 (left) and ε > 0 (right). Bottom row:Corresponding coordinates' path of x ⋆ λ as a function of λ. The solid lines correspond to the coordinates in l 1 and l 4 , and the dashed ones to the coordinates in l 2 and l 3 .

Figure 10 . 3 :

 103 Figure 10.3: Total Variation Compressed Sensing with q/n = 1/3. Top: u F such that Φ * p F = div u F . Bottom: u A such that Φ * p A = div u A .

Figure 10 . 4 :

 104 Figure 10.4: Behavior of IC for a noiseless deconvolution scenario with a Gaussian blur and ℓ 1 -analysis sparsity regularization in a shift invariant Haar dictionary with J max = 4. IC is plotted as a function of the Gaussian blurring kernel size σ ∈ [0.5, 3.0] for the total variation dictionary and the Haar wavelet dictionary with two normalization exponents τ. Dash-dotted line: τ = 1 (unit-normed). Dashed line: τ = 1/2 (tight-frame). Solid line: total variation.

Figure 10 . 5 :

 105 Figure 10.5: Behavior of IC for a compressed sensing scenario matrix with a Gaussian measurement matrix and the Fused Lasso regularization. Empirical probability of the event IC < 1 as a function of the sampling ratio q/n ∈ [0.5, 1] and the support size η ∈ [0.025, 0.15] with ε = 50/N.

Figure 10 . 6 :

 106 Figure 10.6: Behavior of IC for a compressed sensing scenario matrix with a Gaussian measurement matrix and the Fused Lasso regularization. Empirical probability of the event IC < 1 as a function of the parameter ε ∈ [1/n, 200/n] and the support size η ∈ [0.025, 0.15] with q/n = 0.8.

T

  x = x ′ | x ′ I = ρx I for some ρ ∈ R , where I = {i | x i = ||x|| ∞ }.This reflects that fact that J = || • || ∞ favors signal having the same saturation pattern as x, see Proposition 3.15.

F 2 .

 2 and Φ * S are independent. Thus, for some τ > 0 Pr (IC(x) 1) Pr IC(x)1 ||p F || τ + Pr (||p F || τ) . As soon as m dim(T ) = n -|I| + 1, Φ T is full-column rank. Thus ||p F || 2 = e, (Φ * T Φ T ) -1 e . Φ * T Φ T-1 is an inverse Wishart matrix with m degrees of freedom. To estimate the deviation of this quadratic form, we use classical results on inverse χ 2 random variables with mn + |I| degrees of freedom and we get the tail boundPr ||p F || 1 mn + |I|t ||e|| efor t > 0. Now, conditionally on p F , the entries of α S = P S Φ * p F are i.i.d. N(0, ||p F || 2 ) and so are those of -α S by trivial symmetry of the centered Gaussian. Thus, using a union bound, we getPr IC(x) 1 ||p F || τ Pr max i∈I (-(α S x ) i ) + 1/|I| ||p F || τ Pr max i∈I ((α S x ) i ) + 1/|I| ||p F || τ |I| Pr ((z) + 1/(τ|I|))|I| Pr (z 1/(τ|I|))Observe that (α S ) i = 0 for all i ∈ I c . Choosingτ = 1 |I|(mn + |I|t)where we used that ||e|| = 1/ √ I, and inserting in the above probability terms, we getPr (||p F || τ) e

2

 2 One has df A (y) = E Z ( ν(Z), Φ * A * AZ ) (10.7)where Z ∼ N(0, Id p ), and where for any z ∈ R p , ν = ν(z) solves the following linear system We havetr AΦΓ [Λ] Φ * A * = tr ΦΓ [Λ] Φ * A * A .

Figure 10 . 7 :

 107 Figure 10.7: Illustration of the selection of λ by minimizing GSURE Π in a superresolution problem (q/n = 0.5) with anisotropic total variation regularization. (a) The observed image y. (b) A solution x ⋆ (y) of (P y,λ ) at the optimal λ (the one minimizing GSURE Π ). (c) The underlying true image x 0 . (d) Projection risk Risk Π and its GSURE Π estimate obtained from (10.9) using k = 1 random realization.

Figure 10 . 8 :

 108 Figure 10.8: Illustration of the selection of λ by minimizing GSURE Π in a compressed sensing problem (q/n = 0.5) by an ℓ 1 -analysis regularization in a shift-invariant Haar wavelet dictionary. (a) The MLE x ML . (b)A solution x ⋆ (y) of (P y,λ ) at the optimal λ (the one minimizing GSURE Π ). (c) The underlying true image x 0 . (d) Projection risk Risk Π and its GSURE Π estimate obtained from (10.9) using k = 1 random realization.

  min

  x∈R n F(x, y) + J(x), For instance, one thinks of the analysis ℓ p regularization, i.e. Φx|| 2 + λ||D * x|| p p ,

2 2 OS

 22 Φx|| 2 + λG(x) subject to ∀i, x i 0, Φx|| 2 + λJ(x), where J = G + ι C and C = {x | ∀i, x i 0}. Unfortunately, the function J is unbounded, due to the presence of ι C , hence not covered by our results.Continuous setting. The continuous problem, defined in (cP y,λ ),f ⋆ ∈ argmin f∈H 1 2 ||y -Ψf|| 2 + λJ(f) (cP λ (y))leads to the same questions as (P y,λ ). For instance, when J is the total variation of a function (1.14), what can be said about the stability of the discontinuity set of a solution ? This would extend the result of Caselles et al. (2007) obtained when Φ = Id and w = 0Operator bound of A with respect to J 1 and J An Projection onto the first k components. P T Orthogonal projection of the vector space T SL x (•) Set of partly smooth functions at x with linear manifold S x (M) Set of partly smooth functions at x relative to M , x 0 ) Bregman divergence between x and x 0 with respect to η Projection of x on T , x T = P T x (C x,y ) (General) restricted injectivity condition (INJ T ) Restricted injectivity condition on T

  est une fonction de R n dans R + que l'on considérera dans cette thèse convexe. Il s'agit de réaliser un compromis entre fidélité aux données (terme quadratique ici) et régularisation, représentée ici par J. Ce compromis est dicté par le choix du paramètre λ. L'opérateur Φ n'étant généralement pas injectif, il est important de garder en mémoire le fait que x ⋆ n'est pas uniquement déterminé. Quand w = 0, c'est-à-dire en absence de bruit, (P y,λ ) se réduit sous la formex ⋆ ∈ Argmin x∈R n J(x) sujet à Φx = y. (P y,0 )Le choix de la pénalité J est un problème de recherche actif. Une des directions possibles est de considérer J comme promouvant des données dites de faible complexité. Plus précisément, en se donnant une collection de sous-espaces vectoriels T de R n , nous sommes amenés à considérer le problème de sélection de modèle non-convexe inf T ∈T,x∈T ||y -Φx|| 2 + λpen(T ), o ù typiquement pen(T ) = dim T . Ce problème étant non seulement nonconvexe, mais souvent également NP-difficile, il est nécessaire de considérer une méthode d'approximation. Dans cette thèse, nous nous consacrons aux relaxations convexes. Ainsi la fonction de comptage (taille du support d'un vecteur) est relâchée par la norme ℓ 1 , le rang d'une matrice est approché par la norme nucléaire, etc. Ayant fixé une régularisation convexe J, nous définissons le vecteur modèle associé à x ∈ R n comme e x = argmin e∈aff ∂J(x) ||e||, o ù aff ∂J(x) est l'espace affine engendré par la sous-différentielle de J en x. Nous définissons également l'espace modèle tangent commeT x = span(∂J(x)) ⊥ , o ù span C est l'espace linéaire engendré par C. Par exemple, si J = || • || 1 , alors T x = {η | supp(η) ⊆ supp(x)} et e x = sign(x).

  ||x ⋆ -x 0 || 2 Cε , o ù C = C 1 (2 + c||η|| 2 ) + C 2 (1 + c||η|| 2 /2) 2 cC η , C 1 > 0 et C 2 > 0 étant deux constantes indépendantes de η et 0 < C η < 1.Nous dirons que J est une fonction partiellement lisse[START_REF] Lewis | The Convex Analysis of Unitarily Invariant Matrix Functions[END_REF] pour une variété M si, pour tout point x ∈ M, J restreinte à M est C 2 autour de x, l'espace tangent T M (x) à M en x est T x et que l'application multivoque ∂J est continue au point x relativement à M. Pour ce type de régularisation, incluant les normes ℓ 1 , ℓ 1ℓ 2 , nucléaire ou encore ℓ ∞ , nous montrons le résultat suivant de sélection de modèle.Th éor ème 2 Soient x 0 ∈ R n , T = T x 0 et e = e x 0 . Supposons que :• J est partiellement lisse pour la variété M et x 0 ∈ M,• Φ * Φ +, * T e ∈ ri ∂J(x 0 ), • Ker Φ ∩ T = {0}.Alors il existe des constantes positives C, C ′ telles que, si w et λ sont choisis tels que||w|| C et λ = C ′ ||w||, (.1)la solution x ⋆ du problème (P y,λ ) est unique et satisfaitx ⋆ ∈ M et ||x 0x ⋆ || = O(||w||).

2 .

 2 o ù H T = bd(Π n+p,n (A T )), Π n+p,n est la projection canonique sur les n premières composantes, bd C est le bord de C, etA T = (y, x T ) ∈ R n × T | Φ * T (Φx Ty) ∈ rbd ∂J(x T ) .Notre première contribution est de déterminer le comportement local des solutions du problème (P y,λ ) à l'extérieur de cet ensemble. Nous notons J T la restriction de J à T .Th éor ème 3 Soit J une fonction 1-homogène partiellement lisse pourM = T x 0 . Soient y ∈ H et x ⋆ une solution de (P y,λ ) telle que Ker Φ T ∩ Ker D 2 J T (x ⋆ ) = {0} (I x ⋆ ) o ù T = T x ⋆ .Alors il existe un voisinage V ⊂ R n de y et une application x : V → T tels que :(i) pour tout ȳ ∈ V, x( ȳ) est une solution de (P y,λ ), et x(y) = x ⋆ , (ii) l'application x est C 1 (V) et ∀ ȳ ∈ V, D 1 x( ȳ) = -(Φ * T Φ T + D 2 J T (x ⋆ )) -1 Φ T , L'application y → µ(y) = Φx ⋆ est univoque et C 1 (R n \ H). Pour tout y ∈ H, il existe une solution x ⋆ de (P y,λ ) telle que (I x ⋆ ) est satisfaite. De plus, pour tout y ∈ H, div( µ)(y) = tr(∆(y)) o ù ∆(y) = -Φ T (Φ T * Φ T + D 2 J T (x ⋆ )) -1 • Φ T * . Soit Y = Φx 0 + W avec W ∼ N(0, σ 2 Id n ).Le degré de liberté (DOF) d'une procédure statistique quantifie la complexité de celle-ci. Suivant la définition d'Efron (1986), le DOF est défini comme df = n i=1 cov(Y i , µ i (Y)) σ Dans ce cadre, nous montrons le théorème suivant d'estimation du risque. Th éor ème 4 Soit J une fonction partiellement lisse pour M = T x 0 définissable dans une structure o-minimale. Alors : (i) H est de mesure de Lebesgue nulle, (ii) µ est une fonction Lipschitz, donc faiblement différentiable, avec un gradient borné p.p, (iii) df = tr(∆(Y)) est un estimateur sans biais de df = E(div( µ(Y)), (iv) le SURE, défini par SURE( µ)(Y) =||Yµ(Y)|| 2 + 2σ 2 dfnσ 2 , (.2) est un estimateur non biaisé de E || µ(Y)µ 0 || 2 .

2 Mathematical Background Contents 2.1 Convex Analysis

	2.1.1 Functions
	2.1.2 Convexity
	2.1.3 Special Convex Sets
	2.1.4 Multivalued Mappings
	2.1.5 Asymptotic Cone and Function
	2.2 Differential Properties
	2.2.1 Subdifferential
	2.2.2 Minimizers Gradients for Composite Problems
	2.2.3

Smooth Manifolds 2.3 Gauges 2.3.1 Gauge and its Polar

	2.3.2 Polar Calculus
	2.3.3 Lift to Matrix Spaces
	2.3.4 Operator Bounds
	2.4 O-minimality
	2.4.1 Definition
	2.4.2 Properties

  and is finite everywhere if, and only if, C is a bounded set. We have the following property. P roposi ti on 2 .1 Let C 1 , C 2 two nonempty closed convex subsets. Then,

  or equivalently γ C is coercive if, and only if, C is compact. (v) γ C is bounded and coercive on dom γ C = span C if, and only if, C is compact and 0 ∈ ri C. In particular, γ C is bounded and coercive if, and only if, C is compact and 0 ∈ int C.

p ro o f (i)-(iii) are obtained from (Hiriart-Urruty et al. 2001, Theorem V.1.2.5). (iv) is obtained by combining (Hiriart-Urruty et al. 2001, Corollary V.1.2.6 and Proposition IV.3.2.5). (v): the second statement follows by combining (iii)-(iv), while the first part is the second one written in dom γ C

  where Π : R n+1 → R n is the projection on the first n. (v) o-minimality: O 1 is precisely the finite unions of intervals and points. D efi ni ti on 2 . 33 -D efi nable Set and Functi on. Let O be an ominimal structure. The elements of O n are called the definable subsets of R n

Table 4 . 1 :

 41 Examples of Partly Smooth Functions. For all these regularizations, τ x = ξ x = 0.

	Contents
	5.1 Primal Problem	132
	5.1.1 Existence of Solutions	133
	5.1.2 Convergence of the Primal Problem	134
	5.2 Certificates and Restricted Injectivity	135
	5.2.1 Fenchel-Rockafellar Duality	135
	5.2.2 Dual Certificates	136
	5.2.3 Restricted Injectivity	140
	5.3 Uniqueness	141
	5.3.1 Sublevel Set and its Cones	141
	5.3.2 The Strong Nullspace Property	142
	5.3.3 Topological Conditions	144
	5.4 Construction of Non-Degenerate Certificates	146
	5.4.1 Linearized Precertificate	146
	5.4.2 Analysis Precertificate	149

Relation to Previous Works 159

	Contents
	6.1 Bregman Rate	154
	6.2 Linear Convergence Rate	156
	6.2.1 Main Result	156
	6.2.2 Proof of Theorem 6.1	158
	6.3

, the authors look after ∃ω s.t. Dω ∈ Im Φ * and ω ∈ ∂J 0 (D * x), and its non-degenerate version, which implies the source condition.

Contents 7.1 Selection Against Small Noise: General Case 164

	7.1.1 Sensitivity of the Lagrangian Problem	164
	7.1.2 Main Result	165
	7.1.3 Proof of Theorem 7.2	167
	7.1.4 Proof of Proposition 7.1	171
	7.2 Selection of Linear Manifold	172
	7.2.1 Main Result	172
	7.2.2 Proof of Theorem 7.3	173
	7.3

Relation to Previous Works 179

Relation to Previous Works 199

	Contents
	8.1 Main Assumptions	187
	8.1.1 Assumptions on the Regularizer	187
	8.1.2 Assumption on the Data Fidelity	188
	8.1.3 Restriction and Second-Order Derivative of the Reg-
	ularizer	189
	8.2 Local Behavior of a Solution Mapping	190
	8.2.1 Restricted Injectivity	190
	8.2.2 Transition Space	191
	III 8.2.3 Main Result	192
	8.3 Local Behavior of the Prediction Mapping	196
	8.3.1 Single-Valued Prediction Mapping	197
	8.3.2 Well-Posedness of the Restricted Injectivity Condition 197 Sensitivity 8.3.3 Main Result 199
	8.4
	183

  asserts that, if y → µ(y) is weakly differentiable function (i.e. typically in a Sobolev space over an open subset of R n ), such that each coordinate y → µ i (y) ∈ R has an essentially bounded weak derivative 1

  allows us to define an unbiased estimate of the degrees of freedom, which is computable in closed form. By Proposition 9.2, µ is Lipschitz continuous. From(Evans et al. 1992, Theorem 5, Section 4.2.3), weak differentiability follows. Rademacher theorem asserts that a Lipschitz continuous function is differentiable Lebesgue a.e. and its derivative and weak derivative coincide Lebesgue a.e.,[START_REF] Evans | Measure theory and fine properties of functions[END_REF], Theorem 2, Section 6.2). Its weak derivative, whenever it exists, is upper-bounded by the Lipschitz constant. Thus

	Theorem 9 .2 Suppose that conditions (C O ), (C τ ) and (C L ) hold. Then,
	df(y) = tr ∆(y) a.e.,
	where ∆ is defined in (8.5). Hence, tr ∆(y) is an unbiased estimate of df(y).
	p ro o f E

  10.2.4 Fused Lasso Compressed Sensing Fused Lasso was introduced in Tibshirani et al. 2005. It corresponds to taking D = D DIF εId , and J = ||D * • || 1 in (P y,λ ), where ε > 0

+ λ||α|| 1 . (1.[START_REF] Vaiter | Robust Polyhedral Regularization[END_REF]

The meaning of sensitivity is different here from what is usually intended in satistical sensitivity and uncertainty analysis.

To be understood here as a set-valued mapping.

We write the same symbol as for the derivative, and rigorously speaking, this has to be understood to hold Lebesgue-a.e.

Any other norm has its own subscript in this manuscrit.

The objective function is the function to be optimized.

The boundness assumption does not play any role in section. It will however be important in our results in

Again, we could define this notion without assumption of boundness from below.

Note that here the parameter λ is absorbed within the fidelity term F.

Strictly speaking, the minimization may have to be over a convex subset of R n .

Recall that e(x ⋆ ) is always different from the origin unless x ⋆ = 0.

9.4 Relation to Previous Works

9.4.1 Degrees of Freedom

9.4.2 Generalized Stein Unbiased Risk Estimator

We write the same symbol as for the derivative, and rigorously speaking, this has to be understood to hold Lebesgue-a.e.

σ 2 tr (Φ * Φ) -1 .

Remerciements

3

Model Tangent Subpace Main contributions of this chapter

• Introduction of the model tangent space and model vector in Definition 3.1 and the subdifferential gauge in Definition 3.2.

• Theorem 3.1 provides a pointwise decomposition of the subdifferential of any function in Γ + c (R n ).

5

Certificates and Uniqueness

Main contributions of this chapter

• Introduction of (non-degenerate) dual certificates (Definition 5.1), minimal norm certificate (Definition 5.2), linearized precertificate (Definition 5.4) and its associated identifiability criterion (Definition 5.5).

• Introduction of the restricted injectivity assumption (Definition 5.3).

• Theorem 5.3 gives a sufficient condition for uniqueness for (P y,λ ) or (P y,0 ).

Part II Robustness 151 6

Noise ℓ 2 Robustness Main contributions of this chapter

• Theorem 6.1 shows that if both the non-degenerate source condition and the restricted injectivity hold, then (P y,λ ) enjoys a linear convergence rate with respect to the estimation error.

7

Model Selection

Main contributions of this chapter • Theorem 7.2 ensures that for a partly smooth function J, if the restricted injectivity holds and that the linearized precertificate is a non-degenerate certificate, then for a certain regime of small noise, (P y,λ ) has a unique solution which belongs in the model manifold of x 0 .

• Theorem 7.3 proves a similar result for partly smooth functions with linear manifold with explicit constants.

Local Differentiability of the Optimal Solutions

Main contributions of this chapter

• Theorem 8.1 constructs a smooth map of solutions to (P F y,λ ) on an open neighborhood of some solution x ⋆ , and computes its derivative.

• Theorem 8.2 shows that the prediction map is well-defined outside the transition space and gives its derivative.

Restriction and Second-Order Derivative of the Regularizer

For a subspace T ⊂ R n , and any function g ∈ C 2 (T × R q ), we denote

which can be understood as the Hessian of the mapping x ∈ T → g(x, y), i.e. the restriction of g(•, y) to T . Of course, when T is the whole space, we recover the "full" Hessian.

We also denote D 2 12 g(x, y) the Jacobian of the mapping y ∈ R q → ∇ 1 g(x, y) with respect to y, and ∇ 1 g(x, y) is the gradient of g w.r.t the first variable at (x, y).

We denote

the restriction of J to T for some subspace T ⊂ T. Hence the hessian of J T is well-defined on T. Observe that ∇J T (x) = e x for x ∈ T . We illustrate this definition on several examples.

Lasso and general Lasso.

and thus, D 2 J T (x T ) = 0. This is also the case for the analysis ℓ 1 -penalty (general Lasso), see for instance [START_REF] Vaiter | Local behavior of sparse analysis regularization: Applications to risk estimation[END_REF]. This property basically reflects the fact that these regularizers are polyhedral, hence piecewise affine.

Group Lasso. For J = || • || 1,2 as defined in (1.13), we have

Single-Valued Prediction Mapping

Under this condition, the following immediate lemma gives a convenient re-writing of condition (C x ⋆ ,y ).

L emma 8 .2 We assume that condition (C strict ) holds. For x ∈ R n , and T = T x , the two following conditions are equivalent.

Furthermore, if x ⋆ 0 and x ⋆ 1 are two solutions of P(y), then Φx ⋆ 0 = Φx ⋆ 1 .

p ro o f The first part of the lemma come from the following equivalent statements.

The second part is contained in Proposition 2.2.

This lemma allows us to define the prediction µ :

without ambiguity given any solution x ⋆ (y), which in turn defines a singlevalued mapping µ.

Well-Posedness of the Restricted Injectivity Condition

The following lemma proves that (C x ⋆ ,y ) is not restrictive, and in particular, there exists always a solution x ⋆ such that it holds.

Main Result

The following theorem provides a closed-form expression of the local variations of µ(y) with respect to the observation y.

Theorem 8 .2 We assume that condition (C strict ) holds. The mapping y → µ(y) is C 1 (R n \ H). For all y ∈ H, there exists a solution x ⋆ of (P F y ) such that (C x ⋆ ,y ) is satisfied. Moreover, for all y ∈ H, D µ(y) = ∆(y) (8.5) where

where x ⋆ is any solution of (P F y ) such that (C x ⋆ ,y ) holds and T = T x ⋆ .

p ro o f We can now prove Theorem 8.2. At any y / ∈ H, using the previous Lemma 8. 3 we consider x ⋆ a solution of (P F y ) such that (C x ⋆ ,y ) holds. According to Theorem 8.1, one can construct a mapping x( ȳ) which coincides with x ⋆ at y, and is C 1 for ȳ in a neighborhood of y. Since µ( ȳ) = Φx( ȳ) on this neighborhood, this shows that µ is in turn C 1 at y, and its divergence is equal to tr(∂ y µ(y)). Note that this shows that this computation is independent of the particular choice of x ⋆ provided that (C x ⋆ ,y ) holds.

Relation to Previous Works

Sensitivity analysis 5 is a major branch of optimization and optimal control theory. Comprehensive monographs on the subject are [START_REF] Bonnans | Perturbation analysis of optimization problems[END_REF][START_REF] Mordukhovich | Sensitivity analysis in nonsmooth optimization[END_REF]). The focus of sensitivity analysis is the dependence and the regularity properties of the optimal solution set and the optimal values 5. The meaning of sensitivity is different here from what is usually intended in statistical sensitivity and uncertainty analysis.

Main contributions of this chapter

• Proposition 9.2 proves that the prediction is Lipschitz continuous with respect to the observation.

• Theorems 9.1 and 9.2 prove that the (G)SURE is an unbiased estimator of the risk for non-linear Gaussian regression and generalized linear model.

family and its density reads

where the canonical parameter vector is the linear predictor Φx 0 . In this case,

, where h i is the inverse of the so-called link function in the language of GLM. Each h i is a monotonic differentiable function, and a typical choice is the canonical link h i = ϕ ′ i , where ϕ ′ i is known to be oneto-one if the family is regular [START_REF] Brown | Fundamentals of Statistical Exponential Families with Applications in Statistical Decision Theory[END_REF]. Well-known examples are the identity link h i (t) = t (Gaussian distribution, linear model), the reciprocal link h i (t) = -1/t (Gamma and exponential distributions), and the logit link

Transformations The second example is where h plays the role of a transformation such as variance-stabilizing transformations (VSTs), symmetrizing transformations, or bias-corrected transformations. There is an enormous body of literature on transformations, going back to the early 1940s. A typical example is when Y i are indepedent Poisson random variables ∼ P ((Φx 0 ) i ), in which case h i takes the form of the Anscombe bias-corrected VST. See (Das-Gupta 2008, Chapter 4) for a comprehensive treatment and more examples.

Degrees of Freedom and Unbiased Risk Estimation

The degrees of freedom (DOF) of an estimator quantifies the complexity of a statistical modeling procedure [START_REF] Efron | The estimation of prediction error: Covariance penalties and cross-validation[END_REF]. It is at the heart of several risk estimation procedures and thus allows one to perform parameter selection through risk minimization.

In this section, we will assume that F 0 in (8.1) is strictly convex, so that the response (or the prediction) µ(y) = Φx ⋆ (y) is uniquely defined as a singlevalued mapping of y (see Lemma 8.2). That is, it does not depend on a particular choice of solution x ⋆ (y) of (P F y ). More generally, the degrees of freedom could be defined for any estimator of the prediction. Let µ 0 = Φx 0 .

Prediction, Projected and Estimation Risk

All estimators of the form GSURE B with B such that BΦ = AΦ share the same expectation given by Theorem 9.1. Hence, there are several ways to estimate the risk in reconstructing Aµ 0 . For the estimation of the prediction, projection and estimation risks, we now give the corresponding expressions and associated estimators (with subscript notations) as direct consequences of Theorem 9.1:

which provides an unbiased estimate of the prediction risk

This coincides with the classical SURE.

∂y .

It provides an unbiased estimate of the projection risk

If Φ is the synthesis operator of a Parseval tight frame, i.e. ΦΦ * = Id, the projection risk coincides with the prediction risk and so do the corresponding GSURE estimates

Section 2.4 argues that this condition is not restrictive.

We assume in this section that F takes the form (8.1) and that

and

Obviously, assumption (C τ ) implies (C strict ), and thus the claims of the previous section remain true. Moreover, this assumption holds for the squared loss, but also for some losses of the exponential family (8.2), possibly adding a small quadratic term in β. As far as assumption (C L ) is concerned, it is easy to check that it is fulfilled with L = 1 for any loss of the exponential family (8.2), since D 2 12 F 0 (µ, y) = Id.

The Transition Space has Zero-Measure

P roposi ti on 9 .1 Suppose that conditions (C O ) and (C T ) hold. Then, H is of Lebesgue measure zero.

p ro o f We obtain this assertion by proving that all H T are of zero measure for all T and that the union is over a finite set, because of (C T ). Let C ⊂ R n be the set whose gauge is J, and C • its polar.

• Since J is definable by (C O ), ∇ 1 F(x, y) is also definable by virtue of Proposition 2.11.

• Given T ∈ T, T is also definable. Indeed, T can be equivalently written

which involves algebraic (in fact linear) sets, whence definability follows after interpreting the logical notations (conjunction and universal quantifiers) in

10

Numerical Considerations

Main contributions of this chapter

• We prove in Theorem 10.1 that under the same hypothesis of non-degeneracy and partial smoothness as those of Theorem 7.2, the forward-backward algorithm identifies the correct manifold after a finite number of steps.

• We discuss in Sections 10.2 and 10.3 how the linearized precertificate behaves in different imaging applications.

• We investigate further the behavior of total variation denoising in Theorem 10.2 and the compressed sensing with ℓ ∞ regularization in Theorem 10.3. T he effective computation of a solution to (P y,λ ) is not the main scope of this thesis 1 . However, it seems important to give insight on how to numerically solve with such a problem in high dimension and we give basics understanding of optimization in order to compute the linearized precertificate p F (see Chapter 5).

Contents

Introduction to Proximal Splitting

Suppose that one seeks solutions of

1. However, we developped a Python module, coined pyprox, available on Github: http: //github.com/svaiter/pyprox including forward-backward, Douglas-Rachford, ADMM and generalized forward-backward [START_REF] Raguet | A Generalized Forward-Backward Splitting[END_REF] algorithms. It was used for the numerical experiments of this chapter. We also refer the reader to the Numerical Tours website http: //www.numerical-tours.com.

(ii) The set of fixed-points x ∈ R n | prox f (x) = x is the set of solutions of (10.1).

(iii) For every u, x ∈ R n , one has

(iv) The Moreau identity is satisfied, i.e. for every x ∈ R n ,

We shall give some examples of proximity operators.

Indicator function. Let C ⊆ R n be a non-empty closed convex set and f = ι C its indicator function. Then, prox f = P C the euclidian projection onto C. Note that when C is a linear subspace, the Moreau decomposition (10.2) reads Id = P T + P T ⊥ which accounts for the decomposition of R n = T + T ⊥ into orthogonal subspaces Quadratic objective. Let f(x) = 1 2 Ax, x + b, x + c be a quadratic function, where A is a positive symmetric matrix. Its proximity operator reads

⊲ backward-step end for algorithm [START_REF] Daubechies | An iterative thresholding algorithm for linear inverse problems with a sparsity constraint[END_REF]) are special cases of Algorithm 1. A typical case of applications of this algorithm is to solve (P y,λ ) when f is smooth loss such as the quadratic loss, and g = J a convex regularizer. Cases where this algorithm can be applied is when

Note however that for more complicated regularizers, for instance J = ||D * • || 1 such as the total variation, it is not possible to compute prox J in closed form, so one needs to use more advanced algorithms.

Douglas-Rachford

The forward-backward algorithm works when one of the two functions is differentiable with a uniformly Lipschitz gradient. Suppose now that f, g ∈ Γ 0 (R n ) such that ri dom f ∩ ri dom g = ∅ and f(x) + g(x) → +∞ when ||x|| → +∞. For any function f in Γ 0 (R n ), we write rprox f (x) = 2 prox f (x)x. The Douglas-Rachford has been introduced by [START_REF] Lions | Splitting Algorithms for the Sum of Two Nonlinear Operators[END_REF], in a special case, and further studied by [START_REF] Eckstein | On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators[END_REF]. The algorithm reads as follows. Every sequence x (k) generated by Algorithm 2 converges to a solution to the

) end for problem (10.3), see for instance [START_REF] Bauschke | Convex analysis and monotone operator theory in Hilbert spaces[END_REF]. A typical example of application is to solve (P y,0 ), where f = ι {x| Φx=y} and g = J.

where ∆ = DD * is a discrete Laplacian operator. This implies that for i 1 < k < i 2 where i 1 , i 2 are consecutive indices of I, m is obtained by linearly interpolating (see Figure 10.1) the values η i 1 and η i 2 , i.e

Hence, if x 0 does not contain a staircase subsignal, one has IC(x 0 ) < 1. On the contrary, if there is i 1 such that s i 1 = s i 2 , where i 1 and i 2 are consecutive indices of I, then for every i 1 < j < i 2 , η j = s i 1 = ±1 which implies that IC(x 0 ) = 1.

This theorem together with Theorem 7.2 shows that if a signal x 0 does not have a staircase subsignal, then TV denoising identifies correctly the jump set when the noise is small. This means that if w is small enough, for λ proportional to the noise level, the TV denoised version of y contains the same jumps as x 0 .

To gain a better understanding of the latter situation, we build an instructive family of signals x 0 for which the IC criterion is equal to 1. It turns out that depending on the structure of the noise w, the D-support of x 0 , supp(D * x 0 ), can be either stably identified or not. For n a multiple of 4, we split {1, . . . , n} into 4 sets l k = {(k -1)M + 1, ..., kM} of cardinality M = n/4. Let 1 l k be the boxcar signal whose support is l k . We define the staircase signal x 0 = -1 l 1 + 1 l 4 degraded by a deterministic noise w of the form w = ε(1 l 3 -1 l 2 ), where ε ∈ R. The observation vector y = x 0 + w reads y = -1 l 1 -ε1 l 2 + ε1 l 3 + 1 l 4 .

Suppose that ε > 0, then the solution x ⋆ λ of P λ (y) is In this section, we exemplify the usefulness of our GSURE estimator, see Definition 9.1, which can serve as a basis for automatically tuning the value of λ in the case of analysis ℓ 1 -sparsity, i.e. J = ||D * • || 1 . This is achieved by computing, from a single realization of the noise w ∼ N(0, σ 2 Id), the parameter λ that minimizes the value of GSURE when solving (P y,λ ) from y = Φx 0 + w for various scenarios on Φ and x 0 . Note that this method can be adapted to other analysis regularizers.

Specializing Theorem 9.1 to this case, we have the following result.

Corollary 10 .1 We assume that the observation model is Y

Ker D * Λ ∩ Ker Φ = {0}.

Computing the GSURE

According to Lemma 8.3, there always exists a solution of (P y,λ ) such that (INJ T ) holds, and this solution can be computed, see [START_REF] Vaiter | Local behavior of sparse analysis regularization: Applications to risk estimation[END_REF] for the analysis ℓ 1 prior. With assumption (INJ T ) at hand, we now define the following matrix whose role will be clarified shortly.

List of Notations

(SC x )

Non-degenerate source condition