Laurent El Gahoui

Alexandre Gramfort

Rémi Gribonval

Zaid Har- Chaoui

Laurent Jacob

Jinzhu Jia

Percy Liang

Vincent Michel

Variable Selection and Sparsity-Inducing Regularizations

. In the remainder of this thesis, we shall focus instead on the concept of regularization, and more precisely, sparsity-inducing regularization. Let us introduce more formally this concept.

Sparsity-Inducing Regularization

 for a thorough description of loss functions).

To express our a priori assumption that the learned vector w should be sparse in order to perform variable selection, we also consider a regularization term Ω : R p → R + , so that our formulation becomes

(1.1)

The scalar λ ≥ 0 is known as the regularization parameter and it controls the effect of Ω, while W ⊆ R p is a convex set that can possibly encode further properties of the problem, such as the non-negativity of the coefficients of w. To promote sparse solutions, Ω should intuitively penalize vectors w that have many nonzero coefficients. Thus, a natural candidate to consider is the ℓ 0 pseudo-norm, that is,

 in the context of least-squares regression). This computational challenge prompts the need for surrogates (or relaxations) of the ℓ 0 pseudo-norm that would not only preserve the desired sparsity-inducing properties, but would also be amenable to optimization.

Sparsity-Inducing Norms and the Example of the ℓ 1 -Norm

In this section, we present one way of reformulating ℓ 0 -based problems via sparsityinducing norms 1 . In particular, we first focus on the ℓ 1 -norm which is the most popular 1. A norm . on a real-valued vector space V is a function V → R+ such that (positive homogeneity) for any (a, v) ∈ R × V, we have av = |a| v , (triangular inequality) for any u, v ∈ V, it holds u + v ≤ u + v , and (separability of points) v = 0 if and only if v = 0 for all v ∈ V.

. Moreover, for this alternating scheme to be provably convergent, the variables (η g )g∈G have to be bounded away from zero, resulting in solutions whose entries may have small values, but not "true" zeros.

Some Ingredients to Study Statistical Properties of Sparse Methods

Chapters 2 and 6 study some statistical properties of sparse structured linear models. This section aims at providing some background material which should be useful in this perspective.

Understanding the Properties of Structured

Sparsity-Inducing Norms

Chapter abstract: We consider the empirical risk minimization problem for linear supervised learning, with regularization by structured sparsity-inducing norms. These are defined as sums of Euclidean norms on certain subsets of variables, extending the usual ℓ 1 -norm and the group ℓ 1norm by allowing the subsets to overlap. This leads to a specific set of allowed nonzero patterns for the solutions of such problems. We first explore the relationship between the groups defining the norm and the resulting nonzero patterns, providing both forward and backward algorithms to go back and forth from groups to patterns. This allows the design of norms adapted to specific prior knowledge expressed in terms of nonzero patterns. We also present an efficient active set algorithm, and analyze the consistency of variable selection for least-squares linear regression in low and high-dimensional settings.

résumé

De nombreux domaines issus de l'industrie et des sciences appliquées ont été, au cours des dernières années, les témoins d'une révolution numérique. Cette tendance s'est accompagnée d'une croissance continue du volume des données-vidéos, musiques et images, dont le traitement est devenu un véritable défi technique. Par exemple, il est aujourd'hui fréquent de prendre des centaines de photographies de plusieurs millions de pixels, la moindre application de méthodes du traitement de l'image devenant alors une opération difficile. Dans ce contexte, la parcimonie est apparue comme un concept central en apprentissage statistique et traitement du signal. Il est en effet naturel de représenter, analyser et exploiter les données disponibles à travers un nombre réduit de paramètres. Par exemple, on peut imaginer effectuer de la reconnaissance d'objets sur des images de hautes résolutions en n'utilisant qu'un petit sous-ensemble pertinent de pixels. Alors que les approches générales favorisant la parcimonie ont déjà été l'objet de nombreux travaux-débouchant sur d'élégantes fondations théoriques, des outils algorithmiques efficaces et plusieurs succès pratiques-cette thèse se concentre sur une forme particulière et plus récente de parcimonie, nommée parcimonie structurée.

Comme son nom l'indique, nous considérerons des situations où nous ne serons pas simplement intéréssés par la parcimonie, mais où nous aurons également à disposition des connaissances a priori nous renseignant sur certaines propriétés structurelles. En continuant d'exploiter l'exemple de la reconnaissance d'objets mentioné ci-dessus, nous savons que des pixels voisins sur une image ont tendance à partager des propriétés similaires, telles que la classe de l'objet à laquelle ils appartiennent. Ainsi, une approche encourageant la parcimonie devrait tirer partie de cette information spatiale.

L'objectif de cette thèse est de comprendre et analyser le concept de parcimonie structurée, en se basant sur des considérations statistiques, algorithmiques et appliquées. Nous commencerons par introduire une famille de normes structurées parcimonieuses dont les propriétés sont étudiées en détail. En particulier, nous montrerons à quel type d'information structurelle ces normes correspondent, et nous présenterons sous quelles conditions statistiques elles sont capables de produire une séléction consistente de variables. Nous étudierons ensuite l'apprentissage de dictionnaires parcimonieux et structurés, où nous exploiterons les normes introduites précédemment dans un cadre de factorisation de matrices. L'approche qui en résulte est fléxible et versatile, et nous montrerons que les éléments de dictionnaire appris exhibent une structure parcimonieuse adaptée à la classe de signaux considérée. Concernant l'optimisation, nous proposerons différents outils algorithmiques efficaces et capables de passer à l'échelle, tels que des stratégies à ensemble de variables actives ou encore des méthodes proximales. Grâce à ces outils algorithmiques, nous illustrerons sur de nombreuses applications issues de domaines variés, quand et pourquoi la parcimonie structurée peut être bénéfique. Ces illustrations contiennent par exemple, des tâches de restauration en traitement de l'image, la modélisation de documents textuels sous la forme d'une hiérarchie de thèmes, la prédiction de la taille d'objets à partir de signaux d'imagerie par résonance magnétique fonctionnelle, ou encore des problèmes de segmentation d'images en vision par ordinateur.

iii introduction et contributions de la thèse Une grande variété de problèmes en apprentissage statistique peuvent se résumer par l'apprentissage, à partir de données, d'un jeu de paramètres maximisant un critère préétabli, d'une manière soit supervisée ou non supervisée. De tels problèmes apparaissent par exemple lorsqu'on désire expliquer un ensemble de réponses à partir de certaines observations, ou lorsqu'on souhaite résumer, organiser, ou compresser un grand volume de données. Dans tous les cas, et de manière similaire à d'autres domaines des sciences appliquées, une solution simple, parcimonieuse, est souvent privilégiée face à d'autres alternatives plus "complexes". Ce biais en faveur de la parcimonie peut être justifié par deux arguments : On peut soit croire a priori que le phénomène étudié admet effectivement une solution parcimonieuse, ou alternativement, et sans cette connaissance a priori, on peut chercher une explication simple parce qu'elle se prête bien à l'interprétation, à la compréhension et conduit à des post-traitements moins coûteux.

En apprentissage statistique, on fait référence aux approches parcimonieuses pour désigner les maniéres de résoudre un problème en n'utilisant qu'un nombre limité de paramètres.

Le thème principal de cette thèse est la parcimonie structurée. Comme son nom l'indique, nous allons nous concentrer sur des configurations où non seulement la parcimonie est pertinente, mais aussi où des connaissances a priori suggèrent que la solution attendue exhibe certaines structures intéressantes.

Des connaissances a priori de nature structurelle peuvent se manifester sous différentes formes, et reflètent par exemple des propriétés ordinales, temporelles, ou encore spatiales du problème étudié. Cette importante idée est probablement plus simple à comprendre à travers une série d'exemples concrets.

Dans le contexte de la vision par ordinateur, la ségmentation consiste à partitioner des images en sous-parties pertientes (e.g., premier plan et arrière plan). Dans ce cas, il semble naturel de prendre en compte le fait que des pixels voisins dans l'image sont susceptibles de partager des attributs similaires. Les résultats de ségmentation sont effectivement améliorés grâce à cette information spatiale (e.g., voir [START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF]. Pour l'étude de séries temporelles non-alignées d'expression de gênes, l'ordonnancement dans le temps des motifs de régulation (e.g., motifs montants et descendants) jouent un rôle majeur. Comme précédemment, modéliser cette information ordinale conduit à de meilleures performances [START_REF] Tibau Puig | Order-preserving factor analysis-application to longitudinal gene expression[END_REF]. De même, pour le traitement du langage naturel, les données sont générallement représentées par des motifs qui, par exemple, peuvent désigner des pairs de mots contigus labélisés par leurs classes syntaxiques. De tels motifs peuvent être combinés ensemble pour alors former de nouveaux motifs plus complexes. Dans un cadre parcimonieux, il semble naturel de considérer d'abord les paramètres associés aux motifs élémentaires avant d'analyser des motifs plus sophistiqués. En pratique, cette approche hiérarchique se révèle être performante [START_REF] Martins | Structured sparsity in structured prediction[END_REF]. v Toutes les illustrations mentionées ci-dessus constituent des exemples où la parcimonie structurées est bénéfique. Dans cette thèse, nous essayons de comprendre ce concept que nous étudions sous de divers angles, algorithmique, statistique et appliqué.

Résumé des Contributions de la Thèse

Nous listons ci-dessous les contributions qui émergent de la thèse :

• Chapter 2 : Ce chapitre introduit et étudie précisemment le principal objet de cette thèse, à savoir, les normes encourageant la parcimonie structurée. Ces dernières sont définies comme combinaisons linéaires de normes élémentaires construites à partir de groupes de variables. Nous considérons l'ensemble de tous les groupes possibles et charactérisons exactement quel type de connaissances a priori elles sont capables d'encoder. Avant de décrire comment on peut passer des groupes à l'ensemble des motifs parcimonieux induits, nous montrons qu'il est possible de construire automatiquement une unique norme minimale, adaptée à la famille de motifs parcimonieux désirés, comme par exemple, l'ensemble des rectangles sur une grille en deux dimensions. Par ailleurs, nous étudions sous quelles conditions statistiques, ces normes permettent d'estimer de manière consistente un motif parcimonieux structuré, aussi bien en faible qu'en haute dimensions. Enfin, nous introduisons un algorithmique efficace et rapide basé sur la construction d'un ensemble de variables actives. • Chapter 3 : Cette seconde partie est dédiée à l'apprentissage de dictionnaires structurés, exploitant et étendant le schéma de régularisation introduit au Chapitre 2.

Nous montrons comment on peut apprendre des dictionnaires dont les atomes ont des motifs parcimonieux structurés qui sont adaptés à la classe de signaux considérés. Pour cela, nous présentons une technique d'optimisation simple et efficace qui reposent sur des méthodes de descente "par bloque" disposant de mises à jour explicites. On applique finalement notre outil à un problème de reconnaissance de visages, où nous sommes capables d'apprendre des déscripteurs robustes aux occlusions. • Chapter 4 : Ce chapitre s'intéresse à des méthodes algorithmiques efficaces pour résoudre des problèmes de parcimonie structurée où nous supposons que les variables possèdent une structure hiérarchique. Plus précisemment, nous considérons les méthodes proximales pour lesquelles nous montrons que l'opérateur proximal associé aux normes hiérarchiques peut être calculé exactement via une approche duale. Notre procédure a une complexité linéaire, ou presque linéaire, par rapport au nombre d'atomes, ce qui rend l'application des méthodes proximales accélérées rapides et efficaces. Nous illustrons notre approche par des applications variées, telles que le débruitage d'images naturelles, ou encore la représentation de documents textuels sous forme de hiérarchies de thématiques, établissant par la même occasion des liens avec les modèles probabilistiques utilisés dans ce même cadre. • Chapter 5 : Ce quatrième chapitre considère deux applications de la parcimonie structurée à la neuroimagerie. vi D'une part, nous nous intéressons à la prédiction de la taille d'objets à travers dix sujets, en nous basant sur des signaux d'imagerie par résonance magnétique fonctionelle (IRMf). Pour cela, nous introduisons une régularisation hiérarchique structurée construite à partir des données d'entrainement, via une procédure non supervisée exploitant des contraintes spatiales spécifiques. Cette construction permet de prendre en compte la structure spatiale multi-échelle des signaux IRMf, augmentant ainsi la robustesse aux variabilitées inter-sujets. Nous réalisons une comparaison expérimentale impliquant plusieurs algorithmes et formulations, et nous illustrons la capacité de notre approche à localiser des régions du cerveau dédiées au traitement des stimulis visuels. D'autre part, nous introduisons un modèle génératif pour étudier des séries temporelles du cerveau au repos. Cette classe de signaux est modélisée grâce à l'apprentissage de dictionnaires structurés où les atomes que nous apprenons présentent des formes compactes et localisées en trois dimensions. De plus, notre approche se prête naturellement à la séléction de modèles et à une évaluation quantitative ; dans ce cadre, nous obtenons des améliorations par rapport aux techniques non structurées, mesurées par une vraisemblance sur des données de test. • Chapter 6 : Le cinquième et dernier chapitre de la thèse se situe légèrement en marge de la thématique principale de parcimonie structurée. L'objectif de ce partie est de caractériser les minima locaux du problème (non convexe) de l'apprentissage de dictionnaires parcimonieux, utilisé par exemple dans des extensions structurées aux Chapitres 3 et 4. En particulier, nous considérons un modèle probabiliste de signaux parcimonieux, et nous montrons qu'avec forte probabilité, le problème de l'apprentissage de dictionnaires parcimonieux admet un minimum local suivant certaines courbes passant par le dictionnaire de référence ayant généré les signaux. Notre analyse couvre les cas des dictionnaires redondants et des signaux bruités, étendant ainsi les travaux précédants limités à une configuration sans bruit. L'étude que nous réalisons est non asymptotique et permet de mieux comprendre comment les grandeurs clé du problème, telles que la cohérence ou le bruit, peuvent varier avec la dimension des signaux, le nombre d'observations, le niveau de sparsité et le nombre d'atomes. Ce travail en cours constitue un premier pas vers la preuve plus complexe de l'existence d'un minimum local dans l'ensemble de la variété des dictionnaires normalisés.

vii abstract Numerous fields of applied sciences and industries have been witnessing a process of digitisation over the past few years. This trend has come with a steady increase in the amount of available digital data whose processing was become a challenging task. For instance, it is nowadays common to take thousands of pictures of several millions of pixels, which makes any subsequent image-processing/computer-vision task a computationally demanding exercise.

In this context, parsimony-also known as sparsity-has emerged as a key concept in machine learning, statistics and signal processing. It is indeed appealing to represent, analyze, and exploit data through a reduced number of parameters, e.g., performing object recognition over high-resolution images based only on some relevant subsets of pixels. While general sparsity-inducing approaches have already been well-studied-with elegant theoretical foundations, efficient algorithmic tools and successful applications, this thesis focuses on a particular and more recent form of sparsity, referred to as structured sparsity.

As its name indicates, we shall consider situations where we are not only interested in sparsity, but where some structural prior knowledge is also available. Continuing the example of object recognition, we know that neighbouring pixels on images tend to share similar properties-e.g., the label of the object class to which they belong-so that sparsity-inducing approaches should take advantage of this spatial information.

The goal of this thesis is to understand and analyze the concept of structured sparsity, based on statistical, algorithmic and applied considerations. To begin with, we introduce a family of structured sparsity-inducing norms whose properties are closely studied. In particular, we show what type of structural prior knowledge they correspond to, and we present the statistical conditions under which these norms are capable of consistently performing structured variable selection. We then turn to the study of sparse structured dictionary learning, where we use the aforementioned norms within the framework of matrix factorization. The resulting approach is flexible and versatile, and it is shown to learn representations whose structured sparsity patterns are adapted to the considered class of signals. From an optimization viewpoint, we derive several efficient and scalable algorithmic tools, such as, working-set strategies and proximal-gradient techniques. With these methods in place, we illustrate on numerous real-world applications from various fields, when and why structured sparsity is useful. This includes, for instance, restoration tasks in image processing, the modelling of text documents as hierarchy of topics, the inter-subject prediction of sizes of objects from fMRI signals, and background-subtraction problems in computer vision. ix acknowledgments It is commonly said that superlatives are inappropriate for scientific writing; in this section, I will have to violate this rule.

The second sentence of these acknowledgements is naturally dedicated to my advisors Jean-Yves Audibert and Francis Bach. I would like to deeply thank them for all their help, support, kindness and enthusiasm over the past three years. It is an invaluable privilege to have had the opportunity to work with them.

I would like to warmly thank Laurent El Ghaoui and Massimiliano Pontil for having reviewed my manuscript. I am also grateful to Rémi Gribonval, Eric Moulines, Guillaume Obozinski, and Bertrand Thirion for having accepted to be part of the jury.

Back in 2008 and while leaving the finance industry and arriving in the Willow (and later, Sierra) Project team, I could not expect such an exceptional environment. Jean Ponce along with Francis Bach have been succeeding in creating great teams with a unique atmosphere of work and with worldwide synergies. It is hard to describe how nice it is to be only surrounded by extremely friendly and smart colleagues, all at the cutting edge in their respective fields of research. I therefore want to thank (in alphabetical order), Sylvain Arlot (for all my annoying questions about concentration of measure), Louise Benoit, Y-Lan Boureau, Florent Couzinié-Devy, Vincent Delaitre, Olivier Duchenne, Cécile Espiègle, Loïc Février, Jan Van Gemert, Edouard Grave, Warith Harchaoui, Toby Hocking, Armand Joulin (for all my measure-theoretic questions that he always considered in a good mood), Ivan Laptev, Augustin Lefèvre (for the pleasure it was to share his office, and for his patience regarding my numerous questions), Nicolas Le Roux, Bryan Russell, Mark Schmidt, Josef Sivic, Mathieu Solnon, and Oliver Whyte.

I also have a special thank for my two good friends, Julien Mairal and Guillaume Obozinski, with whom I closely worked and collaborated. They have this rare ability to convey their passions for research through their charisma and enthusiasm. They have really been role models to look up to, and I am more than happy to see that they are becoming the next worldwide experts in their fields.

Introduction and Related Work

A large variety of problems in machine learning and computational statistics amount to learning, from some data, a set of parameters that maximizes a predefined criterion, in either a supervised or unsupervised fashion. Such problems for instance arise when we desire to relate some input observations to some output response, or when we simply wish to summarize, organize or compress large amounts of data. In any case, and similarly to other fields of applied sciences, a simple, parsimonious solution is often preferred over more "complex" ones. This bias towards parsimony can be underpined in two ways: First, we may a priori believe that the studied phenomenon has indeed a parsimonious solution, or second, and without this prior knowledge, we may seek a simple explanation because it lends itself well to understanding, interpretability and less processing.

In machine learning and statistics, parsimony is also known as sparsity, and, in a broad sense, we shall refer to sparsity-inducing approaches as ways of tackling problems by only resorting to a reduced number of parameters.

The central theme of this thesis is structured sparsity. As its name indicates, we shall concentrate on settings where not only sparsity is relevant, but also where prior knowledge suggests that the expected solution exhibits some structure of interest.

Structural prior knowledge can come up under various forms and may for instance reflect ordering, spatial or temporal properties of the problem at hand. This important idea is probably easier to understand through concrete examples.

In the context of computer vision, segmentation consists in partitioning images into meaningful parts (e.g., background versus foreground). In this case, it seems natural to take into account the fact that neighboring pixels on the image are likely to share similar attributes. Better segmentation results are indeed obtained thanks to this spatial prior knowledge (e.g., see Boykov et al., 2001, and references therein). While studying misaligned time series of gene expressions across different subjects, the temporal ordering of the regulation patterns (e.g., up-and down-regulations) plays a prominent part. Again, modeling this ordinal information leads to improved performance [START_REF] Tibau Puig | Order-preserving factor analysis-application to longitudinal gene expression[END_REF]. In natural language processing, the features associated with the data are usually built from templates which may, for instance, represent pairs of contiguous words labeled by their part-of-speech tags. Such templates can be combined with each other, hence forming more complex features. In this context, it seems natural to first consider parameters relative to elementary templates before looking at more sophisticated ones; this hierarchical approach turns out to be powerful in practice [START_REF] Martins | Structured sparsity in structured prediction[END_REF].

Introduction and Related Work

All the aforementioned illustrations constitute examples where structured sparsity is beneficial. In this thesis, we try to understand this concept and we thoroughly study it from various viewpoints-algorithmic, statistical as well as applied.

The remainder of the introduction is organized as follows: We first summarize the contributions made by this thesis before introducing more formally the framework of structured sparsity and the key object in our analysis, namely structured sparsityinducing norms. We then present some background material about dictionary learning and its variants, which are techniques used throughout the manuscript. Finally, since most chapters make heavy use of concepts from convex optimization and/or from statistical learning theory, we dedicate two sections of this introduction to present the necessary material.

Summary of the Contributions of the Thesis

We list the contributions that emerge from this thesis: • Chapter 2: This chapter introduces and studies in details the core object of this thesis, namely, structured sparsity-inducing norms. These are defined as linear combinations of norms built over groups of variables. We consider all possible sets of groups and characterize exactly what type of prior knowledge can be encoded by considering families of overlapping groups of variables. Before describing how to go from groups to nonzero patterns, we show that it is possible to "reverseengineer" a given set of nonzero patterns, i.e., to build the unique minimal set of groups that will generate these patterns. This allows the automatic design of sparsity-inducing norms, adapted to target sparsity patterns such as rectangles on a two-dimensional grid. We moreover study under which conditions we can obtain a consistent estimation of structured nonzero patterns, both in low-and high-dimensional settings. Finally, we introduce an efficient and scalable active-set algorithm. • Chapter 3: This second part is dedicated to sparse structured dictionary-learning, leveraging and extending the regularization scheme introduced in Chapter 2. We show how we can learn dictionaries whose atoms have structured nonzero patterns adapted to the class of signals considered. To this end, we propose an efficient and simple optimization technique based on nested block-coordinate descents with closed-form updates. We eventually apply our unsupervised method to face recognition, where we learn localized features robust to occlusions. • Chapter 4: This chapter focuses on designing efficient algorithmic tools to solve sparse decomposition problems where we assume some hierarchical structure among the variables. Specifically, we resort to proximal methods for which we show that the proximal operator associated with the hierarchical norm is computable exactly via a dual approach. Our procedure has a complexity linear, or close to linear, in the number of atoms, which makes accelerated proximalgradient techniques highly scalable and efficient. We illustrate our method on various problems, such as the denoising of natural image patches and the repre-1.2. Notation sentation of text documents over hierarchies of topics, thus establishing interesting connections with probabilistic topic models. • Chapter 5: This fourth section concentrates on two applications of structured sparsity to neuroimaging.

On the one hand, we consider the prediction of sizes of objects across ten subjects based on data investigating object coding in high-level visual cortex. To this end, we introduce a sparse hierarchical structured regularization built in a data-driven fashion, from a spatially-constrained agglomerative clustering. This construction makes it possible to take into account the multi-scale spatial structure of fMRI data, thus resulting in more robustness to inter-subject variability. We conduct an experimental comparison of several algorithms and formulations and illustrate the ability of the proposed method to localize in space and in scale some brain regions involved in the processing of visual stimuli.

On the other hand, we introduce a generative model to study brain resting-state time series. This class of signals is modeled by sparse structured dictionary learning where we learn atoms that exhibit three-dimensional localized clusters. Moreover, the proposed approach provides a natural framework for model selection and quantitative evaluation, where we obtain improvements over unstructured methods, as measured by some likelihood on held-out data. • Chapter 6: The fifth and last chapter of this thesis is slightly next to the scope of the main theme developed so far, namely, structured sparsity. In this part, we aim at characterizing the local minima of sparse coding, also known as sparse dictionary learning (see Chapters 3 and 4 for some structured extensions) which relies on a non-convex procedure whose local minima have not been fully analyzed yet. To this end, we consider a probabilistic model of sparse signals, and show that, with high probability, sparse coding admits a local minimum along some curves passing through the reference dictionary generating the signals.

Our study takes into account the case of over-complete dictionaries and noisy signals, thus extending previous work limited to noiseless settings and/or undercomplete dictionaries. The analysis we conduct is non-asymptotic and makes it possible to understand how the key quantities of the problem, such as the coherence or the level of noise, are allowed to scale with respect to the dimension of the signals, the number of atoms, the sparsity and the number of observations. This work in progress constitutes a first step towards the more involved proof of the existence of the local minimum over the entire manifold of normalized dictionaries.

Notation

Throughout the manuscript, we shall refer to vectors as bold lower case (possibly Greek) letters, and matrices by bold upper case ones. For any integer j in the set 1; p {1, . . . , p}, we denote the j-th coefficient of a p-dimensional vector w ∈ R p by w j . Similarly, for any matrix W ∈ R n×p , we refer to the entry at the i-th row and j-th 1. Introduction and Related Work column as W ij , for any (i, j) ∈ 1; n × 1; p . We will also have to describe sub-vectors of w ∈ R p , that is, for any J ⊆ 1; p , we denote by w J ∈ R |J| the vector formed by the entries of w indexed by J. Likewise, for any I ⊆ 1; n , J ⊆ 1; p , we denote by W IJ ∈ R |I|×|J| the sub-matrix of W formed by the rows (respectively, the columns) indexed by I (respectively by J).

We extensively manipulate norms in this thesis. We thus define the ℓ q -norm for any vector w ∈ R p by w q p j=1 |w j | q 1/q for q ∈ [1, ∞), and w ∞ max j∈ 1;p |w j |.

For q ∈ (0, 1), we extend the definition above to ℓ q quasi-norms. In the same vein, for any matrix W ∈ R n×p , we define the Frobenius norm of W by

W F n i=1 p j=1 W 2 ij 1/2
. Also, we refer to the set of nonnegative real numbers as R + {t ∈ R; t ≥ 0}. If need be, additional and more specific notation may be introduced at the beginning of some chapters.

Variable Selection and Sparsity-Inducing Regularizations

In a broad sense, a significant fraction of the work presented in this thesis is about variable selection, also known as feature selection. Throughout this manuscript, we should understand "variable" and "feature" as being a descriptor used to represent the data, for instance, the intensity of a pixel taken from an image or the frequency of a word in a document. Nowadays, data are not only becoming abundant in many scientific and industrial fields (e.g., the web and finance industries), but they are also being made available through richer and more complex representations (e.g., the ever increasing resolution of images).

In this context, variable selection is an essential tool that has primarily three purposes [START_REF] Guyon | An introduction to variable and feature selection[END_REF]: (1) "summarizing" the description of the data to make it more interpretable and understandable, (2) obtaining a more compact and effective representation-e.g., for compression, (3) and finally, concentrating the predictive power of the different features when prediction accuracy matters.

In this thesis, we are interested in these three aspects, laying the emphasis on the first and last points. More concretely, variable selection shall amount to finding a subset of relevant features among a total of p variables, that is, learning a sparse vector of parameters w in R p whose set of nonzero coefficients characterizes the corresponding set of selected features. Note that the formulations of the underlying learning problems will be made more formal in the subsequent sections. There exist numerous ways of 1. Introduction and Related Work example of such reformulations, and which makes it possible to understand the key properties common to more general sparsity-inducing norms.

Regularizing by the ℓ 1 -norm has been a topic of intensive research over the last decade. This line of work has witnessed the development of nice theoretical frameworks [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF][START_REF] Chen | Atomic decomposition by basis pursuit[END_REF][START_REF] Mallat | A wavelet tour of signal processing[END_REF][START_REF] Tropp | Greed is good: Algorithmic results for sparse approximation[END_REF][START_REF] Tropp | Just relax: Convex programming methods for identifying sparse signals in noise[END_REF][START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF][START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF][START_REF] Zhang | Some sharp performance bounds for least squares regression with l1 regularization[END_REF] and the emergence of many efficient algorithmic tools [START_REF] Efron | Least angle regression[END_REF][START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Friedman | Pathwise coordinate optimization[END_REF][START_REF] Wu | Coordinate descent algorithms for lasso penalized regression[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Wright | Sparse reconstruction by separable approximation[END_REF][START_REF] Needell | CoSaMP: Iterative signal recovery from incomplete and inaccurate samples[END_REF][START_REF] Yuan | Comparison of optimization methods and software for large-scale l1-regularized linear classification[END_REF]. Moreover, various applications were considered, for instance, in compressed sensing [START_REF] Candes | Decoding by linear programming[END_REF], for the structure estimation of graphical models [START_REF] Meinshausen | High-dimensional graphs and variable selection with the lasso[END_REF] and for several reconstruction tasks involving natural images (e.g., see [START_REF]Sparse coding for machine learning, image processing and computer vision[END_REF], for a review).

Within the context of least-squares regression, ℓ 1 -norm regularization is known as Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] in statistics and basis pursuit in signal processing [START_REF] Chen | Atomic decomposition by basis pursuit[END_REF]. Formulation (1.1) thus reduces to min

w∈R p 1 2 y -Xw 2 2 + λ w 1 (1.2)
in the Lasso case, and becomes min

α∈R p 1 2 x -Dα 2 2 + λ α 1 (1.3)
for basis pursuit. Though similar from an optimization viewpoint, we have explicitly written down the two formulations (1.2) and (1.3) to point out the conceptual differences there exist in statistics and signal processing settings. On the one hand, we use X ∈ R n×p to denote a set of n observations described by p variables, while y ∈ R n represents the corresponding set of n targets that we try to predict. For instance, y may have discrete entries in the context of classification. On the other hand, and for basis pursuit, we consider a m-dimensional signal x ∈ R m that we express as a linear combination of p dictionary elements composing the dictionary D [d 1 , . . . , d p ] ∈ R m×p . While the matrix X is assumed fixed and given beforehand, we shall see in Section 1.4 that the dictionary may correspond to either some pre-defined basis (e.g., see Mallat, 1999, for wavelet basis) or to some learned representations [START_REF] Olshausen | Emergence of simple-cell receptive field properties by learning a sparse code for natural images[END_REF]. To avoid confusion, we will try to respect as much as possible this notation throughout the manuscript.

Since the ℓ 1 -norm is convex (as any norm), it is possible to design tractable algorithms to solve (1.1) in this case. We shall review some of these methods in Section 1.5. Nonetheless, we have not justified yet why the ℓ 1 -norm is supposed to have the same sparsity-inducing behavior as the ℓ 0 pseudo-norm. While a formal answer will be provided in Section 1.5 based on adapted optimization tools, we first give "intuitive" arguments.

Variable Selection and Sparsity-Inducing Regularizations

One-Dimensional Solution to Lasso: Soft-Thresholding Operator

In order to understand why the ℓ 1 -regularization promotes sparse solutions, it is convenient to consider the Lasso formulation (1.2) when the data matrix X is orthogonal, in which case the minimization decouples into p independent one-dimensional problems of the form min w∈R 1 2 (uw) 2 + λ|w| for some real number u ∈ R.

(1.4)

Although the function w → φ u,λ (w) 1 2 (uw) 2 + λ|w| is convex, its minimization is not straightforward since it is non-smooth, with the presence of the absolute value. If the minimum does not happen at zero, we can take the derivative of φ u,λ and obtain a closed-form solution for the minimizer. On the other hand, if the function φ u,λ has its smallest value at zero, its left/right derivative should be positive to guarantee first order optimality conditions. These facts can be summarized in the following statement, that is arg min w∈R 1 2 (uw) 2 + λ|w| = sign(u) max{0, |u| -λ}.

As a matter of fact, the minimizer of (1.4) is well-known and corresponds to the softthresholding operator introduced by [START_REF] Donoho | Adapting to unknown smoothness via wavelet shrinkage[END_REF]. We shall discuss at greater length in Section 1.5 why this operator is important and how it constitutes a building block of efficient algorithms. It is interesting to see how the solution ŵ varies with respect to the regularization parameter. As long as the absolute value |u| is smaller than λ, the solution ŵ remains equal to zero, and sparsity is indeed promoted. Then, when |u| goes beyond the threshold λ, the minimizer becomes equal to a "shifted" version of u.

To understand how the ℓ 1 -norm mimics the effect of the ℓ 0 pseudo-norm, we consider as well the following problem min w∈R 1 2 (uw) 2 + λ1 {w =0} , (1.5)

where the indicator function 1 {w =0} is equal to one if w is nonzero, and zero otherwise.

Again, problem (1.5) can be proved to have a closed-form minimizer whose expression is given by ŵ = u.1 {|u|≥ √ 2λ} . We display on Figure 1.1 the profiles of the minimizers for both (1.4) and (1.5).

In order to gain more insight into the sparsity-inducing behavior of the ℓ 1 -norm, we now consider some intuitive geometrical argument.

Geometrical Intuition Through the ℓ 1 -Norm Ball

Although we consider in (1.1) a regularized formulation, we could equivalently focus on a constrained problem, that is, min w∈R p f (w) such that Ω(w) ≤ µ,

(1.6) for some µ ∈ R + , and where we have assumed for simplicity that W = R p . The set of solutions of (1.6) parametrized by µ is the same as that of (1.1), as described by some value of λ µ depending on µ (e.g., see Section 3.2 in [START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF]. At optimality, the opposite of the gradient of f evaluated at any solution ŵ of (1.6) is known to belong to the normal cone to B = {w ∈ R p ; Ω(w) ≤ µ} at ŵ [START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF]. In other words, for sufficiently small values of µ, i.e., so that the constraint is active, the level set of f for the value f ( ŵ) is tangent to B.

Introduction and Related Work

As a consequence, the geometry of the ball B is directly related to the properties of the solutions ŵ. If Ω is taken to be the ℓ 2 -norm, then the resulting ball B is the standard, isotropic, "round" ball that does not favor any specific direction of the space. On the other hand, when Ω is the ℓ 1 -norm, B corresponds to a diamond-shaped pattern in two dimensions, and to a pyramid in three dimensions. In particular, B is anisotropic and exhibits some singular points due to the non-smoothness of Ω. Moreover, these singular points are located along the axis of R p , so that if the level set of f happens to be tangent at one of those points, sparse solutions are obtained. In order to better understand why such constraints favour solutions located at vertices or some degenerate parts of the domain boundaries, we refer the interested reader to [START_REF] Barvinok | Problems of distance geometry and convex properties of quadratic maps[END_REF]; [START_REF] Pataki | On the rank of extreme matrices in semidefinite programs and the multiplicity of optimal eigenvalues[END_REF] for related results in the context of cone programming and low-rank matrices. We display on Figure 1.2 the balls B for both the ℓ 1 -and ℓ 2 -norms.

After having introduced the ℓ 1 -norm, we now turn to more sophisticated sparsityinducing norms capable of encoding additional information about the data and the problem at hand.

Structured Sparsity-Inducing Norms

We have so far focused on the ℓ 1 -norm as a computationally-tractable surrogate for the ℓ 0 -pseudo-norm. The primary goal of these regularizers is to penalize dense vectors of parameters, that is, those whose number of nonzero coefficients is large. In other words, these regularization schemes only care about cardinality: they treat each variable individually and they are blind to potential relationships that may exist between the features. A simple way of justifying the latter property is to see that for any permutation 1.3. Variable Selection and Sparsity-Inducing Regularizations One objective of this thesis is to come up with sparsity-inducing norms capable of encoding some additional structure about the variables. We shall assume this structural information available and known a priori. In addition, we will loosely speak about structure, without providing with a formal definition of what we mean by this term; instead, our statements will be motivated and illustrated by concrete examples.

Sparsity-Inducing Norms with Non-Overlapping Groups of Variables

Let us introduce a first extension of the ℓ 1 -norm. Assume that we consider twodimensional data-points of the form x = [x 1 , x 2 ], described by two categorical variables taking values in the set {c 1 , c 2 , c 3 }. One way of encoding these unordered categories consists of introducing an augmented space of dummy variables, so that a data-point x can represented as 1} 6 , with x c l 1. Introduction and Related Work scheme as triplets is likely to be lost. It is therefore appealing to keep the tripletstructure while performing variable selection. This setting is actually an example where group sparsity can be beneficial. Let us consider the set G = {{1, 2, 3}, {4, 5, 6}} of subsets of 1; 6 , and introduce the norm Ω(w) = g∈G w g 2 = w 2 1 + w 2 2 + w 2 3 + w 2 4 + w 2 5 + w 2 6 .

x aug = [x c 1 1 , x c 2 1 , x c 3 1 , x c 1 2 , x c 2 2 , x c 3 2 ] ∈ {0,
Intuitively, Ω acts as an ℓ 1 -norm on the vector { w g 2 } g∈G . Regularizing by Ω therefore causes some w g 2 (and equivalently w g ) to be zeroed out for some g in G. Conversely, within each groups of variables g in G, the ℓ 2 -norm does not induce sparsity. As a result, the norm Ω promotes sparsity at the level of the groups of variables g in G. Back to our example, this choice of Ω makes it possible to preserve the triplet-structure of the coding scheme while performing variable selection. We illustrate the properties of such a norm on Figure 1.3-(a) where we display the unit ball associated with Ω, which is to be contrasted with the ℓ 1 -norm ball represented on Figure 1.2. The singular points appearing on these balls describe the sparsity-inducing behavior of the underlying norms Ω.

We have introduced here a specific instance of structured sparsity-sometimes referred to as group sparsity-where structure is understood as non-overlapping block of variables. More generally, if G denotes a partition of 1; p , we can extend the definition of Ω as Ω(w) = g∈G w g q for any q ∈ (1, ∞].

(1.7)

Variable Selection and Sparsity-Inducing Regularizations

Thus defined, Ω is usually referred to as a mixed ℓ 1 /ℓ q -norm, and in practice, popular choices for q are {2, ∞}. In the context of least-squares regression, this regularization is known as group Lasso [START_REF] Turlach | Simultaneous variable selection[END_REF][START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF]. It has be shown to improve the prediction performance and/or interpretability of the learned models when the block structure is relevant [START_REF] Roth | The group-Lasso for generalized linear models: uniqueness of solutions and efficient algorithms[END_REF][START_REF] Stojnic | On the reconstruction of block-sparse signals with an optimal number of measurements[END_REF][START_REF] Lounici | Taking advantage of sparsity in multi-task learning[END_REF][START_REF] Huang | The benefit of group sparsity[END_REF]. Moreover, applications of this regularization scheme arose in the context of multi-task learning [START_REF] Obozinski | Joint covariate selection and joint subspace selection for multiple classification problems[END_REF][START_REF] Quattoni | An efficient projection for ℓ 1 /ℓ ∞ regularization[END_REF][START_REF] Liu | Blockwise coordinate descent procedures for the multi-task lasso, with applications to neural semantic basis discovery[END_REF] to account for features shared across tasks, and multiple kernel learning (Bach, 2008b) for the selection of different kernels. Specifying in advance the exact partition of the set features is in many situations too strong a requirement. We now consider a more general family of norms that notably reduces this constraint and makes it possible to encode richer structures.

Sparsity-Inducing Norms with Overlapping Groups of Variables

Sparsity-inducing norms with overlapping groups of variables constitutes an essential building block of this thesis and one of its contributions. Chapter 2 is dedicated to a thorough and formal analysis of the properties of these norms; we give in this section a rather informal overview of these objects and present in which practical circumstances they might prove to be interesting.

Starting from the definition of Ω in Eq. (1.7), it is natural to study what happens when the set of groups G is allowed to contain elements that overlap. In fact, and as shown in Chapter 2, the sparsity-inducing behavior of Ω remains the same. As a result, when regularizing by Ω, some entire groups of variables g in G are set to zero. We illustrate the properties of such a norm on Figure 1 .3-(b) where we display the unit ball associated with Ω. While the resulting patterns of nonzero variables-also referred to as supports, or nonzero patterns-were obvious in the non-overlapping case, it is interesting to understand here the relationship that ties together the set of groups G and its associated set of possible nonzero patterns. Let us denote by P the latter set.

Under mild conditions, it can be proved (see Chapter 2) that given any intersectionclosed2 family of patterns P of variables, such as all the rectangles on a two-dimensional grid of variables, it is possible to build an ad-hoc set of groups G-and hence, a regularization norm Ω-that enforces the support of the solutions of (1.1) to belong to P. Moreover, the converse is also true, meaning that given any norm of the form (1.7), we can characterize the set of possible nonzero patterns that the solutions of (1.1) can have.

These properties have important practical implications and make it possible to design norms that are adapted to the structure of the problem at hand; we give below interesting examples:

One-dimensional Sequence. Given p variables organized in a sequence, if we want to select only contiguous nonzero patterns, we represent on Figure 1.4 the set of groups G to consider. In this case, we have |G| = O(p). Imposing the contiguity of the nonzero patterns is for instance relevant in the context of time series, or for the diagnosis of tumors, based on the profiles of arrayCGH [START_REF] Rapaport | Classification of arrayCGH data using fused SVM[END_REF]. Indeed, because of the specific spatial organization of bacterial artificial chromosomes along the genome, the set of discriminative features is expected to have specific contiguous patterns. Note that the sparsifying effect we obtain here is to be contrasted with that of a penalty based on total-variation [START_REF] Rudin | Nonlinear total variation based noise removal algorithms[END_REF] plus the ℓ 1 -norm, where contiguous patterns would be encouraged but without the guarantee of selecting a single contiguous sequence. Moreover, the profiles of the learned coefficients will be piecewise constant in this case, a property which may not be desirable. Two-dimensional Grid. In the same way, assume now the p variables are organized on a two-dimensional grid. If we want the possible nonzero patterns P to be the set of all rectangles on this grid, the appropriate groups G to consider can be shown (see Chapter 2) to be those represented on Figure (1.5). In this setting, we have |G| = O( √ p).

Introduction and Related Work

Figure 1.5: Vertical and horizontal groups: (Left) the set of blue and green groups to penalize in order to select rectangles. (Right) In red, an example of nonzero pattern recovered in this setting, with its corresponding zero pattern (hatched area).

Sparsity-inducing regularizations built upon such group structures have resulted in good performances for background subtraction [START_REF] Cevher | Sparse signal recovery using markov random fields[END_REF][START_REF] Baraniuk | Model-based compressive sensing[END_REF]1.3. Variable Selection and Sparsity-Inducing Regularizations Huang et al., 2009;Mairal et al., 2010b), topographic dictionary learning [START_REF] Kavukcuoglu | Learning invariant features through topographic filter maps[END_REF][START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF], wavelet-based denoising [START_REF] Rao | Convex approaches to model wavelet sparsity patterns[END_REF], and for face recognition with corruption by occlusions (Jenatton et al., 2010b). Hierarchical Structure. A third interesting example assumes that the variables have a hierarchical structure. Specifically, we consider that the p variables correspond to the nodes of tree T (or a forest of trees). Moreover, we assume that we want to select the variables according to a certain order: a feature can be selected only if all its ancestors in T are already selected. This hierarchical rule can be shown to lead to the family of groups displayed on Figure 1.6.

Figure 1.6: Left: example of a tree-structured set of groups G (dashed contours in red), corresponding to a tree T with p = 6 nodes represented by black circles. Right: example of a sparsity pattern induced by the tree-structured norm corresponding to G: the groups {2, 4}, {4} and {6} are set to zero, so that the corresponding nodes (in gray) that form subtrees of T are removed. The remaining nonzero variables {1, 3, 5} form a rooted and connected subtree of T . This sparsity pattern obeys the following equivalent rules: (i) if a node is selected, the same goes for all its ancestors. (ii) if a node is not selected, then its descendant are not selected.

Chapter 4 largely discusses the properties of the norm Ω in this case, and presents efficient ways of optimizing the corresponding problem (1.1). This penalty was first used in [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]; since then, this type of groups has led to numerous applications, for instance, wavelet-based denoising [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF][START_REF] Baraniuk | Model-based compressive sensing[END_REF][START_REF] Huang | Learning with structured sparsity[END_REF]Jenatton et al., 2011c), hierarchical dictionary learning for both topic modeling and image restoration (Jenatton et al., 2010a(Jenatton et al., , 2011c)), log-linear models for the selection of potential orders [START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF], bioinformatics, to exploit the tree structure of gene networks for multi-task regression [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF], and multiscale mining of fMRI data for the prediction of some cognitive task (Jenatton et al., 2011b).

Extensions. The possible choices for the sets of groups G are not limited to the aforementioned examples. More complicated topologies can be considered, for instance, three-

Introduction and Related Work

dimensional spaces discretized in cubes or spherical volumes discretized in slices. An application to neuroimaging in Chapter 5 pursues this idea.

Before introducing more material in this introduction, we next review alternative approaches to structured sparsity that have emerged in the literature.

Related Approaches to Structured Sparsity

We classify these parallel approaches into three categories: Convex Formulations. As mentioned in the previous section, the family of norms defined in (1.7) is adapted to intersection-closed sets of nonzero patterns. However, some applications exhibit structures that can be more naturally modelled by unionclosed families of supports. This idea was developed by [START_REF] Jacob | Group Lasso with overlaps and graph Lasso[END_REF] who, given a set of groups G, introduced the following norm Ω union (w) min

ξ∈R p×|G| g∈G ξ g 2 such that g∈G ξ g = w, ∀g ∈ G, ξ g j = 0 if j / ∈ g. (1.8)
As it will be discussed at length below, there also are non-convex formulations adapted to union-closed families of supports. We now turn to another convex way of inducing structured sparsity. Unlike the approaches previously presented, the work from [START_REF] Micchelli | A family of penalty functions for structured sparsity[END_REF] does not rely on a set of groups G. Instead, for a given nonempty convex cone C ⊆ R p , the following norm is considered

Ω C (w) = inf c∈C 1 2 p j=1 w 2 j c j + c j .
The authors from [START_REF] Micchelli | A family of penalty functions for structured sparsity[END_REF] show that Ω C penalizes less the vectors w which satisfy {|w j |} j∈ 1;p ∈ C. As a result, and for judicious choices of C, the norm Ω C promotes some form of structure, as encoded by C. Examples of such structures are contiguous, or ordered, coefficients on a sequence.

Submodular Formulations.

Another approach for structured sparsity draws connections with submodular analysis (Bach, 2010a). Starting from non-decreasing, submodular3 set-functions F of the supports of the parameter vector w-i.e., w → F ({j ∈ 1; p ; w j = 0})-structured sparsity-inducing norms can be built by considering the convex envelope of F on the unit ℓ ∞ -norm ball. By selecting the appropriate set-function F , similar structures to those described above can be obtained. This idea was further extended to symmetric, submodular set-functions of the level sets of w, that is, w → max ν∈R F ({j ∈ 1; p ; w j ≥ ν}), thus leading to different types of structures (Bach, 2010b).

Dictionary Learning with Structured Sparsity-Inducing Penalties

Non-convex Formulations. We end this review of structured sparsity approaches by listing non-convex formulations. In the same flavor as the norm (1.8), [START_REF] Huang | Learning with structured sparsity[END_REF] considered the penalty

ψ(w) min H⊆G g∈H ω g , such that {j ∈ 1; p ; w j = 0} ⊆ g∈H g,
where G is a given set of groups, and {ω g } g∈G is a set of positive weights which defines a coding length. In other words, the penalty ψ measures from an information-theoretic viewpoint, "how much it costs" to represent w. A related approach was considered in [START_REF] Haupt | Signal reconstruction from noisy random projections[END_REF]. Finally, in the context of compressed sensing, the work of [START_REF] Baraniuk | Model-based compressive sensing[END_REF] also focuses on union-closed families of supports. In this case, however, the formulation does not bring into play a penalty derived from informationtheoretic considerations.

Dictionary Learning with Structured Sparsity-Inducing Penalties

So far, we not have not discussed how we should choose the representations of the available signals and data. For instance, in the basis pursuit formulation (1.3), we have implicitly assumed that we were given a fixed dictionary D-e.g., a basis of wavelets [START_REF] Mallat | A wavelet tour of signal processing[END_REF])-over which it was sensible to decompose the signal x. This section is dedicated to the introduction of matrix-factorization and dictionary-learning techniques which make it possible to learn a representation adapted to the considered class of signals.

Background Material on Matrix Factorization and Dictionary Learning

Throughout this introduction to dictionary learning, we take the notation and the viewpoint from signal processing, following the basis-pursuit formulation (1.3). Let us consider a set of n signals X = [x 1 , . . . , x n ] ∈ R m×n described by m features, typically n natural image patches composed of m pixels. The goal of the methods we next present consists of expressing each of these signals as linear combinations of dictionary elements, also known as atoms, taken from a learned dictionary.

Concretely, let us denote by D = [d 1 , . . . , d p ] ∈ R m×p the dictionary and introduce A = [α 1 , . . . , α n ] ∈ R p×n the matrix containing the decompositions, also referred to as codes, for each of the n signals. With this notation in place, we try to simultaneously learn (A, D) in order to obtain X ≈ DA, as measured by some data-fitting term. Most of the work in this thesis focuses on the square loss function4 , but some fields of applications benefit from other data-fitting

Introduction and Related Work

terms, e.g., in audio processing [START_REF] Févotte | Nonnegative matrix factorization with the itakura-saito divergence: With application to music analysis[END_REF]Lefèvre et al., 2011a). Stated in this way, the problem of finding some interesting pair (A, D) is still very general; it is therefore appealing to reduce the space of candidate pairs (A, D) by further adding some constraints on A and/or D. Those constraints should reflect some expected properties of the problem at hand. We list below some interesting and well-known examples of such formulations:

Nonnegative Matrix-Factorization

This type of factorization is well-tailored for nonnegative signals. Nonnegative matrixfactorization (NMF) [START_REF] Lee | Learning the parts of objects by non-negative matrix factorization[END_REF] has notably become popular thanks to its application to face images. In this context, it was observed to retrieve sets of variables that are partly localized on the face and capture some features or parts of the face which seem intuitively meaningful given our a priori (for a more detailed discussion and comparisons to structured approaches, see Chapter 3). More formally, the formulation of NMF reads min

A∈R p×n + , D∈R m×p + 1 2 X -DA 2 F .
Thus, the coefficients of both A and D are forced to be nonnegative. This standard formulation was extended in many ways, see, e.g., [START_REF] Hoyer | Non-negative matrix factorization with sparseness constraints[END_REF]; [START_REF] Cai | Graph regularized non-negative matrix factorization for data representation[END_REF].

K-means Clustering

Another example of matrix-factorization problem is K-means [START_REF] Lloyd | Least squares quantization in PCM[END_REF]. The objective of K-means is to find clusters and cluster centers, also referred to as centroids, starting from a set of unlabeled data-points. Once the desired number of clusters is chosen, say p ∈ N, the formulation of K-means is given by min

A∈{0,1} p×n D∈R m×p 1 2 X -DA 2 F , such that for any i ∈ 1; n , α i 1 = 1.
The additional constraints on the columns of A enforce that one signal belongs to a single cluster out of the p possibles ones. The previous list of examples is of course not exhaustive, and could be further completed, e.g., by principal component analysis [START_REF] Jolliffe | Principal component analysis[END_REF] which is probably one of the most popular tools for data analysis and unsupervised dimensionality reduction. Moreover, and as it shall be discussed at length in Chapter 4, several probabilistic topic models [START_REF] Blei | Latent dirichlet allocation[END_REF] and other stochastic block models [START_REF] Airoldi | Mixed membership stochastic blockmodels[END_REF] can be viewed as matrix-factorization problems as well [START_REF] Buntine | Variational Extensions to EM and Multinomial PCA[END_REF]. For further discussions, we refer the interested readers to some reviews about matrix factorization and dictionary learning, e.g., [START_REF] Singh | A Unified View of Matrix Factorization Models[END_REF]; [START_REF]Sparse coding for machine learning, image processing and computer vision[END_REF]; [START_REF] Tosic | Dictionary learning[END_REF].

We now turn to the formulation of dictionary learning we adopt throughout this manuscript.

1.4. Dictionary Learning with Structured Sparsity-Inducing Penalties

Dictionary-Learning with (Structured) Sparsity-Inducing Norms

Let us introduce two convex sets A ⊆ R p×n and D ⊆ R m×p . We shall now focus on the following formulation min

A∈A, D∈D 1 2 X -DA 2 F + λΩ(A) , (1.9)
where Ω is a sparsity-inducing regularizer applied to the matrix A, which will usually decompose into a sum of vector-based regularizers for each of the columns (or rows) of A. By introducing Ω in the cost function, we create a unbalance that can cause the coefficients of the matrix A to end up with small magnitudes; we therefore impose the set D to be bounded to avoid any degenerated solution. As a side comment, we know from [START_REF] Bach | Convex sparse matrix factorizations[END_REF] that problem (1.9) has equivalent formulations, and we can go from regularized to constrained objective functions, and conversely. However, some of the resulting formulations are more suitable for optimization than others, and we shall mostly consider the form of problem (1.9).

Optimization. In terms of optimization, the presence of the product DA in problem (1.9) implies that there is unfortunately no joint convexity in the pair (A, D). However, when one of the matrices is kept fixed, the problem is convex with respect to the other one; this property can be exploited within an alternative optimization scheme which leads to good results in practice and has become a commonly-used procedure [START_REF] Olshausen | Sparse coding with an overcomplete basis set: A strategy employed by V1?[END_REF][START_REF] Aharon | The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representations[END_REF][START_REF] Lee | Efficient sparse coding algorithms[END_REF]Mairal et al., 2009b). Details about the computations of these two steps-i.e., the optimization over A for D fixed, and vice-versa-are discussed in Chapter 3 and 4. Furthermore, an efficient online optimization method was recently designed for dictionary learning (Mairal et al., 2010a), making it possible to handle extremely large sets of signals.

Sparse Dictionary Learning.

A popular instance of (1.9) is sparse coding (Olshausen andField, 1996, 1997), where we not only want each signal to be well explained, but we also seek sparse decompositions over the dictionary; this leads to min

A∈R p×n , D∈D 1 2 X -DA 2 F + λ n i=1 α i 1 , (1.10)
where D is typically taken to be the set of dictionaries whose atoms have their ℓ 2 -norm less than, or equal to one. Sparse coding has received a great deal of attention over the past few years and has led to state-of-the-art performances in computer vision [START_REF] Boureau | Learning mid-level features for recognition[END_REF], as well as in various image-processing tasks, e.g., see [START_REF] Peyré | Sparse modeling of textures[END_REF]; [START_REF]Sparse coding for machine learning, image processing and computer vision[END_REF] and numerous references therein. It is worth noting that sparse dictionary learning can alternatively be formulated as convex [START_REF] Bach | Convex sparse matrix factorizations[END_REF]Bradley and Bagnell, 2009b), submodular [START_REF] Krause | Submodular dictionary selection for sparse representation[END_REF], and non-parametric Bayesian [START_REF] Zhou | Non-parametric Bayesian dictionary learning for sparse image representations[END_REF] problems.

Introduction and Related Work

Related to sparse coding is the problem of sparse principal component analysis [START_REF] Jolliffe | A modified principal component technique based on the Lasso[END_REF][START_REF] Zou | Sparse principal component analysis[END_REF][START_REF] Moghaddam | Spectral bounds for sparse PCA: Exact and greedy algorithms[END_REF][START_REF] Zass | Nonnegative sparse PCA[END_REF][START_REF] Aspremont | Optimal solutions for sparse principal component analysis[END_REF][START_REF] Lounici | Taking advantage of sparsity in multi-task learning[END_REF][START_REF] Witten | A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis[END_REF]. In this setting, the decompositions of the signals are not encouraged to be sparse anymore, but instead, the dictionary elements are forced to involve only a few variables (i.e., a small fraction of the m features). For further details and discussions about the underlying optimization problems, we refer the interested readers to the review from [START_REF] Zhang | Sparse pca: Convex relaxations, algorithms and applications[END_REF].

Sparse Structured Dictionary Learning. The framework of dictionary learning lends itself well to structured sparsity. Indeed, by having access to the factorization DA, it possible to encode prior knowledge in various ways: through the features (via the columns of D), over the latent variables (through the columns of A and the rows of D), and also across the different signals (this time, thanks to the rows of A).

Moreover, there is an important subtlety which is worth being exposed. In the traditional basis pursuit setting, as presented Eq. (1.3), the dictionary is assumed fixed, and we therefore need strong prior information to be able to organize the atoms according to the considered regularizers. Although such situations do exist, e.g., see Chapter 4 with some wavelet base that naturally exhibits a hierarchical structure, they remain rather uncommon. Of course, in some cases, we can still apply an adapted pre-processing step to make sure the data exactly match the structure encoded by the regularization, e.g., see Chapter 5 where a hierarchical clustering of the features is performed beforehand.

On the contrary, in the context of dictionary learning, since the dictionary elements are learned, we can argue that the atoms will have to match well the prior that is imposed by the regularization. In other words, combining structured regularization with dictionary learning has precisely the advantage that the dictionary elements will selforganize and self-adapt to match the prior.

Sparse structured dictionary learning has been successfully applied to various modalities, such as, for instance, misaligned gene-expression time series [START_REF] Tibau Puig | Order-preserving factor analysis-application to longitudinal gene expression[END_REF], hierarchical topic modeling (Jenatton et al., 2010a(Jenatton et al., , 2011c)), the design of topographic dictionaries [START_REF] Kavukcuoglu | Learning invariant features through topographic filter maps[END_REF][START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF] and localized features for face recognition (Jenatton et al., 2010b), the denoising of natural image patches (Jenatton et al., 2010a(Jenatton et al., , 2011c)), and speaker/instrument identification (Sprechmann et al., 2010a) as well as source separation (Lefèvre et al., 2011b).

Some Elements of Convex Analysis and Convex Optimization for Sparse Methods

Some Elements of Convex Analysis and Convex

Optimization for Sparse Methods Some parts of this section are built upon the material developed in the following book chapter: F. Bach, R. Jenatton, J. Mairal and G. Obozinski. Convex Optimization with Sparsity-Inducing Norms. In S. Sra, S. Nowozin, and S. J. Wright, editors, Optimization for Machine Learning, 2011. The goal of this section is twofold. On the one hand, we shall recall results from convex analysis that are important to study the properties of structured sparsity-inducing norms. Most of these results are well-known in the literature, and we refer the readers interested in a more extensive coverage of this topic to classical books, such as, e.g., [START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF]; [START_REF] Boyd | Convex Optimization[END_REF]; [START_REF] Bertsekas | Nonlinear programming[END_REF]. On the other hand, we will focus on optimization techniques that are well-suited to address problem (1.1) in the context of structured sparsity-inducing norms.

Background Material of Convex Analysis

We start this section by introducing the concept of subgradient which generalizes the notion of derivative for non-smooth functions, and which is essential to describe optimality conditions. As a reminder, a function h : R p → R is said to be convex if and only if (1) dom(h) {w ∈ R p ; h(w) < +∞} is a convex set, and (2) for any (v, w, t) ∈ dom(h) × dom(h) × [0, 1], we have

h(tv + (1 -t)w) ≤ t h(v) + (1 -t)h(w).
Moreover, if the domain dom(h) is nonempty, h is said to be proper. In particular, this definition allows convex functions to take the value +∞, such as the indicator function of a convex set.

Subgradients and Optimality conditions

Given a convex function h : R p → R and a vector w in R p , let us define the set of subgradients-or simply, the subdifferential-of h at w as

∂h(w) {z ∈ R p ; h(w) + z ⊤ (v -w) ≤ h(v) for any vector v ∈ R p }.
(1.11)

The elements of ∂h(w) are called the subgradients of h at w. For convex functions and for all points that lie in the interior of the domain, the subdifferential at this point is nonempty. We present interesting examples of subdifferentials that will prove important in our analysis:

Introduction and Related Work

• For a convex and differentiable function f : In this case, the subdifferential is a singleton that reduces to the gradient of f , that is, ∂f (w) = {∇f (w)}. • For a norm Ω: A little calculation shows in this case the following equality

∂Ω(w) = {z ∈ R p ; Ω * (z) ≤ 1} if w = 0, {z ∈ R p ; Ω * (z) ≤ 1 and z ⊤ w = Ω(w)} otherwise.
(1.12)

The previous equation brings into play an object central to this thesis, namely the dual norm of Ω, referred to as Ω * and which is defined by

Ω * (z) max w∈R p z ⊤ w such that Ω(w) ≤ 1. (1.13)
The dual norm notably arises in the analysis of estimation bounds [START_REF] Negahban | A unified framework for highdimensional analysis of M-estimators with decomposable regularizers[END_REF], in the definition of proper stopping criteria while minimizing convex functions, and in the design of working-set strategies [START_REF] Bach | Convex optimization with sparsityinducing norms[END_REF], as will be shown in Section 1.5.2. Moreover, the dual norm of Ω * is Ω itself, and as a consequence, the formula above holds also if the roles of Ω and Ω * are exchanged.

It is easy to show that in the case of an ℓ q -norm, for q ∈ [1; +∞], the dual norm is the ℓ q ′ -norm, with q ′ in [1; +∞] such that 1 q + 1 q ′ = 1. In particular, the ℓ 1 -and ℓ ∞ -norms are dual to each other, and the ℓ 2 -norm is self-dual (dual to itself). The previous list is not exhaustive, and other examples of useful subdifferentials include the indicator function of a convex set C, whose set of subgradients is given by the normal cone to C. We now turn to the main application of subgradients in this thesis, that is, the characterization of optimality conditions for non-smooth optimization problems:

Proposition 1 (Subgradients at Optimality) For any proper convex function h : R p → R, a point w in R p is a global minimum of h if and only if the condition 0 ∈ ∂h(w) holds.

Note that the concept of subdifferential is mainly useful for non-smooth functions. If h is differentiable at w, the set ∂h(w) is indeed the singleton {∇h(w)}, and the condition 0 ∈ ∂h(w) reduces to the classical first-order optimality condition ∇h(w) = 0.

With these technical results in place, we can characterize the optimality conditions of problem (1.1) when the data-fitting term is assumed to be smooth and Ω is taken to be a norm. Putting together the pieces with Eq. (1.12) and Proposition 1, and applying standard convex calculus rules [START_REF] Rockafellar | Convex analysis[END_REF], we have that w ∈ R p is optimal if and only if

- 1 λ ∇f (w) ∈ ∂Ω(w).
As a consequence, the vector 0 is solution if and only if Ω * ∇f (0) ≤ λ. The general optimality condition above can be specified to the ℓ 1 -norm. Recalling the definition of the subdifferential of a norm in Eq. (1.12) along with the fact that the ℓ ∞ -and ℓ 1 -norms are dual to each other, we obtain that w ∈ arg min

v∈R p f (v)+λ v 1 ⇔    ∇f (w) j ≤ λ,
for all j ∈ 1; p with w j = 0, ∇f (w) j = sign(w j )λ, otherwise.

Some Elements of Convex Analysis and Convex Optimization for Sparse Methods

These optimality conditions provide with a formal justification to the sparsity-inducing property of the ℓ 1 -norm: depending on the strength of the regularization parameter λ, the directions along which the variation of the data-fitting term is not large enough are kept to zero, and sparsity is indeed promoted. In the case of the square loss function, note that we get back the traditional optimality conditions for Lasso and basis pursuit [START_REF] Fuchs | Recovery of exact sparse representations in the presence of bounded noise[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF].

We now introduce additional material to derive duality gaps and monitor the progress of optimization algorithms.

Fenchel Conjugate and Duality Gaps

Let us denote by f * the Fenchel conjugate of f [START_REF] Rockafellar | Convex analysis[END_REF], defined by

f * (z) sup w∈R p [z ⊤ w -f (w)].
Regardless of the nature of the function f , the conjugate is always convex. Moreover, under mild conditions (convexity and closedness of the level sets of f ), the functions f and its biconjuguate f * * coincide. For many functions, the conjugate is available in closed form [START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF], and for the square loss function, we notably have that t → 1 2 t 2 is self-conjugate. In the context of this thesis, it is notably useful to specify the expression of the conjugate of a norm. Perhaps surprisingly and misleadingly, 5 the conjugate of a norm is not equal to its dual norm, but corresponds instead to the indicator function of the unit ball of its dual norm. More formally, we have the result (e.g., see Example 3.26 in [START_REF] Boyd | Convex Optimization[END_REF]:

Proposition 2 (Fenchel conjugate of a norm)
Let Ω be a norm on R p . The following equality holds for any z ∈ R p

sup w∈R p [z ⊤ w -Ω(w)] = 0 if Ω * (z) ≤ 1 +∞ otherwise.
Fenchel conjugates are particularly useful to derive duality gaps. As a brief reminder, the duality gap of a minimization problem is defined as the difference between the primal and dual objective functions, evaluated for a feasible pair of primal/dual variables (Boyd and Vandenberghe, 2004, see Section 5.5). This gap serves as a certificate of (sub)optimality: if it is equal to zero, then the optimum is reached, and provided that strong duality holds, the converse is true as well [START_REF] Boyd | Convex Optimization[END_REF], see Section 5.5). Dual problems and feasible dual variables can be obtained through the following result (Borwein and Lewis, 2006, Theorem 3.3.5):

Introduction and Related Work

f and the dual norm of Ω, then we have

max z∈R p : Ω * (z)≤λ -f * (z) ≤ min w∈R p f (w) + λΩ(w).
Moreover, equality holds as soon as the domain of f has nonempty interior.

Hence, for any feasible w, z in R p , we can compute the difference between the value of the primal objective function f (w) + λΩ(w) and the dual objective function -f * (z), which results in the duality gap:

f (w) + λΩ(w) + f * (z).
Proposition 3 shows that it is always positive, and that it vanishes at optimality, where we then speak about strong duality. Duality gaps are important in convex optimization because they provide an upper bound on the difference between the current value of an objective function and the optimal value, which makes it possible to set proper stopping criteria for iterative optimization algorithms. Given a current iterate w, computing a duality gap requires choosing a "good" candidate for z (and in particular a feasible one). Given that at optimality, z ⋆ = ∇f (w ⋆ ) is the unique solution to the dual problem, a natural choice of dual variable is z = min 1, λ Ω * (∇f (w)) ∇f (w), which reduces to z ⋆ at the optimum and therefore yields a zero duality gap at optimality.

The next section provides with an alternative tool to deal with non-smooth convex formulations, especially adapted to the structure (1.7) of Ω.

Conic Duality and Sparsity-Inducing Norms

Because of the structure of the regularizers we study in this thesis-namely linear combinations of (generally non-smooth) norms-conic duality (see [START_REF] Boyd | Convex Optimization[END_REF], and references therein) appears as a powerful and appealing tool. In fact, conic duality makes it possible to derive representations of (1.7) 6 that may be more amenable to optimization (see, e.g., [START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF]Jenatton et al., 2010a) and/or better suited for theoretical analysis (see, for instance, Bach, 2008b;Jenatton et al., 2010a).

We review here simple properties of cones and give an example of reformulation of (1.7) that is particularly useful when dealing with overlapping groups of variables (Jenatton et al., 2010a;[START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF].

Let us consider C ⊆ R p a cone, that is, a set such that if c ∈ C, then tc ∈ C for any nonnegative scalar t. If the cone C is further assumed to be proper, 7 it induces a partial ordering C on R p , so that a C b ⇔ ba ∈ C.

6. In fact, using conic duality in place of classical Fenchel duality is more natural in our context. Both approaches can lead to the same dual formulations, but conic duality is usually more straightforward. For these reasons, we have made the decision to present both frameworks.

7. A cone is said to be proper if it is convex, closed, has no empty interior and does not contain any line [START_REF] Boyd | Convex Optimization[END_REF].

Some Elements of Convex Analysis and Convex Optimization for Sparse Methods

A well-known example of such ordering is the componentwise inequality between pdimensional vectors built from the proper cone C = R p + . Based on the cone C, a dual counterpart can be defined in the following way:

C * {a ∈ R p ; a ⊤ b ≥ 0 for all b ∈ C}.
Moreover, it can be shown that if C is proper, the same goes for the dual cone C * [START_REF] Boyd | Convex Optimization[END_REF], which notably implies that a dual ordering C * can also be defined, similarly to C . Interesting properties tie these two orderings together, among which

[a C b] ⇔ [c ⊤ a ≤ c ⊤ b for all c C * 0]. (1.14)
Relationship (1.14) is central to extend Lagrangian duality and Karush-Kuhn-Tucker optimality conditions to convex programs where inequality constraints are specified via generalized conic orderings, also known as generalized inequalities.

In the context of this thesis, it is natural to consider the following proper cone

C {(a, t) ∈ R p+1 ; a ≤ t}, (1.15)
where . denotes any norm on R p . Interestingly, the dual cone of C has a simple expression that brings into play the dual norm of . (see Boyd and Vandenberghe, 2004, Example 2.25), namely

C * {(α, τ ) ∈ R p+1 ; α * ≤ τ }. (1.16)
For instance, when the underlying norm is chosen to be the ℓ 2 -norm, we speak about second-order cone, or "ice-cream" cone [START_REF] Boyd | Convex Optimization[END_REF]. In addition, we have by definition of the conic orderings

[ a ≤ t] if and only if [(a, t) C 0], and [ α * ≤ τ ] if and only if [(α, τ ) C * 0].
The two previous relationships are the starting point to reformulate (1.1). If Ω(w) = g∈G w g , a first approach consists of rewriting (1.1) under (convex) conic constraints to bypass the non-smoothness of Ω, that is, min

w∈R p ,t∈R |G| f (w) + λ g∈G t g , such that ∀ g ∈ G, (w g , t g ) C 0.
(1.17)

Provided that f is smooth and convex, the overall objective function is also smooth and convex, so that projected-gradient schemes [START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF]) can be applied. For instance, when . is chosen to be the ℓ 2 -norm, projecting onto a single conic constraint can be performed efficiently with the following closed-form solution:

Proj C ((a, t)) =        (0, 0) if a 2 ≤ -t, (a, t) if a 2 ≤ t,

Introduction and Related Work

When the set of conic constraints is separable, the situation is simple since it is sufficient to perform the individual projections sequentially. Moreover, when separability does not hold anymore, i.e., when the groups in G overlap, Dykstra's cyclic algorithm can be used instead [START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF]. Generalized inequalities can be further exploited to obtain dual formulations of (1.1) (Jenatton et al., 2010a;[START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF]. In particular, we have the following result:

Proposition 4 (Duality through generalized inequalities)

Let f be a proper convex function. Let us consider the maximization problem

max ξ∈R p×|G| , ξ∈R p -f * ( ξ) such that        ∀ g ∈ G, ξ g j = 0 for j / ∈ g, ∀ g ∈ G, (ξ g , λ) C * 0, ξj = g∈G,g∋j ξ g j .
(1.18)

Problems (1.1) and (1.18) are dual to each other and strong duality holds.

The previous proposition turns out to be useful to derive efficient algorithms when groups in G are overlapping (Jenatton et al., 2010a;[START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF] and to make interesting connections between sparsity-inducing norms and optimization problems encountered in operations research (for more details, see [START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF].

We next present algorithms that are well adapted to solve problem (1.1) with structured sparsity-inducing norms.

Optimization Methods with Structured Sparsity-Inducing Norms

While the previous section was dedicated to the theoretical study of problem (1.1) with tools borrowed from convex analysis, we now present practical optimization procedures. In particular, we shall pay attention to the (if known) convergence-rate guarantees of the presented algorithms and their ability to properly handle sparsity. By the latter statement, we mean that the procedures should be able to output solutions with an identifiable set of nonzero entries, without requiring a somewhat arbitrary thresholding operation. Finally, Chapter 4 contains some speed comparisons involving the different techniques presented below.

Generic Approaches

The first class of methods we consider here is blind to the structure of problem (1.1). If we assume we can compute the gradient of f , and since we can always obtain a subgradient for Ω as defined in (1.7), we can then resort to subgradient descent to solve problem (1.1) (see, e.g., Bertsekas, 1999, and references therein). In this case, we know from [START_REF] Nesterov | Introductory lectures on convex optimization: a basic course[END_REF] that after k iterations, the cost function f + λΩ will be ε-close to the optimum, with ε = O(1/ √ k). This guarantee is rather weak, and we shall see afterwards methods with faster rates. In addition, subgradient descent cannot typically handle sparsity, except when combined with other strategies, e.g., some truncation step [START_REF] Langford | Sparse online learning via truncated gradient[END_REF].

Some Elements of Convex Analysis and Convex Optimization for Sparse Methods

Before reviewing other techniques, it is worth mentioning that if f is chosen to be the square loss, and when Ω is a mixed ℓ 1 /ℓ 2 -or ℓ 1 /ℓ ∞ -norm, then problem (1.1) can be cast as a second-order cone program (SOCP) and a quadratic program (QP) respectively, for which there are generic interior-point solvers that are known to be accurate but not highly scalable. As a brief reminder, a SOCP is defined as a convex optimization problem with a linear cost function, under second-order cone constraints (see Section 1.5.1) and linear equality constraints [START_REF] Boyd | Convex Optimization[END_REF], that is,

min u∈R p u ⊤ 0 u such that A i u + b i 2 ≤ c ⊤ i u + d i , for i ∈ 1; m , Eu = f , where u 0 ∈ R p , A i ∈ R n×p , b i ∈ R n , c i ∈ R p , d i ∈ R, E ∈ R q×p
and f ∈ R q are fixed parameters defining the set of constraints.

Reweighted-ℓ 2 Approaches

This second class of methods deals with the non-smoothness of Ω by expressing the norm as the minimum over a set of smooth functions. At the cost of adding new variables (to describe the set of smooth functions), the problem becomes more amenable to optimization. In particular, reweighted-ℓ 2 schemes consist of approximating the norm Ω by successive quadratic upper bounds [START_REF] Argyriou | Multi-task feature learning[END_REF][START_REF] Rakotomamonjy | SimpleMKL[END_REF]Jenatton et al., 2010b;[START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF][START_REF] Daubechies | Iteratively reweighted least squares minimization for sparse recovery[END_REF][START_REF] Micchelli | A family of penalty functions for structured sparsity[END_REF]. Indeed, based on the inequality of arithmetic and geometric means 2 √ ab ≤ a + b, it can be shown that

Ω(w) = min (η g ) g∈G ∈R |G| + 1 2 g∈G w g 2 2 η g + η g = min η∈R |G| + 1 2 p j=1 g∈G g∋j 1 η g w 2 j + g∈G η g , (1.19)
where we have assumed that Ω is a mixed ℓ 1 /ℓ 2 -norm. Interestingly, equality holds above if and only if η g = w g 2 for all g in G. Plugging the previous relationship into Eq. (1.1), the optimization can then be performed by alternating between the updates of w and the additional variables (η g ) g∈G . 8 We shall see in Chapter 3 extensions of (1.19) to the case of more sophisticated norms. When the norm Ω is defined as a linear combination of ℓ ∞ -norms, we are not aware of the existence of such variational formulations. Unlike subgradient descent or proximal methods (see next section), reweighted-ℓ 2 methods do not have simple non-asymptotic guarantees on their convergence rate (e.g., in Daubechies et al., 2010, local convergence rates are studied).

Introduction and Related Work

Proximal Methods

Proximal methods constitute a class of first-order techniques well-adapted to deal with problem (1.1) [START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF]. In fact, and as opposed to the generic approaches presented above, proximal schemes will specifically take advantage of the structure of (1.1), namely, the sum of two convex terms where only one of these components is assumed smooth. Thus, we will typically assume that the data-fitting function f is convex differentiable, with Lipschitzcontinuous gradient. On the other hand, Ω is only asked to be convex, and problem (1.1) with structured sparsity-inducing norms therefore matches these requirements.

Proximal methods have become increasingly popular over the past few years, in both the signal processing (e.g., [START_REF] Becker | NESTA: A Fast and Accurate First-order Method for Sparse Recovery[END_REF][START_REF] Wright | Sparse reconstruction by separable approximation[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF], and numerous references therein) and the machine learning communities (e.g., Bach et al., 2011, and references therein). In a broad sense, these methods can be seen as a natural extension of gradient-based techniques when the objective function to minimize has a non-smooth part. Proximal methods are iterative procedures. The simplest version of this class of methods linearizes at each iteration the function f around the current estimate ŵ, and this estimate is updated as the (unique by strong convexity) solution of the proximal problem, defined as follows:

min w∈R p f ( ŵ) + (w -ŵ) ⊤ ∇f ( ŵ) + λΩ(w) + L 2 w -ŵ 2 2 .
(1.20)

A quadratic term is added in order to keep the update in a neighborhood where f is close to its linear approximation, and L > 0 is a parameter which is an upper bound on the Lipschitz constant of ∇f . Provided that we can solve efficiently the proximal problem of Eq. (1.20), this first iterative scheme constitutes a simple way of solving problem (1.1). It appears under various names in the literature: proximal-gradient techniques [START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF], forwardbackward splitting methods [START_REF] Combettes | Proximal splitting methods in signal processing[END_REF], and iterative shrinkagethresholding algorithm [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]. Furthermore, convergence rates of the function value f + λΩ can be proved [START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], and after k iterations, the precision obtained is in the order of O(1/k), which is to be contrasted with the subgradient case O(1/ √ k). Interestingly, this first iterative scheme was extended to "accelerated" versions [START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]. In these extensions, the update is not taken to be exactly the result from (1.20); instead, it consists of solving the proximal problem applied to a well-chosen linear combination of the previous estimates. In this case, the function value f + λΩ converges to the optimum with a rate of O(1/k 2 ), where k is the iteration number. From [START_REF] Nesterov | Introductory lectures on convex optimization: a basic course[END_REF], we know that this rate is optimal within the class of first-order techniques; in other words, accelerated proximal-gradient methods behave as if there were no non-smooth component to handle.

Various proximal schemes can be found in the literature, depending on the considered setting (e.g., batch versus stochastic) and/or the assumptions made on the function f . For online/stochastic frameworks, we refer the interested readers to Duchi and 1.5. Some Elements of Convex Analysis and Convex Optimization for Sparse Methods Singer (2009); [START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF]; [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF]. Moreover, if we only assume that f is convex, without satisfying smoothness properties, Douglas-Rachford splitting algorithm can be applied [START_REF] Combettes | Proximal splitting methods in signal processing[END_REF]. For additional work in this direction, see also [START_REF] Duchi | Efficient online and batch learning using forward backward splitting[END_REF]; [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF].

We have so far given an overview of proximal methods, without specifying how we precisely handle its main building block, namely the computation of the proximal problem, as defined in (1.20).

Proximal Problem. We start by equivalently rewriting problem (1.20) as

min w∈R p 1 2 w -ŵ - 1 L ∇f ( ŵ) 2 2 + λ L Ω(w).
Under this form, we can readily observe that when λ = 0, the solution of the proximal problem amounts to the standard gradient update rule. The problem above can be more generally viewed as an instance of the proximal operator [START_REF] Moreau | Fonctions convexes duales et points proximaux dans un espace hilbertien[END_REF] associated with λΩ:

Prox λΩ : u ∈ R p → arg min v∈R p 1 2 u -v 2 2 + λΩ(v).
This operator enjoys many attractive properties, e.g., continuity and nonexpansivity [START_REF] Combettes | Signal recovery by proximal forward-backward splitting[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF]. In particular, there is a relation that ties Prox λΩ together with the Euclidean projection Proj {v∈R p ; Ω * (v)≤λ} which projects onto the ball {v ∈ R p ; Ω * (v) ≤ λ}. Specifically, we have for any u ∈ R p (see Example 2.17 in [START_REF] Combettes | Signal recovery by proximal forward-backward splitting[END_REF]:

Prox λΩ (u) = u -Proj {v∈R p ; Ω * (v)≤λ} (u).

(1.21)

For many choices of regularizers Ω, the proximal operator leads to a closed-form solution, which makes proximal methods especially efficient. If Ω is chosen to be the ℓ 1 -norm, we get back the soft-thresholding operator (1.4) applied elementwise. Similarly, if Ω is a mixed ℓ 1 /ℓ 2 -norm with its underlying set of groups G forming a partition of 1; p , we obtain the group soft-thresholding operator [START_REF] Turlach | Simultaneous variable selection[END_REF]:

For any u ∈ R p , for any g ∈ G, Prox λΩ (u) g =    0 if u g 2 ≤ λ, ug 2 -λ ug 2 u g otherwise.
In some cases, the direct computation of Prox λΩ is not easy, while the projection Proj {v∈R p ; Ω * (v)≤λ} is; we can then exploit (1.21). This situation arises for instance with a mixed ℓ 1 /ℓ ∞ -norm, where G still defines a partition of 1; p . We indeed have

For any u ∈ R p , for any g ∈ G, Prox λΩ (u)

g = u g -Proj {v∈R |g| ; v 1 ≤λ} (u g ),
and the Euclidean projection onto the ℓ 1 -norm ball can be performed in linear time [START_REF] Brucker | An O(n) algorithm for quadratic knapsack problems[END_REF][START_REF] Duchi | Efficient projections onto the ℓ 1 -ball for learning in high dimensions[END_REF].

Introduction and Related Work

As soon as groups in G overlap, the situation becomes difficult since no closedforms are available, even by resorting to (1.21); Chapter 4 focuses on these settings and provides efficient algorithmic solutions. A related question of interest is whether it is possible to compute Prox λΩ only approximately, while still enjoying convergence, and possibly guaranteeing the same rates as in the approximation-free case (Schmidt et al., 2011b).

Block-Coordinate Methods

The main idea of (block-) coordinate descent techniques is to solve problem (1.1) by sequentially optimizing with respect to one variable (or block of variables) while keeping the other ones fixed. This approach is appealing when the subproblem involving a single variable (or, a single block of variables) can be computed efficiently (either exactly or approximately). Moreover, since several cycles over the p variables at play may be necessary, (block-) coordinate descent techniques can be naturally coupled to working-set strategies (see next section).

Coordinate descent is particularly suited for the Lasso (1.2) and basis pursuit (1.3) problems [START_REF] Fu | Penalized regressions: the bridge versus the lasso[END_REF][START_REF] Friedman | Pathwise coordinate optimization[END_REF][START_REF] Wu | Coordinate descent algorithms for lasso penalized regression[END_REF]. In this case, it can be shown that each subproblem amounts to solving a proximal operator associated with the ℓ 1 -norm, whose solution is available in closed-form, as defined by the soft-thresholding operator (1.4). Convergence in this setting is guaranteed by results from Tseng (2001). 9 In this context, the simplicity of application of coordinate descent lies in the fact that (1) the data-fitting is quadratic, and (2) the ℓ 1 -norm is separable with respect to the p variables.

The situation is more complex when considering more general loss functions (for instance, a logistic loss function) and/or a broader family of regularizers (for example, separable mixed ℓ 1 /ℓ q -norms with G forming a partition of 1; p ). We briefly mention the extensions from [START_REF] Tseng | A coordinate gradient descent method for nonsmooth separable minimization[END_REF]; [START_REF] Wright | Accelerated block-coordinate relaxation for regularized optimization[END_REF] whose algorithms consist in using local quadratic approximations of the data-fitting term. 10 In both papers, convex block-separable regularizers can be handled by solving a proximal problem within each block. Such a method was for instance applied in [START_REF] Meier | The group lasso for logistic regression[END_REF] for logistic regression with an ℓ 1 /ℓ 2 -norm regularization.

For structured sparsity, we are interested in regularizers with overlapping groups of variables (see Section 1.3.3), in which case we are not aware of efficient block-coordinate methods. However, and as developed in Chapter 4, some dual representations of Ω can circumvent the issue of overlapping groups (see, e.g., Proposition 4, page 24). 9. Note that the result from [START_REF] Bertsekas | Nonlinear programming[END_REF] could also be applied, after reformulating the regularization term in the form of separable convex constraints.

10. The authors from [START_REF] Tseng | A coordinate gradient descent method for nonsmooth separable minimization[END_REF] consider more general quadratic approximations (i.e., with non-diagonal Hessians) for which an inexact line-search can be conducted.

Some Elements of Convex Analysis and Convex Optimization for Sparse Methods

Working-set Approaches

Working-set algorithms address optimization problems by solving an increasing sequence of small subproblems of (1.1). The working set, that we will denote by J ⊆ 1; p , refers to the subset of variables involved in the optimization of these subproblems.

Working-set algorithms proceed as follows: after computing a solution to the problem restricted to the variables in J, global optimality is checked to determine whether the algorithm has to continue. If this is the case, new variables enter the working set J according to a strategy that has to be defined. Note that we only consider forward algorithms, i.e., where the working set grows monotonically. In other words, there are no backward steps where variables would be allowed to leave the set J. Provided this assumption is met, it is easy to see that these procedures stop in a finite number of iterations.

This class of algorithms takes advantage of sparsity from a computational point of view [START_REF] Lee | Efficient sparse coding algorithms[END_REF][START_REF] Szafranski | Hierarchical penalization[END_REF]Bach, 2008a;[START_REF] Roth | The group-Lasso for generalized linear models: uniqueness of solutions and efficient algorithms[END_REF][START_REF] Obozinski | Joint covariate selection and joint subspace selection for multiple classification problems[END_REF]Jenatton et al., 2011a;[START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF], since the subproblems that need to be solved are typically much smaller than the original one.

Working-set algorithms require three ingredients:

• Inner-loop solver: At each iteration of the working-set algorithm, problem (1.1) has to be solved on J, i.e., subject to the additional equality constraint that w j = 0 for all j in J c :

min w∈R p f (w) + λΩ(w), such that w J c = 0. (1.22)
The computation can be performed by any of the methods presented in this section. Working-set algorithms should therefore be viewed as "meta-algorithms". Since solutions for successive working sets are typically close to each other (except for the newly-active variables), the approach is efficient if the method chosen can use warm-restarts. Finally, even though problem (1.22) is formally defined in R p , in practice, adding the equality constraint on J c amounts to manipulating vectors in R |J| (with hopefully |J| ≪ p). • Computing the optimality conditions: Given a solution ŵ of problem (1.22), it is then necessary to check whether ŵ is also a solution for the original problem (1.1), without explicitly forcing the components in J c to be equal to zero. This test relies on the duality gaps of problems (1.22) and (1.1). In particular, if ŵ is a solution of problem (1.22), it follows from Proposition 3 that

f ( ŵ) + λΩ( ŵ) + f * (∇f ( ŵ)) = 0.
In fact, the Lagrangian parameter associated with the equality constraint ensures the feasibility of the dual variable formed from the gradient of f at ŵ. In turn, this guarantees that the duality gap of problem (1.22) vanishes. The candidate ŵ is now a solution of the full problem (1.1), i.e., without the equality constraint, if and only if

Ω * (∇f ( ŵ)) ≤ λ.
(1.23)

Introduction and Related Work

Condition (1.23) points out that the dual norm Ω * is a key quantity to monitor the progress of the working-set algorithm (Jenatton et al., 2011a). In simple settings, for instance when Ω is the ℓ 1 -norm, checking condition (1.23) can be easily computed since Ω * is just the ℓ ∞ -norm. In this case, condition (1.23) becomes | ∇f ( ŵ) j | ≤ λ, for all j in {1, . . . , p}.

Note that by using the optimality of problem (1.22), the components of the gradient of f indexed by J are already guaranteed to be no greater than λ.

For more general sparsity-inducing norms with overlapping groups of variables, the dual norm Ω * cannot be computed easily anymore, prompting the need for approximations and upper-bounds of Ω * (Bach, 2008a;Jenatton et al., 2011a;[START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF]. • Strategy for the growth of the working set: If condition (1.23) is not satisfied for the current working set J, some inactive variables in J c have to become active. This point raises the questions of how many and how these variables should be chosen.

First, depending on the structure of Ω, a single or a group of inactive variables have to be considered to enter the working set. Furthermore, one natural way to proceed is to look at the variables that violate condition (1.23) most. In the example of ℓ 1 -regularized least squares regression with normalized predictors, this strategy amounts to selecting the inactive variable that has the highest correlation with the current residual. The working-set algorithms we have described so far aim at solving problem (1.1) for a fixed value of the regularization parameter λ. However, for specific types of loss and regularization functions (e.g., see [START_REF] Rosset | Piecewise linear regularized solution paths[END_REF], the set of solutions of problem (1.1) can be obtained efficiently for all possible values of λ, which is exploited by homotopy algorithms (Osborne et al., 2000b) such as LARS [START_REF] Efron | Least angle regression[END_REF].

Extensions

Other methods could be considered for solving efficiently problem (1.1) when Ω is a structured sparsity-inducing norm. For instance, we may design quasi-Newton strategies (Schmidt et al., 2011a), or further develop augmented-Lagrangian techniques [START_REF] Combettes | Proximal splitting methods in signal processing[END_REF][START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF], as already pioneered by [START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF]; [START_REF] Qin | Structured sparsity via alternating directions methods[END_REF]. In the same vein as augmented-Lagrangian techniques, there is a last extension which would be interesting to consider in the context of norms with overlapping groups of variables. Starting from the objective min w∈R p f (w) + λ g∈G w g , for any norm . , we may introduce duplicate variables for each group g ∈ G with the constraint of being equal to w g , hence resulting in the new problem:

min w∈R p ,u∈R s G f (w) + λ g∈G u g , such that u = Gw,

Some Ingredients to Study Statistical Properties of Sparse Methods

where s G g∈G |g| and the matrix G ∈ R s G ×p encodes the group structure of G. The nice feature of the problem above is to replace the norm with overlaps by one penalty without overlapping groups of variables. This is however achieved at the cost of (1) dealing with a larger problem (i.e., with p + g∈G |g| parameters), and (2) being able to properly handle (e.g., via projections) the new equality constraint. Noticing that G is full column-rank when G spans the set 1; p and does not contain two identical groups, we may go one step further and try to solve the equivalent problem

min u∈R s G f ((G ⊤ G) -1 G ⊤ u) + λ g∈G u g , such that u ∈ span(G).
Interesting questions for future work would consist in studying to what extent do we need to keep the constraint u ∈ span(G) to go back to the domain of w, and when the inverse of G ⊤ G can be computed efficiently thanks to the structure of G.

Conclusions

We briefly summarize this review of optimization techniques in the light of the difficulties related to the use of norms with overlapping groups of variables. The optimization involving Ω built from overlapping groups of variables is deemed challenging because the computations of its proximal operator and of its dual norm are both nontrivial sub-problems, with a computational cost typically in the same order as that of the full initial problem. In particular, we do not have closed-form solutions available anymore. As a result, the applications of schemes based on proximal operators (or projections) and/or tests of optimality via the dual norm (e.g., in working-set/homotopy strategies) become tricky, or at least, not straightforward. Moreover, and as explained above, block-coordinate descent methods do not apply directly to the case of overlapping groups of variables. Finally, reweighted-ℓ 2 schemes are generally limited by the inversion of linear systems they have to repeatedly perform.

Our different simulations and experiments have shown that proximal methods appear so far as the most efficient and versatile approach. Coupling them with working-set strategies (see Section 2.4 in Chapter 2) is of course possible, but we sometimes observed that the bounds on the optimality conditions on which these strategies rely (see Propositions 7 and 8 in Chapter 2) might not be as tight as what we would need in practice.

This section concludes the introduction to the concepts from convex optimization which we require in the remainder of the thesis. We now review tools and notions to analyze some statistical properties of the solutions of problem (1.1).

Introduction and Related Work

For convenience, we adopt in the subsequent paragraphs the notations from the statistics/machine-learning setting, similarly to the Lasso problem in Eq. (1.2).

Several Criteria and Measures of Quality

From now on, let us concentrate on linear models. We assume that the output, or response, y is generated from the linear combination of a p-dimensional observation x ∈ R p with a true model w ⋆ ∈ R p . In addition, we shall assume some corruption from noise, in the form of, typically sub-Gaussian, centered random variable ε. In other words, we have y = x ⊤ w ⋆ + ε.

(1.24)

Stated in this way, the only source of randomness corresponds to the noise, and we then speak about fixed-design setting. Random-design analysis (e.g, [START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF], in the case of Lasso) can also be conducted by adding some probabilistic assumptions on the observations. We will assume that the true model w ⋆ has a sparse, structured nonzero pattern

J ⋆ = {j ∈ 1; p ; w ⋆ j = 0}, with |J ⋆ | ≪ p.
As an estimator for w ⋆ , it is thus natural to take any solution of problem (1.1), with Ω being a well-chosen structured sparsityinducing norm. Let us denote by ŵ one of these solutions.11 There are now several criteria of interest (for more details and formal definitions, see, e.g., [START_REF] Liu | Nonparametric Learning in High Dimensions[END_REF]:

• Estimation performance: For this first criterion, we are interested in how far ŵ is from the true model, that is, how large ŵw ⋆ is for some norm . . • Prediction performance: In this case, we might well be far from the true model w ⋆ ; instead, we want predictions to be accurate, that is, the quantity X( ŵ-w ⋆ ) to be small for some norm . , where X ∈ R n×p stands for a set of n observations in R p stacked row-wise. • Model-selection consistency: This last criterion, also referred to as support consistency or sparsistency, focuses on recovering the true sparsity pattern J ⋆ of w ⋆ . In other words, we want to have Pr {j ∈ 1; p ; ŵj = 0} = J ⋆ ≈ 1.

The probability above has to be understood as being taken with respect to all sources of randomness in the problem. In this thesis, we mostly study the third criterion. Furthermore, we ideally wish to obtain a non-asymptotic characterization of this criterion in order to better understand the roles and the scaling of (n, p, |J ⋆ |). Finally, even tough it has not been listed above, stability (e.g., of the estimated nonzero pattern) is another important measure. It has lately been the focus of a great deal of research, see, e.g., Bach (2008c); [START_REF] Meinshausen | Stability selection[END_REF]; Liu et al. (2010b).

We now describe an important structure of proof which is recurring in the study of sparse models where estimators result from a convex program.

1.6. Some Ingredients to Study Statistical Properties of Sparse Methods

The Primal-Dual "Witness" Proof

In this section, we present a method to characterize model-selection consistency (i.e., the third criterion above) when the estimator is derived from a convex program. Our situation is exactly in this scope, since we consider any solution of problem (1.1) for some structured sparsity-inducing norm.

This scheme of proof appeared in many places, for instance, to study the structures of graphical models [START_REF] Wainwright | High-dimensional graphical model selection using ℓ 1 -regularized logistic regression[END_REF][START_REF] Ravikumar | Model selection in gaussian graphical models: High-dimensional consistency of l1-regularized mle[END_REF][START_REF] Jalali | On learning discrete graphical models using group-sparse regularization[END_REF] and to prove exact support recovery of Lasso [START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF], group Lasso (Bach, 2008b;[START_REF] Nardi | On the asymptotic properties of the group Lasso estimator for linear models[END_REF] and other sparse estimators (Jenatton et al., 2011a).

We now describe the different steps of the primal-dual "witness" proof:

• Construction of an "oracle" solution: This first step consists of building a primal candidate starting from the knowledge of J ⋆ . More precisely, we consider ŵoracle = arg min

w∈R p f (w) + λΩ(w) , such that w j = 0 for any j ∈ [J ⋆ ] c , (1.25)
where we have assumed that the solution ŵoracle is unique (typically through assumptions on the observations and choice of λ). By construction, we know that ŵoracle has the correct sparsity pattern; we will have to prove that ŵoracle is also solution to the full problem, without the "oracle" equality constraint. In the Lasso case, problem (1.25) is equivalent to

min w J ⋆ ∈R |J ⋆ | 1 2 y -X J ⋆ w J ⋆ 2 2 + λ w J ⋆ 1 .
• Defining a candidate for the dual variable: Let us denote by ŵ a solution of problem (1.25) without the equality constraint. We know from Proposition 1 that we can characterize through a subgradient condition the optimality of ŵ. We therefore need to construct a subgradient candidate z whose entries in J ⋆ are in agreement with the optimality of ŵoracle . By separately treating z J ⋆ and z [J ⋆ ] c , we implicitly assume that the regularizer Ω and its subdifferential can be decomposed in this way; this is indeed the case for our structured sparsity-inducing norms (see Chapter 2). Generally, we will be able to express (explicitly or implicitly) z [J ⋆ ] c with respect to z J ⋆ . • Check the validity of the candidate: The third step of the procedure checks whether ŵoracle is optimal for the full problem, using the subgradient z previously defined as a certificate. In particular, we need to guarantee that z ∈ ∂λΩ( ŵoracle ). • Probabilistic control: While the reasoning in the previous steps was deterministic, it remains to define and control the probabilistic events over which the construction and the condition z ∈ ∂λΩ( ŵoracle ) hold with high probability. This technique of proof has to be related to working-set algorithms (see Section 1.5.2) in that they share a similar construction and use the same tools from convex analysis.

Introduction and Related Work

Zoology of Conditions Imposed on the Design Matrix

In the statistical analysis of the sparse models we are interested in, 12 theoretical guarantees-for prediction, estimation or support recovery-require some assumptions on the correlations of the p variables. Since Chapters 2 and 6 bring into play such assumptions, it is worth reviewing and presenting these concepts. For simplicity, we consider the ℓ 1 -norm setting, but the quantities we describe below can be extended to structured scenarios [START_REF] Negahban | A unified framework for highdimensional analysis of M-estimators with decomposable regularizers[END_REF].

In the case of noiseless linear models, i.e., Eq. (1.24) with ε = 0, it is interesting to study when the true vector w ⋆ is exactly recovered by a procedure based on ℓ 1 -norm minimization. For a sparsity level of |J ⋆ |, a necessary and sufficient condition for exact recovery is known as the null space property with parameter |J ⋆ | [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF][START_REF] Feuer | On sparse representation in pairs of bases[END_REF][START_REF] Cohen | Compressed sensing and best k-term approximation[END_REF]. As soon as the data are corrupted by some noise, exact recovery becomes impossible and it is then sensible to focus on the estimation error ŵw ⋆ 2 . We thus analyze under which sufficient condition of the null space property, this error can be proved to be small.

We review below several sufficient conditions, starting from the most restrictive one (but also the simplest one). We notably illustrate these conditions by their sampling complexity in the case of random Gaussian matrices with i.i.d. entries, that is, under which scaling of (n, p, |J ⋆ |) the conditions hold with high probability. In the subsequent, we use the notation a b if there exists some constant K > 0 such that a ≥ Kb holds. 13 Moreover, we shall assume that the data matrix X ∈ R n×p composed of n observations stacked row-wise has its columns normalized with unit ℓ 2 -norm.

• Elementwise incoherence condition: This first sufficient condition is rather intuitive, and explicitly asks for a control of the off-diagonal terms of the covariance matrix X ⊤ X [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF][START_REF] Feuer | On sparse representation in pairs of bases[END_REF][START_REF] Lounici | Sup-norm convergence rate and sign concentration property of lasso and dantzig estimators[END_REF], that is,

max j,k∈ 1;p ,j =k 1 n [X ⊤ X] jk 1 |J ⋆ | .
For random Gaussian matrices, this inequality is known to be satisfied with high probability for the following scaling n |J ⋆ | 2 log(p) (e.g., see [START_REF] Candès | Near-ideal model selection by l1 minimization[END_REF].

• Restricted isometry: The introduction of this second condition is slightly more involved. For s ∈ N, we introduce the s-restricted isometry constant δ s > 0 (Candes and Tao, 2005) as the smallest scalar such that

(1 -δ s ) a 2 2 ≤ Xa 2 2 ≤ (1 + δ s ) a 2 2 ,
hold for any vector a ∈ R p with |{j ∈ 1; p ; a j = 0}| ≤ s. In words, the previous equation says that all sub-matrices involving at most s columns of X behave like 12. We do not discuss the body of work that do not assume any computational limit, for instance, aggregated estimators (e.g., see Section 3.5 in Audibert, 2010, and references therein). In this case, no correlation-based assumptions are needed.

13. The usual notation is a = Ω(b), but we keep this symbol to represent the regularization term.

1.6. Some Ingredients to Study Statistical Properties of Sparse Methods an isometry. It can be shown [START_REF] Candes | Decoding by linear programming[END_REF][START_REF] Baraniuk | A simple proof of the restricted isometry property for random matrices[END_REF] that if the following inequality is valid

δ s + δ 2s + δ 3s < 1,
then, the null space property with parameter s is also satisfied. Moreover, for random Gaussian matrices, we obtain the scaling n |J ⋆ | log(p/|J ⋆ |) [START_REF] Mendelson | Uniform uncertainty principle for bernoulli and subgaussian ensembles[END_REF], where we can note the improvement with respect to the incoherence condition. [START_REF] Zhou | Restricted eigenvalue conditions on subgaussian random matrices[END_REF]. Formally, there exist (γ, κ) strictly positive scalars such that

1 n Xa 2 2 ≥ γ 2 a 2 2 for any vector a ∈ R p with a J ⋆ 1 ≤ κ a [J ⋆ ] c 1 .
The scaling for random Gaussian matrices was recently proved by [START_REF] Zhou | Restricted eigenvalue conditions on subgaussian random matrices[END_REF] 14 and leads to n |J ⋆ | log(p/|J ⋆ |). Again, there is an improvement upon the incoherence condition. Before concluding this section, there is another correlation condition worth being discussed. The irrepresentability condition [START_REF] Fuchs | Recovery of exact sparse representations in the presence of bounded noise[END_REF][START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Tropp | Just relax: Convex programming methods for identifying sparse signals in noise[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF] is central when the criterion of interest is the model-selection consistency, that is, the ability of the estimator to retrieve the nonzero pattern of w ⋆ . This is to be contrasted with the previous conditions which are primarily introduced to control the estimation error. Moreover, the irrepresentability condition is tight since it is proved to be necessary and sufficient [START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF]. Formally, it can be stated as follows:

[X ⊤ X] [J ⋆ ] c J ⋆ [X ⊤ X] J ⋆ J ⋆ -1 sign(w ⋆ ) J ⋆ ∞ < 1.
As opposed to the previously-described criteria that depend only on |J ⋆ |, the condition above explicitly brings into play the sign vector sign(w ⋆ ) and the set J ⋆ . At the cost of being more restrictive, we can further maximize out over all possible sign vectors and supports of size |J ⋆ | [START_REF] Tropp | Just relax: Convex programming methods for identifying sparse signals in noise[END_REF]; the resulting criterion is shown to be weaker than the one based on restricted eigenvalue [START_REF] Van De Geer | On the conditions used to prove oracle results for the Lasso[END_REF]. Eventually, we discuss in Chapter 2 how the irrepresentability condition is modified to adapt to structured settings.

Thus far, we have not taken into account any computational aspect. While it is relevant to look at ensembles of random matrices in the context of compressed sensing, the situation is different in statistics/machine-learning where the design matrix X is fixed. It would therefore be interesting to be able to compute some tests of these sufficient 1. Introduction and Related Work conditions: Some works recently investigated this direction of research, e.g., d 'Aspremont et al. (2008); [START_REF] Juditski | Accuracy guarantees for l1-recovery[END_REF].

We refer the readers to [START_REF] Van De Geer | On the conditions used to prove oracle results for the Lasso[END_REF] for a complete comparison of various assumptions made in the analysis of the Lasso.

Introduction

Sparse linear models have emerged as a powerful framework to deal with various supervised estimation tasks, in machine learning as well as in statistics and signal processing. These models basically seek to predict an output by linearly combining only a small subset of the features describing the data. To simultaneously address this variable selection and the linear model estimation, ℓ 1 -norm regularization has become a popular tool, that benefits both from efficient algorithms (see, e.g., [START_REF] Efron | Least angle regression[END_REF][START_REF] Lee | Efficient sparse coding algorithms[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Yuan | Comparison of optimization methods and software for large-scale l1-regularized linear classification[END_REF], and multiple references therein) and well-developed theory for generalization properties and variable selection consistency [START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF][START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF][START_REF] Zhang | Some sharp performance bounds for least squares regression with l1 regularization[END_REF].

When regularizing by the ℓ 1 -norm, sparsity is yielded by treating each variable individually, regardless of its position in the input feature vector, so that existing relationships and structures between the variables (e.g., spatial, hierarchical or related to the 2. Understanding the Properties of Structured Sparsity-Inducing Norms physics of the problem at hand) are merely disregarded. However, many practical situations could benefit from this type of prior knowledge, potentially both for interpretability purposes and for improved predictive performance.

For instance, in neuroimaging, one is interested in localizing areas in functional magnetic resonance imaging (fMRI) or magnetoencephalography (MEG) signals that are discriminative to distinguish between different brain states [START_REF] Gramfort | Improving M/EEG source localization with an intercondition sparse prior[END_REF]Xiang et al., 2009, and references therein). More precisely, fMRI responses consist in voxels whose three-dimensional spatial arrangement respects the anatomy of the brain. The discriminative voxels are thus expected to have a specific localized spatial organization [START_REF] Xiang | Boosting with spatial regularization[END_REF], which is important for the subsequent identification task performed by neuroscientists. In this case, regularizing by a plain ℓ 1 -norm to deal with the ill-conditionedness of the problem (typically only a few fMRI responses described by tens of thousands of voxels) would ignore this spatial configuration, with a potential loss in interpretability and performance.

Similarly, in face recognition, robustness to occlusions can be increased by considering as features, sets of pixels that form small convex regions on the face images (Jenatton et al., 2010b). Again, a plain ℓ 1 -norm regularization fails to encode this specific spatial locality constraint (Jenatton et al., 2010b). The same rationale supports the use of structured sparsity for background subtraction tasks [START_REF] Cevher | Sparse signal recovery using markov random fields[END_REF][START_REF] Huang | Learning with structured sparsity[END_REF]Mairal et al., 2010b). Still in computer vision, object and scene recognition generally seek to extract bounding boxes in either images [START_REF] Harzallah | Combining efficient object localization and image classification[END_REF] or videos [START_REF] Dalal | Human detection using oriented histograms of flow and appearance[END_REF]. These boxes concentrate the predictive power associated with the considered object/scene class, and have to be found by respecting the spatial arrangement of the pixels over the images. In videos, where series of frames are studied over time, the temporal coherence also has to be taken into account. An unstructured sparsityinducing penalty that would disregard this spatial and temporal information is therefore not adapted to select such boxes.

Another example of the need for higher-order prior knowledge comes from bioinformatics. Indeed, for the diagnosis of tumors, the profiles of array-based comparative genomic hybridization (arrayCGH) can be used as inputs to feed a classifier [START_REF] Rapaport | Classification of arrayCGH data using fused SVM[END_REF]. These profiles are characterized by plenty of variables, but only a few samples of such profiles are available, prompting the need for variable selection. Because of the specific spatial organization of bacterial artificial chromosomes along the genome, the set of discriminative features is expected to have specific contiguous patterns. Using this prior knowledge on top of a standard sparsity-inducing method leads to improvement in classification accuracy [START_REF] Rapaport | Classification of arrayCGH data using fused SVM[END_REF]. In the context of multi-task regression, a genetic problem of interest is to find a mapping between a small subset of single nucleotide polymorphisms (SNP's) that have a phenotypic impact on a given family of genes [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF]. This target family of genes has its own structure, where some genes share common genetic characteristics, so that these genes can be embedded into a underlying hierarchy [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF]. Exploiting directly this hierarchical information in the regularization term outperforms the unstructured approach with a standard ℓ 1 -norm. Such hierarchical structures have been likewise useful in the context 2.1. Introduction of wavelet regression [START_REF] Baraniuk | Model-based compressive sensing[END_REF][START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF][START_REF] Huang | Learning with structured sparsity[END_REF]Jenatton et al., 2011c), kernel-based non linear variable selection (Bach, 2008a) and for topic modeling (Jenatton et al., 2011c).

These real world examples motivate the need for the design of sparsity-inducing regularization schemes, capable of encoding more sophisticated prior knowledge about the expected sparsity patterns.

As mentioned above, the ℓ 1 -norm focuses only on cardinality and cannot easily specify side information about the patterns of nonzero coefficients ("nonzero patterns") induced in the solution, since they are all theoretically possible. Group ℓ 1 -norms [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF][START_REF] Roth | The group-Lasso for generalized linear models: uniqueness of solutions and efficient algorithms[END_REF][START_REF] Huang | The benefit of group sparsity[END_REF] consider a partition of all variables into a certain number of subsets and penalize the sum of the Euclidean norms of each one, leading to selection of groups rather than individual variables. Moreover, recent works have considered overlapping but nested groups in constrained situations such as trees and directed acyclic graphs [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]Bach, 2008a;[START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF]Jenatton et al., 2010aJenatton et al., , 2011c;;[START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF].

In this chapter, we consider all possible sets of groups and characterize exactly what type of prior knowledge can be encoded by considering sums of norms of overlapping groups of variables. Before describing how to go from groups to nonzero patterns (or equivalently zero patterns), we show that it is possible to "reverse-engineer" a given set of nonzero patterns, i.e., to build the unique minimal set of groups that will generate these patterns. This allows the automatic design of sparsity-inducing norms, adapted to target sparsity patterns. We give in Section 2.3 some interesting examples of such designs in specific geometric and structured configurations, which covers the type of prior knowledge available in the real world applications described previously.

As will be shown in Section 2.3, for each set of groups, a notion of hull of a nonzero pattern may be naturally defined. For example, in the particular case of the twodimensional planar grid considered in this chapter, this hull is exactly the axis-aligned bounding box or the regular convex hull. We show that, in our framework, the allowed nonzero patterns are exactly those equal to their hull, and that the hull of the relevant variables is consistently estimated under certain conditions, both in low and high-dimensional settings. Moreover, we present in Section 2.4 an efficient active set algorithm that scales well to high dimensions. Finally, we illustrate in Section 2.6 the behavior of our norms with synthetic examples on specific geometric settings, such as lines and two-dimensional grids.

Notation. For any finite set A with cardinality |A|, we also define the |A|-tuple (y a ) a∈A ∈ R p×|A| as the collection of p-dimensional vectors y a indexed by the elements of A. Furthermore, for two vectors x and y in R p , we denote by x•y = (x 1 y 1 , . . . , x p y p ) ⊤ ∈ R p the elementwise product of x and y.

Understanding the Properties of Structured Sparsity-Inducing Norms

Regularized Risk Minimization

We consider the problem of predicting a random variable Y ∈ Y from a (potentially non random) vector x ∈ R p , where Y is the set of responses, typically a subset of R. We assume that we are given n observations (x i , y i ) ∈ R p × Y, for i ∈ 1; n . We define the empirical risk of a loading vector w ∈ R p as L(w) = 1 n n i=1 ℓ y i , w ⊤ x i , where ℓ : Y × R → R + is a loss function. We assume that ℓ is convex and continuously differentiable with respect to the second parameter. Typical examples of loss functions are the square loss for least squares regression, i.e., ℓ(y, ŷ) = 1 2 (yŷ) 2 with y ∈ R, and the logistic loss ℓ(y, ŷ) = log(1 + e -y ŷ) for logistic regression, with y ∈ {-1, 1}.

We focus on a general family of sparsity-inducing norms that allow the penalization of subsets of variables grouped together. Let us denote by G a subset of the power set of 1; p such that g∈G g = 1; p , i.e., a spanning set of subsets of 1; p . Note that G does not necessarily define a partition of 1; p , and therefore, it is possible for elements of G to overlap. We consider the norm Ω defined by

Ω(w) = g∈G j∈g (ω g j ) 2 |w j | 2 1 2 = g∈G ω g • w 2 , ( 2.1)
where (ω g ) g∈G is a |G|-tuple of p-dimensional vectors such that ω g j > 0 if j ∈ g and ω g j = 0 otherwise. A same variable w j belonging to two different groups g 1 , g 2 ∈ G is allowed to be weighted differently in g 1 and g 2 (by respectively ω g 1 j and ω g 2 j ). We do not study the more general setting where each ω g would be a (non-diagonal) positivedefinite matrix, which we defer to future work. Note that a larger family of penalties with similar properties may be obtained by replacing the ℓ 2 -norm in Eq. (2.1) by other ℓ q -norm, q > 1 [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]. Moreover, non-convex alternatives to Eq. (2.1) with quasi-norms in place of norms may also be interesting, in order to yield sparsity more aggressively (see, e.g., Jenatton et al., 2010b).

This general formulation has several important sub-cases that we present below, the goal of this chapter being to go beyond these, and to consider norms capable to incorporate richer prior knowledge.

• ℓ 2 -norm: G is composed of one element, the full set 1; p .

• ℓ 1 -norm: G is the set of all singletons, leading to the Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] for the square loss. • ℓ 2 -norm and ℓ 1 -norm: G is the set of all singletons and the full set 1; p , leading (up to the squaring of the ℓ 2 -norm) to the Elastic net [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF] for the square loss. • Group ℓ 1 -norm: G is any partition of 1; p , leading to the group-Lasso for the square loss [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF]). • Hierarchical norms: when the set 1; p is embedded into a tree [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF] or more generally into a directed acyclic graph (Bach, 2008a), then a set of p groups, each of them composed of descendants of a given variable, is considered. 40

Groups and Sparsity Patterns

We study the following regularized problem:

min w∈R p 1 n n i=1 ℓ y i , w ⊤ x i + µΩ(w), (2.2)
where µ ≥ 0 is a regularization parameter. Note that a non-regularized constant term could be included in this formulation, but it is left out for simplicity. We denote by ŵ any solution of Eq. 2.2. Regularizing by linear combinations of (non-squared) ℓ 2 -norms is known to induce sparsity in ŵ [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]; our grouping leads to specific patterns that we describe in the next section.

Groups and Sparsity Patterns

We now study the relationship between the norm Ω defined in 2.1 and the nonzero patterns the estimated vector ŵ is allowed to have. We first characterize the set of nonzero patterns, then we provide forward and backward procedures to go back and forth from groups to patterns.

Stable Patterns Generated by G

The regularization term Ω(w) = g∈G ω g •w 2 is a mixed (ℓ 1 , ℓ 2 )-norm [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]. At the group level, it behaves like an ℓ 1 -norm and therefore, Ω induces group sparsity. In other words, each ω g • w, and equivalently each w g (since the support of ω g is exactly g), is encouraged to go to zero. On the other hand, within the groups g ∈ G, the ℓ 2 -norm does not promote sparsity. Intuitively, for a certain subset of groups G ′ ⊆ G, the vectors w g associated with the groups g ∈ G ′ will be exactly equal to zero, leading to a set of zeros which is the union of these groups, g∈G ′ g. Thus, the set of allowed zero patterns should be the union-closure of G, i.e. (see Figure 2.1 for an example):

Z = g∈G ′ g; G ′ ⊆ G . (2.
3)

The situation is however slightly more subtle as some zeros can be created by chance (just as regularizing by the ℓ 2 -norm may lead, though it is unlikely, to some zeros). Nevertheless, Theorem 1 shows that, under mild conditions, the previous intuition about the set of zero patterns is correct. Note that instead of considering the set of zero patterns Z, it is also convenient to manipulate nonzero patterns, and we define

P = g∈G ′ g c ; G ′ ⊆ G = z c ; z ∈ Z . (2.4)
We can equivalently use P or Z by taking the complement of each element of these sets.

The following two results characterize the solutions of the problem (2.2). We first gives sufficient conditions under which this problem has a unique solution. We then

Understanding the Properties of Structured Sparsity-Inducing Norms

Figure 2.1: Groups and induced nonzero pattern: three sparsity-inducing groups (middle and right, denoted by {g 1 , g 2 , g 3 }) with the associated nonzero pattern which is the complement of the union of groups, i.e., (g 1 ∪ g 2 ∪ g 3 ) c (left, in black).

formally prove the aforementioned intuition about the zero patterns of the solutions of (2.2), namely they should belong to Z. In the following two results (see proofs in Appendix A.1.1 and Appendix A.1.2), we assume that ℓ : (y, y ′ ) → ℓ(y, y ′ ) is nonnegative, twice continuously differentiable with positive second derivative with respect to the second variable and non-vanishing mixed derivative, i.e., for any y, y

′ in R, ∂ 2 ℓ ∂y ′2 (y, y ′ ) > 0 and ∂ 2 ℓ ∂y∂y ′ (y, y ′ ) = 0. Proposition 5 (Uniqueness) Let Q denote the Gram matrix 1 n n i=1 x i x ⊤ i .
We consider the optimization problem in (2.2) with µ > 0. If Q is invertible or if the group 1; p belongs to G, then the problem in (2.2) admits a unique solution.

Note that the invertibility of the matrix Q requires p ≤ n. For high-dimensional settings, the uniqueness of the solution will hold when {1, . . . , p} belongs to G, or as further discussed at the end of the proof, as soon as for any j, k ∈ {1, . . . , p}, there exists a group g ∈ G which contains both j and k. Adding the group {1, . . . , p} to G will in general not modify P (and Z), but it will cause G to lose its minimality (in a sense introduced in the next subsection). Furthermore, adding the full group {1, . . . , p} has to be put in parallel with the equivalent (up to the squaring) ℓ 2 -norm term in the elastic-net penalty [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF], whose effect is to notably ensure strong convexity. For more sophisticated uniqueness conditions that we have not explored here, we refer the readers to Osborne et al. (2000a, Theorem 1, 4 and 5), Rosset et al. (2004, Theorem 5) or Dossal (2007, Theorem 3) in the Lasso case, and [START_REF] Roth | The group-Lasso for generalized linear models: uniqueness of solutions and efficient algorithms[END_REF] for the group Lasso setting. We now turn to the result about the zero patterns of the solution of the problem in (2.2):

Theorem 1 (Stable patterns)

Assume that Y = (y 1 , . . . , y n ) ⊤ is a realization of an absolutely continuous probability distribution. Let k be the maximal number such that any k rows of the matrix (x 1 , . . . , x n ) ∈ R p×n are linearly independent. For µ > 0, any solution of the problem in (2.2) with at most k-1 nonzero coefficients has a zero pattern in Z = g∈G ′ g; G ′ ⊆ G

Groups and Sparsity Patterns

almost surely.

In other words, when Y = (y 1 , . . . , y n ) ⊤ is a realization of an absolutely continuous probability distribution, the sparse solutions have a zero pattern in Z = g∈G ′ g; G ′ ⊆ G almost surely. As a corollary of our two results, if the Gram matrix Q is invertible, the problem in (2.2) has a unique solution, whose zero pattern belongs to Z almost surely. Note that with the assumption made on Y , Theorem 1 is not directly applicable to the classification setting. Based on these previous results, we can look at the following usual special cases from Section 2.2 (we give more examples in Section 2.3.5):

• ℓ 2 -norm: the set of allowed nonzero patterns is composed of the empty set and the full set 1; p . • ℓ 1 -norm: P is the set of all possible subsets. • ℓ 2 -norm and ℓ 1 -norm: P is also the set of all possible subsets.

• Group ℓ 1 -norm: P = Z is the set of all possible unions of the elements of the partition defining G. • Hierarchical norms: the set of patterns P is then all sets J for which all ancestors of elements in J are included in J (Bach, 2008a). Two natural questions now arise: (1) starting from the groups G, is there an efficient way to generate the set of nonzero patterns P; (2) conversely, and more importantly, given P, how can the groups G-and hence the norm Ω(w)-be designed?

General Properties of G, Z and P

We now study the different properties of the set of groups G and its corresponding sets of patterns Z and P.

Closedness. The set of zero patterns Z (respectively, the set of nonzero patterns P) is closed under union (respectively, intersection), that is, for all K ∈ N and all z 1 , . . . , z K ∈ Z, K k=1 z k ∈ Z (respectively, p 1 , . . . , p K ∈ P, K k=1 p k ∈ P). This implies that when "reverse-engineering" the set of nonzero patterns, we have to assume it is closed under intersection. Otherwise, the best we can do is to deal with its intersection-closure. For instance, if we consider a sequence (see Figure 2.4), we cannot take P to be the set of contiguous patterns with length two, since the intersection of such two patterns may result in a singleton (that does not belong to P).

Minimality. If a group in G is the union of other groups, it may be removed from G without changing the sets Z or P. This is the main argument behind the pruning backward algorithm in Section 2.3.3. Moreover, this leads to the notion of a minimal set G of groups, which is such that for all G ′ ⊆ Z whose union-closure spans Z, we have G ⊆ G ′ . The existence and uniqueness of a minimal set is a consequence of classical results in set theory [START_REF] Doignon | Knowledge Spaces[END_REF]. The elements of this minimal set are usually referred to as the atoms of Z.

Minimal sets of groups are attractive in our setting because they lead to a smaller number of groups and lower computational complexity-for example, for 2 dimensional-

Understanding the Properties of Structured Sparsity-Inducing Norms

Figure 2.2: G-adapted hull: the pattern of variables I (left and middle, in red) and its hull (left and right, hatched square) that is defined by the complement of the union of groups that do not intersect I, i.e., (g 1 ∪ g 2 ∪ g 3 ) c . grids with rectangular patterns, we have a quadratic possible number of rectangles, i.e., |Z| = O(p 2 ), that can be generated by a minimal set G whose size is

|G| = O( √ p).
Hull. Given a set of groups G, we can define for any subset I ⊆ 1; p the G-adapted hull, or simply hull, as:

Hull(I) = g∈G, g∩I=∅ g c ,
which is the smallest set in P containing I (see Figure 2.2); we always have I ⊆ Hull(I) with equality if and only if I ∈ P. The hull has a clear geometrical interpretation for specific sets G of groups. For instance, if the set G is formed by all vertical and horizontal half-spaces when the variables are organized in a 2 dimensional-grid (see Figure 2.5), the hull of a subset I ⊂ {1, . . . , p} is simply the axis-aligned bounding box of I. Similarly, when G is the set of all half-spaces with all possible orientations (e.g., orientations ±π/4 are shown in Figure 2.6), the hull becomes the regular convex hull1 . Note that those interpretations of the hull are possible and valid only when we have geometrical information at hand about the set of variables.

Graphs of patterns.

We consider the directed acyclic graph (DAG) stemming from the Hasse diagram [START_REF] Cameron | Combinatorics: Topics, Techniques, Algorithms[END_REF] of the partially ordered set (poset) (G, ⊃). By definition, the nodes of this graph are the elements g of G and there is a directed edge from g 1 to g 2 if and only if G 1 ⊃ G 2 and there exists no g ∈ G such that g 1 ⊃ g ⊃ g 2 [START_REF] Cameron | Combinatorics: Topics, Techniques, Algorithms[END_REF]. We can also build the corresponding DAG for the set of zero patterns Z ⊃ G, which is a super-DAG of the DAG of groups (see Figure 2.3 for examples). Note that we obtain also the isomorphic DAG for the nonzero patterns P, although it corresponds to the poset (P, ⊂): this DAG will be used in the active set algorithm presented in Section 2.4. Prior works with nested groups [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]Bach, 2008a;[START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF]Jenatton et al., 2010a;[START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF] have also used a similar DAG structure, with the slight difference that in these works, the corresponding hierarchy of variables is built from the prior knowledge about the problem at hand (e.g., the tree of wavelets in [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF], the decomposition of kernels in Bach (2008a) or the hierarchy of genes in [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF]). The DAG we introduce here on the set of groups naturally and always comes up, with no assumption on the variables themselves (for which no DAG is defined in general).

From Patterns to Groups

We now assume that we want to impose a priori knowledge on the sparsity structure of a solution ŵ of our regularized problem in Eq. 2.2. This information can be exploited by restricting the patterns allowed by the norm Ω. Namely, from an intersection-closed set of zero patterns Z, we can build back a minimal set of groups G by iteratively pruning away in the DAG corresponding to Z, all sets which are unions of their parents. See Algorithm 1. This algorithm can be found under a different form in Doignon and Falmagne (1998)-we present it through a pruning algorithm on the DAG, which is natural in our context (the proof of the minimality of the procedure can be found in Appendix A.1.3). The complexity of Algorithm 1 is O(p|Z| 2 ). The pruning may reduce significantly the number of groups necessary to generate the whole set of zero patterns, sometimes from exponential in p to polynomial in p (e.g., the ℓ 1 -norm). In Section 2.3.5, we give other examples of interest where |G| (and |P|) is also polynomial in p. Output: Collection of zero patterns Z and nonzero patterns P.

Understanding the Properties of Structured Sparsity-Inducing Norms

Initialization: Z = {∅}. for m = 1 to M do C = {∅} for each Z ∈ Z do if (g m Z) and (∀g ∈ {g 1 , . . . , g m-1 }, g ⊆ Z ∪ g m ⇒ g ⊆ Z) then C ← C ∪ {Z ∪ g m } . end if end for Z ← Z ∪ C. end for P = {Z c ; Z ∈ Z}.

From Groups to Patterns

The forward procedure presented in Algorithm 2, taken from [START_REF] Doignon | Knowledge Spaces[END_REF], allows the construction of Z from G. It iteratively builds the collection of patterns by taking unions, and has complexity O(p|Z||G| 2 ). The general scheme is straightforward. Namely, by considering increasingly larger sub-families of G and the collection of patterns already obtained, all possible unions are formed. However, some attention needs to be paid while checking we are not generating a pattern already encountered. Such a verification is performed by the if condition within the inner loop of the algorithm. Indeed, we do not have to scan the whole collection of patterns already obtained (whose size can be exponential in |G|), but we rather use the fact that G generates Z.

Groups and Sparsity Patterns

Note that in general, it is not possible to upper bound the size of |Z| by a polynomial term in p, even when G is very small (indeed, |Z| = 2 p and |G| = p for the ℓ 1 -norm).

Examples

We now present several examples of sets of groups G, especially suited to encode geometric and temporal prior information.

Sequences.

Given p variables organized in a sequence, if we want only contiguous nonzero patterns, the backward algorithm will lead to the set of groups which are intervals [1, k] k∈{1,...,p-1} and [k, p] k∈{2,...,p} , with both |Z| = O(p 2 ) and |G| = O(p) (see Figure 2.4). Imposing the contiguity of the nonzero patterns is for instance relevant for the diagnosis of tumors, based on the profiles of arrayCGH [START_REF] Rapaport | Classification of arrayCGH data using fused SVM[END_REF]. Figure 2.4: (Left) The set of blue groups to penalize in order to select contiguous patterns in a sequence. (Right) In red, an example of such a nonzero pattern with its corresponding zero pattern (hatched area).

Two-dimensional grids. In Section 2.6, we notably consider for P the set of all rectangles in two dimensions, leading by the previous algorithm to the set of axis-aligned half-spaces for G (see Figure 2.5), with |Z| = O(p 2 ) and |G| = O( √ p). This type of structure is encountered in object or scene recognition, where the selected rectangle would correspond to a certain box inside an image, that concentrates the predictive power for a given class of object/scene [START_REF] Harzallah | Combining efficient object localization and image classification[END_REF]. Larger set of convex patterns can be obtained by adding in G half-planes with other orientations than vertical and horizontal. For instance, if we use planes with angles that are multiples of π/4, the nonzero patterns of P can have polygonal shapes with up to 8 faces. In this sense, if we keep on adding half-planes with finer orientations, the nonzero patterns of P can be described by polygonal shapes with an increasingly larger number of faces. The standard notion of convexity defined in R 2 would correspond to the situation where an infinite number of orientations is considered [START_REF] Soille | Morphological Image Analysis: Principles and Applications[END_REF]. See Figure 2.6. The number of groups is linear in √ p with constant growing linearly with the number of angles, while |Z| grows more rapidly (typically non-polynomially in the number of angles). Imposing such convex-like regions turns out to be useful in computer vision. For

Understanding the Properties of Structured Sparsity-Inducing Norms

instance, in face recognition, it enables the design of localized features that improve upon the robustness to occlusions (Jenatton et al., 2010b). In the same vein, regularizations with similar two-dimensional sets of groups have led to good performances in background subtraction tasks (Mairal et al., 2010b), where the pixel spatial information is crucial to avoid scattered results. Another application worth being mentioned is the design of topographic dictionaries in the context of image processing [START_REF] Kavukcuoglu | Learning invariant features through topographic filter maps[END_REF][START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF]. In this case, dictionaries self-organize and adapt to the underlying geometrical structure encoded by the two-dimensional set of groups. Extensions. The sets of groups presented above can be straightforwardly extended to more complicated topologies, such as three-dimensional spaces discretized in cubes or spherical volumes discretized in slices. Similar properties hold for such settings. For instance, if all the axis-aligned half-spaces are considered for G in a three-dimensional space, then P is the set of all possible rectangular boxes with |P| = O(p2 ) and |G| = O(p 1/3 ). Such three-dimensional structures are interesting to retrieve discriminative and local sets of voxels from fMRI/MEEG responses. In particular, they have recently proven useful for modeling brain resting-state activity (Varoquaux et al., 2010c). Moreover, while the two-dimensional rectangular patterns described previously are adapted to find bounding boxes in static images [START_REF] Harzallah | Combining efficient object localization and image classification[END_REF], scene recognition in videos requires to deal with a third temporal dimension [START_REF] Dalal | Human detection using oriented histograms of flow and appearance[END_REF]. This may be achieved by designing appropriate sets of groups, embedded in the three-dimensional space obtained by tracking the frames over time. Finally, in the context of matrixbased optimization problems, e.g., multi-task learning and dictionary learning, sets of groups G can also be designed to encode structural constraints the solutions must respect. This notably encompasses banded structures [START_REF] Levina | Sparse estimation of large covariance matrices via a nested Lasso penalty[END_REF] and simultaneous row/column sparsity for CUR matrix factorization [START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF].

Representation and computation of G. The sets of groups described so far can actually be represented in a same form, that lends itself well to the analysis of the next section. When dealing with a discrete sequence of length l (see Figure 2.4), we have

G = {g k -; k ∈ {1, . . . , l-1}} ∪ {g k + ; k ∈ {2, . . . , l}}, = G left ∪ G right , with g k -= {i; 1 ≤ i ≤ k} and g k + = {i; l ≥ i ≥ k}.
In other words, the set of groups G can be rewritten as a partition 2 in two sets of nested groups, G left and G right .

The same goes for a two-dimensional grid, with dimensions h×l (see Figure 2.5 and Figure 2.6). In this case, the nested groups we consider are defined based on the following groups of variables

g k,θ = {(i, j) ∈ {1, . . . , l} × {1, . . . , h}; cos(θ)i + sin(θ)j ≤ k},
where k ∈ Z is taken in an appropriate range. The nested groups we obtain in this way are therefore parameterized by an angle 3 θ, θ ∈ (-π; π]. We refer to this angle as an orientation, since it defines the normal vector cos(θ) sin(θ) to the line {(i, j) ∈ R 2 ; cos(θ)i + sin(θ)j = k}. In the example of the rectangular groups (see Figure 2.5), we have four orientations, with θ ∈ {0, π/2, -π/2, π}. More generally, if we denote by Θ the set of the orientations, we have

G = θ∈Θ G θ ,
where θ ∈ Θ indexes the partition of G in sets G θ of nested groups of variables. Although we have not detailed the case of R 3 , we likewise end up with a similar partition of G.

Understanding the Properties of Structured Sparsity-Inducing Norms

Optimization and Active Set Algorithm

For moderate values of p, one may obtain a solution for Eq. (2.2) using generic toolboxes for second-order cone programming (SOCP) whose time complexity is equal to O(p 3.5 + |G| 3.5 ) [START_REF] Boyd | Convex Optimization[END_REF], which is not appropriate when p or |G| are large. This time complexity corresponds to the computation of Eq. ( 2.2) for a single value of the regularization parameter µ.

We present in this section an active set algorithm (Algorithm 3) that finds a solution for Eq. ( 2.2) by considering increasingly larger active sets and checking global optimality at each step. When the rectangular groups are used, the total complexity of this method is in O(s max{p 1.75 , s 3.5 }), where s is the size of the active set at the end of the optimization. Here, the sparsity prior is exploited for computational advantages. Our active set algorithm needs an underlying black-box SOCP solver; in this chapter, we consider both a first order approach (see Appendix A.1.8) and a SOCP method 4 -in our experiments, we use SDPT3 [START_REF] Toh | SDPT3-a MATLAB software package for semidefinite programming, version 1.3[END_REF][START_REF] Tütüncü | Solving semidefinite-quadratic-linear programs using SDPT3[END_REF]. Our active set algorithm extends to general overlapping groups the work of Bach (2008a), by further assuming that it is computationally possible to have a time complexity polynomial in the number of variables p.

We primarily focus here on finding an efficient active set algorithm; we defer to future work the design of specific SOCP solvers, e.g., based on proximal techniques (see, e.g., [START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF], and numerous references therein), adapted to such non-smooth sparsity-inducing penalties.

Optimality Conditions: from Reduced Problems to Full Problems

It is simpler to derive the algorithm for the following regularized optimization problem 5 which has the same solution set as the regularized problem of Eq. ( 2.2) when µ and λ are allowed to vary [START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF], see Section 3.2):

min w∈R p 1 n n i=1 ℓ y i , w ⊤ x i + λ 2 [Ω(w)] 2 .
(2.5)

In active set methods, the set of nonzero variables, denoted by J, is built incrementally, and the problem is solved only for this reduced set of variables, adding the constraint w J c = 0 to Eq. (2.5). In the subsequent analysis, we will use arguments based on duality to monitor the optimality of our active set algorithm. We denote by L(w) = 1 n n i=1 ℓ y i , w ⊤ x i the empirical risk (which is by assumption convex and 4. The C++/Matlab code used in the experiments may be downloaded from the authors website. 5. It is also possible to derive the active set algorithm for the constrained formulation min w∈R p 1 n n i=1 ℓ yi, w ⊤ xi such that Ω(w) ≤ λ. We however observed that the penalized formulation tends to be empirically easier to tune (e.g., via cross-validation), as the performance is usually quite robust to small changes in µ, while it is not robust to small changes in λ. 50 2.4. Optimization and Active Set Algorithm continuously differentiable) and by L * its Fenchel-conjugate, defined as [START_REF] Boyd | Convex Optimization[END_REF][START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF]:

L * (u) = sup w∈R p {w ⊤ u -L(w)}.
The restriction of L to R |J| is denoted L J (w J ) = L( w) for wJ = w J and wJ c = 0, with Fenchel-conjugate L * J . Note that, as opposed to L, we do not have in general

L * J (κ J ) = L * (κ) for κJ = κ J and κJ c = 0.
For a potential active set J ⊂ 1; p which belongs to the set of allowed nonzero patterns P, we denote by G J the set of active groups, i.e., the set of groups G ∈ G such that G ∩ J = ∅. We consider the reduced norm Ω J defined on R |J| as

Ω J (w J ) = g∈G ω g J • w J 2 = g∈G J ω g J • w J 2 ,
and its dual norm

Ω * J (κ J ) = max Ω J (w J )≤1 w ⊤ J κ J , also defined on R |J| .
The next proposition (see proof in Appendix A.1.4) gives the optimization problem dual to the reduced problem (Eq. 2.6 below):

Proposition 6 (Dual Problems) Let J ⊆ 1; p . The following two problems min w J ∈R |J| L J (w J ) + λ 2 [Ω J (w J )] 2 , (2.6) max κ J ∈R |J| -L * J (-κ J ) - 1 2λ [Ω * J (κ J )] 2 , (2.7)
are dual to each other and strong duality holds. The pair of primal-dual variables {w J , κ J } is optimal if and only if we have

κ J = -∇L J (w J ), w ⊤ J κ J = 1 λ [Ω * J (κ J )] 2 = λ [Ω J (w J )] 2 .
As a brief reminder, the duality gap of a minimization problem is defined as the difference between the primal and dual objective functions, evaluated for a feasible pair of primal/dual variables [START_REF] Boyd | Convex Optimization[END_REF], see Section 5.5). This gap serves as a certificate of (sub)optimality: if it is equal to zero, then the optimum is reached, and provided that strong duality holds, the converse is true as well [START_REF] Boyd | Convex Optimization[END_REF], see Section 5.5).

The previous proposition enables us to derive the duality gap for the optimization problem Eq. 2.6, that is reduced to the active set of variables J. In practice, this duality gap will always vanish (up to the precision of the underlying SOCP solver), since we will sequentially solve (2.6) for increasingly larger active sets J. We now study how, starting from the optimality of the problem in (2.6), we can control the optimality, or

Understanding the Properties of Structured Sparsity-Inducing Norms

equivalently the duality gap, for the full problem Eq. (2.5). More precisely, the duality gap of the optimization problem Eq. 2.6 is

L J (w J ) + L * J (-κ J ) + λ 2 [Ω J (w J )] 2 + 1 2λ [Ω * J (κ J )] 2 = L J (w J ) + L * J (-κ J ) + w ⊤ J κ J + λ 2 [Ω J (w J )] 2 + 1 2λ [Ω * J (κ J )] 2 -w ⊤ J κ J ,
which is a sum of two nonnegative terms, the nonnegativity coming from the Fenchel-Young inequality [START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF]Boyd and Vandenberghe, 2004, Proposition 3.3.4 and Section 3.3.2 respectively). We can think of this duality gap as the sum of two duality gaps, respectively relative to L J and Ω J . Thus, if we have a primal candidate w J and we choose κ J = -∇L J (w J ), the duality gap relative to L J vanishes and the total duality gap then reduces to

λ 2 [Ω J (w J )] 2 + 1 2λ [Ω * J (κ J )] 2 -w ⊤ J κ J .
In order to check that the reduced solution w J is optimal for the full problem in Eq. ( 2.5), we pad w J with zeros on J c to define w and compute κ = -∇L(w), which is such that κ J = -∇L J (w J ). For our given candidate pair of primal/dual variables {w, κ}, we then get a duality gap for the full problem in Eq. ( 2.5) equal to

λ 2 [Ω(w)] 2 + 1 2λ [Ω * (κ)] 2 -w ⊤ κ = λ 2 [Ω(w)] 2 + 1 2λ [Ω * (κ)] 2 -w ⊤ J κ J = λ 2 [Ω(w)] 2 + 1 2λ [Ω * (κ)] 2 - λ 2 [Ω J (w J )] 2 - 1 2λ [Ω * J (κ J )] 2 = 1 2λ [Ω * (κ)] 2 -[Ω * J (κ J )] 2 = 1 2λ [Ω * (κ)] 2 -λw ⊤ J κ J .
Computing this gap requires computing the dual norm which itself is as hard as the original problem, prompting the need for upper and lower bounds on Ω * (see Propositions 7 and 8 for more details).

Active set algorithm

We can interpret the active set algorithm as a walk through the DAG of nonzero patterns allowed by the norm Ω. The parents Π P (J) of J in this DAG are exactly the patterns containing the variables that may enter the active set at the next iteration of Algorithm 3. The groups that are exactly at the boundaries of the active set (referred to as the fringe groups) are

F J = {g ∈ (G J ) c ; ∄g ′ ∈ (G J ) c , g ⊆ g ′ }, i.e.
, the groups that are not contained by any other inactive groups.

Figure 2.7: The active set (black) and the candidate patterns of variables, i.e. the variables in K\J (hatched in black) that can become active. The fringe groups are exactly the groups that have the hatched areas (i.e., here we have

F J = K∈Π P (J) G K \G J = {g 1 , g 2 , g 3 }).
In simple settings, e.g., when G is the set of rectangular groups, the correspondence between groups and variables is straightforward since we have F J = K∈Π P (J) G K \G J (see Figure 2.7). However, in general, we just have the inclusion ( K∈Π P (J) G K \G J ) ⊆ F J and some elements of F J might not correspond to any patterns of variables in Π P (J) (see Figure 2.8).

We now present the optimality conditions (see proofs in Appendix A.1.5) that monitor the progress of Algorithm 3: Proposition 7 (Necessary condition) If w is optimal for the full problem in Eq. ( 2.5), then

max K∈Π P (J) ∇L(w) K\J 2 h∈G K \G J ω h K\J ∞ ≤ -λw ⊤ ∇L(w) 1 2 . (N ) Proposition 8 (Sufficient condition) If max g∈F J    k∈g ∇L(w) k h∋k, h∈(G J ) c ω h k 2    1 2 ≤ λ(2ε -w ⊤ ∇L(w)) 1 2 , (S ε )
then w is an approximate solution for Eq. ( 2.5) whose duality gap is less than ε ≥ 0.

Note that for the Lasso, the conditions (N ) and (S 0 ) (i.e., the sufficient condition taken with ε = 0) are both equivalent (up to the squaring of Ω) to the condition

Understanding the Properties of Structured Sparsity-Inducing Norms

Figure 2.8: The active set (black) and the candidate patterns of variables, i.e. the variables in K\J (hatched in black) that can become active. The groups in red are those in K∈Π P (J) G K \G J , while the blue group is in F J \( K∈Π P (J) G K \G J ). The blue group does not intersect with any patterns in Π P (J).

∇L(w) J c ∞ ≤ -w ⊤ ∇L(w), which is the usual optimality condition [START_REF] Fuchs | Recovery of exact sparse representations in the presence of bounded noise[END_REF][START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF]. Moreover, when they are not satisfied, our two conditions provide good heuristics for choosing which K ∈ Π P (J) should enter the active set.

More precisely, since the necessary condition (N ) directly deals with the variables (as opposed to groups) that can become active at the next step of Algorithm 3, it suffices to choose the pattern K ∈ Π P (J) that violates most the condition.

The heuristics for the sufficient condition (S ε ) implies that, to go from groups to variables, we simply consider the group g ∈ F J violating the sufficient condition the most and then take all the patterns of variables K ∈ Π P (J) such that K ∩ g = ∅ to enter the active set. If g ∩ ( K∈Π P (J) K) = ∅, we look at all the groups h ∈ F J such that h ∩ g = ∅ and apply the scheme described before (see Algorithm 4).

A direct consequence of this heuristics is that it is possible for the algorithm to jump over the right active set and to consider instead a (slightly) larger active set as optimal. However if the active set is larger than the optimal set, then (it can be proved that) the sufficient condition (S 0 ) is satisfied, and the reduced problem, which we solve exactly, will still output the correct nonzero pattern.

Moreover, it is worthwhile to notice that in Algorithm 3, the active set may sometimes be increased only to make sure that the current solution is optimal (we only check a sufficient condition of optimality).

Algorithm 3 Active set algorithm

Input: Data {(x i , y i ), i = 1, . . . , n}, regularization parameter λ, Duality gap precision ε, maximum number of variables s. Output: Active set J, loading vector ŵ.

Initialization: J = {∅}, ŵ = 0. while (N ) is not satisfied and |J| ≤ s do Replace J by violating K ∈ Π P (J) in (N ). Solve the reduced problem min w J ∈R |J| L J (w J ) + λ 2 [Ω J (w J )] 2 to get ŵ. end while while (S ε ) is not satisfied and |J| ≤ s do Update J according to Algorithm 4. Solve the reduced problem min w J ∈R |J| L J (w J ) + λ 2 [Ω J (w J )] 2 to get ŵ. end while
Convergence of the active set algorithm. The procedure described in Algorithm 3 can terminate in two different states. If the procedure stops because of the limit on the number of active variables s, the solution might be suboptimal. Note that, in any case, we have at our disposal a upperbound on the duality gap.

Otherwise, the procedure always converges to an optimal solution, either (1) by validating both the necessary and sufficient conditions (see Propositions 7 and 8), ending up with fewer than p active variables and a precision of (at least) ε, or ( 2) by running until the p variables become active, the precision of the solution being given by the underlying solver.

Algorithm 4 Heuristics for the sufficient condition (S ε )

Let g ∈ F J be the group that violates (S ε ) most.

if (g ∩ ( K∈Π P (J) K) = ∅) then for K ∈ Π P (J) such that K ∩ g = ∅ do J ← J ∪ K. end for else for H ∈ F J such that H ∩ g = ∅ do for K ∈ Π P (J) such that K ∩ H = ∅ do J ← J ∪ K.

end for end for end if

Algorithmic complexity. We analyze in detail the time complexity of the active set algorithm when we consider sets of groups G such as those presented in the examples of Section 2.3.5. We recall that we denote by Θ the set of orientations in G (for more details, see Section 2.3.5). For such choices of G, the fringe groups F J reduces to the 2. Understanding the Properties of Structured Sparsity-Inducing Norms largest groups of each orientation and therefore |F J | ≤ |Θ|. We further assume that the groups in G θ are sorted by cardinality, so that computing F J costs O(|Θ|).

Given an active set J, both the necessary and sufficient conditions require to have access to the direct parents Π P (J) of J in the DAG of nonzero patterns. In simple settings, e.g., when G is the set of rectangular groups, this operation can be performed in O(1) (it just corresponds to scan the (up to) four patterns at the edges of the current rectangular hull).

However, for more general orientations, computing Π P (J) requires to find the smallest nonzero patterns that we can generate from the groups in F J , reduced to the stripe of variables around the current hull. This stripe of variables can be computed as g∈(G J ) c \F J g c \J, so that getting

Π P (J) costs O(s2 |Θ| + p|G|) in total.
Thus, if the number of active variables is upper bounded by s ≪ p (which is true if our target is actually sparse), the time complexity of Algorithm 3 is the sum of:

• the computation of the gradient, O(snp) for the square loss.

• if the underlying solver called upon by the active set algorithm is a standard SOCP solver, O(s max J∈P,|J|≤s |G J | 3.5 + s 4.5 ) (note that the term s 4.5 could be improved upon by using warm-restart strategies for the sequence of reduced problems).

• t 1 times the computation of (N ), that is O(t 1 (s2 |Θ| +p|G|+sn 2 θ )+p|G|) = O(t 1 p|G|).
During the initialization (i.e., J = ∅), we have |Π P (∅)| = O(p) (since we can start with any singletons), and

|G K \G J | = |G K | = |G|,
which leads to a complexity of O(p|G|) for the sum G∈G K \G J = G∈G K . Note however that this sum does not depend on J and can therefore be cached if we need to make several runs with the same set of groups G.

• t 2 times the computation of (S ε ), that is O(t 2 (s2 |Θ| + p|G| + |Θ| 2 + |Θ|p + p|G|)) =
O(t 2 p|G|), with t 1 + t 2 ≤ s. We finally get complexity with a leading term in O(sp|G| + s max J∈P,|J|≤s |G J | 3.5 + s 4.5 ), which is much better than O(p 3.5 + |G| 3.5 ), without an active set method. In the example of the two-dimensional grid (see Section 2.3.5), we have |G| = O( √ p) and O(s max{p 1.75 , s 3.5 }) as total complexity. The simulations of Section 2.6 confirm that the active set strategy is indeed useful when s is much smaller than p. Moreover, the two extreme cases where s ≈ p or p ≪ 1 are also shown not to be advantageous for the active set strategy, since either it is cheaper to use the SOCP solver directly on the p variables, or we uselessly pay the additional fixed-cost of the active set machinery (such as computing the optimality conditions). Note that we have derived here the theoretical complexity of the active set algorithm when we use an interior point method as underlying solver. With the first order method presented in Appendix A.1.8, we would instead get a total complexity in O(sp 1.5 ).

Intersecting Nonzero Patterns

We have seen so far how overlapping groups can encore prior information about a desired set of (non)zero patterns. In practice, controlling these overlaps may be delicate and hinges on the choice of the weights (ω g ) G∈G (see the experiments in Section 2.6).

Pattern Consistency

In particular, the weights have to take into account that some variables belonging to several overlapping groups are penalized multiple times.

However, it is possible to keep the benefit of overlapping groups whilst limiting their side effects, by taking up the idea of support intersection (Bach, 2008c;[START_REF] Meinshausen | Stability selection[END_REF]. First introduced to stabilize the set of variables recovered by the Lasso, we reuse this technique in a different context, based on the fact that Z is closed under union.

If we deal with the same sets of groups as those considered in Section 2.3.5, it is natural to rewrite G as θ∈Θ G θ , where Θ is the set of the orientations of the groups in G (for more details, see Section 2.3.5). Let us denote by ŵ and ŵθ the solutions of Eq. ( 2.5), where the regularization term Ω is respectively defined by the groups in G and by the groups6 in G θ .

The main point is that, since P is closed under intersection, the two procedures described below actually lead to the same set of allowed nonzero patterns: a) Simply considering the nonzero pattern of ŵ. b) Taking the intersection of the nonzero patterns obtained for each ŵθ , θ in Θ. With the latter procedure, although the learning of several models ŵθ is required (a number of times equals to the number of orientations considered, e.g., 2 for the sequence, 4 for the rectangular groups and more generally |Θ| times), each of those learning tasks involves a smaller number of groups (that is, just the ones belonging to G θ ). In addition, this procedure is a variable selection technique that therefore needs a second step for estimating the loadings (restricted to the selected nonzero pattern). In the experiments, we follow Bach (2008c) and we use an ordinary least squares (OLS). The simulations of Section 2.6 will show the benefits of this variable selection approach.

Pattern Consistency

In this section, we analyze the model consistency of the solution of Eq. ( 2.2) for the square loss. Considering the set of nonzero patterns P derived in Section 2.3, we can only hope to estimate the correct hull of the generating sparsity pattern, since Theorem 1 states that other patterns occur with zero probability. We derive necessary and sufficient conditions for model consistency in a low-dimensional setting, and then consider a high-dimensional result.

We consider the square loss and a fixed-design analysis (i.e., x 1 , . . . , x n are fixed). The extension of the following consistency results to other loss functions is beyond the scope of the chapter (see for instance [START_REF] Bach | Bolasso: model consistent Lasso estimation through the bootstrap[END_REF]. We assume that for all i ∈ 1; n , y i = x ⊤ i w ⋆ + ε i where the vector ε is an i.i.d. vector with Gaussian distributions with mean zero and variance σ 2 > 0, and w ⋆ ∈ R p is the population sparse vector; we denote by J ⋆ the G-adapted hull of its nonzero pattern. Note that estimating the G-adapted hull of w ⋆ is equivalent to estimating the nonzero pattern of w ⋆ if and only if this nonzero pattern belongs to P. This happens when our prior information has led us to consider

Understanding the Properties of Structured Sparsity-Inducing Norms

an appropriate set of groups G. Conversely, if G is misspecified, recovering the hull of the nonzero pattern of w may be irrelevant, which is for instance the case if

w ⋆ = w ⋆ 1 0 ∈ R 2 and G = {{1}, {1, 2}}.
Finding the appropriate structure of G directly from the data would therefore be interesting future work.

Consistency Condition

We begin with the low-dimensional setting where n is tending to infinity with p fixed. In addition, we also assume that the design is fixed and that the Gram matrix

Q = 1 n n i=1 x i x ⊤
i is invertible with positive-definite (i.e., invertible) limit:

lim n→∞ Q = Q ⋆ ≻ 0.
In this setting, the noise is the only source of randomness. We denote by r ⋆ J ⋆ the vector defined as

∀j ∈ J ⋆ , r ⋆ j = w ⋆ j g∈G J ⋆ ,g∋j (ω g j ) 2 ω g • w ⋆ -1
for the group Lasso, i.e., the quantity

max g∈(G ⋆ J ) c Q ⋆ g Hull(J ⋆ ) [Q ⋆ Hull(J ⋆ )Hull(J ⋆ ) ] -1 w ⋆ g w ⋆ g 2 g∈G ⋆ J 2
must be less or strictly less than one.

High-Dimensional Analysis

We prove a high-dimensional variable consistency result (see proof in Appendix A.1.7) that extends the corresponding result for the Lasso [START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF], by assuming that the consistency condition in Theorem 2 is satisfied.

Theorem 3 Assume that Q has unit diagonal, κ = λ min (Q J ⋆ J ⋆ ) > 0 and (Ω c J ⋆ ) * [Q [J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ r ⋆ J ⋆ ] < 1 -τ , for some τ > 0. If τ µ √ n ≥ σC 3 (G, J ⋆ ), and µ|J ⋆ | 1/2 ≤ C 4 (G, J ⋆ ), then the probability of incorrect hull selection is upper bounded by: exp - nµ 2 τ 2 C 1 (G, J ⋆ ) 2σ 2 + 2|J| exp - nC 2 (G, J ⋆ ) 2|J ⋆ |σ 2 ,
where

C 1 (G, J ⋆ ), C 2 (G, J ⋆ ), C 3 (G, J ⋆ ) and C 4 (G, J ⋆ ) are constants defined in Appendix A.1.7,
which essentially depend on the groups, the smallest nonzero coefficient of w ⋆ and how close the support {j ∈ J ⋆ : w ⋆ j = 0} of w ⋆ is to its hull J ⋆ , that is the relevance of the prior information encoded by G.

In the Lasso case, we have

C 1 (G, J ⋆ ) = O(1), C 2 (G, J ⋆ ) = O(|J ⋆ | -2 ), C 3 (G, J ⋆ ) = O((log p) 1/2 ) and C 4 (G, J ⋆ ) = O(|J ⋆ | -1 ), leading to the usual scaling n ≈ log p and µ ≈ σ(log p/n) 1/2 .
We can also give the scaling of these constants in simple settings where groups overlap. For instance, let us consider that the variables are organized in a sequence (see Figure 2.4). Let us further assume that the weights (ω g ) g∈G satisfy the following two properties:

a) The weights take into account the overlaps, that is,

ω g j = β(|{h ∈ G ; h ∋ j, h ⊂ g and h = g}|), with t → β(t) ∈ (0, 1] a non-increasing function such that β(0) = 1, b) The term max j∈ 1;p g∋j,g∈G ω g j
is upper bounded by a constant K independent of p. Note that we consider such weights in the experiments (see Section 2.6). Based on these assumptions, some algebra directly leads to

u 1 ≤ Ω(u) ≤ 2K u 1 , for all u ∈ R p .

Understanding the Properties of Structured Sparsity-Inducing Norms

We thus obtain a scaling similar to the Lasso (with, in addition, a control of the allowed nonzero patterns). With stronger assumptions on the possible positions of J ⋆ , we may have better scalings, but these are problem-dependent (a careful analysis of the groupdependent constants would still be needed in all cases).

Experiments

In this section, we carry out several experiments to illustrate the behavior of the sparsity-inducing norm Ω. We denote by Structured-lasso, or simply Slasso, the models regularized by the norm Ω. In addition, the procedure (introduced in Section 2.4.3) that consists in intersecting the nonzero patterns obtained for different models of Slasso will be referred to as Intersected Structured-lasso, or simply ISlasso.

Throughout the experiments, we consider noisy linear models y = Xw ⋆ + ε, where w ∈ R p is the generating loading vector and ε is a standard Gaussian noise vector with its variance set to satisfy Xw 2 = 3 ε 2 . This consequently leads to a fixed signalto-noise ratio. We assume that the vector w is sparse, i.e., it has only a few nonzero components, that is, |J ⋆ | ≪ p. We further assume that these nonzero components are either organized on a sequence or on a two-dimensional grid (see Figure 2.9). Moreover, we consider sets of groups G such as those presented in Section 2.3.5. We also consider different choices for the weights (ω g ) g∈G that we denote by (W1), (W2) and (W3) (we will keep this notation throughout the following experiments):

(W1): Uniform weights, ω g j = 1, (W2): Weights depending on the size of the groups, ω g j = |g| -2 , (W3): Weights for overlapping groups, ω g j = ρ |{h∈G ; h∋j, h⊂g and h =g}| , for some ρ ∈ (0, 1). For each orientation in G, the third type of weights (W3) aims at reducing the unbalance caused by the overlapping groups. Specifically, given a group g ∈ G and a variable j ∈ g, the corresponding weight ω g j is all the more small as the variable j already belongs to other groups with the same orientation. Unless otherwise specified, we use the third type of weights (W3) with ρ = 0.5. In the following experiments, the loadings w ⋆ J , as well as the design matrices, are generated from a standard Gaussian distribution with identity covariance matrix. The positions of J ⋆ are also random and are uniformly drawn.

Consistent hull estimation

We first illustrate Theorem 2 that establishes necessary and sufficient conditions for consistent hull estimation. To this end, we compute the probability of correct hull estimation when we consider diamond-shaped generating patterns of |J ⋆ | = 24 variables on a 20×20-dimensional grid (see Figure 2.9h). Specifically, we generate 500 covariance matrices Q ⋆ distributed according to a Wishart distribution with δ degrees of freedom, generating patterns that we use for the 20×20-dimensional grid experiments; again, those patterns all form the same hull of 24 variables, i.e., the diamond-shaped convex in (h). The positions of these generating patterns are randomly selected during the experiments. For the grid setting, the hull is defined based on the set of groups that are half-planes, with orientations that are multiple of π/4 (see Section 2.3.5).

Understanding the

Structured variable selection

We show in this experiment that the prior information we put through the norm Ω improves upon the ability of the model to recover spatially structured nonzero patterns. We are looking at two situations where we can express such a prior through Ω, namely (1) the selection of a contiguous pattern on a sequence (see Figure 2.9g) and ( 2) the selection of a convex pattern on a grid (see Figure 2.9h).

In what follows, we consider p = 400 variables with generating patterns w ⋆ whose hulls are composed of |J ⋆ | = 24 variables. For different sample sizes n ranges in the set {100, 200, 300, 400, 500, 700, 1000}, we consider the probabilites of correct recovery and 7. We have empirically observed that this choice of degrees of freedom enables to cover well the consistency transition regime around zero in Figure 2.10. the (normalized) Hamming distance to the true nonzero patterns. For the realization of a random setting {J ⋆ , w ⋆ , X, ε}, we compute an entire regularization path over which we collect the closest Hamming distance to the true nonzero pattern and whether it has been exactly recovered for some µ. After repeating 50 times this computation for each sample size n, we report the results in Figure 2.11.

First and foremost, the simulations highlight how important the weights (ω g ) g∈G are. In particular, the uniform (W1) and size-dependent weights (W2) perform poorly since they do not take into account the overlapping groups. The models learned with such weights do not manage to recover the correct nonzero patterns (in that case, the best model found on the path corresponds to the empty solution, with a normalized Hamming distance of |J ⋆ |/p = 0.06-see Figure 2.11c).

Although groups that moderately overlap do help (e.g., see Slasso with the weights 2. Understanding the Properties of Structured Sparsity-Inducing Norms (W3) on Figure 2.11c), it remains delicate to handle groups with many overlaps, even with an appropriate choice of (ω g ) g∈G (e.g., see Slasso on Figure 2.11d). The ISlasso procedure does not suffer from this issue since it reduces the number of overlaps whilst keeping the desirable effects of overlapping groups. Another way to yield a better level of sparsity, even with many overlaps, would be to consider non-convex alternatives to Ω (see, e.g., Jenatton et al., 2010b). Moreover, adding the ±π/4-groups to the rectangular groups enables to recover a nonzero pattern closer to the generating pattern. This is illustrated on Figure 2.11d where the error of ISlasso with only rectangular groups (in black) corresponds to the selection of the smallest rectangular box around the generating pattern.

Prediction error and relevance of the structured prior

In the next simulation, we start from the same setting as Section 2.6.2 where we additionally evaluate the relevance of the contiguous (or convex) prior by varying the number of zero variables that are contained in the hull (see Figure 2.9). We then compute the prediction error for different sample sizes n ∈ {250, 500, 1000}. The prediction error is understood here as X test (w ⋆ŵ) 2 2 / X test w ⋆ 2 2 , where ŵ denotes the OLS estimate, performed on the nonzero pattern found by the model considered (i.e., either Lasso, Slasso or ISlasso). The regularization parameter is chosen by 5-fold cross-validation and the test set consists of 500 samples. For each value of n, we display on Figure 2.12 the median errors over 50 random settings {J ⋆ , w ⋆ , X, ε}, for respectively the sequence and the grid. Note that we have dropped for clarity the models that performed already poorly in Section 2.6.2.

The experiments show that if the prior about the generating pattern is relevant, then our structured approach performs better that the standard Lasso. Indeed, as soon as the hull of the generating pattern does not contain too many zero variables, Slasso/ISlasso outperform Lasso. In fact, the sample complexity of the Lasso depends on the number of nonzero variables in w ⋆ [START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF] as opposed to the size of the hull for Slasso/ISlasso. This also explains why the error for Slasso/ISlasso is almost constant with respect to the number of nonzero variables (since the hull has a constant size).

Active set algorithm

We finally focus on the active set algorithm (see Section 2.4) and compare its time complexity to the SOCP solver when we are looking for a sparse structured target. More precisely, for a fixed level of sparsity |J ⋆ | = 24 and a fixed number of observations n = 3500, we analyze the complexity with respect to the number of variables p that varies in {100, 225, 400, 900, 1600, 2500}. We consider the same experimental protocol as above except that we display the median CPU time based only8 on 5 random settings {J ⋆ , w, X, ε}. We assume that we have a rough idea of the level of sparsity of the .12: For the sample size n ∈ {250, 500, 1000}, we plot the prediction error versus the proportion of nonzero variables in the hull of the generating pattern. In the grid case, two sets of groups G are considered, the rectangular groups with or without the ±π/4-groups (denoted by (π/4) in the legend). The points, the lower and upper error bars on the curves respectively represent the median, the first and third quartile, based on 50 random settings {J ⋆ , w ⋆ , X, ε}.

true vector and we set the stopping criterion s = 4|J ⋆ | (see Algorithm 3), which is a rather conservative choice. We show on Figure 2.13 that we considerably lower the 2. Understanding the Properties of Structured Sparsity-Inducing Norms Active set Active set (π/4) SOCP SOCP (π/4) Figure 2.13: Computational benefit of the active set algorithm: CPU time (in seconds) versus the number of variables p, displayed in log-log scale. The points, the lower and upper error bars on the curves respectively represent the median, the first and third quartile. Two sets of groups G are considered, the rectangular groups with or without the ±π/4-groups (denoted by (π/4) in the legend). Due to the computational burden, we could not obtain the SOCP's results for p = 2500. computational cost for the same level of performance9 . As predicted by the complexity analysis of the active set algorithm (see the end of Section 2.4), considering the set of rectangular groups with or without the ±π/4-groups results in the same complexity (up to constant terms). We empirically obtain an average complexity of ≈ O(p 2.13 ) for the SOCP solver and of ≈ O(p 0.45 ) for the active set algorithm.

Not surprisingly, for small values of p, the SOCP solver is faster than the active set algorithm, since the latter has to check its optimality by computing necessary and sufficient conditions (see Algorithm 3 and the discussion in the algorithmic complexity paragraph of Section 2.4).

Conclusion

We have shown how to incorporate prior knowledge on the form of nonzero patterns for linear supervised learning. Our solution relies on a regularizing term which linearly combines ℓ 2 -norms of possibly overlapping groups of variables. Our framework brings into play intersection-closed families of nonzero patterns, such as all rectangles on a two-dimensional grid. We have studied the design of these groups, efficient algorithms and theoretical guarantees of the structured sparsity-inducing method. Our experiments have shown to which extent our model leads to better prediction, depending on the relevance of the prior information.

A natural extension to this work is to consider bootstrapping since this may improve theoretical guarantees and result in better variable selection (Bach, 2008c;[START_REF] Meinshausen | Stability selection[END_REF]. In order to deal with broader families of (non)zero patterns, it would be interesting to combine our approach with recent work on structured sparsity: for instance, [START_REF] Baraniuk | Model-based compressive sensing[END_REF]; [START_REF] Jacob | Group Lasso with overlaps and graph Lasso[END_REF] consider union-closed collections of nonzero patterns, [START_REF] He | Exploiting structure in wavelet-based Bayesian compressive sensing[END_REF] exploit structure through a Bayesian prior while [START_REF] Huang | Learning with structured sparsity[END_REF] handle non-convex penalties based on information-theoretic criteria.

More generally, our regularization scheme could also be used for various learning tasks, as soon as prior knowledge on the structure of the sparse representation is available, e.g., for multiple kernel learning [START_REF] Micchelli | Learning the kernel function via regularization[END_REF], multi-task learning [START_REF] Argyriou | Convex multi-task feature learning[END_REF][START_REF] Obozinski | Joint covariate selection and joint subspace selection for multiple classification problems[END_REF][START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF] and sparse matrix factorization problems (Mairal et al., 2010a;Jenatton et al., 2010bJenatton et al., , 2011c)).

Finally, although we have mostly explored in this chapter the algorithmic and theoretical issues related to these norms, this type of prior knowledge is of clear interest for the spatially and temporally structured data typical in bioinformatics [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF], computer vision (Jenatton et al., 2010b;Mairal et al., 2010b) and neuroscience applications (see, e.g., Varoquaux et al., 2010c).

Structured Sparse Principal Component Analysis

Chapter abstract: We present an extension of sparse principal component analysis (PCA), or sparse dictionary learning, where the sparsity patterns of all dictionary elements are structured and constrained to belong to a prespecified set of shapes. This structured sparse PCA is based on a structured regularization recently introduced by Jenatton et al. (2011a). While classical sparse priors only deal with cardinality, the regularization we use encodes higher-order information about the data. We propose an efficient and simple optimization procedure to solve this problem. Experiments with two practical tasks, the denoising of sparse structured signals and face recognition, demonstrate the benefits of the proposed structured approach over unstructured approaches.

The material of this chapter is based on the following publication: R. Jenatton, G. Obozinski, F. Bach. Structured sparse principal component analysis. In International Conference on Artificial Intelligence and Statistics (AISTATS). 2010

Introduction

Principal component analysis (PCA) is an essential tool for data analysis and unsupervised dimensionality reduction. Its goal is to find, among linear combinations of the data variables, a sequence of orthogonal factors that most efficiently explain the variance of the observations.

One of PCA's main shortcomings is that, even if it finds a small number of important factors, the factor themselves typically involve all original variables. In the last decade, several alternatives to PCA which find sparse and potentially interpretable factors have been proposed, notably non-negative matrix factorization (NMF) [START_REF] Lee | Learning the parts of objects by non-negative matrix factorization[END_REF] and sparse PCA (SPCA) [START_REF] Jolliffe | A modified principal component technique based on the Lasso[END_REF][START_REF] Zou | Sparse principal component analysis[END_REF][START_REF] Zass | Nonnegative sparse PCA[END_REF][START_REF] Witten | A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis[END_REF].

However, in many applications, only constraining the size of the factors does not seem appropriate because the considered factors are not only expected to be sparse but also to have a certain structure. In fact, the popularity of NMF for face image analysis owes essentially to the fact that the method happens to retrieve sets of variables that are partly localized on the face and capture some features or parts of the face which

Structured Sparse Principal Component Analysis

seem intuitively meaningful given our a priori. We might therefore gain in the quality of the factors induced by enforcing directly this a priori in the matrix factorization constraints. More generally, it would be desirable to encode higher-order information about the supports that reflects the structure of the data. For example, in computer vision, features associated to the pixels of an image are naturally organized on a grid and the supports of factors explaining the variability of images could be expected to be localized, connected or have some other regularity with respect to that grid. Similarly, in genomics, factors explaining the gene expression patterns observed on a microarray could be expected to involve groups of genes corresponding to biological pathways or set of genes that are neighbors in a protein-protein interaction network.

Recent research on structured sparsity has highlighted the benefit of exploiting such structure in the context of regression and classification (Jenatton et al., 2011a;[START_REF] Jacob | Group Lasso with overlaps and graph Lasso[END_REF][START_REF] Huang | Learning with structured sparsity[END_REF], compressed sensing [START_REF] Baraniuk | Model-based compressive sensing[END_REF], as well as within Bayesian frameworks [START_REF] He | Exploiting structure in wavelet-based Bayesian compressive sensing[END_REF]. In particular, Jenatton et al. (2011a) show that, given any intersection-closed family of patterns P of variables, such as all the rectangles on a 2-dimensional grid of variables, it is possible to build an ad hoc regularization norm Ω that enforces that the support of the solution of a least-squares regression regularized by Ω belongs to the family P.

Capitalizing on these results, we aim in this chapter to go beyond sparse PCA and propose structured sparse PCA (SSPCA), which explains the variance of the data by factors that are not only sparse but also respect some a priori structural constraints deemed relevant to model the data at hand. We show how slight variants of the regularization term from Jenatton et al. (2011a) can be used successfully to yield a structured and sparse formulation of principal component analysis for which we propose a simple and efficient optimization scheme.

The rest of the chapter is organized as follows: Section 3.2 casts the SSPCA problem in the dictionary learning framework, summarizes the regularization considered by Jenatton et al. (2011a) and its essential properties, and presents some simple variants which are more effective in the context of PCA. Section 3.3 is dedicated to our optimization scheme for solving SSPCA. Our experiments in Section 3.4 illustrate the benefits of our approach through the denoising of sparse structured synthetic signals and an application to face recognition.

Notations.

For any matrix Y ∈ R n×p , we write y j ∈ R n for the j-th column of Y, while we write Y i ∈ R p for its i-th row. We refer to the set {j ∈ 1; p ; w j = 0} as the support, or nonzero pattern of the vector w ∈ R p . For any finite set A with cardinality |A|, we also define the |A|-tuple (y a ) a∈A ∈ R p×|A| as the collection of p-dimensional vectors y a indexed by the elements of A. Furthermore, for two vectors x and y in R p , we denote by x • y = (x 1 y 1 , . . . , x p y p ) ⊤ ∈ R p the elementwise product of x and y. Finally, we extend b → a b by continuity in zero with a 0 = ∞ if a = 0 and 0 otherwise.

Problem Statement

It is useful to distinguish two conceptually different interpretations of PCA. In terms of analysis, PCA sequentially projects the data on subspaces that explain the largest fraction of the variance of the data. In terms of synthesis, PCA finds a basis, or orthogonal dictionary, such that all signals observed admit decompositions with low reconstruction error. These two interpretations recover the same basis of principal components for PCA but lead to different formulations for sparse PCA. The analysis interpretation leads to sequential formulations (d 'Aspremont et al., 2008;[START_REF] Moghaddam | Spectral bounds for sparse PCA: Exact and greedy algorithms[END_REF][START_REF] Jolliffe | A modified principal component technique based on the Lasso[END_REF] that consider components one at a time and perform a deflation of the covariance matrix at each step (see [START_REF] Lounici | Taking advantage of sparsity in multi-task learning[END_REF]. The synthesis interpretation leads to non-convex global formulations [START_REF] Zou | Sparse principal component analysis[END_REF]Mairal et al., 2010a;[START_REF] Moghaddam | Spectral bounds for sparse PCA: Exact and greedy algorithms[END_REF][START_REF] Lee | Efficient sparse coding algorithms[END_REF] which estimate simultaneously all principal components, often drop the orthogonality constraints, and are referred to as matrix factorization problems [START_REF] Singh | A Unified View of Matrix Factorization Models[END_REF] in machine learning, and dictionary learning in signal processing.

The approach we propose fits more naturally in the framework of dictionnary learning, whose terminology we now introduce.

Matrix Factorization and Dictionary Learning

Given a matrix X ∈ R m×n of n columns corresponding to n observations in R m , the dictionary learning problem is to find a matrix D ∈ R m×p , called the dictionary, such that each observation can be well approximated by a linear combination of the p columns (d k ) k∈ 1;p of D called the dictionary elements. If A ∈ R p×n is the matrix of the linear combination coefficients or decomposition coefficients, the matrix product DA is called a decomposition of X.

Learning simultaneously the dictionary D and the decomposition A corresponds to a matrix factorization problem (see Witten et al., 2009, and reference therein). As formulated by [START_REF] Bach | Convex sparse matrix factorizations[END_REF] or [START_REF] Witten | A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis[END_REF], it is natural, when learning a decomposition, to penalize or constrain some norms or quasi-norms of A and D, say Ω A and Ω D respectively, to encode prior information -typically sparsity -about the decomposition of X. This can be written generally as

min A∈Rp×n, D∈R m×p 1 2nm X-DA 2 F + λ p k=1 Ω D (d k ), such that Ω A (A k ) ≤ 1, ∀ k ∈ 1; p , (3.1)
where the regularization parameter λ ≥ 0 controls to which extent the dictionary is regularized 1 . If we assume that both regularizations Ω A and Ω D are convex, problem (3.1) is convex with respect to A for fixed D and vice versa. It is however not jointly convex in the pair (A, D).

The formulation of sparse PCA considered by [START_REF] Lee | Efficient sparse coding algorithms[END_REF] corresponds to a particular instance of this problem, where the dictionary elements are required to be sparse 1. From [START_REF] Bach | Convex sparse matrix factorizations[END_REF], we know that our formulation is also equivalent to two other ones, penalized respectively by λ

2 p k=1 [ΩD(d k )] 2 +[ΩA(A k )] 2 and λ p k=1 ΩD(d k )ΩA(A k ).

Structured Sparse Principal Component Analysis

(without the orthogonality constraint D ⊤ D = I). This can be achieved by penalizing the columns of D by a sparsity-inducing norm, such as the ℓ 1 -norm:

Ω D (d k ) = d k 1 .
In the next section we consider a regularization Ω D which controls not only the sparsity but also the structure of the supports of dictionary elements.

Structured Sparsity-Inducing Norms

The work of Jenatton et al. (2011a) considered a norm which induces structured sparsity in the following sense: the solutions to a learning problem regularized by this norm have a sparse support which moreover belongs to a certain set of groups of variables. Interesting sets of possible supports include sets of variables forming rectangles when arranged on a grid and more generally convex subsets2 .

The framework of Jenatton et al. (2011a) can be summarized as follows: if we denote by G a subset of the power set of 1; p , such that g∈G g = 1; p , we define the mixed ℓ 1 /ℓ 2 norm Ω on a vector y ∈ R p as

Ω(y) = g∈G j∈g (ω g j ) 2 |y j | 2 1 2 = g∈G ω g • y 2 ,
where (ω g ) g∈G ∈ Rp × |G| is a |G|-tuple of p-dimensional vectors such that ω g j > 0 if j ∈ g and ω g j = 0 otherwise. This norm Ω linearly combines the ℓ 2 norms of possibly overlapping groups of variables, with variables in each group being weighted by (ω g ) g∈G . Note that a same variable y j belonging to two different groups g 1 , g 2 ∈ G is allowed to be weighted differently in g 1 and g 2 (by respectively ω g 1 j and ω g 2 j ). For specific choices of G, Ω leads to standard sparsity-inducing norms. For example, when G is the set of all singletons, Ω is the usual ℓ 1 norm (assuming that all the weights are equal to one).

We focus on the case of a 2-dimensional grid where the set of groups G is the set of all horizontal and vertical half-spaces (see Figure 3.1 taken from Jenatton et al., 2011a). As proved by Jenatton et al. (2011a, Theorem 3.1), the ℓ 1 /ℓ 2 norm Ω sets to zero some groups of variables ω g • y 2 , i.e., some entire horizontal and vertical half-spaces of the grid, and therefore induces rectangular nonzero patterns. Note that a larger set of convex patterns can be obtained by adding in G half-planes with other orientations. In practice, we use planes with angles that are multiples of π 4 , which enables the nonzero patterns to have polygonal shapes with up to 8 faces.

Among sparsity inducing regularizations, the ℓ 1 norm is often privileged since it is convex. However, so-called concave penalizations, such as penalization by an ℓ α quasinorm, which are closer to the ℓ 0 quasi-norm and penalize more aggressively small coefficients can be preferred, especially in a context where the unregularized problem, here dictionary learning is itself non convex. In light of recent work showing the advantages of addressing sparse problems through concave penalization (e.g., see [START_REF] Zou | One-step sparse estimates in nonconcave penalized likelihood models[END_REF], Figure 3.1: (Left) The set of blue and green groups with their (not displayed) complements to penalize to select rectangles. (Right) In red, an example of recovered pattern in this setting.

we therefore generalize Ω to a family of non-convex regularizers as follows: for α ∈ (0, 1), we define the quasi-norm Ω α for all vectors y ∈ R p as

Ω α (y) = g∈G ω g • y α 2 1 α = ( ω g • y 2 ) g∈G α ,
where we denote by ( ω g • y 2 ) g∈G ∈ R 1×|G| the |G|-tuple composed of the different blocks ω g • y 2 . We thus replace the (convex) ℓ 1 /ℓ 2 norm Ω by the (neither convex, nor concave) ℓ α /ℓ 2 quasi-norm Ω α . While leading to the same set of (non)zero patterns, the ℓ α quasi-norm yields sparsity at the group level more aggressively.

Optimization

We consider the optimization of Eq. (3.1) where we use Ω D = Ω α to regularize the dictionary D. We discuss in Section 3.3.3 which norms Ω A we can handle in this optimization framework.

Formulation as a Sequence of Convex Problems

We now consider Eq. (3.1) where we take

Ω D to be Ω α , α ∈ (0, 1), that is, min A∈Rp×n, D∈R m×p 1 2nm X-DA 2 F + λ p k=1 Ω α (d k ), such that Ω A (A k ) ≤ 1, ∀ k ∈ 1; p , (3.2)
Although the minimization problem in Eq. (3.2) is still convex in A for D fixed, the converse is not true anymore because of Ω α . Indeed, the formulation in D is non-

Structured Sparse Principal Component Analysis

differentiable and non-convex. To address this problem, we use the variational equality based on the following lemma that is related3 to ideas from [START_REF] Micchelli | Learning the kernel function via regularization[END_REF]:

Lemma 1 (Variational formulation) Let α ∈ (0, 2) and β = α 2-α . For any vector y ∈ R p , we have the following equality

y α = min z∈R p + 1 2 p j=1 y 2 j z j + 1 2 z β ,
and the minimum is uniquely attained for

z j = |y j | 2-α y α-1 α , ∀j ∈ 1; p .
Proof. Let ψ : z → p j=1 y 2 j z -1 j + z β be the continuously differentiable function defined on R p + . We have lim z β →∞ ψ(z) = +∞ and lim z j →0 ψ(z) = +∞ if y j = 0 (for y j = 0, note that min z∈R p + ψ(z) = min z∈R p + ,z j =0 ψ(z)). Thus, the infimum exists and it is attained. Taking the derivative with respect to z j (for z j > 0) leads to the expression of the unique minimum, expression that is still correct for z j = 0.

To reformulate problem (3.2), let us consider the |G|-tuple (η g ) g∈G ∈ R p×|G| of r- dimensional vectors η g that satisfy for all k ∈ 1; p and g ∈ G, η g k ≥ 0. It follows from Lemma 1 that 2 p k=1 Ω α (d k ) is equal to min (η g ) G∈G ∈R p×|G| + p k=1 (η g k ) g∈G β + g∈G d k • ω g 2 2 (η g k ) -1 ,
that can be rewritten in turn as

min (η g ) g∈G ∈R p×|G| + p k=1 (d k ) ⊤ Diag ζ k -1 d k + (η g k ) g∈G β ,
where we have introduced the matrix ζ ∈ R m×p defined by4 ζ jk g∈G, g∋j

(ω g j ) 2 (η g k ) -1 -1 .
This leads to the following formulation

min A, D, Ω A (A k )≤1 (η g ) g∈G ∈Rp×|G| + 1 2nm X-DA 2 F + λ 2 p k=1 (d k ) ⊤ Diag ζ k -1 d k + (η g k ) g∈G β , (3.3)
which is equivalent to Eq. (3.2) and which is now quadratic with respect to D.

Sharing Structure among Dictionary Elements

So far, the regularization quasi-norm Ω α has been used to induce a structure inside each dictionary element taken separately. Nonetheless, some applications may also benefit from a control of the structure across dictionary elements. For instance it can be desirable to impose the constraint that several dictionary elements share the exact same nonzero patterns. In the context of face recognition, this could be relevant to model the variability of faces as the combined variability of several parts, with each part having a small support (such as eyes), and having its variance itself explained by several dictionary elements (corresponding for example to the color and the shape of the eyes).

To this end, we consider M, a partition of 1; p . Imposing that two dictionary elements d k and d k ′ share the same sparsity pattern is equivalent to imposing that d k i and d k ′ i are simultaneously zero or non-zero. Following the approach used for joint feature selection [START_REF] Obozinski | Joint covariate selection and joint subspace selection for multiple classification problems[END_REF] where the ℓ 1 norm is composed with an ℓ 2 norm, we compose the norm Ω α with the

ℓ 2 norm d M i = (d k i ) k∈M 2
, of all i th entries of each dictionary element of a class M of the partition M, leading to the regularization:

M ∈M Ω α (d M i ) = M ∈M g∈G (ω g i d k i ) i∈g, k∈M α 2 1 α . (3.4)
In fact, not surprisingly given that similar results hold for the group Lasso (Bach, 2008b), it can be shown that the above extension is equivalent to the variational formulation

min A, D, Ω A (A k )≤1 (η g ) g∈G ∈R |M|×|G| + 1 2nm X-DA 2 F + λ 2 M ∈M k∈M (d k ) ⊤ Diag ζ M -1 d k + (η g M ) g∈G β ,
with class specific variables {η M } M ∈M , {ζ M } M ∈M defined in a similar way to {η k } k∈ 1;p and {ζ k } k∈ 1;p .

Algorithm

The main optimization procedure described in Algorithm 5 is based on a cyclic optimization over the three variables involved, namely (η g ) g∈G , A and D. We use Lemma 1 to solve (3.2) through a sequence of problems that are convex in A for fixed D (and conversely, convex in D for fixed A). For this sequence of problems, we then present efficient optimization procedures based on block coordinate descent (BCD) (Bertsekas, 1999, Section 2.7). We describe these in detail in Algorithm 5. Note that we depart from the approach of Jenatton et al. (2011a) who use an active set algorithm. Their approach does not indeed allow warm restarts, which is crucial in our alternating optimization scheme.

Update of (η g ) g∈G . The update of (η g ) g∈G is straightforward (even if the underlying minimization problem is non-convex), since the minimizer (η g ) * in Lemma 1 is given in closed-form. In practice, following [START_REF] Micchelli | Learning the kernel function via regularization[END_REF], we avoid numerical instability near zero with the smoothed update η g k ← max{(η g k ) * , ε}, with ε ≪ 1.

Update of A.

The update of A follows the technique suggested by Mairal et al. (2010a). Each row A k of A is constrained separately through Ω A (A k ). Furthermore, if we assume that D and {A j } j =k are fixed, some basic algebra leads to

arg min Ω A (A k )≤1 1 2nm X-DA 2 F = arg min Ω A (A k )≤1 A k -d k -2 2 [d k ] ⊤ X- j =k d j A j 2 2 = arg min Ω A (A k )≤1 A k -w 2 2 , (3.5)
which is simply the Euclidean projection Π Ω A (w) of the row vector w onto the unit ball of Ω A . Consequently, the cost of the BCD update of A depends on how fast we can perform this projection; the ℓ 1 and ℓ 2 norms are typical cases where the projection can be computed efficiently [START_REF] Brucker | An O(n) algorithm for quadratic knapsack problems[END_REF][START_REF] Duchi | Efficient projections onto the ℓ 1 -ball for learning in high dimensions[END_REF]. In the experiments, we take Ω A to be the ℓ 2 norm. In addition, since the function A k → 1 2nm X-DA 2 F is continuously differentiable on the (closed convex) unit ball of Ω A , the convergence of the BCD procedure is guaranteed since the minimum in Eq. (3.5) is unique (Bertsekas, 1999, Proposition 2.7.1). The complete update of A is given in Algorithm 5.

Update of D.

A fairly natural way to update D would be to compute the closed form solutions available for each row of D. Indeed, both the loss 1 2nm X-DA 2 F and the penalization on D are separable in the rows of D, leading to m independent ridgeregression problems, implying in turn m matrix inversions.

However, in light of the update of A, we consider again a BCD scheme on the columns of D that turns out to be much more efficient, without requiring any non-diagonal matrix inversion. The detailed procedure is given in Algorithm 5. The convergence follows along the same arguments as those used for A.

Algorithm 5 Main procedure for solving Eq. (3.3).

Input: Dictionary size p, data matrix X. Initialization: Initialization of A, D (possibly random). while ( stopping criterion not reached ) Update (η g ) G∈G : closed-form solution.

Update A by BCD:

for t = 1 to T u , for k = 1 to p: A k ← Π Ω A A k + d k -2 2 [d k ] ⊤ (X-DA) . Update D by BCD: for t = 1 to T v , for k = 1 to p: d k ← Diag ζ k Diag A k 2 2 ζ k +nmλ1 -1 (X ⊤ A ⊤ k -DAA ⊤ k + A k 2 2 d k ). Output: Decomposition A, D.
Our problem is not jointly convex in (η g ) g∈G , A and D, which raises the question of the sensitivity of the optimization to its initialization. This point will be discussed in 3.4. Experiments Section 3.4. In practice, the stopping criterion relies on the relative decrease (typically 10 -3 ) in the cost function in Eq. (3.2).

Algorithmic complexity. The complexity of Algorithm 5 can be decomposed into three terms, corresponding to the update procedures of (η g ) g∈G , A and D. We denote by T u (respectively T v ) the number of updates of A (respectively D) in Algorithm 5. First, computing (η g ) g∈G and

ζ costs O(p|G|+(|G| + p) g∈G |g|) = O(mp|G| + m|G| 2 ). The update of U requires O((m + T u n)p 2 + (nm + C Π T u )p) operations
, where C Π is the cost of projecting onto the unit ball of Ω A . Similarly, we get for the update of D a complexity of O((n + T v m)p 2 + nmp). In practice, we notice that the BCD updates for both A and D require only few steps, so that we choose T u = T v = 5. In our experiments, the algorithmic complexity simplifies to O(m 2 +p 2 max{n, m}+pm max{m 1/2 , n}) times the number of iterations in Algorithm 5. Note that the complexity is linear in n and is quadratic in p, which is empirically the computational bottleneck.

Extension to NMF. Our formalism does not cover the positivity constraints of nonnegative matrix factorization, but it is straightforward to extend it at the cost of an additional cheap threshold operation (to project onto the positive orthant) in the BCD updates of A and D.

Experiments

We first consider the denoising of synthetic signals to illustrate the effect of our regularization. We then focus on the application of SSPCA to a face recognition problem and we show that, by adding a sparse structured prior instead of a simple sparse prior, we gain in robustness to occlusions. In preliminary experiments, we considered the exact regularization from Jenatton et al. (2011a), i.e., with α = 1, but found that the obtained patterns were not sufficiently sparse and salient. We therefore turned to the setting where the parameter α is in (0, 1). We chose α = 0.5, since much smaller or larger values yield either not sparse enough solutions or numerical instability.

By definition, dictionary learning belongs to unsupervised learning; in that sense, our method may appear first as a tool for exploratory data analysis, which leads us naturally to qualitatively analyze the results of our decompositions (e.g., by visualizing the learned dictionaries). This is obviously a difficult and subjective exercise, beyond the assessment of the consistency of the method in artificial examples where the "true" dictionnary is known. For that reason, we endeavor in the experiments to compare our method objectively and quantitatively with other techniques. Specifically, we apply our method within either a denoising or a classification setting, and assess its performance respectively by the obtained increase in explained variance or classification accuracy.

A Matlab toolbox implementing our method can be downloaded from http://www.di.ens.fr/ jenatton/.

Denoising of Synthetic Signals

In this first experiment, we consider signals generated by the following noisy linear model

α 1 d 1 + α 2 d 2 + α 3 d 3 + ε ∈ R 400 , (3.6) where D = [d 1 , d 2 , d 3 ] ∈ R 400×3
are sparse and structured dictionary elements organized on a 20 × 20-dimensional grid (D is represented on the top row of Figure 3.2). The components of the noise vector ε are independent and identically distributed according to a centered Gaussian distribution with its variance set to obtain a signal-to-noise ratio (SNR) of 0.5. The coefficients [α 1 , α 2 , α 3 ] that linearly combine the dictionary elements of D are generated according to a centered Gaussian distribution, with the following covariance matrix

   1 0 0.5 0 1 0.5 0.5 0.5 1    .
From 250 of such signals, we learn a decomposition D Â with p = 3 dictionary elements, which seems a reasonable choice of p in an attempt to recover the underlying (in this case, known) structure of D. For SPCA and SSPCA, the regularization parameter λ is selected by 5-fold cross-validation on the reconstruction error. Based on the learned dictionary D, we denoise 1000 new signals generated in the same way. We report in Table 3.1 the results of the denoising, for PCA, SPCA and SSPCA.

The difficulty of this task is essentially twofold and lies in (1) the high level of noise and in ( 2) the small number of signals (i.e., 250 signals against 400 variables) available to learn the decomposition.

As displayed on Figure 3.2, PCA and SPCA learn very scattered and uninterpretable dictionary elements. On the other hand, the sparse structured prior we put through Ω α helps to recover the initial structure of D, which, in turn, improves upon the denoising performance of SSPCA (see Table 3.1). Note that in order to assess the statistical significance of the differences between the average denoising performances of Table 3.1, one has to consider the sample standard deviation divided by √ 1000 [START_REF] Lehmann | Testing statistical hypotheses[END_REF], i.e., roughly ≈ 0.007.

The setting we consider here raises the interesting question of model identifiability, i.e., whether we can recover the true dictionary elements that generated the signals, which we defer to future work.

PCA SPCA SSPCA

Estimation error: 0.41 ± 0.22 0.40 ± 0.22 0.34 ± 0.21

Face Recognition

We apply SSPCA on the cropped AR Face Database [START_REF] Martinez | PCA versus LDA[END_REF]) that consists of 2600 face images, corresponding to 100 individuals (50 women and 50 men). For each subject, there are 14 non-occluded poses and 12 occluded ones (the occlusions are due to sunglasses and scarfs). We reduce the resolution of the images from 165×120 pixels to 38×27 pixels for computational reasons.

Figure 3.3 shows examples of learned dictionaries (for p = 36 elements), for NMF, SSPCA and SSPCA with shared structure (see Section 3.3.2). While NMF finds sparse but spatially unconstrained patterns, SSPCA select sparse convex areas that correspond to a more natural segment of faces. For instance, meaningful parts such as the mouth and the eyes are recovered by the dictionary.

We now quantitatively compare SSPCA, SPCA, PCA and NMF on a face recognition problem. We first split the data into 2 parts, the occluded faces and non-occluded ones. For different sizes of the dictionary, we apply each of the aforementioned dimensionality reduction techniques to the non-occluded faces. Keeping the learned dictionary D, we decompose both non-occluded and occluded faces on D. We then classify the occluded faces with a k-nearest-neighbors classifier (k-NN), based on the obtained low-dimensional representations A. Given the size of the dictionary, we choose the number of neighbor(s) and the amount of regularization λ by cross-validation5 on the non-occluded faces.

The formulations of NMF, SPCA and SSPCA are non-convex and as a consequence, the local minima reached by those methods might a priori be sensitive to the initialization. To evaluate this sensitivity, we repeat the protocol described above 10 times and display in Figure 3.4 the median, first and third quartile of the classification scores obtained in this way. In practice we found the performance on the test set to be pretty stable as a function of the initialization. We denote by shared-SSPCA (resp. shared-SPCA) the models where we impose, on top of the structure of Ω α , to have only 10 different nonzero patterns among the learned dictionaries (see Section 3.3.2). We performed a Wilcoxon signed-rank test [START_REF] Lehmann | Testing statistical hypotheses[END_REF] between the classification scores of NMF and SSPCA, and for dictionary sizes greater than 100 (up to 150), our approach performs better than NMF at the 5% significance level. For smaller dictionaries, NMF and SSPCA perform similarly. The other methods, including PCA and SPCA, obtained overall lower scores than NMF and can also be shown to perform significantly worse than SSPCA.

As a baseline, we also plot the classification score that we obtain when we directly apply k-NN on the raw data, without preprocessing. Because of its local dictionary, SSPCA proves to be more robust to occlusions and therefore outperforms the other methods on this classification task. On the other hand, SPCA, that yields sparsity without a structured prior, performs poorly. Sharing structure across the dictionary elements (see Section 3.3.2) seems to help SPCA for which no structure information is otherwise available. The goal of our chapter is not to compete with state-of-the-art techniques of face recognition, but to demonstrate the improvement obtained between the ℓ 1 norm and more structured norms. We could still improve upon our results using non-linear classification (e.g., with a SVM) or by refining our features (e.g., with a Laplacian filter). 3). The dictionary elements are sorted in decreasing order of explained variance. While NMF gives sparse spatially unconstrained patterns, SSPCA finds convex areas that correspond to more natural face segments. SSPCA captures the left/right illuminations and retrieves pairs of symmetric patterns. Some displayed patterns do not seem to be convex, e.g., nonzero patterns located at two opposite corners of the grid. However, a closer look at these dictionary elements shows that convex shapes are indeed selected, and that small numerical values (just as regularizing by ℓ 2 norm may lead to) give the visual impression of having zeroes in convex nonzero patterns. This also shows that if a nonconvex pattern has to be selected, it will be, by considering its convex hull.

Structured Sparse Principal Component Analysis

Conclusions

We proposed to apply a non-convex variant of the regularization introduced by Jenatton et al. (2011a) to the problem of structured sparse dictionary learning. We present an efficient block-coordinate descent algorithm with closed-form updates. In a denoising task of sparse structured signals, our approach led to better performance and to a more interpretable decomposition of the data. For face recognition, the dictionaries learned have increased robustness to occlusions compared to NMF.

In future work, we would like to investigate Bayesian frameworks that would define similar structured priors and allow the principled choice of the regularization parameter and the number of dictionary elements [START_REF] Zhou | Non-parametric Bayesian dictionary learning for sparse image representations[END_REF]. Moreover, although we focus in this work on controlling the structure of the dictionary D, we could instead impose structure on the decompostion coefficients A and study the induced effect on the dictionary D [START_REF] Kavukcuoglu | Learning invariant features through topographic filter maps[END_REF]. This could be straightforward to do with the same formulation, by transposing the data matrix X. Finally, we intend to apply this structured sparsity-inducing regularization for multi-task learning, in order to take advantage of the structure between tasks.

Proximal Methods for Structured Sparsity-Inducing Norms

Abstract of the chapter: Sparse coding consists of representing signals as sparse linear combinations of atoms selected from a dictionary. We consider an extension of this framework where the atoms are further assumed to be embedded in a tree. This is achieved using a recently introduced tree-structured sparse regularization norm, which has proven useful in several applications. This norm leads to regularized problems that are difficult to optimize, and in this chapter, we propose efficient algorithms for solving them. More precisely, we show that the proximal operator associated with this norm is computable exactly via a dual approach that can be viewed as the composition of elementary proximal operators. Our procedure has a complexity linear, or close to linear, in the number of atoms, and allows the use of accelerated gradient techniques to solve the tree-structured sparse approximation problem at the same computational cost as traditional ones using the ℓ 1 -norm. Our method is efficient and scales gracefully to millions of variables, which we illustrate in two types of applications: first, we consider fixed hierarchical dictionaries of wavelets to denoise natural images. Then, we apply our optimization tools in the context of dictionary learning, where learned dictionary elements naturally organize in a prespecified arborescent structure, leading to better performance in reconstruction of natural image patches. When applied to text documents, our method learns hierarchies of topics, thus providing a competitive alternative to probabilistic topic models.

The work presented in this chapter was achieved with the collaboration of Julien Mairal, Guillaume Obozinski and Francis Bach, with equal contribution between Julien Mairal and myself. The material of this chapter is based on the following work: R. Jenatton

Introduction

Modeling signals as sparse linear combinations of atoms selected from a dictionary has become a popular paradigm in many fields, including signal processing, statistics, and machine learning. This line of research, also known as sparse coding, has witnessed the development of several well-founded theoretical frameworks [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF][START_REF] Chen | Atomic decomposition by basis pursuit[END_REF][START_REF] Mallat | A wavelet tour of signal processing[END_REF][START_REF] Tropp | Greed is good: Algorithmic results for sparse approximation[END_REF][START_REF] Tropp | Just relax: Convex programming methods for identifying sparse signals in noise[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF][START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF] and the emergence of many efficient algorithmic tools [START_REF] Efron | Least angle regression[END_REF][START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Wright | Sparse reconstruction by separable approximation[END_REF][START_REF] Needell | CoSaMP: Iterative signal recovery from incomplete and inaccurate samples[END_REF][START_REF] Yuan | Comparison of optimization methods and software for large-scale l1-regularized linear classification[END_REF].

In many applied settings, the structure of the problem at hand, such as, e.g., the spatial arrangement of the pixels in an image, or the presence of variables corresponding to several levels of a given factor, induces relationships between dictionary elements. It is appealing to use this a priori knowledge about the problem directly to constrain the possible sparsity patterns. For instance, when the dictionary elements are partitioned into predefined groups corresponding to different types of features, one can enforce a similar block structure in the sparsity pattern-that is, allow only that either all elements of a group are part of the signal decomposition or that all are dismissed simultaneously (see [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF][START_REF] Stojnic | On the reconstruction of block-sparse signals with an optimal number of measurements[END_REF].

This example can be viewed as a particular instance of structured sparsity, which has been lately the focus of a large amount of research [START_REF] Baraniuk | Model-based compressive sensing[END_REF][START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF][START_REF] Huang | Learning with structured sparsity[END_REF][START_REF] Jacob | Group Lasso with overlaps and graph Lasso[END_REF]Jenatton et al., 2011a;[START_REF] Micchelli | A family of penalty functions for structured sparsity[END_REF]. In this chapter, we concentrate on a specific form of structured sparsity, which we call hierarchical sparse coding: the dictionary elements are assumed to be embedded in a directed tree T , and the sparsity patterns are constrained to form a connected and rooted subtree of T [START_REF] Donoho | CART and best-ortho-basis: a connection[END_REF][START_REF] Baraniuk | Optimal tree approximation with wavelets[END_REF][START_REF] Baraniuk | Near best tree approximation[END_REF][START_REF] Baraniuk | Model-based compressive sensing[END_REF][START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF][START_REF] Huang | Learning with structured sparsity[END_REF]. This setting extends more generally to a forest of directed trees. 1In fact, such a hierarchical structure arises in many applications. Wavelet decompositions lend themselves well to this tree organization because of their multiscale structure, and benefit from it for image compression and denoising [START_REF] Shapiro | Embedded image coding using zerotrees of wavelet coefficients[END_REF][START_REF] Crouse | Wavelet-based statistical signal processing using hidden Markov models[END_REF][START_REF] Baraniuk | Optimal tree approximation with wavelets[END_REF][START_REF] Baraniuk | Near best tree approximation[END_REF][START_REF] Baraniuk | Model-based compressive sensing[END_REF][START_REF] He | Exploiting structure in wavelet-based Bayesian compressive sensing[END_REF][START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF][START_REF] Huang | Learning with structured sparsity[END_REF]. In the same vein, edge filters of natural image patches can be represented in an arborescent fashion [START_REF] Zoran | The "tree-dependent components" of natural scenes are edge filters[END_REF]. Imposing these sparsity patterns has further proven useful in the context of hierarchical variable selection, e.g., when applied to kernel methods (Bach, 2008a), to log-linear models for the selection of potential orders [START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF], and to bioinformatics, to exploit the tree structure of gene networks for multi-task regression [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF]. Hierarchies of latent variables, typically used in neural networks and deep learning architectures (see Bengio, 2009, and references therein) have also emerged as a natural structure in several applications, notably to model text documents. In particular, in the context of topic models [START_REF] Blei | Latent dirichlet allocation[END_REF], a hierarchical model of latent variables based on Bayesian 4.1. Introduction non-parametric methods has been proposed by [START_REF] Blei | The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies[END_REF] to model hierarchies of topics.

To perform hierarchical sparse coding, our work builds upon the approach of [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF] who first introduced a sparsity-inducing norm Ω leading to this type of tree-structured sparsity pattern. We tackle the resulting nonsmooth convex optimization problem with proximal methods (e.g., [START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Wright | Sparse reconstruction by separable approximation[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF] and we show in this chapter that its key step, the computation of the proximal operator, can be solved exactly with a complexity linear, or close to linear, in the number of dictionary elements-that is, with the same complexity as for classical ℓ 1 -sparse decomposition problems [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF][START_REF] Chen | Atomic decomposition by basis pursuit[END_REF]. Concretely, given an m-dimensional signal x along with a dictionary D = [d 1 , . . . , d p ] ∈ R m×p composed of p atoms, the optimization problem at the core of our chapter can be written as

min α∈R p 1 2 x -Dα 2 2 + λΩ(α), with λ ≥ 0.
In this formulation, the sparsity-inducing norm Ω encodes a hierarchical structure among the atoms of D, where this structure is assumed to be known beforehand. The precise meaning of hierarchical structure and the definition of Ω will be made more formal in the next sections. A particular instance of this problem-known as the proximal problem-is central to our analysis and concentrates on the case where the dictionary D is orthogonal.

In addition to a speed benchmark that evaluates the performance of our proposed approach compared to other convex optimization techniques, two types of applications and experiments are carried out. First, we consider settings where the dictionary is fixed and given a priori, corresponding for instance to a basis of wavelets for the denoising of natural images. Second, we show how one can take advantage of this hierarchical sparse coding in the context of dictionary learning [START_REF] Olshausen | Sparse coding with an overcomplete basis set: A strategy employed by V1?[END_REF][START_REF] Aharon | The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representations[END_REF]Mairal et al., 2010a), where the dictionary is learned to adapt to the predefined tree structure. This extension of dictionary learning is notably shown to share interesting connections with hierarchical probabilistic topic models.

To summarize, the contributions of this chapter are threefold:

• We show that the proximal operator for a tree-structured sparse regularization can be computed exactly in a finite number of operations using a dual approach.

Our approach is equivalent to computing a particular sequence of elementary proximal operators, and has a complexity linear, or close to linear, in the number of variables. Accelerated gradient methods (e.g., [START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF] can then be applied to solve largescale tree-structured sparse decomposition problems at the same computational cost as traditional ones using the ℓ 1 -norm. • We propose to use this regularization scheme to learn dictionaries embedded in a tree, which, to the best of our knowledge, has not been done before in the context of structured sparsity. • Our method establishes a bridge between hierarchical dictionary learning and hierarchical topic models [START_REF] Blei | The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies[END_REF], which builds upon the interpretation 4. Proximal Methods for Structured Sparsity-Inducing Norms of topic models as multinomial PCA [START_REF] Buntine | Variational Extensions to EM and Multinomial PCA[END_REF], and can learn similar hierarchies of topics. This point is discussed in Sections 4.5.5 and 4.6. The rest of this chapter is organized as follows: Section 4.2 presents related work and the problem we consider. Section 4.3 is devoted to the algorithm we propose, and Section 4.4 introduces the dictionary learning framework and shows how it can be used with tree-structured norms. Section 4.5 presents several experiments demonstrating the effectiveness of our approach and Section 4.6 concludes the chapter.

Problem Statement and Related Work

Let us consider an input signal of dimension m, typically an image described by its m pixels, which we represent by a vector x in R m . In traditional sparse coding, we seek to approximate this signal by a sparse linear combination of atoms, or dictionary elements, represented here by the columns of a matrix D [d 1 , . . . , d p ] in R m×p . This can equivalently be expressed as x ≈ Dα for some sparse vector α in R p , i.e, such that the number of nonzero coefficients α 0 is small compared to p. The vector α is referred to as the code, or decomposition, of the signal x. In the rest of the chapter, we focus on specific sets of nonzero coefficients-or simply, nonzero patterns-for the decomposition vector α. In particular, we assume that we are given a tree2 T whose p nodes are indexed by j in {1, . . . , p}. We want the nonzero patterns of α to form a connected and rooted subtree of T ; in other words, if ancestors(j) ⊆ {1, . . . , p} denotes the set of indices corresponding to the ancestors3 of the node j in T (see Figure 4.1), the vector α obeys the following rule

α j = 0 ⇒ [ α k = 0 for all k in ancestors(j) ]. (4.1)
Informally, we want to exploit the structure of T in the following sense: the decomposition of any signal x can involve a dictionary element d j only if the ancestors of d j in the tree T are themselves part of the decomposition.

Problem Statement and Related Work

We now review previous work that has considered the sparse approximation problem with tree-structured constraints (4.1). Similarly to traditional sparse coding, there are basically two lines of research, that either (A) deal with nonconvex and combinatorial formulations that are in general computationally intractable and addressed with greedy algorithms, or (B) concentrate on convex relaxations solved with convex programming methods.

Nonconvex Approaches

For a given sparsity level s ≥ 0 (number of nonzero coefficients), the following nonconvex problem

min α∈R p α 0 ≤s 1 2 x -Dα 2 2 such that condition (4.1) is respected, (4.2)
has been addressed by [START_REF] Baraniuk | Optimal tree approximation with wavelets[END_REF]; [START_REF] Baraniuk | Near best tree approximation[END_REF] in the context of wavelet approximations with a greedy procedure. A penalized version of problem (4.2) (that adds λ α 0 to the objective function in place of the constraint α 0 ≤ s) has been considered by [START_REF] Donoho | CART and best-ortho-basis: a connection[END_REF], while studying the more general problem of best approximation from dyadic partitions (see Section 6 in [START_REF] Donoho | CART and best-ortho-basis: a connection[END_REF]. Interestingly, the algorithm we introduce in Section 4.3 shares conceptual links with the dynamic-programming approach of [START_REF] Donoho | CART and best-ortho-basis: a connection[END_REF], which was also used by [START_REF] Baraniuk | Model-based compressive sensing[END_REF], in the sense that the same order of traversal of the tree is used in both procedures. We investigate more thoroughly the relations between our algorithm and this approach in Appendix A.2.1. Problem (4.2) has been further studied for structured compressive sensing [START_REF] Baraniuk | Model-based compressive sensing[END_REF], with a greedy algorithm that builds upon [START_REF] Needell | CoSaMP: Iterative signal recovery from incomplete and inaccurate samples[END_REF]. Finally, [START_REF] Huang | Learning with structured sparsity[END_REF] have proposed a formulation related to (4.2), with a nonconvex penalty based on an information-theoretic criterion.

Convex Approach

We now turn to a convex reformulation of the constraint (4.1), which is the starting point for the convex optimization tools we develop in Section 4.3.

Hierarchical Sparsity-Inducing Norms

Condition (4.1) can be equivalently expressed by taking its contrapositive, thus leading to an intuitive way of penalizing the vector α to obtain tree-structured nonzero patterns. More precisely, defining descendants(j) ⊆ {1, . . . , p} analogously to ancestors(j) for j in {1, . . . , p}, condition (4.1) amounts to saying that if a dictionary element is not used in the decomposition, its descendants in the tree should not be used either. Formally, this writes down

α j = 0 ⇒ [ α k = 0 for all k in descendants(j) ].
(4.3)

Proximal Methods for Structured Sparsity-Inducing Norms

From now on, we denote by G the set defined by G {descendants(j); j ∈ {1, . . . , p}}, and refer to each member g of G as a group (Figure 4.2). To obtain a decomposition with the desired property (4.3), one can naturally penalize the number of groups g in G that are "involved" in the decomposition of x, i.e., that record at least one nonzero coefficient of α: g∈G δ g , with δ g 1 if there exists j ∈ g such that α j = 0, 0 otherwise. (4.4)

While this intuitive penalization is nonconvex (and not even continuous), a convex proxy has been introduced by [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]. It was further considered by Bach (2008a); [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF]; [START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF] in several different contexts. For any vector α ∈ R p , let us define Ω(α)

g∈G ω g α |g ,
where α |g is the vector of size p whose coordinates are equal to those of α for indices in the set g, and 0 otherwise4 . The notation . stands in practice either for the ℓ 2 -or ℓ ∞norm, and (ω g ) g∈G denotes some positive weights5 . As analyzed by [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF] and Jenatton et al. (2011a), when penalizing by Ω, some of the vectors α |g are set to zero for some g ∈ G.6 Therefore, the components of α corresponding to some complete subtrees of T are set to zero, which exactly matches condition (4.3), as illustrated in Figure 4.2.

Note that although we have presented for simplicity this hierarchical norm in the context of a single tree with a single element at each node, it can easily be extended to the case of forests of trees, and/or trees containing arbitrary numbers of dictionary elements at each node (with nodes possibly containing no dictionary element). More broadly, this formulation can be extended with the notion of tree-structured groups, which we now present:

Definition 1 (Tree-structured set of groups.) A set of groups G {g} g∈G is said to be tree-structured in {1, . . . , p}, if g∈G g = {1, . . . , p} and if for all g, h ∈ G, (g ∩ h = ∅) ⇒ (g ⊆ h or h ⊆ g). For such a set of groups, there exists a (non-unique) total order relation such that:

g h ⇒ g ⊆ h or g ∩ h = ∅ .
Given such a tree-structured set of groups G and its associated norm Ω, we are interested throughout the chapter in the following hierarchical sparse coding problem,

min α∈R p f (α) + λΩ(α), (4.5)
Figure 4.2: Left: example of a tree-structured set of groups G (dashed contours in red), corresponding to a tree T with p = 6 nodes represented by black circles. Right: example of a sparsity pattern induced by the tree-structured norm corresponding to G: the groups {2, 4}, {4} and {6} are set to zero, so that the corresponding nodes (in gray) that form subtrees of T are removed. The remaining nonzero variables {1, 3, 5} form a rooted and connected subtree of T . This sparsity pattern obeys the following equivalent rules: (i) if a node is selected, the same goes for all its ancestors. (ii) if a node is not selected, then its descendant are not selected.

where Ω is the tree-structured norm we have previously introduced, the non-negative scalar λ is a regularization parameter controlling the sparsity of the solutions of (4.5), and f a smooth convex loss function (see Section 4.3 for more details about the smoothness assumptions on f ). In the rest of the chapter, we will mostly use the square loss f (α) = 1 2 x -Dα 2 2 , with a dictionary D in R m×p , but the formulation of Eq. (4.5) extends beyond this context. In particular one can choose f to be the logistic loss, which is commonly used for classification problems (e.g., see [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition[END_REF].

Before turning to optimization methods for the hierarchical sparse coding problem, we consider a particular instance. The sparse group Lasso was recently considered by Sprechmann et al. (2010b) and [START_REF] Friedman | A note on the group lasso and a sparse group lasso[END_REF] as an extension of the group Lasso of [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF]. To induce sparsity both groupwise and within groups, Sprechmann et al. (2010b) and [START_REF] Friedman | A note on the group lasso and a sparse group lasso[END_REF] add an ℓ 1 term to the regularization of the group Lasso, which given a partition P of {1, . . . , p} in disjoint groups yields a regularized problem of the form

min α∈R p 1 2 x -Dα 2 2 + λ g∈P α |g 2 + λ ′ α 1 .
Since P is a partition, the set of groups in P and the singletons form together a treestructured set of groups according to definition 1 and the algorithm we will develop will therefore be applicable to this problem.

Optimization for Hierarchical Sparsity-Inducing Norms

While generic approaches like interior-point methods [START_REF] Boyd | Convex Optimization[END_REF]) and subgradient descent schemes [START_REF] Bertsekas | Nonlinear programming[END_REF] might be used to deal with the non-4. Proximal Methods for Structured Sparsity-Inducing Norms smooth norm Ω, several dedicated procedures have been proposed.

In [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF], a boosting-like technique is used, with a path-following strategy in the specific case where . is the ℓ ∞ -norm. Based on the variational equality

u 1 = min z∈R p + 1 2 p j=1 u 2 j z j + z j , (4.6)
Kim and Xing (2010) follow a reweighted least-square scheme that is well adapted to the square loss function. To the best of our knowledge, a formulation of this type is however not available when . is the ℓ ∞ -norm. In addition it requires an appropriate smoothing to become provably convergent. The same approach is considered by Bach (2008a), but built upon an active-set strategy. Other proposed methods consist of a projected gradient descent with approximate projections onto the ball {u ∈ R p ; Ω(u) ≤ λ} [START_REF] Schmidt | Convex structure learning in log-linear models: Beyond pairwise potentials[END_REF], and an augmented-Lagrangian based technique (Sprechmann et al., 2010b) for solving a particular case with two-level hierarchies.

While the previously listed first-order approaches are (1) loss-function dependent, and/or ( 2) not guaranteed to achieve optimal convergence rates, and/or (3) not able to yield sparse solutions without a somewhat arbitrary post-processing step, we propose to resort to proximal methods7 that do not suffer from any of these drawbacks.

Optimization

We begin with a brief introduction to proximal methods, necessary to present our contributions. From now on, we assume that f is convex and continuously differentiable with Lipschitz-continuous gradient. It is worth mentioning that there exist various proximal schemes in the literature that differ in their settings (e.g., batch versus stochastic) and/or the assumptions made on f . For instance, the material we develop in this chapter could also be applied to online/stochastic frameworks [START_REF] Duchi | Efficient online and batch learning using forward backward splitting[END_REF][START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF][START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] and to possibly nonsmooth functions f (e.g., [START_REF] Duchi | Efficient online and batch learning using forward backward splitting[END_REF][START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF]Combettes and Pesquet, 2010, and references therein). Finally, most of the technical proofs of this section are presented in Appendix A.2.2 for readability.

Proximal Operator for the Norm Ω

Proximal methods have drawn increasing attention in the signal processing (e.g., [START_REF] Becker | NESTA: A Fast and Accurate First-order Method for Sparse Recovery[END_REF][START_REF] Wright | Sparse reconstruction by separable approximation[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF], and numerous references therein) and the machine learning communities (e.g., Bach et al., 2011, and references therein), especially because of their convergence rates (optimal for the class of first-order techniques) and their ability to deal with large nonsmooth convex problems (e.g., [START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]. In a nutshell, these methods can be seen as a natural extension of gradient-based techniques when the objective function to minimize has a nonsmooth part. Proximal methods are iterative procedures. The simplest version of this class of methods linearizes at each iteration the function f around the current estimate α, and this estimate is updated as the (unique by strong convexity) solution of the proximal problem, defined as follows:

min α∈R p f ( α) + (α -α) ⊤ ∇f ( α) + λΩ(α) + L 2 α -α 2 2 .
The quadratic term keeps the update in a neighborhood where f is close to its linear approximation, and L > 0 is a parameter which is an upper bound on the Lipschitz constant of ∇f . This problem can be equivalently rewritten as:

min α∈R p 1 2 α -α - 1 L ∇f ( α) 2 2 + λ L Ω(α).
Solving efficiently and exactly this problem is crucial to enjoy the fast convergence rates of proximal methods. In addition, when the nonsmooth term Ω is not present, the previous proximal problem exactly leads to the standard gradient update rule. More generally, we define the proximal operator:

Definition 2 (Proximal Operator)
The proximal operator associated with our regularization term λΩ, which we denote by Prox λΩ , is the function that maps a vector u ∈ R p to the unique solution of

min v∈R p 1 2 u -v 2 2 + λΩ(v). (4.7)
This operator was initially introduced by [START_REF] Moreau | Fonctions convexes duales et points proximaux dans un espace hilbertien[END_REF] to generalize the projection operator onto a convex set. What makes proximal methods appealing for solving sparse decomposition problems is that this operator can be often computed in closed-form. For instance,

• When Ω is the ℓ 1 -norm-that is, Ω(u) = u 1 , the proximal operator is the well-known elementwise soft-thresholding operator,

∀j ∈ 1; p , u j → sign(u j )(|u j | -λ) + = 0 if |u j | ≤ λ sign(u j )(|u j | -λ) otherwise.
• When Ω is a group-Lasso penalty with ℓ 2 -norms-that is, Ω(u) = g∈G u |g 2 , with G being a partition of 1; p , the proximal problem is separable in every group, and the solution is a generalization of the soft-thresholding operator to groups of variables:

∀g ∈ G , u |g → u |g -Π . 2 ≤λ [u |g ] =    0 if u |g 2 ≤ λ u |g 2 -λ u |g 2 u |g otherwise,
where Π . 2 ≤λ denotes the orthogonal projection onto the ball of the ℓ 2 -norm of radius λ.

Proximal Methods for Structured Sparsity-Inducing Norms

• When Ω is a group-Lasso penalty with ℓ ∞ -norms-that is, Ω(u) = g∈G u |g ∞ , the solution is also a group-thresholding operator:

∀g ∈ G, u |g → u |g -Π . 1 ≤λ [u |g ],
where Π . 1 ≤λ denotes the orthogonal projection onto the ℓ 1 -ball of radius λ, which can be solved in O(p) operations [START_REF] Brucker | An O(n) algorithm for quadratic knapsack problems[END_REF][START_REF] Maculan | A linear-time median-finding algorithm for projecting a vector on the simplex of Rn[END_REF]. Note that when u |g 1 ≤ λ, we have a group-thresholding effect, with u |g -Π . 1 ≤λ [u |g ] = 0. More generally, a classical result (see, e.g., [START_REF] Combettes | Proximal splitting methods in signal processing[END_REF][START_REF] Wright | Sparse reconstruction by separable approximation[END_REF] says that the proximal operator for a norm . can be computed as the residual of the projection of a vector onto a ball of the dual-norm denoted by . * , and defined for any vector κ in R p by κ * max z ≤1 z ⊤ κ. 8 This is a classical duality result for proximal operators leading to the different closed forms we have just presented. We have indeed that Prox λ . 2 = Id -Π . 2 ≤λ and Prox λ . ∞ = Id -Π . 1 ≤λ , where Id stands for the identity operator. Obtaining such closed forms is, however, not possible anymore as soon as some groups in G overlap, which is always the case in our hierarchical setting with tree-structured groups.

A Dual Formulation of the Proximal Problem

We now show that Eq. (4.7) can be solved using a dual approach, as described in the following lemma. The result relies on conic duality [START_REF] Boyd | Convex Optimization[END_REF], and does not make any assumption on the choice of the norm . :

Lemma 2 (Dual of the proximal problem) Let u ∈ R p and let us consider the problem

max ξ∈R p×|G| - 1 2 u - g∈G ξ g 2 2 -u 2 2 s.t. ∀g ∈ G, ξ g * ≤ λω g and ξ g j = 0 if j / ∈ g, (4.8)
where ξ = (ξ g ) g∈G and ξ g j denotes the j-th coordinate of the vector ξ g in R p . Then, problems (4.7) and (4.8) are dual to each other and strong duality holds. In addition, the pair of primal-dual variables {v, ξ} is optimal if and only if ξ is a feasible point of the optimization problem (4.8), and

v = u -g∈G ξ g and ∀g ∈ G, ξ g = Π . * ≤λωg (v |g + ξ g ), (4.9)
where we denote by Π . * ≤λωg the Euclidean projection onto the ball {κ ∈ R p ; κ * ≤ λω g }.

Optimization

Note that we focus here on specific tree-structured groups, but the previous lemma is valid regardless of the nature of G. The rationale of introducing such a dual formulation is to consider an equivalent problem to (4.7) that removes the issue of overlapping groups at the cost of a larger number of variables. In Eq. (4.7), one is indeed looking for a vector v of size p, whereas one is considering a matrix ξ in R p×|G| in Eq. (4.8) with g∈G |g| nonzero entries, but with separable (convex) constraints for each of its columns.

This specific structure makes it possible to use block coordinate ascent [START_REF] Bertsekas | Nonlinear programming[END_REF]. Such a procedure is presented in Algorithm 6. It optimizes sequentially Eq. (4.8) with respect to the variable ξ g , while keeping fixed the other variables ξ h , for h = g. It is easy to see from Eq. (4.8) that such an update of a column ξ g , for a group g in G, amounts to computing the orthogonal projection of the vector u |gh =g ξ h |g onto the ball of radius λω g of the dual norm . * .

Algorithm 6 Block coordinate ascent in the dual

Inputs: u ∈ R p and set of groups G.

Outputs: (v, ξ) (primal-dual solutions).

Initialization: ξ = 0. while ( maximum number of iterations not reached

) do for g ∈ G do ξ g ← Π . * ≤λωg ( u -h =g ξ h |g )
. end for end while v ← u -g∈G ξ g .

Convergence in One Pass

In general, Algorithm 6 is not guaranteed to solve exactly Eq. (4.7) in a finite number of iterations. However, when . is the ℓ 2 -or ℓ ∞ -norm, and provided that the groups in G are appropriately ordered, we now prove that only one pass of Algorithm 6, i.e., only one iteration over all groups, is sufficient to obtain the exact solution of Eq. (4.7). This result constitutes the main technical contribution of the chapter and is the key for the efficiency of our procedure.

Before stating this result, we need to introduce a lemma showing that, given two nested groups g, h such that g ⊆ h ⊆ {1, . . . , p}, if ξ g is updated before ξ h in Algorithm 6, then the optimality condition for ξ g is not perturbed by the update of ξ h . Lemma 3 (Projections with nested groups) Let . denote either the ℓ 2 -or ℓ ∞ -norm, and g and h be two nested groups-that is, g ⊆ h ⊆ {1, . . . , p}. Let u be a vector in R p , and let us consider the successive projections

ξ g Π . * ≤tg (u |g ) and ξ h Π . * ≤th (u |h -ξ g ), with t g , t h > 0. Let us introduce v = u -ξ g -ξ h . The following relationships hold ξ g = Π . * ≤tg (v |g + ξ g ) and ξ h = Π . * ≤th (v |h + ξ h ).

Proximal Methods for Structured Sparsity-Inducing Norms

The previous lemma establishes the convergence in one pass of Algorithm 6 in the case where G only contains two nested groups g ⊆ h, provided that ξ g is computed before ξ h . Let us illustrate this fact more concretely. After initializing ξ g and ξ h to zero, Algorithm 6 first updates ξ g with the formula ξ g ← Π . * ≤λωg (u |g ), and then performs the following update: ξ h ← Π . * ≤λωh (u |hξ g ) (where we have used that ξ g = ξ g |h since g ⊆ h). We are now in position to apply Lemma 3 which states that the primal/dual variables {v, ξ g , ξ h } satisfy the optimality conditions (4.9), as described in Lemma 2. In only one pass over the groups {g, h}, we have in fact reached a solution of the dual formulation presented in Eq. (4.8), and in particular, the solution of the proximal problem (4.7).

In the following proposition, this lemma is extended to general tree-structured sets of groups G: Proposition 9 (Convergence in one pass) Suppose that the groups in G are ordered according to the total order relation of Definition 1 and that the norm . is either the ℓ 2 -or ℓ ∞ -norm. Then, after initializing ξ to 0, a single pass of Algorithm 6 over G with the order yields the solution of the proximal problem (4.7).

Proof. The proof largely relies on Lemma 3 and proceeds by induction. By definition of Algorithm 6, the feasibility of ξ is always guaranteed. We consider the following induction hypothesis

H(h) ∀g h, it holds that ξ g = Π . * ≤λωg ([u -g ′ h ξ g ′ ] |g + ξ g ) .
Since the dual variables ξ are initially equal to zero, the summation over g ′ h, g ′ = g is equivalent to a summation over g ′ = g. We initialize the induction with the first group in G, that, by definition of , does not contain any other group. The first step of Algorithm 6 easily shows that the induction hypothesis H is satisfied for this first group. We now assume that H(h) is true and consider the next group h ′ , h h ′ , in order to prove that H(h ′ ) is also satisfied. We have for each group g ⊆ h,

ξ g = Π . * ≤λωg ([u -g ′ h ξ g ′ ] |g + ξ g ) = Π . * ≤λωg ([u -g ′ h ξ g ′ + ξ g ] |g ). Since ξ g |h ′ = ξ g for g ⊆ h ′ , we have [u -g ′ h ξ g ′ ] |h ′ = [u -g ′ h ξ g ′ ] |h ′ + ξ g -ξ g = [u -g ′ h ξ g ′ + ξ g ] |h ′ -ξ g ,
and following the update rule for the group h ′ ,

ξ h ′ = Π . * ≤λω h ′ ([u -g ′ h ξ g ′ ] |h ′ ) = Π . * ≤λω h ′ ([u -g ′ h ξ g ′ + ξ g ] |h ′ -ξ g ).
At this point, we can apply Lemma 3 for each group g ⊆ h, which proves that the induction hypothesis H(h ′ ) is true. Let us introduce v u -g∈G ξ g . We have shown that for all g in G, ξ g = Π . * ≤λωg (v |g + ξ g ). As a result, the pair {v, ξ} satisfies the optimality conditions (4.9) of problem (4.8). Therefore, after one complete pass over g ∈ G, the primal/dual pair {v, ξ} is optimal, and in particular, v is the solution of problem (4.7).

Optimization

We recall that the total order relation introduced in Definition 1 is defined so that when a group h is included in a group g, then h should be processed before g. We illustrate in Figure 4.3 the practical implications of Proposition 9. More precisely, we consider Algorithm 6 with both the "right" order for G (as advocated by Proposition 9), and random orders. We then monitor the cost function of the primal proximal problem and its dual counterpart, respectively given in (4.7) and (4.8). Using conic duality, we have derived a dual formulation of the proximal operator, leading to Algorithm 6 which is generic and works for any norm . , as long as one is able to perform projections onto balls of the dual norm . * . We have further shown that when . is the ℓ 2 -or the ℓ ∞ -norm, a single pass provides the exact solution when the groups G are correctly ordered. We show however in Appendix A.2.3, that, perhaps surprisingly, the conclusions of Proposition 9 do not hold for general ℓ q -norms, if q / ∈ {1, 2, ∞}. Next, we give another interpretation of this result.

Interpretation in Terms of Composition of Proximal Operators

In Algorithm 6, since all the vectors ξ g are initialized to 0, when the group g is considered, we have by induction uh =g ξ h = uh g ξ h . Thus, to maintain at each iteration of the inner loop v = uh =g ξ h one can instead update v after updating ξ g according to v ← vξ g . Moreover, since ξ g is no longer needed in the algorithm, and since only the entries of v indexed by g are updated, we can combine the two updates into v |g ← v |g -Π . * ≤λωg (v |g ), leading to a simplified Algorithm 7 equivalent to Algorithm 6.

Proximal Methods for Structured Sparsity-Inducing Norms

Algorithm 7 Practical Computation of the Proximal Operator for ℓ 2 -or ℓ ∞ -norms.

Inputs: u ∈ R p and an ordered tree-structured set of groups G.

Outputs: v (primal solution).

Initialization: v = u. for g ∈ G, following the order , do v |g ← v |g -Π . * ≤λωg (v |g ). end for
Actually, in light of the classical relationship between proximal operator and projection (as discussed in Section 4.3.1), it is easy to show that each update

v |g ← v |g -Π . * ≤λωg (v |g ) is equivalent to v |g ← Prox λωg . [v |g ].
To simplify the notations, we define the proximal operator for a group g in G as Prox g (u)

Prox λωg . (u |g ) for every vector u in R p .

Thus, Algorithm 7 in fact performs a sequence of |G| proximal operators, and we have shown the following corollary of Proposition 9:

Corollary 4 (Composition of Proximal Operators) Let g 1 . . . g m such that G = {g 1 , . . . , g m }. The proximal operator Prox λΩ associated with the norm Ω can be written as the composition of elementary operators:

Prox λΩ = Prox gm • . . . • Prox g 1 .

Efficient Implementation and Complexity

Since Algorithm 7 involves |G| projections on the dual balls (respectively the ℓ 2and the ℓ 1 -balls for the ℓ 2 -and ℓ ∞ -norms) of vectors in R p , in a first approximation, its complexity is at most O(p 2 ), because each of these projections can be computed in O(p) operations [START_REF] Brucker | An O(n) algorithm for quadratic knapsack problems[END_REF][START_REF] Maculan | A linear-time median-finding algorithm for projecting a vector on the simplex of Rn[END_REF]. But in fact, the algorithm performs one projection for each group g involving |g| variables, and the total complexity is therefore O g∈G |g| . By noticing that if g and h are two groups with the same depth in the tree, then g ∩h = ∅, it is easy to show that the number of variables involved in all the projections is less than or equal to dp, where d is the depth of the tree:

Lemma 4 (Complexity of Algorithm 7) Algorithm 7 gives the solution of the primal problem Eq. (4.7) in O(pd) operations, where d is the depth of the tree.

Lemma 4 should not suggest that the complexity is linear in p, since d could depend of p as well, and in the worst case the hierarchy is a chain, yielding d = p -1. However, in a balanced tree, d = O(log(p)). In practice, the structures we have considered experimentally are relatively flat, with a depth not exceeding d = 5, and the complexity is therefore almost linear.

Moreover, in the case of the ℓ 2 -norm, it is actually possible to propose an algorithm with complexity O(p). Indeed, in that case each of the proximal operators Prox g is a Algorithm 8 Fast computation of the Proximal operator for ℓ 2 -norm case. Require: u ∈ R p (input vector), set of groups G, (ω g ) g∈G (positive weights), and g 0 (root of the tree).

1: Variables: ρ = (ρ g ) g∈G in R |G| (scaling factors); v in R p (output, primal variable).

2: computeSqNorm(g 0 ).

3: recursiveScaling(g 0 ,1). 4: return v (primal solution).

Procedure computeSqNorm(g)

1:
Compute the squared norm of the group:

η g ← u root(g) 2 2 + h∈children(g) computeSqNorm(h). 2:
Compute the scaling factor of the group: ρ g ← 1λω g / √ η g + .

3: return η g ρ 2 g . Procedure recursiveScaling(g,t)

1: ρ g ← tρ g . 2: v root(g) ← ρ g u root(g) . 3: for h ∈ children(g) do 4:
recursiveScaling(h,ρ g ). 5: end for scaling operation: v |g ← 1λω g / v |g 2 + v |g . The composition of these operators in Algorithm 6 thus corresponds to performing sequences of scaling operations. The idea behind Algorithm 8 is that the corresponding scaling factors depend only on the norms of the successive residuals of the projections and that these norms can be computed recursively in one pass through all nodes in O(p) operations; finally, computing and applying all scalings to each entry takes then again O(p) operations.

To formulate the algorithm, two new notations are used: for a group g in G, we denote by root(g) the indices of the variables that are at the root of the subtree corresponding to g, 9 and by children(g) the set of groups that are the children of root(g) in the tree. For example, in the tree presented in Figure 4.2, root({3, 5, 6}) = {3}, root({1, 2, 3, 4, 5, 6}) = {1}, children({3, 5, 6}) = {{5}, {6}}, and children({1, 2, 3, 4, 5, 6}) = {{2, 4}, {3, 5, 6}}. Note that all the groups of children(g) are necessarily included in g. The next lemma is proved in Appendix A.2.2.

Lemma 5 (Correctness and complexity of Algorithm 8)

When . is chosen to be the ℓ 2 -norm, Algorithm 8 gives the solution of the primal problem Eq. (4.7) in O(p) operations.

So far the dictionary D was fixed to be for example a wavelet basis. In the next section, we apply the tools we developed for solving efficiently problem (4.5) to learn a dictionary D adapted to our hierarchical sparse coding formulation. 9. As a reminder, root(g) is not a singleton when several dictionary elements are considered per node.

Application to Dictionary Learning

We start by briefly describing dictionary learning.

The Dictionary Learning Framework

Let us consider a set X = [x 1 , . . . , x n ] in R m×n of n signals of dimension m. Dictionary learning is a matrix factorization problem which aims at representing these signals as linear combinations of the dictionary elements, that are the columns of a matrix D = [d 1 , . . . , d p ] in R m×p . More precisely, the dictionary D is learned along with a matrix of decomposition coefficients A = [α 1 , . . . , α n ] in R p×n , so that x i ≈ Dα i for every signal x i .

While learning simultaneously D and A, one may want to encode specific prior knowledge about the problem at hand, such as, for example, the positivity of the decomposition [START_REF] Lee | Learning the parts of objects by non-negative matrix factorization[END_REF], or the sparsity of A [START_REF] Olshausen | Sparse coding with an overcomplete basis set: A strategy employed by V1?[END_REF][START_REF] Aharon | The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representations[END_REF][START_REF] Lee | Efficient sparse coding algorithms[END_REF]Mairal et al., 2010a). This leads to penalizing or constraining (D, A) and results in the following formulation:

min D∈D,A∈A 1 n n i=1 1 2 x i -Dα i 2 2 + λΨ(α i ) , ( 4.10)
where A and D denote two convex sets and Ψ is a regularization term, usually a norm or a squared norm, whose effect is controlled by the regularization parameter λ > 0. Note that D is assumed to be bounded to avoid any degenerate solutions of Problem (4.10).

For instance, the standard sparse coding formulation takes Ψ to be the ℓ 1 -norm, D to be the set of matrices in R m×p whose columns have unit ℓ 2 -norm, with A = R p×n [START_REF] Olshausen | Sparse coding with an overcomplete basis set: A strategy employed by V1?[END_REF][START_REF] Lee | Efficient sparse coding algorithms[END_REF]Mairal et al., 2010a). However, this classical setting treats each dictionary element independently from the others, and does not exploit possible relationships between them. To embed the dictionary in a tree structure, we therefore replace the ℓ 1 -norm by our hierarchical norm and set Ψ = Ω in Eq. (4.10).

A question of interest is whether hierarchical priors are more appropriate in supervised settings or in the matrix-factorization context in which we use it. It is not so common in the supervised setting to have strong prior information that allows us to organize the features in a hierarchy. On the contrary, in the case of dictionary learning, since the atoms are learned, one can argue that the dictionary elements learned will have to match well the hierarchical prior that is imposed by the regularization. In other words, combining structured regularization with dictionary learning has precisely the advantage that the dictionary elements will self-organize to match the prior.

Learning the Dictionary

Optimization for dictionary learning has already been intensively studied. We choose in this chapter a typical alternating scheme, which optimizes in turn D and A = [α 1 , . . . , α n ] while keeping the other variable fixed [START_REF] Aharon | The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representations[END_REF]Lee et al., 4.4. Application to Dictionary Learning 2007; Mairal et al., 2010a).10 Of course, the convex optimization tools we develop in this chapter do not change the intrinsic non-convex nature of the dictionary learning problem. However, they solve the underlying convex subproblems efficiently, which is crucial to yield good results in practice. In the next section, we report good performance on some applied problems, and we show empirically that our algorithm is stable and does not seem to get trapped in bad local minima. The main difficulty of our problem lies in the optimization of the vectors α i , i in {1, . . . , n}, for the dictionary D kept fixed. Because of Ω, the corresponding convex subproblem is nonsmooth and has to be solved for each of the n signals considered. The optimization of the dictionary D (for A fixed), which we discuss first, is in general easier.

Updating the dictionary D. We follow the matrix-inversion free procedure of Mairal et al. (2010a) to update the dictionary. This method consists in iterating block-coordinate descent over the columns of D. Specifically, we assume that the domain set D has the form

D µ {D ∈ R m×p , µ d j 1 + (1 -µ) d j 2 2 ≤ 1, for all j ∈ {1, . . . , p}}, (4.11) or D + µ D µ ∩ R m×p + , with µ ∈ [0, 1].
The choice for these particular domain sets is motivated by the experiments of Section 4.5. For natural image patches, the dictionary elements are usually constrained to be in the unit ℓ 2 -norm ball (i.e., D = D 0 ), while for topic modeling, the dictionary elements are distributions of words and therefore belong to the simplex (i.e., D = D + 1 ). The update of each dictionary element amounts to performing a Euclidean projection, which can be computed efficiently (Mairal et al., 2010a). Concerning the stopping criterion, we follow the strategy from the same authors and go over the columns of D only a few times, typically 5 times in our experiments. Although we have not explored locality constraints on the dictionary elements, these have been shown to be particularly relevant to some applications such as patch-based image classification [START_REF] Yu | Nonlinear learning using local coordinate coding[END_REF]. Combining tree structure and locality constraints is an interesting future research.

Updating the vectors α i . The procedure for updating the columns of A is based on the results derived in Section 4.3.3. Furthermore, positivity constraints can be added on the domain of A, by noticing that for our norm Ω and any vector u in R p , adding these constraints when computing the proximal operator is equivalent to solving

min v∈R p 1 2 [u] + -v 2 2 + λΩ(v).
This equivalence is proved in Appendix A.2.2. We will indeed use positive decompositions to model text corpora in Section 4.5. Note that by constraining the decompositions α i to be nonnegative, some entries α i j may be set to zero in addition to those already zeroed out by the norm Ω. As a result, the sparsity patterns obtained in this way might not satisfy the tree-structured condition (4.1) anymore.

Proximal Methods for Structured Sparsity-Inducing Norms

Experiments

We next turn to the experimental validation of our hierarchical sparse coding.

Implementation Details

In Section 4.3.3, we have shown that the proximal operator associated to Ω can be computed exactly and efficiently. The problem is therefore amenable to fast proximal algorithms that are well suited to nonsmooth convex optimization. Specifically, we tried the accelerated scheme from both [START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF] and [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], and finally opted for the latter since, for a comparable level of precision, fewer calls of the proximal operator are required. The basic proximal scheme presented in Section 4.3.1 is formalized by [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF] as an algorithm called ISTA; the same authors propose moreover an accelerated variant, FISTA, which is a similar procedure, except that the operator is not directly applied on the current estimate, but on an auxiliary sequence of points that are linear combinations of past estimates. This latter algorithm has an optimal convergence rate in the class of first-order techniques, and also allows for warm restarts, which is crucial in the alternating scheme of dictionary learning. 11 Finally, we monitor the convergence of the algorithm by checking the relative decrease in the cost function. 12 Unless otherwise specified, all the algorithms used in the following experiments are implemented in C/C++, with a Matlab interface. Our implementation is freely available at http://www.di.ens.fr/willow/SPAMS/.

Speed Benchmark

To begin with, we conduct speed comparisons between our approach and other convex programming methods, in the setting where Ω is chosen to be a linear combination of ℓ 2 -norms. The algorithms that take part in the following benchmark are: • Proximal methods, with ISTA and the accelerated FISTA methods [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF].

• A reweighted-least-square scheme (Re-ℓ 2 ), as described by Jenatton et al. (2011a); [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF]. This approach is adapted to the square loss, since closed-form updates can be used. 13 • Subgradient descent, whose step size is taken to be equal either to a/(k + b) or a/( √ k + b) (respectively referred to as SG and SG sqrt ), where k is the iteration number, and (a, b) are the best 14 parameters selected on the logarithmic grid (a, b) ∈ {10 -4 , . . . , 10 3 } × {10 -2 , . . . , 10 5 }.

11. Unless otherwise specified, the initial stepsize in ISTA/FISTA is chosen as the maximum eigenvalue of the sampling covariance matrix divided by 100, while the growth factor in the line search is set to 1.5.

12. We are currently investigating algorithms for computing duality gaps based on network flow optimization tools (Mairal et al., 2010b).

13. The computation of the updates related to the variational formulation (4.6) also benefits from the hierarchical structure of G, and can be performed in O(p) operations.

14. "The best step size" is understood as being the step size leading to the smallest cost function after 500 iterations. 100 4.5. Experiments • A commercial software (Mosek, available at http://www.mosek.com/) for secondorder cone programming (SOCP). Moreover, the experiments we carry out cover various settings, with notably different sparsity regimes, i.e., low, medium and high, respectively corresponding to about 50%, 10% and 1% of the total number of dictionary elements. Eventually, all reported results are obtained on a single core of a 3.07Ghz CPU with 8GB of memory. The experiment is small scale, m = 256, p = 151, and shows the performances of six optimization methods (see main text for details) for three levels of regularization. The curves represent the relative value of the objective to the optimal value as a function of the computational time in second on a log 10 / log 10 scale. All reported results are obtained by averaging 5 runs.

Hierarchical dictionary of natural image patches

In this first benchmark, we consider a least-squares regression problem regularized by Ω that arises in the context of denoising of natural image patches, as further exposed in Section 4.5.4. In particular, based on a hierarchical dictionary, we seek to reconstruct noisy 16×16-patches. The dictionary we use is represented on Figure 4.9. Although the problem involves a small number of variables, i.e., p = 151 dictionary elements, it has to be solved repeatedly for tens of thousands of patches, at moderate precision. It is therefore crucial to be able to solve this problem quickly and efficiently.

We can draw several conclusions from the results of the simulations reported in Figure 4.4. First, we observe that in most cases, the accelerated proximal scheme performs better than the other approaches. In addition, unlike FISTA, ISTA seems to suffer in non-sparse scenarios. In the least sparse setting, the reweighted-ℓ 2 scheme is the only method that competes with FISTA. It is however not able to yield truly sparse solutions, and would therefore need a subsequent (somewhat arbitrary) thresholding operation. As expected, the generic techniques such as SG and SOCP do not compete with dedicated algorithms. . Three levels of regularization are considered. The curves represent the relative value of the objective to the optimal value as a function of the computational time in second on a log 10 / log 10 scale. In the highly regularized setting, tuning the step-size for the subgradient turned out to be difficult, which explains the behavior of SG in the first iterations.

Proximal Methods for Structured Sparsity-Inducing Norms

Multi-class classification of cancer diagnosis

The second benchmark explores a different supervised learning setting, where f is no longer the square loss function. The goal is to demonstrate that our optimization tools apply in various scenarios, beyond traditional sparse approximation problems. To this end, we consider a gene expression dataset15 in the context of cancer diagnosis. More precisely, we focus on a multi-class classification problem where the number m of samples to be classified is small compared to the number p of gene expressions that characterize these samples. Each atom thus corresponds to a gene expression across the m samples, whose class labels are recorded in the vector x in R m .

The dataset contains m = 308 samples, p = 30 017 variables and 26 classes. In addition, the data exhibit highly-correlated dictionary elements. Inspired by [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF], we build the tree-structured set of groups G using Ward's hierarchical clustering [START_REF] Johnson | Hierarchical clustering schemes[END_REF] on the gene expressions. The norm Ω built in this way aims at capturing the hierarchical structure of gene expression networks [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF].

Instead of the square loss function, we consider the multinomial logistic loss function that is better suited to deal with multi-class classification problems (see, e.g., [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition[END_REF]. As a direct consequence, algorithms whose applicability crucially depends on the choice of the loss function f are removed from the benchmark. This is the case with reweighted-ℓ 2 schemes that do not have closed-form updates anymore. Importantly, the choice of the multinomial logistic loss function leads to an optimization problem over a matrix with dimensions p times the number of classes (i.e., a total of 30 017 × 26 ≈ 780 000 variables). Also, due to scalability issues, generic interior point solvers could not be considered here.

The results in Figure 4.5 highlight that the accelerated proximal scheme performs overall better that the two other methods. Again, it is important to note that both proximal algorithms yield sparse solutions, which is not the case for SG.

Denoising with Tree-Structured Wavelets

We demonstrate in this section how a tree-structured sparse regularization can improve classical wavelet representation, and how our method can be used to efficiently solve the corresponding large-scale optimization problems. We consider two wavelet orthonormal bases, Haar and Daubechies3 (see [START_REF] Mallat | A wavelet tour of signal processing[END_REF], and choose a classical quad-tree structure on the coefficients, which has notably proven to be useful for image compression problems [START_REF] Baraniuk | Optimal tree approximation with wavelets[END_REF]. This experiment follows the approach of Zhao et al. ( 2009) who used the same tree-structured regularization in the case of small onedimensional signals, and the approach of [START_REF] Baraniuk | Model-based compressive sensing[END_REF] and [START_REF] Huang | Learning with structured sparsity[END_REF] images where images were reconstructed from compressed sensing measurements with a hierarchical nonconvex penalty.

We compare the performance for image denoising of both nonconvex and convex approaches. Specifically, we consider the following formulation

min α∈R m 1 2 x -Dα 2 2 + λψ(α) = min α∈R m 1 2 D ⊤ x -α 2 2 + λψ(α),
where D is one of the orthonormal wavelet basis mentioned above, x is the input noisy image, Dα is the estimate of the denoised image, and ψ is a sparsity-inducing regularization. Note that in this case, m = p. We first consider classical settings where ψ is either the ℓ 1 -norm-this leads to the wavelet soft-thresholding method of Donoho and Johnstone (1995)-or the ℓ 0 -pseudo-norm, whose solution can be obtained by hardthresholding (see [START_REF] Mallat | A wavelet tour of signal processing[END_REF]. Then, we consider the convex tree-structured regularization Ω defined as a sum of ℓ 2 -norms (ℓ ∞ -norms), which we denote by Ω ℓ 2 (respectively Ω ℓ∞ ). Since the basis is here orthonormal, solving the corresponding decomposition problems amounts to computing a single instance of the proximal operator. As a result, when ψ is Ω ℓ 2 , we use Algorithm 8 and for Ω ℓ∞ , Algorithm 7 is applied. Finally, we consider the nonconvex tree-structured regularization used by [START_REF] Baraniuk | Model-based compressive sensing[END_REF] denoted here by ℓ tree 0 , which we have presented in Eq. (4.4); the implementation details for ℓ tree 0 can be found in Appendix A.2.1. Compared to [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF], the novelty of our approach is essentially to be able to solve efficiently and exactly large-scale instances of this problem. We use 12 classical standard test images, 16 and generate noisy versions of them corrupted by a white Gaussian noise of variance σ. For each image, we test several values of λ = 2 i 4 σ √ log m, with i taken in a specific range. 17 We then keep 16. These images are used in classical image denoising benchmarks. See Mairal et al. (2009b). 17. For the convex formulations, i ranges in {-15, -14, . . . , 15}, while in the nonconvex case i ranges in {-24, . . . , 48}. [START_REF] Mallat | A wavelet tour of signal processing[END_REF], for two nonconvex approaches (ℓ 0 and ℓ tree 0 ) and three different convex regularizationsthat is, the ℓ 1 -norm, the tree-structured sum of ℓ 2 -norms (Ω ℓ 2 ), and the tree-structured sum of ℓ ∞ -norms (Ω ℓ∞ ). Best results for each level of noise and each wavelet type are in bold. Bottom part of the tables: Average improvement in PSNR with respect to the ℓ 0 nonconvex method (the standard deviations are computed over the 12 images). CPU times (in second) averaged over all images and noise realizations are reported in brackets next to the names of the methods they correspond to. the parameter λ giving the best reconstruction error. The factor σ √ log m is a classical heuristic for choosing a reasonable regularization parameter (see [START_REF] Mallat | A wavelet tour of signal processing[END_REF]. We provide reconstruction results in terms of PSNR in Table 4.1. 18 We report in this table

Proximal Methods for

Experiments

the results when Ω is chosen to be a sum of ℓ 2 -norms or ℓ ∞ -norms with weights ω g all equal to one. Each experiment was run 5 times with different noise realizations. In every setting, we observe that the tree-structured norm significantly outperforms the ℓ 1 -norm and the nonconvex approaches. We also present a visual comparison on two images on Figure 4.6, showing that the tree-structured norm reduces visual artefacts (these artefacts are better seen by zooming on a computer screen). The wavelet transforms in our experiments are computed with the matlabPyrTools software. This experiment does of course not provide state-of-the-art results for image denoising (see Mairal et al., 2009b, and references therein), but shows that the tree-structured regularization significantly improves the reconstruction quality for wavelets. In this experiment the convex setting Ω ℓ 2 and Ω ℓ∞ also outperforms the nonconvex one ℓ tree 0 .20 We also note that the speed of our approach makes it scalable to real-time applications. Solving the proximal problem for an image with m = 512 × 512 = 262 144 pixels takes approximately 0.013 seconds on a single core of a 3.07GHz CPU if Ω is a sum of ℓ 2 -norms, and 0.02 seconds when it is a sum of ℓ ∞ -norms. By contrast, unstructured approaches have a speed-up factor of about 7-8 with respect to the tree-structured methods.

Dictionaries of Natural Image Patches

This experiment studies whether a hierarchical structure can help dictionaries for denoising natural image patches, and in which noise regime the potential gain is significant. We aim at reconstructing corrupted patches from a test set, after having learned dictionaries on a training set of non-corrupted patches. Though not typical in machine 4. Proximal Methods for Structured Sparsity-Inducing Norms learning, this setting is reasonable in the context of images, where lots of non-corrupted patches are easily available.21 noise 50 % 60 % 70 % 80 % 90 % flat 19.3 ± 0.1 26.8 ± 0.1 36.7 ± 0.1 50.6 ± 0.0 72.1 ± 0.0 tree 18.6 ± 0.1 25.7 ± 0.1 35.0 ± 0.1 48.0 ± 0.0 65.9 ± 0.3 Table 4.2: Quantitative results of the reconstruction task on natural image patches. First row: percentage of missing pixels. Second and third rows: mean square error multiplied by 100, respectively for classical sparse coding, and tree-structured sparse coding. We extracted 100 000 patches of size m = 8×8 pixels from the Berkeley segmentation database of natural images [START_REF] Martin | A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[END_REF], which contains a high variability of scenes. We then split this dataset into a training set X tr , a validation set X val , and a test set X te , respectively of size 50 000, 25 000, and 25 000 patches. All the patches are centered and normalized to have unit ℓ 2 -norm.

For the first experiment, the dictionary D is learned on X tr using the formulation of Eq. (4.10), with µ = 0 for D µ as defined in Eq. (4.11). The validation and test sets are corrupted by removing a certain percentage of pixels, the task being to reconstruct the missing pixels from the known pixels. We thus introduce for each element x of the validation/test set, a vector x, equal to x for the known pixel values and 0 otherwise. Similarly, we define D as the matrix equal to D, except for the rows corresponding to missing pixel values, which are set to 0. By decomposing x on D, we obtain a sparse code α, and the estimate of the reconstructed patch is defined as Dα. Note that this procedure assumes that we know which pixel is missing and which is not for every element x.

Experiments

The parameters of the experiment are the regularization parameter λ tr used during the training step, the regularization parameter λ te used during the validation/test step, and the structure of the tree. For every reported result, these parameters were selected by taking the ones offering the best performance on the validation set, before reporting any result from the test set. The values for the regularization parameters λ tr , λ te were selected on a logarithmic scale {2 -10 , 2 -9 , . . . , 2 2 }, and then further refined on a finer logarithmic scale with multiplicative increments of 2 -1/4 . For simplicity, we chose arbitrarily to use the ℓ ∞ -norm in the structured norm Ω, with all the weights equal to one. We tested 21 balanced tree structures of depth 3 and 4, with different branching factors p 1 , p 2 , . . . , p d-1 , where d is the depth of the tree and p k , k ∈ {1, . . . , d -1} is the number of children for the nodes at depth k. The branching factors tested for the trees of depth 3 where p 1 ∈ {5, 10, 20, 40, 60, 80, 100}, p 2 ∈ {2, 3}, and for trees of depth 4, p 1 ∈ {5, 10, 20, 40}, p 2 ∈ {2, 3} and p 3 = 2, giving 21 possible structures associated with dictionaries with at most 401 elements. For each tree structure, we evaluated the performance obtained with the tree-structured dictionary along with a non-structured dictionary containing the same number of elements. These experiments were carried out four times, each time with a different initialization, and with a different noise realization.

Quantitative results are reported in Table 4.2. For all fractions of missing pixels considered, the tree-structured dictionary outperforms the "unstructured one", and the most significant improvement is obtained in the noisiest setting. Note that having more dictionary elements is worthwhile when using the tree structure. To study the influence of the chosen structure, we report in Figure 4.7 the results obtained with the 13 tested structures of depth 3, along with those obtained with unstructured dictionaries containing the same number of elements, when 90% of the pixels are missing. For each dictionary size, the tree-structured dictionary significantly outperforms the unstructured one. An example of a learned tree-structured dictionary is presented on Figure 4.9. Dictionary elements naturally organize in groups of patches, often with low frequencies near the root of the tree, and high frequencies near the leaves.

Text Documents

This last experimental section shows that our approach can also be applied to model text corpora. The goal of probabilistic topic models is to find a low-dimensional representation of a collection of documents, where the representation should provide a semantic description of the collection. Approaching the problem in a parametric Bayesian framework, latent Dirichlet allocation (LDA) [START_REF] Blei | Latent dirichlet allocation[END_REF] model documents, represented as vectors of word counts, as a mixture of a predefined number of latent topics that are distributions over a fixed vocabulary. LDA is fundamentally a matrix factorization problem: [START_REF] Buntine | Variational Extensions to EM and Multinomial PCA[END_REF] shows that LDA can be interpreted as a Dirichlet-multinomial counterpart of factor analysis. The number of topics is usually small compared to the size of the vocabulary (e.g., 100 against 10 000), so that the topic proportions of each document provide a compact representation of the corpus. For instance, these new features can be used to feed a classifier in a subsequent classification task. We similarly use our dictionary learning approach to find low-dimensional representations of text corpora. The dictionary is learned on 50, 000 patches of size 16 × 16 pixels.

Proximal Methods for Structured Sparsity-Inducing Norms

Suppose that the signals X = [x 1 , . . . , x n ] in R m×n are each the bag-of-word representation of each of n documents over a vocabulary of m words, the k-th component of x i standing for the frequency of the k-th word in the document i. If we further assume that the entries of D and A are nonnegative, and that the dictionary elements d j have unit ℓ 1 -norm, the decomposition (D, A) can be interpreted as the parameters of a topic-mixture model. The regularization Ω induces the organization of these topics on a tree, so that, if a document involves a certain topic, then all ancestral topics in the tree are also present in the topic decomposition. Since the hierarchy is shared by all documents, the topics at the top of the tree participate in every decomposition, and should therefore gather the lexicon which is common to all documents. Conversely, the deeper the topics in the tree, the more specific they should be. An extension of LDA to model topic hierarchies was proposed by [START_REF] Blei | The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies[END_REF], who introduced a non-parametric Bayesian prior over trees of topics and modelled documents as convex combinations of topics selected along a path in the hierarchy. We plan to compare our approach with this model in future work.

Visualization of NIPS proceedings

We qualitatively illustrate our approach on the NIPS proceedings from 1988 through 1999 [START_REF] Griffiths | Finding scientific topics[END_REF]. After removing words appearing fewer than 10 times, the dataset is composed of 1714 articles, with a vocabulary of 8274 words. As explained above, we consider D + 1 and take A to be R p×n + . Figure 4.10 displays an example of a learned dictionary with 13 topics, obtained by using the ℓ ∞ -norm in Ω and selecting manually λ = 2 -15 . As expected and similarly to [START_REF] Blei | The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies[END_REF], we capture the stopwords at the root of the tree, and topics reflecting the different subdomains of the conference such as neurosciences, optimization or learning theory.

Posting classification

We now consider a binary classification task of n postings from the 20 Newsgroups data set. 22 We learn to discriminate between the postings from the two newsgroups alt.atheism and talk.religion.misc, following the setting of [START_REF] Lacoste-Julien | DiscLDA: Discriminative learning for dimensionality reduction and classification[END_REF] and [START_REF] Zhu | MedLDA: maximum margin supervised topic models for regression and classification[END_REF]. After removing words appearing fewer than 10 times and standard stopwords, these postings form a data set of 1425 documents over a vocabulary of 13312 words. We compare different dimensionality reduction techniques that we use to feed a linear SVM classifier, i.e., we consider (i) LDA, with the code from [START_REF] Blei | Latent dirichlet allocation[END_REF], (ii) principal component analysis (PCA), (iii) nonnegative matrix factorization (NMF), (iv) standard sparse dictionary learning (denoted by SpDL) and (v) our sparse hierarchical approach (denoted by SpHDL). Both SpDL and SpHDL are optimized over D + 1 and A = R p×n + , with the weights ω g equal to 1. We proceed as follows: given a random split into a training/test set of 1 000/425 postings, and given a number of topics p (also the number of components for PCA, NMF, SpDL and 4.6. Discussion SpHDL), we train an SVM classifier based on the low-dimensional representation of the postings. This is performed on a training set of 1 000 postings, where the parameters, λ ∈ {2 -26 , . . . , 2 -5 } and/or C svm ∈ {4 -3 , . . . , 4 1 } are selected by 5-fold cross-validation. We report in Figure 4.11 the average classification scores on the test set of 425 postings, based on 10 random splits, for different number of topics. Unlike the experiment on image patches, we consider only complete binary trees with depths in {1, . . . , 5}. The results from Figure 4.11 show that SpDL and SpHDL perform better than the other dimensionality reduction techniques on this task. As a baseline, the SVM classifier applied directly to the raw data (the 13312 words) obtains a score of 90.9±1.1, which is better than all the tested methods, but without dimensionality reduction (as already reported by [START_REF] Blei | Latent dirichlet allocation[END_REF]. Moreover, the error bars indicate that, though nonconvex, SpDL and SpHDL do not seem to suffer much from instability issues. Even if SpDL and SpHDL perform similarly, SpHDL has the advantage to provide a more interpretable topic mixture in terms of hierarchy, which standard unstructured sparse coding does not.

Discussion

We have applied this approach in various settings, with fixed/learned dictionaries, and based on different types of data, namely, natural images and text documents. A line of research to pursue is to develop other optimization tools for structured norms with general overlapping groups. For instance, Mairal et al. (2010b) have used network flow optimization techniques for that purpose, and Bach (2010a) submodular function optimization. This framework can also be used in the context of hierarchical kernel learning (Bach, 2008a), where we believe that our method can be more efficient than existing ones. This work establishes a connection between dictionary learning and probabilistic topic models, which should prove fruitful as the two lines of work have focused on different aspects of the same unsupervised learning problem: Our approach is based on convex optimization tools, and provides experimentally more stable data representations. Moreover, it can be easily extended with the same tools to other types of structures corresponding to other norms (Jenatton et al., 2011a;[START_REF] Jacob | Group Lasso with overlaps and graph Lasso[END_REF]. It should be noted, however, that, unlike some Bayesian methods, dictionary learning by itself does not provide mechanisms for the automatic selection of model hyper-parameters (such as the dictionary size or the topology of the tree). An interesting common line of research to pursue could be the supervised design of dictionaries, which has been proved useful in the two frameworks (Mairal et al., 2009a;Bradley and Bagnell, 2009a;[START_REF] Blei | Supervised topic models[END_REF]. 4.7. Extension: General Overlapping Groups and ℓ 1 /ℓ ∞ -norms Without loss of generality, 23 we assume from now on that the scalars u j are all non-negative, and we constrain the entries of ξ to be non-negative. Such a formulation introduces p|G| dual variables which can be much greater than p, the number of primal variables, but it removes the issue of overlapping regularization. We now associate a graph with problem (4.13), on which the variables ξ g j , for g in G and j in g, can be interpreted as measuring the components of a flow.

Graph Model

Let G be a directed graph G = (V, E, s, t), where V is a set of vertices, E ⊆ V × V a set of arcs, s a source, and t a sink. Let c : E → R + and c ′ : E → R + ∪ {+∞} be two functions on the arcs, where c is a cost function and c ′ is a non-negative capacity function. As done classically in the network flow literature [START_REF] Ahuja | Network Flows[END_REF][START_REF] Bertsekas | Linear network optimization: algorithms and codes[END_REF], we define a flow as a non-negative function on arcs that satisfies capacity constraints on all arcs (the value of the flow on an arc is less than or equal to the arc capacity) and conservation constraints on all vertices (the sum of incoming flows at a vertex is equal to the sum of outgoing flows) except for the source and the sink. We now introduce the canonical graph G associated with our optimization problem:

Definition 3 (Canonical Graph)
Let G ⊆ 2 1;p be a set of groups, and (η g ) g∈G be positive weights. The canonical graph G = (V, E, s, t) is the unique graph defined as follows:

1. V = V u ∪ V gr , where V u is a vertex set of size p, one vertex being associated to each index j in 1; p , and V gr is a vertex set of size |G|, one vertex per group g in G. We thus have |V | = |G| + p. For simplicity, we identify groups g in G and indices j in 1; p with vertices of the graph, such that one can from now on refer to "vertex j" or "vertex g".

2. For every group g in G, E contains an arc (s, g). These arcs have capacity λη g and zero cost.

3. For every group g in G, and every index j in g, E contains an arc (g, j) with zero cost and infinite capacity. We denote by ξ g j the flow on this arc. 4. For every index j in 1; p , E contains an arc (j, t) with infinite capacity and a cost 1 2 (u jξ j ) 2 , where ξ j is the flow on (j, t). Examples of canonical graphs are given in Figures 4.12a-(c) for three simple group structures. The flows ξ g j associated with G can now be identified with the variables of problem (4.13). Since we have assumed the entries of u to be non-negative, we can now 23. Let ξ ⋆ denote a solution of Eq. (4.13). Optimality conditions of Eq. (4.13) derived in Jenatton et al. (2010aJenatton et al. ( , 2011c) ) show that for all j in 1; p , the signs of the non-zero coefficients ξ ⋆g j for g in G are the same as the signs of the entries uj. To solve Eq. ( 4.13), one can therefore flip the signs of the negative variables uj, then solve the modified dual formulation (with non-negative variables), which gives the magnitude of the entries ξ ⋆g j (the signs of these being known). Indeed,

Proximal Methods for

• the only arcs with a cost are those leading to the sink, which have the form (j, t), where j is the index of a variable in 1; p . The sum of these costs is

p j=1 1 2 (u j -ξ j ) 2
, which is the objective function minimized in Eq. (4.14); • by flow conservation, we necessarily have ξ j = g∈G ξ g j in the canonical graph; • the only arcs with a capacity constraints are those coming out of the source, which have the form (s, g), where g is a group in G. By flow conservation, the flow on an arc (s, g) is j∈g ξ g j which should be less than λη g by capacity constraints; • all other arcs have the form (g, j), where g is in G and j is in g. Thus, ξ g j = 0 for j / ∈ g. Therefore we have shown that finding a flow minimizing the sum of the costs on such a graph is equivalent to solving problem (4.13). When some groups are included in others, the canonical graph can be simplified to yield a graph with a smaller number of edges. Specifically, if h and g are groups with h ⊂ g, the edges (g, j) for j ∈ h carrying a flow ξ g j can be removed and replaced by a single edge (g, h) of infinite capacity and zero cost, carrying the flow j∈h ξ g j . This simplification is illustrated in Figure 4.12d, with a graph equivalent to the one of Figure 4.12c. This does not change the optimal value of ξ ⋆ , which is the quantity of interest for computing the optimal primal variable w ⋆ . These simplifications are useful in practice, since they reduce the number of edges in the graph and improve the speed of our algorithms.

Computation of the Proximal Operator

Quadratic min-cost flow problems have been well studied in the operations research literature [START_REF] Hochbaum | About strongly polynomial time algorithms for quadratic optimization over submodular constraints[END_REF]. One of the simplest cases, where G contains a single group as in Figure 4.12a, is solved by an orthogonal projection on the ℓ 1 -ball of radius λη g . It has been shown, both in machine learning [START_REF] Duchi | Efficient projections onto the ℓ 1 -ball for learning in high dimensions[END_REF] and operations research [START_REF] Hochbaum | About strongly polynomial time algorithms for quadratic optimization over submodular constraints[END_REF][START_REF] Brucker | An O(n) algorithm for quadratic knapsack problems[END_REF], that such a projection can be computed in O(p) operations. When the group structure is a tree as in Figure 4.12d, strategies developed in the two communities are also similar (Jenatton et al., 2010a;[START_REF] Hochbaum | About strongly polynomial time algorithms for quadratic optimization over submodular constraints[END_REF], 24 and solve the problem in O(pd) operations, where d is the depth of the tree.

The general case of overlapping groups is more difficult. [START_REF] Hochbaum | About strongly polynomial time algorithms for quadratic optimization over submodular constraints[END_REF] have shown that quadratic min-cost flow problems can be reduced to a specific parametric max-flow problem, for which an efficient algorithm exists [START_REF] Gallo | A fast parametric maximum flow algorithm and applications[END_REF]. 25 While 4.7. Extension: General Overlapping Groups and ℓ 1 /ℓ ∞ -norms

s g ξ g 1 +ξ g 2 +ξ g 3 ≤ λη g u 2 ξ g 2 u 1 ξ g 1 u 3 ξ g 3 t ξ 1 , c 1 ξ 2 , c 2 ξ 3 , c 3 (a) G = {g = {1, 2, 3}}. s g ξ g 1 +ξ g 2 ≤ λη g h ξ h 2 +ξ h 3 ≤ λη h u 2 ξ h 2 ξ g 2 u 1 ξ g 1 u 3 ξ h 3 t ξ 1 , c 1 ξ 2 , c 2 ξ 3 , c 3 (b) G = {g = {1, 2}, h = {2, 3}}. s g ξ g 1 +ξ g 2 +ξ g 3 ≤ λη g h ξ h 2 +ξ h 3 ≤ λη h u 2 ξ h 2 ξ g 2 u 1 ξ g 1 u 3 ξ g 3 ξ h 3 t ξ 1 , c 1 ξ 2 , c 2 ξ 3 , c 3 (c) G = {g = {1, 2, 3}, h = {2, 3}}. s g ξ g 1 +ξ g 2 +ξ g 3 ≤ λη g h ξ h 2 +ξ h 3 ≤ λη h ξ g 2 +ξ g 3 u 2 ξ g 2 +ξ h 2 u 1 ξ g 1 u 3 ξ g 3 +ξ h 3 t ξ 1 , c 1 ξ 2 , c 2 ξ 3 , c 3 (d) G = {g = {1} ∪ h, h = {2, 3}}.
Figure 4.12: Graph representation of simple proximal problems with different group structures G. The three indices 1, 2, 3 are represented as grey squares, and the groups g, h in G as red discs. The source is linked to every group g, h with respective maximum capacity λη g , λη h and zero cost. Each variable u j is linked to the sink t, with an infinite capacity, and with a cost c j 1 2 (u jξ j ) 2 . All other arcs in the graph have zero cost and infinite capacity. They represent inclusion relations in-between groups, and between groups and variables. The graphs (c) and (d) correspond to a special case of tree-structured hierarchy in the sense of Jenatton et al. (2010a). Their min-cost flow problems are equivalent.

Proximal Methods for Structured Sparsity-Inducing Norms

this generic approach could be used to solve Eq. (4.13), we propose to use Algorithm 9 that also exploits the fact that our graphs have non-zero costs only on edges leading to the sink. As further shown in [START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF] through speed benchmarks, it it has a significantly better performance in practice. This algorithm clearly shares some similarities with existing approaches in network flow optimization such as the simplified version of [START_REF] Gallo | A fast parametric maximum flow algorithm and applications[END_REF] presented by [START_REF] Babenko | Experimental evaluation of a parametric flow algorithm[END_REF] that uses a divide and conquer strategy. Moreover, an equivalent algorithm exists for minimizing convex functions over polymatroid sets [START_REF] Groenevelt | Two algorithms for maximizing a separable concave function over a polymatroid feasible region[END_REF]. This equivalence, a priori non trivial, is uncovered through a representation of structured sparsity-inducing norms via submodular functions, which was recently proposed by Bach (2010a).

Algorithm 9 Computation of the proximal operator for overlapping groups.

1: Input: u ∈ R p , a set of groups G, positive weights (η g ) g∈G , and λ (regularization parameter).

2: Build the initial graph G 0 = (V 0 , E 0 , s, t) as explained in Section 4.7.2.

3: Compute the optimal flow: ξ ← computeFlow(V 0 , E 0 ). 4: Return: w = uξ (optimal solution of the proximal problem).

Function computeFlow(V = V u ∪ V gr , E) 1: Projection step: γ ← arg min γ j∈Vu 1 2 (u j -γ j ) 2 s.t. j∈Vu γ j ≤ λ g∈Vgr η g . 2:
For all nodes j in V u , set γ j to be the capacity of the arc (j, t). 3: Max-flow step: Update (ξ j ) j∈Vu by computing a max-flow on the graph (V, E, s, t).

4: if ∃ j ∈ V u s.t. ξ j = γ j then 5:
Denote by (s, V + ) and (V -, t) the two disjoint subsets of (V, s, t) separated by the minimum (s, t)-cut of the graph, and remove the arcs between V + and V -. Call E + and E -the two remaining disjoint subsets of E corresponding to V + and V -.

6:

(ξ j ) j∈V + u ← computeFlow(V + , E + ).

7:

(ξ j ) j∈V - u ← computeFlow(V -, E -). 8: end if 9: Return: (ξ j ) j∈Vu .
The intuition behind our algorithm, computeFlow (see Algorithm 9), is the following: since ξ = g∈G ξ g is the only value of interest to compute the solution of the proximal operator w = uξ, the first step looks for a candidate value γ for ξ by solving the following relaxed version of problem (4.14):

arg min γ∈R p j∈Vu 1 2 (u j -γ j ) 2 s.t. j∈Vu γ j ≤ λ g∈Vgr η g . (4.15)
The cost function here is the same as in problem (4.14), but the constraints are weaker: Any feasible point of problem (4.14) is also feasible for problem (4.15). This problem can be solved in linear time [START_REF] Brucker | An O(n) algorithm for quadratic knapsack problems[END_REF]. Its solution, which we denote γ for simplicity, provides the lower bound uγ 2 2 /2 for the optimal cost of problem (4.14).

Extension: General Overlapping Groups and ℓ 1 /ℓ ∞ -norms

The second step tries to construct a feasible flow (ξ, ξ), satisfying additional capacity constraints equal to γ j on arc (j, t), and whose cost matches this lower bound; this latter problem can be cast as a max-flow problem [START_REF] Goldberg | A new approach to the maximum-flow problem[END_REF]. If such a flow exists, the algorithm returns ξ = γ, the cost of the flow reaches the lower bound, and is therefore optimal. If such a flow does not exist, we have ξ = γ, the lower bound is not achievable, and we build a minimum (s, t)-cut of the graph [START_REF] Ford | Maximal flow through a network[END_REF] defining two disjoints sets of nodes V + and V -; V + is the part of the graph that could potentially have received more flow from the source (the arcs between s and V + are not saturated), whereas V -could not (all arcs linking s to V -are saturated). At this point, it is possible to show that the value of the optimal min-cost flow on all arcs between V + and V -is necessary zero. Thus, removing them yields an equivalent optimization problem, which can be decomposed into two independent problems of smaller sizes and solved recursively by the calls to computeFlow(V + , E + ) and computeFlow(V -, E -). A formal proof of correctness of Algorithm 9 and further details can be found in [START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF].

Experiments

In this section, we present two experiments demonstrating the applicability and the benefits of our methods for solving large-scale sparse and structured regularized problems (additional applications can be found in [START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF].

CUR-like Matrix Factorization

In this experiment, we show how our tools can be used to perform the so-called CUR matrix decomposition [START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF]. It consists of a low-rank approximation of a data matrix X in R n×p in the form of a product of three matricesthat is, X ≈ CUR. The particularity of the CUR decomposition lies in the fact that the matrices C ∈ R n×c and R ∈ R r×p are constrained to be respectively a subset of c columns and r rows of the original matrix X. The third matrix U ∈ R c×r is then given by C + XR + , where A + denotes a Moore-Penrose generalized inverse of the matrix A [START_REF] Horn | Matrix analysis[END_REF]. Such a matrix factorization is particularly appealing when the interpretability of the results matters [START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF]. For instance, when studying gene-expression datasets, it is easier to gain insight from the selection of actual patients and genes, rather than from linear combinations of them.

In [START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF], CUR decompositions are computed by a sampling procedure based on the singular value decomposition of X. In a recent work, [START_REF] Bien | CUR from a Sparse Optimization Viewpoint[END_REF] have shown that partial CUR decompositions, i.e., the selection of either rows or columns of X, can be obtained by solving a convex program with a group-Lasso penalty. We propose to extend this approach to the simultaneous selection of both rows and columns of X, with the following convex problem:

min W∈R p×n 1 2 X -XWX 2 F + λ row n i=1 W i ∞ + λ col p j=1 W j ∞ . (4.16)

Proximal Methods for Structured Sparsity-Inducing Norms

In this formulation, the two sparsity-inducing penalties controlled by the parameters λ row and λ col set to zero some entire rows and columns of the solutions of problem (4.16). Now, let us denote by W I J in R |I|×|J| the submatrix of W reduced to its nonzero rows and columns, respectively indexed by I ⊆ {1, . . . , p} and J ⊆ {1, . . . , n}. We can then readily identify the three components of the CUR decomposition of X, namely

XWX = CW I J R ≈ X.
Problem (4.16) has a smooth convex data-fitting term and brings into play a sparsityinducing norm with overlapping groups of variables (the rows and the columns of W). As a result, it can be handled with the optimization tools introduced in this section. We now compare the performance of the sampling procedure from [START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF] with our proposed sparsity-based approach. To this end, we consider the four geneexpression datasets 9_Tumors, Brain_Tumors1, Leukemia1 and SRBCT, with respective dimensions (n, p) ∈ {(60, 5727), (90, 5921), (72, 5328), (83, 2309)}. 26 In the sequel, the matrix X is normalized to have unit Frobenius-norm while each of its columns is centered.

To begin with, we run our approach27 over a grid of values for λ row and λ col in order to obtain solutions with different sparsity levels, i.e., ranging from |I| = p and |J| = n down to |I| = |J| = 0. For each pair of values [|I|, |J|], we then apply the sampling procedure from [START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF]. Finally, the variance explained by the CUR decompositions is reported in Figure 4.13 for both methods. Since the sampling approach involves some randomness, we show the average and standard deviation of the results based on five initializations. The conclusions we can draw from the experiments match the ones already reported in [START_REF] Bien | CUR from a Sparse Optimization Viewpoint[END_REF] for the partial CUR decomposition. We can indeed see that both schemes perform similarly. However, our approach has the advantage not to be randomized, which can be less disconcerting in the practical perspective of analyzing a single run of the algorithm. It is finally worth being mentioned that the convex approach we develop here is flexible and can be extended in different ways. For instance, we may imagine add further low-rank/sparsity constraints on W thanks to sparsity-promoting convex regularizations.

Background Subtraction

Following [START_REF] Cevher | Sparse signal recovery using markov random fields[END_REF]; [START_REF] Huang | Learning with structured sparsity[END_REF], we consider a background subtraction task. Given a sequence of frames from a fixed camera, we try to segment out foreground objects in a new image. If we denote by y ∈ R n this image composed of n pixels, we model y as a sparse linear combination of p other images X ∈ R n×p , plus an error term e in R n , i.e., y ≈ Xw + e for some sparse vector w in R p . This approach is reminiscent of [START_REF] Wright | Robust face recognition via sparse representation[END_REF] .13: Explained variance of the CUR decompositions obtained for our sparsitybased approach and the sampling scheme from [START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF]. For the latter, we report the average and standard deviation of the results based on five initializations. From left to right and top to bottom, the curves correspond to the datasets 9_Tumors, Brain_Tumors1, Leukemia1 and SRBCT.

present in both y and X, while e contains specific, or foreground, objects in y. The resulting optimization problem is given by

min w∈R p ,e∈R n 1 2 y-Xw-e 2 2 + λ 1 w 1 + λ 2 { e 1 + Ω(e)}, with λ 1 , λ 2 ≥ 0. (4.17)
In this formulation, the only ℓ 1 -norm penalty does not take into account the fact that neighboring pixels in y are likely to share the same label (background or foreground), which may lead to scattered pieces of foreground and background regions (Figure 4.14).

We therefore put an additional structured regularization term Ω on e, where the groups in G are all the overlapping 3×3-squares on the image. This optimization problem can be viewed as an instance of least-squares problem regularized by Ω, with the particular design matrix [X, I] in R n× (p+n) , defined as the columnwise concatenation of X and the identity matrix. As a result, we could directly apply the same procedure as the one used in the other experiments. Instead, we further exploit the specific structure of problem (4.17): Notice that for a fixed vector e, the 4. Proximal Methods for Structured Sparsity-Inducing Norms optimization with respect to w is a standard Lasso problem (with the vector of observations ye), 28 while for w fixed, we simply have a proximal problem associated to the sum of Ω and the ℓ 1 -norm. Alternating between these two simple and computationally inexpensive steps, i.e., optimizing with respect to one variable while keeping the other one fixed, is guaranteed to converge to a solution of (4.17). 29 In our simulations, this alternating scheme has led to a significant speed-up compared to the general procedure.

A dataset with hand-segmented images is used to illustrate the effect of Ω. 30 For simplicity, we use a single regularization parameter, i.e., λ 1 = λ 2 , chosen to maximize the number of pixels matching the ground truth. We consider p = 200 images with n = 57600 pixels (i.e., a resolution of 120 × 160, times 3 for the RGB channels). As shown in Figure 4.14, adding Ω improves the background subtraction results for the two tested images, by removing the scattered artifacts due to the lack of structural constraints of the ℓ 1 -norm, which encodes neither spatial nor color consistency.

28. Since successive frames might not change much, the columns of X exhibit strong correlations. As a result, we use the LARS algorithm [START_REF] Efron | Least angle regression[END_REF] whose complexity is independent of the level of correlation in X.

29. More precisely, the convergence is guaranteed since the non-smooth part in (4.17) is separable with respect to w and e [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF]. The result from [START_REF] Bertsekas | Nonlinear programming[END_REF] may also be applied here, after reformulating (4.17) as a smooth convex problem under separable conic constraints.

30. http://research.microsoft.com/en-us/um/people/jckrumm/wallflower/testimages.htm 5

Application of Structured Sparsity to Neuroimaging

Multi-scale Mining of fMRI Data with Hierarchical Structured Sparsity

Abstract of the first section of the chapter: Inverse inference, or "brain reading", is a recent paradigm for analyzing functional magnetic resonance imaging (fMRI) data, based on pattern recognition and statistical learning. By predicting some cognitive variables related to brain activation maps, this approach aims at decoding brain activity. Inverse inference takes into account the multivariate information between voxels and is currently the only way to assess how precisely some cognitive information is encoded by the activity of neural populations within the whole brain. However, it relies on a prediction function that is plagued by the curse of dimensionality, since there are far more features than samples, i.e., more voxels than fMRI volumes.

To address this problem, different methods have been proposed, such as, among others, univariate feature selection, feature agglomeration and regularization techniques. In this paper, we consider a sparse hierarchical structured regularization. Specifically, the penalization we use is constructed from a tree that is obtained by spatially-constrained agglomerative clustering. This approach encodes the spatial structure of the data at different scales into the regularization, which makes the overall prediction procedure more robust to inter-subject variability. The regularization used induces the selection of spatially coherent predictive brain regions simultaneously at different scales. We test our algorithm on real data acquired to study the mental representation of objects, and we show that the proposed algorithm non only delineates meaningful brain regions but yields as well better prediction accuracy than reference methods.

The material of this first section is based on the following papers: R.

Introduction

Functional magnetic resonance imaging (or fMRI) is a widely used functional neuroimaging modality. Modeling and statistical analysis of fMRI data are commonly done through a linear model, called general linear model (GLM) in the community, that incorporates information about the different experimental conditions and the dynamics of the hemodynamic response in the design matrix. The experimental conditions are typically modelled by the type of stimulus presented, e.g., visual and auditory stimulation, which are included as regressors in the design matrix. The resulting model parameters-one coefficient per voxel and regressor-are known as activation maps. They represent the local influence of the different experimental conditions on fMRI signals at the level of individual voxels. The most commonly used approach to analyze these activation maps is called classical inference. It relies on mass-univariate statistical tests (one for each voxel), and yields so-called statistical parametric maps (SPMs) [START_REF] Friston | Statistical parametric maps in functional imaging: A general linear approach[END_REF]. Such maps are useful for functional brain mapping, but classical inference has some limitations: it suffers from multiple comparisons issues and it is oblivious of the multivariate structure of fMRI data. Such data exhibit natural correlations between neighboring voxels forming clusters with different sizes and shapes, and also between distant but functionally connected brain regions.

To address these limitations, an approach called inverse inference (or "brain-reading") [START_REF] Dehaene | Inferring behavior from functional brain images[END_REF][START_REF] Cox | Functional magnetic resonance imaging (fMRI) "brain reading": detecting and classifying distributed patterns of fMRI activity in human visual cortex[END_REF] was recently proposed. Inverse inference relies on pattern recognition tools and statistical learning methods to explore fMRI data. Based on a set of activation maps, inverse inference estimates a function that can then be used to predict a target (typically, a variable representing a perceptual, cognitive or behavioral parameter) for a new set of images. The challenge is to capture the correlation structure present in the data in order to improve the performance of the mapping learnt, which is measured through the resulting prediction accuracy. Many standard statistical learning approaches have been used to construct prediction functions, among them kernel machines (SVM, RVM) [START_REF] Schölkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF] or discriminant analysis (LDA, QDA) [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition[END_REF]. For the application considered in this chapter, earlier performance results [START_REF] Cox | Functional magnetic resonance imaging (fMRI) "brain reading": detecting and classifying distributed patterns of fMRI activity in human visual cortex[END_REF][START_REF] Laconte | Support vector machines for temporal classification of block design fMRI data[END_REF] indicate that we can restrict ourselves to mappings that are linear functions of the data.

Throughout the chapter, we shall consider a training set composed of n pairs (x, y) ∈ R p × Y, where x denotes a p-dimensional fMRI signal (p voxels) and y stands for the target we try to predict. In the experiments we carry out in Section 5.2.4, we will encounter both the regression and the multi-class classification settings, where Y denotes respectively the set of real numbers and a finite set of integers. In this chapter, we aim at learning a weight vector w ∈ R p and an intercept b ∈ R such that the prediction of y can be based on the value of w ⊤ x + b. This is the case for the linear regression and logistic regression models that we use in Section 5.2.4. It is useful to rewrite these quantities in matrix form; more precisely, we denote by X ∈ R n×p the design matrix assembled from n fMRI data points and by y ∈ R n the corresponding n targets. In other words, each row of X is a p-dimensional sample, i.e., an activation map of p voxels related to one stimulus presentation.

Multi-scale Mining of fMRI Data with Hierarchical Structured Sparsity

Learning the parameters (w, b) remains challenging since the number of features (10 4 to 10 5 voxels) exceeds by far the number of samples (a few hundreds of images). The prediction function is therefore prone to the phenomenon of overfitting in which the learning set is predicted precisely whereas the algorithm provides very inaccurate predictions on new samples (the test set). To address this issue, dimensionality reduction attempts to find a low dimensional subspace that concentrates as much of the predictive power of the original set as possible for the problem at hand.

Feature selection is a natural approach to perform dimensionality reduction in fMRI, since reducing the number of voxels potentially allows to identify a predictive region of the brain. This corresponds to discarding some columns of X. This feature selection can be univariate, e.g., analysis of variance (ANOVA) [START_REF] Lehmann | Testing statistical hypotheses[END_REF], or multivariate. While univariate methods ignore joint information between features, multivariate approaches are more adapted to inverse inference since they extract predictive patterns from the data as a whole. However, due to the huge number of possible patterns, these approaches suffer from combinatorial explosion, and some costly suboptimal heuristics (e.g., recursive feature elimination [START_REF] Guyon | Gene selection for cancer classification using support vector machines[END_REF][START_REF] Martino | Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns[END_REF]) can be used. That is why ANOVA is usually preferred in fMRI. Alternatively, two more adapted solutions have been proposed: regularization and feature agglomeration.

Regularization is a way to encode a priori knowledge about the weight vector w. Possible regularizers can promote for example spatial smoothness or sparsity which is a natural assumption for fMRI data. Indeed, only a few brain regions are assumed to be significantly activated during a cognitive task. Previous contributions on fMRI-based inverse inference include [START_REF] Carroll | Prediction and interpretation of distributed neural activity with sparse models[END_REF]; [START_REF] Rissman | Detecting individual memories through the neural decoding of memory states and past experience[END_REF]; [START_REF] Ryali | Sparse logistic regression for wholebrain classification of fMRI data[END_REF]; [START_REF] Yamashita | Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns[END_REF]. They can be presented through the following minimization problem: min where λΩ(w) is the regularization term, typically a non-Euclidean norm, and the fit to the data is measured through a convex loss function (w, b) → L(y, X, w, b) ∈ R + . The choice of the loss function will be made more specific and formal in the next sections. The coefficient of regularization λ balances the loss and the penalization term. In this notation, a common regularization term in inverse inference is the so-called Elastic net [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF][START_REF] Grosenick | Interpretable classifiers for FMRI improve prediction of purchases[END_REF], which is a combined ℓ 1 and ℓ 2 penalization:

λΩ(w) = λ 1 w 1 + λ 2 w 2 2 = p j=1 λ 1 |w j | + λ 2 w 2 j .
(5.2)

For the square loss, when setting λ 1 to 0, the model is called ridge regression, while when λ 2 = 0 it is known as Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] or basis pursuit [START_REF] Chen | Atomic decomposition by basis pursuit[END_REF]. The essential shortcoming of the Elastic net is that it does not take into account the spatial structure of the data, which is crucial in this context [START_REF] Michel | Total variation regularization for fMRI-based prediction of behaviour[END_REF]. Indeed, due to the intrinsic smoothing of the complex metabolic pathway underlying the difference of blood oxygenation measured with fMRI [START_REF] Ugurbil | How accurate is magnetic resonance imaging of brain function[END_REF], statistical learning approaches should be informed by the 3D grid structure of the data.

Application of Structured Sparsity to Neuroimaging

In order to achieve dimensionality reduction, while taking into account the spatial structure of the data, one can resort to feature agglomeration. It constructs new features, called parcels, by averaging neighboring voxels exhibiting similar activations. The advantage of agglomeration is that no information is discarded a priori and that it is reasonable to hope that averaging might reduce noise. Although, this approach has been successfully used in previous work for brain mapping [START_REF] Flandin | Improved detection sensitivity in functional MRI data using a brain parcelling technique[END_REF][START_REF] Thirion | Dealing with the shortcomings of spatial normalization: Multi-subject parcellation of fMRI datasets[END_REF], it often does not consider the supervised information (i.e., the target y) while constructing the parcels. A recent approach has been proposed to address this issue, using a supervised greedy top-down exploration of a tree obtained by hierarchical clustering [START_REF] Michel | A supervised clustering approach for extracting predictive information from brain activation images[END_REF]. This greedy approach has proven to be effective especially for inter-subject analyses, i.e., when the training and the evaluation sets are related to different subjects. In this context, methods need to be robust to intrinsic spatial variations that exist across subjects: despite being co-registered into a common space, some variability remains between subjects, which implies that there is no perfect voxelto-voxel correspondence between volumes. As a result, the performances of traditional voxel-based methods are strongly affected. Therefore, averaging in the form of parcels is a good way to cope with inter-subject variability. This greedy approach is nonetheless suboptimal, as it explores only a subpart of the whole tree.

Based on these considerations, we propose to integrate the multi-scale spatial structure of the data within the regularization term Ω, while preserving convexity in the optimization. This notably guarantees global optimality and stability of the obtained solutions. To this end, we design a sparsity-inducing penalty that is directly built from the hierarchical structure of the spatial model obtained by Ward's algorithm [START_REF] Ward | Hierarchical grouping to optimize an objective function[END_REF]. Such a penalty has already been successfully applied in several contexts, e.g., in bioinformatics, to exploit the tree structure of gene networks for multi-task regression [START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF], and also for topic models and image inpainting (Jenatton et al., 2010a).

We summarize here the contributions of our chapter:

• We explain how the multi-scale spatial structure of fMRI data can be taken into account in the context of inverse inference through the combination of a spatially constrained hierarchical clustering procedure and a sparse hierarchical regularization. • We provide a convex formulation of the problem and propose an efficient optimization procedure. • We conduct a experimental comparison of several algorithms and formulations on fMRI data and illustrate the ability of the proposed method to localize in space and in scale some brain regions involved in the processing of visual stimuli.

The rest of the chapter is organized as follows: we first present the concept of structured sparsity-inducing regularization and then describe the different regression/classification formulations we are interested in. After exposing how we handle the resulting large-scale convex optimization problems thanks to proximal methods, we validate our approach on both a synthetic setting and a real dataset.

Combining agglomerative clustering with sparsity inducing regularizers

Hierarchical clustering allows to construct a tree-structured hierarchy of features on top of the original voxels features. Moreover, the underlying voxels corresponding to each of these features correspond to localized spatial patterns on the brain of the form we hope to retrieve [START_REF] Chklovskii | Maps in the brain: What can we learn from them?[END_REF]. Instead of selecting features in the tree greedily, we propose to cast the feature selection problem as supervised learning problem of the form (5.1). It is natural to require of the regularizer Ω that it should respect the tree structure of the hierarchy so as to induce the selection of localized patterns.

Constructing the sparsity-inducing norm

The structured sparsity-inducing term Ω is built from the result of the hierarchical clustering of the voxels. The latter yields a hierarchy of clusters represented as a tree T (or dendrogram) [START_REF] Johnson | Hierarchical clustering schemes[END_REF]. The root of the tree is the unique cluster that gathers all the voxels, while the leaves are the clusters with a single voxel. Among different hierarchical agglomerative clustering procedures, we use the variance-minimizing approach of Ward's algorithm [START_REF] Ward | Hierarchical grouping to optimize an objective function[END_REF], since it minimizes the loss of information at each step of clustering. In short, two clusters are merged if the resulting parcellation minimizes the sum of squared differences within all clusters (also known as inertia criterion).

In order to take into account the spatial information, we also add connectivity constraints in the hierarchical clustering algorithm, so that only neighboring clusters can be merged together. The resulting clusters are thus called parcels. Each node of the tree T either corresponds to a voxel if it is a leaf, or defines a parcel, as the union of its children's clusters of voxels (see Figure 5.1).

Figure 5.1: Example of a tree T when p = 5, with three voxels and two parcels. The parcel 2 is defined as the averaged intensity of the voxels {1, 2}, while the parcel 1 is obtained by averaging the parcel 2 and voxel 3. In red dashed lines are represented the five groups of variables that compose G. For instance, if the group containing the parcel 2 is set to zero, the voxels {1, 2} are also (and necessarily) zeroed out. Best seen in color.

We now consider the augmented space of variables (also known as features), formed by not only the voxels, but also by the parcels. This approximately doubles the number of features of the fMRI signals. In other words, p does not denote the number of voxels anymore, but instead, the total number of nodes of T .1 In the following, the level of activation of each parcel is (recursively) defined by the averaged intensity of the voxels it is composed of (i.e., local averages) [START_REF] Flandin | Improved detection sensitivity in functional MRI data using a brain parcelling technique[END_REF][START_REF] Thirion | Dealing with the shortcomings of spatial normalization: Multi-subject parcellation of fMRI datasets[END_REF]. This produces a multi-scale representation of the fMRI data that becomes increasingly invariant to spatial shifts of the encoding regions within the brain volume. More formally, if j is a node of T and P j stands for the set of voxels of the corresponding parcel (i.e., the set of leaves of the subtree rooted at node j), we consider the mean of the parcel that we denote by x P j . In this notation, the linear model we use is of the form

f w (x) = w ⊤ x = j∈T w j x P j = i∈V j∈A(i) w j |P j | x i ,
where A(i) is the set of ancestors of a node i in T (including itself), and V corresponds to the leaves of the tree. To lighten notations, in the remainder of the chapter, we will denote by X instead of X the matrix of features from the augmented space.

In the perspective of inter-subject validation, the augmented space of variables can be exploited in the following way: since the information of single voxels may be unreliable, the deeper the node in T , the more variable the corresponding parcel's intensity is likely to be across subjects. This property suggests that, while looking for sparse solutions of (5.1), we should preferentially select the variables near the root of T , before trying to access smaller parcels located further down in T .

Traditional sparsity-inducing penalties, e.g., the ℓ 1 -norm Ω(w) = p j=1 |w j |, yield sparsity at the level of single variables w j , disregarding potential structures-for instance, spatial-existing between larger subsets of variables. We leverage here the concept of structured sparsity where Ω penalizes some predefined subsets, or groups, of variables that reflect prior information about the problem at hand [START_REF] Baraniuk | Model-based compressive sensing[END_REF][START_REF] Huang | Learning with structured sparsity[END_REF]Jenatton et al., 2011a;[START_REF] Jacob | Group Lasso with overlaps and graph Lasso[END_REF]. In particular, we follow [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF] that first introduced hierarchical sparsity-inducing penalties. Given a node j of T , we denote by g j ⊆ {1, . . . , p} the set of indices that record all the descendants of j in T , including itself. In other words, g j contains the indices of the subtree rooted at j; see Figure 5.1. If we now denote by G the set of all g j , j ∈ {1, . . . , p}, that is, G {g 1 , . . . , g p }, we can define our hierarchical penalty as

Ω(w) g∈G w g 2 g∈G j∈g w 2 j 1/2 .
(5.3)

As shown in Jenatton et al. (2011a), Ω is a norm, and it promotes sparsity at the level of groups g ∈ G, in the sense that it acts as a ℓ 1 -norm on the vector ( w g 2 ) g∈G .

Regularizing by Ω therefore causes some w g 2 (and equivalently w g ) to be zeroed out for some g ∈ G. Moreover, since the groups g ∈ G represent rooted subtrees of T , this implies that if one node/parcel j ∈ g is set to zero by Ω, the same occurs for all its descendants [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]. To put it differently, if one parcel is selected, then all the ancestral parcels in T will also be selected. This property is in accordance with our concern of robustness with respect to voxel misalignments between subjects, since large parcels are considered before smaller ones.

The family of norms with the previous property is actually slightly larger and we consider throughout the chapter norms Ω of the form [START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF]: (5.4) where w g denotes either the ℓ 2 -norm w g 2 or the ℓ ∞ -norm w g ∞ max j∈g |w j | and (η g ) g∈G are (strictly) positive weights that can compensate for the fact that some features are overpenalized as a result of being included in a larger number of groups than others. In light of the results from Jenatton et al. (2010a), we will see in Section 5.1.4 that a large class of optimization problems regularized by Ω-as defined in (5.4)-can be solved efficiently.

Ω(w) g∈G η g w g ,

Supervised learning framework

In this section, we introduce the formulations we consider in our experiments. As further discussed in Section 5.2.4, the target y we try to predict corresponds to (discrete) sizes of objects, i.e., a one-dimensional ordered variable. It is therefore sensible to address this prediction task from both a regression and a classification viewpoint.

Regression

In this first setting, we naturally consider the square loss function, so that problem (5.1) can be reduced to

min w∈R p 1 2n y -Xw 2 2 + λΩ(w) with λ ≥ 0.
Note that in this case, we have omitted the intercept b since we can center the vector y and the columns of X instead.

Classification

We now look at our prediction task from a multi-class classification viewpoint. Specifically, we assume that Y is a finite set of integers {1, . . . , c}, c > 2, and consider both multi-class and "one-versus-all" strategies [START_REF] Rifkin | In defense of one-vs-all classification[END_REF]. We need to slightly extend the formulation (5.1): To this end, we introduce the weight matrix W [w 1 , . . . , w c ] ∈ R p×c , composed of c weight vectors, along with a vector of intercepts b ∈ R c .

A standard way of addressing multi-class classification problems consists in using a multi-logit model, also known as multinomial logistic regression (see, e.g., (Hastie et al.,129

Application of Structured Sparsity to Neuroimaging

2009) and references therein). In this case, class-conditional probabilities are modeled for each class by a softmax function and leads to min

W∈R p×c b∈R c 1 n n i=1 log c k=1 e x ⊤ i (w k -w y i )+b k -by i + λ c k=1 Ω(w k ) .
Whereas the regularization term is separable with respect to the different weight vectors w k , the loss function induces a coupling in the columns of W. As a result, the optimization has to be carried out over the entire matrix W.

In Section 5.2.4, we consider another multi-class classification scheme. The "oneversus-all" strategy (OVA) consists in training c different (real-valued) binary classifiers, each one being trained to distinguish the examples in a single class from the observations in all remaining classes. In order to classify a new example, among the c classifiers, the one which outputs the largest (most positive) value is chosen. In this framework, we consider binary classifiers built from both the square and the logistic loss functions. If we denote by Ȳ ∈ {-1, 1} n×c the indicator response matrix defined as

Ȳk i 1 if y i = k and -1 otherwise, we obtain min W∈R p×c 1 2n c k=1 Ȳk -Xw k 2 2 + λ c k=1 Ω(w k ),
and

min W∈R p×c b∈R c 1 n n i=1 c k=1 log 1 + e -Ȳk i (x ⊤ i w k +b k ) + λ c k=1 Ω(w k ).
By invoking the same arguments as in Section 5.1.3, the vector of intercepts b is again omitted in the above problem with the square loss. The formulations we reviewed in this section can be solved efficiently within the same optimization framework we now introduce.

Optimization

The convex minimization problem (5.1) is challenging, since the penalty Ω as defined in (5.4) is non-smooth and the number of variables to deal with is large (about p ≈ 10 5 voxels in the following experiments). To this end, we resort to proximal methods (see, e.g., [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Combettes | Proximal splitting methods in signal processing[END_REF][START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF][START_REF] Wright | Sparse reconstruction by separable approximation[END_REF]). In a nutshell, these methods can be seen as a natural extension of gradient-based techniques when the objective function to minimize has an amenable non-smooth part. They have increasingly drawn the attention of a broad research community because of their convergence rates (optimal within the class of first-order techniques) and their ability to deal with large non-smooth convex problems. We assume from now on that the convex loss function L(y, X, .) is differentiable with Lipschitz-continuous gradient, which notably covers the cases of the square and simple/multinomial logistic functions (introduced in Section 5.1.3).

Multi-scale Mining of fMRI Data with Hierarchical Structured Sparsity

The simplest version of this class of methods linearizes at each iteration the function L(y, X, .) around the current estimate w 0 ,2 and this estimate is then updated as the (unique by strong convexity) solution of the proximal problem:

min w∈R p L(y, X, w 0 ) + (w -w 0 ) ⊤ ∇L w (y, X, w 0 ) + λΩ(w) + L 2 w -w 0 2 2 .
The quadratic term keeps the update in a neighborhood where L(y, X, w 0 ) is close to its linear approximation, and L > 0 is a parameter which is a upper bound on the Lipschitz constant of the gradient of L. This problem can be equivalently rewritten as:

min w∈R p 1 2 w 0 - 1 L ∇L w (y, X, w 0 ) -w 2 2 + λ L Ω(w).
(5.5)

Solving efficiently and exactly this problem is crucial to enjoy the fastest convergence rates of proximal methods. In addition, when the non-smooth term Ω is not present, the previous proximal problem exactly leads to the standard gradient update rule. In simple settings, the solution of problem (5.5) is given in closed form: For instance, when the regularization Ω is chosen to be the ℓ 1 -norm, we get back the well-known soft-thresholding operator [START_REF] Donoho | Adapting to unknown smoothness via wavelet shrinkage[END_REF]. The work of Jenatton et al. (2010a) recently showed that the proximal problem (5.5) could be solved efficiently and exactly with Ω as defined in (5.4). The underlying idea of this computation is to solve a well-ordered sequence of simple proximal problems associated with each of the terms w g for g ∈ G. We refer the interested readers to Jenatton et al. (2011c) for further details.

In our experiments, we will use the accelerated proximal gradient scheme (FISTA) taken from [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], which is a similar procedure as the one described above, except that the proximal problem (5.5) is not solved for the current estimate, but for an auxiliary sequence of points that are linear combinations of past estimates. 3 In terms of computational complexity, such proximal schemes are guaranteed to be ε close to the optimal objective function in O( L/ε) iterations [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF][START_REF] Nesterov | Gradient methods for minimizing composite objective function[END_REF]. The cost of each iteration is dominated by the computation of the gradient (e.g., O(np) for the square loss) and the proximal operator, whose time complexity is linear, or close to linear, in p for the tree-structured regularization (Jenatton et al., 2011c).

Experiments and results

We now present experimental results on simulated data and real fMRI data.

Simulations

In order to illustrate the proposed method, the hierarchical regularization with the ℓ 2norm and η g = 1 for all g was applied in a regression setting on a small two-dimensional simulated dataset consisting of 300 square images (40 × 40 pixels i.e. X ∈ R 300×1600 ). The weight vector w used in the simulation-itself an image of the same dimensionis presented in Fig. 5.2-a. It consists of three localized regions of two different sizes that are predictive of the output. The images x (i) are sampled so as to obtain a correlation structure which mimics fMRI data. Precisely, each image x (i) was obtained by smoothing a completely random image -where each pixel was drawn i.i.d from a normal distribution -with a Gaussian kernel, which introduces spatial correlations between neighboring pixels. Subsequently, correlations between the regions corresponding to the three patterns were introduced in order to simulate co-activations between different brain regions (0.3 correlation between the two bigger patterns, and -0.2 correlation between the smallest and lower-corner patterns).

The choice of the weights and of the correlation introduced in images aim at illustrating how the hierarchical regularization estimates weights at different resolutions in the image. The targets were simulated by forming w ⊤ x (i) corrupted with an additive white noise (SNR=10dB). The loss used was the square loss as detailed in Section 5.1.3. The regularization parameter was estimated with two-fold cross-validation (150 images per fold) on a logarithmic grid of 30 values between 10 3 and 10 -3 .

The weights estimated are presented in Fig. 5.2 at different scales, i.e., different depths in the tree. It can be observed that all three patterns are present in the weight vector but at different depth in the tree. The small activation in the top-right hand corner shows up mainly in scale 3 while the bigger patterns appear higher in the tree in scales 5 and 6. This simulation clearly illustrates the ability of the method to capture informative spatial patterns at different scales. We now present results on real data.

Description on the data

We apply the different methods to analyze the data of ten subjects from an fMRI study originally designed to investigate object coding in high-level visual cortex (see Eger et al., 2008, for details). During the experiment, twelve healthy volunteers viewed objects of two categories (each one of the two categories is used in half of the subjects) with four different exemplars in each category. Each exemplar was presented at three different sizes (yielding 12 different experimental conditions per subject). Each stimulus was presented four times in each of the six sessions. We averaged data from the four repetitions, resulting in a total of n = 72 images by subject (one image of each stimulus by session). Functional images were acquired on a 3-T MR system with eight-channel head coil (Siemens Trio, Erlangen, Germany) as T2*-weighted echo-planar image (EPI) volumes. Twenty transverse slices were obtained with a repetition time of 2s (echo time, 30ms; flip angle, 70 • ; 2 × 2 × 2-mm voxels; 0.5-mm gap). Realignment, normalization to MNI space, and GLM fit were performed with the SPM5 software4 . In the GLM, the time course of each of the 12 stimuli convolved with a standard hemodynamic response function was modeled separately, while accounting for serial auto-correlation with an AR(1) model and removing low-frequency drift terms with a high-pass filter with a cut- off of 128s. In the present work we used the resulting session-wise parameter estimate images. All the analysis are performed on the whole brain volume.

The four different exemplars in each of the two categories were pooled, leading to images labeled according to the three possible sizes of the object. By doing so, we are interested in finding discriminative information to predict the size of the presented object.

This can be reduced to either a regression problem in which our goal is to predict a simple scalar factor (size or scale of the presented object), or a three-category classification problem, each size corresponding to a category. We perform an inter-subject analysis on the sizes both in regression and classification settings. This analysis relies on subject-specific fixed-effects activations, i.e., for each condition, the six activation maps corresponding to the six sessions are averaged together. This yields a total of 12 images per subject, one for each experimental condition. The dimensions of the real data set are p ≈ 7×10 4 and n = 120 (divided into three different sizes). We evaluate the performance of the method by cross-validation with a natural data splitting, leave-one-subject-out. Each fold consists of 12 volumes. The parameter λ of all methods is optimized over a grid of 30 values of the form 2 k , with a nested leave-one-subject-out cross-validation on the training set. The exact scaling of the grid varies for each model to account for different Ω.

Application of Structured Sparsity to Neuroimaging

Methods involved in the comparisons

In addition to considering standard ℓ 1 -and squared ℓ 2 -regularizations in both our regression and multi-class classification tasks, we compare various methods that we now review.

First of all, when the regularization Ω as defined in (5.4) is employed, we consider three settings of values for (η g ) g∈G which leverage the tree structure T . More precisely, we set η g = ρ depth(g) for g in G, with ρ ∈ {0.5, 1, 1.5} and where depth(g) denotes the depth of the root of the group g in T . In other words, the larger ρ, the more averse we are to selecting small (and variable) parcels located near the leaves of T .

The greedy approach from Michel et al. ( 2010) is included in the comparisons, for both the regression and classification tasks. It relies on a top-down exploration of the tree T . In short, starting from the root parcel that contains all the voxels, we choose at each step the split of the parcel that yields the highest prediction score. The exploration step is performed until a given number of parcels is reached, and yields a set of nested parcellations with increasing complexity. Similarly to a model selection step, we chose the best parcellation among those found in the exploration step. The selected parcellation is thus used on the test set. In the regression setting, this approach is combined with Bayesian ridge regression, while it is associated with a linear support vector machine for the classification task (whose value of C is found by nested cross-validation in {0.01, 0.1, 1}).

Regression setting.

In order to evaluate whether the level of sparsity is critical in our analysis, we implemented a reweighted ℓ 1 -scheme [START_REF] Candes | Enhancing sparsity by reweighted l1 minimization[END_REF]. In this case, sparsity is encouraged more aggressively as a multi-stage convex relaxation of a concave penalty. Specifically, it consists in using iteratively a weighted ℓ 1 -norm, whose weights are determined by the solution of previous iteration.

To better understand the added value of the hierarchical norm (5.4) over unstructured penalties, we consider another variant of weighted ℓ 1 -norm, this time defined in the augmented space of features. The weights are manually set and reflect the underlying tree structure T . By analogy with the choice of (η g ) g∈G made for the tree-structured regularization, we take exponential weights depending on the depth of the variable j, with ρ = 1.5. 5 We also tried weights (η g ) g∈G that are linear with respect to the depths, but those led to worse results. We now turn to the models taking part in the classification task.

Classification setting. As discussed in Section 5.1.3, the optimization in the classification setting is carried out over a matrix of weights W ∈ R p×c . This makes it possible to consider other regularization schemes.

In particular, we apply ideas from multi-task learning [START_REF] Obozinski | Joint covariate selection and joint subspace selection for multiple classification problems[END_REF]) by viewing each class as a task. More precisely, we use a regularization norm defined by Ω multi-task (W) p j=1 W j , where W j denotes either the ℓ 2 -or ℓ ∞ -norm of the j-th row of W. The rationale for the definition of Ω multi-task is to assume that the set of relevant voxels is the same across the c different classes, so that sparsity is induced simultaneously over the columns of W. As a remark, in the "one-versus-all" setting, although the loss functions for the c classes are decoupled, the use of Ω multi-task induces a relationship that ties them together.

Note that the tree-structured regularization Ω we consider does not impose a joint pattern-selection across the c different classes. Although a multi-task extension of Ω with ℓ ∞ -norms has recently been proposed (Mairal et al., 2010b), the cost of the corresponding proximal operator is significantly higher, which is likely to raise some computational issues in our large-scale experiments.

Results

We present result of the comparison of our approach based on the hierarchical sparsity-inducing norm (5.4) with the models presented in the previous section. For each method, we computed the cross-validated prediction accuracy and the percentage of non-zero coefficients, i.e., the level of sparsity of the models.

Regression results.

The results for the inter-subject regression analysis are given in Table 5.1. The lowest error in prediction accuracy is obtained by the proposed hierarchical structured sparsity approach (Tree ℓ 2 with ρ = 1), that also yields one of the lowest (along with greedy) standard deviation indicating that the results are most stable. This can be explained by the fact that the use of local signal averages in the proposed algorithm is a good way to get some robustness to inter-subject variability. We also notice that the sparsity-inducing approaches (Lasso and reweighted ℓ 1 ) have the highest error in prediction accuracy, probably because the obtained solutions are too sparse, and suffer from the absence of perfect voxel-to-voxel correspondences between subjects.

In terms of sparsity, we can see, as expected, that ridge regression does not yield any sparsity and that the Lasso solution is very sparse (in the feature space, with approximately 7 × 10 4 voxels). Our method yields a median value of 9.36% of non-zero coefficients (in the augmented space of features, with about 1.4 × 10 5 nodes in the tree). The maps of weights obtained with Lasso and the hierarchical regularization for one fold, are given in Fig. 5.3. The Lasso yields a scattered and overly sparse pattern of voxels, that is not easily readable, while our approach extracts a pattern of voxels with a compact structure, that clearly outlines brain regions expected to activate differentially for stimuli with different low-level visual properties, e.g., sizes; the early visual cortex in the occipital lobe at the back of the brain. Interestingly, the patterns of voxels show some symmetry between left and right hemispheres, especially in the primary visual cortex which is located at the back and center of the brain. Such an observation matches very well with existing neurosciences knowledge of this brain region that processes the visual contents of both visual hemifields. The weights obtained at different depth level in the tree, corresponding to different scales, show that the largest coefficients are concentrated From the left, the first column contains the mean and standard deviation of the test error (unexplained variance), computed over leave-one-subject-out folds. The best performance is obtained with the hierarchical ℓ 2 penalization (ρ = 1) constructed from the Ward tree. Statistical significance is assessed with a Wilcoxon two-sample paired signed rank test. The superscript * indicates a rejection at 5%.

at the higher scales (scale 6 in Fig. 5.3), showing that the object size cannot be well decoded at the voxel level but requires features formed by more macroscopic clusters of voxels.

Classification results. The results for the inter-subject classification analysis are given in Table 5.2. The best performance is obtained with a multinomial logistic loss function, also using the hierarchical ℓ 2 penalization (ρ = 1).

For both ℓ 1 and hierarchical regularizations, one of the three vectors of coefficients obtained for one fold are presented in Fig. 5.4. While for ℓ 1 , the active voxels are scattered all over the brain, the tree ℓ 2 regularization yields clearly delineated sparsity patterns located in the visual areas of the brain. Like for the regression results, the highest coefficients are obtained at scale 6 showing how spatially extended is the brain region involved in the cognitive task. The symmetry of the pattern at this scale is also particularly striking in the primary visual areas. It also extends more anteriorly into the inferior temporal cortex, known for high-level visual processing.

Multi-scale Mining of fMRI Data with Hierarchical Structured Sparsity

Loss function:

Square ("one-versus-all")

Error (mean,std) P-value w.r.t. Tree ℓ2 (ρ = 1)-ML Median fraction of non-zeros (%) Regularization: ℓ2 (Ridge) (29. -d) tree regularization at different scales -In this case, the regularization algorithm extracts a pattern of voxels with a compact structure, that clearly outlines early visual cortex which is expected to discriminate between stimuli of different sizes.

Conclusion

In this chapter, we introduced a hierarchically structured regularization, which takes into account the spatial and multi-scale structure of fMRI data. This approach copes with inter-subject variability in a similar way as feature agglomeration, by averaging neighboring voxels. Although alternative agglomeration strategies do exist, we simply used the criterion which appears as the most natural, Ward's clustering, and which builds parcels with little variance.

Results on a real dataset show that the proposed algorithm is a promising tool for mining fMRI data. It yields higher prediction accuracy than reference methods, and the map of weights it obtains exhibit a cluster-like structure. It makes them easily readable compared to the overly sparse patterns found by classical sparsity-promoting approaches.

For the regression problem, both the greedy method from [START_REF] Michel | A supervised clustering approach for extracting predictive information from brain activation images[END_REF] and the proposed algorithm yield better results than unstructured and non-hierarchical regularizations. However, in both regression and classification settings, the convex formulation introduced here leads to the best performance while enjoying the guarantees of convex optimization. In particular, while the greedy algorithm relies on a two-step approach that may be far from optimal, the hierarchical regularization induces simultaneously the selection of the optimal parcellation and the construction of the optimal predictive model, given the initial hierarchical clustering of the voxels. Moreover, convex methods yield predictors that are essentially stable with respect to perturbations of the design or the initial clustering, which is typically not the case of greedy methods.

Finally, it should be mentioned that the performance achieved by this approach in inter-subject problems suggests that it could potentially be used successfully in medical diagnosis problems, where brain images -not necessarily functional images-are used to classify individuals into diseased or control population. Indeed, for difficult problems of that sort, where the reliability of the diagnostic is essential, the stability of models obtained from convex formulations and the interpretability of sparse and localized solutions are useful properties to have in order to provide a credible diagnostic.

Sparse Structured Dictionary Learning for Brain Resting-State Activity Modeling

Abstract of the second section of the chapter: We introduce a generative model to study brain resting-state time series. These signals are represented as linear combinations of latent spatial maps, which we obtain via matrix factorization techniques developed for dictionary learning.

In particular, we propose to learn the spatial components with specific structural constraints, e.g, small localized clusters of voxels, which can be achieved with sparsity-inducing regularization schemes recently used for dictionary learning. While brain resting-state time-series are generally the object of exploratory data analysis, our model provides a natural framework for model selection and quantitative evaluation. We show that our approach yields improved estimates

Introduction

Functional magnetic resonance imaging (fMRI) yields in-vivo time-resolved measurements of brain activity. As the workhorse of neuroimaging, it has been the basis for much research on brain function, such as the investigation of neural coding corresponding to specific behavior. The study of intrinsic brain function is receiving increasing interest from the neuroscience community [START_REF] Biswal | Toward discovery science of human brain function[END_REF]. For this purpose, the spontaneous brain activity of subjects in the absence of a task is recorded in so-called resting-state experiments. In this setting, the statistical modeling of the signals is an unsupervised problem, and independent component analysis (ICA) has been widely used to study decomposition of the brain-activation time series in a set of interpretable spatial components [START_REF] Beckmann | Probabilistic independent component analysis for functional magnetic resonance imaging[END_REF][START_REF] Beckmann | Investigations into restingstate connectivity using independent component analysis[END_REF]Varoquaux et al., 2010e). More recently, graphical models learned from the correlation matrices between distant activation time series have been shown to contain discriminative features of pathologies or cognitive processes [START_REF] Cecchi | Discriminative network models of schizophrenia[END_REF]Varoquaux et al., 2010a;[START_REF] Richiardi | Decoding brain states from fMRI connectivity graphs[END_REF]. These two approaches are complementary: on the one hand, spatial ICA extracts spatial patterns of coactivation, but forgoes any modeling of the correlation structure of the time series, since it operates on whitened data; on the other hand, learning statistically sound correlation matrices with the small sample size of neuroimaging data must rely on the definition of regions of interest to extract the relevant signal [START_REF] Varoquaux | Detection of brain functional-connectivity difference in post-stroke patients using group-level covariance modeling[END_REF].

Here we propose to simultaneously learn spatial patterns as well as the associated correlation structure by fitting a single model, leveraging recent work on sparse structured principal component analysis (SSPCA) (Jenatton et al., 2010b). Specifically, the time series are represented as linear combinations of factors, or dictionary elements, that are not simply constrained to be sparse (i.e., to use only a small number of voxels) but that are rather explicitly constrained to be supported by spatially-compact regions. The motivation for introducing this structure is the high degree of local spatial organization of the neuronal maps corresponding to different cognitive functions [START_REF] Chklovskii | Maps in the brain: What can we learn from them?[END_REF]. In addition, the spatial variability of function across different subjects limits the ability of brain imaging to resolve functional processes and dictates the use of brain regions learned from the data for inter-subject comparisons [START_REF] Thirion | Dealing with the shortcomings of spatial normalization: Multi-subject parcellation of fMRI datasets[END_REF]. As detailed in the conclusion, an important consequence of selecting for structured com-

Application of Structured Sparsity to Neuroimaging

ponents, as with SSPCA, rather than for independent components, as with ICA, is that it gives a two level representation, segmenting localized brain regions regardless of the fluctuations of large scale cognitive networks that have been shown to encode cognitive or pathology-related brain states.

We introduce a new generative model for spontaneous brain activity that is suitable for model selection. This model provides a consistent framework for covariance estimation as well as various matrix decomposition methods, such as ICA, the reference method in neuroimaging [START_REF] Beckmann | Probabilistic independent component analysis for functional magnetic resonance imaging[END_REF]. As illustrated in our experiments, our approach seeking structure in the data leads to improvements over other unstructured decompositions, with both (1) an increase in model likelihood on left-out time series corresponding to unseen subjects, as well as ( 2) decomposition in spatial components that matches known brain areas of interest and is thus interpretable. Our model is consistent with current neuroscientific understanding of spontaneous brain activity.

Model and Problem Statement

We start by introducing our generative model and by motivating the use of dictionary learning. We consider brain resting-state time series measured on m voxels, and denote by x ∈ R m one of the time points, a three-dimensional image of brain activity. We assume that the signal x can be expressed as a linear combination of p dictionary elements that are the columns of the dictionary matrix D ∈ R m×p . More precisely, we have

x = Dα + ε, ( 5.6)
where ε ∼ N (0, Σ ε ) is a zero-mean Gaussian noise vector identically and independently distributed (i.i.d.) across the different data points acquired at different times. In the sequel, we shall assume that there is no spatial correlation between different voxels in the noise term, so that Σ ε is chosen to be diagonal. The decomposition of the signal x on the dictionary D is given by the vector α ∼ N (0, Σ α ) which follows a zero-mean multivariate Gaussian distribution, also i.i.d. across the observations. We further assume independence between ε and α, and x is thus Gaussian with covariance matrix

Σ x DΣ α D ⊤ + Σ ε .
(5.7)

In the experiments, we will use this generative model to evaluate the quality of fit of our approach, as measured by the corresponding likelihood function. Now, given a set of n observations of the brain activity images (i.e., n times points) represented by the rows of the data matrix X ∈ R m×n , we are interested in expressing X as X ≈ DA, (5.8)

where the n decomposition vectors are stacked in the matrix A ∈ R p×n . Finding such a pair of matrices (A, D) corresponds precisely to a matrix factorization problem [START_REF] Singh | A Unified View of Matrix Factorization Models[END_REF][START_REF] Witten | A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis[END_REF], also known as dictionary learning in signal processing (Mairal et al., 2010a). Various penalties and constraints can be imposed on A and/or D in order to specify prior knowledge about the target factorization, e.g., positivity [START_REF] Lee | Learning the parts of objects by non-negative matrix factorization[END_REF] or sparsity [START_REF] Lee | Efficient sparse coding algorithms[END_REF]Mairal et al., 2010a). Independent Component Analysis (ICA), widely used in neuroimaging, can also be formulated as a dictionary-learning problem, with the constrain of minimum mutual information between the columns of D.

The wide success that ICA has enjoyed in neuroimaging comes from the interpretability of the dictionary elements that it extracts. Indeed, unlike dictionary elements learned by PCA, ICA components display localized and contrasted features that match functional neuroanatomical knowledge. For instance the component shown on Figure 5.6 corresponds to the brain regions composing the well-known default mode network. More recently, it has been argued that the key to the success of ICA in fMRI is not the independence of the components, but their sparsity [START_REF] Daubechies | Independent component analysis for brain fMRI does not select for independence[END_REF]Varoquaux et al., 2010d). In addition, systematic studies of all ICA components learn from resting-state brain activity show that they are composed of localized features, that correspond either to functional brain units or to structured artifacts in the signal generated by anatomical features (Varoquaux et al., 2010e). It thus appears that, for the interpretation of the decomposition on dictionary elements (i.e., the columns of D), of the spatial compactness of the salient features that they display plays an important role: imaging neuroscientists think in terms of brain activation areas forming small localized clusters. Interestingly, this property can be encoded by using the structured sparsity-inducing regularization recently introduced in Jenatton et al. (2011a). We now briefly present this regularization scheme and explain how it has been further exploited for dictionary learning (Jenatton et al., 2010b).

Structured Sparse Dictionary Learning

The work of Jenatton et al. Jenatton et al. (2011a) considered a norm Ω which induces structured sparsity in the following sense: the solutions to a learning problem regularized by this norm have a sparse support which moreover belongs to a predefined set of possible nonzero patterns. Interesting examples of such sets of supports include sets of variables forming contiguous segments or rectangles when arranged respectively on a line or on a grid. More formally, the norm Ω can be defined on R m by introducing a set G of subsets of {1, . . . , m} that we refer to as groups. The choice of the set of groups G defines the nature of the possible sparsity patterns associated with Ω. For any vector

d ∈ R m , Ω(d) g∈G d g 2 = g∈G j∈g d 2 j 1/2 .
(5.9)

In other words, Ω consists of a sum of non-squared ℓ 2 -norms on subsets of variables specified by g ∈ G. The definition (5.9) covers interesting subcases: for example, when G is the set of singletons, we get back the ℓ 1 -norm. As analyzed by (Jenatton et al., 2011a), Ω promotes sparsity at the level of groups, in the sense that it acts as a ℓ 1 -norm on the vector ( d g 2 ) g∈G . Regularizing by Ω therefore causes some d g 2 (and equivalently d g ) to be zeroed out for some g in G.

In the experiments, we design a particular set of overlapping groups G that is adapted to the 3-dimensional structure of the voxels. Our choice of G constrains the sparsity patterns to form rectangular boxes, in order to favor the selection of clustered voxels. This is achieved by considering for G all axis-aligned half-spaces of the discrete 3-dimensional space of voxels. The norm Ω can naturally fit in the dictionary-learning framework (Jenatton et al., 2010b). Suppose that we look for a pair of matrices A = [α 1 , . . . , α n ] = [A 1 , . . . , A p ] and D = [d 1 , . . . , d p ] such that (5.8) holds. If in addition we require to obtain sparse structured dictionary elements, we can consider the formulation of Jenatton et al. (2010b), 6 that is,

min A∈R p×n ,D∈R m×p 1 2 X-DA 2 F +λ p k=1 Ω(d k ) such that A k 2 ≤ 1 for all k ∈ 1; p . (5.10)
The positive scalar λ controls the strength of the regularization, while the constraints on the rows of the matrix A are required to avoid degenerate solutions. The specific choice of the ℓ 2 -norm follows Jenatton et al. (2010b). Problem (5.10) is not jointly convex in A and D, but convex with respect to one matrix while the other one is kept fixed, and vice versa. This property calls for an alternate optimization scheme, more precisely a block coordinate descent scheme; we refer the reader to Jenatton et al. (2010b) and references therein for more details on the optimization procedure.

Applications and Experiments

We present in this section the experimental validation of our approach. To this end, we consider the set of brain resting-state time series used in Varoquaux et al. (2010e). Twelve healthy volunteers were scanned at rest, eyes closed, for a period of 20 minutes. Each individual dataset is made of n = 820 volumes (time points) with a 3 mm isotropic resolution, corresponding to approximately 50 000 voxels within the brain. Standard neuroimaging preprocessing was applied using the SPM5 software 7 : after slice-timing interpolation and motion correction, cerebral volumes were realigned to an inter-subject template and smoothed with a 6 mm isotropic Gaussian kernel.

The objective of the experiments is to compare our sparse structured approach (SSPCA) with a standard sparse dictionary learning formulation (Sparse PCA, SPCA)i.e., using the ℓ 1 -norm in place of Ω in the formulation (5.10)-and with principal component analysis (PCA). This validation scheme is based solely on the decomposition DA and thus makes no distinction between ICA and PCA. Indeed, the matrices A, D found by ICA and PCA are identical up to a rotation, therefore leading to the same likelihood on unseen data 8 . For a set of dictionary sizes, p ∈ {10, 40, 70, . . . , 220}, we perform a 6. In fact, Jenatton et al. (2010b) considers a weighted non-convex variant of Ω that leads to the same set of sparsity patterns, but more aggressively. In the experiments, we adopt the very same setting as Jenatton et al. (2010b); we do not detail this variant for simplicity. Moreover, though non-convex, the optimization with the variant of Ω follows along the same line thanks to a variational formulation.

7. Wellcome Department of Cognitive Neurology; www.fil.ion.ucl.ac.uk/spm 8. However, the spatial components obtained by these two methods are different.

Sparse Structured Dictionary Learning for Brain Resting-State Activity Modeling

6-fold cross-validation that respects splits of the data by subjects, i.e., each fold contains all the time points for two subjects. After learning a decomposition ( Â, D) on 5 out of the 6 folds, we compute an estimate Σx of Σ x as defined in (5.7), where Σα is the sample covariance for Â, and Σε is obtained from the residuals. If we now denote by Σ held-out x the sample covariance matrix of the time-series data points computed from the held-out fold, we evaluate the different approaches based on the likelihood L of the held out data in our Gaussian generative model, which is

L( Σx , Σ held-out x ) -log | Σx | -Tr( Σ-1 x Σ held-out x
).

(5.11)

To lower the computational burden, the space of voxels is down-sampled to m ≈ 5 000, where the activation of a single voxel is replaced by the average activation of a 2×2 ×2cube of voxels. The cross-validation scores are displayed on Figure 5.5. Both SPCA and SSPCA are non-convex, and solutions therefore depend on the initialization of the algorithms. We experimentally observed that the cross-validation scores are quite stable under random initialization of A and D. Moreover, we assess the statistical significance of the cross-validation scores in Table 5. On Figure 5.5, we can see that, as the number of elements in the dictionary learned increases, the various learning procedures extract a more detailed representation and the 5.10 -2 3.10 -3 1.10 -3 7.10 -5 1.10 -4 4.10 -6 8.10 -6 1.10 -5 SSPCA vs. PCA/ICA 4.10 -2 3.10 -3 5.10 -3 2.10 -3 1.10 -3 8.10 -4 6.10 -4 6.10 -4

Application of Structured Sparsity to Neuroimaging

Table 5.3: P-values from paired t-tests carried out on the series of likelihood results over the 6 cross-validation scores. For the models learned using SSPCA, the average likelihood on the held-out data is significantly higher than those obtained using SPCA and PCA at the 1% significance level for dictionary sizes greater than 40.

likelihood of unseen data increases. However for PCA and ICA, this generalization score reaches a plateau and decreases as the algorithms overfit and learn dictionary elements from signal that is not reproducible across folds, that is between different subjects. Interestingly, the optimal number of ICA components set by this model-selection procedure, 50, corresponds roughly to the number selected on the same dataset by a different procedure unrelated to the proposed generative model (Varoquaux et al., 2010e). For the penalized methods, SPCA and SSPCA, the generalization score keeps increasing with the number of elements in the dictionary: the penalization controls over-fitting. SSPCA generalizes best for large dictionary sizes and is thus best suited for obtaining a detailed description of resting-state brain covariance.

In our experiments, the generalization scores of models learned by SSPCA and SPCA do not reach a maximum. This could indicate that the best description of the resting brain involves a higher number of regions. Indeed, [START_REF] Tucholka | Probabilistic anatomo-functional parcellation of the cortex: how many regions?[END_REF] argues that, in a task-based fMRI study, the optimal number of regions lies around 500. As the computational cost of these methods scales in p 2 , our experiments were limited by available computational resources and we plan to explore larger dictionary sizes in the future.

Discussion

The key idea behind the study of resting-state brain activity is that the correlations in the signal observed by fMRI reveal regions of the brain that function together. This is why modeling the covariance matrix of fMRI time series is receiving increasing attention [START_REF] Varoquaux | Detection of brain functional-connectivity difference in post-stroke patients using group-level covariance modeling[END_REF][START_REF] Cecchi | Discriminative network models of schizophrenia[END_REF]. However, with the limited number of samples, learning a covariance between the time series of all the 50 000 brain voxels is an ill-posed problem. The generative model defined by equations (5.7) and (5.8) can be interpreted as decomposing the covariance of the observed signal in a structured part, given by the dictionary elements D and the covariance Σ α , and an unstructured part, given by the diagonal covariance Σ ε . As such, this model is well-suited for the study of resting-state functional connectivity.

The dictionary elements estimated by SSPCA are spatial maps displaying one or two localized features. As such, they define a set of regions that explain well the signal. As it is usually done in neuroimaging, the spatial map learned by decomposition of the functional images is displayed, thresholded, on an anatomical brain image. (right) Three dictionary elements corresponding to the same brain regions. The different colormaps (red, green and blue) correspond to the different dictionary elements learned. The transparency reflects the structured sparsity, rather than an arbitrary threshold 9 . This is why the different elements displayed on the figure have a block-like outline that reflect the shape of the groups used in norm Ω. Some displayed patterns do not seem to be rectangular boxes (e.g., the green patterns). However, a closer look at these dictionary elements shows that boxes are indeed selected, and that small numerical values (just as regularizing by ℓ 2 -norm may lead to) give the visual impression of having zeroes between the two green areas.

Figure 5.5 shows the outline of all the regions defined by such a decomposition. These regions either correspond to functional brain regions, located in the gray matter, or are characteristic of non-neural signals, such as movement, most-often considered as artifacts to be accounted for in fMRI studies. We find that the brain regions segmented by SSPCA are well positioned on functionally-relevant anatomical structures, such as cortical folds, and correspond well to the regions forming the well-known brain networks extracted most-often by ICA (see Figure 5.6).

Compared to standard ICA-based procedures for fMRI, our approach based on SSPCA offers several improvements that go beyond better cross-validation performance for large dictionaries. First of all, it naturally extracts regions: no thresholding is necessary to recover sparsity as in Varoquaux et al. (2010d) and spatially scattered dictionary elements are penalized. In addition, ICA has no intrinsic notion of noise: it forgoes explained variance and explains all the data at hand by estimating a rotation matrix, called the mixing matrix, that maps whitened signals to dictionary elements. As a result, ICA procedures used in fMRI rely on PCA to select a subspace maximizing the explained variance [START_REF] Beckmann | Probabilistic independent component analysis for functional magnetic resonance imaging[END_REF]Varoquaux et al., 2010e). On the contrary, SPCA and SSPCA perform subspace selection and signal decomposition in a single step, relying on the penalization to model noise. Thus, when using SSPCA, the proportion of the observed signal modelled as noise is set by the structured-sparsity prior.

9. The optimization procedure we rely on (Jenatton et al., 2010b) is not based on thresholding to set coefficients explicitly to zero, but on successive re-weighing of coefficients, some of which converge to zero. As a result the sparsity of the dictionary element is given by the error tolerance set in the optimization.

147

Application of Structured Sparsity to Neuroimaging

But the most important improvement of SSPCA over ICA is probably to differentiate the estimation of localized regions from the grouping distant regions in networks: D encodes only local information and all long-distance structure can be found in Σ α . Indeed, a large body of work has shown that the amount of correlation between distant brain regions is modulated by cognitive brain processes [START_REF] Richiardi | Decoding brain states from fMRI connectivity graphs[END_REF] or pathologies [START_REF] Vanhaudenhuyse | Default network connectivity reflects level of consciousness in non-communicative brain-damaged patients[END_REF][START_REF] Cecchi | Discriminative network models of schizophrenia[END_REF]Varoquaux et al., 2010a). When estimating dictionary elements on multiple subjects with different long-distance correlation structures, brain regions extracted by ICA will be affected while there is no reason to believe that the underlying localized functional units differ. SSPCA is more robust to such variations in the data. As such, it is more suited to extract regions that will be used in a second step to perform principled inter-subject inference on the correlation structure (Varoquaux et al., 2010a).

Conclusion

We introduced in this paper a generative model for brain resting-state activity. We use the dictionary-learning framework to learn this model from fMRI time-series. In these settings, we can compare several dictionary-learning techniques using cross-validation. We apply sparse-structured PCA, an approach with assumptions that match current understanding of brain spontaneous activity, and show that it generalizes better than alternative methods, including ICA, the reference method in neuroimaging. These preliminary results suggest that SSPCA is a good candidate for learning regions from the data, which is a challenging problem, critical for brain covariance modeling [START_REF] Varoquaux | Detection of brain functional-connectivity difference in post-stroke patients using group-level covariance modeling[END_REF]. Our proposed model could be extended in further work to estimating a subject-specific covariance matrix with common dictionary elements. This would enable principled inter-subject comparison on brain covariance using a consistent framework based on dictionary learning.

Local Analysis of Sparse Coding in Presence of Noise

Abstract of the chapter: A popular approach within the signal processing and machine learning communities consists in modelling signals as sparse linear combinations of atoms selected from a learned dictionary. While this paradigm has led to numerous empirical successes in various fields ranging from image to audio processing, there have only been a few theoretical arguments supporting these evidences. In particular, sparse coding, or sparse dictionary learning, relies on a non-convex procedure whose local minima have not been fully analyzed yet. In this chapter, we consider a probabilistic model of sparse signals, and show that, with high probability, sparse coding admits a local minimum along some curves passing through the reference dictionary generating the signals.

Our study takes into account the case of over-complete dictionaries and noisy signals, thus extending previous work limited to noiseless settings and/or under-complete dictionaries. The analysis we conduct is non-asymptotic and makes it possible to understand how the key quantities of the problem, such as the coherence or the level of noise, are allowed to scale with respect to the dimension of the signals, the number of atoms, the sparsity and the number of observations. This work in progress constitutes a first step towards the more involved proof of the existence of the local minimum over the entire manifold of normalized dictionaries.

The material of this chapter is based on some work in progress which has been achieved with the collaboration of Francis Bach and Rémi Gribonval.

Introduction

Modelling signals as sparse linear combinations of atoms selected from a dictionary has become a popular paradigm in many fields, including signal processing, statistics, and machine learning. This line of research has witnessed the development of several well-founded theoretical frameworks (e.g., see [START_REF] Tropp | Just relax: Convex programming methods for identifying sparse signals in noise[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF][START_REF] Zhang | Some sharp performance bounds for least squares regression with l1 regularization[END_REF] and efficient algorithmic tools (e.g., see Bach et al., 2011, and references therein).

However, the performance of such approaches hinges on the representation of the signals, which makes the question of designing "good" dictionaries prominent. A great deal of effort has been dedicated to come up with efficient predefined dictionaries, e.g., the various types of wavelets [START_REF] Mallat | A wavelet tour of signal processing[END_REF]. These representations have notably contributed to many successful image processing applications such as compression. More recently,

Local Analysis of Sparse Coding in Presence of Noise

the idea of simultaneously learning the dictionary and the sparse decompositions of the signals -also known as sparse dictionary learning, or simply, sparse coding-has emerged as a powerful framework, with state-of-the-art performances in many tasks, including denoising, inpainting and image classification (e.g., see Mairal et al., 2010a, and references therein).

Although sparse dictionary learning can alternatively be formulated as convex [START_REF] Bach | Convex sparse matrix factorizations[END_REF]Bradley and Bagnell, 2009b), non-parametric Bayesian [START_REF] Zhou | Non-parametric Bayesian dictionary learning for sparse image representations[END_REF] and submodular [START_REF] Krause | Submodular dictionary selection for sparse representation[END_REF] problems, the most popular and widely used definition of sparse coding brings into play a non-convex optimization problem. Despite its empirical and practical success, there has only been a little theoretical analysis of the properties of sparse dictionary learning. For instance, [START_REF] Maurer | k-dimensional coding schemes in hilbert spaces[END_REF]; [START_REF] Vainsencher | The sample complexity of dictionary learning[END_REF] derive generalization bounds which quantify how much the expected signal-reconstruction error differs from the empirical one, computed from a random and finite-size sample of signals. The bounds obtained in [START_REF] Maurer | k-dimensional coding schemes in hilbert spaces[END_REF][START_REF] Vainsencher | The sample complexity of dictionary learning[END_REF] are non-asymptotic and uniform with respect to the whole class of dictionaries considered (e.g., those with normalized atoms). As discussed later, the questions raised in this chapter explore a different and complementary direction.

Another theoretical aspect of interest consists of characterizing the local minima of sparse coding, in spite of the non-convexity of its formulation. This problem is closely related to the question of identifiability, that is, whether it is possible to recover a reference dictionary that is assumed to generate the observed signals. The authors of [START_REF] Gribonval | Dictionary identification-sparse matrix-factorization via ℓ 1 -minimization[END_REF] pioneered research in this direction by considering noiseless signals in the case where the reference dictionary forms a basis. Still in a noiseless setting, [START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF] extended the analysis to over-complete dictionaries, i.e., these composed of more atoms than the dimension of the signals. To the best of our knowledge, comparable analysis have not been carried out yet for noisy signals. In particular, the structure of the proofs of [START_REF] Gribonval | Dictionary identification-sparse matrix-factorization via ℓ 1 -minimization[END_REF]; [START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF] hinges on the absence of noise and cannot be straightforwardly transposed to take into account some noise; this point will be made more formal in the subsequent sections.

In this chapter, we therefore analyze the local minima of sparse coding in presence of noise and make the following contributions:

• Within a probabilistic model of sparse signals, we derive a non-asymptotic characterization of the probability of finding a local minimum along some curves passing through the reference dictionary. • The analysis we conduct makes it possible to better understand (a) how many signals are required to hope for identifiability, (b) what the impact of the degree of over-completeness is, (c) what parameters contribute to the curvature around minima, and (d) what level of noise appears as manageable. • We show that under deterministic coherence-based assumptions, such a local minimum along a curve is guaranteed to exist with high probability. This work is in progress and represents a first milestone towards the more involved 6.2. Problem statement result guaranteeing the existence of a local minimum in a neighborhood of the reference dictionary. Importantly, the neighborhood has to be understood as being within the entire manifold of normalized dictionaries, as opposed to be just along single curves. Arguments based on (1) concentration of the measure uniformly over all such curves, and ( 2) the discretization of this set of curves through ǫ-nets (Massart, 2003) seem to indicate that the current conclusions of this chapter shall still apply with small changes.

Notation. For all vectors v ∈ R p , we denote by sign(v) ∈ {-1, 0, 1} p the vector such that its j-th entry [sign(v)] j is equal to zero if v j = 0, and to one (respectively, minus one) if v j > 0 (respectively, v j < 0). We extensively manipulate matrix norms in the sequel. For any matrix A ∈ R n×p , we define its Frobenius norm by A F [ n i=1 p j=1 A 2 ij ] 1/2 ; similarly, we denote the spectral norm of A by |||A||| 2 max x 2 ≤1 Ax 2 , and refer to the operator ℓ ∞ -norm as

|||A||| ∞ max x ∞≤1 Ax ∞ = max i∈ 1;n p j=1 |A ij |.
For any square matrix B ∈ R n×n , we denote by diag(B) ∈ R n the vector formed by extracting the diagonal terms of B, and conversely, for any b ∈ R n , we use Diag(b) ∈ R n×n to represent the (square) diagonal matrix whose diagonal elements are built from the vector b.

Moreover, we introduce the ordering such that for any scalar a, b ∈ R, the relationship a b holds if and only if there exists a non-negative universal constant ω satisfying a ≤ ωb.

Problem statement

We introduce in this section the material required to define our problem and state our results.

Background material on sparse coding

Let us consider a set of n signals X

[x 1 , . . . , x n ] ∈ R m×n of dimension m, along with a dictionary D [d 1 , . . . , d p ] ∈ R m×p composed of p atoms-also referred to as dictionary elements. Sparse coding simultaneously learns a dictionary D and a set of n sparse vectors A [α 1 , . . . , α n ] ∈ R p×n such that each signal x i can be well approximated by x i ≈ Dα i for i in 1; n . By sparse, we mean that the vector α i has k ≪ p nonzero coefficients, so that we aim at reconstructing x i from only a few atoms. Before introducing the standard formulation of sparse coding [START_REF] Olshausen | Sparse coding with an overcomplete basis set: A strategy employed by V1?[END_REF][START_REF] Lee | Efficient sparse coding algorithms[END_REF]Mairal et al., 2010a), we define the following elements:

Definition 4

For any dictionary D ∈ R m×p and signal x ∈ R m , we define the function f x as

f x (D) min α∈R p 1 2
x -Dα 2 2 + λ α 1 . (6.1)

Local Analysis of Sparse Coding in Presence of Noise

Similarly for any set of n signals X [x 1 , . . . , x n ] ∈ R m×n , we introduce

F n (D) 1 n n i=1 f x i (D). (6.2)
The standard approach to perform sparse coding [START_REF] Olshausen | Sparse coding with an overcomplete basis set: A strategy employed by V1?[END_REF][START_REF] Lee | Efficient sparse coding algorithms[END_REF]Mairal et al., 2010a) where the regularization parameter λ in Eq. ( 6.1) controls the level of sparsity, while D ⊆ R p×n denotes a compact (generally convex) set; in this chapter, D is chosen to be the set of dictionaries with unit ℓ 2 -norm atoms, which is a natural choice in image processing (Mairal et al., 2010a). Note however that other choices for the set D may also be relevant depending on the application at hand (e.g., see Jenatton et al., 2011c, where in the context of topic models, the atoms in D belong to the unit simplex).

The goal of the chapter is to characterize some local minima of the function F n under a generative model for the signals x i . Although the function F n is Lipschitz continuous (Mairal et al., 2010a), its minimization is challenging since it is non-convex and subject to the constraints of D. Moreover, F n is defined through the minimization over the vectors A, which, at first sight, does not lead to a simple and convenient expression. We next show that F n has a simple form in some specific favorable scenarios.

Closed-form expression for F n :

We leverage here a key property of the function f x . Let denote by α ∈ R p a solution of problem (6.1), that is, the minimization defining f x . If the sign of α is known in advance, say ŝ ∈ {-1, 0, 1} p with support defined by J {j ∈ 1; p ; ŝj = 0}, then α has a simple closed-form expression (e.g., see [START_REF] Fuchs | Recovery of exact sparse representations in the presence of bounded noise[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF]. In particular, if we use the notation D J ∈ R p×|J| to denote the dictionary restricted to the |J| atoms indexed by J, and assuming that D ⊤ J D J is invertible, we have

αJ = D ⊤ J D J -1 D ⊤ J x -λŝ J ∈ R |J| and αJ c = 0.
This property is appealing in that it makes it possible to obtain a closed-form expression for f x (and hence, F n ), provided that we can control the sign patterns of α. In the light of this remark, it is natural to define the following function:

Definition 5

Let s ∈ {-1, 0, 1} p be a sign vector with support J. For any dictionary D ∈ R m×p such that D ⊤ J D J is invertible, and for any signal x ∈ R m , we define the function φ x as

φ x (D|s) 1 2 x 2 2 -(D ⊤ J x -λs J ) ⊤ (D ⊤ J D J ) -1 (D ⊤ J x -λs J ) . (6.4)
We define Φ n analogously to F n for a sign matrix S ∈ {-1, 0, 1} p×n .

Problem statement

Based on the previous definition, note that we have the relationship

f x (D) = 1 2 x -D α 2 2 + λ α 1 = φ x (D|ŝ).
As it will be further discussed later, showing that the functions F n and Φ n coincide will amount to studying the sign recovery property of ℓ 1 -regularized least-squares problems, which is a topic already well-understood (e.g., see Wainwright, 2009, and references therein). We now introduce the generative model of sparse signals.

Probabilistic model of sparse signals

Throughout the chapter, we assume the signals we observe are generated independently according to a specific probabilistic model. Let us consider a fixed reference dictionary D 0 ∈ D. Each signal x ∈ R m is built from the following independent steps:

(1) Draw without replacement k atoms out of the p available in D 0 . This procedure thus defines a support J {j ∈ 1; p ; δ(j) = 1} whose size is |J| = k, and where δ(j) denotes the indicator function equal to one if the j-th atom is selected, zero otherwise. Note that E[δ(j)] = k p , and for i = j, we further have

E[δ(j)δ(i)] = k(k-1) p(p-1) ≤ k 2 p 2 .
Our result also holds for any sampling scheme for which the conditions above on the expectation are satisfied.

(2) Define a sparse vector α 0 ∈ R p whose entries in J are generated i.i.d. according to a symmetric distribution with a compact support bounded away from zero. Formally, there exist some strictly positive scalars (α, α) such that for all j ∈ J, it holds that |[α 0 ] j | ∈ [α, α] almost surely. On the other hand, for j not in J, [α 0 ] j is set to zero. As discussed later in Section 6.3.1, our analysis can probably be extended to the class of distributions with non-compact supports containing zero, with a sufficiently small mass at zero. (3) Eventually generate the signal x = D 0 α 0 + ε, where the entries of the additive noise ε ∈ R m are assumed i.i.d. Gaussian with zero-mean and variance σ 2 . The Gaussian assumption is made here for simplicity; our study can be similarly conducted for more general sub-Gaussian noises. With this generative model in place, we can state more precisely our objective: we want to show that Pr F n has a local minimum in a "neighborhood" of D 0 ≈ 1, where the probability is with respect to the two sources of randomness, i.e., α 0 and ε. Note that we loosely refer to a certain "neighborhood" since in our regularized formulation, a local minimum cannot appear exactly at D 0 . The proper meaning of this neighborhood is the subject of the next section.

Importantly, we have so far referred to D 0 as the reference dictionary generating the signals. However, and as already discussed in [START_REF] Gribonval | Dictionary identification-sparse matrix-factorization via ℓ 1 -minimization[END_REF]; [START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF], we know that for any permutation matrix Σ and any diagonal matrix Diag(s) with the vector s in {-1, 1} p , we have D 0 α 0 = (D 0 Σ -1 Diag(s) -1 )(Diag(s)Σα 0 ) 6. Local Analysis of Sparse Coding in Presence of Noise with Diag(s)Σα 0 1 = α 0 1 . As a result, while solving (6.3), we cannot hope for the identifiability of the specific D 0 , but we focus instead on the identifiability of the whole equivalence class defined by the transformations described above. From now on, we simply refer to D 0 to denote one element of this equivalence class.

Oblique manifold and tangent space

The minimization of F n is carried out over D, which is the set of dictionaries with unit ℓ 2 -norm atoms. This set turns out to be a manifold, known as the oblique manifold (Absil et al., 2008). Since D 0 is assumed to belong to D, it is therefore natural to consider the behavior of F n according to the geometry of D. More specifically, let us consider the set of matrices

W D 0 W ∈ R m×p ; diag(W ⊤ D 0 ) = 0 and diag(W ⊤ W) = 1 .
Now, for any matrix W ∈ W D 0 , for any velocity vector v ∈ R p with v 2 = 1, and for all t ∈ R, we introduce the parameterized dictionary:

D(t) D 0 Diag[cos(vt)] + WDiag[sin(vt)], (6.5)
where Diag[cos(vt)] and Diag[sin(vt)] ∈ R p×p stand for the diagonal matrices with diagonal terms equal to {cos(v j t)} j∈ 1;p and {sin(v j t)} j∈ 1;p respectively. By construction In this case, the set W D 0 is given by

W D 0 = W = [w 1 , w 2 ] ∈ R 2×2 ; w 1 = ± 1 0 and w 2 = ± 0 1 ,
while the entire tangent space at D 0 is the following vector space In particular, we can see that the matrix M = 1 0 0 0 does belong to the tangent space whereas we obviously have M / ∈ W D 0 . Moreover, the matrix M can be written as M = WDiag(v), with the vector v = 1 0 , v 2 = 1, and the matrix

W = [w 1 , w 2 ] ∈ R 2×2 ; (w 1 ) ⊤ d 1 0 = 0,
W = 1 0 0 1 ∈ W D 0 .
We can then describe the neighborhood of D 0 through the triplet (t, v, W). In particular, instead of studying the local minima of D → F n (D), we focus on the function t → F n (D(t)) for any pair (v, W). Loosely speaking, this corresponds to explore every curve of D passing through D 0 ; see Figure 6.1. We next describe the last ingredient required to state our results.

Mutual coherence

Our analysis needs the reference dictionary D 0 to respect some coherence-based properties. We thus define the mutual coherence of the dictionary D 0 (e.g., see [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF][START_REF] Feuer | On sparse representation in pairs of bases[END_REF][START_REF] Lounici | Sup-norm convergence rate and sign concentration property of lasso and dantzig estimators[END_REF] by

µ 0 max i,j∈ 1;p ,i =j |[d i 0 ] ⊤ [d j 0 ]| ∈ [0, 1].
The term µ 0 gives a measure of the level of correlation in D 0 , and it is for instance equal to zero in the case of an orthogonal dictionary. Similarly, we introduce µ(t) for the dictionary D(t). For any t ∈ R, we notably have the simple inequality:

µ(t) max i,j∈ 1;p ,i =j |[d i (t)] ⊤ [d j (t)]| ≤ µ 0 + 3|t|. (6.6)

Local Analysis of Sparse Coding in Presence of Noise

In particular, note that we have µ(0) = µ 0 . We now see how the coherence is exploited in our study.

Assumption on the reference dictionary D 0 : For the theoretical analysis we conduct here, we consider a deterministic coherence-based assumption, such that the coherence µ 0 and the level of sparsity k should be inversely proportional. Such an assumption was considered for instance in the previous work by [START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF].

The need for this constraint on µ 0 appears twice in the analysis, first to control the supports of the solutions of (6.1) (see Section 6.3.3 for more details), and second, to be able to guarantee for the function F n some curvature in a certain neighborhood of D 0 . More specifically, we shall assume that

• (Support recovery) For some η 0 ∈ (0, 1), it holds that

µ 0 ≤ 1 -η 0 2 -η 0 1 k .
Based on this assumption, we have for all J ⊆ 1; p with |J| ≤ k,

|||[D 0 ] ⊤ J c [D 0 ] J [D 0 ] ⊤ J [D 0 ] J -1 ||| ∞ ( * ) ≤ kµ 0 1 -kµ 0 ≤ 1 -η 0 , (6.7)
where the first inequality ( * ) is proved in Lemma 16, while the second inequality follows from the assumption. The condition displayed in Eq. (6.7) is well-known and is often referred to as the irrepresentability condition (e.g., see [START_REF] Fuchs | Recovery of exact sparse representations in the presence of bounded noise[END_REF][START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF]. As further discussed in Section 6.3.3, it is a central element to control the supports of the solutions of ℓ 1 -regularized least-squares problems. As a side comment, it worth noting that we impose the irrepresentability condition via a condition on the coherence, which is a stronger requirement (Van de Geer and Bühlmann, 2009). • (Curvature) We also need the following condition to hold

1 - k p 1 + pµ 0 1 -kµ 0 > 0, or equivalently, µ 0 < p -k 2pk . (6.8)
where the second inequality is equivalent to µ 0 ≤ 1-η 0 2-η 0 1 k . Note that the first inequality ( * ) is proved in Lemma 16.

• (Curvature) We also need the following condition to hold

1 - k p 1 + pµ 0 1 -kµ 0 > 0, or equivalently, µ 0 < p -k 2pk .
This second assumption is required to show that the second derivative of F n is positive in some neighborhood of D 0 . Lemma 11 makes this statement more formal and precise.

Main result and structure of the proof

In the main results from Section 6.3.1, we shall further simplify the assumptions above through numerical constants (for more details, see Assumption (1) in Theorem 5), so that k p 1+pµ 0 1-kµ 0 and terms depending only on kµ 0 will be regarded as universal constants. After having described the different components of our problem, we now present the core contribution of this chapter.

Main result and structure of the proof

This section contains the statement of the main result of this chapter (see Theorem 5 and Corollary 6) which shows that under appropriate scalings of (n, p, k, m, σ, µ 0 ), it is possible to prove that, with high probability, problem (6.3) admits a local minimum along some curves passing through D 0 . The proofs of the results of this section may be found in Appendix 6.6.

Main result

We begin with a complete statement of our result, before giving a simplified (but also easier to interpret) corollary:

Theorem 5 (Local minimum of sparse coding along curves) Consider n independent signals following the generative model from Section 6.2.2, with the reference dictionary D 0 ∈ R m×p . Denoting by N (n) log 2 (n)/ √ n and σ(t) max{σ; α|t|}, let us introduce the following set of assumptions for some t ∈ R:

(1) [Coherence]: kµ 0 ≤ min 1 2 ; p-2k 3p

(2) [Dictionary perturbation]:

t min |t| 1 k with        t min λp α 2 k max λ; α k p kµ 0 ; αN (n) t min p α 2 k (α 2 k + σ 2 m) λ 2 σ 2 (t) exp(-ζ λ 2 σ 2 (t) ) 1 2
for some universal constant ζ > 0

λ max λ; α k p kµ 0 ; αN (n) λ 2 m + λ √ m √ α 2 k + σ 2 m + α 2 k + σ 2 m α 4 k 2 p 2
(5) [Taylor-expansion perturbation]:

λ 2 σ 2 (t) exp(-ζ λ 2 σ 2 (t) ) α 6 k 3 p 3 1 (α 2 k+σ 2 m)(λ 2 m+λ √ m √ α 2 k+σ 2 m+α 2 k+σ 2 m) 2 (6) [Sample complexity]: N (n) min α 2 α 2 k p ; λ 2 σ 2 (t) exp(-ζ 2 λ 2 σ 2 (t)
) . Consider some W ∈ W D 0 and v 2 = 1 parameterizing D(u). Under Assumptions ( 1)-( 6), u → F n (D(u)) admits a local minimum in [-t, t] with probability exceeding 1c 0 /n, for some universal constant c 0 > 0.

Local Analysis of Sparse Coding in Presence of Noise

We discuss in the next section the main building blocks of this result and give a high-level structure of the proof.

The statement of Theorem 5 presents the precise conditions under which the existence of a local minimum for sparse coding can be proved to happen with high probability. We now derive a corollary that constitutes a simplified version of Theorem 5 and that makes it possible to better understand how the different quantities involved in our problem, such that the coherence µ 0 and the noise σ, are allowed to vary with (n, p, k, m). ; log(p) p β/2 . Consider some W ∈ W D 0 and v 2 = 1 parameterizing D(u). Under Assumptions (1)-( 6), u → F n (D(u)) admits a local minimum in [-1/p, 1/p] with probability exceeding 1c 0 /n, for some universal constant c 0 > 0.

Some comments and remarks are in order:

Assumption (1): While we hoped for a coherence scaling in O(1/k), the relationship between the first, second and third derivatives of Φ n (see Assumption (4) in Theorem 5) has led us to consider a coherence in O(1/ kp log(p)), which unfortunately brings into play the full size of the dictionary, as opposed to just the sparsity level as in the noiseless setting of [START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF]. We could improve this weakness of our analysis by refining the upper bound of the third derivative of Φ n (which is currently scaling in O(α 2 k)). Assumption ( 2): We observe a quite natural scaling of the noise which should decrease in O(1/p) as the number of atoms increases. The noiseless setting can be easily handled and leads to the same scalings as those displayed in Corollary 6. In fact, it suffices to follow the proof of Corollary 6, noticing that σ(t) becomes equal to α|t| when σ is set to zero. In both the noisy and noiseless settings, the perturbation of the reference dictionary is chosen to be in the order of |t| = O(1/p). Assumption (3): The choice of the regularization parameter is driven by the constraints imposed in Proposition 10 and the fact that we want the perturbation of the Taylor expansion-which is proportional to λ 2 σ 2 (t) exp(-ζ λ 2 σ 2 (t) ), to remain small. Assumption (4): Perhaps surprisingly, we need to impose a lower bound on the 6.3. Main result and structure of the proof sparsity level k. We do so in order to guarantee enough curvature for the function Φ n . In fact, the second derivative of t → Φ n (D(t)|sign(A 0 )) evaluated at t = 0 is proportional to k/p, as proved in Lemmas 10 and 11. Assumption ( 5): Our study relies on the exact recovery of the supports of {α i 0 } i∈ 1;n . This is only possible when the minimum magnitude of the signals α is bounded sufficiently away from zero, as quantify by the inequality log(p) p β-3 1/6 α α . Note that this inequality points out that it is likely to extend our result to the case of distributions of α 0 with some small enough mass at zero. Interestingly, and more generally, a question of interest is whether we can conduct a similar analysis without having to go through exact recovery, since, in the end, we essentially care about the dictionary and the function F n , regardless of the behavior of the learned coefficients α. Assumption ( 6): The sample complexity we obtain indicates we need in the order of O(log 2 (p)p 3 ) ≈ O(k 2 p 3 ) signals to be able to prove the existence of a local minimum, compared to O(kp 3 ) in [START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF]. We believe that by refining the concentration bounds, it is possible to replace the term N (n) = log 2 (n)/

√ n by log(n)/ √ n, which would thus lead to the same scaling as in [START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF].

We now look at a more detailed description of the proof of Theorem 5.

Architecture of the proof

The proof is based upon three building blocks that we now present. The main idea consists in first focusing on the study of the function t → f x (D(t)), before looking at the behavior of t → F n (D(t)) whose properties will stem from the concentration of the average of the independent random variables {f x i (D(t))} i∈ 1;n . We give below some details regarding the three steps we follow:

(1) Proving that the functions t → φ x (D(t)|sign(α 0 )) and t → f x (D(t)) coincide for suitable values of t ∈ T coincide ⊆ R, and that this event happens with high probability. The consequence of this point is that we can study a smooth function with an explicit representation instead of dealing with the intricate function f x . This step is fully described in Section 6.3.3. ( 2) Computing the second-order Taylor expansion of t → φ x (D(t)|sign(α 0 )) at t = 0, along with a uniform upper bound on the third derivative of this function. Again, the terms in this expansion and the upper bound should be viewed as random variables whose value need to be controlled with high probability. Importantly, since t → φ x (D(t)|sign(α 0 )) and t → f x (D(t)) may differ, we need to account for this event (even though it happens with low probability). This points results in a perturbed Taylor development for the function t → f x (D(t)), as analyzed in Lemma 7. Details of this second ingredient are given in Section 6.3.4. (3) Aggregating the results for n (independent) signals, based on the computation of step [START_REF] Varoquaux | Sparse Structured Dictionary Learning for Brain Resting-State Activity Modeling as assessed by the likelihood on unseen data, while exhibiting interpretable spatial components, that match known areas of interest in the brain[END_REF]. For all the random quantities, we proceed in the same way: we first show they are bounded with high probability, and conditionally to these events, we apply Hoeffding's and Bernstein's inequalities for a concentration around the expectation: We then have the expression of the perturbed Taylor development for the function t → F n (D(t)), which is valid with high probability.

The final conclusion follows by invoking Lemma 7 that guarantees the existence of a local minimum within a set T min ⊆ R. It finally remains to check that T min ⊆ T coincide . This third building block is presented in details in Section 6.3.5. The presence of noise in our analysis has led us to consider a structure of proof radically different from the schemes proposed in the related work [START_REF] Gribonval | Dictionary identification-sparse matrix-factorization via ℓ 1 -minimization[END_REF]; [START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF]. In particular, our formulation in problem (6.3) differs from that of [START_REF] Gribonval | Dictionary identification-sparse matrix-factorization via ℓ 1 -minimization[END_REF]; [START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF] where the ℓ 1 -norm of A is minimized over the equality constraint AD = X and the dictionary normalization D ∈ D. Optimality is then characterized through the linearization of the equality constraint, a technique that could not be easily extended to the noisy case.

The remainder of this section is dedicated to the description of the three key ingredients we referred to above.

Exact sign recovery for perturbed dictionaries

The objective of this section is to determine the conditions under which the two functions t → Φ n (D(t)|sign(A 0 )) and t → F n (D(t)) coincide. As briefly exposed in Section 6.2.1, it turns out that this question comes down to studying exact recovery for some ℓ 1 -regularized least-squares problems. Let us momentarily assume that

f x (D) = min α∈R p 1 2 x -Dα 2 2 + λ α 1 ,
has a solution α with support J, and that the matrix D ⊤ J D J is invertible. In this case, and according to Definition 6.4 of φ x , we have the equality

f x (D) = φ x (D|sign( α)).
Within our probabilistic model for the signal x = D 0 α 0 + ε, the sign vector sign( α) is a random variable that depends on both the coefficients α 0 and the noise ε, which is obviously difficult to control. It is therefore interesting to study when sign( α) = sign(α 0 ) holds.

Exact sign recovery in the problem associated with f x (D 0 ) has already been wellstudied (e.g., see [START_REF] Fuchs | Recovery of exact sparse representations in the presence of bounded noise[END_REF][START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF]. In particular, exact recovery is guaranteed to happen with high probability provided that (a) the dictionary D 0 satisfies the so-called irrepresentability condition-as assumed in Eq. (6.7), and that (b) the non-zero coefficients of α 0 are far enough from the noise level.

However, in our context, we need the same conclusion to hold not only at the dictionary D 0 , but also at D(t) = D 0 for some values of t ∈ R. We make this statement precise in the following proposition: Proposition 10 (Exact recovery for perturbed dictionaries) Let us consider some W ∈ W D 0 and some normalized vector v ∈ R p , v 2 = 1, parametrizing D(t). Assume that the signal x ∈ R m is generated according to the model from Section 6.2.2. Let η ∈ (0, η 0 ). Let T coincide ⊆ R be the set

T coincide t ∈ R; |t| ≤ min 1, 1 3 1 -η 2 -η - 1 -η 0 2 -η 0 1 k .
Define for any t ∈ T coincide the regularization parameter

λ ≥ 3 max{σ, α|t|} η 6 log(p) and threshold(λ) (2 √ 2 + 1)λ 1 -kµ(t)
.

Moreover, consider the noise condition threshold(λ) < α.

There exists a universal constant ζ > 0 such that for any t ∈ T coincide with probability exceeding

1 -8 exp -ζ λ 2 max{σ 2 , α 2 t 2 } , the vector α(t) ∈ R p defined by α(t) = [D(t)] ⊤ J [D(t)] J -1 [D(t)] ⊤ J x -λsign([α 0 ] J ) 0 ,
is the unique solution of min α∈R p [ 1 2 x -D(t)α 2 2 + λ α 1 ], and exact sign recovery holds, namely, sign( α(t)) = sign(α 0 ).

The structure of the proof closely follows that of [START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF], except that additional terms measuring the perturbation between D 0 and D(t) have to be controlled. Interestingly, this perturbation measured by the parameter |t| acts as a second source of noise, where the variance σ 2 is comparable to α 2 t 2 . In addition, the result from Proposition 10 indicates that the perturbation should remain small-that is, D(t) close to D 0 with |t| in the order of O(1/k)-if we want to guarantee exact recovery at the dictionary D(t). In the same vein, some precaution needs to be taken to prevent the non-zero coefficients of α 0 from being too small.

In the light of Definition 6.4, the previous proposition shows that for suitable values of t, the functions t → φ x (D(t)|sign(α 0 )) and t → f x (D(t)) coincide with probability greater than 1 -8 exp(-ζ λ 2 max{σ 2 ,α 2 t 2 } ); we therefore refer to this event as E coincide . In other words, with probability Pr(E coincide ), we can equivalently study the smooth function t → φ x (D(t)|sign(α 0 )) in lieu of t → f x (D(t)), which constitutes one of the building block of our approach.

Importantly, the event E coincide is only concerned with a single signal; when we consider a collection of n independent signals, we should instead study the event n i=1 E i coincide to guarantee that Φ n and F n do coincide. However, as the number of observations n becomes large, it is unrealistic and not possible to ensure that exact recovery will hold simultaneously-and with high probability-for the n signals. To get around this issue,

Local Analysis of Sparse Coding in Presence of Noise

we will exploit the fact that the quantities we need to control are averages over n random variables.

It is now natural to consider the local behavior of t → φ x (D(t)|sign(α 0 )) through its Taylor expansion around t = 0.

Computation of the Taylor expansion of

t → φ x (D(t)|sign(α 0 ))
We want to characterize some local minima of t → f x (D(t)) via the function t → φ x (D(t)|sign(α 0 )). The assumptions we make through the coherence µ 0 and the perturbation parameter t guarantee that for any support J ⊆ 1; k , the matrix [D(t)] ⊤ J [D(t)] J is invertible; under these conditions, and according to Definition 6.4, t → φ x (D(t)|sign(α 0 )) is a smooth function for which we can compute a Taylor expansion. As precisely described in Lemma 15 and Section 6.8, we then obtain an expression of the form

φ x (D(t)|sign(α 0 )) -φ x (D 0 |sign(α 0 )) + a x t + b x t 2 ≤ L x |t| 3 , (6.9)
which is valid for any t that preserves the smoothness of φ x . The quantities a x , b x and L x are random variables that are combinations of linear, quadratic and bilinear forms with respect to α 0 and/or ε. These random variables can be shown to concentrate around their expectation, as further discussed in Propositions 11, 12 and 13. On the event E coincide (see Section 6.3.3), the functions t → f x (D(t)) and t → φ x (D(t)|sign(α 0 )) coincide, so that the development in Eq. (6.9) directly applies for f x . When the previous property is not true anymore, i.e., on the event E c coincide , we use the fact that the difference between f x and φ x can always be upper bounded. More specifically, we can notice from the definitions of both f x and φ x that for any D ∈ R m×p and any s ∈ {-1, 0, 1} p such that φ x is properly defined, we have

f x (D) ≤ 1 2 x 2 2 and φ x (D|s) ≤ 1 2 x 2 2 .
We thus end up with the following perturbed Taylor expansion

f x (D(t)) -f x (D 0 ) + a x t + b x t 2 ≤ L x |t| 3 + r x , with r x = 21 E c coincide x 2
2 . (6.10) In order to prove the existence of a local minimum around D 0 , we will subsequently have to probabilistically control a upper bound of the gradient a x , a lower bound of the curvature b x , along with upper bounds on the third derivative L x and the perturbation r x .

Aggregation of n signals and existence of a local minimum

Starting from the expansion in Eq. (6.10) for n independent signals {x i } i∈ 1;n , the final step of the proof consists in concentrating the random variables

1 n n i=1 a x i , 1 n n i=1 b x i , 1 n n i=1 L x i and 1 n n i=1 r x i .
To this end, we make use of Hoeffding's and Bernstein's inequalities conditioned to an appropriate event where the random variables at stake are bounded. The discussion of the existence of a local minimum then follows from the next lemma: Let us define

t 0 max |a| b , 1 √ 1 -θ r b ,
and assume that for all t ∈ [-t 0 , t 0 ],

|f (t) -(f (0) + at + 2bt 2 )| ≤ L|t| 3 + r.
If the following condition holds

r < b 3 L 2 1 -θ ,
then f admits a local minimum in T min (-t 0 , t 0 ).

After enforcing the conditions required by Lemma 7, it remains to check a last inequality. Indeed, while Proposition 10 says we cannot go too far from D 0 to guarantee exact recovery, we need at the same time, and according to Lemma 7, to go far enough to ensure the existence of a local minimum. This results in testing the inclusion T min ⊆ T coincide , or equivalently, the inequality (with the notation from Lemma 7):

max |a| b , 1 √ 1 -θ r b ≤ |t| ≤ min 1, 1 3 1 -η 2 -η - 1 -η 0 2 -η 0 1 k .

Some experimental validations

In this section, we try to illustrate the results from Section 6.3.1 through some simulations. Although we do not manage to exactly highlight the scalings found in Corollary 6, our experiments still underline the main interesting trends put forward by our results.

In the setting we consider, we take p = 20, m = 10 and the reference dictionary D 0 has its entries generated i.i.d. according to a standard Gaussian distribution. Moreover, the sparsity level k is fixed to 4, while the non-zero coefficients of α 0 are uniformly drawn over the segment [-α, -α] ∪ [α, α], with α = 1 and α = 0.1. The level of noise is set according to Corollary 6, i.e., σ = α/p, whereas the number of signals is equal to n = 2500.

We study the behavior of the solutions of problem (6.3), by looking at the variations with respect to one parameter while keeping the other ones fixed (to the values mentioned above). This protocol leads to the series of results displayed in Figure 6.2, indexed by the letters (a) to (e).

We detail two important aspects of the experiment, namely, the choice of the regularization parameter λ, and how we deal with the invariances of problem (6.3) (see 6.3), versus some varying parameters. The curves represent the average error and standard deviation based on 10 runs, for random and fixed initializations. Details about the setting can be found in the text. details in the end of Section 6.2.2). On the one hand, since our analysis relies on exact recovery and we know in advance the correct sparsity level (i.e., the "oracle" value of k), we first tune λ over a logarithmic grid of values so that the learned α's match this sparsity level. Note that this tuning step is performed over an auxiliary set of signals.

On the other hand, we know that the dictionary D that we learn by minimizing problem (6.3) may differ from D 0 up to sign flips and atom permutations. To account for these possible transformations, we solve, as a post-processing step, the following problem min

s∈{-1,1} p Σ∈P D -D 0 ΣDiag(s) 2 F ,
where Σ belongs to the set of permutation matrices P. Let us denote by Σ(j) the result of the permutation of the index j by Σ. Since both D and D 0 have normalized atoms, the previous problem is equivalent to

max s∈{-1,1} p Σ∈P p j=1 s j [ dj ] ⊤ d Σ(j) 0 = max Σ∈P p j=1 [ dj ] ⊤ d Σ(j) 0
.

We recognize here an assignment problem based on the absolute correlation matrix D⊤ D 0 , which can be efficiently solved using the Hungarian algorithm [START_REF] Kuhn | The hungarian method for the assignment problem[END_REF].

To solve problem (6.3), we use the stochastic algorithm from Mairal et al. (2010a) 1 where the batch size is fixed to 512, while the number of epochs is chosen so as to pass over each signal 25 times (on average). We consider two types of initialization, one starting from a random dictionary, and the other one using the correct dictionary D 0 as a starting point.

Let us comment on Figure 6.2. The measure we consider in the experiments is the normalized Frobenius distance between D 0 and the (transformed) solution of problem (6.3). The results from Figure 6.2 show the error averaged over 10 runs, corresponding to 10 different initial dictionaries in the random initialization case. 2 Overall, it is interesting to see that the curves based on the random and fixed initialization roughly have the same behavior. Plot (a) shows that the number of signals is indeed beneficial to obtain a local minimum around D 0 . The scaling in O(p 3 ) advocated by Corollary 6 (in this case, about 8000 signals) seems reasonable, though slightly pessimisitc, in that the error does not decrease much and stabilizes from about 4000 signals.

Plot (b) shows that, when the coherence is small enough (µ 0 decreases as m increases with p fixed, and hence the problem becomes easier), the initialization does not seem to matter anymore. However, we still observe an increasing error since the number of signals remains fixed to 2500.

As for plot (c), it is interesting to see that the quality of the solution does not seem to be affected as α decreases. This contradicts Corollary 6 and confirms the fact that we should probably not go through exact recovery to prove our results. Moreover, notice that the case α = α = 1 corresponding to α 0 distributed on the hypercube appears as more complex, which our theory is not able to predict so far.

We now turn to plot (d). This simulation confirms that the noise is an important factor; we observe a sharp transition around values close to 1/k, which may indicate that our scaling in O(1/p) is too pessimistic.

Finally, the figure in plot (e) seems to point out that the sparser the signals, the better the learned dictionary. It may be interesting to better understand in future work why dense signals harm the identifiability of the dictionary. One potential explanation would be that as k increases, we need more signals, while the simulation kept n fixed.

Conclusion

We have conducted a non-asymptotic analysis of the local minima of sparse coding in the presence of noise, thus extending prior work which focused on noiseless settings [START_REF] Gribonval | Dictionary identification-sparse matrix-factorization via ℓ 1 -minimization[END_REF][START_REF] Geng | On the Local Correctness of L1 Minimization for Dictionary Learning[END_REF]. Within a probabilistic model of sparse signals, we have shown that a local minimum exists with high probability along some curves passing through the reference dictionary. The natural next step to undertake is to prove the existence of a local minimum in a neighborhood of the reference dictionary, i.e., within the entire manifold of normalized dictionaries, as opposed to just along single curves. Arguments based on (1) concentration of the measure uniformly over all such curves, and (2) the discretization of this set of curves through ǫ-nets (Massart, 2003) shall be useful to this end.

Our study can be further developed in multiple ways. On the one hand, while we have assumed deterministic coherence-based conditions scaling in O(1/k), it may interesting to consider non-deterministic assumptions [START_REF] Tropp | Norms of random submatrices and sparse approximation[END_REF][START_REF] Candès | Near-ideal model selection by l1 minimization[END_REF], which are likely to lead to improved scalings.

On the other hand, we may also use more realistic generative models for α 0 , for instance, spike and slab models [START_REF] Ishwaran | Spike and slab variable selection: frequentist and bayesian strategies[END_REF], or signals with compressible priors [START_REF] Cevher | Learning with compressible priors[END_REF]. Similarly, it may be interesting to look at misspecified models, e.g., with a certain fraction of non-sparse signals.

Moreover, a more ambitious question is about the possible characterization of a global minimum for sparse coding (modulo the inevitable invariances of the problem).

Finally, it is also of interest to study the case of sparse structured dictionary learning (e.g., see Jenatton et al., 2011c, and references therein), where the ℓ 1 -norm would be replaced by other sparsity-inducing norms capable of modelling classes of structured signals.

Proofs of the main results

This section contains the detailed proofs of the main results of the chapter.

Proof of Theorem 5

The proof is based on the different building blocks presented in Section 6.3.2. Note that the assumption of the theorem on kµ 0 makes it possible to upper bound the terms depending only on kµ 0 by universal constants.

To begin with, we consider the event E taylor over which we control the first-, second-, third-order and perturbation terms displayed in Eq. (6.10) for each of the n signals. These terms are respectively controlled in Propositions 11, 12, 13 and 14. Using the union bound and setting τ = 2 log(n), the probability of the event of interest is lower bounded by

Pr(E taylor ) ≥ 1 - y∈ c (0) ax ,c (0) bx ,c (0) Lx ,2 1 -1 - y n 2 n+1 ≥ 1 - c 0 n ,
for some universal constant c 0 > 0.

On the event E taylor , and according to Propositions 11, 13 and Lemmas 8, 12, we have that

1 n n i=1 a x i λ max λ; α k p kµ 0 ; αN (n) , and 1 n n i=1 L x i max{1, N (n)}(λ 2 m + λ √ m α 2 k + σ 2 m + α 2 k + σ 2 m).
Still on the event E taylor , and since the theorem assumes that N (n) α p , so that we obtain the following lower bound for 6.6. Proofs of the main results the curvature term (see Proposition 12 and Lemma 10):

1 n n i=1 b x i α 2 k p .
It now remains to handle the perturbation term. To this end, we apply Proposition 14 by considering the random variables {1 [E i coincide ] c x i 2 2 } i∈ 1;n . Combining the results from both Proposition 10 and Lemma 14, we have the following upper bounds of the expectation and the variance

E 1 [E i coincide ] c x i 2 2 (α 2 k + σ 2 m) λ 2 σ 2 (t) exp -ζ λ 2 σ 2 (t) E 1 [E i coincide ] c x i 4 2 (α 2 k + σ 2 m) 2 λ 4 σ 4 (t) exp -ζ λ 2 σ 2 (t)
,

where the universal constant ζ > 0 comes from Proposition 14. In addition, since we assume that the sample complexity

N (n) satisfies N (n) λ 2 σ 2 (t) exp(-ζ 2 λ 2 σ 2 (t)
), we can apply Proposition 14 by setting τ = log(n) and using for ς 2 the upper bound above, which leads to

1 n n i=1 r x i (α 2 k + σ 2 m) λ 2 σ 2 (t) exp -ζ λ 2 σ 2 (t)
.

With the previous elements in place, we can apply Lemma 7 which explicits the relationships that the first-, second-, third-order and perturbation terms must satisfy to guarantee the existence of a local minimum. Writing directly these relationships leads to the inequalities numbered (4) and (5) in the theorem. Note that the term θ in Lemma 7 can be guaranteed to be no greater than 1/2 by setting the multiplicative constant of λ small enough, so that terms in 1/(1θ) can be seen as universal constants.

The remaining conditions, that is, inequalities ( 2) and (3), result, on the one hand, from the constraints imposed on λ and t by Proposition 10, and, on the other hand, from the test T min ⊆ T coincide , as explained in Section 6.3.5.

Proof of Corollary 6

The proof of Corollary 6 mostly consists in checking that the proposed choices of scalings satisfy the conditions required by Theorem 5.

We first discuss the choice of λ = O(α log(p)/p). Let β be a scalar greater than 3. At the cost of having a more stringent constraint on the noise σ(t), we can always have the equality β log(p) = ζ λ 2 σ 2 (t) , which leads in turn to the advertised scaling of σ(t) and the choice |t| = O(1/p). Note that we will have to check that the conditions imposed by t min in Theorem 5 are valid with |t| = O(1/p). Moreover, since we constrain the sparsity level by log(p) α 4 α 4 k, we indeed satisfy condition (3) from Theorem 5. We now turn to the coherence condition (1) from Theorem 5: We need to check the inequality α 4 α 4 1 log(p)kp p -2k 3kp .

Local Analysis of Sparse Coding in Presence of Noise

To this end, it suffices to notice that α α ≤ 1 and k < √ kp < log(p)kp, along with the fact that for some universal constant z ∈ (0, 1), we have z 1 k ≤ p-2k 3kp . In the light of the previous points, and since we ask for log(p) m p 2 k, we obtain the following simplifications:

(α 2 k + σ 2 m) α 2 k (λ 2 m + λ √ m α 2 k + σ 2 m + α 2 k + σ 2 m) α 2 k λ 2 σ 2 (t) exp -ζ λ 2 σ 2 (t) log(p) p β λ α k p kµ 0 .
In addition, the constraint of the corollary on the sample complexity N (n) implies that

N (n) α 4 α 4 k p log(p) = O k p kµ 0 α 2 α 2 k p ,
as required by condition (6) in Theorem 5. Similarly, the inequality N (n)

λ 2 σ 2 (t) exp(-ζ 2 λ 2
σ 2 (t) ) from condition (6) in Theorem 5 exactly corresponds to condition (6) in the corollary where N (n) log(p)/p β/2 . At this stage of the proof, it remains to check conditions ( 2), ( 4) and ( 5

α 2 k (α 2 k + σ 2 m) λ 2 σ 2 (t) exp(-ζ λ 2 σ 2 (t) ) 1 2 p α 2 k α 2 k log(p) p β 1 2 α α log(p) p (β-1)/2 . Having α α √ log(p) p (β-1)/2 |t| = 1 p is then equivalent to log(p) p β-3
1/2 α α , which is true thanks to condition (5) in the corollary. Condition (4): We have

λ max λ; α k p kµ 0 ; αN (n) λ 2 m + λ √ m α 2 k + σ 2 m + α 2 k + σ 2 m λα k p kµ 0 α 2 k α 4 k 2 p 2 ,
as required by Theorem 5.

Proofs of the main results

Condition (5):

We first have

α 6 k 3 p 3 1 (α 2 k + σ 2 m)(λ 2 m + λ √ m √ α 2 k + σ 2 m + α 2 k + σ 2 m) 2 α 6 α 6 1 p 3 , so that it suffices to check that λ 2 σ 2 (t) exp -ζ λ 2 σ 2 (t) = log(p) p β α 6 α 6 1 p 3 ,
which is in turn guaranteed thanks to condition (5) in the corollary. We have examined all the conditions required by Theorem 5 for which the scalings proposed in the corollary apply; this concludes the proof.

Proof of Proposition 10

First and foremost, we look at the assumption made on the parameter t. For any t ∈ T coincide , we have

|t| ≤ 1 3 1 -η 2 -η - 1 -η 0 2 -η 0 1 k ≤ 1 3 1 -η 2 -η 1 k -µ 0 ,
which, by Eq. (6.6) and (6.7), leads to

µ(t) ≤ 1 -η 2 -η 1 k < 1 k .
This inequality shows that the matrix [D(t)] ⊤ J [D(t)] J is invertible for any J ⊆ 1; p with |J| ≤ k, and

|||[D(t)] ⊤ J c [D(t)] J [D(t)] ⊤ J [D(t)] J -1 ||| ∞ ≤ kµ(t) 1 -kµ(t) ≤ 1 -η.
The core of the proof relies on the control of four probabilistic events:

E 1 = ε ∈ R m ; [D(t)] ⊤ J ε ∞ ≤ √ 2λ , E 2 = ε ∈ R m ; [D(t)] ⊤ J c (I -[P(t)] J )ε ∞ ≤ λ η 3 , E 3 = α 0 ∈ R p ; [D(t)] ⊤ J c (I -[P(t)] J )[D 0 ] J [α 0 ] J ∞ ≤ λ η 3 , E 4 = α 0 ∈ R p ; [D(t)] ⊤ J ([D(t)] J -[D 0 ] J )[α 0 ] J ∞ ≤ √ 2λ .
First, notice that given our noise assumption, Lemma 18 shows that

[ α(t) -α 0 ] J ∞ ≤ threshold(λ) < α ≤ min j∈J |[α 0 ] j |,
which implies in turn that sign( α(t)) = sign(α 0 ). It remains to prove that α(t) is the unique solution of the Lasso program. To this end, we take advantage of Lemma 19.

Local Analysis of Sparse Coding in Presence of Noise

On the event 4 i=1 E i , we have

[D(t)] ⊤ J c (I -[P(t)] J )x ∞ + λ|||[D(t) J c ] ⊤ [D(t)] J [[D(t)] ⊤ J [D(t)] J ] -1 ||| ∞ which is first upper bounded by [D(t)] ⊤ J c (I -[P(t)] J )ε ∞ + [D(t)] ⊤ J c (I -[P(t)] J )[D 0 ] J [α 0 ] J ∞ + λ(1 -η)
, and then by

λ η 3 + λ η 3 + λ(1 -η) = λ(1 - η 3 ) < λ.
Putting together the pieces with sign( α(t)) = sign(α 0 ), Lemma 19 leads to the desired conclusion.

The remainder of the proof is dedicated to the control of the four events E i , i in 1; 4 . Note that we always proceed in the same way: conditionally on the draw of J, we compute uniform bounds, i.e., that depend only on k, leading to the conclusion without conditioning.

Event E 1 : Consider the centered Gaussian variable u j = ε ⊤ [d(t)] j for j in J. Since D(t) has unit ℓ 2 -norm columns, the variance of u j is σ 2 . Applying Lemma 6 along with the union bound, we have

Pr(E 1 ) ≥ 1 -2k exp - λ 2 σ 2 .
Event E 2 : We proceed similarly and consider u j = ε ⊤ (I -[P(t)] J )[d(t)] j for j in J c . Since D(t) is normalized and the spectral norm of a projector is bounded by one, the variance of u j is upper bounded by σ 2 . We obtain,

Pr(E 2 ) ≥ 1 -2(p -k) exp - η 2 18 λ 2 σ 2 .
Events E 3 and E 4 : For the treatment of these events, we simply invoke Lemma 17. Finally, by independence of α 0 and ε combined with the union bound, we have

Pr(∩ 4 i=1 E i ) = Pr(E 1 ∩E 2 )Pr(E 3 ∩E 4 ) ≥ 1-Pr([E 1 ] c )-Pr([E 2 ] c ) 1-Pr([E 3 ] c )-Pr([E 4 ] c ) .
Putting the different pieces together, the probability of exact recovery is lower bounded by

Pr(∩ 4 i=1 E i ) ≥ 1 -8 max{p -k, k} exp -min 1, η 2 18 , 1 2 , η 2 54 λ 2 max{σ 2 , α 2 t 2 } = 1 -8 exp - η 2 54 λ 2 max{σ 2 , α 2 t 2 } + log max{p -k, k} ,
so that there exists a universal constant ζ > 0 such that

Pr(∩ 4 i=1 E i ) ≥ 1 -8 exp -ζ λ 2 max{σ 2 , α 2 t 2 } .

Proof of Lemma 7

The function f is lower and upper bounded by the functions φ l and φ u defined as

φ l (t) = f (0) + at + 2bt 2 -L|t| 3 -r and φ u (t) = f (0) + at + 2bt 2 + L|t| 3 + r.
Let us first assume that a < 0. We recall that θ = |a|L/b 2 and it is assumed to belong to (0, 1). Observe that

φ l (|a|/b) -f (0) = -r + a |a| b + 2b |a| 2 b 2 -L |a| 3 b 3 = -r + L |a| 3 b 3 b 2 |a|L -1 = -r + |a| 2 b (1 -θ). As a result, if the condition r < |a| 2 b (1 -θ) is satisfied, we have φ l (|a|/b) > f (0). Let us now consider that r ≥ |a| 2 b (1 -θ) holds, or equivalently, t ′ 1 √ 1 -θ r b > |a| b . (6.11)
We study the sign of φ l (t ′ )f (0), that is,

φ l (t ′ ) -f (0) = -r + at ′ + 2b[t ′ ] 2 -L[t ′ ] 3 = r ν(1 -θ) -θ √ 1 -θ + (1 + θ)ν - 1 √ 1 -θ ν 2 = r ν(1 -θ) ψ θ (ν),
where we have introduced the quantity ν L √ r/b 3/2 . The second-order polynomial function ψ θ admits two distinct positive roots

u 1 = θ √ 1 -θ and u 2 = √ 1 -θ,
and for any u ∈ (u 1 , u 2 ), ψ θ (u) is strictly positive. Moreover, using Eq. (6.11), we already know that ν ≥ u 1 . It is then sufficient to notice that ν ≤ u 2 is equivalent to imposing the condition

r ≤ b 3 L 2 (1 -θ).
To summarize, we have proved so far that, under the conditions of the lemma, it holds that φ l (t 0 ) > f (0). Now, notice that φ l (-t 0 )φ l (t 0 ) = -2a t 0 , so that for any a < 0, we end up with the following chains of inequalities

f (0) < φ l (t 0 ) ≤ f (t 0 ) and f (0) < φ l (t 0 ) < φ l (-t 0 ) ≤ f (-t 0 ).
On the compact set [-t 0 ; t 0 ], since f is continuous, it has a global minimum. Moreover, given that f (0) < min{f (-t 0 ), f (t 0 )} and 0 ∈ [-t 0 ; t 0 ], this minimum lies in the interior of T min = (-t 0 ; t 0 ).

Eventually, the case a > 0 can be dealt with in a similar fashion.

Control of the Taylor expansion

This section of the appendix is dedicated to the intermediate results necessary to probabilistically control the Taylor expansion of the function t → φ x (D(t)|sign(α 0 )). We begin with the control of the first order term. Note that the concrete computation of the Taylor expansion is postponed to Section 6.8.

Concentration of the first-order terms

Lemma 8 (Upper bound of the first-order term a x ) There exist a universal constant c (0) ax and coefficients {c (j) ax } j∈ 1;3 depending only on kµ 0 such that for any τ ≥ 2,

Pr |a x | ≤ c (1) ax max{λ 2 , λα, λσ, σ 2 , σα}τ ≥ 1 -c (0) ax exp(-τ ),
and

|E[a x ]| ≤ c (2) ax λα k p kµ 0 + λ 2 c (3) ax .
Proof. The first part of the proof simply consists in gathering the concentration results of all the lemmas from Section 6.7.1, and applying the union bound. The second conclusion follows by putting together the non-vanishing expectation terms appearing in the lemmas from Section 6.7.1, and using the upper bound from Lemma 9.

Lemma 9 (Upper bound of the leading expectation term) For any W ∈ W D 0 and for any vector v with unit ℓ 2 -norm, we have

E Tr [D 0 ] ⊤ J [D 0 ] J -1 [D 0 ] ⊤ J [WDiag(v)] J ≤ k p kµ 0 1 -kµ 0 1 + kµ 0 .
Proof. By definition of W, we know that diag(W ⊤ D 0 ) = 0, and since Diag(v) is diagonal, it holds that diag(D ⊤ 0 WDiag(v)) = 0. As a consequence, we can freely subtract the identity matrix in the trace, and we have

Tr [D 0 ] ⊤ J [D 0 ] J -1 [D 0 ] ⊤ J [WDiag(v)] J = Tr I -[D 0 ] ⊤ J [D 0 ] J -1 [D 0 ] ⊤ J [WDiag(v)] J ≤ I -[D 0 ] ⊤ J [D 0 ] J -1 F [D 0 ] ⊤ J [WDiag(v)] J F ≤ kµ 0 1 -kµ 0 [D 0 ] ⊤ J [WDiag(v)] J F ≤ kµ 0 1 -kµ 0 |||[D 0 ] J [D 0 ] ⊤ J ||| 1/2 2 [WDiag(v)] J F ≤ kµ 0 1 -kµ 0 1 + kµ 0 [WDiag(v)] J F ,
where we have repeatedly used Lemma 16. Now, we have by definition of the sampling procedure and the normalization of both v and W,

E[ [WDiag(v)] J 2 F ] = E p j=1 δ(j)v 2 j w j 2 2 = k p .
The conclusion follows by Jensen's inequality.

a x i -E[a x ] ≤ c (1) ax max{λ 2 , λα, λσ, σ 2 , σα} τ 2 √ n ≥ 1 -c (0) ax exp(-τ ) n+1 .
Proof. The proof consists in conditioning for each independent signal x i to the event defined in Lemma 8 where the first order terms we want to concentrate are bounded. For any τ 2 ≥ 2, and with probability exceeding 1c (0) ax exp(-τ 2 ) n , we thus have for all

i ∈ 1; n |a x i | ≤ c (1)
ax max{λ 2 , λα, λσ, σ 2 , σα}τ 2 . We then consider the collection of independent, bounded variables {a x i } i∈ 1;n , and the conclusion follows by applying Hoeffding's inequality (see Lemma 23), that is,

Pr 1 n n i=1 a x i -E[a x ] ≤ c (1) ax max{λ 2 , λα, λσ, σ 2 , σα} τ 1 τ 2 √ n ≥ 1-2 exp(-τ 2 1 ) 1-c (0) ax exp(-τ 2 ) n ,
for any τ 1 ≥ 0 and τ 2 ≥ 2. We can further simplifying the right-hand side since for any

τ ≥ 1, 1 -2 exp(-τ 2 ) 1 -exp(-τ ) n ≥ 1 -2 exp(-τ ) n+1 .

Details of the concentration of the first-order terms

This section contains all the intermediate results necessary to concentrate the firstorder term of the Taylor expansion. In particular, we will have to make use of the concentration inequalities from Lemmas 22, 25 and 26. In any case, we need to control bilinear or quadratic forms that bring into play the Frobenius norm of a matrix. Upper bounds of this norm are obtained by exploiting the following facts

• According to Lemma 16, we have

[D ⊤ J D J ] -1 F ≤ 1/(1 -kµ 0 ) and |||D J D ⊤ J ||| 2 ≤ 1 + kµ.
• Projectors have their spectral norm bounded by one.

• The dictionary D 0 and the matrix W have unit ℓ 2 -norm columns, so that in combination with the fact that the vector v is also normalized, we have D 0 Diag(v) F = 1 and WDiag(v) F = 1.

Control of the Taylor expansion

such that for any τ ≥ 1,

Pr |b x | ≤ c (1) bx max{λ 2 , λα, λσ, σ 2 , σα, α 2 }τ ≥ 1 -c (0) bx exp(-τ ), and
α 2 k p 1 - k p 1 + pµ 0 1 -kµ 0 -λα c (2) bx -λ 2 c (3) bx ≤ E[b x ] ≤ α 2 k p + λ α c (2) bx + λ 2 c (3) bx .
Proof. The first part of the proof simply consists in gathering the concentration results of all the lemmas from Section 6.7.2, and applying the union bound. The second conclusion follows by putting together the non-vanishing expectation terms appearing in the lemmas from Section 6.7.2, and using the lower/upper bounds from Lemma 11.

Lemma 11 (Upper bound of the leading expectation term) Let P 0 be the orthogonal projector that projects onto the span of [D 0 ] J . For any W ∈ W D 0 and for any vector v with unit ℓ 2 -norm, we have

k p 1 - k p 1 + pµ 0 1 -kµ 0 ≤ E Tr [WDiag(v)] ⊤ J (I -P 0 )[WDiag(v)] J ≤ k p .
Proof. First, by definition of the sampling procedure and the normalization of both v and W, we have

E[ [WDiag(v)] J 2 F ] = E p j=1 δ(j)v 2 j w j 2 2 = k p .
As a result, the right-hand side is easily proved by upper bounding the spectral norm of I -P 0 by one. We now turn to the lower bound. We start with the following inequality

Tr [WDiag(v)] ⊤ J P 0 [WDiag(v)] J ≤ |||[D 0 ] ⊤ J [D 0 ] J ] -1 ||| 2 [D 0 ] ⊤ J [WDiag(v)] J 2 F ≤ 1 1 -kµ 0 [D 0 ] ⊤ J [WDiag(v)] J 2 F ,
where we have used Lemma 16. Developing the Frobenius norm with the property Diag(W ⊤ D 0 ) = 0, we obtain

[D 0 ] ⊤ J [WDiag(v)] J 2 F = p j=1 p i=1 i =j δ(j)δ(i)v 2 j ([w j ] ⊤ d i 0 ) 2 , so that E [D 0 ] ⊤ J [WDiag(v)] J 2 F = k(k -1) p(p -1) D ⊤ 0 WDiag(v) 2 F .
Since k(k-1) p(p-1) ≤ k 2 p 2 and WDiag(v) 2 F = 1, the expectation above is upper bounded by

E [D 0 ] ⊤ J [WDiag(v)] J 2 F ≤ k 2 p 2 |||D 0 D ⊤ 0 ||| 2 .
Applying Lemma 16, we have |||D 0 D ⊤ 0 ||| 2 ≤ 1+pµ 0 , and the advertised conclusion follows.

Local Analysis of Sparse

x i -E[b x ] ≤ c (1) bx max{λ 2 , λα, λσ, σ 2 , σα, α 2 } τ 2 √ n ≥ 1 -c (0) bx exp(-τ ) n+1 .
Proof. The proof essentially follows that of Proposition 11. It uses Lemma 10 in place of Lemma 8, which leads to

Pr 1 n n i=1 b x i -E[b x ] ≤ c (1) bx max{λ 2 , λα, λσ, σ 2 , σα, α 2 } τ 1 τ 2 √ n ≥ 1 -2 exp(-τ 2 1 ) 1 -c (0) bx exp(-τ 2 ) n ,
for any τ 1 ≥ 0 and τ 2 ≥ 2. We apply simplifications similar to those found in the proof of Proposition 11.

Details of the concentration of the second-order terms

This section contains all the intermediate results necessary to concentrate the secondorder term of the Taylor expansion. In particular, we will have to make use of the concentration inequalities from Lemmas 22, 25 and 26. In any case, we need to control bilinear or quadratic forms that bring into play the Frobenius norm of a matrix. Upper bounds of this norm are obtained by exploiting the following facts

• According to Lemma 16, we have

[D ⊤ J D J ] -1 F ≤ 1/(1 -kµ 0 ) and |||D J D ⊤ J ||| 2 ≤ 1 + kµ.
• Projectors have their spectral norm bounded by one.

• The dictionary D 0 and the matrix W have unit ℓ 2 -norm columns, so that in combination with the fact that the vector v is also normalized, we have D 0 Diag(v) F = 1 and WDiag(v) F = 1. The names of the next Lemma refer to the notation introduced in Lemma 15.

Lemma (Concentration of B αα )

Let P 0 be the orthogonal projector that projects onto the span of [D 0 ] J . There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, we have

B αα -E[([α 0 ] 1 ) 2 ] Tr([WDiag(v)] ⊤ J (I -P 0 )[WDiag(v)] J ) ≤ α 2 τ
with probability exceeding 1c 0 exp(-c 1 τ ).

Lemma (Concentration of B sα,1 )

For any τ ≥ 2, the term B sα,1 is upper bounded by 33λατ with probability exceeding 1 -4 exp(-τ ).

Local Analysis of Sparse Coding in Presence of Noise

with probability exceeding 1c 0 exp(-c 1 τ ).

Lemma (Concentration of B εs,4 )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term B εs,4 is upper bounded by λσ 1 (1kµ 0 ) 5/2 τ with probability exceeding 1c 0 exp(-c 1 τ ).

Lemma (Concentration of B ss )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term B ss is upper bounded by

12λ 2 √ 1 + kµ 0 (1 -kµ 0 ) 3 τ with probability exceeding 1 -c 0 exp(-c1τ ).

Lemma (Concentration of B εε,1 )

There exist universal constants c 0 , c 1 > 0 such that for any τ > 1, the term B εε,1 is upper bounded by

σ 2 1 1 -kµ 0 τ
with probability exceeding 1c 0 exp(-c 1 τ ).

Lemma (Concentration of B εε,2 )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term B εε,2 is upper bounded by

2σ 2 1 + kµ 0 (1 -kµ 0 ) 3/2 τ
with probability exceeding 1c 0 exp(-c 1 τ ).

Control of the third derivative

Proposition 13 (Upper bound on the third derivative of Φ n ) There exist a constant c Lx > 0 depending only on kµ 0 such that for any sign matrix S ∈ {-1, 0, 1} p×n and for all |t| ≤ 1 such that t → Φ n (D(t)|S) is three times differentiable, we have

1 n n i=1 ∇ 3 φ x i (D(t)|s i ) -E ∇ 3 φ x (D(t)|s) ≤ c Lx λ 2 m + λ √ m(α 2 k + σ 2 m) 1/2 √ τ + (α 2 k +σ 2 m)τ τ √ n ,
with probability exceeding [1 -2 exp(-τ )] n+1 , for any τ ≥ 1.

Control of the Taylor expansion

Proof. The proof essentially follows that of Proposition 11. It uses Lemma 12 in place of Lemma 8, which leads to

1 n n i=1 ∇ 3 φ x i (D(t)|s i ) -E ∇ 3 φ x (D(t)|s) ≤ c Lx λ 2 m + λ √ m(α 2 k + σ 2 m) 1/2 √ τ 2 + (α 2 k +σ 2 m)τ 2 τ 1 √ n , with probability exceeding [1 -2 exp(τ 2 1 )][1 -exp(-τ 2 )
] n , for any τ 1 ≥ 0 and any τ 2 ≥ 1. We apply simplifications similar to those found in the proof of Proposition 11.

Lemma 12 (Upper bound on the third derivative of φ x ) There exist a constant c Lx > 0 depending only on kµ 0 such that for any sign vector s ∈ {-1, 0, 1} p and for all |t| ≤ 1 with t → φ x (D(t)|s) three times differentiable, we have

|∇ 3 φ x (D(t)|s) | ≤ c Lx λ 2 m + λ √ m(α 2 k + σ 2 m) 1/2 √ τ + (α 2 k + σ 2 m)τ .
with probability exceeding 1exp(-τ ), for any τ ≥ 1. Moreover, we have

E ∇ 3 φ x (D(t)|s) ≤ c Lx λ 2 m + λ √ m(α 2 k + σ 2 m) 1/2 + (α 2 k + σ 2 m) .
Proof. The core of the proof consists of upper bounding the derivative of the three parts T xx , T sx and T ss composing φ x (see Section 6.8 for their complete expressions). We proceed by using Cauchy Schwartz's inequality, upper bounding the ℓ 2 -norm of sign vectors by √ m, and applying the bounds from Section 6.8, which leads to

|∇ 3 φ x (D(t)|s) | ≤ c Lx λ 2 m + λ √ m x 2 + x 2 2 ,
for some positive constant c Lx depending only on kµ 0 . Now, combined with Lemma 13, we obtain the first desired conclusion. Finally, since

E[ x 2 2 ] ≤ (α 2 k + σ 2 m),
we obtain by Jensen's inequality E[ x 2 ] ≤ (α 2 k + σ 2 m) 1/2 , and the second conclusion follows.

Control of the residual term

Proposition 14

Consider n independent draws {(α i 0 , ε i )} i∈ 1;n following the generative model from Section 6.2.2. Consider also n independent events {E i } i∈ 1;n defined on the same probability space as that of (α 0 , ε). Let us define n non-negative scalars {ς i } i∈ 1;n such that for any i ∈ 1; n , ς 2 i is greater than, or equal to, the variance of 1

E i x i 2 2 . Moreover, introduce ς 2 1 n n i=1 ς 2 i . For any τ ≥ 1, if we have τ 2 ≤ 3 20 ς (α 2 k + σ 2 m) √ n,

Local Analysis of Sparse Coding in Presence of Noise

then it holds that

Pr 1 n n i=1 1 E i x i 2 2 -E[1 E x 2 2 ] ≤ 2ς τ √ n ≥ 1 -2 exp(-τ ) n+1 .
Proof. The proof consists in conditioning to the event (see Lemma 13)

n i=1 (α i 0 , ε i ) ∈ R p × R m ; x i 2 2 ≤ 5(α 2 k + σ 2 m)τ ,
and then considering the collection of independent, zero-mean, bounded variables

{1 E i x i 2 2 - E[1 E x 2 2 ]} i∈ 1;n .
The conclusion follows by applying Bernstein's inequality (see Lemma 24), that is, for any τ 1 ≥ 0 and for any τ ≥ 1, if it holds that

τ 1 ≤ 3 2 ς 10(α 2 k+σ 2 m)τ √ n, we then have Pr 1 n n i=1 1 E i x i 2 2 -E[1 E x 2 2 ] ≤ 2ς τ 1 √ n ≥ 1 -2 exp(-τ 2 1 ) 1 -exp(-τ ) n .
Setting τ 1 = τ , and further simplifying the right-hand side since for any τ ≥ 1,

1 -2 exp(-τ 2 ) 1 -exp(-τ ) n ≥ 1 -2 exp(-τ ) n+1 ,
we obtain the desired conclusion.

Lemma 13 (Control of the ℓ 2 -norm of a signal) Let x be a signal following the generative model from Section 6.2.2. For any τ ≥ 1, we have Pr(

x 2 2 ≤ 5(α 2 k + σ 2 m)τ ) ≥ 1 -exp(-τ ).
Proof. The result is a direct application of Lemma 20. We recall that we have x = [D 0 ] J [α 0 ] J + ε, and that the norm of x can be expressed as follows

x 2 = [ α[D 0 ] J σI ] 1 α [α 0 ] J 1 σ ε 2 .
Lemma 14 (Truncated expectation of the ℓ 2 -norm of a signal) Let x be a signal following the generative model from Section 6.2.2. Consider an event E defined on the same probability space as that of (α 0 , ε). For any τ ≥ 2, we have

E[1 E x 2 2 ] ≤ 5(α 2 k + σ 2 m)τ Pr(E) + 9 exp(-τ ) .
Moreover, it holds that Proof. Let fix some τ ≥ 2. We introduce the event

E[1 E x 4 2 ] ≤ 25(α 2 k + σ 2 m)
K (α 0 , ε) ∈ R p × R m ; x 2 2 ≤ 5(α 2 k + σ 2 m)τ ,
and define l τ as the largest integer such that τ ∈ [l τ , l τ + 1). We can then "discretize" the event K c as

K c ⊆ ∞ l=lτ K c l , with K c l = (α 0 , ε) ∈ R p × R m ; x 2 2 5(α 2 k + σ 2 m) ∈ [l, l + 1) .
We now have

E[1 E x 2 2 ] = E[1 E∩K x 2 2 ] + E[1 E∩K c x 2 2 ] ≤ 5(α 2 k + σ 2 m)Pr(E)τ + ∞ l=lτ E[1 E∩K c l x 2 2 ] ≤ 5(α 2 k + σ 2 m) Pr(E)τ + ∞ l=lτ (l + 1)E[1 E∩K c l ] ≤ 5(α 2 k + σ 2 m) Pr(E)τ + ∞ l=lτ (l + 1)E[1 {(α 0 ,ε)∈R p ×R m ; x 2 2 ≥5(α 2 k+σ 2 m)l} ] .
Applying Lemma 13, we continue with the upper bound

E[1 E x 2 2 ] ≤ 5(α 2 k + σ 2 m) Pr(E)τ + ∞ l=lτ (l + 1) exp(-l) .
We recognize here the derivative of the Taylor series of u → u lτ +1 /(1u) for u ∈ (0, 1), which leads to

∞ l=lτ (l + 1)u l = l τ -l τ u + 1 (1 -u) 2 u lτ .
When setting u = 1/e, we obtain with τ ∈ [l τ , l τ + 1)

∞ l=lτ (l + 1) exp(-l) ≤ 2 l τ 1 -1/e exp(-l τ ) ≤ 2 eτ 1 -1/e exp(-τ ) ≤ 9τ exp(-τ ).
We thus reach the first advertised conclusion. The second result follows along the same lines, except that we end up with

E[1 E x 4 2 ] ≤ 25(α 2 k + σ 2 m) 2 Pr(E)τ 2 + ∞ l=lτ (l + 1) 2 exp(-l) .
As above, and noticing that (1 + l) 2 = (1 + l)(2 + l) -(1 + l), we can recognize the difference of the second and first derivatives of the Taylor series of u → u lτ +1 /(1u) for u ∈ (0, 1), which gives

∞ l=lτ (l + 1) 2 u l = l 2 τ (1 -u) 2 + 2l τ (1 -u) + u + 1 (1 -u) 3 u lτ .

Local Analysis of Sparse Coding in Presence of Noise

∇Σ(t) = Π ⊤ J -VSD ⊤ 0 D 0 C -CD ⊤ 0 D 0 SV + VCW ⊤ WSV + VSW ⊤ WCV -VSD ⊤ 0 WS + CD ⊤ 0 WCV -SW ⊤ D 0 SV + VCW ⊤ D 0 C Π J ∇ 2 Σ(t) = Π ⊤ J -V 2 CD ⊤ 0 D 0 C + 2VSD ⊤ 0 D 0 SV -CD ⊤ 0 D 0 CV 2 -V 2 SW ⊤ WS + 2VCW ⊤ WCV -SW ⊤ WSV 2 -V 2 CD ⊤ 0 WS -2VSD ⊤ 0 WCV -CD ⊤ 0 WSV 2 -SW ⊤ D 0 CV 2 -2VSW ⊤ D 0 CV -V 2 SW ⊤ D 0 C Π J ∇ 3 Σ(t) = Π ⊤ J V 3 SD ⊤ 0 D 0 C + 3V 2 CD ⊤ 0 D 0 SV + 3VSD ⊤ 0 D 0 CV 2 + CD ⊤ 0 D 0 SV 3 -V 2 CW ⊤ WS -3V 2 SW ⊤ WCV -3VCW ⊤ WSV 2 -SW ⊤ WCV 2 V 3 SD ⊤ 0 WS -3V 2 CD ⊤ 0 WCV + 3VSD ⊤ 0 WSV 2 -CD ⊤ 0 WCV 3 -V 3 CW ⊤ D 0 C + 3V 2 SW ⊤ D 0 SV + -3VCW ⊤ D 0 CV 2 + SW ⊤ D 0 SV 3 Π J and ∇[Σ(t) -1 ] = -Σ(t) -1 ∇Σ(t)Σ(t) -1 ∇ 2 [Σ(t) -1 ] = -∇[Σ(t) -1 ]∇Σ(t)Σ(t) -1 -Σ(t) -1 ∇ 2 Σ(t)Σ(t) -1 -Σ(t) -1 ∇Σ(t)∇[Σ(t) -1 ]
∇ 3 [Σ(t) -1 ] = -Σ(t) -1 ∇ 3 Σ(t)Σ(t) -1 -2∇[Σ(t) -1 ]∇ 2 Σ(t)Σ(t) -1 -2Σ(t) -1 ∇ 2 Σ(t)∇[Σ(t) -1 ] -2∇[Σ(t) -1 ]∇Σ(t)∇[Σ(t) -1 ] -∇ 2 [Σ(t) -1 ]∇Σ(t)Σ(t) -1 -Σ(t) -1 ∇Σ(t)∇ 2 [Σ(t) -1 ].

Derivatives of t → D(t).

We gather here the formula related to the derivatives of D(t): Derivatives of t → φ x (D(t)|s). The different derivatives of the three terms T xx , T sx and T ss can be computed by applying the product rule, based on the derivatives of D(t) and Σ(t) -1 (see above). We only give the formula for the first derivatives, namely

∇D(t)
∇T xx = x ⊤ (∇D(t)Π J Σ(t) -1 Π J D(t) ⊤ + D(t)Π J ∇[Σ(t) -1 ]Π J D(t) ⊤ +D(t)Π J Σ(t) -1 Π J [∇D(t)] ⊤ )x ∇T sx = λs ⊤ J (∇[Σ(t) -1 ]Π ⊤ J D(t) ⊤ + Σ(t) -1 Π ⊤ J [∇D(t)] ⊤ )x
∇T ss = λ 2 2 s ⊤ J ∇[Σ(t) -1 ]s J .

Computation of the Taylor expansion

We obtain the results advertised in Lemma 15 by evaluating the different derivatives at t = 0. Simplifications arise by making use of the following facts:

x = D 0 α 0 + ε

I = Π ⊤ J Π J α 0 = Π J Π ⊤ J α 0 P 0 D 0 Π J = D 0 Π J .

Some details about the upper bound of the third derivative

We need to upper bound quantities of the form D 0 CV h Π J F and D 0 SV h Π J F for h in 0; 3 . Thanks to the normalization of the vector v and the columns of D 0 , it easy to obtain upper bounds by constant terms when h ≥ 1. Moreover, since | sin(t)| ≤ |t|, it is also simple to derive upper bounds scaling in |t| as soon as the sine term (i.e., S) is present. In the case of D 0 CΠ J F , we can notice that for any matrix

A AD 0 CΠ J 2 F = Tr(Π ⊤ J C ⊤ D ⊤ 0 A ⊤ AD 0 CΠ J ) = Tr(AD 0 CΠ J Π ⊤ J C ⊤ D ⊤ 0 A ⊤ ) ≤ Tr(AD 0 Π J Π ⊤ J D ⊤ 0 A ⊤ ) ≤ |||D 0 Π J Π ⊤ J D ⊤ 0 ||| 2 A 2
∇[Σ(t) -1 ] F ≤ 8 √ 1 + kµ 0 (1 -kµ 0 ) 2 ∇ 2 [Σ(t) -1 ] F ≤ 144(1 + kµ 0 ) (1 -kµ 0 ) 3 ∇ 3 [Σ(t) -1 ] F ≤ 3872(1 + kµ 0 ) 3/2
(1kµ 0 ) 4 .

Local Analysis of Sparse Coding in Presence of Noise

Based on these upper bounds, we can further control the terms appearing in T sx and T xx , namely

∇ 3 [Σ(t) -1 Π ⊤ J D(t) ⊤ ] F ≤ 8 ∇ 3 [Σ(t) -1 ]Π ⊤ J D(t) ⊤ F ≤
30976(1 + kµ 0 ) 3/2 ((1 + kµ 0 ) 1/2 + 1) (1kµ 0 ) 4 , and

∇ 3 [D(t)Π J Σ(t) -1 Π ⊤ J D(t) ⊤ ] F ≤ 27 D(t)Π J ∇ 3 [Σ(t) -1 ]Π ⊤ J D(t) ⊤ F ≤ 104544(1 + kµ 0 ) 3/2 ((1 + kµ 0 ) 1/2 + 1) 2 (1 -kµ 0 ) 4 ,
where we use the fact there are respectively 8 and 27 terms appearing in ∇ 3 T sx and ∇ 3 T xx . Moreover, the upper bound by ((1 + kµ 0 ) 1/2 + 1) exploits that for any matrix A

D(t)Π

J A F ≤ D 0 CΠ J A F + WSΠ J A F ≤ Tr(A ⊤ Π ⊤ J C ⊤ D ⊤ 0 D 0 CΠ J A ⊤ ) 1/2 + WSΠ J F A F ≤ (|||Π ⊤ J CD ⊤ 0 D 0 CΠ J ||| 1/2 2 + 1) A F ≤ ((1 + kµ 0 ) 1/2 + 1) A F ,
where we can notice in the last inequality that the coherence of D 0 C is lower than that of D 0 , and we can therefore make use of Lemma 16.

Technical lemmas

The final section of this appendix gathers different technical lemmas required by the main results of the chapter.

Properties related to the mutual coherence

Lemma 16

Let D ∈ R m×p be a dictionary with coherence µ and normalized columns (i.e., with unit ℓ 2 -norm). For any J ⊆ 1; p with |J| ≤ k, We have

|||D J D ⊤ J ||| 2 = |||D ⊤ J D J ||| 2 ≤ 1 + kµ.
Similarly, it holds

|||D ⊤ J D J ||| ∞ ≤ 1 + kµ and |||D ⊤ J c D J ||| ∞ ≤ kµ.
Moreover, if we further have kµ < 1, then D ⊤ J D J is invertible and

max |||[D ⊤ J D J ] -1 ||| ∞ , |||[D ⊤ J D J ] -1 ||| 2 , [D ⊤ J D J ] -1 F ≤ 1 1 -kµ ,
6.9. Technical lemmas with, in addition,

max |||[D ⊤ J D J ] -1 -I||| ∞ , |||[D ⊤ J D J ] -1 -I||| 2 , [D ⊤ J D J ] -1 -I F ≤ kµ 1 -kµ .
Proof. These properties are already well-known (see, e.g. [START_REF] Fuchs | Recovery of exact sparse representations in the presence of bounded noise[END_REF]. We briefly prove them. First, we introduce H = D ⊤ J D J -I. A straightforward elementwise upper bound leads to H F ≤ |J|µ ≤ kµ. As a consequence, we have

|||D ⊤ J D J ||| 2 ≤ 1 + |||H||| 2 ≤ 1 + H F ≤ 1 + kµ.
By definition of |||.||| ∞ , we also have

|||D ⊤ J D J ||| ∞ ≤ 1 + |||H||| ∞ = 1 + max i∈J j∈J,j =i |[d i ] ⊤ d j | ≤ 1 + kµ.
Note that for |||D ⊤ J c D J ||| ∞ , there are no diagonal terms to take into account. Now, if kµ < 1 holds, then we have max{|||H||| ∞ , |||H||| 2 , H F } < 1 and there are convergent series expansion of [I + H] -1 in each of these norms [START_REF] Horn | Matrix analysis[END_REF]. By sub-multiplicativity, we obtain

[D ⊤ J D J ] -1 = ∞ t=0 (-1) t H t ≤ ∞ t=0 H t ≤ 1/(1 -kµ),
where . stands for one the three aforementioned matrix norms. The last result lies in the fact that [D ⊤ J D J ] -1 -I = ∞ t=1 (-1) t H t ≤ kµ/(1kµ).

6.9.2 Lemmas related to the study of problem (6.1)

J -[D 0 ] J )[α 0 ] J ∞ > τ ) ≤ 2k exp -τ 2 4α 2 t 2 .

Local Analysis of Sparse Coding in Presence of Noise

Proof. We focus on the first inequality. Consider for j ∈ J c the zero-mean sub-Gaussian variable (v j t) ≤ 3(v j t) 2 , which leads to j∈J tan 2 (v j t) ≤ 3 v 2 2 t 2 = 3t 2 . For the second inequality, we proceed similarly with u j [d(t) j ] ⊤ ([D(t)] J -[D 0 ] J )[α 0 ] J for j ∈ J. In this case, the variance of u j is upper bounded by where we have used the same arguments as above, along with the inequality 2(1cos(z)) ≤ z 2 . The conclusion results from the application of Lemma 6 along with the union bound over |J| = k terms.

E[u 2 j ] ≤ α 2 [d(t) j ] ⊤ ([D(t)] J -[D 0 ] J )([D(t)] J -[D 0 ] J ) ⊤ d(t) j = α 2 ([D(t)] J -[D 0 ] J ) ⊤ d(t) j 2

Lemma 18

Let J ⊆ 1; p and s ∈ {-1, 0, 1} |J| . Consider a dictionary D ∈ R m×p such that D ⊤ J D J is invertible. Consider also the vector α ∈ R p defined by

α = [D ⊤ J D J ] -1 [D ⊤ J x -λs] 0 ,
with x ∈ R m and λ a nonnegative scalar. If the vector x can be written as x = [D 0 ] J [α 0 ] J + ε for some (D 0 , α 0 , ε) ∈ R m×p × R p × R m , then we have

[α -α 0 ] J ∞ ≤ |||[D ⊤ J D J ] -1 ||| ∞ λ + D ⊤ J (D J -[D 0 ] J )[α 0 ] J ∞ + D ⊤ J ε ∞ .

Technical lemmas

Proof. The proof consists of simple algebraic manipulations. We first plug the expression of x into that of α and then use the triangle inequality for . ∞ , along with the definition and the sub-multiplicativity of |||.||| ∞ .

Lemma 19

Let x ∈ R m be a signal. Consider J ⊆ 1; p and a dictionary D ∈ R m×p such that D ⊤ J D J is invertible. Consider also a sign vector s ∈ {-1, 1} |J| and define α ∈ R p by

α = [D ⊤ J D J ] -1 [D ⊤ J x -λs] 0 ,
for some regularization parameter λ ≥ 0. Let us introduce the projector P J D J [D ⊤ J D J ] -1 D ⊤ J that projects onto the span of D J . If the following two conditions hold

   sign [D ⊤ J D J ] -1 [D ⊤ J x -λs] = s, D ⊤ J c (I -P J )x ∞ + λ|||D ⊤ J c D J [D ⊤ J D J ] -1 ||| ∞ < λ,
then α is the unique solution of min α∈R p [ 1 2 x-Dα 2 2 +λ α 1 ] and we have sign( αJ ) = s.

Proof. We first check that α is a solution of the Lasso program. It is well-known (e.g., see [START_REF] Fuchs | Recovery of exact sparse representations in the presence of bounded noise[END_REF][START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF] that this statement is equivalent to the existence of a subgradient z ∈ ∂ α 1 such that -D ⊤ (x -D α) + λz = 0, where z j = sign( αj ) if αj = 0, and |z j | ≤ 1 otherwise. We now build from s such a subgradient. Given the definition of α and the assumption made on its sign, we can take z J s. It now remains to find a subgradient on J c that agrees with the fact that αJ c = 0. More precisely, we define z J c by λz J c D ⊤ J c (x -D α) = D ⊤ J c (I -P J )x + λD ⊤ J c D J [D ⊤ J D J ] -1 s.

Using our assumption, we have z J c ∞ < 1. We have therefore proved that α is a solution of the Lasso program. The uniqueness comes from Lemma 1 in [START_REF] Wainwright | Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ 1 -constrained quadratic programming[END_REF].

Some useful concentration results

In this section, we first recall some known concentration results before proving more specific lemmas.

Compilation of some known results

Definition 6 (Sub-Gaussian variables, e.g., see [START_REF] Buldygin | Metric characterization of random variables and random processes[END_REF]; Vershynin (2010)) A zero-mean random variable z is sub-Gaussian with parameter σ > 0 if and only if for any t ≥ 0

E[exp(tz)] ≤ exp σ 2 t 2 2 .
where M off stands for the matrix M with its diagonal terms set to zero. By symmetry, the magnitude of y i is independent of its sign.

Conclusion

The main thread of this thesis is structured sparsity which we have studied from various angles, with statistical, algorithmic and applied considerations. We have first introduced structured sparsity-inducing norms. They are capable of encoding high-order structural information about the problem at hand, unlike standard sparsity-promoting penalties based on cardinality. We have precisely characterized what type of prior knowledge they can model, and have derived analysis for consistent structured variable selection, in both low-and high-dimensional settings.

Our second contribution lies in the use of structured sparsity-inducing norms within the framework of dictionary learning. We have shown how the nonzero patterns of the decompositions and/or dictionary elements can self-organize to best adapt to the considered class of signals. In addition, we have proposed an efficient and simple optimization tool built upon nested block-coordinate descent procedures. These points are eventually illustrated by an application to face recognition where we learn localized features more robust to occlusions.

The third section of the thesis is dedicated to convex optimization. It proves that problems regularized by structured sparsity-inducing norms can be efficiently solved by proximal methods. In particular, we have shown how the proximity operator can be computed exactly in various settings, based on dual formulations. This algorithmic development notably paves the way for numerous large-scale applications, ranging from topic models to background subtraction.

The fourth chapter discusses at greater length two applications of structured sparsity to neuroimaging, in both supervised and unsupervised settings. We have first considered the inter-subject prediction of sizes of objects from fMRI signals. A sparse hierarchical regularization is shown to well capture the multi-scale aspects of the data, while dealing properly with the inter-subject variability. Furthermore, we have studied brain restingstate time series where structured dictionary learning has proved to be a promising tool.

The last chapter of this thesis is slightly next to the scope of structured sparsity developed throughout the four previous chapters. It studies the local minima of the standard non-convex formulation of sparse coding. In particular, we have proposed a non-asymptotic analysis within a probabilistic model of sparse noisy signals, thus extending earlier work limited to noiseless settings and/or under-complete dictionaries. Our main result notably shows that, with high probability, a local minimum indeed exists along some curves passing through the reference dictionary generating the signals.

Conclusion

This thesis has explored several questions related to structured sparsity, hence providing with some answers, either full or partial. These answers raise in turn new challenges and new directions for future research which we now discuss.

Throughout this thesis, we make the key assumption that we have at disposal some structural prior knowledge that justifies the use of our norms. This point can of course be debatable. Even though we have seen that dictionary learning relaxes in some sense this constraint, it is rather rare to have an exact match between the structure we can actually model and what the real data dictate. This remark suggests that an important question is to be able to learn from the data the adequate structure. This is an obviously difficult question, notably due to the combinatorial and discrete nature of the problem. Further exploring submodular tools (Bach, 2010a,b) in this case seems an interesting avenue of research.

Moreover, this thesis focuses on the use of structured sparsity only via the regularization. It would also be of interest to consider structured data-fitting terms, for instance, to model specific forms of noise and outliers (Liu et al., 2010a).

On the applied side, the last few years have been witnessing an increasing number of applications of structured sparsity in various fields. Only recently, the domain of natural language processing (NLP) has been considered [START_REF] Martins | Structured sparsity in structured prediction[END_REF]. Further developments of structured sparsity in NLP seem natural since it is a field where a great deal of structural information is available, e.g., hierarchical in n-grams models. Moreover, the large-scale problems typical from NLP could benefit from recent stochastic extensions of proximal methods [START_REF] Duchi | Efficient online and batch learning using forward backward splitting[END_REF][START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF][START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF]. The contribution from this thesis regarding the computation of proximal operators for structured norms could be directly applied.

A Proofs

This section of the appendix gathers the proofs and some technical elements encountered throughout the manuscript.

A.1 Proofs and Technical Elements of Chapter 1 A.1.1 Proof of Proposition 5

We recall that L(w) = 1 n n i=1 ℓ y i , w ⊤ x i . Since w → Ω(w) is convex and goes to infinity when w 2 goes to infinity, and since L is lower bounded, by Weierstrass' theorem, the problem in (2.2) admits at least one global solution.

•First case: Q invertible. The Hessian of L is

1 n n i=1 x i x ⊤ i ∂ 2 ℓ ∂y ′ 2 (y i , w ⊤ x i ).
It is positive definite since Q is positive definite and min i∈{1,...,n} ∂ 2 ℓ ∂y ′2 (y i , w ⊤ x i ) > 0. So L is strictly convex. Consequently the objective function L+µΩ is strictly convex, hence the uniqueness of its minimizer.

•Second case: {1, . . . , p} belongs to G. We prove the uniqueness by contradiction. Assume that the problem in (2.2) admits two different solutions w and w. Then one of the two solutions is different from 0, say w = 0.

By convexity, it means that any point of the segment [w, w] = aw + (1a) w; a ∈ [0, 1] also minimizes the objective function L + µΩ. Since both L and µΩ are convex functions, it means that they are both linear when restricted to [w, w]. Now, µΩ is only linear on segments of the form [v, tv] with v ∈ R p and t > 0. So we necessarily have w = tw for some positive t. We now show that L is strictly convex on [w, tw], which will contradict that it is linear on [w, w]. Let E = Span(x 1 , . . . , x n ) and E ⊥ be the orthogonal of E in R p . The vector w can be decomposed in w = w ′ + w ′′ with w ′ ∈ E and w ′′ ∈ E ⊥ . Note that we have w ′ = 0 (since if it was equal to 0, w ′′ would be the minimizer of µΩ, which would imply w ′′ = 0 and contradict w = 0). We thus have (w ⊤ x 1 , . . . , w ⊤ x n ) = (w ′ ⊤ x 1 , . . . , w ′ ⊤ x n ) = 0.

This implies that the function s → ℓ(y i , sw ⊤ x i ) is a polynomial of degree 2. So it is not linear. This contradicts the existence of two different solutions, and concludes the proof of uniqueness.

A. Proofs

Remark 7

Still by using that a sum of convex functions is constant on a segment if and only if the functions are linear on this segment, the proof can be extended in order to replace the alternative assumption "{1, . . . , p} belongs to G" by the weaker but more involved assumption: for any (j, k) ∈ {1, . . . , p} 2 , there exists a group g ∈ G which contains both j and k.

A.1.2 Proof of Theorem 1

For w ∈ R p , we denote by Z(w) its zero pattern (i.e., the indices of zero-components of w). To prove the result, it suffices to prove that for any set I ⊂ {1, . . . , p} with I c / ∈ Z and |I| ≤ k -1, the probability of By construction, we have |J| ≤ k, and thus, by assumption, the matrix X J = (x 1 ) J , ..., (x n ) J ⊤ ∈ R n×|J| has rank |J|. As in the proof of Proposition A.1.1, this implies that the function L J is strictly convex, and then, the uniqueness of the minimizer of L J + µΩ, and also the uniqueness of the point at which the gradient of this function vanishes. So the equation f (w, ỹ) = 0 on w ∈ R J : w j = 0 for any j ∈ I has a unique solution, which we will write w ỹ J . On a small enough ball around (w ỹ J , ỹ), f is continuously differentiable since none of the norms vanishes at w ỹ J . Let (f j ) j∈J be the components of f and H JJ =

E I = Y ∈

A.1.4 Proof of Proposition 6

The proposition comes from a classic result of Fenchel Duality (Borwein and Lewis, 2006, Theorem 3.3.5 and Exercise 3.3.9) when we consider the convex function

h J : w J → λ 2 [Ω J (w J )] 2 ,
whose Fenchel conjugate h * J is given by κ J → 1 2λ [Ω * J (κ J )] 2 (Boyd and Vandenberghe, 2004, example 3.27). Since the set {w J ∈ R |J| ; h J (w J ) < ∞} ∩ {w J ∈ R |J| ; L J (w J ) < ∞ and L J is continuous at w J } is not empty, we get the first part of the proposition. Moreover, the primal-dual variables {w J , κ J } is optimal if and only if -κ J ∈ ∂L J (w J ),

κ J ∈ ∂[ λ 2
[Ω J (w J )] 2 ] = λΩ J (w J )∂Ω J (w J ), where ∂Ω J (w J ) denotes the subdifferential of Ω J at w J . The differentiability of L J at w J then gives ∂L J (w J ) = {∇L J (w J )}. It now remains to show that κ J ∈ λΩ J (w J )∂Ω J (w J ) (A.1) is equivalent to

w ⊤ J κ J = 1 λ [Ω * J (κ J )] 2 = λ [Ω J (w J )] 2 . (A.2)
As a starting point, the Fenchel-Young inequality (Borwein and Lewis, 2006, Proposition 3.3.4) gives the equivalence between (A.1) and

λ 2 [Ω J (w J )] 2 + 1 2λ [Ω * J (κ J )] 2 = w ⊤ J κ J . (A.3)
In addition, we have [START_REF] Rockafellar | Convex analysis[END_REF] ∂Ω J (w J ) = {u J ∈ R |J| ; u ⊤ J w J = Ω J (w J ) and Ω * J (u J ) ≤ 1}.

(A.4) Thus, if κ J ∈ λΩ J (w J )∂Ω J (w J ) then w ⊤ J κ J = λ [Ω J (w J )] 2 . Combined with (A.3), we obtain w ⊤ J κ J = 1 λ [Ω * J (κ J )] 2 .

A. Proofs so that we get for all u K\J ∈ R |K\J| , u ⊤ K\J ∇L(w * ) K\J + λΩ(w * )

g∈G K \G J ω g K\J ∞ u K\J 2 ≥ 0.
Finally, Proposition 6 gives λΩ(w * ) =λw * ⊤ ∇L(w * ) 1 2 , which leads to the desired result.

Proof of Proposition 8

The goal of the proof is to upper bound the dual norm Ω * (κ) by taking advantage of the structure of G; we first show how we can upper bound Ω * (κ) by (Ω c J ) * [κ J c ]. We indeed have:

Ω * (κ) = max g∈G J ω g •v 2 + g∈(G J ) c ω g •v 2 ≤1 v ⊤ κ ≤ max g∈G J ω g J •v J 2 + g∈(G J ) c ω g •v 2 ≤1 v ⊤ κ = max Ω J (v J )+(Ω c J )(v J c )≤1 v ⊤ κ = max {Ω * J (κ J ), (Ω c J ) * [κ J c ]}
, where in the last line, we use Lemma 33. Thus the duality gap is less than

1 2λ [Ω * (κ)] 2 -[Ω * J (κ J )] 2 ≤ 1 2λ max{0, [(Ω c J ) * [κ J c ]] 2 -[Ω * J (κ J )] 2 },
and a sufficient condition for the duality gap to be smaller than ε is

(Ω c J ) * [κ J c ] ≤ (2λε + [Ω * J (κ J )] 2 ) 1 2 .
Using Proposition 6, we have -λw ⊤ ∇L(w) = [Ω * J (κ J )] 2 and we get the right-hand side of Proposition 8. It now remains to upper bound (Ω c J ) * [κ J c ]. To this end, we call upon Lemma 29 to obtain:

(Ω c J ) * [κ J c ] ≤ max g∈(G J ) c    j∈g κ j h∈j,h∈(G J ) c ω h j 2    1 2
.

Among all groups g ∈ (G J ) c , the ones with the maximum values are the largest ones, i.e., those in the fringe groups F J = {g ∈ (G J ) c ; ∄g ′ ∈ (G J ) c , g ⊆ g ′ }. This argument leads to the result of Proposition 8.

A.1.6 Proof of Theorem 2

Necessary condition: We mostly follow the proof of [START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF]; Bach (2008b). Let ŵ ∈ R p be the unique solution of min w∈R p L(w) + µ Ω(w) = min w∈R p F (w).

This gives the necessary condition.

Sufficient condition:

We turn to the sufficient condition. We first consider the problem reduced to the hull J ⋆ , min

w∈R |J ⋆ | L ⋆ J (w J ⋆ ) + µΩ J (w J ⋆ ).
that is strongly convex since Q J ⋆ J ⋆ is positive definite and thus admits a unique solution ŵJ ⋆ . With similar arguments as the ones used in the necessary condition, we can show that ŵJ ⋆ tends in probability to the true vector w J ⋆ . We now consider the vector ŵ ∈ R p which is the vector ŵJ ⋆ padded with zeros on [J ⋆ ] c . Since, from Theorem 1, we almost surely have Hull({j ∈ 1; p , ŵj = 0}) = {j ∈ 1; p , ŵj = 0}, we have already that the vector ŵ consistently estimates the hull of w ⋆ and we have that ŵ tends in probability

A.1.7 Proof of Theorem 3

Since our analysis takes place in a finite-dimensional space, all the norms defined on this space are equivalent. Therefore, we introduce the equivalence parameters a(J ⋆ ), A(J ⋆ ) > 0 such that

∀u ∈ R |J ⋆ | , a(J ⋆ ) u 1 ≤ Ω J ⋆ [u] ≤ A(J ⋆ ) u 1 .
We similarly define a([J ⋆ ] c ), A([J ⋆ ] c ) > 0 for the norm (Ω c J ⋆ ) on R |[J ⋆ ] c | . In addition, we immediatly get by order-reversing:

∀u ∈ R |J ⋆ | , A(J ⋆ ) -1 u ∞ ≤ (Ω J ⋆ ) * [u] ≤ a(J ⋆ ) -1 u ∞ .
For any matrix Γ, we also introduce the operator norm Γ m,s defined as

Γ m,s = sup u s≤1 Γu m . A. Proofs Defining q [J ⋆ ] c |J ⋆ q [J ⋆ ] c -Q [J ⋆ ] c J ⋆ Q -1
J ⋆ J ⋆ q J ⋆ , we can write the gradient of L on [J ⋆ ] c as

∇L( ŵ) [J ⋆ ] c = -q [J ⋆ ] c |J ⋆ -µQ [J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ rJ ⋆ = -q [J ⋆ ] c |J ⋆ -µQ [J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ (r J ⋆ -r ⋆ J ⋆ ) -µQ [J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ r ⋆ J ⋆
, which leads us to control the difference rJ ⋆r ⋆ J ⋆ . Using Lemma 30, we get

rJ ⋆ -r ⋆ J ⋆ 1 ≤ ŵJ ⋆ -w ⋆ J ⋆ ∞ g∈G J ⋆ ω g J ⋆ 2 2 ω g •w 2 + g∈G J ⋆ ω g •ω g •w 2 1 ω g •w 2 3
, where w = t 0 ŵ + (1t 0 )w ⋆ for some t 0 ∈ (0, 1).

Let J = {k ∈ J ⋆ : w ⋆ k = 0} and let ϕ be defined as ϕ = sup

u∈R p :J⊂{k∈J ⋆ :u k =0}⊂J ⋆ g∈G J ⋆ ω g • ω g • u 1 ω g J • ω g J • u J 1 ≥ 1.
The term ϕ basically measures how close J ⋆ and J are, i.e., how relevant the prior encoded by G about the hull J ⋆ is. By using Eq. (A.8), we have

ω g • w 2 2 ≥ ω g J • w J 2 2 ≥ ω g J • ω g J • w J 1 ν 3 ≥ ω g • ω g • w 1 ν 3ϕ ,
along with

ω g • w 2 ≥ ω g J • w J 2 ≥ ω g J 2 ν 3 ≥ ω g J ⋆ 2 ν 3 √ ϕ and w ∞ ≤ 5 3 w ⋆ ∞ . Therefore we have rJ ⋆ -r ⋆ J ⋆ ∞ ≤ ŵJ ⋆ -w ⋆ J ⋆ ∞ g∈G J ⋆ ω g J ⋆ 2 2 ω g •w ∞ + 5ϕ ν w ⋆ ∞ ω g J ⋆ •ω g J ⋆ 1 ω g •w 2 ≤ A. Proofs
On the other hand, since Q has unit diagonal and Q

[J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ Q J ⋆ [J ⋆
] c has diagonal terms less than one, Q [J ⋆ ] c [J ⋆ ] c |J ⋆ also has diagonal terms less than one, which implies that Q

1/2 [J ⋆ ] c [J ⋆ ] c |J ⋆ ∞,2 ≤ 1.
Hence ψ is a Lipschitz function with Lipschitz constant upper bounded by σn -1/2 a([J ⋆ ] c ) -1 . Thus by concentration of Lipschitz functions of multivariate standard random variables (Massart, 2003, Theorem 3.4), we have for t > 0:

Pr (Ω c J ⋆ ) * [q [J ⋆ ] c |J ⋆ ] ≥ t+σn -1/2 Q 1/2 2 E [(Ω c J ⋆ ) * (v)] ≤ exp - nt 2 a([J ⋆ ] c ) 2 2σ 2 . Applied for t = µτ /2 ≥ 2σn -1/2 Q 1/2 2 E [(Ω c J ⋆ ) * (v)],
we get (because (u -1) 2 ≥ u 2 /4 for u ≥ 2):

Pr (Ω c J ⋆ ) * [q [J ⋆ ] c |J ⋆ ] ≥ t ≤ exp - nµ 2 τ 2 a([J ⋆ ] c ) 2 32σ 2 .
It finally remains to control the term Pr( q J ⋆ ∞ ≥ ξ), with

ξ = κν 3 min 1, 3τ κ 1/2 a([J ⋆ ] c ) 4αν .
We can apply classical inequalities for standard random variables (Massart, 2003, Theorem 3.4) that directly lead to

Pr( q J ⋆ ∞ ≥ ξ) ≤ 2|J ⋆ | exp - nξ 2 2σ 2 .
To conclude, Theorem 3 holds with .20) and

C 1 (G, J ⋆ ) = a([J ⋆ ] c ) 2 16 , (A.18) C 2 (G, J ⋆ ) = κν 3 min 1, τ κ 1/2 a([J ⋆ ] c )ν 24ϕ 3/2 w ⋆ ∞ g∈G J ⋆ ω g J ⋆ 2 2 , (A.19) C 3 (G, J ⋆ ) = 4 Q 1/2 2 E [(Ω c J ⋆ ) * (v)] , (A
C 4 (G, J ⋆ ) = κν 3A(J ⋆ ) min 1, τ κ 1/2 a([J ⋆ ] c )ν 24ϕ 3/2 w ⋆ ∞ g∈G J ⋆ ω g J ⋆ ∞
, where we recall the definitions: v a centered Gaussian random variable with unit covariance matrix, J = {j ∈ J ⋆ : w ⋆ j = 0}, ν = min{|w ⋆ j |; j ∈ J}, ϕ = sup

u∈R p :J⊂{k∈J ⋆ :u k =0}⊂J ⋆ g∈G J ⋆ ω g • ω g • u 1 ω g J • ω g J • u J 1 , κ = λ min (Q J ⋆ J ⋆ ) > 0 and τ > 0 such that (Ω c J ⋆ ) * [Q [J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ r ⋆ ] < 1 -τ .
A.1.8 A first order approach to solve Eq. 2.2 and Eq. 2.5

Both regularized minimization problems Eq. 2.2 and Eq. 2.5 (that just differ in the squaring of Ω) can be solved by using generic toolboxes for second-order cone programming (SOCP) [START_REF] Boyd | Convex Optimization[END_REF]. We propose here a first order approach that takes up ideas from [START_REF] Micchelli | Learning the kernel function via regularization[END_REF]; [START_REF] Rakotomamonjy | SimpleMKL[END_REF] and that is based on the following variational equalities: for x ∈ R p , we have where ζ j is defined as above. The reformulations Eq. A.21 and Eq. A.22 are jointly convex in {w, (η g ) g∈G } and lend themselves well to a simple alternating optimization scheme between w (for instance, w can be computed in closed-form when the square loss is used) and (η g ) g∈G (whose optimal value is always a closed-form solution). If the variables (η g ) g∈G ∈ R |G| + are bounded away from zero by a smoothing parameter, the convergence of this scheme is guaranteed by standard results about block coordinate descent procedures [START_REF] Bertsekas | Nonlinear programming[END_REF].

This first order approach is computationally appealing since it allows warm-restart, which can dramatically speed up the computation over regularization paths. Moreover, it does not make any assumptions on the nature of the family of groups G.

A.1.9 Technical lemmas

In this last section of the appendix, we give several technical lemmas. We consider I ⊆ 1; p and G I = {g ∈ G; g ∩ I = ∅} ⊆ G, i.e., the set of active groups when the variables I are selected.

A. Proofs

We begin with a dual formulation of Ω * obtained by conic duality [START_REF] Boyd | Convex Optimization[END_REF]: This primal problem is convex and satisfies Slater's conditions for generalized conic inequalities, which implies that strong duality holds [START_REF] Boyd | Convex Optimization[END_REF]. We now consider the Lagrangian L defined as such that for all g ∈ G I , ξ g j = 0 if j / ∈ G and ξ g I 2 ≤ τ g . The dual function is obtained by taking the derivatives of L with respect to the primal variables v I and (α g ) g∈G I and equating them to zero, which leads to ∀j ∈ I, u j + g∈G I ,G∋j ω g j ξ g j = 0 ∀g ∈ G I , γτ g = 0.

After simplifying the Lagrangian, the dual problem then reduces to min γ,(ξ g I ) g∈G I γ s.t. ∀j ∈ I, u j + g∈G I ,G∋j ω g j ξ g j = 0 and ξ g j = 0 if j / ∈ G, ∀g ∈ G I , ξ g I 2 ≤ γ, which is equivalent to the displayed result.

Since we cannot compute in closed-form the solution of the previous optimization problem, we focus on a different but closely related problem, i.e., when we replace the objective max g∈G I ξ g I 2 by max g∈G I ξ g I ∞ , to obtain a meaningful feasible point: , where we denote by j 0 an argmax of the latter maximization. We notably have for all g ∋ j 0 :

|ξ g j 0 | < |u j 0 | h∈j 0 ,h∈G I ω h j 0
.

By multiplying both sides by ω g j 0 and by summing over g ∋ j 0 , we get .

Proof. We simply plug the minimizer obtained in Lemma 28 into the problem of Lemma 27.

We now derive a lemma to control the difference of the gradient of Ω J evaluated in two points:

A. Proofs

Lemma 30 Let u J , v J be two nonzero vectors in R |J| . Let us consider the mapping w J → r(w J ) =

g∈G J ω g J •ω g J •w J ω g J •w J 2 ∈ R |J| .
There exists z J = t 0 u J + (1t 0 )v J for some t 0 ∈ (0, 1) such that

r(u J ) -r(v J ) 1 ≤ u J -v J ∞ g∈G J ω g J 2 2 ω g J •z J 2 + g∈G J ω g J •ω g J •z J 2 1 ω g J •z J 3 2
.

Proof. For j, k ∈ J, we have

∂r j ∂w k (w J ) = g∈G J (ω g j ) 2 ω g J • w J 2 I j=k - g∈G J (ω g j ) 2 w j ω g J • w J 3 2 (ω g k ) 2 w k ,
with I j=k = 1 if j = k and 0 otherwise. We then consider t ∈ [0, 1] → h j (t) = r j (tu J + (1t)v J ). The mapping h j being continuously differentiable, we can apply the mean-value theorem: there exists t 0 ∈ (0, 1) such that h j (1)h j (0) = ∂h j (t) ∂t (t 0 ).

We then have

|r j (u J ) -r j (v J )| ≤ k∈J ∂r j ∂w k (z) |u k -v k | ≤ u J -v J ∞   g∈G J (ω g j ) 2 ω g J • z J 2 + k∈J g∈G J (ω g j ) 2 |z j | ω g J • z J 3 2 (ω g k ) 2 |z k |   ,
which leads to

r(u J ) -r(v J ) 1 ≤ u J -v J ∞ g∈G J ω g J 2 2 ω g J •z J 2 + g∈G J ω g J •ω g J •z J 2 1 ω g J •z J 3 2
.

Given an active set J ⊆ 1; p and a direct parent K ∈ Π P (J) of J in the DAG of nonzero patterns, we have the following result:

Lemma 31 For all g ∈ G K \G J , we have K\J ⊆ g.

Proof. We proceed by contradiction. We assume there exists g 0 ∈ G K \G J such that K\J g 0 . Given that K ∈ P, there exists G ′ ⊆ G verifying K = g∈G ′ g c . Note that g 0 / ∈ G ′ since by definition g 0 ∩ K = ∅.

We can now build the pattern K = g∈G ′ ∪{g 0 } g c = K ∩ g c 0 that belongs to P. Moreover, K = K ∩ g c 0 ⊂ K since we assumed g c 0 ∩ K = ∅. In addition, we have that J ⊂ K and J ⊂ g c 0 because K ∈ Π P (J) and g 0 ∈ G K \G J . This results in J ⊂ K ⊂ K, which is impossible by definition of K.

Algorithm 10 Computation of the Proximal Operator for the Nonconvex Approach Inputs: u ∈ R p , a tree-structured set of groups G and g 0 (root of the tree).

Outputs: v (primal solution). Initialization: v ← u.

Call recursiveThresholding(g 0 ).

Procedure recursiveThresholding(g)

1: η ← min 0, -1 2 u root(g) 2 2 + λ + h∈children(g) recursiveThresholding(h) .

A.2.2 Proofs

Proof of Lemma 2

Proof. The proof relies on tools from conic duality [START_REF] Boyd | Convex Optimization[END_REF]. Let us introduce the cone C {(v, z) ∈ R p+1 ; v ≤ z} and its dual counterpart C * {(ξ, τ ) ∈ R p+1 ; ξ * ≤ τ }. These cones induce generalized inequalities for which Lagrangian duality also applies. We refer the interested readers to [START_REF] Boyd | Convex Optimization[END_REF] for further details.

We can rewrite problem (4. This primal problem is convex and satisfies Slater's conditions for generalized conic inequalities (i.e., existence of a feasible point in the interior of the domain), which implies that strong duality holds [START_REF] Boyd | Convex Optimization[END_REF]. We now consider the Lagrangian L defined as

L(v, z, τ , ξ) = 1 2 u -v 2 2 + λ g∈G ω g z g - g∈G z g v |g ⊤ τ g ξ g ,
with the dual variables τ = (τ g ) g∈G in R |G| , and ξ = (ξ g ) g∈G in R p×|G| , such that for all g ∈ G, ξ g j = 0 if j / ∈ g and (ξ g , τ g ) ∈ C * . The dual function is obtained by minimizing out the primal variables. To this end, we take the derivatives of L with respect to the primal variables v and z and set them to zero, which leads to vu -g∈G ξ g = 0 and ∀g ∈ G, λω gτ g = 0.

arg min y * ≤t w-y 2 is that the residual w-κ lies in the normal cone of the constraint set [START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF], that is, for all y such that y * ≤ t, (wκ) ⊤ (yκ) ≤ 0. The displayed result then follows from the definition of the dual norm, namely κ * = max z ≤1 z ⊤ κ.

Proof of Lemma 3

Proof. First, notice that the conclusion ξ h = Π . * ≤λωh (v |h + ξ h ) simply comes from the definition of ξ h and v, along with the fact that ξ g = ξ g |h since g ⊆ h. We now examine ξ g .

The proof mostly relies on the optimality conditions characterizing the projection onto a ball of the dual norm • * . Precisely, by Lemma 34, we need to show that either

ξ g = u |g -ξ h |g , if u |g -ξ h |g * ≤ t g , or
ξ g * = t g and ξ g⊤ (u |gξ h |gξ g ) = ξ g * u |gξ h |gξ g . Note that the feasibility of ξ g , i.e., ξ g * ≤ t g , holds by definition of κ g . Let us first assume that ξ g * < t g . We necessarily have that u |g also lies in the interior of the ball of . * with radius t g , and it holds that ξ g = u |g . Since g ⊆ h, we have that the vector u |hξ g = u |hu |g has only zero entries on g. As a result, ξ h g = 0 (or equivalently, ξ h |g = 0) and we obtain ξ g = u |g = u |gξ h |g , which is the desired conclusion. From now on, we assume that ξ g * = t g . It then remains to show that ξ g⊤ (u |gξ h |gξ g ) = ξ g * u |gξ h |gξ g . We now distinguish two cases, according to the norm used. ℓ 2 -norm: As a consequence of Lemma 34, the optimality condition reduces to the conditions for equality in the Cauchy-Schwartz inequality, i.e., when the vectors have same signs and are linearly dependent. Applying these conditions to individual projections we get that there exists ρ g , ρ h > 0 such that ρ g ξ g = u |gξ g and ρ h ξ h = u |hξ gξ h .

(A.26)

Note that the case ρ h = 0 leads to u |hξ gξ h = 0, and therefore u |gξ gξ h |g = 0 since g ⊆ h, which directly yields the result. The case ρ g = 0 implies u |gξ g = 0 and therefore ξ h |g = 0, yielding the result as well. Now, we can therefore assume ρ h > 0 and ρ g > 0. From the first equality of (A.26), we have ξ g = ξ g |g since (ρ g + 1)ξ g = u |g . Further using the fact that g ⊆ h in the second equality of (A.26), we obtain (ρ h + 1)ξ h |g = u |gξ g = ρ g ξ g .

Figure 1 . 1 :

 11 Figure 1.1: Comparison between soft-and hard-thresholding, respectively on the left and right figures.

  Figure 1.2: Comparison between the ℓ 2 -norm and ℓ 1 -norm balls in three dimensions, respectively on the left and right figures. The ℓ 1 -norm ball presents some singular points located along the axis of R 3 .

  Figure 1.3: Comparison between two mixed ℓ 1 /ℓ 2 -norm balls in three dimensions, without and with overlapping groups of variables, respectively on the left and right figures. The singular points appearing on these balls describe the sparsity-inducing behavior of the underlying norms Ω.

Figure 1

 1 Figure 1.4: (Left) The set of blue groups to penalize in order to select contiguous patterns in a sequence. (Right) In red, an example of such a nonzero pattern with its corresponding zero pattern (hatched area).

Figure 2 .

 2 Figure 2.3: The DAG for the set Z associated with the 2×2-grid. The members of Z are the complement of the areas hatched in black. The elements of G (i.e., the atoms of Z) are highlighted by bold edges.

Algorithm 1

 1 Backward procedure Input: Intersection-closed family of nonzero patterns P. Output: Set of groups G. Initialization: Compute Z = {P c ; P ∈ P} and set G = Z. Build the Hasse diagram for the poset (Z, ⊃). for t = min g∈Z |g| to max g∈Z |g| do for each node g ∈ Z such that |g| = t do if C∈Children(g) C = g then if (Parents(g) = ∅) then Connect children of g to parents of g. end if Remove g from G. Set of groups G = {g 1 , . . . , g M }.

Figure 2 . 5 :

 25 Figure2.5: Vertical and horizontal groups: (Left) the set of blue and green groups with their (not displayed) complements to penalize in order to select rectangles. (Right) In red, an example of nonzero pattern recovered in this setting, with its corresponding zero pattern (hatched area).

Figure 2 . 6 :

 26 Figure 2.6: Groups with ±π/4 orientations: (Left) the set of blue and green groups with their (not displayed) complements to penalize in order to select diamond-shaped patterns. (Right) In red, an example of nonzero pattern recovered in this setting, with its corresponding zero pattern (hatched area).

  Figure 2.9: Examples of generating patterns (the zero variables are represented in black, while the nonzero ones are in white): (Left column, in white) generating patterns that are used for the experiments on 400-dimensional sequences; those patterns all form the same hull of 24 variables, i.e., the contiguous sequence in (g). (Right column, in white)generating patterns that we use for the 20×20-dimensional grid experiments; again, those patterns all form the same hull of 24 variables, i.e., the diamond-shaped convex in (h). The positions of these generating patterns are randomly selected during the experiments. For the grid setting, the hull is defined based on the set of groups that are half-planes, with orientations that are multiple of π/4 (see Section 2.3.5).

  ISlasso -W3 (π/4) (d) Distance to the true pattern for the grid

Figure 2 .

 2 Figure 2.11: For different sample sizes, the probabilites of correct recovery and the (normalized) Hamming distance to the true nonzero patterns are displayed. In the grid case, two sets of groups G are considered, the rectangular groups with or without the ±π/4-groups (denoted by (π/4) in the legend). The points and the error bars on the curves respectively represent the mean and the standard deviation, based on 50 random settings {J ⋆ , w ⋆ , X, ε}.

Figure 2

 2 Figure 2.12: For the sample size n ∈ {250, 500, 1000}, we plot the prediction error versus the proportion of nonzero variables in the hull of the generating pattern. In the grid case, two sets of groups G are considered, the rectangular groups with or without the ±π/4-groups (denoted by (π/4) in the legend). The points, the lower and upper error bars on the curves respectively represent the median, the first and third quartile, based on 50 random settings {J ⋆ , w ⋆ , X, ε}.

  of variables) log(CPU time) (seconds)

Figure 3 . 2 :

 32 Figure 3.2: Top row: dictionary D = [d 1 , d 2 , d 3 ] ∈ R 400×3 used to generate the signals Eq. (3.6). From the second to the bottom row: dictionary elements recovered from 250 signals by PCA, SPCA and SSPCA (best seen in color).

Figure 3

 3 Figure 3.3: Top left, examples of faces in the datasets. The three remaining images represent learned dictionaries of faces with p = 36: NMF (top right), SSPCA (bottom left) and shared-SSPCA (bottom roght) (i.e., SSPCA with |M| = 12 different patterns of size 3). The dictionary elements are sorted in decreasing order of explained variance. While NMF gives sparse spatially unconstrained patterns, SSPCA finds convex areas that correspond to more natural face segments. SSPCA captures the left/right illuminations and retrieves pairs of symmetric patterns. Some displayed patterns do not seem to be convex, e.g., nonzero patterns located at two opposite corners of the grid. However, a closer look at these dictionary elements shows that convex shapes are indeed selected, and that small numerical values (just as regularizing by ℓ 2 norm may lead to) give the visual impression of having zeroes in convex nonzero patterns. This also shows that if a nonconvex pattern has to be selected, it will be, by considering its convex hull.

Figure 3

 3 Figure 3.4: Classification accuracy versus dictionary size: each dimensionality reduction technique is used with k-NN to classify occluded faces. SSPCA shows better robustness to occlusions. The points, lower and upper error bars on the curves respectively represent the median, first and third quartile, based on 10 runs.

Figure 4 . 1 :

 41 Figure 4.1: Example of a tree T when p = 6. With the rule we consider for the nonzero patterns, if we have α 5 = 0, we must also have α k = 0 for k in ancestors(5) = {1, 3, 5}.

  Figure4.3: One pass convergence: the cost function of the primal proximal problem and its dual counterpart are monitored with respect to the number of group updates in Algorithm 6. In this setting, G corresponds to a complete binary tree of depth 4, with a total of p = |G| = 31 nodes. With the correct order, one pass is sufficient to reach the exact solution (note that log 10 (31) ≈ 1.49). For the random orders of G, we display the average of the cost functions based on 20 different orders.

97 4 .

 4 Proximal Methods for Structured Sparsity-Inducing Norms

  scale: small, regul.: high

Figure 4 . 4 :

 44 Figure 4.4: Benchmark for solving a least-squares regression problem regularized by the hierarchical norm Ω. The experiment is small scale, m = 256, p = 151, and shows the performances of six optimization methods (see main text for details) for three levels of regularization. The curves represent the relative value of the objective to the optimal value as a function of the computational time in second on a log 10 / log 10 scale. All reported results are obtained by averaging 5 runs.

Figure 4 . 5 :

 45 Figure4.5: Benchmark for solving a large-scale multi-class classification problem for four optimization methods (see details about the datasets and the methods in the main text). Three levels of regularization are considered. The curves represent the relative value of the objective to the optimal value as a function of the computational time in second on a log 10 / log 10 scale. In the highly regularized setting, tuning the step-size for the subgradient turned out to be difficult, which explains the behavior of SG in the first iterations.

  19

( a )

 a Lena, σ = 25, ℓ1 (b) Lena, σ = 25, Ω ℓ 2 (c) Barb., σ = 50, ℓ1 (d) Barb., σ = 50, Ω ℓ 2

Figure 4 . 6 :

 46 Figure 4.6: Visual comparison between the wavelet shrinkage model with the ℓ 1 -norm and the tree-structured model, on cropped versions of the images Lena and Barb.. Haar wavelets are used.

Figure 4

 4 Figure4.7: Mean square error multiplied by 100 obtained with 13 structures with error bars, sorted by number of dictionary elements from 16 to 401. Red plain bars represents the tree-structured dictionaries. White bars correspond to the flat dictionary model containing the same number of dictionary as the tree-structured one. For readability purpose, the y-axis of the graph starts at 50.

Figure 4 . 8 :

 48 Figure 4.8: Learned dictionary with tree structure of depth 4. The root of the tree is in the middle of the figure. The branching factors are p 1 = 10, p 2 = 2, p 3 = 2. The dictionary is learned on 50, 000 patches of size 16 × 16 pixels.

Figure 4 . 9 :

 49 Figure 4.9: Learned dictionary with a tree structure of depth 5. The root of the tree is in the middle of the figure. The branching factors are p 1 = 10, p 2 = 2, p 3 = 2, p 4 = 2.The dictionary is learned on 50, 000 patches of size 16 × 16 pixels.

Figure 4 .

 4 Figure 4.10: Example of a topic hierarchy estimated from 1714 NIPS proceedings papers(from 1988 through 1999). Each node corresponds to a topic whose 5 most important words are displayed. Single characters such as n, t, r are part of the vocabulary and often appear in NIPS papers, and their place in the hierarchy is semantically relevant to children topics.

Figure 4 .

 4 Figure 4.11: Binary classification of two newsgroups: classification accuracy for different dimensionality reduction techniques coupled with a linear SVM classifier. The bars and the errors are respectively the mean and the standard deviation, based on 10 random splits of the dataset. Best seen in color.

  Figure4.13: Explained variance of the CUR decompositions obtained for our sparsitybased approach and the sampling scheme from[START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF]. For the latter, we report the average and standard deviation of the results based on five initializations. From left to right and top to bottom, the curves correspond to the datasets 9_Tumors, Brain_Tumors1, Leukemia1 and SRBCT.

Figure 4 .

 4 Figure4.14: The original image y (line 1), the background (i.e., Xw) reconstructed by our method (line 2), and the foreground (i.e., the sparsity pattern of e as a mask on the original image) detected with ℓ 1 (line 3) and with ℓ 1 + Ω (line 4). The bottom line is another foreground found with Ω, on a different image, with the same values of λ 1 , λ 2 as for the previous image. For the top left image, the percentage of pixels matching the ground truth is 98.8% with Ω, 87.0% without. As for the top right image, the result is 93.8% with Ω, 90.4% without (best seen in color).

  (w,b)∈R p+1 L(y, X, w, b) + λΩ(w) with λ ≥ 0, (5.1)

Figure 5 . 2 :

 52 Figure 5.2: Weights estimated in the simulation study. The true coefficients are presented in a) and the estimated weights at different scales, i.e., different depths in the tree, are presented in b)-h). The results are best seen in color.

  Figure5.3: Maps of weights obtained using different regularizations in the regression setting. (a) ℓ 1 regularization -We can notice that the predictive pattern obtained is excessively sparse, and is not easily readable despite being mainly located in the occipital cortex. (b-d) tree ℓ 2 regularization (ρ = 1) at different scales -In this case, the regularization algorithm extracts a pattern of voxels with a compact structure, that clearly outlines early visual cortex which is expected to discriminate between stimuli of different sizes. 3D images were generated with Mayavi[START_REF] Ramachandran | Mayavi: 3d visualization of scientific data[END_REF].

Figure 5

 5 Figure 5.5: (left) Cross-validation scores: Average likelihood on unseen data obtained for SSPCA, SPCA and PCA/ICA over the 6 folds, with different dictionary sizes. The likelihood corresponds here to L normalized by 10 4 . No error bars are displayed on this figure, the statistical significance of the results being assessed in Table 5.3. (right) Outline of the regions defined by the different dictionary elements for p = 100 displayed on 2D cuts, as well as a 3D view with a representation of the cortical folds. The outlines drawn correspond to 25% of the maximum value of the dictionary elements. A poorlystructured pink region stands out outside the upper frontal regions: it is a well known movement-pattern often extracted in ICA analysis (map 9 on Figure 2 of Varoquaux et al. (2010e)).

Figure 5

 5 Figure5.6: (left) Dictionary element estimated by ICA, corresponding to the brain network known as the default mode network. As it is usually done in neuroimaging, the spatial map learned by decomposition of the functional images is displayed, thresholded, on an anatomical brain image. (right) Three dictionary elements corresponding to the same brain regions. The different colormaps (red, green and blue) correspond to the different dictionary elements learned. The transparency reflects the structured sparsity, rather than an arbitrary threshold 9 . This is why the different elements displayed on the figure have a block-like outline that reflect the shape of the groups used in norm Ω. Some displayed patterns do not seem to be rectangular boxes (e.g., the green patterns). However, a closer look at these dictionary elements shows that boxes are indeed selected, and that small numerical values (just as regularizing by ℓ 2 -norm may lead to) give the visual impression of having zeroes between the two green areas.

  , we have D(t) ∈ D for all t ∈ R and D(0) = D 0 . Moreover, in the notation of our parametrization, the set of matrices given by M ∈ R m×p ; M = ∂D(t) ∂t t=0 WDiag(v), with W ∈ W D 0 and v 2 = 1 corresponds to the tangent space of D at D 0 (Absil et al., 2008), intersected with the set of matrices in R m×p with unit Frobenius norm (indeed, we have WDiag(v) F = 1). Note that the parameter v is essential in that the set W D 0 does not contain every direction from the tangent space of D at D 0 , as illustrated in the example below: Example: Why is the parameter v important? Consider the dictionary D 0 = [d 1 0 , d 2 0 ] with m = p = 2 and the following choices of atoms

  and (w 2 ) ⊤ d 2 0 = 0 .

Figure 6

 6 Figure 6.1: Illustration of the parametrization of the set D of dictionaries with unit ℓ 2norm atoms. The dictionary D(t) is a curve of D passing through the reference dictionary D 0 (in red). The curve is parametrized by W ∈ W D 0 and v, so that WDiag(v) defines a direction of the tangent space at D 0 (in gray).

( 3 )

 3 [Regularization]: σ(t) log(p) λ α min 1; α α k p (4) [First/second/third-derivative conditions]:

Corollary 6 (

 6 Local minimum of sparse coding, specific scaling) Consider n independent signals following the generative model from Section 6.2.2, with the reference dictionary D 0 ∈ R m×p . Denoting by N (n) log 2 (n)/ √ n, let us introduce the following set of assumptions for any β > 3:

6. 4 .

 4 Some experimental validations Lemma 7 (Local minimum from perturbed Taylor expansion) Let f be a continuous real-valued function. Assume there exist (a, b, L, r) such that |a|, b, L and r are strictly positive, with θ |a|L b 2 ∈ (0, 1).

Figure 6 . 2 :

 62 Figure6.2: Estimation error, i.e., the normalized Frobenius distance between D 0 and the solution of problem (6.3), versus some varying parameters. The curves represent the average error and standard deviation based on 10 runs, for random and fixed initializations. Details about the setting can be found in the text.

2

 2

  = [-D 0 Diag(sin(vt)) + WDiag(cos(vt))]Diag(v) ∇ 2 D(t) = -D(t)Diag(v) 2 ∇ 3 D(t) = [D 0 Diag(sin(vt)) -WDiag(cos(vt))]Diag(v) 3 .

F≤ 2 F

 2 (1 + kµ 0 ) A In the light of the remarks above, a little calculation leads to the following upper bounds:∇Σ(t) F ≤ 2( 1 + kµ 0 + |t| 1 + kµ 0 + |t| + t 2 ) |t|≤1 ≤ 8 1 + kµ 0 ∇ 2 Σ(t) F ≤ 2( 1 + kµ 0 + 1 + |t| 1 + kµ 0 + 3|t| + 2t 2 ) |t|≤1 ≤ 16 1 + kµ 0 ∇ 3 Σ(t) F ≤ 2( 1 + kµ 0 + 3 + |t| 1 + kµ 0 + 7|t| + 4t 2 ) |t|≤1 ≤ 32 1 + kµ 0 ,and for |t| ≤ 1,

2 ≤ α 2 2 ≤ α 2 2 ≤ α 2 W 2 F = α 2 j∈J tan 2

 222222222 u j [d(t) j ] ⊤ (I -[P(t)] J )[D 0 ] J [α 0 ] J . The variance of u j is upper bounded by E[u 2 j ] ≤ α 2 [d(t) j ] ⊤ (I -[P(t)] J )[D 0 ] J [D 0 ] ⊤ J (I -[P(t)] J )d(t) j = α 2 [D 0 ] ⊤ J (I -[P(t)] J )d(t) j 2 Diag(tan(v J t))W ⊤ J (I -[P(t)] J )d(t) j 2 |||W J Diag(tan(v J t))||| 2 J Diag(tan(v J t)) (v j t),where the first inequality exploits the definition of D 0 , i.e., D 0 = Diag(cos(vt)) -1 D(t) -WDiag(tan(vt)), combined with the relation (I -[P(t)] J )[D(t)] J = 0, while the other upper bounds use the normalization of d(t) and W J , along with the fact that projectors have their spectral norm bounded by one. The final result follows by applying Lemma 6 and the union bound over |J c | = pk terms. Moreover, if |t| ≤ 1, we have |v j t| ≤ 1 and tan 2

2 ≤ α 2 2 ≤ α 2 2 ≤ 2 F = 2α 2 j∈J(

 2222222 |||[D(t)] J -[D 0 ] J ||| 2 |||[D 0 ] J (Diag(cos(v J t)) -I) + W J Diag(sin(v J t))||| 2 2α 2 [D 0 ] J (Diag(cos(v J t)) -I) 2 F + W J Diag(sin(v J t)) cos(v j t) -1) 2 + sin 2 (v j t) = 2α 2 j∈J 2(1cos(v j t)) ≤ 2α 2 j∈J (v j t) 2 ≤ 2α 2 t 2

.

  R n : there exists w solution of the problem in(2.2) with Z(w) = I c is equal to 0. We will prove this by contradiction: assume that there exists a set I ⊂ {1, . . . , p} with I c / ∈ Z, |I| ≤ k -1 and Pr(E I ) > 0. Since I c / ∈ Z, there exists α ∈ Hull(I) \ I. Let J = I ∪ {α} and G I = {g ∈ G : g ∩ I = ∅} be the set of active groups. Define R J = {w ∈ R p : w J c = 0}. The restrictions L J : R J → R and Ω J : R J → R of L and Ω are continuously differentiable functions on w ∈ R J : w I = 0 with respective gradients∇L J Let f (w, Y ) = ∇L J (w) + µ∇Ω J (w), where the dependence in Y of f (w, Y ) is hidden in ∇L J (w) = 1 n n i=1 (x i ) J ∂ℓ ∂y ′ (y i , w ⊤ x i ).For Y ∈ E I , there exists w ∈ R J with Z(w) = I c , which minimizes the convex function L J + µΩ J . The vector w satisfies f (w, Y ) = 0. So we have proved E I ⊂ E ′ I , whereE ′ I = Y ∈ R n :there exists w ∈ R J with Z(w) = I c and f (w, Y ) = 0 . Let ỹ ∈ E I . Consider the equation f (w, ỹ) = 0 on w ∈ R J : w j = 0 for any j ∈ I .

  ∂f j ∂w k j∈J,k∈J . The matrix H JJ is actually the sum of:A. ProofsAs for the second point, we proceed by contradiction. If there exists another baseG * that spans Z such that G * ⊂ G, then ∃ e ∈ G, e / ∈ G * .By definition of the set Z, there exists in turn G ′ ⊆ G * , G ′ = {e} (otherwise, e would belong to G * ), verifying e = g∈G ′ g, which is impossible by construction of G whose members cannot be the union of elements of Z.

  uniquely attained for z j = |x j |/ x 1 . Similarly, we have whose minimun is uniquely obtained for z j = |x j |. Thus, we can equivalently rewrite Eq. ζ j = ( g∋j (ω g j ) 2 (η g ) -1 ) -1 . In the same vein, Eq. 2.5 is equivalent to min w∈R p , (η g ) g∈G ∈R

  Lemma 27Let u I ∈ R |I| . We have(Ω I ) * [u I ] By definiton of (Ω I ) * [u I ], we have (Ω I ) * [u I ] = max Ω I (v I )≤1 u ⊤ I v I .By introducing the primal variables (α g ) g∈G I ∈ R |G I | , we can rewrite the previous maximization problem as(Ω I ) * [u I ] = max v, g∈G I αg≤1 u ⊤ I v I , s.t. ∀ g ∈ G I , ω g I • u G∩I 2 ≤ α g ,which is a second-order cone program (SOCP) with |G I | second-order cone constraints.

L

  (v I , α g , γ, τ g , ξ g I ) = u ⊤ I v I + γ(1variables {γ, (τ g ) g∈G I , (ξ g I ) g∈G I } ∈ R + ×R |G I | ×R |I|×|G I |

  We proceed by contradiction. Let us assume there exists (ξ g I ) g∈G I such that max

  g j 0 | < |u j 0 |, which leads to a contradiction.We now give an upperbound on Ω * based on Lemma 27 and Lemma 28:Lemma 29 Let u I ∈ R |I| . We have (Ω I ) * [u I ] ≤ max g∈G I

  g , such that (v |g , z g ) ∈ C, ∀g ∈ G, by introducing the primal variables z = (z g ) g∈G ∈ R |G| , with the additional |G| conic constraints (v |g , z g ) ∈ C, for g ∈ G.

Table 3

 3

.1: Average and standard deviation of the normalized estimation error, computed over 1000 signals for PCA, SPCA and SSPCA.

  , G. Obozinski, F. Bach. Proximal Methods for Sparse Hierarchical Dictionary Learning. In Proceedings of the International Conference on Machine Learning (ICML). 2010 R. Jenatton * , J. Mairal * , G. Obozinski, F. Bach. Proximal Methods for Hierarchical Sparse Coding. In Journal of Machine Learning Research, 12, 2297-2334. 2011 (long version of the previous article) (

* , J. Mairal * * equal contributions)

Table 4

 4 Structured Sparsity-Inducing Norms

	Haar

.1: Top part of the tables: Average PSNR measured for the denoising of 12 standard images, when the wavelets are Haar or Daubechies3 wavelets (see

  Jenatton, A. Gramfort, V. Michel, G. Obozinski, F. Bach, and B. Thirion. Multiscale Mining of fMRI Data with Hierarchical Structured Sparsity.

	In International
	Workshop on Pattern Recognition in Neuroimaging (PRNI). 2011
	R. Jenatton, A. Gramfort, V. Michel, G. Obozinski, E. Eger, F. Bach, and B. Thirion.
	Multi-scale Mining of fMRI Data with Hierarchical Structured Sparsity. Preprint
	arXiv:1105.0363 Submitted to SIAM Journal on Imaging Sciences. 2011 (long version
	of the previous paper)

Table 5 .

 5 1: Prediction results obtained on fMRI data (see text) for the regression setting.

  Table 5.2: Prediction results obtained on fMRI data (see text) for the multi-class classification setting. From the left, the first column contains the mean and standard deviation

		2, 5.9)	0.004 *	100.00
	ℓ1	(33.3, 6.8)	0.004 *	0.10
	ℓ1/ℓ2 (Multi-task)	(31.7, 9.5)	0.004 *	0.12
	ℓ1/ℓ∞ (Multi-task)	(33.3,13.6)	0.009 *	0.22
	Tree ℓ2 (ρ = 0.5)	(25.8, 9.2)	0.004 *	99.93
	Tree ℓ2 (ρ = 1)	(25.0, 5.5)	0.027 *	10.08
	Tree ℓ2 (ρ = 1.5)	(24.2, 9.9)	0.130	0.05
	Tree ℓ∞ (ρ = 0.5)	(30.8, 8.8)	0.004 *	59.49
	Tree ℓ∞ (ρ = 1)	(24.2, 7.3)	0.058	1.21
	Tree ℓ∞ (ρ = 1.5)	(25.8, 10.7)	0.070	0.04
	Loss function:		Logistic ("one-versus-all")
		Error (mean,std) P-value w.r.t. Tree ℓ2 (ρ = 1)-ML Median fraction of non-zeros (%)
	Regularization:
	ℓ2 (Ridge)	(25.0, 9.6)	0.008 *	100.00
	ℓ1	(34.2, 15.9)	0.004 *	0.55
	ℓ1/ℓ2 (Multi-task)	(31.7, 8.6)	0.002 *	47.35
	ℓ1/ℓ∞ (Multi-task)	(33.3, 10.4)	0.002 *	99.95
	Tree ℓ2 (ρ = 0.5)	(25.0, 9.6)	0.007 *	99.93
	Tree ℓ2 (ρ = 1)	(20.0, 11.2)	0.250	7.88
	Tree ℓ2 (ρ = 1.5)	(18.3, 6.6)	0.500	0.06
	Tree ℓ∞ (ρ = 0.5)	(30.8, 10.4)	0.004 *	59.42
	Tree ℓ∞ (ρ = 1)	(24.2, 6.1)	0.035 *	0.60
	Tree ℓ∞ (ρ = 1.5)	(21.7, 8.9)	0.125	0.03
	Loss function:		Multinomial logistic (ML)
		Error (mean,std) P-value w.r.t. Tree ℓ2 (ρ = 1)-ML Median fraction of non-zeros (%)
	Regularization:
	ℓ2 (Ridge)	(24.2, 9.2)	0.035 *	100.00
	ℓ1	(25.8, 12.0)	0.004 *	97.95
	ℓ1/ℓ2 (Multi-task)	(26.7, 7.6)	0.007 *	30.24
	ℓ1/ℓ∞ (Multi-task)	(26.7, 11.6)	0.002 *	99.98
	Tree ℓ2 (ρ = 0.5)	(22.5, 8.8)	0.070	83.06
	Tree ℓ2 (ρ = 1)	(16.7, 10.4)	-	4.87
	Tree ℓ2 (ρ = 1.5)	(18.3, 10.9)	0.445	0.02
	Tree ℓ∞ (ρ = 0.5)	(26.7, 11.6)	0.015 *	48.82
	Tree ℓ∞ (ρ = 1)	(22.5, 13.0)	0.156	0.34
	Tree ℓ∞ (ρ = 1.5)	(21.7, 8.9)	0.460	0.05
		Error (mean,std) P-value w.r.t. Tree ℓ2 (ρ = 1)-ML Median fraction of non-zeros (%)
	Greedy	(21.6, 14.5)	0.001 *	0.01

of the test error (percentage of misclassification), computed over leave-one-subject-out folds. The best performance is obtained with the hierarchical ℓ 2 penalization (ρ = 1) constructed from the Ward tree, coupled with the multinomial logistic loss function. Statistical significance is assessed with a Wilcoxon two-sample paired signed rank test. The superscript * indicates a rejection at 5%.

  2 τ 2 Pr(E) + 33 exp(-τ ) .

	6.7. Control of the Taylor expansion
	180

  J . Fix τ > 0, and consider a random vector α 0 that follows the generative model from Section 6.2.2. For any J ⊆ 1; p with |J| = k, and for all |t| ≤1 such that [D(t)] ⊤ J [D(t)] J is invertible, we have Pr( [D(t)] ⊤ J c (I -[P(t)] J )[D 0 ] J [α 0 ] J ∞ > τ ) ≤ 2(pk) exp -τ 2 6α 2 t 2 ,

	and
	Pr( [D(t)] ⊤ J ([D(t)]

Lemma 17 (Control of the perturbation of D 0 ) For any J ⊆ 1; p , we introduce the orthogonal projector

[P(t)] J [D(t)] J [[D(t)] ⊤ J [D(t)] J ] -1 [D(t)] ⊤ J that projects onto the span of [D(t)]

  So, conditionally on the values of |y i |, we can apply Lemma 21, that is, for all t ≥ 2,Pr(|sign(y) ⊤ Diag(|y|)M off sign(y)| ≤ 45σ M off F t) ≥ 1 -2 exp(-t),where we have used the fact 64 × 65/96 ≤ 45, and Diag(|y|)M off F ≤ σ M off F . Putting the pieces together with the union bound, the inequality M off F + diag(M)

	√ the desired conclusion. 2[ M off 2 F + diag(M) 2 2 ] 1/2 =	√	2 M F , and the simplification 45 √	2 ≤ 2 ≤ 64, we obtain

A finite set A is said to be intersection-closed, if for any two elements a1, a2 ∈ A, we have a1 ∩ a2 ∈ A. In words, this means that the the set A is stable with respect to the intersection.

Let S be a finite set. A function F : 2 S → R is said to be submodular if for any subset A, B ⊆ S, we have the inequalityF (A ∩ B) + F (A ∪ B) ≤ F (A) + F (B); seeBach (2010a) and references therein.

More precisely, since we manipulate matrices, we measure how close DA is from X via the Frobenius norm, that is, 1 2 X -DA 2 F .

1+t/ a 2 2(a, t) otherwise.

The problem of identifiability (and uniqueness) is further discussed in Chapter 2. Usually, the assumptions made on the observations lead to a unique estimator ŵ.

Simultaneously, the authors from Raskutti et al. (2010) came up with the weaker scaling that imposes n |J ⋆ | log(p).

We use the term convex informally here. It can however be made precise with the notion of convex subgraphs[START_REF] Chung | Spectral Graph Theory[END_REF].

Note the subtlety: the setsG θ are disjoint, that is G θ ∩ G θ ′ = ∅ for θ = θ ′ , but groups in G θ and G θ ′ can overlap.

Due to the discrete nature of the underlying geometric structure of G, angles θ that are not multiple of π/4 (i.e., such that tan(θ) / ∈ Z) are dealt with by rounding operations.

To be more precise, in order to regularize every variable, we add the full group 1; p to G θ , which does not modify P.

.In the Lasso and group Lasso setting, the vector r ⋆ J ⋆ is respectively the sign vector sign(w ⋆ J ) and the vector defined by the blocks (wg wg 2 ) g∈G ⋆ J . We define Ω c J ⋆ (w [J ⋆ ] c ) = g∈(G ⋆ J ) c ω g [J ⋆ ] c • w [J ⋆ ] c 2(which is the norm composed of inactive groups) with its dual norm (Ω c J ⋆ ) * ; note the difference with the norm reduced to [J ⋆ ] c , defined asΩ [J ⋆ ] c (w [J ⋆ ] c ) = g∈G ω g [J ⋆ ] c • w [J ⋆ ] c 2 .The following Theorem gives the sufficient and necessary conditions under which the hull of the generating pattern is consistently estimated. Those conditions naturally extend the results of[START_REF] Zhao | On model selection consistency of Lasso[END_REF] andBach (2008b) for the Lasso and the group Lasso respectively (see proof in Appendix A.1.6).Theorem 2 (Consistency condition)Assume µ → 0, µ √ n → ∞ in Eq.(2.2). If the hull is consistently estimated, then(Ω c J ⋆ ) * Q ⋆ [J ⋆ ] c J ⋆ [Q ⋆ J ⋆ J ⋆ ] -1 r ⋆ J ⋆ ≤ 1. Conversely, if (Ω c J ⋆ ) * Q ⋆ [J ⋆ ] c J ⋆ [Q ⋆ J ⋆ J ⋆ ] -1 r ⋆ J ⋆ < 1, then the hull is consistently estimated, i.e., Pr {j ∈ 1; p , ŵj = 0} = J ⋆ -→ n→+∞ 1.The two previous propositions bring into play the dual norm (Ω cJ ⋆ ) * that we cannot compute in closed form, but requires to solve an optimization problem as complex as the initial problem in Eq. (2.5). However, we can prove bounds similar to those obtained in Propositions 7 and 8 for the necessary and sufficient conditions.Comparison with the Lasso and group Lasso. For the ℓ 1 -norm, our two bounds lead to the usual consistency conditions for the Lasso, i.e., the quantityQ ⋆ [J ⋆ ] c J ⋆ [Q ⋆ J ⋆ J ⋆ ] -1 sign(w ⋆ J ) ∞must be less or strictly less than one. Similarly, when G defines a partition of 1; p and if all the weights equal one, our two bounds lead in turn to the consistency conditions

Figure 2.10: Consitent hull estimation: probability of correct hull estimation versus the consistency condition(Ω c J ⋆ ) * [Q ⋆ [J ⋆ ] c J ⋆ [Q ⋆ J ⋆ J ⋆ ] -1 r ⋆ J ⋆ ].The transition appears at zero, in good agreement with Theorem 2.where δ is uniformly drawn in {1, 2, . . . , 10p}.7 The diagonal terms of Q are then re-normalized to one. For each of these covariance matrices, we compute an entire regularization path based on one realization of {J ⋆ , w ⋆ , X, ε}, with n = 3000 samples. The quantities {J ⋆ , w, ε} are generated as described previously, while the n rows of X are gaussian with covariance Q ⋆ . After repeating 20 times this computation for each Q ⋆ , we eventually report in Figure2.10 the probabilites of correct hull estimation versus the consistency condition(Ω c J ⋆ ) * [Q ⋆ [J ⋆ ] c J ⋆ [Q ⋆ J ⋆ J ⋆ ] -1 r ⋆ J ⋆ ]. In good agreement with Theorem 2, comparing (Ω c J ⋆ ) * [Q ⋆ [J ⋆ ] c J ⋆ [Q ⋆ J ⋆ J ⋆ ] -1 r ⋆ J ⋆ ]to 1 determines whether the hull is consistently estimated.

Note that it already corresponds to several hundreds of runs for both the SOCP and the active set algorithms since we compute a 5-fold cross-validation for each regularization parameter of the (approximate) regularization path.

We have not displayed this second figure that is just the superposition of the error curves for the SOCP and the active set algorithms.

Although we use the term convex informally here, it can however be made precise with the notion of convex subgraphs[START_REF] Chung | Spectral Graph Theory[END_REF].

Note that we depart from[START_REF] Micchelli | Learning the kernel function via regularization[END_REF] who consider a quadratic upperbound on the squared norm. We prefer to remain in the standard dictionary learning framework where the penalization is not squared.

For the sake of clarity, we do not specify the dependence of ζ on (η g )g∈G.

We perform 5-fold cross-validation and the number of nearest neighbor(s) is searched in {1, 3, 5} while log 10 (λ) is in {-11, -10.5, . . . , -7}. For the dictionary, we consider the sizes p ∈ {10, 20, . . . , 150}.

A tree is defined as a connected graph that contains no cycle (see[START_REF] Ahuja | Network Flows[END_REF].

Our analysis straightforwardly extends to the case of a forest of trees; for simplicity, we consider a single tree T .

We consider that the set of ancestors of a node also contains the node itself.

Note the difference with the notation αg, which is often used in the literature on structured sparsity, where αg is a vector of size |g|.

For a complete definition of Ω for any ℓq-norm, a discussion of the choice of q, and a strategy for choosing the weights ωg (see[START_REF] Zhao | The composite absolute penalties family for grouped and hierarchical variable selection[END_REF][START_REF] Kim | Tree-guided group Lasso for multi-task regression with structured sparsity[END_REF].

It has been further shown byBach (2010a) that the convex envelope of the nonconvex function of Eq. (4.4) is in fact Ω with . being the ℓ∞-norm.

Note that the authors of[START_REF] Chen | An efficient proximalgradient method for single and multi-task regression with structured sparsity[END_REF] have considered proximal methods for general group structure G when . is the ℓ2-norm; due to a smoothing of the regularization term, the convergence rate they obtained is suboptimal.

It is easy to show that the dual norm of the ℓ2-norm is the ℓ2-norm itself. The dual norm of the ℓ∞ is the ℓ1-norm.

Note that although we use this classical scheme for simplicity, it would also be possible to use the stochastic approach proposed byMairal et al. (2010a).

The dataset we use is 14_Tumors, which is freely available at http://www.gems-system.org/.

Denoting by MSE the mean-squared-error for images whose intensities are between 0 and 255, the PSNR is defined as PSNR = 10 log 10 (255 2 /MSE) and is measured in dB. A gain of 1dB reduces the MSE by approximately 20%.

http://www.cns.nyu.edu/ eero/steerpyr/.

It is worth mentioning that comparing convex and nonconvex approaches for sparse regularization is a bit difficult. This conclusion holds for the classical formulation we have used, but might not hold in other settings such as[START_REF] Coifman | Translation-invariant de-noising[END_REF].

Note that we study the ability of the model to reconstruct independent patches, and additional work is required to apply our framework to a full image processing task, where patches usually overlap[START_REF] Elad | Image denoising via sparse and redundant representations over learned dictionaries[END_REF] Mairal et al., 2009b).

Available at http://people.csail.mit.edu/jrennie/20Newsgroups/.

Note however that, while[START_REF] Hochbaum | About strongly polynomial time algorithms for quadratic optimization over submodular constraints[END_REF] only consider a tree-structured sum of ℓ∞-norms, the results fromJenatton et al. (2010a) also apply for a sum of ℓ2-norms.

By definition, a parametric max-flow problem consists in solving, for every value of a parameter, a max-flow problem on a graph whose arc capacities depend on this parameter.

The datasets are freely available at http://www.gems-system.org/.

More precisely, since the penalties in problem (4.16) shrink the coefficients of W, we follow a two-step procedure: We first run our approach to determine the sets of nonzero rows and columns, and then compute W I J = C + XR + .

We can then identify nodes (and parcels) of T with indices in {1, . . . , p}.

For simplicity and clarity of the presentation, we do not consider the optimization of the intercept that we let unregularized in all our experiments.

The Matlab/C++ implementation we use is available at http://www.di.ens.fr/willow/SPAMS/.

http://www.fil.ion.ucl.ac.uk/spm/software/spm5.

Formally, the depth of the feature j is equal to depth(gj), where gj is the smallest group in G that contains j (smallest is understood here in the sense of the inclusion).

The code is available at http://www.di.ens.fr/willow/SPAMS/.

Note that due to the stochastic optimization tool used here, even the curves for the fixed initialization exhibit some small variations.

hence for any j ∈ [J ⋆ ] c µ ∆j = ( ŵw ⋆ ) j = 0. This implies that the nonrandom vector∆ * verifies ∆ * [J ⋆ ] c = 0. As a consequence, ∆ * J ⋆ minimizes ∆ ⊤ J ⋆ Q ⋆ J ⋆ J ⋆ ∆ J ⋆ + [r ⋆ J ⋆ ] ⊤ ∆ ⋆ J , hence r ⋆ J ⋆ = -Q J ⋆ J ⋆ ∆ * J ⋆ .Besides, since ∆ * is the minimizer of Flim , by taking the directional derivatives as in the proof of Lemma 32, we have(Ω [J ⋆ ] c ) * [Q ⋆ [J ⋆ ] c J ⋆ ∆ * J ⋆ ] ≤ 1.

u ⊤ [J ⋆ ] c ∇L( ŵ) [J ⋆ ] c = u ⊤ [J ⋆ ] c -µ Q [J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ rJ ⋆ + Q [J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ q J ⋆q [J ⋆ ] c .The terms related to the noise vanish, having actuallyq = o p (1). Since Q → Q ⋆ and rJ ⋆ → r ⋆ J ⋆ , we get for all u [J ⋆ ] c ∈ R |[J ⋆ ] c | u ⊤ [J ⋆ ] c ∇L( ŵ) [J ⋆ ] c = -µ u ⊤ [J ⋆ ] c Q ⋆ [J ⋆ ] ⋆ J ⋆ [Q ⋆ J ⋆ J ⋆ ] -1 r ⋆ J ⋆ + o p (µ).Since we assume (Ω[J ⋆ ] c ) * [Q ⋆ [J ⋆ ] c J ⋆ [Q ⋆ J ⋆ J ⋆ ] -1 r ⋆ J ⋆ ] < 1, we obtain -u ⊤ [J ⋆ ] c ∇L( ŵ) [J ⋆ ] c < µ(Ω c J ⋆ )[u [J ⋆ ] c ] + o p (µ),which proves the optimality condition of Lemma 32 relative to the inactive variables: ŵ is therefore optimal for the full problem.

√ ϕ ŵJ ⋆ -w ⋆ J ⋆ ∞ ν 1 + 5ϕ w ⋆ ∞ ν g∈G J ⋆ ω g J ⋆ 2.Introducing α = 18ϕ 3/2 w ⋆ ∞ ν 2 g∈G J ⋆ ω g J ⋆ 2 , we thus have proved rJ ⋆r ⋆ J ⋆ 1 ≤ α ŵJ ⋆w ⋆ J ⋆ ∞ . (A.10)By writing the Schur complement of Q on the block matricesQ [J ⋆ ] c [J ⋆ ] c and Q J ⋆ J ⋆ , the positiveness of Q implies that the diagonal terms diag(Q [J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ Q J ⋆ [J ⋆ ] c ) are less than one, which results in Q [J ⋆ ] c J ⋆ Q -1/2 J ⋆ J ⋆ ∞,2 ≤ 1. We then have Q [J ⋆ ] c J ⋆ Q -1 J ⋆ J ⋆ (r J ⋆r ⋆ J ⋆ ) ∞ = Q [J ⋆ ] c J ⋆ Q -1/2 J ⋆ J ⋆ Q -1/2 J ⋆ J ⋆ (r J ⋆r ⋆ J ⋆ ) ∞ (A.11) ≤ Q [J ⋆ ] c J ⋆ Q -1/2 J ⋆ J ⋆ ∞,2 Q -1/2 J ⋆ J ⋆ 2 rJ ⋆r ⋆ J ⋆ 2 (A.12) ≤ κ -1/2 rJ ⋆r ⋆

2 where v has the same value as during the iteration g of Algorithm 6. Therefore, after calling the

Proximal Methods for Structured Sparsity-Inducing Norms

Extension: General Overlapping Groups and

ℓ 1 /ℓ ∞ -norms

The work presented in this extension section was achieved with the collaboration of Julien Mairal, Guillaume Obozinski and Francis Bach, with In this section, we consider an extension of the setting studied so far in the chapter. In particular, we now take Ω to be a linear combination of ℓ ∞ -norms, while we do not assume anymore that G is tree-structured; G thus corresponds to a general set of overlapping groups. As it will be discussed at length later, the ℓ ∞ -norm is piecewise linear, a property we will fully take advantage of. From now on, we focus on the computation of the following proximal operator:

For all the detailed proofs of the results presented subsequently, we refer the interested readers to the long version [START_REF] Mairal | Convex and network flow optimization for structured sparsity[END_REF]. We start by specifying the dual formulation from Lemma 2 to the setting of Eq. (4.12); we shall see that this dual problem can be reformulated as a quadratic min-cost flow problem for which we present an efficient algorithm.

Lemma 6 (Dual of the proximal problem (4.12)) Given u in R p , consider the problem

s.t. ∀g ∈ G, ξ g 1 ≤ λη g and ξ g j = 0 if j / ∈ g, (4.13)

where ξ = (ξ g ) g∈G is in R p×|G| , and ξ g j denotes the j-th coordinate of the vector ξ g . Then, every solution ξ ⋆ = (ξ ⋆g ) g∈G of Eq. (4.13) satisfies w ⋆ = u-g∈G ξ ⋆g , where w ⋆ is the solution of Eq. (4.12).

Local Analysis of Sparse Coding in Presence of Noise

The names of the next Lemma refer to the notation introduced in Lemma 15.

Lemma (Concentration of A εα )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term A εα is upper bounded by σατ, with probability exceeding 1c 0 exp(-c 1 τ ).

Lemma (Concentration of A sα )

For any τ ≥ 2, we have

with probability exceeding 1 -4 exp(-τ ).

Lemma (Concentration of A εs,1 )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term A εs,1 is upper bounded by λσ 1 1kµ 0 τ with probability exceeding 1c 0 exp(-c 1 τ ).

Lemma (Concentration of A εs,2 )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term A εs,2 is upper bounded by λσ √ 1 + kµ 0 (1kµ 0 ) 3/2 τ with probability exceeding 1c 0 exp(-c 1 τ ).

Lemma (Concentration of A ss )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term A ss is upper bounded by

with probability exceeding 1c 0 exp(-c 1 τ ).

Lemma (Concentration of A εε )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term A εε is upper bounded by

with probability exceeding 1c 0 exp(-c 1 τ ).

Concentration of the second-order terms

with probability exceeding 1 -4 exp(-τ ).

Lemma (Concentration of B εα,1 )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term B εα,1 is upper bounded by

Lemma (Concentration of B εα,2 )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term B εα,2 is upper bounded by

with probability exceeding 1c 0 exp(-c 1 τ ).

Lemma (Concentration of B εs,1 )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term B εs,1 is upper bounded by

Lemma (Concentration of B εs,2 )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term B εs,2 is upper bounded by

Lemma (Concentration of B εs,3 )

There exist universal constants c 0 , c 1 > 0 such that for any τ ≥ 1, the term B εs,3 is upper bounded by

Local Analysis of Sparse Coding in Presence of Noise

Again, when setting u = 1/e, and with similar upper bounds as before, we obtain

which concludes the proof.

Computation of the Taylor expansion

For any sign vector s with support J, the function D → φ x (D|s) as defined in (6.4) is smooth as long as the matrix D ⊤ J D J is invertible. In this section, we derive a secondorder Taylor expansion of the composition t → φ x (D(t)|s) around the value t = 0, assuming that the smoothness conditions described above are satisfied for values of t small enough. Moreover, we are interested in this expansion in the specific setting from Section 6.2.2 where each signal x is equal to x = D 0 α 0 + ε, and where the sign vector s is such that s = sign(α 0 ).

Throughout the different computations, we will need to refer to the orthogonal projector onto the span of [D 0 ] J , which we denote by P 0 . In addition, the matrix Π J ∈ R p×|J| that projects onto the columns indexed by J, i.e., D J = DΠ J , is represented by Π to ease notation. For short, we refer to Diag(v), Diag(cos(vt)) and Diag(sin(vt)) as respectively V, C and S. Finally, we denote by Θ 0 the inverse of [D 0 ] ⊤ J [D 0 ] J . Lemma 15 (Taylor expansion of φ x (.|s)) Assuming the invertibility conditions for D(t) around t = 0, the function t → φ x (D(t)|s) admits a second-order Taylor expansion around t = 0, and we have

with the following expressions for a x and b x :

.

Some details about the computation of the Taylor expansion

In this section, we detail the computations underlying Lemma 15. For convenience, we rewrite φ x (D(t)|s) as

where we have introduced

Derivatives of t → Σ(t) -1 . We gather here the formula related to the derivatives of Σ(t) and Σ(t) -1 .

We notably have for any t ≥ 0 the tail probability

Moreover, if {z j } j∈ 1;p is a collection of p independent sub-Gaussian variables with parameters σ j , then p j=1 z j is also sub-Gaussian, with parameter p j=1 σ 2 j . Lemma 20 (From Hsu et al. (2011)

Lemma 23 (Hoeffding's Inequality, e.g., see [START_REF] Boucheron | Concentration inequalities. Advanced Lectures on Machine Learning[END_REF]) Let {z j } j∈ 1;p be a collection of independent, bounded random variables such that for any j ∈ 1; p , it holds that z j ∈ [z j , z j ] almost surely. For any t ≥ 0, we have

Lemma 24 (Bernstein's Inequality, e.g., see [START_REF] Boucheron | Concentration inequalities. Advanced Lectures on Machine Learning[END_REF]) Let {z j } j∈ 1;p be a collection of independent, zero-mean and bounded random variables such that for any j ∈ 1; p , it holds that |z j | ≤ z almost surely. Let us denote by σ 2 j the variance of z j and define σ 2 1 p p j=1 σ 2 j . For any t ≥ 0, we have

.

Technical lemmas

In particular, for any t ≤ 3σ 2z √ p, we have

Some more specific tail bounds

Lemma 25

Let M ∈ R n×p be a matrix. Consider two independent sub-Gaussian vectors y ∈ R n and x ∈ R m with i.i.d. entries of respective parameters σ x and σ y . There exist universal constants c 0 , c 1 > 0 such that we have for all t ≥ 1

Proof. We simply apply the result from Lemma 22 on the quadratic form

Notice that the Frobenius norm of the augmented matrix is bounded by

Lemma 26

Let M ∈ R n×n be a matrix. Consider a random vector y ∈ R n with i.i.d. entries. Assume the distribution of these entries to be symmetric and bounded, with y ∞ ≤ σ almost surely. For all t ≥ 2, it holds that

Proof. We start by splitting y ⊤ M sign(y) into

The expectation of the first term is equal to E[|y 1 |]Tr(M), while the second term is centered. Since the first term consists of a sum of independent bounded random variables, we can apply Hoeffding's inequality (see Lemma 23), that is, for all t ≥ 0,

where we have used the fact that |y i | ∈ [0, σ] almost surely. Let us focus on the second residual term. Notice that

a) the Hessian of L J , which is positive definite (still from the same argument as in the proof of Theorem A.1.1), b) the Hessian of the norm Ω J around (w ỹ J , ỹ) that is positive semidefinite on this small ball according to the Hessian characterization of convexity (Borwein and Lewis, 2006, Theorem 3.1.11).

Consequently, H JJ is invertible. We can now apply the implicit function theorem to obtain that for Y in a neighborhood of ỹ,

with ψ = (ψ j ) j∈J a continuously differentiable function satisfying the matricial relation

Let c α denote the column vector of H -1 JJ corresponding to the index α, and let D the diagonal matrix whose

Now, since X J has full rank, c α = 0 and none of the diagonal elements of D is null (by assumption on ℓ), we have ∇ψ α = 0. Without loss of generality, we may assume that ∂ψ α /∂y 1 = 0 on a neighborhood of ỹ.

We can apply again the implicit function theorem to show that on an open ball in R n centered at ỹ, say B ỹ, the solution to ψ α (Y ) = 0 can be written y 1 = ϕ(y 2 , . . . , y n ) with ϕ a continuously differentiable function.

By Fubini's theorem and by using the fact that the Lebesgue measure of a singleton in R n equals zero, we get that the set A(ỹ) = Y ∈ B ỹ : ψ α (Y ) = 0 has thus zero probability. Let S ⊂ E I be a compact set. We thus have S ⊂ E ′ I . By compacity, the set S can be covered by a finite number of ball B ỹ. So there exist ỹ1 , . . . , ỹm such that we have S ⊂ A(ỹ 1 ) ∪ • • • ∪ A(ỹ m ). Consequently, we have Pr(S) = 0.

Since this holds for any compact set in E I and since the Lebesgue measure is regular, we have Pr(E I ) = 0, which contradicts the definition of I, and concludes the proof.

A.1.3 Proof of the minimality of the Backward procedure (see Algorithm 1)

There are essentially two points to show: (1) G spans Z, and (2) G is minimal.

The first point can be shown by a proof by recurrence on the depth of the DAG. At step t, the base G (t) verifies { g∈G ′ G, ∀G ′ ⊆ G (t) } = {g ∈ Z, |g| ≤ t} because an element g ∈ Z is either the union of itself or the union of elements strictly smaller. The initialization t = min g∈Z |g| is easily verified, the leafs of the DAG being necessarily in G.

A.1. Proofs and Technical Elements of Chapter 1

Reciprocally, starting from (A.2), we notably have

In light of (A.4), it suffices to check that Ω * J (κ J ) ≤ λΩ J (w J ) in order to have Eq. (A.1). Combining Eq. (A.2) with the definition of the dual norm, it comes

which concludes the proof of the equivalence between Eq. (A.1) and Eq. (A.2).

A.1.5 Proofs of Propositions 7 and 8

In order to check that the reduced solution w J is optimal for the full problem in Eq. ( 2.5), we complete with zeros on J c to define w, compute κ = -∇L(w), which is such that κ J = -∇L J (w J ), and get a duality gap for the full problem equal to

By designing upper and lower bounds for Ω * (κ), we get sufficient and necessary conditions.

Proof of Proposition 7

Let us suppose that w * = w * J 0 J c is optimal for the full problem in Eq. (2.5). Following the same derivation as in Lemma 32 (up to the squaring of the regularization Ω), we have that w * is a solution of Eq. (2.5) if and only if for all u ∈ R p ,

We project the optimality condition onto the variables that can possibly enter the active set, i.e., the variables in Π P (J). Thus, for each K ∈ Π P (J), we have for all

By combining Lemma 31 and the fact that G K\J ∩ (G J ) c = G K \G J , we have for all G ∈ G K \G J , K\J ⊆ g and therefore u g∩K\J = u K\J . Since we cannot compute the dual norm of u K\J → ω g K\J •u K\J 2 in closed-form, we instead use the following upperbound

A.1. Proofs and Technical Elements of Chapter 1

The quantity ∆ = ( ŵw ⋆ )/µ is the minimizer of F defined as

where

Since µ → 0, we also have by taking the directional derivative of Ω at w in the direction of ∆

The limiting function Flim being stricly convex (because Q ⋆ ≻ 0) and F being convex, we have that the minimizer ∆ of F tends in probability to the unique minimizer of Flim [START_REF] Fu | Asymptotics for Lasso-type estimators[END_REF] referred to as ∆ * .

By assumption, with probability tending to one, we have

A. Proofs to w ⋆ . From now on, we consider that ŵ is sufficiently close to w ⋆ , so that for any g ∈ G J ⋆ , ω g • ŵ 2 = 0. We may thus introduce

It remains to show that ŵ is indeed optimal for the full problem (that admits a unique solution due to the positiveness of Q). By construction, the optimality condition (see Lemma 32) relative to the active variables J ⋆ is already verified. More precisely, we have

Moreover, for all u

, by using the previous expression and the invertibily of Q, we have Moreover, our proof will rely on the control of the expected dual norm for isonormal vectors:

] with v a centered Gaussian random variable with unit covariance matrix. In the case of the Lasso, it is of order (log p) 1/2 .

Following Bach (2008b) and [START_REF] Nardi | On the asymptotic properties of the group Lasso estimator for linear models[END_REF], we consider the reduced problem on J ⋆ , min w∈R p L J ⋆ (w J ⋆ ) + µΩ J (w J ⋆ ) with solution ŵJ ⋆ , which can be extended to J c with zeros. From optimality conditions (see Lemma 32), we know that

where the vector q ∈ R p is defined as q

We denote by ν min{|w ⋆ j |; w ⋆ j = 0} the smallest nonzero components of w ⋆ . We first prove that we must have with high probability ŵg ∞ > 0 for all g ∈ G J ⋆ , proving that the hull of the active set of ŵJ ⋆ is exactly J ⋆ (i.e., no active group is missing). We have

) , hence from (A.5) and the definition of A(J ⋆ ),

so that for all g ∈ G J ⋆ , ŵg ∞ ≥ ν 3 , hence the hull is indeed selected. This also ensures that ŵJ ⋆ satisfies the equation (see Lemma 32)

We now prove that the ŵ padded with zeros on [J ⋆ ] c is indeed optimal for the full problem with high probability. According to Lemma 32, since we have already proved (A.9), it suffices to show that

where the last line comes from Eq. (A.6) and (A.10). We get

Thus, if the following inequalities are verified

we obtain

i.e., J ⋆ is exactly selected. Combined with earlier constraints, this leads to the first part of the desired proposition.

We now need to make sure that the conditions Eq. (A.7), (A.16) and (A.17) hold with high probability. To this end, we upperbound, using Gaussian concentration inequalities, two tail-probabilities. First, q [J ⋆ ] c |J ⋆ is a centered Gaussian random vector with covariance matrix

where

and v a centered Gaussian random variable with unit covariance matrix.

Since for any u we have

, by using Sudakov-Fernique inequality [START_REF] Adler | An Introduction to Continuity, Extrema, and Related Topics for General Gaussian Processes[END_REF], Theorem 2.9), we get:

In addition, we have

A.1. Proofs and Technical Elements of Chapter 1

We give below an important Lemma to characterize the solutions of (2.2).

Lemma 32

The vector ŵ ∈ R p is a solution of

≤ µ, with Ĵ the hull of {j ∈ 1; p , ŵj = 0} and the vector r ∈ R p defined as

In addition, the solution ŵ satisfies

Proof. The problem min w∈R p L(w) + µ Ω(w) = min w∈R p F (w) being convex, the directional derivative optimality condition are necessary and sufficient (Borwein and Lewis, 2006, Propositions 2.1.1-2.1.2). Therefore, the vector ŵ is a solution of the previous problem if and only if for all directions u ∈ R p , we have

Some algebra leads to the following equivalent formulation

The first part of the lemma then comes from the projections on Ĵ and Ĵc . An application of the Cauchy-Schwartz inequality on u ⊤ Ĵ rĴ gives for all u ∈ R p

Combined with the equation (A.23), we get ∀u ∈ R p , u ⊤ ∇L( ŵ) + µ Ω(u) ≥ 0, hence the second part of the lemma.

We end up with a lemma regarding the dual norm of the sum of two disjoint norms (see [START_REF] Rockafellar | Convex analysis[END_REF]:

Lemma 33

Let A and B be a partition of 1; p , i.e., A ∩ B = ∅ and A ∪ B = 1; p . We consider two norms

A. Proofs

A.2 Proofs and Technical Elements of Chapter 2 A.2.1 Links with Tree-Structured Nonconvex Regularization

We present in this section an algorithm introduced by [START_REF] Donoho | CART and best-ortho-basis: a connection[END_REF] in the more general context of approximation from dyadic partitions (see Section 6 in [START_REF] Donoho | CART and best-ortho-basis: a connection[END_REF] for solving the following problem

where the u in R p is given, λ is a regularization parameter, G is a set of tree-structured groups in the sense of Definition 1, and the functions δ g are defined as in Eq. ( 4.4)-that is, δ g (v) = 1 if there exists j in g such that v j = 0, and 0 otherwise. This problem can be viewed as a proximal operator for the nonconvex regularization g∈G δ g (v). As we will show, it can be solved efficiently, and in fact it can be used to obtain approximate solutions of the nonconvex problem presented in Eq. ( 4.1), or to solve tree-structured wavelet decompositions as done by [START_REF] Baraniuk | Model-based compressive sensing[END_REF].

We now briefly show how to derive the dynamic programming approach introduced by Donoho (1997). Given a group g in G, we use the same notations root(g) and children(g) introduced in Section 4.3.5. It is relatively easy to show that finding a solution of Eq. (A.24) amounts to finding the support S ⊆ {1, . . . , p} of its solution and that the problem can be equivalently rewritten with the abusive notation δ g (S) = 1 if g ∩ S = ∅ and 0 otherwise. We now introduce the quantity

After a few computations, solving Eq. (A.25) can be shown to be equivalent to minimizing ψ g 0 (S) where g 0 is the root of the tree. It is then easy to prove that for any group g in G, we have min S⊆{1,...,p}

which leads to the following dynamic programming approach presented in Algorithm 10. This algorithm shares several conceptual links with Algorithm 7 and 8. It traverses the tree in the same order, has a complexity in O(p), and it can be shown that the whole procedure actually performs a sequence of thresholding operations on the variable v.

A. Proofs

After simplifying the Lagrangian and flipping (without loss of generality) the sign of ξ, we obtain the dual problem in Eq. (4.8). We derive the optimality conditions from the Karush-Kuhn-Tucker conditions for generalized conic inequalities [START_REF] Boyd | Convex Optimization[END_REF]. We have that {v, z, τ , ξ} are optimal if and only if

Combining the complementary slackness with the definition of the dual norm, we have

Furthermore, using the fact that ∀g ∈ G, (v |g , z g ) ∈ C and (ξ g , τ g ) = (ξ g , λω g ) ∈ C * , we obtain the following chain of inequalities

for which equality must hold. In particular, we have v ⊤ |g ξ g = v |g ξ g * and z g ξ g * = λz g ω g . If v |g = 0, then z g cannot be equal to zero, which implies in turn that ξ g * = λω g . Eventually, applying Lemma 34 gives the advertised optimality conditions.

Conversely, starting from the optimality conditions of Lemma 2, and making use again of Lemma 34, we can derive the Karush-Kuhn-Tucker conditions displayed above. More precisely, we define for all g ∈ G, τ g λω g and z g v |g .

The only condition that needs to be discussed is the complementary slackness condition.

If v |g = 0, then it is easily satisfied. Otherwise, combining the definitions of τ g , z g and the fact that v ⊤ |g ξ g = v |g ξ g * and ξ g * = λω g , we end up with the desired complementary slackness.

Optimality condition for the projection on the dual ball

Lemma 34 (Projection on the dual ball)

Proof. When the vector w is already in the ball of . * with radius t, i.e., w * ≤ t, the situation is simple, since the projection Π . * ≤t (w) obviously gives w itself. On the other hand, a necessary and sufficient optimality condition for having κ = Π . * ≤t (w) =

A. Proofs

This implies that u |gξ gξ h |g = ρ g ξ gρg ρ h +1 ξ g , which eventually leads to

The desired conclusion follows

. ℓ ∞ -norm: In this case, the optimality corresponds to the conditions for equality in the ℓ ∞ -ℓ 1 Hölder inequality. Specifically, ξ g = Π . * ≤tg (u |g ) holds if and only if for all ξ g j = 0, j ∈ g, we have

Looking at the same condition for ξ h , we have that ξ h = Π . * ≤th u |hξ g holds if and only if for all ξ h j = 0, j ∈ h, we have

From those relationships we notably deduce that for all j ∈ g such that ξ g j = 0, sign(ξ g j ) = sign(u j ) = sign(ξ h j ) = sign(u jξ g j ) = sign(u jξ g jξ h j ). Let j ∈ g such that ξ g j = 0. At this point, using the equalities we have just presented,

|g ∞ (which can be shown using the sign equalities above), and

and therefore for all ξ g j = 0, j ∈ g, we have u jξ g jξ h j = u |gξ gξ h |g ∞ sign(ξ g j ), which yields the result.

Proof of Lemma 5

Proof. Notice first that the procedure computeSqNorm is called exactly once for each group g in G, computing a set of scalars (ρ g ) g∈G in an order which is compatible with the convergence in one pass of Algorithm 6-that is, the children of a node are processed prior to the node itself. Following such an order, the update of the group g in the original Algorithm 6 computes the variable ξ g which updates implicitly the primal variable as follows

It is now possible to show by induction that for all group g in G, after a call to the procedure computeSqNorm(g), the auxiliary variable η g takes the value v |g A.2. Proofs and Technical Elements of Chapter 2 procedure computeSqNorm(g 0 ), where g 0 is the root of the tree, the values ρ g correspond to the successive scaling factors of the variable v |g obtained during the execution of Algorithm 6. After having computed all the scaling factors ρ g , g ∈ G, the procedure recursiveScaling ensures that each variable j in {1, . . . , p} is scaled by the product of all the ρ h , where h is an ancestor of the variable j.

The complexity of the algorithm is easy to characterize: Each procedure computeSqNorm and recursiveScaling is called p times, each call for a group g has a constant number of operations plus as many operations as the number of children of p. Since each child can be called at most one time, the total number of operation of the algorithm is O(p).

Sign conservation by projection

The next lemma specifies a property for projections when . is further assumed to be a ℓ q -norm (with q ≥ 1). We recall that in that case, . * is simply the ℓ q ′ -norm, with q ′ = (1 -1/q) -1 .

Lemma 35 (Projection on the dual ball and sign property)

Let w ∈ R p and t > 0. Let us assume that . is a ℓ q -norm (with q ≥ 1). Consider also a diagonal matrix S ∈ R p×p whose diagonal entries are in {-1, 1}. We have Π . * ≤t (w) = SΠ . * ≤t (Sw).

Proof. Let us consider κ = Π . * ≤t (w). Using essentially the same argument as in the proof of Lemma 34, we have for all y such that y q ′ ≤ t, (wκ) ⊤ (yκ) ≤ 0. Noticing that S ⊤ S = I and y q ′ = Sy q ′ , we further obtain (Sw -Sκ) ⊤ (y ′ -Sκ) ≤ 0 for all y ′ with y ′ q ′ ≤ t. This implies in turn that SΠ . * ≤t (w) = Π . * ≤t (Sw), which is equivalent to the advertised conclusion.

Based on this lemma, note that we can assume without loss of generality that the vector we want to project (in this case, w) has only nonnegative entries. Indeed, it is sufficient to store beforehand the signs of that vector, compute the projection of the vector with nonnegative entries, and assign the stored signs to the result of the projection.

Non-negativity constraint for the proximal operator

The next lemma shows how we can easily add a non-negativity constraint on the proximal operator when the norm Ω is absolute (Stewart and Sun, 1990, Definition 1.2), that is, a norm for which the relation Ω(u) ≤ Ω(w) holds for any two vectors w and u ∈ R p such that |u j | ≤ |w j | for all j.

Lemma 36 (Non-negativity constraint for the proximal operator)

Let κ ∈ R p and λ > 0. Consider an absolute norm Ω. We have arg min

Proof. Let us denote by ẑ+ and ẑ the unique solutions of the left-and right-hand side of (A.27) respectively. Consider the normal cone N R p + (z 0 ) of R p + at the point z 0 [START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF] and decompose κ into its positive and negative parts, κ = [κ] + + [κ] -. We can now write down the optimality conditions for the two convex problems above [START_REF] Borwein | Convex Analysis and Nonlinear Optimization: Theory and Examples[END_REF]: ẑ+ is optimal if and only if there exists w ∈ ∂Ω(ẑ + ) such that ẑ+ -[κ] + + λw = 0. Similarly, ẑ is optimal if and only if there exists (s, u) ∈ ∂Ω(ẑ)×N R p + (ẑ) such that ẑ-κ+λs+u = 0. We now prove that

We proceed by contradiction. Let us assume that there exists z

This implies that there exists j ∈ {1, . . . , p} for which [κ j ] -< 0 and z j -ẑ+ j < 0. In other words, we have 0

With the assumption made on Ω and replacing ẑ+ j by z j , we have found a solution to the left-hand side of (A.27) with a stricly smaller cost function than the one evaluated at ẑ+ , hence the contradiction. Putting the pieces together, we now have

which shows that ẑ+ is the solution of the right-hand side of (A.27).

A.2.3 Counterexample for ℓ q -norms, with q / ∈ {1, 2, ∞}.

The result we have proved in Proposition 9 in the specific setting where . is the ℓ 2 -or ℓ ∞ -norm does not hold more generally for ℓ q -norms, when q is not in {1, 2, ∞}. Let q > 1 satisfying this condition. We denote by q ′ (1q -1 ) -1 the norm parameter dual to q. We keep the same notation as in Lemma 3 and assume from now on that u |g q ′ > t g and u |h q ′ > t g + t h . These two inequalities guarantee that the vectors u |g and u |hξ g do not lie in the interior of the ℓ q ′ -norm balls, of respective radius t g and t h .

We show in this section that there exists a setting for which the conclusion of Lemma 3 does not hold anymore. We first focus on a necessary condition of Lemma 3: Lemma 37 (Necessary condition of Lemma 3) Let . be a ℓ q -norm, with q / ∈ {1, 2, ∞}. If the conclusion of Lemma 3 holds, then the vectors ξ g |g and ξ h |g are linearly dependent.

Proof. According to our assumptions on u |g and u |hξ g , we have that ξ g q ′ = t g and ξ h q ′ = t h . In this case, we can apply the second optimality conditions of Lemma 34, which states that equality holds in the ℓ q -ℓ q ′ Hölder inequality. As a result, there exists ρ g , ρ h > 0 such that for all j in g:

If the conclusion of Lemma 3 holds-that is, we have ξ g = Π . * ≤tg (u |gξ h |g ), notice that it is not possible to have the following scenarios, as proved below by contradiction: A.2. Proofs and Technical Elements of Chapter 2

|g q ′ = t g , then we would have for all j in g, |ξ h j | q ′ = ρ h |u j -ξ g j -ξ h j | q = 0, which implies that ξ h |g = 0 and u |g q ′ = t g . This is impossible since we assumed u |g q ′ > t g . We therefore have u |gξ h |g q ′ > t g and using again the second optimality conditions of Lemma 34, there exists ρ > 0 such that for all j in g, |ξ g j | q ′ = ρ|u jξ g jξ h j | q . Combined with the previous relation on ξ h |g , we obtain for all j in g, |ξ g j | q ′ = ρ ρ h |ξ h j | q ′ . Since we can assume without loss of generality that u only has nonnegative entries (see Lemma 35), the vectors ξ g and ξ h can also be assumed to have nonnegative entries, hence the desired conclusion.

We need another intuitive property of the projection Π . * ≤t to derive our counterexample:

Lemma 38 (Order-preservation by projection)

Let . be a ℓ q -norm, with q / ∈ {1, ∞} and q ′ 1/(1q -1 ). Let us consider the vectors κ, w ∈ R p such that κ = Π . * ≤t (w) = arg min y q ′ ≤t yw 2 , with the radius t satisfying w q ′ > t. If we have w i < w j for some (i, j) in {1, . . . , p} 2 , then it also holds that κ i < κ j .

Proof. Let us first notice that given the assumption on t, we have κ q ′ = t. The Lagrangian L associated with the convex minimization problem underlying the definition of Π . * ≤t can be written as L(y, α) = 1 2 yw 2 2 + α y q ′ q ′t q ′ , with the Lagrangian parameter α ≥ 0.

At optimality, the stationarity condition for κ leads to ∀ j ∈ {1, . . . , p}, κ jw j + αq ′ |κ j | q ′ -1 = 0.

We can assume without loss of generality that w only has nonnegative entries (see Lemma 35). Since the components of κ and w have the same signs (see Lemma 35), we therefore have |κ j | = κ j ≥ 0, for all j in {1, . . . , p}. Note that α cannot be equal to zero because of κ q ′ = t < w q ′ . Let us consider the continuously differentiable function ϕ w : κ → κw + αq ′ κ q ′ -1 defined on (0, ∞). Since ϕ w (0) = -w < 0, lim κ→∞ ϕ w (κ) = ∞ and ϕ w is strictly nondecreasing, there exists a unique κ * w > 0 such that ϕ w (κ * w ) = 0. If we now take w < v, we have

With ϕ v being strictly nondecreasing, we thus obtain κ * w < κ * v . The desired conclusion stems from the application of the previous result to the stationarity condition of κ.

A. Proofs

Based on the two previous lemmas, we are now in position to present our counterexample:

Proposition 15 (Counterexample)

Let . be a ℓ q -norm, with q / ∈ {1, 2, ∞} and q ′ 1/(1-q -1 ). Let us consider G = {g, h}, with g ⊆ h ⊆ {1, . . . , p} and |g| > 1. Let u be a vector in R p that has at least two different nonzero entries in g, i.e., there exists (i, j) in g × g such that 0 < |u i | < |u j |. Let us consider the successive projections ξ g Π . * ≤tg (u |g ) and ξ h Π . * ≤th (u |hξ g ) with t g , t h > 0 satisfying u |g q ′ > t g and u |h q ′ > t g + t h . Then, the conclusion of Lemma 3 does not hold.

Proof. We apply the same rationale as in the proof of Lemma 38. Writing the stationarity conditions for ξ g and ξ h , we have for all j in g ξ g j + αq ′ (ξ g j ) q ′ -1u j = 0, and ξ h j + βq ′ (ξ h j ) q ′ -1 -(u jξ g j ) = 0, (A.29)

with Lagrangian parameters α, β > 0. We now proceed by contradiction and assume that ξ g = Π . * ≤tg (u |gξ h |g ). According to Lemma 37, there exists ρ > 0 such that for all j in g, ξ h j = ρξ g j . If we combine the previous relations on ξ g and ξ h , we obtain for all j in g, ξ g j = C(ξ g j ) q ′ -1 , with C q ′ (αβρ q ′ -1 ) ρ .

If C < 0, then we have a contradiction, since the entries of ξ g and u |g have the same signs. Similarly, the case C = 0 leads a contradiction, since we would have u |g = 0 and u |g q ′ > t g . As a consequence, it follows that C > 0 and for all j in g, ξ g j = exp log(C) 2-q ′ , which means that all the entries of the vector ξ g g are identical. Using Lemma 38, since there exists (i, j) ∈ g × g such that u i < u j , we also have ξ g i < ξ g j , which leads to a contradiction.